Clustered Federated Spatio-Temporal Graph Attention Networks for Skeleton-Based Action Recognition

Abstract

Federated learning (FL) for skeleton-based action recognition remains underexplored, particularly under strong client heterogeneity where regular FedAvg tends to cause client drift and unstable convergence. We introduce Clustered Federated Spatio-Temporal Graph Attention Networks (CF-STGAT), a clustered FL framework that leverages attention-derived spatio-temporal statistics from local STGAT models to dynamically group clients and perform attention-weighted inter-cluster fusion that gently align cluster models. Concretely, the server periodically extracts multi-head parameter-based attention descriptors, normalizes and projects them via PCA, and applies K-means to form clusters; a global reference is then computed by attention–similarity weighting and used to regularize each cluster model with a lightweight fusion step. On NTU RGB+D 60/120(NTU 60/120), CF-STGAT consistently outperforms strong FL baselines with the STGAT backbone, yielding absolute top-1 gains of +0.84/+4.09 (NTU 60, X-Sub/X-Setup) and +7.98/+4.18 (NTU 120, X-Sub/X-Setup) over FedAvg, alongside smoother per-client trajectories and lower terminal test loss. Ablations indicate that attention-guided clustering and inter-cluster fusion are complementary: clustering reduces within-group variance whereas fusion limits cross-cluster divergence. The approach keeps local training unchanged and adds only server-side statistics and clustering.

1. Introduction

Skeleton-based human action recognition (HAR) is an active area in computer vision with applications in intelligent surveillance and robotic perception [1,2]. Given a sequence of 3D skeletal joints, the goal is to predict the action label by jointly modeling spatio-temporal configurations and dynamics. Early CNN/RNN approaches often treated skeletons as images or plain sequences and thus underutilized the explicit skeletal graph structure. Graph neural networks (GNNs) address this by operating on graph-structured data; the landmark ST-GCN [3] constructs a spatio-temporal skeleton graph and achieves strong centralized performance. Also, recent work [4] has demonstrated the effectiveness of spatial–temporal graph convolutional networks for skeleton-based activity recognition in practical applications. Subsequent advances enrich spatial topology modeling and temporal expressiveness through two-stream or channel-wise variants and multi-scale graph designs [5,6,7]. To capture long-range interactions across distant joints and timesteps, the Spatio-Temporal Graph Attention Network (STGAT) provides interpretable attention patterns across joints and time, yielding compact structural descriptors [8]. However, most methods are developed and evaluated in centralized settings and do not address privacy constraints.

In practice, skeleton data are compartmentalized between devices or institutions, and privacy rules impede centralization. Federated learning (FL) enables collaborative training without sharing raw data, thereby preserving privacy and supporting personalization [9]. A baseline method is established by integrating STGAT with the standard FedAvg algorithm, referred to as Fed-STGAT (see Figure 1). This configuration preserves data privacy by sharing model parameters, thereby keeping the raw skeleton data streams on the client devices. Nevertheless, non-IID heterogeneity which span subjects, viewpoints, and recording setups induces client drift and unstable convergence when updates are naively averaged. In skeleton HAR the effect is exacerbated: the same action may manifest distinct dynamics across subjects and capture setups, inflating variance and degrading global averaging. Existing FL-HAR and clustered FL partially alleviate these issues. Prior efforts with regularized or contrasting objectives (e.g., FedProx [10], MOON [11]) constrain local training, while clustered FL (e.g., FedCLAR [12]) groups similar clients before aggregation to reduce cross-client variance and stabilize optimization. However, few methods explicitly leverage attention statistics a privacy-friendly, structure-aware signals for clustering and coordination. Compared to sample-level features, an attention descriptor is a low-dimensional statistic derived from the trained model that summarizes cross-joint and cross-time importance patterns and aligns naturally with the skeleton graphs.

We therefore propose Clustered Federated Spatio-Temporal Graph Attention Networks (CF-STGAT). Clients train STGAT locally; the server periodically extracts lightweight multi-head attention statistics and performs K-means to obtain several structure-aware, homogeneous clusters. Beyond standard within-cluster aggregation, we compute a global reference by attention–similarity weighting (via cosine similarity between cluster-level attention centroids and a global prototype) and apply a lightweight fusion to the intra-cluster aggregates. This design preserves personalization while mitigating cross-cluster drift and adds only minimal server-side statistical and clustering over-head.

The contributions of this paper are summarized as follows:

• We introduce an attention–descriptor-driven federated clustering paradigm for skeleton HAR, turning spatio-temporal attention into a privacy-friendly client representation for dynamic cluster assignment.
• We propose an attention–similarity-weighted cross-cluster fusion that balances personalization and consistency via a lightweight update, without changing the client-side training pipeline.
• We perform comprehensive experiments on NTU 60/120 with X-Sub/X-Setup, benchmarking against FedAvg, FedProx, MOON, and clustered baselines FedCLAR. CF-STGAT achieves consistent accuracy gains and smoother convergence.

2. Related Work

2.1. Spatio-Temporal Model for Skeleton-Based Action Recognition

Modeling spatio-temporal dependencies is central to skeleton-based action recognition. Beyond early CNN/TCN pipelines [13,14,15], recent 3D-CNNs designs revisit skeleton encoding with task-specific representations, for example, PoseConv3D [16] encodes poses as 3D heatmap volumes and achieves strong cross-dataset robustness. Graph-based methods continue to advance topology modeling: InfoGCN [17] integrates an information-bottleneck objective with attention-based graph convolution to infer context-dependent topology, HD-GCN hierarchically decomposes skeleton graphs to capture semantically meaningful edges [18], ref. [19] reviewed skeleton graph neural network-based human action recognition, and BlockGCN [20] redefines topology awareness via block-wise graph convolutions with improved efficiency. Transformer-style models further extend receptive fields, FG-STFormer [21] balances focal/local and global attention, STAR-Transformer employs cross-attention to fuse spatio-temporal cues [22], and SkateFormer partitions joints and frames to efficiently model skeletal–temporal relations [23]. While attention improves global dependency modeling, its computational cost can be high, efficient self-attention and local dynamic STGAT reduce over-head while maintaining accuracy [8,24]. Recent surveys synthesize the field’s shift from RNN/LSTM to GCN/GAT/Transformer families and highlight the need for structure-aware, efficient, and privacy-preserving learning, which is the target setting of our framework CF-STGAT [25,26].

2.2. Federated Learning for Skeleton-Based Action Recognition

Federated learning (FL) enables privacy-preserving collaboration by keeping raw data on device and exchanging only model updates, thereby reducing both data-exposure risk and the costs of centralized collection; in practice, it is often paired with secure aggregation and lightweight communication protocols, making FedAvg [27] a widely adopted, communication-efficient baseline for deep models across modalities [28,29,30]. To address non-IID data, regularized and contrastive formulations improve robustness to statistical heterogeneity, FedProx [10] augments the local objective with a proximal term to constrain client drift under variable local steps, while MOON [11] introduces a model-contrastive objective that aligns client and global representations to reduce drift. Variance-reduction and normalization strategies such as SCAFFOLD and FedNova further correct client-specific bias and the objective inconsistency induced by unbalanced local updates [31,32], yet these methods remain structure-agnostic to skeleton topology. Furthermore, clustered and personalized schemes address client heterogeneity that is typical in skeleton HAR, where agglomerative clustered FL groups clients by update similarity to stabilize training [12,33], while structured FL via clustered additive modeling provides a principled formulation to reconcile cluster-wise personalization with global coordination [34]; however, they rely on generic similarity measures that do not explicitly reflect spatio-temporal skeleton dependencies. When skeleton streams and their topologies are explicitly modeled as graphs, federated graph learning (FGL) [35] offers task-tailored aggregation and topology-aware updates, recent systems and algorithms demonstrate gains by combining global coordination with local graph-specific objectives [36,37]. Related cross-silo variants such as vertical FL and split learning are also relevant when different parties hold complementary features or wish to offload parts of the network; recent surveys and studies document privacy risks and corresponding defenses, as well as practical split feature-partitioned training pipelines that can be combined with FL for spatio-temporal data [38,39,40]. In contrast, we adopt FedAvg as the baseline privacy boundary and inject structure awareness through leveraging parameter-based attention features from STGAT to drive clustering and attention–similarity-weighted fusion, achieving cluster-wise personalization while gently regularizing cross-cluster drift.

3. Methods

3.1. Preliminary

We adopt STGAT as the local model due to its strong empirical performance in capturing short-term features for skeleton sequences via spatio-temporal patterns. Below, we formalize the graph construction and summarize the network components relevant to our method.

3.1.1. Spatio-Temporal Graph Building

Formally, the human skeleton graph is always represented as $G=(\mathcal{V},A)$, where $\mathcal{V}=\{v_1,v_2,\dots ,v_N\}$ denotes skeleton joints and $A\in {\mathbb{R}}^{N\times N}$ is the adjacency matrix. $A_{i,j}\ne 0$ indicates a connection between joint $v_i$ and $v_j$, otherwise $A_{i,j}=0$ without connections. Regarding the input, the action sequences are characterized by a set of node features, represented as a feature tensor $X\in {\mathbb{R}}^{C\times T\times N}$ where each joint is described by a C dimensional feature vector across T frames. To enhance the capture of local cross-spacetime joint relationships without necessitating multiple transmissions, the construction of a spatio-temporal graph is proposed in which joints within a localized cross-spacetime neighborhood are represented as nodes. These nodes are connected by edges not only to other nodes within their original spatio graph but also to their counterparts in adjacent temporal frames. For a given time point t and window size $\tau \in {\mathbb{N}}^{*}$, the concatenation of features within the time window can be expressed as $X_{\tau }^t=[X_{:,t-{\displaystyle \frac{\tau }{2}},:};X_{:,t-{\displaystyle \frac{\tau }{2}}+1,:};\dots ;X_{:,t+{\displaystyle \frac{\tau }{2}},:}]\in {\mathbb{R}}^{C\times \tau \times N}$. Then, a time-aware adjacency is defined as $\widetilde{A_{\tau }^t}=\left[\widetilde{A_1^t};\dots ;\widetilde{A_{\tau }^t}\right]\in {\mathbb{R}}^{\tau \times N\times N}$.

3.1.2. STGAT Network

As is widely recognized, for a graph G, with the node feature matrix X serving as the input, the formulation for a new feature matrix H derived from a layer within a graph convolutional network (GCN) at each timestep can be described by

$$H=\sigma \left({\widehat{D}}^{-{\displaystyle \frac{1}{2}}}\widehat{A}{\widehat{D}}^{-{\displaystyle \frac{1}{2}}}XW\right),$$

(1)

The degree matrix $\tilde{D}$ corresponds to $\tilde{A}$ and is a diagonal matrix whose elements represent the sum of the connections, including self-loops, that incorporate its own features into the calculation. Also, W typically represents the trainable weight matrix, and $\sigma$ denotes a non-linear activation function. In parallel to the development of the spatio-temporal graph, by integrating $\widetilde{A_{\tau }^t}$ into graph operations, the output for each timestep input $X_{\tau }^t$ can be derived as

$$H_{\tau }^t=\sigma \left(\widetilde{A_{\tau }^t}{X}_{\tau }^{t}W\right),$$

(2)

Furthermore, STGAT utilizes multi-head attention modules to learn various kinds of edge weights. Specifically, independent self-attention modules S are employed to discern diverse graph structures. Consequently, the aggregated output from all heads across various spatio-temporal graphs can be expressed as

$$Y^t=\sigma \left({\displaystyle \frac{1}{S}}\sum _{s=1}^S{\widehat{A}}_{\tau }^t{,}_s{X}_{\tau }^t{W}_s\right),$$

(3)

where ${\widehat{A}}_{\tau }^t{,}_s$ denotes the unique spatio-temporal graph computed at timestamp t, with a temporal length of $\tau$ for the $s_{th}$ head.

3.2. Fed-STGAT

As illustrated in Figure 1, the structure comprises N distributed clients and a central aggregation server. Each client trains a local STGAT on private data and uploads model parameters $\mathrm{\Theta }$ after local updates. The server aggregates the received parameters and broadcasts the global model for the next round, repeating until convergence or a fixed communication budget.

3.2.1. Local Updates

For each client $i\in N$, at the beginning of each communication round, the local STGAT model is initialized with the global model parameters ${\mathrm{\Theta }}^t$ received from the server. The model is then trained on the client’s private dataset using multiple steps of SGD to perform parameter updates. The detailed procedure is as follows:

$${\mathrm{\Theta }}_i^{t+1}={\mathrm{\Theta }}^t-\eta ·{\nabla }_{\mathrm{\Theta }}\mathcal{L}\left({\mathrm{\Theta }}^t;D_i\right).$$

(4)

where $\eta$ is the learning rate and ${\nabla }_{\mathrm{\Theta }}\mathcal{L}\left({\mathrm{\Theta }}^t;D_i\right)$ is the set of gradient updates of the local dataset on client i. Upon completion of the local training, each client uploads its updated parameters to the central server for aggregation.

3.2.2. Aggregation

The server performs global model aggregation when receiving the locally updated model parameters ${\left\{{\mathrm{\Theta }}_i^t\right\}}_{i=1}^N$ from N participating clients. The global model is updated on FedAvg as follows:

$${\mathrm{\Theta }}^{t+1}=\sum _{i=1}^N{\displaystyle \frac{n_i}{n}}{\mathrm{\Theta }}_i^t,$$

(5)

where $n_i$ denotes the number of the local data samples on client i, and n is the total number of samples from all selected clients. The updated global parameters are redistributed to clients for the next round.

3.2.3. Limitation of Fed-STGAT

Under non-IID partitions, FedAvg can induce client drift and unstable convergence. The lack of structure-aware aggregation and personalization further limits cross-subject generalization. Thus, we introduce dynamic clustering in Section 3.3 and a light fusion step in Section 3.4 to mitigate drift and reduce inter-client variance.

3.3. Dynamic Cluster Adjustment

The approach extends FedAvg by grouping clients dynamically into clusters and training cluster-specific models. To obtain a structure-aware similarity, we extract attention-layer parameters from each client’s STGAT as privacy-preserving features that reflect spatio-temporal patterns. Every $I_c$ rounds, the server reclusters participating clients based on these features. Before clustering, we normalize the feature vector of each client ${\mathbf{a}}_i=[a_{i1},\dots ,a_{im}]$ by Z-score, then we fit a PCA model on the normalized attention vectors to reduce the dimensionality from m to d automatically by retaining 90% of the cumulative explained variance. This yields low-dimensional descriptors that preserve most of the information in the original attention parameters while reducing noise and improving the efficiency and stability of the subsequent clustering step. Having assembled the matrix $\tilde{\mathbf{A}}=[{\tilde{\mathbf{a}}}_1,\dots ,{\tilde{\mathbf{a}}}_n]$, the empirical covariance matrix is computed as $\Sigma ={\displaystyle \frac{1}{n}}{\tilde{\mathbf{A}}}^{\top }\tilde{\mathbf{A}}$. Solving the eigenproblem $\Sigma {\mathbf{U}}_d={\mathbf{U}}_d{\mathrm{\Lambda }}_d$ yields the top-deigenvectors ${\mathbf{U}}_d=[{\mathbf{u}}_1,\dots ,{\mathbf{u}}_d]$ associated with the largest eigenvalues in ${\mathrm{\Lambda }}_d$. Each normalized vector ${\tilde{\mathbf{a}}}_i$ is then projected as

$${\mathbf{z}}_i={\mathbf{U}}_d^{\top }{\tilde{\mathbf{a}}}_i,$$

(6)

yielding the final d-dimensional clustering feature for client i. If the current round is one of the designated clustering intervals $I_c$, the server uses the collected feature vectors from all participating clients to recompute the client clusters. If a client is assigned to another cluster, it means the characteristics of its model have shifted or the clustering decision boundary has changed.

We adopt a K-means approach for the clustering algorithm due to its simplicity and efficiency in the federated setting, partitioning the clients into C clusters by minimizing the within-cluster variance in the feature space. As shown in Figure 2, the outcome is a set of clusters $C_1,C_2,\dots ,C_C$, where clients within each cluster have high internal similarity in their attention features ${\mathbf{z}}_i$. After clustering, each cluster $C_k$ is associated with its own sub-model. Clients do not remain tied to a single cluster for the entire training; they are allowed to change clusters if their attention-based representation shifts or the clustering boundaries are refined over time.

Complexity note. Let N denote the number of participating clients in a clustering step and m the dimensionality of the raw attention features. Computing the covariance matrix and its eigendecomposition for PCA on the $N\times m$ matrix $\tilde{\mathbf{A}}$ has complexity $\mathcal{O}(N{m}^2+m^3)$. Since m is a small, fixed hyperparameter in our setup, this cost is effectively linear in N. Performing K-means in the reduced d-dimensional space has complexity $\mathcal{O}\left(NCId\right)$, where C is the number of clusters and I is the number of iterations. In practice, m, d, C, and I are all small constants, and clustering is only performed every $I_c$ rounds on the subset of active clients, so the server-side over-head is negligible compared to local STGAT training and communication.

3.4. Cluster Federated Aggregation

While clustered federated learning provides personalization by clustering models to groups of similar clients, it is important to prevent these cluster-specific models from diverging too far from each other, which could harm overall generalization. Therefore, for Cluster $C_k$, the server forms a size-weighted average ${\overline{\mathbf{w}}}_{C_k}$ of the received client models; for inter-cluster regularization aggregation, to maintain global coherence between clusters and prevent cluster models from diverging excessively.

We construct a global attention prototype ${\mathbf{a}}_G$ from cluster attention centroids and compute cosine similarities $s_k=cosine({\mathbf{a}}_k,{\mathbf{a}}_G)$ between cluster $C_k$. Attention-aware weights

$${\alpha }_k={\displaystyle \frac{N_k{s}_k}{{\sum }_j{N}_j{s}_j}},\sum _k{\alpha }_k=1.$$

(7)

which yield a reference model:

$${\mathbf{w}}_G^{\left(t\right)}=\sum _k{\alpha }_k{\overline{\mathbf{w}}}_{C_k}^{\left(t\right)}.$$

(8)

Each cluster model is then updated by a light fusion:

$${\overline{\mathbf{w}}}_{C_k}^{(t+1)}\leftarrow (1-\gamma ){\overline{\mathbf{w}}}_{C_k}^{\left(t\right)}+\gamma {\mathbf{w}}_G^{\left(t\right)},$$

(9)

with $\gamma$ controlling inter-cluster regularization. Instead of treating the fusion coefficient $\gamma$ as a fixed hyperparameter, we adapt it automatically at each communication round based on the discrepancy between the previous global model and the current cluster models. In practice, this makes $\gamma$ increase when cluster models deviate more from the global model (encouraging stronger fusion) and decrease when the models are already similar (allowing more specialization). This step preserves local training pipelines while aligning clusters through periodic server-side coordination to effectively balance model personalization with overall consistency in a federated learning setting.

The above formulation can be understood as follows. The server clusters clients through similar attention-based descriptors which form the uploaded spatio-temporal structure, so that each cluster-wise model is trained on a group of clients that share similar dependency patterns. The fusion regularization term softly couples these cluster-specific models, allowing them to share transferable knowledge while still adapting to the particular characteristics of each cluster. In this way, the method balances global sharing and local specialization in a structure-aware manner.

4. Experimental Results

4.1. Datasets

NTU 60 [41]. A widely adopted benchmark for 3D skeleton-based human action recognition. Captured with Microsoft Kinect v2 sensors in a controlled laboratory setting, it comprises 56,880 samples across 60 action classes performed by 40 subjects. Each sample encodes a sequence of 25 skeletal joints, with each joint represented by its (x, y, z) coordinates in 3D space. The dataset offers two standardized evaluation protocols: cross-subject (X-Sub), which trains on half of the subjects and tests on the other half, and cross-view (X-View). We follow the standard X-Sub split; for X-View, we instead use a setup-based split for stress-testing distribution shift.

NTU 120 [42]. An extended version of NTU 60 that includes 120 action classes, each performed twice by 60 subjects (40 male, 20 female) across four distinct recording setups. NTU 120 retains the same data modalities as NTU 60, offering synchronized RGB and depth streams along with the core 3D skeleton data (25 joints per frame). NTU 120 introduces the Cross-Setup (X-Setup) protocol: even-ID setups for training and odd-ID setups for testing. Each setup features a distinct location and background and, as in NTU 60, it employs three horizontally spaced cameras to capture each action from three different side-view angles simultaneously.

We also adopt a federated-by-dataset setting, where federated clients are constructed from the underlying dataset structure. Concretely, for both NTU 60 and NTU 120, each $(\mathrm{subject},\mathrm{setup})$ pair thus defines a federated client, inducing a naturally non-IID label and setup distribution across clients. The statistics of federated data partitions are shown in

Table 1. Statistics of the federated data partitions on NTU 60 and NTU 120.

Dataset	Protocol	Classes	Train Clients	Test Clients
NTU 60	X-Sub	60	20	20
NTU 60	X-Setup	60	8	9
NTU 120	X-Sub	120	53	53
NTU 120	X-Setup	120	16	16

. The central server evaluates on a global test set defined over subject–setup pairs. For NTU 60, in addition to the standard X-Sub, we define a cross-setup protocol (NTU 60 X-Setup) to stress-test distribution shifts across recording setups: following the rule of NTU 120 X-Setup, sequences from even-numbered setups are used for training and those from odd-numbered setups for testing.

4.2. The Details of Implementation

All experiments are conducted using the PyTorch 1.12.1 on a single NVIDIA RTX 4090 GPU. The backbone network is based on STGAT, consisting of eight graph attention layers, each with eight attention heads. We use batch size 32 and weight decay $5\times {10}^{-4}$. From each sequence, we first sample 150 frames and then randomly crop to 128 frames for both training and testing to match sequence length. Within FL, each client performs $E=1$ local epoch per communication cycle. The initial learning rate is set to 0.1 and is reduced by a factor of 10 at communication rounds $\{60,90,120\}$, respectively. The total number of communication rounds is fixed at $R=200$. For clustering, the interval is $I_c=5$ communication rounds. We set $C=4$ on NTU 60 and $C=6$ on NTU 120. The fusion regularization coefficient $\gamma$ is selected from the set $\{0.1,0.2,0.5\}$ based on a small grid search.

4.3. Experiments Results and Discussion

4.3.1. General Results

Table 2. Overall comparison on NTU RGB+D 60/120 under X-Sub and X-Setup. We report top-1 accuracy (%) on each protocol. Bold denotes the best. Values in parentheses indicate the absolute margin vs. FedAvg (baseline row marked with *).

Model Backbone: STGAT	NTU 60		NTU 120
	X-Sub (%)	X-Setup (%)	X-Sub (%)	X-Setup (%)
Fed-STGAT *	89.72	86.46	72.35	78.71
FedProx	89.31 (−0.41)	87.49 (+1.03)	75.36 (+3.01)	80.65 (+1.94)
MOON	88.01 (−1.71)	86.96 (+0.50)	73.31 (1.04)	79.86 (+1.15)
FedCLAR	89.88 (+0.16)	87.24 (+0.78)	75.55 (+3.20)	80.83 (+2.12)
CF-STGAT (our)	90.56 (+0.84)	90.55 (+4.09)	80.33 (+7.98)	82.89 (+4.18)

presents a comparative performance analysis on NTU 60/120 under the X-Sub and X-Setup protocols with STGAT as the backbone. (1) CF-STGAT achieves the best overall performance, surpassing FedAvg by +0.84 (NTU 60, X-Sub), +4.09 (NTU 60, X-Setup), +7.98 (NTU 120, X-Sub), and +4.18 (NTU 120, X-Setup), yielding an average gain of +4.27 percentage points (pp). The largest single-cell improvement occurs on NTU 120 (X-Sub) (+7.98). (2) CF-STGAT also outperforms FedCLAR across all settings, with gains of +0.68, +3.31, +4.78, and +2.06 pp on the same four protocol–dataset combinations (mean +2.71 pp). Notably, the margin is largest on NTU 120 (X-Sub) (+4.78). (3) CF-STGAT Improvements over FedAvg are larger on NTU 120 (mean +6.08 pp) than on NTU 60 (mean +2.47 pp), suggesting greater benefits under the more heterogeneous setting. Overall, relative to optimizer-based baselines (FedAvg, FedProx, MOON) and clustered FL (FedCLAR), CF-STGAT attains the highest accuracy across all protocols, with robust gains on NTU 120.

4.3.2. Clustering Stability Analysis

To analyze how client cluster assignments evolve over communication rounds, we measure round-to-round agreement between consecutive reclustering checkpoints using the Adjusted Rand Index (ARI) and report a churn rate (the fraction of clients that change clusters between two checkpoints). Across NTU 60 X-Sub configurations, the reclustering dynamics stabilize after a short warm-up. As shown in

Table 3. Round-to-round clustering stability on NUT 60 (X-Sub). Values are mean ± std over consecutive checkpoint pairs.

Setup	ARI	Churn
$I_c=5$, $C=4$	0.846 ± 0.213	0.040 ± 0.056
$I_c=5$, $C=3$	0.674 ± 0.326	0.064 ± 0.077
$I_c=3$, $C=4$	0.750 ± 0.236	0.064 ± 0.064

, the $I_c=5$, $C=4$ variant yields the strongest consistency (ARI = 0.846 ± 0.213) and the lowest churn (0.040 ± 0.056), outperforming $C=3$ (ARI = 0.674 ± 0.326; churn = 0.064 ± 0.077) and $I_c=3$ (ARI = 0.750 ± 0.236; churn = 0.064 ± 0.064). This indicates that too small C conflates heterogeneous clients, producing fuzzy boundaries. Conversely, when C is too large, semantically similar clients are over-partitioned, leading to cluster fragmentation and elevated churn. Also, overly frequent reclustering (smaller $I_c$) can introduce transient boundary oscillations, even if the fraction of migrating clients is moderate, the induced pairwise disagreements depress ARI.

In addition, we visualize the row-normalized average transition matrix to reveal where changes concentrate. The transition matrices in Figure 3 are strongly diagonal on both datasets, confirming persistent cluster identities. On NTU 120, clusters 1–4 exhibit near-perfect self-transition (≥0.98), with localized drift from $0\to 1$ (0.24) and $5\to 1$ (0.12), that means a single fuzzy boundary while the rest remain stable. On NTU 60, overall stability is also high, but the boundary between clusters shows more activity: $3\to 1\approx 0.26$ and $2\to 1\approx 0.08$, consistent with the slightly lower ARI mean and the small but non-zero churn pulses. Overall, the table and heatmaps show that instability is concentrated at a few adjacent-cluster boundaries rather than reflecting global oscillations.

4.3.3. Effectiveness of Clustering

As shown in Figure 4, in the absence of clustering (treating all clients as a single group), the baseline method FedAvg and its regularized extensions FedProx and MOON exhibit slow and uneven convergence under non-IID partitions. Training a single global model with FedAvg on highly non-IID clients can induce large variance in the aggregated update, since local gradients are optimized for very different data distributions. This variance manifests as oscillatory behavior of the global model and unstable performance across communication rounds. Although MOON accelerates the initial rise, the jagged behavior remains, indicating that optimizer-level regularization alone is insufficient to eliminate client drift. Consistently, the test loss shown in Figure 5 decreases more slowly and stabilizes at a higher asymptote, particularly on NTU 120, where label and domain shifts are stronger. By contrast, our clustering mechanism groups clients with similar attention-based descriptors, i.e., similar spatio-temporal dependency patterns over the skeleton graph, so that each cluster-wise model is trained on a more homogeneous subset of clients. Within a cluster, local updates are better aligned and the variance of the aggregated update is reduced, which mitigates client drift and yields smoother training trajectories. The resulting cluster-specific models thus exhibit more stable convergence behavior than a single global model trained on all heterogeneous clients.

4.3.4. Effectiveness of Fusion

With the clustered-only method FedCLAR, most clients indeed benefit from more homogeneous aggregation; however, the persistent low-accuracy tail and residual jaggedness indicate that inter-cluster drift is not fully controlled. Compared with our approach, this limitation is reflected in a higher late-stage test loss (Figure 5). By contrast, CF-STGAT, clustering augmented with global fusion, produces tighter and smoother client trajectories, elevates the plateau height, and compresses the long tail (Figure 4). On the test set, fusion yields faster loss decay and a lower terminal loss, with the largest margin observed on NTU 120. Overall, these results show that our method complements clustering by gently yet effectively aligning clusters, thereby delivering additional accuracy and stability beyond clustered-only federated learning.

4.3.5. Discussion and Future Work

A limitation of the present study is that we do not include explicit empirical comparisons with some recent personalized federated learning methods, such as FedPer [43] and FedRep [44]. These approaches adopt a complementary perspective: they typically share a global feature extractor while learning client-specific heads or representations, thereby personalizing predictions at the client level without explicitly modeling relationships among clients. In contrast, our method focuses on structure-aware clustering of clients based on attention-derived descriptors and trains cluster-specific STGAT models to capture shared spatio-temporal patterns within each group. This design is orthogonal to head- or representation-based personalization and could, in principle, be combined with FedPer/FedRep-style techniques by applying them within each discovered cluster. We leave a systematic empirical comparison and such hybrid combinations as important directions for future work. From a privacy perspective, the server only observes compressed attention-based descriptors aggregated over each client, rather than raw data or per-sample activations; while this reduces the risk of direct data leakage, our method is complementary to secure aggregation and differential-privacy mechanisms, and a formal privacy analysis of such descriptors is left for future work.

5. Conclusions

In summary, this work introduced CF-STGAT, a clustered federated framework for skeleton-based action recognition that integrates multi-head spatio-temporal attention with dynamic client clustering and a lightweight, attention-weighted inter-cluster fusion. The approach preserves cluster-level personalization while providing gentle cross-cluster alignment. Empirically, CF-STGAT delivers higher accuracy, smoother and faster convergence, and consistently lower terminal test loss, with the advantages becoming more pronounced under stronger heterogeneity. The method leaves client-side training unchanged and adds only modest server-side statistics and clustering.