Gaussian Topology Refinement and Multi-Scale Shift Graph Convolution for Efficient Real-Time Sports Action Recognition

Abstract

Skeleton-based action recognition is a critical technology for intelligent sports analysis. Although the human skeletal structure exhibits inherent bilateral symmetry, sensor noise on resource-constrained edge devices frequently induces geometric distortion and topological asymmetry. Consequently, achieving a balance between high accuracy and real-time performance remains a significant challenge. To this end, we propose EMS-GCN, an Efficient Multi-scale Shift Graph Convolutional Network that integrates geometric priors. Specifically, we design a Gaussian kernel-driven topology refinement module to mitigate structural noise inherent in sensor data. By leveraging geometric symmetry and Gaussian distances among nodes, this module dynamically constrains graph topology learning, thereby effectively rectifying the structural asymmetry and ambiguity induced by noise. Furthermore, we construct a Multi-scale Shift Linear Attention (MSLA) module to replace computationally intensive temporal convolutions. Leveraging temporal shift invariance, this module captures multi-scale contexts via parameter-free shift operations. Furthermore, we introduce a linear temporal attention mechanism to model global temporal dependencies with linear complexity, effectively resolving the information asymmetry inherent in long-range interactions. Finally, EMS-GCN incorporates a dual-branch attention structure to adaptively calibrate feature responses. Extensive experiments demonstrate that our model maintains high recognition accuracy with only 0.56 M parameters, representing a reduction of over 60% compared to mainstream baselines. These results validate the efficacy of leveraging geometric and temporal symmetries to enhance real-time sports analysis.

1. Introduction

The rapid evolution of Micro-Electro-Mechanical Systems (MEMS) and computer vision has established intelligent sports analysis as a cornerstone of modern competitive sports [1,2]. Human skeleton data inherently forms a topological graph structure characterized by high geometric symmetry, such as the bilateral symmetry of limbs. Acquired through motion capture sensors or pose estimation algorithms, skeleton data encodes human kinematic features in a compact topological form. Owing to its innate robustness against illumination changes, background clutter, and clothing textures, this modality is extensively applied in action diagnosis, tactical analysis, and AI-assisted scoring [3,4]. However, unlike traditional offline analysis, real-time sports scenarios impose stringent dual requirements on the response speed and recognition accuracy of sensing systems. This challenge is particularly pronounced on resource-constrained edge computing devices, such as smart cameras and embedded analysis terminals. Consequently, effectively leveraging spatiotemporal symmetry and structural information of skeletons within millisecond-level time windows remains a critical problem that demands an urgent solution [5,6].

Graph Convolutional Networks (GCNs) have emerged as the standard framework for skeleton-based action recognition, effectively replacing traditional CNN and RNN architectures for processing non-Euclidean data [7]. Leading approaches, such as CTR-GCN [8], utilize dynamic adjacency matrices to bypass the limitations of physical body connections and enhance spatial modeling. However, existing GCN architectures still confront three critical bottlenecks from the perspective of symmetry and structural stability.

First, the structural uncertainty inherent in sensor data undermines latent geometric symmetry. During the acquisition of high-speed movements, such as gymnastics turns and fencing, limitations in sensor sampling rates or occlusion often cause skeletal nodes to exhibit high-frequency jitter or drift. Consequently, the intrinsic topological symmetry of the human body is disrupted [9]. Existing dynamic graph learning methods lack geometric prior constraints, making them susceptible to generating spurious connections that disrupt structural balance under noise interference. Consequently, leveraging spatial distance symmetry among nodes to calibrate dynamic topology is essential for enhancing model robustness.

Second, there is an inherent contradiction between long-term temporal dependency modeling and computational efficiency. Sports actions involve complex temporal evolution, ranging from instantaneous bursts like jumping to prolonged tactical sequences. Traditional methods primarily rely on Temporal Convolutional Networks (TCNs) with fixed kernel sizes [10,11]. These fixed structures, however, fail to capture multi-scale receptive fields necessary for diverse temporal dynamics. Although Transformers effectively capture global dependencies [12], the quadratic computational complexity $O\left(T^2\right)$ of self-attention mechanisms restricts real-time deployment on resource-constrained sensor terminals. This limitation creates an urgent demand for an efficient mechanism that combines linear complexity with global context modeling.

Third, the feature representation capability of existing lightweight designs remains limited. While approaches like Shift-GCN [13] adopt shift operations to replace convolutions for efficiency, they rely on simple element-wise summation for feature fusion. Such mechanisms lead to the confusion and loss of fine-grained motion details between adjacent frames in high-speed sports videos, failing to meet the rigorous demands of precise action analysis.

To address these challenges, we propose the Efficient Multi-scale Shift Graph Convolutional Network (EMS-GCN), a framework tailored for real-time sensor data processing. This method achieves high-precision spatiotemporal modeling with minimal computational cost. First, we design a Gaussian Topology Refinement Module (GTRM). This component calculates inter-node spatial distances using learnable Gaussian kernels. By injecting strong geometric priors into dynamic topology generation, the module effectively suppresses structural ambiguity stemming from sensor noise. Second, to capture both long-term and short-term temporal features with low latency, we develop a Multi-scale Shift Linear Attention (MS-LTA) module. Specifically, we utilize temporal shift operations with diverse dilation rates to construct multi-scale receptive fields. To prevent feature aliasing, we adopt a concatenation strategy rather than simple summation. Uniquely, we introduce a linear temporal attention mechanism that employs the kernel trick to reduce global temporal modeling complexity to a linear level. This design maintains high inference speeds while achieving long-range dependency modeling comparable to Mamba-like architectures. Finally, a dual-branch attention module adaptively calibrates feature responses for critical frames and joints.

The main contributions of this work are summarized as follows:

• We propose EMS-GCN, a highly efficient skeleton-based action recognition network tailored for edge sensor applications. This model reconstructs the underlying spatiotemporal interaction mechanisms. As illustrated in Figure 1, EMS-GCN achieves high-precision real-time inference with only 0.56 M parameters.
• We design a Gaussian topology refinement module. Distance-based symmetry constraints are utilized. Topological structure uncertainty induced by acquisition jitter and drift in sensor data is effectively resolved. Geometric stability of the skeleton graph is restored.
• We propose a temporal modeling scheme integrating multi-scale shift and linear attention. By employing a feature concatenation strategy, this approach eliminates the temporal aliasing risks inherent in traditional shift operations. Furthermore, it achieves precise capture of global temporal dependencies with linear computational complexity, striking an optimal balance between long-range context modeling and real-time response requirements.

2. Related Work

2.1. Skeleton-Based Action Recognition with GCNs

Graph convolutional networks have reshaped the paradigm of skeleton-based action recognition due to their effectiveness in modeling non-Euclidean data [5,14,15,16,17]. Yan et al. introduced ST-GCN, which represents the human skeleton as a spatio-temporal graph and applies graph convolution to capture spatial dependencies among joints as well as temporal dynamics, thereby reducing reliance on manually designed features [18]. In intelligent sports and wearable sensing systems, skeleton sequences are typically generated by pose estimation modules driven by sensors such as RGB-D cameras, mmWave radars, or IMUs. These sequences frequently suffer from artifacts including joint jitter, occlusion-induced missing data, and structural noise. Consequently, ensuring the robustness of topological modeling and the efficiency of real-time inference becomes a critical constraint. However, ST-GCN is constrained by predefined physical connection graphs, which limit its ability to model dependencies between joints without direct physical links, such as coordinated hand movements in clapping actions, and consequently restricts the representation of complex motions.

To overcome the limitations of fixed graph topologies, adaptive graph convolution methods have been developed. Shi et al. proposed 2S-AGCN, which learns data-driven graph topologies in an end-to-end manner and improves model flexibility [19]. This framework was further extended to a multi-stream adaptive architecture that integrates complementary cues such as bone length and motion velocity [20]. Li et al. introduced AS-GCN, which captures latent dependencies by inferring action links and employs an encoder–decoder structure to predict future poses [21]. Considering the asymmetry and directionality of human motion, directed graph neural networks have been introduced for skeleton-based action recognition. Directed graph models were adopted by Shi et al. and Korban et al. to represent asymmetric dependency relationships within skeletal data [22,23]. More recently, Li et al. proposed DD-GCN, in which a directed diffusion mechanism is employed to capture deeper contextual features at the vertex level [24]. In addition, DG-STGCN, proposed by Duan et al., achieves finer motion modeling by dynamically updating spatiotemporal graph structures [25]. Nevertheless, under conditions of sensor-induced structural noise and pose estimation errors, highly adaptive topologies often suffer from fluctuating adjacency relationships and semantic ambiguity. These structural inconsistencies compromise both cross-scenario generalization and stability.

With continued investigation, the extraction of more discriminative features and the optimization of computational topology have become central concerns. A context-aware graph convolution was developed by Zhang et al. to strengthen local representations through the integration of contextual information from all joints [26]. MS-GCN was proposed by Liu et al., in which a decoupled multi-scale aggregation strategy is adopted to reduce interference among features from different neighborhoods [27]. Under the assumption that distinct channels may exhibit different topological patterns, CTR-GCN was introduced by Chen et al. Channel-specific joint correlations are modeled through topology refinement along the channel dimension [28]. Peng et al. designed optimal GCN architectures using Neural Architecture Search (NAS). Chi et al. introduced InfoGCN, which enhances action representation robustness by incorporating information bottleneck theory and modality-agnostic meta-learning [29,30]. Lee et al. employed a hierarchical decomposition strategy to capture action features at multiple levels of abstraction [31]. Different from previous studies that prioritize modeling capacity, we focus on structural noise and topological ambiguity within sensor-based skeleton sequences. We introduce a Gaussian Topology Refinement Module integrated into the CTR-style dynamic topology framework. By learning Gaussian distance priors for joint pairs, this module imposes gating constraints on adjacency relationships, effectively mitigating noise-induced structural ambiguity and enhancing topological stability.

Although notable improvements in recognition accuracy have been achieved by the aforementioned approaches, most existing models rely on computationally intensive spatiotemporal convolution operations. This reliance results in increased parameter counts and substantial computational overhead, which limits their applicability in real-time scenarios. To mitigate this limitation, Shift-GCN was introduced by Cheng et al. [7]. In this method, spatial graph convolutions and temporal convolutions are replaced with zero-parameter feature shift operations, enabling a significant reduction in computational complexity while maintaining competitive performance. However, existing shift convolution approaches primarily focus on single-scale temporal modeling. Consequently, they often fail to simultaneously capture local details and long-term dependencies when processing rapid and rhythmically complex sports actions. While traditional self-attention mechanisms effectively model long-range dependencies, their substantial computational and memory overheads hinder low-latency deployment on edge sensing devices. This limitation drives us to develop an architecture that retains the efficiency of shift operations while flexibly accommodating multi-scale temporal dynamics. To this end, we integrate linear temporal attention into the multi-scale shift framework. This design captures global temporal dependencies with linear complexity, effectively reconciling the trade-off between efficiency and discriminability in real-time sports analysis.

2.2. Lightweight Graph Convolutional Networks

The increasing demand for deploying skeleton-based action recognition models on edge devices and real-time systems has driven recent efforts to reduce computational complexity and parameter size while maintaining performance [32,33,34]. Traditional graph convolutional networks rely on computationally intensive operations to model complex spatiotemporal dependencies, limiting their applicability in resource-constrained environments. To address this, lightweight GCN design paradigms have been explored, including efficient operator design, topology optimization, and neural architecture search.

Among various lightweight architectures, Shift-GCN [7] introduced the pioneering concept of shift graph convolution. By replacing computationally intensive spatial convolutions with parameter-free feature shift operations, this method preserves spatial receptive fields while significantly reducing computational overhead. Owing to its high parameter efficiency, this paradigm has attracted considerable attention and has motivated a series of subsequent studies. For example, Roy et al. proposed a knowledge-driven shift graph convolutional network that enhances feature representations through prior knowledge [35], while Lu et al. incorporated attention mechanisms to guide shift operations and improve the expressiveness of topological structures [36]. Chaudhuri and Bhattacharya integrated state space models with a U-shaped Shift-GCN to capture long-range dependencies in extended sequences [37]. While shift-based methods effectively reduce parameter counts, existing shift strategies are often constrained by fixed temporal receptive fields or simple summation-based fusion, which limits their ability to preserve fine-grained motion features in sports actions with complex temporal dynamics. Furthermore, the representation of long-term action structures is often constrained when relying on shift operations without effective global dependency modeling mechanisms.

Beyond shift operations, the pursuit of lightweight models has extended to network topology reconstruction and adaptive mechanisms. Feature propagation efficiency is improved in BlockGCN through a redefinition of topology-aware computation [8]. A lighter and faster multi-scale adaptive GCN was introduced by Jiang and Deng to balance accuracy with efficiency across scales [11]. Targeting embedded systems, an efficient recognition framework was developed by Noor et al., demonstrating the feasibility of deploying complex models on low-power devices [12]. To address the specific characteristics of sports scenarios, Wu et al. proposed ASTM-Net, which focuses on lightweight recognition of sports activities [38]. Wang et al. further reduced model redundancy by leveraging multi-view topology [39]. In automated design, Zhou et al. introduced HGNAS, which applies hardware-aware neural architecture search to identify graph structures suitable for edge devices [40]. Nevertheless, existing studies often struggle to balance temporal modeling capacity with computational efficiency. While approaches utilizing multi-scale or large-kernel convolutions introduce unavoidable latency, oversimplified designs tend to compromise the representation of mixed long- and short-term dynamics.

Although existing lightweight approaches have shown notable improvements on general action recognition benchmarks, challenges remain when handling high-speed and complex sports actions [41]. Methods that emphasize aggressive parameter reduction often struggle to capture multi-scale temporal dependencies spanning both short-term and long-term patterns. In contrast, approaches that introduce large-kernel convolutions or complex inference mechanisms to strengthen temporal modeling inevitably increase computational cost, which conflicts with the requirements of real-time analysis [42,43]. As illustrated in Figure 2, unlike existing approaches, EMS-GCN is designed to break the trade-off between accuracy and efficiency. For temporal modeling, we build upon parameter-free multi-scale temporal shifts, forming stable temporal context representations via a concatenation-based fusion strategy. We further integrate linear temporal attention to capture long-range dependencies. This design circumvents the prohibitive computational and memory access costs associated with standard self-attention on edge devices. In terms of topological modeling, we incorporate a Gaussian Topology Refinement Module (GTRM) into the CTR-based channel-wise dynamic graph framework. This module utilizes pairwise Gaussian distances of joint features to apply gating constraints on dynamic adjacencies, effectively suppressing structural noise and topological ambiguity stemming from sensor or pose estimation errors. Consequently, EMS-GCN captures both fine-grained action dynamics and global temporal dependencies with a compact model size of approximately 0.56 M parameters, enabling efficient and stable recognition for real-time sports analysis.

3. Proposed Method

3.1. Overall Architecture

In skeleton-based action recognition tasks, sensor data is typically modeled as a spatiotemporal graph structure. Formally, let $X\in {\mathbb{R}}^{N\times C\times T\times V}$ denotes the input skeleton sequence, where $N$, $C$, $T$, and $V$ represent the batch size, number of channels, sequence length, and number of nodes. Our objective is to design a lightweight and efficient network $F(·)$ to accurately map the input $X$ to the corresponding action labels $Y$.

We introduce EMS-GCN to alleviate the excessive computational burden of traditional Graph Convolutional Networks on edge devices and mitigate structural noise inherent in sensor data. As depicted in Figure 3, the framework consists of multiple stacked spatiotemporal processing blocks. Diverging from conventional GCNs, we fundamentally reconstruct both the spatial and temporal streams:

• Spatial Dimension: We introduce the Gaussian Topology Refinement Module (GTRM). By leveraging inter-node statistical distribution properties to constrain dynamic topology learning, this module addresses topological ambiguity caused by positional drift in sensor data.
• Temporal Dimension. We propose the Multi-scale Shift Linear Attention Module (MS-LTA). This component captures local temporal features via parameter-free shift operations and models global dependencies through a linear-complexity attention mechanism, effectively replacing computationally intensive standard temporal convolutions.

The overall forward inference process of EMS-GCN is shown in Algorithm 1.

Algorithm 1: Forward Procedure of the Gaussian-Refined CTR-GCN with Multi-Scale Shift and Linear Temporal Attention

Input: Skeleton tensor $\mathit{X}\in {\mathbb{R}}^{N\times C\times T\times V\times M}$, predefined graph partitions $\{{\mathit{A}}^{\left(k\right)}{\}}_{k=1}^K$

Output: Class logits $\widehat{\mathit{y}}\in {\mathbb{R}}^{N\times num\_class}$

1 Reorder $\mathit{X}$ to merge person and joint dimensions, then apply batch normalization over $(M,V,C)$;

2 Reshape normalized features into $\mathit{F}\in {\mathbb{R}}^{\left(NM\right)\times C\times T\times V}$;

3 for $\mathcal{l}=1$ to $6$ do

4 |  // Spatial graph reasoning with multi-subset CTRGC

5 |  $\mathbf{Y}\leftarrow 0$;

6 |  for $k=1$ to $K$ do

7 |  |  Project $\mathit{F}$ by $1\times 1$ convolutions to obtain relation features and node features;

8 |  |  Compute dynamic topology ${\tilde{\mathit{A}}}_{ctr}^{(k)}=Conv(\mathit{tanh}({\varphi }_1(\mathit{F})-{\varphi }_2(\mathit{F})))\cdot \alpha +\mathit{P}{\mathit{A}}^{\left(k\right)}$;

9 |  |  Compute Gaussian prior from temporally averaged features: $\mathit{G}=\mathit{exp}\left(-{\displaystyle \frac{d\left(\overline{\mathit{F}}\right)}{2{\sigma }^2}}\right)$;

10 |  |  Refine topology: ${\tilde{\mathit{A}}}^{\left(k\right)}={\tilde{\mathit{A}}}_{ctr}^{\left(k\right)}+\beta \left({\tilde{\mathit{A}}}_{ctr}^{\left(k\right)}\odot \mathit{G}\right)$;

11 |  |  Aggregate subset response: $\mathit{Y}\leftarrow \mathit{Y}+einsum({\tilde{\mathit{A}}}^{(k)},\psi (\mathit{F}))$;

12 |  end

13 |  Fuse subset responses with residual graph mapping and activation: $\mathit{F}\leftarrow ReLU(BN(\mathit{Y})+{Res}_{gcn}(\mathit{F}))$;

14 |  // Temporal modeling block

15 |  ${\mathit{F}}_{shift}\leftarrow MultiScaleShiftFusion\left(\mathit{F}\right)$;

16 |  ${\mathit{F}}_{lin}\leftarrow LinearTemporalAttention\left(\mathit{F}\right)$;

17 |  $\mathit{F}\leftarrow {\mathit{F}}_{shift}+{\mathit{F}}_{lin}$;

18 |  Apply dropout, channel-spatial attention, temporal downsampling (if stride $>1$), and residual fusion: $\mathit{F}\leftarrow ReLU\left(Pool\right(Attn\left(Drop\right(\mathit{F}\left)\right))+{Res}_{tcn}(\mathit{F}\left)\right)$;

19 end

20 Apply global average pooling over temporal and joint axes;

21 Average features across the person dimension $M$, then apply dropout;

22 Return logits $\widehat{\mathit{y}}=FC\left(\mathit{F}\right)$;

3.2. Gaussian-Refined Channel-Wise Topology Modeling

When processing sensor-acquired skeleton sequences, conventional adaptive adjacency matrices are susceptible to overfitting specific noise patterns in the training data. To enhance topological robustness, we integrate a Gaussian Topology Refinement Module (GTRM) [28] into the channel-wise topology refinement framework of standard GCNs, as illustrated in Figure 3.

Static Feature Aggregation: To capture inherent inter-node geometric relationships, we first extract static feature representations from the input. Considering that anatomical constraints remain stable despite variations in relative joint positions during actions, we perform aggregation along the temporal dimension.

$$\overline{X}={\displaystyle \frac{1}{T}}\sum _{t=1}^T{X}_{\mathrm{i}\mathrm{n}}\left[:,:,t,:\right]\in {\mathbb{R}}^{N\times C\times V}$$

(1)

where $t$ denotes the specific time frame index. By applying average pooling along the temporal dimension $T$, this equation aims to eliminate transient motion noise, thereby aggregating global static spatial features for subsequent topological modeling.

Gaussian Topological Priors: We compute Euclidean distances between node pairs to construct physically constrained topologies. Given arbitrary nodes $v_i$ and $v_j$ with feature vectors $x_i$, $x_j\in {\mathbb{R}}^{\mathrm{C}}$, we formulate the pairwise distance matrix $D\in {\mathbb{R}}^{\mathrm{N}\times \mathrm{V}\times \mathrm{V}}$ as follows:

$$D_{ij}={‖\overline{x_i}-\overline{x_j}‖}_2^2=\sum _{c=1}^C{\left({\overline{x}}_{i,c}-{\overline{x}}_{j,c}\right)}^2$$

(2)

Based on the distance matrix, we construct the topological prior $G_{map}$ using the Gaussian Radial Basis Function. To adapt to scale variations across different joints, we introduce a learnable bandwidth parameter $\mathsf{\Sigma }\in {\mathbb{R}}^{v\times v}$:

$$\left\{\begin{array}{c}{\sigma }_{ij}=max\left({\mathsf{\Sigma }}_{i,j},ϵ\right)\\ G_{ij}=exp\left(-{\displaystyle \frac{D_{ij}}{2{\sigma }_{ij}^2}}\right)\end{array}\right.$$

(3)

Here, we set the threshold $ϵ=0.1$ to prevent division by zero and ensure numerical stability. A value of $G_{ij}$ approaching 1 indicates high geometric proximity between nodes $i$ and $j$, corresponding to a stronger connection. Where $\mathsf{\Sigma }$ denotes the tolerance parameter for the Gaussian distance penalty. Formulating $\mathsf{\Sigma }$ as a learnable matrix enables the model to adaptively allocate specific distance thresholds across various joint pairs. Utilizing a Gaussian kernel, the GTRM applies an exponential decay penalty to physically distant node connections. Consequently, this strong geometric prior compels the model to focus primarily on locally reliable joint connections, even in the presence of severe noise, which effectively eliminates structural distortions induced by drift.

Fused Graph Convolution: Traditional GCNs typically generate dynamic adjacency matrices $A_t\in {\mathbb{R}}^{N\times C^{\prime }\times T\times V}$ via channel-specific correlation modeling. Building upon this foundation, EMS-GCN integrates a Gaussian gating mechanism for topology refinement. Let $A_{init}$ denote the initial adjacency matrix defined by natural connections, and $A_{dyn}$ represent the dynamic component generated by the model. The fundamental topology is formulated as follows:

$$A_t=A_{dyn}+A_{init}$$

(4)

The refined adjacency matrix $A_{\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}$ of EMS-GCN is defined as:

$$A_{\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}}=A_t+\beta \cdot \left(A_t\odot G_{\mathrm{m}\mathrm{a}\mathrm{p}}\right)$$

(5)

where $\odot$ denotes the Hadamard product. The broadcasting mechanism is applied along the channel dimension. The contributions of the original topology and the Gaussian-corrected topology are adaptively balanced by a learnable scalar parameter $\beta$. The model relies more heavily on the geometric prior by increasing $\beta$ under conditions of severe sensor noise. The final spatial feature update is formulated as follows:

$$X_{\mathrm{o}\mathrm{u}\mathrm{t}}=Einsum\left(\prime ncuv,nctv \to nctu\prime ,A_{\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{d}},X_{\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{d}}\right)$$

(6)

Gaussian topological priors are essential for preserving human bilateral symmetry and geometric stability. Occlusion or motion noise easily induces asymmetric anomalous connections within purely feature-driven dynamic graphs. These anomalies subsequently corrupt the inherent symmetric topology of the skeleton. To resolve this issue, GTRM calculates time-smoothed feature distances to generate a Gaussian distribution. Due to inherent physical symmetries, mapping these features into the Gaussian probability space yields balanced distance responses for symmetric joints. This process effectively preserves bilateral symmetry at the prior level. The Gaussian gating mechanism of GTRM acts as a stable geometric anchor against noise. Ultimately, the model restores geometric consistency by suppressing spurious high-frequency edges that corrupt symmetry and forcing the refined topology to regress toward physically compliant spatial distributions.

The conventional CTR-GCN primarily infers dynamic topologies by applying linear transformations to channel-level feature differences. In contrast, GTRM models pairwise relationships using Gaussian distribution functions. This formulation compensates for the spatial distance constraints absent in CTR-GCN and effectively circumvents spurious connections. Unlike refinement mechanisms that recalibrate individual feature channels, the Gaussian topological prior in GTRM operates exclusively on the spatial dimension to regulate joint connectivities independently. Conceptually, GTRM acts as a data-driven structural prior. By mapping spatial distances into a Gaussian probability space, it introduces a physical constraint functioning as a topological regularizer. Ultimately, this mechanism penalizes anomalous dynamic edges produced by the baseline model to reduce ambiguity and prevent overfitting to noisy actions.

3.3. Multi-Scale Shift Linear Temporal Attention

Action recognition relies on temporal evolution alongside spatial structure. Standard Temporal Convolutional Network structures incur high computational overheads. Furthermore, the self-attention mechanism in standard Transformers exhibits $O\left(T^2\right)$ complexity and struggles to process long sequences. To address these challenges, we propose a hybrid Multi-scale Shift Linear Attention Module (MSLTAM) that combines multi-scale shifts with linear attention as illustrated in Figure 3.

Multi-scale Temporal Shift. We employ a temporal shift strategy to capture local temporal context with negligible computational overhead. We define the shift operator $S_d\left(\cdot \right)$ with dilation rate $d$ for the input feature $Z\in {\mathbb{R}}^{N\times C\times T\times V}$. The shift operation displaces features along the temporal axis for each channel $c$ as described below:

$$Z_{out}^{\left(t\right)}=\left[S_d^{+}{\left(Z\right)}^{\left(t\right)},Z^{\left(t\right)},S_d^{-}{\left(Z\right)}^{\left(t\right)}\right]$$

(7)

Specifically, bidirectional fill shifting is implemented in EMS-GCN:

$$S_d^{+}{\left(Z\right)}^{\left(t\right)}=\left\{\begin{array}{cc}{Z}^{(t+d)},\hfill & \hfill t<T-d\\ 0,\hfill & \hfill t\ge T-d\end{array}\right.,S_d^{-}{\left(Z\right)}^{\left(t\right)}=\left\{\begin{array}{cc}0,\hfill & \hfill t<d\\ Z^{(t-d)},\hfill & \hfill t\ge d\end{array}\right.$$

(8)

We employ parallel multi-path shift operations with dilation rates $d\in \left\{\mathrm{1,2}\right\}$ to capture multi-scale features. This design significantly expands the temporal receptive field without increasing multiplication operations. The formulation is expressed as follows:

$$F_{\mathrm{s}\mathrm{h}\mathrm{i}\mathrm{f}\mathrm{t}}={Conv}_{1\times 1}\left(Concat\left(\left[S_1\left(Z\right),S_2\left(Z\right),Z\right]\right)\right)$$

(9)

Efficient Linear Attention: Shift operations specialize in local features while linear attention modules address global temporal dependencies. The standard attention calculation follows $Softmax\left(Q{K}^T\right)V$. This formulation suffers from high computational costs and extensive memory usage. We adopt a linear attention approximation based on the kernel trick. We denote the query, key and value matrices as $K,Q,V\in {\mathbb{R}}^{N\times H\times T\times (C∕H)}$. The variable $H$ represents the number of heads. We replace the Softmax kernel with the feature mapping function $\varphi \left(\cdot \right)$ as follows:

$$\varphi \left(\cdot \right)=GELU\left(x\right)+l$$

(10)

The function $\varphi \left(\cdot \right)$ guarantees non-negative feature values, a prerequisite for kernel function approximation within the attention mechanism. By leveraging the associative property of matrix multiplication, we reorder the computation from $\left(Q{K}^T\right)V$ to $Q\left(K^{T}V\right)$. This optimization reduces the complexity from $O\left(T^2\right)$ to $O\left(T\right)$. To aggregate key-value information, we first compute the global context matrix $M_{kv}$ as follows:

$$M_{kv}=\sum _{t=1}^T\varphi {\left(K\right)}_t^T{V}_t\hspace{1em}\in {\mathbb{R}}^{H\times C\prime \times C\prime }$$

(11)

We subsequently compute the normalization factor $Z_k$ as follows:

$$Z_k=\sum _{t=1}^T\varphi {\left(K\right)}_t\hspace{1em}\in {\mathbb{R}}^{H\times C\prime }$$

(12)

We preserve the spatial dimension during the summation process. This strategy enables independent learning of temporal attention for each joint. We subsequently utilize the query matrix to retrieve global information for query response and re-weighting as follows.

$${Attn}_{num}=\varphi {\left(Q\right)}_t{M}_{kv}, {Attn}_{den}=\mathsf{\varphi }{\left(\mathrm{Q}\right)}_{\mathrm{t}}{\mathrm{Z}}_{\mathrm{k}}^{\mathrm{T}}$$

(13)

The final attention output can be obtained as follows:

$$Y_{\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}={\displaystyle \frac{\varphi \left(Q\right)\left(K^{T}V\right)}{\varphi \left(Q\right)K^{T}l}}={\displaystyle \frac{{Attn}_{num}}{{Attn}_{den}+ϵ}}$$

(14)

This mechanism enables the network to capture global context across the entire video sequence at each frame with linear computational cost, significantly enhancing the recognition of long-term actions. Finally, we derive the final temporal feature $X_{\mathrm{t}\mathrm{e}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{l}}$ by summing the shift features and linear attention features:

$$X_{\mathrm{t}\mathrm{e}\mathrm{m}\mathrm{p}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{l}}=F_{\mathrm{s}\mathrm{h}\mathrm{i}\mathrm{f}\mathrm{t}}+Y_{\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}$$

(15)

Efficient Channel Attention: To further enhance feature discriminability, we integrate a dual-branch efficient channel attention module at the end of each residual block. This component calibrates feature responses from both global and local perspectives. As illustrated in Figure 4, the global branch is designed to capture inter-channel correlations and is formulated as follows:

$$g=AvgPool(X)\in {\mathbb{R}}^{N\times C\times 1\times 1}, {\alpha }_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}}=\sigma (W_2\delta (W_{1}g))$$

(16)

The symbols $\delta$ and $\sigma$ denote the ReLU and Sigmoid activation functions, respectively. The local branch focuses on fine-grained features across adjacent channels and spatiotemporal positions. We employ a $3\times 1$ convolution kernel to facilitate feature interaction as follows:

$${\alpha }_{\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}=\sigma \left(BN\left({Conv}_{1\times 1}\left(\sigma \right(BN\left({Conv}_{3\times 1}\right(X\left)\right)\left)\right)\right)\right)$$

(17)

Therefore, the final fusion output is obtained through residual connections and attention weighting:

$$X_{final}=X·{\alpha }_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}}+X·{\alpha }_{\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}+X$$

(18)

This design allows the model to adaptively emphasize keyframes and key joints that are crucial for action classification, while suppressing background noise.

Figure 4. Schematic of the proposed efficient channel attention module. This module comprises a global branch and a local branch operating in parallel. The global branch utilizes global average pooling to compress spatiotemporal dimensions and captures long-range channel dependencies via a squeeze-and-excitation mechanism. The local branch employs convolutional layers to preserve spatial structures while extracting fine-grained spatiotemporal context. The final features are fused via residual weighting to adaptively enhance the responses of key action features.

Figure 4. Schematic of the proposed efficient channel attention module. This module comprises a global branch and a local branch operating in parallel. The global branch utilizes global average pooling to compress spatiotemporal dimensions and captures long-range channel dependencies via a squeeze-and-excitation mechanism. The local branch employs convolutional layers to preserve spatial structures while extracting fine-grained spatiotemporal context. The final features are fused via residual weighting to adaptively enhance the responses of key action features.

3.4. Loss Function

We formulate the task as a multi-class classification problem and optimize the network parameters using the standard cross-entropy loss function. The training objective is to minimize the error between the predicted and ground truth labels. Let $Y\in {\mathbb{R}}^{N\times K}$ denote the output logits and $y$ represent the ground truth labels. The loss function is defined as follows:

$${\mathcal{L}}_{cls}=-{\displaystyle \frac{1}{N}}\sum _{I=1}^N\sum _{k=1}^K\mathbb{I}(y_i=k)log({\displaystyle \frac{exp\left(Y_{i,k}\right)}{{\sum }_{j=1}^{K}exp\left(Y_{i,j}\right)}})$$

(19)

The variable $K$ denotes the total number of action categories and $\mathbb{I}(\cdot )$ represents the indicator function. Auxiliary reconstruction tasks are referenced in partial configurations. However, we primarily rely on classification loss for end-to-end supervised training in the core experimental settings. We adopt this strategy to ensure real-time performance and prioritize the enhancement of classification accuracy.

4. Experiments

4.1. Experimental Setup

Datasets: The effectiveness and generalization of EMS-GCN for action recognition are evaluated on three widely used benchmark datasets in action understanding, namely Penn Action [44], NTU RGB+D [45], and NW-UCLA [46]. The Penn Action dataset consists of standard sports motion sequences and provides detailed annotations of human poses and action categories, which enables evaluation of the model performance in modeling pose variations and fine-grained motion patterns in sports activities. NTU RGB+D is a widely used large-scale dataset for 3D skeleton-based action recognition, covering diverse action categories and scene variations. The NW-UCLA dataset represents a multi-view 3D action recognition benchmark captured synchronously by Kinect cameras from three distinct angles. This dataset encompasses RGB, depth and skeleton data modalities. It is widely employed to evaluate algorithm generalization capabilities in cross-view scenarios. By conducting joint tests on these three datasets, the performance of EMS-GCN can be objectively verified from two perspectives: fine-grained motion modeling capability and robustness in real-world complex environments.

Evaluation Metrics: Top-1 Accuracy is used as the primary quantitative metric to maintain consistency with mainstream action recognition evaluation methods. This metric measures the proportion of correct predictions, where the top predicted label matches the ground truth. It directly reflects the model’s recognition ability under standard classification conditions and ensures clear consistency and comparability across different experimental settings and datasets.

Implementation Details: The EMS-GCN architecture is implemented using PyTorch 2.1, with training and inference conducted on a single NVIDIA RTX 3090 GPU. To ensure reproducibility, detailed parameter settings and training strategies are provided. The model is trained with the Stochastic Gradient Descent (SGD) optimizer, incorporating Nesterov momentum (0.9) to accelerate convergence. The initial learning rate is set to 0.1, with reductions by a factor of 0.1 at the 30th and 50th epochs. A warm-up strategy is employed for the first 5 epochs to stabilize the training process. A weight decay coefficient of 0.0005 is applied, and dropout with a rate of 0.5 is used to prevent overfitting. Data augmentation techniques, including random rotation, scaling, and temporal sampling, are introduced to enhance generalization.

4.2. Performance Comparison

We benchmark the proposed method against state-of-the-art approaches on public datasets, with detailed results provided in

Table 1. Performance comparison results on the NTU RGB+D dataset, where red and blue markers denote the first- and second-ranked performances under each individual metric, respectively.

Methods	NTU-60 (%)		NTU-120 (%)		Penn Action (%)	NW-UCLA (%)	Params (M)	Flops (G)
	x-Sub	x-View	x-Sub	x-Set
MS-G3D [27]	91.5	96.2	86.9	88.4	96.1	-	2.8 ($\downarrow$80%)	5.2
CTR-GCN [28]	92.4	96.4	88.9	90.4	96.9	96.5	1.5 ($\downarrow$62%)	2.0
EfficientGCN [47]	91.7	95.7	88.3	89.1	96.7	-	2.0 ($\downarrow$72%)	15.2
InfoGCN [30]	92.8	96.7	89.2	90.7	96.5	96.6	1.6 ($\downarrow$65%)	1.8
FRHead [48]	93.1	96.8	89.5	90.9	97.0	96.8	2.0 ($\downarrow$72%)	-
BlockGCN [8]	92.4	97.0	90.3	91.5	96.8	96.9	1.3 ($\downarrow$57%)	1.6
DeGCN [10]	93.3	97.4	91.0	92.1	97.6	97.2	5.6 ($\downarrow$90%)	-
ST-TR [49]	90.8	96.3	85.1	87.1	96.3	-	12.1 ($\downarrow$96%)	259.4
TranSkeleton [50]	92.8	97.0	89.4	90.5	96.7	-	2.2 ($\downarrow$75%)	9.2
Hyperformer [51]	92.9	96.5	89.9	91.3	97.1	96.7	2.7 ($\downarrow$80%)	9.6
SkeMixFormer [52]	93.0	97.1	90.1	91.3	99.2	97.4	2.1 ($\downarrow$73%)	4.8
SkateFormer [5]	93.5	97.4	89.8	91.4	98.4	98.3	2.0 ($\downarrow$72%)	3.6
FreqMixFormer [6]	93.6	97.4	90.5	91.9	99.7	97.4	2.0 ($\downarrow$72%)	64.4
EMS-GCN	92.6	97.3	88.7	91.8	99.4	97.2	0.56	1.3

. The experimental results demonstrate that our method exhibits significant advantages across all datasets. By optimizing the topology of information flow, the framework maintains or even enhances the mutual information of action semantics while substantially reducing model entropy capacity. Notably, on large-scale datasets such as NTU RGB+D and NTU RGB+D 120, the proposed model achieves state-of-the-art performance despite a reduction in parameter count of over 50%.

Specifically, EMS-GCN achieves state-of-the-art performance on the NTU RGB+D dataset with a model size of only 0.56 M parameters. We train and evaluate the model exclusively on skeleton data, reaching 92.6% accuracy on the NTU RGB+D x-sub benchmark. Notably, our approach trails the leading FreqMixFormer method by only 1.0%, despite a 72% reduction in parameter count. Moreover, EMS-GCN achieves 97.1% accuracy on the NTU RGB+D x-view benchmark, delivering the second-best performance among state-of-the-art methods. On the more extensive NTU RGB+D 120 dataset, the model shows a marginal performance gap compared to top-tier approaches. These results are particularly noteworthy given the highly efficient architectural design of our framework.

EMS-GCN exhibits remarkable efficacy on small-scale datasets such as Penn Action and NW-UCLA. On the Penn Action dataset, our model achieves second-best performance, trailing the leading FreqMixFormer by a mere 0.3%. Furthermore, EMS-GCN delivers competitive results on NW-UCLA, where the accuracy gap remains within 1.1% compared to SkateFormer. These outcomes underscore the robust generalization capabilities of our architecture across varied data scales. While performance on large-scale datasets provides potential for further improvement, EMS-GCN reduces the total parameter count by more than 60% compared to state-of-the-art methods. We achieve a successful balance between model volume and recognition performance. Consequently, our approach maintains a competitive advantage over various state-of-the-art benchmarks.

4.3. Ablation Study

We perform a series of ablation studies on the Penn Action and NW-UCLA datasets to validate the contribution of each component in EMS-GCN and assess its impact on recognition accuracy and computational cost. To ensure a fair comparison, we maintain identical hyperparameter configurations across all trials, modifying only the specific module under evaluation.

Contribution of Individual Components: We extensively modify the CTR-GCN baseline to evaluate the performance gains from the Gaussian Topology Refinement Module (GTRM) and the Multi-scale Shift Linear Attention Module (MS-LTA).

Table 2. Comparison results of the contributions of core components. The red and blue color markers indicate the first and second place rankings for each metric, respectively.

Method	+GTRM	+MS-LTA	Action Penn (%)	NW-UCLA (%)
Baseline	×	×	96.9	96.5
Baseline + GTRM	√	×	98.2	96.7
Baseline + MS-LTA	×	√	99.1	96.9
EMS-GCN	√	√	99.4	97.2

shows that both components yield clear improvements over the baseline. Specifically, our architecture outperforms the baseline by 2.5% and 0.7% on the Penn Action and NW-UCLA datasets, respectively. These results confirm the effectiveness of the individual components within EMS-GCN.

Comparison of Temporal Modeling Strategies: Efficiency in temporal modeling is vital for edge computing in smart sports analysis.

Table 3. Ablation results of Temporal Modeling Strategies. The red and blue color markers indicate the first and second place rankings for each metric, respectively.

Strategy	Complexity	Flops (G)	Penn Action (%)
TCN	$\mathrm{O}(\mathrm{K}\cdot \mathrm{T})$	1.2	96.9
Self-Attention	$\mathrm{O}\left({\mathrm{T}}^2\right)$	1.8	97.7
MS-LTA	$\mathrm{O}\left(\mathrm{T}\right)$	0.6	99.4

presents a comparison of performance and resource consumption across various temporal modules. We design MS-LTA to achieve linear complexity by leveraging kernel function approximation. Experimental results show that our approach surpasses the standard TCN in accuracy while requiring only 50% of the FLOPs. This effective trade-off between precision and latency positions MS-LTA as a preferred solution for real-time action recognition.

Sensitivity Analysis of Multi-Scale Shift Configurations: To capture action features across various temporal spans, we evaluate the impact of different dilation rate combinations in shift operations.

Table 4. Ablation results of the Sensitivity Analysis of Multi-Scale Shift Configurations. The red and blue color markers indicate the first and second place rankings for each metric, respectively.

Dilations Configuration	Receptive Field	Penn Action (%)
Single Scale	Small	98.4
Dual Scale	Medium	99.4
Triple Scale	Large	99.2

shows that a single scale is insufficient for capturing the temporal context of slow motions. Although more complex scale combinations theoretically expand the receptive field, they introduce redundant information at this parameter scale, leading to slight performance drops and increased memory overhead. We adopt a dual-scale configuration in EMS-GCN to achieve an optimal balance between feature richness and model compactness. This choice validates the effectiveness of our design for resource-constrained scenarios.

Impact of Dual-Branch Attention Calibration: We evaluate the dual-branch attention module integrated at the end of the network through a series of ablation tests.

Table 5. Ablation results of the Impact of Dual-Branch Attention Calibration. The red and blue color markers indicate the first and second place rankings for each metric, respectively.

Branch Configuration	Focus Area	Penn Action (%)
w/o Attention	-	98.4
Global Only	Channel	99.2
Local Only	Temporal	99.1
Ours	Channel + Temporal	99.4

shows that using the global or local branch independently leads to performance improvements. Our parallel dual-branch structure adaptively calibrates feature responses by emphasizing critical feature channels and key action frames. This integration produces a synergistic effect, yielding a 0.9% accuracy improvement over the baseline with almost no additional computational burden.

5. Conclusions

We propose EMS-GCN, an efficient graph convolutional network tailored for real-time sports analysis on edge devices. To address structural noise in sensor-based skeleton data, we design the Gaussian Topology Refinement Module (GTRM). This module utilizes learnable Gaussian distances to effectively constrain topological ambiguity. Additionally, we develop a multi-scale shift linear attention mechanism that integrates parameter-free shift operations with linear-complexity attention. This design enables long-term temporal dependency modeling with minimal computational overhead. Experiments demonstrate that EMS-GCN attains high accuracy with a minimal parameter count of 0.56 M. Our model significantly surpasses mainstream baseline models, offering an efficient, lightweight solution for intelligent sensing systems. We will direct future efforts toward optimizing deployment on heterogeneous embedded hardware and exploring multi-modal fusion.