Graph Convolutional Network with Multi-View Topology for Lightweight Skeleton-Based Action Recognition

Abstract

Skeleton-based action recognition is an important subject in deep learning. Graph Convolutional Networks (GCNs) have demonstrated strong performance by modeling the human skeleton as a natural topological graph, representing the connections between joints. However, most existing methods rely on non-adaptive topologies or insufficiently expressive representations. To address these limitations, we propose a Multi-view Topology Refinement Graph Convolutional Network (MTR-GCN), which is efficient, lightweight, and delivers high performance. Specifically: (1) We propose a new spatial topology modeling approach that incorporates two views. A dynamic view fuses joint information from dual streams in a pairwise manner, while a static view encodes the shortest static paths between joints, preserving the original connectivity relationships. (2) We propose a new MultiScale Temporal Convolutional Network (MSTC), which is efficient and lightweight. (3) Furthermore, we introduce a new temporal topology strategy by modeling temporal frames as a graph, which strengthens the extraction of temporal features. By modeling the human skeleton as both a spatial and a temporal graph, we reveal a topological symmetry between space and time within the unified spatio-temporal framework. The proposed model achieves state-of-the-art performance on several benchmark datasets, including NTU RGB + D (XSub: 92.8%, XView: 96.8%), NTU RGB + D 120 (XSub: 89.6%, XSet: 90.8%), and NW-UCLA (95.7%), demonstrating the effectiveness of our GCN module, TCN module, and overall architecture.

1. Introduction

Skeleton-based action recognition plays a crucial role in real-world applications and has long been a prominent topic in the field of deep learning. It is widely used in areas such as intelligent surveillance, human–computer interaction, and medical rehabilitation. Early studies on action recognition primarily relied on video or RGB data as input [1,2,3]. With the advancement of various sensor technologies, alternative modalities have emerged, including skeleton data [4,5,6], depth maps [7], and infrared sequences [8]. Among these, skeleton data has attracted increasing attention due to its advantages in preserving user privacy, being robust against background noise, and enabling lightweight network designs.

Initially, skeleton-based action recognition methods were built upon hand-crafted features [9], which often performed well on specific datasets but lacked generalization ability. Subsequently, deep learning approaches were introduced. RNN-based methods [10,11] treated skeleton sequences as ordered vectors to preserve temporal information but struggled to model spatial dependencies and suffered from slow training. CNN-based methods [12] transformed skeleton data into pseudo-2D images, which also lacked explicit spatial modeling and failed to capture complex joint interactions while offering limited temporal modeling. Since the human skeleton naturally forms a graph structure, GCN-based methods [13] have been proposed to aggregate features based on the topological relationships between joints, effectively capturing intricate inter-joint dependencies and achieving leading performance in both spatial and temporal feature extraction.

Yan et al. [14] made a pioneering contribution by proposing ST-GCN, the first work to apply graph convolutional networks to skeleton-based action recognition. In ST-GCN, joints are treated as graph nodes, and bones (including spatial and temporal connections) are treated as graph edges, thereby establishing a graph-based representation of the human skeleton. However, ST-GCN relies on manually defined skeleton topologies, which are not adaptive and may overlook important connections between joints that are not naturally linked. Subsequent methods [15,16,17] made improvements in topology learning, but most of them either used simplistic strategies or focused only on a single perspective, leading to incomplete and inaccurate topology representations. BlockGCN [17] pointed out that simple interactions or learning topologies from a single perspective can cause the loss of original skeleton connection information and deviate from the physical topology, which severely impacts recognition performance.

Moreover, as GCN-based methods continue to evolve and recognition accuracy improves, the ever-growing number of parameters and computational demands have emerged as major challenges. Although models such as CTRGCN [16], FR-Head [18], EfficientGCN-B4 [19], and MST-GCN [20] have achieved promising results, they still suffer from heavy model complexity and high computational costs. Therefore, this paper aims to design a high-accuracy, lightweight, and computationally efficient model.

In terms of spatial modeling (GCN), we propose a novel Multi-view Topology Refinement Graph Convolutional Network (MTRGC). In the dynamic view, we dynamically learn pairwise topology relationships among all joints using a dual-stream feature input strategy. Features are projected onto multiple heads to capture richer relational representations. Within each head, we introduce a dimension expansion mechanism that enables pairwise interaction between dual-stream nodes, allowing channel-wise information exchange. We then fuse the multi-head topological information to form a refined representation. For the static view, inspired by BlockGCN [17], we incorporate a topology encoding mechanism that solely depends on the original physical connections and is independent of the input data. By encoding relative positions between statically connected joints, we preserve inherent skeletal structures while complementing the dynamic view.

For temporal modeling, aiming for a lightweight and efficient design, we propose a new MultiScale Temporal Convolutional (MSTC) Network with temporal topology modeling: We employ depthwise separable convolution to extract multi-scale temporal features and integrate a pooling module to capture averaged feature representations. Additionally, another branch is introduced to preserve the original input features. To mitigate the limited receptive field of depthwise separable convolution, we adopt large kernels for broader context modeling. (b) We propose Gated Channel-wise Temporal Topology (GCTT), which models time series as graphs to explicitly capture temporal topology relationships. This approach significantly enhances temporal feature extraction.

We conduct extensive experiments on large-scale skeleton action recognition datasets [21,22,23]. Experimental results show that our model achieves superior performance across multiple modalities. The main contributions of this paper are summarized as follows:

• We propose a multi-view topology modeling strategy that captures dynamic joint relationships via a novel pairwise interaction mechanism, while preserving original skeletal connectivity through a complementary static view.
• We propose a novel MultiScale Temporal Convolutional Network that employs depthwise separable convolutions with larger kernels for temporal feature extraction. By incorporating a pooling module and a branch preserving input information, it captures richer feature representations. The proposed temporal module achieves a lightweight design while maintaining high accuracy.
• We propose a novel Gated Channel-wise Temporal Topology (GCTT) that further improves temporal feature extraction on top of the lightweight design. Extensive experiments demonstrate that each component of our model achieves remarkable performance, and the overall model surpasses many state-of-the-art methods. Furthermore, our results highlight the importance of simultaneously leveraging dynamic and static topology information.

(For abbreviations, refer to

Table A1. Acronym table.

Acronym	Full Form
GCN	Graph Convolutional Network
TCN	Temporal Conbolutional Nwtwork
MTRGC	Multi-view Topology Refinement Graph Convolution
MSTC	MutiScale Temporal Convolution
GCTT	Gated Channel-wise Temporal Topology

.)

2. Related Work

2.1. Skeleton-Based Action Recognition

With the early development of action recognition, most methods relied on videos or RGB data as input. However, with the advancement of various types of sensors, action recognition has expanded to other modalities. Among them, skeleton data, due to its unique advantages, has emerged as a prominent research focus in the field of action recognition.

Early skeleton-based action recognition methods using deep learning mainly adopted RNNs and CNNs for feature extraction. However, these models often lacked effective spatial or temporal modeling capabilities, making it difficult to capture complex and deep inter-joint relationships, which led to a bottleneck in the progress of skeleton-based recognition.

Since human joints naturally form a graph structure in a non-Euclidean space, Graph Convolutional Networks (GCNs) offer an appropriate tool to handle such structures. Yan et al. [14] first introduced Graph Convolutional Networks (GCNs) to extract topological relations among skeletal joints, significantly enhancing feature representation for skeleton-based action recognition.

Further studies built upon this idea. However, these early GCN-based models used non-adaptive topological structures, limiting their ability to flexibly model spatial relationships and preventing dynamic topology updates.

Subsequent works focused on adaptive learning of topology: 2s-AGCN [15] parameterized skeleton nodes into learnable vectors, allowing the network to adaptively learn adjacency weights. CTRGCN [16] proposed a dual-stream subtraction strategy to capture channel-wise topological relationships between joints. This channel-level topological modeling significantly improved model performance and became a baseline choice for many models due to its simplicity. The SA-GC module proposed by InfoGCN [24], based on dot-product attention, can infer a context-dependent intrinsic topology in the spatial modeling of a skeleton. Self-GCN [25] similarly uses attention-based topological modeling, and both demonstrate the widespread application and effectiveness of the attention mechanism in skeleton-based action recognition.

For temporal modeling, early methods like ST-GCN [14] employed simple temporal convolutions. Later, more advanced models were proposed: MST-GCN [20] partitioned feature maps along the channel dimension to better capture features across different temporal ranges. CTRGCN [16] incorporated a multi-scale temporal convolution module combined with dilated convolutions and pooling layers, significantly enhancing the network’s ability to model long-term temporal dependencies. Similarly, FR-Head [18], Koopman [26], and others utilized enhanced TCNs that played crucial roles in improving the extraction of temporal features.

2.2. Relative Position Encoding

Although the aforementioned models are capable of capturing complex relationships between joints, their methods for node interaction remain suboptimal. Moreover, since these models often learn topology from a single perspective, the dynamically learned topology tends to gradually deviate from the original physical structure of the skeleton.

To address this issue, BlockGCN [17] introduced a relative positional encoding mechanism that preserves the inherent skeletal connections during dynamic topology learning.

Building upon this idea, we propose a novel multi-view topology representation, which further enhances the modeling of spatial relationships by integrating multiple perspectives.

2.3. Lightweight Temporal Convolutional Networks

Although the aforementioned models have achieved notable breakthroughs in performance, these improvements often come at the cost of increased parameter counts and higher computational complexity. Moreover, relatively little attention has been paid to enhancing temporal feature extraction.

MobileNet [27] use depthwise separable convolutions to build light-weight deep neural networks. A depthwise separable convolution consists of two steps: first, a depthwise convolution is applied to each input channel separately; second, a pointwise (1 × 1) convolution is used to fuse features across channels. This structure significantly reduces the complexity of convolutional operations and has been widely adopted in various computation-constrained visual tasks.

In the field of skeleton-based action recognition, methods such as TSGCNext [28] have also applied depthwise separable convolutions. However, the usage was limited to single-branch designs, which restricted the capacity for rich temporal feature extraction.

Meanwhile, models such as CTRGCN [16], FR-Head [18], Koopman [26], and BlockGCN [17] employed multi-scale temporal modules, but they suffered from excessive parameter sizes, low efficiency, and overly complex computations.

Inspired by MobileNet’s [27] depthwise separable convolution, our research combines multi-scale temporal modeling and proposes a novel MultiScale Temporal Convolutional Network (MSTC).

On this basis, we further integrate a Gated Channel-wise Temporal Topology (GCTT) module, significantly improving the model’s performance while maintaining a lightweight and efficient design.

2.4. Symmetric Topology Modeling

Human skeletons are naturally represented as graphs and processed using Graph Convolutional Networks (GCNs), with the adjacency matrix playing a crucial role. We perform symmetric topology modeling by treating the skeleton as both a spatial and a temporal graph, capturing joint relationships from both dimensions: (a) For spatial modeling, our MTRGC module performs multi-view topology learning, preserving the original static connections while dynamically capturing evolving spatial dependencies. (b) For temporal modeling, we treat skeleton sequences as temporal graphs to learn topological structures along the time dimension. Experimental results demonstrate that our symmetric modeling of spatial and temporal topologies significantly enhances the GCN’s ability to capture structural relationships.

3. Methods

In this section, we first present the graph representation of the skeleton data (Section 3.1), followed by a description of the Graph Convolutional Network (GCN) from different views (Section 3.2). Subsequently, we analyze the lightweight design of the Temporal Convolutional Network (TCN) from a mathematical standpoint, along with the modeling process of our Gated Channel-wise Temporal Topology (GCTT) (Section 3.3.2). Finally, we outline the overall structure of the proposed model (Section 3.4).

3.1. Preliminaries

Human skeleton data inherently exhibits a graph structure represented by $\mathcal{G}=(\mathcal{V},\mathcal{E})$ [14]. Specifically, $\mathcal{V}=\left\{v_1,v_2,\dots \dots ,v_N\right\}$ denotes the set of joints. The set of edges $\mathcal{E}$, representing the connections between each pair of joints, is described by the adjacency matrix $\mathbf{A}\in {\mathbb{R}}^{N\times N}$. Each element $a_{ij}$ in $\mathbf{A}$ indicates the strength of the connection between $v_i$ and $v_j$.

In this work, A is the initialized adjacency matrix representing the human skeleton topology, defined following the design in ST-GCN [14]. A contains three types of edges, corresponding to three channels: self-link, inward, and outward. These three channels represent different types of topologies, enriching the expression of topological relationships. A[i] refers to the i-th channel of A. In our GCN module, we loop through MTRGC three times, passing in one channel of A each time.

Our work in Section 3.2 aims to optimize the weights of the adjacency matrix to accurately model the relationships between joints. The original skeleton data is represented as $\mathbf{X}\in {\mathbb{R}}^{C\times T\times V}$, where C denotes the number of input channels, T denotes the number of input frames, and V denotes the number of joints.

3.2. Multi-View Topology Refinement Graph Convolution

As shown in Figure 1, MTRGC is divided into two parts: a dynamic view and a static view, each capturing different topological relationships. The following sections will introduce these two views separately.

3.2.1. Dynamic View

The dynamic view adopts a two-stream strategy with input features ${\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}}\in {\mathbb{R}}^{C\times T\times V}$. To reduce computational cost, we first reduce the feature dimension, followed by a temporal pooling layer to obtain the averaged features along the T dimension. This process can be expressed as follows:

$${\mathbf{x}}_1={\displaystyle \frac{1}{T}}\sum _{i=1}^T\varphi \left({\mathbf{x}}_{:,t,:}\right),{\mathbf{x}}_2={\displaystyle \frac{1}{T}}\sum _{i=1}^T\phi \left({\mathbf{x}}_{:,t,:}\right),$$

(1)

although ‘conv1’ and ‘conv2’ have the same shape and structure, they are two independent layers with different parameters (initialized and updated independently). During model training, they do not share parameters and do not affect each other. Therefore, we use two different function symbols, $\varphi$ and $\phi$, to represent ‘conv1’ and ‘conv2’, respectively. Now ${\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}}\in {\mathbb{R}}^{V\times C_r}$.

To enrich feature extraction, the features are projected onto multiple heads, and the channels of each head are given by $C_h$:

$${\mathbf{x}}_1=\left[{\mathbf{x}}_{11},\dots ,{\mathbf{x}}_{1h}\right],{\mathbf{x}}_2=\left[{\mathbf{x}}_{21},\dots ,{\mathbf{x}}_{2h}\right],$$

(2)

Subsequently, we expand the dimensions and concatenate the features to fuse the channel information of each pair of nodes, thereby constructing a topology matrix that fully encodes the interactions among all nodes. As illustrated in Figure 2, we use the first head as an example to demonstrate the process of pairwise topology modeling; the inputs are represented by ${\mathbf{x}}_{\mathbf{11}}\in {\mathbb{R}}^{1\times V\times C_h}$ and ${\mathbf{x}}_{\mathbf{21}}\in {\mathbb{R}}^{1\times V\times C_h}$. As shown in Figure 2, the V node characteristics of ${\mathbf{x}}_{\mathbf{11}}$ are treated as a column and replicated V times, while the V node characteristics of ${\mathbf{x}}_{\mathbf{21}}$ are treated as a row and replicated V times:

$$\begin{array}{c}{\mathbf{x}}_{11}^{exp}={\mathbf{x}}_{11}\otimes {\mathbf{1}}_V^{\top }\in {\mathbb{R}}^{V\times V\times C_h},\hfill \\ {\mathbf{x}}_{21}^{exp}={\mathbf{1}}_V\otimes {\mathbf{x}}_{21}\in {\mathbb{R}}^{V\times V\times C_h},\hfill \end{array}$$

(3)

where ${\mathbf{x}}_{11}^{exp}$ and ${\mathbf{x}}_{21}^{exp}$ represent the expanded joint matrices, ${\mathbf{1}}_V$ represents a matrix of shape $V\times 1$ with all elements equal to one, ${\mathbf{1}}_V^{\top }$ represents the transpose of the matrix, and ⊗ represents matrix multiplication.

Then, ${\mathbf{x}}_{11}^{exp}$ and ${\mathbf{x}}_{21}^{exp}$ are concatenated along the channel dimension. In this way, the feature of each node in ${\mathbf{x}}_{11}^{exp}$ is concatenated with the feature of each node in ${\mathbf{x}}_{21}^{exp}$, including itself. As a result, all nodes from the two-stream inputs are fused along the channel dimension to form a new feature representation with shape $1\times V\times V\times 2C_h$. Finally, a linear layer is applied to fuse the channel information of each pair of nodes, resulting in the desired spatial topology matrix:

$$\begin{array}{c}\begin{array}{cc}\hfill {\mathbf{Z}}_{i,j}& =\left[{\left({\mathbf{x}}_{11}^{exp}\right)}_{i,j}\Vert {\left({\mathbf{x}}_{21}^{exp}\right)}_{i,j}\right]\in {\mathbb{R}}^{1\times V\times V\times 2C},\hfill \\ \hfill {\mathbf{S}}_{i,j}& ={\mathbf{W}}_s{\mathbf{Z}}_{i,j}\in {\mathbb{R}}^{1\times V\times V}.\hfill \end{array}\hfill \end{array}$$

(4)

In this process, ‖ denotes the concatenation operation between each joint in ${\mathbf{x}}_{11}^{exp}$ and ${\mathbf{x}}_{21}^{exp}$. As illustrated in Figure 2, each joint in ${\mathbf{x}}_{11}^{exp}$ is concatenated with each joint in ${\mathbf{x}}_{21}^{exp}$. ${\mathbf{W}}_s$ represents the fusion of channel information, ensuring thorough interaction between each joint pair in the dual-stream data. Since we employ h heads, the final derived pairwise topology is denoted as ${\mathrm{A}}_p\in {\mathbb{R}}^{h\times V\times V}$.

After performing pairwise topology modeling, we further enhance the extraction of topological relationships by fusing the multi-head topology information, projecting it into a high-level representation:

$${\mathrm{A}}_d={\mathrm{A}}_p\times W_c,{\mathrm{A}}_d\in {\mathbb{R}}^{C^{\prime }\times V\times V},$$

(5)

where $W_c$ denotes the weights of 2d convolution, ${\mathrm{A}}_p$ represents the topology obtained from the pairwise topology modeling, and ${\mathrm{A}}_d$ represents the final channel-wise topology obtained from the dynamic view.

3.2.2. Static View

BlockGCN [17] identified that the original skeletal connection information may be lost during the dynamic weight updating process, and proposed a relative positional encoding method to mitigate this issue. In this work, we incorporate the encoding strategy from BlockGCN as the static view to complement and enhance the dynamic view.

As illustrated in Figure 1, the numbers in the joint matrix indicate the minimum hop count between two joints on the skeletal graph ${\mathcal{G}}_{\mathrm{S}}$. The Hop Params Table, which can be dynamically updated, provides weight parameters $e_i$ based on the hop value $d_{i,j}$ between two joints. By selecting the corresponding hop parameter according to the hop value $d_{i,j}$, we construct a topology matrix $B\in {\mathbb{R}}^{\left|\mathcal{V}\right|\times \left|\mathcal{V}\right|}$ that preserves the original skeletal connectivity. Since the hop parameters are determined by the fixed, unchanging skeletal structure, this representation is referred to as the static view. The encoding process can be expressed as follows:

$$\begin{array}{cc}\hfill B_{ij}& =e_{d_{i,j}}\mathrm{with}\hfill \\ \hfill d_{i,j}& =\underset{D\in Distances\left({\mathcal{G}}_{\mathbf{s}}\right)}{min}\left\{\left|D\right|,D_1=v_i,D_{\left|D\right|}=v_j\right\},\hfill \end{array}$$

(6)

where $D_1$ and $D_D$ denote the starting point and the ending point, $\left|D\right|$ denotes all the distance values from $D_1$ to $D_D$, and hop value $d_{i,j}$ indicates the shortest distance between $D_1$ and $D_D$. We denote the topology obtained from the static view as ${\mathrm{A}}_s\in {\mathbb{R}}^{C^{\prime }\times V\times V},$

3.2.3. Multi-View

As shown in Figure 1 and Figure 3, by fusing the topology from the two views and refining them using the CTRGCN method, we obtain a multi-view refinement topology:

$${\mathrm{A}}_m=\left({\mathrm{A}}_d\times \alpha \right)+{\mathrm{A}}_s+\mathrm{A}\left[i\right],{\mathrm{A}}_m\in {\mathbb{R}}^{C^{\prime }\times V\times V},$$

(7)

where $\alpha$ is a learnable parameter initialized to 0, used to control the strength of ${\mathrm{A}}_m$.

This part of feature transformation performs dimensional expansion on the original input, producing a feature tensor ${\mathbf{x}}_{\mathbf{3}}\in {\mathbb{R}}^{C^{\prime }\times T\times V}$ that matches the dimension of $A_m$. Finally, the multi-view topology $A_m$ is used to aggregate the features of ${\mathbf{x}}_{\mathbf{3}}$ to produce the new feature representation:

$$\mathbf{Z}=\mathbf{E}({\mathbf{X}}_3,{\mathbf{A}}_m)=\left[{\mathbf{A}}_{m1}{{\mathbf{x}}_3}_{:,1}∥{\mathbf{A}}_{m2}{{\mathbf{x}}_3}_{:,2}∥\dots \parallel {\mathbf{A}}_{{\mathbf{mC}}^{\prime }}{{\mathbf{x}}_3}_{:,{\mathbf{C}}^{\prime }}\right]$$

(8)

Here, E denotes the einsum summation operation. In fact, in our MTRGC, the topology from each channel of ${\mathrm{A}}_{\mathrm{m}}$ is used to aggregate the features of each channel in ${\mathbf{x}}_{\mathbf{3}}$: ${\mathrm{A}}_{\mathrm{m}}$ and ${\mathbf{x}}_{\mathbf{3}}$ for each channel are denoted as 1, 2, …, ${\mathbf{C}}^{\prime }$, and matrix multiplication is performed between them. The symbol ‖ denotes the concatenation operation. Finally, we obtain our new representation of the characteristics $\mathbf{Z}\in {\mathbb{R}}^{{\mathbf{C}}^{\prime }\times T\times V}$.

3.3. MultiScale Temporal Convolutional Network (MSTC)

In this section, we introduce the lightweight implementation and the Gated Channel-wise Temporal Topology (GCTT) separately.

3.3.1. Lightweight Implementation

Considering the inconsistent durations of different actions, we propose a lightweight multi-scale temporal modeling strategy to effectively capture features across various temporal spans. In contrast to [29], our method employs fewer branches. As illustrated in the right part of Figure 5, each branch initially applies a convolutional layer for reducing dimensions. The first two branches further pass through a depthwise convolution (dw Conv) and a pointwise convolution (pw Conv), followed by GCTT module to refine the features. The third branch applies a pooling operation, while the fourth branch directly retains the input features without modification. These four branches jointly extract features from multiple perspectives, substantially enhancing the richness of the feature representation.

As pointed out in Section 2.3, although previous research has attempted to utilize multi-branch structures or depthwise separable convolutions for temporal modeling, these approaches still exhibit certain deficiencies. In contrast, our proposed module achieves a lightweight and efficient design without compromising accuracy. Compared with methods such as CTRGCN [16], FR-Head [18], Koopman [26], and BlockGCN [17], which employ conventional convolution operations for feature extraction, our approach significantly reduces both parameters and computational cost.

For conventional 2d convolution:

$$\begin{array}{cc}\hfill & \mathrm{Parameters}:{\mathrm{Params}}_{2d}=C_{\mathrm{in}}\times C_{\mathrm{out}}\times k_t,\hfill \\ \hfill & \mathrm{Computational}\mathrm{Complexity}:{\mathrm{C}}_{2d}=O\left(T\times V\times C_{\mathrm{in}}\times C_{\mathrm{out}}\times k_t\right),\hfill \\ \hfill & \mathrm{FLOPs}:{\mathrm{FLOPs}}_{2\mathrm{d}}=2\times T\times V\times k_t\times C_{\mathrm{in}}\times C_{\mathrm{out}},\hfill \end{array}$$

(9)

For our depthwise separable convolution:

$$\begin{array}{cc}\hfill & \mathrm{Parameters}:{\mathrm{Params}}_{\mathrm{DSC}}=C_{\mathrm{in}}\times k_t+C_{\mathrm{in}}\times C_{\mathrm{out}},\hfill \\ \hfill & \mathrm{Computational}\mathrm{Complexity}:{\mathrm{C}}_{\mathrm{DTC}}=O\left(T\times V\times \left(C_{\mathrm{in}}\times k_t+C_{\mathrm{in}}\times C_{\mathrm{out}}\right)\right),\hfill \\ \hfill & \mathrm{FLOPs}:{\mathrm{FLOP}}_{\mathrm{DSC}}=2\times T\times V\times \left(k_t\times C_{\mathrm{in}}+C_{\mathrm{in}}\times C_{\mathrm{out}}\right),\hfill \end{array}$$

(10)

$C_{\mathrm{in}}$ denotes input channels, $C_{\mathrm{out}}$ denotes output channels, $k_t$ denotes kernel size, T denotes time dimension, and V denotes number of joints. Here, $O(·)$ denotes the asymptotic computational complexity, characterizing the growth rate of computation with respect to the input size.

From Equations (9) and (10), it can be observed that our depthwise separable convolution significantly outperforms the conventional 2d convolution in terms of parameter count, computational complexity, and FLOPs, achieving substantial reductions across all three metrics.

Moreover, dilated convolutions are applied in the first two branches to enhance the receptive field. Considering that depthwise separable convolutions operate on individual channels and naturally suffer from a limited receptive field, we utilize larger kernel sizes to mitigate this drawback.

3.3.2. Gated Channel-Wise Temporal Topology Representation

While Section 3.3.1 has achieved lightweight and efficient temporal modeling, there remains potential for further improvement in temporal feature extraction. To address this, we propose Gated Channel-wise Temporal Topology (GCTT). As depicted on the right side of Figure 5, GCTT processes the outputs from the depthwise separable convolutions of the first two branches. The detailed modeling procedure is illustrated in Figure 4. Initially, we use a 1 × 1 convolution to change the input channel size from C to 2C, and then apply a chunk operation along the channel dimension to split the input into q and k, where the input feature is denoted as $X\in {\mathbb{R}}^{C_b\times T\times V}$, and $C_b$ gives the branch channels in MSTC. The dimensional changes are shown in the figure:

$$\left(Q,K\right)=split\left(Conv\left(X\right)\right)\mathrm{where}Q,K\in {\mathbb{R}}^{C_b\times T\times V},$$

(11)

Then, we split Q and K along the channel dimension into h heads, with each head having a feature dimension denoted as $C_{bh}={\displaystyle \frac{C_b}{h}}$:

$$Q,K\in {\mathbb{R}}^{h\times C_{bh}\times T\times V},$$

(12)

and subsequently, spatial pooling is performed to compute the average features over all joints, and Q is transposed:

$$\begin{array}{cc}\hfill Q& ={\displaystyle \frac{1}{V}}\sum _{v=1}^{V}Q(:,:,:,v),K={\displaystyle \frac{1}{V}}\sum _{v=1}^{V}K(:,:,:,v),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill Q& \in {\mathbb{R}}^{h\times T\times C_{bh}},K\in {\mathbb{R}}^{h\times C_{bh}\times T}\hfill \end{array}$$

(14)

After this, we perform matrix multiplication to model dynamic temporal relations and fuse multi-head topological information. To further enhance the feature representation, we introduce a gating mechanism and a residual connection, resulting in the proposed Gated Channel-wise Temporal Topology (GCTT) denoted as ${\mathrm{A}}_{\mathrm{T}}$:

$${\mathrm{A}}_T=\mathsf{\Lambda }[softmax\left({\displaystyle \frac{QK}{\sqrt{C_{bh}}}}\right)],{\mathrm{A}}_T\in {\mathbb{R}}^{C_b\times T\times T},$$

(15)

where $\mathsf{\Lambda }$ denotes a convolution layer used to fuse the information from multiple heads.

Finally, feature extraction is performed on the initial input, and our GCTT module is used to aggregate these features, resulting in a feature representation that captures temporal topological relationships:

$$\left.Y=X+g\times \left(\mathrm{einsum}(’\mathrm{nctt},nctv\to nct{v}^{\prime },errorConv\left(X\right)\right)\right)$$

(16)

Here, X denotes the original input, and g represents the gating coefficient. The einsum operation indicates a tensor contraction over the temporal dimension.

We provide a further explanation for the gating coefficient: Our gating coefficient ‘gate’ is defined as a learnable parameter, initialized to one. As shown in Figure 4, after extracting features using the channel-wise temporal topology, the ‘gate’ controls the output. During training, the value of ‘gate’ is continuously updated. On top of the residual connection that preserves the original input information, it controls the influence of the temporal topology on the final result. In other words, the ‘gate’ can adjust the strength of the impact of the proposed GCTT on the outcome, thus finding an optimal level of influence.

3.4. Model Architecture

The basic block of our model is illustrated in the left part of Figure 5. Each Graph Convolutional Network (GCN) consists of three parallel Multi-View Topology Refinement Graph Convolution (MTRGC) modules, whose outputs are summed. Each Temporal Convolution Network (TCN) is implemented using a MultiScale Temporal Convolution (MSTC) module. A basic block is composed of one GCN and one TCN. The complete model is illustrated in Figure 6. Our model is constructed with nine basic blocks, followed by a global average pooling layer and a fully connected layer to generate the final recognition results.

Figure 5. The left side shows our basic block, which consists of a GCN and a TCN, while the right side illustrates the structure of the TCN.

Figure 5. The left side shows our basic block, which consists of a GCN and a TCN, while the right side illustrates the structure of the TCN.

Figure 6. Overall architecture of our model.

Figure 6. Overall architecture of our model.

4. Experiments

4.1. Datasets

NTU RGB + D. NTU RGB + D [21] is one of the most classical benchmarks for skeleton-based human action recognition. It contains a total of 56,880 skeleton sequences spanning 60 action categories, including daily activities, health-related actions, and person-to-person interactions. The actions were performed by 40 subjects aged between 10 and 35 and were simultaneously recorded from three different horizontal angles using Microsoft Kinect v2 sensors. Each skeleton sequence consists of up to two subjects, and each human body is represented by 25 3D joints tracked over time. To evaluate the generalization ability of models, two standard protocols are adopted: (1) Cross-Subject (X-Sub): A total of 20 subjects are used for training and the remaining 20 for testing. (2) Cross-View (X-View): Samples captured from camera views 2 and 3 are used for training, while those from camera view 1 are reserved for testing.

NTU RGB + D 120. NTU RGB + D 120 [22] is an extension of the original NTU RGB + D dataset and is currently the largest skeleton-based human action recognition benchmark. It consists of 113,945 skeleton sequences spanning 120 action classes, including both the original 60 classes and an additional 60 newly introduced ones. The actions were performed by 106 subjects and were captured using Microsoft Kinect v2 sensors across 32 distinct camera setups, each representing a different location and background environment. To ensure a fair and comprehensive evaluation, the dataset provides two standard evaluation protocols: (1) Cross-Subject (X-Sub): The subjects are evenly split, with 53 used for training and the remaining 53 for testing. (2) Cross-Setup (X-Set): The dataset is divided based on the setup IDs, where even-numbered setups are used for training and odd-numbered setups for testing.

Northwestern-UCLA. The Northwestern-UCLA dataset [23] is a multi-view skeleton-based action recognition dataset comprising 1494 video clips across 10 action classes, each performed by 10 different subjects. The data was captured simultaneously from three distinct viewpoints using Kinect cameras, providing 3D skeleton sequences composed of 20 joints per subject. The dataset follows a cross-view evaluation protocol recommended by the authors: sequences captured from the first two cameras are used for training, while those from the third camera are used for testing

4.2. Implementation Details

All experiments were conducted on one RTX 3090 GPU with the PyTorch 2.0.0 deep learning framework. Our models are trained with Stochastic Gradient Descent (SGD) with momentum 0.9 and weight decay 0.0004. The training epoch is set to 110 and a warmup strategy is used in the first 5 epochs to make the training procedure more stable. The learning rate is set to 0.05 and decays with a factor 0.1 at epoch 90 and 100. For NTU RGB + D [21] and NTU RGB + D 120 [22], the batch size is 64, each sample is resized to 64 frames, and we adopt the data pre-processing in CTRGCN [16]. For Northwestern-UCLA [23], the batch size is 16, and we also adopt the data pre-processing in [30].

4.3. Ablation Study

The experiments in this subsection were conducted based on the X-sub benchmark of the NTU RGB + D 120 dataset [22] using the joint modality. The results demonstrate the effectiveness of each proposed GCN and TCN module.

We adopt ST-GCN [14] as our baseline, using the same official implementation provided by CTRGCN [16]. ST-GCN employs non-adapative topology for aggregating spatial features (GCN) and utilizes conventional 2D convolutions for temporal modeling (TCN). However, it suffers from significant limitations in both topology modeling and feature extraction. Building upon ST-GCN, we introduce our model and demonstrate through extensive experiments that our approach achieves advantages of lightweight design, high computational efficiency, and superior recognition accuracy.

As shown in

Table 1. Ablation study results. ‘✓’ indicates the module is used, ‘-’ indicates it is not used, and ‘↑’ indicates the improvement in percentage points.

Model	Multi-View		MSTC	GCTT	Params	Flops	Acc(%)
	Dynamic	Static
STGCN (baseline)	-	-	-	-	2.09 M	2.34 G	84.6
+ dynamic only	✓	-	-	-	2.25 M	2.54 G	85.0 (↑0.4)
+ static only	-	✓	-	-	2.09 M	2.34 G	84.9 (↑0.3)
+ Multi-view	✓	✓	-	-	2.26 M	2.54 G	85.7 (↑1.1)
+ MSTC	✓	✓	✓	-	1.27 M	1.55 G	85.6
Whole model	✓	✓	✓	✓	1.37 M	1.65 G	86.1 (↑1.5)

,

• Adding only the dynamic topology on top of the baseline improves the accuracy by 0.4%, demonstrating that our pairwise modeling strategy for dynamically capturing spatial joint relationships is highly effective.
• Adding only the static topology improves the accuracy by 0.3%, indicating that modeling the skeletal topology based on the distance between joints can effectively preserve static connection information.
• Incorporating the multi-view topology leads to a 1.1% improvement in accuracy, which exceeds the combined gains from 1 and 2. This result proves that the two views are not merely additive but synergistic, validating both the necessity and effectiveness of multi-view topology modeling, as well as the generalizability and universality of the static topology.
• After introducing the MSTC module, the accuracy slightly drops by 0.1%; however, the model size is reduced by 44% (0.99 M parameters), and FLOPs decrease by 39% (0.99 G), demonstrating the lightweight nature of the proposed design.
• Finally, with the addition of the GCTT module, the accuracy improves by 1.5% compared to the baseline, proving that our channel-wise temporal topology modeling effectively captures the relationships between different temporal frames, further enriching temporal feature extraction based on MSTC.

Also, we conduct experiments on the kernel sizes used in the depthwise convolutions within the MSTC module. The results in

Table 2. Ablation study on TCN kernel sizes.

Kernel Size 1	Kernel Size 2	Acc (%)
5	5	85.7
5	7	86.1
7	7	85.2
7	9	85.6
9	9	85.5

show that using different kernel sizes for the two branches is more beneficial for capturing motion features across different temporal scales. Furthermore, in order to compensate for the limited receptive field of depthwise convolutions, we adopt a larger kernel size of 7 for the second branch.

4.4. Comparison with State-of-the-Art

We follow the same fusion strategy as in CTRGCN [16] and same ensemble strategy in FRhead [18], combining four modalities: joint, bone, joint motion, and bone motion for comparison.To ensure fairness and eliminate randomness, we follow the fixed training protocol of CTRGCN. All experimental results of our model are obtained using a single RTX 3090 GPU. We compare our models with the state-of-the-art methods on NTU RGB + D 120, NTU RGB + D in

Table 3. Comparison with state-of-the-art methods on NTU60 and NTU120 datasets (2S: fusion of joint and bone streams; 4S: fusion of joint, bone, joint-motion, and bone-motion streams).

Methods	NTU60-XSub	NTU60-XView	NTU120-XSub	NTU120-XSet
STGCN [14]	81.5	88.3	–	–
SGN [31]	89.0	94.5	79.2	81.5
AS-GCN [32]	86.8	94.2	–	–
2s-AGCN [15]	88.5	95.1	–	–
DGNN [13]	89.9	96.1	–	–
Shift-GCN [30]	90.7	96.5	85.9	87.6
MS-G3D [29]	91.5	96.2	86.9	88.4
Dynamic-GCN [33]	91.5	96.0	87.3	88.6
MST-GCN [20]	91.5	96.6	87.5	88.8
CTRGCN [16]	92.4	96.8	88.9	90.6
InfoGCN (4S) [24]	–	–	89.4	90.7
Efficient-G4 [19]	92.1	–	88.7	88.9
FRhead [18]	92.8	96.8	89.5	90.9
MTR-GCN (2S)	92.3	96.4	89.2	90.4
MTR-GCN (4S)	92.8	96.8	89.6	90.8

and NW-UCLA in

Table 4. Comparison with state-of-the-art methods on NW-UCLA dataset.

Methods	NW-UCLA (%)
Lie Group [34]	74.2
HBRNN-L [35]	78.5
Actionlet Ensemble [36]	76.0
Ensemble TS-LSTM [37]	89.2
AGC-LSTM [38]	93.3
Shift-GCN [30]	94.6
DC-GCN + ADG [39]	95.3
MTR-GCN (Ours)	95.7

. On the NTU120 dataset, our model outperforms the state-of-the-art methods under both cross-subject and cross-setup settings. As shown in the Figure 7, our model achieves this with significantly fewer parameters and lower computational cost. Furthermore, Table 3 demonstrates that the fusion of joint and bone modalities in our model achieves performance comparable to the best results. As shown in Table 4, our model also surpasses a wide range of state-of-the-art methods on the NW-UCLA dataset. The comparison with state-of-the-art methods demonstrates that our model is a lightweight, efficient, and high-performance solution.

5. Conclusions

This work addresses the limitations in spatial and temporal modeling for skeleton-based action recognition. For spatial modeling, we propose the Multi-view Topology Refinement Graph Convolution (MTRGC), which integrates both dynamic and static perspectives to overcome the issues of catastrophic forgetting of skeletal topology and insufficient relational modeling capacity in conventional GCNs. Experimental results demonstrate that MTRGC achieves a synergistic effect—greater than the sum of its individual views—rather than a simple additive gain. For temporal modeling, we introduce the MultiScale Temporal Convolution (MSTC), which enables lightweight design without compromising accuracy; building on this, we propose Gated Channel-wise Temporal Topology (GCTT) to perform topological modeling along the temporal dimension, effectively enhancing temporal feature extraction.

Our model achieves state-of-the-art performance across multiple benchmarks. However, there still exists the issue of incomplete feature extraction. It remains a challenge whether better skeleton features can be extracted using methods other than topology modeling, or if improvements can be made in data preprocessing. These are the challenges we face. Future work may focus on further improving training efficiency and exploring more advanced multi-relational modeling techniques.

6. Visualization

As shown in Figure 8, the Static Topo exhibits a clear diagonal pattern, indicating strong self-connections. The presence of brighter regions near the diagonal further suggests that it effectively captures the relationships between adjacent joints. Overall, its structure is regular and well-aligned with the true human skeleton topology. In contrast, the Dynamic Topo appears more random, reflecting a topology learned entirely from inter-joint relations without relying on any prior skeletal structure. Our Multi-view Topo retains the prior structural knowledge of the original skeleton while dynamically learning the global topology. This results in a more regular and smoother distribution of weights, effectively reducing the impact of noise.

As shown in Figure 9, we visualize the static topology across different layers. It can be observed that our static topology is continuously optimized, exhibiting distinct representational characteristics at each layer. Through iterative refinement and updates, the model ultimately learns an optimal static topology representation.