ActionMamba: Action Spatial–Temporal Aggregation Network Based on Mamba and GCN for Skeleton-Based Action Recognition

Abstract

Skeleton-based action recognition networks have widely adopted the approach of Graph Convolutional Networks (GCN) due to their superior capabilities in modeling data topology, but several key issues still require further investigation. Firstly, the graph convolutional network extracts action features by applying temporal convolution to each key point, which causes the model to ignore the temporal connections between different important points. Secondly, the local receptive field of graph convolutional networks limits their ability to capture correlations between non-adjacent joints. Motivated by the State Space Model (SSM), we propose an Action Spatio-temporal Aggregation Network, named ActionMamba. Specifically, we introduce a novel embedding module called the Action Characteristic Encoder (ACE), which enhances the coupling of temporal and spatial information in skeletal features by combining intrinsic spatio-temporal encoding with extrinsic space encoding. Additionally, we design an Action Perception Model (APM) based on Mamba and GCN. By effectively combining the excellent feature processing capabilities of GCN with the outstanding global information modeling capabilities of Mamba, APM is able to comprehend the hidden features between different joints and selectively filter information from various joints. Extensive experimental results demonstrate that ActionMamba achieves highly competitive performance on three challenging benchmark datasets: NTU-RGB+D 60, NTU-RGB+D 120, and UAV–Human.

1. Introduction

Action recognition is a fundamental task in video understanding. In earlier research efforts, a variety of modalities have been employed to represent features, including RGB frames [1], optical flow information [2,3], and human skeleton representations [4,5]. Among these different modalities, action recognition methods that rely on skeletal data have attracted growing attention, and this is largely due to their unique focus on the coordinate sequences of human joints. In the early stages of relevant research, skeleton-centered methodologies mainly converted skeletal data into sequences of joint vectors or pseudo-images. These converted representations were subsequently used to train Convolutional Neural Networks (CNNs) [6] or Recurrent Neural Networks (RNNs) [7]. Nevertheless, these vector-based sequences had limitations: they lacked the ability to model the structural connections between keypoints and failed to sufficiently capture the spatial dependencies that exist between adjacent joints. To address this problem, Graph Neural Networks (GNNs) were introduced into the field. Specifically, the research by [8] proposed a framework where human joints and their physical connections are defined as nodes and edges in a graph, respectively. On this basis, Graph Convolutional Networks (GCNs) are applied to extract spatio-temporal features from the pre-defined graph structure. Through the process of aggregating information from connected nodes, this method enables the direct modeling of joint adjacency and the interactions between joints. Furthermore, skeleton-based technical approaches show remarkable robustness when facing contextual variations such as background noise and changes in lighting conditions. This robustness stems from the compact and low-dimensional nature of skeletal data representations. These inherent advantages of skeleton-based methods lay a solid foundation for the advancement of subsequent research in the domain of skeleton-based action recognition.

For improved modeling of interactions among distant joints, AS-GCN was introduced by [9]. This model enhances classification results in action recognition by using fundamental blocks that combine action–structure graph convolution with temporal convolution for learning features across space and time. In an effort to uncover latent connections between joints, Ref. [10] utilized a neural architecture search method to refine the GCN model’s structure. Their method expands the search parameters by incorporating a range of dynamic graph substructures and higher-order connections, and it utilizes Chebyshev polynomial approximations to implicitly represent the associations between joints. The work from [11] successfully captures dependencies over extended ranges by integrating contextual information via feature fusion. With the aim of lowering the computational burden of GCNs, Shift-GCN was presented by [12], adapting the principle of shift convolution for graph-based networks. This architecture leverages shift-graph operations combined with lightweight point-wise convolutions to create adaptable receptive fields for spatial and temporal graph convolutions, thereby augmenting representational power while decreasing complexity. Furthermore, Ref. [13] put forward a multi-stream GCN design that merges input from joint positions, motion velocities, and bone characteristics in its initial layers. The framework utilizes separable convolution layers alongside composite scaling methods to diminish superfluous parameters and concurrently enhance the model’s overall capacity.

Subsequently, inspired by Transformers, Ref. [14] decomposed the data into spatial and temporal dimensions and proposed the Decoupled Spatial–Temporal Attention Network (DSTA-Net). This network encodes the two streams sequentially with attention modules, modeling the spatial and temporal dependencies between joints without involving positional or mutual connection information. Ref. [15] introduced a multimodal Transformer-based action recognition network that fuses the encoding results of spatial–temporal skeletal models and acceleration models. To leverage the advantages of both Transformers and GCNs simultaneously, Ref. [16] proposed a multi-order multimodal Transformer (3Mformer) that applies high-order Transformers to process skeletal data’s spatio-temporal features to better capture higher-order motion patterns between joints. SkeleTR [17] initially captures individual dynamic information with GCNs and then models human interaction with stacked Transformer encoders. Ref. [18] proposed a new triple attention module (TAM) to guide GCN in perceiving significant changes in local motion. BlockGCN [19] encodes bone connections by utilizing the feature of graph distance to describe physical topology, preserving important topological details that are often lost in traditional GCNs and improving the performance of the original GCN.

Despite the promising results achieved in previous research, existing technical methods still exhibit significant limitations, particularly in balancing efficiency and effectiveness when addressing behavior recognition tasks. GCN-based approaches face challenges in capturing global contextual information due to their inherently limited receptive fields—a constraint that hinders the development of high-quality skeletal features. In contrast, Transformer-based models and hybrid frameworks excel at global feature modeling but are constrained by considerable computational costs. This elevated cost is mainly caused by the self-attention mechanism, whose complexity exhibits a quadratic relationship with the number of tokens.

Fortunately, the improved S4 model [20], namely Mamba [21], offers a new solution to the aforementioned problems through selection mechanisms and parallel algorithm design. It facilitates the model to attain Transformer’s long-distance context modeling capabilities while maintaining linear complexity. Recently, some pioneering work has applied Mamba to tasks such as remote sensing [22,23], visual recognition [24,25], and medical imaging [26,27], but the potential of Mamba in action recognition tasks has not been fully explored due to the lack of designs like cross-attention.

Therefore, we propose an Action Spatio-temporal Aggregation Network (ActionMamba) based on GCN and Mamba. It mainly consists of two parts: the Action Characteristic Encoder (ACE) and the Action Perception Model (APM). The Action Characteristic Encoder, consists of Inner Spatial–Temporal Embedding and External Space Embedding. By introducing explicit coordinate features and emphasizing the importance of the intrinsic skeleton in the input features, this module significantly improves the spatial geometric modeling effect of the model and mitigates the model’s disregard for spatio-temporal connections between different key points. In the network’s backbone stage, we designed the Action Perception Model. To fully utilize GCN’s excellent early-stage processing power and Mamba’s efficiency in extracting remote features, we divided APM into two stages. First, the skeletal features are fed into the Shift Mamba-GCN module (SM-GCN), which expands the graph convolutional receptive field and extracts intrinsic skeletal features through shift graph convolution and Mamba block processing. This endows the model with the capability to capture implicit dependencies among different joints. Subsequently, we introduce the Spatio-Temporal Mamba block (ST-Mamba), which integrates residual connections with state space modeling (SSM) modules. This design is engineered to capture long-term relationships at both the intra-frame and inter-frame levels, enabling the acquisition of feature representations that are more precise, discriminative, and resilient. To assess the performance of our proposed model, comprehensive experiments were performed using three benchmark datasets for skeleton-based action recognition: NTU RGB+D 60, NTU RGB+D 120, and UAV–Human.

The primary contributions of this work are outlined as follows:

• We propose ActionMamba, an action spatio-temporal aggregation network featuring a global receptive field and dynamic weighting mechanism. By integrating Mamba with a Graph Convolutional Network, ActionMamba adaptively captures global spatio-temporal properties of human skeletons with linear complexity, thereby improving model performance.
• To boost the model’s comprehension of spatial relationships, we have devised an efficient action feature encoder. This module incorporates extrinsic skeletal characteristics and attention-based mechanisms, facilitating a more accurate representation of the human body’s structural configuration.
• Furthermore, to discern implicit relationships among joints, we put forward a novel Action Perception Model. This new model synergizes the global feature extraction capacity of Mamba with the local feature aggregation proficiency of GCNs.
• Extensive experiments conducted on multiple benchmark datasets demonstrate that our approach consistently outperforms existing state-of-the-art methods.

The structure for the remainder of this manuscript is as follows. In Section 2, we provide a review of the pertinent literature. Section 3 is dedicated to detailing our methodology, encompassing the SSM model and our proposed framework, which consists of the APM and ACE modules. The comprehensive evaluation and the resulting outcomes are presented in Section 4. Finally, Section 5 concludes the paper by summarizing our proposed contributions and suggesting avenues for future investigation.

2. Related Work

2.1. GCN-Based Action Recognition

Skeleton-based action recognition has garnered considerable attention in research due to the compactness of skeleton data and the strong correlations among features. Graph Convolutional Networks, particularly ST-GCN [8], have achieved remarkable success in this domain by exploiting the intrinsic connectivity and feature correlation of the human body. This model pioneered graph convolutions and temporal convolutions, which encode the connections among skeleton joints using adjacency matrices and extract the dynamic information of individual joints in the temporal dimension through 1D convolution. To enhance the adaptability of the skeleton graph, AGCN [28] introduced an adaptive adjacency matrix that not only emphasizes existing skeletal connections but also captures potential relationships between nodes. By focusing on contextually related intrinsic topology modeling, Dynamic GCN [29] integrates the contextual features of all joints into the learning of joint relationships. Channel-Topology Refined GCN (CTR-GCN) [5] focuses on embedding joint topology across different channels. In contrast, InfoGCN [30] introduces an attention-based graph convolution mechanism that captures contextually relevant topologies through potential representations learned via the information bottleneck principle. Spatial Graph Diffusion Convolution (S-GDC) networks [31] aim to learn dynamic graph structures through diffusion, enabling the model to capture both long-range dependencies within a single body and interactions between multiple bodies. In summary, while GCN-based approaches are effective at modeling joint interactions via topological structures, they often remain constrained to local spatial–temporal regions.

2.2. Transformers-Based Action Recognition

The Transformer architecture, originally developed for machine translation tasks in the natural language processing domain, subsequently had its range of applications broadened to include the field of computer vision. The Vision Transformer (ViT) [32] took the lead in this transition, as it adopts Transformer encoders to extract feature information from images. Compared with GCN-based methods, Transformer-centered approaches offer more efficient global topological modeling capabilities and can better emphasize the significance of joints that have no physical adjacency. To make full use of the self-attention mechanism, DSTA-Net [14] proposed a decoupled framework specifically designed for spatio-temporal self-attention. Following a comparable research direction, STAR [33] combines sparse attention applied in the spatial domain with segmented linear attention used in the temporal domain; this combination allows it to efficiently handle skeleton action sequences with variable lengths. Ref. [15] put forward a multimodal action recognition network based on Transformers, which integrates the encoded feature representations derived from both spatio-temporal skeleton models and acceleration-focused models. As a hybrid approach, SkeleTR [17] initially leverages GCNs to capture the dynamic information of individual skeletons, after which it utilizes stacked Transformer encoders to model the interaction relationships among humans. While Transformer-based methods are capable of capturing global topological information and emphasizing the relationships between non-adjacent joints, the self-attention mechanism they depend on introduces computational complexity that exhibits a quadratic relationship with the number of tokens. This intrinsic characteristic significantly restricts the scalability of these methods and their practical deployment in real-world application scenarios [34].

2.3. State Space Model

Benefiting from its excellent mathematical properties, the state space model (SSM) has gained significant attention in recent times. A key distinguishing feature of this model is its ability to effectively retain historical input information, while only relying on the latest input during both training and inference processes. When compared with recurrent neural networks (RNNs) and Transformers, architectures constructed based on SSMs have shown significantly better performance in tasks requiring long-context understanding—with the Long-Range Arena (LRA) benchmark [35] serving as a typical example. In recent developments, prominent models like Mamba [21] have demonstrated improved performance and higher computational efficiency compared to advanced Transformer models in long-context scenarios, its structure is shown in Figure 1. This achievement strongly confirms the great potential of SSMs to act as core architectural components. Consequently, the application scope of Mamba has expanded beyond its initial use in natural language processing to cover a range of other fields. For example, within the Vision Transformer framework, Vim [25] makes use of Mamba to develop a new, general-purpose vision backbone, which is built on bidirectional Mamba blocks. This particular backbone embeds positional data into the sequence of images and utilizes the bidirectional state space model for compressing visual feature representations. For the specific application of biomedical image segmentation, a hybrid architecture known as U-Mamba was introduced by [36]. This model combines convolutional networks with state space models and was explicitly created to manage long-range dependencies. For multimodal image fusion tasks, [37] incorporated Mamba to capture long-range features from single-modal images: it first extracts low-level features using CNNs and high-level features using Mamba blocks, and then utilizes a reconstruction module to produce the fused output. Within the domain of action recognition, Simba [38] has also presented a skeleton-based action recognition model based on the U-ShiftGCN framework. This method embeds Mamba blocks into the core part of the U-ShiftGCN architecture to process temporal features between skeletal connections.

3. Method

In Section 3.1, we introduce the basics of the SSM and outline the motivation behind designing ActionMamba. Next, in Section 3.2, we describe the Action Characteristic Encoder, and in Section 3.3, we present the architectural details of the Action Perception Model.

3.1. Preliminaries

State space models originated from classical control theory and were initially utilized in the fields of control and computational neuroscience for modeling dynamic system frameworks. They were later attempted to be discretized through reconstruction and convolution for computational processing. In recent years, SSMs have gained popularity among researchers due to their exceptional performance in analyzing long-term data. Particularly, an SSM defines a linear mapping from an input signal, denoted as $x\left(t\right)$ (a function of time t), to an output signal, denoted as $y\left(t\right)$, through a latent state $h\left(t\right)$. This relationship can be represented by Equation (1):

$$\begin{array}{cc}\hfill h^{\prime }\left(t\right)& =\mathbf{A}h\left(t\right)+\mathbf{B}x\left(t\right),\hfill \\ \hfill y\left(t\right)& =\mathbf{C}h\left(t\right)+\mathbf{D}x\left(t\right).\hfill \end{array}$$

(1)

where $h\left(t\right),h^{\prime }\left(t\right)\in {\mathbb{R}}^{d\times 1}$ represent complex-valued states, $x\left(t\right),y\left(t\right)\in {\mathbb{R}}_q$ represent input and output, respectively, and t represents a continuous time index. $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, $\mathbf{D}$ are complex-valued matrices of appropriate dimensions. Additionally, the SSM model requires the discretization of model parameters before being applied to deep learning algorithms. Specifically, the continuous parameters $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, $\mathbf{D}$ are transformed into their discrete counterparts $\overline{A},\overline{B},\overline{C},\overline{D}$ using a time scale $\mathrm{\Delta }\in \mathbb{R}$ and the Zero-order Hold method, as depicted in Equation (2):

$$\begin{array}{ccc}\hfill \overline{A}=& exp(\mathrm{\Delta }\mathbf{A}),\overline{B}=\hfill & \hfill {(\mathrm{\Delta }\mathbf{A})}^{-1}exp(\mathrm{\Delta }\mathbf{A}-\mathbf{I})\mathrm{\Delta }\mathbf{B}\approx \mathrm{\Delta }\mathbf{B},\\ \hfill \overline{C}=& C,\overline{D}=D.\hfill \end{array}$$

(2)

For the ease of subsequent discretization treatment, we refer to this process as Disc.

$$\begin{array}{c}\hfill \overline{\mathbf{A}},\overline{\mathbf{B}},\overline{\mathbf{C}},\overline{\mathbf{D}}=Disc\left(\mathbf{x}\right)\end{array}$$

(3)

The discrete state space model reformulates Equation (1) as follows:

$$\begin{array}{cc}\hfill h_i& =\overline{A}{h}_{i-1}+\overline{B}{x}_i,\hfill \\ \hfill y_i& =\overline{C}{h}_i+\overline{D}{x}_i.\hfill \end{array}$$

(4)

where, $\overline{A},\overline{B},\overline{C},$ and $\overline{D}$ are matrices representing discrete-time dynamics that have been discretized using the time step $\mathrm{\Delta }\in \mathbb{R}$, and i represents a discrete time index. Additionally, Mamba proposed a modified version of a parallel scanning algorithm designed specifically for training and inference. This update effectively overcomes the time-varying nature of the SSM and addresses the issue of high computing cost in real applications.

Despite Mamba’s impressive efficiency in handling various visual task inputs, its introduction also presents significant challenges. Firstly, Mamba inherently operates in a causal manner, requiring images to be converted into shifted sequences and processed cyclically. This limits the receptive field of each token to its preceding sequence and introduces additional latency. Secondly, as mentioned by [39], Mamba’s choice of the state space model resembles a single-head linear attention mechanism and does not incorporate a multi-head design. Although this strategy increases efficiency, it may not be appropriate for non-causal data structures which include graph structures.

As a result of this, we redesigned the Mamba structure. Mamba is effectively applied to non-causal data structures by combining global shift convolution and the Mamba structure, resulting in omnidirectional spatio-temporal feature mixing. Furthermore, to compensate for the reduction in receptive field produced by the calculation of the Mamba shift sequence, we incorporated an action feature encoder. By incorporating the fundamental properties of joints as well as the intrinsic spatio-temporal connection, Mamba gains local bias and location information, which improves model performance.

3.2. Action Characteristic Encoder

In action recognition tasks, the spatial position and associated features of the skeleton play a crucial role due to the significant variations in data from human body joints across different actions. To capture and incorporate the dynamic spatial–temporal relationship inherent in human skeleton data, we have designed the Action Characteristic Encoder module. As illustrated in Figure 2a, we employ Inner Spatial–Temporal Embedding and External Space Embedding to embed the input human skeleton features.

Inner Spatial–Temporal Embedding. The temporal information’s shifting rhythm is closely linked to skeletal features, hence reliable spatial–temporal encoding is critical for action recognition tasks. To effectively integrate and process these spatial–temporal dynamics, we propose the Inner Spatial–Temporal Embedding module (ISTE). This module employs importance as a guiding factor to aggregate spatial–temporal information among skeletons. As shown in Figure 2a, the primary idea behind the intrinsic spatial–temporal connection unit involves calculating attention scores between adjacent vertices using an attention mechanism. Subsequently, the vertex features are aggregated and updated. This operation prioritizes nodes that carry essential features and selectively establishes implicit dependencies by aggregating joint features over long distances. As a result, the aggregate representation $z_i$ of node i can be expressed as:

$$\begin{array}{c}\hfill {\mathbf{z}}_i^{\left(t\right)}=ReLU\left({\alpha }_{i,i}\mathbf{W}{\mathbf{x}}_i^{\left(t\right)}+\sum _{j\in \mathcal{N}\left(i\right)}{\alpha }_{i,j}\mathbf{W}{\mathbf{x}}_j^{\left(t\right)}\right)\end{array}$$

(5)

where, ${\mathbf{x}}_i^{\left(t\right)}\in {\mathbb{R}}^w$ represents the input features of node $\mathrm{i}$. $\mathcal{N}\left(i\right)=\left\{j\mid A_{ji}>0\right\}$ indicates the collection of neighboring nodes of $\mathrm{i}$ obtained from the learned adjacency matrix A. $\mathrm{W}$ is initialized using a uniform initialization method and trained as a model parameter. The calculation of the attention coefficients ${\alpha }_{i,j}$ is shown in Equation (6):

$$\begin{array}{cc}\hfill \pi (i,j)& =LeakyReLU\left({\mathbf{a}}^{\top }\left(\mathbf{W}{\mathbf{x}}_i^{\left(t\right)}\oplus \mathbf{W}{\mathbf{x}}_j^{\left(t\right)}\right)\right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\alpha }_{i,j}& ={\displaystyle \frac{exp\left(\pi \right(i,j\left)\right)}{{\sum }_{k\in \mathcal{N}\left(i\right)\cup \left\{i\right\}}exp\left(\pi (i,k)\right)}}\hfill \end{array}$$

(7)

where, ⊕ denotes the concatenation operation, and $\mathbf{a}$ represents trainable coefficient vector. We employ the LeakyReLU activation function to introduce non-linearity when computing attention coefficients, which are subsequently normalized using the softmax function.

External Space Embedding. Following the ISTE module, we concatenate the C-dimensional features with the X, Y, and Z coordinates of the 3D skeletal data to help the model better understand the spatial positions of individual joints. Detailed specifics are illustrated in Figure 2a. This strategy enables the network to explicitly observe local geometric patterns of the human body, thereby facilitating the learning of its structural configuration across the entire network. The final integrated output feature $\mathrm{Z}\in {\mathbb{R}}^{N\times (C+3)\times T}$ is expressed in Equation (8):

$$\begin{array}{c}\hfill {\mathbf{Z}}_i^{\left(t\right)}=Concat\left({\mathbf{z}}_i^{\left(t\right)},{\mathbf{z}}_i^{\left(3\right)}\right)\end{array}$$

(8)

where, Concat represents the concatenation operation. The ${\mathbf{z}}_i^{\left(3\right)}$ vector belongs to the set ${\mathbb{R}}^{N\times 3\times T}$ and represents the X-Y-Z coordinates of the three-dimensional skeleton data.

The rationale behind our External Space Embedding can be understood from the work of [12], where Shift-GCN illustrates that augmenting skeletal data with spatial coordinates can substantially improve the model’s capacity for geometric representation. Our method adopts a similar principle. However, in contrast to the architecture proposed by [40], we confine the concatenation of the skeleton’s X-Y-Z coordinates with the input features to the entry point of the network’s first block, rather than applying this structure globally. Although this technique involves a less explicit encoding of external spatial position, the Mamba model’s exceptional proficiency in capturing long-range dependencies allows the network to more accurately discern geometric patterns within the human form. This assertion is empirically validated through the experiments detailed in Section 4.4.2.

3.3. Action Perception Model

To develop a Mamba architecture suitable for non-causal data, we integrated the Shift-GCN design with Mamba and proposed the Action Perception Model (APM). On one hand, the model leverages Shift-GCN’s flexible receptive field over spatial and temporal graphs, enabling it to adaptively capture implicit relationships among local joints. On the other hand, it benefits from Mamba’s selective scanning mechanism and dynamic adjustment capabilities, which help model long-range spatio-temporal dependencies within human skeletal structures. The overall structure of the APM is depicted in Figure 2. The APM comprises two main stages: Shift Mamba-GCN Block (SM-GCN) and Spatial–Temporal Mamba Block (ST-Mamba).

Shift Mamba-GCN Block. The Shift-GCN method utilizes non-local shift graph operations, enabling each node to collect information from all other nodes. However, this approach uniformly blends the features across different nodes and artificially equalizes the connection strength between them. Consequently, we have introduced a more effective module known as the Shift Mamba-GCN Block. This module establishes global joint dependency relations via a structural state space model and diminishes the influence of insignificant nodes at each iteration, leading to the heterogeneous mixture modeling of features.

As depicted in Figure 2b, the SM-GCN block represents a fusion of Shift-GCN and Mamba, incorporating Spatial-Mamba (S-Mamba) and Temporal-Mamba (T-Mamba) modules to improve the integration of spatio-temporal data. More precisely, for a given spatial skeleton feature map designated as $\mathbf{F}\in {\mathbb{R}}^{C\times T\times N}$, N corresponds to the quantity of skeleton points and C denotes the feature channel count. The shift displacement for the i-th channel is calculated as $i;mod;N$. This feature map is then processed by an S-Shift block, where a Shift operation and a subsequent convolution transform it into a spiral feature map containing global information, as illustrated in Figure 3. Next, these spiral features are input to the S-Mamba module. Within this block, Mamba establishes long-range dependencies at both intra-frame and inter-frame levels. By merging local detailed features from various joints, this process ultimately yields an omnidirectional, mixed spatio-temporal feature. In contrast, the T-Mamba module is designed to aggregate temporal information across the skeleton by inverting the spatial and temporal selection scanning mechanisms.

The computational process of the S-Mamba block is depicted in Figure 4, and the pseudocode is presented in Algorithm 1. To effectively utilize the scanning mechanism of Mamba and the advantages of hardware-aware parallel algorithms, we reshape the input feature ${\mathbf{F}}_{in}\in {\mathbb{R}}^{C\times T\times N}$ through a multilayer perceptron (MLP) into tensors X and Z. This process is designed to align and concatenate slices from each time step t, as shown below:

$$\begin{array}{c}\hfill \mathbf{X},\mathbf{Z}={\mathbf{Conv}}_x\left(\mathbf{Norm}\left({\mathbf{F}}_{in}\right)\right),{\mathbf{Conv}}_z\left(\mathbf{Norm}\left({\mathbf{F}}_{in}\right)\right).\end{array}$$

(9)

where $\mathbf{Norm}(·)$ represents layer normalization, and ${\mathbf{Conv}}_x(·)$ and ${\mathbf{Conv}}_z(·)$ represent two convolutional layers. Afterwards, $\mathbf{X}$ is flattened in three directions, producing $x_0$, $x_1$, and $x_2$. The dimensions of $x_0$ and $x_1$ are $\mathrm{CT}\times \mathrm{N}$, and $x_2$ is $\mathrm{CN}\times \mathrm{T}$. By reshaping the data along different dimensions, we obtain a new set of feature vectors that help the model to process spatial and temporal information in conjunction. This allows for a more effective capturing of complex patterns in the data. Subsequently, these 1D sequences are processed through 1D convolution and SSM blocks for feature extraction, resulting in three outputs: $y_0$, $y_1$, and $y_2$.

$$\left\{\begin{array}{c}{x}_i={\mathrm{Flatten}}_i\left(\mathbf{X}\right),\hfill \\ y_i={\mathbf{SSM}}_i\left(Con{v}_i\left(x_i\right)\right).\hfill \end{array}\right.i=0,1,2.$$

(10)

where ${\mathrm{Flatten}}_i$ denotes the flattening operation in the ith direction, ${\mathbf{SSM}}_i$ represents the ith SSM block. In Section 3.1, we detailed the process and discretization of the SSM blocks. Subsequently, we unflatten the outputs of the SSM blocks to obtain $y_0$, $y_1$, and $y_2$. The features are then gated and added with Z to derive the fused output ${\mathbf{F}}_{\mathrm{mid}}\in {\mathbb{R}}^{C\times T\times N}$.

$$\left\{\begin{array}{c}{\mathbf{Y}}_i={\mathrm{UnFlatten}}_i\left(y_i\right),\hfill \\ {\mathbf{F}}_{mid}=\sum _i^2\left({\mathbf{Y}}_i·\mathrm{SiLU}\left(\mathbf{Z}\right)\right)\hfill \end{array}\right.i=0,1,2.$$

(11)

where ${\mathrm{UnFlatten}}_i$ represents the unfolding operation along the i-th direction. $\mathrm{SiLU}(·)$ denotes the SiLU activation function. The final output of the module ${\mathbf{F}}_{out}\in {\mathbb{R}}^{C\times T\times N}$ can be described as:

$$\begin{array}{cc}\hfill {\mathbf{F}}_{out}& ={\mathbf{Conv}}_o\left({\mathbf{F}}_{mid}\right).\hfill \end{array}$$

(12)

where ${\mathbf{Conv}}_o(·)$ represents the convolutional layer. In contrast to adaptive time-shifted graph convolution [12] and dynamic offset graph convolution [40], SM-GCN emphasizes the potential features between joints and connections. The comparison with other traditional methods is shown in Figure 5. This allows the network to prioritize important joints after the feature shift transformation. Therefore, this method not only addresses the coarse kernel space exploration problem caused by dynamic convolution but also enables the Mamba module to effectively process non-causal data. Ultimately, the model can selectively filter relevant information based on the input features of each joint, thereby enhancing overall performance.

Algorithm 1 S-Mamba Block.

Input: $F_{in}:(C,T,N)$ Output: $F_{out}:(C,T,N)$   1: $\mathbf{X}:(C,T,N)\leftarrow {\mathbf{Conv}}_x\left(LayerNorm\left(X\right)\right)$   2: $\mathbf{Z}:(C,T,N)\leftarrow {\mathbf{Conv}}_z\left(LayerNorm\left(Z\right)\right)$   3: for $i=0$; $i<3$; $i++$do   4:    if $i<2$ then   5:     ${\mathbf{x}}_i:(CT,N)\leftarrow {Flatten}_i\left(X\right)$   6:     ${\mathbf{y}}_i:(CT,N)\leftarrow {SSM}_i\left({Conv}_i\left(X\right)\right)$   7:    else   8:     ${\mathbf{x}}_i:(CN,T)\leftarrow {Flatten}_i\left(X\right)$   9:     ${\mathbf{y}}_i:(CN,T)\leftarrow {SSM}_i\left({Conv}_i\left(X\right)\right)$   /* $\mathrm{SSM}$ represents Equation (4) implemented   by selective scan [21] */ 10:    end if 11: end for 12: for $i=0$; $i<3$; $i++$do 13:    ${\mathbf{Y}}_i:(C,T,N)\leftarrow {\mathbf{UnFlatten}}_i\left({\mathbf{x}}_i\right)$ 14: end for 15: ${\mathbf{F}}_{mid}:(C,T,N)\leftarrow {\sum }_i^2\left({\mathbf{Y}}_i·\mathrm{SiLU}\left(\mathbf{Z}\right)\right)$ 16: ${\mathbf{F}}_{out}:(C,T,N)\leftarrow {\mathbf{Conv}}_0\left({\mathbf{F}}_{mid}\right)$ $\hspace{1em}\hspace{0.17em}\mathbf{return}{F}_{out}$

Spatial–Temporal Mamba Block. Through the design of the SM-GCN block, the model is able to perform a global feature overview at an early stage and obtain the intrinsic hidden features between different joints. However, due to the inherent characteristics of the GCN method, the SM-GCN network is still insufficient in modeling the spatio-temporal global context, making it difficult to generate high-quality and reliable features. Therefore, we propose a Mamba-based spatio-temporal fusion module, called the ST-Mamba block, which is placed at the end of the APM structure. It fully leverages the excellent processing capability of the GCN in the early stage of the task and the high efficiency of Mamba in extracting distant features. This ensures that the network has strong input adaptability and global information modeling capability while maintaining lower complexity.

As illustrated in Figure 2, the core idea of the ST-Mamba block is to effectively integrate different levels of cross-scale features, thereby capturing various layers of detail and contextual information to enhance the overall feature representation. Consequently, the ST-Mamba branches the input $\mathbf{x}\in {\mathbb{R}}^{C\times T\times N}$ into two branches after layer normalization.

$$\begin{array}{c}\hfill {\mathbf{x}}_1=Norm\left(x\right)\end{array}$$

(13)

In the first branch, a multi-layer perceptron is utilized for projection and convolution, with SiLU activation applied to obtain an intermediate output $\mathbf{X}$. This intermediate output, $\mathbf{X}$, is then used as input to the state space model (Formula (2)) to generate the output $\mathbf{f}$. In the other branch, the input $\mathbf{x}$ is propagated through an MLP layer and SiLU activation to acquire a gating factor, ${\mathbf{f}}^{\prime }$, which controls the output $\mathbf{f}$. Finally, via an MLP layer and a residual connection, the final output $\mathbf{y}\in {\mathbb{R}}^{C\times T\times N}$ is obtained. The specific computation process of the first branch is depicted in Formulas (14)–(16):

$$\begin{array}{c}\hfill \mathbf{X}=SiLU(Conv\left({MLP}_1\left({\mathbf{x}}_1\right)\right))\end{array}$$

(14)

$$\begin{array}{c}\hfill \overline{\mathbf{A}},\overline{\mathbf{B}},\overline{\mathbf{C}},\overline{\mathbf{D}}=Disc\left(\mathbf{X}\right)\end{array}$$

(15)

$$\begin{array}{c}\hfill \mathbf{f}=SSM(\overline{\mathbf{A}},\overline{\mathbf{B}},\mathbf{C})\left(\mathbf{X}\right)\end{array}$$

(16)

where LayerNorm represents layer normalization, SiLU represents the activation function, ${MLP}_1$ stands for the multi-layer perceptron, and Conv denotes the convolution layer. Disc and SSM are as depicted in Section 3.1. The precise calculation process of the second branch is illustrated in Equation (17):

$$\begin{array}{c}\hfill {\mathbf{f}}^{\prime }=SiLU\left({MLP}_2\left({\mathbf{x}}_1\right)\right)\end{array}$$

(17)

where, SiLU denotes the activation function, ${MLP}_2$ represents the multi-layer perceptron, and ${\mathbf{f}}^{\prime }$ indicates the intermediate state. The final output of the ST-Mamba block is given in Equation (18):

$$\begin{array}{c}\hfill \mathbf{y}={MLP}_3\left({\mathbf{f}}^{\prime }\odot \mathbf{f}\right)+\mathbf{x}\end{array}$$

(18)

where ⊙ represents the element-wise multiplication mechanism, and ${MLP}_3$ denotes the multilayer operation. Ultimately, this approach enables the model to capture a broader range of contextual information and strengthens its ability to focus on salient features, thereby enhancing overall performance.

4. Experiments

In this section, we first introduce the datasets and experimental settings. Then, we compare our ActionMamba architecture with the current state-of-the-art skeleton-based human action recognition benchmarks, including some recent models based on Mamba, to demonstrate the superior performance of our model. Next, we conduct extensive ablation studies to verify the effectiveness and practicability of the proposed modules.

4.1. Dataset

NTU RGB+D. NTU RGB+D [41] is a prominent and extensively utilized benchmark for assessing skeleton-based action recognition systems. The dataset contains a total of 56,880 sequences of skeletal actions, which are grouped into 60 distinct categories and were performed by 40 different subjects. Each sample portrays a single action, involving a maximum of two individuals. All data was captured concurrently from three separate viewpoints using Microsoft Kinect v2 sensors. For standardized evaluation, the dataset’s creators have established two official protocols: (1) Cross-Subject (X-Sub), where the training set includes sequences from 20 subjects and the test set comprises data from the remaining 20 subjects; and (2) Cross-View (X-View), which uses data from camera perspectives 2 and 3 for training, while data from the first camera perspective is reserved for testing.

NTU-120 RGB+D. As the largest available dataset for human action recognition with three-dimensional joint annotations, NTU RGB+D 120 [42] encompasses 114,480 action samples distributed across 120 categories. Data for these samples were captured from 106 different subjects using three distinct camera viewpoints. The collection consists of 32 unique setups, each defined by a particular location and background environment. The creators have established two benchmark evaluation protocols: (1) Cross-Subject (X-Sub), for which the 106 participants are split equally into training and testing sets of 53 subjects each; and (2) Cross-Setup (X-Setup), where samples for training are selected from even-numbered setup identifiers, and samples for testing are sourced from odd-numbered ones.

UAV–Human. The UAV–Human [43] dataset is a comprehensive, large-scale collection created for analyzing human behavior from aerial viewpoints captured by drones. Its data was gathered over a three-month duration through drone flights in a variety of urban and rural settings, under both daytime and nighttime conditions. This collection features a wide spectrum of variability, including diverse objects, backgrounds, lighting scenarios, weather patterns, occlusions, camera movements, and drone flight orientations. UAV–Human is designed to support several research tasks, such as action recognition, pose estimation, re-identification, and attribute recognition. For our assessment, we adopt the evaluation methodology proposed in [44]. The dataset offers two distinct evaluation benchmarks, Cross-Subject v1 (CSv1) and Cross-Subject v2 (CSv2), both of which utilize a split of 89 subjects for training purposes and 30 subjects for testing.

4.2. Implementation

All experimental procedures were implemented within the PyTorch(Cuda 11.8 version) framework and executed on an NVIDIA GTX 3090 GPU (Nvidia Corporation, Santa Clara, CA, USA). For the model’s training, we utilized the Stochastic Gradient Descent (SGD) optimizer with a Nesterov momentum value of 0.9 and a weight decay of 0.0002. Our architecture was configured with an 8-layer SM-GCN module and a 2-layer ST-Mamba module. Each skeleton sequence was normalized to a consistent length of 300 frames, with sequences falling short of this length being appended with zero frames. During the training phase, a batch size of 16 was used, and the model was trained for a total of 140 epochs. The learning rate was initialized to 0.01 and underwent a linear warmup for the first 10 epochs. Subsequently, this rate was reduced by a factor of 10 at the 60th, 80th, and 100th epoch milestones.

4.3. Comparison with SOTA Methods

To validate the overall performance of our proposed ActionMamba method, we conducted experiments on three datasets: NTU RGB+D, NTU-120 RGB+D, and UAV–Human. Many advanced GCN methods have adopted a multi-stream fusion framework. We also use this framework for comparison.

NTU RGB+D. The experimental results on NTU RGB+D 60 are summarized in

Table 1. Comparison with the state-of-the-art methods on the NTU RGB+D 60 dataset. † represents the result we achieved using the officially released code, reproducing the environment as described in Section 4.2. The gray background indicates the method using Mamba structure.

Model	Year	Conf.	Param(M)	FLOPs(G)	X-Sub	X-View
ST-GCN [8]	2018	AAAI	3.10	16.3	81.5	88.3
AS-GCN [9]	2019	CVPR	6.99	35.5	86.8	94.2
2s-AGCN [28]	2019	CVPR	6.94	37.3	88.5 †	95.1 †
2s NAS-GCN [10]	2020	AAAI	13.1	−	89.4	95.7
CA-GCN [11]	2020	CVPR	−	−	83.5	91.4
Js Shift-GCN [12]	2020	CVPR	1.52	−	87.8	95.1
2s Shift-GCN [12]	2020	CVPR	3.04	−	89.7	96.0
Js MS-G3D [45]	2020	CVPR	3.20	48.8	89.4	95.0
2s MS-G3D [45]	2020	CVPR	6.44	98.0	91.5 †	96.2 †
SGN [46]	2020	CVPR	1.8	15.4	86.6	93.4
PA-ResGCN [47]	2020	ACMMM	3.6	−	90.9 †	96.0 †
ST-TR [48]	2021	ICPR	12.10	259.4	90.3	96.1
2s-PST-GCN [49]	2021	ICASSP	1.84	−	88.6	95.1
SparseShift-GCN [50]	2021	PRL	−	−	90.9	96.6
2s EDGN [8]	2022	CVIU	−	−	88.7	95.2
2s InfoGCN [8]	2022	CVPR	−	−	90.8	95.2
4s InfoGCN [8]	2022	CVPR	−	−	92.3	96.9
IST-GCN [51]	2023	AI	1.62	17.54	90.8	96.2
2s STF-Net [52]	2023	PR	3.4	−	90.8	96.2
RMMD [53]	2023	NC	−	−	83.0	90.5
SARGCN [54]	2023	AI	1.09	5.37	88.9 †	94.8 †
3Mformer [16]	2023	CVPR	4.37	58.45	94.8	98.7
RSA-Net [55]	2023	PR	3.5	−	90.9 †	95.3 †
$H^2$GCN [56]	2024	JKSU	−	−	88.9	94.8
Js Simba [38]	2024	−	−	−	91.3 †	96.1 †
Bs Simba [38]	2024	−	−	−	88.48	93.41
Js ActionMamba		Ours	1.9	5.94	89.16	94.92
Bs ActionMamba		Ours	1.9	5.94	88.79	94.18
2s ActionMamba		Ours	3.8	12.73	91.83	96.67

, where we compare ActionMamba with some of the most advanced models (including recent models based on Mamba). The results show that ActionMamba achieves superior performance in the two benchmark tests on the NTU-RGB+D dataset. It reached an accuracy of 91.83% on the X-Sub benchmark, achieving a 0.33% improvement compared to 2s MS-G3D. On the X-View benchmark, ActionMamba achieved an accuracy of 96.87%, a 0.47% improvement over the best-performing STF-Net. Notably, the 2s strategy adopted by ActionMamba garnered results comparable to those of the 4s strategy InfoGCN, fully demonstrating the effectiveness of our designed model. The accuracy results for each category in the NTU-RGB+D 60 X-Sub dataset are shown in Figure 6.

NTU RGB+D 120. ActionMamba’s accuracy on the NTU-RGB+D 120 dataset is equally impressive. As shown in

Table 2. Comparison with the state-of-the-art methods on the NTU RGB+D 120 dataset. † represents the result we achieved using the officially released code, reproducing the environment as described in Section 4.2. The gray background indicates the method using Mamba structure.

Model	X-Sub	X-Setup
ST-GCN [8]	70.7 †	73.2 †
Js Tripool [57]	80.1	82.8
AS-GCN [9]	77.9 †	78.5 †
2s AGCN [28]	82.5 †	84.2 †
4s CDGC [58]	86.3	87.8
Shift-GCN [12]	85.9	87.6
2s MS-G3D [45]	86.2	88.0
SGN [46]	79.2	81.5
2s UNIK [59]	80.8	86.5
2s ST-TR [48]	85.1	87.1
2s STF-Net [52]	84.9 †	87.7 †
JS Simba [38]	79.75	86.28
2s ActionMamba	86.48	88.15

, ActionMamba achieved accuracies of 86.48% and 88.1% on X-Sub and X-Setup, respectively. Compared to 2s MS-G3D, 2s ActionMamba improved by 0.28% and 0.15%, respectively. Additionally, even when compared to the 4-stream strategy CDGC algorithm, 2s ActionMamba maintained a competitive advantage, effectively validating our design concept.

UAV–Human. Test results on the UAV–Human dataset further substantiate the effectiveness of ActionMamba. As shown in

Table 3. Classification accuracy comparison against state-of-the-art methods on the UAV–Human dataset. The symbol † indicates that the results were obtained using the officially released codes for our implementation. The gray background indicates the method using Mamba structure.

Model	UAV–Human
	CSv1 (%)	CSv2 (%)
ST-GCN [8]	30.3 †	56.1 †
2s-AGCN [28]	34.8	66.7
HARD-Net [60]	37.0	−
Shift-GCN [12]	38.0 †	67.0 †
DGCN [61]	29.9	−
TFC-GCN [62]	39.6	64.7
IDGAN [44]	43.4 †	68.3 †
ActionMamba	44.3	69.5

, ActionMamba outperformed previous algorithms on the UAV–Human dataset. Compared with IDGAN, ActionMamba improves by 0.9% on the CSv1 benchmark and 1.2% on the CSv2 benchmark, effectively demonstrating the practicality and adaptability of our design.

4.4. Ablation Study

To evaluate the effectiveness of our proposed components, we conducted a thorough ablation study on the X-Sub subset of the NTU RGB+D 60 dataset. We used Shift-GCN as the baseline model and systematically modified the network modules to assess the contribution of each individual component.

4.4.1. Effectiveness of Different Components of ActionMamba

We utilized Shift-GCN as the baseline and sequentially integrated ACE, ST-Mamba, and SM-GCN to verify the effectiveness of the algorithm components. As shown in

Table 4. Ablation analysis of several ActionMamba components on the NTU RGB+D 60 dataset.The ↑ represents the numerical increase in the experimental results relative to the baseline results.

Model	X-Sub
Shift-GCN	89.70
Shift-GCN+ACE	89.88 ($\uparrow 0.18$)
Shift-GCN+ACE+ST-Mamba	90.39 ($\uparrow 0.69$)
Shift-GCN+ACE+SM-GCN	91.43 ($\uparrow 1.73$)
ActionMamba	91.83 ($\uparrow 2.13$)

, ACE enhanced Shift-GCN by 0.18%, while ST-Mamba further improved it by 0.69%. To further demonstrate the performance of our proposed model, we excluded the ST-Mamba module and exclusively incorporated the SM-GCN module on top of ACE. This adjustment led to a 1.73% increase in accuracy for Shift-GCN, effectively confirming the potential of our designed SM-GCN module.

4.4.2. Effectiveness of Action Characteristic Encoder

To evaluate the impact of the Action Characteristic Encoder on overall performance, a series of encoding experiments were designed. After adding ISTE and ESE to Shift-GCN, respectively, the model achieved slight improvements of 0.11% and 0.07% as shown in

Table 5. Effectiveness of the Action Characteristic Encoder. The symbol * signifies that we are placing the module to each layer of backbone.The ↑ represents the numerical increase in the experimental results relative to the baseline results.

Model	X-Sub
Shift-GCN	89.70
Shift-GCN+ISTE	89.81 $\uparrow 0.11$)
Shift-GCN+ESE	89.77 ($\uparrow 0.07$)
Shift-GCN+ACE	89.91 ($\uparrow 0.21$)
Shift-GCN+ACE *	90.21 ($\uparrow 0.51$)
ActionMamba w/o ACE	91.36 ($\uparrow 1.66$)
ActionMamba	91.83 ($\uparrow 2.23$)
ActionMamba+ACE *	91.84 ($\uparrow 2.24$)

. By combining the sub-modules and adding them to Shift-GCN, the model’s performance improved by 0.21%. In order to verify the effectiveness of external space encoding and provide comparison for subsequent experiments, ACE modules (i.e., ISTE and ESE were applied simultaneously) were added to each stage of Shift-GCN. Experimental results show that external space encoding can provide additional human skeleton information for model performance and bring a 0.51% performance improvement to the model. Under the same conditions, when ACE was added to our designed ActionMamba, the performance of this model has been improved by 0.57%. This effectively proved that the Mamba structure can capture encoded information through the state space and apply it to subsequent feature expansion, demonstrating the effectiveness and practicality of our designed module.

In addition, it is worth noting that ACE modules were integrated at each stage of ActionMamba, leading to a further performance improvement of 0.01%. This enhancement, although marginal, underscores the cumulative benefit of refined architectural components. Compared to ST-GCN, this observation supports the efficacy of the proposed SM-GCN module in capturing and preserving the relative positional relationships of human skeletal joints across different stages of the feature extraction process. These results validate the effectiveness of our architectural design and highlight the advantages of incorporating Mamba mechanisms within the network.

4.4.3. Complexity Comparison

For a comprehensive evaluation of our method’s inference efficiency relative to established techniques, we undertook a meticulous comparative analysis of model complexity. The results of this analysis are consolidated in

Table 6. Complexity comparison of the Model.

Model	FLOPs (G)	X-Sub	Inference (ms)
ST-TR	259.4	90.3	29.446
2s InfoGCN	-	90.8	33.736
4s InfoGCN	-	92.3	49.881
IST-GCN	17.54	90.8	23.825
3MFormer	58.45	94.8	76.482
JS Simba	-	91.3	22.846
JS ActionMamba	5.94	89.70	16.224
2s ActionMamba	12.73	91.83	18.753

. The performance of all models was benchmarked on an NVIDIA GeForce RTX 3090 GPU using the ONNX temporal inference framework. To ensure an equitable comparison, specific parameters were employed for the evaluation: the inference batch size was set to one, and each model was subjected to 50 preheating iterations before being tested over 200 inference runs on the same hardware. Our proposed approach exhibits a distinct advantage in both inference time and parameter efficiency, leading to a significant reduction in the demand for computational resources while preserving a competitive level of accuracy.

Notably, when contrasted with the leading baseline model, 3MFormer, our approach demonstrates a 75.48% enhancement in inference velocity while incurring only a minimal reduction in accuracy. Moreover, in a direct comparison with the Mamba-based Simba algorithm, our technique is not only faster (18.753 versus 22.846) but also achieves superior accuracy (91.83 compared to 91.3). Collectively, these outcomes indicate that the proposed method attains an effective equilibrium between model performance and computational demands, which can be attributed to its meticulously crafted, fine-grained architecture.

4.4.4. The Impact of Different Levels of Numbers

To determine how network depth impacts performance, we systematically adjusted the number of SM-GCN and ST-Mamba layers, and evaluated our proposed model on the NTU RGB+D 60 dataset. As shown in

Table 7. Impact of the number of SM-GCN/ST-Mamba layers on model performance.

Method	SM-GCN	ST-Mamba	FLOPs (G)	X-Sub
ActionMamba	5	1	8.92	87.41
ActionMamba	6	1	10.18	89.93
ActionMamba	7	1	11.34	90.87
ActionMamba	8	1	12.41	91.61
ActionMamba	9	1	13.28	91.38
ActionMamba	8	2	12.73	91.83
ActionMamba	8	3	13.17	91.74
ActionMamba	8	4	13.54	90.91

, a larger number of SM-GCN layers typically corresponds to improved performance. This observation implies that deeper network architectures are more capable of capturing complex spatio-temporal relationships. Nevertheless, the gains in performance begin to weaken once the architecture exceeds a certain depth. Specifically, when the number of SM-GCN layers reaches nine, the model exhibits signs of overfitting or gradient vanishing—and this phenomenon results in a slight drop in accuracy. In a concurrent analysis, we carried out a systematic investigation of the ST-Mamba layers, and found that a two-layer setup yielded the best performance. Drawing on these results, we reached the conclusion that an architecture consisting of eight SM-GCN layers and two ST-Mamba layers achieves the optimal balance between the model’s expressive capacity and computational cost. For this reason, this configuration was set as the final architecture for all subsequent experiments.

4.4.5. Confusion Matrix

To conduct a more detailed evaluation of the proposed method’s recognition performance, we performed an in-depth study on 15 action categories known for being highly confusable with similar activities. The confusion matrices presented in Figure 7 illustrate the classification outcomes on the NTU-60 dataset for both Shift-GCN and our methodology. An examination of these matrices reveals that our approach obtains superior recognition accuracy across almost all categories. This enhancement is especially notable in several challenging action classes, highlighted in red, such as the differentiation between “put on shoes” and “take off shoes,” as well as “reading” and “writing”. Figure 8 provides illustrations of these difficult instances, where precise classification is challenging without leveraging temporal data. The critical importance of modeling spatio-temporal information for these particular actions is evident. By aggregating spatio-temporal information, our proposed technique boosts the model’s multimodal learning capacity. This showcases the Mamba algorithm’s advantages in extracting spatio-temporal representation features from video actions and proves the efficacy of the algorithm designed in this paper.

5. Conclusions and Future Work

We present an action spatio-temporal aggregation network termed ActionMamba, which combines the strengths of Graph Convolutional Networks (GCNs) with the Mamba architecture. Through the integration of Mamba into the Shift-GCN framework using shift convolution and a spatio-temporal scanning mechanism, we have developed an enhanced SM-GCN structure. This design enables the model to efficiently utilize spatio-temporal information while maintaining a global receptive field and reducing computational complexity. To further strengthen Mamba’s ability to model long-distance dependencies, we propose the Action Characteristic Encoder, which merges external spatial encoding with intrinsic spatio-temporal encoding. This component enhances the model’s understanding of the human body’s spatial arrangement and helps capture the underlying relationships between non-adjacent joints. Experimental results from assessments on the NTU-RGB+D 60, NTU-RGB+D 120, and UAV–Human benchmark datasets verify that ActionMamba achieves an outstanding balance between performance accuracy and inference speed, even outperforming current Mamba-based methods.

In our future research, we aim to explore the broader applicability of Mamba and other state space models in the field of action recognition. Specifically, we plan to tackle multi-person interaction recognition by employing dynamic feature encoders to model the spatio-temporal dynamics of such interactions, with the goal of achieving effective motion disentanglement in group scenarios. Regarding real-time implementation on low-power hardware, we will focus on model optimization by developing lightweight Mamba variants and adopting dynamic pruning strategies. These improvements are expected to facilitate deployment across a wider range of wearable sensing devices, thereby contributing to advancements in the field of human activity recognition.