Abstract
Graph convolutional networks (GCNs) have received widespread attention in skeleton-based human action recognition because of their remarkable advantages in both recognition accuracy and computational efficiency. However, most existing GCN-based methods rely on a fixed, physically defined graph structure, which fails to capture the rich and adaptive correlations among joints that are critical for understanding complex actions, particularly those involving subtle movements or long-range dependencies. In this paper, we propose a Cascade Graph Convolutional Network (CGCN) based on a Motion-Aware Hierarchical Adjacency Matrix (MAHAM). The MAHAM integrates both physical and non-physical joint relationships by learning a dynamic correlation matrix from skeletal data, which enhances the model’s representational capacity and interpretability. Furthermore, a cascade network is proposed to refine action recognition in a stepwise manner, progressively employing skeleton graphs of increasing joint numbers. This cascaded design allows most simple actions to be classified at early stages with fewer joints, thereby substantially reducing the computational overhead. Moreover, a feature matrix constructed from a priori geometric and kinematic features, such as joint displacements, velocities, and accelerations, is used to form a more comprehensive representation of human actions, further improving the discrimination of fine-grained activities. We perform extensive experiments on three public datasets: Florence3D-Actions, UTKinect-Action3D, and a newly collected HanYue-Action3D. Across the three datasets, our method achieves Accuracy–Latency Balance Index (ALBI) values of 0.990, 0.953, and 0.985, respectively, which demonstrates that our method achieves good performance in the joint evaluation of accuracy and inference efficiency.