A Hybrid Human Activity Recognition Method Using an MLP Neural Network and Euler Angle Extraction Based on IMU Sensors

Abstract

Inertial measurement unit (IMU) technology has gained popularity in human activity recognition (HAR) due to its ability to identify human activity by measuring acceleration, angular velocity, and magnetic flux in key body areas like the wrist and knee. It has propelled the extensive application of HAR across various domains. In the healthcare sector, HAR finds utility in monitoring and assessing movements during rehabilitation processes, while in the sports science field, it contributes to enhancing training outcomes and preventing exercise-related injuries. However, traditional sensor fusion algorithms often require intricate mathematical and statistical processing, resulting in higher algorithmic complexity. Additionally, in dynamic environments, sensor states may undergo changes, posing challenges for real-time adjustments within conventional fusion algorithms to cater to the requirements of prolonged observations. To address these limitations, we propose a novel hybrid human pose recognition method based on IMU sensors. The proposed method initially calculates Euler angles and subsequently refines them using magnetometer and gyroscope data to obtain the accurate attitude angle. Furthermore, the application of FFT (Fast Fourier Transform) feature extraction facilitates the transition of the signal from its time-based representation to its frequency-based representation, enhancing the practical significance of the data. To optimize feature fusion and information exchange, a group attention module is introduced, leveraging the capabilities of a Multi-Layer Perceptron which is called the Feature Fusion Enrichment Multi-Layer Perceptron (GAM-MLP) to effectively combine features and generate precise classification results. Experimental results demonstrated the superior performance of the proposed method, achieving an impressive accuracy rate of 96.13% across 19 different human pose recognition tasks. The proposed hybrid human pose recognition method is capable of meeting the demands of real-world motion monitoring and health assessment.

1. Introduction

With the continuous advancement of technology and the increasing focus on physical well-being, sensor-based human activity recognition (HAR) has gained significant attention and has been widely applied. Human activity refers to the motion status of the human body in space, encompassing its position, angles, and trajectories. The recognition of human activity holds great potential across diverse domains, including medical rehabilitation [1], sports science, human–computer interactions, and gaming. In these fields, the attainment of precise and real-time human activity recognition plays a pivotal role in attaining predefined objectives.

The practical uses of traditional camera-based activity recognition systems are severely constrained by their low recognition accuracy, susceptibility to ambient interference, and high processing complexity. Contrarily, sensor-based human position identification techniques, such as those using depth cameras, audio signals, and inertial measurement units (IMUs), offer important benefits like high accuracy, low latency, and robustness to environmental factors [2]. These benefits support the quick advancement and extensive application of sensor-based activity recognition techniques. However, there also exist certain drawbacks to the sensor-centric approach. Firstly, sensor acquisition and integration amplify hardware costs and system intricacy, rendering it unsuitable for scenarios with constrained budgets or space requirements. Secondly, specific motions or actions might necessitate intricate algorithmic parsing, potentially leading to performance degradation. The processing and fusion of sensor data necessitate meticulous calibration and adjustments for varying sensor types, involving intricate technical challenges.

Human activity recognition entails monitoring and analyzing the body’s movement status through the utilization of sensors, and automatically recognizing activity based on spatial or trajectory characteristics. The versatility of this approach allows for its application in various scenarios, bringing with it distinct advantages and specific conditions for implementation. Furthermore, the fusion of multiple sensor technologies enables a more thorough and comprehensive recognition and analysis of human activity, providing valuable insights into activity-related information. For example, placing sensors on different parts of the body (such as the wrist, waist, and ankle) can obtain more comprehensive information about human activity, helping to identify more complex behaviors such as bending, turning, etc. Human activity recognition can be achieved through different sensor-based methods. Inertial measurement units (IMUs) calculate body movements based on acceleration, angular velocity, and magnetic flux data from key body areas like wrists and knees [3]. Depth cameras, using infrared encoding, accurately recognize whole-body activities and joints [4]. Video sequence technology analyzes high-speed camera footage for activity recognition in sports and medical rehabilitation. Acoustic signal analysis estimates body activity by analyzing sound signals in the context of human–machine interactions and virtual reality scenarios. These methods enhance human–computer interactions, optimize sports training, aid medical rehabilitation, and advance virtual reality applications.

Although the IMU (inertial measurement unit) has been widely used in HAR, it still faces limitations in practical applications. The raw data from a single accelerometer is too simplistic to independently determine activity. Meanwhile, gyroscopes accumulate small errors over time with prolonged usage, leading the integrated values from the sensor to progressively deviate from true values. This phenomenon can result in the drift of integrated values over time, ultimately impacting the accuracy of applications [5]. As a result, many researchers have proposed multi-sensor fusion algorithms based on Kalman filters, combining attitude data from accelerometers, gyroscopes, and magnetometers. However, establishing stable observation equations for long-term monitoring is challenging due to the algorithm’s inclusion of intricate mathematical principles, multi-dimensional state observation handling, and iterative updates. This complexity becomes particularly pronounced when dealing with high-dimensional state spaces or frequent data updates, resulting in escalated computational demands, and calculating measurement noise covariance and state noise covariance adds to the algorithm’s complexity, leading to excessive overhead.

Therefore, this paper proposes a method of human pose angle extraction using an accelerometer and Euler angle calculation to recognize 19 kinds of human physical activities. Utilizing a sliding window technique, the method removes high-frequency noise to mitigate the impact of integration drift. Simultaneously, it introduces a module known as GAM to emphasize the temporal characteristics of sensor data, further counteracting integration drift. By adaptively adjusting model weights, it learns feature representations of the sensor data, reducing complex mathematical computations and enhancing recognition accuracy [6].

The contributions of the paper:

• A human feature extraction model is introduced to convert multi-dimensional sensor data into one-dimensional features;
• An approach for feature group division and classifier network construction is proposed to improve group correlation analysis and human action recognition accuracy;
• The impact of the K-nearest neighbors algorithm (KNN) and sliding window size on evaluation results is examined;
• The study investigates the influence of GAM, transformer block, and classifier block on the experimental accuracy.

The subsequent sections of this paper are organized as follows: The second section delineates recent advancements in the realm of human behavior recognition. Moving forward, the third section elucidates the transformation of Euler angles within human posture, alongside the methodology of integrating GAM-MLP information. Progressing further, the fourth section details the groundwork undertaken for the forthcoming experiments. In the subsequent fifth section, we present a comprehensive compilation of experimental findings, meticulously analyzed. Lastly, the sixth section encapsulates the entirety of the article through a concise summary, while also delving into an insightful analysis of potential avenues for future research opportunities.

2. Related Work

Recently, computer vision applied to image-based HAR has made significant progress. Bayraktar et al. [7]. have provided valuable insights into visual semantic analysis using deep neural networks and load analysis through sensors. This is noteworthy for harnessing computer vision and sensor load for human behavior recognition. However, visual-based HAR is susceptible to environmental factors such as lighting and angle variations. Therefore, the application of low-cost sensor-based HAR methods has become more prominent in this field. Yigit et al. [8]. proposed three comprehensive algorithms for low-cost variable stiffness robotics mechanisms. These algorithms enhance recognition accuracy and reduce costs in the adjustment of robot joint stiffness. In [9], an innovative approach was introduced to address the estimation of external forces and torques on elastic joints. This method leverages the inherent elastic characteristics of joints, eliminating the need for expensive sensors. These approaches provide some cost-effective solutions for joint sensor design in HAR.

When utilizing sensors for HAR, achieving a sufficient level of accuracy is crucial [10]. In scenarios where only a single sensor is employed for recognition, Anazco et al. developed a HAR system [11], in which individual IMUs are attached to the dominant wrist. They employed a variational autoencoder to automatically denoise the IMU signals, enhancing recognition accuracy. This approach achieved a recognition accuracy of 95.09% in everyday smartphone-based motion recognition. However, relying solely on sensors worn on the wrist has limitations when it comes to recognizing complex human gait behaviors. The choice of sensor placement significantly impacts the accuracy of human gait recognition. Abdelhafiz et al. devised a sensor selection approach using the feature selection criteria of maximum relevance and minimum redundancy. This method identifies optimal sensor placement for gait recognition [12]. Additionally, they implemented a two-layer classifier to differentiate interfered activities and incorporated physical features into the feature dictionary, thereby enhancing recognition accuracy.

Through collecting data using sensors placed at appropriate positions, neural network methods have demonstrated strong recognition capabilities when processing such data, Rivera et al. presented an approach for recognizing hand activities in daily life [13]. They proposed a deep autoencoder based on ARMA (Auto-Regressive Moving Average) and a deep recursive network using GRU (Gated Recurrent Unit). The ARMA-based deep autoencoder effectively denoises the unprocessed time-series signals, while the deep RNN-GRU utilizes the output from the encoder to identify seven hand gestures. This approach yielded a 12.8% increase in accuracy compared to traditional classifiers for gesture recognition. Nevertheless, effective preprocessing techniques for sensor data remain essential to further enhance recognition accuracy.

Applying data preprocessing and model pretraining techniques often leads to remarkable results in dealing with complex human activity recognition tasks. Hashim et al. proposed a method to transform raw accelerometer and gyroscope sensor data into the visual domain [14]. To address the issue of high computational complexity during this process, they employed fine-tuning through pretraining a CNN and transfer learning. On several online human activity recognition datasets, they achieved a classification accuracy of 98.36%. Tahir [15] combined data preprocessing techniques with the primary domain features of human activities, such as time, frequency, wavelet, and time-frequency features. They employed a random forest classifier to monitor human body activities. Results on five commonly used HAR datasets demonstrated that their method exhibited a certain superiority compared to cutting-edge approaches. However, the challenge lies in how to process and fuse data from multiple sensors to achieve accurate human behavior recognition. This has emerged as a new challenge in the HAR field.

Fusing data from different sensors can enhance recognition accuracy. Chakraborty addressed the issue of recognition errors due to sensor placement and developed a heterogeneous sensor system [16]. This system utilized low-cost leg sensors and fingertip-based pulse sensors to acquire multimodal data. They employed a one-dimensional deep convolutional neural network for system performance evaluation. Through feature fusion, this approach achieved a 97% accuracy in recognizing the walking movements of the human body.

By monitoring and analyzing body movements through sensors and recognizing activities based on spatial and trajectory characteristics, the methods mentioned above open up versatile applications and offer valuable insights into activity-related information. For instance, the positioning of IMU sensors, data preprocessing techniques, and adjustments to network modules have all served as references in our approach presented in this paper. However, multi-sensor data fusion continues to face challenges associated with high complexity [17]. These challenges include overcoming the problem of the unified processing of multi-sensor data, concise attitude angle transformation, and behavioral feature extraction to avoid high complexity in the recognition process.

In order to address these issues, this paper introduces a human feature extraction model based on sensor data, aiming to collect and extract human posture angles to the maximum extent. The model uses dimensionality reduction technology to fuse the Euler angles of multiple parts of the human body into the overall pose angle, and utilizes the FFT algorithm for transforming time-domain characteristics into frequency-domain features, enhancing the accuracy of human activities recognition. Additionally, a novel technique called the Group Attention Module (GAM) is introduced, utilizing a multi-layer perceptron to share and fuse information among different features within the same group, effectively extracting the behavioral features of activities. This approach effectively improves the accuracy and robustness of HAR.

3. Methods

3.1. Enhancing the Human Pose Recognition Model Structure

The HAR system architecture involves data preprocessing to remove sensor noise and calculate Euler angles using roll, pitch, and yaw angles. Magnetometer and gyroscope data correct the initial Euler angles and reduce dimensionality, forming the attitude angle model. Feature extraction methods, including FFT, involve shifting to the frequency domain, reflecting human activity periodicity and frequency for practical significance [18]. Information fusion through a multi-layer perceptron (MLP) and a group attention module enhances feature fusion. Manual classification combined with conventional neural network techniques achieve accurate HAR. The fully connected and SoftMax layers produce the final classification results, enabling precise human activity recognition. Figure 1 illustrates the identification process of the hybrid human pose recognition method using an MLP neural network and Euler angle extraction based on IMU sensors.

3.2. Extraction of Human Pose Euler Angles

3.2.1. Initial Attitude Angle Calculation

We use Euler angles to describe the orientation of several body parts [19,20,21], including the torso, left hand, right hand, left leg, and right leg, in order to calculate the initial torso rolling angle, pitch angle, and yaw angle. The acceleration vectors of various body parts must be converted into a single coordinate system in order to calculate the Euler angles with accuracy, assuring consistency in angle calculation and dimensionality reduction.

To get the torso’s Euler angles, the following procedures are conducted, assuming the common coordinate system is represented by $xyz$:

First of all, the roll angle of the torso, indicated as ${\varnothing }_t^{torso}$, is calculated as follows:

$${\mathrm{\varnothing }}_t^{torso}={tan}^{-1}\frac{a_{y,t}^{torso}}{a_{z,t}^{torso}}$$

(1)

where, in the common coordinate system, $a_{y,t}^{torso}$ denotes the acceleration along the Y-axis of the acceleration vector and $a_{z,t}^{torso}$ signifies the acceleration along the Z-axis. Furthermore, the pitch angle of the torso ${\theta }_t^{torso}$ is computed using the formula:

$${\theta }_t^{torso}={tan}^{-1}\frac{-a_{x,t}^{torso}}{\sqrt{{\left(a_{y,t}^{torso}\right)}^2+{\left(a_{z,t}^{torso}\right)}^2}}$$

(2)

In this equation, the term, $a_{x,t}^{torso}$ denotes the acceleration of the torso’s acceleration vector in the common coordinate system along the X-axis. The yaw angle of the torso, shown by the symbol ${\phi }_t^{torso}$, is then calculated. The initial yaw angle is set to zero during the computation since accelerometers can only detect an object’s acceleration in relation to a fixed coordinate system and not its absolute direction:

$${\phi }_t^{torso}=0$$

(3)

Once the Euler angles of the torso in the common coordinate system have been computed, the Euler angles for the other body parts can be calculated using specific methods tailored to each part. For instance, the acceleration vector of the right hand must be rotated so that it is perpendicular to the xz plane of the torso in order to calculate the Euler angles of the right hand. In order to accurately calculate the Euler angles in the torso coordinate system, this rotation seeks to remove the right hand’s projection in the xz plane of the torso. Figure 2 is right hand coordinate transformation, gyroscope, and accelerometer schematic for the human body model, in which we describe the 3D human model.

The process involves first calculating the rotation angle of the right hand vector, denoted as ${\alpha }_t^{rightarm}$, relative to the coordinate system of the torso:

$${\alpha }_t^{rightarm}={tan}^{-1}\frac{{-a}_{z,t}^{torso}}{a_{y,t}^{torso}}$$

(4)

where, $a_{z,t}^{torso}$ represents the acceleration along the Z-axis of the acceleration vector of the right hand position in the common coordinate system, and $a_{y,t}^{torso}$ denotes the acceleration along the Y-axis.

Next, the acceleration vector of the right hand after rotation, represented as ${\alpha }_t^{rightarm}$, is obtained by rotating the original acceleration vector ${\overrightarrow{\alpha }}_t^{rightarm}$ about the x-axis of the torso coordinate system, using the rotation matrix $R_x({\alpha }_t^{rightarm})$ [22,23]:

$${\overrightarrow{\alpha }}_t^{rightarm}=R_x({\alpha }_t^{rightarm}){\overrightarrow{\alpha }}_t^{rightarm}$$

(5)

$$R_x({\alpha }_t^{rightarm})=\left[\begin{array}{ccc}1& 0& 0\\ 0& cos({\alpha }_t^{rightarm})& -sin({\alpha }_t^{rightarm})\\ 0& sin({\alpha }_t^{rightarm})& cos({\alpha }_t^{rightarm})\end{array}\right]$$

(6)

To begin with, the magnetometer data is utilized to compensate for the yaw angle in the original attitude angle by determining the direction of the horizontal plane of Earth’s magnetic field. Since the magnetic field component perpendicular to the horizontal plane does not impact the calculation of the yaw angle, the orientation of the magnetic field within the horizontal plane is employed for compensation. The formula for compensating the yaw angle is given as follows:

$${\phi }_t=0+atan2(m_y,m_x)$$

(7)

Additionally, the original Euler angles are compensated using gyroscope data from various body parts, following a similar calculation approach. The compensation method for the torso’s original Euler angle using gyroscope data is described below.

Gyroscope data can be integrated to compensate for the torso’s Euler angles. Specifically, the gyroscope data compensation formula is as follows:

$$\left\{\begin{array}{c}{roll}_{comp}={roll}_{prev}+\frac{{gyro}_x · dt}{cos\left({pitch}_{prev}\right) · cos\left({yaw}_{prev}\right)}\\ {pitch}_{comp}={pitch}_{prew}+\frac{{gyro}_y · dt}{cos\left({roll}_{prev}\right) · cos\left({yaw}_{prev}\right)}\\ {yaw}_{comp}={yaw}_{prew}+\frac{{gyro}_z · dt}{cos\left({roll}_{prev}\right) · cos\left({pitch}_{prev}\right)}\end{array}\right.$$

(8)

where, $dt$ represents the sampling time interval, ${roll}_{prev}$, ${pitch}_{prew}$, and ${yaw}_{prew}$ are the Euler angles of the torso at the previous sampling moment. These formulas employ the integration of gyroscope data to update the current Euler angles of the torso, thereby compensating for the gyroscope data.

Generally, the Euler angle principle is an early method for attitude determination due to its straightforward physical interpretation. However, it involves complex operations with many trigonometric calculations. To simplify the practical application process, we introduce the quaternion method as an optimization of the Euler angle approach.

3.2.2. Euler Angle Correction for Quaternion and Rodriguez Parameters

Given the roll $\mathrm{\varnothing }$, pitch $\theta$, and yaw $\phi$ Euler angles for each body part, these can be converted into rotation quaternions. The quaternion representing the orientation of a body part in the common coordinate system would be in the form:

$$q=cos\left(\frac{\mathrm{\varnothing }}{2}\right)cos\left(\frac{\theta }{2}\right)cos\left(\frac{\phi }{2}\right)+sin\left(\frac{\mathrm{\varnothing }}{2}\right)sin\left(\frac{\theta }{2}\right)sin\left(\frac{\phi }{2}\right)i+sin\left(\frac{\mathrm{\varnothing }}{2}\right)cos\left(\frac{\theta }{2}\right)cos\left(\frac{\phi }{2}\right)j+cos\left(\frac{\mathrm{\varnothing }}{2}\right)sin\left(\frac{\theta }{2}\right)cos\left(\frac{\phi }{2}\right)k$$

(9)

$$\left\{\begin{array}{c}\varnothing =atan2\left(2\right(q_0{q}_3+q_1{q}_2),1-({q_2}^2+{q_3}^2\left)\right)\\ \theta =asin\left(2\right(q_0{q}_2+q_1{q}_3\left)\right)\\ \phi =atan2\left(2\right(q_0{q}_1+q_2{q}_3),1-({q_1}^2+{q_2}^2\left)\right)\end{array}\right.$$

(10)

where, $q_0{, q}_1,q_2, and q_3$ are the components of the quaternion.

Compared to the Euler angle method, the quaternion description method further simplifies the calculation process and effectively avoids the singularity problem of the Euler angle due to a trigonometric function operation, but quaternions possess redundant parameters and quaternion normalization constraints. Quaternion tracing was compared to the Euler angle method singularity problem of the angle function operation. Lastly, to facilitate the extraction of human activity posture features, a dimensionality reduction technique is applied to the optimized Euler angles obtained through the above steps. The resulting reduced Euler angles are defined as the attitude angle, denoted as attitude. The formula for computing the attitude angle is as follows:

$$attitude=arccos(cos\mathrm{\varnothing }cos\theta cos\phi +sin\mathrm{\varnothing }sin\theta sin\phi )$$

(11)

where, $\mathrm{\varnothing }$, $\theta ,$ and $\phi$ are the roll, pitch, and yaw angles, respectively. This formulation combines the three-dimensional Euler angles into a single one-dimensional attitude angle, providing a concise representation of the overall posture.

3.2.3. Attitude Angle Calculation Algorithm Design

A new method of human attitude angle calculation based on multiple attitude parameters compensating each other is proposed, which makes up for the redundant parameters and the singular values of trigonometric functions. The attitude angle calculation procedure begins with the initialization process and unit conversion of acceleration data (lines 1–2). Subsequently, Euler angles are computed based on the sampled frequency-processing sequence data, with compensation provided through gyroscope and accelerometer data (lines 3–12). Following this, the Euler angle vector is transformed into a rotation matrix, and from there into a quaternion and Rodrigues parameters (lines 13–15). Finally, the ultimate pose angles are calculated and displayed (lines 16–19).

Algorithm 1: The human attitude angle calculation method based on multiple attitude parameters
Input: Sample time series T, Acceleration data Adata (ax, ay, az), Gyroscope data Gdata,
Magnetometer data Mdata;
Output: Attitude Angle attitude;
(1)	Initialization Adata, Mdata, Quaternion Q and Rodrigues Parameters r;
(2)	Conversion unit of accelerometer data to the acceleration of gravity
(3)	for i = 0,0.04,0.08…T do:
(4)	Calculate roll angle $\varnothing$t and pitch angle $\theta$t
(5)	Compensate for Mdata to correct Attitude Angle with magnetometer
(6)	Calculate yaw angle $\phi$t
(7)	Convert angles to radians
(8)	Update Euler Angle vector Eangle
(9)	Calculate the Gyroscope Euler Angle variation:
(10)	Gchange = Gdata * dt
(11)	Compensate the Euler Angle Ccomp:
(12)	Ccomp = Eangle + Gdata;
(13)	Transform Euler Angle vector to rotation matrix
(14)	Convert Euler Angle vector to quaternion
(15)	Convert quaternion to Rodrigues Parameters
(16)	Calculate Attitude Angle attitude
(17)	Convert radians to angles
(18)	end for
(19)	Output the final Attitude Angle attitude

3.3. Human Pose Feature Extraction

FFT is applied to the data [24], yielding the frequency-domain conversion results depicted in Figure 3.

$$X\left[k\right]={\sum }_{n=0}^{N-1}x\left[n\right]\times e^{\frac{-j2\pi kn}{N}}$$

(12)

where, $X\left[k\right]$ represents the frequency-domain signal, $x\left[n\right]$ denotes the denoised time-domain signal of length $N$, $N$ represents the number of sampled data points, and $j$ represents the imaginary unit $\left(\sqrt{-1}\right)$.

The first 45 FFT coefficients are then selectively extracted as the 45 eigenvalues, which represent the human activity.

$$X_{eigenvalues}=\left[X\left[0\right],X\left[1\right],\cdots ,X\left[44\right]\right]$$

(13)

These eigenvalues are subsequently harnessed for deep learning-based human body activity recognition.

3.4. GAM-MLP Information Fusion

3.4.1. Human Pose Feature Information Fusion

In order to better process the spatiotemporal features of IMU sensor data, the neural network model that is suggested in this paper incorporates information fusion [25], attention evaluation [26], and classification recognition [27]. Figure 4 depicts the model architecture of GAM-MLP and the process of action classification. The data of different sensors in each part of the human body are put into the GAM-MLP model as one-dimensional features to recognize human activities, and finally the recognition results are obtained. The data format represented by (, 64) in the figure represents (batch_size, data dimension), due to batch_ size is not fixed and is therefore represented as blank.

GAM-MLP’s primary objective is to extract posture features of the utmost significance for action recognition from a 20-dimensional dataset. This extraction process is intended to facilitate effective information fusion and utilize these features as the primary foundation for classification. To accomplish this objective, the paper introduces a component called the GAM.

In conventional neural network classification methodologies, each pose feature is uniformly trained, with their significance determined solely by the weights assigned to individual neurons. However, this method suffers from inadequate communication among distinct pose features, ultimately yielding less impactful results. This paper introduces a novel approach that combines manual partitioning and automatic learning. This hybrid approach empowers the neural network model to allocate varying degrees of importance to different categories of pose features, ultimately resulting in an enhanced classification accuracy.

In the GAM, the pose features are manually categorized into distinct groups. During the manual grouping process, following feature extraction via FFT, it becomes evident that there exist resemblances in the characteristics among features associated with various activities. For instance, the frequency features of “Sitting” and “Standing” are characterized by frequencies of less than 4 Hz and exhibit a unimodal pattern. On the other hand, activities such as “Exercising on a stepper” and “Playing Basketball” demonstrate characteristic frequencies reaching approximately 10 Hz, marked by multiple peaks and substantial fluctuations. Consequently, human motion activities can be effectively classified into dynamic and static groups. The static group conforms to low-frequency attributes with a single wave peak, while the dynamic group encompasses high-frequency elements, multiple peaks, and significant fluctuations. As depicted in the accompanying Figure 5, we provide a visual comparison of feature images extracted from several activity features.

Furthermore, adhering to the principles of the periodicity inherent in dynamic motion, the dynamic group can be further categorized into two subgroups: periodic dynamic activity and random dynamic activity. Within the periodic dynamic activity category, distinctions can be made based on the unique patterns and trends of peak characteristics. To summarize, the criteria and guidelines for manual partitioning can be systematically delineated, progressing from higher-order groupings to lower-order ones, in accordance with the specific characteristics of the dataset. The schematic representation of the group attention module is illustrated in Figure 6. The blocks with different colors in the figure represent the weights calculated by different groups.

To address the inherent nonlinear relationships and patterns within both the dynamic and static groups of features, this paper introduces the “Transform Block”, which combines elements from MLP and the Transformer model. Both MLP and the Transform Block are integral components of deep learning models, commonly employed in tandem to handle intricate datasets and tasks.

MLP serves the purpose of introducing nonlinearity into the model, while the Transform Block, a crucial part of the Transformer architecture, is utilized for effectively modeling sequential data. This integration is geared towards enhancing the model’s capacity for representation learning and nonlinear modeling, thereby enabling it to adaptively generate distinct weights between modules and effectively capture the intricate relationships and patterns present in the data.

The Multi-Layer Perceptron (MLP) serves as a conduit for interchanging and amalgamating distinct features within the same group. It encompasses a series of layers including the fully connected, activation, and standardization layers [28,29,30]. The training process employs backpropagation, iteratively adjusting weights and biases. This fosters incremental learning and the integration of pertinent information across features in the same group.

The fully connected layer embodies input and output neurons. Given the ith input, $x_i,$ and the jth output, $y_j$, their relationship is governed by the weight, $w_{ij}$, and bias, $b_j$, represented as

$$y_j={\sum }_{i=1}^{10}{w}_{ij}{x}_i+b_j$$

(14)

Two fully connected layers are followed by the activation layer and the standardization layer, respectively, to achieve the nonlinearity of network representation and improve training speed, generalization ability, and robustness, while reducing overfitting, gradient disappearance or gradient explosion problems. For the activation layer, the Rectified Linear Unit (ReLU) is selected, which is more efficient and safer than other activation functions while making the input of neurons a nonlinear transformation. Its function expression is shown as follows:

$$\mathrm{ReLU}\left(x\right)=\max \left(0,x\right)$$

(15)

where max represents taking the larger value of the two numbers, and 0 is the cutoff point of ReLU, that is, when x < 0, the value of ReLU is 0.

For the normalization layer, we utilize batch normalization. The characteristics of each sample, $x^{\left(i\right)}=\left(x_1^{\left(i\right)},x_2^{\left(i\right)},\cdots ,x_m^{\left(i\right)}\right)$, where $m$ indicates feature count, are transformed according to

$$\mathrm{Mean}:{\mathsf{\mu }}_B=\frac{1}{m}{\sum }_{i=1}^m{x}_i$$

(16)

$$\mathrm{Variance}:{\mathsf{\sigma }}_B^2=\frac{1}{m}{\sum }_{i=1}^m{\left(x_i-{\mathsf{\mu }}_B\right)}^2$$

(17)

$$\mathrm{Normalization}:\widehat{x_i}=\frac{x_i-{\mathsf{\mu }}_B}{\sqrt{{\mathsf{\sigma }}_B^2+\mathsf{ϵ}}}$$

(18)

$$\mathrm{Output}:y_i=\mathsf{\gamma }\widehat{x_i}+\mathsf{\beta }$$

(19)

where ${\mathsf{\mu }}_B$ and ${\mathsf{\sigma }}_B^2$ denote batch mean and variance, respectively, $\widehat{x_i}$ is the normalized feature, $\mathsf{\gamma }$ and $\mathsf{\beta }$ are learnable parameters for scaling and shifting, and ϵ prevents division by zero. This process standardizes data, maintains distribution information, and optimizes data representation.

The synergetic integration of these layers fortifies the MLP’s capacity to comprehend and amalgamate interrelationships among features within the same group. This amalgamation enhances activity recognition accuracy through a robust and sophisticated learning process.

Subsequently, we employ transformer blocks to enhance attention evaluation and sequential modeling, leveraging the self-attention mechanism to improve the model’s performance. Specifically, the transformer block facilitates establishing global dependencies across various points within the input sequence, enabling the capture of longer-range semantic dependencies and enhancing the model’s efficacy in human motion recognition tasks.

Transformer blocks typically comprise several sub-layers, including a self-attention layer, a fully connected layer, and a normalization layer. The self-attention layer calculates attention weights at different positions within the input sequence, facilitating the correlation of distinct parts of the input sequence. The fully connected layer further processes and transforms the outcome from the self-attention layer. After the self-attention and fully connected layers, the input and output are combined through a residual connection, while the neural network training process is expedited via batch normalization. This helps to prevent issues like gradient vanishing or explosion, ultimately improving the model’s generalization capabilities. Multiple transformer blocks can be stacked together to construct a deeper neural network, further enhancing the model’s overall performance.

Self-attention is a core component of the transformer model, playing a crucial role in machine learning tasks in natural language processing and other domains. It can associate and interact with information from different positions in the input sequence to capture the long-term dependencies in the sequence.

The self-attention [31] calculation processes a sequence $(x_1,x_2,\dots ,x_n)$, with each $x_i$ being a vector. The self-attention layer yields an output sequence $\left(y_1,y_2,\dots ,y_n\right)$, where each $y_i,$ is also a vector. The self-attention mechanism can be explained as follows:

$$\mathrm{SelfAttention}\left(X\right)=\mathrm{softmax}\left(\frac{{X·W}_Q{\left({X·W}_K\right)}^T}{\sqrt{d_k}}\right)X·W_V$$

(20)

where $X\in R^{n\times d}$ represents the input sequence, $W_Q,W_K, \mathrm{and} W_V\in R^{d\times d_k}$ are weight matrices that map input vectors to query, key, and value vectors. The output of self-attention is a weighted average of input position vectors, with weights calculated through similarity (dot product) between position vectors after scaling and Softmax normalization.

Residual connection addresses gradient vanishing in deep neural network training by allowing input data to flow directly to subsequent layers during forward propagation. This preserves input information, alleviating information loss and distortion. This concept applies beyond convolutional neural networks and enhances various neural network structures’ training.

Finally, the weighted feature group representations were added together to obtain the final attention fusion representation. The proposed group attention module and transformer module empower the neural network to focus on different input parts, enhancing adaptability across tasks and scenes. In activity recognition, it aids the network in emphasizing activity-related input information, improving accuracy and generalization.

3.4.2. Activity Classification and Recognition

The GAM-MLP model concludes with a fully connected layer followed by a SoftMax layer for classification purposes [32]. The diagram illustrating the classification and recognition process in the MLP of layer plus SoftMax layer is shown in Figure 7 below.

The function of the SoftMax layer is to transform neural network output scores into a probability space, ensuring category probabilities range from 0 to 1, summing to 1. For neural network output scores $z=\left(z_1,z_2,\dots ,z_n\right)$, the Softmax layer’s output is given by

$$y_i=\frac{e^{z_i}}{{\sum }_{j=1}^n{e}^{z_j}}$$

(21)

Here, $e$ is the base of the natural logarithm, and $y_i$ represents the probability for the ith class.

4. HAR Datasets and Experiment Settings

4.1. Experimental Environment and Data Acquisition

For human body pose recognition using IMU sensors, we have meticulously assembled a specialized hardware and software configuration. The system features an Intel Core i9-12900K CPU at 3.50 GHz and an Nvidia GeForce RTX 3080Ti GPU with 12 GB VRAM, ensuring high-speed computation for accurate pose analysis. With 64 GB DDR4 3000 RAM, memory-intensive operations are efficiently handled. TensorFlow 2.5 serves as the deep learning framework, supported by Python 3.9 as the primary development language. PyCharm 2020.1 IDE offers a user-friendly platform for coding and debugging. Windows 10 Professional OS provides stability and reliability for precise pose recognition. This setup optimizes IMU sensor capabilities, benefiting sports biomechanics, healthcare, and animation applications. The relevant configurations of the experiment are shown in

Table 1. High-performance AI workstation for advanced deep learning applications.

Name	Specific Configuration
CPU	Intel Core i9-12900K@3.50 GHZ
Graphics card	Nvidia Geforce RTX 3080Ti (12 GB) GPU
Memory	64 GB DDR4 3000
Deep learning framework	TensorFlow 2.5
Development language	Python 3.9
Developing an IDE	Pycharm 2020.1
Operating system	Window10 Professional

.

Sensor-based human activity recognition is a sophisticated technology that utilizes sensor technology to monitor and analyze the spatial positioning and motion characteristics of the human body. It automatically recognizes and interprets various human activities, making it applicable in diverse scenarios. We employ an inertial measurement unit (IMU) sensor technology to achieve a more comprehensive and in-depth recognition and analysis of human activity information.

The datasets used in the experiment are the PAMAP2 dataset and the MultiportGAM dataset, where the PAMAP2 dataset is a public dataset that can be publicly obtained. The MultiportGAM dataset is our self-made dataset, which includes 19 different types of activities and generates a total of 11,034 sample data.

We invited 10 volunteers to collect sensor data for the MultiportGAM dataset. To ensure the accurate collection of human activity data, we utilized specific sensors for different body parts. An accelerometer is employed to measure the acceleration of the human torso, left hand, right hand, left leg, and right leg. Additionally, a magnetometer is used to measure the magnetic field strength and direction of these body parts. Furthermore, a gyroscope is utilized to measure the angular velocity of the aforementioned body segments. The eight types of daily activities collected are shown in Figure 8. By compensating for the data from the magnetometer and gyroscope, more precise localization of human joints is achieved, enhancing the accuracy of human activity identification.

The sampling frequency of the sensor is 30 Hz, and the number of collected action types is 19. Each volunteer completed these actions within 1–2 h, with an average duration of 3–5 min for each action. In order to keep the behavioral data collected by sensors closer to the real-world human behavior, some specific actions such as cycling, shooting, paddling, etc., in Figure 8 are collected using sensors with the help of relevant equipment. Daily actions such as standing, sitting, walking, going upstairs, lying down, etc., are carried out in an orderly manner under indoor conditions to ensure the stability of the collection environment and avoid magnetic field interference. In addition, each row of data includes sensors for calibration to eliminate bias and improve measurement accuracy. The collected raw action data includes action labels and timestamps for subsequent data analysis and alignment.

Table 2. Comprehensive human activity recognition datasets with sensor information.

Dataset	Action Categories	Sample Size	Sensor Type	Acquisition Location
PAMAP2 Dataset	18	65,052	inertial measurement units, acceleration sensors, magnetometer, gyroscope	Ankle Chest Wrist
MultiportGAM Dataset	19	11,034	accelerometer, magnetometer, spirometer	Torso Left Arm Right Arm Left Leg Right Leg

records the basic information of the two datasets, including activity category, sample size, and sensor type. Compared to the PAMAP2 dataset, the MultiportGAM dataset has fewer samples, but it encompasses a greater variety of sensor placement. Both datasets capture common human activities from daily life.

4.2. Sliding Window Segmentation Signal Processing

Continuous data signals gathered by sensors in practical applications are frequently plagued by noise and abrupt interference, which can dramatically reduce measurement accuracy. In the process of human activity recognition, the data of the sensor will present certain gait characteristics when the human body performs repetitive unit actions. After noise reduction via sliding window, the smoothness of the data time series is preserved and the motion characteristics are highlighted, as shown in Figure 9. The implementation of a sliding window denoising approach improves the accuracy and efficiency of the process.

To identify the optimal sliding window size and sampling frequency values, noise reduction is applied to the specific torso data adopted by the tester. The reason for adopting the torso data here is that when the human body performs periodic activities, the torso differs from the rest of the body in typical general gait characteristics. At the same time, in this study, the sliding window size was varied, ranging from 1 to 9 s. The sampling frequency was adjusted within the range of 15 to 30 Hz, with a step size of 5 Hz. The average accuracy was used as the scoring Index to solve the optimal value of the sliding window size and sampling frequency, and the evaluation scores of different machine learning algorithms were obtained as shown in Figure 10.

From the results, it can be found that the different algorithms have a peak at a window size of 5 s and a frequency of 25 Hz which indicates that choosing one particular algorithm over another does not make a difference. Also, an accuracy of 0.92–0.99 is good enough for our purposes. Therefore, these values will be used for further experiments and discussion.

5. Result and Analysis

5.1. Accuracy and Loss of the 10-Fold Cross Validation on Both Training and Test Sets

The Euler angle data collected by sensors and calculated using Algorithm 1 is used as input for the information fusion experiment after separating the training and test sets. Figure 11 shows that the upgraded GAM-MLP method’s Train acc and Test acc curves exhibit a better match.

Additionally, the accuracy of the test set and training set stabilize as the number of iteration rounds approaches 120, demonstrating outstanding robustness and generalization capabilities.

From the accuracy curves of the training and testing sets in Figure 11, it can be seen that with the increase of iteration rounds, the curves representing the accuracy of the training and testing sets steadily increase and maintain alignment. This indicates that GAM-MLP performs as well as the training data when facing new data, demonstrating good generalization ability.

5.2. Performance on Different Datasets

We proceed to compare our hybrid human activity recognition method with both baseline techniques and several cutting-edge techniques on the MultiportGAM and PAMAP2 dataset. We evaluate each model’s performance using three key metrics: accuracy, recall, and F1-score. Accuracy can intuitively measure the model’s ability to accurately predict human activity categories. The recall rate represents the proportion of the number of behavior categories correctly predicted by the model to the actual number of human behaviors. A high recall rate means that the model can more comprehensively detect human behavior and reduce missed recognition. The F1-score provides a more comprehensive evaluation of the model’s performance in human behavior recognition tasks. A model with a high F1-score demonstrates strong performance in terms of both precision and recall, being able to more accurately recognize human behavior and possessing good overall performance. This comprehensive analysis aims to uncover the effectiveness of our hybrid human pose recognition approach in the realm of human activity recognition.

The results of our comparisons on the MultiportGAM dataset are presented in

Table 3. Comparison of different methods for the MultiportGAM dataset.

Method	Accuracy (%)	Recall (%)	F1-Score (%)
GAM-MLP	96.13	96.12	96.13
SVM	82.70	82.70	82.69
DCL [33]	92.11	92.10	92.11
IN [34]	92.72	92.72	92.71
LSTM-CNN [35]	94.23	94.10	94.17

, offering a clear perspective on the respective performances. Similarly, the comparison outcomes with the PAMAP2 dataset are summarized in

Table 4. Comparison of different methods for the PAMAP2 dataset.

Method	Accuracy (%)	Recall (%)	F1-Score (%)
GAM-MLP	93.96	93.89	93.91
SVM	82.84	82.43	82.58
CNN-M [36]	93.74	93.28	93.85
LSTM-CNN	92.63	92.61	92.89
FE-CNN [37]	91.66	91.43	91.40
DCL	92.49	92.42	92.30
CE-HAR [38]	92.14	92.43	92.18
IN	91.77	91.76	91.47
TL-HAR [39]	92.33	91.83	92.08
ConvAE-LSTM [40]	94.33	-	94.46

. By examining the metrics across these datasets, we gain valuable insights into how our hybrid human pose recognition method performs in comparison to other established and advanced techniques.

Based on the comparative results, it is evident that leveraging the human pose angle model constructed from raw sensor data and the GAM-MLP model’s attention mechanism, we achieved remarkable performance on the MultiportGAM Dataset. Our hybrid approach attained an accuracy of 96.13%, a recall rate of 96.12%, and an F1-score of 96.13%. In contrast to the baseline methods, our approach demonstrated notable improvements of 13.43%, 13.42%, and 13.44% in the respective accuracy, recall, and F1-score metrics.

The hybrid approach surpasses the currently established cutting-edge methods, showcasing improvements in recognition accuracy ranging from 1.65% to 5.54%. These findings underscore the efficacy of our hybrid approach in advancing the field of human activity recognition, substantiating its superiority over both conventional baselines and advanced methodologies.

The comparative results of the PAMAP2 dataset show that the hybrid approach demonstrates certain advantages, with its recognition performance surpassing that of other state-of-the-art methods except for ConvAE-LSTM. Even when compared to ConvAE-LSTM, the hybrid approach lags behind in recognition accuracy by only 0.39%. However, the hybrid approach boasts lower model parameters and complexity, Our method reduces the overall parameter count by 19–21%, making it more advantageous for deployment on mobile devices. In summary, our human body pose model excels at extracting posture angles of various body parts to the fullest extent. By efficiently utilizing sensor data information, GAM-MLP effectively distinguishes and relates the differences and connections between actions when focusing on intra-group features. This process leads to precise recognition outcomes.

5.3. Recognition of Common Human Movements

Across the various human activity categories, the most prevalent ones include walking, standing, climbing upstairs, descending stairs, and lying down. In most scenarios, recognition often revolves around these primary actions. Therefore, we proceeded to extract these six key movement categories from both the MultiportGAM dataset and the PAMAP2 dataset. This focused evaluation allows us to assess our hybrid human pose recognition method’s classification performance specifically on these commonly encountered activities.

The experimental outcomes, illustrated in Figure 12 and Figure 13, highlight how the hybrid human pose recognition method performs when tasked with classifying these frequently observed actions within the MultiportGAM dataset and the PAMAP2 dataset.

It is evident that due to variations in the number and placement of sensors, the performance of GAM-MLP varies across different datasets. Comparing the two datasets, unlike the PAMAP2 dataset, the MultiportGAM dataset collects accelerometer, magnetometer, and gyroscope data from the chest and limbs, using two additional IMUs. In terms of daily behavior, the MultiportGAM has a longer duration, resulting in higher recognition accuracy.

On the MultiportGAM dataset, our GAM-MLP model achieves an average accuracy of 97.45% across the six common daily activities. This represents a 1.04–10.8% enhancement over other models. Notably, the highest accuracy is achieved in recognizing the “sitting down” action, reaching an accuracy of 98.94%. Even in the challenging “descending stairs” action, the model reaches an impressive precision of 93.67%. Similarly, on the PAMAP2 dataset, the “sitting down” action boasts the highest recognition rate of 99.02%, while the “descending stairs” action presents the lowest recognition rate at 93.74%. The average accuracy improves by 1.59–11.06% across the board. Through the combination of the multi-layer perceptron and group attention module, we achieve the information exchange and fusion between different features of the same group. In the training process, the neural network automatically learns and optimizes the weight and bias through the back-propagation algorithm, so that the neurons can better capture the correlation information between different features, in order to improve the accuracy and reliability of the classification.

5.4. GAM Ablation Comparison

The experiment to evaluate the optimization effect of GAM was conducted on the MultiportGAM dataset and the PAMAP2 dataset, with GAM modules removed to compare the recognition performance of human movements. The comparison results on the MultiportGAM dataset are shown in Figure 14a, while the comparison results on the PAMAP2 dataset can be observed in Figure 14b.

The experimental results showed that on the MultiportGAM dataset, compared to the original structure without adding GAM, GAM achieved an accuracy improvement of 9.38%, a recall improvement of 9.37%, and an F1-score improvement of 9.37%. On the PAMAP2 dataset, these three indicators increased by 7.15%, 7.17%, and 7.16%, respectively.

To assess the generalization capability of the GAM module, we conducted a comparative analysis with other attention modules. These alternative attention modules were integrated into the residual blocks within the framework for experimentation, and the attention modules we compared included SE and CBAM. The MultiportGAM dataset and PAMAP2 dataset were employed as the dataset for our experiments. The experimental results are presented in

Table 5. Comparison of different methods for the MultiportGAM dataset.

Attention Module	Accuracy (%)	Recall (%)	F1-Score (%)
GAM-MLP	96.13	96.12	96.13
SE-MLP	92.74	92.13	92.48
CBAM-MLP	91.11	92.02	92.08

and

Table 6. Comparison of different methods for the PAMAP2 dataset.

Attention Module	Accuracy (%)	Recall (%)	F1-Score (%)
GAM-MLP	93.96	93.89	93.91
SE-MLP	89.34	88.75	89.16
CBAM-MLP	88.75	89.13	88.96

.

It can be observed that compared to GAM, the application of CE and CBAM, two general attention mechanisms, in the hybrid method, results in an accuracy improvement of about 3% compared to the proposed framework without the application of attention mechanisms. However, there is still a gap of 3.4–5% compared to GAM-MLP, indicating that GAM has a higher adaptability to the proposed framework. This stems from the manual grouping module of GAM, which reduces errors between specified groups while adapting to the active features of the dataset.

Thanks to GAM’s ability to extract temporal features of sensor data and dynamically adjust the weights of accelerometer, gyroscope, and magnetometer data, GAM-MLP can better capture the associations between multiple sensors in different time periods, thereby improving the accuracy of action recognition.

5.5. Identification of 19 Types of Diverse Actions

In the recognition of 19 activities such as walking, jumping, and shooting baskets, GAM-MLP demonstrates exceptional capabilities due to the extraction of spatiotemporal features from raw sensor data that has been processed for activity construction. The model achieves an impressive average recognition accuracy of 96.13%. Notably, in the 3rd (Sitting), 4th (Lying), 15th (Cycling), and 19th (Rowing) activity categories, characterized by distinct features, the model achieves nearly complete recognition. While accuracy may slightly decrease in other activity categories, it still maintains a high precision of 93.54%. The confusion matrix for 19 types of action recognition is shown in Figure 15.

In the recognition process of 19 types of activities, we used the attention mechanism to dynamically adjust the focus of the neural network, so as to better capture the key information and improve the accuracy of action recognition. This mechanism enables the neural network to adaptively focus on the important feature information, thereby avoiding the information redundancy and unnecessary computational overhead caused by the independent learning of the weight for each neuron in the conventional neural network.

6. Conclusions and Future Work

This paper offered a hybrid HAR approach using an MLP neural network and Euler angle extraction based on IMU sensors. In the method, by employing accelerometers, we precisely capture the acceleration of the torso, left arm, right arm, left leg, and right leg. This meticulous data collection approach is supplemented by gyroscopes, which monitor the angular velocity of various bodily components, and magnetometers, which gauge the direction and strength of magnetic fields. Refining joint placement through compensation of magnetometer and gyroscope data enhances the accuracy of human activity identification. Experimental findings, derived from tests conducted on widely used datasets, underscore the superiority of our proposed GAM-MLP model over existing deep learning-based models. On the PAMAP2 dataset and the MultiportGAM dataset, accuracy rates of 93.96% and 96.13% were achieved, respectively. Particularly noteworthy is the 97.45% accuracy achieved in recognizing six common daily activities within the MultiportGAM dataset. Compared to traditional methods and some advanced techniques, an improvement of 1.04% to 10.8% in accuracy was attained. Nevertheless, there is still room for improvement in GAM-MLP in identifying confusing actions

Looking forward, our future endeavors are geared towards exploring the realm of online deep learning models and expanding the horizons of both human activity recognition (HAR) and GAM-MLP to adeptly handle vast volumes of multivariate data. Specifically, in the future, additional inference modules can be considered in the GAM-MLP network to accelerate recognition speed and accuracy. When dealing with multivariate data, increasing the depth of the network appropriately may improve the recognition effect. Furthermore, the attention mechanism is intricately connected to the challenges posed by big data. Big data often exhibits diversity and variability, and attention mechanisms can adapt to various data distributions and characteristics, thereby enabling models to better accommodate shifts in the data. We are committed to enhancing the robustness of attention mechanisms in HAR recognition. They can assist models in focusing on crucial information, suppressing noise and outliers, and dynamically adjusting weights based on changing data, thereby maintaining model stability within the context of big data.