Exploring Cutout and Mixup for Robust Human Activity Recognition on Sensor and Skeleton Data ^†

Abstract

Human Activity Recognition (HAR) is an essential area of research in Artificial Intelligence and Machine Learning, with numerous applications in healthcare, sports science, and smart environments. While several advancements in the field, such as attention-based models and Graph Neural Networks, have made great strides, this work focuses on data augmentation methods that tackle issues like data scarcity and task variability in HAR. In this work, we investigate and expand the use of mixup and cutout data augmentation methods to sensor-based and skeleton-based HAR datasets. These methods were first widely used in Computer Vision and Natural Language Processing. We use both augmentation techniques, customized for time-series and skeletal data, to improve the robustness and performance of HAR models by diversifying the data and overcoming the drawbacks of having limited training data. Specifically, we customize mixup data augmentation for sensor-based datasets and cutout data augmentation for skeleton-based datasets with the goal of improving model accuracy without adding more data. Our results show that using mixup and cutout techniques improves the accuracy and generalization of activity recognition models on both sensor-based and skeleton-based human activity datasets. This work showcases the potential of data augmentation techniques on transformers and Graph Neural Networks by offering a novel method for enhancing time series and skeletal HAR tasks.

1. Introduction

Human Activity Recognition (HAR) has become a crucial area of research within Artificial Intelligence and Machine Learning due to its wide range of practical applications [1,2,3], driven by wearable devices and smart sensors, in our daily lives. HAR often involves automatically detecting and classifying human activities based on data collected from various sensors, such as accelerometers and gyroscopes, typically worn by subjects. Recognizing human activities holds significant potential in many areas. In healthcare, various Machine Learning technologies have been used, including deep neural networks for activity detection [4,5,6], to ensure the welfare of hospital patients and the general population. When it comes to sports science, deep learning models have made a positive impact in modeling sports like football [7] and karate [8], on fall detection [9], posture recognition [10], and many others [11]. HAR has also been implemented in various areas, such as home automation and energy efficiency [12,13,14,15], anomaly detection and public safety [16,17], and gesture recognition and virtual reality [18,19].

Recent advancements in HAR have leveraged deep learning architectures to improve recognition accuracy [11,20,21,22]. For sensor-based HAR, Convolutional Neural Networks (CNNs) have been effective in capturing spatial features from sensor data, while Long Short-Term Memory (LSTM) networks model temporal dependencies [23,24,25,26]. Transformer models, known for their self-attention mechanisms, have also been applied to handle long-range temporal dependencies in time-series data [27,28]. Hybrid models combining CNNs and LSTMs have shown improved performance by leveraging both spatial and temporal features [24,26,29]. Despite their outstanding performance in an experimental environment, the real-world environment still poses a problem, mainly because of poor data distribution [22].

In skeleton-based HAR, Graph Neural Networks (GNNs) have gained prominence due to their ability to model the human body’s skeletal structure as a graph. By representing human body joints as nodes and the connections between them as edges, GNNs can effectively capture both spatial and temporal dependencies in skeleton data, making them well-suited for HAR tasks. Moreover, GNNs can handle the non-Euclidean structure of skeleton data, which provides a significant advantage over traditional methods that assume grid-like data structures [30,31,32,33,34]. Models like the Spatial–Temporal Graph Convolutional Network (ST-GCN) capture spatial relationships between joints and temporal dynamics across frames [35]. Adaptive Graph Convolutional Networks (AGCNs) and attention mechanisms have further enhanced the capacity to focus on relevant joints and motions [36]. Despite these advancements, the models often require large, diverse datasets to perform well and may not generalize effectively to new subjects or environments due to data scarcity and variability [37].

In spite of the amount of research done in the area of Human Activity Recognition, several challenges in the field need to be addressed, such as lack of data [38], variability of data measuring hardware and techniques [39], and misalignment of activities [40,41]. While sensors generate vast amounts of data, labeled and diverse datasets for HAR remain limited. On top of that, human motions are complex and varied, making accurate recognition challenging. In this research, we want to address the issue of HAR data scarcity and low diversity to provide a potential solution. This issue has been addressed by T. Alafif et al. [42], who discussed the issue of data scarcity in abnormal behavior detection in crowds and tried to address the problem by using generative adversarial networks to augment the dataset with artificial samples. On the other hand, Q. Zhu et al. [43] proposed using semi-supervised Machine Learning to use unlabeled data alongside labeled datasets to improve activity recognition with smartphone inertial sensors.

However, a significant challenge in HAR is the scarcity and limited diversity of available datasets. Collecting large-scale, annotated HAR datasets is resource-intensive due to the need for specialized equipment, privacy considerations, and the inherent variability of human movements. Traditional data augmentation techniques used in image processing do not directly apply to time-series and skeleton data, leaving a gap in effectively enhancing HAR datasets. In this research, we explore the use of mixup and cutout data augmentation [44,45] alongside attention-based architecture, which was first proposed by A. Vaswani et al. [46], and Graph Neural Networks, which have gained popularity in the HAR space [34], in order to tackle the problem of low diversity and potential scarcity when training activity recognition models. Our experimental results show that this technique improves model accuracy in state-of-the-art activity recognition models.

To summarize, the contributions of this paper are:

• We introduce and adapt the mixup data augmentation technique for sensor-based time-series data in HAR, improving data diversity and model generalization while preserving the temporal structure.
• We extend the cutout data augmentation technique to skeleton-based HAR datasets to simulate partial occlusions and missing data in joint positions, thereby improving model robustness.
• We empirically demonstrate that integrating these techniques with attention-based models and Graph Neural Networks generally improves performance and accuracy across various HAR datasets.

2. Background Literature

Human Activity Recognition (HAR) is a field within Artificial Intelligence and Machine Learning that focuses on the automatic detection and classification of human activities based on data collected from various methods, including both sensor-based and skeleton-based approaches [1,38]. Sensor-based HAR typically involves accelerometers, gyroscopes, or other wearable devices to capture movement and orientation data. At the same time, skeleton-based HAR uses data derived from visual sensors such as RGB cameras or depth sensors, which track the positions of skeleton joints over time. HAR has significant applications in diverse areas, including healthcare, where it can monitor patient activities and detect falls, sports science for performance analysis, and smart home environments to enhance automation and security [4,11]. Understanding and recognizing human activities is crucial for developing systems that provide personalized feedback, improve safety, and support independent living.

Traditional approaches to HAR often involve classical Machine Learning techniques such as k-Nearest Neighbors (k-NN) [47,48], Support Vector Machines (SVM), and decision trees [29,49]. These methods rely heavily on manual feature extraction and selection processes, where domain experts identify relevant features from the raw sensor data. While these approaches have been effective to some extent, they face challenges, such as the labor-intensive nature of feature engineering and the difficulty in capturing complex patterns in the data. Additionally, these models may struggle to scale and generalize across different datasets and environments due to their reliance on handcrafted features [38]. The advancement towards deep learning has significantly improved HAR by enabling models to learn features from the raw data automatically [11].

Convolutional Neural Networks (CNNs) have been used to capture spatial dependencies in the data [50], while Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks are adept at modeling temporal dependencies [51]. These deep learning models have shown superior performance compared to traditional methods, as they can handle large volumes of data and extract hierarchical features that represent complex patterns in human activities. Similarly, attention-based models have revolutionized the fields of Computer Vision (CV) and Natural Language Processing (NLP) by introducing the transformer architecture that relies on self-attention mechanisms [46]. Unlike traditional sequence models, transformers can capture long-range dependencies and complex patterns in data without the need for recurrent connections. This capability has led to the development of state-of-the-art models in NLP, such as BERT and GPT-3 [52,53], and in Computer Vision, such as Vision Transformers (ViTs) [54]. The scalability and efficiency of transformers make them promising candidates for applications in time-series data like HAR, where capturing temporal dependencies is crucial.

On the other hand, Graph Neural Networks (GNNs), first proposed by M. Gori et al. in 2005 [33] and later developed by F. Scarselli et al. in 2009 [32], have also shown great promise when it comes to activity recognition tasks. For skeleton data, typically represented as sequences of joints or key-points in a graph structure, GNNs are a natural fit due to their ability to capture complex relationships between body joints over time. Their usability was shown in one of the initial works of Yan et al. [55], who implemented GNNs on top of skeleton sequences and proposed a model they named the Spatial–Temporal Graph Convolutional Neural Network (ST-GCN). GNNs can model both spatial and temporal dependencies by treating each joint as a node and the connections between them as edges, allowing for more precise activity recognition. Studies have demonstrated the effectiveness of GNNs in extracting both local and global body movement patterns, which are essential for recognizing complex human activities. In particular, approaches such as the aforementioned ST-GCNs and Dynamic Graph Convolutional Networks (DGCNs) [56] have been particularly successful in skeleton-based HAR, where the model is designed to simultaneously learn spatial features from the skeleton’s structure and temporal features from the sequential nature of the data.

Data augmentation is a technique used to improve the generalization and robustness of Machine Learning models by artificially increasing the diversity of the training dataset. The effectiveness of data augmentation has been demonstrated across various domains, including vision and natural language, which have led to improvements in model performance and stability [57]. Mixup is a specific data augmentation technique that generates new training samples by linearly interpolating between pairs of existing samples and their corresponding labels [44]. Initially introduced for image classification tasks, mixup has been shown to improve model generalization and robustness by encouraging the model to behave linearly between training examples. The mathematical formulation of mixup involves creating a new sample as a weighted combination of two samples and their labels, effectively smoothing the decision boundary of the classifier, according to the authors H. Zhang et al. in their research [44]. Previous studies have extended mixup data augmentation to other domains, including NLP, where it has also demonstrated significant benefits [58].

Another powerful data augmentation technique is cutout, which enhances model robustness by randomly masking out sections of the input data during training. Initially proposed for image classification by T. DeVries et al. in 2017 [45], cutout works by occluding parts of the input samples, forcing the model to rely on surrounding information to make accurate predictions. This technique has been shown to improve generalization by making the model less reliant on specific features and more capable of capturing broader patterns in the data. In the context of Human Activity Recognition (HAR), cutout data augmentation can be adapted to mask out key-point sections in skeleton data, simulating real-world conditions where some sensor data might be noisy or missing. This research shows that introducing such variability during training cutout helps models become more robust to partial occlusions and missing data, ultimately leading to improved performance in HAR tasks. By demonstrating the effectiveness of our approach through our experiments, we provide a potential solution to some of the critical challenges in HAR. The promising results from our experiments clearly show the value of this approach, paving the way for further exploration and application of data augmentation techniques in different domains of Human Activity Recognition.

3. Methodology

Our proposed method applies both sensor-based and skeleton-based Human Activity Recognition (HAR) models, incorporating mixup and cutout data augmentation techniques. The datasets used in this study consist of time-series sensor data collected from devices such as accelerometers and gyroscopes, and skeleton data extracted from visual sensors like RGB cameras or depth sensors. Each dataset contains sequences of sensor readings from subjects performing various activities for sensor-based HAR, while skeleton-based HAR tracks the positions of important body joints over time. The continuous sensor and skeleton data are segmented into fixed-length windows to create consistent input samples for the models, with each segment labeled according to the corresponding activity.

This research applies the mixup data augmentation technique for time-series sensor data to address data scarcity and improve model robustness. Mixup generates new training samples by linearly interpolating between pairs of existing sensor data samples and their corresponding labels, enhancing the diversity of the data. For skeleton-based HAR, we employ the cutout data augmentation technique, which involves masking random parts of the skeleton data to improve model robustness by forcing it to focus on incomplete or missing joint information, ultimately enhancing the generalization capabilities of the model.

Building on this approach, we first apply the mixup data augmentation technique to the sensor-based HAR data, which generates new training samples by blending existing data points and their labels. For each training iteration, pairs of time-series samples $(X_i,y_i)$ and $(X_j,y_j)$ are randomly selected from the training dataset. A mixing coefficient $\lambda ϵ[0,1]$ is sampled from Beta distribution $\lambda ~Beta(\alpha ,\alpha )$. The selected pairs are linearly combined to generate new samples:

$$\tilde{X}=\lambda X_i+(1-\lambda )X_j$$

(1)

$$\tilde{y}=\lambda y_i+(1-\lambda )y_j$$

(2)

Here, $X_i$ and $X_j$ are time-series matrices of the shape $\left(T,C\right)$, where $T$ is the sequence length and $C$ is the number of sensor channels. This implies $X_i{,X}_jϵ{\mathbb{R}}^{T\times C}$ and $y_i{,y}_jϵ{\mathbb{R}}^K$, where $K$ is the number of activity classes. The labels $y_i$ and $y_j$ are one-hot encoded vectors representing the activities.

The graphs in Figure 1 and Figure 2 display sensor readings from a human activity sample before (Figure 1) and after (Figure 2) applying the mixup technique. The data are presented across six separate channels, each representing different sensor readings over time. The x-axis of each plot denotes time, while the y-axis indicates the sensor reading values. For each channel, the signal exhibits a combination of patterns derived from two different samples, reflecting the linear interpolation characteristic of mixup.

The mixup augmentation technique blends the time-series data from two separate activities, creating synthetic samples that enhance the training dataset’s diversity. The representation of each class after dimensionality reduction with t-SNE [59], similar to [22], can be seen in Figure 3. We can see in the figure that mixup provides additional data points to improve model generalization and provide additional data points for classes with a lower number of samples.

We employ transformer-based models proposed by S. Ek et al. [22] and by I. Luptáková [60]. Their transformer architecture is a customized form of the Vision Transformer [54], with some modifications to adapt it to HAR data. For the sake of clarity, we simplify the model architecture and explain its components below. First, the input time-series data is projected into a higher-dimensional space using a linear embedding layer, transforming the raw sensor readings into a suitable format for the Vision Transformer and then applying positional embeddings to keep temporal order, as shown in Equations (3) and (4):

$$E=Embedding\left(X\right)$$

(3)

$$E_{pos}=E+PositionalEncoding(T,d_{model})$$

(4)

where $d_{model}$ is the dimentionality of the embeddings. The model consists of multiple Vision Transformer encoder layers, each comprising multi-head self-attention mechanisms and feed-forward networks:

$$H_l=TransformerEncoderLayer\left(H_{l-1}\right)$$

(5)

for $l=1,\dots ,L$, where $H_0=E_{pos}$ and $L$ is the number of encoder layers. The output of the final encoder layer is passed through a global average pooling layer to aggregate the sequence information into a fixed-size vector.

$$z=GlobalAveragePooling1D\left(H_L\right)$$

(6)

The pooled vector is then passed through fully connected layers to produce the final activity classification. The cross-entropy loss function is used to train the model, comparing the predicted activity probabilities $\widehat{y}$ with the mixup-augmented labels $\tilde{y}$:

$$\mathcal{L}=-{\sum }_k{\tilde{y}}_k\log {\widehat{y}}_k$$

(7)

This methodology of implementing mixup on sensor-based HAR data is empirically validated through experiments on four different HAR datasets, comparing the results to baseline models without mixup augmentation. The findings demonstrate that mixup augmentation leads to improved accuracy and robustness in transformer-based HAR models.

After applying mixup for sensor-based HAR, we turn our focus to skeleton-based HAR, where we utilize the cutout data augmentation technique to improve model robustness by randomly masking portions of the skeleton data. Figure 4 shows that the skeleton key-points are randomly masked as a data augmentation technique. Given a skeleton sample $X\in {\mathbb{R}}^{T\times J\times D}$, where $T$ is the sequence length, $J$ is the number of joints in the skeleton, and $D$ is the dimensionality of each joint, cutout for skeleton data can be defined as randomly selecting a set of joints ${\mathfrak{I}}_{cutout}\subseteq \{1,2,3,J\}$ to be masked, where the size of ${\mathfrak{I}}_{cutout}$ is controlled by a hyperparameter $p$, representing the proportion of joints to be cutout. Then, for each selected joint ${J\in \mathfrak{I}}_{cutout}$, set its coordinates to zero or an alternative value, such as the mean of the dataset across all frames, such that ${\mathrm{x}}_{\mathrm{t},\mathrm{j}}=0$ for all $t\in \left\{1,2,\dots ,T\right\},j\in {\mathfrak{I}}_{cutout}$. The masking function $M$ can be represented as shown in Equation (8):

$$\mathrm{M}{\left(\mathrm{x}\right)}_{t,j}=\left\{\begin{array}{l}{\mathrm{x}}_{\mathrm{t},\mathrm{j}}ifj\notin {\mathfrak{I}}_{cutout},\\ 0j\in {\mathfrak{I}}_{cutout}\end{array}\right.$$

(8)

This cutout augmentation technique effectively masks or “removes” selected joints from the skeleton data, simulating missing or occluded parts of the body, as shown in Figure 5. By doing so, the model is trained to recognize activities using only the remaining visible joints, encouraging it to focus on broader movement patterns rather than relying on specific joints. This enhances the model’s robustness, improving its ability to generalize to real-world scenarios where sensor data might be incomplete or noisy.

4. Results

Our experimental results were obtained on an NVIDIA Ampere A100 GPU with 80 gigabytes of video RAM, manufactured by NVIDIA Corporation, based in Santa Clara, CA, USA. When it comes to vision mixup methods for sensor-based data, we tested our method on four datasets, namely HHAR [61], MotionSense [62], UCI-HAR [63], and the RealWorld dataset [64]. In our experiments, we show that models trained on mixup augmented datasets produce better results than the baseline transformer-based models [22,60], as shown in

Table 1. Comparison of transformer-based HAR model accuracy using clean, mixup, and combined training data across different datasets.

Architecture	Training Data	HHAR	MotionSense	UCI	RealWorld	Average
HART [22]	Clean dataset	98.01%	98.17%	93.32%	93.94%	95.86%
	Mixup dataset	98.15%	97.98%	94.50%	94.54%	96.29%
	Clean + Mixup	98.22%	98.30%	94.44%	94.66%	96.41%
HAR-Transformer [60]	Clean dataset	94.99%	97.33%	89.62%	88.57%	92.63%
	Mixup dataset	96.53%	97.40%	92.26%	91.49%	94.42%
	Clean + Mixup	97.04%	97.91%	91.75%	91.52%	94.56%

and, more succinctly, in Figure 6.

The table illustrates the accuracy of the transformer model across three different training data scenarios: clean data, mixup data, and a combination of mixup and clean data. For the HHAR dataset, the model trained on clean data achieved an accuracy of 98.01%, while the model trained on mixup augmented data reached a slightly higher accuracy of 98.15%. The highest accuracy of 98.22% was observed when the model was trained on a combination of mixup and clean data, demonstrating the effectiveness of our approach in leveraging both augmented and original data to enhance model performance. Similarly, the MotionSense dataset showed a slight decrease in accuracy when trained solely on mixup data (97.98%) compared to clean data (98.17%). However, the combination of mixup and clean data resulted in an improved accuracy of 98.3%, highlighting the robustness of our method in integrating augmented data for better generalization. For the UCI-HAR dataset, the model trained on mixup data achieved a significantly higher accuracy of 94.5%, compared to 93.32% with clean data. This substantial improvement underscores the potential of mixup augmentation in scenarios with limited training data or higher variability. The combination of mixup and clean data also performed well, achieving an accuracy of 94.44%, close to the mixup-only scenario. The RealWorld dataset also benefited from mixup augmentation. The model trained on clean data achieved an accuracy of 93.94%, while the mixup augmented data resulted in an accuracy of 94.54%. The highest accuracy of 94.66% was observed with the combination of mixup and clean data, reinforcing the trend seen across other datasets.

On the other hand, we used the PYSKL library proposed in [65] to test cutout augmentation on the STGCN [55] and STGCN++ [65] models.

Table 2. Performance comparison of ST-GCN and ST-GCN++ models with and without cutout augmentation on the NTU 60 and NTU 120 datasets.

Dataset Configuration	NTU RGB + D		NTU RGB + D 120
	XSubject	XView	XSubject	XView
STGCN [55]	88.31%	94.89%	83.05%	88.45%
STGCN with cutout	88.50%	95.48%	83.25%	88.30%
STGCN++ [65]	88.02%	94.93%	84.35%	88.64%
STGCN++ with cutout	88.23%	94.98%	84.16%	88.68%

compares the performance of the ST-GCN and ST-GCN++ models with and without the application of cutout augmentation on the NTU RGB + D [66] and NTU RGB + D 120 [67] action recognition datasets, evaluated in both cross-subject (XSubject) and cross-view (XView) settings. The NTU RGB + D dataset is one of the most extensive skeleton-based action recognition datasets, consisting of 60 different actions performed by 40 subjects in front of an RGB + D camera, with data captured from multiple angles. The extended NTU RGB + D 120 dataset includes 120 action classes, covering a more comprehensive range of activities and more diverse scenarios. The results demonstrate that cutout augmentation provides a noticeable improvement in several cases, particularly in the XView setting. For instance, in NTU RGB + D, the accuracy for ST-GCN with cutout increased from 94.89% to 95.48%, while ST-GCN++ with cutout showed a slight improvement from 94.93% to 94.98%. Similarly, for the NTU RGB + D 120 dataset, cutout improved the performance of ST-GCN from 83.05% to 83.25% in the XSubject setting, and the accuracy of ST-GCN++ in the XView setting increased from 88.64% to 88.68%. These results suggest that the cutout augmentation technique helps the models generalize better, particularly in scenarios involving different views or subjects, by forcing the model to focus on incomplete data and improving its robustness.

Overall, the results clearly indicate that both mixup and cutout augmentations, whether used alone or in combination with clean data, enhance the accuracy of models for HAR tasks. Both mixup applied to sensor-based data and cutout to skeleton-based data contribute to improving model robustness. The combination of augmented and clean data often yields the best performance, suggesting that integrating these augmentation techniques can effectively address data scarcity and variability issues, leading to more robust and generalizable models across different HAR modalities.

5. Conclusions

In this research, we explored the application of transformer models, specifically Vision Transformers (ViTs), for Human Activity Recognition (HAR) using time-series data. To address the prevalent data scarcity issue and enhance HAR model robustness, we adapted the mixup data augmentation technique for time-series data. Our experiments on four diverse HAR datasets (HHAR, MotionSense, UCI-HAR, and RealWorld) consistently demonstrated that models trained with mixup-augmented data outperform those trained on clean data alone.

In addition to mixup, we implemented cutout augmentation on Graph Neural Network (GNN) models trained on skeleton-based HAR datasets. By masking random joints from the skeleton data, cutout forces the GNN models to learn activity patterns from incomplete data, improving generalization and robustness, particularly in scenarios where skeleton data may be noisy or incomplete. The application of cutout augmentation on skeleton datasets such as NTU RGB + D 60 and NTU RGB + D 120 showed noticeable performance gains, particularly in cross-subject and cross-view evaluations.

Our approach of combining mixup augmentation with Vision Transformer models and cutout augmentation with GNN models for skeleton data presents a novel and effective methodology for advancing HAR. This combination leverages the strengths of both techniques: the powerful temporal dependency capture of transformers for time-series data and the spatial–temporal representation learning of GNNs for skeleton data. Together, these approaches address key challenges in HAR, such as variability in sensor data, incomplete skeleton data, and the need for robust, scalable models.