Search Results (96)

Autism Spectrum Disorder (ASD) poses significant challenges in diagnosis due to its diverse symptomatology and the complexity of early detection. Atypical gait and gesture patterns, prominent behavioural markers of ASD, hold immense potential for facilitating early intervention and optimising treatment outcomes. These patterns can be efficiently and non-intrusively captured using modern computational techniques, making them valuable for ASD recognition. Various types of research have been conducted to detect ASD through deep learning, including facial feature analysis, eye gaze analysis, and movement and gesture analysis. In this study, we optimise a dual-stream architecture that combines image classification and skeleton recognition models to analyse video data for body motion analysis. The first stream processes Skepxels—spatial representations derived from skeleton data—using ConvNeXt-Base, a robust image recognition model that efficiently captures aggregated spatial embeddings. The second stream encodes angular features, embedding relative joint angles into the skeleton sequence and extracting spatiotemporal dynamics using Multi-Scale Graph 3D Convolutional Network(MSG3D), a combination of Graph Convolutional Networks (GCNs) and Temporal Convolutional Networks (TCNs). We replace the ViT model from the original architecture with ConvNeXt-Base to evaluate the efficacy of CNN-based models in capturing gesture-related features for ASD detection. Additionally, we experimented with a Stack Transformer in the second stream instead of MSG3D but found it to result in lower performance accuracy, thus highlighting the importance of GCN-based models for motion analysis. The integration of these two streams ensures comprehensive feature extraction, capturing both global and detailed motion patterns. A pairwise Euclidean distance loss is employed during training to enhance the consistency and robustness of feature representations. The results from our experiments demonstrate that the two-stream approach, combining ConvNeXt-Base and MSG3D, offers a promising method for effective autism detection. This approach not only enhances accuracy but also contributes valuable insights into optimising deep learning models for gesture-based recognition. By integrating image classification and skeleton recognition, we can better capture both global and detailed motion patterns, which are crucial for improving early ASD diagnosis and intervention strategies. Full article

Soft sensors are designed to be flexible, making them ideal for wearable devices as they can conform to the human body during motion, capturing pertinent information effectively. However, once these wearable sensors are constructed, modifying them is not straightforward without undergoing a re-prototyping process. In this study, we introduced a novel design for a modular soft sensor unit (M2SU) that incorporates a short, wire-shaped sensory structure made of eutectogel, with magnetic blocks at both ends. This design facilitates the easy assembly and reversible integration of the sensor directly onto a wearable device in situ. Leveraging the piezoresistive properties of eutectogel and the dual conductive and magnetic characteristics of neodymium magnets, our sensor unit acts as both a sensing element and a modular component. To explore the practical application of M2SUs in wearable sensing, we equipped a glove with 8 M2SUs. We evaluated its performance across three common gesture recognition tasks: numeric keypad typing (Task 1), symbol drawing (Task 2), and uppercase letter writing (Task 3). Employing a 1D convolutional neural network to analyze the collected data, we achieved task-specific accuracies of 80.43% (Top 3: 97.68%) for Task 1, 88.58% (Top 3: 96.13%) for Task 2, and 79.87% (Top 3: 91.59%) for Task 3. These results confirm that our modular soft sensor design can facilitate high-accuracy gesture recognition on wearable devices through straightforward, in situ assembly. Full article

This study aims to explore the potential of exergaming (which can be used along with prescriptive medication for children with spinal muscular atrophy) and examine its effects on monitoring and diagnosis. The present study focuses on comparing models trained on joint data for gesture detection, which has not been extensively explored in previous studies. The study investigates three approaches to detect gestures based on 3D Microsoft Azure Kinect joint data. We discuss simple decision rules based on angles and distances to label gestures. In addition, we explore supervised learning methods to increase the accuracy of gesture recognition in gamification. The compared models performed well on the recorded sample data, with the recurrent neural networks outperforming feedforward neural networks and decision trees on the captured motions. The findings suggest that gesture recognition based on joint data can be a valuable tool for monitoring and diagnosing children with spinal muscular atrophy. This study contributes to the growing body of research on the potential of virtual solutions in rehabilitation. The results also highlight the importance of using joint data for gesture recognition and provide insights into the most effective models for this task. The findings of this study can inform the development of more accurate and effective monitoring and diagnostic tools for children with spinal muscular atrophy. Full article

Video emotion recognition faces significant challenges due to the strong spatiotemporal coupling of dynamic expressions and the substantial variations in cross-scale motion patterns (e.g., subtle facial micro-expressions versus large-scale body gestures). Traditional methods, constrained by limited receptive fields, often fail to effectively balance multi-scale correlations between local cues (e.g., transient facial muscle movements) and global semantic patterns (e.g., full-body gestures). To address this, we propose an enhanced attention module integrating multi-dilated convolution and dynamic feature weighting, aimed at improving spatiotemporal emotion feature extraction. Building upon conventional attention mechanisms, the module introduces a multi-branch parallel architecture. Convolutional kernels with varying dilation rates (1, 3, 5) are designed to hierarchically capture cross-scale the spatiotemporal features of low-scale facial micro-motion units (e.g., brief lip tightening), mid-scale composite expression patterns (e.g., furrowed brows combined with cheek raising), and high-scale limb motion trajectories (e.g., sustained arm-crossing). A dynamic feature adapter is further incorporated to enable context-aware adaptive fusion of multi-source heterogeneous features. We conducted extensive ablation studies and experiments on popular benchmark datasets such as the VideoEmotion-8 and Ekman-6 datasets. Experiments demonstrate that the proposed method enhances joint modeling of low-scale cues (e.g., fragmented facial muscle dynamics) and high-scale semantic patterns (e.g., emotion-coherent body language), achieving stronger cross-database generalization. Full article

Human gesture image recognition is the process of identifying, deciphering, and classifying human gestures in images or video frames using computer vision algorithms. These gestures can vary from the simplest hand motions, body positions, and facial emotions to complicated gestures. Two significant problems affecting the performance of human gesture picture recognition methods are ambiguity and invariance. Ambiguity occurs when gestures have the same shape but different orientations, while invariance guarantees that gestures are correctly classified even when scale, lighting, or orientation varies. To overcome this issue, hand-crafted features can be combined with deep learning to greatly improve the performance of hand gesture image recognition models. This combination improves the model’s overall accuracy and dependability in identifying a variety of hand movements by enhancing its capacity to record both shape and texture properties. Thus, in this study, we propose a hand gesture recognition method that combines Reset50 model feature extraction with the Tamura texture descriptor and uses the adaptability of GAM to represent intricate interactions between the features. Experiments were carried out on publicly available datasets containing images of American Sign Language (ASL) gestures. As Tamura-ResNet50-OptimizedGAM achieved the highest accuracy rate in the ASL datasets, it is believed to be the best option for human gesture image recognition. According to the experimental results, the accuracy rate was 96%, which is higher than the total accuracy of the state-of-the-art techniques currently in use. Full article

Body posture dynamics have garnered significant attention in recent years due to their critical role in understanding the emotional states conveyed through human movements during social interactions. Emotions are typically expressed through facial expressions, voice, gait, posture, and overall body dynamics. Among these, body posture provides subtle yet essential cues about emotional states. However, predicting an individual’s gait and posture dynamics poses challenges, given the complexity of human body movement, which involves numerous degrees of freedom compared to facial expressions. Moreover, unlike static facial expressions, body dynamics are inherently fluid and continuously evolving. This paper presents an effective method for recognizing 17 micro-emotions by analyzing kinematic features from the GEMEP dataset using video-based motion capture. We specifically focus on upper body posture dynamics (skeleton points and angle), capturing movement patterns and their dynamic range over time. Our approach addresses the complexity of recognizing emotions from posture and gait by focusing on key elements of kinematic gesture analysis. The experimental results demonstrate the effectiveness of the proposed model, achieving a high accuracy rate of 91.48% for angle metric + DNN and 93.89% for distance + DNN on the GEMEP dataset using a deep neural network (DNN). These findings highlight the potential for our model to advance posture-based emotion recognition, particularly in applications where human body dynamics distance and angle are key indicators of emotional states. Full article

In this work, a conductive composite film composed of multi-walled carbon nanotubes (MWCNTs) and multi-layer Ti₃C₂Tx MXene nanosheets is used to construct a strain sensor on sandpaper Ecoflex substrate. The composite material forms a sophisticated conductive network with exceptional electrical conductivity, resulting in sensors with broad detection ranges and high sensitivities. The findings indicate that the strain sensing range of the Ecoflex/Ti₃C₂Tx/MWCNT strain sensor, when the mass ratio is set to 5:2, extends to 240%, with a gauge factor (GF) of 933 within the strain interval from 180% to 240%. The strain sensor has demonstrated its robustness by enduring more than 33,000 prolonged stretch-and-release cycles at 20% cyclic tensile strain. Moreover, a fast response time of 200 ms and detection limit of 0.05% are achieved. During application, the sensor effectively enables the detection of diverse physiological signals in the human body. More importantly, its application in a data glove that is coupled with machine learning and uses the Support Vector Machine (SVM) model trained on the collected gesture data results in an impressive recognition accuracy of 93.6%. Full article

Over recent years, automated Human Activity Recognition (HAR) has been an area of concern for many researchers due to its widespread application in surveillance systems, healthcare environments, and many more. This has led researchers to develop coherent and robust systems that efficiently perform HAR. Although there have been many efficient systems developed to date, still, there are many issues to be addressed. There are several elements that contribute to the complexity of the task, making it more challenging to detect human activities, i.e., (i) poor lightning conditions; (ii) different viewing angles; (iii) intricate clothing styles; (iv) diverse activities with similar gestures; and (v) limited availability of large datasets. However, through effective feature extraction, we can develop resilient systems for higher accuracies. During feature extraction, we aim to extract unique key body points and full-body features that exhibit distinct attributes for each activity. Our proposed system introduces an innovative approach for the identification of human activity in outdoor and indoor settings by extracting effective spatio-temporal features, along with a Multi-Class Support Vector Machine, which enhances the model’s performance to accurately identify the activity classes. The experimental findings show that our model outperforms others in terms of classification, accuracy, and generalization, indicating its efficient analysis on benchmark datasets. Various performance metrics, including mean recognition accuracy, precision, F1 score, and recall assess the effectiveness of our model. The assessment findings show a remarkable recognition rate of around 88.61%, 87.33, 86.5%, and 81.25% on the BIT-Interaction dataset, UT-Interaction dataset, NTU RGB + D 120 dataset, and PKUMMD dataset, respectively. Full article

Flexible electronics is pursuing a new generation of electronic skin and human–computer interaction. However, effectively detecting large dynamic ranges and highly sensitive human movements remains a challenge. In this study, flexible strain sensors with a self-assembled PDMS/MXene/MWCNT structure are fabricated, in which MXene particles are wrapped and bridged by dense MWCNTs, forming complex sliding conductive networks. Therefore, the strain sensor possesses an impressive sensitivity (gauge factor = 646) and 40% response range. Moreover, a fast response time of 280 ms and detection limit of 0.05% are achieved. The high performance enables good prospects in human detection, like human movement and pulse signals for healthcare. It is also applied to wearable smart data gloves, in which the CNN algorithm is utilized to identify 15 gestures, and the final recognition rate is up to 95%. This comprehensive performance strain sensor is designed for a wide array of human body detection applications and wearable intelligent systems. Full article

Estimating human joint angles is a crucial task in motion analysis, gesture recognition, and motion intention prediction. This paper presents a novel model-based approach for generating reliable and accurate human joint angle estimation using a dual-branch network. The proposed network leverages combined features derived from encoded sEMG signals and RGB-D image data. To ensure the accuracy and reliability of the estimation algorithm, the proposed network employs a convolutional autoencoder to generate a high-level compression of sEMG features aimed at motion prediction. Considering the variability in the distribution of sEMG signals, the proposed network introduces a vision-based joint regression network to maintain the stability of combined features. Taking into account latency, occlusion, and shading issues with vision data acquisition, the feature fusion network utilizes high-frequency sEMG features as weights for specific features extracted from image data. The proposed method achieves effective human body joint angle estimation for motion analysis and motion intention prediction by mitigating the effects of non-stationary sEMG signals. Full article

The research of multimodal emotion recognition based on facial expressions, speech, and body gestures is crucial for oncoming intelligent human–computer interfaces. However, it is a very difficult task and has seldom been researched in this combination in the past years. Based on the GEMEP and Polish databases, this contribution focuses on trimodal emotion recognition from facial expressions, speech, and body gestures, including feature extraction, feature fusion, and multimodal classification of the three modalities. In particular, for feature fusion, two novel algorithms including supervised least squares multiset kernel canonical correlation analysis (SLSMKCCA) and sparse supervised least squares multiset kernel canonical correlation analysis (SSLSMKCCA) are presented, respectively, to carry out efficient facial expression, speech, and body gesture feature fusion. Different from the traditional multiset kernel canonical correlation analysis (MKCCA) algorithms, our SLSKMCCA algorithm is a supervised version and is based on the least squares form. The SSLSKMCCA algorithm is implemented by the combination of SLSMKCCA and a sparse item (L1 Norm). Moreover, two effective solving algorithms for SLSMKCCA and SSLSMKCCA are presented in addition, which use the alternated least squares and augmented Lagrangian multiplier methods, respectively. The extensive experimental results on the popular public GEMEP and Polish databases show that the recognition rate of multimodal emotion recognition is superior to bimodal and monomodal emotion recognition on average, and our presented SLSMKCCA and SSLSMKCCA fusion methods both obtain very high recognition rates, especially for the SSLSMKCCA fusion method. Full article

Compared with the traditional gel electrode, the dry electrode is being taken more seriously in bioelectrical recording because of its easy preparation, long-lasting ability, and reusability. However, the commonly used dry AgCl electrodes and silver cloth electrodes are generally hard to record through hair due to their flat contact surface. Claw electrodes can contact skin through hair on the head and body, but the internal claw structure is relatively hard and causes discomfort after being worn for a few hours. Here, we report a conductive Velcro electrode (CVE) with an elastic hook hair structure, which can collect biopotential through body hair. The elastic hooks greatly reduce discomfort after long-time wearing and can even be worn all day. The CVE electrode is fabricated by one-step immersion in conductive silver paste based on the cost-effective commercial Velcro, forming a uniform and durable conductive coating on a cluster of hook microstructures. The electrode shows excellent properties, including low impedance (15.88 kΩ @ 10 Hz), high signal-to-noise ratio (16.0 dB), strong water resistance, and mechanical resistance. After washing in laundry detergent, the impedance of CVE is still 16% lower than the commercial AgCl electrodes. To verify the mechanical strength and recovery capability, we conducted cyclic compression experiments. The results show that the displacement change of the electrode hook hair after 50 compression cycles was still less than 1%. This electrode provides a universal acquisition scheme, including effective acquisition of different parts of the body with or without hair. Finally, the gesture recognition from electromyography (EMG) by the CVE electrode was applied with accuracy above 90%. The CVE proposed in this study has great potential and promise in various human–machine interface (HMI) applications that employ surface biopotential signals on the body or head with hair. Full article

Over the last few decades, a growing number of studies have used wearable technologies, such as inertial and pressure sensors, to investigate various domains of music experience, from performance to education. In this paper, we systematically review this body of literature using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) method. The initial search yielded a total of 359 records. After removing duplicates and screening for content, 23 records were deemed fully eligible for further analysis. Studies were grouped into four categories based on their main objective, namely performance-oriented systems, measuring physiological parameters, gesture recognition, and sensory mapping. The reviewed literature demonstrated the various ways in which wearable systems impact musical contexts, from the design of multi-sensory instruments to systems monitoring key learning parameters. Limitations also emerged, mostly related to the technology’s comfort and usability, and directions for future research in wearables and music are outlined. Full article

In the era of inclusive education, students with attention deficits are integrated into the general classroom. To ensure a seamless transition of students’ focus towards the teacher’s instruction throughout the course and to align with the teaching pace, this paper proposes a continuous recognition algorithm for capturing teachers’ dynamic gesture signals. This algorithm aims to offer instructional attention cues for students with attention deficits. According to the body landmarks of the teacher’s skeleton by using vision and machine learning-based MediaPipe BlazePose, the proposed method uses simple rules to detect the teacher’s hand signals dynamically and provides three kinds of attention cues (Pointing to left, Pointing to right, and Non-pointing) during the class. Experimental results show the average accuracy, sensitivity, specificity, precision, and F₁ score achieved 88.31%, 91.03%, 93.99%, 86.32%, and 88.03%, respectively. By analyzing non-verbal behavior, our method of competent performance can replace verbal reminders from the teacher and be helpful for students with attention deficits in inclusive education. Full article

Hand gesture recognition is pivotal in facilitating human–machine interaction within the Internet of Things. Nevertheless, it encounters challenges, including labeling expenses and robustness. To tackle these issues, we propose a semi-supervised learning framework guided by pseudo-label consistency. This framework utilizes a dual-branch structure with a mean-teacher network. Within this setup, a global and locally guided self-supervised learning encoder acts as a feature extractor in a teacher–student network to efficiently extract features, maximizing data utilization to enhance feature representation. Additionally, we introduce a pseudo-label Consistency-Guided Mean-Teacher model, where simulated noise is incorporated to generate newly unlabeled samples for the teacher model before advancing to the subsequent stage. By enforcing consistency constraints between the outputs of the teacher and student models, we alleviate accuracy degradation resulting from individual differences and interference from other body parts, thereby bolstering the network’s robustness. Ultimately, the teacher model undergoes refinement through exponential moving averages to achieve stable weights. We evaluate our semi-supervised method on two publicly available hand gesture datasets and compare it with several state-of-the-art fully-supervised algorithms. The results demonstrate the robustness of our method, achieving an accuracy rate exceeding 99% across both datasets. Full article