Search Results (40)

This study presents a Digital Twin–Mixed Reality (DT–MR) framework for the immersive and interactive supervision of automated container terminals (ACTs), addressing the fragmented data and limited situational awareness of conventional 2D monitoring systems. The framework employs a middleware-centric architecture that integrates heterogeneous subsystems—covering terminal operation, equipment control, and information management—through standardized industrial communication protocols. It ensures synchronized timestamps and delivers semantically aligned, low-latency data streams to a multi-scale Digital Twin developed in Unity. The twin applies level-of-detail modeling, spatial anchoring, and coordinate alignment (from Industry Foundation Classes (IFCs) to east–north–up (ENU) coordinates and Unity space) for accurate registration with physical assets, while a Microsoft HoloLens 2 device provides an intuitive Mixed Reality interface that combines gaze, gesture, and voice commands with built-in safety interlocks for secure human–machine interaction. Quantitative performance benchmarks—latency ≤100 ms, status refresh ≤1 s, and throughput ≥10,000 events/s—were met through targeted engineering and validated using representative scenarios of quay crane alignment and automated guided vehicle (AGV) rerouting, demonstrating improved anomaly detection, reduced decision latency, and enhanced operational resilience. The proposed DT–MR pipeline establishes a reproducible and extensible foundation for real-time, human-in-the-loop supervision across ports, airports, and other large-scale smart infrastructures. Full article

STEM (Science, Technology, Engineering, and Mathematics) education faces the challenge of incorporating advanced technologies that foster motivation, collaboration, and hands-on learning. This study proposes a portable system capable of transforming ordinary surfaces into interactive learning spaces through gamification and spatial perception. A prototype based on multi-agent architecture was developed on the PANGEA (Platform for automatic coNstruction of orGanizations of intElligent agents) platform, integrating LIDAR (Light Detection and Ranging) sensors for gesture detection, an ultra-short-throw projector for visual interaction and a web platform to manage educational content, organize activities and evaluate student performance. The data from the sensors is processed in real time using ROS (Robot Operating System), generating precise virtual interactions on the projected surface, while the web allows you to configure physical and pedagogical parameters. Preliminary tests show that the system accurately detects gestures, translates them into digital interactions, and maintains low latency in different classroom environments, demonstrating robustness, modularity, and portability. The results suggest that the combination of multi-agent architectures, LIDAR sensors, and gamified platforms offers an effective approach to promote active learning in STEM, facilitate the adoption of advanced technologies in diverse educational settings, and improve student engagement and experience. Full article

Background: Upper limb loss significantly impacts quality of life, and whereas myoelectric prostheses restore some function, conventional surface electromyography (sEMG) systems face challenges like poor signal quality, high cognitive burden, and suboptimal control. Phantom X, a novel implantable electrode-array-based system leveraging machine learning (ML), aims to overcome these limitations. This feasibility study assessed Phantom X’s performance using non-invasive surface EMG electrodes to approximate implantable system behavior. Methods: This single-arm, non-randomized study included 11 participants (9 able-bodied, 2 with transradial amputation) fitted with a 32-electrode cutaneous array around the forearm. EMG signals were processed through an ML algorithm to control a desk-mounted prosthesis. Performance was evaluated via gesture accuracy (GA), modified gesture accuracy (MGA), and classifier gesture accuracy (CGA) across 11 hand gestures in three arm postures. User satisfaction was also assessed among the two participants with transradial amputation. Results: Phantom X achieved an average GA of 89.0% ± 6.8%, MGA of 96.8% ± 2.0%, and CGA of 93.6% ± 4.1%. Gesture accuracy was the highest in the Arm Parallel posture and the lowest in the Arm Perpendicular posture. Thumbs Up had the highest accuracy (100%), while Index Point and Index Tap gestures showed lower performance (70% and 79% GA, respectively). The mean latency between EMG onset and gesture detection was 250.5 ± 145.9 ms, with 91% of gestures executed within 500 ms. The amputee participants reported high satisfaction. Conclusions: This study demonstrates Phantom X’s potential to enhance prosthesis control through multi-electrode EMG sensing and ML-based gesture decoding. The non-invasive evaluation suggests high accuracy and responsiveness, warranting further studies with the implantable system to assess long-term usability and real-world performance. Phantom X may offer a superior alternative to conventional sEMG-based control, potentially reducing cognitive burden and improving functional outcomes for upper limb amputees. Full article

Sign language recognition plays a crucial role in enabling communication for deaf individuals, yet current methods face limitations such as sensitivity to lighting conditions, occlusions, and lack of adaptability in diverse environments. This study presents a wearable multi-channel tactile sensing system based on smart textiles, designed to capture subtle wrist and finger motions for static sign language recognition. The system leverages triboelectric yarns sewn into gloves and sleeves to construct a skin-conformal tactile sensor array, capable of detecting biomechanical interactions through contact and deformation. Unlike vision-based approaches, the proposed sensor platform operates independently of environmental lighting or occlusions, offering reliable performance in diverse conditions. Experimental validation on American Sign Language letter gestures demonstrates that the proposed system achieves high signal clarity after customized filtering, leading to a classification accuracy of 94.66%. Experimental results show effective recognition of complex gestures, highlighting the system’s potential for broader applications in human-computer interaction. Full article

Robust and efficient contactless human–machine interaction is critical for integrated sensor systems in clinical settings, demanding low-power solutions adaptable to edge computing platforms. This paper presents a real-time hand gesture recognition system using a low-power Frequency-Modulated Continuous Wave (FMCW) radar sensor, featuring a novel Multiple Feature Fusion (MFF) framework optimized for deployment on edge devices. The proposed system integrates velocity profiles, angular variations, and spatial-temporal features through a dual-stage processing architecture: an adaptive energy thresholding detector segments gestures, followed by an attention-enhanced neural classifier. Innovations include dynamic clutter suppression and multi-path cancellation optimized for complex clinical environments. Experimental validation demonstrates high performance, achieving 98% detection recall and 93.87% classification accuracy under LOSO cross-validation. On embedded hardware, the system processes at 28 FPS, showing higher robustness against environmental noise and lower computational overhead compared with existing methods. This low-power, edge-based solution is highly suitable for applications like sterile medical control and patient monitoring, advancing contactless interaction in healthcare by addressing efficiency and robustness challenges in radar sensing for edge computing. Full article

Autonomous drone navigation presents challenges for users unfamiliar with manual flight controls, increasing the risk of collisions. This research addresses this issue by developing a multifunctional drone control system that integrates hand gesture recognition, obstacle avoidance, and 3D mapping to improve accessibility and safety. The system utilizes Google’s MediaPipe Hands software library, which employs machine learning to track 21 key landmarks of the user’s hand, enabling gesture-based control of the drone. Each recognized gesture is mapped to a flight command, eliminating the need for a traditional controller. The obstacle avoidance system, utilizing the Flow Deck V2 and Multi-Ranger Deck, detects objects within a safety threshold and autonomously moves the drone by a predefined avoidance distance away to prevent collisions. A mapping system continuously logs the drone’s flight path and detects obstacles, enabling 3D visualization of drone’s trajectory after the drone landing. Also, an AI-Deck streams live video, enabling navigation beyond the user’s direct line of sight. Experimental validation with the Crazyflie drone demonstrates seamless integration of these systems, providing a beginner-friendly experience where users can fly drones safely without prior expertise. This research enhances human–drone interaction, making drone technology more accessible for education, training, and intuitive navigation. Full article

Intrinsic conductive ionic hydrogels, endowed with excellent mechanical properties, hold significant promise for applications in wearable and implantable electronics. However, the complexity of exercise and athletics calls for mechanical tunability, facile processability and high conductivity of wearable sensors, which remains a persistent challenge. In this study, we developed a mechanically tunable and high ionic conductive hydrogel patch to approach multi-gesture or motion monitoring. Through adjustment of the ratio of amino trimethylene phosphonic acid (ATMP) and poly(vinyl alcohol) (PVA), the composite hydrogel attains tunable mechanical strength (varying from 50 kPa to 730 kPa), remarkable stretchability (reaching up to 1900% strain), high conductivity (measuring 15.43 S/m), and strong linear sensitivity (with a gauge factor of 2.34 within 100% strain). Benefitting with the tunable mechanical sensitivity, the composite hydrogel patch can perform subtle movement monitoring, such as epidermal pulses or pronounced muscle vibrations; meanwhile, it can also recognize and detect major motions, such as hand gestures. The mechanically tunable composite hydrogel contributes a versatile sensing platform for health or athletic monitoring, with wide and sensitive adoptability. Full article

Autism Spectrum Disorder (ASD) poses significant challenges in diagnosis due to its diverse symptomatology and the complexity of early detection. Atypical gait and gesture patterns, prominent behavioural markers of ASD, hold immense potential for facilitating early intervention and optimising treatment outcomes. These patterns can be efficiently and non-intrusively captured using modern computational techniques, making them valuable for ASD recognition. Various types of research have been conducted to detect ASD through deep learning, including facial feature analysis, eye gaze analysis, and movement and gesture analysis. In this study, we optimise a dual-stream architecture that combines image classification and skeleton recognition models to analyse video data for body motion analysis. The first stream processes Skepxels—spatial representations derived from skeleton data—using ConvNeXt-Base, a robust image recognition model that efficiently captures aggregated spatial embeddings. The second stream encodes angular features, embedding relative joint angles into the skeleton sequence and extracting spatiotemporal dynamics using Multi-Scale Graph 3D Convolutional Network(MSG3D), a combination of Graph Convolutional Networks (GCNs) and Temporal Convolutional Networks (TCNs). We replace the ViT model from the original architecture with ConvNeXt-Base to evaluate the efficacy of CNN-based models in capturing gesture-related features for ASD detection. Additionally, we experimented with a Stack Transformer in the second stream instead of MSG3D but found it to result in lower performance accuracy, thus highlighting the importance of GCN-based models for motion analysis. The integration of these two streams ensures comprehensive feature extraction, capturing both global and detailed motion patterns. A pairwise Euclidean distance loss is employed during training to enhance the consistency and robustness of feature representations. The results from our experiments demonstrate that the two-stream approach, combining ConvNeXt-Base and MSG3D, offers a promising method for effective autism detection. This approach not only enhances accuracy but also contributes valuable insights into optimising deep learning models for gesture-based recognition. By integrating image classification and skeleton recognition, we can better capture both global and detailed motion patterns, which are crucial for improving early ASD diagnosis and intervention strategies. Full article

Dynamic gesture detection is a key topic in computer vision and deep learning, with applications in human–computer interaction and virtual reality. However, traditional methods struggle with long sequences, complex scenes, and multimodal data, facing issues such as high computational cost and background noise. This study proposes an Attention-Enhanced dual-layer LSTM (Long Short-Term Memory) network combined with Grounding SAM (Grounding Segment Anything Model) for gesture detection. The dual-layer LSTM captures long-term temporal dependencies, while a multi-head attention mechanism improves the extraction of global spatiotemporal features. Grounding SAM, composed of Grounding DINO for object localization and SAM (Segment Anything Model) for image segmentation, is employed during preprocessing to precisely extract gesture regions and remove background noise. This enhances feature quality and reduces interference during training. Experiments show that the proposed method achieves 96.3% accuracy on a self-constructed dataset and 96.1% on the SHREC 2017 dataset, outperforming several baseline methods by an average of 4.6 percentage points. It also demonstrates strong robustness under complex and dynamic conditions. This approach provides a reliable and efficient solution for future dynamic gesture-recognition systems. Full article

As healthcare costs rise due to aging populations and chronic illnesses, optimized care solutions are urgently needed. Gesture recognition and fall detection are critical for intelligent companion robots in healthcare. However, current deep learning models struggle with accuracy and real-time performance in complex backgrounds due to high computational demands. To address this, we propose an improved RT-DETR R18 model tailored for companion robots. This lightweight, efficient design integrates YOLOv9’s ADown module, the RepNCSPELAN4 module, and custom attention-based AdaptiveGateUpsample and AdaptiveGateDownsample modules for enhanced multi-scale feature fusion, reducing weight and complexity while optimizing real-time detection. Experiments show our model achieves a 51.7% reduction in parameters, a 46.7% decrease in GFLOPS, and higher FPS compared to RT-DETR R18, with mAP@0.5, mAP@0.5-0.95, precision, and recall improving to 99.4%, 86.4%, 99.6%, and 99.4%, respectively. Testing in complex indoor environments confirms its high accuracy for gesture recognition and fall detection, reducing manual workload and offering a novel solution for human behavior recognition in intelligent companionship. Full article

Surface electromyography (sEMG) is an important non-invasive method used in muscle function assessment, rehabilitation and human–machine interaction. However, existing commercial devices often lack sufficient channels, making it challenging to simultaneously acquire signals from multiple muscle sites.In this acticle, we design a portable multi-node sEMG acquisition system based on the TCP protocol to overcome the channel limitations of commercial sEMG detection devices. The system employs the STM32L442KCU6 microcontroller as the main control unit, with onboard ADC for analog-to-digital conversion of sEMG signals. Data filtered by analogy filter is transmitted via an ESP8266 WiFi module to the host computer for display and storage. By configuring Bluetooth broadcasting channels, the system can support up to 40 sEMG detection nodes. A gesture recognition algorithm is implemented to identify grasping motions with varying channel configurations. Experimental results demonstrate that with two channels, the Gradient Boosting Decision Tree (GBDT) algorithm achieves a recognition accuracy of 99.4%, effectively detecting grasping motions. Full article

Unlike traditional remote control systems for controlling unmanned aerial vehicles (UAVs) and drones, active research is being carried out in the domain of vision-based hand gesture recognition systems for drone control. However, contrary to static and sensor based hand gesture recognition, recognizing dynamic hand gestures is challenging due to the complex nature of multi-dimensional hand gesture data, present in 2D images. In a real-time application scenario, performance and safety is crucial. Therefore we propose a hybrid lightweight dynamic hand gesture recognition system and a 3D simulator based drone control environment for live simulation. We used transfer learning-based computer vision techniques to detect dynamic hand gestures in real-time. The gestures are recognized, based on which predetermine commands are selected and sent to a drone simulation environment that operates on a different computer via socket connectivity. Without conventional input devices, hand gesture detection integrated with the virtual environment offers a user-friendly and immersive way to control drone motions, improving user interaction. Through a variety of test situations, the efficacy of this technique is illustrated, highlighting its potential uses in remote-control systems, gaming, and training. The system is tested and evaluated in real-time, outperforming state-of-the-art methods. The code utilized in this study are publicly accessible. Further details can be found in the “Data Availability Statement”. Full article

In this work, a conductive composite film composed of multi-walled carbon nanotubes (MWCNTs) and multi-layer Ti₃C₂Tx MXene nanosheets is used to construct a strain sensor on sandpaper Ecoflex substrate. The composite material forms a sophisticated conductive network with exceptional electrical conductivity, resulting in sensors with broad detection ranges and high sensitivities. The findings indicate that the strain sensing range of the Ecoflex/Ti₃C₂Tx/MWCNT strain sensor, when the mass ratio is set to 5:2, extends to 240%, with a gauge factor (GF) of 933 within the strain interval from 180% to 240%. The strain sensor has demonstrated its robustness by enduring more than 33,000 prolonged stretch-and-release cycles at 20% cyclic tensile strain. Moreover, a fast response time of 200 ms and detection limit of 0.05% are achieved. During application, the sensor effectively enables the detection of diverse physiological signals in the human body. More importantly, its application in a data glove that is coupled with machine learning and uses the Support Vector Machine (SVM) model trained on the collected gesture data results in an impressive recognition accuracy of 93.6%. Full article

Over recent years, automated Human Activity Recognition (HAR) has been an area of concern for many researchers due to its widespread application in surveillance systems, healthcare environments, and many more. This has led researchers to develop coherent and robust systems that efficiently perform HAR. Although there have been many efficient systems developed to date, still, there are many issues to be addressed. There are several elements that contribute to the complexity of the task, making it more challenging to detect human activities, i.e., (i) poor lightning conditions; (ii) different viewing angles; (iii) intricate clothing styles; (iv) diverse activities with similar gestures; and (v) limited availability of large datasets. However, through effective feature extraction, we can develop resilient systems for higher accuracies. During feature extraction, we aim to extract unique key body points and full-body features that exhibit distinct attributes for each activity. Our proposed system introduces an innovative approach for the identification of human activity in outdoor and indoor settings by extracting effective spatio-temporal features, along with a Multi-Class Support Vector Machine, which enhances the model’s performance to accurately identify the activity classes. The experimental findings show that our model outperforms others in terms of classification, accuracy, and generalization, indicating its efficient analysis on benchmark datasets. Various performance metrics, including mean recognition accuracy, precision, F1 score, and recall assess the effectiveness of our model. The assessment findings show a remarkable recognition rate of around 88.61%, 87.33, 86.5%, and 81.25% on the BIT-Interaction dataset, UT-Interaction dataset, NTU RGB + D 120 dataset, and PKUMMD dataset, respectively. Full article

This paper presents a collaborative manufacturing cell implemented in a laboratory setting, focusing on developing learning-based interaction abilities to enhance versatility and ease of use. The key components of the system include 3D real-time volumetric monitoring for safety, visual recognition of hand gestures for human-to-robot communication, classification of physical-contact-based interaction primitives during handover operations, and detection of hand–object interactions to anticipate human intentions. Due to the nature and complexity of perception, deep-learning-based techniques were used to enhance robustness and adaptability. The main components are integrated in a system containing multiple functionalities, coordinated through a dedicated state machine. This ensures appropriate actions and reactions based on events, enabling the execution of specific modules to complete a given multi-step task. An ROS-based architecture supports the software infrastructure among sensor interfacing, data processing, and robot and gripper controllers nodes. The result is demonstrated by a functional use case that involves multiple tasks and behaviors, paving the way for the deployment of more advanced collaborative cells in manufacturing contexts. Full article