Next Article in Journal
Wind Turbine Main Bearing Fault Prognosis Based Solely on SCADA Data
Next Article in Special Issue
Vision and RTLS Safety Implementation in an Experimental Human—Robot Collaboration Scenario
Previous Article in Journal
Electronics for Sensors
Previous Article in Special Issue
Design and Implementation of Universal Cyber-Physical Model for Testing Logistic Control Algorithms of Production Line’s Digital Twin by Using Color Sensor
Article

A Deep Learning Framework for Recognizing Both Static and Dynamic Gestures

1
LIRMM, Université de Montpellier, CNRS, 34392 Montpellier, France
2
Cognitive Robotics Department, Delft University of Technology, 2628 CD Delft, The Netherlands
*
Author to whom correspondence should be addressed.
Academic Editor: Frantisek Duchon
Sensors 2021, 21(6), 2227; https://doi.org/10.3390/s21062227
Received: 12 February 2021 / Revised: 10 March 2021 / Accepted: 17 March 2021 / Published: 23 March 2021
(This article belongs to the Collection Sensors and Data Processing in Robotics)
Intuitive user interfaces are indispensable to interact with the human centric smart environments. In this paper, we propose a unified framework that recognizes both static and dynamic gestures, using simple RGB vision (without depth sensing). This feature makes it suitable for inexpensive human-robot interaction in social or industrial settings. We employ a pose-driven spatial attention strategy, which guides our proposed Static and Dynamic gestures Network—StaDNet. From the image of the human upper body, we estimate his/her depth, along with the region-of-interest around his/her hands. The Convolutional Neural Network (CNN) in StaDNet is fine-tuned on a background-substituted hand gestures dataset. It is utilized to detect 10 static gestures for each hand as well as to obtain the hand image-embeddings. These are subsequently fused with the augmented pose vector and then passed to the stacked Long Short-Term Memory blocks. Thus, human-centred frame-wise information from the augmented pose vector and from the left/right hands image-embeddings are aggregated in time to predict the dynamic gestures of the performing person. In a number of experiments, we show that the proposed approach surpasses the state-of-the-art results on the large-scale Chalearn 2016 dataset. Moreover, we transfer the knowledge learned through the proposed methodology to the Praxis gestures dataset, and the obtained results also outscore the state-of-the-art on this dataset. View Full-Text
Keywords: gestures recognition; operator interfaces; human activity recognition; commercial robots and applications; cyber-physical systems gestures recognition; operator interfaces; human activity recognition; commercial robots and applications; cyber-physical systems
Show Figures

Figure 1

MDPI and ACS Style

Mazhar, O.; Ramdani, S.; Cherubini, A. A Deep Learning Framework for Recognizing Both Static and Dynamic Gestures. Sensors 2021, 21, 2227. https://doi.org/10.3390/s21062227

AMA Style

Mazhar O, Ramdani S, Cherubini A. A Deep Learning Framework for Recognizing Both Static and Dynamic Gestures. Sensors. 2021; 21(6):2227. https://doi.org/10.3390/s21062227

Chicago/Turabian Style

Mazhar, Osama, Sofiane Ramdani, and Andrea Cherubini. 2021. "A Deep Learning Framework for Recognizing Both Static and Dynamic Gestures" Sensors 21, no. 6: 2227. https://doi.org/10.3390/s21062227

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop