Assistive Data Glove for Isolated Static Postures Recognition in American Sign Language Using Neural Network
Abstract
:1. Introduction
- Developing an assistive glove based on flex and gyroscope sensors;
- Collecting datasets for numeric, alphabetic, and alphanumeric (i.e., numbers and alphabet) ASL;
- Training NN models;
- Analyzing the impact of activation functions on the performance of neural models;
- Testing the trained models.
2. Literature Review
3. Methodology
4. Materials and Methods
4.1. Hardware Components
4.1.1. Flex Sensor
4.1.2. MPU 6050
4.1.3. Arduino Microcontroller
4.2. Dataset Generation
4.3. Neural Network Architecture
4.4. Scaled Conjugate Gradient Back Propagation Algorithm
5. Results and Discussion
- a.
- Number datasets
- b.
- Alphabets dataset
- c.
- Alphanumeric dataset
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Amin, M.S.; Rizvi, S.T.H.; Hossain, M. A Comparative Review on Applications of Different Sensors for Sign Language Recognition. J. Imaging 2022, 8, 98. [Google Scholar] [CrossRef] [PubMed]
- Amin, M.S.; Rizvi, S.T.H. Sign Gesture Classification and Recognition Using Machine Learning. Cybern. Syst. 2022, 1–15. [Google Scholar] [CrossRef]
- Amin, M.T.; Latif, M.Y.; Jathol, A.A.; Ahmed, N.; Tarar, M.I.N. Alphabetical Gesture Recognition of American Sign Language using E-Voice Smart Glove. In Proceedings of the 2020 IEEE 23rd International Multitopic Conference (INMIC), Bahawalpur, Pakistan, 5–7 November 2020; pp. 1–6. [Google Scholar] [CrossRef]
- Kim, S.; Kim, J.; Ahn, S.; Kim, Y. Finger language recognition based on ensemble artificial neural network learning using armband EMG sensors. Technol. Health Care 2018, 26, 249–258. [Google Scholar] [CrossRef] [PubMed]
- Verdadero, M.S.; Cruz, J.C.D. An Assistive Hand Glove for Hearing and Speech Impaired Persons. In Proceedings of the 2019 IEEE 11th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM), Laoag, Philippines, 29 November–1 December 2019; pp. 1–6. [Google Scholar] [CrossRef]
- Sarker, S.; Hoque, M.M. An Intelligent System for Conversion of Bangla Sign Language into Speech. In Proceedings of the 2018 International Conference on Innovations in Science, Engineering and Technology (ICISET), Chittagong, Bangladesh, 27–28 October 2018; pp. 513–518. [Google Scholar] [CrossRef]
- Ahmed, M.A.; Zaidan, B.B.; Zaidan, A.A.; Salih, M.M.; bin Lakulu, M.M. A Review on Systems-Based Sensory Gloves for Sign Language Recognition State of the Art between 2007 and 2017. Sensors 2018, 18, 2208. [Google Scholar] [CrossRef]
- Al-Qurishi, M.; Khalid, T.; Souissi, R. Deep Learning for Sign Language Recognition: Current Techniques, Benchmarks, and Open Issues. IEEE Access 2021, 9, 126917–126951. [Google Scholar] [CrossRef]
- Rao, G.A.; Syamala, K.; Kishore, P.V.V.; Sastry, A.S.C.S. Deep convolutional neural networks for sign language recognition. In Proceedings of the 2018 Conference on Signal Processing And Communication Engineering Systems (SPACES), Vijayawada, India, 4–5 January 2018; pp. 194–197. [Google Scholar] [CrossRef]
- Dutta, N.; Saha, J.; Sarker, F.; Zaman, H.U. A Novel Design of a Multi-DOF Mobile Robotic Helping Hand for Paralyzed Patients. In Proceedings of the 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Bangalore, India, 19–22 September 2018; pp. 2219–2225. [Google Scholar] [CrossRef]
- Shah, F.; Shah, M.S.; Akram, W.; Manzoor, A.; Mahmoud, R.O.; Abdelminaam, D.S. Sign Language Recognition Using Multiple Kernel Learning: A Case Study of Pakistan Sign Language. IEEE Access 2021, 9, 67548–67558. [Google Scholar] [CrossRef]
- Subedi, B.; Dorji, K.U.; Wangdi, P.; Dorji, T.; Muramatsu, K. Sign Language Translator of Dzongkha Alphabets Using Arduino. In Proceedings of the 2021 Innovations in Power and Advanced Computing Technologies (i-PACT), Kuala Lumpur, Malaysia, 27–29 November 2021; pp. 1–6. [Google Scholar] [CrossRef]
- Kos, A.; Umek, A.; Tomazic, S.; Zhang, Y. Identification and Selection of Sensors Suitable for Integration into Sport Equipment: Smart Golf Club. In Proceedings of the 2016 International Conference on Identification, Information and Knowledge in the Internet of Things (IIKI), Beijing, China, 20–21 October 2016; pp. 128–133. [Google Scholar] [CrossRef]
- Chowdhury, T.H.; Mamun, M.; Islam, A.; Tabassum, S.N.; Karim, T. Verbalink-A Gesture Based Speech Generating Device. In Proceedings of the 2020 11th International Conference on Electrical and Computer Engineering (ICECE), Dhaka, Bangladesh, 17–19 December 2020; pp. 375–378. [Google Scholar] [CrossRef]
- Jani, A.B.; Kotak, N.A.; Roy, A.K. Sensor Based Hand Gesture Recognition System for English Alphabets Used in Sign Language of Deaf-Mute People. In Proceedings of the 2018 IEEE SENSORS, New Delhi, India, 28–31 October 2018; pp. 1–4. [Google Scholar] [CrossRef]
- Faisal, A.A.; Abir, F.F.; Ahmed, M.U. Sensor Dataglove for Real-time Static and Dynamic Hand Gesture Recognition. In Proceedings of the 2021 Joint 10th International Conference on Informatics, Electronics & Vision (ICIEV) and 2021 5th International Conference on Imaging, Vision & Pattern Recognition (icIVPR), Kitakyushu, Japan, 16–20 August 2021; pp. 1–7. [Google Scholar] [CrossRef]
- Silva, B.; Calixto, W.; Furriel, G. Devices analysis and artificial neural network parameters for sign language recognition. In Proceedings of the 2017 CHILEAN Conference on Electrical, Electronics Engineering, Information and Communication Technologies (CHILECON), Pucon, Chile, 18–20 October 2017; pp. 1–5. [Google Scholar] [CrossRef]
- Suri, A.; Singh, S.K.; Sharma, R.; Sharma, P.; Garg, N.; Upadhyaya, R. Development of Sign Language using Flex Sensors. In Proceedings of the 2020 International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 10–12 September 2020; pp. 102–106. [Google Scholar] [CrossRef]
- Mardiyanto, R.; Utomo, M.F.R.; Purwanto, D.; Suryoatmojo, H. Development of hand gesture recognition sensor based on accelerometer and gyroscope for controlling arm of underwater remotely operated robot. In Proceedings of the 2017 International Seminar on Intelligent Technology and Its Applications (ISITIA), Surabaya, Indonesia, 28–29 August 2017; pp. 329–333. [Google Scholar] [CrossRef]
- Nyaga, C.N.; Wario, R.D. Sign Language Gesture Recognition through Computer Vision. In Proceedings of the 2018 IST-Africa Week Conference (IST-Africa), Gaborone, Botswana, 9–18 May 2018; Cunningham, P., Cunningham, M., Eds.; Africa: IIMC International Information Management Corporation: Dublin, Ireland, 2018; p. 8. [Google Scholar]
- Mummadi, C.K.; Leo, F.P.P.; Verma, K.D.; Kasireddy, S.; Scholl, P.M.; Kempfle, J.; Van Laerhoven, K. Real-Time and Embedded Detection of Hand Gestures with an IMU-Based Glove. Informatics 2018, 5, 28. [Google Scholar] [CrossRef]
- Deriche, M.; Aliyu, S.O.; Mohandes, M. An Intelligent Arabic Sign Language Recognition System Using a Pair of LMCs With GMM Based Classification. IEEE Sens. J. 2019, 19, 8067–8078. [Google Scholar] [CrossRef]
- Kudrinko, K.; Flavin, E.; Zhu, X.; Li, Q. Wearable Sensor-Based Sign Language Recognition: A Comprehensive Review. IEEE Rev. Biomed. Eng. 2020, 14, 82–97. [Google Scholar] [CrossRef]
- Huang, S.; Ye, Z. Boundary-Adaptive Encoder With Attention Method for Chinese Sign Language Recognition. IEEE Access 2021, 9, 70948–70960. [Google Scholar] [CrossRef]
- Jiang, S.; Lv, B.; Guo, W.; Zhang, C.; Wang, H.; Sheng, X.; Shull, P.B. Feasibility of Wrist-Worn, Real-Time Hand, and Surface Gesture Recognition via sEMG and IMU Sensing. IEEE Trans. Ind. Inform. 2017, 14, 3376–3385. [Google Scholar] [CrossRef]
- Kamal, S.M.; Chen, Y.; Li, S.; Shi, X.; Zheng, J. Technical Approaches to Chinese Sign Language Processing: A Review. IEEE Access 2019, 7, 96926–96935. [Google Scholar] [CrossRef]
- Lee, B.G.; Lee, S.M. Smart Wearable Hand Device for Sign Language Interpretation System With Sensors Fusion. IEEE Sens. J. 2017, 18, 1224–1232. [Google Scholar] [CrossRef]
- Li, L.; Jiang, S.; Shull, P.B.; Gu, G. SkinGest: Artificial skin for gesture recognition via filmy stretchable strain sensors. Adv. Robot. 2018, 32, 1112–1121. [Google Scholar] [CrossRef]
- Muneer, A.-H.; Ghulam, M.; Wadood, A.; Mansour, A.; Mohammed, B.; Tareq, A.; Amine, M.M. Deep Learning-Based Approach for Sign Language Gesture Recognition With Efficient Hand Gesture Representation. IEEE Access 2020, 8, 192527–192542. [Google Scholar]
- Aly, S.; Aly, W. DeepArSLR: A Novel Signer-Independent Deep Learning Framework for Isolated Arabic Sign Language Gestures Recognition. IEEE Access 2020, 8, 83199–83212. [Google Scholar] [CrossRef]
- Kishore, P.; Kumar, D.A.; Sastry, A.C.; Kumar, E.K. Motionlets Matching With Adaptive Kernels for 3-D Indian Sign Language Recognition. IEEE Sens. J. 2018, 18, 3327–3337. [Google Scholar] [CrossRef]
- Maraqa, M.; Abu-Zaiter, R. Recognition of Arabic Sign Language (ArSL) using recurrent neural networks. In Proceedings of the 2008 First International Conference on the Applications of Digital Information and Web Technologies (ICADIWT), Ostrava, Czech Republic, 4–6 August 2008; pp. 478–481. [Google Scholar] [CrossRef]
- Adithya, V.; Vinod, P.R.; Gopalakrishnan, U. Artificial neural network based method for Indian sign language recognition. In Proceedings of the 2013 IEEE Conference on Information & Communication Technologies, Thuckalay, India, 11–12 April 2013; pp. 1080–1085. [Google Scholar] [CrossRef]
- Akmeliawati, R.; Ooi, M.P.-L.; Kuang, Y.C. Real-Time Malaysian Sign Language Translation using Colour Segmentation and Neural Network. In Proceedings of the 2007 IEEE Instrumentation & Measurement Technology Conference IMTC 2007, Warsaw, Poland, 1–3 May 2007; pp. 1–6. [Google Scholar] [CrossRef]
- Calado, A.; Errico, V.; Saggio, G. Toward the Minimum Number of Wearables to Recognize Signer-Independent Italian Sign Language WITH Machine-Learning Algorithms. IEEE Trans. Instrum. Meas. 2021, 70, 2513809. [Google Scholar] [CrossRef]
- Papastratis, I.; Dimitropoulos, K.; Konstantinidis, D.; Daras, P. Continuous Sign Language Recognition Through Cross-Modal Alignment of Video and Text Embeddings in a Joint-Latent Space. IEEE Access 2020, 8, 91170–91180. [Google Scholar] [CrossRef]
- De Castro, G.Z.; Guerra, R.R.; Guimarães, F.G. Automatic translation of sign language with multi-stream 3D CNN and generation of artificial depth maps. Expert Syst. Appl. 2023, 215, 119394. [Google Scholar] [CrossRef]
- DelPreto, J.; Hughes, J.; D’Aria, M.; de Fazio, M.; Rus, D. A Wearable Smart Glove and Its Application of Pose and Gesture Detection to Sign Language Classification. IEEE Robot. Autom. Lett. 2022, 7, 10589–10596. [Google Scholar] [CrossRef]
- Montemurro, K.; Brentari, D. Emphatic fingerspelling as code-mixing in American Sign Language. Proc. Linguistic Soc. Am. 2018, 3, 61. [Google Scholar] [CrossRef]
Neural Network Specifications | Dataset Type | |||
---|---|---|---|---|
Digit | Alphabet | Alphanumeric | ||
BiLayered | Dataset Split | 80% training, 10% validation, and 10% testing | ||
Training Algorithm | Scaled Conjugate Gradient based Back Propagation | |||
Training accuracy | 97.7% | 95.3% | 96.5% | |
Testing Accuracy | 94.3% | 90.7% | 91.5% | |
Prediction speed | 170,000 obs/s | 260,000 obs/s | 270,000 obs/s | |
Training time | 1.9319 s | 21.79 s | 50.77 s | |
Connected Layers | 2 | |||
Each layer size | 10 | |||
Regularization strength | Lambda | |||
Performance | Cross Entropy Error | |||
Activation Functions | ReLU, Tanh, Sigmoid | |||
Iteration Limit | 1000 | |||
ReLU Accuracy | 98.7% | 97.5% | 95.1% | |
Tanh Accuracy | 95.5% | 94.8% | 93.3% | |
Sigmoid Accuracy | 91.9% | 92.6% | 90.2% | |
TriLayered | Connected Layers | 3 | ||
Each layer size | 10 | |||
Regularization strength | Lambda | |||
Performance | Cross Entropy Error | |||
Activation Functions | ReLU, Tanh, Sigmoid | |||
Iteration Limit | 1000 | |||
ReLU Accuracy | 96.8% | 93.2% | 97.6% | |
Tanh Accuracy | 94.7% | 92.5% | 95.9% | |
Sigmoid Accuracy | 90.4% | 87.9% | 78.5% |
Sr. No | Literature-Based Recognition Models | Accuracy |
---|---|---|
1 | Support Vector Machine (SVM) [1] | 91.93% |
2 | Template-matching approach [5] | 83.58% |
3 | Template-matching approach [6] | 99.5% |
4 | DTW and Nearest Mapping [7] | 96.5% |
5 | LDA, KNN and SVM [10] | 98% |
6 | Template-matching approach [12] | 92% |
7 | Wrist-based gesture recognition system [13] | 92.66% and 88.8% |
8 | Local Fusion algorithm on motion sensor [15] | 91%, 92%, and 93% |
9 | K-Nearest Neighbor (KNN) [17] | 99.53% for static gestures and 98.64% for dynamic gestures |
10 | Multilayer Perceptron [23] | 96.1% |
11 | Template-matching algorithm [26] | 98% |
12 | Convolutional Neural Network [31] | 92.88% |
13 | Recurrent Neural Network [36] | 95% |
14 | Feed-forward Artificial Neural Network [32] | 91.11% |
15 | Color segmentation and Neural Network [33] | 90% |
16 | Multistream 3D CNN [34] | 91% |
17 | Long Short. Term Memory Networks [37] | 92.8% |
18 | Artificial Neural Network [38] | 93.91% |
19 | 3-branch Convolutional Neural Network [35] | 90% |
20 | Bilayered NN (digit dataset) | 98.7% |
21 | Bilayered NN (alphabet dataset) | 97.5% |
22 | Bilayered NN (alphanumeric dataset) | 95.1% |
23 | Trilayered NN (digit dataset) | 96.8% |
24 | Trilayered NN (alphabet dataset) | 93.2% |
25 | Trilayered NN (alphanumeric dataset) | 97.6% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Amin, M.S.; Rizvi, S.T.H.; Mazzei, A.; Anselma, L. Assistive Data Glove for Isolated Static Postures Recognition in American Sign Language Using Neural Network. Electronics 2023, 12, 1904. https://doi.org/10.3390/electronics12081904
Amin MS, Rizvi STH, Mazzei A, Anselma L. Assistive Data Glove for Isolated Static Postures Recognition in American Sign Language Using Neural Network. Electronics. 2023; 12(8):1904. https://doi.org/10.3390/electronics12081904
Chicago/Turabian StyleAmin, Muhammad Saad, Syed Tahir Hussain Rizvi, Alessandro Mazzei, and Luca Anselma. 2023. "Assistive Data Glove for Isolated Static Postures Recognition in American Sign Language Using Neural Network" Electronics 12, no. 8: 1904. https://doi.org/10.3390/electronics12081904
APA StyleAmin, M. S., Rizvi, S. T. H., Mazzei, A., & Anselma, L. (2023). Assistive Data Glove for Isolated Static Postures Recognition in American Sign Language Using Neural Network. Electronics, 12(8), 1904. https://doi.org/10.3390/electronics12081904