Continuous Recognition of Teachers’ Hand Signals for Students with Attention Deficits
Abstract
:1. Introduction
2. Proposed Method
3. Experimental Results
4. Discussion
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Hurwitz, S.; Watson, L.R. Joint attention revisited: Finding strengths among children with autism. Autism 2016, 20, 538–550. [Google Scholar] [CrossRef] [PubMed]
- Lai, Y.H.; Chang, Y.C.; Ma, Y.W.; Huang, S.Y.; Chao, H.C. Improvement of ADHD Behaviors with AI Perception Technology. In Proceedings of the International Cognitive Cities Conference, Kyoto, Japan, 3–6 September 2019. [Google Scholar]
- Al Hazmi, A.N.; Ahmad, A.C. Universal design for learning to support access to the general education curriculum for students with intellectual disabilities. World J. Educ. 2018, 8, 66–72. [Google Scholar] [CrossRef]
- Lidstone, D.E.; Mostofsky, S.H. Moving toward understanding autism: Visual-motor integration, imitation, and social skill development. Pediatr. Neurol. 2021, 122, 98–105. [Google Scholar] [CrossRef] [PubMed]
- Shkedy, G.; Shkedy, D.; Sandoval-Norton, A.H.; Fantaroni, G.; Castro, J.M.; Sahagun, N.; Christopher, D. Visual communication analysis (VCA): Implementing self-determination theory and research-based practices in special education classrooms. Cogent Psychol. 2021, 8, 1875549. [Google Scholar] [CrossRef]
- Baragash, R.S.; Al-Samarraie, H.; Alzahrani, A.I.; Alfarraj, O. Augmented reality in special education: A meta-analysis of single-subject design studies. Eur. J. Spec. Needs Educ. 2020, 35, 382–397. [Google Scholar] [CrossRef]
- Garzón, J.; Pavón, J.; Baldiris, S. Systematic review and meta-analysis of augmented reality in educational settings. Virtual Real. 2019, 23, 447–459. [Google Scholar] [CrossRef]
- Zhong, D.; Chen, L.; Feng, Y.; Song, R.; Huang, L.; Liu, J.; Zhang, L. Effects of virtual reality cognitive training in individuals with mild cognitive impairment: A systematic review and meta-analysis. Int. J. Geriatr. Psychiatry 2021, 36, 1829–1847. [Google Scholar] [CrossRef] [PubMed]
- Sam, C.; Naicker, N.; Rajkoomar, M. Meta-analysis of artificial intelligence works in ubiquitous learning environments and technologies. Int. J. Adv. Comput. Sci. Appl. 2020, 11, 603–613. [Google Scholar] [CrossRef]
- Cao, Z.; Simon, T.; Wei, S.E.; Sheikh, Y. Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Cao, Z.; Hidalgo, G.; Simon, T.; Wei, S.E.; Sheikh, Y. OpenPose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 43, 172–186. [Google Scholar] [CrossRef]
- Liu, T.; Chen, Z.; Wang, X. Automatic Instructional Pointing Gesture Recognition by Machine Learning in the Intelligent Learning Environment. In Proceedings of the 2019 4th International Conference on Distance Education and Learning, Shanghai, China, 24–27 May 2019. [Google Scholar]
- Wang, J.; Liu, T.; Wang, X. Human hand gesture recognition with convolutional neural networks for K-12 double-teachers instruction mode classroom. Infrared Phys. Tech. 2020, 111, 103464. [Google Scholar] [CrossRef]
- Hernández Correa, J.; Farsani, D.; Araya, R. An Application of Machine Learning and Image Processing to Automatically Detect Teachers’ Gestures. In Proceedings of the International Conference on Computational Collective Intelligence, Da Nang, Vietnam, 30 November–3 December 2020. [Google Scholar]
- Gu, Y.; Hu, J.; Zhou, Y.; Lu, L. Online Teaching Gestures Recognition Model Based on Deep Learning. In Proceedings of the 2020 International Conference on Networking and Network Applications, Haikou, China, 10–13 December 2020. [Google Scholar]
- Araya, R.; Sossa-Rivera, J. Automatic detection of gaze and body orientation in elementary school classrooms. Front. Robot. AI 2021, 8, 729832. [Google Scholar] [CrossRef] [PubMed]
- Yoon, H.Y.; Kang, S.; Kim, S. A non-verbal teaching behaviour analysis for improving pointing out gestures: The case of asynchronous video lecture analysis using deep learning. J. Comput. Assist. Learn. 2024, 40, 1006–1018. [Google Scholar] [CrossRef]
- Liu, H.; Yao, C.; Zhang, Y.; Ban, X. GestureTeach: A gesture guided online teaching interactive model. Comput. Animat. Virtual Worlds 2024, 35, e2218. [Google Scholar] [CrossRef]
- Chen, Z.; Feng, X.; Liu, T.; Wang, C.; Zhang, C. A Computer-Assisted Teaching System with Gesture Recognition Technology and Its Applications. In Proceedings of the International Conference on Digital Technology in Education, Taipei, Taiwan, 6–8 August 2017. [Google Scholar]
- Chiang, H.H.; Chen, W.M.; Chao, H.C.; Tsai, D.L. A virtual tutor movement learning system in eLearning. Multimed. Tools Appl. 2019, 78, 4835–4850. [Google Scholar] [CrossRef]
- Goto, T.; Sakurai, D.; Ooi, S. Proposal of Feedback System Based on Skeletal Analysis in Physical Education Classes. In Proceedings of the 4th International Conference on Education and Multimedia Technology, Kyoto, Japan, 19–22 July 2020. [Google Scholar]
- Amrutha, K.; Prabu, P.; Paulose, J. Human Body Pose Estimation and Applications. In Proceedings of the 2021 Innovations in Power and Advanced Computing Technologies, Kuala Lumpur, Malaysia, 27–29 November 2021. [Google Scholar]
- Farsani, D.; Lange, T.; Meaney, T. Gestures, systemic functional linguistics and mathematics education. Mind Cult. Act. 2022, 29, 75–95. [Google Scholar] [CrossRef]
- Kure, A.E.; Brevik, L.M.; Blikstad-Balas, M. Digital skills critical for education: Video analysis of students' technology use in Norwegian secondary English classrooms. J. Comput. Assist. Learn. 2023, 39, 269–285. [Google Scholar] [CrossRef]
- Kim, Y.; Soyata, T.; Behnagh, R.F. Towards emotionally aware AI smart classroom: Current issues and directions for engineering and education. IEEE Access 2018, 6, 5308–5331. [Google Scholar] [CrossRef]
- Swain, D.; Satapathy, S.; Acharya, B.; Shukla, M.; Gerogiannis, V.C.; Kanavos, A.; Giakovis, D. Deep learning models for yoga pose monitoring. Algorithms 2022, 15, 403. [Google Scholar] [CrossRef]
- Connie, T.; Aderinola, T.B.; Ong, T.S.; Goh, M.K.O.; Erfianto, B.; Purnama, B. Pose-based gait analysis for diagnosis of Parkinson’s disease. Algorithms 2022, 15, 474. [Google Scholar] [CrossRef]
- Gesnouin, J.; Pechberti, S.; Bresson, G.; Stanciulescu, B.; Moutarde, F. Predicting intentions of pedestrians from 2D skeletal pose sequences with a representation-focused multi-branch deep learning network. Algorithms 2020, 13, 331. [Google Scholar] [CrossRef]
- Martinez, J.; Hossain, R.; Romero, J.; Little, J.J. A Simple Yet Effective Baseline for 3D Human Pose Estimation. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017. [Google Scholar]
- Luvizon, D.C.; Picard, D.; Tabia, H. 2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018. [Google Scholar]
- Rogez, G.; Weinzaepfel, P.; Schmid, C. LCR-Net++: Multi-person 2D and 3D pose detection in natural images. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 42, 1146–1161. [Google Scholar] [CrossRef] [PubMed]
- Li, W.; Liu, H.; Ding, R.; Liu, M.; Wang, P.; Yang, W. Exploiting temporal contexts with strided transformer for 3D human pose estimation. IEEE Trans. Multimed. 2023, 25, 1282–1293. [Google Scholar] [CrossRef]
- Zheng, C.; Wu, W.; Chen, C.; Yang, T.; Zhu, S.; Shen, J.; Kehtarnavaz, N.; Shah, M. Deep Learning-based human pose estimation: A survey. ACM Comput. Surv. 2023, 56, 11. [Google Scholar] [CrossRef]
- Bazarevsky, V.; Grishchenko, I.; Raveendran, K.; Zhu, T.; Zhang, F.; Grundmann, M. BlazePose: On-Device Real-Time Body Pose Tracking. In Proceedings of the CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, USA, 15 June 2020. [Google Scholar]
- MediaPipe. Available online: https://developers.google.com/mediapipe/solutions/vision/pose_landmarker/ (accessed on 2 April 2024).
- Zhang, F.; Bazarevsky, V.; Vakunov, A.; Tkachenka, A.; Sung, G.; Chang, C.L.; Grundmann, M. MediaPipe Hands: On-device Real-Time Hand Tracking. In Proceedings of the CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, USA, 15 June 2020. [Google Scholar]
- Lugaresi, C.; Tang, J.; Nash, H.; McClanahan, C.; Uboweja, E.; Hays, M.; Zhang, F.; Chang, C.L.; Yong, M.; Lee, J.; et al. MediaPipe: A Framework for Perceiving and Processing Reality. In Proceedings of the CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Long Beach, CA, USA, 17 June 2019. [Google Scholar]
- Grandini, M.; Bagli, E.; Visani, G. Metrics for Multi-Class Classification: An Overview. arXiv 2020, arXiv:2008.05756. [Google Scholar]
- Yu, M.; Kim, N.; Jung, Y.; Lee, S. A frame detection method for real-time hand gesture recognition systems using CW-radar. Sensors 2020, 20, 2321. [Google Scholar] [CrossRef] [PubMed]
- Choi, J.W.; Ryu, S.J.; Kim, J.H. Short-range radar based real-time hand gesture recognition using LSTM encoder. IEEE Access 2019, 7, 33610–33618. [Google Scholar] [CrossRef]
- Ryumin, D.; Ivanko, D.; Ryumina, E. Audio-visual speech and gesture recognition by sensors of mobile devices. Sensors 2023, 23, 2284. [Google Scholar] [CrossRef] [PubMed]
- Sidiya, K.; Alzanbagi, N.; Bensenouci, A. Google Glass and Apple Watch Will They Become Our Learning Tools? In Proceedings of the 12th Learning and Technology Conference, Jeddah, Saudi Arabia, 12–13 April 2015.
- Lai, M.C.; Chiang, M.S.; Shih, C.T.; Shih, C.H. Applying a vibration reminder to ameliorate the hyperactive behavior of students with attention deficit hyperactivity disorder in class. J. Dev. Phys. Disabil. 2018, 30, 835–844. [Google Scholar] [CrossRef]
- Zarraonandia, T.; Díaz, P.; Montero, Á.; Aedo, I.; Onorati, T. Using a google glass-based classroom feedback system to improve students to teacher communication. IEEE Access 2019, 7, 16837–16846. [Google Scholar] [CrossRef]
- Ayearst, L.E.; Brancaccio, R.; Weiss, M.D. An open-label study of a wearable device targeting ADHD, executive function, and academic performance. Brain Sci. 2023, 13, 1728. [Google Scholar] [CrossRef]
- Whitmore, N.; Chan, S.; Zhang, J.; Chwalek, P.; Chin, S.; Maes, P. Improving Attention Using Wearables via Haptic and Multimodal Rhythmic Stimuli. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 11–16 May 2024. [Google Scholar]
Number | Name | Number | Name | Number | Name |
---|---|---|---|---|---|
0 | Nose | 11 | Left shoulder | 22 | Right thumb |
1 | Left eye inner | 12 | Right shoulder | 23 | Left hip |
2 | Left eye | 13 | Left elbow | 24 | Right hip |
3 | Left eye outer | 14 | Right elbow | 25 | Left knee |
4 | Right eye inner | 15 | Left wrist | 26 | Right knee |
5 | Right eye | 16 | Right wrist | 27 | Left ankle |
6 | Right eye outer | 17 | Left pinky | 28 | Right ankle |
7 | Left ear | 18 | Right pinky | 29 | Left heel |
8 | Right ear | 19 | Left index | 30 | Right heel |
9 | Mouth left | 20 | Right index | 31 | Left foot index |
10 | Mouth right | 21 | Left thumb | 32 | Right foot index |
Video | Recognized Signal | Accuracy | Sensitivity | Specificity | Precision | F1 Score |
---|---|---|---|---|---|---|
1 | Pointing toleft | 90.16% | 100.00% | 98.10% | 89.47% | 94.44% |
Pointing toright | 97.67% | 88.61% | 82.35% | 89.36% | ||
Non-pointing | 82.26% | 98.33% | 98.08% | 89.48% | ||
Macro Average | - | 93.31% | 95.01% | 89.97% | 91.09% | |
2 | Pointing toleft | 91.30% | 100.00% | 97.64% | 78.57% | 88.00% |
Pointing toright | 100.00% | 89.29% | 85.71% | 92.31% | ||
Non-pointing | 83.56% | 100.00% | 100.00% | 91.04% | ||
Macro Average | - | 94.52% | 95.64% | 88.09% | 90.45% | |
3 | Pointing toleft | 83.66% | 100.00% | 93.79% | 69.44% | 81.96% |
Pointing toright | 89.19% | 89.06% | 82.50% | 85.71% | ||
Non-pointing | 75.73% | 91.92% | 90.70% | 82.54% | ||
Macro Average | - | 88.31% | 91.59% | 80.88% | 83.40% | |
4 | Pointing toleft | 90.18% | 92.93% | 96.00% | 94.85% | 93.88% |
Pointing toright | 95.83% | 95.50% | 71.88% | 82.14% | ||
Non-pointing | 86.14% | 93.50% | 91.58% | 88.78% | ||
Macro-Average | - | 91.63% | 95.00% | 86.10% | 88.27% | |
5 | Pointing toleft | 86.24% | 85.11% | 93.55% | 90.91% | 87.91% |
Pointing toright | 92.68% | 97.18% | 88.37% | 90.47% | ||
Non-pointing | 84.34% | 87.41% | 80.46% | 82.35% | ||
Macro Average | - | 87.38% | 92.71% | 86.58% | 86.91% | |
Average | Pointing toleft | 88.31% | 95.61% | 95.82% | 84.65% | 89.24% |
Pointing toright | 95.07% | 91.93% | 82.16% | 88.00% | ||
Non-pointing | 82.41% | 94.23% | 92.16% | 86.84% | ||
Macro-Average | - | 91.03% | 93.99% | 86.32% | 88.03% |
Method | Classification Task | Classes | Keypoint Extraction | Classification | Accuracy |
---|---|---|---|---|---|
[12] | Pointing or not | 2 | OpenPose | Non-linear neural network | 90% |
[13] | Pointing or not | 2 | Convolutional neural network | Convolutional neural network | over 90% |
[14] | Gesticulating or not | 2 | OpenPose | Machine learning | 54~78% |
Proposed method | Pointing to left Pointing to right Non-pointing | 3 | MediaPipe | Simple rules | 88.31% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, I.D.S.; Yang, C.-M.; Wu, S.-S.; Yang, C.-K.; Chen, M.-J.; Yeh, C.-H.; Lin, Y.-H. Continuous Recognition of Teachers’ Hand Signals for Students with Attention Deficits. Algorithms 2024, 17, 300. https://doi.org/10.3390/a17070300
Chen IDS, Yang C-M, Wu S-S, Yang C-K, Chen M-J, Yeh C-H, Lin Y-H. Continuous Recognition of Teachers’ Hand Signals for Students with Attention Deficits. Algorithms. 2024; 17(7):300. https://doi.org/10.3390/a17070300
Chicago/Turabian StyleChen, Ivane Delos Santos, Chieh-Ming Yang, Shang-Shu Wu, Chih-Kang Yang, Mei-Juan Chen, Chia-Hung Yeh, and Yuan-Hong Lin. 2024. "Continuous Recognition of Teachers’ Hand Signals for Students with Attention Deficits" Algorithms 17, no. 7: 300. https://doi.org/10.3390/a17070300
APA StyleChen, I. D. S., Yang, C. -M., Wu, S. -S., Yang, C. -K., Chen, M. -J., Yeh, C. -H., & Lin, Y. -H. (2024). Continuous Recognition of Teachers’ Hand Signals for Students with Attention Deficits. Algorithms, 17(7), 300. https://doi.org/10.3390/a17070300