Research on Contactless Bio-Signal Measurement Technology for Improving Social Awareness of Individuals with Communication Challenges
Abstract
:1. Introduction
2. Integrated Interface Implementation
3. Measurement of State Data Based on Non-Contact Image Analysis
3.1. Optical Blood Flow (Heart Rate) Signal Acquisition
3.2. Respiration Signal Acquisition
3.3. Face Feature Point Detection and Facial Expression Recognition Implementation
3.4. Contactless Interaction Implementation
3.4.1. Gaze and Facial Movement Tracking Interaction
3.4.2. Hand Movement Tracking Interaction
4. Experiment
- Left area: 10 < x <60, 250 < y < 350
- Center area: 295 < x < 345, 250 < y < 350
- Right area: 580 < x < 630, 250 < y < 350
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Myles, B.S.; Simpson, R.L. Asperger Syndrome: A Guide for Educators and Parents, 2nd ed.; Shoal Creek Blvd: Austin, TX, USA, 2003.
- Schopler, E.; Mesibov, G.B. High-Functioning Individuals with Autism; Springer: Boston, MA, USA, 1992. [Google Scholar] [CrossRef]
- Diehl, J.J.; Bennetto, L.; Watson, D.; Gunlogson, C.; McDonough, J. Resolving ambiguity: A psycholinguistic approach to understanding prosody processing in high-functioning autism. J. Brain Lang. 2008, 106, 144–152. [Google Scholar] [CrossRef] [Green Version]
- Diehl, J.J.; Paul, R. Acoustic differences in the imitation of prosodic patterns in children with autism spectrum disorders. J. Res. Autism Spectr. Disord. 2012, 6, 123–134. [Google Scholar] [CrossRef] [Green Version]
- Grossman, R.B.; Edelson, L.R.; Tager-Flusberg, H. Emotional facial and vocal expressions during story retelling by children and adolescents with high-functioning autism. J. Speech Lang. Hear. Res. 2013, 56, 1035–1044. [Google Scholar] [CrossRef] [Green Version]
- Stagg, S.D.; Slavny, R.; Hand, C.; Cardoso, A.; Smith, P. Does facial expressivity count? how typically developing children respond initially to children with autism. Autism 2013. [Google Scholar] [CrossRef] [PubMed]
- Brewer, R.; Biotti, F.; Catmur, C.; Press, C.; Happ’e, F.; Cook, R.; Bird, G. Can neurotypical individuals read autistic facial expressions? Atypical production of emotional facial expressions in autism spectrum disorders. Autism Res. 2016, 9, 262–271. [Google Scholar] [CrossRef] [Green Version]
- Eldevik, S.; Hastings, R.P.; Hughes, J.C.; Jahr, E.; Eikeseth, S.; Cross, S. Meta-analysis of Early Intensive Behavioral Intervention for children with autism. J. Clin. Child Adolesc. Psychol. 2009, 38, 439–450. [Google Scholar] [CrossRef] [PubMed]
- Lange, C.G.; James, W. A Series of Reprints and Translations. The Emotions; Williams & Wilkins Co.: Philadelphia, PA, USA, 1922; Volume 1. [Google Scholar] [CrossRef]
- Cannon, W.B. The James-Lange Theory of Emotions: A Critical Examination and an Alternative Theory. Am. J. Psychol. 1927, 39, 106. [Google Scholar] [CrossRef]
- Blair, R.J.R.; Coles, M. Expression recognition and behavioral problems in early adolescence. Cogn. Dev. 2000, 15, 421–434. [Google Scholar] [CrossRef]
- Chung, S.Y.; Yoon, H.J. A Framework for Treatment of Autism Using Affective Computing. In Proceedings of the 18th Medicine Meets Virtual Reality (MMVR), Newport Beach, CA, USA, 8–12 February 2011. [Google Scholar]
- Billeci, L.; Sicca, F.; Maharatna, K.; Apicella, F.; Narzisi, A.; Campatelli, G.; Calderoni, S.; Pioggia, G.; Muratori, F. On the Application of Quantitative EEG for Characterizing Autistic Brain: A Systematic Review. Front. Hum. Neurosci. 2013, 7, 442. [Google Scholar] [CrossRef] [Green Version]
- Marco, E.J.; Hinkley, L.B.N.; Hill, S.S.; Nagarajan, S.S.; Hinkley, L.B.N. Sensory processing in autism: A review of neurophysiologic findings. Pediatr. Res. 2011, 69, R48–R54. [Google Scholar] [CrossRef] [PubMed]
- Wang, Y.; Hensley, M.K.; Tasman, A.; Sears, L.; Casanova, M.F.; Sokhadze, E.M. Heart Rate Variability and Skin Conductance During Repetitive TMS Course in Children with Autism. Psychophysiol. Biofeedback 2016, 41, 47–60. [Google Scholar] [CrossRef]
- Cabibihan, J.-J.; Javed, H.; Aldosari, M.; Frazier, T.; Elbashir, H. Sensing Technologies for Autism Spectrum Disorder Screening and Intervention. Sensors 2016, 17, 46. [Google Scholar] [CrossRef] [Green Version]
- Jang, E.-H.; Park, B.-J.; Park, M.-S.; Kim, S.-H.; Sohn, J.-H. Analysis of physiological signals for recognition of boredom, pain, and surprise emotions. J. Physiol. Anthropol. 2015, 34, 25. [Google Scholar] [CrossRef] [Green Version]
- Yu, S.-N.; Chen, S.-F. Emotion state identification based on heart rate variability and genetic algorithm. In Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milano, Italy, 25–29 August 2015; pp. 538–541. [Google Scholar]
- Nakano, T.; Tanaka, K.; Endo, Y.; Yamane, Y.; Yamamoto, T.; Nakano, Y.; Ohta, H.; Kato, N.; Kitazawa, S. Atypical gaze patterns in children and adults with autism spectrum disorders dissociated from developmental changes in gaze behaviour. Proc. Biol. Sci. R. Soc. B Biol. Sci. 2010, 277, 2935–2943. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Bird, G.; Catmur, C.; Silani, G.; Frith, C.; Frith, U. Attention does not modulate neural responses to social stimuli in autism spectrum disorders. Neuroimage 2006, 31, 1614–1624. [Google Scholar] [CrossRef] [PubMed]
- Luo, Y.; Cheong, L.F.; Cabibihan, J.J. Modeling the Temporality of Saliency. In Computer Vision—Proceedings of the ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, 1–5 November 2014; Cremers, D., Reid, I., Saito, H., Yang, M.H., Eds.; Revised Selected Papers, Part III; Springer International Publishing: Cham, Switzerland, 2015; pp. 205–220. [Google Scholar]
- Boraston, Z.; Blakemore, S.J. The application of eye-tracking technology in the study of autism. J. Physiol. 2007, 581, 893–898. [Google Scholar] [CrossRef] [PubMed]
- Jolliffe, T.; Baron-Cohen, S. A test of central coherence theory: Can adults with high-functioning autism or Asperger syndrome integrate objects in context? Vis. Cogn. 2001, 8, 67–101. [Google Scholar] [CrossRef]
- Pelphrey, K.A.; Sasson, N.J.; Reznick, J.S.; Paul, G.; Goldman, B.D.; Piven, J. Visual scanning of faces in autism. J. Autism Dev. Disord. 2002, 32, 249–261. [Google Scholar] [CrossRef]
- Klin, A.; Jones, W.; Schultz, R.; Volkmar, F.; Cohen, D. Defining and quantifying the social phenotype in autism. Am. J. Psychiatry 2002, 159, 895–908. [Google Scholar] [CrossRef]
- Joseph, R.M.; Tanaka, J. Holistic and part-based face recognition in children with autism. J. Child Psychol. Psychiatry 2003, 44, 529–542. [Google Scholar] [CrossRef]
- Spezio, M.L.; Adolphs, R.; Hurley, R.S.E.; Piven, J. Abnormal use of facial information in high-functioning autism. J. Autism Dev. Disord. 2007, 37, 929–939. [Google Scholar] [CrossRef] [PubMed]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. SSD: Single Shot MultiBox Detector. In European Conference on Computer Vision, Proceedings of the ECCV 2016, Amsterdam, The Netherlands, 11–14 October 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 21–37. [Google Scholar] [CrossRef] [Green Version]
- Viola, P.; Jones, M. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), Kauai, HI, USA, 8–14 December 2001. [Google Scholar] [CrossRef]
- Brox, T.; Bruhn, A.; Papenberg, N.; Weickert, J. High Accuracy Optical Flow Estimation Based on a Theory for Warping. In European Conference on Computer Vision, Proceedings of the ECCV 2004, Prague, Czech Republic, 11–14 May 2004; Springer: Berlin/Heidelberg, Germany, 2004; pp. 25–36. [Google Scholar] [CrossRef]
- Hartigan, J.A.; Wong, M.A. Algorithm AS 136: A K-Means Clustering Algorithm. Appl. Stat. 1979, 28, 100. [Google Scholar] [CrossRef]
- Ester, M.; Kriegel, H.-P.; Sander, J.; Xu, X. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, Portland, OR, USA, 2–4 August 1996; Simoudis, E., Han, J., Fayyad, U.M., Eds.; AAAI Press: Palo Alto, CA, USA, 1996; pp. 226–231, ISBN 1-57735-004-9. [Google Scholar]
- Zadeh, A.; Baltrusaitis, T.; Morency, L.-P. Convolutional Experts Constrained Local Model for Facial Landmark Detection. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 22–29 October 2017; pp. 2051–2059. [Google Scholar] [CrossRef] [Green Version]
- Dalal, N.; Triggs, B. Histograms of oriented gradients for human detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, 20–25 June 2005; pp. 886–893. [Google Scholar] [CrossRef] [Green Version]
- Wrnch. What is Human-Centric Computer Vision? Available online: https://wrnch.ai/technology/ (accessed on 4 April 2021).
- Mavadati, S.M.; Mahoor, M.H.; Bartlett, K.; Trinh, P.; Cohn, J.F. DISFA: A Spontaneous Facial Action Intensity Database. IEEE Trans. Affect. Comput. 2013, 4, 151–160. [Google Scholar] [CrossRef]
Classification | Category | Comparative Group | Control Group | Total |
---|---|---|---|---|
Gender | Male | 3 | 8 | 11 |
Female | 11 | 11 | ||
Age | 12~15 | 1 | 2 | 3 |
16~19 | 5 | 4 | 9 | |
20~24 | 6 | 1 | 7 | |
25~29 | 2 | 1 | 3 | |
Education | Junior Highschool | 2 | 2 | 4 |
Attending Highschool | 4 | 2 | 6 | |
Graduated Highschool | 1 | 2 | 3 | |
Attending University | 5 | 2 | 7 | |
Graduated University | 2 | 2 | ||
Disability | ASD | 2 | 2 | |
ID | 6 | 6 |
Num | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | Mean Err | RMSE | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | contact | 90 | 81 | 79 | 81 | 77 | 81 | 82 | 85 | 91 | 90 | 79 | 73 | 82 | 78 | 88 | 83 | 80 | 78 | 75 | 76 | 1.673 | |
ours | 88 | 81 | 78 | 80 | 80 | 82 | 80 | 85 | 90 | 90 | 81 | 73 | 78 | 80 | 87 | 82 | 81 | 78 | 77 | 74 | |||
err | 2 | 0 | 1 | 1 | 3 | 1 | 2 | 0 | 1 | 0 | 2 | 0 | 4 | 2 | 1 | 1 | 1 | 0 | 2 | 2 | 1.3 | ||
2 | contact | 78 | 77 | 88 | 78 | 82 | 86 | 85 | 86 | 87 | 88 | 83 | 85 | 88 | 85 | 95 | 89 | 82 | 80 | 84 | 83 | 1.774 | |
ours | 77 | 77 | 86 | 76 | 81 | 87 | 85 | 86 | 85 | 89 | 80 | 85 | 86 | 85 | 90 | 89 | 84 | 78 | 83 | 83 | |||
err | 1 | 0 | 0 | 2 | 1 | 1 | 0 | 0 | 2 | 1 | 3 | 0 | 2 | 0 | 5 | 0 | 2 | 2 | 1 | 0 | 1.15 | ||
3 | contact | 83 | 82 | 86 | 70 | 88 | 81 | 81 | 83 | 82 | 87 | 85 | 86 | 82 | 86 | 85 | 84 | 85 | 89 | 78 | 85 | 1.244 | |
ours | 84 | 81 | 84 | 71 | 87 | 81 | 82 | 82 | 83 | 86 | 84 | 85 | 83 | 85 | 85 | 86 | 86 | 89 | 77 | 82 | |||
err | 1 | 1 | 2 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 2 | 1 | 0 | 1 | 3 | 1.05 | ||
4 | contact | 84 | 87 | 86 | 86 | 89 | 86 | 86 | 79 | 84 | 77 | 79 | 99 | 78 | 80 | 82 | 79 | 82 | 78 | 86 | 80 | 2.626 | |
ours | 84 | 87 | 86 | 87 | 87 | 86 | 84 | 74 | 84 | 76 | 82 | 90 | 77 | 81 | 80 | 78 | 83 | 77 | 86 | 78 | |||
err | 0 | 0 | 0 | 1 | 2 | 0 | 2 | 5 | 0 | 1 | 3 | 9 | 1 | 1 | 2 | 1 | 1 | 1 | 0 | 2 | 1.6 | ||
5 | contact | 84 | 88 | 79 | 87 | 90 | 85 | 93 | 86 | 86 | 80 | 84 | 81 | 86 | 82 | 86 | 87 | 86 | 91 | 87 | 88 | 1.466 | |
ours | 84 | 86 | 78 | 86 | 87 | 85 | 90 | 85 | 84 | 81 | 82 | 82 | 85 | 83 | 85 | 86 | 85 | 90 | 86 | 87 | |||
err | 0 | 2 | 1 | 1 | 3 | 0 | 3 | 1 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1.25 | ||
mean | 1.27 | 1.756 |
Num | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | Mean Err | RMSE | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | contact | 19 | 19 | 20 | 18 | 18 | 17 | 15 | 16 | 16 | 15 | 16 | 16 | 17 | 15 | 15 | 21 | 21 | 17 | 16 | 17 | 0.591 | |
ours | 19 | 20 | 20 | 18 | 19 | 17 | 16 | 16 | 16 | 15 | 16 | 17 | 17 | 15 | 15 | 22 | 21 | 16 | 16 | 18 | |||
err | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0.35 | ||
2 | contact | 14 | 15 | 11 | 14 | 11 | 14 | 13 | 12 | 13 | 15 | 22 | 21 | 22 | 15 | 15 | 14 | 13 | 12 | 14 | 15 | 0.447 | |
ours | 14 | 15 | 11 | 14 | 11 | 14 | 13 | 12 | 13 | 15 | 21 | 22 | 21 | 15 | 15 | 14 | 13 | 12 | 14 | 14 | |||
err | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0.2 | ||
3 | contact | 13 | 15 | 14 | 13 | 15 | 13 | 14 | 11 | 15 | 14 | 14 | 14 | 14 | 23 | 26 | 23 | 20 | 20 | 20 | 18 | 0.387 | |
ours | 14 | 15 | 14 | 13 | 15 | 13 | 14 | 11 | 15 | 14 | 14 | 14 | 14 | 23 | 27 | 23 | 20 | 20 | 21 | 18 | |||
err | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0.15 | ||
4 | contact | 17 | 14 | 17 | 16 | 13 | 15 | 13 | 13 | 13 | 11 | 16 | 14 | 14 | 14 | 12 | 15 | 13 | 13 | 16 | 16 | 1.140 | |
ours | 17 | 19 | 17 | 16 | 13 | 15 | 13 | 13 | 13 | 11 | 16 | 14 | 14 | 15 | 12 | 15 | 13 | 13 | 16 | 16 | |||
err | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0.3 | ||
5 | contact | 15 | 15 | 14 | 15 | 14 | 13 | 25 | 20 | 19 | 18 | 15 | 14 | 14 | 13 | 13 | 13 | 13 | 18 | 13 | 13 | 1.396 | |
ours | 15 | 15 | 14 | 16 | 14 | 13 | 25 | 20 | 19 | 18 | 14 | 14 | 14 | 13 | 12 | 13 | 13 | 12 | 13 | 13 | |||
err | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 6 | 0 | 0 | 0.45 | ||
mean | 0.29 | 0.792 |
Status | Ground-Truth | Number of Trials | T | F | Ratio | |
---|---|---|---|---|---|---|
1 | joy | 100 ea | 95 | 5 | 95 | |
2 | surprise | 100 | 0 | 100 | ||
3 | disgust | 98 | 2 | 98 | ||
4 | sadness | 97 | 3 | 97 | ||
5 | fear | 89 | 11 | 89 | ||
6 | neutral | 95 | 5 | 95 | ||
mean | 95.7 |
Method | Ground-Truth | Number of Trials | T | F | Ratio | |
---|---|---|---|---|---|---|
1 | gaze | 100 ea | 100 | 0 | 100 | |
2 | hand | 100 | 0 | 100 | ||
mean | 100 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Nam, S.; Song, H.; Kim, Y. Research on Contactless Bio-Signal Measurement Technology for Improving Social Awareness of Individuals with Communication Challenges. Appl. Sci. 2021, 11, 5169. https://doi.org/10.3390/app11115169
Nam S, Song H, Kim Y. Research on Contactless Bio-Signal Measurement Technology for Improving Social Awareness of Individuals with Communication Challenges. Applied Sciences. 2021; 11(11):5169. https://doi.org/10.3390/app11115169
Chicago/Turabian StyleNam, Seonghyeon, Hayoung Song, and Youngwon Kim. 2021. "Research on Contactless Bio-Signal Measurement Technology for Improving Social Awareness of Individuals with Communication Challenges" Applied Sciences 11, no. 11: 5169. https://doi.org/10.3390/app11115169