Fall Detection Based on Key Points of Human-Skeleton Using OpenPose
Abstract
:1. Introduction
2. Related Work
2.1. Skeleton Estimation
2.2. Fall Detection
2.2.1. Inertial Sensor(s)-Based Fall Detection
2.2.2. Context-Based Fall Detection
2.2.3. RF-Based Fall Detection
2.2.4. Sensor Fusion-Based Fall Detection
3. Methods
3.1. OpenPose Gets the Skeleton Information of the Human Body
3.2. Decision Condition One (the Speed of Descent at the Center of the Hip Joint)
3.3. Decision Condition Two (the Angle between the Centerline of the Human and the Ground)
3.4. Decision Condition Three (the Width to Height Ratio of the Human Body External Rectangular)
3.5. Determine Whether a Person Can Stand after a Fall
4. Experimental Results
4.1. Experiment Data and Test
- True positive (TP): a fall occurs, the device detects it.
- False positive (FP): the device announces a fall, but it did not occur.
- True negative (TN): a normal (no fall) movement is performed, the device does not declare a fall.
- False negative (FN): a fall occurs but the device does not detect it.
4.2. Analysis of the Experimental Results
5. Conclusions and Future Work
Conclusions
- (a)
- The environment of daily life is complex, there may be situations in which peoples’ actions cannot be completely captured by surveillance. In the future, we can study the estimation and prediction of peoples’ behavior and actions in the presence of partial occlusion.
- (b)
- In this paper, the action is identified from the side, and the other directions are not considered. Future research can start with multiple directions recognition and then comprehensively judge whether to fall.
- (c)
- Building a fall alarm system for people. In the event of a fall, the scene, time, location, and other detailed information shall be timely notified to the rescuer, to speed up the response speed of emergency rescue.
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- WHO. Number of People over 60 Years Set to Double by 2050; Major Societal Changes Required. Available online: https://www.who.int/mediacentre/news/releases/2015/older-persons-day/en/ (accessed on 17 March 2020).
- Lapierre, N.; Neubauer, N.; Miguel-Cruz, A.; Rincon, A.R.; Liu, L.; Rousseau, J. The state of knowledge on technologies and their use for fall detection: A scoping review. Int. J. Med. Inform. 2018, 111, 58–71. [Google Scholar] [CrossRef] [PubMed]
- Christiansen, T.L.; Lipsitz, S.; Scanlan, M.; Yu, S.P.; Lindros, M.E.; Leung, W.Y.; Adelman, J.; Bates, D.W.; Dykes, P.C. Patient activation related to fall prevention: A multisite study. Jt. Comm. J. Qual. Patient Saf. 2020, 46, 129–135. [Google Scholar] [CrossRef] [PubMed]
- Grossman, D.C.; Curry, S.J.; Owens, D.K.; Barry, M.J.; Caughey, A.B.; Davidson, K.W.; Doubeni, C.A.; Epling, J.W.; Kemper, A.R.; Krist, A.H. Interventions to prevent falls in community-dwelling older adults: US Preventive Services Task Force recommendation statement. JAMA 2018, 319, 1696–1704. [Google Scholar] [PubMed]
- Gates, S.; Fisher, J.; Cooke, M.; Carter, Y.; Lamb, S. Multifactorial assessment and targeted intervention for preventing falls and injuries among older people in community and emergency care settings: Systematic review and meta-analysis. BMJ 2008, 336, 130–133. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Faes, M.C.; Reelick, M.F.; Joosten-Weyn Banningh, L.W.; Gier, M.D.; Esselink, R.A.; Olde Rikkert, M.G. Qualitative study on the impact of falling in frail older persons and family caregivers: Foundations for an intervention to prevent falls. Aging Ment. Health 2010, 14, 834–842. [Google Scholar] [CrossRef] [PubMed]
- Johansson, G. Visual perception of biological motion and a model for its analysis. Percept. Psychophys. 1973, 14, 201–211. [Google Scholar] [CrossRef]
- Chen, T.; Li, Q.; Fu, P.; Yang, J.; Xu, C.; Cong, G.; Li, G. Public opinion polarization by individual revenue from the social preference theory. Int. J. Environ. Res. Public Health 2020, 17, 946. [Google Scholar] [CrossRef] [Green Version]
- Chen, T.; Li, Q.; Yang, J.; Cong, G.; Li, G. Modeling of the public opinion polarization process with the considerations of individual heterogeneity and dynamic conformity. Mathematics 2019, 7, 917. [Google Scholar] [CrossRef] [Green Version]
- Chen, T.; Wu, S.; Yang, J.; Cong, G. Risk Propagation Model and Its Simulation of Emergency Logistics Network Based on Material Reliability. Int. J. Environ. Res. Public Health 2019, 16, 4677. [Google Scholar] [CrossRef] [Green Version]
- Chen, T.; Shi, J.; Yang, J.; Li, G. Enhancing network cluster synchronization capability based on artificial immune algorithm. Hum. Cent. Comput. Inf. Sci. 2019, 9, 3. [Google Scholar] [CrossRef]
- Jiang, C.; Chen, T.; Li, R.; Li, L.; Li, G.; Xu, C.; Li, S. Construction of extended ant colony labor division model for traffic signal timing and its application in mixed traffic flow model of single intersection. Concurr. Comput. Pract. Exp. 2020, 32, e5592. [Google Scholar] [CrossRef]
- Chen, T.; Wu, S.; Yang, J.; Cong, G.; Li, G. Modeling of emergency supply scheduling problem based on reliability and its solution algorithm under variable road network after sudden-onset disasters. Complexity 2020, 2020. [Google Scholar] [CrossRef] [Green Version]
- Ye, Q.; Dong, J.; Zhang, Y. 3D Human behavior recognition based on binocular vision and face–hand feature. Optik 2015, 126, 4712–4717. [Google Scholar] [CrossRef]
- Alagoz, B.B. Obtaining depth maps from color images by region based stereo matching algorithms. arXiv 2008, arXiv:0812.1340. [Google Scholar]
- Foix, S.; Alenya, G.; Torras, C. Lock-in time-of-flight (ToF) cameras: A survey. IEEE Sens. J. 2011, 11, 1917–1926. [Google Scholar] [CrossRef] [Green Version]
- Zhang, Z. Microsoft kinect sensor and its effect. IEEE Multimed. 2012, 19, 4–10. [Google Scholar] [CrossRef] [Green Version]
- Newell, A.; Yang, K.; Deng, J. Stacked hourglass networks for human pose estimation. In Proceedings of the Computer Vision—14th European Conference, Amsterdam, The Netherlands, 18 October 2016; pp. 483–499. [Google Scholar]
- Insafutdinov, E.; Pishchulin, L.; Andres, B.; Andriluka, M.; Schiele, B. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. In Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018; pp. 34–50. [Google Scholar]
- Jeong, S.; Kang, S.; Chun, I. Human-skeleton based Fall-Detection Method using LSTM for Manufacturing Industries. In Proceedings of the 2019 34th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), Jeju Shinhwa World, Korea, 23–26 June 2019; pp. 1–4. [Google Scholar]
- Xu, Q.; Huang, G.; Yu, M.; Guo, Y. Fall prediction based on key points of human bones. Phys. A Stat. Mech. Its Appl. 2020, 540, 123205. [Google Scholar] [CrossRef]
- Koshmak, G.; Loutfi, A.; Linden, M. Challenges and issues in multisensor fusion approach for fall detection. J. Sens. 2016, 2016. [Google Scholar] [CrossRef] [Green Version]
- Mubashir, M.; Shao, L.; Seed, L. A survey on fall detection: Principles and approaches. Neurocomputing 2013, 100, 144–152. [Google Scholar] [CrossRef]
- Ren, L.; Peng, Y. Research of fall detection and fall prevention technologies: A systematic review. IEEE Access 2019, 7, 77702–77722. [Google Scholar] [CrossRef]
- Shahzad, A.; Kim, K. FallDroid: An automated smart-phone-based fall detection system using multiple kernel learning. IEEE Trans. Ind. Inform. 2018, 15, 35–44. [Google Scholar] [CrossRef]
- Fino, P.C.; Frames, C.W.; Lockhart, T.E. Classifying step and spin turns using wireless gyroscopes and implications for fall risk assessments. Sensors 2015, 15, 10676–10685. [Google Scholar] [CrossRef] [PubMed]
- Light, J.; Cha, S.; Chowdhury, M. Optimizing pressure sensor array data for a smart-shoe fall monitoring system. In Proceedings of the 2015 IEEE SENSORS, Busan, Korea, 1–4 November 2015; pp. 1–4. [Google Scholar]
- Han, H.; Ma, X.; Oyama, K. Flexible detection of fall events using bidirectional EMG sensor. Stud. Health Technol. Inform. 2017, 245, 1225. [Google Scholar] [PubMed]
- Sun, J.; Wang, Z.; Pei, B.; Tao, S.; Chen, L. Fall detection using plantar inclinometer sensor. In Proceedings of the 2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom), Beijing, China, 10–14 August 2015; pp. 1692–1697. [Google Scholar]
- Droghini, D.; Principi, E.; Squartini, S.; Olivetti, P.; Piazza, F. Human fall detection by using an innovative floor acoustic sensor. In Multidisciplinary Approaches to Neural Computing; Springer: Berlin/Heidelberg, Germany, 2018; pp. 97–107. [Google Scholar]
- Chaccour, K.; Darazi, R.; el Hassans, A.H.; Andres, E. Smart carpet using differential piezoresistive pressure sensors for elderly fall detection. In Proceedings of the 2015 IEEE 11th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), Abu Dhabi, United Arab Emirates, 19–21 October 2015; pp. 225–229. [Google Scholar]
- Fan, X.; Zhang, H.; Leung, C.; Shen, Z. Robust unobtrusive fall detection using infrared array sensors. In Proceedings of the 2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), Daegu, Korea, 16–18 November 2017; pp. 194–199. [Google Scholar]
- Fan, K.; Wang, P.; Hu, Y.; Dou, B. Fall detection via human posture representation and support vector machine. Int. J. Distrib. Sens. Netw. 2017, 13, 1550147717707418. [Google Scholar] [CrossRef]
- Liu, Y.; Wang, N.; Lv, C.; Cui, J. Human body fall detection based on the Kinect sensor. In Proceedings of the 2015 8th International Congress on Image and Signal Processing (CISP), Shenyang, China, 14–16 October 2015; pp. 367–371. [Google Scholar]
- Kong, X.; Meng, L.; Tomiyama, H. Fall detection for elderly persons using a depth camera. In Proceedings of the 2017 International Conference on Advanced Mechatronic Systems (ICAMechS), Xiamen, China, 6–9 December 2017; pp. 269–273. [Google Scholar]
- Rafferty, J.; Synnott, J.; Nugent, C.; Morrison, G.; Tamburini, E. Fall detection through thermal vision sensing. In Ubiquitous Computing and Ambient Intelligence; Springer: Berlin/Heidelberg, Germany, 2016; pp. 84–90. [Google Scholar]
- Tang, Y.; Peng, Z.; Ran, L.; Li, C. iPrevent: A novel wearable radio frequency range detector for fall prevention. In Proceedings of the 2016 IEEE International Symposium on Radio-Frequency Integration Technology (RFIT), Taipei, Taiwan, 24–26 August 2016; pp. 1–3. [Google Scholar]
- Wang, H.; Zhang, D.; Wang, Y.; Ma, J.; Wang, Y.; Li, S. RT-Fall: A real-time and contactless fall detection system with commodity WiFi devices. IEEE Trans. Mob. Comput. 2016, 16, 511–526. [Google Scholar] [CrossRef]
- Lu, C.; Huang, J.; Lan, Z.; Wang, Q. Bed exiting monitoring system with fall detection for the elderly living alone. In Proceedings of the 2016 International Conference on Advanced Robotics and Mechatronics (ICARM), Macau, China, 18–20 August 2016; pp. 59–64. [Google Scholar]
- De Quadros, T.; Lazzaretti, A.E.; Schneider, F.K. A movement decomposition and machine learning-based fall detection system using wrist wearable device. IEEE Sens. J. 2018, 18, 5082–5089. [Google Scholar] [CrossRef]
- Kepski, M.; Kwolek, B. Event-driven system for fall detection using body-worn accelerometer and depth sensor. IET Comput. Vis. 2017, 12, 48–58. [Google Scholar] [CrossRef] [Green Version]
- Ramezani, R.; Xiao, Y.; Naeim, A. Sensing-Fi: Wi-Fi CSI and accelerometer fusion system for fall detection. In Proceedings of the 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), Las Vegas, NV, USA, 4–7 March 2018; pp. 402–405. [Google Scholar]
- Cao, Z.; Simon, T.; Wei, S.-E.; Sheikh, Y. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 7291–7299. [Google Scholar]
- Shiba, K.; Kaburagi, T.; Kurihara, Y. Fall detection utilizing frequency distribution trajectory by microwave Doppler sensor. IEEE Sens. J. 2017, 17, 7561–7568. [Google Scholar] [CrossRef]
Inertial Sensor(s)-Based | Context-Based | RF-Based | Sensors Fusion-Based | ||
---|---|---|---|---|---|
Ambient-Based | Vision-Based | ||||
Advantages | Easy to implement Few privacies issue High accuracy Real time | Least intrusive Few privacy and security issues | Convenient Accurate | Real-time Contactless Low-cost Nonintrusive | Accurate Significant performance |
Shortcoming | Intrusive | Limited detection range Easier affected by the external environment | Considerable computing Privacy issue Limited capture space | Coverage issue Limited range | Information redundancy Robust fusion algorithm |
Joint Number | Accuracy | X-Coordinate | Y-Coordinate |
---|---|---|---|
0 | 0.97517216 | 333 | 93 |
1 | 0.86759883 | 355 | 113 |
2 | 0.86547723 | 351 | 113 |
3 | 0.70167869 | 343 | 154 |
4 | 0.89546448 | 331 | 184 |
5 | 0.92826366 | 360 | 112 |
6 | 0.91062319 | 372 | 148 |
7 | 0.95447254 | 376 | 180 |
8 | 0.73973876 | 355 | 190 |
9 | 0.90461838 | 377 | 238 |
10 | 0.92706913 | 398 | 284 |
11 | 0.80891138 | 355 | 189 |
12 | 0.92123324 | 337 | 239 |
Action | Action Description | Sample Size |
---|---|---|
Falling actions | Fall | 40 |
Stand up after a fall | 20 | |
Similar falling actions | Squat/Stoop | 20 |
Daily actions | Walk/Sit down | 20 |
Experiment Number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
---|---|---|---|---|---|---|---|---|---|---|
Height (cm) | 172 | 170 | 175 | 163 | 172 | 175 | 177 | 178 | 183 | 177 |
Weight (kg) | 65 | 64 | 72.5 | 67 | 67.5 | 67 | 68 | 57 | 67 | 60 |
Number | Action | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Stoop | Squat | Walk | Sit Down | Fall | Fall1 | Fall2 | Fall3 | Stand Up after a Fall1 | Stand Up after a Fall2 | |
1 | ✕ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
2 | ✓ | ✓ | ✓ | ✕ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
3 | ✕ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
4 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
5 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
7 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
8 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
9 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
10 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Sensitivity | Specificity | Accuracy | |
---|---|---|---|
Result | 98.3% | 95% | 97% |
Algorithm | Classification | Features | Accuracy (%) |
---|---|---|---|
Mel-Frequency Cepstral Coefficients + SVM [30] | Ambient-based | Acoustic waves | 99.14–100% |
Threshold + SVM [25] | Inertial sensor (s)-based | Acceleration, Magnitude variation, Max peak | 91.7–97.8% |
Microwave Doppler + Markov model [44] | RF-based | Velocity Frequency | 95% |
Threshold + Madgwick’s decomposition [40] | Fusion-based | Acceleration, Velocity Displacement | 91.1% |
OpenPose + LSTM [20] | Vision-based | Coordinate, Speed | 98.7% |
OpenPose + Convolutional neural network [21] | Vision-based | skeleton map | 91.7% |
Our proposed OpenPose + three thresholds | Vision-based | Velocity, Angle, Ratio | 97% |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, W.; Jiang, Z.; Guo, H.; Ni, X. Fall Detection Based on Key Points of Human-Skeleton Using OpenPose. Symmetry 2020, 12, 744. https://doi.org/10.3390/sym12050744
Chen W, Jiang Z, Guo H, Ni X. Fall Detection Based on Key Points of Human-Skeleton Using OpenPose. Symmetry. 2020; 12(5):744. https://doi.org/10.3390/sym12050744
Chicago/Turabian StyleChen, Weiming, Zijie Jiang, Hailin Guo, and Xiaoyang Ni. 2020. "Fall Detection Based on Key Points of Human-Skeleton Using OpenPose" Symmetry 12, no. 5: 744. https://doi.org/10.3390/sym12050744