Quality Evaluation Algorithm for Chest Compressions Based on OpenPose Model

Abstract

Aiming at the problems of the low evaluation efficiency of the existing traditional cardiopulmonary resuscitation (CPR) training mode and the considerable development of machine vision technology, a quality evaluation algorithm for chest compressions (CCs) based on the OpenPose human pose estimation (HPE) model is proposed. Firstly, five evaluation criteria are proposed based on major international CPR guidelines along with our experimental study on elbow straightness. Then, the OpenPose network is applied to obtain the coordinates of the key points of the human skeleton. The algorithm subsequently calculates the geometric angles and displacement of the selected joint key points using the detected coordinates. Finally, it determines whether the compression posture is standard, and it calculates the depth, frequency, position and chest rebound, which are the critical evaluation metrics of CCs. Experimental results show that the average accuracy of network behavior detection reaches 94.85%, and detection speed reaches 25 fps.

1. Introduction

In recent years, cardiac arrest (CA) has had an extremely high morbidity and mortality rate worldwide [1]. CPR is the most effective means of rescuing patients with Outside of Hospital Cardiac Arrest (OHCA) [2]. Therefore, it is very urgent to increase the popularity of CPR among the public. A complete CPR process includes five parts: on-site environmental safety assessment, patient injury assessment and calling for help, CCs, artificial respiration and defibrillation. For CPR procedures, CCs have the most extended duration [3]. Interruption of compressions, unstable compression speed, and intensity can all affect the effectiveness of CCs, resulting in poor resuscitation quality and low patient survival [4]. Effective and continuous CCs can improve coronary artery perfusion and myocardial ischemia and hypoxia [5,6]. Therefore, for the training of CPR, standard action training and assessment should be the main objective.

At present, the most effective method in the traditional CPR training mode is to have trainees perform CCs on CPR manikins [7], as shown in Figure 1. The manikin, which is equipped with various sensors, can provide feedback on compression depth, frequency and whether the chest fully rebounds during the operation. However, it cannot monitor the operating position and whether the elbow is straight, which is essential to ensure CC quality. Therefore, monitoring elbow straightness and operating position is generally the instructor’s work. However, one instructor can only train six people simultaneously at most, which is inefficient and unable to guarantee the quality of the supervision. Therefore, the automatic judgment of CCs is an essential step in realizing innovative CPR training and evaluation.

As the name suggests, the aim of HPE [8] is to locate key points (such as the head, right hand, and left shoulder) of the human body from an image or video and to track the pose. With the development of deep learning, HPE is applied in a wide range of fields, such as virtual reality, sports motion analysis, human–computer interactions and medical health care.

In this paper we introduce an intelligent approach for evaluating CC quality using video imagery. OpenPose [9] is applied to the evaluation process in CPR. The recognition algorithm is realized based on the OpenPose network by extracting the coordinates of key points from video frames recorded by monocular cameras, preprocessing the data and using the joint point coordinates for corresponding calculations. Since the key indicators (depth, frequency, position, chest rebound and elbow straightness) can be calculated from the coordinate data, the operator does not need to install any sensors in the manikin body. Based on the rules of CPR guidelines from the American Heart Association (AHA) [10], the European Resuscitation Council (ERC) [11], the European Society of Cardiology (ESC) [12] and the Extracorporeal Life Support Organization (ELSO) [13], and based on our research on compression poses, the algorithm finally makes an assessment. The first section introduces the system architecture and the OpenPose model. Then, we conduct research to define reasonable CC criteria. In the second section, we give a detailed introduction to algorithm implementation based on the principles mentioned above. Finally, we design an experiment to prove the effectiveness of the method.

2. System Architecture

2.1. Design

In order to evaluate the quality of CCs, this paper estimates the human pose in a two-dimensional (2D) image by applying OpenPose. By analyzing the motion features of crucial parts of the human body, this paper assesses the depth, frequency, position, chest rebound and elbow straightness of compression postures. As shown in Figure 2, the system architecture consists of four stages: image acquisition, pose estimation, essential motion feature assessment and quality evaluation. The details of each step are introduced below.

A camera is installed to record the front view of the CC motions of trainees. The camera’s field of view should include the operator’s whole shoulders and elbows as well as the upper body of the lying CPR manikin. Sample CCs video is provided in Supplementary Material (Video S1). FFmpeg [14] is a computer program that can record and convert video and audio. We used the FFmpeg package in python to vary video format, resolution and frame rate [15].

After the format conversion is completed, the pose estimation can be performed. The OpenPose we selected is a 2D human estimation algorithm that detects multiple people’s poses in real time and provides valuable information using multi-GPU-based deep learning methods. OpenPose uses the bottom-up method, first returning all the joint points in a picture and returning the part affinity field (PAF) simultaneously. According to the PAF, the joint points are assigned to the corresponding people. Example processed frame is provided in Supplementary Material (Figure S1). Therefore, its computational complexity is not related with the number of people in the image, and its computational efficiency is higher than the top-down method. Based on the VGG-19 network [16], OpenPose extracts a feature map. It generates a part confidence map to detect each joint of the human body and to determine the relationship between the extracted joints [17].

We used OpenPose to recognize the coordinate information of 18 key points and the confidence of the key points of the trainee from the processed video. However, during the experiment, the key points of the CPR manikin are also extracted together. To judge the standardization of the trainee’s motions, we must first distinguish the key points of the trainee from them. The value of confidence provided by OpenPose can differentiate the key-point information of the trainee in the video from the mixed coordinate information and can generate the key-point coordinate file simultaneously. As shown in Figure 3b, the multi joints and links of the trainee’s body are recognized and generated.

2.2. Research

The AHA Guidelines point out that the compression depth of the chest should be between 5 and 6 cm [10]. Usually, the efficiency of standard CPR is only about 30% of the efficiency of autonomous cardiac pumping. When the compression depth is less than 5 cm, the efficiency of CPR is worse, which may cause the death of the patient.

ERC has pointed out that, in the process of CCs, it is essential to maintain chest rebound [11]. Chest rebound can allow blood to flow back sufficiently to be fully “ejected” during the subsequent compression [18].

In the ERC guidelines from 2021, the frequency is at least 100 times/min [11]. Furthermore, it must be 100–120 times/min in the AHA guidelines [10]. If the frequency is less than 100 times/min, the return of spontaneous circulation (ROSC) is significantly reduced. When the compression frequency is more than 120 times/min, due to the limitation of chest rebounds, myocardial perfusion and coronary blood flow both decrease, which reduces the success rate of the ROSC [19].

The correct compression position is the midpoint of the connection between the two nipples, which can withstand greater force [12]. If the position is offset during CPR, it is easy to fracture the patient’s ribs [20].

Although there is no explicit requirement in major international CPR guidelines that states that the operator’s elbows must be straight during compressions, during the actual operation, it was found that elbow straightness significantly influences the compression depth and frequency of the chest, which in turn affects the quality. At present, the relationship between them has not been accurately studied. Therefore, this section examines the relationship between elbow straightness and compression depth, frequency and chest rebound in a certain period, and then it explores the effects of elbow straightness on CC quality.

To test the influence on depth, frequency, rebound and position of different elbow straightness, and to determine the appropriate elbow angle used for our algorithm, we recruited 34 volunteers to perform compression on a CPR manikin with real-time feedback under random conditions (referring to random within three groups of angles of 151–160°, 161–170°, and 171–180°).

As shown in

Table 1. The quality of CC under different elbow angles.

Observation Indicators	151–160°	161–170°	171–180°	p
Average depth (mm)	50.62 ± 5.16	52.48 ± 5.39	53.79 ± 5.54	>0.05
Average frequency (times/min)	99	106	117	<0.05
Correct rate of depth (%)	75.82 ± 11.68	79.21 ± 11.10	84.35 ± 8.97	<0.05
Full rebound rate (%)	76.32 ± 15.32	78.64 ± 18.73	83.96 ± 13.47	<0.05
Correct rate of frequency (%)	72.32 ± 20.32	80.96 ± 15.67	86.89 ± 16.73	<0.05
Correct rate of compression position (%)	100 ± 0.00	100 ± 0.00	100 ± 0.00	>0.05

, a t-test was used for the measurement data, and a chi-squared test was used for the enumeration data. When p < 0.05, the difference was statistically significant. Continuous variables are expressed as mean ± standard deviation (i.e., x ± s).

It can be seen from Figure 4 that the correct proportion of compression depth in the range of 171–180° is the highest, and the difference is statistically significant (p < 0.05) compared with the other two groups. Moreover, the trend of correct rates in the frequency group and in the chest rebound group is consistent with the depth group. Therefore, it can be concluded that, when the elbow is bent during operation, the error rates of depth, frequency, and rebounds increase, affecting CC quality.

In this study, experiments show that elbows should be kept as straight as possible to ensure quality and to improve the success rate of rescue, and elbow angles in the 171–180° range are acceptable. In the following study, operator elbow straightness is one of the key indicators to measure quality.

Based on the above research on the five indicators, we specify five standardized operating parameters and eligibility requirements in

Table 2. Criteria for chest compressions.

Indicators	Desired Value Range	Sub-Requirement	Overall Requirement
Operating position	The midpoint of the connection between the two nipples	The correct rate is more than 70%	Every indicator should be qualified
Elbow straightness	171–180°	The correct rate of the left and right elbows should both reach more than 70%
Frequency	100–120 times/min	The correct rate is more than 70%
Depth	5–6 cm
Chest rebound	Must be fully rebound

. It is worth noting that we set a fault tolerance rate of 30%. Specifically, a qualified CC video requires that the correct rate of each inspection index must reach 70%, and each inspection index is qualified.

2.3. Key Motion Feature Assessment

2.3.1. Depth of Compression

OpenPose can identify the coordinates of the key points of the human hand, so we use the max displacement of the coordinates to determine compression depth. During CCs, both hands are crossed together and pressed downwards vertically, and they have the same moving distances. Thus, we only consider the compression depth of a single hand. The initial position of the right wrist is selected as the starting position of the compression. Since the x-coordinate of the corresponding key point has no reference meaning of depth, only its y-coordinate needs to be considered when calculating.

As shown in Figure 5, the depth of compression means the distance from the starting compression position to the deepest position in a cycle. Therefore, we took the order number of the frame as the abscissa and the y-coordinate of the right wrist key point in the corresponding frame as the ordinate, and we drew the curve by the plt.plot function in python. Then, we found the compression depth in a cycle by the argrelextrema function of scipy.signal., which calculates the relative extrema of input array data of y-coordinates and returns a tuple with maximum value. As shown in Figure 6, in actual experiments, it was found that local minimum points appear in one cycle due to the fast frame rate and the accuracy of OpenPose. As a result, multi values are selected incorrectly in one cycle. To avoid this problem and to prevent missing extreme points, after testing, the y-value is selected to output every two frames to prevent the problem of local minimum points. Example curve is provided in Supplementary Material (Figure S2). Since the point coordinates in the image are in pixels, it is necessary to transform the depth to the actual depth through a corresponding scale transformation to discriminate whether the compression depth is 5–6 cm.

2.3.2. Chest Rebound

We use the same methods to find the position of the minimum value in the scattered point image and to avoid the appearance of local minimum points. To maintain adequate rebound of the chest after each compression, we must ensure that the starting position of each compression is close to the initial position. Based on the weak perspective model, we can determine whether the chest fully rebounds by judging whether the Euclidean distance between the starting position and the initial position of the wrist joint point in each CC is within the expected range, as shown in Figure 7.

2.3.3. Frequency of Compression

We obtained an initial video with a 30 fps frame rate and judged the compression frequency by the order numbers of frames in which the minimum point appears, i.e., the end of one single compression. For example, assuming that the abscissa is e and that there is a minimum y value, the frequency f = 1800/e. To eliminate the influence of local minimum points during the calculation, the y value is taken every two frames. Thus, in practice, the frequency f = 900/e. As shown in Figure 8, if the minimum values appear at the three horizontal axis points a, b and c from the starting position, it means that the three compressions end at the a, b and c frames. Then, we can calculate the corresponding compression frequencies f1 = 900/a, f2 = 900/(b − a) and f3 = 900/(c − b).

If the calculated frequency is lower than 100 times/min, the algorithm regards the operation as too slow; if the frequency is higher than 120 times/min, the operation is deemed to be too fast. If it is between 100 and 120 times/min, the operation is judged as correct.

2.3.4. Position of Compression

To discriminate whether the CC position is correct, based on the weak perspective model, a circular area on the chest of the CPR manikin is marked, and to determine whether the position is correct, the coordinate of the starting position is determined to be outside or within the circular area. By analyzing the coordinates of the positions of 213 correct CC samples, we found that, except for individual errors, they are essentially distributed in the circular area, as described in (1).

$${\mathrm{x}}^2+{\mathrm{y}}^2-374\mathrm{x}-54\mathrm{y}+\mathrm{35,662}=0$$

(1)

2.3.5. Elbow Angles

In order to judge whether both elbows are straight when performing CCs, we introduce the elbow angle index, which determines whether a pose is correct. Due to individual differences, we can consider the posture with the corresponding angle to be between 170 and 180° as qualified, as shown in Figure 9a. As shown in Figure 10, by knowing the coordinates of the three elbow key points A, B and C, the angle of elbows can be measured by the cosine formula.

Knowing A(x₁,y₁), B(x₂,y₂) and C(x₃,y₃), we first calculate $\overrightarrow{\mathrm{AB}}$ and $\left|\mathrm{AB}\right|$ with a formula, as shown in Equations (2) and (3). In the same way, $\overrightarrow{\mathrm{BC}}$ and $\left|\mathrm{BC}\right|$ can be calculated, as shown in Equations (4) and (5).

$$\overrightarrow{\mathrm{AB}}=({\mathrm{x}}_2-{\mathrm{x}}_1,{\mathrm{y}}_2-{\mathrm{y}}_1),$$

(2)

$$\left|\mathrm{AB}\right|={({(({\mathrm{x}}_2-{\mathrm{x}}_1)}^2+({\mathrm{y}}_2-{\mathrm{y}}_1)}^2){)}^{\frac{1}{2}},$$

(3)

$$\overrightarrow{\mathrm{BC}}=({\mathrm{x}}_3-{\mathrm{x}}_2,{\mathrm{y}}_3-{\mathrm{y}}_2),$$

(4)

$$\left|\mathrm{BC}\right|={({(({\mathrm{x}}_3-{\mathrm{x}}_2)}^2+({\mathrm{y}}_3-{\mathrm{y}}_2)}^2){)}^{\frac{1}{2}}.$$

(5)

Finally, the angle between the two vectors formed by the three elbow key points can be measured by applying the law of cosines, as described in (6).

$$\angle \mathrm{B}=\arccos (\overrightarrow{\mathrm{AB}}\times \overrightarrow{\mathrm{BC}}÷(\left|\mathrm{AB}\right|\left|\mathrm{AC}\right|))$$

(6)

3. Results and Analysis

3.1. Experimental Environment and Methods

The experimental environment consists of an Honor V30 mobile phone, a CPR manikin, and a PC with an i7 Intel core processor and 16 GB RAMs with GTX-1080-Ti GPUs to operate OpenPose and other computation processes. The human–computer interaction interfaces are the Browser/Server architecture web page. The front-end uses the Vue framework to realize function design and page development. The back-end uses the NET framework to build a 5-layer structure of an entity layer, a data processing layer, a business logic processing layer, an API layer and a view layer. Our system functions include user registration and login, information management and maintenance, video capturing, uploading and management, and motion evaluation.

This article uses a monocular camera to experiment. The test set includes correct actions and seven kinds of incorrect actions. The incorrect CC actions consist of elbow-flexing, too deep, too shallow, too fast, too slow, insufficient chest rebound and wrong operating position. If a CC meeting the five requirements is detected, it is judged to be correct. Otherwise, our browser page prompts the corresponding specific error reason. A total of five experimenters (three males and two females) were invited to conduct CCs on CPR manikins with sensors that can feedback depth, frequency, rebound and position. In order to make the experiment more convincing, the height and weight of the experimenters were chosen to have a significant difference, with a weight range of 50–80 kg and a height range of 1.6–1.8 m. Each person performed each of the above 8 kinds of actions a total of 30 times. A total of 1200 (5 × 8 × 30) test actions were performed, including 1050 incorrect and 150 correct actions. According to the feedback data from CPR manikin’s sensors, 1200 samples were manually labeled as correct or incorrect. The judgment result from our algorithm was compared with the labels to detect our method’s accuracy.

3.2. Results and Analysis

The above experimental results in

Table 3. The experimental results.

Action Type	The Number of Successes per Person in 30 Trials					The Success Rate/%
	1	2	3	4	5
Correct	29	27	28	29	28	94.00%
Elbow-flexing	28	30	28	28	28	94.67%
Too deep	27	28	27	29	29	93.33%
Too shallow	30	28	29	29	30	97.33%
Too fast	28	27	28	28	27	92.00%
Too slow	29	30	30	29	30	98.67%
Insufficient chest rebound	30	29	28	28	29	96.00%
Wrong operating position	26	28	27	28	27	90.67%

show that, when the camera is at a fixed position, the detection accuracy of the evaluation algorithm based on the OpenPose model reaches 94.85%, and the operating speed is up to 25 fps. When the algorithm is running, the recognition effect of each key point of the human body is good, and it can correctly distinguish the key points of the human body and the manikin body. In order to verify the effectiveness of the algorithm, this paper uses four performance indicators of accuracy, precision, recall and F-measure to evaluate the performance of the proposed algorithm. The calculation methods of the four indicators are shown by Formulas (7)–(10).

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}},$$

(7)

$$\mathrm{Precision} =\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},$$

(8)

$$\mathrm{Recall} =\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},$$

(9)

$$\mathrm{F}-\mathrm{measure} =2\times \frac{\mathrm{Recall} \times \mathrm{Precision}}{\mathrm{Recall} + \mathrm{Precision}}.$$

(10)

As shown in

Table 4. The performance of the proposed method.

Parameter	Value	Parameter	Value
TP	0.118	Accuracy	94.58%
FP	0.047	Precision	71.57%
FN	0.008	Recall	94.00%
TN	0.828	F-measure	81.27%

, the accuracy of the proposed method was 94.58%, and TP and TN were measured to be 0.118 and 0.828, respectively. In general, precision measures the fraction of positive examples from the group that the proposed method predicts to be positive, and recall measures the fraction of positive instances from the positive group of the actual class. F-measure is a harmonic mean of precision and recall. The results of precision, recall and F-measure were calculated to be 71.57%, 94.00% and 81.27%, respectively. The difference in precision and recall occurred because the FP was slightly higher than the FN by 0.039. This means that incorrect actions were counted as correct more often than correct actions as incorrect. Further studies are needed to improve the performance of the method.

4. Conclusions

An automatic CC evaluation approach was developed by using video imagery for people who want to learn how to perform CCs, such as students, drivers and police. We applied OpenPose to extract multiple joints and links of the human body in 2D images and then analyzed critical body parts that influence quality. In order to consider the individual physical differences of each person, a total of 1200 samples were generated from 40 videos recorded by monocular cameras. Five criteria were defined based on our research and on rules from major CPR guidelines. The angles of elbows, depth, frequency, position and chest rebound were parameterized to discriminate between correct and incorrect actions.

Our approach has certain advantages by comparing several indicators, such as comprehensiveness, applicability, cost, performance and flexibility.

• In addition to detecting the general indicators of compression, we propose automated discrimination of operator posture for the first time.
• Our algorithm is not affected by the surrounding environment. It uses a typical monocular RGB camera as a sensor to collect data, which has better applicability than a depth camera and has a lower cost.
• The performance of our algorithm is 94.58%, 71.57%, 94.00%, and 81.27% in accuracy, precision, recall and F-measure, respectively.
• The application of the HPE method in CPR assessment can not only reduce the problem of incorrect scores caused by an instructor’s lack of energy, but it can also allow trainees to understand their own mistakes in private practice to better master CPR techniques.

For future studies, improvements can be made in the following two aspects. Firstly, experimentation can be conducted to explore better methods to eliminate the local minimum problem, such as using the median filter. Furthermore, 3D analysis of the human body as well as deep learning approaches are to be further researched to improve its reliability in public CPR popularization.