A Random Forest-Based Accuracy Prediction Model for Augmented Biofeedback in a Precision Shooting Training System

In the military, police, security companies, and shooting sports, precision shooting training is of the outmost importance. In order to achieve high shooting accuracy, a lot of training is needed. As a result, trainees use a large number of cartridges and a considerable amount of time of professional trainers, which can cost a lot. Our motivation is to reduce costs and shorten training time by introducing an augmented biofeedback system based on machine learning techniques. We are designing a system that can detect and provide feedback on three types of errors that regularly occur during a precision shooting practice: excessive hand movement error, aiming error and triggering error. The system is designed to provide concurrent feedback on the hand movement error and terminal feedback on the other two errors. Machine learning techniques are used innovatively to identify hand movement errors; the other two errors are identified by the threshold approach. To correct the excessive hand movement error, a precision shot accuracy prediction model based on Random Forest has proven to be the most suitable. The experimental results show that: (1) the proposed Random Forest (RF) model achieves the prediction accuracy of 91.27%, higher than any of the other reference models, and (2) hand movement is strongly related to the accuracy of precision shooting. Appropriate use of the proposed augmented biofeedback system will result in a lower number of rounds used and shorten the precision shooting training process.


Introduction
Accuracy is the most important parameter in precision shooting. To achieve the desired level of shooting accuracy, trainees have to practice a lot, which takes a lot of rounds (cartridges) and time. Advances in science and technology today provide the means to improve and develop training techniques. In recent years, many sensors such as accelerometers [1,2], gyroscopes [3,4], and pressure sensors [5,6] have been widely used in various sports activities. For example, ultrasound sensors, which represent a reasonable compromise between cost and accuracy, have been used in archery to analyze the stability of the archer's hands [7]. Based on signal features collected from 3D accelerometers, activities such as football, Nordic walking, and cycling were recognized [8]. Inertial sensors were attached to the upper arm to prevent shoulder injuries in baseball and volleyball [9]. The use of sensor Figure 1. The distributed architecture of the biofeedback system. The acquired sensor signals are processed by the processing device. The user can receive concurrent feedback on unwanted or excessive hand movements (solid arrows) or terminal feedback on the shooting results (dashed arrow). All signals and processed data are stored in the database.

Measurement of Precise Shooting Performance
In precise shooting practice, shots are fired at the standardized pistol target shown in Figure 2a. The outer nine rings are 50 mm wide and the inner two rings are 25 mm wide. The results are scored according to the distance from the center from 0 (miss) to 11 (center). We define the shooting session as a group of N fired shots. Usually a shooting session consists of N = 5 shots. There are two basic measures for evaluating shooting performance: accuracy and precision, graphically shown in Figure  2b.
Accuracy refers to how well the shot impact point is centered on the standardized pistol target; the closer to the center of the target, the better the accuracy. The measures of shooting performance are the accuracy, i.e., the extent to which the center of the group of shots is close to the center of the target, and the precision, i.e., the tightness or the size of the group [12]. Precision describes the spread of individual shots around the center of the shot group; it should be noted that in this paper we define Figure 1. The distributed architecture of the biofeedback system. The acquired sensor signals are processed by the processing device. The user can receive concurrent feedback on unwanted or excessive hand movements (solid arrows) or terminal feedback on the shooting results (dashed arrow). All signals and processed data are stored in the database.

Measurement of Precise Shooting Performance
In precise shooting practice, shots are fired at the standardized pistol target shown in Figure 2a. The outer nine rings are 50 mm wide and the inner two rings are 25 mm wide. The results are scored according to the distance from the center from 0 (miss) to 11 (center). We define the shooting session as a group of N fired shots. Usually a shooting session consists of N = 5 shots. There are two basic measures for evaluating shooting performance: accuracy and precision, graphically shown in Figure 2b.
Sensors 2020, 20, x FOR PEER REVIEW 4 of 15 a group of 5 shots as one shooting session. An excellent shooter can achieve a high shooting result for all 5 shots if the shots are placed at a small distance from the center of the target. As already mentioned in the Introduction section, only shooting accuracy is discussed in this paper.
(a) (b) Figure 2. (a) Standardized pistol target; (b) the distribution of accuracy and precision: (i) high accuracy and high precision, (ii) low accuracy but high precision, (iii) low precision but high accuracy, (iv) low accuracy and low precision.
The shot score, read from the target, is not a suitable measure of the shooting performance when we compare the results at different target distances [23]. The three most commonly used distances in shooting practice are 6 m, 10 m and 15 m, as shown in Figure 3. Obviously, it is much easier to achieve a good result at shorter distances than at longer distances. A relatively good shot at 6 m would be a poor shot at 10 m and a miss at 15 m, see the thick dotted lines in Figure 3. In order to fairly evaluate the precision shooting performance from different shooting distances, we use the shot result angle as a measure instead of the absolute shot result in numerical values from 0 to 11, which are read from the target. The shot result angle is defined by Eq. (1), where r is the distance between the point of shot impact and the center of the target and d is the shooting distance.

• arctan
(1) In our research we define only two types of shot results: the good and the bad. The test subjects shoot from three different distances: 6 m, 10 m and 15 m. We define a good shot based on the angle α. The threshold value is defined according to the shot result of 8 at a shooting distance of 15 m, which gives α 0.5°. For example, for the same shot angle α, the shot score can be classified as good (8) at 6 m, medium (4) at 10 m and bad (out of target) at 15 m shooting distance; see the dotted lines in Figure 3. Accuracy refers to how well the shot impact point is centered on the standardized pistol target; the closer to the center of the target, the better the accuracy. The measures of shooting performance are the accuracy, i.e., the extent to which the center of the group of shots is close to the center of the target, and the precision, i.e., the tightness or the size of the group [12]. Precision describes the spread of individual shots around the center of the shot group; it should be noted that in this paper we define a group of 5 shots as one shooting session. An excellent shooter can achieve a high shooting result for all 5 shots if the shots are placed at a small distance from the center of the target. As already mentioned in the Introduction section, only shooting accuracy is discussed in this paper.
The shot score, read from the target, is not a suitable measure of the shooting performance when we compare the results at different target distances [23]. The three most commonly used distances in shooting practice are 6 m, 10 m and 15 m, as shown in Figure 3. Obviously, it is much easier to achieve a good result at shorter distances than at longer distances. A relatively good shot at 6 m would be a poor shot at 10 m and a miss at 15 m, see the thick dotted lines in Figure 3.
Sensors 2020, 20, x FOR PEER REVIEW 4 of 15 a group of 5 shots as one shooting session. An excellent shooter can achieve a high shooting result for all 5 shots if the shots are placed at a small distance from the center of the target. As already mentioned in the Introduction section, only shooting accuracy is discussed in this paper.
(a) (b) Figure 2. (a) Standardized pistol target; (b) the distribution of accuracy and precision: (i) high accuracy and high precision, (ii) low accuracy but high precision, (iii) low precision but high accuracy, (iv) low accuracy and low precision.
The shot score, read from the target, is not a suitable measure of the shooting performance when we compare the results at different target distances [23]. The three most commonly used distances in shooting practice are 6 m, 10 m and 15 m, as shown in Figure 3. Obviously, it is much easier to achieve a good result at shorter distances than at longer distances. A relatively good shot at 6 m would be a poor shot at 10 m and a miss at 15 m, see the thick dotted lines in Figure 3. In order to fairly evaluate the precision shooting performance from different shooting distances, we use the shot result angle as a measure instead of the absolute shot result in numerical values from 0 to 11, which are read from the target. The shot result angle is defined by Eq. (1), where r is the distance between the point of shot impact and the center of the target and d is the shooting distance.

• arctan
(1) In our research we define only two types of shot results: the good and the bad. The test subjects shoot from three different distances: 6 m, 10 m and 15 m. We define a good shot based on the angle α. The threshold value is defined according to the shot result of 8 at a shooting distance of 15 m, In order to fairly evaluate the precision shooting performance from different shooting distances, we use the shot result angle as a measure instead of the absolute shot result in numerical values from 0 Sensors 2020, 20, 4512 5 of 16 to 11, which are read from the target. The shot result angle is defined by Equation (1), where r is the distance between the point of shot impact and the center of the target and d is the shooting distance.
In our research we define only two types of shot results: the good and the bad. The test subjects shoot from three different distances: 6 m, 10 m and 15 m. We define a good shot based on the angle α. The threshold value is defined according to the shot result of 8 at a shooting distance of 15 m, which gives α = 0.5 • . For example, for the same shot angle α, the shot score can be classified as good (8) at 6 m, medium (4) at 10 m and bad (out of target) at 15 m shooting distance; see the dotted lines in Figure 3.

Data Acquisition and Preprocessing
Our sensor device mounted onto the bottom of the pistol grip includes a 3-axis accelerometer and a 3-axis gyroscope. The sampling frequency f s of our sensor device is 250 Hz and the number of recorded samples for each shot is 1000; 750 samples (3 s) are recorded before the firing and 250 samples (1 s) are recorded after the firing. Figure 4 shows the absolute value of the acceleration signal of a fired shot. For a better readability of the graph, only the signal between sample numbers 500 and 1000 is plotted. As shown in Figure 4, there is an evident change in acceleration at sample number 750, which represents the firing moment of the pistol. Each shot is divided into four phases: the aiming, triggering, firing and recoil phases. It was proved in [10] that the hand movement before the firing phase is strongly negatively correlated with the accuracy of the precision shooting.

Data Acquisition and Preprocessing
Our sensor device mounted onto the bottom of the pistol grip includes a 3-axis accelerometer and a 3-axis gyroscope. The sampling frequency fs of our sensor device is 250 Hz and the number of recorded samples for each shot is 1000; 750 samples (3 s) are recorded before the firing and 250 samples (1 s) are recorded after the firing. Figure 4 shows the absolute value of the acceleration signal of a fired shot. For a better readability of the graph, only the signal between sample numbers 500 and 1000 is plotted. As shown in Figure 4, there is an evident change in acceleration at sample number 750, which represents the firing moment of the pistol. Each shot is divided into four phases: the aiming, triggering, firing and recoil phases. It was proved in [10] that the hand movement before the firing phase is strongly negatively correlated with the accuracy of the precision shooting. We can acquire the angular velocity signals from the 3-axis gyroscope of the sensor device. The angular velocity signal around the X-axis is shown in Figure 5a. Since we use the shot angle result as a measure of performance, we use to calculate the angle signal by the expression Equation (2), where is the sampling time. The angle signal is shown in Figure 5b, which is calculated from the angular velocity signal from Figure 5a. We can acquire the angular velocity signals ω[n] from the 3-axis gyroscope of the sensor device. The angular velocity signal around the X-axis is shown in Figure 5a. Since we use the shot angle result as a measure of performance, we use ω[n] to calculate the angle signal β[n] by the expression Equation (2), where T s is the sampling time. The angle signal β[n] is shown in Figure 5b, which is calculated from the angular velocity signal ω[n] from Figure 5a.
We can acquire the angular velocity signals from the 3-axis gyroscope of the sensor device. The angular velocity signal around the X-axis is shown in Figure 5a. Since we use the shot angle result as a measure of performance, we use to calculate the angle signal by the expression Equation (2), where is the sampling time. The angle signal is shown in Figure 5b, which is calculated from the angular velocity signal from Figure 5a.  In total, we have six measured signals: angular velocity (Gx,Gy,Gz) acquired by a 3-axis gyroscope, acceleration (Ax, Ay, Az) acquired by an accelerometer. The angle signals β (Ax, Ay, Az) are derived from angular velocity signals (Equation (2)). For each shot we also calculate an angle variation value (peak-to peak) in triggering phase, just before firing; as discussed in detail in Section 2.4.

Classification
In this paper, we perform a binary classification by defining only two types of shot results: the good and the bad, according to the shot angle threshold of 0.5 • as defined by Equation (1). If the shot angle exceeds the threshold, the shot is classified as bad, otherwise it is classified as good. The movement of the pistol is acquired through the attached sensor device including a 3-axis accelerometer and a 3-axis gyroscope. The measure for pistol movement is the standard deviation of the measured and derived sensor signals; for example, the standard deviation of the rotational speed around the X-axis. We use the Pearson correlation coefficient method [24] and the importance score of each feature obtained by the random forest (RF) model to obtain the optimized feature set to train a model for model training.
To obtain the data of the training model, we have to predefine the conditions and measures for data cleaning and the removal of outliers. There are three basic types of errors in precision shooting that can influence the shot results: hand movement errors, aiming errors and triggering errors [11]. The hand movement error occurs when the pistol movement exceeds a predefined threshold, which is caused by the shooter's conscious or unconscious body/arm swaying. We can detect this error by calculating the variability of the sensor signals, as shown in Figure 6a. We measure the variability of the signal values between sample numbers 500 and 700 to give the users enough time to react to possible feedback on the motion error. It usually takes about 180 ms for people to take action when they receive the feedback [25]. The triggering error occurs when the trigger is pulled incorrectly, resulting in an unwanted pistol movement just before the shot is fired. We can monitor this error by analyzing the change in the pistol angle in the last 100 ms before firing, in Figure 6b this happens at angle signals with sample numbers between 720 and 745. The aiming error occurs due to an incorrect aiming technique. It cannot be detected by analyzing sensor signals, but by statistically analyzing a number of shots from the same user after the shooting session. The aiming error scenario is shown in Figure 6c, which shows that the aiming error cannot be distinguished from the signal of a good shot.
aiming technique. It cannot be detected by analyzing sensor signals, but by statistically analyzing a number of shots from the same user after the shooting session. The aiming error scenario is shown in Figure 6c, which shows that the aiming error cannot be distinguished from the signal of a good shot.
Significantly, the hand movement before the firing phase is strongly negatively correlated with the accuracy of the precision shot, as demonstrated in [10]. However, there is one exception-where the shot result is good even if the hands move too much-we called this lucky shots.  Significantly, the hand movement before the firing phase is strongly negatively correlated with the accuracy of the precision shot, as demonstrated in [10]. However, there is one exception-where the shot result is good even if the hands move too much-we called this lucky shots.
Our goal is to establish a model for hand movement error based on machine learning using the sensor signals and data. To achieve this, it is necessary to eliminate the interference of the other two errors-triggering error and aiming error as well as lucky shots. Any data that interferes with the model training for a just reason are defined as outliers. The outlier elimination flowchart is shown in Figure 7. Our goal is to establish a model for hand movement error based on machine learning using the sensor signals and data. To achieve this, it is necessary to eliminate the interference of the other two errors-triggering error and aiming error as well as lucky shots. Any data that interferes with the model training for a just reason are defined as outliers. The outlier elimination flowchart is shown in Figure 7. First, we have to identify and eliminate the triggering error outliers. The triggering error typically occurs within the last 100 ms (25 samples) before the firing phase, so we analyze all signals between sample numbers 720 and 745. The guard interval of the trigger phase is 5 samples (20 ms). In most cases the triggering error causes a vertical and/or lateral movement of the gun. The expression for the detection of a triggering error is defined by Equation (3), where R is the difference between the maximum and minimum value of where the value of n is a sample number between 720 and 745. The shot score is influenced by ; if is greater than the predefined score shot angle α 0.5°, a triggering error is very likely. For this reason, we define the threshold value for the triggering error angle at 0.5°.
, 720 745 (3) First, we have to identify and eliminate the triggering error outliers. The triggering error typically occurs within the last 100 ms (25 samples) before the firing phase, so we analyze all signals between sample numbers 720 and 745. The guard interval of the trigger phase is 5 samples (20 ms). In most cases the triggering error causes a vertical and/or lateral movement of the gun. The expression for the detection of a triggering error is defined by Equation (3), where R is the difference between the maximum and minimum value of β[n] where the value of n is a sample number between 720 and 745. The shot score is influenced by R; if R is greater than the predefined score shot angle α = 0.5 • , a triggering error is very likely. For this reason, we define the threshold value for the triggering error angle at R > 0.5 • .
After we have eliminated the triggering error outliers, we detect and remove some other known outliers. The most important datum collected by the sensors is the angular velocity (ω[n]) around the X-axis, since the movement of the pistol most is often in a vertical and lateral direction. Therefore, we focus primarily on the gyroscope signals around the X-axis. In Figure 8, the standard deviation of the pistol angle around the X-axis, calculated from the β[n] signal, is plotted against the shot result expressed by its angle. The threshold value for the shot result is 0.5 • , which was defined to distinguish between good and bad shots. Both thresholds divide the graph into four areas. The shots in area A are lucky shots. Area B and area C correspond to the conclusions in [10], where the precision shooting result is inversely proportional to the standard deviation of the hand accelerations and rotations. Area D most likely contains the shots with the aiming error, because there must be no triggering errors and the hand movement is also small, but the shot result is still bad. Figure 8. Detection of outliers denoted as lucky shots and the shots with aiming error. Area A is the area of lucky shots, the points in Area B and C represent the hand movement detection regions, and Area D most probably is the area denoting points of aiming error.
Using the above method, we can remove shots with aiming errors, triggering errors and lucky shots. After this procedure we can successfully establish the accuracy prediction model of precision shooting based on the random forest.
The random forest (RF) is an ensemble tree-based algorithm, which is specified as follows: (a) Utilize bagging to randomly generate k diversified subsets ( , , , … , ) of the entire training set . (b) For each subset , grow an unpruned classification tree . During the splitting of each node, rather than choosing the best split among all predictors , randomly selects ( ≪ ) of these predictors , and then choose the best split among those variables.
(c) Predict new data by aggregating the predictions of the k trees following the majority decision rule.

Bayesian Hyper-Parameter Optimization
The classification algorithms usually have several hyperparameters that strongly influence the accuracy of the model. The most common methods for selecting hyperparameters are mainly Grid Search [26], Random Search [27] and Bayesian Optimization. Among them, Grid Search, also known As can be seen in Figure 8, there is another threshold value, which is defined at 0.18 • of the standard deviation of the pistol angle around the X-axis. In [10], the authors have shown that hand movement and shot score are negatively correlated; higher values of hand movement lead to a lower shot score and consequently to a higher shot angle. The current study has established that more experienced shooters have smaller shot angles, as shown in Section 3. Based on the above, it is unlikely that a high hand movement will give a good shot score, but in some cases, it happens-we call such shots lucky shots. Based on our experience with shooting tests, we set the estimated probability of lucky shots at 5%. Under this assumption, we calculate the threshold value of 0.18 • , which meets our criterion. In order to validate our assumption, our system should be supplemented by an optical measuring system, which was not within the scope of our study.
Both thresholds divide the graph into four areas. The shots in area A are lucky shots. Area B and area C correspond to the conclusions in [10], where the precision shooting result is inversely proportional to the standard deviation of the hand accelerations and rotations. Area D most likely contains the shots with the aiming error, because there must be no triggering errors and the hand movement is also small, but the shot result is still bad.
Using the above method, we can remove shots with aiming errors, triggering errors and lucky shots. After this procedure we can successfully establish the accuracy prediction model of precision shooting based on the random forest.
The random forest (RF) is an ensemble tree-based algorithm, which is specified as follows: (a) Utilize bagging to randomly generate k diversified subsets (D 1, D 2 , D 3 , . . . , D k ) of the entire training set D. (b) For each subset D i , grow an unpruned classification tree T i . During the splitting of each node, rather than choosing the best split among all predictors M, randomly selects m try (m try M) of these predictors M, and then choose the best split among those variables. (c) Predict new data by aggregating the predictions of the k trees T k following the majority decision rule.

Bayesian Hyper-Parameter Optimization
The classification algorithms usually have several hyperparameters that strongly influence the accuracy of the model. The most common methods for selecting hyperparameters are mainly Grid Search [26], Random Search [27] and Bayesian Optimization. Among them, Grid Search, also known as brute force search method, is currently the most common one. It runs through the hyperparameter combination in a loop, searching for all values in the range of candidate parameters and uses the most powerful parameters as optimal parameters of the model. Random Search does not loop through all candidate parameter values, but randomly selects a fixed number of parameters from the given parameter distribution. The disadvantage of these two methods is that each new estimate of the optimal parameter is independent of the previous training. If the amount of data is large, many parameters must be tuned, which requires excessive computational resources.
Bayesian optimization [19], which achieves state-of-the-art results for a few global optimization problems, is an effective solution for hyperparameter tuning. The application scenario of the Bayesian optimization method is that only the input and output values are known while the function equation, function structure and the mathematical properties are unknown. Known points are used to learn the form of the objective function. The specific method consists of increasing the sample points and updating the posterior probability distribution of the objective function until the posterior distribution is basically consistent with the real distribution. Compared to grid and random search, it is better to use the last parameter information to adjust the current parameters. In this paper, we use a sequential model-based optimization (SMBO) algorithm [28], which is an advanced Bayesian optimization scheme for tuning the hyperparameters. SMBO usually builds a model to select the optimal hyperparameters in a defined search space, and is particularly common when evaluating the cost function is expensive. The SMBO algorithm is shown in Figure 9.
Sensors 2020, 20, x FOR PEER REVIEW 9 of 15 candidate parameter values, but randomly selects a fixed number of parameters from the given parameter distribution. The disadvantage of these two methods is that each new estimate of the optimal parameter is independent of the previous training. If the amount of data is large, many parameters must be tuned, which requires excessive computational resources. Bayesian optimization [19], which achieves state-of-the-art results for a few global optimization problems, is an effective solution for hyperparameter tuning. The application scenario of the Bayesian optimization method is that only the input and output values are known while the function equation, function structure and the mathematical properties are unknown. Known points are used to learn the form of the objective function. The specific method consists of increasing the sample points and updating the posterior probability distribution of the objective function until the posterior distribution is basically consistent with the real distribution. Compared to grid and random search, it is better to use the last parameter information to adjust the current parameters. In this paper, we use a sequential model-based optimization (SMBO) algorithm [28], which is an advanced Bayesian optimization scheme for tuning the hyperparameters. SMBO usually builds a model to select the optimal hyperparameters in a defined search space, and is particularly common when evaluating the cost function is expensive. The SMBO algorithm is shown in Figure 9.
Fit a new model ℋ. 7 return ℋ Figure 9. The pseudo-code of sequential model-based optimization, where is the objective function, is a cheaper-to-evaluate surrogate model, S is an auxiliary criterion function.

Biofeedback Application
The methods described can be used in a biofeedback application as follows: (a) The accuracy prediction model, gives the user concurrent feedback on hand movement error when working in a real-time scenario; if the result of the prediction is a bad shot, the

Biofeedback Application
The methods described can be used in a biofeedback application as follows: (a) The accuracy prediction model, gives the user concurrent feedback on hand movement error when working in a real-time scenario; if the result of the prediction is a bad shot, the application advises the user to suspend the shot, calm down and try again later; (b) After each shot, the application checks for possible triggering errors in post-processing mode and gives the terminal feedback to the user; (c) At the end of the shooting session, the application calculates the statistical values of precision and accuracy and provides terminal feedback about possible aiming errors.

Experimental Setup
The collection of the shooting results was carried out with the help of 61 subjects, of different ages, genders and shooting skill level. The total sample of respondents included beginners, intermediate, experienced, professional and sport shooters. The majority of shooters were men (43), while 18 shooters were women. Shootings were carried out in five measurement sessions on different dates. Descriptive statistics, including mean value, standard deviation, coefficient of variance (CV), maximum and minimum shot angle, are presented for each session and skill level group in Tables 1 and 2, respectively. The shooters used both hands to support the CZ99 pistol model, as shown in Figure 10a. The placement of the sensor device and the orientation of the sensor coordinate system is shown in Figure 10b. The shooters conducted 193 shooting sessions, with each session comprising 5 shots, so that a total of 965 shots were fired. Shots were fired from distances of 6 m, 10 m and 15 m. The shot result angle was used as a performance measure at all distances. The shot results are divided into two groups, the good and the bad, according to the shot angle threshold of 0.5 • . If the shot angle exceeds the threshold value, the shot result is bad, otherwise it is good. We use 1 and 0 to mark the good and bad results. placement of the sensor device and the orientation of the sensor coordinate system is shown in Figure  10b. The shooters conducted 193 shooting sessions, with each session comprising 5 shots, so that a total of 965 shots were fired. Shots were fired from distances of 6 m, 10 m and 15 m. The shot result angle was used as a performance measure at all distances. The shot results are divided into two groups, the good and the bad, according to the shot angle threshold of 0.5°. If the shot angle exceeds the threshold value, the shot result is bad, otherwise it is good. We use 1 and 0 to mark the good and bad results. The initial feature set includes the standard deviation of the angular velocity signal (StdGx, StdGy, and StdGz), the standard deviation of the angle signal β (StdAx, StdAy, StdAz), the standard deviation of the acceleration signal (StdAccx, StdAccy, StdAccz) , and the angle range (Rx, Ry, and Rz) defined as the difference between the maximum and minimum of β from the sample before 700 for three dimensions. Combining these 12 features with a shot result finally yields a data set with a matrix size of 965 × 13.
After we have eliminated the outliers, we randomly shuffle the remaining data set (720 × 13) and randomly split it into a training set and a test set with a ratio of 8:2. The training set is used to build the RF-based accuracy prediction model of precision shooting, whose parameters are selected by 10-fold cross-validation and Bayesian hyperparameter optimization. The test set is also used for evaluation. The classification algorithms used in this paper are trained with the Scikit learning The initial feature set includes the standard deviation of the angular velocity signal (StdGx, StdGy, and StdGz), the standard deviation of the angle signal β (StdAx, StdAy, StdAz), the standard deviation of the acceleration signal (StdAccx, StdAccy, StdAccz), and the angle range (Rx, Ry, and Rz) defined as the difference between the maximum and minimum of β from the sample before 700 for three dimensions. Combining these 12 features with a shot result finally yields a data set with a matrix size of 965 × 13.
After we have eliminated the outliers, we randomly shuffle the remaining data set (720 × 13) and randomly split it into a training set and a test set with a ratio of 8:2. The training set is used to build the RF-based accuracy prediction model of precision shooting, whose parameters are selected by 10-fold cross-validation and Bayesian hyperparameter optimization. The test set is also used for evaluation. The classification algorithms used in this paper are trained with the Scikit learning package of the Python software. Bayesian hyperparameter optimization is performed using the hyperopt [29] package for Python.

Feature Selection
In order to obtain the key features related to the performance of precision shooting, we carry out a feature selection. First, we calculate the Pearson correlation coefficient [24] for each feature and the corresponding shot result. Table 3 shows that the results of all features are negative, indicating that the greater the values of all features, the worse the shot result. It also proves that the hand movement before the firing phase is strongly related to the accuracy of the precision shooting [10]. Next, we use the initial feature set to build a model based on RF. The scores of the feature importance are shown in Figure 11. The variability of the β[n] of the X-axis achieves the highest score, which proves that the angle feature is strongly related to the shot result and the movement of pistol often appears in the upward direction (X-axis), as mentioned in Section 2. The reason why the variability of β[n] can achieve a higher score than the angle range is that it can better reflect the hand movement of changes before firing. pistol often appears in the upward direction (X-axis), as mentioned in Section 2. The reason why the variability of can achieve a higher score than the angle range is that it can better reflect the hand movement of changes before firing.
Finally, we use the Embedded method which is performed by Select From Model function [30] in the feature selection module of scikit-learn. The selected estimator is RF and the threshold is set to 0.1, so the remaining features set are StdGx, StdAccx, StdAx, StdAz, Rx and Rz .

Algorithm Testing and Selection
To evaluate the performance of the proposed accuracy prediction model optimized with Bayesian hyperparameter optimization, it is compared with a number of baseline models, including logistic regression (LR) [31], support vector machine (SVM) [32], decision tree (DT) [33], k-nearest neighbors (KNN) [34], random forest (RF) [18], Adaboost (AB) [35] and gradient boosting decision tree (GBDT) [36]. Because of an insufficient amount of data, deep learning methods cannot be used. Finally, we use the Embedded method which is performed by Select From Model function [30] in the feature selection module of scikit-learn. The selected estimator is RF and the threshold is set to 0.1, so the remaining features set are StdGx, StdAccx, StdAx, StdAz, Rx and Rz.

Algorithm Testing and Selection
To evaluate the performance of the proposed accuracy prediction model optimized with Bayesian hyperparameter optimization, it is compared with a number of baseline models, including logistic regression (LR) [31], support vector machine (SVM) [32], decision tree (DT) [33], k-nearest neighbors (KNN) [34], random forest (RF) [18], Adaboost (AB) [35] and gradient boosting decision tree (GBDT) [36]. Because of an insufficient amount of data, deep learning methods cannot be used. After the 10-fold cross-validation and Bayesian hyperparameter optimization on the training set, we can determine the optimal parameters for the models. The results of the 10-fold cross-validation of the models with the optimal parameters are shown in Figure 12. After the 10-fold cross-validation and Bayesian hyperparameter optimization on the training set, we can determine the optimal parameters for the models. The results of the 10-fold cross-validation of the models with the optimal parameters are shown in Figure 12. Next, we evaluate the performance of the models with the optimal parameters on the test set. The results are shown in Figure 13, the RF-based accuracy prediction model achieves an accuracy of 91.27%, which outperforms all baseline models. GBDT is the best baseline model with the second highest accuracy of 89.36% and AB is close to GBDT. The result shows that ensemble models Next, we evaluate the performance of the models with the optimal parameters on the test set. The results are shown in Figure 13, the RF-based accuracy prediction model achieves an accuracy of 91.27%, which outperforms all baseline models. GBDT is the best baseline model with the second highest accuracy of 89.36% and AB is close to GBDT. The result shows that ensemble models including RF, AB and GBDT perform better than the individual models including LR, SVM, DT, and KNN. The latter are simple algorithms that are widely used in machine learning, but suffer performance losses due to the overfitting problem. GBDT is easier to learn than RF because GBDT learns the residuals from the previous trees, and RF follows the majority decision rule. Many results are combined to give good generalization and higher accuracy. Next, we evaluate the performance of the models with the optimal parameters on the test set. The results are shown in Figure 13, the RF-based accuracy prediction model achieves an accuracy of 91.27%, which outperforms all baseline models. GBDT is the best baseline model with the second highest accuracy of 89.36% and AB is close to GBDT. The result shows that ensemble models including RF, AB and GBDT perform better than the individual models including LR, SVM, DT, and KNN. The latter are simple algorithms that are widely used in machine learning, but suffer performance losses due to the overfitting problem. GBDT is easier to learn than RF because GBDT learns the residuals from the previous trees, and RF follows the majority decision rule. Many results are combined to give good generalization and higher accuracy. In addition to accuracy, a number of other criteria for classification performance such as precision, recall and score are frequently used. We use these criteria to further evaluate the In addition to accuracy, a number of other criteria for classification performance such as precision, recall and F 1 score are frequently used. We use these criteria to further evaluate the performance of each classification method, the results of which are presented in Table 4. The F 1 score is the harmonic mean of precision and recall. The higher the F 1 score, the more robust the classifier. The ROC (Receiver Operating Characteristics) curve of the proposed RF model is shown in Figure 14, which is one of the important evaluation metrics for testing the performance of the classification model. AUC (Area Under the Curve) represents the separability of the measure. The higher the AUC, the better the performance of this classifier. An excellent model usually has an AUC that is close to 1, which means it has a good measure of separability. On the contrary, a poor model has an AUC that is close to 0, which means that it has the worst performance. As shown in Figure 14, the AUC of the proposed RF model is 0.914, which proves that it is effective. Figure 14, which is one of the important evaluation metrics for testing the performance of the classification model. AUC (Area Under the Curve) represents the separability of the measure. The higher the AUC, the better the performance of this classifier. An excellent model usually has an AUC that is close to 1, which means it has a good measure of separability. On the contrary, a poor model has an AUC that is close to 0, which means that it has the worst performance. As shown in Figure 14, the AUC of the proposed RF model is 0.914, which proves that it is effective.

Conclusions
This paper proposes an accuracy prediction model of precision shooting based on the RF algorithm. Experimental results prove that the proposed model with RF achieves higher prediction accuracy than the other existing reference models. The results also prove that the hand movement is negatively correlated with the accuracy of precision shooting and that the angle feature is more relevant to the shot result than the angular velocity feature. Based on this model, a biofeedback application is designed that operates in concurrent feedback mode and terminal feedback mode. If such a system detects excessive pistol movement and the proposed model provides a prediction that the result will be poor, the system gives the user a concurrent feedback and suggests that the user suspend the current shot and try again later. Such training can lead to a reduced number of poorly executed shots and consequently saving rounds, money and time. In addition, the system can provide terminal feedback after each shot, informing users and trainers of their errors and performance.
There is still a lot of work to be done in the future. The first step is to collect more data, including many more subjects and shooting sessions from all distances (6 m, 10 m and 15 m). With enough data we should be able to achieve higher prediction accuracy, the model of shooting precision could be

Conclusions
This paper proposes an accuracy prediction model of precision shooting based on the RF algorithm. Experimental results prove that the proposed model with RF achieves higher prediction accuracy than the other existing reference models. The results also prove that the hand movement is negatively correlated with the accuracy of precision shooting and that the angle feature is more relevant to the shot result than the angular velocity feature. Based on this model, a biofeedback application is designed that operates in concurrent feedback mode and terminal feedback mode. If such a system detects excessive pistol movement and the proposed model provides a prediction that the result will be poor, the system gives the user a concurrent feedback and suggests that the user suspend the current shot and try again later. Such training can lead to a reduced number of poorly executed shots and consequently saving rounds, money and time. In addition, the system can provide terminal feedback after each shot, informing users and trainers of their errors and performance.
There is still a lot of work to be done in the future. The first step is to collect more data, including many more subjects and shooting sessions from all distances (6 m, 10 m and 15 m). With enough data we should be able to achieve higher prediction accuracy, the model of shooting precision could be improved and deep learning methods could be applied. The next steps we are planning are to test a method for reliable threshold determination for more accurate outlier detection and to build a multi-classification accuracy prediction model of all eleven shot scores.