# Pilot Behavior Recognition Based on Multi-Modality Fusion Technology Using Physiological Characteristics

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Methods

#### 2.1. Data Construction

#### 2.1.1. Participants

#### 2.1.2. Flight Platform and Task Details

#### 2.1.3. Data Acquisition

#### 2.2. Data Pre-Processing

#### 2.2.1. Normalization

#### 2.2.2. Ectopic and Missing Value Processing

#### 2.2.3. Downsampling and Filtering

#### 2.3. Data Conversion

#### 2.3.1. Time-Domain Features

#### 2.3.2. Frequency-Domain Features

#### 2.3.3. Multi-Modal Features Conversion

#### 2.3.4. Correlation Analysis

#### 2.4. Modeling

Algorithm 1. Weighted voting scheme. |

Input: |

${\mathrm{C}}_{\mathrm{i}}$: Classifier |

${\mathrm{L}}_{\mathrm{j}}$: Labels of Data Set |

m: Ensemble Size |

n: the Number of Labels |

Output: |

the predicted class ${\mathrm{y}}_{\mathrm{j}}$ from a single classifier ${\mathrm{C}}_{\mathrm{i}}$ |

the predicted class y* |

for i = 1: m |

for j = 1: n |

compute ${\mathrm{p}}_{{\mathrm{C}}_{\mathrm{i}}{\mathrm{L}}_{\mathrm{j}}}$, the probability assigned by ${\mathrm{C}}_{\mathrm{i}}$ to class ${\mathrm{L}}_{\mathrm{j}}$ |

$\mathsf{\mu}$ = ${\mathrm{arg}\text{}\mathrm{max}}_{\mathrm{L}=1,\dots ,\mathrm{r}}$ ${\mathrm{p}}_{{\mathrm{C}}_{\mathrm{i}}{\mathrm{L}}_{\mathrm{j}}}$ |

${\mathrm{y}}_{{\mathrm{C}}_{\mathrm{i}}}$ = ${\mathrm{y}}_{\mathsf{\mu}}$ |

for j = 1: n |

$\mathrm{y}\left({\mathrm{L}}_{\mathrm{j}}\right)$ = { i = 1,…,m: ${\mathrm{y}}_{{\mathrm{C}}_{\mathrm{i}}}$ == ${\mathrm{y}}_{{\mathrm{L}}_{\mathrm{j}}}$} |

if $\mathrm{y}\left({\mathrm{L}}_{\mathrm{j}}\right)$ == $\varnothing $ |

${\mathrm{g}}_{{\mathrm{L}}_{\mathrm{j}}}$ = 0 |

else |

for i in $\mathrm{y}\left({\mathrm{L}}_{\mathrm{j}}\right)$ do |

${\mathsf{\omega}}_{{\mathrm{C}}_{\mathrm{i}}}$ = ${\mathrm{max}}_{\mathrm{j}=1,\dots ,\mathrm{n}}$ ${\mathrm{p}}_{{\mathrm{C}}_{\mathrm{i}}{\mathrm{L}}_{\mathrm{j}}}$ |

${\mathrm{g}}_{{\mathrm{L}}_{\mathrm{j}}}$= $\sum}_{\mathrm{i}=1}^{\mathrm{m}}{\mathsf{\omega}}_{{\mathrm{C}}_{\mathrm{i}}$, |

$\mathsf{\mu}$ = ${\mathrm{arg}\text{}\mathrm{max}}_{\mathrm{L}=1,\dots ,\mathrm{r}}$ ${\mathrm{g}}_{{\mathrm{L}}_{\mathrm{j}}}$ |

y* = ${\mathrm{y}}_{\mathsf{\mu}}$ |

return y* |

## 3. Results and Discussion

#### 3.1. Data Measures

#### 3.2. Classification Model Performance

#### 3.3. Classification Improvement

## 4. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## Abbreviations

MFT | Multi-modal Fusion Technology |

ECG | Electro Cardio |

EMG | Myoelectricity |

GSR | Galvanic Skin Response |

RESP | Respiration |

SKT | Skin Temperature |

EEG | Electroencephalogram |

MSE | Mean Square Error |

ROC | Receiver Operating Characteristic Curve |

GBDT | Gradient Boosting Decision Tree |

SVM | Support Vector Machine |

MLP | Multilayer Perceptron |

RRI | RR Interval |

NN | Normal-to-Normal |

SDSD | Standard Deviation of the Difference between Adjacent NN Intervals |

SDNN | Standard Deviation of NN intervals |

RMSSD | Root Mean Square of Successive Differences |

pNN50 | Percentage of Mean R–R Intervals Greater than 50 MS |

pNN20 | Percentage of Mean R–R Intervals Greater than 20 MS |

VLF | Very Low Frequency (0.0033–0.04 Hz) |

ULF | Ultra Low Frequency (0–0.0033 Hz) |

LF | Low Frequency (0.04–0.15 Hz) |

HF | High Frequency (0.15–0.4 Hz) |

LF/HF | Energy Ratio of Low Frequency to High Frequency |

RMS | Root Mean Square |

iEMG | Integral EMG |

## Appendix A

## References

- Greiser, S.; Lantzsch, R.; Wolfram, J.; Wartmann, J.; Müllhäuser, M.; Lüken, T.; Döhler, H.-U.; Peinecke, N. Results of the pilot assistance system “Assisted Low-Level Flight and Landing on Unprepared Landing Sites” obtained with the ACT/FHS research rotorcraft. Aerosp. Sci. Technol.
**2015**, 45, 215–227. [Google Scholar] [CrossRef] - Smaili, M.H.; Rouwhorst, W.F.J.A.; Frost, P. Conceptual Design and Evaluation of Upset-Recovery Systems for Civil Transport Aircraft. J. Aircr.
**2018**, 55, 947–964. [Google Scholar] [CrossRef] - Rydlo, K.; Rzucidlo, P.; Chudy, P. Simulation and prototyping of FCS for sport aircraft. Aircr. Eng. Aerosp. Technol.
**2013**, 85, 475–486. [Google Scholar] [CrossRef] - Xi, P.; Law, A.; Goubran, R.; Shu, C. Pilot Workload Prediction from ECG Using Deep Convolutional Neural Networks. In Proceedings of the IEEE International Symposium on Medical Measurements and Applications (IEEE MeMeA), Istanbul, Turkey, 26–28 June 2019. [Google Scholar]
- Dai, J.; Luo, M.; Hu, W.; Ma, J.; Wen, Z. Developing a fatigue questionnaire for Chinese civil aviation pilots. Int. J. Occup. Saf. Ergon.
**2018**, 26, 37–45. [Google Scholar] [CrossRef] [PubMed] - Hajra, S.G.; Xi, P.; Law, A. A comparison of ECG and EEG metrics for in-flight monitoring of helicopter pilot workload. In Proceedings of the 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Toronto, ON, Canada, 11–14 October 2020. [Google Scholar]
- Law, A.; Jennings, S.; Ellis, K. A Comparison of Control Activity and Heart Rate as Measures of Pilot Workload in a Helicopter Tracking Task. In Proceedings of the Vertical Flight Society 75th Annual Forum, Philadelphia, PA, USA, 16 May 2019. [Google Scholar]
- Feleke, A.G.; Bi, L.; Fei, W. Detection of Driver Emergency Steering Intention Using EMG Signal. In Proceedings of the 2020 IEEE International Conference on Real-time Computing and Robotics (RCAR), Asahikawa, Japan, 28–29 September 2020; pp. 280–285. [Google Scholar]
- Bhardwaj, R.; Natrajan, P.; Balasubramanian, V. Study to Determine the Effectiveness of Deep Learning Classifiers for ECG Based Driver Fatigue Classification. In Proceedings of the 2018 IEEE 13th International Conference on Industrial and Information Systems (ICIIS), Rupnagar, India, 1–2 December 2018. [Google Scholar]
- Masters, M.; Schulte, A. Investigating the Utility of fNIRS to Assess Mental Workload in a Simulated Helicopter Environment. In Proceedings of the 2020 IEEE International Conference on Human-Machine Systems (ICHMS), Rome, Italy, 7–9 September 2020. [Google Scholar]
- Pamela, Z.; Antonio, A.; Riccardo, B.; Alessandro, P.; Roberto, R.; Fabio, F.; Diego, M.; Michela, M. Car Driver’s Sympathetic Reaction Detection Through Electrodermal Activity and Electrocardiogram Measurements. IEEE Trans. Biomed. Eng.
**2020**, 67, 3413–3424. [Google Scholar] [CrossRef] - Šalkevicius, J.; Damaševicius, R.; Maskeliunas, R.; Laukienė, I. Anxiety Level Recognition for Virtual Reality Therapy System Using Physiological Signals. Electronics
**2019**, 8, 1039. [Google Scholar] [CrossRef] - Petrescu, L.; Petrescu, C.; Mitruț, O.; Moise, G.; Moldoveanu, A.; Moldoveanu, F.; Leordeanu, M. Integrating Biosignals Measurement in Virtual Reality Environments for Anxiety Detection. Sensors
**2020**, 20, 7088. [Google Scholar] [CrossRef] - Chen, L.-L.; Zhao, Y.; Ye, P.-F.; Zhang, J.; Zou, J.-Z. Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers. Expert Syst. Appl.
**2017**, 85, 279–291. [Google Scholar] [CrossRef] - Qi, M.S.; Yang, W.J.; Ping, X.; Liu, Z.-J. Driver fatigue Assessment Based on the Feature Fusion and Transfer Learning of EEG and EMG. In Proceedings of the 2018 Chinese Automation Congress (CAC), Changsha, China, 1 November 2018. [Google Scholar]
- He, J.; Zhang, C.Q.; Li, X.Z.; Zhang, D.H. A review of research on multimodal fusion techniques for deep learning. Comput. Eng.
**2020**, 32, 186–189. [Google Scholar] - Wang, D.; Shen, P.; Wang, T.; Xiao, Z. Fatigue Detection of Vehicular Driver through Skin Conductance, Pulse Oximetry and Respiration: A Random Forest Classifier. In Proceedings of the 2017 IEEE 9th International Conference on Communication Software and Networks (ICCSN), Guangzhou, China, 6–8 May 2017. [Google Scholar]
- Hu, J.; Min, J. Automated detection of driver fatigue based on EEG signals using gradient boosting decision tree model. Cogn. Neurodynamics
**2018**, 12, 431–440. [Google Scholar] [CrossRef] - Vargas-Lopez, O.; Perez-Ramirez, C.; Valtierra-Rodriguez, M.; Yanez-Borjas, J.; Amezquita-Sanchez, J. An Explainable Machine Learning Approach Based on Statistical Indexes and SVM for Stress Detection in Automobile Drivers Using Electromyographic Signals. Sensors
**2021**, 21, 3155. [Google Scholar] [CrossRef] [PubMed] - Jiawei, H.; Micheline, K.; Jian, P. Data Mining Concepts and Techniques; Machinery Industry Press: Beijing, China, 2012. [Google Scholar]
- Kim, K.H.; Bang, S.W.; Kim, S.R. Emotion recognition system using short-term monitoring of physiological signals. Med Biol. Eng. Comput.
**2004**, 42, 419–427. [Google Scholar] [CrossRef] [PubMed] - Dan, B. Data Mining for Design and Manufacturing: Methods and Applications; Kluwer Academic Publishers: Alphen am Rhine, The Netherlands, 2001. [Google Scholar]
- Rahman, N. A Taxonomy of Data Mining Problems. Int. J. Bus. Anal.
**2018**, 5, 73–86. [Google Scholar] [CrossRef] - Bai, J.; Shen, L.; Sun, H.; Shen, B. Physiological Informatics: Collection and Analyses of Data from Wearable Sensors and Smartphone for Healthcare. Adv. Exp. Med. Biol.
**2017**, 1028, 17–37. [Google Scholar] - Cheng, K.S.; Chen, Y.S.; Wang, T. Physiological parameters assessment for emotion recognition. In Proceedings of the Biomedical Engineering and Sciences (IECBES), Langkawi, Malaysia, 15 April 2013. [Google Scholar]
- Patel, M.; Lal, S.; Kavanagh, D.; Rossiter, P. Applying neural network analysis on heart rate variability data to assess driver fatigue. Expert Syst. Appl.
**2010**, 38, 7235–7242. [Google Scholar] [CrossRef] - Healey, J.; Picard, R.W. Detecting Stress during Real-World Driving Tasks Using Physiological Sensors. IEEE Trans. Intell. Transp. Syst.
**2005**, 6, 156–166. [Google Scholar] [CrossRef] - Ying, L.; Shan, F. Brief review on physiological and biochemical evaluations of human mental workload. Hum. Factors Ergon. Manuf. Serv. Ind.
**2012**, 22, 177–187. [Google Scholar] - Tian, X.T. Evaluation of Driver Mental Load Based on Heart Rate Variability and Its Application. Master’s Thesis, Northern Polytechnic University, Beijing, China, 2016. [Google Scholar]
- Chieng, T.M.; Yuan, W.H.; Omar, Z. The Study and Comparison between Various Digital Filters for ECG De-noising. In Proceedings of the 2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), Sarawak, Malaysia, 3–6 December 2018. [Google Scholar]
- Aladag, S.; Guven, A.; Ozbek, H.; Dolu, N. A comparison of denoising methods for Electrodermal Activity signals. In Proceedings of the Medical Technologies National Conference, Sarawak, Malaysia, 15–18 October 2015. [Google Scholar]
- Ghalyan, I.; Abouelenin, Z.; Kapila, V. Gaussian Filtering of EMG Signals for Improved Hand Gesture Classification. In Proceedings of the 2018 IEEE Signal Processing in Medicine and Biology Symposium, Philadelphia, PA, USA, 1 December 2018. [Google Scholar]
- Akay, M. Wavelets in biomedical engineering. Ann. Biomed. Eng.
**1995**, 23, 531–542. [Google Scholar] [CrossRef] - Arozi, M.; Caesarendra, W.; Ariyanto, M.; Munadi, M.; Setiawan, J.D.; Glowacz, A. Pattern Recognition of Single-Channel sEMG Signal Using PCA and ANN Method to Classify Nine Hand Movements. Symmetry
**2020**, 12, 541. [Google Scholar] [CrossRef] - Behbahani, S.; Ahmadieh, H.; Rajan, S. Feature Extraction Methods for Electroretinogram Signal Analysis: A Review. IEEE Access
**2021**, 9, 116879–116897. [Google Scholar] [CrossRef] - Purnamasari, P.D.; Martmis, R.; Wijaya, R.R. Stress Detection Application based on Heart Rate Variability (HRV) and K-Nearest Neighbor (KNN). In Proceedings of the 2019 International Conference on Electrical Engineering and Computer Science (ICECOS), Jakarta, Indonesia, 24–25 October 2019. [Google Scholar]
- Esener, I.I. Subspace-based feature extraction on multi-physiological measurements of automobile drivers for distress recognition. Biomed. Signal Process. Control
**2021**, 66, 102504. [Google Scholar] [CrossRef] - Edelmann, D.; Móri, T.F.; Székely, G.J. On relationships between the Pearson and the distance correlation coefficients. Stat. Probab. Lett.
**2021**, 169, 108960. [Google Scholar] [CrossRef] - Liu, X.S. A probabilistic explanation of Pearson′s correlation. Teach. Stat.
**2019**, 41, 115–117. [Google Scholar] [CrossRef] - Yu, J.; Benesty, J.; Huang, G.; Chen, J. Daily Activity Feature Selection in Smart Homes Based on Pearson Correlation Coefficient. Int. J. Electr. Power Energy Syst.
**2019**, 5, 38–40. [Google Scholar] - Mitchell, J. Three machine learning models for the 2019 Solubility Challenge. ADMET DMPK
**2020**, 8, 215–250. [Google Scholar] [CrossRef] [PubMed] - Molan, M.; Molan, G. Maintaining Sustainable Level of Human Performance with Regard to Manifested Actual Availability; Springer: Cham, Switzerland, 2018. [Google Scholar]
- Sirikul, W.; Buawangpong, N.; Sapbamrer, R.; Siviroj, P. Mortality-Risk Prediction Model from Road-Traffic Injury in Drunk Drivers: Machine Learning Approach. Int. J. Environ. Res. Public Health
**2021**, 18, 10540. [Google Scholar] [CrossRef] [PubMed] - Veeramachaneni, K.; Yan, W.; Kai, G.; Osadciw, L. Improving Classifier Fusion Using Particle Swarm Optimization. In Proceedings of the IEEE Symposium on Computational Intelligence in Multicriteria Decision Making, Honolulu, HI, USA; 2007. [Google Scholar]
- Aytu, O.A.; Serdar, K.B.; Hasan, B.B. A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification—ScienceDirect. Expert Syst. Appl.
**2016**, 62, 1–16. [Google Scholar] - Delgado, R. A semi-hard voting combiner scheme to ensemble multi-class probabilistic classifiers. Appl. Intell.
**2022**, 52, 3653–3677. [Google Scholar] [CrossRef] - Dong, X.; Zhiwen, Y.U.; Cao, W.; Shi, Y.; Ma, Q. A survey on ensemble learning. Front. Comput. Sci.
**2019**, 14, 241–258. [Google Scholar] [CrossRef] - Gomes, H.M.; Barddal, J.P.; Enembreck, F.; Bifet, A. A Survey on Ensemble Learning for Data Stream Classification. Acm Comput. Surv.
**2017**, 50, 1–36. [Google Scholar] [CrossRef] - Parvin, A.; MirnabiBaboli, B.; Alinejad-Rokny, E. Proposing a classifier ensemble framework based on classifier selection and decision tree. Eng. Appl. Artif. Intell.
**2015**, 37, 34–42. [Google Scholar] [CrossRef] - Wang, H.; Suh, J.W.; Das, S.R.; Pluta, J.B.; Craige, C.; Yushkevich, P.A. Multi-Atlas Segmentation with Joint Label Fusion. IEEE Trans. Pattern Anal. Mach. Intell.
**2013**, 35, 611–623. [Google Scholar] [CrossRef] - Pawiak, P.; Acharya, U. Novel deep genetic ensemble of classifiers for arrhythmia detection using ECG signals. Neural Comput. Appl.
**2020**, 32, 11137–11161. [Google Scholar] [CrossRef] - Shen, H.; Lin, Y.; Tian, Q.; Xu, K.; Jiao, J. A comparison of multiple classifier combinations using different voting-weights for remote sensing image classification. Int. J. Remote Sens.
**2018**, 39, 3705–3722. [Google Scholar] [CrossRef] - Bonett, D.G.; Wright, T.A. Sample size requirements for estimating pearson, kendall and spearman correlations. Psychometrika
**2000**, 65, 23–28. [Google Scholar] [CrossRef] - Olbrich, S.; Sander, C.; Matschinger, H.; Mergl, R.; Hegerl, U. Brain and body: Associations between EEG vigilance and the autonomous nervous system activity during rest. J. Psychophysiol.
**2011**, 25, 190. [Google Scholar] [CrossRef] - Wanyan, X.; Zhuang, D.; Zhang, H. Improving pilot mental workload evaluation with combined measures. Bio-Medical Mater. Eng.
**2014**, 24, 2283–2290. [Google Scholar] [CrossRef] - Machado, A.V.; Pereira, M.G.; Souza, G.G.; Xavier, M.; Aguiar, C.; de Oliveira, L.; Mocaiber, I. Association between distinct coping styles and heart rate variability changes to an acute psychosocial stress task. Sci. Rep.
**2021**, 11, 24025. [Google Scholar] [CrossRef] - Castaldo, R.; Montesinos, L.; Wan, T.S.; Serban, A.; Massaro, S.; Pecchia, L. Heart Rate Variability Analysis and Performance during a Repeated Mental Workload Task. In Proceedings of the EMBEC & NBC 2017, Tampere, Finland, 11–15 June 2017. [Google Scholar]
- Satti, A.T.; Kim, J.; Yi, E.; Cho, H.Y.; Cho, S. Microneedle Array Electrode-Based Wearable EMG System for Detection of Driver Drowsiness through Steering Wheel Grip. Sensors
**2021**, 21, 5091. [Google Scholar] [CrossRef] - Wrle, J.; Metz, B.; Thiele, C.; Weller, G. Detecting sleep in drivers during highly automated driving: The potential of physiological parameters. IET Intell. Transp. Syst.
**2019**, 13, 1241–1248. [Google Scholar] [CrossRef] - Skibniewski, F.W.; Dziuda, Ł.; Baran, P.M.; Krej, M.K.; Guzowski, S.; Piotrowski, M.A.; Truszczyński, O.E. Preliminary Results of the LF/HF Ratio as an Indicator for Estimating Difficulty Level of Flight Tasks. Aerosp. Med. Hum. Perform.
**2015**, 86, 518–523. [Google Scholar] [CrossRef] [PubMed] - Rosa, E.; Lyskov, E.; Grönkvist, M.; Kölegård, R.; Dahlström, N.; Knez, I.; Ljung, R.; Willander, J. Cognitive performance, fatigue, emotional and physiological strains in simulated long-duration flight missions. Mil. Psychol.
**2022**, 34, 224–236. [Google Scholar] [CrossRef] - Borghini, G.; Di Flumeri, G.; Aricò, P.; Sciaraffa, N.; Bonelli, S.; Ragosta, M.; Tomasello, P.; Drogoul, F.; Turhan, U.; Acikel, B.; et al. A multimodal and signals fusion approach for assessing the impact of stressful events on Air Traffic Controllers. Sci. Rep.
**2020**, 10, 8600. [Google Scholar] [CrossRef] [PubMed] - Koskimaki, H. Avoiding Bias in Classification Accuracy—A Case study for Activity Recognition. In Proceedings of the 2015 IEEE Symposium Series on Computational Intelligence, Cape Town, South Africa, 7–10 December 2015; pp. 301–306. [Google Scholar]

Category | Aim | Methods |
---|---|---|

Data cleaning | Handling of anomalies in data values | Missing value processing (abandon/replacement) |

Ectopic values processing | ||

Outlier and noise handling | ||

Data integration | Increase sample data size | Combining multiple data sets into a single data set |

Data standardization | Scales the sample values to a specified range | Discretization |

Dualization | ||

Normalization (min–max, z-score) | ||

Function transformation |

ECG | GSR | EMG | RESP | SKT | |
---|---|---|---|---|---|

Noise reduction | Wavelet | Gaussian | Wavelet | Wavelet | Sliding average |

High pass | 1 Hz | / | 5 Hz | / | 5 Hz |

Band stop | 50 Hz | 50 Hz | 50 Hz | 50 Hz | 50 Hz |

Low pass | 40 Hz | 5 Hz | 500 Hz | 20 Hz | 200 Hz |

Parameters | Description |
---|---|

Mean | $\overline{x}=\frac{i}{{N}_{s}}{\displaystyle \sum}_{i=1}^{{N}_{s}}x\left(i\right)$ |

Standard Deviation | ${F}_{s}=(\frac{1}{{N}_{s}}\sqrt{{\displaystyle \sum}_{i=1}^{{N}_{s}}{\left(x\left(i\right)-\overline{x}\right)}^{2}}$ |

Root Mean Square (RMS) | ${F}_{s}=\sqrt{\frac{1}{{N}_{s}}{\displaystyle \sum}_{i=1}^{{N}_{s}}{\left(x\left(i\right)\right)}^{2}}$ |

Parameters | Description |
---|---|

Power | Power in the frequency band |

Median Frequency | ${{\displaystyle \int}}_{0}^{MF}P\left(\omega \right)d\omega ={{\displaystyle \int}}_{MF}^{\infty}P\left(\omega \right)d\omega =\frac{1}{2}{{\displaystyle \int}}_{0}^{\infty}P\left(\omega \right)d\omega $ |

Mean Power Frequency | $MPF=\frac{{{\displaystyle \int}}_{0}^{\infty}\mathsf{\omega}P\left(\omega \right)d\omega}{{{\displaystyle \int}}_{0}^{\infty}P\left(\omega \right)d\omega}$ |

ECG | GSR | EMG | RESP | SKT | |
---|---|---|---|---|---|

ECG value | HR value | SC value | EMG value | RESP value | SKT value |

SDSD | NN | mean | standard deviation | standard deviation | |

SDNN | RMSSD | standard deviation | RMS | power | |

pNN50 | pNN20 | Integral EMG | mean | ||

VLF | ULF | median frequency | |||

LF | HF | mean power frequency | |||

LF/HF | mean |

Model | Mean Accuracy | Lowest Accuracy | MSE |
---|---|---|---|

Logistic Regression | 0.430 | 0.417 | 4.0109 |

Naive Byes | 0.362 | 0.339 | 4.1813 |

AdaBoost | 0.373 | 0.355 | 4.0822 |

SVM | 0.441 | 0.432 | 3.9632 |

K-Nearest Neighbor | 0.952 | 0.947 | 0.2989 |

ETC | 0.965 | 0.962 | 0.1765 |

DTC | 0.964 | 0.960 | 0.1733 |

GBC | 0.968 | 0.965 | 0.1755 |

XGBC | 0.967 | 0.962 | 0.1847 |

Stall | Somersault | Takeoff | Turn and Hover | Level Flight | Roll | |
---|---|---|---|---|---|---|

Subject 1 | 10 | 8 | 6 | 6 | 4 | 3 |

Subject 2 | 7 | 6 | 8 | 5 | 3 | 4 |

Subject 3 | 8 | 9 | 5 | 6 | 4 | 4 |

Subject 4 | 9 | 8 | 3 | 5 | 3 | 3 |

Subject 5 | 8 | 5 | 2 | 3 | 2 | 2 |

Subject 6 | 8 | 4 | 1 | 4 | 1 | 3 |

Subject 7 | 6 | 8 | 1 | 5 | 2 | 2 |

Subject 8 | 3 | 7 | 2 | 3 | 3 | 6 |

Subject 9 | 8 | 9 | 5 | 6 | 5 | 5 |

Subject 10 | 7 | 8 | 3 | 6 | 4 | 4 |

Subject 11 | 8 | 7 | 3 | 5 | 4 | 3 |

Subject 12 | 5 | 7 | 2 | 6 | 5 | 2 |

Subject 13 | 7 | 9 | 3 | 7 | 5 | 3 |

Subject 14 | 8 | 8 | 4 | 5 | 4 | 3 |

Mean | 7.29 | 7.36 | 3.43 | 5.14 | 3.50 | 3.36 |

ETC | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.96 | 0.91 | 0.97 | 1.00 | 0.99 | 0.97 |

recall | 0.95 | 0.92 | 0.98 | 1.00 | 0.99 | 1.00 |

F1 | 0.96 | 0.91 | 0.97 | 1.00 | 0.99 | 0.98 |

average accuracy for 10-fold CV | 0.9652 | MSE for 10-fold CV | 0.1765 | |||

average accuracy for LOO CV | 0.7817 | MSE for LOO CV | 1.1199 | |||

DTC | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.96 | 0.91 | 0.97 | 1.00 | 0.99 | 0.97 |

recall | 0.95 | 0.92 | 0.97 | 1.00 | 0.99 | 1.00 |

F1 | 0.96 | 0.91 | 0.97 | 1.00 | 0.99 | 0.98 |

average accuracy for 10-fold CV | 0.9642 | MSE for 10-fold CV | 0.1733 | |||

average accuracy for LOO CV | 0.7062 | MSE for LOO CV | 1.5760 | |||

GBC | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.96 | 0.91 | 0.97 | 1.00 | 0.99 | 0.97 |

recall | 0.95 | 0.93 | 0.98 | 1.00 | 0.99 | 0.93 |

F1 | 0.96 | 0.92 | 0.98 | 1.00 | 0.99 | 0.95 |

average accuracy for 10-fold CV | 0.9677 | MSE for 10-fold CV | 0.1755 | |||

average accuracy for LOOCV | 0.7064 | MSE for LOO CV | 1.4856 | |||

XGBC | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.96 | 0.91 | 0.97 | 1.00 | 0.99 | 0.97 |

recall | 0.96 | 0.92 | 0.98 | 1.00 | 0.99 | 1.00 |

F1 | 0.96 | 0.92 | 0.98 | 1.00 | 0.99 | 0.98 |

average accuracy for 10-fold CV | 0.9674 | MSE for 10-fold CV | 0.1847 | |||

average accuracy for LOO CV | 0.7473 | MSE for LOO CV | 1.4894 | |||

Proposed Model | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.96 | 0.93 | 0.97 | 1.00 | 0.99 | 0.97 |

recall | 0.96 | 0.92 | 0.98 | 1.00 | 0.99 | 1.00 |

F1 | 0.96 | 0.93 | 0.98 | 1.00 | 0.99 | 0.98 |

average accuracy for 10-fold CV | 0.9693 | MSE for 10-fold CV | 0.1693 | |||

average accuracy for LOO CV | 0.8094 | MSE for LOO CV | 1.0606 |

ETC | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.97 | 0.95 | 0.99 | 1.00 | 1.00 | 1.00 |

recall | 0.97 | 0.96 | 0.99 | 1.00 | 0.99 | 0.91 |

F1 | 0.97 | 0.96 | 0.99 | 1.00 | 0.99 | 0.95 |

average accuracy for 10-fold CV | 0.9792 | MSE for 10-fold CV | 0.1093 | |||

average accuracy for LOO CV | 0.7889 | MSE for LOO CV | 1.1933 | |||

DTC | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.96 | 0.92 | 0.99 | 1.00 | 0.99 | 0.97 |

recall | 0.96 | 0.94 | 0.98 | 1.00 | 0.99 | 0.88 |

F1 | 0.96 | 0.93 | 0.98 | 1.00 | 0.99 | 0.92 |

average accuracy for 10-fold CV | 0.9728 | MSE for 10-fold CV | 0.1579 | |||

average accuracy for LOO CV | 0.7301 | MSE for LOO CV | 1.3759 | |||

GBC | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.98 | 0.93 | 0.99 | 1.00 | 0.99 | 0.97 |

recall | 0.97 | 0.96 | 0.98 | 1.00 | 0.99 | 0.90 |

F1 | 0.97 | 0.94 | 0.99 | 1.00 | 0.99 | 0.93 |

average accuracy for 10-fold CV | 0.9726 | MSE for 10-fold CV | 0.1266 | |||

average accuracy for LOO CV | 0.7306 | MSE for LOO CV | 1.2341 | |||

XGBC | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.96 | 0.93 | 0.98 | 1.00 | 1.00 | 1.00 |

recall | 0.97 | 0.94 | 0.98 | 1.00 | 0.99 | 0.94 |

F1 | 0.97 | 0.94 | 0.98 | 1.00 | 0.99 | 0.97 |

average accuracy for 10-fold CV | 0.9741 | MSE for 10-fold CV | 0.1151 | |||

average accuracy for LOO CV | 0.7697 | MSE for LOO CV | 1.3256 | |||

Proposed Model | ||||||

level | roll | turn and hover | takeoff | somersault | stall | |

precision | 0.98 | 0.95 | 0.99 | 1.00 | 1.00 | 0.97 |

recall | 0.98 | 0.96 | 0.98 | 1.00 | 1.00 | 0.93 |

F1 | 0.98 | 0.95 | 0.99 | 1.00 | 1.00 | 0.95 |

average accuracy for 10-fold CV | 0.9815 | MSE for 10-fold CV | 0.1026 | |||

average accuracy for LOO CV | 0.8273 | MSE for LOO CV | 0.9601 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Li, Y.; Li, K.; Wang, S.; Chen, X.; Wen, D.
Pilot Behavior Recognition Based on Multi-Modality Fusion Technology Using Physiological Characteristics. *Biosensors* **2022**, *12*, 404.
https://doi.org/10.3390/bios12060404

**AMA Style**

Li Y, Li K, Wang S, Chen X, Wen D.
Pilot Behavior Recognition Based on Multi-Modality Fusion Technology Using Physiological Characteristics. *Biosensors*. 2022; 12(6):404.
https://doi.org/10.3390/bios12060404

**Chicago/Turabian Style**

Li, Yuhan, Ke Li, Shaofan Wang, Xiaodan Chen, and Dongsheng Wen.
2022. "Pilot Behavior Recognition Based on Multi-Modality Fusion Technology Using Physiological Characteristics" *Biosensors* 12, no. 6: 404.
https://doi.org/10.3390/bios12060404