Fuzzy Comprehensive Evaluation of Pilot Cadets’ Flight Performance Based on G1 Method

Abstract

In this paper, to better evaluate the flight performance of pilot cadets, a flight performance evaluation index system was constructed based on the task of the traffic pattern, the flight training manual, and interviews with instructors. The fuzzy comprehensive evaluation model established by the G1 method was used to evaluate the flight performance of pilot cadets. The flight data of 30 flight cadets were collected to verify the applicability of the fuzzy comprehensive evaluation model. The results showed that the index system established in this paper can meet the requirements of flight performance evaluation. In addition, the fuzzy comprehensive evaluation results were consistent with the evaluation results of experts. Therefore, the system is effective and feasible for the evaluation of pilot cadets’ flight performance through the fuzzy comprehensive evaluation model established by the index system and G1 method.

1. Introduction

Safety is critical to the development of various industries, particularly civil aviation. The civil aviation industry plays an important role in the modern economy. An efficient and safe civil aviation transport network can promote commercial development and economic growth. At present, air travel is one of the most important means of long-distance travel and international communication. Ensuring safe flights is essential to protect the lives and property of passengers. In addition, maintaining a good safety record and implementing proactive safety measures can increase passenger confidence and loyalty, thereby contributing to the long-term growth and sustainability of the business. Aviation safety is, therefore, of vital importance to passengers, airlines, the economy, national security and the international community. A safe, reliable and sustainable civil aviation system can provide people with a better travel experience.

However, in current civil aviation transportation, there have been frequent unsafe flying incidents caused by crew factors. Statistical data show that the proportion of accidents caused by mechanical reasons is decreasing each year [1,2], whilst accidents caused by human factors account for over 70% of the total [3,4]. Pilot errors account for around 60% [5,6,7]. For example, the flight crew of Air China Flight 129 lost situational awareness during the flight and disregarded the height restrictions around the airport [8], while the captain of Asiana Airlines Flight 214 demonstrated poor manual flight skills and heavily relied on the autopilot system during the landing of a wide-body aircraft [9], which led to inadequate flight capabilities causing aviation accidents. These unsafe flight incidents reflect the insufficient knowledge and performance of existing pilots, the mismatch between pilot evaluation systems and industry development, and highlight the importance of conducting evaluations of pilot ability.

In recent research on pilot performance evaluations, researchers and scholars have primarily focused on methods and models. Hebbar and Pashilkar [10] performed a realistic approach and landing flight scenario using a reconfigurable flight simulator, and made subjective and quantitative measurements of pilot performance. Wojcik et al. [11] adapted the Demand Resource Evaluation Scores (DRES) as a metacognitive indicator to assess pilot students’ perceptions during simulated training of a novel manoeuvre. The research found that individual metacognitive evaluations of a stressful aviation manoeuvre might be important for progress in flight performance.

Some researchers also used quick access recorder (QAR) data to evaluate pilots’ flight performance. Wang et al. [12] developed an evaluation method for pilot’s performance during the landing phase based on flight QAR data. A flight landing operation performance evaluation system (FLOPES) was set up based on the evaluation model. Chen and Huang [13] used the Bayesian Network to perform flight crew performance evaluation. Based on an analysis of 484 aviation accidents caused by human factors, a flight crew performance model was constructed. Zhang et al. [14] studied the relationship between QAR data and pilot performance, and put forward one-dimensional convolutional neural networks (1-D CNN) that consider QAR metrics in an integrated manner. Then an approach was developed to achieve the state of pilot performance evaluations.

For this research’s method, studies found that fuzzy comprehensive evaluation refers to the use of fuzzy mathematics methods to assess the possibility of evaluating fuzzy objects influenced by multiple factors based on certain evaluation criteria [15,16,17]. The fuzzy comprehensive evaluation method has been widely applied in the research of evaluation modelling in various fields and has achieved good results [18,19,20].

In weight calculation, subjective weighting methods like the analytic hierarchy process (AHP) method and the order relationship analysis method (G1 method) are commonly used. The AHP method is a simple and practical method for handling complex problems with multiple objectives, criteria and levels. The AHP involves experts assigning values in the range from 1 to 9, or their reciprocals, to express the relative importance between two indicators [21]. After these valuations, a consistency check is performed [22]. However, when dealing with a large number of indicators, errors in assessing their importance can occur, resulting in inconsistent judgements [23], such as A being more important than B, B being more important than C, and C being more important than A. Thus, determining the importance between A and C becomes problematic.

In contrast, the G1 method is an improved index weighting method based on the AHP algorithm [24]. The G1 method avoids the consistency test required by the AHP algorithm by using the non-inferior relationship between indicators, and it has the characteristics of simple operation and wide applicability [25]. The G1 method initially requires experts to rank the importance of indicators according to their judgments and subsequently assign values to express the importance between them. This approach effectively circumvents the drawbacks of the AHP method. In the context of designing a flight trainee skill assessment, this study acknowledges that there may be a substantial number of indicators to consider. If the AHP method was employed, it could lead to consistency check failures. Moreover, repeatedly convening experts for the assessment of indicator importance is not very convenient. Consequently, the G1 method was chosen as the approach to calculating the indicator weights in this research.

Some scholars combined the G1 method with fuzzy comprehensive evaluation method for the construction research of evaluation models [26,27,28]. Thus, applying the G1 method [29] and the fuzzy evaluation method [30] to the assessment of pilot flight performance enables a comprehensive consideration of various factors exhibited by pilot flight performance. Based on the importance and evaluation results of each factor, the original qualitative evaluation can be quantified to better handle the multifactorial, fuzzy, and subjective judgments in the assessment of pilot flight performance.

Meanwhile, traffic pattern flight encompasses all flight processes, from take-off to landing, and flight training for pilot cadets primarily focuses on traffic pattern flight training [31]. Therefore, the detailed contributions and novelty of this paper are as follows:

(1) Based on the characteristics of flight tasks in the traffic pattern and the flight training manual, a flight performance evaluation index system that can accurately assess the flight performance of cadets in traffic pattern flight is derived.

(2) By determining the weight relations of various indicators through the G1 method, an evaluation model of flight performance for pilot cadets using the fuzzy comprehensive evaluation method is established.

(3) The flight data of pilot cadets are obtained through experiments. The evaluation results of flight performance of pilot cadets are calculated by inputting the data into the evaluation model. The results show that the evaluation index system and the method proposed in this paper are suitable for a flight performance evaluation of pilot cadets.

2. Flight Performance Evaluation Index System for Pilot Cadets

The traffic pattern flight is a major course for pilot cadets in their flight training. They can acquire flight performance such as take-off, climb, turn, cruise, descent, and landing from the traffic pattern flight training. Standard thresholds and scoring rules for monitoring indicators are set in advance to evaluate the flight performance. The real-time monitoring of flight data and overshoot situations is performed during traffic pattern flight training for pilot cadets.

2.1. Flight Process of Traffic Pattern

The traffic pattern flight is a flight subject that contains a complete flight process. The flight altitude of this subject generally does not exceed 1500 feet. For example, an airport with a runway magnetic heading of 359° is used for traffic pattern flight training. In this flight training, the maximum flight altitude is 1100 ft, and the pilot cadets are required to complete 11 flight phase tasks in sequence, including take-off, upwind flight, crosswind turn, crosswind flight, downwind turn, downwind flight, base turn, base flight, final turn, final flight and landing. Figure 1 shows the traffic pattern training process.

There are a number of tasks to be completed during the traffic pattern flight. In the take-off phase, the pilot cadet’s control of the aircraft’s status is examined. The pilot cadet needs to control the aircraft to accelerate along the centerline of the runway and to pull the nose up when reaching the rotation speed. During various lateral flights, the pilot cadet needs to manage the aircraft’s track and energy, control the aircraft at the predetermined altitude and heading, and prevent the aircraft’s speed from exceeding the threshold. At each turning point during the flight, the pilot cadet needs to maneuver the aircraft to form a proper turning attitude at the appropriate position, monitor the bank angle so that it does not exceed the limit and promptly correct it before the turn is completed.

The approach and landing phase is the most challenging phase for the pilot cadet. At the end of the final turn, the pilot cadet should position the aircraft on the extended centerline of the runway and parallel to the runway heading. During the approach, the pilot cadet controls the energy, heading, and altitude of the aircraft based on the position of the glide slope and reference points. Before touchdown, the pilot cadet needs to ensure they will land within the touchdown area on the runway. The pilot cadet then controls the aircraft to slow down and taxi along the centerline of the runway.

2.2. Flight Performance Evaluation Index System

The selection of evaluation indicators is a crucial step in the evaluation of flight performance, as it directly affects the evaluation results of the pilot performance. In traffic pattern flight, the pilot cadet follows the training manual for flight operations, and each flight phase has corresponding operational requirements. The pilot cadet’s performance can be intuitively reflected by the relevant flight parameters [32].

The selected indicators should reflect the accuracy of the pilot cadet’s control of the aircraft during the traffic pattern flight, and achieve comprehensive monitoring. By considering the importance of various flight parameters and their practical value in flight performance evaluations, and based on the characteristics of flight tasks in various phases and the training manual, the evaluation indicators were selected.

According to the principles of systematicity, scientific quality, comparability, and practicability [33], the pilot cadet’s flight performance can be reflected by the control of the aircraft’s energy, flight attitude, flight trajectory during different flight phases in the traffic pattern flight. Therefore, based on the pilot cadet’s control over the aircraft’s energy, flight attitude, flight trajectory, a preliminary selection of evaluation indicators for the pilot cadet’s flight performance was made.

In terms of flight energy control, the pilot cadet’s assessment is based on parameters such as altitude, speed, and load factor. In terms of flight attitude control, the pilot cadet’s assessment is based on parameters such as pitch angle and bank angle. For flight trajectory control, the pilot cadet’s assessment is mainly based on trajectory parameters. Thus, for the flight energy control of the pilot cadet, nine evaluation indicators were selected, including altitude deviation in crosswind turn, altitude deviation in downwind, altitude deviation in runway threshold, rotation speed, maximum climb rate in upwind, maximum descent rate in approach, touchdown rate, touchdown overload, maximum flight overload. For the flight attitude control of the pilot, six evaluation indicators were selected, including maximum pitch angle in upwind, maximum pitch angle in final, maximum bank in crosswind turn, maximum bank in downwind turn, maximum bank in turning base, maximum bank in turning final. For the flight trajectory control of the pilot, six evaluation indicators were selected, including track deviation in upwind, track deviation in crosswind, track deviation in downwind, track deviation in approach, touchdown deviation, and taxing deviation.

Based on the characteristics of flight tasks in different phases and the flight training manual, this study provisionally identified 21 evaluation indicators for evaluating the flight performance of pilot cadets. To make the indicators more rational and scientific, the Delphi method was used to interview 168 flight school instructors, airline captains, instructors, experts and academics in the field of pilot training. After two rounds of expert interviews, 90.5% of the initially selected indicators were established as the evaluation criteria for the traffic pattern flight performance of the pilot cadet. The statistics of the experts including numbers and percentages for the interviews are shown in

Table 1. The statistics of experts.

Expert	Number	Percentage
Flight instructor from flight school	28	16.7%
Flight instructor from airlines	25	14.9%
Captain	83	49.4%
Experts and scholars	32	19.0%
Total	168	100%

. The statistics for the Delphi questionnaires are presented in

Table 2. The statistics for the Delphi questionnaires.

Rounds	Specialists	Real Answer Number
First round	168	104
Second round	168	131

.

Through the collaborative analysis of expert opinions, two indicators of flight attitude control, maximum pitch angle in upwind and maximum pitch angle in final, were found to be repetitive, with two indicators of flight energy control, maximum climb rate in upwind and maximum descent rate in approach. The remaining 19 indicators had low repetitiveness and could effectively achieve the evaluation of flight performance for the pilot cadet’s traffic pattern flight throughout the entire process.

Based on the expert interviews, two flight attitude control indicators, maximum pitch angle in upwind and maximum pitch angle in final, were removed. The evaluation indicators for the flight performance of pilot cadets were then optimized. As a result, a total of 19 evaluation indicators in three dimensions were finalized. The flight performance evaluation index system for pilot cadets is shown in Figure 2.

3. Method

The evaluation of pilot cadets’ flight performance is a complex system, and the selection of appropriate evaluation methods and the establishment of a proper evaluation model will lead to a more accurate assessment of pilot cadets’ traffic pattern flight performance. In this study, the G1 method and the fuzzy comprehensive evaluation method are used to establish an evaluation model of pilot cadets’ traffic pattern flight performance and to evaluate their flight performance.

3.1. Subjective Weight Calculation by G1 Method

To divide a complex problem into a hierarchical structure model with interrelation and subordination, the complex problem is decomposed into several indicators, level by level. The weights can be obtained by determining the importance ranking and degree of each indicator in the hierarchical structure model. In this paper, $W_i$ is used to represent the weight of the main criteria indicators’ level and $w_i$ is used to represent the weight of the sub-criteria indicators’ level. The implementation method of the G1 method was as follows:

• Determine the order relationship of indicators. For a set of flight performance evaluation indicators $u_1,u_2,\cdots u_n$, experts ranked the importance of these evaluation indicators from high to low based on their personal understanding and experience. Let this ranking be denoted as $u_1^{}>u_2^{}>\cdots >u_{k-1}^{}>u_k^{}>u_n^{}$.

• Assign importance values to adjacent indicators. $u_{k-1}^{}$ and $u_k^{}$ are two adjacent indicators, and the relative importance between the two indicators can be represented by $r_k$. The assigned values of $r_k$ are shown in

  Table 3. Relative importance assignment for adjacent indicators.

  ${\mathit{r}}_{\mathit{k}}$	Inscription
  1.0	$u_{k-1}$ is as important as $u_k$
  1.1	Intermediate value
  1.2	$u_{k-1}$ is slightly more important than $u_k$
  1.3	Intermediate value
  1.4	$u_{k-1}$ is more important than $u_k$
  1.5	Intermediate value
  1.6	$u_{k-1}$ is much more important than $u_k$
  1.7	Intermediate value
  1.8	$u_{k-1}$ is extremely more important than $u_k$

  .

  $$r_k=\frac{w_{k-1}^{}}{w_k^{}}$$

  (1)

  where $k=n,n-1,\cdots ,2$, $w_{k-1}$ and $w_k$ represent the weights of the $k-1$ indicator and the $k$ indicator.

• Calculate the weight of each indicator. Based on the relative importance value $r_k$ of adjacent indicators, the weight $w_n$ of the evaluation indicator $u_n$ is calculated using Equation (2):

  $$w_n^{}=\frac{1}{1+{\displaystyle \sum _{k=2}^n{\displaystyle \prod _{i=k}^n{r}_i}}}$$

  (2)

• After obtaining the weight $w_n^{}$ of indicator $u_n^{}$, the weight $w_i$ of the remaining indicator $u_i^{}$ can be calculated step by step using Equation (3):

  $$w_{k-1}^{}=r_k{w}_k^{}$$

  (3)

• An expert group determines the weight vector. If there are $h$ experts involved in the process of determining the indicator weights, different experts may provide different order relationships and weight coefficients among the indicators. To solve this problem, Equation (4) is used to calculate the total weight coefficients $w_i^{*}$ for all experts regarding the indicators:

  $$w_i^{*}=\frac{1}{h}{\displaystyle \sum _{q=1}^h{w}_i^q}\left(q=1,2,\cdots ,h;i=1,2,\cdots ,n\right)$$

  (4)

  where $w_i^q$ represents the weight coefficient of the expert $q$ on the evaluation indicator $u_i$.

3.2. Multi-Level Fuzzy Comprehensive Evaluation

3.2.1. Grading Standard

The grading standard of evaluation indicators is the basis of the evaluation system for the flight performance of flight cadets. We divided the grading standard into four levels: excellent, good, medium, and bad. Therefore, the comment set V = { Excellent, Good, Medium, Bad} was established for fuzzy comprehensive evaluation.

By consulting the flight training manual for the maximum threshold requirements of flight control and flight parameters in the traffic pattern flight, the threshold relationship between the flight performance level of the pilot cadet and the corresponding evaluation indicators for each flight performance is derived.

Table 4. Flight performance evaluation threshold standards.

Evaluation Indicator	Standard Value	Deviation Value
		Excellent	Good	Medium	Bad
Altitude deviation in crosswind turn (u1)/ft	800	0	−40	−90	−150
Altitude deviation in downwind (u2)/ft	1100	0	±40	±90	±150
Altitude deviation in runway threshold (u3)/ft	50	±10	±20	±35	±50
Rotation speed (u4)/kt	55	±1	±3	±5	±7
Maximum climb rate in upwind (u5)/(ft/min)	500	0	+40	+90	+150
Maximum descent rate in approach (u6)/(ft/min)	500	0	+40	+90	+150
Touchdown rate (u7)/(ft/min)	40	0	+15	+30	+50
Touchdown overload (u8)/G	1.20	0	+0.15	+0.3	+0.5
Maximum flight overload (u9)/G	1.5	0	+0.3	+0.6	+0.9
Maximum bank in crosswind turn (u10)/(°)	30	0	+4	+9	+15
Maximum bank in downwind turn (u11)/(°)	30	0	+4	+9	+15
Maximum bank in turning base (u12)/(°)	30	0	+4	+9	+15
Maximum bank in turning final (u13)/(°)	30	0	+4	+9	+15
Track deviation in upwind (u14)/(°)	359	±1	±2	±3.5	±5
Track deviation in crosswind (u15)/(°)	89	±1	±3	±6	±10
Track deviation in downwind (u16)/(°)	179	±1	±3	±6	±10
Track deviation in approach (u17)/(°)	359	±1	±2	±3.5	±5
Touchdown deviation (u18)/m	0	±1	±2	±3	±5
Taxing deviation (u19)/m	0	±0.3	±0.7	±1.1	±1.6

shows the relationship between the grading standards of 19 evaluation indicators and the flight performance levels. Each grading standard corresponds to a deviation value, which serves as the basis for the membership function.

3.2.2. Membership Function

The fuzzy membership degrees of each factor are determined using membership functions. Figure 3 shows a graphical representation of the triangular fuzzy membership function Figure 3a and the trapezoidal fuzzy membership function Figure 3b.

As shown in Table 4, the deviation value thresholds corresponding to the different levels of the 19 criteria can be categorized into two scenarios. In the first scenario, the thresholds have a center-symmetric distribution around zero. In the second scenario, the thresholds are spread over a range and spread in both directions. In this context, if the deviation value of flight data falls to the left of the threshold of the Excellent level or to the right of the threshold of the Bad level, it can also be considered Excellent or Bad, with a membership degree of 1. If the deviation value of flight data falls between the thresholds of two levels, this study postulates that the value is a membership with the level to the left and simultaneously with the level to the right. Furthermore, when the deviation value is exactly at the threshold of a particular level threshold, this research assumes that the value has the highest degree of membership with that particular level, with a membership degree of 1. Therefore, the trapezoidal membership function is used to deal with the boundary thresholds and the triangular membership functions for the intermediate levels.

By using a combination of triangular fuzzy membership function and trapezoidal fuzzy membership function, a hybrid membership function is established to evaluate the flight performance of pilot cadets in the traffic pattern flight. The hybrid membership function is shown in Equation (5):

$$A\left(x\right)=\left\{\begin{array}{cc}1& x\le a\\ \begin{array}{c}\frac{b-x}{b-a}\end{array}& a\le x\le b\\ 0& b\le x\end{array}\right\}$$

$$A\left(x\right)=\left\{\begin{array}{cc}\frac{x-a}{b-a}& a\le x\le b\\ \frac{c-x}{c-b}& b\le x\le c\\ 0& \begin{array}{ccc}x\le a& \mathrm{or}& c\le x\end{array}\end{array}\right\}$$

(5)

$$A\left(x\right)=\left\{\begin{array}{cc}0& x\le a\\ \frac{x-a}{b-a}& a\le x\le b\\ 1& b\le x\end{array}\right\}$$

The graphical representation of the hybrid membership function is shown in Figure 4.

3.2.3. Evaluation Matrix

Based on the experimental data that were obtained and the hybrid membership functions, the main criteria element evaluation matrix of each pilot cadet is calculated. Each $R_i$ is considered as a main criteria element and is denoted as $R={\left[R_1,R_2,R_3\right]}^T$. For example, Equation (6) is the evaluation matrix of $R_1$:

$$R_1=\left[\begin{array}{cccc}{r}_{11}& r_{12}& \cdots & r_{1j}\\ r_{21}& r_{22}& \cdots & r_{2j}\\ ⋮& ⋮& \ddots & ⋮\\ r_{i1}& r_{i2}& \cdots & r_{ij}\end{array}\right]\left(i=1,2,\cdots ,9\right);\left(j=1,2,\cdots ,4\right)$$

(6)

where $R_1$ means the evaluation matrix of the main criteria element $U_1$. The $r_{ij}$ in $R_1$ is calculated based on the deviation data obtained from the experiment and the membership function. $i$ refers to the number of indices of $U_1$ and $j$ refers to the number of grading standard in the comment set $V$.

3.2.4. Fuzzy Comprehensive Evaluation

The fuzzy comprehensive evaluation model $B_i$ of a main criteria element is calculated by Equation (7):

$$B_i=W\circ R_i=\left[\begin{array}{cccc}{w}_1& w_2& \cdots & w_i\end{array}\right]\circ \left[\begin{array}{cccc}{r}_{11}& r_{12}& \cdots & r_{1j}\\ r_{21}& r_{22}& \cdots & r_{2j}\\ ⋮& ⋮& \ddots & ⋮\\ r_{i1}& r_{i2}& \cdots & r_{ij}\end{array}\right]=\left[\begin{array}{cccc}{b}_{i1}& b_{i2}& \cdots & b_{ij}\end{array}\right]$$

(7)

where symbol $\circ$ is the fuzzy synthesis operator. $b_j$ represents the membership degree corresponding to each grading. By using the maximum membership rule to deal with $B_i$, the final fuzzy comprehensive evaluation result can be obtained. In addition, to better reflect the impact of various indices in the evaluation of pilot cadet’s flight performance on traffic pattern, the weighted average fuzzy synthesis operator $M\left(\cdot ,+\right)$ was chosen in this paper, and the $M\left(\cdot ,+\right)$ is shown in Equation (8):

$$M\left(\cdot ,+\right)=\left(w_1\cdot r_{1j}\right)+\cdots +\left(w_i\cdot r_{ij}\right)={\displaystyle \sum _{i=1}^n{w}_i}{r}_{ij}$$

(8)

Equation (8) can comprehensively consider the evaluation indicators and comprehensive evaluation results, which makes the fuzzy comprehensive evaluation results more practical. After obtaining the evaluation results $B_i$ of the main criteria element, the evaluation results $B_i$ of the main criteria element need to be normalized, while the ${\displaystyle \sum _{i=1}^n{b}_{{}_i}}\ne 1$ in the calculation result. A new fuzzy comprehensive evaluation matrix $R$ is formed using the normalized results. Then, the product-weighted fuzzy synthesis operator is used to recalculate and obtain the comprehensive evaluation results $B$ of the pilot cadet’s flight performance on the traffic pattern flight.

4. Results and Discussions

The pilot cadets evaluated are selected from a flight school in China that trains approximately 320 pilot cadets per year. In terms of training equipment, the school is equipped with a professional flight simulator training room that can accommodate up to 90 pilot cadets for simulator training. At present, the evaluation of the cadets’ traffic pattern flight performance is mainly based on subjective scoring, which has low accuracy and is highly influenced by the evaluators’ subjectivity. In order to obtain more accurate evaluation results, this study attempts to utilize the flight data collected during the cadets’ training and apply a fuzzy comprehensive evaluation method to objectively evaluate the cadets’ traffic pattern flight performance.

4.1. Experimental Design and Data Pre-Processing

In order to obtain flight data and expert judgements on the training performance of the pilot cadets, which were used to validate the flight performance evaluation model, a flight experiment was designed specifically for the pilot cadets in the traffic pattern flight. The specific experimental procedure is shown in Figure 5.

A total of 30 male student pilots aged between 18 and 20 years old, healthy and well-rested prior to the test, and three flight instructors with more than 5 years’ experience and valid flight certificates, participated in the experiment.

The flight experiment used the Cessna 172R flight simulator. During the experiment, control deviation data were recorded as the pilot cadets operated the flight simulator. After the experiment, the three instructors evaluated the flight performance of the cadets based on the maximum deviation values recorded during the flight experiments, using the evaluation threshold table for flight evaluation indicators. The expert evaluation results were then obtained by calculating the average scores given by the three instructors. The maximum deviation values obtained for each indicator and the expert evaluation results were then verified, and the complete and accurate data were stored. Finally, the dataset of the maximum deviation values and the expert evaluation results for the 30 pilot cadets in the 19 evaluation indicators were obtained. The statistical analysis of the maximum deviation values for the 30 cadets across the 19 evaluation indicators is shown in

Table 5. The statistical analysis of the maximum deviation values for the 30 cadets across the 19 evaluation indicators.

Indicator	Maximum Deviation Value	Indicator	Maximum Deviation Value
$u_1$	−143.24	$u_{11}$	12.52
$u_2$	−153.80	$u_{12}$	12.23
$u_3$	56.04	$u_{13}$	15.81
$u_4$	9.68	$u_{14}$	−5.42
$u_5$	163.40	$u_{15}$	11.87
$u_6$	143.20	$u_{16}$	11.53
$u_7$	47.86	$u_{17}$	5.96
$u_8$	0.40	$u_{18}$	5.98
$u_9$	0.67	$u_{19}$	−1.55
$u_{10}$	11.48	-	-

.

4.2. Calculation of Indicator Weights

The weights of the 19 evaluation indicators were calculated using the G1 method. A total of 168 experts were invited to rate the importance of the indicators, and 131 survey responses were collected. Based on the collected survey results, the weights of each indicator were calculated and are presented in

Table 6. The weights of each indicator.

Element	${\mathit{W}}_{\mathit{i}}$	Indicator	${\mathit{w}}_{\mathit{i}}$
Aircraft Energy Control ($U_1$)	0.4065	$u_1$	0.0279
		$u_2$	0.0316
		$u_3$	0.0863
		$u_4$	0.0437
		$u_5$	0.0886
		$u_6$	0.2081
		$u_7$	0.1684
		$u_8$	0.1296
		$u_9$	0.2158
Aircraft attitude control ($U_2$)	0.2765	$u_{10}$	0.2559
		$u_{11}$	0.2004
		$u_{12}$	0.2228
		$u_{13}$	0.3209
Aircraft path control ($U_3$)	0.3170	$u_{14}$	0.1251
		$u_{15}$	0.0882
		$u_{16}$	0.1295
		$u_{17}$	0.2089
		$u_{18}$	0.2440
		$u_{19}$	0.2043

.

In Table 6, the most important concern is aircraft energy control $W_1=0.4065$, the second is the aircraft path control $W_3=0.3170$, and the least important element is the aircraft attitude control $W_2=0.2765$. The weight of the 19 indicators is $W_1$ = [0.0279, 0.0316, 0.0863, 0.0437, 0.0886, 0.2081, 0.1684, 0.1296, 0.2158], $W_2$ = [0.2559, 0.2004, 0.2228, 0.3209], $W_3$ = [0.1251, 0.0882, 0.1295, 0.2089, 0.2440, 0.2043].

4.3. Flight Performance Evaluation

Based on the calculation results of the weights, the evaluation results of the main criteria element of the pilot cadets’ traffic pattern flight performance are calculated using a fuzzy evaluation model. Taking the pilot cadet 1 as an example, a comprehensive evaluation of this pilot cadet’s traffic pattern flight performance is conducted using a fuzzy comprehensive evaluation method. The maximum deviations of the first pilot cadet in the 19 evaluation indicators are shown in

Table 7. The maximum deviations of the first pilot cadet in the 19 evaluation indicators.

Indicator	Value	Indicator	Value
$u_1$	−15.05	$u_{11}$	−11.13
$u_2$	58.38	$u_{12}$	−8.82
$u_3$	6.93	$u_{13}$	−8.55
$u_4$	1.38	$u_{14}$	−1.20
$u_5$	−135.63	$u_{15}$	0.31
$u_6$	−39.39	$u_{16}$	−4.94
$u_7$	−12.63	$u_{17}$	0.01
$u_8$	0.14	$u_{18}$	−1.87
$u_9$	−0.14	$u_{19}$	−0.15
$u_{10}$	−8.44	-	-

.

Based on the deviations shown in Table 7, the evaluation matrix (see Equations (9)–(11)) of the main criteria elements for the pilot cadet 1 is calculated using the membership function Equation (5).

$$R_1=\left[\begin{array}{cccc}0.6238& 0.3762& 0& 0\\ 0& 0.6324& 0.3676& 0\\ 1& 0& 0& 0\\ 0.81& 0.19& 0& 0\\ 1& 0& 0& 0\\ 1& 0& 0& 0\\ 1& 0& 0& 0\\ 1& 0& 0& 0\\ 1& 0& 0& 0\end{array}\right]$$

(9)

$$R_2=\left[\begin{array}{cccc}1& 0& 0& 0\\ 1& 0& 0& 0\\ 1& 0& 0& 0\\ 1& 0& 0& 0\end{array}\right]$$

(10)

$$R_3=\left[\begin{array}{cccc}0.8& 0.2& 0& 0\\ 1& 0& 0& 0\\ 0& 0.3533& 0.6467& 0\\ 1& 0& 0& 0\\ 0.13& 0.87& 0& 0\\ 1& 0& 0& 0\end{array}\right]$$

(11)

According to Equation (7), the evaluation results $B_i$ of the pilot cadet 1 can be calculated as follows:

$$B_1=\left[\begin{array}{cccc}0.9496& 0.0388& 0.0116& 0\end{array}\right]$$

(12)

$$B_2=\left[\begin{array}{cccc}1& 0& 0& 0\end{array}\right]$$

(13)

$$B_3=\left[\begin{array}{cccc}0.6332& 0.2831& 0.0837& 0\end{array}\right]$$

(14)

Following the principle of maximum membership degree, the evaluation results of the pilot cadet 1 at the level of the main criteria elements are shown in

Table 8. The evaluation results of the pilot cadet 1 at the main criteria elements’ level.

Element	Evaluation Result
$U_1$	Excellent
$U_2$	Excellent
$U_3$	Excellent

.

Since the calculation results show that ${\displaystyle \sum _{i=1}^n{b}_{{}_i}}=1$, it is not necessary to normalize the evaluation results of the main criteria elements when constructing a comprehensive evaluation matrix for the pilot cadet 1. Therefore, the comprehensive evaluation matrix $R$ for flight performance can be obtained.

$$R=\left[\begin{array}{cccc}0.9496& 0.0388& 0.0116& 0\\ 1& 0& 0& 0\\ 0.6332& 0.2831& 0.0837& 0\end{array}\right]$$

(15)

By synthesizing the comprehensive evaluation matrix $R$ with the weight matrix using the weighted average fuzzy synthesis operator, the comprehensive evaluation results of pilot cadet 1 can be obtained.

$$B=\left[\begin{array}{cccc}0.8632& 0.1055& 0.0313& 0\end{array}\right]$$

(16)

Based on the maximum membership degree obtained from the calculation results, the flight performance evaluation result of pilot cadet 1 is excellent.

Then, by repeating the steps of the above fuzzy comprehensive evaluation of flight performance, the evaluation results of traffic pattern flight performance for all 30 pilot cadets can be calculated. The evaluation results of the flight performance for the 30 pilot cadets are shown in

Table 9. The evaluation results of flight performance for the 30 pilot cadets.

Cadet	Result	Cadet	Result
1	Excellent	16	Bad
2	Bad	17	Medium
3	Medium	18	Medium
4	Medium	19	Excellent
5	Excellent	20	Excellent
6	Bad	21	Medium
7	Medium	22	Excellent
8	Excellent	23	Bad
9	Excellent	24	Excellent
10	Bad	25	Medium
11	Medium	26	Good
12	Excellent	27	Good
13	Excellent	28	Excellent
14	Excellent	29	Medium
15	Excellent	30	Excellent

.

4.4. Results Analysis

The evaluation results show that among the 30 pilot cadets that were evaluated, 14 cadets’ evaluation results are excellent, 2 cadets’ evaluation results are good, 9 cadets’ evaluation results are medium, and 5 cadets’ evaluation results are poor. By examining the flight theory learning and simulated flight training performance of the selected pilot cadets during their daily schooling, it is clear that the evaluation results have a strong correlation with their performance. The cadets who are rated as excellent have better theoretical learning performance and a better performance in the hard simulated flight training test while the cadets with poor evaluation results have a weak learning performance and poor training performance. Therefore, it can be verified that the evaluation model of pilot cadets’ traffic pattern flight performance established by the fuzzy comprehensive evaluation method in this paper can evaluate pilot cadets’ handling performance in traffic pattern flights. To verify the reliability of the model evaluation results and their consistency with the expert evaluation results, the model evaluation results were compared with the expert evaluation results.

Figure 6 shows the evaluation results by the experts and the method proposed in this paper. By comparing the model evaluation results with the expert evaluation results, it can be concluded that the model evaluation results obtained in this paper are basically consistent with the expert evaluation results, which proves the accuracy of the modelling method for evaluating the flight performance of pilot cadets in traffic pattern flight. It also confirms the good practical application effect of the fuzzy evaluation method used in this paper. Through the evaluation of pilot cadets’ flight performance, flight instructors and flight schools can clearly identify the differences in aircraft energy control, flight attitude control, and flight trajectory control among different pilot cadets, as well as the overall level of pilot cadets’ flight performance within the group. This provides an important theoretical basis for instructors to develop personalized training plans in a timely and accurate manner.

5. Conclusions

In this paper, an evaluation model of pilot cadets’ flight performance in traffic pattern flight was established using the Delphi method, the G1 method, and the fuzzy comprehensive evaluation method. For the evaluation of pilot cadets’ flight performance, a traffic pattern flight was selected as the flight subject for the evaluation of flight performance. Combined with the flight training manual and expert interviews with flight instructors, a total of 19 flight parameters were selected as evaluation indicators, forming a flight performance evaluation index system for pilot cadets. The weight of the evaluation indicators determined by the G1 method incorporates the judgments of numerous experts on the degree of impact of the indicators. The evaluation model established by the fuzzy comprehensive evaluation method avoids the errors in the evaluation results caused by the subjective factors of the instructors. Then, the flight data of pilot cadets were obtained through simulated flight training experiments. By inputting the flight data into the evaluation model, the flight performance evaluation results were calculated for each pilot caded. The results show the evaluation index system established in this paper accurately reflects the flight performance of the pilot cadets. The evaluation model based on the G1 method and the fuzzy comprehensive evaluation method established in this paper has consistent evaluation results with expert results. Therefore, the evaluation model has good rationality, accuracy, and applicability in evaluating the performance of pilot cadets.

The contribution of this study is to propose an evaluation model that assesses the flight performance of pilot cadets during their traffic pattern flight training. By using this evaluation model to assess their current level of flight proficiency, it could help them to identify weaknesses in their flight operations and provide scientific support for the development of personalized training plans. In addition, the evaluation of pilot cadets’ flight performance will also help to promote safer and more sustainable civil aviation operations in the future.