Wind Shear Operation-Based Competency Assessment Model for Civil Aviation Pilots

: Undesirable meteorological conditions are typical aviation flight safety threats. Although most meteorological radar and flight augmentation computers have avionic system wind shear alarms, the preferred approach is that pilots avoid unsafe wind shear events. Therefore, effective pilot competency evaluations are needed to assess pilots’ abilities to deal with these events. This study developed a wind shear operation competency model that includes observable behavior indicators, sub-task decomposition, and competency check items. An adapted competency model and a quantitative data-driven competency evaluation criteria optimization method were then developed using three-dimensional competency feature modeling, after which wind shear simulation flight training data were used to verify the effectiveness of the proposed method. The competency assessment grades were significantly correlated with the results from experienced examiners with a 93.33% evaluation accuracy. The proposed quantitative data-driven competency assessment method can provide effective pilot competency assessments for a range of aviation meteorological threats.


Introduction
Wind shear, which is a complex weather phenomenon characterized by sudden horizontal or vertical changes in wind speed, seriously threatens flight safety.When an aircraft encounters wind shear, its airspeed, climb rate, angle of attack, lift coefficient, and other parameters can suddenly change, which can seriously affect maneuvering and altitude control, and, if not properly handled, can result in an aviation accident.Low-level wind shear, in particular, which can cause drastic horizontal or vertical changes in wind vector directions below 600 m [1,2], is a significant takeoff and approach risk factor.Low-level wind shear usually occurs at low altitudes because of severe convection, frontal systems, radiative inversions, and special airspace, all of which can result in a drastic loss of aircraft energy [3,4].Therefore, if the crew does not take effective, timely responses, accidents may occur.These low-level wind shear hazards have been defined by the International Civil Aviation Organization (ICAO) as unsafe events that require additional monitoring.The International Air Transport Association (IATA) reported that 31% of all accidents in 2022 were the result of adverse weather conditions, with the most often cited contributing factors (18% of the accidents) being wind shear and thunderstorms [5].In China, the flight quality-monitoring big data platform reported that 21.31% of the red events in 2022 were also wind shear alarm events.Recent typical unsafe wind shear-related events are shown in Table 1.Although flight crews are given meteorological information in advance and through onboard forward-looking radars (predictive wind shear systems, or PWSs), automatic terminal information services (ATISs) and flight augmentation computers (FACs) can Aviation meteorological situations focused on aircraft movement and wind vectors, divide wind shear events into four types: downwind, upwind, side, and vertical [8].Each of these requires different pilot operations.When downwind shear is encountered, there is a sudden decrease in airspeed, which causes the airplane to lose altitude due to the reduction in lift.When encountered during takeoff, the roll is characterized by a slow increase in speed and an inability to leave the ground.If encountered at a high altitude, the airplane loses altitude after passing through the shear line, but the crew has a chance to recover.However, if the plane is at a low altitude or action is delayed, the plane may crash or be forced to touch down outside the runway.When upwind shear is encountered, the airplane may depart from its normal climb or descent path because of sudden increases in airspeed and lift, especially if it is passing through an upwind wind shear area during the approach phase, at which point the crew must deal with the airplane's higher altitude or speed.Lateral wind shear can cause the airplane to slip, undergo a slope change, and deviate from its original intended trajectory, which can cause it to land on a slope, land with yaw, or even run off the runway during landing.Depending on the landing distance available (LDA), a strong tailwind can adversely affect the landing; therefore, if this risk is not quickly dealt with, the landing could be dangerous.Finally, the most dangerous of all wind shear types is vertical wind shear.When an airplane is subjected to a sudden and strong downdraft, it can suddenly fall abnormally and deviate from its original trajectory.In general, because unexpected meteorological conditions, such as wind shear, can threaten flight safety, knowing how to handle and manage such threats is vital to ensuring flight safety.Based on many incident investigations and line operation safety assessment (LOSA) data, the ICAO developed a pilot competency framework to ensure that pilots have the skills to effectively respond to aviation threats.The competencies include the application of knowledge, procedural operations, regulatory compliance, automated flight path management, and manual control flight path management, as well as non-technical competencies, such as communication, leadership and teamwork, problem solving and decision making, situation awareness, and workload management [9].It can be seen that the non-technical abilities are mainly the capacities of Crew Resource Management (CRM).In fact, the ability to handle complex weather threats such as wind shear depends more on their performance of specific behaviors in CRM.
Consequently, the effective assessment of a pilot's abilities to handle these meteorological threats in flight has attracted widespread attention and is also the core of the data-driven evidence-based training and assessment training proposed by the ICAO, the aims of which are to effectively identify pilot skill deficiencies and reduce the potential risk of unsafe events [9,10].Current pilot competency assessment methods comprise two categories.The first is a comprehensive evaluation method that assesses pilot competencies using an assessment index system, an analytical hierarchy process, and fuzzy comprehensive evaluations [11][12][13].However, because the determination of the indicator system weights is somewhat subjective, the assessment consistency is limited.The second type is a data-driven assessment method that exploits flight parameter data or pilot behavior data to construct a prediction model [14,15] and utilizes machine learning algorithms to predict the pilot's competency scores [16].Although this method can improve the prediction accuracy when the machine learning models are combined, it has poorer interpretability in practical applications and is unable to provide a specific pilot competency evaluation standard.In this study, we focused on the three-dimensional competency assessment (3DCA) criteria in the competency assessment framework model.To assess pilot competency in a wind shear scenario, we propose a method that optimizes the assessment criteria using existing assessment data, which allows the assessment criteria to be automatically generated.Competency assessments based on effectively constructing data-driven quantitative assessment criteria using competency feature representation have rarely been examined.
To assess a pilot's competency in managing aviation meteorology threats, we propose an adapted competency model that has observable behaviors (OBs) based on wind shear operations and corresponding competency check items.We developed a data-driven competency pilot assessment optimization model based on three-dimensional competency feature modeling and an optimization algorithm to ensure that the model meets the competency quantification assessment standards.Finally, we conducted experiments using simulated wind shear flight training data to verify the effectiveness of the proposed method.
The remainder of this paper is organized as follows.Section 2 describes the characteristics of different wind shear events and the competency assessment criteria for dealing with these meteorological threats.Section 3 introduces the wind shear operation-based competency model and the competency feature modeling for the competency assessment method.Section 4 details the wind shear simulated flight training evaluation experiments and provides the results of a comprehensive analysis to verify the performance of the proposed data-driven competency assessment method, and Section 5 presents the conclusions.

Problem Statements
In actual flight, wind shear is categorized into two types based on the warning methods: predictive wind shear (PWS) and reactive wind shear (RWS) [6,17].PWS refers to the transmission of coherent impulses by the airborne meteorological radar to a certain wind field range in front of the aircraft and then receiving echoes to detect the wet air wind vectors and their distance; that is, PWS detects an imminent wind shear in the plane's flight path.RWS refers to a wind shear that exists at the aircraft's current location, the presence of which is determined by the aircraft's FAC, which collects system and environmental signals from the flight environment.
Table 2 shows the 131 wind shear flight alert events that occurred in China from January 2020 to June 2020, the data for which were extracted from a Flight Operational Quality Assurance system [18].As can be seen, there are significant distribution differences in the altitude and flight phases for the PWS and RWS alert events.Of the 131 wind shear events shown in Table 2, 86 were RWS events, accounting for 65.65%, and 45 were PWS events, accounting for 34.35%.As there were no correlations between them, more than two-thirds of these actual wind shear events could not be detected in advance by the airborne meteorological radars.The flight phase data for the wind shear events show that there were 48 takeoff and climb phase events, most of which were PWS (34 events), and 83 approach and landing phase events, 72 of which were RWS.The reasons for these differences may be because of the parameters obtained when the wind shear threat factor was being assessed.If the aircraft was already in the wind shear region when the RWS alarm was triggered, the aircraft altitude at the time that the RWS alarm was triggered was the height of the wind shear event.Table 2 also indicates that 80% of the wind shear events occurred below 200 ft, that is, at a low altitude, which, if improperly handled by the pilots, has a stronger possibility of a dangerous crash.
However, regardless of the wind shear threat and the wind shear alarm, pilots must initiate timely and effective operations [19].Pilots can use different methods to identify RWS wind shear events, such as predicting them in advance; observing the wind direction, speed and other characteristics during the phases when the FAC is unable to provide wind shear detection; and paying attention to changes in the avionic system's flight parameters when the FAC provides wind shear detection information.When an RWS warning is triggered, the flight crew must immediately initiate procedural operations.Airborne meteorological radars can detect PWS events; however, pilots need to acknowledge PWS warnings before takeoff, during takeoff, and when climbing and landing, and they must execute the correct operations to avoid entering the wind shear area.The choice of wind shear procedure indicates a pilot's competence in managing these types of threats.Effectively assessing these wind shear pilot competencies, therefore, requires an analysis of the entire threat and error management process and assessments of the pilot's competence behavioral indicators for the various wind shear handling sub-tasks.
At present, the IATA has a 3DCA criteria guideline for assessing pilot competency.The pilot competency assessment needs to measure three competency feature dimensions: the number of OBs demonstrated by pilots (how many); the frequency of the demonstrated OBs (how often); and the threat and error management outcomes (TEM outcomes) [20].Thus, as the 3DCA criteria provide three dimensions of competency assessment characteristics, they are sufficient.However, because the grading standard descriptions are uncertain, examiners vary in their subjective understanding of the assessment guidelines, which can lead to assessment misalignments and inconsistencies.Therefore, a competency behavioral feature-based quantitative assessment model based on real flight training data can be used to evaluate pilot competency effectively when there are undesirable weather conditions, such as wind shear.

Wind Shear Operation-Based Competency Model
The corresponding dispositions for the two wind shear events are different [21,22].RWS wind shear identification requires pilots to pay attention to the airborne avionic system alarms; if the RWS warning is triggered, the pilot flying (PF) should immediately activate the RWS procedure operations while coordinating with the pilot monitoring (PM).If the wind shear warning occurs during the takeoff phase before reaching the takeoff decision V1 speed, it is necessary to perform a rejected takeoff procedure.If wind shear is encountered before V1, there may not be sufficient runway remaining to stop if a rejected takeoff is initiated at V1. Therefore, before making this decision, flight crews must assess whether the remaining runway is sufficient to safely perform the rejected takeoff.If the remaining runway is judged to be insufficient, the takeoff must continue to avoid a runway overrun.If the RWS alarm is triggered after reaching the takeoff decision V1 speed, the VR speed must be sufficient to lift the wheels and takeoff.However, in special cases when the remaining runway is insufficient to accelerate to normal takeoff speed or perform an interrupted takeoff, the pilot may need to take the wheels off below the VR speed to maximize aircraft safety.The checking criteria for handling an RWS encountered after takeoff and departure or during the approach and landing phase are as follows: (1) decisively set the TOGA (Take Off/Go Around) thrust to ensure aircraft performance; (2) maintain the current configuration until recovery to minimize the aircraft altitude and speed loss caused by the wind shear; and (3) adjust the pitch attitude as required.
For PWS, it is first necessary to combine the wind shear information displayed on the PFD (primary flight display), the ND (navigation display), and the audio warnings to confirm the PWS warning trigger level and judge the wind shear authenticity and the corresponding risk, after which the PWS operations are performed.The PWS handling procedures are as follows: (1) if a PWS warning is triggered before takeoff, the crew should delay takeoff or choose the most favorable runway for the takeoff; (2) if a PWS warning is triggered during the takeoff roll, the crew should execute a rejected takeoff; (3) if a PWS warning is triggered during the approach or landing phase, the crew should perform a go-around maneuver to avoid the aircraft being unable to recover due to the low altitude; and (4) if a PWS warning is triggered during flight and the aircraft does not enter the wind shear, the crew should adjust the thrust and speed reference system (SRS) settings and improve the aircraft's performance by changing the wing flaps to counteract the wind shear effect on the aircraft.The wind shear procedural operation check items for the different wind shear modes and flight phases are shown in Table 3, which does not specify specific aircraft types but gives the required capability check items for different representative wind shear conditions.Traditional training wind shear scenario flight assessments are conducted by experienced examiners who rate the check items in Table 3 and subjectively rate the pilots; however, because this assessment is subjective, examiners are not required to consider the wind shear response competencies.Analyses of actual flight wind shear events and decompositions of the key behavioral performances for wind shear handling conditions reveal that wind shear handling requires five main competencies: problem solving and decision making, procedural application, flight path management, workload management, and leadership and teamwork.Problem-solving and decision-making skills refer to the ability to accurately recognize and deal with problems using appropriate decision-making processes.In wind shear conditions, the pilot's problem-solving and decision-making skills need to be centered on the establishment of situational awareness of wind shear timeliness and accurate wind shear handling operations.Procedural application refers to a comprehensive application of knowledge to execute standard operating procedures (SOPs).The flight path management capability refers to the ability to control the aircraft's path using automated or manual controls, which for wind shear events involves maintaining the aircraft's altitude.Workload management refers to effective information processing and the correct prioritization and assignment of tasks.When wind shear is encountered, there is an increase in the quantity of information that needs to be processed, which means that it is important that pilots appropriately process the meteorological and flight state information.Leadership and teamwork refer to the ability to be receptive to suggestions and feedback from the team and ensure effective communication by considering the opinions of others and allowing others to participate in developing relevant plans.When confronted with wind shear events, pilots are expected to actively communicate with and provide feedback to the controllers and collaborate with the crew to overcome the wind shear threat.Although the adapted competency model that we provide for wind shear scenarios is based on expert opinions, the formulation methodology for the proposed competency quantitative assessment criteria is based on a data-driven algorithm.The adapted competency model based on these wind shear operations is shown in Figure 1.

Competency Feature Representation with Wind Shear Operations
The competency model developed in 3.1 is then correlated with the check items or observations, which involves correlating the check items corresponding to the capability items in Figure 1 with the required observable behaviors.During wind shear operation training, an examiner uses a checklist to record the student's completion of the observation items for each subject and then scores the results.Wind shear flight training scoring is usually conducted on a 5-point scale, with higher scores representing higher performances.The obtained wind shear operation-based subject check worksheet data item scores can be denoted as  = ( ) × = ( ,  , ⋯ ,  ) ,  = 1,2, . . ., , where  is the total number of check items,  is the score for the th check item evaluated, and the corresponding  is the full value of the actual score; therefore, the upper bound of the scores for all observations can be denoted as  = ( ,  , ⋯ ,  ) .The TEM outcome wind shear operation performance on the check worksheet can be denoted as  = ( ) × = ( ,  , ⋯ ,  ) ， = 1,2, . . .,  , where  is the total number of observations on the checklist,  is the score for the th observation, the maximum value  is the upper bound of the score for that item, and the observation vector when all check items are taken as full scores is  = ( ,  , ⋯ ,  ) .
After the competency behavioral indicator observation, each check item corresponds to an OB in the wind shear competency model; that is, each check item reflects the performance of a corresponding OB.The association between any check item  and competency  is constructed using the Delphi survey method to solicit flight experts' opinions.In the development process, we obtained suggestions from twenty pilot competency assessment experts about the correlations between the competency check items and the observable behaviors, with the corresponding elements in the correlation matrix being 0 or 1.If more than half of the experts believed there was a correlation, the  was 1, that is, there are correlations between the competence check items and the observable behaviors.Therefore, to assess the wind shear operational performance, a check item and OB association matrix was constructed, notated as , which is determined using Equation (1):

Competency Feature Representation with Wind Shear Operations
The competency model developed in Section 3.1 is then correlated with the check items or observations, which involves correlating the check items corresponding to the capability items in Figure 1 with the required observable behaviors.During wind shear operation training, an examiner uses a checklist to record the student's completion of the observation items for each subject and then scores the results.Wind shear flight training scoring is usually conducted on a 5-point scale, with higher scores representing higher performances.The obtained wind shear operation-based subject check worksheet data item scores can be denoted as A = (a i ) m×1 = (a 1 , a 2 , • • • , a m ) T , i = 1, 2, . . ., m, where m is the total number of check items, a i is the score for the ith check item evaluated, and the corresponding a max i is the full value of the actual score; therefore, the upper bound of the scores for all observations can be denoted as The TEM outcome wind shear operation performance on the check worksheet can be denoted as where k is the total number of observations on the checklist, c i is the score for the ith observation, the maximum value c max i is the upper bound of the score for that item, and the observation vector when all check items are taken as full scores is After the competency behavioral indicator observation, each check item corresponds to an OB in the wind shear competency model; that is, each check item reflects the performance of a corresponding OB.The association between any check item i and competency OB j is constructed using the Delphi survey method to solicit flight experts' opinions.In the development process, we obtained suggestions from twenty pilot competency assessment experts about the correlations between the competency check items and the observable behaviors, with the corresponding elements in the correlation matrix being 0 or 1.If more than half of the experts believed there was a correlation, the b ij was 1, that is, there are correlations between the competence check items and the observable behaviors.Therefore, to assess the wind shear operational performance, a check item and OB association matrix was constructed, notated as B, which is determined using Equation (1): Similarly, the observation term and OB association matrix D for the TEM result worksheet was as shown in Equation ( 2): where n denotes the number of observable behaviors in the wind shear operation competency model, p denotes the number of observable behaviors for threat and error management on the TEM checklist, and b ij and d ij denote the association attributes of the ith observation with the jth OB, with b ij or d ij = 1 indicating that the ith observation contains the jth OB performance; otherwise 0 is taken.
Similarly, to obtain the competency assessment matrix Y based on the TEM outcome for the relevant OB competency composition, the TEM score vector C outcome is multiplied with each correlation matrix D column, as shown in Equation ( 4): In ( 3) and ( 4), a i b ij is the contribution value of the jth OB to the ith required wind shear check item, and c i d ij is the contribution value of the jth OB to the ith TEM outcome.Because each wind shear scenario or sub-task involves multiple competency behavioral indicators, extracting the competency-based quantitative assessment criteria from historical check data requires the modeling of multiple competency feature dimensions [23].Based on the 3DCA criteria, f many , f o f ten , and f tem are the competency behavioral performance feature dimensions for the wind shear events, which are, respectively, how many, how often, and the TEM outcome.As the contribution value for the observed OB demonstration in the competency assessment matrix was higher than 50% of the maximum, it has not yet been assessed.Therefore, the OB observed quantity was modeled using the vector norms in Equation ( 5): where ⊙ is the Hadamard product.
To determine the 'how often' information in the OBs, the actual number of occurrences is expressed as the sum of the non-zero elements in the competency assessment matrix using Equation (6): To determine the TEM outcome, which represents the security management margin for the threat disposal operations, the lowest score of all competency OBs for each check item is taken, with the OB-based threat and error outcomes determined using Equation ( 7):

Wind Shear Competency Assessment Optimization Model
With P denoting the competency level, a five-level grade is usually used for competency assessments.Therefore, the ratings for each check item level rating in P exa are as follows: exemplary, effectively, adequately, minimal acceptable, ineffectively = { 5, 4, 3, 2, 1} .
Because the performance expectations and completion criteria settings for the operational check items are different in different training scenarios, the three-dimensional competency features information is standardized.The maximum values for f many , f o f ten and f tem are then obtained based on the competency assessment matrices shown in Equations ( 8)-( 10): The normalizations of the three-dimensional features from the 3DCA criteria are expressed as follows: , and f tem = f tem f max tem , where f o f ten , f many , The aim is to minimize any expected deviations between the competency assessment score and the competency rating given by the experienced flight examiners.P o f ten , P many , and P tem are the competency scores under different dimensions, and P OB is the competency rating based on the 3DCA criteria competency features modeling; therefore, the competency assessment optimization model can be expressed as shown in ( 11) and ( 12): The original optimization model in (11) to ( 13) is a nonlinear stochastic optimization problem, which is difficult to solve directly.Therefore, the original stochastic optimization objective is replaced by the sample mean deviation using empirical error minimization; that is, the optimization objective is replaced with min∑ L l=1 P OB (l) − P exa (l) .Based on the wind shear operation evaluation checklist data, a vector of scores for the check items is obtained, such as A = (a i ) 27×1 = (4, 3, • • • 3, 2, 4) T .Similarly, based on the TEM outcomes, a score vector C is obtained for the corresponding TEM results.

Subject (Sub) Observation (No) Evaluation Criteria Levels
(2) Construct the correlation matrix for check items and OBs We used a Delphi survey to solicit the opinions of flight experts and instructors and construct the correlation matrix for the twenty-seven check item observations, the six TEM outcome observations, and the eleven wind shear operation competency OBs, with the value of the corresponding elements being 1 if there was an influential relationship, and 0 if there was not.The final obtained observations OB competency correlation matrix constructed from the flight expert information is shown below, where the wind shear operation-based competency OBs demonstrated are concentrated from OB1.1 to OB5.2.
The observation term-OB correlation matrix for the corresponding TEM outcomes was similarly derived.
(3) Build model of competency assessment feature representation and optimization The observation vector A and the correlation matrix B were combined to construct the OB-based competency assessment matrix X as follows: Based on the competency feature representations in the 3DCA criteria-based competency assessment model framework, for each pilot evaluation sample, the how many, how often, and TEM outcome information for the OB performances was determined using Equations ( 5)- (10).For instance, when f many = 5, f often = 363, and f tem = 4, the three-dimensional features after normalization were as follows:

Competency Assessment Model Solution
The historical evaluation checklist data for 200 pilots were divided into a training set (85%) and a test set (15%), after which it was easy to determine the wind shear operationbased competency assessment matrices for all pilots.Based on the assessment method flow, the optimization problem in the previous steps needed to be solved.As the optimization approximation objective for the training set was small (in this case, the training set sample K = N * 85% = 170), the problem was then transformed into twelve non-negative real numbers (α 1 (i) , α 2 (i) , α 3 (i) , α 4 (i) ) for the solution, which made the approximate optimization objective extremely small.
Based on the actual training set score data in the training set, the score distribution from 1 to 5 for the 'how often' characteristic dimension is shown in Figure 2.
Aerospace 2024, 11, x FOR PEER REVIEW 12 of 18 Based on the competency feature representations in the 3DCA criteria-based competency assessment model framework, for each pilot evaluation sample, the how many, how often, and TEM outcome information for the OB performances was determined using Equations ( 5)- (10).For instance, when

Competency Assessment Model Solution
The historical evaluation checklist data for 200 pilots were divided into a training set (85%) and a test set (15%), after which it was easy to determine the wind shear operationbased competency assessment matrices for all pilots.Based on the assessment method flow, the optimization problem in the previous steps needed to be solved.As the optimization approximation objective for the training set was small (in this case, the training set sample K = N * 85% = 170), the problem was then transformed into twelve non-negative real numbers ( α α α α ) for the solution, which made the approximate optimi- zation objective extremely small.Based on the actual training set score data in the training set, the score distribution from 1 to 5 for the 'how often' characteristic dimension is shown in Figure 2. In Figure 2, the 3DCA criteria-based competence behavior indicators show a singlepeak distribution for the 'how often' characteristic dimension grades, of which there were 1 with 1 point, 2 with 2 points, 56 with 3 points, 103 with 4 points, and 8 with 5 points.In Figure 2, the 3DCA criteria-based competence behavior indicators show a singlepeak distribution for the 'how often' characteristic dimension grades, of which there were 1 with 1 point, 2 with 2 points, 56 with 3 points, 103 with 4 points, and 8 with 5 points.
In Figure 3, the 3DCA model of competence assessment indicators still shows a singlepeak distribution for the 'how many' characteristic dimension grades, of which there were 0 with 1 point, 1 with 2 points, 2 with 3 points, 157 with 4 points, and 10 with 5 points.
Aerospace 2024, 11, x FOR PEER REVIEW 13 of 18 In Figure 3, the 3DCA model of competence assessment indicators still shows a single-peak distribution for the 'how many' characteristic dimension grades, of which there were 0 with 1 point, 1 with 2 points, 2 with 3 points, 157 with 4 points, and 10 with 5 points.In Figure 4, the 3DCA model competency behavior indicators for the 'TEM outcome' dimensional grades were more concentrated, for which there were a 1-point count of 0, a 2-point count of 2, a 3-point count of 3, a 4-point count of 156, and a 5-point count of 9.In Figure 4, the 3DCA model competency behavior indicators for the 'TEM outcome' dimensional grades were more concentrated, for which there were a 1-point count of 0, a 2-point count of 2, a 3-point count of 3, a 4-point count of 156, and a 5-point count of 9.
Aerospace 2024, 11, x FOR PEER REVIEW 13 of 18 In Figure 3, the 3DCA model of competence assessment indicators still shows a single-peak distribution for the 'how many' characteristic dimension grades, of which there were 0 with 1 point, 1 with 2 points, 2 with 3 points, 157 with 4 points, and 10 with 5 points.In Figure 4, the 3DCA model competency behavior indicators for the 'TEM outcome' dimensional grades were more concentrated, for which there were a 1-point count of 0, a 2-point count of 2, a 3-point count of 3, a 4-point count of 156, and a 5-point count of 9.The wind shear operation-based competency assessment levels for the test sample data were validated against the real levels given by the flight instructors.Using the obtained competency assessment criteria in Table 5, the assessment results are given in Appendix A, Table A1.The smallest of the three dimensions for each sample is the model assessment level.Consistency analyses often use nonparametric hypothesis testing to calculate the Spearman rank correlation coefficient [24], with a Spearman correlation coefficient range of [−1, +1].A Spearman correlation coefficient less than 0 represents a negative correlation, a Spearman correlation coefficient greater than 0 represents a positive correlation, and a Spearman correlation coefficient equal to 0 represents no correlation; the closer the correlation coefficient is to 0, the weaker the correlation, and the closer the correlation coefficient is to −1 or +1, the stronger the correlation.The correlations between the proposed model-based evaluation levels and the real levels were analyzed (Table 6).Spearman's correlation coefficient between the levels evaluated from the model and the real levels was r = 0.854, p < 0.01 (**), which indicates that the proposed model and real levels were significantly correlated.

Accuracy Analysis
To verify the accuracy of the results for the competency assessment model, the test set evaluation data were used for validation by calculating the f many , f o f ten , and f tem for each pilot, from which the P o f ten , P many , and P tem , based on the competency assessment model criterion thresholds, were obtained by taking the minimum of the three as the pilot's levels.Model assessment level comparisons for the samples and the deviations from the actual levels are shown in Figure 5.
To verify the accuracy of the results for the competency assessment model, the test set evaluation data were used for validation by calculating the  ,  , and  for each pilot, from which the  ,  , and  , based on the competency assessment model criterion thresholds, were obtained by taking the minimum of the three as the pilot's levels.Model assessment level comparisons for the samples and the deviations from the actual levels are shown in Figure 5.The analysis based on the proposed model found that 93.33% of the sample agreed with the actual levels, and the other level deviations were all within 1 level of 6.67%, which was an acceptable deviation range because of the possible subjectivity in the instructor's original checklist observations.Therefore, the proposed competency assessment optimization model based on historical data information provides good results within acceptable rating deviations.

Conclusions
Complex and unexpected aviation meteorological conditions are typical flight safety threats.Therefore, an effective pilot competency evaluation method is needed to enhance pilots' comprehensive skills in dealing with meteorological aviation threats.The 3DCA criteria are a general criterion framework for assessing the competency of the pilots.Because of the uncertain grading descriptions and subjective evaluator judgment, the consistency of the assessment results is not ensured when using the 3DCA criteria directly.In this study, we developed an evaluation method that automatically generates quantitative assessment criteria based on the existing assessment data.With this focus, we constructed an adapted competency model and observable behavior indicators, conducted competency check item and sub-task decomposition, and then, with simulated flight training evaluation data under wind shear conditions, developed a competency assessment criteria optimization model based on three-dimensional competency feature modeling and an algorithm to determine the competency evaluation criteria solution.The consistency and accuracy of the proposed competency evaluation method were validated on actual wind shear simulation flight training sample data.
While pilots with different preferred aircraft types behave differently in wind shear restrictions, they have behavioral performance commonalities.Our approach is based on The analysis based on the proposed model found that 93.33% of the sample agreed with the actual levels, and the other level deviations were all within 1 level of 6.67%, which was an acceptable deviation range because of the possible subjectivity in the instructor's original checklist observations.Therefore, the proposed competency assessment optimization model based on historical data information provides good results within acceptable rating deviations.

Conclusions
Complex and unexpected aviation meteorological conditions are typical flight safety threats.Therefore, an effective pilot competency evaluation method is needed to enhance pilots' comprehensive skills in dealing with meteorological aviation threats.The 3DCA criteria are a general criterion framework for assessing the competency of the pilots.Because of the uncertain grading descriptions and subjective evaluator judgment, the consistency of the assessment results is not ensured when using the 3DCA criteria directly.In this study, we developed an evaluation method that automatically generates quantitative assessment criteria based on the existing assessment data.With this focus, we constructed an adapted competency model and observable behavior indicators, conducted competency check item and sub-task decomposition, and then, with simulated flight training evaluation data under wind shear conditions, developed a competency assessment criteria optimization model based on three-dimensional competency feature modeling and an algorithm to determine the competency evaluation criteria solution.The consistency and accuracy of the proposed competency evaluation method were validated on actual wind shear simulation flight training sample data.
While pilots with different preferred aircraft types behave differently in wind shear restrictions, they have behavioral performance commonalities.Our approach is based on available evaluation data, and the obtained competency quantitative assessment criteria are not affected by the operational wind shear differences for various aircraft types.Meanwhile, because different airlines have their own checklist data for these different training subjects, the quantitative competency assessment criteria need to be generated based on appropriate evaluation data.Therefore, the proposed data-driven competency assessment method can be applied to competency assessment models for other conditions; that is, it can be used for pilot competency assessments under various training stages and to improve pilot abilities to deal with sudden aviation threats.Further, due to the large number of complex conditions that must be simulated for competency assessments, except for the skills that need to be verified in real airplanes, training is generally conducted in simulators.Although the conditions in a real aircraft and a simulator are not the same, the observable behavior data from the simulator training scenarios provide good information about whether a pilot has the competencies to deal with the situation in a real aircraft.However, further verification could be the direction of future research.

Sub 1 : 4 No. 2 : Monitor speed 5 : 5 :
PWS after takeoff lift front wheel No. 1: Set thrust 5: Set and verified TOGA in a timely manner 4: Set TOGA in a timely manner 3: Reached TOGA 2: Did not set TOGA in place 1: Forgot to set TOGA Discovered multilevel change in a timely manner 4: Discovered change in a timely manner 3: Discovered change 2: Did not discover change in time 1Accurately and conditionally maintained configuration 4: Maintained configuration in a timely manner 3: Maintained configuration 2: Failed to maintained configuration 1: Put down landing gear 3 No. 27: Call and report 5: Efficiently and accurately conveyed information 4: Conveyed information in a timely manner 3: Conveyed information 2: Failed to convey information 1: Forgot to convey information 3

Figures 2 -
Figures 2-4 show that the competency feature distributions from the actual flight training sample across all three dimensions are unimodal.The distribution characteristics

Figure 5 .
Figure 5.A comparison of the model assessment levels and the real levels.

Figure 5 .
Figure 5.A comparison of the model assessment levels and the real levels.

Table 1 .
Typical unsafe wind shear incidents in recent years.

Table 2 .
Distribution of wind shear alarm events in China (from January to June 2020).

Table 3 .
Wind shear operation check items.

Table 4 .
Wind shear operation evaluation checklist.

Table A1 .
The scores on three dimensions with the test samples.