Civil Aircraft Landing Attitude Ultra-Limit Warning System Based on mRMR-LSTM

Abstract

To achieve the forward movement of the aircraft landing attitude ultra-limit, this paper builds a deep learning-based aircraft landing attitude warning system. The early warning system includes four modules: data pretreatment, feature dimensionality reduction, prediction, and judgment. Subsequently, through data pretreatment methods such as data cleaning, frequency normalization, data standardization, and feature classification, the experimental dataset is transformed into a form recognizable by machine learning algorithms and neural network models. The necessary feature parameters are extracted to form a deep learning training dataset. Then, the Max-Relevance and Min-Redundancy algorithm was applied to screen the QAR (Quick Access Recorder) parameters with the highest correlation with the predictor variables, and the LSTM network model was established to predict the pitch and roll angles of the aircraft landing, respectively. Evaluation metrics are used to determine the optimal model parameters. Finally, the confusion matrix is introduced to test the prediction effect of the model, and through the secondary indicators of the confusion matrix, the prediction accuracy of the established landing attitude warning system is 94.83% for the pitch angle and 91.18% for the roll angle. It also provides pilots with a 5 s time margin to avoid risks. The system can effectively issue early warnings for ultra-limit landing attitude events and, based on the prediction results, identify the types of risks.

1. Introduction

A vital component of the civil aviation sector is the assurance of aviation safety. In our previous work, we compiled data on civil aviation accidents in the United States from 1993 to 2023. The data were primarily derived from accident reports published by the National Transportation Safety Board (NTSB) and the Federal Aviation Administration (FAA). Over the span of three decades, a total of 2572 accidents occurred in the U.S. civil aviation sector, underscoring the critical importance of aviation safety. When the entire flight journey is categorized by flight phases, as illustrated in Figure 1, 31% of the total accidents occurred during the landing phase, which accounts for only 1% of the total flight duration. Among these incidents, 19.91% and 31.09% of accidents for aircraft under Parts 121 and 135 regulations, respectively, occurred during the landing phase. This indicates that the landing phase of aircraft is prone to various accidents and poses a higher risk of potential hazards. Therefore, conducting in-depth analysis and research on this flight phase is essential to ensuring safe and reliable aircraft landings.

To address the landing safety issue, most of the current studies start from the flight unsafe events such as hard landing, long landing, aircraft tail strike, and runway excursions or exceedances. For example, Wang Lei et al. established a quantitative evaluation model of hard landing risk based on the Pauta criterion to calculate the possibility and severity of hard landing with the distribution function of the vertical load at the grounding moment of the aircraft [1]. Sun Ruisan et al. constructed a quantitative evaluation model of flight ultra-limits by using QAR ultra-limits as an evaluation index and obtaining the weights of the indexes based on the grey clustering principle of deviation maximization [2]. Based on the Gaussian mixture model, Li et al. analyzed a large number of flight parameters of QAR data samples and compared the single abnormal flight data with the clustering results to mine the abnormal patterns in flight operation [3,4,5,6,7]. Ayra et al., based on the Bayesian network, modeled and analyzed runway ultra-limits and excavated the high-risk factors leading to the accidents [7]. Sun H and J Xie et al. combined QAR data, geographic data, and meteorological data to explore the spatial and temporal distribution characteristics of unstable approach events and used Pearson’s correlation coefficient and geographically weighted correlation coefficient to mine the causes of unstable approach events at different levels and scales [8,9]. Xu Li et al. transformed the classical K-means clustering into a semi-supervised algorithm and applied it to curve clustering and constructed a re-landing risk assessment model based on the clustering results. And applied it to curve clustering and constructed a hard landing risk assessment model “CurveCluster+” based on the clustering results. It exhibits good interpretability and has a 92.99% recognition accuracy for hard landing events [10,11].

Aircraft wiped runway incident is one of the frequently occurring unsafe events, belonging to the serious signs of transportation aviation, the essence of this event is that the landing attitude of the aircraft is over the limit, resulting in the tail or wingtip of the aircraft and the runway at the time of grounding and other parts of the runway cuts, as can be seen in Figure 2 and Figure 3 shown in detail.

Aircraft wiped runway incident not only causes damage to the aircraft structure, affects flight performance and maneuverability, and seriously threatens flight safety, but also causes damage to the runway and other ground facilities, which increases the maintenance cost, and may also affect the landing safety of other aircraft. At present, scholars’ research on the aircraft landing attitude ultra-limit is mainly divided into two categories:

The first type of research is mainly based on the theories of flight principles, mechanics, and automatic control, and uses numerical simulation and simulation analysis to analyze the aircraft landing attitude ultra-limit events. For example, Chatterji et al. used a Kalman filter to integrate the video images acquired by an onboard camera and the attitude angle information sensed by a roll sensor to invert the relative position and pitch attitude of the aircraft to the runway during landing [12]. Ricky et al. developed a visual warning system for the tail wipe event of an aircraft landing and taking off, which can prompt the pilot with a warning symbol in the cockpit main flight display [13]. Chester used numerical simulation to analyze the vertical loads on the landing gear of a landing aircraft under different attitude conditions and found that the landing impact strength of the front landing gear is very sensitive to the initial pitch angle and pitch inertia value of the aircraft [14]. Wen et al. analyzed the effect of the damping force of the landing gear of a carrier aircraft on the landing attitude and established a dynamic model of the landing of a carrier aircraft on a moving deck, which was used to simulate the landing of carrier aircraft in relatively complex attitudes [15]. Zhang et al. designed a method combining self-impedance control and optimal control to control the landing attitude of aircraft under turbulence, gusts of wind, and modeling errors, and simulated based on the longitudinal nonlinear model of Boeing 747, which verified the control effect of the system [16]. Zhang et al. analyzed the effects of landing gear damping force on the landing attitude of aircraft and established a dynamic model for the landing on moving deck, which was used to simulate the landing of carrier aircraft in relatively complex attitude, Zhang et al. analyzed the impact of landing gear damping force on the landing attitude of aircraft, and established a dynamic model for the landing of aircraft on moving deck. Zhang Yishan et al. analyzed qualitatively the risk factors that can easily lead to the aircraft takeoff and landing tail wipe from the perspective of flight principles [17]. Su Ze introduced the modal superposition method and ellipsoidal joint probability distribution and gave the calculation method of the landing impact load of the aircraft by considering the influence of the landing speed and the attitude [18]. Li C et al. constructed the geometrical constraints model of the aircraft tail wipe for takeoff and landing according to the geometrical shape of the aircraft to achieve the risk identification for the aircraft tail strike event. The risk identification of the tail scratch event is realized at [19]. Ioannis Katsidimas et al. developed both static and dynamic algorithms to assess the risk of runway-excursion exceedances during landing [20].

The second type of research focuses more on data, which analyzes and mines the flight record data to study the aircraft’s landing attitude. For example, based on QAR data, Wang Lei et al. used Monte Carlo simulation to obtain the distribution of pitch angle and the risk prediction curves of tail wiping for different fleets of aircrafts during landing, and established a prediction model of aircraft landing pitch angle, which can accurately predict the probability of exceeding the limit of the fleet’s pitch angle, and provide effective technical support for the management of tail wiping events [21]. Yang Xingyue et al. extracted the pilots’ operation parameters from QAR data and established a human-caused operation analysis model of tail wiping in the landing phase of aircraft, which realized the analysis of tail wiping events in the landing phase of aircraft. The human factor operation analysis model of aircraft tail wipe during the landing stage, which realizes the visualization and analysis of human factors in aircraft tail strike events [22]. Wang Lei et al. used a Bayesian network analysis model to analyze the causal factors of landing attitude, load, distance and other overlimit events, and then determined the ideal flight parameters to reduce the risk of landing overlimit [23]. Zhang N et al. investigated the aircraft tail strike event from the perspective of human factors and verified it on the basis of the QAR data, and the results showed that roughly maneuvering the pilot’s stick before grounding and continually over-manipulating the pilot’s stick after grounding would lead to excessive pitch attitude [24]. Using non-exceedance QAR data, Lu Fei et al. applied a random-forest algorithm to construct a predictive model for tail strike events [25].

A summary of the current research status reveals that studies on Aircraft landing attitude ultra-limit have the following three limitations:

(1) The landing attitude of an aircraft consists of two dimensions: pitch angle and roll angle, but most of the current studies focus on the pitch attitude, especially on the analysis of tail strike events, and there are fewer discussions and research on the roll attitude. Tail strike events are only one of the serious consequences of aircraft landing attitude ultra-limits, while roll attitude ultra-limits can also lead to serious accidents such as wingtip and engine wipe, so it is necessary to conduct in-depth research on both attitude dimensions at the same time.

(2) At present, QAR data are widely used in the analysis of flight unsafe events such as hard landing, long landing, runway excursions or exceedances, unstable approach, and so on. Compared with the above unsafe events, the current research on landing attitude ultra-limit is mostly based on the analysis of mechanics, control, and flight principles. Studies focusing on QAR-based data mining for this topic remain limited. Most existing research results are limited to post-event research and qualitative analysis, and there are few results on early warning of landing attitude ultra-limit and quantitative classification of ultra-limit risk.

(3) With the development of big data mining and artificial intelligence technology, the research on flight unsafe events based on QAR data has been increasing. However, several limitations persist in this field. First, QAR data are the sensitive data of airlines, which involves the company’s safety management information, and thus is not shared within the civil aviation community. Second, QAR data are essentially a high-dimensional time series and are closely integrated with flight specialties, which have high professional barriers. Third, although the flight quality management software used by airlines will raise an alarm for ultra-limit events, many borderline critical events are filtered out, resulting in substantial data wastage. Moreover, the information from these borderline exceedance events is often not effectively integrated, analyzed, or applied.

In summary, it is necessary to combine QAR data to conduct in-depth analysis and research on the ultra-limit risk of aircraft landing attitude, explore potential risk factors and laws, and provide reliable technical support for the control and prevention of landing attitude ultra-limit events.

2. Aircraft Landing Attitude Ultra-Limit Risk Warning System

2.1. Early Warning System Framework

The aircraft landing attitude early warning system proposed in this paper includes four sub-modules: a data preprocessing module, a feature dimension reduction module, a prediction module, and a judgment module. The overall framework of the early warning system is illustrated in Figure 4.

The data preprocessing module converts the QAR database into a format suitable for deep learning models by performing data segmentation, cleaning, gap filling, and normalization.

The feature dimension reduction module is based on the mutual information of the parameters, and the QAR parameters with the highest correlation with the pitch angle and the roll angle are selected by Max-Relevance and Min-Redundancy(mRMR) algorithm [26] as the input parameters of the early warning system.

The prediction module employs a Long Short-Term Memory (LSTM) neural network [27] to construct separate prediction models for pitch and roll angles, respectively, and outputs the landing attitude sequence of the flight.

The judgment module visualizes the output landing attitude sequence into the landing attitude wrap line graph. By comparing the position of the maximum attitude data points in relation to the safety zone and wrap line in the graph, it can be determined whether the landing attitude of the flight will exceed the limit, and outputs the specific risk type.

2.2. Data Pre-Processing

The Quick Access Recorder (QAR), a widely used onboard flight data recorder, is capable of collecting comprehensive flight information and performance metrics during the whole process from engine start to shut down after landing. It records thousands of parameters such as flight trajectory, aircraft attitude, engine performance, and cockpit input. As such, QAR data serve as a critical resource for pilot training, flight quality monitoring, flight accident investigation, while also providing data guarantee for the analysis and research of unsafe events.

Due to issues such as numerous parameters, huge data volume, unguaranteed integrity, and inconsistent sampling frequency of QAR data, the original QAR data cannot be directly applied to data mining and must undergo preprocessing. To address the above problems, this paper converts the experimental data into an applicable form by means of data cleaning, feature selection, data segmentation, unified frequency, and normalization to convert the raw data into a usable format. This process results in the construction of a dedicated landing-phase database and a deep learning training dataset, providing a data basis for subsequent research. The detailed procedure is shown in Figure 5.

2.2.1. QAR Parameter Distribution Pattern

The pauta criterion, also known as the 3σ criterion, is an outlier data discrimination principle based on probability theory [28,29]. Since the pauta criterion is applicable to the data that obey normal distribution or nearly normal distribution, in order to reasonably select the outlier detection algorithm, this paper statistically analyzes each parameter in the QAR data set used, which was collected from a Part 121-certified aircraft type—Boeing 737–800 and plots the distribution histograms of the 46 parameters by taking 172,524 groups of data points, with the number of bars uniformly set to 300. Six QAR parameters whose distribution patterns are most nearly normal are filtered by the histograms. The six QAR parameters whose distribution patterns are most similar to the normal distribution were selected from the histograms and are shown in Figure 6.

In Figure 6, the histogram characteristics of the two parameters, left aileron angle (Figure 6a) and lateral acceleration (Figure 6e), have the highest degree of approximation to the normal distribution. To further assess whether these QAR parameters follow a normal distribution, a normality test was performed for the above two types of QAR parameters using the Quantile–Quantile Plot method [30]. The results obtained can be seen in Figure 7.

From the analysis of the Q-Q plot in Figure 7, the sample scatters of both left aileron angle and lateral acceleration are not fitted to a straight line, and the sample scatters are biased above the theoretical quantile line in the upper right corner and below the theoretical quantile line in the lower left corner, which is an S-shaped curve. This indicates that the tails of the above two types of parameters are more heavily weighted than the normal distribution, and they are both “peak long tail” type data distributions, which are significantly different from the normal distribution.

In summary, all parameters in the QAR data set used in this experiment show a poor fit to the pauta criterion. Therefore, the Local Outlier Factor (LOF) algorithm is adopted for outlier detection.

2.2.2. Outlier Detection

The local outlier factor algorithm was proposed by Breunig M M in 2000 [31], it is a density based local outlier detection algorithm. It evaluates the degree of outlierness by calculating the local reachable density of each data point and determines whether a point is an outlier based on this value. The detailed calculation steps of this algorithm are presented below.

Step 1: Calculate the kth distance of each point (in the case of point p) from the kth reachable distance of each point in the neighborhood of each point in the input data point set X by means of Equation (1) reach-dist_k, where point o is the kth farther away point from point p, d_k (o) is the kth distance from point o, and d(o,p) is the distance between data points o and p.

$$reach-dis{t}_k(o,p)=\max \{d_k(o), d(o,p)\}.$$

(1)

Step 2: Calculate the kth local reachable density lrd_k for each point in the data point set X, where is the kth distance neighborhood of p points.

$$lr{d}_k(p)={\displaystyle \frac{\left|N_k(p)\right|}{{\displaystyle {\sum }_{o\in N_k(p)}reach-dis{t}_k(o,p)}}}.$$

(2)

Step 3: Calculate the kth local outlier factor LOF_k for each point in the data point set X.

$$LO{F}_k(p)={\displaystyle \frac{{\displaystyle {\sum }_{o\in N_k(p)}\frac{lr{d}_k(o)}{lr{d}_k(p)}}}{\left|N_k(p)\right|}}$$

(3)

Step 4: Output the data points to which the largest n local outliers belong.

In the QAR dataset, variations exist in both the distribution and attributes of individual parameters. The LOF algorithm exhibits simplicity and intuitiveness, capable of quantifying the degree of abnormality at each sample point. It is not constrained by the underlying data distribution when identifying outliers and exhibits good applicability to QAR data. Therefore, this paper employs the LOF algorithm in Sklearn library to detect the abnormal values across different QAR parameters. For the identified and localized abnormal values and missing values, linear interpolation is applied to replace and fill the gaps. The specific results are shown in Figure 8.

2.2.3. Data Standardization

In neural network modeling, data normalization is crucial. A suitable data scale can significantly improve the performance of deep learning models, accelerate convergence, and reduce the risk of gradient vanishing and explosion. The parameters in the QAR data set have large differences in data scales and are not comparable. Therefore, this paper adopts Min–Max Scaling method to map the original QAR data to the range of [0, 1], preserving the original distribution characteristics while rendering the data dimensionless.

Let X_ij represent the jth sampling value of the ith parameter in the whole QAR data set, and is the standardized data of X_ij, which is calculated as shown in Equation (4).

$$X_{ij}^{\prime }={\displaystyle \frac{X_{ij}-X_{min}}{X_{max}-X_{min}}}$$

(4)

where X_max and X_min are the maximum and minimum values of all the sampled values of this parameter i.

2.3. Feature Dimensionality Reduction

In the training and modeling of neural networks, excessively high input dimensionality can lead to increased model complexity, higher computational resource demands, and a greater risk of overfitting. The QAR dataset used in this paper involves multiple parameters, such as flight performance, pilot operation, and atmospheric environment, with large parameter dimensions, and the correlation between each parameter and the predicted target varies in strength and weakness. Therefore, in order to take into account the model prediction effect and training complexity, this paper chooses Max-Relevance and Min-Redundancy (mRMR) algorithm to reduce the dimensionality of the input parameters and selects the parameter with the largest correlation with the target features as the input of the model.

2.3.1. Max-Relevance and Min-Redundancy Algorithm

The Max-Relevance and Min-Redundancy algorithm is a feature selection algorithm that takes into account the degree of redundancy between variables on the basis of mutual information. It is commonly applied to data analysis and machine learning tasks to select the most relevant features to improve the performance of the trained model. The main idea of the algorithm is to maximize the correlation between the features to be selected and the target features while minimizing the degree of redundancy among the features to be selected. This approach ensures that the selected features carry essential information as well as avoid overlapping content, thereby improving the generalization ability of the model.

The mRMR algorithm is able to filter n features (n ≤ m) with maximum correlation and minimum redundancy between them and the target feature c from a sample space N of dimension m, and constitutes a feature subspace S. The definitions of maximum correlation and minimum redundancy can be seen in Equations (5) and (6):

$$\max D(S,c),D={\displaystyle \frac{1}{\left|S\right|}}{\displaystyle \sum _{x_i\in S}I(x_i,c)}$$

(5)

$$\min R(S),R={\displaystyle \frac{1}{{\left|S\right|}^2}}{\displaystyle \sum _{x_i,x_j\in S}I(x_i,x_j)}$$

(6)

In Equation (5): |S| is the dimension of the filtered subspace, I (x_i,c) is the mutual information between the feature (x_i contained in S and the target feature c, where i {1,2,3,…,n}, in Equation (6): I (x_i,x_j) is the mutual information between feature (x_i and feature x_j contained in S, where j∈{1,2,3,…,n}.

The combination of maximum relevance D and minimum redundancy R is the principle of mRMR algorithm, i.e., the combination of Equations (5) and (6), which is recorded as $\max \Phi (D,R)$, and the Mutual Information Quotient (MIQ) is chosen as the judgmental basis for the screening of features by the mRMR algorithm, the definition of which can be seen in Equation (7).

$$\left\{\begin{array}{l} \max \mathit{\Phi }(\mathit{D},\mathit{R})\\ \mathit{\Phi }(\mathit{D},\mathit{R})={\displaystyle \frac{\mathit{D}}{\mathit{R}}}\end{array}\right.$$

(7)

On the basis of this criterion, if the subspace S_n−1 containing n − 1 features has been filtered out, then filtering out the last relevant feature in the remaining feature set {N−S_n−1} needs to satisfy the following equation:

$$\underset{x_j\in \{N-S_{n-1}\}}{\max }[{\displaystyle \frac{I(x_j,c)}{\frac{1}{n-1}{\displaystyle \sum _{x_i\in S_{n-1}}I(x_j,x_i)}}}]$$

(8)

2.3.2. Results of Feature Dimensionality Reduction

In this paper, the mRMR algorithm is used to perform feature dimensionality reduction on the candidate parameter sets of pitch and roll angles, respectively. The candidate parameters are arranged in descending order according to the feature correlation scores. The top 20 parameters with the highest correlation to both the pitch and roll angles are selected and visually represented in horizontal bar charts, as depicted in Figure 9 and Figure 10, respectively.

An analysis of Figure 9 reveals that since the control column directly controls the elevator and rudder to manipulate the pitch attitude of the aircraft. Specifically, the control column force (CCFF) has the highest correlation with the pitch angle, and its correlation score reaches 1.38. In addition, parameters such as the left angle of attack (AOA_LH), rudder angle (ELEV_1), the vertical acceleration (VRTG), and the right angle of attack (AOA_LH) also show strong correlation with the pitch angle, with all their correlation scores exceeding 1. These parameters are therefore considered strongly correlated with pitch attitude.

Analysis of Figure 10 reveals that there are fewer parameters with strong correlation with roll angle than pitch angle. However, the right aileron deflection angle (AILERON_2) shows the highest correlation with roll angle, with a correlation score of 1.21. This is due to the fact that the roll attitude of the aircraft is directly manipulated by the aileron deflection, so the aileron position is closely correlated with the roll attitude. Additionally, the wind speed (WIN_LAT), longitudinal wind speed (WIN_LONG), and cockpit wheel force (CWF) also show moderate correlation with the roll angle, with correlation scores of 0.8, 0.65, and 0.56, respectively. This aligns with the common sense of flight that the aircraft needs to coordinate the rudder to offset the interference of the ground wind when landing.

2.4. Predictive Modeling

2.4.1. Model Training Process

The training process of the LSTM network model constructed in this paper comprises three main steps, as illustrated in the flowchart in Figure 11.

Step 1: Define the structure of the LSTM network model, including the selection of an appropriate number of hidden layers, activation functions, loss functions, and optimizer types, tailored to the characteristics of the QAR dataset.

Step 2: Training the model. The model generates predictions through forward propagation, then refines and optimizes the model parameters via backpropagation. Following this, assess the model’s performance on the validation set. Once the evaluation meets the criteria, the model is saved.

Step 3: Cycle through Step 2 and output the model that performs best on the validation set after reaching the training count threshold.

2.4.2. Definition of Model Structure

Based on testing and analysis, it was determined that constructing separate LSTM models for pitch and roll angles yields better prediction performance. Additionally, the optimal results were achieved when using a single-layer LSTM network. Therefore, this paper adopts a one-layer LSTM network architecture for building the warning model.

The activation function of the LSTM layer is the Tanh function, and the training batch size is set to 64, the length of the output sequence is set to 12, the length of the input sequence (Seq_len), the dimension of the input parameter (Input_size), the learning rate (lr) and the dimension of the hidden layer (Hidden_size) are analyzed and adjusted with multiple values according to the demand. To prevent overfitting, a Dropout layer is added after the LSTM layer, with a dropout rate of 0.5, which randomly sets 50% of the LSTM outputs to zero during training. Two fully connected layers are set up, and the number of neurons in the hidden layer is set to 128. And the activation function is chosen to be the Relu function and the Sigmoid function, respectively. The model is trained and optimized using the Adam optimizer. Adam optimizer for training optimization, the structure of the whole model is shown in Figure 12.

2.4.3. Optimization of Model Parameters

After determining the model structure, in order to realize the best pitch and roll angle prediction effect, this paper carries out training and parameterization analysis for different numbers of input parameters and input sequence lengths to determine the optimal ratio of the number of input parameters to the length of the input sequence. Based on the results of feature dimensionality reduction, the first 11, 16, and 21 parameters (including the target itself) with the strongest correlation with pitch angle and roll angle are selected as input parameters, and the length of the input sequence is analyzed by taking 12, 18, 24 and 30 s. The optimal ratio of the number of input parameters to the length of the input sequence of the evaluation indexes is bolded, and the specific results are shown in

Table 1. Analysis of number and sequence length of pitch angle input parameters.

Length of Input List/s	Number of Input Parameters
	11			16			21
	MSE	RMSE	MAE	MSE	RMSE	MAE	MSE	RMSE	MAE
12	0.00505	0.0711	0.0523	0.00505	0.0710	0.0523	0.00502	0.0709	0.0521
18	0.00508	0.0713	0.0525	0.00506	0.0711	0.0524	0.00503	0.0709	0.0521
24	0.00460	0.0678	0.0500	0.00457	0.0676	0.0498	0.00456	0.0675	0.0497
30	0.00461	0.0679	0.0501	0.00456	0.0675	0.0497	0.00454	0.0674	0.0495

and

Table 2. Analysis of number and sequence length of roll angle input parameters.

Length of Input List/s	Number of Input Parameters
	11			16			21
	MSE	RMSE	MAE	MSE	RMSE	MAE	MSE	RMSE	MAE
12	0.00272	0.0522	0.0402	0.00271	0.0521	0.0400	0.00270	0.0519	0.0401
18	0.00296	0.0544	0.0432	0.00427	0.0653	0.0509	0.00374	0.0612	0.0492
24	0.00132	0.0363	0.0285	0.00137	0.0370	0.0292	0.00134	0.0366	0.0287
30	0.00135	0.0367	0.0288	0.00137	0.0370	0.0291	0.00140	0.0374	0.0294

.

Based on the analysis of Table 1, it can be concluded that when training the pitch angle prediction model, the more input parameters, the better the prediction effect of the model. Therefore, the number of input parameters is set to be 21. Further analysis of each column shows that when the input sequence length is set to 30 s, the evaluation indexes of the three categories of MSE, RMSE, and MAE reach the minimum value, indicating the model’s performance is the best on the test set. Therefore, in this paper, the pitch angle prediction model is trained with 21 input parameters, and the input sequence length is 30 s, resulting in an input feature dimension is 21 × 30. The learning rate of the pitch angle prediction model is set to 0.002, and the hidden layer dimension is set to 64 after the tuning parameter analysis.

Similarly, based on the analysis in Table 1, the roll angle prediction model achieves the best performance on the test set when the number of input parameters is taken as 11 and the length of input sequence is taken as 24 s, with the corresponding MSE, RMSE, and MAE values being 0.00132, 0.0363, and 0.0285, respectively. Therefore, the dimensions of the input features for training the roll angle prediction model are 11 × 24. After tuning the analysis, the learning rate is set to 0.0001, and the hidden layer dimension is set to 64. The learning rate of the roll angle prediction model is set to 0.0001, and the hidden layer dimension is set to 64.

2.4.4. Model Training Effects

During the training process of deep learning models, loss functions are typically utilized to measure the model’s performance. The trajectory of the loss function curve is essential for understanding the training quality and convergence of the model. Therefore, this paper selects the Mean Squared Error (MSE) as the loss function to track the performance of the predictive model on both the training and validation sets, thereby assessing the training efficacy. The loss function curves for the pitch and roll angle prediction models are depicted in Figure 13 and Figure 14, respectively.

An analysis of Figure 13 and Figure 14 reveals that the loss function curves of the training set and validation set of the pitch and roll angle prediction model are continuously decreasing, which indicates that the model learns normally, and it is an ideal training state. The consistent decline in loss further confirms the favorable training outcomes and suggests that the selected model parameters are well-suited for the prediction task.

Figure 15 and Figure 16 illustrate the pitch and roll angle predictions for 100 flights in the test set compared to the actual data, respectively. The blue curves in the two figures show the predicted sequence of 100 flights, each with an output sequence length of 12 s for a total of 1200 s, while the yellow curves show the actual values of the pitch and roll angle sequences. The vertical axis represents the pitch and roll angle values, while the horizontal axis denotes the timestamp in seconds.

Figure 15 and Figure 16 demonstrate that the predicted curves basically coincide with the actual flight data recording curves, which visually verifies the accuracy of the pitch and roll angle prediction model.

2.5. Landing Attitude Ultra-Limit Judgment Criteria

Excessive or insufficient pitch and roll attitudes of an aircraft upon touchdown can readily result in hazardous ground contact, thereby posing a risk to aviation. Therefore, in this paper, the landing attitude envelope of Boeing 737–800 is used to judge whether there is any abnormality in the attitude of the aircraft during landing. If the pitch angle and roll angle of the aircraft at the time of landing reach the attitude angle envelope, it means that it touches the ground with an abnormal attitude, or even wipes out the runway, which is a serious flight accident. However, such cases are rare and not reflected in the dataset for the time being, so this paper adds the pitch and roll angle ultra-limit boundary to the landing attitude envelope diagram according to the flight quality monitoring standard [32], as presented in Figure 17.

In Figure 17, the light green rectangle represents the range of the safety zone. According to the Bureau of Flight Quality Control Standard [33], the upper boundary of the safety zone takes the value of 80% of the wake angle of the Boeing 737–800, which corresponds to 7.6 degrees. The lower boundary is set at a pitch angle of 1 degree, while the right boundary corresponds to a roll angle of 4 degrees. Therefore, this paper takes the landing attitude envelope diagram in Figure 18 as the standard and basis for judging whether the aircraft landing attitude is normal. If the aircraft’s pitch and roll attitude is within the light green rectangle, it is considered to be a normal landing. If the attitude reaches or exceeds the boundary of the safety zone but does not touch the envelope, it is considered that the aircraft’s grounded attitude exceeds the limit, but it does not constitute a serious flight accident, such as wiping the runway.

3. System Performance Verification and Analysis

It is not comprehensive to illustrate the prediction effect of the model only with the relevant evaluation indexes. Therefore, this paper further validates the constructed landing attitude warning system with the ultra-limited flights in actual operation as the test object.

3.1. Visualization of Evaluations

According to the operational standards of the Bureau [32], a landing roll angle exceeding ±4 degrees is classified as a roll angle ultra-limit, with leftward deviations denoted as negative and rightward as positive. Additionally, a pitch angle exceeding 80% of the aircraft’s tail angle is considered a pitch angle ultra-limit, particularly for the Boeing 737–800 model, where this threshold corresponds to an angle of 7.6 degrees. The dataset provided by the airline includes 17 instances of roll angle ultra-limit flights, comprising 10 flights with rightward deviation and 7 with leftward deviation. However, the dataset is limited in terms of pitch angle overlimit flights, with only one recorded instance of a 7.88-degree overlimit landing pitch attitude. To compensate for the limited number of pitch-related exceedance cases, the present study conducts a statistical analysis of all available QAR datasets, ranks all flights by the maximum landing pitch attitude in descending order, and identifies 29 flights with a landing pitch angle exceeding 6 degrees. Consequently, this paper adopts a landing pitch angle of 6 degrees as the threshold for pitch angle exceedance and considers these 29 groups of flights as candidates for the pitch angle ultra-limit. This approach is utilized to evaluate the early warning performance of the system concerning flights with large pitch attitude exceedances.

To evaluate the accuracy rate of the landing attitude warning system and other related indices, this paper adds an equal number of normal flights as a control in the test set, respectively. Specifically, when verifying the pitch angle prediction effect of the landing attitude warning system, 29 groups of pitch angle ultra-limit flight data and 29 groups of normal flight data are used, totaling 58 groups, when verifying the roll angle prediction effect, 17 groups of roll angle ultra-limit flight data and 17 groups of normal flight data are used, totaling 34 groups.

To facilitate the evaluation of the prediction performance, the prediction results of pitch angle and roll angle are exported separately. The maximum values of the grounding phase, defined as 1 s before touchdown, the moment of touchdown, and 5 s after touchdown, are extracted and plotted into the off-ground analysis diagram to visualize the prediction results. The pitch angle prediction results correspond to Figure 18. The roll angle prediction results correspond to Figure 19.

Analysis of Figure 18a reveals that among the 29 sets of pitch angle exceedance data, three sets were incorrectly predicted with results below 6 degrees, while the remaining 26 sets were accurately predicted. In Figure 18b, all 29 sets of normal flight data were correctly predicted.

The analysis in Figure 19a reveals that of the 17 sets of roll angle exceedance data, two sets were incorrectly predicted with results below 4 degrees, while the remaining 15 sets were accurately predicted. In Figure 19b, one set of normal roll angle data was incorrectly predicted as exceeding the limit, whereas the remaining 16 sets were correctly identified.

3.2. Quantitative Evaluation

To quantitatively evaluate the prediction performance of the system, this paper introduces the Confusion Matrix to evaluate the reliability of the model [32], also known as the likelihood matrix or error matrix, which is a visualization tool for comparing the classification results with the actual measured values. A schematic representation of the confusion matrix is presented in Figure 20.

Since the confusion matrix actually counts the number of classification of prediction results, its primary indicators are not sufficient to directly evaluate the strengths and weaknesses of the model. Therefore, secondary evaluation metrics are derived from the basic statistical results of the confusion matrix. In this paper, we use the accuracy rate, false alarm rate, and omission rate to evaluate the prediction effect of the model.

In this paper, the ultra-limit events are regarded as positive samples, while the normal events are regarded as negative samples, and the outputs of the pitch and roll angle prediction models are used to construct the confusion matrix, and the results are shown in

Table 3. Confusion matrix for pitch angle prediction results.

Pitch Confusion Matrix		Projected Value
		Ultra-Limit	Normalcy
Actual value	Ultra-limit	26	3
	Normalcy	0	29

and

Table 4. Confusion matrix for roll angle prediction results.

Roll Confusion Matrix		Projected Value
		Ultra-Limit	Normalcy
Actual value	Ultra-limit	15	2
	Normalcy	1	16

.

From the confusion matrix of the pitch angle prediction results, it can be derived that the pitch angle prediction accuracy of the system is 94.83%, the false alarm rate is 0%, and the omission rate is 10.34%.

From the confusion matrix of the roll angle prediction results, it can be derived that the roll angle prediction accuracy of the system is 91.18%, the false alarm rate is 5.88%, and the missed detection rate is 11.76%.

4. Conclusions

This study proposes an mRMR–LSTM-based landing attitude warning system that moves the risk gate of landing attitude ultra-limit events forward, enabling prior alerts. The overall framework is first introduced; then, the mRMR algorithm selects the 20 QAR features most strongly correlated with pitch and roll. Using these features, an LSTM network is trained, and optimal parameters are identified by comparing different input dimensions and sequence lengths. Validation results show detection accuracies of 94.83% for pitch angle and 91.18% for roll angle, giving pilots a five-second reaction margin and effectively warning of impending exceedance events. The approach provides a data-driven foundation for rapid post-flight identification of high-risk sorties and, eventually, real-time cockpit advisories. Current limitations include reliance on a single aircraft type and the absence of in-flight trials. Future work will broaden the dataset to multiple fleets and conduct small-scale flight tests to enhance generalizability and support airworthiness certification.