A Multi-Head Attention-Based Transformer Model for Predicting Causes in Aviation Incidents

Abstract

The timely identification of probable causes in aviation incidents is crucial for averting future tragedies and safeguarding passengers. Typically, investigators rely on flight data recorders; however, delays in data retrieval or damage to the devices can impede progress. In such instances, experts resort to supplementary sources like eyewitness testimonies and radar data to construct analytical narratives. Delays in this process have tangible consequences, as evidenced by the Boeing 737 MAX accidents involving Lion Air and Ethiopian Airlines, where the same design flaw resulted in catastrophic outcomes. To streamline investigations, scholars advocate for natural language processing (NLP) and topic modelling methodologies, which organize pertinent aviation terms for rapid analysis. However, existing techniques lack a direct mechanism for deducing probable causes. To bridge this gap, this study trains and evaluates the performance of a transformer-based model in predicting the likely causes of aviation incidents based on long-input raw text analysis narratives. Unlike traditional models that classify incidents into predefined categories such as human error, weather conditions, or maintenance issues, the trained model infers and generates the likely cause in a human-like narrative, providing a more interpretable and contextually rich explanation. By training the model on comprehensive aviation incident investigation reports like those from the National Transportation Safety Board (NTSB), the proposed approach exhibits promising performance across key evaluation metrics, including BERTScore with Precision: (M = 0.749, SD = 0.109), Recall: (M = 0.772, SD = 0.101), F1-score: (M = 0.758, SD = 0.097), Bilingual Evaluation Understudy (BLEU) with (M = 0.727, SD = 0.33), Latent Semantic Analysis (LSA similarity) with (M = 0.696, SD = 0.152), and Recall Oriented Understudy for Gisting Evaluation (ROUGE) with a precision, recall and F-measure scores of (M = 0.666, SD = 0.217), (M = 0.610, SD = 0.211), (M = 0.618, SD = 0.192) for rouge-1, (M = 0.488, SD = 0.264), (M = 0.448, SD = 0.257), M = 0.452, SD = 0.248) for rouge-2 and (M = 0.602, SD = 0.241), (M = 0.553, SD = 0.235), (M = 0.5560, SD = 0.220) for rouge-L, respectively. This demonstrates its potential to expedite investigations by promptly identifying probable causes from analysis narratives, thus bolstering aviation safety protocols.

1. Introduction

Establishing the cause of an aviation incident or accident to prevent it from re-occurring in the future is the core goal of any aviation safety occurrence investigation and analysis. Hereafter, aviation accidents will be considered a subset of aviation incidents. Conventionally, whenever an investigation is deemed necessary in the event of an aviation incident or accident, the primary source of information is usually the Cockpit Voice Recorder (CVR) and Flight Data Recorder (FDR) devices [1]. Data from these two devices are vital in giving an account of what was happening within the cockpit and the input to the aircraft received from the pilot, respectively, minutes before and at the time of the incident. However, retrieving these two devices can take months or even years and in the worst case, the devices become severely damaged during or after the incident, making the data irretrievable [2]. In such cases, where the data on the devices do not give conclusive findings or are not readily available for the investigations to start, the experts often divert their attention to other sources which can include eyewitnesses, pilot reports, air traffic controllers, satellite images, radar information, damaged aircraft components, and weather stations readings at the time of the incident [3]. This gathered information is often prepared and presented as a narrative describing the series of events and conditions under which the incident occurred. This information is then analysed by experts to establish the likely cause of the incident [4], allowing them to suggest possible measures that can deter such incidents from happening again.

However, this entire process is time-consuming, and in the event of a design flaw, until the cause is established and a preventative measure designed and implemented, the lives of passengers flying with such an aircraft model remain at risk. As an example, the flaw in the design of Boeing 737 MAX’s Manoeuvring Characteristics Augmentation System (MCAS) feature which in certain circumstances counteract the pilots’ input caused two fatal accidents including the crash of Lion Air (JT610) [5] flight followed, five months later, by Ethiopian Airlines Flight 302 (ET-302). Both aircraft crashed a few minutes after taking-off killing all 189 and 157 people on board respectively [5,6]. If the cause of the Lion Air accident had been established quickly and acted upon appropriately, the ET-302 [6] accident would likely have been avoided. With the aim of shortening aviation incident investigation time, and allowing the quick establishment of the cause, researchers have proposed various natural language processing (NLP) and topic modelling-based approaches like latent Dirichlet allocation (LDA), latent semantic analysis (LSA), parallel latent Dirichlet allocation (PLDA) [7,8,9,10,11]. These proposed schemes analyse and group aviation terms with related meanings or that are connected to a given phase of flight, field of aviation, flight conditions, and/or causes into related topics. Such approaches could help the investigation team to establish the area of concentration and consequently, lead to quick establishment of the causes.

While previous studies have demonstrated remarkable success in predicting accident severity and identifying likely causal factors through classification and term clustering techniques, such as topic modelling, the outputs of these models often lack interpretability and fail to provide contextually rich explanations of probable causes. This study investigates the potential of a transformer-based model to infer the likely cause of aviation incidents in a human-like narrative, thereby enhancing interpretability and contextual depth by generating explanations directly from the initial analysis narrative. Transformer models like bidirectional encoder representations from transformers (BERTs) [12] and their variants [13,14,15,16,17,18] have demonstrated cutting-edge performances across various challenging NLP tasks. Since analysis narratives often contain long textual paragraphs, the researchers hypothesized that the resulting model would produce enhanced performance if based upon recent studies on long-input transformers [19,20,21,22] which have revealed that increasing the transformer’s input length positively correlates with model performance.

The training methodology employed for the transformer model in this study adheres to the core principles of language translation transformers. However, it diverges in a key aspect: rather than using the masked input to the transformer’s decoder as the target language during training, it uses the target probable cause. When evaluated on the NTSB dataset, the model demonstrated the potential of transformers to enhance the efficiency of aviation incident investigations by generating probable causes based on the analysis narrative. This study makes a two-fold contribution:

• A generative model based on a multi-head attention transformer is trained and evaluated to generate the probable cause of an aviation incident based on the raw textual analysis of event narratives preceding, during, or following an accident. Given that the training dataset comprises both long and short input narratives, the model effectively processes diverse input lengths, thereby expediting the investigation process and enhancing air transport safety.
• Many aviation incident datasets have instances with analysis narratives but no corresponding entries of the probable cause. Consequently, a significant number of instances are removed during data preprocessing, thereby reducing the volume of training data that could otherwise improve model learning. This data reduction negatively impacts the model’s ability to generalize to new instances. By using the model trained in this work, the missing probable cause entries can be inferred from the available analysis narratives, ultimately enhancing model performance and improving generalization to unseen data.

The rest of this paper is organized as follows: Section 2 presents a review of prior related work followed by Section 3 where a detailed description of our approach is presented. In Section 4, the findings of this study are presented. In Section 5, a detailed discussion of the findings is presented, highlighting the contributions and limitations of our study. Finally, Section 6 gives concluding remarks, highlighting the direction of future work.

2. Related Work

The utilization of machine learning and deep learning methods and techniques in aviation analysis and prediction has garnered increasing attention from aviation safety researchers. This interest is driven by objectives such as expediting aviation incident investigations, promptly determining the causes of incidents for swift mitigation of future occurrences, predicting incidents, and extracting knowledge to enhance air transport safety. This section delves into key prior studies that have employed AI-based techniques in alignment with aviation safety.

Burnett et al. [23] trained four conventional ML classifiers, including decision trees, KNN, SVM, and ANN with back propagation for prediction of aviation injuries and fatalities. The authors employed a cross validation training approach with 10 folds and looked at how factors like pilots’ accumulated flight hours and age impacted the rate of injuries and fatalities. Experimental results revealed ANN to be superior for the task when evaluated on datasets sourced from Federal Aviation Administration (FAA) between 1975 and 2002 inclusive.

Nanyonga et al. [24,25] utilized NLP and other AI to analyze text narratives, aiming to determine aircraft damage levels from safety incidents. Four learning models: long short-term memory (LSTM), bidirectional LSTM (BLSTM), and gated recurrent units (GRU), simple recurrent neural networks (sRNNs), and hybrid architecture models including GRU+LSTM, sRNN+BLSTM+GRU, etc., were assessed on 27,000 NTSB reports. Results indicated all models achieved over 87.9% accuracy, surpassing random guessing (25%) for a four-class problem. However, LSTMs and RNN-based models have inherent limitations in handling long-text dependencies due to their sequential nature, which could hinder performance on longer aviation narratives.

Another study [26] assessed the risk created by various anomalies in aviation events using of a hybrid classifier constituting proposed a hybrid model comprising an SVM and several neural networks. The four-step method involved all events being categorized into five risk-level groups, followed by application of an SVM model to determine the link between textual event synopses and the resulting consequences. Next, the hybrid model was trained to capture the correlations between contextual event attributes and risk-level groups. A fusion rule was then proposed to combine outcomes from the two models and finally, a stochastic-base decision tree was used to predict the risk level. The limitation of this approach lies in its dependence on a hybrid of traditional ML models, which may not capture complex relationships in textual data as effectively as deep learning models like transformers.

Zhang and Mahadevan [27] and Valdés et al. [28] deployed Bayesian inference-based techniques for aviation incident modelling and analysis. their study aimed to forecast aircraft safety incidents by employing an inventive statistical method. This method utilized Bayesian inferences and hierarchical structures to build learning models of varying complexities and goals. In contrast, Valdés et al. [27] focused on analysing commercial aviation accidents spanning the period between 1982 and 2006, as documented by the NTSB. This second study proposed a four-phase approach to build a Bayesian network capable of capturing the relationship between the sequence of events that led to the accidents. The methodology encompassed creating a graphical representation for visualizing aviation accident events, forming a Bayesian network representation by amalgamating the graphical representations of all accidents while accounting for the causal and dependent relationships between aircraft damage and personnel injury.

In study [29], two models—ResNet and simple RNN—were trained and evaluated to classify the phase of flight during which the incident happened. Various NLP-based techniques were sequentially deployed including word tokenization, punctuation, unwanted character and stopword removal, lemmatization operations, and word2vec transformation of the unstructured textual analysis narratives extracted from the NTSB aviation incident investigation reports. The models recorded a classification accuracy of more than 68% on a seven-class classification problem.

In study [30], Nanyonga et al., carried out a comparative study of two topic modelling analysis techniques: LDA and non-negative matrix factorization (NMF) regarding aviation accident reports. Using the coherence value for performance evaluation the quality of generated topics was evaluated with LDA, displaying superior topic coherence and indicating its robustness in extracting semantic connections among words within topics. NMF, on the other hand, showcased exceptional performance in line with generating unique and detailed topics, facilitating a more targeted examination of particular aspects of aviation accidents.

Their study [31] showcased an automated text classification approach utilizing machine learning that could enhance analysts’ efficiency by accurately categorizing “Occurrence” in aviation incident reports, thereby enabling more precise querying of reporting databases. Using a random forest algorithm to classify more than 45,000 textural reports, an accuracy of 80–93% was recorded based on the ICAO “Occurrence” Category. The authors also conducted text cleaning that encompassed use of standard NLP techniques including stemming, removal of irrelevant words and symbols like stop words, punctuation characters and other special symbols, and then deployed the n-gram techniques including bi-gram, tri-gram, etc., for feature extraction prior to passing the reports to the ML algorithm for classification.

Studies including [32,33,34,35] deployed NLP-based techniques including topic modelling and text classification for information extraction from and analysis of aviation incident reports and have reported competitive results regarding causal factor analysis like human factor analysis, aviation incident risk classification, aircraft damage classification, aviation report clustering and grouping, and many other AI-based tasks. While these studies offer valuable insights into the use of NLP for aviation safety, they often rely on conventional NLP techniques such as LDA or SVMs, which are limited in their ability to capture nuanced contextual relationships in lengthy incident narratives.

Recent studies have proposed innovative approaches to improving predictive models in aviation and engineering. Ning Shen et al. [36] developed a combined model for aero-engine life prediction, integrating long short-term memory (LSTM) networks with a multi-headed attention mechanism and an autoregressive integrated moving average (ARIMA) model. This hybrid approach enhances accuracy in predicting the remaining useful life (RUL) of engines by focusing on critical time features and leveraging the advantages of linear feature extraction.

Also, Zhaofei Li et al. [37] introduced the transformer–TCN self-attention network (TTSNet) for RUL prediction of aircraft engines, utilizing exponential smoothing, normalization, and multi-branch attention mechanisms to capture both global and local time-series features. Their model demonstrated superior performance compared to traditional methods, as evidenced by improved RMSE and score metrics across various datasets. Xiuxun Liu et al. [38] proposed an enhanced GNSS positioning error estimation model based on a multi-layer perceptron and attention mechanisms. Their approach significantly reduced positioning errors, outperforming state-of-the-art models like LSTM and CNN. Additionally, Huali Cai et al. [39] addressed the challenge of multi-label classification for customer complaints in the aviation industry. They introduced the MAG model, which combines BERT, attention mechanisms, and multi-channel feature extraction networks to improve classification accuracy. Their model’s ability to better extract text features and learn inter-feature relationships provides new insights into the optimization of service quality in aviation.

While previous NLP architectures such as LSTMs have been widely used for text generation tasks, they exhibit limitations when handling long-text dependencies due to their sequential nature. The multi-head attention mechanism in transformers allows for better parallelization and contextual understanding, making them well-suited for analyzing extensive aviation narratives. Additionally, long-input transformer models have demonstrated superior performance in processing lengthy text sequences [22,40], making them an optimal choice for our task.

One research gap revealed in our literature review concerns attention-based transformers. Despite the attention-based transformer models achieving outstanding performance on various NLP tasks, including machine translation [41,42,43], text summarization [44,45], text simplification [46,47], grammatical error correction [48,49], and question answering [50], little-to-no attention has been paid to their deployment in the field of aviation safety to establish the likely causes of an aviation incident given the raw text analysis narrative. The work in this study aims to close this knowledge gap by proposing and training a transformer-based model for such tasks.

3. Proposed Approach

3.1. Dataset

Several aviation and transport safety agencies such as the Australian Transport Safety Bureau (ATSB), Aviation Safety Reporting System (ASRS), and the NTSB, actively gather and release reports detailing aviation incident investigations. This study utilized aviation incident reports provided by the NTSB. These reports, along with accompanying metadata, are available on the NTSB’s website in a variety of formats, such as monthly published .pdf documents, .json files, or by querying individual reports through their online platform. A summarized version in .csv format can also be obtained. For this study, the researchers focused on .json files containing detailed incident investigations from the years 2001 to 2020. Importantly, the only included incidents were investigations that had been concluded, resulting in a dataset comprising 29,676 cases. From each report, the analysisNarrative were extracted and probableCause sections to facilitate model training and validation processes. Additionally, a comprehensive statistical analysis was carried out to assess the distribution of text lengths within these fields. It was found that the average length of the analysisNarrative was 1116 words, while the probableCause field averaged 165 words, with standard deviations of 858.36 and 93.12, respectively. Further examination revealed that the shortest analysisNarrative entry contained only 4 words, while the longest reached 36,544 words. In comparison, the probableCause field ranged from 7 to 1600 words. Figure 1 and Figure 2 visually represent the distribution of text lengths for both the analysisNarrative and probableCause fields.

3.2. Data Pre-Processing

Data pre-processing involved removing HTML tags and urls, transforming wrongly encoded characters; that is, characters encoded with the ASCII equivalent codes were decoded to their natural language characters. Also, although several statistical methods including the five-point summary-based multiplier method, the Z-score method, and extreme percentile methods, can be employed to establish the upper bound for outlier removal, given the highly skewed distribution of text length, the five-point summary-based multiplier method and the Z-score method were found to be inefficient for this study. Consequently, the extreme percentile method was utilized to determine the upper bound for out-of-range text narratives. The 99.9 th percentile $P_{99.9}$ was applied, yielding upper bounds of 10,022.45 for “analysisNarrative” and $909.71$ for “probableCause”. These values were subsequently rounded to 10,000 and 1000, respectively, and reports whose analysisNarrative entries were longer than 10,000 words, and probableCause entries longer than 1000 words, were treated as outliers and discarded for this study.

3.3. The Transformer

The transformer architecture, introduced in study [40], represented a revolutionary advancement in the NLP domain, yielding remarkable outcomes. Departing from traditional RNN models, the transformer employs multi-head self-attention, enabling parallel processing and overcoming the limitations of sequential training inherent in conventional RNNs. This self-attention mechanism not only enhances computational efficiency but also captures intricate dependencies among various text components. As described by the authors, the attention process involves associating a given query (Q), with key (K)–value (V) pairs for sequence generation. Within this framework, Q, keys K, V, and the prediction are expressed as vectors. The resulting sequence is computed through a weighted summation of the V entries, with each value’s weight determined by a passing a scaled-dot product of Q and K vectors through a softmax function as depicted in Equation (1). Figure 3 shows the architecture of the transformer model and the architectural components of its encoder, decoder and output blocks.

$$Attention(Q,K,V)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(1)

where;

• $d_k$ is the projection dimension of keys, K.
• T transposes K to allow matrix multiplication.

In order to facilitate concurrent processing, the multi-head self-attention mechanism utilizes several linear projections of Q, K, and V, each mapped to dimensions $d_k$, $d_k$ and $d_v$ respectively. These parallel operations generate outputs within the $d_v$-dimensional space, which are combined and mapped again to derive the ultimate V entries. This approach results in a model capable of simultaneously attending to information across many representational vector subspaces at various locations. The multi-head self-attention mechanism, featuring p heads, is defined as presented in Equation (2).

$$MultiHead(Q,K,V)=cancat(hea{d}_i,\dots ,hea{d}_p)W^a$$

(2)

where

• $W^a\in {\mathbb{R}}^{p{d}_v\times d_{model}}$ represents a learnable weight matrix that is used to project the concatenated outputs of all attention heads into the desired output dimension.
• $hea{d}_i=Attention(Q_i,K_i,V_i)$
• $d_{\mathit{model}}$ is the dimension of the input and output embeddings through the transformer model.

3.4. Experimental Setup

Various studies, including [51,52,53], have demonstrated that training large transformer models has led to significant advancements in natural language processing (NLP) and computer vision. Specifically, these studies trained multi-head attention-based transformers with 8 attention heads and exhibited improved performance. Consequently, the model architecture comprised 8 attention heads, with an embedding dimension of 1024 [54,55] to accommodate long input sequences and 8 encoders and decoder layers to handle the inherent complexity of the long row analysis narrative inputs. A dropout rate of $0.1$ was applied at each batch normalization layer and feed-forward layer to enhance regularization and mitigate overfitting. Additionally, the inner-layer dimension of the feed-forward network was configured to 2048 and the vocabulary sizes for both the “analysisNarrative” and “probableCause” components were set to 100,000—determined using a word counter function. The model was trained on $90\%$ of the dataset while the remaining $10\%$ was used for testing following study [56] in which this split ratio produced the best prediction results. Training was performed for 50 epochs using a learning rate of $0.001$ with the Adam optimizer, betas were set to $(0.95,0.96)$, epsilon set to $1\times {10}^{-10}$, batch-size set to 64, and cross-entropy was the loss function.

All coding was implemented in Python 3.8.10 on a server with 256 cores and 512 GB of RAM. Used libraries include: Pandas (1.5.0) for reading and managing dataframes, Numpy (1.22.4) for performing mathematical operations and categorical data transformations, and matplotlib for performance visualization. The Pytorch framework was used for building the model while libraries including sentence-bleu, score, and rouge-score were used for model performance scoring. All experiments were conducted in a Jupyter notebook on a Linux server with 256 CPU cores and 256 GB RAM, running Ubuntu 5.4.0-169-generic.

3.5. Performance Metrics

To evaluate the quality of generated probable cause, three metrics commonly used in tasks involving natural language generation problems such as text summarizing, machine translation, question answering, and grammatical error correction are used in this study.

3.5.1. Bilingual Evaluation Understudy (BLEU)

BLEU [57] deploys an n-gram based evaluation metric approach that is extensively utilized in machine translation assessment. It is precision-centric and assesses the degree of overlap between n-grams from the target and generated texts. This overlap is insensitive to word position, except for n-gram term associations. However, BLEU imposes a brevity penalty when the generated text is substantially shorter than the reference text. In addition to machine translation, BLEU finds application in problems where the input and output use the same natural language, including grammatical error correction [58,59], summarization [60,61], and text simplification [47,62], which involves rewriting a sentence into one or more simpler sentences. The BLEU score can be computed using Equation (3) [57].

$$BLEU=BP·exp(\sum _i^N(w_i·ln{p}_i))$$

(3)

where;

• $BP\to$ Brevity Penalty, calculated using Equation (4);
• $w_i\to$ order i n-gram precision’s weight;
• $p_i\to$n-gram’s modified precision score of order i;
• $N\to$ maximum n-gram order to consider.

$$BP=exp(1-{\displaystyle \frac{l_p}{l{r}_{avg}}})$$

(4)

• $l_p\to$ length of predicted cause
• $l{r}_{avg}\to$ average length of reference cause.

3.5.2. Recall Oriented Understudy for Gisting Evaluation (ROUGE)

ROUGE [63] applies a definition similar to that of BLEU. However, unlike BLEU which emphasizes precision, ROUGE’s emphasis is on recall. ROUGE comes in three main versions [64,65]: n-rouge, primarily examining n-gram overlap (such as 2-rouge and 1-rouge for 2-grams and 1-gram, respectively); L-rouge, which evaluates the Longest Common Text Sub-sequence; and s-rouge, emphasizing skip grams. Like BLEU, ROUGE finds application in both machine translation and in problems where the input and output use the same natural language, including summarizing [66,67,68], grammatical error correction [65,69], and text simplification [70,71,72], which involves rewriting a sentence into one or more simpler sentences. For each of rouge-1, rouge-2 and rouge-L, the precision, recall, and F-measure are calculated using Equations (5)–(7) [63].

$$Precision={\displaystyle \frac{Coun{t}_{mn-gram-ap}}{Coun{t}_{n-gram-p}}}$$

(5)

$$Recall={\displaystyle \frac{Coun{t}_{mn-gram-ap}}{Coun{t}_{n-gram-a}}}$$

(6)

$$F1-Score=2\times {\displaystyle \frac{precison\times recall}{precison+recall}}$$

(7)

where

• $Coun{t}_{mn-gram-ap}$ is the number of n-grams from the target probable cause matching with the predicated probable cause.
• $Coun{t}_{n-gram-p}$ is the count of n-grams in predicted probable cause.
• $Coun{t}_{n-gram-a}$ is the count of n-grams in actual probable cause.

3.5.3. Latent Semantic Analysis (LSA)

LSA [73], presented in 1997 by Landauer and Dumais in [74], calculates the semantic similarity between a reference sentence and the model’s generated sentence. It relies on pre-computed word co-occurrence counts from a large corpus. Employing the bag of words (BOW) approach, it treats word order as irrelevant. Unlike ROUGE and BLEU, LSA is lenient on variations in word choice, such as “hard” versus “difficult”. In essence, LSA encodes sentences or documents into vectors using a bag of words technique. These vectors enable the computation of similarity metrics, such as cosine similarity, to assess the likeness between generated and target texts. Like BLEU and ROUGE, LSA has seen application in measuring the output quality of various natural language generation models including text summarizing, grammatical correction, translation, and text simplification [75,76,77,78,79]. The cosine similarity between sequences, $s_1$ and $s_2$, can be obtained by converting the sequences to numeric vectors, $v_1$ and $v_2$, and then using Equation (8) for similarity calculation [80].

$$Similarity(v_1,v_2)={\displaystyle \frac{dot(v_1,v_2)}{\left|\right|v_1\left|\right|\times \left|\right|v_2\left|\right|}}$$

(8)

3.5.4. BERTScore

The BERTScore [81] performance metric was introduced as an alternative to conventional evaluation metrics, including BLEU, LSA, and ROUGE. It is particularly advantageous for assessing the quality of text summarization by measuring the semantic similarity between a generated summary and the original text. Traditional evaluation metrics often fail to accurately match paraphrases, as they rely on surface-level text comparisons and may misjudge semantically equivalent expressions that differ in wording, resulting in inaccurate performance assessments. Additionally, n-gram-based models are limited in their ability to capture long-range dependencies and tend to penalize meaningful reordering of text, further compromising evaluation accuracy.

4. Results

For model inference, instances from the test set were fed into the model, and it generated a probable cause for each analysis narrative. The example of training cases are shown in Figure 4, where random samples of analysis narratives from the test set are passed to the model. The model generated almost semantically perfect probable causes concerning each input narrative.

4.1. Model Performance Based on the BERTScore

The BERTScore was computed for each test sample and the average and standard deviation of the BERT-precision ($BER{T}_P$), BERT-recall ($BER{T}_R$), and BERT-F1 ($BER{T}_{F1}$) were recorded as (M = 0.749, SD = 0.109), (M = 0.772, SD = 0.101), and (M = 0.758, SD = 0.097), respectively. A scatter distribution of the obtained BERTScore values between (probable cause, predicted probable cause) pairs for the first 1000 incident narratives is shown in Figure 5.

4.2. Model Performance Based on the BLEU Score

The BLEU Score was used to measure how closely the predicted probable cause matched the reference probable cause. For each pair of sentences, BLEU gives a value between 0 and 1, with 1 indicating a perfect match. The minimum n-gram order was set to 1 while N was set to 4 for this work. After a series of evaluations with various random samples of size 500 from the test set, in comparison with results from other metrics, the weight vector, w was set to $(0.1,0.1,0,0)$.

For each instance in our test set, the BLEU score was computed, recording a mean score of $0.727$ with a standard deviation of $-/+$$0.330$. A scatter distribution of the obtained BLEU scores between the first 1000 (probable cause, predicted probable cause) pairs is shown in Figure 6.

4.3. Model Performance Based on the LSA Similarity Score

LSA similarity gives the semantic similarity between vector representations of the output probable cause and target probable cause. It represents the semantic similarity rather than lexical similarity. A high similarity score implies that the sequences have closer meanings. This is similar to the case for BLEU scores, as the (probable_cause, predicted_probable_cause) pair was obtained for each instance in the test set.

Each component of the pair was then converted into its numeric vector representation using Google’s pretrained Universal Sentence Encoder version 4, which is the latest version at the time of writing this paper. Universal Sentence Encoder models were introduced by Google Researchers in study [82], where the cosine similarity was deployed consequently placing vector embeddings of semantically similar words close to each other. The pretrained Universal- Sentence Encoder model used in this work can be downloaded from the TensorFlow hub. The Universal Sentence Encoder was accessed on 11 December 2020 at (https://tfhub.dev/google/universal-sentence-encoder/4). Our model recorded a mean LSA similarity score of $0.697$ with a standard deviation of −/+$0.153$. A distribution of the obtained similarity scores is visualized in Figure 7.

4.4. Model Performance Based on the ROUGE Scores

For ROUGE Scores, this study considered n-rouge (Rouge-1, Rouge-2) and L-rouge (Rouge-L). These scores measure the overlap of n-grams between the candidate and reference sentences. Rouge-1 gives score from unigrams, Rouge-2 gives score from bi-grams, while Rouge-L gives the score from the longest common sub-sequence. Higher scores indicate better overlap between the sentences.

4.5. analysisNarrative Length vs. BLEU/LSA Scores

Further investigations were carried out on how the length of the input analysis narrative impacted the model’s output in terms of the BLEU and LSA similarity scores. The results revealed that the analysis narrative length had no direct correlation with the model’s BLEU score as shown in Figure 8. On the other hand, the LSA similarity score shows no correlation with the length of the input analysis narrative for shorter inputs. However, it tends to converge to the mean score as the length of the analysis pattern increases as shown in Figure 9. This finding emphasizes the researchers’ hypothesis which stated that working with long input sequences would enhance the model’s predictive performance. This also emphasises the finding of prior studies on long-input transformers including [19,20,21,22] which revealed that increasing the transformer’s input length positively correlates with model performance.

5. Discussion

Having the ability to predict the probable causes of an aviation incident can greatly expedite the investigation process. The results from this study revealed that a multi-head attention-based transformer model is a tool for solving this problem. However, although the model recorded commendable results across all metrics, by their formula, the LSA similarity score is more reliable compared to the BLEU and ROUGE metrics. This is because the model’s output and reference sentence can constitute a different set words for the same semantic content. Since the LSA similarity score computes the overall semantic similarity between the sentences, it will more likely produce a high score if the two sentences are similar and vice versa. On the other hand, the BLEU score requires that the weight vector, w for each n-gram is manually determined. This means that the final BLEU score greatly depends on the accuracy of the values of w which requires human expert and if wrongly determined can lead to misleading results. Also, the computation of BLEU and ROUGE scores like the uni-gram, bi-gram, etc., depend on the overlap of words between the reference and predicted sentence, that is, observed probable cause and predicted probable cause for this study.

For instance, considering the output in the screenshot in Figure 10, the reference probable cause as given in the dataset is as follows:

“The mechanic’s improper maintenance of the main transmission aft pinion nut and belt drive system, which resulted in the uncoupling of the tail rotor driveshaft and the subsequent loss of helicopter control.”

While the model’s prediction, given the same analysis narrative, is as follows:

“The failure of the main rotor drive belts due to a loss of belt tension on the main rotor drive system as a result of maintenance personnel’s failure to properly secure the nut and the helicopters main rotor drive belts.”

Although the semantic meanings of the two narratives are close and would both draw the incident investigator’s attention to the same component and attribute the failure to the maintenance personnel’s not properly securing the nut and belt drive system, BLEU scores differed across different weight vector values. On the other hand, the ROUGE scores were Rouge-1: precision = 0.476, recall = 0.606, F-measure = 0.533; Rouge-2: precision = 0.146, recall = 0.188, F-measure = 0.164; and Rouge-L: precision = 0.310, recall = 0.394, F-measure = 0.347 as shown in

Table 1. ROUGE results: precision, recall, and F-measure from Rouge-1, Rouge-2, and Rouge-L.

Metric	Precision		Recall		F-Measure
	Mean	Stddev	Mean	Stddev	Mean	Stddev
rouge-1	0.666	0.217	0.610	0.211	0.618	0.192
rouge-2	0.488	0.264	0.448	0.257	0.452	0.248
rouge-L	0.602	0.241	0.553	0.235	0.560	0.220

.

As it can be seen in

Table 2. BLEU scores for various weight vector values.

Weight Vector	BLEU Score
$[0.1,0.1,0.1,0.1)]$	$8.67\times {10}^{-32}$
$[0.01,0.01,0.01,0.01]$	$7.83\times {10}^{-4}$
$[0.25,0.25,0,0]$	0.459
$[0.1,0.1,0,0]$	0.732
$[0.01,0.01,0,0]$	0.969

, the results from the BLEU score largely depend on the values of vector w. It is also clear that the score greatly degrades when w contains entries for the tri-gram and quad-gram which correspond to the third and fourth entries of w, respectively. The value is also misleading for very small entries of the uni-gram and bi-gram as seen when w is set to $(0.01,0.01,0,0)$.

After a performance comparison with results for BERTscore, LSA-score, and ROUGE, and considering BLEU scores for the different the weight vectors in Table 2, $w=(0.01,0.01,0,0)$ was chosen for BLEU scoring.

Generally, the recorded scores in the case of ROUGE metrics are relative more reliable for the Rouge-1 and Rouge-L. The Rouge-2 has recorded poor performance due to the fact that the word sequence in the reference text does not always overlap with the word sequence in the model’s output. For the example output in Figure 10, the recorded ROUGE scores are poor in terms of precision, recall, and F-measure for all the three n-grams used in this study despite the semantic meaning being very similar. On the other hand, because the LSA returns the semantic similarity between two text sequences, its output is considerably high ($0.757$) for this particular example indicating that despite the discrepancies in the used set of words, the semantic meaning is greatly similar.

Finally, the LSA Similarity score’s input length-model performance analysis indicated that training the model with long inputs can result in stable model performance as the score converged to the mean score with increasing input length (see Figure 9).

Examining just research article titles for keywords in machine learning for aviation safety shows there has been significant concurrent research on the topic.

Table 3. Comparison of NLP models for predicting probable causes in aviation incidents.

Authors	NTSB Dataset	NLP	Transformer-Based Approach	BERT Score	ROUGE Score	BLEU Score	LSA Similarity
Nanyonga et al. [83]	Y	Y	N	N	N	N	N
Darveau et al. [84]	N	Y	N	N	N	N	N
Zhao et al. [85]	Y	Y	N	N	N	N	N
Xiong et al. [86]	N	Y	N	N	N	N	N
Hou et al. [87]	N	N	N	N	N	N	N
Ni et al. [88]	N	N	N	N	N	N	N
Xiong et al. [89]	Y	N	N	N	N	N	N
Liu et al. [90]	Y	N	N	N	N	N	N
Akiko Aizawa [91]	N	N	N	N	N	N	N
Xiong et al. [92]	N	N	N	N	N	N	N
Perboli et al. [35]	N	Y	N	N	N	N	N
Katragadda et al. [93]	N	Y	N	N	N	N	N
Dong et al. [4]	N	Y	N	N	N	N	N
Our study	Y	Y	Y	Y	Y	Y	Y

shows 12 research publications alongside the current study, one of which is by the team here. The table also maps the key features of this study to show its significance against current research. Three of the studies have none of our selected features in common such as transformer approach to predicting probable cause.

Limitations

Training a highly efficient transformer model necessitates a substantial amount of training data, which posed a significant limitation in this study. As a result, while the model generated correct predictions for the majority of “analysisNarrative” entries, it also produced instances of insufficient information and, in some cases, incorrect predictions.

Table 4. Examples where the model produced insufficient information or incorrect predictions.

Observed “probableCause”	Predicted “probableCause”
The pilot’s inadequate compensation for the crosswind condition and failure to maintain directional control of the aircraft on landing.	The pilot’s failure to maintain directional control during landing.
The incapacitation of the pilot during high-altitude cruise flight for undetermined reasons.	The pilot’s intentional flight into terrain as a result of his impairment due to alcohol consumption.
The pilot’s failure to monitor the balloon’s altitude, which resulted in the balloon impacting an airport rotating beacon tower.	The pilot’s failure to maintain clearance from a powerline during landing.

presents examples where the model generated incomplete or inaccurate predictions in relation to the observed “probableCause” entries.

The second limitation is that the primary objective of this study was limited to assessing the effectiveness of the standard transformer model architecture in predicting the probable cause of an aviation incident based on the raw-text incident analysis narratives while the exploration of alternative models, such as Longformer, LongT5, BigBird, and BigBirdPegasus, among others, was deferred to future research.

Finally, the standard multi-head self-attention transformer architecture exhibits quadratic scaling with respect to the input size, n, with a time complexity of $O\left(n^2{d}_{\mathrm{model}}\right)$. Consequently, both the training and inference times in this study increased quadratically with the input size. Furthermore, the large projection dimension $d_{\mathrm{model}}$, combined with hardware constraints—specifically, the use of a CPU instead of a GPU—contributed to the model’s slow average inference time of $0.322$ s per instance.

6. Conclusions

Quick identification of the potential cause of an aviation incident is crucial for preventing future tragedies. While flight data recorders are commonly used, delays or damage can obstruct their effectiveness. The Boeing 737 MAX accidents with Lion Air and Ethiopian Airlines highlight the impact of such delays. To improve investigation efficiency, this study trained a transformer-based model for predicting the probable cause of an aviation incident given an analysis narrative of the series of events that can be collected from sources including eyewitnesses, radar systems, Air traffic controllers that were in charge of the flight under investigation, maintenance history/logs, etc. The model was trained on extensive NTSB aviation incident reports and allows short- and long-input narratives.

Despite the approach demonstrating potential in expediting investigations and enhancing aviation safety—evidenced by its strong performance across evaluation metrics such as BERTScore, BLEU, ROUGE, and LSA—the model’s output can be further improved through training on a more comprehensive dataset. As a direction for future research, analysis narratives from additional aviation investigation bureaus, such as the ATSB, could be integrated with the NTSB narratives. Retraining the model on this expanded dataset is expected to enhance its predictive capabilities and overall performance.

While the evaluation primarily focused on quantitative performance metrics, a critical next step is to incorporate expert assessment of the model’s predictions. Their feedback is essential for validating the model’s applicability, ensuring the accuracy and usefulness of the generated probable cause statements, and aligning the model with real-world aviation safety analysis and decision-making processes. Furthermore, the model was designed to support aviation incident investigators during the preliminary analysis phase by generating probable causes from raw incident narratives, thereby enhancing the efficiency of the investigation process by providing quick insights. However, it is not intended for use during the data collection or final report preparation phases. Moreover, the deployment of AI models in safety-critical aviation systems requires adherence to established regulatory frameworks, such as those set by the FAA and the European Union Aviation Safety Agency (EASA). Future research is needed before use in operational contexts to further explore how the model aligns with existing regulatory standards, particularly in terms of transparency, interpretability, and reliability.