Aircraft Engine Performance Monitoring and Diagnostics Based on Deep Convolutional Neural Networks

Round 1

Reviewer 1 Report

This paper introduced a novel deep learning framework for aircraft engine fault diagnosis. It's a well-written paper with detailed figures and algorithms, and solid experiments. I only have a few minor comments:

1. Please improve the main contribution discussion in the introduction. The current ones are not concise enough.
2. Please provide more related works about solving fault diagnosis problems using deep learning approaches.
3. Please specify how many trials were run to obtain the experimental results. Please also provide the variance to the results if multiple trials were executed.

Author Response

Response to the reviewer’s comments

• Title: Aircraft Engine Performance Monitoring and Diagnostics based on Deep Convolutional Neural Networks.
• Manuscript ID: machines-1472713

We would like to thank the reviewer for his/her effort and comments, which are very helpful to improve the quality of the paper. We have taken these comments into consideration in the revised version of the manuscript. In the following answers, all reviewer’s comments are in black, and our answers in blue. We hope that the reviewer will find our answered satisfactory, and we are also willing to consider any further suggestion that the reviewer might have on the revised manuscript.

Best Regards

Reviewer 1

This paper introduced a novel deep learning framework for aircraft engine fault diagnosis. It's a well-written paper with detailed figures and algorithms, and solid experiments. I only have a few minor comments:

1. Please improve the main contribution discussion in the introduction. The current ones are not concise enough.

Improvements have been made (page 3, paragraph 2).

1. A novel physics assisted CNN framework is proposed for three-shaft turbofan engines fault diagnostics. The framework can discriminate between gradual and rapid gas turbine deterioration followed by a successful isolation of gas path faults at the component level. The physics-based scheme can also update itself for baseline changes caused by maintenance events. This avoids the need for retraining the CNN algorithm after every overhaul. However, using the physics-based scheme alone for diagnostics has some accuracy deficiencies due to measurement uncertainty and model smearing effects. Hence, the CNN technique is coupled to offset these limitations and enhance the overall diagnostic accuracy.
2. As demonstrated by the experimental results, the proposed method can deal with multiple-fault scenarios, which will increase the significance of the method in real-life situations [34].

• Benefits of applying modular CNN framework for gas turbine FDI is verified through comparison with a similar LSTM framework and with a single CNN based FDI scheme. It is shown that the method proposed outperforms the other methods.

1. It is also verified that the method proposed is advantageous to handle a considerable disparity between training and test datasets, that is difficult for most of traditional data-driven methods [35]. This robustness is important to accommodate engine-to-engine degradation profile differences.
2. Please provide more related works about solving fault diagnosis problems using deep learning approaches.

The yellow highlighted related works are added in the revised manuscript (page 2, paragraph 3).

The rapid advancement of machine learning (ML) methods opens extended re-search access to investigate the contribution to the gas turbine application domain. For instance, a deep autoencoder was utilized by Yan and Yu [20] for measurement noise removal and a gas turbine combustor anomaly detection. A re-optimized deep auto-encoder based anomaly detection method was also demonstrated in [21] for fleet gas turbines. On a different study, a long short-term memory network based autoencoder (LSTM-AE) framework was developed for aircraft engine sensor and actuator faults detection and classification using raw time-series data [22]. Combined GPA and LSTM based gas turbine fault diagnostics and prognostics method was devised by Zhou et al. [23]. The GPA was dedicated to estimate performance health indices of the target gas path components of the case study engine through a performance adaptation process. Whereas the LSTM method was employed to forecast the future degradation profile of the components based on the estimated health indices. In recent years, there has been a rapid rise in the use of convolutional neural networks (CNNs) for rotating machinery diagnostics inspired by their powerful feature learning and classification ability [24]. A considerable number of applications can also be found in gas turbine prognostics, such as [25-27]. Nevertheless, there have been only a few attempts on gas turbine diagnostics. Liu et al. [28] proposed a CNN based technique to monitor the performance of a gas turbine engine hot components based on exhaust gas temperature (EGT) profiles. Guo et al. [29] used a 2D-CNN algorithm for a gas turbine vibration monitoring using transformed vibration signals as input. Grouped convolutional denoising autoencoders were used to reduce measurement noise and extract useful features from aircraft communications, addressing and reporting system (ACARS) data [30]. A 1D CNN was employed for abrupt-fault diagnostics based on timeseries data [31]. Zhong et al. [32] and Yang et al. [33] evaluated the effectiveness of the transfer-learning principle with CNN for engine fault diagnostics with limited fault samples. In both studies, the au-thors considered single fault scenarios only. Conversely, the benchmark studies con-ducted within the Glenn Research Center in NASA [34] recommended to consider multiple faults as well for more reliable diagnostic solutions.

3. Please specify how many trials were run to obtain the experimental results. Please also provide the variance to the results if multiple trials were executed.

To determine the hyperparameters of the CNN model with the best performance, we executed extensive experiments according to the following controlling parameters:

1. Number of layers and their order of arrangements in the CNN structure.

• Convolution → Relu → Dropout → Pooling → Dropout → Flattened → Fully-connected
• Convolution → Relu → Dropout → Pooling → Dropout → Convolution → Relu → Dropout → Pooling → Flattened → Fully-connected
• Convolution → Relu → Convolution → Relu → Dropout → Pooling → Pooling → Dropout → Flattened → Fully-connected
• Convolution + Relu → Convolution → Relu → Convolution → Relu → Droput → Pooling → Pooling → Droput → Flattened → Fully-connected
• Convolution → Convolution → Relu → Droupout → Pooling → Dropout → Flattened → Fully-connected

2. Number of filters for each convolutional layer: from 1 to 500
3. Filter size: from 1x1 to 1x12
4. Dropout values: 10%, 20%, 30%, 40% and 50%.
5. Pooling operations: maximum pooling and average pooling
6. Pooling layer size and stride: Pool size from [1 1] to [1 10] and stride from [1 1] to [1 5]
7. Three different optimization algorithms: Adam, sgdm, and RMSProp
8. Different training and test dataset ratios: 60/40, 70/30, and 80/20
9. Number of epochs: from 3 to 1000
10. Learning rate: 0.1, 0.01, and 0.001

However, since we did not save the results for each iteration, we are afraid that it is difficult for us to remember how many trials were run to obtain the experimental results, and to provide the variance for the results.

Thank you once again!

Best regards

Authors

Author Response File: Author Response.pdf

Reviewer 2 Report

In this paper, a new method is proposed to solve the problem that aircraft engine performance monitoring and diagnostics. Overall, the quality of this work is good. Thus, a minor revision is necessary. 

1. 1.For the convenience of readers' understanding, it is necessary to briefly explain the relationship between aircraft engines and gas turbines.
2. Please explain what factors mainly refer to noise in the fault diagnosis ofaircraft engines, or how it is generated?
3. The author mentions the advantages of model-based and data-driven methods in the introduction, but do not mention the disadvantages. Please elaborate on this content.
4. Please further elaborate on the advantages of the method proposed in this paper, especially compared to traditional model-based and data-driven methods.
5. Please improve the background of data-driven methods by referring, such as, to Data-driven fault diagnosis for traction systems in high-speed trains: A survey, challenges, and perspectives.
6. 6.For the convenience of readers' understanding, it is necessary to summarize the flow chart of the algorithm.

Author Response

Response to the reviewer’s comments

• Title: Aircraft Engine Performance Monitoring and Diagnostics based on Deep Convolutional Neural Networks.
• Manuscript ID: machines-1472713

We would like to thank the reviewer for his/her effort and comments, which are very helpful to improve the quality of the paper. We have taken these comments into consideration in the revised version of the manuscript. In the following answers, all reviewer’s comments are in black, and our answers in blue. We hope that the reviewer will find our answers satisfactory, and we are also willing to consider any further suggestion that the reviewer might have on the revised manuscript.

Best Regards

Reviewer 2

In this paper, a new method is proposed to solve the problem that aircraft engine performance monitoring and diagnostics. Overall, the quality of this work is good. Thus, a minor revision is necessary.

1. For the convenience of readers' understanding, it is necessary to briefly explain the relationship between aircraft engines and gas turbines.

Normally, aircraft engine is a gas turbine. But aiming to avoid any confusion for readers, we tried to be as much consistent as possible in using these words in the revised manuscript.

2. Please explain what factors mainly refer to noise in the fault diagnosis of aircraft engines, or how it is generated?

In this paper, noise refers to measurements’ uncertainty. This has been explicitly specified in the revised manuscript. The following text is also added to clarify how it is generated.

Description of each measurement used for the engine diagnostics with the associated level of noise considered is provided in Table 5. A Gaussian noise is added for each measurement with a standard deviation (σ) in percent of the measured values. (Page 14)

3. The author mentions the advantages of model-based and data-driven methods in the introduction, but do not mention the disadvantages. Please elaborate on this content.

The below text is added in the revised manuscript (Page 2, paragraph 2).

Model-based methods have some accuracy deficiencies due to measurement uncertainty and model smearing effects. Data-driven methods, on the other hand, lack interpretability of their internal working (they are “black-box” models), require large amount of data for training, and the training process can be excessively time-consuming [1, 14].

4. Please further elaborate on the advantages of the method proposed in this paper, especially compared to traditional model-based and data-driven methods.

Further elaborations on the advantages of the method (page 3):

1. A novel physics assisted CNN framework is proposed for three-shaft turbofan engines fault diagnostics. The framework can discriminate between gradual and rapid gas turbine deterioration followed by a successful isolation of gas path faults at the component level. The physics-based scheme can also update itself for baseline changes caused by maintenance events. This avoids the need for retraining the CNN algorithm after every overhaul. However, using the physics-based scheme alone for diagnostics has some accuracy deficiencies due to measurement uncertainty and model smearing effects. Hence, the CNN technique is coupled to offset these limitations and enhance the overall diagnostic accuracy.
2. As demonstrated by the experimental results, the proposed method can deal with multiple-fault scenarios, which will increase the significance of the method in real-life situations [34].

• Benefits of applying modular CNN framework for gas turbine FDI is verified through comparison with a similar LSTM framework and with a single CNN based FDI scheme. It is shown that the method proposed outperforms the other methods.

1. It is also verified that the method proposed is advantageous to handle a considerable disparity between training and test datasets, that is difficult for most of traditional data-driven methods [35]. This robustness is important to accommodate engine-to-engine degradation profile differences.

5. Please improve the background of data-driven methods by referring, such as, to Data-driven fault diagnosis for traction systems in high-speed trains: A survey, challenges, and perspectives.

Done. Page 2, paragraph 2.

6. For the convenience of readers' understanding, it is necessary to summarize the flow chart of the algorithm.

Section 3.1.1 has been revised for more clarity, and the following paragraph is added in the text to summarize the flow chart:

To estimate the net measurement deltas induced by the underlying fault(s), first, a new baseline should be established at some previous flight α and compute the performance parameter deltas backwards from the current flight k to flight α+1, as illustrated in Figure 4 and Figure 5. Second, use the estimated performance parameter deltas to predict corrected measurements through the engine performance model in adaptive mode at reference conditions (i.e., T_Ref = 288.15K and P_Ref = 1.01325 bar). Then, set the corrected measurements at flight α as new baseline measurements. For the subsequent flights (from flight α+1 to flight k), take the actual measurement at each flight and run the gas path analysis to estimate the associated performance parameter deltas respect to the new baseline. Repeat the second step and predict the associated corrected measurements. Finally, compute the measurement deltas based on Eq. (1).

Thank you once again!

Best regards

Authors

Author Response File: Author Response.pdf

Reviewer 3 Report

The authors present in their paper the application of a CNN towards the prediction and monitoring of faults and engine performance for an aircraft engine. The particular force of these networks is considered promising for correct, but also robust fault diagnostics on gas turbine engines, however, have not yet been applied before. In this paper, the authors developed such a method and tested its capabilities on multi-fault diagnostic cases. Moreover, the method was compared with two alternative methods, clearly showing that it outperforms both methods. 

Overall, the paper is very well written, structured, and easy to read and understand. The methodology is well explained, easy to follow and read. Moreover, the results show indeed the force of the used method and hence support the conclusion. The quality of the paper is very high and apart from some minor errors listed below, I believe the paper can be accepted for publication (after having made these minor corrections). 

Minor errors:

1. Line 77: the acronym CNNs is not introduced
2. I found some minor spelling/grammar mistakes: 
   1. line 108: is also provide must be are also provided
   2.  line 132: and is usually caused by
   3. Line 356: the propose method are 
   4. line 587: are responbile for 
3. In equation 4, it is not clear why you subtract y_min from x_i^j. Please check if this is correct and explain
4. Equations 5 --> 9: what is the considered sigma function here (this is not specified in the text)
5. Lines 481-->489 as well as table 8: can the authors present some additional information on how the ranges for fault magnitude have been set (in particular why 4% and 5% max errors have been selected, since this appears as arbitrarily). 
6. Table 10 and lines 572 to 578: it is not clear to me how to interpret this: does it mean one has to work with larger datasets? What is the cost in terms of required computational time?
7.  Section 4.1 (idem for 4.2 and 4.3): any comment on the additional cost of a multiple CNN compared to these alternative methods with whom you compared, as opposed to its gain in accuracy? 

Author Response

Response to the reviewer’s comments

• Title: Aircraft Engine Performance Monitoring and Diagnostics based on Deep Convolutional Neural Networks.
• Manuscript ID: machines-1472713

We would like to thank the reviewer for his/her effort and comments, which are very helpful to improve the quality of the paper. We have taken these comments into consideration in the revised version of the manuscript. In the following answers, all reviewer’s comments are in black, and our answers in blue. We hope that the reviewer will find our answers satisfactory, and we are also willing to consider any further suggestion that the reviewer might have on the revised manuscript.

Best Regards

Reviewer 3

The authors present in their paper the application of a CNN towards the prediction and monitoring of faults and engine performance for an aircraft engine. The particular force of these networks is considered promising for correct, but also robust fault diagnostics on gas turbine engines, however, have not yet been applied before. In this paper, the authors developed such a method and tested its capabilities on multi-fault diagnostic cases. Moreover, the method was compared with two alternative methods, clearly showing that it outperforms both methods.

Overall, the paper is very well written, structured, and easy to read and understand. The methodology is well explained, easy to follow and read. Moreover, the results show indeed the force of the used method and hence support the conclusion. The quality of the paper is very high and apart from some minor errors listed below, I believe the paper can be accepted for publication (after having made these minor corrections). 

Minor errors:

1. Line 77: the acronym CNNs is not introduced.
2. I found some minor spelling/grammar mistakes: 

• Line 108: is also provide must be are also provided
• Line 132: and is usually caused by
• Line 356: the propose method are 
• line 587: are responbile for 

3. In equation 4, it is not clear why you subtract y_min from x_i^j. Please check if this is correct and explain. It was a typo, the minus sigh between and  should not be there. So, the equation is corrected as 
4. Equations 5 --> 9: what is the considered sigma function here (this is not specified in the text)

σ denotes a sigmoid activation function, and this is now added in the text.

5. Lines 481-->489 as well as table 8: can the authors present some additional information on how the ranges for fault magnitude have been set (in particular why 4% and 5% max errors have been selected, since this appears as arbitrarily). 

According to Diakunchack [i], compressor fouling can cause 5% flow capacity and 1.8% isentropic efficiency loss. That is equivalent to a modular fault magnitude of 5.3%. Due to their exposure to the inlet air contaminants, gas path components before the combustor are often prone to higher fault magnitudes than the components after. As can be seen in Ref. [ii] & [iii] bellow, turbine faults are represented by a fault magnitude of 4.5%. For this reason, we considered 5% for the FAN, IPC, and HPC and 4% for the HPT, IPT, and LPT. However, since there is no consistency in the literature in this regard [iv], the authors would like to live it as users defined.

168. Diakunchak, I.S. Performance deterioration in industrial gas turbines. J. Eng. Gas Turbines Power 1992, 114, 161–168.
169. Mohammadi, E., and Montazeri-Gh, M. (April 18, 2014). "Simulation of Full and Part-Load Performance Deterioration of Industrial Two-Shaft Gas Turbine." ASME. J. Eng. Gas Turbines Power. September 2014; 136(9): 092602. https://doi.org/10.1115/1.4027187

• Qingcai, Yang, et al. "Full and part-load performance deterioration analysis of industrial three-shaft gas turbine based on genetic algorithm." Turbo Expo: Power for Land, Sea, and Air. Vol. 49828. American Society of Mechanical Engineers, 2016.

83. Fentaye, A.D.; Baheta, A.T.; Gilani, S.I.; Kyprianidis, K.G. A review of gas turbine gas-path diagnostics: State-of-the-Art methods, challenges and opportunities. Aerospace 2019, 6, 83.
84. Table 10 and lines 572 to 578: it is not clear to me how to interpret this: does it mean one has to work with larger datasets? What is the cost in terms of required computational time?

As we tried to explain under the sub-section “Effect of data distribution”, the data used to train and test the CNN model should be sufficient to represent the distribution of both the fault and fault-free classes. It should also be balanced to avoid model biasedness towards the majority class. With this in mind, we considered different datasets and the experiment results showed that the detection accuracy increases with the data size, while the data is also becoming more balanced.

intel Core i7 CPU, windows 10 operating system was the hardware used. As can be seen in the table below, the computational time increases with the training data size. But this would not be a problem since a GPU or HPC can be used for real-life applications.

The below sentence is included in the revised version to clarify this.

Using a high-performance computing (HPC) or GPU can overcome this problem.

1. Section 4.1 (idem for 4.2 and 4.3): any comment on the additional cost of a multiple CNN compared to these alternative methods with whom you compared, as opposed to its gain in accuracy?

The complexity of the framework and space requirement may increase with increasing number of faults considered in the system. Such as sensor faults, actuator faults, and some other simultaneous fault scenarios, for instance, sensor-component, sensor-actuator, sensor-sensor, component-component-component, actuator-component etc. But the breakthrough of hardware can overcome this problem. 

Thank you once again!

Best regards

Authors

Author Response File: Author Response.pdf

Reviewer 4 Report

The paper investigates a very interesting topic in gas turbines diagnostics. It is well written with a good structure and use of English language. It clearly explains the methodology and how it is implemented in the gas turbine diagnostics.

            I only have minor comments:

• Line 481 - Rapid degradation/fault simulation: how the authors evaluate the depth of the maintenance action (figure 2) and re-evaluate the baseline performance to re-use the model.
• Table 8: "on top of the gradual degradation" how the method responds when corrective maintenance action is taken place? Was that scenario examined?
• Line 268: NASA report should be reference 29
• Line 502: Usually, 1st person is avoided

Author Response

Response to the reviewer’s comments

 Title: Aircraft Engine Performance Monitoring and Diagnostics based on Deep Convolutional Neural Networks.

• Manuscript ID: machines-1472713

We would like to thank the reviewer for his/her effort and comments, which are very helpful to improve the quality of the paper. We have taken these comments into consideration in the revised version of the manuscript. In the following answers, all reviewer’s comments are in black, and our answers in blue. We hope that the reviewer will find our answers satisfactory, and we are also willing to consider any further suggestion that the reviewer might have on the revised manuscript.

Best Regards

Reviewer 4

The paper investigates a very interesting topic in gas turbines diagnostics. It is well written with a good structure and use of English language. It clearly explains the methodology and how it is implemented in the gas turbine diagnostics.

I only have minor comments:

• Line 481 - Rapid degradation/fault simulation: how the authors evaluate the depth of the maintenance action (figure 2) and re-evaluate the baseline performance to re-use the model.

In real-life situations, the depth of performance recovery depends on the type and effectiveness of the maintenance action take place. It could be evaluated by comparing with the performance of the engine when it was a brand new, or by comparing it with the re-baselined performance just before the fault occurs (since the performance change due to gradual degradation during the fault event would not be significant). If there is unrestored performance due to poor maintenance, re-baselining will take place after the maintenance event to re-use the model ahead.

Once the gas turbine is tested in a test cell after the maintenance action, test cell measurements could be used to update the baseline performance model.

The below paragraph is included in the revised manuscript (page 6, paragraph 2).

If maintenance actions take place after a rapid or abrupt fault event, the level of performance recovered needs to be assessed. The level of the recovery depends on the type and effectiveness of the maintenance action take place. It could be evaluated by comparing with the performance of the engine when it was a brand new, or by comparing it with the re-baselined performance just before the fault occurs. If there is a considerable unrecovered performance left after the maintenance, re-baselining will take place after the maintenance event to re-use the model ahead.

• Table 8: "on top of the gradual degradation" how the method responds when corrective maintenance action is taken place? Was that scenario examined?

Once the gas turbine is tested in a test cell after the corrective action, the results could be compared to performance tests from when the gas turbine was newly produced. If changes are detected, re-baselining will take place through the model adaptation process. Changes in performance after the overhaul should then be tracked with reference to the new baseline.

However, since this requires real-time maintenance activities and test cell data, this scenario was not examined in the current work.

• Line 268: NASA report should be reference 29. We have double checked, and the used reference was correct.
• Line 502: Usually, 1^st person is avoided. Corrected.

Thank you once again!

Best regards

Authors

Author Response File: Author Response.pdf