Aircraft Fleet Health Monitoring with Anomaly Detection Techniques
Round 1
Reviewer 1 Report
This paper focuses on the application of well know ML methods to anomaly detection problem using considerable volume of real sensor data (and maintenance data including messages and actual services) from a cooling unit system on a wide body aircraft. In that sense, the three basic autoencoder models (namely fully-connected autoencoder, convolutional autoencoder and a long short-term memory (LSTM) autoencoder) are well know and applied/developed for similar problems. In that respect the paper does not provide novelty beyond application. However the fact that the work is carried out on real (and complex) data and the results are real-life based is evaluated as positive. As such, the paper would have benefitted from a more methodological approach (and sensitivity analysis) towards the number of layers, units per layer, activation functions and selection of thresholds. Currently the designs come across as one shot solution and there is limited insight on many of the design selections. In addition the paper is missing considerable amount of details on the actual plots of training process of the designs as to decipher cases of overfit or underfit. As such discussions on the realism of the model (and model performance) beyond the data that is collected and the models ability to generalize (or not) could have been interesting.
The fact that there are cases where "faulty signals are correctly reconstructed (false negatives) and healthy signals are not (false positives)" suggests that there are further contributing factors (maybe specific sensors) that contribute to such cases. Again the paper would have benefitted from analysis.
Nevertheless, the paper is evaluated as positive.
Author Response
Please see the attachment.
Author Response File: 
Author Response.docx
Reviewer 2 Report
Title: Aircraft Fleet Health Monitoring using Anomaly Detection Techniques
Technical Comments:
- A proper literature review of the current state of the art including limitations or gaps in existing approaches should be presented. This review should: direct future readers to relevant useful other works in the area, give the readers a perspective of the place of their work in the wider field that includes other approaches, provide a sense of the dynamics of an evolving field. The current manuscript seems to lack this level of rigor. It would be beneficial for a reader of the manuscript to understand the different models that have been used in literature for aviation anomaly detection (clustering, SVM, Bayesian neural networks, etc.) in order to fully understand and appreciate the power of autoencoders.
- Page 13 line 353 - A few flights with NaN values in sensor data were also removed - how many and what % of total flights was this. Were the nans at each point in the sensor reading or only a subset of points? Could it have been filled with mean value/forward fill/backward fill?
- Figure 8 needs to be improved in quality. Showing the labels for the data in the left side plot does not serve any purpose and is making the figure cluttered
- Figure 14,15,16 are very poor quality and hard to read/follow. Recommend re-plotting these with better font size, and higher DPI
- It is recommended that the authors add a flowchart or graphic at the beginning that outlines the overall paper and the thought process as well as original contributions so that it is easy for the reader to follow
- While the paper does a good job of describing the autoencoder results and implications, these are not the innovations of the authors. The discussion seems to be lacking on the implications of this work on the aircraft fleet health monitoring problem. Specifically, what do we know now after the study that we didn't know before and how can that knowledge be useful to future efforts.
- The authors should also add some concrete recommendations on model selection, impact of features/sensors on this choice etc. This discussion seems limited in the manuscript right now.
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Round 2
Reviewer 2 Report
Thank you for the responses and for addressing the comments raised.
