Formulating an Engineering Framework for Future AI Certification in Aviation

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This is an exciting and well-established paper that fits the journal's scope. The topic covered is extremely important, and I highly rate this paper. TurnItIn similarity index is just 12%, which is very low. Reviewing this manuscript was a joy, but I have a few issues that the authors should consider. Thus, I recommend a minor review.

1. Can the authors provide a qualitative discussion on the cost of implementing the proposed approach?
2. In my opinion, the monitoring step should be described in more detail as this is one of the most crucial (if not the most vital) in the whole process (page 16, lines 707-714).
3. Can the authors provide a graph for the MLDL description?
4. Can the authors elaborate more on the drawbacks of their proposed approach in the discussion section?

I hope that those comments will help authors to improve their manuscript.

Author Response

We thank the reviewer for the positive feedback and a glad that they enjoyed reviewing our manuscript. We are thankful for the time and effort put into it.

Comment 1: Can the authors provide a qualitative discussion on the cost of implementing the proposed approach?
Response 1: A qualitative discussion of the potential cost of implementing our approach is, in our opinion, outside the scope of this paper (as it will depend on many factors governed by the use case). However, we are already working on a follow-up paper (to be submitted again in MDPI Aerospace), which will utilize our approach and apply it to a real-world problem and include the requested qualitative discussion. Nevertheless, we added some first thoughts on the issue to the discussion.

Comment 2: In my opinion, the monitoring step should be described in more detail as this is one of the most crucial (if not the most vital) in the whole process (page 16, lines 707-714).
Response 2: We added more details about the monitoring step to both the general description (section 5.2, p. 14, 614-628) and the proposed section (p. 17, 737-749).

Comment 3: Can the authors provide a graph for the MLDL description?
Response 3: Unfortunately, we weren't able to acquire the corresponding rights to publish the MLDL graph under the journal's CC BY license.

Comment 4: Can the authors elaborate more on the drawbacks of their proposed approach in the discussion section?
Response 4: We thank the reviewer for critically questioning our approach. We added some potential drawbacks to the discussion.

Reviewer 2 Report

Comments and Suggestions for Authors

The paper addressed developing a comprehensive engineering framework for certifying AI-based systems in aviation, particularly by integrating agile, DevOps principles with the existing W-shaped process proposed by EASA. As AI adoption in safety-critical domains like aviation grows, having a robust framework for certification becomes crucial. This paper addresses a pressing regulatory and engineering need. The paper provides a critical evaluation and extension of the W-shaped process.

1. The result is a combined framework that introduces several new phases.
2. Builds on existing literature and identifies gaps in current research.

This is a theoretical example. Are there pilot study examples/descriptions that can be included? Add more color or typographical distinctions for the new elements (ConOps, ODD, etc.) vs. traditional W-process steps in Figure 3 to make it stand out.

The phrase “Normally hull loss” in table 1 seems grammatically incorrect.

Figure 3 needs to be recreated better in terms of visibility and reader convenience. Currently figure text is small and the color scheme makes it difficult to read.   The phrase “Normally hull loss” in table 1 seems grammatically incorrect.   Is it possible to add a graphic in the earlier section that shows some a depiction of literature review or bibliometric analysis?

 

Author Response

We thank the reviewer for the positive feedback on our manuscript. We are pleased that the reviewer also sees the problems addressed by our manuscript.

Comment 1: This is a theoretical example. Are there pilot study examples/descriptions that can be included?
Response 1: Unfortunately, not; however, we are already working on a follow-up paper (to be submitted again in MDPI Aerospace), which will utilize our approach and apply it to a real-world problem.

Comment 2: Figure 3 needs to be recreated better in terms of visibility and reader convenience. Currently figure text is small and the color scheme makes it difficult to read. Add more color or typographical distinctions for the new elements (ConOps, ODD, etc.) vs. traditional W-process steps in Figure 3 to make it stand out.
Response 2: We thank the reviewer for the feedback on improving Figure 3. As suggested, we highlighted the new elements in a different color and explained the corresponding color coding in the text.

Comment 3: The phrase “Normally hull loss” in table 1 seems grammatically incorrect.
Response 3: The phrase has been incorrectly taken from EASA's Certification Specifications for Large Aeroplanes CS-25. It has been corrected to the phrase used in the primary literature: "Normally with hull loss"

Comment 4: Is it possible to add a graphic in the earlier section that shows some a depiction of literature review or bibliometric analysis?
Response 4: We, the authors, believe that visual depictions of our literature review do not add any significant value to the paper compared to the increased mental load required from the reader. Those kinds of figures are often hard to read and are not necessarily relevant to the context of our paper. For this reason, we would like to avoid adding the requested figure.

Reviewer 3 Report

Comments and Suggestions for Authors

The work formulated an engineering framework for future AI certification in aviation. Here are a few comments.

1) The authors are advised to list down key challenges and also list down key contributions that address those challenges in the introduction. This will enhance the readability of the paper.

2) What can be the possible steps to standardize operational design?

3) What can be the best criteria to achieve a balanced tradeoff between real data and simulated data? How can work contribute to safety-critical issues while adopting AI? Especially functional safety issues.

4) The literature needs to be further strengthened, such as by addressing critical aviation operations. The best papers that can be referenced to support aviation operations are a novel parallel series data-driven model for IATA-coded flight delay prediction and feature analysis, and a second about the prediction of aircraft trajectory and the associated fuel consumption using covariance bidirectional extreme learning machines.

Author Response

We thank the reviewer for the time taken to review our manuscript and for providing valid points, helping us improve our work.

Comment 1: The authors are advised to list down key challenges and also list down key contributions that address those challenges in the introduction. This will enhance the readability of the paper.
Response 1: We thank the reviewer for the feedback to improve the readability of the paper. We added a summary of the key challenges and our contributions to address those challenges in the introduction.

Comment 2: What can be the possible steps to standardize operational design?
Response 2: We assume the reviewer meant the operational domain (OD) and operational design domain (ODD). If so, those have been standardized in the context of automated driving, see https://www.asam.net/standards/detail/openodd/. For aviation, however, the definition of both the OD and ODD is part of current research. See, for example, https://elib.dlr.de/197957/ and https://doi.org/10.1109/dasc62030.2024.10749684.

Comment 3: What can be the best criteria to achieve a balanced tradeoff between real data and simulated data? How can work contribute to safety-critical issues while adopting AI? Especially functional safety issues.
Response 3: We, the authors, feel the first question is outside the scope of our manuscript, as we believe this question is highly dependent on the use case and cannot be answered in general. Regarding the second question, we added some points to the discussion to address this question.

Comment 4: The literature needs to be further strengthened, such as by addressing critical aviation operations. The best papers that can be referenced to support aviation operations are a novel parallel series data-driven model for IATA-coded flight delay prediction and feature analysis, and a second about the prediction of aircraft trajectory and the associated fuel consumption using covariance bidirectional extreme learning machines.
Response 4: We thank the reviewer for the valuable recommendation and included the two papers in Section 5.

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

Th authors have addressed my all comments.