A Neural Network with Physical Mechanism for Predicting Airport Aviation Noise

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

I have no further comments and recommend this paper for publication.

Author Response

Dear Reviewer,

Thank you very much for your hard work in reviewing this paper. We submit a revised version of our paper with the title " A Neural Network with Physical Mechanism for Predicting Airport Aviation Noise," ID: 3141481, which has been modified according to your comments and suggestions.

 

The following is a point-by-point response to your comments and suggestions:

 

Comment #1: I have no further comments and recommend this paper for publication.

Author response: Thank you for your review and recognition of the paper's contribution.

 

With best regards,

 

Dan Zhu

College of Civil Aviation

Nanjing University of Aeronautics and Astronautics

E-mail: zwlnuaa@nuaa.edu.cn

Aug. 20, 2024

 

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

Dear authors,

please find attached my review. 

Comments for author File: Comments.pdf

Author Response

Dear Reviewer, 

Thank you very much for your hard work in reviewing this paper. We submit a revised version of our paper with the title " A Neural Network with Physical Mechanism for Predicting Airport Aviation Noise," ID: 3141481, which has been modified according to your comments and suggestions. 

The following is a point-by-point response to your comments and suggestions: 

Comment #1: The paper describes the use of different neural network modeling to predict the emitted noise in the airport surroundings by single airplane takeoff or landing events. A methodologies able to include existing physical models for noise prediction in the neural networks is proposed and compared to a pure data-driven approach, a different physics-data hybrid neural network and the original physical modeling. The objective of the article is clear and relevant, the combination of data driven and physical models is a very interesting path in the research of noise modeling. The paper is overall well written and the text is almost always easy to read. 
Author response: Thank you for your review and recognition of the paper's contribution.

Comment #2: In Table 2, page 10, row 350, it is clearly stated under step 8 that the test set data are used to train the PGNN. In particular, it is stated that if the error is not as expected, the hyperparameters of the neural network, such as the number of hidden layers, neurons, and the physically constrained weights of the loss function, are adjusted and the training repeated. This is in contrast with the basic rules of AI and Machine Learning approaches: the test set MUST be used only for evaluating the test error, which CAN’T be used to further improve the predictions of the model, for example adjusting the hyperparameters of the network. The neural network MUST be completely blind with respect to the test set. If the authors want to implement an automatic hyperparameter adjustment procedure, they must create a validation set from the data, aside from the training and test sets. The validation set can be used for the mentioned purpose, and the test set must be kept isolated only for the final performance evaluation before step 9 of the authors’ algorithm. If the test set is used to train the model, the evaluated metrics on the test set do not give any hint on the generalization capability of the network, since it has already seen those data. If the flaw is present also in the training of the DNN and HPNN, please fix them as well.
Author response: Thank you for your comment. Changed the description in Table 2, page 10, line 336, under Step 8: Use random sampling to divide the dataset into training set, validation set, and test set in a ratio of 8:1:1. The training set is used for model training, the validation set is used for performance evaluation during model optimization, and the test set is used for performance evaluation of the final model. During model training, the training set is first used for model training, and then the validation set is used to evaluate model performance. If the validation error is as expected, model training is complete; otherwise, adjust the model parameters or hyperparameters and repeat steps 6-8 (without re-dividing the dataset) until the model performance reaches the expected standard. 

Comment #3: The authors should fix some graphical issues in their images. Most of the pictures appear poorly cut and in low resolution. Panels with multiple images suffer of subfigures with different sizes.
Author response: Thank you for your valuable feedback on our manuscript. We have carefully reviewed and addressed the graphical issues as suggested. Specifically, we have improved the resolution of the images, ensured that all subfigures are of consistent size, and corrected any poorly cut images. We hope these changes meet the journal's standards.

Comment #4: Figures numbers have some issues: there are two figure 1, two figure 2, and two figure 3.
Author response: Thank you for the detailed review. We found two instances each of figures 1, 2 and 3. We have corrected the figure numbers to ensure that each figure has a unique and consecutive number throughout the manuscript.

Comment #5: There is no figure 11 in the manuscript. Figure 11 is cited in the text and should show one of the main results of the paper. Please fix it.
Author response: Thank you for your advice. We note that Figure 11 is missing from the manuscript. We have now included Figure 11, which shows one of the main results of the paper. (Please see page 14, row 450, Fig. 11).

Comment #6: Please homogenize scales in figure 1.
Author response: Thank you for your suggestion regarding the scale. We have standardized the scales as you requested to ensure consistency and clarity. (Please see page 13, row 431, Fig. 8).

Comment #7: Figures 1, 2, and 3 on page 13 should be repeated for the results on the test set.
Author response: Thank you for your suggestion. We would like to clarify that Figures 1, 2, and 3 in the original article already display the results of the experiments on the test set. However, we have now redrawn and described these figures as Figures 8, 9, and 10 in the revised manuscript to enhance clarity and presentation. We appreciate your attention to detail and your valuable feedback.

Comment #8: Results should be clearly separated between training/validation and test set. It must be stated clearly in the text and in the captions (for example, are figs 9 and 10 referring to the training set?).
Author response: Thank you for your valuable feedback. We have clearly distinguished between the results between the training/validation set and the test set throughout the manuscript. It is clearly stated which results belong to the training/validation set and which results belong to the test set. In addition, the titles of Figures 8, 9 and 10 (and any other relevant figures) have been changed to make it clear whether they refer to the training/validation set or the test set.

Comment #9: This is just a suggestion. There is no actual need to split figure 2 at page 13 and fig 10: you can effectively show the results in terms of absolute value and percentage error using two y-axes, one on the left and one on the right, in the same figure.
Author response: Thank you for your suggestion. We have implemented your suggestion by combining the results into a single graph with two y-axes - one showing the absolute values and the other showing the percent error. This approach effectively presents the data in a clear and concise manner. (Please see page 13, Fig. 9 and Fig 10).

Comment #10: Table 2 should be repeated also for DNN and HPNN, clarifying the algorithm used for all the methods.
Author response: Thank you for your suggestion. We appreciate the importance of providing a comprehensive comparison. However, we would like to clarify that the focus of this paper is on the innovative PGNN model. The DNN and HPNN models are included primarily for performance comparison with the PGNN. Therefore, we have adjusted the structure of the manuscript to include a brief introduction of these two models in section 3.3. Model training and Experimental Design, We believe this approach effectively highlights the PGNN model while still providing the necessary context for the comparisons. We hope this meets your expectations, and we are open to further suggestions if needed. (Please see page 12, row384-395).

Comment #11: It would be nice to see the training and validation error convergence during the training phase for the final network configurations.
Author response: Thank you for your suggestions. We have added Figure 7 which shows the convergence of the training and validation errors during the training phase. This will clearly illustrate how the error evolves as the network is trained. (Please see page 11, Fig. 7).

Comment #12: How is the dataset divided into training and test sets? What criterion is used?
Author response: Thank you for your valuable feedback. The dataset is divided into training set, validation set and test set using random sampling method with a ratio of 8:1:1. (Please see page 10, step 8).

Comment #13: Are the operations in the test set from days that are also contained in the training set? This must be avoided for the same reason as the major flaw.
Author response: Thank you for highlighting this important concern. We have carefully reviewed the dataset splitting process to ensure that the operations in the test set do not overlap with those in the training set.

Comment #14: Please, explain all the abbreviations in the paper the first time you use it (a non-exhaustive example is ADS-B which is not defined in the manuscript).
Author response: Thank you for your valuable suggestion. We have carefully reviewed the manuscript and ensured that all abbreviations are defined the first time they are used. For example:"ADS-B" (Automatic Dependent Surveillance-Broadcast) has now been clearly defined when it is first mentioned in the text, along with any other abbreviations that required clarification.

Comment #15: Sentence in row 45 " Machine learning has poor generalization capability due to the lack of clear physical meaning " may be confirmed (or not) by a notable increase in the test error compared with the training error for the DNN model. The authors may further comment on this in the revised manuscript.
Author response: Thank you for your insightful comments. We carefully checked the generalization ability of the DNN model by its performance on new monitoring stations: the purely data-driven DNN model showed weaker performance than the ECAC model on other monitoring datasets, which shows that pure machine learning models are highly dependent on data quality and perform poorly on inferring data. (See page 14, line 441).

 

Comment #16: I disagree with the sentence in row 75. How can the PGNN model reduce the effort required to calibrate the physical-driven model parameters if its training is based on the previous existence of a reliable and calibrated physical-driven model?
Author response: Thank you for your feedback regarding the sentence in row 75.  we agree with your assessment and have removed the sentence from the manuscript.We appreciate your careful review and insightful feedback.

Comment #17: I disagree with the sentence in row 95. The prediction results from PGNN is not ”ensured” to conform to the law of physics as the training is only guided towards this objective, but not enforced. Strictly speaking, that is a soft constraint that can be violated by the PGNN model. Do the authors have further proof of their statement?
Author response: Thank you for your insightful feedback. I agree with you that the soft constraints in the PGNN model may be violated. This is evident from the modified box plots in Figures 10 and 11, which show significant outliers in the results of both the HPNN and PGNN models. These outliers indicate that soft constraints may be violated in some cases. However, it is worth noting that the overall predictive performance of the PGNN model remains high, with only occasional deviations under specific conditions.

Comment #18: I partially disagree with the sentence in row 90. On one side it is true that the PGNN prediction is guided by the physical model. On the other side, the ”correction” that the network provides on top of the ECAC model can be considered the new black box, as it is guided by the input features. This is fine, but I think this aspect should be better clarified.
Author response: Thank you for your thoughtful feedback. I appreciate your perspective regarding the statement in row 90.You are correct that while the PGNN prediction is guided by the physical model, the "correction" introduced by the network on top of the ECAC model can indeed be seen as a new black box, driven by the input features. This is an important nuance, and I agree that it would benefit from further clarification in the manuscript. In the revised version, I will elaborate on this point, highlighting that while the PGNN leverages physical knowledge to guide the network, the corrections it applies are not fully transparent, and this aspect introduces an additional layer of complexity. I will ensure that this distinction is clearly communicated to avoid any potential misunderstanding. (Please see page 3, row 98).

Comment #19: In the end of the introduction the authors say that they proposed the PGNN. It seems reductive to me, as they are proposing also the HPNN for the noise prediction at the airports, and the two approaches are compared also against a pure data-driven DNN. This aspect can be better emphasized.
Author response: Thank you for your insightful feedback. The main focus of our paper is the PGNN model, which is innovative compared to the DNN and HPNN models. In our study, HPNN and DNN models are used for comparative experiments, so we have simplified the description of HPNN and DNN models in our paper.

Comment #20: As far as I know figure 1 is valid for PGNN only. If so, I feel that the first part of the ”Methodology” section should be more general, and less focused on only one of the methods used in the work (in fact, the authors correctly dedicated a separate subsection to each of the methods, and the introductory part of ”Methodology” feels more appropriate in the PGNN subsection).
Author response: Thank you for your feedback. We have revised the structure of the paper to better align with our focus on the PGNN model. Specifically, the "Methodology" section now exclusively discusses the PGNN model, reflecting its role as the main contribution of our work. We have moved the discussion of the DNN and HPNN models to section 3.3. Model training and Experimental Design, where they are introduced and compared as part of our experimental framework. This structural change aims to provide a clearer and more focused presentation of our methodologies and to highlight the innovations of the PGNN model.

Comment #21: Can the methods used by the authors discriminate between different aircraft, for example B777 of different generations, or A320 with different aerodynamic evolutions or engine options? In other words, parameters such as aircraft name, engines type and number, ... are considered for the single event prediction? Can they be added in the methodology? Do the authors think that may help in further improving the methodology?
Author response: Thank you for your thoughtful question. The current method of distinguishing between different aircraft types is through the ANP (Aircraft Noise Performance) database, which is an international data repository following the ICAO 9911 document, developed and maintained by the U.S. Department of Transportation, the European Union Agency for the Safety of Air Navigation (EASA), and the European Civil Aviation Conference (ECAC). It provides flight performance parameters for more than 150 airplane models, including aircraft parameters, engine parameters, aerodynamic parameters, etc. These parameters will enable the model to better consider the aircraft model. These parameters will allow the model to better account for noise and performance variations associated with different aircraft configurations. (Please see page 4, row 170).

Comment #22: Similar to the previous question, are airplane settings such as flap-slat settings, landing gear deployment, ..., included in the modeling? Do the authors think that may help in further improving the methodology?
Author response: Thank you for your question. Our current modeling approach uses the general flap settings provided in the ANP (Aircraft Noise Performance) database. Due to data limitations, we are unable to accurately capture and include specific flap settings, landing gear deployments, and other similar parameters for each flight. While our current model relies on generic data, we recognize the value of including more flight-specific parameters and will consider exploring ways to incorporate these data in future work, if available. 

Comment #23: I do not agree with the sentence in row 321. ”Physical knowledge is embedded in the model to simulate the stochastic correlation between aviation noise and data” can, in my opinion, be attributed to the HPNN approach, but not to PGNN. In HPNN, in fact, the introduction of meteorological data in the features may cause the same trajectory to provide different noise footprints for different ambient conditions. However, in PGNN the physical model is used as a regularization technique guiding the training towards networks whose predictions are closer to the physical model ones, limiting the ”physical inconsistency”, as correctly stated by the authors. Can the authors comment on that?
Author response: Thank you for your detailed feedback. We agree that the sentence in question could be more accurate. The sentence is intended to convey the role of physics knowledge in guiding the model, rather than directly modeling stochastic correlations. In Physics-Guided Neural Networks (PGNN), physical knowledge is indeed used as a regularization technique. This regularization helps guide the training process by ensuring that the network's predictions are consistent with the established physical model, thus reducing physical inconsistency. However, it does not directly model random correlations. We have revised the manuscript to clarify this distinction and to better reflect the specific role of physical knowledge in PGNN. (Please see page 8, row 302).

Comment #24: In row 336 I suggest to replace ”1:η” with ”η = 1
Author response: Thank you for your suggestion on the symbols in line 336. In our study, the weights of the empirical and physical errors are set to 1 and η, respectively, where η is a hyperparameter representing the weight of the physical error. Table 3 provides experimentally determined values for η. (Please see page 12, row 392).

With best regards,

Dan Zhu
College of Civil Aviation
Nanjing University of Aeronautics and Astronautics
E-mail: zwlnuaa@nuaa.edu.cn 
Aug. 20, 2024

 

Author Response File: Author Response.docx

Reviewer 3 Report

Comments and Suggestions for Authors

The paper introduces an innovative approach to overcoming the limitations of physics-guided and data-driven methods in predicting airport noise. The proposed PGNN model represents a novel and potentially influential advancement in the field by integrating the ECAC model with deep neural networks. The authors effectively demonstrate its superior performance compared to existing techniques. The paper presents a clear problem statement and a strong motivation for the research. It effectively combines physics-based knowledge with data-driven techniques. The PGNN model shows impressive improvements over both physics-driven and data-driven models. The experimental evaluation is thorough, including comparisons with multiple models.

To enhance the paper's reproducibility and depth, the authors should provide more specific details about the neural network architecture and training process, including visualizations of learned representations. A more comprehensive understanding of the dataset, including its size, distribution, and preprocessing steps, is essential. Additionally, reporting a wider range of evaluation metrics and conducting ablation studies would strengthen the model's assessment. Finally, analyzing the model's performance under various conditions would demonstrate its generalizability and robustness.

   

Further suggestions:

• In Figure 1, include detailed notation for the ECAC model within the figure legend itself.
• In Table 1, represent weights using the correct format (e.g., xxx, xxx).
• In Figure 7, adding geographical information about the stations would help readers understand distances and locations better.
• In Section 3.3, consider representing the experimental design pictorially instead of using text.
• Verify the figure numbers on page 13 (after line 435) and page 14 for accuracy.

It is recommended that the authors review the manuscript to ensure all information is correctly cross-referenced, as it is possible an older version of the manuscript was submitted.

       

Author Response

Dear Reviewer,

 

Thank you very much for your hard work in reviewing this paper. We submit a revised version of our paper with the title " A Neural Network with Physical Mechanism for Predicting Airport Aviation Noise," ID: 3141481, which has been modified according to your comments and suggestions.

 

The following is a point-by-point response to your comments and suggestions:

 

Comment #1: The paper introduces an innovative approach to overcoming the limitations of physics-guided and data-driven methods in predicting airport noise. The proposed PGNN model represents a novel and potentially influential advancement in the field by integrating the ECAC model with deep neural networks. The authors effectively demonstrate its superior performance compared to existing techniques. The paper presents a clear problem statement and a strong motivation for the research. It effectively combines physics-based knowledge with data-driven techniques. The PGNN model shows impressive improvements over both physics-driven and data-driven models. The experimental evaluation is thorough, including comparisons with multiple models.

Author response: Thank you for your review and recognition of the paper's contribution.

 

Comment #1: To enhance the paper's reproducibility and depth, the authors should provide more specific details about the neural network architecture and training process, including visualizations of learned representations. A more comprehensive understanding of the dataset, including its size, distribution, and preprocessing steps, is essential. Additionally, reporting a wider range of evaluation metrics and conducting ablation studies would strengthen the model's assessment. Finally, analyzing the model's performance under various conditions would demonstrate its generalizability and robustness.

Author response: Thank you for your thorough review of our manuscript and your valuable comments. We have carefully considered your comments and made the necessary changes to improve the quality of the manuscript. In the following, we will provide detailed responses to your questions one by one:

(1) Neural network architecture and training process. We have added more details about the PGNN neural network architecture and training process in the revised manuscript. Specifically, we describe the number of layers, activation function, optimizer type, learning rate, batch size, and elapsed time. This information will now be presented in section 3.3. Model training and Experimental Design. (Please see page 11, row 374).

(2) Visualization of Learning Representations. We have added visualizations to illustrate the learning representations in the neural network. These visualizations are included in Figure 7. (Please see page 11, row 381).

(3) Dataset details. We have extended the description of the dataset in 3.1. Data description. This includes information about the size of the dataset, the types of datasets, and the methods we used. (Please see page 10, row 338).

(4) Broader assessment metrics. Thank you for your suggestion for additional assessment metrics. After careful consideration, we believe that the current set of metrics provides a comprehensive and robust assessment of the performance of our model, especially in the context of the specific goals of our study.

(5) Ablation studies. We appreciate your suggestion to include more ablation studies. In the article experiments, the ECAC model is a purely physical approach, the DNN model is a purely neural network approach, and the HPNN model integrates the outputs of the physical model as input features to create a hybrid physical-neural network. Given this setup, we believe that for the PGNN model (where the loss function incorporates a physically-driven model), comparison with the above three models can be effective for the purpose of ablation studies.

(6) Model performance under various conditions. We have tested the model under various conditions, including different datasets and environmental settings. The results are presented in section 3.4. Analysis, demonstrating the robustness and generalizability of the model under different conditions. (Please see page 12, row 394).

 

Comment #3: In Figure 1, include detailed notation for the ECAC model within the figure legend itself.

Author response: Thank you for your suggestion to include detailed notation for the ECAC model within the figure legend of Figure 1. We would like to clarify that Figure 1 is intended as a schematic illustration. To maintain clarity and brevity, we have not included detailed notations in the figure legend. However, we have provided a comprehensive explanation of the symbols and notation used in the ECAC model later in the main text. We appreciate your understanding and are happy to make further adjustments if necessary.

 

Comment #4: In Table 1, represent weights using the correct format (e.g., xxx, xxx).

Author response: Thank you for pointing out the formatting issue in Table 1. We have revised the table to represent the weights using the correct format as suggested. The updated version is now consistent and clear. (Please see page 7, row 276).

 

Comment #5: In Figure 7, adding geographical information about the stations would help readers understand distances and locations better.

Author response: Thank you for your suggestion. We have included latitude and longitude information for each point in the Figure 7 caption to provide a clearer context for the data presented. (Please see page 10, row 349, Fig.6).

 

Comment #6: In Section 3.3, consider representing the experimental design pictorially instead of using text.

Author response: Thank you for your suggestion to represent the experimental design pictorially in Section 3.3. While we appreciate the idea, we believe that due to the complexity and the specific details involved, it is more effective and clear to present this information in text form. This approach allows us to convey the necessary details with precision. We appreciate your understanding and are open to further feedback.

 

Comment #7: Verify the figure numbers on page 13 (after line 435) and page 14 for accuracy.

Author response: Thank you for bringing this issue to our attention. We have verified and corrected the figure numbers on page 13 (after line 435) and page 14 to ensure accuracy. The figure numbers in the manuscript are now correct.

 

 

With best regards,

 

Dan Zhu

College of Civil Aviation

Nanjing University of Aeronautics and Astronautics

E-mail: zwlnuaa@nuaa.edu.cn

August 20, 2024

 

Author Response File: Author Response.docx

Reviewer 4 Report

Comments and Suggestions for Authors

The authors propose a new model based on physic-guided neural networks combining the classical ECAC model with machine learning techniques to improve the accuracy of aircraft noise. The proposed model is interesting

 Here are some comments

1-The authors should double check figures 8, 9 and 11. They are wrongly named figures 1, 2 and 3.

2-Can you provide references for equations 2, 3, 4 and 5? How do you get equations 4 and 5?

3-How are the results in Table 1 obtained? Is there any reference ?

4-In the airport, you have multiple aircrafts. What is the effect of the interactions between aircrafts and how do you consider this in your model?

5-The noise monitoring equipment continuously and uninterruptedly receives noise information every second. What is this equipment? What kind of sensors are used and how are they calibrated? How do you ensure that these measurements are not affected by external conditions?

 

Author Response

Dear Reviewer,

 

Thank you very much for your hard work in reviewing this paper. We submit a revised version of our paper with the title " A Neural Network with Physical Mechanism for Predicting Airport Aviation Noise," ID: 3141481, which has been modified according to your comments and suggestions.

 

The following is a point-by-point response to your comments and suggestions:

 

Comment #1: The authors propose a new model based on physic-guided neural networks combining the classical ECAC model with machine learning techniques to improve the accuracy of aircraft noise. The proposed model is interesting

Author response: Thank you for your review and recognition of the paper's contribution.

 

 

Comment #2: The authors should double check figures 8, 9 and 11. They are wrongly named figures 1, 2 and 3.

Author response: Thank you for pointing out the problem with the numbering of the figures. We have double-checked and corrected the names of figures 8, 9 and 11. These figures are now correctly named in the manuscript.

 

Comment #3: Can you provide references for equations 2, 3, 4 and 5? How do you get equations 4 and 5?

Author response: Thank you for your question regarding the references for Equations 2, 3, 4, and 5. These equations are derived from the ECAC DOC. 29 report. Source: Ecac, E. CEAC DOC. 29: Report on Standard Method of Computing Noise Contours Around Civil Airports, Volume 2: Technical Guide. In Proceedings of the European Civil Aviation Conference (ECAC): Neuilly-sur-Seine, France, 2016.

 

Comment #4: How are the results in Table 1 obtained? Is there any reference?

Author response: Thank you for your question regarding the results in Table 1. The data presented in Table 1 are obtained from the ANP (Aircraft Noise Performance) database. We have included this information in the manuscript to clarify the source of the data. (Please see page 7, row 276).

 

Comment #5: In the airport, you have multiple aircrafts. What is the effect of the interactions between aircrafts and how do you consider this in your model?

Author response: Thank you very much for your interest in the issue of aircraft interactions at the airport. At Hefei Xinqiao International Airport, due to the nature of its single-runway operation, all takeoff and landing flights are required to meet established horizontal and vertical separation requirements before they can be executed, a measure that greatly reduces the possibility of direct aircraft interactions within the airport area. As a result of these stringent air traffic management rules, the effect of aircraft interactions on noise levels is significantly attenuated and can be considered almost non-critical. Therefore, in our model, in order to maintain its simplicity and accuracy, it has been decided not to take the contribution of aircraft interactions to noise into account.

 

Comment #6: The noise monitoring equipment continuously and uninterruptedly receives noise information every second. What is this equipment? What kind of sensors are used and how are they calibrated? How do you ensure that these measurements are not affected by external conditions?

Author response: Thank you for your inquiry about noise monitoring equipment. The airport uses OSEN-Z01 Class 1 sound level meter for noise monitoring. The sound level meter complies with GB/T3785.1-2010/IEC61672-1:2013, with a frequency range of 10HZ~20KHZ and a measurement range of 30DB(A)~130DB(A). It can continuously record the noise level without interruption every second. The device utilizes a high sensitivity microphone to detect and measure sound levels, and is initially calibrated, periodically calibrated and field calibrated to ensure precision and accuracy. To ensure that measurements are not affected by external conditions, the transducer is housed in a weatherproof enclosure to protect it from environmental factors such as rain, wind and temperature fluctuations.

 

With best regards,

 

Dan Zhu

College of Civil Aviation

Nanjing University of Aeronautics and Astronautics

E-mail: zwlnuaa@nuaa.edu.cn

Aug. 20, 2024

 

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Dear authors, please find attached my review. 

I am asking the editor to give the authors a sufficient time to perform the review of their work because addressing all the points thoroughly will require a non negligible effort. 

Comments for author File: Comments.pdf

Author Response

Dear Reviewer,

 

Thank you very much for your hard work in reviewing this paper. We submit a revised version of our paper with the title " A Neural Network with Physical Mechanism for Predicting Airport Aviation Noise," ID: aerospace-3141481, which has been modified according to your comments and suggestions.

 

The following is a point-by-point response to your comments and suggestions:

 

Comment 1: My biggest comment in this second round is still connected to the use of training/validation/test sets. In particular I sill have some doubts about what the authors really did and how did they use the data in the models training. The authors replied to my question changing the point 7 in their algorithm description. But they did not perform any new train of the neural networks. This can be clearly seen comparing figures 1 (page 13) of the old manuscript and figure 8 of the revised one, or figures 2 and 10 of the old with figures 9 and 10 of the revised, or again figure 13 (page 14) of the old version and figure 11 of the new one. The mentioned figures are showing exactly the same data and results, confirming that the models were not changed or trained differently during the revision. Should I assume that the authors performed a correct training also in their original work, and the only error was done in writing the manuscript? However, the authors did not state clearly that this was the case in their response. I, as a reviewer, have not the time, the data, and the tools to check this: I cannot replicate the authors’ numerical experiments. In addition, the paragraph in row 437, referred to figure 11, further confuses me about the issue: “Next, the four models are tested for noise prediction on new monitoring stations (independent of the four monitoring stations in the training set) to evaluate their generalization performance.” Since the figure 11 caption clearly states “test set”, is the test set composed of the same events of the training and validation but different monitoring stations are used? Or figure 11 is showing the results obtained on a subset of the test set, which is composed of independent event and including only monitoring stations not included in the training and validation sets? Moreover, the authors’ response to my comment #11 cast a shadow on the authors’ honesty:

Comment: Are the operations in the test set from days that are also contained in the training set? This must be avoided for the same reason as the major flaw. Author response: Thank you for highlighting this important concern. We have carefully reviewed the dataset splitting process to ensure that the operations in the test set do not overlap with those in the training set. If the authors really changed the dataset splitting process, how can the previously mentioned figures show the exact same results between the original and revised manuscript? It is practically impossible that two different training and validation set provides the same model and hence predictions on different test sets (even just because the test set is changed!). Since this was and still is a diriment issue, I ask the authors to deeply and clarify it once for all.

Author response : Thank you for your detailed feedback. I apologize for the confusion caused by the unclear description of the data splitting and training process in the manuscript. It should be noted that due to a misunderstanding of your suggestions, there were some errors in our responses and revisions, which led to further confusion.

In the original manuscript, Table 2, page 10, row 350 states: “7. Train the PGNN model with an established physically guided loss function. 8. Test the optimized model using the test set data. If the error is as expected, the model training is complete. Otherwise, repeat steps 6-8 until the desired training model performance is obtained.” This should be revised to: “1. Back propagation optimization is performed with the goal of minimizing the Loss function construction. The Adam optimizer is used to update the model weight pa-rameters and continuously adjust the parameters to optimize the model.By using a K-fold cross-validation method (specifically set to k=5), we iteratively train the model using the training set data. In each round of cross-validation, the model is trained on a training subset and evaluated on a validation subset. 2. If the error metrics obtained after cross-validation meet the expected performance criteria, it indicates that the model training is complete and meets the predefined accuracy requirements. Conversely, if the error does not meet expectations, we will repeat the training and optimization steps of the model until we obtain results that satisfy the desired training model performance.” (Please see page 12 “Neural network training”).

In the original manuscript, page 10, row 358 states: “The dataset comprises a total of 3,195 flight information records, with 80% of the data reserved for training and the remaining 20% for testing.” This should be revised to: “The dataset contains a total of 3,195 flight information records, of which the data from monitoring point 2 is divided in the ratio of 80% for training and 20% for testing. Meanwhile, the data from monitoring points 1, 3, and 4 are dedicated to testing the generalization performance of the model.” (Please see page 12 row 392).

For page 14, row 442 in the original manuscript, which states: “After constructing the models, they were tested on the independent monitoring datasets at monitoring points 1, 3, and 4 to evaluate their generalization performance,” it should be revised to: “Next, A comprehensive generalization performance evaluation of the four models was performed using the datasets collected at monitoring points 1, 3, and 4.” (Please see page 17 row 502).

For page 12, row 400 in the original manuscript, which states: “Table 4 compares the average prediction performance of three metrics for four models using data from monitoring point 2: the purely physics-driven model ECAC, the purely data-driven model DNN, and the two physical fusion models HPNN and PGNN.” it should be revised to: “Table 6 compares the average predictive performance of the four models for three metrics using the test set data from monitoring point 2: the purely physically-driven model ECAC, the purely data-driven model DNN, and the two physically-fused models HPNN and PGNN. ” (Please see page 15 row 460).

The experimental results and pictures in the original manuscript are accurate, but there are ambiguities in the written expression, which has now been revised.

 

Comment #2: In the introduction, row 74, the authors claimed that the PGNN ”[...] model accurately captures the uncertainty in the prediction process, resulting in a more comprehensive and robust performance”. My comment is about the words ”robustness” and ”uncertainty”. I interpret the ”robustness” part in this way: the authors are claiming that the PGNN is able to improve the generalization capability of the method compared to HPNN and DNN, and that, moreover, it is capable to address uncertainties due to some aleatory variable. To support the claim of improved ”robustness”, the authors should include in their manuscript evidences that the training and validation errors of PGNN, HPNN and DNN is somehow similar, but the error by PGNN on the test set is better than the others. The last part of the proof is already present (PGNN outperforms HPNN and DNN on test set), but the first part is not sufficiently highlighted by the shown results. In addition, another issue is the use of the word ”uncertainty” in the prediction process. This can be used in presence of aleatory variables, or when some variables of the problem are not modeled and still have an influence on the value to be predicted, such that an apparently identical inputs give different outputs. To support the claim of ”capturing uncertainties”, the authors should extract from the data the footprints of a single type of aircraft, sharing the same input vector (doing the same nominal trajectory, at the same nominal speed, with the same nominal settings, etc.), and compare the predictions from all the models on this subset of data. In this regard, moreover, my previous comment 21 still applies (”I do not agree with the sentence in row 321. ”Physical knowledge is embedded in the model to simulate the stochastic correlation between aviation noise and data” can, in my opinion, be attributed to the HPNN approach, but not to PGNN. In HPNN, in fact, the introduction of meteorological data in the features may cause the same trajectory to provide different noise footprints for different ambient conditions. However, in PGNN the physical model is used as a regularization technique guiding the training towards networks whose predictions are closer to the physical model ones, limiting the “physical inconsistency”, as correctly stated by the authors. Can the authors comment on that?”). The authors’ response to the comment of the first review round suggests that they do not mean uncertainty as I do (”However, it does not directly model random correlations.”). In this case, although a better clarification of the authors’ point will be sufficient, my curiosity still suggest to perform the analysis stated in the above, its results would be rather interesting by the way!

Author response: Thank you for your detailed comments and valuable suggestions. We value your suggestions and see them as opportunities to further improve the manuscript.

Regarding the question of “robustness”, the exact question you mentioned is important. Our goal is indeed to show the advantages of the PGNN model in generalization ability compared to HPNN and DNN, which can be regarded as a kind of robustness. However, we realize that the training and validation errors between PGNN, HPNN and DNN are not fully described in the current manuscript. In fact, when training the three models, we adjust their loss values ​​to the same level to ensure the comparability of performance. This is described in the section “3.3. Model Training and Experimental Design”. (Please see page 14 row 452).

Regarding the aspect of “uncertainty”, the PGNN model proposed in this paper combines the advantages of the HPNN model, especially keeping the physical guidance consistent with the HPNN input and taking it as the input vector. Therefore, PGNN also has the ability to simulate the random correlation between aviation noise and data.

 

Comment #3: row 443: “While the HPNN and PGNN models exhibited a decline in their overall metrics, their predictive performance was still better than the physics-driven ECAC model, indicating that data-driven models based on physical knowledge can learn the overall output tendency of the system and have good adaptability to the uncertainty of real data.”, see previous comment

Author response: Thank you for your valuable feedback. We have readjusted the description to make it easier to understand. (Please see page 17 row 507).

 

Comment #4: In row 88, the Physics Guided Graph Neural Network (PGNN) is introduced. However, PGNN is already used as abbreviation for Physics Guided Neural Network. Moreover, in the original manuscript I never read about Physics Guided Graph Neural Network, and its difference with Physics Guided Neural Network is never stated. I think it is just a typo, but please confirm it.

Author response: Thank you for pointing this out. You are correct; this was indeed a typo. The term "Physics Guided Graph Neural Network (PGNN)" in row 88 should actually be "Physics Guided Neural Network (PGNN)."I have corrected this mistake in the manuscript. I appreciate your attention to detail.

 

Comment #5: The new arrangement of the Section 2 “Methodology” partly satisfy my previous comments on it. However, since the article uses ECAC, DNN, HPNN, and PGNN, all of these methods must be described in the section, as was done in the original manuscript although with some criticality, and the differences between them must be highlighted.

Author response: Thank you for your feedback and for acknowledging the improvements made in Section 2, "Methodology." I understand the need to include a comprehensive description of all methods used in the article, i.e., ECAC, DNN, HPNN, and PGNN, as well as highlighting the differences between them. I will ensure that the revised manuscript thoroughly meets this requirement. Specifically, I will expand "Section 2.2.2 Neural Network" to provide a detailed description of DNN, HPNN, and PGNN, similar to the original manuscript. (Please see page 17 row 298).

 

Comment #6: As part of the consequences of the previous comment, I disagree with your response to my comment #8 of the previous review round: Comment: ”Table 2 should be repeated also for DNN and HPNN, clarifying the algorithm used for all the methods.” Author response: Thank you for your suggestion. We appreciate the importance of providing a comprehensive comparison. However, we would like to clarify that the focus of this paper is on the innovative PGNN model. The DNN and HPNN models are included primarily for performance comparison with the PGNN. Therefore, we have adjusted the structure of the manuscript to include a brief introduction of these two models in section 3.3. Model training and Experimental Design, We believe this approach effectively highlights the PGNN model while still providing the necessary context for the comparisons. We hope this meets your expectations, and we are open to further suggestions if needed. (Please see page 12, row384-395). I agree that the main focus is on the PGNN model, however, all the results of a scientific paper should be repeatable by an interested reader, and all the necessary details to obtain the same results of the original authors should be included in the manuscript. For this reason, I repeat the comment.

Author response: I understand and agree with your concern about the reproducibility of the results presented in the manuscript. All methods used in the study must be described in detail so that interested readers can replicate the results. Regarding your comment 8 in the previous review, I appreciate your reiteration and acknowledge the importance of providing comprehensive details for all methods discussed. I will modify Table 2 to include a detailed description of the algorithms used for the DNN and HPNN in addition to the information already provided for the PGNN model. This will ensure that the methods for all models are clearly presented and comparable. (Please see page 10 Tables 2 and 3).

 

Comment #7: Functional dependencies in Eq. 2 differ from the description of the subsequent equation terms. For example, Lre(P, d) is described as a term depending on temperature, pressure, humidity, while Eeng(ϕ) is stated to depend also from β and ϵ. Probably the description paragraph should be adjusted.

Author response: Thank you for your feedback. The description of formula 2 and its parameters is from the official document "[31]. Ecac, E. CEAC DOC. 29: report on standard method of computing noise contours around civil airports, volume 2: technical guide. In Proceedings of the European Civil Aviation Conference (ECAC): Neuilly-sur-Seine, France, 2016.". Since there are too many descriptions in the original text, we have simplified some unimportant descriptions. If this simplification leads to inconsistencies in some descriptions, we will make corresponding adjustments to ensure accuracy.

 

Comment #8: row 213, I think that ”Numbered lists can be added as follows” is a typo

Author response: Thank you for catching this mistake. You are correct; "Numbered lists can be added as follows" is a typo and was not intended to be part of the manuscript. I have removed this text in the revised version. I appreciate your attention to detail.

 

Comment #9: The features composing the input vector are described in separated subsections. This is good as it simplifies the reading, however I’d like to have a clear definition of each subvector and of the final input vector used by each of the models, such as for example  and ,,

Author response: Thank you for your feedback, I understand the need to clearly define each sub-vector and the final input vector used by each model. To address this, I will clearly write out the input of each model in Table III, ensuring that the input vector of each model is well defined and easy to understand. (Please see page 10 Tables 2 and 3).

 

Comment #10: Figure 8 still show a different scale between the subfigures. In addition, the subfigures do not have an identification letter behind them, it is hence not univoquely determined which subfigure is (a), (b), (c). and (d).

Author response: Thank you for your feedback on Figure 8.I apologize for the scale issues and missing identification letters. I will adjust the scale of all subfigures in Figure 8 to ensure consistency across all subfigures. I will also add identification letters to each subfigure to clearly label them. (Please see page 16 Fig. 9).

 

Comment #11: in row 359 the authors write: ”When processing ADS-B data, special attention is paid to multiple aircraft signals in the same time window. Through precise calculations, the aircraft closest to the ground monitoring station is selected as the analysis object, and the aircraft position is determined in real time based on the latest ADS-B signal received every second”. The noise measurements in this case possibly present the effect of all the aircraft present in the area. Did the author just neglect this and associate the noise measurements to the aircraft closest to the observation point? Or did they preprocess somehow the noise measurement data in this case?

Author response: Thank you for your valuable insights. In our analysis, we specifically considered the aircraft closest to the ground monitoring point as the primary noise contributor. Hefei Xinqiao International Airport operates with a single runway, and to ensure flight safety, all takeoffs and landings adhere to strict horizontal and vertical separation standards. Given that our noise monitoring points are located near the airport, these safety measures effectively minimize the likelihood of noise interference or overlap between different flights, making the closest aircraft the dominant source of noise.

 

Comment #12: Step 8 of the algorithm in Table 2 do not describe how the (hyper)parameters update is performed. Row 375 says ”The choice of hyperparameters is determined by parameter optimization experiments”. Is it a proper optimization process (in this case, some information such as the algorithm used and its settings are needed)? A ”manual” try and error tuning has been performed? Is it an automatic procedure within the 200 epochs? Are 200 epochs used for each hyperparameters try? This should be further clarified.

Author response: Thank you for pointing out the need to clarify the hyperparameter update process described in step 8 of the algorithm in Table 2. This is described in detail in 3.3. Model training and Experimental Design(Please see page 14 row 437).

 

Comment #13: Partially connected to the comment above: figure 7 caption “Loss value curves for the PGNN model” does not specify which hyperparameters configuration is referring. I guess the answer is “the final configuration” but this should be clarified.

Author response: Thank you for your observation. You are correct; the caption in Figure 7 refers to the loss value curves for the PGNN model using the final hyperparameters configuration.

 

Comment #14: The authors introduced three error metrics in Section 3.2. Did the author had a precise aim using more than a single error metric? Why did they select MAE, MAPE, and RMSE? Do these three particular metrics monitor complementary aspects of the solution and of the reproduction error?

Author response: Thank you for your review and feedback. Regarding the error metrics used in Section 3.2, I would like to explain their selection and purpose. We choose multiple error metrics to comprehensively evaluate the performance of the model from different perspectives. Each metric has its own unique characteristics and can reveal different aspects of the model's performance. Therefore, by using multiple metrics, we can gain a deeper understanding of the strengths and weaknesses of the model and ensure the comprehensiveness and reliability of the results.

MAE (Mean Absolute Error): MAE is an intuitive error metric that represents the average of the absolute errors between the predicted values ​​and the actual values. It is insensitive to outliers and can provide a uniform distribution of errors.

MAPE (Mean Absolute Percentage Error): MAPE is used to measure the relative size of the prediction error and can provide a percentage form of the prediction error, so that data of different sizes can be compared.

RMSE (Root Mean Square Error): RMSE emphasizes the penalty of large errors because it is the square root of the mean of the squared errors. It can more strongly reflect the impact of large errors on the overall performance, so it is suitable for application scenarios where large prediction errors need to be avoided.

MAE and RMSE combined can help us understand the overall situation of errors and sensitivity to large errors. The combination of MAE and MAPE can help us understand both the absolute amount of error and the impact of relative error. The combination of RMSE and MAPE can comprehensively evaluate the accuracy of the model and the actual impact of the error.

 

Comment #15: This is a very minor comment: I’d find interesting to see the comparison between the methods predictions for a single test case, i.e. showing the models noise predictions (not the errors) for a single event at the monitoring points. I don’t know which form would it be applicable/better: noise footprint maps, a set of grouped histograms (each group showing the prediction of all the methods for a different observation point), whatever the authors think is best.

Author response: Thank you for your insightful comments. We appreciate your suggestion to compare the methods’ predictions for a single test case. To address this, we will add a section to the manuscript showing the noisy predictions of each model for a specific event. (Please see page 18 row 521).

 

Comment #16: row 476, the authors say, “Data Availability Statement: This study did not report any data.”. Please, correct.

Author response: Thank you for your feedback, the data used in this study are closely related to a specific project. Due to the sensitivity of the data and the confidentiality agreement related to the project, the accessibility of the data is strictly restricted and we are unable to disclose this data publicly.

 

With best regards,

 

Dan Zhu

College of Civil Aviation

Nanjing University of Aeronautics and Astronautics

E-mail: zhu85dan@nuaa.edu.cn

Sep. 6, 2024

Reviewer 3 Report

Comments and Suggestions for Authors

In the revised manuscript, the authors have addressed most of the queries and incorporated the suggestions provided by the reviewers. Given the thorough revisions and the improvements made, it can be recommended that the manuscript be accepted for publication. The authors have satisfactorily addressed the issues raised, and the manuscript now mostly meets the journal's standards for publication.

Author Response

Comment 1: In the revised manuscript, the authors have addressed most of the queries and incorporated the suggestions provided by the reviewers. Given the thorough revisions and the improvements made, it can be recommended that the manuscript be accepted for publication. The authors have satisfactorily addressed the issues raised, and the manuscript now mostly meets the journal's standards for publication.

Response 1: We sincerely appreciate the time and effort you and the reviewers have dedicated to evaluating our manuscript. We understand that the review process is both thorough and demanding, and we are grateful for your commitment to ensuring the quality and integrity of our research.

Reviewer 4 Report

Comments and Suggestions for Authors

The authors addressed all the comments

Author Response

Comment 1: The authors addressed all the comments.

Response 1: We sincerely appreciate the time and effort you and the reviewers have dedicated to evaluating our manuscript. We understand that the review process is both thorough and demanding, and we are grateful for your commitment to ensuring the quality and integrity of our research.

Round 3

Reviewer 2 Report

Comments and Suggestions for Authors

The authors clarified all the points I raised in my previous review round.

I hence recommend the publication of the manuscript after a small minor revision.

I just have two minor comments:

- regarding comment #14: thank you for the explanation, I think adding that to the manuscript would be valuable for the readers, adding some insight 

- regarding comment #16: I understand that the data can't be made available, but declaring that the work "did not report any data" is simply wrong. probably you can adjust the statement with something similar to "The data that support the findings of this study are available from [THIRD PARTY NAME] but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of [THIRD PARTY NAME]" or other more appropriate statements, that can be easily found online (for example https://authorservices.taylorandfrancis.com/data-sharing/share-your-data/data-availability-statements/)

 

 

 

Author Response

Dear Reviewer,

 

Thank you very much for your hard work in reviewing this paper. We submit a revised version of our paper with the title " A Neural Network with Physical Mechanism for Predicting Airport Aviation Noise," ID: aerospace-3141481, which has been modified according to your comments and suggestions.

 

The following is a point-by-point response to your comments and suggestions:

 

Comment 1: The authors clarified all the points I raised in my previous review round. I hence recommend the publication of the manuscript after a small minor revision.

Author response: We sincerely appreciate the time and effort you have dedicated to evaluating our manuscript. Thank you for your review and recognition of the paper's contribution.

 

Comment 2: - regarding comment #14: thank you for the explanation, I think adding that to the manuscript would be valuable for the readers, adding some insight

Author response: Author response: Thank you for your positive feedback and suggestions. We have revised the manuscript accordingly and explained the three error metrics as follows: “Among them, MAE is an intuitive error metric that represents the average of the absolute errors between the predicted and actual values. It is insensitive to outliers and can provide a uniform error distribution. MAPE measures the relative size of the prediction error and displays it as a percentage to facilitate comparison of different data scales. RMSE emphasizes large errors because it is the square root of the mean of the squared errors. It can better reflect the impact of large errors on the overall performance.” (Please see page 13 row 422).

 

Comment 3: - regarding comment #16: I understand that the data can't be made available, but declaring that the work "did not report any data" is simply wrong. probably you can adjust the statement with something similar to "The data that support the findings of this study are available from [THIRD PARTY NAME] but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of [THIRD PARTY NAME]" or other more appropriate statements, that can be easily found online (for example https://authorservices.taylorandfrancis.com/data-sharing/share-your-data/data-availability-statements/)

Author response: Thank you for your insightful comments. We acknowledge that the previous statement was unclear and may have caused some confusion. Based on your suggestions, we have revised the data availability statement to more accurately reflect the accessibility of the data. The revised statement is as follows: “The data supporting the findings of this study are available from Hefei Xinqiao International Airport. However, the availability of these data is restricted, and the data were used for this study under license and are therefore not publicly available. However, the authors can provide the data upon reasonable request and with the permission of Hefei Xinqiao International Airport.” (Please see page 19 row 573).

 

With best regards,

 

Dan Zhu

College of Civil Aviation

Nanjing University of Aeronautics and Astronautics

E-mail: zhu85dan@nuaa.edu.cn

Sep. 10, 2024

Author Response File: Author Response.docx