Hybrid 4-Dimensional Trajectory Prediction Model, Based on the Reconstruction of Prediction Time Span for Aircraft en Route

Round 1

Reviewer 1 Report

Overall Notes to the Authors:

The presented manuscript describes the a method for constructing a hybrid trajectory prediction model to increase en-route flight safety and improve airspace management capabilities from improved predictions of the aircraft’s four dimensional flight trajectory. The presentation of various trajectory prediction methods is thorough. However, I recommend some revisions in the manuscript, primarily in the initial setup of the paper’s objectives in the abstract, as well as the correction of fragmented sentences throughout the paper.

1. Section introductions and conclusions: The authors should consider adding conclusions and introductions to the start and end of Sections 2, 3, and 4 to make the objectives and the conclusions of each section more clear. For example, in section 3, Trajectory prediction experiments, the authors mention extracting 589 ADS-B flights as the first topic. I would recommend setting up the example; for example, why choose those specific flights for your demonstration and what are you going to do with them?

2. Section 4 presentation of your final method, it would be useful to include a figure similar to figure 3 to summarize your final hybrid method.

Additional formatting comments:

• I think you should connect the ADSB data and your method more clearly in the abstract. Should avoid using undefined acronyms like MSE in the abstract.
• Page 2, line 75: formulated shouldn’t be capitalized.
• Page 2, line 84: fragment sentence, should revise.
• I stopped giving specific grammar comments after page 1 because there are grammar mistakes throughout the manuscript. In particular, the manuscript is full of fragmented sentences. Please revise. I recommend having the paper reviewed by an editor for grammar errors.
• Should use the full page width for figures to make the text bigger and easier to read.

Author Response

  Thanks for your review!
  I have responded to your points in the document "respond1.docx" detailly.
  Please read my attached document "respond1.docx" for further review.
  If you have any other questions, please feel free to contact me at any time.
  Sincerely,Jinlun Zhou.

Author Response File: Author Response.pdf

Reviewer 2 Report

In this work, the authors propose a hybrid model consisting of strategically switching between several different trajectory prediction models, depending on which one minimizes an along-track prediction error. Motivated by the continual adaptation/implementation of trajectory-based operations (TBOs) in many FIRs throughout the world, the authors' research is relevant, and could have significant impacts on trajectory prediction/modelers.

My major concerns regarding the paper is the motivation behind the individual models chosen to be included within the hybrid model, as well as the generality of the results, given the limited case study scope. My major comments are given in detail below, followed by minor comments.

One comment I would like to make is that I very much appreciate the authors taking the time and effort to release their trajectory prediction data and models. This will be beneficial to future researchers looking to reproduce and expand on the work herein.

In summary, I find the work to be well-motivated, and I believe that a publishable manuscript could be produced after major revisions by the authors.

MAJOR COMMENTS

1. While the motivation behind using a hybrid model may be apparent (e.g., taking advantage of the benefits offered by several models by switching from one model to another strategically), it is less apparent why the authors chose the set of models/methods that they did choose (e.g., LSTM, BPNN, etc.)

   The authors jump right into Section 2 detailing the prediction model without motivating their selection of the various pieces of such a hybrid prediction model, nor explaining the pros/cons of each model, why one might be desirable versus another, and in what circumstances. Furthermore, combined with a rather sparse literature review, it is unclear to the reader what previous trajectory modeling research has leveraged, e.g., velocity trend extrapolation, LSTM, BPNN, etc.

   I would suggest that the authors significantly expand on the motivation (why they choose the models they did, and what are the pros/cons), the research gap they are trying to fill, and what their specific contributions are in this work.

2. Currently, the pre-processing step prior to input into the various machine learning models within the hybrid model is very confusing, as detailed in equations (6)-(9) and described in page 5, lines 194-202. I suggest the authors write this as a pseudo-code algorithm, and describe the workflow of the algorithm much more clearly than the way it is currently laid out.

   Since this is a critical step in the entire process (i.e., outputs/results/predictions will be strongly dependent on the input data format for training/validation), this procedure should be laid out as clear as possible, for later reproduction or extension.

3. For the stateful-LSTM validation results, the authors do not clearly describe/discuss why the MSE loss on the validation set increases dramatically after a certain training epoch. No mention of this is given in lines 457-465, even though this result is rather unintuitive at first glance. It would be great if the authors could expand on this point.

4. It appears that, at least for this route (and these performances will be extremely route- and aircraft type/engine type dependent) ODOPI.XEBUL..P101 used by the author as an experiment case study, the MSE is often lower-bounded by either VTE or stateful-LSTM, and the other models always perform worse in terms of MSE. However, because this is only the performance across one route, it's hard to speak with generality about the performance of these hybrid models. Currently, according to the authors' own selection criterion, BP, LSTM, KF, and 1D-CNN will never be selected as the active model in the hybrid model at any time period along the trajectory.

   Thus, I would strongly suggest that the authors broaden out their study and include additional cases wherein other models might outperform the VTE/stateful-LSTM duo, because otherwise the motivation for such a rich hybrid model becomes weaker, as it is dominated entirely by 2 models. If that is the case, why not simply propose a 2-model hybrid model, instead of sacrificing additonal computation time/resources to also run the 4 other models?

MINOR COMMENTS

1. Page 2, lines 75-78: For the flight plan-based model, it would be great if the authors could provide a more in-depth explanation on what some of the relevant past research has been. For example, research involving multi-flight common routings or trajectory option sets/collaborative trajectory options should be cited, as these directly influence the flight plan creation/modification process, and is a joint process involving airlines, ANSPs, etc.
2. In Section 2.1.1, the authors should provide intuition on what they mean by the trajectory intention deviation factor. For example, typically, aircraft trajectory intentions are, e.g., waypoints entered directly into the flight management computer. The authors should give examples of what kind of deviations may arise, why these deviations happen, etc.
   
   
3. The authors should define QNE, as it is not defined anywhere.
   
   
4. In Figure 2, it is unclear what the values represent, as there is no legend provided with this figure. I suggest the authors provide a more focused picture (i.e., the entirety of the graphical user interface is not necessary), and add a legend for the figure.
   
   
5. In equation (11), the authors should be clear about the norm on the right-hand side, especially if they are measuring a mean squared error. To be explicit, they should denote that they are taking a (vector) 2-norm, and should also denote a square, because otherwise the 2-norm includes a square root, and the authors are not measuring RMSE.
   
   
6. There are several smaller suggestions for Section 3.1.2 involving the DL weight update process. I would first suggest that the authors make explicit the norm used in equation (17), then turn the Adam weight update procedure into a pseudo-code algorithm. Finally, in line 302, I believe that the dot-circle symbol should represent an element-wise vector product, as "vector product" as currently used by the authors is vague (e.g., it could be a dot product, or an elemement-wise product).
   
   
7. There is some confusion in Figure 14, as the caption depicts probability densities for 500s and 1000s of aircraft position (in terms of prediction timespans), but in the actual figure, the label is 100s, not 1000s. If the authors could clarify on this point, that would be great.
   
   
8. In Figure 27(a), the authors should include unit labels for the altitude (is it in feet? meters?)
   
   
9. In the conclusion, I'm not sure if the authors can claim that their hybrid model's performance will always be "better theoretically," because a better hybrid model could always be produced if an additional model is added. Thus, I think it would be more accurate to state that the hybrid model will at least perform no worse than any of the individual models that comprise it.

Author Response

  Thanks for your review!
  I have responded to your points in the document "respond2.docx" detailly.
  Please read my attached document "respond2.docx" for further review.

  If you have any other questions, please feel free to contact me at any time.
  Sincerely, Jinlun Zhou.

Author Response File: Author Response.pdf

Reviewer 3 Report

I would suggest the authors to add more evidence for the three-item classification being provided in the introduction (line44 onwards). Reference 1 and 2 do not seem to reflect the statement explicitly. 

"The above-mentioned various trajectory prediction methods have different prediction performances under different time spans of prediction". I agree with this statement, but I would recommend the authors to synthetize the main expected differences and timing. A syntetic table is recommended with the key variables to allow the comparison.
It seems the content in line 100-110 can be syntethize in bullet points to increase readability

With respect to the candidate pool of prediciton models. I would have expected more emphasis on the criteria for the selection, as well as comments on the excluded results (line 232 onwards). 

"Since it is impossible to test all of the possible structures of the BP neural network, this paper only tests several hyperparameters of the BP neural network structure, as shown in Table 2." This statement is partly true: a grid search/random search/ or even bayesian optimization could have been used for hyperparameters optimization. Can the authors clarify why a systematic approach has not been followed, or further document/extend their analysis in this regard?

I suggest he authros to provide a summary of the obtained results in the conclusions, as well as a frank account of the study limitations.

About LSTM, the authors indicate the risk of over-fitting, but they it seesm they do not have any formal check in this regard. I strongly recommend to verify over-fitting potentials in their solution through some dedicated algorithm (e.g. cross validation)
Language revision required:
- *What’s more
- flight plan is jointly *Formulated
- *And Section5 is the conclusion of this research.
- Then *build a candidate pool that includes (missing subject)
- This paper *chose seven. It is not the paper itself that made the choice.
- a little *deviate from it
etc.

Check also font style/levels/capital letters/paragraphs throught the text. There are several inconsitencies.

Author Response

  Thanks for your review!
  I have responded to your points in the document "respond3.docx" detailly.
  Please read my attached document "respond3.docx" for further review.
  If you have any other questions, please feel free to contact me at any time.
  Sincerely,Jinlun Zhou.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

I appreciate the efforts undertaken by the authors to address my concerns.I believe that the manuscript is publishable, conditioned on the authors including the appendix with additional experiments and working the appendix into their discussion/conclusion.

In their response, the authors said that including the appendix would weaken their conclusions -- however, one must include all experimental results, as that promotes transparency, and adds additional nuance. Trajectory prediction is inherently noisy/stochastic, so it should not be surprising that on certain routes/conditions, one method out-performs another.

Author Response

Another experiment of trajectory prediction has been added to this paper as Section6: Appendix: an additional case.

Thank you very much for your points makes the paper more reasonable!

Sincerely,

  Jinlun Zhou.

Reviewer 3 Report

I thank the authors for revising the article following my suggestions and recommendations.

Author Response

  Thank you very much for your points makes the paper more reasonable!

Sincerely,

  Jinlun Zhou.