Measures and Methods for the Evaluation of ATO Algorithms
Round 1
Reviewer 1 Report
The topic addressed in this paper is interesting, however, the paper is long and not well structured. The contributions of the paper are very vague. The reviewer would suggest the authors state the contributions explicitly and reorganize the contents. The detailed comments are given as follows:
- There are more than 10 paragraphs in the conclusion sections, which makes it difficult to really understand the main conclusions of the paper.
- It seems that the proposed approach proposed by the authors are the 6-layer model and the framework of the scenarios. However, only two or three scenarios are tested in the case study, other than a more or less complete set or framework of the ATO system. In addition, the tested scenarios are just a few simulating runs for the speed profile optimization algorithm and the criteria are also normal criteria for speed profile optimization algorithms in the literature, which weakens the contribution of this paper. So the reviewer is a bit confusing about the contributions of the paper. Please state this clearly in the introduction section.
- In the abstract and many places in the paper, the authors mentioned about “the proposed approach”. What is this approach exactly? It is very unclear to the reviewer.
- What does the ATO algorithms mean? Are they the speed profile optimization algorithm and the traction and braking force control algorithm? There are many papers in the literature addressed these two type of algorithms and they presented many criteria for the evaluation of these algorithms. What are the differences between these presented in this paper and those proposed in the literature?
- Title of Section “literature review” should be “Literature review” with capital L. What’s the relationship between Section 2.1 and 2.2? Moreover, Section 2.2 is too long. It would be better to add subtitles to organize the contents better. What are the conclusions of the literature review? How do they connect with Section 3?
Author Response
Dear Reviewer 1, we thank you for your time and your efforts to improve our paper. Please find our responses below.
Point 1: There are more than 10 paragraphs in the conclusion sections, which makes it difficult to really understand the main conclusions of the paper.
Response 1: Thank you for your valuable advice. We simplified the conclusion section by transferring some statements to other locations where they are originally referring to. This allows the conclusion section to focus more on the most important elements.
Point 2: It seems that the proposed approach proposed by the authors are the 6-layer model and the framework of the scenarios. However, only two or three scenarios are tested in the case study, other than a more or less complete set or framework of the ATO system. In addition, the tested scenarios are just a few simulating runs for the speed profile optimization algorithm and the criteria are also normal criteria for speed profile optimization algorithms in the literature, which weakens the contribution of this paper. So the reviewer is a bit confusing about the contributions of the paper. Please state this clearly in the introduction section.
Response 2: The approach consists of the transformed and adapted 6-layer model and the scenario types, also adapted for the special needs in the context of ATO performance evaluation. The other two components are the KPIs which are commonly used in the railway sector and taken from the literature and the list of possible performance influencing factors. KPIs have been extended by sub-level indicators specifically interesting for ATO performance evaluation. You are right, the case study does only cover a few simulation runs and one specific influencing factor. It shall serve as a demonstration of how to apply the approach. It is not a complete evaluation of the mentioned ATO algorithm BEAOnline. A comprehensive evaluation is in the scope for further research as mentioned in the paper. We adapted the paper to make this clearer.
Point 3: In the abstract and many places in the paper, the authors mentioned about “the proposed approach”. What is this approach exactly? It is very unclear to the reviewer.
Response 3: Thank you for pointing out that more clarification is needed to ensure the reader easily understands what the proposed approach is. Please see the answer to your second question for a short overview. We adapted the description in the introduction and abstract specifically listing the contributions.
Point 4: What does the ATO algorithms mean? Are they the speed profile optimization algorithm and the traction and braking force control algorithm? There are many papers in the literature addressed these two type of algorithms and they presented many criteria for the evaluation of these algorithms. What are the differences between these presented in this paper and those proposed in the literature?
Response 4: It is correct that the literature is addressing several performance indicators. We clarified that the criteria are taken from the literature and have been supplemented by a multi-level structure implementing sub-indicators that refer to ATO algorithm performance testing. Thanks to your review we also made clear that the approach focuses on the speed profile optimization algorithm, which is one main part of ATO systems. Also it is envisaged to emphasize the need for a structured test framework for ATO performance tests that allows the comparison of multiple algorithms or one algorithm in different challenging situations of varying complexity.
Point 5: Title of Section “literature review” should be “Literature review” with capital L. What’s the relationship between Section 2.1 and 2.2? Moreover, Section 2.2 is too long. It would be better to add subtitles to organize the contents better. What are the conclusions of the literature review? How do they connect with Section 3?
Response 5: Thank you for the advice. We restructured subsection 2.3 with subsubsections. The connections to the following sections are described in the subsubsection conclusion. There we state a summary and which findings we want to use and/or adapt in our approach (measures, 6-layer model, scenario testing).
Reviewer 2 Report
This is an interesting application of applied simulation design to ATO. While well-written, the paper fails short in providing additional information on the methods used. We suggest the authors to provide not only a pseudo code of the simulation used, but also report a table on the basic descriptive statistics of the inputs and outputs used in the model, to make the paper more readable to readers.
Author Response
Dear Reviewer 2, we thank you for your positive response and the suggestions you made. Please find our response below.
Point 1: This is an interesting application of applied simulation design to ATO. While well-written, the paper fails short in providing additional information on the methods used. We suggest the authors to provide not only a pseudo code of the simulation used, but also report a table on the basic descriptive statistics of the inputs and outputs used in the model, to make the paper more readable to readers.
Response 1: Thank you for your positive feedback. For our tests, we used a proprietary simulator provided by ProRail. It is not possible to add pseudo-code of the simulator or the ATO algorithm due to the IP rights. The full evaluation of BEAonline was not in the scope of the paper rather than developing a method to apply in further investigations.
Reviewer 3 Report
An exceptionally high-quality scientific article deals with the topic of the application of train movement modeling with the help of appropriate simulations. All elements of ATO system modeling are discussed, including all indicators and scenarios. The article is extremely extensive and, in my opinion, is more in the form of a chapter in a scientific book than a scientific article. I think the scope of the article could contribute to less interest in reading it. Appropriately (too) much emphasis is placed on the chapter of reviewing the scientific literature and the approach to ATO testing.
Congratulations to the authors for such an extensive article; I suggest that they consider possible changes in terms of producing a shorter but still effective article.
I also suggest the following corrections:
- Chapter 3.3.4 mentions that a comprehensive overview of the influencing factors is presented in "section A" - you probably meant "Appendix A"?
- the graphical simulation results presented in Figure 5, a - f are not readable; I suggest choosing the right color and font size to make the article readable.
Author Response
Dear Reviewer 3, we highly appreciate your positive response and your suggestions. Due to the shortness of time, we were not able to produce a shorter version. Nevertheless, we hope that the paper fits the needs of the readers and that the provided information is valuable.
Point 1: Chapter 3.3.4 mentions that a comprehensive overview of the influencing factors is presented in "section A" - you probably meant "Appendix A"?
Response 1: Thank you, that was indeed a wrong wording. We changed it to "Appendix A". We also checked (and corrected) the case of the section titles (all main words beginning with an uppercase letter) and the subsequent subsections (all lowercase except the first letter).
Point 2: the graphical simulation results presented in Figure 5, a - f are not readable; I suggest choosing the right color and font size to make the article readable.
Response 2: We adapted the font size and the color and also edited the underlying grid.
