Scenario Identification and Classification to Support the Assessment of Advanced Driver Assistance Systems

Reviewer 2: Point-by-point response to Comments and Suggestions for Authors

The authors present approach to identify cut-in, cut-out, and cut-through manoeuvres from a real-world dataset. A rule-based classification approach is compared with an AI-based classification approach, where the results suggest that the AI-based approach performs better in terms of correct identification of the manoeuvres. The approach is indeed interesting. However, I think the comparison approach is questionable, please refer to my detail comments.

1.      Comments: There are several existing work on scenario identification and extraction, especially on cut-in and cut-out, that were not considered by the authors. A few examples are provided below. H. Muslim et al., "Cut-Out Scenario Generation With Reasonability Foreseeable Parameter Range From Real Highway Dataset for Autonomous Vehicle Assessment," in IEEE Access, vol. 11, pp. 45349-45363, 2023, doi: 10.1109/ACCESS.2023.3268703. A. Zlocki et al., "Logical Scenarios Parameterization for Automated Vehicle Safety Assessment: Comparison of Deceleration and Cut-In Scenarios From Japanese and German Highways," in IEEE Access, vol. 10, pp. 26817-26829, 2022, doi: 10.1109/ACCESS.2022.3154415. Ma, X., Ma, Z., Zhu, X., Cao, J. et al., "Driver Behavior Classification under Cut-In Scenarios Using Support Vector Machine Based on Naturalistic Driving Data," SAE Technical Paper 2019-01-0136, 2019, https://doi.org/10.4271/2019-01-0136.

Thank you for your remark. References have been included.

2.      Comments: There are many questionable points in the authors' approach to motivate that the TSF-based model is better than the rule-based decision tree. It is an unfair comparison in my opinion, please refer to my detail comments.

The rule-based approach as shown is currently industrial standard, and in order to be not unfair we only used lateral information in combination with TSF as alternative. Unfair would have been to use more information like longitudinal and lateral signals.

3.      Comments: Many of the references are in German (as far as I understood). That makes it difficult to judge their relevancy and it is not very useful to the readers in an international context. Some references referred to the arXiV version instead of the properly published version in proceeding (e.g., reference 6).

Unfortunately, the strongest automotive research and developments in the ADAS field are located in Germany (e.g. PEGASUS, Mercedes-Benz Drive Pilot,..), which is why much literature is in German language.

4.      Comments: It is not clear why the rule-based classification used time window of 10 seconds, while the AI-based approach used 20 seconds. This makes a comparison unfair in my opinion (the AI-based approach already has more information for identification of the manoeuvre).

The rule-based approach looks for signal jumps at every time step and doesn’t need any window. The 10s you mentioned are only a criterion for detection cut-through maneuvers (within 10s the combination of cut-in & cut-out shall occur).

5.      Comments: If the lateral distance is available in the data, why does the rule-based classification use the longitudinal distance, while the AI-based approach use lateral distance? In other words, why not compare the approach based on the same feature (i.e., longitudinal v.s. longitudinal)? This is unclear to the reader and not well-motivated.

In principle you are right. In order to motivate our choice, we have extended the beginning of chapter 3. In a later research we may combine both signals for becoming more robust. Näher am echten Fahrer (dieser bewertet auch anhand des Lateralabstands).

6.      Comments: I may have missed it, but I did not find a comment on how much data were required (or used) to train this model to achieve this level of accuracy.

Thank you for the hint. We included this information in the revised paper.

7.      Comments: It is quite interesting to see that both approaches failed to classify the "other" manoeuvre as "other". Perhaps the authors should add a discussion on that.

Thank you for the valuable remark. We included a discussion in the revised paper.

8.       Comments: The author mentioned using additional feature in the conclusion (a combination of longitudinal distance and lateral distance). I wonder if a rule-based approach using both features would also perform better. It would be interesting for the reader if the authors could add a discussion about this. 

You are absolutely right. We were also thinking about this topic already, see response Nr.5. However, investigating all combinations would have blasted the paper.

Reviewer 3: Point-by-point response to Comments and Suggestions for Authors

The paper presents results about the application of two strategies for high-precision risk assessment of Advanced Driver Assistance Systems. The first strategy is a traditional rule-based approach (similar to an expert system) while the second is based on Artificial Intelligence approaches.

The paper is weel written and organized. Results are convincing and supports both discussion and conclusions.

Thank you for your favorable assessment.

1.      Comments: Results are well presented: both tables and figures are clear and easy-to-read.

Thank you for your favorable assessment.

2.      Comments: I have just a curiosity -- not to be intended as a concern. Clearly, the AI-based approach (i.e., Time-Series Forest, TSF) outperforms the traditional rule-based decision tree (DT) strategy, by improving in terms of both precision and recall on the three classes. However, it is not clear if there is any overlap -- and in that case which is its amount -- between the classification errors of the two methods. In other terms, can TSF correctly classify all the instances which are also correctly classified by DT (plus others)? or does it improve accuracy by changing the set of correctly classified instances?

3.      Comments: This could be really interesting to investigate, even because, if the two strategies are complementary, they could be optimally combined by further improving accuracy.

You raise indeed a really interesting question, which is why we made the counts and included it in the revised paper in Table 2.

Reviewer 2: Point-by-point response to Comments and Suggestions for Authors

I would like to thank the authors for considering my comments. However, not all my comments are addressed and I would encourage the authors to revise their paper again.

1.       Comments: State-of-the-art review. Perhaps I was not clear in the previous review, my apologies for that. I gave those references only as examples in the previous review, my main concern is that I do not feel that the authors gave a complete overview about the state of knowledge in the area of research.

Response: Thank you for your remark. For my dissertation, I conducted a thorough literature review. In this paper, however, I initially cited only the most relevant sources. Nevertheless, I have now included two additional references that may be important for this paper as well.

2.       Comments: Please try to use the proper reference where possible instead of the version from ArXiV. For instance, the reference 6 has a proper published version:

Response: Thank you for the hint again. I adjusted the references accordingly, where it was possible I added the proper published version.  

3.       Comments: I can understand the argument that some references are only available in German, please at least provide the link to the documents. It would be great if the authors also attempt to find an equivalent version in English, which I assume might be available since some of the projects mentioned are international projects.

Response: Where it was possible, I provided an equivalent version in English. As ATZ is a  widely recognized automotive journal in Germany, it makes it in the field of ADAS important to include in the references.

4.       Comments: Thank you for the explanation added to the beginning of the Chapter 3. However, that does not clarify why the rule-based method used longitudinal distance and the AI-based used the lateral distance. That is, if the standard rule-based approach classifies the manoeuvres based on longitudinal distance, why not use the longitudinal distance for the AI-based approach as well, so that one can see a clear comparison of performance based on the same type of input data.

Response: Thank you for the clarification. However, our intention is to demonstrate that AI can recognize patterns that are difficult or even impossible to describe with rule-based approaches. This capability opens entirely new possibilities. The purpose of our work is not to make a direct comparison, but rather to highlight the potential of these new methods and to inspire future developments.