Review Reports - Exploration of Traffic Accident-Based Pilot Zones for Autonomous Vehicle Safety Validation

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

- Abbreviations need to be defined when they appear for the first time in the text

- Move the figures to be after mentioning it in the text

- 4. Methodology: "Subsequently, using the CNN+BiGRU model, we classify data from the KNPA dataset, which consists of human-caused traffic accidents, to identify those similar to AV-caused accidents". (the purpose of using KNPA dataset and this stage of classification is not clear how will it enhance the performance what is the need for it?)

- Fig. 2 stages needs more explanation in the text

- Section 4.4.4 paragraph 2 and 3 are repeated

- The presentation and contribution are commendable, but the results cannot rely on due to the small dataset size and the overfitting problem that it may cause.

- Also the author needs to show how the proposed model outperforms the previous work mention in the related work in terms of performance and complexity for example.

Author Response

** Response to the Reviewers/Summary of Changes **

We would like to thank the editor and reviewers for their valuable comments. Taking all the comments into consideration, the revised texts, apart from the references in the revised manuscript, have been changed and highlighted with red(green) lettering.

(a) Abbreviations need to be defined when they appear for the first time in the text.

▸ As the reviewer commented, we identified some incorrect abbreviations in the manuscript. We have revised and corrected all abbreviation-related issues throughout the revised manuscript.

(b) Move the figures to be after mentioning it in the text.

▸ As the reviewer commented, we have adjusted the placement of all figures to ensure they appear after being mentioned in the text.

(c) 4. Methodology: "Subsequently, using the CNN+BiGRU model, we classify data from the KNPA dataset, which consists of human-caused traffic accidents, to identify those similar to AV-caused accidents". (the purpose of using KNPA dataset and this stage of classification is not clear how will it enhance the performance what is the need for it?).

▸ As the reviewer mentioned, we have provided a detailed explanation of the purpose for using the two datasets (DMV dataset and KNPA dataset). Specifically, the KNPA dataset, used as the test data, serves as the basis for identifying AV-like data. Given the current global lack of large-scale datasets related to AV traffic accidents, we propose an alternative approach by identifying AV-like cases from human-caused traffic accidents to explore potential pilot zones.

▸ In the revised manuscript, we have revised the content on pages 4-5, lines 185-206.

(d) Fig. 2 stages needs more explanation in the text.

▸ As the reviewer commented, we have added further details to the revised manuscript to explain the methodology outlined in Figure 2 as thoroughly as possible. Additionally, we have modified the figure to enhance the readers' understanding.

▸ In the revised manuscript, we have revised the content on pages 5-6, lines 208-229.

(e) Section 4.4.4 paragraph 2 and 3 are repeated.

▸ As the reviewer commented, we have removed the third paragraph that was repeated in the revised manuscript. We appreciate your attention to detail in catching this significant oversight.

(f) The presentation and contribution are commendable, but the results cannot rely on due to the small dataset size and the overfitting problem that it may cause.

▸ As the reviewer mentioned, it is true that the DMV dataset used in this study is relatively small, which increases the likelihood of overfitting during the training process of the CNN+BiGRU classification model. We have addressed this limitation, including the potential for overfitting, in the ‘Conclusion’ section of the revised manuscript.

▸ In the revised manuscript, we have revised the content on page 15, lines 521-530.

▸ Additionally, we have outlined future research directions to address these limitations. In our upcoming studies, we aim to gather more extensive data on AV traffic accidents and conduct further research to improve reliability.

▸ In the revised manuscript, we have revised the content on page 15, lines 531-533.

(g) Also the author needs to show how the proposed model outperforms the previous work mention in the related work in terms of performance and complexity for example.

▸ As the reviewer mentioned, we have thoroughly analyzed the issues observed in previous scenario-based studies, such as limited scenario variety, gap from reality, high cost and time requirements, and limited interaction testing. Sequentially, we have compared these issues to highlight the advantages of the proposed method in detail.

▸ In the revised manuscript, we have revised the content on pages 3-4, lines 127-154.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors,

I gladly accepted to review your work on ”Exploration of Traffic Accident-Based Pilot Zones for Autonomous Vehicle Safety Validation”.

In order to publish your work, there are a few issues that need attention so that your work will receive the deserved credit in the field:

1. My first concern is about the reproducibility of your work. In order to achieve I recommend that you add a subsection describing your test setup, hardware and software used and also you should consider making a repository with your data under license of course, in order to backup your results of CNN+BiGRU model on the DMV dataset. I hope this is possible and thus your work will have a better impact in your field.

2. My second concern, resides on the citing formulation. This starts with the reference [15]. Although: In [] ,.... is accepted in many papers, I recommend placing the brackets form at the end of sentence of phrase. If the referenced work is key work in your manuscript you can also add names as in Rocklage et al. ... This will boost the readability of your work and improve citing other's work.

3. My third and final concern is on the comments on the results in section 4. Please provide further explanations on figure 8 and each involved cluster. You can also repeat some of your previous explanations to support the context. It also would be nice a figure before applying DBSCAN. This will highlight your results even move.

I hope that your work in the field will gather more impact and autonomous companies around the world will put the in autonomous road transportation.

Best regards

Author Response

** Response to the Reviewers/Summary of Changes **

(a) My first concern is about the reproducibility of your work. In order to achieve I recommend that you add a subsection describing your test setup, hardware and software used and also you should consider making a repository with your data under license of course, in order to backup your results of CNN+BiGRU model on the DMV dataset. I hope this is possible and thus your work will have a better impact in your field.

▸ As the reviewer commented, we have added the "4.1. Experimental Setup" subsection and provided a detailed description of the research environment used in this study. Additionally, we reflected the use of various software modules, such as GloVe, NLTK, CNN+BiGRU, and DBSCAN, at each stage in the revised manuscript.

▸ In the revised manuscript, we have revised the content on page 6, lines 230-240.

▸ As the reviewer mentioned, all data used in this study, including the embedding vectors (csv) and classification models (md), can be accessed via the following repository. By making this repository available to researchers, we aim to facilitate further studies and encourage broader research.

▸ https://github.com/Ez-Sy01/Exploration-of-Traffic-Accident-Based-Pilot-Zones-for-Autonomous-Vehicle-Safety-Validation

▸ In the revised manuscript, “Data Availability Statement” are provided on page 16, lines 566-567.

(b) My second concern, resides on the citing formulation. This starts with the reference [15]. Although: In [] ,.... is accepted in many papers, I recommend placing the brackets form at the end of sentence of phrase. If the referenced work is key work in your manuscript you can also add names as in Rocklage et al. ... This will boost the readability of your work and improve citing other's work

▸ As the reviewer pointed out, we noticed inconsistencies in citation formatting. In the revised manuscript, we have standardized the citation style by placing citation brackets [] at the end of sentences. Additionally, where citations appeared in the subject position, we revised them to follow the 'author et al.' format for better readability.

▸ In the revised manuscript, all modifications related to references have been highlighted in green.

(c) My third and final concern is on the comments on the results in section 4. Please provide further explanations on figure 8 and each involved cluster. You can also repeat some of your previous explanations to support the context. It also would be nice a figure before applying DBSCAN. This will highlight your results even move.

▸ As the reviewer suggested, we added Figure 8a, which illustrates the polygon before applying the DBSCAN clustering method. Additionally, in the revised manuscript, we provided an in-depth analysis of each cluster's meaning and demonstrated the validity of our proposed method by comparing the individual and total areas of the clusters.

▸ In the revised manuscript, we have revised the content on pages 13-14, lines 459-491.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

This paper suggests a data-driven approach to address the challenges of selecting effective pilot zones for Autonomous Vehicle safety verification. By analyzing historical traffic accident data, the authors aim to identify road segments that exhibit characteristics similar to AV operating conditions. This methodology offers a more efficient and comprehensive alternative to traditional methods, which often involve costly and time-consuming searches for suitable pilot zones after pilot districts have been determined.

The paper is nice and interesting; however, I have several concerns:

1. In Figure 2, the author needs to specify what information goes through each arrow drawn in the figure.

2. In equation 1, the sigma should be split into two sigmas - one where i is from 1 to n and another where j is from 1 to n.

3. In equation 2, OK for C_max; however, what is C_max' (I.e. C_max with apostrophe)?

4. Equation 3 and equation 4 are the same. Why in equation 3 the result is called "Recall" and in equation 4 the result is called "Precision"?

5. The term "epoch" is unclear. The authors write "the epoch is around 80" and "when the epoch was 100". So, what is "epoch"? Which unit does "epoch" use?

6. The computational complexity of Algorithm 1 should be analyzed.

7. In Ghorai, P., Eskandarian, A., Abbas, M., & Nayak, A., "A Causation Analysis of Autonomous Vehicle Crashes" IEEE Intelligent Transportation Systems Magazine, 2024, the authors claim that the accidents of autonomous vehicles and of traditional vehicles are different and therefore it seems impractical to learn from accidents of traditional vehicles. I would encourage the authors to cite this article and challenge its argument.

8. I would encourage the author to cite the autonomy levels of self-driving vehicles that can be found here: Wiseman Y., "Autonomous Vehicles", Encyclopedia of Information Science and Technology, Fifth Edition, Vol. 1, Chapter 1, pp. 1-11, 2020, available online at: https://u.cs.biu.ac.il/~wisemay/Autonomous-Vehicles-Encyclopedia.pdf and explain to which autonomy level their model can be applied.

9. The paper only emphasizes the advantages of the work done, but does not elaborate on the shortcomings of the work done and the follow-up prospects.

10. The format of references should be consistent.

Author Response

** Response to the Reviewers/Summary of Changes **

(a) In Figure 2, the author needs to specify what information goes through each arrow drawn in the figure.

▸ As the reviewer suggested, we revised Figure 2 to clearly depict the input and output values at each stage and modified it overall to enhance clarity for the readers.

▸ We have added further details to the revised manuscript to explain the methodology outlined in Figure 2 as thoroughly as possible.

▸ In the revised manuscript, we have revised the content on pages 5-6, lines 208-229.

(b) In equation 1, the sigma should be split into two sigmas - one where i is from 1 to n and another where j is from 1 to n. 252~265.

▸ As the reviewer mentioned, since i and j are grouped under a single sigma, we revised the equation to use a double sigma notation accordingly.

▸ In the revised manuscript, we have revised the content on page 9, lines 333.

▸ As the reviewer pointed out, we have confirmed that C_max was indeed a typo. Additionally, we identified several other corrections that needed to be made in the formulas. Consequently, we have revised all symbol notations, formula representations, and explanations related to GloVe. This improvement was made possible thanks to the reviewer's valuable feedback, for which we are sincerely grateful.

▸ In the revised manuscript, we have revised the content on page 9, lines 349.

(d) Equation 3 and equation 4 are the same. Why in equation 3 the result is called "Recall" and in equation 4 the result is called "Precision"?

▸ The denominators for Recall and Precision are (True Positive + False Negative) and (True Positive + False Positive), respectively. Since the terms in the latter part of the formulas differ, the resulting equations are also distinct. In the revised manuscript, we've highlighted the differing terms in red for clarity.

▸ In the revised manuscript, we have revised the content on page 11, Equation 3 and Equation 4.

(e) The term "epoch" is unclear. The authors write "the epoch is around 80" and "when the epoch was 100". So, what is "epoch"? Which unit does "epoch" use?

▸ As the reviewer pointed out, the term 'epoch' was not explained in detail. In the revised manuscript, we have clearly defined it as used in deep learning training to ensure clarity and avoid any confusion.

▸ In the revised manuscript, we have revised the content on page 11, lines 396-400.

(f) The computational complexity of Algorithm 1 should be analyzed.

▸ To calculate the time complexity of this algorithm, we analyze the operations at each step. The input dataset X has a size of N, with each data point represented as 2D coordinates. The outer for loop iterates $N$ times, from i = 1 to N. Within this loop, the inner for loop also iterates N times for each i, where the distance between two points is computed. The Manhattan distance computation between two points takes constant time O(1). The number of neighbors is counted inside the inner loop, and the algorithm checks whether the count is below the threshold k. These operations are also performed in constant time O(1). Therefore, the complexity of the inner loop is O(N), and since the outer loop runs N times, the overall time complexity of the algorithm becomes O(N) x O(N) x O(1) x O(1) = x O(N²).

▸ In the revised manuscript, we have revised the content on page 12, lines 429-438.

(g) In Ghorai, P., Eskandarian, A., Abbas, M., & Nayak, A., "A Causation Analysis of Autonomous Vehicle Crashes" IEEE Intelligent Transportation Systems Magazine, 2024, the authors claim that the accidents of autonomous vehicles and of traditional vehicles are different and therefore it seems impractical to learn from accidents of traditional vehicles. I would encourage the authors to cite this article and challenge its argument.

▸ As the reviewer pointed out, we have cited the mentioned paper in the revised manuscript and presented a counterargument to their claims in the 'Introduction' section.

▸ Contrary to their claim, AVs and human-driven vehicles operate in similar environments and face overlapping challenges. As a result, analyzing traditional vehicle accident data can offer valuable insights into human error, environmental factors, and road design, helping to predict potential risks for AVs. Since AVs continue to interact with human drivers, pedestrians, and cyclists, understanding traditional accident patterns is crucial for improving AV algorithms and ensuring safer operations in diverse scenarios.

▸ In the revised manuscript, we have revised the content on page 2, lines 61-70.

(h) I would encourage the author to cite the autonomy levels of self-driving vehicles that can be found here: Wiseman Y., "Autonomous Vehicles", Encyclopedia of Information Science and Technology, Fifth Edition, Vol. 1, Chapter 1, pp. 1-11, 2020, available online at: https://u.cs.biu.ac.il/~wisemay/Autonomous-Vehicles-Encyclopedia.pdf and explain to which autonomy level their model can be applied..

▸ According to the paper recommended by the reviewer, Level 4 autonomy in self-driving vehicles refers to high automation, where the vehicle can operate independently under certain conditions while ensuring passenger safety. At this level, a human driver is not required to intervene in emergencies, provided the vehicle operates within its specified limits. Given that most autonomous vehicles registered with the California DMV are classified as Level 4, the proposed methodology is expected to be highly useful for future pilot programs involving Level 4 autonomous vehicles with restricted Operational Design Domains (ODD).

▸ In the revised manuscript, we have revised the content on page 15, lines 514-520.

(i) The paper only emphasizes the advantages of the work done, but does not elaborate on the shortcomings of the work done and the follow-up prospects.

▸ As the reviewer mentioned, we have detailed the reliance on limited information, the model's overfitting due to insufficient training data, and the limitations in exploring pilot zones in the revised manuscript.

▸ In the revised manuscript, we have revised the content on page 15, lines 521-530.

▸ Additionally, we have proposed future work directions to address these challenges.

▸ In the revised manuscript, we have revised the content on page 15, lines 531-549.

(j) The format of references should be consistent.

▸ In the revised manuscript, all modifications related to references have been highlighted in green.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors,

I agree with all modifications.

Congratulations on this follow up manuscript.

Best of luck with your research!

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have addressed all my concerns. The revised manuscript is ready for publication.