Evaluation of Different Generative Models to Support the Validation of Advanced Driver Assistance Systems

Round 1

Reviewer 1 Report

This paper investigates the use of GAN models to generate realistic driving scenarios. The authors compare four models: a variational autoencoder (VAE), a generative adversarial network (GAN), a Wasserstein GAN (WGAN), and a temporal GAN ​​(TimeGAN), evaluating their performance in terms of realism, diversity, and statistical similarity. 

The article is a supplement to the article: Generation of Realistic Cut-In Maneuvers to Support Safety Assessment of Advanced Driver Assistance Systems. In the previous article, you showed that VAE works great for the chosen task, and in this one you compare it with other methods and show how much better VAE is than others. The article contains many useful metrics for comparison and, in general, the structure of the article is very well assembled.

However, I am a little confused by the lack of any new contribution to the research area. The comparison is made with fairly standard methods, I would like to see a comparison with some existing work.

ome Questions:
1) The paper lacks a literature review on the topic under study. What modern methods are currently used in addition to Generative Networks? Why are Generative Networks the best fit for this task? The choice is also not explained.
2) Why does this paper focus on the tie-in scenario? What scenarios are typically studied and which ones are the most difficult to predict?

Some pictures are repeated in both the previous and the new article. Here you either need to add links to the pictures or redraw them. 

Author Response

1. Comments: The paper lacks a literature review on the topic under study. What modern methods are currently used in addition to Generative Networks? Why are Generative Networks the best fit for this task? The choice is also not explained.
Answer: The introduction has been revised to clarify the scope of the paper.

2. Comments: Why does this paper focus on the tie-in scenario? What scenarios are typically studied and which ones are the most difficult to predict?
Answer: Thank you for the hint. We focused on the cut-in scenario because it is both common in real-world driving and widely regarded as one of the most difficult to predict due to its sudden nature, limited reaction time, and high variability in driver behavior (added in the introduction). We agree with the reviewer that also other maneuvers should be investigated in future as mentioned in the conclusions.

3.Comments: Some pictures are repeated in both the previous and the new article. Here you either need to add links to the pictures or redraw them.
Answer: The authors agree that your suggestion is good scientific style. Therefore, in two cases links are added, in all other cases figures are new or may only in parts look like old figures but are combined in a new way.
Reviewer

Author Response File: Author Response.docx

Reviewer 2 Report

Thank you for inviting me to evaluate the article titled “Evaluation of Different Generative Models to Support the Validation of Advanced Driver Assistance Systems”. In this paper, four kinds of scene generation methods based on artificial intelligence, including Variational Autoencoder, a basic Generative Adversarial Network, Wasserstein GAN, and Time-Series GAN, are compared and analyzed. The ability to cut into the scene to generate realistic and diverse trajectories for the cut-in scenario is evaluated, which provides a direction for the continuous progress of autonomous driving technology and has certain research significance. I recommend reconsidering after major revisions.

My detailed comments are as follows:

1. The introduction directly points out that this study applies and evaluates four AI-based scene generation models. Please supplement the reasons for choosing these four models, whether there are other traditional methods and corresponding limitations.
2. In the second chapter, only 25% of the measured cut-in maneuvers are selected as the training set. Please explain the reason for the selection of this proportion. Whether few training sets have an impact on the accuracy of the prediction results.
3. The source of the data and the description of the extracted scene are too brief and can be appropriately supplemented. Whether more experimental data sets be added to evaluate the adaptability and generalization ability of different models.
4. Experiments show that the VAE method exhibits excellent performance. The performance differences of each model on different indicators can be analyzed in more depth to help readers have a more detailed understanding of the advantages and disadvantages of each model.
5. As mentioned in the abstract, VAE enhanced with a Convolutional Neural Network shows excellent performance. However, the VAE method performs best among the four potential artificial intelligence models in the conclusion. Please unify the expression of the VAE method in the article.
6. There are many formatting problems in the article, which need to be further improved. For example, the indent of the first line of the paragraph, the symbol of the formula, the citing format of the reference, and so on.

Author Response

1. Comments: The introduction directly points out that this study applies and evaluates four AI-based scene generation models. Please supplement the reasons for choosing these four models, whether there are other traditional methods and corresponding limitations.

Additional comments are given in the introduction.

2. Comments: In the second chapter, only 25% of the measured cut-in maneuvers are selected as the training set. Please explain the reason for the selection of this proportion. Whether few training sets have an impact on the accuracy of the prediction results.

Thank you for the hint. While we initially trained our models using a more conventional data split with more data, we deliberately reduced the training set size in later experiments to simulate a more challenging situation for the generative models (additional comments given in the paper).

3. Comments: The source of the data and the description of the extracted scene are too brief and can be appropriately supplemented. Whether more experimental data sets be added to evaluate the adaptability and generalization ability of different models.

The dataset used in this study was already described in detail in our previous work [10]. The data cannot be provided as a free supplement as it is classified by the company.

4. Comments: Experiments show that the VAE method exhibits excellent performance. The performance differences of each model on different indicators can be analyzed in more depth to help readers have a more detailed understanding of the advantages and disadvantages of each model.

 Discussion of results is extended.

5. Comments: As mentioned in the abstract, VAE enhanced with a Convolutional Neural Network shows excellent performance. However, the VAE method performs best among the four potential artificial intelligence models in the conclusion. Please unify the expression of the VAE method in the article.

Thank you for the hint. The term VAE is now used more consistently.

6. Comments: There are many formatting problems in the article, which need to be further improved. For example, the indent of the first line of the paragraph, the symbol of the formula, the citing format of the reference, and so on.

Indention and reference style is according to the MDPI format. Rechecking of formula revealed some minor printing errors which have been corrected. Could you please clarify your observations if we missed some more errors?

Author Response File: Author Response.docx

Reviewer 3 Report

accepted

accepted

Author Response

1. Comment: Accepted.

Thank you for your favorable assessment.

Author Response File: Author Response.docx

Reviewer 4 Report

The article attempts to highlight the work being done to train AI-based methods for cut-in manuevers in ADAS systems. It is an interesting topic and of great importance, especially with the move towards autonomous vehicles. I had two suggestions for the authors to further improve the article.

1. In line 101, Was there a reason for using so little percentage data for the training? In many ML-based papers, the training dataset appears to be random data, but at least 50-60%?
2. It is commendable the extent to which the authors have described the different methods and the underlying mathematics. However, I would recommend adding citations to relevant works and reducing the verbose nature in those sections if possible.

Author Response

1. Comments: In line 101, Was there a reason for using so little percentage data for the training? In many ML-based papers, the training dataset appears to be random data, but at least 50-60%?

Thank you for the hint. While we initially trained our models using a more conventional data split with more data, we deliberately reduced the training set size in later experiments to simulate a more challenging situation for the generative models (additional comments given in the paper).

2. Comments: It is commendable the extent to which the authors have described the different methods and the underlying mathematics. However, I would recommend adding citations to relevant works and reducing the verbose nature in those sections if possible.

We have added some more citations regarding the procedures and the models used in the paper.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The introduction section now includes links to modern methods: (15-18). The authors also added a brief comparison of traditional and AI approaches to testing autonomous systems.
1) However, there is no direct comparison with previous works where similar methods were used (e.g., [7][9][10]).

2) There is still no information on other common scenarios (e.g., lane change, braking, crossing intersection).
It is not specified which of them are considered the most difficult and why. There is no citation of sources confirming that cut-in is indeed one of the most difficult scenarios.

3) I would like the authors to clearly indicate the novelty of their work in the text of the article, in contrast to the previous article:
Generation of Realistic Cut-In Maneuvers to Support Safety Assessment of Advanced Driver Assistance Systems

The figures and tables look good.

Author Response

1. Comments: However, there is no direct comparison with previous works where similar methods were used (e.g., [7][9][10]).

Answer 1: Thank you for the comments. In the revised paper we already discussed the mentioned literature with their limitations within lines 44-52. A more detailed comparison would not increase the information value. 

2. Comments: There is still no information on other common scenarios (e.g., lane change, braking, crossing intersection). It is not specified which of them are considered the most difficult and why. There is no citation of sources confirming that cut-in is indeed one of the most difficult scenarios.

Answer 2: This information is well known from the expert in the ADAS field and is commonly used as a base scenario for ADAS validation ([10], where we refer to research projects like PEGASUS,..). Even a comprehensive explanation is included in the revised paper in lines 69-73.

3. Comments: I would like the authors to clearly indicate the novelty of their work in the text of the article, in contrast to the previous article: Generation of Realistic Cut-In Maneuvers to Support Safety Assessment of Advanced Driver Assistance Systems

Answer 3: The novelty of this paper is mentioned in the revised paper in lines 56-66, as we are now using other network architectures for comparison purposes.

Author Response File: Author Response.docx

Reviewer 2 Report

This article explores the application of different generative models, including Variational Autoencoder, Generative Adversarial Network, Wasserstein GAN, and Time Series GAN, in supporting the validation of Advanced Driver Assistance Systems, and evaluates the authenticity, diversity, and statistical similarity of the trajectories generated by these models through qualitative and quantitative analysis, which has certain research significance. I recommend accepting after minor revisions.

My detailed comments are as follows:

1. When introducing the network topology structures of different models, it is recommended to supplement the analysis and explanation of the selection of hyperparameters (such as learning rate, regularization parameters, etc.) and optimizer configuration.
2. In the section of quantitative metrics for the comparison of generated results, the mean of the incoming variance of the outgoing and the Hungarian distance are selected as quantitative evaluation indicators. It is recommended to explain the reasons for selecting these two metrics and whether they are sufficient to evaluate the quality and diversity of the generated trajectories.
3. In addition to summarizing the advantages of the VAE method in generating trajectories, the conclusion section should also include a discussion of the limitations of different artificial intelligence models (including VAE, GAN, WGAN, and TimeGAN) to more comprehensively explain the applicability and improvement directions of each model.

Author Response

1. Comments: When introducing the network topology structures of different models, it is recommended to supplement the analysis and explanation of the selection of hyperparameters (such as learning rate, regularization parameters, etc.) and optimizer configuration.

Answer 1: Thank you for your valuable comments. We added the learning rates and the grid search with the corresponding hyperparameters within the new revised paper in lines 343-350. 

 

2. Comments: In the section of quantitative metrics for the comparison of generated results, the mean of the incoming variance of the outgoing and the Hungarian distance are selected as quantitative evaluation indicators. It is recommended to explain the reasons for selecting these two metrics and whether they are sufficient to evaluate the quality and diversity of the generated trajectories.

Answer 2: In an unpublished Master thesis of one of the authors and in an unpublished dissertation there were more metrics investigated, but the information value was too low to integrate it within this paper. 

 

3. Comments: In addition to summarizing the advantages of the VAE method in generating trajectories, the conclusion section should also include a discussion of the limitations of different artificial intelligence models (including VAE, GAN, WGAN, and TimeGAN) to more comprehensively explain the applicability and improvement directions of each model.

Answer: The limitations of each AI model are already investigated with the analysis of the used metrics within the revised paper. An improvement of each model for the ADAS application can be analysed in future investigations.

Author Response File: Author Response.docx