Review Reports - YOLOv8-Based Drone Detection: Performance Analysis and Optimization

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1.There is a spelling error in Table 1 where "Confussion Matrix" should be corrected to "Confusion Matrix". On page 14, the word "comparision" in the sentence “The term 'True Positive values' is used for comparision in the title because the performances of other models are compared using this parameter.” should be corrected to "comparison". There are other typos in this article, and it is recommended to check them carefully and correct them.

2.The paper uses a publicly available drone dataset and applies data augmentation and model training. However, the description of dataset diversity and experimental design is not detailed enough. For example, the impact of shooting angles and background complexity on model performance is not discussed. It is recommended to provide more information on the diversity and complexity of the dataset, specifically detailing the distribution of data under different environmental conditions.

3.The paper compares the performance of YOLOv8 with other models such as YOLOv4, YOLOv5, and Mask R-CNN, demonstrating the superiority of YOLOv8. However, the analysis of the reasons behind the performance differences is not in-depth. It is suggested to provide a more detailed analysis of the performance differences between the models, exploring the strengths and weaknesses of different models in various detection scenarios.

4.In the fifth line of the abstract, "... dataset collected by Mehdi Ö zel for a UAV competition is sourced from 5 GitHub." The "are" here should be changed to "is" because "dataset" is a singular noun.

5.In the third line of the second paragraph on page two, "Many different studies have been executed to detect drones via using radars and alternative methods [10]," the word "via using" is repeated and should be simplified to "using". Correct to: "Many different studies have been executed to detect drones using radars and alternative methods [10]."

6.The sentence "we utilized the freely available dataset collected by Mehdi Ö zel for a UAV competition was used" on the first line of the seventh paragraph on page four is structurally incorrect. It should be "we utilized the freely available dataset collected by Mehdi Ö zel for a UAV competition" [19] Remove 'was used' to make the sentence smoother.

Comments on the Quality of English Language

There are some grammatical errors that need to be corrected.

Author Response

Dear Editor,

Thank you very much for your valuable feedback. Below, I am providing the responses and update information.

Comments 1: There is a spelling error in Table 1 where "Confussion Matrix" should be corrected to "Confusion Matrix". On page 14, the word "comparision" in the sentence “The term 'True Positive values' is used for comparision in the title because the performances of other models are compared using this parameter.” should be corrected to "comparison". There are other typos in this article, and it is recommended to check them carefully and correct them.

Response 1: The two spelling errors mentioned have been corrected and the document has been reviewed again. The content has been updated with a few modifications based on recent discussions.

Comments 2: The paper uses a publicly available drone dataset and applies data augmentation and model training. However, the description of dataset diversity and experimental design is not detailed enough. For example, the impact of shooting angles and background complexity on model performance is not discussed. It is recommended to provide more information on the diversity and complexity of the dataset, specifically detailing the distribution of data under different environmental conditions.

Response 2: Detailed information about the dataset is not provided in the abstract or introduction, but rather under the section 3.1 Experimental Environment and Dataset. This section includes graphs related to the location and size distribution of the dataset. Various augmentation methods have been applied to the images in the dataset, and their effects are detailed under section 4.2 Data Augmentation.

Comments 3: The paper compares the performance of YOLOv8 with other models such as YOLOv4, YOLOv5, and Mask R-CNN, demonstrating the superiority of YOLOv8. However, the analysis of the reasons behind the performance differences is not in-depth. It is suggested to provide a more detailed analysis of the performance differences between the models, exploring the strengths and weaknesses of different models in various detection scenarios.

Response 3: An explanation has been added under the section 4.4 Comparison With Other Models in this regard. However, a detailed analysis of the points mentioned in the discussion could be included under the Future Works section. It would be beneficial to consider this in the 5. FUTURE WORKS section and include information on how such a study would be advantageous.

Comments 4: In the fifth line of the abstract, "... dataset collected by Mehdi Ö zel for a UAV competition is sourced from 5 GitHub." The "are" here should be changed to "is" because "dataset" is a singular noun.

Response 4: The suggestion has been implemented.

Comments 5: In the third line of the second paragraph on page two, "Many different studies have been executed to detect drones via using radars and alternative methods [10]," the word "via using" is repeated and should be simplified to "using". Correct to: "Many different studies have been executed to detect drones using radars and alternative methods [10]."

Response 5: The suggestion has been implemented.

Comments 6: The sentence "we utilized the freely available dataset collected by Mehdi Ö zel for a UAV competition was used" on the first line of the seventh paragraph on page four is structurally incorrect. It should be "we utilized the freely available dataset collected by Mehdi Ö zel for a UAV competition" [19] Remove 'was used' to make the sentence smoother.

Response 6: The suggestion has been implemented.

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript only showed the evaluation results of YOLOv8 for the drone image dataset but did not propose novel a method for drone detection. For publication, it is needed to improve existing methods or propose a new approach.

1. The pros and cons should be described for image-based detection methods and other methods that are radar-based, radio frequency-based, and sound signal-based methods.

2. The analysis of existing studies on image-based methods for drone detection via neural networks needs to be supplemented. In other words, Section 2 should handle more related studies.

3. The captions of Fig. 3 (a) and (b) are the same. Fig. 3. (a) is meaningless because the dataset has only one class. The meaning of many red squares in Fig. 3 (b) is ambiguous.

4. In Table 2, what is the difference between ‘YOLOv8’ and ‘Proposed method’? In addition, the experiment results seem to use different dataset for the models. For strict evaluation, the experiment should be performed with the same dataset.

Comments on the Quality of English Language

Minor editing of English language required.

Author Response

Dear Editor,

Thank you very much for your valuable feedback. Below, I am providing the responses and update information.

A more detailed description of the methodology has been updated under the abstract and distributed across subheadings.

Comments 1: The pros and cons should be described for image-based detection methods and other methods that are radar-based, radio frequency-based, and sound signal-based methods.

Response 1: Content has been updated under the 1. Introduction section in light of this view.

Comments 2: The analysis of existing studies on image-based methods for drone detection via neural networks needs to be supplemented. In other words, Section 2 should handle more related studies.

Response 2: Information on new studies has been added under the 2. Literature Review section.

Comments 3: The captions of Fig. 3 (a) and (b) are the same. Fig. 3. (a) is meaningless because the dataset has only one class. The meaning of many red squares in Fig. 3 (b) is ambiguous.

Response 3: In the context of the discussion, the figure has been updated to provide a clearer version.

Comments 4: In Table 2, what is the difference between ‘YOLOv8’ and ‘Proposed method’? In addition, the experiment results seem to use different dataset for the models. For strict evaluation, the experiment should be performed with the same dataset.

Response 4: The method referred to as “YOLOv8” is the model that reflects the results without any augmentation or hyperparameter tuning applied." The “Proposed Model” is the model that hyperparameter tuning and data augmentation techniques were applied to maximize the performance of the model. The experiments were conducted with the same dataset, and the model was developed using appropriate augmentation methods.

Author Response File: Author Response.docx

Reviewer 3 Report

Comments and Suggestions for Authors

The paper discusses the application of deep learning techniques, specifically the YOLOv8 model, for drone detection. Given the widespread use of drones, the study aims to mitigate the associated risks through early detection. The dataset for training the model was sourced from GitHub and labeled using Roboflow. The model was trained on Google Colab, incorporating dataset augmentation techniques such as rotation and blurring. The model achieved high performance metrics, including a precision of 0.946, recall of 0.9605, and a precision-recall curve value of 0.978, outperforming other popular models like Mask CNN, CNN, and YOLOv5.

The paper addresses a relevant and timely issue, leveraging deep learning for enhancing drone detection, which is crucial for security purposes. The reported high precision, recall, and precision-recall curve values indicate the effectiveness of the YOLOv8 model and the proposed methodology. The study employs a systematic approach, from dataset collection and labeling to model training and performance enhancement through data augmentation techniques.

Please find some comments and suggestion below:

Provide more details about the dataset used, including the number of images, variety of drone types, and different environmental conditions. Explain the training process in more detail, including the hyperparameters used, the number of epochs, and the computational resources required.

Include a detailed comparison with other models (Mask CNN, CNN, YOLOv5) in terms of performance metrics and computational efficiency. Introduce more recent work on the application of lightweight neural networks for detection tasks, such as: https://doi.org/10.1007/s00170-022-10335-8

Conduct ablation studies to demonstrate the impact of different augmentation techniques (rotation, blurring) on the model's performance. This will help to understand the contribution of each technique to the overall improvement in performance metrics.

Suggest potential future work or enhancements that could further improve the detection accuracy or reduce the computational load. Explore the possibility of integrating the system with real-time drone monitoring systems.

Overall, the manuscript presents a significant advancement in drone detection using the YOLOv8 model. With additional details on the methodology, comparative analysis, ablation studies, and real-world applicability, the paper has the potential to make a substantial contribution to the field. Minor improvements in English writing quality will enhance the manuscript's overall readability. The paper presents valuable and innovative approaches to improving drone detection, but it requires additional details and comprehensive evaluation to fully substantiate the claimed improvements and ensure clarity for the readers.

Comments on the Quality of English Language

The quality of English in the manuscript is generally good, with clear and coherent language.

Author Response

Dear Editor,

Thank you very much for your valuable feedback. Below, I am providing the responses and update information.

Please find some comments and suggestion below:

Comments 1: Provide more details about the dataset used, including the number of images, variety of drone types, and different environmental conditions. Explain the training process in more detail, including the hyperparameters used, the number of epochs, and the computational resources required

Response 1: Detailed information about the dataset is not provided in the abstract or introduction, but rather under the section 3.1 Experimental Environment and Dataset. This section includes graphs related to the location and size distribution of the dataset. Various augmentation methods have been applied to the images in the dataset, and their effects are detailed under section 4.2 Data Augmentation.

Comments 2: Explain the training process in more detail, including the hyperparameters used, the number of epochs, and the computational resources required.

Response 2: Detailed information about training process is provided in the section 4.1. Hyperparameter Settings and 4.2. Dataset Augmentation Method.

Comments 3: Include a detailed comparison with other models (Mask CNN, CNN, YOLOv5) in terms of performance metrics and computational efficiency. Introduce more recent work on the application of lightweight neural networks for detection tasks, such as: https://doi.org/10.1007/s00170-022-10335-8

Response 3: Detailed comparison contents are provided under Section 4.4 Comparision with other models section. Recent articles related to the subject have been added under 2. Literatur Review section.

Comments 4: Conduct ablation studies to demonstrate the impact of different augmentation techniques (rotation, blurring) on the model's performance. This will help to understand the contribution of each technique to the overall improvement in performance metrics.

Response 4: The ablation techniques briefly outlined in the abstract are discussed in detail in Section 4. The most effective results achieved in this study are presented as the proposed model.

Comments 5: Suggest potential future work or enhancements that could further improve the detection accuracy or reduce the computational load. Explore the possibility of integrating the system with real-time drone monitoring systems.

Response 5: A similar scope of future work was covered in Section 5. Future Works, but the significance was emphasized within the context of your perspective.

Author Response File: Author Response.docx

Reviewer 4 Report

Comments and Suggestions for Authors

The paper presents a study on drone detection using the YOLOv8 model. The authors focus on optimizing the model's performance through hyperparameter tuning and data augmentation techniques. They evaluate the model's effectiveness using a publicly available dataset and compare its performance against other models like YOLOv5, YOLOv4, and Mask R-CNN. The study concludes that YOLOv8 outperforms these models in terms of precision and recall.

How does YOLOv8 compare to previous YOLO versions in terms of architectural improvements?
What specific data augmentation techniques contributed most to the performance gains?
How can the results of this study be applied in real-world drone detection scenarios?
Critical Inquiry:
What are the limitations of using the YOLOv8 model for drone detection, particularly in adverse weather conditions?
Is the dataset used in the study sufficiently diverse to ensure the generalizability of the model?
How do the findings of this study align with or contradict previous research on drone detection using deep learning methods?

Author Response

Comments 1: How does YOLOv8 compare to previous YOLO versions in terms of architectural improvements?

Response 1: The introduction section has been updated with a new paragraph as part of this discussion.

Comments 2: What specific data augmentation techniques contributed most to the performance gains?

Response 2: As part of this discussion, information has been added under the abstract section.

Comments 3: How can the results of this study be applied in real-world drone detection scenarios?

Response 3: Information has been provided within this context under the Introduction heading, and updates have also been made as part of this review.

Comments 4: What are the limitations of using the YOLOv8 model for drone detection, particularly in adverse weather conditions?

Response 4: In this discussion, a new paragraph has been added under the Introduction heading.

Comments 5: Is the dataset used in the study sufficiently diverse to ensure the generalizability of the model?

Response 5: This study was conducted on the same dataset in order to obtain comparable results with other studies in the literature. This approach was preferred to ensure methodological consistency and to ensure that the results obtained can be meaningfully compared with other studies. If a different dataset had been used, the comparisons would have lost their reliability, and it would have been difficult to establish a meaningful relationship between the results. Therefore, it is very important to continue with the current dataset to solidify my work in the literature. It is thought that a larger dataset could significantly improve the performance of my model. However, to obtain results that are compatible and comparable with existing studies in the literature, we preferred to stay with the current dataset in this study.

Comments 6: How do the findings of this study align with or contradict previous research on drone detection using deep learning methods?

Response 6: The findings of this study are generally consistent with previous studies and in some ways take these studies further. Previous studies have shown that models such as YOLOv4 and YOLOv5 are effective in drone detection, but these models have limitations in terms of accuracy. YOLOv8, as shown in this study, offers significant improvements in accuracy. These findings show that deep learning models are constantly evolving and becoming more suitable for complex tasks such as drone detection. The study also emphasizes the importance of data augmentation and hyperparameter optimization in improving model performance, confirming the effectiveness of these techniques, consistent with previous studies.

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Although the manuscript has been modified, it is still hard to find novelty. The hyperparameter tuning and the data augmentation presented in the manuscript are the common techniques for network improvement. In image-based drone detection, it is important to accurately detect small-sized objects. That is, it should be possible to distinguish between drones and other objects of similar shape for small size. However, the drone image dataset used in the manuscript appears to be mostly for large-size drone objects.

Comments on the Quality of English Language

Minor editing of English language required.

Author Response

Comments 1: Although the manuscript has been modified, it is still hard to find novelty. The hyperparameter tuning and the data augmentation presented in the manuscript are the common techniques for network improvement. In image-based drone detection, it is important to accurately detect small-sized objects. That is, it should be possible to distinguish between drones and other objects of similar shape for small size. However, the drone image dataset used in the manuscript appears to be mostly for large-size drone objects.

Response 1: This study was conducted on the same dataset in order to obtain comparable results with other studies in the literature. This approach was preferred to ensure methodological consistency and to ensure that the results obtained can be compared meaningfully with other studies. If a different dataset was used, the comparisons could lose their reliability, and it would be difficult to establish a meaningful relationship between the results. Therefore, we consider that proceeding on the dataset used in methods such as YOLOv5, YOLOv4 and Mask R-CNN in the literature is very important to solidify the place of my study in the literature. The scope of our study was primarily to evaluate the performance improvements in the criteria determined with the existing dataset. The detection of small objects is of course of critical importance in the field of drone detection, and we understand your concerns in this regard. However, the primary goal of our study is to optimize the performance of YOLOv8 on larger objects in the existing dataset and compare this performance with widely accepted methods. In this context, collecting a separate dataset for small object detection or modifying the existing dataset would not only exceed the scope of the study, but also affect the validity of the comparison. In addition to these explanations and the previous updates, revisions have been made to the Introduction and abstract sections to strengthen the novelty.

Reviewer 4 Report

Comments and Suggestions for Authors

Accept in present form

Round 3

Reviewer 2 Report

Comments and Suggestions for Authors

I agree that reliable performance comparisons between various detection models can be made even using existing drone dataset. However, the manuscript does not propose a novelty approach or a modification of the network structure in order to improve the traditional YOLOv8. The manuscript appears to be limited to verifying the detection performance of YOLOv8 for drone objects.

In addition, it has been already well known that image transformation for data augmentation and hyper-parameter setting can improve the network performance, which is difficult to consider as the novelty or contribution of the proposed method.

Comments on the Quality of English Language

Minor editing of English language required.