Flight Test Analysis of UTM Conflict Detection Based on a Network Remote ID Using a Random Forest Algorithm
Round 1
Reviewer 1 Report (Previous Reviewer 2)
I thank the authors for addressing my comments in the re-submitted version.
Please revise the numbering of sections because e.g., 2.3 is present twice in the document.
The language throughout the document is mostly fine.
Author Response
Thank you for pointing out a mistake. We change it accordingly in line 282. Also, we checked for the rest of the text and found the mistake only on that point.
Please mind that there are other changes in the revised manuscript for comments from other reviewers.
Author Response File: Author Response.docx
Reviewer 2 Report (Previous Reviewer 1)
In this paper, there are still need a mathematical model to know that the intruder UAV is inside the area of the inverted teardrop shape or not. Please add it.
Author Response
Thank you for the feedback. The algorithm of conflict detection is shown in Figure 3. We add the detailed mathematical calculation for this algorithm below Figure 3 in subsection 2.1 as shown in Equations 1 to 7. The changes are highlighted in light blue color.
Please mind that there are other changes in the revised manuscript for comments from other reviewers.
Author Response File: Author Response.docx
Reviewer 3 Report (New Reviewer)
The manuscript needs a lot of work before going ahead. Some impornant changes in the structured are necessary in order to be much more clear despite of it depicts an interesting topic. Some re-arrangements and changes are necessary before going ahead. If authors well follow the suggestion given I will give a chance to the manuscript to be published on Drones.
Firstly, I suggest you to imporve the introduction section and in particular discussing about the application of Drones and the novelty that you discuss in some multidiciplinary context. To do so and help you I suggest to consider to include the following references:
- https://doi.org/10.3390/life13040987
- https://doi.org/10.1080/01431161.2018.1523832
Then, at the end of the introduction section I suggest you to improve the aim of your work better describing the novelty of the work. You have partially done but i suggest to remark the framework.
In material and methods are completely missing add this sections as well as results and discussion. Consider to include the versions of the software and produce a summary workflow of each phase you done as an image. Moreover, in figure that report maps please introduce the EPSG or the datum.
Report validation, uncertianity and accuracies in resuts.
Consider to divide conclusions and reccomandations hoewever as you structured the paper it seems a review but it is not clear the typology you reported article...
Moderate editing of English language required.
Author Response
Thank you for the comments. We are trying our best to answer it and the changes are highlighted in yellow color in the revised manuscript.
Comment: Firstly, I suggest you to imporve the introduction section and in particular discussing about the application of Drones and the novelty that you discuss in some multidiciplinary context. To do so and help you I suggest to consider to include the following references:
- https://doi.org/10.3390/life13040987
- https://doi.org/10.1080/01431161.2018.1523832
Answer: We add the suggestion to the beginning part of Chapter 1 with additional references, but your 1st recommended reference is not related to UAV or UTM context.
Comment: Then, at the end of the introduction section I suggest you to improve the aim of your work better describing the novelty of the work. You have partially done but i suggest to remark the framework.
Answer: We revised our research aim and novelty in the last paragraph of Chapter 1.
Comment: In material and methods are completely missing add this sections as well as results and discussion.
Answer: we used unconventional naming for the manuscript sections. Actually, chapter 2 is about the framework of the research, chapter 3 is about the methodology, and Chapter 4 is about results and discussion. We revised the section names to be more familiar terms in chapters 2, 3, and 4
Comment: Consider to include the versions of the software and produce a summary workflow of each phase you done as an image. Moreover, in figure that report maps please introduce the EPSG or the datum.
Answer: Versioning of software Python, Bokeh, and SKLEARN is added as suggested in the first paragraph of subsection 3.1., and the second paragraph of subsection 4.4. We do believe most of the process is supported by a flowchart or a diagram for better understanding, except in the machine learning section. Thus, we add the diagram of the Random Forest algorithm in Figure 5. The subsequent figure numbering is adjusted accordingly. Figures 8 and 9 are updated to depict the datum as requested.
Comment: Report validation, uncertianity and accuracies in resuts.
Answer: Additional parameters and their explanations are added for the validation, uncertainty, and accuracy of the result in subsections 4.4.1 and 4.4.2. Uncertainty plots are shown in Figures 16 and 19.
Comment: Consider to divide conclusions and reccomandations hoewever as you structured the paper it seems a review but it is not clear the typology you reported article...
Answer: We split this chapter into subsection 5.1 for the conclusions and 5.2 for the recommendations.
Please mind that there are other changes in the revised manuscript for comments from other reviewers.
Author Response File: Author Response.docx
Round 2
Reviewer 3 Report (New Reviewer)
The manuscript has been improved however some changes in introduction still be necessary. In particular better discussing the role of UTM Conflict using Random Forest Algorithm especially in remote sensing product. To do so and help you conisder to include the following references:
- https://doi.org/10.3390/rs15092348
- https://books.google.it/books?hl=it&lr=&id=TII0EAAAQBAJ&oi=fnd&pg=PT8&dq=utm+drone&ots=AivmSiF5wp&sig=VvNFlwDTYzzwGAhIQIMoAAkSiXY&redir_esc=y#v=onepage&q=utm%20drone&f=false
after this the manuscript can be considered suitable for pubblication.
Minor editing of English language required
Author Response
Thank you for the feedback. We modified the introduction chapter in paragraph 3 to add several examples of remote sensing applications referring to the suggested references. We also add a discussion about UTM conflict detection and its possibility to use machine learning such as random forest algorithm in paragraphs 6 and 7 of the introduction chapter as recommended. The changes are highlighted in yellow in the revised manuscript.
Author Response File: Author Response.docx
This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.
Round 1
Reviewer 1 Report
1. According to Figure 5, it can be seen that the conflict detection area is a region composed of multiple circles stacked together, which is inconsistent with the inverted teardrop shape described in 2.3. Please provide additional explanation.
2. Assuming the two unmanned aerial vehicles flying in opposite directions, the angle betwwen their motion is 0 to 90 degrees. And the distance between them is less than or equal to 6r, greater than 3r. According to Figure 3, the area detected by the conflict detection algorithm should be an annual sector, which is inconsistent with the inverted teardrop shape area described in 2.3. Please provide additional explanation.
3. It is recommended to add scenarios to verify the effectiveness of the algorithm, such as the conflict scenario of two drones flying in the same direction at the same or different altitude, the conflict scenario of two drones flying in the opposite direction at the same or different altitude and so on.
4. According to the content of the article, two drones are operating and only one drone is instructed to maneuver for collision avoidance. How could we determine which drone will perform collision avoidance in multiple drones flying situation? From Figure 5, it could be seen that one drone with green area is not indicated for warning. How to ensure the safety level of this drone?
5. According to conclusion 4.2.2, the average latency time is about 1.57 seconds, and what is the longest latency time? What is the impact of latency time on drones warning when the speed of the drone is fast?
Author Response
- According to Figure 5, it can be seen that the conflict detection area is a region composed of multiple circles stacked together, which is inconsistent with the inverted teardrop shape described in 2.3. Please provide additional explanation.
Thank you for this feedback. Actually, the most challenging process in the UI design of our UTM monitoring is how to draw the inverted teardrop shape because there is no built-in teardrop shape in the Python-Bokeh module. However, the best option found is using multiple circles with different radii arranged in the direction of the UAV flight to form an inverted teardrop shape. We add this additional information in the first paragraph of subsection 3.1.
- Assuming the two unmanned aerial vehicles flying in opposite directions, the angle between their motion is 0 to 90 degrees. And the distance between them is less than or equal to 6r, greater than 3r. According to Figure 3, the area detected by the conflict detection algorithm should be an annual sector, which is inconsistent with the inverted teardrop shape area described in 2.3. Please provide additional explanation.
Based on the algorithm in Figure 3, when an intruder UAV is at a distance between 3r and 6r from the own UAV, it will trigger a warning if the intruder location is inside the area of the inverted teardrop shape. Otherwise, no warning will be produced. Following the shape of an inverted teardrop, the closer the intruder UAV is to the flight path of the own UAV, the longer the detection distance is considered. This is the difference between this teardrop shape area detection algorithm compared to the circle detection area. We add this additional information in subsection 2.2 in the paragraph just before Figure 3.
- It is recommended to add scenarios to verify the effectiveness of the algorithm, such as the conflict scenario of two drones flying in the same direction at the same or different altitude, the conflict scenario of two drones flying in the opposite direction at the same or different altitude and so on.
Thank you for the recommendation. In scenario 2, we conducted several flights with two UAVs flying together without strict flight patterns. At some points, the UAV could be in the opposite direction or in the same direction or other patterns. This is one of the reasons we selected a machine learning analysis technique that is able to detect a hidden pattern of the data without a clear input setting in analyzing the warning detection. We believe this scenario accommodates your suggestion. Thus, no additional information is added to the text.
- According to the content of the article, two drones are operating and only one drone is instructed to maneuver for collision avoidance. How could we determine which drone will perform collision avoidance in multiple drones flying situation? From Figure 5, it could be seen that one drone with green area is not indicated for warning. How to ensure the safety level of this drone?
In our inverted teardrop method, the own UAV that has UAV intruders in its detection area is obliged to make the maneuver. For the case mentioned in the question, only one UAV will make an avoidance maneuver. We believe a maneuver from one UAV is enough to avoid a conflict. Also, this scenario will avoid a case where both UAVs execute avoidance maneuvers that leads to another conflict. We add in this additional information in the last paragraph of subsection 3.3.
- According to conclusion 4.2.2, the average latency time is about 1.57 seconds, and what is the longest latency time? What is the impact of latency time on drones warning when the speed of the drone is fast?
According to Figure 12.d, the longest latency time is around 24 s. It is considered an outlier in the boxplot because the value is much larger than the average and its standard deviation. Actually, the contribution to his large value comes from the query time which is affected by the internet signal quality. This is one of the findings in our conclusion. We understood that latency time is proportional to the uncertainty of the UAV position and become more severe when the UAV speed is fast. It affects the safety of UAV operation and we will consider this finding for our future research. We add in this additional information in the last paragraph of subsection 4.2.2 and also in the last paragraph of conclusion chapter
Author Response File: Author Response.pdf
Reviewer 2 Report
The authors consider the case of UTM systems used for conflict detection exploiting network remote id for UAV identification.
Overall, the work has merit. Some parts need careful proofreading and a more streamlined exposition.
Some comments:
* please rephrase the abstract to introduce the scope of the work, your approach, and the significance of the submission
* define what a conflict is before delving into the literature
* the state of the art could be better organised and subsections should be used to distinguish among V2V, V2G, and mixed approaches
* clarify the novelty in your work
* section 2.3: RF is superior in accuracy ... . state that RF, according to [25], is better in this context, not in a general sense
* please use total time or latency time to avoid confusing the reader. is it defined as the sum of the broadcast plus query plus process time?
* safe zones around the UAVs should be enlarged or reduced according to the total time because the timing of warnings is strictly dependent on it. is it correct? elaboration is needed on this point
* boxplots of different quantities (e.g., broadcast time, process time) should not use the same scale for the sake of improved readability. Figure 11b (or 12b), for instance, cannot be fully appreciated as it is
* please elaborate on the significance of section 4.4 to motivate your choice of RF and not e.g. PCA
* the authors conclude that the results from RF are not surprising. could the same conclusions be reached by statistical analysis only?
please proceed to a careful proofreading and streamlining of the language
Author Response
Some comments:
* please rephrase the abstract to introduce the scope of the work, your approach, and the significance of the submission
Thank you for the feedback. The scope of the UTM-based network Remote ID is using the communication channel of Taiwan’s cellular communication network in suburban and rural areas. The approach is already mentioned in the explanation of our research method. The significance of our research is that the result and findings can be used as a reference for aviation authorities and other stakeholders for the development of future UTM systems. This additional information is added to our abstract.
* define what a conflict is before delving into the literature
We add the conflict definition in paragraph 2 of chapter 1. The conflict is defined as “an event in which two or more aircraft experience a loss of minimum separation” [J. K. Kuchar and L. C. Yang, “A Review of Conflict Detection and Resolution Modeling Methods,” IEEE Trans. Intell. Transp. Syst., vol. 1, no. 4, pp. 179–189, 2000, doi: 10.1109/6979.898217.]
* the state of the art could be better organised and subsections should be used to distinguish among V2V, V2G, and mixed approaches
We revise Chapter 1 with additional formatting for Subchapter 1.1. V2G, 2.2. V2V, and 2.3 mixed communications
* clarify the novelty in your work
The novelties of our research are the usage of network Remote ID to develop a UTM monitoring system and the conflict detection algorithm based on an inverted teardrop shape. So far, no research is found on those topics. We add this information in the last paragraph of Chapter 1.
* section 2.3: RF is superior in accuracy ... . state that RF, according to [25], is better in this context, not in a general sense
We refer to the information to the reference [25] from MDPI Aerospace Journal about critical parameter identification in commercial aviation using machine learning. The reference makes a comparison of several candidates of classification algorithms, namely Boosting Ensemble, Decision Tree, K-Nearest Neighbour, Naive Bayes, and Random Forest. We agree with you that the RF algorithm is recommended for this kind of research. Thus, we follow their recommendation to use Random Forest because our analysis context is similar to their research. We revise one sentence in the first paragraph of subsection 2.3 to mention this context of recommendation.
* please use total time or latency time to avoid confusing the reader. is it defined as the sum of the broadcast plus query plus process time?
We change all total time terms into latency time such as in subsections 2.2 and 4.1.
* safe zones around the UAVs should be enlarged or reduced according to the total time because the timing of warnings is strictly dependent on it. is it correct? elaboration is needed on this point
The fixed radius of the near-collision circle is referred to the research from MIT Lincoln Laboratory (MIT LL) and the John Hopkins University Applied Physics Laboratory (JHU APL) [19], [20], [21] about the minimum fixed separation. We found additional reasoning that this fixed separation concept has been demonstrated successfully in Airborne Collision Avoidance Systems for small UAS (ACAS sXu) systems to mitigate the likelihood of a loss of separation. Thus we add this additional information in the second paragraph of subsection 2.2.
* boxplots of different quantities (e.g., broadcast time, process time) should not use the same scale for the sake of improved readability. Figure 11b (or 12b), for instance, cannot be fully appreciated as it is
We re-adjust and change the boxplot in Figures 11b, 12b, and 13f as suggested.
* please elaborate on the significance of section 4.4 to motivate your choice of RF and not e.g. PCA
We found that the RF algorithm is an ensemble of multi-decision trees with the advantages of versatile uses, easy-to-understand hyperparameters, and avoiding overfit with enough trees compared to other machine learning techniques. It is also one of the most-used algorithms, due to its simplicity and diversity that can be used for both classification and regression tasks. We add these significances of the RF algorithm in the first paragraph of subsection 4.4.
* the authors conclude that the results from RF are not surprising. could the same conclusions be reached by statistical analysis only?
In our study, the machine learning and statistical results are supporting each other. However, no guarantee that other cases could have a similar result. For this question, we did not add additional information to the text.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
1. According to Figure 5, it can be seen that the conflict detection area is a region composed of multiple circles stacked together, which is not the strict inverted teardrop shape. Also, this question was mentioned last time, there is no additional explanation or amendment.
2. Assuming the two unmanned aerial vehicles flying in opposite directions, the angle between their motion is 0 to 90 degrees. And the distance between them is less than or equal to 6r, greater than 3r. According to Figure 3, the area detected by the conflict detection algorithm should be an annual sector, which is inconsistent with the inverted teardrop shape area described in 2.3. Also, this question was mentioned last time, there is no additional explanation or amendment.
3. It is recommended to add scenarios to verify the effectiveness of the algorithm, such as the conflict scenario of two drones flying in the same direction at the same or different altitude, the conflict scenario of two drones flying in the opposite direction at the same or different altitude and so on. Also, this question was mentioned last time, there is no additional explanation or amendment.
4. According to the content of the article, two drones are operating and only one drone is instructed to maneuver for collision avoidance. How could we determine which drone will perform collision avoidance in multiple drones flying situation? From Figure 5, it could be seen that one drone with green area is not indicated for warning. How to ensure the safety level of this drone? Also, this question was mentioned last time, there is no additional explanation or amendment.
5. Figure 11 (b) is missing.
6. Figure 12 (b) is different from the last time, but the summing-up is the same. It is hard to understand.
Author Response
Dear Reviewer,
Thank you for your second feedback. Actually, we answered all your first feedback and accommodated it in the amendments of our revised submitted manuscript. I am not sure why it is not shown in your received file. Perhaps because we are advised to use the Track Change function in MS Word, you can see the differences when you turn the function on. I am sorry for the loophole. Here, I add in the screenshot of the revised page for each of the feedback.
1. According to Figure 5, it can be seen that the conflict detection area is a region composed of multiple circles stacked together, which is not the strict inverted teardrop shape. Also, this question was mentioned last time, there is no additional explanation or amendment.
Answer: Actually, the most challenging process in the UI design of our UTM monitoring is how to draw the inverted teardrop shape because there is no built-in teardrop shape in the Python-Bokeh module. However, the best option found is using multiple circles with different radii arranged in the direction of the UAV flight to form an inverted teardrop shape. We add this additional information in the first paragraph of subsection 3.1. The screenshot of the amendment is:
2. Assuming the two unmanned aerial vehicles flying in opposite directions, the angle between their motion is 0 to 90 degrees. And the distance between them is less than or equal to 6r, greater than 3r. According to Figure 3, the area detected by the conflict detection algorithm should be an annual sector, which is inconsistent with the inverted teardrop shape area described in 2.3. Also, this question was mentioned last time, there is no additional explanation or amendment.
Answer: Based on the algorithm in Figure 3, when an intruder UAV is at a distance between 3r and 6r from the own UAV, it will trigger a warning if the intruder location is inside the area of the inverted teardrop shape. Otherwise, no warning will be produced. Following the shape of an inverted teardrop, the closer the intruder UAV is to the flight path of the own UAV, the longer the detection distance is considered. This is the difference between this teardrop shape area detection algorithm compared to the circle detection area. We add this additional information in subsection 2.2 in the paragraph just before Figure 3. The screenshot of amendment is:
3. It is recommended to add scenarios to verify the effectiveness of the algorithm, such as the conflict scenario of two drones flying in the same direction at the same or different altitude, the conflict scenario of two drones flying in the opposite direction at the same or different altitude and so on. Also, this question was mentioned last time, there is no additional explanation or amendment.
Answer:
Thank you for the recommendation. In scenario 2, we conducted several flights with two UAVs flying together without strict flight patterns. At some points, the UAV could be in the opposite direction or in the same direction or other patterns. This is one of the reasons we selected a machine learning analysis technique that is able to detect a hidden pattern of the data without a clear input setting in analyzing the warning detection. We believe this scenario accommodates your suggestion. Thus, no additional information is added to the text.
4. According to the content of the article, two drones are operating and only one drone is instructed to maneuver for collision avoidance. How could we determine which drone will perform collision avoidance in multiple drones flying situation? From Figure 5, it could be seen that one drone with green area is not indicated for warning. How to ensure the safety level of this drone? Also, this question was mentioned last time, there is no additional explanation or amendment.
Answer: In our inverted teardrop method, the own UAV that has UAV intruders in its detection area is obliged to make the maneuver. For the case mentioned in the question, only one UAV will make an avoidance maneuver. We believe a maneuver from one UAV is enough to avoid a conflict. Also, this scenario will avoid a case where both UAVs execute avoidance maneuvers that leads to another conflict. We add in this additional information in the last paragraph of subsection 3.3. The screenshot of the amendment is:
5. Figure 11 (b) is missing.
Answer: Actually, another reviewer requests to re-scale Figures 11.b, 12.b, and 13.f for the sake of improved readability. Unfortunately, I miss to re-paste Figure 11.b since I used the Track Change function which seems to be the Figure there. Now, I add the Figures. The screenshot is:
6. Figure 12 (b) is different from the last time, but the summing-up is the same. It is hard to understand.
Answer: It is the same as in the previous question. The screenshot is:
I hope now, you can see all the amendments in our revised manuscript and consider the answers. Thank you for your feedback.
Author Response File: Author Response.docx