Strategies for Scaleable Communication and Coordination in Multi-Agent (UAV) Systems
Round 1
Reviewer 1 Report
The work focuses on a very timely topic. Multi-UAV control and communication are key issues in the impending air mobility spaces (for example, in UTM).
I consider that authors describe all the complexities associated with these two issues in a very clear manner and provide very focused possible solutions existing in the literature, in the context of reinforcement learning.
On the other hand, the proposed architecture, based on interactive state spaces and multi-level clustering seems very reasonable for this scenario.
My only objection is that out of the 57 references only 3 correspond to the last three years. Since 2020 there has been no work in this line?
Author Response
Thank you for your positive feedback to the manuscript! Your comments have been very helpful for improving the paper. Below we attempt to respond to the issues highlighted by the reviewer.
Point 1: My only objection is that out of the 57 references only 3 correspond to the last three years. Since 2020 there has been no work in this line?
Response 1: The authors thank the reviewer for noticing this. Part of this imbalance is because research in ad-hoc wireless networking reached peak intensity during the late 90s and early 2000s. Reinforcement learning experienced a breakthrough in 2014 after the publication of:
Mnih, V., Kavukcuoglu, K., Silver, D. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236.
The paper above (which has over 20 thousand citations) presents a general approach to deep reinforcement learning. Subsequent follow-up papers refined and expanded this general approach and are also included in the references. We have found the majority of very recent work in deep RL to be more application specific and less directly relevant to the general problem of tracking interest-points. However, we have added three recent references on swarm robotics in section 5.1 that are somewhat narrower in scope.
Author Response File: 
Author Response.docx
Reviewer 2 Report
The analogy is drawn between the control and communications problems, both present in multi-agent UAV and AUV work, is incisive and extremely well-rooted in the literature. The setout of the argument and the architectural conclusion appear sound. Notwithstanding, for a journal rooted in aerospace research, the argument and certainly the architecture remains distant without a case study, at least in simulation. The paper comes to a somewhat abrupt ending, arguing that the detail is not apropos, in deference to an architecture solution at this time. Yet, that architecture is defined through a discussion of the similarities of the well-setout control problem with the well-setout communication problem. This architecture needs exemplifying and illustration with slightly more than the discussion given, hence the recommended case study, or at a minimum an outline of a structured approach on how to research the proposed architecture (i.e., recommended approach). Which universities are closest, what computing is required, and so forth? Finally, the control problem literature is more recent than the communication literature leading to a slight imbalance that perhaps needs addressing in limitations.
Author Response
Thank you for your positive feedback to the manuscript! Your comments have been very helpful for improving the paper. Below we attempt to respond to the issues highlighted by the reviewer.
Point 1: The analogy is drawn between the control and communications problems, both present in multi-agent UAV and AUV work, is incisive and extremely well-rooted in the literature. The setout of the argument and the architectural conclusion appear sound. Notwithstanding, for a journal rooted in aerospace research, the argument and certainly the architecture remains distant without a case study, at least in simulation.
Response 1: We have added a case-study in section 5.3 that investigates one component of the overall architecture. The purpose of this case-study is to demonstrate the kind of simulation and forensic analysis that makes the architecture more plausible and less distant. However, we acknowledge that an overall proof-of-concept is still a vast undertaking. Our objective with the case-study is to inspire efforts towards that end with a tentative first step.
Point 2: The paper comes to a somewhat abrupt ending, arguing that the detail is not apropos, in deference to an architecture solution at this time. Yet, that architecture is defined through a discussion of the similarities of the well-setout control problem with the well-setout communication problem. This architecture needs exemplifying and illustration with slightly more than the discussion given, hence the recommended case study, or at a minimum an outline of a structured approach on how to research the proposed architecture (i.e., recommended approach). Which universities are closest, what computing is required, and so forth?
Response 2: We have added a structured approach to an architecture in Section 5.2 consisting of five milestones to help bridge the gap towards a proof-of-concept. These milestones are based on our personal experience working in this space and reflect our best insights, but they are not the final word. The case-study in Section 5.3 addresses one part of the first milestone. Each milestone is a problem in its own right, but we have argued that the literature suggests a proof-of-concept is possible.
Point 3: Finally, the control problem literature is more recent than the communication literature leading to a slight imbalance that perhaps needs addressing in limitations.
Response 3: We acknowledge the imbalance in the dating between the communication and control literature. This imbalance can be attributed in part to the wave of interest in wireless networking during the late 90s early 2000s followed by another wave in artificial intelligence after 2010. The paper below is a landmark in reinforcement learning and appears within the last 10 years.
Mnih, V., Kavukcuoglu, K., Silver, D. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236.
We have not explicitly mentioned this imbalance in the paper, but are willing to do so if the reviewer feels it necessary.
Reviewer 3 Report
The article "Strategies for Scalable Communication and Coordination in Multi-Agent (UAV) Systems" provides new control and communication architecture for multi-UAV tracking systems. The topic is actual and fits into the Aerospace scope, and I find it very interesting. Each section of the article reviews part of the overall problem and helps the reader to understand the problematics. The manuscript is well organised, the chapter's length is adequate to its contents, and it's technically sound and readable with ordinary effort.
However, this manuscript needs to be modified as, in its current state, I'm finding it more as a conference contribution than as the Current Contents journal paper. The most relevant chapter – "A Proposed Control/Communication Architecture for Multi-Agent Systems", should be widened and discussed deeper. In addition, the manuscript lacks any simulation/measurement/experiment. The same goes for the conclusion, as the statement "Much work remains to be done" makes this manuscript look like incomplete work rather than the possible solution for the problem depicted in the introduction/abstract. I understand that this is only the start of the long process of solving a complex problem, but the conclusion should give the reader a clear view of what was done and what the authors find innovative or where they succeeded.
The overall quality of pictures should be increased (if the problem with the low resolution was not caused by conversion to the PDF format). There are also two blank pages at the end of the manuscript. Authors should pay attention to reference formatting as there are different styles for conferences, journals and books. Please re-check the formatting and styles.
I recommend accepting and publishing the manuscript after careful authors' revision.
Author Response
Thank you for your positive feedback to the manuscript! Your comments have been very helpful for improving the paper. Below we attempt to respond to the issues highlighted by the reviewer.
Point 1:
The article "Strategies for Scalable Communication and Coordination in Multi-Agent (UAV) Systems" provides new control and communication architecture for multi-UAV tracking systems. The topic is actual and fits into the Aerospace scope, and I find it very interesting. Each section of the article reviews part of the overall problem and helps the reader to understand the problematics. The manuscript is well organised, the chapter's length is adequate to its contents, and it's technically sound and readable with ordinary effort.
However, this manuscript needs to be modified as, in its current state, I'm finding it more as a conference contribution than as the Current Contents journal paper. The most relevant chapter – "A Proposed Control/Communication Architecture for Multi-Agent Systems", should be widened and discussed deeper.
Response 1: We have added section 5.2 which proposes five cumulative milestones towards a proof-of-concept of the architecture and section 5.3 which provides a case-study of the first milestone. While an actual proof-of-concept outside the scope of this paper, we attempt to inspire efforts towards that end by providing a survey of a broad selection of the literature that suggests the proposed milestones are achievable. This paper largely functions as a survey/review of the literature.
Point 2: In addition, the manuscript lacks any simulation/measurement/experiment.
Response 2: The case-study added in Section 5.3 includes simulations of the training process for a single-agent multi-interest-point scenario using q-learning. We identify obstacles/challenges through a forensic analysis of the learning process and its convergence properties. We argue that some of the techniques discussed in previous sections (i.e., hierarchical reinforcement learning) offer a solution to these obstacles. This case-study only applies to the first milestone and is the first step in a much larger endeavor (a proof-of-concept of the whole architecture).
Point 3: The same goes for the conclusion, as the statement "Much work remains to be done" makes this manuscript look like incomplete work rather than the possible solution for the problem depicted in the introduction/abstract. I understand that this is only the start of the long process of solving a complex problem, but the conclusion should give the reader a clear view of what was done and what the authors find innovative or where they succeeded.
Response 3: Following the reviewer’s suggestion, the authors have removed that statement. We have added the following:
“The case-study included in this survey provides some perspective of the work required to develop a proof-of-concept for the overall architecture.”
Point 4: The overall quality of pictures should be increased (if the problem with the low resolution was not caused by conversion to the PDF format).
Response 4: Per the reviewer’s suggestion, we have examined all figures in the paper (using on the PDF version on the MDPI website). All of the figures in the paper use vector graphics, allowing infinite zooming. The only partial exception to this is the background of Figure 1.a. and 4.a., however, upon examination, these do not seem to have an issue up to 800% zoom. We have tried this using Adobe Acrobat Reader, Sumatra PDF, Firefox PDF viewer (built-in to the browser), and Chrome PDF viewer (built-in to the browser). We are happy to further explore this issue with the MDPI editorial staff before final publishing.
Point 5: There are also two blank pages at the end of the manuscript.
Response 5: The authors thank the reviewer for noticing this. The two superfluous pages at the end have been removed.
Point 6: Authors should pay attention to reference formatting as there are different styles for conferences, journals and books. Please re-check the formatting and styles.
Response 7: Following the reviewer’s recommendation, we have gone through all the references to be compliant to the formatting and style standard
Round 2
Reviewer 2 Report
Well done, excellent response and changes!
