Áika: A Distributed Edge System for AI Inference

Round 1

Reviewer 1 Report

The subject of this paper is exciting and valuable for readers. From a practical perspective, the approach is understandable, but from a scientifical point of view, we have some remarks to make:

First of all, we observe some issues concerning the methodological approach. The organisation of research is unclear; even though the approach is oriented toward experimentation is necessary to include a model of research with the explanation of specific scenarios for each experiment. It is not very clear the logic of each experiment and their motivation to justify the inclusion in the paper. In plus, it is not clear if the experimentations must prove something and if these results are or are not significant for demonstrating the goal of this research. In this context, we ask from authors to include a model of research with a clear explanation of the motivation for the inclusion of each experiment. For each experiment, the authors must explain the operating parameters, the scenario, the results, and the utility of these results for the research goal.

In Abstract is not illustrated the originality of this research.
In the Introduction is included too more details concerning the Aika system. We recommend including the objective of this research and the structure of this paper in the last paragraph. Please, exclude from the last paragraph the contributions. The contributions must be the subject of discussions or conclusions. 
In the Background and Motivation, we identified only motivation. Even in this context, we have not identified scientific references in this motivation. Thus, we can see personal ideas concerning motivation. It is necessary to include references to other scientifical studies.  
In the System Overview, we identified the architecture of the system with a description. 
In section 4, it is not understandable why this structure and why it is necessary par. 4.5. Distributed Word Counter. We ask from authors to explain/motivate this structure and the utility of par. 4.5. 
In the title of some figures, there is an explanation for the content of this. We consider that it is more appropriate to do this explanation under each Figure as a comment. Please, carefully analyze Figure 15 and the explanation of this because there seems to be something missing in this Figure in correlation with the explanation from the title.
We think that the section 'Related work' must be included in the introduction or in a special section as part of the literature review. It is necessary to extend the Conclusion because these are superficially presented here.
  

Author Response

Dear reviewer,
Thank you for your valuable feedback regarding our manuscript submission. We have taken your points to heart and have worked to meet your concerns. What follows is a response to your suggestions with a description of the steps we have taken to improve our manuscript. 

 - We observe some issues concerning the methodological approach. The organisation of research is unclear; even though the approach is oriented toward experimentation is necessary to include a model of research with the explanation of specific scenarios for each experiment. It is not very clear the logic of each experiment and their motivation to justify the inclusion in the paper. In plus, it is not clear if the experimentations must prove something and if these results are or are not significant for demonstrating the goal of this research. In this context, we ask from authors to include a model of research with a clear explanation of the motivation for the inclusion of each experiment. For each experiment, the authors must explain the operating parameters, the scenario, the results, and the utility of these results for the research goal

It has become clear to us that the intention and purpose behind our experiments should be communicated better. Furthermore, the paper and its introductory sections did not clearly state the research direction that resulted in the design and implementation of our system. To improve this, we have rewritten parts of the background section to more clearly state the motivation and scenarios that has driven the research direction to implement this system. Additionally, the explanations and reasoning behind our experiments have also been extended and improved at several points.

- In Abstract is not illustrated the originality of this research.

It has become apparent that the Abstract of this article did not sufficiently describe the novelty of our research, and was vague in describing the overall system. To improve this, we have rewritten most of the abstract, to include the most important details of the research and the system architecture, and to summarize the novelty of the paper.

- In the Introduction is included too more details concerning the Aika system. We recommend including the objective of this research and the structure of this paper in the last paragraph. Please, exclude from the last paragraph the contributions. The contributions must be the subject of discussions or conclusions.  In the Background and Motivation, we identified only motivation. Even in this context, we have not identified scientific references in this motivation. Thus, we can see personal ideas concerning motivation. It is necessary to include references to other scientifical studies.  

We agree that the overall structure of the first sections of the paper was not adequate to introduce and present our research. To improve this, we have reworked the Introduction and Background and Motivation sections (which is now renamed to Background) to be more consise and better structured. The bullet point list of contributions has been replaced by an explanation of the structure of the paper and the research it presents. The Background section has been extended with more references to existing work that motivates the direction and requirements of our system.

- In section 4, it is not understandable why this structure and why it is necessary par. 4.5. Distributed Word Counter. We ask from authors to explain/motivate this structure and the utility of par. 4.5. 

We realize that the value of our Distributed Word Counter experiment was not adequately explained in our Experiments section. The intended utility of this experiment, that it represents a more generalized workload that can be expressed using a graph computation model, has been more clearly stated in the paper.

- Please, carefully analyze Figure 15 and the explanation of this because there seems to be something missing in this Figure in correlation with the explanation from the title.

The description of Figure 15 mistakenly did not explain every metric shown in the graph. We have modified it to properly describe that the numbers indicated on the data points are the standard deviation of each measurement. This is also shown by error bars in the plot, but the error bars are small at many points.

- We think that the section 'Related work' must be included in the introduction or in a special section as part of the literature review

We agree, and after reviewer feedback we have merged the Related Work section into the revised Background section, which comes after the Introduction section in the beginning of the paper. The Related Work section that was previously found towards the end of the paper has been removed in the revised manuscript.

- It is necessary to extend the Conclusion because these are superficially presented here.

It has become clear to us that the Conclusion section was not adequate to conclude the paper and describe our work and results. We have extended this section with more information about the system design and the results of our experiments.

Overall, we have revised our manuscript to improve language, clarity, and to show the relevance between our work, our application domain, and the presented background work. In addition to the changes outlined above, we have made adjustments to the grammar and tone of some sections of the paper in accordance to reviewer feedback. 

We hope you find this revised version of the manuscript to have addressed some of the concerns with the initial submission.

Reviewer 2 Report

The paper describes an agent based system architecture. 

It is well structured and good to read. Given the trend in edge computing the work is highly relevant. 

A few points where improvements should be made: 

Section Distributed Deep Feature Extraction is unclear: with the other performance evaluations the number of agents is used on the x-axis. Why not here. Its also not clear what a sub-graph is. Please be more detailed here.

In the same section, the authors are surprised "sequential feature extraction approach performs better compared to the single machine benchmark". From the text in this section I assumed multiple agents (N) share the work. Where each agents here do the three classifications. But still multiple agents are used. But I might be misguided by the description. 

More details on the implementation (language, size of library, memory needs, ...) would be useful. 

Given a distributed system I am missing performance checks with different connectivity. More details on the serialization of the data would be helpful. This aspect can put some heavy load on small computing devices. 

With respect to related work I am missing a discussion of the akka (Java, SCALA) and akka.net (C#) platforms. 

Author Response

Dear reviewer,
Thank you for your valuable feedback regarding our manuscript submission. We have taken your points to heart and have worked answer your questions. What follows is a response to your suggestions with a description of the steps we have taken to improve our manuscript. 

 - Section Distributed Deep Feature Extraction is unclear: with the other performance evaluations the number of agents is used on the x-axis. Why not here. Its also not clear what a sub-graph is. Please be more detailed here.

It has become clear to us from reviewer feedback that the description of the metrics in this experiment was not sufficient, and that an explanation of sub-graphs is missing. In short, a sub-graph is a smaller component of the computation DAG that is split off so that it can be executed in parallel with other sub-graphs. We have extended this experiment evaluation with a better description of this.

- In the same section, the authors are surprised "sequential feature extraction approach performs better compared to the single machine benchmark". From the text in this section I assumed multiple agents (N) share the work. Where each agents here do the three classifications. But still multiple agents are used. But I might be misguided by the description. 

At this data point (sequential feature extraction with one sub graph), the sequential feature extraction is essentially single-threaded, as multiple sub-graphs is needed to effectively share the work. The explanation of this has been expanded in the paper.

- Given a distributed system I am missing performance checks with different connectivity. More details on the serialization of the data would be helpful. This aspect can put some heavy load on small computing devices. 

We agree that connectivity performance evaluations would be interesting, particularily so because of the targeted domain of this system. However, we were unfortunately not able to perform additional experiments for this revision.

- With respect to related work I am missing a discussion of the akka (Java, SCALA) and akka.net (C#) platforms. 

We have unfortunately not been able to add a discussion of this particular system, but in order to expand our background and related work section for this revision of the manuscript, other systems has been added.

Overall, we have revised our manuscript to improve language, clarity, and to show the relevance between our work, our application domain, and the presented background work.

We hope you find this revised version of the manuscript satisfactory and that you found your questions adequately answered.

Round 2

Reviewer 1 Report

In this version, the paper is presented in an appropriate form reporting us to the experimental approach used as a research method.

I analyzed all the explanations on the authors' part and I appreciate positive the current form of the paper.