Embedded Federated Learning for VANET Environments

Round 1

Reviewer 1 Report

I would feel more confortable with a typical distribution of the article. Meaning that Introduction and related work share the same section, and that results and discussion should address different questions. But since the work seems well done, the article is well written, and probably it is only a matter of my personal taste, I feel that this article should be published. And only if authors find it relevant, they can organize it in a more structured manner, but only if they need it.

Thanks for you effort for making the paper understandable. Good job.

Author Response

I would feel more confortable with a typical distribution of the article. Meaning that Introduction and related work share the same section, and that results and discussion should address dierent questions. But since the work seems well done, the article is well written, and probably it is only a matter of my personal taste, I feel that this article should be published. And only if authors find it relevant, they can organize it in a more structured manner, but only if they need it.

Authors’ Response:

We thank Reviewer #1 for his/her comments that have helped us revise and improve the quality of the manuscript. We appreciate the reviewer’s recognition about the strengths of the paper.
1. Meaning that Introduction and related work share the same section, and that results and discussion should address dierent questions.

Authors’ Response:

We understand the reviewer’s suggestion, but as we all agree that the article is easy to read, we chose not to change its construction. However, we appreciate the suggestion and will consider it in our next articles.

Author Response File: Author Response.pdf

Reviewer 2 Report

The paper needs to improve overall english language. The Citation are poorly discussed. The author has used the comparison of 2 models where LSTM and CNN. Why does other models not compared as the timeseries analysis can be suggested using ARIMA and Prophet algorithms. According to my suggestion the others models should be compared to measure the accuracy among the data.  

Author Response

The paper needs to improve overall english language. The Citation are poorly discussed. The author has used the comparison of 2 models where LSTM and CNN. Why does other models not compared as the timeseries analysis can be suggested using ARIMA and Prophet algorithms. According to my suggestion the others models should be compared to measure the accuracy among the data.
Authors’ Response: We thank Reviewer #2 for his/her comments that have helped us revise and improve the quality of the manuscript.
1. The paper needs to improve overall english language.
Authors’ Response: Thanks to the reviewer for the alert. Regarding the English writing, the revised version of the manuscript has been carefully and deeply revised following all the reviewers’ comments.
2. The Citation are poorly discussed.
Authors’ Response: We revised the Related Work Section and included more details on the works that we believe to be the most relevant. The additions and changes made are highlighted in blue.
3. The author has used the comparison of 2 models where LSTM and CNN. Why does other models not compared as the timeseries analysis can be suggested using ARIMA and Prophet algorithms. According to my suggestion the others models should be compared to measure the accuracy among the data.
Authors’ Response: We appreciate the reviewer’s suggestion, but the main goal of our work was to develop a framework that makes federated learning available quickly and simply. Our framework can work with the most used machine learning models such as Recurrent Neural Networks (RNN), Neural Network (NN), Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU), both for use in edge devices and centralized in the datacenter/cloud. We also provide the main methods/algorithms for model aggregation in the federated option. To speed up testing, we chose to use literature references on use cases related to traffic/vehicle datasets. Thus, as highlighted in Section V, we support our choice of CNN/LSTM in works such as [34] and [35]. In particular, the Table 2 in Benidis et. al [35] presents a very illustrative summary where it is possible to notice that for the data types related to traffic, most of the solutions highlighted in the table indicated better results associated with CNN/LSTM. Based on these studies, we chose to discuss the performance of our solution only with CNN and LSTM, with which the best results are obtained for our use cases and datasets. We reinforce this motivation also at the beginning of Section 5.

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper provides interesting overview of several approaches to the decentralized ML. It contains good experimental results and comparative analysis both from the algorithmic and hardware points of view. It also provides good overview of the current approaches and technological capabilities. I have two suggestions, that could slightly improve the paper:

1. Your results seems to be a little bit out of the context. You provide the comparative analysis of several systems but is it the first such performance evaluation? Maybe there were other studies with different test data? Are there any dependency on the problem type - it works good with CNN, but what about other models and problems, etc. Last but not least, you confirm the efficiency of the approach but what are the next steps? Does this mean that one can start using your approach in practice? Or if one would like to improve the framework, what would the possible direction be - to wait for better hardware?

2.  It would be interesting to see the analysis of the time vs number of clients. What would be the dependency if you continue increasing the number of clients?  

Author Response

The paper provides interesting overview of several approaches to the decentralized ML. It contains good experimental results and comparative analysis both from the algorithmic and hardware points of view. It also provides good overview of the current approaches and technological capabilities. I have two suggestions, that could slightly improve the paper: 1. Your results seems to be a little bit out of the context. You provide the comparative analysis of several systems but is it the first such performance evaluation? Maybe there were other studies with different test data? Are there any dependency on the problem type - it works good with CNN, but what about other models and problems, etc. Last but not least, you confirm the effciency of the approach but what are the next steps? Does this mean that one can start using your approach in practice? Or if one would like to improve the framework, what would the possible direction be - to wait for better hardware? 2. It would be interesting to see the analysis of the time vs number of clients. What would be the dependency if you continue increasing the number of clients?
Authors’ Response: We thank Reviewer #3 for his/her comments that have helped us revise and improve the quality of the manuscript. We appreciate the reviewer’s recognition about the strengths of the paper.
1.a Your results seems to be a little bit out of the context. You provide the comparative analysis of several systems but is it the first such performance evaluation? Maybe there were other studies with different test data?
Authors’ Response: In the performance evaluations, we always aimed to analyze each step of the federated process, that is, the cost of training on edge devices, sending the models for aggregation in the cloud/datacenter, the cost of aggregating the edge, distributing the aggregated models to edge devices, and training with local data. We understand that each step is important and should be considered by administrators when choosing between the distributed/federated option and the centralized option. Thus, we show the overall performance of the federated option, summarized in Table 3, and then evaluate the individual performance of each of the federated steps. In our Related Works Section, we highlight the most recent works that involve machine learning and federated learning solution deployment in real devices [9-22]. The differential aspects of our work is that we really evaluate all the steps in a real environment, and compare different equipments as edge devices, including the evaluation of the devices (overhead and RAM/CPU consumption). To clarify this point, we extended the opening paragraph of Section 5.
1.b Are there any dependency on the problem type - it works good with CNN, but what about other models and problems, etc.
Authors’ Response: We appreciate the reviewer’s suggestion, but the main goal of our work was to develop a framework that makes federated learning available quickly and simply. Our framework can work with the most used machine learning models such as Recurrent Neural Networks (RNN), Neural Network (NN), Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU), both for use in edge devices and centralized in the datacenter/cloud. We also provide the main methods/algorithms for model aggregation in the federated option. To speed up testing, we chose to use literature references on use cases related to traffic/vehicle datasets. Thus, as highlighted in Section V, we support our choice of CNN/LSTM in works such as [34] and [35]. In particular, the Table 2 in Benidis et. al [35] presents a very illustrative summary where it is possible to notice that for the data types related to traffic, most of the solutions highlighted in the table indicated better results associated with CNN/LSTM. Based on these studies, we chose to discuss the performance of our solution only with CNN and LSTM, with which the best results are obtained for our use cases and datasets. We reinforce this motivation also at the beginning of Section 5.
1.c Last but not least, you confirm the effciency of the approach but what are the next steps? Does this mean that one can start using your approach in practice? Or if one would like to improve the framework, what would the possible direction be - to wait for better hardware?
Authors’ Response: FedFramework is designed to be the federated learning tool for the city use cases (Section 4.2). The deployment of FedFramework to the project’s edge devices is ongoing. Therefore, it is already being used in practice. We believe that the evaluation we discussed in the article can show alternatives to simply expanding hardware capabilities. In our view, the applications that consume the service offered by FedFramework will dictate the paths. But by allowing a detailed assessment of resource consumption, it is possible to scale the solution. Because it is a composition of services, it is possible to distribute them in different pieces of equipment that would be the “edge device”, in addition to facilitating strategies based on Fog Computing. Therefore, the answer to the question is ’Yes’, FedFramework can indeed start to be used in practice. Although this discussion is very interesting, we decided not to include it in the article.
2. It would be interesting to see the analysis of the time vs number of clients. What would be the dependency if you continue increasing the number of clients?
Authors’ Response: We think we have made that assessment. In all our tests, we progressively increased the number of clients, reaching a maximum of 70, which was supported by the hardware we used, the Jetsons. We varied clients both in relation to the number of containers on each device, and also the total number of clients participating in the aggregation cycles (FL rounds), and we gathered the time it took for 1 round and the total time. Table 3 presents an overview of clients vs devices vs rounds and the time required.

Author Response File: Author Response.pdf