Federated Deep Reinforcement Learning for Energy-Efficient Edge Computing Offloading and Resource Allocation in Industrial Internet

Round 1

Reviewer 1 Report

1. The study presented in the paper focuses on optimizing the power consumption of peripheral computing devices. It is known that уfficient power consumption of a computing device is inextricably related with the degree of its load. However, the power consumption model for peripheral computing devices is poorly considered in the paper. In particular, different modes of device power consumption such as the device is turned off, the device is on and in waiting state, the device is used for executing tasks and has a low computing load, the device is used for executing tasks and has a high computing load, etc. are not studied. Moreover, different operations performed by devices cause different power consumption. Consideration of the aforementioned aspects of energy consumption would be desirable in the paper. In addition, please explain in detail the meaning of the CPU’s effective capacitance parameter ku. From the text of the paper, it seems that ku is a constant.

2. Please provide graphs of changes in the efficiency of using the resources of computing devices and edge servers with an increase in the number of the servers for the compared algorithms.

3. Often, not always justified and explained use of superscripts of variables makes it difficult to read the text of the paper. If possible, please remove the superscripts of variables that do not carry significant meaning.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The paper presents an innovative approach for optimizing energy consumption in Industrial Internet mobile edge computing systems by combining federated learning and deep reinforcement learning.

However, there are several minor issues with the manuscript that need to be addressed to improve its quality, which are noted below.

 1.     Line 39-40 – “This challenge leads to communication overhead such as energy and delay”.  Energy itself is not a communication problem. Instead, it is a challenge or concern related to mobile edge computing systems. Maybe you meant energy consumption but again it’s not communication overhead. Rewrite to bring clarity.

2.     Line 63 – The mention of “You” is misleading, although he is the first author. May be use You et.al or The authors.

3.     Line 81 – “Li et al. [23] proposed to use of the DRL algorithm “. Grammatical error in the sentence.

4.     Line 107-108 – “However, there is insufficient of related research on multi-edge-terminal collaboration” – Check syntax error “of” not required.

5.     Line 120 -121 – Which system are we pointing here? MEC ? The context is unclear.

6.     Line 124 – Provide suitable reference to DDPG and FL.

7.     The Figure 1 provides a clear representation of the system model and the interactions between different components. However, to improve readability and avoid repetitive labeling, it would be beneficial to include a common strip or legend in the device layer that provides a clear description or symbol for the "Mobile industrial intelligent terminal devices." This would enhance the clarity and aesthetics of the figure by reducing redundant labeling while still conveying the necessary information.

8.     Line 168-169 – “Firstly, the tasks can be computed partly of fully on the terminal device locally.” – Grammatical error again 

9.     Line 201-202 – Missing clarity. “The device utilizes multi-core computing for tasks simultaneously, where the regulable core k ∈{1, 2, ..., K}.” Did you mean to process tasks?  Also, what regulable means here? 

10.  Line 221 – “The delay can be expressed ass” – Typos

11.  Line 257 – 260 – Quite lengthy sentence and lacks clarity.

12.  Line 268 – Recommend adding citation for MDP.

13.  Line 333- – Recommend adding citation for DQN.

14.  Line 432 – Algorithm number is missing.

15.  Line 483, 449, 493-494 – Mention the scheme name also as in the figures there is no detail of β & α to look at.

16.  Line 506 – Mention the figure number.

17.  Line 541 – “Because the IF-DDPG is an improved version of the DDPG.” – Seems incomplete

The authors must address the comments and resolve few issues in order to improve the quality of their paper before accepting it for publication.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This paper presented federated deep reinforcement learning to overcome energy efficiency and resource allocation using edge computing in industrial internet. The article is relevant and interesting, but in overall aspects it lacks in different aspects.

1.      Authors states that “the terminal device has the option to either execute the task locally or offload it partially or entirely to the edge server”, this need to be clarified because the working of federated learning (where all the training is performed locally at the device end) is different.

2.      The motivation of this paper is not clear. Deep reinforcement learning (DRL) is used for sequential decision making and control problems. What is the motivation for using the DRL with federated learning?

3.      The authors have not included a review of the existing literature, which is necessary to provide context and highlight the issues and problems present in the current body of work.

4.      It is crucial for the authors to address this gap by incorporating a comprehensive review of relevant literature. This will enable readers to better understand the research landscape, identify the challenges that exist, and appreciate the novelty and contributions of the authors' work. Authors are required to cite some of the recent research articles such as

a.       S. Yu, X. Chen, Z. Zhou, X. Gong and D. Wu, "When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multitimescale Resource Management for Multiaccess Edge Computing in 5G Ultradense Network," in IEEE Internet of Things Journal, vol. 8, no. 4, pp. 2238-2251, 15 Feb.15, 2021, doi: 10.1109/JIOT.2020.3026589.

b.      H. Wang, Z. Kaplan, D. Niu and B. Li, "Optimizing Federated Learning on Non-IID Data with Reinforcement Learning," IEEE INFOCOM 2020 - IEEE Conference on Computer Communications, Toronto, ON, Canada, 2020, pp. 1698-1707.

c.       Yaqoob, M.M.; Nazir, M.; Khan, M.A.; Qureshi, S.; Al-Rasheed, A. Hybrid Classifier-Based Federated Learning in Health Service Providers for Cardiovascular Disease Prediction. Appl. Sci. 2023, 13, 1911.

d.      Khodadadian, S., Sharma, P., Joshi, G. and Maguluri, S.T., 2022, June. Federated reinforcement learning: Linear speedup under markovian sampling. In International Conference on Machine Learning (pp. 10997-11057). PMLR.

e.       Z. Tianqing, W. Zhou, D. Ye, Z. Cheng and J. Li, "Resource Allocation in IoT Edge Computing via Concurrent Federated Reinforcement Learning," in IEEE Internet of Things Journal, vol. 9, no. 2, pp. 1414-1426, 15 Jan.15, 2022, doi: 10.1109/JIOT.2021.3086910.

5.      Why did authors use federated averaging at the cloud end? To reduce communication costs and optimize the convergence speed of the algorithm.

6.      Authors are required to compare their performance with this paper “Yaqoob, M.M.; Nazir, M.; Yousafzai, A.; Khan, M.A.; Shaikh, A.A.; Algarni, A.D.; Elmannai, H. Modified Artificial Bee Colony Based Feature Optimized Federated Learning for Heart Disease Diagnosis in Healthcare. Appl. Sci. 2022, 12, 12080.”

7.      Authors are advised to present a summarized table about five schemes/settings along with the hyperparameters of that scheme.

8.      The reviewer is interested in knowing the number of clients considered by the authors and how this could potentially impact the performance of their proposed method. It is crucial to provide information regarding the number of clients involved in the study and to discuss the potential effects that varying the number of clients may have on the performance outcomes of the proposed method.

9.      How “State Space” and “Action Space” affects the performance of their proposed method if the reward function is wrongly computed?

10.  Authors did not provide any information about the dataset used to train, test, and validate the performance of their proposed method. A subsection is required to be added to the article about the dataset used.

11.  In figures 5 and 7, does the energy refer to the energy of the terminal nodes or the edge server nodes?

12.  Authors are required to comment on the energy model used for the terminal nodes. For energy model please refer to the following articles and include in your literature

a.       Jiang, C., Fan, T., Gao, H., Shi, W., Liu, L., Cérin, C. and Wan, J., 2020. Energy aware edge computing: A survey. Computer Communications, 151, pp.556-580.

b.      M. Mateen Yaqoob, W. Khurshid, L. Liu, S. Zulqarnain Arif, I. Ali Khan et al., "Adaptive multi-cost routing protocol to enhance lifetime for wireless body area network," Computers, Materials & Continua, vol. 72, no.1, pp. 1089–1103, 2022.

13.  All sections should have a proper introduction before subsections start (e.g., Section 2 starts right away).

14.  The current writing style of this article assumes that the readers are already well-versed in the specific topic and familiar with the proposed method, which is unlikely to be the case for most, if not all, readers. As a result, it is challenging to discern the actual novelty put forth by the authors. It is imperative to provide a clear and concise description of the proposed method to make it accessible and comprehensible to a broader audience.

15.  The entire paper's English should be double-checked and improved.

Minor english edits are required. Some long sentences need to be concised. It is also recommended to used Grammarly business edition.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

My previous concerns are addressed in well manner. The article is accepted from my side and I have no objection to process this article further.