Review Reports - Virtual Machine Placement in Edge Computing Based on Multi-Objective Reinforcement Learning

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

In brief: interesting article, but some statements need justification or need to be removed

The advise is therefore: Accept after minor but mandatory revisions

The article proposes an algorithm called EVMPRL for VM placement in edge computing;

it gives a decent description; and gives a comparison with L-EVMPRL and MRLVMP in simulations.

It is shown that EVMPRL is behaving correctly; the simulation shows comparable results with respect to L-EVMPRL and MRLVMP, but ‘out performing’ seems to be not justified as a conclusion

With the popularization of internet of things (IoT),

Energy consumption of mobile edge computing (MEC) servers is growing rapidly.
The high real-time requirements of IoT make it imperative for edge computing to reduce the response latency of tasks.
Virtual machine (VM) placement can reduce the response latency of VM requests and the energy consumption of MEC servers effectively.

The three bullet points may be true in some cases, but the generalisation here is not justified: not all IoT is real-time, in general energy consumption of MEC server is not growing rapidly, VM by itself is creating degrees of freedom at a cost (the degrees of freedom may be used to optimise one or multiple aspects, but to suggest that all can be optimised at the same time may be a few bridges too far.

The core of the article is appreciated, but the ‘packaging should refrain from claims and statements that are not justified by the research and the text:

Accept after minor but mandatory revisions

Author Response

1.‘Out performing’ seems to be not justified as a conclusion.

Authors’ response:

Thanks for your comment.

The last sentence of the original version of the conclusion Section was, “Through experiments, this paper demonstrates that EVMPRL outperforms an existing algorithm and L-EVMPRL based on the linear scalarization function.”

In the revised version, we rewrite this sentence. “The experimental results demonstrate that EVMPRL is more capable of achieving the trade-off between the response latency and energy consumption while obtaining higher-quality solution sets compared to MRLVMP and L-EVMPRL. The results also demonstrate EVMPRL's convergence and scalability.”

Please review the above explanations and the modifications in Section 7. Thank you.

2.Not all IoT is real-time.

Authors’ response:

Thanks for your comment.

The original version was overly generalized.

In the revised version, we highlight some important latency sensitive/real-time IoT applications, such as autonomous driving, smart manufacturing, and smart wearables.

In the second paragraph of Section 1, we use the expression of “real-time IoT applications”.

Please review the above explanations and the modifications in the Abstract, the first two paragraphs of Section 1, and the first paragraph of Subsection 2.2. Thank you.

3.In general energy consumption of MEC server is not growing rapidly.

Authors’ response:

Thanks for your comment.

We change the expression of “is growing rapidly” to “is also on the rise”

Please review the above explanations and the modifications in the Abstract, and the first paragraph of Section 1. Thank you.

4.VM by itself is creating degrees of freedom at a cost (the degrees of freedom may be used to optimise one or multiple aspects, but to suggest that all can be optimised at the same time may be a few bridges too far.

Authors’ response:

Thanks for your comment.

The original version was overly generalised.

Optimized at the same time or simultaneously is inappropriate.

In the revised version, we use “tradeoff” instead.

“Our aim is to find the Pareto approximate solution set that achieves the tradeoff of response latency of VM requests and energy consumption of MEC servers.”

“The goal of EVMPRL is to find a Pareto approximate solution set that can achieve the tradeoff of reducing the response latency of VM requests and the energy consumption of MEC servers.”

These modifications are in the Abstract, the fourth paragraph of Section 1, Section 2, and Section 7. Thank you.

In the revised version, we change “VM placement can optimize various metrics” to “VM placement can optimize many important metrics”.

These modifications are in the second paragraph of Section 1 and the first paragraph of Section 2. Thank you.

(3) In the revised version, we have restructured the section on related work and provided a more detailed description of it. Moreover, we provide a table of related work to summarize and compare the related work.

In Table 1, we summarize and compare the related work in the environmental, framework, and optimization objectives aspects. Energy consumption, resource wastage, latency, and SLA violation rate have been discussed in Section 2.

This paper focuses on energy consumption and response latency, which we have explained in the Introduction and Subsection 2.2 in the revised version.

Please review the above explanations and the modifications in Section 1 and Section 2. Thank you.

We really appreciate these valuable comments and suggestions by the reviewer.

Author Response File: Author Response.docx

Reviewer 2 Report

Comments and Suggestions for Authors

The article investigates an interesting topic and aims to provide evidence from a context where the topic is of relevance: article proposes an algorithm called EVMPRL for virtual machine placement in edge computing based on reinforcement learning

The need for this investigation is extended in the introduction section and the gap in the literature is clear. Yet, the authors should try to enhance the contribution of their work in the introduction section. This will add clarity to the originality of their work and will set the ground for the study carried.

The literature review section is well elaborated and sets the ground for the investigation.

The methods employed are well described and elaborated by the researcher. The rationale is clear, and the measurement instrument development is also clear.

We recommend that the source be specified for each figure.

The results are clearly explained by the authors.

The discussion and conclusion section highlight once again the importance of this study’s findings. It enhances the contribution both to literature and to the practical context.

Comments on the Quality of English Language

Overall, the article is well-written. Yet, authors should pay attention to few syntax errors and possible typos found across the manuscript. There is a need to correct grammatical errors and to convey information in an easy understandable and explicit way. This will enhance the readability of their work

Author Response

1.We recommend that the source be specified for each figure.

Authors’ response:

Thanks for your comment.

In the original version, each figure and the content that introduces it are too far apart. It was less reader-friendly.

In the revised version, we place each figure near its relevant content.

In the revised version:

(1)Figure 1 is between lines 187 and 188.

The relevant text is in line 188.

(2) Figure 2 is between lines 389 and 390.

The relevant text is in line 393.

(3) Figure 3 is between lines 403 and 404.

The relevant text is in line 405.

(4)Figure 4 is between lines 412 and 413.

The relevant text is in line 413.

(5) Figure 5 is between lines 446 and 447.

The relevant text is in line 447.

(6) Figure 6 is between lines 484 and 485.

The relevant text is in line 480.

(7) Figure 7 is between lines 491 and 492.

The relevant text is in line 488.

(8) Figure 8 is between lines 495 and 496.

The relevant text is in line 494.

Please review the above explanations and the modifications in each figure and its relevant content. Thank you.

2.Comments on the Quality of English Language：

Authors’ response:

Thanks for your comment.

We have checked the spelling and grammar throughout the manuscript and made corrections accordingly. The typos and grammatical errors, as suggested by the reviewers, have been corrected.

Please review the above explanations. The modifications have been marked in yellow. Thank you.

We really appreciate these valuable comments and suggestions by the reviewer.

Author Response File: Author Response.docx

Reviewer 3 Report

Comments and Suggestions for Authors

This is the review of the manuscript titled: "Virtual machine placement in edge computing based on multi-objective reinforcement learning". The paper has some contributions, however, requires significant improvement in writing. I have a few comments that should be addressed to improve the quality of the manuscript and consider for publication.

The authors did not develop a good motivation for the proposed work in introduction section. It is suggested to properly highlight the problem, and gaps in the existing research.

For instance:

What is the motivation of using RL over other optimization techniques.

The novelty and contribution in using Chebyshev scalarization function is not clear.

There is no information about the data or simulation setup used for training the reinforcement learning model? Is it representative of real-world MEC environments?

To show the significance of improvement, it is suggested to provide quantitative results demonstrating the superiority of proposed algorithms over existing ones.

What are the limitations of this work? Did the authors test any specific scenarios or edge cases where EVMPRL might not perform as well as other approaches?

Also comment about the scalability of the proposed work. How does it scale with the increasing number of IoT devices or servers? What about heterogeneous environment, where the servers have different resource capacities of energy efficiency levels?

Minor comments:

There are many writing issues. The sentences are not formed properly, which will cause difficulty for the readers. (Lines 53-55)

Throughout the paper, avoid using "[18] proposes/investigated..". The references should be listed in a way that is classified or clearly stated with the purpose. Summarization is needed in terms of popular methods, distinctions and similarities between various applications.

The term "order-of-magnitude" is not used properly in the manuscript. For instance Line 124 can be written as "the energy consumption is order-of-magnitude higher than..."

Line 133, a new paragraph is not needed.

There are several typos and grammatical errors throughout the manuscript. To name a few:

Line 25. 2.46 billion

Line 39

Line 50

Line 135

Rewrite (Lines 83-86)

Lines 160-161

Line 236

"SLA" is not defined

Consider rearranging the figures. Place them near the text they are referred in.

Comments on the Quality of English Language

My comments about the quality of English are included in the general comments.

Author Response

1.What is the motivation of using RL over other optimization techniques.

Authors’ response:

Thanks for your comment.

(1)“The VM placement problem belongs to the discrete decision-making problem, which is suitable for solving by RL [19].”

The above explanation is added to the fourth paragraph of Section 1. In addition, we cite the literature [19] to support this explanation.

[19]Gábor, Z.; Kalmár, Z.; Szepesvári, C. Multi-criteria reinforcement learning. In Proceedings of the ICML, 1998, Vol. 98, pp. 197–205.

The previous work[42] also demonstrates the applicability of RL to discrete decision-making problems by citing [19].

Thank you.

(2)Our previous work has shown, through experiments, that using the Chebyshev scalarization function in RL algorithms can solve the weighting coefficient selection problem.

Although other algorithms, such as the ant colony algorithm and the evolutionary algorithm, are also capable of solving discrete decision problems, there is no sufficient evidence to prove that the weighting coefficient selection problem can be solved when using the Chebyshev scalarization function in these algorithms.

We have cited our previous work in the revised version to support the above property when using the Chebyshev scalarization function in RL. The previous work cited in the revised version is as follows:

[11]Qin, Y.; Wang, H.; Yi, S.; Li, X.; Zhai, L. A multi-objective reinforcement learning algorithm for deadline constrained scientific workflow scheduling in clouds. Frontiers of Computer Science 2021, 15, 1–12.

[12]Yi, S.; Li, X.; Wang, H.; Qin, Y.; Yan, J. Energy-aware disaster backup among cloud datacenters using multiobjective reinforcement learning in software defined network. Concurrency and Computation: Practice and Experience 2022, 34, e6588.

Please review the above explanations and the modifications in Section 1. Thank you.

2.The novelty and contribution in using Chebyshev scalarization function is not clear.

Authors’ response:

Thanks for your comment.

(1)Contributions

①Our previous work has shown that multi-objective reinforcement learning (RL) algorithms utilizing the Chebyshev scalarization function can effectively select weighting coefficients [11,12].

②The Chebyshev scalarization function not only overcomes the problems of the linear scalarization function but also searches for solutions in a bigger space rather than in convex regions. The method finds Pareto-optimal solutions in various Pareto frontier shapes [13]. Thus, it can find the high-quality solution set under most combinations of weighting coefficients.

For the limited space, we just give a brief introduction of Chebyshev scalarization function in the first two paragraphs in Section 5.2.2. Interested readers can obtain more detailed information through the above references, especially [13].

(2)Novelty

The problem of selecting weighting coefficients for the VM placement problem in edge computing environments has not been considered. The Chebyshev scalarization function is well-suited to address this problem. So, we incorporate innovations from our previous work.

The novelty and contributions are marked in yellow.

Please review the above explanations and the modifications in the third paragraph of Section 1, and Section 5.2.2. Thank you.

3.There is no information about the data or simulation setup used for training the reinforcement learning model? Is it representative of real-world MEC environments?

Authors’ response:

Thanks for your comment.

The data or the experimental environment in this paper is based on related work, such as [16],[24],[33],[35], etc. The server type and VM requests in this paper are common in the references of VM placement.

Three types of MEC servers, HP Proliant ML110 G5, HP Proliant DL360 G7, and HP Proliant DL360 Gen9, are common in real-world server rooms and datacenters.

In this paper, VM requests are randomly generated according to the methodology in the above related work.

From the aspect of dataset or the experimental environment, this paper can reflect the heterogeneity of real-world MEC servers. As in previous work[16,33,35], we randomly generate VM requests to simulate the diversity of demand data in reality. We used randomly generated data because of the need to study the problem on different scales. Therefore, the settings in this paper can reflect real environments well.

Please review the above explanations and the modifications in Subsection 6.1. Thank you.

4.To show the significance of improvement, it is suggested to provide quantitative results demonstrating the superiority of proposed algorithms over existing ones.

Authors’ response:

Thanks for your comment.

Because the superiority degree is related to the number of MEC servers and the number of VM requests, it differs significantly in different experimental environments.

Let us take the third paragraph in Subsection 6.4 as an example.

“Figure 4(a) depicts the case when the MEC server number is 80. When the VM request number is 100, the total response latency calculated by EVMPRL is 1.48% and 5.51% lower than that of L-EVMPRL and MRLVMP, respectively; when the VM request number rises to 700, the total response latency calculated by EVMPRL is 4.54% and 10.31% lower than that of L-EVMPRL and MRLVMP, respectively.”

This paragraph and Figure 4(a) describe the comparison of the algorithms in terms of response latency. We can see that the superiority degree of EVMPRL varies with the number of MEC servers and the number of VM requests.

5.What are the limitations of this work? Did the authors test any specific scenarios or edge cases where EVMPRL might not perform as well as other approaches?

Authors’ response:

Thanks for your comment.

In the revised version, we have restructured the section on related work and provided a more detailed description of it. Moreover, we provide a table of related work to summarize and compare the related work. In Table 1, we summarize and compare the related work in the environmental, framework, and optimization objectives aspects.

This paper focuses on energy consumption and response latency, which we have explained in the Introduction and Subsection 2.2 in the revised version.

However, the problem in this paper also involves resource utilization. This will be further investigated in future work. We refer to the issue in the section of conclusion and will address it in future work.

To illustrate the limitations and directions for improvement, we have added a paragraph at the end of the conclusion section.

“This paper only addresses the problem in a static environment and does not consider metrics such as resource utilization. In future work, we will investigate VM placement problems in turbulent changing environments, and will consider as many metrics as possible that might be involved in that environment.”

Please review the above explanations and the modifications in Section 1, Subsection 2.2, and Section 7. Thank you.

6.Also comment about the scalability of the proposed work. How does it scale with the increasing number of IoT devices or servers? What about heterogeneous environment, where the servers have different resource capacities of energy efficiency levels?

(1) Also comment about the scalability of the proposed work. How does it scale with the increasing number of IoT devices or servers?

Authors’ response:

Thanks for your comment.

To demonstrate the scalability of EVMPRL, we have added the subsection of scalability analysis in the revised version. We verify the scalability of EVMPRL based on the methodology in [21].

Although the running time increases with the larger VM request scale, EVMPRL still can obtain good convergence because of the following effective measures:

①Based on the reward functions for response latency and energy consumption, EVMPRL prefers servers that produce the least possible response latency and energy consumption for the current VM request. At the cost of narrowing search space, response latency and energy consumption can approximate to relatively optimal values.

②In action selection, the scalarized strategy can remove low-quality solutions. It evaluates the distance from the current solution to the utopian point, and prefers the action with smallest SQ-value to accelerate algorithm convergence.

③Further to avoid falling into local optimal, we balance exploration and exploitation by using the scalarized ε-greedy policy.

Please review the above explanations and Subsection 6.6. Thank you.

(2) What about heterogeneous environment, where the servers have different resource capacities of energy efficiency levels?

Authors’ response:

Thanks for your comment.

①Heterogeneity reflected by total response latency

In our mathematical formulation, the demand for CPU resources by a VM request is the demand for the number of CPU cores. The processing speed of a VM is determined by its assigned server type. Its processing time (response latency) is calculated based on the processing speed according to equation (2). Its processing speed varies with its assigned server type. The total response latency is the sum of the response latency of all VM requests, which is shown in equation (5).

Besides, the reward function corresponding to total response latency is equation (15). The smaller the response latency generated by the currently placed VM, the greater the reward. Therefore the algorithm can select the server type based on the response latency.

As a result, our first objective can reflect the heterogeneity of server types.

②Heterogeneity reflected by total energy consumption

As shown in equation (7), the energy consumption of a server type is calculated based on its power consumption and response latency. Based on equation (6) and equation (7), both of them are dependent on the current server type.

Besides, the reward function corresponding to total energy consumption is equation (16). The smaller the energy consumption generated by the currently placed VM, the greater the reward value. Therefore the algorithm can select the server type based on energy consumption.

Please review the above explanations and the modifications in Section 5.1.3. Thank you.

7.There are many writing issues. The sentences are not formed properly, which will cause difficulty for the readers. (Lines 53-55)

Authors’ response:

Thanks for your comment.

In the revised version, we reorganize this sentence and explained it in detail using the two optimization objectives in this paper as examples. The modifications are as follows:

“Besides, these algorithms sum the objective functions without considering the gap between the order-of-magnitude differences of objective values [14–18]. It makes the optimization objective with a smaller order of magnitude have a limited impact on the final result. For example, the energy consumption in our experiments is much greater than the latency. If we scalarize the objective functions and add them up, the latency will have a negligible effect on the final result.”

Please review the above explanations and the modifications in Section 1. Thank you.

8.Throughout the paper, avoid using "[18] proposes/investigated..". The references should be listed in a way that is classified or clearly stated with the purpose. Summarization is needed in terms of popular methods, distinctions and similarities between various applications.

Authors’ response:

Thanks for your comment.

(1) In the revised version, we have restructured the section of related work and provided a more detailed description of the related work.

Please review the above explanations and the modifications in Section 2. Thank you.

(2) In the revised version, we provide a table of related work to summarize and compare the related work.

In Table 1, we summarize and compare the related work in the aspects of the environment, the framework, and optimization objectives.

Please review the above explanations and the modifications in Section 2. Thank you.

9.The term "order-of-magnitude" is not used properly in the manuscript. For instance Line 124 can be written as "the energy consumption is order-of-magnitude higher than..."

Authors’ response:

Thanks for your comment.

In the revised version, we reorganize this sentence and its next sentence.

“For instance, the energy consumption is significantly order-of-magnitude higher than the response latency in [15], resulting in a negligible effect of the response latency on the results. The energy consumption is significantly order-of-magnitude higher than the resource utilization in [18], resulting in a negligible effect of the resource utilization on the results.”

Please review the above explanations and the modifications in Subsection 2.2. Thank you.

10.Line 133, a new paragraph is not needed.

Authors’ response:

Thanks for your comment.

In the revised version, we place the sentence at the end of the preceding paragraph.

Please review the above explanations and the modifications in the last sentence of Section 2. Thank you.

11.There are several typos and grammatical errors throughout the manuscript. To name a few:

Line 25. 2.46 billion

Line 39

Line 50

Line 135

Rewrite (Lines 83-86)

Lines 160-161

Line 236

Authors’ response:

Thanks for your comment.

We have checked the spelling and grammar throughout the manuscript and made corrections accordingly. The typos and grammatical errors, as suggested by the reviewers, have been corrected.

Please review the above explanations and the modifications have been marked in yellow. Thank you.

12."SLA" is not defined

Authors’ response:

Thanks for your comment.

SLA is the abbreviation of Service Level Agreement. We have explained it in the first paragraph of Section 2.

Please review the above explanations and the modifications in Section 2. Thank you.

13.Consider rearranging the figures. Place them near the text they are referred in. Authors’ response:

Thanks for your comment.

In the original version, each figure and the text it is referred in are too far apart. It was less reader-friendly.

In the revised version, we have placed each figure near its text.

In the revised version:

(1)Figure 1 is between lines 187 and 188.

The relevant text is in line 188.

(2) Figure 2 is between lines 389 and 390.

The relevant text is in line 393.

(3) Figure 3 is between lines 403 and 404.

The relevant text is in line 405.

(4)Figure 4 is between lines 412 and 413.

The relevant text is in line 413.

(5) Figure 5 is between lines 446 and 447.

The relevant text is in line 447.

(6) Figure 6 is between lines 484 and 485.

The relevant text is in line 480.

(7) Figure 7 is between lines 491 and 492.

The relevant text is in line 488.

(8) Figure 8 is between lines 495 and 496.

The relevant text is in line 494.

Please review the above explanations and the modifications in each figure and its relevant text. Thank you.

We really appreciate these valuable comments and suggestions by the reviewer.

Author Response File: Author Response.docx

Reviewer 4 Report

Comments and Suggestions for Authors

Dear authors,

This study focus on the trade-off between the mitigation of response latency of VM requests and the energy consumption of MEC servers in edge computing, through a multi-objective VM placement problem.

Nonetheless, the study do not advance solutions for a myriad of limitations in the model of Pareto front for large-scale MEC systems, such as the potential scalability that might demand considerable resources and optimization algorithms; the irrelevance of such model in turbulent changing contexts; or even the heterogeneity of MEC servers and users, difficult to standardize a Pareto Front.

Those issues need either to be included in the main text, or referred to in a subheading ‘limitations of the study’ in the conclusions section.

Also, some grammar errors need fixing e.g. line 92.

Best wishes,

Author Response

1.Nonetheless, the study do not advance solutions for a myriad of limitations in the model of Pareto front for large-scale MEC systems, such as the potential scalability that might demand considerable resources and optimization algorithms; the irrelevance of such model in turbulent changing contexts; or even the heterogeneity of MEC servers and users, difficult to standardize a Pareto Front.

Those issues need either to be included in the main text, or referred to in a subheading ‘limitations of the study’ in the conclusions section.

(1) The potential scalability that might demand considerable resources and optimization algorithms

Authors’ response:

To demonstrate the scalability of EVMPRL, we have added the subsection of scalability analysis in the revised version. We verify the scalability of EVMPRL based on the methodology in [21].

Although the running time increases with the larger VM request scale, EVMPRL still can obtain good convergence because of the following effective measures:

③Further to avoid falling into local optimal, we balance exploration and exploitation by using the scalarized ε-greedy policy.

Please review the above explanations and Subsection 6.6. Thank you.

(2) The irrelevance of such model in turbulent changing contexts

Authors’ response:

Thanks for your comment.

In fact, the problem addressed in this paper is in a static context.

The static contexts are also very important in the field of VM placement. Some representative related work includes[14],[15],[21], etc.

To illustrate the limitations and directions for improvement, we have added a paragraph at the end of the conclusion section.

Please review the above explanations and the modifications in Section 7. Thank you.

(3) The heterogeneity of MEC servers and users, difficult to standardize a Pareto Front.

Authors’ response:

①Heterogeneity reflected by total response latency

As a result, our first objective can reflect the heterogeneity of server types.

②Heterogeneity reflected by total energy consumption

Please review the above explanations and the modifications in Section 5.1.3. Thank you.

2.Also, some grammar errors need fixing e.g. line 92.

Authors’ response:

Thanks for your comment.

(1)In the revised version, we have removed the redundant “is”. “VM placement is critical for cloud computing.”

Please review the above explanations and the modifications in the first paragraph of Subsection 2.1. Thank you.

(2) Besides, we have checked the spelling and grammar throughout the manuscript and made corrections accordingly. The typos and grammatical errors, as suggested by the reviewers, have been corrected.

We really appreciate these valuable comments and suggestions by the reviewer.

Author Response File: Author Response.docx

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have addressed the comments and improved the quality of manuscript.
I have no further comments.