Optimizing Electric Vehicle Charging Recommendation in Smart Cities: A Multi-Agent Reinforcement Learning Approach
Mohammad Ekramul Kabir
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe authors addressed an interesting topic, especially while EV industry is booming through over the world. They choose a real-world example of Thailand. However, the presentation of the manuscript requires significant improvement. The specific comments are as follows-
1) Literature review section is way too long: i) EV is known to everyone, so this was not necessary to mention that long. ii) trends of using EVs can be placed in introduction part (in a shorter format) to motivate the work, iii) deploying charging station problem also needs to consider other things like mentioned in https://ieeexplore.ieee.org/abstract/document/9090360
2) This would be interesting to add the mathematical model for the optimization problem along with its constraints.
3) The result section is not interesting. This would be great to illustrate the findings and vary different parameters to see the impact.
4) What is the red rectangle (right one) in Fig 1?
5) Fig 3 could be improved, e.g., use text color to match with arrow color.
6) There are some typos, e.g., Page 4, line 185, "Here, it would be the location of qt". qt must not be the location, its charging request
Comments on the Quality of English Language1) The major problem is the orientation. There are too many redundancy that makes the manuscript unnecessarily long. Moreover, sections F and G have the same title!!! "Multiple objectives optimization"!!! why?
2) There are many typos, as I mentioned earlier. So, please check and edit accordingly.
Author Response
Expressing gratitude for the valuable comments and suggestions received, we have duly incorporated the necessary corrections and revisions in accordance with your guidance. Please find the revised details highlighted in green in the subsequent sections of this manuscript.
1) Literature review section is way too long: i) EV is known to everyone, so this was not necessary to mention that long. ii) trends of using EVs can be placed in introduction part (in a shorter format) to motivate the work, iii) deploying charging station problem also needs to consider other things like mentioned in https://ieeexplore.ieee.org/abstract/document/9090360.
Answer 1. We have eliminated extraneous information in this section, including the summary, and condensed the content to succinctly delineate the essence of electric vehicles (EVs) and the prevailing trends. The revised material now spans from (line 250 to 277).
2. For citation purposes, we have referred to the work authored by [77] Kristian Sevdari, Lisa Calearo, Peter Bach Andersen, Mattia Marinelli, "Ancillary services and electric vehicles: An overview from charging clusters and chargers technology perspectives," Renewable and Sustainable Energy Reviews, vol. 167, 112666, 2022, ISSN 1364-0321, https://doi.org/10.1016/j.rser.2022.112666.
2) This would be interesting to add the mathematical model for the optimization problem along with its constraints.
Answer: We have incorporated a mathematical model to address the optimization problem, along with its corresponding constraints, within the designated section titled "Mathematical Model for Optimization." The pertinent details can be found in lines 535 to 581 of the document.
3) The result section is not interesting. This would be great to illustrate the findings and vary different parameters to see the impact.
Answer We have included the evaluation results of various parameters, emphasizing the significant influence of diverse algorithms on key metrics. This addition provides essential insights into the effectiveness of these algorithms concerning EV charging station recommendations. The specific findings are outlined in lines 1073 to 1112 of the document.
4) What is the red rectangle (right one) in Fig 1?
Answer We have rectified the typing error. The red rectangle illustrating the EV Charging System in the smart city has been repositioned or redesigned as needed.
5) Fig 3 could be improved, e.g., use text color to match with arrow color.
Answer Figure 3 has been removed from the previous version, and a novel research framework, represented in Figure 2, has been established to expound upon the steps taken in this study. This diagram meticulously outlines a systematic research framework crafted for the optimization of Electric Vehicle Charging Stations (EVCS) through the application of Multi-Agent Reinforcement Learning (MARL), as detailed in lines 408 to 433.
6) There are some typos, e.g., Page 4, line 185, " Here, it would be the location of qt ". qt must not be the location, its charging request.
Answer We have changed the thoroughly reviewed all equations and formulas incorporated in this paper.
Comments on the Quality of English Language
1) The major problem is the orientation. There are too many redundancy that makes the manuscript unnecessarily long. Moreover, sections F and G have the same title!!! " Multiple objectives optimization "!!! why?
Answer We apologize for the typographical error involving the duplication of content in two sections. The redundant content has been removed.
2) There are many typos, as I mentioned earlier. So, please check and edit accordingly.
Answer We have successfully rectified all typographical errors throughout the entire paper.
Reviewer 2 Report
Comments and Suggestions for AuthorsThe paper is of interest, I have a few comments to make:
1. I would expect more results and more enriched actually instead of that long text with the algorithms text representation. Is it possible for the authors to enrich the content with provifing additional results?
2. The literature survey is quite extended, I would suggest a few more recent papers on this topic as well. Example: Paraskevas, Athanasios, et al. "Optimal management for EV charging stations: A win–win strategy for different stakeholders using constrained Deep Q-learning." Energies 15.7 (2022): 2323.
3. I would like to see some benchmarking of the proposed implementation against pre-published research
4. I would also like to see some discussion on the computational effort/burden of this solution
Author Response
Expressing gratitude for the valuable comments and suggestions received, we have duly incorporated the necessary corrections and revisions in accordance with your guidance. Please find the revised details highlighted in green in the subsequent sections of this manuscript.
1) I would expect more results and more enriched actually instead of that long text with the algorithms text representation. Is it possible for the authors to enrich the content with providing additional results?
Answer We have enriched the content by including additional details concerning the flowchart depicting algorithm execution, aiming to provide a more elaborate representation than an extensive textual description. Additionally, we have incorporated supplementary content focusing on the evaluation results, highlighting the significant influence of various algorithms on key parameters and offering valuable insights into their effectiveness within the realm of EV charging station recommendations (Lines 1072-1110). Numeric results are presented in terms of numerical indicators (Lines 1222-1233). Moreover, a detailed explanation has been provided regarding the notable advantages of MADDPG, emphasizing its unique capabilities (Lines 1234-1245).
2) The literature survey is quite extended, I would suggest a few more recent papers on this topic as well. Example:
Paraskevas, Athanasios, et al. "Optimal management for EV charging stations: A win–win strategy for different stakeholders using constrained Deep Q-learning." Energies 15.7(2022): 2323.
Answer We greatly appreciate your feedback. In response, we have streamlined the content in this section by removing extraneous details, including the summary. The material has been succinctly restructured to capture the fundamental aspects of electric vehicles (EVs) and the prevailing trends, with a particular focus on brevity (lines 250 to 277).
Additionally, we have incorporated a reference to a relevant work: [78] Paraskevas, A.; Aletras, D.; Chrysopoulos, A.; Marinopoulos, A.; Doukas, D.I. "Optimal Management for EV Charging Stations: A Win–Win Strategy for Different Stakeholders Using Constrained Deep Q-Learning," published in Energies 2022, volume 15, page 2323.
3) I would like to see some benchmarking of the proposed implementation against pre-published research.
Answer We have introduced a benchmarking analysis of the proposed implementation against previously published research within the Results section, spanning from line 1073 to 1135. This addition aims to provide a comparative assessment of the performance of our implementation in relation to existing studies, offering valuable insights into its effectiveness.
4) I would also like to see some discussion on the computational effort/burden of this solution.
Answer We have expanded the discussion on the computational effort of the proposed solution, providing more detailed insights, particularly in lines 1223 to 1246.
Reviewer 3 Report
Comments and Suggestions for AuthorsAn improved approach for recommending charging stations of electrical vehicles based on multi-agent reinforcement learning algorithms is proposed in the paper. The results presented in the manuscript demonstrate the superior performance of the proposed method compared to the methods proposed in other research papers. The conclusions are consistent with the evidence and arguments presented. English language is clear with some minor grammar mistakes. There are no excessive self-citations in the manuscript.
There are several comments and suggestions to improve quality of the paper.
1) There are several figures copied from other papers. Do the authors of the manuscript have permission to copy the images from the copyright holders?
2) It is suggested to explain the proposed algorithm in more clear way. For example, show a flowchart to better understand the algorithm.
3) The paper has much text and very low number of figures and tables, so it is difficult to catch the idea for a reader. It is suggested to show more relevant figures and tables.
Comments on the Quality of English LanguageMinor editing of English language required
Author Response
An improved approach for recommending charging stations of electrical vehicles based on multi-agent reinforcement learning algorithms is proposed in the paper. The results presented in the manuscript demonstrate the superior performance of the proposed method compared to the methods proposed in other research papers.
The conclusions are consistent with the evidence and arguments presented.
English language is clear with some minor grammar mistakes. There are no excessive self-citations in the manuscript.
Answer
Expressing gratitude for the valuable comments and suggestions received, we have duly incorporated the necessary corrections and revisions in accordance with your guidance. Please find the revised details highlighted in green in the subsequent sections of this manuscript.
There are several comments and suggestions to improve quality of the paper.
1) There are several figures copied from other papers. Do the authors of the manuscript have permission to copy the images from the copyright holders?
Answer Figure 1 is sourced from the Thailand government, and we have obtained the copyright permission for its usage. Additionally, we have removed Figure 2, as it is deemed unnecessary for the scope of this paper.
2) It is suggested to explain the proposed algorithm in more clear way. For example, show a flowchart to better understand the algorithm.
Answer We have incorporated flowcharts detailing the steps of each algorithm and provided explanations on their functionality. Specifically:
Figure 3: Flowchart illustrating the steps of the Real Algorithm (Lines 586-610).
Figure 4: Flowchart illustrating the steps of the Random Algorithm (Lines 586-610).
Figure 5: Flowchart illustrating the steps of the DQN Algorithm (Lines 639-686).
Figure 6: Flowchart illustrating the steps of the DDPG Algorithm (Lines 696-758).
Figure 7: Flowchart illustrating the steps of the MADDPG Algorithm (Lines 760-826).
3) The paper has much text and very low number of figures and tables, so it is difficult to catch the idea for a reader. It is suggested to show more relevant figures and tables.
Answer We have introduced pivotal figures, namely Figures 3 through 7, to visually encapsulate ideas and succinctly summarize the textual content. These figures specifically serve to illustrate the step-by-step flowcharts for each algorithm, enhancing the comprehension of the algorithms' processes. The relevant information can be found in lines 585 to 826 of the paper.
Reviewer 4 Report
Comments and Suggestions for AuthorsThis article's strength is handling essential topics such as optimising electric vehicle charging in intelligent cities using a new approach as multiagent DDPG (MADDPG). It gives new advantages described in the article.
1. In row 183, the indexes are in subscribe, not in capital letters.
2. In row 185, „the location of qt „is mentioned. In row 184 it was defined as a charging request. Symbol t is mentioned as time, not place.
3. All symbols might be in italics.
4. In row 193, „ci“is present. Index i might be as subscript. All such places need to improve.
5. In row 200, the symbol C does not mention a description, what it mentions.
6. Please describe what the abbreviation MARL algorithm means in row 218. It needs to describe in first appearance
7. In row 224, what means the ECCS?
8. Present the MADDPG as flowchart.
9. In conclusion needs to present some numerical indicators, what is advantage behind others the MADDPG algorithm.
Author Response
This article's strength is handling essential topics such as optimizing electric vehicle charging in intelligent cities using a new approach as multiagent DDPG (MADDPG). It gives new advantages described in the article.
Answer
Expressing gratitude for the valuable comments and suggestions received, we have duly incorporated the necessary corrections and revisions in accordance with your guidance. Please find the revised details highlighted in green in the subsequent sections of this manuscript.
1) In row 183, the indexes are in subscribed, not in capital letters.
Answer The adjustment to normal letter case in row 183 within the green highlight has been successfully made.
2) In row 185, the location of qt is mentioned. In row 184 it was defined as a charging request. Symbol t is mentioned as time, not place.
Answer The relocation of "qt" has been successfully implemented as per your instruction.
3) All symbols might be in italics.
Answer The conversion of all symbols to italics within the green highlight has been successfully executed.
4) In row 193, „ci“is present. Index i might be as subscript. All such places need to improve.
Answer We have thoroughly reviewed all equations and formulas, and in accordance with this scrutiny, we have made the correction of "ci" to ci in row 193.
5) In row 200, the symbol C does not mention a description, what it mentions.
Answer We have provided definitions for all symbols employed in this paper, encompassing the range from line 183 to line 212.
6)Please describe what the abbreviation MARL algorithm means in row 218. It needs to describe in first appearance.
Answer We have furnished a comprehensive elucidation of the Multi-Agent Reinforcement Learning (MARL) algorithm at row 214-222.
7)In row 224, what means the ECCS?
Answer We have provided definitions for key terms and expounded on the Electric Vehicle (EV) charging recommendation problem, incorporating the following:
Definition 1: Charging request: a charging request is defined as the t-th request (i.e., step t) of a day.
Definition 2: CWT: This was defined as the sum of the travel time from the location of the charging request q_t to the target charging station c^i, and the queuing time at c^i, until q_t had finished charging.
Definition 3: CP: This was defined as the cost per unit of electricity, which was typically a combination of the electricity cost and service fee.
Definition 4: Charging Failure Ratio (CFR): CFR is defined as the ratio of charging requests that accepted the authors’ recommendation but failed to charge, divided by the total number of charging requests that accepted the recommendation. The authors aimed to recommend the most suitable charging station (Line 181-212).
8)Present the MADDPG as flowchart.
Answer We have incorporated the flowcharts detailing the steps of each algorithm and provided explanations on their functionality. Specifically:
Figure 3: Flowchart illustrating the steps of the Real Algorithm (Lines 586-610).
Figure 4: Flowchart illustrating the steps of the Random Algorithm (Lines 586-610).
Figure 5: Flowchart illustrating the steps of the DQN Algorithm (Lines 639-686).
Figure 6: Flowchart illustrating the steps of the DDPG Algorithm (Lines 696-758).
Figure 7: Flowchart illustrating the steps of the MADDPG Algorithm (Lines 760-826).
9)In conclusion needs to present some numerical indicators, what is advantage behind others the MADDPG algorithm.
Answer We have included numerical indicators in the conclusion section, spanning from line 1222 to 1233. Additionally, we have appended a discussion highlighting the advantages of the MADDPG algorithm compared to others in lines 1234 to 1245.
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsThe quality of flowcharts is poor. Please improve the drawing quality before the final submission.
Comments on the Quality of English LanguagePlease read the manuscript carefully to avoid typos and other anomalies.
Author Response
The quality of flowcharts is poor. Please improve the drawing quality before the final submission.
Answer: We have redrawn Figure 2-7 into a new format, enhancing pixel clarity to meet the guidelines outlined by the journal. Additionally, we have highlighted Figure 2-7 in green.
Figure 2 Line 431
Figure 3 Line 586
Figure 4 Line 612
Figure 5 Line 639
Figure 6 Line 695
Figure 7 Line 759
Reviewer 2 Report
Comments and Suggestions for Authorsno further comments, thanks
Author Response
Thank you so much.
Reviewer 3 Report
Comments and Suggestions for AuthorsThe reviewer is satisfied with the answers given and the corrections made. The paper is accepted.
Author Response
Thank you so much.
