Review Reports - Adaptive Multi-Agent Reinforcement Learning for Optimizing Dynamic Electric Vehicle Charging Networks in Thailand

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

I thank you for giving me the opportunity to have read with much pleasure your paper entitled “Adaptive Multi-Agent Reinforcement Learning for Optimizing Dynamic Electric Vehicle Charging Networks in Thailand”.

The research topic is certainly timely as it addresses issues related to the decarbonization of mobility systems. The goal of the work is to evaluate how the use of MARL can be used to manage dynamic vehicle charging networks. The study considers the vehicle recharging network in Thailand.

The paper is well organized in an organic way into 6 sections: i) introduction, ii) Literature Review, iii) Methodology, iv) Simulation and Results, v) Discussion and vi) Conclusions.

The following are my major revisions. I hope they can be helpful to you in improving and enhancing the paper:

1. At the end of section 1, I suggest including a few lines to describe the organization of the paper (reminder).

2. Figures 1 and 2 are not cited in the text.

3. In line 120 the authors write: Current State of EV Adoption: The adoption of EVs in Thailand is growing, driven by government incentives and increasing consumer awareness. The dominance of specific EV brands and charging point manufacturers shapes the infrastructure landscape. It would be helpful to include some numbers (graph) to justify this statement.

4. Concerning possible strategies for electrification of mobility systems considering user habits, I suggest reading this paper where you can find some interesting insights:

Decarbonizing transportation: A data-driven examination of ICE vehicle to EV transition (2024). Cleaner Engineering and Technology, 21. https://doi.org/10.1016/j.clet.2024.100782

Regards

Author Response

Reviewer 1

The paper is well organized in an organic way into 6 sections: i) introduction, ii) Literature Review, iii) Methodology, iv) Simulation and Results, v) Discussion and vi) Conclusions.

Answer: Dear Reviewer, the author extends sincere appreciation to the reviewers for their kindness, valuable suggestions, and constructive feedback, which have significantly enhanced the quality of our paper. In response to your recommendations, we have thoroughly revised and amended the manuscript. The changes are red color, with additional information indicated in yellow highlight.

-----------------------------------------------------------------------------------------------------------------------

The following are my major revisions. I hope they can be helpful to you in improving and enhancing the paper:

At the end of section 1, I suggest including a few lines to describe the organization of the paper (reminder).

Answer: Thank you very much for your valuable suggestion. We have already added a description of the organization of the paper at the end of the Introduction section. (Lines 221-232).

Figures 1 and 2 are not cited in the text.

Answer: Thank you for your valuable suggestion to include citations in the text for Figures 1 and 2 to enhance the completeness of the content. (Lines 115,334)

In line 120 the authors write: Current State of EV Adoption: The adoption of EVs in Thailand is growing, driven by government incentives and increasing consumer awareness. The dominance of specific EV brands and charging point manufacturers shapes the infrastructure landscape. It would be helpful to include some numbers (graph) to justify this statement. (Lines 122-130)

Answer: Thank you for your recommendation to provide additional information on this topic. “This influenced the market estimates about Thailand's prospects of selling EVs have been revised and the burden increased significantly primarily because of better sales growth achieved in the first half of the year 2024 than expected. In this period, a total of 49,319 EVs were sold, which is above the original estimate for the sales of the whole year. Therefore, the sales forecasts have been revised to reflect sales of 80,700 units with an increase of 151% compared to 20.7% estimated last time. Even though most of the forecasts expect an overall contraction in total vehicle sales, it is anticipated that there will be a surge in sales of EVs because of the attractive subsidies which will maintain a high demand for them [22]”.

Concerning possible strategies for electrification of mobility systems considering user habits, I suggest reading this paper where you can find some interesting insights:

Decarbonizing transportation: A data-driven examination of ICE vehicle to EV transition (2024). Cleaner Engineering and Technology, 21. https://doi.org/10.1016/j.clet.2024.100782

Answer: Thank you once again for your valuable recommendation for further reading from the paper you suggested. We have reviewed it and provided directions for future research based on this study. (1031-1043)

“Furthermore, subsequent studies should consider examining the shift from ICE vehicles to EVs as an effective way to reduce carbon emissions in the transport industry. Such a research activity should use in-depth data analysis focusing on environmental assessment, cost estimation, and technological aspects of this shift. It ought to showcase the considerable prospects of EVs in mitigating greenhouse gas emissions while tackling the obstacles of infrastructure, policy, and the awareness of people for sustainable development [101]. In particular, future studies should propose a new method of tracking control of nonlinear dynamic systems based on, which is enriched by neural network NN – reinforcement learning RL. This method solves the problems of tracking and optimization simultaneously. The effectiveness of the approach is confirmed through numerical simulations, which produce positive results and thus prove the effectiveness of the method to optimize tracking control of nonlinear dynamic systems [102-103].

The authors wish to express their sincere gratitude to the Reviewer for the invaluable suggestions, which have significantly contributed to enhancing the quality of this paper. We also deeply appreciate the encouragement to further our research on the application of advanced technologies in the field. Thank you for your support.

Reviewer 2 Report

Comments and Suggestions for Authors

This paper explores the use of an adaptive Multi-Agent Reinforcement Learning (MARL) approach to optimise electric vehicle (EV) charging networks in Thailand. However, there are several modifications needed for the authors:

[1] When describing the MARL framework, the specific implementation details of each component, such as the hyperparameter settings of the DQN algorithm, the optimisation strategy during training, etc., can be described in more detail.

[2] When presenting the experimental results, please add further charts and visual data to compare the performance differences between different scenarios more intuitively.

[3] Please double-check the language and format of your paper to ensure clarity of expression, logical coherence and formatting.

[4] Please further specify the advantages and innovations of the methods herein over traditional or existing methods.

[5] In this paper, the author focuses on using artificial intelligence methods to optimize the scheduling of electric vehicles, which can compare and analyze different artificial intelligence methods to demonstrate your advantages, which can refer to:

[a] IEEE Transactions on Industrial Informatics, vol. 15, no. 14, pp. 2008-2022, 2019

[b] IEEE Transactions on Power Electronics, vol. 36, no. 1, pp. 73-77, Jan. 2021

Comments on the Quality of English Language

A proof reading is needed.

Author Response

Reviewer 2

-----------------------------------------------------------------------------------------------------------------------

[1] When describing the MARL framework, the specific implementation details of each component, such as the hyperparameter settings of the DQN algorithm, the optimization strategy during training, etc., can be described in more detail.

Answer: Thank you for your insightful suggestion. We have included a description of the MARL framework and detailed the specific implementation aspects of each component, including the hyperparameter settings for the DQN algorithm and the optimization strategy employed during training, in Section 3.3. This section contains the following subsections (Lines 541-607).

3.3 Algorithm Design

3.3.1 Hyperparameters

3.3.2 Optimization Strategy

3.3.3 Simulation Parameters

3.3.4 Evaluation Metrics

[2] When presenting the experimental results, please add further charts and visual data to compare the performance differences between different scenarios more intuitively.

Answer: Thank you for your thoughtful suggestion. We have included new Figures 16-21 to illustrate the results and intuitively compare the performance differences across various scenarios. These can be found in Lines 820-867. (Lines 820-867)

Figure 16. Reward Comparison per Episode for Different Scenarios

Figure 17. Convergence Time Across Scenarios

Figure 18. Exploration vs. Exploitation Analysis

Figure 19. Reward Maximization Over Time

Figure 20. State Variable Evolution Comparison

Figure 21. Computational Efficiency

[3] Please double-check the language and format of your paper to ensure clarity of expression, logical coherence and formatting.

Answer: Thank you for your recommendation. We will conduct a thorough review of the language and formatting throughout the paper to ensure clarity of expression, logical coherence, and compliance with the required formatting standards.

[4] Please further specify the advantages and innovations of the methods herein over traditional or existing methods.

Answer: Thank you very much for your valuable suggestion. We have added information regarding the advantages and innovations of the methods presented in this paper compared to traditional or existing approaches. A new section titled "5.2 Advantages and Innovations of Adaptive MARL Methods" has been included in the discussion section to outline these advantages and innovations. This section can be found in Lines 868-912.

[a] IEEE Transactions on Industrial Informatics, vol. 15, no. 14, pp. 2008-2022, 2019

[b] IEEE Transactions on Power Electronics, vol. 36, no. 1, pp. 73-77, Jan. 2021

Answer: Thank you for your constructive feedback. In this paper, we have included future research directions to emphasize the use of artificial intelligence methods for optimizing the scheduling of electric vehicles. We propose to compare and analyze various artificial intelligence methodologies to highlight their respective advantages. These references will support our discussion and provide a solid foundation for our analysis, as indicated by references [101-103].

To support this analysis, we have considered the following sources: (Lines 1031-1043)

[a] IEEE Transactions on Industrial Informatics, vol. 15, no. 14, pp. 2008-2022, 2019

We review the literature and add information

[102] Wen, G.; Chen, C.L.P.; Ge, S.S.; Yang, H.; Liu, X. Optimized Adaptive Nonlinear Tracking Control Using Actor–Critic Rein-forcement Learning Strategy. IEEE Trans. Ind. Informat. 2019, 15(9), 4969–4977. https://doi.org/10.1109/TII.2019.2894282.

[b] IEEE Transactions on Power Electronics, vol. 36, no. 1, pp. 73-77, Jan. 2021

We review the literature and add information

[103]Zhang, C.; Chen, J.; Liu, Y.; Wang, X.; Li, Z.; Zhang, S. A Strong Misalignment-Tolerance Wireless Power Transfer System Based on Dynamic Diffusion Magnetic Field for Unmanned Aerial Vehicle Applications. IEEE Trans. Power Electron. 2024, 39(11), 14129–14134. https://doi.org/10.1109/TPEL.2024.3426609.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear Authors,

thank you for considering my suggestions. I hope my recommendations have helped to improve your research.

You have responded in a timely and comprehensive manner to the four requests for further study and evaluation.

Therefore, I believe that the research is publishable without further modification.

Thank you

Author Response

Thank you so much for your suggestions.