Deep Reinforcement Learning-Based Real-Time Joint Optimal Power Split for Battery–Ultracapacitor–Fuel Cell Hybrid Electric Vehicles

Round 1

Reviewer 1 Report

The manuscript presented a method to optimize power split for a hybrid energy storage based EV. Neural network was used to predict speed and acceleration; the DRL was used to optimize power split between UC, FC, and battery to improve vehicle performance. The topic is interesting, some improvements are suggested as follows.

1. Please add more details about the neural network model and its performance.
2. Please provide parameters of electric drives including motors, inverters, etc. in experimental test part.
3. Did you consider inverter loss?

Author Response

Response to Reviewer Comments

<< Reviewer's comments to the author(s) >>

The manuscript presented a method to optimize power split for a hybrid energy storage based EV. Neural network was used to predict speed and acceleration; the DRL was used to optimize power split between UC, FC, and battery to improve vehicle performance. The topic is interesting, some improvements are suggested as follows.

1. Please add more details about the neural network model and its performance.
2. Please provide parameters of electric drives including motors, inverters, etc. in the experimental test part.
3. Did you consider inverter loss?

After examining the reviewers’ comments carefully, we have revised our manuscript according to the reviewers' comments as follows.

(Italic letters indicate reviewers' comments. To indicate the revised part of our paper, we also added information such as section number or subsection number, paragraph number or (page number, line number). For example, Subsection 2.2 (p. 6, l. 10) means that there are revised parts on the page number 6, and line 10 in Sect 2.2 of the paper.)

Point 1: Please add more details about the neural network model and its performance.

Response 1: In sections 3 and 4 more details about the neural network model and its performance were added.

• Subsection 3.3 (p. 13): explanation about shaped reward functions used to accelerate the training process
• Section 4 (pages 14 to 19): the results section was rewritten and it now includes many more figures and tables describing the performance of the model, such as:
  • Comparison between the traditional rule-based technique and the proposed DRL technique (Figures 11 to 14)
  • Comparison of the performance of the model for different initial SOC values (Table 9)
  • Comparison between the usage of a single motor and two complementary motors (Table 9)
  • Results of the prediction step and comparison between the predicted and actual values (Figures 16 to 18 and Table 10)

Point 2: Please provide parameters of electric drives including motors, inverters, etc. in the experimental test part.

Response 2: We added parameters for electric drives including motors and inverters and also explained how to use them to calculate power consumption.

• Subsection 2.5 (p. 6-7)
• Table 2
• Subsection 2.6 (p. 11)
• Table 5

Point 3: Did you consider inverter loss?

Response 3: We obtained the efficiency value of the inverter through the inverter datasheet, and added the formula for calculating the inverter loss.

• Subsection 2.6 (p. 11)
• Table 5
• Equation (32)

Author Response File: Author Response.pdf

Reviewer 2 Report

M.S. No.: electronics-1730071

Title: Deep reinforcement learning-based real-time joint optimal power split for battery/ultracapacitor/fuel cell hybrid electric vehicles

In this paper, the authors have designed an efficient power split optimization algorithm for hybrid electric vehicles (HEVs). I have some comments regarding this manuscript (MS).

1. The abbreviation for hybrid electric vehicles should be HEVs (add s if plural).
2. The abbreviations used in abstract should be defined separately, eg. UDDS, SOS, WLTC, etc.
3.  In the introduction, some information regarding experiments supercapacitors and Li-ion batteries should be mentioned. Eg. https://pubs.rsc.org/en/content/articlehtml/2021/ra/d1ra07235h, https://www.sciencedirect.com/science/article/pii/S1385894719306266, https://onlinelibrary.wiley.com/doi/10.1002/sstr.202000064
4. Also references related to solid-state batteries and their future application should be included. Eg. https://www.frontiersin.org/articles/10.3389/fchem.2019.00522/full, https://pubs.acs.org/doi/10.1021/acsenergylett.0c01977#
5. Font size of the texts in the all the Figures should be increased.
6. The experiments and simulation methodology should be provided in more detail.
7. The deviation between experimental and simulation should be provided.
8. The abstract and conclusion should be reconstructed. Authors should follow as

First importance of the field, work done, the gap, strategy to achieve the goal, and outcomes.

Author Response

Response to Reviewer Comments

<< Reviewer's comments to the author(s) >>

Title: Deep reinforcement learning-based real-time joint optimal power split for battery/ultracapacitor/fuel cell hybrid electric vehicles

In this paper, the authors have designed an efficient power split optimization algorithm for hybrid electric vehicles (HEVs). I have some comments regarding this manuscript (MS).

1. The abbreviation for hybrid electric vehicles should be HEVs (add s if plural).
2. The abbreviations used in abstract should be defined separately, eg. UDDS, SOS, WLTC, etc.
3. In the introduction, some information regarding experiments supercapacitors and Li-ion batteries should be mentioned. Eg. https://pubs.rsc.org/en/content/articlehtml/2021/ra/d1ra07235h, https://www.sciencedirect.com/science/article/pii/S1385894719306266, https://onlinelibrary.wiley.com/doi/10.1002/sstr.202000064
4. Also references related to solid-state batteries and their future application should be included. Eg. https://www.frontiersin.org/articles/10.3389/fchem.2019.00522/full, https://pubs.acs.org/doi/10.1021/acsenergylett.0c01977#
5. Font size of the texts in the all the Figures should be increased.
6. The experiments and simulation methodology should be provided in more detail.
7. The deviation between experimental and simulation should be provided.
8. The abstract and conclusion should be reconstructed. Authors should follow as: First importance of the field, work done, the gap, strategy to achieve the goal, and outcomes.

After examining the reviewers’ comments carefully, we have revised our manuscript according to the reviewers' comments as follows.

(Italic letters indicate reviewers' comments. To indicate the revised part of our paper, we also added information such as section number or subsection number, paragraph number or (page number, line number). For example, Subsection 2.2 (p. 6, l. 10) means that there are revised parts on the page number 6, and line 10 in Sect 2.2 of the paper.)

Point 1: The abbreviation for hybrid electric vehicles should be HEVs (add s if plural).

Response 1: As suggested by the reviewer, we revised the occurrences of the words HEV and HEVs and corrected them accordingly.

• Abstract (p. 1)
• Introduction (pp.1-2)

Point 2: The abbreviations used in abstract should be defined separately, eg. UDDS, SOS, WLTC, etc.

Response 2: As suggested by the reviewer, we revised the abbreviations used in the abstract and defined them where necessary.

• Abstract (p. 1)

Point 3: In the introduction, some information regarding experiments supercapacitors and Li-ion batteries should be mentioned. Eg. https://pubs.rsc.org/en/content/articlehtml/2021/ra/d1ra07235h, https://www.sciencedirect.com/science/article/pii/S1385894719306266, https://onlinelibrary.wiley.com/doi/10.1002/sstr.202000064

Response 3: As suggested by the reviewer, we added references and explanations in the introduction.

• Introduction (p. 2, l.13-14)
• References

Point 4: Also references related to solid-state batteries and their future application should be included. Eg. https://www.frontiersin.org/articles/10.3389/fchem.2019.00522/full, https://pubs.acs.org/doi/10.1021/acsenergylett.0c01977#

Response 4: As suggested by the reviewer, we added references and explanations in the introduction.

• Introduction (p. 2, l.15-17)
• References

Point 5: Font size of the texts in the all the Figures should be increased.

Response 5: The font size of the texts in the Figures were quite small. Thus, according to the reviewer’s suggestion, we increased their font size. Thank you for pointing that out.

• All Figures

Point 6: The experiments and simulation methodology should be provided in more detail.

Response 6: In sections 3 and 4 more details about the neural network model and its performance were added.

• Subsection 3.3 (p. 13): explanation about shaped reward functions used to accelerate the training process
• Section 4 (pages 14 to 19): the results section was rewritten and it now includes many more paragraphs, figures and tables describing the performance of the model, such as:
  • Comparison between the traditional rule-based technique and the proposed DRL technique (Figures 11 to 14)
  • Comparison of the performance of the model for different initial SOC values (Table 9)
  • Comparison between the usage of a single motor and two complementary motors (Table 9)
  • Results of the prediction step and comparison between the predicted and actual values (Figures 16 to 18 and Table 10)

Point 7: The deviation between experimental and simulation should be provided.

Response 7: In this paper we didn’t use an actual testbed consisting of an actual hybrid electric vehicle. We only performed simulations and the new results obtained were explained in Point 6.

Point 8: The abstract and conclusion should be reconstructed. Authors should follow as First importance of the field, work done, the gap, strategy to achieve the goal, and outcomes.

Response 8: As suggested by the reviewer, we rewrote the abstract and the conclusion trying to follow the structure suggested.

• Abstract
• Conclusion

Author Response File: Author Response.pdf

Reviewer 3 Report

1, The authors need to revise the grammar of this manuscript.

 For example, the last sentence of the abstract. "...fuel efficiency than other....". ;  "... charging time, high price, and severely affected on temperature."

2, Please improve the quality of the figures. I cannot recommend to publish this manuscript until the authors improve it. 

3, Non-standard abbreviations/acronyms should be written out in full on first use. E.g. SDC, RDC and etc. 

4, I don't think the authors provide details of the EMS. Why are there two motors? Without that information, it is difficult for readers to follow this paper. 

5, Is it necessary to present details of UDDS and WLTC? 

6, In figure 5, why is the unit of power loss m/s? 

7, Section 2.6 is not completed. Why?

8, In this optimization problem, what is the onjective function? 

Author Response

Response to Reviewer Comments

<< Reviewer's comments to the author(s) >>

1. The authors need to revise the grammar of this manuscript. For example, the last sentence of the abstract. "...fuel efficiency than other....". ;  "... charging time, high price, and severely affected on temperature."
2. Please improve the quality of the figures. I cannot recommend to publish this manuscript until the authors improve it.
3. Non-standard abbreviations/acronyms should be written out in full on first use. E.g. SDC, RDC and etc.
4. I don't think the authors provide details of the EMS. Why are there two motors? Without that information, it is difficult for readers to follow this paper.
5. Is it necessary to present details of UDDS and WLTC?
6. In figure 5, why is the unit of power loss m/s?
7. Section 2.6 is not completed. Why?
8. In this optimization problem, what is the objective function? 

After examining the reviewers’ comments carefully, we have revised our manuscript according to the reviewers' comments as follows.

(Italic letters indicate reviewers' comments. To indicate the revised part of our paper, we also added information such as section number or subsection number, paragraph number or (page number, line number). For example, Subsection 2.2 (p. 6, l. 10) means that there are revised parts on the page number 6, and line 10 in Sect 2.2 of the paper.)

Point 1: The authors need to revise the grammar of this manuscript. For example, the last sentence of the abstract. "...fuel efficiency than other....". ;  "... charging time, high price, and severely affected on temperature."

Response 1: According to the reviewer’s suggestion, we revised the grammar of the manuscript. Thank you for pointint that out.

Point 2: Please improve the quality of the figures. I cannot recommend to publish this manuscript until the authors improve it.

Response 2: The font size of the texts in the Figures were quite small and, due to their low resolution, it was difficult to see and analyze them. Therefore, we revised all of them and did our best to improve their quality. Thank you very much for your suggestion.

Point 3: Non-standard abbreviations/acronyms should be written out in full on first use. E.g. SDC, RDC and etc.

Response 3:  we checked the manuscript and tried our best to write the acronyms in full on first use. Thank you for the suggestion.

Point 4: I don't think the authors provide details of the EMS. Why are there two motors? Without that information, it is difficult for readers to follow this paper.

Response 4: We described previous studies using two motors in the introduction.

• Introduction (p. 2, l.27-31)

Point 5: Is it necessary to present details of UDDS and WLTC?

Response 5: As suggested by the reviewer, we removed the Figures regarding UDDS and WLTC.

Point 6: In figure 5, why is the unit of power loss m/s?

Response 6: Thank you for pointing that out. We changed the unit to Watts (W).

Point 7: Section 2.6 is not completed. Why?

Response 7: We had previously made a mistake when writing a dissertation and failed to complete it. While writing the thesis again this time, we moved the location of Section 2.6 to Section 2.4 and added pictures.

• Subsection 2.4 (p. 5-6)
• Figure 3

Point 8:  In this optimization problem, what is the objective function? 

Response 8: The objective function is now included in section 3.2.

Author Response File: Author Response.pdf

Reviewer 4 Report

The manuscript entitled 'Deep reinforcement learning-based real-time joint optimal power split for battery/ultracapacitor/fuel cell hybrid electric vehicles' has been reviewed. I have some suggestions that may help to improve the paper quality further:
1- In abstract and main context, some acronyms and nomenclatures such as UDDS, WLTC, and SoC should be stated at the first sight. Also, the gap between the literature and this paper is not mentioned. Authors only stated that the model jointly learns the optimal power split. What is the strategy after that?
2- Does your focus on using deep reinforcement learning-based or any new method or results which have new prominent advantages compared to other approaches of controlling systems?
3- The number of abbreviations in the article is very large. So those topics that are only used in the article should be stated and the rest should be deleted.
4- How do you argue that this method works in other HEV? What is the fundamental difference between optimal power split used in your deep reinforcement learning and other prediction approaches that makes your article unique?
5- A significant advantage and results are not observed from the simulation to reflect the performance of the presented deep learning method. What is the main contribution of the analysis in this work?
6- Where do the speed, power, and power loss values in the figures come from?
7- What will the simulation results be like by changing the battery / ultracapacitor / fuel cell values?
8- What software is used to validate the simulation? How confident are you in the results?

Author Response

Response to Reviewer Comments

 << Reviewer's comments to the author(s) >>

Title: Deep reinforcement learning-based real-time joint optimal power split for battery/ultracapacitor/fuel cell hybrid electric vehicles

In this paper, the authors have designed an efficient power split optimization algorithm for hybrid electric vehicles (HEVs). I have some comments regarding this manuscript (MS).

1. In abstract and main context, some acronyms and nomenclatures such as UDDS, WLTC, and SoC should be stated at the first sight. Also, the gap between the literature and this paper is not mentioned. Authors only stated that the model jointly learns the optimal power split. What is the strategy after that?
2. Does your focus on using deep reinforcement learning-based or any new method or results which have new prominent advantages compared to other approaches of controlling systems?
3. The number of abbreviations in the article is very large. So those topics that are only used in the article should be stated and the rest should be deleted.
4. How do you argue that this method works in other HEV? What is the fundamental difference between optimal power split used in your deep reinforcement learning and other prediction approaches that makes your article unique?
5. A significant advantage and results are not observed from the simulation to reflect the performance of the presented deep learning method. What is the main contribution of the analysis in this work?
6. Where do the speed, power, and power loss values in the figures come from?
7. What will the simulation results be like by changing the battery / ultracapacitor / fuel cell values?
8. What software is used to validate the simulation? How confident are you in the results?

After examining the reviewers’ comments carefully, we have revised our manuscript according to the reviewers' comments as follows.

(Italic letters indicate reviewers' comments. To indicate the revised part of our paper, we also added information such as section number or subsection number, paragraph number or (page number, line number). For example, Subsection 2.2 (p. 6, l. 10) means that there are revised parts on the page number 6, and line 10 in Sect 2.2 of the paper.)

Point 1: In abstract and main context, some acronyms and nomenclatures such as UDDS, WLTC, and SoC should be stated at the first sight. Also, the gap between the literature and this paper is not mentioned. Authors only stated that the model jointly learns the optimal power split. What is the strategy after that?

Response 1: We corrected the acronyms and nomenclatures in the abstract. Thank you for pointing that out. We have also rewritten the introduction to mention more explicitly the gap between the paper and the literature, as explained below.

• Introduction (p.3)
  • Hybrid energy storage system consisting of battery/ultracapacitor/fuel cell and a HEV consisting of two complementary propulsion machines
  • SAC with shaped reward functions
  • Prediction of the future velocity and load power to help the system plan in advance the power split

Point 2:  Does your focus on using deep reinforcement learning-based or any new method or results which have new prominent advantages compared to other approaches of controlling systems?

Response 2: We compared the deep reinforcement learning-based method to the traditional fixed rule-based method. One of the biggest advantages of the deep reinforcement method is that it can adapt well to new data that it has never seen without the need for retraining. We have also included many new results, as shown below:

• Subsection 3.3 (p. 13): explanation about shaped reward functions used to accelerate the training process
• Section 4 (pages 14 to 19): the results section was rewritten and it now includes many more paragraphs, figures and tables describing the performance of the model, such as:
  • Comparison between the traditional rule-based technique and the proposed DRL technique (Figures 11 to 14)
  • Comparison of the performance of the model for different initial SOC values (Table 9)
  • Comparison between the usage of a single motor and two complementary motors (Table 9)
  • Results of the prediction step and comparison between the predicted and actual values (Figures 16 to 18 and Table 10)

Point 3: The number of abbreviations in the article is very large. So those topics that are only used in the article should be stated and the rest should be deleted.

Response 3: We tried to reduce the number of abbreviations, but there may still be quite a few of them. Thank you for pointing that out.

Point 4: How do you argue that this method works in other HEV? What is the fundamental difference between optimal power split used in your deep reinforcement learning and other prediction approaches that makes your article unique?

Response 4: We used a hybrid electric vehicle structure that we have seen in only one paper, which did not focus on reinforcement learning. Additionally, we have also proposed a method to predict the load power of the system to better plan the power allocation. The detailed description of our approach is mentioned in Point 2.

Point 5: A significant advantage and results are not observed from the simulation to reflect the performance of the presented deep learning method. What is the main contribution of the analysis in this work?

Response 5: We have included many more paragraphs, figures and tables regarding the results of the method. They are described in Point 2.

Point 6: Where do the speed, power, and power loss values in the figures come from?

Response 6: We added explanation to Figure 5 by adding Figure 2 and the text in section 2.5. We hope this is enough to clarify the reviewer’s question.

Point 7: What will the simulation results be like by changing the battery / ultracapacitor / fuel cell values?

Response 7: We included results for different initial state of charge values to show the robustness of the model. They are included in Table 9 and Figures 13 and 14.

Point 8: What software is used to validate the simulation? How confident are you in the results?

Response 8: We used the library TF-agents within the TensorFlow framework and the Python programming language. We are confident that the results are good enough and, if necessary, we can submit the source code.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Improved. Thanks.

Reviewer 2 Report

The authors implemented the comments raised by the reviewer. The manuscript can be accepted.

Reviewer 4 Report

Thanks for your response and clarification.