Power Battery Scheduling Optimization Based on Double DQN Algorithm with Constraints
Abstract
:1. Introduction
2. Mathematical Description of Optimization Problem
3. Battery Model and Verification
3.1. Dataset
3.2. Regression Model Based on Deep Learning
3.3. Capacity Estimation Model Based on GPR
3.4. Model Verification
4. Methodology
4.1. Deep Reinforcement Learning Based on Double DQN
4.2. Principal Component Analysis Algorithm
4.3. The Relationship between the Optimization Problem and Double DQN
4.4. Overall Framework
5. Experimental Evaluation and Discussion
5.1. Cycle Life and Cumulative Rewards
5.2. SOC Curve Change during Convergence
5.3. Adaptive Capability of the Algorithm
5.4. Results and Discussion
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Chen, S.; Gao, Z.; Sun, T. Safety challenges and safety measures of Li-ion batteries. Energy Sci. Eng. 2021, 9, 1647–1672. [Google Scholar] [CrossRef]
- Gallardo-Lozano, J.; Romero-Cadaval, E.; Milanes-Montero, M.I.; Guerrero-Martinez, M.A. Battery equalization active methods. J. Power Sources 2014, 246, 934–949. [Google Scholar] [CrossRef]
- Hu, C.; Youn, B.D.; Chung, J. A multiscale framework with extended Kalman filter for lithium-ion battery SOC and capacity estimation. Appl. Energy 2012, 92, 694–704. [Google Scholar] [CrossRef]
- Li, J.; Adewuyi, K.; Lotfi, N.; Landers, R.G.; Park, J. A single particle model with chemical/mechanical degradation physics for lithium ion battery State of Health (SOH) estimation. Appl. Energy 2018, 212, 1178–1190. [Google Scholar] [CrossRef]
- Li, Y.; Liu, K.L.; Foley, A.M.; Zulke, A.; Berecibar, M.; Nanini-Maury, E.; Van Mierlo, J.; Hoster, H.E. Data-driven health estimation and lifetime prediction of lithium-ion batteries: A review. Renew. Sustain. Energy Rev. 2019, 113, 109254. [Google Scholar] [CrossRef]
- Lee, S.J.; Kee, M.; Park, G.H. Sensor data compression and power management scheme for low power sensor hub. IEICE Electron. Express 2017, 14, 20170974. [Google Scholar] [CrossRef] [Green Version]
- Fang, X.; Hodge, B.M.; Bai, L.Q.; Cui, H.T.; Li, F.X. Mean-Variance Optimization-Based Energy Storage Scheduling Considering Day-Ahead and Real-Time LMP Uncertainties. IEEE Trans. Power Syst. 2018, 33, 7292–7295. [Google Scholar] [CrossRef]
- Gao, Y.; Yang, J.J.; Yang, M.; Li, Z.S. Deep Reinforcement Learning Based Optimal Schedule for a Battery Swapping Station Considering Uncertainties. IEEE Trans. Ind. Inform. 2020, 56, 5775–5784. [Google Scholar] [CrossRef]
- Xu, G.N.; Du, X.W.; Li, Z.J.; Zhang, X.J.; Zheng, M.X.; Miao, Y.; Gao, Y.; Liu, Q.S. Reliability design of battery management system for power battery. Microelectron. Reliab. 2018, 88–90, 1286–1292. [Google Scholar] [CrossRef]
- Saleeb, H.; Sayed, K.; Kassem, A.; Mostafa, R. Power Management Strategy for Battery Electric Vehicles. IET Electr. Syst. Transp. 2019, 9, 65–74. [Google Scholar] [CrossRef]
- Jan, K.U.; Dubois, A.M.; Diallo, D. Hybrid Battery-SC and Battery-Battery Multistage Design and Energy Management for Power Sharing. In Proceedings of the IECON 2021—47th Annual Conference of the IEEE Industrial Electronics Society, Toronto, ON, Canada, 13–16 October 2021. [Google Scholar] [CrossRef]
- Shaheen, A.M.; Spea, S.R.; Farrag, S.M.; Abido, M.A. A review of meta-heuristic algorithms for reactive power planning problem. Ain Shams Eng. J. 2018, 9, 215–231. [Google Scholar] [CrossRef] [Green Version]
- Paliwal, N.K. A day-ahead Optimal Scheduling Operation of Battery Energy Storage with Constraints in Hybrid Power System. Procedia Comput. Sci. 2020, 167, 2140–2152. [Google Scholar] [CrossRef]
- Ouyang, Q.; Wang, Z.S.; Liu, K.L.; Xu, G.T.; Li, Y. Optimal Charging Control for Lithium-Ion Battery Packs: A Distributed Average Tracking Approach. IEEE Trans. Ind. Inform. 2020, 16, 3430–3438. [Google Scholar] [CrossRef]
- Liu, Z.L.; Wang, X.; Zhang, F. Full Life-Cycle Optimal Battery Scheduling for Maximal Lifetime Value Considering Degradation. IEEE Trans. Energy Convers. 2022, 37, 1379–1393. [Google Scholar] [CrossRef]
- Waschneck, B.; Reichstaller, A.; Belzner, L.; Altenmuller, T.; Bauernhansl, T.; Knapp, A.; Kyek, A. Optimization of global production scheduling with deep reinforcement learning. Procedia CIRP 2018, 72, 1264–1269. [Google Scholar] [CrossRef]
- Rugwiro, U.; Gu, C.H.; Ding, W.C. Task Scheduling and Resource Allocation Based on Ant-Colony Optimization and Deep Reinforcement Learning. J. Internet Technol. 2019, 20, 1463–1475. [Google Scholar] [CrossRef]
- Mbuwir, B.V.; Kaffash, M.; Deconinck, G. Battery Scheduling in a Residential Multi-Carrier Energy System Using Reinforcement Learning. In Proceedings of the 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), Aalborg, Denmark, 29–31 October 2018. [Google Scholar] [CrossRef]
- Sui, Y.; Song, S.M. A Multi-Agent Reinforcement Learning Framework for Lithium-ion Battery Scheduling Problems. Energies 2020, 13, 1982. [Google Scholar] [CrossRef] [Green Version]
- Huang, B.; Wang, J.H. Deep-Reinforcement-Learning-Based Capacity Scheduling for PV-Battery Storage System. IEEE Trans. Smart Grid 2021, 12, 2272–2283. [Google Scholar] [CrossRef]
- Severson, K.A.; Attia, P.M.; Jin, N.; Perkins, N.; Jiang, B.; Yang, Z.; Chen, M.H.; Aykol, M.; Herring, P.K.; Fraggedakis, D.; et al. Data-driven prediction of battery cycle life before capacity degradation. Nat. Energy 2019, 4, 383–391. [Google Scholar] [CrossRef] [Green Version]
- Li, X.Y.; Wang, Z.P.; Zhang, L.; Zou, C.F.; Dorrell, D.D. State-of-health estimation for Li-ion batteries by combing the incremental capacity analysis method with grey relational analysis. J. Power Sources 2019, 410–411, 106–114. [Google Scholar] [CrossRef]
- Che, Y.H.; Deng, Z.W.; Tang, X.L.; Lin, X.K.; Nie, X.H.; Hu, X.S. Lifetime and Aging Degradation Prognostics for Lithium-ion Battery Packs Based on a Cell to Pack Method. Chin. J. Mech. Eng. 2022, 35, 4. [Google Scholar] [CrossRef]
- Roman, D.; Saxena, S.; Robu, V.; Pecht, M.; Flynn, D. Machine learning pipeline for battery state-of-health estimation. Nat. Mach. Intell. 2021, 3, 447–456. [Google Scholar] [CrossRef]
- Zhu, J.G.; Wang, Y.X.; Huang, Y.; Gopaluni, R.B.; Cao, Y.K.; Heere, M.; Muhlbauer, M.J.; Mereacre, L.; Dai, H.F.; Liu, X.H.; et al. Data-driven capacity estimation of commercial lithium-ion batteries from voltage relaxation. Nat. Commun. 2022, 13, 2261. [Google Scholar] [CrossRef] [PubMed]
- de Winter, J.C.F.; Gosling, S.D.; Potter, J. Comparing the Pearson and Spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychol Methods. 2016, 21, 273–290. [Google Scholar] [CrossRef] [PubMed]
- Che, Y.H.; Deng, Z.W.; Lin, X.K.; Hu, L.; Hu, X.S. Predictive battery health management with transfer learning and online model correction. IEEE Trans. Veh. Technol. 2021, 70, 1269–1277. [Google Scholar] [CrossRef]
- Zhang, Y.Z.; Xiong, R.; He, H.W.; Pecht, M.G. Long short-term memory recurrent neural network for remaining useful life prediction of lithium-ion batteries. IEEE Trans. Veh. Technol. 2018, 67, 5695–5705. [Google Scholar] [CrossRef]
- Hu, X.S.; Che, Y.H.; Lin, X.K.; Onori, S. Battery health prediction using fusion-based feature selection and machine learning. IEEE Trans. Transp. Electrif. 2021, 7, 382–398. [Google Scholar] [CrossRef]
- Deng, Z.W.; Hu, X.S.; Lin, X.K.; Xu, L.; Che, Y.H.; Hu, L. General discharge voltage information enabled health evaluation for lithium-ion batteries. IEEE/ASME Trans. Mechatron. 2021, 26, 1295–1306. [Google Scholar] [CrossRef]
- Hu, X.S.; Che, Y.H.; Lin, X.K.; Deng, Z.W. Health prognosis for electric vehicle battery packs: A data-driven approach. IEEE-ASME Trans. Mechatron. 2020, 25, 2622–2632. [Google Scholar] [CrossRef]
- Deng, Z.W.; Hu, X.S.; Lin, X.K.; Che, Y.H.; Xu, L.; Guo, W.C. Data-driven state of charge estimation for lithium-ion battery packs based on Gaussian process regression. Energy 2020, 205, 118000. [Google Scholar] [CrossRef]
- Willmott, C.J.; Matsuura, K. Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Clim. Res. 2005, 30, 79–82. [Google Scholar] [CrossRef]
- Vazquez-Canteli, J.R.; Nagy, Z. Reinforcement learning for demand response: A review of algorithms and modeling techniques. Appl. Energy 2019, 235, 1072–1089. [Google Scholar] [CrossRef]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef] [PubMed]
- van Hasselt, H.; Guez, A.; Silver, D. Deep Reinforcement Learning with Double Q-Learning. In Proceedings of the Association-for-the-Advancement-of-Artificial-Intelligence (AAAI) Conference on Artificial Intelligence, Phoenix, AZ, USA, 12–17 February 2016. [Google Scholar]
- Ait-Sahalia, Y.; Xiu, D.C. Principal Component Analysis of High-Frequency Data. J. Am. Stat. Assoc. 2019, 114, 287–303. [Google Scholar] [CrossRef] [Green Version]
- Park, S.; Pozzi, A.; Whitmeyer, M.; Perez, H.; Kandel, A.; Kim, G.; Choi, Y.; Joe, W.T.; Raimondo, D.M.; Moura, S. A Deep Reinforcement Learning Framework for Fast Charging of Li-Ion Batteries. IEEE Trans. Transp. Electrif. 2022, 8, 2770–2784. [Google Scholar] [CrossRef]
- Zuo, G.Y.; Du, T.T.; Lu, J.H. Double DQN Method For Object Detection. In Proceedings of the Chinese Automation Congress (CAC), Jinan, China, 20–22 October 2017. [Google Scholar]
- Han, B.A.; Yang, J.J. Research on Adaptive Job Shop Scheduling Problems Based on Dueling Double DQN. IEEE Access 2020, 8, 186474–186495. [Google Scholar] [CrossRef]
- Huang, R.C.; He, H.W.; Zhao, X.Y.; Wang, Y.L.; Li, M.L. Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm. Appl. Energy 2022, 321, 119353. [Google Scholar] [CrossRef]
- Ramdania, D.R.; Irfan, M.; Alfarisi, F.; Nuraiman, D. Comparison of genetic algorithms and Particle Swarm Optimization (PSO) algorithms in course scheduling. J. Phys. Conf. Ser. 2019, 1402, 022079. [Google Scholar] [CrossRef]
Battery | Var | Ske | Kur | |||
---|---|---|---|---|---|---|
PCC | SCC | PCC | SCC | PCC | SCC | |
CH5 | 0.9867 | 0.9789 | 0.9998 | 0.9878 | 0.9764 | 0.9777 |
CH11 | 0.9986 | 0.9686 | 0.9999 | 0.9799 | 0.9915 | 0.9837 |
CH17 | 0.9976 | 0.9966 | 0.9954 | 0.9733 | 0.9824 | 0.9968 |
CH22 | 0.9971 | 0.9974 | 0.9877 | 0.9843 | 0.9962 | 0.9695 |
CH32 | 0.9846 | 0.9732 | 0.9865 | 0.9812 | 0.9921 | 0.9999 |
Variables | Description | Value |
---|---|---|
Discount factor | 0.99 | |
The learning rate of neural network | 0.001 | |
greed strategy, exploration, and exploitation | 0.9 |
Experimental Input | Sequential | PSO | GA | Q−Learning | DDQN | PCA−DDQN |
---|---|---|---|---|---|---|
4.4 A, 2.01 V, 1.06 Ah (cycle = 10) 4 C−2 V | 898 | 903 | 874 | 840 | 879 | 907 |
7.7 A, 3.5 V, 0.19 Ah (cycle = 5) C1−Q1 | 515 | 494 | 526 | 499 | 512 | 506 |
3.3 A, 3.5 V, 0.86 Ah (cycle = 6) C2−80 | 733 | 716 | 682 | 712 | 701 | 719 |
0 A, 3.3 V, 0.88 Ah (cycle = 7) rest | 561 | 502 | 573 | 502 | 549 | 564 |
1.1 A, 3.43 V, 0.94 Ah (cycle = 9) 1 C−3.6 V | 408 | 459 | 372 | 410 | 438 | 453 |
0.00016 A, 3.29 V, 0.88 Ah (cycle = 8) start | 631 | 662 | 597 | 581 | 612 | 659 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Xiong, H.; Chen, J.; Rong, S.; Zhang, A. Power Battery Scheduling Optimization Based on Double DQN Algorithm with Constraints. Appl. Sci. 2023, 13, 7702. https://doi.org/10.3390/app13137702
Xiong H, Chen J, Rong S, Zhang A. Power Battery Scheduling Optimization Based on Double DQN Algorithm with Constraints. Applied Sciences. 2023; 13(13):7702. https://doi.org/10.3390/app13137702
Chicago/Turabian StyleXiong, Haijun, Jingjing Chen, Song Rong, and Aiwen Zhang. 2023. "Power Battery Scheduling Optimization Based on Double DQN Algorithm with Constraints" Applied Sciences 13, no. 13: 7702. https://doi.org/10.3390/app13137702