A Deep Reinforcement Learning-Based MPPT Control for PV Systems under Partial Shading Condition
Abstract
:1. Introduction
- Two proposed efficient and robust MPPT controllers for PV systems based on DRL are proposed and simulated in MATLAB/Simulink, including DQN and DDPG.
- Eight scenarios under different weather conditions are considered for testing the performances of the two proposed methods. They are divided into four scenarios under uniform conditions and four other scenarios under partial shading conditions, as shown in Table 3.
- A comparison between the proposed method and the P&O method is also investigated.
2. Modelling of PV Module under PSC
2.1. Mathematical Model of PV Module
2.2. Partial Shading System Effect
2.3. PV System Introduction
3. Deep Reinforcement Learning based MPPT Control
3.1. Basic Concept of DRL
3.2. Markov Decision Process Model of a PV System
3.3. Methodology of the DQN MPPT Control
3.4. Methodology of the DDPG MPPT Control
4. Simulation and Results
4.1. Simulation Set up
4.2. Training Results and Performance under STC
4.3. Performance under Varying Operating Conditions
4.4. Performance under PSC
5. Conclusions
Supplementary Materials
Author Contributions
Funding
Conflicts of Interest
Nomenclature
G | Irradiation () |
T | Temperature () |
q | Electronic charge (C) |
k | Boltzmann’s constant |
A policy | |
Value function | |
J | Objective function |
L | Loss function |
Action-value function | |
a | Action |
r | Reward |
s | State |
Weight matrix | |
Discount factor | |
Exploration rate | |
I | Current (A) |
V | Voltage (V) |
P | Power (W) |
References
- Lin, C.E.; Phan, B.C. Optimal Hybrid Energy Solution for Island Micro-Grid. In Proceedings of the 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) (BDCloud-SocialCom-SustainCom), Atlanta, GA, USA, 8–10 October 2016; pp. 461–468. [Google Scholar]
- Belhachat, F.; Larbes, C. A review of global maximum power point tracking techniques of photovoltaic system under partial shading conditions. Renew. Sustain. Energy Rev. 2018, 92, 513–553. [Google Scholar] [CrossRef]
- Ramli, M.A.M.; Twaha, S.; Ishaque, K.; Al-Turki, Y.A. A review on maximum power point tracking for photovoltaic systems with and without shading conditions. Renew. Sustain. Energy Rev. 2017, 67, 144–159. [Google Scholar] [CrossRef]
- Rezk, H.; Fathy, A.; Abdelaziz, A.Y. A comparison of different global MPPT techniques based on meta-heuristic algorithms for photovoltaic system subjected to partial shading conditions. Renew. Sustain. Energy Rev. 2017, 74, 377–386. [Google Scholar] [CrossRef]
- Danandeh, M.A.; Mousavi, G.S.M. Comparative and comprehensive review of maximum power point tracking methods for PV cells. Renew. Sustain. Energy Rev. 2018, 82, 2743–2767. [Google Scholar] [CrossRef]
- Karami, N.; Moubayed, N.; Outbib, R. General review and classification of different MPPT Techniques. Renew. Sustain. Energy Rev. 2017, 68, 1–18. [Google Scholar] [CrossRef]
- Mohapatra, A.; Nayak, B.; Das, P.; Mohanty, K.B. A review on MPPT techniques of PV system under partial shading condition. Renew. Sustain. Energy Rev. 2017, 80, 854–867. [Google Scholar] [CrossRef]
- Ahmed, J.; Salam, Z. An improved perturb and observe (P&O) maximum power point tracking (MPPT) algorithm for higher efficiency. Appl. Energy 2015, 150, 97–108. [Google Scholar]
- Al-Majidi, S.D.; Abbod, M.F.; Al-Raweshidy, H.S. A novel maximum power point tracking technique based on fuzzy logic for photovoltaic systems. Int. J. Hydrogen Energy 2018, 43, 14158–14171. [Google Scholar] [CrossRef]
- Kassem, A.M. MPPT control design and performance improvements of a PV generator powered DC motor-pump system based on artificial neural networks. Int. J. Electr. Power Energy Syst. 2012, 43, 90–98. [Google Scholar] [CrossRef]
- Belhachat, F.; Larbes, C. Global maximum power point tracking based on ANFIS approach for PV array configurations under partial shading conditions. Renew. Sustain. Energy Rev. 2017, 77, 875–889. [Google Scholar] [CrossRef]
- Mumtaz, S.; Ahmad, S.; Khan, L.; Ali, S.; Kamal, T.; Hassan, S. Adaptive Feedback Linearization Based NeuroFuzzy Maximum Power Point Tracking for a Photovoltaic System. Energies 2018, 11, 606. [Google Scholar] [CrossRef] [Green Version]
- Shaiek, Y.; Smida, M.B.; Sakly, A.; Mimouni, M.F. Comparison between conventional methods and GA approach for maximum power point tracking of shaded solar PV generators. Sol. Energy 2013, 90, 107–122. [Google Scholar] [CrossRef]
- Ahmed, J.; Salam, Z. A Maximum Power Point Tracking (MPPT) for PV system using Cuckoo Search with partial shading capability. Appl. Energy 2014, 119, 118–130. [Google Scholar] [CrossRef]
- Titri, S.; Larbes, C.; Toumi, K.Y.; Benatchba, K. A new MPPT controller based on the Ant colony optimization algorithm for Photovoltaic systems under partial shading conditions. Appl. Soft Comput. 2017, 58, 465–479. [Google Scholar] [CrossRef]
- Benyoucef, A.S.; Chouder, A.; Kara, K.; Silvestre, S.; Sahed, O.A. Artificial bee colony based algorithm for maximum power point tracking (MPPT) for PV systems operating under partial shaded conditions. Appl. Soft Comput. 2015, 32, 38–48. [Google Scholar] [CrossRef] [Green Version]
- Kaced, K.; Larbes, C.; Ramzan, N.; Bounabi, M.; Dahmane, Z.E. Bat algorithm based maximum power point tracking for photovoltaic system under partial shading conditions. Sol. Energy 2017, 158, 490–503. [Google Scholar] [CrossRef] [Green Version]
- Yang, B.; Zhong, L.; Zhang, X.; Chun, H.; Yu, T.; Li, H.; Jiang, L.; Sun, L. Novel bio-inspired memetic salp swarm algorithm and application to MPPT for PV systems considering partial shading condition. J. Cleaner Prod. 2019, 215, 1203–1222. [Google Scholar] [CrossRef]
- Jiang, L.L.; Srivatsan, R.; Maskell, D.L. Computational intelligence techniques for maximum power point tracking in PV systems: A review. Renewable Sustainable Energy Rev. 2018, 85, 14–45. [Google Scholar] [CrossRef]
- Koad, R.B.A.; Zobaa, A.F.; El-Shahat, A. A Novel MPPT Algorithm Based on Particle Swarm Optimization for Photovoltaic Systems. IEEE Trans. Sustain. Energy 2017, 8, 468–476. [Google Scholar] [CrossRef] [Green Version]
- Suryavanshi, R.; Joshi, D.R.; Jangamshetti, S.H. PSO and P&O based MPPT technique for SPV panel under varying atmospheric conditions. In Proceedings of the 2012 International Conference on Power, Signals, Controls and Computation, Thrissur, Kerala, India, 3–6 January 2012. [Google Scholar]
- Garg, H. A hybrid PSO-GA algorithm for constrained optimization problems. Appl. Math. Comput. 2016, 274, 292–305. [Google Scholar] [CrossRef]
- Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction, 2nd ed.; MIT Press: Cambridge, MA, USA, 2018. [Google Scholar]
- Glavic, M. (Deep) Reinforcement learning for electric power system control and related problems: A short review and perspectives. Annu. Rev. Control 2019, 48, 22–35. [Google Scholar] [CrossRef] [Green Version]
- Kofinas, P.; Doltsinis, S.; Dounis, A.I.; Vouros, G.A. A reinforcement learning approach for MPPT control method of photovoltaic sources. Renew. Energy 2017, 108, 461–473. [Google Scholar] [CrossRef]
- Wei, C.; Zhang, Z.; Qiao, W.; Qu, L. Reinforcement-Learning-Based Intelligent Maximum Power Point Tracking Control for Wind Energy Conversion Systems. IEEE Trans. Ind. Electron. 2015, 62, 6360–6370. [Google Scholar] [CrossRef]
- Nambiar, A.; Anderlini, E.; Payne, G.; Forehand, D.; Kiprakis, A.; Wallace, A. Reinforcement Learning Based Maximum Power Point Tracking Control of Tidal Turbines. In Proceedings of the 12th European Wave and Tidal Energy Conference, Cork, Ireland, 27 August–September 2017. [Google Scholar]
- Hsu, R.; Liu, C.T.; Chen, W.Y.; Hsieh, H.-I.; Wang, H.L. A Reinforcement Learning-Based Maximum Power Point Tracking Method for Photovoltaic Array. Int. J. Photoenergy 2015, 2015. [Google Scholar] [CrossRef]
- Youssef, A.; Telbany, M.E.; Zekry, A. Reinforcement Learning for Online Maximum Power Point Tracking Control. J. Clean Energy Technol. 2016, 4, 245–248. [Google Scholar] [CrossRef] [Green Version]
- Phan, B.C.; Lai, Y.-C. Control Strategy of a Hybrid Renewable Energy System Based on Reinforcement Learning Approach for an Isolated Microgrid. Appl. Sci. 2019, 9, 4001. [Google Scholar] [CrossRef] [Green Version]
- Chou, K.-Y.; Yang, S.-T.; Chen, Y.-P. Maximum Power Point Tracking of Photovoltaic System Based on Reinforcement Learning. Sensors 2019, 19, 5054. [Google Scholar] [CrossRef] [Green Version]
- Zhang, X.; Li, X.; He, T.; Yang, B.; Yu, T.; Li, H.; Jiang, L.; Sun, L. Memetic reinforcement learning based maximum power point tracking design for PV systems under partial shading condition. Energy 2019, 174, 1079–1090. [Google Scholar] [CrossRef]
- Dong, M.; Li, D.; Yang, C.; Li, S.; Fang, Q.; Yang, B.; Zhang, X. Global Maximum Power Point Tracking of PV Systems under Partial Shading Condition: A Transfer Reinforcement Learning Approach. Appl. Sci. 2019, 9, 2769. [Google Scholar] [CrossRef] [Green Version]
- Lapan, M. Deep Reinforcement Learning Hands-On: Apply Modern RL Methods, with Deep Q-Networks, Value Iteration, Policy Gradients, TRPO, AlphaGo Zero and More; Packt Publishing Ltd.: Birmingham, UK, 2018. [Google Scholar]
- Gu, S.; Holly, E.; Lillicrap, T.; Levine, S. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Marina Bay Sands, Singapore, 29 May–2 June 2017; pp. 3389–3396. [Google Scholar]
- Lillicrap, T.P.; Hunt, J.J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; Wierstra, D. Continuous control with deep reinforcement learning. arXiv Preprint 2015, arXiv:1509.02971. [Google Scholar]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, L.; Wierstra, D.; Riedmiller, M. Playing atari with deep reinforcement learning. arXiv Preprint 2013, arXiv:1312.5602. [Google Scholar]
- Kahn, G.; A Villaflor, B.D.; Abbeel, P.; Levine, S. Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 20–25 May 2018; pp. 1–8. [Google Scholar]
- He, J.; Chen, J.; He, X.; Gao, J.; Li, L.; Deng, L.; Ostendorf, M. Deep reinforcement learning with a natural language action space. arXiv Preprint 2015, arXiv:1511.04636. [Google Scholar]
- Mohamed Shakeel, P.; Baskar, S.; Dhulipala, V.R.S.; Mishra, S.; Jaber, M.M. Maintaining Security and Privacy in Health Care System Using Learning Based Deep-Q-Networks. J. Med. Syst. 2018, 42, 186. [Google Scholar] [CrossRef] [PubMed]
- Zhang, D.; Han, X.; Deng, C. Review on the research and practice of deep learning and reinforcement learning in smart grids. CSEE J. Power Energy Syst. 2018, 4, 362–370. [Google Scholar] [CrossRef]
- Zhang, Z.; Zhang, D.; Qiu, R.C. Deep reinforcement learning for power system: An. overview. CSEE J. Power Energy Syst. 2019, 1–12. [Google Scholar]
- Wei, C.; Zhang, Z.; Qiao, W.; Qu, L. An Adaptive Network-Based Reinforcement Learning Method for MPPT Control of PMSG Wind Energy Conversion Systems. IEEE Trans. Power Electron. 2016, 31, 7837–7848. [Google Scholar] [CrossRef]
- Saenz-Aguirre, A.; Zulueta, E.; Fernandez-Gamiz, U.; Lozano, J.; Lopez-Guede, J. Artificial Neural Network Based Reinforcement Learning for Wind Turbine Yaw Control. Energies 2019, 12, 436. [Google Scholar] [CrossRef] [Green Version]
- Ram, J.P.; Babu, T.S.; Rajasekar, N. A comprehensive review on solar PV maximum power point tracking techniques. Renewable Sustainable Energy Rev. 2017, 67, 826–847. [Google Scholar] [CrossRef]
- Bendib, B.; Belmili, H.; Krim, F. A survey of the most used MPPT methods: Conventional and advanced algorithms applied for photovoltaic systems. Renewable Sustainable Energy Rev. 2015, 45, 637–648. [Google Scholar] [CrossRef]
- Mirza, A.F.; Ling, Q.; Javed, M.Y.; Mansoor, M. Novel MPPT techniques for photovoltaic systems under uniform irradiance and Partial shading. Sol. Energy 2019, 184, 628–648. [Google Scholar] [CrossRef]
- Prasanth Ram, J.; Rajasekar, N. A new global maximum power point tracking technique for solar photovoltaic (PV) system under partial shading conditions (PSC). Energy 2017, 118, 512–525. [Google Scholar] [CrossRef]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Venes, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef] [PubMed]
- Casas, N. Deep deterministic policy gradient for urban traffic light control. arXiv Preprint 2017, arXiv:1703.09035. [Google Scholar]
- Li, Y. Deep Reinforcement Learning: An Overview. arXiv Preprint 2018, arXiv:1810.06339. [Google Scholar]
- Wu, J.; He, H.; Peng, J.; Li, Y.; Li, Z. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus. Appl. Energy 2018, 222, 799–811. [Google Scholar] [CrossRef]
- Fan, J.; Wang, Z.; Xie, Y.; Yang, Z. A Theoretical Analysis of Deep Q-Learning. arXiv Preprint 2019, arXiv:1901.00137v3. [Google Scholar]
- Wu, Y.; Tan, H.; Peng, J.; Zhang, H.; He, H. Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus. Appl. Energy 2019, 247, 454–466. [Google Scholar] [CrossRef]
Specifications | Value |
---|---|
Maximum Power (W) | 334.905 |
Voltage at MPP (V) | 41.5 |
Current at MPP (A) | 8.07 |
Open circuit voltage, Voc (V) | 49.9 |
Short circuit current, Isc (A) | 9 |
Temperature coefficient of Voc (%/°C) | −0.36 |
Temperature coefficient of Isc (%/°C) | 0.09 |
Specifications | Value |
---|---|
Replay memory size | |
Batch size | 512 |
Discount factor () | 0.9 |
DQN | |
Exploration rate () | 1 |
Decay of exploration rate | 0.0001 |
Exploration rate minimum () | 0.001 |
DDPG | |
Initial variance | 0.4 |
Decay of initial variance | 0.0001 |
Smoothing factor () | 0.001 |
Scenarios | Weather Conditions | DQN | DDPG |
---|---|---|---|
1 | Uniform with 1000 | 5.83% | 3.21% |
2 | G changes | 1.24% | 0.96% |
3 | T changes | 2.74% | 2.55% |
4 | Both T and G change | 1.62% | 1.58% |
5 | 900,900,350 | 38.3% | 44.6% |
6 | 900,350,300 | 25.9% | 22.1% |
7 | 500,800,600 | 0.56% | 0.92% |
8 | 900,300,250 | 17.9% | 15.4% |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Phan, B.C.; Lai, Y.-C.; Lin, C.E. A Deep Reinforcement Learning-Based MPPT Control for PV Systems under Partial Shading Condition. Sensors 2020, 20, 3039. https://doi.org/10.3390/s20113039
Phan BC, Lai Y-C, Lin CE. A Deep Reinforcement Learning-Based MPPT Control for PV Systems under Partial Shading Condition. Sensors. 2020; 20(11):3039. https://doi.org/10.3390/s20113039
Chicago/Turabian StylePhan, Bao Chau, Ying-Chih Lai, and Chin E. Lin. 2020. "A Deep Reinforcement Learning-Based MPPT Control for PV Systems under Partial Shading Condition" Sensors 20, no. 11: 3039. https://doi.org/10.3390/s20113039
APA StylePhan, B. C., Lai, Y.-C., & Lin, C. E. (2020). A Deep Reinforcement Learning-Based MPPT Control for PV Systems under Partial Shading Condition. Sensors, 20(11), 3039. https://doi.org/10.3390/s20113039