# Adaptive Online-Learning Volt-Var Control for Smart Inverters Using Deep Reinforcement Learning

^{*}

## Abstract

**:**

## 1. Introduction

- No need for additional equipment: this saves installation effort and costs.
- Reduction of data flows: with a self-learning individual volt-var control there is no need for data exchange with other grid actors.
- Individual application at the point of demand: this enables DGOs to progressively adapt the distribution grid to higher shares of DG feed-in.
- Flexible adaptation to changing environments through online-learning: the ongoing exploration in the learning process allows a continuous adaption to the actual reactive power demand.

## 2. Proposed DRL Volt-Var Control Algorithm

## 3. Simulation Framework

#### 3.1. 21-Bus Test Feeder

#### 3.2. Reactive Power Demand in the Test Feeder

## 4. Simulation Results—Application of the DRL Volt-Var Control Algorithm

#### 4.1. Static Grid Behavior

#### 4.2. Dynamic Grid Behavior

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Ueda, Y.; Kurokawa, K.; Tanabe, T.; Kitamura, K.; Sugihara, H. Analysis Results of Output Power Loss Due to the Grid Voltage Rise in Grid-Connected Photovoltaic Power Generation Systems. IEEE Trans. Ind. Electron.
**2008**, 55, 2744–2751. [Google Scholar] [CrossRef] - Tahir, M.; Nassar, M.E.; El-Shatshat, R.; Salama, M.M.A. A review of Volt/Var control techniques in passive and active power distribution networks. In Proceedings of the 2016 IEEE Smart Energy Grid Engineering (SEGE), Oshawa, ON, Canada, 21–24 August 2016; pp. 57–63. [Google Scholar] [CrossRef]
- Woyte, A.; Thong, V.; Belmans, R.; Nijs, J. Voltage Fluctuations on Distribution Level Introduced by Photovoltaic Systems. IEEE Trans. Energy Convers.
**2006**, 21, 202–209. [Google Scholar] [CrossRef] - Chen, H.; Chen, J.; Shi, D.; Duan, X. Power flow study and voltage stability analysis for distribution systems with distributed generation. In Proceedings of the 2006 IEEE Power Engineering Society General Meeting, Montreal, QC, Canada, 18–22 June 2006; p. 8. [Google Scholar] [CrossRef]
- Dong, F.; Chowdhury, B.; Crow, M.; Acar, L. Improving Voltage Stability by Reactive Power Reserve Management. IEEE Trans. Power Syst.
**2005**, 20, 338–345. [Google Scholar] [CrossRef] - Niknam, T.; Ranjbar, A.; Shinari, A. Impact of distributed generation on volt/var control in distribution networks. In Proceedings of the 2003 IEEE Bologna Power Tech Conference Proceedings, Bologna, Italy, 23–26 June 2003; Volume 3, pp. 210–216. [Google Scholar] [CrossRef]
- Hietpas, S.; Naden, M. Automatic voltage regulator using an AC voltage-voltage converter. IEEE Trans. Ind. Appl.
**2000**, 36, 33–38. [Google Scholar] [CrossRef] - Chamana, M.; Chowdhury, B.H. Optimal Voltage Regulation of Distribution Networks With Cascaded Voltage Regulators in the Presence of High PV Penetration. IEEE Trans. Sustain. Energy
**2018**, 9, 1427–1436. [Google Scholar] [CrossRef] - Singh, B.; Solanki, J. A Comparison of Control Algorithms for DSTATCOM. IEEE Trans. Ind. Electron.
**2009**, 56, 2738–2745. [Google Scholar] [CrossRef] - Smith, J.W.; Sunderman, W.; Dugan, R.; Seal, B. Smart inverter volt/var control functions for high penetration of PV on distribution systems. In Proceedings of the 2011 IEEE/PES Power Systems Conference and Exposition, Phoenix, AZ, USA, 20–23 March 2011; pp. 1–6. [Google Scholar] [CrossRef]
- Malekpour, A.R.; Pahwa, A. Reactive power and voltage control in distribution systems with photovoltaic generation. In Proceedings of the 2012 North American Power Symposium (NAPS), Champaign, IL, USA, 9–11 September 2012; pp. 1–6. [Google Scholar] [CrossRef]
- Demirok, E.; González, P.C.; Frederiksen, K.H.B.; Sera, D.; Rodriguez, P.; Teodorescu, R. Local Reactive Power Control Methods for Overvoltage Prevention of Distributed Solar Inverters in Low-Voltage Grids. IEEE J. Photovoltaics
**2011**, 1, 174–182. [Google Scholar] [CrossRef] - Keane, A.; Ochoa, L.F.; Vittal, E.; Dent, C.J.; Harrison, G.P. Enhanced Utilization of Voltage Control Resources With Distributed Generation. IEEE Trans. Power Syst.
**2011**, 26, 252–260. [Google Scholar] [CrossRef] [Green Version] - Pereira, B.R.; Martins da Costa, G.R.M.; Contreras, J.; Mantovani, J.R.S. Optimal Distributed Generation and Reactive Power Allocation in Electrical Distribution Systems. IEEE Trans. Sustain. Energy
**2016**, 7, 975–984. [Google Scholar] [CrossRef] [Green Version] - Satsangi, S.; Kumbhar, G. Review on Volt/VAr Optimization and Control in Electric Distribution System. In Proceedings of the 2016 IEEE 1st International Conference on Power Electronics, Intelligent Control and Energy Systems (ICPEICES), Delhi, India, 4–6 July 2016; pp. 1–6. [Google Scholar] [CrossRef]
- Manbachi, M.; Farhangi, H.; Palizban, A.; Arzanpour, S. Smart grid adaptive volt-VAR optimization: Challenges for sustainable future grids. Sustain. Cities Soc.
**2017**, 28, 242–255. [Google Scholar] [CrossRef] - Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; Riedmiller, M. Playing Atari with Deep Reinforcement Learning. arXiv
**2013**, arXiv:1312.560. [Google Scholar] - Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. Nature
**2015**, 518, 529–533. [Google Scholar] [CrossRef] - Kober, J.; Bagnell, J.A.; Peters, J. Reinforcement learning in robotics: A survey. Int. J. Robot. Res.
**2013**, 32, 1238–1274. [Google Scholar] [CrossRef] [Green Version] - Gu, S.; Holly, E.; Lillicrap, T.; Levine, S. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017; pp. 3389–3396. [Google Scholar] [CrossRef] [Green Version]
- Cao, D.; Hu, W.; Zhao, J.; Zhang, G.; Zhang, B.; Liu, Z.; Chen, Z.; Blaabjerg, F. Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review. J. Mod. Power Syst. Clean Energy
**2020**, 8, 1029–1042. [Google Scholar] [CrossRef] - Perera, A.; Kamalaruban, P. Applications of reinforcement learning in energy systems. Renew. Sustain. Energy Rev.
**2021**, 137, 110618. [Google Scholar] [CrossRef] - Zhang, D.; Han, X.; Deng, C. Review on the research and practice of deep learning and reinforcement learning in smart grids. CSEE J. Power Energy Syst.
**2018**, 4, 362–370. [Google Scholar] [CrossRef] - Zhang, Z.; Zhang, D.; Qiu, R.C. Deep reinforcement learning for power system: An overview. CSEE J. Power Energy Syst.
**2019**. [Google Scholar] [CrossRef] - Glavic, M. (Deep) Reinforcement learning for electric power system control and related problems: A short review and perspectives. Annu. Rev. Control
**2019**, 48, 22–35. [Google Scholar] [CrossRef] [Green Version] - Wang, W.; Yu, N.; Gao, Y.; Shi, J. Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems. IEEE Trans. Smart Grid
**2020**, 11, 3008–3018. [Google Scholar] [CrossRef] - Yang, Q.; Wang, G.; Sadeghi, A.; Giannakis, G.B.; Sun, J. Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning. IEEE Trans. Smart Grid
**2020**, 11, 2313–2323. [Google Scholar] [CrossRef] [Green Version] - Li, C.; Jin, C.; Sharma, R. Coordination of PV Smart Inverters Using Deep Reinforcement Learning for Grid Voltage Regulation. In Proceedings of the 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), Boca Raton, FL, USA, 16–19 December 2019; pp. 1930–1937. [Google Scholar] [CrossRef] [Green Version]
- Zhang, Y.; Wang, X.; Wang, J.; Zhang, Y. Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems. IEEE Trans. Smart Grid
**2021**, 12, 361–371. [Google Scholar] [CrossRef] - Cao, D.; Hu, W.; Zhao, J.; Huang, Q.; Chen, Z.; Blaabjerg, F. A Multi-Agent Deep Reinforcement Learning Based Voltage Regulation Using Coordinated PV Inverters. IEEE Trans. Power Syst.
**2020**, 35, 4120–4123. [Google Scholar] [CrossRef] - Wang, S.; Duan, J.; Shi, D.; Xu, C.; Li, H.; Diao, R.; Wang, Z. A Data-Driven Multi-Agent Autonomous Voltage Control Framework Using Deep Reinforcement Learning. IEEE Trans. Power Syst.
**2020**, 35, 4644–4654. [Google Scholar] [CrossRef] - Lillicrap, T.P.; Hunt, J.J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; Wierstra, D. Continuous control with deep reinforcement learning. arXiv
**2015**, arXiv:1509.02971. [Google Scholar] - Liu, H.; Wu, W. Two-stage Deep Reinforcement Learning for Inverter-based Volt-VAR Control in Active Distribution Networks. IEEE Trans. Smart Grid
**2020**. [Google Scholar] [CrossRef] - Silver, D.; Lever, G.; Heess, N.; Degris, T.; Wierstra, D.; Riedmiller, M. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning (ICML 2014), Beijing, China, 21–26 June 2014; Volume 1, pp. 605–619. [Google Scholar]
- Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; Zaremba, W. OpenAI Gym. arXiv
**2016**, arXiv:1606.01540v1. [Google Scholar] - Plappert, M. keras-rl. Github Repos. 2016. Available online: https://github.com/keras-rl/keras-rl (accessed on 19 February 2021).
- Köppl, S.; Bruckmeier, A.; Böning, F.; Hinterstocker, M.; Kleinertz, B.; Konetschny, C.; Mueller, M.; Samweber, F.; Schmid, T.; Zeiselmair, A. Projekt MONA 2030: Grundlage für die Bewertung von Netzoptimierenden Maßnahmen: Teilbericht Basisdaten; Forschungsstelle für Energiewirtschaft e.V. (FfE): München, Germany, 2017. [Google Scholar]
- Dassault Systèmes. FMIKit for Simulink. 2020. Available online: https://github.com/CATIA-Systems/FMIKit-Simulink (accessed on 19 February 2021).
- Tjaden, T.; Joseph, B.; Weniger, J.; Quaschning, V. Repräsentative Elektrische Lastprofile für Einfamilienhäuser in Deutschland auf 1-Sekündiger Datenbasis; Technical Report; Hochschule für Technik und Wirtschaft Berlin: Berlin, Germany, 2015. [Google Scholar] [CrossRef]
- Weniger, J.; Quaschning, V. Begrenzung der Einspeiseleistung von netzgekoppelten Photovoltaiksystemen mit Batteriespeichern. In Proceedings of the 28. Symposium Photovoltaische Solarenergie, Staffelstein, Germany, 6–8 March 2013. [Google Scholar]

**Figure 6.**Reactive power output from the deep reinforcement learning (DRL) volt-var control (VVC) at the nodes N1, N5, and N10 after 80,000 training steps together with the calculated reactive power demand as a reference.

**Figure 7.**Moving average rewards over 1000 successive learning steps over the number of steps during static learning application of the proposed DRL VVC algorithm.

**Figure 8.**Voltage topology in the test feeder with reactive power injection at node N10 (red circle), the transformer is circled in black, the households in gray.

**Figure 9.**Voltage along the line to nodes N1, N5, and N10 without reactive power feed-in and with DRL VVC at the considered node.

Parameter | Value |
---|---|

Actor | 5 layer à 32 nodes |

Critic | 6 layer à 64 nodes |

activation function | ReLu, output layer: linear |

learning rate $\alpha $ | 0.001 |

target model update $\tau $ | 0.001 |

discout factor $\gamma $ | 1 |

random process | Ohrnstein-Uhlenbeck process ($\theta =0.01$, $\sigma =0.01$) |

warmup steps | 10,000 |

memory | 100,000 |

**Table 2.**Load and photovoltaic (PV) profile numbers and input values for the dynamic and static case studies. For the static case, the data from 21 June, 11.06 h 40 s was used.

Node N | Load Profile No. | PV Factor/kWp | P${}_{\mathit{load}}$/W (Static) | P${}_{\mathit{PV}}$/W (Static) |
---|---|---|---|---|

N1 | 1 | 6.5 | 53 | 1932 |

N2 | 2 | 4.9 | 567 | 1456 |

N3 | 3 | 3.4 | 169 | 1010 |

N4 | 4 | 5.8 | 393 | 1723 |

N5 | 5 | 2.7 | 54 | 802 |

N6 | 6 | 1.5 | 11 | 446 |

N7 | 7 | 6.7 | 16 | 1991 |

N8 | 8 | 7.9 | 747 | 2348 |

N9 | 9 | 4.6 | 186 | 1367 |

N10 | 10 | 6.9 | 121 | 2050 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Beyer, K.; Beckmann, R.; Geißendörfer, S.; von Maydell, K.; Agert, C.
Adaptive Online-Learning Volt-Var Control for Smart Inverters Using Deep Reinforcement Learning. *Energies* **2021**, *14*, 1991.
https://doi.org/10.3390/en14071991

**AMA Style**

Beyer K, Beckmann R, Geißendörfer S, von Maydell K, Agert C.
Adaptive Online-Learning Volt-Var Control for Smart Inverters Using Deep Reinforcement Learning. *Energies*. 2021; 14(7):1991.
https://doi.org/10.3390/en14071991

**Chicago/Turabian Style**

Beyer, Kirstin, Robert Beckmann, Stefan Geißendörfer, Karsten von Maydell, and Carsten Agert.
2021. "Adaptive Online-Learning Volt-Var Control for Smart Inverters Using Deep Reinforcement Learning" *Energies* 14, no. 7: 1991.
https://doi.org/10.3390/en14071991