# A Frequency–Pressure Cooperative Control Strategy of Multi-Microgrid with an Electric–Gas System Based on MADDPG

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- (1)
- A frequency–pressure cooperative control structure of multi-microgrid with electric-gas system is proposed. Based on the analysis of the operating characteristics of the natural gas network and the coupling equipment, the natural gas network model and the P2G system coupling model are established. The coordinated transformation of the electrical coupling system in the two modes of the gas determined by power (GDP) and the power determined by gas (PDG) is realized, which provides a model basis for the coordinated control of frequency and natural gas pressure.
- (2)
- In order to coordinate power transmission between multiple microgrids and realize coordination and cooperation, the MADDPG algorithm with the cooperative control idea of “centralized training, decentralized execution” is proposed to design the frequency controller for the multi-microgrid. Moreover, on the basis of the two areas of the IEEE standard, the area control error (ACE) is used as one of the state spaces in the design process of the controller so as to obtain the optimal solution of multi-microgrid coordination quickly and accurately in the control process.
- (3)
- In order to coordinate the control of the frequency of the microgrid and the air pressure of the natural gas network, the structure and the reward function of the MADDPG controller is designed according to the two control objectives in order for it to meet the two control goals of the frequency of the microgrid and the air pressure of the natural gas network together.

## 2. The Model of the Power-to-Gas System

#### 2.1. The Model of Natural Gas Network

_{in}and p

_{out}are the air pressure at the inlet and outlet of the natural gas pipeline, M

_{in}, M

_{out}are the flow at the inlet and outlet of the pipeline, c is the sound propagation speed in the natural gas pipeline, D is the diameter of the pipeline, S is the cross-sectional area of the pipeline, L is the length of the pipeline, f is the friction coefficient of the pipeline, ρ is the density of natural gas, ω is the flow rate of natural gas in the pipeline, and x is the distance.

_{in}, p

_{out}, M

_{in}and M

_{out}. For a section of natural gas pipeline, when two variables are given, the values of the other two variables can be solved.

_{r}is the proportional control coefficient in the pressure regulating valve, and p

_{out}

_{_ref}is the air pressure reference value at the outlet of the pressure regulating valve.

#### 2.2. Coupling Relationship of Power-to-Gas System

_{DDPG}is the control signal sent by the agent DDPG, ∆X

_{MT}is the state quantity that characterizes the valve position change of the fuel system, T

_{f}and T

_{t}are the time constants of the fuel system and the gas turbine respectively, ±δ

_{MT}is the power climbing constraint of the micro gas turbine, ±μ

_{MT}is the power variation constraint. The working mode of the micro gas turbine control system is adjusted by the DDPG controller.

## 3. Load Frequency Control Model of Multi-Microgrid

#### 3.1. Load Frequency Control Model of P2G Equipment

_{P2G}is the load frequency control signal received by the P2G equipment, T

_{ele}is the time constant of the electrolytic cell, ±δ

_{P2G}is the upper and lower limits of the power change rate constraint, ±μ

_{P2G}is the upper and lower limits of the power increment constraint, and ∆P

_{P2G}is the power increment of the P2G device.

#### 3.2. Load Frequency Control Structure of a Multi-Microgrid Based on MADDPG

_{L}is the load disturbance power, ΔP

_{W}is the wind disturbance power, ΔP

_{MT}is the power variation of MT, ΔP

_{EV}is the power variation of EVs, ΔP

_{P2G}is the power variation of P2G, ΔP

_{line}is the power variation of tie-line, H

_{t}is the constant of inertia of the microgrid, T

_{sij}is the coupling link parameter, ACE is the area control error, which is a signal used to coordinate the two controllers.

## 4. Load Frequency Controller of Islanded Microgrid Based on MADDPG

#### 4.1. Theoretical Analysis of DDPG

_{t}by the ACN. The action a

_{t}

_{+1}at the t+1 time can be generated according to the subsequent state of the environment by the ATN. The value R

_{t}corresponding to the status s

_{t}and action a

_{t}can be calculated by the CCN. The value of Q’ (s

_{t}

_{+1}, a

_{t}

_{+1}|ω’), which is used to calculate the target value y, can be generated by the CTN based on subsequent state s

_{t+1}and action a

_{t}

_{+1}, as shown in the Formula (8):

_{j}is the target value of the j sample, Q(s

_{j},a

_{j},ω) is the output value of the CCN for the j sample, π

_{θ}(⋅) is the output value of the ACN.

_{m}.

#### 4.2. Theoretical Analysis of MADDPG

- (1)
- State S, which can describe all possible configurations of all agents;
- (2)
- Actions A
_{1},…,A_{N}of each agent; - (3)
- Observations O
_{1},…,O_{N}of each agent.

_{i}is the reward value of each agent.

_{1}, s

_{2},…,s

_{n}) and global actions (a

_{1}, a

_{2},…,a

_{n}); Actor (policy network) can only make actions based on local observations. The learning process of MADDPG is similar to the above-mentioned DDPG algorithm, and its objective function is shown in Formula (13) [32]:

#### 4.3. Definition of State Space and Action Space

_{i}(Δa

_{EVi}, Δa

_{MTi}, Δa

_{P2Gi}) to each unit in the multi-microgrid so as to control the output power of MT, EV and P2G in multi-microgrid, and achieve rapid suppression of frequency fluctuations and pressure deviations.

_{MT}(t), ∆A

_{EV}(t), ∆A

_{P2G}(t)]

#### 4.4. Design of Reward Function

_{i}is the global reward in MG i, r

_{f}is the reward of microgrid frequency, r

_{p}is the reward of air pressure, μ

_{1}, μ

_{2}, μ

_{3}and μ

_{4}are the parameters of each control area in the frequency reward r

_{f}, and δ

_{1}, δ

_{2}, δ

_{3}and δ

_{4}and are the parameters of each control area in the air pressure reward r

_{p}, the ACE(t) represents the instantaneous value of ACE at time t. η represents the weight of ACE, which is 0.5 in this paper. And the convergence effect and the learning speed can be affected by the size of the reward value, so it is necessary to perform simulation tests according to actual examples, and the specific process would be discussed in next section.

## 5. Simulation Results

#### 5.1. Pre-Learning Stage

_{φ(s,a)}. The training process is shown in Figure 14.

#### 5.2. Case Study

#### 5.2.1. Case 1: The Response of Microgrid Load and Wind Power Combined Disturbance

#### 5.2.2. Case 2: The Response of Natural Gas Load Disturbance

## 6. Conclusions

- The model of P2G system with MT and P2G equipment as the coupling device is constructed, and the great interaction characteristics between the microgrid and the natural gas network are proved through simulation. The MADDPG algorithm with the cooperative regulation idea of “centralized training and decentralized execution” can meet the coordinated control of multi-microgrid.
- Compared with traditional controller, the MADDPG controller with the ability of online learning and experience playback can more effectively deal with the random disturbance of the multi-microgrid, the |Δf| can be limited in 0.03 Hz, and the excellent rate can reach 100%, which is significantly better than the PI controller and Fuzzy controller.
- The MADDPG controller can greatly coordinate the frequency recovery and air pressure adjustment of the multi-microgrid. When the node air pressure of the natural gas network changes, the MADDPG controller can ensure that the |Δf| of the microgrid is kept within 0.03 Hz, and quickly bring the node air pressure deviation close to 0.

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Chu, S.; Majumdar, A. Opportunities and challenges for a sustainable energy future. Nature
**2012**, 488, 294–303. [Google Scholar] [CrossRef] - Li, C. Proposal and parametric analysis of an innovative natural gas pressure reduction and liquefaction system for efficient exergy recovery and LNG storage. Energy
**2021**, 233, 120022. [Google Scholar] [CrossRef] - Qin, G.; Zhang, M.; Yan, Q.; Xu, C.; Kammen, D.M. Comprehensive evaluation of regional energy internet using a fuzzy analytic hierarchy process based on cloud model: A case in China. Energy
**2021**, 1, 120569. [Google Scholar] [CrossRef] - Li, Y.; Zhang, F.; Li, Y.; Wang, Y. An improved two-stage robust optimization model for CCHP-P2G microgrid system considering multi-energy operation under wind power outputs uncertainties. Energy
**2021**, 223, 120048. [Google Scholar] [CrossRef] - Wang, C.; Wang, S.; Liu, F.; Bi, T.; Wang, T. Risk-loss coordinated admissibility assessment of wind generation for integrated electric-gas systems. IEEE Trans. Smart Grid
**2020**, 11, 4454–4465. [Google Scholar] [CrossRef] - Martinez-Mares, A.; Fuerte-Esquivel, C.R. A unified gas and power flow analysis in natural gas and electricity coupled networks. IEEE Trans. Power Syst.
**2012**, 27, 2156–2166. [Google Scholar] [CrossRef] - Bao, Z.; Ye, Y.; Wu, L. Multi-timescale coordinated schedule of interdependent electricity-natural gas systems considering electricity grid steady-state and gas network dynamics. Int. J. Electr. Power Energy Syst.
**2019**, 118, 105763. [Google Scholar] [CrossRef] - Zhang, Z.; Wang, C.; Yang, M.; Chen, X.; Lv, H. Day-ahead optimal dispatch for integrated energy system considering power-to-gas and dynamic pipeline networks. IEEE Trans. Ind. Appl.
**2021**, 99, 1. [Google Scholar] [CrossRef] - Chaudry, M.; Jenkins, N.; Strbac, G. Multi-time period combined gas and electricity network optimisation. Electr. Power Syst. Res.
**2008**, 78, 1265–1279. [Google Scholar] [CrossRef] - Liu, Y.; Liu, T. Research on system planning of gas-power integrated system based on improved two-stage robust optimization and non-cooperative game method. IEEE Access
**2021**, 9, 79169–79181. [Google Scholar] [CrossRef] - Zhong, Y.; Xie, W.; Zhang, X. A neural network compound control algorithm for complex nonlinear electric gas pressure regulating system. In 2016 Chinese Control and Decision Conference (CCDC); IEEE: Piscataway, NJ, USA, 2016; pp. 3055–3060. [Google Scholar]
- Rokrok, E.; Shafie-Khah, M.; Catalão, J.P. Review of primary voltage and frequency control methods for inverter-based islanded microgrids with distributed generation. Renew. Sustain. Energy Rev.
**2018**, 82, 3225–3235. [Google Scholar] [CrossRef] - Bevrani, H.; Feizi, M.R.; Ataee, S. Robust frequency control in an islanded microgrid: H∞ and μ-Synthesis approaches. IEEE Trans. Smart Grid.
**2015**, 7, 706–717. [Google Scholar] [CrossRef] [Green Version] - Jiang, Y.; Guo, L. Research on wind power accommodation for an electricity-heat-gas integrated microgrid system with power-to-gas. IEEE Access
**2019**, 7, 87118–87126. [Google Scholar] [CrossRef] - Masuta, T.; Yokoyama, A. Supplementary load frequency control by use of a number of both electric vehicles and heat pump water heaters. IEEE Trans. Smart Grid
**2012**, 3, 1253–1262. [Google Scholar] [CrossRef] - Long, B.; Liao, Y.; Chong, K.T.; Rodriguez, J.; Guerrero, J.M. Enhancement of frequency regulation in ac microgrid: A fuzzy-MPC controlled virtual synchronous generator. IEEE Trans. Smart Grid
**2021**, 12, 3138–3149. [Google Scholar] [CrossRef] - Wang, X.; Yang, J.; Chen, L.; He, J. Application of liquid hydrogen with SMES for efficient use of renewable energy in the energy internet. Energies
**2017**, 10, 185. [Google Scholar] [CrossRef] [Green Version] - Vachirasricirikul, S.; Ngamroo, I. Robust controller design of microturbine and electrolyzer for frequency stabilization in a microgrid system with plug-in hybrid electric vehicles. Int. J. Electr. Power Energy Syst.
**2012**, 43, 804–811. [Google Scholar] [CrossRef] - Kelkoul, B.; Boumediene, A. Stability analysis and study between classical sliding mode control (SMC) and super twisting algorithm (STA) for doubly fed induction generator (DFIG) under wind turbine. Energy
**2020**, 214, 118871. [Google Scholar] [CrossRef] - Rezaei, N.; Mazidi, M.; Gholami, M.; Mohiti, M. A new stochastic gain adaptive energy management system for smart microgrids considering frequency responsive loads. Energy Rep.
**2020**, 6, 914–932. [Google Scholar] [CrossRef] - Zhang, Y.; Fu, L.; Zhu, W.; Bao, X.; Liu, C. Robust model predictive control for optimal energy management of island microgrids with uncertainties. Energy
**2018**, 164, 1229–1241. [Google Scholar] [CrossRef] - Salehi, J.; Namvar, A.; Gazijahani, F.S.; Shafie-khah, M.; Catalão, J.P. Effect of power-to-gas technology in energy hub optimal operation and gas network congestion reduction. Energy
**2022**, 240, 122835. [Google Scholar] [CrossRef] - Zhang, Y.; Yang, J.; Pan, X.; Zhu, X.; Zhan, X.; Li, G.; Liu, S. Data-driven robust dispatch for integrated electric-gas system considering the correlativity of wind-solar output. Int. J. Electr. Power Energy Syst.
**2022**, 134, 107454. [Google Scholar] [CrossRef] - Wu, X.; Zhang, Y.; Arulampalam, A.; Jenkins, N. Electrical stability of large scale integration of micro generation into low voltage grids. Int. J. Electron.
**2005**, 1, 299–320. [Google Scholar] - Goerguen, H. Dynamic modelling of a proton exchange membrane (PEM) electrolyzer. Int. J. Hydrog. Energy
**2006**, 31, 29–38. [Google Scholar] [CrossRef] - Yang, J.; Zeng, Z.; Tang, Y.; Yan, J.; He, H.; Wu, Y. Load frequency control in isolated micro-grids with electrical vehicles based on multivariable generalized predictive theory. Energies
**2015**, 8, 2145–2164. [Google Scholar] [CrossRef] - Fan, P.; Ke, S.; Kamel, S.; Yang, J.; Li, Y.; Xiao, J.; Xu, B.; Rashed, G.I. A frequency and voltage coordinated control strategy of island microgrid including electric vehicles. Electronics
**2021**, 11, 17. [Google Scholar] [CrossRef] - Rao, Y.; Yang, J.; Xiao, J.; Xu, B.; Liu, W.; Li, Y. A frequency control strategy for multimicrogrids with V2G based on the improved robust model predictive control. Energy
**2021**, 222, 119963. [Google Scholar] [CrossRef] - Huang, L.; Fu, M.; Qu, H.; Wang, S.; Hu, S. A deep reinforcement learning-based method applied for solving multi-agent defense and attack problems. Expert Syst. Appl.
**2021**, 176, 114896. [Google Scholar] [CrossRef] - Yang, Q.; Zhu, Y.; Zhang, J.; Qiao, S.; Liu, J. UAV air combat autonomous maneuver decision based on ddpg algorithm. In Proceedings of the 2019 IEEE 15th International Conference on Control and Automation (ICCA) IEEE, Edinburgh, Scotland, 16–19 July 2019. [Google Scholar]
- Li, Y.; Ma, G.; Yang, J.; Wang, H.; Feng, J.; Ma, Y. Dynamic equivalent modeling for power converter based on LSTM neural network in wide operating range. Energy Rep.
**2021**, 7, 477–484. [Google Scholar] [CrossRef] - Yang, Z.; Wang, J.; Gao, S.; Zhou, D. A method to estimate speed droop ratios for primary frequency control responses in power generation units. Int. J. Electr. Power Energy Syst.
**2020**, 119, 105868. [Google Scholar] [CrossRef] - Yu, T.; Zhou, B.; Chan, K.W.; Yuan, Y.; Yang, B.; Wu, Q.H. R(λ) imitation learning for automatic generation control of interconnected power grids. Automatica
**2012**, 48, 2130–2136. [Google Scholar] [CrossRef]

**Figure 6.**The DDPG control signal during frequency modulation in the mode of gas determined by power.

**Figure 7.**The DDPG control signal during gas pressure modulation in the mode of power determined by gas.

**Figure 14.**Random perturbation function in the pre-learning phase: (

**a**) Random disturbances in wind power generation; (

**b**) Random disturbance of natural gas pipeline flow. (

**c**) The output power increment function of the EV stations.

**Figure 15.**The complete trend graph of the reward function: (

**a**) The complete trend graph of the reward function; (

**b**) The trend graph of the reward function for the last 50 iterations.

Unit | Parameter | Meaning | Value |
---|---|---|---|

MT | T_{f} | time constant of governor | 10 s |

T_{t} | time constant of generator | 0.1 s | |

R_{f} | speed regulation factor | 0.005 Hz/p.u. | |

λ_{mtd} | lower limit of active power variation | −0.015 p.u. | |

λ_{mtp} | upper limit of active power variation | 0.02 p.u. | |

EV_{MG1} | T_{ev1} | time constant of EV station of MG1 | 1 s |

±λ_{ev1} | the limit of active power variation | ±0.018 p.u. | |

n_{ev1} | Initial number of EVs in station of MG1 | 36 | |

EV_{MG2} | T_{ev2} | time constant of EV station of MG2 | 1 s |

±λ_{ev2} | the limit of active power variation | ±0.016 p.u. | |

n_{ev2} | Initial number of EVs in station of MG2 | 32 | |

P2G | T_{ele} | time constant of P2G | 0.5 s |

P_{CL}_{max} | maximum controllable amount | 0.01 p.u. | |

P_{CL}_{min} | minimum controllable amount | −0.01 p.u. | |

Other | H_{t} | Microgrid inertia parameters | 7.11 s |

T_{sij} | Microgrid coupling link parameters | 0.545 p.u. |

Parameter | Meaning | Value |
---|---|---|

μ_{i} (i = 1, 2, 3, 4) | time constant of governor | 3, 8, 20, 45 |

δ_{i} (i = 1, 2, 3, 4) | time constant of generator | 1, 3, 8, 20 |

γ | The discount factor | 0.95 |

α | The learning rate | 0.001 |

N_{e} | The maximum episode number | 500 |

n_{s} | The number of steps in each round | 300 |

l | The step size | 0.1 |

SN | Parameter Settings | Average Reward | Final Award |
---|---|---|---|

1 | h = 3, u = 50 | −34.037 | −1.83622 |

2 | h = 3, u = 200 | −26.673 | −1.20923 |

3 | h = 5, u = 50 | −21.096 | −0.65307 |

4 | h = 5, u = 200 | −27.075 | −1.35723 |

5 | h = 10, u = 50 | −34.572 | −2.54778 |

6 | h = 10, u = 200 | −40.922 | −3.04643 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Fan, P.; Hu, J.; Ke, S.; Wen, Y.; Yang, S.; Yang, J.
A Frequency–Pressure Cooperative Control Strategy of Multi-Microgrid with an Electric–Gas System Based on MADDPG. *Sustainability* **2022**, *14*, 8886.
https://doi.org/10.3390/su14148886

**AMA Style**

Fan P, Hu J, Ke S, Wen Y, Yang S, Yang J.
A Frequency–Pressure Cooperative Control Strategy of Multi-Microgrid with an Electric–Gas System Based on MADDPG. *Sustainability*. 2022; 14(14):8886.
https://doi.org/10.3390/su14148886

**Chicago/Turabian Style**

Fan, Peixiao, Jia Hu, Song Ke, Yuxin Wen, Shaobo Yang, and Jun Yang.
2022. "A Frequency–Pressure Cooperative Control Strategy of Multi-Microgrid with an Electric–Gas System Based on MADDPG" *Sustainability* 14, no. 14: 8886.
https://doi.org/10.3390/su14148886