# Optimal Operation Control of PV-Biomass Gasifier-Diesel-Hybrid Systems Using Reinforcement Learning Techniques

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

_{2}emissions, and various social criteria: creating jobs, effects on human health, human development index, etc. [5,6,7].

- The control problem of an isolated microgrid is formulated as an MDP. The modified open-source RL framework is employed for the modeling of an off-grid microgrid to investigate how state-of-the-art RL techniques can utilize the simulated data in order to learn an operation policy that minimizes the total system cost.
- The biomass gasification unit is employed to obtain producer gas. At the same time, the operation of the internal combustion engine (generator) is considered only in producer gas and dual-fuel mode (producer gas and diesel fuel). They operate as steerable generators of different configurations of a microgrid.

## 2. Microgrid MDP-Based Environment Simulator

#### 2.1. Dynamics

#### 2.1.1. Storage

#### 2.1.2. Steerable Generator Model

#### 2.2. Stochastic Optimization Formulation

_{t}∈ T:

## 3. Reinforcement Learning for Energy Microgrids Management

#### 3.1. Problem Statement

#### 3.2. Reinforcement Learning Agents

- fuel costs for the generation,
- curtailment cost for the excess of generation that had to be curtailed, and
- load shedding cost for the excess of load that had to be shed in order to maintain balance in the microgrid.

#### 3.2.1. MILP-Based Optimizer

#### 3.2.2. Deep Q-Network Agent

#### 3.2.3. Monte-Carlo Tree Search Agent

#### 3.2.4. Proximal Policy Optimization Agent

## 4. Results

#### 4.1. Microgrid Simulator Description

- If the total possible production (i.e., PV production, active steerable generators capacity, and the storages maximum discharge rate) is lower than the total consumption, a steerable generator is activated at its minimum stable generation. This instruction is repeated until the total load can be served or until all steerable generators are active. In a few words, the generators are activated one by one at their minimum stable generation until the total load can be served. Given the lower flexibility of the gasifier biomass generator compared to the diesel generator, it is assumed that the biomass generator does not turn off completely but continues to operate in idle mode. For the co-fired generator, the possibility of autonomous start-up on diesel fuel remains to ensure ignition of the gasifier biomass generator [80,81,82].
- Once all active steerable generators are known, the net generation can be calculated based on their minimum stable generation, the PV production, and the total consumption.
- If the net generation is positive, the storages (with charge instruction) charges the excess of energy until the net generation becomes zero. The storages with discharge or idle instructions do not do anything. The remaining excess of energy is curtailed.
- If the net generation is negative, the storages (with discharge instruction) discharges the deficit of energy until the net generation becomes zero. The storages with charge or idle instructions do not do anything. The remaining deficit of energy is then compensated by the active steerable generators which can be adjusted at a higher production level than their minimum stable power. If, in addition, steerable generators cannot handle the remaining deficit, this deficit is considered as lost load.

#### 4.2. Analysis of Different Microgrid Configuration Efficiency

- Configuration 1 (case 1)—PV (10 kW), diesel generator (10 kW), two storage devices (2 × 10 kWh), and three loads (3 × 10 kW).
- Configuration 2 (case 2)—PV (10 kW), gasifier biomass generator (10 kW), two storage devices (2 × 10 kWh), and three loads (3 × 10 kW).
- Configuration 3 (case 3)—PV (10 kW), co-fired generator (10 kW), two storage devices (2 × 10 kWh), and three loads (3 × 10 kW).
- Configuration 4 (case 3)—co-fired generator (20 kW), two storage devices (2 × 10 kWh), and three loads (3 × 10 kW).

#### 4.3. Comparative Study of RL-Based Models

## 5. Discussion and Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Siddaiah, R.; Saini, R.P. A review on planning, configurations, modeling and optimization techniques of hybrid renewable energy systems for off grid applications. Renew. Sustain. Energy Rev.
**2016**, 58, 376–396. [Google Scholar] [CrossRef] - Chauhan, A.; Saini, R.P. A review on Integrated Renewable Energy System based power generation for stand-alone applications: Configurations, storage options, sizing methodologies and control. Renew. Sustain. Energy Rev.
**2014**, 38, 99–120. [Google Scholar] [CrossRef] - Anvari, S.; Khalilarya, S.; Zare, V. Exergoeconomic and environmental analysis of a novel configuration of solar-biomass hybrid power generation system. Energy
**2018**, 165, 776–789. [Google Scholar] [CrossRef] - Cuesta, M.A.; Castillo-Calzadilla, T.; Borges, C.E. A critical analysis on hybrid renewable energy modeling tools: An emerging opportunity to include social indicators to optimise systems in small communities. Renew. Sustain. Energy Rev.
**2020**, 122, 109691. [Google Scholar] [CrossRef] - Rajbongshi, R.; Borgohain, D.; Mahapatra, S. Optimization of PV-biomass-diesel and grid base hybrid energy systems for rural electrification by using HOMER. Energy
**2017**, 126, 461–474. [Google Scholar] [CrossRef] - Sawle, Y.; Gupta, S.C.; Bohre, A.K. Socio-techno-economic design of hybrid renewable energy system using optimization techniques. Renew. Energy
**2018**, 119, 459–472. [Google Scholar] [CrossRef] - El-Emam, R.S.; Dincer, I. Assessment and Evolutionary Based Multi-Objective Optimization of a Novel Renewable-Based Polygeneration Energy System. J. Energy Res. Technol.
**2017**, 139. [Google Scholar] [CrossRef] - Guo, S.; Liu, Q.; Sun, J.; Jin, H. A review on the utilization of hybrid renewable energy. Renew. Sustain. Energy Rev.
**2018**, 91, 1121–1147. [Google Scholar] [CrossRef] - de Oliveira Vilela, A.; Lora, E.S.; Quintero, Q.R.; Vicintin, R.A.; Souza, T.P.D.S. A new technology for the combined production of charcoal and electricity through cogeneration. Biomass Bioenergy
**2014**, 69, 222–240. [Google Scholar] [CrossRef] - Kohsri, S.; Meechai, A.; Prapainainar, C.; Narataruksa, P.; Hunpinyo, P.; Sin, G. Design and preliminary operation of a hybrid syngas/solar PV/battery power system for off-grid applications: A case study in Thailand. Chem. Eng. Res. Des.
**2018**, 131, 346–361. [Google Scholar] [CrossRef] [Green Version] - Singh, A.; Baredar, P. Techno-economic assessment of a solar PV, fuel cell, and biomass gasifier hybrid energy system. Energy Rep.
**2016**, 2, 254–260. [Google Scholar] [CrossRef] [Green Version] - Zhang, X.; Zeng, R.; Mu, K.; Liu, X.; Sun, X.; Li, H. Exergetic and exergoeconomic evaluation of co-firing biomass gas with natural gas in CCHP system integrated with ground source heat pump. Energy Convers. Manag.
**2019**, 180, 622–640. [Google Scholar] [CrossRef] - González, A.; Riba, J.R.; Rius, A. Optimal sizing of a hybrid grid-connected photovoltaic–wind–biomass power system. Sustainability
**2015**, 7, 12787–12806. [Google Scholar] [CrossRef] [Green Version] - Perez-Navarro, A.; Alfonso, D.; Álvarez, C.; Ibáñez, F.; Sanchez, C.; Segura, I. Hybrid biomass-wind power plant for reliable energy generation. Renew. Energy
**2010**, 35, 1436–1443. [Google Scholar] [CrossRef] - Mago, P.J.; Chamra, L.M. Analysis and optimization of CCHP systems based on energy, economical, and environmental considerations. Energy Build.
**2009**, 41, 1099–1106. [Google Scholar] [CrossRef] - Parihar, A.K.S.; Sethi, V.; Banerjee, R. Sizing of biomass based distributed hybrid power generation systems in India. Renew. Energy
**2019**, 134, 1400–1422. [Google Scholar] [CrossRef] - Li, L.; Yao, Z.; You, S.; Wang, C.H.; Chong, C.; Wang, X. Optimal design of negative emission hybrid renewable energy systems with biochar production. Appl. Energy
**2019**, 243, 233–249. [Google Scholar] [CrossRef] [Green Version] - Chauhan, A.; Dwivedi, V.K. Optimal sizing of a stand-alone PV/wind/MHP/biomass based hybrid energy system using PSO algorithm. In Proceedings of the 2017 6th International Conference on Computer Applications in Electrical Engineering-Recent Advances (CERA), Roorkee, India, 5 October 2017; pp. 7–12. [Google Scholar] [CrossRef]
- Munuswamy, S.; Nakamura, K.; Katta, A. Comparing the cost of electricity sourced from a fuel cell-based renewable energy system and the national grid to electrify a rural health centre in India: A case study. Renew. Energy
**2011**, 36, 2978–2983. [Google Scholar] [CrossRef] - Banerjee, R. Comparison of options for distributed generation in India. Energy Policy
**2006**, 34, 101–111. [Google Scholar] [CrossRef] - Mahapatra, S.; Dasappa, S. Rural electrification: Optimising the choice between decentralised renewable energy sources and grid extension. Energy Sustain. Dev.
**2012**, 16, 146–154. [Google Scholar] [CrossRef] - Electric Microgrid on Mount Athos [Electronic Document]. Available online: https://energynet.ru/?p=articles&id=319 (accessed on 12 March 2020).
- Kartite, J.; Cherkaoui, M. Study of the different structures of hybrid systems in renewable energies: A review. Energy Procedia
**2019**, 157, 323–330. [Google Scholar] [CrossRef] - Al Ghaithi, H.M.; Fotis, G.P.; Vita, V. Techno-economic assessment of hybrid energy off-grid system—A case study for Masirah island in Oman. Int. J. Power Energy Res.
**2017**, 1, 103–116. [Google Scholar] [CrossRef] - Bhandari, B.; Lee, K.T.; Lee, G.Y.; Cho, Y.M.; Ahn, S.H. Optimization of hybrid renewable energy power systems: A review. Int. J. Pr. Eng. Man-Gt.
**2015**, 2, 99–112. [Google Scholar] [CrossRef] - Kurbatsky, V.G.; Sidorov, D.N.; Spiryaev, V.A.; Tomin, N.V. The hybrid model based on Hilbert-Huang Transform and neural networks for forecasting of short-term operation conditions of power system. IEEE Trondheim Power Tech.
**2011**, 1–7. [Google Scholar] [CrossRef] - Arun, P. Optimum Design of Biomass Gasifier Integrated Hybrid Energy Systems. Int. J. Energy Res.
**2015**, 5, 891–895. [Google Scholar] - Sansaniwal, S.K.; Pal, K.; Rosen, M.A.; Tyagi, S.K. Recent advances in the development of biomass gasification technology: A comprehensive review. Renew. Sustain. Energy Rev.
**2017**, 72, 363–384. [Google Scholar] [CrossRef] - García, R.; Pizarro, C.; Lavín, A.G.; Bueno, J.L. Biomass sources for thermal conversion. Techno-economical overview. Fuel
**2017**, 195, 182–189. [Google Scholar] [CrossRef] - Molino, A.; Chianese, S.; Musmarra, D. Biomass gasification technology: The state of the art overview. J Energy Chem.
**2016**, 25, 10–25. [Google Scholar] [CrossRef] - Castaldi, M.; Van Deventer, J.; Lavoie, J.M.; Legrand, J.; Nzihou, A.; Pontikes, Y.; Py, X.; Vandecasteele, C.; Vasudevan, P.T.; Verstraete, W. Progress and prospects in the field of biomass and waste to energy and added-value materials. Waste Biomass Valorization
**2017**, 8, 1875–1884. [Google Scholar] [CrossRef] [Green Version] - Santanu, D.; Avinash, K.A.; Moholkar, V.S.; Thallada, B. Coal and Biomass Gasification. Recent Advances and Future; Springer: Singapore, 2018; Volume 524. [Google Scholar] [CrossRef]
- Heidenreich, S.; Foscolo, P.U. New concepts in biomass gasification. Prog. Energy Combust. Sci.
**2015**, 46, 72–95. [Google Scholar] [CrossRef] - Hupa, M.; Karlstrom, O.; Vainio, E. Biomass combustion technology development—It is all about chemical details. Proc. Combust. Inst.
**2017**, 36, 113–134. [Google Scholar] [CrossRef] - Kozlov, A.N.; Svishchev, D.A.; Khudiakova, G.I.; Ryzhkov, A.F. A kinetic analysis of the thermochemical conversion of solid fuels (A review). Solid Fuel Chem.
**2017**, 51, 205–213. [Google Scholar] [CrossRef] - Ramos, A.; Monteiro, E.; Silva, V.; Rouboa, A. Co-gasification and recent developments on waste-to-energy conversion: A review. Renew. Sustain. Energy Rev.
**2018**, 81, 380–398. [Google Scholar] [CrossRef] - Baroudi, D.; Ferrantelli, A.; Li, K.Y.; Hostikka, S. A thermomechanical explanation for the topology of crack patterns observed on the surface of charred wood and particle fibreboard. Combust. Flame
**2017**, 182, 206–215. [Google Scholar] [CrossRef] [Green Version] - Costa, F.F.; Costa, M. Particle fragmentation of raw and torrefied biomass during combustion in a drop tube furnace. Fuel
**2015**, 159, 530–537. [Google Scholar] [CrossRef] - Tolvanen, H.; Keipi, T.; Raiko, R. A study on raw, torrefied, and steam-exploded wood: Fine grinding, drop-tube reactor combustion tests in N
_{2}/O_{2}and CO_{2}/O_{2}atmospheres, particle geometry analysis, and numerical kinetics modeling. Fuel**2016**, 176, 153–164. [Google Scholar] [CrossRef] - Kortelainen, M.; Jokiniemi, J.; Nuutinen, I.; Torvela, T.; Lamberg, H.; Karhunen, T.; Tissari, J.; Sippula, O. Ash behaviour and emission formation in a small-scale reciprocating-grate combustion reactor operated with wood chips, reed canary grass and barley straw. Fuel
**2015**, 143, 80–88. [Google Scholar] [CrossRef] - Lanzerstorfer, C. Grate-Fired Biomass Combustion Plants Using Forest Residues as Fuel: Enrichment Factors for Components in the Fly Ash. Waste Biomass Valorization
**2017**, 8, 235–240. [Google Scholar] [CrossRef] [Green Version] - Hirka, I.; Zivny, O.; Hrabovsky, M. Numerical Modelling of Wood Gasification in Thermal Plasma Reactor. Plasma Chem. Plasma Process.
**2017**, 37, 947–965. [Google Scholar] [CrossRef] - Materazzi, M.; Lettieri, P.; Mazzei, L.; Taylor, R.; Chapman, C. Reforming of tars and organic sulphur compounds in a plasma-assisted process for waste gasification. Fuel Process. Technol.
**2015**, 137, 259–268. [Google Scholar] [CrossRef] - Yakaboylu, O.; Harinck, J.; Smit, K.G.; De Jong, W. Testing the constrained equilibrium method for the modeling of supercritical water gasification of biomass. Fuel Process. Technol.
**2015**, 138, 74–85. [Google Scholar] [CrossRef] - González, A.M.; Jaén, R.L.; Lora, E.E.S. Thermodynamic assessment of the integrated gasification-power plant operating in the sawmill industry: An energy and exergy analysis. Renew. Energy
**2020**, 147, 1151–1163. [Google Scholar] [CrossRef] - Sutar, K.B.; Kohli, S.; Ravi, M.R. Design, development and testing of small downdraft gasifiers for domestic cookstoves. Energy
**2017**, 124, 447–460. [Google Scholar] [CrossRef] - Susastriawan, A.A.P.; Saptoadi, H. Small-scale downdraft gasifiers for biomass gasification: A review. Renew. Sustain. Energy Rev.
**2017**, 76, 989–1003. [Google Scholar] [CrossRef] - Elsner, W.; Wysocki, M.; Niegodajew, P.; Borecki, R. Experimental and economic study of small-scale CHP installation equipped with downdraft gasifier and internal combustion engine. Appl. Energy
**2017**, 202, 213–227. [Google Scholar] [CrossRef] - Renzi, M.; Riolfi, C.; Baratieri, M. Influence of the syngas feed on the combustion process and performance of a micro gas turbine with steam injection. Energy Procedia
**2017**, 105, 1665–1670. [Google Scholar] [CrossRef] - Obernberger, I.; Brunner, T.; Mandl, C.; Kerschbaum, M.; Svetlik, T. Strategies and technologies towards zero emission biomass combustion by primary measures. Energy Procedia
**2017**, 120, 681–688. [Google Scholar] [CrossRef] - Wang, T.; Stiegel, G.J. Integrated gasification combined cycle (IGCC) technologies. Woodhead Publ.
**2017**, 929. [Google Scholar] - Thattai, A.T.; Oldenbroek, V.; Schoenmakers, L.; Woudstra, T.; Aravind, P.V. Experimental model validation and thermodynamic assessment on high percentage (up to 70%) biomass co-gasification at the 253 MWe integrated gasification combined cycle power plant in Buggenum, The Netherlands. Appl. Energy
**2016**, 168, 381–393. [Google Scholar] [CrossRef] [Green Version] - Cormos, A.-M.; Dinca, C.; Cormos, C.-C. Multi-fuel multi-product operation of IGCC power plants with carbon capture and storage (CCS). Appl. Therm. Eng.
**2015**, 74, 20–27. [Google Scholar] [CrossRef] - Howaniec, N.; Smolinski, A.; Cempa-Balewicz, M. Experimental study on application of high temperature reactor excess heat in the process of coal and biomass co-gasification to hydrogen-rich gas. Energy
**2015**, 84, 455–461. [Google Scholar] [CrossRef] - Francois-Lavet, V.; Tarella, D.; Ernst, D.; Forteneau, R. Deep Reinforcement Learning Solutions for Energy Microgrids Management. In European Workshop on Reinforcement Learning; 2016; Available online: http://hdl.handle.net/2268/203831 (accessed on 31 January 2020).
- Sidorov, D.; Panasetsky, D.; Tomin, N.; Karamov, D.; Zhukov, A.; Muftahov, I.; Dreglea, A.; Liu, F.; Li, Y. Toward Zero-Emission Hybrid AC/DC Power Systems with Renewable Energy Sources and Storages: A Case Study from Lake Baikal Region. Energies
**2020**, 13, 1226. [Google Scholar] [CrossRef] [Green Version] - Shang, Y.; Wu, W.; Guo, J.; Lv, Z.; Ma, Z.; Sheng, W.; Chen, R. Stochastic Dispatch of Energy Storage in Microgrids: A Reinforcement Learning Approach Incorporated with MCTS. arXiv
**2019**, arXiv:1910.04541. [Google Scholar] - Mocanu, E.; Mocanu, D.C.; Nguyen, P.H.; Liotta, A.; Webber, M.E.; Gibescu, M.; Slootweg, J.G. On-line building energy optimization using deep reinforcement learning. IEEE Trans. Smart Grid.
**2018**, 10, 3698–3708. [Google Scholar] [CrossRef] [Green Version] - Mbuwir, B.V.; Ruelens, F.; Spiessens, F.; Deconinck, G. Battery Energy Management in a Microgrid Using Batch Reinforcement Learning. Energies
**2017**, 10, 1846. [Google Scholar] [CrossRef] [Green Version] - Li, F.D.; Wu, M.; He, Y.; Chen, X. Optimal control in microgrid using multi-agent reinforcement learning. ISA Trans.
**2012**, 51, 743–751. [Google Scholar] [CrossRef] - Sogabe, T.; Malla, D.B.; Takayama, S.; Shin, S.; Sakamoto, K.; Yamaguchi, K.; Okada, Y. Smart grid optimization by deep reinforcement learning over discrete and continuous action space. In Proceedings of the 2018 IEEE 7th World Conference on Photovoltaic Energy Conversion (WCPEC) (A Joint Conference of 45th IEEE PVSC, 28th PVSEC & 34th EU PVSEC), Waikoloa Village, HI, USA, 10–15 June 2018; pp. 3794–3796. [Google Scholar] [CrossRef]
- Bollinger, L.A.; Evins, R. Multi-Agent Reinforcement Learning for Optimizing Technology Deployment in Distributed Multi-Energy Systems; EG-ICE Workshop: Krakow, Poland, 2016. [Google Scholar]
- Duan, J.; Yi, Z.; Shi, D.; Lin, C.; Lu, X.; Wang, Z. Reinforcement-Learning-Based Optimal Control for Hybrid Energy Storage Systems in Hybrid AC/DC Microgrids. IEEE Trans. Ind. Inform.
**2019**. [Google Scholar] [CrossRef] - Ji, Y.; Wang, J.; Xu, J.; Fang, X.; Zhang, H. Real-Time Energy Management of a Microgrid Using Deep Reinforcement Learning. Energies
**2019**, 12, 2291. [Google Scholar] [CrossRef] [Green Version] - Boukas, I.; El Mekki, S.; Cornélusse, B. Data-driven Parameterized Policies for Microgrid Control. Unpublished work. 2019. [Google Scholar]
- An, L.N.; Tuan, T.Q. Dynamic Programming for Optimal Energy Management of Hybrid Wind–PV–Diesel–Battery. Energies
**2018**, 11, 3039. [Google Scholar] [CrossRef] [Green Version] - Zhuo, W. Microgrid Energy Management Strategy with Battery Energy Storage System and Approximate Dynamic Programming. In Proceedings of the 2018 37th Chinese Control Conference (CCC), Wuhan, China, 25–27 July 2018; pp. 7581–7587. [Google Scholar] [CrossRef]
- Jahangir, H.; Ahmadian, A.; Golkar, M.A. Optimal design of stand-alone microgrid resources based on proposed Monte-Carlo simulation. In Proceedings of the 2015 IEEE Innovative Smart Grid Technologies—Asia (ISGT ASIA), Bangkok, Thailand, 3–6 November 2015; pp. 1–6. [Google Scholar] [CrossRef]
- Sidorov, D.N.; Muftahov, I.R.; Tomin, N.; Karamov, D.N.; Panasetsky, D.A.; Dreglea, A.; Liu, F.; Foley, A. A Dynamic Analysis of Energy Storage with Renewable and Diesel Generation using Volterra Equations. IEEE Trans. Ind. Inf.
**2020**, 3451–3459. [Google Scholar] [CrossRef] [Green Version] - Sutton, R.S.; Barto, A.G. Introduction to Reinforcement Learning; MIT Press: Cambridge, MA, USA, 2018. [Google Scholar]
- Watkins, C.J.C.H.; Dayan, P. Technical Note: Q-Learning. Mach. Learn.
**1992**, 8, 279–292. [Google Scholar] [CrossRef] - Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; Riedmiller, M. Playing atari with deep reinforcement learning. arXiv
**2013**, arXiv:1312.5602. [Google Scholar] - Browne, C.B.; Powley, E.; Whitehouse, D.; Lucas, S.M.; Cowling, P.I.; Rohlfshagen, P.; Colton, S. A survey of Monte Carlo tree search methods. IEEE Trans. Comput. Intell. AI Games
**2012**, 4, 1–43. [Google Scholar] [CrossRef] [Green Version] - Kartal, B.; Hernandez-Leal, P.; Taylor, M.E. Action Guidance with MCTS for Deep Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Atlanta, GA, USA, 8–12 October 2019; Volume 15, pp. 153–159. [Google Scholar]
- Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal policy optimization algorithms. arXiv
**2017**, arXiv:1707.06347. [Google Scholar] - Sutton, R.S.; McAllester, D.A.; Singh, S.P.; Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems; Massachusetts Institute of Technology Press: Cambridge, MA, USA, 2000; pp. 1057–1063. [Google Scholar]
- Kalbande, S.R.; Deshmukh, M.M.; Wakudkar, H.M.; Wasu, G. Evaluation of gasifier based power generation system using different woody biomass. ARPN J. Eng. Appl. Sci.
**2010**, 5, 82–88. [Google Scholar] - Available online: https://github.com/bcornelusse/microgridRLsimulator (accessed on 12 March 2020).
- Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; Zaremba, W. Openai gym. arXiv
**2016**, arXiv:1606.01540. [Google Scholar] - Zysin, L.V.; Koshkin, N.L.; Orlov, E.I.; Sergeev, V.V.; Steshenkov, L.P. A study of the joint operation of a diesel engine and a gas generator processing plant biomass. Therm. Eng.
**2002**, 49, 14–19. [Google Scholar] - Martínez, J.D.; Mahkamov, K.; Andrade, R.V.; Lora, E.E.S. Syngas production in downdraft biomass gasifiers and its application using internal combustion engines. Renew. Energy
**2012**, 38, 1–9. [Google Scholar] [CrossRef] - Sharma, M.; Kaushal, R. Performance and emission analysis of a dual fuel variable compression ratio (VCR) CI engine utilizing producer gas derived from walnut shells. Energy
**2020**, 192, 116725. [Google Scholar] [CrossRef]

**Figure 1.**The main reinforcement learning (RL)-based approach for the energy microgrids’ optimal management.

**Figure 4.**Total costs (left) and generation/load mix – right (The load mix on the graph here does not mean the entire total load of the microgrid, but only an illustration of what components of the electricity consumption (load, battery, or curtailment) the generated power were used to ensure balance) of different microgrids’ configurations for optimal policies, ${\pi}^{*}$ obtained using the Monte-Carlo tree search (MCTS) for the one-week testing period.

**Figure 5.**Total costs (left) and generation/load mix (right) of different microgrids with co-fired generators for optimal policies, ${\pi}^{*}$ obtained using MCTS for the one-week testing period.

**Figure 6.**Dynamics of the charge and discharge of batteries for Case 1 for optimal policies, ${\pi}^{*}$ obtained using PPO and MCTS algorithms for the one-week testing period.

**Figure 7.**Dynamics of the charge and discharge of batteries for Case 4 for optimal policies, ${\pi}^{*}$ obtained using PPO algorithm for the one-week testing period.

Unit | Parameter | Value |
---|---|---|

Diesel generator | lower heating value, $LH{V}_{fuel}$ [MJ/kg] | 43.2 |

fuel density ${\rho}_{fuel}$ [kg/l] | 820 | |

fuel (diesel) price, ${\pi}_{g}^{fuel}$ [euro/l] | 1 | |

minimal power ratio | 0.25 | |

capacity, ${P}_{st}$ [kW] | 10 | |

Gasifier biomass generator | lower heating value, $LH{V}_{fuel}$ [MJ/m^{3}] | 6.17 |

biomass flow rate, ${\dot{m}}_{gas}$ [kg/h] | 15 | |

fuel (pellets) price, ${\pi}_{g}^{fuel}$ [euro/kg] | 0.11 | |

minimal power ratio | 0.20 | |

capacity, ${P}_{st}$ [kW] | 10 | |

Co-fired generator | minimal power ratio | 0.20 |

producer substitution ratio, ${z}_{gas}$ | 8.5 | |

fuel (pellets) price, ${\pi}_{g}^{fuel}$ [euro/kg] | 0.11 | |

available producer flow rate [kW/h] | 28 | |

capacity, ${P}_{st}$ [kW] | 10/20 * | |

Storage device | battery capacity, [kWh] | 12 |

charge/discharge efficiency, ${\eta}^{charge}$, ${\eta}^{discharge}$ | 0.95/0.89 | |

maximum/minimum charge rate, [kW] | 4.0 |

Models | Total Costs (Euro) | |||
---|---|---|---|---|

PV + Co-Fired Generator (Case 1) | PV + Gasifier Biomass Generator (Case 2) | PV + Diesel Generator (Case 3) | Co-Fired Generator (Case 4) | |

MCTS | 181 | 144 | 630 | 240 |

DQN | 1042 | 975 | 1619 | 2140 |

PPO | 417 | 846 | 1478 | 1110 |

MILP (ideal model) | 131 | 122 | 347 | 161 |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kozlov, A.N.; Tomin, N.V.; Sidorov, D.N.; Lora, E.E.S.; Kurbatsky, V.G.
Optimal Operation Control of PV-Biomass Gasifier-Diesel-Hybrid Systems Using Reinforcement Learning Techniques. *Energies* **2020**, *13*, 2632.
https://doi.org/10.3390/en13102632

**AMA Style**

Kozlov AN, Tomin NV, Sidorov DN, Lora EES, Kurbatsky VG.
Optimal Operation Control of PV-Biomass Gasifier-Diesel-Hybrid Systems Using Reinforcement Learning Techniques. *Energies*. 2020; 13(10):2632.
https://doi.org/10.3390/en13102632

**Chicago/Turabian Style**

Kozlov, Alexander N., Nikita V. Tomin, Denis N. Sidorov, Electo E. S. Lora, and Victor G. Kurbatsky.
2020. "Optimal Operation Control of PV-Biomass Gasifier-Diesel-Hybrid Systems Using Reinforcement Learning Techniques" *Energies* 13, no. 10: 2632.
https://doi.org/10.3390/en13102632