OptiMEMS: An Adaptive Lightweight Optimal Microgrid Energy Management System Based on the Novel Virtual Distributed Energy Resources in Real-Life Demonstration

Abstract

As microgrids have gained increasing attention over the last decade, more and more applications have emerged, ranging from islanded remote infrastructures to active building blocks of smart grids. To optimally manage the various microgrid assets towards maximum profit, while taking into account reliability and stability, it is essential to properly schedule the overall operation. To that end, this paper presents an optimal scheduling framework for microgrids both for day-ahead and real-time operation. In terms of real-time, this framework evaluates the real-time operation and, based on deviations, it re-optimises the schedule dynamically in order to continuously provide the best possible solution in terms of economic benefit and energy management. To assess the solution, the designed framework has been deployed to a real-life microgrid establishment consisting of residential loads, a PV array and a storage unit. Results demonstrate not only the benefits of the day-ahead optimal scheduling, but also the importance of dynamic re-optimisation when deviations occur between forecasted and real-time values. Given the intermittency of PV generation as well as the stochastic nature of consumption, real-time adaptation leads to significantly improved results.

1. Introduction

Microgrids (MG), foreseen as the avant-garde building blocks of the future Distribution Network (DN), have attracted much attention from various research fields, ranging from power electronics [1] and control systems [2] to machine learning [3]. As such, it is essential to move past simulations [4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23] and hardware-in-the-loop (HIL) configurations [24,25,26,27,28,29,30] towards the evaluation of technologies in real-life applications [31,32,33,34]. In general, MG operation must be technically reliable and economically feasible, regardless of the stochastic nature of Renewable Energy Sources (RES), or the connection status with the DN [16,35]. The MG day-ahead scheduling problem essentially consists of the implementation of an optimised energy management strategy. Optimisation techniques lead to improved MG operation compared to rule-based approaches [36], which, though simple and direct, limit the MG operation [24]. The optimisation objectives may vary: cost [37] or profit-efficient MG operation [38], via stochastic programming [39,40], or a combination of integrated Two-Stage Stochastic Programming (ITSP) and Worst-Case Value at Risk (VCVaR) [18]; social benefit maximisation via Mixed Integer Point Solver (MIPS) [14]; extension of equipment’s lifetime [8,41]; load feeding reliability, utilisation of resources, e.g., RES production [42]. They are solved using common optimisation techniques, e.g., Dynamic, Linear/Mixed-Integer Linear Programming (LP/MILP) [43]; Nonlinear (NLP)/Quadratic Programming [15,34]; or heuristics ones, such as Genetic Algorithm [44], Isolation Niche Immune Genetic Algorithm [45], Non-Dominated Sorting Genetic Algorithm type II [46], Niching Evolutionary Algorithm [47], Particle Swarm Optimisation [37,48], Ant Colony Optimisation [49], or even Binary Backtracking Search Algorithm [20] and Glowworm Swarm Optimisation [12]. Heuristic algorithms are rarely applied beyond simulation. Due to their structure, they are susceptible to local minima entrapment [50]. Moreover, they are quite complex to implement, leading to long computation time. Ergo, they are not quite deemed suitable for real-time operation but rather for cases of long-term planning and assessment. Simpler, deterministic approaches, such as LP/MILP have been identified as more suited for real-site demonstration [9,13,26,51,52].

As presented in Table 1, there is a significant gap in the literature regarding actual deployment of the introduced solutions, either in terms of Day-Ahead (DA) and Real-Time (RT) optimisation schemes, which could very well lead to completely different results than the ones extracted from simulations, or even HIL setups. In general, simulations and HIL setups are good at understanding system margins, making design trade-offs and verification. However, each simulation software makes a series of assumptions in regards to the equipment modelling and operating conditions. Similarly, small-scale experimental test-beds, though closer to reality, still constitute a controlled environment, which, especially in the presence of purely stochastic phenomena (such as loading and weather conditions), cannot describe reality accurately [53]. For these reasons, this work aims towards presenting the real-life application of a microgrid optimal scheduler, where real weather conditions and real home users are involved for a period of 9 days.

Aiming to create a real-world applied optimal scheduler, in this paper, an Optimal Energy Management System (OptiMEMS) has been developed, deployed and assessed upon a residential MG. OptiMEMS framework includes four components: an MILP-based optimisation engine that produces the optimal day- or steps-ahead MG schedules; a real-time supervising tool that monitors and validates the optimal schedule application; a machine-learning-based load consumption forecasting tool and a hybrid deterministic/stochastic PV generation predicting tool (Figure 1). The optimised scheduler solves an adjusted Unit Commitment (UC) problem, tailored for MGs operating in grid-connected mode, with the objective being the minimisation of the operation cost. Finally, a customised module is responsible for applying the optimised schedule whenever it is allowed and, for triggering its recalculation in real-time, in case of significant divergence between real load consumption, RES generation and the forecasted ones. The application of the OptiMEMS is realised on a fully RES-based, grid-connected AC microgrid [54].

This paper is organised as follows. In Section 2.1, the problem formulation is analysed. In Section 2.2, the System Architecture is presented. In Section 3.1, the experimental setup is described, and in Section 3.2, the results are presented. Finally, Section 4 is devoted to conclusions and future work.

Table 1. Day-ahead MG optimisation approaches.

Ref No.	Objective	Technique	MG Mode	DERs	Implementation	Schedule
[4]	EM - min cost	MILP	Islanded Grid Connected	PV, Wind Turbines, Fuel Cells, Micro Turbines, Diesel Generators, ESS	Simulations	DA RT
[5]	EM - min losses	Dynamic Programming	Grid Connected	Wind Turbines, ESS, PV, ESS	Simulation	DA
[6]	EM - min cost	MILP	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA
[7]	EM	MILP/NLP	Isolated	PV, Wind Turbines, Fuel Cells, Micro Turbines, Diesel Generators, ESS	Simulations	DA
[8]	multi-objective	MILP	Isolated	PV, Diesel Generators, ESS	Simulations	DA
[10]	min cost (operation)	MILP	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA
[11]	min cost (operation)	MILP	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA
[12]	min losses + min fuel consumption	Heuristic	Grid Connected	PV, Micro Turbines	Simulations	DA
[14]	Social benefit max	Quadratic Programming	Grid Connected	PV, Wind Turbines, Fuel Cells, Micro Turbines, Pico Hydel ESS	Simulations	DA RT
[15]	min cost (operation)	Quadratic Programming	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA RT
[16]	min cost (operation)	Stochastic/Robust optimisation	Grid Connected	PV, Wind Turbines, Diesel Generators, ESS	Simulations	DA
[17]	min cost (operation)	Quadratic Programming	Grid Connected	PV, ESS	Simulations	DA
[18]	min cost	ITSP and WCVaR	Grid Connected	PV, Wind Turbines, Micro Turbines, CHP, ESS	Simulations	DA
[20]	EM	Heuristic	Grid Connected	PV, Wind Turbines, Fuel Cells, Diesel Generators, ESS	Simulations	DA
[21]	min cost (operation)	MILP	Isolated	Heavy/Light fuel units, PV, Wind Turbines, ESS	Simulations	DA
[22]	min cost (operation)	MILP	Islanded	PV, Wind Turbines, Diesel Generators, ESS	Simulations	DA Intra-DA
[23]	min cost—max ESS utilisation efficiency	MILP	Gird Connected	PV, Wind Turbines, Micro Turbines, Diesel Generators, ESS	Simulations	HA MA
[24]	min cost	Dynamic Programming	Grid Connected	PV, ESS	HIL	DA
[25]	min cost (operation)	Rule-based	Isolated	PV, Fuel Cells	HIL	DA
[26,27]	min cost (operation)	LP/MILP	Islanded Grid Connected	PV, Wind Turbines, ESS	HIL	DA
[30]	min cost (operation)	MILP	Islanded Grid Connected	PV, Wind Turbines, Fuel Cells, ESS	HIL	DA
[32]	min cost	MILP	Isolated	PV, Wind Turbines, microCHP, ESS	Real-life Prototype	DA
[33]	min cost (operation + emissions)	Rule-based	Isolated	PV, Diesel Generators	Simulations	Yearly
[34]	min cost (operation + emissions)	NLP/Sequential Quadratic Programming	Islanded Grid Connected	PV, Wind Turbines, Micro Turbines, CHP, ESS	HIL	DA
[36]	EM - min cost	Rule-based	Grid Connected	PV, Wind Turbines, Micro Turbines, ESS	Simulations + HIL	DA RT
[39]	min cost (operation)	Stochastic Programming	Grid Connected	PV, Wind Turbines, Micro Turbines, ESS	Simulations	DA
[37,48]	min cost (operation)	Heuristic	Grid Connected	PV, Wind Turbines, CCHP, ESS	Simulations	DA RT
[38]	max profit	two-stage Stochastic Programming	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA
[40]	max power production	MINLP	Grid Connected	PV, ESS	Simulations + HIL	DA RT
[41,51]	min cost (operation + extension of hybrid ESS lifetime)	MILP	Isolated	PV, Diesel Generators, ESS	Simulations + HIL	DA
[43]	min fuel consumption + min cost (operation)	LP/MILP	Isolated	PV, Diesel Generators, ESS	Simulations	DA
[44]	min cost (operation)	Heuristic	Isolated	PV, Wind Turbines, Diesel Generators, ESS	Simulations	DA
[45]	min cost (operation)	Heuristic	Islanded Grid Connected	PV, Wind Turbines, HT, Fuel Cells, ESS, GT	Simulation	DA
[46]	min embodied energy + min LPSP	NLP/Sequential Quadratic Programming	Grid Connected	PV, Wind Turbines, ESS	Simulations	hours-ahead
[47]	min cost (operation + emission)	Heuristic	Grid Connected	ESS	Simulations	hours-ahead
[49]	min cost	Heuristic	Grid Connected	PV, Wind Turbines, Micro Turbines, ESS	Simulations + HIL	DA RT
[52]	max profit	MILP	Grid Connected	PV, ESS	Simulations + HIL	DA RT

Table 1. Day-ahead MG optimisation approaches.

Ref No.	Objective	Technique	MG Mode	DERs	Implementation	Schedule
[4]	EM - min cost	MILP	Islanded Grid Connected	PV, Wind Turbines, Fuel Cells, Micro Turbines, Diesel Generators, ESS	Simulations	DA RT
[5]	EM - min losses	Dynamic Programming	Grid Connected	Wind Turbines, ESS, PV, ESS	Simulation	DA
[6]	EM - min cost	MILP	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA
[7]	EM	MILP/NLP	Isolated	PV, Wind Turbines, Fuel Cells, Micro Turbines, Diesel Generators, ESS	Simulations	DA
[8]	multi-objective	MILP	Isolated	PV, Diesel Generators, ESS	Simulations	DA
[10]	min cost (operation)	MILP	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA
[11]	min cost (operation)	MILP	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA
[12]	min losses + min fuel consumption	Heuristic	Grid Connected	PV, Micro Turbines	Simulations	DA
[14]	Social benefit max	Quadratic Programming	Grid Connected	PV, Wind Turbines, Fuel Cells, Micro Turbines, Pico Hydel ESS	Simulations	DA RT
[15]	min cost (operation)	Quadratic Programming	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA RT
[16]	min cost (operation)	Stochastic/Robust optimisation	Grid Connected	PV, Wind Turbines, Diesel Generators, ESS	Simulations	DA
[17]	min cost (operation)	Quadratic Programming	Grid Connected	PV, ESS	Simulations	DA
[18]	min cost	ITSP and WCVaR	Grid Connected	PV, Wind Turbines, Micro Turbines, CHP, ESS	Simulations	DA
[20]	EM	Heuristic	Grid Connected	PV, Wind Turbines, Fuel Cells, Diesel Generators, ESS	Simulations	DA
[21]	min cost (operation)	MILP	Isolated	Heavy/Light fuel units, PV, Wind Turbines, ESS	Simulations	DA
[22]	min cost (operation)	MILP	Islanded	PV, Wind Turbines, Diesel Generators, ESS	Simulations	DA Intra-DA
[23]	min cost—max ESS utilisation efficiency	MILP	Gird Connected	PV, Wind Turbines, Micro Turbines, Diesel Generators, ESS	Simulations	HA MA
[24]	min cost	Dynamic Programming	Grid Connected	PV, ESS	HIL	DA
[25]	min cost (operation)	Rule-based	Isolated	PV, Fuel Cells	HIL	DA
[26,27]	min cost (operation)	LP/MILP	Islanded Grid Connected	PV, Wind Turbines, ESS	HIL	DA
[30]	min cost (operation)	MILP	Islanded Grid Connected	PV, Wind Turbines, Fuel Cells, ESS	HIL	DA
[32]	min cost	MILP	Isolated	PV, Wind Turbines, microCHP, ESS	Real-life Prototype	DA
[33]	min cost (operation + emissions)	Rule-based	Isolated	PV, Diesel Generators	Simulations	Yearly
[34]	min cost (operation + emissions)	NLP/Sequential Quadratic Programming	Islanded Grid Connected	PV, Wind Turbines, Micro Turbines, CHP, ESS	HIL	DA
[36]	EM - min cost	Rule-based	Grid Connected	PV, Wind Turbines, Micro Turbines, ESS	Simulations + HIL	DA RT
[39]	min cost (operation)	Stochastic Programming	Grid Connected	PV, Wind Turbines, Micro Turbines, ESS	Simulations	DA
[37,48]	min cost (operation)	Heuristic	Grid Connected	PV, Wind Turbines, CCHP, ESS	Simulations	DA RT
[38]	max profit	two-stage Stochastic Programming	Grid Connected	PV, Wind Turbines, ESS	Simulations	DA
[40]	max power production	MINLP	Grid Connected	PV, ESS	Simulations + HIL	DA RT
[41,51]	min cost (operation + extension of hybrid ESS lifetime)	MILP	Isolated	PV, Diesel Generators, ESS	Simulations + HIL	DA
[43]	min fuel consumption + min cost (operation)	LP/MILP	Isolated	PV, Diesel Generators, ESS	Simulations	DA
[44]	min cost (operation)	Heuristic	Isolated	PV, Wind Turbines, Diesel Generators, ESS	Simulations	DA
[45]	min cost (operation)	Heuristic	Islanded Grid Connected	PV, Wind Turbines, HT, Fuel Cells, ESS, GT	Simulation	DA
[46]	min embodied energy + min LPSP	NLP/Sequential Quadratic Programming	Grid Connected	PV, Wind Turbines, ESS	Simulations	hours-ahead
[47]	min cost (operation + emission)	Heuristic	Grid Connected	ESS	Simulations	hours-ahead
[49]	min cost	Heuristic	Grid Connected	PV, Wind Turbines, Micro Turbines, ESS	Simulations + HIL	DA RT
[52]	max profit	MILP	Grid Connected	PV, ESS	Simulations + HIL	DA RT

2. Materials and Methods

2.1. Optimisation Problem Formulation

2.1.1. Virtual Distributed Energy Resources

In the context of OptiMEMS, the optimisation module is responsible for solving an adjusted UC problem tailored to MG requirements. Its basis is a novel concept called: virtual Distributed Energy Resource (vDER). Using the Passive Sign Convention, a vDER can be a power source (provider: negative power) or a power load (consumer: positive power). Therefore, in the present optimisation problem, the vDERs “ESS charge” and “Grid Export” are positive, while “PV”, “ESS Discharge” and “Grid Import” are negative; this information is given on the superscript symbols. The accumulated output of all vDERs must satisfy the load demand in each time slot.

Regarding the formulation of vDERs, unidirectional DERs, such as PV modules, remain as they were, whereas DERs with a bidirectional energy flow, such as ESSs, are split into two complementary virtual ones. In this study, the formulation of the vDERs concerns a grid-connected MG, consisting of PV modules, ESS modules (assigned index S) and the Point of Common Coupling (PCC), with the latter being treated as a bidirectional DER (assigned index G for Grid). For example, ESS Discharge and Grid Import are modelled as vDERs that provide power, and ESS Charge and Grid Export as vDERs that consume power. During grid-connected operation, aiming towards PV production maximisation, PV modules can be modelled as negative loads: forecasted production is subtracted from the forecasted consumption, leading to a new artificial curve of “Net Demand”. Generally, if PV curtailment is possible, PV modules are modelled as an additional vDER (negative power). In the following paragraphs, the latter, more general approach is employed.

2.1.2. Adjusted Unit Commitment Problem Formulation

Since the distances among the physical assets of the considered MG are limited, the physical aspects of the MG circuitry can be omitted and thus, the need for any kind of optimal power flow related constraints. Therefore, the single bus model approach of the UC can be adopted, without loss of reality [49,55]. It is noted that all the variables and symbols used in the following paragraphs are given in the Abbreviations section at the end of the document. The problem contains two sets of optimisation variables, shown below: the continuous vDER energies as a function of time and a set of auxiliary integer (binary) optimisation variables modelling the mutually exclusive operation of the complementary vDERs related to PCC and batteries. In the general case of an MG consisting of N batteries, M PV modules and a PCC, the optimisation variables for a time horizon T are:

$$\begin{array}{cc}\hfill {\mathbf{E}}_S\left(t\right)& =\left[\begin{array}{ccc}{E}_{S1}^{+}\left(t\right)& \cdots & E_{S_N}^{+}\left(t\right)\\ E_{S1}^{-}\left(t\right)& \cdots & E_{S_N}^{-}\left(t\right)\end{array}\right]=\left[\begin{array}{c}\overrightarrow{E_S^{+}}{\left(t\right)}^{\top }\\ \overrightarrow{E_S^{-}}{\left(t\right)}^{\top }\end{array}\right]\hfill \end{array}$$

(1a)

$$\begin{array}{cc}\hfill \overrightarrow{E_G}{\left(t\right)}^{\top }& =\left[\begin{array}{cc}{E}_G^{-}\left(t\right)& E_G^{+}\left(t\right)\end{array}\right]\hfill \end{array}$$

(1b)

$$\begin{array}{cc}\hfill \overrightarrow{E_{PV}}{\left(t\right)}^{\top }& =\left[\begin{array}{ccc}{E}_{PV1}\left(t\right)& \cdots & E_{P{V}_M}\left(t\right)\end{array}\right]\hfill \end{array}$$

(1c)

$$\begin{array}{cc}\hfill {\beta }_S\left(t\right)& =\left[\begin{array}{ccc}{\beta }_{S1}^{+}\left(t\right)& \cdots & {\beta }_{S_N}^{+}\left(t\right)\\ {\beta }_{S1}^{-}\left(t\right)& \cdots & {\beta }_{S_N}^{-}\left(t\right)\end{array}\right]=\left[\begin{array}{c}\overrightarrow{{\beta }_S^{+}}{\left(t\right)}^{\top }\\ \overrightarrow{{\beta }_S^{-}}{\left(t\right)}^{\top }\end{array}\right]\hfill \end{array}$$

(2a)

$$\begin{array}{cc}\hfill {\overrightarrow{\beta }}_G{\left(t\right)}^{\top }& =\left[\begin{array}{cc}{\beta }_G^{-}\left(t\right)& {\beta }_G^{+}\left(t\right)\end{array}\right]\hfill \end{array}$$

(2b)

The optimisation objective, expressed by Equation (3), is to minimise the operation cost of a grid-connected MG. The main beneficiary can either be the MG owner or an aggregator depending on the proper energy price: a Real-Time Pricing in the case of the MG owner and the System Marginal Price for the aggregator. In this work, the MG owner is chosen as the end-user.

$$\begin{array}{c}\hfill OF=\min \sum _{t=1}^T\left[{\overrightarrow{j_G}}^{\top }·({\mathbf{E}}_S\left(t\right)·\overrightarrow{LCO{E}_S})-{\overrightarrow{E_G}}^{\top }\left(t\right)·\overrightarrow{RTP}\left(t\right)+{\overrightarrow{E_{PV}}}^{\top }\left(t\right)·\overrightarrow{LCO{E}_{PV}}\right]\end{array}$$

(3)

The costs that have been taken into account are: a) the energy importing cost, b) the energy exporting price and c) the Levelised Cost of Energy (LCOE) for the PV and the ESS modules. Regarding costs (a) and (b), an RTP scheme has been used in order to generalise the problem formulation, which is defined as $\overrightarrow{RTP}\left(t\right)={\left[RTP{\left(t\right)}^{-}RTP{\left(t\right)}^{+}\right]}^{\top },t\in [1,T]$. The LCOE calculations are done as in [56], since the usage of more elaborate schemes is considered to be out of the scope of the current study. All constraints are time-dependent since they should be applied in every time instant during the whole optimisation time horizon. Hence, for all the following equations: $t\in [1,T]$.

The first set of constraints in (4) expresses the equilibrium between energy supply and demand:

$${\overrightarrow{j_G}}^{\top }·({\mathbf{E}}_S\left(t\right)·\overrightarrow{j_S})+{\overrightarrow{E_G}}^{\top }\left(t\right)·\overrightarrow{j_G}+{\overrightarrow{E_{PV}}}^{\top }\left(t\right)·\overrightarrow{j_{PV}}=E_L\left(t\right)$$

(4)

The second constraint set (5) describes the vDERs minimum/maximum allowed limits, multiplied (using the ⊙ Hadamard, or the element-wise product) with the respective operation variables:

$$\begin{array}{cccc}\hfill \overrightarrow{E_S^{min}}\odot \overrightarrow{{\beta }_S^{+}}\left(t\right)& \le \overrightarrow{E_S^{+}}\left(t\right)\hfill & \hfill \le & \overrightarrow{E_S^{max}}\odot \overrightarrow{{\beta }_S^{+}}\left(t\right)\hfill \\ \hfill \overrightarrow{E_S^{min}}\odot \overrightarrow{{\beta }_S^{-}}\left(t\right)& \le \overrightarrow{E_S^{-}}\left(t\right)\hfill & \hfill \le & \overrightarrow{E_S^{max}}\odot \overrightarrow{{\beta }_S^{-}}\left(t\right)\hfill \\ \hfill \overrightarrow{E_G^{min}}\odot {\overrightarrow{\beta }}_G\left(t\right)& \le \overrightarrow{E_G}\left(t\right)\hfill & \hfill \le & \overrightarrow{E_G^{max}}\odot {\overrightarrow{\beta }}_G\left(t\right)\hfill \\ \hfill \overrightarrow{E_{PV}^{max}}\left(t\right)& \le \overrightarrow{E_{PV}}\left(t\right)\hfill & \hfill \le & \overrightarrow{0}\hfill \end{array}$$

(5)

The set of constraints in (6) and (7) ensure that at all times, the energy stored in each battery follows the natural continuity and also, that the energy during both charge and discharge operation, is defined by preset limits, according to charge and discharge C-rates:

$$\overrightarrow{E_S^{min}}\le \sum _{\tau =0}^{t-1}\left[\overrightarrow{E_S^{+}}\left(\tau \right)+\overrightarrow{E_S^{-}}\left(\tau \right)\right]+\overrightarrow{E_S^{+}}\left(t\right)+\overrightarrow{E_S^{-}}\left(t\right)\le \overrightarrow{E_S^{max}}$$

(6)

where, for $\tau =0$, the sum $\overrightarrow{E_S^{+}}\left(\tau \right)+\overrightarrow{E_S^{-}}\left(\tau \right)$ corresponds to the initially stored energy ($\overrightarrow{E_S}\left(0\right)$) for the N microgrid batteries right before the beginning of the optimisation horizon start, and these values are given as input to the algorithm.

$$\begin{array}{cc}\hfill \overrightarrow{0}& \le \overrightarrow{E_S^{+}}\left(t\right)\le \overrightarrow{R_C^{+}}\odot \overrightarrow{Cap}\odot \overrightarrow{DoD}·r_t\hfill \\ \hfill \overrightarrow{0}& \ge \overrightarrow{E_S^{-}}\left(t\right)\ge \overrightarrow{R_C^{-}}\odot \overrightarrow{Cap}\odot \overrightarrow{DoD}·(-r_t)\hfill \end{array}$$

(7)

The final constraint (8) dictates the mutually exclusive operation of the PCC and battery-related vDERs as follows:

$$\overrightarrow{{\beta }_S^{+}}\left(t\right)+\overrightarrow{{\beta }_S^{-}}\left(t\right)=\overrightarrow{{\beta }_G^{-}}\left(t\right)+\overrightarrow{{\beta }_G^{+}}\left(t\right)=\overrightarrow{j_T}$$

(8)

Since all optimisation expressions are linear and the optimisation variables are either continuous or binary, the optimisation problem is formed as an MILP problem. This is considered an asset, because LP/MILP problems are solved much faster than their non-linear equivalents. Moreover, re-running the optimisation algorithm can be achieved quickly, thus allowing for real-time applications. Since the OptiMEMS framework is accompanied by two auxiliary machine-learning based modules responsible for the forecast of the upcoming consumption and generation (presented briefly in the following subsection), there is no need to employ the Stochastic Optimisation approach since stochasticity is tackled in these two separate software modules.

2.2. System Architecture

2.2.1. Optimal Scheduling Engine for Microgrids

The OptiMEMS system architecture is depicted in Figure 1. It consists of software modules interconnected in the OptiMEMS platform, which interacts with web-based services in the cloud to fetch necessary input time series and also, with the MG assets for the setpoint signal dispatch and monitoring. The OptiMEMS platform is comprised of the following four sub-components: (a) one core module called Optimal Scheduling Engine for Microgrids (OSEM), which implements the Adjusted Unit Commitment Problem described in the previous section, (b) two forecasting engines providing the necessary production and generation time series forecasts to OSEM and finally, (c) the Real-Time Validator/Applicator of Optimal Schedule (RT-VAOS) module, which is responsible for the MG monitoring, dispatch of the produced setpoints and, whenever necessary, the trigger of recalculation of the optimal schedule. Automation-wise, all OptiMEMS components operate on a 24 h basis automatically, with 15-min granularity without human interference. In the following paragraphs, each module is described in terms of functional details, implementation and data flows.

The main component of OptiMEMS is the Optimal Scheduling Engine for MGs (OSEM) module, which formulates the optimisation problem described in the previous paragraph. OSEM is designed to produce an optimal schedule for either the whole day (24 h, 15 min of resolution), or upon request, from the next quarter of an hour till the end of the day. The OSEM engine is purely developed in Python 3.7. The mathematical formulation presented in Section 2.1 is realised with the help of the open-source library “PuLP” [57], while the MILP problem solver is the C++ open-source COIN-OR Branch and Cut (CBC) solver [58], characterised by high solving accuracy and computational speed. The choices of PuLP and CBC systems do not affect the functionalities of the OSEM module. Other optimisation frameworks, e.g., the usage of GUROBI, GAMS or MATLAB have also been tried; however, the authors chose the creation of an open-source optimisation engine.

The OSEM module needs three input variables: (a) the day-ahead RTP, (b) the load consumption and (c) PV production forecast time series, both with a predefined time granularity (e.g., quarterly, hourly etc.). When OptiMEMS runs at midnight, it receives the day-ahead forecasts, while, in case a recalculation of the optimised schedule is requested by RT-VAOS due to deviations, it receives the short-term forecasts, which correspond to a time series starting from the upcoming time slot up until the end of the day, and they are both updated periodically every hour so that even the latest deviations from normality are taken into consideration. The output of OSEM is a set of time series containing setpoints in terms of energy to be provided or consumed by the MG vDERs at each time slot of the optimisation horizon with the same granularity (e.g., quarterly), i.e., either day-ahead or for the remainder of the day in case recalculation is triggered.

2.2.2. Real-Time Validation and Application of Optimal Schedule

The module responsible for the application of day-ahead scheduling and coordination of its potential calibration is called “Real-Time Validation and Application of Optimal Schedule” (RT-VAOS). The term “real-time” is used in this work in order to describe the fact that at the moment of measurements logging (monitoring), control actions (either optimal schedule application or schedule bypass) are applied upon the microgrid inverters. RT-VAOS has a one-minute resolution, meaning that each minute measurements are monitored and control actions (following either optimal schedule or safety limits) are applied upon the microgrid inverters. The key feature of RT-VAOS is the use of real-time measurements to make the MG less vulnerable to possible errors deriving from forecasting tools. This component validates, in real-time, whether the current MG production and consumption are aligned or not (binary logic) with the pre-issued forecasted time series, within some tolerance limits. Their calculation has been performed via trial-and-error for various loading cases of the examined MG, leading to the creation of a customised “Tolerance Limits Look-up Table” (TL-LuT). In case of absence of significant deviations and if the MG DERs are capable of meeting the required energy demand, then appropriate setpoints are forwarded to the DER inverters. More specifically, RT-VAOS evaluates a series of predefined different scenarios in order to dispatch the proper signals. These scenarios are formed by the amount and type of vDERs defining the MG. RT-VAOS assigns each segment of the optimisation schedule in a specific case and calculates the forwarded setpoints to MG assets, which in all cases respect the battery’s safety limits.

In case of significant deviation of actual real-time overall load and/or PV measurements from the forecasted values, RT-VAOS enforces the “MG-AUTO” mode and consequently, demands recalculation of the optimised vDER scheduling for the remainder of the day, based on short-term forecasts. “MG-AUTO” operates the MG PV modules in their MPP, satisfies all connected loads and, whenever needed, charges the MG ESS, keeping them constantly fully charged. Any excess of energy is injected uncontrollably back to the grid. Figure 2 depicts the RT-VAOS algorithm flow chart.

Finally, RT-VAOS calculates the actual operating cost derived by the application of the optimal schedule, which, at the end of the day, is compared to the expected cost. This comparison is used for readjusting TL-LuT if necessary.

2.2.3. Load and PV Forecasting Engines

The MG load characteristics are highly related to seasonality, user behaviour and weather conditions. These parameters make load forecasting a challenging task, and extensive research has been applied to this area, a comprehensive review of which can be found in [3]. There is a rivalry between deterministic and machine-learning based approaches, with the latter prevailing in cases of limited knowledge, regarding each load, their types, the precise scheduling of MG users, etc. Thus, and due to the proposed OptiMEMS requirements, a machine-learning load-forecasting engine has been designed that exploits one-year-long real-time energy consumption measurements from an extensive smart-metering system of prime importance for a higher accuracy of the prediction. The engine produces a 24 h ahead load forecast with a 15-min resolution (long-term) once per day, at midnight. Day-ahead load forecast is considered a multi step time series forecasting problem; thus, multiple models for each time horizon are developed (Direct Multi-Step Forecast). Except for the load consumption, time features (hour, day, month, year) and temperature are also used as additional features in order to capture the seasonality and improve the efficiency of the module. Furthermore, this module has the functionality to run upon request in shorter time intervals (short-term, every quarter of an hour), thus reducing the mismatches between forecasted and real-time measurements. The load forecasting models are produced by using gradient boosting trees via the extreme gradient boosting (XGBoost) system [59]. For more information on the XGBoost model design, readers are encouraged to refer to the work of Xue et al. [60]. The design, implementation and testing of the load forecasting engine produced for the needs of OptiMEMS have been submitted by the authors for publication, and by the time this paper is written, it will be under review.

Regarding the PV forecast, a hybrid deterministic/machine-learning-based approach is adopted, as presented by the authors in previous works [61,62]. Briefly, the forecast engine is based on the open-source python library PVlib [63], adjustable to every PV installation, while two data-driven machine-learning models correct the forecasts PV production according to the expected weather forecast. More precisely, the first step is to feed a numerical weather prediction (NWP) to the PVlib engine in order to calculate the long-term (day-ahead) or short-term (few hours ahead) PV generation time series, with a 15-min time interval. The production curve is further improved by being fitted into two models that were trained utilising past data of actual and forecasted power generation. The first model takes into consideration the PV forecast errors with respect to the solar angles assuming ideal weather conditions, i.e., complete sunshine. The latter incorporates weather information concerning clouds, analysing past errors on different levels of cloudiness. Hence, the weather stochasticity and thus the PV volatile generation are taken into account within this sub-module. In terms of training the PV forecast models, a set of year-long data is required, which must contain an energy production time series from the PV modules of the selected MG. A detailed analysis regarding the hybrid deterministic/machine-learning-based PV forecasting module can be found in the recently published works of the authors of [61,62].

3. Experimental Validation

3.1. Experimental Setup

The application of the proposed OptiMEMS was realised on the CERTH/ITI SmartHome [54], a small-scale MG with AC-coupled DERs. This pilot is a real-life residential establishment with commercial (e.g., inverters) and custom equipment (e.g., smart meters, etc.) where various smart grid related technologies are evaluated for R&D purposes.

3.1.1. Microgrid Infrastructure

The MG (see Figure 3) consists of 9.57 kWp PV modules on a 10 kW inverter, a 5 kWh Li-ion battery with 3 single-phase 3 kVA inverters/chargers configured as Master-Slave-Slave in order to form a three-phase system, a variety of over 50 kWp controllable and non-controllable loads and a smart LV bus (fully controllable PCC). The system was configured to allow real-time monitoring and control on multiple levels using various wireless and wired communication protocols (e.g., MODBUS, TCP/IP). A detailed description of the CERTH/ITI SmartHome Microgrid can be found in [64]. This MG configuration allows control over the DER inverters as dictated by the proposed OptiMEMS under real conditions. Considering that the examined MG has AC-coupled DERs, and in order to highlight the performance of the algorithm without any load or PV curtailment, the mainly controlled assets are the battery inverters. A Raspberry Pi 3b running RT-VAOS was installed on-site in order to monitor and control the battery/PV inverters, along with the PCC.

3.1.2. Local Customisation—Web Services

As shown in Figure 1, both weather and RTP time series were fetched through online web-based Java APIs using RESTful services. The weather API collects current measurements every 15 min and the day-ahead predicted conditions every hour in order to be used as input at the two forecasting engines. The necessary variables are local temperature, irradiance (Diffuse Horizontal Irradiation—DHI, Direct Normal Irradiation—DNI, Global Horizontal Irradiation—GHI), relative cloud coverage in various atmospheric altitudes, relative humidity and wind speed. To improve prediction, two different weather forecasts were used (weatherbit.io and darksky.net), with empirically selected levels of trust, at 30% and 70%, respectively. The daily RTP scheme fed to the OSEM module was fetched by a German Utility Provider, namely Stadtwerk Haßfurt (eex.stwhas.de).

3.1.3. Experimental Scenarios

Towards providing not only proof-of-concept but also real-life experimental results, OptiMEMS was deployed on the aforementioned infrastructure over a period of nine days, during which the following scenarios were tested:

• Baseline Operation: A fully automatic state, without any optimisation scheme applied, aiming at maximising PV generation and keeping batteries fully charged.
• Scenario A—Optimal Day-Ahead Scheduling (“Opt Mode”): Right before midnight, OSEM creates the day-ahead optimal schedule as described in Section 2.2.1. Within the day and if the scheduling is applicable, the RT-VAOS follows that schedule and sends commands every minute to each asset with the appropriate setpoints.
• Scenario B—Adaptive Optimal Day-Ahead Scheduling (“reOpt Mode”): As in Scenario A, OSEM creates the day-ahead schedule for the examined day. RT-VAOS monitors the status of MG DERs. When an out-of-limits deviation occurs between actual and forecasted load, or PV generation, RT-VAOS triggers a recalculation of the schedule, for the remainder of the day (see Section 2.2.2).

Towards presenting financial metrics to better assess the performance of the proposed framework, Scenarios A and B are compared to each other, using the Baseline Operation as reference. Since OptiMEMS is deployed in a real-life establishment, it should be noted that it is not possible to apply Opt and reOpt mode on the same day. Thus, two different sets of days have been chosen; in particular, Scenario A (Opt) was applied during 15–17th of July, and Scenario B (reOpt), on 19, 22–23, 25–26 and 29th. In order to be able to extract comparable results in regards to the performance of OptiMEMS, scenarios A and B were deployed in days with similar weather and operational conditions. It should be noted that during all experiments, CERTH/ITI SmartHome users were unaware of the particular days that the experiment was taking place; thus, the load profile remained intact. Therefore, it can be considered that the OptiMEMS prototype has been successfully demonstrated in an operational environment (Technology Readiness Level—TRL: 7).

3.2. Experimental Results

To comprehend how OptiMEMS works, the MG scheduling and actual operation are demonstrated for two sample summer days: Figure 4 and Figure 5 depict the MG operation under Experimental Scenario A on the 15th of July, while Figure 6 and Figure 7 show the MG operation under Scenario B on 22 July. To highlight the importance of MG operation monitoring and the adaptive nature of the optimal MG scheduling provided in Scenario B, the following interesting remark is made: the drop on the PV production curve shown with the black line in Figure 4b corresponds to the actual production, which was not accurately predicted by the PV forecasting engine in the day ahead horizon. Under Scenario A operation, the deviations between the PV forecast and actual production force the MG to bypass the originally optimal schedule calculated in the beginning of the day and turn to MG-AUTO during the slots of the deviations. This fact can be observed in Figure 5a, which shows the battery operation of that day: around the times 16:00–17:30, the battery was originally scheduled to charge (shown by the dashed black lines), however, due to the PV output drop, the OptiMEMS engine does not allow to proceed to charging so that unexpected expenses from excessive energy imports are avoided. At first glance, it is evident that in reOpt mode, the MG vDERs follow the dictated optimised schedule closer. In addition, in Figure 7a,b, the differentiation between Opt and reOpt modes is illustrated: until the first triggering at 09:30 a.m., i.e., request for a new optimised schedule (orange lines), the optimal schedule calculated at the beginning of the day (black dashed line) is identical to that actually applied (continuous grey line); however, after re-optimisation is triggered, the dispatched schedules diverge significantly. Overall, for these days, during Scenario A, the optimal scheduling was ignored by the RT-VAOS module and the MG was set to MG-AUTO mode in a total of 14–15-min slots. On the other hand, in Scenario B, the optimal schedule was not followed in only 11–15-min slots. It is interesting that the merits of reOpt mode are not entirely based upon “missing” fewer time slots but on the calculation of more accurate scheduling. This improved performance lies predominantly on the accuracy of the two short-term forecasts and is quantified using the Mean Absolute Error (MAE) of each forecasting engine (long-term and short-term forecasts for Scenario A and B, respectively) in

Table 2. MAE of load and PV forecasting tools (kW per hour quarter “kWh-q”).

Opt Dates	15/7	16/7	17/7				Mean Values
MAE Load	0.192	0.232	0.379				0.268
MAE PV	0.092	0.265	0.523				0.293
reOpt Dates	19/7	22/7	23/7	25/7	26/7	29/7	Mean Values
MAE Load	0.153	0.126	0.124	0.114	0.131	0.097	0.124
MAE PV	0.111	0.049	0.139	0.122	0.125	0.101	0.107

. Indeed, the average MAE for long-term load and PV forecast are 0.268 and 0.293, respectively, while in the case of short-term forecasts, they become 0.124 and 0.107, i.e., relative reductions of MAE equal to $(0.268-0.124)/0.268\times 100\%=53.73\%$ and $(0.293-0.107)/0.293\times 100\%=63.48\%$.

In order to assess the overall performance of Opt and reOpt Modes throughout the experiments, in Figure 8a, a key performance indicator (KPI) named “MG_A” (%) is defined as the relative error of actually achieved to expected daily profits, with respect to baseline daily costs (9). As observed, when the MG operated in Opt mode, this KPI is always negative, meaning that it is impossible to reach the initially set goal since time slots are being “lost” when the schedule is disregarded (MG-AUTO). On the contrary, on certain days, reOpt mode may lead to positive values of the MG_A KPI, meaning that the actually achieved daily profit was eventually greater than what originally predicted, a fact justified by the improved accuracy of both short-term forecasts as this derives from their re-calibration upon real-time measurements. This observation highlights the superiority of reOpt mode over the Opt mode. This statement is further strengthened by the analysis of daily cost reductions relative to the baseline daily costs (Figure 8b). Indeed, in both Opt and reOpt modes, the overall MG cost is less than what it would be without any optimisation. Specifically, Opt mode leads to approximately 5.5% daily cost reduction, whereas by employing reOpt mode, cost reduction fluctuates between 6% and 13%, with an average value of 9%. Additionally, for all Scenario B days, the daily cost reduction that would be reached if Opt mode was applied is calculated for comparison reasons. For example, during the demonstrated day with Scenario B, the daily profit was reached €0.198 (7.11% relative profit), while if Scenario A was applied on the same day, the daily profit would be equal to €0.09 (3.25% relative profit). Generally, as seen in Figure 8b, reOpt mode leads to 1.5–2 times greater relative cost reductions compared to Opt mode.

$$M{G}_A=\frac{{\Pi }_a-{\Pi }_e}{{\Pi }_a}·100$$

(9)

4. Conclusions

This work presented the design and application of an optimal scheduling scheme, tailored to grid-connected, fully RES-based MG, with AC-coupled DERs. The core OSEM module is accompanied by load and PV forecasting engines, cloud services APIs and the RT-VAOS module controls of commercially available inverters. The OptiMEMS framework has been applied successfully and continuously on a real-life MG over nine summer days, and the results of its application are presented. Initial findings are very encouraging with respect to minimising daily cost during grid-connected mode and, at the same time, thanks to the developed real-time monitoring system, ensuring proper MG operation even in the case of deviations between forecasted and real-time measurements. A comparative analysis of the two possible operation modes, day-ahead-Opt mode/intra-day-reOpt mode clearly indicate the predominance of the adaptive scheduling logic. Another significant merit of the proposed methodology is the applicability of the overall framework on real world MGs due to (a) OptiMEMS coupling with commercial inverters, (b) limited infrastructure prerequisites (internet access, one computer and a Raspberry Pi) and minimal computational requirements for all developed software.

Furthermore, OptiMEMS is easily expandable, not only in terms of numbers of controllable MG DERs but also through additional functionalities, such as the inclusion of optimal vehicle-to-grid operation, a feature soon to be available by the authors as an expansion pack. Future work involves optimal load shedding capabilities for islanded operation and testing more optimisation techniques (e.g., metaheuristic techniques, or robust optimisation) for further improving the final outcome. By integrating these extra features, OptiMEMS efficiency is expected to increase, providing an even more effective tool for MG management. Finally, it is noteworthy that throughout the testing phase, it became evident that the accuracy, and thus, the overall framework effectiveness heavily depends on accurate forecasting engines. Thus, relevant research endeavours should intensify.