Jump Linear Quadratic Control for Microgrids with Commercial Loads

: Due to the aging power-grid infrastructure and increased usage of renewable energies, microgrids ( µ Grids) have emerged as a promising paradigm. It is reasonable to expect that they will become one of the fundamental building blocks of a smart grid, since e ﬀ ective energy transfer and coordination of µ Grids could help maintain the stability and reliability of the regional large-scale power-grid. From the control perspective, one of the key objectives of µ Grids is load management using local generation and storage for optimized performance. Accomplishing this task can be challenging, however, particularly in situations where local generation is unpredictable both in quality and in availability. This paper proposes to address that problem by developing a new optimal energy management scheme, which meets the requirements of supply and demand. The method that will be described in the following models µ Grids as a stochastic hybrid dynamic system. Jump linear theory is used to maximize storage and renewable energy usage, and Markov chain theory is applied to model the intermittent generation of renewable energy based on real data. Although the model itself is quite general, we will focus exclusively on solar energy, and will deﬁne the performance measure accordingly. We will demonstrate that the optimal solution in this case is a state feedback law with a piecewise constant gain. Simulation results are provided to illustrate the e ﬀ ectiveness of such an approach.


Introduction
The power industry landscape is changing rapidly, due to the emergence of Distributed Energy Resources (DERs) and new storage technologies, active demand-side participation, and the growing need for green/renewable energy. At the same time, it is clear that the grid infrastructure is aging, and that maintaining its resiliency and reliability will require significant resources. These two factors have created the need to re-examine the existing power-grid structure (which is now over 100 years old).
The proliferation of microgrids (µGrids) has been steadily growing in recent years, and this is likely to be an important part of the grid of the future. It is widely believed that the next generation of µGrids will not be used just for supplying backup power, and will also provide a complex network configuration that possesses all the essential attributes of a large-scale grid (such as the ability to balance electrical demand with sources, schedule the dispatch of resources and maintain grid reliability). It is reasonable to expect that these new µGrids will enable greater use of DERs, limit greenhouse gas emissions, improve local grid reliability and reduce operating costs [1]. We should also mention in this context that µGrids can become a potentially useful tool for managing the variability of intermittent Renewable Energy Sources (RESs) such as solar energy, due to the decreasing cost of DERs [1].
Considering that 90% of all power outages and disturbances originate in the distribution system due to component failures and load imbalances [2], and recognizing that occurrences of natural disasters have increased in the last few years [3], it is imperative to move towards a decentralized grid that allows for more flexibility and scalability. It is becoming increasingly clear that the grid of the future will involve a considerable amount of "plug and play", in which µGrids will play an essential role (particularly when it comes to optimized interoperability) [4]. The power industry has already embarked on this path, and has considered DERs as a viable alternative to centralized generation [5].
Integration of DER units in general (and µGrids in particular) gives rise to several operational challenges, such as bidirectional power flows, potential instability, uncertainty and low inertia [6]. A considerable amount of research is currently being conducted on the design and management of µGrids, and the problems that arise when they are connected to the grid. The solutions that have been proposed so far take many different forms, and depend on how the µGrid is being used. In general, it is possible to classify these techniques into three broad categories, based on the type of problem that they consider. Some of them deal with utility µGrids (where a city district operates as a µGrid), others with commercial and industrial µGrids (such as commercial buildings, university campuses, factories, and manufacturing plants, for example) and there are also those that focus on remote µGrids (which operate only in an islanded mode) [7]. The method proposed in this paper is aimed exclusively at commercial µGrids, since there is a considerable amount of publicly available data that pertains to their load patterns.
Energy management in µGrids is usually considered to be a hierarchical control system with three levels [8,9], the first of which is known as primary control. Primary control is concerned with the interface between generators, storage devices, and loads. The control actions are based on local measurements, and must be performed quickly (often within a few milliseconds). Secondary control (which is also referred to as local energy management of individual µGrids) is responsible for correcting the errors produced by the primary control. It acts over a longer period of time, and ensures the reliable operation of µGrids. Tertiary control (which we will focus on in this paper) coordinates the operation of multiple µGrids that interact with each other, as well as with the utility grid. It regulates active and reactive power flow, and supplies secondary control units with reference voltages. By doing so, tertiary control provides an optimal power schedule which allows individual µGrids to use their resources in the most effective way.
Because standard energy management schemes for µGrids were typically based on deterministic control strategies and optimization techniques, they were not able to manage random energy generation efficiently [10][11][12][13][14][15][16][17][18][19]. In [10][11][12] decentralized control was considered to be a strategy for energy management of µGrids. A robust decentralized servomechanism controller for power management of an autonomous multi-DER was analyzed in [10], and a decentralized bi-level algorithm for energy management of networked µGrids was considered in [11]. In [12], decentralized control schemes were proposed for µGrids to maintain the desired temperature in local buildings at a minimum cost.
Reference [13] examines a centralized approach for controlling the power flow in hybrid µGrids, and [14] uses dynamic programing to control energy storage devices by solving the optimal power flow problem on the tertiary control level. State of the art of control strategies and the control challenges that they introduce for distributed energy storage systems in µGrids are discussed in [15]. Parisio et al. [16] apply model predictive control by using a mixed-integer linear programming formulation to optimize µGrids operation, while [17] considers a scenario-based robust energy management method for uncertain renewable generation and load. In this context, we should also mention references [18] and [19], which use an alternating-direction multiplier method to optimize the operation of multiple µGrids.
Although these deterministic solutions were important in the planning of µGrid resources over time, increased use of intermittent renewable energy has encouraged researchers to study µGrid energy scheduling in a stochastic framework. Several such algorithms have been developed in recent years [20][21][22][23][24][25][26], each of which takes into account the uncertainty of non-dispatchable generation and Energies 2020, 13, 4997 3 of 21 loads. It should be noted that in most of these cases, the proposed solutions rely on a stochastic programming approach, and the intermittent RESs are described using empirical data (or are modeled as Gaussian processes). In [20,21], for example, energy scheduling of µGrids with integrated RES is formulated as a stochastic problem, and a two-stage stochastic optimization is proposed for day ahead transactions and dispatching. The focus in [22] is an online energy management strategy for real-time operation of µGrids. In this paper, online energy management is modeled as a stochastic optimal power flow problem using a Lyapunov optimization algorithm. Han et al. [23] give an overview of distributed coordinated control strategies for µGrids with high integration of RES, which are based on multiagent systems. In [24,25], a hierarchical control structure is proposed which uses stochastic model predictive control for energy management of individual µGrids, and reference [26] considers a one-step method for optimizing the long term operation cost of a µGrid. An interesting feature of this method is that it does not require statistical knowledge of uncertainties in the load, the renewable energy generation or electricity prices. It is important to keep in mind that all the methods mentioned above rely on iterative techniques to find a viable solution. In that respect they are different from the approach proposed in this paper.
Papers that have used Markov processes to model random RES and battery energy storage systems are of particular interest for this study. In [27], a semi-Markov process model was proposed to model photovoltaic (PV) power and a rule-based controller was used to compute average generator and battery power during each scheduling interval. Belloni et al. [28], use Markov chains to model uncertainties in the RES generation, and propose a stochastic dynamic programming algorithm to minimize the cost of energy consumption in wind powered µGrids with energy storage system. In [29], a Markov jump process was used to model the stochastic changes in distributed energy storage systems.
In [30], a stochastic model predictive control was applied to design optimal control for wind turbines that are subject to random wind speed. The authors formulated a Markov jump linearization model to describe random switching characteristics of the wind turbine. Markov chains have also been used by Li and Roche [31] to develop a two-stage scheduling algorithm for multi-energy supply microgrids, and an online learning prediction method was introduced to anticipate the short-term load demands and renewable outputs in real-time dispatching. In a more recent work [32], a solar generation model was proposed based on the Markov chain approach, which predicts the power generation of solar cells.
From the above discussion, it is clear that energy management systems for µGrids need to be designed in a way that reduces the impact of load variations, natural disasters, and the inherent randomness of RES. The difficulty, however, is that most of the energy management strategies proposed to date use online optimization techniques that are computationally demanding and costly to implement. To address this issue, we propose an optimal control scheme which relies on a simplified model that represents multiple µGrids as multivariate stochastic hybrid dynamic systems. The controller design is based on jump linear theory, which is used to maximize storage and renewable energy usage.
It is important to recognize in this context that the proposed approach does not attempt to predict weather parameters (such as the availability of solar or wind energy, for example) on a daily basis [32][33][34]. Instead, it relies on yearly regional weather data to build a probabilistic model that is used to compute appropriate control gains offline.
The main contribution of this paper is the application of Jump Linear Quadratic Control (JLQC) to a model that combines deterministic and stochastic elements. This hybrid representation (coupled with a quadratic cost function) provides the necessary mathematical framework for optimizing the use of storage and solar energy by piecewise control action. An important feature of this control strategy is that it does not require online optimization. As a result, it is computationally efficient, and is relatively easy to implement compared to other techniques proposed in the literature. This is particularly important in cases where the system includes many interacting microgrids.
The paper is organized as follows. In Section 2, we present a simplified model for a flexible grid structure. We then proceed to develop a stochastic optimal control strategy, which is described in Energies 2020, 13, 4997 4 of 21 Section 3. Simulation results and their analysis are provided in Section 4, while Section 5 offers some conclusions (as well as a brief discussion of possible directions for future work).

System Description and Mathematical Modeling
We begin this section, by offering a general system description that addresses some recent challenges in power-grid energy management. We will then introduce a simple mathematical model for the grid that will allow us to develop a stochastic optimal control strategy. Figure 1 provides a schematic representation of multiple µGrids that can be connected to and disconnected from the utility grid via a medium voltage (MV) distribution substation. In this configuration, µGrids that are connected can be viewed as a cluster that is linked to the grid through a point of common coupling (PCC), while those that are disconnected are considered to be islanded. A continuous dynamic model for structures of this type will be proposed in the next section. Depending on the grid requirements, a communication link may exist between the grid and the µGrids for the exchange of information.

The System Description
Energies 2020, 13, x FOR PEER REVIEW 4 of 20 Section 3. Simulation results and their analysis are provided in Section 4, while Section 5 offers some conclusions (as well as a brief discussion of possible directions for future work).

System Description and Mathematical Modeling
We begin this section, by offering a general system description that addresses some recent challenges in power-grid energy management. We will then introduce a simple mathematical model for the grid that will allow us to develop a stochastic optimal control strategy. Figure 1 provides a schematic representation of multiple Grids that can be connected to and disconnected from the utility grid via a medium voltage (MV) distribution substation. In this configuration, Grids that are connected can be viewed as a cluster that is linked to the grid through a point of common coupling (PCC), while those that are disconnected are considered to be islanded. A continuous dynamic model for structures of this type will be proposed in the next section. Depending on the grid requirements, a communication link may exist between the grid and the Grids for the exchange of information. As noted in the Introduction, this study will focus on commercial Grids (including university campuses). Systems of this sort have two important features that allow us to develop an effective and robust control strategy:

The System Description
• Although each Grid load may be comprised of multiple buildings, their aggregated power has a predictable profile • The load demand reaches its peak during the day (when solar energy is available), and reaches its minimum at night.
Data that validates these two points is provided in Figure 2, which shows the hourly power usage for a typical commercial load [35] and for Santa Clara University [36]. The basic layout for a system with multiple µGrids that can connect and disconnect from the utility grid.
As noted in the Introduction, this study will focus on commercial µGrids (including university campuses). Systems of this sort have two important features that allow us to develop an effective and robust control strategy:

•
Although each µGrid load may be comprised of multiple buildings, their aggregated power has a predictable profile • The load demand reaches its peak during the day (when solar energy is available), and reaches its minimum at night.
Data that validates these two points is provided in Figure 2, which shows the hourly power usage for a typical commercial load [35] and for Santa Clara University [36].
In modeling the dynamic behavior of µGrids that can operate both in the islanded mode and in the connected mode, we will not consider the possibility of bidirectional energy transfer between the utility grid and µGrids. This assumption helps prevent grid instability due to the stochastic nature of the net load, which represents the difference between the total load and the solar generation (in this instance and throughout the paper, the term "stability" is understood in the sense of Lyapunov). In modeling the dynamic behavior of Grids that can operate both in the islanded mode and in the connected mode, we will not consider the possibility of bidirectional energy transfer between the utility grid and Grids. This assumption helps prevent grid instability due to the stochastic nature of the net load, which represents the difference between the total load and the solar generation (in this instance and throughout the paper, the term "stability" is understood in the sense of Lyapunov).
Each cluster of Grids can be connected to or disconnected from the grid in a scheduled or random manner, depending on their load and their generation needs. We will assume that islanded Grids are controlled using only locally available information, and our objective will be to stabilize each disconnected Grid using a stochastic optimal control law. This control law must ensure that stability (and an acceptable level of suboptimality) is retained when one or more Grids are reconnected to the grid. Transitions of this sort are bound to occur, since Grids are not designed to operate indefinitely in isolation. Figure 3 shows the basic structure of each Grid, which consists of a specific load (commercial or a university campus), battery storage, and an array of solar cells. When Grid operates in the connected mode, the utility grid provides energy both to the load and to the battery. Unused solar energy by the load may also charge the battery if needed. When it is in the islanded mode, solar energy is provided internally, and is distributed to the battery and the load in a way that reflects the load requirements. During the night, it is assumed that the battery is the sole source of energy. This is a reasonable assumption, since most large consumers require less energy during the night than during the day (see Figure 2). In situations where the required battery storage is not available, a secondary generation source (such as fuel cell, for example) may be considered. This addition has little effect on the complexity of the model and the control strategy, so we will disregard it in our analysis. Each cluster of µGrids can be connected to or disconnected from the grid in a scheduled or random manner, depending on their load and their generation needs. We will assume that islanded µGrids are controlled using only locally available information, and our objective will be to stabilize each disconnected µGrid using a stochastic optimal control law. This control law must ensure that stability (and an acceptable level of suboptimality) is retained when one or more µGrids are reconnected to the grid. Transitions of this sort are bound to occur, since µGrids are not designed to operate indefinitely in isolation. Figure 3 shows the basic structure of each µGrid, which consists of a specific load (commercial or a university campus), battery storage, and an array of solar cells. When µGrid operates in the connected mode, the utility grid provides energy both to the load and to the battery. Unused solar energy by the load may also charge the battery if needed. When it is in the islanded mode, solar energy is provided internally, and is distributed to the battery and the load in a way that reflects the load requirements. During the night, it is assumed that the battery is the sole source of energy. This is a reasonable assumption, since most large consumers require less energy during the night than during the day (see Figure 2). In situations where the required battery storage is not available, a secondary generation source (such as fuel cell, for example) may be considered. This addition has little effect on the complexity of the model and the control strategy, so we will disregard it in our analysis. In modeling the dynamic behavior of Grids that can operate both in the islanded mode and in the connected mode, we will not consider the possibility of bidirectional energy transfer between the utility grid and Grids. This assumption helps prevent grid instability due to the stochastic nature of the net load, which represents the difference between the total load and the solar generation (in this instance and throughout the paper, the term "stability" is understood in the sense of Lyapunov).
Each cluster of Grids can be connected to or disconnected from the grid in a scheduled or random manner, depending on their load and their generation needs. We will assume that islanded Grids are controlled using only locally available information, and our objective will be to stabilize each disconnected Grid using a stochastic optimal control law. This control law must ensure that stability (and an acceptable level of suboptimality) is retained when one or more Grids are reconnected to the grid. Transitions of this sort are bound to occur, since Grids are not designed to operate indefinitely in isolation. Figure 3 shows the basic structure of each Grid, which consists of a specific load (commercial or a university campus), battery storage, and an array of solar cells. When Grid operates in the connected mode, the utility grid provides energy both to the load and to the battery. Unused solar energy by the load may also charge the battery if needed. When it is in the islanded mode, solar energy is provided internally, and is distributed to the battery and the load in a way that reflects the load requirements. During the night, it is assumed that the battery is the sole source of energy. This is a reasonable assumption, since most large consumers require less energy during the night than during the day (see Figure 2). In situations where the required battery storage is not available, a secondary generation source (such as fuel cell, for example) may be considered. This addition has little effect on the complexity of the model and the control strategy, so we will disregard it in our analysis.  In the simplified scenario that we just described, the primary goal is to manage the energy demand. This is done by ensuring that the power generated by the utility grid (P grid ), the PV array (P solar ) and the battery (P B ) is balanced with the power required by the load (P load ). We can express this relationship as: It is assumed that P load (t) follows a preassigned trajectory, whose characteristics will be defined in the following section. P solar (t) (which represents the power generated by the PV array) can be viewed as the result of a stochastic process whose characteristics depend on cloud coverage as well as the time of day. Since the battery can deliver power to the load (or can absorb it when charging), we will treat P B (t) as a control input. Power losses that are mostly due to transmission lines are neglected in this model, since µGrids are considered to be sufficiently close to substations (less than 10 miles).
The PV array should be designed in a way that reflects the daytime load requirements (which is becoming increasingly feasible, due to decreasing cost of solar panels). Doing so ensures full load satisfaction when the weather is sunny, and the excess energy can be transferred to the battery. Because of that, the battery can act as a buffer, which allows the system to use the stored solar energy when the PV array is inactive.
In addition to the relationship described in (1), our model incorporates the following three assumptions: • Information about the system state is readily available, including instantaneous power flow, acceptable limits for generation levels, and maximum and minimum battery levels. This can be ensured by using a State of Charge (SoC) tracker for the battery, in conjunction with voltage, current, or phasor monitoring devices.

•
Only active power is considered, and voltages and phase angles are assumed to be regulated by controllers/inverters on the PV array and the battery.

•
The load for each µGrid conforms to the pattern provided in the National Renewable Energy Laboratory (NREL) System Advisor Model (SAM) dataset. This data can be found online [35], and represents the "average" commercial load in the United States.

Solar Generation Model
Although solar power is clearly a desirable form of generation (because of its passive generation profile and the decreasing cost of panels), it has certain intrinsic shortcomings which stem from its intermittent nature. The most important one is that PV generation requires sunny weather for maximal efficiency, which means that generation can be severely impacted by heavy cloud coverage. To model the stochastic nature of solar generation, we will use a continuous Markov chain to describe cloud coverage patterns (following the methodology proposed in [37]).
In general, cloud coverage classification should be based on regional weather patterns. As shown in Figure 4, in northern California, the level of cloud coverage during the day can be classified as sunny, partially cloudy, and overcast. Since there is no solar generation at night, the power generated by the PV array can be represented as a piecewise constant function: (2)  he term ( ) that appears in (2) represents the mode of the Markov chain that correspon vel of cloud coverage. This means that r(t) evolves according to a continuous time Markov c restricted to three possible values. The transition probability matrix P indicating the probab transitioning from state i to state j, = { } is given by: Coefficients represent the (non-negative) transition rate from ), and is defined as iven this notation, the transition rate matrix Π will have the form Π = ( ) ( , = 1,2,3) ata for the cloud coverage transition probability matrix was obtained from San ational Airport's meteorological reports [39], (also known as METAR). This data con ation about precipitation and cloud coverage, and is routinely collected at all major airp overnment buildings in the United States. For the purposes of our study the hourly c age data was "quantized" in a way that differentiates between three levels: sunny, part y, and overcast. The transition probability and the transition rates were calculated using et (which contains information gathered between 2008 and 2017). he data analysis confirmed that the transition probability from one state to the other state ange significantly from year to year. The transition probability matrix obtained for 20 n in Figure 5. It is interesting to note how the magnitudes on the main diagonal compare t agonal elements in the transition matrix. It is evident that the cloud coverage tends to rem  The actual values that P solar (r(t)) takes depend on the number of panels, efficiency, and the cloud coverage.
The term r(t) that appears in (2) represents the mode of the Markov chain that corresponds to the level of cloud coverage. This means that r(t) evolves according to a continuous time Markov chain, and is restricted to three possible values. The transition probability matrix P indicating the probability p ij of transitioning from state i to state j, P = p ij is given by: where ∆ > 0 and lim ∆→0 0(∆) ∆ = 0. Coefficients π ij represent the (non-negative) transition rate from i to j (i j), and π i is defined as (4) Given this notation, the transition rate matrix Π will have the form Π = (π ij ) (i, j = 1, 2, 3) [38]. Data for the cloud coverage transition probability matrix was obtained from San Jose International Airport's meteorological reports [39], (also known as METAR). This data contains information about precipitation and cloud coverage, and is routinely collected at all major airports and government buildings in the United States. For the purposes of our study the hourly cloud coverage data was "quantized" in a way that differentiates between three levels: sunny, partially cloudy, and overcast. The transition probability and the transition rates were calculated using this data set (which contains information gathered between 2008 and 2017).
The data analysis confirmed that the transition probability from one state to the other state does not change significantly from year to year. The transition probability matrix obtained for 2016 is shown in Figure 5. It is interesting to note how the magnitudes on the main diagonal compare to the off-diagonal elements in the transition matrix. It is evident that the cloud coverage tends to remain in one state for prolonged periods of time. These long stretches of continuous weather are clearly consistent with California's stable weather patterns.
Energies 2020, 13, 4997 8 of 21 hown in Figure 5. It is interesting to note how the magnitudes on the main diagonal compare to the ff-diagonal elements in the transition matrix. It is evident that the cloud coverage tends to remain n one state for prolonged periods of time. These long stretches of continuous weather are clearly onsistent with California's stable weather patterns.

Battery Model
Battery energy storage systems are commonly used to supplement generation from the PV array, (which is inherently intermittent). Relevant battery characteristics include its energy capacity (more specifically SoC), discharge/charge powers, safe operating temperatures, and life cycle. The SoC of a battery represents its available capacity expressed as a percentage of its rated capacity. Since depleting or overcharging the battery can have detrimental effects, the SoC of the battery should always be maintained within proper limits. Voltage and current measurements can be used to estimate SoC, since they cannot be measured directly. In the following, we will assume that the energy stored in the battery, E B (t), and the charge power P B (t) can be used to provide adequate information about the SoC. The battery energy storage has the following limits, E Bmin ≤ E B (t) ≤ E Bmax , as well as discharging and charging limits, 0 ≤ P dis.
B . In the second and third inequality, P ch.
B and P dis. B represent the maximal allowable charge and discharge values, respectively. Variations in the energy stored in the battery can be expressed as [14] dE B (t) dt = P ch.
where P ch. B = P solar (t) + P grid (t), and γ denotes the rate of self-discharge. We will assume that batteries have the same efficiency when charging and discharging, and that they can do so concurrently. This feature becomes important in cases when µGrids exchange energy directly. To ensure normal battery operation (and increase its lifespan), it is standard practice to maximize the lifespan of the battery by setting E Bmin ≥ 10% and E Bmax ≤ 90% of the maximum energy that can be stored.

Simplified Mathematical Model
A simplified model that describes the energy management of a single µGrid in an n µGrid system has the form: In (6), E Bm represents energy stored in battery m, P Lm is the load demand of µGrid m, P solarm denotes solar power generated for µGrid m, and P gridm is the amount of power delivered from the Energies 2020, 13, 4997 9 of 21 utility grid to µGrid m. The quantity P discm represents the power delivered from the battery to the load. Parameter α m is the fraction of solar power delivered to load m, and β m is the fraction of power delivered from the main grid to µGrid m (it is assumed that any unused energy is absorbed by the battery). The values of these two constants are chosen based on the solar panel size and battery storage capacity.
Coefficient γ m denotes the rate of self-charge for battery m, and P grid (t) is the power allocated by the utility grid for the n µGrids. This power can be expressed as P grid (t) = P grid1 (t) + P grid2 (t) + · · · + P gridn (t) (7) and can be bounded as 0 ≤ P grid (t) ≤ P M G . To balance the power in the network, P gridm must satisfy where vector ψ = [θ mp ] represents the tuning coefficients for µGrid m and component p (which can be a battery or a load). Vector E = [E Bl (t), E Ll (t)] T , (l = 1, . . . , n; l m), consists of components E Bl and E Ll , which denote the energy stored in battery l and the energy absorbed by load l, respectively. In the state space, the mathematical model described above can be represented as with with m = 1, . . . , n; l = 1, . . . , n; l m. In this expression (which uses the fact that P L (t) = . E L (t)), x 2k−1 and x 2k denote the energy stored by battery k, and the energy demand in µGrid k, respectively. The components of vector u represent the power delivered by each battery, and are considered to be control variables.
The system described in (9) can be represented in compact form as .
When µGrids switch to islanded mode, this system becomes decoupled, and A reduces to a 2n × 2n block diagonal matrix To illustrate the feasibility of the ideas proposed in Section 2.2, we will consider the simplified configuration shown in Figure 6, where a pair of µGrids (n = 2) is connected to the utility grid. µGrids in this diagram can be connected or disconnected from the utility grid at any point in time, and must be able to operate for a few days in islanded mode.
When Grids switch to islanded mode, this system becomes decoupled, and A reduces to a 2 × 2 block diagonal matrix To illustrate the feasibility of the ideas proposed in Section 2.2, we will consider the simplified configuration shown in Figure 6, where a pair of Grids (n = 2) is connected to the utility grid. Grids in this diagram can be connected or disconnected from the utility grid at any point in time, and must be able to operate for a few days in islanded mode. Figure 6. Basic layout for a system with two Grids. Figure 6. Basic layout for a system with two µGrids.
In the special case when we have only two µGrids, the proposed model for the energy management has the form x 4 (t) = α 2 P solar2 (t) + β 2 P grid2 (t) + u 2 (t) (11) with P grid (t) = P grid1 (t) + P grid2 (t). When P grid (t) = 0, both µGrids are in islanded mode. For the purposes of this study, we will assume that P grid (t) = P M G , where P M G is a known constant. To balance the power in the network, P grid1 and P grid2 must satisfy where These coefficients are computed in such a way that (12) and (13) are guaranteed to be satisfied. What this means is that the power delivered by the utility to the common bus is shared between the two µGrids (see Figure 6).
Replacing P grid1 and P grid2 in (11) by the expressions given in (12), we obtain the following interconnected system: Energies 2020, 13, 4997 In this expression, x 1 denotes the energy stored by battery 1, x 2 is the energy demand in µGrid1, x 3 is the energy stored by battery 2 and x 4 is the energy demand in µGrid2. Inputs u 1 and u 2 represent the power delivered by each battery, and are considered to be control variables.
System (14) can be equivalently represented as .
A more compact description of this system has the form where In this model, x denotes the system state, u = (u 1 u 2 ) T represents the control action, y is the output, and P corresponds to the power delivered by the grid and the solar panels, a fraction of which goes to the battery.
When µGrids 1 and 2 switch to the islanded mode the system becomes decoupled, and matrices A, B, C and P take the form: where I denotes a 2 × 2 identity matrix. Microgrids 1 and 2 can therefore be modeled as two independent subsystems . z 1 (t) = A 11 z 1 (t) + B 1 u 1 (t) + P 1 (t) y 1 (t) = z 1 (t) (17) . with For the purposes of this study, we will assume that the µGrids are not connected (this is not necessary, but it simplifies the analysis considerably). It should be observed that the system described in (16) represents an interconnection of the two subsystems in (17) and (18).
As discussed in the previous section the solar generation can be modeled as a continuous time Markov chain with three states-sunny, partly cloudy, and overcast. In view of that, we can model the power delivered to the system as For notational simplicity, we will use r(t) = i as superscripts to represent different Markov states, where i = 1, 2, 3. It is assumed that P i solarm takes different constant values during the day depending on the state i, and that it is zero at night. We will assume that the solar radiation is constant during the day. Coefficients α i m and β i m are weather dependent, and represent fractions of the power that is transferred. Since solar energy production can be viewed as a continuous Markov chain with 3 states, the power delivered (P i ) will be piecewise constant, with the changes coinciding with jumps. Thus, the system represented by (16) can be structured as a continuous linear time invariant system with Markovian jumps and a hybrid (continuous-discrete) state space [x i] T . Such a model has the form . with where matrix A is piecewise constant and changes as the system jumps from one state of solar energy to the next state. Matrices B and C do not depend on solar generation states and are the same as in (15). P i is the same as in (19). In the following section, we will show how this model can be used to design an optimal stochastic controller for the system to optimize the energy usage of µGrids.

The Jump Linear Quadratic Control (JLQC) Formulation
As we noted earlier, the ability to connect and disconnect µGrids from utility grid is potentially beneficial, since it can increase the use of renewable energy by maintaining stability and improving Energies 2020, 13, 4997 13 of 21 the robustness of the system. In this section, we describe a JLQC strategy that can help achieve this objective.
One of the advantages of the Markov chain approach is that it makes the resulting model more tractable mathematically. It also allows us to use the existing body of work in this field [37,38,40,41]. In the following we will briefly describe the optimal regulator used in this analysis, and will subsequently transform the model developed in Section 2.2 so that JLQC control can be applied directly.

An Overview of Jump Linear Systems
In the state-space domain, a jump linear system can be described as .
where x(t) ∈ R n and u(t) ∈ R m denote the state and the input vectors, respectively. In this model, A(r(t)) and B(r(t)) are n × n and n × m matrices, respectively, and r(t) represents the mode in which the system operates (which is determined by a finite state Markov jump process). To simplify the notation, in the following we will refer to matrices [A(r(t)), B(r(t))] r(t)=i as [A i , B i ] when the system operates in the i th mode [40]. Since r(t) reflects the cloud coverage during the day, it can be viewed as a random variable whose evolution is governed by a continuous time Markov chain (as discussed in Section 2.1.2).
To design an optimal controller for systems of this type, it is desirable to minimize a cost function of the form where t 0 and t f denote the initial and final time, respectively, and E{.} represents the expected value [38]. Matrices, Q(r(t)) and R(r(t)), depend on the mode in which the system operates, and can be used to ensure that its response meets a given set of performance criteria. In the following, we will denote these matrices as [Q i , R i ] when the system is operating in its i th mode, with the added assumption that Q i ≥ 0 and R i > 0 and symmetric.
For the finite horizon problem with the cost function given in (22), the optimal controller can be obtained using the stochastic maximum principle and has the general form where matrices K i (t) (i = 1, 2, 3) satisfy the set of coupled differential matrix Riccati equations with S i = B i R −1 i B T i and K i t f = 0. In this study, we will be primarily interested in the steady state values of K i (t). When stochastic controllability and observability conditions are satisfied [40], matrices K i (t) will converge to the unique positive definite solutions of system Coupled matrix equations of this sort can be solved using the numerical algorithm proposed in [42]. The optimal controller that is obtained in this manner minimizes the cost function (26) and the resulting control law becomes:

Control Strategy for Connected and Islanded µGrids
Our objective in the following will be to determine control laws that optimize the power needed to satisfy the energy demand of the two µGrids (this accounts for the coordination of their operation both in the connected and in the islanded mode). We will provide a separate mathematical model for each mode, taking into account the requirements of the JLQC paradigm.

The Connected Mode
To adapt our model in the standard framework proposed in [38], we will replace with the piecewise constant vector P i in (19) using the change of variables x(t) = x(t) − x i 0 and x i 0 represents the desired energy setpoints for the battery and the load (which are mode dependent in general). Set point changes are designed to address two types of variations in the system: the discrepancy between the load demand during the day and at night, and changes in power generation due to different cloud coverage. The vector P i that appears in (28) is related to P i as Using the new variables, the system can be represented as .
which conforms to the framework proposed in [38].
It is important to recognize that this transformation creates discontinuities in the state vector at times when the jump occurs. These discontinuities can be attributed to variations in solar power levels (which differ from one mode to another). As a result, the obtained model corresponds to (30) when r(t) = i, and to with at points when the system transitions from mode i to mode j. In expression (31), x(t − ) denotes the left limit of x(τ) as τ approaches t. The term q ij should be interpreted as the change in the operating points due to variations in cloud coverage [37,43].
For the infinite horizon problem with performance measure , the optimal regulator is given as a time-varying feedback law where the bias vector η i (t) evolves according to equation The presence of the bias term η i (t) is a result of the fact that transitions from one mode to another coincide with discontinuities in the state vector (which are described in (31)). In the steady state, η i (t) can be easily computed by solving three equations in three unknowns. The obtained optimal control law will minimize the error between the energy demand and the energy delivered, while maximizing the energy stored by the batteries.

The Islanded Mode
As noted earlier, in this mode the utility grid cannot deliver power to islanded µGrids. Given that the cost of battery storage and solar panels is decreasing yearly, their cost is not considered in this study. Such a system is fully decoupled, so maintaining stability and optimizing its performance is not an easy task. To do that, it is necessary to precondition the system by stabilizing each µGrid before computing the optimal control.
For the sake of simplicity, in the following we will focus only on the equations for µGrid1 (with the understanding that equations for µGrid2 can be easily derived from those that describe µGrid1).
We begin by observing that system (35) is unstable, and that matrix A 11 is singular. Our first step will therefore be to stabilize it by moving the eigenvalues of A 11 to desired values (using standard pole placement techniques). This is done by setting with v 2 (t) = −Kz 1 (t) where K denotes the gain matrix. The closed-loop system then becomes where A 11 = (A 11 + B 1 K) is stable and nonsingular. We can now apply the method described in Section 3.1. to compute the optimal control action as v 1 The term z(t) in (38) can be expressed as where z i 0 denotes the desired energy setpoint), and the bias vector η i1 (t) and matrix K i1 ∞ are obtained using (25) and (34).
When evaluating the relative merits of this strategy, it is important to keep in mind that the resulting control law does not analytically take into account constraints on battery storage. One can compensate for this by using the weighting matrices Q and R, (together with the setpoints in the various modes) as additional degrees of freedom. In this way, we can help keep system trajectories within the desired limits [42].

Simulation Results
In this section, we provide simulation results using solar energy data from San Jose airport [39] and load data from [35]. As noted earlier, it is assumed that the utility grid can allocate a known amount of power to the µGrids in the connected mode. In general, µGrids 1 and 2 can have different types of solar panels and battery storage. Since they are connected to the same distribution substation, it is reasonable to assume that they are physically close to each other (and therefore have the same cloud coverage).
When the system is in the connected mode, the utility grid provides power to both µGrids according to their needs. The available weather data was used to generate random cloud coverage patterns, which determined the overall amount of solar generation at any given point in time. As mentioned in Section 1, we divided cloud coverage levels into three categories (which is representative of the weather patterns in northern California). For our simulation, we chose the power generated by PV arrays on a sunny day to be 900 kW for µGrid 1 and 1200 kW for µGrid 2. Table 1 provides additional information about the PV sources and batteries for each cloud coverage mode.  Figure 7 shows simulation results that cover a period of 5 business days. Since commercial load behavior is more predictable due to its nature, we were able to use the load profile as a collection of setpoints, and not as an input or as a disturbance. In that respect, our approach differs from most other methods described in the literature. We should also note in this context that due to low or zero inertia related to batteries and solar energy, any small imbalance between the supply and demand can be handled by the batteries and PV arrays.
Another distinguishing feature of the proposed control strategy is that it does not rely on weather forecasts or load forecasts-instead, it considers cloud coverage, the hour of the day (or night) and battery energy setpoint in each mode, and uses this information to compute the required control action. The choice of setpoints is dependent on cloud coverage, since on sunny days we want the battery fully charged, and on rainy days (or at night) we want to allow the battery to discharge as much as possible. In fact, once r(t) jumps from i to j, the optimal control action is to switch from K i to K j . The only requirement is that the information be available when needed (this is not difficult to do with existing computing technology).
The two batteries are chosen to have different storage capacity, and their energy output represents the control action. The storage capacity for batteries 1 and 2 is 1000 kWh and 1100 kWh, respectively. We assumed, however, that the SoC for both batteries lies between 10% and 90%, (these limits are shown by dashed line in Figure 7 top graph). The existence of such limits prevents deep charging/discharging, which can adversely affect the life span of the battery. setpoints, and not as an input or as a disturbance. In that respect, our approach differs from most other methods described in the literature. We should also note in this context that due to low or zero inertia related to batteries and solar energy, any small imbalance between the supply and demand can be handled by the batteries and PV arrays. Another distinguishing feature of the proposed control strategy is that it does not rely on weather forecasts or load forecasts-instead, it considers cloud coverage, the hour of the day (or night) and battery energy setpoint in each mode, and uses this information to compute the required control action. The choice of setpoints is dependent on cloud coverage, since on sunny days we want the battery fully charged, and on rainy days (or at night) we want to allow the battery to discharge as much as possible. In fact, once r(t) jumps from i to j, the optimal control action is to switch from Ki to Kj. The only requirement is that the information be available when needed (this is not difficult to do with existing computing technology).
The two batteries are chosen to have different storage capacity, and their energy output represents the control action. The storage capacity for batteries 1 and 2 is 1000kWh and 1100kWh, respectively. We assumed, however, that the SoC for both batteries lies between 10% and 90%, (these limits are shown by dashed line in Figure 7 top graph). The existence of such limits prevents deep charging/discharging, which can adversely affect the life span of the battery.
Our simulation results show that on sunny days both batteries are charged to their maximum level using solar energy and energy provided by the utility grid. During the night, they discharge to satisfy the demand. Figure 7 also indicates that batteries cannot fully charge during cloudy or rainy intervals (and during the night they may discharge below 50% of their capacity). It is important to recognize, however, that neither battery discharges completely, since both Grids are connected to the utility grid. Figure 8 shows how the system behaves when the Grids operate in the islanded mode. It is readily observed that in this case the battery must discharge completely in order to satisfy the demand, although the load demand during the night is only about 1/3 of the energy required for the day. We should also note that on a sunny day the battery will be fully charged (which is similar to the connected mode), but on partly cloudy or overcast days it discharges more than in the connected mode. Unless the effects these large daily swings are mitigated in some way, they can reduce the Our simulation results show that on sunny days both batteries are charged to their maximum level using solar energy and energy provided by the utility grid. During the night, they discharge to satisfy the demand. Figure 7 also indicates that batteries cannot fully charge during cloudy or rainy intervals (and during the night they may discharge below 50% of their capacity). It is important to recognize, however, that neither battery discharges completely, since both µGrids are connected to the utility grid. Figure 8 shows how the system behaves when the µGrids operate in the islanded mode. It is readily observed that in this case the battery must discharge completely in order to satisfy the demand, although the load demand during the night is only about 1/3 of the energy required for the day. We should also note that on a sunny day the battery will be fully charged (which is similar to the connected mode), but on partly cloudy or overcast days it discharges more than in the connected mode. Unless the effects these large daily swings are mitigated in some way, they can reduce the battery's life span, and perhaps even damage it. This is obviously undesirable, and needs to be addressed. In our future work, we plan to do so by adding controlled secondary generation.
To properly evaluate the simulation results presented in this section, it is important to keep in mind that they were based on three novel ideas. First, a simple continuous multivariate dynamic model was obtained which permits the use of an optimal stochastic control strategy. What is particularly appealing about this approach is the fact that the model does not have to be exact, and that the obtained analytical control law guarantees optimal use of solar energy and battery storage. Second, we use the fact that commercial loads are predictable and can afford to become self-sustainable, because the cost of outages outweighs the cost of solar panels and storage. The fact that loads are grouped by µGrids, allows for a more uniform load distribution. Finally, we should point out that the proposed control design does not rely on weather prediction, since the JLQC method allows us to obtain appropriate control actions using offline weather data. A Monte Carlo method is used to generate a set of possible scenarios of cloud coverage using the obtained transition probability matrix. battery's life span, and perhaps even damage it. This is obviously undesirable, and needs to be addressed. In our future work, we plan to do so by adding controlled secondary generation. To properly evaluate the simulation results presented in this section, it is important to keep in mind that they were based on three novel ideas. First, a simple continuous multivariate dynamic model was obtained which permits the use of an optimal stochastic control strategy. What is particularly appealing about this approach is the fact that the model does not have to be exact, and that the obtained analytical control law guarantees optimal use of solar energy and battery storage. Second, we use the fact that commercial loads are predictable and can afford to become selfsustainable, because the cost of outages outweighs the cost of solar panels and storage. The fact that loads are grouped by Grids, allows for a more uniform load distribution. Finally, we should point out that the proposed control design does not rely on weather prediction, since the JLQC method allows us to obtain appropriate control actions using offline weather data. A Monte Carlo method is used to generate a set of possible scenarios of cloud coverage using the obtained transition probability matrix.
When comparing the results obtained using the proposed control strategy with other existing methods (such as stochastic model predictive control, multi-agent systems distributed control, and the various stochastic optimization techniques mentioned in the Introductory section), it should be recognized that this approach does not require online optimization, nor does it rely on day ahead scheduling using load or weather forecast. Furthermore, the control actions are not obtained iteratively, so there are no convergence issues. These features constitute key advantages, since they eliminate the need for intensive computation and online prediction. Because of that, the proposed control strategy is both robust and economical. There is broad consensus in the literature that this is desirable, and that offline controller design is preferred whenever possible (see e.g., [44]).

Conclusions and Future Work
In this paper, we proposed a simple stochastic hybrid dynamic model for Grids that can connect and disconnect from the utility grid. This mathematical framework was subsequently used to design an optimal stochastic control strategy, whose main objective was to maintain the stability of the grid by optimizing storage and solar energy usage. The obtained gain matrix is piecewise constant, and can be calculated offline by applying the stochastic maximum principle.
It is important to recognize in this context that the resulting closed-loop system is potentially capable of self-healing when a fault occurs, or when a natural disaster impacts the grid. To the best of our knowledge, no similar model exists in the literature. Our future work in this area will focus on increasing the number of Grids and removing the assumption that they must be physically close to When comparing the results obtained using the proposed control strategy with other existing methods (such as stochastic model predictive control, multi-agent systems distributed control, and the various stochastic optimization techniques mentioned in the Introductory section), it should be recognized that this approach does not require online optimization, nor does it rely on day ahead scheduling using load or weather forecast. Furthermore, the control actions are not obtained iteratively, so there are no convergence issues. These features constitute key advantages, since they eliminate the need for intensive computation and online prediction. Because of that, the proposed control strategy is both robust and economical. There is broad consensus in the literature that this is desirable, and that offline controller design is preferred whenever possible (see e.g., [44]).

Conclusions and Future Work
In this paper, we proposed a simple stochastic hybrid dynamic model for µGrids that can connect and disconnect from the utility grid. This mathematical framework was subsequently used to design an optimal stochastic control strategy, whose main objective was to maintain the stability of the grid by optimizing storage and solar energy usage. The obtained gain matrix is piecewise constant, and can be calculated offline by applying the stochastic maximum principle.
It is important to recognize in this context that the resulting closed-loop system is potentially capable of self-healing when a fault occurs, or when a natural disaster impacts the grid. To the best of our knowledge, no similar model exists in the literature. Our future work in this area will focus on increasing the number of µGrids and removing the assumption that they must be physically close to each other. We will also consider the effects of adding controlled secondary generation to the subsystems, and examine how the system dynamics change when the µGrids are connected.
A potential limitation of the proposed control strategy is that it does not explicitly incorporate constraints on battery storage capacity. To ensure that these constraints are met, we used weighting matrix Q and R (as well as the setpoints on battery storage capacity) as tuning parameters. Our simulations established that this approach is effective, and that the resulting control law can accommodate discontinuities (jumps) which are due to sudden changes in solar generation, battery capacity, and load demand.
We should also note that the approach described in this paper implicitly assumes that the 24-h load demand follows a predictable pattern. When that is not the case, the gain matrix becomes time dependent, and the control design becomes considerably more complex (in part because of convergence issues related to the Riccati equations). We plan to investigate this problem in our future research. Optimal control gain matrix for mode i P Load (t) Power required by the load (kW) P B (t) Power generated by the battery (kW) P solar (t) Power generated by the PV array (kW) r(t) Cloud coverage mode P grid (t) Total power allocated by the utility grid (kW) P gridm (t) Power delivered from the utility grid to µGrid m (kW) E Bm (t) Energy stored in battery m (kWh) P solarm (t) Solar power generated for µGrid m (kW) P discm (t) Power delivered from the battery m to the load m (kW) P(t) Power delivered by the grid and the solar panels (kW)