# Hierarchical, Grid-Aware, and Economically Optimal Coordination of Distributed Energy Resources in Realistic Distribution Systems

^{1}

^{2}

^{3}

^{4}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

#### 1.1. Motivation

#### 1.2. Related Literature

#### 1.3. Summary of Proposed Research Contributions

- The presented GML-FOL-STL-DER hierarchical scheme represents a novel, scalable, and practically implementable approach to the Market DSO’s task of coordinating DERs while accounting for individual device and AC grid constraints;
- The scheme employs optimization-based methods within each layer to ensure that DERs are utilized optimally and in a “grid-aware” manner, and then integrates the layers with feedback-based control schemes to be robust against model-mismatch and forecast errors.
- Simulation-based analysis is conducted based on realistic network models from a New York DSO which validates the coupled GML-FOL-STL operations and highlights the role and value of the proposed hierarchical scheme.

## 2. System Models and Consideration

#### 2.1. Market Signals for the GML

#### 2.2. Grid Signals for the FOL

#### Modeling Unbalanced Feeders

#### 2.3. Device Signals

**AC Model:**The operation of a residential AC is governed by thermal dynamics of the room temperature, as represented by [32,33],

**WH Model:**The operation of an electric WH is governed by the thermal dynamics of the water temperature. In the simplistic ‘one-mass’ thermal model which assumes that the temperature inside the water-tank is spatially uniform (valid when the tank is nearly full or nearly empty) [33,34,35], the water temperature dynamics can be expressed in the form of:

**VB Model:**This flexibility can be represented by a virtual battery (VB) model [22,23,24,25,26] that captures the power (response) and energy (duration) limits on the aggregate control offered by the DERs. A VB is typically modeled in the form of first-order dynamics to represent the temporal evolution of the virtual energy state driven by changes in the power consumption as a control input, with constraints specified on the power set-points and the (virtual) energy states [22,24,26]. In this work, for illustrative purpose, we present a deterministic VB model with closed-form expressions for its parameters, leveraging full DER information from the local controller to reasonably assume availability of required device-specific parameters of all the DERs. Thus, consider an aggregation of N thermostatic loads behind a service transformer, where each thermostatic load is indexed by $i=1,\dots ,N.$ Note that the hybrid dynamical model of the i-th thermostatic load, described by either (6) or (7), can be compactly represented in the following generalized form [22,23,24]:

## 3. Overview of Hierarchical DER Control Scheme

**Grid market layer (GML)**employs a TSO’s market signals to optimize the dispatch of available, aggregated flexibility from all feeders and deliver economically optimal power set-points for each feeder’s headnode in the DSO’s system. Since we use market signals from New York’s TSO (NYISO), we consider the GML on a timescale of 5 minutes, which matches the update rate of NYISO’s “real-time market.”- -
- Input: market signals (from TSO); bounds on flexibility for aggregated feeders (from FOL)
- -
- Output: economic feeder power reference (to FOL)

**Feeder operational layer (FOL)**employs the GML’s desired power reference trajectory at each headnode, the DSO’s unbalanced distribution network models, and the STL’s VB model to optimize the dispatch of controllable assets within a feeder so as to minimize power deviations from the headnode reference. The controllable assets include groups of DERs (i.e., a VB) and PV inverters that together track the GML’s economic power reference at the feeder’s head-node while maintaining an acceptable voltage profile throughout the feeder. Since the FOL responds to forecast errors and that solar PV variability is on the order of minutes, the FOL’s timescale has been selected as 1 min.- -
- Input: economic feeder head-node power reference (from GML); VB model parameters and VB state of charge (from STL)
- -
- Output: bounds on flexibility for aggregated feeders (to GML); VB power set-points (to STL)

**Service transformer layer (STL)**employs the FOL’s optimal resource dispatch signal at each (primary) node in the feeder along with DER data to coordinate small, local groups of DERs while accounting for local device constraints on power and energy (e.g., temperature bounds prescribed by users). Since we need to update the DER dispatch often to reject any un-modeled disturbances (e.g., inflexible, background demand), we have selected a timescale of 1 s for the STL’s dispatch loop.- -
- Input: VB power set-points (from FOL); DER data (from DER)
- -
- Output: updated VB state of charge estimate (to FOL); DER control signal (to DER)

## 4. Grid Market Layer (GML)

#### 4.1. Operational Constraints

#### 4.2. GML Power Flow Model

#### 4.3. GML Formulation and Implementation

**subject to**

#### 4.4. Peak-Shaving Mode

**subject to**

#### 4.5. Illustration of GML

- Scenario #1: This baseline scenario assumes that no VB is available, i.e., ${B}_{f}^{max}\left(t\right)=0,\phantom{\rule{3.33333pt}{0ex}}\forall f,t$, and that all solar runs at full capacity, i.e., ${P}_{f}^{g}\left(t\right)={P}_{f}^{max}\left(t\right),\phantom{\rule{3.33333pt}{0ex}}\forall t$, for both the day-ahead and real-time markets.
- Scenario #2: In this GML scenario, the GML has the ability to curtail the solar usage and charge/discharge the VB.
- Scenario #3: In this GML+peak-shaving scenario, the peak-shaving mode is implemented, and the unit price for peak demand charge is set to be $\gamma =10,000$$\$/$MW.

## 5. Feeder Operational Layer (FOL)

**Remark**

**1**

#### 5.1. FOL Multi-Period Formulation

#### Ensuring AC Feasible Optimal Solution

#### 5.2. Robust FOL Formulation

#### 5.2.1. Nature of Uncertainty in Solar PV Forecasts

#### 5.2.2. Chance-Constraints

#### 5.3. Illustration of FOL with Solar PV Forecasts

`HSL_MA86`[55] and the total compute time for the SOCP-NLP is no worse than 15 seconds to ensure viable methodology. Based on the FOL’s optimal dispatch and the actual demand and solar PV injections, AC load flows are computed with GridLab-D [56].

## 6. Service Transformer Layer (STL)

## 7. Inter-Layer Communication and Control

- The full network data for the FOL’s optimization-based dispatch of VBs.
- Live SCADA and power flow information from distribution substations.
- Secure communication infrastructure for corrective inter-feeder and intra-feeder control.

#### 7.1. Communications between Layers

#### 7.2. Feedback Control between Layers

#### 7.2.1. Inter-Feeder Control System

#### 7.2.2. Intra-Feeder Control System

#### 7.3. Proof of Concept: Inter-Layer Feedback Control

#### 7.4. Proof of Concept: Communications between Layers

## 8. Large-Scale Coupled Simulation Results

#### 8.1. Simulation Setup

#### 8.2. Results

#### Peak Shaving

## 9. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Lazard. Levelized Cost of Energy Analysis-Version 13.0; Technical Report; Lazard: Hamilton, Bermuda, 2019. [Google Scholar]
- United Nations Environment Programme. Emissions Gap Report; UNEP: Nairobi, Kenya, 2019. [Google Scholar]
- Goldenberg, C.; Dyson, M.; Masters, H. Demand Flexibility: The Key To Enabling A Low-Cost, Low-Carbon Grid; Technical Report; Rocky Mountain Institute: Bassault, CO, USA, 2018. [Google Scholar]
- Hledik, R.; Faruqui, A.; Lee, T.; Higham, J. The National Potential for Load Flexibility: Value and Market Potential Through 2030; Technical Report; The Brattle Group: Boston, MA, USA, 2019. [Google Scholar]
- IEA. Digitalisation and Energy; Technical Report; International Energy Agency: Paris, France, 2017. [Google Scholar]
- DeMartini, P. Future of U.S. Electric Distribution: Part II; Technical Report; Edison Electric Institute: Washington, DC, USA, 2012. [Google Scholar]
- Wu, F.F.; Varaiya, P.P.; Hui, R.S. Smart Grids with Intelligent Periphery: An Architecture for the Energy Internet. Engineering
**2015**, 1, 436–446. [Google Scholar] [CrossRef] [Green Version] - Taft, J.D. Architectural Basis for Highly Distributed Transactive Power Grids: Frameworks, Networks, and Grid Codes; Technical Report; Pacific Northwest National Lab. (PNNL): Richland, WA, USA, 2016. [Google Scholar]
- Taft, J.D. Grid Architecture 2; Technical Report; Pacific Northwest National Lab. (PNNL): Richland, WA, USA, 2016. [Google Scholar]
- De Martini, P. Operational coordination architecture: New models and approaches. IEEE Power Energy Mag.
**2019**, 17, 29–39. [Google Scholar] [CrossRef] - Martini, P.D.; Kristov, L.; Schwartz, L. Distribution Systems in A High Distributed Energy Resources Future: Planning, Market Design, Operation and Oversight; Technical Report; Lawrence Berkeley National Laboratory: Berkeley, CA, USA, 2015. [Google Scholar]
- Kristov, L.; De Martini, P.; Taft, J.D. A tale of two visions: Designing a decentralized transactive electric system. IEEE Power Energy Mag.
**2016**, 14, 63–69. [Google Scholar] [CrossRef] - Kok, K.; Widergren, S. A Society of Devices: Integrating Intelligent Distributed Resources with Transactive Energy. IEEE Power Energy Mag.
**2016**, 14, 34–45. [Google Scholar] [CrossRef] [Green Version] - Nazir, M.S.; Hiskens, I.A. A dynamical systems approach to modeling and analysis of transactive energy coordination. IEEE Trans. Power Syst.
**2018**, 34, 4060–4070. [Google Scholar] [CrossRef] [Green Version] - Consolidated Edison Company of New York. Consolidated Edison Distributed System Implementation Plan; Con Edison DSIP Filing: New York, NY, USA, 2018; pp. 1–7. [Google Scholar]
- Molzahn, D.; Roald, L.A. Grid-Aware versus Grid-Agnostic Distribution System Control: A Method for Certifying Engineering Constraint Satisfaction. In Proceedings of the Hawaii International Conference on System Sciences, Honolulu, HI, USA, 8–11 January 2019. [Google Scholar]
- Dall’Anese, E.; Guggilam, S.S.; Simonetto, A.; Chen, Y.C.; Dhople, S.V. Optimal regulation of virtual power plants. IEEE Trans. Power Syst.
**2018**, 33, 1868–1881. [Google Scholar] [CrossRef] - Arnold, D.B.; Sankur, M.D.; Negrete-Pincetic, M.; Callaway, D.S. Model-Free Optimal Coordination of Distributed Energy Resources for Provisioning Transmission-Level Services. IEEE Trans. Power Syst.
**2018**, 33, 817–828. [Google Scholar] [CrossRef] - Bidram, A.; Davoudi, A. Hierarchical structure of microgrids control system. IEEE Trans. Smart Grid
**2012**, 3, 1963–1976. [Google Scholar] [CrossRef] - Baker, K.; Bernstein, A.; Dall’Anese, E.; Zhao, C. Network-Cognizant Voltage Droop Control for Distribution Grids. IEEE Trans. Power Syst.
**2018**, 33, 2098–2108. [Google Scholar] [CrossRef] [Green Version] - Dörfler, F.; Simpson-Porco, J.W.; Bullo, F. Breaking the Hierarchy: Distributed Control and Economic Optimality in Microgrids. IEEE Trans. Control Netw. Syst.
**2016**, 3, 241–253. [Google Scholar] [CrossRef] [Green Version] - Hao, H.; Sanandaji, B.M.; Poolla, K.; Vincent, T.L. Aggregate flexibility of thermostatically controlled loads. IEEE Trans. Power Syst.
**2014**, 30, 189–198. [Google Scholar] [CrossRef] - Hughes, J.T.; Domínguez-García, A.D.; Poolla, K. Identification of Virtual Battery Models for Flexible Loads. IEEE Trans. Power Syst.
**2016**, 31, 4660–4669. [Google Scholar] [CrossRef] - Chakraborty, I.; Nandanoori, S.P.; Kundu, S. Virtual battery parameter identification using transfer learning based stacked autoencoder. In Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA, 17–20 December 2018; pp. 1269–1274. [Google Scholar]
- Nandanoori, S.P.; Chakraborty, I.; Ramachandran, T.; Kundu, S. Identification and validation of virtual battery model for heterogeneous devices. In Proceedings of the 2019 IEEE Power & Energy Society General Meeting (PESGM), Atlanta, GA, USA, 4–8 August 2019; pp. 1–5. [Google Scholar]
- Chakraborty, I.; Nandanoori, S.P.; Kundu, S.; Kalsi, K. Stochastic Virtual Battery Modeling of Uncertain Electrical Loads Using Variational Autoencoder. In Proceedings of the 2020 American Control Conference (ACC), 1–3 July 2020; pp. 1305–1310. [Google Scholar]
- Nazir, N.; Racherla, P.; Almassalkhi, M. Optimal Multi-Period Dispatch of Distributed Energy Resources in Unbalanced Distribution Feeders. IEEE Trans. Power Syst.
**2020**, 35, 2683–2692. [Google Scholar] [CrossRef] [Green Version] - Nazir, N.; Almassalkhi, M. Receding-Horizon Optimization of Unbalanced Distribution Systems with Time-Scale Separation for Discrete and Continuous Control Devices. In Proceedings of the 2018 Power Systems Computation Conference (PSCC), Dublin, Ireland, 11–15 June 2018; pp. 1–7. [Google Scholar]
- Nazir, N.; Almassalkhi, M. Stochastic multi-period optimal dispatch of energy storage in unbalanced distribution feeders. In Proceedings of the 2020 Power Systems Computation Conference (PSCC), Porto, Portugal, 29 June–3 July 2020; pp. 1–7. [Google Scholar]
- New York ISO. Real-Time Market LBMP - Zonal. Available online: http://mis.nyiso.com/public/P-24Alist.htm (accessed on 24 October 2020).
- Neukomm, M.; Nubbe, V.; Fares, R. Grid-interactive Efficient Buildings Technical Report Series: Overview of Research Challenges and Gaps; Technical Report; U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy: Washington, DC, USA, 2019.
- Perfumo, C.; Kofman, E.; Braslavsky, J.H.; Ward, J.K. Load management: Model-based control of aggregate power for populations of thermostatcally controlled loads. Energy Convers. Manag.
**2012**, 55, 36–48. [Google Scholar] [CrossRef] - Nandanoori, S.P.; Kundu, S.; Vrabie, D.; Kalsi, K.; Lian, J. Prioritized threshold allocation for distributed frequency response. In Proceedings of the 2018 IEEE Conference on Control Technology and Applications (CCTA), Copenhagen, Denmark, 21–24 August 2018; pp. 237–244. [Google Scholar]
- Diao, R.; Lu, S.; Elizondo, M.; Mayhorn, E.; Zhang, Y.; Samaan, N. Electric water heater modeling and control strategies for demand response. In Proceedings of the 2012 IEEE Power and Energy Society General Meeting, San Diego, CA, USA, 22–26 July 2012; pp. 1–8. [Google Scholar] [CrossRef]
- Kundu, S.; Hansen, J.; Lian, J.; Kalsi, K. Assessment of Optimal Flexibility in Ensemble of Frequency Responsive Loads. In Proceedings of the IEEE International Conference on Smart Grid Communication, Dresden, Germany, 23–27 October 2017. [Google Scholar]
- Duffaut Espinosa, L.; Khurram, A.; Almassalkhi, M. A Virtual Battery Model for Packetized Energy Management. In Proceedings of the IEEE Conference on Decision and Control (to appear), Jeju Island, Korea, 14–18 December 2020. [Google Scholar]
- Zhou, Z.; Levin, T.; Conzelmann, G. Survey of US Ancillary Services Markets; Technical Report; Argonne National Lab.: Argonne, IL, USA, 2016. [Google Scholar]
- Coffrin, C.; Bent, R.; Sundar, K.; Ng, Y.; Lubin, M. PowerModels jl: An Open-Source Framework for Exploring Power Flow Formulations. In Proceedings of the 2018 Power Systems Computation Conference (PSCC), Dublin, Ireland, 11–15 June 2018; pp. 1–8. [Google Scholar]
- Nazir, N.; Almassalkhi, M. Voltage positioning using co-optimization of controllable grid assets. arXiv
**2019**, arXiv:1911.00338. [Google Scholar] - Kron, G. Tensor Analysis of Networks; J. Wiley & Sons, Incorporated: New York, NY, USA, 1939. [Google Scholar]
- Blumsack, S.; Hines, P.; Patel, M.; Barrows, C.; Sanchez, E.C. Defining power network zones from measures of electrical distance. In Proceedings of the 2009 IEEE Power Energy Society General Meeting, Calgary, AB, Canada, 26–30 July 2009; pp. 1–8. [Google Scholar] [CrossRef]
- Dorfler, F.; Bullo, F. Kron reduction of graphs with applications to electrical networks. IEEE Trans. Circuits Syst. I Regul. Pap.
**2012**, 60, 150–163. [Google Scholar] [CrossRef] [Green Version] - Shukla, S.R.; Paudyal, S.; Almassalkhi, M.R. Efficient Distribution System Optimal Power Flow with Discrete Control of Load Tap Changers. IEEE Trans. Power Syst.
**2019**, 34, 2970–2979. [Google Scholar] [CrossRef] - Christakou, K.; LeBoudec, J.; Paolone, M.; Tomozei, D. Efficient Computation of Sensitivity Coefficients of Node Voltages and Line Currents in Unbalanced Radial Electrical Distribution Networks. IEEE Trans. Smart Grid
**2013**, 4, 741–750. [Google Scholar] [CrossRef] [Green Version] - Amini, M.; Almassalkhi, M. Optimal Corrective Dispatch of Uncertain Virtual Energy Storage Systems. IEEE Trans. Smart Grid
**2020**, 11, 4155–4166. [Google Scholar] [CrossRef] [Green Version] - Garifi, K.; Baker, K.; Christensen, D.; Touri, B. Convex Relaxation of Grid-Connected Energy Storage System Models With Complementarity Constraints in DC OPF. IEEE Trans. Smart Grid
**2020**, 11, 4070–4079. [Google Scholar] [CrossRef] - UMBRELLA-F97-Project. Toolbox for Common Forecasting, Risk Assessment, and Operational Optimisation in Grid Security Cooperations of Transmission System Operators(TSOs). 2015. Available online: http://www.e-umbrella.eu (accessed on 19 November 2020).
- Haupt, S.E.; Kosovic, B.; Jensen, T.; Lee, J.; Jimenez, P.; Lazo, J.; Cowie, J.; Mccandless, T.; Pearson, J.; Weiner, G.; et al. The Sun4cast
^{®}Solar Power Forecasting System: The Results of the Public-Private-Academic Partn Partnership to Advance Solar Power Forecasting; No. NCAR/TN-526+STR; National Center for Atmospheric Research: Boulder, CO, USA, 2016; pp. 1–287. [Google Scholar] - Perez, R.; Schlemmer, J.; Hemker, K.; Kivalov, S.; Kankiewicz, A.; Dise, J. Solar energy forecast validation for extended areas & economic impact of forecast accuracy. In Proceedings of the 2016 IEEE 43rd Photovoltaic Specialists Conference (PVSC), Portland, OR, USA, 5–10 June 2016; pp. 1119–1124. [Google Scholar] [CrossRef]
- Bing, J.; Krishnani, P.; Bartholomy, O.; Hoff, T.; Perez, R. Solar monitoring, forecasting, and variability assessment at SMUD. In Proceedings of the World Renewable Energy Forum, Denver, CO, USA, 13–17 May 2012. [Google Scholar]
- Roald, L.; Andersson, G. Chance-constrained AC optimal power flow: Reformulations and efficient algorithms. IEEE Trans. Power Syst.
**2017**, 33, 2906–2918. [Google Scholar] [CrossRef] [Green Version] - Stellato, B. Data-Driven Chance Constrained Optimization. Master’s Thesis, ETH-Zürich, Zürich, Switzerland, 2014. [Google Scholar]
- Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual. 2020. Available online: http://www.gurobi.com (accessed on 24 October 2020).
- Wächter, A.; Biegler, L.T. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math. Program.
**2006**, 106, 25–57. [Google Scholar] [CrossRef] - HSL, A. Collection of Fortran Codes for Large-Scale Scientific Computation. 2007. Available online: http://www.hsl.rl.ac.uk (accessed on 24 October 2020).
- Chassin, D.P.; Schneider, K.; Gerkensmeyer, C. GridLAB-D: An open-source power systems modeling and simulation environment. In Proceedings of the Transmission and Distribution Conference and Exposition, Chicago, IL, USA, 21–24 April 2008; pp. 1–5. [Google Scholar]
- Lougee-Heimer, R. The Common Optimization INterface for Operations Research. IBM J. Res. Dev.
**2003**, 47, 57–66. [Google Scholar] [CrossRef] - Botkin-Levy, M.; Engelmann, A.; Mühlpfordt, T.; Faulwasser, T.; Almassalkhi, M. Distributed Control of Charging for Electric Vehicle Fleets under Dynamic Transformer Ratings. arXiv
**2020**, arXiv:eess.SY/2007.10304. [Google Scholar] - IEEE. Guide for Smart Grid Interoperability of Energy Technology and Information Technology Operation with the Electric Power System (EPS), End-Use Applications, and Loads. In IEEE Std 2030–2011; IEEE: Piscataway, NJ, USA, 2011; pp. 1–126. [Google Scholar]
- Distributed Energy Resources Task Force. Distributed Energy Resources: Connection, Modeling and Reliability Considerations; Technical Report; North American Electric Reliability Council (NERC): Atlanta, GA, USA, 2016. [Google Scholar]
- Schweizer Engineering Laboratories, Inc. SEL-3530-4 Real-Time Automation Controller (RTAC) Spec Sheet. Available online: https://cms-cdn.selinc.com/assets/Literature/Product%20Literature/Data%20Sheets/3530_DS_20200224.pdf?v=20200305-193540 (accessed on 24 October 2020).
- Applegate, C.J.; Chapin, S.J. Requirements and Capabilities Needed for Robust DERMS Control Verification; Technical Report; Lawrence Livermore National Laboratory (LLNL): Livermore, CA, USA, 2019. [Google Scholar]
- Arghandeh, R.; Bariya, M.; Cotter, G.; Deka, D.; Konakalla, S.A.R.; Seyedi, Y.; Shasavari, A. Synchronized Measurements and Their Applications in Distribution Systems: An Update. Available online: https://www.naspi.org/sites/default/files/reference_documents/naspi_distt_synchro_measure_apps_20200716.pdf (accessed on 24 October 2020).
- Lee, P. The Software-Defined Power Grid Is Here. 2020. Available online: https://spectrum.ieee.org/energy/the-smarter-grid/the-softwaredefined-power-grid-is-here (accessed on 24 October 2020).
- Mackiewicz, R.E. Overview of IEC 61850 and benefits. In Proceedings of the 2006 IEEE Power Engineering Society General Meeting, Montreal, QC, Canada, 8–22 June 2006; pp. 1–8. [Google Scholar]
- Adrah, C.M.; BjØrnstad, S.; Kure, Ø. Fusion networking technology for IEC 61850 inter substation communication. In Proceedings of the 2017 IEEE International Conference on Smart Grid and Smart Cities (ICSGSC), Singapore, 23–26 July 2017; pp. 152–156. [Google Scholar]
- Aftab, M.A.; Hussain, S.S.; Ali, I.; Ustun, T.S. IEC 61850 based substation automation system: A survey. Int. J. Electr. Power Energy Syst.
**2020**, 120, 106008. [Google Scholar] [CrossRef] - Taft, J. Electric Grid Resilience and Reliability for Grid Architecture; Pacific Northwest National Laboratory (PNNL): Richland, WA, USA, 2017. [Google Scholar]
- Brunner, C. IEC 61850 for power system communication. In Proceedings of the 2008 IEEE/PES Transmission and Distribution Conference and Exposition, Bogota, Columbia, 13–15 August 2008; pp. 1–6. [Google Scholar]
- Guay, F.; Cardinal, J.; Lemiex, E.; Guerette, S. Digital real-time simulator using IEC 61850 communication for testing devices. In Proceedings of the CIGRE Canada Conference, Montreal, QC, Canada, 24–26 September 2012. [Google Scholar]
- Nazir, N.; Almassalkhi, M. Grid-aware aggregation and realtime disaggregation of distributed energy resources in radial networks. arXiv
**2019**, arXiv:math.OC/1907.06709. [Google Scholar]

**Figure 1.**Hierarchical distributed energy resources (DER) control scheme along qualitative spatio-temporal scales.

**Figure 2.**An example of scenario clustering for real-time price changes and reserve prices. (

**a**) Real-time price change scenarios. (

**b**) Reserve price scenarios.

**Figure 5.**The aggregate virtual battery (VB) discharge with the real-time price. When the price is high, the VB tends to discharge to avoid a high real-time cost, e.g., 18:15.

**Figure 6.**Process for network reduction of electrical circuits by partitioning the network into clusters of similar nodes with the same color. The largest nodes in each cluster is the designated super-node.

**Figure 7.**Illustrating the effect of the intra-hour forecast error model for solar PV forecast over the prediction horizon. The thick dashed line (

**- -**) represent the expected solar PV generation. The forecasts provide a 60-minute preview window and are updated every 30 m.

**Figure 8.**Left: (

**a**) Tracking of the reference grid market layer (GML) head-node power by the feeder operational layer (FOL) through the control of VBs showing acceptable tracking performance for the period 13:00–14:00 Right: (

**b**) Histogram of the voltages obtained from the stochastic AC OPF. Clearly, the voltages are within the ANSI limits given by the red dashed vertical lines.

**Figure 9.**Scalability of real-time control and service transformer layer (STL) coordination of distributed energy resources (DERs).

**Figure 10.**Hierarchical Real Time Control Scheme. (

**a**) Inter-feeder Controller. (

**b**) Intra-feeder Controller.

**Figure 11.**Simulation of intra- and inter-feeder controllers correcting static set-points to improve tracking.

**Figure 12.**A schematic diagram showing the interacting elements in the validation environment. Here a single feeder and service transformer unit are considered. A total of 60 DER assets are coordinated by the system using International Electrotechnical Commission (IEC) 61850 compliant information models and communication protocols.

**Figure 13.**Communication throughput: (

**a**). Generic, object-oriented substation events (GOOSE) exchanges between STL and DERs (average latency ≈5 ms). (

**b**). Load request correction from FOL to STL (average latency ≈ 10 ms). (

**c**). Load estimate query from STL to FOL.

**Figure 14.**The integrated co-simulation environment for numerical validation of the coupled hierarchical stochastic control algorithms (STL, FOL and GML) with large-scale simulations of distribution feeder models populated with solar PV and other DERs.

**Figure 15.**Total nominal active power demand and supply at the headnode of each feeder. (

**a**) Feeder 1 nominal head node demand. (

**b**) Feeder 1 available solar PV generation. (

**c**) Feeder 2 nominal head node demand. (

**d**) Feeder 2 available solar PV generation. (

**e**) Feeder 3 nominal head node demand. (

**f**) Feeder 3 available solar PV generation.

**Figure 16.**The 24-h GML real power set-point trajectory for three fully modelled feeders starting from 11:00 in the peak-shaving mode.

**Figure 17.**GML setpoint tracking performance for the fully modelled feeders in the FOL. (

**a**) Feeder 1. (

**b**) Feeder 2. (

**c**) Feeder 3.

**Figure 18.**Distribution of voltage magnitudes for all nodes during peak hour (11:00 to 12:00). (

**a**) Feeder 1. (

**b**) Feeder 2. (

**c**) Feeder 3.

**Figure 19.**Evolution of the VBs’ normalized state of charge during peak hour in the FOL. (

**a**) Feeder 1. (

**b**) Feeder 2. (

**c**) Feeder 3.

**Figure 20.**Total solar PV output from all nodes during peak hour (11:00 to 12:00) in the FOL. (

**a**) Feeder 1. (

**b**) Feeder 2. (

**c**) Feeder 3.

Without VB | VB: 75 MW + 187.5 MWh | VB: 150 MW + 375 MWh | |||
---|---|---|---|---|---|

Scenario | #1 | #2 | #3 | #2 | #3 |

Real-time cost ($) | 428,330 | 425,981 | 426,053 | 424,322 | 424,486 |

Solar curtailment cost ($) | 0 | 0 | 0 | 0 | 0 |

Peak cost ($) | 12,609,000 | 13,299,920 | 12,150,240 | 14,061,360 | 11,881,330 |

Total cost ($) | 13,037,330 | 13,725,901 | 12,576,293 | 14,485,682 | 12,305,816 |

VB: 75 MW + 187.5 MWh | VB: 150 MW + 375 MWh | |||
---|---|---|---|---|

Scenario | #2 | #3 | #2 | #3 |

Real-time saving | 12.59 $/MWh | ∖ | 10.72 $/MWh | ∖ |

Peak saving | ∖ | 2448.80 $/MWh | ∖ | 1941.49 $/MWh |

Capacity (MWh) | Power Rating (MW) | |
---|---|---|

Feeder 1 | 0.45 | 2.26 |

Feeder 2 | 0.29 | 1.45 |

Feeder 3 | 0.80 | 3.66 |

**Table 4.**Tracking root-mean-square error (RMSE) error and 95th percentile of the nodal voltage distribution when virtual batteries are utilized.

Tracking RMSE | Voltage u.b. with VB (95th Percentile) | Voltage u.b. without VB (95th Percentile) | |
---|---|---|---|

Feeder 1 | 14 kW | 1.053 p.u | 1.058 p.u |

Feeder 2 | 20 kW | 1.035 p.u | 1.041 p.u |

Feeder 3 | 90 kW | 1.038 p.u | 1.047 p.u |

Mean Curtailment with VB | Mean Curtailment without VB | |
---|---|---|

Feeder 1 | 0.2 MW | 0.5 MW |

Feeder 2 | 1.2 MW | 1.6 MW |

Feeder 3 | 0.3 MW | 0.6 MW |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Almassalkhi, M.; Brahma, S.; Nazir, N.; Ossareh, H.; Racherla, P.; Kundu, S.; Nandanoori, S.P.; Ramachandran, T.; Singhal, A.; Gayme, D.;
et al. Hierarchical, Grid-Aware, and Economically Optimal Coordination of Distributed Energy Resources in Realistic Distribution Systems. *Energies* **2020**, *13*, 6399.
https://doi.org/10.3390/en13236399

**AMA Style**

Almassalkhi M, Brahma S, Nazir N, Ossareh H, Racherla P, Kundu S, Nandanoori SP, Ramachandran T, Singhal A, Gayme D,
et al. Hierarchical, Grid-Aware, and Economically Optimal Coordination of Distributed Energy Resources in Realistic Distribution Systems. *Energies*. 2020; 13(23):6399.
https://doi.org/10.3390/en13236399

**Chicago/Turabian Style**

Almassalkhi, Mads, Sarnaduti Brahma, Nawaf Nazir, Hamid Ossareh, Pavan Racherla, Soumya Kundu, Sai Pushpak Nandanoori, Thiagarajan Ramachandran, Ankit Singhal, Dennice Gayme,
and et al. 2020. "Hierarchical, Grid-Aware, and Economically Optimal Coordination of Distributed Energy Resources in Realistic Distribution Systems" *Energies* 13, no. 23: 6399.
https://doi.org/10.3390/en13236399