Real-Time Co-Simulation Implementation for Voltage and Frequency Regulation in Standalone AC Microgrid with Communication Network Performance Analysis across Traffic Variations

Abstract

Effective communication networks are crucial for ensuring reliable and stable operation and control in smart microgrids (MGs). This paper proposes a comprehensive analysis of the interdependence between power and communication networks in the real-time control of a standalone AC microgrid to address this vital need. Thus, the role of communication network design is emphasized in facilitating an effective centralized secondary control to regulate the voltage and frequency of an MG. Consequently, voltage and frequency deviations from the droop-based primary control should be eliminated. This study employs a real-time co-simulation testbed setup that integrates OPAL-RT and network simulator (ns-3), supporting a rigorous evaluation of the interplay between the communication networks and control within the MG. Experiments have been conducted to demonstrate the effectiveness of the designed communication infrastructure in seamlessly enabling real-time data exchange among the primary and secondary control layers. Testing scenarios have been implemented, encompassing low-traffic patterns with minimal load variations and high traffic characterized by more frequent and severe load changes. The experimental results highlight the significant impact of traffic variations on communication network performance. Despite the increase in traffic, the effectiveness and reliability of the designed communication network have been validated, underscoring the vital role of communication in ensuring the resilient and stable operation of cyber–physical standalone AC microgrids.

1. Introduction

With the advancements in information technologies, conventional power systems have become smart, sophisticated networks known as cyber–physical power systems (CPPSs), which accommodate the power system (physical layer) and the communication and computation networks (cyber layer) [1]. The significant integration of cutting-edge information and communication technologies (ICTs) with physical power infrastructure distinguishes this evolution and provides substantial automation, efficiency, and resilience enhancement [2]. These communication networks are the core foundation of CPPSs, which allow real-time information exchange among diverse entities, including generation systems, transmission infrastructures, distribution networks, and control centers [3]. As a result, enabling the deployment of advanced control algorithms helps enhance the overall power system stability, sustainability, and resiliency. In particular, reliable communication infrastructures have greater significance in controlling the decentralization of power systems that rely on the high penetration of renewable energy sources (RESs) [4]. These networks facilitate the quick and precise transmission of control signals necessary for the system’s stability with RESs, characterized by variable and unpredictable behavior [5]. Thus, implementing ICT in power systems is an avenue for constructing resilient and adaptive CPPSs that can cope with the challenges of the contemporary energy environment [6]. The interdependence and the bidirectional interactions among the cyber and physical layers characterize the operation of cyber–physical power systems. This results in complexity and challenges in ensuring the resiliency and stability of power systems while considering the influence of communication networks on their physical system performance [1]. Thus, any degradation in the communication networks within the cyber layer, such as failures, delays, and cyber-attacks, will significantly impact the power system’s sustainable, reliable, and stable operation [4].

Microgrids (MGs) have emerged as a vital development within the broader architecture of CPPSs, establishing localizing power resources that strengthen the reliability and sustainability of power systems. MGs can be identified as small-scale controllable power entities integrating various power resources to satisfy the load requirements [7,8]. These systems can operate with the main utility grid or independently, offering a sustainable and flexible solution in the modern energy market [9]. Microgrids are distinguished by their willingness to incorporate a wide range of distributed energy resources (DERs), including photovoltaic (PV) systems and wind generation systems, in addition to energy storage systems (ESSs) [10,11,12]. However, one of these systems’ significant challenges is managing and controlling these resources with different characteristics within a unified framework [13,14]. These DERs are inverter-based resources (IBRs) that cannot directly connect to the power system but interface through controlled power electronics converters (PECs) that help ensure the MG’s operation within the acceptable voltage and frequency limits [15]. The designed control system of MGs depends on the MG mode of operation, which is either autonomous or grid-connected [16]. In the grid-connected mode, the inverter-based resources work in a grid-following way, where the DERs adapt their output to be synchronized with the grid voltage and frequency [17]. On the contrary, in autonomous operation, the MGs operate in a grid-forming manner independently, establishing their grid with the stabilization of the voltage and frequency values to ensure meeting load requirements, while decreasing reliance on the main grid [18]. Notably, the islanded operation of AC microgrids has technical challenges regarding the internal adaptation among diverse resources, with their intermittent characteristics, to maintain the MG voltage and frequency stability. Additionally, managing the operation of multiple IBRs to satisfy the load demand requires a robust and reliable control system to ensure each source can dynamically adjust its output to maintain the overall MG stability [19].

A hierarchical control structure has been deployed to fully control the intricate dynamics of islanded AC microgrids for more efficient and stable operation. It accommodates three control levels, each involving a particular function, known as primary, secondary, and tertiary control [20]. The primary control level is the base of this structure, and it is responsible for regulating the voltage and frequency values locally, employing a droop control mechanism within the inverter-based MG and facilitating power sharing between the parallel-connected inverters. This communication-less control level is sufficient for the short-term stabilization of the MG, though it needs to effectively eliminate voltage and frequency deviations for long-term stable operation [20,21]. Although the primary control layer can effectively control the voltage and frequency responses resulting from dynamic changes in the consumed power or generation, it has limitations in eliminating the steady-state errors that occur over time. Consequently, an additional control layer is essential for fine-tuning the microgrid operation across several operational conditions to ensure the electric power quality and stability of the MG over long-term operational conditions. Therefore, secondary control is established to overcome these constraints, forming a supervisory control center that communicates with the local droop controllers of each DER within the primary control layer [22]. Thus, the voltage and frequency remain within their rated values during the long-term operation without any steady-state errors across various operational conditions. On the other hand, the secondary control is a communication-based architecture, which requires a reliable communication network that seamlessly allows the bidirectional flow of data in real time between the primary and secondary control levels within the AC microgrid [23]. The reliable design of these communication networks has a vital role in maintaining the quality of the electrical system. A delay or packet loss in the communicated signals may introduce further voltage and frequency deviations, impacting the MG’s ability to fulfill operational requirements and maintain the voltage and frequency at their rated values. Consequently, considering the interdependence between the communication and power networks and analyzing the MGs’ operation as a cyber–physical power system is crucial for maintaining the overall system’s resilient and stable operation.

Communication networks within the hierarchical architecture play a vital role in enhancing the resilient and stable operation of microgrids, offering a stable medium that supports the transmission of measurements and control signals accurately in real time across various traffic patterns [2,24,25]. Consequently, the communication network performance affects the stabilization and resiliency of the MG. The National Institute of Standards and Technology (NIST) has specified the evaluation of communication network performance by examining key performance indicators (KPIs), such as the mean delay, the percentage of packet loss and the transmitted, and the received bitrate within the communication infrastructure [26], to ensure how accurately information is exchanged within the smart cyber–physical MG architecture. Throughout this communication-based control architecture, any degradation in the communication network performance directly affects the stabilization and functionality of the MG. Accordingly, ensuring reliable communication performance and the capability of handling low and high traffic patterns is crucial, especially in practical scenarios where the islanded MG is subjected to frequent load variations.

1.1. Review of Relevant Literature

Several investigations have been carried out to examine the functioning, managing, and resilient control of AC microgrids. The authors of [27] have investigated the voltage regulation in an islanded MG by developing a centralized secondary voltage controller cascaded with a module for generating the voltage regulating value. Upon the load requirement and the MG operational circumstances, the proposed secondary voltage control scheme sends the appropriate control signal to the primary level to adapt the MG voltage. Simulation results employing PSCAD have demonstrated the effectiveness of the proposed technique under different operating conditions in enhancing voltage regulation and reactive power sharing. The proposed work in [28,29] has targeted the frequency regulation in AC microgrid systems, including communication delays’ impact on MG stability. A centralized model predictive control (MPC) for frequency adaptation in an inverter-based MG is introduced in [28] to compensate for the limitations of droop-based control in grid-forming inverters. Based on the Frequency Divider (FD) concept in transmission systems, the authors have developed a modified version of a conventional MPC to improve the frequency performance. Several MATLAB simulation scenarios were conducted to ensure the effectiveness of the proposed scheme in controlling the system frequency compared to traditional PI and MPC controllers. Additionally, they assessed the performance of the MG frequency across communication delays. An enhancement of frequency performance, employing a genetic algorithm PID-based virtual inertia (VI) control (GA-PID-based VI) scheme within an islanded AC microgrid is developed in [29]. Simulation results have presented the superiority of the proposed technique over utilizing conventional VI control schemes under different disturbances, including delays in communication channels between the primary and secondary controllers. Though [28,29] have assessed the effectiveness of the proposed frequency control schemes under latency in communication issues, they did not include any modeling of the communication infrastructure between the primary and secondary control levels, which affects the practicality of the proposed scheme in real-world applications.

Analyzing the operation and control of microgrids under autonomous operation and grid integration is depicted in [30]. An enhanced centralized control algorithm has been presented to control the MG operation fully, including active/reactive power sharing and frequency/voltage regulation under several real-world scenarios. A hardware-in-the-loop setup utilizing Typhoon HIL has been implemented to validate the effectiveness of the proposed control scheme in enhancing the MG transient stability. An analysis of the communication infrastructure and its quality of service (QoS) within the introduced MG system must be conducted. In [31], researchers have presented an artificial neural network (ANN)-based secondary voltage and frequency control in a centralized way within an islanded AC microgrid. The simulation analysis is illustrated to show the efficiency of the proposed algorithm in adapting the voltage and frequency under step-load change. Researchers in [32] have proposed a distributed secondary control architecture to eliminate any frequency deviation within an autonomous inverter-based MG. An adaptive neuro-fuzzy inference system (ANFIS) has been designed as the base of the proposed control system architecture to enhance the MG performance with its online adaptability characteristics compared to the traditional PI controller. Despite the research work introduced in [27,28,29,30,31,32] having introduced various communication-based control schemes for enhancing MG performance across numerous operational conditions, they overlooked the modeling and evaluation of the communication networks among the different controllers within the context of the smart MG system.

The authors of [33,34,35,36,37] have investigated the resilient and stable performance of autonomous AC microgrids across several operational conditions while considering the impact of cyber-attacks, such as denial of service (DoS) and false data injection (FDI). A robust distributed secondary control algorithm is proposed in [33] to ensure the resilient operation of an inverter-based MG under an FDI attack that targets the secondary controller. The effectiveness of the proposed scheme has been evaluated under FDI attack on several points within the MG, including attacks on actuators, sensing nodes, and communication channels. Power system-based simulation analyses have been implemented to validate the proposed control scheme while ensuring the voltage and frequency stability of the microgrid, with a notable absence in validating the real cyber–physical platform that includes a representation of power and communication networks. The impact of non-ideal communication links and data manipulation on voltage and frequency stability in an islanded AC microgrid have been studied in [34]. The authors presented an adaptive, resilient secondary control algorithm in a distributed manner for regulating voltage and frequency values under normal operation and cyber threats. MATLAB/Simulink has been utilized as a validation medium, highlighting a crucial gap in the practicality of the proposed scheme due to the need for the real-time emulation of the communication networks to ensure the effectiveness of the proposed attack-resilient control scheme in actual operational conditions. In [35], researchers have developed an event-triggered adjusting scheme for a cyber–physical AC microgrid subjected to a DoS attack. An evaluation of the developed control algorithm to enhance MG resiliency has been implemented through the MATLAB simulation environment. This work mainly targets the effect of a need for more communication on MG stability, with an evident scarcity of cyber–physical hardware validation and accurate communication network emulation.

Real-time power system-based simulation research studies have been conducted in [36,37]. Examining the MG as a CPPS while considering the cybersecurity aspect is presented in [36]. The authors have assessed MG operation when equipped with a centralized secondary controller under islanded/grid-connected modes of operation, considering the influence of FDI attacks and faults. Then, they developed a long short-term memory (LSTM)-based approach for detecting cyber-attacks during MG operation. The introduced control technique has been validated in the real-time digital simulator (OPAL-RT). Frequency regulation on a shipboard MG across hybrid FDI/DoS attacks affecting the measuring and actuating data signals is illustrated in [37]. A machine learning (ML)-based attack detection scheme incorporating an active disturbance rejection control (ADRC) algorithm has been developed to improve the cyber resiliency of the MG under study. Irrespective of the notable relevance of the introduced work in [36,37], there is a need to validate the introduced scheme within a cyber–physical framework that integrates a real-time emulation of communication and power networks, not only the power system real-time simulators as they were exhibited.

Even though these studies referenced vital aspects of the resilient operation of AC microgrids, they should have considered the modeling and performance examination of the communication infrastructure that facilitates the proposed communication-based control algorithms within the cyber–physical MG system. A communication network emulation for smart microgrid applications employing the NetSim software tool is presented in [38,39,40]. In [38], the authors have designed a simulation model of communication networks among smart MG monitoring systems. They provided a detailed quantitative analysis of the network performance metrics, including the loss in delivered packets, throughput, and delays across the deployment of various communication protocols, such as ZigBee, wired local area networks (LANs), and wireless LANs (WLANs). Performance analysis of IEC 61850-based reactive power sharing in an MG system implemented on a long-term evolution (LTE) stack is introduced in [39]. The simulation results have proved the effectiveness and superiority of implementing LTE technology with IEC-based communication within microgrid infrastructures over Wi-Fi and WiMAX technologies. The communication network model consists of four managing levels within a five-bus MG, and its performance analysis, in terms of transmission rates, latency, and packet loss percentage under several technologies, has been demonstrated in [40].

The evaluation of communication networks within a networked microgrid system using the User Datagram Protocol (UDP) and Transmission Control Protocol (TCP) is introduced in [41]. A detailed communication network QoS analysis has been presented in diverse traffic loads to ensure the resiliency of the developed model using network simulator (ns-3). Researchers in [42] have leveraged ns-3 to establish and validate a synchrophasor communication network (SCN) framework in smart grids. A detailed design, modeling, validation, and performance evaluation have been illustrated under various real-time experiments to ensure the reliability and resiliency of the proposed network in achieving the smart grid communication requirements. Researchers in [43] have developed a co-simulation platform that integrates power and communication networks, utilizing the MATLAB/Simulink and OMNeT++ simulators, respectively, for emulating the dynamics of cyber–physical smart grids. A communication infrastructure that uses hypertext transfer protocol (HTTP) for data exchange among MG components has been developed. Nonetheless, using MATLAB/Simulink to represent the power system limits the applicability of this platform in real-world scenarios due to the lack of real-time implementation. The introduced work [38,39,40,41,42,43] highlights a notable research gap in the interdependence analysis of the communication networks’ influence on MG control performance. The proposed work focused on the design, modeling, and numerical quantitative analysis for communication infrastructures for distinct MG deployments without studying the MG behavior from the power system point of view, such as its stability and resiliency across diverse practical scenarios.

The resilient and stable operation of the microgrid as a CPPS, while considering the real-time modeling of communication and power system infrastructures, has been showcased in [44,45,46,47]. In [44], a comprehensive analysis of frequency regulation in a hierarchically controlled islanded inverter-based MG, composed of two parallel grid-forming inverters, was depicted with detailed communication network modeling among the control levels. The effectiveness of the proposed communication-based control infrastructure has been validated in real-time operation, employing the integration between OPAL-RT and ns-3 in a unified framework. Nonetheless, they passed over the proposed communication network QoS evaluation under the implemented experiments. Furthermore, the voltage control aspect within the cyber–physical MG system was not covered. The work proposed in [45] targets the foundation of a co-simulation framework that accommodates power and communication systems. The resiliency of an autonomous MG system under the condition of generation outage, while varying the communication technologies within the developed communication network, was examined. A dynamic mode decomposition (DMD)-based data-driven control scheme has been implemented in [46] to ensure MG frequency stability under communication latency. In [47], researchers have developed an ANN-based protection scheme in a centralized manner for a multi-MG system. Experimental validation of the introduced protection scheme has been presented in a cyber–physical framework that includes MATLAB for power system representation and NetSim for communication emulation.

1.2. Paper Contribution and Organization

Inspired by the prior findings [27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47], a significant research gap emerged in thoroughly exploring the interdependence within cyber–physical MG systems, particularly recognizing the integration of communication infrastructure into the MG control architectures. Figure 1 presents a graphical demonstration categorizing the literature studies, highlighting the focus of the proposed work. Consequently, this paper presents an inventive interdependence analysis that bridges the research gap in exploring AC microgrids as a CPPS. This work targets the voltage and frequency regulation in an islanded AC microgrid system, employing a communication-based centralized secondary control scheme and meticulously evaluating the communication network performance within the implemented hierarchical control structure under different traffic patterns. Experimental validation of the proposed cyber–physical standalone AC microgrid system is implemented in a co-simulation hardware testbed setup that integrates OPAL-RT and ns-3 to emulate the physical MG system and the communication layer. Several experiments have demonstrated the effectiveness of the developed communication-based secondary control scheme compared to communication-less droop control in enhancing voltage and frequency performance, as well as the reliability and resiliency of the proposed communication infrastructure across diverse traffic loads with quantitative performance metrics.

The paper is structured as follows: Section 2 outlines the system modeling and communication-based hierarchical control structure within a standalone AC microgrid. The detailed modeling of the cyber–physical MG components, the interfacing between various real-time simulation environments, and the acquisition of communication performance data within the real-time co-simulation testbed setup are demonstrated in detail in Section 3. Section 4 presents the experimental results for the MG control system under different load patterns and investigates the communication network performance evaluation across low and high traffic loads. The conclusion is presented in Section 5.

Figure 1. Summary of the mentioned literature studies and the focus of the proposed work.

Figure 1. Summary of the mentioned literature studies and the focus of the proposed work.

2. Standalone AC Microgrid and Hierarchical Control Description

A microgrid system is considered a cyber–physical power system, including a physical layer containing the physical devices, measuring units, and local controllers, alongside a communication layer facilitating the coordination between various control levels. The overall configuration of an AC microgrid under an islanded mode of operation is presented in Figure 2. The physical construction of the MG system accommodates n-parallel distributed generation (DG) units, including a constant DC link, a three-phase inverter, a filter, and an output connector for each unit. The controlled DG units are connected to the common AC bus while operating in an autonomous mode of operation, forming the desired voltage and frequency levels while satisfying the load requirements.

A communication-based hierarchical control architecture is implemented to maintain the voltage and frequency at specified nominal values (208 V, 60 Hz). The communication layer within the hierarchical control structure offers a resilient and reliable medium for bidirectional data exchange within the communication-based hierarchical structure. This control architecture incorporates multiple control loops, and their correlated functions can be outlined as mentioned below:

• Inner control loops: Concentrate on adjusting the output voltage and current of the voltage source inverter (VSI) within each DG unit under varying load conditions.
• Primary control loop: This control level can be considered a communication-free control that locally controls the power sharing among the parallel-connected VSIs, known as a droop control in inverter-based MGs. This way of control is initiated from the conventional primary control loop in synchronous generator-based high inertia power systems that control power generation via regulating the turbine governor.
• Secondary control loop: This level is responsible for restoring any deviation in the MG voltage and frequency to eliminate the limitations of the droop-based primary control in keeping the voltage and frequency at their rated values. A communication-based centralized secondary control for regulating the voltage and frequency values is implemented in this work.

The following subsections provide a detailed mathematical representation of each control level within the islanded AC inverter-based MG under investigation.

2.1. Inner Control Loop

The inner control loop follows a cascaded control architecture designed to regulate the current and voltage outputs of the VSI. It includes inner current and outer voltage control loops that operate on a fast timescale to maintain MG stability, and the production of the inner loop imitates the required pulsing signals of the switches of the VSI. These loops are designed to operate in a synchronous dq-frame; the three-phase AC signals are simplified and transformed in the dq reference frame by applying the Clark and Park transformations. The transformation of the three-phase input voltage $e_a$, $e_b$, and $e_c$ from the stationary phase system into $e_d$ and $e_q$ in the dq-frame is expressed as follows [48,49]:

$$e_d={\displaystyle \frac{2}{3}}\left[v_a\mathit{cos}\left(\phi \right)+v_b\mathit{cos}\left(\phi -{\displaystyle \frac{2\Pi }{3}}\right)+v_c\mathit{cos}\left(\phi +{\displaystyle \frac{2\Pi }{3}}\right)\right]$$

(1)

$$e_q=-{\displaystyle \frac{2}{3}}\left[v_a\mathit{sin}\left(\phi \right)+v_b\mathit{sin}\left(\phi -{\displaystyle \frac{2\Pi }{3}}\right)+v_c\mathit{sin}\left(\phi +{\displaystyle \frac{2\Pi }{3}}\right)\right]$$

(2)

where $\phi =\omega t+{ʎ}_i$ is the rotating reference frame angle, and it is a function of the angular speed of the rotating frame ω, and the initial value of the voltage phase shift angle is ${ʎ}_i$. The inner current control loop adapts the current output in the dq-frame from the inverter, and the dynamic equations are given by

$$\mathrm{L}{i}_d^{\prime }=e_d-R_f{i}_d-\omega L_f{i}_q+e_{od}$$

(3)

$$\mathrm{L}{i}_q^{\prime }=e_q-R_f{i}_q+\omega L_f{i}_d+e_{oq}$$

(4)

The $i_{d }$ and $i_q$ are the VSI output current in the dq-frame, while the values of output voltage in the dq-frame are $e_{od}$ and $e_{oq}$. The control function of this loop is to track the reference currents $i_d^{\mathrm{*}}$ and $i_q^{\mathrm{*}}$; thus, a conventional PI controller is used with fixed proportional and integral gains $K_{Pi}$ and $K_{Ii}$. The mathematical equations represent this control objective described as [49]:

$$e_d=R_f{i}_d^{*}+\omega L_f{i}_q^{*}+L_f{i}_d^{\mathrm{*}\prime }{+e}_{od}+K_{Pi}\left(i_d^{*}-i_d\right)+K_{Ii}\int \left[i_d^{*}-i_d\right]dt$$

(5)

$$e_q=R_f{i}_q^{\prime }-\omega L_f{i}_d^{*}+L_f{i}_q^{\mathrm{*}\prime }{+e}_{oq}+K_{Pi}\left(i_q^{*}-i_q\right)+K_{Ii}\int \left[i_q^{*}-i_q\right]dt$$

(6)

The outer voltage control adjusts the output voltage of the inverter by regulating the reference current values of $i_d^{\mathrm{*}}$ and $i_q^{\mathrm{*}}$ for the inner current loop. The following equations represent the reference current values corresponding to the required regulated voltage to ensure precise voltage regulation under load variations:

$$i_d^{\mathrm{*}}=C_f{e}_{od}^{\mathrm{*}\prime }+i_{od}+K_{PE}\left(e_{od}^{\mathrm{*}}-e_{od}\right)+K_{IE}\int \left[e_{od}^{\mathrm{*}}-e_{od}\right]dt$$

(7)

$$i_q^{\mathrm{*}}=C_f{e}_{oq}^{\mathrm{*}\prime }+i_{oq}+K_{PE}\left(e_{oq}^{\mathrm{*}}-e_{oq}\right)+K_{IE}\int \left[e_{oq}^{\mathrm{*}}-e_{oq}\right]dt$$

(8)

The values of the reference voltage in the dq-frame are $e_{od}^{\mathrm{*}}$ and $e_{oq}^{\mathrm{*}}$; in addition, the employed PI controller for the outer voltage loop has constant gains $K_{PE}$ and $K_{IE}$.

2.2. Local Primary Control Loop

The objective of this control level is to manage the accurate power sharing among the parallel-connected VSIs in a distributed manner based on the loading condition without direct communication between the parallel inverters. Each grid-forming VSI has its droop controller to adapt the active and reactive power sharing (P/ω and Q/V) droop ratios; thus, it is a communication-free control layer. Nevertheless, this control layer cannot maintain the voltage and frequency at their rated values, as the droop characteristics considered a proportional controller cannot keep zero steady-state errors for the MG voltage and frequency values.

Consider that there are several DG units (n number) connected through a feeder link with impedance $Z_{li}=R_{li}+j{X}_{li}$ to the common AC bus within the islanded MG. The output voltage of ${\mathrm{D}\mathrm{G}}_i$ equals $V_i⌊{\rho }_i$ (i = 1, 2, ……, n), and the voltage at the common bus is $V_{bus}⌊{\rho }_{bus}$; thus, the i-th voltage source inverter’s active and reactive power outputs are given as [27]:

$$P_{DGi}={\displaystyle \frac{R_{li}\left[V_i{V}_{bus}\mathit{cos}{\rho }_i-V_{bus}^2\right]+{ X}_{li}{V}_i{V}_{bus}\mathit{sin}{\rho }_i}{{\left|Z_{li}\right|}^2}}$$

(9)

$$Q_{DGi}={\displaystyle \frac{x_{li}\left[V_i{V}_{bus}\mathit{cos}{\rho }_i-V_{bus}^2\right]-{ R}_{li}{V}_i{V}_{bus}\mathit{sin}{\rho }_i}{{\left|Z_{li}\right|}^2}}$$

(10)

Aligned with a minor power angle ${\rho }_i$, it is acceptable to assume that $\mathit{cos}{\rho }_i\approx 1$ and $\mathit{sin}{\rho }_i\approx {\rho }_i$, additionally, assuming the value of $X_{li}$≫$R_{li}$; thus, the resistive component of the feeder can be neglected. Consequently, the output active and reactive powers can be approximated as follows:

$$P_{DGi}={\displaystyle \frac{V_i{V}_{bus}{\rho }_i}{{ X}_{li}}}$$

(11)

$$Q_{DGi}={\displaystyle \frac{V_{bus}\left[{V_i-V}_{bus}\right]}{{ X}_{li}}}$$

(12)

As a result, the values of generated active and reactive power from each VSI can be regulated by controlling the power angle ${\rho }_i$ and the DG voltage $V_i$, respectively, with a trade-off in the voltage and frequency deviations according to the operational condition. To achieve this objective, the reference values of the voltage amplitude and frequency for the outer voltage control loop are adjusted, employing the droop control mechanism [50]:

$${\omega }_i={\omega }_{des}-m_{droop}{P}_i$$

(13)

$$V_i=V_{des}-n_{droop}{Q}_i$$

(14)

The active and reactive droop factors are $m_{droop}$ and $n_{droop}$, respectively, and the desired rated values of the MG frequency and voltage are ${\omega }_{des}$ and $V_{des}$.

2.3. Communication-Based Secondary Control Loop

This work implements a communication-based centralized secondary controller to correct the MG voltage and frequency deviations resulting from the droop control in the primary control layer. The secondary controller is designed to maintain the voltage and frequency levels constant at their nominal values and correct deviations from the primary control loop, as demonstrated in Figure 3, under any operational scenarios to ensure the islanded MG’s power quality and stability. The following equations represent the voltage and frequency control signals (${∆\mathrm{v}}_{control}$ and ${∆\omega }_{control})$, utilizing traditional PI controllers:

$${∆\mathrm{v}}_{control}=K_{PCV\_sec}{∆\mathrm{V}}_{MG}+K_{I_{CV\_sec}}\int {∆V}_{MG}dt$$

(15)

$${∆\omega }_{control}=K_{P_{CF\_sec}}{∆\omega }_{MG}+K_{I_{CF\_sec}}\int {∆\omega }_{MG}dt$$

(16)

where the voltage and frequency deviations at the common AC bus are ${∆V}_{MG}$ and ${∆\omega }_{MG}$. $K_{PCV\_sec}$, $K_{I_{CV\_sec}}, K_{P_{CF\_sec}},\mathrm{a}\mathrm{n}\mathrm{d} K_{I_{CF\_sec}}$ are the voltage and frequency PI-based centralized secondary controller gains. Through the communication layer within the hierarchical control architecture, these control signals are transmitted to the local droop controllers within the physical layer of the standalone AC microgrid.

2.4. Communication Network Graph Theory

The communication layer has a crucial role in facilitating the control system operation and coordinating the different control levels to achieve the stable operation of the MG and meet the load requirements. Communication graph theory can mathematically model the communication network within a multi-agent MG system, where the interactions between the centralized secondary controller and the local droop controllers can be effectively identified. An undirected graph $G_{Com}=f(C, \mu , A)$ describes the network, where C denotes a set of components and $\mathrm{\mu }$ identifies the connection edges among these components. The existence of these edges is expressed by the adjacency matrix A.

The main components of the set C are the centralized secondary controller $c_{sec}$ and the local droop controllers $c_{droop\_i}$ for i = 1, 2, ……, n, within each n-parallel-connected DG unit. Therefore, the set of components is identified as $C=\left[c_{sec}, c_{droop\_1}, c_{droop\_2}, \dots \dots , c_{droop\_n}\right]$. In this work, the communication network is a star topology where the centralized secondary controller (CC) acts as the system hub, as depicted in Figure 4. The edges describe the links between the controllers within the hierarchical structure $\mu =\left[\left(c_{sec}, c_{droo{p}_i}\right), i=1, 2, \dots \dots , n\right]$, which demonstrates the bidirectional communication channels among the droop and central controllers. These communication links are represented in matrix A = [$a_{ji }]$ where each element $a_{ji}$ describes the linking edge between the central controller node $c_{sec}$ and droop nodes $c_{droop\_i}$, as identified in (17). In the star topology, the centralized secondary controller is connected to all the local droop controllers; thus, the adjacency matrix is expressed in (18), where the first column and row represent the secondary controller, and the local droop controllers are identified in the remaining matrix.

$$a_{ji }=\left\{\begin{array}{c}1, if \left(c_{sec},c_{droo{p}_i}\right)\in \mu \\ 0, otherwise\end{array}\right.$$

(17)

$$A=\left[\begin{array}{c} 0 \\ 1 \\ \begin{array}{c}1 \\ \begin{array}{c}⋮ \\ 1 \end{array}\end{array}\end{array}\begin{array}{c} 1 \\ 0 \\ \begin{array}{c}0 \\ \begin{array}{c}⋮\\ 0\end{array} \end{array}\end{array}\begin{array}{c}1\\ 0\\ \begin{array}{c}0\\ \begin{array}{c}⋮\\ 0\end{array}\end{array}\end{array} \begin{array}{c} \cdots \\ \dots \\ \begin{array}{c} \dots \\ \begin{array}{c} \ddots \\ \dots \end{array}\end{array}\end{array} \begin{array}{c}1\\ 0 \\ \begin{array}{c}0\\ \begin{array}{c}⋮\\ 0\end{array}\end{array}\end{array}\right]$$

(18)

3. Cyber–Physical Standalone AC Microgrid Co-Simulation Design

For a precise analysis of the cyber–physical AC microgrid system, addressing the interdependence between the communication network and the physical system, this section will illustrate the real-time modeling for each element within the MG under study. Additionally, it demonstrates the integration of the different cyber and physical environments in a unified co-simulation platform, which facilitates evaluating the communication and control systems’ performance. Figure 5 depicts this by co-simulating the electric and communication networks utilizing OPAL-RT (OP4610XG model) and ns-3 (ns-3.35 version), respectively. The real-time simulators are interfaced through the deployment of docker containers.

3.1. Communication Network Emulation

As detailed in the hierarchical control structure in Section 2, the communication network facilitates the operation of the centralized secondary controller. Thus, each local droop controller (LC) alongside the central controller (CC), represented by nodes in ns-3, seamlessly facilitates the control objective of regulating the AC microgrid voltage and frequency under various operating conditions.

Network simulator ns-3 offers a scalable environment that emulates multi-agent systems with diverse communication technologies and protocols [51]. The ability to simulate real communication network devices, such as routers, switches, and communication channels, provides a detailed representation of the interdependence between the electrical and communication networks in real-time CPPSs. Moreover, external physical systems can interfere with the emulated communication model via the virtual tab bridges inside the software, fulfilling the research gap in studying the power systems as a cyber–physical system [41,51,52]. Moreover, it can emulate cyber-attacks, such as DoS, FDI, and man-in-the-middle attacks, to assess their impact on the power system resiliency, which can be investigated in detail in future work. All the previously mentioned scalable features are in open-source software that runs on the Linux operating system (OS), making it easily used by the research community without increasing the computational costs.

In this work, the communication infrastructure is designed as an ethernet-wired network in ns-3, supporting the implemented hierarchical control structure. The developed communication topology, as illustrated in Figure 5, consists of two local area networks (LANs) corresponding to the primary and secondary control layers within the AC microgrid system. Each node within the communication model represents a control agent in the MG system connected to the router through network switches within each LAN.

The Internet Protocol (IP) establishes communication between the network’s nodes, and each node within the model has its identified IP address. According to the MG structure under investigation, we have four DG units, and each unit has its local droop controller; thus, we create four nodes in the first LAN representing each control agent in the primary layer. For the secondary control layer, there is a centralized secondary controller; consequently, there is one node in the second LAN that performs as the introduced communication-based centralized secondary controller. Communication between the two LANs is established through two routers, forming the designed network’s foundation. Therefore, it streamlines the information exchange through the hierarchical control architecture to enable the continuous real-time adaptation of the MG voltage and frequency.

Leveraging the communication graph outlined in Section 2.4, the network is coded in ns-3 using C++. This network can be scalable for different MG architectures with multiple agents. The scalability can be achieved by expanding the number of nodes, routers, and the designed LAN based on the power system configuration. Let $N_{{LAN}_z}$ denote the nodes in each ${LAN}_z$ (z is the number of local area networks), $N_{{LAN}_z}=\left\{n_1, n_2, \dots ..,n_{N_{total}}\right\}$, and R is the set of routers where $R_{{LAN}_z}=\left\{r_1, r_2, \dots ..,r_{R_{total}}\right\}$. For the system under study, z = 2; thus, $N_{{LAN}_1}=\left\{n_{11}, n_{12}, n_{13}, n_{14}\right\}, N_{{LAN}_2}=\left\{n_{21}\right\}, R_{{LAN}_1}=\left\{r_1\right\},\mathrm{a}\mathrm{n}\mathrm{d} R_{{LAN}_2}=\left\{r_2\right\}$. After, the connection links L between the network devices are identified as follows: Point-to-Point (P2P) communication between the implemented LANs among the backbone routers $L_{r_1, r_2}^{p2p}=\left\{r_1, r_2\right\}$, and Carrier Sense Multiple Access (CSMA) within each LAN, where $L_{r_1, N_{{LAN}_1}}^{CSMA}=\left\{r_1, n_{11}, n_{12}, n_{13}, n_{14}\right\}$ and $L_{r_2, N_{{LAN}_2}}^{CSMA}=\left\{r_2, n_{21}\right\}$.

Then, each designed node and router is assigned its unique IP address from subnets; the base IP for ${LAN}_1 {\mathrm{i}\mathrm{s} BaseIP}_{{LAN}_1}=\mathrm{10.1.1.0}$ and for ${LAN}_2 {\mathrm{i}\mathrm{s} BaseIP}_{{LAN}_2}=\mathrm{10.1.2.0}$. Therefore, each node within ${LAN}_1$ and ${LAN}_2$ can be identified as in (19).

$${IP}_{{node}_{{LAN}_1}}^j={BaseIP}_{{LAN}_1}+j \mathrm{and} {IP}_{{node}_{{LAN}_2}}^j={BaseIP}_{{LAN}_2}+j$$

(19)

where j is an integer that represents the position of each node within the subnet; for example, at $j=1 \mathrm{i}\mathrm{n} {LAN}_1$ this identifies $n_{11}:IP \mathrm{10.1.1.1}$.

For making the designed network in ns-3 able to interact with external physical systems, tab bridges $T_i$ have has been set up at each node, where $T_i=\left\{T_1, T_2, \dots ,T_{T_{total}}\right\}$; here, we have four nodes representing the local droop controllers $\left\{n_{11}, n_{12}, n_{13}, \mathrm{a}\mathrm{n}\mathrm{d} n_{14}\right\}$ in addition to one node representing the centralized controller $n_{21}$; therefore, five tab bridges have been implemented. These bridges enable the interface with the physical system (OPAL-RT) through docker containers described in the following subsection.

3.2. Docker Containers for Data Exchange and System Integration

To implement the integration in real time between the physical MG in OPA-RT and the designed communication network ns-3, docker containers have been utilized to facilitate smooth data exchange within the cyber–physical framework. Docker is a containerization technology developed to encapsulate an application into a unified unit (container), including libraries, configuration files, and binaries [53]. These containers are an efficient and lightweight virtualization technology, sharing the host machine’s operating system (OS) kernel. This will eliminate the need for a separate OS and provide a faster start-up than virtual machines (VMs), with less consumed resources. Moreover, they are independent systems that can operate consistently within different computing frameworks, ensuring the same action of the deployed application inside the container regardless of any changes between environments [54]. Therefore, they are an ideal choice in real-time cyber–physical simulations where fast action and a dynamic performance among distinct operating environments are crucial.

In this study, docker containers are implemented on Linux OS and ns-3 in the same host computer, serving as an interface medium, as illustrated in Figure 5. Consequently, this enables the real-time exchange of information between the local control agents in the physical MG runs on OPAL-RT and their corresponding nodes in the designed communication network operating in ns-3. Each node in ns-3 interfaces with external systems through a tap bridge; thus, these implemented tab bridges can interface with the container through a virtual ethernet (veth) pair. The veth pair establishes the connection link between each node in the designed communication model and docker containers, supporting the bidirectional data flow through the emulated communication model. Figure 6 depicts the integration between the nodes inside ns-3 and the docker containers. Modbus TCP communication is used between the external centralized secondary controller (server) and the local controllers’ nodes in ns-3 (clients) through the four implemented docker containers, as demonstrated in Figure 5. A shared memory is deployed within the same Linux host computer to enhance the interfacing process further. Various sequences can interact with the same memory, making it straightforward for the docker containers and the ns-3 nodes to communicate simultaneously.

3.3. Data Extraction and Communication Network Performance Analysis

The network simulator ns-3 is crucial in supporting the detailed analysis of the communication network performance within the cyber–physical MG system. It provides multiple tools and methods for this purpose, ensuring the resilience and stability of the system. This section comprehensively assesses the interdependence between the communication network performance and the power system behaviors and describes in detail the approaches for data extraction from the designed communication network. To analyze the performance of the developed communication network and ensure its resilience and reliability in handling the system requirements in diverse conditions, information streaming among the control agents needs to be gathered online during the system operation. These tools can be seamlessly integrated with the designed code for the network model, facilitating the packet flow statistics being captured during the cyber–physical system operation. A post-analysis is then implemented to calculate the KPIs, such as end-to-end delay, percentage packet loss, and transmitted/received bitrates, which accurately describe the communication performance metrics at each node within the designed network.

Packet Tracing (PCAP) and FlowMonitor are highlighted as powerful tools within ns-3 [41,51]. They play a significant role in enabling data capturing and a detailed analytical visualization analysis within the developed network model. Packet Tracing, as an available technique in ns-3, records all system events during the real-time operation associated with packets corresponding to the flow of information. The PCAP tracing stores the packet-level data in the PCAP file format, which can be analyzed externally by tools, such as Wireshark [55], for post-real-time simulation analysis. The simulation model script enables PCAP tracing by including the command (EnablePcap) at the desired nodes to start data capture at the beginning of system operation.

FlowMonitor, a modular tool, complements PCAP tracing by monitoring high-level data [51]. It is designed to offer an overall view of communication performance through aggregating statistics metrics per flow. In the network context, each flow is identified by unique data, including the IP addresses for the source and destination nodes, the port for the source and destination, and the implemented transmission protocol. The link within the designed system among the control agents is identified by Flow ID, and during the system operation, we can gather the statistics for each Flow ID by installing the FlowMonitor module within the communication model script (FlowHelper. InstallAll()). The collected data during the online operation are exported in XML format for thorough analysis after the end of the simulation time [56,57].

In addition to collecting data internally from ns-3, another available way within our flexible system is to extract data by streaming it from docker containers. As described before, the containers are implemented to interface each local control node with the external physical system, setting up packet capture tools, such as tcpdump and tshark, with encapsulated applications inside the container. The collected data in PCAP files provide detailed network traffic statistics during the simulation time, facilitating the communication performance analysis. Accordingly, a thorough systematic study of the communication network performance within the MG system can be obtained by implementing these multi-faceted approaches for data extraction, as depicted in Figure 6. In alignment with NIST recommendations [26] in this work, we assessed the communication network performance by analyzing the end-to-end delay ${(Delay}_{E2E})$, transmitted bitrate (${BR}_{tr}$), received bitrate (${BR}_{rec}$), and percentage packet loss (${\%PL}_{com})$ at each node with the designed network, featuring a thorough quantitative analysis. These KPIs can be mathematically calculated from the data captured as follows [56]:

$${Delay}_{E2E}(ms)={\displaystyle \frac{\sum _{k=1}^{N_p}(T_{re, k}-T_{tr, k})}{N_p}}$$

(20)

$${BR}_{tr}(\mathrm{b}\mathrm{i}\mathrm{t}/\mathrm{s})={\displaystyle \frac{{Bits}_{tr}}{T_{tr, k}}}$$

(21)

$${BR}_{rec}(\mathrm{b}\mathrm{i}\mathrm{t}/\mathrm{s})={\displaystyle \frac{{Bits}_{rec}}{T_{re, k}}}$$

(22)

$${PL}_{com}(\%)={\displaystyle \frac{P_l}{P_{total}}}$$

(23)

where $T_{re, k} \mathrm{a}\mathrm{n}\mathrm{d} T_{tr, k}$ are the reception and transmission times of packet k, and $N_p$ is the total number of analyzed packets. ${Bits}_{tr}$ and ${Bits}_{rec}$ denote the total transmitted and received bits, $P_l$ is the lost packets, and $P_{total}$ presents the total number of packets.

3.4. Implementing the Standalone AC Microgrid in Real Time

The standalone AC microgrid integrates a four-parallel VSI operating under a grid-forming mode and is connected to a common AC bus to share the total load demand equally. For validating the efficient operation of the cyber–physical MG in real-time, the physical MG layer is implemented in the OPAL-RT (OP4610XG) real-time digital simulator through RT-LAB (version 2021.3.4) that seamlessly hyperlinks between MATLAB/Simulink and OPAL-RT. The implemented MG model in RT-LAB is converted to C-code appropriate for processing in real time; then, the converted model is loaded into the OP4610XG simulator via ethernet connection, allowing the operation in real time and the deployment of diverse operational conditions, as illustrated in Figure 7. This hardware-in-the-loop setup is integrated with the external centralized secondary controller computer through the implemented cyber layer to form the co-simulation cyber–physical MG system, as described in the previous subsections.

4. Experimental Results and Discussion

The experimental validation of the proposed standalone AC microgrid system equipped with the designed communication network is demonstrated in this section. Figure 8 presents the implemented co-simulation hardware setup for the cyber–physical AC microgrid system under study at the FIU smart grid testbed. The implemented setup integrates the cyber layer (ns-3) with the physical layer (OPAL-RT) of the AC microgrid system through the docker containers, facilitating the interdependence analysis of communication and control for smart microgrid systems, as explained in Section 3. Table 1 illustrates the cyber–physical system parameters in this study. It is worth mentioning that, with the scalability and flexibility of the utilized power systems and communication real-time simulators, this setup serves as a sustainable tool that can be adapted to accommodate several power system applications by redesigning the control and communication systems.

Table 1. The cyber–physical system parameters.

Parameters	Value
Common bus line-to-line voltage $\left(V_{MG}\right)$	208 V
Common bus frequency $\left(F_{MG}\right)$	60 Hz
VSI filter $(R_f{L}_f{C}_f$)	0.05 Ω, 3.5 mH, 50 μf
Output connector $\left(R_c{L}_c\right)$	0.05 Ω, 0.35 mH
Total connected load $(P_l, Q_l)$	28 kW, 800 VAR
Inner voltage controller ${(K}_{PE}, K_{IE})$	0.286, 500
Inner current controller ${(K}_{Pi}, K_{Ii})$	55, 1570
Secondary voltage controller $(K_{Pv\_sec}, K_{Iv\_sec})$	0.5, 50
Secondary frequency controller $(K_{Pf\_sec}, K_{If\_sec})$	0.15, 9
Number of docker containers	4
Number of tab bridges	4
Number of local area networks (LANs)	2
Number of controller nodes (n)	5
Number of routers (r)	2
Links between routers	P2P communication
Links in each LAN	CSMA communication

Table 1. The cyber–physical system parameters.

Parameters	Value
Common bus line-to-line voltage $\left(V_{MG}\right)$	208 V
Common bus frequency $\left(F_{MG}\right)$	60 Hz
VSI filter $(R_f{L}_f{C}_f$)	0.05 Ω, 3.5 mH, 50 μf
Output connector $\left(R_c{L}_c\right)$	0.05 Ω, 0.35 mH
Total connected load $(P_l, Q_l)$	28 kW, 800 VAR
Inner voltage controller ${(K}_{PE}, K_{IE})$	0.286, 500
Inner current controller ${(K}_{Pi}, K_{Ii})$	55, 1570
Secondary voltage controller $(K_{Pv\_sec}, K_{Iv\_sec})$	0.5, 50
Secondary frequency controller $(K_{Pf\_sec}, K_{If\_sec})$	0.15, 9
Number of docker containers	4
Number of tab bridges	4
Number of local area networks (LANs)	2
Number of controller nodes (n)	5
Number of routers (r)	2
Links between routers	P2P communication
Links in each LAN	CSMA communication

Figure 8. Hardware testbed setup for the cyber–physical AC microgrid under study.

Figure 8. Hardware testbed setup for the cyber–physical AC microgrid under study.

Extensive experiments have examined the resiliency and stability of the implemented cyber–physical system in diverse operational scenarios. The effectiveness of the developed communication network and control system is examined under variations in traffic patterns to achieve the required voltage and frequency adjustments. Low and high traffic load scenarios have been implemented to assess their impact on the MG’s stability and the communication network performance, as explained in the following subsections. The flow chart in Figure 9 streamlines the experimental validation process under each loading condition in the following subsections to obtain quantitative performance matrices for both the communication network and the power system performances. A comparative analysis was conducted to evaluate the impact of different traffic patterns on the communication network quality of services in terms of the delay values, percentage packet loss, and transmitted/received bitrates at each node within the developed communication network.

4.1. Scenario 1: Low-Traffic Pattern

Throughout this experiment, the performance of the cyber–physical MG system was evaluated under low-traffic conditions, characterized by a step change in total load demand. Specifically, the total power requested at the common bus was increased by 20 kW connected at t = 25 s, and this additional load was disconnected at t = 65 s. The experiment commenced by analyzing the MG performance, employing primary control only, and then the interdependence analysis of the communication-based control scheme was illustrated.

4.1.1. Doop-Based Control Scheme (Low-Traffic)

Under this loading pattern, we initially assessed the MG performance, utilizing the droop-based control (communication-free control scheme) to validate the robustness of the designed primary control layer in regulating the MG common bus’s voltage and frequency to the designed ratings of 208 V and 60 Hz. Figure 10 and Figure 11 depict that the four parallel-connected grid-forming VSIs, utilizing droop-based control, effectively met the load requirements while equally sharing the total active and reactive power demands.

Regarding the voltage and frequency stability of the MG, the droop-based control, along with the inner control loops, have successfully regulated the voltage and frequency at the common AC bus under the tested load profile. As illustrated in Figure 12, the AC bus voltage is stable; however, it does not reach the rated line-to-line voltage of 208 V, employing the primary control loop only. Continuous voltage deviations ${\mathsf{\Delta }V}_{MG}$ exist, and these deviations increment as the load demand rises. A similar limitation in the droop control for eliminating the steady-state errors observed in the frequency of the MG is shown in Figure 13. While the frequency remains stable and dynamically adjusted with load variations, it cannot be constant at 60 Hz.

Under the specified load pattern in this test, the observed droop in voltage (approximately 1.5%) and frequency deviation ${\Delta F}_{MG}$ (ranging from 0.22 Hz to 0.35 Hz) can be accepted as they are within the permissible limits. Nevertheless, under sudden high-power changes, only deploying a droop-based primary control layer would be insufficient to keep the MG stable. Accordingly, the standalone MG will experience variations in voltage and frequency beyond the acceptable limits. This limitation in the droop control highlights the need to utilize the secondary control layer to ensure the MG’s stability and resiliency across various operational scenarios.

4.1.2. Communication-Based Centralized Secondary Control (Low-Traffic)

The communication-based centralized secondary control layer for regulating the MG’s voltage and frequency is established and verified in this test, employing the developed communication network model (Section 3.1) within the implemented setup, as presented in Figure 8. The secondary control is successfully implemented following the steps in their order, as illustrated in Figure 9. The bidirectional information flow between the physical MG (local controllers) and the centralized controller through the developed cyber layer has been initiated, as illustrated in Figure 14, to regulate the MG’s voltage and frequency simultaneously in real time. Consequently, this fulfills the research gap in analyzing the MG operation as a CPPS, while including the real-time implementation of both power and communication networks.

As demonstrated in Figure 15 and Figure 16, the voltage and frequency deviations have been effectively eliminated, ensuring the cyber–physical MG operates without steady-state errors at the common bus. The designed communication network has enabled the implemented secondary controller to regulate the common bus voltage and frequency at their nominal values 208 V and 60 Hz, even under the considered load changes. This addresses the limitations of the droop control that, while functioning as a proportional controller, could not eliminate steady-state errors in voltage and frequency.

Figure 14. Bidirectional data flow through the communication network (Wireshark capture).

Figure 14. Bidirectional data flow through the communication network (Wireshark capture).

4.1.3. Communication Network Performance Analysis (Low-Traffic)

To achieve the study’s objective of providing a comprehensive interdependence examination among the communication networks and the control system within the microgrid framework, this section introduces the performance analysis of the communication network. The performance metrics used to assess the designed network behavior include the delay $\left({Delay}_{E2E}\right)$, the transmitted bitrate (${BR}_{tr}$), received bitrate (${BR}_{rec}$), and percentage packet loss (${\mathrm{\%}PL}_{com})$ at each node within the network. Using a combination of data extraction approaches, as illustrated in Section 3.3, the network performance is evaluated under low-traffic conditions, which involve one step up/down in the total connected load. This analysis aims to assess the designed communication network’s effectiveness in supporting resilient and reliable microgrid control in alignment with the communication requirements.

During operation, continuous communication is established between the four droop-based controllers in the primary layer and the external centralized secondary control through the communication network, supporting real-time voltage and frequency adaptation. Accordingly, the effectiveness of the network in transferring control commands is evaluated at each local controller (LC) node, identified as n11, n12, n13, and n14 in LAN 1, as shown in Figure 5. Under a low traffic load scenario, the communication network exhibits optimal performance due to the modest control actions requested, limiting the data flow within the network with a light burden on the network. The network experienced a low value of ${Delay}_{E2E}$ at each local controller node, consistently ranging from 12 ms up to 15 ms, as illustrated in Figure 17a, which is significantly below the 100 ms acceptable threshold for communication delays in the control of microgrids, ensuring the promptness of the designed network in handling control actions, with rapid response to keep the MG stable.

In terms of the ${BR}_{tr} \mathrm{a}\mathrm{n}\mathrm{d} {BR}_{rec}$, their values are stable at each node within the designed network capacity (10 Mb/s) between the local area networks without contingency signs. The noticed bitrates at each node range from around 0.2 Mb/s to approximately 0.4 Mb/s, as Figure 17b depicts, indicating the efficient utilization of the permissible bandwidth for bidirectional data flow. Consequently, the communication network does not experience noticed data loss, confirming the robustness and reliability of the designed communication model.

Figure 17. Communication network performance metrics at each node (low-traffic scenario): (a) mean delay (ms), (b) transmitted and received bitrates (Mb/s).

Figure 17. Communication network performance metrics at each node (low-traffic scenario): (a) mean delay (ms), (b) transmitted and received bitrates (Mb/s).

4.2. Scenario 2: High-Traffic Pattern

The cyber–physical MG’s functionality was investigated in this test across high-traffic load circumstances, which were identified by substantial alterations in the magnitude of the total connected load and the frequency of variations. This server operational condition involves the power demand’s incrementation and subtraction with a value of 45 Kw in addition to the total connected load value, with a high rate of change that equals a five times more frequent step up/down than the low-traffic scenario. Consequently, the effectiveness and resiliency of the hierarchical control system and the communication network model are validated in their ability to sustain voltage and frequency stability in diverse practical applications. The validation process follows the steps described in Figure 9, initiated by MG performance evaluation, using the primary control only, followed by operation with the communication-based secondary control, with a comprehensive assessment of the communication network quality of services.

4.2.1. Doop-Based Control Scheme (High-Traffic)

The primary control layer was assessed in handling the heavy traffic scenario without relying on the communication infrastructure. Thus, we can validate to what extent the droop-based control assigned to each grid-forming VSI can keep the voltage and frequency close to their limits at the common bus under these severe load fluctuations. The voltage and frequency responses at the common AC bus are presented in Figure 18 and Figure 19, respectively. It should be noticed that the primary control layer stabilizes the bus voltage and frequency. However, it cannot satisfy the MG’s voltage and frequency nominal values.

The increments in the total demand were shared equally among the parallel-connected DG units, as depicted in Figure 20 and Figure 21 under this test; however, the results revealed restrictions on the efficacy of the droop control, and the deviations started to be beyond the limits tolerated in standalone AC microgrids, unlike the first scenario. The observed frequency deviations are more pronounced, reaching almost 0.66 Hz after connecting the additional load. Additionally, the drop in bus voltage has notably increased by approximately 3%. These results demonstrate that relying on conventional droop control only in some situations leads to destabilization consequences and underscores the need for the secondary control layer to restore MG stability.

4.2.2. Communication-Based Centralized Secondary Control (High-Traffic)

To enhance the MG resiliency and stability under this severe loading condition and address the shortcomings with the deployment of the primary control only, the developed communication network model was activated in this phase to facilitate the connection of the secondary control layer. Thus, a validation of the efficiency and reliability of the designed communication network under high-traffic conditions will be illustrated to confirm to what extent the designed network met the communication requirements for controlling microgrids in actual implementations. Figure 22 and Figure 23 prove the effectiveness of the communication-based secondary control in eliminating the deviations in voltage and frequency introduced by the primary control layer. Thus, the nominal value for the common bus voltage is satisfied at 208 V, and the frequency is 60 Hz, thanks to the timely transfer of control commands, even under this loading condition, which is characterized by high dynamics.

4.2.3. Communication Network Performance Analysis (High-Traffic)

Reliable communication is the backbone for facilitating the control objectives. Thus, evaluating the communication networks’ performance is crucial in cyber–physical microgrid control studies to present a thorough overview of the cyber–physical system operation. In this section, the communication performance is analyzed under the implemented high-traffic pattern in terms of the key performance indicators, ${Delay}_{E2E}$, ${BR}_{tr}$, ${BR}_{rec},\mathrm{a}\mathrm{n}\mathrm{d}$ ${\mathrm{\%}PL}_{com}$, at each local control node. Thus, the impact of increasing the traffic load on the designed communication network resiliency can be evaluated compared to the low-traffic scenario.

Due to the frequent rapid variations in the total load demand, the communication network experienced a substantial boost in data transmission to achieve the voltage and frequency regulation process. This affects the overall network performance, with significant variations in the KPIs values compared to the low-traffic scenario. As illustrated in Figure 24a, the delays at each local controller node ranged from 50 ms to 57 ms, which was considerably longer by approximately four times than their values in the low-traffic scenario.

Despite the stress on the communication network under high-traffic operational conditions, the designed network performed effectively and reasonably maintained stability, with bitrates below its capacity limits. As depicted in Figure 24b, the transmitted and received bitrates span around 1 to 1.3 Mb/s, demonstrating the effective use of the accessible bandwidth with the proposed network. With the increased strain and contingency on the network, a drop in transmitted packets exists compared to zero packet loss under low-traffic conditions. However, the observed percentage packet loss, as shown in Figure 24c, was around 0.08% to 0.12%, which can be accepted and does not considerably impact the MG stability.

Overall, the communication network performance metrics reinforce the effectiveness and reliability of the proposed communication network design in handling the control objectives under the studied traffic variations, without degradation under high-traffic conditions. Consequently, it experimentally verified that the proposed cyber–physical system is well designed to accommodate future scenarios, with probable extensions. Additionally, these findings indicate the need for continuous traffic monitoring to ensure operation within the network capacity. Thus, further optimization methods on the communication network design can be considered as future work, to enhance the overall performance and sustain MG stability and resiliency.

Figure 24. Communication network performance metrics at each node (high-traffic scenario): (a) mean delay (ms), (b) transmitted and received bitrates (Mb/s), (c) percentage packet loss (%).

Figure 24. Communication network performance metrics at each node (high-traffic scenario): (a) mean delay (ms), (b) transmitted and received bitrates (Mb/s), (c) percentage packet loss (%).

5. Conclusions

This study provides a comprehensive analysis of the interdependence between communication networks and control systems in standalone AC microgrids, highlighting the critical role of communication in supporting the resilient and stable operation of cyber–physical microgrids. A communication network model that enables a hierarchical control architecture, integrating primary and centralized secondary control layers, is proposed in this study. The effectiveness of the designed communication model and the communication-based centralized secondary controller are experimentally validated through the employment of ns-3 for emulating the communication network and opal-rt for presenting the MG’s physical layer in a co-simulation testbed setup. Extensive experiments have been conducted to ensure the reliability and resiliency of the proposed cyber–physical system performance in the real-time regulation of the MG’s voltage and frequency under diverse traffic variations.

The experiments’ findings illustrate that while the droop-based primary control can effectively manage the power sharing among the parallel-connected grid-forming inverters, it fails to eliminate the steady-state errors in voltage and frequency, especially under high-traffic patterns. The voltage and frequency deviations have notably boosted compared to the low-traffic scenario, affecting MG stability. Accordingly, the introduced real-time centralized secondary controller supported by a reliable and flexible communication structure has effectively regulated the voltage and frequency, keeping them at their nominal values in instances of severe fluctuations in load. The quantitative analysis of the communication network performance, regarding the delay, percentage packet loss, and transmitted/received bitrates, reveals that the designed network remained reliable and stable across diverse traffic variations. Consequently, it underscores the critical need for communication to support the effective control of standalone microgrids and ensure the overall cyber–physical system’s resiliency and stability. In our future work, further exploration regarding the influence of communication network challenges, such as delays or packet loss, in addition to that of cyber vulnerabilities on the MG performance, will be investigated. Moreover, various intelligent control schemes can be developed to ensure MG resiliency and stability across several practical challenges.