An Empirical Evaluation of Communication Technologies and Quality of Delivery Measurement in Networked MicroGrids

Abstract

Networked microgrids (NMG) are gaining popularity as an example of smart grids (SG), where power networks are integrated with communication technologies. Communication technologies enable NMGs to be monitored and controlled via communication networks. However, ensuring that communication networks in NMGs satisfy quality of delivery (QoD) metrics such as the round trip time (RTT) of NMG control data is necessary. This paper addresses the communication network types and communication technologies used in NMGs. We present various NMG deployments to demonstrate real-life applicability in different contexts. We develop a real-time NMG testbed using real hardware, such as Cisco 4331 Integrated Services Routers (ISR). We evaluate QoD in NMG control data by measuring RTT under varying relative network congestion levels. The results reveal that high-variance background traffic leads to greater RTTs, surpassing the industrial communication response time requirement specified by the European Telecommunications Standards Institute (ETSI) by over 25 times.

1. Introduction

Traditional grids (TG), designed over 100 years ago, were built to supply electricity to communities [1]. Their centralised structure comprises generation, transmission, distribution, and end-users phases. However, they no longer meet the modern world’s needs adequately due to increasing energy demands, increasing distributed energy resources (DER) deployments, and extreme-weather-related power outages. Smart grids (SG) are promising, resilient, efficient, and controllable solutions.

SGs have various applications, including advanced smart metering infrastructure (AMI) [2], vehicle-to-grid (V2G) [3], electric vehicles (EV) [4], and networked microgrids (NMG) [5]. The main idea of SGs is to facilitate distributed power grids that provide a two-way exchange of information and electricity. We illustrate the transition from TGs to SGs to highlight the role of communication networks from generation to end-users in Figure 1. NMGs are defined in IEEE 2030.7–2017 as a group of interconnected loads and DERs with clearly defined electrical boundaries that act as a single controllable entity with respect to the grid and can connect and disconnect from the grid to enable operation in both grid-connected or off-grid modes [6]. They constitute a critical component of community electrification and may help to achieve net zero carbon emission targets. While NMGs and SGs employ similar technologies, such as photovoltaic (PV) panels, wind turbines, and battery storage, NMGs have the ability to operate independently, running as self-sufficient power systems [7].

Figure 2 represents the communication and power networks integration in NMGs. Communication networks are crucial for two-way control data exchange between NMG endpoints. Wired and wireless communication technologies are utilised to develop communication networks in the NMG configurations [8,9]. The selection of communication technology depends on the requirements of NMG deployment. Integration of these technologies into NMGs enables system management, including power line monitoring, control data transmission, and supply–demand management [10]. The selected communication network needs to consider quality of delivery (QoD) requirements, including round trip time (RTT), packet loss, and throughput [11]. As various traffic types share the same communication network with NMG control data, large and variable RTTs become a QoD concern. Increased RTTs negatively impact control data transmission, potentially causing delayed control signal propagation. These challenges in the communication network compromise the effective operation of NMGs.

Communication between NMG endpoints, i and j, in Figure 3 consists of two types of messages: control data and acknowledgment (ACK) messages. At time index n, sender endpoint i sends control data to receiver endpoint j at time $t_{ij}\left[n\right]$. Once the control data are received, the receiver endpoint responds by sending an ACK message back to the sender endpoint. The sender endpoint receives the ACK message at $t_{ij}[n+2]$. During this process, both the control data and the ACK message are subject to transmission $t_t$ and propagation $t_p$ delays. The total elapsed time between sending the control data and receiving the ACK message is defined as the RTT. The RTT between NMG endpoints i and j is denoted as $r_{ij}$ and is measured as follows:

$$r_{ij}(t_{ij}\left[n\right],t_{ij}[n+2])=2t_t+2t_p.$$

(1)

One-way delay in NMGs is measured in [12,13]. The RTT measurements are used to assess network performance and network congestion in the transmission control protocol (TCP) [14]. Given that NMGs allow bidirectional transmission of control data and ACK messages, RTT measurement provides a more comprehensive approach to understanding QoD measurement in NMGs. Few papers measure RTT as a QoD metric [15], while many papers employ simulation tools to represent NMGs in different areas of communication networks [16]. The majority of papers overlook the impact of background traffic on the communication network between NMG endpoints. We address this gap by measuring the RTTs of NMG control data using a real hardware testbed under varying network congestion levels. We investigate whether background traffic adversely affects the QoD of NMG control data. To this end, we inject background traffic with varying bit rates into Layer 3 (L3) of the open systems interconnection (OSI) network model between NMG endpoints. We hypothesise that background traffic with high bit rate variance leads to RTTs surpassing industrial latency requirements.

To set the context for our empirical evaluation of RTT in NMGs, we begin by highlighting the limitations of existing approaches that focus on one-way delay measurement or simulation-based studies. To this end, in Section 2 we classify the different types of NMG. In Section 3, we discuss current QoD measurement papers in the area of NMGs. This establishes the need for a more realistic assessment of QoD under dynamic network conditions. However, it is crucial to note that this paper is not a review article; rather, it presents a novel, empirical study using a real hardware testbed to measure RTTs under varying levels of background traffic. This background allows us to then bridge the gap between theoretical models and practical NMG deployments, offering insights into the impact of real-world network congestion on control data transmission. We explore the different types of communication networks deployed to support NMGs in Section 4 and present real-life NMG deployments to highlight the usage of communication technologies in NMG projects. In Section 5, we present the NMG testbed design and provide RTT measurements derived from various experimental cases to examine our hypothesis. The aim of this work is to empirically measure RTT in a realistic, hardware NMG testbed. The main contribution is the RTT measurements, not the testbed’s design. Finally, in Section 6, we present future work and challenges in NMGs.

2. NMG Classification

NMGs are classified based on system topology, control strategy, operation mode, generation scale, and application type. The system topology categorises NMGs based on power supply, such as alternating current (AC), direct current (DC), and hybrid [17]. Control strategies divide NMGs into three groups: centralised, decentralised, and hybrid [18]. In the centralised control strategy, all endpoints are controlled by the same networked microgrid controller (NMGC). In the decentralised strategy, endpoints are managed by local controllers. The hybrid strategy combines both strategies, with endpoints being connected to local controllers, and all local controllers being controlled by an NMGC. Operation modes consist of grid-connected and off-grid modes based on the main grid connection status. Off-grid mode ensures continuous electricity flow by operating independently when main grid faults occur [19]. Based on a generation scale, NMGs are classified as small, up to 10 MW; medium, from 10 MW to 100 MW; and large, greater than 100 MW [20]. Finally, NMGs can be used in various areas, including communities, military bases, and business parks. Different generation and storage assets such as solar panels, wind turbines, and flow batteries can be used to develop NMGs, as illustrated in Figure 4. While electricity is shared between NMG endpoints, control data that hold instant current, voltage, and power values are transmitted between endpoints.

3. QoD Measurement for NMGs

As NMGs gained popularity in recent years, they introduced several research topics that require examination. One of these topics is QoD measurement in NMGs. The power engineering community typically uses latency as a metric for QoD measurements, specifically measuring the one-way delay between NMG endpoints. In contrast, the SG proposes two-way control data communication, introducing RTT as a measurable metric of delay. Table 1 highlights the research gap that this paper addresses. We classify papers using physical equipment, e.g., switches, routers, and OPAL-RT, for power and communication networks as real implementations. Communication network simulators, e.g., Network Simulator-3 (NS-3) and OPNET, are not considered real implementations because they partially represent real-life scenarios. We consider online gaming, video streaming (VS), and similar applications as real-life background traffic. We also examine whether papers address network experiences congestion.

In [21], the authors propose a communication network emulation model that represents the communication network using Network Simulator-3 (NS-3) to examine communication performance between NMG endpoints. Throughout the real-time simulation, the User Datagram Protocol (UDP) and the TCP are used to transfer NMG control data between MGCC and NMG endpoints under different intensity bit rate traffic levels. The proposed model reveals that TCP causes higher one-way delay and packet loss rates, which affects the QoD between NMG endpoints. However, since background traffic is not injected into the communication network in this paper, the one-way delay remains at a millisecond level. In modern communication networks, various background traffic flows are present. The lack of such traffic is unrealistic and limits the understanding of QoD measurement in NMGs.

An Ethernet-based real-time simulation NMG testbed is implemented in [22]. The proposed NMG testbed is comprised of two layers: a power network, which is simulated by the OPAL-RT power network simulator, and a communication network, which is developed using the OPNET networking simulator. This study examines how latency affects the rate of convergence. The authors reveal that latency increases with increased levels of congestion. They also highlight that a 90% level of congestion causes the rate of convergence to slow down in the NMG setting, which indicates that latency between NMG endpoints negatively affects the QoD in NMG control data. The authors of [23] develop an NMG simulation testbed integrating power and communication networks. The NMG testbed employs GridLAB-D for power network simulation and NS-3 for communication network implementation. This study investigates the interdependencies between NMGs and the communication network by evaluating the impact of varying levels of communication delays to the stability of NMGs. The results demonstrate that congested network conditions lead to increased communication delays. These increased communication delays negatively affect the power generation capacity, resulting in power loss.

An NMG testbed that employs both wired and wireless communication is simulated in [13]. The authors evaluate the one-way delay between NMG endpoints in home-based and neighbourhood-based networks while varying the bandwidth levels and the number of smart consumers. The authors reveal that one-way wireless communication causes slight variability in one-way delay, which is measured to be greater than that of wired communication. As expected, the one-way delay in both wired and wireless communication is measured to be lower when the available bandwidth is increased. However, the authors do not introduce additional traffic into the communication network, which leads to the one-way delay being measured at very low millisecond levels.

One-way delay and packet loss rates in NMG control data are measured under various levels of network congestion in [24]. The paper utilises a real-time testbed that facilitates real hardware components. The authors examine how different network perturbation levels affect QoD in NMG control data by measuring one-way delay and packet loss rates. The following experimental scenarios are carried out using both wired and wireless communication technologies for NMG control data transmission between NMG endpoints: (1) NMG control data are sent over an Ethernet connection exclusively; (2) the control data are transmitted via a Wi-Fi connection without any interference perturbation; (3) the control data are transmitted over a Wi-Fi connection with continuous interference perturbation that leads to noticeable communication disruption; (4) the Wi-Fi connection experiences persistent high-level interference, resulting in near-total communication disruption during the NMG control data transmission; (5) the control data are sent over a Wi-Fi connection with intermittent interference where interference fluctuates between the levels of cases (3) and (4). As a result, the packet loss rates are found to be the highest when communication between NMG endpoints is almost totally disrupted. The authors also observe that slightly higher one-way delay values occur when a Wi-Fi connection is used, which is expected, as high-level persistent perturbation exists in the connection. However, none of these experiments cause greater one-way delays in NMG control data because the perturbation levels remain stable during the experiments.

A small-scale NMG testbed is developed in [25]. The RTT measurement is carried out without injected background traffic. The authors measure the RTTs of NMG control data using two different packet sizes, 1 byte and 109 bytes, in separate experiments. However, these constant packet sizes in NMG control data do not represent real-life traffic, which involves varying packet sizes. Packet size distributions in real-life video streaming applications, as measured in [26], show variability. For instance, packet sizes vary between approximately 100 bytes and 1400 bytes in Google Hangouts, with variance in traffic distribution. The limited packet size variance in NMG control data causes RTTs to be measured low and remain at the millisecond level. In [27], RTT measurement is performed in the presence of various levels of background traffic. The authors design an NMG testbed using real hardware that represents NMG endpoints in a small area network and inject different types of background traffic, including VS. Although control data are transmitted quickly in the small area network, the authors conclude that greater RTTs occur in the presence of high-variance background traffic. A software-defined networking (SDN)-based campus NMG testbed is developed to analyse the communication channels for NMG control data traffic during cyber–physical disruptions [28]. The authors examine the robustness of SDN in NMGs by measuring RTTs of NMG control data. The RTT measurement is performed under two experimental scenarios, including no congestion and constant congestion levels in the SDN channel. The results demonstrate that measured RTTs remain at millisecond levels with the help of SDN infrastructure as NMG control packets are diverted to other paths when the congestion level is increased in the communication links. Nevertheless, only NMG control data are transmitted over the SDN channel, and no additional background traffic injection is considered.

One-way delay is measured using either simulation tools or a real-life testbed in many papers. However, there are two main points that limit the realism of QoD measurements in current papers. First, NMGs are designed to transfer control data bidirectionally so that the RTT measurement gives a better understanding of data exchange between NMG endpoints. Second, simulation tools simulate real-life scenarios in NMGs, but they are not capable of representing them effectively.

Table 1. Summary of QoD Measurements in Current Papers.

Reference	Real Power Network	Real Communication Network	Congested Network	Real-Life Background Traffic	RTT Measurement
[13]	✓
[21]	✓		✓
[22]	✓		✓		✓
[23]			✓
[24]	✓	✓	✓
[25]		✓			✓
[28]	✓	✓	✓		✓
This paper	✓	✓	✓	✓	✓

Table 1. Summary of QoD Measurements in Current Papers.

Reference	Real Power Network	Real Communication Network	Congested Network	Real-Life Background Traffic	RTT Measurement
[13]	✓
[21]	✓		✓
[22]	✓		✓		✓
[23]			✓
[24]	✓	✓	✓
[25]		✓			✓
[28]	✓	✓	✓		✓
This paper	✓	✓	✓	✓	✓

4. Communication Networks and Technologies in NMGs

Communication networks in NMG are categorised into three main categories based on coverage distance and data rate [29]: Wide Area Networks (WANs), Field Area Networks (FANs) or Industrial Area Networks (IANs), and Home Area Networks (HANs) or building area networks (BANs), as illustrated in Figure 5. A relation between coverage distance and data rate requirements in NMG communication networks is presented in Figure 6.

We summarise the communication technologies used in different network types in

Table 2. Wired and wireless communication technologies used in NMGs [19,24,31,32,33,34,35].

Connection Type	Coverage	Data Rate	Applicability	Advantages	Disadvantages	Application
Wired Technologies
Ethernet	Up to 100 m	Up to 100 Gbps	WAN, FAN, HAN	Widely available, suitable for low coverage	Limited coverage distance	Home automation
Fiber-optic	Up to 60 km	Up to several Tbps	WAN, FAN	High data rate, fast transmission	More expensive than copper cables and some wireless technologies	Communication channel in generation site
Narrowband PLC	Up to 3 km	Up to 500 kbps	WAN, FAN, HAN	Affordable, widely available	Greater electromagnetic noise, susceptible to disruptions, low data rate	Transmission of electricity
Broadband PLC	Up to $1.5$ km	Up to 300 Mbps	WAN, FAN, HAN	Affordable, widely available	Greater electromagnetic noise, susceptible to disruptions	Transmission of electricity
HomePlug	Up to 200 m	Up to 10 Mbps	HAN	Affordable, energy efficient	Limited coverage distance	Home automation
Asymmetric Digital Subscriber Line (ADSL)	Up to 5 km	Up to 8 Mbps	HAN	Affordable, energy efficient	Limited coverage distance, low data rate	Home automation
3G	Up to 75 km	Up to $7.2$ Mbps	WAN, FAN, HAN	Widely available, existing infrastructure	Shared network infrastructure, risk of congestion	Smart meter data collection
4G	Up to 12 km	Up to 100 Mbps	WAN, FAN, HAN	Wide coverage distance, fast data transmission	Shared network infrastructure, risk of congestion	Two-way communication between smart device and control centre
Wireless Technologies
5G	Up to 50 km	Up to 20 Gbps	WAN, FAN	High data rate and reliability, low latency	Costly infrastructure, security issues	Data exchange between MGs
ZigBee (IEEE 802.15.4)	Up to 100 m	Up to 250 kbps	HAN	Affordable, low complexity	Limited coverage distance, low data rate, slow processing rate	Home automation
Bluetooth (IEEE 802.15.1)	Up to 100 m	Up to 721 kbps	HAN	Affordable, power efficient, low complexity	Short coverage distance, low data rate, poor security	Home automation
LoRaWAN	Up to 15 km	Up to 50 kbps	WAN, FAN	Wide coverage distance, affordable and secure communication	Low data rate	Monitoring of electricity transmission towers
Wi-Fi (IEEE 802.11 b/g/n)	Up to 1 km	Up to 600 Mbps	WAN, FAN, HAN	Effective in short range, affordable	Limited coverage distance	V2G

. Both wired and wireless communication technologies are utilised in NMGs, offering distinct capabilities in terms of coverage, data rates, and applicability. Wired technologies, such as Ethernet cables, offer high data rates with a limited coverage distance, while 4G cellular networks provide extended coverage at lower data rates. Various communication technologies are used in each type of communication network. Real-life NMGs, including communication technologies used, scale, and generation components, are tabulated in

Table 3. Real-life NMG Deployments and the Communication Technologies used.

Organisation	Location	Communication Technology	Scale	NMG Component(s)	Reference(s)
Schneider Electric	Grenoble, France	IoT (e.g., ZigBee) and Cloud	450 kW	PV, micro-CHP, Battery	[36]
NEDO	Hachinohe, Japan	Ethernet and Fiber-optic cables	600 kW	PV, Wind, Bio-gas, Battery	[37,38,39]
NEDO	Kyoto, Japan	Ethernet	650 kW	PV, Wind, Bio-gas, Battery	[37,38,39]
Hitachi Europe Limited	Isles of Scilly, England	IoT (e.g., 3G), Ethernet and Cloud	800 kW	PV and Battery	[40,41]
Schneider Electric	Fairfield, CT, USA	IoT (e.g., ZigBee) and Cloud	$3.25$ MW	PV, Fuel Cell, Natural Gas	[42]
Commonwealth Edison Company	Bronzeville, IL, USA	Fiber-optic cables, ZigBee	$7.5$ MW	PV, Bio-gas, Battery	[43,44]
United Kingdom Research and Innovation	Orkney, Scotland	4G, Wi-Fi and Cloud	8 MW	PV, Wind, Tidal, Battery	[45,46,47]
Horizon Power	Onslow, Australia	4G	11 MW	PV and Battery	[48]
Scottish and Southern Electricity Networks	Shetland, Scotland	IoT and Cloud	$12.5$ MW	Wind, Tidal and Battery	[49,50]
Siemens	Tampere, Finland	Ethernet, Cloud, ZigBee	16 MW	PV, Fuel Cell, Battery	[51,52]

. Practical deployments demonstrate that multiple communication technologies are employed in NMGs to manage diverse generation components, with the generation capacity reaching up to 16 MW.

WANs are the highest tier of communication networks in NMGs, covering large geographical areas where substantial electricity generation occurs. They also cover the transmission of power generated from generation sites to electricity substations. Wide-area monitoring, wide-area protection, and wide-area control are examples of WANs in NMGs [29]. The coverage distance of WANs ranges from 10 km to 100 km. A large number of data points are needed to transmit control data for system stability and decision-making in NMGs at a fast rate, ranging from 10 Mbps to 1 Gbps. Given that wide area coverage is needed in WANs, communication technologies providing broad geographical coverage are employed in NMGs. These include cellular, satellite communication, LoRaWAN, NB-IoT, WiMAX, cloud solutions, and fiber-optic cables [30,53].

FANs are the intermediate stage, and they facilitate information flow between WANs and HANs. For example, they can be situated between power stations and industrial business parks. FANs are a critical layer of communication networks in NMGs, receiving control data from distribution sites, such as electricity substations, and transmitting them to customer premises. The area coverage of FANs varies from a few meters up to 10 km, and the data rate ranges from 100 kbps to 10 Mbps. WiMAX, cellular, power line communication (PLC), RF-Mesh, Bluetooth, ZigBee, and Ethernet are communication technologies in FANs for NMGs [54,55].

The smallest type of network for NMGs are HANs, also called BANs. These types of network represent homes and buildings where small-scale NMGs are deployed. They cover short distances and facilitate low-bandwidth technologies. Smart meters in HANs enable local-level control data collection for NMGs [56]. Given resource limitations in smart meters [57], computationally low-cost communication technologies are necessary in HANs [58]. Both wired and wireless technologies are utilised to gather data from homes and buildings. Typical communication technologies in HANs include ZigBee, Wi-Fi, Bluetooth, cellular, and Ethernet [59]. In Ireland, Electricity Supply Board networks (ESBN) and Greencom networks employ Wi-Fi-based units for smart meter monitoring in HANs in the Dingle Peninsula electrification project [60].

5. Designing an NMG Testbed and Numerical Results

In this section, we present an empirical evaluation of RTT measurements using a representative NMG testbed. The focus of this work is on the RTT measurements obtained under realistic network conditions, rather than the novelty of the testbed’s design. The reliability of our RTT measurements is directly tied to the realistic nature of our testbed. Therefore, we start by detailing its configuration. In closing, we discuss the implications of these measurements for real-world deployments.

We develop the NMG testbed, as shown in Figure 7 and Figure 8, using two Cisco 4331 Integrated Services Routers (ISR) to facilitate connectivity between NMG endpoints. The clocks of ISRs are synchronised using Network Time Protocol (NTP) to accurately measure the QoD of NMG control data. Each NMG endpoint comprises a 48 V DC source, and each DC source is connected to a $0.242$ $\Omega$ resistor. A variable resistor with a maximum of $12.7$ $\Omega$ provides load to the entire testbed. Each DC source is connected to a PC using an analog-to-digital converter (ADC). This configuration represents an NMG deployment in a FAN, operating between customer premises and the utility data center [61]. HANs are typically implemented using layer-2 (L2) switches of the OSI model. They represent NMG generation sites in limited areas such as homes. While L2 switches are fast communication network devices, they experience low variance in background traffic compared to FANs and WANs. FANs employ multiple L3 ISRs to link NMG generation sites over larger areas. The ISRs handle complex network traffic with high-variance background traffic. A greater number of ISRs are used in WANs to connect diverse NMG generation sites and to manage high-variance background traffic, such as large wind farms that electrify cities.

Three DC sources are linked to desktop PCs using fast Ethernet cables, providing 100 Mbps bandwidth. The ISRs are interconnected via a serial cable that provides 45 Mbps bandwidth. This serial connection is the bottleneck link that limits maximum bandwidth to 45 Mbps [62]. To increase the relative congestion level and to inject various background traffic, we integrate a laptop into the NMG testbed. Background traffic in this paper characteristically represents real-life VS services, such as Apple TV, which suggests a minimum of 25 Mbps for 4K VS [63]. Both the background traffic and the maximum available bandwidth provide a realistic representation of real-life scenarios.

LabVIEW, a graphical programming language, is utilised to transfer control data within NMG endpoints and to monitor the entire NMG testbed. We stream real-time 4K 120 frames-per-second (FPS) and 8K 60 FPS videos between NMG endpoints for 2 min in each experiment. We then measure the RTTs of NMG control data in the presence of VS. We stream real-time videos using VLC media player. VLC is an open-source multimedia player that operates across different operating systems [64]. VS traffic is injected using various networking protocols, including the TCP, the UDP, and the Real-Time Streaming Protocol (RTSP). In some cases, we inject real-time videos into the communication network between NMG endpoints using a combination of these protocols. We also inject Internet Control Message Protocol (ICMP) packets by using a script that generates with random bit rate and interval times. In total, we conduct over 40 experiments. We develop the network streams (NS) algorithm to send NMG control data between endpoints. This algorithm employs writer–reader functions at each endpoint. The reference current, voltage, and power values are encapsulated within control data and transmitted between NMG endpoints via data streams. The writer and reader functions incorporate a 1 ms delay to process the incoming control data. We consider this delay as the transmission delay. Once the communication between NMG endpoints is complete, the connection is terminated.

We monitor the communication network between NMG endpoints to sniff packets and collect control data using the WinDump sniffing tool. Windump is a TCPDump adaptation for the Windows environment [65]. We measure the mean RTT, ${\mu }_{ij}$, between NMG endpoints, i and j, using a sequence of RTT measurements, $M+1$, which is represented as $r_{ij}\left[0\right],r_{ij}\left[1\right],\dots ,r_{ij}\left[M\right]$. The mean RTT is calculated as

$${\mu }_{ij}={\displaystyle \frac{{\sum }_{n=0}^M{r}_{ij}\left[n\right]}{M+1}}.$$

(2)

The standard deviation of RTT between NMG endpoints, i and j, is represented as ${\sigma }_{ij}$ and measured as

$${\sigma }_{ij}=\sqrt{{\displaystyle \frac{{\sum }_{n=0}^M{(r_{ij}\left[n\right]-{\mu }_{ij})}^2}{M}}}.$$

(3)

We propose a novel formula to quantify the relative congestion level between NMG end-points in Equation (4). We represent the relative congestion level between NMG endpoints, i and j, as c. We consider the following parameters: total transmitted data in bits, ${\alpha }_{td}$; maximum available bandwidth in Mbps, $\Lambda$; experiment duration, $\tau$; and a constant scale-up coefficient, $\omega$, which is 100. The relative congestion level is determined as

$$c=\left({\displaystyle \frac{{\alpha }_{td}}{\Lambda \times t}}\right)\times 100\times \omega .$$

(4)

We evaluate the QoD of NMG control data by measuring the mean and standard deviation of the RTTs at different relative congestion levels. These evaluations are compared with industrial communication response time requirements as tabulated in

Table 4. Communication response time requirements in intelligent electronic devices (IED) [66,67].

Standard	Protection Message	Control Message	Monitoring Message	Operation Message	Billing Message
ETSI TR 102 935 [68]	Up to 10 ms	100 ms	1 s	N/A	1 h to 1 day
IEC 61850 [69]	4 ms	100 ms	1 s	1 s	N/A

. These requirements are specified by the European Telecommunications Standards Institute (ETSI) and the International Electrotechnical Commission (IEC). Communication response time requirements vary, with the IEC defining a 4 ms maximum threshold for protection messages, whereas ETSI specifies a 10 ms threshold. Both standards align with the control message requirement of 10 ms. We illustrate the changes in the mean and standard deviation of RTTs across different relative congestion levels in Figure 9 and Figure 10. Notably, the highest mean and standard deviation of RTTs occur when the communication channel is approximately 50% congested. When TCP is used to deliver VS packets, it leads to higher mean and standard deviation of RTTs compared to other protocols, such as UDP, RTSP, and ICMP, as shown in Figure 11 and Figure 12. This suggests that high-variance bit rate in background traffic results in higher mean and standard deviation of RTTs.

We analyse the relationship between the mean of RTTs and the standard deviation of RTTs by computing Pearson’s correlation coefficient. A scatter plot in Figure 13 presents the relationship between the mean and standard deviation of RTTs, where the correlation coefficient is computed as $0.89$. This suggests that there is a strong positive linear relationship. Further analysis in Figure 14 reveals a Pearson’s correlation of $0.84$ between the mean and standard deviation of RTTs that exceed the ETSI requirement. This strong positive linear relationship persists when the industrial communication response time requirements are considered. As a result, RTTs exceed industrial requirements by more than 25 times for NMG control data. Greater mean and standard deviation of RTTs are measured when high-variance background traffic is present. These findings support our hypothesis that high-variance background traffic negatively affects the QoD of NMG control data.

6. Future Work and Open Challenges

A key result of this paper is that the mean RTT of NMG control data exceeds the industrial communication response time requirements by a factor of 25 in the presence of VS. Maintaining the RTTs of NMG control data at tolerable levels is crucial to prevent system-wide stability and performance issues [70] for operational functions. As future work, machine learning (ML) prediction algorithms will be used to forecast future RTTs to ensure that QoD remains within industrial latency requirements.

Various techniques exist for predicting RTTs in communication networks [71]. Recent papers explore ML algorithms for the future states of QoD metrics. In [72], an ML ensemble prediction model is employed for RTT prediction in 4G and 5G networks. The experiments demonstrate that the developed ensemble prediction model effectively predicts future RTTs. The authors of [73] implement a recurrent neural network (RNN) model that forecasts future RTTs in a communication network. They demonstrate that the implemented model outperforms traditional RTT prediction techniques. In [74], various ML models are employed for RTT prediction in cellular networks. The authors conduct two experimental scenarios where a smartphone is connected to the real cellular network in a suburban area in Italy. The results show that the random forest (RF) and decision tree (DT) prediction models achieve precision scores of $0.84$ and $0.92$ for the two experimental scenarios, respectively.

Several ML techniques have been implemented to predict other QoD metrics in congested communication networks. A packet loss predictor, using two XGboost ML models, is developed for the TCP Reno and the cubic congestion control algorithms in [75]. The mininet network emulator is used to represent experimental scenarios where the background traffic is present. The results demonstrate successful packet loss prediction, and explicit congestion notification (ECN) is used to regulate sender packet rates before packet loss occurs. As only two congestion control algorithms are considered; the performance of models remains untested with other control algorithms. The authors of [76] address the limitation of autoregressive filters in jitter prediction for QoD measurement of VS. To improve QoD measurement in VS services, they introduce a novel jitter estimator that improves ML classification algorithms’ learning capabilities across different network congestion levels. By using off-the-shelf ML, RF, and DT, their approach achieves classification accuracy rates of $95.1\%$ for high, $45.2\%$ for medium, and $95.1\%$ for low congestion levels. In [77], an RNN-based model is developed to predict jitter in a congested communication network. The novel jitter prediction algorithm, called ASAPjitter, is implemented to predict gradient changes in a jitter measurement of VS. These predictions are classified by a convolutional neural network (CNN). The jitter classification algorithm, FEATjitter, is implemented to convert jitter time series measurements into the high-dimensional feature space. This conversion aims to enhance the accuracy of predicting and classifying future jitter measurement for VS. The results show that an accuracy of $84.5\%$ is obtained for ASAPjitter and $91.6\%$ for FEATjitter in total.

The challenges associated with ML prediction algorithms need to be addressed. NMGs are characterised by complex components and numerous variables. This complexity presents a challenge for ML prediction algorithms in accurately learning and representing the relationships among these variables [78]. Moreover, ML prediction algorithms require adequate computational resources, including large memory capacity and processing power, when trained on large-scale datasets [79]. These pose a challenge for the scalability of ML prediction algorithms in the context of NMGs.

We assume that greater RTTs will occur when NMGs are deployed in a WAN, where the variance of background traffic is high. One of the open challenges is the need to examine RTTs of NMG control data in a WAN. We anticipate that the QoD of NMG control data would degrade if the experimental scenarios run for longer durations, as the impact of high-variance background traffic increases on the communication network.

On the other hand, NMGs are dependent upon communication networks to operate as efficient, smart, and robust small-scale power systems. However, this reliance transforms NMGs into a more complex, bidirectionally peer-to-peer (P2P) power system, making them vulnerable to cyber attacks [80]. These cyber attacks pose various threats and negatively impact various parts of NMGs. For instance, aurora attacks can damage physical NMG components, such as generators, and lead to degraded operational performance [81]. Despite the declining costs of lithium-ion batteries in recent years, they remain an upfront expense and are considered an ongoing challenge in NMGs [82]. Environmental concerns arise regarding the disposal of these storage components [83]. Emerging storage technologies, such as flow batteries, thermal storage, ${\mathrm{CO}}_2$-based storage, and other long-duration energy storage (LDES) products, are gaining popularity as cost-effective and environmentally friendly solutions [84,85]. As an emerging technology, NMGs are constrained by regulatory limitations. This regulatory gap presents a significant challenge, as NMGs directly facilitate DERs into the main grid. The absence of these regulations prevents the effective usage of NMGs [86].

7. Conclusions

In this paper, we addressed the communication network types and communication technologies used in NMGs. We evaluated papers that address QoD measurement of NMG control data. These papers measured one-way delay using either simulation tools or real-life testbeds. There are two main points that limit the realism of QoD measurement in current papers. First, NMG are designed to transfer control data in bidirectionally so that the RTT measurement yields a better understanding of data exchange between NMG endpoints. Second, simulation tools provide a partial representation of real-life scenarios in NMGs.

To further investigate the effect of background traffic on the QoD of NMG control data, this paper addressed the RTT measurements in NMGs using a real hardware testbed. We injected various levels of background traffic into the communication network. In summary, we concluded that high-variance in background traffic, caused by TCP-based VS, led to the RTTs of NMG control data surpassing industrial requirements by more than 25 times.

The results of this paper have significant consequences for sustainable development in energy systems. By analysing the impact of various levels of background traffic on QoD of NMG control data, this paper highlights the need for optimised communication networks in NMGs. The empirical evaluation using the real hardware NMG testbed presents tangible evidence supporting the implementation of robust communication networks that ensure reliable communication between NMG endpoints. Enhanced QoD in NMGs enables the effective integration of renewables, which is crucial for achieving net-zero carbon targets. As renewable integration in NMGs increases, optimised communication networks become 419 essential to facilitate key functionalities such as effective power-sharing, improved NMG controller performance, and optimal NMG operations. These improvements strengthen the role of NMGs within sustainable energy systems, supporting both environmental sus-tainability and economic development through more reliable and resilient power systems.