Toward Wireless Smart Grid Communications: An Evaluation of Protocol Latencies in an Open-Source 5G Testbed

Abstract

Fifth-generation networks promise wide availability of wireless communication with inherent security features. The 5G standards also outline access for different applications requiring low latency, machine-to-machine communication, or mobile broadband. These networks can be advantageous to numerous applications that require widespread and diverse communications. One such application is found in smart grids. Smart grid networks, and Operational Technology (OT) networks in general, utilize a variety of communication protocols for low-latency control, data monitoring, and reporting at every level. Transitioning these network communications from wired Wide Area Networks (WANs) to wireless communication through 5G can provide additional benefits to their security and network configurability. However, introducing these wireless capabilities may also result in a degradation of network latency. In this paper, we propose utilizing 5G for smart grid communications, and we evaluate the latency impacts of encapsulating GOOSE, Modbus, and DNP3 for transmission over a 5G network. The OpenAirInterface open-source library is utilized to deploy an in-lab 5G Core Network and gNB for testing with off-the-shelf User Equipment (UE). This creates an effective 5G test platform for experimenting with different OT protocols such as GOOSE. The results are validated by measuring two different Intelligent Electronic Devices’ contact closure times for each network configuration. These tests are also conducted for varying packet sizes in order to isolate different sources of network latency. Our study outlines the latency impact of communication over 5G for time-critical and non-critical applications regarding their transition toward private 5G-based OT network implementations. The conducted experiments illustrate that in the case of GOOSE packets, simple encapsulation may exceed the protocol’s time-critical nature, and, therefore, additional measures must be taken to ensure a viable transition of GOOSE to 5G services. However, non-critical applications are shown to be viable for migration to 5G.

1. Introduction

Smart grid communication protocols have a wide range of applications to serve the growing complexity of modern electrical grids. As Distributed Energy Resources (DERs) become more commonplace, grid complexity and the associated network complexity for monitoring and control increase in order to meet the increase in demand and ensure the safe operation of grid resources. Some commonly used non-proprietary protocols for Supervisory Control and Data Acquisition (SCADA) and substation communication in North America are DNP3, Modbus, and IEC-61850 [1]. The IEC-61850 protocol suite is designed for greater visibility into smart grid devices at the process, bay, and station levels. At its core, it consists of the Generic Object-Oriented Substation Event (GOOSE), Sampled Value (SV), and Manufacturing Message Specification (MMS) protocols [2]. Among these protocols, GOOSE is capable of both substation and SCADA communications. It utilizes an 802.1Q multi-casting feature that transmits unencrypted data at the station level, whereas Routable GOOSE (R-GOOSE) encapsulates GOOSE packets for delivery to SCADA systems. Modbus and DNP3 are often used for reporting and have higher latency tolerances. They utilize session-based TCP communications and support TLS for security [3,4].

The use of wireless networks for smart grid operations can ease the support of communications for DERs or different grid components over large geographical distances and make networks easier to re-configure without changing the grid infrastructure or the cost associated with installing large wired networks. As one prominent example of wireless technologies of interest for such wireless transitions, 5G also contains policy enforcement capabilities for security, which can be leveraged to transition toward a secure zero-trust architecture [5]. The capabilities of 5G can, in part, be attributed to advancements in antenna design, providing higher gain with smaller footprints [6]. These advancements allow for a higher density of connected devices on wireless networks compared to previous cellular generations [7]. The added security of 5G can also mitigate vulnerabilities for protocols such as GOOSE [8], which multi-cast plain text messages over their networks. Full implementation of the recommended security features of 5G would also mitigate vulnerabilities found in the Modbus and DNP3 protocols, which have been shown to be susceptible to Man-in-the-Middle (MiTM) attacks and False Data Injection Attacks (FDIAs) [9,10].

In this paper, we evaluate the latency impact of incorporating 5G communication for OT protocols (both with and without low-latency requirements) commonly used in substation communications. Specifically, we examine the latency impact on the communications of IEC-61850 GOOSE, Modbus, and DNP3, considering the latency both in a standard deployment and when communicating over 5G. For the testbed in our study, we utilize the implementation provided by the OpenAirInterface (OAI) open-source project [11,12], which allows us to establish a full 5G core and Radio Access Network (RAN) to represent, for example, a private 5G network for substation communications. We also perform in-depth latency measurements for each component of the 5G testbed, isolating the factors critical to our measured latency. We utilize these obtained results to investigate the feasibility of transitioning these protocols directly to 5G operations. Our research results presented in this paper show that the latency introduced through simple encapsulation of OT protocols using 5G as the transport mechanism is suitable for non-critical smart grid protocols but not for time-sensitive protocols like IEC-61850 GOOSE. Our study shows that a further reduction in latency is required to accommodate time-critical smart grid applications. The remainder of the paper is structured as follows. Section 2 reviews the publications related to the research area, whereas Section 3 outlines our research methodology, including the testbed setup, encapsulation procedure, and latency measurements. Section 4 presents the results we obtained from the testbed, including the end-to-end latency and a breakdown of the individual factors contributing to latency. Section 5 outlines our future work, and Section 6 concludes this paper.

2. Related Works

Numerous works have been published in the scientific literature investigating the advantages of using 5G for smart grid communications. Examples include the works in [13,14,15]. Improving communications and application performance with Multi-access Edge Computing (MEC) provides the advantage of more reliable systems with higher fault tolerance and better end-user experience. Such efforts focus particularly on protocols with low-latency requirements, such as IEC-61850 GOOSE. The authors of [16] encapsulated GOOSE packets for transmission over a virtualized 5G core in a Non-Stand-Alone (NSA) configuration and reported the simulated delay of a virtual network. Latency measurements were also reported for R-GOOSE in [17], which utilized a testbed with both 4G and 5G subscriptions through a Finnish cellular service provider. The whole suite of protocols in IEC-61850 was encapsulated into UDP and TCP packets in [18], showing promising results for the latency of these protocols operating over cellular services. The GOOSE protocol’s performance has been measured in a Hardware-In-the-Loop (HIL) simulator, targeting a physical IED’s response time to virtual IED faults over a simulated 5G network [19]. The authors of [20] investigated the capabilities and constraints encountered when utilizing the IEC-61850 protocol suite for protection systems communicating over 5G. All of these results show promise for GOOSE communications between substations over wide geographical areas. However, none of these reviews investigated the specific latency effects of replacing standard GOOSE communication within a substation with a 5G wireless approach while maintaining GOOSE’s strict latency requirements.

Modbus has also been studied over a 5G network for additional industrial applications. For example, the authors of [21] utilized a Raspberry Pi as a protocol gateway for a TCP-to-5G conversion. Additionally, 5G has been utilized for other smart grid applications. The authors of [22] proposed using an IOT gateway that communicates with a controller server utilizing 5G and translates decisions made by the server into Modbus protocol messages. In [23], the authors utilized a 5G network to communicate with a server external to a smart grid, which uses a neural network to detect attacks on the Modbus TCP/IP protocol. However, less research has been directed toward the transition of the DNP3 protocol. In [24], the authors suggested a neural network for “artifact for security in 5G”, which improves the security of common attacks in multiple OT protocols. The authors of [25] provided an overview of the potential challenges in moving critical infrastructure communication to 5G, including for DNP3. These reviews investigated improving the security of Modbus and DNP3 without investigating the operational impact of using 5G as a transportation method.

3. Materials and Methods

The initial testbed utilized for our protocol evaluations was outlined in our previous work [26]. This testbed, which is also shown in Figure 1, is logically divided into three different sections: the test controller, 5G network, and Device Under Test (DUT). Two Intelligent Electronic Devices (IEDs) from two reputable manufacturers were utilized for our initial GOOSE tests. Each device was configured to close or open a contact in relation to a specific boolean value within a GOOSE packet. The test controller operates on a Raspberry Pi 4 single-board computer (SBC), allowing access to the physical measurements on General-Purpose Input/Output (GPIO) pins. Packet generation for this step was completed with the Libiec61850 [27] software library, which handles all scheduling and updating of packets. For the Modbus and DNP3 protocols, Scapy [28] was utilized with an additional user-contributed DNP3 library [29]. The User Equipment (UE) is described in detail in the subsection below. However, we are refraining from publicly disclosing the precise devices due to security considerations related to ongoing efforts.

3.1. 5G Network

This approach and the testing of smart grid protocols over 5G was implemented using a local, in-lab 5G testbed setup located at the Advanced Telecommunication Engineering Laboratory (TEL) at the University of Nebraska-Lincoln. The testbed architecture was designed and developed to accurately represent regular private 5G network deployments, incorporating a 5G Core Network (CN), gNodeB (gNB), and User Equipment (UE). OAI was utilized as the software foundation for establishing and operating this network, allowing for software control of configurations and hardware flexibility in the Radio Access Network (RAN) operation.

OAI provides full support for operating the UE, gNB, and CN, all of which are provided as an open-source solution. Furthermore, all components and features are compliant with the 3GPP standard. The provided RAN currently complies with Release 15 [30], and the CN complies with Release 16 [31], with development continuing for this project. The CN’s Virtualized Network Functions (VNFs) are implemented as individual Docker images hosted on top of a Ubuntu 22.04 LTS Linux server with the updated low-latency kernel installed. The gNB is implemented using software radio principles using GNURadio v3.10.7.0, with the signal processing aspects performed by a bare-metal (non-Dockerized) implementation on a server with operating system and kernel specifications identical to those of the CN. The radio frontend utilized is an Ettus Research B210 Universal Software Radio Peripheral (USRP) [32]. This USRP is configured by the OAI software v1.5.1 to operate on band n78 in Time Division Duplex (TDD) mode at a center frequency of 3.6192 GHz and a 40 MHz wide channel. The network architecture of the 5G network is shown in Figure 2. Communication between the USRP and its host machine is facilitated through USB 3.0 to allow for high-bandwidth communications. The operations of the USRP are controlled via the interface between the OAI software and Ettus’ UHD (USRP Hardware Drivers) v4.4.0.

In our testbed, the UE functionality is provided by a Raspberry Pi 4 with Ubuntu 22.04 LTS installed, along with a Quectel RM520N-GL 5G HAT Modem, as shown in Figure 3. Quectel provides proprietary firmware and Linux drivers in order to seamlessly integrate the radio modem with the desktop environment as a cellular interface. Like the USRP, the modem is connected to the Raspberry Pi via USB 3.0. As the UE is the only component facilitating communication between the 5G network and the DUT, the UE runs additional software to forward any applicable OT packets from the test controller. In the case of the GOOSE protocol, encapsulated packets are extracted and forwarded onto the switch for the DUT. With Modbus and DNP3, we incorporate the UE as the DUT for the packets since the UE is not required to extract and forward any payload.

3.2. Low-Latency Protocol Testing—GOOSE

The initial latency tests run with our testbed are outlined in [26]. These tests take initial GOOSE network latency measurements against two separate IEDs, a protection system and a general-purpose relay, and compare these latency measurements against those obtained using encapsulated R-GOOSE packets. In our current 5G communications tests, these R-GOOSE packets were additionally prepared and transported over 5G. Since these tests were performed against IEDs that only support GOOSE and not R-GOOSE, our evaluation additionally considered and measured the latency impact of encapsulation and extraction. Since a number of such factors contribute to the overall end-to-end latency, a total of 1000 cycles were completed for each test to ensure statistical reliability for our results.

Latency measurements create a log-normal distribution, causing the mean value to be skewed by including samples from the upper sample range. For this reason, we collected a sufficiently large number of samples to be statistically valid and applied the Central Limit Theorem. This theorem states that the mean will be normally distributed for high sample counts. In this study, our target error margin was $\pm 3\%$ with a 95% confidence interval. To calculate the required number of samples, the equation for normal distributions shown in Equation (1) was utilized. Here, p is the population proportion, z is the z-score of a normal distribution to achieve a 95% confidence interval (CI), and E is the targeted error margin. Since we operate without a known population standard deviation, we utilized $p(1-p)$ instead of our population’s variance ${\sigma }_P^2$.

$$n={\sigma }_P^2{(\frac{z}{E})}^2\approx p(1-p){(\frac{z}{E})}^2$$

(1)

This equation is derived from the standard error equation for a normal distribution, as shown in Equation (2). Since we have an undefined population, we implement $0.5$ to give the largest variance for the population. Given a desired confidence interval of 95%, we use the standard normal 1.96 for the z-score. Since 1068 samples would be required for a 3% error margin, we calculated that 1000 samples, as shown in Equation (2), would be sufficient for a 3.1% error margin, which is sufficient for our experiments.

$$E=z·\sqrt{\frac{{\sigma }_P^2}{n}}\approx z·\sqrt{\frac{p(1-p)}{n}}$$

(2)

3.3. Latency Evaluation

We evaluated several test cases to comprehensively study the impact on communication latency introduced by 5G. These scenarios were selected to provide a deeper understanding of the contributions to the overall latency resulting from each processing step within the end-to-end communications process. Specifically, the steps taken to transmit a GOOSE protocol packet over 5G include the encapsulation of the GOOSE packet within TCP, the 5G network processing, sending the packet over 5G using the GPRS Tunneling Protocol (GTP), and finally, the UE extracting and forwarding the GOOSE packet over Ethernet. A timing diagram of this process can be seen in Figure 4. To isolate the effects of each network component on the latency of the overall system in the absence of 5G, we also conducted additional tests for direct TCP communication between both sides, referred to as the “TCP” test scenario, as well as TCP communication with an additional hop to the Core Network workstation, which we call “TCP 2-Hop” in this paper. For each scenario outlined, a GPIO on the test controller was configured to read the state of an IED contact. The contact was configured to reflect the value of a boolean variable within the GOOSE dataset. Each of these network configurations is explained in more detail in the following paragraphs.

L2-Eth—This network configuration measures the delay of the standard GOOSE network. GOOSE operates by sending Ethernet frames directly over a wired link. In this configuration, the test controller is connected to the same managed switch as the DUT. GOOSE packets are generated by the test controller and multi-cast to a pre-defined MAC address of the DUT subscription. This scenario is used as a baseline for all tests moving forward and outlines the difference between the subsequent tests and the IED’s baseline response time. These results are also used to characterize the stability of the measured 5G network latency and to detect and contrast the occurrence of outliers within a standard GOOSE configuration compared to the 5G-encapsulated packets.

TCP—The network configuration for both the TCP and TCP 2-Hop configurations can be seen in Figure 5. In this configuration, the GOOSE Publisher creates the same GOOSE packet that would occur in the L2-Eth scenario. However, after the packet is generated, it is encapsulated within a TCP/IP frame structure, similar to the procedure required for R-GOOSE communications. The GOOSE publisher establishes a TCP session with the UE before sending any encapsulated GOOSE packets. Since GOOSE packets are scheduled at a rate not exceeding once per second throughout all of our tests, the same TCP session is used for all packets. In this configuration, the gNB machine is removed from the network setup, and instead, the UE is directly connected to the switch, thereby providing a direct TCP connection from the publisher to the UE for the extraction and publishing of the GOOSE packet. This is illustrated in Figure 6.

TCP 2-Hop— In the TCP 2-Hop configuration, an additional routing step is added at the Core Network workstation. This configuration is utilized to identify any latency contribution that is introduced through the additional routing step required to communicate between the GOOSE Publisher and gNB in cases where the 5G core tunnels traffic using GTP. This test generates packets at the same publisher, but it first sends the TCP packet to the Core Network instead, which is then tasked with forwarding the packet to the UE through a second TCP session, representing the GTP tunnel as a second network hop.

5G—With the initial 5G test configuration, a standard TCP session over 5G eMBB is utilized. The test controller establishes a regular TCP session with the UE that traverses through the 5G network by virtue of a GTP tunnel within the 5G realm of this network configuration. For this to be successful, routes are manually added to each device’s IP table to indicate the UE sessions that must be routed through the Core Network. Each packet is thereby routed from the GOOSE Publisher to the Core Network. The Core Network then further encapsulates the TCP packet into the GTP packet for tunneling it to the gNB. The gNB base station handles all scheduling of packets to the UE over the wireless interface. As the final step in this network scenario, the UE needs to extract all received TCP packets from GTP, then further extract the encapsulated GOOSE packet, and finally publish it over the wired Ethernet interface so that it can reach the DUT. Compared to the TCP 2-Hop configuration, the GTP encapsulation and gNB scheduling result in additional latencies that are combined and labeled as 5G-specific latency, as shown in Figure 4.

5G-TLS—This configuration utilizes the same topology and routing as the previous 5G scenario but adds TLS encryption through the OpenSSL library. This scenario is used to measure the latency effect of adding security features to the GOOSE communication rather than sending unencrypted data over the network. Similar to the 5G TCP configuration, the session is opened before GOOSE traffic is sent, so the TLS handshake is not re-occurring for each individual packet transmission. However, since encryption can be a computationally costly operation, this step aims to quantify the latency impact of this processing.

Latency Isolation—For further isolation of the variance within the packet latency, separate network taps were placed on the Core Network, gNB, and OT device to compare transmission times. Each tap was connected between the workstation and switch, with traffic captured in a single Wireshark [33] session. This method allows us to compare the recorded epoch time of each packet reception for timing measurements. For the timing diagram shown in Figure 4, the original GOOSE message identified as message generation, the GTP packet from the Core Network to gNB, and UE to OT device GOOSE packet are all captured. We then compared the original GOOSE packet, the GTP packet, and the forwarded GOOSE packets for timing comparison, as shown in Figure 7.

3.4. Session-Based Protocols—Modbus and DNP3

In contrast to the GOOSE protocol, Modbus and DNP3 do not have low-latency requirements for their respective protocol transmissions. These protocols utilize a simple request/response model, with packets often scheduled through the internal logic of an IED or external signaling. The utilized network configuration for these protocols is the same as that of the 5G TCP test case described above. Instead of a GOOSE publisher, a test controller is implemented through the Scapy library [28]. In contrast to the previous tests against production IEDs, the 5G UE for these tests directly acts as an endpoint for these protocols. The UE utilizes the same Scapy library and responds to each Modbus and DNP3 request with a valid response.

Modbus—The Modbus protocol was initially created for serial wire communication in a bus configuration. The protocol is frequently used to interact with Programmable Logic Controllers (PLCs) or for communication between a SCADA system and Remote Terminal Units (RTUs). It commonly utilizes RS-485 for signaling, and the messaging exchange follows a request/response model. The “Client” device is the requesting device for all communication, while a “Server” device generates a response packet to the corresponding request. The packet structure contains a 1-byte server address, function code, relevant data, and the CRC error-check field for a request or response. Although packets can be as small as 5 bytes, the protocol limits the size of a packet to 256 bytes [3].

As Modbus transitioned to TCP communication, the error-check field was removed and a Modbus Application (MBAP) header was introduced. The header contains the server address and function code, but the packet adds a transaction identification field for devices to identify individual request packets and their responses. Containing the same number of 253 available data bytes, the largest Modbus packet in TCP communication is 260 bytes.

DNP3—Similar to Modbus’s request/response structure, DNP3 utilizes a “Master” for generating requests, with an “Outstation” creating the corresponding response. DNP3 is commonly used for communication between SCADA systems and a variety of components, including PLCs, RTUs, and IEDs. DNP3 allows for usage over a serial bus, such as RS-485, and TCP sessions over Ethernet, without changing the message structure. This is achieved through message fragmentation, as no DNP3 packet will ever exceed 292 bytes [4]. DNP3 utilizes link, transport, and application layer headers to determine message priority, including source and destination addresses, and for packet reassembly requirements. Given the ability to reassemble packets, more data can be read from a single request than is possible in Modbus.

DNP3 also allows flexibility in data representation with defined object-oriented data types. Designed as a polling protocol, it enables scheduled reports to be requested at set intervals. However, the DNP3 protocol offers several levels of conformance [4], with higher levels allowing for the optional use of unsolicited messages from an “Outstation” to the “Master”. For the purposes of this paper, we configure each DNP3 endpoint to refuse unsolicited requests.

Latency Measurements—For latency measurements in each test case, a network tap was placed on the Core Network workstation and connected to the UE. On the UE device, Wireshark was utilized to capture packets on both the Core Network interface and the UE’s 5G interface. This allowed for time-stamped measurements as the OT protocol packets arrived on the Core Network, during GTP packet forwarding to the gNB, and when OT protocol packets arrived at the UE.

4. Testbed Results

The results of the testbed are separated into the GOOSE results and the results for the Modbus and DNP3 protocols. With the GOOSE protocol, all five network configuration results are compared, contrasting the latency of LAN configurations with the 5G configurations. Optimization is then performed on the 5G components of the testbed to evaluate these changes for any achievable improvements to the testbed’s latency. For Modbus and DNP3, initial tests are performed, and the results are compared to the GOOSE results. Given the similarity of results between each protocol, different packet sizes were also tested and evaluated in order to isolate the cause of latency related to gNB wireless scheduling.

4.1. GOOSE Results

The results we obtained from conducting all initial latency tests, as described in Section 3.3, are shown in

Table 1. Mean latency measurements (ms).

		L2-Eth	TCP	TCP 2-Hop	5G	5G-TLS
Dev 1	Latency	14.152	15.316	15.797	19.214	19.056
	CI 95 % Spread ±	0.171	0.391	1.476	0.205	0.226
	Difference from L2-Eth	-	1.164	1.645	5.062	4.904
Dev 2	Latency	5.797	6.415	6.478	10.267	10.325
	CI 95 Spread % ±	0.082	0.302	0.299	0.105	0.110
	Difference from L2-Eth	-	0.618	0.681	4.470	4.528

. For each of these tests, 1000 cycles were executed following the defined transportation topology and method. We expected these tests to yield similar values for the “Difference to L2-Eth” row between the two devices, as this row demonstrates the added latency compared to a standard GOOSE network’s latency. This would indicate a relative independence of device implementation regarding the observed behavior. We also expected the latency difference for each test to fall within both devices’ calculated 95% confidence intervals, indicating a high confidence that the mean latency is within the given range of values. We hypothesized that our tests would demonstrate a small increase in latency for each routing step added across the different test scenarios, with a comparatively larger increase in latency resulting from the introduction of the 5G network. While our tests conclusively confirmed this hypothesis, we also observed a notable difference in the average response time between the different devices.

From these results, we can observe that the discrepancy in the difference rows can be explained by the 95% confidence intervals of each individual device. Device 1 exhibited 1.164 ms added latency for the TCP test, whereas Device 2 exhibited 0.618 ms. These results may come across as inconsistent latency across devices. However, the confidence intervals show Device 1’s ± 0.391 ms and Device 2’s ± 0.302 ms, giving a full range of 0.693 ms. This implies that our dataset contained a large amount of noise or outliers, which was investigated through network optimization. The main point of interest in this table is the noticeable increase in latency from utilizing TCP routing and also from adding 5G communication. These results are shown graphically in Figure 8. Although there was added latency resulting from TCP, as shown in the TCP and TCP 2-Hop scenarios, there was a far greater impact on the latency resulting from 5G. When comparing the scenarios with and without 5G, we observed a 3.898 ms latency increase for Device 1 and a 3.852 ms latency increase for Device 2.

The relatively large confidence interval spread is likely a result of the IED-specific behavior for message processing and contact closing. As shown in Figure 8, the 5G and 5G-TLS scenarios exhibited similar latency increases for both devices when compared to the configurations not utilizing 5G. In these results, an interesting point to note is the lower latency exhibited by Device 1 in the 5G-TLS scenario compared to the 5G-TCP scenario. We attribute this to the wider range of response times for Device 1 compared to Device 2 and the existence of outliers in our data. Given the confidence interval spread for both devices’ 5G communication, we conclude that the latency difference between 5G TCP and 5G TLS is smaller than our confidence interval’s resolution.

This is further illustrated in Figure 9, where a comparison of the results for each configuration is shown separately for both IEDs. In this figure, it can be seen that each device maintained a relatively stable distribution of the measured response times across all tested configurations. Overall, Device 1 exhibited a wider range of response times, with an average response time of just below 15 ms for all non-5G scenarios, but it also exhibited regular responses timed at close to 20 ms. In contrast to these results, Device 2 achieved a response time closer to 6ms, with nearly all samples concentrated below 10 ms, even in the TCP configurations. However, these differences in response variance also remained stable across the 5G results, as the measured delay deltas ${\delta }_1$ and ${\delta }_2$, which represent the difference between the median non-5G and median 5G scenario, had similar values. In these tests, ${\delta }_1=3.945$ ms and ${\delta }_2=3.934$ ms. One particularly interesting result from this test is the addition of outliers for Device 2 and the increase in outliers for Device 1. This implies that although the network configurations closely maintained overall device performance, even over 5G, adding 5G increased the possibility for outliers in end-to-end latency. This was observed for both devices.

To explain the introduction of outliers in these results, the latency isolation outlined in the previous section was measured, as shown in Figure 10. In this figure, the GTP encapsulation step includes the duration from when a GOOSE packet is sent to the Core Network to the GTP-encapsulated packet being forwarded to the gNB. The GTP-to-GOOSE extraction measures the timing from the GTP packet sent to the gNB to the final forwarding of the extracted GOOSE Ethernet frame to the OT device, whereas the end-to-end latency is the sum of these two values. From these results, we can observe that the latency of each step has a large number of outliers, with GTP encapsulation ranging from below 1 ms to over 10 ms, and GTP-to-GOOSE extraction averaging 5 ms but also repeatedly exceeding 10 ms. From these results, we can see that not all packets were guaranteed to be transmitted within the required latency for GOOSE messaging within a substation. However, we observed that the latencies were sufficient for less stringent R-GOOSE applications in OT WAN applications [17].

The results shown in Figure 9, Figure 10 and Figure 11 consistently show outliers, pointing to the fact that it may not be possible to guarantee latencies to be within a well-defined range, which, in turn, impacts the viability of directly supporting time-sensitive networking. Figure 9 outlines the difference in processing between Device 1 and Device 2, which we will address in our future research by studying the device’s internal protocol parsing and message handling operations. The separate message preparation steps, as shown in Figure 10, display a large latency spread for the GTP-to-GOOSE extraction and the GOOSE end-to-end latency measurements, which indicates the possibility that 5G resource-block allocations, frame timing, and other air-interface considerations negatively impact the observed performance and introduce non-deterministic considerations that contribute to this large spread, such as the possibility for retransmissions due to bit errors. Additionally, we can also observe a large number of outliers in this figure. To reduce the number of outliers stemming from 5G network operations, we further optimized the configuration of Ubuntu’s low-latency kernel on both the gNB and Core Network workstations. The results from this test can be seen in Figure 11. While there are still a number of outliers within this latency test, no outliers exceed 10 ms latency. We attribute the latency spread and remaining outliers to scheduling within the OT devices and their protocol processing, within the 5G network, and other 5G-related non-deterministic factors not covered by our performed optimizations. Therefore, we aim to study these issues further in our future research, including the investigation of additional optimization strategies and their benefits to latency and the observed end-to-end latency jitter.

4.2. Modbus and DNP3

The testing for Modbus and DNP3 followed the same test pattern as the GOOSE protocol’s 5G TCP configuration. Each protocol was tested for 1000 cycles for statistical reliability, with the test coordinator generating packets to be encapsulated by the Core Network, forwarded to the gNB in GTP format, and transmitted over the wireless interface for UE reception. The same requesting packet was generated for each protocol: a Modbus “Write Registers” packet with a 78-byte payload and a DNP3 “Direct Operate” packet with an 87-byte payload. These packets were chosen due to their small packet size for the protocol. The “Write Registers" packet for Modbus contains a single address, a size byte, and 2 data bytes to be written, whereas the “Direct Operate” packet for DNP3 contains a single 16-bit item payload. Both packets represent the smallest supported packets according to each protocol’s TCP specification. The results shown in Figure 12 show that each protocol achieved similar results, but DNP3 exhibited a slightly lower latency compared to Modbus/TCP and GOOSE. Due to this discrepancy, additional testing was conducted for multiple packet sizes for both Modbus and DNP3 packets. Within the Modbus and DNP3 standards, individual packet sizes are defined for TCP communications. For Modbus, the maximum size packet is 260 bytes [3], whereas DNP3 allows for 292 bytes [4]. For testing the impact of packet size on latency, individual packets with lengths of 103, 145, 221, and 255 bytes were tested in the same manner as the smaller packet sizes used in the initial testing.

The results of these packet size tests, once again comparing the measured latency, can be seen in Figure 13. The results in this figure indicate that the latency did not correlate with the limited packet size of Modbus and the DNP3 protocol. Since each packet exhibited a similar average latency and similar latency distribution, we can conclude that there was no linear relation between the packet size and the measured latency. Instead, we posit that packets adhering to the Modbus and DNP3 standards would always exhibit a similar latency for delivery. This implies that the gNB scheduling of wireless packets is the main constraint on packet latency, and packet scheduling for these studied protocols is expected to fall within a similar latency range for similar network topologies and technologies.

5. Discussion

From the results presented in this paper, showing a median latency of 10–20 ms observed across the tested devices and topologies, simply encapsulating GOOSE packets for transmission over a 5G network may not meet the protocol’s stringent substation latency requirements of less than 4 ms. However, the measured latencies are indeed sufficient for most other smart grid protocols, such as the tested Modbus and DNP3 protocols. In our future work focused on 5G networking for smart grid applications, the evaluation of different frequency bands and channel widths will be incorporated as additional avenues of exploration, along with frame optimization, 5G resource scheduling, and overall timing in radio operations, to further reduce the achievable end-to-end latency. There are also promising solutions within recent 3GPP releases that can be evaluated for GOOSE and other smart grid protocols, namely Sidelink Relaying [31] and Sidelink Enhancement [34]. Rather than requiring the utilization of a gNB, sidelink communication is often used in V2X for smart cities. This technique could be repurposed for use in smart grids, where lower latency communication is desired within the physical substation. The standard TCP over 5G communications outlined in this paper can be used in conjunction with sending traffic to a SCADA system or different substations. We plan to conduct further research into additional mechanisms to reduce the latency impact discussed in this paper by leveraging new and existing 5G mechanisms and parameter optimizations utilized within URLLC while expanding the supported packet length to achieve support for transporting OT protocol payloads.

We also aim to further expand the scope and capabilities of our testbed, specifically in terms of scale, and embrace a zero-trust architecture. To address the scale of the testbed, incorporating additional base stations and UE would allow for further testing of fairness and latency evaluation within a congested network. This platform would also allow for modifying existing Core Network functions and adding new VNFs, which would be leveraged for realizing a zero-trust architecture [5]. Our research will also focus on identifying and resolving any challenges in applying our findings to real-world implementations of 5G for smart grid OT equipment. For example, we aim to explore the impact of using 5G gateways, rather than directly communicating over 5G for grid communications, and a viable migration path for already deployed smart grid systems toward 5G for an aging grid infrastructure [35,36].

6. Conclusions

As 5G network deployments are growing in popularity, utilizing reliable wireless communication for smart grid operations increases data visibility and network configurability with limited additional infrastructure investment. Utilizing an open-source package, OpenAirInterface, we tested the response latency of two production IEDs by comparing the end-to-end latency of GOOSE, DNP3, and Modbus packets sent over a standard wired Ethernet setup to the latency achieved over a 5G network. Here, the end-to-end latency was measured for different test cases consisting of TCP-encapsulated packets, TCP encapsulation with multiple hops, and TLS encryption, both with and without a 5G communication session. Although the results presented in this paper indicate that in its current form, we would fail to meet GOOSE’s strict latency requirements, they nevertheless validate the merit of this approach for OT WAN monitoring applications and other applications using Modbus or DNP3, where latency is of lower criticality compared to factors such as ease of centralized monitoring or secure communications. This was confirmed by evaluating their end-to-end latency. Furthermore, our results indicate the potential to meet GOOSE’s latency requirements through optimization of the network segments and operations, which is the focus of our ongoing efforts.