Development of Traffic Scheduling Based on TSN in Smart Substation Devices

Abstract

Smart substations are an important trend in substation construction. With increasing data traffic, it is difficult for the traditional Ethernet network to meet the real-time requirements of control information in smart substations. Hence, in this paper, a deterministic network architecture for substations based on time-sensitive networks (TSN) has been developed in order to realize the domain-wide time synchronization and efficient real-time communication of the “three-layer and two-network” model in smart substations. Furthermore, a design scheme for substation automation equipment based on TSN is proposed. The proposed device realizes the timely transmission of real-time control information packets by utilizing the Earliest TxTime First (ETF) Qdisc technology of Linux and the timing sending capability of Intel 210 NIC. Furthermore, it collaborates with the time-aware shaper (TAS) traffic scheduling mechanism of TSN switches to ensure the end-to-end deterministic delay of time-sensitive traffic. As a result, it provides efficient real-time communication services with low latency and jitter for smart substation automation systems.

1. Introduction

The development of computer and network technology has promoted the development and application of smart technology. The communication architecture of a substation automation system has also evolved gradually with the development of computer technology and network communication technology [1,2]. At the beginning of the 21st century, the IEC 61850 communication protocol was applied to smart substation automation systems, and the “three-layer and two-network” object-oriented communication model, which contains complex IT/OT data traffic, was used for the digitalization, intelligence and communication standardization of substations [3,4]. Traditional Ethernet transmits data in a “best-effort” way, which can only reduce the end-to-end delay jitter by tens of milliseconds, and it is difficult to meet the communication service requirements of low latency, low jitter, and high reliability in smart substations [5,6]. Therefore, there is an urgent need to establish a new generation of networks that can provide “on-time and accurate” quality of service for data transmission [7,8].

Currently, the domestic and international technical solutions that can provide deterministic transmission mainly include Flexible Ethernet (FlexE), Time-Sensitive Networking (TSN), Deterministic Networking (DetNet) [9], Deterministic IP (DIP) technology, Deterministic WiFi (DetWiFi) [10,11], and 5G Deterministic Networking (5GDN) [12], etc. Among them, FlexE inserts an intermediate layer, FlexE Shim, between the physical layer and the link layer to decouple the service rate and the physical channel rate, providing a more flexible bandwidth granularity than traditional Ethernet and supporting the evolution of high-speed large-port 400GE, 1TE, etc. However, it is a little worse in terms of deterministic latency guarantee, and substations have not yet reached the bandwidth demands of 400GE and 1TE. However, in a smart substation environment, network requirements are more about ensuring the reliability and timeliness of critical control information (such as protection information) than simply bandwidth expansion. In addition, the deployment of FlexE may affect the interoperability of existing devices because not all devices can support the FlexE standard. DetNet and DIP both provide three-layer deterministic network transmission, and deterministic WiFi and 5GDN both focus on wireless transmission, which is difficult to deploy in substation automation systems that currently transmit data traffic through cables. The deterministic low-latency, low-jitter transmission capability that TSN provides to Layer 2 (the link layer) is the basis for the deterministic network technologies described above.

TSN can better address Layer 2 network deterministic guarantees than traditional Ethernet. It achieves transmission without congestion and packet loss through a set of protocol standards, provides low latency and jitter with upper bound guarantees, and provides deterministic transmission guarantees for latency-sensitive traffic. To achieve deterministic transmission in Layer 2 networks, TSN can achieve sub-microsecond time synchronization by IEEE 802.1 AS and the newer revision IEEE 802.1 AS-REV, and it can achieve priority scheduling of time-sensitive traffic by using traffic-shaping mechanisms such as IEEE 802.1 Qav (Credit-based Shaper, CBS), IEEE 802.1Qbv (Time-aware Shaper, TAS), 802.1Qch (Cyclic Queuing and Forwarding, CQF), IEEE 802.1Qbu (Frame Preemption), and IEEE 802.1Qcr (Asynchronous Traffic Shaping, ATS). In addition, IEEE 802.1 Qcc provides a centralized set of tools for globally managing and controlling the network, which can perform resource reservation, scheduling, and other configurations via remote management protocols (such as NETCONF or RESTCONF), enabling quality of service guarantees for time-sensitive data streams.

According to the network architecture of substation automation systems and the characteristics of power grid service traffic, a design scheme for smart substation Ethernet networking based on TSN characteristics has been proposed [13,14,15,16]. In the three-layer and two-network substation network architecture, traditional Ethernet switches are replaced with TSN switches, and Qbv and Qbu technologies are utilized to provide low-latency and low-jitter services for some time-sensitive flows. The application of TSN can not only ensure the safety of large smart substations in network congestion but can also increase the scalability of small network-scale substations, facilitating subsequent access to more equipment and services. However, if only the Qbv scheduling technique of TSN switches is available and the end-side devices do not have TSN characteristics and cannot collaborate with the TSN switches for traffic scheduling optimization, the upper bound of the end-to-end delay of the data streams will not be optimal.

In the three-layer two-network smart substation architecture that introduces TSN switches, this paper proposes a design scheme for a smart substation end-side device based on TSN technology. In terms of implementation, the device utilizes Linux’s Earliest TxTime First (ETF) Qdisc and the Intel 210 NIC to achieve the time-aware capability of hardware-timed message transmission, which, together with the Qbv technology of the TSN switch, is able to provide a bounded delay for time-sensitive streams and offers a new solution for Quality of Service (QoS) enhancement in smart substation automation systems. This scheme can not only optimize network resources, reduce redundant communication and improve network efficiency in large-scale substation networks but it can also be applied in small-scale substations to facilitate subsequent system expansion.

To summarize, the main contributions of the paper are as follows: (1) By using a TSN switch in smart substation networks, security problems arising from network congestion can be solved, and the scalability of small-scale network substations can be improved. (2) By using the ETF queue scheduling mechanism and NIC cards that support hardware timing, the proposed end-side devices can collaborate with TSN switches to provide bounded delay transmission for time-sensitive data streams. (3) The results show that the delay for time-sensitive messages is significantly reduced, and jitter is stable enough to achieve deterministic transmission.

2. Smart Substation Delay Analysis and TSN Technology Analysis

In the smart substation network architecture, Ethernet infrastructure is used to facilitate information interaction across three layers of devices. Because various types of service traffic are intertwined within two networks, the traffic scheduling technology of TSN is used to optimize QoS for different types of service traffic.

2.1. Smart Substation Delay Analysis

The IEC 61850 standard is widely used to construct smart substation network systems. This standard defines a communication interface model consisting of three layers, in which each layer adopts an open structure to facilitate network connection. The overall network architecture can be succinctly summarized as “three layers and two networks”, as shown in Figure 1. In this model, the station control layer and the interval layer devices are connected to each other through the station control layer network, and the interval layer and the process layer devices are connected to each other through the process layer network. As the data traffic within the substation network increase, various network issues emerge, including network congestion, network storms and latency jitter.

The smart grid requires a high QoS, especially for the security of relay protection within the grid. Currently, the general relay protection method used within the grid is still the differential protection method, in which Electronic Devices (IEDs) at the interval layer collect current data from multiple measurement points under the Ethernet network in the process layer and then send differential protection commands and recognize fault point information by analyzing these collected data. Hence, the reliability and real time of the communication network are good enough to promptly respond to the differential protection, namely, lower latency, slighter jitter and less packet loss. These all require the communication network of the smart grid to have low latency, a slight jitter and less packet loss, viz. time-critical sensitive streams. According to the IEC 61850, the fast-breaking stream and sensor data information stream require a delay of less than 3 ms (see

Table 1. Message transmission delay requirements.

Message	Signal Source	Low Real-Time Latency Requirement	High Real-Time Latency Requirement
fast:trip	trip signal	10 ms	3 ms
fast:other	other signal	100 ms	20 ms
medium speed	measured signal	100 ms	100 ms
low speed	parametric signal	500 ms	500 ms
Sampled value	transformer transmission	10 ms	3 ms

).

In order to meet the requirements of low latency and high reliability for the various traffic within the smart grid, TSN switches are needed to replace traditional Ethernet switches, and the Qbv scheduling mechanism of TSN switches is also utilized to satisfy the QoS of various traffic.

2.2. TSN QBV Technical Analysis

Among the traffic-shaping technologies employed by TSN, only Qbv technology can achieve fine-grained scheduling for each time-sensitive flow. However, its traffic scheduling is particularly complicated, especially during peak traffic.

In 2016, the TSN working group released the IEEE 802.1Qbv standard, which introduced the time-aware shaper (TAS) as a means to schedule time-sensitive and non-time-sensitive flows. The fundamental concept of TAS is the time-aware gating mechanism, shown in Figure 2. Within this mechanism, TAS sets up a transmission gate at the egress of each sending queue. Different types of flows are then mapped onto the corresponding sending queues according to the VLAN priority of the Ethernet frames. The switching state of the gate determines whether the packets in the transmit queue are used for transmission or not. If the gate state is set to “on”, the packets can be transmitted, whereas if the gate state is set to “off”, the packets are still cached in the queue. The switching state of the transmission gate is controlled by a time-based Gate Control List (GCL), which consists of multiple time slot table entries. The duration of all the time slots combines to form a cycle, and the GCL operates cyclically within this cycle to set the switching state of all the transmission gates within each time slot window. Therefore, by programing all the time slots, all time-sensitive flows can leave the switch sequentially at their scheduled time without being blocked by unscheduled traffic. This implies that the latency of time-sensitive packets passing through the switch at each hop is deterministic.

The TAS provides a “proprietary fast channel” for the time-sensitive traffic and ensures a deterministic transmission delay. Because the scheduling mechanism of TAS is time-triggered, it is imperative to establish clock synchronization among all bridges from the sender to the receiver for the scheduling mechanism to function properly, and IEEE 802.1AS is responsible for offering a shared global clock across all devices within a specific time domain.

Nevertheless, in order to establish a TSN network that encompasses complete domain connectivity and fully exploits the performance of TSN, it is crucial for the end-side device of TSN to be configured in conjunction with the TSN switch. The end-side device of TSN needs to possess the time-aware capability to send messages at precise hardware times. Taking into account the immense size of the grid and the considerable traffic it handles, the GCL gating table configured in each TSN switch will contain a substantial number of table entries, probably exceeding 2000 entries. Furthermore, the scheduling granularity is exceptionally fine, resulting in very short time slots corresponding to time-sensitive streams. If the end-side device of TSN fails to precisely control the timing of the message reaching the first-hop TSN switch and the message misses the corresponding time slot, the message will remain in the first-hop switch for an additional period, potentially up to one GCL cycle. This, in turn, will result in increasing delay and jitter throughout the end-to-end transmission. Smart substations need to transmit data not only horizontally across various layers, but also vertically between layers. This is especially essential for devices situated in the interval layer, which consists of a smart electronic device IED. The IED receives commands from the station control layer, measures the process layer devices, makes control decisions, and transmits the collected data to other IEDs or sends the data to the supervisory control and data acquisition system configured in the station control layer for further processing. All the traffic converges in the IED’s NIC, and the IED’s TSN NIC must have the capability of traffic shaping, enabling it to send messages at specific hardware-timed intervals. This ensures an enhanced guarantee of the end-to-end delay and jitter of time-sensitive flow. In this paper, we propose a TSN design scheme for end-side devices of a smart substation to enable devices to have the capability to send messages at specific intervals.

3. TSN End-Side Equipment Design Scheme

To ensure that the substation network achieves low latency and predictability, the TSN switches at each layer utilize strict prioritization and the TAS scheduling mechanism. Furthermore, IEEE 802.1AS is employed to synchronize the clocks of all the devices to form a global clock and drive the switch GCL gating table to run periodically. To ensure the predictable end-to-end delay and jitter of time-sensitive streams, the time at which an end device sends messages must match the forwarding schedule of the TSN switch. The end device needs to create a sending schedule that is coordinated with the TSN switch gating cycle to ensure that real-time periodic data are sent accurately and on time.

3.1. Overall Structure

In order to achieve the scheduling requirements of sending different time-sensitive streams at regular intervals in substation-side devices, this paper proposes an egress scheduling scheme that integrates both software and hardware components. The software adopts the open-source Linux system, and the hardware adopts the Intel 210 network card. The overall architecture is depicted in Figure 3. For each packet that needs to be sent at regular intervals, the application program working in the user state generates two data structures: a buffer to store the packet payload and a socket buffer (SKB) to store the packet’s sending time. The packet enters the Linux kernel state space through the socket subsystem, where the protocol stack wraps the packet into a complete Ethernet frame. Subsequently, the packet is processed by the packet scheduler of the corresponding NIC device I210, known as queuing disciplines (Qdisc), which is also referred to as the software TSN scheduler in this context. The Qdisc is configured using the TC tool provided by Linux, and it queues, stores, and forwards incoming packets according to the configured rules. When the packet is forwarded by the Qdisc to the corresponding NIC driver igb, the NIC driver notifies the I210 NIC driver Direct Memory Access (DMA) to transfer the packet to the NIC Random Access Memory, while also passing the packet sending time to the internal NIC. The I210 NIC then sends the packet from the internal NIC according to the predetermined sending time of the packet at regular intervals. In this case, the selection of ETF Qdisc determines the moment when the packet leaves the Linux kernel space, and the choice of the I210 NIC determines the time when the packet leaves the NIC. This integration of hardware and software components ensures that the data stream is dispatched from the end device at the specified time, thereby working with the time scheduling table of the TSN switch to achieve a defined end-to-end delay and jitter.

3.2. Software Implementation

The Linux kernel provides many flow control modules, among which Qdiscs are mainly used to control the transmission of kernel packets and manage an intermediate queue for caching packets being sent to the NIC device. Different Qdiscs have different rules for entering and exiting the queue, in which the Multi-Queue Priority Queueing (MQPRIO) manages traffic classes and maps traffic with different priorities to different NIC hardware queues. This priority is the SKB priority and also corresponds to the traffic VLAN priority tag. The ETF is responsible for accurately controlling the moment that packets leave the Linux kernel. As TSN time-sensitive flows are based on time triggering, the MQPRIO and ETF are very suited to the choices of Qdiscs.

In this scheme, we used MQPRIO and ETF Qdisc to implement Linux kernel 5.10 software TSN scheduling. As shown in Figure 4, real-time and non-real-time applications generate time-sensitive and best-effort flows, respectively. The time-sensitive flows need to set the traffic VLAN priority and the sending time (TxTime) according to the configuration, and the configuration here is in conjunction with the GCL table of the TSN switch. When the packets leave the kernel network stack, they enter the flow control module Qdisc. MQPRIO maps different priorities to different traffic classes according to the configured mapping rules, and different traffic classes are bound to different queues, and here the software queues correspond one-to-one with the NIC hardware queues. Different classes of flows enter the queues of different sub-Qdiscs for management and control, time-sensitive flows enter the ETF sub-Qdisc, and best-effort flows enter the Priority FIFO (PFIFO) sub-Qdisc. ETF Qdisc handles packets with TxTime, and its queue is a prioritized queue, which stores packets in the order of packet sending time TxTime and sends packets out of the queue and into the NIC driver queue and thus into the NIC in the time interval (called delta) before the sending time T. That is, the packet leaves the Linux kernel at the T-delta moment. PFIFO Qdisc follows a first-in-first-out (FIFO) rule, making it suitable for managing best-effort traffic.

3.3. Hardware Coordination

In order to improve the accuracy of message transmission timing, this scheme uses an Intel I210 as the network card. This NIC has four transmit queues, with queue numbers 0 and 1 having the ability to send hardware-timed messages, as shown in

Table 2. I210 NIC transmit descriptors.

63–57	56–32	31–16	15–9	8–0
Reserved	LaunchTime	VLAN	MACLEN	IPLEN

. The LaunchTime field in the transmit descriptor is used to set the time for sending packets in queues 0 and 1. LaunchTime is a 25-bit field defined in units of 32 ns, which can ensure that the packets leave the NIC device at a precise and controllable time. Therefore, NIC queues 0 and 1 can be used for the transmission of time-sensitive streams, and queues 2 and 3 can be used for the transmission of best-effort streams.

The hardware-timed sending capability of the I210 NIC works in conjunction with the Linux ETF Qdisc software scheduling function. ETF Qdisc sends packets to the NIC driver’s queue at the T-delta moment. The NIC driver writes the packet send time T to the LaunchTime field of the send descriptor and notifies the NIC driver DMA to carry the packet to queue 0 or queue 1 within the NIC. The NIC extracts the LaunchTime field of the descriptor to obtain the packet send time. Until the current time is greater than the send time T, the packet will leave the NIC hardware port. Queues 2 and 3 are used to cache the best-effort streams and transmit the messages in the queue out of the network port in FIFO order.

It is worth noting that the sending time of the message is set in the user application, the time base used is the Linux system clock, and the software scheduling performed in the ETF also uses the Linux system clock as the time base, whereas the current time reference is the hardware clock of the NIC when performing the timed sending in the hardware of the NIC, so the system clock and hardware clock of the TSN-side device must be kept synchronized. Here, linuxptp is used to synchronize the I210 NIC with the Linux system hardware and software clocks.

4. Experiment and Data Analysis

4.1. Experiment

In order to verify the TSN characteristics of the end-side device of the smart substation proposed in this paper, two different industrial controllers were used to simulate the interval layer device and the process layer device, respectively. The interval layer device serves the purpose of sending packets, whereas the process layer device is responsible for receiving packets. These simulations imitated the transmission process of the differential protection commands, and the TSN performance of the end-side device of the smart substation was verified through the measurement of the latency of the time-sensitive flow containing the differential protection commands.

The experimental scheme is depicted in Figure 5, involving both devices running the Linux-rt 5.3 real-time operating system, which is also equipped with ETF Qdisc and MQPRIO Qdisc. On the interval layer device, a periodic real-time packet-sending application is running with a sending period of 1 ms, a message length of 200 bytes, and a VLAN priority of 5. The I210 NIC is employed to send the message out at regular intervals according to the process illustrated in Figure 3. The two devices are connected through two MOXA TSN switches and one HIRSCHMANN TSN switch. The TSN switches turn on the Qbv scheduling function with a cycle period of 1 ms. Its GCL table is shown in

Table 3. GCL table of TSN switch.

Time Slot Entry	Slot Length	Gate Status
T0	0–200 μs	11011111
T1	200–600 μs	00100000
T2	600–1000 μs	11011111

, and the gating row from left to right indicates the switching status of queues Q7~Q0, respectively, where “1” signifies “on” and “0” signifies “off”. Meanwhile, the process layer device uses the Tcpdump tool to capture the corresponding message and record its reception time.

The offset of the message sending time TxTime is set to 200 μs on the interval layer device. This coincides with the moment when the TSN switch opens the corresponding time slot. This arrangement ensures that the message achieves the shortest sojourn time when it passes through the TSN switch. Additionally, the process layer device uses the I210 NIC to capture the hardware time slot of the message, known as the receive time ArriveTime. This allows for the calculation of the delay of the time-sensitive stream using the following formula:

delay = ArriveTime − Txtime

(1)

The MOXA TSN switch supports IEEE 802.1AS, and both devices run linuxptp to complete clock synchronization across all devices. After completing the clock synchronization, two sets of comparative experiments were conducted. In one set, the interval layer device activated the ETF Qdisc and the timed sending function of I210, whereas in the other set, these features were deactivated. Each group sent a total of 600,000 packets.

4.2. Data Analysis

The experimental results, as shown in Figure 6 and Figure 7, exhibit a significant contrast in outcomes. When the TSN functionality of the interval layer device remains inactivated, the departure time of time-sensitive packets from the network card interface becomes uncertain. These packets may enter the TSN switch within time slot T0, T1, or T2. When time-sensitive packets fall within time slot T1, the time-sensitive flow encounters a negligible queuing delay in the output queue Q5 of the switch. It is instantaneously dispatched, resulting in a 50 μs delay for time-sensitive flow. Correspondingly, when time-sensitive packets fall within time slot T0, the time-sensitive flow is buffered in the output queue Q5 for an additional period. It waits for the arrival of time slot T1, at which point the corresponding door for queue Q5 opens, allowing the packet to be transmitted. In this scenario, the maximum delay for the packet is 250 μs, which is 200 μs longer than the delay for packets falling within time slot T1. When a time-sensitive packet falls within time slot T2, it is buffered in queue Q5 for a more extended period. It is only forwarded when time slot T1 of the next cyclic period arrives. The maximum delay for such packets is approximately 1100 μs. Consequently, when the interval layer device lacks TSN capabilities, both the delay and jitter of time-sensitive flows are relatively high.

As illustrated in Figure 7, upon activating the TSN functionality in the interval layer device, the set time-sensitive message is sent just in time slot T1 of the GCL table of the TSN switch. Consequently, its delay is significantly reduced to approximately 60 μs and its stable delay jitter is around 140 μs when configured with Qbv scheduling of TSN switches. In addition, when TSN is enabled, the packet loss rate can be reduced from 0.11% to 0%, so network transmission is more reliable. More relevant data can be found in

Table 4. Statistical properties of TSN on and off.

Status	Packet Loss Rate	Min	Max	Jitter	Avg	Std Dev
TSN OFF	0.11%	30.65 μs	1133.59 μs	1102.94 μs	454.18 μs	342.85 μs
TSN ON	0%	14.34 μs	158.21 μs	143.87 μs	60.25 μs	29.37 μs

.

5. Conclusions

The continuous increase in IT/OT data traffic in smart substations gives rise to the problem that the transmission delay and jitter of time-sensitive streams also experience an increase. In this paper, within the network architecture of a TSN-based smart substation automation system, a traffic scheduling scheme utilizing TSN has been proposed. This scheme not only introduces TSN switches into the structure of a three-layer two-network intelligent substation, but also incorporates an end-side device combining Qbv, Qbu technology and the ETF Qdisc of Linux, which provides low latency and low jitter services for some time-sensitive data streams on the basis of realizing hardware-timed message transmission capability. The results show that the proposed scheme can significantly reduce transmission delay to 60 μs and stable delay jitter to 140 μs without losing packets. Compared to traditional best-effort transmission, this scheme is able to provide a bounded delay for time-sensitive streams, thereby significantly improving the reliability of data transmission. Additionally, it presents a reference for enhancing the QoS of the smart substation automation system.

For future research, efforts can be concentrated on real-time optimization and compatibility as well as interoperability. On one hand, further algorithm optimization can be pursued to achieve lower latency and more stable jitter, thereby enhancing the system’s real-time performance. On the other hand, addressing compatibility issues between TSN devices from different providers is necessary to ensure seamless interoperability within the system.