Research on Real-Time Communication Algorithm of Substation Based on Time-Sensitive Network

Abstract

A time-sensitive network (TSN) extends the conventional Ethernet to support time-sensitive data flow. Thus, it enables simultaneous transmission of high reliability (HR) flow, medium reliability (MR) flow, and low reliability (LR) flow on the same network, thereby improving the reliability of data transmission. A TSN is a symmetric network that connects sensors and other facilities. As a backbone network, it can efficiently connect the underlying sensors and other levels of facilities, as well as ensure the quality of service of the network. For modern supervisory control and data acquisition (SCADA) systems, several types of sensors are widely used. The acquisition cycle of sensors for different purposes varies significantly from milliseconds to seconds. Moreover, these data also have different real-time requirements. Based on satisfiability modulo theories (SMT), this study proposes a TSN routing and scheduling method by adding related scheduling constraints. Compared with other methods, the proposed method can realize the routing and scheduling of hybrid flow in a hyper period and consider MR flow and LR flow, which improves the feasibility and certainty of data flow interaction between substations.

1. Introduction

With the development of big data and edge computing, smart grids [1,2] have been promoted. The core of the smart grid provides a cyber-physical system (CPS) [3], which transmits the information collected by the sensing device in the power system to the controller. An increasing number of intelligent algorithms [4,5,6,7] have been introduced into this field. It applies the result of instruction analysis to the power field equipment to ensure the stability of the power field equipment. To realize the reliability of the CPS, we should design a real-time communication network to provide a guarantee for it. A time-sensitive network (TSN) is based on the conventional Ethernet. It combines clock synchronization and resource reservation to simultaneously transmit high reliability (HR) flow, medium reliability (MR) flow, and low reliability (LR) in the network; thus, it provides deterministic data interaction and delay for embedded instrument devices in the network [8]. However, a wide range of sensor networks may need to be connected to each other. TSN has good compatibility and scalability, and it is decentralized and symmetric because of the uplink and downlink as well as architectural designs. It can be used as a backhaul network or backbone network of multiple sensor networks or directly connect sensors, controllers, and actuators. For the data acquired from various sensors, different real-time requirements should be individually met. The TSN network architecture has significant natural advantages in meeting several real-time requirements of sensors, and it has wide adaptability and easy deployment. Artificial intelligence algorithms, such as Convolutional Neural Networks (CNN) and genetic algorithms, are often involved in network scheduling [9,10,11,12,13].

In the substation communication process based on IEC61850 [6,14], the data flow is divided into three types: express flow, moderato flow, and slow flow. The response delay corresponding to the three types of messages is 10–500 ms. To further improve the certainty of communication between stations, we introduced a TSN to reduce the aforementioned delay to a microsecond level. The three data flows correspond to HR flow, MR flow, and LR flow in our study.

To ensure the deterministic delay and jitter of HR flow, we should consider MR flow and LR flow in the routing and scheduling process to improve scheduling performance considerably. The joint routing and scheduling problem calculates the offline gate control list (GCL) of all flows [15] using an algorithm before deploying the network. When the network is running, the switch schedules the data flow through the scheduling list to realize the deterministic data flow transmission. Additionally, the GCL can be online, which could cause a certain degree of complex calculations and destroy the existing scheduling strategy, thereby generating a new GCL to satisfy the needs of new scheduling tasks.

The structure of the paper is as follows: Section 2 introduces related works on TSN routing and scheduling, which was the main topic of this study. Section 3 briefly describes the relevant standards of TSN and the substation communication configuration scheme based on TSN. Section 4 presents the proposed system model designs and constraints of the flow. Section 5 elaborates on the routing and scheduling processes of the algorithm, including preprocessing, scheduling calculation, and online monitoring. Section 6 comprehensively considers the influence of the flow, quantity, and scale of network topology on the algorithms’ performance and compares it with other algorithms [16,17,18]. Section 7 summarizes the study.

2. Related Work

TSN routing and scheduling research can be divided into two types: offline and online scheduling [19]. Most researchers design offline schedules because the process is static, and all data flow, topology, scheduling constraints, and other data are known. In contrast, the online scheduling process is dynamic and variable, wherein dynamic adjustments are made according to changes in the data flow, network topology, and scheduling tasks. However, this process is often complicated.

Regardless of offline or online scheduling, the core task of the TSN scheduling algorithm is to calculate the GCL available for data scheduling. The solution method of the scheduling list is solved using approaches such as satisfiability modulo theories (SMT) [20], ILP [21], network calculus [18], or swarm intelligence optimization [17,22,23,24]. The authors of [17] proposes an improved ant colony algorithm, which aims to optimize the queuing delay in the data flow scheduling process and evaluates the average transmission delay and jitter of data flow scheduling by adding port and data flow constraints; the experimental results have achieved good performance indicators. The authors of [20] proposes an offline scheduling algorithm based on SMT. By adding data flow constraints, the maximum deadline of data flow transmission is used as a measurement index, and the schedulability of BE (best effort) flow is taken into account. In addition, the algorithm is also performing TSN simulation based on the INET framework of OMNET++, and the simulation results meet the performance indicators related to scheduling. The authors of [22] proposes an improved genetic algorithm, which considers both the routing and scheduling of data flows, effectively improves the scheduling performance of TT flows, and rationally optimizes network resource allocation. The authors of [24] proposes a hybrid genetic algorithm, which fully considers scheduling constraints, network topology scale, and data flow scale, effectively improving timeslot utilization and ensuring the real-time nature of data flow transmission. The scheduling constraint of the data flow in the solution process may be a single HR flow or the joint scheduling of HR and MR flow [16]. Finally, hardware or software verifies the algorithm according to a specific scenario, wherein the software verification uses a simulation program [25] or OMNET++ simulation software (omnetpp-5.5.1) [26] that simulates the hardware environment.

The main contributions of this study are as follows:

(1)

This study designed an offline and online TSN hybrid data flow routing and scheduling algorithm. Compared with previous studies [19], the offline and online scheduling algorithms designed in this study consider changes in the network topology and data flow. Previous studies [19] considered the change in network topology in the online scheduling mode and combined it with the offline mode to reconstruct a new network topology. The algorithm can prevent the failure of scheduling calculations caused by new equipment access;

(2)

This study presents an integration method of TSN and IEC61850. Simultaneously, three types of messages in the substation: express flow, moderato flow, and slow flow, are mapped to HR flow, MR flow, and LR flow in the TSN network. The introduction of time-sensitive technology can effectively reduce the response time during flow transmission, from milliseconds to microseconds. In the context of energy interconnection, the introduction of a reliable TSN communication technology can improve the certainty of communication between substations;

(3)

Finally, this study discusses the forwarding and dispatching problems of three different data flows, which is more in line with the real-time scenario of hybrid data flow transmission and is of great significance to research. Compared with previous studies [16,17,19], this study not only considers HR flow and MR flow but also considers the schedulability of LR flow. This is necessary for flow transmission between the substations.

3. Configuration of TSN Substation Communication

3.1. TSN Protocol

TSN contains an IEEE802.1 protocol cluster, which can provide the minimum delay for the data flow transmitted in a nondeterministic Ethernet and improve the certainty of the network. Section 3.1.1, Section 3.1.2, Section 3.1.3, Section 3.1.4, Section 3.1.5 introduce the relevant standards of TSN technology, which is the basis for configuring TSNs. A specific TSN configuration scheme is introduced in Section 3.2.

3.1.1. IEEE802.1 AS

IEEE802.1 AS can realize high-precision clock synchronization for all time-sensing systems in VLAN. Based on IEEE1588, this standard provides clock synchronization for the TSN domains requiring high clock accuracy. As shown in Figure 1, each TSN domain contains end nodes and switch nodes, among which the best master clock can be selected through switching clock synchronization packets to achieve global clock synchronization.

3.1.2. IEEE802.1 Qat

The IEEE802.1 Qat stream reservation protocol (SRP) defines a set of resource reservation strategies that effectively solve the problem of quality of service (QoS) drop caused by resource competition for different data flow. The SRP consists of two parts: multiple registration protocol (MRP) and stream reservation. In the process of the flow, the destination node needs to initialize the MRP to establish multiple spanning tree forwarding paths between source and sink. The SRP is implemented on the reserved path in the receiving and forwarding of the data flow. Simultaneously, the subtrees of the spanning tree detect changes in nodes in the network topology and make dynamic adjustments to adapt to changes in the topology.

3.1.3. IEEE802.1 Qbv

IEEE802.1 Qbv plan traffic scheduling enhancement protocol defines a time-aware shaper (TAS) mechanism. As illustrated in the following Figure 2, by defining different priority queues and adding queue gates controlled by a GCL in different transmission queues, it is realized that each provides a deterministic timeslot scheduling strategy for the flow in a transmission period.

3.1.4. IEEE802.1 Qbu

The IEEE802.1 Qbu frame preemption protocol optimizes the time-sensitive data stream queuing problem caused by uninterrupted data stream transmission. When the data flow is greater than 124 B, the express frame can interrupt the slow frame [27].

3.1.5. IEEE802.1 Qcc

The IEEE 802.1 Qcc stream reservation enhancement and performance improvement protocol provides three configurations for the TSN. It includes a fully distributed model of independent deployment on the user and network sides, centralized deployment on the network side, distributed user/centralized network configuration model of distributed deployment on the user side, and centralized network configuration model of centralized deployment on the network and user sides.

3.2. Configuration of TSN Substation Communication

Figure 3 shows the steps of configuring the TSN, which are as followsFirst: during substation communication, the constructed TSN network topology may be a complex dynamic topology system, which may contain multiple subsystems. To achieve deterministic flow, it is necessary to ensure that the network clock is synchronized [28]. An excessive clock offset may affect the QoS of data flow. Second: provide the largest possible bandwidth reservation for data flow to ensure that the transmission of time-sensitive data flow is reliable [29]. Third: to reasonably allocate the timeslot in each scheduling period, the TAS of the substation node and switching node should be designed reasonably [30], and the GCL should be generated for scheduling. Data flow with different QoS requirements is distributed in the corresponding port. fourth: combined with scheduling requirements, time-sensitive data flow can preempt the transmission of low-priority data flow [31]. Finally: consider the actual size of the network and whether to store configuration information in each node in the topology or set up an independent configuration node. Generally, a complex network structure requires separate network configuration nodes to improve scheduling performance.

In this study, we aimed to design a reasonable time scheduling GCL for substation nodes and switching nodes based on the provided data flow. The timeslot utilization rate of each period was improved considerably while ensuring schedulability. Considering that our entire simulation process was a small-scale TSN, the configuration was performed on each node in the network. Simultaneously, we assumed that the clock of the simulated network topology was synchronized.

4. System Model

In Section 4.1, we establish a system, scheduling task, and traffic models. In Section 4.2, we add transmission constraints to the traffic model. The model and constraints established in Section 4 are applied to the routing and scheduling algorithm described in Section 5. For specific design methods, please refer to Section 5.1, Section 5.2, Section 5.3, Section 5.4, Section 5.5, Section 5.6.

4.1. System Model

For the convenience of representation, we used the tuple, $\mathrm{G}(\mathrm{V},\mathrm{E})$, to represent the network topology [15]. Herein $\mathrm{V}={\mathrm{E}}_{\mathrm{n}}\cup {\mathrm{S}}_{\mathrm{n}}$ is a set of nodes in the topology graph, including substation node ${\mathrm{E}}_{\mathrm{n}}$ and switching node ${\mathrm{S}}_{\mathrm{n}}$; $\mathrm{E}$ denotes a set of full-duplex links. In this study, the transmission rate of the full-duplex physical link was 1 Gbps. $\mathrm{G}(\mathrm{T},\mathrm{F})$ represents the application [32], $\mathrm{T}$ represents the scheduling task running on the node, and $\mathrm{F}$ represents the virtual link connection established between tasks.

A real-time scheduling task is initiated by the substation node and arrives at the next substation node within the deadline. In practice, we try to allow more data flow to be scheduled within the deadline to improve the overall scheduling performance.

In the process of flow routing and scheduling, we considered three types of flow, including HR flow, MR flow, and LR flow. Under normal circumstances, HR flow has the highest QoS priority, and the unschedulability of data flow occurs primarily in the MR and LR flow. We used set S to represent the three types of traffic: $\mathrm{S}=\{{\mathrm{S}}^{\mathrm{T}\mathrm{T}},{\mathrm{S}}^{\mathrm{A}\mathrm{V}\mathrm{B}},{\mathrm{S}}^{\mathrm{B}\mathrm{E}}\}$, where ${\mathrm{S}}^{\mathrm{T}\mathrm{T}}$ represents the HR flow set, ${\mathrm{S}}^{\mathrm{A}\mathrm{V}\mathrm{B}}$ represents the MR flow set, and ${\mathrm{S}}^{\mathrm{B}\mathrm{E}}$ represents the BE flow set. The dictionary ${\mathrm{S}}_{\mathrm{i}}({\mathrm{S}}_{\mathrm{i}}\in {\mathrm{S}}^{\mathrm{T}\mathrm{T}} \mathrm{o}\mathrm{r} {\mathrm{S}}^{\mathrm{A}\mathrm{V}\mathrm{B}} \mathrm{o}\mathrm{r} {\mathrm{S}}^{\mathrm{B}\mathrm{E}})$ stores the corresponding attributes of the data flow, ${\mathrm{S}}_{\mathrm{i}}=\{{\mathrm{f}}_{\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}},{\mathrm{f}}_{\mathrm{o}\mathrm{f}\mathrm{f}\mathrm{s}\mathrm{e}\mathrm{t}},{\mathrm{f}}_{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}},{\mathrm{f}}_{\mathrm{d}\mathrm{e}\mathrm{a}\mathrm{d}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{e}}\}$, where ${\mathrm{f}}_{\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}}$ represents the frame length, ${\mathrm{f}}_{\mathrm{o}\mathrm{f}\mathrm{f}\mathrm{s}\mathrm{e}\mathrm{t}}$ represents the offset of the frame, ${\mathrm{f}}_{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}$ represents the sending period, and ${\mathrm{f}}_{\mathrm{d}\mathrm{e}\mathrm{a}\mathrm{d}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{e}}$ represents the sending deadline.

The integration of network topology and application programs determined a reasonable routing and scheduling way for a flow. Routing determined the path of data flow routing between nodes. Scheduling determined whether the data flow could be scheduled based on the routing path and combined with parameters, such as the size of the data flow, transmission period, and deadline. These parameters were defined when designing the model of the flow attribute ${\mathrm{S}}_{\mathrm{i}}$. When the flow could not be scheduled, timeslot optimization was used to determine whether the data flow could arrive within the deadline.

4.2. Related Constraints

Here, we add transmission constraints based on the three types of flows. The application ensures that each node should reasonably allocate the timeslot of the task within a hyper period, define the relevant constraints and forwarding requirements, determine the switching node and the substation node, and send the data flow to the destination simultaneously.

Related constraints are as follows:

Constraint1: As shown in Figure 4 and Equation (1), because the transmission of the flow is periodic, transmission should be completed within this period. In addition, ${\mathit{t}}_0$ to ${\mathit{t}}_1$ represent a transmission period; the timeslot occupied by the data flow should be within this period to ensure that it does not overflow. Herein, ${\mathit{t}}_{\mathit{period}}$ represents one transmission period and ${\mathit{t}}_{\mathit{frame}}$ represents the specific timeslot allocated by the data flow in the period

$${\mathit{t}}_{\mathit{p}\mathit{e}\mathit{r}\mathit{i}\mathit{o}\mathit{d}}\le {\mathit{t}}_{\mathit{f}\mathit{r}\mathit{a}\mathit{m}\mathit{e}}\le {\mathit{t}}_{\mathit{p}\mathit{e}\mathit{r}\mathit{i}\mathit{o}\mathit{d}+1}$$

(1)

Constraint2: As shown in Figure 5 and Equation (2), the transmission offset of the data flow should be between the minimum offset and the maximum period. This is considering that the periodically transmitted data flow will deviate from the fixed transmission timeslot owing to jitter. We must ensure that when an offset occurs, the data flow does not overflow the cycle owing to the offset. This may cause packet loss or unschedulability, which is very serious for the system. In addition, ${\mathit{t}}_1$ to ${\mathit{t}}_2$ represent the offset of the data flow and ${\mathit{t}}_2$ to ${\mathit{t}}_3$ represent the timeslot of the data flow. After determining the offset of the data flow, we should ensure that the data flow will not overflow its period. Here, ${\mathit{t}}_{\mathit{offset}}$ represent data flow offset.

$${\mathit{t}}_{\mathit{o}\mathit{f}\mathit{f}\mathit{s}\mathit{e}\mathit{t}}\le {\mathit{t}}_{\mathit{f}\mathit{r}\mathit{a}\mathit{m}\mathit{e}}\le {\mathit{t}}_{\mathit{p}\mathit{e}\mathit{r}\mathit{i}\mathit{o}\mathit{d}}$$

(2)

Constraint3: As shown in Figure 6, the transmission interval of the frame must be equal to the period. In addition, the data flow with ${\mathit{t}}_0$ as the transmission starting point should ensure that the transmission starting point of the data flow is ${\mathit{t}}_1$. Additionally, the transmission interval must be one period. Essentially, the data flow at ${\mathit{t}}_1$ should not be transmitted prior to ${\mathit{t}}_1$. The periodicity of the data flow transmission must be guaranteed.

Constraint4: The link is exclusive to the frame during transmission, and only one flow is allowed on the link. Essentially, other data flow cannot be transmitted in a certain occupied timeslot;

Constraint5: As shown in Figure 7 and Equation (3), when the data flow passes through the switch, the transmission of the next data flow should be greater than the delay of the switch in the previous data flow. In addition, the delay caused by the data flow passing through the switch is from ${\mathit{t}}_7$ to ${\mathit{t}}_8$; therefore, we should wait for a processing delay before sending the next data flow. Essentially, the switch node in the figure will have a delay from receiving the data flow to transmitting the data flow. This issue must be considered during the simulation.

$${\mathit{t}}_{\mathit{f}\mathit{r}\mathit{a}\mathit{m}\mathit{e}+1}\ge {\mathit{t}}_{\mathit{f}\mathit{r}\mathit{a}\mathit{m}\mathit{e}}+{\mathit{t}}_{\mathit{p}\mathit{r}\mathit{o}\mathit{c}\mathit{e}\mathit{s}\mathit{s}\mathit{i}\mathit{n}\mathit{g}}$$

(3)

Constraint6: As shown in Figure 8, the delay in the entire transmission process of the flow transmitted between nodes should be less than the entire scheduling period of the application. Essentially, the hyper-period from ${\mathit{t}}_0$ to ${\mathit{t}}_1$ is composed of the period of data flow1 formed by ${\mathit{t}}_0$ to ${\mathit{t}}_1$ and the period of the flow2 formed by ${\mathit{t}}_5$ to ${\mathit{t}}_6$. Both flow1 and flow2 should not exceed the overall scheduling period of the data flow. Thus, all the data flow can be scheduled;

Constraint7: As shown in Equation (4), the ${\mathit{t}}_{\mathit{s}\mathit{c}\mathit{h}\mathit{e}\mathit{d}\mathit{u}\mathit{l}\mathit{i}\mathit{n}\mathit{g}}$ generated during the entire transmission process of the flow transmitted between nodes should be less than ${\mathit{t}}_{\mathit{d}\mathit{e}\mathit{a}\mathit{d}\mathit{l}\mathit{i}\mathit{n}\mathit{e}}$; herein, ${\mathit{t}}_{\mathit{s}\mathit{c}\mathit{h}\mathit{e}\mathit{d}\mathit{u}\mathit{l}\mathit{i}\mathit{n}\mathit{g}}$ represents the total scheduling time (corresponding to ${\mathit{t}}_1$ in the figure), and ${\mathit{t}}_{\mathit{d}\mathit{e}\mathit{a}\mathit{d}\mathit{l}\mathit{i}\mathit{n}\mathit{e}}$ represents the deadline (corresponding to ${\mathit{t}}_1$ in the figure):

$${\mathit{t}}_{\mathit{s}\mathit{c}\mathit{h}\mathit{e}\mathit{d}\mathit{u}\mathit{l}\mathit{i}\mathit{n}\mathit{g}}\le {\mathit{t}}_{\mathit{d}\mathit{e}\mathit{a}\mathit{d}\mathit{l}\mathit{i}\mathit{n}\mathit{e}}$$

(4)

As shown in Equation (5), if there is no data flow queued in the transmission queue, the maximum offset of the data flow transmission is less than the ${\mathit{t}}_{\mathit{max}1}$.

$${\mathit{t}}_{\mathit{m}\mathit{a}\mathit{x}\_\mathit{o}\mathit{f}\mathit{f}\mathit{s}\mathit{e}\mathit{t}}\le {\mathit{t}}_{\mathit{m}\mathit{a}\mathit{x}1}={\mathit{t}}_{\mathit{p}\mathit{e}\mathit{r}\mathit{i}\mathit{o}\mathit{d}}-{\mathit{t}}_{\mathit{f}\mathit{r}\mathit{a}\mathit{m}\mathit{e}}$$

(5)

As shown in Equation (6), if there are multiple data flows in the queue, the scheduling time of the flow is the task scheduling period, and the scheduling time of the next flow should be less than ${\mathit{t}}_{\mathit{max}2}$; herein, ${\mathit{t}}_{\mathit{max}\_\mathit{o}\mathit{f}\mathit{f}\mathit{s}\mathit{e}\mathit{t}}$ represents the maximum offset, ${\mathit{t}}_{\mathit{p}\mathit{e}\mathit{r}\mathit{i}\mathit{o}\mathit{d}}$ represents the data flow period, and ${\mathit{t}}_{\mathit{q}\mathit{u}\mathit{e}\mathit{u}\mathit{i}\mathit{n}\mathit{g}}$ represents the queuing delay.

$${\mathit{t}}_{\mathit{f}\mathit{r}\mathit{a}\mathit{m}\mathit{e}}\le {\mathit{t}}_{\mathit{f}\mathit{r}\mathit{a}\mathit{m}\mathit{e}+1}\le {\mathit{t}}_{\mathit{m}\mathit{a}\mathit{x}2}={\mathit{t}}_{\mathit{p}\mathit{e}\mathit{r}\mathit{i}\mathit{o}\mathit{d}}+{\mathit{t}}_{\mathit{q}\mathit{u}\mathit{e}\mathit{u}\mathit{i}\mathit{n}\mathit{g}}+{\mathit{t}}_{\mathit{m}\mathit{a}\mathit{x}\_\mathit{o}\mathit{f}\mathit{f}\mathit{s}\mathit{e}\mathit{t}}$$

(6)

5. Routing and Scheduling Framework

The purpose of the algorithm is to generate a GCL that satisfies the constraints in [33,34]. Here, we propose a novel routing and scheduling approach architecture based on SMT [35] to generate an effective scheduling list. The framework included six parts: mapping, stream processing, timeslot occupancy, task scheduling, communication scheduling, and dynamic monitoring. As shown in Figure 9, we used a process chart to describe offline and online routing and scheduling algorithms (Section 5.1, Section 5.2, Section 5.3, Section 5.4, Section 5.5, Section 5.6).

As an example, Figure 10 briefly describes the computing tasks of the routing and scheduling frameworks. We assumed that the preprocessing work had been completed, including mapping, stream processing, and timeslot occupancy. The scheduling GCL was obtained through task and communication scheduling. The TAS running on each terminal node (substation node1–4) contained eight queues. However, we described only three queues in the figure to simplify the process. The substation nodes sent three data flows—HR, MR, and LR. Each substation node periodically sent data flow according to an offline dispatch GCL. One switching node received the mixed transmission data flow, combined the hyper period and scheduling constraints to schedule the task, scheduled the hybrid transmission data flow according to the calculated GCL, and transmitted it to substation node4.

5.1. Mapping

To enable the substation nodes in the network topology to perform scheduled calculation tasks, it was necessary to map the application program to the network topology. In the initial input file, the network topology and applications were separated. The network topology was written in XML, including all nodes (substation and switching nodes) and the connection status of the nodes; however, the nodes were not divided. This implies that we had to obtain the corresponding substation nodes and switch nodes through corresponding processing. By traversing the number of adjacent node connections of the node, we could obtain the substation and switching nodes, establish a link to them, and obtain the network topology with the node identification. Subsequently, the application contained scheduled calculation tasks that had to be executed on the substation node, and we could obtain the attributes of different types of data flow. The input application program was traversed, virtual links were established among the tasks of different substation nodes, and the tasks were initialized as well as stored in the corresponding substation nodes.

5.2. Stream Processing

After mapping, we obtained a scheduling task that contained the attributes of the data flow. Thereafter, we processed the data flow of the scheduled computing task executed on the substation node. First, it was crucial to obtain the hyper period of the flow set. The hyper period was used to evaluate the feasibility index for a group of data flow to satisfy the schedules. As the name implies, if there are several flow tasks, L.C.M. of these tasks is named by hyper period. When a group of periodic flow can be scheduled in a hyper period, they repeat every other hyper period and run uninterrupted.

5.3. Timeslot Occupancy

We simulated the scheduling of eight periodic data flows to design an offline GCL. The timeslot occupancy was obtained by calculating the worst time occupied by the arrival time in the worst case of different data flow. Note that to increase the occupancy rate of the hyper period timeslot, we must compress the deadline of each data flow considerably to achieve compression of the overall data flow scheduling period. Finally, we used the SMT solver to add constraints to obtain the maximum timeslot occupancy value.

HR flow is the most stable, with an almost constant scheduling period and execution time. Therefore, the compression deadline discussed above refers to LR and MR flows because they are the key factors for evaluating scheduling performance.

5.4. Task Scheduling

Task scheduling determines the GCL on the substation node and reasonably arranges the timeslots occupied by different tasks. The hyper period of all data flow to be scheduled was obtained in the stream processing part. During task scheduling calculation, first, the data flow was divided. For non-HR flow, the offset and cutoff times were calculated again. This result was obtained in the timeslot occupancy part. Second, we traversed the task to calculate whether the data flow satisfied the bandwidth occupation. For a data flow that exceeded bandwidth occupation, reasonable timeslots were reallocated. Finally, we obtained the task scheduling strategy that satisfied the terminal node and timeslot allocation of each data flow.

5.5. Communication Scheduling

A communication schedule establishes communication and data stream transmissions between different nodes. First, we determined whether the offset and deadline between nodes and the period of the data flow satisfied the requirements. Second, the fusion of the mapping section to the topology map, application program, and substation node scheduling GCL was obtained from the task scheduling section as input. Third, the stream constraint conditions between nodes were established to determine whether the communication schedule was satisfied. Finally, the final scheduling link sequence was obtained through a communication scheduling calculation, and the communication schedule was completed.

5.6. Dynamic Monitoring

Dynamic monitoring provides flexibility and diversity in scheduling. We could monitor the connected switching nodes and substation nodes at any time and change the scale of the initial topology to achieve online scheduling.

In the topology initialization process, we input the newly accessed node information into the configuration file to update the network topology. The scheduler re-executed procedures 5.1 to 5.5 to calculate the schedule list that satisfied the constraints and used it as the new schedule criterion.

6. Experimental Result

To verify the superiority of the scheduling algorithm, Python was used to write the offline and online scheduling algorithms. The algorithm included network topology, task scheduling, and related constraints, and it was tested on a PC with a hardware specification of Intel Core i7-8750 H and 16 GB RAM. Moreover, Python supports SMT, ILP, and network calculus in the programming process. Although the abovementioned methods are effective for TSN data flow routing and scheduling calculations, SMT provides better support for Python, and the library functions are clear and simple, which is convenient for us to develop. Therefore, in the solution process, we used an SMT-based Z3 solver to complete the TSN data flow for routing and scheduling calculation tasks. Compared with ILP and network calculus, the execution process of the Z3 solver based on SMT is simple and clear and has a relatively short algorithm execution time.

We used the network topology shown in Figure 11, which includes four substation nodes (Substation node1, Substation node2, Substation node3, and Substation node4) and a switch node. As summarized in

Table 1. Application flow properties.

Flow	Type	R	T	D	P
S1	HR	r1	200 µs	200 µs	750 B
S2	HR	r1	200 µs	200 µs	750 B
S3	HR	r1	200 µs	200 µs	750 B
S4	HR	r1	200 µs	200 µs	750 B
S5	MR	r2	200 µs	200 µs	750 B
S6	MR	r2	200 µs	200 µs	750 B
S7	MR	r2	200 µs	200 µs	750 B
S8	LR	r3	200 µs	200 µs	750 B

, the scheduling task includes eight data flows, which are four HR flows (S1–S4), three MR flows (S5–S7), and one LR flow (S8). The MR flow was all Class A, and the default idle slope was 75%. Herein, R represents the sending path of the flow, T represents the period of the flows, D represents the deadline of the flows, and P represents the length of the flows. The transmission times of HR flow, MR flow, and LR flow were 6, 12, and 12 µs, respectively.

The core task of routing and scheduling calculated the GCL, which is cyclic and can realize scheduling tasks for all data flow in a hyper-period. All data flows to be scheduled were sent to the link by the same port of the node.

We scheduled the data flow according to their priority and generated the GCL. In this case [16,17,22], we ran the algorithm 30 times and noted the average time. The overall TSN switch scheduling periods were 206.3, 209.4, and 211.3 µs, which caused the MR flow not to be scheduled, thereby resulting in scheduling failure. On the contrary, we ran our algorithm 30 times in the same case, and our result in an average scheduling period was 198.4 µs for the hybrid data flow, which satisfied the worst-case deadline, and all data flow could be scheduled.

For the purpose of verifying the general applicability of the scheduling approach, we changed the count of flow and the scale of the network topology, and we applied the algorithm to verify its scheduling performance. The scheduling results when the network topologies were 10, 20, 40, and 100, and when there were eight data flows are presented in

Table 2. The influence of network topology scale on scheduling.

Network Topology Scale	Number of Flow	Scheduling Performance
10	8	Y
20	8	Y
40	8	Y
100	8	Y

(Y means that data flow can be scheduled).

It can be observed from Table 2 that our algorithm can maintain the schedulability of the data flow in dynamically changing topologies, which improves the reliability of the algorithm. In addition, in the 10-node network topology, there were five source nodes, three destination nodes, and two switch nodes. The 20-node network topology contained 10 source nodes, five destination nodes, and five TSN switch nodes. In the 40-node network topology, there were 25 source nodes, 10 destination nodes, and five TSN switch nodes. The 100-node network topology contained 65 source nodes, 30 destination nodes, and five TSN switch nodes.

The network topology models of scales 10 and 100 are shown in Figure 12a,b, respectively. The network topology models of scales 20 and 40 were identical in the structure of the switch node. In four cases, switch node1 connected to the source node was selected as the test node, and the algorithm was run 30 times on average. The average scheduling period was 194.3 µs.

In addition, as summarized in

Table 3. The influence of the number of data flow on scheduling.

Number of Flow	Network Topology Scale	Scheduling Performance
8	10	Y
10	10	Y
20	10	Y
40	10	Y

, when the number of flows is 8, 10, 20, and 40, and the network topology scale is 10, we verified the impact of the number of flows on the performance of the algorithm scheduling (Y means that data flow can be scheduled).

As summarized in Table 3, we selected four different flow models, among which the data flow with a scale of 10 contained six HR flows, three MR flows, and one LR flow. The flow with a scale of 20 contained 12 HR flows, six MR flows, and two LR flows. The flow of size 40 contained 23 HR flows, 12 MR flows, and five LR flows. We ran the algorithm 30 times on average, and all data flows were scheduled.

In addition, we compared the algorithm execution times with those reported in the literature [16,17]. The execution time of the algorithm when the scale of network topology remained unchanged at 10, and the scales of data flow were 8, 10, 20, and 40 is shown in Figure 13. In a small-scale data flow, there was little difference in the execution times of the three algorithms. However, with an increase in the data flow scale, our algorithm (A) exhibited a better algorithm execution time than those in [16] (B) and [17] (C).

Similarly, the execution time of the algorithm when the data flow scale remains unchanged at eight and the data flow scales were 10, 20, 40, and 100 is shown in Figure 14. In small-scale networks, there is little difference in the execution time of the three algorithms, but with an increase in the network topology scale, our algorithm (A) exhibited a better algorithm execution time compared with those in [16] (B) and [17] (C).

7. Conclusions

In the research on TSN hybrid data flow transmission and scheduling, a design based on SMT is suitable for the GCL of TSN hybrid data flow transmission. The algorithm can realize both offline and online operating modes. When the network topology and scheduling data mode stream are fixed, a priori calculation can be used to solve the offline scheduling table to realize the scheduling of the data flow; the network topology and the scheduling data flow are not fixed. Additionally, our algorithm could identify new access devices by modifying the configuration file, and it could perform scheduling calculations to generate a new GCL.

Compared with other algorithms [16,17,23], the algorithm based on TSN substation communication considers HR flow, MR flow, and LR flow simultaneously and ensures that MR flow can be scheduled. In addition, our algorithm considered scheduling problems in online situations. The configurability of online scheduling can effectively solve the problem of scheduling table failure when topology changes. Considering the executing duration of the algorithm, our proposed algorithm can effectively reduce the executing duration of scheduling calculations and improve the overall availability of the algorithm in medium- and large-scale network topologies or flows.

The next step of this study is to apply the algorithm to a hardware platform for testing and to further improve the algorithm.