Software-Defined Time-Sensitive Networking for Cross-Domain Deterministic Transmission

Abstract

With the rapid development of Industrial 4.0, massive emerging time-critical applications pose new demands for industrial networks, such as strict latency boundaries, ultra-reliable transmission, and so on. Time-Sensitive Networking (TSN) is well suited for these demanding scenarios due to its support for deterministic transmission based on Ethernet. However, many use cases in real industrial scenarios involve non-TSN domains, and, thus, the network requires providing deterministic transmission across non-TSN domains to support them. To this end, this paper proposes a novel Software-Defined Time-Sensitive Networking (SD-TSN) framework that integrates the determinism of TSN and scalability of Software-Defined Networking (SDN). SD-TSN exploits the coordination between the coordinated controller and different domain controllers to bound non-deterministic queuing delays in a way that guarantees determinism. In particular, we designed a multi-domain time-aware traffic scheduling model, which harmonizes the time-aware shaper (TAS) in TSN and time-slots in non-TSN domains based on a global view, generating transmission schedules for each domain. A prototype testbed was built to conduct the experiments. The evaluation results demonstrate that the proposed method can efficiently achieve bounded delay and low delay variations in the transmission across non-TSN domains.

1. Introduction

With the rapid development of Industry 4.0, the number of time-sensitive applications is increasingly growing [1]. These applications pose new requirements for the network in terms of determinism, i.e., the data need to be transmitted with a bounded delay and low delay variation [2,3]. Ethernet is popular in various fields in terms of scalability and cost. However, satisfying these demanding requirements over standard Ethernet is hard due to the best-effort (BE) transmission mechanism [4].

Time-Sensitive Networking (TSN) is a promising technology that provides deterministic, real-time, and ultra-reliable transmission through IEEE 802 networks [5]. Within TSN, the IEEE 802.1Qbv time-aware shaper (TAS) introduces a transmission gate per queue in front of the egress port to activate or deactivate the queue [6]. The gate is in the open state to activate the queue such that the queued frames in the queue are permitted to be transmitted, and in the case where the gate is in a closed state, the frames of this queue are not transmitted [7]. The transmission order of the frames in the signal queue depends on the arrival order, and the transmission selection among multiple open queues is a strict priority algorithm based on priority [8]. Furthermore, the state of all gates is controlled by the gate control list (GCL). Overall, all frames are assigned to the corresponding queues based on the priority code point in their headers, then GCL determines the transmission behavior of the frames such that it guarantees determinism. Although TSN is a rather novel technology, it is being widely adopted in a closed network to establish deterministic connections, e.g., in-vehicle networks and industrial automation [9].

However, the end-to-end transmission of many applications in real scenarios tends to involve non-TSN domains. For instance, within a factory or industrial park, diverse devices (e.g., sensors, actuators, machines, and robots) will continuously interact with each other through the network, and they require deterministic transmission guarantees to facilitate intelligent production and service with data and information [10]. TSN struggles to provide determinism for these applications due to the benefits of it typically being limited to closed networks [9]. First, it would be impractical to enable all equipment to support hardware TSN mechanisms for achieving large-scale deterministic transmission since replacing all legacy devices with TSN-capable equipment is highly expensive. On the other hand, under the wave of Information Technology (IT) and Operational Technology (OT) convergence, various terminals (e.g., Human–Machine Interface and laptops) from different vendors cannot fully support TSN capabilities. Moreover, even enabling all devices to support hardware TSN mechanisms regardless of cost, guaranteeing determinism in a large-scale network through TSN, is still challenging due to the strict requirements for time synchronization and short link distances [11]. Therefore, the need to extend TSN’s capability to provide deterministic transmission across non-TSN domains is urgent.

Software-Defined Networking (SDN) is an attractive networking paradigm that separates the network control logic from the data plane, promoting the ability to program the network and facilitating network evolution [12]. After years of development and evolution, SDN has become quite mature and is being used in various fields, e.g., industrial [13], vehicle [14], and 5G [15]. Moreover, the fully centralized configuration model of TSN is highly fitting for the centralized control paradigm of SDN. Hence, this study combines the SDN paradigm and the fully centralized model of TSN, proposing a novel Software-Defined Time-Sensitive Networking (SD-TSN) framework to provide deterministic cross-domain transmission. Within SD-TSN, each domain controller is responsible for the network policy of its respective network, and the coordinated controller is responsible for the communication among them to establish the cross-domain transmission. SD-TSN exploits the global view to coordinate the domain controllers and the coordinated controller to bound queuing delays of time-sensitive traffic flows in non-TSN domains in a way that provides deterministic guarantees. In particular, we present a multi-domain time-aware scheduling model that generates transmission schedules for time-sensitive traffic flows in each domain by harmonizing the TAS in TSN domains and time slots in non-TSN domains. In general, the main contributions of this paper are as follows:

(1)

We propose an SD-TSN framework that builds end-to-end transmission across non-TSN domains for time-sensitive traffic flows. It exploits the coordination between the coordinated controller and different domain controllers to bound queuing delays in non-TSN domains to provide deterministic guarantees.

(2)

Within SD-TSN, we design a multi-domain, time-aware traffic scheduling model that harmonizes the TAS in TSN domains and time slots in non-TSN domains based on the global view to generate transmission schedules for time-sensitive traffic flows in each domain.

(3)

We conducted extensive experiments in the simulation environment and prototype testbed. The experimental results demonstrate that the proposed method can provide bounded delay and low delay variation guarantees for transmission across non-TSN domains.

This paper is organized as follows. Section 2 presents the related work. Section 3 describes the design of the SD-TSN framework. Section 4 presents the multi-domain time-aware traffic scheduling model. Section 5 evaluates the performance of our proposed model and algorithm. Finally, Section 6 concludes this paper and presents future work.

2. Related Work

With the rise of TSN, extensive studies on deterministic transmission have been published [16,17,18,19,20,21,22,23,24]. The approaches in these studies are feasible but typically oriented to TSN networks and thus cannot provide deterministic transmission across non-TSN domains. For this issue, some works investigated the extension of TSN through various advanced technologies to provide determinism. In general, they can be divided into two categories: wired-based solutions and wireless-based solutions.

Wire-based solution: The IETF deterministic networking (DetNet) working group defines an overall architecture to support transmission in layer three with deterministic latency, jitter, and very low/zero packet loss. [9]. Therefore, Peng et al. [25] designed a large-scale deterministic network flow-shaping mechanism to guarantee deterministic worst-case latency and zero packet loss for time-sensitive flows in Large-scale Deterministic Networks (LDNs). Krolikowski et al. [26] formulated the joint routing and scheduling problem for DetNet and provide efficient algorithms in large-scale deterministic networks. DIP is a scalable three-layer deterministic network that has the potential to become an LDN architecture due to its support for remote deterministic transmission services and substantial traffic management [27]. Tan et al. [11] proposed a hierarchical network containing access networks and a core network, which leverages Deterministic IP (DIP) in the core network to connect TSN networks for achieving long-distance deterministic transmission. Huang et al. [28] proposed a C-TSDN architecture that combines source routing, network slicing, and cycle-based scheduling technologies to ensure end-to-end deterministic transmission. Moreover, the integration of TSN and SDN is a promising solution that exploits centralized control to build the interoperability between TSN networks [9]. For instance, Xue et al. [7] propose an interconnection scheme following a software-defined TSN (SD-TSN) paradigm to enable deterministic communications among multiple closed networks. Zhong et al. [29] proposed an SDN-enable optical transport network simulation platform to support the end-to-end service resource prejudge in cross-domain TSN.

Wireless-based solution: This part describes the related work regarding extending TSN over wireless networks. First, given that both 802.11 and 802.3 are based on the same family of IEEE 802 standards [30], extending TSN over 802.11 is a natural and seamless step [31]. For instance, Seijo et al. [32] discuss the integration challenges of wired TSN and wireless local area network technologies and propose a hybrid TSN device architecture. Miranda et al. [33] present a modular, multi-domain controller architecture to provide end-to-end TSN-enabled control over LAN and WLAN domains. Cavalcanti et al. [31] describe the approaches to extend TSN to wireless with IEEE 802.11, containing the wireless network management model, wireless time synchronization, time-aware scheduling, wireless link reliability, etc. Furthermore, extending TSN through 5G is considered one of the best options to support Industry 4.0 as it supports ultra-reliable and ultra-low latency communications (URLLC). Wang et al. [34] proposed an integrated TSN and 5G industrial network architecture, where the 5G system acts as a logical TSN-capable bridge support deterministic communication. Atiq et al. [35] analyzed the recent standardization efforts and developments in IEEE 802.11 and 5G to enable low-latency, deterministic communications and present the current status of their integration with wired TSN. Larrañaga et al. [36] discuss current research and standardization work on 5G-TSN integration and quantify the 5GS bridge delay for a closed loop control application.

The above literature shows the research progress on deterministic transmission across non-TSN domains, summarized in

Table 1. Summary of related work.

Solution	Key Non-TSN Technology	Reference	Short Description
Wire-based	DetNet	[25]	Design an LFS mechanism that guarantees deterministic worst-case latency and zero packet loss for time-sensitive flows in Large-scale Deterministic Networks.
		[26]	Formulates the Deterministic Networking (DN) planning problem and presents an effective solution based on column generation and dynamic programming for Large-Scale Deterministic Networks.
	DIP	[11]	Proposes a hierarchical network containing access networks and a core network, in which access networks perform TAS to aggregate time-sensitive traffic and the core network exploits DIP to achieve long-distance deterministic transmission.
		[28]	Proposes the C-TSDN that enables cycle-based end-to-end deterministic transmission with multi-layer collaboration supported by the source routing and network slicing.
	SDN	[7]	Proposes an interconnection scheme following a software-defined TSN (SD-TSN) paradigm to enable deterministic communications among multiple closed networks.
	OTN	[29]	Proposes an SDN-enable optical transport network simulation platform to support the end-to-end service resource prejudge in cross-domain TSN.
Wireless-based	5G	[34]	Constructs a TSN-5G architecture, where the 5G system acts as a logical TSN-capable bridge, and then designs a Double Q-learning-based hierarchical particle swarm optimization algorithm (DQHPSO) to search for the optimal scheduling solution.
		[35]	Describes the recent standardization and developments in TSN and 5G and discusses the integration between TSN and 5G to enable bounded low-latency communications.
		[36]	Discusses current research and standardization work on 5G-TSN integration and quantifies, for a closed loop control application, the 5GS bridge delay.
	IEEE 802.11	[31]	Describes the approaches to extend TSN to wireless with IEEE 802.11, containing the wireless network management model, wireless time synchronization, time-aware scheduling, wireless link reliability, etc.
		[32]	Discusses the integration challenges of wired TSN and wireless local area network technologies and proposes a hybrid TSN device architecture.
		[33]	Presents a modular, multi-domain controller architecture to provide end-to-end TSN-enabled control over LAN and WLAN domains

. They mainly focus on extending TSN in the data plane with advanced technologies, which may be difficult and introduce extra costs. For example, the presented hierarchical architecture in [11] requires DIP to connect different TSN networks such that it enables long-distance deterministic transmission. In addition, 5G-based solutions typically need the device-side TSN translator (DS-TT) and network-side TSN translator (NW-TT) on the input and output ports of the 5GS, respectively, to enable process and forward data like Ethernet ports. This increases the cost of realistic deployment, and the transmission performance is easily restricted by the capabilities of these TSN translators. Unlike the above related works, the proposed SD-TSN focuses on extending TSN through the control-plane capabilities without imposing demanding requirements on the data plane. Depending on application requirements, environmental constraints and other factors, it can be wire-based, wireless-based, or a hybrid of both. Within SD-TSN, based on the global view and centralized control capabilities, the non-TSN domain acts as a logic bridge whose delay can be measured. Incorporated with the advanced tools developed for TSN, the controller will specify schedules for different domains in a way that guarantees deterministic transmission across non-TSN domains.

3. Software-Defined Time-Sensitive Networking Framework

Following the paradigm of network control and data forwarding separation in SDN and the fully centralized configuration model of TSN [37], the SD-TSN framework consists of the physical infrastructure plane, the orchestration and configuration plane, and the application plane, as shown in Figure 1.

3.1. Physical Infrastructure Plane

The physical infrastructure plane includes TSN switches, non-TSN switches, and diverse end stations, such as the sensors, actuators, devices, and so on. TSN switches must support the forwarding process specified by IEEE 802.1Q (mainly including frame filtering, queuing frames, transmission selection, etc.) in a way that enables the implementation of the precise shaping and scheduling of time-sensitive traffic flows. These end stations will generate a variety of time-sensitive traffic flows, such as control data traffic from PLC to the robotic arms and cyclic data from controller to input/output communication. In SD-TSN, directly interconnected TSN-capable switches composed of a TSN domain and non-TSN domain are similar to this. We assume that the non-TSN domain is in charge of connecting different TSN domains, and each end station only connects with the TSN domain. There are several candidate technologies for the non-TSN domain, e.g., DetNet [38], Deterministic Dynamic Networks (DDNs) [39], DIP [40], 5G [34], and OpenFlow switches [7], etc.

3.2. Orchestration and Configuration Plane

This plane is logically centralized, consisting of the domain controllers, the coordinated controller, and the Centralized User Configuration (CUC). The domain controllers control the network domains, such as the Centralized Network Configuration (CNC) for the TSN domain. The coordinated controller communicates with all domain controllers, coordinating the network control and management orders among them. CUC communicates with end stations and the application plane, being responsible for retrieving the capabilities of end stations and transferring the requirements of the application plane to domain controllers or the coordinated controller.

The domain controller/CNC mainly consists of four modules, as described below:

• Topology discovery: This module exploits and collects the Link Layer Discovery Protocol (LLDP) to discover the network topology of the domain to support the path computation. Moreover, it sends LLDP messages periodically in a way that ensures the real-time discovery of the changes in topology, such as node addition and deletion.
• Path computation: Based on the topology information, it is responsible for computing the transmission path for each traffic flow according to the strategies from the coordinated controller. Various routing algorithms can be used for path computation, such as the Dijkstra shortest-path algorithm.
• Traffic scheduling: It is in charge of generating schedules for the domain to indicate the transmission behavior of traffic flows, such that it guarantees determinism. Based on the requirements from the CUC and strategies from the coordinated controller, this module leverages scheduling algorithms to compute the schedules, such as integer linear programming (ILP)-based and heuristic-based algorithms.
• Resource pool: It records all flows’ transmission behavior (e.g., sending time on the node) and the network state (e.g., bandwidth, throughput, and utilization) in the domain, and it updates them to the coordinated controller to form the global view.

The domain controllers communicate with the physical infrastructure through management protocols, such as NETCONF, SNMP, and RESTCONF [23].

The coordinated controller is composed of six main modules.

• Resource management: This module records the network resources from all domain controllers and formulates a global view of all network domains.
• Topology management: This module collects the topology information of each domain to build the topology of all network domains.
• Path management: It computes the path for the end-to-end connection across multiple domains based on the global topology information.
• Connection management: It builds the end-to-end connection for services across multiple network domains and manages the connections.
• Time synchronization: It manages and configures the time synchronization functions between multiple network domains.
• Traffic policy: Based on the global view of the network, it coordinates the transmission schedules among network domains.

The CUC translates and transfers the services requirements such as the deadline and other types of information to the domain controllers or coordinated controller for network configuration. When the CUC receives the service requirements, it identifies the end stations that generate and receive the traffic flows (namely, talkers and listeners) for each service. Then, the talkers group, listeners group, and requirements are sent to the domain and coordinated controllers to generate the transmission schedules for all traffic. After the schedules are computed, the CUC is in charge of informing the corresponding end stations according to the schedules. For instance, it classifies the services into different types with an identification (ID) and assigns sending time for each flow. Within the TSN domain, the service types correspond to the traffic classes, and the traffic classes are distinguished through the VLAN ID and priority code point (PCP).

3.3. Application Plane

The application plane covers an array of colorful and diverse applications, such as smart manufacturing and automotive networks. With the capability of the orchestration and configuration plane, operators can distribute the requirements through high-level language for various demands without caring about the underlying details. These service requirements are transferred to the CUC through standard Application Programming Interface (API), such as the RESTful API.

4. Multi-Domain Time-Aware Traffic Scheduling

In this section, we propose a multi-domain time-aware traffic scheduling model to generate transmissions schedule for each domain.

4.1. System Model

Consider an SD-TSN network in which two TSN domains are connected through a non-TSN domain, as shown in Figure 1. The network can be modeled as a graph $G^n=(V^n,E^n),n=0,1,2$ where $V^n$ is the set of network nodes in the nth network, and $E^n$ is the set of network links in corresponding networks. $G^0$ denotes the non-TSN domain and $G^n(n\ge 1)$ denotes the TSN domains. A link $[v_a^n,v_b^n]$ of $E^n$ connects the source node $v_a^n$ and the destination node $v_b^n$. We assume an equivalence between the port and its associated link since any egress port is connected to at most one link. Hence, a physical queue in source node $v_a^n$ of the link $[v_a^n,v_b^n]$ is denoted as $q_m^{\left[v_a^n{v}_b^n\right]}$.

A set of periodic time-sensitive traffic flows F is considered. Each flow $f_i\in F$ is defined as

$$f_i=\{sr{c}_i,ds{t}_i,d{l}_i,T_i,l_i\}$$

(1)

where $sr{c}_i$ is the source node, $ds{t}_i$ is the destination node, $d{l}_i$ is the deadline requirement, $T_i$ is the period of the flow, and $l_i$ is the flow length. The transmission path of $f_i$ between its source and destination is denoted by $pat{h}_i$, containing an ordered sequence of nodes from $sr{c}_i$ to $ds{t}_i$. Within SD-TSN, it is split into several path segments, and each path segment is defined by $s{p}_i^n$. More specifically, $s{p}_i^{v_a^n}$ represents a portion of the path of $f_i$ in $G^n$, which ends at $v_a^n$. For each path segment, the egress and ingress nodes are denoted as $v_{x_i}^n$ and $v_{y_i}^n$, respectively. In this paper, we assume the paths are known before the traffic scheduling.

In the transmission of a flow, the propagation delay and queuing delay on link $[v_a^n,v_b^n]$ of $f_i$ are denoted as $p{d}^{[v_a^n,v_b^n]}$ and $q{d}_i^{[v_a^n,v_b^n]}$, respectively. The transmission delay and processing delay of flow $f_i$ at node $v_a^n$ are denoted as $t{d}_i^{v_a^n}$ and $f{d}_i^{v_a^n}$, respectively. Denoting the bandwidth as B, we have $t{d}_i^{v_a^n}=l_i/B$. Given that the propagation, processing, and transmission delays in the network elements are deterministic and constant [41], we have

$$c{d}_i^{[v_a^n,v_b^n]}=t{d}_i^{v_a^n}+p{d}^{[v_a^n,v_b^n]}+f{d}_i^{v_b^n}$$

(2)

In TSN domains, the gate open time for flow $f_i$ in queue $q_m^{[v_a^n,v_b^n]}$ is denoted as $o{t}_{i,m}^{[v_a^n,v_b^n]}$. The length of time that the gate is on the state of open is defined as the open window, which is denoted as $o{w}_{i,m}^{[v_a^n,v_b^n]}$. The cycle of the GCL, i.e, the interval over which the operations in a GCL repeats, is denoted as $g{c}^{[v_a^n,v_b^n]}$. It is computed as the least common multiple of periods of flows to be transmitted on link $[v_a^n,v_b^n]$. The notation details are shown in

Table 2. Notations.

Symbol	Definition
$G^n$	the nth network domain
$V^n$	the set of network nodes in $G^n$
$E^n$	the set of network links in $G^n$
$v_a^n$	the ath network node in $V^n$
$[v_a^n,v_b^n]$	the link that connects source node $v_a^n$ and destination node $v_b^n$
F	the set of time-sensitive traffic flow
$f_i$	the ith flow in F
$ds{t}_i$	the destination node of $f_i$
$d{l}_i$	the deadline requirement of $f_i$
$T_i$	the period of $f_i$
$l_i$	the length of $f_i$
$pat{h}_i$	the path of $f_i$
$s{p}_i^n$	the path segment of $f_i$ in $G^n$
$s{p}_i^{v_a^n}$	the path of $f_i$ in $G^n$ before $v_a^n$ in $G^n$
$v_{x_i}^n$	the egress node of $s{p}_i^n$
$v_{y_i}^n$	the ingress node of $s{p}_i^n$
$F^{[v_a^n,v_b^n]}$	the set of time-sensitive traffic flows routed through link $[v_a^n,v_b^n]$
$p{d}^{[v_a^n,v_b^n]}$	the propagation delay on $[v_a^n,v_b^n]$
$t{d}_i^{v_a^n}$	the transmission delay of $f_i$ at $v_a^n$
$f{d}_i^{v_a^n}$	the processing delay of $f_i$ at $v_a^n$
$q{d}_i^{[v_a^n,v_b^n]}$	the queuing delay of $f_i$ on $[v_a^n,v_b^n]$
$c{d}_i^{[v_a^n,v_b^n]}$	the sum of the transmission delay, processing delay, and propagation delay of $f_i$ on $[v_a^n,v_b^n]$
$e{d}_i^n$	the end-to-end delay between the ingress node and the egress node of $s{p}_i^n$
B	the link bandwidth
$q_m^{[v_a^n,v_b^n]}$	the mth queue at $v_a$ of $[v_a^n,v_b^n]$
$o{t}_{i,m}^{[v_a^n,v_b^n]}$	the open time of the gate for $f_i$
$o{w}_{i,m}^{[v_a^n,v_b^n]}$	the open window of the gate for $f_i$
$g{c}^{[v_a^n,v_b^n]}$	the gating cycle at node $v_a^n$ of $[v_a^n,v_b^n]$
$t_{ig}^{[v_a^n,v_b^n]}$	the ingress timestamps on $[v_a^n,v_b^n]$
$t_{eg}^{[v_a^n,v_b^n]}$	the egress timestamps on $[v_a^n,v_b^n]$
${\delta }_i$	the difference in transmission delay between $f_i$ and $f_0$.
${\theta }_i^0$	the worst-case end-to-end delay of $f_i$ on $s{p}_i^0$
$s{t}_i^n$	the starting time of $f_i$ in $G^n$

.

4.2. Problem Formulation

In the end-to-end transmission of a time-sensitive traffic flow, the end-to-end delay can be divided into four parts, namely, the propagation delay, the transmission delay, the processing delay, and the queuing delay. The first three delays can be calculated directly based on the global view and, thus, can be considered to be deterministic. Queuing occurs in a switch when multiple traffic flows attempt to transmit through the same output port simultaneously [42]. In real scenarios, a wide variety of traffic flows will arrive at the switch at any time and demand to be transmitted; thus, the queuing delay is non-deterministic. TSN enhances the hardware mechanism of Ethernet-based switches and employs various shaping and scheduling techniques to achieve bounded queuing delays in TSN domains. However, time-sensitive traffic flows may suffer non-deterministic queuing delays when they are transmitted in non-TSN domains.

SD-TSN aims to bound the non-deterministic queuing delay to provide deterministic guarantees. It can be achieved as long as no traffic flows transmit over the same output port at the same time in non-TSN domains, i.e., the queuing delay of these flows on each node in the non-TSN domain is equal to zero. To this end, SD-TSN exploits traffic shaping and scheduling to allocate the time for all traffic to be injected into the non-TSN domains in the TSN domain before the time-sensitive traffic enters the non-TSN domain. With the allocated injection times, all flows do not interfere with each other during the transmission in non-TSN domains. However, traffic outside the schedule (e.g., best-effort traffic) will inevitably appear in the non-TSN domain which impairs the deterministic guarantees. For this issue, SD-TSN exploits the logical centralized control to restrict and compensate for the non-deterministic queuing delay caused by these flows. Specifically, we define the following constraints to guarantee deterministic communication in the SD-TSN.

In the TSN domain, the scheduling model leverages GCLs to specify flow transmission behavior for determinism. For each gate event of all GCLs, it must be able to accommodate the transmission of corresponding flows, which can be describe as follows:

$$\begin{array}{cc}\hfill & \forall G^n,n\ge 1,\forall [v_a^n,v_b^n]\in E^n,\forall f_i\in F^{[v_a^n,v_b^n]}:\hfill \\ \hfill & \frac{l_i}{B}\le o{w}_{i,m}^{[v_a^n,v_b^n]}\le g{c}^{[v_a^n,v_b^n]}.\hfill \end{array}$$

(3)

In addition, the interval between gate open events on two adjacent nodes must allow for the normal transmission of time-sensitive flows. A prematurely closed gate will accidentally terminate the flow transmission with adverse effect. Thus, we have

$$\begin{array}{cc}\hfill & \forall G^n,n\ge 1,\forall \left[v_a^n,v_b^n\right]\in E^n,\forall f_i\in F^{[v_a^n,v_b^n]}:\hfill \\ \hfill & o{t}_{i,m}^{[v_b^n,v_b^n+1]}\ge o{t}_{i,m}^{[v_a^n,v_b^n]}+c{d}_i^{[v_a^n,v_b^n]}.\hfill \end{array}$$

(4)

If gates of more than one physical queue are open, the interference of the flows can occur. For example, when a flow in the queue with a lower priority is transmitting, flows in other queues wait until a frame in that flow finishes transmitting. In addition, the flow transmission of the queue with a lower priority is interrupted. To avoid this phenomenon among multiple physical queues, the open windows of their gates are not allowed to overlap. That is,

$$\begin{array}{cc}\hfill & \forall G^n,n\ge 1,\forall \left[v_a^n,v_b^n\right]\in E^n,\forall f_i,f_j\in F^{[v_a^n,v_b^n]},i\ne j,\hfill \\ \hfill & \forall \alpha \in \left[0,g{c}^{[v_a^n,v_b^n]}/T_i-1\right],\forall \beta \in \left[0,g{c}^{[v_a^n,v_b^n]}/T_j-1\right]:\hfill \\ \hfill & (o{t}_{i,m}^{[v_a^n,v_b^n]}+\alpha \times T_i\ge o{t}_{j,m^{\prime }}^{[v_a^n,v_b^n]}+o{w}_{j,m^{\prime }}^{[v_a^n,v_b^n]}+\beta \times T_i)\vee \hfill \\ \hfill & (o{t}_{j,m^{\prime }}^{[v_a^n,v_b^n]}+\beta \times T_j\ge o{t}_{i,m}^{[v_a^n,v_b^n]}+o{w}_{i,m}^{[v_a^n,v_b^n]}+\alpha \times T_i)\hfill \end{array}$$

(5)

For each time-sensitive traffic flow, it is required to arrive at the destinations within the deadline, then we have

$$\begin{array}{c}\hfill \forall f_i\in F:0<o{t}_{i,m}^{[ds{t}_i-1,ds{t}_i]}+c{d}_i^{[ds{t}_i-1,ds{t}_i]}-o{t}_{i,m}^{[sr{c}_i,sr{c}_i+1]}+c{d}_i^{[sr{c}_i,sr{c}_i+1]}\le d{l}_i\end{array}$$

(6)

For network domains that do not support the hardware TSN mechanism, we aim to ensure that all time-sensitive traffic flows do not interfere with each other during the transmission in these domains by allocating isolated time slots for them when they enter the domain. First of all, the injection time of each time-sensitive flow into the non-TSN domain should not overlap with each other to prevent adverse results. With TAS, the constraint can be described as follows:

$$\begin{array}{cc}\hfill & \forall G^n,n\ge 1,\forall f_i,f_j\in F,v_{x_i}^n=v_{x_j}^n,i\ne j,\hfill \\ \hfill & \forall \alpha \in \left[0,g{c}^{[v_{x_i}^n,v_{y_i}^0]}/T_i-1\right],\forall \beta \in \left[0,g{c}^{[v_{x_j}^n,v_{y_j}^0]}/T_j-1\right]:\hfill \\ \hfill & (o{t}_{i,m}^{[v_{x_i}^n,v_{y_i}^0]}+\alpha \times T_i\ge o{t}_{j,m^{\prime }}^{[v_{x_j}^n,v_{y_j}^0]}+o{w}_{j,m^{\prime }}^{[v_{x_j}^n,v_{y_j}^0]}+\beta \times T_i)\vee \hfill \\ \hfill & (o{t}_{j,m^{\prime }}^{[v_{x_j}^n,v_{y_j}^0]}+\beta \times T_j\ge o{t}_{i,m}^{[v_{x_i}^n,v_{y_i}^0]}+o{w}_{i,m}^{[v_{x_i}^n,v_{y_i}^0]}+\alpha \times T_i)\hfill \end{array}$$

(7)

Incorporating the global view of the coordinated controller, the propagation delay, transmission delay, processing delay, and transmission path are known. Thus, the goal of transmission with zero queuing delay can be achieved by further adjusting the departure time of the flow at the egress node in the TSN domain; thus, the constraint is as follows:

$$\begin{array}{cc}\hfill & \forall \left[v_a^0,v_b^0\right]\in E^0,\forall f_i,f_j\in F^{[v_a^0,v_b^0]},i\ne j\hfill \\ \hfill & \forall \alpha \in \left[0,hp(i,j)/T_i-1\right],\forall \beta \in \left[0,hp(i,j)/T_j-1\right]:\hfill \\ \hfill & (s{t}_i^0+\sum _{k\in s{p}_i^{v_a^0}}c{d}_i^{[k,k+1]}+t{d}_i^{v_a^0}+\alpha \times T_i\ge s{t}_j^0+\sum _{k\in s{p}_j^{v_a^0}}c{d}_j^{[k,k+1]}+\beta \times T_j)\vee \hfill \\ \hfill & (s{t}_j^0+\sum _{k\in s{p}_j^{v_a^0}}c{d}_j^{[k,k+1]}+t{d}_j^{v_a^0}+\beta \times T_j\ge s{t}_i^0+\sum _{k\in s{p}_i^{v_a^0}}c{d}_i^{[k,k+1]}+\alpha \times T_i)\hfill \end{array}$$

(8)

where $s{t}_i^0=o{t}_{i,m}^{[v_{x_i}^n,v_{y_i}^0]}+c{d}_i^{[v_{x_i}^n,v_{y_i}^0]}$, representing the starting time of flow $f_i$ in a non-TSN domain, and $hp(i,j)=lcm(T_i,T_j)$ is the least common multiple of $f_i$ and $f_j$, also known as the hyperperiod of $f_i$ and $f_j$.

Considering the influence of unscheduled best-effort traffic flows, we use the meter mechanism to limit the maximum forwarding traffic at the ingress port. In the worst case, a time-sensitive traffic flow waits in a non-TSN node until the best-effort traffic is transmitted. Denote the maximum traffic volume allowed for forwarding in one gate control period as $\rho$, the volume of best-effort traffic can vary between 0 and $\rho$ at each non-TSN node. Then, the queuing can occur randomly, resulting in a random end-to-end delay in the non-TSN domain. In the proposed model, we utilize the mechanism in [43] to measure the delay in the non-TSN domain. In the non-TSN domain, the flow for the delay measurement, denoted as $f_0$, is sent periodically. The period of $f_0$ is named as a test cycle. The ingress timestamps $t_{ig}^{[v_a^n-1,v_a^n]}$ and the egress timestamps $t_{eg}^{[v_a^n,v_a^n+1]}$ for each network node $v_a^n$ are recorded and constructed as the timestamp pairs $(t_{ig}^{[v_a^n-1,v_a^n]},t_{eg}^{[v_a^n,v_a^n+1]})$. The timestamp pairs are collected by the domain controller and sent to the coordinated controller to construct a delay view. With timestamp, the delay of path segment $s{p}_i^0$ in the hth test cycle is given as

$$e{d}_{i,h}^0=t_{eg,h}^{[v_{x_i}^0,v_{x_i}^0+1]}-t_{ig,h}^{[v_{y_i}^0-1,v_{y_i}^0]}-{\delta }_i,$$

(9)

where ${\delta }_i={\displaystyle \sum _{k\in s{p}_i^0}}(c{d}_i^{[k,k+1]}-c{d}_0^{[k,k+1]})$ reflects the difference in constant delay between $f_i$ and $f_0$. Furthermore, the worst-case delay of flow $f_i$ in the non-TSN domain is given as

$${\theta }_i^0=max\left\{e{d}_{i,h}^0\right|h=1,2,\cdots ,H\}.$$

(10)

When a time-sensitive traffic flow is transmitted from the non-TSN domain to a TSN domain, the random queuing delay in the non-TSN domain results in a delay variation between the flow periods. To eliminate the delay variation, the departure time of flow $f_i$ at the ingress switch of the TSN domain has the following constraint:

$$\begin{array}{cc}\hfill & \forall f_i\in F,\forall [v_{x_i}^n,v_{y_i}^0],[v_{x_i}^0,v_{y_i}^{n^{\prime }}],n\ge 1,n^{\prime }\ge 1:\hfill \\ \hfill & o{t}_{i,m}^{[v_{y_i}^{n^{\prime }},v_{y_i}^{n^{\prime }}+1]}\ge s{t}_i^0+{\theta }_i^0+p{d}^{[v_{x_i}^0,v_{y_i}^{n^{\prime }}]}+f{d}_i^{v_{y_i}^{n^{\prime }}}.\hfill \end{array}$$

(11)

4.3. Scheduling Algorithm

This subsection elaborates on the implementation of the proposed scheduling model. The most straightforward method to compute the schedule is simply building constraints all at once and then putting them into an SMT solver. However, this approach is highly time-consuming and even fails especially in the case of the number of flows being large. Hence, we present a Tabu search-based multiple-domain scheduling (TMDS) Algorithm 1 for the proposed scheduling model. Compared to the SMT-based method, our method aims to find a feasible solution for flows one by one, rather than solving all constraints together for all flows, and the pseudocode of the algorithm is shown in Algorithm 1. Here, we simply create an equivalence between the sending time and the GCL since the transformation is straightforward [44].

Algorithm 1 Tabu search-based multiple-domain scheduling algorithm

Input: Flow set F, Network domain set $G^n$ Output: Schedule set S        1: $S\leftarrow \left\{\right\}$        2: $F^{*}\leftarrow SortFlow\left(F\right)$$▹$ Sort F by flow period in ascending order        3: for all $f_i\in F^{*}$ do        4:     Get worst-case delay ${\theta }_i^0$ according to Equation (10)        5:     $S_i^{*}\leftarrow InitialSchedule(f_i,S,{\theta }_i^0)$$▹$ Initial the earliest sending time of $f_i$ on nodes over the path        6:     $TC\leftarrow TestConflict(G^n,f_i,S,S_i^{*})$        7:     while $TC\ne ⌀$ do        8:        $Neighbor{S}_i^{*}\leftarrow GenerateNeighborSolution(S_i^{*},TC)$        9:        $NeighSolution\leftarrow \left\{\right\}$        10:       for all $N{S}_i^{*}\in Neighbor{S}_i^{*}$ do        11:          $NeighSolution\leftarrow TestConflict(G^n,f_i,S,N{S}_i^{*})+NeighSolution$        12:       end for        13:       $S_i^{*}\leftarrow BestSolution\left(NeighSolution\right)$        14:       $TC\leftarrow TestConflict(G^n,f_i,S,S_i^{*})$        15:     end while        16:     $S\leftarrow S+S_i^{*}$        17: end for        18: return S

As shown in Algorithm 1, all flows are first sorted by period from frequent to infrequent (line 2), and the larger flow will be in the former when the periods of flows are equal. This is because larger and more frequent flows are harder to be scheduled later. Then, it initializes $S_i^{*}$ for $f_i$ through $InitialSchedule\left(\right)$ (line 5), that is, a set of sending times of $f_i$ on all nodes over the path. A set of sending times can be computed by the starting time at the source node according to the proposed constraints, and $InitialSchedule\left(\right)$ assigns an earliest possible starting time for $f_i$ according to the existing S. Next, $TestConflict\left(\right)$ tests the validity of $S_i^{*}$ by putting it into $G^n$ (line 6). Each conflict generated during testing can be regarded as a duration, e.g., if the $f_i$ with $S_i^{*}$ and the scheduled flows with S have a conflict according to Equation (5), the conflict value is equal to the length of the overlap between $f_i$ and the scheduled flows. If $TC$ satisfies all constraints, $S_i^{*}$ will be merged into S to update the schedule (line 16). Otherwise, the algorithm generates the neighborhood of the schedule according to the conflict, i.e., offset the initial starting time by the conflict (line 8). Each solution in the neighborhood is verified to comply with the constraints through $TestConflict\left(\right)$, and the best solution is selected for the next iteration (lines 10–14). This procedure will iterate until a feasible $S_i^{*}$ is found. The abovementioned steps are repeated until all flows have been scheduled.

5. Performance Evaluation

5.1. Evaluation in Simulated Environments

5.1.1. Simulation Setup

In this section, we evaluate the performance of our approach and compare it with the widely-applied SMT approach [16]. The evaluations are mainly focused on the schedulability and execution time, where schedulability is defined as the ratio between the number of scheduled flows and the number of flows to be scheduled. The simulation topology is the same as that in Figure 1. The bandwidth of the network is 1 Gbps. Each time-sensitive traffic flow transmits from a TSN domain to another TSN domain, and the non-TSN domain is composed of non-TSN capable switches. The propagation delay is set as 0.5 $\mathsf{\mu }$s, and the forwarding delay is set as 1 $\mathsf{\mu }$s. The period of the time-sensitive traffic is selected randomly from the set {1 ms, 2 ms, 4 ms, 8 ms, 16 ms, 32 ms, 64 ms}. Each time-sensitive traffic flow is required to finish its transmission in one period. The frame length is 64 bytes, and a flow contains $\gamma$ frames, $\gamma$ = 1, 2, 4, or 8 as in [20]. The time limit for scheduling approaches is 20 min [45]. For each flow, the value of $\gamma$ is selected randomly with uniform distribution. These scheduling approaches were implemented in Python 3.10 on a PC with 64-bit 8-core 3.0 GHz Intel Core-i7 and 16 GB memory.

5.1.2. Simulation Results

Various WCDs in the non-TSN domain: Figure 2a compares the schedulability of the TMDS under different WCDs. The WCD in the non-TSN domain $\theta$ is set as 0.2 ms, 0.4 ms, 5 ms, and 10 ms. The results show the effectiveness of the proposed model in different WCDs, and the smaller $\theta$ has an advantage in the schedulability. If the flow period is similar to the worst-case delay in the non-TSN domain, the flow is less possible to be scheduled. Moreover, the smaller $\theta$ can bring the larger delay bounds in the TSN domains. A larger delay bound means the larger queuing delay is tolerant, which means more flows can be scheduled when they transmit through the same link. Figure 2b presents the relationship of the execution time and the flow number. It shows that the more the number of flows increases, the more the execution time increases. This is because the excessive flow will lead to a much larger neighborhood space, which will lead to a much larger computation time. Moreover, the execution time is the longest when $\theta =0.2$, since the schedulable flow is the highest. Considering that different WCDs generate results with similar trends, we assume $\theta$ = 10 ms for the following tests.

Various flow numbers: A comparison of the schedulability for two scheduling approaches is shown in Figure 3a. The SMT-based approach can schedule all flows when the number of flows is less than 2200 and then decrease as the flow number increases. This is because the runtime by it reaches the time limit when scheduling about 2151 flows, i.e., no more new flows can be scheduled as the flow number increases. Our approach can maintain 100% schedulability until 2200 flows and then decrease similarly thereafter. Note that the downward trend of TMDS is slightly faster than that of the SMT-based after the flow number larger than 2200. This is due to the fact that TMDS attempts to find a feasible solution for flows one by one, which may yield suboptimal solution in terms of the whole flows. Figure 3b compares the execution time of these approaches for scheduling various flow numbers. Because all flows in a hyper-period of flow periods need to be computed according to given constraints, the SMT-based approach takes a longer time to schedule all flows. For instance, it spends approximately 1042 s to schedule 2000 flows; meanwhile, the runtime of our approach is only 3.84% of it, i.e., about 40 s. For the same reason, the SMT-based technique hits the time limit when scheduling approximately 2151 flows. The TMDS tries to search for a feasible solution through a Tabu search method-based trial rather than calculating all flows in a hyper-period according to the given constraints, which significantly reduces the execution time, e.g., it spends about 313 s when the flow number is 5000. The comparison of the bandwidth utilization for two scheduling approaches is given in Figure 3c. As expected, the bandwidth utilization increases as the flow number increases when the flow number is less than 2200. After that, the bandwidth utilization of TMDS gradually decreases as the flow increases since the schedulable flow decreases gradually. It is worth noting that the maximum bandwidth utilization is less than 10% by two approaches, which reserves sufficient resources for the transmission of BE traffic. Furthermore, the time-sensitive traffic flows are typically periodic and small according to the descriptions of flow properties in IEC/IEEE 60802 [46]; thus, their transmission typically would not block BE traffic by occupying the egress port for a long time.

Various bandwidth utilizations: To further evaluate the robustness of our approach, we design a scheduling test in various bandwidth utilizations. Specifically, we add a certain number of time-sensitive flows to the networks with different bandwidths, respectively. We choose 10 different bandwidth utilizations of the networks: 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%. The number of added flows is randomly generated between 1800 and 2000, and these flows need to be scheduled by two scheduling approaches. Figure 4a compares the schedulability of two approaches under different network bandwidths. It shows that both scheduling approaches can schedule all flows when the remaining resources are sufficient. For instance, the TMDS and the SMT-based approach can maintain 100% schedulability before the bandwidth utilization less than 60% and 70%, respectively. Furthermore, the schedulability by TMDS is slightly lower than the SDM-based after the bandwidth utilization greater 60%. This is because the TMDS following a heuristic strategy may miss some feasible solutions, making it more sensitive to the remaining resources. The comparison of execution time for scheduling approaches is shown in Figure 4b. The TMDS only takes an average of 32 s to schedule these flows while the SMT-based approach spends around 907 s for scheduling. Overall, the TMDS drastically reduces execution time while ensuring that schedulability is not degraded.

5.2. Evaluation on Testbed in Real Environment

5.2.1. Experimental Setup

In this subsection, we present the prototype testbed we built to verify the effectiveness of the proposed approach in a realistic environment, as shown in Figure 5. The testbed is composed of two parts, which are the control plane for centralized control by the controller and the data plane for transmitting the time-sensitive flows, respectively. In the control plane, we develop the CNC for the TSN domain on a server with an Intel Xeon E2224 processor (Intel, Beijing, China) running at 3.4 GHz and 16 GB RAM. The proposed model is inserted into the traffic scheduling module for the GCLs’ computation. The OpenDaylight is deployed as the non-TSN domain controller to configure flow tables and meter mechanism in OpenFlow switches. In the data plane, we assume two TSN domains (Domain A and Domain C) connected by a non-TSN domain (Domain B that is composed of OpenFlow switches) to enable transmission across a non-TSN domain. All switches in the TSN domain support IEEE 802.1Qbv and IEEE 802.1AS for precise traffic scheduling and time synchronization. Two TSN domains are synchronized through the cross-domain time synchronization scheme while the TSN switches in each domain are synchronized through IEEE 802.1AS [47].

Table 3. Hardware devices and configuration.

Unit	Module	Configuration
TSN switch	ETX TSN Switch (Beijing University of Posts and Communications, Beijing, China)	NETCONF, GE
OpenFLow switch	Centec V350 (Centec, Suzhou, China)	Openflow1.3, GE
Grand master	OSA 5421 (Oscilloquartz, Neuchâtel, Switzerland)	GE, Rubidium
Talker/Listner	VIAVI MTS 5800 (VIAVI, Beijing, China)	GE

shows the details of the hardware devices.

In the experiment, we assumed all time-sensitive flows were sent from the talker to the listener, and the flow properties were the same as those in the previous subsection. Considering that typical industrial networks are usually based on three topologies: linear, ring, and tree [48]. We selected the linear topology as the test topology to evaluate our approach to avoiding the optimization space brought by various routing methods, and other topologies can be denoted as a set of linear topologies. In domains A and C, two TSN switches are linearly linked via a 1 Gbps full-duplex Ethernet cable. The traffic generator sends background traffic to the traffic receiver. The background traffic includes two parts, constant-rate traffic of 800 Mb/s and burst traffic of 200 Mb/s. All parts are assigned with the lowest priority, and the burst occurs randomly in the experiment. The propagation delay, the processing delay, and the worst-case delay in the non-TSN domain are measured through the measurement mechanism in [43].

Two test cases were designed to measure the end-to-end delay and delay variation of the time-sensitive traffic flows. The delay variation is defined as the end-to-end delay offset relative to the minimum delay in all periods. All test traffic flows were transmitted from the talker to the listener. In Case 1, two OpenFlow switches are connected linearly with fiber in domain B. In Case 2, twenty-six OpenFlow switches are linearly connected through fiber in domain B, and the length of each fiber segment is 80 km. Thus, the distance of the non-TSN domain in case 2 is equal to 2000 km.

5.2.2. Experimental Results

Table 4. Experimental results on the testbed.

Approach	Flow Number	Case 1			Case 2
		Schedulability (%)	Response Time (ms)	Bandwidth Utilization (%)	Schedulability (%)	Response Time (ms)	Bandwidth Utilization (%)
TMDS	10	100	79.602	0.47	100	181.201	0.51
	30	100	139.033	1.37	100	242.589	1.36
	50	100	271.356	2.35	100	379.265	2.41
	70	100	334.203	3.38	100	445.621	3.4
	90	100	413.025	5.01	100	528.624	4.98
SMT-based	10	100	1025.621	0.51	100	1159.325	0.5
	30	100	4253.625	1.38	100	4401.389	1.38
	50	100	7358.251	2.33	100	7468.256	2.39
	70	100	10,254.163	3.4	100	10,373.673	3.43
	90	100	15,425.025	5.03	100	15,626.674	5.02

illustrates the experimental results of our approach with the increasing flow. It shows that the controller with the TMDS can respond to changes in traffic within one second, which is applicable to most industrial scenarios [9]. The response time in Case 2 is slightly longer than that in Case 1. This is because the messages take more time in domain B, thus increasing the time for the measurement of the WCD. The bandwidth utilization of these approaches increases as the flow number increase, but all are less than 5.03%. Therefore, the transmission of these flows would not interfere with or block the transmission of background traffic. In addition, the controller can schedule all flows for each test in Cases 1 and 2. For simplicity, we selected 50 flows for the next experiments.

Figure 6 shows the end-to-end delay and delay variation of the flow in Case 1. The worst-case delay in the SDN domain is about 780 $\mathsf{\mu }$s according to the delay measurement mechanism. In this condition, the burst best-effort traffic flows result in congestion. The minimum queuing delay result from the best-effort traffic flows in the SDN domain is about 12.2 $\mathsf{\mu }$s when the time-sensitive traffic flow waits for a frame to be transmitted. The maximum end-to-end delay in Case 1 is below 0.867 ms, demonstrating that the computed schedules achieved the bounded end-to-end delay. The delay variation in Case 1 is less than 1 $\mathsf{\mu }$s, which is much more smaller than the delay variation results from the best-effort traffic flows. The maximum delay variation is about 1/1000 of the end-to-end delay, showing the low delay variation feature of the proposed method.

Figure 7 shows the end-to-end delay and delay variation of Case 2. The results show that the proposed model is effective in the long-distance test case. The maximum end-to-end delay is less than 10.97 ms, which is bounded, although much larger than that in Case 1 due to the propagation delay in the SDN domain. The delay variation in Case 2 is less than 10 $\mathsf{\mu }$s, and the maximum delay variation is also about 1/1000 of the end-to-end delay. Although the delay variation is ten times that in Case 1, it is still much smaller than the delay variation results from the best-effort traffic flows in the SDN domain. This demonstrates the capability of the proposed method in alleviating delay variation. The delay variations in Case 2 mostly derive from time errors between two TSN networks. According to the time-synchronization measurement, the maximum time error of the end-to-end time synchronization in Case 1 is 300 ns, while it is 8 $\mathsf{\mu }$s in Case 2.

6. Conclusions and Future Works

The emerging industrial applications makes it imperative to guarantee the determinism of the end-to-end connection among multiple domains. In this paper, we propose a multi-domain time-aware traffic scheduling model in an SD-TSN framework. SD-TSN is designed based on the SDN paradigm in which the coordinated controller performs the cooperation among network domains. The proposed scheduling model unifies schedule time-sensitive traffic flows by harmonizing TAS in TSN domains and time-slots in non-TSN domains. The experimental results demonstrate that our proposed method can guarantee bounded delay and low delay variations for end-to-end transmission across non-TSN domains. The delay variation in a 2000 km end-to-end connection is below 10 $\mathsf{\mu }$s.

However, the deployment of SD-TSN in the real environment still faces several challenges. First, the realistic application scenarios are typically flexible, dynamic, and involve many devices from different vectors, making WCD measurement a difficult task. A direct approach to this issue is to enhance the capability of the controller to collect timestamp information from individual nodes with a high frequency. This provides real-time WCD measurements to cope with complex and dynamic applications. However, too frequently, measurement message interactions will interfere with the transmission of other time-sensitive traffic, such as network control data and safe-critical data. Moreover, enhancing diverse low-level devices (e.g., sensors and actuators) enables them to support frequent interactions, which is not feasible in terms of feasibility or economically. Thus, a promising solution is a more refined design of domain controllers for non-TSN domains. For non-TSN domains with delay guarantees (e.g., 5G URLLC), the domain controller can straightforwardly measure the WCD. For other non-TSN domains, it adapts advanced techniques to measure the WCD without high-frequency interactions, e.g., relaxing the accuracy of WCD measurements and predicting WCD through artificial intelligence techniques.

On the other hand, time synchronization is a factor that cannot be ignored since it directly affects the performance of traffic scheduling. For similar reasons, it is a trade-off problem between performance and costs. SD-TSN needs to find a feasible solution for better deployment, e.g., maintaining a low synchronization precision while enabling traffic scheduling. Furthermore, the downgrade deployment of SD-TSN is also an open issue. Our work is dedicated to guarantee deterministic transmission across non-TSN domains. However, in real environments, the talkers or listeners likely directly connect to the non-TSN domains, and their requirements are not demanding at the same time. The proposed scheduling model is not suitable for this case, but SD-TSN has a high potential to cope with this issue based on enhancements to the capabilities of the control plane. The key challenge lies in the design of the scheduling model and corresponding algorithm, which is the focus of future work.