1. Introduction
A large-scale communication network is a crucial aspect of smart factories. It not only enables communication among actuators and sensors, enabling an adaptive and flexible process, but also an integration of large computing resources such as edge cloud and factory cloud systems. Factory cloud refers to a local server system near the shopfloor with cloud capabilities. Artificial intelligence, big data analysis, and cloud-based control systems increase production efficiency and quality by enabling overall orchestration and optimization. Combined with wireless communication, the digitalization of production systems can achieve high flexibility, allowing customized products with a batch size of one. Especially autonomous guided vehicles and mobile robots can benefit from wireless communication and cloud systems. Moving path planning and control algorithms to a factory cloud system enables smart and collaborative decision making and fast reaction without efficiency loss.
Industrial manufacturing communication needs to meet rigorous requirements regarding reliability, availability, and real-time capability [
1,
2]. State-of-the-art real-time communication is dominated by fieldbus protocols, such as EtherCat, PROFINET/PROFIBUS, or CC-Link. These protocols were designed for local connections inside a machine or a production cell with the focus on the high reliability and short cycle times. As a result, they are mostly manufacturer-specific and at the most compatible on the physical layer. Furthermore, all these solutions for industrial communication are wired based, creating static connections. Wireless solutions have only been used for a limited number of applications in industry, which are caused by the high requirements in the production environment [
3,
4]. Current state-of-the-art solutions such as Wi-Fi or Bluetooth cannot serve these requirements; therefore, communication networks are limited to wired communication. In consequence, industrial networks are rigid and do not provide the flexibility and connectivity needed for Industry 4.0. Two main deficits have been identified: Existing communication solutions are vendor specific and therefore not compatible with each other, and they do support wireless solutions.
With IEEE 802.1 Time-Sensitive Networking (TSN) [
5], a set of standards for deterministic communication has been introduced to IEEE 802.1 and IEEE 802.3 to support real-time communication over Ethernet networks. With TSN, large-scale Ethernet networks can transmit real-time critical data suitable for industrial applications [
6,
7]. This reduces the need for specific fieldbus protocols and vendor-specific communication systems. TSN introduces three main areas to Ethernet: time synchronization, sending scheduled traffic, and centralized, automated system configuration [
8]. In the prototype, TSN is used for the wired connection of the field devices with the factory cloud. Owing to its compatibility with enterprise networks, its vendor independence, and its scalability, TSN is well suited for large-scale networks for production in future.
The 5G system is seen as a key enabling technology for wireless communication in industrial use cases. Meanwhile, 3GPP, the standardization body for 5G, has made design considerations from the very beginning to define technology features that allow ultra-reliable low-latency communication (URLLC) for industrial use cases [
9,
10]. The 3GPP Releases 15 and 16 specify various technology features to enhance reliability and provide low-latency communication, including faster signaling schemes, traffic prioritization, redundancy and robustness for control and data transmission, etc. [
11]. Furthermore, 3GPP has specified the technological framework to allow the integration of 5G with TSN [
12,
13,
14]. The basic functionalities and features to allow 5G integration with TSN have been completed in 3GPP Releases 16 and 17. The 5G system is treated as a TSN bridge in the 5G-TSN integration framework. Features such as native support for Layer-2 (Ethernet) traffic, efficient transportation of TSN Ethernet traffic with compression schemes in the 5G network, End-to-End (E2E) time synchronization, Quality of Service (QoS) differentiation, priority class handling to map QoS flows of TSN traffic to 5G QoS, and translation functionalities for 5G-TSN interworking are supported. The 5G-ACIA (5G Alliance for Connected Industries and Automation) industrial forum has been investigating the detailed requirements, interfaces, and architectures that allow the integration of 5G with TSN-enabled networks in industrial use cases, bringing both automation industry and information communication technology players together [
15].
We have carried out a prototype setup integrating 5G in a TSN network to evaluate 5G-TSN performance for industrial applications, mainly in real-time analysis of the communication. The industrial application requirements are defined for virtual Programmable Logic Controllers (PLC) in mobile robotics use cases communicating with cyclic and synchronous signals. This prototype setup consists of a 5G URLLC standard-compliant pre-commercial system and a wired network supporting different TSN standards. The integration of the 5G URLLC prototype system has been carried out as a virtual bridge between the TSN switches as described in [
15].
The structure of this paper is as follows:
Section 2 describes the mobile robotics use case and its requirements toward the communication infrastructure. In
Section 3, the architecture and implementation of the 5G-TSN communication prototype are given. The detailed performance measurements carried out on the integrated prototype system for the use case requirements and a discussion of the results are presented in
Section 4.
Section 5 summarizes the paper and provides an outlook.
2. Use Case Mobile Robotics and Requirements
Mobility is an essential requirement of flexible lineless assembly systems to meet the market demand for highly customized products. Recent advancements in robotics, sensor systems, and communication infrastructure push toward ultra-flexible production systems [
16]. Industrial mobile platforms enable temporary production cells in assembly stations. Mobile robots enable new sets of use cases, and to realize them, reliable real-time communication is a must for mobile robots. Reliable real-time communication between the mobile robots and the factory cloud system enables collaborative decision-making tasks and advanced AI/ML methods, which was impossible with onboard processing on the mobile robots. In the simplest case, a mobile robot consists of a power supply, a platform, a manipulator, a computing and control unit, and several sensors. Depending on the applications and use cases, the mobile robot can be equipped with specific components, such as a loading unit or conveyor belt for transport. Mobile robots for more complex industrial applications are often equipped with a manipulator, enabling several possible assembly tasks such as pick and place, handling (storage, placing, holding, setting, etc.), and supporting processes (marking, cleaning, inspection, etc.). Recent publications [
17,
18] highlight the following advantages of mobile robots compared to fixed robots:
Greater reconfigurability and flexibility of the production system;
Increased production efficiency with high product variability;
Greater reliability and less downtime.
2.1. Factory Cloud-Based Control of Mobile Robotics
In cloud robotics, the control functionality of the robot is offloaded from the robot platform into an edge-cloud computing platform, such as a factory cloud, as shown in
Figure 1. The control functionality includes motion planning, PLC, localization and mapping, and all the monitoring and computational demanding processes. The robot controller in the factory cloud computes the trajectory and predicts how to act in the environment with obstacles. This decision is based on the data that a mobile robot transmits from internal and external sensors to the factory cloud system for processing via wireless communication [
19]. The evaluation and accuracy of the data are strictly connected with the choice of sensors. There are many approaches for data evaluation, since the correct and quick perception of the environment is seen as the most important characteristic of a mobile robot. Especially, approaches from computer vision and deep learning in connection with various camera-based sensors have shown a high potential for this.
Although mobile robots have great potential for many applications, they are still at the beginning of development and have only found their way into production chains in isolated cases. This is, among other things, due to some important challenges of current mobile robots. The most important challenges are listed below [
20]:
Localization, navigation, and trajectory planning are highly complex and are still the subject of current research;
Up to now, safe cooperation with people has only been possible in restricted movement modes;
Detection of obstacles is not possible in the required time, depending on the sensor technology used.
The limitation of the safe operation of a robot in a production environment is usually not specified by the moving mass but by the speed, detection, and reaction time. However, the sophisticated algorithms for obstacle detection and reaction often demand extensive computational resources that are generally unavailable on mobile robots, e.g., for energy consumption considerations. To establish complex anatomies, the decentralized processing of data acquired by the mobile robots enables high flexibility and supplies the computational resources required for complex tasks but has not been possible in the past due to the latency and reliability limitations of conventional wireless communication systems. As a new wireless technology, 5G is designed to satisfy both low-latency and high-reliability communication demands and provides enough bandwidth to resolve the issues mentioned above.
Both 5G and TSN enable the real-time connectivity of mobile robots to the factory cloud systems, and with that, mobile robots can transfer computationally intensive tasks to the factory cloud systems [
21]. By offloading such tasks from mobile robots, sophisticated and complex algorithms are possible for the localization tasks, safe cooperation, and real-time obstacle detection on the factory cloud systems. In this research, the cooperation between two mobile robots is targeted to be analyzed. Both 5G and TSN play a crucial role in mobile robots’ synchronized movement when picking the material cooperatively in this use case.
2.2. Requirements toward the Communication Infrastructure
In addition to wireless communication, mobile robots need reliable and deterministic communication. In this use case, the robot has a LiDAR (Light Detection and Ranging) system for detecting the environment, a camera system for precise positioning of the tool, and a cloud-based PLC. These three different data streams have different requirements regarding reliability and real time.
To define the requirements, the traffic types and characteristics given in [
22] and 5G-ACIA are taken as reference, and the requirements are extended based on our industrial use case.
Table 1 shows the traffic profiles for our use case. The Data Delivery Guarantee in the table indicates the aspect of the traffic’s requirement which needs to be guaranteed to make sure the traffic does not experience any errors or interruptions. In a cloud-based PLC, Data Delivery Guarantee indicates the maximum bound of communication latency between the transmitter and receiver to make sure the connection is not dropped. However, in LiDAR, besides the latency bound, it indicates the minimum required throughput to transfer the data. For camera streaming, Data Delivery Guarantee shows the minimum required throughput to function properly. The camera traffic is a video stream with low criticality, giving information for the positioning of the tool. The LiDAR sensor sends continuous data triggering an event-based reaction in case of an identified dangerous situation. The cloud-based PLC sends cyclic data to the robot, controlling the movement and behavior of the system. Since mobile robots need to collaborate with other robots or machines, a shared time zone is required, and synchronization between the end devices is needed. Therefore, the traffic types and requirements can be defined as shown in
Table 1. Generally, the cyclic PLC data need to be transmitted in a shorter time than the latency bound. The robot is not generally sensitive to jitter, or packet delay variations, as long as the latency bound is kept. However, if two mobile robots are collaborating, the jitter in the transmission can lead to some varying offsets in the movements of the two individual robots. Depending on the required synchronicity in movement of the robots, this may require an upper bound on the tolerable jitter. Alternatively, the control of the two robots can be made with a common time reference that is shared among the robots, e.g., by time synchronization over the network.
4. Discussion and Results
We have evaluated the performance of the integrating 5G with TSN in terms of the time synchronization accuracy, overall end-to-end latency and jitter behavior. All the measurements have been conducted in an industrial shopfloor with over-the-air transmissions. For validation, the data traffic with the highest requirements has been tested with 50 bytes message sizes and a cycle time of 2 ms, simulating the control data traffic in the mobile robotics use case described in
Section 2.
4.1. Time Synchronization Accuracy
There should be a common time reference for field devices and the communication endpoints in the network. Time synchronization is a fundamental requirement for many of the TSN protocols. Reliable and accurate communication in industrial use cases require accurate time synchronization. As described in
Section 3, we have used the NetTimeLogic PPS analyzer for precise time synchronization accuracy measurements of the prototype 5G-TSN integrated setup. The onboard FPGA on the PPS analyzer computes the time difference between the PPS signals at the two endpoints of the setup of 5G integrated with TSN, as shown in
Figure 8 with nanosecond accuracy. We have carried out PPS measurements for the time synchronization error of the 5G system integrated with TSN for over 6 h. As shown in the histogram in
Figure 11, the mean time synchronization accuracy in the setup of 5G integrated with TSN remains below 3 µs, while the maximum error remains below 8 µs. This level of time synchronization accuracy in the integrated setup (including over-the-air time synchronization in the URLLC test system) is sufficiently well suited for the IEEE 802.1Qbv based gating scheme implemented in our prototype setup for de-jittering purposes, as described in the next section.
4.2. Dejittering of the Transmission
In
Section 4, the scheduled traffic and traffic shaping for the integrated 5G and TSN communication pipeline has been shown, especially the shaping of the PC output and the shaping of the over-the-air transmission time. To validate the shaping of the output, measurements with cyclic data traffic and a fixed message size have been performed. A message size of 50 bytes and a cycle time of 2 ms have been tested with at least one hundred thousand samples. To determine the performance of the 5G-TSN bridge independent of the PC performance, the jitter has been measured at the output of the PC sending the data, the egress gate of TSN-Switch 1 and the egress gate of TSN-Switch 2. The performance of the URLLC testbed can be found in
Section 3.2.1.
The shaping of the PC output can be seen in
Figure 12. On the left side, the jitter of the PC output is shown, and on the right side, the jitter at the egress gate of TSN-Switch 1 is shown. Since the PC is not a dedicated real-time system, the computation of the application and the cyclic sending of the data generates a certain output jitter that can go up to 50 µs and with a 99.9% value at ≈21 µs. Using the TSN-Switch 1 to reduce the jitter by holding the messages for a certain time and allowing the PC to send within a certain time window, the probability distribution of the messages is reduced to a 99.9% value of 125 ns, as shown on the right side of
Figure 12. This is a significant reduction in the jitter caused by the PC, thereby enabling deterministic communication with narrow time arrival windows. Especially for fieldbus systems or scheduled traffic in industrial Ethernet networks, the time windows are short to enable fast communication. Our empirical results signify that without requiring expensive real-time hardware, factory cloud and edge systems could satisfy real-time application requirements with appropriate traffic shaping.
In
Figure 13, the measurement results of the integrated 5G and TSN communication pipeline without IEEE 802.1Qbv are shown. On the left side, the output jitter of the PC is shown with a 99.9% value of ≈35 µs, and on the right side, the jitter measured at TSN-Switch 2 is shown with a 99.9% value of ≈373 µs. Considering the latency values of the 5G-URLLC-testbed presented in
Section 3.2.1, it can be seen that the jitters of the PC and the URLLC-testbed are accumulated in the pipeline. The overall end-to-end jitter is increasing with each node in the communication chain. As a consequence, in a large-scale network without traffic shaping mechanisms to reduce the jitter, such as traffic scheduling according to IEEE 802.1Qbv, significant jitter can accumulate over the network and make it possibly unsuitable for certain jitter-sensitive industrial usage.
In
Figure 14, the measurement results of the integrated 5G and TSN communication pipeline with IEEE 802.1Qbv scheduled traffic are shown. Again, the PC is introducing a jitter with a 99.9% value of 16 µs but the egress jitter of TSN-Switch 2 is reduced to a 99.9% value of ≈920 ns. With the reduction in jitter, it can be seen that not only the jitter introduced by the PC but also the jitter introduced by the 5G system is compensated by the traffic scheduling mechanism. Using IEEE 802.1Qbv, the wireless 5G bridge can be integrated in the network in a way that the value of the transmission jitter is similar as in a wired connection; see
Figure 12. By decoupling components introducing larger jitter with TSN switches, the jitter can be significantly reduced, making 5G suitable for jitter-sensitive deterministic communication for industrial usage. Compared to
Figure 13 with the 99.9% value of the jitter being at 372 µs, the 99.9% jitter over the integrated 5G and TSN network can be reduced by a factor of 400 to only 922 ns by applying IEEE 802.1Qbv, as shown in
Figure 14. These results clearly show the potential of 5G integrated with TSN for the industrial usage.
4.3. Evaluation of the Implementation Regarding the Use Case
In
Section 2, we discussed the motivation behind the use case and utilizing 5G in combination with TSN for cooperative tasks between mobile robots. As mentioned before, the reliable real-time communication between the factory cloud system and the mobile robots is vital to enable the synchronized movements of mobile robots. The factory cloud system calculates and transmits the Robot Operating System (ROS) based commands related to the path planning, manipulator movement, obstacle avoidance, and advanced AI/ML algorithms for object detection to the mobile robots. The factory cloud system transmits control commands to the mobile robots periodically with the cycle time of 7 ms with the packet size ranging from 32 to 80 bytes. Since control commands are critical, the PLC is configured such that the data should arrive at the receiver side within a latency bound of 6.3 ms (90% of 7 ms cycle time). Otherwise, the mobile robot will come to a halt, since the controller raises a critical error due to the safety requirement. Hence, communication between the factory cloud system and the mobile robots—including the TSN network and the 5G system—needs to fulfill this upper latency bound. As measurement tests indicate, the 5G system integrated with the TSN network provides a latency below 0.8 ms with high reliability. Such reliability ensures that the critical traffic is transmitted within the required time window for the use case.
In our use case, mobile robots transmit their LiDAR data and camera data to the factory cloud system. LiDAR data are classified as a medium traffic. LiDAR data periodically transmit data to the factory cloud system every 25 to 100 ms (which is configurable) in order to provide essential data for obstacle avoidance and emergency stop. LiDAR data are part of the uplink communication in our 5G and TSN setup with the required throughput of around 1 to 3 Mbps. LiDAR data are transmitted to the factory cloud system in real time with high reliability within the latency below 1 ms. On the other hand, camera traffic is considered as background traffic with large packet sizes, which requires higher throughput compared to LiDAR but without strict latency requirement on the wireless communication. The challenge for the camera data is the high throughput, which is limited in the uplink direction by the uplink capacity of around 150 Mbps. In the use case, the resolution of the camera is reduced to keep its throughput within 20 to 50 Mpbs while not yet degrading the performance of AI/ML-based object detection running on the factory cloud system. Based on the measurement test for the data packet size of 1042 bytes, the maximum latency for over-the-air communication stays around 1 ms, which is much lower than what is required for data transmission of the camera.
The coordination of mobile robots demands 5G and TSN to fulfill the traffic requirement of both mobile robots at the same time. In this case, the robot controller in the factory cloud system receives the necessary data from both mobile robots and schedules the movement of each of the mobile robots to execute the coordination tasks. The challenge of coordination tasks in our use case is syncing the movement of mobile robots specially when they handle sensitive materials such as glasses, since any jitter that causes the latency of communication between the controller and the mobile robot to go beyond the maximum latency bound causes one mobile robot to stop operating. In this case, the robot controller in the factory cloud needs to transmit a halt command to the other mobile robot within 7 to 10 ms; otherwise, the glass will break. The aforementioned time period includes any processing time of factory cloud system, which needs to be accounted for determining the permitted latency for the 5G system to ensure the safe operation of mobile robots in coordination tasks. During the measurement test period, we have not experienced any latency exceeding the maximum latency bound, and both mobile robots stayed in operating mode during the execution of the task during the experimentations.
5. Conclusions
In this paper, a prototype 5G system integrated with a TSN network has been presented for a typical industrial mobile robotics use case. The prototype consists of a 5G URLLC test system and commercially available TSN switches. Different TSN protocols were integrated with the 5G system for deterministic over-the-air communication. The paper describes the architectural details and step-by-step analysis on jitter reduction mechanisms applied in the communication chain between the application hosted in a factory cloud and the mobile robot. The selected use case of a mobile robot will play an increasing role in manufacturing in the future to increase production flexibility by, e.g., introducing line-less production systems. It puts high requirements on communication with regard to low latency, high reliability and low jitter. The experimental validation took place in an industrial shopfloor environment. In the context of 5G system integration with TSN, these are the first measurements taken in such an environment.
With the TSN standards used, the performance results of the prototype satisfies the requirements of the use case, enabling the wireless control of mobile robots. Time synchronization over the 5G system can be achieved at high precision; the time error introduced by synchronizing over the TSN network—and including the 5G system—was observed to be lower than 8 µs with a mean value of below 3 µs in our experimental validation over the test period of more than 6 h. On the industrial shopfloor, the 5G URLLC test system demonstrated time-critical communication performance; we observed an end-to-end latency over the TSN network and the 5G system that was below 0.8 ms with a 99.9% reliability. However, a jitter in the transmission over the 5G system in the order of 500 µs was experienced, which is due to the characteristics of the 5G design and the wireless characteristics. We studied an end-to-end TSN configuration using TSN time scheduling according to IEEE 802.1Qbv, where all network nodes are synchronized to a common reference clock. One key measurement result is that the jitter introduced by different entities in the communication path can be isolated and compensated in a TSN-based network, reducing the overall jitter end-to-end. In particular, by having an IEEE 802.1Qbv-configured TSN bridge located after a node with relatively large jitter, the IEEE 802.1Qbv-configured bridge can compress this jitter. In our experiments, we could reduce the end-to-end jitter over the TSN and 5G network from ≈370 to ≈0.9 µs through IEEE 802.1Qbv configuration in the network.
From our test setup and validation, we conclude that 5G, and in particular when integrated with an industrial TSN network, is capable of supporting demanding industrial automation use cases. We have implemented and validated this for the example of cloud-controlled mobile robotics in an industrial shopfloor environment.