A 5G-Based VR Application for Efficient Port Management

Abstract

In recent years, the throughput of cargo ports has increased rapidly. It is urgent to improve the operating efficiency of ports for its increasing demands. Both industry and academia have shown great interest in adopting 5G and Virtual Reality (VR) technologies to improve the efficiency and safety of industrial operations. However, such technologies have not been well explored in port operations. This paper proposes a 5G-based VR smart port to support intelligent management for five typical port operations. The overall architecture of the smart port and its key processes, major advantages, and limitations are discussed in this paper. An application of the proposed smart port system is demonstrated. The performance study based on network Quality of Service (QoS) and Quality of user Experience (QoE) proves the feasibility of the proposed architecture. The architecture supports efficient interaction in real-time, making comprehensive decisions, and staff training. The smart port architecture is suitable for tasks of high working intensity and can dramatically increase operational efficiency.

1. Introduction

With the development of the shipping industry, the number of ships sailing abroad has increased dramatically in the past few years. At the same time, the governments of port states have to face severe maritime security situations brought by foreign ships. In particular, large cargo ships and cargo trucks can cause accidents due to bad weather and human factors. What is more, it can endanger the lives of people and cause large-scale economic losses [1].

International ports have a high demand for smart applications to save cost and increase productivity when they serve as an important hub of international trade, logistics, and global supply chain [2,3]. The requirements of a smart port include autonomous loading and unloading, all-around visibility, intelligent management and control, intelligent decision-making, and flexible collaboration [4,5]. The key to addressing the requirements is to enable comprehensive perception and extensive interconnection of the smart port. The traditional smart port mainly relies on physical camera equipment to support the monitoring of terminal information and a closed-circuit television (CCTV) network to obtain local information [6,7,8]. The action for emergencies relies on remote operation through the information gathered from the cameras or local operation by nearby workers. However, both solutions suffer a long latency because of communication delay or visual impairment caused by bad weather. Late action will make the port face the potential risk of economic loss such as damage to machines. The smart port requires a short delay, large bandwidth, and high reliability for the perception and the requirements of flexible operation methods to react to emergencies.

The in-depth integration of a new generation of information technology (e.g., the Internet, big data, cloud computing, artificial intelligence, etc.) and industrial enterprises will promote the industrial digital transformation. With the commercial deployment of the 5G networks, Mobile Edge Computing (MEC) has become a promising infrastructural technology to support various business models and applications for many small companies in the industry [4]. MEC platforms integrate computation and communication technology, which have greatly empowered vertical industries by exploring their new applications in vertical industries [9,10,11]. By pushing computation services close to the end-users, MEC can better satisfy the needs of various vertical application domains for high bandwidth, low latency, high computing power, and high security [4,5,6].

Novel applications over 5G such as virtual reality (VR) [12,13] bring new opportunities to smart ports [14,15]. VR-based smart port applications emerge to promote efficiency or sustainability for transportation services [15]. They provide a 360-degree panoramic picture, allowing users to immerse themselves in the scene, with a better sense of space and distance, to achieve a truly immersive interaction. However, the existing attempts on such kinds of applications can only provide very limited functions. Their application architectures are still more cloud-based, which fails to unleash the full potential of the 5G infrastructure. For example, many roaming requirements in a smart port, such as interactive repair of local equipment and remote multi-person collaboration, will need frequent interaction with the local environment. Cloud-based architecture can easily cause network congestion when multiple VR hardware devices keep requesting services such as positioning and model rendering. In addition, heterogeneous equipment in a smart port requires extra communication costs to achieve remote control, which will eventually affect the Quality of Experience (QoE) for the users. The encryption process for data transmission will also cause a latency increase.

This paper proposes a novel smart port application that is enabled by 5G and virtual reality (VR) technologies. The proposed smart port application allows users to obtain an immersive experience in a completely virtual environment while still achieving operations such as grabbing and moving through a VR handle. To leverage 5G technology to enhance mobile broadband for the VR-based application, we proposed an Edge-Cloud architecture for VR applications. It extends the traditional cloud-edge-end model [4,16,17] by a domain-specific topology for smart ports.

This paper demonstrates a series of applications of smart port in assisted driving, remote manipulation, intelligent security, emergency systems, and intelligent cargo management systems. Specifically, this paper aims at ship information visualization, modeling and simulation on ships, and applies the simulation data to the visualization of ship information. Taking a large cruise ship as an example, we established a model of the cruise ship and cabin and a virtual ocean scene. The traditional elements of the terminal such as cranes, cargo, trucks, and containers were visualized using the Unity3D virtual reality engine [18]. The end devices are connected to the Huawei VR Glass. The operation of a container crane is with high precision requirements in port operations, and the rash operation by unskilled workers will bring certain risks [19,20,21]. We built a three-dimensional (3D) virtual environment of each board of the port, which can visualize the operation process planning of the port. It can greatly optimize the human labor cost, time consumption, and safety level without the requirement of a specific field/room in the training of machinery operation and port operation process, ship operation, etc. The system can move the experiments, practical training, and other skills training to the virtual environment. The proposed application can also aptly demonstrate complex, abstract, internal structures, and phenomena that are not suitable for direct observation. They show the operation process in an all-around and multi-angle way and create an interactive virtual scenario to provide viewers immersive experience, thus dramatically enriching the intelligent development of the port.

The problem this paper tries to solve is to study the feasibility of adopting 5G and VR in smart ports. This paper not only shows the possibility of smart port applications based on 5G + VR, but also demonstrates its reliability and QoS constraints in real-world smart port operations.

The biggest difference between the system architecture proposed in this paper compared to other port systems is the immersive construction based on a real port in a VR environment. The system proposed in this paper can be used to assist inexperienced operators to obtain familiar with various scenarios. It can also be used to concatenate information from various departments in the terminal so that operators at remote locations can arrange paths, plans, etc.

The main problems with current VR technology are the high latency, high data volume, and low video quality that cause poor user experience. With these disadvantages, traditional VR technology cannot be applied to the industry such as port and remote-controlling. However, thanks to the development of 5G technology, the network data transmission rate, packet loss rate, and reliability are greatly improved compared to 4G. It provides the possibility to use VR technology in industrial scenarios, to improve efficiency, safety and lower human labor cost. This paper performs an elaborate analysis on data transmission, packet loss, and reliability in VR applications under 5G networks. The specific experimental verifications are listed in Chapter 4 of this paper.

The problems addressed in this paper are as follows.

• A new 5G + VR smart port architecture is proposed, and the feasibility analysis is carried out under certain conditions through the upload rate, packet loss rate, and stability.
• The paper constructed a set of physical systems and simulations of autonomous driving and remote-control equipment in a 5G + VR environment, and further demonstrated the feasibility of this architecture by practical operation. However, not all applications mentioned in Chapter 3 has been fully implemented. More performance studies are still required to rigorously verify every aspect of the applications, as well as promote new possible applications using 5G + VR in smart ports.
• This paper conducted a set of QoE subjective tests that analyzes the advantages and disadvantages of this system architecture performed by 35 participants. The results can provide clues for the future improvement of such systems.

2. Smart Ports’ Demand Analysis

2.1. G Features and Application Scenarios

The International Telecommunication Union (ITU) has defined the three major application scenarios of 5G as enhanced mobile broadband (eMBB), massive Machine Type Communications (mMTC), and ultra-reliable low latency communications (uRLLC). eMBB mainly provides much higher bandwidth than that of 4G. mMTC is mainly to meet the communication needs of a large number of Internet of Things (IoT) nodes, and is oriented to application scenarios that target sensing and data collection. uRLLC is based on 5G technology with low latency (1 ms) and high reliability (99.999%) characteristics, mainly for the special application needs of vertical industries such as smart ports.

With high speed, multiple connections, and low latency, the 5G network can contribute to the development of the digital economy and is an important foundation for promoting the implementation of “a Community of Shared Future for Mankind” strategy [21,22,23]. The 5G network will be extremely conducive to creating a cross-industry integrated ecosystem. The 5G network is rapidly gaining popularity in smart transportation, smart manufacturing, smart education, autonomous driving, and so on. Among these, the typical applications of enhanced mobile broadband are 4K/8K/3D high-definition video, VR, augmented reality (AR), etc. The ultra-low latency and ultra-high bandwidth of 5G solve a series of users’ dizziness and unclear frame problems.

As an extension of the 4G technology, 5G has the advantages of higher speed, larger capacity, and lower delay, and is an important carrier to support the interconnection of everything. 5G technology comprehensively uses key technologies such as large-scale antennas, ultra-dense networking, flexible duplex, full-duplex, and spectrum sharing. In addition, 5G also uses network slicing [24] that allows the creation of a logically independent network on general physical information infrastructure.

5G network technology can support the mixed bearing of industrial control signals with millisecond-level delay and multi-channel high-definition camera video data. 5G network technology can give full play to the differentiated performance advantages in the construction and operation of smart terminals, breaking through the limitations of traditional methods. What is more, 5G networks significantly improve the operating efficiency of automated terminals and inject innovation power into the transformation of traditional artificial terminals.

2.2. Edge Computing

MEC is the foundation of 5G technology and is indispensable for smart ports to make real-time interactions with the real world through VR. The schematic diagram of the edge computing capability network topology is shown in Figure 1. MEC provides multiple network access, computing platforms, and edge computing platform capabilities for industry users to support the digital transformation of industry customers.

(1)

Network connection. MEC, based on 4G/5G converge, can provide high-bandwidth, low-latency network access services for industry and enterprise customers. At the same time, it can be extended to provide the existing wired, narrowband Internet of Things (IoT) and other network connections in the park through plug-ins.

(2)

Computing platform. Based on the general X86 server architecture, mobile edge computing provides an open-source cloud computing management platform and a container-based multi-core cloud management framework. It adopts a lightweight platform and provides different forms such as virtual machines, bare machines, and containers for industry applications.

(3)

Edge computing platform capabilities. We further superimpose the convenience of wireless communication and Platform as a Service (PaaS) capability, providing wireless network information services, Quality of Service (QoS), bandwidth management, DNS processing, routing distribution, and other network capabilities. At the same time, it provides edge application capabilities such as application lifecycle control, application registration, routing policy management, indoor positioning, and video rendering for third-party applications. It also provides Application Programming Interfaces (API) to open network capabilities and services to the outside world. PaaS is a software development platform as a service, and the development environment is relatively perfect. However, the application supported by PaaS is limited and lacking in universality, and the enterprise has no access to the underlying business logic. As a result, it is difficult for terminal devices providers to develop PaaS, which cannot meet all the needs of the enterprise.

MEC provides edge cloud Infrastructure as a Service (IaaS) platform service, which can host third-party services. The IaaS edge cloud includes the underlying hardware, servers, storage, and other equipment. The virtualization layer uses an open-source cloud computing management platform architecture and a standardized x86 architecture underlying server. The edge cloud IaaS platform adopts lightweight resource management Virtual Infrastructure Management (VIM), and only basic components are retained. The consumption of physical resources is reduced by reducing the number of component service worker threads. The computing unit supports the dual-core solution of virtual machine and container to meet the different resource requirements of supporting applications and facilitate rapid deployment and upgrades. The edge cloud is small in scale, the bearer business is mainly local processing, and the required storage capacity is small, whereas either local storage or distributed cloud storage can be adopted.

IaaS offers computing infrastructure (servers, networking technology, storage, and data center space) as a service to customers. It includes providing operating systems and virtualization technologies to manage resources. Consumers can obtain services from a sophisticated computer infrastructure over the Internet. IaaS is more versatile and less difficult to develop than PaaS. However, IaaS is not economical and more expensive to maintain. Therefore, in our scenario, IaaS and PaaS were both used to save costs as much as possible on the premise of industrialization. The relationship between IaaS and PaaS is shown in Figure 2:

2.3. VR Development

Virtual reality technology originated in the 1960s and has become a technology of concern to the scientific and engineering communities [8,9]. It involves multiple disciplines, such as computer science, mathematics, mechanics, acoustics, optics, mechanics, biology, and even aesthetics and social sciences [10]. Essentially, the 3D virtual environment allows people to feel the immersive space with natural functions such as sight, hearing, touch, and smell, just like being on the scene. In this way, people can use the abilities and methods that people are accustomed to observing, analyzing, manipulating and controlling the generated objective world, and, finally, immerse in it. VR is not creating a real-world, but an alternate environment. People can enter and interact with the environment through various media of the computer. As a systematic technology, VR has already made a qualitative leap in the way of thinking. It does not consider the problem from only one aspect as in a single technology but requires all components to pursue the best overall performance of the system in content acquisition, processing, transmission, and rendering processes.

VR applies many next-generation information and communication technologies such as near-eye display, perception interaction, rendering processing, network transmission, and content production, and building a virtual immersive and interactive environment for users through somatosensory interaction. It constructs a business form spanning the cloud-edge-things continuum [4] to meet the needs of the immersive movement of the user’s viewport, and to promote information consumption. An effective virtual realization experience in the smart port will make port operators unaware of the real environment and focus on the existence of objects in the virtual environment. The use of VR technologies can help improve the effectiveness of the virtual reality experience and achieve system goals. The pivotal technologies of the 5G network include mobile edge computing, network slicing, and large-scale antennas, which can provide QoS guarantees for smart ports. Smart ports based on a 5G network and VR can provide a variety of different business opportunities.

3. Application of 5G + VR in Smart Ports

The average latency of VR under a 5G network is less than 20 ms in real-world tests, which provides the possibility for many different smart port applications [23,24,25,26]. To further explore the feasibility of VR for smart ports, we built the entire virtual environment with Unity3D and operated a VR terminal under 5G network conditions for normal operations. Various sensors, such as temperature and motion speed, were used to inter-communicate information with the terminal, and various weather and adverse conditions were also simulated for the terminal staff to learn the operation. After the training of terminal operations in the VR environment with minimum cost, operator errors can be effectively reduced, and the risks caused by human factors at the terminal can be significantly reduced. Based on the verification and improvement of comprehensive simulation, we propose the following “5G + VR Smart Port” system architecture.

3.1. Key Process

The 5G + VR smart port framework proposed in this paper has two VR modes as shown in Figure 3, where different modes can be selected depending on the network QoS and application requirement.

In Mode 1, the virtual reality is reconstructed by building every real-world model in the virtual scene, in which real-world related parameters such as movement speed, angle, and collision are rendered through Unity3D. Mode 1 is suitable for limited networks with low bandwidth, as only the interaction commands are needed to be transmitted between the real world and the virtual scene. Its latency is typically more than 15ms in a 4G network environment and can be reduced to less than 5ms in a 5G network environment while having a 0% packet loss rate.

In Mode 2, as shown in Figure 4, the immersive VR experience is achieved by real-time video captured by the 360° camera and transmitted through the 5G network. The terminal operators can observe the real-time 360° video of the captured scene by wearing a VR device or using a PC monitor. In this way, the VR experience is degraded from the original 6 Degrees of Freedom (DoF) to 3 DoF, where the viewers are limited to a 360° view at a fixed point in the remote scene without the feasibility to move around. This is due to the extremely high data volume (about 10 times that of 3DoF) and video rendering computation capability limitations in implementing 6 DoF VR video streaming. According to the QoE evaluation mode, results show that the video resolution in the VR environment must reach at least a 4K resolution level for the viewers to avoid feeling fatigued when using for a long period (e.g., 15~20 min). This mode requires a high transmission rate (e.g., above 20 Mbps), and the latency is generally 20ms under the 5G network, which is fully supportable for normal work in most departments of smart ports. For some specific scenarios, for example, assisted driving, higher safety can be achieved by having an ultra-low delay in 5G slicing networks. This requires higher-speed networks or edge deployment of 5G with theoretical latency of 5ms or even less than 1ms. The main process of Mode 2 consists of three stages: video capture, transmission processing, and video playback.

(1)

Video capture: Panoramic cameras are used to shoot panoramic images, connecting to the base station through a 5G CPE (Customer Premise Equipment), 5G mobile phone, or the 5G module that is equipped with a camera, and they are responsible for uploading the generated RTSP or RTMP format video stream through the 5G access network.

(2)

Transmission processing: The 5G bearer network guarantees the port video content through the “virtual private network” to ensure that the 360° video stream is smoothly transferred to the content platform located in the cloud. After the content platform performs a series of operations such as local storage, resolution conversion, coding rate conversion, and encapsulation format conversion on the live 360° stream, it will generate a new 360° video stream and transmit it to the terminal through the bearer network. In a traditional network such as WLAN and 4G, the video streaming quality is easily affected by network fluctuations and its QoE cannot be guaranteed for remote control. Industrial 5G networks can provide a slicing network that preserves sufficient network resources for steady 360° video streaming with a high QoE level.

(3)

Video playback: 360° video streams are transmitted to TVs, mobile phones, VR helmets, and other terminal devices through the 5G base station downlink channel for further decoding and playback. The requirements for video stream bit rate, resolution, and other parameters obtained by different terminal devices vary according to their computation capability, network connection, screen size, application scenario, etc.

Combined with 5G’s multi-access edge computing network elements, cloud processing can sink to network edge nodes and be implemented on edge servers. When the 360° video stream reaches the mobile edge server of the 5G network, the edge server transcodes the video stream content in real-time according to the current users’ viewing orientation, device hardware configuration, and network bandwidth status to provide the highest possible QoE level to the end-user. For example, User A is currently facing the front, the terminal device is a dual-eye VR all-in-one device and the downlink network bandwidth is greater than 1 Gbps, then the mobile edge computing server will perform 360º tile-based transcoding on the current received 360° video stream. The video is transcoded to produce video segments with only a 110° × 120° area directly in front, and transmits to the user device for playback. When the user turns his head, the server will transcode the content that needs to be played on the screen in real-time according to the viewing direction, and deliver it to the user’s terminal device.

Additionally, Mode 1 and Mode 2 can be adopted simultaneously. The main VR scene is built based on Mode 1 and a limited amount of 2D video or the 360° video viewport center is transmitted via Mode 2 to provide real-world tracking information in a small window within the virtual scene.

3.2. System Architecture

The overall architecture of the “5G + VR Smart Port” system consists of four layers, including the terminal device layer, network layer, cloud layer, and platform service layer. The terminal layer includes high-definition cameras, VR helmets, smartwatches, and smart devices, to support services such as remote access, video capture, scene capture, action recognition, and operation presentation. The network layer includes 5G networks, intelligent IoT components, providing data connections for various business scenarios [22,23,24,25]. The cloud platform provides extended network and computation capabilities for complex tasks using cloud computing technology. The platform layer includes various components such as broadcast control, content delivery, and mobile edge computing to enable port business control, cargo distribution, video analysis, and processing. Through VR, the scene in the port can be integrated virtualization, and the virtual port environment can be very intuitive. It can not only simulate various weather conditions well but also understand drainage systems, power supply systems, road traffic, and so on, at a glance. In the virtual environment, a variety of business drills can be comprehensively simulated, which can increase the fault tolerance rate of the smart port when completing various tasks. By controlling the VR handle or a mouse connected to a PC, the staff can control trucks and cranes to move goods to any warehouse efficiently. According to the business characteristics of smart ports and the requirements for transmission rate and delay, the major applications of 5G + VR in smart ports include autonomous driving, remote control of mechanical equipment, smart security, and emergency command, which are discussed elaborately in the following context. Figure 5 shows the smart port architecture.

The system architecture first utilizes various types of sensors to sense and pre-process environmental information, which provide basic data support for positioning and complex environment awareness in the port, These sensors include intelligent beacons, as well as video surveillance, radar, satellite positioning, VDES, and other means equipped on autonomous navigation vessels, trucks loaded and allocated around smart port park. When facing some severe visual impaired conditions such as rain and fog in the port, the system may suffer temporary loss of visual information. The cumulative error of the Inertial Measurement Unit (IMU) is corrected by fusing millimeter wave radar, vision and BeiDou satellites. The multi-sensor based localization of multiple targets (e.g., obstacles, ships, bridges, etc.) can be used to solve ambiguous localization and generate target localization information in complex waters. This information is then integrated with systems based on VDES, radar, and AIS to generate information of fairway traffic and other vessels. In order to meet the needs of smart ports for comprehensive and accurate perception of complex environment, multi-sensors are combined with the hydrological and meteorological information released by air, sky, land, and water communication systems based on wireless mesh network, 5G, Beidou, and VDES to carry out cross-validation. Finally, it can provide a sensory transmission network with full coverage of the complex environment of the smart port.

This paper focuses on the verification and analysis of these applications’ feasibility. We have realized the model of autonomous driving, remote control, intelligent security, emergency management, and intelligent cargo management system in VR. Particularly, this paper implemented and performed an in-depth analysis on the remote control of mechanical equipment, controlling remote PLC to grasp cargo through VR under 5G network. The applications are described in detail as follows.

• Autonomous Driving in Smart Port

Autonomous trucks in smart ports can maintain depth perception, accuracy, and effectiveness even under terrible working conditions such as rain, snow, and night. Autonomous driving can improve operational efficiency and safety, and reduce human labor costs and human involvement. However, in the current port development, the coverage of intelligent road facilities is still very limited. It lacks network facilities such as road-side communication systems to support the interconnection of vehicle-to-vehicle and vehicle-to-pedestrian. The road condition cannot meet the requirements of autonomous driving.

For the relatively closed scenario of the smart port, it is easy to virtualize the scene, which makes remote controlling an autonomous truck possible. The autonomous truck is based on 5G + VR technology to drive on a specific road in a closed industrial park [16,27,28]. Specific routes are planned in smart terminals remotely by human manipulators, and autonomous trucks can load and deliver goods by themselves, which greatly reduces the flow of personnel and improves operational efficiency. As the information of each module of the port can be stored and visualized in the reconstructed VR environment, it is more convenient and faster to help the manipulator to learn and carry out technical route planning. Data collected by multiple sensors deployed in real-world road facilities can be mapped in a VR environment using the MQTT network interface through an industrial 5G network that achieves real-time communication between vehicles, people, and roads. The specific display interface is shown in Figure 6 below.

The remote manipulator can monitor the movement of the truck through the driver’s perspective and supports the real-time retrieval and visual reading of the information of each part of the smart port, thereby greatly improving the work efficiency. Port environmental information is collected and sent to VR systems from warehouses, trucks, cranes, etc., which can make the remote working principle more comprehensive. Such information can also support overall planning and task orchestration for smart ports, which can greatly improve the working efficiency of port tasks. 5G networks’ high speed, large capacity, and low latency features can enable smart port autonomous driving with timely and high reliability of the transmission of remote-control commands for driving trucks. This supports the need for manual remote control of autonomous driving trucks under abnormal conditions and achieves L3 or L4 automatic driving, automatic guidance, precise parking, automatic loading, and other functions in various port scenarios. Autonomous trucks are seamlessly connected with the remote control of quay cranes and quay bridge mechanical equipment. By integrating with the information for intelligent port machinery and equipment, material, and environment, autonomous driving can better understand the scenario where it resides and make better decisions. Remote manipulators can make decisions based on the feedback in the port and the vehicle condition of the autonomous driving truck to complete a series of tasks, thereby supporting the fully automated operation of the terminal. The remote-control scheme of autonomous driving trucks is shown in Figure 7.

• Remote Control of Mechanical Equipment

Controlling quay cranes and yard bridges remotely can greatly reduce human labor, improve operational efficiency and safety, and improve working conditions for the operators in a smart port. This requires the remote-control communication system for quay cranes and yard bridges to satisfy the synchronization and reliable transmission of control data. The communication system for remote control of quay cranes in traditional ports can only rely on an optical fiber network, which is not only at the risk of a single point of failure, but also hard to deploy for remote-control cranes. Taking advantage of the high speed and large capacity of the 5G network, the data connection between quay cranes, yard bridges, and remote-control centers adopts a mix of 5G and wired networking methods, which can improve both reliability and flexibility of network transmission [17,29,30]. The system combines the terminal operation management system, the quay crane automatic control system, and the quay bridge automatic control system to enable the remote-control operation of the quay crane and the quay bridge.

• Smart Security

The smart security system is one of the important components of the smart port. The system transfers various parameters of sensors in real world to the VR environment not only for storage but making a comprehensive evaluation for risk avoidance decisions. The parameters transmission models have been described in Section 3.1.

By setting up video surveillance equipment in areas such as terminal berths, storage yards, gates, roads, and inspection areas, real-time monitoring of quay bridges, yard bridges, personnel, vehicles, containers, and boundaries is achieved. Traditional smart ports cannot detect port information with high accuracy in bad weather conditions, especially in the case of damaged cameras. The smart security system proposed in this paper reconstructs the entire port environment in VR to avoid the security risks existing in traditional intelligent ports. With different types of IoT sensors, the VR system can effectively prevent inaccurate monitoring data of smart ports due to camera aging, sight blocking, or bad weather conditions. Considering the instability of the port’s equipment, i.e., the potential risk caused by inaccurate data, a VR environment can simulate weather condition changes and equipment damages to predict the potential safety hazards. Through the combination of physical monitoring and VR, various risk events of the port can be more accurately grasped, which can support higher precision operation. Most existing port’s end devices of their video surveillance application can transmit data through the port optical fiber network system, but for areas that do not have wired network access conditions, 5G network transmission can be adopted. The smart security content is shown in Figure 8.

• Emergency Response

The traditional intelligent port security system is mainly supported by end devices through real-time monitoring, but this method cannot support emergency response, such as disaster warning and disaster prevention [31,32,33,34]. The security system of smart port in this paper is based on 5G + VR, which can effectively avoid losses caused by monitoring equipment damage in the event of accidents. As shown in Figure 9, end devices such as drones can be transmitted back to the smart port emergency command and dispatch center to enable real terminal scenes synchronized in one screen. By simulating the status of each part of the ports in real-time with VR, an operator can simulate the decision in response to an accident under any harsh conditions. The operator can also plan the information collection scheme of the whole port through VR, which is not available in traditional intelligent ports. This greatly improves the flexibility of terminal commanders to observe and judge emergencies in the port area and improves the efficiency of decision-making and command execution. 5G + VR can support the response of delays at a millisecond-level for video surveillance data, and the staff can control the wharf warehouse information, truck running condition, logistics information, and make adjustments in real-time. Once a problem is located in the port, the most appropriate contingency plan can be adopted immediately, while minimizing the risk and potential loss. Briefly, the intelligent security system proposed in this paper is mainly used to simulate the user’s decision when danger occurs, as well as the discovery of potential risks, integrated information from all aspects to make comprehensive decisions to reduce human labor and time costs.

• Intelligent Container Tally System

The intelligent container tally system is mainly composed of an identification system, a transmission system at the edge of the network, and a management system at a higher level of the network [35,36,37]. From the implementation perspective, as shown in Figure 10, the recognition system and the management system are only different at the technical level, and usually not a bottleneck. The transmission system is a key of the QoS of an intelligent container tally system. Most of the quay cranes of the traditional wharf are not equipped with optical fiber for data transmission, and can only transmit the signal of the identification system to the back-end through a wireless network [20]. From the operating point of view, the limitations of the wireless Access Point (AP) solution are mainly manifested in the following three aspects:

• Wireless AP’s limited roaming support makes the quay cranes easy to experience packet loss in handover processes during their movement.
• The maintenance effort of the wireless AP increases the cost of the terminal.
• The overall QoS (i.e., bandwidth, packet loss, delay, jitter, etc.) level of wireless AP cannot simultaneously carry the transformation business requirements of quay crane tally and quay crane remote control.

Judging from the technical characteristics of 5G and the current industry application demonstrations, 5G communication technology can better solve the above problems. eMBB is the enhanced mobile broadband of 5G to provide support for the high bandwidth required for both intelligent container tally system and quay crane remote control. At present, the measured speed under the No-Standalone network Architecture (NSA) can reach 1.2 Gbit/s. At the same time, 5G’s high-reliability and low-latency connection can provide a cost-effective solution for the remote-control transformation of traditional quay cranes.

A customized 5G industrial router can be installed on each quay crane to customize 5G access terminals so the data on the quay crane can be transmitted to the back-end management platform through the 5G network. The system can be extended to support more complex tasks such as cargo management, route planning, and time management of cargo ships through all aspects of gathered information.

3.3. QoS Based System Optimization

The core QoS requirements of vehicle communication service types are classified in

Table 1. Vehicle communication service classification.

Business Types	Business Characteristics	Core QoS Requirements
Vehicle positioning and posture Beacon Traffic status	Low latency Low upstream and downstream rates Small data packets Intermittent transmission, resulting in idle channels	Upstream: 100 kbps Downlink: 150 Mbps 20 ms end-to-end delay
Video surveillance Radar signals Ship vision signals	Uplink high speed Large data packets Continuous transmission Long distance from the base station brings video quality degradation	50 Mbps uplink
Vehicle collaboration Control system	Low data rates Small packets Intermittent Large number of connected nodes Random channel accesses bring low channel utilization	Random channel access Massive connections

. Through QoS tracking and prediction, the system can dynamically determine the optimal network path, select transmitted data, and allocate channel resources for different types of businesses. Based on the characteristics of the truck business and user QoE requirements, we propose optimal path planning model that is shown in Figure 11.

According to the core QoS requirements, we perform QoE tracking and modeling for each communication link of the specific application. Using the set E to represent the set of links in the network, for any one transmission link e belonging to E, four attribute functions are defined, transmission link delay function delay(e), cost function cost(e), transmission link available bandwidth function bandwidth(e), and transmission link delay jitter function delay jitter(e). For any node n, four attributes are defined, the node delay function delay(n), the node cost function cost(n), the packet loss function packet loss(n), and the delay jitter function delay_jitter(n) in the node. For some given source node s and destination node d belonging to the set of nodes, Pr(s,d) is the transmission path between the source node s and the destination node ground. The QoE constraints to be satisfied are shown below.

$$f_1\left(P_r\left(s,d\right)\right)={\displaystyle \sum }_{c\in P_r\left(s,d\right)}delay\left(e\right)+{\displaystyle \sum }_{n\in P_r\left(s,d\right)}delay\left(n\right)\le M_1$$

$$f_2\left(P_r\left(s,d\right)\right)=min\left\{\mathrm{bandwidth}(e),e\in P_r\left(s,d\right)\right\}\ge M_2$$

$$f_3\left(P_r\left(s,d\right)\right)={\displaystyle \sum }_{e\in P_r\left(s,d\right)}\mathrm{delay}\text{\_}\mathrm{jitter}(e)+{\displaystyle \sum }_{n\in P_r\left(s,d\right)}\mathrm{delay}\text{\_}\mathrm{jitter}(n)\le M_3$$

$$f_4\left(P_r\left(s,d\right)\right)=1-{\displaystyle \prod }_{n\in P_r\left(s,d\right)}\left(1-{\mathrm{packet}}_{-}loss(n)\right)\le M_4$$

$$f_5\left(P_r\left(s,d\right)\right)=cost\left(P_r\left(s,d\right)\right)$$

M₁ represents the service constraint on the maximum end-to-end latency, M₂ represents the minimum available bandwidth of the network transmission link, M₃ indicates the maximum delay jitter that the system can tolerate, and M₄ represents the maximum packet loss rate constraint. The QoE path selection algorithm looks for a path with the smallest network cost from all possible network paths (i.e., 4G, 5G, WiFi, optical fiber, network cable, etc.), represented as Pr(s,d) if the four QoE constraint is met.

3.4. Advantages

Compared with wired broadband networks or 4G networks, smart ports with industrial 5G architecture have better support for mobility, more stable QoS, and low overhead. The equipment can be connected to the network through the 5G module, which neither of the current systems used optical fiber, video cable, and other wired connection methods that have mobility restrictions nor suffers the unstable connection provided by Wi-Fi. Compared with traditional smart terminals, it can reduce the disturbance caused by limited bandwidth, network congestion, unstable QoS, etc. 5G services provides independent network resources for the carried VR video stream and emergency data, avoiding the influence of resource preemption by other services and ensuring smooth video streaming and data transmission. The introduction of multi-access edge computing servers also saves downstream network bandwidth for users. The original 360° video is compressed into 1/3–1/2 of the original video at the edge of the network, saving the traffic cost for users. At the same time, the transmission scheme has low requirements for the configuration of user hardware equipment, which indirectly reduces the hardware cost and thus reduces the overhead.

In fact, this system requires a network structure with large bandwidth to support high volume data transmission. As described in Chapter 3, a complete interaction between the VR environment and the real environment requires continuous and fast transmission of data from multiple sensors. Thus, 5G is required to ensure the efficiency and stability of the system. What distinguishes the system proposed in this paper from traditional port systems is that we have built a whole set of sensor information networks to support the various interactive information needed in both VR environment and the real world. Most current VR port systems [38]. are limited to the implementation of a certain aspect of functionality, such as safety training for operators [39]. We have built a complete set of sensor networks and system architecture that can realize the requirements of the entire terminal assembly line operations. Under the VR environment, operators are able to make decisions based on the integrated information from all aspects of sources. It can help save more time than in traditional terminals [40], from the time the goods enter the port to the time they enter the warehouse.

4. Demonstration and Performance Evaluation

4.1. Materials and Methods

The experimental material mainly includes physical production of 3D printing materials, VR equipment, wireless network, and 5G environment.

We choose ABS plastic for 3D printing of the ship body. ABS plastic is generally opaque, with good dimensional stability, electrical properties, wear resistance, chemical resistance, abrasion resistance, molding process, and machining. ABS plastic can better adapt to the changes in various situations in a complex environment. This system uses 5G technology to overcome the latency problem of traditional VR technology. The VR perception is provided by Huawei VR glasses with 70/90 FPS (mobile mode/PC mode), 3K screen resolution and 1058 PPI. The performance of Huawei VR glasses is stable, cost-effective, and adapted to the 5G network. The wireless communication transmission technology adopts HTTPS. Through 5G transmission rate, it can ensure data transmission at high speed and encrypt and decrypt the data to prevent the transmission data from being illegally stolen. The experiments of this system are conducted in Wuhan, which is rich in 5G base stations and can provide a complete and accurate 5G network.

This experimental VR environment is built-in Unity3D, which simulates the construction of a whole modern port. Various weather conditions can be simulated, and modes such as inspection or teaching can be conducted. In addition, we make cargo ships and car models by 3D printing to create a set of simple physical models. Moreover, we also designed a user experience test and participants were given ten minutes to familiarize themselves with Huawei VR glasses, using simple interactions such as moving the objects in the VR environment, calling out, and canceling the main menu through gestures and voice. After that, detailed instructions on how to use the system will be provided for each user. When the preparation work is completed, users will start to experience the system. The experiment continued about 20 min per person, including the time to familiarize themselves with the device, conduct tests, and fill out questionnaires. Users will control the objects in the VR environment, reach the specified location and carry a certain amount of goods. In the VR environment, we tested a series of data such as data transmission rate, packet loss rate, and latency, and scored the user’s experience to verify the feasibility of the system as well as to provide experience and direction for subsequent system development. The data and process obtained from the experiments will be described in more detail in the subsequent chapters.

4.2. Experiment Setup

The new smart port architecture proposed in this paper can simulate the real-world scenario through VR streaming integrated with additional information about equipment, machine, and road in the port so that the operator can view immersive information of any area of the port at any time and from anywhere. Through 5G, the data collected by the cameras can be transmitted to the remote operator side with ultra low delay thus effectively improving the operation efficiency. It can greatly facilitate the loading and unloading of port cargo, support autonomous driving, reduce labor intensity, and avoid operational safety risks. Virtualizing the information in the port through VR supports the interconnection of everything in the port while observing and retrieving information in real-time.

In the proposed system, multiple sources of information are collected from massive sensors deployed around the smart port. The staff can remotely monitor and control the objectives such as an autonomous truck in real-time through the VR glasses. The system leverages mature driverless trucks technology that can achieve autonomous driving under specific routes in specific environments, and the staff only needs to regularly overhaul the ground as well as the sensors on the trucks. In Figure 12, we demonstrate easy control of a truck traveling around the smart port and observe the container information of each workshop in the port in real-time through the first-person operating platform and simulate the operation of the port in various situations in real-time.

Besides the demonstration on the entire smart port level, a series of experiments are conducted to investigate the feasibility of the proposed 5G + VR architecture within the application of remote crane control. The following section first describes how the real-world environment of the smart port is mapped to the VR environment, and how the data collected from various sensors and PLC controllers are transmitted between both environments via 5G + HTTP to synchronize the states in controlling the truck to move the cargo and the crane to grab the cargo. Detailed network analysis is then performed to study the QoS of the network and its feasibility to support the proposed architecture. Section 4.3 illustrates the QoE analysis of the demonstrated smart port VR system with an extensive subjective test.

4.3. Remote Control of Mechanical Equipment Demonstration

The trolley and crane system are shown in Figure 13. This experiment builds a simplified system to verify the feasibility of the proposed structure and the optimization of this structure. The construction of the system will be gradually enhanced in future projects. The VR system is connected to the PLC device and the AGV device, and commands are sent to the PLC device and the AGV device to implement the ship-to-shore cargo integration process. The overall process is as follows:

• The laden ship and the AGV cart start from their respective starting points, go to the unloading point and stop when they arrive.
• After both the ship and the AGV arrive at the unloading point, the robotic arm R1 starts to unload the cargo.
• After the unloading is completed, the cargo carrier and the AGV trolley go to the loading point and stop when they arrive.
• After both the cargo ship and the trolley arrive at the loading point, the mechanical arm R2 starts loading.
• After the loading is completed, the carrier and the trolley return to the starting point, and the next round of the process is about to start.

The real-time data of the trolley and the cargo ship are synchronized in the VR environment and the corresponding commands, such as forward, turn, and crane to grab the goods, etc., can be sent directly to the VR system. Figure 14 shows the delay measured by the time difference between sending the command in VR and the reaction time of the actual trolley and crane. We compare the latency and packet loss rate of this system in 5G and 4G network environments, and the latency rate obtained from the experiment is shown in Figure 14. The latency of the 4G network is on average 80 ms, which is far from the satisfaction level of most industrial operation requirements. The latency of the 5G network is below 20ms, which is negligible for VR users. This latency can be further reduced to below 5 ms in a dedicated industrial 5G network designed for time-sensitive applications such as remote control and autonomous driving.

Operators can act remotely at the port side through the 5G + VR system, which greatly reduced the movement of operators and will further reduce the occurrence of accidents. For example, a remote-control operator can control a crane to move goods after arriving at a designated area. The crane can effectively avoid a series of problems such as the grab bucket instability caused by improper manual operation in the VR hoisting process. According to the investigation, the maximum data transmission rate of 4G and WiFi is generally not more than 30 M/s, while the transmission rate of 5G is above 200 M/s. The comparison of upload rate, download rate, network fluctuation, and delay between 4G and 5Gz is shown in Figure 15. The test site is in Hongshan District, Wuhan, China. With the low latency and high-speed advantages of 5G communication technology, operators can support real-time control of the entire hoisting process, avoiding errors caused by delays in the operation process. The demonstration of grabbing goods in the VR scenario is synchronized with the demonstration of a real crane.

From the above 5G and 4G network comparison experiments we can conclude that, whether in terms of effectiveness, reliability, latency or packet loss, etc., 5G networks are far better than that of 4G to meet the needs of the smart port. We experimented with driverless cars and remote control in a VR environment with a 4G network, and the network latency was basically above 100 ms, which is completely unable to meet normal production and life needs. Additionally, in the 5G network environment, the latency is almost stable below 30 ms. When the QoS level is high, the VR video streaming tile-based scheme of Mode 2 can be supported, and when the QoS is poor, the scene modeling can be switched to Mode 1, and only the control signal and scene parameters can be transmitted, thus realizing the flexibility of the port work.

5. QoE Analysis of the Smart Port VR System

In this experiment, the experimenter mainly through the VR headset experience in a virtual environment to experience the manipulation of autonomous trucks and manipulate the crane to grab the goods and a series of operations. The system QoE framework is shown in Figure 16.

(1)

Content quality

Content quality is the core part of the augmented reality system and is directly related to the user experience. In terms of content quality, this paper boils down to two important secondary influencing factors, one is the richness of the content, and the other is the authenticity of the information. The ship terminal information visualization system must contain the information that customs officers need to verify in their daily work, and, also, provide some additional auxiliary content. For the authenticity of the information, augmented reality needs to be able to vividly display the content and give the user enough immersion. Therefore, the container, cargo, and ship models need to be as realistic as possible.

(2)

Hardware quality

Since users need to wear VR devices for interaction, the quality of the hardware of the device will greatly affect the user’s experience. Therefore, it is mainly affected by two factors, the user’s freedom of movement and wearing comfort. Users need to be able to freely observe virtual objects from any head perspective, while the user’s head has been supporting the weight of the device in the case of long-term wear work, therefore, the overall quality of the device and the wrapping will directly affect the user experience.

(3)

User interaction

The human-computer interaction under augmented reality is different from the general mobile end of manually clicking and sliding on the screen, it needs to go to use all the physical information input of the user, so user interaction is an important reference index for evaluating the system.

The first is the naturalness of the interaction. The interaction should be as close as possible to the interaction methods used by users in reality, such as voice and gestures. On the other hand, it is the response speed of the system to the user’s operation, for example, when the user speaks a voice quality or clicks a button, if the system is too slow to respond, it will directly reduce the user’s experience, and even make the user lose interest in continuing to use the device experience.

(4)

Environmental understanding

In this paper, environmental understanding is considered one of the important first-level parameters of the system evaluation model. Environmental understanding is further divided into two subcategories. Firstly, realistic elements need to fit with their surroundings in terms of content and location in order to make sense of them. Secondly, as users often change their position and orientation, VR applications should be able to respond to environmental changes quickly to avoid mismatches with the real world.

We have conducted user experience evaluation experiments on this system, the experimenters and experimental steps are as follows:

The testers should ensure that they have some knowledge about port operations and VR so that they can better gauge the feasibility of the system. Thirty-five volunteers were invited to the test. The volunteers include undergraduate and master’s students in computer science, telecommunication, and navigation technology, as well as administrative staff, teachers, and others with different professional backgrounds, ranging from 18 to 50 years old.

The participants were first invited to attend a 10 min warmup to familiarize themselves with Huawei VR glasses, using gestures as well as voice to perform brief interactions such as moving their heads, calling out, and canceling the main menu. After participants were familiarized with the VR device, each user was provided with detailed instructions on how to use the system. After the preparation was completed, users started to test the system, and the entire experiment was controlled to take about 20 min per person, including the time to familiarize with the device, perform the test, and fill out the questionnaire (full version can be found in Appendix A: Users QoE Questionnaire). Participants knew in advance whether they were operating under 4G or 5G networks, and the results were indisputable. In the 4G network, the latency was basically above 80ms, and it was unstable, jittery, and prone to packet loss, which could not support the application’s QoS requirements. Participants generally responded that they were completely unable to perform normal operations under 4G network conditions, and the data obtained was not meaningful for the study. In the 5G network environment, the participants all gave their evaluation results, and the experimental results are shown in Figure 17.

The degree of correlation between the four aspects of the QoE architecture and the overall user scores can be seen in Figure 17. We can obtain a Pearson correlation coefficient of 0.52 between content quality and overall user ratings, which indicates that the content quality parameters maintain a positive correlation with users’ final ratings. Based on numerical analysis, it can be noted that the content quality has a strong correlation with users’ ratings, and when the content quality is higher, users’ final ratings will receive corresponding positive feedback. The correlation coefficients of hardware quality, user interaction and user rating are 0.63 and 0.57, respectively, which have positive effects on the system rating as well as content quality. Among all the ratings, user interaction has the strongest influence on user rating, and the improvement in user interaction will benefit the system more significantly. The lowest correlation between environmental understanding and user score is 0.70, which indicates that environmental understanding and user present a strongest correlation and can be increased in priority for system optimization.

6. Conclusions

This paper introduces the application of 5G + VR in smart ports by elaborately discussing the application prospects of new-generation information technologies such as edge cloud computing, big data, Internet of Things, mobile Internet, and intelligent control. At the same time, this paper shows a demonstration of building a smart port with in-depth integration of information technology and port business based on modern infrastructure and equipment. The major contribution of this paper is that it firstly studied the feasibility and reliability of 5G + VR in smart ports. Through the network QoS tracking, it is confident to say that the current 5G network is feasible for the transmission of VR, only if the virtual scene is carefully designed to transmit minimum data according to real-time QoS level. At the same time, the overall design plan for the smart port is constructed from multiple aspects such as autonomous driving, remote control of mechanical equipment, smart security, emergency command, and an intelligent container tally system. The smart port architecture combines key VR technologies such as video collection, transmission processing, and video playback. By building a target platform for the loading and unloading operations of general cargo terminals, operators can remotely view the operation scenario, and take action on the real terminal devices in the loading and unloading operations of the smart port. With the help of 5G’s massive bandwidth, ultra-low delay, and high reliability, the live VR video of the remote scene and the control commands can meet the requirement for data transmission and remote control. The smart port operation safety and efficiency are highly improved in comparison with traditional ports. The current traditional ports have a high turnover of personnel. What is more, the risk avoidance measures are not perfect, which may cause major accidents and losses in cases such as extreme weather. Compared with traditional ports and other port systems with VR, the proposed architecture adopts a full aspect of information from multiple sources based on 5G network and make interactions through immersive VR. The system supports efficient interaction in real-time, making comprehensive decisions, and, also, simulating the training of workers remotely regardless of restricted environments and weather conditions.

The limitation in this paper is that not all applications are fully implemented and verified by a full set of QoS and QoE analysis. In future work, it is hoped that based on the proposed smart port architecture, more specific parameters (e.g., QoS requirement, control channel stability) for each operation scenario will be determined, providing a scientific basis for the selection of port operating equipment and a reasonable number of decisions.