Collaborative Augmented Digital Twin: A Novel Open-Source Augmented Reality Solution for Training and Maintenance Processes in the Shipyard of the Future ^†

Abstract

Large companies use a lot of resources on workshop operator training and industrial machinery maintenance since the lack of this practice or its poor implementation increases the cost and risks of operating and handling sensitive and/or hazardous machinery. Industrial Augmented Reality (IAR), a major technology in the Industry 4.0 paradigm that may enhance worker performance, minimize hazards and improve manufacturing processes, could be beneficial in this situation. This paper presents an IAR solution that allows for visualizing and interacting with the digital twin of a critical system. Specifically, the augmented digital twin of an industrial cooler was developed. The proposed IAR system provides a dynamic way to perform operator training with a full-size model of the actual equipment and to provide step-by-step guidance so that maintenance processes can be performed more safely and efficiently. The proposed system also allows several users to use devices at the same time, creating a new type of collaborative interaction by viewing the model in the same place and state. Performance tests with many simultaneous users have been conducted, with response latency being measured as the number of connected users grows. Furthermore, the suggested IAR system has been thoroughly tested in a real-world industrial environment.

1. Introduction

The introduction of new technologies that optimize production processes has increased growth to the point of being considered a fourth industrial revolution. Thus, the term Industry 4.0 was established to name this new stage in which industry integrates manufacturing processes, information systems and communications technologies. In particular, Industrial Internet of Things (IIoT) and Industrial Augmented Reality (IAR) are key for the Industry 4.0 paradigm [1].

In addition, cyber-physical systems enable the interconnection between the real and virtual worlds. Although it is not a new term [2], digital twins [3] have experienced an increase in interest in recent years. In the case of the shipbuilding industry, processes are constantly evolving by introducing new technologies to adapt to Industry 4.0.

The aim of this paper is to demonstrate in a practical scenario the capabilities of Industry 4.0 for shipbuilding by mixing three key technologies (digital twin, IAR and IIoT) with the objective of optimizing maintenance, production and operator training tasks. To this end, an application has been developed for Microsoft Hololens Augmented Reality glasses [4]. It consists of a digital twin of a cooler and it allows the visualization of animations and the interaction with the model in a virtual way. In addition, the application displays real-time information obtained from sensors embedded in the cooler.

2. Design of the System

The architecture of the designed system is depicted in Figure 1, where the multiple modules of the system can be distinguished clearly. Both the AR/MR and IoT layers make use of the Service Layer to communicate with each other. In addition, the Service Layer is in charge of coordinating all the processes as well as managing the different protocols used by the heterogeneous devices involved in the system.

In order to show the capabilities of the proposed system, an application was developed for Microsoft HoloLens smartglasses. Such an application allows for visualizing the 3D model of a cooler, as well as to receiving indications on repair and maintenance processes and operating parameters measured in real time by embedded sensors. The application is shown in Figure 2, which has been captured by one of the users, who watches in real time how the other user interacts with the digital twin of the cooler. Thus, both users share the same collaborative experience.

3. Experiments

In order to measure the performance of the system, a series of tests were carried out in a real environment. Taking advantage of the fact that Unity [5] allows the development of applications for different environments and operating systems, a mock-up version of the IAR application was developed to emulate its behavior when running multiple instances on a computer. Thus, a desktop computer can perform operations as if it were multiple users using the IAR application. In this way, the network load can be simulated in a realistic scenario without needing to purchase multiple pairs of HoloLens smart glasses.

The performed test cases were carried out for scenarios with 5, 10, 15 and 20 simultaneous clients. Average latencies were measured using 2.4 GHz and 5 GHz IEEE 802.11 ac networks. In all cases, the measured latencies included the processing and rendering times of the information contained in the packets. In other words, the latency is measured since a packet is generated at the source device and until it is processed and the information is rendered on the screen of the destination device. This includes waiting times related to the frame rate limits imposed by the smart glasses graphics engine. Measuring times in such a way ensures that the results provide an accurate representation of the waiting times that would be experienced by a real user that executes the application.

The average obtained latencies are shown in

Table 1. The 2.4 GHz and 5 GHz average latency measured for a device on the network (ms).

Clients	Average	Standard Deviation	Variance
5	24.0804	13.4883	0.25171
10	40.777	20.925	0.53113
15	61.4403	30.2245	1.02500
20	86.0936	38.2146	1.61572

. As it can be observed, latency increases with the number of users. Additionally, the variance is notably low, especially for a lower number of users, due to network stability (i.e., users generate less interference). The differences between the 2.4 GHz and 5 GHz networks are approximately 3–5 ms, always higher in the case of the 2.4 GHz network due to the saturation of the spectrum in terms of the frequency and the characteristics of the physical layer.

4. Discussion

The developed digital twin application was created and tested to develop shipyard assembly instructions and support. The proposed features allow for the detailed visualization of virtual elements in their full size, positioned precisely in the location where the real elements would be positioned. Moreover, the proposed application facilitates IIoT real-time interactions, collaborative training, and assistance tasks, all of which help to speed up manufacturing operations.