A Collaborative Augmented Reality Application for Training and Assistance during Shipbuilding Assembly Processes ^†

Abstract

This paper presents the development of a novel Microsoft HoloLens collaborative application that allows shipyard operators to interact with a virtual clutch during its assembly in a real Turbine workshop. Such an Augmented Reality (AR) experience acts as a virtual guide while assembling different parts of a ship. In particular, the proposed application allows operators to position the clutch on a real environment and interact with it. The application also provides information about the documentation of each part of the clutch, showing its blueprints and physical measurements. The proposed AR application enables collaborative AR experiences, allowing users to visualize the same content and animations at the same time and interact simultaneously with 3D objects from multiple devices. Furthermore, the application is integrated with an Industrial Internet of Things (IIoT) framework, resulting on an AR-IIoT application that is able to receive and display real-time sensor data on information panels, as well as to trigger actions through actuators by making use of virtual user interfaces.

1. Introduction

Shipbuilding companies are updating and integrating new technologies on their working processes leading to what is called Shipyards 4.0. Among the different Industry 4.0 technologies that can be used in a Shipyard 4.0, one of the most promising is Augmented Reality (AR), which provides a wide range of features that can be leveraged in order to develop powerful and useful applications that project virtual elements into real scenarios. These features enable improving industrial processes, maximizing worker efficiency, as well as incorporating new learning and training methods that can be integrated on a shipbuilding company.

The objective of this work is to study the potential of Industrial Augmented Reality (IAR) to facilitate, support and optimize production and assembly tasks. With this aim, an IAR application has been developed with Microsoft HoloLens glasses [1] to guide operators and to facilitate training tasks in a visual and practical way.

2. Materials and Methods

2.1. Hardware and Software

The proposed IAR application was developed using two Microsoft HoloLens 1st generation smart glasses. The application development was carried out with Unity 2019.3.3f1 using HoloToolkit 2017.4.3.0, as well as Visual Studio 2017 for building and deploying the Unity project into the HoloLens glasses. In addition, in order to be able to compile the project for Universal Windows Platform (UWP), Windows 10 Creators Update and Microsoft UWP Software Development Kit (SDK) were used. All the 3D models were processed with Blender 2.8.

2.2. Methodology

For the development of the IAR application an iterative methodology was followed, in which the different functionalities were added incrementally. In the first iteration the basic functionality was implemented, including the visualization of gear movements and assembly animations, the addition of simple buttons to perform actions, User Interface (UI) translation and 3D model simplification (polygon reduction). In the second iteration the User Experience (UX) was improved, replacing the buttons with a panel that makes use of the holographic buttons provided by HoloToolkit and adding billboard and tag-along functionality to the panel (this implies that the panel follows the user and it is always facing towards his/her direction). In the third iteration the documents associated with the production order and the component blueprints were added, so that when the user taps on them, a panel is shown with such a documentation. In the fourth iteration the shared experience was implemented and integrated. Finally, in the fifth iteration, an Industrial Internet of Things (IIoT) framework was integrated [2], including the corresponding panel modifications, from which the connected sensor values can be seen and interacted with.

3. Results

Figure 1 shows, on the left, a screenshot of the developed IAR application where a step sequence for the correct assembly of a clutch is visualized through animations and together with the blueprints and documentation related to its components. The application can be executed simultaneously and in a synchronized way by different smart glasses that are in the same physical location, so all users see the same virtual objects and animations at the same time and in the same position. In addition, the application is capable of receiving and displaying information from sensors in real time as well as triggering actions through actuators by making use of a virtual user interface.

The developed solution was tested in the Turbine workshop of Navantia in Ferrol during the assembly process of a clutch (illustrated in Figure 1 on the right), which consisted of a high number of mobile parts with varied shapes and sizes. Its aim was to optimize the process by reducing assembly time and the risks associated with human errors. Therefore, the proposed IAR application allows for illustrating visually the contents of the assembly manuals that are currently printed on paper, connecting the clutch with external IIoT devices and as well as easing collaborative training tasks.

4. Discussion

The proposed IAR application was designed and tested for training and assistance during shipbuilding assembly. The developed functionalities allow for visualizing virtual elements in detail at their real size, placed in a precise way in a position where the real elements would be integrated, even before the production of such elements starts. Moreover, the proposed application enables IIoT interactions, collaborative training and assistance tasks to speed up manufacturing processes.