Development and Assessment of Augmented Reality Technology for Using in an Equipment Maintenance and Diagnostic System

Abstract

This study presented work considering the development and initial assessment of an augmented reality approach to provide a user interface for operators that could be a part of an equipment maintenance and diagnostics system. Its purpose was to provide an equipment system for graduate students of engineering and technology to experiment with the design of augmented reality technology. The proposed system took place three hours per week over a period of four weeks of corrective actions that were triggered in the Department of Industrial Education and Technology at the National Changhua University of Education, Taiwan. The students adopted augmented reality technology to achieve big data acquisition and analysis for pre-diagnostic and maintenance applications. Preliminary assessment of the proposed system was encouraging and showed that it achieved success in helping students understand concepts and using augmented reality technology for equipment maintenance and diagnostics. The study provided important initial insights into its impact on student learning.

1. Introduction

With the era of the Internet of Things (IoT) and Industry 4.0 coming, plus COVID-19, the global industry has been impacted, leading to changes in the industrial structure and in industrial management models. With rapid industrial development, making judgments based solely on experience and intuition is no longer sufficient. It is necessary to enhance industrial competitiveness with the help of big data and software and hardware integration system platforms [1].

The impact of COVID-19 highlights the dependence of traditional industries on human labor. Thus, optimizing manpower distribution and division of labor is relatively important. Suppose tasks that are prone to human errors are assigned to IoT platforms. In this case, the equipment can independently capture and analyze the work order information, production data, and its own production parameters. As equipment needs to be regularly maintained in order to achieve high productivity, high efficiency, and high mobility of production lines, on-site operation and maintenance are critical. Therefore, augmented reality (AR) is increasingly used in manufacturing and industrial engineering to promote Industry 4.0 and the Industrial Internet of Things (IIoT) in the development of smart factories [2].

To maintain pace with rapid market transformation and technological changes to enhance competitiveness, industries must respond quicker, adopt new technologies, and fully understand their applications. Often, industries must achieve self-transformation and upgrade without replacing existing equipment architecture or investments. Industrial production data must be known at all times. Therefore, building a set of architecture that can flexibly expand without changing the existing equipment is crucial. Lehnhoff et al. [3] proposed open platform communications unified architecture (OPC UA) to simplify machine-to-machine (M2M) communications to help industries solve the problems of inter-system data reorganization.

Virtual and real integration technology mainly employs 3D teaching materials to carry out situational simulation learning. This helps students acquire knowledge from situational backgrounds constructed by teachers. Situational simulation learning emphasizes that students can learn by combining situational learning with simulation teaching. Students can make judgments after careful thinking through their logic, cognition, and observation. They can obtain immediate feedback on different decisions. Chau et al. [4] found that situational simulation can obtain high satisfaction for students and inspire student interest compared with traditional learning modes. Safaei and Shafieiyoun [5] found that the virtual learning environment has been widely used in various medical, business, and education fields. Teachers can use a virtual learning environment to give students a sense of presence, such as personal experience. Situation simulations can improve students’ interests in learning.

With the improvement in process technology in the technology industry, the requirements for process accuracy are increased, making it challenging to improve the product yield, which is a crucial problem to solve in the manufacturing industry. In addition, with the progress and development of manufacturing technologies, production machines are becoming increasingly complex. After long-term operation, parts of production machines may age or be damaged, reducing the production yield and quality of processed products. Therefore, to maintain production machines and workpiece accuracy, real-time intelligent fault diagnostics and pre-diagnostic abilities of machines, such as machine fault detection and intelligent management systems, should be used. This will ensure the reliability and production quality of production machines and reduce machine maintenance costs.

Virtual and real integration technology, a key technology of Industry 4.0, uses computers and sensors to connect various devices, machines, and digital systems through the network technology so that they can communicate with each other and integrate the virtual world with the real world. The virtual and real integration system can acquire big data from entities, environments, and activities to integrate network space with real space, and then achieve the overall intelligence of production and manufacturing. Frolov et al. [6] highlighted that intelligent manufacturing in Industry 4.0 covers many aspects. The virtual and real production system is an important part, which can achieve flexible system development, improve market competitiveness, and help Taiwan enter the era of intelligent manufacturing more quickly.

Virtual and real integration technology uses display technology to combine creative visual design and artificial intelligence technology. Through the virtual plant demonstration model, 3D plant simulation software is applied to integrate virtual and real learning with experience, combine network learning and entity teaching, analyze the big data of on-site teaching, and count students’ learning difficulties. Teachers can simulate layout, process, mechatronics, and automatic manufacturing plants and verify mechatronics control programs through offline programming to reduce the damage probability of automatic warehouse equipment and apply the virtual and real integration technology to automatic warehouse equipment.

Augmented reality (AR) is a combination between real and virtual reality (VR). AR is defined as a real environment “augmented” through virtual (computer graphic) objects. It is a technology capable of enhancing a person’s perception of their physical environment through interactive digital information virtually superimposed on the real world [7,8].

Augmented reality (AR) is increasingly applied by enterprises in manufacturing and industrial engineering, and is the key to realizing smart factories and promoting Industry 4.0 and IIoT. For example, wearing AR glasses for warehouse inventory and safety helmets combined with AR glasses for equipment maintenance. As AR technology is relatively new, industries are cautious about it under current conditions. As such, it is not yet widespread across industries. However, augmented reality technology combines virtual and real integration technology to diagnose automatic warehouse equipment and to simulate how to produce products on production lines in the future. This is necessary to avoid risks that may be caused in the production process and industrial safety during actual production line deployment, reduce the damage to automatic warehouse equipment, and achieve cross-domain integration.

This study integrated equipment communication protocols with augmented reality technology, including the integration of PLC and CNC controller data, to integrate equipment information into the virtual and real integration system platform through the OPC UA communication protocol [9]. Technicians used AR glasses to connect to the plant network through WIFI. The virtual–real integrated network system platform in the back-end was used to send equipment information captured by AR glasses to on-site technicians for troubleshooting [10].

From a pedagogical point of view, the proposed system would provide a consistent and complete learning environment. From a technological perspective, this study focuses on the adaptation of concepts and technologies developed in the field of augmented reality technology. In this proposed system, students will learn the following: (1) to plan an augmented reality technology project, (2) to implement programs, (3) to monitor processes, (4) to grasp automation concepts, and (5) to understand how augmented reality technology processes work. In addition to augmented reality (AR) technology, the virtual–real integrated network system platform in this study integrated communication technologies of plants to use IIoT in Industrial 4.0 intelligent manufacturing. Specifically, the purposes of this study were as follows: (1) to build a virtual–real integrated network system platform centered on the augmented reality technology for industrial equipment maintenance and diagnostics, (2) to provide various equipment integration applications to the virtual–real integrated network system platform, and (3) to provide engineers with equipment resource integration for information integration and development.

2. Literature Review

2.1. Augmented Reality (AR) in Industry

Human–machine interfaces are devices that allow for the interaction between a human and a machine [11,12,13]. Technological advances in creating holograms have recently made it possible to have devices and software tools that allow augmented reality (AR) applications in industrial sectors [14].

There are several examples of the usage of augmented reality (AR) in industry settings. The survey of [15] overviews AR smart glasses for assembly operators in the manufacturing industry. Lambrecht et al. [16] demonstrated that AR-based robot programming could be more efficient than classic teach-in and offline programming, and described the possibilities by combining the benefits of online and offline programming into an AR system [17]. Makris et al. [18] presented an augmented reality system for operator support in human–robot collaborative assembly. AR is also applied to aspects of manufacturing other than providing support for operators. For example, Karlsson et al. [19] combined augmented reality and simulation-based optimization for decision-maker support in manufacturing.

With bibliometric and network visualization techniques, Goel et al. [20] synthesized the apparel industry into the literature on augmented reality and virtual reality. After examining the extant gaps in the literature, they set the trajectory for future research by integrating the seemingly unrelated literature of augmented and virtual reality in the apparel industry through mapping the intellectual structure.

2.2. Augmented Reality (AR) in Education

Lorenzo et al. [21] applied the Web of Science and Scopus databases to develop a comprehensive review of how augmented reality (AR) helped autistic students in learning from 1996 to 2020, which set the foundation for designing an action protocol of AR activities for autistic students. In addition, Cheng et al. [22] concluded that the AR educational program could improve students’ knowledge and thinking ability. On the basis of situated learning, Li and Liu [23] blended mobile learning and augmented reality, making students not only have access to information content in a real situation, but also exposing them to such information via augmented reality. As it turned out, participants gave positive feedbacks of six aspects of the proposed approach.

An augmented reality (AR) system was also implemented to improve maintenance technicians’ performance from the training stage to daily interventions by Oliveira et al. [24]. Such an attempt was believed to have made great contributions towards the development of AR and their application in industrial environments. With an AR-guided product disassembly framework being proposed, Chang et al. [25] excluded the help of expert intervention and enhanced the efficiency of the disassembly process performed by human operators. With the guidance of AR, core-retrieval during remanufacturing can be accomplished with great quality. In addition, augmented reality visualization can have a great impact on maintenance [26]. AR can close the information loop between maintenance information systems and the supported operations, which can help in not only boosting performance, but also retrieving feedback. Semi-structured interviews and surveys with maintainers were also conducted to determine the maintenance challenges and validate the proposed framework.

Diao and Shih [27] integrated a smartphone-based platform with the scenario in a real-world old campus building with a modeling-based augmented reality maintenance system. By following the animated instruction, management protocols and maintenance routes can be suggested, which also ensured a worker’s safety. Such implementation received positive user responses.

In the Industry 4.0 era, predictive maintenance is the key to boosting the reliability of machine tools in manufacturing. An intelligent predictive maintenance approach for machine tools was proposed [28] in order to implement a highly reliable maintenance plan, as well as the fault prediction, the decision-making process of maintenance, and the AR-based auxiliary maintenance. Moreover, remote expert service is also integrated in the AR-supported auxiliary maintenance so that unanticipated faults that have no occurrence in maintenance experience database can be dealt with. In sum, the proposed maintenance approach was considered to be of great assistance.

2.3. OPC UA Communication Protocol Architecture and Its Principle

Figure 1 shows the OPC UA system architecture [9]. Plant devices may include various controllers and components such as PLC, CNC, robot arms, and sensors. The field device communication protocol can be converted into the OPC UA protocol through the IIoT gateway, to integrate information from the back-end database, mobile guidance, web presentation, and enterprise situation rooms, and to easily expand plant equipment and back-end management integration applications. Moreover, augmented reality (AR) communication data issues field device information and integrates picture information to receive alarm messages, status information, and log the data of field devices. Enterprise situation rooms present data in a visual form, track and analyze them layer by layer, and help managers improve the timeliness of decision-making and problem-solving.

3. AR-Based Remotely Collaborative Equipment Network Architecture

3.1. Proposed AR-Based User Interface

Figure 2 shows the AR-based equipment network system architecture, which can be used for preventive maintenance and for diagnosing metal forming punches. Users adopted wearable devices for equipment inspection and operation. At the same time, the virtual and real integration system platform connected with the field equipment to obtain information and an equipment action state for the real application of IIoT. In traditional industrial applications, personnel operation and equipment information are separated and cannot correspond to the relevant information, resulting in personnel disoperation and no fool-proofing or a protection mechanism. Through the system platform developed in this study, mutual supervision and an interlocking mechanism could be achieved to ensure personnel safety and equipment maintenance, thereby improving plant operation efficiency and reducing the shutdown duration caused by failure or human factors.

This study combined augmented reality technology with virtual and real integration technology to diagnose metal forming punches, and simulated how to produce products on production lines in the future. This method could help avoid the risks that may be caused in the production process and in industrial safety during actual production line deployment, which would reduce damage to the automatic warehouse equipment and achieve cross-domain integration.

3.2. Augmented Reality Wearable Hardware

An augmented reality device primarily consists of a wearable bracket, a video lens, a microphone audio receiver, an image screen, and a built-in WIFI or USB link device for connecting to data in a back-end virtual–real integration platform system. Therefore, the hardware and software should be matched accordingly, as other hardware is available. The system should only comply with the OS development environment and can be applied to the system of this study.

3.2.1. Specifications of Wearable Hardware

The specifications of wearable hardware [29] in this study are shown in

Table 1. Specifications of the wearable hardware.

Built-in Display (Built-in Audio)
AR View	Monocular
FOV (Horizontal)	16.8°
Weight	180 g
Built-in audio	Yes (ear speaker)
Microphone	Yes (noise canceling)
Connectivity	Wi-Fi, Bluetooth
Charging	USB-C
Camera	12 MP and 4K 30 FPS video
Memory	6 GB RAM—64 GB internal memory
Battery	1000 mAh internal battery
Battery life	2–3 h
Controls	Touchpad, headmotion, and voice
Operating system	Android 8.1
Chip	8 Core 2.52 Ghz Qualcomm XR1
Compliances	IP 67, water, dust and drop resistant, and PPE
Manufacturer	Vrexpert
Country	Netherlands
OS version	Android 10
Creator	Transition technologies PSC
Software source	Transition technologies PSC in Poland

. The built-in operating system of this AR wearable device is Android OS. Based on this operating system environment, this study developed APPs on this device to link them with the back-end virtual and real integration system through the WIFI network or USB communication device to integrate equipment networking information. The back-end server releases images and guidance information as a demonstration.

The AR wearable device adopted in this study is shown in Figure 3. The device can be integrated into the research system of this study if it complies with the hardware specifications, is highly compatible, and can be applied across platforms.

3.2.2. Wearable Device System Description

The AR wearable device is used to scan the actual images of the field equipment. Through data capture of the virtual and real integration system server, the image capture operation and the judgment of label creation are carried out. Then, the system performs logical guidance and outputs guidance information to online users who wear AR devices in the field. Through this logical action cycle, the preventive maintenance, verification, and inspection of equipment can be completed to achieve equipment maintenance and operation. During the process, the operator only needs to face the direction of the equipment switch and the operation area, and the system will determine the equipment in the current operation [30]. The AR wearable device has six main functions, namely: (1) help the operator (OP) perform operations and inspect equipment parts, (2) cooperate with remote experts to guide operations and inspect machines or parts, (3) create working procedures to help other operators or novices complete tasks according to the steps, (4) scan-build digital copies of physical locations, (5) volume label by using digital volume labels to mark additional information for entity objects, and (6) guidance by providing information about the equipment body.

3.3. AR-Based Remote Collaboration System Architecture

The AR-based remote collaboration system architecture is comprised of subscription and publication. Through the data calculation, a system command, video in and video out of the virtual and real integration system, the call of the field AR-based remote collaboration system architecture is completed. This virtual and real integration system can realize the support function of on-site subscription by remote experts, computer vision, or third-party systems to solve the equipment operation, maintenance, and testing on the site [31]. The system can meet the following six functional requirements: environment identification, data portability, work instruction, video teaching, remote collaboration, and labeling at any time.

4. AR-Introduced Equipment Maintenance and Diagnostic System

The global manufacturing industry is accelerating towards smart factories. With the increasing demand for remote operation and automation, emerging technologies such as big data, artificial intelligence, virtual and real integration, and IoT are used for capacity improvement and optimization. Therefore, after statistics and analysis, the visual sensor data from the plant equipment are dynamically displayed on the portable device through the AR interface, which can show the possible failures of the current equipment and even lists the historical maintenance data and various details of each equipment so that technicians can simply learn about and reduce the losses caused by improper operation.

4.1. Introduction Examples

This study introduced the designed technology into the metal forming punch production line to reduce the defective products of sheet metals. AR technology can also monitor the production status of equipment, production capacity, production parameters, abnormal records, equipment maintenance diagnostics, and mold life. Moreover, employees carry out maintenance warnings and abnormal records of equipment through augmented reality inspection and operation, which reduces troubleshooting only when the equipment fails and it improves the real-time grasp of the production status. When the manager inspects the production line, the AR portable wearable device can directly reflect the plant equipment’s actual data and conditions. In the case of an emergency, decisions and responses can be made immediately to reduce unnecessary losses.

4.2. System Integration Test

The general aim of this study was to develop augmented reality technology to achieve equipment maintenance and diagnostic system for maintaining the normal operation of equipment in production lines. The proposed system was managed by a teacher and graduate assistant. Five male students participated, who were drawn from the population of graduate students. The students were required to prepare themselves before the system and write a report upon completing the system. The proposed system experiment took place three hours per week over a period of four weeks.

4.2.1. Expert Evaluations

Three domain expert evaluations were used to help determine the accuracy of the embedded knowledge and the effectiveness of the teaching system. All were University professors and/or researchers with an average more than 3 years of experience in teaching augmented reality technology; therefore, each expert had a strong background in engineering education. Overall, the expert evaluations were generally positive.

4.2.2. Student Evaluations

Evaluations by the five participating students were used to help determine the acceptability of the project system according to the following criteria: (1) usability of the system, (2) the extent to which the students gained hands-on experience, and (3) whether the project system helped students learn about augmented reality technology. The evaluation form was distributed to all students who had taken a course at the Department of Industrial Education and Technology at the National Changhua University of Education, Taiwan. Most students successfully undertook the experiment, and responded positively to the system. They reported that they enjoyed performing the task and found it interesting.

The students also provided suggestions to improve the laboratory, for example, (1) it would be better to have more assignments and exercises because it is good to practice on the assignments, and (2) it would help if animations on difficult topics were included. As a result of the interviews, which were conducted with the students in an unstructured manner, it was determined that students could use the system covered by augmented reality technology learning presented in this study.

4.3. System Architecture Application and Description

The proposed augmented reality technology could can be set up by designing experimental content, such as a user interface, for the device on site with step-by-step instructions. The students followed the instructions to conduct the use of the device on site. The students used the software to prepare their own control programs for the augmented reality technology system.

Figure 4 shows the architecture of this study, which can flexibly integrate and expand various devices and architectures. In addition to the characteristics of edge computing, it integrates the applications of cloud servers and the ability to provide feedback in real-time. In this study, augmented reality technology was used for integrated applications of wearable devices, and augmented reality technology was introduced in the production line of the punch to combine with the virtual and real system architecture. Furthermore, the IIoT gateway [32] and the punch equipment PLC were used for communication data acquisition. The communication architecture shows the manual and automatic production and operation, and the production parameters set for PLC and HMI. In addition, if detailed mold information cannot be obtained from PLC, transverse and longitudinal vibration sensors were added to obtain production data, to detect the vibration arising from pressing of sheet metals and models during production, for cross-comparison with pressure data returned by PLC, and to conduct the quantitative statistical analysis. Then, a quantitative statistical analysis is conducted to learn about the actual production parameters so as to know the best time for future equipment and mold maintenance in order to optimize the capacity.

The equipment information collected by the IIoT gateway is sent back to the virtual and real system server via the plant’s internal entity network for data storage, collation, selection, classification, statistics, analysis, diagnostics, and early warnings. Important information, such as the production quantity, capacity data, abnormal statistics, sensor status, production time, and maintenance status, is displayed on the war situation information billboard, which can directly help the lean management of the production line in real-time. In addition, the manager or inspector can use the AR wearable device on-site, and the data displayed on the AR wearable device screen are calculated by machine equipment in the virtual and real system.

Figure 5 shows the war information billboard through an analysis of the sensor status alarm. After viewing the statistics of the abnormal status of all equipment sensors in the production line, the frequent failure points of the equipment are analyzed, repaired quikly, and troubleshot to offer immediate help for future maintenance and diagnosis so that the equipment can improve the production value of the plant in real-time.

Figure 6 shows the AR wearable device maintenance diagnostic screen. Through the AR wearable device for equipment maintenance diagnostic operations, the operator can scan the inside of the device’s electrical control box through the lens, and the virtual integrated system server will prompt the device normal initial setting, as well as the previously created warning label diagram, reminding the operator to warn of electrical hazards or the setting through which the location of the internal configuration components should be located. If the operator has unclear parts, they can also call the back desk for remote guidance from the professional engineers, or do so through a standard operation process (SOP).

Figure 7 shows the data screen displayed by the AR wearable device. The equipment maintenance can be completed according to the SOP, which can reduce improper operations or failures.

The virtual representations give more information to the user, providing insights that can be helpful for device maintenance diagnostics, debugging, and safety. The main visualization elements are virtual system visualization, live command dialogue, joint tooltip, and equipment targets, as follows:

(1)

The virtual equipment can mimic the actual equipment. The virtual equipment anticipating the movement of the actual equipment could support safety in a collaborative environment, as the predictive movement of the equipment could be seen in advance so as to prevent possible risks.

(2)

The live command dialogue indicates the current line of code executed by the controller. This can allow users to identify different moments on the production and connect them, which could be useful for debugging or introduced changes introduced in the code.

(3)

The joint information spline tooltip displays the position and state of the joint.

(4)

The equipment targets are visualized with the use of frames, which also indicate orientation.

(5)

The equipment targets can be visualized to understand the use of frames and to predict the equipment’s subsequent movements.

All of these graphical user interface (GUI) elements can be hidden or displayed depending on the aim of the AR wearable device operator.

4.4. Analysis and Discussion

Table 2. Comparison before and after the introduction of the system.

Item Description	Before the Introduction of the System	After the Introduction of the System
Human resource	High	Low
Paper operation	High	No paper operation
Input cost	High human cost	High system construction cost
Data integrity	Low	Actual equipment feedback
Timeliness	Production data at work are the report data at the end of the previous day	Real-time feedback
Degree of visualization	None	High
Condition of equipment in the production line	Low degree of grasp or no real-time	A high degree of grasp/real-time feedback/alert notification
Production arrangement	Failure to remove equipment faults without delay	Grasp real-time status, fault report
Product yield	Low	High

shows the comparison of the plant’s production line before and after the actual introduction of the AR system. The table shows that the production data can be obtained in real-time after the system is established. Before the system is introduced, human factors may cause low data integrity, thus increasing production costs. As a result of the system’s advantages, the real data of the equipment can be known, and the system will conduct data collection, storage, statistics, and analysis automatically, without staff reports. Therefore, the sensor data of the equipment are important. After the data collection through the IIoT gateway, the real-time data can be transmitted to the virtual and real system servers for analyses and applications.

This study developed and assessed augmented reality technology to provide a user interface for operators that could be a part of an equipment maintenance and diagnostics system. The students achieved big data acquisition and analysis for pre-diagnostic and maintenance applications. Figure 8 shows the production data for the vibration analysis (normal). The normal-state data under long-term system observation indicate the normal vibration data in the process of production by the punch.

On the contrary, if abnormal waveforms are detected, it can be confirmed that the data are different due to mold aging, as shown in Figure 9. Therefore, the system will issue warnings, but will not stop the equipment in real time. Engineers are required to diagnose and maintain the equipment, and the normal production capacity will not be affected at random. After expert evaluation, whether the mold is aging can be confirmed, and a future AI learning model for equipment diagnostics can be built to improve the judgment accuracy of the system.

5. Conclusions

5.1. Conclusions

In this study, augmented reality technology was applied to industrial equipment maintenance and diagnostics. The virtual and real integration network system platform was combined with IIoT applications to conform to the trend of current industrial applications and to adapt to the rapidly changing international market environment, especially under the impact of the global COVID-19 pandemic on industrial development. In this study, automatic industrial equipment applications reflected the Industrial 4.0 equipment network, equipment data collection, and real-time online interactive decision-making. Augmented reality technology can realize real-time online recognition and real-time data feedback in order to reduce operating costs or equipment losses for enterprises.

This study integrated the complete equipment information into the virtual and real integration network system platform through the OPC UA communication protocol. The conclusions are as follows: (1) Automatic industrial equipment diagnostics and maintenance were introduced to industrial applications, and various equipment can be integrated and applied into the virtual and real integration network system platform. (2) The IIoT gateway was provided to the control system for OPC UA protocol conversion and cloud data concatenation and integration, including for communication devices, sensors, and remote servers. (3) In this study, the vertically integrated infrastructure of Industry 4.0 was realized, and equipment information in the perception layer and network layer were exchanged through the IIoT gateway.

5.2. Limitations

Although these results provide insight into effective learning initiatives, some limitations must be addressed when interpreting them. First, this study represents the test of a theoretical model, and should be subjected to further testing with different participants, contexts, and technological architectures. Second, the participants were graduate students who were completing the course as part of their degree requirement, so these results may not generalize to other settings and contexts. Issues of motivation for research participation by graduates can also influence the results. Third, owing to the course requirements, this study could not completely capture the richness of the reciprocal relationship between social presence and interaction.

5.3. Recommendations

This study only focused on integrating augmented reality technology and virtual and real systems for local equipment system architecture in industry. There are two suggestions for future studies: (1) A universal system architecture standard operating procedure (SOP) and a universal online equipment modularization approach regarding AR technology can be developed, which will be beneficial for the rapid introduction and establishment of industrial systems. (2) Modular and normative specifications are suggested to be established among the layers of the information system in order to reduce the development costs of customized projects. (3) Another AR system to quantify the actual effect is an advanced human–machine interface, which will be welcome in the future.