Real-Time Remote Maintenance Support Based on Augmented Reality (AR)
Round 1
Reviewer 1 Report
It is not necessary to show literature search methodology, any graduate student will know this. Suggest delete. Not sure what is new in this submission compared to several papers already published by the authors and his group. Criticizing other approaches are outdated is an insult to the researchers mentioned in the reference and this is ethically unacceptable and uncalled for. There is no real case study. Readers can only see framework. It is claimed that the methodology has been tested in vitro in a lab-based workshop and real life industrial scenario, such claims are not substantiated. The formula (3) is not meaningful and its use in the study not well demonstrated. Table 1 and Figure 7 are not credible as they could be any figures, and there is no proof that they are obtained from industrial case studies. In summary, the submission appears to be conceptual, and there is a serious lack of demonstration that the methodology (framework) has been well tested in industry. The authors should describe this in much greater details to make it convincing.Author Response
Comment:
It is not necessary to show literature search methodology, any graduate student will know this. Suggest delete. Not sure what is new in this submission compared to several papers already published by the authors and his group.
Reply:
Thank you for this comment. The review methodology has been limited only to the absolute necessary information. In contrast to past publications, this approach proposes a real-time AR remote maintenance system. The past publications were focused on the development of AR remote maintenance instructions based on predeveloped AR scenes. By extension, this new framework facilitates engineers to provide real-time support in unexpected scenarios, by minimizing the time to create and share AR content.
Comment:
Criticizing other approaches are outdated is an insult to the researchers mentioned in the reference and this is ethically unacceptable and uncalled for.
Reply:
The authors would like to apologize for that misunderstanding, as there was no intention in insulting other researchers. In the manuscript it is clearly stated that these research work have sparked the interest of the authors in conducting the present research work. However, in order to sufficiently address that issue the term outdated has been replaced by the following phrase:
“is based on technologies and equipment that currently can be considered as obsolete”
Comment:
There is no real case study. Readers can only see framework. it is claimed that the methodology has been tested in vitro in a lab-based workshop and real life industrial scenario, such claims are not substantiated.
Reply:
The case study section has been revised and enriched with more material.
Comment:
The formula (3) is not meaningful and its use in the study not well demonstrated. Table 1 and Figure 7 are not credible as they could be any figures, and there is no proof that they are obtained from industrial case studies. In summary, the submission appears to be conceptual, and there is a serious lack of demonstration that the methodology (framework) has been well tested in industry. The authors should describe this is much greater details to make it convincing.
Reply:
The data used for the calculations of the maintenance and operation costs have been provided by the industrial partner. However, these data cannot be shared publicly as per the industrial partner’s request. The framework has been developed and test both in-vitro and in the industrial environment as described in the manuscript.
Reviewer 2 Report
In this paper a framework for remotely supporting maintenance operations based on augmented reality is proposed. In general, the paper is very well organized, written and presented, focusing on an important issue for the manufacturing enterprises, by proposing to approach it through innovative and Industry 4.0 oriented approaches and technologies. Moreover, the papers appear to be sound and complete, by putting forward an original contribution.
Although, some improvements are still needed, such as:
Mainly in the introduction, but also further along the text, some references are lacking, for instance when stating: in lines 39 to 41 “… a considerable amount of research effort has been put on the design and development of real-time maintenance support tools and applications based on mobile devices to prevent unnecessary errors from happening”. Also next, in lines 41 to 44: “With the evolution of Information and 42 Communication Technologies (ICT) as well as the massive break out of Augmented Reality (AR), the 43 research community extended that approach on exploiting the advantages of AR for data projection 44 during the maintenance operations.” And also, on lines 45 to 47: “However, most of the research works available on the web cover basic aspects of Maintenance 46 and Repair Operations (MRO) and provide flexible and accurate troubleshooting, the constant 47 innovation of cutting-edge technologies unveils new opportunities for remote maintenance support”.Additionally, although the authors provide a literature about more or less closely related work done on remote maintenance services, they argue that no novel and real-time based approaches or platforms do exist, and do not more closely compare their contribution to some existing ones, which appears to limit the credibility of the work.
Therefore, I would like to suggest the authors to provide some kind of comparison between features (advantages and disadvantages) of their proposed framework regarding the state of the art.
Author Response
Comment:
Mainly in the introduction, but also further along the text, some references are lacking, for instance when stating: in lines 39 to 41 “… a considerable amount of research effort has been put on the design and development of real-time maintenance support tools and applications based on mobile devices to prevent unnecessary errors from happening”. Also next, in lines 41 to 44: “With the evolution of Information and Communication Technologies (ICT) as well as the massive break out of Augmented Reality (AR), the research community extended that approach on exploiting the advantages of AR for data projection during the maintenance operations.” And also, on lines 45 to 47: “However, most of the research works available on the web cover basic aspects of Maintenance and Repair Operations (MRO) and provide flexible and accurate troubleshooting, the constant innovation of cutting-edge technologies unveils new opportunities for remote maintenance support”.
Reply:
Thank you, the missing references have been added to the pointed sections. The changes are visible below.
Besides the failure forecast, a major issue identified, is the completion of accurate and error-free maintenance operations, and thus the reassurance of fully functional machines in the earliest time possible. Based on that concept, a considerable amount of research effort has been put on the design and development of real-time maintenance support tools and applications based on mobile devices to prevent unnecessary errors from happening [8,16,17,23]. With the evolution of Information and Communication Technologies (ICT) as well as the massive break out of Augmented Reality (AR), the research community extended that approach on exploiting the advantages of AR for data projection during the maintenance operations [23-28]. Generally, under the framework of Industry 4.0 in an attempt to bridge the physical world with virtual worlds, new technologies have emerged, such as Augmented Reality (AR), Mixed Reality (MR) and Virtual Reality (VR) [3]. These digital technologies lie under the umbrella term Extended Reality (XR). The term XR is often used by practitioners and scientists in order to describe the three immersive technologies mentioned above [4].
However, most of the research works available on the web cover basic aspects of Maintenance and Repair Operations (MRO) and provide flexible and accurate troubleshooting [8,16,17,23], the constant innovation of cutting-edge technologies unveils new opportunities for remote maintenance support. Therefore, there is an apparent need for the development of real-time AR frameworks for the support of MRO.
Comment:
Additionally, although the authors provide a literature about more or less closely related work done on remote maintenance services, they argue that no novel and real-time based approaches or platforms do exist, and do not more closely compare their contribution to some existing ones, which appears to limit the credibility of the work.
Therefore, I would like to suggest the authors to provide some kind of comparison between features (advantages and disadvantages) of their proposed framework regarding the state of the art.
Reply:
Thank you for this meaningful comment. As suggested, a comparison of the proposed framework other similar frameworks has been presented and discussed in Section 6 “Discussion”.
Reviewer 3 Report
Author propose original approach to remote co-operation between engineer-expert and technician on the shop floor. The problem is interesting for both sides scientists and technicians. This document only presents a general look at the problem and indicates how to solve it using the Cloud bridging the different working groups involved in the production lifecycle.
Next paper should focus on the substantive and technical details of proposed solutions
Hoever the idea is interesting, and will find readers
Author Response
Comment:
Author propose original approach to remote co-operation between engineer-expert and technician on the shop floor. The problem is interesting for both sides scientists and technicians. This document only presents a general look at the problem and indicates how to solve it using the Cloud bridging the different working groups involved in the production lifecycle.
Next paper should focus on the substantive and technical details of proposed solutions
However the idea is interesting, and will find readers
Reply:
The authors would like to thank the reviewer, for their time and effort put, on reviewing the manuscript and providing meaningful comments.
Reviewer 4 Report
I do not consider myself an expert in engineering, but rather an augmented reality in general. From this point of view, I also evaluated this paper. I cannot judge to what extent augmented reality is usable / beneficial / innovative in the engineering domain.
In general, I recommend to better describe the research part, system verification and add discussion section. It would be useful to specify some information in the description of the system architecture, too.
Abstract
The term " Extended Reality (XR) " as used in the abstract, however, it does not appear anywhere else in the text (e. g. in the introduction). Either it should be deleted from the abstract, or it should mention with its definition in the introduction.
Introduction
Page 2, row 58, formula 2 –Variables (A, MTTR) are not explained in the list. It should be the same as formula 1. Once introduced, the abbreviation should be used consistently (Information and Communication Technologies – ICT). It is relevant both for the Introduction and Literature Review sections.
Literature Review
I am surprised that the Web of Science, which is considered to be the most prestigious in many domains (and countries), was not used as a source for systematic research. Were the described queries (page 2, rows 94-101) used within ResearchGate (page 2, row 87), too? In the first attempt, I found relevant results that are not used in the bibliography, i.e. https://www.researchgate.net/publication/224189086_Exploring_the_Benefits_of_Augmented_Reality_Documentation_for_Maintenance_and_Repair
System architecture
How does communication between shop-floor technician and engineering department work within the system described? is it only visual or audio is also supported (audio-visual communication)? Which is also a relevant way to improve communication. Page 7, row 247, figure 2 – Firs step in sequence – “1) Establish communication …” is bidirectional. I thought (based on schema in figure1) that communication is initiated by a shop-floor technician. Page 7, row 247, figure 2 – “Cloud service” and “Cloud storage/toolbox” are connected in some way or they are independent parts of the system? What are the hardware requirements of the described system? Is the app designed for tablets, smartphones, or other devices?
Case Study and Results
Why was the application not user-tested? User testing could provide valuable information to improve the GUI and also the overall architecture. Figures 7 and 8 are redundant. The information they contain is also in Table 1. Page 10, row 344, formula 3 – Variable ??? is not explained. Page 10, row 360, figure 6 – Why is the graph interlaced with a polynomial function? The text should justify why the linear function was not used. Page 11, row 387, figure 8 – Graph does not have units for Y-axis. Used terms should be unified (Scenario 1 = AS-IS SITUATION; Scenario 2 = PROPOSED METHODOLOGY).
Conclusions
The conclusion should be preceded by a discussion that will comment the advantages and disadvantages of the created system or augmented reality in general, eventually the further extension of the system in the future, or its further testing (including user evaluation).
Typos
Page 1, row 14 – … Reality (AR).Furthermore … Page 10, row 335 – Desgin of Experiments
Author Response
Comment:
The term " Extended Reality (XR) " as used in the abstract, however, it does not appear anywhere else in the text (e.g. in the introduction). Either it should be deleted from the abstract, or it should mention with its definition in the introduction.
Reply:
The following text has been added in the introduction area, in order to support the concept of XR.
“Generally, under the framework of Industry 4.0 in an attempt to bridge the physical world with virtual worlds, new technologies have emerged, such as Augmented Reality (AR), Mixed Reality (MR) and Virtual Reality (VR) [3]. These digital technologies lie under the umbrella term Extended Reality (XR). The term XR is often used by practitioners and scientists in order to describe the three immersive technologies mentioned above [4].”
Comment:
Page 2, row 58, formula 2 –Variables (A, MTTR) are not explained in the list. It should be the same as formula 1. Once introduced, the abbreviation should be used consistently (Information and Communication Technologies – ICT). It is relevant both for the Introduction and Literature Review sections.
Reply:
These two variables have been discussed in the context of the manuscript. And under equation (2), the symbols used are explicitly explained. More specifically, A represents the machine availability for given values of MTBF and MTTR. Regarding MTTR, it represents the Mean Time To Repair a given machine, and it is commonly expressed in hours. It is a value calculated from recording the duration of maintenance operations, by maintenance engineers.
Comment:
I am surprised that the Web of Science, which is considered to be the most prestigious in many domains (and countries), was not used as a source for systematic research. Were the described queries (page 2, rows 94-101) used within ResearchGate (page 2, row 87), too? In the first attempt, I found relevant results that are not used in the bibliography, i.e. https://www.researchgate.net/publication/224189086_Exploring_the_Benefits_of_Augmented_Reality_Documentation_for_Maintenance_and_Repair
Reply:
Thank you for this comment, Web of Science has been used in the revised version of the manuscript in order to enhance the completeness of the literature review. The queries presented in the beginning of Section 2 have been applied to all of the platforms listed in the manuscript. However, the queries in the revised version of the manuscript have been deleted since they were not considered necessary.
Comment:
How does communication between shop-floor technician and engineering department work within the system described? Is it only visual or audio is also supported (audio-visual communication)? Which is also a relevant way to improve communication. Page 7, row 247, figure 2 – Firs step in sequence – “1) Establish communication …” is bidirectional. I thought (based on schema in figure1) that communication is initiated by a shop-floor technician.
Reply:
As it is stated in Section 3 “Proposed System Architecture”, the communication between the shop-floor technician and the expert engineer, is not limited. More specifically, the communication of the two parties can be in the form of teleconference, meaning that audio and video transmission is supported, as well as file exchange is also feasible.
Regarding Figure 2, the bidirectional arrow is used in order to mention that audio and video communication is bidirectional. The reviewer has well-conceived the fact that the shop-floor technician is the one to initiate the communication.
Comment:
Page 7, row 247, figure 2 – “Cloud service” and “Cloud storage/toolbox” are connected in some way or they are independent parts of the system? What are the hardware requirements of the described system? Is the app designed for tablets, smartphones, or other devices?
Reply:
The cloud services are independent to the Cloud storage/toolbox; thus the dashed line is used in Figure 2.
Regarding the hardware, in the current version of the app, a Microsoft HoloLens is required for the shop-floor technician. For the expert engineer, a common desktop PC is capable of supporting the framework and its functionalities. As a reference, the development and testing of the application has been accomplished, by using a laptop PC equipped with an Intel Core I7 processor, and 8GB of RAM memory.
The developed application is compatible with HMDs as well as smartphones and tablets. However, in order to successfully implement a markerless AR system, the use of HMD, such as Microsoft HoloLens is preferred due to its spatial recognition capabilities. In Section 4, lines 299-301, it has been mentioned, the framework has been developed so that a variety of devices can be utilized.
Comment:
Why was the application not user-tested? User testing could provide valuable information to improve the GUI and also the overall architecture.
Reply:
We apologize for this misunderstanding. The developed application has been tested with actual users who have provided insightful feedback regarding crucial aspects of the framework’s functionalities, including GUI layout and functionalities, communication between the involved parties etc.. The feedback was taken into consideration and the framework has been updated accordingly. Therefore, the following paragraph has been added, in order to explicitly present the testing and validation process of the framework.
Comment:
Figures 7 and 8 are redundant. The information they contain is also in Table 1.
Reply:
Thank you for this comment. Following the reviewer’s comment, Table 1 has been deleted, as Figure 7 and Figure 8 are considered to provide more insightful information in the context of the manuscript. In addition to that, Figure 7 has been modified and replaced by the following figure.
Figure 7 Distribution of time AS-IS situation vs Proposed Methodology
Comment:
Page 10, row 344, formula 3 – Variable ??? is not explained.
Reply:
A short explanation of the variable has been added as follows.
“ Is the number of machines on hand, in the work center i, for period t.”
Comment:
Page 10, row 360, figure 6 – Why is the graph interlaced with a polynomial function? The text should justify why the linear function was not used.
Reply:
As it has been explained in the paragraph, see lines 348-356 in page 10, in Figure 6 the cost of maintenance for the machine used for the validation of the industrial scenario is reflected, in a time horizon of 5 years. In order to construct this graph, confidential data from the industrial have been utilized. What is more, in an attempt to predict how the maintenance costs evolve over a wider time horizon, i.e. 10 years, statistical regression has been utilized and the two polynomial curves have been produced. Regarding the degree of the polynomial, it depends on the non-linear relation between the individual costs regarding the maintenance of machine tools. The following text has been added to support the calculation of the prediction model.
“It is stressed out that due to the non-linear relation between the individual costs of operation and maintenance of machine tools during their lifecycle [32], a 2nd degree polynomial has been calculated.”
Comment:
Page 11, row 387, figure 8 – Graph does not have units for Y-axis.
Reply:
The image has been updated, according to the reviewer recommendation.
Comment:
Used terms should be unified (Scenario 1 = AS-IS SITUATION; Scenario 2 = PROPOSED METHODOLOGY).
Reply:
The terms have been successfully unified, please refer to reply of comment 8.
Comment:
The conclusion should be preceded by a discussion that will comment the advantages and disadvantages of the created system or augmented reality in general, eventually the further extension of the system in the future, or its further testing (including user evaluation).
Reply:
Thank you for this comment. In the revised version of the manuscript a discussion section has been added, summarizing the stages of the framework’s development as well as its strengths in contrast to similar approaches.
6. Discussion
The outcome of the current research work, is a multi-sided, multi-device application, enabling the remote maintenance of complex machine tools under the framework of Industry 4.0. The term multi-sided is used, in order to describe that multiple operators can connect to the platform simultaneously and discuss on a malfunction that has occurred unexpectedly. On the other hand, the term multi-device, is used in order to describe the compatibility of the developed framework in a variety of devices. It is stressed out that the most common devices used for AR implementations are HDMs and handheld devices, including smart-phones and tablets. However, as the scope of the current research was mainly focused on the provision of a totally markerless AR solution, the use of Microsoft’s HoloLens HMD is preferred, since the device supports spatial recognition, which in turn has facilitated the development of the framework. It is stressed out that the use of the HMD is initially intended only for the shop-floor technician, as a means to capture their surrounding and based on that, the expert engineer is capable of placing/registering the virtual content on the user’s physical environment. The development of the framework was performed in a set of consecutive stages. The first stage was based on the conceptualization of the system’s architecture, the discussion of the modules comprising the architecture as well as the tools to be used for the final development. Then in stage two, the basic modules were developed, utilizing the software described in Section 4. As soon as the alpha version of the framework was released, the framework has been internally tested in vitro, in the lab-based machine shop described in Section 5. During this stage, colleague engineers have been asked to test the functionalities of the developed application either from the side of the expert engineer or from the side of the shop-floor technician, aiming at gathering insightful feedback. Then, based on the feedback gathered during the previous stage, the framework has been improved in certain areas, regarding the layout of the GUIs and the functioning of the communication services. Then the beta version of the developed framework has been prepared and tested with the industrial partner in their premises.
Throughout the development and validation stages, meaningful feedback has been gathered. The main strengths of the proposed framework can be summarized to the minimum development effort required by the expert engineer. More specifically, the remote maintenance frameworks are based on the development of AR scenes which are then implemented on an AR-ready device, thus requiring advanced development skills and a certain amount of preparation time. On the contrary with the proposed framework, an expert engineer is capable of creating AR content on the user’s FoV with the use of simple Drag-and-Drop actions. As such, no special skills are required and the time for AR content preparation and implementation is eliminated. Another strength of this framework is the ability to create online sessions. The online sessions are based on the Unity 3D multiplayer API, offering the remote connection of multiple users. As discussed in previous paragraphs, the developed application is compatible with most devices, thus there is no practical limitation on the use of equipment. In comparison with other approaches, presented in [8,16,17,23], the functionality of remote connectivity combined with the broad compatibility of the app, further extends mobility of the user, i.e. the shop-floor technician in an industrial environment. More specifically, in [16] the technicians have to carry a laptop computer along with a USB web camera, in order to establish communication with the expert engineer and proceed with the maintenance operations. The frameworks presented in the above-mentioned publications are using frame markers for the registration of the content on the physical environment. With that said, the performance of the system relies heavily on the quality of the camera used for the detection of the marker, the user’s mobility is limited, as the camera has to maintain visibility of the marker at all times, otherwise the augmentations cannot be registered and updated as the user moves around. In addition to that, the proposed framework, since it utilizes newer equipment with increased computational power, the use of more vivid/realistic virtual objects is feasible. Not only that, but in previous approaches the augmentations are limited to the display of simple schematics, e.g. arrows, text boxes or indicate the component of interest via a wireframe model. On the contrary the current approach can recognize the objects, by the utilization of Microsoft HoloLens.
Comment:
Page 1, row 14 – … Reality (AR).Furthermore … Page 10, row 335 – Desgin of Experiments
Reply:
All the typographic errors have been addressed.
Round 2
Reviewer 1 Report
Revision acceptable, although not 100% response to the questions raised by the reviewer.
Author Response
The authors would like to thank the reviewer for the time and effort put as well as for the constructive comments they raised.
Reviewer 4 Report
Most of my suggestions have been responded appropriately in the text of the paper. I have only two minor comments.
Which technology (software) can be used for communication in the form of a teleconference between the shop-floor technician and the expert engineer.
Information about used hardware and compatible one (Microsoft HoloLens etc.), which are included in the cover letter, should be briefly mentioned also in the text of the paper.
Author Response
Thank you for that comment. The manuscript has been revised so that under Section 4 “Implementation” the following text has been added clarifying the points raised by the reviewer.
“As far as the communication module is concerned and more specifically the teleconference functionality, a variety of solutions has been examined, including Skype, Agora.io [34], GIGA Video Streamer [35] as well as the development of a custom teleconference tool. Among the available solutions, the latter was selected, i.e. the development of a custom teleconference interface. To begin with, the integration of Skype, although it is an appealing solution, and the quality of the service itself is sufficient, it required that it was run as an external application on the Microsoft HoloLens HMD, which had a negative effect on the performance of the app, as well as it was not pleasant for the end-users to switch over to two different applications, one for the communication and one for the AR remote support. On the other hand, the Agora.io and the GIGA Video Streamer are very capable SDKs (Software Development Kit) that could be integrated in the proposed framework. However, these two SDKs are not free. Therefore, the development of a custom module based on the UNet API, discussed above, was the only choice satisfying the framework’s requirements. Concretely, from the expert engineer side, the video stream from the camera and the microphone of the PC was captured and transmitted to the Microsoft Hololens HMD, utilizing the . As it can be seen in Figure 7 (b), from the technician side, a pop-up window appears where the technician can view the expert engineer and through the embedded speakers of the HMD to hear the expert’s voice.
From a hardware point of view, for the on-cite technician a Microsoft HoloLens [36] device has been utilized in order to take advantage of its four environment understanding cameras and the mixed reality capture in terms of sensors, whereas for the development of the framework, and the validation experiments, a laptop PC has been utilized. More specifically the laptop PC is equipped with an Intel core i7 processor clocked at 2.20GHz, 8GB DDR4 RAM and a NVIDIA GeForce 1060 GPU with 6GB dedicated memory.”
