Implementation of Augmented Reality in a Mechanical Engineering Training Context

Abstract

Global industry is at the heart of its fourth industrial revolution, being driven by the emergence of new digital solutions: Augmented reality allows us to consider the evolution towards the “the augmented operator”. This technology is currently little used in higher education, especially for mechanical engineers. We believe that it can facilitate learning and develop autonomy. The objective of this work is to assess the relevance of augmented reality in this context, as well as its impact on learning. The difficulties for a student approaching a technical system are related to reading and understanding 2D and even 3D representations, lack of knowledge on components functions, and the analysis of the chain of power transmission and transformation of movement. The research is intended to see if AR technologies are relevant to answer these issues and help beginners get started. To that end, several AR scenarios have been developed on different mechanical systems, using the relevant features of the AR interfaces that we have identified. Otherwise, these experiences have enabled us to identify specific issues linked to the implementation of AR. Our choice of AR devices and software allows us to have an integrated digital chain with digital tools and files used by mechanical engineers. Finally, we sought to assess how this technology made it possible to overcome the difficulties of learners, in different learning situations.

1. Introduction

After a state of the art on AR in an industrial and higher education context, this article presents the context of our study and our issues, the process of defining an AR scenario with its technical constraints, and finally, the experiments carried out with "Arts et Métiers" students and the results obtained.

We work with engineering students in mechanical engineering. We are focusing our study on the uses and needs in mechanical engineering, as well as on the contributions that AR can have.

1.1. Issues of Augmented Reality in an Industrial and Educational Context

The fourth great industrial revolution underway is driven by the emergence of new digital technologies, including augmented reality (AR) which has pride of place there. The capitalization of information gathered from the Gartner institute makes it possible to visualize the evolution of AR on the Hype Cycle (Figure 1). This technology is now reaching a stage of maturity and is beginning to be used industrially, beyond experimentation.

The use of AR is part of the skills of the 21st century, engineers must be trained. From 2013, augmented reality has shifted from being a technological novelty, to a practical instrument of instruction [1]. Moreover, the HORIZON report places augmented reality as an educational technology with significant advances [2].

A study carried out by Cap Gemini in 2018 indicates that more than 600 companies use augmented reality and find it a potential and a relevance superior to virtual reality [3]. It, therefore, appears essential to prepare the engineers of tomorrow for the use of this technology and for the difficulties it entails.

1.2. Industrial Context

In an industrial context, augmented reality makes it possible to overcome the difficulty of reading technical documents because AR eases making the link between representations and reality when superimposing virtual parts on a real device: AR shows its relevance for assembly operations (time saving), maintenance (reduces human errors and task completion time) [5], as well as training. The presence of invisible risks (heat, electrical voltage, etc.) can also be highlighted.

For maintenance purposes, AR provides both on-the-job support and a training platform:

• Technical support: in a classic context, the user and maintenance manuals available to operators are most of the time subject to interpretation. So, using a maintenance manual requires experience and training. “Paper” manuals are also cumbersome and inconvenient. The cognitive load due to the permanent shifting between the maintenance manual and the real system where the parts must be situated is reduced: AR allows to directly locate the instructions at the right place in the system to maintain, gives intuitive access to the product data, and so enhance the work environment [5].
• AR is also an effective technology platform for training in maintenance field [6,7], even for highly demanding industrial maintenance tasks [8], thanks to the reduction of the required mental workload. Tumler et al. showed that working with an optimal AR system could mean a decreased strain compared to traditional work assistance systems [9].

1.3. Education Context

In the context of engineer training, we find that students have difficulties in mechanical design, for reading technical plans or 3D models, to make the link with the real system, to identify the kinematic chain of a mechanism. A first experiment with students has highlighted the interest of AR [10].

Wang et al. have made a review of AR in education and training [5]. They assume that AR allows learners to have instant access to relevant information for just-in-time learning. They also highlight the saving in training time necessary to master certain tasks.

AR allows for training by doing, providing an engaging experience that allows learners to interact with content while immersed in the real environment. Thus, AR increases motivation and the pleasure of learning. It is motivating for a student to be able to explore and visualize complex phenomena [11]. This results in significantly improved retention [12]. Even though learners may be distracted by the effect of technological novelty, Wang et al. indicate that learners’ attention rose and picked while using AR in learning [5]. Compared to virtual reality (VR), AR increases the feeling of “presence” and improves memorization [13]; this performance delta is also highlighted by Liou et al. [14]. In addition, the potential physiological trauma in VR users cannot be ruled out, particularly following neurological changes [15].

Many other reviews of the literature reveal promising trends and have identified positive impacts of using AR in education [16,17,18]; the advantages include increased motivation and positive learner attitude, improved understanding and learning performance, increased learner engagement, superior performance in physical tasks, improved communication, and group collaboration.

While it is not easy to estimate the actual use rate of augmented reality in an educational context, it is possible to be interested in the number of articles published on this subject. Sirakaya and Alsancak Sirakaya observed an increasing evolution (Figure 2) of this number between 2011 and 2016 [19]. Their work also highlights the different sectors of education involved in these publications. Biology leads with 19.8% of articles while engineering comes second with a total of 12.8%.

The bibliographic study carried out highlighted the potential of the use of augmented reality in an educational context. Additionally, even if AR applications are developed by universities [20], on the other hand, we did not find any study or precise information on the link between the functionalities offered by AR solutions and the needs for technical representation in mechanical engineering. We, therefore, propose to analyze these functionalities, to assess their relevance and their impact on learning.

So our research objectives are:

• Identify the relevant AR specifics in the context of mechanical engineering teaching,
• Implement different scenarios to respond to each educational difficulty,
• Evaluate and quantify the contributions and the effectiveness of AR in helping students overcome difficulties.

2. Study Context and Material Aspects

Now that we studied the literature as a whole, in this paragraph, we describe the context of this work, the process of creating an AR scenario from CAD data, and the technical constraints encountered.

Augmented reality was initially defined by Milgram and Kishimo as a technique for enriching reality through the superposition of information or digital content [21]. Mixed reality fits into the reality/virtuality continuum between tangible interfaces and VR; Figure 3 specifies the interfaces studied in our work. We use wearable devices (head-mounted display with gesture recognition) and handled AR devices (tablets).

Within AR technologies, projection enriches the environment, while the techniques we are testing increase the human, in a way. Projection AR devices are not suitable in our case, because we wish to go around the real system.

2.1. Numerical Process, from CAD Model to AR Experiment

In the context of our use of augmented reality as an educational tool, it was essential to adopt a technical solution allowing teachers as well as learners to build on skills relating to the profession of a mechanical engineer. The different actors should not be forced into time-consuming programming or graphics activities.

The Diota software suite makes it possible to rely directly on a Catia CAD model that students and teachers alike are used to handling. The associated digital chain (described in Figure 4) is as follows:

• Modeling: If you want to superimpose the reality of virtual 3D parts, AR implementation can be complicated, especially in medium-sized companies. Indeed, the models that we want to display must be modeled in digital 3D. As part of our study, the CADs are carried out with Catia.
• Definition of the scenario: Through the Catia Composer software it is possible to define a scenario, where at each step we define the parts to be displayed, information or explanatory texts offered to the user... This step is very important because the scenario must be progressive and take into account the capacity of the students.
• Definition of the tracking model and export: Diota for Composer allows you to define the tracking model for each view. This tool offers different types of project export, either in .diotaplayer for use on Surface tablets, or .diotaproject for use on Hololens glasses. In both cases, the tracking model is converted into a mesh whose fineness we must determine.
• Use of AR: Once the export has been carried out and imported into the final user interface, the scenario is played using the Diota Player (tablet or Hololens version).

If augmented reality is based on reality and adds a virtual overlay to it, it is not necessary to display everything. Following our first study [10], we found that an information overload on a view can confuse the user. It is necessary to highlight essential information on the one hand without masking important aspects of reality on the other hand [22]. We applied this principle to the different scenarios designed for this study, for example with the turbofan engine.

2.2. The Issue of Tracking

The display of information (text or 3D representation) in space without connection to reality constitutes much less added value. This consideration is the heart of many augmented reality works [23,24]. Tracking consists of hooking a real element by identification to a defined virtual element, in order to position the entire virtual overlay in space. Each view can have its own tracking model, which will serve as a reference for the entire augmented representation.

We wanted to use a software solution without programming, which is compatible with the Dassault System CAD solution, so teachers and students can use it without a huge amount of programming knowledge, programming time and without losing contact with their core work tools. Among the available platforms, only the Diota solution allows this.

However, it offers tracking by recognition of 3D part shape. Whether through the multiple cameras of Hololens glasses or the main camera of a Surface tablet, the Diota Player identifies the real tracking part(s) defined in Diota for Composer. Compared to 2D targets, this offers several advantages: Regardless of the user’s positioning in relation to the real system, the tracking works. Moreover, virtual parts can be perfectly positioned in 3D compared to real parts.

This nevertheless requires carrying out a prior manual approach, but this initialization does not have to be repeated.

In order for this process to work, it is necessary to have an extremely faithful 3D model of the parts, and this is where a paradox comes in: if the tracking model is too simple, it cannot be recognized in a singular way (a portion of the tube not being unique on the real model for example) but if it is too complex, on the one hand, it will have a significant weight (causing slowness in the use of the scenario) and on the other hand the parts concerned will be more difficult to model. To overcome the problem of the complexity of modeling, or if we do not have a digital definition of the part, we carried out tests with the scan of the housing of a gearbox allowing us to obtain a cloud of points in 3D. If we define this scan as a tracking model, it works perfectly well (Figure 5). The visual rendering of the part is not aesthetic but this is not disturbing insofar as in the case of the scenario explaining the operation of a car gearbox or its disassembly/assembly, we do not need to display the casing.

Depending on the final user interface, this notion of tracking should be approached in a different way.

2.2.1. With the Surface Tablet

• Performance aspects: The storage and calculation capacities of the tablet allow fine parameterization of the tracking model mesh (around 10,000 polygons).
• Tracking aspects: The tracking model is the only reference taken into account by the tablet’s Diota Player. This means that this model must always remain in its entirety in the field of view of the camera of the tablet. This constraint can be problematic, especially when it is necessary to get closer to observe details, or when we have to step back. One way to partially overcome this problem is to define different reference parts (tracking model) depending on the scenario views. For example, it is the casing of the machine tool that is used first when we want to give an overall presentation of the machine. Then in a second step, to visualize the interior of the machine (its axes, internal components, etc), it is necessary to get closer and therefore to find another tracking object: an internal plate was used. We can see in Figure 6 the tracking model chosen (in blue) superimposed on a real machine: on the left of the figure, it is the machine body and on the right the internal plate.

2.2.2. With Hololens

• Performance aspects: The storage space, as well as the available computing power (Hololens1), makes it necessary to limit the fineness of the mesh of the tracking object (around 1000 polygons). It must be chosen precisely enough to allow identification but not too much in order to limit the technical strain on the eyeglasses.
• Tracking aspects: The entire environment (the room, tables, chairs, etc.) is constantly scanned by the eyeglasses. This allows great stability of the virtual representation once the hooking has been made, whether or not the tracking model remains visible by the cameras of the glasses. However, this technology is therefore in essence not applicable to a system whose position would change in relation to its environment. Thus, a tracking calibration may become null and void following a change in the layout of the room in which the system studied is located.

It is therefore necessary to adapt the tracking object according to the objective of the scenario phase and to consider several tracking objects during an AR scenario.

3. Pedagogical Constraints

We describe here the educational difficulties encountered, and the AR functionalities tested to remedy them.

In the context of engineering training, we find that students have difficulties in mechanical design: reading technical sketches or 3D models, making the link with the real system, or even identifying the kinematic chain of a mechanism. We have identified relevant AR functionalities to overcome these difficulties.

The AR solution used includes several functionalities that can support the needs of perception or technical representations, in the teaching of mechanical engineering:

• Increased perception of 3D shapes: The immersive aspect of using AR is contingent on thinking about what you want to show when making the script. Indeed, part of the parts of the system can be visible directly in the real world and it is not necessary to display the virtual version. However, it may be interesting to display the bottom of the housing, or to let the real hide part of the virtual to reinforce the feeling of depth and thus multiply the immersive aspect of the experience. The representation in the context of a virtual solid piece, around which one can move around also facilitates the perception of all its forms. Overall, spatial acuity is improved [25,26].
• Designate a part or specific point of a mechanism: Highlighting can be achieved through the addition of labels, text, or access to the nomenclature. Certain parts can also be highlighted thanks to a particular color, also to visualize an internal part not normally visible (Figure 7).
• Help with the analysis of a functioning: AR allows the contextualization of a mechanism, within a system represented virtually (Figure 8). The circuits, the flows can be highlighted. By transparency effect, it is possible to display parts or mechanisms internal to the system, to add colors and/or indications, or even to implement animations in order to explain various functioning. It can be, for example, the input and output of a system, its kinematic chain, its operating modes.
• Work instructions (help with the implementation): It is possible to visualize how to actuate a mechanism, to position a tool in the right place and with the right gesture. The AR can also make it possible to identify the tool to be taken from a tool trolley (Figure 9) or to ensure the conformity of an assembly (verification of the presence or position of components). AR is very relevant for procedural purposes: The gestures are superimposed directly on the real system that the operator is working on, at the right place. The working instructions are given step by step. Access to numerical datasheets can be linked to the system components. For more details please refer to [27]
• Visualization of an invisible characteristic: In order to ensure the safety of workers, dangerous areas or rooms can be highlighted (high temperature or high voltage for example).
• Remote expertise: Synchronous collaboration between the operator and an expert. The guidance and targeted information are provided by a distant expert, thanks to AR glasses. The expert sees the scene by the glasses camera, and he can give the operator audio and visual instructions superimposed to his direct vision.

4. Experiments and Results

In the following section, we, therefore, propose to evaluate the gains for different teaching situations, through tree scenarios. In order to ensure the relevance of the experiments, they were all carried out on real industrial systems. So, for example, we worked with an airplane engine or even an automobile gearbox rather than on fictitious educational models.

4.1. Electric Gate Actuator Experiment

A first experiment with students highlighted the interest in AR and its impact on learning [10]. This study consists of evaluating the ability of second-year bachelor and first-year engineering school students ("Arts et Métiers" Sciences and technologies), to understand a mechanical system with or without augmented reality. The chosen students had a profile with poor knowledge of mechanical design at the time of the experience.

4.1.1. Conditions of Experiment

This takes the form of a 2 h practical work, where students should try to understand an electric gate actuator, as well as an internal release mechanism (Figure 10).

We provided to each student:

• The functional system put in situation (Figure 10),
• All the system spare parts,
• The technical documentation (plan, exploded view) and a digital 3D representation (on computer),
• A guide for the study of the mechanical system: Students were asked to analyze the functioning, and model and calculate values.

Students were divided into two groups of identical size and they worked at different times with or without AR scenario help:

• Half students had to study the system from only the “paper” documentation and CAD representation,
• The other half had the same elements, but also an augmented reality scenario, available on Hololens glasses and for Surface Pro tablet (Figure 10).

4.1.2. Procedure

They work during the study for about 2 h, handling the operating system, spare parts, documents, and AR interfaces for some.

Independently of the deliverable (practical work report), the same quick individual questionnaire was distributed to all participants immediately after their study of the mechanism.

From their answers, a score (out of 20) was assigned to them to estimate their level of “hot” understanding of the system, and thus identify the impact of the use of AR devices on analysis and comprehension. For the sake of relevance and to avoid any bias, the rating was carried out blind.

The questions were related in particular to:

• Inputs/outputs, energies used.
• The location of components fulfilling certain functions.
• Identification of the kinematic chain and movement transformations.
• Understanding of the internal release mechanism.

Results of a first iteration were published in [10], but since then the experiment has been enriched: results were refined with new cohorts (110 students in all). However, beyond understanding the system, we wanted to assess the usability of AR devices (user interface, hardware using), ergonomics (“how did you feel with the device”), and utility feel.

The impressions and comments (of users of AR tools) were collected via another part of the questionnaire. Students were asked if:

• The AR was a help for the understanding of the mechanism,
• The AR interfaces were easy to use/ergonomic/heavy/uncomfortable/useful/fun/relevant,
• Compared to the use of paper plan, the use of AR has made it possible to:

  –

  Better perceive the location of parts,

  –

  Better identify the shapes of parts,

  –

  Better identify the kinematic chain of transmission and mobility,

  –

  More easily locate the location of the tool and the gesture to disengage,

  –

  Better understand the release mechanism,

  –

  Better understand the functions of parts or sub-assemblies,

  –

  Save time.

4.1.3. Results

• Understanding analysis

Figure 11 represents the breakdown of student scores: for the students who used the AR scenario (in blue), the shift demonstrates a best understanding and apprehension of the system. Thus, it appears that on average the students who had access to augmented reality obtained scores 22.6% higher than the students who worked with traditional documentation.

• Users Experience Analysis

Overall user satisfaction with the technology is 93.3%, 84.4% find it useful and 91.1% consider it to save time. A total of 73.3% believe that compared to the use of the paper documentation, the use of AR made it possible to better perceive the location of the parts.

4.2. Aircraft Turbofan Engine Experiment

This second experiment concerned the study of the operating principle of a double-flow turbo reactor. This included identifying components, high- and low-pressure kinematic sub-assemblies, and identifying air and gas flows. The 3 h of practical work allocated to this experience took place as follows:

–

Study of the system from classical documentation

–

Assessment of the understanding of the subjects using a questionnaire

–

System study with an AR scenario developed for Hololens glasses and Surface Pro tablet (Figure 12)

–

Assessment of the understanding of the subjects using the same questionnaire.

The scenario proposed to the learners consisted of four stages (called “views” in the Diota solution):

–

View 01: Macroscopic presentation of the various components of the engine (Figure 13)

–

View 02: Highlighting of the high pressure assembly (HP): high pressure compressor + high pressure turbine (Figure 12 and Figure 14)

–

View 03: Highlighting of the low pressure assembly (LP): low pressure turbine + epicyclic reduction gear + fan

–

View 04: Animation of rotation of the HP and LP shafts. This animation makes it possible to explain the functioning of the engine thanks to the different rotation speeds for the 3 elements concerned (HP assembly, LP assembly, fan).

Each view is accompanied by an explanatory text. This text is present on the Diota dashboard. In the case of the Hololens, it is in a window displayed in augmented reality, which can be placed anywhere in space (Figure 14).

However, the low number of participants for the moment does not allow us to identify representative trends. While the quantitative aspect of this study cannot be exploited, the difference in student interest and motivation between the two study phases was obvious.

4.3. Automobile Gearbox Experiment

During the presentation of a car gearbox without AR, students were not able to correctly identify the different shafts that made it up. The AR scenario eases the understanding by highlighting the characteristics of the system (kinematic chain, nomenclature, the principle of gear shifting, etc.) and facilitates the transmission of certain information (Figure 15). The scenario tested also allows to guide the phases of dismantling the components of the gearbox, without any errors observed on the parts to be handled at each step. This scenario remains to be improved, particularly with regard to the notion of occlusion. It is necessary to correctly manage the occlusion of the real by the virtual, as shown by [27]. Thus, the user can better perceive where the part to be added is placed during assembly, compared to the parts already in place.

4.4. Analysis of Uses and Correlation Technical Needs vs. AR Features

It is essential that an AR scenario is adapted to the targeted skills (learning outcomes) [5]. Beyond the immersive technology hype, we wanted to contextualize its use in order to assess its added value.

After all our carried-out scenarios iterations, we capitalized on our feedback in order to match technical and educational needs with the optimal AR feature. The matrix (Figure 16) highlights the relevance of the functionalities offered by the AR solutions that we tested. (The mark (X) is used when a feature can be used in some particular case but in general, it is not the best one to use.)

This correlation will now help AR scenario designers to focus on their operational objectives, by quickly identifying the AR features that best meet their needs. It may also encourage non-initiated people to get started using AR.

5. Conclusions and Perspectives

The first contribution of augmented reality in terms of support for engineering education in higher education is to familiarize the engineers of tomorrow with a technology with which they will most likely be confronted in their careers, in order to perceive its potential and imagine its uses in the industry of the future.

Our goal is not to promote a tool, but to seek to improve the transmission of knowledge. This is the second major contribution of augmented reality. Indeed, we have been able, through various experiments, to measure that this technology really helps in understanding complex mechanical systems.

Moreover, unlike traditional documentation which provides the user with a large amount of information in which he can get lost, augmented reality scenarios only show what their designer has decided, and at the appropriate time.

Thus, augmented reality is proving to be a relevant support tool for pedagogy in an engineering context:

• In terms of user experience for learners, 93.3% were satisfied,
• Generating an easier understanding of complex systems since on average, the learners who benefited from the AR obtained 22.6% better results than others.

While each time the interest of the learners in augmented reality technologies was palpable, it is however important to distinguish the understanding of the functioning of a system in particular, and the intrinsic ability of the student to understand complex systems. Indeed, if augmented reality makes it possible to transmit knowledge about a specific system, this does not affect instantly its analytical skills. It is more appropriate to see this technology as a promising support tool, allowing to guide the student, to facilitate the analysis of a mechanism and be more efficient; but not as a magic tool multiplying the capacities of the learner. However, it does not seem realistic for the moment to envisage a complete replacement of conventional tools in favor of AR.

Other works remain to be done around augmented reality tools, in the context of educational use. We aim in particular to explore the possibilities offered by this technology in terms of remote collaboration.