Novel Mixed Reality Use Cases for Pilot Training

Abstract

This study explored novel mixed reality (MR) use cases for pilot training using a mix of methods rooted in the general innovation theory of dynamic capabilities. The aim was to identify areas of improvement for various aspects of the flight training based on MR, in a socially and economically sustainable manner. Multiple surveys and workshops have been conducted with flight instructors, administrative staff, pilots and student pilots. The main result of this study is a systematic identification of the three most promising MR use cases: interactive theory training, cockpit procedure, and outside check training. These results are important because they inform the development of technical didactic tools for pilot training. The applicability of MR technologies to accommodate diverse user needs and preferences is addressed, while also considering aspects of economical sustainability.

1. Introduction

Pilot training today is not significantly different from what it was decades ago. It is mainly based on synchronous learning and on-site training. However, efficiency and a high training quality are now more important than ever. Depending on the type of pilot training, students and instructors may come from anywhere in the world, and this means traveling and scheduling should be as efficient as possible. Similarly, the expensive training equipment of approved training organisations (ATOs) should be utilized efficiently and at full capacity. New technological advances such as off-the-shelf MR devices provide promising possibilities to improve and innovate use cases in pilot education. Nonetheless, novel MR use cases in pilot training have to cover multiple aspects to be useful in practice. Business-relevant factors must be taken into account as well as the training process, and the diversity of students’ learning needs and preferences.

This study aims to explore promising MR use cases while taking multiple stakeholder views including students, pilot instructors, ATOs and pilots into account. This diverse group of stakeholders helped to include gender, business, technical and also domain specific aspects. Attention was paid to formulate use cases independent of the current state of the art, such as currently available MR devices, to define use cases which will be relevant for the next generation of devices or the one after that.

1.1. Business Innovation in Pilot Education

Flight training is a time and resource consuming process for both initial and recurrent pilot education. A high-quality flight training creates value and positively impacts the trained pilots and the passengers, and should also be economically sustainable for both commercial aircraft operators as well as ATOs. ATOs aim to provide optimal training while also meeting economic goals. This requires them to optimize their processes and strive for innovation in the sense of Schumpeter [1] in order to be competitive on the market. Novel technological developments such as MR provide manifold opportunities in various domains for process improvement and even new business models [2,3]. Guided by the general innovation theory of dynamic capabilities [4], and the use-case technology-mapping (UCTM) innovation framework [5], which is based on this theory, we explore novel MR use cases for pilot training. We thereby aim to identify areas of improvement for various aspects of the flight training business ecosystem [6] and seek to identify use cases where MR can be used to improve the value proposition in the sense of the wide-spread business model canvas concept [7,8] to the pilots in a gender-sensitive way, while the value creation aspects for ATOs can also be optimized.

1.2. Augmented/ Mixed Reality

The concepts of augmented reality (AR), virtual reality (VR) and MR can be structured in a common context as the reality–virtuality continuum defined by Milgram et al. [9] (see Figure 1). Chronologically, the first term which appeared was VR, coined by Jaron Lanier [10]. However, systems which present an immersive six degrees of freedom virtual environment already existed [11]. The term AR was defined later in the 1990s by Caudell and Mizell during their work at Boeing [12]. They aimed to improve aircraft manufacturing by using a see-through head-mounted display. A few years later in 1994, Milgram et al. created the reality–virtuality continuum [9]. Since MR describes a large section between the real and a virtual environment, AR is also a part of MR. AR applications in aviation share a long common history, and manufacturing and maintenance use cases are investigated with particular frequency [13,14,15,16,17]. In this area of application, use cases based on AR process guidance are common. For example, step by step assembling of aircraft components or guided step by step maintenance/repair tasks for workers. The step by step protocol for learning in virtual environments in particular ensures a long-term retention of knowledge [18]. Another area of interest is education [19,20,21]. Research has uncovered the potential of AR technology to change and improve learning and teaching environments and experiences [22,23,24]. This potential can also be identified for AR-supported pilot training [25,26,27], as shown in various implementations [28,29]. The capabilities of modern MR/AR devices can thus be used to enrich training materials and to create an engaging learning environment for meeting the needs of the next generation of pilots [30,31,32].

1.3. Pilot Education

Type rating (TR) training gives an advanced qualification to pilots that is required for obtaining, maintaining or renewing the entitlement to fly a particular type of complex aircraft, listed in the “EASA type rating and licence endorsement lists”, as pilot in command (PIC) or first officer (FO). A TR course consists of a theoretical part and practical training. For the practical part mock-ups, flight simulators and/or real aircraft can be used. Furthermore, type rating instructors (TRI) and examiners (TRE) are needed. TREs relate to EASA regulations for all EASA member states in Europe. The requirements for TRIs and TREs are regulated within the European Union (EU) by the Commission Regulation (EU) No 1178/2011, Annex I —Part FCL, Subpart J and K [33]. The regulations valid in the United States can be found in 14 CFR Part 61 [34]. The TR course program is dense and intensive. Some TRI/TREs and participants travel long distances to attend the course, because there are very few simulators anywhere for some specific aircraft types. TRI/TREs and trainees are often already employed by an airline or business aviation company. TRI/TREs need to be up to date and experienced in flying the aircraft type, and this mean TRI/TREs often have multiple job assignments (e.g., pilot and instructor/examiner). TR trainings are planned on a yearly basis and trainees can register for these fixed dates. The challenge for the ATOs is to organize TRI/TREs for these courses and to consider their rosters, so the ATOs are facing various constraints in terms of time, availability, content and costs. Various course contents are taught by different TRIs. The absence or delay of a TRI/TRE (e.g., due to illness or travel issues) can be problematic for a course.

These conditions are reflected in the experience and performance of the trainees during the training. In a survey report, Moesl et al. [26] noted that a number of TR pilots and instructors mentioned difficulties related to the learning conditions of a TR course, such as stress and time pressure, issues with self-study and differences in the previous knowledge trainees possess. The main mitigations proposed by instructors to address these difficulties were an extension of the course duration, or a diversification of training tools that allow asynchronous learning (e.g., instructional videos, mock-up or procedure trainers) complementary to attending the full-flight simulator sessions with an instructor. The main mitigations proposed by trainees were an extension of the classroom instruction, the use of additional computer-based training (CBT), allocating more time to difficult topics and deviating from the course plan to address individual needs.

1.4. Gender Aspects Related to Flight Training

Male pilots represent more than 90 percent of the pilot population worldwide [35]. Learning needs and preferences of male pilots tend to be generalized to both genders. Although competency standards in aviation are the same for both genders, an interesting research question is how each gender group reaches those standards and how they cope with more or less conservative training methods. Gender differences in flight training have been described in various studies [36,37,38,39,40]. Designing an inclusive and socially sustainable pilot training program requires that diversity aspects are identified, communicated and addressed [40]. Many gender differences related to flight training are rooted in different experiences and starting conditions typical for each gender [36]. As research on cognitive processing in the general population shows, women perform better in verbal tasks and men perform better in visuospatial tasks such as mental rotation of objects [41]. However, gender differences in these starting conditions can vanish when the learning methods are appropriately designed. Both the practice and research show that trainees of both genders can reach similar levels of performance despite initial differences. An experiment on mental rotation tasks has shown that gender differences in the performance of two-dimensional (2D) visuospatial mental rotation tasks decreased after training [42]. Even more remarkable, when the mental rotation tasks were performed in 3D, gender differences were absent [42], perhaps because no mental reconstruction of the 3rd dimension was required. These findings are confirmed by gender sensitive flight training research on upset recovery, which requires the reestablishing of straight-and-level flight by managing an aircraft’s 3D rotation and speed [43]. Bauer et al. [43] investigated an inclusive flight training method for upset recovery allowing for 3D handling in a flight simulator, as compared to theoretical instruction that was state-of-the-art for particular recovery characteristics. Their results [43] showed that trainees from both gender groups reached similar levels of performance in upset recovery during post-test, after receiving precisely the same training, and this despite initial gender differences in the pre-test. Thus, the appropriate, inclusive method of instruction can support trainees in reaching similar levels of performance with the same amount of training.

2. Materials and Methods

2.1. Research Approach

This study aims to explore innovations for pilot training based on MR technologies using a gender-sensitive, socially sustainable approach. Another aim is to support ATOs in their innovation efforts of providing more value to their customers in an economically sustainable fashion. This innovation process is guided by the dynamic capabilities theory [4] and the UCTM framework [5], which facilitate a structured business model and process innovation approach.

The dynamic capabilities theory is a strategic management concept, which consists of three main process steps (1) Sensing, (2) Seizing, and (3) Reconfiguration, which can be subsumed by constantly checking for new technical developments and business opportunities (Sensing), designing and developing concrete concepts (e.g., business models and investment decisions) which may realize the identified opportunities (Seizing), and implementing the innovation into the real-world organization to finally profit from the innovation (Reconfiguration) [4].

The UCTM framework [5] (see Figure 2) is an operational business process and business model innovation concept. It consists of a technology-driven (right part of the framework in Figure 2) and a human-centered and process-driven (left part of the framework in Figure 2) approach for opportunity scouting, and aims to map technologies with promising use cases to ultimately create process, service, product, or business model innovations (center of the framework in Figure 2) [5].

In this study, the technology that forms the basis of the innovation process was predetermined in the form of MR. Consequently, the study focused on the human-centered and process driven part of the framework to identify use cases, that could benefit from the novel technological opportunities. In order to identify these potential use cases and create novel forms of pilot training, we followed the detailed use case scouting and the process and needs analysis concept depicted on the left part of Figure 2.

Figure 2. The UCTM framework modified from [5] guiding the service innovation process.

Figure 2. The UCTM framework modified from [5] guiding the service innovation process.

Different methods and data sources were used in this study, in order to develop use cases from a holistic perspective. As illustrated in Figure 3 module 1, high-level workshops were conducted with ATOs, where the focus was on understanding the challenges in pilot education, and on identifying potential areas for improvement. With this information, use cases were specified at a high level without detailed content. In addition, the results of two studies [26], ref. [25] were used as a basis for determining the potential content of the use cases (see Figure 3 module 2 and 4). In module 3, the promising syllabus content was determined, and in module 5, outcomes of all sources were merged to specify detailed use cases for pilot training.

2.2. Participants

Different groups and participants were involved depending on the data collection methods. Notwithstanding the gender imbalance in aviation, an intention was set and particular efforts have been done to identify and include as many female participants as possible for this study.

Participants of the initial workshops (see Figure 3 module 1) were two senior managers (two men, one with a business degree, and the other with an engineering degree and also an active airline pilot) and two middle managers (one woman with a business education degree and one man with an engineering degree and also an active pilot instructor) from different ATOs.

The workshops of module 3 and 5 were attended by the same group of participants as in module 1, with one exception. The male middle manager (Chief-of-Training) was not available for the workshop and was replaced by a flight instructor, also male. All participants of module 1, 3 and 5 had a basic understanding of AR and VR, furthermore, before the first workshop they received an introduction to Microsoft HoloLens.

In module 2, a survey was conducted that is described in detail in [25]. Participants of the survey were 48 male and 12 female pilots or student pilots.

In module 4, another survey was conducted with pilots and instructors, which is described in detail by Moesl et al. [26]. Participants of the latter survey were 24 male pilots and 7 female pilots, holding type ratings e.g., for B767, A319/320/321, or Cessna C525, C500 series. In addition, 22 Type Rating Instructors participated in the latter survey.

3. Results

The present study describes three MR use cases for pilot training, which are shown in Table 1, Table 2 and Table 3. The needs of ATOs, pilots, pilot instructors and pilot students were taken into account, by merging results from different studies and workshops. An overview of the method is outlined in Figure 3 which is subdivided in five modules, where the interim results of each module are discussed in Section 3.4.1, Section 3.4.2, Section 3.4.3 and Section 3.4.4. The final results represented as module 5 are shown below in Section 3.1, Section 3.2 and Section 3.3.

3.1. Use Case—Interactive Theory Training

Interactive theory training was the use case best rated by ATOs during the high-level workshops (module 1). The “flight management system” was the best rated syllabus topic according to feedback from pilots, instructors and ATOs representatives. However, the participants of the final workshop (module 5) did not propose to pursue this topic because they did not expect it to have a high impact on training. Instead, they selected the “electrical power supply” as one of the most difficult topics based on feedback from TR trainees in module 4 [26]. The participants argued that for trainees, the practical relevance of this knowledge is often underestimated because it is only needed in rare emergency situations. However, this training content and its improvement through MR technology were considered important. Thus, the electrical power system was selected as a high impact MR development topic.

Table 1. Use case interactive theory.

Triggering Event	First time, during the classroom training, after the introduction of the electrical system. A second time, during the simulator training where the electrical system is explained. Syllabus content “electrical power supply”.
Description	The effects of a fuse and generator failure on the electrical system will be explained in an interactive manner.
Actors	TR student (TRI (configures simulator in simulator training), (helps students in classroom training)).
Precondition	The student sits in a simulator of a business jet, or in a classroom in front of a paper/wood mock-up and is wearing the MR device (e.g., HoloLens).
Flow of activities	For the classroom training, all buttons and controls will be holographic. For the simulator training, the physical buttons will be highlighted using MR, the student must use the physical buttons of the simulator. The application workflow is in both scenarios the same. After 15 s of training, the failure indicators of the electrical system get illuminated. The student must initiate the correct actions to prevent a failure of essential instruments. Subsequently, the MR application will show the student a schematic representation of the electrical system and the student is asked to identify the failure in the schematics and point out which instruments are affected.

Table 1. Use case interactive theory.

Triggering Event	First time, during the classroom training, after the introduction of the electrical system. A second time, during the simulator training where the electrical system is explained. Syllabus content “electrical power supply”.
Description	The effects of a fuse and generator failure on the electrical system will be explained in an interactive manner.
Actors	TR student (TRI (configures simulator in simulator training), (helps students in classroom training)).
Precondition	The student sits in a simulator of a business jet, or in a classroom in front of a paper/wood mock-up and is wearing the MR device (e.g., HoloLens).
Flow of activities	For the classroom training, all buttons and controls will be holographic. For the simulator training, the physical buttons will be highlighted using MR, the student must use the physical buttons of the simulator. The application workflow is in both scenarios the same. After 15 s of training, the failure indicators of the electrical system get illuminated. The student must initiate the correct actions to prevent a failure of essential instruments. Subsequently, the MR application will show the student a schematic representation of the electrical system and the student is asked to identify the failure in the schematics and point out which instruments are affected.

3.2. Use Case—Outside Check

Different elements of an outside check use case emerged during the high-level workshops (module 1). Although multiple slightly different versions of outside checks rated better than other high-level use cases, only one outside check use case was defined in detail to have more diversity in the results. “Engine including auxiliary power unit” was the best rated syllabus topic for this high-level use case by pilots, instructors and ATOs representatives. The discussion between the participants, however, did not show a sufficient advantage of an MR utilization for this content. The most beneficial topic for AR from TR students and TR instructors (see [26]) was selected: “aeroplane external visual inspection; location of each item and purpose of inspection”. For the ATOs, this topic is also interesting from an economic and organisational perspective, because the aircraft which is needed for the training is a valuable and limited resource. Depending on the level of training (basic, advanced) the aircraft used differ in size, complexity and, of course, costs. For basic training with light aircraft, the access to the aircraft at an airport can be a logistic issue, because of restrictions (security checks, access cards). Furthermore, at the airport there are also insufficient opportunities to find aircraft defects during checks, and simulation is the more appropriate environment for experiencing safety critical training contents [44,45]. In advanced training (e.g., TR) normally only simulators of complex aircraft are used, and the visual inspection is not taught on a real aircraft. Complex aircraft are much more expensive, and the objective for the operator of such an aircraft is to maximize the flight hours. Ground time is non-earning and therefore must be minimized. This results in the effect that an operator always prefers an aircraft to be flying and has only little interest in providing it for outside-check training, unless the ATO would be ready to pay for this, which would create additional costs and availability issues. Consequently, it was decided that the implementation of this use case should not depend on a real aircraft, but should instead make a visual representation that would be as precise as possible. The actual size and proportions of the aircraft would be maintained. A physical walk around of the virtual aircraft should allow the student to train as realistically as possible.

Table 2. Use case outside check.

Triggering Event	During the classroom training while discussing syllabus topic “Aeroplane external visual inspection; location of each item and purpose of inspection”.
Description	Description of elements and the outside check are carried out on a virtual aircraft model. During the check, the aircraft is augmented with virtual/holographic elements, showing what should be checked next and how. Interactive elements with explanations are included.
Actors	TR Trainee
Precondition	The trainee is standing in a large enough room in front of a virtual model of a business jet (e.g., Cessna C525 CJ1+).
Flow of activities	A full scale virtual model of a business jet is displayed in front of the trainee. For this, a sufficiently large room is necessary (e.g., Cessna C525 CJ1+ has a model length of approximately 13 m, wingspan approximately 14.5 m). The trainee walks around the aircraft model and inspects specific parts of the aircraft according to the procedure. The MR application will indicate actions that need to be performed during the inspection such as turning handles, opening doors, etc. Should any parts be out of reach (e.g., if you must climb around the aircraft model in order to reach them) the virtual aircraft model will be orientated and moved accordingly.

Table 2. Use case outside check.

Triggering Event	During the classroom training while discussing syllabus topic “Aeroplane external visual inspection; location of each item and purpose of inspection”.
Description	Description of elements and the outside check are carried out on a virtual aircraft model. During the check, the aircraft is augmented with virtual/holographic elements, showing what should be checked next and how. Interactive elements with explanations are included.
Actors	TR Trainee
Precondition	The trainee is standing in a large enough room in front of a virtual model of a business jet (e.g., Cessna C525 CJ1+).
Flow of activities	A full scale virtual model of a business jet is displayed in front of the trainee. For this, a sufficiently large room is necessary (e.g., Cessna C525 CJ1+ has a model length of approximately 13 m, wingspan approximately 14.5 m). The trainee walks around the aircraft model and inspects specific parts of the aircraft according to the procedure. The MR application will indicate actions that need to be performed during the inspection such as turning handles, opening doors, etc. Should any parts be out of reach (e.g., if you must climb around the aircraft model in order to reach them) the virtual aircraft model will be orientated and moved accordingly.

3.3. Use Case—Procedure Training

The procedure training use case and the preliminary airport exercise (familiarization with an airport and its infrastructure/conditions) were rated equally well in module 1. Procedure training was also identified as a promising MR application area in module 2 [25]. Thus, this was chosen as the third use case. The participants in module 5 did not agree with the syllabus topic that was top-rated in the pairwise comparison (module 3). Instead of this, the topic “Use of checklist prior to starting engines, starting procedures, radio and navigation equipment check, selection and setting of navigation and communication frequencies” which TR instructors rated as most beneficial for AR [26] was selected. Based on this topic, the MR application could allow the trainees to learn some elements remotely and asynchronously, in advance of their on-site TR course. This would relieve the time pressure experienced by some trainees in conditions of synchronous learning [26], while in addition it could also spare resources of the ATOs.

Table 3. Use case pre-flight procedure training—cockpit poster.

Triggering Event	Student wants to practice cockpit preparation and APU ground starting for a business jet.
Description	Cockpit preparation and in-flight starting procedure are shown step by step to a trainee. A cockpit poster is enriched with virtual/ holographic and animated content to increase the immersion and to visualize each step of the procedure in more detail. The trainee can navigate through the individual steps and interact with certain cues of the cockpit model at each step.
Actors	TR trainees
Precondition	The trainee sits in front of a business jet cockpit poster (e.g., Cessna 560 XLS) and wears the MR device with the application installed.
Flow of activities	The flow of activities is based on the “cockpit preparation” and “APU ground starting” checklists. The activities can be divided into verification, setting changes and conducting function tests. Using the MR application, the trainee navigates through the individual steps in the correct order by watching the animations. The MR device will highlight and animate the corresponding switches and control elements. The trainee interacts with specific elements of the cockpit in each step (e.g., press a button and continue to the next step).

Table 3. Use case pre-flight procedure training—cockpit poster.

Triggering Event	Student wants to practice cockpit preparation and APU ground starting for a business jet.
Description	Cockpit preparation and in-flight starting procedure are shown step by step to a trainee. A cockpit poster is enriched with virtual/ holographic and animated content to increase the immersion and to visualize each step of the procedure in more detail. The trainee can navigate through the individual steps and interact with certain cues of the cockpit model at each step.
Actors	TR trainees
Precondition	The trainee sits in front of a business jet cockpit poster (e.g., Cessna 560 XLS) and wears the MR device with the application installed.
Flow of activities	The flow of activities is based on the “cockpit preparation” and “APU ground starting” checklists. The activities can be divided into verification, setting changes and conducting function tests. Using the MR application, the trainee navigates through the individual steps in the correct order by watching the animations. The MR device will highlight and animate the corresponding switches and control elements. The trainee interacts with specific elements of the cockpit in each step (e.g., press a button and continue to the next step).

3.4. Interim Results

This subsection presents the interim results from the modules 1–4 (see Figure 3) and provides detailed insights into the data collected from multiple stakeholder groups.

3.4.1. Interim Results Module 1—High-Level Workshops

This section discusses the interim results shown in Figure 3, module 1. In a first step the overall training process and general statistical data related to generic flight training were documented. This was the foundation for the workshops to narrow down the areas where use cases should take place. The workshops revealed 11 high-level use cases which are shown in Table 4. The ATOs prioritized the use cases, based on nine criteria:

• frequency of occurrence,
• expected training quality improvement,
• expected negative training quality impact,
• financial benefit for pilot school,
• promotion of gender diversity,
• resource savings regarding pilot instructors,
• increased flexibility for general resource planning,
• benefits for the ATO and
• benefits for the student.

The most promising use case of the interactive theory was rated with the maximum possible points for training quality improvement and benefits for student, with minimal negative impact for training. Three different variants of the outside check followed in ranking. On the fifth place there were two equally-rated use cases: “Cockpit procedure training—Poster” and “Preliminary exercise—Airport”.

Table 4. High-level use cases for flight training.

Rank	Name	Description
1	Interactive theory	Complex objects and procedures are displayed interactively, e.g., exploded view drawings of a engine, interactive representation of the lift equation etc.
2	Outside check—VR quiz	The students are given tasks to do on a virtual aircraft. e.g., to find the drain valves, or to carry out a process sequence in the correct order. There are points for correct answers (gamification).
3	Outside check—VR introduction	The outside check is carried out on a virtual aircraft. During the check, the systems indicate what should be checked next. Interactive elements with explanations are included, e.g., how the cable works on an aileron.
4	Outside check—AR introduction	The outside check is carried out on a real aircraft. During the check, an AR device indicates what should be checked next. Interactive elements with explanations are included, e.g., how the cable works on an aileron.
5	Cockpit procedure training—poster	The cockpit procedure is shown step by step and visualized using AR on a cockpit poster. For example, a button which needs to be pressed is marked red. No haptic feedback is included for this.
5	Preliminary exercise—airport	The airport is displayed in detail using an MR device. The student can familiarize himself/herself with the traffic pattern, entry points, exit points, approach and departure.
6	Outside check—AR quiz	The students are given tasks in AR to do on a real aircraft. e.g., to find the drain valves, or to carry out a process sequence in the correct order. There are points for correct answers (gamification).
7	Preliminary exercise—mental image	Student experiences a traffic pattern in 3D that can be stopped at any time. The student can experience when to send radio messages, when to press a button or push levers.
8	Cockpit procedure training—flight simulator	The cockpit procedure is shown step by step and visualized using AR on a flight simulator cockpit. For example, a button which needs to be pressed is marked red.
9	Supporting cues—flight simulator	During the flight, instruments are expanded using AR to show additional information, e.g., display cues to fly into a holding or to fly a standard curve.
9	Cockpit procedure training—virtual	The cockpit procedure is shown step by step and visualized using a VR cockpit. For example, a button which needs to be pressed is marked red.

Table 4. High-level use cases for flight training.

Rank	Name	Description
1	Interactive theory	Complex objects and procedures are displayed interactively, e.g., exploded view drawings of a engine, interactive representation of the lift equation etc.
2	Outside check—VR quiz	The students are given tasks to do on a virtual aircraft. e.g., to find the drain valves, or to carry out a process sequence in the correct order. There are points for correct answers (gamification).
3	Outside check—VR introduction	The outside check is carried out on a virtual aircraft. During the check, the systems indicate what should be checked next. Interactive elements with explanations are included, e.g., how the cable works on an aileron.
4	Outside check—AR introduction	The outside check is carried out on a real aircraft. During the check, an AR device indicates what should be checked next. Interactive elements with explanations are included, e.g., how the cable works on an aileron.
5	Cockpit procedure training—poster	The cockpit procedure is shown step by step and visualized using AR on a cockpit poster. For example, a button which needs to be pressed is marked red. No haptic feedback is included for this.
5	Preliminary exercise—airport	The airport is displayed in detail using an MR device. The student can familiarize himself/herself with the traffic pattern, entry points, exit points, approach and departure.
6	Outside check—AR quiz	The students are given tasks in AR to do on a real aircraft. e.g., to find the drain valves, or to carry out a process sequence in the correct order. There are points for correct answers (gamification).
7	Preliminary exercise—mental image	Student experiences a traffic pattern in 3D that can be stopped at any time. The student can experience when to send radio messages, when to press a button or push levers.
8	Cockpit procedure training—flight simulator	The cockpit procedure is shown step by step and visualized using AR on a flight simulator cockpit. For example, a button which needs to be pressed is marked red.
9	Supporting cues—flight simulator	During the flight, instruments are expanded using AR to show additional information, e.g., display cues to fly into a holding or to fly a standard curve.
9	Cockpit procedure training—virtual	The cockpit procedure is shown step by step and visualized using a VR cockpit. For example, a button which needs to be pressed is marked red.

3.4.2. Interim Results Module 2—Survey–Focus AR

The qualitative and quantitative analysis from Schaffernak et al. (module 2) analysed the responses to closed and open-ended questions obtained from pilots, student pilots, and flight instructors regarding the potential utilization of AR for pilot training [25]. To illustrate the possibilities of AR, demonstration videos from application domains distinct to aviation were shown to the participants. The videos presented how AR could support real world tasks, such as technical maintenance of terrestrial communication infrastructure, indoor navigation and visualisation of planned buildings. The analysis revealed seven application areas:

• Pre-flight—briefing,
• Pre-flight—outside-check,
• Pre-flight—procedure training,
• Practical flight training—flight training,
• Practical flight training—procedure training,
• Theory,
• AR navigation

3.4.3. Interim Results Module 3—Detailed Level Workshops

Based on application areas from Section 3.4.2, the content topics of the current syllabus [46] for pilot training were classified by the ATOs. This resulted in a 7 × 104 matrix (see extract

Table 5. Extract of pilot training syllabus assigned to application areas.

Syllabus Content Topics	Application Areas
	Pre-Flight			PFT *
	Briefing	Outside-Check	Procedure Training	Flight Training	Procedure Training	Theory	AR Navigation
Cockpit, cabin and cargo compartment		x				x	x
Pneumatic system						x	x
Aeroplane external visual inspection;
Location of each item and purpose of inspection		x					x
Cockpit inspection	x	x					x
Taxiing in compliance with air traffic
control or instructions of instructor				x			x
Adherence to departure and arrival
routes and ATC instructions					x		x
Holding procedures					x		x
Circling approach				x	x		x
Traffic pattern and landing without
extended or with partly extended flaps and slats				x			x

* PFT … Practical flight training

). Pre-flight—briefing included three topics, pre-flight—outside-check six topics, pre-flight—procedure training six topics, practical flight training—flight training 41 topics, practical flight training—procedure training 45 topics, theory 29 topics, and AR navigation nine topics. The syllabus topics were prioritized per application area using a pairwise comparison. The top-rated topics per application area can be seen in

Table 6. Best three syllabus topics per application area.

Application Area	Syllabus Content Topics
Pre-flight—briefing	Cockpit inspection Flight preparation Performance calculation
Pre-flight—outside-check	Engine including auxiliary power unit Landing gear Flight controls and high lift devices
Pre-flight—procedure training	Between V1 and V2 Rejected take-off at a reasonable speed before reaching V1 Shortly after reaching V2
Practical flight training—flight training	Crosswind landings (a/c, if practicable) Wind shear at takeoff/landing ACAS event
Practical flight training—procedure training	Crosswind take-off Manual go-around with the critical engine simulated inoperative after an instrument approach on reaching DH, MDH or MAPt Rejected landing at 15 m (50 ft) above runway threshold and go-around
Theory	Flight management systems Emergency equipment operation and correct application of the following emergency equipment in the aeroplane Special requirements for extension of a type rating for instrument approaches down to decision heights of less than 200 ft (60 m)
AR navigation	Aeroplane external visual inspection; location of each item and purpose of inspection Traffic pattern and landing without extended or with partly extended flaps and slats Cockpit, cabin and cargo compartment

.

3.4.4. Interim Results Module 4—Survey–Focus Syllabus

In a survey among 31 TR pilots (24 male pilots and seven female pilots) and 22 TRI (all male), [26] investigated the syllabus content of the TR training. The experience of the participating TR pilots and instructors is shown in Figure 4. The most difficult content to learn was missed approach procedures for both female and male pilots. TRIs rated take-offs as the most difficult content to learn. Flight maneuvers was rated as the easiest content to learn over all pilots, female pilots top-rated limitations and male pilots flight maneuvers. TRIs also rated limitations as the easiest content to learn. Women selected special requirements as the content which could benefit the most from AR, whereas men selected flight management system, which was also the top-rated content over all pilots. Aircraft structure and equipment was top-rated by TRIs [26].

4. Ecosystem Analysis

Innovating in complex business environments such as the flight training business, in which various economic entities collaborate, requires detailed analysis of individual goals and needs in order to create a sustainable business network [6,47]. In addition to identifying the highly relevant use cases described above, we also considered the prospective business ecosystem of these use cases in a pre-conceptual ecosystem case study. This pre-conceptual ecosystem helps to gain insights in relevant stakeholders and the values exchanged between them. Based on this model, pre-conceptual analysis of value exchange relations (e.g., reciprocity) can be performed. This enables first insights in basic structures of the prospective ecosystem and fosters the anticipation of value-generating and value-hampering value exchange relations in an early design stage [48].

In the course of a workshop with an ATO, we interactively created a prospective ecosystem of economic entities, which would be required to implement the identified use cases. We used the stakeholder map blueprint of Fassin [49] to identify relevant economic entities, also designated as actors, in a structured way, and then the EcoVis modeling tool to visually represent the ecosystem including the value exchange relations between the economic entities [50]. Economic entities were clustered according to their role (e.g., customers, ATO and its sub entities (e.g., technical personnel), financial partners, etc.) and visually highlighted with colored ‘actor’ symbols.

The focus was on the value exchange and resource layer of the ecosystem analysis framework [47], which provides information on the actors involved and the values they exchange. Actors are illustrated as tripartite circles, the exchange values are shown as directed arcs between the actors. Arcs with continuous lines represent provision links, with dashed lines as revenue links. Labels in colored triangles give more information on the type of value exchanged (e.g., product, service, money) [50].

This helped us to better understand the complex ecosystem in a pre-conceptual phase and to identify aspects which could require particular attention, when implementing and embedding the use cases in real-life business environments. The resulting ecosystem (see Figure 5) consists of various actors who directly exchange values and thereby enable the realization of the identified use cases. One key-finding of this ecosystem analysis was that the role change of technical personnel working at ATOs would need to be addressed. The introduction of the new MR service leads to additional tasks for the ATO staff, particularly for technical personnel. These MR-related tasks would need additional resources and incentives (e.g., compensation), in order to foster a viable value exchange relation between ATO and technical personnel. This insight informs future system and incentive design phases and helps to overcome potential problems when deploying the service. Another finding was that MR-related services and applications could increase understanding of and identification with the “service” training of ATOs employees, who are not pilots.

5. Discussion

Training methods in aviation still heavily rely on synchronous learning and on-site training today. These pose constraints on the attendance of student pilots and lead to bottlenecks in the availability of expensive training equipment and staff. On the other hand, current and future developments in MR technology generate new opportunities for novel use cases to improve pilot education. Several authors have recognized the advantages of AR technology for learning and teaching [22,23,24]. Studies which investigated MR, VR or AR in pilot education [29,30,31,32] aimed on a specific use case, technology or presenting a general idea. This study addresses potential MR-based improvements in pilot training using a holistic gender-sensitive approach and explored three novel MR use cases for pilot training. The aim was to identify areas of improvement based on MR for various aspects of the business ecosystem [6], in a socially and economically sustainable manner. In detail, an iterative method was used for analyzing current flight training processes and evaluating the potential of MR technology to improve them. A mix of methods, such as surveys of pilots and instructors, and also workshops with ATO representatives, were applied for analysis, comparison, and ranking of various data sets.

Research [26] shows that stress, time pressure and issues with self-study are reported by a number of pilots. Proposed mitigations, such as extension of the course duration, could create additional costs for the pilots or their employers. However, these mitigations could also be achieved by improving the training methods in a socially and economically sustainable manner.

Traditionally, the learning needs and preferences of male pilots tend to be generalized to both genders because aviation is a male dominated domain [35]. Nevertheless, gender differences in flight training exist and need to be addressed [36,37,38,39,40]. Ref. [43] showed that trainees from both gender groups reached similar levels of flight performance in post-test, after the same amount of training, despite initial gender differences in pre-test. This shows that with the same amount of training, a similar level of performance can be reached with an inclusive training method. Designing an inclusive and socially sustainable pilot training program requires that diversity aspects are identified, communicated and addressed [40].

The ecosystem modelling highlighted individual goals and needs that in accordance with [6,47] should be addressed for creating a sustainable business network where various economic entities collaborate in the complex flight training environment. ATOs strive to provide optimal training and meet economic goals, they need to optimize their processes and thrive for innovation. This is required for market competitiveness as described by Schumpeter [1]. Since ATOs are part of a complex ecosystem, the innovation process must take all actors of the system into account. For example, authorities such as EASA regulate the training content and thus need to be considered in the implementation of innovative use cases. Simulation equipment is certified, and a change in the code basis leads to an expansive and time consuming re-certification process. However, the integration of MR technology in the training process of ATOs can bear competitive advantages, such as a more efficient use of resources or a unique selling proposition compared to the competition. For example, a more efficient training process can be established if some training parts are not limited by the accessibility of a particular aircraft type at an airport or by the availability of an instructor. Additionally, applying MR technology to pilot training in a playful way can increase the engagement and attention of the trainee and lead to an improved training experience for the students [25,51]. Thus, MR could improve the value proposition for student pilots, in a gender-sensitive way, while the value creation aspects for ATOs could also be optimized. Furthermore, the work also presents some limitations, e.g., the data collection involved all relevant stakeholders, but the number of participating ATO stakeholders could be further increased. A further issue was that the constraint of limited resources meant that some top-rated high-level use cases were not detailed in this work, such as ‘outside check—AR introduction’ or ‘preliminary exercise—airport’. These might be addressed in future studies. A further issue is that the use cases developed in this study require further validations. This could be accomplished by quantitative comparative studies with prototype applications, based on commercial off the shelf MR devices such as Microsoft HoloLens. However, when implementing these use cases, further technical and financial limitations must be considered, such as cost per unit, development costs, motion sickness, narrow field of view, missing haptic, etc.

6. Conclusions

This study shows that MR technologies offer new possibilities for pilot education and business innovation. As a main result, three MR use cases have been identified and specified that have potential both for improving pilot training, and for promoting gender diversity and accessibility in an economically sustainable manner. The use of multiple data collection methods and the involvement of various stakeholders provided a holistic perspective for designing new use cases. This method of merging various results from the different data collection methods allowed the identification of the three most promising use cases. First, interactive theory training—which helps students with an interactive AR explanation of the electrical power supply. Second, outside check—which guides students while carrying out the outside check procedure on a virtual aircraft with augmented cues. Last, procedure training—which supports students while training cockpit preparation and in-flight starting procedure using an AR enriched cockpit poster. Furthermore, a prospective ecosystem of economic entities, which would be required to implement the identified use cases was jointly created with representatives of two ATOs. This analysis indicated that the introduction of MR related services could require additional incentives for technical experts working at the ATOs, and could increase ATO non-pilot employees’ understanding of and identification with the pilot training.