Potential Augmented Reality Application Areas for Pilot Education: An Exploratory Study

Abstract

The goal of carrying out this work was to identify potential application areas for augmented reality (AR) in pilot education by addressing gender preferences. Like the field of engineering, the aviation industry is dominated by men. Because the aviation industry forecasts a high demand for pilots, it is highly desirable to address gender diversity and improve teaching methods in pilot education. In this study, potential application areas for AR-supported pilot training were investigated by conducting a survey with 60 pilots and flight instructors (including 12 women). Typical AR use cases were presented in videos, and the pilots reported their preferences regarding similar or other AR applications used in different parts of the flight training program. AR navigation was the use case that was most frequently preferred by both female and male pilots. The majority of pilots agreed that AR could potentially be used in theoretical instruction, pre-flight aircraft inspection, and procedure training. In addition, both gender groups showed similar preferences for various gaming concepts that make learning more interesting and engaging, such as receiving positive feedback. However, a higher percentage of women than men reported that achieving a target or receiving points to successfully finish a task and answering questions during the game were satisfying. Including a story in the game to attract attention was preferred by a higher percentage of men than women. The results of this study can be used to design AR educational concepts that support gender diversity in pilot education and other technical domains.

1. Introduction

Within the next years, a global pilot shortage is expected for several reasons, including an increasing demand that leads to growing fleets, new rules that were introduced for First Officers (by the Federal Aviation Administration (FAA)) in 2013, retirements, and attrition. In particular, smaller regions are already experiencing shortages that affect commercial air services. In 2016, the University of North Dakota predicted a shortage of around 15,000 airline pilots by 2026, but only in the United States (US) airline fleet [1,2,3,4]. This shortage is additionally intensified due to the underrepresentation of women in the airline industry. Halleran [5] showed that women make up only a small percentage of those working in Science, Technology, Engineering, Mathematics (STEM) fields in general, and in the aviation industry in particular (e.g., there are only 5% female pilots with an Airline Transport Pilot License (ATPL) in the US). The Boeing Company [6] forecasts a demand for 804,000 new pilots (commercial, business, and helicopter) over the next 20 years in their report “Pilot and Technician Outlook 2019–2038”. Furthermore, the company stated: “The aviation industry will need to adopt innovative training solutions to enable optimum learning and knowledge retention. Immersive technologies, adaptive learning, schedule flexibility, and new teaching methods will be needed to effectively meet a wide range of learning styles. The growing diversity and mobility of aviation personnel will also require instructors to have cross-cultural, cross-generational, and multilingual skills to engage with tomorrow’s workforce.” [6].

In the 1910s, the Antoinette Simulator was the first mechanical flight simulator training device [7]. In 1929, the first electromechanical simulator was created [8] and used by World War II pilots for training in Instrument Flight Rules (IFR) [9]. Flight simulation enabled faster, safer, and cheaper training of pilots, becoming widely established in commercial aviation in the 1960s [7]. Currently, a wide range of training simulators is used, including part-task desktop devices, flight procedure training devices, and high-fidelity motion-based full flight simulators [10]. More recently, networks of simulators have been used in research on pilot training in both military [11] and civil aviation [12]. In addition, serious games, such as the computer game The Space Fortress, have been successfully used to train pilots’ allocation of attention in conditions of multitasking, skills that pilots could transfer to real flight situations [13].

In a current handbook for Flight Instructor (FIs) published by the FAA, future developments are outlined and new trends in training are stated. The handbook mentions a shift from traditional classrooms to laboratory environments using devices such as computers, and briefly describes Virtual Reality (VR), but does not mention augmented reality (AR) [14].

In 2017, Brown [15] talked about transforming aviation training with AR. She distinguished between VR, AR, and Mixed Reality (MR). These new techniques appear promising and could be used to bridge the gap between classroom, simulation, and also practical operations. They allow users to examine and interact with an engine, for example, in order to better understand the internal processes. Other possible use cases envisioned by Brown [15] include: Procedure training, aircraft systems training, aircraft familiarization, maintenance training operations, cabin familiarization and training, virtual manuals, and hands-free remote assistance. Airlines and military organizations are being forced to improve and adapt their training programs to address the needs of the Next Generation of Aviation Professionals (NGAP), on the one hand, to shorten training programs and also reduce costs, and, on the other hand, to fulfill the higher requirements of a new generation with more native digital skills [16,17,18].

VR is characterized by its potential to create a completely synthetic and artificial world, in which the user is completely immersed and with which he or she can interact. Towards the other extreme—the real world—(see Figure 1) subclasses of VR exist, which are known as MR. Both of these worlds, real and virtual, are merged [19]. According to Azuma [20], the 3D-technology AR can be used to enhance the real world with virtual objects, but does not replace the real world, unlike VR. In addition, AR is characterized by the fact that it offers the possibility for a real-time interaction with the user to take place [20]. In other words, real-world and virtual objects can coexist in a common environment in real time.

AR has recently gained more attention as a new technology that can be used to enhance training and learning in the field of education, as compared to other industries, where its use has been investigated since the early 1990s [21,22]. Bower et al. [21] pointed out that quite a few researchers had identified a huge potential for AR in education. In their article, Wu et al. [23] identified aspects influencing the use of AR in education. These include the learning content in 3D perspectives, which allows students to investigate a system from different perspectives and improve their understanding, and visualizing the invisible to explore abstract concepts or phenomena, such as airflow. The technology of AR can be used with various devices; for example, smartphones, tablets, and glasses, such as the Microsoft HoloLens.

This paper is structured in six sections. In the introduction, we address the problem of the pilot shortage and the demand for innovative and gender-sensitive training programs and technologies. In the next section, a review of the literature on relevant technology, education, and gender issues is presented. The third section includes information on pilot education. In Section four, the survey is explained, describing the methods, materials, and analytical techniques used. Section five presents the quantitative and qualitative results of the survey. Finally, the results are discussed, and an outlook for future work is given.

2. Theoretical Background

AR has a long history in the field of aviation. For example, the term AR was first mentioned by [24] to support aircraft manufacturing at Boeing. Subsequently, the utilization of AR was transferred to further application fields. A large number of existing studies in the broader literature have been carried out to examine and conduct AR use cases in the area of education to reveal the potential of this technology. One major aspect of VR and of AR is that of (partial) immersion. The work of Dede [25] indicated that students who are immersed in a digital environment develop sophisticated problem-finding skills. The research of El Sayed et al. [26] showed that the utilization of AR in education increases the visualization ability of students and decreases the education expenses incurred by schools. Furthermore, AR or mixed environmental training can be used to develop psychomotor and cognitive skills [27]. In 1995, Copolo and Hounshell [28] showed that student groups who used both 3D computer models and physical models during training performed better than groups who had used only one of these.

Previous studies have been conducted to review the existing literature on this topic and reveal its common advantages and disadvantages. Radu [29] conducted a review of journal articles, describing the advantages and disadvantages of AR- and non-AR applications (e.g., traditional books, video) used in educational environments for student learning. He also mentioned that digital learning experiences are becoming more easily available for students because of the use of smartphones and tablets. He identified certain benefits of AR-supported learning, such as learning spatial structure and function (e.g., learning components of an aircraft turbine engine), long-term memory retention, improved physical task performance (e.g., maintenance activities), and increased student motivation. In his review, Radu [29] also acknowledged evidence of deterioration through the use of AR, such as attention tunneling, usability difficulties, ineffective classroom integration, and learner differences. Wu et al. [23] investigated 54 relevant articles, asking different guiding questions, and concluded that AR has a great potential to support learning and teaching. However, technological, pedagogical, and learning issues still require further investigation. Learning challenges in AR environments often emerge as a result of the different student skill levels (e.g., spatial navigation, collaboration, problem solving skills). As shown by [30], students also express concerns about the quality and adequacy of the material provided to learn in such environments. In a literature review on the use of AR in the area of STEM, Ibánez and Delgado-Kloos [31] found that applications were most often used for exploration or simulation purposes; in fewer cases, these applications were game-based, and most were visually stimulating. They also investigated the learning outcomes of students using AR, noting that motivation was the most frequently cited emotional outcome, followed by attitude, enjoyment, and engagement. As Ibánez and Delgado-Kloos [31] observed, the studies hardly considered diversity. Hence, this issue needs to be addressed to support all users adequately.

Over time, an extensive amount of literature has developed on aviation-related AR/VR use cases. De Crescenzio et al. [32] showed an improvement in task efficiency when AR was used for aircraft maintenance training. However, a primary focus was placed on the technical implementation in this study. A similar study was conducted by [33], in which the feasibility of AR solutions in aeronautical maintenance tasks was investigated. The authors identified different technical disadvantages of the current state of technology, and concluded that the number of errors occurring during maintenance could be reduced by using AR. Another article reported the results of an analysis of the utilization of smart glasses to guide the pilot during flight with additional visual cues. As a result, Haiduk [34] concluded that smart glasses have the potential to support pilots in the future. Koglbauer et al. [35] showed that flight performance could be significantly increased by simulator training using augmented cues, and addressed the potential use of AR cues for the pilots’ orientation in the 3D space. Oberhauser et al. [36] investigated the benefits and limitations of VR training and concluded that VR can partly be used to supplement simulator training.

Research results show that technology or use cases are often designed with a typical user in mind, often favoring specific characteristics, such as a gender [37]. The field of aviation has especially been historically shaped by men; thus, the stereotype of a masculine domain is widespread, leading to potential gender issues [38]. Therefore, special attention should be paid to this aspect. Research results have shown that men generally perform better than women when processing visuospatial information, such as mental rotation, whereas women are generally superior to men when processing verbal information [39]. However, Neubauer et al. [40] showed that gender differences were absent in 3D tasks, and these diminished after training 2D visuospatial mental rotation tasks. These findings were confirmed by Koglbauer and Braunstingl [12], who evaluated a flight simulator training program and did not find gender differences in terms of situational awareness and the performance of flight tasks that involved processing both visuospatial and verbal information. Women who took part in simulator training perceived a lower workload than men when the workload was associated with the need to coordinate with other aircraft in the traffic circuit [12]. The intensity of the positive emotions experienced by the trainees of both genders was high, indicating that practical training was more enjoyable and effective than classroom instruction when teaching visual airport procedures [12]. Based on gender-specific gaming behavior, previous research showed that different types of games are preferred by women and men. Men tend to play more competitive games. Women prefer logical- and skill-training-based games. Already in 2013, 46% of the most frequent purchasers of video games were women, with growing participation [41]. A few years earlier, this increase in female players was mentioned by Joiner et al. [42]. More importantly, the work points out that women and men can benefit equally from game-based learning. However, the design of the game features is crucial for success.

Overall, the findings in the literature suggest that the use of AR has a huge potential to improve education in general and flight training in particular. Although many studies on AR in education have been carried out, little research has been conducted on potential AR use cases in pilot education. The implications of taking a more holistic approach in this education, which takes gender aspects into consideration, has rarely been addressed in the literature.

3. Pilot Education

In this section, we briefly explain the pilot education program, limiting the explanation to airplanes, regulations set by the European Union Aviation Safety Agency (EASA), and the license types of participants included in the current survey: Private Pilot License (Airplane) (PPL(A)), Type Rating (TR), and the instructors for both. As can be seen in Figure 2, a PPL(A) is considered as a basic level in pilot education. Pilots with a PPL(A) are allowed to fly single-engine piston airplanes up to a maximum take-off weight of 2000 kg under good weather conditions (Visual Flight Rules (VFR)), but only for non-commercial purposes. In order to obtain a PPL(A), the student pilot has to complete around 100 hours of theoretical lessons and receive 45 hours of flight instruction [43]. Theoretical lessons comprise the following subjects: Aviation law, human performance, meteorology, communications, principles of flight, operational procedures, flight performance and planning, aircraft general knowledge, and navigation. The syllabus is strictly defined by the EASA. Detailed regulations can be found in [44].

A TR authorizes the pilot to fly the type of aircraft specified in the rating and is considered as an extra privilege. It requires additional training and is supplementary to the Commercial Pilot License (CPL) or ATPL. For instance, a pilot flying an Airbus A320 must obtain an ATPL as an initial license and, additionally, a TR for the type A320. Theoretical lessons and flight instruction hours must be completed by the pilot to obtain a TR.

The practical flight instruction is performed with a certified instructor, who is allowed to provide training for the class or type of aircraft for which he or she has a license. The FI PPL(A) is allowed to instruct student pilots during their PPL(A) training. A Type Rating Instructor (TRI) is approved to perform more advanced training on certain aircraft types. To obtain an instructor certificate, certain conditions have to be met (i.e., the pilot has to provide evidence of a certain amount of flight experience). He or she holds the license or TR for which he or she applies as an instructor, and he or she has to complete an instructor training course. These conditions are not exhaustive; therefore, more detailed requirements are described in [44].

All of the previously mentioned licenses, ratings, or certificates share common features, namely that the training consists of both a theoretical and a practical portion (i.e., the flight instruction). Different but traditional options are normally used for the theoretical part of the training: Traditional classroom training, Computer-based Training (CBT), and Web-based Training (WBT). Normally, a mixture of at least classroom training and CBT or WBT is used to comply with the regulations.

Traditional classroom instruction can be described as teacher-centered face-to-face classroom learning, carried out according to a fixed time schedule. The content provision and the learning process are controlled by the teacher, and students have a mainly passive learning behavior [45,46].

In contrast, e-learning is characterized by learner-centered and self-paced training that is independent of a time and location, as it is asynchronous in nature [46,47]. Bedwell and Salas [48] summarized CBT as “a self-contained, interactive, often asynchronous, computer-based program designed for self-paced instruction that uses features of learner control coupled with predesigned material, required responses, and feedback”. The content of CBTs is often delivered by CD/DVD. Compared to this, the content of WBTs is delivered via the World Wide Web (WWW). This content can be understood, in a rather simple manner, as online documents with hyperlinks that help the students navigate through the course. The documents contain multimedia content, such as text and pictures. More enhanced WBTs offer services such as functions that facilitate information search, synchronous communication, collaboration, or knowledge level testing [49].

The flight instruction has three phases: A briefing phase, the flight training including pre-flight inspections, and the debriefing phase. In practice, two devices are mainly used: The real aircraft and the simulator (with different designs and qualities, depending on the requested license or rating). During the basic education phases, training in a real aircraft is very common and also mandatory. For instance, students can complete five hours (of 45 h practice) in a Flight Simulation Training Device (FSTD) during the PPL(A)-training. As the students’ education advances, the amount of practical training received on different types of simulators increases (e.g., Full Flight Simulator (FFS), Flight and Navigation Procedures Trainer (FNPT)). The flight instruction is performed with an instructor or at least under the supervision of an instructor, which means that the student pilot has to finish a certain number of solo flights (depending on the desired license).

This explorative study was carried out to identify potential application areas for AR in pilot education (PPL(A) and TR) and to identify gender preferences.

4. Materials and Methods

4.1. Participants

Forty-eight male pilots with a mean age of 41.74 years (SD = 1.79) and twelve female pilots with a mean age of 38.60 years (SD = 2.51) participated in the survey.

Table 1. Distribution of licenses for each gender group.

License	PPL(A)	PPL(A) FI	TR	TR FI	Total
Male	9	5	19	15	48
Female	6	0	6	0	12
Total	15	5	25	15	60

shows the distribution of licenses within each gender group.

A multi-national European sample of pilots was recruited from flight training organizations, as well as via the personal contacts of the investigators. All participants received information about the purpose of the survey and gave their informed consent. The access to the electronic survey form was enabled by a four-digit code generated by the investigator.

4.2. AR Survey

The AR survey was organized in three main parts. The first part included questions about the pilot’s knowledge and experience with AR (What do you know about AR? What do you know about VR? Did you use one or more of these or similar AR applications: Ikea Product Fitting in Your Place, Poke’mon Go, Playstation Playroom, MeasureKit—AR Ruler Tape, Xbox Kinect Games, Snapchat, no or other applications used. Answer choices were: Yes, no, or not applicable). The pilot was also asked to describe their motivation to install or try the AR application (Why did you install or try that/those App(s) in the first place: Curious about AR, wanted to try something new out, met an actual need for information, advertised in public, suggested by a friend, suggested by App Store, other. Answer choices were: Yes, no, or not applicable).

The second part of the survey included questions about the applicability of typical cross-domain AR use cases. To encourage the pilots to think of possible use cases for AR examples of the following, typical AR use cases were presented:

• Remote Consultation and Assistance. During a remote consultation, an expert helps a novice complete a task remotely. A video was shown that demonstrated maintenance assistance provided to an on-site worker by a specialist.
• AR Planning. Testing and designing complex structures in real-world environments without the need of physical production. The video showed an example of ship construction site planning in a real environment.
• AR Navigation. Navigation using AR allows the user to display virtual navigation cues directly on the real environment (e.g., the street). The video showed navigation cues that could be used to navigate through a huge indoor space.

For each use case, the respondents were asked if they believed that there were beneficial use cases in pilot training. The answer choices were: Strongly disagree, disagree, not sure, agree, and strongly agree. In addition, the participants were asked to comment on the applicability of the particular use cases in pilot training.

Furthermore, the pilots were asked to identify parts of the pilot training in which AR could be used: Theoretical parts, pre-flight inspection, and practical pilot training (e.g., the landing). The answer choices were: Strongly disagree, disagree, not sure, agree, and strongly agree. In addition, the pilots were asked to comment on the possible use of AR in pilot training.

In the third part of the survey, the pilots were asked to rate their preferences regarding the following game concepts: Achieving a target to finish tasks, receiving points if you successfully finish a task, answering questions during the game, having a time limit to finish tasks, including a story to attract a trainee’s attention, collecting assets or information to proceed in the game, solving puzzles to proceed in the game, and receiving feedback for correct actions. For each game concept, the following answer choices were presented: Very unsatisfying, unsatisfying, neutral, satisfying, and very satisfying.

4.3. Descriptive Data Analysis

In this study, we aimed to identify gender-specific aspects and preferences. Therefore, due to size differences between the gender groups, the results are presented descriptively by using percentages. To conduct the data analysis, the answer choices with five options (e.g., strongly disagree, disagree, not sure, agree, strongly agree) were reduced to three categories (e.g., disagree, not sure, agree) by merging the extreme answers, such as “strongly disagree” and “disagree”, into a single category, such as “disagree”.

4.4. Qualitative Method

The qualitative analysis was primarily conducted to gain insights into the potential AR-based use cases for flight training. Furthermore, the respondents’ statements were also analyzed to classify their state of knowledge about AR. The qualitative data analysis was structured around the qualitative content analysis framework of Mayring [50] and can be described for the potential AR use case identification as follows:

• Analysis material: The material that was analyzed qualitatively was generated by conducting an online survey (see Section 4). While the quantitative analysis described in Section 5.1 was conducted on responses to the survey questions, which resulted in ratings, the qualitative analysis was conducted on the text provided by the respondents as answers to the open-ended questions in the survey.
• Situation of data generation: The respondents independently chose when they filled in the online survey. All respondents participated voluntarily.
• Formal characteristics of the material: The material that was analyzed qualitatively was generated by the respondents in a textual form during the online survey.
• Guiding question: The guiding question for the analysis of potential AR use cases was: What are potential AR-based flight training use cases? Based on the online survey, the respondents were able to rate pre-defined AR use cases and to describe potentially interesting AR use cases for flight training on their own.
• Interpretation technique: Frequency analysis was used as a technique to interpret the outcomes. This technique is applied by identifying selected text items based on the categories developed and, thereby, gaining insights based on the calculated frequency.
• Analysis units: The units of analysis were defined by relevant statements concerning potential AR-based flight training use cases.
• Coding system: The coding system (see

  Table 2. Categorization codes used for text analysis to classify the area in which augmented reality (AR) application can be used in pilot training. These codes are used for the use-case-specific questions (See Section 5.2).

  Codes	Description
  Theory	Classroom training without further specification
  Theory_TT	Classroom with technical and physical focus
  Theory_TL	Classroom with airspace focus
  Training	Simulator or aircraft training without further specification
  Training_PC	Cockpit procedure training
  Training_PE	Emergency procedure training
  Training_PP	Training for pre-flight tasks/procedures such as briefing
  Training_PF	Familiarization of new airports
  Training_PD	Training for debriefing
  Training_PN	Training related to navigation
  System_PS	Utilization in simulator
  System_PA	Utilization in aircraft
  System_TM	Utilization with training mock-ups
  Practice	Support in real-world scenarios without further specification
  Practice_PT	Support for technical problems in real-world scenarios
  Practice_NT	Support for non-technical problems in real-world scenarios
  None	Cannot be used during pilot training
  Bad	Would worsen pilot training

  ) was generated inductively and iterated until a consensus was reached among three analysts.
• Coding: The analyzed material was coded by these three analysts. Initial disagreements among the analysts regarding the coding strategy were discussed until a full consensus was reached.
• Interpretation: The results were interpreted as presented in Section 5.2.

The qualitative analysis of the responses regarding the state of knowledge about AR was conducted using the same process, but by using the categorization codes presented in

Table 3. Categorization codes for text analysis to classify the participants’ states of knowledge.

Codes	Description
NO	No general knowledge of the topic existing
KH	General knowledge of the topic existing
UK	State of knowledge cannot be judged

.

5. Results

5.1. Quantitative Results from the Survey

The results show that about 66.70% of female and 60.42% of male pilots had no experience with AR applications. Few male pilots reported having had an experience with Ikea Product Fitting in Your Place (12.50%), Poke’mon Go (10.42%), Playstation Playroom (8.33%), MeasureKit—AR Ruler Tape (6.25%), or Xbox Kinect games (6.25%). Female pilots reported having experience only with Ikea Product Fitting in Your Place (8.33%) and MeasureKit—AR Ruler Tape (8.33%). Male pilots reported having installed or tried these AR applications because they either wanted to try out something new (10.42%), meet an actual need for information (20.83%), it was advertised in public (6.25%), respond to a suggestion by a friend (10.42%), respond to a suggestion by the App Store (10.42%), or for other reasons (4.17%). Female pilots reported having installed or tried those AR applications because they wanted to meet an actual need for information (16.67%) or because the AR was advertised in public (8.33%). As Figure 3 shows, most male and female pilots believed that pilot training would benefit from the use cases of Remote Consultation and Assistance, Augmented Reality Planning, and Augmented Reality Navigation. These results are presented in

Table 4. Preferred utilization of industrial AR use cases in pilot training and usage in syllabus.

Usage of Industrial Use Cases	Gender	Disagree (%)	Not Sure (%)	Agree (%)
Remote Consultation and Assistance	Men	18.75	31.25	50.00
	Women	0.00	45.45	54.55
Augmented Reality Planning	Men	25.00	20.83	54.17
	Women	0.00	36.36	63.64
Augmented Reality Navigation	Men	14.58	22.92	62.50
	Women	0.00	9.09	90.91
Usage in Syllabus
Theoretical parts	Men	16.67	22.92	60.42
	Women	0.00	9.09	90.91
Pre-flight aircraft inspection	Men	14.58	18.75	66.67
	Women	9.09	0.00	90.91
Practical flight training	Men	22.92	25.00	52.08
	Women	9.09	27.27	63.64

. As illustrated in Figure 4, most men agreed with the possible use of AR in theoretical instruction (60.42%), pre-flight aircraft inspection (66.67%), and practical flight training (52.08%). As compared to the men, a higher percentage of women agreed with a possible use of AR in theoretical instruction (90.91%), pre-flight aircraft inspection (90.91%), and practical flight training (63.64%). Nevertheless, as Table 4 shows, a certain percentage of women and men disagreed or expressed neutral opinions about the possible use of AR in pilot training.

The results for the pilots’ preferences for different game concepts are illustrated in Figure 5. Achieving a target to finish tasks was considered satisfying by 72.73% of the women and 60.42% of the men. Receiving points if you successfully finish a task was considered satisfying by 72.73% of the women and 56.25% of the men. Answering questions during the game was considered satisfying by 72.73% of the women and only 37.50% of the men. Setting a time limit to finish tasks was considered unsatisfying by 29.17% of the men and 27.27% of the women, with only 27.08% of the men and 9.09% of the women reporting this as a satisfying game concept. Including a story in the game to attract attention was considered satisfying by 68.75% of the men and 45.45% of the women. Collecting assets or information to proceed was considered satisfying by 45.83% of the men and by 54.55% of the women. Solving puzzles to proceed was considered unsatisfying by 25.00% of the men and 27.27% of the women, with 39.58% of the men and 45.45% of the women reporting this as a satisfying game concept. Receiving feedback for correct actions met the preferences of 68.75% of the men and 72.73% of the women (details are shown in

Table 5. Preferences for game concepts in detail.

Game Concepts	Gender	Unsatisfying (%)	Neutral (%)	Satisfying (%)
Achieve a target to finish tasks	Men	10.42	29.17	60.42
	Women	0.00	27.27	72.73
Receiving points if you successfully finish a task	Men	6.25	37.50	56.25
	Women	0.00	27.27	72.25
Answer questions during the game	Men	16.67	45.83	37.50
	Women	0.00	27.27	72.73
Time limit to finish tasks	Men	29.17	43.75	27.08
	Women	27.27	63.64	9.09
The game includes a story to attract your attention	Men	8.33	22.92	68.75
	Women	27.27	27.27	45.45
Collecting assets or information to proceed	Men	12.50	41.67	45.83
	Women	27.27	18.18	54.55
Solving puzzles to proceed	Men	25.00	35.42	39.58
	Women	27.27	27.27	45.45
Receiving feedback for correct actions	Men	6.25	25.00	68.75
	Women	0.00	27.27	72.73

).

5.2. Qualitative Results from the Survey

The survey contained open-ended questions regarding the topic. Here, the participants could add feedback or describe their thoughts. These answers were categorized using the method described in Section 4.

An open-ended question was asked to evaluate the pilots’ previous knowledge regarding AR. The results are illustrated in

Table 6. Participants’ knowledge of the term AR.

Gender	Basic Understanding	No Understanding	Cannot be Assessed
Men	20 (41.67%)	10 (20.83%)	18 (37.50%)
Women	7 (58.33%)	1 (8.33%)	4 (33.33%)

. Seven (58.33%) of the women and twenty (58.33%) of the men showed a basic understanding of the term AR. Only one (8.33%) response from a women and ten (20.83%) responses from the men show no understanding of the concept of AR.

The open-ended question “Where would you use AR during the pilot training?” was answered by 47 out of 60 participants (see

Table 7. Categorized free-text responses, structured by gender.

Description	Predefined Use Cases												Any Use Case
	Remote Consultation Assistance				AR Navigation				AR Planing				Where Would you Use AR during the Pilot Training?
	Men		Women		Men		Women		Men		Women		Men		Women
Total responses	14	(100.00%)	4	(100.00%)	15	(100.00%)	3	(100.00%)	13	(100.00%)	4	(100.00%)	36	(100.00%)	11	(100.00%)
Classroom training without further specification	1	(7.14%)	2	(50.00%)	0	(0.00%)	1	(33.33%)	7	(53.84%)	1	(25.00%)	8	(22.22%)	5	(45.45%)
Classroom with technical and physical focus	0	(0.00%)	2	(50.00%)	0	(0.00%)	1	(33.33%)	5	(38.46%)	1	(25.00%)	5	(13.88%)	5	(45.4 %)
Classroom with airspace focus	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	1	(7.69%)	1	(25.00%)	0	(0.00%)	0	(0.00%)
Simulator or aircraft training without further specification	6	(42.85%)	0	(0.00%)	13	(86.66%)	3	(100%)	5	(38.46%)	2	(50.00%)	25	(69.44%)	8	(72.72%)
Cockpit procedure training	1	(7.14%)	0	(0.00%)	3	(20.00%)	0	(0.00%)	1	(7.69%)	0	(0.00%)	9	(25.00%)	3	(27.27%)
Emergency procedure training	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	4	(11.11%)	5	(45.45%)
Training for pre-flight tasks/procedures such as briefing	1	(7.14%)	0	(0.00%)	1	(6.66%)	1	(33.33%)	1	(7.69%)	0	(0.00%)	9	(25.00%)	0	(0.00%)
Familiarization of new airports	0	(0.00%)	0	(0.00%)	0	(0.00%)	1	(33.33%)	0	(0.00%)	1	(25.00%)	2	(5.55%)	0	(0.00%)
Training for debriefing	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	1	(7.69%)	0	(0.00%)	1	(2.77%)	0	(0.00%)
Training related to navigation	0	(0.00%)	0	(0.00%)	11	(73.33%)	1	(33.33%)	2	(15.38%)	0	(0.00%)	10	(27.77%)	2	(18.18%)
Utilization in simulator	2	(14.28%)	0	(0.00%)	1	(6.66%)	0	(0.00%)	1	(7.69%)	0	(0.00%)	5	(14.88%)	0	(0.00%)
Utilization in aircraft	7	(50.00%)	0	(0.00%)	1	(6.66%)	0	(0.00%)	1	(7.69%)	0	(0.00%)	1	(2.77%)	0	(0.00%)
Utilization with training mock-ups	0	(0.00%)	0	(0.00%)	1	(6.66%)	0	(0.00%)	1	(7.69%)	0	(0.00%)	0	(0.00%)	0	(0.00%)
Support in real-world scenarios without further specification	5	(35.71%)	1	(25.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)
Support for technical problems in real-world scenarios	3	(21.42%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)
Support for none technical problems in real-world scenarios	2	(14.28%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)
Cannot be used during pilot training	3	(21.42%)	1	(25.00%)	1	(6.66%)	0	(0.00%)	1	(7.69%)	1	(25.00%)	2	(5.55%)	1	(9.09%)
Would worsen pilot training	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)	0	(0.00%)

, column five). A categorization was possible for 38 of these answers, ten of which were from women and 28 were from men. Thirty-three answers could be categorized as “Simulator or aircraft training without further specification”; therefore, this category contains most of the answers. All 33 answers could be assigned to sub-categories, whereby “Cockpit procedure training”, “Emergency procedure training”, “Training related to navigation”, and “Training for pre-flight tasks/procedures such as briefing” contained the most answers. Most answers that were classified in the category “Cockpit procedure training” indicated that learning flows and cockpit procedures would be a good use case, but the term “checklists” was also mentioned in four of the twelve answers. Two participants also stated that an AR familiarization with the cockpit would be helpful. More than half of the answers were given by pilots with TR and/or FI certificates. Half of the answers categorized in the category “Emergency procedure training” described or mentioned directly training for emergency procedures. The second half of these answers described general procedures, where it was not clear if the participant was also considering emergency procedures. Especially female participants suggested this use case; furthermore, only three participants in this group were pilots with a TR and/or FI certificates. Nine answers could be categorized as “Training for pre-flight tasks/procedures such as briefing”. Seven participants suggested the use case “outside check”, where the pilot inspects her or his aircraft before takeoff. The rest of the participants gave more generic answers, such as “de/briefing” or “pre-/post-flight.”

Thirteen answers could be categorized as “Classroom training without further specification", ten of which are in the category “Classroom with technical and physical focus”. Here, most participants mentioned that AR could be helpful for explaining the mechanical components of the aircraft and how the system works, but one person also said that it could be used to illustrate aerodynamics. This category included 45.45% of the female responses but only 13.88% of the male responses.

Three answers of participants could be categorized as “Can not be used during pilot training,” two commented that they did not think that AR could help with training, and one stated that it could only be beneficial if aircrafts were equipped with AR devices. No participant mentioned that AR would worsen pilot training.

The suggested use case of AR Navigation showed the user how AR could be used to navigate through unknown environments, such as large office spaces. Eighteen of the sixty participants provided an answer to the open-ended question “Please explain use cases you have in mind [… utilizing AR navigation for pilot training].” (see Table 7, column three). A categorization was possible for seventeen of these answers, of which three were from females and fourteen were from males. Most responses described a use case that was categorized as “Training related to navigation”. Five answers in this category mentioned that supporting the approach would be useful, and four suggested showing a virtual flight path. The answers were given mainly by male participants with TR and or FI certificates.

The suggested use case of AR Planning showed the user how AR could be used to plan objects in real environments. Seventeen of the sixty participants gave answers to the open-ended question “Please explain use cases you have in mind. [… utilizing AR planning for pilot training]” (see Table 7, column four). A categorization was possible for sixteen answers, four of which were from females and twelve from males. Answers categorized with “Classroom training without further specification” mainly suggested a technical use case where the components of the aircraft are explained using AR. Only two participants indicated that AR could also be used to explain airspace-related topics as well as technical components.

The suggested use case of Remote Consultation and Assistance showed the participants how a remote specialist can support a person during a complex procedure using AR. Eighteen of the sixty participants gave answers to the open-ended question “Please explain use cases you have in mind. [… utilizing AR remote consultation and assistance for pilot training]” (See Table 7, column one). A categorization was possible for seventeen answers, of which four were from females and thirteen were from males. Six answers were categorized as “Simulator or aircraft training without further specification”, and three answers suggested utilizing the Remote Consultation in a solo flight to support the student pilot. Four participants described why Remote Consultation is not a good use case for pilot training. Overall, these results show that pilot tasks are often time-critical, and that too little time is available to seek consultation; furthermore, a haptic feedback is necessary for the consultant to provide effective support. The utilization of this use case in a real aircraft was mentioned by seven participants, and the usage in the simulator by two participants.

6. Discussion and Outlook

The aim of conducting this research was to identify promising application areas for AR in pilot education. Previous research results show that women and men have different learning preferences; thus, special attention was given to gender aspects ([51,52]). The survey was designed in such a way that it did not require the participants to have previous knowledge of AR. For instance, questions were enriched with further content, such as videos, to illustrate the possibilities of the technology. Nevertheless, 33.30% of the women and 39.58% of the men stated that they had already had an experience with AR, and the results of the qualitative evaluation showed that 58.33% of the women and 41.67% of the men were able to describe its basic principles. Men had experience with five of the six listed AR applications, whereas women had experience with only two. This might be due to the type of application, which may be more interesting for one gender than for the other [41]. A further reason could be that women spend less time on playing games than men [53].

Transferring and adapting established use cases from one industry to another is a common approach. In this survey, we showed the participants AR use cases which are already applied in other industries, and allowed them to rate whether they thought AR has beneficial applications in pilot training. The use case Remote Consultation and Assistance, where a remote specialist supports a novice user, was rated by 54.55% of the women and 50.00% of the men as desirable. Additional comments were given by four women and fourteen men. Support in real-world scenarios and in simulator or aircraft training is the most frequently mentioned utilization. AR Planning was described as a beneficial use case for pilot training by 63.64% of the women and 54.17% of the men. Additional comments showed that utilization is preferred, firstly, for classroom instruction and, secondly, in the area of simulator or aircraft training. Out of the three use cases, AR Navigation was considered the most promising, whereby 90.91% of the women and 62.50% of the men stated that they thought that there are beneficial applications.

The next section of the survey showed the applicability of AR to the three main parts of pilot education, namely the theoretical parts, pre-flight inspection, and practical flight training. Among the participants, 90.91% of the women and 60.42% of the men agreed that the theoretical parts of pilot training could benefit from AR. These results are supported by the qualitative data, where ten women and thirteen men mentioned the utilization of AR in classroom training, or more specifically, in classroom training related to technical and physical parts of the syllabus. Similar results could be found for the pre-flight aircraft inspection, whereby 90.91% of the women and 66.67% of the men agreed that AR is a promising technology that can be used to improve this area of pilot training. The analyzed responses to open-ended questions show that nine men (25.00%) would use AR-supported pre-flight training for tasks and procedures such as briefing during pilot training. The utilization of AR in practical flight training was considered by 63.64% of the women and 52.08% of the men as a promising area. An analysis of the comments given did not reveal this utilization area. However, eight women (72.27%) and 25 men (69.44%) thought that utilization in simulator or aircraft training would be beneficial. In particular, cockpit procedure training and emergency procedure training were mentioned in the responses.

Making educational content more interesting and easier to learn by connecting such content with game concepts is an often-used approach. Implementation details are often crucial for success, and thus the last part of the questionnaire focuses on revealing preferences for such game concepts. As the results show, “receiving feedback for correct actions”, “achieve a target to finish task”, and “receiving points if you successfully finish a task” are highly valued by both women and men, but by a higher percentage of women as compared to men. On the other hand, “The game includes a story to attract your attention” was preferred by more men.

These results show the potential for AR to be used to improve the content and content delivery in pilot education. However, the limitations of this study are the relatively small sample size, especially the small size of the female group, and the lack of randomization in the selection of participants. Because of these limitations, the statistical methods that could be used were restricted and, therefore, the data are presented descriptively and no statistical tests have been performed. Another limitation is that the participants were mainly citizens of the European Union.

In conclusion, the use case of AR Navigation as well as the areas “pre-flight inspection” and “theoretical training” could be applied to pilot education. In addition, the analysis of the qualitative data indicated that “simulator or aircraft training” and especially procedure training, were considered as relevant for future AR applications. Based on these findings, future research is needed to define AR use cases in detail. Further experiments carried out to investigate the effects of AR on pilot training could provide research findings that can serve as a foundation for proof-of-concept applications. Furthermore, future investigations should be done to validate the results of this survey on a larger scale. An equally large group of both genders chosen through random sampling would especially allow statistical tests and inference.