Next Article in Journal
Osteonecrosis of Humeral Head after Arthroscopic Capsular Release for Postoperative Shoulder Joint Stiffness: A Case Report
Next Article in Special Issue
Video-Based Gaze Detection for Oculomotor Abnormality Measurements
Previous Article in Journal
Two-Step Relaxation of Non-Equilibrium Electrons in Graphene: The Key to Understanding Pump–Probe Experiments
Previous Article in Special Issue
The Impact of Reading Modalities and Text Types on Reading in School-Age Children: An Eye-Tracking Study
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 14
Issue 3
10.3390/app14031251
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Differences between Experts and Novices in the Use of Aircraft Maintenance Documentation: Evidence from Eye Tracking

by
Florence Paris
1,2,*,
Remy Casanova
2,†,
Marie-Line Bergeonneau
1 and
Daniel Mestre
2,†
1
Airbus Helicopters, 13700 Marignane, France
2
Aix Marseille University, CNRS, ISM, 13700 Marseille, France
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Appl. Sci. 2024, 14(3), 1251; https://doi.org/10.3390/app14031251
Submission received: 14 December 2023 / Revised: 24 January 2024 / Accepted: 29 January 2024 / Published: 2 February 2024
(This article belongs to the Special Issue Eye-Tracking Technologies: Theory, Methods and Applications)
Browse Figures
Review Reports Versions Notes

Abstract

:
Maintenance is a highly procedural activity requiring motor and cognitive engagement. The aim of this experimental study was to examine how expertise affects maintenance tasks, in particular, the use of procedural documents. A total of 22 aircraft maintenance technicians were divided into two groups according to their level of expertise. Helicopter maintenance was evaluated in a real work environment, using an eye tracker, a fixed camera, and NASA-TLX to measure workload. Both groups reported a high mental load. Novices showed elevated levels of effort and mental demand. Experts were faster at all levels of the task and spent less time consulting maintenance documentation. The acquisition of procedural information was greater at the start of the task, where the gap between groups was more pronounced. This may be related to the overall planning of the task, in addition, the task was atomized, with frequent back-and-forth between execution and information intake, for all participants. Novices had a longer document consultation duration, spread over a greater number of consultations, but did not have a higher average consultation time. The results indicate a higher mental load for novices, potentially linked to an increased atomization of the task, as shown by the frequency of consultations.

1. Introduction

1.1. Context

Maintenance refers to a set of tasks aimed at preserving the physical condition of equipment or a system, allowing it to operate in accordance with its specifications [1,2]. The maintenance activity can be understood as the result of a dynamic interaction between external determinants (maintenance task) and internal determinants (maintenance operator) [3,4]. In the aircraft industry context, this activity is crucial in ensuring the airworthiness of aircraft, guaranteeing the safety of passengers, equipment, and people on the ground [5,6,7]. Various determinants emerge, contributing to the inherent complexity of the task. The aircraft maintenance task, as well as the maintenance task in general, is characterized by hazardous environments, with a high degree of inherent uncertainty and limited repeatability [8,9,10]. It is important to note the complexity involved in aircraft design. Aircraft consist of multiple interconnected and interdependent systems, each containing sub-systems that are vulnerable to cascading effects. Any malfunction in one system can have an impact on other systems or the whole aircraft [11]. Additionally, access to the part being serviced may also require the removal and re-installation of other systems. Assembly tasks have many parallels with maintenance; indeed, this is because assembly and installation tasks are integral parts of the maintenance task.

1.2. The Procedural Documentation

The complex design of aircraft systems and the severity of the consequences of errors make it subject to stringent regulatory requirements. Maintenance tasks exhibit the attributes of a “well-structured task”, as explained by Simon [12]. They are characterized by a testable standard for solutions, descriptions of problem states using objectives and sub-objectives, achievable state modifications, and knowledge integration [12]. The procedure determines the hierarchical structure of the task by sequencing it into sub-tasks with integrated detailed instructions. As a prescriptive document, the aircraft maintenance manual (AMM) explicitly defines all procedures required to maintain an aircraft. Organized into chapters and sub-chapters, this hierarchical document integrates illustrations and text. Managing uncertainty is a significant challenge for safety-critical organizations, such as nuclear power plants, the oil and chemical industries, and aviation [13]. Rules and procedures guide and structure activities by defining objectives, decision-making processes, and constraints, ultimately improving reliability and safety.

1.3. Experts versus Novices

Aircraft maintenance is carried out by aircraft maintenance technicians (AMTs). Given the complexity of the task and safety challenges inherent in the aviation industry, aircraft maintenance is carried out by a highly skilled and specialized workforce [14]. Maintenance requires a wide variety of skills and knowledge [10]. Comprehending the impact of expertise can provide valuable information to identify areas of improvement [15,16].
Expertise has been studied extensively in a wide range of fields [17,18,19]. The literature highlights the following aspects: expertise is inherently domain-specific because individuals acquire specialized problem-solving strategies within a particular domain through experiences that provide opportunities to use and organize domain-specific information [20]. In their domain, experts often develop an increased ability to perceive significant information patterns that evade those without this skill. For instance, expert electronics technicians were shown to be able to reproduce significant parts of complex circuit diagrams after only a few seconds of exposure, whereas novices were unable to do so [21]. Experts detect specific patterns, which allows them to memorize and process complex information faster than beginners. In Chase and Simon (1973) [22], experts demonstrated enhanced memory for structured stimuli (patterns of chess pieces), but did not show the same ability to recall random unstructured stimuli. These results have been replicated in various fields, including medicine and electronics [21,23,24]. Experts tend to execute actions faster and more efficiently than non-experts [25,26,27,28]. Expertise allows individuals to use previously learned rules and procedures, eliminating the need to engage in a step-by-step reasoning process for each task [29]. This phenomenon has implications for cognitive engagement. As individuals gain experience and knowledge within a domain, they construct mental models and heuristics that facilitate more efficient task performance [19,25,29]. Consequently, this leads to a reduction in the mental load required for task completion.

1.4. The Use of Procedural Information

In the context of maintenance, there is a cognitive side of the activity but in contrast to chess, mental calculation, or programming, there is also a physical side of the activity involving the execution of motor tasks. Therefore, the activity must be measured globally, but also through information gathering and execution.
There is a specificity in the cognitive aspects involved in the maintenance activity; indeed, reading a procedure has a pragmatic objective: execution. This directly affects how the procedure is used [30]. The processing of procedural documents occurs in multiple informational contexts, including the user’s prior knowledge of the system [31]. In the context of maintenance, unlike other situations where access to the procedural document at the time of the task is difficult or impossible [32,33], the operator refers to the document while performing the procedure [34,35,36]. A common phenomenon described in the literature on the use of procedure is the “atomization of actions” [37]. The activity is sequenced around two actions: procedural information intake and task execution. Participants interrupt the progression of their actions (execution of instructions) through information-gathering activities carried out on the procedural document. Atomization has been studied in a variety of contexts, including medicine [32,38,39,40], cooking recipes [37], and the use of everyday objects [41,42]. Atomization would minimize the cognitive cost of translating instructions into actions [37,43]. In multimedia learning, segmenting information into chunks serves as a strategy to reduce the cognitive load on learners engaged in information processing [44,45,46].
With regard to maintenance, several studies have studied maintenance documentation [34,47,48,49,50]. However, to date, no quantitative data have been collected on the use of documentation in a real maintenance task, looking at differences between experts and novices. Moreover, given the complexity of the procedures involved in operational aircraft maintenance tasks, it is expected that the atomization phenomenon will be observed, but we are missing empirical data.
There are various techniques for measuring procedural information use, considering the constraints associated with the maintenance activity. First, the tools used must have the least impact on the operator. For example, the Think-Aloud method is embedded in the context of qualitative research, its main limitation being the direct influence on the execution of the task, which requires the operator to perform a secondary task consisting of the verbal expression of his thoughts [51]. Ganier and his collaborators have developed a specialized software, known as TIP-EXE [32,52]. This software can blur the prescribed document. Therefore, the user can selectively deflect the desired segment of the document with a single click. At the same time, the software generates time data that provide details about the viewing order as well as the time spent exploring each specific segment. This method has many advantages; however, it also generates a cost related to an additional action for the operator and is not adapted to the study of interactions with a non-digital prescription. The method of measuring visual attention through an eye tracker is currently used in expert–novice paradigms and in the context of the study of procedural information intake [51]. As part of the experiments conducted in Jannin’s thesis [43] on suture learning, two distinct approaches were adopted to evaluate the use of the procedure: the use of TIP-EXE software and the implementation of an eye-tracking measure. The author underlines the relevance of the eye-tracking method because of its contribution to a more realistic and ecological experience. Other studies have also used eye tracking [53,54] to evaluate the application of the procedure in contexts involving static participants. The adoption of mobile eye-tracking technology offers a relevant solution for assessing the second sequencing between execution and procedure [51]. By measuring the duration of gaze within an area of interest (AOI) [55,56], it is possible to quantify the time and duration during which visual attention is devoted to maintenance documentation, in a non-binding way and via paper format. Eye movement tracking has emerged as the optimal approach for exploring visual cognitive strategies; it allows for accurate measurement in the context of complex tasks within their dynamic environments, extending to fields such as medicine [57], sports [58,59], transport [60,61,62,63,64], and construction [65,66,67].

1.5. Our Approach

This study is based on a previous study [68,69], and centered on the development of a tool to measure the use of procedure documents in aircraft maintenance. The results from one participant showed an important information intake phase at the beginning of the operation. In this study, one goal is to see if these results are generalizable and if there are specific patterns associated with expertise in procedural information acquisition. The primary aim of our study is to quantify the impact of expertise on procedural information intake in the context of a real maintenance task carried out in a maintenance hangar. The maintenance procedure theoretically provides a comprehensive description of the task, enabling its execution without additional prerequisites [3] and providing the same framework for all AMTs in the maintenance process [34,48]. We intend to measure the operators’ activity and workload, to evaluate the effectiveness of procedural documentation in bridging the gap between novices and experts. This investigation was framed by the following research questions and hypotheses:
RQ1. 
Is there an impact of expertise on the procedural information intake in the context of a maintenance task?
H1. 
We expect experts to be faster than novices in both the execution and procedural information intake.
The operationalization of this hypothesis will be based on behavioral indicators (absolute duration, document consultation duration, execution duration, percentage of time spent on the document). To provide a more detailed analysis of the use of the procedure, a closer examination of the atomization phenomenon, as described in the literature, will be proposed.
RQ2. 
Is procedural information intake atomization dependent on the expertise level?
H2. 
If the phenomenon of atomization of the task is due to resource-intensive processes associated with procedure execution, then we expect expertise to have an impact on the number of consultations and the average time spent consulting the procedure.
Finally, we wish to address the workload.
RQ3. 
Are there differences in the workload experienced by operators?
H3. 
Maintenance seems to have physical [70,71,72] and mental dimensions. Based on the literature, the difference between novices and experts is anticipated in the mental dimension.

2. Materials and Methods

2.1. Participants

The target population for our study is men and women who are helicopter AMTs and over the age of 18. The population consists of 22 participants who have been divided into two groups based on their experience. The inclusion criteria for the study are as follows: (i) be over 18 years of age; and (ii) be actively engaged in AMT work or training at the time of the study. Participants are divided into two groups according to their experience. The expert group consisted of 10 male participants with a mean age of 51.6 years ( S D = 7.6 ) and a mean experience of 30 years ( S D = 7.2 ). Inclusion in this group required a minimum of 20 years of experience as an AMT. The novice group consisted of 12 participants, with a mean age of 22.7 years ( S D = 3.3 ) and a mean experience of 7 months ( S D = 1.8 ). Participants in this group were in training at the time of the experiment and had some field experience during their training. Of the 12 participants in the novice group, 67% were male.

2.2. Materials

Data collection included a scene camera, a mobile eye tracker, and a questionnaire. A GoPro Hero 4 camera (GoPro, San Mateo, CA, USA), affixed within the maintenance area (Figure 1), captured the working area surrounding the helicopter at 1080p resolution and 30 frames per second. The Tobii Pro Glasses 2 (Tobii AB, Danderyd, Sweden) was utilized as a head-mounted mobile eye tracker to acquire data on eye fixation, and scene video captured the working area surrounding the helicopter at 1080p resolution and 30 frames per second. The Tobii Pro Glasses 2 was utilized as a head-mounted mobile eye tracker to acquire data on eye fixation and scene video. The device has an accuracy of 0.5° and a sampling rate of 100 Hz. This tool is unobtrusive and suitable for real-world data collection [61,73]. Workload measurement was undertaken using the weighted NASA Task Load Index [74] questionnaire due to its standardized and validated tool status. The 15 pairwise comparisons of dimensions were presented to the operator. The frequency with which each dimension is chosen determines its weight or importance. These weights are then multiplied by the raw ratings for each dimension, according to the Hart and Staveland procedure [74].

2.3. Protocol

Before starting the experiment, the experimenter provided a detailed explanation of the study and answered any questions the participants might have had. Each participant signed a consent form before being equipped with the eye tracker. During the task, the technician initiated it as soon as they received the procedural document, and participants were free to take as much time as they needed to complete the task.
The validation of maintenance quality was primarily anchored in the successful completion of the task and the final compliance of the helicopter with established navigability standards.
All common maintenance tools and specific tools referenced in the procedural document were available in the hangar. Upon completion, participants signaled the end of the task and were disconnected from the eye tracker. Afterward, they completed the NASA-TLX questionnaire. Aircraft maintenance technicians were tasked with inspecting the components of the right rear landing gear brake unit on an H215/225 helicopter (Airbus Helicopters, Marignane, France). This task involved a removal phase to access the area to be inspected, followed by an installation phase to return the helicopter to its original configuration. The task could be completed by one technician.
The prescribed reference for this task was the aircraft maintenance manual (AMM), which contained the procedure for inspecting brake unit parts on the helicopter model (ensuring the absence of scratches, wear, corrosion, impact marks, leaks, and the verification of their condition). The procedure was 21 pages in a hard copy format, spread over five work cards. The information in the document was either contained entirely in a work card or a chapter within a work card. There were 15 pages of text and 6 pages of figures in the procedural documentation.

2.4. Data Analysis

2.4.1. Previous Result

Prior to this study, we conducted previous research [69] aimed at defining a methodology to characterize the use of procedural documents using an eye-tracking device. It is based on a temporal qualification of the data, allowing us to relate the consultation times within the procedure and the main steps of the tasks materialized by milestones (Figure 2). The methodology developed was tested on the same task as in this experiment with one participant. The division of the task into temporal sequences based on observable milestones makes it possible to enrich the analysis of variables and to compare participants with varying overall task durations.
The milestones presented in Figure 2 and used in this study were selected because of their succinct yet recognizable actions (observable by priority through the egocentric camera of the eye tracker or by default through the scene camera), and the specific order in which they are realized. Each removal/installation is described in the procedure and was observed for each participant. The visualization of the distribution of the use of the document during the task (Figure 3) allowed us to see the need to characterize the back-and-forth between reading times and execution times. The upper part of the figure shows the breakdown of the task into three phases and nine steps. The three main phases of the maintenance operation are removal, inspection, and installation. The steps are numbered from I to IX. The steps are delimited by the start and end of the operation and by eight numbered milestones whose labels are shown in white frames and presented in Figure 2. The lower part of the figure shows the evolution of the document consultation for a participant based on the data of the previous research [69]. The background is divided into segments colored according to the task step (shown in the upper part). The width of the colored areas in the background shows the relative duration of each step. The gray strips represent the time the participant spent looking at the maintenance document. The black curve represents the cumulative percentage of time the participant spent viewing the document. Based on this observation, we measured the number of consultations and the average consultation duration.

2.4.2. Variables

We identify four independent variables (expertise, phases, steps, NASA TLX dimensions) and seven dependent variables. The definitions of the dependent variables are presented in Table 1. All dependent variables, except the weighted NASA TLX score, are extracted either globally or for each phase or step.

2.4.3. Data Quality

One participant did not complete the NASA-TLX questionnaire, resulting in a sample size of 21 for the workload analysis (9 experts and 12 novices). The temporal variables analysis excluded two experts and two novices who had a low gaze sample percentage (<60%) and one novice due to incomplete eye data recording (8 experts and 9 novices). The gaze sample percentages for both groups were as follows: between 60% and 70% for two experts, between 70% and 80% for three experts and one novice, and more than 80% for three experts and nine novices. The decision to exclude participants demonstrating a gaze sample percentage below 60% was rooted in our commitment to precision and to avoid potential biases in the analysis.

2.4.4. Extraction of Variables Related to Instruction Consultation

To characterize the acquisition of procedural information [75,76], temporal behavioral data were collected to obtain all dependent variables related to the use of procedural documentation. In order to detect document consultation, we used the area of interest (AOI) technique on all eye-tracking data collected in the field. An AOI was defined [56], for all pages of the document, for all participants. We used Tobii Pro Lab 1.152 ® analysis software, which supports both manual and automated AOI mapping. We manually checked fixations for all data collected because a validity test of the automated mapping technique on the previous research [69] yielded unsatisfactory results. The data processing resulted in a time series, indicating when the participants consulted the procedural documents. Gaze fixations were temporally qualified to associate them with the corresponding phase and step (Figure 4). We used a filter threshold on the mapped ocular data to group the fixations made on the document that were spaced less than 3 s apart to form a consultation. We tested a threshold range from 0.5 to 10 s on the data, and we found that the selection of the filter threshold did not affect the inter-participant differences. The decision to use a 3-second threshold was based on the observations made during the previous research [69]. This allowed us to extract the following variables: the number of consultations and average consultation time.

2.4.5. Statistical Analysis

We conducted a statistical analysis on the three levels of the maintenance task: entire task, phase level, and step level. To test our hypotheses, we used multiple analyses of variance (ANOVAs) on the dependent variables. At the entire task level, we performed one-way ANOVA with the factor of expertise (novice, expert). At the phase level, we used repeated measure two-way ANOVA with factors of expertise (novice, expert) and phase (removal, inspection, installation). At the step level, we used repeated measure two-way ANOVA with factors of expertise (novice, expert) and Step (I to IX). Similarly, for the weighted NASA TLX scores, we used repeated measure two-way ANOVA with factors of expertise (novice, expert) and dimensions (mental, physical, temporal, performance, effort, frustration). In the case of significant ANOVA results, we conducted post hoc analyses using Student’s Newman–Keuls post hoc tests to determine significant differences. A significance level of α = 0.05 was used for all analyses.

2.4.6. Data Processing

For data analysis, three software tools were used to process and analyze the data. Tobii Pro Lab (1.152, Tobii AB, Danderyd, Sweden) was used for processing ocular data and raw data exports. Matlab scripts (R2020b, The MathWorks Inc., Natick, MA, USA), were used to obtain dependent variables. Finally, RStudio scripts (1.3.959, RStudio, PBC, Boston, MA, USA) were used to calculate statistical indicators, perform statistical tests, and generate figures.

3. Results

3.1. Workload

This section aims to investigate the effect of expertise on workload. Table 2 presents the mean values and interquartile ranges for the NASA TLX dimension among both experts and novices. The ANOVA revealed no significant effect of expertise on the weighted NASA-TLX. However, the dimension had a significant effect on workload ( F ( 5 , 95 ) = 14.8 , p < 0.001 ), with the mental dimension being significantly higher than the other dimensions for all groups (all p < 0.001 ). Additionally, an interaction effect of expertise × dimension was observed ( F ( 5 , 95 ) = 5.7 , p < 0.001 ), with novices reporting significantly higher load than experts in the effort ( p < 0.05 ) and mental ( p < 0.05 ) dimensions.

3.2. Procedural Information Intake

All participants used the AMM during the maintenance task. Table 3 presents the mean values and interquartile ranges for the three variables among both experts and novices.

3.2.1. Main Effect of Phase/Step

The ANOVA analysis revealed significant effects of the phase on both document consultation duration ( F ( 2 , 30 ) = 55.2 , p < 0.001 ) and the percentage of time spent consulting the document ( F ( 2 , 30 ) = 17.6 , p < 0.001 ). Post hoc analyses revealed significant differences (all p < 0.001 ) in document consultation duration across all phases. The removal phase exhibited the longest duration, followed by installation and inspection. Additionally, post hoc analysis revealed that the removal phase was significantly higher than the other phases in terms of the percentage of time spent consulting the document ( p < 0.001 ).
A main effect of phase was found on both absolute duration ( F ( 2 , 30 ) = 62.1 , p < 0.001 ) and execution duration ( F ( 2 , 30 ) = 51.2 , p < 0.001 ). Post hoc analyses revealed significant differences between all phases for both absolute duration (p < 0.05) and execution duration ( p < 0.001 ). We observed that for both variables, the installation phase had the longest duration, followed by the removal and inspection phases.
On a more microscopic scale, a significant main effect of the step was observed for all four variables: document consultation duration ( F ( 8 , 120 ) = 25.3 , p < 0.001 ), percentage of time spent on the document ( F ( 8 , 120 ) = 33.8 , p < 0.001 ), absolute duration ( F ( 8 , 120 ) = 6.4 , p < 0.001 ), and execution duration ( F ( 8 , 120 ) = 6.2 , p < 0.001 ). Post hoc analyses revealed that step I exhibited a significantly higher percentage of time allocated to consulting documentation and document consultation duration ( p < 0.001 for both indicators). Step VII exhibited significantly longer absolute duration and execution duration compared to the other steps ( p < 0.01 and p < 0.001 , respectively).

3.2.2. Main Effect of Expertise

Data analysis revealed a significant main effect of expertise on the duration of document consultation ( F ( 1 , 15 ) = 32.2 , p < 0.001 ), the percentage of time spent consulting documentation ( F ( 1 , 15 ) = 15.5 , p < 0.01 ), absolute duration ( F ( 1 , 15 ) = 21.1 , p < 0.001 ), and execution ( F ( 1 , 15 ) = 9.26 , p < 0.01 ).

3.2.3. Interaction Effect of Phase/Step x Expertise

Regarding the interaction effect between phase and expertise, the results showed no significant interaction effect for execution or the percentage of time spent on documentation. However, a significant interaction effect was observed for the consultation duration ( F ( 2 , 30 ) = 23.6 , p < 0.001 ) and absolute duration ( F ( 2 , 30 ) = 22.1 , p < 0.05 ). In a post hoc analysis of the two variables, significant differences in document consultation duration emerged between the novice and expert groups for both the removal and installation phases ( p < 0.001 ). During the removal phase, experts showed an average document consultation duration of 5.6 min ( S D = 2.7 min), while novices showed a significantly longer average of 24.7 min ( S D = 9.7 min). Similarly, in the installation phase, experts showed an average document consultation duration of 4.3 min ( S D = 2.8 min), while novices showed an average duration of 12.9 min ( S D = 3.3 min). Furthermore, a significant difference was observed within the novice group across all phases ( p < 0.001 ). This was not observed in the expert group.
When examining the interaction effect of step x expertise, the results show a significant effect on the duration of the consultation ( F ( 8 , 120 ) = 14.6 , p < 0.001 ), as well as a significant effect on the percentage of time spent on documentation ( F ( 8 , 120 ) = 8.3 , p < 0.001 ). No interaction effect is found for absolute and execution duration.
Post hoc analyses were conducted for consultation duration and the percentage of time spent on documentation. In relation to the duration of document consultation during Step I (Figure 5), a significant difference was found between the novice and expert groups at this point ( p < 0.001 ). Additionally, in terms of the percentage of time spent on documentation during Step I of the task (as shown in Figure 5), both novices ( σ = 55.6 % , S D = 14.3 % ) and experts ( σ = 22.1 % , S D = 11.8 % ) allocated the highest percentage of time to document consultation. In particular, Step I stands out as significantly higher than all other steps for novices (all p < 0.001 ) and, with the exception of Step II, for experts (all p < 0.05 ).
In this section, we investigated how expertise influences the variables that compose absolute duration. The results demonstrate that expertise has a significant impact on absolute duration, with a greater impact on consultation than on execution. More specifically, experts perform the maintenance task at an average of 45.8 % faster than novices. This distinction becomes more apparent when consulting instructions, where experts are 71.6 % faster than novices. For execution, the difference is 35.9 % .

3.3. Atomization

The analysis focused on indicators measuring the phenomenon of atomization, including the number of consultations and the average duration of consultations.

3.3.1. Main Effect of Phase/Step

For both variables, there are significant main effects on phase, with ( F ( 2 , 30 ) = 25.4 , p < 0.001 ) for consultation and ( F ( 2 , 30 ) = 13.3 , p < 0.001 ) for average consultation duration. The number of consultations is lower in the inspection phase (all p < 0.001 ). The average consultation duration is longer in removal (all p < 0.001 ). There is also a significant main effect on the step, with ( F ( 8 , 120 ) = 21.4 , p < 0.001 ) for consultation and ( F ( 8 , 120 ) = 13.7 , p < 0.001 ) for the average consultation duration. There is more consultation on steps V and VII compared to all other steps (all p < 0.05 ). The average consultation duration is longer in Step I ( p < 0.001 ).

3.3.2. Main Effect of Expertise

Only the number of consultations was significantly impacted by expertise ( F ( 1 , 15 ) = 22.7 , p < 0.001 ). Novice participants conducted an average of 202 consultations ( S D = 47.2 ), while expert participants performed 88 consultations ( S D = 50.8 ).
The average consultation duration ( F ( 1 , 15 ) = 2.1 , p > 0.05 ) for all participants was 10.8 s, with experts averaging 9.1 s and novices averaging 12.2 seconds.

3.3.3. Interaction Effect of Phase/Step x Expertise

The phase-level ANOVA demonstrated a significant interaction effect solely on the number of consultations ( F ( 8 , 120 ) = 5.9 , p < 0.01 ). There is an expert–novice difference in the removal and installation phases (all p < 0.001 ). For novice operators, there is a lower number of consultations during inspection (all p < 0.001 ).
The step-level ANOVA demonstrated a significant interaction effect solely on the average consultation duration ( F ( 8 , 120 ) = 2.3 , p < 0.05 ) (Figure 6). For Step I, post hoc analyses indicated that average consultation times were longer than for all other steps. This result applies to both novices (all p < 0.001 ) and experts (all p < 0.001 ), with the exception of Step II for experts.

4. Discussion

In this study, we investigated the impact of expertise on aircraft maintenance activity.
The key findings of our study can be summarized as follows: Our indicators on activity duration are consistent with the body of literature on the subject [19,26,27,28,77,78]. Experts have a lower task completion time than novices (H1). The maintenance task encompasses the execution of motor actions that are prescribed by the procedural instructions outlined in the aircraft maintenance manual. Experts are faster on both sides of the task (execution duration and document consultation). Differences between novices and experts are more important in the time dedicated to procedural information intake. On average, novices took 35.9% longer to execute the task and 71.6% longer to consult the documentation. All participants consulted the procedural document, regardless of the level of expertise of the participants, with frequent back-and-forth between the documentation consultation activity and execution throughout the task. These results are consistent with the context of the maintenance task, particularly with the safety aspect inherent in the aviation field, but also with the complexity of the procedures [34] necessary for the maintenance of a helicopter, which makes it impossible to fully memorize it [50]. Expertise has a global effect on all tasks of the maintenance activity, but its effect on document consultation is more complex.
Concerning the NASA-TLX dimensions, novices showed a significantly higher score on both the effort and the mental workload (H3) dimensions as compared to experts. These findings can be explained by considering the influence of the knowledge and skills of the experts on the efficiency of the maintenance activities: both in the extraction of crucial information relative to the current task stage and in the planning of the execution based on the information derived from the procedural document. Our study shows that even if the maintenance activity generates physical constraints (displacement, awkward postures, load bearing) [9,70,72,79], the use of procedures requires a strong mobilization of attentional and cognitive resources, corroborating the findings of [45,50,80,81,82].
Moreover, our results show that the amount of procedural information intake is not linear during the task. During the removal phase, operators, regardless of their level of expertise, place a greater emphasis on procedural information intake, particularly during the first step of removal for novices and the first and second steps for experts. Removal is theoretically less complicated than installation [83]. Access to certain parts may remain restricted during the removal phase until other components have been deposited. These accessibility constraints are inherent in the system and embody the interdependent relationships that structure the instructions within the procedure. They provide additional information by clarifying the sequence of actions to be followed by the technician. We suggest that the longer duration and greater importance of information gathering at the beginning of a task are not solely related to the execution of instructions during those intervals, but rather to the overall planning of the maintenance task. In this study, procedural information intake would serve two purposes, an execution purpose throughout the task and a general planning purpose at the beginning of the task.
The data relating to the number of consultations and average consultation duration illustrates the phenomenon of atomization in the maintenance task. The aircraft maintenance activity observed for all participants shows a phenomenon observed in the context of other tasks involving a procedural document in other domains [37,40,51]. The whole consultation duration of the document is segmented into multiple short consultations. The maintenance activity consists of multiple processes of information intake and action, resulting in a multitude of specific planning periods linked to the part of instruction consulted.
In our task, novices exhibit a prolonged duration of procedural document consultation, and this extension is manifested in a higher frequency of consultations (H2). However, we do not observe a longer average duration of consultations for novices, if we consider the task in its entirety. These results can be explained by the fact that novices tend to break the overall task into more information acquisition cycles in order to reduce cognitive load, i.e., to process smaller chunks of information at once [45]. Previous work has shown that this process limits the amount of information in working memory in order to reduce the cost of processing instructions [42,80]. However, when focusing on what we identified as the primary planning phase of the task. It is observed that the largest discrepancies between the groups occur in Step I. These discrepancies were observed in terms of consultation duration, the percentage of time devoted to consultation, and the average consultation duration. It is important to note that this effect among novices is not counterbalanced by a decrease in consultation duration during subsequent steps of the task. Our study aligns with the existing literature on the process of forming a mental representation of a problem-solving task environment, referred to as the ’basic problem space’ [84,85]. Novices appear to invest more time in procedural information intake for task planning. This suggests that experts, who are familiar with a particular problem type, can efficiently draw upon previous problem spaces and distinguish only necessary information for general planning without going into detailed instructions that will be processed later during execution. This is in contrast to novices who must allocate additional time and effort to construct the problem space from scratch due to their lack of familiarity with the problem type.

4.1. Practical Implications

Our study presents empirical findings on procedural information intake behaviors in the authentic context of maintenance tasks. The research involved using the procedure in its natural paper format, without any imposed restrictions or specific usage instructions. The maintenance task was extensive, lasting over an hour and requiring the execution of multiple instructions from a document comprising numerous pages. Understanding how aircraft maintenance technicians use aircraft maintenance manuals has practical implications for procedural design and AMT training. The information intake within AMMs is achieved through multiple short consultations, averaging a duration of 10.8 s. Good readers can process information at a rate of 200 to 400 words per minute [86,87]. This indicates that the amount of information absorbed during each consultation is relatively small. In document design, it is important to optimize the physical format to facilitate the efficient location of relevant information [54,88]. Our study highlights the potential challenges faced by novice individuals when initiating the maintenance task; training programs can be designed to emphasize the initial interaction with the global procedural document at the start of a task. Consequently, workers could learn to extract the important cues for planning their entire task. Our study shows that mobile eye tracking is suitable for field studies in the context of aircraft maintenance, even over long periods of measurement with procedural documents in paper format.

4.2. Limitations and Future Studies

There are some limitations associated with this study that need to be mentioned. We believe that the main limitation of this study is our attempt to be as close as possible to real maintenance conditions by not imposing any constraints on the operator’s activities. This has implications for methodological choices. The mental workload measurement might have been addressed using physiological indicators [89], such as heart rate or pupil diameter. However, the lighting conditions of our hangar or the movement inherent to maintenance activity made these indicators difficult to set up.
Regarding the depth of the analysis of eye-tracking data, our study considers only basic indicators and procedural documentation in a single object of interest. It would be beneficial to include an analysis of the information extracted from the documentation to further develop the initial contributions presented in this study. For example, it might be interesting to analyze the effect of expertise on document navigation.
Finally, while the primary focus of our study is on procedural aspects, we recognize the importance of including the physical dimension, which may provide a perspective for a more comprehensive understanding of expertise in maintenance tasks.

5. Conclusions

This study investigated the effect of expertise on aircraft maintenance tasks, including the use of procedural documents. The analysis of activity, using step and phase division, as well as the measurement of gaze behavior and workload, enabled us to assess the influence of expertise in a real-life industrial context. The results show that experts are faster than novices at all levels of the task (whole task, phases, steps), as well as in the execution and information intake. The study allowed us to characterize the way procedural information is acquired about a population of AMTs. The results show that the acquisition of procedural information is more important at the beginning of the task and that there is a back-and-forth between execution and information acquisition. This procedural information-gathering time at the beginning of the task can be attributed to the overall planning of the maintenance task prior to the start of its execution. This is when the expert–novice gap is the most important. The extra time spent by novices during this phase is not offset by a less significant use of procedures afterward. The novices had to exert more mental effort than the experts to accomplish the task; it is still important to note that both groups reported a mental dimension that was superior to the other scales measured. The findings indicate that novices experience a higher workload during the maintenance task compared to experts. This could be attributed to novices exhibiting increased task atomization, as evidenced by the greater number of consultations. These results have important implications for AMT training. They demonstrate the importance of focusing on the intake of procedural information and the use of procedures in the planning of maintenance tasks. The findings suggest that training programs for novice AMTs place a strong emphasis on the effective intake of procedural information and the use of procedures in the planning of maintenance tasks. The results suggest that the design of procedures should focus on facilitating the acquisition of procedural information. Our study suggests that the information extraction process within the AMM differs between the planning and completion phases of the task. This could impact the development of procedures. This study paves the way for future research on the effect of expertise on the aircraft maintenance task. Future research could explore the effect of expertise on information gathering within the maintenance procedure.

Author Contributions

Conceptualization, F.P., R.C., M.-L.B. and D.M.; methodology, F.P., R.C. and D.M.; software, F.P.; validation, R.C. and D.M.; formal analysis, F.P., R.C. and D.M.; investigation, F.P.; resources, M.-L.B.; data curation, F.P.; writing—original draft preparation, F.P.; writing—review and editing, F.P., R.C., M.-L.B. and D.M.; visualization, F.P.; supervision, R.C. and D.M.; project administration, D.M.; funding acquisition, R.C., M.-L.B. and D.M. All authors have read and agreed to the published version of the manuscript.

Funding

Florence Paris is employed by Airbus Helicopters under the industrial convention CIFRE n° 2020/0749 between ANRT (Association Nationale de la Recherche et de la Technologie) and Airbus Helicopters. She is affiliated with Aix-Marseille University as a Doctoral Candidate, under collaboration agreement n° IPAH13430 between Airbus Helicopters and Aix-Marseille University (Institute of Movement Sciences).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data of the resulting values of measurements are not publicly available. For further information regarding the data, contact the authors of this article.

Conflicts of Interest

Florence Paris and Marie-Line Bergeonneau were employed by the company Airbus Helicopters. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
AMMAircraft Maintenance Manual
AMTaircraft maintenance technician
ANOVAanalysis of variance
AOIarea of interest
IQRinterquartile range
MSDmusculoskeletal disorder

References

  1. Blaise, J.C.; Levrat, E.; Iung, B. Process approach-based methodology for safe maintenance operation: From concepts to SPRIMI software prototype. Saf. Sci. 2014, 70, 99–113. [Google Scholar] [CrossRef]
  2. Jasiulewicz-Kaczmarek, M. The role of ergonomics in implementation of the social aspect of sustainability, illustrated with the example of maintenance. In Occupational Safety and Hygiene; CRC Press: London, UK, 2013; pp. 1–6. [Google Scholar] [CrossRef]
  3. Leplat, J.; Hoc, J.M. Tâche et activité dans l’analyse psychologique des situations. Cah. Psychol. Cogn. Anc Cah. Psychol. Marseille 1983, 3, 49–63. [Google Scholar]
  4. Darses, F. Activity analysis: Not what it was! Trav. Hum. 2016, 79, 193–208. [Google Scholar] [CrossRef]
  5. Latorella, K.; Prabhu, P. A review of human error in aviation maintenance and inspection. Int. J. Ind. Ergon. 2000, 26, 133–161. [Google Scholar] [CrossRef]
  6. Gunes, T.; Turhan, U.; Açıkel, B. An Assessment of Aircraft Maintenance Technician Competency. Int. J. Aviat. Sci. Technol. 2020, 1, 22–29. [Google Scholar] [CrossRef]
  7. Hobbs, A. An Overview of Human Factors in Aviation Maintenance; Aviation Research and Analysis AR-2008-055; Australian Transport Safety Bureau: Canberra, Australia, 2008; Issue: AR-2008-055. [Google Scholar]
  8. De La Garza, C.; Weill-Fassina, A. Méthode d’analyse des difficultés de gestion du risque dans une activité collective: L’entretien des voies ferrées. Saf. Sci. 1995, 18, 157–180. [Google Scholar] [CrossRef]
  9. Grusenmeyer, C. Les accidents liés à la maintenance. Etude bibliographique; Technical Report NS 248; Institut National de Recherche et de Sécurité (INRS): Paris, France, 2005. [Google Scholar]
  10. Reiman, T. Understanding maintenance work in safety-critical organisations—Managing the performance variability. Theor. Issues Ergon. Sci. 2011, 12, 339–366. [Google Scholar] [CrossRef]
  11. Federal Aviation Administration. Aviation Maintenance Technician Handbook–General; Technical Report FAA-H-8083-30B; U.S. Department of Transportation: Oklahoma City, OK, USA, 2023; Issue: (FAA-H-8083-30B). [Google Scholar]
  12. Simon, H.A. The structure of ill structured problems. Artif. Intell. 1973, 4, 181–201. [Google Scholar] [CrossRef]
  13. Oedewald, P.; Reiman, T. Special characteristics of safety critical organizations. Work Psychol. Perspect. VTT Publ. 2007, 633, 1–126. [Google Scholar]
  14. Haritos, T.; Macchiarella, N. A mobile application of augmented reality for aerospace maintenance training. In Proceedings of the 24th Digital Avionics Systems Conference, Washington, DC, USA, 30 October–3 November 2005; Volume 1, pp. 5.B.3–5.1. [Google Scholar] [CrossRef]
  15. Charness, N.; Tuffiash, M. The Role of Expertise Research and Human Factors in Capturing, Explaining, and Producing Superior Performance. Hum. Factors 2008, 50, 427–432. [Google Scholar] [CrossRef]
  16. Farrington-Darby, T.; Wilson, J.R. The nature of expertise: A review. Appl. Ergon. 2006, 37, 17–32. [Google Scholar] [CrossRef]
  17. Ericsson, K.A. The Influence of Experience and Deliberate Practice on the Development of Superior Expert Performance. In The Cambridge Handbook of Expertise and Expert Performance; Cambridge University Press: Cambridge, UK, 2006; pp. 683–703. [Google Scholar] [CrossRef]
  18. Ericsson, K.; Krampe, R.; Tesch-Roemer, C. The Role of Deliberate Practice in the Acquisition of Expert Performance. Psychol. Rev. 1993, 100, 363–406. [Google Scholar] [CrossRef]
  19. Chi, M.T.; Glaser, R.; Rees, E. Expertise in Problem Solving; Technical Report ADA100138; University of Pittsburgh, Pa. Learning Researchand Development Center, National Inst. of Education (ED): Washington, DC, USA; Office of Naval Research: Arlington, VA, USA, 1981; Issue: LRDC-1981/3. [Google Scholar]
  20. Voss, J.F.; Greene, T.R.; Post, T.A.; Penner, B.C. Problem-Solving Skill in the Social Sciences. Psychol. Learn. Motiv. 1983, 17, 165–213. [Google Scholar] [CrossRef]
  21. Egan, D.E.; Schwartz, B.J. Chunking in recall of symbolic drawings. Mem. Cogn. 1979, 7, 149–158. [Google Scholar] [CrossRef]
  22. Chase, W.G.; Simon, H.A. Perception in chess. Cogn. Psychol. 1973, 4, 55–81. [Google Scholar] [CrossRef]
  23. Norman, G.R.; Brooks, L.R.; Allen, S.W. Recall by expert medical practitioners and novices as a record of processing attention. J. Exp. Psychol. Learn. Mem. Cogn. 1989, 15, 1166–1174. [Google Scholar] [CrossRef] [PubMed]
  24. Didierjean, A.; Ferrari, V.; Cauzinille-Marmèche, E. L’expertise cognitive au jeu d’échecs: Quoi de neuf depuis De Groot (1946)? L’Année Psychol. 2004, 104, 771–793. [Google Scholar] [CrossRef]
  25. Chi, M.T.H.; Glaser, R.; Farr, M.J. The Nature of Expertise; Psychology Press: New York, NY, USA, 1988; Google-Books-ID:Nwx8AgAAQBAJ. [Google Scholar]
  26. Haji, F.A.; Khan, R.; Regehr, G.; Drake, J.; de Ribaupierre, S.; Dubrowski, A. Measuring cognitive load during simulation-based psychomotor skills training: Sensitivity of secondary-task performance and subjective ratings. Adv. Health Sci. Educ. Theory Pract. 2015, 20, 1237–1253. [Google Scholar] [CrossRef]
  27. Judkins, T.N.; Oleynikov, D.; Stergiou, N. Objective evaluation of expert and novice performance during robotic surgical training tasks. Surg. Endosc. 2009, 23, 590–597. [Google Scholar] [CrossRef]
  28. Kieras, D.; Tibbits, M.; Bovair, S.; Psychology, A.U.T.D.O. How Experts and Nonexperts Operate Electronic Equipment from Instructions; Technical Report ADA139451; Department of Psychology, University of Arizona: Tucson, AZ, USA, 1984. [Google Scholar]
  29. Lesgold, A.M. Acquiring Expertise; Technical Report PDS-5; University of Pittsburgh, Learning Research and Development Center: Pittsburgh, PA, USA, 1983. [Google Scholar]
  30. Köhler, C.; Kékenbosch, C.; Verstiggel, J.C. La compréhension d’un texte procédural: Un processus à profondeur variable. Int. J. Psychol. 2000, 35, 258–269. [Google Scholar] [CrossRef]
  31. Ganier, F.; Gombert, J.; Michel, F. Effets du format de présentation des instructions sur l’apprentissage de procédures à l’aide de documents techniques. Le Trav. Hum. 2000, 63, 121–152. [Google Scholar]
  32. Ganier, F.; de Vries, P. Are instructions in video format always better than photographs when learning manual techniques? The case of learning how to do sutures. Learn. Instr. 2016, 44, 87–96. [Google Scholar] [CrossRef]
  33. Biard, N. L’apprentissage de procédures médicales par vidéo: Effets de la segmentation et du contrôle du rythme par l’apprenant. Ph.D. Thesis, Université de Bretagne occidentale, Brest, France, 2019. [Google Scholar]
  34. Robin, D. Usages et Bénéfices de la 3D en Maintenance Aéronautique. Ph.D. Thesis, Toulouse II Le Mirail, Toulouse, France, 2012. [Google Scholar] [CrossRef]
  35. Leplat, J. Eléments pour l’étude des documents prescripteurs. In Repères pour L’analyse de L’activité en Ergonomie; Le Travail humain, Presses Universitaires de France: Paris, France, 2008; pp. 93–130. [Google Scholar]
  36. Anderson, J.R. Rules of the Mind; Psychology Press: New York, NY, USA, 2014. [Google Scholar] [CrossRef]
  37. Vermersch, P. Données d’observation sur l’utilisation d’une consigne écrite: L’atomisation de l’action. Le Trav. Hum. 1985, 48, 161–172. [Google Scholar]
  38. Chaniaud, N.; Natacha, M.; Emilie, L.E.; Olga, M. Impact of the format of user instructions on the handling of a wrist blood pressure monitor. Cogn. Process. 2021, 22, 261–275. [Google Scholar]
  39. Jannin, L.; Ganier, F.; de Vries, P. Ego or Heterocentric? Effects of The Perspective of Video or Pictures Instructions on Procedural Learning. L’Année Psychol. 2022, 122, 439–470. [Google Scholar] [CrossRef]
  40. Jannin, L.; Ganier, F.; De Vries, P. Atomized or delayed execution? An alternative paradigm for the study of procedural learning. J. Educ. Psychol. 2019, 111, 1406–1415. [Google Scholar] [CrossRef]
  41. Duggan, G.B.; Payne, S.J. Interleaving reading and acting while following procedural instructions. J. Exp. Psychol. Appl. 2001, 7, 297–307. [Google Scholar] [CrossRef] [PubMed]
  42. Ganier, F. Le Traitement Cognitif: Déterminant de la Conception des Documents Procéduraux. Ph.D. Thesis, Université de Bourgogne, Dijon, France, 1999. [Google Scholar]
  43. Jannin, L. Approche Psycho-Ergonomique de L’usage de la Simulation en e-Learning Pour L’apprentissage de Procédures: Le cas du Point de Suture. Ph.D. Thesis, Université de Bretagne Occidentale, Brest, France, 2020. [Google Scholar]
  44. Mayer, R.E.; Moreno, R. Nine Ways to Reduce Cognitive Load in Multimedia Learning. Educ. Psychol. 2003, 38, 43–52. [Google Scholar] [CrossRef]
  45. Fraser, K.L.; Ayres, P.; Sweller, J. Cognitive Load Theory for the Design of Medical Simulations. Simul. Healthc. J. Soc. Fo Simul. Healthc. 2015, 10, 295–307. [Google Scholar] [CrossRef]
  46. Khacharem, A.; Spanjers, I.A.E.; Zoudji, B.; Kalyuga, S.; Ripoll, H. Using segmentation to support the learning from animated soccer scenes: An effect of prior knowledge. Psychol. Sport Exerc. 2013, 14, 154–160. [Google Scholar] [CrossRef]
  47. Zafiharimalala, H.; Tricot, A. Text signals in the aircraft maintenance documentation. In Proceedings of the 8th MAD Multidisciplinary Perspectives on Signalling Text Organisation, Moissac, France, 17–20 March 2010. [Google Scholar]
  48. Zafiharimalala, H. Étude Ergonomique Pour la Consultation sur écran de Petite Taille de la Documentation de Maintenance Aéronautique. Ph.D. Thesis, Université Toulouse le Mirail–Toulouse II, Toulouse, France, 2011. [Google Scholar]
  49. Zafiharimalala, H.; Tricot, A. Vers une Prise en Compte de L’utilisateur dans la Conception de Documents en Maintenance Aéronautique. In Proceedings of the Prise en Compte de l’Utilisateur dans les Systèmes d’Information, Toulouse, France, 26 May 2009; pp. 63–74. [Google Scholar]
  50. Cippelletti, E. Aide à la Conception, test de L’usage et de L’acceptation D’un Logiciel de Maintenance. Ph.D. Thesis, Université Grenoble Alpes, Grenoble, France, 2017. [Google Scholar]
  51. Meng, M. Using Eye Tracking to Study Information Selection and Use in Procedures. IEEE Trans. Prof. Commun. 2023, 66, 7–25. [Google Scholar] [CrossRef]
  52. Ganier, F. 3-Méthodes pour l’étude de la compréhension d’instructions et l’évaluation des documents procéduraux. In Comprendre la Documentation Technique; Le Travail Humain, Presses Universitaires de France: Paris, France, 2013; pp. 61–86. [Google Scholar]
  53. Alamargot, D.; Chesnet, D.; Dansac, C.; Ros, C. Eye and Pen: A new device for studying reading during writing. Behav. Res. Methods 2006, 38, 287–299. [Google Scholar] [CrossRef]
  54. Schmid, S.; Baccino, T. Stratégies de lecture pour les textes à consigne. Langages 2001, 35, 105–124. [Google Scholar] [CrossRef]
  55. Duchowski, A. Suggested Empirical Guidelines. In Eye Tracking Methodology: Theory and Practice; Springer: London, UK, 2007; pp. 171–179. [Google Scholar] [CrossRef]
  56. Jacob, R.J.; Karn, K.S. Commentary on Section 4—Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. In The Mind’s Eye; Hyönä, J., Radach, R., Deubel, H., Eds.; North-Holland: Amsterdam, The Netherlands, 2003; pp. 573–605. [Google Scholar] [CrossRef]
  57. Bapna, T.; Valles, J.; Leng, S.; Pacilli, M.; Nataraja, R.M. Eye-tracking in surgery: A systematic review. ANZ J. Surg. 2023, 93, 2600–2608. [Google Scholar] [CrossRef]
  58. Hüttermann, S.; Noël, B.; Memmert, D. Eye tracking in high-performance sports: Evaluation of its application in expert athletes. Int. J. Comput. Sci. Sport 2018, 17, 182–203. [Google Scholar] [CrossRef]
  59. Espino Palma, C.; Luis del Campo, V.; Muñoz Marín, D. Visual Behaviours of Expert Padel Athletes When Playing on Court: An In Situ Approach with a Portable Eye Tracker. Sensors 2023, 23, 1438. [Google Scholar] [CrossRef]
  60. Kapitaniak, B.; Walczak, M.; Kosobudzki, M.; Jóźwiak, Z.; Bortkiewicz, A. Application of eye-tracking in drivers testing: A review of research. Int. J. Occup. Med. Environ. Health 2015, 28, 941–954. [Google Scholar] [CrossRef]
  61. Martinez-Marquez, D.; Pingali, S.; Panuwatwanich, K.; Stewart, R.A.; Mohamed, S. Application of Eye Tracking Technology in Aviation, Maritime, and Construction Industries: A Systematic Review. Sensors 2021, 21, 4289. [Google Scholar] [CrossRef] [PubMed]
  62. Vrzakova, H.; Bednarik, R. Hard lessons learned: Mobile eye-tracking in cockpits. In Proceedings of the 4th Workshop on Eye Gaze in Intelligent Human Machine Interaction, New York, NY, USA, 26 October 2012; Gaze-In ’12. pp. 1–6. [Google Scholar] [CrossRef]
  63. Madlenak, R.; Chinoracky, R.; Stalmasekova, N.; Madlenakova, L. Investigating the Effect of Outdoor Advertising on Consumer Decisions: An Eye-Tracking and A/B Testing Study of Car Drivers’ Perception. Appl. Sci. 2023, 13, 6808. [Google Scholar] [CrossRef]
  64. Madlenak, R.; Masek, J.; Madlenakova, L.; Chinoracky, R. Eye-Tracking Investigation of the Train Driver’s: A Case Study. Appl. Sci. 2023, 13, 2437. [Google Scholar] [CrossRef]
  65. Hasanzadeh, S.; Esmaeili, B.; Dodd, M.D. Measuring Construction Workers’ Real-Time Situation Awareness Using Mobile Eye-Tracking. In Proceedings of the Construction Research Congress 2016, San Juan, Puerto Rico, 31 May–2 June 2016; pp. 2894–2904. [Google Scholar] [CrossRef]
  66. Hasanzadeh, S.; Esmaeili, B.; Dodd, M.D. Examining the Relationship between Construction Workers’ Visual Attention and Situation Awareness under Fall and Tripping Hazard Conditions: Using Mobile Eye Tracking. J. Constr. Eng. Manag. 2018, 144, 04018060. [Google Scholar] [CrossRef]
  67. Fu, H.; Xia, Z.; Tan, Y.; Guo, X. Influence of Cues on the Safety Hazard Recognition of Construction Workers during Safety Training: Evidence from an Eye-Tracking Experiment. J. Civ. Eng. Educ. 2024, 150, 04023009. [Google Scholar] [CrossRef]
  68. Paris, F.; Casanova, R.; Bergeonneau, M.L.; Mestre, D. Characterizing the expertise of aircraft maintenance technicians using eye-tracking. In Proceedings of the 2022 Symposium on Eye Tracking Research and Applications, Seattle, WA, USA, 8–11 June 2022; ETRA ’22. pp. 1–3. [Google Scholar] [CrossRef]
  69. Paris, F.; Casanova, R.; Bergeonneau, M.; Mestre, D. Analysis of the Use of the Maintenance Documentation Using Eye-Tracking: A Pilot Study. In Proceedings of the 1st International Conference on Cognitive Aircraft Systems—ICCAS, Toulouse, France, 1–2 June 2022; Volume 1, pp. 57–60. [Google Scholar] [CrossRef]
  70. Asadi, H.; Yu, D.; Mott, J.H. Risk factors for musculoskeletal injuries in airline maintenance, repair & overhaul. Int. J. Ind. Ergon. 2019, 70, 107–115. [Google Scholar]
  71. Sugiharto, F.M. The Relationship between Mental Workload and Occupational Stress among Aircraft Maintenance Officers at PT X. Indones. J. Occup. Saf. Health 2019, 8, 233. [Google Scholar] [CrossRef]
  72. Yazgan, E.; Ozkan, N.F.; Ulutas, B.H. A questionnaire-based musculoskeletal disorder assessment for aircraft maintenance technicians. Aircr. Eng. Aerosp. Technol. 2022, 94, 240–247. [Google Scholar] [CrossRef]
  73. Duchowski, A.T. A breadth-first survey of eye-tracking applications. Behav. Res. Methods Instrum. Comput. 2002, 34, 455–470. [Google Scholar] [CrossRef] [PubMed]
  74. Hart, S.G.; Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload; Advances in Psychology; Hancock, P.A., Meshkati, N., Eds.; North-Holland: Amsterdam, The Netherlands, 1988; Volume 52, pp. 139–183. [Google Scholar] [CrossRef]
  75. Ganier, F. Évaluer l’efficacité des documents techniques procéduraux: Un panorama des méthodes. Le Trav. Hum. 2002, 65, 1–27. [Google Scholar] [CrossRef]
  76. Hegarty, M.; Carpenter, P.A.; Just, M.A. Diagrams in the comprehension of scientific texts. In Handbook of Reading Research; Lawrence Erlbaum Associates, Inc.: Hillsdale, NJ, USA, 1991; Volume 2, pp. 641–668. [Google Scholar]
  77. Narazaki, K.; Oleynikov, D.; Stergiou, N. Objective Assessment of Proficiency with Bimanual Inanimate Tasks in Robotic Laparoscopy. J. Laparoendosc. Adv. Surg. Tech. 2007, 17, 47–52. [Google Scholar] [CrossRef] [PubMed]
  78. Suebnukarn, S.; Phatthanasathiankul, N.; Sombatweroje, S.; Rhienmora, P.; Haddawy, P. Process and outcome measures of expert/novice performance on a haptic virtual reality system. J. Dent. 2009, 37, 658–665. [Google Scholar] [CrossRef]
  79. Nogueira, H.C.; Diniz, A.C.P.; Barbieri, D.F.; Padula, R.S.; Carregaro, R.L.; de Oliveira, A.B. Musculoskeletal disorders and psychosocial risk factors among workers of the aircraft maintenance industry. Work 2012, 41, 4801–4807. [Google Scholar] [CrossRef]
  80. Glover, J.A.; Harvey, A.L.; Corkill, A.J. Remembering Written Instructions: Tab a Goes into Slot C, or Does It? Br. J. Educ. Psychol. 1988, 58, 191–200. [Google Scholar] [CrossRef]
  81. Ganier, F.; Hoareau, C.; Devillers, F. Évaluation des performances et de la charge de travail induits par l’apprentissage de procédures de maintenance en environnement virtuel. Le Trav. Hum. 2013, 76, 335–363. [Google Scholar] [CrossRef]
  82. Ganier, F. Factors Affecting the Processing of Procedural Instructions: Implications for Document Design. IEEE Trans. Prof. Commun. 2004, 47, 15–26. [Google Scholar] [CrossRef]
  83. Reason, J.; Maddox, M. Human Error. In Human Factors Guide for Aviation Maintenance; Federal Aviation Association (FAA), Ed.; Galaxy Scientific Corporation: Washington, DC, USA, 1998; p. 48. [Google Scholar]
  84. Hayes, J.R.; Simon, H.A. Understanding written problem instructions. Knowl. Cogn. 1974, 1, 167–200. [Google Scholar]
  85. Newell, A.; Simon, H.A. Human Problem Solving; Prentice-Hall: Englewood Cliffs, NJ, USA, 1972; Volume 104. [Google Scholar]
  86. Brysbaert, M. How many words do we read per minute? A review and meta-analysis of reading rate. J. Mem. Lang. 2019, 109, 30. [Google Scholar] [CrossRef]
  87. Rayner, K.; Schotter, E.R.; Masson, M.E.J.; Potter, M.C.; Treiman, R. So Much to Read, So Little Time: How Do We Read, and Can Speed Reading Help? Psychol. Sci. Public Interest 2016, 17, 4–34. [Google Scholar] [CrossRef]
  88. Cellier, J.M.; Terrier, P. Le rôle de la mise en forme matérielle dans le traitement cognitif de consignes. Langages 2001, 35, 79–91. [Google Scholar] [CrossRef]
  89. Orlandi, L.; Brooks, B. Measuring mental workload and physiological reactions in marine pilots: Building bridges towards redlines of performance. Appl. Ergon. 2018, 69, 74–92. [Google Scholar] [CrossRef]
Applsci 14 01251 g001
Figure 1. Aircraft maintenance activity analysis setup.
Figure 1. Aircraft maintenance activity analysis setup.
Applsci 14 01251 g001
Applsci 14 01251 g002
Figure 2. Key parts for task milestones definition: pin (a), wheel (b), brake unit (c), and bolt (d). The yellow arrows indicate the location of the part in the image.
Figure 2. Key parts for task milestones definition: pin (a), wheel (b), brake unit (c), and bolt (d). The yellow arrows indicate the location of the part in the image.
Applsci 14 01251 g002
Applsci 14 01251 g003
Figure 3. Decomposition of the maintenance operation and evolution of document consultation for one participant.
Figure 3. Decomposition of the maintenance operation and evolution of document consultation for one participant.
Applsci 14 01251 g003
Applsci 14 01251 g004
Figure 4. Temporal qualification of document consultation of the fixation. On the left, an illustration of the AMT wearing the mobile eye tracker with the gaze fixation on the maintenance documentation. On the right, the temporal qualification of the document consultation of the fixation is presented, where the first point represents the qualification of the fixation within the documentation and the second point indicates its temporal qualification in the decomposition presented in Figure 3.
Figure 4. Temporal qualification of document consultation of the fixation. On the left, an illustration of the AMT wearing the mobile eye tracker with the gaze fixation on the maintenance documentation. On the right, the temporal qualification of the document consultation of the fixation is presented, where the first point represents the qualification of the fixation within the documentation and the second point indicates its temporal qualification in the decomposition presented in Figure 3.
Applsci 14 01251 g004
Applsci 14 01251 g005
Figure 5. Boxplots of the percentage of time spent consulting documentation per step and expertise group.
Figure 5. Boxplots of the percentage of time spent consulting documentation per step and expertise group.
Applsci 14 01251 g005
Applsci 14 01251 g006
Figure 6. Box plots comparing per step and per expertise group (A) the average consultation time, and (B) the number of consultations.
Figure 6. Box plots comparing per step and per expertise group (A) the average consultation time, and (B) the number of consultations.
Applsci 14 01251 g006
Table 1. Dependent variables and their definitions.
Table 1. Dependent variables and their definitions.
VariableUnitsDefinition
Absolute durationminDifference between the timestamps of the milestones bordering the time period studied.
Document consultation durationminSum of gaze fixation duration performed within the AMM during the time period studied
Execution durationminDifference between document consultation duration and absolute duration
Percentage of time spent consulting the documentation%Ratio between the documentation consultation duration and the absolute duration
Number of consultations∅Number of consultations in the time period studied.
Average consultation durationsAverage duration of consultations over the time period studied.
NASA-TLX scores∅The weighted scores on the six dimensions of the NASA-TLX.
Table 2. Mean and interquartile range (IQR) values for each NASA TLX dimension (9 experts and 12 novices).
Table 2. Mean and interquartile range (IQR) values for each NASA TLX dimension (9 experts and 12 novices).
Mental
Demands		Physical
Demands		Temporal
Demands		PerformanceEffortFrustration
MeanIQRMeanIQRMeanIQRMeanIQRMeanIQRMeanIQR
Weighted NASA
TLX Score		Experts170.6175.0138.985.088.980.050.650.059.450.029.45
Novices234.2197.548.347.543.336.25126.230.0163.3191.328.87.5
Table 3. Mean and interquartile range (IQR) values for each of the dependent variables grouped by groups and phases. Each column represents a phase and is divided into two sections, showing the median and interquartile ranges. Similarly, each row represents a dependent variable and is divided into two sections, showing the values for each expertise group (8 experts and 9 novices).
Table 3. Mean and interquartile range (IQR) values for each of the dependent variables grouped by groups and phases. Each column represents a phase and is divided into two sections, showing the median and interquartile ranges. Similarly, each row represents a dependent variable and is divided into two sections, showing the values for each expertise group (8 experts and 9 novices).
RemovalInspectionInstallationTotal
MeanIQRMeanIQRMeanIQRMeanIQR
Absolute Duration (min)Expert40.69.911.216.255.120.910733.2
Novice78.610.920.99.990.421.7189.935.9
Execution Duration (min)Expert35.07.29.813.450.816.795.724.9
Novice53.916.617.812.277.419.5149.118.7
Document consultation duration (min)Expert5.64.01.52.74.33.811.59.7
Novice24.78.63.12.912.93.240.715.7
Percentage of time spent on documentation (%)Expert14.17.98.812.67.63.910.65.0
Novice31.413.020.113.015.16.922.08.4
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Paris, F.; Casanova, R.; Bergeonneau, M.-L.; Mestre, D. Differences between Experts and Novices in the Use of Aircraft Maintenance Documentation: Evidence from Eye Tracking. Appl. Sci. 2024, 14, 1251. https://doi.org/10.3390/app14031251

AMA Style

Paris F, Casanova R, Bergeonneau M-L, Mestre D. Differences between Experts and Novices in the Use of Aircraft Maintenance Documentation: Evidence from Eye Tracking. Applied Sciences. 2024; 14(3):1251. https://doi.org/10.3390/app14031251

Chicago/Turabian Style

Paris, Florence, Remy Casanova, Marie-Line Bergeonneau, and Daniel Mestre. 2024. "Differences between Experts and Novices in the Use of Aircraft Maintenance Documentation: Evidence from Eye Tracking" Applied Sciences 14, no. 3: 1251. https://doi.org/10.3390/app14031251

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop