Differences between Experts and Novices in the Use of Aircraft Maintenance Documentation: Evidence from Eye Tracking

Round 1

Reviewer 1 Report

Maybe the "Aircraft maintenance" should be put into the title to make the title more

Too many keywords contained in the current version. 5 keywords are enough. The keywords of "Task load" and "Cognitive load" have the nearly same meaning and only one of them should be remained.

A seperate literature review secion or a seperate theoretical background section should be divided from the current redundant introduction section. Many recent published eye-tracking related papers like "https://doi.org/10.1061/JCEECD.EIENG-1882" have been neglected in the references.

The figure name of the figure 1 is too long. It is hard for the readers to get the key point.

It is really confusing to see such long figure name in the figure 3.

Are the authors sure about so many variable in the table 1 are all dependent variables. It is not normal to find so many dependent variables in a single technical paper.

The logical sequence of the paper is confusing. I could even not know which key points are the authors focusing on. The names of the sub-sections should be meaningful, not statistical phrases. The readers could know the main logical streamline by only read the subsection name in the results section.

Each sub-figures should have names, such as figure 6.

The discussion section is not thoughtprovoking enough. The section of the discussion should be consistent with the sections of the results section. As the discussion should be on the basis of the research results.

If the authors want to compare the expertise with the noise in different process stages, the results sould be reported by the order of the process stages. In a word, the logical sequence of the current version, especially in the results section, is really confusing. It makes the paper hard to read. 

Maybe the "Aircraft maintenance" should be put into the title to make the title more

Too many keywords contained in the current version. 5 keywords are enough. The keywords of "Task load" and "Cognitive load" have the nearly same meaning and only one of them should be remained.

A seperate literature review secion or a seperate theoretical background section should be divided from the current redundant introduction section. Many recent published eye-tracking related papers like "https://doi.org/10.1061/JCEECD.EIENG-1882" have been neglected in the references.

The figure name of the figure 1 is too long. It is hard for the readers to get the key point.

It is really confusing to see such long figure name in the figure 3.

Are the authors sure about so many variable in the table 1 are all dependent variables. It is not normal to find so many dependent variables in a single technical paper.

The logical sequence of the paper is confusing. I could even not know which key points are the authors focusing on. The names of the sub-sections should be meaningful, not statistical phrases. The readers could know the main logical streamline by only read the subsection name in the results section.

Each sub-figures should have names, such as figure 6.

The discussion section is not thoughtprovoking enough. The section of the discussion should be consistent with the sections of the results section. As the discussion should be on the basis of the research results.

If the authors want to compare the expertise with the noise in different process stages, the results sould be reported by the order of the process stages. In a word, the logical sequence of the current version, especially in the results section, is really confusing. It makes the paper hard to read. 

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Article:

Difference between experts and novices in the use of maintenance documentation: Evidence from eye-tracking.

Introduction should be separated from theory. Main topics of introduction could be maintenance and atomization of maintenance.  Possible to mention: There are clear parallels between maintenance and assembly.

Main topic of theory is information intake from documentation (operationalized by document consultation) and mental task load (operationalized by TLX). Expertise here is defined by better capability of information intake (less consultations, more per consultation, faster consultation etc.), physical capabilities are more or less neglected.     

Theoretically, the use of maintenance documentation is seen as an activity of looking for informations or instructions how to proceed in a maintenance process before more or less skilled workers (AMT divided in experts, novices) perform maintenance activities. Each maintenance process is a sequence of two classes of activities: mentally collecting information from documentation (procedural information intake, line 54) and physically executing steps of maintenance (task execution, line 55). Steps and Phases are sequences of smaller or larger sub-parts of maintenance. Further on, information intake is not only looking at the document, but is a kind of active searching or navigating for relevant information in a larger document, an attentional fixation of so called areas of interest, a decision and a preparation of execution. The article suggests that experts and novices use same AOIs, but remain more or less long in these areas. Perhaps there are different AOIs used (?).

The concept of maintenance expertise should be analysed in some more extent. Experts are not only elder persons, but are also more experienced persons with much more training.  They can handle much more complexity, without having more mental load. Thus, expertise also can imply that several or longer steps are taken together or that even no information beyond type of aircraft is needed. (On the other hand, advantages of expertise can be reduced when consultation is enforced by a specific rule or a time-consuming procedure). Some more attention should be directed to the relation of expertise definition and results of empirical investigations concerning conceptual constituents of expertise. If an expert works worse than a novice, he probably is no expert ( or he is ill).

Hypothesis 1 concerns time advantages of experts. Experts work faster in a defined, stable context. Interesting: Perhaps not in all steps, or in different amounts.  Hypothesis 2 claims that time spent on information intake differs with task progress, independent of worker status. This hypothesis needs some more theoretical foundation, especially concerning the interaction of work progress and status of worker.  Perhaps the interaction should be treated in an additional hypothesis. At the first glance it seems that time spent on information depends on the kind of maintenance (e.g. removal is faster than installation? Line 114). Novices could work as fast as experts when there is no choice and only one possible way of proceeding. Hypothesis 3 sounds a little bit poor, it should be upgraded a little bit (especially as there are some surprising results concerning different steps and there is a back and forth of information intake and performance). Theoretically, load is a little bit neglected.  Mental load itself is a rather controversal theme in cognitive ergonomics, the same is valid for a clear demarcation of mental and physical load.

Part 2 Materials and Methods contains a lot of information concerning expert and novice participants, used  variables and instruments, research procedure and statistical analysis.  Fig. 1 and 2 give some visual insight in the maintenance task, fig. 3 shows phases and steps of atomization of aircraft maintenance. The research projects is realized in a workshop context, not in a real working context. Could this be a motivational problem for experts? Please check “Of the 12 participants in the novice group, 73% were male” (line 145).

Description of maintenance, maintenance document and concrete use of document is remains rather unclear. How many pages contains the document, is there any order of pages, what is the relation of written texts and illustrations, what do subjects do at what place during all the time? Are subjects using specific tools? What exactly is to be done in the phase  of inspection? Why are there no steps of inspection? What is done during start before step 1 and finally after step 8? What kind of activities contains  “looking at the maintenance document” (line 175)? Are there timelines for different steps, as figure 3 might suggest? Why are experts excluded, when they show a lower gaze sample percentage than 60%? Is there any failure control?

Results  are presented concerning duration of maintenance, number of consultations and workload. Table 2 should be revised concerning data, it would be helpful to have a column concerning the number of participants in each group. In total, data show that experts need less time for maintenance, have a lower percentage of consultations and show less mental load.  This result is not really surprising. Perhaps some aspects should be delved more intensely. In this context, what is about  possibilities of generalization of the results. Recommendation to include a new figure like figure 3 with progress curves for experts and novices.   

Is there any information concerning a weighted NASA-TLX-values in the paper? Without this information it is difficult to understand values in Table 3. There is a better description of data and some more explanation needed.  

Discussion

Is atomization a process? Or is it a kind of given structure, which is described in the document?  “Novices atomize more than experts” (line 367): This implies that subjects give different structures to the maintenance process. However, in a structural view novices and experts are confronted with the same structure, however they need different amounts of information seeking in order to understand structure and to make correct procedural decisions. This is the basis of back-and forth (line 414) processes (movements?). What means consultation and documentation in line 414? There should be a better coordination of introduction and discussion concerning the term atomization. Experts are not to be seen as good readers, but they know where they find relevant information in larger documents. For them the document is in large parts less informative. The same is true for illustrations.

What is totally missing in this discussion are limitations of the study. These concern for example the number of participants, the structure of activities during 2-3 hours of work, unclear relation of inspection as a phase and a step, unclear description of maintenance and documentation use, and finally, missing reasons why eye tracking did not imply measures of pupil dilation and other physiological parameters were excluded. It seems possible that TLX is not the best instrument for this task in this context.  

Information intake seems to have the same meaning as acquisition, consultation, gathering. I do have my doubts. Document navigation or navigation between consultation and execution could be  interesting future projects. 

What is also missing are some more concrete proposals for industrial enterprises, how to better design and present information in documents. For experts there is probably too much, for novices perhaps too little information in the used document. Experts often prefer illustrations, novices more concrete prescription and pictures of working steps. Researchers like Wickens,  Rasmussen or Hollnagel worked rather intensely on topics like these.  Perhaps there is a need to prefer modern electronical media like AR-glasses in order to better adapt the amount and kind of information to individual needs. It could be an interesting research project to investigate how it becomes possible to change novices to experts more quickly via better designed instructions.      

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

All my suggestions have been tackled and the current version could meet the requirements of the journal. Therefore, I recommend to accpet the paper as it is.

All my suggestions have been tackled and the current version could meet the requirements of the journal. Therefore, I recommend to accpet the paper as it is.

Reviewer 2 Report

Authors  made a fine job. Thank you.