Effect of Cognitive Distractors on Neonatal Endotracheal Intubation Performance: Insights from a Dual-Task Simulator

Abstract

Neonatal endotracheal intubation (ETI) is a complex medical procedure that demands extensive training before practicing on real patients. Clinical studies indicate that the conventional training approach, typically conducted in idealized conditions with task trainers, has a low skill transferability rate compared to performance in the dynamic environments common in intensive care units (ICUs). According to cognitive load theory, novices encounter difficulties in multitasking scenarios, exhibiting performance declines due to competition among tasks for cognitive resources; experts, having achieved automaticity, have more cognitive resources to handle additional tasks present in high-stress environments and therefore exhibit less performance degradation. Current ETI skill assessment methods do not capture these differences in expertise. To bridge this gap, we develop an innovative dual-task mixed-reality (MR) simulator to evaluate the influence of cognitive distractors on ETI and substantiate effective performance measurement metrics. Results affirm that experts demonstrate superior proficiency in handling extraneous cognitive loads compared to novices. This has important implications for understanding how to measure novice performance in ETI settings. Taken together, the dual-task ETI training simulator and the associated automated skill evaluation metric system hold promise for enhancing training in neonatal ETI practice and ultimately leading to improved patient care outcomes.

1. Introduction

Neonatal endotracheal intubation (ETI) is a commonly employed medical intervention to establish an artificial airway for newborns. This procedure becomes imperative when infants face challenges in autonomous respiration or require mechanical ventilation assistance. Given its time-sensitive nature, the successful execution of this procedure mandates the precise placement of the endotracheal tube by the operator within a narrow window of 30 s [1]. However, pediatric residents maintain low intubation success rates (23–25%) throughout three years of residency, with an average of only three supervised clinical ETI opportunities [2]. This is well below the 26–75 procedures needed to achieve a consistent 90% success rate [3,4]. To foster skill proficiency before clinical exposure, conventional training methods have relied upon task trainers coupled with instructor feedback. Complementing these approaches, diverse simulators have been developed for ETI training [5,6], providing a secure environment for trainees to refine intubation skills free from the constraints of a neonatal intensive care unit (NICU). Nevertheless, the apparent inadequacy in the transfer of skills acquired through task trainer-based simulators to the actual NICU underscores shortcomings in current training paradigm [7]. Our hypothesis is that traditional training frameworks evaluate trainees’ procedural skills within idealized conditions, devoid of the environmental stressors inherent in medical procedures involving real patients. Novices can learn to perform manual skills relatively easily when no distractors are competing for cognitive resources. However, when required to perform two tasks simultaneously, performance often drops because the secondary task consumes the attentional resources that are so crucial for good performance at early skill levels [8]. This perspective contends that insufficient cognitive demands during training may result in an overestimation of trainees’ proficiency levels. This overestimation, in turn, may lead to the premature advancement of trainees, or their untimely cessation of training, well before they have garnered the requisite proficiency and experience for skill transfer.

Cognitive load theory explains how the human cognitive system processes and learns new information [9]. It points to a crucial difference between experts and novices in how they allocate attention resources. Experts exhibit the ability to adeptly manage multiple tasks simultaneously with minimal performance degradation, a characteristic referred to as “automaticity” [10]. The evaluation of automaticity, indicative of skilled performance, often involves the assessment of secondary tasks that consume attention resources. Secondary tasks, when used for performance assessment, have proven sensitive enough to discern performance variations among skilled individuals compared to traditional metrics [11,12,13,14]. The attainment of automaticity, as indicated by secondary task measures, typically demands more prolonged training periods compared to proficiency determined by single-task metrics.

These findings underscore the need for more sophisticated training methods and nuanced performance metrics, which encompass secondary tasks, to address the transfer of skills learned in simulated settings to real-world medical scenarios. To substantiate these assertions, we designed and implemented a mixed-reality ETI simulator equipped with a dual-task training module, followed by an empirical assessment of participants’ performance. Our study compares procedural skills using both single-task and dual-task metrics, providing insights into ETI skill automaticity in complex environments. Data analysis results confirm the efficacy of our dual-task simulator in discerning differences between expert and novice ETI performances, pointing to a direction for future training regimen design for ETI and other medical skill development.

The key contributions of this study are outlined as follows:

• Identify the limitations inherent in the current single-task ETI training paradigm, particularly its inadequacy in transferring skill from simulation to clinical settings.
• Develop the first dual-task mixed-reality (MR) simulator for neonatal endotracheal intubation. The simulator effectively captures expert-novice performance differences that traditional training paradigms fail to detect.
• Facilitate seamless transition between single-task and dual-task modes in future works using the MR simulator.
• Develop an automated skill evaluation strategy for trainee performance.
• Ground the efficacy of the multitask ETI training method in theoretical and empirical research from cognitive psychology, thereby paving the way for the evaluation of novel training regimens in future endeavors.

The subsequent sections of this paper are structured as follows: Section 2 provides a comprehensive review of existing literature pertaining to ETI training and the utilization of dual-task training in cognitive psychology study. Section 3 outlines the innovative design of a dual-task MR training ETI simulator, automated scoring system, and data collection process. In Section 4, we propose a validation methodology aiming at assessing the difference between experts and novices. Subsequently, Section 5 discusses data analysis results. Finally, Section 6 concludes our work and introduces future research.

2. Related Work

Medical simulators serve as invaluable tools for medical professionals, spanning from students to seasoned practitioners, facilitating the practice and refinement of techniques within a controlled environment. For instance, simulators for otolaryngology, bone surgery, and vascular reconstruction have been shown to significantly improve surgical outcomes [15,16,17]. Advanced technologies such as virtual reality (VR) and augmented reality (AR) have been integrated into simulator design to further enhance surgical training and performance [18,19,20,21]. These innovations contribute substantially to improving proficiency and confidence, laying a strong foundation for clinical practice and ultimately leading to better patient outcomes.

Infants admitted to the NICU are highly vulnerable to respiratory insufficiency and often require intubation and mechanical ventilation. Proficiency in ETI is a critical resuscitation skill for pediatric trainees and is essential for effective infant care [22]. Figure 1 illustrates the intubation procedure. First, insert the laryngoscope with the left hand in the oral cavity; second, press down the tongue and the epiglottis until the vocal cords are in sight; lastly, hold the left hand steadily and insert the intubation tube with the right hand. To ensure competency, trainees must undergo ETI training prior to clinical exposure [23,24]. Among the various training modalities, physical task trainer simulators are the most widely utilized for ETI instruction. However, current training protocols employing these simulators demonstrate limited skill transferability to real clinical settings [7]. Despite successfully completing task trainer-based training, practitioners frequently encounter challenges in achieving optimal performance during actual patient encounters. This discrepancy can be attributed to several factors. Firstly, the lack of comprehensive understanding of the oral cavity in task trainers limits the efficacy of training guidance and feedback, and the mechanical proficiency gained from blindly memorizing procedural movements within task trainer oral cavities may not translate effectively to real-life patient scenarios. Secondly, the haptic tactile feedback provided by task trainers often lacks fidelity compared to that encountered with real neonates [25]. Lastly, the training environment within a classroom setting differs significantly from the dynamic and distraction-laden ambiance of actual clinical settings in NICU [26].

To improve training outcomes, several computer-aided simulators have been developed to augment the task-trainer-based training experiences [6,27,28]. While VR ETI simulators promise more versatile and realistic training experiences, the low-fidelity versions remain unsuitable for comprehensive educational purposes [27,28,29,30]. Conversely, high-fidelity VR simulators, though promising, are relatively new, expensive to develop, and have not been widely adopted within pediatric training programs to investigate their skill transferability [5,31].

Recognizing the need for enhanced training methodologies, our group has developed both AR and VR simulators to elevate training experiences and outcomes. However, existing studies and simulators have yet to comprehensively address the environmental distraction factor impacting skill transferability. In this study, we bridge this gap by introducing a dual-task simulator designed to replicate the complexities of real-world clinical settings. The primary objective of this study is to validate the discriminative capacity of a dual-task simulator in delineating subtle differences between expert practitioners and novices, where conventional single-task simulators have proven inadequate. Moreover, we aim to establish robust metrics capable of quantifying these differentiating factors. These insights will pave the way for developing more effective training paradigms in medical education.

Dual-task training is a method used in cognitive psychology and neuroscience to improve cognitive functions by simultaneously engaging in two different tasks. The idea behind dual-task training is based on the concept of cognitive load theory, which suggests that cognitive tasks impose demands on our mental resources. According to this theory, when our mental resources are overloaded, our performance in these tasks may suffer [9,32]. Novices typically experience higher cognitive load when learning a new skill due to the inherent complexity of the task and the need to actively process and integrate new information. As individuals gain expertise and proficiency in a skill, the intrinsic cognitive load decreases as tasks become more automated and require less conscious effort. However, this reduction in intrinsic load may be offset by increases in extraneous load if the task environment becomes more complex or if instructional materials are poorly designed. Nonetheless, with continued practice and skill refinement, individuals can achieve a state of automaticity where tasks are performed with minimal cognitive effort [33].

The dual-task paradigm is the most widely used method to measure automaticity. It typically involves performing a primary task, which is usually the task that the individual wants to improve, along with a secondary task that is designed to place additional demands on cognitive resources. By training under conditions of increased cognitive load, the theory posits that individuals can improve their ability to manage multiple tasks simultaneously, leading to enhancements in cognitive performance and skill automaticity [34]. Therefore, the development of a dual-task simulator not only mirrors real-world clinical scenarios but also serves as a tool for discerning experts and novices, when single-task measurement fails, by measuring the secondary task performance and potentially can be integrated into new medical training regimens.

3. Study Design

We conducted the dual-task neonatal intubation study in collaboration with the Children’s National Hospital. The human subject research and data collection were approved by their Institutional Review Board.

3.1. Simulator Design

We designed the ETI dual-task simulator in alignment with best-practice guidelines for healthcare simulation [35], and iteratively refined the design based on user feedback from neonatologists. The ETI dual-task simulator provides automatic skill assessment and dual-task settings. It consists of four distinct modules: a primary task ETI module, a secondary audio-based number reading and response module, a skill analysis and visualization module, and an AR module designed to provide visual guidance during training. This study aims to differentiate expertise levels using a dual-task paradigm; the AR module is not employed during the data collection process as it is specifically utilized in training sessions.

The primary task ETI simulation module includes a Miller 1 blade laryngoscope manufactured by Teleflex in Wayne, PA, USA, an endotracheal tube, and a full-term Laerdal^® task trainer from Ellerslie Auckland, New Zealand. The CT-scanned virtual model of the task trainer is visualized by volume rendering. A virtual laryngoscope was modeled by 3D scanning both the Miller 1 blade and handle. Kinematic data for both the task trainer and the laryngoscope are tracked and recorded by an Ascension 3D Guidance trakStar^™ 6 degrees of freedom (DOF) tracking system with 3 electromagnetic (EM) sensors. It operates at a default measurement rate of 80 Hz, with a positional accuracy of 1.40 mm and angular accuracy of 0.5°. An EM tracking system comprises a transmitter that emits an electromagnetic field and sensors attached to body segments or instruments to detect the field, in our study, one EM is attached to the laryngoscope handle tip, one to the tube, and one to the task trainer head. These sensors capture field characteristics and relay the data to a central management unit, which processes the information to compute the position and orientation of the sensors, thereby determining the spatial configuration of the tracked body segments in three-dimensional space. The motion data are registered and mapped to their virtual counterparts in the simulator. Figure 2 shows the primary-task ETI module GUI, it can record a new trial or review existing trials. On the left pane, user can select the existing trials to review, the timer is used for time tracking with current frame being shown on the bottom; the main window shows the intubation view, where the user can see the anatomy information of the task trainer (the green and red bars are optional trajectory visual aids of the current and suggested motions); the bottom pane include a view for pitch or force change over time, an ISO threshold for controlling the transparency of the task trainer volume, and a head ratio to set the visible portions of the task trainer.

The AR module is developed using HoloLens^TM from Microsoft, which enables users to visualize a superimposed representation of the internal anatomy over a real task trainer. The system employs a distributed framework to facilitate data streaming between the computer and the HoloLens. The visual representation from the user’s perspective is illustrated in Figure 3, highlighted with a black circle.

The grading feedback visualization module displays classification results using a polar chart, supplemented by key performance feature values over time. The polar chart is constructed on a five-point scale, with each radial axis representing a distinct performance feature. The enclosed area, determined by the assigned grades for each feature, serves as an indicator of the trainee’s overall performance, where a larger enclosed area corresponds to a higher level of proficiency. An overview of this visualization is shown in Figure 4. This visual representation provides trainees with an intuitive feedback mechanism, offering clear insights into their performance and highlighting areas for improvement.

In intubation scenarios, environmental distractions are primarily auditory, as practitioners need process information communicated by other team members. To simulate this, our secondary task module incorporates a speech synthesizer and a footswitch to emulate auditory distractions and capture user responses [36]. Participants are instructed to tap the footswitch during the ETI procedure upon detecting a sequence that meets predefined criteria, referred to as a “hit event”. The secondary task incorporates two discernible levels of difficulty:

• 1-back: Upon detecting a number’s immediate repetition, participants are instructed to activate the footswitch, enabling the MR simulator to capture the corresponding timestamp for a hit event recording. For example, a user will tap the footswitch upon hearing the second occurrence of the number 3 within the sequence {...6, 2, 3, 3, 5,...}. The number and order of the sequence are randomized for each trial.
• 2-back: Upon detecting a number’s recurrence after a different one, participants are instructed to activate the footswitch for a hit event recording. For example, a user will hit the footswitch when hearing the second occurrence of the number 8 within the sequence {...2, 7, 8, 5, 8,...}. The number and order of the sequence are randomized for each trial.

We employ the term “0-back” to denote single-task setting, where subjects only perform primary ETI tasks.

Figure 5 shows the ETI dual-task simulator during the data capturing process.

3.2. Dataset Collection

ETI procedures were conducted under both single-task and dual-task conditions, involving 10 experts and 19 novices (average $29.8\pm 6.9$ years old). The expert group is drawn from neonatology and pediatric residents and fellows with on average 10.7 years of experience. The novice group is drawn from medical students and pediatric residents with on average 3.3 years of experience.

Each participant completed a total of 15 trials. The initial 10 trials alternated between 0-back and 1-back, with a starting trial of 0-back or 1-back (counterbalanced across participants). The remaining 5 trials were 2-backs. To reduce variance associated with participants differently prioritizing performance on the primary versus the secondary task, participants were instructed to emphasize performance on ETI while allocating attention to the secondary task as their cognitive resources permitted. These instructions, coupled with engagement in the secondary task as a nominal motivator, have been shown to be effective in past studies measuring medical simulator proficiency [12].

After purging the data of anomalies in cases when the EM lost track, we curated a dataset of 295 trials. The 3D motions of these trials were then replayed and evaluated by three expert neonatologists. They conducted an unbiased assessment of overall scores through blind review using a 5-point Likert scale, where 1 represents poor, and 5 represents excellent. We used these scoring results to train a deep learning model for automated ETI skill assessment [6].

4. Methods

We conducted a two-way group (expert vs. novice) × task (0-back vs. 1-back vs. 2-back) repeated measures analysis of variance (ANOVA) [37]. The dependent measures include (i) primary ETI task performance, and (ii) the task engagement ratio and performance of the secondary task. Presentation of all data is in the form of means with statistical significance established at $p<0.05$. The F-statistic (for group and task performance variances) and t-statistic (for group and task performance means) were used to investigate if there were statistically significant differences between expertise levels and task complexities.

4.1. Primary ETI Task Metrics

The ETI performance was evaluated with an overall score graded by experts on a 5-point Likert scale, and the scores of the individual features were recommended by expert neonatologists. These individual features include trial time, duration of glottis visibility, force applied against the gum, penetration depth into the tongue plane, pitch peaks, and yaw peaks as defined in

Table 1. Primary ETI task performance metrics, which are evaluated on a 5-point Likert scale.

Metrics	Descriptions
Overall score	The performance is evaluated by experts based on the entire procedure.
Trial time	Time used to complete an ETI procedure. The shorter the better.
Duration of glottis visibility	The duration of glottis being in the line of sight of the laryngoscope. The longer the better.
Force	The force exerted by the laryngoscope on the upper gum. The smaller the better.
Penetration depth	The penetration depth of the laryngoscope into the tongue plane. It has to be the right amount to view glottis, but not too deep to damage the tongue.
Pitch peaks	The pitch angle peaks of the laryngoscope to measure the smoothness of motion.
Yaw peaks	The yaw angle peaks of the laryngoscope to measure the smoothness of motion.

. Fast Fourier transformation was used to filter out high frequency data in pitch and yaw peaks. They were automatically graded on a 5-point Likert scale with a multi-task convolutional neural network classification algorithm [6].

4.2. Secondary Task Metrics

The performance measurement of the secondary task was conveyed by participant engagement ratio as in Equation (1) and footswitch hit accuracy in (2):

$$\begin{array}{c}\hfill Second\_task\_engage={\displaystyle \frac{Totaluserhits}{Systemgivenhits}}\end{array}$$

(1)

$$\begin{array}{c}\hfill accuracy={\displaystyle \frac{Correctuserhits}{Systemgivenhits}}\end{array}$$

(2)

The secondary task engagement ratio measure functions as an indicator of cognitive resource allocation, independent of response accuracy, whereas accuracy serves as a metric for assessing the proficiency of performance in the execution of the secondary task.

4.3. Statistic Metrics

The t-test was used to determine whether there exists a statistically significant difference in the means of two groups.

$$\begin{array}{c}\hfill t\left(df\right)={\displaystyle \frac{{\overline{x}}_1-{\overline{x}}_2}{\sqrt{s^2(\frac{1}{n_1}+\frac{1}{n_2})}}}\end{array}$$

(3)

where $df$ is the degree of freedom of the test, which is the number of samples for equal-sized groups, and weighted value of the sample sizes from the two unequal-sized groups. s is the standard error, $n_1$ and $n_2$ are the sample size of the two groups. If the corresponding p-value for this t-value is less than $0.05$, the null hypothesis that the means of the two groups are the same is rejected.

The ANOVA result was reported as an F-statistic, accompanied by its associated degrees of freedom and p-value. The F-value measures the ratio of between-group variance and within-group variance (4).

$$\begin{array}{c}\hfill F_{d{f}_1,d{f}_2}={\displaystyle \frac{Betweengroupsvariance}{Withingroupvariance}}\end{array}$$

(4)

where $d{f}_1$ and $d{f}_2$ are the degree of freedom of the numerator and the degree of freedom of the denominator. If the corresponding p-value for this F-value is less than $0.05$, then the null hypothesis that there is no significant effect of the factors on the dependent variables is rejected.

We conducted data analysis in the following aspects to see the performance difference between experts and novice, the impact of the dual-task on both groups, and the effect of repetition on performance. The null hypotheses for these tests are as follows:

• There is no performance difference between experts and novices for each task setting.
• There is no performance difference under different task settings for the same group (expert or novice).
• There is no difference between experts and novices on how the dual-task impacts their performance.
• There is no difference in ETI performance under intensive training.

5. Results and Discussion

The results of the data analysis are reported categorically in accordance with the respective hypotheses. Table cells that show statistical significance are colored in purple.

5.1. Expert vs. Novice

5.1.1. Primary Task Performance Comparison

The overall performance of the expert and novice groups under 0-back, 1-back, and 2-back conditions are plotted in Figure 6, and the corresponding t-test and F-test results are shown in

Table 2. Mean score for expert and novice group, t-value, F-value, and their corresponding p-value.

Test	0-Back	1-Back	2-Back	Overall
Expert score	$4.00$	$3.74$	$4.09$	$3.94$
Novice score	$3.03$	$2.85$	$4.05$	$3.29$
t-value	$t\left(98\right)=-6.35$	$t\left(98\right)=-4.02$	$t\left(93\right)=-0.2134$	$t\left(293\right)=-5.55$
F-value	$F_{1,99}$ = 40.32	$F_{1,99}=16.13$	$F_{1,94}=0.05$	$F_{1,294}=30.79$
p-value	$6.74\times {10}^{-9}$	$1.16\times {10}^{-4}$	$0.83$	$6.44\times {10}^{-8}$

. The results show that the overall performance is significantly different between experts and novices for 0-back (single task) and 1-back, whereas they are statistically the same for 2-back settings. All conditions considered, experts scored better than novices.

It is worth noting that novice group performance decreased first at 1-back, then increased at 2-back, whereas the expert group was less affected by task conditions. We will further analyze the within-group performance in later sections. For the current between-group analysis result, it is necessary to further examine why 1-back and 2-back affect experts and novices differently: Is it because there are differences in attention allocation? Or did the novice recalibrate and find the optimal path over trials? This can be determined by comparing the secondary task engagement rate and accuracy. The repetition effects will be discussed in later sections.

5.1.2. Secondary Task Performance Comparison

The comparison of secondary task engagement ratios and accuracy statistics between expert and novice groups is depicted in Figure 7 and summarized in

Table 3. Mean secondary task engagement ratio and accuracy for expert and novice group, and between-group difference by t-value, F-value, and their corresponding p-value.

Test	1-Back	2-Back	Overall
Expert engagement	$0.30$	$0.26$	$0.28$
Novice engagement	$0.27$	$0.20$	$0.24$
Engagement t-value	$t\left(98\right)=-1.24$	$t\left(93\right)=-2.78$	$t\left(193\right)=2.50$
Engagement F-value	$F_{1,99}=1.55$	$F_{1,94}=7.74$	$F_{1,194}=6.23$
Engagement p-value	$0.23$	$0.01$	$0.01$
Expert accuracy	$0.27$	$0.23$	$0.25$
Novice accuracy	$0.11$	$0.07$	$0.10$
Accuracy t-value	$t\left(98\right)=-8.06$	$t\left(93\right)=-9.63$	$t\left(193\right)=-12.01$
Accuracy F-value	$F_{1,99}=64.94$	$F_{1,94}=92.73$	$F_{1,194}=144.18$
Accuracy p-value	$1.90\times {10}^{-12}$	$1.24\times {10}^{-15}$	$3.61\times {10}^{-25}$

. The overall secondary task engagement ratio is low for both expert and novice groups in both dual-task conditions. Specifically, in the 1-back setting, both experts and novices allocate a similar level of attention, with novices exhibiting significantly lower accuracy compared to experts; the collective primary task mean score for both groups decreases. We will discuss the statistical significance of this effect in subsequent within-group analyses. In the 2-back setting, both groups experience a further reduction in secondary task engagement ratios and accuracy, with novice performance significantly lagging behind that of experts.

5.2. Dual-Task Interference Effects on Expert

One-way repeated measures ANOVA (dual-task setting as factor) is employed to assess the influence of the secondary task on the primary ETI task. A comprehensive comparison of both overall performance and individual features is illustrated in Figure 8 and

Table 4. The impact of secondary task on expert ETI performance measured by t-value, F-value, and their corresponding p-value.

Feature	0 vs. 1-Back $\mathit{t}\left(34\right)$ (p)	0 vs. 2-Back $\mathit{t}\left(34\right)$ (p)	1 vs. 2-Back $\mathit{t}\left(34\right)$ (p)	${\mathit{F}}_{2,68}$ (p)
Overall Score	$1.27\left(0.21\right)$	$-0.25\left(0.61\right)$	$-2.24\left(0.03\right)$	$2.08\left(0.13\right)$
Trial time	$-0.90\left(0.38\right)$	$1.26\left(0.21\right)$	$2.04\left(0.048\right)$	$2.38\left(0.10\right)$
Glottis view	$0.47\left(0.64\right)$	$0.50\left(0.62\right)$	$0.05\left(0.96\right)$	$0.16\left(0.85\right)$
Force	$-0.63\left(0.53\right)$	$0.25\left(0.80\right)$	$0.76\left(0.45\right)$	$0.37\left(0.69\right)$
Depth	$0.67\left(0.51\right)$	$0.45\left(0.66\right)$	$-0.13\left(0.90\right)$	$0.22\left(0.80\right)$
Pitch Peaks	$-0.88\left(0.39\right)$	$0.31\left(0.76\right)$	$1.03\left(0.31\right)$	$0.81\left(0.45\right)$
Yaw peaks	$-1.44\left(0.048\right)$	$0.69\left(0.51\right)$	$1.68\left(0.10\right)$	$2.21\left(0.12\right)$

. The colored table cells indicate features that yielded significantly different performance metrics between task settings. The F-value statistics reveal that the simultaneous execution of two tasks has an insignificant impact on both the overall performance and various features. Notably, the yaw peaks feature demonstrates a statistically significant difference between the 0-back and 1-back conditions ($t\left(34\right)=1.44$, $p=0.048$, 1 vs. 0-back t-value is a mirror of the 0- vs. 1-back t in Table 4), indicating an increase in mean yaw peaks during the 1-back dual-task condition. A noteworthy observation arises when examining the performance of experts: a significant increase is found in their overall score ($t\left(34\right)=2.24,p=0.03$) after switching from 1-back to 2-back. Furthermore, trial times exhibit a significant reduction ($t\left(34\right)=-2.04,p=0.048$, 1- vs. 2-back), suggesting that, initially, experts may display slight susceptibility to dual-task interference in the experiment. However, as they accumulate more trial experience and progressively allocate less attention to the secondary task, their skill proficiency improves.

5.3. Dual-Task Interference Effects on Novice

A comprehensive comparison of both overall performance and individual features of novices under different task conditions is illustrated in Figure 9 and

Table 5. The impact of secondary task on novice ETI performance measured by t-value, F-value, and their corresponding p-value.

Feature	0 vs. 1-Back $\mathit{t}\left(59\right)$ (p)	0 vs. 2-Back $\mathit{t}\left(59\right)$ (p)	1 vs. 2-Back $\mathit{t}\left(59\right)$ (p)	${\mathit{F}}_{2,118}$ (p)
Overall Score	$0.98\left(0.33\right)$	$-8.70(3.62\times {10}^{-12})$	$-7.95(6.48\times {10}^{-11})$	$45.30(2.52\times {10}^{-15})$
Trial time	$-1.28\left(0.04\right)$	$2.31\left(0.02\right)$	$-3.79(3.52\times {10}^{-4})$	$6.95\left(0.001\right)$
Glottis view	$1.66\left(0.03\right)$	$2.77\left(0.008\right)$	$-3.45\left(0.001\right)$	$7.01\left(0.001\right)$
Force	$-0.40\left(0.69\right)$	$3.55(7.69\times {10}^{-4})$	$3.81(3.30\times {10}^{-4})$	$10.28(7.65\times {10}^{-5})$
Depth	$0.45\left(0.66\right)$	$2.76\left(0.01\right)$	$2.57\left(0.01\right)$	$5.60\left(0.005\right)$
Pitch Peaks	$-0.82\left(0.42\right)$	$2.41\left(0.02\right)$	$3.28\left(0.02\right)$	$5.59\left(0.005\right)$
Yaw peaks	$-1.64\left(0.045\right)$	$1.67\left(0.10\right)$	$3.30\left(0.001\right)$	$5.49\left(0.005\right)$

. The tabulated data reveal that, during the initial phase of the experiment, novices exhibited susceptibility to the distractions posed by the alternating execution of 0-back and 1-back task settings. This was reflected in extended execution times ($t\left(59\right)=1.28,p=0.04$) for the ETI task and a reduction in the duration of glottis being visible ($t\left(59\right)=-1.66,p=0.03$), alongside an increase in yaw peaks ($t\left(59\right)=1.64,p=0.045$). However, as the subjects transitioned to the 2-back dual-task setting and accumulated practice, novices began to allocate reduced attention to the secondary task. This resulted in improvements in all key performance features, with the exception of yaw peaks, which remain the same level as the beginning of the training.

5.4. ETI Performance Variation over Trials

The preceding sections have substantiated the effectiveness of the dual-task simulator in capturing the differences between experts and novices, even when they exhibit equivalent levels of proficiency in the primary task. Meanwhile, over the course of 15 trials, participants demonstrated notable enhancements in their primary task performance, particularly evident among novice participants. In order to quantitatively investigate this, we conducted a one-way repeated measures ANOVA (time as factor). To mitigate the potential interference stemming from the relative heightened secondary task engagement observed in the 1-back dual-task condition, we deliberately omitted the 1-back and selected four performance measures from the trial sequence: 0-back trial 1, 0-back trial 5, 2-back trial 11, and 2-back trial 15.

5.4.1. Expert Performance Change over Trials

In Section 5.2, our data analysis reveals that experts do not exhibit significant degradation under dual-task conditions. Within this section, we investigate whether these experts’ performance benefited in any discernible way over the course of the testing session. While we did not observe statistically significant changes in individual performance metrics, as illustrated in

Table 6. Expert performance change over trials. The significance is measured by t-value, F-value, and their corresponding p-value.

Feature	Trial 5 vs. 1 $\mathit{t}\left(6\right)$ (p)	Trial 11 vs. 1 $\mathit{t}\left(6\right)$ (p)	Trial 15 vs. 1 $\mathit{t}\left(6\right)$ (p)	${\mathit{F}}_{3,18}$ (p)
Overall Score	$1.16\left(0.29\right)$	$0.55\left(0.60\right)$	$3.29\left(0.02\right)$	$2.57\left(0.09\right)$
Trial time	$-0.10\left(0.93\right)$	$-1.12\left(0.31\right)$	$-0.61\left(0.56\right)$	$0.37\left(0.77\right)$
Glottis view	$-0.75\left(0.48\right)$	$-0.39\left(0.71\right)$	$-1.15\left(0.29\right)$	$0.80\left(0.51\right)$
Force	$1.75\left(0.13\right)$	$0.69\left(0.51\right)$	$1.82\left(0.12\right)$	$1.39\left(0.28\right)$
Depth	$0.35\left(0.74\right)$	$-0.06\left(0.95\right)$	$0.26\left(0.80\right)$	$0.11\left(0.95\right)$
Pitch Peaks	$0.17\left(0.87\right)$	$0.09\left(0.93\right)$	$-0.60\left(0.57\right)$	$0.33\left(0.80\right)$
Yaw peaks	$-0.32\left(0.76\right)$	$-0.81\left(0.45\right)$	$-1.15\left(0.29\right)$	$0.75\left(0.54\right)$

, there is a cumulative effect wherein these metrics collectively demonstrate improvement over time, resulting in a significant increase in the overall score ($t\left(6\right)=3.29,p=0.02$) when compared to the initial stage of the training program. Figure 10 illustrates these individual performance feature changes over time, wherein the blue box chart represents the 0-back (single-task) condition from time points 1 to 5, the orange box chart corresponds to the 1-back condition from time points 1 to 5, and the blue box chart from time points 6 to 10 signifies the 2-back condition. The grouping of 0-back and 1-back conditions is necessitated by the distinct alternative dual-task patterns for each subject, which do not align with the time axis. The mean value curve is computed from the means of the 0-back and 2-back conditions. At the lower part of the overall score chart in Figure 10a, we include a plot of the secondary task engagement ratio to provide an estimate of the combined cognitive load. The secondary task engagement ratio has been augmented by 1.5 for clarity in visualization. We observe that the secondary task engagement ratio remains the same level over time.

5.4.2. Novice Performance Change over Trials

In Section 5.3, the data analysis result shows that novice performance is substantially influenced by dual-task conditions, with a significant decline at the beginning, which gradually improves in subsequent 2-back trials with significantly reduced secondary task engagement ratio. In this section, we study the impact of repetition on novice performance. As detailed in

Table 7. Novice performance change over trials. The significance is measured by t-value, F-value, and their corresponding p-value.

Feature	Trial 5 vs. 1 $\mathit{t}\left(12\right)$ (p)	Trial 11 vs. 1 $\mathit{t}\left(12\right)$ (p)	Trial 15 vs. 1 $\mathit{t}\left(12\right)$ (p)	${\mathit{F}}_{3,33}$ (p)
Overall Score	$2.92\left(0.01\right)$	$7.00(2.37\times {10}^{-5})$	$8.04(6.21\times {10}^{-6})$	$18.11(4.00\times {10}^{-7})$
Trial time	$-1.36\left(0.20\right)$	$-2.52\left(0.03\right)$	$-3.23\left(0.01\right)$	$3.91\left(0.02\right)$
Glottis view	$-2.99\left(0.10\right)$	$-2.87\left(0.02\right)$	$-2.56\left(0.03\right)$	$6.40\left(0.002\right)$
Force	$-0.16\left(0.87\right)$	$-0.66\left(0.52\right)$	$-3.54\left(0.005\right)$	$1.83\left(0.16\right)$
Depth	$-0.19\left(0.85\right)$	$-1.21\left(0.25\right)$	$-0.40\left(0.70\right)$	$0.87\left(0.47\right)$
Pitch Peaks	$-2.07\left(0.06\right)$	$-1.93\left(0.08\right)$	$-3.13\left(0.01\right)$	$4.12\left(0.01\right)$
Yaw peaks	$-1.30\left(0.22\right)$	$-1.22\left(0.25\right)$	$-3.19\left(0.01\right)$	$2.37\left(0.09\right)$

, our observations indicate that at the initial stage of their testing session, following five single-task practice trials, a cumulative enhancement across various individual performance features becomes evident, resulting in a significant increase in the overall score. As the number of practice trials progresses, by the 11th trial, a substantial reduction in the time required to complete a given procedure becomes evident, contributing to a significant overall score increase. By the conclusion of the testing session, all performance features exhibit noticeable improvement, except for the depth feature.

The mean value plot for the novice group in Figure 11 provides a clear illustration of their skill development over time. In line with Section 5.4.1, we maintain consistent color-coding and grouping strategies for different task conditions, and the secondary task engagement ratio is adjusted by an increment of 1 to enhance visualization clarity. We observe that the secondary task engagement ratio is lower in the 2-back setting compared to 1-back.

This in-depth data analysis reveals that while novice participants achieved high ETI procedure outcomes, their secondary task performance indicates they have not yet reached automaticity. According to cognitive load theory, this suggests a high cognitive load that limits available working memory resources for concurrent tasks. The findings highlight the limitations of traditional single-task assessment metrics, which may overlook the cognitive demands trainees face in real clinical settings. Incorporating multitask evaluation provides a more comprehensive measure of performance and readiness by capturing the cognitive load experienced during complex procedures.

5.5. Discussion

The data analysis results are in line with our initial hypotheses:

• Single-task performance measurements are insufficient to evaluate practitioner’s readiness for clinical practice.
• Individuals with different levels of expertise exhibit distinct responses when subjected to dual-task conditions.
• Experts exhibit greater resilience to the adverse effects of elevated cognitive load, as compared to novices.

With multiple factors affecting the ETI performance, it is of practical interest to study which factor plays a pivotal role in the overall performance, or if it is the combined effect of all factors. To address this, we employed a stepwise linear regression approach to iteratively derive an optimal regression model [38]. The resulting model is presented in Figure 12. Our analysis clearly demonstrates that the subject group (isExpt), the dual-task mode (NMode), the trial order (orderID), and secondary task engagement ratio (SecondTaskEngage) are independently significant factors influencing the final performance scores. Furthermore, the interactions of expertise level with dual-task mode and expertise level with secondary task engagement also contribute to the overall performance.

6. Conclusions

In this paper, we present, to the best of our knowledge, the first practical and efficient dual-task ETI simulator that offers an alternative medical skill training paradigm. The incorporation of a high extrinsic load secondary task during training not only mirrors the complexities of the real-world clinical environment but also facilitates the differentiation between learners and expedites the learning process. Statistical analysis results demonstrate the efficacy of dual-task performance metrics in assessing the automaticity of skills. As expertise increases, ETI can be executed more “automatically”, resulting in better outcomes even when attention is divided among multiple tasks.

Achieving true automaticity on a task as complex as ETI is understood to necessitate intensive practice over many training intervals. The subsequent phase of our research involves the in-depth examination of skill retention. Our ongoing studies focus on the comprehensive investigation of the long-term retention of acquired skills, facilitated by the systematic assessment of trainee performance at regular intervals throughout the participants’ residency. This approach allows us to track the durability and maintenance of these skills over time. We are currently engaged in the data preprocessing stage, and the results of our research will be reported in a future publication.