Improving Pilot Situational Awareness Using a Gaze-Based Assisted Adaptive Interface ^†

Abstract

Modern flight decks incorporate advanced automation and real-time data systems to improve safety and efficiency. However, maintaining situational awareness under a high workload remains a critical challenge, particularly when attentional resources are stretched. This study investigates how adaptive interfaces, using real-time eye-tracking data, can help pilots maintain their concentration and stay aware of all indications on large-area displays. To this end, we developed a gaze-responsive display that monitors whether the pilot has visually focused on objects displayed on a moving map. When the system detects that the pilot has not noticed an object on the map, it automatically adjusts the interface, highlighting the missed object, thus helping to prevent critical information from being overlooked. The interface was evaluated in a controlled simulation study with 34 participants. By capturing pilots’ gaze behaviour, the system reveals how attention shifts under workload and how adaptive visual cues can complement natural scanning patterns. Participant feedback indicated that adaptive behaviour provided supportive guidance without imposing additional cognitive load. Overall, the study highlights the potential of adaptive gaze-based interfaces to enhance attention strategies and contribute to more resilient situational awareness in aviation.

1. Introduction

Current flight decks are becoming increasingly advanced, featuring high levels of automation, networked sensor suites, and real-time data-driven systems designed to optimize operational effectiveness, reduce pilot workload, and enhance flight safety [1]. Avionics today provide pilots with high-resolution, second-by-second information on navigation, weather, aircraft systems, and traffic management displayed in integrated digital displays. Although these technologies have improved aircraft reliability and operational accuracy, they have also introduced new levels of complexity in human-machine interaction [2].

Human factor studies in aviation have shown that maintaining situational awareness in high demand or unexpected circumstances is particularly challenging. Pilots are increasingly exposed to absorbing large amounts of information, handling multiple tasks, and making rapid decisions within a time limit [3]. When cognitive demands exceed the mental resources available, the probability of overlooking vital signals or making suboptimal choices increases, most notably during complex or stressful phases of flight, such as descent, approach, or emergency manoeuvrers [4].

The development of cockpit technology also raises issues of pilot overreliance on automated systems. As the autonomy and forecasting abilities of systems improve, there is a greater chance of automation complacency and decreased vigilance. Cognitive workload and neuroergonomics research have shown that experienced pilots are also prone to attentional narrowing, misallocation of focus, and brief failures in monitoring when faced with workload [5]. Such limitations mean that simply presenting more information for pilots to see does not necessarily lead to enhanced awareness or decision-making.

Eye-tracking technology yields precise measurements of visual attention, identifying the spatial focus of the gaze as well as its temporal characteristics, while the ongoing analysis of gaze behaviour allows for the estimation of a subject’s cognitive state, offering indicators of workload, attentional shifts, and potential loss of situational awareness [6]. This information enables a deeper understanding of what the pilot is actively attending to, as well as what critical elements may be overlooked.

Leveraging such data, adaptive interfaces can dynamically respond in real time, emphasizing essential information or downplaying less pertinent details as needed [7]. By guiding user attention more effectively, these systems help minimize the risk of missed visual cues and support improved situational awareness. This adaptability is particularly crucial in high-demand domains like military aviation, where split-second decisions and awareness of visual details are vital.

In response to such problems, we introduced a gaze-adaptive mapping interface designed to support pilot situational awareness through real-time eye-tracking data. The proposed adaptive display responds to the pilot’s gaze behaviour by identifying whether specific objects have been visually acknowledged. As objects move across the display, the system monitors whether the pilot fixates on them before they reach a predefined central threshold. If an object has not been viewed by that point, a subtle white outline appears around its symbol, serving as a gentle, non-intrusive cue that the item may require attention. Once the pilot looks at the object, the indicator is immediately removed. Conversely, if the object is observed early, no cue is triggered. This approach offers supportive guidance without imposing additional cognitive burden, allowing the pilot to maintain primary control of the aircraft while still benefiting from adaptive assistance.

2. System Design

The adaptive interface was designed to support the detection of dynamic visual stimuli during a dual-task flight scenario while remaining unobtrusive and compatible with natural scanning behaviour. The display presents a continuously scrolling map filled with blue and red moving objects, whose movement is synchronized with the simulated aircraft, as shown in Figure 1. This creates a dynamic visual environment in which pilots must maintain constant situational awareness while simultaneously managing flight controls. The interface was developed in Python 3.10 using the PyQt6 framework, which provided the necessary flexibility for real-time performance, event handling, and precise control of graphical elements.

Real-time gaze tracking and processing were implemented using the Tobii Research Software Development Kit (SDK) 2.1, which enabled continuous monitoring of the user’s gaze position throughout the task. Gaze data were captured by a Tobii Pro Spectrum eye tracker, sampling at 300 Hz. This sampling rate is suitable for detecting brief fixations and rapid saccadic movements that are typical of tracking tasks. The eye-tracking system monitored whether participants visually fixated on specific objects by applying a dwell-time threshold of 0.1 s [8]. This threshold, commonly used in eye tracking studies, balances sensitivity with the need to avoid false detections resulting from momentary shifts in gaze. The fixation detection process was performed in real time, allowing the system to continuously evaluate whether each object entering the central area of the map had received visual attention.

The adaptive behaviour of the developed interface focused on detecting objects that the pilot was not focused on. When an object reached the centre of the large area display, which was chosen for its visual and functional significance, the system checked whether a lock had already been recorded for that particular item. If not, the interface generated a distinctive visual signal: a white halo around the objects, as shown in Figure 2. This halo was deliberately distinctive so as to guide attention without being distracting or competing with other cockpit information. The signal remained only until the eye tracker detected the first stabilization on the objects, at which point it automatically disappeared. This ensured that the system only intervened when necessary and remained passive when the participant’s natural scanning behaviour was sufficient.

3. Methodology

The study was designed to evaluate the effectiveness of the adaptive gaze-responsive interface during a normal test flight, where the pilot is not under high pressure. A controlled laboratory experiment was conducted with a within-subjects design, so that each participant completed both the adaptive and non-adaptive interface conditions. The methodology prioritized ecological relevance incorporating a flight simulation environment with a dynamic map object detection task, allowing observation of how the adaptive system supported attention in multitasking conditions.

3.1. Setup

For the experiment, we used a dual-screen workstation configuration designed to mimic the vertical layout commonly found in aircraft cockpits, as shown in Figure 3. The primary monitor displayed the Microsoft Flight Simulator 2020 (version 1.2.7.0), Microsoft Corporation, Redmond, WA, USA. environment, which participants controlled using standard flight equipment consisting of a joystick. Directly below, the secondary monitor was positioned to replicate the cockpit instrument layout, as shown in Figure 3. This layout allowed participants to transition naturally from monitoring flight parameters on the main screen to visual detection displayed on the secondary screen.

Eye movement data was recorded using the Tobii Pro Spectrum eye tracker (Tobii AB, Stockholm, Sweden) operating at 300 Hz. The device was placed directly below the secondary screen and calibrated individually for each participant using a standard calibration procedure, with a fixed viewing distance of approximately 65 cm. This placement ensured accurate measurement of gaze behaviour both during map viewing and during the task.

3.2. Scenario

The experimental scenario consisted of two flight sessions, each lasting five minutes. During both sessions, participants were required to complete the “First Solo Flight” mission of Microsoft Flight Simulator. Throughout the flights, participants were responsible for maintaining stable control of the aircraft by monitoring key cockpit indicators such as altitude, airspeed, and heading. These flight parameters needed to be kept within acceptable ranges, requiring ongoing attention to the primary display.

In addition to maintaining the aircraft’s trajectory, participants were also required to monitor a secondary map display, where a visual detection task took place. Various objects appeared on the map throughout each trial, and participants were tasked with recognising and identifying them as quickly and accurately as possible. The objects were either blue or red, and their symbols were derived from the NATO standard symbol set, providing operationally relevant visual cues familiar to many surveillance and aviation contexts. Objects appeared in a random sequence so that participants would not anticipate their timing or location.

Each participant completed the scenario twice: once with the adaptive version of the map interface and once with the non-adaptive version. The order of conditions was counterbalanced to minimise sequence effects. In the adaptive condition, the interface responded to the participant’s gaze behaviour to assist in the detection process, while in the non-adaptive condition, the map remained static and offered no support. This structure ensured that each participant interacted with both systems under equivalent flight and detection demands.

3.3. Experiment Protocol

An announcement was first distributed within the university inviting volunteers to participate in an experiment involving an eye tracker and Microsoft Flight Simulator. Individuals who wished to take part registered for a session through Google Calendar. Upon arriving at the laboratory, participants were asked to read and complete an informed consent form. They were then instructed to fill in a pre-experiment questionnaire that collected basic demographic information.

Following the administrative procedures, participants were given time to familiarise themselves with both the flight controls and the flight simulator environment. This training period emphasised the importance of maintaining stable flight while simultaneously scanning the secondary display for emerging objects. After completing the familiarisation phase, the experimental tasks were explained in detail to ensure that each participant clearly understood the required actions during the trials.

Participants then completed an eye-tracker calibration procedure. Once calibration was successful, they proceeded to perform the two experimental scenarios described earlier, completing both the adaptive and the non-adaptive interface conditions.

After finishing the flight sessions, participants took part in a semi-structured interview. During this interview, they reflected on the usefulness of the adaptive cues, their perceived workload, the realism of the experience, and any instances in which the adaptive behaviour influenced their visual scanning strategy.

4. Results

The experiment involved 34 participants (that were self-identified as 26 male, 7 female, 1 other; mean age = 21.7 years, range 18–29). Most participants were undergraduate students, and only a small percentage had any previous experience with flight simulators. Around one-third had interacted with eye-tracking technologies before, typically through prior research participation.

4.1. System Effectiveness

To evaluate the technical consistency of the adaptive highlighting mechanism, system effectiveness was assessed using two confusion matrices. These represent how accurately the system responded to whether participants saw or missed visual objects.

Table 1. Initial Adaptation Condition.

System/Participant	Did Not See the Object	Saw the Object	Outcome
Adapted	402	278	TP = 402, FP = 278
Did not adapt	0	40	FN = 0, TN = 40

describes how the system behaved when deciding to trigger adaptation (i.e., highlight a missed object), while

Table 2. Adaptation-Stopping Condition.

System/Participant	Did Not See the Object	Saw the Object	Outcome
Adaptation didn’t stop	468	219	TP = 468, FP = 219
Adaptation stop	0	33	FN = 0, TN = 33

represents its behaviour when determining whether to stop adaptation once the object was seen.

These tables show that the system always adapted when necessary, without ever failing to flag overlooked objects (no false negatives). Although the number of false positives was high, this redundancy ensured that participants did not overlook visual cues. It is important to note that participants maintained similar overall performance despite the false positives, suggesting that the system’s feedback did not distract them. Instead, it likely reinforced users’ confidence, as they trusted the system to help them whenever an object was overlooked.

4.2. Comparison Between Adaptive and Non-Adaptive Conditions

We first compared the total number of objects detected in the two main conditions. This comparison provides a direct measure of how adaptive map affected detection results.

Frequency analysis showed that half of the participants (50%) detected more objects using the adaptive system, 38.9% detected more objects without it, and 11.1% saw the same number in both conditions. Although the overall results were similar between the groups, the presence of adaptive support benefited the majority of the participants, helping them remain attentive and confident that any items they missed would be flagged by the system.

We then examined how participants located objects on the map. More specifically, we analysed the number of objects they located in the first half of the screen and in the second half. This approach was chosen because, in the adaptive interface, objects that had not been located by the pilot were highlighted by the developed system when the object passed the middle of the screen. In this approach, it was investigated whether the adaptive map contributed to maintaining detection performance as the session progressed.

The mean number of detected objects per condition is shown in

Table 3. 2 (Condition: Non-Adaptive, Adaptive) × 2 (Screen Half: First, Second) ANOVA.

Condition	First Half Mean (SD)	Second Half Mean (SD)
Non-Adaptive	9.33 (5.21)	5.69 (2.64)
Adaptive	8.83 (5.34)	7.00 (3.67)

below.

The results showed that participants generally detected more objects in the first half of the screen in both conditions. However, adaptive map improved detection in the second half, where participants typically experienced difficulties. These results indicate that the adaptive map helped maintain situational awareness over time.

To further verify these findings, we performed separate McNemar tests for both conditions, examining whether the number of objects observed changed significantly between the first and second halves, as shown in

Table 4. Non-Adaptive Condition.

Non-Adaptive	Second Half Saw	Second Half Not Seen	Row Total
First Half Saw	101	235	336
First Half Not Seen	104	280	384
Column Total	205	515	720

below.

The results showed that the participants missed significantly more symbols in the second half of the screen compared to the first, as shown in

Table 5. Adaptive Condition.

Adaptive	Second Half Saw	Second Half Not Seen	Row Total
First Half Saw	121	197	318
First Half Not Seen	131	271	402
Column Total	252	468	720

below.

In contrast, under the adaptive condition, detection in the second half improved. Adaptive visualization reduced the decline, allowing participants to recover symbols that had not been previously observed. These results suggest that the adaptive map helped maintain engagement and prevented the typical decline in detection accuracy over time.

Although the system produced a number of false positives, the participants’ overall detection remained stable. This suggests that the extraneous cues did not distract users, but rather supported their visual tracking, contributing to better overall consistency.

4.3. Detection by Symbol Colour (Blue vs. Red)

We examined whether the colour of the symbols affected users’ ability to locate them. To evaluate the effect of colour on detection, we compared performance between blue and red symbols. This analysis also investigated whether adaptive highlighting benefited one colour more than the other.

The results showed that participants detected blue symbols equally well in both conditions, suggesting that there was no particular advantage to using the adaptive version. In contrast, detection of red symbols improved slightly when the adaptive map was active.

Although colour alone did not significantly affect overall performance, adaptive map appeared to provide a small benefit for red symbols, which are often more visually demanding. This suggests that the system’s symbols may be particularly useful for objects with less natural prominence.

4.4. User Feedback

Qualitative feedback from participants provided valuable insights into user experience. Most participants reported that the adaptive highlighting made the task easier and reduced their concern about missing important objects. The white-ring cues were described as subtle yet effective, providing reassurance without introducing distraction. Some participants noted that under high workload, the cues could occasionally be missed, and suggested adding dynamic brightness or optional sound indicators to enhance visibility.

Overall, users described feeling more comfortable and confident when using the adaptive version. They expressed that knowing the system would highlight any missed object allowed them to focus more effectively on the main flying task. The combination of strong reliability, minimal distraction, and user trust highlights the potential of adaptive attention-support systems in future cockpit environments.

5. Conclusions and Future Work

The statistical analyses demonstrate that the adaptive eye-tracking system effectively enhanced user performance, particularly in the second half of the task where attentional fatigue typically occurs. The frequency analysis showed that half of the participants saw more objects with the adaptive interface, while McNemar’s test confirmed a clear improvement in second-half detections under the adaptive condition. The repeated-measures ANOVA further supported this result, revealing a significant interaction between condition and screen position, indicating that the adaptive visualization successfully compensated for the decline in attention over time.

Despite a relatively high number of false positives, the system achieved perfect recall, never failing to highlight an unseen object. This combination of complete detection coverage and moderate precision suggests that users were fully supported without being overwhelmed. Importantly, user performance did not degrade due to the additional visual cues, confirming that the adaptive feedback did not distract but instead reinforced confidence and sustained situational awareness.

User feedback reinforced the statistical outcomes: participants reported feeling more comfortable and confident with the adaptive system, trusting that missed objects would be highlighted automatically. This psychological assurance, paired with measurable improvements in second-half detection, underscores the practical potential of adaptive gaze-based interfaces in high-demand aviation settings.

Future work will focus on enhancing the system’s precision by refining fixation-detection algorithms, reducing false positives, and modelling peripheral vision to better distinguish between seen and unseen objects. Additionally, integrating multimodal feedback, such as subtle auditory or haptic cues, could further improve detection consistency under high workload. Testing the enhanced version in realistic cockpit environments will provide essential validation for future pilot training and operational applications.