Using Eye Movement Data Visualization to Enhance Training of Air Traffic Controllers: A Dynamic Network Approach

Abstract

The Federal Aviation Administration (FAA) forecasted substantial increase in the US air traffic volume creating a high demand in Air Traffic Control Specialists (ATCSs). Training times and passing rates for ATCSs might be improved if expert ATCSs’ eye movement (EM) characteristics can be utilized to support effective training. However, effective EM visualization is difficult for a dynamic task (e.g., aircraft conflict detection and mitigation) that includes interrogating multi-element targets that are dynamically moving, appearing, disappearing, and overlapping within a display. To address the issues, a dynamic network-based approach is introduced that integrates adapted visualizations (i.e., time-frame networks and normalized dot/bar plots) with measures used in network science (i.e., indegree, closeness, and betweenness) to provide in-depth EM analysis. The proposed approach was applied in an aircraft conflict task using a high-fidelity simulator; employing the use of veteran ATCSs and pseudo pilots. Results show that, ATCSs’ visual attention to multi-element dynamic targets can be effectively interpreted and supported through multiple evidences obtained from the various visualization and associated measures. In addition, we discovered that fewer eye fixation numbers or shorter eye fixation durations on a target may not necessarily indicate the target is less important when analyzing the flow of visual attention within a network. The results show promise in cohesively analyzing and visualizing various eye movement characteristics to better support training.

1. Introduction

The Federal Aviation Administration (FAA) forecasted a 1.4% annual increase in the US air traffic volume; from currently 43.2 million aircraft to 60.3 million by 2040 (FAA, 2018). However, the currently available number of expert air traffic control specialists (ATCSs) might not be sufficient to handle the anticipated increase of air traffic volume. Additionally, the current training completion time of the air traffic controllers takes many years of intensive training (Hampton, 2016). Therefore, the FAA has been trying to find ways to efficiently train the FAA Academy candidates.

One of the critical tasks of ATCSs is to detect and mitigate possible aircraft conflicts (i.e. possible collisions) through visually scanning the radar screen. The ATCSs look for conflicting situations between aircraft pairs (or groups) to resolve it and guide them to their destination in a safe/timely manner. Thus, the ATCS’s task involves a significant amount of visual scanning of the radar display and, subsequently, cognitive processing of the observed information to take necessary actions. Eye-mind hypothesis (Just & Carpenter, 1976) showed there exists high correlation between the eye movement (EM) data and the cognitive process of an observer. Kang and Landry (2014) demonstrated that exposing novice controllers to the visual scanpath of the expert ATCSs improved their overall scanning efficiency by reducing their false positive cases of conflict detection among aircraft. On Similar lines, Rudi, Kiefer and Raubal (2018) demonstrated that visualization of EM data of pilot’s working in a cockpit might prove useful for flight instruction purposes. Therefore, if we could effectively analyze, visualize, and interpret experts’ eye movement characteristics, we might be able to use those findings to train the candidates or novices.

Effective analysis of ATCSs’ visual scanning process is challenging. The Radar display has, a large number of dynamic targets (i.e. aircraft on radar display) which have dynamic properties (e.g. radar representation of an aircraft can change their shape and position with time). Figure 1 represents the dynamic aspect of the radar representation of aircraft.

Visualization of ATCSs’ EM data includes two steps. The first step involves developing a time-ordered mapping between the eye fixations (EFs) and the aircraft on the display. The second step consists of characterization of the developed scanpath sequences. To develop the mapping function in case of moving and overlapping targets, Dynamic Areas of Interest (AOIs) can be created which are dynamic convex boundaries that fits the moving targets and considers the visual angle accuracy (Kang & Bass, 2014; Kang, Mandal, Crutchfield, Millan, & McClung, 2016; Papenmeier & Huff, 2010). However, as the number of targets increases, visualization of the scanpath sequences becomes challenging using the widely used visualization methods such as point based and AOI based methods (Blascheck, Kurzhals, Raschke, Burch, Weiskopf, & Ertl, 2014) (explained below in Background section), since these methods might create visual clutter when numerous targets are visualized. In addition, air traffic has a dynamic nature, meaning that an aircraft can dynamically move within the radar display for a certain amount of time, and/or two or more aircraft can overlap on one another. As a result, any aircraft’s relevance from the visual scanning point of view evolves with time. The prevalent visualization methods are unable to handle this dynamic aspect of the ATCS’s visual scanning process.

In addition, other existing pivotal researches (Blascheck, Raschke, & Ertl, 2013; Burch, Beck, Raschke, Blascheck, & Weiskopf, 2014; Goldberg & Kotval, 1999) focus more on EF numbers and EF durations or simpler forms of scanpaths (explained in Background section); however, for a dynamic task such as air traffic control task, how the multiple targets (e.g. aircraft) are observed as a network (using the various EM transition characteristics) can also be important, meaning that even if there were fewer number of EFs or less durations, a target can be considered important if it plays a crucial role in the EM flow among multiple targets or acts as a bridge between two disconnected groups of targets. Furthermore, if we only consider the number of EFs on the aircraft, it might produce an incorrect interpretation about the important aircraft. For example, consider a case when an aircraft has just entered the radar display. There is high chance that this new entrant aircraft might receive a substantial amount of EF duration, as the ATCSs might want to know about its destination, altitude, and other details; however, this new aircraft might not be important in terms of conflict resolution with the already existing set of aircraft on the display. As a result, the ATCSs will not fixate again on this aircraft, rendering it unimportant in terms of the overall scanning strategy. Zhang, Ren and Wu (2014) provided valuable findings using static networks in the air traffic control domain, but the dynamic aspects and the issues raised above were not addressed.

Therefore, we need an improved analysis framework which will help us (1) develop visualization methods which can represent the EM data with a large number of targets with less visual clutter, (2) find measures that can accommodate the dynamic aspects of the moving targets, and (3) integrate the visualizations and measures for effective analyses and interpretations.

In this paper, we provide several approaches to address the issues raised above. First is to adapt the dynamic network (DNet) approach (Burch et al., 2014) and modify its structural components for visualizing the EM data of the ATCSs. A DNet is a collection of time-ordered static networks. The DNet visualization enables easy handling of a large number of targets, thereby reducing visual clutter. Being a collection of several networks, the DNet can easily represent the evolution of a target’s importance over time and the dynamics of visual scanning characteristics. The adapted DNet is aligned with three “vertex importance measures” such as “indegree,” “closeness,” and “betweenness” (Freeman, 1978) to better determine important targets. Furthermore, two types of normalization procedures (i.e. percent normalization and distance normalization) are introduced that calculate the relative amount of visual attention given to a target in comparison to the maximum values obtained for a specific task. Finally, we adapted the dot plots and bar plots to either better represent the evolution of the important targets or compare the vertex importance measures among the participants. Note that we will replace the term “vertex” with “target” or “AOI” for easier understanding.

2. Background

2.1. Eye movement (EM) visualizations: Point based, area of interest (AOI) based, and hybrid

Blascheck et al. (2014) have categorized the various EM visualization methods into point based, AOI based, and hybrid visualizations. Summaries and issues are as follows.

Existing point based visualization methods, e.g. timeline visualization (Kurzhals, Heimerl, & Weiskopf, 2014), scanpath visualization (Goldberg & Helfman, 2010), attention maps (Kurzhals & Weiskopf, 2013), space-time cubes (Kurzhals & Weiskopf, 2013) represent the time-ordered horizontal and vertical coordinates of the EFs occurring on the display. These methods are effective on visualizing the exact EF locations to unravel important regions (in absence of predefined targets) when given static stimuli. However, due to the visual angle error of the eye trackers, it is challenging to map the EFs with small and dynamic multi-element targets making it difficult to apply the point based methods (Mandal, Kang, & Millan, 2016). In addition, our interest is in investigating which moving targets were focused upon rather than the physically fixed area within a display.

On the other hand, existing AOI based visualization methods allows EM analysis based on either pre-defined region or target on the display. The AOI based methods have been categorized into timeline and relational AOI visualizations (Blascheck et al., 2014). Timeline AOI visualizations such as parallel scanpath (Raschke, Chen, & Ertl, 2012), scarf plot (Kurzhals, Fisher, Burch, & Weiskopf, 2015), and AOI river plot (Burch, Kull, & Weiskopf., 2013) focus on developing effective methods to represent the AOIs that have been fixated upon at various time intervals. However, these methods are challenging to apply for long duration tasks having large number of targets (e.g. twenty or more targets) and frequent EF transitions between them (e.g. air traffic control task).

Relational AOI visualization methods are more appropriate to handle the issues raised above through visualizations using circular heat map transition diagram (Blascheck et al., 2013), transition matrix (Goldberg & Kotval, 1999) and network visualization (Burch et al., 2014). These methods visualize the aggregated EM data by showing the relationship that exists between he AOIS in terms of the EF transitions between them, unlike the timeline approaches. Relational AOI based approaches do not represent the physical location of the AOIs on the display. In detail, in circular heat map visualization (Blascheck et al., 2013), AOIs are represented as segments of a circular layout (using different colors and sizes) and the EF transitions are shown by directed arrows between the circular segments. Transition matrix visualization (Goldberg & Kotval, 1999) represents the EF transitions among the AOIs in a tabular fashion. The most appropriate approach to address the issues of a dynamic task is through the network visualization that shows the AOIs as vertices and the EF transitions between AOIs as the directed edges between the vertices of a network (Burch et al., 2014; Holmqvist, Holsanova, Barthelson, & Lundqvist, 2003; Mandal et al., 2016; Tory, Atkins, Kirkpatrick, Nicolaou, & Yang, 2005).

However, if we try to apply the relational AOI visualization methods, we often run into possible visual clutter issues if there are large number of targets and it can be difficult to represent all the EM characteristics using the existing network approach. The following subsections 2.2. and 2.3 provide summaries of the DNet mathematical framework and how various EM network characteristics can be integrated based on time intervals.

2.2. Mathematical framework of DNet

A DNet is as a sequence of static networks (also called networks), where each constituent network is associated with a time interval (Beck, Burch, Diehl, Weiskopf, 2014). If the total time duration of the collected EM data is divided into T time intervals, then a DNet representing such a data is written as DynN = {N₁, N₁, .., N_T}, where N_t is the network for time interval t, where t = 1,2, .., T.

A network N_t is written as N_t = (V_t, E_t, M_t), where, V_t is the set of vertices (AOIs for the present study), E_t is the set of edges (EF transitions for the present study) between the vertices, and M_t is the adjacency matrix which contains all edge weights (amount of EF transitions between AOI pairs).

The set of vertices is written as V_t = (v₁, v₂, …, v_mt), where m_t is the number of vertices for time interval t. A network can either have directed or undirected edges, although for EM visualization we only consider directed edges. The set E_t consists of ordered pairs of vertices (v_i, v_j) showing that there exists a directed edge from the vertex v_i towards vertex v_j Thus, E_t = {e_ij(t)|v_i, v_j ∈ V_t, i ≠ j}. Lastly, the adjacency matrix is written as $M_t={\left[w_{ij}\left(t\right)\right]}_{m_t\times m_t}$, where, w_ij(t) states of multiple vertices over time (Saraiya, Lee, & North, 2005; von Landesberger et al., 2011).

2.3. DNet for EM visualization

Beck et al. (2014) provided an exhaustive list of various DNet visualization approaches representing EM data. Depending on the representation of the time variable, various visualization approaches have been categorized into two groups: Animation, and timeline visualization. Animation visualization refers to representing a DNet as an animated sequence of networks. Timeline visualization refers to representing a DNet as a sequence of networks in a single image showing the complete sequence of interactions between the targets. In the present work, we have applied the node-link based timeline approach for representing the DNet, because this visualization helps to preserve the mental map and reduces the cognitive load of the observer (Beck, Burch, & Diehl, 2013).

As noted by various researchers (Archambault & Purchase, 2013; Ghani, Elmqvist, & Yi, 2012; Purchase, Hoggan, & Görg, 2006), preserving the mental map (i.e. the abstract structural information layout about a network’s elements that an analyst develops in their mind as they visually scan the visualization) helps in tracing the change in vertex properties and edge paths across different time intervals. Additionally, the timeline visualization, using the node-link approach, provides an intuitive and efficient framework for analyzing the change of states of multiple vertices over time (Saraiya, Lee, & North, 2005; von Landesberger et al., 2011).

However, the existing DNet approach only uses the number and duration of EFs on the AOI to measures the AOI importance. As noted in the introduction, these two raw measures may lead to misleading results in case of dynamic targets. Therefore, it is required to consider other target importance measures to address the highlighted issue.

The next section discusses the three target importance measures (from the network science domain) that can be adapted for analysing AOI importance for dynamic scenarios.

2.4. Target (or AOI) importance measures

The three most popular target importance measures in a given network are indegree, closeness, and betweenness (Freeman, 1978; Newman, 2004; Opsahl, Agneessens, & Skvoretz, 2010). Mandal et al. (2016) have shown some possibilities in applying the abovementioned measures to build a basic foundation for the proposed approach in this article. It is noted that we introduce the “time” element (“t”) within the three vertex importance measures to consider the dynamicity.

Indegree of a vertex is defined as the sum of all incoming weights to it from all other vertices in the network. For the present study, incoming weights can be interpreted as the incoming EF transitions to a given AOI. Thus, indegree for the j^th AOI is given as $I_j={\sum }_{k=1}^m{w}_{kj}$ (Newman, 2004), where, w_kj is the number of EF transitions from the k^th AOI to the j^th AOI, and m is the total number of AOIs on the display.

We should note that the indegree measure, shown above, is static in nature. As a result, we modified it to develop the dynamic analogous, where the indegree for an AOI is defined for each of the time interval considered in the DNet framework. Thus, the modified indegree measure for the j^th AOI for time interval t is calculated as:

$$I_j\left(t\right)={\sum }_{\begin{array}{c}k=1\\ k\ne j\end{array}}^{m_t}{w}_{kj}\left(t\right)$$

(i)

Where, w_kj(t) is the number of EF transitions coming for the k^th AOI to j^th AOI and m_t is the number of unique AOIs in the AOI fixation sequence for time interval t. Large indegree value suggests higher importance for an AOI, as it received large number of EFs. Thus, indegree can be interpreted as a measure of direct attention received by an AOI.

However, indegree measure only considers the local structure (direct EF transitions) around a vertex but ne glects the global structure of the network (Borgatti, 2005; Opsahl et al., 2010). This issue is addressed by the two measures named “closeness” and “betweenness.”

Note that the network science (or graph theory) includes the concept of “outdegree” however, the indegree values and outdegree values are always the same within the EM network since there is no possibility of an inward single EM transition dividing into two or more outward transitions.

Before we understand the two measures, we need to define the concept of distance between AOIs in a network visualizing EM data. In the present case, we define the distance from one AOI (e.g. ‘A’) to another AOI (e.g. ‘B’) as the inverse of the number of EF transitions from AOI A towards B. Thus, a large number of EF transitions between AOIs results in a smaller distance between them in terms of the visual scanning strategy.

The closeness of a vertex measures its distance from all other vertices in the network. Thus, higher closeness value for a given vertex means it is easier to access any part of the network from it. In the present study, higher accessibility of an AOI can be interpreted as a greater association (both direct and indirect) with other AOIs in the network. High closeness value suggests that the AOI lies in the central location in terms of the observer’s visual scanning strategy. The closeness for the j^th AOI is given as $C_j={\sum }_{k=1}^m(1/{d_{jk}}^{*})$ (Opsahl et al., 2010), where, ${d_{jk}}^{*}$ is the minimum distance from the j^th AOI to the k^th AOI (if multiple paths exist) and m is the total number of AOIs on the display. Like the previous indegree measure case, the static closeness measure is also modified to develop its dynamic analogous. Thus, the modified closeness measure for the j^th AOI for time interval t is defined as follows:

$$C_j\left(t\right)={\sum }_{\begin{array}{c}k=1\\ k\ne j\end{array}}^{m_t}\frac{1}{{d_{jk}}^{*}\left(t\right)}$$

(ii)

Where, ${d_{jk}}^{*}\left(t\right)$ is the minimum distance from the j^th AOI to the k^th AOI (if multiple paths exist) and m_t Is the number of unique AOIs in the AOI fixation sequence for time interval t.

In a dynamic scenario, there are instances where due to visual scanning strategy of the observer, an AOI despite receiving small amount of direct attention (low indegree measure) and being present in a non-central location (low closeness value) can still play a significant role by connecting (acting as a bridge between) two groups of AOIs. Such an AOI plays a crucial role in controlling the flow of attention among the other AOIs on the display, and this aspect is measured through the concept of betweenness explained below.

Betweenness for the j^th AOI is defined as $B_j={\sum }_{l=1}^m{\sum }_{k=1,j\ne l\ne k}^m\left({SP}_{kl}^j/{SP}_{kl}\right)$ (Opsahl et al., 2010), where SP_kl represents the total number of shortest paths (if multiple paths exist) from the k^th AOI to the l^th AOI, and ${SP}_{kl}^j$ represents the number of such shortest paths that pass through the j^th AOI. Thus, the modified betweenness measure for the j^th AOI for time interval t is defined as follows:

$$B_j\left(t\right)={\sum }_{k=1}^{m_t}{\sum }_{\begin{array}{c}l=1\\ j\ne l\ne k\end{array}}^{m_t}\frac{{SP}_{kl}^j\left(t\right)}{{SP}_{kl}\left(t\right)}$$

(iii)

Where, SP_kl(t) represents the total number of shortest paths (if multiple paths exist) from the k^th AOI to the l^th AOI, and ${SP}_{kl}^j\left(t\right)$ represents the number of such shortest paths which pass through the j^th AOI for the time interval t.

3. Proposed approach

Figure 2 represents the various steps in the proposed methodology for analyzing ATCS’s EM data.

3.1. STEP 1. Collect observer’s EM and targets’ location data

The input for the first step is the simulation experimental data. As output for this step, two types of data are obtained: (a) EM data, that consists of horizontal and vertical coordinates of the EF and its associated fixation duration, and (b) target location data, that consists of the pixel coordinates of the various targets on the display.

3.2. STEP 2. Divide the experiment time into several smaller time intervals

The input for step 2 is the time duration (in minutes or hours) for which the experiment has been conducted, and the output are several equal (unequal) sized time intervals which sum to the experimental time duration. For example, consider that the total experiment duration is divided into four equal time intervals (i.e. T = 4, see section 2.2). Note that the time intervals can be chosen based on the task characteristics or the researcher’s judgment (i.e. fixed or event-based time intervals).

3.3. STEP 3. Select first time interval

The inputs for this step are the various time intervals obtained in the previous step 2. The intervals are indexed and arranged in a time-order sequence. In this step, we start with the the first time interval to initiate developing the AOI fixation sequence.

3.4. STEP 4. Develop AOI fixation sequence (i.e. scanpath sequence) for the selected time interval

As input, step 4 receives three things, the time interval selected in step 3, and the EM and target location data for this time interval. As output from this step, we obtain the AOI fixation sequence for the time interval considered. The size of the AOI fixation sequence is directly proportional to the number of EFs in the time interval. The AOI fixation sequence is created for the selected time interval by adapting the approach suggested by Kang et al. (2016). Creating the AOI fixation sequence involves mapping the EFs with the AOIs. Only those EFs falling within any of the AOI boundaries are considered for AOI fixation sequence development, else they are ignored. AOIs are coded by assigning uppercase letters followed by lowercase alphabets (i.e. A, B, …, a, b). The developed AOI fixation sequence is the collapsed form of a raw AOI fixation sequence. In the collapsed form of the sequence, multiple consecutive fixations of the same AOI is collapsed to a single fixation case (e.g. AAA is collapsed into A). Thus, a raw AOI fixation sequence AABCC is collapsed to ABC. In addition, an overlapping AOI case is shown in parenthesis with individual constituent AOIs separated by a semi-colon. For example, if an EF falls within the overlapped region of AOIs A and B, it is represented as (A; B). After mapping all the EFs to the AOIs, we obtain a time-ordered sequence of AOIs, that shows which AOIs were fixated and in which order.

Table 1. Samples AOI fixation sequences having overlaps.

Time interval	AOI fixation sequence
1	ABABABEBAEAEACACAEA
2	EAEAEAEABABCBC(D; G)C(D;G)
3	C(D;G)C(D;G)C(D;G)CEAEAEFEF
4	EFEFEFCFC(D;G)C(D;G)FC(D;G)

shows a sample AOI fixation sequence for four time intervals in a hypothetical scenario with seven AOIs.

3.5. STEP 5. Develop AOI transition matrix for the selected time interval

Step 5 receives the AOI fixation sequence developed in step 4 as input. Afterwards, the AOI fixation sequence is transformed, as per the approach suggested by Noton and Stark (1971), for developing the associated AOI transition matrix, which is the output of this step. The size of the transition matrix depends on the number of unique AOI states (single and overlapped both) in the fixation sequence. The AOI transition matrix shows, in a tabular manner, how many transitions have occurred between various AOIs pairs. For example,

Table 2. AOI transition matrix developed from the AOI fixation sequence for time interval 1 within Table 1.

From AOI	To AOI
	A	B	C	E
A	0	3	2	3
B	3	0	0	1
C	2	0	0	0
E	3	1	0	0

represents the AOI transition matrix associated with the AOI fixation sequence for time interval 1 in Table 1. The sequence shows there are three EF transitions from AOI A to B highlighted in grey within Table 2. A different AOI transition matrix is required for each time interval; thus, before moving to the next step, we need to create the associated AOI transition matrix for each time interval.

3.6. STEP 6. Develop DNet visualization

The inputs for step 6 are the AOI transition matrix for all the time intervals obtained in step 2. As output from this step, we obtain the DNet representation of the EM data. Developing the DNet involves two steps. First, developing a static network for each of the time intervals considered in STEP 2. Second, arranging the static networks in a time-based order to visualize the DNet. The details of both the steps are given below.

3.6.1. Develop static network for each time interval

The static network is developed by adopting the design principles suggested by Mandal et al. (2016). Thus, a network’s vertex size is drawn proportional to the number of EFs received by the corresponding AOI and the vertex’s color is based on a sequential multihued color scale, where red color means high EF duration and yellow color means low EF duration occurring on the associated AOI. The thickness of an edge between a vertex pair is proportional to the number of EF transitions occurring between the vertices in the edge’s direction. Therefore, for the example shown in Table 1, we have four time intervals, thus the DNet is written as DynN = {N₁, N₂, N₃, N₄}, where N_i is the static network for time interval i (i = 1, …, 4).

3.6.2. Visualize the DNet

Before visualizing the DNet, the vertices of the component static networks are to be arranged in a specific layout for mental map preservation. We have used the rectangular grid layout for this purpose; in which, we start from the left bottom corner and move towards the right, ending at the top right corner. We sort the AOI positions using the natural ordering of English letters and putting uppercase AOI groups first and then lowercase AOI groups. The single AOIs are followed by overlapped AOI cases in the grid layout. The overlapped AOI cases are arranged in an increasing order of constituent AOI numbers, i.e. an overlapped AOI case with two AOIs comes before a case with three AOIs. For example, overlapped AOI (A;B) comes before (A;B;C), which comes before (A;B;C;D), and so on.

Consider an AOI fixation sequence that has n unique AOI states (including both single and overlapped AOIs). Therefore, the number of columns (number of AOIs in a single row) in the grid layout will be the smallest integer greater equal to n. Figure 3 represents a sample ordering scheme of six AOIs in a rectangular grid layout.

Once the component static networks are arranged in a grid layout, we visualize the DNet by arranging the static networks in a time-ordered sequence. Figure 4 represents a sample DNet visualization of the EM data for a hypothetical scenario (see section 3.4). Note that the relative location of all AOIs in Figure 4 remains constant in each of the networks corresponding to various time intervals. For example, AOI A is placed at the bottom row first column in all the constituent networks. This constant relative position of each AOI helps in their navigation across various time intervals and thus helps to preserve the mental map of the observer. The DNet has four time intervals, and the AOI fixation sequences for all these intervals is shown in Table 1. We should note that, due to the dynamic nature of the AOIs and typical scanning strategy of ATCSs, some AOIs may not be fixated upon despite being present on the display. In addition, the overlapped AOI cases arise in the AOI fixation sequence only if they are fixated upon by the ATCSs. Theoretically, there can be many possible overlapped AOI cases as compared to what is observed in real life experimental data. For example, with n unique AOIs, theoretically, we only if they are fixated upon by the ATCSs. Theoretically, there can be many possible overlapped AOI cases as compared to what is observed in real life experimental data. For example, with n unique AOIs, theoretically, we can have ${\sum }_{r=1}^n\left(\begin{array}{c}n\\ r\end{array}\right)={\sum }_{r=1}^n{\displaystyle \frac{n!}{r!(n-r)!}}$ possible AOIs cases (including both single and overlapped cases) on the display. Although, not all overlapped AOI states will appear on the display and even if they occur not all of them will be fixated upon by the ATCSs. This is also the case with single AOI cases.

Thus, it is computationally expensive and inefficient approach to consider all those AOI states which have not being fixated at all and thus does not appear in the AOI fixation sequence. Consequently, we only consider those AOI states for visualization and analysis which appear at least once in the AOI fixation sequence of the ATCS’s. As a result, we ignore those AOIs which, despite being present on the display, received no EFs from the ATCSs. Therefore, not all AOIs appear in the DNet visualization for each of the ATCSs, which results in a change in the relative position of each AOI in the grid structure of the DNet visualization. In addition, for comparing various visual scanning strategies, we only consider the AOI cases which are common to all the AOI fixation sequences of various ATCS’s.

3.7. STEP 7. Calculate target (or AOI) importance measures

Step 7 involves calculating the measures using equations (i), (ii), and (iii). As a result, we provide the DNet as the input to this step, and as output we obtain all three importance measure values for each of the AOIs with respect to all the time intervals considered in the DNet. For example, consider the DNet visualization in Figure 4, where the indegree value of AOI B changes from 4 to 3 as we move from the time interval 1 to 2 (Figure 4a,b). For the given DNet in Figure 4, to calculate the closeness and betweenness value for AOI B, we first demonstrate how to calculate the distance between the AOIs. For example, in the first time interval, the distance from AOI B to A is given by d_BA(t) = 1/w_BA(t) and substituting t =1, we get d_BA(1) = 1/3.

To calculate the minimum distance between two AOIs consider Figure 4a, where there are two EF transition paths from AOI B to E: Direct path from AOI B to E, which has edge weight 1; indirect path from AOI B to A and then to E, which has edge weights 3 and 3, respectively. Thus, the shortest distance from AOI B to E is defined as the path having the minimum distance of d_BE* (t) = min[d_BE(t), (d_BA (t) + d_AE(t))]. For the first time interval (t = 1), we obtain d_BE* (1) = min[1/ 1, (1/3 + 1/3)] = min [1, 2/3] = 2/3. Thus, for the first time interval, the closeness and betweenness value of AOI B is given by 2.375 and 8, respectively.

3.8. STEP 8. Normalize and visualize target (or AOI) importance measures

The last step involves normalizing the calculated measures and subsequently visualizing the normalized measure values. Thus, the obtained AOI importance measures in step 7 acts as input for this step, and as output we obtain the normalized measure values accompanied by their visualization. Equation (i), (ii), and (iii) shows that the three measures both depend on the number of AOIs and the amount of EFs. In addition, these measures also have different units, as a result, they are incommensurable. For the present dynamic scenario, both the number of AOIs and EFs are not constant for the different time intervals considered in the DNet analysis. In addition, to compare the AOI importance values across various time intervals and across multiple observers, we need to eliminate the units of importance measures, thereby bringing them to a similar scale. To address this issue, we present two normalization options.

3.8.1. Percent normalization

Percent normalization refers to dividing an importance measurement of an AOI by the sum of the same measurement of all the AOIs. The percent normalized indegree value of an AOI shows the percentage share of the total number of EFs received by an AOI. Thus, it can be interpreted as the percentage of the total attention attributed to an AOI. The percent normalized indegree value of the j^th AOI for time interval t is calculated as:

$${\overline{I}}_j\left(t\right)={\displaystyle \frac{I_j\left(t\right)}{{\sum }_{j=1}^{m_t}{I}_j\left(t\right)}}$$

iv

Where, ${\overline{I}}_j\left(t\right)$ is the percent normalized indegree value for time interval t. We get, $0\le {\overline{I}}_j\left(t\right)\le 1$ and ${\sum }_{j=1}^{m_t}{\overline{I}}_J\left(t\right)=1$.

3.8.2. Distance normalization

Distance normalization refers to calculating how far away a given importance measurement of an AOI is from the maximum value observed for that measurement within a time interval. The distance normalization process is defined for all the three target importance measures. The distance normalized measure value of an AOI can be interpreted as the relative amount of attention given to an AOI as compared to the maximum amount of attention given to any AOI. The distance normalized measure for the j^th AOI, for time interval t, is calculated as:

$$\widetilde{{\varnothing }_j}(t)={\displaystyle \frac{{\varnothing }_j(t)-{\min }_j{\varnothing }_j(t)}{{\max }_j{\varnothing }_j(t)-{\min }_j{\varnothing }_j(t)}}$$

(v)

Where, ${\mathrm{m}\mathrm{a}\mathrm{x}}_j{\varnothing }_j(t)$, ${\mathrm{m}\mathrm{i}\mathrm{n}}_j{\varnothing }_j(t)$ and ${\tilde{\varnothing }}_j(t)$ is the maximum, minimum, and distance normalized value of the measure ${\varnothing }_j\left(t\right)$ respectively ($0\le {\tilde{\varnothing }}_J(t)\le 1$).

${\varnothing }_j\left(t\right)$ is applicable for representing any of the three target importance measures (i.e. indegree, closeness, and betweenness) whereas percent normalization is useful for only indegree due to manner in which the measures are calculated (see sections 2.4 and 3.7).

3.8.3. Target (or AOI) importance measure visualization

Once the values of the target importance measures are normalized, the last step involves their subsequent visualization. We provide a couple of examples of visualization approaches: Dot plot based on multiple time intervals (for a single observer and single vertex importance measure) and bar plot based on multiple participants (for a single time interval and single vertex importance measure). There can be various ways to visually represent the combinations of importance measures, normalization methods, multiple time intervals, and number of participants. Another example of a bar plot that compares all importance measures of a single participant for a single time frame is also provided in the Results section.

3.8.3.1. Adapted dot plot visualization

Dot plots can better represent the evolution of the visual attention across various time intervals. We have adapted the dot plot to visualize the percent normalized indegree measure for all the time intervals considered in the DNet framework. Using the normalized measurements helps in comparing the relative significance of AOIs within a single time interval and how an AOI’s importance is changing across various time intervals. Figure 5 shows a sample dot plot for the normalized indegree measure for all the AOIs presented in the DNet in Figure 4. In Figure 5, the size of each dot is proportional to the percent normalized indegree value of the AOI for the given time interval.

3.8.3.2. Adapted bar plot visualization

Bar plots can better assist in comparing a single importance measure among multiple participants or multiple importance measures for a single participant. Figure 6 shows an example of the adapted bar plot that compares the indegree measurements among multiple participants

4. Experiment

The proposed approach was implemented into an air traffic control task. The task involved expert ATCSs observing the radar display to detect possible aircraft collisions. Details are as follows.

4.1. Participants

Three retired ATCSs (P1, P2 and P3) with over thirty years of experience were participants for the experiment. The ATCSs were recruited with the help of FAA Civil Aerospace Medical Institute (CAMI). Also, three FAA CAMI employees were also involved in the experiment as pseudo-pilots. They made the necessary maneuverings by following the ATCS’s voice commands. A simulated radio connection was used as the communication channel between the pseudo-pilots and ATCSs.

4.2. Apparatus

Hardware: A 19.83 × 19.83-inch monitor, with a 2048 × 2048-pixel active display area, was used for displaying the simulated air traffic scenarios. Apart from this, an additional monitor (kept to the right of the simulation monitor) was used to display textual information about the aircraft, e.g. trajectory and possible future conflicts it might encounter (also known as the Enroute Automation Modernization tool). The ATCS’s used a keyboard, placed beneath the simulation display, to provide necessary input commands. FaceLab 5 eye tracker with an accuracy in the range of 0.5 – 1.0 degree visual angle error and sampling rate of 60 Hz (Eyetracking.com, 2017) was used to collect ATCSs’ EM data. A threshold of 100 milliseconds was used for defining fixations. The participants were seated within a range of 50-70 cm from the simulated radar display. Software: A customized ISim software used by FAA CAMI was used for simulating the enroute air traffic scenario (with an update rate of 1 sec). EyeWorks software was used to process the raw eye tracking data collected through FaceLabs eye trackers.

4.3. Scenario and task

4.3.1. Scenario

For the experiment we used one 20 minutes long simulated enroute air traffic scenario, which was developed and provided by the FAA. This scenario had a total of thirty nine unique aircraft (named as A, B, …, Z, a, b, …, m) with an average of twenty aircraft per frame. The scenario had a minimum of seven and maximum of thirty aircraft per frame. Figure 7 represents a screen capture of the simulated scenario. The scenario update rate was 1 second. The display shows various aircraft and a weather patch in blue color. The aircraft representation shows the direction in which the aircraft is moving (shown by the white line) and data block which contains information about the aircraft’s computer code name, altitude, current speed, and destination airport. For example, in Figure 7, the aircraft N7890 (AOI U) is flying at a constant altitude of 19,000 ft (as shown by 190C) with a speed of 402 knots and it going to KGPT airport (i.e. Gulfport–Biloxi International Airport in Mississippi).

4.3.2. Task

The task involved controlling the simulated low altitude enroute airspace using the ERAM system. The ATCSs are required to fulfil two objectives: (1) Route the aircraft through the sector within the display and (2) avoid any conflict scenario by preventing loss of separation (vertical and horizontal separation of 1000 ft and 5 knots respectively) between aircraft. To achieve these objectives, the ATCSs gave voice commands (e.g. change is altitudes, speed and direction of the aircraft) to the pseudo pilots, using the simulated radio, for necessary maneuvering of the aircraft.

4.4. Data analysis

For the DNet analysis, the 20 minutes simulated scenario was divided manually, on trial and error basis, into four equal time intervals of 5 minutes. This choice was motivated by the fact that we observed on average an aircraft spends around 5 minutes on the radar display. Event based intervals were not applied in this research since many aircraft appeared and disappeared throughout.

Matlab was used to develop the AOI fixation sequence for each of the four time intervals mentioned above. In addition, igraph package (Csardi & Nepusz, 2006) in R software was used to create the dynamic network representation of the AOI fixation sequences and also for AOI importance metric calculation and visualization.

Figure 2 shows there are eight major steps in the proposed methodology. The time complexity (TC) of steps 1-4 for one time interval can be written as TC_1–4 = α₁e_tn_t, where, e_t and n_t are the number of eye fixations and number of AOIs in time interval t respectively, and α₁ is some positive constant. Due to human physiological limitations the value of e_t cannot increase arbitrability. Therefore, the governing factor of the time complexity for steps 1-4 is the number of AOIs. Thus, we can approximately write TC_1–4 = O(n_t). Taking the worst case scenario, we can find an upper bound for this value by taking the maximum number of AOIs across all time intervals. Let, n_max be the maximum number of AOIs present during any of the time intervals. Thus, we get TC_1–4 = α₁n_max. Similarly, the time complexity of step 5 can be written as TC₅ = α₂n_max², where α₂ is some positive constant. For T time intervals the time complexity of step 1-4 and 5 can be written as TC_1–4 = Tα₁n_max and TC₅ = Tα₂n_max² respectively.

As mentioned in section 3.6.2, in a given DNet visualization, the relative location of each AOI remains constant within the various constituent networks for various time intervals. As a result, for drawing the DNet, we need to consider all the unique AOI states that arise in the AOI fixation sequence of an ATCS. Therefore, the time complexity for step 6 can be written as, TC₆ = Tα₃n_uniq, where, n_uniq is the number of unique AOI states (both single and overlapped) in the AOI fixation sequence and α₃ is some positive constant. Step 7 consists of calculating three AOI importance measure. The time complexity for indegree, closeness and betweenness metric are Tα₄n_max, Tα₅(n_max² − n_max), Tα₆n_max(n_max − 1)(n_max − 2) respectively. Thus, the time complexity for step 7 can be written as TC₇ = Tα₄n_max + Tα₅ (n_max² − n_max) + Tα₆n_max(n_max − 1)(n_max − 2). The time complexity of Step 8 can be written as TC₈ = Tα₇n_max, where, α₇ is some positive constant. Adding all the time complexity for step 1-8, the total time complexity for the overall process can be written as: TC_tot = Tα₁n_max + α₂n_max² + α₃n_uniq + α₄n_max + α₅(n_max² − n_max) + α₆n_max (n_max − 1)(n_max – 2) + α₇n_max]..

On simple algebraic reorganization, we get that: TC_tot = T[α₆n_max³ + n_max² (α₂ + α₅ − 3α₆) + n_max (α₁ + α₄ + 2α₆+α₇ − α₅) + α₃n_uniq].

Thus, neglecting the lower order terms of n_max, the approx. time complexity of the overall data analysis process reduces to the order of O(Tn_max³ + Tn_uniq). Thus, we can see that the number of time interval, maximum number of AOIs within any time interval and the number of unique AOI states in the AOI fixation sequence do impact the processing time of the proposed approach.

5. Results

5.1. Dynamic graph visualization

Figure 8 represents the DNet visualization of the EM data of one ATCS for the simulated scenario shown in Figure 7. In Figure 8a (i.e. time interval 0-5 minutes), AOI F (i.e. AAL68) and U (i.e. N7890) are the most important AOIs as they both have most EF numbers (circle size) and longest EF duration (circle color). In addition, there are highest EF transitions between AOI F and AOI U (based on the thickness of the link). AOI K (i.e. EJA33), despite having a small number of EFs, has substantially longer EF duration. AOI b and AOI d can also be considered as important AOIs based on how a researcher wants to set the threshold.

For the second time interval (i.e. 5-10 minutes) shown in Figure 8b, the important AOIs have changed to AOI G (newly appeared aircraft not shown in Figure 8a and AOI K. Notice that AOI F has moved out of the display (see Figure 7b) and AOI U is still within the display but AOI U is not visually attended any longer.

In addition, notice that AOI d (i.e. UAL1322) has been receiving consistent visual attention throughout the two time intervals 0-5 minutes and 5-10 minutes. Similarly, the important AOIs change for the next two time intervals. In the last time interval, the overlapping AOI (i.e. AOI (J;f)) receives much visual attention. As we move across time intervals from 1 to 4, a visible trend is the increase in the complexity of the network, with increase in the number of AOIs and EF transitions among them.

5.2. Adapted dot plot

Figure 9 represents the dotplot visualization of the normalized indegree measure for all the AOIs present in DNet shown in Figure 8. For example, in Figure 9a, AOI F and U has high importance in first time interval (i.e. 0-5), although, their importance reduces drastically in the subsequent intervals. AOI d receives consistent visual attention throughout the first two intervals. The indegree results are in accordance with the DNet results in Figure 8 since indegree measures the number of EFs received by an AOI.

The adapted dot plot better shows the evolution of important AOIs. Considering AOI K, we see that its importance initially grows as we move from the first time interval to the second, where it reaches its maximum and then starts decreasing for the last two time intervals. Another noticeable fact is that majority of the overlapped AOI cases have significant importance only in one time interval. These trends took more time to identify when observing the DNet.

5.3. Adapted bar plot

Figure 10 represents the relative importance of various AOIs present in the first time interval of participant 1 (i.e. P1). Again, AOI F and U are important, but also, we can identify that AOI b (i.e. SWA340) and AOI d can also be important AOIs when we additionally consider the closeness and betweenness values. We can also observe that AOI K had small EF numbers during the first interval, however, it can be considered as an important AOI due to the long EF duration and relatively high closeness value. Furthermore, we can see that the AOI Q and AOI a might be an important AOI considering that AOI (Q;a) also has moderate indegree and closeness values.

Figure 11 shows the measures (i.e. indegree, closeness, and betweenness) visualized based on participants for the first time interval. Although there are slight variations, we can see a consistent trend among the participants. In addition, it becomes more evident that AOI Q and AOI a can be important AOIs (considering the values of AOI (Q;a)) in addition to AOI F, AOI K, AOI b and AOI d. In detail, overlapping AOI (Q;a) received substantial EF duration (as shown by the vertex color in the DNet visualization) although it received moderate number of EFs on it (as shown by the vertex size in the DNet visualization). On the other hand, comparing the three importance measurements, AOI (Q;a), in spite of having moderate indegree and closeness value, has insignificant betweenness value.

This might be due to the short lifespan of the overlapped AOI state since AOI a (i.e. SWA2920 flying at 21000 ft with speed 424 knots overtakes AOI Q (i.e. N46332 flying at 7000 ft with speed 163 knots very quickly on the display. As a result, it is highly unlikely that it will play a crucial role as a bridge for the flow of attention between other groups of aircraft.

6. Discussion

Results obtained from the DNet visualization coupled with multiple time interval visualizations enabled us to identify which are the important targets in each of the time intervals and how their importance evolves as we move across time intervals. Application of the three adapted target importance measures (i.e. indegree, closeness, and betweenness) along with the various ways to visualize those values showed that we can better analyze target importance measures that accounts for the interaction among targets.

In addition, we could identify various EM network characteristics of each participant and how the relative importance of various targets differs among them.

DNet was better to visualize the EM flow of the overall network and EF transitions, whereas the adapted dot plot was better to visualize the evolution of importance of each AOI across various time intervals. It is noted that the results could have been different if we had used a different time interval. For example, if the time interval was to 10 minutes, the most important AOI would turn out to be AOI d and the importance of AOI F and AOI U would have been less substantial. However, such results can be misleading since AOI F and AOI U were important AOIs during the first 5 minutes.

Selection of time interval thresholds (whether fixed or varied based on events/triggers) can be tricky and depend on the scenario characteristics.

The adapted bar plots were better when multiple target importance measures and/or multiple participants’ measures were needed to be compared side-by-side. The normalized and adapted measure plots show that, despite receiving substantial amount of EFs, an AOI may not be significant in terms of the flow of visual attention across the various AOIs within the display as shown by the closeness and betweenness measures. The proposed approach can be useful in increasing the training efficiency of the novice controllers. Novice controllers can know which are the important targets that needs to be focused upon and how to move the attention across various targets as the scenario characteristics change. Furthermore, the trainees can better understand which targets are highly correlated for conflict mitigation through observing the EF transition characteristics using the DNet and evaluating the closeness/ betweenness values.

7. Conclusion

In this research, we integrated the DNet framework with three target important measures (i.e. indegree, closeness, and betweenness). During the integration, we normalized the measurements and then adapted the dot plot and bar plot to better visualize the outputs. The approach facilitated the understanding of how visual attentions occur on the dynamic network (i.e. EM network created from an aircraft conflict task) from various perspectives. The results obtained showed that, the traditional approach of using the raw EM data measures (i.e. number and duration of EFs) might be misleading for dynamic scenarios where the targets’ lifespan on the display are nonuniform. The proposed approach enabled us to better understand how the observers’ attention were devoted to the various targets including the overlapping targets on the display. In addition, in case of dynamic targets, to understand target importance we need to also consider which targets are integral in the smooth flow of attention across the various targets within the display.

8. Limitations and Future work

One challenging issue was determining the time intervals for effective analysis. We have used five minute time intervals for the experiment based on trial and errors and we could have applied time intervals based on specific events (e.g. when target appears or disappears, or when a verbal control command is issued by an expert ATCS).

However, event-based time intervals can create issues if too many events occur within very short times or too few events occur in very long times. Instead, we could try applying shorter time intervals. For example, reducing the 5 minute interval to 1 minute interval). However, we might be burdened with visualizing and analyzing too many outputs. Therefore, formal sensitivity analysis needs to be performed as part of future study to determine an optimal range of values for the intervals.

In addition, we have not used pre-determined thresholds to identify important AOIs and only identified them if all measures were relatively higher than others. Therefore, a statistical method should be developed that clearly differentiate the important AOIs from the non-important one.

Moreover, as a future study, we can also analyze the variation of the saccade length and eye fixation duration with time and various air traffic scenario characteristics. For example, we can consider how the distance between several aircraft on the display affects the saccade length of the ATCS. This might help us analyze the type of search behaviors undertaken by them, i.e. is the search goal oriented where they search for targets having similar characteristics or is it a random one.

With increase in the number of AOIs and EF transition between them, the visual scalability of the DNet visualization gets impacted negatively, as there are more instances of edge crossings in the network representations. Furthermore, increase in the number of time intervals also increases the number of static networks within a DNet framework. This leads to increase in the cognitive load of the observers as they have to keep track of an AOI across more number of network representations. Thus, in terms of visualization scalability, we can apply various graph simplification processes, also known as graph filtering processes, where the unimportant edges (i.e. edges representing low EF transitions) are not considered for visualization purpose, thus reducing the visual clutter.

Furthermore, it can be valuable to analyze the community structure of AOIs (cluster of AOIs having high amount of EF transitions between them) and their evolution formed in the network representation of the EF data. Finally, we are planning to develop a mapping scheme between the visual scanning pattern classifications (Mcclung & Kang, 2016) with the DNet results. This might help in understanding how different visual scanning strategies are related to the overall target’s importance.