Trajectory Predictor and Conflict Detection Figures of Merit for a Performance-Based Adaptive Air Traffic Monitoring System

Abstract

This paper investigates the impact of trajectory predictor performance on the encounter probability generated by an adaptive conflict detection tool and examines the flexibility of the tool dependent on its adjustable thresholds, using historical radar track data. To achieve these objectives, two figures of merit were proposed: System Dynamic Range and System Tuning Envelope. To examine the conflict detection’s performance variability under different uncertainty levels and predictor types, simple multi-horizon trajectory predictors trained with two machine learning techniques of different characteristics are assessed at various look-ahead times: extreme gradient boosting with a discrete nature and a multi-layer perceptron regressor with a continuous nature. The results highlight the interdependence between the performances of the trajectory predictor and the conflict detector, and the quantification of this relationship can be represented through a sigmoid function. In addition, the two proposed figures of merit are effective for selecting suitable operating points in an adaptive conflict detector, based on dynamic thresholds and the performance requirements necessary for the trajectory predictors to achieve the expected detection performance at different look-ahead time.

1. Introduction

The accommodation of increasing levels of air traffic has long been limited by the insufficient capacity of the ATM (Air Traffic Management) system, leading to a significant number of delays caused by air traffic flow and management capacity constraints. This not only results in longer flight times but also contributes to increased fuel consumption and emissions [1].

The COVID-19 pandemic caused a significant drop in air traffic, but as the global situation improved, traffic now is rising once again to pre-pandemic levels [2]. To meet the increasing demand, airspace capacity must increase. Capacity increments can be approached from various perspectives. The main driver of capacity limits is the workload of air traffic controllers (ATCos). The workload of ATCos is tied to the tasks they perform during their shifts, and it varies based on the number of flights within their sector and the level of complexity each flight presents [3]. For example, in an en-route sector, a flight may only require minimal attention if it is simple to monitor, or it may require a significant amount of attention if it interacts with other flights and potentially causes conflicts that require additional clearances [4,5]. Therefore, employing additional supporting tools for ATCos to handle potential conflicts has long stood as one of the solutions to reduce workload and increase sector capacity.

Efforts are underway, particularly through the SESAR 3 Joint Undertaking programme in Europe and the NextGen program in the United States, to address this challenge. In particular, a project of the SESAR programme, PJ18-W2 4DSkyways, developed a solution named “Improved Ground Trajectory Predictions enabling future automation tools”, which aimed to increase the quality of separation management service. The primary objective of the solution was to validate enhanced conflict detection and resolution (CDR) tools, rooted in the improvement of the performance of ground trajectory predictors (TPs), thereby enhancing the overall quality of service [6]. The present paper was developed within this project. Therefore, this paper presents metrics to assess the interrelation between the performance of TPs and CDs (Conflict Detectors), to facilitate establishing performance requirements for TP services to support the separation management function.

The sections of this paper are arranged as follows: Section 1 includes a brief introduction and Section 1.1 presents a literature review of the related topic; the methodology adopted for the construction of the TP and CD tools and proposed figures of merit is described in Section 2, including the outline of the selected use case for the analysis; the results of the figures of merit are displayed in Section 3; and finally, the findings of this work and the next steps are summarised in Section 4.

1.1. The State of the Art

A study carried out by EUROCONTROL Experimental Centre using fast time simulation demonstrated that, when controllers are equipped with conflict detection tools such as medium-term conflict detection (MTCD) and a tactical controller tool with a perfect accuracy level, their task load is reduced by up to 21%, leading to a more balanced distribution of task load between planner and tactical controllers [7]. Consequently, one viable approach to reducing air traffic controller workload and increasing capacity is to provide them with more advanced decision support tools (DSTs) for CDR [8].

However, CDR functionality is dependent on an accurate trajectory prediction. As a result, the performance of the CDR tool and the overall ATC (Air Traffic Control) system is directly impacted by the accuracy of the TP [8].

Trajectory prediction was defined in [9] as the process that estimates a future trajectory of an aircraft through computation, and it is performed by a trajectory predictor. The TP can adopt various methodological approaches [10]. Ref. [11] conducted a general review of 282 papers, with a detailed examination of 20 of them. These selected studies showcased diverse methodologies, including mathematical models such as point-mass models with the BADA (Base of aircraft data) database [12], kinematic models, and kinetic models, among others. Recent studies have facilitated the integration of novel input sources and methodologies. Regarding data, the inclusion of onboard information through extended projected profiles enables prediction enhancement [13]. Regarding the method, the utilisation of data-driven predictors trained using machine learning (ML) algorithms has become widespread in present times [14,15]. These ML techniques have exhibited superior performances to conventional methods, given an appropriate dataset and training pipeline [16].

Predictions are always subject to error. The inaccuracies of predictions stem from diverse sources, leading to a high level of uncertainty [17,18,19,20], and their applicability depends on the operational context [21]. The consideration of this uncertainty is crucial in decision-making processes within ATM. High uncertainty leads to an amplification of the workload, due to the increasing need to monitor potential conflicts, an increase in both false and missed alerts [19,22], and results in more drastic clearances at a small look-ahead time (LAT) [23]. On the contrary, low uncertainty implies high reliability, enabling anticipated resolutions.

Ehrmanntraut [22] highlighted the importance of considering the uncertainty of trajectory prediction in conflict detection tools. CDR tools generally operate by identifying the breach of minimum separation between two aircraft, indicated by a specific measure falling below a predetermined threshold. The commonly used approach involves projecting the trajectory over a specified look-ahead time and determining if the two aircraft will lose separation by evaluating the probability of their relative distance being below a previously set threshold for each time point in the series [24].

Regarding the requirements of TP and CDR, basic metrics to evaluate TP performance encompass longitudinal, lateral, and vertical prediction error [25]. In [26], Mondoloni introduced a framework to define metrics, providing a list of metrics for assessing TP performance and presenting the system operating characteristics (SOC) curve, which depicts the trade-off between false alert (FA) and missed alert (MA) of a CD tool. Additionally, Schuster et al. [8] conducted a review on the requirements of TP and CDR for each element of the SESAR Concept of Operations and proposed performance metrics such as TP accuracy, TP integrity, CD false alert, and CD integrity.

Concerning the link between TP and CDR, ref. [27] proposed a methodology for the functional verification and performance validation of TPs. The proposed process categorises TP requirements into two groups: direct and indirect. Direct requirements are directly related to the TP itself, such as prediction accuracy, while indirect requirements relate to the impact on the overall system performance, like meeting the required CDR performance. Ref. [27] highlighted the difficulty in validating indirect TP requirements using recorded radar track data, primarily due to controller-issued clearances for deconfliction, which complicate the computation of false alerts and positive alerts. Therefore, ref. [27] proposed a validation approach relying on simulations that have a high level of fidelity, although the cost is expensive.

Many studies have used simulations to assess the impact of TP accuracy on CD performance. Ref. [28] evaluated the relationship between different levels of longitudinal error and the performance of a conflict probe using Monte Carlo simulations, and ref. [29] used simulations with a time shift from [30] to emulate conflict events, evaluating several scenarios of different TP accuracy levels using the hit rate and the FA rate. The latter study led to the conclusion that the conflict probe performance depends on TP accuracy, showcasing reduced false alarm rates and increased true positive rates as TP accuracy improves. Furthermore, the CD performance was estimated using a linear regression function based on TP accuracy.

Based on the SOC curve [26] and the conflict probability [24], this study proposes quantitative figures of merit using historical traffic data to evaluate the influence of TP performance on the encounter probability of a dynamic CD tool, without the need for expensive simulation effort. Apart from revealing their relation, these metrics inform the selection of the operating point of the CD for different LATs, based on the trade-off between false alerts and missed alerts, establishing requirements for TP performance and CD settings.

In this context, an encounter is considered to be a state which may develop into a loss of separation between two flights in the absence of ATCos intervention, and dynamic CD refers to a detection tool with dynamic variables in the setting capable of adjusting its operational performance to meet task-specific requirements.

Two TPs trained with different ML algorithms are selected for comparative evaluation regarding TP accuracy and prediction stability: extreme gradient boosting (XGB), a member of boosting algorithm family, and a multi-layer perceptron regressor (MLP), a type of artificial neural network.

This paper focuses on analysing the dependency between the performance of TPs and CDs, considering performance in terms of their accuracy and stability. The impact of accuracy is quantitatively assessed with two figures of merit, each supporting the selection of operating points in adaptive CD and defining performance requirements.

2. Methodology

This section provides an overview of the trajectory predictor and conflict detector tools developed for the analysis, including the corresponding figures of merit to assess their performance and their relationships. As the analysis focuses on the horizontal plane, the predictor is referred to as the horizontal trajectory predictor (HTP). To facilitate comprehension, the nomenclature used in this paper is summarised in

Table 1. Nomenclature—Relevant Abbreviations.

Abbreviation	Definition	Abbreviation	Definition
ANN	Artificial Neural Network	MLP	Multi-Layer Perceptron Regressor
AoI/R	Area of Interest/Responsibility	NM	Nautical Miles
ATCo	Air Traffic Controller	OSED	Operational Service and Environment Definition
CA/P	Critical Area/Point	RP	Real positive
CDR	Conflict Detection and Resolution	s	Seconds
DCB	Demand and Capacity Balancing	SDR	System Dynamic Range
DST	Decision Support Tool	STE	System Tuning Envelope
FA/MA	False Alert/Missed Alert	TA	True Alert
FL	Flight Level	TN	True negative
HTP	Horizontal Trajectory Predictor	TP	Trajectory Predictor
LAT	Look-Ahead Time	WV	Wake Vortex
ML	Machine Learning	XGB	eXtreme Gradient Boosting regressor

,

Table 2. Notation—Indexes.

Index	Definition	Index	Definition
i	Recurrent pattern of flow 1	k	Critical point within a critical area
j	Recurrent pattern of flow 2	p	Present or current

,

Table 3. Notation—Trajectory Predictor.

Notation	Definition	Notation	Definition
$\gamma$	Arc-length along the trajectory of each flow pattern [NM]	$t_e$	Elapsed time at sector entry [s]
${\gamma }_p$	Current position of the flight, measured as elapsed distance $\gamma$ of the flight from the entry point [NM]	$t_p$	Elapsed time from entry to ${\gamma }_p$ [s]
$\delta \gamma$	Target distance from the current location to make prediction [NM]	$F{L}_p$	Flight level at ${\gamma }_p$ [FL]
${\widehat{t}}_{\gamma ,{\gamma }_p}$	Predicted required time to reach the target point $\delta \gamma$ from the current position ${\gamma }_p$ [s]	$v_p$	Ground speed at ${\gamma }_p$ [knots (kts)]

and

Table 4. Notation—Conflict Detector.

Notation	Definition	Notation	Definition
$f_1$	Flight 1 of an interacting flight pair	${\gamma }_p$	Arc-length of evaluation, indicating ${\gamma }_{p2}$
$f_2$	Flight 2 of an interacting flight pair	${\gamma }_{C{P}_k}^{i,f_1}\left({\gamma }_{C{P}_k}^{j,f_2}\right)$	$\gamma$ of pattern i (j) of $f_1$ ($f_2$) to reach the critical point k [NM]
$d_{THR}$	Horizontal distance threshold to identify CP [NM]	$C{A}_{i,j}$	Critical point k between patterns [i, j], composed of the location of $f_1$ and $f_2$ along patterns i and j, respectively, [NM]
$\Delta t_{THR}$	Time difference threshold to obtain real conflict and encounter probability [s]	$C{P}_k^{i,j}$	Critical point k between patterns [i, j], composed of the location of $f_1$ and $f_2$ along patterns i and j, respectively, [NM]
$P_{THR}$	Probability threshold and operation threshold to decide whether notify an alert	${\widehat{t}}_{{\gamma }_p,{\gamma }_{C{P}_k}}^{i,f_1}\left({\widehat{t}}_{{\gamma }_p,{\gamma }_{C{P}_k}}^{j,f_2}\right)$	Time predicted to reach ${\gamma }_{C{P}_k}^{i,f_1}\left({\gamma }_{C{P}_k}^{j,f_2}\right)$ at ${\gamma }_p$ through pattern i (j) [s]
$\Delta t_{min}$	Real-time difference [s]	$\Delta {\widehat{t}}_{res,{\gamma }_p,C{P}_k}^{i,j}$	Difference between the predicted time to reach $C{P}_k^{i,j}$ of both flights for the patterns [i, j] [s]
$\Delta d_{min}$	Real minimum horizontal separation [NM]	${\sigma }_{{\gamma }_p,{\gamma }_{C{P}_k}}^{i,f_1}\left({\sigma }_{{\gamma }_p,{\gamma }_C{P}_k}^{j,f_2}\right)$	Standard deviation of HTP to predict ${\gamma }_{C{P}_k}^{i,f_1}\left({\gamma }_{C{P}_k}^{j,f_2}\right)$ at ${\gamma }_p$ through pattern i (j) [s]
$\Delta h_{min}$	Real minimum vertical separation [ft]	${\sigma }_{res,{\gamma }_p,C{P}_k}^{i,j}$	Combined standard deviation of the predictions of both flights for $C{P}_k$ [s]
$\mu$	Mean	$P_{{\gamma }_p,C{P}_k}^{i,j,\Delta t_{THR}}$	Encounter probability at $C{P}_k$ of the patterns [i, j] at ${\gamma }_p$
$\sigma$	Standard deviation	$P_{{\gamma }_p,CA}^{i,j,\Delta t_{THR}}$	Encounter probability of the patterns [i, j] of the flight pair
${\gamma }_{p1}\left({\gamma }_{p2}\right)$	Current position of $f_1$ ($f_2$) [NM]	$P_{{\gamma }_p}^{\Delta t_{THR}}$	Encounter probability of the flight pair

.

In this study, the measures are derived from the time separation at a specified location where minimum separation will be violated if both aircraft arrive concurrently, based on one of the cognitive abstractions of air traffic control operators (ATCOs) [4], who concentrate on critical points in an air traffic sector. Consequently, the probabilistic definition of that metric between two aircraft also considers uncertainty in the temporal dimension instead of the spatial dimension. This perspective was explored in [23].

Hence, the primary reference variable for the models is $\gamma$, which represents the arc-length along the trajectory of each pattern [23].

2.1. Horizontal Trajectory Predictor

The HTPs trained in this study are termed multi-horizon HTPs; this term refers to trajectory predictors that can update the prediction in accordance with variations in current position information by updating the inputs accordingly. This unique characteristic enables the assessment of performance at various positions without requiring the training of multiple individual predictors.

The objective of the HTP is to predict the time a flight requires (${\widehat{t}}_{\gamma ,{\gamma }_p}$) to reach a given distance $\gamma$ from the current position ${\gamma }_p$. The inputs include the following:

• aircraft characteristics: wake vortex (WV);
• information collected at the current position: current position (${\gamma }_p$), current flight level ($F{L}_p$), current speed ($v_p$) and elapsed time from the entry ($t_p$);
• target point: distance from current location to the target distance for prediction ($\delta \gamma$).

Regarding training, each identified traffic flow possesses its own predictor. As pointed out in [31], each flow exhibits recurring patterns—with the most commonly flown trajectories within it—with the planned route being one of them. Hence, it is worth noting that the same $\gamma$ can refer to different geographic coordinates, depending on the specific pattern and flow that are being considered, as shown in Figure 1. However, for each recurrent pattern, the relation between the coordinates and $\gamma$ is bijective.

The left-hand side of Figure 1 illustrates the concept of traffic flow, with two recurrent patterns for a single flow, and depicts the bijective relation between $\gamma$ and the geographical coordinates, varying according to the pattern. The right hand side of Figure 1 shows the concept of multi-horizon HTP, which depends on the current position and the target point.

The entry to the sector is considered to be the starting point of the trajectory of analysis. When a flight reaches its entry point at instance $T_e$, its current position or elapsed distance within the sector ${\gamma }_p$ is 0 NM, and its associated elapsed time of entry $t_e$ is also 0 s. As the flight progresses, its position ${\gamma }_p$ changes accordingly, along with the associated elapsed time $t_p$.

To present performance differences and their implications, two multi-horizon HTPs were trained using different types of ML algorithms that used the same features:

• Extreme gradient boosting regressor: a tree-based algorithm capable of providing reasonable performance with a small training sample. This HTP is referred to as XGB-HTP.
• Multi-layer perceptron regressor: a feed-forward artificial neural network (ANN). This HTP is referred to as MLP-HTP.

2.2. Conflict Detector

The aim of the CD tool is to detect flight pairs that could evolve into an encounter without ATCo intervention. Each identified flight pair is considered as potential conflict. This paper only considers flight pairs that belong to different traffic flows, excluding potential conflicts within the same flow.

The CD used for the study has three user-defined thresholds, namely distance ($d_{THR}$), time ($t_{THR}$), and probability ($P_{THR}$), for its functioning. The latter two parameters are always modifiable, while $d_{THR}$ is immutable during the functioning of the tool; it can only be set at the configuration phase. The incorporation of the two dynamic variables within the system architecture endows the CD tool with adaptive capabilities, enabling it to adjust its responses to meet different operational needs.

The CD tool identifies potential conflicts by calculating the time separation between two flights at their critical area (CA), a specific spatial location where an infringement in horizontal separation occurs if two aircraft arrive simultaneously or with small time separation. This is inherent information for the CD tool that is calculated based on the recurrent patterns of flow pairs. Thus, a critical area between the pattern i of a flow and the pattern j of another flow ($C{A}_{i,j}$) is an area that contains a set of tuples of two values of $\gamma$ - one for each flow - where the horizontal separation is below the selected distance threshold ($d_{THR}$). Each tuple is a critical point (CP), $C{P}_k^{i,j}$, and the number of CPs depends on the convergence between the two pattern trajectories.

Let us consider two flights: Flight 1 ($f_1$) as the first flight entering the sector of analysis, and Flight 2 ($f_2$) as the second. The mathematical representation of the critical area is provided as follows in Equation (1):

$$C{A}_{i,j}=\left[\begin{array}{c}{C{P}_1}^{i,j}\\ {C{P}_2}^{i,j}\\ \dots \end{array}\right]=\left[\begin{array}{c}({\gamma }_{C{P}_1}^{i,f_1},{\gamma }_{C{P}_1}^{j,f_2})\\ ({\gamma }_{C{P}_2}^{i,f_1},{\gamma }_{C{P}_2}^{j,f_2})\\ \dots \end{array}\right]$$

(1)

The identification of the potential conflicts relies on comparing between the time separation estimated at CA and the minimum time separation ($\Delta t_{THR}$).

As the estimated time separation is the difference between the two predictions—which has an associated uncertainty—the output of the CD tool is also a probabilistic variable. The formula for estimating the probability of conflict, presented in Equations (2)–(4), was demonstrated in [23], which resembles those demonstrated in [24] for distance-based thresholds. The probability, as estimated using Equation (2) as follows, is derived by applying the error function, denoted as erf.

$$\begin{array}{c}\hfill P(x\le \Delta t_{THR})=\frac{1}{2}\left[erf\left(\frac{\Delta t_{THR}+\mu }{\sqrt{2}\sigma }\right)+erf\left(\frac{\Delta t_{THR}-\mu }{\sqrt{2}\sigma }\right)\right]\end{array}$$

(2)

Based on the previous work of [23,24], when the prediction data follows a normal distribution centred around the prediction $\mu$ with its corresponding standard deviation $\sigma$, the encounter probability is defined as the probability of the time separation falling below $\Delta t_{THR}$.

Due to the probabilistic nature of the output, a probability threshold ($P_{THR}$) is set to select the sample of infringement notifications; flight pairs whose encounter probability is below $P_{THR}$ will be notified and require attention. Therefore, given the same encounter probability, the notifications vary according to $P_{THR}$.

Conflict Detection Process

Given two flights, the evaluation of their encounter probability begins when the second flight enters the sector. Hence, the arc-length of evaluation depends on the location of $f_2$ (${\gamma }_{p2}$). For facilitation, the location of the evaluation is named ${\gamma }_p$ in the sections of CD (${\gamma }_p$ = ${\gamma }_{p2}$).

The detection process at the arc-length of evaluation ${\gamma }_p$ encompasses the selection of potential interacting flight pairs and the calculation of the encounter probability.

The steps to identify the flights that potentially interact with $f_2$ are as follows (see Figure 2A):

1.

Select the concurrent flights of $f_2$; these are the flights that are also in the sector at ${\gamma }_p$. Figure 2A shows an example of the instant when $f_2$ enters the sector. The triangles represent the flights and the three coloured lines represent the recurrent patterns of $f_2$’s flow; these are possible trajectories that the flight might take. $f_2$ has five concurrent flights at the entry (A, B, D, E, and F), and flight C is discarded because it is still out of the sector.

2.

Select from the concurrent flights those that belong to flows that interact with $f_2$’s flow and do not belong to the same flow as $f_2$. In the example of Figure 2A, A and F are discarded because A belongs to $f_2$’s flow and F does not interact with $f_2$.

3.

Estimate the present location of the concurrent flights: ${\gamma }_{p1}$.

4.

Select those flights that have not reached any of their CA with $f_2$’s flow, i.e., flights whose ${\gamma }_{p1}<min({\gamma }_{C{P}_1}^{i,f_1},{\gamma }_{C{P}_2}^{i,f_1},\dots )$. The flights that have already overflown the critical areas will not lose separation with $f_2$. In the example of Figure 2A, B has already overflown CA.

For each remaining flight $f_1$, the encounter probability ($P_{{\gamma }_p,CA}^{i,j,\Delta t_{THR}}$) is evaluated for each pattern combination [i, j], because the pattern is unknown in advance.

1.

For each $C{P}_k^{i,j}$ of the $C{A}_{i,j}$ (see Figure 2B):

(a)

Predict the time each flight crosses $C{P}_k^{i,j}$ for both flights: ${\widehat{t}}_{{\gamma }_p,{\gamma }_{C{P}_k}}^{i,f_1}$ and ${\widehat{t}}_{{\gamma }_p,{\gamma }_{C{P}_k}}^{j,f_2}$. Figure 2B takes the pattern 1 (blue lines) of flight D and $f_2$, as an example. The HTP of each flow will predict the time each flight crosses the corresponding ${\gamma }_{C{P}_k}^{i,f_1}$ (${\gamma }_{C{P}_k}^{j,f_2}$) of each $C{P}_k^{i,j}$.

(b)

Determine the standard deviation of the following predictions: ${\sigma }_{{\gamma }_p,{\gamma }_{C{P}_k}}^{i,f_1}$ and ${\sigma }_{{\gamma }_p,{\gamma }_{C{P}_k}}^{j,f_2}$.

(c)

Obtain $\Delta {\widehat{t}}_{res,{\gamma }_p,C{P}_k}^{i,j}$ and the corresponding prediction error ${\sigma }_{res,{\gamma }_p,C{P}_k}^{i,j}$ (see Equations (3) and (4), as follows).

$$\mu =\Delta {\widehat{t}}_{res,{\gamma }_p,C{P}_k}^{i,j}={\widehat{t}}_{{\gamma }_p,{\gamma }_{C{P}_k}}^{i,f_1}-{\widehat{t}}_{{\gamma }_p,{\gamma }_{C{P}_k}}^{j,f_2}$$

(3)

$$\sigma ={\sigma }_{res,{\gamma }_p,C{P}_k}^{i,j}=\sqrt{{\left({\sigma }_{{\gamma }_p,{\gamma }_{C{P}_k}}^{i,f_1}\right)}^2+{\left({\sigma }_{{\gamma }_p,{\gamma }_{C{P}_k}}^{j,f_2}\right)}^2}$$

(4)

(d)

Based on the selected $\Delta t_{THR}$, evaluate $P_{{\gamma }_p,C{P}_k}^{i,j,\Delta t_{THR}}$ using Equation (2).

$$\begin{array}{c}\hfill P_{{\gamma }_p,C{P}_k}^{i,j,\Delta t_{THR}}=P(x\le \Delta t_{THR})\end{array}$$

(5)

2.

The final encounter probability between patterns [i,j] ($P_{{\gamma }_p,CA}^{i,j,\Delta t_{THR}}$) is calculated using the mean of the probabilities obtained from each $C{P}_k^{i,j}$ (see Figure 2C).

Finally, flight pairs whose $P_{{\gamma }_{p2},CA}^{i,j,\Delta t_{THR}}$ exceeds $P_{THR}$ are considered to be detected potential conflicts that could lead to horizontal infringement if the flights continue to follow their patterns [i,j].

2.3. Evaluation Metrics and Figures of Merits

For the assessment of HTP accuracy, the prediction errors are used. The target point for prediction is always a critical point, enabling the analysis of the encounter probability between flights.

Once evaluated, the impact of different HTP accuracies on the CD functioning is presented by two figures of merit. Beyond the quantification of the impact, each exhibits a specialised application.

The figures of merits are both defined in the Operational Service and Environment Definition (OSED) document of the project PJ18-W2 4DSkyways:

• System dynamic range (SDR): an element to support the possible encounter classification and to prioritise resolution, as well as evaluating the performance of the CD tool. It mainly represents the impact of the HTP accuracy on the outputs of the CD tool.
• System tuning envelope (STE): an element to facilitate the tuning of the CD tool using the parameters $\Delta t_{THR}$ and $P_{THR}$. It assesses the performance of the CD tool, supporting the selection of operating points.

These metrics are further detailed in Section 2.3.1, Section 2.3.2 and Section 2.3.3.

2.3.1. Horizontal Trajectory Predictor Performance Metrics

The performance of the HTP considered in this paper includes mainly the prediction accuracy, but the stability of the prediction is also analysed, with stability defined as the tendency to maintain the predictions’ consistencies over time.

The performance of the trajectory predictors is assessed by analysing the residuals of the test sample, which are the differences between the prediction (${\widehat{t}}_{{\gamma }_p,\gamma }$) and the real value ($t_{{\gamma }_p,\gamma }$), as follows:

$$res={\widehat{t}}_{{\gamma }_p,\gamma }-t_{{\gamma }_p,\gamma }$$

(6)

Based on the residuals obtained from the test sample, the metrics that facilitate the visualisation of the performance are as follows:

• Heatmap of residuals’ mean (${\mu }_{res,{\gamma }_p,\gamma }$), which represents the residual mean for each current position and possible target point. Ideally the mean is zero for every combination of $\gamma$ and ${\gamma }_p$.
• Heatmap of residuals’ standard deviation (${\sigma }_{res,{\gamma }_p,\gamma }$): ideally the deviation is zero for every combination of $\gamma$ and ${\gamma }_p$.

2.3.2. System Dynamic Range

Quoting the definition from the OSED: “… the dynamic range can be considered as the rates at which the conflict probability decreases as the predicted minimum separation increases…”. Hence, the SDR shows the relationship between the outputs of the TP and the CD, which are the predictions and the encounter probability, respectively. It is an extension of the system operating characteristic curves [26].

The graphical representation of the SDR is the encounter probability $P_{{\gamma }_p,CA}^{i,j,\Delta t_{THR}}$ versus the peak probability-predicted time separation $\Delta {\widehat{t}}_{{\gamma }_p,C{P}_{maxP}}^{i,j}$.

Figure 3 shows a draft of possible SDR behaviours, adapted from the OSED. Each curve represents a SDR curve using predictors of a given accuracy, from the perfect TP in red to the green curve with a large TP error. The coloured area from the highest to the lowest probability of the SDR is named transition area.

If the trajectory prediction error is large—indicating large uncertainty—the CD performance shows a slow transition from high to low encounter probabilities with a broader transition area, suggesting a low dynamic range and a shallow curve. On the contrary, high-performance predictors will exhibit steep curves and consequently, a large dynamic range and a quick transition. In the case of a perfect detection tool without error, the curve will resemble a step function, with an encounter probability equal to 100% for predictions below the “safe” time-over-difference threshold, and an abrupt drop to zero thereafter.

When the uncertainty follows a normal distribution and when considering the absolute predicted time difference, the resulting probability has a folded normal distribution [24]. Hence, the representation of the SDR should follow a S-shaped function. Based on this theoretical property, the shape of the SDR distribution can be approximated with the following logistic function:

$$f\left(x\right)=\frac{L}{1+e^{-k(x-x_0)}}+b$$

(7)

The parameters that define the logistic functions are as follows:

• L: the maximum value of the curve (b = 0) when the logistic curve is increasing, or the minimum value when it is decreasing.
• k: the steepness of the curve.
• b: the displacement of the curve over the y-axis.
• $x_0$: the x-axis value of the midpoint.

2.3.3. System Tuning Envelope

Due to the presence of uncertainty, conflict detection also has associated errors. Hence, a notification from the tool can be genuine or a false alert (FA), and a hidden instance can be correctly hidden or a missed alert (MA). These elements reflect the accuracy and reliability of the CD tool. STE, as a figure of merit, enables the assessment of CD accuracy using the proportion of FAs and MAs.

The adaptive conflict detector provides conflict alerts based on the estimation of the encounter probability, which depends on $\Delta t_{THR}$ and $d_{THR}$, and the operation threshold $P_{THR}$. Every possible interaction with an estimated probability higher than $P_{THR}$ will be notified, while those with an estimated probability lower than $P_{THR}$ will be filtered out.

However, $P_{THR}$ is a dynamic threshold. Therefore, a low $P_{THR}$ will result in a large number of notifications, including most of the genuine infringements, in exchange for overloading the controller with excessive nuisance notifications. This increases the workload and diminishes user confidence in the tool. Conversely, a high $P_{THR}$ implies fewer false alerts, but risks missing critical notifications.

The conditions to detect the nuisances and missed notifications are visualised in Figure 4. The horizontal and vertical grey lines indicate the $\Delta t_{THR}$. All instances falling into the green area represent real infringements, and those in the red area do not. Those on the left side of the vertical line are notified alerts, and the rest are hidden. Based on Figure 4, the following variables related to CD accuracy are derived and used for the representation of STE:

• False alerts or nuisance: incorrectly notified alerts. Flight pairs whose real minimum separation is above the threshold ($|\Delta t_{min}|>\Delta t_{THR}$) but are notified as potential conflicts because the estimated probability is above the decision threshold ($P_{{\gamma }_p,C{P}_{maxP}}^{i,j}>P_{THR}$).
• Missed alerts or missed: incorrectly hidden alerts. Flight pairs whose real minimum separation is below the threshold ($|\Delta t_{min}|<\Delta t_{THR}$) but the estimated probability is below the decision threshold ($P_{{\gamma }_p,C{P}_{maxP}}^{i,j}\le P_{THR}$).
• True alerts (TA): correctly notified alerts. Flight pairs whose real minimum separation is below the threshold ($|\Delta t_{min}|<\Delta t_{THR}$) and estimated probability is above the decision threshold ($P_{{\gamma }_p,C{P}_{maxP}}^{i,j}>P_{THR}$).
• True negative (TN): correctly filtered-out alerts. Flight pairs whose real minimum separation is above the threshold ($|\Delta t_{min}|>\Delta t_{THR}$) and estimated probability is below the decision threshold ($P_{{\gamma }_p,C{P}_{maxP}}^{i,j}\le P_{THR}$).
• Real positive (RP) or real conflict: flight pairs whose real minimum separation is below the threshold ($|\Delta t_{min}|<\Delta t_{THR}$).

The proportions of nuisance and missed alerts used in STE are calculated using the following equations:

$$Nuisance\%=\frac{FA}{TN+FA}$$

(8)

$$Missed\%=\frac{MA}{TA+MA}$$

(9)

The graphical representation of STE illustrates the trade-off between nuisance and missed alerts for every $P_{THR}$. These are reflected as proportions of FAs and MAs, respectively. Changes in the output notifications resulting from adjustments to $P_{THR}$ are captured in the STE, which helps balance the trade-off between nuisance and missed alerts. Consequently, STE supports the selection of the most suitable operating point for the CD tool at different LATs to align with specific operational needs.

2.4. Use Cases

To demonstrate the applicability of the proposed methodology, we have selected a use case that includes a complex crossing point. This use case has been chosen based on its relevance and adequacy to highlight our methodology.

The selected use case is located in the Spanish airspace. The area of responsibility (AoR) for the analysis is LECMPAU (Pamplona Upper Sector), where the interactions between flows take place. For the planning controller of LECMPAU, adjacent sectors should also be considered. Therefore, the collapsed sector LECMBDP is selected as the area of interest (AoI) for the analysis to anticipate possible encounters. LECMBDP comprises LECMPAU, LECMBLU (Bilbao Upper Sector), and LECMDGU (Domingo Upper Sector).

Two cruising flows are considered, named F4 and F24 (see Figure 5). F4 is a northward flow with 2890 flights, and F24 is eastbound with 1432 flights. F24 interacts with F4 within the AoR, close to the boundary.

This use case was selected because the traffic densities are significant and the critical points between the flows are close to the sector boundary. Clearances issued to the flights during their crossing through BLU or DGU could induce a conflict at PAU. An incident between a similar pair of flows occurred in 2018 [32].

Evaluation Contextual Parameters

The following four parameters are considered to analyse the adaptiveness and corresponding variability in performance: the independent variable ${\gamma }_p$ and the three adjustable parameters $d_{THR}$, $\Delta t_{THR}$, and $P_{THR}$.

For the analysis, the following values are selected:

• $d_{THR}$: context threshold that influences the resulting critical area.

  –

  1 NM.

• $\Delta t_{THR}$: internal parameter of CD that impacts the encounter probability.

  –

  10 s, 30 s, 60 s, 90 s, and 120 s.

• $P_{THR}$: internal parameter of CD that affects the notified alerts.

  –

  from 0.1 to 0.99, incremented by 0.1.

• ${\gamma }_p$: independent variable that indicates LAT and consequently, the level of prediction error.

  –

  0, 20, 40, 60, 80, 100, and 120 NM from the entry point in the sector of analysis.

Due to the characteristics of the two flows, flight pairs with the second flight belonging to F24 are more likely to evolve into a conflict. Therefore, the analysis is focused on these instances and along the trajectory of F24. The arc-length of evaluation (${\gamma }_p$) is referred to LECMPAU’s boundary; that is, the entry at LECMPAU is considered to be ${\gamma }_p=$ 0 NM. The planned trajectory of F24 from entry at LECMBDP to LECMPAU is 108 NM long, which equates to approximately 15 min, considering the mean required time of the traffic sample (see Figure 6).

3. Results

This section presents the results of the study, which are divided into two subsections. Section 3.1 analyses and compares the performance of the two HTPs, and Section 3.2 details the analysis of the metrics SDR and STE.

3.1. Horizontal Trajectory Predictor Performance

This subsection presents the performance comparison between the four trained HTPs for the flows F4 and F24.

For both flows, the data were partitioned, with 75% reserved for training and the remaining 25% designated as the test sample, to assess the models’ performance on unseen data. The XGB-HTPs were trained by fine-tuning their parameters, setting the depth of trees to five for F24 and six for F4, with the learning rate set to 0.055 in both cases. On the other hand, the network architecture of the MLP-HTPs consisted of two hidden layers, each with 12 neurons, and used the ReLU (Rectified Linear Unit) activation function.

Regarding the HTP accuracy, it is dependent on the current position of the flight and the prediction point; a smaller $\delta \gamma$ implies reduced uncertainty and increased predictability. Details of the prediction errors of the MLP and XGB-HTPs are shown in Figure 7 and Figure 8.

In each figure, columns represent flows; the top row illustrates the mean of the residuals, and the second row shows the standard deviation. Each heatmap displays a matrix containing the corresponding value for each pair of ${\gamma }_p$ and $\gamma$. As the target point $\gamma$ should be larger than ${\gamma }_p$, only values in the upper triangular area are available.

For both flows, ${\sigma }_{res,{\gamma }_p,\gamma }$ escalates with an increase in the distance between ${\gamma }_p$ and $\gamma$ because the uncertainty increases with $\gamma$, whereas ${\mu }_{res,{\gamma }_p,\gamma }$ predominantly fluctuates around zero. Statistically, the maximum value of ${\sigma }_{res,{\gamma }_p,\gamma }$ across the four HTPs is comparable, approximately 28 s, while ${\mu }_{res,{\gamma }_p,\gamma }$ varies within a range of −6 s to 6 s.

In comparison, the difference in ${\sigma }_{res,{\gamma }_p,\gamma }$ for F4’s HTPs is minimal. However, for F24’s HTPs, particularly when the ${\gamma }_p$ of F24 is high, ${\sigma }_{{\gamma }_p,\gamma }$ is notably lower in the XGB than in the MLP model. Additionally, the deviation of ${\mu }_{res,{\gamma }_p,\gamma }$ from zero is more pronounced in F4.

Diverse regions are discernible in the ${\mu }_{res,{\gamma }_p,\gamma }$ heatmaps of the four HTPs. These areas are related to trajectory patterns and common manoeuvres of the aircraft. A contributing factor to the formation of these regions is the speed variation. Figure 9 illustrates these speed variations from the entry point to the subsequent points ($V_{{\gamma }_p}$ − $V_{{\gamma }_p=0\mathrm{NM}}$) for each flight of the test sample.

Regarding the prediction stability, a clear difference between the XGB-HTPs and MLP-HTPs is evidenced by the presence of diagonal bands in XGB-HTPs heatmaps (see Figure 8), indicating prediction instability over $\gamma$. This phenomenon stems from the discrete nature of decision tree algorithms, where similar inputs ranges lead to identical prediction allocation, despite differences in the actual input values.

However, when analysing the prediction stability of particular flights, predictions of MLP-HTPs also exhibit fluctuation over $\gamma$, despite the continuous nature of the algorithm. The cause of the fluctuation could be due to various reasons, such as the non-linearity of the problem.

Figure 10 shows two flight pairs selected as examples, which present the predictions of the total time from the entry to their corresponding $C{P}_1^{1,1}$ at each ${\gamma }_p$.

In most cases, the predictions for XGB-HTPs exhibit more variation, although MLP-HTPs also show fluctuation.

3.2. Conflict Detection

One derivation from Equation (2) is that the performance of the CD is impacted by the uncertainty of the associated predictors, which increases over $\gamma$. Consequently, the performance tendency of the CD varies for each pair of patterns.

This subsection is focused on the horizontal potential conflicts between the pattern pair 1–1 of the flow pair F4–F24, where the second flight belongs to F24 (i.e., the flight of F4 is already within LECMBDP).

The CD performance is evaluated from two perspectives. Firstly, the HTP performance impact assessment considers the two differently modeled HTPs, alongside a selection of ${\gamma }_p$, which are selected as they represent different levels of uncertainty. Secondly, the examination of the effects of dynamic thresholds involves the $P_{THR}$, which sets the notification threshold, and $\Delta t_{THR}$, which influences the estimation of encounter probability.

The analyses of the metrics SDR and STE are detailed in Section 3.2.1 and Section 3.2.2.

3.2.1. System Dynamic Range

The estimated probability, and consequently the CD performance, are intimately related to the performance of the trajectory predictors used in the detection process. This dependency can be observed in the SDR graphs, which compare the predicted time separation and the estimated encounter probability.

Figure 11 presents the SDR graphs of the pair F4–F24 using different HTPs, time difference thresholds, and evaluation arc-lengths. It contains four graphs. The figures of column A are obtained using MLP-HTPs whereas those for column B are obtained using XGB-HTPs. The two upper graphs show the SDR curves across ${\gamma }_p$ when $\Delta t_{THR}$ = 60 s, and the lower two show the variation in $\Delta t_{THR}$ when ${\gamma }_p$ = −108 NM (entry point at LECMBDP).

In the four graphs, each point represents a flight pair whose encounter probability $P_{{\gamma }_p,C{P}_{maxP}}^{1,1,\Delta t_{THR}}$ is greater than zero. Points of the same colour form the SDR distribution of the same evaluation arc-length ${\gamma }_p$ or threshold $\Delta t_{THR}$. Each distribution has its corresponding logistic fit in the same colour.

Regarding the variation by ${\gamma }_p$, all logistic fits intersect with the black step function, which represents the ideal CD performance, around a midpoint. At this intersection, the $|\Delta {\widehat{t}}_{{\gamma }_p,C{P}_{maxP}}^{1,1}|$ is approximately $\Delta t_{THR}$ and the encounter probability is about 0.5. Due to the characteristic of the probability estimation formula, Equation (2) simplifies when $|\Delta {\widehat{t}}_{{\gamma }_p,C{P}_{maxP}}^{1,1}|=\Delta t_{THR}$, as shown in Equation (10), as follows:

$$\begin{array}{c}\hfill P(x\le \Delta t_{THR})=\frac{1}{2}\left[erf\left(0\right)+erf\left(\frac{2\Delta t_{THR}}{\sqrt{2}\sigma }\right)\right]=\frac{1}{2}erf\left(\frac{2\Delta t_{THR}}{\sqrt{2}\sigma }\right)\end{array}$$

(10)

As error function is bounded within the range of −1 and 1 and $\Delta t_{THR}$ is positive, the resulting probability is 0.5 at this point in the distribution. Therefore, the transition area is centred around this midpoint; as $|\Delta t_{{\gamma }_p,C{P}_{maxP}}^{1,1}|$ increasingly deviates from $\Delta t_{THR}$, the resulting probabilities asymptotically approach the extremities of zero and one.

In addition, the prediction uncertainty diminishes as ${\gamma }_p$ increases, because the flights are approaching the critical point. Consequently, the SDR distribution progressively approximates ideal behaviour, characterised by a steeper slope.

Regarding the variation by $\Delta t_{THR}$, it is noteworthy that the maximum probability of the SDR escalates with increasing $\Delta t_{THR}$ until it reaches 1. The maximum encounter probability occurs when $|\Delta \widehat{t}|=$ 0 s, where the probability estimation (Equation (2)) simplifies to the following form:

$$\begin{array}{c}\hfill P(x\le \Delta t_{THR})=\frac{1}{2}\left[erf\left(\frac{\Delta t_{THR}}{\sqrt{2}\sigma }\right)+erf\left(\frac{\Delta t_{THR}}{\sqrt{2}\sigma }\right)\right]=erf\left(\frac{\Delta t_{THR}}{\sqrt{2}\sigma }\right)\end{array}$$

(11)

As result, the encounter probability will equal the error function of $\frac{\Delta t_{THR}}{\sqrt{2}\sigma }$. The ideal result will be P = 1 for $|\Delta \widehat{t}|=$ 0 s. Given the characteristics of the error function, the larger $\frac{\Delta tTHR}{\sqrt{2}\sigma }$ is, the closer to one the output of the error function will be. Therefore, the maximum probability achievable is determined by the selected threshold and the performance of the predictor and $\Delta t_{THR}>\sqrt{2}\sigma$ is a necessary condition to approach optimum performance.

To check the condition, the highest ${\sigma }_{res,{\gamma }_p,\gamma }$ of both flows for the first critical point are selected, which correspond to the prediction errors at their corresponding entry point (see

Table 5. Variation of $\sigma$ in both HTPs for different ${\gamma }_p$.

${\mathit{\gamma }}_{\mathit{p}\mathbf{1}},{\mathit{\gamma }}_{\mathit{p}\mathbf{2}}$	${\mathit{f}}_{\mathbf{1}}$ MLP ${\mathit{\sigma }}_{\mathbf{res},{\mathit{\gamma }}_{\mathit{p}\mathbf{1}},\mathit{\gamma }}$	${\mathit{f}}_2$ MLP ${\mathit{\sigma }}_{\mathbf{res},{\mathit{\gamma }}_{\mathit{p}2},\mathit{\gamma }}$	${\mathit{f}}_1$ XGB ${\mathit{\sigma }}_{\mathbf{res},{\mathit{\gamma }}_{\mathit{p}1},\mathit{\gamma }}$	${\mathit{f}}_2$ XGB ${\mathit{\sigma }}_{\mathbf{res},{\mathit{\gamma }}_{\mathit{p}2},\mathit{\gamma }}$	MLP $\mathit{\sigma }$	XGB $\mathit{\sigma }$	MLP $\mathit{\sigma }$–XGB $\mathit{\sigma }$
0, 0	20.78	17.73	21.15	15.56	38.63	37.13	1.50
20, 20	17.35	14.27	17.74	11.47	31.77	29.88	1.89
40, 40	14.06	12.91	14.30	9.69	27.00	24.42	2.57
60, 60	11.48	12.08	11.54	6.64	23.57	18.82	4.74
80, 80	9.07	11.41	9.05	5.34	20.61	14.86	5.75
100, 100	4.18	10.65	3.87	1.64	16.18	6	10.23

). The ${\sigma }_{res,{\gamma }_p,\gamma }$ of MLP-HTPs for F4 and F24 are 20.78 and 17.73 s, respectively, resulting in a $\sigma$ (using Equation (4)) of 27.31 and $\sqrt{2}\sigma \approx 38.63$. On the other side, the ${\sigma }_{res,{\gamma }_p,\gamma }$ of XGB-HTPs are 21.15 and 15.56 s, respectively, resulting in $\sqrt{2}\sigma \approx 37.13$. Thus, the results of both predictors reached a probability of one when the time separation equals 60 s. However, for a $\Delta t_{THR}$ of 10 and 30 s, the maximum probability achievable is less than one, because $\sqrt{2}\sigma >\Delta t_{THR}$, resulting in a distribution with a reduced maximum value.

Although the overall deviation of the predictors is slightly lower in MLP-HTPs for F4 for low ${\gamma }_{p1}$ and ${\gamma }_{p2}$, the low ${\sigma }_{res,{\gamma }_p,\gamma }$ in XGB-HTPs for F24 compensates for the error and leads to a set of lower $\sigma$, as reflected in Table 5. As a result, the XGB-HTPs’ distributions exhibit a more pronounced slope than those of the MLP-HTPs.

The analysis of the SDR graphs is further detailed by examining the parameters of the logistic functions (see Figure 12). The values that lead to a optimum SDR are:

• L: −1. The negative sign indicates the direction of the function.
• $x_0$: $\Delta t_{THR}$.
• k: the higher the better.
• b: 1.

In Figure 12, the optimal values of L, k, and b are achieved in the CD tool for the $\Delta t_{THR}$ values of 60, 90, and 120 s, and they remain constant. This constancy is because they all satisfy the condition of $\Delta t_{THR}>\sqrt{2}\sigma$ for any given ${\gamma }_p$, whereas for $\Delta t_{THR}$ values of 10 s and 30 s, the parameters are far away from optimal. As uncertainty decreases with increasing ${\gamma }_p$, the parameters of these curves approach the optimal. Specifically, for 30 s, the SDR curve meets the optimal values for L, k, and b at −28 NM. For the same reason, the logistic parameters of CD using XGB-HTPs reach the optimal values earlier than those using MLP-HTPs.

As result, the SDR and its associated logistic parameters facilitate setting the performance requirements for the trajectory predictors used in the detection process. By selecting the appropriate logistic parameters, the SDR curve that aligns with the expected CD output distribution and performance (to be analysed in Section 3.2.2) can be constructed, and the key points of the SDR will delineate the required TP accuracy. In the adaptive CD tool, where the dynamic parameters are adjustable during operation, the TP requirements may vary accordingly, to sustain CD performance across various CD configurations. This approach ensures flexibility in selecting the CD configuration while consistently upholding CD performance.

3.2.2. System Tuning Envelope

For the variable STE, the CD performance variations, mainly the accuracy, arising from HTP performance and dynamic thresholds are examined.

The resulting STE curves are presented in Figure 13. Each curve is composed of several points, which represent the proportion of nuisances and missed alerts for a given $P_{THR}$. The two graphs on the first row present the variation of the STE curves by ${\gamma }_p$ for a fixed $\Delta t_{THR}$ = 30 s, and the other two illustrate the variation by $\Delta t_{THR}$ for a fixed ${\gamma }_p$ = −108 NM. In this subsection, the selected $\Delta t_{THR}$ is 30 s to visualise distinct variations, because it is the closest value to the maximum $\sigma$, as presented in Section 3.2.1.

In general, despite the fluctuations in the curves, the STE curves exhibit a clear trend across ${\gamma }_p$, where the proportion of nuisances and missed alerts decreases as ${\gamma }_p$ increases, approaching the critical point. This phenomenon is due to the reduction in the size of the sample of potential conflicts; as ${\gamma }_p$ increases, many flights have already flown over the critical point, and the potential interaction is cleared.

Regarding the HTP-induced variation, XGB-HTPs generally exhibit a higher proportion of FAs, whereas MLP-HTPs show a greater proportion of MAs. This phenomenon, taking into account the smaller standard deviation of XGB-HTPs’ residuals, is caused by the use of the error function ($erf\left(\frac{\Delta t_{THR}+\mu }{\sqrt{2}\sigma }\right)$ and $erf\left(\frac{\Delta t_{THR}-\mu }{\sqrt{2}\sigma }\right)$) in the probability estimation, using Equation (2). The error function is a sigmoid function in nature, whose slope depends on $\sigma$. A lower $\sigma$ in XGB-HTPs causes the function to have a steep slope, leading to a rapid transition to extreme values as the predicted time separation deviates from $\Delta t_{THR}$, resulting in a sharp probability transition and thereby an increase in notified positives and FAs. In contrast, MLP-HTPs, with higher $\sigma$ values, induce a gradual slope in the error function, manifesting a slower probability transition over the predicted separation range and a tendency towards more MAs due to a less pronounced probability change.

The CD performance variation due to the larger disparity of the predictors’ accuracy is assessed by comparing results at different evaluation points. As presented in Figure 13, the FA and MA proportions both decrease with an increase in ${\gamma }_p$. This reduction is caused by two factors. Firstly, a higher ${\gamma }_p$ leads many flights to overfly the critical point, making the interaction impossible and consequently reducing the interaction sample size. Second, the uncertainty diminishes as ${\gamma }_p$ increases, enhancing the accuracy of probability estimations and resulting in higher rates of True Positive (TP) and True Negative (TN). Figure 14 provides two flight pair examples, showcasing significant probability variation over ${\gamma }_p$, converting what initially appear as FAs and MAs into TNs and TAs, respectively. In the blue one, although it is a conflict-free flight pair, whose $\Delta t_{min}$ lies above $\Delta t_{THR}$ = 30 s, the estimated probability at ${\gamma }_p$ = −108 NM is relatively high at 0.61, making it possible to be notified and then classified as an FA. However, as the uncertainty decreases, this probability decreases to zero when ${\gamma }_p$ reaches −28 NM, allowing the tool to accurately identify this case as conflict-free. The second flight pair presents the opposite case: a RP with an initial low estimated probability of 0.2 that escalates to 0.84 at ${\gamma }_p$ = −8 NM, close to the sector boundary of PAU.

Concerning the variation by $\Delta t_{THR}$ at ${\gamma }_p=$−108 NM, the observed trend is inconsistent. The FA and MA proportions initially decrease with increasing $\Delta t_{THR}$ up to 60 s, followed by a slight increase at 90 s and 120 s. The sample sizes and the error function again influence the results. RP and CD notifications are both dependent on $\Delta t_{THR}$; the size of both samples increases with less strict $\Delta t_{THR}$. In terms of the error function, unlike the variation by ${\gamma }_p$, changing the $\Delta t_{THR}$ does not affect the overall steepness of the curve; it changes the point along the existing sigmoid curve. Given the same $\sigma$ and predicted separation, the difference between $\Delta t_{THR}$ and the predicted separation increases with $\Delta t_{THR}$, where the estimated point is likely to be located at the extreme steep regions.

Table 6. Number of flight pairs with $\Delta t_{min}<\Delta t_{THR}$ at ${\gamma }_p$ = −108 NM and $P_{THR}$ = 0.5.

$\Delta {\mathit{t}}_{\mathbf{THR}}$	No. RP	No. FA	No. MA
10	35	0	35
30	92	26	21
60	173	12	15
90	226	21	23
120	325	18	28

shows an example, illustrating changes in analysis sample size with different $\Delta t_{THR}$.

These graphs could assist the user in selecting the most appropriate CD operating point for the operation, depending on LAT, by choosing suitable dynamic thresholds. First, this involves selecting the most appropriate $\Delta t_{THR}$ that exhibits the lowest proportions of FAs and MAs.

Secondly, the selection of $P_{THR}$ determines the notification of predicted alerts, representing the trade-off between FAs and MAs. The dynamic nature of $P_{THR}$ is beneficial during the operation. A low $P_{THR}$ implies a higher proportion of nuisances and fewer missed alerts, while a high $P_{THR}$ implies fewer nuisances and more missed alerts. The trade-off is tailored to the specific roles and tasks of the executive and planner controller, as well as to the look-ahead time.

On one hand, a large LAT allows for the consideration of a low rate of MAs and a reduction in the number of missed alerts. This is considering the greater TP prediction error and more available time for the controller to further evaluate the situation at a large LAT. Consequently, this implies selecting a low $P_{THR}$.

On the other hand, as flights approach the critical point, a higher operating point that prioritises highly probable encounters becomes more appropriate. This is aimed at short LATs, to reduce false alerts by notifying interactions that require immediate actions and thus avoiding overwhelming the controllers.

Hence, the STE facilitates the selection of different operating points for the adaptive CD tool, depending on the corresponding LAT.

CD Stability

The stability of the estimated probability cannot be directly discerned from the previous figures of merit; it requires the examination of specific instances. The analysis, based on the two flight pairs selected in Section 3.1, demonstrates that CD stability is influenced by the stability of the predictions, as depicted in Figure 15. Moderate fluctuations in the predictions result in smooth variation in the estimated probability with ${\gamma }_p$, while large irregularities result in high-frequency fluctuations.

4. Discussion

In this paper, the primary objectives included assessing the impact of trajectory predictor performance on the encounter probability generated by an adaptive conflict detection tool and examining the flexibility of the CD tool dependent on its adjustable thresholds. Two figures of merit were proposed to address these objectives, which also assist in selecting the most suitable trajectory predictors and dynamic CD thresholds. These selections ultimately determine the operating point necessary to meet operational requirements.

For the analysis conducted, trajectory predictors were used to compute the time separation at critical points where minimum separation might be infringed. The conflict detection tool then evaluated the difference between the predicted time and the actual time to compute the encounter probability.

Two simple machine-learning-based horizontal trajectory predictors were employed for each flow of analysis: one trained using extreme gradient boosting, and the other using a multi-layer perceptron regressor. By leveraging these predictors and multiple evaluation points with varying uncertainty, changes in the figures of merit due to the prediction error could be visualised.

Flexibility was attributed to the conflict detection tool through its three adjustable thresholds: the distance threshold, defining the critical point; the time difference threshold, influencing the resulting encounter probability; and the probability threshold, determining the notification of alerts. The latter two thresholds were examined with various values to illustrate their effects on alert notifications and the implications for different operating points.

To assess the impact, predictor performance was quantified using the mean and standard deviation of prediction errors, evaluated at each location and for every subsequent segment. Distinct regions were identified in the metrics’ heatmaps, with locations of splits typically corresponding to ground speed variations.

Furthermore, the relationship between trajectory predictors and the CD tool was demonstrated using two figures of merit: system dynamic range and system tuning envelope. System dynamic range illustrates the variation of encounter probability with predicted time separation. It demonstrates that the rate at which the encounter probability decreases with increasing predicted separation is higher for high-performance predictors, and it declines with the deterioration of TP performance. The distribution of the system dynamic range follows a logistic function, and the analysis of its parameters aids in selecting appropriate horizontal trajectory predictors.

The remaining figure of merit, system tuning envelope, illustrates the trade-off between the proportion of nuisances and missed alerts. It demonstrated that high-performance predictors provide a smaller proportion of nuisances and missed alerts. However, its main utility lies in the selection of operating points, allowing for the fine-tuning of the system to better align with specific requirements for managing the airspace safely. The notifications are mainly impacted by the probability threshold; lower thresholds result in a larger proportion of nuisances and a lower proportion of missed alerts, while higher thresholds enhance the reliability of notified alerts at the expense of increased missed alerts. The trade-off decision rests with the controllers and depends on their specific tasks.

The selection of the time difference threshold depends on the performance of the trajectory predictors. It has been observed that when the ratio of the time separation threshold for conflict detection to the prediction errors exceeds $\sqrt{2}$, the behaviour of the conflict detector converges towards the ideal, following a stepped function.

In conclusion, this study demonstrated that the performance of the trajectory predictor and the conflict detector are intrinsically related, and the figures of merit are useful in selecting appropriate trajectory predictors and operating points for an adaptive conflict detection tool. Additionally, given a specified operating point, these figures of merit facilitate the specification of prediction performance requirements and conflict detection configurations to achieve the desired operational outcomes.

Building upon the findings of this study, the next stages can be divided into three primary blocks. The first block encompasses further research and development to enhance trajectory predictors, aligned with operational requirements that may be elicited from the application of this study. This part can involve extending investigations to vertical predictors, as well as assessing other impacts on the proposed figures of merit, such as flight Intent variations. To achieve this, it may entail the exploration of advanced machine learning algorithms and the integration of additional input sources, such as airborne information through extended projected profiles.

The second block focuses on the further development of the conflict detector, expanding the detection process to encompass not only interactions between flights within the same flow but also those between singular flights that do not belong to any standard flows, and between singular flights.

Following this, the third block will focus on integrating the figures of merit into the decision making process within the existing separation management workflow. The results of this study can contribute to the evolution of human–machine interactions of air traffic controllers, enabling the dynamic selection of operating points of conflict detectors, based on their operational needs. Additionally, incorporating uncertainty management and improving trajectory predictors could enhance the controller’s trust in these tools, thus underpinning the deployment of future concepts such as a reduction of the separation minima, which would bring significant performance benefits [33].