Enhancing UAS Integration in Controlled Traffic Regions Through Reinforcement Learning

Abstract

Controlled Traffic Regions (CTRs) around major airports pose an important challenge to Unmanned Aerial System (UAS) traffic management. Current regulations highly restrict UAS missions in these areas by confining them to segregated areas. This paper makes a proposal to allow more ambitious UAS missions inside CTRs, such as paths across the CTR or between heliports inside the CTR, based on self-separation. This proposal faces two important problems: on the one hand, the adaptive response to the dynamic airspace reconfiguration of a CTR without necessarily terminating the flight, and on the other, a self-managed conflict resolution that allows maintaining traffic separations without the intervention of air traffic controllers. This paper proposes a solution named Reinforcement Learning Multi-Agent Separation Management (RL-MASM). It employs a multi-agent reinforcement learning system with a fully decentralized decision-making scheme, although it uses a common information source of the environment. The proposed system is evaluated against classical control algorithms for obstacle avoidance to determine the potential benefits of AI-based methods. Results show that AI-based methods can benefit from knowing the intent of a UAS. This leads to increased performance in intrusions into no-fly zones or collisions, and also solves some challenging scenarios for classical control algorithms. From the aeronautical point of view, the proposed solution also introduces important advantages in terms of efficiency, scalability, and decentralization.

1. Introduction

Unmanned Aerial System (UAS) operations inside a Controlled Traffic Region (CTR) are of high interest, since these regions include most of the urban areas of big cities where UAS are intended to be used. A CTR is a controlled zone around an airport which usually has a high air-traffic density and includes different types of traffic: commercial flights flying under Instrument Flight Rules (IFR) [1], general aviation (small aircraft, helicopters, etc.) flying under Visual Flight Rules (VFR), and now UAS and some other types of advanced air mobility need also to be included. That makes a CTR a specially complex airspace where different types of traffic fly under different flight rules.

The safe integration of UAS traffic into different types of airspaces, especially CTRs, is one of the big challenges for the growth of unmanned air traffic. This problem must be addressed from several perspectives: legislative, technological, and organizational [2,3]. The permissiveness of the legislative perspective is usually in line with the degree of development achieved from the other two perspectives: technological, and organizational. For example, most European Union state members’ legislation [4] is quite restrictive and only allows UAS traffic in confined areas currently. That implies constraining UAS operations, such as intracity transportation or inspection missions inside a CTR.

Both EUROCONTROL and the Federal Aviation Administration (FAA) have addressed the problem of integrating UAS into the airspace in a safe and standardized manner through the ConOps framework [5,6]. This framework defines key concepts like UAS types and categories, their operational environments, Unmanned Traffic Management (UTM), and their integration with Air Traffic Management (ATM). However, one of the most interesting aspects is the use of U-Space services. They provide a set of services and technologies that enable drones to fly beyond the visual line of sight (BVLOS). U-space services are intended to be available through a real-time communication network. Among the services that will be available, this paper assumes the presence of a U-space monitoring service to deliver traffic information in a given airspace. This service will rely on technological facilities based on the “electronic conspicuity” concept. Based on that, this work contributes to providing support for some other U-space services, specifically, No-Fly Zones (NFZs) and UTM. NFZs are restricted areas where UASs cannot penetrate. In our case, these areas are only allowed for manned aviation. This restriction will be enforced through facilities like “geo-fencing”. On the other hand, UTM will ensure that drones can detect and avoid other aircraft, manage congestion, and maintain safe separation from other drones and manned aviation.

The Dynamic Airspace Reconfiguration (DAR) [7] concept proposed by U-space is the basis for a first approach to deal with integrated operations. DAR refers to the process of adjusting or modifying the structure and boundaries of airspace to optimize its use based on particular factors, like, for example, current traffic demands or weather conditions. In our case, DAR is used to define a number of NFZ into a CTR where UAS operation is restricted, so only manned aviation is allowed. These NFZs can be dynamically redefined to maintain safety based on traffic, weather or contingencies, and to provide UASs the maximum operational level. The proposal of this work provides support for DARs.

The DAR concept can be implemented using obstacle avoidance and collision avoidance techniques. These techniques have been extensively studied in robotics [8,9]. The Conflict Resolution (CR) strategies proposed in this paper are based on a field of Artificial Intelligence (AI) called Reinforcement Learning (RL) and, more precisely, on a sub-field of RL known as Multi-Agent Reinforcement Learning (MARL). In MARL, multiple agents interact in the same environment. Several MARL structures are possible depending on whether the relationship between the agents is cooperative, competitive, or mixed. Structures also depend on whether the agents can communicate with each other or not.

The main advantage of using MARL over control algorithms is their adaptability to complex and dynamic environments. They are designed to balance exploration (trying new actions to discover better strategies) and exploitation (relying on known strategies that have worked well in the past). This makes RL suitable for environments where optimal actions are not always clear and need to be discovered over time.

Once manned aircraft and UAS traffic have been separated using DAR, the other issue is how to keep unmanned traffic separated. Several CR strategies have been proposed in previous research [10,11,12]. They range from fully centralized to fully decentralized. This work advocates a decentralized strategy based on self-separation, similar to that proposed for VFR separation. Centralized schemes are generally more efficient, but suffer from reduced reliability when faced with uncertainty and communication failures, and tend to require higher amounts of computing, leading to scaling issues. The proposed approach avoids providing a centralized unit and enables each drone to execute the CR process independently, enhancing responsiveness.

The problem of CR among multiple autonomous mobile agents has been addressed using AI. Solutions can be broadly classified into pair-wise CR and Multi-Aircraft Conflict Resolution (MACR). However, using some heuristics, MACR can be transformed into a set of pair-wise CRs. One proposal for this transformation is to process the conflicts hierarchically, computing the envelope and restricting it in further conflicts [13]. Solutions can also be classified according to the decentralization level. The work of [14] uses neural network architectures that employ inter-agent communication for decision making. This scheme either requires being executed in a centralized way and resolutions broadcast to all agents, or using a highly reliable peer-to-peer communication facility between UASs, which seems unrealistic to be implemented in a U-space service. On the other hand, decentralized CR algorithms are based on defining an observation space with a reduced number of intruders. Delimiting this set is usually based on heuristics [15]. This technique has been proven to be effective in manned aviation, but determining this observation space becomes more complex as traffic density increases and traffic with highly different dynamics coexist in the airspace.

The main contribution of this work is to propose and to validate a scheme for UAS operation in a CTR based on AI techniques. This technique allows overcoming the current restrictions of UAS to segregated areas. The work also contributes to the definition of a simulation framework for drones in AI. An important issue in this definition is that the proposed architecture enables the use of environments with a variable number of drones, allowing the definition of policies and observation spaces that are not constrained to a predetermined number of agents. This work also defines a number of key scenarios where collision avoidance techniques should be validated and compares a reference method used in classical control for routing and collision avoidance with our proposed AI-based algorithm, detailing their performance and advantages.

The proposal to separate UAS from manned aviation is based on the DAR concept. It uses self-separation for separating UASs among them. The implementation relies on a U-space service which is a central facility that provides traffic information and it also assumes that UASs are equipped with some device that provides electronic conspicuity to implement this U-space service. It also assumes that every UAS has submitted a flight-plan. The solution provides the optimal path for every UAS in a completely decentralized way. This solution dynamically avoids conflicts with DARs and other UASs.

This paper is organized as follows: Section 2 discusses the problem of organizing air traffic into a CTR and our proposal to integrate UASs in such a scenario. Section 3 describes the modeling of the components and the environment to be implemented as a MARL algorithm. Section 4 details the formulation of the problem in terms of the RL agent. Section 5 deals with the training process of the RL agents. Section 7 is the major contribution of this paper and validates the results through several critical scenarios. It also compares the performance of the AI approach to traditional control techniques. Lastly, Section 8 contains the conclusions and consideration of future work.

2. Introducing UAS Traffic into CTRs

Figure 1 shows an example of a CTR of a Spanish airport (LEVC, Valencia) showing the routes followed by different traffic types and how the urban area of the biggest city is totally included in it. Additionally, it also includes a large number of small cities in the metropolitan area. The CTR vertically expands from the terrain to 6000 ft. Commercial and military traffic flying under IFR take-off and land along routes converging on the runway axis. General aviation flying under VFR fly through corridors to join the Aerodrome (AD) traffic pattern in the approaches. Since the CTR is class-D, the airport tower performs positive control over IFR traffic and clears VFR traffic to enter the CTR, although VFR traffic is responsible for self-separating from the rest of the traffic. Airports with a high traffic density (as many have in the USA) are class-B, so the tower is responsible for separating both IFR and VFR traffic.

In this context, new types of traffic related to Urban Air Mobility (UAM) and especially UAS need to be introduced. Some UAS missions can be enclosed inside a well defined and small region. These types of missions can be easily managed using the corresponding permits to temporarily segregate an airspace that is not flown by manned aviation. This is the current situation. However, we want to introduce missions where UASs fly between different points scattered around the CTR, like hospitals, small heliports, cities, etc., where UAS would fly a route defined by a mission plan. The mission plans would go through a deconfliction process in the strategic phase (pre-flight), which is beyond the scope of this paper. But conflicts will also arise in the tactical phase due to uncertainty and the dynamic nature of the environment. As discussed in [16], an airspace reconfiguration can cause strategically deconflicted flight plans to no longer be conflict-free, so routes need to be modified to preserve the separations.

The proposed solution is based on separating UASs from manned aviation (IFR or VFR) using NFZ where only manned aviation can fly and dynamically reconfiguring them when necessary according to the DAR concept. Figure 1 shows these protected areas with well-defined horizontal and vertical limits. There might be some static areas that are always activated, but most of them are dynamically activated when necessary. The separation of one UAS from other UASs would be based on self-separation.

The implementation of the proposed scheme is based on MARL.

This is a solution based on self-separation, with fully decentralized decision-making. A UAS is modeled as an RL agent. RL is a type of machine learning where an agent learns to make decisions by interacting with an environment. The goal is for the agent to maximize a reward signal by taking actions that lead to the best outcomes over time. This way agents can learn the “optimal” strategy in a highly dynamic environment.

However, some central information source that provides information about the environment is assumed. An alternative to a centralized information source would be one in which this information is supplemented, or even fully replaced, by readings of onboard sensors providing information about surrounding traffic. These are usually referred to as DAA (Detect and Avoid) devices, but they are not required in our system. The observation space consists of environment information, encoded as an image, supplemented with a vector containing the kinematic state of the surrounding UASs. The environment image provides the following information: the layout of the NFZs, the last known position of the surrounding traffic, their protecting volumes (separation minima), and their intents in the form of mission plans. This centralized information source could be realistically implemented using U-space services like the Network Identification Service (NIS), which allows obtaining information about nearby aircraft, and the Flight Authorisation Service (FAS), that allows submission of mission plan information to UTM.

Figure 2 shows the role of AI in this problem. In the example, a UAS is trying to get to the next waypoint in the mission plan when an NFZ, corresponding to a CTR corridor, is activated through DAR. Two extreme strategies are possible to reach the target: holding until the NFZ is hopefully deactivated and can be reached following a straight route, or avoiding the obstacle, contouring its perimeter following a long trajectory. The adopted strategy will obviously depend on the goals that the reward function tries to optimize (distance, time, etc.). But even with a given reward function, it will also depend on the environment: the time that the NFZ remains activated or the size of the NFZ. That is something that the RL agent should learn through training, so the design of the agents’ training must also be carefully designed to optimize its behavior.

3. Simulation Environment

The Reinforcement Learning Multi-Agent Separation Management (RL-MASM) algorithm requires a simulation environment so the agents may interact with it and gather experiences to learn the optimal policy. This simulation environment is divided into four main blocks: the agents, the airspace, the separation model, and the mission plans. In this section, each block is formally defined.

3.1. Agent Modeling

The multi-agent separation management problem presented in this work is formally defined as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) [17]. Dec-POMDP is a model for decision-making across multiple agents in which each agent receives a partial observation of the current state of the environment and chooses an action accordingly. Dec-POMDP is a generalization of the Markov Decision Process (MDP).

Formally, Dec-POMDPs are defined by the tuple $\langle \mathcal{N},\mathcal{S},\mathbb{A},T,\mathbb{O},O,R,\gamma \rangle$ [18], where $\mathbb{N}$ is the set of n agents, $\mathcal{S}$ is the set of environment states, $\mathbb{A}$ is the set of joint actions, T is the transition probability function, $\mathbb{O}$ is the set of joint observations with O being the observation probability function, R is the reward function, and $\gamma$ is the discount function.

The agents use a stochastic policy ($\pi$) to map their local observation to a distribution over their action space $\pi :{\mathbb{O}}_i\times {\mathbb{A}}_i\to [0,1]$. Through training, this policy is tuned to maximize the discounted expected cumulative reward $J\left(\pi \right)$.

$$J(\pi )=E[\sum _{t=0}^{\infty }{\gamma }^{t}R(s^t,a^t)|\pi ]$$

(1)

RL techniques are widely used to find the optimal policy in MDPs environments. Our solution presented is based on the Proximal Policy Optimization (PPO) algorithm [19], which has been proven to work in cooperative environments with homogeneous agents [20].

The agent requires a model of the drone to simulate the interactions with the environment. The state s of the drone i at time t is represented in Equation (2). It is formed by the drone’s position ($x_i,y_i$), its speed ($v_i$), and heading (${\chi }_i$).

$$s_i^t={\left[\begin{array}{c}{x}_i\\ y_i\\ v_i\\ {\chi }_i\end{array}\right]}^t$$

(2)

Equations (3) and (4) define the state transition for an action composed of a heading and speed change ($\Delta {\chi }_i,\Delta v_i$).

$${\left[\begin{array}{c}{v}_i\\ {\chi }_i\end{array}\right]}^{t+1}=\left[\begin{array}{c}max\left(v_{min},min\left(v_{max},v_i^t+\Delta v_i^t\right)\right)\\ ({\chi }_i^t+\Delta {\chi }_i^t)mod360\end{array}\right]$$

(3)

$${\left[\begin{array}{c}{x}_i\\ y_i\end{array}\right]}^{t+1}={\left[\begin{array}{c}{x}_i\\ y_i\end{array}\right]}^t+{\left[\begin{array}{c}cos\left({\chi }_i\right)v_i\\ sin\left({\chi }_i\right)v_i\end{array}\right]}^{t+1}$$

(4)

All agents are homogeneous and share the same performance limits, with

Table 1. Drone performance parameters.

Parameter	Description	Value
$\Delta v_{max}$	Acceleration	1 m/${\mathrm{s}}^2$
$\Delta {\chi }_{max}$	Track change rate	17 $°$/s
$v_{min}$	Minimum speed	0 m/s
$v_{nom}$	Nominal speed	10 m/s
$v_{max}$	Maximum speed	15 m/s

showing the main parameters for the drone model. The values were derived from the typical drone outlined in [21].

This implementation allows drones to hover. Future work may consider incorporating drones with different performance models in the same environment, including both multi-copter and fixed-wing drones.

The drones start each episode stationary, with their position defined by the first waypoint on their mission plan as described in Section 3.4.

3.2. Airspace Modeling

The actions of the Air Traffic Controls (ATCs) are simulated by dynamically segregating areas of the airspace, defined by independently activated NFZs. Drones must exit and avoid entering any activated NFZ. In this paper, several different models of the airspace have been defined for training and evaluation of the system.

In order to encourage generalization to any airspace configuration, the agents are trained on airspaces with randomly generated NFZs. $n_{obstacle}$ NFZs are created and randomly located. Overlap between two or more NFZs is allowed. Each NFZ is a regular polygon with n sides, with $n\in \mathbb{N}:3\le n\le ob{s}_{sides}$, and a circumradius of $r\in \mathbb{R}:0.5\ast obs\_siz{e}_{max}\le n\le obs\_siz{e}_{max}$.

This process is repeated at the start of each training episode. The airspace models used in the evaluation are detailed in Section 7.

3.3. Separation Modeling

The safe separation modeling follows the work presented in [22]. The authors present a solution that defines a loss of separation as the intersection of the separation threshold volumes between two drones, with different separation thresholds depending on several factors, such as the drone category, the current traffic density, and the performance of the Communications, Navigation, and Surveillance (CNS) infrastructure. The separation threshold distances used in this article were sourced from a similar urban use case in the BUBBLES project validation exercises, concretely, S2E2 [23], and are present in

Table 2. Horizontal separation distances.

Category	Threshold (m)	Category	Threshold (m)
A1	102.00	SAIL III–IV	263.64
A2	102.00	SAIL V–VI	281.90
A3	198.36	Certified (no passenger)	471.01
SAIL I–II	234.20	Certified (passenger)	640.19

.

Drones are randomly given a category at the start of each episode.

3.4. Mission Plan Creation

Mission plans represent the intent of each agent, providing a set of targets or waypoints that the drone has to reach. The mission plans are important as they provide a desired trajectory that the agents must optimize and also inform the intent of the surrounding agents. Having the intent of the rest of the traffic allows the agents to plan ahead and avoid future conflicts.

Random mission plans start on a random point, and subsequent waypoints are created within $W{P}_{maxDist}$ of the previous waypoint. To ensure that the agents can finish the mission if an ATC segregates an area, the intermediate waypoints are only created outside of segregable areas. The first and last waypoints of the mission plan are the take-off and landing sites. Mission plans are completed when the drone reaches each waypoint on its mission plan in order.

4. Reinforcement Learning Framework

In this section, we illustrate the different system components to solve the environment described in the previous Section.

RL-MASM is agnostic to the total number of participants in the airspace, enabling decentralized decision-making that takes into account all relevant hazards and acts accordingly.

4.1. Observation Space

The observation space for the agents was designed with two objectives in mind. First, the observation space must convey enough information to the agent so that it is able to learn the optimal policy. Second, the observation space must be compatible with an indefinite number of agents, so that it may be deployed in environments with varying numbers of participants.

To fulfil the previous restrictions, a hybrid observation space was defined. This hybrid observation space is composed by an image that contains information about the current airspace surrounding the agent and a vector with information about the current state of the agent and its target. This is defined as $o^t=[o_a^t,o_s^t]$, with $o_a^t$ being the airspace image and $o_s^t$ being the state vector observation.

The $o_a^t$ observation is a three-channel, 256 by 256 pixel square image that represents a 5 ${\mathrm{km}}^2$ area surrounding the agent, resulting in a resolution of 19.5 m/pxl. This image contains the next waypoints in the mission plan, the active NFZs in the area and the position, speed, intent, and separation minima of other drones.

Taking into account the state and intents of surrounding agents allows the agent to optimize its trajectory, reducing conflicts and distance traveled. Figure 3 represents the image observation received by an agent, showing itself, an NFZ and another drone. The image is centered in the current agent’s position and is aligned in the current direction of flight. The intent of the current agent appears as a green line towards the next waypoints. The separation minima of both drones is shown as a blue area and the intent of the other drones is shown with fading blue lines. The red areas represent active NFZs.

This representation of the airspace offers an observation space that is not dependent on the number of participants, allowing for an indefinite number of participants in the airspace, as well as representing NFZs of indefinite shapes. Previous approaches only allow for a set number of intruders and use heuristics to extract the most relevant actors to use in the decision-making step [15].

The area covered by the image has been adapted to the performance and separation minima used in this article and may require increases for faster drones.

Information about the current state of the drone is also provided in the $o_s^t$. Equation (5) defines this observation:

$$o_s^t=\left[\begin{array}{c}dis{t}_{target}/en{v}_{size}\\ cos({\chi }_i-{\psi }_i)\\ sin({\chi }_i-{\psi }_i)\\ v/v_{max}\end{array}\right]$$

(5)

with $dis{t}_{target}/en{v}_{size}$ being the normalized distance to the next waypoint, ${\chi }_i-{\psi }_i$ the relative heading to the next waypoint, and $v/v_{max}$ the normalized velocity.

4.2. Action Space

As the environment proposed in this paper is limited to a 2D space, the action space is limited to the acceleration and turn rate. The action is formulated as $a_i\left(t\right)=[a_T,a_{\chi }]:a_T,a_{\chi }\in \mathbb{R},-1\le a\le 1$. $a_T$ acts over the acceleration and $a_{\chi }$ over the turn rate.

The acceleration and turn rate change for the state update are obtained by multiplying the action with the performance parameters of the drone, as in Equation (6)

$${\left[\begin{array}{c}\Delta v_i\\ \Delta {\chi }_i\end{array}\right]}^t=a_i^t\left[\begin{array}{c}\Delta v_{max}\\ \Delta {\chi }_{max}\end{array}\right]$$

(6)

4.3. Reward Function

We developed a multi-objective reward function to describe the desired behavior of the agents. The reward function for the agent i is formulated in Equation (7). The reward is divided into the following sub-components: the reward for progressing on the mission plan ($r_i^{target}$), the avoidance reward ($r_i^{conflict}$), the nominal speed reward ($r_i^{vel}$), and the efficiency reward ($r_i^{eff}$).

$$r_i=r_i^{target}+r_i^{conflict}+r_i^{vel}+r_i^{eff}$$

(7)

The reward calculation uses the weights $W_{target},W_{conf},W_{vel},W_{thust}$ and $W_{\chi }$. The weights can be tuned to give priority to an objective over the others.

Table 3. Environment parameters.

Parameter	Description	Value
$n_{agents}$	Number of agents	12
$ma{x}_{steps}$	Step limit per episode	1750
$en{v}_{size}$	Environment size	25 km
$n_{obstacle}$	Number of NFZs	15
$obs\_side{s}_{max}$	Maximum number of sides of an NFZ	6
$obs\_siz{e}_{max}$	Maximum size of random NFZs	4 km
$W{P}_{maxDist}$	Maximum distance between waypoints	7.5 km
$W_{target}$	Weight for target reward	0.75
$W_{conf}$	Weight for conflict avoidance reward	−1.1
$W_{vel}$	Weight for nominal speed reward	−0.15
$W_{thust}$	Weight for efficiency reward for speed changes	−0.05
$W_{\chi }$	Weight for efficiency reward for track changes	−0.05

contains the weights used in the experiments.

The reward for progressing on the mission plan is defined in Equation (8). It is based on the change in distance to the next waypoint during the last step.

$$r_i^{target}=W_{target}{\displaystyle \frac{dis{t}_{i-target}^t-dis{t}_{i-target}^{t+1}}{v_{nom}}}$$

(8)

If the agent reaches a waypoint at the end of the step, $r_i^{target}$ is set to 1.

The avoidance reward ensures that the agents remain sufficiently separated during the episode. This reward penalises the agent for getting closer to another agent or NFZ. The discount function $f_d\left(dist\right)$ is used to give more importance to obstacles closer to the current drone.

The resulting $r_i^{conflict}$ is presented below:

$$\begin{array}{cc}\hfill r_{c}onflict& =W_{conf}\times \hfill \\ \hfill & {\displaystyle \sum _{j\in \mathcal{N}\backslash \left\{i\right\}}}{r}_f{f}_d\left(dis{t}_{i-j}^{t+1}\right){\displaystyle \frac{dis{t}_{i-j}^t-dis{t}_{i-j}^{t+1}}{v_{max}}}\hfill \end{array}$$

(9)

$$r_f=\left\{\begin{array}{cc}1\hfill & \mathrm{if}dis{t}_{i-j}^t-dis{t}_{i-j}^{t+1}>0\hfill \\ 0.5\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(10)

The distance for each obstacle is calculated ($dis{t}_{i-j}$) differently for drones or NFZs. For drones, the distance is measured from the exterior of their separation thresholds. For NFZs, the distance is measured between the center of the drone and the exterior of the NFZ. Figure 4a,b display examples of how this distance is calculated.

The reward is reduced by the factor $r_f$ if the agent is moving away from the obstacle j, ensuring that an agent that approaches and afterwards moves away from it has a lower cumulative reward than another that does not approach the obstacle.

And the discount function $f_d\left(dist\right)$ reduces the conflict penalty exponentially with the distance:

$$f_d\left(dist\right)=\left(1-{\displaystyle \frac{1}{1+e^{\alpha (-dist+\beta )}}}\right)$$

(11)

This tunable logistic function is bounded to $[0,1]$. We set the logistic growth rate $\alpha$ at 0.01 and the midpoint $\beta$ at 500.

The reward function has to be compatible with states in which there is an intrusion or loss of separation, as these events may occur during training and evaluation. The path of the highest reward in these cases is the shortest to the resolution of the conflict. To further penalize these cases, $W_{conf}$ is added to $r_{c}onflict$ once per instance of separation loss or intrusion until they are resolved.

The drones are encouraged to fly at nominal speeds by $r^{vel}$:

$$r_i^{vel}=W_{vel}{\displaystyle \frac{|v_i^t-v_{nom}|}{v_{nom}}}$$

(12)

The efficiency reward discourages changes to the current velocity or direction, preventing oscillations and promoting consistent trajectories.

$$r_i^{eff}=W_{thust}\left|a_T^t\right|+W_{\chi }\left|a_{\chi }^t\right|$$

(13)

4.4. Neural Network

The PPO algorithm uses an actor network that outputs the policy distribution and a critic network that estimates the value of the current state. In our case, both use the same central architecture. We have used a hybrid architecture to accommodate the observations ($o_a^t,o_s^t$). First, we implemented a custom feature extractor to obtain features from the image of the airspace. This feature extractor uses a series of two-dimensional convolutional layers (conv2D) with max pooling (MaxPool2d). Then, the drone state observation ($o_s^t$) is concatenated to the output of the last layer and three subsequent fully connected layers of 256, 128, and 64 units with ReLU activation. Similar architectures have been used in the literature to process hybrid observations [24].

The action $a_i^t$ is sampled from the distribution generated by the actor network, and the critic network outputs the expected value of a given state.

Figure 5 summarizes the network architecture for the actor and critic.

5. RL-MASM Algorithm Training

With the environment and framework described in previous sections, the RL-MASM was trained using the PPO algorithm [19,25]. During training, a single policy is trained with experiences from all agents. Later, the agents are independently deployed during evaluation, following the Centralized Training Decentralized Execution paradigm.

RL-MASM was trained for 4.5 million total steps with 12 agents per environment. The parameters for the training environment are detailed in

Table 4. Training hyperparameters.

Hyperparameter	Value
Learning rate	$3\times {10}^{-4}$
Number of training steps	4,500,000
Batch size	512
Rollout buffer	9216
Discount factor	0.99
Generalized Advantage Estimator (GAE) parameter	0.95

. Episodes are terminated if all agents have finished their missions or the maximum number of steps ($ma{x}_{steps}$) is reached.

Each environment step corresponds to 2 s of simulated time.

The starting point for the reward weights was defined analytically and then refined through successive training iterations. The weights were selected to ensure that the reward values remained within the same order of magnitude throughout training, which helped minimize instability. Furthermore, in scenarios involving loss of separation, the reward for avoiding the other drone consistently outweighs the combined rewards of all other objectives.

In Section 6, we evaluate the effect each parameter has on the behavior of the system.

In Figure 6, we show the total reward, intrusions, and losses of separation per episode gathered during training. There is an increase in the total reward per episode as the policy is trained, suggesting the policy is learning from the interactions with the environment. The losses of separation and intrusions reduce over time as well. These metrics show the number of simulation steps that a drone was within the separation threshold of other drones and within an NFZ, respectively.

An important point is that the initial conditions of the environment may lead to inevitable intrusions and losses of separation, independently of the performance of the agent. For example, agents may start the episode within an NFZ or in very close proximity to another agent. This forces the agents to learn how to solve these situations and, as the training progresses, their numbers are reduced until a baseline level is reached.

6. Reward Function Sensitivity Analysis

In this section, we evaluate the sensitivity of the system to the parameters in the reward function ($W_{target}, W_{conf}, W_{vel}, W_{thrust},\mathrm{and}{W}_{\chi }$). We performed 20 Monte Carlo training runs varying the values of the reward weights. The values for each run were sampled from a uniform distribution with the ranges shown in

Table 5. Weight ranges.

Parameter	Range
$W_{target}$	0.5	1
$W_{conf}$	−0.25	−1.5
$W_{vel}$	−0.25	0
$W_{thrust}$	−0.25	0
$W_{\chi }$	−0.25	0

. The range of values were defined analytically.

Figure 7 shows the linear correlation between the changes in the different parameters and the total distance traveled by all agents, the total number of separation losses, and the success rate. The success rate is defined as the ratio of episodes where all agents complete their flight plans.

The results indicate that, as expected, there is a trade-off between $W_{target}$ and $W_{conf}$ in distance and losses of separation. Giving more weight to the separation in the reward function leads to safer missions, but it negatively impacts the total distance needed and success rate as agents need to deviate from their mission plan to avoid other drones and NFZs.

The impact of the rest of the parameters is negative in both metrics, as the behavior that they enforce is not represented in the metrics obtained. For the $W_{thrust}$ and $W_{\chi }$ parameters, the results analyzed without those terms show erratic flights with changes in speed and heading. In the case of $W_{vel}$, the agents tend to hover or fly at their maximum speed. Although these behaviors perform well according to the presented metrics, they are undesirable for operators as they affect the overall flight efficiency and increase the aircraft’s maintenance cost.

The reward weights used in the evaluation are present in Table 3.

7. Model Validation

Once the policy had been trained in a random environment, the policy was evaluated in a series of scenarios created to assess the performance of the separation management system. One of the goals of this study is to assess the performance of AI-based methods against classical control algorithms. For that reason, RL-MASM was compared to the classical control method Dynamic Window Approach (DWA) [26]. DWA is a non-communicating obstacle avoidance algorithm used extensively for mobile robot routing [27]. DWA offers great computational efficiency and robustness to changes in configuration parameters while not requiring explicit communication between drones.

Simulations with a basic agent that ignores NFZs and other participants were used as the baseline algorithm.

The algorithms were validated in the following environments:

• Random Obstacle: This environment is similar to the training environment; however, for the evaluation, the NFZs may activate and deactivate dynamically as the episode progresses.
• LEVC CTR with DAR: uses the Valencia CTR as defined in Airac 2310 [16]. This environment was chosen because it was well studied in the AURA project [16] as an example of UAS integration with ATM through DAR and offers interesting features for routing.
• Converging Pattern: In this environment, the drones are distributed radially and have to traverse to the opposite side. It is a challenge for the conflict avoidance performance of the algorithms, as all mission plans intersect at the center of the environment. This environment has no NFZs.
• Narrow Passage: In this environment, the agents must cross a narrow passage formed by two NZFs that remain active for the duration of the episode. It simulates a constrained navigation scenario with high congestion.

Figure 8 shows an example of each environment. In the figures, the NFZs are represented as golden shapes; each drone is represented by a triangle, with a blue circle around it, showing its separation threshold. The trajectory of each drone is shown in the same color as the drone, with its mission plan displayed as a line and the waypoints as squares. A bigger square is used to represent the next waypoint for each drone. In the CTR environment, the LEVC CTR profile is added for visualization purposes.

The validation is performed on 15 instances of each environment with the parameters defined in

Table 6. Validation environment parameters.

Parameter	CTR with DAR	Random	Converging	Narrow
$n_{agents}$	15	15	10	8
$ma{x}_{steps}$	2500
$en{v}_{size}$	30 km	30 km	30 km	25 km

. The parameters used for DWA are in

Table 7. DWA parameters.

Parameter	Value
Simulation time	10 s
Time step	1 s
Velocity space samples	7
Heading space samples	7
Weight path	0.75
Weight separation/intrusion	1

.

For each episode, the NFZs of the Random and CTR environments start with a 50% chance of being active and are periodically toggled with a randomized period $T_{act}\in \mathbb{R}:250\le T_{act}\le 750$ s. A random period is sampled per NFZ per episode.

Several metrics were collected to evaluate the performance of the algorithms. The metrics are as follows: (a) “Success rate”, the ratio of episodes in which all agents complete their missions, reaching all the waypoints in the plan; (b) Total distance traveled; (c) Episode length; the number of simulation steps intruding a NFZ; and (d) Total steps with separation loss.

Table 8. Performance metrics of the algorithms with the different environment types. Mean/$\sigma$ are shown for distance and length. Best result per environment highlighted in bold.

Environment	Algorithm	Success Rate	Distance (km)	Length (Steps)
CTR with DAR	RL-MASM	93%	455.67/7.11	1683.60/329.03
	DWA	87%	366.76/6.62	2121.13/229.89
	Baseline		299.22/5.4	774.87/21.41
Random	RL-MASM	87%	460.58/11.01	1800.47/407.16
	DWA	73%	370.13/7.36	2178.60/251.95
	Baseline		297.31/5.83	767.53/30.90
Converging	RL-MASM	100%	165.88/6.82	1012.87/71.79
	DWA	87%	180.69/8.81	2046.93/248.13
	Baseline		127.158/3.35	700.07/46.82
Narrow	RL-MASM	100%	125.80/4.53	964.67/50.59
	DWA	67%	133.23/9.30	1966.73/385.82
	Baseline		103.51/3.63	636.60/30.02

shows the results of the validation with the DWA and the proposed algorithm, as well as the basic algorithm used as a baseline. The table shows the mean distance traversed by all the agents in each episode and the number of simulation steps required by the agents to achieve their mission. If the maximum number of simulation steps is reached, the episode is stopped and marked as unsuccessful.

Episodes finish if all agents successfully finish their missions or if the maximum number of steps ($ma{x}_{steps}$) is reached.

The results show that RL-MASM is able to solve the environment at a higher rate than DWA, with a lower mean episode length. The DWA algorithm tends to deadlock with high traffic densities and with convex NFZ shapes. Such a situation can be observed in Figure 9a, which shows the DWA algorithm in the narrow passage environment. Due to the sizes of the drone’s separation thresholds and NFZs, none of the drones can progress without loss of separation and enter a deadlock state, hindering the successful completion of the episode. In environments where the NFZs may deactivate, this effect leads to a lower total traveled distance, but an increased episode length. This is especially significant in the CTR environment. In turn, the proposed algorithm can observe the NFZ and the intent of other agents, leveraging this information to optimize the path and complete the assigned mission, as can be seen in Figure 9b.

The Converging Pattern environment challenges the agents to plan ahead to avoid conflicts with other drones. The DWA method fails to complete some instances on time as the drones become surrounded in the center and find themselves in similar deadlocks as in the Narrow environment, increasing the time required to solve the episode dramatically. The proposed algorithm is able to complete the missions within a lower timeframe while avoiding most of the conflicts, thus reducing the mean steps with losses of separation by 93% versus no separation management. Figure 10a shows that the trajectories followed by the agents solve the scenario successfully, with some agents following the shortest path and others preemptively routing around the center to avoid conflicts.

Figure 10b shows the cumulative trajectories of the agents in the Narrow Passage environment. Although there are a small number of intrusions on the NFZs, agents are able to adapt their trajectories to the incoming traffic, often circling around to allow other drones to pass, avoiding potential deadlocks. The number of intrusions and losses of separation in this environment shows that further improvement to the methods is available, especially in situations with high traffic density.

As an example of the CTR environment, Figure 11a shows the initial configuration, and Figure 11b displays the end of the episode as the agents are reaching their last waypoints, with the tracks flown by each drone. Throughout the episode, the CTR NFZs have been toggled with random periods. This shows the agents preemptively modifying their trajectories to avoid potential losses of separation and exiting activated NFZs as soon as possible.

The proposed algorithm also outperforms DWA in the number of intrusions at the cost of a higher number of separation losses in the NFZ. The results in Figure 12b show a marked improvement over the baseline in the number of separation losses, which is especially notable considering the density of the airspace.

8. Conclusions

This work presents RL-MASM, a novel framework for integrating heterogeneous traffic on a CTR based on multi-agent reinforcement learning, with a particular focus on the dynamic airspace reconfiguration scheme proposed by SESAR. This solution assumes a high level of autonomy of the participants, requiring minimum effort from ATC to maintain separation.

RL-MASM implements an observation space that represents the current airspace and nearby traffic, without constraints on the number of participants or predefined NFZ geometries. It was validated across diverse scenarios involving various airspace structures and traffic densities, including a recreation of the Valencia CTR, with a higher success rate and improved separation management when compared to DWA.

While current evaluations assume ideal communication conditions, real-world operations face latency, data loss, and sensor inaccuracies. Future work will address these challenges and explore vertical separation strategies to further enhance robustness.

As urban air mobility continues to expand, intelligent airspace management systems like RL-MASM will be critical to ensuring the safe and scalable integration of UAS worldwide.