Autonomous Cooperative Drone Swarms for Countering Drones via Multi-Agent Deep Reinforcement Learning ^†

Abstract

The integration of artificial intelligence (AI), particularly deep reinforcement learning (DRL), promises to enhance the autonomy and adaptability of drones in complex environments. This research explores the implementation of a cooperative counter-drone swarm solution using multi-agent DRL, such as Multi-Agent Proximal Policy Optimization (MAPPO), and the aim is to enhance public security. In this paper, an open-source simulation platform, AirSim, is utilized to train and test the proposed method. A centralized critic architecture within a multi-agent reinforcement learning (MARL) framework using Proximal Policy Optimization (PPO) is implemented. A PettingZoo–Ray RLlib integration provides scalable multi-agent training using shared policies to encourage collaboration. A centralized critic is trained by observing the joint state and action space of all drone agents, while drone agents execute decentralized policies during deployment. We observed that increasing the number of cooperative drones improves performance, achieving a 66.7% increase in episode reward, a 42% improvement in team success rate, and a 65% reduction in geofence violations compared to the two-drone configuration. The proposed framework provides a scalable foundation for real-world cooperative counter-unmanned aerial system (C-UAS) operations using deep reinforcement learning.

1. Introduction

The growing number of drones operating in civilian and commercial airspace has demonstrated the importance of counter-unmanned aerial systems to ensure airspace safety, privacy, and critical infrastructure protection. In the European Union, these efforts are closely aligned with the development of U-space [1], a regulatory framework to enable safe and efficient integration of drones into low-altitude airspace. Effective C-UAS solutions that can interact with U-space services are therefore essential for coordinated detection, interception, and traffic management of unauthorized drones.

Autonomous multi-agent systems have revolutionized applications such as aerial surveillance, disaster response, and environmental monitoring. Especially, drone swarms offer scalable, flexible platforms for real-time decision-making in dynamic environments. However, decentralized coordination remains challenging due to partial observability, spatial constraints, and limited communication.

Multi-Agent Reinforcement Learning (MARL) provides a promising framework for developing such coordination policies by enabling agents to learn optimal behaviors by interacting with their environment [2]. However, MARL presents several challenges: learning stable policies in the presence of multiple learners, dealing with sparse and delayed rewards, and ensuring effective collaboration without explicit communication.

Recent studies have explored reinforcement learning (RL) and drone swarm intelligence for aerial systems, including drone control systems, swarm coordination, and counter-drone technologies. For instance, Gergal [3] investigates drone swarming tactics using reinforcement learning like proximal policy optimization and soft actor-critic. Additionally, Gavin et al. [4] explored the benefits of hardware acceleration and parallelization for multi-agent reinforcement learning in swarm systems. In addition, drone swarms are also utilized in disaster relief. Ref. [5] explores the development of a decentralized drone swarm system designed for the dynamic search, detection, and monitoring of wildfires. This study demonstrates that the proposed system can handle increases or decreases in the number of drones, with the swarm dynamically adjusting to the available agents.

In counter-unmanned aerial systems (C-UAS), Yang et al. [6] conduct a bibliometric analysis and scoping review of recent developments of C-UAS technologies. The findings of their review highlight key topics and emerging trends, focusing on multisensory fusion and machine-learning approaches for enhancing drone detection and mitigation. Additionally, Castrillo et al. [7] review cooperative defensive drone team technologies for countering drones and suggest improvements for real-time swarm defense in terms of team coordination, communication network, and simulation framework. Moreover, Simonjan et al. [8] studied RL-based countermeasures against target-oriented attacks of intelligent unmanned aerial vehicle (UAV) swarms.

This paper extends the previous research on counter drones [9,10] to a multi-agent framework using Multi-Agent Proximal Policy Optimization. This paper integrates a centralized critic and decentralized actor architecture to train cooperative drone agents in the AirSim simulator. To promote collaboration among the agents, PettingZoo and Ray RLlib are used for supporting scalable distributed training with shared policies. The objective is to evaluate how increasing the number of agents influences performance metrics such as interception success rate, collision frequency, and geofence violations, and to establish a foundation for scalable real-world C-UAS drone swarm deployment.

2. Methods

2.1. Reinforcement Learning

Reinforcement learning is an approach to AI based on trial-and-error experiences. In RL, the agent makes a decision and takes an action. The agent interacts with the environment. The environment provides states, which are information about the current status of the agent. Each action updates the environment and its state. Finally, a reward is submitted by the environment, informing about the benefit of using the action at that moment. The objective of the agent is to maximize the final reward value. The interaction between the agent and the environment is shown in Figure 1. State is represented as $S_t$, and the state space is represented as S. The interaction between the agent and the environment is in discrete time steps t. Action and action space are represented as $A_t$ and $A\left(S_t\right)$ respectively. Reward values are updated each time, $R_{t+1}$, and a new state becomes $S_{t+1}$.

In RL, states are mapped to the probability of the possible actions in each time step t, and this is called a policy. The policy is chosen to maximize the cumulative reward over time. The fundamental concepts for RL are explained in detail in [2].

2.2. Proximal Policy Optimization

Proximal Policy Optimization (PPO) [11] is a model-free on-policy RL algorithm that works well for both discrete and continuous action space environments. PPO utilizes an actor-critic framework, where there are two networks: an actor (policy network) and a critic network (value function). The PPO algorithm is a highly sample-efficient method that learns from each interaction with the environment. This is important in our case since our environment is computationally expensive (i.e., slow), and we aim for the drone agents to learn as much as possible from fewer interactions with the environment.

The PPO algorithm that uses fixed-length trajectory segments is shown in Algorithm 1 below. At each iteration, each of N (parallel) actors collects T timesteps of data. Then PPO constructs the surrogate loss function on these NT timesteps of data and optimizes it with minibatch Stochastic Gradient Descent (SGD) or the Adam method for stochastic optimization [12], for K epochs.

Algorithm 1 PPO, Actor-Critic Style

1: for iteration = 1, 2, … do 2:     for actor = 1, 2, …, N do 3:         Run policy ${\pi }_{{\theta }_{\mathrm{old}}}$ in environment for T timesteps 4:         Compute advantage estimates ${\widehat{A}}_1,\dots ,{\widehat{A}}_T$ 5:     end for 6:     Optimize surrogate L w.r.t. $\theta$, with K epochs and minibatch size $M\le NT$ 7:     ${\theta }_{\mathrm{old}}\leftarrow \theta$ 8: end for

2.3. Simulation and Framework Integration

The training environment is implemented in AirSim [13], a high-fidelity simulator for autonomous vehicles. In this paper, the neighborhood environment is selected for training and testing the deep reinforcement learning algorithm. The custom AirSim environment follows the Gymnasium API [14], the modern continuation of OpenAI Gym [15], which is utilized for compatibility with reinforcement learning frameworks to integrate into the multi-agent training framework.

The multi-agent environment follows the PettingZoo Parallel API [16] interfaced with Ray RLlib [17,18] to enable scalable distributed MARL training by connecting PettingZoo’s multi-agent API with RLlib’s distributed PPO training loop. Agents share a common policy to promote cooperative behavior while maintaining decentralized execution. Figure 2 illustrates the experimental setup.

The experimental setup diagram is illustrated in Figure 2.

2.4. Multi-Agent DRL Model

DRL Model includes Multi-Agent Proximal Policy Optimization (MAPPO) with a centralized critic and decentralized execution. The actor processes the local image of the agent, and the centralized critic utilizes the concatenated global state of all agents. Actor and critic do not share weights since image features are unrelated to global state features. In other words, drone agents act based only on the local images, but the critic uses the full global state to stabilize learning in the multi-agent setting. A shared policy is implemented for all the agents.

The Neural Network Architecture of the actor consists of a convolutional neural network (CNN) with three convolutional layers with rectified linear unit (ReLU) activation functions and a 512-unit fully connected layer, mapping local images to action logits. Neural Network (NN) critic has four Multilayer Perceptron (MLP) with three hidden layers, mapping global state to value estimate $V\left(s\right)$, the state value function, shared across all agents. MAPPO architecture is shown in Figure 3. If there are N agents, the full global state is a concatenation of all their positions in x and y coordinates, such as global_state $\in {\mathbb{R}}^{2N}$.

2.4.1. Drone Agent States

Each agent observes a local depth image from AirSim ($84\times 84\times 1$) and a global state vector containing the normalized 2D positions of all agents.

2.4.2. Drone Control Actions

Each agent performs three discrete 2D actions: moving forward with fixed speed, yawing left (rotate counter-clockwise (CCW)), and right (rotate clockwise (CW)). The actions are represented in detail in

Table 1. Actions.

Action	Movement	Motion
0	5 m/s in +x direction	Move Forward
1	30 deg CCW	Yaw Left
2	25 deg CW	Yaw Right

. The target drone is stationary and keeps its altitude at 4 m.

2.4.3. Reward Function

The reward function, as detailed in

Table 2. Intermediate rewards.

Component	Weight/Value	Description
Progress $\Delta d$	$0.6$	Positive if agent moves closer to target
Proximity $f_{\mathrm{prox}}$	$0.3$	Optimal radius 5 m; penalized beyond 20 m
Coverage $f_{\mathrm{cov}}$	$0.3$	Team angular spread around target
Smoothness $P_{\mathrm{smooth}}$	$0.05$	Penalizes turns or large control magnitudes
Time cost	$0.01$	Constant per-step penalty

and

Table 3. Terminal events (team-cooperation rewards).

Event	Reward	Notes
Target captured	$+100$	Shared across agents
Collision (obstacle)	$-50$	Shared across agents
Geofence violation	$-25$	Shared across agents
Timeout (>180 steps)	$-25/N$ each	Episode truncation

, includes both intermediate rewards and terminal rewards that reward or penalize the agent during the episode. Reward shaping is designed to encourage agents to move closer to the target (progress), maintain an optimal distance (proximity), and spread out around the target (coverage), while penalizing sudden turns (smoothness) and time elapsed (efficiency). The intermediate rewards are clipped to $[-2,2]$ for stability.

Terminal rewards include team-level rewards: a large bonus for capturing the target, penalties for collisions or geofence violations, and a time limit penalty. The geofence extends from −5 to +25 along the x-axis and from −10 to +15 along the y-axis. Rewards from terminal events are shared equally among agents. This design is implemented to balance each agent’s individual movements with team cooperation.

The reward function for each time step becomes

$$\begin{array}{cc}\hfill R_i^{\mathrm{episode}}& ={\displaystyle \sum _{t=0}^T}(0.6\Delta d_i\left(t\right)+0.3f_{\mathrm{prox},i}\left(t\right)+0.3f_{\mathrm{cov}}\left(t\right)\hfill \\ \hfill & -0.05P_{\mathrm{smooth},i}\left(t\right)-0.01{)}_{\mathrm{clipped}\mathrm{to}[-2,2]}+{\displaystyle \frac{1}{N}}{R}_{\mathrm{terminal}},\hfill \end{array}$$

(1)

Terminal rewards are as follows:

$$R_{\mathrm{terminal}}=\left\{\begin{array}{cc}+100,\hfill & \mathrm{if}\mathrm{target}\mathrm{captured},\hfill \\ -50,\hfill & \mathrm{if}\mathrm{collision},\hfill \\ -25,\hfill & \mathrm{if}\mathrm{geofence}\mathrm{violation},\hfill \\ -25,\hfill & \mathrm{if}\mathrm{time}\mathrm{limit}.\hfill \end{array}\right.$$

(2)

3. Results

The drone agents are trained by utilizing a different number of agents. By using the same MAPPO setup, training is accomplished with two scenarios: one with two drone agents and another with five drone agents.

The drone agent’s performances are evaluated by using the performance metrics such as episode reward mean, team success rate, geofence violations, and collision frequency. The training results of two scenarios are shown in Figure 4 and

Table 4. Overall training performance comparison between 2-drone and 5-drone swarms.

Metric	2 Drones	5 Drones	Improvement
Episode Reward Mean	29.79	49.67	+66.7%
Team Success Rate	0.393	0.558	+42%
Collision Reduction (%)	96.9%	78.7%	−18.8%
Geofence Hits	0.616	0.216	−65%

.

The results show that the increasing number of cooperative drones improves performance, with a 42% improvement in team success rate. For example, the episode reward mean graph shows that the five drone agents performed higher average episode rewards than the two drone agents, demonstrating efficient cooperation catching the target with a performance improvement of approximately 66.7%. Additionally, the team’s success rate was higher with the five drone agents as the training proceeded, although two drones had a higher success rate at the beginning. Moreover, the agents learned to remain within the operational area, and the five-drone agent scenario had fewer geofence violations in the later stage of the training, achieving a 65% decrease in geofence hits. Collisions caused by environmental obstacles, such as trees, or by colliding with other drone agents, also decreased over training in both cases. Collision rates were initially higher with five drone agents due to a higher exploration rate but dropped significantly as coordination improved. Both configurations demonstrate a massive reduction in collisions. Even in the more complex five-drone agent scenario, the system successfully eliminated nearly 79% of initial accidents by the 100th iteration. In summary, the training results show that increasing the number of cooperating drones improves target elimination performance, providing higher rewards, fewer collisions, and better team success rates. Reward shaping and centralized critic are also effective in coordinating multi-agent, counter-drone systems.

Our results suggest that effective counter-drone swarm defense can be achieved without a one-to-one ratio between defenders and intruders, considering the 42% improvement in team success rate after increasing the defender count from two to five drones. A relatively small team of cooperative, intelligent drone agents can counter larger intruding drones by allowing the system to scale sub-linearly, achieving higher operational safety and mission success through collective intelligence.

Figure 5 also shows convergence trends during training. To balance exploration and exploitation, an entropy coefficient schedule was implemented, decaying linearly from 0.010 to 0.001 over the course of training. It is seen that as entropy regularization decays, policy entropy decreases, and episode length mean stays between 26 and 32 steps, which indicates more consistent and efficient pursuit behavior.

Hyperparameters

The hyperparameters presented in

Table 5. PPO Hyperparameters.

Parameter	Description	Value
Lr	Learning rate	3 $\times {10}^{-4}$
train_batch_size	Samples collected per update	512
minibatch_size	Data per gradient step	64
num_epochs	Reuse passes over data	3
gamma	Future reward discount	0.99
lambda_	Advantage smoothing factor	0.95
vf_clip_param	Limit value estimate change	2.0
vf_loss_coeff	Weight of value loss	1.5
entropy_coeff_schedule	Exploration decay schedule	0.010–0.001
grad_clip	Cap gradient magnitude	1.0

represent a customized configuration rather than standard default values. These parameters are tuned for training the MAPPO-based multi-agent system within AirSim to ensure stable convergence and efficient learning, while maintaining computational efficiency.

4. Conclusions and Future Work

This study presents a cooperative counter-drone system trained by the MAPPO algorithm. We have observed that cooperative drone swarms for countering drones powered by multi-agent DRL significantly enhance learning in complex urban environments, achieving higher episode rewards and a significant improvement in team success rate. This provides a robust foundation for transitioning from simulation to real-world deployment.

With coordinated positioning, coverage control, and collective reward structures, the system exhibits sub-linear scaling between defender count and intruder swarm size. As a direction for future work, we plan to investigate the extent to which these properties generalize to large-scale scenarios involving hundreds or thousands of intruding drones.

Future research will also focus on enhancing the proposed counter-drone solution. Firstly, a drone detection model will be implemented to enable real-time identification, classification, localization, and tracking of unauthorized drones within controlled airspace. This model is expected to improve drone detection accuracy by using advanced AI techniques under different environmental and operational conditions. Moreover, alternative algorithms will be explored to improve sample efficiency with off-policy methods or to enhance coordination in discrete action settings with value-decomposition approaches. Finally, U-Space services with counter-drone systems will be integrated. This integration aims to establish a coordinated framework that supports safe and secure low-altitude air traffic management. By linking U-Space with counter-drone operations, authorities can respond faster to potential drone threats.