Interception of a Single Intruding Unmanned Aerial Vehicle by Multiple Missiles Using the Novel EA-MADDPG Training Algorithm

Abstract

This paper proposes an improved multi-agent deep deterministic policy gradient algorithm called the equal-reward and action-enhanced multi-agent deep deterministic policy gradient (EA-MADDPG) algorithm to solve the guidance problem of multiple missiles cooperating to intercept a single intruding UAV in three-dimensional space. The key innovations of EA-MADDPG include the implementation of the action filter with additional reward functions, optimal replay buffer, and equal reward setting. The additional reward functions and the action filter are set to enhance the exploration performance of the missiles during training. The optimal replay buffer and the equal reward setting are implemented to improve the utilization efficiency of exploration experiences obtained through the action filter. In order to prevent over-learning from certain experiences, a special storage mechanism is established, where experiences obtained through the action filter are stored only in the optimal replay buffer, while normal experiences are stored in both the optimal replay buffer and normal replay buffer. Meanwhile, we gradually reduce the selection probability of the action filter and the sampling ratio of the optimal replay buffer. Finally, comparative experiments show that the algorithm enhances the agents’ exploration capabilities, allowing them to learn policies more quickly and stably, which enables multiple missiles to complete the interception task more rapidly and with a higher success rate.

1. Introduction

Unmanned aerial vehicles (UAVs), especially drones, have been widely applied and played a key role in several local wars due to their low cost, flexible maneuverability, strong concealment, and ability to perform tasks in harsh environments. Although drone technology has shown great potential in areas such as reconnaissance, logistics, and environmental monitoring, its security issues have also raised widespread concern, especially how to effectively address the threat of hostile drones [1]. Therefore, the study of UAV interception is essential, especially in the context of the rapid development and widespread application of UAV technology today.

The algorithms are mainly divided into two major categories: rule-based approaches [2,3,4,5,6,7,8,9,10,11] and learning-based approaches [12,13,14,15,16,17,18,19,20,21,22]. Within these categories, the methods can be further divided into single and multiple types. Single interception guidance methods: Command to line-of-sight guidance adjusts the interceptor’s path based on external commands, precisely controlling the interception process [4]. Optimal guidance designs guidance laws by minimizing a cost function to find the best path under given performance metrics [5]. Augmented proportional navigation adjusts the guidance gain to adapt to different interception conditions, enhancing efficiency against high-speed or highly maneuverable targets [6]. Predictive guidance adjusts the flight path of the interceptor based on predictions of the target’s movements, combating fast-maneuvering targets [7]. Multiple interception guidance methods: The leader–follower strategy involves one UAV acting as the leader, formulating the interception strategy, while the other UAVs follow and execute the plan, simplifying the decision-making process and ensuring team actions are synchronized [8]. The coordinated attack strategy requires high coordination among UAVs to determine the best angles and timings for the attack, effectively utilizing the team’s collective strength to increase the success rate of interceptions [9]. The sequential or alternating attack strategy allows UAVs to attack in a predetermined order or timing, maintaining continuous pressure on the target and extending the duration of combat [10]. The distributed guidance strategy enables each UAV to make independent decisions based on overall team information, enhancing the team’s adaptability and robustness, suitable for dynamic battlefield environments [11].

In recent decades, the rapid development and success of reinforcement learning have made it an ideal tool for solving complex problems, particularly in the field of UAV interception applications. The development of single-agent reinforcement learning algorithms began in the early 1980s, with initial explorations focusing on simple learning control problems [12]. Into the 21st century, with enhancements in computational power and data processing capabilities, deep learning techniques began to be integrated into reinforcement learning. Mnih et al. pioneered the deep q-network, achieving control levels in visually rich Atari games that matched or exceeded human performance for the first time [13]. Additionally, the deep deterministic policy gradient (DDPG) algorithm [14], proposed by Lillicrap et al., and the proximal policy optimization algorithm [15] by Schulman et al., further expanded the application of reinforcement learning in continuous action spaces. Haarnoja et al. introduced the soft actor–critic algorithm, which optimized policies within a maximum entropy framework, demonstrating its superiority in handling tasks involving high dimensions and complex environments [16]. However, single-agent algorithms face challenges such as the exponential explosion in state-action space and local optima when dealing with multi-agent systems. To overcome these challenges, researchers have extended single-agent algorithms to multi-agent algorithms, such as multi-agent deep deterministic policy gradient (MADDPG), multi-agent actor–critic, multi-agent proximal policy optimization (MAPPO), and multi-agent twin-delayed deep deterministic policy gradient (MATD3) [17,18,19,20]. In the realms of single UAV and multiple UAV interception, reinforcement learning has emerged as a significant research focus. For a single UAV, as studied by Koch et al., deep reinforcement learning is employed to train the UAV for autonomous navigation and obstacle avoidance in environments with obstacles [21]. In the context of multiple UAVs, Qie et al. proposed an artificial intelligence method named simultaneous target assignment and path planning based on MADDPG to train the system to solve target assignment and path planning simultaneously according to a corresponding reward structure [22].

1.1. Related Works

In the field of UAV interception, various studies have explored effective interception strategies for both single UAV and multiple UAV environments to tackle challenging missions. Garcia et al. focused on designing minimal-time trajectories to intercept malicious drones in constrained environments, breaking through the limitations of traditional path planning in complex settings [23]. Tan et al. introduced a proportional-navigation-based switching strategy for quadrotors to track maneuvering ground targets, addressing the limitations of traditional proportional derivative control. By dynamically switching between proportional navigation and proportional derivative controls, the proposed method reduced tracking errors and oscillations, demonstrating improved performance and robustness against measurement noise. The optimal switching points, derived analytically, ensured minimal positional errors between the UAV and the target [24]. Cetin et al. addressed the visual quadrotor interception problem in urban anti-drone systems by employing a model predictive control (MPC) method with terminal constraints to ensure engagement at a desired impact angle. The modified MPC objective function reduced the interceptor’s maneuvering requirements at the trajectory’s end, enhancing efficiency. The proposed guidance methodology was successfully tested in various scenarios, demonstrating effective interception of both maneuvering and non-maneuvering targets with minimal maneuvering at the interception’s conclusion [25]. Xue et al. introduced a fuzzy control method for UAV formation trajectory tracking, simplifying control logic and improving stability. Simulations confirm the method’s effectiveness in achieving rapid and stable formation docking [26]. Li et al. developed an enhanced real proportional navigation guidance method for gun-launched UAVs to intercept maneuvering targets, addressing limitations such as saturation overload and capture region constraints. By integrating an extended Kalman filter for data fusion and trajectory prediction, the method improved guidance accuracy and overcame system delay issues, resulting in more effective interception of “Low–slow–small” targets [27]. Liu et al. explored a cooperative UAV countermeasure strategy based on an interception triangle, using a geometric formation formed by three UAVs to enhance interception efficiency [28]. Tong et al., inspired by the cooperative hunting behavior of Harris’s hawks, proposed a multi-UAV cooperative hunting strategy, optimizing the interception process by mimicking natural cooperative behaviors [29]. In addition to drones, research on the interception of other moving objects also offers valuable insights. Shaferman et al. focused on developing and optimizing guidance strategies that enable multiple interceptors to coordinate their operations to achieve a predetermined relative intercept angle. By employing advanced optimization algorithms, this research significantly enhanced mission efficiency, particularly in scenarios where the intercept angle was crucial for successful interception [30]. Chen et al. explored and developed methods for enhancing the accuracy of target recognition and matching in missile guidance systems that used television-based seekers [31].

While rule-based methods have laid a solid foundation for UAV interception, the dynamic and unpredictable nature of modern aerial environments necessitates more adaptable and intelligent approaches. This is where reinforcement learning comes into play, offering significant advantages in UAV interception tasks. Reinforcement learning allows UAVs to learn from interactions with the environment, enabling them to make decisions in real time and adapt to changes dynamically.

In the single-agent domain, to address the issue of using drones to intercept hostile drones, Ting et al. explored the use of a deep q-network (DQN) with a graph neural network (GNN) model for drone-to-drone interception path planning, leveraging the flexibility of GNNs to handle variable input sizes and configurations. The proposed DQN-GNN method effectively trained a chaser drone to intercept a moving target drone, demonstrating the feasibility of integrating GNNs with deep reinforcement learning for this purpose. Results showed that the chaser drone could intercept the target in a reasonable time, suggesting that this approach enhanced the efficiency of drone interception tasks [32]. Furthermore, to counter the threat of small unmanned aerial systems to airspace systems and critical infrastructure, Pierre et al. applied deep reinforcement learning to intercept rogue UAVs in urban airspace. Using the proximal policy optimization method, they verified its effectiveness in improving interception success rates and reducing collision rates [33]. In addition, research on unmanned surface vessels (USVs) and missile interception is also of important reference significance. Du et al. introduced a multi-agent reinforcement learning control method using safe proximal policy optimization (SPPO) for USVs to perform cooperative interception missions. The SPPO method incorporated a joint state–value function to enhance cooperation between defender vessels and introduced safety constraints to reduce risky actions, improving the stability of the learning process. Simulation results demonstrated the effectiveness of the SPPO method, showing high performance in reward and successful cooperative interception of moving targets by the USVs [34]. Liu et al. introduced a combined proximal policy optimization and proportional–derivative control method for USV interception, enhancing interception efficiency. Simulation results showed that the proposed method significantly reduced the interception time compared to traditional approaches [35]. Hu et al. presented the twin-delayed deep deterministic policy gradient (TD3) algorithm to develop a guidance law for intercepting maneuvering targets with UAVs. The proposed method improved upon traditional proportional navigation guidance by directly mapping the line-of-sight rate to the normal acceleration command through a neural network, resulting in enhanced accuracy and convergence of the guidance system [36]. Li et al. proposed a covertness-aware trajectory design method for UAVs based on an enhanced TD3 algorithm. By integrating multi-step learning and prioritizing experience replay techniques, the method enabled UAVs to adaptively select flight velocities from a continuous action space to maximize transmission throughput to legitimate nodes while maintaining covertness. Considering the impact of building distribution and the uncertainty of the warden’s location, this approach effectively addressed issues that were challenging for standard optimization methods and demonstrated significant performance advantages over existing deep reinforcement learning baselines and non-learning strategies in numerical simulations [37]. Li et al. employed a quantum-inspired reinforcement learning (QiRL) approach to optimize the trajectory planning in UAV-assisted wireless networks without relying on prior environmental knowledge. QiRL introduced a novel probabilistic action selection policy inspired by quantum mechanics’ collapse and amplitude amplification, achieving a natural balance between exploration and exploitation without the need for tuning exploration parameters typical in conventional reinforcement learning methods. Simulation results validated the effectiveness of QiRL in addressing the UAV trajectory optimization problem, demonstrating faster convergence and superior learning performance compared to traditional q-learning methods [38]. Li et al. introduced TD3 and quantum-inspired experience replay (QiER) to optimize the trajectory planning for UAVs in cellular networks, enhancing performance in reducing flight time and communication outages and improving learning efficiency by effectively balancing exploration and exploitation [39].

For multi-agent systems, Wan et al. presented an enhanced algorithm based on MADDPG called mixed-experience MADDPG (ME-MADDPG), which enhanced sample efficiency and training stability by introducing artificial potential fields and a mixed-experience strategy. Experiments demonstrated that ME-MADDPG achieved faster convergence and better adaptability in complex dynamic environments compared to MADDPG, showing superior performance in multi-agent motion planning [40]. In order to enhance navigation and obstacle avoidance in multi-agent systems, Zhao et al. proposed the MADDPG-LSTM actor algorithm, which combined long short-term memory (LSTM) networks with the MADDPG algorithm, and the simplified MADDPG algorithm to improve efficiency in scenarios with a large number of agents. Experimental results showed that these algorithms outperformed existing networks in the OpenAI multi-agent particle environment, and the LSTM model demonstrated a favorable balance in handling data of varying sequence lengths compared to transformer and self-attention models [41]. Huang et al. proposed and validated the MADDPG for its effectiveness and superior performance in multi-agent defense and attack scenarios [42]. For collaborative drone tasks, a cooperative encirclement strategy based on multi-agent reinforcement learning was proposed, using the attention-mechanism MADDPG algorithm, which addressed the problem of surrounding airborne escape targets in a collaborative drone attack scenario [43]. Using the all-domain simulation 3D war game engine from China Aerospace System Simulation Technology, Wei et al. conducted simulations of confrontations between UAVs and radar stations. By designing rewards and integrating LSTM networks into multi-agent reinforcement learning, the recurrent deterministic policy gradient (RDPG) method was improved, and a combination of MADDPG and RDPG algorithm was developed, significantly enhancing the algorithm’s effectiveness and accuracy [44]. Jeon et al. introduced a fusion–multiactor–attention–critic model for energy-efficient navigation control of UAVs. The model incorporated a sensor fusion layer and a dissimilarity layer to optimize information processing, resulting in improved energy efficiency and 38% more deliveries within the same time steps compared to the original multiactor–attention–critic model [45]. Yue et al. proposed a method for multi-object tracking by a swarm of drones using a multi-agent soft actor–critic approach [46]. For the pursuit–evasion game of multi-rotor drones in obstacle-laden environments, Zhang et al. proposed the CBC-TP Net, a multi-agent bidirectional coordination target prediction network, which incorporated a vectorized extension of the MADDPG method to ensure effective task execution even if the “swarm” system was compromised [47]. The priority-experience replay-based multi-agent deep deterministic policy gradient algorithm (PER-MADDPG) has shown excellent performance in increasing the success rate of pursuits and reducing response times, featuring faster convergence and reduced oscillations compared to the MADDPG algorithm [48]. Zhu et al. applied deep reinforcement learning to the cluster control problems of multi-robot systems in complex environments, proposing the PER-MADDPG algorithm, which significantly enhanced learning efficiency and convergence speed. Experimental results validated the effectiveness of this algorithm in completing cluster tasks in obstacle-laden environments [49]. Jiang et al. integrated self-attention into the MADDPG algorithm to enhance the stability of learned policies by adapting to the dynamic changes in the number of adversaries and allies during confrontations. They presented a simplified 2D simulation environment and conducted experiments in three different collaborative and confrontational scenarios, demonstrating improved performance over baseline methods [50]. Zhang et al. tested the distributed decision-making and collaboration tasks of heterogeneous drones in a Unity3D cooperative combat environment, proposing an improved MAPPO algorithm and enhancing the algorithm’s generalizability through the course learning [51]. Huang et al. developed a collaborative path planning method for multiple drones based on the MATD3 algorithm with dual experience pools and a particle swarm optimization algorithm, significantly reducing the age of information [52]. Some typical methods are summarized in Table 1.

1.2. Contributions

This paper investigates the guidance of multiple missiles in three-dimensional space for intercepting a single UAV with random initial positions and uncertain trajectories. We use an improved algorithm called equal-reward and action-enhanced multi-agent deep deterministic policy gradient (EA-MADDPG) based on the MADDPG algorithm for training the control policy of the missiles to enhance their ability to complete interception tasks. The main contributions of this paper are as follows:

• In multi-agent collision avoidance tasks, MADDPG-LSTM outperforms standard MADDPG due to the LSTM’s capability to handle sequential data. This advantage arises from the LSTM’s underlying principle of maintaining long-term dependencies and capturing temporal patterns through its specialized memory cells and gating mechanisms. Compared to LSTM, the EA-MADDPG in this paper, based on an equal reward setting and an optimal replay buffer with a special storage mechanism, enhances the utilization of important experiences during learning, thereby improving the collision avoidance training effectiveness in multi-missile interception tasks.
• Prioritized experience replay techniques can enhance exploration performance by prioritizing and sampling experiences based on their significance within the replay buffer. In contrast, this paper introduces a new method to enhance exploration performance by incorporating an action filter. During training, agents use the action filter for action selection with a certain probability, executing these actions to enhance exploration and acquire unique experiences for storage.
• ME-MADDPG [40] generates special experiences using an artificial potential field with a certain probability and incorporates them into the buffer for sampling and training. In this paper, the EA-MADDPG uses an action filter for action selection to generate unique experiences for storage and learning. All actions are derived from the output of the agents’ actor networks.

Table 1. Typical Methods.

Article	Method
[41,44]	MADDPG with long short-term memory networks
[37,48,49]	Prioritized experience replay techniques
[40]	Artificial potential field and mixed-experience mechanism

Table 1. Typical Methods.

Article	Method
[41,44]	MADDPG with long short-term memory networks
[37,48,49]	Prioritized experience replay techniques
[40]	Artificial potential field and mixed-experience mechanism

The remaining parts of this paper are structured as follows. Section 2 introduces the basic notations. Section 3 presents the kinematic models of missiles and the UAV, as well as the description of the problem. Section 4 details the adopted EA-MADDPG algorithm. Section 5 conducts simulations and discusses the results, and Section 6 summarizes the conclusions of the paper.

2. Notation

N represents the number of agents within the environment. The set of states $\mathcal{X}$ represents the possible configurations of all agents. The set $\mathcal{A}=\{{\mathcal{A}}_1,\cdots ,{\mathcal{A}}_N\}$ denotes the possible action space of all agents, and ${\mathcal{A}}_i$ denotes the action space of agent i. The set $\mathcal{O}=\{{\mathcal{O}}_1,\cdots ,{\mathcal{O}}_N\}$ denotes the observation space of all agents, and ${\mathcal{O}}_i$ denotes the observation space of agent i. $\mathbf{x}\in \mathcal{X}$ denotes the state of all agents at a certain moment. $o_i\in {\mathcal{O}}_i$ denotes the information observed by agent i at a certain moment. $a=\{a_1,a_2,\cdots ,a_N\}$, where $a\in \mathcal{A}$ represents the joint action vector of all agents at a certain moment, and $a_i\in {\mathcal{A}}_i$ denotes the action executed exclusively by agent i within the environment at a certain moment. ${\mathit{\mu }}_{{\theta }_i}:{\mathcal{O}}_i\mapsto {\mathcal{A}}_i$ (abbreviated as ${\mathit{\mu }}_i$) denotes the policy generated by the actor network of the ith agent, with ${\theta }_i$ representing the actor network’s parameters. It takes observations as input and produces actions as output. ${{\mathit{\mu }}^{\prime }}_{{\theta }_i^{\prime }}$ (abbreviated as ${{\mathit{\mu }}^{\prime }}_i$) denotes the policy generated by the target actor network of the ith agent, with ${\theta }_i^{\prime }$ representing the target network’s parameters. ${\mathit{Q}}_{{\varphi }_i}^{\mathit{\mu }}(\mathbf{x},a):\mathcal{X}\times \mathcal{A}\mapsto \mathbb{R}$, it is the action value function generated by the critic network of the ith agent, with ${\varphi }_i$ representing the critic network’s parameters and $\mathit{\mu }$ representing the set $\{{\mathit{\mu }}_1,\cdots ,{\mathit{\mu }}_N\}$. ${\mathit{Q}}_{{\varphi }_i^{\prime }}^{{\mathit{\mu }}^{\prime }}(\mathbf{x},a)$ is the action value function generated by the target critic network of the ith agent, with ${\varphi }_i^{\prime }$ representing the target critic network’s parameters and ${\mathit{\mu }}^{\prime }$ representing the set $\{{\mathit{\mu }}_1^{\prime },\cdots ,{\mathit{\mu }}_N^{\prime }\}$. ${\mathbf{x}}^{\prime }$ denotes the subsequent state reached by agents after executing their current actions. $r=\{r_1,r_2,\cdots ,r_N\}$, where $r\in {\mathbb{R}}^N$, denotes the joint reward vector of all agents, and $r_i\in \mathbb{R}$ represents the immediate reward obtained by agent i within the environment. $r_i^{total}={\sum }_{k=0}^{k_{max}-1}{\gamma }^k{r}_i^k$ denotes the total expected return of agent i, where $k_{max}\in {\mathbb{N}}^{+}$ denotes the max step in an episode. $\mathcal{N}\in {\mathcal{A}}_i$ represents the noise added to actions during reinforcement learning training to enhance exploration. $\mathcal{D}$ denotes the replay buffer used to store the past experience tuples collected. $\gamma \in \mathbb{R}$ denotes the discount rate used in calculating cumulative rewards. $\tau \in \mathbb{R}$ denotes the parameters used for updating the target network. $S\in \mathbb{R}$ denotes the number of experience tuples used for training at one time. ${▽}_{x}y$ denotes the derivative of y with respect to x.

3. Modeling and Problem Description

This paper considers the problem of a group of N missiles intercepting a single intruding UAV which aims to attack some ground target. In this section, the models of the missiles and the UAV are introduced first, which is then followed by the detailed description of the intercepting problem in rigorous mathematics.

3.1. Modeling of Missiles and the UAV

Define $\underline{N}=\{1,\cdots ,N\}$. For $i\in \underline{N}$, the kinematic model of the ith missile is given by

$$p_i(k+1)=p_i\left(k\right)+T{v}_i\left(k\right)$$

(1)

where $p_i\left(k\right),v_i\left(k\right)\in {\mathbb{R}}^3$ denote the position and velocity of the ith missile at the kth step, respectively. $T\in \mathbb{R}$ denotes the duration of each step. Furthermore, the velocity $v_i\left(k\right)$ is given by

$$v_i\left(k\right)=\left(\begin{array}{c}\parallel v_i\left(k\right)\parallel cos\left({\beta }_i^v\left(k\right)\right)cos\left({\alpha }_i^v\left(k\right)\right)\\ \parallel v_i\left(k\right)\parallel cos\left({\beta }_i^v\left(k\right)\right)sin\left({\alpha }_i^v\left(k\right)\right)\\ \parallel v_i\left(k\right)\parallel sin\left({\beta }_i^v\left(k\right)\right)\end{array}\right)$$

(2)

Here, the speed $\parallel v_i\left(k\right)\parallel$ satisfies $\parallel v_i\left(k\right)\parallel =v_{max}$, where $v_{max}\in \mathbb{R}$ denotes the maximum linear velocity. Moreover, as shown in Figure 1, $v_{i-xy}\left(k\right)\in {\mathbb{R}}^2$ denotes the projection of $v_i\left(k\right)$ onto the plane parallel to O–$xy$. ${\alpha }_i^v\left(k\right)\in (-\pi ,\pi ]$ denotes the angle of $v_{i-xy}\left(k\right)$ with respect to the positive semi-axis of x, with counterclockwise being positive and clockwise being negative, and this rule of sign for angles is adopted throughout this paper. ${\beta }_i^v\left(k\right)\in (-{\displaystyle \frac{\pi }{2}},{\displaystyle \frac{\pi }{2}}]$ denotes the angle of $v_i\left(k\right)$ with respect to $v_{i-xy}\left(k\right)$. The angles of ${\alpha }_i^v\left(k\right)$ and ${\beta }_i^v\left(k\right)$ are governed by

$$\left\{\begin{array}{c}\hfill {\beta }_i^v(k+1)={\beta }_i^v\left(k\right)+T{u}_{{\beta }_i}\left(k\right)\hfill \\ \hfill {\alpha }_i^v(k+1)={\alpha }_i^v\left(k\right)+T{u}_{{\alpha }_i}\left(k\right)\hfill \end{array}\right.$$

(3)

where the corresponding angular velocities $u_{{\beta }_i}\left(k\right),u_{{\alpha }_i}\left(k\right)\in \mathbb{R}$ are considered as the control inputs of the missiles. To make the control practical, it is assumed that $|u_{{\beta }_i}\left(k\right)|<\Theta$ and $|u_{{\alpha }_i}\left(k\right)|<\Theta$ for some control input saturation threshold $\Theta >0$.

Figure 1. The heading parameters of the ith missile.

The kinematic model of the UAV is given by

$$p_m(k+1)=p_m\left(k\right)+T{v}_m\left(k\right)$$

(4)

where $p_m\left(k\right),v_m\left(k\right)\in {\mathbb{R}}^3$ denote the position and velocity of the UAV at the kth step, respectively. The velocity $v_m\left(k\right)$ is given by

$$v_m\left(k\right)=\left(\begin{array}{c}{v}_{max}cos({\beta }^m\left(k\right)+{\mathcal{U}}_{\beta })cos({\alpha }^m\left(k\right)+{\mathcal{U}}_{\alpha })\\ v_{max}cos({\beta }^m\left(k\right)+{\mathcal{U}}_{\beta })sin({\alpha }^m\left(k\right)+{\mathcal{U}}_{\alpha })\\ v_{max}sin({\beta }^m\left(k\right)+{\mathcal{U}}_{\beta })\end{array}\right)$$

(5)

As shown in Figure 2, $v_{m-xy}\left(k\right)\in {\mathbb{R}}^2$ denotes the projection of $v_m\left(k\right)$ onto the plane parallel to O–$xy$. ${\alpha }^m\left(k\right)\in (-\pi ,\pi ]$ denotes the angle of $v_{m-xy}\left(k\right)$ with respect to the positive semi-axis of x. ${\beta }^m\left(k\right)\in (-{\displaystyle \frac{\pi }{2}},{\displaystyle \frac{\pi }{2}}]$ denotes the angle of $v_m\left(k\right)$ with respect to $v_{m-xy}\left(k\right)$. ${\mathcal{U}}_{\alpha },{\mathcal{U}}_{\beta }\in [-{\displaystyle \frac{\pi }{36}},{\displaystyle \frac{\pi }{36}}]$ represent the noise subject to a uniform distribution. Note that the speed of the UAV is the same as the maximal speed of the missiles.

Figure 2. The heading parameters of the UAV.

As shown in Figure 3, $l_i^n\left(k\right)\in {\mathbb{R}}^2$ denotes the projection of $p_m\left(k\right)-p_i\left(k\right)$ onto the plane parallel to O–$xy$, and $l_i^f\left(k\right)\in {\mathbb{R}}^2$ denotes the projection of $p_m\left(k\right)-p_i(k-1)$ onto the plane parallel to O–$xy$. Note that $l_i^n\left(k\right)$ represents the current relative position between the ith missile and the UAV, whereas $l_i^f\left(k\right)$ represents the semi-historical relative position. ${\alpha }_i^n\left(k\right)\in (-\pi ,\pi ]$ denotes the angle of $l_i^n\left(k\right)$ related to the positive semi-axis of x. ${\beta }_i^n\left(k\right)\in (-{\displaystyle \frac{\pi }{2}},{\displaystyle \frac{\pi }{2}}]$ denotes the angle of $p_m\left(k\right)-p_i\left(k\right)$ related to $l_i^n\left(k\right)$. ${\alpha }_i^f\left(k\right)\in (-\pi ,\pi ]$ denotes the angle of $l_i^f\left(k\right)$ related to the positive semi-axis of x. ${\beta }_i^f\left(k\right)\in (-{\displaystyle \frac{\pi }{2}},{\displaystyle \frac{\pi }{2}}]$ denotes the angle of $p_m\left(k\right)-p_i(k-1)$ related to $l_i^f\left(k\right)$. Based on these angles, we define

$${\alpha }_i\left(k\right)=\left\{\begin{array}{cc}2\pi -({\alpha }_i^n\left(k\right)-{\alpha }_i^v\left(k\right))\hfill & \mathrm{if}{\alpha }_i^n\left(k\right)-{\alpha }_i^v\left(k\right)>\pi \hfill \\ {\alpha }_i^n\left(k\right)-{\alpha }_i^v\left(k\right)\hfill & \mathrm{if}\pi \ge {\alpha }_i^n\left(k\right)-{\alpha }_i^v\left(k\right)>-\pi \hfill \\ 2\pi +({\alpha }_i^n\left(k\right)-{\alpha }_i^v\left(k\right))\hfill & \mathrm{if}{\alpha }_i^n\left(k\right)-{\alpha }_i^v\left(k\right)\le -\pi \hfill \end{array}\right.$$

(6)

$${\beta }_i\left(k\right)={\beta }_i^n\left(k\right)-{\beta }_i^v\left(k\right)$$

(7)

where ${\alpha }_i\left(k\right)$ denotes the angle of $l_i^n\left(k\right)$ with respect to $v_{i-xy}\left(k\right)$. ${\beta }_i\left(k\right)$ denotes the angle of $p_m\left(k\right)-p_i\left(k\right)$ with respect to $v_i\left(k\right)$ after they are both rotated around the z-axis to align within the same vertical plane. Moreover, for $k=1,2,\cdots$,

$${\alpha }_i^{\prime }\left(k\right)=\left\{\begin{array}{cc}2\pi -({\alpha }_i^f\left(k\right)-{\alpha }_i^v(k-1))\hfill & \mathrm{if}{\alpha }_i^f\left(k\right)-{\alpha }_i^v(k-1)>\pi \hfill \\ {\alpha }_i^f\left(k\right)-{\alpha }_i^v(k-1)\hfill & \mathrm{if}\pi \ge {\alpha }_i^f\left(k\right)-{\alpha }_i^v(k-1)>-\pi \hfill \\ 2\pi +({\alpha }_i^f\left(k\right)-{\alpha }_i^v(k-1))\hfill & \mathrm{if}{\alpha }_i^f\left(k\right)-{\alpha }_i^v(k-1)\le -\pi \hfill \end{array}\right.$$

(8)

$${\beta }_i^{\prime }\left(k\right)={\beta }_i^f\left(k\right)-{\beta }_i^v(k-1)$$

(9)

where ${\alpha }_i^{\prime }\left(k\right)$ denotes the angle of $l_i^f\left(k\right)$ with respect to $v_{i-xy}(k-1)$. ${\beta }_i^{\prime }\left(k\right)$ denotes the angle of $p_m\left(k\right)-p_i(k-1)$ with respect to $v_i(k-1)$ after they are both rotated around the z-axis to align within the same vertical plane. In addition, we simply set ${\alpha }_i^{\prime }\left(0\right)=0$ and ${\beta }_i^{\prime }\left(0\right)=0$.

Figure 3. The current and semi-historical heading parameters between the ith missile and the UAV.

3.2. Problem Description

In Figure 4, the blue curves represent a hemisphere $\mathbb{P}$ with center O and radius $\mathcal{R}$. The orange curves represent a cone with apex O and inner angle $\phi \in [0,{\displaystyle \frac{\pi }{2}}]$. The cone intersects with $\mathbb{P}$, and let the upper hemisphere be denoted by ${\mathbb{P}}_u$. In this paper, it is assumed that $p_m\left(0\right)\in {\mathbb{P}}_u$, i.e., the UAV will appear randomly on the upper hemisphere. Let $p_{Target}={[p_{Target,x},p_{Target,y},p_{Target,z}]}^T\in {\mathbb{R}}^3$ denote the position of the target, satisfying

$$p_{Target,z}=0,p_{Target,x}^2+p_{Target,y}^2<{\mathcal{R}}_r^2$$

(10)

for some ${\mathcal{R}}_r>0$. For all k, the angles of of ${\beta }^m\left(k\right)$ and ${\alpha }_m\left(k\right)$ are set to be such that the vector ${[cos\left({\beta }^m\left(k\right)\right)cos\left({\alpha }^m\left(k\right)\right),cos\left({\beta }^m\left(k\right)\right)sin\left({\alpha }^m\left(k\right)\right),sin\left({\beta }^m\left(k\right)\right)]}^T$ has the same direction as the vector $p_{Target}-p_m\left(k\right)$. For $i\in \underline{N}$, let $p_i\left(k\right)={[p_{ix}\left(k\right),p_{iy}\left(k\right),p_{iz}\left(k\right)]}^T$. Suppose

$$p_{iz}\left(0\right)=0,p_{ix}{\left(0\right)}^2+p_{iy}{\left(0\right)}^2<{\mathcal{R}}_g^2$$

(11)

for some ${\mathcal{R}}_g>0$. Moreover, for $i\in \underline{N}$, ${\beta }_i^v\left(0\right)$ and ${\alpha }_i^v\left(0\right)$ are set to be such that the vector ${[cos\left({\beta }_i^v\left(k\right)\right)cos\left({\alpha }_i^v\left(k\right)\right),cos\left({\beta }_i^v\left(k\right)\right)sin\left({\alpha }_i^v\left(k\right)\right),sin\left({\beta }_i^v\left(k\right)\right)]}^T$ has the same direction as the vector $p_m\left(0\right)-p_i\left(0\right)$. The control objective is to design the control inputs of the missiles, i.e., $u_{{\beta }_i}\left(k\right),u_{{\alpha }_i}\left(k\right)$ such that there exist $i\in \underline{N}$ and $k^{*}$ satisfying the following two conditions simultaneously,

$$\parallel p_i\left(k^{*}\right)-p_m\left(k^{*}\right)\parallel \le {\mathcal{R}}_d$$

(12)

$$\parallel p_m\left(k^{*}\right)-p_{Target}\parallel >{\mathcal{R}}_s$$

(13)

and for $k\le k^{*}$, at least one of these two conditions does not hold. Here, ${\mathcal{R}}_d$ denotes the intercepting distance of the UAV. and ${\mathcal{R}}_s$ denotes the safety distance of the target. At the same time, we define:

$$d=\parallel p_m\left(k^{*}\right)-p_{Target}\parallel$$

(14)

which represents the distance between the UAV and the target at the time when the UAV is successfully intercepted.

Figure 4. Environment description.

For the multi-missile guidance and interception problem described above, we chose the MADDPG algorithm as the fundamental method. Here were the reasons for this choice: First, in multi-agent systems, missiles need to work together to achieve a common goal. Each missile must optimize its own behavior while considering the actions of other missiles to avoid mutual interference. MADDPG is designed to handle such coordination by using shared global information during training. This allows each missile to consider the actions of other missiles and learn how to coordinate effectively to approach the UAV faster while avoiding collisions. Second, the action space of missiles is continuous because the missiles are controlled by flight angular velocities. This requires the algorithm to handle continuous action parameters rather than discrete choices. MADDPG can effectively handle continuous action spaces and is very suitable for multi-agent systems that require precise control. Third, MADDPG allows for centralized training where all missiles can access global information, which helps in learning effective coordination strategies. During execution, each missile makes decisions based only on local information, which reduces the computational burden and improves real-time decision-making. Finally, for the MATD3 algorithm that uses twin critics, as the number of missiles increases, the complexity of training and maintaining more network parameters rises. Therefore, in situations with a larger number of agents, the simplicity and efficiency of MADDPG make it a better choice, as it can reduce the computational cost associated with training and maintaining critic networks.

We used the EA-MADDPG algorithm to train actor networks for each missile, with ${\beta }_i\left(k\right)$, ${\alpha }_i\left(k\right)$, and $\parallel p_m\left(k\right)-p_i\left(k\right)\parallel$ as inputs for each actor and $u_{{\beta }_i}\left(k\right)$ and $u_{{\alpha }_i}\left(k\right)$ as the continuous action outputs. The goal was for the missiles to move toward the UAV more quickly while avoiding collisions with each other and with the ground, thereby completing the interception task. Each missile was assigned a separate critic network, which used the inputs and outputs of the corresponding actor network as inputs and output q-value.

4. EA-MADDPG

Figure 5 shows the framework of EA-MADDPG. Compared to MADDPG, EA-MADDPG has two additional components: the action filter and the optimal replay buffer ${\mathcal{D}}^{*}$. During the experience collection phase, we introduce the guided mode, as indicated by the blue arrows in the figure. After the policy network of agent i outputs the action $a_i$ based on the observation $o_i$, $a_i$ is input into the action filter which outputs filtered action $a_i^{Guided}$ where $a_i^{Guided}\in {\mathcal{A}}_i$ and $i=1,\cdots ,N$. Once agents have completed the filtered actions, we obtain an experience tuple $(\mathbf{x},a^{Guided},r,{\mathbf{x}}^{\prime })$ that is stored in optimal replay buffer ${\mathcal{D}}^{*}$ where $a^{Guided}=\{a_1^{Guided},\cdots ,a_N^{Guided}\}$. The steps indicated by the brown arrows in the figure are essentially the same as those in the MADDPG algorithm, with two differences: first, experience tuples $(\mathbf{x},a,r,{\mathbf{x}}^{\prime })$ were stored simultaneously in both buffers, and second, sampling was conducted from the two buffers in proportion $\eta$. Next, we provide a detailed introduction to the EA-MADDPG algorithm.

Figure 5. The structure of the EA-MADDPG algorithm.

4.1. Equal Reward Setting

In the training of the MADDPG algorithm, a set of reward functions $\left\{R_i(\mathbf{x},a,{\mathbf{x}}^{\prime })\right\}$ is established, enabling agents to receive rewards through these functions each time they complete actions within the environment. In EA-MADDPG, we introduce an additional set of reward functions $\left\{R_i^{eq}(\mathbf{x},a)\right\}$ and a reward setting called the equal reward setting to further enhance the training capabilities.

The key characteristic of the equal reward setting is that every state has the same maximum reward:

$$R_i^{eq}(\mathbf{x},a)\le C_{max}$$

(15)

$$R_i(\mathbf{x},a,{\mathbf{x}}^{\prime })\le C_{max}$$

(16)

where $C_{max}\in \mathbb{R}$ is a constant value. For any given state $\mathbf{x}$, the maximum reward value that can be obtained is equal.

Next, we propose two specific reward functions that meet the above setting. For each agent i:

$$\begin{array}{cc}{R}_i^{eq}\hfill & =f({\beta }_i\left(k\right),u_{{\beta }_i}\left(k\right),\lambda )+f({\alpha }_i\left(k\right),u_{{\alpha }_i}\left(k\right),\lambda )\hfill \\ & +\sum _{j\ne i}{\varsigma }_1[f({\beta }_j\left(k\right),u_{{\beta }_j}\left(k\right),\lambda )+f({\alpha }_j\left(k\right),u_{{\alpha }_j}\left(k\right),\lambda )]\hfill \end{array}$$

(17)

$$\begin{array}{cc}{R}_i\hfill & =f({\beta }_i^{\prime }\left(k\right),u_{{\beta }_i}(k-1),\lambda )+f({\alpha }_i^{\prime }\left(k\right),u_{{\alpha }_i}(k-1),\lambda )\hfill \\ & +\sum _{j\ne i}{\varsigma }_2[f({\beta }_j^{\prime }\left(k\right),u_{{\beta }_j}(k-1),\lambda )+f({\alpha }_j^{\prime }\left(k\right),u_{{\alpha }_j}(k-1),\lambda )]\hfill \\ & +R_i^{col}+R_i^{ground}\hfill \end{array}$$

(18)

where

$$f(\psi ,\kappa ,\Lambda )=\left\{\begin{array}{cc}{\displaystyle \frac{\kappa }{\Lambda }}+1\hfill & \mathrm{If}\psi \ge \Lambda \hfill \\ -{\displaystyle \frac{\kappa }{\Lambda }}+{\displaystyle \frac{\psi }{\Lambda }}+2\hfill & \mathrm{If}\Lambda >\kappa \ge \psi \mathrm{and}\psi >-\Lambda \hfill \\ {\displaystyle \frac{\kappa }{\Lambda }}-{\displaystyle \frac{\psi }{\Lambda }}+2\hfill & \mathrm{If}\psi >\kappa >-\Lambda \mathrm{and}\psi <\Lambda \hfill \\ -{\displaystyle \frac{\kappa }{\Lambda }}+1\hfill & \mathrm{If}\psi \le -\Lambda \hfill \end{array}\right.$$

(19)

$\lambda =Tmax|u_{{\beta }_i}|=Tmax|u_{{\alpha }_i}|$ denotes the maximum angular velocity that can change the agent’s direction from both vertical and horizontal perspectives in one step. ${\varsigma }_1,{\varsigma }_2\in \mathbb{R}$. $R_i^{col}$ denotes the penalty for agent i after agent i collides with other agents, and $R_i^{ground}$ denotes the penalty for agent i after agent i hits the ground:

$$R_i^{col}=\left\{\begin{array}{cc}-100\hfill & \mathrm{Agent} i \mathrm{collides}\hfill \\ 0\hfill & \mathrm{else} \hfill \end{array}\right.$$

(20)

$$R_i^{ground}=\left\{\begin{array}{cc}-100\hfill & \mathrm{Agent} i \mathrm{hits} \mathrm{the} \mathrm{ground}\hfill \\ 0\hfill & \mathrm{else} \hfill \end{array}\right.$$

(21)

Next, let us take $f({\alpha }_i\left(k\right),u_{{\alpha }_i}\left(k\right),\lambda )$ in $R_i^{eq}$ as an example to explain why $R_i^{eq}$ and $R_i$ satisfy equal reward setting.

As shown in Figure 6, it is the plane parallel to O–$xy$ and the angle between the two blue dashed lines in the figure represents the range within which the missile can change its velocity direction in one step from horizontal perspectives. The blue dashed lines about $v_{i-xy}\left(k\right)$ are symmetric. When ${\alpha }_i\left(k\right)$ is greater than $\lambda$, the reward is maximized at 2 when $u_{{\alpha }_i}\left(k\right)=\lambda$ and decreases as $u_{{\alpha }_i}\left(k\right)$ decreases. Similarly, if ${\alpha }_i\left(k\right)$ is less than $-\lambda$, the reward is also maximized at 2 when action $u_{{\alpha }_i}\left(k\right)=-\lambda$ and decreases as $u_{{\alpha }_i}\left(k\right)$ increases. If the angle ${\alpha }_i\left(k\right)$ is within $[-\lambda ,\lambda ]$, the reward is maximized at 2 when action $u_{{\alpha }_i}\left(k\right)={\alpha }_i\left(k\right)$ and decreases as $u_{{\alpha }_i}\left(k\right)$ changes to both sides. The above is the general logic of the function $f({\alpha }_i\left(k\right),u_{{\alpha }_i}\left(k\right),\lambda )$ and ensures that the maximum reward obtained from it is 2. Therefore, extending to each term of $R_i^{eq}$, they all meet the equal reward setting, meaning that $R_i^{eq}$ satisfies the equal reward setting. $R_i$ has a similar composition to $R_i^{eq}$; therefore, $R_i$ also possesses this property.

Figure 6. Schematic of $f({\alpha }_i\left(k\right),u_{{\alpha }_i}\left(k\right),\lambda )$.

4.2. Optimal Replay Buffer ${\mathcal{D}}^{*}$

MADDPG uses a single experience replay buffer $\mathcal{D}$. In ME-MADDPG, we introduce an additional replay buffer ${\mathcal{D}}^{*}$ called optimal replay buffer. Figure 7 shows that $e_{new}$ is a tuple containing the experience saved in the current step. $e_1,\cdots ,e_{B_s^{*}}$ are all experience tuples stored in the buffer ${\mathcal{D}}^{*}$, and $B_s^{*}\in \mathbb{R}$ denotes the maximum storage size of ${\mathcal{D}}^{*}$. When storing experiences, the method of storing experience tuples in empty slots is the same as $\mathcal{D}$. When experience tuples in ${\mathcal{D}}^{*}$ are being overwritten, ${\mathcal{D}}^{*}$ requires an additional step of judgment. For example, as shown in the figure, when $e_{new}$ is about to cover $e_4$ since r contains the rewards of all agents at this step, we select the minimum reward value $min\left(r\right)$ in $e_{new}$ and compare it with the corresponding minimum reward value $min\left(r_{e_4}\right)$ in $e_4$. The experience tuple corresponding to the larger of the two compared reward values is then stored.

Figure 7. Optimal replay buffer ${\mathcal{D}}^{*}$.

During training, samples can be extracted from $\mathcal{D}$ and ${\mathcal{D}}^{*}$ in proportion $\eta ={\displaystyle \frac{N_{{\mathcal{D}}^{*}}}{S}}$ and then mixed for learning:

$$\eta ={\eta }_0{e}^{-\hslash c_{episode}}$$

(22)

$$S=N_{{\mathcal{D}}^{*}}+N_{\mathcal{D}}$$

(23)

$N_{{\mathcal{D}}^{*}}\in \mathbb{R}$ denotes the number of experience tuples collected from ${\mathcal{D}}^{*}$. $N_{\mathcal{D}}\in \mathbb{R}$ denotes the number of experience tuples collected from $\mathcal{D}$. $c_{episode}\in \mathbb{R}$ denotes the current episode value. $\hslash \in \mathbb{R}$ denotes the decay rate of $\eta$. ${\eta }_0\in \mathbb{R}$ denotes the initial proportion we set.

4.3. Action Filter

In the training of reinforcement learning, the exploration performance of agents is critical to the speed and effectiveness of learning. Therefore, we introduce an action filter. In the introduction above, we introduced the guided mode, using the action filter. As shown in Figure 8, after agents derive the action set $\{a_1,a_2,\cdots ,a_N\}$ and input it into the action filter, we generate a batch of random noise $\{{\mathcal{N}}_1,\cdots ,{\mathcal{N}}_{Nq}\}$ from $\mathcal{N}$, where $q\in \mathbb{R}$ denotes the repeated exploration coefficient, and integrate it with the action set $\{a_1,a_2,\cdots ,a_N\}$ to create a collection of exploratory actions $\{a_1^1,a_1^2,\cdots ,a_1^q,a_2^1,\cdots ,a_N^q\}$. The observations $\{o_1,o_2,\cdots ,o_N\}$, the actions $\{a_1,a_2,\cdots ,a_N\}$, and these exploratory actions are then fed into the set of reward functions $\left\{R_i^{eq}\right\},i=1,\cdots ,N$ mentioned above to generate a set of exploratory rewards $\{r_i^t=R_i^{eq}(\mathbf{x},a_1,a_2,\cdots ,a_i^t,a_{i+1},\cdots ,a_N)\}$, $i=1,\cdots ,N$,$t=1,\cdots ,q$. We input them into the filter:

$$a_i^{Guided}=arg\underset{a_i^t}{max}\left(R_i^{eq}(\mathbf{x},a_1,a_2,\cdots ,a_i^t,a_{i+1},\cdots ,a_N)\right)$$

(24)

Subsequently, agents execute $\left\{a_i^{Guided}\right\},i=1,\cdots ,N$, obtain rewards $\left\{r_i\right\},i=1,\cdots ,N$, and move to the next state ${\mathbf{x}}^{\prime }$.

Figure 8. Action filter under guided mode.

The noise $\mathcal{N}$ is generated by:

$$\mathcal{N}={\mathcal{N}}_{a}K{e}^{-\vartheta c_{episode}}$$

(25)

where $K\in {\mathcal{A}}_i$ and the values in K follow a standard normal distribution. ${\mathcal{N}}_a\in \mathbb{R}$ denotes the amplitude of noise. $\vartheta \in \mathbb{R}$ denotes the noise decay rate.

During training, we have a certain probability $p^{Guided}$ of choosing the guided mode. As the training progresses, the probability gradually approaches zero:

$$p^{Guided}=p_0^{Guided}{e}^{-\delta c_{episode}}$$

(26)

where $\delta \in \mathbb{R}$ denotes the probability decay rate we set. $p_0^{Guided}\in \mathbb{R}$ denotes the initial probability. Algorithm 1 presents the pseudocode for EA-MADDPG.

Algorithm 1 EA-MADDPG

for episode = 1 to M do    Obtain $p^{Guided}$ and $\eta$    Initialize a random process $\mathcal{N}$ for action exploration    Receive initial state $x$    Determine whether to enter guided mode based on probability $p^{Guided}$    if Guided Mode then      for $t=1$ to max-episode-length do         for each agent i, select action $a_i={\mathit{\mu }}_{{\theta }_i}\left(o_i\right)$         Obtain the exploratory actions $\{a_1^1,a_1^2,\cdots ,a_1^q,a_2^1,\cdots ,a_N^q\}$ by integrating it with a set of random noise $\{{\mathcal{N}}_1,\cdots ,{\mathcal{N}}_{Nq}\}$         Input to the action filter to obtain $\left\{a_i^{Guided}\right\}$: $$a_i^{Guided}=arg\underset{a_i^t}{max}\left(R_i^{eq}(\mathbf{x},a_1,a_2,\cdots ,a_i^t,a_{i+1},\cdots ,a_N)\right)$$         Execute actions $\left\{a_i^{Guided}\right\}$ and observe reward r and new state ${\mathbf{x}}^{\prime }$         Store $(\mathbf{x},a^{Guided},r,{\mathbf{x}}^{\prime })$ in replay buffer ${\mathcal{D}}^{*}$         $\mathbf{x}\leftarrow {\mathbf{x}}^{\prime }$      end for    else      for $t=1$ to max-episode-length do         For each agent i, select action $a_i={\mathit{\mu }}_{{\theta }_i}\left(o_i\right)+\mathcal{N}$ with respect to the current policy and exploration         Execute actions $a=\{a_1,a_2,\cdots ,a_N\}$ and observe reward r and new state ${\mathbf{x}}^{\prime }$         Store $(\mathbf{x},a,r,{\mathbf{x}}^{\prime })$ in replay buffer $\mathcal{D}$ and optimal replay buffer ${\mathcal{D}}^{*}$         $\mathbf{x}\leftarrow {\mathbf{x}}^{\prime }$         for agent $i=1$ to N do           Sample a random mini-batch of S samples $({\mathbf{x}}^j,a^j,r^j,{\mathbf{x}}^{\prime j})$ in proportion $\eta$ from $\mathcal{D}$ and ${\mathcal{D}}^{*}$           Set $y^j=r_i^j+\gamma {\mathit{Q}}_{{\varphi }_i^{\prime }}^{{\mathit{\mu }}^{\prime }}({\mathbf{x}}^{\prime j},a_1^{\prime },a_2^{\prime },\cdots ,a_N^{\prime }){|}_{a_k^{\prime }={\mathit{\mu }}_{\mathbf{k}}^{\prime }\left(o_k^j\right)}$           Update critic by minimizing the loss: $$\mathcal{L}\left({\varphi }_i\right)={\displaystyle \frac{1}{S}}\sum _{j=1}^S{\left(y^j-{\mathit{Q}}_{{\varphi }_i}^{\mathit{\mu }}({\mathbf{x}}^j,a_1^j,\cdots ,a_N^j)\right)}^2$$           Update actor using the sampled policy gradient: $${▽}_{{\theta }_i}J\approx {\displaystyle \frac{1}{S}}\sum _{j=1}^S{▽}_{{\theta }_i}{\mathit{\mu }}_i\left(o_i^j\right){▽}_{a_i}{\mathit{Q}}_{{\varphi }_i}^{\mathit{\mu }}({\mathbf{x}}^j,a_1^j,\cdots ,a_i,\cdots ,a_N^j){|}_{a_i={\mathit{\mu }}_i\left(o_i^j\right)}$$         end for         Update target network parameters for each agent i: $$\begin{gathered} {\theta }_i^{\prime }\leftarrow \tau {\theta }_i+(1-\tau ){\theta }_i^{\prime } \\ {\varphi }_i^{\prime }\leftarrow \tau {\varphi }_i+(1-\tau ){\varphi }_i^{\prime } \end{gathered}$$      end for    end if end for

5. Simulation and Discussion

Comprehensive simulation results are given in this section to examine the performance of the EA-MADDPG algorithm. Two scales were considered, namely $N=3$ and $N=6$.

5.1. Simulation Setting

Table 2, Table 3 and Table 4 below list the values of relevant parameters for two training environments. Table 5 lists the values of parameters related to training.

Table 2. Initial missile positions when $N=3$.

Parameter	Value	Explanation of Parameter
$(p_{01}^x,p_{01}^y,p_{01}^z)$	(10, 10, 0)	Initial coordinates of missile 1 (m)
$(p_{02}^x,p_{02}^y,p_{02}^z)$	(0, −100, 0)	Initial coordinates of missile 2 (m)
$(p_{03}^x,p_{03}^y,p_{03}^z)$	(−50, 200, 0)	Initial coordinates of missile 3 (m)

Table 3. Initial missile positions when $N=6$.

Parameter	Value	Explanation of Parameter
$(p_{01}^x,p_{01}^y,p_{01}^z)$	(10, 10, 0)	Initial coordinates of missile 1 (m)
$(p_{02}^x,p_{02}^y,p_{02}^z)$	(0, -100, 0)	Initial coordinates of missile 2 (m)
$(p_{03}^x,p_{03}^y,p_{03}^z)$	(−50, 200, 0)	Initial coordinates of missile 3 (m)
$(p_{04}^x,p_{04}^y,p_{04}^z)$	(10, 100, 0)	Initial coordinates of missile 4 (m)
$(p_{05}^x,p_{05}^y,p_{05}^z)$	(−30, −40, 0)	Initial coordinates of missile 5 (m)
$(p_{06}^x,p_{06}^y,p_{06}^z)$	(−150, 0, 0)	Initial coordinates of missile 6 (m)

Table 4. Common parameters for both environments.

Parameter	Value	Explanation of Parameter
$v_{max}$	50	The maximum linear speed (m/s)
$\Theta$	${\displaystyle \frac{\pi }{6}}$	The maximum angular velocity in the plane O–$xy$ and the vertical plane (rad/s)
T	0.05	Time per step (s)
$\phi$	${\displaystyle \frac{\pi }{3}}$	The angle between the generatrix of the cone and the z-axis (rad)
$\mathcal{R}$	2100	The radius of the hemisphere P (m)
${\mathcal{R}}_g$	300	The radius of the green circle (m)
${\mathcal{R}}_r$	200	The radius of the red circle (m)
${\mathcal{R}}_d$	30	Interference distance of the device (m)
${\mathcal{R}}_s$	100	The minimum distance allowed for the UAV to approach the target (m)

Table 5. Training’s parameters.

Parameter	Value	Explanation of Parameter
q	3	Repeated exploration coefficient
${\eta }_0$	0.3	Initial proportion
ℏ	0.001	The decay rate of $\eta$
$p_0^{Guided}$	0.3	Initial probability
$\delta$	0.005	The probability decay rate of $p^{Guided}$
${\mathcal{N}}_a$	0.2	The amplitude of noise
$\vartheta$	0.001	The noise decay rate
$\gamma$	0.95	Discounted rate
$\varrho$	0.02	Learning rate
$\tau$	0.02	Soft update rate
$B_s$	50,000	Buffer size of $\mathcal{D}$
$B_s^{*}$	5000	Buffer size of ${\mathcal{D}}^{*}$
S	64	Batch size
$n_{step}$	200	Training frequency
$\varsigma$	0.5	Gradient clipping

Table 6 and Table 7 list the dimensionality parameters of each layer in the actor networks and critic networks, respectively. For each agent, both the actor and critic networks (including the target networks for actor and critic) employed a classic feedforward neural network structure, a multi-layer perceptron (MLP). Each network comprised an input layer, a hidden layer, and an output layer.

Table 6. The (target) actor network’s parameters.

Layer	Input Dimension	Output Dimension
Input layer	3	128
Hidden layer	128	128
Output layer	128	2

Table 7. The (target) critic network’s parameters.

Layer	Input Dimension	Output Dimension
Input layer	3N	128
Hidden layer	128	128
Output layer	128	1

In the actor network, the input layer was fully connected and used a rectified linear unit (ReLU) as the activation function, with a normalization function applied to the batch input data. In the critic network, the input layer was also fully connected and used a rectified linear unit (ReLU) as the activation function, but it passed the input data directly using an identity function. The hidden layers in both networks were fully connected, with ReLU as the activation function. The output layer in both networks was fully connected; in the actor network, the hyperbolic tangent activation function was used.

For reference, we used the MADDPG [17], ME-MADDPG [40], MADDPG-L [41], MAPPO [19] and MATD3 [20] algorithms in the same environment. At the same time, we replaced the multiple critics in the EA-MADDPG framework with a single universal critic to evaluate the performance of all agents. This framework was called EA-MADDPG-S. The training was conducted on a computer equipped with an AMD Ryzen Threadripper PRO 3975WX 32-core CPU, 128 GB of RAM, and an NVIDIA RTX 4090 GPU.

5.2. Simulation Results and Discussion

5.2.1. Training Result

Through training, we obtained the following results:

Figure 9 illustrates the curves of the average reward of three missiles throughout the training episodes for $N=3$. Figure 10 illustrates the curves of the average reward of six missiles throughout the training episodes for $N=6$.

Figure 9. The courses of reward for missiles for $N=3$.

Figure 10. The courses of reward for missiles $N=6$.

The reward functions (18) for missiles reflected our expectation of a joint optimal strategy for multiple missiles, where all missiles approached the UAV more quickly while avoiding collisions with each other. If one missile performed poorly during training, resulting in a lower individual reward, it also affected the rewards of the other missiles, causing the overall rewards of all missiles to decrease.

From Figure 9 and Figure 10, we can see that the training performance of the MADDPG-L and MAPPO algorithms was not very effective in the current environment, as the reward curves showed little change from the beginning to the end of training. Therefore, in the following convergence analysis of the reward curves we do not include these two algorithms.

In the three vs. one environment, as shown in Figure 9, missile collisions and crashes into the ground occurred at the beginning of the training, leading to low and even negative reward values. Through continuous training, the missiles learned to fly toward the UAV while avoiding collisions, leading to an increase in reward values. In the subsequent training process, the convergence values of the reward curves under the EA-MADDPG algorithm were higher than those of other algorithms. This indicated that under this algorithm, the three missiles could approach the UAV more quickly while avoiding collisions with each other.

In the six vs. one environment, as shown in Figure 10, the reward curves under the EA-MADDPG algorithm converged, and both the convergence speed and convergence values were greater than those of the reward curves under other algorithms. As the number of missiles increased, the difficulty of training began to rise. Within the same episodes, the reward curve of missile 1 was negative under ME-MADDPG. The reward curves of missiles 3, 4, and 5 were negative under MADDPG. The reward curves of missiles 1, 4, and 5 were negative under MATD3. This indicated that the corresponding missiles were still experiencing collisions with each other or crashing into the ground, which also affected the training and reward values of the other missiles. Under EA-MADDPG, the missiles could actively explore and collect good experiences through the action filter, quickly learning new strategies to overcome the aforementioned issues. To prevent excessive learning for this portion of experiences and stabilize subsequent learning effects and reward curves, EA-MADDPG mixed these experiences with normally obtained experiences in optimal replay buffer ${\mathcal{D}}^{*}$. At the same time, it reduced the probability $p^{Guided}$ and the sampling ratio $\eta$ according to certain decay rates.

We compared the reward curves of EA-MADDPG and EA-MADDPG-S in the three vs. one environment. As shown in Figure 11, the reward curve of missile 1 under EA-MADDPG-S converged faster than that under EA-MADDPG. However, the convergence speed of missile 2 was slower under EA-MADDPG-S compared to EA-MADDPG. For missile 3, the convergence speeds of both algorithms were roughly the same. Nonetheless, the final convergence values for all missiles under EA-MADDPG were higher than those under EA-MADDPG-S. From the comparison of the reward curves above, we conclude that training with a single critic was less effective than having a separate critic for each agent.

Figure 11. Comparison of reward curves under the EA-MADDPG and EA-MADDPG-S algorithms.

As shown in Figure 12, EA-MADDPG required additional time consumption compared to MADDPG. Analyzing EA-MADDPG’s training process, in the action filter, we needed to introduce a batch of noise to generate exploratory actions, calculate the $\left\{R_i^{eq}\right\}$, and select $\left\{a_i^{Guided}\right\}$ based on $\left\{R_i^{eq}\right\}$. All these steps required additional time consumption. Additionally, storing and sampling from ${\mathcal{D}}^{*}$ also incurred extra time costs. When $p^{Guided}$ and the sampling ratio $\eta$ are set improperly, the extra time consumption increases. The algorithm analysis and experiments demonstrated that both the action filter and the optimal replay buffer design introduced additional time complexity.

Figure 12. Training time consumption of algorithms in 3 vs. 1 environment.

5.2.2. Policy Networks Testing

Next, we evaluated the trained policy networks in the three vs. one environment. The testing procedure was as follows: First, we selected several initial positions and strike coordinates for the UAV, observed the interception trajectories of three missiles and the UAV, conducted 100 tests for each position, and compared $k_{avg}^{*}$ and $d_{avg}$ under different algorithms for each position, where $k_{avg}^{*},d_{avg}\in \mathbb{R}$ denote the average values of $k^{*}$ and d, respectively. Second, 1000 tests were conducted with randomly generated initial positions and strike coordinates. We compared the percentage ℘ of successful interceptions under different algorithms across these 1000 trials.

As shown in Figure 13, the missile trajectories under the EA-MADDPG algorithm most closely matched the trajectories we expected: three missiles were all able to quickly move towards the UAV that was performing small-amplitude irregular movements. From the trajectories under the ME-MADDPG, it can be seen that although the three missiles no longer crashed into the ground or collided with each other, they were still unable to approach the target UAV quickly. This explained the reason for the lower convergence values of the curves of ME-MADDPG mentioned above. The trajectories under MADDPG showed that missile 2 and missile 3 achieved better training results because they could move towards the UAV, while missile 1 did not perform well. Similar to MADDPG, the missile trajectories under the MATD3 exhibited the same performance. Missile 2 and missile 3 could move towards the UAV, while missile 1 did not perform well and only flew straight up. From the trajectory plots, we can explain why the convergence values of the reward curves for the other algorithms were lower than those of the EA-MADDPG algorithm.

Figure 13. The trajectories of missiles under four different algorithms with varying initial and target positions.

As shown in Figure 14 and Figure 15, the two graphs intuitively demonstrated the speed of UAV interception under three algorithms and did not show data under ME-MADDPG because its interception success rate was too low (the interception success percentage is shown below), making the interception data not referential. Under the EA-MADDPG algorithm, positions 1 to 6 required significantly fewer steps for interception, and the UAV was intercepted at a longer distance, thus showing a greater advantage. Combining the analyses mentioned above, we conclude that EA-MADDPG outperformed other algorithms both in terms of the combined training effect of multiple missiles (i.e., all missiles were ideally trained) and the speed of interception.

Figure 14. $d_{avg}$ under different positions. Positions 1 to 6 correspond to initial positions and target positions in Figure 13a–f.

Figure 15. $k_{avg}^{*}$ under different positions. Positions 1 to 6 correspond to initial positions and target positions in Figure 13a–f.

Next, we conducted 1000 sets of tests to compare the percentage ℘ of successful interceptions under different algorithms.

As shown in Figure 16, EA-MADDPG, with an interception success rate of 99.5%, exceeded MADDPG at 84.3% and MATD3 at 72.3%, while the percentage for ME-MADDPG was only 20.3%. This demonstrated that EA-MADDPG’s training effect in terms of interception success rate surpassed the other three algorithms.

Figure 16. The interception success percentage ℘ among 1000 tests.

6. Conclusions

This paper stemmed from the reality of the progressively increasing threat of drones and the ongoing development of multi-agent reinforcement learning. In response to the growing threat of UAVs, this paper combined multi-agent reinforcement learning technology with the use of missiles to explore the issue of using missiles for UAV defense in three-dimensional space, applying missiles in defense against UAVs.

This paper proposed an improved variant of the MADDPG algorithm called EA-MADDPG. Compared to the original algorithm, the EA-MADDPG algorithm had two distinct advantages: first, it enhanced the exploration of multiple agents through the action filter, allowing the agents to be more “proactive” in exploring and updating their strategies when they were experiencing poor training outcomes; second, through the design of the equal-reward setting and the optimal replay buffer, the agents were able to more efficiently utilize the experiences gained from “proactive” exploration, thereby improving the efficiency and effectiveness of the training.

This study explored a scenario with a single UAV. However, in reality, UAVs often operate in groups to perform tasks. Therefore, in the future, we will further investigate multi-UAV scenarios and multi-agent reinforcement learning strategies for multi-target interception tasks. Additionally, balancing exploration and exploitation during agent training has always been a crucial aspect of reinforcement learning research. In the EA-MADDPG framework presented in this paper, the series of exploratory actions were generated by adding multiple simple random noises. This design still has considerable room for improvement. Future work will consider incorporating Ornstein–Uhlenbeck noise for research and will also refer to the action selection principles based on relevant quantum theory from the QiRL framework and the prioritization principles for experiences in the replay pool from the QiER framework for more in-depth research.