Anti-Jamming Decision-Making for Phased-Array Radar Based on Improved Deep Reinforcement Learning

Abstract

In existing phased-array radar systems, anti-jamming strategies are mainly generated through manual judgment. However, manually designing or selecting anti-jamming decisions is often difficult and unreliable in complex jamming environments. Therefore, reinforcement learning is applied to anti-jamming decision-making to solve the above problems. However, the existing anti-jamming decision-making models based on reinforcement learning often suffer from problems such as low convergence speeds and low decision-making accuracy. In this paper, a multi-aspect improved deep Q-network (MAI-DQN) is proposed to improve the exploration policy, the network structure, and the training methods of the deep Q-network. In order to solve the problem of the $ϵ$-greedy strategy being highly dependent on hyperparameter settings, and the Q-value being overly influenced by the action in other deep Q-networks, this paper proposes a structure that combines a noisy network, a dueling network, and a double deep Q-network, which incorporates an adaptive exploration policy into the neural network and increases the influence of the state itself on the Q-value. These enhancements enable a highly adaptive exploration strategy and a high-performance network architecture, thereby improving the decision-making accuracy of the model. In order to calculate the target value more accurately during the training process and improve the stability of the parameter update, this paper proposes a training method that combines n-step learning, target soft update, variable learning rate, and gradient clipping. Moreover, a novel variable double-depth priority experience replay (VDDPER) method that more accurately simulates the storage and update mechanism of human memory is used in the MAI-DQN. The VDDPER improves the decision-making accuracy by dynamically adjusting the sample size based on different values of experience during training, enhancing exploration during the early stages of training, and placing greater emphasis on high-value experiences in the later stages. Enhancements to the training method improve the model’s convergence speed. Moreover, a reward function combining signal-level and data-level benefits is proposed to adapt to complex jamming environments, which ensures a high reward convergence speed with fewer computational resources. The findings of a simulation experiment show that the proposed phased-array radar anti-jamming decision-making method based on MAI-DQN can achieve a high convergence speed and high decision-making accuracy in environments where deceptive jamming and suppressive jamming coexist.

1. Introduction

With the development of modern electronic countermeasure (ECM) technologies and computing power, radar jamming techniques have become increasingly complex and diverse [1,2]. Consequently, radar anti-jamming decision-making plays an increasingly important role in modern radar applications [3]. In 2006, Haykin introduced the concept of cognitive radar [4], and foresightedly highlighted the application prospects of Bellman’s Dynamic Programming Equation in adaptive algorithms for radar signal processing research. In order to address the problems of intelligent decision-making in radar anti-jamming, existing research analyzes the problem from the perspectives of game theory and reinforcement learning (RL).

Generally, the interaction between radar systems and jammers can be modeled as a two-player zero-sum (TPZS) game [5,6], where both sides attempt to select optimal strategies to maximize their own gains and maximize losses for the opponent [7]. The strategy that brings both sides to a steady state is referred to as the Nash Equilibrium (NE) [8] strategy. However, in complex jamming environments, the model faces a large state and action space, and the information obtained by the model is usually limited. Strategy optimization based solely on game theory struggles to achieve absolute rationality, thus requiring strategies to be gradually improved through continuous learning steps.

As a branch of machine learning (ML), RL primarily addresses exploring and learning optimal strategies. RL, which shows great application potential, has achieved significant progress in energy management [9], robotic control [10], communication network optimization [11], and in many other fields [12,13,14]. RL aims to achieve a stated goal by continuously learning and adapting its strategy in a dynamic environment, and learning how to take actions to maximize rewards. Driven by the wave of intelligent development, the introduction of the RL algorithm provides radar with an adaptive and dynamic anti-jamming strategy optimization method, which can effectively deal with the decision-making difficulties brought about by the diversity of jamming signals. As a typical RL algorithm, Q-learning uses a value function approximation represented by a Q-value table. Q-learning provides an effective solution for low-dimensional decision-making problems, but as the state and action spaces grow, the size of the Q-value table increases exponentially, which not only slows down the computation speed but also severely increases the cost of the decision-making system [15].

Compared to table-based value approximation, the deep Q-network (DQN) employs neural networks to approximate value functions and uses experience replay to reduce temporal correlation among data samples. Experiments have shown that DQN substantially outperforms Q-learning in high-dimensional state and action spaces. With the application of DQN, the radar anti-jamming strategy model can accommodate more anti-jamming actions to cope with the complex jamming environment.

However, there are still drawbacks to DQN. Since action evaluation and selection are performed by the same network, DQN tends to overestimate action values to varying degrees. And this overestimation is magnified, step by step, during the training process, which is called ‘bootstrapping’, causing local optima and convergence difficulties. To address these issues, the double deep Q-network (DDQN) employs separate evaluation and target networks to mitigate action value overestimation. At the same time, DDQN improves the calculation of the target Q-value to further mitigate overestimation compared to DQN. Up to this point, DDQN has made improvements to address the overestimation of the value function, but in more complex environments, the performance of DDQN still faces challenges. First, DDQN relies on the $ϵ$-greedy exploration strategy, which tends to select random rather than optimal actions to some extent. The usual approach to solving the above problem is to gradually decrease $ϵ$ with the number of training steps. But this solution relies heavily on the initial value of $ϵ$ and the rate of descent; inappropriate values will lead to local optima and a slow convergence rate. Second, traditional experience replay collects experiences with equal probability, which ignores differences between experiences. In a dynamic environment, the speed of model convergence will be affected. Lastly, hyperparameters such as learning rate, discount factor, and target network update frequency significantly impact DDQN convergence speed and generalization ability [16], and fixed hyperparameter settings are inadequate for complex tasks. Regarding the above issues, some models optimize the DDQN network structure [17,18,19,20] by employing a dueling double deep Q-network [21], which reconstructs the Q-value as a state-value function and an action advantage value, allowing the agent to learn the state-value more frequently and accurately. Other models choose to improve experience replay [22,23], adopting prioritized experience replay (PER) [24] to preferentially replay high-value experiences, thereby improving data utilization and learning speed.

In existing research, radar anti-jamming decision-making methods mainly focus on three aspects, namely, overall anti-jamming strategy optimization, anti-jamming waveform strategy optimization, and frequency selection strategy optimization. Ref. [19] proposed a cognitive radar anti-jamming strategy generation algorithm based on a dueling double deep Q-network, which shows that the dueling double deep Q-network generates a more effective anti-jamming strategy than DQN. Ref. [20] generates the anti-jamming waveform strategy for airborne radars based on the dueling double deep Q-network as well as PER, and uses an iterative transformation method (ITM) to generate the time domain signals of the optimal waveform strategy. The simulation results demonstrate the improved effect of the dueling double deep Q-network and PER. Yet, these improvements still exhibit slow convergence and potential local optima in complex jamming environments. Ref. [25] established both radar and jammer as agents, and found the optimal power allocation strategy for the radar by using the multi-agent deep deterministic policy gradient (MADDPG) algorithm based on multi-agent reinforcement learning. Ref. [26] presented an NE strategy for radar-jammer competitions involving dynamic jamming power allocation by utilizing an end-to-end deep reinforcement learning (DRL) algorithm, considering that the competition between the radar and the jammer involves imperfect information. Additionally, Ref. [27] proposed a slope-varying linear frequency modulation (SV-LFM) signal strategy generation method based on DQN, enabling radar systems to learn optimal anti-jamming strategies without prior information.

Moreover, compared to other types of radar systems, phased-array radars offer greater flexibility in selecting the positions and number of auxiliary antennas. This capability provides higher adaptability in dynamically changing jamming environments. Therefore, to address the above problems, this paper makes comprehensive improvements to the aforementioned methods and proposes a phased-array radar anti-jamming decision-making approach with high accuracy in environments where multiple types of jamming coexist. A noisy network constructed using factorized Gaussian noise [28] is incorporated to dynamically adjust action selection randomness during training, thereby enhancing exploration efficiency. Moreover, n-step temporal difference learning [29] is employed to update the Q-value using multi-step returns, accelerating reward propagation. The above improvements are combined with a dueling network and DDQN to achieve comprehensive performance enhancement. Additionally, as an improvement of double-depth priority experience replay (DDPER) [30], this paper proposes variable double-depth priority experience replay (VDDPER), which refines experience learning mechanisms by dynamically adjusting the proportion of deep and shallow experiences utilized during training, further accelerating model convergence. The main contributions of this work are as follows:

• To improve the reward convergence speed and decision accuracy, a multi-aspect improved deep Q-network (MAI-DQN) algorithm for phased-array radar anti-jamming decision-making is proposed. In the MAI-DQN, the noisy network based on factorized Gaussian noise is used to improve the exploration policy. And n-step learning and DDQN are used to update the temporal difference target value, thereby improving the reward propagation of long sequence tasks. In addition, the dueling network is used to decompose the Q-value into two parts, thereby introducing the concept of advantage and further improving the network structure. Moreover, target soft update, variable learning rate, and gradient clipping are used to further improve the training effect.
• To enhance the utilization of experience replay across all experiences in the early stages of training and emphasize high-value experiences in the later stages of training, the VDDPER mechanism is proposed as an improvement over DDPER. VDDPER dynamically adjusts the sample sizes for different values of experience to direct the model’s attention toward experiences of different values during training.
• To accurately characterize jamming states in complex environments and enable optimal anti-jamming measures, a novel and effective design of the state space, action space, and reward function—combining data-level and signal-level benefits—is proposed and shown to achieve rapid convergence during training.

The remainder of this paper is organized as follows: In Section 2, the phased-array radar echo signal model, array configuration, and interactive system model for the radar and the jammer competition are introduced. In Section 3, the RL decision-making process and the basic concepts with DQN are analyzed and described. Section 4 presents the implementation of MAI-DQN in phased-array radar anti-jamming decision-making and outlines its improvements in detail. Section 5 describes the experimental setup for several RL algorithms and MAI-DQN, followed by a comparison and analysis of the experimental results. Finally, the conclusion and some future works are shown in Section 6.

2. System Model

In this section, we introduce the basic concept of phased-array radar and analyze the echo signal model. Then, we introduce and analyze the environmental interaction model between the phased-array radar and the jammer.

As a kind of array radar, phased-array radar controls the beam direction by adjusting the phase of individual antenna elements within the array, allowing for rapid scanning, multi-target tracking, and enhanced anti-jamming capabilities. Essentially, it represents a signal processing approach implemented in the analog domain using radio frequency (RF) components and feed networks. This paper focuses on a monostatic uniform linear array (ULA) phased-array radar. Therefore, an accurate echo model is established. When employing a linear frequency modulation (LFM) signal, the single-target echo signal model for an M-element phased-array radar can be expressed as follows:

$$s\left(t\right)=\sqrt{\sigma }\alpha (\Theta )g\left(t\right)exp\left\{j2\pi \left(f_0(t-\tau )+k/2{(t-\tau )}^2+f_{D}t\right)+{\phi }_0\right\},$$

(1)

where $g\left(t\right)$ represents the signal envelope with a pulse width of T; $f_0$ denotes the carrier frequency; $\tau$ represents the target propagation delay; k denotes the signal chirp rate; $f_D$ denotes the target Doppler shift; and ${\phi }_0$ represents the initial phase of the echo signal. Additionally, $\alpha (\Theta )$ denotes the signal steering vector:

$$\alpha \left(\Theta \right)={\left[\begin{array}{cccc}1& e^{-j2\pi d/\lambda sin\Theta }& \dots & e^{-j2\pi \left(M-1\right)d/\lambda sin\Theta }\end{array}\right]}^T,$$

(2)

where d denotes the inter-element spacing of the antenna array, and $\lambda$ denotes the radar wavelength. To avoid grating lobes, most ULA-configured phased-array radars set the element spacing to $2/\lambda$. $\sqrt{\sigma }$ accounts for the influence of the target’s radar cross-section (RCS). In this paper, a log-normal distribution RCS fluctuation model is employed to simulate large naval targets, with the probability density function defined as follows:

$$p\left(\sigma \right)={\displaystyle \frac{exp\left[-{ln}^2(\sigma /{\sigma }_0)/4ln\rho \right]}{\sigma \sqrt{4\pi ln\rho }}},$$

(3)

where ${\sigma }_0$ represents the median RCS value, and $\rho$ denotes the ratio between the average and median RCS values.

Moreover, this paper employs frequency agility (FA) and cover pulse techniques as active anti-jamming strategies. When the radar adopts an FA waveform, the single-target echo signal model can be expressed as follows:

$$s\left(t\right)=\sqrt{\sigma }\alpha (\Theta )g\left(t\right)exp\left\{j2\pi \left(f_m(t-\tau )+k/2{(t-\tau )}^2+f_{D}t\right)+{\phi }_0\right\},$$

(4)

where $f_m$ represents the carrier frequency of the m-th pulse, $f_m=f_0+b\left(m\right)\Delta f$, $b\left(m\right)\in \mathbb{N}$ controls the frequency hopping of the m-th pulse, and $\Delta f$ denotes the minimum frequency hopping interval. When the radar employs a cover pulse waveform, it simultaneously transmits multiple carrier frequencies to deceive jammers into intercepting incorrect frequency points. Assuming that the real signal carrier frequency is $f_0$ this time, the cover pulse waveform echo signal model will not be repeated here.

As shown in Figure 1, when operating in search mode, the radar transmits signals to scan the target region. Part of the signal is reflected by the target and is received as an echo by the radar, while part of the signal is intercepted by the jammer. Using digital radio frequency memory (DRFM) technology, the jammer samples and stores the intercepted signal at high speed. After analyzing the signal characteristics, the jammer decides whether to transmit suppression or deception jamming signals to interfere with the radar in the time, frequency, or energy domains. At the same time, noise signals represented by Gaussian white noise are also received by the radar due to the presence of thermal noise from the electronics in the radar, as well as atmospheric noise and other effects.

Regarding jamming, this paper investigates various types of jamming signals [31,32,33,34], including dense false target jamming, sweep jamming, coherent slice forwarding jamming, and spot jamming, as shown in Figure 2, alongside the target echo and noise signals. Due to the presence of multiple jammers, the above types of jamming may co-exist, and based on the jamming direction, the above types of jamming can be categorized as mainlobe, sidelobe, or combined mainlobe–sidelobe jamming. In addition, random phase perturbations are introduced across all types of jamming to simulate potential uncertainties such as the phase noise from the jammer, phase disturbance strategies, and multipath effects.

Regarding anti-jamming measures and anti-jamming strategies, this paper incorporates frequency agility, cover pulses, the sidelobe canceller (SLC) algorithm, and sidelobe blanking (SLB) as anti-jamming measures. During the game process with the jammer, the model iteratively explores and generates the most effective combination of anti-jamming measures, leading to the derivation of the optimal anti-jamming strategy.

Since electromagnetic waves in free space satisfy the principle of linear superposition, and the target echo signal, jamming signal, and noise are independent, these three signals are generally additive. Under such conditions, the radar’s received signal model can be expressed as follows:

$$R\left(t\right)=A_{s}s\left(t\right)+A_{j}j\left(t\right)+n\left(t\right),$$

(5)

where $s\left(t\right)$ denotes the target echo signal, $j\left(t\right)={\sum }_{h=0}^{H-1}{j}_h\left(t\right)$ represents the jamming signal under one or more jamming, and $n\left(t\right)$ accounts for noise. The amplitude factor of the target echo is given by the following:

$$A_s={10}^{SNR/10}·{\sigma }_n^2,$$

(6)

where $SNR$ represents the signal-to-noise ratio, ${\sigma }_n$ denotes the standard deviation of Gaussian white noise, and the amplitude factor of the jamming signal is as follows:

$$A_j={10}^{JNR/10}·{\sigma }_n^2,$$

(7)

where $JNR$ represents the jamming-to-noise ratio.

After constructing the signal model, additional metrics beyond $SNR$ and $JNR$ are required to comprehensively evaluate the signal quality before and after anti-jamming measures, thereby accurately reflecting the effectiveness of anti-jamming strategies. The first metric is the interference suppression ratio, reflecting the effectiveness of anti-jamming techniques in suppressing jamming signal energy. The interference suppression ratio $ISR$ can be expressed as follows:

$$ISR=10lg\left(P_{j,before}/P_{j,after}\right),$$

(8)

where $P_{j,before}$ denotes the energy of the jamming signal before suppression, and $P_{j,after}$ represents the remaining energy of the jamming signal after suppression.

Another evaluation metric is the radar target detection result after pulse compression and cell-averaging constant false alarm rate (CA-CFAR) processing, which assesses the effectiveness of anti-jamming measures in filtering out false targets. As a classical constant false alarm rate method, CA-CFAR estimates the detection threshold based on a set of neighboring background noise cells and compares it with the signal level in the target cell to determine the presence of a target. Generally, the CA-CFAR detection threshold is given by the following:

$$T_{CA}=\beta ·{\widehat{P}}_n=-ln\left(1-P_{fa}^{1/N_{ref}}\right)·{\sum }_{i=1}^{N_{ref}}{A}_i,$$

(9)

where $\beta$ represents the threshold scaling factor, ${\widehat{P}}_n$ denotes the estimated reference noise level, $P_{fa}$ denotes the predetermined false alarm probability, $N_{ref}$ denotes the number of reference cells, and $A_i$ denotes the amplitude of the echo signal in the i-th reference cell.

Moreover, a set of data-level evaluation metrics is designed within the reward function to assess the effectiveness of anti-jamming measures in the frequency domain.

3. Reinforcement Learning Background

This section begins by outlining several fundamental concepts of reinforcement learning, followed by a description of the algorithmic principles and exploration strategies of DQN.

3.1. Reinforcement Learning Basics

The problems addressed by the RL are generally modeled using a Markov decision process (MDP), which is defined as a four-tuple $\left\{S,A,R,P\right\}$ [35]:

(1)

State: Represents all available information about the environment at any given time. The set of all states forms the state space S. At time step t, the state of the agent is denoted as $s_t$;

(2)

Action: Describes the set of possible actions an agent can take in each state, forming the action space A. At time step t, the action taken by the agent is denoted as $a_t$;

(3)

Reward: Represents the immediate feedback received after taking action $a_t$ at time step t, denoted as $r_{t+1}$, which guides the agent’s behavior optimization;

(4)

Transition probability: Describes the probability of transitioning to state $s_{t+1}$ after executing action $a_t$ in state $s_t$, denoted as $P\left(s_{t+1}|s_t,a_t\right)$. The relationships among S, A, R, and P are shown in Figure 3.

The goal of RL is to find a policy $\pi \left(a|s\right)$ that maximizes the expected reward accumulated by the agent in each state, a process that can be described by the following equation:

$${\pi }^{\ast }={\arg \max }_{\pi }\mathbb{E}\left[G_t\right|\pi ],$$

(10)

where the cumulative reward is defined as follows:

$$G_t=r_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+\dots ={\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k},$$

(11)

where, $\gamma \in \left[0,1\right]$ denotes the discount factor, which measures the importance of future rewards. A value of $\gamma$ closer to 1 implies that the agent emphasizes long-term returns, whereas a value closer to 0 indicates a preference for short-term returns. During optimization, the agent evaluates the quality of policies by learning a value function. Specifically, the state-value function is defined as follows:

$$V^{\pi }\left(s\right)={\mathbb{E}}_{\pi }\left[G_t|s_t=s\right],$$

(12)

which represents the expected cumulative reward obtained from state s when following policy $\pi$. In RL, the state-value function describes the long-term rewards achievable from a given state and, thus, provides an assessment of how favorable a particular state is for the agent. Similarly, under a given policy $\pi$, the action-value function is defined as follows:

$$Q^{\pi }\left(s,a\right)={\mathbb{E}}_{\pi }\left[G_t|s_t=s,a_t=a\right],$$

(13)

which represents the expected cumulative reward obtained by taking action a in state s, subsequently following policy $\pi$. Under an optimal policy ${\pi }^{\ast }$, the state-value function is maximized and referred to as the optimal state-value function:

$$V^{\ast }\left(s\right)={\max }_{\pi }\mathbb{E}\left[r_t+\gamma G_{t+1}\mid s_t=s\right].$$

(14)

Under this condition, the optimal action-value function can be expressed as follows:

$$Q^{\ast }\left(s,a\right)=\mathbb{E}\left[r_t+\gamma {\max }_{a^{\prime }}{Q}^{\ast }\left(s_{t+1},a^{\prime }\right)\mid s_t=s,a_t=a\right].$$

(15)

3.2. Deep Q-Network

The Nature DQN is an improved version of the original DQN algorithm, featuring enhanced network update methods and the use of experience replay. Compared to the original DQN, Nature DQN is more widely adopted and, for convenience, is simply referred to as DQN in the following discussion.

In order to solve the problem of the Q-table being too large due to Q-learning when the dimensions of the states and actions become larger, DQN employs a deep neural network to approximate the Q-function instead of using a traditional Q-table. Specifically, states serve as inputs to the neural network, which outputs Q-values corresponding to each possible action.

DQN uses an evaluation network, denoted as $Q\left(s,a;\theta \right)$, to compute Q-values for actions taken in the current state, and a target network $Q^{\prime }\left(s,a;{\theta }^{\prime }\right)$ to select the optimal action based on these values. Both of them share the same network architecture. Every T steps, the evaluation network synchronizes its parameters with those of the target network. Additionally, DQN uses experience replay and introduces previous data in order to reduce sample correlation and improve convergence speed.

DQN applies the one-step temporal difference (TD) algorithm to calculate the action-value function. During training, the one-step TD target value $y_t$ is used to estimate the cumulative discounted return at each time step:

$$y_t=r_t+{\gamma \max }_{a\prime }Q\left(s_{t+1},a;{\theta }^{\prime }\right),$$

(16)

and the TD error can be expressed as follows:

$${\delta }_t=y_t-Q\left(s_t,a_t;\theta \right).$$

(17)

The goal of the neural network is to minimize the loss function defined by the mean squared error (MSE) of the TD error:

$$L\left(\theta \right)={\displaystyle \frac{1}{2}}E{\left[{\delta }_t\right]}^2.$$

(18)

Additionally, DQN employs gradient descent to optimize the loss function. It computes the partial derivatives of the loss function with respect to each parameter, determining the gradient direction at the current position. Subsequently, parameters are updated in the opposite direction of this gradient, iteratively achieving local minimization of the loss function.

3.3. Exploration Policy in Deep Q-Network

To balance exploration and exploitation and avoid slow network convergence or premature convergence to local optima, DQN employs the $ϵ$-greedy policy for the action selection. The $ϵ$-greedy policy can be formulated as follows:

$$a=\left\{\begin{array}{cc}{\arg \max }_{a^{\prime }}Q\left(s_t,a^{\prime };\theta \right),\hfill & k>ϵ\hfill \\ randomactioninA,\hfill & k\le ϵ\hfill \end{array}\right.,$$

(19)

which indicates that, at training step t, there is a probability of 1-$ϵ$ to select the action with the highest reward, and a probability of $ϵ$ to randomly select an action from the action set A, where k is a random number uniformly sampled from $\left[0,1\right]$, and $ϵ$ denotes the greedy rate. In most models, $ϵ$ is set to 1 at the beginning to encourage exploration during early training steps. As training progresses, $ϵ$ gradually decreases at a fixed rate, which is usually set to 0.99, progressively emphasizing exploitation.

4. MAI-DQN for Anti-Jamming Decision-Making

The traditional DQN algorithm cannot achieve fast reward convergence and an accurate anti-jamming strategy in complex jamming environments. For this reason, the MAI-DQN is proposed.

In this section, the improvements made to the algorithms of this paper for DQN are detailed in terms of network architecture, exploration strategy, and training methods. Then, the state and action spaces of the phased-array radar anti-jamming decision-making model are introduced in detail, and a detailed solution for the phased-array radar anti-jamming strategy based on MAI-DQN is given. As shown in Figure 4, in terms of network architecture, the current signal state $s_t$ of the phased-array radar is input into the network architecture enhanced with DDQN and a dueling network. In terms of exploration policy, the action is selected through the noisy network, and then the TD error values for training are obtained. In terms of training methods, the network parameters are updated using the TD error generated by the n-step TD method. The variable learning rate, soft update, and gradient clipping are used by the model to improve training results. Finally, the Q-value corresponding to the current anti-jamming action is output. In this process, VDDPER stores the current experience in three separate pools based on the reward magnitude for efficient experience replay and adjusts the sample size according to the value of each experience.

4.1. Network Architecture

4.1.1. Double Deep Q-Network

In the network architecture, the MAI-DQN algorithm incorporates a double deep Q-network (DDQN) to further mitigate the overestimation issue observed in DQN. When calculating the one-step TD target value in DDQN, the action corresponding to the maximum Q-value is initially selected using the evaluation network:

$$a^{\prime }={\arg \max }_{a}Q\left(s_{t+1},a;\theta \right),$$

(20)

then, the improved one-step TD target value method can be expressed as follows:

$$y_t^{DDQN}=r_t+\gamma Q\left(s_{t+1},{\arg \max }_{a}Q\left(s_{t+1},a;\theta \right);{\theta }^{\prime }\right).$$

(21)

4.1.2. Dueling Network

MAI-DQN further improves the learning efficiency and generalization capability of DQN by introducing a dueling network structure. In the DQN neural network architecture, the fully connected layers directly connect to the output layer, producing the Q-values corresponding to each action for the current state after training. In contrast, the dueling network introduces two separate branches following the linear layers, forming two evaluation modules—one evaluates the state value $V\left(s\right)$, referred to as the state-value network, and another predicts the advantage of each action $A\left(s,a\right)$, known as the advantage network. This structure ensures that the assessment of the state does not entirely depend on the value of actions, enabling the model to better converge when actions have a limited impact on state value. In the dueling network, the Q-value output can be represented as follows:

$$Q\left(s,a;\theta ,\alpha ,\beta \right)=V\left(s;\theta ,\beta \right)+A\left(s,a;\theta ,\alpha \right),$$

(22)

where $\alpha$ and $\beta$ are parameters corresponding to the advantage network and the state-value network, respectively, used for function optimization.

However, because the above equation overly emphasizes the summed result of $V\left(s\right)$ and $A\left(s,a\right)$, it cannot yield accurate and fixed values of $V\left(s\right)$ and $A\left(s,a\right)$ when given a certain Q-value, potentially leading to unstable or non-convergent training. Therefore, the following improvement is proposed:

$$Q\left(s,a;\theta ,\alpha ,\beta \right)=V\left(s;\theta ,\beta \right)+\left(A\left(s,a;\theta ,\alpha \right)-A_{av}\right),$$

(23)

where $A_{av}={\displaystyle \frac{1}{\left|A\right|}}{\sum }_{a^{\prime }}A\left(s,a^{\prime };\theta ,\alpha \right)$.

According to the improvement of the above equation, both $V\left(s\right)$ and $A\left(s,a\right)$ yield similar solutions for a given Q-value.

4.2. Exploration Policy

As previously mentioned, the traditional DQN employs an $ϵ$-greedy exploration policy. However, both the parameter $ϵ$ and its decay rate require manual configuration. Optimal exploration strategies may vary across different scenarios or different parameter settings within the same scenario, and manually tuning the parameters of the exploration policy may cause slower convergence or suboptimal performance. Additionally, in certain states, some actions may clearly be suboptimal, yet the $ϵ$-greedy policy still randomly selects these actions with probability $ϵ$ and reduces exploration efficiency.

To address this issue, this paper introduces a noisy network to enhance the exploration policy. The noisy network incorporates learnable stochastic noise into the weights of the neural network, enabling the agent to automatically adjust the degree of exploration according to different states. For a neural network, a linear network layer with input p and output q can be represented by the following equation:

$$y=\omega x+b,$$

(24)

where $x\in {\mathbb{R}}^p$, $y\in {\mathbb{R}}^q$, $\omega \in {\mathbb{R}}^{q\times p}$ represents the weight matrix, and $b\in {\mathbb{R}}^{q\times 1}$ denotes the bias vector. Both $\omega$ and b are learnable parameters in the network. When noise is introduced, the equation is as follows:

$$y=({\mu }^{\omega }+{\sigma }^{\omega }\odot {\epsilon }^{\omega })x+{\mu }^b+{\sigma }^b\odot {\epsilon }^b,$$

(25)

where ${\mu }^{\omega }\in {\mathbb{R}}^{q\times p}$, ${\sigma }^{\omega }\in {\mathbb{R}}^{q\times p}$, ${\mu }^b\in {\mathbb{R}}^q$, and ${\sigma }^b\in {\mathbb{R}}^q$. The symbol ⊙ denotes the Hadamard product, and ${\epsilon }^{\omega }\in {\mathbb{R}}^{q\times p}$, ${\epsilon }^b\in {\mathbb{R}}^q$ are random noise variables.

In the noisy network, parameters ${\mu }^{\omega }$, ${\sigma }^{\omega }$, ${\mu }^b$, and ${\sigma }^b$ are learned and updated through gradient descent, whereas the noise variables ${\epsilon }^{\omega }$ and ${\epsilon }^b$ are generated using factorized Gaussian noise. Specifically, p values are sampled independently from a Gaussian distribution to obtain ${\epsilon }_i\in {\mathbb{R}}^{p\times 1}$ firstly, and subsequently, q values are sampled independently from a Gaussian distribution to obtain ${\epsilon }_j\in {\mathbb{R}}^{q\times 1}$. The noise variable ${\epsilon }^{\omega }$ is then calculated as follows:

$${\epsilon }^{\omega }=f\left({\epsilon }_i\right)·f\left({\epsilon }_j^T\right),$$

(26)

where $f\left(\epsilon \right)=sgn\left(\epsilon \right)\sqrt{\left|\epsilon \right|}$, similarly, ${\epsilon }^b$ can be expressed as follows:

$${\epsilon }^b=f\left({\epsilon }_j\right).$$

(27)

In the above equation, ${\epsilon }_i,{\epsilon }_j\sim \mathcal{N}(0,1)$, while the initial values of ${\mu }^{\omega }$ and ${\mu }^b$ are randomly sampled from independent uniform distributions, ${\mu }^{\omega }$,${\mu }^b\sim U(-1/\sqrt{p},1/\sqrt{p})$. In this paper, the initial values of ${\sigma }^{\omega }$ and ${\sigma }^b$ are set to $2.5/\sqrt{p}$.

4.3. Training Methods

4.3.1. Variable Double-Depth Priority Experience Replay

Traditional DQN and DDQN improve training efficiency through experience replay by sampling experiences uniformly with equal probability. But during training, experiences with larger TD errors typically provide greater learning value, making uniform sampling potentially inefficient and resulting in slower convergence. To address this, the PER method replaces uniform sampling with priority-based sampling, improving the agent’s learning efficiency. Additionally, PER employs a Sum Tree data structure to store and retrieve sampling weights, which helps optimize computational complexity. In PER, the sampling probability for each experience is defined as follows:

$$P_i=p_i/{\sum }_{j=1}^N{p}_j,$$

(28)

where $p_i=\left|{\delta }_i\right|+\eta$ represents the sampling priority of the i-th experience, and $\eta$ is a small positive constant used to prevent division by zero when the TD error is zero. Building on the above method, Ref. [31] proposed the DDPER algorithm to further enhance the algorithm’s efficiency in utilizing scarce and high-value experiences. In short, DDPER compares the reward $r_t$ of the current experience with a reward threshold $\Phi$, which is dynamically adjusted during training. In addition to storing experiences in a shallow experience pool, those with rewards exceeding the threshold are classified as positive experiences and stored in a positive deep experience pool. Additionally, DDPER identifies experiences with negative rewards as negative experiences, storing them in a negative deep experience pool. Thus, high-value experiences are distinguished from ordinary experiences.

During training, experiences from the shallow experience pool are sampled according to priorities $p_i$, whereas those from the deep experience pool are sampled uniformly. The sizes of these pools are defined as follows: the shallow-experience pool size is $B_s=B·\left(1-P_{ds}\right)$, the positive deep experience pool size is $B_{dp}=P_{pn}\left(B-B_s\right)$, and the negative deep experience pool size is $B_{dn}=B-B_s-B_{dp}$, where B denotes the batch size.

In DDPER, both $P_{pn}$ and $P_{ds}$ are fixed values. In contrast, in the proposed VDDPER, the model dynamically adjusts the parameters $P_{pn}$ and $P_{ds}$ based on the training steps. This dynamic adjustment enables the experience replay mechanism to better align with the human memory update process. The adjustment strategy for $P_{pn}$ and $P_{ds}$ is described as follows:

$$P_{ds}=min\left(P_{ps}^{max},P_{ps}^{ini}+{\eta }_1\left(n_s/N^{max}\right)\right),$$

(29)

$$P_{pn}=min\left(P_{pn}^{max},P_{pn}^{ini}+{\eta }_2\left(n_s/N^{max}\right)\right),$$

(30)

where $P_{ps}^{ini}$ and $P_{pn}^{ini}$ represent the preset initial values, $P_{ps}^{max}$ and $P_{pn}^{max}$ denote their respective maximum limits, ${\eta }_1$ and ${\eta }_2$ signify the total increments, and $n_s$ and $N^{max}$ are the current training and the maximum training steps, respectively. Through this update mechanism, both $P_{pn}$ and $P_{ds}$ gradually increase from their initial values, $P_{ps}^{ini}$ and $P_{pn}^{ini}$, during the training process, until they reach their maximum limits, $P_{ps}^{max}$ and $P_{pn}^{max}$. At this point, the model achieves maximum efficiency in utilizing high-value experiences.

This approach aims to direct the model’s attention primarily toward ordinary experiences in the initial training stages, enabling the model to adequately learn from a diverse range of experiences. Conversely, in later stages, the model progressively shifts its focus toward high-value experiences, thus enhancing convergence speed and stability.

4.3.2. N-Step Learning

In the one-step TD method, the TD target value only uses a single time step to estimate the cumulative discounted reward, $G_t$. However, when stochastic elements or noise are present in the environment, the TD target value may become biased. Moreover, in tasks with long sequences, the one-step TD method is less effective at incorporating recent experiences into the TD target value. To address these issues, the n-step TD method is proposed, incorporating cumulative rewards from the subsequent N steps. The n-step TD method enables rewards to propagate more rapidly toward earlier decisions, thereby enhancing learning efficiency. The n-step TD target value update method can be expressed as follows:

$$y_t^{NDDQN}={\sum }_{k=0}^{N-1}{\gamma }^k{r}_{r+k}+{\gamma }^{n}Q\left(s_{t+n},{\arg \max }_{a}Q\left(s_{t+n},a;\theta \right);{\theta }^{\prime }\right).$$

(31)

After applying the noisy network, n-step learning, dueling network, and DDQN, the loss function used for training can be expressed as follows:

$$\begin{array}{cc}\hfill & L{\left(\zeta \right)}_{\mathrm{MAI}}=\mathbb{E}\left[\mathbb{E}{\left({\sum }_{k=0}^{n-1}{\gamma }^k{r}_{t+k}+{\gamma }^{n}Q\left(y,b\left(y\right),{\epsilon }^{\prime };{\zeta }^{\prime }\right)-Q\left(x,a,\epsilon ;\zeta \right)\right)}^2\right]\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\left\{\begin{array}{c}(x,a,r,y)\sim D\hfill \\ b\left(y\right)={\arg \max }_{b\in A}Q\left(y,b,{\epsilon }^{″};\zeta \right),\hfill \end{array}\right.\hfill \end{array}$$

(32)

where D denotes the experience replay buffer, $\epsilon$, ${\epsilon }^{\prime }$, and ${\epsilon }^{″}$ represent noise variable samples at different stages, $\zeta$ represents the parameters of the evaluation network, and ${\zeta }^{\prime }$ denotes the parameters of the target network.

4.3.3. Variable Learning Rate, Soft Update, and Gradient Clipping

In addition to the improvements mentioned above, several additional methods are incorporated in this paper to further improve network training performance. Firstly, given the varying demands for learning rates at different stages of training, this paper utilizes a variable learning rate strategy to enhance convergence speed and stability:

$${\alpha }_{lr}=\max \left({\alpha }_{lr}^{min},{\alpha }_{lr}^{ini}-{\alpha }_{lr}^t\left(n_s/N^{max}\right)\right),$$

(33)

where ${\alpha }_{lr}^{ini}$ denotes the initial learning rate, and ${\alpha }_{lr}^t$ represents the total learning rate reduction. As training progresses, the learning rate decreases to ${\alpha }_{lr}^{min}$ gradually.

Secondly, this paper introduces an improved network-update strategy [36], transitioning from a traditional hard update method, in which the evaluation network’s parameters are directly copied to the target network every K steps, to a soft update method, wherein a part of the evaluation network’s parameters is continuously transferred to the target network during each training step. This modification enhances training stability. The parameter update for the target network in the soft update method is given by the following:

$${\theta }^{\prime }=\left(1-\kappa \right){\theta }^{\prime }+\kappa \theta ,$$

(34)

where $\kappa$ is the update ratio of the network.

Thirdly, to prevent gradient explosion, gradient clipping is employed during the neural network’s loss optimization via gradient descent. Specifically, when the $L_2$-norm of the gradient ${∥{\nabla }_{{\zeta }_t}L\left(\zeta \right)∥}_2$ exceeds a predefined threshold, the gradient clipping method proportionally scales the gradient, constraining its norm to the specified threshold.

4.4. State Space

In RL, a state represents all the environmental information accessible to the agent at a specific moment, while the state space denotes the complete set of all possible states. In complex jamming environments, designing an appropriate state space is crucial. In this paper, a state space based on parameters that are necessary for signal-quality assessment is proposed, capturing signal characteristics from multiple perspectives. The state space defined in this study can be expressed as follows:

$$s_t=\left[M_J,SNR,JNR,ISR,n_j,\psi \right],$$

(35)

where the parameter $M_J$ denotes the type of jamming in the current state, $n_j$ represents the number of jammed frequency points in one coherent processing interval (CPI), and $\psi$ indicates the target identification results obtained after the CA-CFAR processing.

As shown in Figure 5, this study selects ten game iterations as a complete game process. During the game between the phased-array radar and the jammer, the radar first transmits a detection signal into the environment. After receiving the radar signal, the jammer transmits a jamming signal back toward the radar. The radar then senses the received signal to obtain the state $s_t$ and subsequently adopts an anti-jamming action, $a_t$, based on the current policy, $\pi \left(a|s\right)$. This forces the jammer to change its jamming type, resulting in a transition to the next state $s_{t+1}$.

4.5. Action Space

In this paper, the action space is defined as a discrete set. Since we focus on optimizing anti-jamming strategies over a fixed time interval, the anti-jamming actions must be structured in a “packaged” form. Considering diverse jamming types, this study incorporates frequency agility and cover pulses as active anti-jamming methods, along with the SLC and SLB as passive anti-jamming methods. Accordingly, the action space is formulated as follows:

$$a_t=\left[f_{num},c_{SLC},c_{SLB}\right],$$

(36)

where $f_{num}$ represents either the number of frequency points used in cover pulses or the number of frequency-agility points within one CPI, and when $f_{num}$ equals 1, it represents conventional LFM signals. The parameters $c_{SLC}$ and $c_{SLB}$ indicate the activation status of SLC and SLB, respectively.

Specifically, SLB exhibits superior target recovery performance against sidelobe spot jamming, whereas SLC is more effective against coherent slice forwarding jamming. Regarding mainlobe deceptive jamming, only active anti-jamming methods can partially mitigate interference effectively. Additionally, pulse compression is utilized as a fixed action for target detection throughout the anti-jamming decision-making optimization process.

4.6. Reward Function

In RL, the reward function guides the agent in learning the values of different actions when given specific states [37]. In radar anti-jamming strategy optimization, a suitably designed reward function directly influences training efficiency and the effectiveness of the final policy. An optimal reward function should incentivize the radar system to learn the best anti-jamming strategies, ensuring effective target detection even in complex jamming environments, without misleading the learning process. After reviewing reward function designs in relevant papers [38,39,40], a reward function at both the signal level and data level is designed in this paper. At the signal level, the reward is based on target detection results within a CPI, defined as follows:

$$R_s={\sum }_{i=1}^I{R}_{\psi ,n},$$

(37)

where I denotes the number of pulses within one CPI, and $R_{\psi ,i}$ is calculated as follows:

$$R_{\psi ,i}=\left\{\begin{array}{cc}1,\hfill & targetdetected\hfill \\ -1,\hfill & targetnotdetected,\hfill \end{array}\right.$$

(38)

which indicates that within a CPI, the radar receives a reward of +1 when the n-th pulse successfully detects the target via CA-CFAR processing, and $-1$ otherwise. Target detection is determined by the radar’s estimation of the current environmental situation.

At the data level, the reward function is designed based on $SNR$, $JNR$, $ISR$, and the number of jammed frequency points:

$$R_d={\alpha }_1{G}_1+{\alpha }_2{G}_2,$$

(39)

where the gain $G_1=10lg\left({10}^{JNR/10}\left(1-P_f·P_r\right)\right)$ quantifies the effectiveness of active anti-jamming measures. The probability of frequency-domain jamming is $P_f=n_j/f_{num}$. The probability of space-domain jamming is defined as $P_r=\left(R_r-R_j\right)/R_r$, with $R_r$ representing the radar’s maximum detection range and $R_j$ denoting the estimated distance to the jammer. The passive anti-jamming measure gain is defined as $G_2={\beta }_1\left(ISR\right)+{\beta }_2\left(SN{R}_a-SN{R}_b\right)$, where $SN{R}_a-SN{R}_b$ denotes the $SNR$ loss resulting from anti-jamming measures.

Considering these signal-level and data-level rewards, the final reward function can be expressed as follows:

$$R={\omega }_1{R}_s+{\omega }_2{R}_d.$$

(40)

In the above equations, ${\omega }_1$, ${\omega }_2$, ${\alpha }_1$, ${\alpha }_2$, ${\beta }_1$, and ${\beta }_2$ represent the weighting factors used to constrain rewards to similar scales. Through RL, the agent progressively discovers the optimal anti-jamming strategy that maximizes the average reward. For comparative purposes, the same reward function is applied to different RL algorithms in simulation experiments. In summary, the pseudocode for the phased-array radar anti-jamming decision-making algorithm based on MAI-DQN is outlined in Algorithm 1.

Algorithm 1 Training procedure of phased-array radar anti-jamming decision-making
Initialize: the state dimension, the size of the action space, hyperparameters for the MAI-DQN, a dueling noisy double evaluation network, the target network with parameters $\zeta$ and ${\zeta }^{\prime }$, priority in the experience pool, the sizes of $B_s$, $B_{dp}$, $B_{dn}$, other parameters for VDDPER, $SNR$, and $JNR$, the settings for the phased-array radar and the jammer, and $t=0$.
1:	for episode = 1 → M do
2:		if $t=0$ then
3:			Generate target echo, random jamming, and noise in the environment, and initialize them as state $s_1$.
4:		end if
5:		Sample a noisy network $\zeta \sim \epsilon$.
6:		Choose action $a_t={\arg \max }_{a}Q\left(s_t,a,\epsilon ;\zeta \right)$.
7:		Observe the environment again and get $r_t$ as well as $s_{t+1}$.
8:		Store experience $\left(s_t,r_t,a_t,s_{t+1}\right)$ in the shallow experience pool.
9:		Store experience $\left(s_t,r_t,a_t,s_{t+1}\right)$ in the positive deep experience pool or negative experience pool, depending on reward threshold $\Phi$ and update $\Phi$ if $r_t\ge \Phi$.
10:		if episode $\ge B_s$ then
11:			Sample experience as $B_s$ in the shallow experience pool by probability matrix $\mathbf{P}$.
12:			Sample the experience as $B_{dp}$ in the positive deep experience pool uniformly.
13:			Sample the experience as $B_{dn}$ in the negative deep experience pool uniformly.
14:			Integrate the experience in $B_s$, $B_{dp}$, $B_{dn}$ into a mini-batch $D_{mini}$.
15:			Update $\mathbf{P}$, $P_{pn}$ and $P_{ds}$.
16:			Update $y_t$ through the n-step TD method.
17:			Train the evaluation network through backpropagation.
18:			Update the priority of the sampling experience by the TD error.
19:			Update the sizes of $B_s$, $B_{dp}$, and $B_{dn}$ by $P_{pn}$, as well as $P_{ds}$.
20:			Train the evaluation network using $D_{mini}$.
21:			Update the target network using the soft update method.
22:		end if
23:		if $t=N$ then
24:			$t=0$.
25:		else
26:			$t=t+1$.
27:			Update the type of jamming.
28:		end if
29:	end for

5. Experiments

This section presents the simulation results of the phased-array radar anti-jamming decision-making model based on the MAI-DQN. The simulation outcomes are analyzed and discussed to illustrate and validate the effectiveness and performance improvements achieved by the proposed algorithm in the decision-making process. Firstly, the comparison result of the loss function verifies the improvement of the variable learning rate on model training. Secondly, the impact of varying the values of the n-step learning parameter, soft update percentage, discount factor, and noisy network parameter on the average reward is analyzed to validate the chosen hyperparameters. Thirdly, compared with the other algorithms, the performance of MAI-DQN is analyzed from three aspects, that is, reward mean, standard deviation, and average decision accuracy. Fourthly, the pulse compression results using MAI-DQN and DDQN are compared. Lastly, an ablation analysis is conducted to compare the complete MAI-DQN with its variants that exclude improvements. The results demonstrate the effectiveness of the proposed MAI-DQN in phased-array radar anti-jamming decision-making.

5.1. Experimental Settings

In the simulation, the radar is capable of employing the following signal transmission modes: conventional LFM signals, frequency agile signals, and cover pulse signals. The jammer can select from a total of nine jamming types, including mainlobe jamming, sidelobe jamming, and combined mainlobe–sidelobe jamming, such as dense false target jamming, sweep jamming, coherent slice forwarding jamming, and spot jamming. Simulation parameters for radar and jammer settings are presented in

Table 1. Simulation parameters.

Parameter	Value
Carrier frequency	3 GHz
Bandwidth	10 MHz
Frequency hopping interval	10 MHz
Pulse width	10 μs
Pulse repetition period	200 μs
Sampling frequency	30 MHz
Standard deviation of noise	0.2
Initial $SNR$	10 dB
Initial $JNR$	30 dB
Initial distance of target	15 km
Number of array elements	24

.

Regarding the RL network structure, this paper adopts two linear network layers as hidden layers, containing 64 units each. The activation function employed in the output layer is the rectified linear unit (ReLU) function [41], defined as follows:

$$f\left(x\right)=\max \left(0,x\right).$$

(41)

As one of the classical neural network activation functions, ReLU introduces nonlinearity into the network, enabling neural networks to effectively learn and approximate complex nonlinear relationships. Compared with Sigmoid and Tanh functions, ReLU maintains a constant gradient of 1 within the positive range, providing better gradient propagation and effectively mitigating the vanishing gradient problem. Moreover, ReLU induces sparsity through neuron deactivation, enhancing the model’s generalization capability and reducing overfitting.

Regarding network training, as previously mentioned, this study utilizes VDDPER, an adaptive learning rate, and soft updating of the target network. The detailed parameter settings for the model are presented in

Table 2. Model parameters.

Parameter	Value
Maximum training steps	10,000
Maximum shallow memory capacity	1000
Maximum positive deep memory capacity	500
Maximum negative deep memory capacity	500
Initial proportion of deep and shallow experiences	0.2
Initial proportion of positive and negative experiences	0.6
Length of observation space	6
Length of action space	14
Initial learning rate	0.01
Minimum learning rate	0.001
Discount factor	0.9
Percentage of soft update	0.005
N-step learning parameter	5
Noisy network parameter	2.5
Mini-batch size	64

.

5.2. Experimental Results and Discussion

To verify the advantages of using an adaptive learning rate compared to a fixed learning rate, this paper compares their respective loss functions while keeping other parameters of the MAI-DQN model unchanged. As illustrated in Figure 6, the training convergence achieved by gradually decaying the learning rate ${\alpha }_{lr}$ from 0.01 to 0.001 shows significant improvements over fixed learning rates of ${\alpha }_{lr}$ = 0.01 and ${\alpha }_{lr}$ = 0.001. Specifically, when a fixed learning rate of ${\alpha }_{lr}$ = 0.001 is employed, the model’s convergence speed noticeably slows down, requiring approximately 6000 steps to achieve convergence. Conversely, with a fixed learning rate of ${\alpha }_{lr}$ = 0.01, although the model initially converges more rapidly, it experiences noticeable oscillations starting at around 1000 training steps.

To verify the appropriateness of the hyperparameter settings in this paper, as shown in Figure 7, we compare the impact of varying the values of the n-step learning parameter, the soft update percentage, the discount factor, and the noisy network parameter on the average reward. The results indicate that when the values listed in Table 2 are used, the model achieves an optimal average reward.

In terms of the algorithm comparison, the proposed MAI-DQN is compared with four other algorithms, which are Nature DQN, DDQN, the DDPER-VGA-DDQN algorithm proposed by Professor Xiao in [31], and MAI-DQN using the DDPER instead of VDDPER(MAI-DQN-DDPER). To ensure a fair comparison, all five algorithms share the same neural network parameters, except for the improvements introduced in this paper. Figure 8a shows the comparison of average rewards during the training process for Nature DQN, DDQN, DDPER-VGA-DDQN, MAI-DQN-DDPER, and the proposed MAI-DQN algorithm.

Firstly, regarding convergence speed, MAI-DQN achieves substantial improvements, reaching convergence at around step 600. In contrast, DDQN and Nature DQN exhibit significantly slower convergence. Meanwhile, MAI-DQN-DDPER and DDPER-VGA-DDQN both converge around step 2200, this notable difference arises because the VDDPER method dynamically adjusts the sampling ratio of experiences with different values during training, thereby enhancing exploratory depth at the initial training stage and significantly accelerating the learning speed.

Secondly, in terms of reward quality and stability, this paper utilizes the mean and standard deviation of rewards to quantify performance, as summarized in

Table 3. Comparison of the mean value and standard deviation of the reward.

Algorithm	Mean Value	Standard Deviation
MAI-DQN	42.79	3.65
MAI-DQN-DDPER	41.91	3.65
DDPER-VGA-DDQN	41.13	4.15
DDQN	39.91	5.35
Nature DQN	39.79	5.61

. Compared with the other four algorithms, MAI-DQN achieves not only rapid convergence but also a higher average reward due to its increased focus on high-value experience in later training stages. Additionally, the multifaceted improvements to the network architecture, particularly the integration of the noisy network structure, result in significantly higher reward stability for MAI-DQN compared to DDQN and DQN.

As shown in Figure 8b, this paper computes the average decision-making accuracy from the beginning to the end of training to evaluate the decision accuracy of the five algorithms. The results indicate that the MAI-DQN algorithm significantly outperforms MAI-DQN-DDPER, DDPER-VGA-DDQN, DDQN, and Nature DQN in terms of average decision accuracy. Furthermore, a notable performance gap appears between Nature DQN and the other four algorithms, attributable to Nature DQN’s tendency to overestimate Q-values in complex jamming environments, causing the model to converge to local optima. It should be noted that none of the algorithms reach 100% accuracy post-convergence, as the results include data collected during the initial exploration stage.

After completing anti-jamming decision-making optimization, this paper compares the pulse compression results using the anti-jamming strategy generated by MAI-DQN with the pulse compression results using the anti-jamming strategy generated by DDQN, thereby validating the effectiveness of the proposed algorithm. To enhance the $SNR$, the following results are obtained by performing incoherent accumulation of pulse compression within one CPI.

As shown in Figure 9, the target signal is masked by the jamming signal in the pulse compression output of the jammed phased-array radar system, resulting in a failure to detect the target.

As shown in Figure 10, in the pulse compression result processed using the anti-jamming strategy generated by MAI-DQN, the target signal is successfully recovered. It is worth noting that although some jamming residue remains in the result, the target can still be detected by CFAR in most cases. However, as shown in Figure 11, in the pulse compression results processed by the anti-jamming strategy generated by DDQN, the results are affected to varying degrees. Specifically, in Figure 11a,c–f, the target signals fail to be recovered to a level detectable by CFAR.

As shown in Figure 12, under 100 Monte Carlo simulations, this paper compares the performance differences among various jamming countermeasures. In the presence of sidelobe jamming or when sidelobe suppression jamming coexists with mainlobe dense false target jamming, the target detection rate achieved after applying the anti-jamming strategies generated by MAI-DQN shows good performance at $JNR$ levels of 30 dB and below. However, when the $JNR$ exceeds 30 dB, the detection rates experience a significant decline under all countermeasure settings.

Lastly, as shown in

Table 4. Ablation analysis on MAI-DQN.

Algorithm	Mean Value	Standard Deviation	Average Target Detection Rate
MAI-DQN	42.79	3.65	86.44%
(w/o) Dueling Network & DDQN	40.85	3.64	80.89%
(w/o) Noisy Network	41.33	4.90	82.67%
(w/o) Training Methods	38.75	4.12	81.00%

, an ablation study was conducted to compare the complete MAI-DQN with its variants that exclude improvements in network architecture, exploration strategy, and training methods, respectively. The comparison focuses on the mean value, standard deviation of the reward, and the average target detection rate after convergence, in order to validate the contribution of each component to the overall model performance. The average target detection rate is obtained from the results of 100 Monte Carlo simulations under all jamming scenarios.

In this paper, the computational hardware includes an Intel(R) Core(TM) i9-12900H CPU @ 2.50 GHz, 32 GB of RAM (Intel, Santa Clara, CA, USA), and an NVIDIA GeForce RTX 3070 Ti Laptop GPU (Nvidia, Santa Clara, CA, USA). The programming environment is Python 3.7, and the framework used is Torch 1.13.1.

Table 5. Comparison of the average time required for each decision-making process.

Algorithm	Value
MAI-DQN	47.8 × 10−2 s
MAI-DQN-DDPER	47.1 × 10−2 s
DDPER-VGA-DDQN	45.9 × 10−2 s
DDQN	42.4 × 10−2 s
Nature DQN	41.9 × 10−2 s

summarizes the average computational time per decision for the MAI-DQN, DDPER-VGA-DDQN, DDQN, and Nature DQN algorithms during the decision-making process.

Compared to the DDPER-VGA-DDQN algorithm, the MAI-DQN employs a more sophisticated network architecture, resulting in a slightly longer average decision-making time. In contrast, Nature DQN, due to its simpler network structure, achieves the shortest decision-making time among the algorithms. Additionally, because phased-array radar systems typically process larger datasets compared to traditional radars, and given that each decision-making instance in this paper spans one CPI, the single decision time observed here is relatively longer. However, in realistic combat scenarios, anti-jamming strategies are usually generated in advance, thus, the increased decision-making time resulting from the adoption of MAI-DQN is generally acceptable.

6. Conclusions

Traditional phased-array radar anti-jamming decision-making methods encounter significant challenges in terms of decision complexity and processing speeds within complex jamming environments. To address this, this paper establishes an RL-based phased-array radar anti-jamming decision-making model, utilizing the proposed MAI-DQN algorithm to identify optimal anti-jamming strategies. This approach effectively enhances both decision-making efficiency and accuracy. Simulation results demonstrate that, compared to the DDPER-VGA-DDQN algorithm, the MAI-DQN in phased-array radar anti-jamming decision-making achieves faster model convergence, improved convergence quality and stability, and higher decision accuracy.

However, since mainlobe suppression jamming typically poses a critical threat to single-station radar systems, future works can focus on extending the anti-jamming strategy optimization from single-station radar systems to multi-station cooperative radar systems, thereby further enhancing anti-mainlobe jamming capabilities. In addition, all experimental results in this paper are derived from simulations, which impose certain limitations; therefore, in future work, real-world measured data can be used to validate the performance of the proposed model.