Intelligent Anti-Jamming Decision Algorithm for Wireless Communication Based on MAPPO

Abstract

A wireless communication intelligent anti-jamming decision algorithm based on Deep Reinforcement Learning (DRL) can gradually optimize communication anti-jamming strategies without prior knowledge by continuously interacting with the jamming environment. This has become one of the hottest research directions in the field of communication anti-jamming. In order to address the joint anti-jamming problem in scenarios with multiple users and without prior knowledge of jamming power, this paper proposes an intelligent anti-jamming decision algorithm for wireless communication based on Multi-Agent Proximal Policy Optimization (MAPPO). This algorithm combines centralized training and decentralized execution (CTDE), allowing each user to make independent decisions while fully leveraging the local information of all users during training. Specifically, the proposed algorithm shares all users’ perceptions, actions, and reward information during the learning phase to obtain a global state. Then, it calculates the value function and advantage function for each user based on this global state and optimizes each user’s independent policy. Each user can complete the anti-jamming decision based solely on local perception results and their independent policy. Meanwhile, MAPPO can handle continuous action spaces, allowing it to gradually approach the optimal value within the communication power range even without prior knowledge of jamming power. Simulation results show that the proposed algorithm exhibits significantly faster convergence speed and higher convergence values compared to Deep Q-Network (DQN), Q-Learning (QL), and random frequency hopping algorithms under frequency sweeping jamming and dynamic probabilistic jamming.

1. Introduction

With the rapid development of communication technology, wireless communication is widely used in various fields, including in society, the economy, and military [1]. However, due to the open nature of wireless communication transmission channels, it is susceptible to various forms of malicious jamming. Therefore, it is essential to design effective anti-jamming methods under jamming threats to ensure the reliability and effectiveness of wireless communication [2].

To ensure that wireless communication networks are protected from jamming threats, mainstream anti-jamming technologies primarily consist of spread spectrum communication techniques such as frequency hopping and direct sequence spread spectrum, often combined with frequency/power/modulation coding adaptation technologies [3,4]. These methods significantly reduce the impact of conventional jamming and improve the reliability of wireless communication in electromagnetic jamming environments to some extent. However, the aforementioned communication anti-jamming techniques have limited adaptability to time-varying jamming environments and struggle to effectively cope with various complex jamming, such as dynamic, cognitive, and intelligent jamming. Artificial intelligence technologies provide new ideas for communication anti-jamming. Currently, many researchers are exploring the integration of conventional communication anti-jamming techniques with artificial intelligence technologies [5,6]. Intelligent agents interact continuously with the spectrum environment to acquire characteristics of jamming signals, such as frequency and power, and autonomously adjust their communication strategies based on reward values to ensure good communication quality, thereby achieving intelligent anti-jamming [7]. In particular, communication anti-jamming technologies based on DRL have gained attention due to their ability to generate anti-jamming strategies without requiring prior knowledge of the jamming environment. This has become an important research direction in intelligent communication anti-jamming. For instance, reference [8] proposed a DRL-based algorithm for communication frequency and power selection, which improved the throughput of communication users while maintaining low power consumption. Reference [9] introduced a DRL-based multi-domain anti-jamming algorithm that learns the patterns of multi-domain dynamic jamming transformations in time, frequency, and spatial domains from spectrum waterfalls, allowing it to select appropriate anti-jamming strategies to evade jamming. These studies demonstrate the effectiveness of applying DRL in the field of communication anti-jamming. However, most current research focuses on single-agent anti-jamming scenarios.

In multi-agent scenarios, each user using a single-agent learning algorithm alone will inevitably increase the difficulty of anti-jamming. For example, in a joint anti-jamming problem in a multi-user scenario, if each user independently employs a DQN algorithm for communication anti-jamming decision-making, the presence of mutual interference among users means that each user can only treat the other users as jamming. This results in a more complex jamming scenario than the actual conditions. If we assume the existence of a centralized agent that can gather information from all users to facilitate learning, decision-making, and information distribution, this requires significant computational power and signaling overhead. Otherwise, these processes may consume excessive time, reducing the actual communication time proportion. However, multi-agent learning algorithms have shown tremendous potential in cooperation and competition, information sharing, and learning efficiency. Their objective is to dynamically learn the optimal strategy for each agent to maximize long-term cumulative rewards [10,11]. MAPPO is a multi-agent algorithm that combines the characteristics of centralized training and decentralized execution [12]. During the learning phase, MAPPO shares the perception, action, and reward information of all agents to obtain a global state. It then uses this global state to compute the value function and advantage function for each agent and optimizes each agent’s independent policy. In the decision-making phase, each agent makes personalized decisions based on local observations and independent policies [13,14].

To address the challenge that conventional wireless communication anti-jamming methods face in effectively resisting various rapidly changing jamming threats in complex electromagnetic environments, this paper proposes a wireless communication intelligent anti-jamming decision algorithm based on MAPPO. The proposed algorithm models the communication anti-jamming problem under malicious jamming threats as a Markov Decision Process (MDP) and uses MAPPO to solve for the optimal joint anti-jamming strategy for the communication system. Historically, research on wireless communication anti-jamming has often been conducted in discrete action spaces, where the simulation parameters set the communication power as discrete values [15,16]. In such models, low communication power can only be used in the absence of jamming, while high communication power is only sufficient to counter low jamming power but fails against high jamming power. These settings are somewhat idealized, requiring prior knowledge of jamming power for the communication parties to set appropriate values in advance, which is often difficult in adversarial communication scenarios. In contrast, the algorithm proposed in this paper can handle continuous action spaces, allowing it to gradually converge to optimal communication power values even in the absence of prior knowledge about jamming power through the training process.

2. System Model and Problem Formulation

2.1. System Model

The article considers a wireless communication environment as shown in Figure 1. This environment includes a jammer, a central base station, $N$ communication users, and $M$ accessible channels. It is assumed that the wireless communication network can provide sufficient channels, hence the condition $N<M$ is established. Due to users operating in the same spectrum range, signals from different users can interfere with each other. This paper mainly considers co-frequency interference, which occurs when multiple users simultaneously occupy the same channel, leading to the phenomenon of mutual interference due to signal overlap.

Assuming that each user has the capability of broadband spectrum sensing, they can perceive the status of the target frequency band [17]. The central base station, as the core of the communication system, possesses learning capabilities and can interact with each user through a protocol-enhanced low-capacity control channel to exchange a small amount of information [18]. During the learning phase, each user transmits information to the central base station, including the status of the target frequency band obtained through interaction with the environment, the actions taken by the transmitter, and the rewards received from communication. The central base station uses the global state to calculate each agent’s value function and advantage function, optimizing each agent’s independent strategy. Users can then make decisions based on local sensing information and their independent strategies, including channel and power selection.

The transmission of communication data is organized into time slots of duration $T_s$, which serve as the minimum time unit for continuous transmission. It is assumed that both jamming and communication change at the end of a time slot, with no changes occurring during the slot itself. The use of a parallel time slot structure as shown in Figure 2 can prevent sensing, learning, and decision-making from occupying the time for data transmission [19]. Specifically, by utilizing modern signal processing technologies, communication devices can perform spectrum sensing and data transmission simultaneously within the same time frame. This means that spectrum sensing and data transmission can be conducted in parallel by the communication devices, with learning carried out by a central base station and decision-making performed by individual intelligent agents. Therefore, the parallel time slot structure illustrated in Figure 2 is theoretically feasible.

2.2. Problem Formulation

This article models the anti-jamming problem of communication systems as a Markov decision process, which can be represented as $〈S, A,P,R,\gamma 〉$ [20,21]. Here, $S$ represents the state space; $A$ represents the action space; and $P$ represents the state transition probability, which is the probability of the external environment transitioning from the current state to a certain state after the transmitter takes an action. In this problem, the jamming is unknown, and the strategies of other users are also unknown, hence the state transition probabilities are also unknown; and $R$ represents the reward function; $\gamma \in \left(\mathrm{0,1}\right)$ represents the discount factor, which reflects the agent’s emphasis on future rewards. The closer the discount factor is to one, the more emphasis is placed on long-term rewards. The specific definitions of each element are as follows:

The state space is the collection of all possible states of the target frequency band. The state of the target frequency band in the time slot is represented by the combination of the average received power of the user on each channel:

$${\mathit{s}}_t=\left\{{\overline{p}}_{1,t},{\overline{p}}_{2,t},\cdots ,{\overline{p}}_{M,t}\right\}\in S$$

(1)

where, ${\overline{P}}_{m,t}$ represents the average power received by the user on the channel $m$.

The action space is the set of all possible independent actions that each user can choose. In this article, each user’s actions include channel selection and power selection. Channel selection involves choosing a single channel for data transmission from a total of $M$ available channels, while power selection involves choosing a value from a continuous interval $\left[p_{min},p_{max}\right]$ to be used as the transmission power. In time slot $t$, the independent actions of user $n$ are defined as follows:

$$a_{n,t}=\left\{f_{n,t},p_{n,t}\right\}\in A$$

(2)

where, $f_{n,t}$ represents the transmission channel and $p_{n,t}$ represents the transmission power.

A joint action is a combination of independent actions taken by all users, which can be expressed as follows:

$${\mathit{a}}_t=\left\{a_{1,t},a_{2,t},\cdots ,a_{N,t}\right\}$$

(3)

The reward function is used to calculate the reward value obtained by taking transmission actions in a specific environmental state. The rewards are influenced by jamming, mutual interference, channel switching, and power loss. Regarding jamming and mutual interference, successful communication must be determined based on the average Signal to Jamming plus Noise Ratio (SJNR) at the receiver and the demodulation threshold. Since users need to interrupt communication and re-establish connections during channel switching, no information can be transmitted during this time. Thus, this article considers channel switching loss and sets the channel switching loss to ${\zeta }_f$. When the selected transmission power exceeds the required minimum transmission power, it is judged that power loss exists, and the power loss factor is set to ${\zeta }_p$.

In time slot $t$, the user’s immediate reward can be expressed as follows:

$$r_{n,t}=\delta ({\theta }_{n,t}\ge {\theta }_{th})-{\zeta }_f\delta (f_{n,t}\ne f_{n,t-1})-{\zeta }_p(p_{n,t}-p_{n,t}^{\prime })\delta (p_{n,t}>p_{n,t}^{\prime })$$

(4)

where $\delta \left(x\right)$ is the indicator function, which is equal to one if the condition $x$ is true, and zero otherwise; ${\theta }_{n,t}$ is the average SJNR received by the receiver on the channel selected by user $n$ in time slot $t$, which includes the power effects of jamming and mutual interference; ${\theta }_{th}$ is the demodulation threshold for the receiver’s SJNR; and $p_{n.t}^{\prime }$ is the minimum transmission power required for the receiver to successfully demodulate.

Joint rewards are a combination of instant rewards for all users, which can be expressed as follows:

$${\mathit{r}}_t=\left\{r_{1,t},r_{2,t},\cdots ,r_{N,t}\right\}$$

(5)

In the MAPPO algorithm, each user has an independent policy, which is represented as ${\pi }_n\left(a_{n,t}{|\mathit{s}}_t\right)$. The joint strategy can be expressed as a combination of independent strategies for all users:

$$\pi \left({\mathit{a}}_t\left|{\mathit{s}}_t\right.\right)=\left\{{\pi }_1\left(a_{1,t}\left|{\mathit{s}}_t\right.\right),{\pi }_2\left(a_{2,t}\left|{\mathit{s}}_t\right.\right),\cdots ,{\pi }_N\left(a_{N,t}\left|{\mathit{s}}_t\right.\right)\right\}$$

(6)

The maximum expected discount reward based on joint strategy can be expressed using a value function:

$$V^{\pi }\left(s\right)=E_{\tau \sim \pi }\left[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^t{\displaystyle \sum _{n=1}^N{r}_{n,t}}}\left|s_0=s\right.\right]$$

(7)

where $\tau$ is a trajectory of the system under the joint strategy.

Assuming that the central base station can obtain global information through interaction with various users, and then learn based on sampling trajectories. Solving the optimal joint strategy is to find a joint strategy that maximizes the value function:

$${\pi }^{\ast }=\arg \underset{\pi }{\max }{V}^{\pi }\left(s\right)$$

(8)

This optimal joint strategy can be approximated by deep reinforcement learning algorithms, such as the MAPPO algorithm used in this paper.

3. Intelligent Anti-Jamming Decision Algorithm for Wireless Communication Based on MAPPO

To obtain the optimal joint strategy, this article proposes a wireless communication intelligent anti-jamming algorithm based on the MAPPO algorithm. The MAPPO algorithm employs a critic-actor architecture, where each agent has a critic network to evaluate the performance of its corresponding actor network during the training process [22]. During execution, each agent independently makes action decisions based on its perceived local information.

Due to the ease of causing significant policy changes when directly using policy gradient updates, a clipping function is employed to limit the magnitude of policy updates, preventing drastic changes in the policy, which helps maintain the stability of learning. The clipping function is expressed as follows:

$$L^{CLIP}\left(\theta \right)=E_t\left[\min \left(r_t\left(\theta \right){\widehat{A}}_t,clip\left(r_t\left(\theta \right),1-\epsilon ,1+\epsilon \right){\widehat{A}}_t\right)\right]$$

(9)

where $\theta$ represents the parameters of the actor network; $r_t\left(\theta \right)$ is the ratio of the new policy to the old policy; ${\widehat{A}}_t$ is the advantage estimate; and $clip\left(·\right)$ is the clipping function that restricts $r_t\left(\theta \right)$ to the range $\left[1-\epsilon ,1+\epsilon \right]$. The ratio of the new to the old policy is expressed as follows:

$$r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left({\mathit{a}}_t|{\mathit{s}}_t\right)}{{\pi }_{{\theta }_{old}}\left({\mathit{a}}_t|{\mathit{s}}_t\right)}}$$

(10)

The advantage function is expressed as follows:

$${\widehat{A}}_t={\delta }_t+\cdots +{(\gamma \lambda )}^{T-t+1}{\delta }_{T-1}$$

(11)

where ${\delta }_t$ is the temporal difference error of the state value function:

$${\delta }_t={\mathit{r}}_t+\gamma V\left({\mathit{s}}_{t+1}\right)-V\left({\mathit{s}}_t\right)$$

(12)

The loss function of the state value function of the critic network is expressed as follows:

$$L^{VF}(\varphi )=E\left[{\left(V\left({\mathit{s}}_t\right)-\left({\mathit{r}}_t+\gamma V\left({\mathit{s}}_{t+1}\right)\right)\right)}^2\right]$$

(13)

where $\varphi$ is the parameters of the critic network.

To encourage exploration of the policy, an entropy term is introduced. The overall loss function is expressed as follows:

$$L^{CLIP+VF+H}=E\left[L^{CLIP}\left(\theta \right)+c_1{L}^{VF}(\varphi )-c_{2}H\left(\pi \left(\mathit{s}\right)\right)\right]$$

(14)

where $H\left(\pi \left(\mathit{s}\right)\right)$ represents the entropy of the current policy, reflecting the uncertainty of the policy distribution. Both $c_1$ and $c_2$ are coefficients, with values set to 0.5.

The wireless communication intelligent anti-jamming algorithm based on MAPPO is shown in Algorithm 1.

Algorithm 1: The wireless communication intelligent anti-jamming algorithm based on MAPPO

initialize: actor network parameters $\theta$, critic network parameters $\varphi$, policy $\pi$, experience replay buffer $D$ for $episode=\mathrm{1,2},\cdots ,L$   for $t=\mathrm{1,2},\cdots ,T$      Each user selects communication channels and transmission power based on independent strategies;      Each user receives rewards $r_{n,t}$ and perception results ${\mathit{s}}_t$;      Environment transitions to the next state ${\mathit{s}}_{t+1}$;   end for   Calculate value function and advantage function;   Store ${\left\{\left[{\mathit{s}}_t,{\mathit{a}}_t,{\mathit{r}}_t,{\widehat{A}}_t\right]\right\}}_{t=1}^T$ to $D$;   for $k=\mathrm{1,2},\cdots K$    Shuffle the data order and renumber it;    for $j=\mathrm{1,2},\cdots ,J$      Randomly sampling the $B$ group data from $D$: ${D_j=\left\{\left[{\mathit{s}}_t,{\mathit{a}}_t,{\mathit{r}}_t,{\widehat{A}}_t\right]\right\}}_{t=1}^B$;      for $b=\mathrm{1,2},\cdots ,B$        Perform gradient updates and update network parameters;      end for    end for   end for   Update independent policies for each user; end for

4. Simulations and Analyses

4.1. Parameter Settings

This algorithm uses PyTorch for simulation experiments to verify the performance of the proposed algorithm. In the simulation, the proposed algorithm is compared with the QL algorithm [23], DQN algorithm [24], and the random frequency hopping algorithm.

QL Algorithm: This algorithm records and updates the Q-values for each state-action pair in a Q-table. By updating the Q-table, the algorithm gradually learns the optimal policy, aiming to maximize the cumulative reward.

DQN Algorithm: This algorithm stores the agent’s interactions with the environment in an experience replay buffer. It uses a deep neural network to replace the Q-table in traditional Q-learning, generating Q-values. The network parameters are adjusted during training by sampling from the experience replay buffer, gradually acquiring the optimal policy.

Random Frequency Hopping Algorithm: this algorithm randomly selects the communication channel and transmission power in each time slot, without considering the behavior of other users.

The comparison algorithm uses a centralized learning approach similar to MAPPO, but all decisions are made by a single agent and then distributed to each user for execution through a low-capacity control channel reinforced by a protocol. The DQN and QL algorithms lack the ability to handle continuous action spaces, so the power selection for comparison algorithms is limited to the upper and lower bounds of the transmission power range. To objectively analyze the performance of the proposed algorithm, the comparative algorithms are set to have the same parameter values as the proposed algorithm. The simulation parameters are shown in

Table 1. Simulation parameters.

Parameters	Numerical Values	Parameters	Numerical Values
$\mathrm{Number} \mathrm{of} \mathrm{users} N$	2	$\mathrm{Number} \mathrm{of} \mathrm{available} \mathrm{channels} M$	5
$\mathrm{Bandwidth} B$	2 MHz	$\mathrm{Communication} \mathrm{time} \mathrm{slot} \mathrm{length} T_s$	20 ms
$\mathrm{Transmission} \mathrm{power} p_C$	8~16 dBm	$\mathrm{Jamming} \mathrm{power} p_L,p_H$	12 dBm, 20 dBm
$\mathrm{Demodulation} \mathrm{threshold} {\theta }_{th}$	4 dB	$\mathrm{Channel} \mathrm{switching} \mathrm{loss} {\zeta }_f$	0.02
$\mathrm{Power} \mathrm{loss} \mathrm{factor} {\zeta }_p$	0.025	Experience pool size	10,000
$\mathrm{Discount} \mathrm{factor} \gamma$	0.3	$\mathrm{Clipping} \mathrm{factor} \epsilon$	0.2
$\mathrm{Learning} \mathrm{rate} \alpha$	0.0005	Optimizer	RMSprop

.

To effectively verify the algorithm proposed in this paper, we consider multiple frequency-sweeping jamming styles: the jamming power of the jamming signals is $p_L,p_H$. Each jamming signal periodically sweeps across all channels in the target frequency band to implement the jamming, as shown in Figure 3, where the two colored blocks represent low-power jamming and high-power jamming, respectively.

At the same time, this paper also considers dynamic probability jamming patterns [25]: the jammer randomly determines the jamming pattern for the current period according to the probability matrix shown in Figure 4. There are three possibilities for each time slot: no jamming, one high-power jamming, or one high-power jamming along with one low-power jamming. The dynamic probability jamming generated based on the probability matrix in Figure 4 is shown in Figure 5.

4.2. Simulation Results and Analysis

Figure 6 shows the time–frequency diagram of the target frequency band under frequency-sweeping jamming, where the blue blocks represent normal communication for users, and the gray blocks indicate jamming affecting user communication. The numbers in the blue blocks represent the communication power, rounded to two decimal places. Figure 6a displays the time–frequency diagram when the algorithm is untrained, showing that users are easily affected by malicious jamming and interference between users, rendering normal communication impossible. In contrast, Figure 6b displays the time–frequency diagram after the algorithm has converged, indicating that users can completely avoid both jamming and mutual interference, and communicate with the lowest communication power.

Figure 7 shows the time–frequency diagram of the target frequency band under dynamic probability jamming. It is evident that after the proposed algorithm converges, it can effectively avoid both jamming and user-to-user mutual interference, and communicate with the lowest communication power.

Figure 8 presents a comparison of the normalized throughput performance of DQN, QL, random hopping frequency, and the proposed algorithm based on MAPPO under frequency-sweeping jamming. The figure assumes that the upper limit of the transmission power range can effectively counter low-power jamming without causing power loss. The performance of random frequency hopping represents the performance achievable by randomly selecting communication channels and power levels in this jamming environment, with an average close to the initial values of the other algorithms, specifically an average of 0.432. The proposed algorithm begins to converge from the 9th episode, achieving a convergence value of 0.983; the DQN algorithm starts to converge at the 39th episode with a convergence value of 0.977; the QL algorithm shows convergence starting from the 37th episode with a convergence value of 0.917. The proposed algorithm converges significantly faster than the compared algorithms and achieves the highest convergence value, which is 0.61% and 7.20% higher than that of the DQN and QL algorithms, respectively.

Figure 9 presents a performance comparison of the algorithms under frequency-sweeping jamming with the preset communication power set relatively high. Compared to the scenario in Figure 8, the only difference is the setting of low-power jamming at 8 dBm. This lower value results in the preset high communication power values of the comparison algorithms being higher than the corresponding optimal communication power, leading to power loss and a decrease in the resulting reward values. The proposed MAPPO algorithm still converges to 0.983, while the DQN and QL algorithms converge to 0.964 and 0.871, respectively. Compared to Figure 8, the convergence value of the proposed MAPPO algorithm remains unchanged, while the convergence values of the DQN and QL algorithms decrease by 1.33% and 5.02%, respectively, thus expanding the advantage of the proposed algorithm. Additionally, the MAPPO algorithm starts to converge as early as the seventh episode, slightly earlier than in Figure 8. This faster convergence is due to the optimal communication power value for low-power jamming being located within the range and users still being able to achieve a high reward even when slightly above the optimal power. Therefore, it can be seen that the proposed MAPPO algorithm is better equipped to handle situations where the preset communication power is incorrectly set due to a lack of prior knowledge about jamming power.

Figure 10 presents a comparison of the normalized throughput performance of the DQN, QL, random frequency hopping, and the proposed MAPPO algorithm under dynamic probability jamming with the preset communication power set relatively high. The proposed MAPPO algorithm converges to 0.984, while the DQN and QL algorithms converge to 0.959 and 0.704, respectively. The proposed algorithm shows a significantly faster convergence rate, achieving convergence values that are 2.61% and 39.77% higher than those of the DQN and QL algorithms, respectively. This demonstrates that the proposed MAPPO algorithm continues to perform excellently under dynamic probability jamming, exhibiting faster convergence, higher convergence values, and greater stability.

5. Conclusions

This paper proposes a wireless communication intelligent anti-jamming algorithm based on MAPPO to address the issues of multi-user scenarios and a lack of prior knowledge about jamming power. Simulation results demonstrate that, compared to the DQN, QL, and random frequency hopping algorithms, the proposed algorithm converges significantly faster and achieves higher convergence values. Furthermore, in situations where the preset communication power is set too high due to the absence of jamming power prior knowledge, the convergence advantage of the proposed algorithm is even more pronounced. The algorithm proposed in this paper performs well in a scenario with two users and also shows potential for application in scenarios with more users and more complex jamming. In future work, we will explore the performance of the proposed algorithm in more complex scenarios, particularly in situations where users cannot completely avoid jamming in the frequency domain.