Multi-AUV Hunting Strategy Based on Regularized Competitor Model in Deep Reinforcement Learning

Abstract

Reinforcement learning has made significant progress in single-agent applications, but it still faces various challenges in multi-agent scenarios. This study investigates the application of reinforcement learning algorithms in a competitive game scenario of multi-autonomous underwater vehicle (multi-AUV) hunting the evaders. We introduce an optimality operator and redefine the objective function of multi-agent reinforcement learning (MARL), transforming the uncertain states of other agents into solvable inference problems, namely the Regularized Competitor Model (RCM). Leveraging RCM, multi-agent systems can optimize strategies in competitive game training more efficiently. We verify and analyze the performance of the proposed algorithm in a multi-AUV hunting scenario. Simulation results demonstrate that the proposed algorithm exhibits strong adaptability and a higher success rate than the baseline in hunting the evaders.

1. Introduction

Deep Reinforcement Learning (DRL) allows agents to have independent exploration and learning capabilities, which enable them to interact continuously with the environment and enhance their behavior through trial and error [1]. Leveraging this, DRL algorithms can assist multiple Autonomous Underwater Vehicles (AUVs) in making independent decisions in complex underwater environments. AUVs are capable of learning optimal behavioral strategies to accomplish specific tasks, such as searching [2,3], tracking [4,5,6], detecting [7], or hunting [8,9]. This ability allows AUVs to make intelligent decisions based on the current context without the need for strict manual control [10]. Given the dynamic nature of underwater environments, DRL algorithms contribute to improving the efficiency of multi-AUV systems by optimizing resource utilization, reducing energy consumption, and shortening task completion time, thereby enhancing overall performance [11].

Unfortunately, traditional reinforcement learning approaches such as Q-Learning or policy gradient are poorly suited to multi-agent environments [12]. One major challenge is that the strategy of each agent evolves during training, making the environment non-stationary from the perspective of any individual agent [13]. This poses learning stability challenges and prevents the direct use of past experience replay [14]. The presence of multiple agents introduces additional uncertainty to the environment. Therefore, the capability to deduce the characteristics of other agents, such as beliefs, private information, behaviors, and strategies, is crucial [15]. Moreover, the actual operational environment of AUVs is inherently complex and poses higher risks. Consequently, it is necessary to perform pre-training to ensure the adaptability and effectiveness of AUVs in challenging conditions. The diversity of the pre-training environment significantly impacts the overall performance of the trained policies [16].

To address these challenges and difficulties, researchers have proposed numerous Multi-Agent Reinforcement Learning (MARL) algorithms. Rashid et al. [17] proposed the QMIX method, which employs a hybrid network to estimate joint action-values as a monotonic combination of per-agent values. By leveraging a hybrid network to combine single-agent local value functions and incorporating global state information during training, the performance of the algorithm is improved. Additionally, this ensures consistency between centralized and decentralized policies. MADDPG [14], an extension of the DDPG algorithm proposed by OpenAI, adopts a centralized training and decentralized execution approach. Its structure mirrors that of DDPG, with each agent utilizing an independent actor network to observe state outputs and determine actions. However, it faces challenges in large-scale MARL with numerous agents due to the need to train separate actor networks for each agent, which potentially leads to increased computational burden and suboptimal performance. Foerster et al. [18] proposed a Bayesian Action Decoder (BAD) in 2019. BAD uses neural networks to allow each agent to infer information about their teammates. To avoid infinite recursive thinking, a public belief system is established by the system [19], which represents all publicly available information about the game state, previous actions, and inferred perspectives of other agents without direct observation. Each agent takes action based on their own observations under the guidance of public belief. Throughout the process, the action space consists of deterministic local strategies that can be sampled for a given common state. In multi-agent training methods, inference related to other agents generally involves strategic interactions among intelligent agents.

Opponent modeling has recently emerged as a crucial component within MARL, enabling agents to effectively anticipate and adapt to the behaviors of other learning agents. Recent studies have framed opponent modeling as a probabilistic inference problem. Wen et al. [20] introduced Probabilistic Recursive Reasoning (PR2), which models each agent’s policy as a conditional distribution, recursively considering the potential future actions of other agents. Another significant contribution is ROMMEO (Regularized Opponent Model with Maximum Entropy Objective) [15], which introduces regularization based on maximum entropy principles to stabilize the learning process and jointly optimize both the agent’s policy and its beliefs about opponents. Additionally, Davies [21] proposed Learning to Model Opponent Learning (LeMOL), which develops a structured model of opponents’ learning dynamics. This structured opponent model is shown to be more accurate and stable than naive behavior cloning baselines and can enhance the performance of algorithmic agents in multi-agent settings.

Game-theoretic approaches facilitate decision-making and strategic interactions across multiple application domains by considering model complexity, decision support capabilities, and employing strategy analysis tools. In the context of MARL, competitive games that incorporate multiple factors, such as the number of participants and the diversity of strategies, facilitate the analysis of the consequences and potential outcomes associated with different strategic choices [22]. This enhances the understanding of the complexity and uncertainty inherent in real-world scenarios, thereby assisting decision-makers in formulating strategies and counterstrategies.

This paper combines game training methods with MARL, and proposes a Regularized Competitor Model-based Algorithm (RCM) to enhance the performance of multi-AUV systems in hunting tasks. The main contributions are as follows:

(i)

Integrating the Competitor Model into the Objective Function. By introducing a binary stochastic variable, denoted as ‘optimality’ (o), this study transforms the strategy modeling of a single agent interacting with other agents into a probabilistic inference problem. The likelihood variational lower bound for achieving optimality is derived as a new objective function for MARL, while the strategies of competitors are modeled using a regularization approach. The competitor model can more accurately learn the strategies of other agents, improve the training efficiency of MARL, and achieve faster convergence.

(ii)

Training within a Competitive Game Framework. When converging to a Nash equilibrium, each intelligent agent selects its optimal strategy given a fixed competitor strategy to maximize its own gains. Applying the same reinforcement learning model to both competing sides enables diverse simulation experiments and improves the generalization ability of training.

The rest of this paper is organized as follows: Section 2 introduces the proposed algorithm framework and outlines the basic model and concepts. It incorporates the competitor model into MARL, enabling the inference of other agents’ strategies by optimizing an optimality variable. This leads to the proposal of a Maximum Entropy Reinforcement Learning Algorithm based on Regularized Competitor Model (RCMAC). Section 3 presents a case study on the application of the algorithm in an underwater multi-AUV hunting scenario involving intelligent evaders. Specific configurations for state, action, and reward functions for this scenario are outlined. Section 4 demonstrates the performance of the algorithm in various hunting scenarios and evaluates RCMAC. Finally, Section 5 presents the conclusions.

2. Maximum Entropy Reinforcement Learning Algorithm Based on Regularized Competitor Model

2.1. The Proposed Framework

This study adopts the centralized training and decentralized execution (CTDE) framework similar to MADDPG. Thus, policies may utilize additional information to simplify training, provided such information is not used during testing. This approach is unsuitable for Q-learning since the Q-function typically cannot incorporate different information between training and testing. Consequently, distinct from MADDPG, a competitor model $\rho$ is introduced to explicitly learn additional information about other agents’ strategies during both training and testing.

As shown in Figure 1, within the CTDE framework, the observation restricts the learned policies to use only local information during execution (i.e., their own observation data). The policy network ${\pi }_i$ of agent $i$ makes decisions based on its own observations, and the improved critic target function comprises two parts. During training, it can optimize the information from other agents. Here, $Q_i$, similar to conventional methods, is used to evaluate the action-state value of the agent itself in the current multi-agent scenario, while ${\rho }_i$ is used to quantify the impact of other agents’ current optimal policies on the agent itself. Figure 1 illustrates the process of distributed execution for each intelligent agent within the framework. It is important to note that the policy network relies not only on the current observation state $o_t^i$ of the agent but also makes decisions based on competitor actions ${\widehat{a}}_t^{-i}$ inferred through the competitor model. This iterative process is followed to complete the planning strategy for each agent.

2.2. Competitive Games

For the n-agent competitive game, a tuple $\left(\mathcal{S},{\mathcal{A}}^1,\dots ,{\mathcal{A}}^n,R^1,\dots ,R^n,p,\mathcal{T},\gamma \right)$ is defined, where $\mathcal{S}$ represents the state space, $p$ denotes the initial state distribution, and $\gamma$ refers to the discount factor, For each agent $i\in \{1,\dots ,n\}$, the action set and reward function are given by ${\mathcal{A}}^i {\mathrm{a}\mathrm{n}\mathrm{d} R}^{\mathrm{i}}=R^i\left(s,a^i,a^{-i}\right)$, respectively. The state transitions are given by the transition function $\mathcal{T}:\mathcal{S}\times \mathcal{A}\to \mathcal{S}$, where $\mathcal{A}=\left\{{\mathcal{A}}^1,\cdots ,{\mathcal{A}}^n\right\}$ is the combined action space.

Agent $i$ selects an action $a^i\in {\mathcal{A}}^i$ based on its policy ${\pi }_{{\theta }^i}^i\left(a^i\mid s\right)$, which is parameterized by ${\theta }^i$ given the state $s\in \mathcal{S}$. We define the joint policy across all agents as ${\pi }_{\theta }$, with $\theta$ representing the joint parameters. For convenience, this joint policy can be viewed from agent $i$’s perspective as ${\pi }_{\theta }=\left({\pi }_{{\theta }^i}^i\left(a^i\mid s\right),{\pi }_{{\theta }^{-i}}^{-i}\left(a^{-i}\mid s\right)\right)$, where $a^{-i}={\left(a^j\right)}_{j\ne i}$, ${\theta }^{-i}={\left({\theta }^j\right)}_{j\ne i}$ and ${\pi }_{{\theta }^{-i}}^{-i}\left(a^{-i}\mid s\right)$ denotes the joint policy of all agents other than $i$. In each step of the game, agents choose their actions simultaneously. The goal of each agent is assumed to maximize its expected cumulative reward, formulated as

$$\max {\eta }^i\left({\pi }_{\theta }\right)=\mathbb{E}\left[\sum _{t=1}^{\mathrm{\infty }}{\gamma }^t{R}^i\left(s_t,a_t^i,a_t^{-i}\right)\right]$$

(1)

with $(a_t^i,a_t^{-i})$ sample from $({\pi }_{{\theta }^i}^i,{\pi }_{{\theta }^{-i}}^{-i})$. For fully cooperative scenarios, it is presumed that at least one joint policy ${\pi }_{\theta }$ exists, enabling all agents to attain the highest possible cumulative reward together.

2.3. Regularized Competitor Model

By introducing a binary random variable $o_t^i$, which signifies the “optimality” of agent $i$ at time $t$, we convert the decision-making process into an inference task. In single-agent scenarios, the reward $R(s_t, a_t)$ has a finite bound, but the exact probability of achieving the highest reward for a given action $a_t$ remains uncertain. Hence, $o_t$ can be viewed as a stochastic variable, and its probability distribution can be described as $P(o_t=1\mid s_t,a_t)$, satisfying $P(o_t=1\mid s_t,a_t)\propto \mathrm{e}\mathrm{x}\mathrm{p}\left(R\right(s_t,a_t\left)\right)$. According to this equation, higher rewards imply greater probabilities of optimality ($o_t=1$). However, the concept of “optimality” in multi-agent settings differs slightly from its interpretation in single-agent problems.

$$P\left(o_t^i=1\mid o_t^{-i}=1,s_t,a_t^i,a_t^{-i}\right)\propto \mathrm{e}\mathrm{x}\mathrm{p}\left(R\left(s_t,a_t^i,a_t^{-i}\right)\right)$$

(2)

Here, we define the symbol $\mathcal{J}$ as a general representation of an objective function. ${\mathcal{J}}_X\left(Y\right)$ denotes the objective function of the variable $X$ with respect to parameter $Y$. In particular, $\mathcal{J}$ without subscripts refers to the general form of the reinforcement learning objective; that is, the maximization of expected cumulative rewards.

In the context of cooperative–competitive games, the optimal solution is achieved when all intelligent agents act according to their optimal strategies, namely obtaining the maximum rewards. Thus, assuming that other agents are following their optimal strategies (i.e.,$o_t^i$=1), the probability $P(o^i=1\mid o^{-i}=1)$ of the likelihood of intelligent agent $i$ also adheres to its optimal strategy and, consequently, obtains the maximum reward.

$$\max \mathcal{J}\triangleq \mathrm{l}\mathrm{o}\mathrm{g}P\left(o_{1:T}^i=1\mid o_{1:T}^{-i}=1\right)$$

(3)

Since optimal strategies and environmental models are presumed unknown beforehand, they are considered as latent variables. To optimize the samples observed as defined in Equation (3), following the derivations in reference [23,24,25,26,27], we employ the method of Variational Inference (VI). These latent variables are represented by an auxiliary distribution, denoted as $q\left(a_{1:T}^i,a_{1:T}^{-i},s_{1:T}\mid o_{1:T}^i=\left.1,o_{1:T}^{-i}=1\right)\right.$. By decomposing the above expression according to the modeling assumptions, we obtain

$$\begin{array}{c}q\left(a_{1:T}^i,a_{1:T}^{-i},s_{1:T}\mid o_{1:T}^i=1,o_{1:T}^{-i}=1\right)\hfill \\ =P\left(s_1\right){\displaystyle \prod _t}P\left(s_{t+1}\mid s_t,a_t\right)q\left(a_t^i\mid a_t^{-i},s_t,o_t^i=o_t^{-i}=1\right)\hfill \\ \times q\left(a_t^{-i}\mid s_t,o_t^i=o_t^{-i}=1\right)\hfill \\ =P\left(s_1\right)\prod P\left(s_{t+1}\mid s_t,a_t\right)\pi \left(a_t^i\mid s_t,a_t^{-i}\right)\rho \left(a_t^{-i}\mid s_t\right)\hfill \end{array}$$

(4)

In this decomposition, a lower bound on the probability density of the optimality for the intelligent agent $i$ can be obtained by utilizing the initial state and state transition probabilities as the prior distribution

$$\begin{array}{l}\log P\left(o_{1:T}^i=1\mid o_{1:T}^{-i}=1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\ge \mathcal{J}(\pi ,\rho )\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\triangleq {\displaystyle \sum _t}{\mathbb{E}}_{\left(s_t,a_t^i,a_t^{-i}\right)\sim q}\left[R^i\left(s_t,a_t^i,a_t^{-i}\right)\right.\left.+H\left(\pi \left(a_t^i\mid s_t,a_t^{-i}\right)\right)-D_{\mathrm{K}\mathrm{L}}\left(\rho \left(a_t^{-i}\mid s_t\right)\parallel P\left(a_t^{-i}\mid s_t\right)\right)\right]\end{array}$$

(5)

$$\begin{array}{l}\log P\left(o_{1:T}^i=1\mid o_{1:T}^{-i}=1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\ge {\displaystyle \sum _t}{\mathbb{E}}_{s_t}[\underset{\mathrm{MEO} }{\underbrace{{\mathbb{E}}_{a_t^i\sim \pi ,a_t^{-i}\sim \rho }\left[R^i\left(s_t,a_t^i,a_t^{-i}\right)+H\left(\pi \left(a_t^i\mid s_t,a_t^{-i}\right)\right)\right]}}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\underset{\mathrm{Regularizer} \mathrm{of} \rho }{\underbrace{{\mathbb{E}}_{a_t^{-i}\sim \rho }\left[D_{\mathrm{K}\mathrm{L}}\left(\rho \left(a_t^{-i}\mid s_t\right)\parallel P\left(a_t^{-i}\mid s_t\right)\right)\right]}}\end{array}$$

(6)

where $\rho \left(a_t^{-i}\mid s_t,o_t^{-i}=1\right)$ represents the competitor model for intelligent agent $i$ and is used to estimate the optimal strategy of its competitor. $\pi \left(a_t^i\mid s_t,a_t^{-i},o_t^i=1,o_t^{-i}=1\right)$ is the conditional strategy of agent $i$ under optimal conditions $\left(o_t^i=o_t^{-i}=1\right)$. $P\left(a_t^{-i}\mid s_t,o_t^{-i}=1\right)$ denotes the prior probability of the competitor’s optimal strategy. The prior $P\left(a_t^{-i}\mid s_t,o_t^{-i}=1\right)$ is set as an empirical distribution of competitor actions in a given state. Since the optimization of the objective function is only concerned with the case $\left(o_t^i=1,o_t^{-i}=1\right)$, this term can be eliminated from $\pi , \rho \mathrm{and} P \left(a_t^{-i}\mid s_t\right)$. Equation (5) provides a variational lower bound on $\mathrm{l}\mathrm{o}\mathrm{g}P\left(o_{1:T}^i=1\mid o_{1:T}^{-i}=1\right)$.

Equation (5) can be expanded further into Equation (6), where the first portion corresponds to the maximum entropy formulation used in the SAC algorithm. It is necessary to clarify which component is associated with reward maximization and which relates to entropy maximization, along with a comparative discussion relative to the SAC formulation. For agent $i$, we enhance the standard expected reward by adding the entropy term of its conditional policy, $H\left(\pi \left(a^i\mid s,a^{-i}\right)\right)$. This combined expression serves as the Maximum Entropy Objective (MEO) for agent $i$. It should be emphasized that optimizing the MEO simultaneously contributes to optimizing the competitor model $\rho$.

2.4. Regularized Competitor Model with Maximum Entropy Objective Actor-Critic (RCMAC)

In this section, we propose an extension of the Soft Q-iteration algorithm [23] to multi-agent scenarios. Although our derivation follows a similar reasoning as that presented in reference [23], generalizing Soft Q-learning to a MARL context introduces additional complexity. To accommodate this, we slightly adjust the objective presented in Equation (6) by introducing an entropy weighting factor, $\alpha$. Setting $\alpha =1$ recovers the original objective formulation.

The multi-agent soft Q-function and V-function are defined, respectively, as below. The conditional policies and competitor models defined in Equations (9) and (10) are the optimal solutions with respect to the objectives defined in Equation (6):

The soft state-action value function of agent $i$ is

$$\begin{array}{l}{Q}_{soft}^{{\pi }^{\mathrm{*}},{\rho }^{\mathrm{*}}}\left(s_t,a_t^i,a_t^{-i}\right)=r_t+{\mathbb{E}}_{\left(s_{t+l},a_{t+l}^i,a_{t+l}^{-i},\dots \right)\sim q}\left[{\displaystyle \sum _{l=1}^{\mathrm{\infty }}}{\gamma }^l\left(r_{t+l}\right.\right.\hfill \\ \left.+\alpha H\left({\pi }^{\mathrm{*}}\left(a_{t+l}^i\mid a_{t+l}^{-i},s_{t+l}\right)\right)-D_{\mathrm{K}\mathrm{L}}\left({\rho }^{\mathrm{*}}\left(a_{t+l}^{-i}\mid s_{t+l}\right)\parallel P\left(a_{t+l}^{-i}\mid s_{t+l}\right)\right)\right]\hfill \end{array}$$

(7)

and the soft state value function is

$$V^{\mathrm{*}}(s)=\mathrm{l}\mathrm{o}\mathrm{g}\sum _{a^{-i}}P\left(a^{-i}\mid s\right){\left(\sum _{a^i}\mathrm{e}\mathrm{x}\mathrm{p}\left({\displaystyle \frac{1}{\alpha }}{Q}_{\mathrm{soft} }^{\mathrm{*}}\left(s,a^i,a^{-i}\right)\right)\right)}^{\alpha }$$

(8)

Then, the optimal conditional policy and competitor models for Equation (6) are

$${\pi }^{\mathrm{*}}\left(a^i\mid s,a^{-i}\right)={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\frac{1}{\alpha }{Q}_{\mathrm{soft} }^{{\pi }^{\mathrm{*}},{\rho }^{\mathrm{*}}}\left(s,a^i,a^{-i}\right)\right)}{{\sum }_{a^i}\mathrm{e}\mathrm{x}\mathrm{p}\left(\frac{1}{\alpha }{Q}_{\mathrm{soft} }^{{\pi }^{\mathrm{*}},{\rho }^{\mathrm{*}}}\left(s,a^i,a^{-i}\right)\right)}}$$

(9)

and

$${\rho }^{\mathrm{*}}\left(a^{-i}\mid s\right)={\displaystyle \frac{P\left(a^{-i}\mid s\right){\left({\sum }_{a^i}\mathrm{e}\mathrm{x}\mathrm{p}\left(\frac{1}{\alpha }{Q}_{\mathrm{soft} }^{\mathrm{*}}\left(s,a^i,a^{-i}\right)\right)\right)}^{\alpha }}{\mathrm{e}\mathrm{x}\mathrm{p}\left(V^{\mathrm{*}}\left(s\right)\right)}}$$

(10)

Similar to the derivation of gradient updates in SAC, we define parameterized Q-networks, conditional policy networks, and competitor models as follows: $Q_{\omega }\left(s,a^i,a^{-i}\right),{\pi }_{\theta }\left(a_t^i\mid s_t,a_t^{-i}\right),{\rho }_{\varphi }\left(a_t^{-i}\mid s_t\right)$. Here, the trainable parameters are $\omega$, $\theta$, and $\varphi$ with the objective of minimizing the target function:

$$\begin{array}{cc}{\mathcal{J}}_Q\left(\omega \right)\hfill & ={\mathbb{E}}_{\left(s_t,a_t^i,a_t^{-i}\right)\sim \mathcal{D}}\left[{\displaystyle \frac{1}{2}}\left(Q_{\omega }\left(s_t,a_t^i,a_t^{-i}\right)\right.\right.\hfill \\ & \left.{\left.-R\left(s_t,a_t^i,a_t^{-i}\right)-\gamma {\mathbb{E}}_{s_{t+1}\sim p_s}\left[\overline{V}\left(s_{t+1}\right)\right]\right)}^2\right]\hfill \end{array}$$

(11)

where

$$\begin{array}{cc}\overline{V}\left(s_{t+1}\right)\hfill & =Q_{\overline{\omega }}\left(s_{t+1},a_{t+1}^i,{\widehat{a}}_{t+1}^{-i}\right)-\log {\rho }_{\varphi }\left({\widehat{a}}_{t+1}^{-i}\mid s_{t+1}\right)\hfill \\ & -\alpha \log {\pi }_{\theta }\left(a_{t+1}^i\mid s_{t+1},{\widehat{a}}_{t+1}^{-i}\right)+\log P\left({\widehat{a}}_{t+1}^{-i}\mid s_{t+1}\right)\hfill \end{array}$$

(12)

Adopting the update mechanism of the twin Q-networks, $Q_{\overline{\omega }}$ represents the target action-value function to ensure a relatively stable target value. ${\widehat{a}}_t^{-i}$ denotes the action sampled from the competitor model of intelligent agent $i$, $\rho \left(a^{-i}\mid s\right)$. It is essential to distinguish this from the actual action taken by the competitor $a_t^i$.

Solving the optimal conditional policy and competitor models in Equations (9) and (10) has proven to be challenging in complex unknown environments. Following the approach outlined in reference [24], we constructed an objective function and train parameters $\theta$ and $\phi$ to minimize the KL divergence:

$$\begin{array}{c}{\mathcal{J}}_{\pi }\left(\theta \right)={\mathbb{E}}_{s_t\sim D,a_t^{-i}\sim \rho }\hfill \\ \left[D_{\mathrm{K}\mathrm{L}}\left({\pi }_{\theta }\left(\cdot \mid s_t,{\widehat{a}}_t^{-i}\right)\parallel {\displaystyle \frac{\exp \left(\frac{1}{\alpha }{Q}_{\omega }\left(s_t,\cdot ,{\widehat{a}}_t^{-i}\right)\right)}{Z_{\omega }\left(s_t,{\widehat{a}}_t^{-i}\right)}}\right)\right]\hfill \end{array}$$

(13)

$$\begin{array}{c}{\mathcal{J}}_{\rho }\left(\varphi \right)={\mathbb{E}}_{\left(s_t,a_t^i\right)\sim D}\hfill \\ [D_{\mathrm{K}\mathrm{L}}(\rho (\cdot \mid s_t)\parallel {\displaystyle \frac{P\left(\cdot \mid s_t\right){\left(\frac{\exp \left(\frac{1}{\alpha }Q\left(s_t,a_t^i,\cdot \right)\right)}{{\pi }_{\theta }\left(a_t^i\mid s_t,\cdot \right)}\right)}^{\alpha }}{Z_{\omega }\left(s_t\right)}})]\hfill \end{array}$$

(14)

where ${\widehat{a}}_t^{-i}=g_{\varphi }\left({ϵ}_t^{-i};s_t\right)$, $a_t^i=f_{\theta }\left({ϵ}_t^i;s_t,{\widehat{a}}_t^{-i}\right)$, therefore ${\mathcal{J}}_{\pi }\left(\theta \right)$ and ${\mathcal{J}}_{\rho }\left(\varphi \right)$can be derived as

$$\begin{array}{cc}{\mathcal{J}}_{\pi }\left(\theta \right)\hfill & ={\mathbb{E}}_{s_t\sim D,{ϵ}_t^i\sim N,{\widehat{a}}_t^{-i}\sim \rho }\left[\alpha \mathrm{l}\mathrm{o}\mathrm{g}{\pi }_{\theta }\left(f_{\theta }\left({ϵ}_t^i;s_t,{\widehat{a}}_t^{-i}\right)\right)\right.\hfill \\ & \left.-Q_{\omega }\left(s_t,f_{\theta }\left({ϵ}_t^i;s_t,{\widehat{a}}_t^{-i}\right),{\widehat{a}}_t^{-i}\right)\right]\hfill \end{array}$$

(15)

$$\begin{array}{cc}{\mathcal{J}}_{\rho }\left(\varphi \right)\hfill & =\mid {\mathbb{E}}_{\left(s_t,a_t\right)\sim D,{ϵ}_t^{-i}\sim N}\left[\mathrm{l}\mathrm{o}\mathrm{g}{\rho }_{\varphi }\left(g_{\varphi }\left({ϵ}_t^{-i};s_t\right)\mid s_t\right)\right.\hfill \\ & -\log P\left({\widehat{a}}_t^{-i}\mid s_t\right)-Q\left(s_t,a_t^i,g_{\varphi }\left({ϵ}_t^{-i};s_t\right)\right)\hfill \\ & \left.+\alpha \mathrm{l}\mathrm{o}\mathrm{g}{\pi }_{\theta }\left(a_t^i\mid s_t,g_{\varphi }\left({ϵ}_t^{-i};s_t\right)\right)\right]\hfill \end{array}$$

(16)

The gradients of $Q_{\omega }\left(s,a^i,a^{-i}\right),{\pi }_{\theta }\left(a_t^i\mid s_t,a_t^{-i}\right),{\rho }_{\varphi }\left(a_t^{-i}\mid s_t\right)$ are

$$\begin{array}{cc}{\nabla }_{\omega }{\mathcal{J}}_Q\left(\omega \right)\hfill & ={\nabla }_{\omega }{Q}_{\omega }\left(s_t,a_t^i,a_t^{-i}\right)\left(Q_{\omega }\left(s_t,a_t^i,a_t^{-i}\right)\right.\hfill \\ & \left.-R\left(s_t,a_t^i,a_t^{-i}\right)-\gamma \overline{V}\left(s_{t+1}\right)\right)\hfill \end{array}$$

(17)

$$\begin{array}{c}{\nabla }_{\theta }{\mathcal{J}}_{\pi }\left(\theta \right)={\nabla }_{\theta }\alpha \log {\pi }_{\theta }\left(a_t^i\mid s_t,{\widehat{a}}_t^{-i}\right)+{\nabla }_{\theta }{f}_{\theta }\left({ϵ}_t^i;s_t,{\widehat{a}}_t^{-i}\right)\\ \left({\nabla }_{a_t^i}\alpha \mathrm{l}\mathrm{o}\mathrm{g}{\pi }_{\theta }\left(a_t^i\mid s_t,{\widehat{a}}_t^{-i}\right)-{\nabla }_{a_t^i}{Q}_{\omega }\left(s_t,a_t^i,{\widehat{a}}_t^{-i}\right)\right)\end{array}$$

(18)

$$\begin{array}{cc}{\nabla }_{\varphi }{\mathcal{J}}_{\rho }\left(\varphi \right)\hfill & ={\nabla }_{\varphi }\log {\rho }_{\varphi }\left({\widehat{a}}_t^{-i}\mid s_t\right)+\left({\nabla }_{{\widehat{a}}_t^{-i}}\mathrm{l}\mathrm{o}\mathrm{g}{\rho }_{{\varphi }^i}\left({\widehat{a}}_t^{-i}\mid s_t\right)\right.\hfill \\ & -{\nabla }_{{\widehat{a}}_t^{-i}}\log P\left({\widehat{a}}_t^{-i}\mid s_t\right)-{\nabla }_{{\widehat{a}}_t^{-i}}{Q}_{{\omega }^i}\left(s_t,a_t^i,{\widehat{a}}_t^{-i}\right)\hfill \\ & \left.+{\nabla }_{{\widehat{a}}_t^{-i}}\alpha \mathrm{l}\mathrm{o}\mathrm{g}{\pi }_{\theta }\left(a^i\mid s_t,{\widehat{a}}_t^{-i}\right)\right){\nabla }_{\varphi }{g}_{\varphi }\left({ϵ}_t^{-i};s_t\right)\hfill \end{array}$$

(19)

3. Multi-AUV Hunting with RCMAC

3.1. Environmental State Description

3.1.1. AUV Motion Model

To facilitate the description of the state and action of an AUV, this section initially presents the motion model of the AUV. It is assumed that the underactuated AUV is controlled by horizontal thrusters, fore and aft symmetrical vertical thrusters, and vertical rudders in a 3-dimensional underwater environment.

Figure 2 illustrates that $\phi , \theta , \mathrm{a}\mathrm{n}\mathrm{d} \psi$ represent the heeling angle, trim angle, and slant angle within the AUV’s inertial coordinate system (positive values indicate counterclockwise rotation). The velocity components of the AUV in the carrier coordinate frame are denoted by $u, v, \mathrm{a}\mathrm{n}\mathrm{d} w$. Additionally, $p, q, \mathrm{a}\mathrm{n}\mathrm{d} r$ represent the angular velocity components in the same carrier coordinate frame. Typically, the AUV’s state is described using six degrees of freedom within the inertial coordinate system, represented as the vector ${\eta }_p=[x,y,z,\phi ,\psi ,\theta ]$ [28]. To simplify calculations, this study assumes a simplified AUV model with vertical thrusters enabling movement along the Z-axis. Since lateral shifts and rolling movements are generally restricted under normal operational conditions, the inertial coordinate system’s attitude reduces to ${\eta }_p=[x,y,z,\psi ]$.

Furthermore, the AUV’s motion within the carrier coordinate frame is represented using the velocity vector ${\eta }_d=(u,w,r)$. Here, $\psi$ indicates the angle between the forward velocity component $u$ and the X-axis, $w$ denotes vertical velocity along the Z-axis, and $r$ corresponds to the rate of change of $\psi$ ($r=\dot{\psi }$). Therefore, the simplified motion model for the AUV can be described as follows:

$$\left\{\begin{array}{c}\dot{x}=u\times \cos \psi \hfill \\ \dot{y}=u\times \sin \psi \hfill \\ \dot{z}=w\hfill \\ \dot{\psi }=r\hfill \end{array}\right.$$

(20)

The velocity change of the AUV is defined as the action, and represented as

$$a=(\mathsf{\Delta }u,\mathsf{\Delta }w,\mathsf{\Delta }\psi )$$

(21)

Based on the above formulas, the self-state of the AUV can be described by ${\eta }_p \mathrm{a}\mathrm{n}\mathrm{d} {\eta }_d$, and expressed as

$$\eta =\left[{\eta }_p,{\eta }_d\right]=[x,y,z,\psi ,u,w,r]$$

(22)

3.1.2. State of the Competitors (Including Cooperators and the Evader)

The position information of cooperators and evader is necessary for the AUV group to encircle the evader [29]. As shown in Figure 3, the relative positions of the cooperating agents and the evader in a three-dimensional environment are indicated by the coordinates $C^{ij}=\left({\varphi }^{ij},d_p^{ij},d_z^{ij}\right)$. $N-1$ vectors extend from the current location of the AUV $i$ toward competitor $j$ (either cooperators or the evader). Where, $d_p^{ij}$ denotes the projected length of vector onto the horizontal (XOY) plane, ${\varphi }^{ij}$ represents the angle formed between vector and the positive X-axis, and $d_z^{ij}$ specifies the vertical projection length along the Z-axis. Consequently, the relative state between the AUV, its cooperators, and the evader can be expressed by the following equation:

$$C^i=\bigcup _{j=1}^{N-1}\left({\varphi }^{ij},d_p^{ij},d_z^{ij}\right)$$

(23)

During the training process of the reinforcement learning algorithm, the state of each AUV is comprised of the pose $\eta$ and the relative positions $C$ with respect to the competitors. Therefore, $s_t$ can be described as $s_t=\left\{{\eta }_t,C_t\right\}$.

3.2. Criteria for a Successful Hunting Progress

In the hunting scenario presented in this paper, a group of AUVs are tasked with hunting an intelligent evader. Assuming there are n AUVs (AUV1, AUV2, …, AUVn), and each AUV is equipped with a special device capable of disrupting the mobility of the evader. It is assumed that the mobility of the evader will be restricted when any hunting AUV successfully approaches it within a range of 1 m. The process of hunting involves the movement of hunting AUVs towards the evader within the hunting cluster. The hunting AUVs need to surround the evader as closely as possible and evenly distribute around them around the evader to restrict their escape direction. At least one hunting AUV attempting to approach the evader to limit its mobility is considered a successful hunting progress.

The multi-AUV hunting of an evader in a three-dimensional underwater environment is depicted in Figure 4. AUV1 to AUVn represent the hunting target points within the encircling circle range, and T denotes the evader. The success of the hunting primarily relies on the following three criteria:

• The distance between the center of the hunting formation and the evader is less than a threshold (5 m);
• The distance between the hunting AUVs and the evader is less than a threshold (10 m);
• There exists at least one hunting AUV with a distance of less than 1 m from the evader.

3.3. Reward Functions of the Hunters

For effective multi-AUV hunting, it is essential to maintain a specific formation so the AUVs can cooperatively surround the evader from multiple directions. Additionally, since the evader will actively try to break through the encirclement, the navigation goals for the AUVs must be updated continuously in response. Considering these complexities, establishing a suitable reward function is critical. In this section, we propose a continuous and modular reward function to enhance the efficiency of training. The reward structure is organized into the following three categories, corresponding to various scenarios encountered during the hunting process.

3.3.1. Cooperating Rewards

To effectively utilize cooperation among AUVs during target detection and pursuit initiation, it is important for the vehicles to maintain a suitable surrounding formation. Ideally, the distance between each AUV and the centroid of the group should remain within a controlled range, and a safe minimum separation must be maintained among individual vehicles to prevent collisions.

The cooperating reward function is therefore defined as follows:

$$r_{\mathrm{encircle} }=\left\{\begin{array}{cc}1& \mathrm{if} d_{\mathrm{center} }\in \left[\mathrm{0,10}\right]\\ 0& \mathrm{else} \end{array}\right.$$

(24)

$$r_{\mathrm{holdingrange} }=\left\{\begin{array}{cc}0& \mathrm{if} min{d}_{\mathrm{other} }>15\\ -0.5& \mathrm{if} min{d}_{\mathrm{other} }\in \left[\mathrm{10,15}\right]\\ -1& \mathrm{if} d_{\mathrm{other} }<10\end{array}\right.$$

(25)

where $d_{\mathrm{center} }$denotes the distance between each hunter and the geometric center of the hunter group, while $d_{\mathrm{other} }$ represents the set of distances between each hunter and the other hunters.

3.3.2. Hunting Rewards

During the training phase, it is assumed that when an AUV detects an evader, it alerts the other hunters, prompting the entire team to initiate pursuit. Hunters are thus motivated to track and move closer to the evader. Additionally, we account for scenarios where the overall group moves closer, but an individual hunter drifts further from the evader. Under such circumstances, the respective AUV receives a positive reward to guide its actions toward fulfilling the group’s collective goal.

$$r_{\mathrm{hunting} }=\left\{\begin{array}{cc}0& \mathrm{if} \mathrm{m}\mathrm{i}\mathrm{n}{d}_{\mathrm{evader}}>20\\ 2& \mathrm{if} d_{\mathrm{evader} }^{t+1}<d_{\mathrm{evader} }^t\\ -1& \mathrm{if} d_{\mathrm{evader}}^{t+1}\ge d_{\mathrm{evader} }^t\\ 1& \mathrm{if} d_{\mathrm{center} \mathrm{to} \mathrm{evader} }^{t+1}<d_{\mathrm{center} \mathrm{to} \mathrm{evader} }^t\end{array}\right.$$

(26)

where $d_{\mathrm{evader}}$ denotes the distance between each hunter and the evader, while $d_{\mathrm{center} \mathrm{to} \mathrm{evader} }^t$ represents he distance between the evader and the geometric center of the hunter group.

3.3.3. Finishing Rewards

An episode terminates under three possible scenarios: (a) any AUV reaches the maximum allowed number of steps for its mission, (b) collision between two or more AUVs, or (c) successful encirclement of the evader by hunter AUVs, effectively restricting its potential escape routes. In each of these scenarios, the hunters receive a final reward $r_{\mathrm{finishing} }$, which signals the end of the current episode. The final rewards are specifically defined as follows:

$$r_{\mathrm{finish} }=\left\{\begin{array}{cc}100& \mathrm{if} \mathrm{situation} \left(\mathrm{c}\right) \\ -100& \mathrm{if} \mathrm{situation} \left(\mathrm{a}\right) \mathrm{or} \left(\mathrm{b}\right) \end{array}\right.$$

(27)

Each AUV receives total rewards based on the following factors:

$$r_{\mathrm{hunter} }=r_{\mathrm{encircle} }+r_{\mathrm{holdingrange} }+r_{\mathrm{hunting} }+r_{\mathrm{finish} }$$

(28)

3.4. Reward Functions of the Evader

In the hunting–evading game scenario presented in this study, the evader’s action and state spaces are configured to be consistent with those of the hunters, and the evader employs the same algorithmic architecture. The key difference lies in the reward function designed for the evader. Specifically, a target point located outside the designated area is assigned to the evader to guide its escape direction. The evader receives rewards when approaching this target point and successfully exiting the boundary of the scene. Moreover, the hunting–evading interaction is formulated as a zero-sum game, meaning that from the evader’s perspective, it additionally receives a reward signal opposite to that of the hunters’ reward. These rewards are defined as follows:

$$r_{\mathrm{evading} }=\left\{\begin{array}{cc}2& \mathrm{if} d_{\mathrm{T} }^{t+1}<d_{\mathrm{T} }^t\\ -1& \mathrm{if} d_{\mathrm{T}}^{t+1}\ge d_{\mathrm{T} }^t\end{array}\right.$$

(29)

$$r_{\mathrm{evader} }=r_{\mathrm{evading} }-r_{\mathrm{hunter}}$$

(30)

where $d_T$ refers to the distance between the evader and the target point.

3.5. Competitive Training Process

The training process of RCMAC employs a centralized training with a distributed execution approach following the procedure below.

Initialize the replay pool and neural network parameters for all AUVs, including the joint policy network ${\pi }_{{\theta }^i}$, competitor model network ${\rho }_{{\varphi }^i}$, and Q-network $Q_{\bar{\omega }}$.

Repeat the following steps until convergence or the maximum iteration count is reached: (a) Each agent predicts the optimal actions of competitors based on the current competitor model. (b) Each agent calculates and selects actions based on the local observations of current state (excluding competitor states) and the competitor actions. (c) All AUVs compute the global environmental state and rewards after executing actions. (d) Each AUV updates the actual competitor actions and stores them in the experience replay buffer. (e) Each AUV samples competitor action samples from the replay pool and updates the competitor model. (f) Calculate target Q-value using the target network, and update the local and target Q-network parameters, as well as the policy network parameters for each agent. Algorithm 1 provides a step-by-step overview of the training process described above.

Algorithm 1 RCMAC in Multi-AUV Hunting

Initialization: Initialize replay buffer $\mathcal{M}$ and its capacity $M$ $\mathrm{Initialize} \mathrm{parameters} {\theta }^i,{\varphi }^i,{\omega }^i,{\psi }^i$$\mathrm{for} \mathrm{each} \mathrm{AUV} i$$\mathrm{and} \mathrm{the} \mathrm{random} \mathrm{process} \mathcal{N}={\left[N\right(\mathrm{0,1}\left)\right]}^3$ for action space $\mathrm{Initialize} \mathrm{learning} \mathrm{rates} {\lambda }_V,{\lambda }_Q,{\lambda }_{\pi },{\lambda }_{\varphi },\alpha$$\mathrm{and} \mathrm{set} \gamma$ as the discount factor. For each iteration do $\mathrm{Initialize} \mathrm{random} \mathrm{process} \mathcal{N}$ for action exploration. $\mathbf{For} \mathrm{each} \mathrm{time} \mathrm{step} t$do $\mathrm{For} \mathrm{the} \mathrm{current} \mathrm{state} s_t$, sample an action and competitor’s action using $\begin{array}{ll}& {\hat{a}}_t^{-i}\leftarrow g_{{\varphi }^{-i}}\left({ϵ}^{-i};s_t\right), \mathrm{where} {ϵ}_t^{-i}\sim \mathcal{N},\\ & a_t^i\leftarrow f_{{\theta }^i}\left({ϵ}^i;s_t,{\hat{a}}_t^{-i}\right), \mathrm{where} {ϵ}_t^i\sim \mathcal{N}.\end{array}$ $\mathrm{Observe} \mathrm{next} \mathrm{state} s_{t+1}$$, \mathrm{competitor} \mathrm{action} a_t^{-i}$$\mathrm{and} \mathrm{reward} r_t^{-i}$, save the new experience in the replay buffer: ${\mathcal{D}}^i\leftarrow {\mathcal{D}}^i\cup \left\{\left(s_t,a_t^i,a_t^{-i},{\widehat{a}}_t^{-i},s_{t+1},r_t^i\right)\right\}$ Update the prior from the replay buffer: ${\psi }^i=\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}{\mathbb{E}}_{{\mathcal{D}}^i}\left[-P\left(a^{-i}\mid s\right)\mathrm{l}\mathrm{o}\mathrm{g}{P}_{{\psi }^i}\left(a^{-i}\mid s\right)\right]$ Sample a mini-batch from the reply buffer: ${\left\{s_t^{\left(n\right)},a_t^{i,\left(n\right)},a_t^{-i,\left(n\right)},{\widehat{a}}_t^{-i,\left(n\right)},s_{t+1}^{\left(n\right)},r_t^{\left(n\right)}\right\}}_{n=1}^N~\mathcal{M}.$ $\mathrm{For} \mathrm{the} \mathrm{state} s_{t+1}^{\left(n\right)}$, sample an action and competitor’s action using $\begin{array}{ll}& {\hat{a}}_{t+\left(n\right)}^{-i,\left(n\right)}\leftarrow g_{{\varphi }^{-i}}\left({ϵ}^{-i};s_{t+1}^{\left(n\right)}\right), \mathrm{where} {ϵ}_{t+1}^{-i}\sim \mathcal{N},\\ & a_{t+1}^{i,\left(n\right)}\leftarrow f_{{\bar{\theta }}^i}\left({ϵ}^i;s_{t+1}^{\left(n\right)},{\hat{a}}_{t+\left(1\right)}^{-i)}\right), \mathrm{where} {ϵ}_{t+1}^i\sim \mathcal{N}.\\ & \end{array}$ ${\bar{V}}^i\left(s_{t+1}^{\left(n\right)}\right)=Q_{\bar{\omega }}\left(s_{t+1}^{\left(n\right)},a_{t+1}^{i,\left(n\right)},{\widehat{a}}_{t+1}^{-i,\left(n\right)}\right)-\alpha \mathrm{l}\mathrm{o}\mathrm{g}{\pi }_{{\theta }^i}\left(a_{t+1}^{i,\left(n\right)}\mid s_{t+1}^{\left(n\right)},{\widehat{a}}_{t+1}^{-i,\left(n\right)}\right)-\mathrm{l}\mathrm{o}\mathrm{g}{\rho }_{{\varphi }^i}\left({\widehat{a}}_{t+1}^{-i,\left(n\right)}\mid s_{t+1}^{\left(n\right)}\right)+\mathrm{l}\mathrm{o}\mathrm{g}{P}_{{\psi }^i}\left({\widehat{a}}_{t+1}^{-i,\left(n\right)}\mid s_{t+1}^{\left(n\right)}\right).$ Set: $y^{\left(n\right)}=\left\{\begin{array}{ll}{r}_t^{\left(n\right)}& \mathrm{for} \mathrm{terminal} s_{t+1}^{\left(n\right)}\\ r_t^{\left(n\right)}+\gamma {\bar{V}}^i\left(s_{t+1}^{\left(n\right)}\right)& \mathrm{for} \mathrm{non}-\mathrm{terminal} s_{t+1}^{\left(n\right)}\end{array}\right.$ $\mathrm{Calculate}: {\nabla }_{{\omega }^i}{\mathcal{J}}_Q\left({\omega }^i\right)$$, {\nabla }_{{\theta }^i}{\mathcal{J}}_{\pi }\left({\theta }^i\right)$$, {\nabla }_{{\varphi }^i}{\mathcal{J}}_{\rho }\left({\varphi }^i\right)$ Update parameters: $\begin{array}{ll}& {\omega }^i={\omega }^i-{\lambda }_Q{\nabla }_{{\omega }^i}{\mathcal{J}}_Q\left({\omega }^i\right)\\ & {\theta }^i={\theta }^i-{\lambda }_{\pi }{\nabla }_{{\theta }^i}{\mathcal{J}}_{\pi }\left({\theta }^i\right)\\ & {\varphi }^i={\varphi }^i-{\lambda }_{{\varphi }^i}{\nabla }_{{\varphi }^i}{\mathcal{J}}_{\rho }\left({\varphi }^i\right)\end{array}$ End for End for

4. Simulation Results

In this section, multiple unbalanced game scenarios are demonstrated to validate the efficacy of the proposed algorithm. Comparisons with the MADDPG algorithm are also presented to showcase the superiority of the proposed algorithm.

4.1. Experimental Environment and Training Parameters

The RCMAC algorithm was implemented on a training workstation equipped with an Intel Core i9 11900K processor (8 cores, 5.1 GHz, Intel Corporation, Santa Clara, CA, USA) and a GPU with 24 GB memory (RTX 3090, NVIDIA Corporation, Santa Clara, CA, USA). The deep neural network models were built using Pytorch (Version 1.13.0) and trained using GPU acceleration. Initially, the RCMAC hyperparameters were set based on reference [15], while the parameters for Maximum Entropy Reinforcement Learning in the hunting scenario mainly followed those outlined in our earlier study [29].

Table 1. Hyperparameters of RCMAC.

Hyperparameter	Value
Learning rate of all networks	0.0003
$\mathrm{Discount} \mathrm{factor} \gamma$	0.97
Mini batch size	256
Replay buffer size M	1024
Training episode num	5000
Testing episode num	500
Max step num	1024

summarizes the hyperparameter settings.

$\mathrm{Discount} \mathrm{factor} \gamma$

4.2. Performance and Analysis

To demonstrate the performance of the proposed algorithm in the pursuit–evasion competitive game, there are three scenarios including 1, 4, and 6 hunters hunting a single evader, respectively. $I(I_x,I_y,I_z)$ denotes the initial position of the hunters and the evader. In each scenario, the evader starts from the initial position $I_T\left(\mathrm{350,55,65}\right)$ and sails towards outer boundary $T\left(\mathrm{100,502,175}\right)$ to escape from the hunting area.

4.2.1. Scenario A: One Hunter–One Evader Game

In scenario A, only one hunter is deployed to pursue the evader. The maximum speeds of the hunter and the evader are 2.7 m/s and 2.5 m/s, respectively. The initial position of the AUV is $I_A\left(\mathrm{200,100,80}\right)$. Following the definition of successful hunting mentioned in Section 3.2, the hunter needs to chase the evader, and the hunting is deemed successful when its distance from the evader is less than 1 m. In the test scenario spanning 500 episodes, the trajectories of the hunter and the evader are depicted in Figure 5, where the pink line represents the trajectory of the evader, and the green line indicate the trajectories of the hunter.

Figure 5a illustrates the trajectory of the hunter’s first successful hunting within the 500-episode experimental scenario, approaching the evader within a range of 1 m. The evader is observed to be hunted as it nears the designated escape boundary, with relatively long pursuit and evasion paths. As the training iterations increase, the pursuit and evasion paths gradually shorten, and the time required for successful hunting decreases, as depicted in Figure 5b,c. The increase in training iterations leads the competitive game training process toward a strategic equilibrium between the pursuer and the evader. The inherent speed advantage of the hunter has been demonstrated to facilitate earlier successful capture of the evader. Figure 6 illustrates the relevant curves of this process, where Figure 6a shows the cumulative hunting success rate and escape success rate for both the hunter and the evader as the number of training iterations increases. Figure 6b–d, respectively, show the conclusion steps, total rewards per episode, and the average rewards per step for both the hunter and the evader. These curves demonstrate that with the progression of the adversarial training, the hunter gradually finds shorter paths to hunt the evader, maximizing the potential for higher rewards.

4.2.2. Scenario B: Four Hunters–One Evader Game

In this scenario, four hunters are deployed to pursue the evader, and are strategically positioned in the designated environment. The initial position of the AUV is $I_A\left(\mathrm{200,100,80}\right),I_B\left(\mathrm{50,200,80}\right),I_C\left(\mathrm{100,450,60}\right),I_D\left(\mathrm{450,450,40}\right)$. The maximum speed of each hunter is reduced to 2.6 m/s, while the evader maintains a maximum speed of 2.5 m/s. Similarly, following the defined criteria for successful hunting in Section 3.2, four hunters progressively approach and encircle the evader during their navigation. The pursuit culminates as the closest hunter attempting to approach and hunt the evader in real-time. In the 500-episode test scenario, the trajectories of the hunters and the evader are depicted in Figure 7, where the pink line represents the trajectory of the evader, and the other lines indicate the trajectories of the hunters.

Figure 7a illustrates the trajectory of the hunter’s first successful hunting in the 500-episode experimental scenario, approaching the evader within a range of 1 m. Similar to the 1v1 pursuit–evasion scenario, the evader is hunted before it approaches the designated escape boundary, resulting in a relatively long pursuit and evasion path. However, the speed advantage of the hunters over the evader is intentionally weakened in this scenario. Under this premise, the hunters achieve their first successful hunting with fewer episodes (21 episodes) than in Scenario A. As training iterations increase, the pursuit and evasion paths gradually shorten, and the time required for successful hunting decreases, as shown in Figure 7b,c. The increase in training iterations determines that the hunters tend to employ shorter paths to achieve faster hunting. Figure 8 presents the relevant curves of this process, and Figure 8a displays the cumulative hunting and escape success rates for both the hunter and the evader as training iterations increase. Figure 8b–d, respectively, show the conclusion steps, total rewards per episode, and the average rewards per step for both the hunter and the evader. In the three curves of Scenario A, due to the weak speed advantage of hunters and the lack of multiple AUVs to restrict the escape of the evader, the reward curves of the evader in Figure 8c,d exhibit a relatively slow declining trend. However, in this scenario, a significant numerical advantage of hunters enables them to better constrain the escape of evader. Compared to the curves in Scenario A, the reward curve of the evader exhibits a more pronounced declining trend.

4.2.3. Scenario C: Six Hunters–One Evader Game

In this scenario, six hunters are deployed to pursue the evader, and strategically positioned in the designated environment. The initial position of the AUV is $I_A\left(\mathrm{200,100,80}\right),$ $I_B\left(\mathrm{50,200,80}\right),I_C\left(\mathrm{450,450,40}\right),I_D\left(\mathrm{100,450,60}\right),I_E\left(\mathrm{100,100,100}\right),I_F\left(\mathrm{300,450,140}\right)$. The maximum speed of each hunter is reduced to 2.6 m/s, while the evader maintains a maximum speed of 2.5 m/s. Other configurations are identical to those in Scenario B. In the 500-episode test scenario, the trajectories of the hunters and the evader are depicted in Figure 9, where the pink line represents the trajectory of the evader, and the other lines indicate the trajectories of the hunters.

In comparison to Scenario B, scenario C aims to explore the algorithm’s capability to support a larger-scale hunting group and whether the performance satisfies expectations. Intuitively, six hunters should hunt the evader faster and easier. Figure 9a indicates that Scenario C successfully hunted for the first time in only three episodes under similar initial conditions. Similarly, as training iterations increase, the pursuit and evasion paths gradually shorten, and the time required for successful hunting decreases, as shown in Figure 9b,c. Figure 10 presents the relevant curves of this process. Compared to the curves in Scenario B, the reward curve of the evader in Scenario C exhibits faster decline, and earlier convergence. This indicates that six hunters impose stronger constraints on the evader’s escape; that is, the algorithm maintains good performance as the number of hunters in the hunting group increases.

4.2.4. Performance Comparison with MADDPG

In the same simulated experimental setting as Scenario C, the training curves of MADDPG and RCMAC are represented as the average rewards obtained by six hunters. Figure 11 illustrates that the average reward obtained by hunters in the RCMAC algorithm is higher than that of the MADDPG algorithm, and the number of steps required for the successful hunting is fewer than that of MADDPG algorithm. This suggests that RCMAC tends to capture the evader faster through more efficient paths. Figure 12 presents the success rate curves of hunting using RCMAC and MADDPG in the aforementioned three scenarios, and it can be observed that MADDPG also exhibits a similar trend. The increase in the number of captors results in earlier detection and successful hunt of the evader. However, RCMAC consistently demonstrates a higher hunting success rate than MADDPG in each scenario.

As shown in Figure 13, the learning curves of policy mean of RCMAC and MADDPG are compared over 500 episodes. It can be observed that RCMAC achieves faster convergence in competitor policy modeling (${\mu }_{\rho 1}$) compared to MADDPG. Specifically, the competitor model in RCMAC reaches its optimal value earlier than its corresponding actual policy (${\mu }_{\pi e}$), indicating that RCMAC can accurately anticipate competitor strategies in advance during training. In contrast, although MADDPG also converges to the optimal policy ultimately, its competitor model exhibits slower convergence and only catches up with the actual policy at later training stages. This implies that RCMAC can establish a more reliable competitor model earlier, providing stronger strategic guidance during policy optimization. Overall, these results validate the effectiveness of the proposed algorithm in enhancing learning efficiency compared to baseline methods.

5. Conclusions

This paper investigates a competitive game process among agents in MARL. An optimality operator is introduced to establish a competitor model during the training process and to reconstruct the objective function. Based on this approach, the RCMAC algorithm is proposed and applied to a scenario involving multiple AUVs hunting evading targets in a 3D underwater environment. The evader is treated as an intelligent agent with learning capabilities; its escape ability is optimized through training, thereby enhancing the randomness of its actions and simulating more complex hunting scenarios. This training strategy enriches the complexity and diversity of the simulated environment, enabling the hunting AUVs’ policies to achieve greater adaptability. In the simulation experiments, RCMAC demonstrated superior convergence and performance across multiple competitive game scenarios. Specifically, as shown in Figure 11, the average cumulative rewards obtained by RCMAC are consistently higher than those achieved by MADDPG across different test scenarios. Figure 12 further shows that RCMAC achieved a higher hunting success rate compared to the baseline MADDPG algorithm. Moreover, the learning curves presented in Figure 13 illustrate that RCMAC converges faster in both policy optimization and competitor modeling.

Overall, these results validate the effectiveness of the proposed RCMAC algorithm in improving learning efficiency and task performance in multi-agent competitive environments. For future work, we plan to address the challenges encountered in larger-scale AUV hunting scenarios, where the computational burden and training time significantly increase due to the expansion of the action and state spaces. Reducing computational complexity and improving training efficiency to enhance the scalability of MARL algorithms in larger team settings will be an important focus of our future research.