Cooperative Patrol Control of Multiple Unmanned Surface Vehicles for Global Coverage

Abstract

The cooperative patrol control of multiple unmanned surface vehicles (Multi-USVs) in dynamic aquatic environments presents significant challenges in global coverage efficiency and system robustness. The study proposes a cooperative patrol control algorithm for multiple unmanned surface vehicles (Multi-USVs) based on a hybrid embedded task state information model and reward reshaping techniques, addressing global coverage challenges in dynamic aquatic environments. By integrating patrol, collaboration, and obstacle information graphs, the algorithm generates kinematically feasible control actions in real time and optimizes the exploration-cooperation trade-off through a dense reward structure. Simulation results demonstrate that the algorithm achieves 99.75% coverage in a 1 km × 1 km task area, reducing completion time by 23% and 74% compared to anti-flocking and partition scanning algorithms, respectively, while maintaining collision rates between agents (CRBAA) and obstacles (CRBAO) below 0.15% and 0.5%. Compared to DDPG, SAC, and PPO frameworks, the proposed training framework (TFMUSV) achieves 28% higher rewards with 40% smaller fluctuations in later training stages. This study provides an efficient and reliable solution for autonomous monitoring and search-rescue missions in complex aquatic environments.

1. Introduction

Multi-agent cooperative patrolling tasks involve multiple agents working together within a predefined area to collect environmental data or monitor the occurrence of sudden events through continuous patrolling. One of the core challenges in this task is the multi-agent cooperative coverage control method. This method requires ensuring full coverage of the entire task area while minimizing the length of the patrolling path through efficient path planning and control strategies, thereby improving patrolling efficiency [1]. To achieve this goal, researchers have proposed various optimization-based cooperative strategies aimed at enhancing the adaptability and execution efficiency of multi-agent systems in complex environments. These strategies not only involve communication and coordination mechanisms between agents but also emphasize how to dynamically adjust patrolling routes in response to sudden situations in a changing environment [2,3].

Given the importance of this task, this paper proposes a cooperative patrol control algorithm for Multi-USVs, aimed at global coverage. The algorithm models the mission scenario using a hybrid embedded task state information model, combined with reward reshaping techniques, enabling each node in the fleet to be trained based on real-time information, thus dynamically generating optimal control commands during mission execution. This modeling approach not only enhances the cooperative patrol efficiency of the Multi-USV system in complex aquatic environments but also provides data support for subsequent cooperative search and interception tasks. Through this method, the study introduces new ideas for decision-making in dynamic environments for Multi-USV systems, with the potential to improve system performance while optimizing the utilization of computational and communication resources.

2. Related Work

There have been many achievements in the research of multi-agent area coverage control methods. Xu et al. [4,5,6] provided a systematic summary and classification of existing area coverage control strategies. They divided the current common area coverage methods into two main categories: one is online algorithms based on a known environmental map, and the other is offline algorithms for unknown environmental maps. Furthermore, based on different approaches to area decomposition, these methods can be further classified into exact-approximate cell decomposition methods and approximate cell decomposition methods. Exact-approximate cell decomposition methods ensure the efficiency and accuracy of the coverage task by precisely dividing the area units, while approximate cell decomposition methods optimize computational complexity and resource consumption, making them suitable for more complex real-world applications. This classification system provides a clear framework for the research of multi-agent area coverage control and offers a theoretical foundation for further algorithm optimization and application promotion, as shown in Figure 1.

2.1. Exact-Approximate Cell Decomposition Method

The exact-approximate cell decomposition method is a method that achieves complete coverage of the entire task area by dividing it into a set of non-overlapping units. The collection of these cells is strictly equal to the free space, and there are no obstacles within them, allowing robots to use simple motion patterns for efficient coverage. Trapezoidal decomposition, plowing decomposition, and Voronoi diagram decomposition are representative techniques of exact-approximate cell decomposition, widely applied in the cooperative patrolling field of multi-agent systems, and have been extensively studied and practically validated.

In the trapezoidal decomposition method, the task area is divided into a series of trapezoidal units (as shown in Figure 2a), with each unit being covered through simple back-and-forth motion. By traversing the adjacency graph related to the decomposition, complete coverage can be ensured, ultimately generating a serrated path for the coverage units. Research on trapezoidal decomposition has been applied across multiple fields. For example, reference [7] proposed a path planning algorithm for water depth mapping that combines trapezoidal decomposition with coverage strategies, designed to generate survey paths for multiple convex regions in coastal areas, showing high reliability in practical applications. Reference [8] further explored the influence of ocean currents on the scanning direction in water depth mapping path planning, optimizing path planning by integrating the trapezoidal decomposition method. Additionally, ref. [9] proposed a coverage path planning framework for unmanned surface vehicle-assisted depth measurement, modeling the depth measurement task as an integer programming problem. This framework optimizes path length, region connectivity, and scanning direction, achieving efficient coverage through binary convex decomposition and layered heuristic algorithms. Performance advantages were validated in lake tests. Ref. [10] introduced a multi-drone flood coverage algorithm that combines deep reinforcement learning with a path dispersion strategy. By using D-infinity water flow estimation techniques to optimize path planning, the dispersion strategy prevents drone clustering in low-altitude areas. This algorithm outperforms existing methods in terms of accumulated rewards and coverage uniformity, demonstrating superior performance. These studies indicate that the trapezoidal decomposition method has immense potential in path planning and area coverage applications.

Due to the larger number of units generated by trapezoidal decomposition, multiple units may merge to form larger units, which can significantly increase the coverage path length and thus affect coverage efficiency. To address this issue, the plow-style unit decomposition method was proposed to improve path planning performance. Similar to trapezoidal decomposition, the plow-style decomposition only extends the vertices of vertical line segments (as shown in Figure 2), thus reducing the number of units and optimizing the coverage path. In related studies, Reference [11] proposed an improved Grey Wolf Algorithm for UAV path planning, optimizing flight efficiency and safety. This method accelerates convergence by dynamically adjusting the weight distribution and micro-oscillation convergence factor, enhancing the global optimum search capability. Under various constraints, this method demonstrated excellent performance, significantly improving UAV path planning efficiency and accuracy, with important application value. Reference [12] proposed an innovative UAV battery management method aimed at optimizing energy-aware task planning. By considering battery replacement requirements, this method optimizes the flight path to reduce overall flight time and avoid unnecessary long-distance flights. Reference [13] proposed a novel deep reinforcement learning method for multi-robot information path planning, aimed at mapping target areas in unknown 3D environments. The key innovation of this method is the introduction of enhanced graphs to model the trajectories of other robots, enabling communication and collision-avoidance planning between robots. Additionally, plow-style decomposition has been widely applied in UAV inspection [14] and Autonomous Underwater Vehicle (AUV) [15] fields. In path optimization, Reference [16] proposed a new meta-heuristic algorithm to solve AUV path planning problems, combining weighted center methods and fitness-based parameter selection schemes, effectively handling AUV path planning tasks in different scenarios. It also demonstrates strong adaptability in underwater environment modeling and threat modeling. Meanwhile, Reference [16] utilized an improved Sparrow Search Algorithm to plan the optimal path and combined consensus algorithms and artificial potential fields for obstacle avoidance and cooperative formation motion. Through golden search optimization and adaptive iteration methods, this approach showed strong feasibility in formation control in obstacle-rich environments. These studies fully validate the superiority and application value of the plow-style unit decomposition method in complex scenarios.

The trapezoidal decomposition and plow-style decomposition methods, by using fixed shapes (such as trapezoids or parallelograms) to partition the task area, are computationally efficient and relatively simple to implement. However, the fixed shapes of these methods limit their adaptability when dealing with complex task areas, especially those with irregular curved boundaries, multiple concave and convex parts, or non-polygonal shapes. These methods struggle to precisely align with the actual boundaries of such areas. For example, when performing mapping or coverage control in natural terrains such as mountains or rivers, these methods may result in partitioned units that do not match the actual terrain, leading to under-coverage or excessive coverage in some regions. This reduces the practical effectiveness and adaptability of the methods. To address these limitations, researchers have proposed the Voronoi diagram decomposition method, as shown in Figure 3. Based on given generator points (such as the initial positions of agents), the Voronoi diagram decomposition naturally divides the task area into multiple irregularly shaped subregions that closely match the actual shape of the area. This method overcomes the limitations of fixed shapes in trapezoidal and plow-style decompositions and is capable of handling task areas with various shapes, including convex polygons, concave polygons, and complex curved boundaries. It reduces the issues of incomplete or redundant coverage caused by decomposition methods and provides a partitioning solution that better meets the requirements of the actual environment.

Soleymani et al. [17] proposed a multi-agent area coverage control method based on the Bellman optimality framework. This approach achieves optimal tracking through an adaptive control strategy combined with actor-critic neural network reinforcement learning techniques, relying on local data rather than a global model. It was demonstrated that when agents converge to Voronoi centers, they can maximize area coverage metrics, which aligns with the solution of the Lloyd algorithm. Papatheodorou et al. [18] studied the area coverage problem for mobile robot networks with imprecise localization. Each robot has a uniform radial sensing capability and is subject to first-order kinematic constraints. This method partitions convex space based on the Voronoi principle, where each robot’s responsibility area corresponds to its Voronoi cell, with boundaries formed by hyperbolic arcs. The proposed control law is distributed and relies on the position information of its Voronoi-Delaunay “neighbors”, inherently ensuring collision avoidance. Yu et al. [19] considered uncertain kinematics and dynamics. By adopting a cost function related to the Voronoi partition, the optimal coverage control was transformed into a tracking control problem, and an adaptive dynamic and kinematic programming approach was used to design an adaptive region-optimal coverage control strategy. Magalhães et al. [20] introduced a control strategy based on Voronoi partitioning for multi-robot systems covering an environment with state and control constraints. At each moment, the Voronoi algorithm partitions the finite convex workspace, assigning each robot non-overlapping navigation areas. The partitioning is adjusted by assigning different weights to each robot, ensuring that robots with higher weights are assigned larger areas. This deployment scheme uses model predictive control and artificial vector fields to guide robots along curves toward the Chebyshev configuration centers of the Voronoi partition, ultimately stabilizing the system and achieving optimal area coverage.

2.2. Approximate Cell Decomposition Method

The approximate cell decomposition method typically achieves effective coverage of an area by dividing the task region into cells of consistent shape and size. However, due to the presence of obstacles or boundary constraints within the task area, some cells may be discarded, meaning that the union of these cells does not fully cover the entire task area. During the partitioning process, the size of the cells is primarily determined by the detection range of the sensors equipped on the agents performing the task. Methods such as spanning tree algorithms, neural networks, and bio-inspired algorithms are typical examples of approximate cell decomposition techniques. Building on research in single-agent systems, the application of this method in multi-agent systems has also received considerable attention.

In the study of coverage in multi-agent systems, Hazon et al. [21] extended the spanning tree method to multi-robot systems using a heuristic approach, proposing the MSTC (Multi-robot Spanning Trees Coverage) method (as shown in Figure 4a). Zkahraman et al. [22] investigated the multi-objective coordination problem in robot task execution, presenting a method that combines control barrier functions with behavior trees to achieve different types of task combinations, while providing performance guarantees for task completion. This method was validated in a simulated underwater coverage task. Hyatt et al. [23] explored the problem of multi-robot online coverage path planning, which involves simultaneously performing coverage and mapping tasks in an unknown environment. They proposed an MCTS (Monte Carlo Tree Search) method based on game theory and machine learning, applying it to multi-robot online coverage path planning. The study implemented the MCTS planner and compared it with an online multi-robot planning method based on the cow-plowing algorithm under different conditions of robot numbers and obstacle densities, while also exploring the adaptability of the MCTS planner for multi-objective tasks such as minimizing turns.

Dong et al. [24] proposed an artificial weighted spanning tree covering algorithm based on the MSTC method. This approach allows each robot to independently construct a spanning tree, gravitates toward the inertia center of uncovered areas, and avoids other robots, thus balancing task distribution. It utilizes Bézier curves to optimize trajectory smoothness, improving real-time performance and dynamic consistency. This method is suitable for distributed multi-robot systems, improving coverage efficiency and enhancing robustness in patrol tasks. Xu et al. [25] proposed a complete coverage path planning algorithm for irregular areas. This method adopts adaptive pruning to optimize local paths, reduce redundancy in coverage, and decrease path complexity and theoretical energy consumption in areas with obstacles. Kim et al. [26] introduced a coverage control strategy for aerial robot formations equipped with downward-facing cameras. Based on the characteristic that camera resolution varies with height, a distributed gradient controller was designed to balance the monitoring area and perception quality, optimizing the target detection probability. Additionally, control barrier functions were used to ensure obstacle avoidance and maintain safe distances during deployment. Chen et al. [27] proposed a fast task planning framework, termed “planning decision trees”, suitable for large-scale multi-robot systems subject to temporal logic constraints. The planning decision tree incrementally constructs a tree structure to represent task progress, modeling the system state in terms of completed positions and times, thereby avoiding complex computations and reducing the search space. This method expands from the root node to the leaf nodes to meet linear temporal logic tasks, ensuring both the feasibility and completeness of the planning. Analysis shows that the solution time of the planning decision tree is linearly related to the number of robots, making it suitable for efficient task planning and offering new research ideas for autonomous coverage and patrol in multi-agent systems. Lu et al. [28] studied the path planning problem of reconfigurable modular robot systems and proposed a hybrid density network method based on obstacle perception to generate connected polygonal nodes. These nodes are then connected using convex polygon trees within a rapidly exploring random tree framework, forming candidate solutions for convex optimization problems. This method ensures connectivity, optimizes computational complexity, and improves planning efficiency in large-scale, complex environments.

In addition to the exact-approximate cell decomposition method and approximate cell decomposition method, several other approaches have also been applied to the field of multi-agent cooperative coverage control. Zhu et al. [29] and Gosrich et al. [30] proposed two coverage control methods for multi-agent systems based on graph neural networks. These methods aggregate local perceptual information from “neighbors” at different hops through the graph neural network, promoting cooperation among robots. The effectiveness of multi-communication in multi-robot coverage was verified, and the scalability and transferability of the learning-based control strategy were evaluated. Danie et al. [31] applied the ant colony optimization algorithm to optimize key parameters in a chaotic ant colony optimization mobility model. This model uses chaotic solutions based on dynamical systems and pheromones to optimize area coverage. Subsequently, genetic algorithms and coevolutionary genetic algorithms were employed to optimize the model parameters, enhancing the system’s operational capabilities. Xin et al. [32] designed a multi-robot cooperative area coverage algorithm, proposing a set of rules to divide the target coverage area into several small regions, and developed a genetic algorithm based on circular encoding to allocate these divided regions to multiple robots. Finally, a zigzag method was used to design the robots’ coverage paths to ensure they cover the assigned areas. Nemer et al. [33] conducted research on the cooperative patrolling tasks of multi-UAV systems, analyzing the flight altitude and characteristics of UAVs using game theory to determine the coverage probability of each node and define a reward function based on this. Through the construction of an evaluation function, the actions of each node and their resulting Nash equilibrium returns were studied. Simulation results show that this method effectively covers the task area, demonstrates strong robustness in responding to sudden failures, and outperforms traditional cooperative visual area coverage methods in terms of coverage rate and computation time.

Although the above methods have made some progress in improving the computational efficiency, adaptability, and task effectiveness of multi-agent systems in collaborative patrolling tasks, there are still some shortcomings in their application. These shortcomings are specifically reflected in the following aspects:

(1) Safety Defects in Motion Patterns: Coverage control algorithms based on deterministic path planning tend to form periodic motion patterns for the agents. The fixed trajectory parameters generated by this pattern (including period length, path shape, and phase characteristics) can be accurately predicted, thus enabling the implementation of targeted countermeasures. This deterministic motion not only reduces the survival probability of agents in dynamic adversarial environments but also causes a cascading decline in task effectiveness.

(2) Lack of Dynamic Coverage Mechanisms: Traditional partition-based coverage control methods are based on the assumption of static optimal position deployment and fail to fully consider the physical constraints of agent platforms. Specifically, in real-world scenarios where perceptual resources are limited (such as finite field of view or intermittent sensor failures) or dynamic mobility capabilities are constrained, agents are forced to adopt periodic back-and-forth movement patterns for area coverage. This strategy not only leads to suboptimal migration path planning between units but also struggles to adapt to real-time changes in the dynamic optimal set of positions, resulting in redundant movement energy consumption.

(3) Insufficient System Robustness: Existing coverage control algorithms (particularly those based on Voronoi diagram spatial partitioning methods) lack a significant node failure tolerance mechanism. This flaw stems from insufficient global situational awareness in distributed architectures and a lack of effective dynamic task reassignment algorithms.

These issues fundamentally reflect three core contradictions in current coverage control methods: challenges in survival in dynamic adversarial environments, the difficulty of sustained coverage under limited resource constraints, and the lack of reliability assurance mechanisms. Therefore, the subsequent chapters of this paper will focus on research from three dimensions: heterogeneous information fusion modeling, dynamic situational awareness reinforcement, and distributed decision optimization, to address the above problems.

3. System Models

In order to accurately and reasonably describe the task situation of the Multi-USV system in patrol missions, this section constructs a hybrid embedded task state model based on information graphs. The model consists of three types of information graphs: the patrol information graph, the cooperative information graph, and the obstacle information graph. Each of these information graphs deeply integrates both the feature information of individual nodes and the feature information of the global task. Through this approach, the organic integration of micro-level information and macro-level information is successfully achieved. The nodes in the system continuously update the model based on real-time patrol results, thereby obtaining the corresponding task situation information. Based on this task situation information, the system trains the nodes to generate corresponding task strategies. This method provides a theoretical foundation for decision-making in Multi-USV systems in complex environments, helping to improve the system’s adaptability and effectiveness.

3.1. Patrol Information Graph

Assume a multi-USV system with $N_U$ nodes is performing a cooperative patrol mission within a rectangular task area $\mathit{A}$ of size $L_M\times L_N$.Given that the nodes are sailing on the water surface, the area $\mathit{A}$ can be simplified into a two-dimensional plane, and the nodes are projected onto $\mathit{A}$ for analysis. When constructing the patrol information graph, the first step is to rasterize the area $\mathit{A}$. The size of each grid $L_x\times L_y$ should be determined based on the detection range of the sensors carried by the nodes. To simplify the analysis, the area measurable by the sensor within a unit of time is approximated as a circular region with a detection radius of $R_s$. Areas outside the detection radius cannot be surveyed for the required task information. After rasterizing area $\mathit{A}$, the task region $g$ to be covered is represented by the center coordinates $m,n$ of each grid:

$$g=\left(m,n\right),m\in \left\{\mathrm{1,2},3\cdots ,{\displaystyle \frac{L_M}{L_x}}\right\},n\in \left\{\mathrm{1,2},3\cdots ,{\displaystyle \frac{L_N}{L_y}}\right\}$$

(1)

The patrol information value obtained by a USV ($US{V}_i$) at time $k$ is represented by ${\lambda }_{i,g}\left(k\right)$. A downward-facing RGB-D camera (FoV 90^° × 60^°, resolution 1280 × 720) captures water surface features. The coverage state ${\lambda }_{i,g}\left(k\right)$ is determined by whether the grid $g$ is within the camera’s frustum and satisfies texture recognition criteria. When ${\lambda }_{i,g}\left(k\right)=1$, it indicates that the patrol information for region $g$ has been acquired by $US{V}_i$ at time $k$. When ${\lambda }_{i,g}\left(k\right)=0$, it means that the patrol information for region $g$ has not yet been acquired by $US{V}_i$ at time $k$. Therefore, as shown in Figure 5, the patrol information acquired by $US{V}_i$ at time $k$ is represented as $E_i\left(k\right)$:

$$E_i\left(k\right)=\left\{{\lambda }_{i,g}\left(k\right)\left|g\in A\right.\right\}$$

(2)

Due to the limitations of the sensor’s perception range, a single node can only acquire partial patrol information for the task area. However, by establishing an appropriate information exchange mechanism, a single node can obtain the patrol information of other nodes, thereby enabling it to gain a more comprehensive understanding of the task situation. Communication between nodes forms the foundation for establishing an information fusion mechanism. Similar to the node’s perception capability, the communication capability of a node is also constrained by both the device and the environment, which means that it can only communicate with other nodes within a specific range. Other nodes located within the communication radius $R_c$ of $US{V}_i$ at time $k$ are defined as $US{V}_i$’s “neighbors”, denoted as $N{e}_i\left(k\right)$:

$${\rm N}{e}_i\left(k\right)=\left\{j|\left\{i,j\right\}\in {\mu }_i\left(k\right)-{\mu }_j\left(k\right)\le R_c\right\}\cup \left\{i\right\}$$

(3)

${\mu }_i\left(k\right)$ and ${\mu }_j\left(k\right)$ represent the positions of $US{V}_i$ and $US{V}_j$, respectively, in the task area at time $k$. At time $k$,$US{V}_i$ patrols and acquires $E_i\left(k\right)$. It then sends $E_i\left(k\right)$ to other neighbors while simultaneously receiving the patrol information from them. $US{V}_i$ merges and updates $E_i\left(k\right)$ with the patrol information obtained from other neighbors according to the following rules: $US{V}_i$ updates the corresponding positions in $E_i\left(k\right)$ where it has not yet acquired patrol information, but other neighbors have. Specifically, $US{V}_i$ updates the patrol information value at those positions from 0 to 1. If $US{V}_i$ has already acquired the patrol information at a given position, no update is made for that position. The updated $E_i\left(k\right)$ is denoted as ${\tilde{E}}_i\left(k\right)$.

3.2. Cooperative Information Graph

As indicated by the previous description, the collaboration between USVs is a critical system characteristic of multi-USV systems, and it represents the greatest advantage of such systems over single USV in practical missions. In the cooperative patrolling task of multi-USV systems, to fully leverage the system’s cooperative features, the nodes in the system, when making task decisions, need to not only maximize their own operational efficiency but also consider the operational information of other nodes (referred to as teammates) within the system as a key reference for decision-making. This approach ensures that the generated algorithm can maximize the overall operational effectiveness of the system.

To address this, a cooperative information graph is introduced in the patrol situation model. This information graph is constructed based on the Long Short-Term Memory (LSTM) artificial neural network model. With the aid of this model, the position of other nodes at the next time step can be predicted based on the information acquired from these nodes, thereby generating the corresponding cooperative information graph. By utilizing the cooperative information graph, nodes can fully consider the operational status of their teammates at the next time step when generating control commands. This approach ensures a rational distribution of tasks among the nodes in the system, avoids conflicts, fosters collaboration between them, and maximizes the system’s task capability. Additionally, it prevents redundant patrolling of the same area by multiple nodes during the mission, thereby improving the operational efficiency of the system.

In the node position dynamic prediction model, the input consists of the dynamic motion information of teammates, which exhibits a clear sequential characteristic. As a result, statistical prediction models are not suitable for this type of input. The LSTM model, however, addresses this issue by incorporating gate structures, including the input gate, forget gate, and output gate, to regulate the flow of information. This mechanism determines which information should be retained and which should be discarded, effectively mitigating the limitations of traditional RNN networks. In the teammate state prediction model, the historical data of teammates’ states are used as input data. The input sequence data is represented as: $X=\left(X_1,X_2,\dots ,X_T\right)$, where $X_T=\left(x_t^1,x_t^2,\dots ,x_t^L\right)\in X$, and $L$ represents the time step, meaning that the teammate’s next state is predicted based on the teammate’s historical state over the past $L$ time steps.The hidden state of the LSTM memory unit is denoted as $H=\left(h_1,h_2,\dots ,h_T\right)$, and the output time series is represented as $\widehat{Y}=\left({\widehat{y}}_1,{\widehat{y}}_2,\dots ,{\widehat{y}}_T\right)$, where ${\widehat{y}}_t=h_T$ serves as the predicted value of the teammate’s next state. The weight of the input weighting layer is defined as:

$$W=\left(w_1,w_2,\dots ,w_L\right)$$

(4)

After importance weighting through the input weighting layer, the input data to the LSTM module can be represented by the Equations (4) and (5) as shown below:

$$\tilde{X}=\left({\tilde{x}}_1,{\tilde{x}}_2,\dots ,{\tilde{x}}_T\right)$$

(5)

$$\widetilde{X_t}=\left(w_1{x}_t^1,w_2{x}_t^2,\dots ,w_L{x}_t^L\right)$$

(6)

The nonlinear mapping function of the memory unit in the LSTM is shown as follows:

$$f_t=\sigma \left(W_f\cdot \left[h_{t-1},{\tilde{X}}_t\right]+b_f\right)$$

(7)

$$i_t=\sigma \left(W_i\cdot \left[h_{t-1},{\tilde{X}}_t\right]+b_i\right)$$

(8)

$$c_t=f_t\cdot c_{t-1}+i_t\cdot \tanh \left(W_c\cdot \left[h_{t-1},{\tilde{X}}_t\right]+b_c\right)$$

(9)

$$o_t=\sigma \left(W_o\cdot \left[h_{t-1},{\tilde{X}}_t\right]+b_o\right)$$

(10)

where $W_f$,$W_i$,$W_c$,$W_o$ are the weight matrices for the forget gate, $b_f$,$b_i$,$b_c$,$b_o$ are the corresponding bias vectors, $\sigma \left(\cdot \right)$ is the sigmoid activation function, $f_t$ is the forget gate state, $i_t$ is the input gate state, $c_t$ is the cell state, $o_t$ is the output gate state, and $h_t$ represents the hidden layer output. Based on the node position prediction model, $US{V}_i$ is capable of obtaining the predicted position of neighboring nodes within its communication radius at time k for the next time step. The predicted position information of neighboring nodes at the next time step is added to the grid map of size $A$ to form the cooperative information graph $C_i\left(k\right)$ at time $k$ for $US{V}_i$, as shown in Equation (11):

$$C_i\left(k\right)=\left\{v_{i,g}\left(k\right)\left|g\in A\right.\right\}$$

(11)

When $v_{i,g}\left(k\right)=1$, it indicates that the region $g$ will be occupied by a teammate at the future time step. When $v_{i,g}\left(k\right)=0$, it indicates that the region $g$ will not be occupied by a teammate at the future time step.

3.3. Obstacle Information Map

In real-world task scenarios, nodes are affected by obstacles during their navigation process. These obstacles can be categorized into two types: the first type is dynamic obstacles, such as other ships sailing on the water surface and other navigation nodes in the system; the second type is static obstacles, such as islands, reefs, large buoys, and so on.

It is assumed that during its navigation, $US{V}_i$’s onboard sensors are capable of extracting and returning obstacle information within a detection radius of $R_s$. For static obstacles, which are typically large in volume, it is necessary to process the static obstacle information obtained by the sensors to ensure the safety of the node’s navigation. Each USV integrates a 360° LiDAR (detection radius $R_s=1$00 m, angular resolution 0.1°) and a side-scan sonar (range 50 m, frequency 500 kHz) for obstacle detection.

The specific processing method is as follows: As shown in Figure 6, if the polygon ABCDE represents an obstacle detected by $US{V}_i$, the returned obstacle information will consist of the latitude and longitude coordinates of the vertices of the polygon ABCDE.

After obtaining the latitude and longitude information of the obstacle vertices, the obstacle data provided by the sensor is extracted into binary information in the raster map using the Scan-Line Fill method [32]. As shown in Figure 6, the polygon ABCDE represents the obstacle detected by $US{V}_i$ in region $R_s$ at time $k$. At this point, the Scan-Line Fill method generates multiple equidistant scan lines, which intersect with the edges of the obstacle at several points. For a closed-shaped obstacle, the intersection points between the equidistant scan lines and the obstacle edges always occur in pairs. From left to right, these intersections alternately enter and exit the obstacle, meaning the region between two intersection points corresponds to the area occupied by the obstacle. The corresponding raster in this region is marked as the obstacle grid. Based on the positional information of these 12 intersection points, the corresponding obstacle information can be formed in the raster map (with the raster information at the intersection points set to 1). For dynamic obstacles, which have a smaller volume, they are treated as nodes for analysis and processing during the obstacle avoidance process. Both dynamic and static obstacles are added to the grid map of size $2R_s\times 2R_s$, resulting in the obstacle information map $O{b}_i\left(k\right)$ for $US{V}_i$:

$$O{b}_i\left(k\right)=\left\{o_{i,g}\left(k\right)\left|g\in A\right.\right\}$$

(12)

When $o_{i,g}\left(k\right)=1$, it indicates that a dynamic obstacle exists within $US{V}_i$’s node safe navigation distance threshold $R_{acollision}$ at time $k$. When $o_{i,g}\left(k\right)=2$, it indicates that a static obstacle exists within $US{V}_i$’s obstacle safe navigation distance threshold $R_{ocollision}$ at time $k$. When $o_{i,g}\left(k\right)=0$, it indicates that no obstacle exists within both $R_{acollision}$ and $R_{ocollision}$ at time $k$ for $US{V}_i$.

4. Proposed Strategy

The complex maritime environment and specific mission context place higher demands on USVs, requiring faster response times to quickly generate corresponding control commands after obtaining situational information, thereby enhancing the system’s real-time control performance. Based on this, this paper proposes a cooperative patrol control strategy based on hybrid embedded task state information, using an improved OODA model control architecture. The strategy combines time-triggered and event-triggered task state information to achieve a more efficient and cooperative patrol control method. By adopting the improved OODA model control architecture, the traditional non-end-to-end decision-making process is transformed into an end-to-end decision-making process. Through reward reshaping and the rational design and adjustment of reward signals, the strategy significantly improves the learning efficiency of the cluster nodes, making the trained strategy particularly suitable for the highly dynamic and real-time response requirements of multi-USV cooperative patrol missions. This section will provide a detailed analysis of the strategy.

4.1. Action State Output of Unmanned Surface Vehicles Based on the Dynamic Window

For an agent, the policy refers to the mapping relationship from state to action. Therefore, the node needs to have the capability to provide accurate and comprehensive mission situational information. Based on the previously constructed mission state information model, the patrol information graph ${\tilde{E}}_i\left(k\right)$ collaboration information graph $C_i\left(k\right)$, and obstacle information graph $O{b}_i\left(k\right)$ obtained by $US{V}_i$ at time $k$ are considered as the mission situational information, or state value $S_i\left(k\right)$:

$$S_i\left(k\right)=\left\{{\tilde{E}}_i\left(k\right),C_i\left(k\right),O{b}_i\left(k\right)\right\}$$

(13)

The motion space of the agent is closely related to the agent’s dynamics and kinematics characteristics. The action $a_i\left(k\right)$ of $US{V}_i$ at time $k$ is expressed as follows:

$$a_i\left(k\right)=\left\{{\tau }_{i,u}\left(k\right),{\tau }_{i,r}\left(k\right)\right\}$$

(14)

In Equations (13) and (14),$S_i\left(k\right)$ serves as the input to policy network, $a_i\left(k\right)$ represents the control output of $US{V}_i$ at time $k$, where ${\tau }_{i,u}\left(k\right)$ and ${\tau }_{i,r}\left(k\right)$ are two continuous variables representing the forward thrust and yaw moment of $US{V}_i$ at time $k$, respectively. Based on the kinematic and dynamic model of the USV established, the motion state information of $US{V}_i$ at time $k$ can be calculated after applying the corresponding $a_i\left(k\right)$. However, due to the limitations of the USV’s dynamic performance, there are certain velocity vectors that the node cannot achieve in the current motion state, regardless of the control input. Therefore, considering the dynamic performance of the USV, the dynamic window algorithm is used to derive the achievable movement speed of the USV within a given time window, referred to as the velocity window $v_{window}$:

$$v_{window}=\left\{v|v\in \left[v_c-\dot{v}{\Delta }_k,v_c+\dot{v}{\Delta }_k\right]\right\}$$

(15)

and the achievable angular velocity, referred to as the angular velocity window ${\omega }_d$:

$${\omega }_{window}=\left\{\omega |\omega \in \left[{\omega }_c-\dot{\omega }{\Delta }_k,{\omega }_c+\dot{\omega }{\Delta }_k\right]\right\}$$

(16)

In Equation (15), $v_c$ is the current movement speed of USV. $\dot{v}$ is the acceleration of the USV, and ${\Delta }_k$ is the duration of the control input.

In Equation (16), ${\omega }_c$ is the current angular velocity of the USV, and $\dot{\omega }$ is the angular acceleration of the USV. Based on Equation (16), the achievable heading angle range of the USV within time ${\Delta }_k$ can be calculated:

$${\theta }_{window}=\left\{\theta |\theta \in \left[{\theta }_h+{\omega }_c{\Delta }_k-{\displaystyle \frac{1}{2}}\dot{\omega }{\Delta }_k^2,{\theta }_h+{\omega }_c{\Delta }_k+{\displaystyle \frac{1}{2}}\dot{\omega }{\Delta }_k^2\right]\right\}$$

(17)

${\theta }_h$ is the current heading of USV. Using Equations (14) and (16), the values of $v_{window}$ and ${\theta }_{window}$ can be determined. However, the elements in the sets of $v_{window}$ and ${\theta }_{window}$ are continuous, which makes engineering calculations difficult. Therefore, as shown in Figure 7, $v_{window}$ is discretized into $M$ speeds, and ${\theta }_{window}$ is discretized into $N$ heading directions. Each discrete speed $v_{discrete}$ and heading ${\theta }_{discrete}$ form a set of velocity vectors $\left(v_{discrete},{\theta }_{discrete}\right)$, and these velocity vectors are referred to as the achievable velocities (RV).

When $US{V}_i$ takes the corresponding action $a_i\left(k\right)$ at time $k$, the velocity vector corresponding to the current node’s motion state is calculated. Then, the closest velocity vector from the RV (reachable velocities) is selected as the velocity vector of $US{V}_i$ after adopting $a_i\left(k\right)$. Based on this chosen velocity vector, the dynamics information after $US{V}_i$ adopts $a_i\left(k\right)$ is computed.

4.2. Reward Function Design

The appropriate reward values not only enable nodes to learn the optimal task strategy based on task situation information, but also effectively prevent nodes from encountering issues such as non-convergence or getting stuck in local optima during the learning process. Before designing the reward values for the nodes, it is essential to first clarify the objectives of the task being executed by the system. The objectives of a multi-unmanned vessel system performing a cooperative patrol task are as follows:

• The system should be able to acquire patrol information of the entire task area.
• Under the condition of acquiring task data for the entire task area, patrol efficiency should be maximized and task costs minimized.
• Ensure the safety of all nodes during navigation, avoiding collisions between nodes and obstacles.

Based on the task environment settings mentioned above, in traditional strategy training, nodes can only receive the final reward $r^{cover}$ after completing full coverage of the task area, and they receive an efficiency reward $r^{effiency}$ according to the completion status. However, these rewards are only given at the end of the task, and the reward structure often exhibits high sparsity, meaning that clear feedback is only provided when the task is fully completed. This delayed feedback mechanism has a significant negative impact on the learning process, leading to the problem of reward sparsity. The core of this issue lies in the fact that during the learning process, agents struggle to establish clear causal relationships and cannot promptly associate specific actions or states with the eventual success of the task. The challenges posed by sparse rewards are multifaceted.

Firstly, it severely hampers the agent’s effective exploration of the environment. In the absence of timely feedback, the agent struggles to determine which actions are beneficial and which are ineffective or harmful. This forces the agent to engage in extensive random exploration, hoping to discover valuable state-action sequences through chance attempts. However, this approach is not only time-consuming but also computationally expensive, especially in tasks with large state spaces or complex action spaces, such as a water patrol task.

Secondly, the issue of sparse rewards can lead to the learning process getting trapped in local optima. Due to the lack of intermediate feedback, the agent may prematurely converge on strategies that seem effective but are not globally optimal. In this case, the agent struggles to break free from its current behavior patterns to explore potentially better solutions, as it lacks sufficient motivation and guidance to try new, higher-risk but higher-reward behaviors.

To address these issues, this paper combines reward reshaping techniques. The core idea of this method is to cleverly design and introduce a series of intermediate reward signals on top of the original sparse rewards. These additional rewards serve as immediate feedback during the task, providing more granular and information-rich guidance for the learning process. Reward reshaping techniques offer more dense and informative feedback by incorporating extra intermediate rewards based on the original reward signals. This approach can significantly accelerate the learning process, helping the agent discover valuable behavioral patterns more quickly. Through the carefully designed reward function, we can integrate domain knowledge or prior information into the reward structure, guiding the agent to explore in the desired direction. This not only improves learning efficiency but also enhances the quality of the final policy.

By combining reward reshaping, we can break down complex long-term goals into a series of more easily learnable short-term sub-goals. This decomposition accelerates the learning process and helps the agent quickly identify and reinforce valuable behaviors. Therefore, by introducing additional reward functions to “shape” the agent’s learning path, the agent receives more feedback during exploration, thus accelerating the learning process. This paper reconstructs the reward structure and assigns corresponding rewards based on the state value obtained by the node. The rewards are divided into patrol reward, cooperative reward, obstacle avoidance reward, and efficiency reward.

4.2.1. Patrol Reward

This section of the reward represents the dense immediate rewards introduced in the reward mapping. The traditional reward $r^{cover}$ is transformed into a coverage reward $r^{cover}\left(k\right)$ that can be obtained at each time step. The strategy assigns patrol rewards to the nodes based on their patrol status in the task area. The reward value is mainly determined by the node’s information collection in the task area. The patrol reward consists of two main components: coverage reward and completion reward. The coverage reward obtained by node $US{V}_i$ at time step $k$ is represented as $r_i^{cover}\left(k\right)$:

$$r_i^{cover}\left(k\right)=\sum _{g\in {\tilde{E}}_i\left(k\right)}{\lambda }_{i,g}\left(k\right) if {\lambda }_{i,g}\left(k-1\right)=0\wedge {\lambda }_{i,g}\left(k\right)=1$$

(18)

The equation indicates that the algorithm will reward the node for acquiring new patrol information in the task area. In other words, when the node’s actions lead to the system obtaining more patrol information, the node will receive the corresponding reward.

When the entire task area is covered, $US{V}_i$ will be given a completion reward $r_i^{finish}\left(k\right)$:

$$r_i^{finish}\left(k\right)={\prod }_{g\in {\tilde{E}}_i\left(k\right)}{\lambda }_{i,g}\left(k\right)$$

(19)

Finally, the patrol reward for $US{V}_i$ at time step $k$ is given by the Equation (20) as follows:

$$r_i^{exploration}\left(k\right)={\beta }_{cover}{r}_i^{cover}\left(k\right)+{\beta }_{finish}{r}_i^{finish}\left(k\right)$$

(20)

where,${\beta }_{cover}$ is the weighting coefficient for the coverage reward, prioritizing exploration of new areas. A higher value encourages nodes to focus on uncovering unexplored regions.${\beta }_{finish}$ is the weighting coefficient for the completion reward, emphasizing full coverage of the task areas. A higher value incentivizes nodes to collectively achieve complete surveillance.

4.2.2. Cooperative Reward

This section of the reward represents the intensive, immediate rewards introduced in the reward mapping. In order to maximize the patrolling capability of each node within the multi-unmanned vehicle system, and to avoid redundant patrolling of the same location by nodes in the system (before the entire task area is covered, redundant patrolling will lead to a waste of system resources and significantly increase the risk of collisions between nodes, which has a significant negative impact on the operational efficiency of the system), a cooperative reward $r_i^{cooperation}\left(k\right)$ is established for each node. This reward is determined based on the cooperative information map $C_i\left(k\right)$ obtained by $US{V}_i$:

After taking the corresponding action $a_i\left(k\right)$ at time $k$, $US{V}_i$ obtains the cooperative information map $C_i\left(k+1\right)$ at time $k+1$. First, the task areas within $C_i\left(k+1\right)$ that have a value of 1 are identified, and the distance $Dis{t}_{ij}\left(k+1\right)$ between these areas and $US{V}_i$ at time $k+1$ is calculated. These distances are then organized into the corresponding distance set:

$$Dis{t}_{cooperation}\left(k+1\right)=\left\{Dis{t}_{i1}\left(k+1\right),Dis{t}_{i2}\left(k+1\right),\dots ,Dis{t}_{ij}\left(k+1\right)\right\}$$

(21)

The calculation of $Dis{t}_{ij}(k+1)$ is shown in Equation (22):

$$Dis{t}_{ij}\left(k+1\right)=\sqrt{{\left(x_i\left(k+1\right)-m\right)}^2+{\left(y_i\left(k+1\right)-n\right)}^2}$$

(22)

where, $\left(x_i\left(k+1\right),y_i\left(k+1\right)\right)$ represents the $x$ and $y$ coordinates of $US{V}_i$’s position at time $k+1$, and $\left(m,n\right)$ refers to the coordinates of the task areas in $C_i\left(k+1\right)$ that satisfy $v_{i,g}\left(k\right)=1$.

$$r_i^{cooperation}\left(k\right)=\left\{\begin{array}{c}\left|{\rm N}{e}_i\left(k\right)\right|,if \forall Dis{t}_{ij}\in \left\{Dis{t}_{cooperation}(k+1)>Dis{t}_{gate}\right\}\\ -{\displaystyle \frac{1-\left(\left(\left|{\rm N}{e}_i\left(k\right)\right|-1\right)/N_U\right)}{e^{T_i(k)}}},else\end{array}\right.$$

(23)

where, $Dis{t}_{gate}$ represents the distance determination threshold, and $T_i\left(k\right)$ denotes the total operation time of $US{V}_i$ up to time $k$.

4.2.3. Obstacle Avoidance Reward

This section of the reward is a dense immediate reward introduced in the reward mapping. In the cooperative patrolling task of a multi-unmanned boat system, obstacle avoidance at the nodes is a critical issue that must be considered. Only by equipping the nodes with effective obstacle avoidance capabilities can the system prevent collisions between nodes and between nodes and obstacles in the task areas, thereby avoiding damage to the nodes and improving the system’s operational efficiency. Therefore, $US{V}_i$ will receive the corresponding obstacle avoidance reward $r_i^{avoid collision}\left(k\right)$ at time $k$, ensuring that the task strategy generated after node training has the necessary obstacle avoidance capability. $r_i^{avoid collision}\left(k\right)$ consists of two parts: the node’s obstacle avoidance reward $r_i^{node collision}\left(k\right)$ and the obstacle’s avoidance reward $r_i^{obstacle collision}\left(k\right)$.

$r_i^{node collision}\left(k\right)$ is designed using the obstacle information map $O{b}_i\left(k\right)$ and based on the Artificial Potential Field (APF) method. Using APF, the node calculates the interaction danger force $F_{ij}\left(k+1\right)$ between each pair of nodes in the system and their corresponding force vectors $F_i\left(k+1\right)$. Then, based on $F_i\left(k+1\right)$, the corresponding action $a_i\left(k\right)$ taken by $US{V}_i$ at time $k$ is determined. The calculation process for $F_{ij}\left(k+1\right)$ is as follows:

After $US{V}_i$ takes the corresponding action $a_i\left(k\right)$ at time $k+1$, it obtains the obstacle information map $O{b}_i\left(k+1\right)$ for time $k+1$. First, the distance $Dis{t}_{ij}\left(k+1\right)$ between the regions with a value of 1 in $O{b}_i\left(k+1\right)$ and $US{V}_i$ at time $k+1$ is calculated, and the corresponding distance set is formed as follows:

$$Dis{t}_{acollision}(k+1)=\left\{Dis{t}_{i1}(k+1),Dis{t}_{i2}(k+1),\dots ,Dis{t}_{ij}(k+1)\right\}$$

(24)

If there are no regions with a value of 1 in $O{b}_i\left(k+1\right)$ then $Dis{t}_{acollision}(k+1)=\left\{\mathrm{\varnothing }\right\}$. The calculation method for $Dis{t}_{ij}(k+1)$ is shown in Equation (25):

$$Dis{t}_{ij}\left(k+1\right)=\sqrt{{\left(x_i\left(k+1\right)-m\right)}^2+{\left(y_i\left(k+1\right)-n\right)}^2}$$

(25)

where,$\left(x_i\left(k+1\right),y_i\left(k+1\right)\right)$ represents the $x$ and $y$ coordinates of $US{V}_i$ at time $k+1$, and $\left(m,n\right)$ refers to the coordinates of the region in $O{b}_i\left(k+1\right)$ that satisfy Equation (26):

$$o_{i,g}\left(k+1\right)=1\wedge o_{i,g}\left(k\right)=1$$

(26)

Therefore, the $F_{ij}\left(k+1\right)$ for $US{V}_j$ is calculated using Equation (27):

$$F_{ij}\left(k+1\right)=\left\{\begin{array}{c}\delta e^{-\alpha Dis{t}_{ij}\left(k+1\right)},\exists Dis{t}_{ij}\left(k+1\right)\in \left\{Dis{t}_{acollision}(k+1)<Dis{t}_{safety}\right\}\\ 0, \forall Dis{t}_{ij}\left(k+1\right)\in \left\{Dis{t}_{acollision}(k+1)>Dis{t}_{safety}\vee Dis{t}_{acollision}(k+1)=\left\{\mathrm{\varnothing }\right\}\right\}\end{array}\right.$$

(27)

where, $\delta$ and $\alpha$ are adjustable parameters. In order to provide sufficient obstacle avoidance reaction time for nodes in relation to other nodes in the region and ensure the safety of the node’s navigation, a “dilation” process is applied. The dilated node becomes a circular region with a radius $r_{USV}$. During training, “pseudo” collisions can be penalized to provide the node with more obstacle avoidance reaction time, thereby enhancing the obstacle avoidance capability between nodes. Based on the above reasons, the safety navigation distance threshold between nodes is:

$$Dis{t}_{safety}=2r_{USV}$$

(28)

$F_i\left(k+1\right)$ is the resultant force vector generated by all the nodes within $O{b}_i\left(k+1\right)$ that meet the conditions, expressed as:

$$F_i\left(k+1\right)={\sum }_{i\ne j}{F}_{ij}\left(k+1\right)$$

(29)

Finally, based on $F_i\left(k+1\right)$, $r_i^{node collision}\left(k\right)$ is calculated using Equation (30):

$$r_i^{node collision}\left(k\right)=\left\{\begin{array}{c}-e^{{\delta }_{dangerous}\left|F_i(k+1)\right|},F_i(k+1)\ne 0,\\ \left|{Na}_i(k)\right|,F_i(k+1)=0.\end{array}\right.$$

(30)

where, ${\delta }_{dangerous}$ is the danger force cost parameter, where ${\delta }_{dangerous}=0.6$. $N{a}_i\left(k\right)$ is the set of regions within $O{b}_i\left(k\right)$ where the value is 1.

$r_i^{obstacle collision}\left(k\right)$ is similarly designed using the information from the obstacle map $O{b}_i\left(k\right)$. First, the distance $O{b}_i\left(k+1\right)$ between the task area with a value of 2 in $O{b}_i\left(k+1\right)$ and $US{V}_i$ is calculated, forming the corresponding distance set:

$$Dis{t}_{ocollision}(k+1)=\left\{Dis{t}_{i1}(k+1),Dis{t}_{i2}(k+1),\dots ,Dis{t}_{ij}(k+1)\right\}$$

(31)

If there are no regions with a value of 2 in $O{b}_i\left(k+1\right)$, then $Dis{t}_{ocollision}(k+1)=\left\{\mathrm{\varnothing }\right\}$. The calculation of $Dis{t}_{ij}(k+1)$ is shown in Equation (32):

$$Dis{t}_{ij}\left(k+1\right)=\sqrt{{\left(x_i\left(k+1\right)-m\right)}^2+{\left(y_i\left(k+1\right)-n\right)}^2}$$

(32)

where, $\left(x_i\left(k+1\right),y_i\left(k+1\right)\right)$ represents the coordinates of $US{V}_i$ at time $k+1$. $\left(m,n\right)$ are the coordinates of the task region in $O{b}_i\left(k+1\right)$ where $o_{i,g}\left(k+1\right)=2\wedge o_{i,g}\left(k\right)=2$. The calculation process for $r_i^{area collision}\left(k\right)$ is shown in Equation (33):

$$r_i^{obstacle collision}\left(k\right)=\left\{\begin{array}{c}-\left(r_{size}-\mathit{min} Dis{t}_{ij}\left(k+1\right)\right) if \exists \left\{Dis{t}_{ij}\left(k+1\right)\in Dis{t}_{ocollision}\left(k+1\right)<r_{size}\right\}\\ \left|N{o}_i\left(k\right)\right|, \forall Dis{t}_{ij}\left(k+1\right)\in \left\{Dis{t}_{ocollision}\left(k+1\right)>r_{size}\vee Dis{t}_{ocollision}\left(k+1\right)=\left\{\mathrm{\varnothing }\right\}\right\}\end{array}\right.$$

(33)

$N{o}_i\left(k\right)$ is the set of regions in $O{b}_i\left(k\right)$ where the value is 2. Similarly, in order to provide sufficient time for the nodes to react to the obstacles in the region and improve the obstacle avoidance effectiveness, the detected static obstacles are “inflated”, which means the obstacle region’s range is expanded. The inflated obstacle becomes a circular region with a radius $r_{area}$. Therefore, when the following condition is met:

$$Dis{t}_{ij}(k+1)<r_{size}$$

(34)

The node is considered to have “collided” with the obstacle region when the condition is met. At this point, the node will receive a corresponding negative reward, where:

$$r_{size}=r_{area}+r_{USV}$$

(35)

The final $r_i^{avoid collision}\left(k\right)$ is shown in Equation (36):

$$r_i^{avoid collision}(k)={\beta }^{node collision}{r}_i^{node e collision}(k)+{\beta }^{area collision}{r}_i^{area collision}(k)$$

(36)

4.2.4. Efficiency Reward

This section of the reward is the sparse terminal reward introduced in the reward mapping. As previously mentioned, one of the main objectives of the task strategy generated by the node is to maximize work efficiency while ensuring full coverage of the task area, thereby reducing the task cost. To achieve this, an appropriate efficiency reward value $r_i^{efficiency}\left(k\right)$ is designed for the node.

$r_i^{efficiency}\left(k\right)$ is used to evaluate the efficiency of the task strategy generated by the node from a time perspective. It is known that, under the same task conditions, the time required for the node to complete a patrol from the start to the stop (i.e., when the system finishes one episode) is called the patrol time $k_{i,episode}$ for $US{V}_i$. The shorter the $k_{i,episode}$ in the system, the higher the efficiency of the patrol strategy used by $US{V}_i$. Therefore, the efficiency reward value received during $US{V}_i$’s learning process will be higher. Conversely, if the patrol strategy used by $US{V}_i$ is less efficient, the efficiency reward value received by $US{V}_i$ will be lower. The efficiency reward $r_i^{efficiency}\left(k\right)$ obtained by $US{V}_i$ at time $k$ is calculated by Equation (37):

$$\begin{array}{l}{r}_i^{efficiency}\left(k\right)\\ =\left\{\begin{array}{c}0,\hspace{1em}{k}_{i,episode}>c_1^T{k}_{Benchmark}\\ {\displaystyle \frac{r_{Benchmark}}{2}}\left[1+cos\left({\displaystyle \frac{\pi \left(T-c_2^T{k}_{Benchmark}\right)}{c_1^T{k}_{Benchmark}-c_2^T{k}_{Benchmark}}}\right)\right]\\ r_{max},k_{i,episode}\le c_1^T{k}_{Benchmark}\end{array}\right.,c_2^T{k}_{Benchmark}<k_{i,episode}\le c_1^T{k}_{Benchmark}\end{array}$$

(37)

In Equation (37), $r_{Benchmark}$ is the baseline efficiency reward for the node when executing the task, and $c_1^T$ and $c_2^T$ are correction factors. $k_{Benchmark}$ is the baseline time for the node to complete the task, which is calculated as follows:

$$k_{Benchmark}=min\left\{{\displaystyle \frac{\left(L_y-1\right)L_M{L}_x}{L_y\left|v_{max}\right|}}+{\displaystyle \frac{\left(L_x-1\right)L_N}{L_x\left|v_{max}\right|}},{\displaystyle \frac{\left(L_x-1\right)L_N{L}_y}{L_x\left|v_{max}\right|}}+{\displaystyle \frac{\left(L_y-1\right)L_M}{L_y\left|v_{max}\right|}}\right\}$$

(38)

In Equation (38), $v_{max}$ represents the maximum speed of the node under ideal conditions. Therefore, the reward value $r_i\left(k\right)$ obtained by $US{V}_i$ at time $k$ is given by Equation (39) as follows:

$$r_i\left(k\right)={\beta }^{T}r={\left[\begin{array}{c}{\beta }_{\mathrm{exploration}}\\ {\beta }_{cooperation}\\ {\beta }_{avoid collision}\\ {\beta }_{efficiency}\end{array}\right]}^T\left[\begin{array}{c}\begin{array}{c}\begin{array}{c}{r}_i^{\mathrm{exploration}}\left(k\right)\\ r_i^{cooperation}\left(k\right)\end{array}\\ r_i^{avoid collision}\left(k\right)\end{array}\\ r_i^{efficiency}\left(k\right)\end{array}\right]$$

(39)

From Equation (39), it can be seen that a constant vector is used to weight various types of reward values. This is because the relative magnitude of the reward values generated by different types of reward terms also has a significant impact on the process of the node’s learning task strategy. Therefore, in order to ensure that different reward terms have distinct effects during the training process, ultimately enabling the system to learn a task strategy that meets the task requirements, a constant vector $\beta$ is used to weight various types of reward values.

During the execution of the patrol task, the primary concern is ensuring the navigational safety of the task nodes, so $r_i^{avoid collision}\left(k\right)$ has the largest weight in $r_i\left(k\right)$. Next, to achieve good operational efficiency, $r_i^{efficiency}\left(k\right)$ also carries a relatively large weight in $r_i\left(k\right)$.

In the early stages of the task, the system’s nodes need to patrol the task area in a more dispersed manner to fully utilize each node’s patrol capabilities. As a result, $r_i^{cooperation}\left(k\right)$ carries more weight than $r_i^{exploration}\left(k\right)$ during this phase. In the later stages of the task, the system needs to maximize the patrol rate in the task area, concentrating the system nodes in the unpatrolled regions, so the weight of $r_i^{exploration}\left(k\right)$ becomes larger in this phase. So:

$${\beta }_{\mathrm{exploration}}=0.5+{\displaystyle \frac{1}{e^{T_i(k)}}}$$

(40)

$${\beta }_{cooperation}=0.5-{\displaystyle \frac{1}{e^{T_i(k)}}}$$

(41)

The weight decays with mission time $T_i(k)$. Early in the mission ($T_i(k)\to 0$),${\beta }_{cover}\approx 1.5$ prioritizes exploration. Later ($T_i(k)\to \mathrm{\infty }$), ${\beta }_{cover}\to 0.5$ shifts focus to collaborative convergence. This mechanism enables real-time strategy adaptation based on mission progress.

5. Strategy Training

5.1. Training Process

The Training Framework for Multi USV(TFMUSV) is shown in Figure 8. The specific training process is as follows:

On ${USV}_i$, the interception strategy ${\mu }_i$ and action evaluation function $Q_i$ run independently.${USV}_i$ predicts the next state of its teammate based on historical state data, using the teammate state prediction model, generating augmented state information $s_i$. Following strategy ${\mu }_i$, it selects action $a_i$ and receives the environmental feedback reward signal $r_i$. The joint state information and joint actions of the USV swarm are input into the action evaluation function in the training framework. After calculating the gradient information, the policy network $P$ and action evaluation network $Q$ in the training framework are updated. The objective function corresponding to the action evaluation network $Q$ is:

$$L\left(\theta \right)=\underset{\left(S_i\left(k\right),a_i\left(k\right),\mathrm{ot}{\mathrm{S}}_i\left(k\right)\_a_i\left(k\right),{\mathrm{r}}_i\left(k\right),S_i{\left(k+1\right)}^{\prime }\right)\sim \mathcal{D}}{E}$$

$$[{\left(\begin{array}{c}Q(S_i\left(k\right),a_i\left(k\right),\mathrm{o}\mathrm{t}{S}_i\left(k\right)\_a_i\left(k\right)\left|\theta \right.)-\\ \left(r_i\left(k\right)+Q(S_i\left(k+1\right),a_i\left(k+1\right),\mathrm{ot}{S}_i\left(k+1\right)\_a_i\left(k+1\right)\left|{\theta }^{\prime }\right.)\right)\end{array}\right)}^2]$$

(42)

In Equation (42), $\mathrm{ot}{S}_i\left(k\right)\_a_i\left(k\right)$ represents the states and actions of other nodes in the multi-agent system. For the state and action at the next time step $\mathrm{ot}{S}_i\left(k+1\right)\_a_i\left(k+1\right)$, the state $\mathrm{ot}{S}_i\left(k+1\right)$ comes from the replay buffer, and the corresponding action $\mathrm{ot}{a}_i\left(k+1\right)$ is obtained through the delayed policy network. The gradient update is as shown in Equation (43). For the policy network, the corresponding objective function is:

$$J(\mu )=\underset{\left(S_i\left(k\right),\mathrm{ot}{\mathrm{S}}_i\left(k\right)\_a_i\left(k\right)\right)\sim \mathcal{D}}{E}[Q(S_i\left(k\right),P\left(S_i\left(k\right),{\mathrm{a}}_i\left(k\right)\left|\mu \right.\right),\mathrm{ot}{\mathrm{S}}_i\left(k\right)\_a_i\left(k\right)\left|\theta \right.)]$$

(43)

The gradient calculation method for Equation (43) is shown in Equation (44):

$${𝛻}_{\mu }J\left(\mu \right)=\underset{\left(S_i\left(k\right),\mathrm{ot}{\mathrm{S}}_i\left(k\right)\_a_i\left(k\right)\right)\sim \mathcal{D}}{E}[{𝛻}_{\theta }P\left(S_i\left(k\right),{\mathrm{a}}_i\left(k\right)\left|\mu \right.\right) {𝛻}_{a}Q(S_i\left(k\right),{\mathrm{a}}_i\left(k\right),\mathrm{ot}{\mathrm{S}}_i\left(k\right)\_a_i\left(k\right)\left|\theta \right.)\left|a_i\left(k\right)=P\left(S_i\left(k\right)\left|\mu \right.\right)\right.]$$

(44)

Then, the parameters $\mu$ in the policy network $P$ are updated through gradient ascent in Equation (45):

$$\mu =\mu +\eta {\displaystyle \frac{\partial }{\partial \mu }}J(\mu )$$

(45)

The pseudocode for the training of the multi-USV cooperative training framework proposed in this paper is shown in Algorithm 1, and the multi-USV cooperative training framework is shown in Figure 8.

Algorithm 1 Multi-USV cooperative Patrol Control algorithm

Require: Multi-agent $N_U$, Policy network ${\mu }_i$, Q Network parameters ${\theta }_i,i=1 to N_U$ Replay buffer $\mathcal{D}$ 1: Assign target network parameters ${\mu }^{\prime }\leftarrow \mu ,{\theta }^{\prime }\leftarrow \theta$ 2: for $i=1 to N_U$ do 3:  ${\mathit{\mu }}_{\mathit{i}}^{\prime }\leftarrow {\mathit{\mu }}_{\mathit{i}}$,${\mathit{\theta }}_{\mathit{i}}^{\prime }\leftarrow {\mathit{\theta }}_{\mathit{i}}$ 4: end for 5: while not stop do 6:   for $k=1:max\_episode\_length$ do 7:      Obtain the current environment state $S_i\left(k\right)$, and obtain $a_i\left(k\right)$ based on ${ P}_i$. 8:      $a_i\left(k\right)=P\left(S_i\left(k\right)\left|\mu \right.\right)+\epsilon ,\epsilon \sim N (N$ is noise) 9:      end for 10:    Execute action $\left(a_1\left(k\right),a_2\left(k\right),\dots a_M\left(k\right)\right)$ to obtain reward $\left(r_1\left(k\right),r_2\left(k\right),\dots r_M\left(k\right)\right)$ and add new state values $\left(s_1\left(k+1\right),s_2\left(k+1\right),\dots s_M\left(k+1\right)\right)$, and add them to $\mathcal{D}$ 11:    if $\left(s_1\left(k+1\right),s_2\left(k+1\right),\dots s_M\left(k+1\right)\right)$ is the terminal state, then 12:      Reset the environment 13:     break 14:    end if 15:  end for 16:  Randomly sample $b$ from $D$, and add it to $B$. 17:  for $i=1 to N_U$ do 18:    Calculate the Q-value, $y_i=r_i\left(k\right)+\gamma Q(S_i\left(k+1\right),a_i\left(k+1\right),other{S}_i\left(k+1\right)\left|{\theta }^{\prime }\right.)$ 19:   Update the Q-value network parameters 20:$\hspace{1em}\theta \leftarrow \theta -{𝛻}_{\theta }{\displaystyle \frac{1}{\left|B\right|}}{\sum }_{\left(S_i\left(k\right),a_i\left(k\right),other{S}_i\left(k\right)\_a_i\left(k\right)\right)\in B}{\left(Q\left(S_i\left(k\right),a_i\left(k\right),other{S}_i{\left(k\right)}_{a_i\left(k\right)\left|{\theta }_i\right.}\right)-y_i\right)}^2$ 21:   Update the policy network parameters 22:   $\mu \leftarrow \mu -{𝛻}_{\mu }{\displaystyle \frac{1}{\left|B\right|}}{\sum }_{\left(S_i\left(k\right),other{S}_i\left(k\right)\_a_i\left(k\right)\right)\in B}Q(S_i\left(k\right),P\left(S_i\left(k\right)\left|{\mu }_i\right.\right),other{S}_i\left(k\right)\_a_i\left(k\right)\left|{\theta }_i\right.)$ 23: for $i=1 to N_U$ do 24:    Update the target network parameters: 25:    ${\mu }_i\prime \leftarrow \rho {\mu }_i^{\prime }+\left(1-\rho \right){\mu }_i^{\prime }$ 26:    ${\theta }_i^{\prime }\leftarrow \rho {\theta }_i^{\prime }+\left(1-\rho \right){\theta }_i$ 27:    end for 28: end while

5.2. Training Result

A simulation training environment based on Python 3.12 is built using Google’s TensorFlow 2.15. All training and simulations were conducted on a server equipped with an Intel Xeon Gold 6248R CPU, 256GB RAM, (manufactured by Intel Corporation, headquartered in Santa Clara, CA, USA) and two NVIDIA RTX A6000 GPUs (manufactured by NVIDIA Corporation, headquartered in Santa Clara, CA, USA). The TensorFlow framework leveraged GPU acceleration for parallelized experience replay and batch gradient computation. This training environment is designed to simulate various task scenarios in a multi-unmanned vehicle system cooperative patrol mission. As shown in Figure 9, a training task area of $1000\times 1000 {\mathrm{m}}^2$ is set in the environment. N obstacles are randomly placed within the task area, represented by the rectangular blocks in Figure 9. The performance parameters and dynamics models of the unmanned vehicles in the simulation environment are set and modeled based on the unmanned vehicles used in actual vessel tests. The hyperparameters required during the training process are provided in

Table 1. Training hyperparameters.

Training Hyperparameters	Value
Batch	1024
Experience replay buffer $\mathcal{D}$	106
Discount factor $\gamma$	0.95
Learning rate $\eta$	0.01
$R_s$	100 m
$R_c$	500 m
Max episode	20,000
Total episode	1 × 107

.

Different types of training frameworks were also used to train the system during the training process. The training results of various algorithms are shown in Figure 10. The changes in the reward values obtained by the nodes throughout the training period are presented for the training frameworks of DDPG, Actor-Critic, SAC, PPO, and TRPO. As shown in the results of Figure 10, when the system used the Actor-Critic, SAC, and TRPO frameworks during training, the reward values obtained by the system could not remain in a stable upward state, and the reward fluctuations were always large, failing to converge throughout the training. In particular, with the SAC algorithm, the reward values obtained by the system even decreased during the later stages of training. This indicates that these three training frameworks are not suitable for training the system, as they cannot help the system learn an effective task strategy.

From the analysis of Figure 10, it can be observed that the three training frameworks (DDPG, TFMUSV, and PPO) can help the system achieve higher rewards during the training process compared to the other three frameworks. However, in the later stages of training, although the reward values tend to converge, the fluctuations remain large. Based on this, using these three training frameworks, and adjusting the weight coefficients of different reward values in the reward function according to the weight coefficient combinations provided in

Table 2. Weight series combination.

	A	B	C	D	E	F
${\beta }_{\mathit{exploration}}$	0.35	0.35	0.43	0.46	0.48	0.25
${\beta }_{cooperation}$	0.43	0.39	0.31	0.35	0.51	0.28
${\beta }_{avoid collision}$	0.39	0.43	0.45	0.51	0.44	0.51
${\beta }_{efficiency}$	0.31	0.41	0.28	0.48	0.35	0.50

, the system is retrained. Figure 11 shows the training results when using the F combination. From Figure 11, it can be seen that, under the DDPG, TFMUSV, and PPO training frameworks, the trends of the node reward values are similar, but further analysis reveals that TFMUSV results in significantly higher initial rewards for the nodes compared to the other two frameworks. Additionally, in the later stages of training, DDPG and PPO lead to reward values that are clearly lower than those obtained with TFMUSV, and the reward value fluctuations in the later stages of training are also significantly smaller in TFMUSV compared to DDPG and PPO. The training results indicate that, compared to the other two training frameworks, TFMUSV enables the system to learn more effective task strategies, thereby allowing the system to achieve higher rewards during task execution.

6. Simulation Test

6.1. Test Evaluation Parameters

To verify whether the algorithm proposed in this paper can help the multi-unmanned vessel system efficiently and accurately complete the patrolling task, simulation testing was conducted on the trained system within the multi-unmanned vessel cooperative patrolling task simulation environment constructed in the earlier stages. During the testing process, several control algorithms currently applied in multi-agent cooperative patrolling tasks were used as comparisons to evaluate the performance of the algorithm proposed in this paper.

Before conducting the test, the performance parameters for evaluating the algorithm were first set. The performance parameters consist of several parts, namely patrol time $T_{total}$, coverage $R_{cover}$, total navigation distance $D_{total}$, CRBAA (Collision Rate Between Agent and Agent), and CRBAO (Collision Rate Between Agent and Obstacle). The calculation methods for each experimental parameter are shown in Equation (46):

$$\left\{\begin{array}{c}{T}_{total}={\displaystyle \frac{{\sum }_{i=1}^{N_{episode}}{T}_i^{episode}}{N_{episode}}},i=\mathrm{1,2},\dots ,N_{episode}\\ D_{total}={\displaystyle \frac{{\sum }_{i=1}^{N_{episode}}{D}_i^{total}}{N_{episode}}},i=\mathrm{1,2},\dots ,N_{episode}\\ R_{cover}={\displaystyle \frac{{\sum }_{i=1}^{N_{episode}}\frac{S_i^{cover}}{S_A}}{N_{episode}}},i=\mathrm{1,2},\dots ,N_{episode}\\ CRBAA={\displaystyle \frac{{\sum }_{i=1}^{N_{episode}}{N}_i^{collisionagents}}{N_i^{step}}},i=\mathrm{1,2},\dots ,N_{episode}\\ CRBAA={\displaystyle \frac{{\sum }_{i=1}^{N_{episode}}{N}_i^{collisionobstacles}}{N_i^{step}}},i=\mathrm{1,2},\dots ,N_{episode}\end{array}\right.$$

(46)

$N_{episode}$ represents the total number of episodes. $T_i^{episode}$ represents the time required for the system to complete a certain episode. When $T_i^{episode}\ge 3T_{Benchmark}$, the episode will be terminated. $D_i^{total}$ represents the total navigation distance of the system nodes during a certain episode. $S_i^{cover}$ represents the area covered by the system in the task region during a certain episode, and $S_A$ represents the actual area of the task region. $N_i^{collisionagents}$ represents the number of collisions between nodes during a certain episode, and $N_i^{step}$ represents the total number of actions taken by the system nodes during that episode. $N_i^{collisionobstacles}$ represents the number of collisions between nodes and obstacles during a certain episode. Additionally, to simulate the patrolling information obtained during the task, different locations in the task area have different depths. When the node sensor covers the corresponding task area, the node will obtain the corresponding water depth data for that area, and the higher the completion of the task, the more complete the system’s water depth data for the task area will be.

In the comparison tests, the chosen algorithms were the anti-flocking-based cooperative patrolling control algorithm (referred to as the anti-flocking algorithm) and the partitioned scanning patrolling control algorithm (referred to as the partitioned scanning patrolling algorithm). Different testing scenarios were set based on different system scales during the test, as follows: Scenario 1: $N_U=3$, Scenario 2: $N_U=4$, Scenario 3: $N_U=5$ Each test scenario was conducted 100 times, $N_{episode}=100$. Every 30 tests were grouped together, and the test results represent the average value of each test group.

6.2. Patrolling Performance Test

6.2.1. Performance Comparison of Three Algorithms

To validate the effectiveness of the proposed algorithm, comparative experiments were conducted in different task areas with three obstacles. The performance of the proposed Training Framework for Multi-USV (TFMUSV) was benchmarked against two existing algorithms: the anti-flocking algorithm and the partition scanning algorithm. In terms of navigation efficiency, the differences in the performance of the three algorithms are reflected in the three performance parameters: $R_{cover}$, $D_{total}$, and $T_{total}$.

As shown in Figure 12, from the rate of increase of the $R_{cover}^i\left(k\right)$ value, the proposed TFMUSV is significantly better than both the anti-flocking algorithm and the partition scanning patrol algorithm. At the same time point, the $R_{cover}^i\left(k\right)\ge 99\%$ value with the proposed algorithm is higher than that of the other two control algorithms. When the system controlled by the proposed TFMUSV achieves the task goal ($R_{cover}^i\left(k\right)\ge 99\%$), the $R_{cover}^i\left(k\right)$ values of the anti-flocking algorithm and the partition scanning patrol algorithm are 94.38% and 85.12%, respectively.

As shown in Figure 13, Figure 14 and Figure 15, it can be observed that the proposed TFMUSV consistently has the smallest $D_{total}$ and $T_{total}$ values in all task scenarios. Although the $R_{cover}$ values under the algorithm proposed in this paper are slightly lower than those of the partition scanning patrol algorithm in various task scenarios, the $R_{cover}^i\left(k\right)$ values still remain above 99%, which is sufficient to meet the predefined task objectives. The proposed TFMUSV demonstrates significant performance improvements, achieving a 23% reduction in $T_{total}$ and a 25% decrease in $D_{total}$ compared to the anti-flocking method. When benchmarked against the partition scanning patrol algorithm, the proposed TFMUSV reduces $T_{total}$ by 74% and $D_{total}$ by 63%, highlighting its superior efficiency in dynamic path planning and resource optimization.

6.2.2. Performance Analysis of the Proposed Algorithm

This section provides a systematic validation of the proposed multi-USV cooperative patrol algorithm, focusing on its core advantages in coverage efficiency, path planning, and dynamic adaptability. Key performance metrics, including $R_{cover}$, $D_{total}$, $T_{total}$, CRBAA and CRBAO, are rigorously analyzed. Scalability tests under varying task areas (1–2.5 km²) and communication radius (200–500 m) further validate the algorithm’s robustness in dynamic environments.

As shown in Figure 16, the navigation trajectories under the proposed algorithm exhibit three distinct operational phases. During the initial stage (Figure 16a,b), the overlap ratio of node trajectories remains below 5%, demonstrating efficient resource dispersion enabled by real-time collaborative information graphs (Equation (11)). This design minimizes redundant exploration while maximizing early-stage coverage efficiency. As the mission progresses, nodes gradually disperse to designated operational zones, highlighting the kinematic smoothing effect of dynamic window constraints (Equations (15)–(17)). During the mission, we observed that the navigation trajectories of some nodes exhibited “drastic” unclosed loops near the boundaries of unexplored regions (as shown by the red trajectory in Figure 16). This phenomenon can be attributed to the following factors: First, the feasible velocity vectors (RV) generated by the dynamic window algorithm (Equations (15)–(17)) only consider short-term kinematic constraints and obstacle avoidance requirements, without fully integrating global coverage objectives. This may cause path oscillations or abrupt turns when nodes approach the boundaries of unexplored regions. Second, although dense rewards have been introduced through reward reshaping (Equations (20), (23) and (36)), the gradient of the exploration reward $R_{cover}^i\left(k\right)$ may become discontinuous at the boundaries of unexplored regions, triggering “sudden” turns. However, in the final phase (Figure 16c), nodes collectively converge on uncovered regions, which validates the dense reward mechanism’s capability to balance exploration and cooperation (Equation (39)).

The algorithm’s environmental perception accuracy is confirmed by the high-fidelity water depth map in Figure 17. Concurrently, obstacle avoidance performance is evidenced by maintained safety margins in trajectory plots and ultralow collision rates (CRBAA $<$ 0.5%, CRBAO $<$ 0.3%, Figure 18). These outcomes are attributed to the integrated artificial potential field model (Equations (27)–(30)) and obstacle inflation strategy (Equation (35)), which enable proactive collision prevention rather than reactive maneuvers.

As shown in Figure 19 and Figure 20, extended evaluations under scaled task areas (1–2.5 km²) and communication radius (200–500 m) demonstrate the algorithm’s adaptability. Although both the $T_{total}$ value and $D_{total}$ increase with the growing task area, the $R_{cover}$ value consistently remains above 99%. Moreover, both the CRBAO and CRBAA values demonstrate that the nodes in the system can ensure safe navigation. This indicates that, even when facing different task areas, the system can still achieve the predefined task objectives under the control of the proposed algorithm.

The patrol performance of the system is influenced by the node communication radius ($R_c$). Expanding the communication radius from 200 m to 500 m reduces $T_{total}$ by 9.3% and $D_{total}$ by 7.8%. The primary reason for the test results mentioned above is that the variation in $R_c$ affects the USV’s ability to update task status information. In other words, a larger $R_c$ allows the node to integrate task information from more nodes, making the task information more complete and accurate, which in turn helps generate more effective actions. Test results from different task areas and communication distances indicate that when the task area and system communication radius change, the algorithm in this paper still enables the system to accomplish the task.

6.3. Robustness Performance Test

This section will test the robustness performance of the system under the algorithm proposed in this paper. The test task scenario is identical to the task scenario in the previous section’s performance testing. During the test, nodes are randomly selected from the system to act as faulty nodes. Once a node is selected as a faulty node, it will immediately stop operating. The selected faulty nodes will immediately stop their operations. In each test scenario, two different configurations for the number of faulty nodes are considered: $N_{break}=1$ and $N_{break}=2$.

Due to the limitations in the length of the paper, the analysis will focus on the testing results of Scenario 2. As shown in Figure 21 and Figure 22, the navigation trajectories of each node under the cooperative patrol control algorithm indicate that, during the early stage of the task, as illustrated in Figure 21a and Figure 22a, one or two nodes in the system stopped operations after performing tasks for a period of time. However, it can be observed from Figure 21 and Figure 22 that after some nodes in the system stopped operating, other remaining nodes took over the tasks of the corresponding faulty nodes and covered their originally assigned operational areas during their continued operation. The work process of the remaining nodes is generally consistent with the work process of the nodes in the previous performance test. Although there is some overlap in the operational areas of the four nodes, the overlapping trajectories are minimal. As the task progresses, the navigation trajectories of the four nodes gradually disperse, heading toward their respective areas of operation. In the final segment of the task, all nodes begin to approach areas that still lack patrol information and carry out operations in those areas. Regardless of whether the number of faulty nodes is $N_{break}=1$ or $N_{break}=2$, the navigation trajectories of the nodes in the system all conform to the aforementioned situation. As shown in Figure 23, the system’s actual acquisition of bathymetric data in the task area under different numbers of faulty nodes demonstrates that the system has successfully obtained the bathymetric data of the task area. This confirms that, under the algorithm proposed in this paper, even in the presence of faulty nodes, the system is still able to complete the set patrol objectives.

The test results of $R_{cover}^i\left(k\right)$ in Scenario 2 are shown in Figure 24. By analyzing the changes in $R_{cover}^i\left(k\right)$, it can be observed that the number of $N_{break}$ has a certain impact on the system’s patrol efficiency. Specifically, in the early stages of the task, the greater the number of $N_{break}$, the more significant the impact on the system’s patrol efficiency, resulting in a decrease in the system’s operational efficiency. However, as the task progresses, the influence of faulty nodes on the system’s operational efficiency gradually diminishes, and the performance gap between different numbers of faulty nodes begins to narrow. Nevertheless, this gap still exists by the end of the task. As seen in the results presented in Figure 20, the number of $N_{break}$ does not have a significant effect on the final $R_{cover}^i\left(k\right)$ value. The $R_{cover}^i\left(k\right)$ values in all test scenarios still reach over 99%. Figure 25 also presents the variations of several test parameters, including $T_{total}$, $D_{total}$, $R_{cover}$, CRBAO, and CRBAA, under different test scenarios. The test results indicate that the algorithm proposed in this paper exhibits strong robustness. When a system node experiences an unexpected failure and can no longer function, the algorithm is still able to provide optimal task actions in real-time based on the current task situation. This minimizes the negative impact of the faulty node on the system’s operations, ensuring the system continues to operate normally with a certain level of operational capability. Furthermore, during the operation, the algorithm ensures the navigational safety of the nodes.

7. Practical Test

7.1. Experimental Scenario

In the cooperative patrol experiment scenario, the base station’s latitude and longitude are (120.876243, 31.079691). A patrol area of 1 km × 1 km is set up within its vicinity, and the task objective is to use a multi-unmanned boat system to collect depth data across the entire mission area. Considering the practical conditions faced by the multi-unmanned boat system and the reliability constraints of the nodes, the real-boat test for the cooperative patrol task is divided into three groups: Experiment 1, with four unmanned boats operating in a task area with four obstacles, without any failure nodes; Experiment 2, with four unmanned boats, in which one failure node occurs (a randomly selected node is instructed to shut down by the shore terminal); Experiment 3, with four unmanned boats, in which two failure nodes occur, in a task area with four obstacles. The scanning patrol algorithm was chosen as the comparison testing algorithm.

To verify the effectiveness of the cooperative patrol control algorithm for the multi-unmanned boat system proposed in this paper, due to actual test conditions, the dynamics and kinematics models of the test boats differ from those used for the simulation experiments. Therefore, in the real-boat test, based on the real-time collective status of the water area, the proposed strategy outputs control actions. After coordinate system alignment and transformation, the control instructions are sent to the decision control system of the unmanned boats. The specific control process is as follows: the current situation (such as the unmanned boats’ coordinates and speed) is transformed into the simulation environment’s coordinate system. The trained task strategy, based on the transformed situation information, outputs the acceleration of the unmanned boats, from which speed and the next time-step coordinates are calculated. These are then converted into latitude, longitude coordinates, and speed in the water area coordinate system. Finally, the waypoint/speed control mode is used to control the unmanned boats’ movement. Due to objective constraints, the sensors carried by the nodes are also simulated or simplified. During the test, the nodes in the system do not carry the sensors needed to detect the depth. Instead, the patrol results of the nodes are provided by the shore-based platform, which constructs the water depth model of the task area and combines it with the real-time position feedback from each node.

7.2. Experimental Results and Analysis

The test results of Experiment 1 are shown in Figure 26 and Figure 27. The focus is on the test situation under the control of the multi-unmanned boat flexible cooperative patrol control strategy, aimed at global coverage. From the navigation trajectories of the nodes in the test process shown in Figure 26, it can be observed that the navigation trajectories of the nodes are evenly distributed within the task area, and no collisions occurred with the virtual obstacles in the test area. The changes in CRBAA and CRBAO data, as shown in

Table 3. Actual vessel test results of the patrol strategy proposed in this paper.

Test Algorithm	${\mathit{T}}_{\mathit{e}\mathit{p}\mathit{i}\mathit{s}\mathit{o}\mathit{d}\mathit{e}}$	${\mathit{D}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}$	${\mathit{R}}_{\mathit{c}\mathit{o}\mathit{v}\mathit{e}\mathit{r}}$	CRBAA	CRBAO
The proposed strategy in this paper	9.5 min	1.7 km	99.75%	0.15%	0.00%
Scanning patrol algorithm	18.8 min	3.3 km	100%	0.00	0.00%

, also demonstrate that the proposed strategy effectively achieves collision avoidance between nodes as well as between nodes and obstacles, ensuring the safe navigation of the nodes. From the depth map of the task area obtained under the control of the proposed strategy, shown in Figure 27, it can be seen that the system has successfully acquired the water depth data of the entire task area by the end of the mission. By mapping the completeness of depth data acquisition, we indirectly verify the algorithm’s ability to ensure global patrol coverage. In the test, the main differences in performance between the two control algorithms are reflected in the performance parameters $T_{episode}$, $R_{cover}$, and $D_{total}$. Table 3 shows the variations of these test parameters ($T_{episode}$, $D_{total}$, $R_{cover}$) under multiple test scenarios. The test results indicate that, compared to the scanning patrol algorithm, the proposed strategy has a similar $R_{cover}$ value, while the $D_{total}$ and $T_{episode}$ values are relatively lower. This suggests that the proposed strategy, in comparison to the scanning patrol algorithm, offers higher patrol efficiency and enables the system to effectively complete the cooperative patrol mission.

The test results of Experiment 2 and Experiment 3 are shown in Figure 28. From the test results, it can be seen that after certain nodes in the system stop working, other nodes continue to operate and take over the patrol tasks in the nearby task areas of the failed nodes. The navigation trajectories of the remaining nodes are evenly distributed within the task area.

Table 4. The actual vessel test results of the patrol strategy’s robustness in this paper.

The Number of Faulty Nodes	${\mathit{T}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}$$, {\mathit{D}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}$$, {\mathit{R}}_{\mathit{c}\mathit{o}\mathit{v}\mathit{e}\mathit{r}}$, CRBAA, CRBAO
$N_{break}=1$ $N_{break}=2$	13.5 min, 8 km, 99.96%, 0.48%, 0.94%, 99.96%
	14.9 min, 9 km, 99.94%, 0.44%, 0.41%, 99.69%

shows the variation of test parameters $T_{episode}$, $D_{total}$, and $R_{cover}$ under multiple tests. It can be observed that the number of $N_{break}$ and the time when a node fails mainly affect the system’s patrol efficiency. The larger the number of $N_{break}$, and the earlier the occurrence of a failed node, the larger the value of $T_{total}$. However, the tests indicate that the number of $N_{break}$ and the time of node failure have no significant impact on the final values of $R_{cover}$, CRBAA, and CRBAO. The system’s coverage of the task area remains above 99%. The test results demonstrate that the proposed strategy is robust, ensuring the system continues to operate normally and maintains a certain level of operational capability even when a node experiences a sudden failure and can no longer function.

8. Conclusions and Future Works

This paper conducts an in-depth study on the cooperative patrol tasks of multi-USV systems. The research first proposes a hybrid embedded task state information modeling method. Based on this, a distributed global collaboration patrol algorithm for multi-USV systems aimed at full-area coverage is proposed, integrating an end-to-end improved OODA decision-making model. By integrating patrol, collaboration, and obstacle information graphs into a unified state representation, the proposed algorithm enables real-time generation of kinematically feasible control actions via dynamic window constraints. The dense reward structure with adaptive weighting coefficients addresses sparse feedback challenges, demonstrating 99.75% coverage rates in simulations and field tests while maintaining collision rates below 0.5%. Experimental validations confirm the framework’s robustness, sustaining operational capability when 50% of nodes malfunction.

Despite these advancements, limitations persist in handling complex hydrodynamic interactions and ultra-large fleet scalability. Future work will focus on integrating environmental current models and wind field estimators to enable real-time compensation for multi-scale disturbances, particularly for typhoon-level winds. And the LSTM-based cooperative information graph demonstrates adaptability in moderate-delay scenarios, its performance under significant communication delays ($∆t>2$s) remains an open challenge. Future work will integrate delay-tolerant mechanisms, such as hybrid Kalman-filter-LSTM prediction and decentralized clock synchronization protocols, to enhance real-time coordination in latency-prone environments. Empirical validation under controlled delay injection will be conducted to quantify the framework’s robustness.

This research establishes a theoretical and practical foundation for autonomous maritime surveillance systems, offering scalable solutions for environmental monitoring and search-rescue missions. The proposed methodology bridges the gap between reinforcement learning theory and marine robotics applications, paving the way for intelligent coordination in complex oceanic scenarios.