Reinforcement Learning-Based Trajectory Planning for Dual-UGV Cooperative Carrying with Adaptive Target Entropy Regulation

Abstract

Trajectory planning for dual unmanned ground vehicles (UGVs) in cooperative carrying remains challenging in complex environments. The relative distance constraint imposed by the shared payload significantly increases the difficulty of cooperative trajectory planning. To address this issue, this paper proposes a reinforcement learning-based dual-UGV cooperative trajectory planning method with adaptive target entropy regulation. Specifically, a task-oriented local observation representation and a cooperative reward function are jointly designed for target guidance, distance maintenance, and obstacle avoidance, so that the learned policy can better satisfy the cooperative control objectives. Moreover, a target entropy regulation mechanism driven by relative distance error is incorporated into the maximum-entropy policy optimization process, enabling adaptive adjustment of the exploration intensity according to the current distance error. These designs are unified within a Multi-Agent Soft Actor-Critic (MASAC) framework under the centralized training and decentralized execution (CTDE) paradigm, forming a complete learning-based solution for dual-UGV cooperative carrying. Simulation results, including validation, comparative, and ablation studies, demonstrate that the proposed method can generate safe coordinated trajectories and achieve high relative-distance maintenance accuracy.

1. Introduction

UGVs play an important role in tasks such as autonomous inspection [1], environmental monitoring [2], infrastructure-related detection and localization [3] and mine search and rescue [4]. In addition, AI-powered monitoring and detection technologies are increasingly used in transportation infrastructure, such as transparent-road sensing and road-condition monitoring [5]. In many cooperative carrying scenarios, such as transporting slender rigid objects, stretchers, or industrial components, the payload geometry imposes a prescribed relative distance constraint between the two UGVs. As a result, maintaining the required inter-UGV distance throughout the motion process becomes a fundamental requirement of the task. Compared with conventional formation control [6], the fixed distance constraint significantly restricts the set of feasible cooperative motions and increases the difficulty of coordination [7]. This makes it more challenging to simultaneously achieve goal reaching, distance maintenance, and obstacle avoidance in complex environments.

Existing trajectory planning research mainly relies on traditional model-driven methods [8,9], including graph search [10,11], sampling-based planning [12], multi-robot cooperative control [13] and collision avoidance methods [14,15]. However, when applied to the dual-UGV cooperative carrying task, these methods still face significant challenges because the shared payload imposes a prescribed relative distance constraint between the two UGVs and strengthens the trajectory coupling between the two UGVs.

Data-driven neural-network models have been widely used for complex engineering modeling and parameter identification tasks, such as elastic-constant determination in composite materials [16]. Reinforcement learning provides a new framework for sequential decision-making in complex environments [17]. This framework has been applied to reconnaissance [18], cooperative navigation [19], collision avoidance [20], and multi-robot task allocation and scheduling [21]. However, applying reinforcement learning to trajectory planning for dual-UGV cooperative carrying tasks remains challenging. Target guidance, distance maintenance, and obstacle avoidance need to be jointly addressed within a unified reinforcement learning framework, thus increasing the difficulty of reward design and coordinated satisfaction of coupled task constraints. In addition, dual-UGV cooperative carrying requires high stability of cooperative motion. Adaptive learning strategies have been explored in dynamic decision-making problems, such as fuzzy transfer learning-based resource allocation [22]. Similarly, in dual-UGV cooperative carrying, the required exploration intensity changes with the cooperative state. If the exploration intensity is set improperly, it will further affect the optimization effect of the policy. Specifically, excessive stochastic exploration may disturb established cooperative motion patterns, and insufficient exploration may limit the policy’s ability to escape locally suboptimal solutions. Therefore, general reinforcement learning methods are often inadequate for this task. Beyond algorithmic performance, the practical deployment of learning-based autonomous decision-making methods also requires attention to reliability, interpretability, and governance issues, as discussed in recent studies on AI governance beyond external constraints [23].

To address this issue, dedicated designs are still required in observation representation, reward modeling, and exploration regulation. Specifically, a task-oriented local observation representation is developed for the dual-UGV cooperative carrying task by integrating goal-guidance information, inter-UGV relative-state information, and local obstacle information into each UGV’s local observation. In addition, a cooperative reward modeling method is constructed to jointly characterize goal guidance, distance maintenance, and safe obstacle avoidance, thus improving the consistency between learning signals and cooperative control objectives. Furthermore, the distance error is incorporated into the target-entropy regulation of the maximum-entropy policy, enabling adaptive adjustment of the exploration intensity according to the current distance error. On this basis, a reinforcement learning framework for dual-UGV cooperative carrying under distance constraints is constructed. As shown in Figure 1, the two UGVs are required to reach their respective target positions without collision while maintaining the prescribed relative distance throughout the motion. The main contributions are as follows:

Figure 1. Schematic of cooperative trajectory planning for dual UGVs in a cooperative carrying task.

• A distance-error-driven adaptive target entropy regulation method is proposed for dual-UGV cooperative carrying tasks. By explicitly incorporating the inter-UGV distance error into the target-entropy regulation of the maximum-entropy policy, the exploration intensity can be adaptively adjusted according to the current cooperative state. This method achieves a better balance between policy exploration and distance maintenance, thus improving training stability and convergence efficiency.
• A joint design method for local observation representation and reward modeling is proposed for dual-UGV cooperative carrying tasks. The observation representation and reward modeling are jointly constructed around the requirements of goal guidance, distance maintenance, and safe obstacle avoidance, thus improving the consistency among state representation, learning signals, and cooperative control objectives.
• Under the CTDE paradigm, the proposed observation–reward joint design and adaptive target entropy regulation mechanism are unified within the MASAC framework. This forms a complete reinforcement learning solution for dual-UGV cooperative carrying under distance constraints. Simulation results verify its effectiveness in relative-distance maintenance accuracy, training stability, convergence speed, and cooperative trajectory quality.

2. Related Work

2.1. Classical Trajectory Planning Methods

Classical trajectory planning methods typically rely on explicit modeling of motion constraints and environmental structures to achieve cooperative planning and navigation. Li et al. introduce adaptive-scaling constrained optimization to improve the feasibility of cooperative trajectory planning in constrained environments [24]. In addition, Ma et al. propose a decentralized planning method for car-like robotic swarms in cluttered environments [25]. Based on [25], Ruiz-Fernandez et al. propose a decentralized formation-control algorithm based on path-planning methods for multi-robot systems in environments with non-convex and dynamic obstacles [14]. Zhao et al. propose the stress response mechanism-artificial potential field (SRM-APF) method for multi-agent formation [26]. These methods integrate local interaction mechanisms with artificial potential fields to enable decentralized formation maintenance and obstacle avoidance. However, their performance often depends heavily on accurate environment modeling and carefully designed rules. When complex environments and coordination constraints are considered simultaneously, the modeling and solution processes become significantly more complicated. Therefore, these methods remain limited in addressing dual-UGV cooperative carrying tasks with fixed inter-UGV distance constraints.

2.2. Reinforcement Learning for Single-Agent Trajectory Planning

Reinforcement learning (RL) has emerged as an effective approach for trajectory planning. In particular, deep reinforcement learning has been successfully applied to mobile robot path planning in unknown environments [27]. He et al. propose an asynchronous multithreading reinforcement learning-based framework for integrated path planning and tracking [28]. Wang et al. incorporate an integral compensation mechanism into the deterministic policy gradient framework to improve robustness [29]. Wang et al. propose a hierarchical framework for real-time trajectory planning and trajectory tracking of gliding robotic dolphins [30]. Overall, these studies demonstrate the effectiveness of RL for trajectory planning in single-agent systems. However, compared with single-agent trajectory planning, the dual-UGV cooperative carrying task involves stricter coordination requirements, as it must jointly consider dual-UGV coordination, fixed-distance maintenance, and obstacle avoidance, thus making the problem more difficult to solve.

2.3. Multi-Agent Reinforcement Learning for Cooperative Trajectory Planning

Multi-agent reinforcement learning (MARL) has become an important approach in multi-robot systems. Chen et al. [31] and Long et al. [32] employ deep reinforcement learning for decentralized multi-agent collision avoidance. Wu et al. proposed the multi-robot dynamic window approach–multi-agent deep deterministic policy gradient (MRDWA-MADDPG) method for distributed multi-agent path planning [33]. This method integrates the dynamic window approach (DWA) with multi-agent deep deterministic policy gradient (MADDPG). Zhan et al. developed the twin delayed multi-agent deep deterministic policy gradient (MATD3) method to alleviate overestimation bias and improve deterministic multi-agent policy learning [34]. Value decomposition methods enhance cooperative multi-agent learning by decomposing the joint action value function, with representative approaches including QPLEX [35] and QMIX [36]. Sun et al. introduce task decomposition mechanisms to enhance cooperative policy learning [37]. To improve exploration efficiency and training stability, maximum-entropy reinforcement learning is introduced into multi-agent learning. Haarnoja et al. propose the soft actor–critic (SAC) algorithm, which improves policy exploration through entropy regularization [38]. He et al. design a hybrid motion planner based on the multi-agent soft actor–critic (MASAC) algorithm [39]. In standard SAC, the target entropy is generally predefined as a constant value, and the temperature coefficient is adaptively updated to make the policy entropy track this fixed entropy target. However, in dual-UGV cooperative carrying tasks, the required exploration intensity is not constant but varies with the inter-UGV distance error. Therefore, a fixed target entropy cannot explicitly characterize the exploration demand associated with cooperative-distance deviations. In addition, existing MARL methods mainly focus on problems such as collision avoidance, cooperative navigation, and task allocation. Limited attention has been paid to dual-UGV trajectory planning for cooperative carrying. This task requires not only target reaching and obstacle avoidance but also maintaining a stable relative distance during motion. As a result, it remains difficult for existing methods to achieve both accurate relative distance maintenance and efficient policy learning simultaneously. To address this issue, this paper proposes a reinforcement learning-based cooperative trajectory planning method for dual UGVs with adaptive target entropy regulation.

3. Problem Formulation and Cooperative Policy Learning Framework Design

For clarity, the main notation used in the cooperative policy learning framework is summarized in Table 1.

Table 1. Main notation used in this paper.

Parameter	Description
${\mathit{s}}_t$	Global state at timestep $t$
${\mathit{o}}_t^i$	Local observation of UGV $i$ at timestep $t$
${\mathit{a}}_t^i$	Action of UGV $i$ at timestep $t$
$Q_{{\phi }_n^i}$	The $n$-th soft Q-network of agent $i$
$Q_{{\overline{\phi }}_n^i}$	The $n$-th target Q-network of agent $i$
${\mathrm{H}}_{\mathrm{target}}(t)$	Target entropy at timestep $t$
$d_{\mathrm{target}}$	Target distance
$d_{ij}$	Relative distance between UGV $i$ and $j$ at timestep $t$
$e_d(t)$	Relative distance error

3.1. Problem Formulation Under the Decentralized Partially Observable Markov Decision Process (Dec-POMDP) Framework

Multi-agent cooperative decision-making tasks under partial observations are formulated as decentralized partially observable Markov decision processes (Dec-POMDPs) [40]:

$$G=⟨\mathit{I},\mathit{S},\mathit{A},\mathit{P},\mathit{R}⟩,$$

(1)

where $\mathit{I}$ denotes the index set of UGVs, $i\in \mathit{I}=\{1,2\dots ,N\}$. $\mathit{S}$ denotes the state space, and ${\mathit{s}}_t\in \mathit{S}$ denotes the global state at timestep $t$. In this work, the state is constructed from the local observations ${\mathit{o}}_t^i$ of both UGVs as ${\mathit{s}}_t=[{\mathit{o}}_t^1,{\mathit{o}}_t^2\dots ,{\mathit{o}}_t^N]\in \mathit{S}$. The joint action space is defined as $\mathit{A}={\mathit{A}}_1\times {\mathit{A}}_2\times \dots {\mathit{A}}_N$, where ${\mathit{a}}_t^i\in {\mathit{A}}_i$ denotes the action of UGV $i$ at timestep $t$. $\mathit{P}({\mathit{s}}_{t+1}|{\mathit{s}}_t,{\mathit{a}}_t)$ is the state transition function, describing the probability of transitioning to the next state. $\mathit{R}:\mathit{S}\times \mathit{A}\to \mathit{R}$ is the reward function used to evaluate the immediate reward. This formulation provides a unified mathematical basis for the subsequent cooperative policy learning framework.

3.2. Multi-Agent Soft Actor-Critic Framework

The dual-UGV cooperative trajectory planning problem is solved using MASAC under the CTDE paradigm. During execution, each UGV makes decisions independently based on its local observation. During training, however, synchronous policy updates may make the learning environment nonstationary. Therefore, CTDE is adopted to improve training stability while preserving decentralized execution. As shown in Figure 2, the global state ${\mathit{s}}_t$ and joint action ${\mathit{a}}_t=({\mathit{a}}_t^1,{\mathit{a}}_t^2)$ are used for centralized value evaluation during training. During execution, each UGV independently outputs its control action ${\mathit{a}}_t^i$ based on its local observation ${\mathit{o}}_t^i$.

Figure 2. The CTDE paradigm adopted for cooperative trajectory planning of dual UGVs.

Under the CTDE paradigm, the cooperative trajectory planning problem is addressed using the MASAC framework [38]. In contrast to deterministic policy methods, MASAC models the policy as a parameterized probability distribution and optimizes it under the maximum entropy objective. This mechanism enables the policy to improve expected cumulative reward and to maintain the necessary exploration during training. As a result, premature convergence to a deterministic policy is alleviated. Based on this formulation, the corresponding actor–critic architecture and parameter update process are detailed in this section. Figure 3 illustrates the overall training structure of MASAC under the CTDE paradigm.

Figure 3. Overall training structure of MASAC under the CTDE paradigm. The arrows indicate the information flow among the policy networks, centralized Q-networks, environment, and replay buffer during training.

The centralized critic of MASAC consists of twin soft $Q$-networks and the corresponding target $Q$-networks. The twin soft $Q$-networks are denoted by $Q_{{\phi }_n^i}$, and the corresponding twin target $Q$-networks are denoted by $Q_{{\overline{\phi }}_n^i}$, where $n\in \{1,2\}$. To construct the soft target, the next state-action pair $({\mathit{s}}_{t+1},{\tilde{\mathit{a}}}_{t+1})$ is evaluated by the twin target $Q$-networks, and the smaller one is selected to alleviate action value overestimation. The soft target value $y_t^i$ is defined as

$$y_t^i=r_t^i+\gamma (\underset{n\in \left\{1,2\right\}}{\min }{Q}_{{\overline{\phi }}_n^i}({\mathit{s}}_{t+1},{\tilde{\mathit{a}}}_{t+1})-{\displaystyle \sum _{i=1}^2\alpha \log {\pi }_{{\theta }_i}({\tilde{\mathit{a}}}_{t+1}^i|{\mathit{o}}_{t+1}^i))},$$

(2)

where $r_t^i$ represents the reward of UGV $i$ at timestep $t$; the discount factor $\gamma \in (0,1)$ is introduced to balance the contributions of immediate and future rewards. ${\tilde{\mathit{a}}}_{t+1}=({\tilde{\mathit{a}}}_{t+1}^1,{\tilde{\mathit{a}}}_{t+1}^2)$ denotes the next joint action generated by the current policies based on the next local observations ${\mathit{o}}_{t+1}^i$. $\alpha$ is a shared temperature coefficient, and its specific design will be detailed in the following subsection.

Using the soft target value $y_t^i$ derived from the soft Bellman equation [41] as the regression target, the current centralized critic networks are updated by minimizing the Bellman regression loss, which is defined as:

$$L_Q({\phi }^i)={\mathrm{{\rm E}}}_{({\mathit{s}}_t,{\mathit{a}}_t,r_t,{\mathit{s}}_{t+1})~D}{\displaystyle \sum _{n=1}^2{(Q_{{\phi }_n^i}({\mathit{s}}_t,{\mathit{a}}_t)-y_t^i)}^2},$$

(3)

where $({\mathit{s}}_t,{\mathit{a}}_t,r_t,{\mathit{s}}_{t+1})$ denotes the rollout data sampled from the replay buffer $D$. Based on the estimates of the current critic networks, the $i$-th policy network is optimized according to the maximum-entropy objective. Its loss function is defined as:

$$L_{{\pi }_i}({\theta }_i)=E_{({\mathit{s}}_t,{\mathit{a}}_t)\sim D,\epsilon \sim N}[\alpha \log {\pi }_{{\theta }_i}({\tilde{\mathit{a}}}_t^i|{\mathit{o}}_t^i)-\underset{n\in \{1,2\}}{\min }{Q}_{{\phi }_n^i}({\mathit{s}}_t,{\tilde{\mathit{a}}}_t)].$$

(4)

Here, ${\tilde{\mathit{a}}}_t=({\tilde{\mathit{a}}}_t^1,{\tilde{\mathit{a}}}_t^2)$ denotes the joint action generated by the current policies of the two UGVs. Reparameterization is adopted to enable stochastic action sampling and to preserve differentiability with respect to the policy parameters. Specifically, the sampled action of the $i$-th UGV is given by

$${\tilde{\mathit{a}}}_t^i=f_{{\theta }_i}({\mathit{o}}_t^i,{ϵ}_t^i)=\tanh ({\mu }_{{\theta }_i}({\mathit{o}}_t^i)+{\sigma }_{{\theta }_i}({\mathit{o}}_t^i)\odot \epsilon ),\epsilon \sim N(0,I),$$

(5)

where $\epsilon$ is a Gaussian noise term, ${\mu }_{{\theta }_i}({\mathit{o}}_t^i)$ and ${\sigma }_{{\theta }_i}({\mathit{o}}_t^i)$ are the mean and standard deviation produced by the policy network, respectively. The $\tanh (\cdot )$ activation function squashes the action into the normalized action space [−1, 1]. This squashing operation is adopted to ensure bound continuous actions, improve training stability, and facilitate mapping to the physical control constraints of UGVs. Through the reparameterization, the sampled action remains differentiable with respect to the policy parameters.

It should be noted that the parameters of the target networks ${\overline{\phi }}_n^i$ are not fixed. Instead ${\overline{\phi }}_n^i$ are slowly updated toward the parameters of the current $Q$ networks ${\phi }_n^i$ through a soft update mechanism:

$${\overline{\phi }}_n^i\leftarrow \tau {\phi }_n^i+(1-\tau ){\overline{\phi }}_n^i,$$

(6)

where $\tau \in (0,1)$ is the soft update coefficient, which controls the update rate of the target $Q$-networks. In this work, $\tau$ is set to 0.01. This mechanism allows the target $Q$-networks to track the parameter variations in the current $Q$-networks smoothly, thus improving training stability.

In the maximum-entropy learning framework, the temperature coefficient $\alpha$ regulates policy stochasticity and balances exploration and exploitation. However, in the dual-UGV cooperative carrying task, a fixed target entropy cannot flexibly adjust the exploration intensity according to task requirements, which limits the effectiveness of policy optimization. Therefore, a distance-error-driven adaptive target entropy regulation mechanism is introduced in the following subsection.

3.3. Distance-Error-Driven Adaptive Target Entropy Regulation

For the dual-UGV cooperative carrying task, when the actual relative distance deviates significantly from the target distance, stronger exploration is required to help the policy escape undesirable cooperative states and search for more effective motion patterns. In contrast, when the relative distance is already close to the target value, excessive policy stochasticity may induce unnecessary oscillatory behavior and reduce the stability of distance maintenance. To address this issue, the current relative distance error is incorporated into the target entropy adjustment process as a feedback signal, enabling adaptive modulation of exploration intensity according to the current cooperative condition. Let $d_{\mathrm{target}}$ denote the target distance, and the distance error $e_d(t)$ is defined as

$$e_d(t)=| d_{ij}(t)-d_{\mathrm{target}} |.$$

(7)

Based on $e_d(t)$, the dynamic target entropy ${\mathrm{H}}_{\mathrm{target}}(t)$ is designed as

$${\mathrm{H}}_{\mathrm{target}}(t)=-\dim (A)+\lambda \cdot \tanh (\eta e_d(t)),$$

(8)

where $\lambda \in (0,\dim (\mathit{A}))$ and $\eta$ is the sensitivity coefficient of the distance error. It should be noted that the proposed mechanism differs from the automatic temperature adjustment in standard SAC. In standard SAC, the target entropy is generally predefined as a constant value, and the temperature coefficient is updated to make the policy entropy track this fixed entropy target. In the proposed method, the temperature-update form is retained, but the target entropy is no longer fixed. Instead, it is dynamically determined by the inter-UGV distance error. Therefore, the temperature coefficient tracks a distance-error-driven target entropy rather than a constant entropy target. The $\tanh (\cdot )$ function is adopted because it is monotonically increasing over the nonnegative interval, globally bounded, and has high sensitivity near zero. Therefore, the target entropy can increase smoothly with the increase in distance error, and its growth range is limited. When approaching the target distance, the function can still respond effectively to small errors. Figure 4 illustrates the effects of λ and η on the dynamic target entropy with respect to the relative-distance error. In Figure 4a, η is fixed at 10 and λ is varied to examine its influence on the adjustment amplitude of the dynamic target entropy. In Figure 4b, λ is fixed at 0.5 and η is varied to examine its influence on the response sensitivity of the target entropy to the relative-distance error.

Figure 4. Effects of λ and η on the dynamic target entropy with respect to the relative-distance error $e_d(t)$. (a) Effect of λ with fixed η = 10. (b) Effect of η with fixed λ = 0.5.

The parameter λ controls the adjustment amplitude of the target entropy. Increasing λ expands the variation range of the target entropy with respect to the distance error, enhancing the capability of regulating exploration intensity. However, an excessively large λ value may amplify entropy fluctuations and introduce additional policy stochasticity, which can impair cooperative motion stability. Conversely, a small λ value restricts the entropy adjustment range and weakens the adaptive regulation effect, making the mechanism closer to a fixed target entropy setting. The parameter η controls the response sensitivity of the target entropy to the relative-distance error. Increasing η accelerates the response of the target entropy to distance-error variations. However, an excessively large η may cause the tanh function to approach saturation within a narrow error range, thereby reducing its ability to distinguish different error levels.

Therefore, λ and η should be selected to provide sufficient exploration regulation while avoiding excessive stochasticity near the desired relative distance. The bounded and monotonic regulation structure enables the proposed mechanism to enhance exploration when the cooperative distance substantially deviates from the target value and to suppress unnecessary stochasticity when the relative distance approaches the desired value.

Then the temperature parameter $\alpha$ is adaptively updated by minimizing the temperature loss function

$$L(\alpha )={\mathrm{{\rm E}}}_{{\mathit{a}}_t\sim \pi }(-\alpha ({\displaystyle \sum _{i=1}^2\log {\pi }_{{\theta }_i}({\mathit{a}}_t^i|{\mathit{o}}_t^i)}+{\mathrm{H}}_{\mathrm{target}}(t))).$$

(9)

To examine the effect of the dynamic target entropy on temperature adjustment, differentiating Equation (9) with respect to $\alpha$ yields

$${\nabla }_{\alpha }L(\alpha )={\mathrm{{\rm E}}}_{{\mathit{a}}_t\sim \pi }-({\displaystyle \sum _{i=1}^2\log {\pi }_{{\theta }_i}({\mathit{a}}_t^i|{\mathit{o}}_t^i)}+{\mathrm{H}}_{\mathrm{target}}(t)).$$

(10)

Accordingly, the gradient-descent update of $\alpha$ can be written as

$$\alpha \leftarrow \alpha +\rho ({\displaystyle \sum _{i=1}^2\log {\pi }_{{\theta }_i}({\mathit{a}}_t^i|{\mathit{o}}_t^i)}+{\mathrm{H}}_{\mathrm{target}}(t))$$

(11)

where $\rho$ is the learning rate. Equation (11) shows that a larger distance error leads to a larger ${\mathrm{H}}_{\mathrm{target}}(t)$, thus enlarging the temperature update term. Since $\alpha$ controls the strength of entropy regularization in Equation (4), this mechanism promotes broader exploration and helps the policy recover from undesirable cooperative states. In contrast, when the distance error becomes small, ${\mathrm{H}}_{\mathrm{target}}(t)$ decreases, which weakens the exploration-driving effect of the temperature update and suppresses excessive stochasticity near the target distance. As a result, the exploration intensity is adaptively regulated according to the current relative distance error, which is expected to reduce oscillations and improve training stability.

The adaptive target entropy mechanism preserves the original actor–critic optimization structure of MASAC. Specifically, the structural forms of the critic loss, policy loss, and temperature-update objective remain unchanged. The main modification lies in replacing the fixed entropy target with the distance-error-driven target entropy defined in Equation (8). Since the relative distance error is nonnegative and the $\tanh (\cdot )$ function is continuous, monotonic, and bounded, ${\mathrm{H}}_{\mathrm{target}}(t)$ is also bounded and varies smoothly within a finite interval. Therefore, the proposed entropy target does not introduce an unbounded entropy regulation term into policy optimization. Instead, the distance-error-driven entropy target serves as a bounded feedback modulation of the entropy objective. This bounded and smooth regulation property provides partial theoretical support for explaining why the proposed mechanism can improve exploration stability while preserving the maximum-entropy learning structure of MASAC. The overall procedure of the proposed MASAC training paradigm with adaptive target entropy regulation is summarized in Algorithm 1.

Algorithm 1: MASAC with Adaptive Target Entropy Regulation
Initialize: policy parameters ${\theta }_i$, critic parameters ${\phi }_n^i$, target critic parameters ${\overline{\phi }}_n^i$, temperature coefficient $\alpha$, replay buffers $D$, and target distance $d_{\mathrm{target}}$.
1	for $ep\mathrm{s}=1$ to $M$ do
2	Reset environment, obtain initial observation $o_i^0,i\in \{1,2\}$;
3	for $t=0$ to $T_{m\mathit{a}x}$ do
4	if $ep\mathrm{s}<M_{init}$ then
5	${\mathit{a}}_i^t\leftarrow \mathrm{RandomAction}(),i\in \{1,2\}$;
6	else
7	${\mathit{a}}_i^t~{\pi }_{{\theta }_i}(\cdot |{\mathit{o}}_i^t),i\in \{1,2\}$;
8	end if
9	${\mathit{a}}_t=({\mathit{a}}_t^1,{\mathit{a}}_t^2)$;
10	Store $({\mathit{s}}_t,{\mathit{a}}_t,{\mathit{r}}_t,{\mathit{s}}_{t+1})$ into $D$;
11	if update then
12	$B\leftarrow \mathrm{Sample}(D)$;
13	Update $Q_{{\phi }_n^i}$, $n\in \{1,2\}$, by Equation (3);
14	Update ${\pi }_{{\theta }_i}$, $i\in \{1,2\}$, by Equation (4);
15	$e_d(t)\leftarrow$ DistanceError($d_{ij}(t),d_{\mathrm{target}}$);
16	${\mathrm{H}}_{\mathrm{target}}(t)\leftarrow$ AdaptiveEntropy($e_d(t)$);
17	Update $\alpha$ by Equation (9);
18	if soft update then
19	${\overline{\phi }}_n^i\leftarrow \tau {\phi }_n^i+(1-\tau ){\overline{\phi }}_n^i,n\in \{1,2\}$;
20	end if
21	end if
22	${\mathit{o}}_i^t\leftarrow {\mathit{o}}_i^{t+1},{\mathit{s}}_t\leftarrow {\mathit{s}}_{t+1}$;
23	if terminal condition is satisfied break;
24	end for
25	end for

4. Task-Oriented Reinforcement Learning Design for Cooperative Carrying of Dual UGVs

4.1. Observation Space Design

In a 2D workspace, the position and velocity of UGV $i$ are denoted as $p_t^i\in {ℝ}^2$ and ${\mathit{v}}_t^i\in {ℝ}^2$, respectively. The target position is denoted by ${\mathit{g}}^i\in {ℝ}^2$. Let $j\in I\backslash \left\{i\right\}$ represent the index of the other UGV. In the considered task scenario, the observation space of each UGV contains only local environmental information and does not include global map information. The local observation ${\mathit{o}}_t^i$ of UGV $i$ is defined as:

$${\mathit{o}}_t^i=[d_{i,g},d_{ij},{\stackrel{\rightharpoonup }{\mathit{e}}}_{ij},{\stackrel{\rightharpoonup }{\mathit{e}}}_{i,g},{\stackrel{\rightharpoonup }{\mathit{v}}}_t^i,{\stackrel{\rightharpoonup }{\mathit{v}}}_{ij},d_{i,o}^{\min },{\stackrel{\rightharpoonup }{\mathit{e}}}_{i,o}], i\ne j$$

(12)

Here, $d_{i,g}$ and $d_{ij}$ denote the distances from UGV $i$ to its target position and to the other UGV, respectively. ${\stackrel{\rightharpoonup }{\mathit{e}}}_{i,g}$ and ${\stackrel{\rightharpoonup }{\mathit{e}}}_{ij}$ denote the unit direction vectors from the position of UGV $i$ to its target and to UGV $j$, respectively. The velocity vector of UGV $i$ is denoted as ${\stackrel{\rightharpoonup }{\mathit{v}}}_t^i$, and the relative velocity vector between the two UGVs is defined as ${\stackrel{\rightharpoonup }{\mathit{v}}}_{ij}={\mathit{v}}_t^i-{\mathit{v}}_t^j$.

Local obstacle information is computed based on the nearest obstacle within the finite sensing radius $r_s$. Let the obstacle set be denoted by ${\left\{{\mathit{o}}_k\right\}}_{k=1}^M$. If there exists at least one obstacle satisfying $| |{\mathit{o}}_k-{\mathit{p}}_t^i| |<r_s$, the index of the nearest obstacle is defined as

$$k^{*}=\arg \underset{| |{\mathit{o}}_k-{\mathit{p}}_t^i| |<r_s}{\min }| |{\mathit{o}}_k-{\mathit{p}}_t^i| |.$$

(13)

The minimum distance between UGV $i$ and the nearest obstacle is denoted as $d_{i,\mathrm{o}}^{\min }$, and the corresponding unit direction vector is denoted as ${\stackrel{\rightharpoonup }{\mathit{e}}}_{i,\mathrm{o}}$. These quantities are computed as:

$$d_{i,\mathrm{o}}^{\min }=| |{\mathit{o}}_{k^{*}}-{\mathit{p}}_t^i| |,{\stackrel{\rightharpoonup }{\mathit{e}}}_{i,\mathrm{o}}={\displaystyle \frac{{\mathit{o}}_{k^{*}}-{\mathit{p}}_t^i}{| |{\mathit{o}}_{k^{*}}-{\mathit{p}}_t^i| |}}.$$

(14)

As illustrated in Figure 5, the local obstacle information is constructed only from the nearest obstacle within the finite sensing radius $r_s$. When no obstacle lies within the sensing radius, the obstacle-related state variables $d_{i,\mathrm{o}}^{\min }$ and ${\stackrel{\rightharpoonup }{\mathit{e}}}_{i,\mathrm{o}}$ are set to 1 and ${[1,1]}^{\top }$.

Figure 5. Illustration of nearest obstacle selection and local obstacle representation within the finite sensing radius $r_s$. The dashed circle represents the finite sensing range, the gray circles represent obstacles, and the arrows indicate the local sensing directions.

4.2. Action Space Design

Based on the local observation representation defined above, each UGV generates a continuous control action. The action space of UGV $i$ is designed as a two-dimensional continuous space. Each action dimension represents the velocity increment of the UGV along the corresponding axis within one timestep, which is defined as

$${\mathit{a}}_t^i=[\Delta {\mathit{v}}_{x,t}^i,\Delta {\mathit{v}}_{y,t}^i], \Delta {\mathit{v}}_{x,t}^i,\Delta {\mathit{v}}_{y,t}^i\in [-\Delta {\mathit{v}}_{\max },\Delta {\mathit{v}}_{\max }]$$

(15)

where $\Delta {\mathit{v}}_{x,t}^i$ and $\Delta {\mathit{v}}_{y,t}^i$ denote the velocity increments of UGV $i$ along the $x$- and $y$-axes at timestep $t$, respectively, and $\Delta {\mathit{v}}_{\max }$ denotes the maximum allowable velocity increment along each direction within one timestep. In this work, $\Delta {\mathit{v}}_{\max }=0.1 \mathrm{m}/\mathrm{s}$. It should be noted that the action defined in Equation (15) is formulated as a high-level planning variable rather than a direct actuator-level control command. This action represents the desired motion adjustment in the 2D workspace and is used for reference trajectory generation. Accordingly, the proposed method focuses on planning-level cooperative decision-making, including goal reaching, distance maintenance, and obstacle avoidance. The nonholonomic kinematic constraints of differential-drive or Ackermann-steering UGVs are not explicitly embedded in the policy action space. For practical implementation, the generated reference trajectories need to be tracked by a lower-level controller that considers vehicle kinematics, actuator constraints, and tracking errors.

4.3. Cooperative Reward Function Design

To achieve cooperative trajectory generation of the dual UGVs, a reward function is designed to characterize target reaching, distance regulation, and safety requirements. Each UGV is required to reach its target positions from the initial position, while maintaining the target distance $d_{\mathrm{target}}$ and avoiding collisions. The reward is defined as

$$r_i^t=r_{\mathrm{goal}}(t)+r_{\mathrm{dist}}(t)+r_{\mathrm{safe}}(t),$$

(16)

where $r_{\mathrm{goal}}(t)$ is the goal guidance reward, $r_{\mathrm{dist}}(t)$ is the distance error penalty, and $r_{\mathrm{safe}}(t)$ denotes the safety penalty for obstacle avoidance.

The goal guidance reward $r_{\mathrm{goal}}(t)$ is designed to encourage both UGVs to approach their respective target positions. If reward is assigned only upon target arrival, the policy receives sparse reward signals over most of the trajectory generation process. As a result, sufficient supervision for learning motion toward the target cannot be obtained. To address this issue, $r_{\mathrm{goal}}(t)$ is formulated as an exponential function of the distance $d_{i,g}(t)$:

$$r_{\mathrm{goal}}(t)=w_g\exp (-k_g{d}_{i,g}(t)),$$

(17)

where $w_g$ is the goal reward weight and $k_g$ is the decay parameter. An exponential form is adopted to establish a nonlinear monotonic relationship between the distance $d_{i,g}(t)$ and the reward value. When the UGV is far from the target, the reward remains small and increases slowly to maintain strategic exploration. As the distance $d_{i,g}(t)$ decreases, the reward increases and rises more rapidly, thus strengthening target guidance and improving target reaching efficiency.

To regulate the actual relative distance $d_{ij}$ toward the target distance $d_{\mathrm{target}}$, the distance error penalty $r_{\mathrm{dist}}(t)$ is defined as

$$r_{\mathrm{dist}}(t)=-w_d{e}_d{(t)}^p+w_b{d}_{ij}(t), 0<p\le 1,$$

(18)

where $w_d$ is the error penalty weight, $p$ is a nonlinear exponent parameter, and $w_b$ is the weight of the distance compensation term. The $-w_d{e}_d{(t)}^p$ dominates the distance regulation process when the error $e_d(t)$ is large, driving the actual distance $d_{ij}$ to rapidly approach the target distance $d_{\mathrm{target}}$. As the error decreases, the penalty magnitude rapidly diminishes, thus avoiding excessive correction. This facilitates faster terminal convergence of the actual relative distance $d_{ij}$ toward the target distance $d_{t\arg et}$ and strengthens the stable locking capability around the target distance. The compensation term $w_b{d}_{ij}(t)$ is introduced to balance the distance regulation structure and to alleviate excessive compression of the inter-UGV distance during close-range cooperative motion. This term may slightly increase the inter-UGV separation. However, its influence remains limited due to the relatively small value assigned to $w_b$. The distance-maintenance objective is still dominated by the nonlinear error penalty term $-w_d{e}_d{(t)}^p$, which directly penalizes error from the target distance. Therefore, the compensation term does not conflict with the distance-keeping objective. Instead, it functions as an auxiliary regulation term that enhances safety and stability during close-range cooperative motion.

To prevent the minimum distance $d_{i,\mathrm{o}}^{\min }(t)$ between the UGV and obstacles from falling below the safety threshold $d_{\mathrm{safe}}$, the collision-avoidance penalty $r_{\mathrm{safe}}(t)$ is formulated as

$$r_{\mathrm{safe}}(t)=\left\{\begin{array}{ll}-w_{\mathrm{o}}(d_{\mathrm{safe}}-d_{i,\mathrm{o}}^{\min }(t)),& d_{i,\mathrm{o}}^{\min }(t)<d_{\mathrm{safe}},\\ 0,\hfill & \hfill d_{i,\mathrm{o}}^{\min }(t)\ge d_{\mathrm{safe}},\end{array}\right.$$

(19)

where $w_{\mathrm{o}}>0$ is the safe penalty weight. When $d_{i,\mathrm{o}}(t)<d_{\mathrm{safe}}$, the penalty severity is proportional to the degree of violation $d_{\mathrm{safe}}-d_{i,\mathrm{o}}^{\min }(t)$, thus providing a stable linear cost gradient until the safety constraint is satisfied. The parameter settings of the reward function are summarized in Table 2.

Table 2. Reward function parameter settings.

Parameter	Value
Goal reward weight $w_{\mathit{g}}$	2
Goal reward decay coefficient $k_{\mathit{g}}$	0.5
Distance error penalty weight $w_d$	1
Nonlinear distance error exponent $p$	0.6
Distance compensation weight $w_b$	0.1
Obstacle safety penalty weight $w_{\mathrm{o}}$	5

To illustrate the joint effect of distance regulation and safety constraints, the response surface $r(e_d,d_{i,\mathrm{o}}^{\min })$ of the combined terms in (18) and (19), evaluated at $d_{\mathrm{target}}=0.5 \mathrm{m}$ and $d_{\mathrm{safe}}=0.3 \mathrm{m}$, is depicted in Figure 6. In this figure, the X-axis denotes the minimum distance to the nearest obstacle $d_{i,\mathrm{o}}^{\min }(t)$, the Y-axis denotes the distance error $e_d(t)$, and the Z-axis denotes the reward value. As shown in Figure 6, when $d_{i,\mathrm{o}}^{\min }\ge d_{\mathrm{safe}}$, the response surface $r(e_d,d_{i,\mathrm{o}}^{\min })$ varies primarily along the distance error dimension. As $e_d$ decreases, the magnitude of the distance error penalty is progressively reduced, leading to a corresponding increase in the reward. When $d_{i,\mathrm{o}}^{\min }<d_{\mathrm{safe}}$, the surface declines along the obstacle clearance dimension, and the reward decreases continuously as the obstacle approaches. Overall, this response surface provides a more intuitive characterization of the joint effect of distance error and the degree of proximity to obstacles on the reward distribution.

Figure 6. Reward response surface with respect to relative distance error and obstacle clearance. The dotted line indicates the safety boundary where $d_{i,\mathrm{o}}^{\min }=d_{\mathrm{safe}}$.

5. Results

5.1. Experimental Environment Design and Parameter Configuration

All simulations are conducted on a laptop equipped with an Intel i7-13620H processor, RTX 4060 GPU, 16 GB RAM, and 1 TB SSD. The simulation environment is implemented in Python 3.8.18 based on the PyTorch 2.3.0 framework under Windows 10, with a fixed timestep of 0.1 s. The simulation scenario includes two UGVs, target positions and several static obstacles. The UGVs are represented by circular equivalent models with a radius of 0.1 m. Different colors are used to visually distinguish the elements: the two UGVs are shown in blue and green, respectively, and obstacles are depicted as gray circles. A representative environment layout is illustrated in Figure 7. The target distance is predefined as $d_{\mathrm{target}}$ = 0.5 m, representing the desired relative distance for cooperative carrying during trajectory planning. The kinematic, perception and safety constraints of dual UGVs are uniformly parameterized, and the corresponding values are summarized in Table 3.

Figure 7. A representative simulation layout for cooperative trajectory planning. The gray circles represent obstacles.

Table 3. Kinematic and perception constraint parameters of the dual-UGV.

Parameter	Value
Maximum velocity $v_{\max }$	0.2 m/s
Sensing range of the sensor $r_s$	0.8 m
Safety threshold for obstacle avoidance $d_{\mathrm{safe}}$	0.3 m
Distance tolerance threshold $d_{\mathrm{tol}}$	0.1 m

5.2. Training Experiments and Results

To evaluate convergence behavior and learning stability, comparative training is conducted for MRDWA-MADDPG, MATD3, MASAC, and the proposed method. To ensure a fair comparison, all learning-based methods are trained under the same simulation setting and evaluated using identical task configurations for each comparison, including the same initial positions, target positions, obstacle layouts, target inter-UGV distance, and task constraints. The kinematics, perception and safety-related parameters are consistent with Table 3. In addition, the common training budget and comparable training hyperparameters are uniformly set for all learning-based methods, as summarized in Table 4. For implementation, the hidden-layer sizes of the actor and critic networks are set to 256 and 256 for the learning-based methods, while the output layers and algorithm-specific modules are configured according to the corresponding algorithmic frameworks. Adam is adopted as the optimizer. Each method is independently trained using three random seeds, and the training curves are reported as the mean and standard deviation over the three runs. Algorithm-specific components are configured according to their original algorithmic frameworks, including entropy regulation for MASAC and the proposed method, and twin-critic estimation, delayed policy update, and target policy smoothing for MATD3.

Table 4. Common training parameter settings for the learning-based methods.

Parameter	Value
Maximum number of training episodes $M$	1000
Maximum number of timesteps per episode $T$	500
Replay buffer capacity $U$	100,000
Batch size $N_{\mathrm{batch}}$	256
Actor learning rate $l_a$	1 × 10−3
Critic learning rate $l_c$	3 × 10−3
Discount factor $\gamma$	0.9
Target network soft update coefficient $\tau$	0.01
Actor hidden-layer sizes	256, 256
Critic hidden-layer sizes	256, 256
Optimizer	Adam
Number of independent random seeds	3

Figure 8 presents the cumulative reward curves of the four learning-based methods. To better illustrate the early-stage convergence behavior, only the first 500 training episodes are shown. Each method is independently trained with three random seeds. The solid curves denote the mean cumulative rewards, and the shaded regions represent the corresponding standard deviations. Therefore, convergence performance and training stability are evaluated based on both the mean reward curves and the standard-deviation bands.

Figure 8. Comparison of cumulative rewards in a static obstacle environment. The solid curves denote the mean cumulative rewards obtained from three independent training runs, and the shaded regions indicate the corresponding standard deviations.

As shown in Figure 8, MRDWA-MADDPG converges much later, at approximately 350 episodes, and exhibits larger reward fluctuations during training, indicating weaker learning stability. The wider standard-deviation band observed during the reward-improvement phase indicates greater variability across different random seeds, suggesting lower policy-learning consistency. This is mainly because its deterministic policy has limited exploration capability for the dual-UGV cooperative carrying task. Compared with MRDWA-MADDPG, MATD3 enters the high-reward region earlier and exhibits relatively better convergence stability, suggesting that twin-critic estimation and delayed policy update help stabilize deterministic policy learning. MASAC reaches a relatively stable reward level after approximately 200 episodes. Its stochastic policy and entropy regularization improve exploration efficiency, but the fixed target entropy still lacks adaptive adjustment to the current cooperative state.

The proposed method reaches a relatively stable high-reward level at approximately 150–180 episodes, earlier than the other learning-based baselines. This faster and more stable convergence is mainly due to the adaptive target entropy regulation mechanism. Compared with baseline methods, the proposed method maintains a narrower standard-deviation band after reaching convergence, indicating better training consistency across different random seeds. When the relative distance error is large, higher entropy promotes broader exploration and accelerates escape from undesirable cooperative states. As the error decreases, lower entropy suppresses unnecessary randomness and reduces oscillatory updates. Therefore, the proposed method reaches the high-reward region earlier and maintains a more stable reward trend during the subsequent training process. These results demonstrate that the proposed method exhibits better training stability than the comparison methods.

5.3. Testing Experiments and Results

5.3.1. Validation Simulations in Different Test Environments

After training, tests are conducted in two different environments to further evaluate the effectiveness of the proposed method in trajectory planning tasks. In the first environment, the planned trajectories are shown in Figure 9, and Figure 10 displays the relative distance between the two UGVs. The results show that both UGVs are navigated smoothly around the obstacles without abrupt heading changes or unnecessary detours and safely reach their respective target positions within 264 decision timesteps. Moreover, the consistent color evolution along the two trajectories indicates good coordination in time. The relative distance is consistently maintained within the range of [0.488, 0.506] m, with the maximum error being less than 2.4% of the target distance, indicating that the method can effectively preserve the distance stability required for cooperative carrying.

Figure 9. Cooperative trajectory results of the proposed method in the first environment, with trajectory colors indicating temporal progression. The gray circles represent obstacles.

Figure 10. Distance results by the proposed method in the first environment. The solid line denotes the actual distance, and the dotted line indicates the target distance.

Figure 11 shows that the two UGVs exhibit highly consistent velocity profiles, rapidly reaching the predefined maximum velocity and only making slight synchronous adjustments during local obstacle avoidance, which demonstrates good coordination during trajectory execution. Overall, the results verify that the proposed method achieves safe obstacle avoidance and target reaching while maintaining distance and motion synchronization for cooperative carrying.

Figure 11. Velocity results by the proposed method in the first environment. The blue and green lines represent the results of UGV 1 and UGV 2, respectively.

Figure 12, Figure 13 and Figure 14 show the planned trajectories, relative distance, and velocity profiles in the second environment. The two UGVs successfully reach their respective target positions within 310 timesteps by considering obstacle avoidance and relative distance stability. Moreover, the color evolution along the two trajectories is highly consistent, indicating that the two UGVs maintain good temporal coordination during cooperative motion. The relative distance remains within the range of [0.491 m, 0.507 m], with the maximum error being less than 1.8% of the target distance. The velocity profile remains stable, with only slight fluctuations during obstacle avoidance, which further reflects stable motion coordination. Overall, the results in both environments demonstrate that the proposed method enables safe and coordinated cooperative carrying while maintaining a stable relative distance.

Figure 12. Cooperative trajectory results of the proposed method in the second environment, with trajectory colors indicating temporal progression. The gray circles represent obstacles.

Figure 13. Distance results by the proposed method in the second environment. The solid line denotes the actual distance, and the dotted line indicates the target distance.

Figure 14. Velocity results by the proposed method in the second environment. The blue and green lines represent the results of UGV 1 and UGV 2, respectively.

5.3.2. Comparative Simulations

To comprehensively evaluate the performance of the proposed method, comparative simulations are conducted by comparing SRM-APF, MRDWA-MADDPG, MATD3, MASAC, and the proposed method. For the learning-based methods, the trained policies are evaluated under identical testing scenarios to ensure a fair comparison. This subsection presents a comparative analysis of trajectory characteristics, relative-distance maintenance, and quantitative metrics.

Trajectory characteristics. Figure 15 presents the planned trajectories of the dual UGVs generated by different methods; Figure 15a–e correspond to SRM-APF, MRDWA-MADDPG, MATD3, MASAC, and the proposed method, respectively. As shown in Figure 15, all methods successfully guide the two UGVs to their respective targets without collision. However, noticeable differences can be observed in obstacle avoidance behavior, trajectory smoothness, and cooperative coordination.

Figure 15. Cooperative trajectories of the dual UGVs generated by different methods. The trajectory color denotes timestep progression. (a) SRM-APF; (b) MRDWA-MADDPG; (c) MATD3; (d) MASAC; (e) the proposed method. The gray circles represent obstacles.

In Figure 15a, the trajectories generated by SRM-APF exhibit noticeable local steering corrections in obstacle-dense regions, resulting in relatively large curvature variations. This is because the local resultant potential field may change abruptly when multiple nearby obstacles exert influence simultaneously, causing repeated heading adjustments during cooperative motion. As a result, the generated trajectories are more sensitive to local geometric interference. In Figure 15b, the trajectories generated by MRDWA-MADDPG exhibit evident local fluctuations and relatively weak overall smoothness. This behavior is mainly due to its deterministic strategy structure, which is more sensitive to local state disturbances during obstacle avoidance. Consequently, frequent control corrections are more likely to be triggered, increasing the probability of falling into suboptimal local motion patterns. As shown in Figure 15c, MATD3 generates smoother trajectories than MRDWA-MADDPG, with fewer abrupt local corrections. This indicates that the twin-critic and delayed-update mechanisms improve the stability of deterministic policy learning. However, slight terminal deviations and inconsistent motion trends can still be observed near the target region, suggesting that its cooperative coordination remains limited.

The trajectories in Figure 15d exhibit better smoothness than those in Figure 15b, although redundant detours are still observed. The improved smoothness is mainly attributed to the stochastic policy, which reduces sensitivity to local state variations and thus alleviates excessive local corrective maneuvers. However, because the entropy regulation is not explicitly adjusted according to the inter-UGV distance requirement, unnecessary exploratory deviations may still occur, leading to redundant path segments during cooperative carrying. By contrast, the trajectories generated by the proposed method in Figure 15e are smoother and contain fewer redundant detours and local corrective maneuvers. In addition, the two UGVs exhibit more consistent obstacle-avoidance behaviors, indicating better coordination during cooperative carrying. This is because the proposed adaptive entropy regulation mechanism can better suppress ineffective exploratory deviations while retaining sufficient flexibility for cooperative adjustment. Consequently, unnecessary local corrections are reduced, and the overall trajectory quality is improved.

Relative distance maintenance comparison. To further evaluate the relative distance maintenance performance of the representative trajectories in Figure 15, Figure 16 presents the corresponding relative distance curves. As shown in Figure 16, SRM-APF exhibits a persistent relative distance error. The inter-UGV distance is alternately below and above the target value at different stages, with occasional abrupt drops. This behavior is mainly attributed to its reliance on instantaneous regulation induced by the resultant potential field force, without an explicit feedback mechanism for the distance error. Therefore, stage deviation is likely to occur. MRDWA-MADDPG exhibits the most pronounced peaks and troughs and the highest oscillation frequency, with repeated rapid overshoots and fallbacks. This indicates high sensitivity to local disturbances, and deterministic policy updates tend to induce error overcorrection, thus degrading distance maintenance stability. Compared with MRDWA-MADDPG, MATD3 shows smaller overall fluctuations, but a noticeable distance peak still appears around the late stage, indicating that its relative-distance maintenance is improved but still not sufficiently stable.

Figure 16. Relative distance between the dual UGVs for the trajectories shown in Figure 15.

MASAC maintains the relative distance near the target value during most of the process, and its overall fluctuation is weaker than that of MRDWA-MADDPG. However, evident oscillation growth and consecutive deviations still appear in the later stage. This suggests that the stochastic policy and entropy regularization help to weaken the violent oscillation, but the fixed target entropy cannot adapt the exploration intensity to the current distance error. Therefore, sustained suppression of later-stage deviations remains limited. In contrast, the proposed method remains closest to the target distance throughout the entire process and exhibits the smallest fluctuation amplitude. This indicates that the distance-error-driven adaptive target entropy regulation mechanism can adaptively modulate the exploration intensity and improve distance maintenance stability, thereby providing more reliable relative distance regulation for dual-UGV cooperative carrying.

Quantitative metric comparison. The quantitative results in Table 5 further support the above observations. In addition to distance-error metrics, trajectory smoothness and path length are further introduced to evaluate trajectory quality and path efficiency. The path length is obtained by accumulating the displacements between consecutive trajectory points of the two UGVs over the entire motion process, which reflects the degree of path redundancy. The smoothness is computed based on the second-order differences in the trajectory position sequence, which characterizes motion continuity and local oscillatory behavior. A smaller value indicates a smoother trajectory.

Table 5. Distance-error, trajectory-quality, and efficiency metrics of different methods for the representative trajectories shown in Figure 15.

Methods	MaxDE (m)	MDE (m)	SDDE (m)	Timesteps	Smoothness	Path Length (m)
SRM-APF	0.0173	0.0052	0.0034	322	0.0002	11.8212
MRDWA-MADDPG	0.0224	0.0056	0.0044	344	0.0140	13.0886
MATD3	0.0231	0.0023	0.0027	315	0.0039	12.2552
MASAC	0.0167	0.0032	0.0039	401	0.0038	13.6933
Proposed method	0.0068	0.0019	0.0015	289	0.0025	11.4105

The proposed method achieves the lowest MaxDE, MDE, and SDDE, with values of 0.0068 m, 0.0019 m, and 0.0015 m, respectively. Compared with SRM-APF, MRDWA-MADDPG, MATD3, and MASAC, its MaxDE is reduced by 60.7%, 69.6%, 70.6%, and 59.3%, respectively. These results indicate that the proposed method achieves both higher distance regulation accuracy and stronger fluctuation suppression during cooperative execution. This advantage can be attributed to the combined effect of the adaptive entropy regulation and the nonlinear distance error penalty, which jointly reduce unnecessary exploratory deviations and enhance local locking around the target distance. As a result, the task is completed in 289 timesteps, which is the smallest value among all compared methods.

As shown in Table 5, the proposed method achieves the shortest path length of 11.4105 m, which is reduced by 3.47%, 12.84%, 6.89%, and 16.67% compared with SRM-APF, MRDWA-MADDPG, MATD3, and MASAC, respectively. This indicates that the proposed method can reduce redundant detours during cooperative motion. In terms of smoothness, the proposed method obtains a lower value than MRDWA-MADDPG, MATD3, and MASAC, indicating fewer local oscillatory adjustments and improved motion continuity. Although SRM-APF achieves the smallest smoothness value, its distance-error metrics are inferior to those of the proposed method. Therefore, the proposed method achieves a more balanced performance in path efficiency, motion continuity, and relative-distance maintenance.

Overall, the results in Figure 15 and Figure 16 and Table 5 consistently demonstrate that the proposed method provides safer and more coordinated cooperative trajectories, higher distance-maintenance accuracy, shorter path length and better stability than the comparison methods.

5.3.3. Statistical Simulations over 60 Test Scenarios

To further evaluate the success rates of different methods under various obstacle distributions, 60 independent tests are conducted for the proposed method, SRM-APF, MASAC, and MRDWA-MADDPG.

To ensure a fair comparison, the initial and target positions of the two UGVs are kept fixed across all test scenarios. Specifically, the initial positions of UGV1 and UGV2 are set to (−2.5, 0) and (−2.0, 0), respectively, while their target positions are set to (1.0, 4.5) and (1.5, 4.5), respectively. The variations among different test scenarios are introduced by randomly generating obstacle distributions within the predefined workspace. All methods are evaluated under the same set of obstacle configurations.

For SRM-APF, the attraction coefficient, repulsion coefficient, inter-UGV distance regulation coefficient, velocity damping coefficient, and obstacle sensing radius were tuned through preliminary experiments. The final parameter set used in this study is $k_{att}$ = 0.4, $k_{rep}$ = 0.6, $k_u$ = 0.5, $k_f$ = 0.5, and $l_{ob}$ = 0.8. These parameters were selected to achieve a balanced performance in target reaching, obstacle avoidance, and relative-distance maintenance. After being determined, the same parameter set is used across all test scenarios without scenario-specific tuning to ensure a fair comparison.

For the dual-UGV cooperative carrying task, this metric has direct physical significance because once the maximum distance error exceeds the prescribed threshold, the cooperative relationship may be disrupted and payload stability may be affected. Thus, the test is regarded as successful if both UGVs reach their target positions without collision and the maximum relative distance error over the whole trajectory satisfies $\max |d_{ij}(t)-d_{\mathrm{target}}|\le ϵ$. To assess robustness under different distance accuracy requirements, the tolerance value $ϵ$ is set to 0.01, 0.02, 0.03, and 0.04 m. Figure 17 presents the resulting success rates.

Figure 17. Success rates of the proposed method, SRM-APF, MASAC, and MRDWA-MADDPG under different distance error tolerance values $ϵ$ over 60 independent tests.

As shown in Figure 17, the success rates of all methods increase as the distance error tolerance $ϵ$ increases. The proposed method consistently achieves the highest success rate over all tolerance values. Its advantage is especially evident under strict tolerance settings. At $ϵ$ = 0.01 m, the proposed method achieves a success rate of 66.67%, exceeding the second-best method by 33.34 percentage points. When $ϵ \ge$ 0.03 m, the success rate of the proposed method stabilizes at 98.33%, demonstrating stronger robustness. This is due to the distance-error-driven adaptive target entropy regulation mechanism, which promotes recovery from large deviations and limits unnecessary exploration near the target distance, reducing violations of the maximum distance error bound.

Although MASAC achieves the second-best overall success rate, its performance remains noticeably inferior under strict tolerance conditions. At $ϵ$ = 0.01 m, its success rate is only 33.33%, and even when $ϵ$ = 0.03 m and 0.04 m, it reaches only 71.67% and 73.33%, respectively. This indicates that although the stochastic policy and entropy regularization improve robustness, the fixed target entropy still limits deviation suppression under strict distance requirements. In contrast, SRM-APF and MRDWA-MADDPG achieve relatively low success rates under small tolerances. At $ϵ$ = 0.01 m, their success rates are only 21.67% and 0.00%, respectively. This indicates the limited error regulation capability of SRM-APF and the disturbance sensitivity of MRDWA-MADDPG under deterministic policy updates.

Overall, these results indicate that the proposed method demonstrates strong robustness and reliability under different tolerance settings, particularly when strict relative-distance accuracy is required.

5.3.4. Ablation Study

To evaluate the contribution of each component in the proposed distance maintenance scheme for the dual-UGV cooperative carrying task, ablation experiments are conducted from the perspectives of observation design, reward design, and entropy regulation. Figure 18 presents representative trajectories under five settings, including the proposed method, No Inter-UGV Relative-State Obs., DistErrorPenalty-L, No DistErrorPenalty, and Fixed Target Entropy. Specifically, No Inter-UGV Relative-State Obs denotes the setting where the inter-UGV relative-state terms $d_{ij}$, ${\stackrel{\rightharpoonup }{\mathit{e}}}_{ij}$, and ${\stackrel{\rightharpoonup }{\mathit{v}}}_{ij}$ in Equation (12) are removed from the observation. DistErrorPenalty-L denotes the setting in which the original nonlinear distance-error penalty is replaced with a linear form, following the linear formation-error penalty formulation in Yan et al. [42]. No DistErrorPenalty denotes the setting in which the distance error penalty term is removed. Figure 19 shows the corresponding relative distance curves. Fixed Target Entropy denotes an ablation setting in which the proposed adaptive target entropy mechanism is replaced with a fixed target entropy, while the observation design and reward function remain the same as those of the proposed method. This setting is equivalent to the MASAC-based variant in which the proposed adaptive target entropy mechanism is disabled. Therefore, the representative trajectory and relative-distance curve of this setting are the same as the corresponding MASAC results in the comparative simulations. It is included in the ablation study to explicitly isolate the effect of the proposed distance-error-driven adaptive entropy regulation mechanism.

Figure 18. Representative cooperative trajectories of the dual UGVs under different ablation settings for the proposed distance maintenance mechanism. The trajectory color denotes timestep progression. (a) Proposed method; (b) No Inter-UGV Relative-State Obs.; (c) DistErrorPenalty-L; (d) No DistErrorPenalty; (e) Fixed Target Entropy. The gray circles represent obstacles.

Figure 19. Relative distance between the dual UGVs for the trajectories shown in Figure 18.

In addition, 60 independent test cases are conducted for each setting in different test environments, and the results are summarized in Table 6. A test is regarded as successful only when both UGVs reach their target regions and the maximum relative-distance error satisfies $\max |d_{ij}(t)-d_{\mathrm{target}}|\le 0.03 \mathrm{m}$. The success rate is calculated over all 60 test cases. For MaxDE, MDE, SDDE, and final timesteps, the statistics are calculated only over the successful test cases and are reported in the form of mean ± standard deviation. If no successful case is obtained, the corresponding statistical entries are denoted by “/”.

Table 6. Statistical results of different ablation settings over 60 independent tests.

Methods	Success Rate (%)	MaxDE (m)	MDE (m)	SDDE (m)	Final Timesteps
No Inter-UGV Relative-State Obs.	0.00	/	/	/	/
DistErrorPenalty-L	75.00	0.0184 ± 0.0053	0.0044 ± 0.0010	0.0037 ± 0.0009	300.62 ± 10.54
No DistErrorPenalty	0.00	/	/	/	/
Fixed Target Entropy	33.33	0.0146 ± 0.0045	0.0034 ± 0.0010	0.0030 ± 0.0009	362.90 ± 50.83
Proposed method	98.33	0.0100 ± 0.0046	0.0025 ± 0.0014	0.0020 ± 0.0010	300.95 ± 19.97

Trajectory analysis. As shown in Figure 18a, the proposed method generates coordinated trajectories that guide both UGVs to their target regions while maintaining good motion consistency. Compared with Figure 18a, the dual-UGV trajectories in Figure 18b exhibit noticeable local oscillations and repeated corrective motions near the initial region, failing to form a stable cooperative progression toward the target region. Without $d_{ij}$, ${\stackrel{\rightharpoonup }{\mathit{e}}}_{ij}$, and ${\stackrel{\rightharpoonup }{\mathit{v}}}_{ij}$, the policy cannot accurately capture the partner UGV’s motion state or its motion trend. As a result, the cooperative constraint in action selection becomes insufficient, which leads to aggravated local wandering, more ineffective adjustments, and difficulty in advancing effectively toward the target region.

Compared with Figure 18a, the trajectories in Figure 18c reach the target region but exhibit more repeated local adjustments during motion. This indicates that replacing the original nonlinear distance-error penalty with a linear form weakens the constraint on deviations near the target distance, thus reducing the stability of distance maintenance and cooperative motion. In Figure 18d, the distance-maintenance performance degrades further, with more evident local separation and compression between the two UGVs. Without the distance-error penalty, policy optimization is mainly driven by goal guidance and obstacle avoidance. Consequently, no explicit mechanism remains for regulating the cooperative distance. As a result, it becomes difficult to maintain a stable relative distance relationship throughout the motion process. Figure 18e shows that the Fixed Target Entropy setting can still guide the two UGVs toward the target positions, but the trajectories contain more frequent local adjustments than those generated by the proposed method. This result indicates that the fixed entropy target has limited adaptability to time-varying relative distance errors. In contrast, the proposed adaptive entropy mechanism adjusts the exploration level according to the distance-maintenance state, thus improving the consistency and stability of cooperative trajectory generation.

Relative distance analysis. As shown in Figure 19, the proposed method stays closest to the target distance and exhibits the smallest fluctuations throughout the motion, indicating better distance-maintenance stability. The Fixed Target Entropy setting maintains the relative distance near the target value during most timesteps. However, compared with the proposed method, it shows larger distance fluctuations as the motion process proceeds. This suggests that the fixed entropy target is insufficiently responsive to variations in the distance maintenance state during cooperative motion. In contrast, the proposed method achieves smaller distance fluctuations by dynamically adjusting the target entropy according to the inter-UGV distance error. Meanwhile, No Inter-UGV Relative-State Obs. shows the largest deviation and strongest fluctuations, with the relative distance quickly exceeding the target value and remaining unstable for a long period. This is consistent with the local wandering and ineffective coordination observed in Figure 18b, indicating that removing the inter-UGV relative-state observations weakens the policy’s ability to perceive and regulate the cooperative state online.

DistErrorPenalty-L still maintains the relative distance around the target value, but with slightly larger fluctuations and more persistent deviations than the proposed method. This indicates that the linear distance-error penalty imposes a weaker constraint around the target distance than the original nonlinear form. By contrast, No DistErrorPenalty exhibits a prolonged downward deviation, indicating degraded distance-maintenance capability once the distance-error penalty is removed.

Quantitative statistical analysis. Table 6 summarizes the results over 60 independent test cases. MaxDE, MDE, SDDE, and final timestep are reported in the form of mean ± standard deviation, while success rate is expressed as the percentage of successful cases. As shown in Table 6, the proposed method achieves a success rate of 98.33% and obtains the lowest values of MaxDE, MDE, and SDDE among all ablation settings. These results indicate strong task-completion capability, high relative-distance maintenance accuracy, and good cooperative stability. Although the proposed method does not achieve the lowest final timestep, its value is very close to that of DistErrorPenalty-L, indicating that the improvement in distance maintenance does not noticeably increase the task-completion time.

Removing the inter-UGV relative-state observations reduces the success rate to 0%, indicating that the policy fails to complete the cooperative transportation task under the prescribed distance-maintenance constraint. Since no successful case is obtained, MaxDE, MDE, SDDE, and final timesteps are not reported for this setting. This result demonstrates that inter-UGV relative-state observations are essential for perceiving the partner UGV’s motion state and maintaining cooperative distance regulation. For DistErrorPenalty-L, the final timestep remains close to that of the proposed method with smaller variation. However, the success rate decreases to 75.00%, and all distance-error metrics deteriorate. This suggests that the linear distance-error penalty weakens the local constraint around the target distance.

For No DistErrorPenalty, the success rate also drops to 0%, indicating that the policy can no longer satisfy the distance-maintenance requirement once the explicit distance-error penalty is removed. The UGVs can still be guided by the goal-reaching and obstacle-avoidance rewards. However, without an explicit distance error constraint, relative distance regulation becomes weaker. This makes it difficult to maintain the inter-UGV spacing required for cooperative carrying. For Fixed Target Entropy, the success rate decreases to 33.33%, and the average final timestep is much larger than that of the proposed method. Although the distance-error metrics remain relatively small for successful cases, the low success rate indicates limited robustness under fixed entropy regulation. In contrast, the proposed adaptive target entropy mechanism improves task-completion reliability and distance-maintenance stability by adjusting exploration according to the inter-UGV distance error.

Overall, the results in Figure 18 and Figure 19 and Table 6 indicate that inter-UGV relative-state observations, nonlinear distance-error penalty design, explicit distance-error penalties, and distance-error-driven adaptive entropy regulation are all important for relative-distance maintenance in dual-UGV cooperative carrying. Compared with the fixed target entropy setting, the proposed adaptive entropy mechanism improves task-completion reliability and reduces distance error fluctuations, demonstrating its effectiveness in cooperative constraint satisfaction.

5.3.5. Evaluation in Dynamic Obstacle Environments

To further evaluate the environmental adaptability and robustness of the proposed method under time-varying conditions, additional simulations are conducted in dynamic obstacle environments. Compared with the static scenarios used in the previous subsections, dynamic obstacles continuously change the local feasible motion space and introduce additional disturbances to cooperative trajectory planning. Therefore, these scenarios impose higher requirements on online trajectory adjustment, collision avoidance, and inter-UGV distance maintenance. In this subsection, two representative dynamic scenarios are considered: a low-density dynamic obstacle environment and a high-density dynamic obstacle environment. These scenarios are used to examine whether the proposed method can generate safe cooperative trajectories while maintaining the desired relative distance.

In the low-density dynamic obstacle environment, Figure 20 presents cooperative motion snapshots of the dual UGVs at steps 50, 150, 250, and the final step. Figure 21 and Figure 22 show the relative-distance and velocity profiles of the two UGVs, respectively. The results show that both UGVs can adjust their trajectories according to the motion of dynamic obstacles and safely reach their respective target positions within 291 decision timesteps. Moreover, the consistent color evolution along the two trajectories indicates good temporal coordination during cooperative motion. The relative distance is maintained within approximately [0.496, 0.504] m, corresponding to a maximum deviation of about 0.004 m, which is less than 0.8% of the target distance. This result indicates that the proposed method achieves accurate distance regulation under dynamic obstacle interference. As shown in Figure 22, the two UGVs exhibit similar velocity trends, with only slight coordinated adjustments during local obstacle avoidance and target approaching.

Figure 20. Cooperative motion process of the dual UGVs in a low-density dynamic obstacle environment. The gray circles represent obstacles. The blue and green circles represent the positions of UGV 1 and UGV 2, respectively.

Figure 21. Relative distance curves of dual UGVs in a low-density dynamic obstacle environment. The solid line denotes the actual distance, and the dotted line indicates the target distance.

Figure 22. Velocity curves of the dual UGVs in a low-density dynamic obstacle environment. The blue and green lines represent the results of UGV 1 and UGV 2, respectively.

In the high-density dynamic-obstacle environment, Figure 23 presents the cooperative motion snapshots of the dual UGVs at representative timesteps, and Figure 24 and Figure 25 display the relative distance and velocity profiles of the two UGVs, respectively. Compared with the low-density case, the high-density environment contains more dynamic obstacles, which increases the difficulty of local obstacle avoidance and cooperative trajectory adjustment. The results show that both UGVs can still avoid collisions and safely reach their respective target positions within 306 decision timesteps. Although more local trajectory adjustments are observed due to the increased obstacle density, the two UGVs maintain coordinated motion during the whole process. The relative distance is maintained within the range of approximately [0.494, 0.508] m, with the maximum error being less than 1.6% of the target distance. This indicates that the proposed method can maintain stable relative distance regulation in more challenging dynamic obstacle environments. As shown in Figure 25, the two UGVs exhibit generally consistent velocity trends, with more noticeable speed adjustments during local obstacle avoidance.

Figure 23. Cooperative motion process in a high-density dynamic obstacle environment. The gray circles represent obstacles. The blue and green circles represent the positions of UGV 1 and UGV 2, respectively.

Figure 24. Relative distance curves of dual UGVs in a high-density dynamic obstacle environment. The solid line denotes the actual distance, and the dotted line indicates the target distance.

Figure 25. Velocity curves of the dual UGVs in a high-density dynamic obstacle environment. The blue and green lines represent the results of UGV 1 and UGV 2, respectively.

Overall, the results in Figure 20, Figure 21, Figure 22, Figure 23, Figure 24 and Figure 25 demonstrate that the proposed method can generate feasible cooperative trajectories in both low-density and high-density dynamic obstacle environments while maintaining stable inter-UGV distance regulation.

6. Conclusions

In this paper, a reinforcement learning-based cooperative trajectory planning method with adaptive target entropy regulation is proposed for dual-UGV cooperative carrying under distance constraints. To address the strong motion coupling and reduced feasible solution space caused by the shared payload, a MASAC-based cooperative learning framework is established under the CTDE paradigm. Within this framework, a distance-error-driven adaptive target entropy regulation mechanism is introduced to adjust the exploration intensity according to the current distance error. Meanwhile, a task-oriented local observation representation and a structured reward design method are developed to jointly account for goal guidance, distance maintenance, and obstacle avoidance. These designs improve the consistency among state representation, learning signals, and cooperative control objectives.

Training results show that the proposed method achieves faster convergence and greater training stability in dual-UGV cooperative carrying tasks. Validation results demonstrate that the proposed method can generate safe and coordinated trajectories while keeping the maximum relative distance error below 2.4% and 1.8% of the target distance in the two test environments, respectively. Comparative results further demonstrate that the proposed method generates smoother cooperative trajectories and completes the task in fewer timesteps than the baseline methods. It also achieves superior relative-distance maintenance accuracy and stability. Multi-scenario statistical results further indicate that the proposed method achieves consistently higher success rates under different distance error tolerances. In particular, it outperforms the second-best method by 33.34 percentage points at 0.01 m and reaches 98.33% at 0.03 m, thereby confirming its strong robustness for cooperative carrying tasks. In addition, the ablation results indicate the effectiveness of the adaptive target entropy mechanism. These results also demonstrate that inter-UGV relative state observations and a nonlinear distance error penalty improve distance maintenance stability and cooperative robustness.

Although the proposed method achieves effective cooperative trajectory planning for dual-UGV cooperative carrying, several issues remain to be further investigated. First, the current action space is designed as a high-level planning variable. It does not explicitly consider the nonholonomic kinematic constraints of differential-drive or Ackermann-steering UGVs. Future work will incorporate realistic vehicle kinematics and lower-level tracking controllers to improve engineering applicability. Second, the physical coupling caused by the carried object is simplified as a prescribed inter-UGV spacing requirement. Future work will further model payload dynamics and mechanical coupling between the UGVs and the carried object. Third, the current validation is mainly conducted in a Python-based two-dimensional simulation environment. Future work will evaluate the proposed method on standard simulation platforms, such as ROS/Gazebo or physics-based simulators. Sensor noise, perception uncertainty, communication delay, actuator constraints, and tracking errors will also be considered. Finally, sim-to-real transfer will be further investigated through domain randomization, noise injection, and online policy adaptation. The extension to multi-UGV cooperative carrying will also be explored in future work.