Direct X-Rudder Path-Following Control for Underactuated AUVs via TIB-CSAC

Abstract

To improve the path-following performance of an underactuated autonomous underwater vehicle (AUV) under varying path geometries and desired velocities, this study proposes a direct X-rudder control method based on Task-Informed Inductive-Bias Conservative Soft Actor–Critic (TIB-CSAC). The proposed method directly learns the X-rudder control policy from the path-following information of the current and subsequent path segments in a data-driven way, thereby avoiding the complex design and manual tuning of guidance laws and attitude controllers for rudder command generation. To support such two-segment policy learning, a task-informed inductive-bias encoder is proposed to construct structured and conditioned state representations, thereby improving sample efficiency and overall training quality. In addition, given the long-tail characteristics of task difficulty in agent training, a multi-head conservative value evaluation mechanism is incorporated to mitigate return drawdowns induced by challenging tasks in the tail stage of training and to enhance tail-stage convergence stability. The path-following performance is validated in three representative scenarios with different path pitch, path heading variations, and desired surge velocity conditions. The results show that, compared with the baseline soft actor–critic (SAC) method, TIB-CSAC improves multiple vertical and horizontal error metrics, including maximum absolute error, mean absolute error, tail error, and error threshold exceedance ratio. These results indicate that TIB-CSAC not only improves overall adherence to the reference path, but also more effectively suppresses extreme errors and tail errors, thereby demonstrating stronger path-following robustness and reliability.

1. Introduction

Owing to their simple structure and high maneuvering efficiency, X-rudder autonomous underwater vehicles (AUVs) have become important platforms for a wide range of underwater applications, including environmental monitoring, offshore infrastructure inspection, and marine resource exploration. As underwater missions become increasingly diverse, AUVs are expected to exhibit strong generalization capability and reliable performance under a broad range of operating conditions.

Path-following is a fundamental capability of AUVs. It enables an AUV to move along a reference path while minimizing its position error. In practice, the reference path is commonly composed of multiple straight-line segments, and the path-following task can be characterized by parameters such as the path heading angle, path pitch angle, and desired surge velocity. However, achieving path-following performance that is both robust and strongly generalizable remains challenging. First, different task-parameter settings lead to substantial variations in task difficulty and impose different maneuvering demands. This challenge is particularly pronounced during path-segment transitions, where the AUV must rapidly adjust its motion and reduce tracking errors within a limited spatial range; these transitions are therefore more difficult than straight-segment following. Moreover, the underactuated nature of X-rudder AUVs, nonlinear hydrodynamic effects, and coupling among multiple motion variables further increase the complexity of unified controller design. As the task-generalization range expands, the workload and complexity associated with multi-parameter tuning also increase.

Due to the absence of direct lateral and vertical actuation in underactuated AUVs, conventional path-following methods typically use a guidance law to transform position errors into desired attitudes. These commands are then tracked by a controller, such as prescribed performance control [1], sliding-mode control [2,3], or model predictive control [4]. However, such model-based methods rely on accurate hydrodynamic models, and parameter identification is difficult and costly. Model-free controllers, such as proportional–integral–derivative (PID) controllers [5,6] and S-plane controllers [7,8,9], are easier to implement but usually rely on expert tuning. Moreover, under significant variations in path geometry and desired surge velocity, these hierarchical schemes often require re-tuning of the guidance law or controller [10], which limits their adaptability and generalization capability.

Deep reinforcement learning (DRL), with its self-learning capability and data-driven decision-making mechanism, provides a promising approach to path-following [5,11]. Sun et al. [12] proposed an improved deep deterministic policy gradient method (OSAM-DDPG) that learns a control policy conditioned on attitude and velocity errors. However, when a line-of-sight (LOS) guidance law with fixed parameters is used, its path-following performance under different operating conditions may be limited. Fang et al. [13] applied DDPG to achieve three-degree-of-freedom control at constant speed with variable pitch and yaw, while realizing path-following via waypoint tracking. Wang et al. [14] developed a simplified DDPG method that does not require prediction of future rewards and validated its performance on a helical path and an inclined straight path under a single constant desired speed. Wang et al. [15] proposed a SAC-based adaptive PID controller and verified its attitude-control performance in LOS-guided path-following, although its effectiveness in direct position-error regulation remains to be further investigated. Zhang et al. [16] proposed an improved DDPG-based path-following method integrated with a LOS guidance law and trained the algorithm on two fixed paths under a single constant desired speed. Zhang et al. [17] proposed a deep interactive reinforcement learning method, and Dong et al. [18] developed an improved SAC-based two-dimensional path-following method. In both studies, the agent was trained and evaluated using the same path geometries; therefore, its path-following performance across diverse path geometries remains an open issue.

Despite these advances, current DRL-based path-following studies still have limitations in state construction when strong generalization is required. On the one hand, many DRL methods rely on manually designed guidance laws with fixed parameters, making it difficult to adapt to path-following tasks with different parameter configurations. On the other hand, multiple types of original path-following information, including position errors, path heading/pitch variations, and progress relative to adjacent segments, are usually mapped by the guidance law into a small number of desired attitude commands [19]. If the agent mainly receives these attitude commands as inputs, task-level contextual information may not be explicitly represented in the agent state. Such local and indirect state representations may create a representation bottleneck, limiting the agent’s conditional learning capability across operating conditions and weakening its generalization performance [20,21].

Existing path-following studies also have limitations in agent training. Some studies restrict agent training to basic control tasks involving attitude and velocity errors, and some methods are trained under only a single surge-speed setting or a single path-geometry condition. Such limited coverage of the training-task distribution can easily lead to policy overfitting, thereby degrading path-following performance in other tasks. In domain-randomized training, challenging operating conditions, in which task parameters such as the path heading angle, path pitch angle, and desired surge velocity simultaneously approach their boundary values, occur with relatively low probability. Meanwhile, compared with straight-segment following, path-segment transitions are more difficult and are associated with sparser experience samples. As a result, task difficulty spans a wide range under domain-randomized training and exhibits a pronounced long-tail distribution. However, existing methods such as DDPG and SAC are essentially risk-neutral expected-return optimization methods. Although they may achieve satisfactory average performance under most regular operating conditions, their path-following performance under more challenging tasks may remain limited.

To address the limited generalization capability and robustness of underactuated X-rudder AUVs under diverse path-following conditions, this study proposes a direct X-rudder control method based on Task-Informed Inductive-Bias Conservative Soft Actor–Critic (TIB-CSAC) for AUV path-following. The main contributions are summarized as follows:

(1)

A direct X-rudder control framework is proposed. In this framework, the X-rudder control policy is learned directly from two-segment path-following information, avoiding the complex hierarchical design of guidance laws and attitude controllers as well as the associated coupled parameter tuning.

(2)

A task-informed inductive-bias state encoder is developed to construct structured and task-conditioned representations of two-segment path-following states for direct policy learning. This design reduces the representation burden caused by direct concatenation of heterogeneous state variables and improves sample efficiency.

(3)

A multi-head conservative value-evaluation mechanism is introduced to improve value assessment for high-uncertainty state–action pairs, thereby reducing return drawdowns induced by challenging path-following tasks and improving tail-stage convergence stability.

(4)

The proposed method is validated in three representative scenarios with distinct path geometries and desired surge velocities. The results show improvements across multiple error metrics, including the maximum absolute error, mean absolute error, tail error, and threshold-exceedance proportion, demonstrating enhanced generalization capability and path-following performance under diverse operating conditions.

The remainder of this paper is organized as follows. Section 2 presents the problem statement. Section 3 describes the proposed method. Section 4 analyzes and discusses the experimental results, and Section 5 concludes the paper and outlines future work.

2. Problem Statement

2.1. Underactuated AUV Model

In this study, the AUV is equipped with a main thruster and an X-rudder configuration. The numbering and rotation directions of the X-rudders, as well as the Earth-fixed (O-XYZ) and body-fixed (O_B-X_BY_BZ_B) coordinate systems, are shown in Figure 1. The kinematic and dynamic model of the AUV is given by [22]

$$\dot{\mathit{\eta }}=\mathit{J}(\mathit{\eta })\mathit{\nu }$$

(1)

$$\mathit{M}\dot{\mathit{\nu }}+\mathit{C}(\mathit{\nu })\mathit{\nu }+\mathit{D}(\mathit{\nu })\mathit{\nu }+\mathit{g}(\mathit{\eta })={\mathit{\tau }}^a+{\mathit{\tau }}^d$$

(2)

where J, M, C, D, and g represent the kinematic transformation matrix, inertia matrix, Coriolis–centripetal matrix, hydrodynamic damping matrix, and restoring-force vector, respectively. η = [x, y, z, θ, ψ]^T and ν = [u, v, w, q, r]^T denote the position-attitude vector in the Earth-fixed frame and the velocity vector in the body-fixed frame, respectively.

In practical AUV design, roll motion is usually expected to remain close to zero to ensure sensor and payload stability. This condition can be maintained passively by the hydrostatic restoring moment and hydrodynamic roll damping. Therefore, under the assumptions that gravity and buoyancy are in equilibrium, the center of buoyancy lies in the vertical plane, and sufficient hydrostatic restoring moment is available for near-zero roll motion, the kinematic and dynamic model can be written as [23,24]

$$\left\{\begin{array}{l}\dot{x}=u\cos (\psi )\cos (\theta )-v\sin (\psi )+w\cos (\psi )\sin (\theta )\\ \dot{y}=u\sin (\psi )\cos (\theta )+v\cos (\psi )+w\sin (\psi )\sin (\theta )\\ \dot{z}=-u\sin (\theta )+w\cos (\theta )\\ \dot{\theta }=q\\ \dot{\psi }=r/\cos (\theta )\end{array}\right.$$

(3)

$$\left\{\begin{array}{l}\dot{u}=(m_{22}vr-m_{33}wq-X_{u}u-X_{u\left|u\right|}u|u|+{\tau }_u^a+{\tau }_u^d)/m_{11}\\ \dot{v}=(-m_{11}ur-Y_{v}v-Y_{v\left|v\right|}v|v|+{\tau }_v^d)/m_{22}\\ \dot{w}=(m_{11}uq-Z_{w}w-Z_{w\left|w\right|}w|w|+{\tau }_w^d)/m_{33}\\ \dot{q}=((m_{33}-m_{11})uw-M_{q}q-M_{q\left|q\right|}q|q|-{\overline{BG}}_{z}W\sin \theta +{\tau }_q^a+{\tau }_q^d)/m_{55}\\ \dot{r}=((m_{11}-m_{22})uv-N_{r}r-N_{r\left|r\right|}r|r|+{\tau }_r^a+{\tau }_r^d)/m_{66}\end{array}\right.$$

(4)

where W is the AUV weight, and ${\overline{BG}}_z$ is the vertical offset between the centers of mass and buoyancy. $m_{(\cdot )}$ denotes the diagonal inertia terms of the mass matrix M, including added mass. ($X_u$, $Y_v$, $Z_w$, $M_q$, $N_r$) and ($X_{u\left|u\right|}$, $Y_{v\left|v\right|}$, $Z_{w\left|w\right|}$, $M_{q\left|q\right|}$, $N_{r\left|r\right|}$) represents linear and nonlinear damping coefficient, respectively. ${\mathit{\tau }}^d={[{\tau }_u^d,{\tau }_v^d,{\tau }_w^d,{\tau }_q^d,{\tau }_r^d]}^{\mathrm{T}}$ is the environmental disturbance. ${\mathit{\tau }}^a={[{\tau }_u^a,0,0,{\tau }_q^a,{\tau }_r^a]}^{\mathrm{T}}$ is the actuator input vector, where ${\tau }_u^a$ is the thruster input, and ${\tau }_q^a$ and ${\tau }_r^a$ are the X-rudder inputs for pitch and yaw motion, respectively.

Unlike a conventional cross-rudder configuration, in which the horizontal and vertical rudders mainly generate pitch and yaw moments separately, the X-rudder contains four inclined control surfaces. As shown in Equation (5), each rudder contributes to both ${\tau }_q^a$ and ${\tau }_r^a$; therefore, the pitch and yaw moments are generated by the combined deflections of all four rudders. This configuration improves rudder-utilization efficiency but also introduces a more complex rudder-moment coupling relationship.

$$\begin{array}{l}{\tau }_q^a=u^2(M_{{\delta }_1}{\delta }_1+M_{{\delta }_2}{\delta }_2+M_{{\delta }_3}{\delta }_3+M_{{\delta }_4}{\delta }_4)\\ {\tau }_r^a=u^2(N_{{\delta }_1}{\delta }_1+N_{{\delta }_2}{\delta }_2+N_{{\delta }_3}{\delta }_3+N_{{\delta }_4}{\delta }_4)\end{array}$$

(5)

where ${\mathit{M}}_{\delta }$ and ${\mathit{N}}_{\delta }$ are the hydrodynamic parameters of the rudder.

2.2. Path Following Objective

The desired path is represented as a polyline of n^p straight-line segments, where the i-th segment with endpoints ${\mathit{P}}_{i-1}^p$ and ${\mathit{P}}_i^p$ (i = 1, 2,…, n^p) is defined by Equation (6).

$$\begin{array}{cc}{\mathit{P}}_i^p=\left[\begin{array}{c}{x}_i^p\\ y_i^p\\ z_i^p\end{array}\right]={\mathit{P}}_{i-1}^p+L_i^p\left[\begin{array}{c}\cos {\psi }_i^p\cos {\theta }_i^p\\ \sin {\psi }_i^p\cos {\theta }_i^p\\ -\sin {\theta }_i^p\end{array}\right]& ,\begin{array}{c}\left|{\psi }_i^p-{\psi }_{i-1}^p\right|\le \mathrm{\Delta }{\psi }_{\max }^p\\ \left|{\theta }_i^p\right|\le {\theta }_{\max }^p\\ L_i^p\ge L_{\min }\end{array}\end{array}$$

(6)

where $\mathrm{\Delta }{\psi }_{\max }^p$ denotes the maximum variation between adjacent path headings ${\psi }_i^p$, and ${\psi }_0^p$ is set to the initial yaw angle of the AUV. ${\theta }_{\max }^p$ represents the maximum absolute value of the path pitch angle ${\theta }_i^p$, and L_min is the minimum path-segment length $L_i^p$.

The vertical error is defined as the perpendicular distance from the AUV to the projection point on the path segment:

$$e_i^v=(z_i^p-z_{i-1}^p){\epsilon }_i+z_{i-1}^p-z$$

(7)

$${\epsilon }_i=(\overrightarrow{{\mathit{P}}_{i-1}^p{\mathit{P}}_{\mathrm{AUV}}}\cdot \overrightarrow{{\mathit{P}}_{i-1}^p{\mathit{P}}_i^p})/{\left|{\mathit{P}}_{i-1}^p{\mathit{P}}_i^p\right|}^2$$

(8)

where ${\epsilon }_i$ is an along-track parameter based on the AUV’s position P_AUV = [x, y, z]^T relative to the path segment ${\mathit{P}}_{i-1}^p{\mathit{P}}_i^p$.

The horizontal error is defined as the horizontal offset with the minimum absolute value among the two adjacent path segments.

$$e_i^h=\left\{\begin{array}{l}{d}_k^h,k=\underset{j\in \{i,i+1\}}{\arg \min }\left|d_j^h\right| \mathrm{if}i<n^p\\ d_{n^p}^h\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \mathrm{if}i=n^p\end{array}\right.$$

(9)

$$d_i^h=\overrightarrow{{\mathit{P}}_{i-1}^h{\mathit{P}}_{\mathrm{AUV}}^h}\times \overrightarrow{{\mathit{P}}_{i-1}^h{\mathit{P}}_i^h}/\left|{\mathit{P}}_{i-1}^h{\mathit{P}}_i^h\right|$$

(10)

where $d_i^h$ denotes the horizontal distance from the AUV to the i-th path segment; and ${\mathit{P}}_i^h={[x_i^p,y_i^p]}^{\mathrm{T}}$ and ${\mathit{P}}_{\mathrm{AUV}}^h$ denote the horizontal position.

In summary, this study aims to minimize both the horizontal and vertical position errors, as shown in Equation (11).

$$\min J={\displaystyle \sum _t(\left|e_{i^e}^h(t)\right|+\left|e_{i^e}^v(t)\right|)}$$

(11)

$$i^e\leftarrow i^e+1, \mathrm{if} {\epsilon }_{i^e}>1$$

(12)

where i^e denotes the path-segment index used to compute the position error, and ${\epsilon }_{i^e}$ is the along-track parameter in Equation (8). The index i^e is initialized to one and is incremented when ${\epsilon }_{i^e}>1$, indicating that the AUV has passed the endpoint of the current segment.

3. X-Rudder Path Following Control Based on TIB-CSAC

3.1. Markov Decision Process Design

3.1.1. States

Let i^o denote the index used for state observation. As shown in Equations (13) and (14), i^o is initialized to one and is incremented when the horizontal error relative to the next target segment remains within a prescribed threshold for a duration $t_{\min }^o$, thereby preserving state continuity during path transitions. To avoid excessive lag behind i^e, i^o is also forcibly incremented when $i^e-i^o>2$. After each update of i^o, t^o is reset to zero.

$$i^o\leftarrow i^o+1, \mathrm{if} t^o\ge t_{\min }^o \mathrm{or} i^e-i^o>2$$

(13)

$$t^o=\left\{\begin{array}{l}{t}^o+\mathrm{\Delta }t \mathrm{if} \left|d_{i^o+1}^h\right|\le d_{\min }^h\\ 0\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{otherwise}\end{array}\right.$$

(14)

The overall observation state vector consists of four components:

$$\mathit{s}= {[{\mathit{s}}_{\mathrm{dyn}}^{\mathrm{T}},{\mathit{s}}_{\mathrm{task}}^{\mathrm{T}},{\mathit{s}}_{i^o}^{\mathrm{T}},{\mathit{s}}_{i^o+1}^{\mathrm{T}}]}^{\mathrm{T}}$$

(15)

where ${\mathit{s}}_{\mathrm{dyn}}$ denotes the general dynamic state describing the current motion and actuator conditions of the AUV.

$${\mathit{s}}_{\mathrm{dyn}}={[{\mathit{\nu }}^{\mathrm{T}},\dot{\psi },{\mathit{\delta }}^{\mathrm{T}}]}^{\mathrm{T}}$$

(16)

where ν = [u, v, w, q, r]^T denotes the velocity vector, and δ = [δ₁, δ₂, δ₃, δ₄]^T denotes the X-rudder-angle vector. ${\mathit{s}}_{\mathrm{task}}$ is the task-level state that provides information on the desired surge velocity, path-geometry variations, and path-following progress:

$${\mathit{s}}_{\mathrm{task}}={[u,\mathrm{\Delta }{\psi }_{i^o}^p,\mathrm{\Delta }{\theta }_{i^o}^p,d_{i^o}^p,{d_{i^o}^p}^{\prime }]}^{\mathrm{T}}$$

(17)

where $\mathrm{\Delta }{\theta }_{i^o}^p={\theta }_{i^o+1}^p-{\theta }_{i^o}^p$ and $\mathrm{\Delta }{\psi }_{i^o}^p={\psi }_{i^o+1}^p-{\psi }_{i^o}^p$ denote the changes in path pitch and path heading, respectively. $d_{i^o}^p$ and ${d_{i^o}^p}^{\prime }$ are the distances from the AUV projection points on segments ${\mathit{P}}_{i^o-1}^p{\mathit{P}}_{i^o}^p$ and ${\mathit{P}}_{i^o}^p{\mathit{P}}_{i^o+1}^p$ to waypoint ${\mathit{P}}_{i^o}^p$, encoding the progress of the AUV along the two adjacent segments.

$$d_{i^o}^p=(\overrightarrow{{\mathit{P}}_{\mathrm{AUV}}{\mathit{P}}_{i^o}^p}\cdot \overrightarrow{{\mathit{P}}_{i^o-1}^p{\mathit{P}}_{i^o}^p})/\left|{\mathit{P}}_{i^o-1}^p{\mathit{P}}_{i^o}^p\right|$$

(18)

$${d_{i^o}^p}^{\prime }=(\overrightarrow{{\mathit{P}}_{i^o}^p{\mathit{P}}_{\mathrm{AUV}}}\cdot \overrightarrow{{\mathit{P}}_{i^o}^p{\mathit{P}}_{i^o+1}^p})/\left|{\mathit{P}}_{i^o}^p{\mathit{P}}_{i^o+1}^p\right|$$

(19)

${\mathit{s}}_i$ (i = i^o, i^o + 1) denotes the segment-specific state that describes the relative path-following error for the current or subsequent path segment. It provides preview information for the upcoming transition and reduces abrupt changes in the policy input.

$${\mathit{s}}_i=[\mathrm{\Delta }{\theta }_i^R,e_i^v,{\dot{e}}_i^v,\mathrm{\Delta }{\psi }_i^R,d_i^h,{\dot{d}}_i^h]$$

(20)

where $\mathrm{\Delta }{\theta }_i^R={\theta }_i^p-\theta$ and $\mathrm{\Delta }{\psi }_i^R={\psi }_i^R-\psi$ represent the relative pitch and yaw angles with respect to segment ${\mathit{P}}_{i-1}^p{\mathit{P}}_i^p$, respectively. $e_i^v$ and ${\dot{e}}_i^v$ are the vertical error defined in Equation (7) and its rate, respectively; and $d_i^h$ and ${\dot{d}}_i^h$ are the horizontal distance defined in Equation (10) and its rate, respectively.

To avoid index overflow when the AUV reaches the final segment, a virtual path point ${\mathit{P}}_{n^p+1}^p$ is appended so that the extended segment ${\mathit{P}}_{n^p}^p{\mathit{P}}_{n^p+1}^p$ maintains the same geometry as ${\mathit{P}}_{n^p-1}^p{\mathit{P}}_{n^p}^p$, with $L_{n^p+1}^p=L_{n^p}^p$, ${\psi }_{n^p+1}^p={\psi }_{n^p}^p$, and ${\theta }_{n^p+1}^p={\theta }_{n^p}^p$.

3.1.2. Actions

The action vector ${\mathit{a}}_t=[\mathrm{\Delta }{\delta }_t^{\theta },\mathrm{\Delta }{\delta }_t^{\psi }]\in {[-1,1]}^2$ is defined as the normalized pitch and yaw control increments at each time step. These increments are mapped to X-rudder angles according to Equation (21). The X-rudder design enables all four rudders to jointly generate pitch and yaw moments, reducing the risk of single-rudder saturation and increasing the available equivalent moment under the rudder-angle limit.

$${\mathit{\delta }}_t=\left[\begin{array}{c}{\delta }_{1,t}\\ {\delta }_{2,t}\\ {\delta }_{3,t}\\ {\delta }_{4,t}\end{array}\right]=\mathrm{\Delta }{\delta }_{\max }\left[\begin{array}{cc}+1& -1\\ -1& -1\\ -1& +1\\ +1& +1\end{array}\right]\left[\begin{array}{c}\mathrm{\Delta }{\delta }_t^{\theta }\\ \mathrm{\Delta }{\delta }_t^{\psi }\end{array}\right]+{\mathit{\delta }}_{t-1},\left|\mathit{\delta }\right|\le {\delta }_{\max }$$

(21)

where Δδ_max and δ_max denote the maximum rudder-angle change per time step and the maximum rudder angle, respectively.

3.1.3. Reward

The reward $r_{\mathrm{RL}}$ consists of two position-error penalties and a velocity penalty that prevents the AUV from moving opposite to the tangent vector ${\mathit{t}}^p$ of the desired path. In addition, the reward terms are normalized to reduce the influence of scale differences among components, preventing the learning process from being dominated by a single error term.

$$r_{RL}=-k_1^r\mathrm{sat}(\left|e_{i^e}^h/e_{\max }^h\right|)-k_2^r\mathrm{sat}(\left|e_{i^e}^v/e_{\max }^v\right|)+k_3^r\mathrm{sat}(r_{i^e}^v/r_{\max }^v)$$

(22)

$$r_{i^e}^v=\min (0,[\dot{x},\dot{y},\dot{z}]\cdot {\mathit{t}}^p)$$

(23)

$${\mathit{t}}^p=[\cos ({\psi }_{i^e}^p)\cos ({\theta }_{i^e}^p),\sin ({\psi }_{i^e}^p)\cos ({\theta }_{i^e}^p),-\sin ({\theta }_{i^e}^p)]$$

(24)

where $k_i^r$ (i = 1, 2, 3) denotes the weighting coefficients of the reward terms, and $e_{\max }^h$, $e_{\max }^v$, and $r_{\max }^v$ denote the corresponding normalization factors. $\mathrm{sat}(x)=\min (1,\max (-1,x))$.

3.1.4. Termination

The episode terminates when $i^e=n^p+1$, indicating that the AUV has passed the endpoint of the final path segment, or when the time exceeds the maximum limit $t_{\max }^t$.

$$d_t=\left\{\begin{array}{l}\mathrm{True},i^e=n^p+1\\ \mathrm{True},t>t_{\max }^t\end{array}\right.$$

(25)

3.2. Task-Informed Inductive-Bias Conservative SAC Algorithm

3.2.1. Task-Informed Inductive-Bias Encoder

In this study, the input state vector consists of dynamic states, segment-specific path-following states, and task-level contextual states, which play distinct functional roles in decision making. Most DRL methods typically flatten and concatenate the state variables into a single unstructured vector and process them using a shared multilayer perceptron (MLP). This approach relies on implicit representation learning to recover the latent structure during training. When the state variables are high-dimensional and heterogeneous, interference among irrelevant or weakly relevant features may increase the burden of representation learning. As a result, sample efficiency and overall training quality may be limited. To mitigate this limitation, a Task-Informed Inductive-Bias (TIB) encoder is proposed to provide structured and task-conditioned state representations.

The overall framework is illustrated in Figure 2. Specifically, the dynamic state vector is first embedded into a compact latent feature $h_{\mathrm{dyn}}$. Two segment-specific branches with identical architectures but independent parameters then encode the path-following states associated with the current and next path segments, respectively. Conditioned on $h_{\mathrm{dyn}}$, each branch extracts segment-specific features, which are fused into a single representation $h_{\mathrm{fuse}}$.

$$h_{\mathrm{dyn}}=f_{\mathrm{dyn}}({\mathit{s}}_{\mathrm{dyn}};{\theta }_{\mathrm{dyn}})$$

(26)

$$h_{\mathrm{cur}}=f_{\mathrm{cur}}\left([h_{\mathrm{dyn}},{\mathit{s}}_{i^o}];{\theta }_{\mathrm{cur}}\right)$$

(27)

$$h_{\mathrm{next}}=f_{\mathrm{next}}\left([h_{\mathrm{dyn}},{\mathit{s}}_{i^o+1}];{\theta }_{\mathrm{next}}\right)$$

(28)

$$h_{\mathrm{fuse}}=f_{\mathrm{fuse}}\left([h_{\mathrm{cur}},h_{\mathrm{next}}];{\theta }_{\mathrm{fuse}}\right)$$

(29)

where $f_{\mathrm{dyn}}(\cdot )$, $f_{\mathrm{cur}}(\cdot )$, $f_{\mathrm{next}}(\cdot )$, and $f_{\mathrm{fuse}}(\cdot )$ are implemented as MLPs, and ${\theta }_{\mathrm{dyn}}$, ${\theta }_{\mathrm{cur}}$, ${\theta }_{\mathrm{next}}$, and ${\theta }_{\mathrm{fuse}}$ denote their parameters.

To adapt encoder features to segment-specific path-following states, a feature-wise linear modulation (FiLM) network $g(\cdot )$ generates scaling and shifting vectors (ω, β) from the task-level state ${\mathit{s}}_{\mathrm{task}}$. The fused feature $h_{\mathrm{fuse}}$ is then modulated element-wise.

$$[\omega ,\beta ]=g({\mathit{s}}_{\mathrm{task}};{\theta }_m)$$

(30)

$$\tilde{h}=\omega \odot h_{\mathrm{fuse}}+\beta$$

(31)

where $\omega ,\beta \in {ℝ}^{\left|h_{\mathrm{fuse}}\right|}$. $g(\cdot )$ is implemented as an MLP and ${\theta }_m$ denotes its parameters. $\odot$ denotes element-wise multiplication.

In general, the overall TIB mapping can be written compactly as

$$\tilde{h}=F_{\mathrm{enc}}(\mathit{s};{\mathrm{\Theta }}_{\mathrm{enc}})$$

(32)

where ${\mathrm{\Theta }}_{\mathrm{enc}}=\{{\theta }_{\mathrm{dyn}},{\theta }_{\mathrm{cur}},{\theta }_{\mathrm{next}},{\theta }_{\mathrm{fuse}},{\theta }_m\}$ denotes all encoder parameters.

Accordingly, the actor encoder produces a latent feature, which is then fed into the policy network to parameterize a squashed Gaussian policy. The action is subsequently sampled using the reparameterization trick.

$${\tilde{h}}^{\pi }=F_{\mathrm{enc}}(\mathit{s};{\mathrm{\Theta }}_{\mathrm{enc}}^{\pi })$$

(33)

$$[\mu (\mathit{s}),\log \sigma (\mathit{s})]=f_{\pi }({\tilde{h}}^{\pi };{\theta }_{\pi })$$

(34)

$$\mathit{a}=\tanh \left(\mu (\mathit{s})+\sigma (\mathit{s})\odot \epsilon \right)$$

(35)

where ${\mathrm{\Theta }}_{\mathrm{enc}}^{\pi }$ and ${\theta }_{\pi }$ denote the parameters of the actor encoder and policy network, respectively; $\epsilon \sim \mathcal{N}(0,I)$, and μ(s) and σ(s) represent the mean vector and standard deviation vector of the diagonal Gaussian distribution, respectively.

Similarly, the critics adopt the same TIB encoder architecture as the actor but use different parameters to avoid gradient interference between policy and critic updates.

$${\tilde{h}}^Q=F_{\mathrm{enc}}(\mathit{s};{\mathrm{\Theta }}_{\mathrm{enc}}^Q)$$

(36)

$$Q_i(\mathit{s},\mathit{a})=f_{Q_i}\left([{\tilde{h}}^Q,\mathit{a}];{\theta }_{Q_i}\right)$$

(37)

where i = 1, 2,…, n^Q, and n^Q denotes the number of critic heads. ${\mathrm{\Theta }}_{\mathrm{enc}}^Q$ and ${\theta }_{Q_i}$ denote the parameters of the critic encoder and the critic heads, respectively.

3.2.2. Conservative Soft Actor–Critic Algorithm

During path-following training, task difficulty spans a broad range and exhibits a long-tail distribution. Conventional SAC is a risk-neutral optimization method that primarily maximizes the overall expected return and lacks an explicit mechanism for handling rare but challenging operating conditions. When sample coverage is insufficient in these difficult regimes, critic value learning for the corresponding state–action pairs relies more heavily on function approximation with sparse data support, resulting in higher estimation uncertainty and lower reliability. Combined with the actor’s tendency to favor actions with high Q-values, this can produce locally optimistic value estimates and thus limit path-following performance in challenging regions.

To address this issue, this paper introduces a multi-head conservative value-evaluation mechanism. Specifically, n^Q (n^Q ≥ 2) critic heads with independent parameters are built on the shared critic TIB encoder. For any state–action pair (s,a), the mean Q-value μ_Q and the critic disagreement σ_Q are given by

$${\mu }_Q(\mathit{s},\mathit{a})={\displaystyle \frac{1}{n^Q}}{\displaystyle \sum _{i=1}^{n^Q}}{Q}_i(\mathit{s},\mathit{a})$$

(38)

$${\sigma }_Q(\mathit{s},\mathit{a})=\sqrt{{\displaystyle \frac{1}{n^Q}}{\displaystyle \sum _{i=1}^{n^Q}{\left(Q_i(\mathit{s},\mathit{a})-{\mu }_Q(\mathit{s},\mathit{a})\right)}^2}}$$

(39)

Policy improvement is guided by μ_Q, which represents the overall Q-value level, and σ_Q, which characterizes estimation uncertainty. This design discourages the policy from favoring actions with high predicted Q-values but large critic disagreement.

$$Q_{\mathrm{C}}(\mathit{s},\mathit{a})={\mu }_Q(\mathit{s},\mathit{a})-\sqrt{\ln n^Q}{\sigma }_Q(\mathit{s},\mathit{a})$$

(40)

$${\mathcal{L}}_{\pi }={\mathbb{E}}_{\mathit{s}\sim \mathcal{D},\mathit{a}\sim \pi (\cdot |\mathit{s})}\left[\begin{array}{c}\alpha \log \pi (\mathit{a}|\mathit{s})-Q_{\mathrm{C}}(\mathit{s},\mathit{a})\end{array}\right]$$

(41)

where $\sqrt{\ln n^Q}$ denotes the uncertainty-penalty coefficient. As n^Q increases, σ_Q can characterize value-estimation uncertainty more stably, and the penalty strength is therefore moderately increased. This growth is kept sublinear to avoid introducing an overly conservative bias that would impair policy-optimization efficiency.

The loss function for the entropy temperature coefficient α is defined as

$${\mathcal{L}}_{\alpha }={\mathbb{E}}_{\mathit{s}\sim \mathcal{D},\mathit{a}\sim {\pi }_{\varphi }(\cdot |\mathit{s})}\left[-\alpha \left(\log \pi (\mathit{a}|\mathit{s})+{\mathcal{H}}_{\mathrm{tar}}\right)\right]$$

(42)

where Ɗ denotes the replay buffer, and ${\mathcal{H}}_{\mathrm{tar}}$ is the target entropy.

To mitigate value overestimation and error propagation in critic bootstrapping, the temporal-difference (TD) target for the i-th critic head is calculated using the minimum of two randomly selected and distinct target critic heads. Let ${\mathit{a}}_{t+1}\sim \pi (\cdot |{\mathit{s}}_{t+1})$; the TD target is

$$y_{t,i}=r_{RL,t}+\gamma (1-d_t)\left(\underset{j=n_1,n_2}{\min }{\overline{Q}}_j({\mathit{s}}_{t+1},{\mathit{a}}_{t+1})-\alpha \log \pi ({\mathit{a}}_{t+1}\mid {\mathit{s}}_{t+1})\right)$$

(43)

where ${\overline{Q}}_j$ denotes the Q-value estimated by the j-th target critic; n₁, n₂∈{1, 2,…, n^Q} and n₁ ≠ n₂; γ is the discount factor; and d_t is the episode termination indicator.

To preserve critic diversity for uncertainty estimation, each critic head is updated using its own TD regression loss on an independently sampled mini-batch:

$${\mathcal{L}}_i^Q={\mathbb{E}}_{({\mathit{s}}_t,{\mathit{a}}_t,r_{RL,t},{\mathit{s}}_{t+1},d_t)\sim {\mathcal{B}}_i,{\mathcal{B}}_i~\mathcal{D}}\left[{(Q_i({\mathit{s}}_t,{\mathit{a}}_t)-y_{t,i})}^2\right]$$

(44)

where ${\mathcal{B}}_i$ denotes the mini-batch independently sampled for the i-th value head from the replay buffer Ɗ.

To ensure consistent state representations, the parameters of the TIB critic encoder are updated using the gradient of the mean TD loss:

$${\mathcal{L}}_{\mathrm{enc}}^Q={\displaystyle \frac{1}{n^Q}}{\displaystyle \sum _{i=1}^{n^Q}}{\mathcal{L}}_i^Q$$

(45)

The target critic encoder and its heads are updated through soft updates:

$${\overline{\mathrm{\Theta }}}_{\mathrm{enc}}^Q\leftarrow \tau {\mathrm{\Theta }}_{\mathrm{enc}}^Q+(1-\tau ){\overline{\mathrm{\Theta }}}_{\mathrm{enc}}^Q$$

(46)

$${\overline{\theta }}_{Q_i}\leftarrow \tau {\theta }_{Q_i}+(1-\tau ){\overline{\theta }}_{Q_i}$$

(47)

The overall algorithm is summarized in Algorithm 1.

Algorithm 1 Training procedure

Input: Critic size nQ; time interval Δt; soft-update rate τ;    discount factor γ; batch size nB; termination flag dt; Initialize: actor $\pi (\mathit{a}|\mathit{s})$; critics ${\left\{Q_i\right\}}_{i=1}^{n^Q}$; temperature α;      target critics ${\left\{{\overline{Q}}_i\right\}}_{i=1}^{n^Q}$; Replay buffer Ɗ; 1: // Interaction 2: for each environment step do 3:  Get current observation st; 4:  Sample action ${\mathit{a}}_t~{\pi }_{\varphi }(\cdot |{\mathit{s}}_t)$; 5:  Execute at, obtain $r_{\mathrm{RL},t}$, st+1, dt; 6:  If dt = true, reset the environment; 7:  Store $\{({\mathit{s}}_t,{\mathit{a}}_t,r_{\mathrm{RL},t},{\mathit{s}}_{t+1},d_t)\}$ into Ɗ; 8: // Parameter updates 9:  if |Ɗ| > nbat then 10:    Sample independent mini-batch ${\left\{{\mathcal{B}}_i\right\}}_{i=1}^{n^Q}$; 11:    Sample next action ${\mathit{a}}_{t+1}~{\pi }_{\varphi }(\cdot |{\mathit{s}}_{t+1})$ for (${\mathit{s}}_{t+1}$ in ${\left\{{\mathcal{B}}_i\right\}}_i^{n^Q}$); 12:    Compute TD target using Equation (43); 13:    Update critic using Equations (44) and (45); 14:    Update actor using Equation (41); 15:    Update α using Equation (42); 16:    Soft update target critics using Equations (46) and (47); 17:  end if 18: end for

4. Experiments

4.1. Basic Settings

To facilitate path-following learning, each training episode consists of a two-segment reference path randomly initialized under the constraints in Equation (6). At the beginning of each episode, the initial AUV velocity is set to zero, and the desired surge speed u_d is randomly sampled from [0.5, 2.0] m/s. This setting covers the acceleration process from zero speed to the desired surge speed, thereby including the initial low-authority phase of the X-rudder in the training distribution. To preserve state continuity and ensure stable policy updates, the observation index i^o is kept fixed at its initial value for most of the episode. Therefore, an episode terminates when either the maximum time limit $t_{\max }^t$ is exceeded or the stabilization-time criterion in Equation (13) is satisfied. The observation index i^o is updated only in the latter case, as shown in Figure 3.

The hydrodynamic parameters are adopted from Ref. [25] and are provided in

Table 1. Hydrodynamic parameters.

Parameter	Value	Parameter	Value
Inertia terms	$(m_{11},m_{22},m_{33})=(215,265,265) \mathrm{kg}$ $(m_{55},m_{66})=80 \mathrm{kg}\cdot {\mathrm{m}}^2$	Rudder rate	Δδmax = 0.3491 rad/s
Damping coefficient	$(X_u,Y_v,Z_w)=(70,100,100) \mathrm{kg}\cdot {\mathrm{m}}^2$ $(M_q,N_r)=50 \mathrm{kg}\cdot {\mathrm{m}}^2/\mathrm{s}$	Rudder limit	δmax = 0.4189 rad
Damping coefficient	$(X_{u\left|u\right|},Y_{v\left|v\right|},Z_{w\left|w\right|})=100\cdot (1,2,2) \mathrm{kg}/\mathrm{m}$ $(M_{q\left|q\right|},N_{r\left|r\right|})=50 \mathrm{kg}\cdot {\mathrm{m}}^2$	AUV weight	W = 1813 N
Rudder coefficients	${\mathit{M}}_{\delta }=0.1595\cdot [+1,-1,-1,+1]$ ${\mathit{N}}_{\delta }=0.1595\cdot [+1,-1,-1,+1]$	Vertical offset	${\overline{BG}}_z=0.02\hspace{0.33em}\mathrm{m}$

; the training task settings are summarized in

Table 2. Training task settings.

Parameter	Value	Parameter	Value
Initial AUV states	η0 = 0, ν0 = 0	Path segments	np = 2
Maximum adjacent heading change	$\mathrm{\Delta }{\psi }_{\max }^p=1.5707 \mathrm{rad}$	Time step	Δt = 0.2 s
Maximum absolute path pitch angle	$\mathrm{\Delta }{\theta }_{\max }^p=0.1222 \mathrm{rad}$	Time threshold	$t_{\max }^o=10 \mathrm{s}$
Minimum segment length	Lmin = 100 m	Error threshold	$d_{\min }^h=0.1 \mathrm{m}$
Normalization factors	$(e_{\max }^h,e_{\max }^v,r_{\max })=(4,2,4)$	Reward weights	$k_i^r=5 \left(i=1,2,3\right)$
Maximum task time limit	$t_{\max }^t=1.3\mathrm{sum}({\mathit{L}}^p/\widehat{u})$	Disturbance phase shifts	$\left|{\phi }_1^d\right|<\pi /8$
Disturbance frequencies	$0.08<k_1^d<0.12$	Disturbance phase shifts	$\pi /8<\left|{\phi }_2^d\right|<3\pi /8$
Disturbance frequencies	$0.18<k_2^d<0.22$	Disturbance phase shifts	$\pi /12<\left|{\phi }_3^d\right|<\pi /4$

. The bounded time-varying external disturbances acting on the AUV hydrodynamics are given in Equation (48). The sinusoidal and cosine terms approximate low-frequency force/moment disturbances with wave-like periodic characteristics.

$$\left\{\begin{array}{l}{\tau }_u^d(t)=u_d/u_{d,\max }(5\sin (k_1^{d}t+{\phi }_1^d)+5\cos (k_1^{d}t+{\phi }_2^d)+5\cos (k_1^{d}t+{\phi }_3^d))\\ {\tau }_v^d(t)=u_d/u_{d,\max }(0.5\sin (k_1^{d}t+{\phi }_1^d)+\cos (k_1^{d}t+{\phi }_2^d)+0.1\cos (k_1^{d}t+{\phi }_3^d))\\ {\tau }_w^d(t)=u_d/u_{d,\max }(0.5\sin (k_1^{d}t+{\phi }_1^d)+\cos (k_1^{d}t+{\phi }_2^d)+0.1\cos (k_1^{d}t+{\phi }_3^d))\\ {\tau }_q^d(t)=u_d/u_{d,\max }(\sin (k_2^{d}t+{\phi }_1^d)+2\cos (k_2^{d}t+{\phi }_2^d)+2\cos (k_1^{d}t+{\phi }_3^d))\\ {\tau }_r^d(t)=u_d/u_{d,\max }(\sin (k_2^{d}t+{\phi }_1^d)+\cos (k_2^{d}t+{\phi }_2^d)+2\cos (k_1^{d}t+{\phi }_3^d))\end{array}\right.$$

(48)

where $k_1^d$ and $k_2^d$ denote the disturbance angular frequencies, while ${\phi }_1^d$, ${\phi }_2^d$, and ${\phi }_3^d$ denote the disturbance phase shifts, which are randomly sampled at the beginning of each episode within the predefined ranges in Table 2. In the subsequent comparative tests, these parameters are fixed to $k_1^d$ = 0.1, $k_2^d=0.2$, ${\phi }_1^d$ = 0, ${\phi }_2^d$ = π/4, and ${\phi }_3^d$= π/6.

In this study, the AUV speed is regulated by an incremental S-plane controller, with the thruster parameters and control law provided in Appendix A. X-rudder path-following control is realized using three off-policy DRL methods: SAC [26], DDPG [27], and the proposed method. SAC is selected as the main baseline to directly evaluate the contributions of the proposed modules within the SAC framework. DDPG is included as another representative off-policy continuous-control algorithm, providing a deterministic-policy comparison. A conventional hierarchical model-free benchmark is also included, combining LOS guidance [18], an S-plane controller, and pseudoinverse-based control allocation. The algorithm settings are summarized in

Table 3. Algorithm settings.

Parameter	Value	Parameter	Value
Learning rate	lr = 2 × 10−4	Dynamic encoder	MLP(2 × 32), ReLU
Discount factor	γ = 0.99	Segment branches	MLP(2 × 64), ReLU
Critic number	nQ = 6	Fusion layer	MLP(1 × 64), ReLU
Soft update coefficient	τ = 1 × 10−3	FiLM modulation	g:MLP(1 × 64), ReLU
Training episodes	neps = 1200	Policy network	MLP(2 × 128), ReLU
Replay Buffer size	nbuf = 1 × 106	Critic head network	MLP(2 × 128), ReLU
Batch size	nbat = 256	Target entropy	${\mathcal{H}}_{\mathrm{tar}}=-2$

. Shared or comparable hyperparameters and network scales are kept consistent wherever applicable so that the comparison is conducted under unified algorithmic configurations and comparable model capacities.

4.2. Training Results

To mitigate stochastic effects and ensure reliable comparisons, each method was trained independently for 1200 episodes using five different random seeds. The training results are shown in Figure 4, where the solid lines represent the episode-wise mean return ${\overline{R}}_j$ across five runs, and the shaded regions indicate the corresponding episode-wise standard deviation ${\sigma }_j$. As illustrated in Figure 4, TIB-CSAC converges faster during training; the magnified view also shows superior late-stage stability, with no sharp declines in episode rewards as observed for DDPG and SAC.

$${\overline{R}}_j={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K{R}_{j,k}}$$

(49)

$${\sigma }_j=\sqrt{{\displaystyle \frac{1}{K-1}}{\displaystyle \sum _{k=1}^K{\left(R_{j,k}-{\overline{R}}_j\right)}^2}}$$

(50)

where $R_{j,k}$ denotes the episode return at episode j in the k-th run, k = 1, 2,…, K, K = 5.

To quantify tail-stage performance, as defined in Equations (51)–(53), let ${\overline{R}}_{\mathrm{tail}}$ denote the mean episode return over the tail stage of training averaged across runs, and let ${\sigma }_{\mathrm{tail}}$ denote the average standard deviation of episode returns over the tail stage within each run. As detailed in

Table 4. Training metric results in the comparative study.

Method	${\overline{\mathit{R}}}_{\mathbf{tail}}$	${\mathit{\sigma }}_{\mathbf{tail}}$	$\overline{\mathbf{TRE}}$	$\overline{\mathbf{AUC}}$
TIB-CSAC	−1.7244 × 102	1.2368 × 102	2.9560 × 102	−2.6398 × 106
SAC	−2.7927 × 102	5.1210 × 102	7.3320 × 102	−6.5231 × 106
DDPG	−5.5372 × 102	4.6465 × 102	9.3920 × 102	−9.2165 × 106

, compared with baseline SAC and DDPG, the proposed method increases ${\overline{R}}_{\mathrm{tail}}$ from −2.7927 × 10² and −5.5372 × 10² to −1.7244 × 10² and reduces ${\sigma }_{\mathrm{tail}}$ from 5.1210 × 10² and 4.6465 × 10² to 1.2368 × 10², indicating improved tail-stage training performance and enhanced tail-stage stability.

$${\overline{R}}_{\mathrm{tail},k}={\displaystyle \frac{1}{N_{\mathrm{tail}}}}{\displaystyle \sum _{j=n^{\mathrm{eps}}-N_{\mathrm{tail}}+1}^{n^{\mathrm{eps}}}}{R}_{j,k}$$

(51)

$${\overline{R}}_{\mathrm{tail}}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K{\overline{R}}_{\mathrm{tail},k}}$$

(52)

$${\sigma }_{\mathrm{tail}}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K\sqrt{\frac{1}{N_{\mathrm{tail}}-1}{\displaystyle \sum _{j=n^{\mathrm{eps}}-N_{\mathrm{tail}}+1}^{n^{\mathrm{eps}}}}{\left(R_{j,k}-{\overline{R}}_{\mathrm{tail},k}\right)}^2}}$$

(53)

where N_tail denotes the number of episodes in the tail-stage training and is set to 200.

To quantify training efficiency, episode returns are first smoothed using a W-episode moving average. For the k-th run, TRE_k is defined as the earliest episode t such that the smoothed return ${\tilde{R}}_{t,k}\ge R_{\mathrm{th}}$ satisfies Equation (55), and the overall TRE is averaged over K runs according to Equation (56). Here, W = 50 and $R_{\mathrm{th}}=-1\times {10}^3$. Moreover, the average area under the training curve is used to quantify overall training quality, as defined in Equation (57).

$${\tilde{R}}_{t,k}={\displaystyle \frac{1}{W}}{\displaystyle \sum _{j=1}^W}{R}_{t-j,k},t\ge W$$

(54)

$${\mathrm{TRE}}_k=\min \left\{t\in \{W,\dots ,n^{\mathrm{eps}}\}\mid {\tilde{R}}_{t,k}\ge R_{\mathrm{th}}\right\}$$

(55)

$$\overline{\mathrm{TRE}}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K{\mathrm{TRE}}_k}$$

(56)

$$\overline{\mathrm{AUC}}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{\displaystyle {\int }_1^{n^{\mathrm{eps}}}{R}_{j,k}}\mathrm{d}j$$

(57)

As detailed in Table 4, TIB-CSAC reaches the near-final performance threshold with fewer training episodes than SAC and DDPG: $\overline{\mathrm{TRE}}$ equal to 2.9560 × 10² compared with 7.3320 × 10² and 9.3920 × 10². It also attains a less negative cumulative return in $\overline{\mathrm{AUC}}$, namely −2.6398 × 10⁶ versus −6.5231 × 10⁶ and −9.2165 × 10⁶, indicating improved sample efficiency and overall training quality.

4.3. Path Following Results

4.3.1. Case 1 Boustrophedon Path

A boustrophedon path is used to assess horizontal path-following performance during aggressive orthogonal turns under varying desired speeds u_d. In the horizontal plane, the path consists of 400 m survey lines connected by 100 m short transects. Moreover, u_d is updated after the AUV completes two consecutive turns and then travels 100 m along the path, with u_d taking values of 2.0, 1.7, 1.4, 1.1, 0.8, and 0.5 m/s. The detailed results are shown in Figure 5, Figure 6, Figure 7 and Figure 8.

As shown by the overall trajectories in Figure 5, all methods accomplish the prescribed boustrophedon path-following task. The zoomed-in views in Figure 5b further indicate that the main differences among these methods are concentrated in a few challenging regions, particularly around orthogonal turns and the subsequent transition phases. As shown in Figure 6 and Figure 7, under different velocity conditions, TIB-CSAC achieves tighter adherence to the reference path, with overall position errors remaining within a relatively small range. During orthogonal-turn transitions, despite variations in rudder hydrodynamic effectiveness across velocities, TIB-CSAC consistently maintains lower horizontal overshoot and faster error decay. By contrast, although the baseline methods keep the horizontal error close to zero under most conditions, they tend to exhibit larger overshoot peaks or oscillations and slower error decay around orthogonal turns. Overall, the results of Case 1 show that TIB-CSAC provides more robust horizontal path-following performance across a range of surge velocities, especially in the small subset of challenging orthogonal-turn conditions.

4.3.2. Case 2 Trapezoidal Path

A 3D trapezoidal path is used to validate the transient path-following response under segment-wise simultaneous variations in the desired path heading, path pitch, and surge velocity. The desired surge velocity u_d is set sequentially to [2, 1.875, 1.75, 1.625, 1.5, 1.375, 1.25, 1.125, 1, 0.875, 0.75, 0.625, 0.5] m/s; the path pitch is set to [−0.1222, 0, 0.1047, 0, −0.0873, 0, 0.0698, 0, −0.0524, 0, 0.0349, 0, −0.0175] rad; and the path heading is set to [0, 1.5708, 0, 1.309, 0, 1.0472, 0, 0.7854, 0, 0.5236, 0, 0.2618, 0] rad. The length of each segment is 100 m. The detailed results are shown in Figure 9, Figure 10, Figure 11 and Figure 12.

As shown in Figure 9, all methods complete the 3D trapezoidal path-following task, and the main discrepancies arise around path-segment transitions. As indicated by Figure 10 and Figure 11, TIB-CSAC yields smaller overall position errors and more stable transitions between path segments than the baseline methods under simultaneous variations in surge velocity and path geometry. In particular, under higher-velocity conditions, TIB-CSAC more effectively suppresses horizontal error peaks caused by large path-heading changes and mitigates persistent vertical-error oscillations associated with large path-pitch variations. As shown in Figure 12, the X-rudder angles of TIB-CSAC remain smooth and continuous except for brief pulse-like adjustments during path-segment transitions. Overall, Case 2 demonstrates that TIB-CSAC provides a more stable transient response under simultaneous variations in surge velocity and path geometry.

4.3.3. Case 3 Random Path

This section evaluates path-following performance on a 3D random path under time-varying surge-velocity conditions.

Table 5. Waypoint Coordinates of Random Paths.

Index	1	2	3	4	5	6	7	8	9	10
$x_i^p$ (m)	111.79	238.38	434.02	546.22	486.21	562.53	674.69	596.73	441.31	433.10
$y_i^p$ (m)	92.60	158.75	179.64	115.61	−47.99	−174.84	−288.56	−370.22	−325.50	−185.18
$z_i^p$ (m)	2.56	14.46	14.68	15.56	3.68	11.19	15.01	20.85	11.28	18.72
Index	11	12	13	14	15	16	17	18	19	20
$x_i^p$ (m)	464.11	375.32	450.57	628.69	794.98	883.33	1001.32	1110.58	1264.07	1387.24
$y_i^p$ (m)	−74.54	99.49	201.61	131.19	68.23	−97.93	−52.65	54.26	135.41	54.28
$z_i^p$ (m)	20.23	24.20	22.87	32.63	31.45	43.51	32.77	35.95	51.05	51.89

summarizes the waypoint configuration of the reference path, which consists of 20 waypoints in total, with the coordinate origin defined as the path-starting point. The velocity is set to $u_d(t)=-0.75\cos (0.01t)+1.25$. The results are shown in Figure 13, Figure 14, Figure 15 and Figure 16.

As shown in Figure 13, Figure 14 and Figure 15, the main differences in the 3D random path-following results are concentrated in challenging regions associated with higher surge velocities and large path-geometry variations. The baseline methods exhibit different error patterns: DDPG and LOS+SP show larger horizontal overshoot and more pronounced vertical-error oscillations in several regions, whereas SAC exhibits horizontal and vertical error fluctuations in certain intervals, accompanied by sustained oscillations in surge velocity. By contrast, under the combined effects of time-varying speed and random path-geometry variations, TIB-CSAC more effectively mitigates horizontal overshoot and vertical oscillations, demonstrating more robust path-following performance under diverse operating conditions.

4.3.4. Metric Analyses

Four metrics are used to quantitatively evaluate path-following performance: the maximum absolute error (MXAE), mean absolute error (MAE), conditional value at risk at 95% (CVaR₉₅), and threshold-exceedance time ratio V_ε. CVaR₉₅ and V_ε are defined in Equations (58)–(60). Specifically, CVaR₉₅ denotes the conditional mean of the position-error magnitude beyond the 95th percentile of |e(t)|, while V_ε denotes the fraction of time for which |e(t)| exceeds the threshold ε = 0.1 m. Detailed quantitative results are summarized in

Table 6. Vertical error metrics.

Case	Metric	TIB-CSAC	LOS+SP	SAC	DDPG
C1	MXAE (m)	1.7299 × 10−1	2.6719 × 10−1	2.6679 × 10−1	4.2489 × 10−1
	MAE (m)	4.1841 × 10−3	1.0881 × 10−2	1.3561 × 10−2	4.7823 × 10−2
	CVaR95 (m)	4.7757 × 10−2	8.6580 × 10−2	6.3900 × 10−2	2.4309 × 10−1
	Vε	7.4956 × 10−3	1.4532 × 10−2	3.0235 × 10−3	1.4487 × 10−1
C2	MXAE (m)	3.6035 × 10−1	4.8607 × 10−1	5.8540 × 10−1	5.5639 × 10−1
	MAE (m)	1.0904 × 10−2	3.8505 × 10−2	2.1844 × 10−2	8.1225 × 10−2
	CVaR95 (m)	1.0198 × 10−1	2.5271 × 10−1	1.5847 × 10−1	4.0743 × 10−1
	Vε	1.7398 × 10−2	1.1855 × 10−1	3.2938 × 10−2	2.5016 × 10−1
C3	MXAE (m)	3.4549 × 10−1	3.7410 × 10−1	4.6598 × 10−1	6.7275 × 10−1
	MAE (m)	1.2928 × 10−2	3.3489 × 10−2	3.1507 × 10−2	9.6232 × 10−2
	CVaR95 (m)	1.2817 × 10−1	2.1243 × 10−1	1.6925 × 10−1	4.4115 × 10−1
	Vε	2.5180 × 10−2	8.2075 × 10−2	6.1138 × 10−2	3.3564 × 10−1

and

Table 7. Horizontal error metrics.

Case	Metric	TIB-CSAC	LOS+SP	SAC	DDPG
C1	MXAE (m)	2.3813 × 100	1.0942 × 101	1.0892 × 101	1.0348 × 101
	MAE (m)	6.1981 × 10−2	5.9013 × 10−1	4.7040 × 10−1	4.8171 × 10−1
	CVaR95 (m)	8.4313 × 10−1	7.1942 × 100	5.3551 × 100	5.1160 × 100
	Vε	6.8683 × 10−2	3.7126 × 10−1	3.8413 × 10−1	7.3974 × 10−1
C2	MXAE (m)	1.0324 × 100	1.0575 × 101	8.3592 × 100	1.4654 × 101
	MAE (m)	3.6557 × 10−2	7.0307 × 10−1	4.6165 × 10−1	7.6099 × 10−1
	CVaR95 (m)	3.8266 × 10−1	6.1709 × 100	4.2428 × 100	7.2433 × 100
	Vε	6.6016 × 10−2	5.7557 × 10−1	4.9173 × 10−1	7.9612 × 10−1
C3	MXAE (m)	2.9708 × 100	1.0196 × 101	6.5206 × 100	1.0790 × 101
	MAE (m)	8.7358 × 10−2	5.9247 × 10−1	5.1517 × 10−1	7.0218 × 10−1
	CVaR95 (m)	1.0650 × 100	4.9855 × 100	4.1604 × 100	5.4241 × 100
	Vε	1.4977 × 10−1	5.8885 × 10−1	4.6221 × 10−1	8.4074 × 10−1

.

$${\mathrm{CVaR}}_{95}=\mathbb{E}\left[\left|e(t)\right|\left|\left|e(t)\right|\ge q_{95}\right.\right]$$

(58)

$$q_{95}=\inf \left\{q:\mathrm{Pr}(\left|e(t)\right|\le q)\ge 0.95\right\}$$

(59)

$$V_{\epsilon }={\displaystyle \frac{1}{T}}{\displaystyle {\int }_0^T\chi \left(|e(t)|>\epsilon \right)dt}$$

(60)

where e(t) denotes the position error, q₉₅ is the 95th percentile of |e(t)|, Pr(⋅) denotes probability, inf{⋅} denotes the infimum, χ(⋅) is the indicator function, and T denotes the task duration.

Averaged over the three cases, TIB-CSAC yields lower overall error metrics than baseline SAC. In terms of MXAE, the average vertical and horizontal MXAE values of TIB-CSAC are reduced by 1.4645 × 10⁻¹ m and 6.4623 m, respectively, indicating a stronger ability to suppress extreme errors. For MAE, the corresponding reductions are 1.2965 × 10⁻² m and 4.2044 × 10⁻¹ m, indicating closer overall adherence to the reference path. For CVaR₉₅, the corresponding reductions are 3.7903 × 10⁻² m and 3.8225 m, indicating more effective suppression of tail errors under difficult operating conditions. For V_ε, the corresponding reductions are 1.5675 × 10⁻² and 3.5120 × 10⁻¹, indicating a lower proportion of time with threshold-exceeding errors and better maintenance of small-error path-following.

Overall, TIB-CSAC exhibits reliable path-following performance under diverse surge-velocity and path-geometry conditions, with clear advantages in overall path adherence, error-peak suppression, tail-error reduction, and persistence within the prescribed error threshold.

Several supplementary metrics are also introduced to quantitatively evaluate X-rudder control behavior and task-level effort. These metrics include the rudder saturation ratio $R_{\delta }^{sat}$, second-order rudder-angle difference index $J_{{\mathrm{\Delta }}^2\delta }$, rudder-rate smoothness index $J_{\dot{\delta }}$, speed-weighted rudder-effort proxy $J_{\delta ,u}$, and AUV energy-consumption proxy $J_{AUV}$, as defined in Equations (61)–(65). Here, N is the number of time steps. The quantitative results are summarized in

Table 8. Control-related metrics.

Case	Metric	TIB-CSAC	LOS+SP	SAC	DDPG
C1	$R_{\delta }^{sat}$	3.6415 × 10−2	3.5432 × 10−2	7.1948 × 10−2	1.9965 × 10−2
	$J_{{\mathrm{\Delta }}^2\delta }$	5.9789 × 10−2	7.9926 × 10−1	2.0554 × 10−2	6.3354 × 100
	$J_{\dot{\delta }}$	7.5725 × 100	1.0895 × 101	7.7465 × 100	4.2052 × 101
	$J_{\delta ,u}$	2.7156 × 101	1.8364 × 101	7.7713 × 101	1.4899 × 101
	$J_{\mathrm{AUV}}$	5.5553 × 103	5.7613 × 103	5.9402 × 103	5.6670 × 103
	$T$	3.7622 × 103	3.8948 × 103	3.9028 × 103	3.8116 × 103
C2	$R_{\delta }^{sat}$	6.0163 × 10−3	1.0941 × 10−2	5.5885 × 10−2	2.7831 × 10−3
	$J_{{\mathrm{\Delta }}^2\delta }$	3.1153 × 10−1	2.3985 × 100	1.0248 × 10−1	6.8936 × 100
	$J_{\dot{\delta }}$	1.8446 × 101	3.1841 × 101	2.1538 × 101	4.7438 × 101
	$J_{\delta ,u}$	3.2003 × 101	2.6966 × 101	1.7262 × 102	1.9909 × 101
	$J_{\mathrm{AUV}}$	2.2942 × 103	2.4531 × 103	2.5507 × 103	2.4031 × 103
	$T$	1.2300 × 103	1.2704 × 103	1.2812 × 103	1.2576 × 103
C3	$R_{\delta }^{sat}$	4.1298 × 10−2	2.0751 × 10−2	9.3597 × 10−2	2.6845 × 10−2
	$J_{{\mathrm{\Delta }}^2\delta }$	6.1293 × 10−1	1.5771 × 100	1.1750 × 10−1	1.3719 × 101
	$J_{\dot{\delta }}$	1.6996 × 101	1.9188 × 101	2.1262 × 101	8.9930 × 101
	$J_{\delta ,u}$	3.4639 × 101	2.3822 × 101	2.0899 × 102	2.6849 × 101
	$J_{\mathrm{AUV}}$	7.5449 × 103	7.5713 × 103	7.7279 × 103	7.5638 × 103
	$T$	2.4384 × 103	2.5830 × 103	2.6988 × 103	2.5480 × 103

.

$$R_{\delta }^{sat}={\displaystyle \frac{1}{4N}}{\displaystyle \sum _{i=1}^4{\displaystyle \sum _{k=1}^N\chi \left(|{\delta }_i(k)|\ge 0.95{\delta }_{\max }\right)}}$$

(61)

$$J_{{\mathrm{\Delta }}^2\delta }={\displaystyle \frac{1}{4(N-2)}}{\displaystyle \sum _{i=1}^4{\displaystyle \sum _{k=3}^N{\left({\delta }_i(k)-2{\delta }_i(k-1)+{\delta }_i(k-2)\right)}^2}}$$

(62)

$$J_{\dot{\delta }}={\displaystyle \frac{1}{4(N-1)}}{\displaystyle \sum _{i=1}^4{\displaystyle \sum _{k=2}^N{\left(({\delta }_i(k)-{\delta }_i(k-1))/\mathrm{\Delta }t\right)}^2}}$$

(63)

$$J_{\delta ,u}={\displaystyle \frac{1}{4N}}{\displaystyle \sum _{i=1}^4{\displaystyle \sum _{k=1}^N{\left(u(k){\delta }_i(k)\right)}^2}}$$

(64)

$$J_{\mathrm{AUV}}=\mathrm{\Delta }t{\displaystyle \sum _{k=1}^N{u}^3(k)}$$

(65)

As shown in Table 8, TIB-CSAC maintains a relatively low rudder saturation ratio in all three cases, although LOS+SP or DDPG achieves lower $R_{\delta }^{sat}$ values in some cases. For the second-order rudder-angle difference $J_{{\mathrm{\Delta }}^2\delta }$, SAC generally yields smaller values, while TIB-CSAC remains lower than LOS+SP and DDPG. This indicates that the proposed method introduces necessary corrective rudder actions during path transitions without causing severe rudder-angle fluctuations. In terms of the rudder-rate smoothness index $J_{\dot{\delta }}$, TIB-CSAC achieves the lowest values in all three cases, suggesting smoother rudder-rate behavior. For the speed-weighted rudder-effort proxy $J_{\delta ,u}$, TIB-CSAC is not always the lowest, but it is consistently lower than SAC. Moreover, TIB-CSAC achieves the lowest AUV energy-consumption proxy $J_{\mathrm{AUV}}$ and the shortest task-completion time T in all three cases. These results indicate that the improved path-following performance of TIB-CSAC is not obtained through persistent rudder saturation or excessive task-level effort but through generally reasonable actuator behavior.

4.4. Ablation Study

To further analyze the contribution of each module, this section conducts an ablation study on the full TIB-CSAC method and three comparison variants: the baseline SAC; CSAC, in which the TIB encoder is replaced by fully connected layers of comparable size; and TIB-SAC, which combines the TIB encoder with the baseline SAC. Each method is trained for 1200 episodes using five random seeds.

4.4.1. Quantitative Analysis of the TIB Encoder

Table 9. Training metric results in the ablation study.

Method	$\overline{\mathrm{TRE}}$	$\overline{\mathrm{AUC}}$	${\mathit{\sigma }}_{\mathrm{tail}}$	${\overline{\mathit{R}}}_{\mathrm{tail}}$
SAC	7.3320 × 102	−6.5231 × 106	5.1210 × 102	−2.7927 × 102
CSAC	6.4180 × 102	−5.4341 × 106	1.3339 × 102	−2.0564 × 102
TIB-SAC	4.3060 × 102	−3.9544 × 106	3.5529 × 102	−2.3032 × 102
TIB-CSAC	2.9560 × 102	−2.6398 × 106	1.2368 × 102	−1.7244 × 102

and Figure 17 quantify the contribution of the TIB encoder through two controlled comparisons: CSAC versus TIB-CSAC and SAC versus TIB-SAC. Compared with TIB-CSAC, the $\overline{\mathrm{TRE}}$ of CSAC increases from 2.9560 × 10² to 6.4180 × 10², corresponding to 2.17 times that of TIB-CSAC. Meanwhile, the $\overline{\mathrm{AUC}}$ decreases from −2.6398 × 10⁶ to −5.4341 × 10⁶, corresponding to 2.06 times the absolute magnitude of TIB-CSAC. A consistent trend is also observed in the comparison between TIB-SAC and SAC. Specifically, the $\overline{\mathrm{TRE}}$ of TIB-SAC decreases from 7.3320 × 10² to 4.3060 × 10², corresponding to 0.59 times that of SAC, while the $\overline{\mathrm{AUC}}$ increases from −6.5231 × 10⁶ to −3.9544 × 10⁶, corresponding to 0.61 times the absolute magnitude of SAC. In addition, the tail-stage metrics show consistent but secondary improvements after introducing the TIB encoder, with ${\sigma }_{\mathrm{tail}}$ decreasing in both SAC versus TIB-SAC and CSAC versus TIB-CSAC. Overall, these results indicate that the structured and conditioned representations induced by the TIB encoder primarily accelerate policy formation and improve overall training quality.

4.4.2. Quantitative Analysis of the Conservative Value Evaluation Mechanism

The contribution of the conservative value-evaluation mechanism is quantified through two controlled comparisons: TIB-SAC versus TIB-CSAC and SAC versus CSAC. Relative to TIB-CSAC, the ${\sigma }_{\mathrm{tail}}$ of TIB-SAC increases from 1.2368 × 10² to 3.5529 × 10², corresponding to 2.87 times that of TIB-CSAC. A similar effect is observed in the comparison between CSAC and SAC, where the ${\sigma }_{\mathrm{tail}}$ of CSAC decreases from 5.1210 × 10² to 1.3339 × 10², corresponding to 0.26 times that of SAC. Collectively, these results indicate that the multi-head conservative value-evaluation mechanism effectively mitigates tail-stage return fluctuations and reduces return drawdowns induced by challenging tasks, thereby improving tail-stage training stability.

4.4.3. Overall Ablation Discussion

Compared with baseline SAC, TIB-CSAC reduces $\overline{\mathrm{TRE}}$ and ${\sigma }_{\mathrm{tail}}$ to 0.40 and 0.24 times those of SAC, respectively, while yielding less negative $\overline{\mathrm{AUC}}$ and ${\overline{R}}_{\mathrm{tail}}$ values whose absolute magnitudes are reduced to 0.40 and 0.62 times those of SAC, respectively. These results indicate that combining the TIB encoder with the conservative value-evaluation mechanism enables TIB-CSAC to achieve faster convergence, higher overall training quality, and better tail-stage performance.

4.5. Sensitivity Study

4.5.1. Sensitivity Analysis on Critic Size

To further investigate the influence of a key hyperparameter on the performance of the proposed algorithm, this section conducts a sensitivity study on the critic size n^Q used in the multi-head conservative value evaluation mechanism. For each parameter setting, training is performed for 1200 episodes using five random seeds.

As shown in Figure 18 and

Table 10. Training metric results in the sensitivity study.

nQ	${\mathit{\sigma }}_{\mathbf{tail}}$	$\overline{\mathbf{TRE}}$	$\overline{\mathit{A}\mathit{U}\mathit{C}}$	${\overline{\mathit{R}}}_{\mathbf{tail}}$
2	2.4269 × 103	1.2000 × 103	−7.9829 × 106	−3.7323 × 103
4	5.3876 × 102	3.8780 × 102	−3.2756 × 106	−2.7346 × 102
6	1.2368 × 102	2.9560 × 102	−2.6398 × 106	−1.7244 × 102
8	1.0508 × 102	3.3480 × 102	−2.6063 × 106	−1.6741 × 102

, when n^Q = 2, the return curves remain in a low-return region with severe fluctuations, indicating that an overly small n^Q is insufficient to provide reliable conservative value evaluation for effective policy learning. With n^Q = 4, the return curves begin to enter the high-return region, but noticeable return drawdowns still persist after convergence, suggesting that value-evaluation robustness remains inadequate. Increasing n^Q from 4 to 6 effectively improves tail-stage stability, with ${\sigma }_{\mathrm{tail}}$ decreasing from 5.3876 × 10² to 1.2368 × 10², corresponding to 0.23 times that of the n^Q = 4 setting. By contrast, further increasing n^Q from 6 to 8 yields only a modest additional reduction in ${\sigma }_{\mathrm{tail}}$, from 1.2368 × 10² to 1.0508 × 10², corresponding to 0.85 times that of the n^Q = 6 setting, while convergence becomes slower, with $\overline{\mathrm{TRE}}$ increasing from 2.9560 × 10² to 3.3480 × 10², corresponding to 1.13 times that of the n^Q = 6 setting. This trend suggests that an overly large n^Q may further suppress tail-stage volatility, but the associated increase in conservatism may also make policy improvement more cautious, thereby reducing sample efficiency. Therefore, under the algorithmic setting adopted in this study, n^Q = 6 provides a favorable trade-off between convergence speed and tail-stage performance.

4.5.2. Sensitivity Analysis on Reward Weights

To further evaluate the influence of reward weights on path-following performance, a sensitivity analysis was conducted for k_r,1, k_r,2, and k_r,3. In addition to the default setting k_r,1 = k_r,2 = k_r,3 = 5, three reward-weight combinations were tested in Cases 1–3: [k_r,1, k_r,2, k_r,3] = [7.50, 3.75, 3.75], [3.75, 7.50, 3.75], and [3.75, 3.75, 7.50]. These settings increase the relative weights of the horizontal-error penalty, vertical-error penalty, and reverse-motion penalty, respectively. The averaged metric results over Cases 1–3 are shown in Figure 19, and the detailed metrics for each case are provided in

Table 11. Horizontal error metrics for sensitivity analysis of reward weights.

Case	[kr,1, kr,2, kr,3]	[7.50, 3.75, 3.75]	[3.75, 7.50, 3.75]	[3.75, 3.75, 7.50]	[5.0, 5.0, 5.0]
C1	MXAE (m)	2.8895 × 100	3.2273 × 100	2.7073 × 100	2.3813 × 100
	MAE (m)	9.3567 × 10−2	1.2752 × 10−1	8.1307 × 10−2	6.1981 × 10−2
	CVaR95 (m)	1.2507 × 100	1.1357 × 100	1.0947 × 100	8.4313 × 10−1
	Vε	8.7584 × 10−2	3.7600 × 10−1	7.3903 × 10−2	6.8683 × 10−2
C2	MXAE (m)	1.1418 × 100	1.6357 × 100	1.1187 × 100	1.0324 × 100
	MAE (m)	4.0079 × 10−2	8.7472 × 10−2	5.9850 × 10−2	3.6557 × 10−2
	CVaR95 (m)	4.0264 × 10−1	7.0743 × 10−1	5.4121 × 10−1	3.8266 × 10−1
	Vε	6.0212 × 10−2	2.0276 × 10−1	1.1075 × 10−1	6.6016 × 10−2
C3	MXAE (m)	2.6098 × 100	2.5967 × 100	2.6334 × 100	2.9708 × 100
	MAE (m)	5.2106 × 10−2	6.1310 × 10−2	6.4482 × 10−2	8.7358 × 10−2
	CVaR95 (m)	5.2805 × 10−1	5.6101 × 10−1	6.0352 × 10−1	1.0650 × 100
	Vε	7.0330 × 10−2	9.8560 × 10−2	9.6979 × 10−2	1.4977 × 10−1

and

Table 12. Vertical error metrics for sensitivity analysis of reward weights.

Case	[kr,1, kr,2, kr,3]	[7.50, 3.75, 3.75]	[3.75, 7.50, 3.75]	[3.75, 3.75, 7.50]	[5.0, 5.0, 5.0]
C1	MXAE (m)	2.5174 × 10−1	1.5607 × 10−1	1.8283 × 10−1	1.7299 × 10−1
	MAE (m)	1.0644 × 10−2	1.0499 × 10−2	6.4236 × 10−3	4.1841 × 10−3
	CVaR95 (m)	1.1995 × 10−1	5.1650 × 10−2	7.6575 × 10−2	4.7757 × 10−2
	Vε	2.4152 × 10−2	4.2438 × 10−4	1.6045 × 10−2	7.4956 × 10−3
C2	MXAE (m)	3.8565 × 10−1	3.0402 × 10−1	3.7452 × 10−1	3.6035 × 10−1
	MAE (m)	1.0350 × 10−2	1.6619 × 10−2	1.0264 × 10−2	1.0904 × 10−2
	CVaR95 (m)	1.1253 × 10−1	9.9581 × 10−2	9.9590 × 10−2	1.0198 × 10−1
	Vε	2.0667 × 10−2	1.5981 × 10−2	1.5450 × 10−2	1.7398 × 10−2
C3	MXAE (m)	3.2619 × 10−1	2.6435 × 10−1	3.2154 × 10−1	3.4549 × 10−1
	MAE (m)	1.0312 × 10−2	1.9053 × 10−2	1.1628 × 10−2	1.2928 × 10−2
	CVaR95 (m)	9.7986 × 10−2	8.2500 × 10−2	9.0378 × 10−2	1.2817 × 10−1
	Vε	1.6776 × 10−2	1.1930 × 10−2	1.5267 × 10−2	2.5180 × 10−2

.

As shown in Figure 19, the relative allocation of reward weights affects the emphasis of path-following performance. Relative to the default setting, increasing the relative weight of k_r,1 improves most horizontal error metrics, especially by reducing the horizontal threshold-exceedance ratio V_ε and horizontal tail error CVaR₉₅. Increasing the relative weight of kr,2 reduces the vertical V_ε, CVaR₉₅, and MXAE compared with the default setting, but it also increases the horizontal error metrics, indicating that emphasizing vertical-error suppression may weaken horizontal path-following performance. After C is increased, the horizontal and vertical error metrics remain generally close to those of the default setting. This occurs because the reverse-motion penalty mainly serves as an auxiliary constraint. When the AUV rarely tends to move opposite to the path direction, increasing k_r,3 has a relatively limited effect on the final path-following error metrics.

Overall, although the setting k_r,1 = k_r,2 = k_r,3 = 5 does not achieve the best value for every individual metric, it provides a more balanced performance between horizontal and vertical path-following errors and avoids significant degradation in either error category. Therefore, this weight setting is adopted as the default reward-function configuration.

4.6. Stochastic Disturbance

To further evaluate the robustness of the proposed controller under stochastic environmental disturbances, an additional test is conducted using a first-order Gauss–Markov disturbance model [28]. The test is performed on a 3D random path with 20 waypoints, where the coordinate origin is defined as the path-starting point. The desired surge velocity is set to $u_d(t)=-0.75\cos (0.01t)+1.25$. The waypoint configuration is summarized in

Table 13. Coordinates of the reference-path waypoints.

Index	1	2	3	4	5	6	7	8	9	10
$x_i^p$ (m)	112.13	307.52	497.67	657.61	813.02	835.02	731.12	619.40	621.43	525.10
$y_i^p$ (m)	0.00	−32.65	−33.77	52.53	155.41	311.17	365.13	209.29	96.76	30.63
$z_i^p$ (m)	4.72	9.98	3.04	8.81	13.64	23.01	20.88	24.59	22.71	19.30
Index	11	12	13	14	15	16	17	18	19	20
$x_i^p$ (m)	408.72	315.79	229.22	138.34	161.95	218.32	283.15	423.69	509.14	595.53
$y_i^p$ (m)	−33.07	−152.37	−264.84	−196.47	−52.90	98.86	197.75	133.86	224.75	297.61
$z_i^p$ (m)	16.14	6.18	4.09	4.13	12.66	12.21	10.74	22.41	14.44	11.01

, and the results are shown in Figure 20, Figure 21 and Figure 22 and

Table 14. Path following error metrics.

	Metric	TIB-CSAC	LOS+SP	SAC	DDPG
Vertical	MXAE (m)	3.0580 × 10−1	3.1465 × 10−1	3.9217 × 10−1	5.5085 × 10−1
	MAE (m)	8.9867 × 10−3	2.3031 × 10−2	2.5653 × 10−2	4.6731 × 10−2
	CVaR95 (m)	6.6192 × 10−2	1.3515 × 10−1	1.3048 × 10−1	2.6511 × 10−1
	Vε	7.9594 × 10−3	4.0323 × 10−2	3.1242 × 10−2	1.2729 × 10−1
Horizontal	MXAE (m)	3.0449 × 100	1.0180 × 101	6.7056 × 100	1.5886 × 101
	MAE (m)	6.2952 × 10−2	5.0292 × 10−1	3.0592 × 10−1	7.7647 × 10−1
	CVaR95 (m)	4.6887 × 10−1	4.9808 × 100	2.6753 × 100	6.6688 × 100
	Vε	1.2621 × 10−1	4.2244 × 10−1	5.8750 × 10−1	9.3249 × 10−1

and

Table 15. Rudder metrics.

Metric	TIB-CSAC	LOS+SP	SAC	DDPG
$R_{\delta }^{sat}$	7.3909 × 10−3	4.1181 × 10−3	5.3944 × 10−2	4.9386 × 10−3
$J_{{\mathrm{\Delta }}^2\delta }$	5.0755 × 10−1	2.2047 × 100	2.1324 × 10−1	1.8090 × 101
$J_{\dot{\delta }}$	1.3140 × 101	2.5033 × 101	3.1093 × 101	1.1789 × 102
$J_{\delta ,u}$	2.5234 × 101	1.4796 × 101	1.7047 × 102	1.6914 × 101
$J_{\mathrm{AUV}}$	7.2190 × 103	7.4023 × 103	7.4040 × 103	7.3943 × 103
$T$	2.2866 × 103	2.3312 × 103	2.3302 × 103	2.3286 × 103

.

$$\begin{array}{l}{\mathit{\tau }}_k^d=\mathrm{clip}({\mathit{\tau }}_k^d,-\mathit{A},\mathit{A})\\ {\mathit{\tau }}_{k+1}^d=\rho {\mathit{\tau }}_k^d+{\mathit{\sigma }}_d\sqrt{1-{\rho }^2}{\mathit{\epsilon }}_k\\ \rho =\exp (-\mathrm{\Delta }t/T_c)\end{array}$$

(66)

where ${\mathit{\tau }}^d={[{\tau }_u^d,{\tau }_v^d,{\tau }_w^d,{\tau }_q^d,{\tau }_r^d]}^T$; A = [15.0, 1.6, 1.6, 5.0, 4.0]^T denotes the theoretical upper bound of each disturbance channel in n Equation (48); ${\mathit{\epsilon }}_k~\mathcal{N}(\mathbf{0},{\mathbf{I}}_5)$; ${\mathit{\sigma }}_d=\mathit{A}/3$; and T_c = 30 s.

As shown in Figure 20, all methods complete the 3D path-following task under stochastic disturbances. The error curves in Figure 22 show that the main error fluctuations are concentrated around path-segment transitions and high-curvature regions. Under stochastic disturbances, TIB-CSAC suppresses both vertical and horizontal error fluctuations more effectively than the baseline methods. As summarized in Table 14, TIB-CSAC achieves the lowest MXAE, MAE, CVaR95, and V_ε, indicating better suppression of peak errors, average errors, tail errors, and threshold-exceeding errors under stochastic environmental disturbances. The supplementary control-related results in Table 15 further show that this improvement is not accompanied by excessive rudder saturation or unreasonable task-level effort. Therefore, the Gauss–Markov disturbance test provides further evidence of the stochastic-disturbance robustness of the proposed method.

5. Conclusions

This study proposes a TIB-CSAC method to improve the X-rudder path-following performance of an underactuated AUV under varying path geometries and desired surge velocities. The proposed method learns the X-rudder control policy directly from two-segment path-following information, thereby avoiding the hierarchical guidance-control architecture and the associated coupled parameter tuning. In addition, a task-informed inductive-bias encoder is introduced to support segment-wise policy learning, improving sample efficiency and overall training quality. A multi-head conservative value-evaluation mechanism is also incorporated to mitigate return drawdowns induced by challenging path-following tasks and to improve tail-stage convergence stability. Overall, compared with baseline SAC, TIB-CSAC reduces $\overline{\mathrm{TRE}}$ and ${\sigma }_{\mathrm{tail}}$ to 0.40 and 0.24 times those of SAC, respectively, while yielding less negative $\overline{\mathrm{AUC}}$ and ${\overline{R}}_{\mathrm{tail}}$ values whose absolute magnitudes are reduced to 0.40 and 0.62 times those of SAC, respectively.

The path-following performance of the proposed method is validated in three representative scenarios with distinct path geometries and desired surge velocities. The results show that TIB-CSAC improves multiple position-error metrics compared with baseline SAC. Averaged over the three cases, the vertical and horizontal MXAE values are reduced by 1.4645 × 10⁻¹ m and 6.4623 m, respectively. The corresponding reductions in MAE are 1.2965 × 10⁻² m and 4.2044 × 10⁻¹ m, while those in CVaR₉₅ are 3.7903 × 10⁻² m and 3.8225 m. In addition, the vertical and horizontal V_ε values are reduced by 1.5675 × 10⁻² and 3.5120 × 10⁻¹, respectively. These results indicate that TIB-CSAC not only improves overall adherence to the reference path but also more effectively suppresses extreme position errors, reduces tail errors, and maintains path-following within the prescribed error threshold, thereby demonstrating improved robustness and reliability.

The current results are mainly obtained through simulation-based evaluation. To reduce the mismatch between the simulator and the real environment, sim-to-real transfer methods provide a promising direction for adapting DRL policies trained in simulation to real AUV systems with limited real-world data [29,30]. For example, Wu et al. improved the real-world deployment of a simulation-trained navigation policy using human-guided interventions and demonstrations [31], while Hsu et al. fine-tuned a simulation-trained policy in a safety-constrained laboratory environment before deploying it on a quadrupedal robot for real-world visual navigation [32]. Therefore, future work will focus on software-system integration, real-world data collection, sim-to-real policy transfer learning, and open-water experiments to further evaluate the proposed method. In addition, the adopted 5-DOF model may be insufficient when rudder faults or asymmetric rudder effectiveness occur. Future work will also investigate fault-tolerant X-rudder control in a full 6-DOF framework by further considering roll dynamics and roll control.