Prediction and Control of Hovercraft Cushion Pressure Based on Deep Reinforcement Learning

Abstract

This paper proposes a deep reinforcement learning-based predictive control scheme to address cushion pressure prediction and stabilization in hovercraft systems subject to modeling complexity, dynamic instability, and system delay. Notably, this work introduces a long short-term memory (LSTM) network with a temporal sliding window specifically designed for hovercraft cushion pressure forecasting. The model accurately captures the dynamic coupling between fan speed and chamber pressure while explicitly incorporating inherent control lag during airflow transmission. Furthermore, a novel adaptive behavior cloning mechanism is embedded into the twin delayed deep deterministic policy gradient with behavior cloning (TD3-BC) framework, which dynamically balances reinforcement learning (RL) objectives and historical policy constraints through an auto-adjusted weighting coefficient. This design effectively mitigates distribution shift and policy degradation in offline reinforcement learning, ensuring both training stability and performance beyond the behavior policy. By integrating the LSTM prediction model with the adaptive TD3-BC algorithm, a fully data-driven control architecture is established. Finally, simulation results demonstrate that the proposed method achieves high accuracy in cushion pressure tracking, significantly improves motion stability, and extends the operational lifespan of lift fans by reducing rotational speed fluctuations.

1. Introduction

During the navigation of a fully skirted hovercraft, the stability of the cushion pressure is crucial for safe operation. However, due to the unique and complex structure of the skirts, modeling and controlling the cushion system poses significant challenges [1,2]. With the rapid advancement of artificial intelligence and the continuous improvement in computational capabilities, researchers are no longer limited to spending extensive time and effort on modeling and controlling such complex systems solely through traditional methods. Instead, deep learning-based modeling [3,4,5] and reinforcement learning-based control [6,7,8] have emerged as increasingly popular research directions. Therefore, exploring how to apply deep reinforcement learning to model and control the complex cushion system of a fully skirted hovercraft constitutes a key focus and challenge of this study.

Numerous studies have been conducted on cushion pressure control for fully skirted hovercrafts [2,9,10]. An adaptive fuzzy sliding-mode control method to regulate the cushion pressure of hovercrafts by controlling the lift fans was proposed in [9]. An adaptive neural network to adjust the parameters of the PID controller was introduced in [2], thereby enhancing the stability control of the cushion pressure. Similarly, reference [10] utilized the DDPG reinforcement learning algorithm to tune the parameters of a PID controller for autonomous cushion height control. However, the aforementioned methods all simplify the complex hovercraft cushion system into a second-order system for controller design, thereby not only neglecting the inherent characteristics of the hovercraft cushion dynamics but also yielding control strategies with limited practical relevance for real-world cushion pressure regulation.

Traditional Reinforcement Learning (RL) relies on interaction with the real environment to make action decisions, and it has achieved preliminary applications in certain low-cost domains such as small unmanned aerial vehicles (UAVs) [11,12] and unmanned ground vehicles (UGVs) [13,14]. A multi-agent reinforcement learning method based on both extrinsic and intrinsic rewards was proposed in [12] to collaboratively control the behavior of UAVs for target encircling tasks. To address the challenge of capturing escaping targets with UGVs, Su et al. [13] decomposed the reward function into individual and cooperative components, optimizing both global and local incentives. By enhancing cooperation among the pursuing UGVs, this approach significantly improves the capture success rate. However, in applications involving large-scale equipment or specific domains where the cost of trial-and-error is prohibitively high, its practical implementation remains challenging [15]. In contrast, offline reinforcement learning eliminates the need for real-time interaction with the environment. Instead, it leverages pre-collected datasets—generated in real or simulated environments—to learn policies and make decisions, thereby mitigating safety risks and reducing training costs associated with online interaction. As a result, offline reinforcement learning has emerged as a growing research direction and has attracted broad attention from researchers and experts across disciplines [16,17].

Deep Learning (DL) models are deep neural network architectures characterized by multiple nonlinear mapping layers, which are capable of extracting features from input signals and uncovering underlying patterns [18]. The Long Short-Term Memory (LSTM) network, an enhanced variant of Recurrent Neural Networks (RNNs), has been widely adopted in numerous time series studies, including applications such as speech recognition [19], multimedia audio and video analysis [20], and road traffic flow prediction [21]. In hovercraft systems, the relationship between fan speed and chamber pressure is highly nonlinear, involving complex physical processes such as fluid dynamics and aerodynamics [2]. However, in the field of hovercraft lift system research, particularly in the use of LSTM for predicting cushion pressure, no relevant studies have been reported to date.

Based on the above analysis, this paper addresses the challenges of modeling the lift system, cushion pressure instability, and control delays in hovercraft by proposing a deep reinforcement learning-based predictive control method for cushion pressure. The main contributions of this work are summarized as follows:

(1)

An LSTM-based predictor with a temporal sliding window is proposed for hovercraft cushion pressure forecasting, effectively capturing the dynamic coupling between fan speed and chamber pressure while explicitly incorporating inherent control lag.

(2)

A novel adaptive behavior cloning mechanism is integrated into the TD3-BC framework, which dynamically balances the RL objective and historical policy constraints via an auto-adjusted weight, thereby mitigating distribution shift and policy degradation in offline settings.

(3)

A fully data-driven control architecture is established by combining the LSTM predictor with the adaptive TD3-BC algorithm, abbreviated as LSTM-TD-BC, enabling accurate cushion pressure tracking, improved motion stability, and extended lift fan operational life through reduced rotational speed fluctuations.

The remainder of this paper is structured as follows: Section 2 introduces the mathematical model of the hovercraft lift system and related preliminary knowledge. Section 3 details the proposed LSTM-TD3-BC-based cushion pressure prediction and control framework. Section 4 presents simulation results and comparative evaluations. Finally, Section 5 concludes the paper and suggests future research directions.

2. Preliminaries and Problem Formulation

2.1. Model Description

The model of the fully skirted hovercraft’s cushion system comprises mathematical representations of the flow and pressure dynamics of the lift fan, the flow and pressure transfer from the booster fan to each individual chamber, and the skirt discharge characteristics and nonlinear flow-pressure balance equations, along with their corresponding solutions [1]. Owing to the structural complexity and modeling challenges associated with the cushion system of a fully skirted hovercraft, the gas within each chamber and the connecting pipelines are assumed to be incompressible. The cushion is modeled as a rectangular structure subdivided in a cross-shaped pattern. A schematic diagram of the overall cushion system layout of the fully skirted hovercraft is provided in Figure 1:

• where $P_i(i=1,2,3,4)$ is the pressure of the four chambers, $P_5$ and $P_6$ are the pressure at the left and right side booster fans, $Q_{FAN1}$ and $Q_{FAN2}$ are the flow rates at the left and right side booster fans, $Q_{INCi}(i=1,2,3,4)$ is the flow rate from the booster fan to each chamber, $Q_{NOZi}(i=1,2)$ is the inflow flow rate of the bow nozzle, $Q_{PUMPi}(i=1,2,3,4)$ is the volume change rate of the air cushion, which is related to the navigation posture and surface characteristics of the hovercraft, $Q_{ICi}(i=1,2,3,4)$ is the flow rate between each chamber, and $Q_i(i=1,2,3,4)$ is the discharge flow rate at the bottom of the apron.

The cushion fans are positioned symmetrically on both sides of the fully skirted hovercraft, with one unit on each side. These fans operate independently yet coordinate to ensure stable motion of the vehicle. The relationship between the flow rate and pressure of the investigated booster fan can be mathematically expressed as follows:

$$\left\{\begin{array}{l}{Q}_{FAN1}=\{-1280\sqrt{\left|P_5-P_{50}\right|}\times sign(P_5-P_{50})-31.6(P_5-P_{50})\}\times N_l/N_f\\ Q_{FAN2}=\{-1280\sqrt{\left|P_6-P_{60}\right|}\times sign(P_6-P_{60})-31.6(P_6-P_{60})\}\times N_r/N_f\end{array}\right.$$

(1)

where $N_l$ and $N_r$ represent the rotational speeds of the left and right fans, $N_f$ represents the rated speed of the fan, and $P_{50}$ and $P_{60}$ represent the static pressure at the raised fan.

2.2. Calculation of Air Cushion Volume Change Rate

During the navigation of a fully cushioned hovercraft, the air cushion pressure and the discharge area interact dynamically. The discharge area is determined by integrating the leakage height along the entire circumference of the bottom apron. To address practical engineering requirements, 35 characteristic points are selected along the bottom periphery of the air cushion to serve as the basis for calculating the discharge area. Figure 2 illustrates the planar structure of the air cushion and the distribution of these characteristic points within the vessel’s coordinate system.

For each chamber, the apron clearance on the X-axis and Y-axis is integrated on the ship’s coordinate system to obtain the apron discharge area, which is expressed as follows:

$$S_i=S_{xi}+S_{yi} (\mathrm{i}=1,2,3,4)$$

(2)

where $S_x={\displaystyle {\int }_0^{3a}{H}_x(x)dx}$, $S_y={\displaystyle {\int }_0^{2b}{H}_y(y)dy}$, $H_x(x)$ and $H_y(y)$ represent the discharge height functions of the flexible apron of the hovercraft on the X and Y axes, respectively. $a$ and $b$ are the cubic fitting coefficients, which are fitted in the X and Y directions as follows:

$$\left\{\begin{array}{l}{S}_x=6.398\times (H_x(0)+H_x(a)+H_x(2a)+H_x(3a))\\ S_y=4.3283\times (H_y(0)+H_y(a)+H_y(2a))\end{array}\right.$$

(3)

Changes in the hovercraft’s attitude or wave-induced motions alter the air volume within the cushion chamber, significantly affecting the air cushion pressure. The rate of change in air volume, in turn, directly influences the mass flow rate throughout the lift system. As illustrated in Figure 2, the air cushion volume is computed based on the distances $T{H}_n (n=1,2,\dots ,35)$ between 35 feature points and the navigation surface. The positions of these feature points in the geodetic coordinate system are given by the following:

$$D{H}_n=-\theta x_n+\phi y_n+z_n$$

(4)

where $(x_n,y_n,z_n)$ is the vertical position of the feature point in the hull coordinate system, $\theta$ and $\phi$ are the longitudinal and transverse inclinations of the hovercraft, respectively. The vertical position of the feature points corresponding to the navigation surface in the geodetic coordinate system is $E{H}_n$; therefore, the distance between the feature points and the navigation surface can be expressed as follows:

$$T{H}_n=D{H}_n-E{H}_n$$

(5)

Then, using the fitting function $OH(T{H}_n,n=1,2,\dots ,35)$, we can calculate the air cushion volume at a certain chamber (taking air cushion 1 as an example) as follows:

$$V=O{H}_1\times S_{c1}$$

(6)

where $O{H}_1$ is the average height between the bottom of air cushion 1 and the sailing surface, given by the following equation:

$$\begin{array}{cc}O{H}_1& =C_{c1}(T{H}_1+T{H}_8+T{H}_{12}+T{H}_{15})+C_{c2}(T{H}_{13}+T{H}_{14}+T{H}_9+T{H}_{10})\hfill \\ & \hspace{0.17em}+C_{c3}(T{H}_{11}+T{H}_{16}+0.25T{H}_{33}+T{H}_{32})\hfill \end{array}$$

(7)

where $C_{c1}$, $C_{c2}$, and $C_{c3}$ are approximately obtained from experiments. If $S_{c1}$ is the horizontal projection area of air cushion 1, then the volume of air cushion 1 at a certain moment can be expressed as follows:

$$V_1=O{H}_1\times S_{c1}$$

(8)

If $\Delta t$ is the sampling time interval and $V(old)$ is the volume of the previous time, then the rate of change in the volume of each air cushion can be obtained as follows:

$$Q_{PUMPi}=(V_i-V_i(old))/\Delta t (\mathrm{i}=1,2,3,4)$$

(9)

Within the booster system, the main duct of the booster fan and the four chambers are collectively described by a set of six nonlinear continuous equations that govern the system dynamics at any given time. These equations can be expressed as follows:

$$\left\{\begin{array}{l}-Q_{INC1}-Q_{INC2}+Q_{NOZ1}+Q_{FAN1}=0\\ -Q_{INC3}-Q_{INC4}+Q_{NOZ2}+Q_{FAN2}=0\\ Q_{INC1}+Q_{IC1}-Q_{IC2}+Q_{PUMP1}+Q_1=0\\ Q_{INC2}+Q_{IC2}-Q_{IC3}+Q_{PUMP2}+Q_2=0\\ Q_{INC3}+Q_{IC3}-Q_{IC4}+Q_{PUMP3}+Q_3=0\\ Q_{INC4}+Q_{IC4}-Q_{IC1}+Q_{PUMP4}+Q_4=0\end{array}\right.$$

(10)

According to Bernoulli’s principle, an increase in fluid velocity corresponds to a decrease in fluid pressure [22]. Based on this relationship, the flow rate in Equation (10) can be expressed in terms of the pressure in each chamber and the corresponding flow cross-sectional area as follows:

$$\left\{\begin{array}{l}{Q}_{INC1}=589\sqrt{\left|P_5-P_1\right|}\mathrm{sgn}(P_5-P_1)\\ Q_{INC2}=589\sqrt{\left|P_5-P_2\right|}\mathrm{sgn}(P_5-P_2)\\ Q_{INC3}=589\sqrt{\left|P_6-P_3\right|}\mathrm{sgn}(P_6-P_3)\\ Q_{INC4}=589\sqrt{\left|P_6-P_4\right|}\mathrm{sgn}(P_6-P_4)\end{array}\right.$$

(11)

$$\left\{\begin{array}{l}{Q}_{NOZ1}=-346\sqrt{\left|P_5\right|}\mathrm{sgn}(P_5)\\ Q_{NOZ2}=-346\sqrt{\left|P_6\right|}\mathrm{sgn}(P_6)\end{array}\right.$$

(12)

$$\left\{\begin{array}{l}{Q}_{IC1}=675\sqrt{\left|P_4-P_1\right|}\mathrm{sgn}(P_4-P_1)\\ Q_{IC2}=338\sqrt{\left|P_1-P_2\right|}\mathrm{sgn}(P_1-P_2)\\ Q_{IC3}=675\sqrt{\left|P_2-P_3\right|}\mathrm{sgn}(P_2-P_3)\\ Q_{IC4}=338\sqrt{\left|P_3-P_4\right|}\mathrm{sgn}(P_3-P_4)\end{array}\right.$$

(13)

$$\left\{\begin{array}{l}{Q}_1=-S_1\times 14.5\sqrt{\left|P_1\right|}\mathrm{sgn}(P_1)\\ Q_2=-S_2\times 14.5\sqrt{\left|P_2\right|}\mathrm{sgn}(P_2)\\ Q_3=-S_3\times 14.5\sqrt{\left|P_3\right|}\mathrm{sgn}(P_3)\\ Q_4=-S_4\times 14.5\sqrt{\left|P_4\right|}\mathrm{sgn}(P_4)\end{array}\right.$$

(14)

We substitute the above flow calculation expressions into Equation (10), and use the apron discharge area $S_i(i=1,2,3,4)$ and chamber volume change rate $Q_{PUMPi}(i=1,2,3,4)$, $N_l$, and $N_r$ as action inputs. The system of equations contains six unknown pressures, which are solved using Newton’s iterative method.

2.3. Markov Decision Process Modeling

The Markov Decision Process (MDP) is widely used to model reinforcement learning tasks. When an intelligent agent performs a task, it begins by interacting with the environment. Starting from a given initial state, a new state is generated according to the state transition probability. Simultaneously, the environment returns a reward signal, based on which the agent updates its action policy. This process repeats iteratively. Through multiple such interactions, the agent gradually learns an optimal policy to accomplish the designated task [23].

First, the air cushion system of the hovercraft is characterized by the following nonlinear state space model:

$$\left\{\begin{array}{l}\dot{X}(t)=F(X(t),U(t))+\zeta (t)\\ Y(t)=C\ast X(t)\end{array}\right.$$

(15)

where $X(t)={[P_1,P_2,P_3,P_4,N_l,N_r]}^T$ is the system state variable vector, and $U(t)={[\Delta N_l,\Delta N_r]}^T$ is the action input, representing the incremental speed of the fan. $Y(t)={[P_1,P_2,P_3,P_4]}^T$ is the output vector, and $\zeta (t)$ represents system noise and external disturbances. $C$ is the output matrix.

Then, transform the control problem into MDP as follows:

$${\rm M}=(S,A,P,R,\gamma )$$

(16)

where $S$ represents the state space, consisting of four chamber pressures and two fan speeds. $A$ represents the action space, consisting of two incremental fan speeds. $P$ represents the probability of state transition, approximated by the LSTM model. $\gamma$ is the discount factor, which is a real number between 0 and 1. $R$ is the reward function, and the goal of pressure prediction control in the air chamber of an air cushion vehicle is to control the pressure amplitude of the air chamber to change within a small range and with a small frequency under the action of two independent lift fans. Therefore, the reward function can be designed as follows:

$$R_t=-\left({\displaystyle \sum _{i=1}^4{w}_i}\left|P_i^{(t+1)}-P_{target}\right|+{\lambda }_a{\displaystyle \sum _{j=1}^2\left|U_j^{(t)}\right|}+{\lambda }_b{\displaystyle \sum _{k=1}^4\left|\Delta P_k^{(t)}\right|}\right)$$

(17)

where $P_{target}$ is the target pressure of the air chamber, which is related to the actual weight and navigation conditions of the hovercraft. The term $|🞄|$ represents absolute value. ${\lambda }_a$ and ${\lambda }_b$ are positive real numbers that, respectively, control the smoothness of the action and the penalty intensity of pressure changes.

3. Prediction and Control of Air Chamber Pressure in Air Cushion Vehicle Based on LSTM-TD3-BC

To enable real-time prediction and active control of the chamber pressure in a fully cushioned hovercraft, this section employs an improved LSTM network to forecast the chamber pressure. This approach is designed to capture the complex spatiotemporal dynamics inherent in the hovercraft’s cushion system, thereby establishing a model foundation for subsequent reinforcement learning-based control of the cushion pressure. Building on the LSTM-based prediction model for the lift system, offline policy optimization is performed using the TD3-BC reinforcement learning algorithm. This enhances both the stability and robustness of the system while maintaining accurate control over the cushion lift pressure.

A schematic diagram of the hovercraft chamber pressure prediction and control framework based on the LSTM-TD3-BC algorithm is presented in Figure 3. The LSTM Prediction Model serves as the environment for the RL agent, forecasting chamber pressures based on the history of system states. The TD3 Algorithm serves as the primary RL controller, generating control actions (fan speed increments) based on the current state and its Q-value estimates. The Behavior Cloning (BC) Framework regularizes the TD3 policy by constraining it towards actions demonstrated in the historical dataset, using an adaptive weight to mitigate distribution shift.

3.1. LSTM Deep Learning Model

LSTM network is an improved model of recurrent neural network (RNN). LSTM learns long-term correlated information by introducing gating mechanism to control the forgetting and flow of information, thereby alleviating the gradient explosion and vanishing problems of RNN during training. Therefore, LSTM is an ideal choice for processing time-series data [24]. The overall structure of LSTM is shown in Figure 4. The LSTM network mainly consists of four modules, namely input gate, forget gate, memory unit, and output gate, and continuously updates the states of the four modules to process the sequence data in each time step.

In Figure 4, $X_t$ is the input data information at time $t$, $H$ is the hidden state information, $C$ is the memory unit state, and $\tanh$ is the hyperbolic tangent activation function. $\sigma$ is a $sigmoid$ activation function, which compresses the input value between 0 and 1 to ensure whether information is retained or forgotten.

The forget gate controls the retention and forgetting of information in the memory unit. Based on the current state and the previous hidden state, obtain an information value representing the degree of forgetting. The higher the value, the greater the probability of the corresponding information being forgotten in memory unit $C_{t-1}$. This mechanism allows LSTM to selectively retain long-term dependent available information, discard low value data, and effectively process complex sequence data. The calculation formula is expressed as follows:

$$F_t=\sigma (W_{xf}{X}_t+W_{hf}{H}_{t-1}+B_f)$$

(18)

where $F_t$ is the output of the forget gate, $W_{xf}$ is the weight matrix between the current input and the forget gate, $W_{hf}$ is the weight matrix between the historical output and the forget gate, and $B_f$ is the bias term of the forget gate.

The input gate serves to update the memory cell state by determining whether new information should be retained. It processes the previous hidden state and the current input through a sigmoid activation function, producing output values between 0 and 1, where 0 signifies insignificance and 1 indicates importance. Meanwhile, the same inputs are processed by a $\tanh$ function to generate candidate memory cell states within the range of −1 to 1. The output of the input gate is expressed as follows:

$$I_t=\sigma (W_{xi}{X}_t+W_{hi}{H}_{t-1}+B_i)$$

(19)

where $I_t$ denotes the output of the input gate, $W_{xi}$ represents the weight matrix between the input gate and the input, $W_{hi}$ is the weight matrix connecting the previous hidden state to the input gate, and $B_i$ denotes the bias term of the input gate.

The candidate cell state represents the extent to which the new input at the current time step can influence the memory cell state. It is computed as follows:

$${\tilde{C}}_t=\tanh (W_{xc}{X}_t+W_{hc}{H}_{t-1}+B_c)$$

(20)

where ${\tilde{C}}_t$ is the candidate cell state, $W_{xc}$ denotes the weight matrix between the input and the candidate state, $W_{hc}$ represents the weight matrix associated with the previous hidden state, and $B_c$ is the bias term.

The memory cell serves as the core component of the entire LSTM network. It is capable of not only regulating the flow and update of information but also storing and transmitting data. After incorporating the information from the forget gate and the input gate, the memory cell state can be updated as follows:

$$C_t=F_t{C}_{t-1}+I_t{\tilde{C}}_t$$

(21)

The output gate regulates the flow of information passed to the next time step. It determines which parts of the cell state should be output Via a sigmoid activation function. The output of the gate is expressed as follows:

$$O_t=\sigma (W_{xo}{X}_t+W_{ho}{H}_{t-1}+B_o)$$

(22)

where $W_{xo}$ denotes the weight matrix between the input and the output gate, $W_{ho}$ represents the weight matrix connecting the previous hidden state to the output gate, and $B_o$ is the bias term of the output gate. Subsequently, the hidden state $H_t$ is obtained by multiplying the output gate with the cell state processed through a $\tanh$ activation function, as follows:

$$H_t=O_t\tanh (C_t)$$

(23)

3.2. TD3-BC Reinforcement Learning Algorithm

To mitigate the high interaction cost and safety risks associated with online reinforcement learning, researchers have introduced offline reinforcement learning [25]. Offline reinforcement learning trains a policy $\pi$ using a pre-collected dataset. The agent executes an action $a$ in state $s$ according to the policy, and the expected cumulative return after following the policy $\pi$ is given by the following:

$$Q_{\pi }(s,a)=E_{P,\pi |s_0=s,a_0=a}\left[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)}\right]$$

(24)

It represents the expected cumulative discounted return when the agent starts from state $s$, executes action $a$, and thereafter follows policy $\pi$. The expectation $E_{P,\pi |s_0=s,a_0=a}$ accounts for the inherent randomness in both the policy and the environment’s state transitions.

Equation (24) is updated Via the Bellman equation as follows:

$$Q(s,a)=r(s,a)+\gamma \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })$$

(25)

This provides a recursive decomposition of the value. It states that the Q-value for a state–action pair $(s,a)$ can be broken down into the immediate reward $r(s,a)$ and the discounted expected value of the next state $\underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime })$, where $s^{\prime }$ is the state resulting from taking action $a$ in state $s$. This recursion is fundamental to most RL algorithms, as it enables iterative estimation and improvement of the Q-function.

Equation (25) can be solved by minimizing the mean squared Bellman error, which is defined as follows:

$$B_l=\frac{1}{2}{E}_{s,a,r,s^{\prime }}{\left[Q(s,a)-(r+\gamma \underset{a^{\prime }}{\max }Q(s^{\prime },a^{\prime }))\right]}^2$$

(26)

This serves as the primary loss function for training the Critic network in value-based RL methods like TD3. By minimizing this loss, the parameters of the Q-network are adjusted so that its estimates become increasingly consistent with the Bellman Equation (25).

Essentially, the objective of offline reinforcement learning is to learn a policy that outperforms the behavior policy, thereby achieving improved performance upon deployment in real-world interactions. However, due to the issue of distribution shift, most offline RL algorithms tend to adopt conservative strategies, which often limits their ability to capture the full complexity of the policy when relying solely on behavioral cloning. Therefore, striking an effective balance between policy improvement and mitigating distribution shift constitutes a central challenge in the field of offline reinforcement learning.

TD3-BC is an offline reinforcement learning algorithm that integrates the advantages of both TD3 and BC [26]. The TD3 component employs an Actor–Critic architecture along with a replay buffer of collected experiences. It updates the policy based on the value of the Q-function, prioritizing state–action pairs with high estimated returns regardless of the current policy behavior. This flexibility enables the algorithm to leverage large, sparse rewards effectively, thereby improving the success rate. Through advanced policy exploration, it derives an optimized policy, denoted as $\pi$. The BC component consists of an expert replay buffer and an Actor network, which learns from expert demonstrations to facilitate knowledge transfer.

TD3-BC incorporates a BC regularization term to ensure that the policy optimizes the Q-function while remaining close to the actions observed in the historical dataset. This prevents the algorithm from taking overly risky actions in regions with insufficient data coverage. Consequently, the policy optimization objective of TD3-BC is to maximize the following policy objective function:

$$L_{\pi }=E_{(s,a)\in D}\left[Q_{\pi }(s,\pi (s))-\rho {\Vert \pi (s)-a\Vert }^2\right]$$

(27)

where $D$ denotes the dataset used for policy training, and $\rho$ is the behavior regularization coefficient that controls the degree of conservatism by constraining the learned policy to remain close to the behavior policy. By default, $\Vert 🞄\Vert$ refers to the $L_2$ norm (Euclidean norm), which has the same meaning when used in subsequent formulas.

3.3. Design of LSTM-TD3-BC-Based Algorithm for Cushion Pressure Prediction and Control

An LSTM network is employed to model the dynamic relationship between fan speeds and chamber pressures. The input to the model is a 6-dimensional state vector $X_t$ over a historical time window of length T, which consists of four chamber pressure values and two fan speed values, denoted as $X_t=[s_{t-T},s_{t-T+1},\dots ,s_{t-1}]$. The output is the predicted four chamber pressure values ${\widehat{Y}}_t$ at the subsequent time step, denoted as ${\widehat{Y}}_t={[\widehat{P_{1t}},\widehat{P_{2t}},\widehat{P_{3t}},\widehat{P_{4t}}]}^T$.

To enhance the robustness of the LSTM prediction model, the Huber loss function is employed as follows:

$$L_{\delta }(Y,\widehat{Y})=\left\{\begin{array}{ll}{\displaystyle \frac{1}{2}}\Vert Y-\widehat{Y}{\Vert }^2\hfill & \mathrm{if} \Vert Y-\widehat{Y}\Vert ⩽\delta \hfill \\ \delta \Vert Y-\widehat{Y}\Vert -{\displaystyle \frac{1}{2}}{\delta }^2\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(28)

where $\delta$ is a positive constant.

The training process of the LSTM prediction model follows systematic data processing and deep learning optimization methodology, with key steps summarized in Algorithm 1. The procedure begins with standardized preprocessing of the raw time-series data from the air cushion vehicle lift system. The mean $\mu$ and standard deviation $\sigma$ of each channel are computed to normalize the dataset into a zero-mean, unit-variance form.

Subsequently, input-output sample pairs are constructed using a temporal window slicing technique to predict the cushion pressure at the next time step. The model architecture employs a two-layer LSTM encoder, with a dropout layer (rate = 0.2) applied after the second LSTM layer to prevent overfitting. The Huber loss function is utilized to combine the advantages of both Mean Absolute Error (MAE) and Mean Square Error (MSE) loss functions, thereby enhancing the model’s robustness to outliers. The Adam optimizer is adopted with a learning rate $\eta =0.001$, and momentum coefficients ${\beta }_1=0.9$ and ${\beta }_2=0.99$. The hyperparameters in Algorithm 1 were chosen based on established practices in deep learning for time-series problems and limited empirical tuning on our validation set. For instance, the time window size T = 20 was selected to balance the need to capture the system’s short-term dynamics with computational efficiency.

Algorithm 1 Training Procedure for LSTM-Based Cushion Pressure Prediction Model
Input:	$\mathrm{Dataset} D_l=\{t,P_1,P_2,P_3,P_4,N_l,N_r\}$
	Time window size T = 20 $\mathrm{Hyperparameters}: \{\eta ,{\beta }_1,{\beta }_2,B,E,\delta \}$
Output:	Trained LSTM prediction model
1	data preprocessing:
2	$\mu ,\sigma \leftarrow$ Calculate the mean and standard deviation of each channel
3	$D\_norm\leftarrow (D_l-\mu )/\sigma$
4	$X\_train,X\_val,Y\_tran,Y\_val\leftarrow$ Split training and testing sets
5	Initialize LSTM model M:
6	Input layer: (T,6) $\leftarrow$ Time window, feature dimension
7	LSTM1: Unit 128
8	Dropout layer: 0.2
9	LSTM2: Unit 128
10	$\mathrm{Output} \mathrm{layer}:Dense(4)\leftarrow$ Fully connected layer, with dimensions of 4
11
12	for epoch to E do:
13	$\mathrm{for} \mathrm{batch} \mathrm{in} \mathrm{Batch} \mathrm{Generator} (X\_train,Y\_train,B$) do:
14	$\widehat{Y}=M(X\_batch)\leftarrow$ forward propagation
15	$L=HuberLoss(Y\_batch,\widehat{Y};\delta )\leftarrow$ Loss calculation
16	$M.backward(Loss)\leftarrow$ backpropagation
17	$\theta \leftarrow Adam(\theta ,\nabla L;\eta ,{\beta }_1,{\beta }_2)$, parameters update
18	end for
19
20	$val\_loss\leftarrow \mathrm{evaluate} (M,X\_val,Y\_val)$
21	$best\_model\leftarrow M$, Save LSTM model
22	end for
24	Return LSTM pressure prediction model $M$

In the pressure control system of the hovercraft lift system, this paper adopts the TD3-BC algorithm as the core reinforcement learning framework. By integrating the TD3 with BC regularization, the algorithm effectively addresses the issues of extrapolation error and policy degradation in conventional offline reinforcement learning. To dynamically balance the strength of policy optimization and historical behavior imitation, and to ensure an effective trade-off between exploring novel actions and maintaining behavioral stability, this paper introduces an adaptive BC regularization method to update the objective function of the Actor network. Therefore, Equation (27) can be rewritten as follows:

$$L_{\phi }=E_{(s,a)\sim D_r}\left[-\lambda Q_{{\theta }_1}(s,{\pi }_{\phi }(s))+\alpha {‖{\pi }_{\phi }(s)-a‖}^2\right]$$

(29)

where $\alpha >0$ is BC regularization strength, and $\lambda$ is the adaptive weighting coefficient defined as follows:

$$\lambda ={\displaystyle \frac{\alpha }{E_{(s,a)\sim D_r}\left[‖Q(s,{\pi }_{\phi }(s))‖\right]+{\lambda }_0}}$$

(30)

where ${\lambda }_0$ is a sufficiently small constant to prevent division by zero.

Remark 1.

The primary role of the adaptive weighting coefficient  ${\lambda }_0$ is to dynamically balance the relative importance between the reinforcement learning objective and the behavior cloning constraint. When the policy performs poorly, indicated by unreliable and low Q-value estimates, ${\lambda }_0$ increases to strengthen the influence of behavioral regularization. This encourages the policy to imitate expert behaviors from the historical dataset, thereby mitigating the risk of risky actions. Conversely, when the policy performs well, ${\lambda }_0$ decreases to relax the behavior cloning constraint, allowing greater exploration for potentially superior strategies beyond historical behaviors.

The Critic network employs a twin Q-network architecture and is trained by minimizing the temporal difference (TD) error, which is derived from the Bellman equation:

$$L({\theta }_i)=E_{(s,a,r,s^{\prime })\sim D_r}\left[{(Q_{{\theta }_i}(s,a)-y)}^2\right]$$

(31)

where $y=r+\gamma (1-d){\min }_{i=1,2}{Q}_{{\theta }_i}(s^{\prime },\mathrm{clip}({\pi }_{{\phi }^{\prime }}( \mathrm{s} \prime )+{\lambda }_0,-1,1))$. The target networks are updated via a soft update mechanism to ensure training stability, thereby guaranteeing control precision in hovercraft lift pressure control task and further enhancing system robustness. The pseudo-code of the TD3-BC-based cushion pressure control algorithm is presented in Algorithm 2:

Algorithm 2 TD3-BC-Based Control for Hovercraft Cushion Pressure
Input:	$\mathrm{Historical} \mathrm{dataset} D_r=\{S,A,R,S^{\prime },done\}$
	LSTM prediction model M $\mathrm{Hyperparameters}:\{\alpha ,\gamma ,\tau ,\sigma ,c,d,B\}$
Output:	$\mathrm{Optimized} \mathrm{control} \mathrm{strategy} {\pi }_{\phi }$
1	Initialization:
2	Experience buffer zone $\leftarrow$ Sampling from D
3	$\mathrm{Randomly} \mathrm{initialize} \mathrm{Actor} \mathrm{network} \pi \_\phi$$, \mathrm{Critic} \mathrm{network} Q\_\theta 1, Q\_\theta 2$
4	$\mathrm{Initialize} \mathrm{target} \mathrm{network}: {\phi }^{\prime }\leftarrow \phi , \theta 1^{\prime }\leftarrow \theta 1, \theta 2 \prime \leftarrow \theta 2$
5
6	for count in maximum iterations do:
7	$\{(s, a, r, s^{\prime }, done)\}\sim B$, Batch sampling
8	$a^{\prime }\leftarrow \pi \_{\phi }^{\prime }(s^{\prime })+clip(N(0,\sigma ),-c,c)$, Target action calculation
9	$Q\_target\leftarrow \min (Q\_\theta 1^{\prime }(s^{\prime },a^{\prime }), Q\_\theta 2^{\prime }(s^{\prime },a^{\prime }))$, Target Q-value calculation
10	Update Critic Network:
11	$\theta 1\leftarrow Adam(\nabla \_\theta 1[{(Q\_\theta 1(s,a)-y)}^2])$
12	$\theta 2\leftarrow Adam(\nabla \_\theta 2[{(Q\_\theta 2(s,a)-y)}^2])$
13	$\mathrm{if} count/d=0$ then:
14	Adaptive BC weight update:
15	$Q\_val\leftarrow Q\_\theta 1(s,\pi \_\phi (s))$
16	$\lambda \leftarrow \alpha /(mean(\left|Q\_vals\right|)+1\times {10}^{-5})$
17	Actor loss for behavior cloning:
18	$L\_actor\leftarrow -\lambda mean(Q\_vals)+\alpha MSE(\pi \_\phi (s),a)$
19	$\phi \leftarrow Adam(\nabla \_\phi \mathrm{L}\_\mathrm{actor})$, Update actor
20	Soft update target network:
21	$\theta 1^{\prime }\leftarrow \tau \theta 1+(1-\tau )\theta 1^{\prime }$
22	$\theta 2^{\prime }\leftarrow \tau \theta 2+(1-\tau )\theta 2^{\prime }$
23	${\phi }^{\prime }\leftarrow \tau \phi +(1-\tau ){\phi }^{\prime }$
24	end if
25	end for
26	$\mathrm{Return} \mathrm{control} \mathrm{strategy} \pi \_\phi$

4. Experimental Results and Discussion

To evaluate the effectiveness of the proposed algorithm, this study utilizes a fully skirted hovercraft simulation platform to generate a cushion dataset during navigation for training the LSTM prediction model. The process is further integrated with the offline reinforcement learning algorithm TD3-BC to make reinforcement learning-based control decisions for cushion pressure regulation. The experiments were conducted in the following environment: hardware setup includes an Intel Core i7-13700KF CPU and an NVIDIA RTX 4060Ti GPU; software environment consists of Python 3.9, PyTorch 1.12.1, and CUDA 11.6; the simulation platform employs PyCharm 2022 for algorithm development and Visual Studio 2022 for hovercraft simulator integration.

The experiments utilized a simulator closely mimicking a real-world hovercraft to generate the dataset, which consists of 101,922 samples. Each sample includes a 7-dimensional feature vector: $X(t)={[t,P_1,P_2,P_3,P_4,N_l,N_r]}^T$. All features, except time, were normalized using min-max scaling. Input sequences were constructed by applying a sliding window method with a history of 20-time steps, and the pressure values at the next time step were used as the output. The dataset was split into training and validation sets for the LSTM model in an 8:2 ratio. Subsequently, an adaptive TD3-BC algorithm was employed, integrating offline data and the LSTM prediction model, to achieve stable cushion pressure control through reinforcement learning-based decision-making.

This study adopts a simulation-based, data-driven experimental design, which is structured into three consecutive phases: (1) data collection, (2) predictive model training, and (3) offline reinforcement learning control. The specific details are as follows:

4.1. Simulation Study on Cushion Lifting Characteristics

The cushion stability of a fully skirted hovercraft is fundamental to its safe navigation. After developing a mathematical model of the lift system, simulation was conducted to acquire pressure variation data of each air chamber under the influence of lift fan speeds and external environmental disturbances. The initial speed of the hovercraft was set to 25 knots, while the rotational speeds of the port and starboard lift fans were fixed at 2020 rpm. Partial simulation results illustrating the pressure changes in the air chambers of the cushion system are shown in Figure 5 and Figure 6.

As can be observed from Figure 5 and Figure 6, under constant lift fan speeds, the hovercraft exhibits significant pressure fluctuations and vibrations due to wave-induced disturbances and skirt air leakage. These effects result in considerable vertical oscillations during navigation, which not only impair crew operational safety but also compromise the hovercraft’s overall seakeeping performance.

4.2. LSTM Air Cushion Pressure Prediction

To validate the effectiveness of the proposed LSTM-based cushion pressure prediction model, its parameter configuration is provided in

Table 1. LSTM Prediction Model Settings.

Parameter Name:	Set Values	Instructions
Input dimension	$(20,6)$	Time step $\times$ number of features
Output dimension	$4$	prediction targets $\left(P1, P2, P3, P4\right)$
network architecture	$128\to 128$	with 128 hidden units per layer
Dropout	0.2	Used to prevent overfitting
Optimizer	Adam	Adaptive Moment Estimation Optimizer
Batch Size	128	Ensure training stability
Number of training epochs	120	Cooperate with the early stop method
loss function	Huber Loss	Huber Loss is more robust to outliers

. Using historical system states from the past 20 time steps and trained over 120 epochs on offline data, the model accurately predicts the pressure values of the four air chambers at the next time step.

The training process of the prediction model is illustrated in Figure 7. This study employs the Huber loss function, which combines the smooth convergence properties of MSE with the robustness of MAE against outliers. As can be observed, both the training loss and validation loss decrease synchronously as the number of iterations increases, converging after approximately 20 epochs without exhibiting overfitting. The stable convergence of the training process lays a foundation for obtaining a high-precision prediction model.

The pressure prediction results are shown in Figure 8. To evaluate the prediction accuracy of the LSTM algorithm for the pressure in each air chamber, the mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R²) were adopted as evaluation metrics, as summarized in

Table 2. Performance Indicators of LSTM Model for Pressure Prediction in Each Gas Chamber.

Air Chamber	MAE (Pa)	RMSE (Pa)	${\mathit{R}}^2$
$P_1$	2.2463	2.9957	0.8302
$P_2$	1.8726	2.5183	0.9004
$P_3$	2.4950	3.5560	0.6110
$P_4$	2.3849	3.0647	0.7079
Average	2.2497	3.0337	0.7624

. The proposed LSTM prediction model demonstrated outstanding performance on the test set: it achieved an average MAE of 2.2497 Pa and an average RMSE of 3.0337 Pa across the four air chambers, with a mean R² value of 0.7624. These results confirm the high prediction accuracy of the proposed method and establish a reliable foundation for subsequent offline reinforcement learning.

4.3. TD3-BC Reinforcement Learning Decision Control

To evaluate the performance of the TD3-BC algorithm in hovercraft cushion pressure control, this study compares the control effectiveness of the LSTM-based prediction model integrated with both the adaptively weighted TD3-BC and the conventional TD3 algorithm. We repeated the training process of the core adaptive TD3-BC algorithm multiple times with different random seeds, while keeping the dataset and LSTM prediction model fixed. All algorithms were executed under the same testing environment, and the parameter settings of the adaptive TD3-BC reinforcement learning approach are provided in

Table 3. Adaptive TD3-BC Reinforcement Learning Parameter Settings.

Parameter Name	Set Values	Instructions
Actor Network Structure	256-256-2	Strategic Network Structure
Critic network structure	256-256-1	Value Network Structure
BC coefficient $\alpha$	2.5	BC regularization strength
Discount factor	0.99	Discount rate for future rewards
$\mathrm{Reward} \mathrm{function} \mathrm{parameters} {\lambda }_a$	0.12	Strategy smoothness parameter
$\mathrm{Reward} \mathrm{function} \mathrm{parameters} {\lambda }_b$	0.6	Stability parameter of chamber pressure
Target network update coefficient $\tau$	0.005	Ensure training stability
Replay buffer size	300,000	Ensure sample diversity
batch size $B$	256	Training batch size
Actor learning rate	3 × 10−4	Adam optimizer learning rate
Critic learning rate	3 × 10−4	Adam optimizer learning rate

.

As shown in Figure 9, compared to the uncontrolled cushion pressure and the LSTM-TD3 control algorithm, the proposed adaptively weighted LSTM-TD3-BC reinforcement learning algorithm effectively maintains the chamber pressure within the desired range with smoother adjustments. Figure 10 illustrates a comparison of lift fan speeds under different control algorithms. In contrast to the LSTM-TD3 method, the proposed adaptive LSTM-TD3-BC results in smaller fluctuations in fan speed, which not only stabilizes cushion pressure but also contributes to prolonged fan service life. The instantaneous reward curves are depicted in Figure 11. Although both methods achieve stable chamber pressure, the adaptive LSTM-TD3-BC algorithm demonstrates more consistent learning performance, owing to the effective adjustment of adaptive weights, as further supported by Figure 12.

In summary, the proposed adaptive LSTM-TD3-BC reinforcement learning-based predictive control method for hovercraft cushion pressure enables accurate prediction of chamber pressure and facilitates stable control decisions for cushion pressure regulation by leveraging historical experiential data and a reliable prediction model. Stable and autonomous control of cushion pressure is essential for the navigation of hovercrafts. Therefore, effective and smooth cushion pressure control lays a critical foundation for surface navigation control tasks. While the overall data collection was a single instance, the policy optimization and evaluation demonstrate stable and repeatable performance under random initialization, and that future work will involve testing across a wider variety of sea states and operational conditions.

5. Conclusions and Future Work

This paper proposed a deep reinforcement learning-based predictive control scheme for cushion pressure in fully skirted hovercraft, aiming to address the challenges of modeling complexity, pressure instability, and control delays in the lift system. Firstly, an LSTM network with a fixed-time window was employed to accurately predict chamber pressure, effectively capturing the dynamic coupling between fan speed and chamber pressure. Secondly, a novel adaptive behavior cloning mechanism was embedded into the TD3-BC framework, which dynamically balances the reinforcement learning objective and historical policy constraints through an auto-adjusted weighting coefficient, thereby effectively mitigating distribution shift and policy degradation in the offline RL setting. By integrating the LSTM prediction model with the adaptive TD3-BC offline reinforcement learning algorithm, stable cushion pressure control was achieved. Finally, simulations demonstrated that the proposed method not only improves the accuracy and robustness of chamber pressure prediction but also achieves smooth pressure control while reducing fluctuations in lift fan speed, thus extending the service life of the equipment. Future research will focus on refining the proposed method and implementing the algorithm in real-world cushion pressure control applications for fully skirted hovercraft.

While this study demonstrates the effectiveness of the proposed method, some limitations remain. Validation was conducted solely via simulation, creating a sim-to-real gap, and control performance depends heavily on the quality of the offline dataset. Future work will focus on implementing the algorithm on a real hovercraft platform through hardware-in-the-loop testing. We will also investigate adaptive data collection strategies, extend the framework to more extreme sea conditions, and incorporate additional practical constraints such as actuator saturation and safety guarantees.