Dual-Objective Pareto Optimization Method of Flapping Hydrofoil Propulsion Performance Based on MLP and Double DQN

Abstract

To address the inherent complexities of underwater operating environments and achieve the design of a highly efficient, energy-saving flapping hydrofoil, this paper proposes an intelligent agent-based model for real-time parametric optimization. A non-parametric surrogate model based on a Multilayer Perceptron (MLP) is established using data samples of multi-dimensional flapping hydrofoil geometric parameters obtained through Computational Fluid Dynamics (CFD) simulations. An improved Double Deep Q-Network (DDQN) algorithm incorporating Pareto frontier information is deployed within the surrogate model to obtain the Pareto optimal solution set for propulsion efficiency and average input power, and a set of propulsion parameter combinations with error ranges between 0.24% and 1.27% across continuous intervals was obtained. Experimental results demonstrate that the proposed MLP-DDQN method is capable of learning the domain-wide optimal solution within the experimental environment, satisfying the Pareto optimality between propulsion efficiency and average input power. Further analysis of the flow field around the flapping hydrofoil under the obtained optimal parameter combination revealed that the presence of stable and continuously attached vortex structures on the wing surface is the intrinsic mechanism responsible for its superior propulsion performance.

1. Introduction

With the growing demand for marine resource exploration and underwater operations, efficient and flexible underwater propulsion systems have become a key research focus [1]. Compared to propellers [2], flapping hydrofoils have emerged as a major research subject in the design of underwater vehicles due to their excellent fluid adaptability and energy conversion efficiency [3]. As a core propulsion component, the hydrodynamic performance of flapping hydrofoils directly determines the overall system performance. Therefore, investigating the propulsion performance of the flapping hydrofoil and the underlying flow mechanism responsible for the enhanced performance is of great significance for developing efficient underwater propulsion systems [4].

The influence of geometric parameters on hydrodynamic performance has consistently been a focal point of research [5]. Previous studies have demonstrated that specific structural configurations, such as the pitch-axis location and leading-edge shape, are crucial for optimizing key performance metrics, including thrust, power coefficient, and propulsion efficiency [6]. Zhao et al. [7] investigated the propulsion performance of NACA 0015, examining the influence of pivotal factors such as the pivot position, maximum thickness, and maximum angle of attack on its hydrodynamic performance. Their study determined that specific structural configurations of the flapping hydrofoil can effectively enhance its cavitation resistance and reduce flow field instability. Zhang et al. [8] conducted a study on the hydrodynamic performance of a three-dimensional flapping hydrofoil, simulating the flow field variations in NACA 0012 under different parameters, including aspect ratios, tip ratios, and forward-swept and back-swept shapes. Through comparative analysis, they identified optimized geometric design parameters for the flapping hydrofoil structure. Gupta et al. [9] demonstrated that the hydrodynamic performance and stability of flapping hydrofoils improve with reduced relative thickness and increased curvature of the leading and trailing edge. Guo et al. [10] conducted hydrodynamic model experiments on a three-dimensional flapping hydrofoil with different leading-edge configurations, investigating their hydrodynamic performance under various oscillating frequencies and analyzing the influence of different leading-edge structures on hydrofoil performance. Zhe et al. [11] introduced vortex-generator (VG) flow-control concepts into tidal-turbine blade design and used CFD to examine how geometric parameters of VGs affect the hydrodynamic performance of NACA 4418.

In recent years, with the rapid development of intelligent optimization algorithms and machine learning technologies, there has been an increasing number of studies focused on integrating these technologies with the optimization of flapping hydrofoil hydrodynamic performance [12]. Song et al. [13] employed an integrated approach combining the Taguchi experimental, neural networks, and CFD to systematically investigate the influence of parameters, including aspect ratio, heaving amplitude, pitching amplitude, and flapping frequency, on the propulsion performance of a three-dimensional NACA 0012. Najafi et al. [14] utilized an artificial neural network (ANN) to predict key performance parameters under different Froude numbers (Fr) and hydrofoil types, with the aim of identifying suitable hydrofoils between two hulls of catamarans. Wang [15] proposed a novel static fluid-structure interaction (FSI) tool based on deep learning, which efficiently solves the FSI problem and demonstrates its capability in accurately predicting the elastic deformation of a flexible hydrofoil. Yang et al. [16] constructed a non-parametric model generalizing across operational environments based on a Gaussian Process Regression (GPR) algorithm and employed Deep Reinforcement Learning (DRL) to acquire the globally optimal combination of motion parameters for flapping hydrofoil propulsion within the entire defined domain under experimental conditions. Kostas et al. [17] developed an optimization framework based on an isogeometric boundary element method (BEM) potential flow solver for hydrofoil geometric parameter optimization.

Although the CFD simulation provides high accuracy in analyzing the hydrodynamic performance of the flapping hydrofoil under various parametric conditions, extensive simulation experiments are highly time-consuming [18]. Furthermore, no definitive correlation has been established between the hydrodynamic performance and geometric parameters, such as the pitch-axis location and curvature [19]. Therefore, rapidly identifying the coupled geometric parameters that correspond to optimal hydrodynamic performance within a defined domain holds significant research value.

To establish an accurate model for flapping hydrofoil motion and account for the strongly nonlinear effects of the underwater environment, this paper employs the MLP for nonlinear modeling of flapping hydrofoil propulsion. Concurrently, the DDQN is used to enable autonomous optimization through the interaction between the underwater flapping hydrofoil and its environment. This paper proposes an MLP-DDQN collaborative optimization method, which utilizes the MLP to construct a flapping hydrofoil surrogate model and incorporates the DDQN algorithm to efficiently identify optimal geometric parameters that maximize propulsion performance and meet specific performance requirements, even under small-sample conditions. This approach offers an efficient and reliable solution for the intelligent design of underwater propulsion systems.

2. Flapping Hydrofoil Propulsion Problem

As the core actuating component of the underwater propulsion system, the geometric features of the flapping hydrofoil directly influence its hydrodynamic performance. The NACA four-digit series airfoils, developed by the U.S. National Advisory Committee for Aeronautics (NACA), are widely used in such research due to their parametric design specifications and extensive experimental data [20]. Therefore, this paper selects the NACA four-digit series as the baseline model for the two-dimensional hydrofoil. Taking the NACA 4418 as an example, its profile and key geometric parameters are illustrated in Figure 1.

By dividing the maximum thickness t, maximum camber f, location of maximum camber $x_f$, and from Figure 1 by the chord length c, their corresponding dimensionless parameters are obtained: namely, relative thickness $t^{*}=t/c$, relative camber $f^{*}=f/c$, and location of relative camber $x_f^{*}=x_f/c$. The pitch-axis location is defined as $x_p^{*}=x_p/c$.

The motion of the flapping hydrofoil primarily consists of coupled heave and pitch, and is governed by the following equations of motion [21]:

$$y\left(t\right)=y\left(0\right)sin\left(\omega t\right)$$

(1)

$$\theta \left(t\right)=\theta \left(0\right)sin(\omega t+\varphi )$$

(2)

where $y\left(t\right)$ is the heave displacement of the flapping hydrofoil, $\theta \left(t\right)$ is the pitch angle of the flapping hydrofoil, $y\left(0\right)$ and $\theta \left(0\right)$ are their amplitudes, and $\varphi$ is the phase lag between heave and pitch motion.

In the study of the flapping hydrofoil, the instantaneous thrust coefficient $C_T\left(t\right)$ and instantaneous lift coefficient $C_L\left(t\right)$ are key metrics for evaluating the hydrodynamic performance of the flapping hydrofoil, and are defined as follows:

$$C_T\left(t\right)={\displaystyle \frac{F_x\left(t\right)}{\frac{1}{2}\rho U_{\infty }^{2}cs}}$$

(3)

$$C_L\left(t\right)={\displaystyle \frac{F_y\left(t\right)}{\frac{1}{2}\rho U_{\infty }^{2}cs}}$$

(4)

where $F_x\left(t\right)$ is the instantaneous thrust in the horizontal direction, $F_y\left(t\right)$ is the instantaneous thrust in the vertical direction, $\rho$ is the fluid density, $U_{\infty }^2$ is the mean incoming velocity, and s is the span length.

Over one flapping period, the mean thrust coefficient $\overline{C_T}$ and mean lift power $\overline{C_L}$ are defined as follows:

$$\overline{C_T}={\displaystyle \frac{1}{T}}{\int }_t^{t+T}{C}_T\left(t\right)dt$$

(5)

$$\overline{C_L}={\displaystyle \frac{1}{T}}{\int }_t^{t+T}{C}_L\left(t\right)dt$$

(6)

This paper addresses the optimization of propulsion performance for the flapping hydrofoil with respect to a set of geometric parameters, denoted as [$x_p^{*}$,$t^{*}$,$x_f^{*}$,$f^{*}$]. The symbols and admissible ranges of these parameters are summarized in

Table 1. Parameters and value ranges.

Parameter	Value Range
relative thickness $t^{*}$	{6, 9, 12, 15, 18, 24}
relative camber $f^{*}$	{0, 1, 2, 4}
location of relative camber $x_f^{*}$	{0, 4}
pitch-axis location $x_p^{*}$	[0, 50]

. We adopt the averaged input power $\overline{P_{\mathrm{in}}}$ and the propulsion efficiency $\eta$ as the propulsion performance metrics.

The mean power coefficient $\overline{C_P}$ is defined as follows:

$$\overline{C_P}={\displaystyle \frac{1}{T}}{\int }_t^{t+T}{C}_P\left(t\right)dt={\displaystyle \frac{2\overline{P_{\mathrm{in}}}}{\rho U_{\infty }^{3}c}}$$

(7)

where $C_P\left(t\right)$ is the instantaneous power coefficient, and $\overline{P_{\mathrm{in}}}$ reflects the energy consumption of the flapping hydrofoil motion, which is defined as follows:

$$\overline{P_{\mathrm{in}}}={\displaystyle \frac{1}{T}}{\int }_t^{t+T}\left[F_y\left(t\right)\dot{y}\left(t\right)+M\left(t\right)\dot{\theta }\left(t\right)\right]dt$$

(8)

where $M\left(t\right)$ is the instantaneous torque of the flapping hydrofoil around the center of rotation.

Ultimately, the propulsion efficiency $\eta$ quantifies the capability of the flapping hydrofoil to convert input energy into propulsive power, which is defined as follows:

$$\eta ={\displaystyle \frac{\overline{C_T}}{\overline{C_P}}}$$

(9)

3. Numerical Method

3.1. Control Equations and Turbulence Modeling

During flapping, vortices are repeatedly generated and shed from the hydrofoil surface, requiring accurate resolution of near-wall boundary-layer dynamics. The Reynolds time-averaged Navier–Stokes (RANS) method is well suited to capturing boundary-layer separation and reattachment with relatively modest mesh requirements and exhibits robust convergence in iterative solvers. Therefore, this paper employs the Reynolds time-averaged Navier–Stokes (RANS) equations to capture the two-dimensional incompressible turbulent flow field. The governing equations for the flow motion are as follows [22]:

$${\displaystyle \frac{\partial {\overline{u}}_i}{\partial x_i}}=0,\left(i=1,2\right)$$

(10)

$${\displaystyle \frac{\partial {\overline{u}}_i}{\partial t}}+{\overline{u}}_j{\displaystyle \frac{\partial {\overline{u}}_i}{\partial x_j}}=-{\displaystyle \frac{1}{\rho }}{\displaystyle \frac{\partial \overline{p}}{\partial x_i}}+{\displaystyle \frac{\partial }{\partial x_j}}\left[\left(\gamma +{\gamma }_t\right)\left({\displaystyle \frac{\partial {\overline{u}}_i}{\partial x_j}}+{\displaystyle \frac{\partial {\overline{u}}_j}{\partial x_i}}\right)\right]$$

(11)

where ${\overline{u}}_i$ denotes the fluid velocity components; $x_i$ represents the control coordinates; p is the fluid pressure; t denotes the time; $\gamma$ is the kinematic viscosity; ${\gamma }_t=c_{\mu }{k}^2/\epsilon$ is the turbulent viscosity coefficient (with $c_{\mu }$ represents the model constant); k stand for the turbulent kinetic energy; and $\epsilon$ is the turbulent dissipation rate.

To more accurately capture the complex characteristics of such flow fields, the realizable k-$\epsilon$ turbulence model was employed in this study to solve the above Reynolds time-averaged Navier–Stokes (N-S) equations; the corresponding literature can be found in [23].

3.2. Computational Domain and Meshing

The size of the computational domain and the influence of far-field boundary conditions cannot be neglected. To ensure full wake development around the hydrofoil, the downstream extent is commonly set to $20c$, where c denotes the chord length. In this study, the total length in the x-direction is $8\mathrm{m}$, with $6\mathrm{m}$ extending in the positive x-direction, and the hydrofoil is centered at the origin. To emulate the effect of channel banks relevant to riverine propulsion, the channel width is set to $0.9\mathrm{m}$. The grid division of the computational domain is shown in Figure 2.

To capture the transient flow during hydrofoil motion, dynamic meshing was performed using an overset-grid approach. An elliptical deforming subdomain centered on the pitch axis was discretized with an unstructured quadrilateral mesh, while the remainder of the domain employed a structured grid. To better resolve the near-wall boundary layer along the hydrofoil and solid walls, a prismatic inflation layer was applied with a first-layer height of $0.0572\mathrm{mm}$, a growth rate of $1.2$, and 15 layers in total.

To assess grid independence, simulations were performed on meshes of three densities: $2.25\times {10}^4$ (coarse), $3.6\times {10}^4$ (medium), and $7.2\times {10}^4$ (fine) cells. The results are shown in Figure 3. The temporal evolution of the instantaneous thrust coefficient obtained with the medium and fine meshes exhibits close agreement, indicating that further refinement has a negligible effect on the solution. Therefore, to balance accuracy and computational cost, the medium mesh is adopted for subsequent simulations.

3.3. Numerical Method Validation

Numerical simulations of hydrofoil propulsion were performed in Ansys Fluent 2022R1 using the realizable k-$\epsilon$ turbulence model and an overset-grid technique. A pressure-inlet boundary condition was prescribed at the domain entrance to represent quiescent water, and a pressure-outlet was applied at the exit. The hydrofoil surface and channel banks were modeled as stationary walls. Hydrofoil motion was imposed via a user-defined function (UDF). Pressure–velocity coupling employed the coupled algorithm and spatial discretization used the second-order upwind scheme. The time step was chosen to be smaller than the ratio of the minimum grid size to the characteristic flow speed and was adaptively adjusted during the computation to maintain numerical stability.

To verify the reliability of the numerical methodology, according to the experimental study on the propulsive performance of flapping hydrofoils in the literature [24], a corresponding simulation model is established to evaluate the propulsion performance of the flapping hydrofoil. The numerical method employed herein is validated by comparison with the experimental results reported in [24], with the computational model and simulation parameters matched to the experimental conditions in that study: the chord length $c=0.1\mathrm{m}$; the mean incoming velocity $U_{\infty }^2$ is taken as $0.4\mathrm{m}/\mathrm{s}$; the phase difference $\varphi ={90}^{\circ }$; the heave amplitudes $y_0=0.075\mathrm{m}$; the maximum angle of attack ${\alpha }_{max}={20}^{\circ }$; the Reynolds number $Re=4.0\times {10}^4$.

Figure 4 shows that the numerical calculation results in this paper are consistent with the experimental results and exhibit the same trends, confirming the effectiveness of the adopted numerical method.

4. Methodology

4.1. Overview of MLP-DDQN Framework

Although traditional CFD simulations accurately resolve flow fields, vortex evolution, and hydrodynamic loads of the flapping hydrofoil, their high computational cost makes them impractical for large-scale, iterative, parametric optimization. Moreover, deep reinforcement learning (DRL) methods that rely on online interaction with such high-cost simulators suffer from prohibitive sample complexity. To address these challenges, we propose a collaborative optimization framework, the MLP-DDQN method, which couples an MLP surrogate model with an enhanced DDQN augmented by Pareto-front information. The MLP efficiently learns the nonlinear mapping from geometric-parameter space to performance metrics, yielding a high-fidelity, low-cost surrogate that replaces the original CFD environment. This surrogate enables rapid prediction of key performance metrics for candidate parameter sets and provides real-time environment feedback to the RL agent. Guided by Pareto-front information, the agent performs dual-objective optimization of propulsion efficiency and averaged input power.

The core workflow comprises three phases: (i) systematic sampling of hydrodynamic performance under different hydrofoil configurations via parameterized CFD simulations; (ii) training an MLP-based surrogate model to learn the nonlinear mapping from geometric parameter combinations to propulsion metrics; and (iii) deploying an improved DDQN algorithm within the surrogate environment to perform global optimization over the geometric design space. The proposed MLP–DDQN framework is illustrated in Figure 5. In this framework, the MLP-based surrogate acts as a compact model of the hydrodynamic environment, mapping each geometric state to the corresponding averaged input power $P_{\mathrm{in}}$ and propulsive efficiency $\eta$, from which the reinforcement-learning reward is computed. The agent’s observed state is a normalized geometric parameter vector ${\mathbf{s}}_t=[x_p^{*},t^{*},x_f^{*},f^{*}]$ representing the current hydrofoil configuration. A 12-dimensional discrete action space is adopted, where each action perturbs one or several components of ${\mathbf{s}}_t$ by a prescribed step size.

4.2. MLP for Hydrofoil Parameter-Performance Mapping

4.2.1. Introduction to MLP Model

The Multilayer Perceptron (MLP) is a feedforward neural network comprising an input layer, one or more hidden layers, and an output layer, each consisting of multiple neurons [25,26,27,28]. The input layer receives external signals and passes them to the first hidden layer; thereafter, each layer takes the previous layer’s outputs as inputs, applies an affine transformation with learnable weights $\mathbf{W}$ and biases $\mathbf{b}$, and then a nonlinear activation to produce its output.

Given a training dataset of N samples ${\left\{({\mathbf{x}}_i,{\mathbf{y}}_i)\right\}}_{i=1}^N$, the MLP defines via layer-wise forward propagation a nonlinear mapping $f_{\theta }:\mathcal{X}\to \mathcal{Y}$ with parameters $\theta =\{\mathbf{W},\mathbf{b}\}$. For each sample, the network is trained to minimize the discrepancy between predictions ${\widehat{\mathbf{y}}}_i=f_{\theta }\left({\mathbf{x}}_i\right)$ and ground-truth labels ${\mathbf{y}}_i$, measured by a regression loss function $\ell ({\widehat{\mathbf{y}}}_i,{\mathbf{y}}_i)$ that is differentiable with respect to the network parameters $\theta$. Gradients of $L\left(\theta \right)$ with respect to $\theta$ are computed by backpropagation and used by a gradient-based optimizer to update the network parameters. Accordingly, the empirical risk minimization objective is

$$minL\left(\theta \right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N\ell \left({\widehat{\mathbf{y}}}_i,{\mathbf{y}}_i\right).$$

(12)

4.2.2. MLP Architecture and Training

For geometric parameter combinations, the MLP accurately captures the complex mapping relationships between geometric parameters and propulsion performance metrics through nonlinear transformations in its hidden layers, thereby providing efficient environmental feedback for reinforcement learning agents.

This paper constructs a data-driven MLP surrogate with three hidden layers: the input layer takes a four-dimensional feature vector; the hidden layers contain 128, 64, and 32 neurons, respectively; and the output layer produces a single performance prediction. To improve training stability, a residual connection projects the input directly to the third hidden layer, alleviating vanishing-gradient effects. Batch-normalization layers are included to reduce overfitting and enhance generalization. During training, we optimize the Huber loss to improve robustness to noisy targets, and use the AdamW optimizer with an initial learning rate of 0.001, together with a learning-rate scheduler to accelerate convergence and avoid poor local minima.

4.3. DDQN Incorporating Pareto Frontier Information

4.3.1. Introduction to Deep Reinforcement Learning

Deep reinforcement learning (DRL) integrates deep neural networks with the reinforcement-learning framework to learn decision policies through continual interaction between an agent and its environment [29,30,31,32]. The decision-making problem is modeled as a Markov decision process (MDP) $\langle \mathcal{S},\mathcal{A},P,R,\gamma \rangle$, where $\mathcal{S}$ is the state space (the agent observes $s_t\in \mathcal{S}$), $\mathcal{A}$ is the action space (the agent selects $a_t\in \mathcal{A}$ according to a policy $\pi \left(a\right|s)$), $P(s_{t+1}\mid s_t,a_t)$ denotes the state-transition probability, $R(s_t,a_t)$ is the reward function, and $\gamma \in [0,1]$ is the discount factor. At each discrete time step t, the agent observes the current state $s_t$, samples an action $a_t\sim \pi (·|s_t)$, and upon executing $a_t$ receives a scalar reward $r_t=R(s_t,a_t)$ together with the next state $s_{t+1}$. This interaction generates experience tuples $(s_t,a_t,r_t,s_{t+1})$, which are used to update the parameters of the decision policy or value function. The objective of DRL is to learn a policy that maximizes the expected cumulative discounted return:

$$G_t=\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}.$$

(13)

4.3.2. DDQN for Hydrofoil Optimization

This paper proposes an improved DDQN algorithm that embeds Pareto-front information into the training process. The detailed workflow is summarized in Algorithm 1. To handle the two propulsion objectives—averaged input power $P_{\mathrm{in}}$ and propulsive efficiency $\eta$, we first construct a coarse approximation of the Pareto front in the objective space using the MLP surrogate. Specifically, the surrogate is evaluated on a set of representative geometric designs, and the corresponding pairs $(P_{\mathrm{in}},\eta )$ are filtered by non-domination to obtain an archive ${\mathcal{F}}_0$ of Pareto-optimal candidates. From this archive, we identify an engineeringly acceptable region in the objective space and define two thresholds: a maximum admissible power ${\tau }_p$ and a minimum acceptable efficiency ${\tau }_e$.

Algorithm 1 DDQN with Pareto Optimization

Require:  Hydrofoil design parameters $\mathcal{P}$, constraint function $\mathcal{C}$, performance models $f_{\overline{P_{\mathrm{in}}}}$, $f_{\eta }$, Pareto thresholds ${\tau }_p$, ${\tau }_e$ 1: Initialize replay buffer $\mathcal{D}$, online network $Q_{\theta }$, target network $Q_{{\theta }^{\prime }}$ 2: for episode $=1,2,\dots ,E$ do 3:       Sample initial design ${\mathbf{s}}_0\in \mathcal{P}$ 4:       for $t=0,1,\dots ,T_{\max }$ do 5:             Select action $a_t$ via $ϵ$-greedy policy from $Q_{\theta }\left({\mathbf{s}}_t\right)$ 6:             Execute $a_t$, observe ${\mathbf{s}}_{t+1}$, and predict: 7:                 $\overline{P_{\mathrm{in}}}\leftarrow f_{\overline{P_{\mathrm{in}}}}\left({\mathbf{s}}_{t+1}\right)$, $\eta \leftarrow f_{\eta }\left({\mathbf{s}}_{t+1}\right)$ 8:             Compute reward with Pareto bonus: 9:                 $r_t\leftarrow (\eta \times 100-\overline{P_{\mathrm{in}}}/10)+\beta ·\mathbb{I}[\overline{P_{\mathrm{in}}}<{\tau }_p\wedge \eta >{\tau }_e]$ 10:             Store transition $({\mathbf{s}}_t,a_t,r_t,{\mathbf{s}}_{t+1})$ in $\mathcal{D}$ 11:             Standard DDQN Update: 12:                 Sample minibatch $\mathcal{B}\sim \mathcal{D}$ 13:                 Compute target $y_j=r_j+\gamma Q_{{\theta }^{\prime }}({\mathbf{s}}_{j+1},arg{max}_{a^{\prime }}{Q}_{\theta }({\mathbf{s}}_{j+1},a^{\prime }))$ 14:                 Update $\theta$ via gradient descent on ${\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum {(y_j-Q_{\theta }({\mathbf{s}}_j,a_j))}^2$ 15:                 Soft update: ${\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime }$ 16:       end for 17:       Pareto Front Analysis: 18:       for all valid designs $\mathbf{s}\in \mathcal{P}$ do 19:             if $f_{\overline{P_{\mathrm{in}}}}\left(\mathbf{s}\right)<{\tau }_p\wedge f_{\eta }\left(\mathbf{s}\right)>{\tau }_e$ then 20:                  Update Pareto front $\mathcal{F}\leftarrow \mathcal{F}\cup \left\{(\mathbf{s},f_{\overline{P_{\mathrm{in}}}}\left(\mathbf{s}\right),f_{\eta }\left(\mathbf{s}\right))\right\}$ 21:             end if 22:       end for 23:       Save $\mathcal{F}$ 24: end for

Within the DDQN framework, the current state $s_t$ is the normalized geometric parameter vector, and the agent selects an action $a_t$ that updates the design to $s_{t+1}$. After querying the MLP surrogate for the corresponding performance metrics $O_P=P_{\mathrm{in}}\left(s_{t+1}\right)$ and $O_{\eta }=\eta \left(s_{t+1}\right)$, the reward is decomposed into a base scalarization term and a Pareto-aware bonus.

The base reward linearly combines the two objectives as follows:

$$R_b=100O_{\eta }-{\displaystyle \frac{1}{10}}{O}_P$$

(14)

where $R_b$ measures the trade-off between propulsive efficiency and averaged input power for the updated design.

The Pareto-front bonus incorporates the threshold information and is defined as follows:

$$R_p=\beta \mathbb{I}\left[P_{\mathrm{in}}\left(s_{t+1}\right)<{\tau }_p\wedge \eta \left(s_{t+1}\right)>{\tau }_e\right]$$

(15)

where $\beta >0$ is a bonus coefficient and $\mathbb{I}[·]$ denotes the indicator function.

In conclusion, the total reward at time step t is then given by the following:

$$R_t=R_b+R_p$$

(16)

This study employs the DDQN architecture with a learning rate of $1\times {10}^{-4}$ to reduce training oscillations and mitigate divergence. Exploration follows an $\epsilon$-greedy policy: the initial exploration rate is set to $0.9$ to ensure sufficient random exploration in early training, and is linearly annealed to $0.1$ to gradually shift toward experience-driven exploitation, thereby balancing exploration and exploitation.

4.3.3. MLP-DDQN for the Fluid Solver

Although DRL can, in principle, train directly within flapping hydrofoil propulsion simulators to discover optimal parameter combinations, its practical deployment is hindered by prohibitive cost. Each CFD evaluation for a hydrofoil propulsion case may require hours, while effective DRL training typically demands thousands of samples, leading to extreme cumulative compute and wall time. Consequently, applying DRL for end-to-end parameter optimization poses substantial engineering feasibility challenges.

In this paper, we introduce an MLP surrogate to learn the mapping from hydrofoil geometry to propulsion performance, and employ an improved DDQN algorithm for optimization on the surrogate. To realize cooperative dual-objective optimization, we incorporate Pareto-front information to refine the solution set. A database linking geometric parameters to hydrodynamic responses is constructed via CFD, and the MLP is trained to approximate the mapping from the geometric parameter combinations to performance metrics. Within the DDQN framework, the agent’s observed state is a normalized vector of hydrofoil parameters, and the objective is to identify Pareto-optimal trade-offs between propulsion efficiency $\eta$ and averaged input power $\overline{P_{\mathrm{in}}}$. The agent conducts autonomous policy search guided by a reward shaping term that biases exploration toward the Pareto front, thereby achieving coordinated improvement in both efficiency and power consumption.

5. Results and Discussion

In this paper, we employ a two-dimensional rigid flapping hydrofoil. Apart from the geometric parameters to be optimized, the following motion parameters are fixed: chord length $c=0.3\mathrm{m}$; flapping frequency $f=1\mathrm{Hz}$; angular frequency $\omega =2\pi \mathrm{rad}{\mathrm{s}}^{-1}$; heave amplitude $y\left(0\right)=0.5c$; pitch amplitude $\theta \left(0\right)={30}^{\circ }$.

All experiments are performed on an Intel Core i5–14600KF CPU @ $3.50\mathrm{GHz}$ with $32\mathrm{GB}$ RAM running Windows 11. The code is implemented in Python (version 3.11.5).

5.1. MLP Surrogate Model

A non-parametric MLP surrogate is constructed to map geometric parameters to propulsion performance metrics. The dataset is generated via CFD simulations, with the flapping-hydrofoil geometric parameters serving as input features and the averaged input power and propulsive efficiency as targets. In total, 624 samples are collected over the design space, and the dataset is randomly split in an 8:2 ratio into a training set and a validation set. During training, the network parameters are optimized using adaptive gradient-based methods with appropriate regularization to enhance generalization and stabilize convergence. The evolution of the training and validation losses exhibits similar decreasing trends and eventually stabilizes at comparably low values, indicating that the surrogate does not suffer from severe overfitting. Parity plots of MLP predictions versus ground truth on the validation set are shown in Figure 6.

In Figure 6a, the MLP surrogate exhibits strong predictive performance on the validation set for propulsive efficiency: the scatter points cluster tightly about the reference line ($y=x$). As shown in Figure 6b, the predictions for averaged input power are similarly accurate. Although a few outliers appear, they constitute less than $2\%$ of all samples. Overall, more than $95\%$ of predictions fall within an acceptable error range, indicating that the model effectively captures the nonlinear mapping between airfoil geometry and propulsion performance. The detailed performance metrics of the MLP surrogate are reported in

Table 2. Performance of the MLP surrogate on the training and validation sets.

Target	Set	MAE	MSE	${\mathit{R}}^2$	Training Time (s)	Inference Time (ms)
averaged input power	Training	1.7485	8.0898	0.9961	5.90	–
–	Validation	1.8794	10.8440	0.9944	–	0.27
propulsive efficiency	Training	0.0026	$1.5\times {10}^{-5}$	0.9867	7.73	–
–	Validation	0.0028	$2.0\times {10}^{-5}$	0.9819	–	0.25

.

In terms of predictive accuracy, the surrogate exhibits consistently high performance on both the training and validation sets. For averaged input power, the validation coefficient of determination reaches $R^2=0.9944$, with the corresponding MAE remaining below $2.0$, indicating only small deviations over the explored power range. Propulsive efficiency is predicted with similarly high accuracy, achieving a validation $R^2=0.9819$ and MAE on the order of ${10}^{-3}$. In both cases, the training errors are only marginally lower than the validation errors, suggesting that the surrogate captures the nonlinear dependence on hydrofoil geometry without noticeable overfitting. Regarding computational efficiency, a single forward pass of the surrogate requires only 0.25–0.27 ms, providing several orders of magnitude speedup over the original CFD simulations, which take on the order of hours per evaluation. Consequently, the MLP surrogate can provide accurate and timely environment feedback for the RL agent, thereby enabling the subsequent dual-objective optimization.

5.2. DDQN Agent

During training, the agent’s observed state is a normalized geometric parameter vector $[x_p^{*},t^{*},x_f^{*},f^{*}]$. The optimization objective is to identify Pareto-optimal solutions that balance propulsive efficiency and averaged input power. A 12-dimensional discrete action space is adopted, supporting both independent parameter updates and coordinated multivariable adjustments. To bias the search toward practically attractive designs, we introduce two scalar thresholds in the reward: a maximum admissible input power ${\tau }_p=140\mathrm{W}$ and a minimum acceptable efficiency ${\tau }_e=38\%$. These values are chosen based on a preliminary surrogate-based Pareto analysis of the CFD database (Section 4.2 and Figure 7), in which they correspond to a knee region of the Pareto front. Within the DDQN reward, $({\tau }_p,{\tau }_e)$ are implemented as soft preference parameters: an additional bonus is issued whenever a candidate simultaneously satisfies the power and efficiency constraints.

Figure 8 shows the DDQN training curves: the averaged reward (right axis) and the Q-network loss (left axis) versus training steps. The loss $L_Q$ denotes the standard DQN value-function loss, i.e., MSE between the current Q-network predictions and the bootstrapped target values. The initial stage exhibits pronounced loss oscillations, reflecting exploratory behavior in the design space; this is followed by a rapid convergence phase, during which the agent discovers high-quality regions and the loss drops sharply. Subsequently, the process stabilizes, with the loss fluctuating around $0.7$, indicating that the agent approaches the global Pareto optimum while local optima remain in the landscape.

The Pareto-front scatter obtained by the DDQN during parameter optimization is shown in Figure 7, with the abscissa denoting averaged input power and the ordinate propulsive efficiency. A local-optimum region is observed for powers of 115–124 W. In this interval, efficiency growth slows and, at times, both power and efficiency decline simultaneously, indicating that the agent becomes trapped in a parameter-space basin without yielding non-dominated improvements. By contrast, the non-dominated set outside this range is more uniformly distributed. For averaged input powers of 103–113 W, efficiency rises rapidly, indicating that the agent has discovered an optimal combination—rearward maximum-camber location, moderate camber, and an optimized pitch-axis position—thereby approaching the optimal solution.

The heatmap analysis of parameter distributions within the Pareto set is shown in Figure 9. The pitch-axis location and relative thickness exhibit strong negative correlations with averaged input power (correlation coefficients $-0.895$ and $-0.516$, respectively), indicating that, within the studied range, appropriately increasing these variables can effectively reduce the required power. By contrast, relative camber and the chordwise location of maximum camber correlate positively with propulsive efficiency (coefficients $0.621$ and $0.242$, respectively), suggesting that a leading-edge geometry with moderate camber and a rearward relative camber position enhances propulsive efficiency. Overall, attaining Pareto optimality in both average input power and efficiency necessitates the joint optimization of the leading-edge shape and the pitch-axis location, thereby balancing low power consumption with high propulsion performance.

The evolution of normalized state variables for the Pareto-front solutions is shown in Figure 10, with all states scaled to $[0,1]$. As the base reward rises from its initial level to the peak, the parameters exhibit pronounced co-evolution. The pitch-axis location drifts continuously rearward from an initial normalized value of $0.58$ (i.e., $29.2\%$ chord), while the relative thickness follows a three-stage trajectory and ultimately reaches $0.8$, corresponding to an actual relative thickness of $18\%$. The DDQN uncovers multiple local optima; in particular, a thin leading edge combined with high camber produces negative coupling that reduces efficiency. The agent then converges toward the optimum, with the peak reward attained at the geometric parameter vector $[0.996,0.8,1.0,1.0]$, representing the physical state $[49.8,18,4,4]$ (in the units defined for pitch-axis location, relative thickness, relative camber location, and relative camber, respectively). The resulting optimum aligns well with the local-optimum region inferred from the MLP surrogate, indicating satisfactory convergence and generalization when using an MLP-based environment model.

The trained model was then used to perform global optimization of parameter combinations over the design space. By adjusting the reward function, efficiency-maximizing designs were obtained under different averaged input-power constraints.

Table 3. Representative Pareto solutions and sample counts for MLP–DDQN vs. traditional CFD.

Category	State Vector	CFD $\left[\mathit{P}\right(\mathbf{W}),\mathit{\eta }(\%\left)\right]$	Prediction $\left[\mathit{P}\right(\mathbf{W}),\mathit{\eta }(\%\left)\right]$	MLP–DDQN Samples	Traditional CFD Samples
Algorithmic optimum	[49.8, 18, 4, 4]	[104.52, 42.05]	[105.85, 41.67]	624	4300
averaged input power $120\mathrm{W}$	[42.5, 15, 4, 4]	[120.04, 41.47]	[118.77, 41.37]	624	4300
averaged input power $130\mathrm{W}$	[34.2, 15, 4, 2]	[129.94, 41.22]	[130.44, 41.64]	624	4300
averaged input power $140\mathrm{W}$	[29.2, 15, 4, 2]	[139.98, 40.86]	[139.13, 40.43]	624	4300

summarizes four representative Pareto-front solutions: the unconstrained algorithmic optimum and three constrained cases with power caps of 120, 130, and $140\mathrm{W}$, which correspond to practical design scenarios with different available input-power budgets. For each case, the table reports the best propulsive efficiency achieved under the specified power limit, together with the corresponding CFD evaluations and MLP–DDQN predictions for the averaged input power $P_{\mathrm{in}}$ and propulsive efficiency $\eta$. The relative errors in both $P_{\mathrm{in}}$ and $\eta$ across these four Pareto solutions lie between $0.24\%$ and $1.27\%$, demonstrating the high fidelity of the MLP–DDQN predictions with respect to the CFD evaluations. Compared with training DDQN directly in the CFD environment (which requires thousands of samples [16]), the MLP–DDQN method uses only about $14.5\%$ of the sample count, yielding markedly higher efficiency. These results indicate that the MLP–DDQN method preserves optimization effectiveness while substantially improving RL training efficiency and reducing computational cost.

5.3. Influence of the Parameter Combinations on the Propulsive Performance

To investigate the intrinsic mechanisms by which parameter combinations affect flapping hydrofoil propulsion performance, three representative settings are selected for detailed analysis, as summarized in

Table 4. Representative geometric parameter combinations and resulting performance.

ID	${\mathit{x}}_{\mathit{p}}^{*}$	${\mathit{t}}^{*}$	${\mathit{x}}_{\mathit{f}}^{*}$	${\mathit{f}}^{*}$	Average Input Power (W)	Propulsive Efficiency (%)
Group 1	20.0	6	0	0	189.62	32.18
Group 2	30.0	15	4	1	140.37	40.32
Group 3	49.8	18	4	4	104.52	42.05

.

These settings are chosen based on the distribution of Pareto-front solutions and the evolution of the reinforcement-learning reward: Group 1 corresponds to a low-efficiency, high-power configuration that lies near the initial design explored by the agent and is clearly dominated on the Pareto front. Group 2 is located in the knee region of the Pareto front, representing a typical engineering compromise in which a substantial gain in propulsive efficiency is achieved while the averaged input power is reduced to a moderate level. Group 3 is the algorithmic optimum identified by the MLP–DDQN agent, exhibiting both the highest propulsive efficiency and the lowest averaged input power among the three.

Together, these three groups span the progression from an unfavorable baseline to an intermediate trade-off and finally to the near-optimal design, enabling a systematic comparison of the associated thrust, lift, pressure distribution, and vortex structures.

Figure 11 shows the evolution of the instantaneous thrust and lift coefficients over one period for the three parameter sets. In Figure 11a, none of the sets produces negative thrust within the cycle; moreover, Group 3 attains both the largest peak $C_T\left(t\right)$ and the highest mean thrust coefficient. In Figure 11b, the instantaneous lift coefficient $C_L\left(t\right)$ exhibits two extrema (one positive and one negative) per cycle; the extremal magnitudes decrease from Group 1 to Group 3, with Group 3 being the smallest. Overall, Group 3 yields the largest mean thrust and the smallest mean lift, indicating more effective conversion of input power into net propulsive thrust rather than into induced-drag-related lift, hence higher propulsive efficiency and lower power demand for a given thrust level.

Adjusting the relative thickness $t^{*}$, relative camber $f^{*}$, and the location of relative camber $x_f^{*}$ primarily targets optimization of the flapping hydrofoil’s leading-edge geometry. Figure 12 shows surface-pressure contours at $t=0.4T$ for the three parameter sets. Distinct pressure patterns emerge: the magnitude and extent of the high-pressure region near the leading edge increase progressively, while the low-pressure region contracts and shifts forward toward the leading edge. Consequently, the pressure difference between the upper and lower surfaces grows from Group 1 to Group 3, with the third configuration attaining the peak thrust.

When the flapping hydrofoil follows its prescribed periodic kinematics, a jetting effect arises: in each cycle, a negative and a positive vortex are shed from the lower and upper surfaces, respectively. The counter-rotating vortex pair convects downstream and dissipates, forming in the wake an alternating street with positive vortices below and negative vortices above. This pattern is opposite to the classical von Kármán vortex street and is therefore termed a reverse von Kármán vortex street.

To further clarify why Group 3 achieves the best propulsive performance, Figure 13 shows vorticity contours at $t=0.4T$ for the three parameter sets. The Group 1 hydrofoil exhibits a relatively loose, disordered vortex street, with large spacing between opposite-sign vortex cores and structures located close to the boundary. By contrast, Group 3 forms a more compact high-vorticity region tightly attached to the surface, with a strict spatial anti-phase arrangement that induces a stronger jet and reduces energy dissipation. Examination of the wake evolution further reveals a monotonic increase in vortex coherence from Groups 1 to 3, culminating in Group 3 with a clear, coherent chain-like vortex structure.

6. Conclusions

This paper proposes an MLP-DDQN method that, under small-sample conditions, rapidly identifies the Pareto-optimal set of flapping hydrofoil designs with respect to propulsive efficiency and averaged input power. An MLP-based surrogate of the simulation environment enables autonomous optimization by a DDQN augmented with Pareto-front information, thereby yielding recommended parameter combinations at prescribed power budgets. The paper further conducts a systematic assessment of how these combinations affect propulsion performance. The principal conclusions are as follows:

• After training, the MLP-DDQN method rapidly locates multiple optima within the design domain using few samples and also returns competitive solutions near local optima. Relative to direct optimization in the original CFD environment, it achieves wall-time speedups of several orders of magnitude and requires only about $14.5\%$ of the samples used by conventional deep reinforcement learning, thereby improving training efficiency.
• The Pareto set produced by the MLP-DDQN method exhibits a mean averaged input power of $106.58\mathrm{W}$ and a mean propulsive efficiency of $41.89\%$. Moreover, with reward shaping, the optimal solutions obtained under different power constraints differ from the corresponding simulation targets by only $0.24$–$1.27\%$.
• Flow-field analyses across geometric parameter combinations indicate that a moderate rearward shift of the pitch-axis location, together with an appropriately shaped leading edge, promotes an orderly reverse von Kármán vortex street over the flapping hydrofoil, enhancing jet momentum transfer while reducing dissipation of shed vortices. This reduces energy loss and yields higher propulsive efficiency.

This study focuses exclusively on geometric-parameter optimization for the two-dimensional flapping hydrofoil and does not incorporate control of kinematic parameters. In addition, the generalization and robustness of the employed simulation environment remain to be validated. Future work will consider joint optimization over state and action parameters in a realistic channel flow to better support practical engineering applications.