Adaptive Sequential Infill Sampling Method for Experimental Optimization with Multi-Fidelity Hamilton Kriging Model

Abstract

Experimental optimization with surrogate models has received much attention for its efficiency recently in predicting the responses of the experimental optimum. However, with the development of multi-fidelity experiments with surrogate models such as Kriging, the traditional expected improvement (EI) in efficient global optimization (EGO) has suffered from limitations due to low efficiency. Only high-fidelity samples to be used in optimizing Kriging surrogate models are infilled, misleading the sequential sampling method in low-fidelity data sets. This recent theory based on multi-fidelity sequential infill sampling methods has gained much attention for balancing the selection of high- or low-fidelity data sets, but ignores the efficiency of sampling in experiments. This article proposes an Adaptive Sequential Infill Sampling (ASIS) method based on Bayesian inference for a multi-fidelity Hamilton Kriging model in the use of experimental optimization, aiming to address the efficiency of sequential sampling. The proposed method is demonstrated by two numerical simulations and one practical aero-engineering problem. The results verify the efficiency of the proposed method over other popular EGO methods in surrogate models, and ASIS can be useful for any other reliability engineering problems due to its efficiency.

1. Introduction

Surrogate modeling is being increasingly applied in experimental science, offering valuable opportunities to optimize experiments and enhance the prediction of complex engineering systems [1,2]. In engineering experiments, the objectives of experimental design are divided into two main categories: development and identification. A well-structured experiment should exhibit repeatability, randomness, and controllability [3]. To achieve the objective of these engineering experiments, field tests are often required. These tests typically necessitate a controlled test environment, a significant number of personnel, and intricate monitoring [4]. A commonly used approach is to employ a surrogate model for approximation. Initially, sampling is conducted using space-filling experimental designs, such as uniform design [5] and Latin hypercube design [6]. By constructing a surrogate model, it becomes possible to predict responses within the sample space, significantly reducing experimental costs. Experimental design utilizing surrogate models can effectively predict responses in sparse sample experiments [7], achieve accurate predictions despite noise [8], and conduct sensitivity analysis in models with multiple inputs [9].

The initial surrogate model exclusively utilized the input and output data from the experiment. This approach facilitated the development of a black box model, a type of modeling technique where the internal workings are not visible or easily interpretable. Several methodologies were employed in this modeling process, including the response surface model (RSM) [10], polynomial chaotic expansion model (PCE) [11], Kriging model [12], radial basis function model (RBF) [13], and support vector machine model (SVM) [14]. Among them, the Kriging surrogate model is commonly used in aerospace design [15] because it effectively provides nonlinear fitting and predicts both the variance of the prediction point and process variance. The Kriging surrogate model, developed by South African mining scientist Danie Krige in 1959, was initially utilized in geostatistics for the exploration of mineral resources. In recent years, the increasing demand for aerodynamic design optimization has prompted the integration of gradient information into Kriging surrogate modeling, leading to the emergence of the gradient-enhanced Kriging (GEK) model [16]. The GEK model utilizes a first-order Taylor expansion, which transforms the partial derivatives at sample points into weighted sums of additional sample function values. However, in instances where gradients cannot be computed at specific sample points, the GEK model reverts to the standard Kriging approach. When experimental data is limited, the model’s response and gradient may not achieve the required accuracy for the design. Surrogate modeling, which depends on a single data set at a single level of fidelity, has encountered limitations. To address this issue, multiple experiments at varying fidelity levels are being applied to experimental design using surrogate models.

With the advancement of simulation technology and complex computing, it has become increasingly common to utilize data from multiple sources concurrently for developmental testing. For example, in the aerodynamic analysis of aero-load predictions [17], wind tunnel testing is recognized as a high-fidelity, costly, and accurate method. Conversely, computational fluid dynamics (CFD) analysis employing large eddy simulation with fine meshes is categorized as a lower-fidelity approach, while CFD analysis based on Reynolds-Averaged Navier–Stokes equations using coarse meshes is considered an even lower-fidelity test. In the development of multi-fidelity modeling using Kriging, Kennedy [18] introduced a framework for multi-fidelity Kriging. This framework employs Bayesian approximation, where the Kriging surrogate model built with low-fidelity data serves as the prior for the high-fidelity Kriging model. In this approach, data from both fidelity levels are utilized for surrogate modeling. Forrester [19] enhances the CoKriging surrogate modeling methodology by applying it to wing optimization and introducing a novel variance estimator that accounts for varying degrees of uncertainty. Han [20] introduced the Hierarchical Kriging model, which effectively integrates low-fidelity models to approximate the global trend. This model employs data of varying fidelity layered accordingly and streamlines the cross-covariance calculations associated with CoKriging. Han [21] analyzed the connections between multiple fidelity levels and further combined gradient-enhanced Kriging with generalized hybrid bridge functions in the field of variable-fidelity modeling (VFM), proposing a multi-fidelity GEK and applying it to the construction of aerodynamic coefficients. Zhang [22] focused on calculating the weight coefficient between gradient weights and fidelity levels in multi-fidelity modeling and proposed a multi-fidelity Kriging model based on Hamilton Monte Carlo (MHK), which improves the efficiency of weight coefficient calculation. Enhancements derived from multi-fidelity modeling strategies represent only a portion of the overall approach to surrogate optimization. In instances where the accuracy requirements are inadequate, it is also imperative to supplement the data set with additional samples.

A crucial component of surrogate-based optimization involves the implementation of infilling sample experiments, which are referred to as sequential experiments in the context of experimental optimization and as acquisition functions [23] in global optimization. For the purposes of this discussion, we shall refer to these methodologies collectively as the infill sampling strategy. Strategies for optimization can be categorized into three main objectives: improvement-based strategies, confidence-based strategies, and entropy-based strategies. Improvement-based strategies, such as Probability Improvement (PI) [24] and expected improvement (EI) [25], focus on selecting evaluation points that maximize the potential enhancement in the current surrogate model’s target extreme value. Confidence-based strategies, also referred to as the least confident bound (LCB) method in Kriging modeling, utilize a lower confidence limit to guide the selection of optimal points. Hertz [26] combines the estimated standard deviation and the estimated response of the surrogate model with a weighting factor to identify points that enhance the lower bound. Entropy-based strategies, also known as the maximum entropy criterion, were initially introduced by Currin [27] to measure the information gained from a single experiment, aiming to find the global optimum by reducing uncertainty. When dealing with multi-fidelity sample data, selecting the appropriate sample set becomes crucial. Zhang [28] developed a variable-fidelity expected improvement (VFEI) method that effectively employs the Hierarchical Kriging model. This method selectively samples from various fidelity data sets, optimizing the sampling process for both accuracy and efficiency while minimizing resource use. Hao [29] proposed an adaptive multi-fidelity expected improvement (AMEI) method that takes into account both prediction accuracy and optimization potential in the context of the multi-fidelity gradient-enhanced Kriging model. Achieving an optimal balance between exploitation and exploration is crucial for effective experimental optimization. Dong [30] proposed a multi-point infill criterion named the Multi-surrogate-based Global Optimization using a Score-based Infill Criterion (MGOSIC) to identify cheap points with scores for selection. Zhang [31] proposed a neighborhood-based Kriging optimization method for sequential experiments, addressing the issue of screening additional samples when the neighborhood domain is small. There remains a gap in effectively utilizing the statistical information from additional samples and balancing the exploration weights across different fidelity levels in multi-fidelity Kriging modeling.

This paper proposes an Adaptive Sequential Infill Sampling (ASIS) method based on the balancing issues mentioned. The method is utilized for multi-fidelity Hamiltonian Kriging modeling when the accuracy of the model cannot be refined. The neighborhood-based Kriging has been expanded to multi-fidelity modeling with the benefit of MHK, and a Probabilistic Nearest Neighborhood (PNN) strategy has been employed to balance the exploration between multi-fidelity models.

The sections of this paper are organized as follows: Section 2 introduces the concept of experimental optimization utilizing the multi-fidelity Hamiltonian Kriging surrogate model. This section encompasses initial experiments, surrogate modeling, sequential design, and the establishment of performance criteria. Section 3 presents the formulation of an Adaptive Sequential Infill Sampling strategy for MHK. It includes a succinct review of both ordinary Multi-fidelity Experimental Information (MFEI) and the Probabilistic Nearest Neighborhood method, as well as a discussion of the adaptive ASIS framework for MHK. Section 4 employs two numerical simulations alongside a case study from aerospace engineering to illustrate the efficacy of the proposed strategy. Lastly, Section 5 offers a comprehensive summary of the ASIS strategy discussed in this work and outlines its potential for future applications.

2. Experimental Optimization Based on Multi-Fidelity Hamiltonian Kriging

This article examines optimization problems in the design of developmental experiments, with a focus on identifying the extreme value of a target response through experimentation. To illustrate the importance of multi-fidelity surrogate models in experimental optimization, this section emphasizes two levels of data fidelity. The use of experimental data with varying levels of fidelity can be beneficial for future extensions.

2.1. Framework of Multi-Fidelity Surrogate-Based Experiment Optimization

The conventional process for optimizing experimental design comprises three critical phases: initial design of experiments (DoE), practice experiments and subsequent optimization, and sequential experimentation. This process is designed to assess the influence of various experimental factors on the outcomes by systematically analyzing the experimental variables through controlled trials. Ultimately, the goal is to determine the optimal combination of these factors that will yield the desired results. The framework illustrating this experimental process is presented in Figure 1.

In instances where experimental costs are excessively high, the establishment of experimental environments is challenging, or labor expenditures associated with experimentation are substantial, simulation experiments grounded in computational analysis represent a feasible alternative. These simulation experiments, referred to as multi-fidelity experiments, are conducted across various scales. Optimization processes relying on multi-fidelity experiments can be effectively implemented through multi-fidelity surrogate models. The overall procedure is primarily divided into four essential phases: initial DoE, implementation of multi-fidelity experiments, surrogate-based optimization, and sequential DoE. This experimental framework is depicted in Figure 2. In the inner loop of the surrogate optimization algorithm, several key components play a crucial role in enhancing the efficiency of surrogate-based experimental optimization. These components include the selection and allocation of initial samples, the development of the surrogate model, and the formulation of criteria for optimization. Each of these elements plays a significant role in enhancing the overall performance of the optimization process.

2.2. Initial Sample Experiments

In the context of both classical and surrogate-based experimental design optimization, the initial sampling phase is of paramount importance. Unlike gradient-based optimization, where the initial sample primarily serves as a starting point, it is essential to ensure that this sample also contributes to effective space filling.

Assume that the true model of a system in region $x\in ℝ$ is

$$y=f(x)$$

(1)

where $x=(x_1,x_2,\dots ,x_s)$ is the factor and $y$ is the response. It is commonly recognized that the experimental sample space is represented as a hypercube, designated as $[a_1,b_1]\times [a_2,b_2]\times \dots \times [a_s,b_s]$. For the purpose of consistency and simplicity, this space is typically defined as the unit cube, referred to as $C^s={[0,1]}^s=\underset{s}{\underbrace{[0,1]\times [0,1]\times \dots \times [0,1]}}$. Let ${\rm P}=\{x_1,x_2,\dots x_n\}$ represent the set of $n$ design points on $C^s$. The objective of the experimental design is to develop a surrogate model through the implementation of design ${\rm P}$.

$$\widehat{y}=\widehat{f}(x)$$

(2)

To estimate the true model presented in Equation (1), it is customary to derive an estimate of parameter $E(y)$ by utilizing the sample mean.

$$E(y)={\displaystyle \underset{C^s}{\int }f(x)dx}$$

(3)

$$\overline{y}(\mathrm{{\rm P}})={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{y}_i}$$

(4)

The Monte Carlo method is recognized as one of the most straightforward random sampling techniques. In this methodology, $\mathrm{{\rm P}}$ signifies $n$ samples taken from a uniform distribution $U(C^s)$ within a defined set $C^s$. The estimated variance is referred to as $Var(f(x))/n$. In accordance with the central limit theorem, it is possible to compute the 95% confidence interval for the variance, which assesses the relationship between the sample mean and the overall mean.

$$\left|\overline{y}(\mathrm{{\rm P}})-E(y)\right|\le 1.96\sqrt{Var(f(x))/n}$$

(5)

In numerous instances, the variance in estimation caused by random sampling can be excessively large. Latin hypercube sampling (LHS), introduced by McKay [32], is a popular method used to minimize this estimation variance. This technique involves dividing the test area $C^s$ into $n$ layers based on variable $x_k$, ensuring that each layer maintains the same marginal probability $1/n$. After this division, a sample is taken from each layer.

When a prior distribution is available, Latin hypercube sampling (LHS) may be conducted according to the specified form of the prior, as elaborated in Appendix A. Figure 3 illustrates an LHS design that utilizes a two-dimensional normal distribution $N[5,{0.05}^2]\times N[8,{0.05}^2]$ as its prior.

2.3. Multi-Fidelity Hamilton Kriging Model

The primary distinction of the Kriging model, in contrast to other models, is its foundation as a non-parametric regression model. Originating from the field of geo-statistics, Kriging posits that a correlation exists between any two exploration points within a specific area, with this correlation being solely dependent on the distance separating them. In the context of a two-dimensional plane, it is assumed that this correlation is pervasive and follows a random stationary distribution. This stationary distribution is termed the correlation function. The strength of the correlation between two known samples is quantified by the correlation coefficient. When a sufficient number of samples are available, it becomes feasible to establish a regression model based on the correlation function, wherein the parameters of the correlation function can be estimated using the maximum likelihood method. For additional details, please refer to Appendix B.

Considering the m-dimensional problem, a small amount of expensive but accurate data (high-fidelity, HF) is established as $y_h:{ℝ}^m\to ℝ$, and a large amount of cheap but low-precision data (low-fidelity, LF) is established as $y_l:{ℝ}^m\to ℝ$.

$$S_h={\left[x^{(1)},\dots ,x^{(n_h)}\right]}^T\in {ℝ}^{n_h\times m}$$

(6)

$$S_l={\left[{x_l}^{(1)},\dots ,{x_l}^{(n_h)}\right]}^T\in {ℝ}^{n_l\times m}$$

(7)

$$Y_h={\left[{y_h}^{(1)},\dots ,{y_h}^{(n_h)}\right]}^T={\left[y\left({x_h}^{(1)}\right),\dots ,y\left({x_h}^{(n_h)}\right)\right]}^T\in {ℝ}^{n_h}$$

(8)

$$Y_l={\left[{y_l}^{(1)},\dots ,{y_l}^{(n_l)}\right]}^T={\left[y\left({x_l}^{(1)}\right),\dots ,y\left({x_l}^{(n_l)}\right)\right]}^T\in {ℝ}^{n_l}$$

(9)

where $S_h$ is the HF input set and $y_h$ is the HF response value set. Similarly, $(S_l,y_l)$ is considered the sampled set of LF data.

The multi-fidelity Hamiltonian Kriging (MHK) can be constructed as follows:

$$\widehat{y}={\displaystyle \sum _i^n{\omega }_i({\rho }_i{\widehat{y}}_i}+(1-{\rho }_i)\nabla {\widehat{y}}_i)$$

(10)

where ${\rho }_i$ is an adaptive parameter of gradient Kriging and ${\omega }_i$ is a scale parameter. The value of $\nabla \widehat{y}(x)$ can be obtained as

$$\nabla \widehat{y}(x)=d{f}^T(x)\widehat{\beta }+d{r}^T(x){\Psi }^{-1}\left(Y-F\widehat{\beta }\right)$$

(11)

where the ith column of $d{f}^T(x)$ is defined as the column that has an equal value in all rows at $\nabla f_i(x),i=1,\dots ,p$, and $d{r}^T(x)$ is defined as the column that has an equal value in ${\psi }^{\prime }\psi \{x-x^{\prime }\},i=1,\dots ,n$.

The selection of MHK is predicated on the capability of the HMC process to effectively simulate the multi-fidelity Kriging target, thereby facilitating faster evaluations. Furthermore, the HMC method’s proficiency in traversing and remaining within the typical distribution permits sampling from the low-fidelity data set $Y_l$, while also enabling adaptive access to the high-fidelity data set $Y$ for the acquisition of new samples.

2.4. Sequential Infill Sampling Strategy

Following the development of a surrogate model utilizing existing sample points, it may be employed to predict the response for any sample point within the designated sample space. However, should the surrogate model fail to meet the requisite accuracy standards, it is essential to manually introduce additional sample points and modify the parameters to enhance accuracy. In general, the optimization of the surrogate model’s accuracy primarily emphasizes global optimization strategies.

$$x^{\ast }=\underset{x\in X}{\arg \max }(\left|\widehat{f}(x)-f_{\min }(x)\right|-\kappa s(x))$$

(12)

The parameter $\kappa$ serves as a tuning variable that establishes a balance between absolute error and prediction variance. When $\kappa \to 0$ denotes the new sample, it corresponds to the widely recognized strategies of EI or PI, as detailed in Appendix C. In contrast, when $\kappa \to \infty$ signifies the new sample, it aligns with the approach aimed at maximizing the squared predicted error.

2.5. Performance Criteria

Quantifying the error in surrogate models can be classified into two primary methods: (1) methods that require additional data, such as test data sets; (2) methods that rely on existing data. The first method is frequently utilized in machine learning, while the second is more prevalent in statistical science. The root-mean-square error (RMSE) serves as an effective measure of global error within the design domain.

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{(y_i-{\widehat{y}}_i)}^2}}$$

(13)

where $y_i$ is the real response of the function and ${\widehat{y}}_i$ is the predicted response of the surrogate model at $x_i$. However, the test data set can be costly, so the Cross-Validation (CV) method provides a solution for this circumstance.

$$RMS{E}_{CV}=\sqrt{{\displaystyle \frac{1}{k}}{\displaystyle \sum _{i=1}^k{({\widehat{y}}_i-{\widehat{y}}_i^{-i})}^2}}$$

(14)

where $k$ is the number of omitted sample points at the kth iteration of surrogate modeling. As the leave-one-out validation (LOO), the k-fold cross-validation [33] method is an extended version of the CV method, and a Predictive Estimation of Model Fidelity (PEMF) is an even more extended of k-fold CV, which can be found in [34].

3. Adaptive Sequential Infill Sampling Strategy for MHK

3.1. Definition of Multi-Fidelity Infill Sampling Strategy

In order to propose an efficient infill sampling strategy for the multi-fidelity surrogate model, we first formulate a definition of the regular multi-fidelity infill sampling strategy.

For a surrogate model as proposed by Equation (10), the prediction of a multi-fidelity model can be written as

$${\widehat{y}}_{MF}={\omega }_1{\widehat{y}}_1+{\omega }_2{\widehat{y}}_2+\dots +{\omega }_n{\widehat{y}}_n$$

(15)

where ${\omega }_i$ is a weight parameter that can be solved by the Lagrange multiplier approach from an information economic function $J$ from MHK via Cholesky decomposition.

The objective of the infill sampling strategy is to choose an appropriate sample point in the design space to promote the accuracy of the surrogate model sequentially. In the process of MHK modeling, a series prediction $\widehat{y}(x,i)$ can be solved and we get

$$\widehat{y}(x,i)\sim N[{\widehat{y}}_i(x),s_i^2(x,i)]$$

(16)

The multi-fidelity model prediction can be formulated as

$$\widehat{\overline{Y}}(x)\sim N[{\widehat{y}}_{MF}(x),{\overline{s}}^2]$$

(17)

Inspired by the EI strategy, Zheng [28] introduced a VFEI strategy specifically designed for multi-fidelity infill sampling. The choice to incorporate LF samples or HF samples is determined by the sequential order of the pairs $E{I}_{VF}\left(x,i\right)$.

$$I_{VF}(x,i)=\max (Y_{MF\min }-\widehat{y}(x,i),0)$$

(18)

$$E{I}_{VF}\left(x,i\right)=\left\{\begin{array}{cc}\left(Y_{\mathrm{MFmin}}-\widehat{y}\left(x\right)\right)\mathrm{\Phi }\left({\displaystyle \frac{Y_{\mathrm{MFmin}}-\widehat{y}\left(x\right)}{s_i\left(x,i\right)}}\right)+s\left(x,l\right)\varphi \left({\displaystyle \frac{Y_{\mathrm{MFmin}}-\widehat{y}\left(x\right)}{s_i\left(x,i\right)}}\right),& ifs\left(x,i\right)>0\\ 0,& ifs\left(x,i\right)=0\end{array}\right.$$

(19)

3.2. Probabilistic Nearest Neighborhood

Probabilistic Nearest Neighborhood (PNN) was initially introduced by Holmes as a method for pattern classification within the field of statistical computation. As an advancement to K-Nearest Neighbor (KNN) classification, PNN offers a probabilistic framework that effectively addresses the uncertainty associated with interactions between neighborhoods. This methodology yields a continuous marginal probability prediction distribution within the interval (0,1). The primary advantage of PNN is its ability to provide a predictive distribution for new observations, resulting in a smoother representation of neighborhoods that extend into regions characterized by sparse data. To enhance the effective utilization of the expected improvement information derived from newly added sample points, it is advisable to employ PNN to deliver the predictive distribution for pre-added samples across various fidelity models. This approach will facilitate informed decision making concerning the addition of multi-fidelity samples.

Consider a set of responses of $\mathbb{Q}=\{(y_1,X_1),(y_2,X_2)\}$ with $n=n_1+n_2$ observations.

$$(y_1,X_1)=\{(y_{11},x_{11}),(y_{12},x_{12}),\dots ,(y_{1n_1},x_{1n_1})\}$$

(20)

$$(y_2,X_2)=\{(y_{21},x_{21}),(y_{22},x_{22}),\dots ,(y_{2n_2},x_{2n_2})\}$$

(21)

where $X_1=(x_{11},x_{12},\dots ,x_{1n_1})$ is the training set and $X_2=(x_{21},x_{22},\dots ,x_{2n_2})$ is the test set. Both of them are matrices of $n\times p$, $y_1$ is an $n_1\times 1$ vector of known class labels for the training data set, and $y_2$ is an $n_2\times 1$ vector of unknown class labels that needs to be recognized.

$$p(y_n|x_n,\beta ,k)={\displaystyle \prod _{i=1}^n\frac{\exp \{\beta (1/k){\displaystyle {\sum }_{j\stackrel{k}{~}i}{\delta }_{y_{1i}{y}_{1j}}}\}}{{\displaystyle \sum _{q=1}^Q\exp \{\beta (1/k){\displaystyle {\sum }_{j\stackrel{k}{~}i}{\delta }_{q{y}_{1j}}}\}}}}$$

(22)

where ${\delta }_{ab}$ is the Dirac function, $\beta$ is an interaction parameter between the neighborhoods of $y_{ni}$ and $y_{nj}$, k is the neighborhood size of a KNN classifier of Q classes, and $(1/k){\displaystyle {\sum }_{j\stackrel{k}{~}i}{\delta }_{q{y}_{1j}}}$ calculates the proportion of training points in the k nearest neighborhoods of $x_{1i}$ of class q. And a new point predictive distribution can be calculated as

$$p(y_{n+1}|x_{n+1},\beta ,k,Y)={\displaystyle \frac{\exp \{\beta (1/k){\displaystyle {\sum }_{j\stackrel{k}{~}n+1}{\delta }_{y_{n+1}{y}_{1j}}}\}}{{\displaystyle \sum _{q=1}^Q\exp \{\beta (1/k){\displaystyle {\sum }_{j\stackrel{k}{~}n+1}{\delta }_{q{y}_{1j}}}\}}}}$$

(23)

An example illustrating the contrast between PNN, KNN, and Discriminant Adaptive Nearest Neighbor (DANN) is shown in Figure 4 and Figure 5 using a set of sine wave-shaped decision boundaries.

3.3. Adaptive Sequential Infill Sampling Strategy

Inspired by the principles of PNN, this paper presents a novel framework for the multi-fidelity addition criterion utilized in MHK. We employ Bayesian inference to determine the optimal number of samples to be incorporated at each fidelity level. Furthermore, we apply PNN to evaluate the neighborhood of the newly added points. The neighborhood points for each fidelity level are ranked according to their mean values, with the highest-scoring points being designated as infill samples for subsequent trials.

Theorem 1.

Bayes formula

• Let $B_1,B_2,\dots ,B_n$ represent a partition of the sample space  $\Omega$. For any event A, it follows that

  $$P(B_i|A)={\displaystyle \frac{p(B_i)p(A|B_i)}{{\displaystyle \sum _{j=1}^{n}p(B_j)p(A|B_j)}}}$$

  (24)

Theorem 2.

Bayesian Prediction

• Assume  $x\sim p(x|\theta )$  for  $X=(x_1,x_2,\dots ,x_n)$, and $\theta \sim p(\theta |\alpha )$; then, the prediction of the $\widehat{x}$ posterior predictive distribution can be calculated as

  $$p(\widehat{x}|X,\alpha )={\displaystyle \int p(\widehat{x}|\theta )}p(\theta |X,\alpha )d\theta$$

  (25)

  With the prediction of MHK as (10), we can calculate the predictive distribution of any fidelity:

  $$p(y_i|{\widehat{y}}_{MF},x)={\displaystyle \int p(y_i|{\omega }_i,{\rho }_i)}p({\omega }_i,{\rho }_i|{\widehat{y}}_{MF},x)d({\omega }_i,{\rho }_i)$$

  (26)

  And with the help of (24),

  $$p(\widehat{y})={\displaystyle \sum _{i=1}^{n}p(y_i)p(\widehat{y}|y_i)}$$

  (27)

  The predictive distribution of any infill sample can be expressed as

  $$p({x_{n+1}}^i)={\displaystyle \frac{\exp \{\beta (1/k){\displaystyle {\sum }_{j\stackrel{k}{~}n+1}{\delta }_{x_{n+1}{x}_{1j}}}\}}{{\displaystyle \sum _{q=1}^Q\exp \{\beta (1/k){\displaystyle {\sum }_{j\stackrel{k}{~}n+1}{\delta }_{q{x}_{1j}}}\}}}}$$

  (28)

  By substituting Equations (27) and (28) into Equation (18), it is possible to obtain a similar structural representation:

  $$I_{ASIS}[{x_{n+1}}^i,i]=\max (p(y_{n+1}^i)-p({\widehat{y}}_{\min }),0)$$

  (29)

Note: The posterior probability density function (PDF) of the sequential samples at the ith fidelity level is subtracted from the current minimum multi-fidelity response. This approach serves two primary objectives: it not only considers the posterior response associated with the newly added sample but also ensures that the responses of the “pre-added” samples reflect the mean response within the vicinity, in accordance with the minimum neighborhood parameter estimate of the PNN.

A multi-fidelity infill decision-making framework is established as follows:

$$E{I}_{ASIS}[{x_{n+1}}^i,i]=\left\{\begin{array}{c}(y_i^{n+1}-y_{\min })\mathrm{\Phi }({\displaystyle \frac{y_i^{n+1}-y_{\min }}{s_i}})+s_i\varphi ({\displaystyle \frac{y_i^{n+1}-y_{\min }}{s_i}}),\begin{array}{cc}if& s_i>0\end{array}\\ 0,\begin{array}{cc}if& s_i=0\end{array}\end{array}\right.$$

(30)

It is noted that ASIS contributes to the improvement in the MF model. In addition, $E{I}_{ASIS}[{x_{n+1}}^i,i]$ means that to calculate any potential improvement for the MF model, after calculation of all infill samples for the MF model, a Bayesian predictor $p({x_{n+1}}^i)$ for $I_{ASIS}[{x_{n+1}}^i,i]$ is added, and then

$$x^{\ast }=\underset{x\in X_i,i=1,2,\dots }{\arg \max E{I}_{ASIS}({x_{n+1}}^i,i)}$$

(31)

Figure 6 presents the optimization structure delineated in this paper, which is primarily categorized into three components: the initial design of experiments (DoE), the multi-fidelity surrogate model, and infill sample optimization. The Initial DoE involves sampling experiments at varying fidelity levels, utilizing Latin hypercube sampling (LHS). The multi-fidelity surrogate model incorporates the MHK model and employs Predictive Estimation of Model Fidelity (PEMF) for root-mean-square error (RMSE) cross-validation, thereby enhancing the accuracy of the model. Additionally, the Adaptive Sequential Infill Sampling (ASIS) strategy proposed in this paper is utilized for effective discrimination among multi-fidelity points.

4. Results and Discussion

In this section, two numerical simulations and one engineering example are used to demonstrate the proposed method.

4.1. Forrestal Function

In this one-dimensional test, Forrestal functions are employed to demonstrate the proposed ASIS strategy. The constants A, B, and C can be adjusted to enhance the accuracy of the low-fidelity function.

$$\begin{array}{l}{y}_h={(6x-2)}^2\sin (12x-4)\\ y_l=A{(6x-2)}^2\sin (12x-4)+B(x-0.5)+C\end{array}$$

(32)

where $x\in \left[0,1\right]$, $A~N(0.5,0.035), B~N(10,0.02), C~N(-5,0.01)$,$y_h$ is a high-fidelity model, and $y_l$ is a low-fidelity model. And there is an optimal solution at $x=0.7573$ with the value of $y_h=-6.0207$.

In this test simulation, we first use 10 low-fi samples at $x_l=\{0;0.1;0.2;0.3;0.4;0.5;0.6;0.7;0.8;0.9;1\}$ and 4 hi-fi samples at $x_h=\left\{0;0.4;0.6;1\right\}$ to build the initial surrogate model by MHK, which can be found in Figure 7.

The infill process of the MHK surrogate model using the ASIS method is illustrated in Figure 8, Figure 9, Figure 10 and Figure 11. Each figure consists of two components: the surrogate functions and the ASIS criterion. During each iteration, the ASIS compares the optimal neighborhood improvement achieved with high-fidelity and low-fidelity samples. It then selects the median of the PNN to add a new sample. The accuracy of each model is assessed using the RMSE of the PEMF. If the accuracy does not meet the set requirement (defined here as 0.01), additional samples are incorporated to update the surrogate model. Throughout the process of four sample additions, two high-fidelity samples ($x_{h1}^{\ast }=0.5226$, $x_{h2}^{\ast }=0.7751$) and two low-fidelity samples ($x_{l1}^{\ast }=0.7726$, $x_{l2}^{\ast }=0.7420$) are added.

To further demonstrate the efficacy of the proposed method, the trial iteration limit is established at 10, with a corresponding RMSE convergence value of 0.01. For the purpose of comparison, we employ several methods, including VF-EI, augmented EI, and AMEI, utilizing the surrogate model that was originally proposed with these methods. Additionally, to control for variables, we incorporate the MHK surrogate model utilized in this paper for a comprehensive comparative analysis. The results of contemporary surrogate models employing various infill sampling strategies are delineated in

Table 1. Results of Forrestal function via modern surrogate models and infill strategies.

Type	Optimal	Iteration	HF Number	Error	Time(s)
Kriging+EI	−6.1020	10	10	0.82	15
CoKriging+augmented EI	−6.0900	10	10	0.52	18
HK+augmented EI	−6.0190	8	8	0.50	10
HK+VF-EI	−6.0195	8	6	0.38	28
HK+AMEI	−6.0198	5	3	0.09	24
MHK+ augmented EI	−6.0191	8	8	0.25	22
MHK+ VF-EI	−6.0197	8	5	0.22	33
MHK+ AM-EI	−6.0200	6	4	0.12	40
MHK+ ASIS	−6.0203	4	2	0.01	20

. The median error predictive distribution of RMSE via PEMF is shown in Figure 12. While the optimal value is denoted as $y_h=-6.0207$, it is observed that both the original EI and the augmented EI methodologies predominantly focus on the incorporation of high-fidelity samples during the sampling process. Conversely, the VF-EI and the AMEI minimize the inclusion of a limited number of high-fidelity samples; however, these approaches still necessitate multiple cycles of sample addition, leading to a significantly high total computational cost. The strategy introduced in this paper, referred to as MHK+ASIS, successfully reduces both the quantity of high-fidelity samples added and the overall number of samples incorporated, while maintaining comparable accuracy.

4.2. Rosebrock Function

A widely recognized 2D optimization function serves as the basis for demonstrating the proposed ASIS. The definitions of the multi-fidelity functions are presented as follows:

$$f_h(x_1,x_2)=100{(x_2-{x_1}^2)}^2+{(1-x_1)}^2$$

(33)

$$f_{l,{\tau }_i}(x)=100{({\tau }_i{x}_2-x_1^2)}^2-2({(x_2-{\tau }_i)}^2+{(x_1-{\tau }_i)}^2)$$

(34)

where $x_1,x_2\in {[-5,10]}^2$, $f_h(x_1,x_2)$ is the high-fidelity version, and the low-fidelity versions are parameterized by ${\tau }_i$ [35]. The cost ratio (CR) for each fidelity by ${\tau }_i$ is settled as

Table 2. Parameters of multi-fidelity Rosebrock function.

Type i	1	2	3	4
$\tau$	1	0.4	0.65	0.9
CR	1	1000	10	100

. The four types fidelity of Kriging surrogate models optimization via GA are shown in Figure 13.

Initially, each surrogate model was formulated based on a single fidelity function. The sampling for each model was conducted in accordance with the CR, utilizing 40, 40,000, 400, and 4000 initial sample points ranging from high to low fidelity to establish the initial Kriging model. The Genetic Algorithm (GA) was employed for optimization, with a specified convergence accuracy of 0.01.

Table 3. Optimization results of different fidelity surrogate models.

TYPE	1	2	3	4
Iteration number	150	220	210	320
MAE	0.0065	0.0052	0.0103	0.093
Sample number	40	40,000	400	4000

outlines the number of iterations and the maximum absolute error (MAE) associated with the convergence of each model.

A high-fidelity surrogate model demonstrates the ability to achieve a satisfactory level of accuracy with a minimal number of iterations. In contrast, a low-fidelity surrogate model, constructed from a substantial number of samples, may attain commendable accuracy; however, it remains inferior to the high-fidelity counterpart. Furthermore, the implementation of a surrogate model that leverages a large sample size can effectively decrease the number of convergence iterations required for optimization using GA. To enhance our analysis, we developed a multi-fidelity MHK surrogate model utilizing the available test samples. Employing the sequential optimization of ASIS, we initially conducted an experiment focused on sample augmentation. Following the attainment of a surrogate model with a relatively high level of accuracy (PEMF = 0.01), we proceeded to execute further optimization, yielding the subsequent results:

The surrogate model based on the MHK was enhanced through the optimization framework outlined in the article, allowing for the incorporation of samples with varying levels of fidelity, shown in Figure 14. The red boxes indicate the new high-fidelity samples (i = 1) introduced by the ASIS method, while the blue boxes denote the collection of low-fidelity samples (i = 2, 3, 4) added. Following the addition of a limited number of high-fidelity samples (i = 14) and low-fidelity samples (i = 95), the PEMF of MHK achieved the stipulated requirement of 0.01. Optimization history of HF and MF surrogate model is shown in Figure 15.

An analysis is conducted to compare the performance of modern surrogate models that utilize various infilling sampling strategies. Results of the comparison are detailed in

Table 4. Results of Rosebrock function via modern surrogate models and infill strategies.

Type	Optimal	Iteration	HF Number	Error	Time(s)
CoKriging + augmented EI	0.0110	30	30	0.170	55
HK+augmented EI	0.0101	30	30	0.103	45
HK + VF-EI	−0.0082	26	12	0.045	28
HK + AMEI	0.0072	27	10	0.060	32
MHK + augmented EI	0.0100	30	30	0.120	40
MHK + VF-EI	−0.0090	27	11	0.055	29
MHK + AM-EI	0.0080	24	10	0.062	30
MHK + ASIS	0.0053	22	9	0.030	29

. The standard Rosebrock function features an extreme point, denoted as ${f_h}^{\ast }(x_1,x_2)=0$, located at (1, 1). However, as the low-fidelity CR scale varies, this extreme point also shifts accordingly. We established the maximum number of infilling iterations at 30 and repeated the experiment 10 times for each strategy to ensure the robustness of the findings. The average error for each experiment is expressed as the root-mean-square error (RMSE). The data indicate that the results for the HK+AMEI, MHK+AMEI, and MHK+ASIS strategies exhibit relative stability and accuracy. Significantly, the MHK+ASIS method proposed in this study minimizes the requirement for additional high-fidelity samples while concurrently reducing the mean error, thereby providing an efficient solution for experimental optimization.

4.3. Naca 0012 Airfoil Validation

The initial application within the field of aerospace engineering pertains to the optimization of a 2D NACA 0012 wing, specifically aimed at determining its optimal lift-to-drag ratio. For the purpose of multi-fidelity validation, a mesh grid family has been supplied by the Langley Research Center [36], as illustrated in Figure 16. Both mesh families share the same leading edge, and the analysis of the airfoil is conducted under conditions characterized by a Mach number of 0.15 and a Reynolds number of 6 million, facilitating free transition. For the validation of experimental results, wind tunnel test data from Ladson is utilized as high-fidelity validation data.

First, the simulation is performed on Xflow, 4 angles of attack (AoAs) are sampled from the Family 2 grid for HF data, and 16 AoAs are sampled from the Family 1 grid for LF data. With the initial LHS data sets, an MHK is built first as shown in Figure 17.

The objective of this multi-fidelity surrogate model is to identify the maximum lift ratio of the NACA 0012 airfoil. Historically, conducting wind tunnel experiments has been a costly endeavor, often succeeded by high-precision large eddy simulations (CFDs). This study utilizes the infill strategy detailed in this paper to incorporate additional samples, comparing its effectiveness with the widely used VFEI and augmented EI strategies. In summary, the proposed ASIS, which integrates three samples (two lf samples and one hf sample), demonstrates a performance that is comparable to the VFEI approach, which adds three samples (one lf sample and two hf samples). Conversely, the augmented EI strategy enhances accuracy by incorporating three hf samples.

The MHK infill sample optimizations via ASIS, VFEI, and augmented EI are shown in Figure 18. A thorough analysis indicates that while the augmented EI method improves overall accuracy and aligns closely with the high-fidelity model, it does neglect the trend information available from low-fidelity samples. The VFEI approach, leveraging a comparative strategy for each added sample, tends to favor high-fidelity samples in the pursuit of improved accuracy. In contrast, the strategy presented in this paper considers the potential probabilistic neighborhood of the added samples, effectively balancing the immediate accuracy enhancements provided by high-fidelity samples with the trend (gradient) improvements that low-fidelity sample points contribute.

Furthermore, the proposed ASIS enables PEMF error prediction to achieve an RMSE accuracy of less than 0.005 after the addition of only eight sample points, which is shown in Figure 19. This finding suggests that, prior to conducting actual tests, the integration of a limited number of samples through ASIS facilitates a progressive prediction of PEMF. Consequently, this method allows for the determination of the optimal number of samples required to attain the desired accuracy, thus minimizing the costs associated with trial and error.

5. Conclusions

In this article, the Adaptive Sequential Infill Sampling (ASIS) strategy was introduced, which significantly enhances the accuracy and efficiency of sample addition experiments in multi-fidelity surrogate modeling, particularly for Multi-fidelity Hamilton Kriging. The ASIS method was validated using two numerical simulations and was applied to a flow optimization analysis problem involving a NACA 0012 airfoil. The conclusions drawn from this study are as follows.

(1)

The Adaptive Sequential Infill Sampling strategy is an advancement of the Best Neighborhood-based Kriging infill method, specifically designed for the optimization of multi-fidelity experiments. This strategy effectively balances accuracy and efficiency in sequential experimental optimization by employing a Probabilistic Neural Network (PNN)-enhanced expected improvement (EI) methodology. Both numerical analyses and practical applications demonstrate the effectiveness of the proposed approach.

(2)

The Adaptive Sequential Infill Sampling strategy is an infill strategy that is used for experimental optimization error prediction. In order to balance the exploration between multi-fidelity models, a Probability Nearest Neighborhood method is used not only for error distribution prediction, but also for criteria optimization. Consequently, the ASIS framework delivers a robust estimate of errors throughout the sequential optimization process.

(3)

The Adaptive Sequential Infill Sampling strategy demonstrates greater utility and cost-effectiveness for multi-fidelity sequential infill sampling compared to certain advanced infill strategies. This is primarily due to its capability to compute posterior predictive distributions, which enhances the estimation of sampling errors in neighboring samples. Moreover, the application of PNN for predictions based on specified error sampling enables the utilization of fewer high-fidelity data points while still meeting the required root-mean-square error (RMSE) criteria.

In addition to the airfoil optimization engineering problem discussed earlier, the ASIS framework is fully capable of addressing a wide range of sequential optimization challenges that necessitate the incorporation of new samples. In future applications, ASIS has the potential to enhance its predictive capabilities in sequential experimental design by leveraging additional Bayesian frameworks.