The Effect of Multi-Additional Sampling for Multi-Fidelity Efﬁcient Global Optimization

: Powerful computer-aided design tools are presently vital for engineering product development. Efﬁcient global optimization (EGO) is one of the most popular methods for design of a high computational cost problem. The original EGO is proposed for only one additional sample point. In this work, parallel computing is applied to the original EGO process via a multi-additional sampling technique. The weak point of the multi-additional sampling is it has slower convergence rate when compared with the original EGO. This paper applies the multi-ﬁdelity technique to the multi-additional EGO process to see the effect of the number of multi-additional sampling points and the converge rate. A co-kriging method and a hybrid RBF/Kriging surrogate model are selected for the surrogate model in the EGO process to show the advantage of the multi-additional EGO process compared with the single-ﬁdelity Kriging surrogate model. In the experiment, single-additional sampling points and two to four number of multi-additional sampling per iteration are tested with symmetry and asymmetry mathematical test functions. The results show the hybrid RBF/Kriging surrogate model can obtain the similar optimal points when using the multi-additional sampling EGO.


Introduction
Presently, computer-aided optimization is one of the most important computer-aided design tools for product development and improvement. In recent years, the optimization design for expensive function was carried out using a combination of global optimization techniques. Due to high-cost computation function, a surrogate model and an optimization technique are combined to increase the efficiency of the optimization algorithm. The optimization with a surrogate model is popular and used in the various fields such as, mechanical engineering [1,2], chemical engineering [3,4], aerospace engineering [5,6], etc. One of the popular methods that combined the surrogate model and the optimization technique to solved an expensive function is an efficient global optimization (EGO) [7] because it can reduce the computational cost for an optimization process. The EGO process starts with generating an initial set of sampling points. The objective of this process is to increase the diversity of the sampling points. The most popular method for generating the initial sampling is the Latin Hypercube Sampling (LHS) [8]. Most of researchers try to increase the efficiency of this process [9][10][11]. Then, the surrogate model is used to estimate the actual function value. The original

Efficient Global Optimization
The EGO [7] procedure, shown in Figure 1, starts with the generated samples for low fidelity and high-fidelity using the design of experiment technique (DoE) [28]. In this research, the LHS [8] is selected for this process, because it could maintain the diversity of the data and can fix the number of experiments by a user. The next process is to construct the surrogate model, in which the Kriging method is selected for a single-fidelity optimization process and co-Kriging and RBF/Kriging hybrid surrogate model are selected for a multi-fidelity optimization process. The schematic diagram of the single-fidelity and multi-fidelity surrogate models is shown in Figure 2. After the surrogate model is constructed, an additional sample point for an optimization method can be found by maximizing the EI [7] via genetic algorithms [29].
The EI(E[I(x)]) at point x can be expressed as where φ is the probability density function representing uncertainty aboutŷ(x), which predicts a function value from a surrogate model. Additional sampling points based on using the EI is repeated until an objective function converges.

Kriging Surrogate Model
An ordinary Kriging model [30] can predict the unknown functionŷ(x) aŝ where µ(x) and ε(x) denote a global model and a local model, respectively. The global model (µ(x)) is expressed as where R is a matrix denoting the correlation between the sample points, and F is a vector containing the evaluation value of each sampling point. For the Kriging surrogate model, µ denotes the constant global model. The local model (ε(x)) is expressed as where r(x) is a vector written in terms of x.r(x) is a vector of sampling points. The correlation between ε(x) and ε(x i ) is a function of the distance between x and x i . In the Kriging surrogate model, the local derivation at an unknown point x is expressed using stochastic processes. Several design points are generated as sampling points and then a surrogate model is constricted using a Gaussian random function as the correlation function for the estimation of the trend using the stochastic process.

Co-Kriging Surrogate Model
A Co-Kriging surrogate model [22] is the extension of a Kriging surrogate model by combining the low-fidelity data and high fidelity data. A co-Kriging method can be predicted an unknown functionŷ(x) by Equation (3). However, the different point of these method is the co-Kriging model estimates the global model µ(x) and the local model ε(x) by combining data of the low-fidelity and the high-fidelity data by using the scaling parameter (ρ) of the different model (d), which is defined as: where F h is a vector that contains the high-fidelity evaluation value of each high-fidelity sample points, andŷ l (x h ) is the estimated value via the Kriging surrogate model of each high-fidelity sample points.

Rbf/Kriging Multi-Fidelity Surrogate Model
A RBF/Kriging multi-fidelity surrogate model [18] is the combination surrogate model between a Radial Basis Function (RBF) [31] and the Kriging surrogate model. In this surrogate model, the RBF is used to predicted the local model (µ(x)) of the Kriging surrogate model in Equation (3). The different point of the RBF/Kriging hybrid surrogate model and co-Kriging surrogate model is the hybrid surrogate model uses the low-fidelity data to predict the global model (µ(x)), but the co-Kriging method uses the low-fidelity data to predict the local model (ε(x)).The hybrid surrogate model can predict the unknown functionŷ(x) aŝ In this study, f RBF (x) is calculated using the data of low-fidelity data and the f RBF (x) at an unknown design point (x) can be expressed as where N is the number of sampling design solutions, w i denotes the weight coefficient of the ith design sampling point, and β represents a positive coefficient.

Multiple Additional Sampling for Multi-Fidelity Optimization
The original multi-fidelity EGO can be obtained one additional sample in each iteration where the process of the original multi-fidelity EGO is illustrated in Figure 1b. A designer can only employ computational resources for one evaluation in this case, whereas they can use a parallel environment in the initial sampling. Here, available computational resources are not employed during the additional sampling stage, as illustrated in Figure 3a.
Here, multi-additional sampling (MAs) for multi-fidelity optimization is proposed. In EGO with MAs, a sub-iteration is additionally included, as illustrated in Figure 3b. In this sub-iteration, an additional sampling solution (obtained by an iterative approach) is included to construct an updated multi-fidelity surrogate model by using the predicted point (x,ŷ(x)) as a temporal function value. Another additional sampling solution is obtained through EI maximization.  A schematic illustration of additional sampling in the proposed MAs method is illustrated in Figure 4. This process begins by having x EImax1 from using EI maximization on the initial surrogate model. The predicted valueŷ a1 is also computed, and then the model is then updated temporally using (x EImax1 ,ŷ a1 ). As the EI value around x EImax1 should not be greater, the next additional point (x EImax2 ) can be obtained. This process is repeated until an arbitrary number of additional sampling solutions are obtained. Thereafter, the exact values y a1 , y a2 , ... are calculated using a parallel evaluation environment for x EImax1 , x EImax2 , ... by using the expensive function. The set of these additional sampling points is finally added to the data set for the improvement of the model. In this sub-iteration, only the valuesŷ(x) predicted by the surrogate model are required. Therefore, multiple additional samples can be rapidly obtained. In the main iteration, the exact value for these additional sampling solutions can be computed using a parallel evaluation. As a result, a design using this EGO method can be completed more quickly compared to the original EGO. The number of sub-iterations can be found using the number of parallel evaluations, where the value of EI or the number of experiments could be the termination condition of the EGO process.

Investigation Problems
Here, the proposed method is investigated by solving test functions. The effect of multi-fidelity surrogate model, which are co-Kriging, RBF/Kriging surrogate model, for the multi-sampling EGO process results are compared with results given by the single-fidelity multi-sampling EGO to demonstrate the efficiency of these methods.
In this study, four test functions [18,32,33] are used. y h denotes a high-fidelity function and y l denoted a low-fidelity function.

•
The six-hump camel-back function (SC) is defined as follows: The Himmelblau function (HIM) is defined as follows: The Rosenbrock function (ROS) is defined as follows: The Colville function (COL) is defined as follows: In each investigation, 10 initial high-fidelity points y h and 200 low-fidelity point y l were acquired using LHS. The number of additional sampling points was set as 12 for all test functions and 2-4 multi-additional sampling were selected to test the effect of multi-additional sampling compared with the results of single-additional sampling for the single-fidelity and multi-fidelity EGO. The function of the low-fidelity data is slightly different from the high-fidelity data because in the real world computation there are some errors between these functions. However, the characteristic of the low-fidelity and high-fidelity function is they, to some extent, have similar function landscapes. To simulate the situation, the low fidelity function is always set as its high fidelity counterpart multiplied by a factor. This implies that the high fidelity function is always more accurate.  Table 1. As shown in Table 1, the optimum value of RBF/Kriging hybrid multi-fidelity EGO is the best optimum solution solved by SAS, and the optimum value by 2-MAs, 3-MAs and 4-MAs were obtained the similar optimum solution. The results from the Co-Kriging method was a good optimum value when solved with the SAS, 2-MAs, and 3-MAs and 4-MAs. However, the results from the RBF/Kriging hybrid multi-fidelity EGO obtained better solution than the Co-Kriging method because the results from the the Co-Kriging method obtained the similar solution as shown in Figure 6 when solving with the MAs shown in Figure 5. As shown in Figure 5b, the results of additional sampling for the same iteration mostly get similar values as that shown in Figure 6. The results of the Co-Kriging method shows that the Co-Kriging multi-fidelity EGO cannot increase the diversity of the additional sampling when solving with the MAs. The results from the ordinary Kriging method obtained the worst solution when solving the optimum problem with the MAs method. The cross-validation results [34] are shown in Figure 7 and Table 2. The cross-validation results show the RBF/Kriging hybrid multi-fidelity surrogate model obtains the high accuracy of the surrogate model for all cases, where the slope of the regression line can maintain a slope close to 1.0 and R 2 greater than 0.95 for all cases. However, for the Co-Kriging multi-fidelity surrogate model results, the surrogate model of 4-MAs cannot maintain the accuracy. The reason for these results is that the additional sampling of this case has low diversity. The Kriging surrogate model results show that the accuracy of this surrogate model is slightly increased when using higher multi-additional sampling, but the accuracy of the Kriging model is still less than the accuracy of the Co-Kriging and RBF/Kriging multi-fidelity surrogate models.    Table 3. As shown in Table 3, the optimum value of Co-Kriging multi-fidelity EGO obtained a good optimum solution with solving by SAS, 2-MAs and 3-MAs. However, the optimum solution of the 4-MAs obtained slightly worse results. For this case, the 4-MAs was not effective because the Co-Kriging method obtained the similar solution when solving with the MAs shown in Figure 8b. This effect was similar when solving the SC problem. The results from the RBF/Kriging multi-fidelity EGO shows the additional sampling has a higher diversity than the results from the Co-Kriging multi-fidelity EGO, but it still obtained good solution for all cases. The results from the ordinary Kriging method show the Kriging method was not good to solve this problem with high MAs. These results could be concluded the multi-fidelity optimization could be helpful to solve the EGO with the MAs because the original EGO will have lower accuracy of the surrogate model. The cross-validation results of the HIM are shown in Table 4. It is seen that the Co-Kriging multi-fidelity EGO method obtained the best solution because the Co-kriging multi-fidelity surrogate model obtained the highest accuracy for SAS, 2-MAs and 3-MAs. However, the Co-Kriging multi-fidelity surrogate model could not maintain the accuracy of the surrogate model for the case 4-MAs. On the other hand, the RBF/Kriging hybird multi-fidelity surrogate model could not obtain the highest accuracy for this test problem, but it can maintain the accuracy of the surrogate model for all cases, while the slope of the regression line can maintain a slope close to 1.0 and R 2 greater than 0.90 for all cases. The results of the Kriging surrogate model reveal that the accuracy of the surrogate model is decreased when using the higher number of MAs. This test problem shows that the multi-fidelity technique could be helpful when using the MAs technique because it could be maintain the accuracy of the surrogate model when using the higher number of MAs. Table 3. Optimum value of the HIM problem.    Figure 9 shows the comparison of additional sampling of the ROS problem across the RBF/Kriging hybrid multi-fidelity EGO, Co-Kriging multi-fidelity EGO and original EGO. The optimum value that obtained by RBF/Kriging hybrid multi-fidelity EGO, Co-Kriging multi-fidelity EGO and original EGO with SAS, 2-MAs, 3-MAs and 4-MAs are shown in Table 5. As shown in Table 5, the optimum value of the RBF/Kriging obtained the best solution for SAS, 3-MAs and 4-MAs. The results from the Co-Kriging obtained the worst optimal solution when solving by SAS, but the solution of higher number of multiple-additional sampling could be accepted. On the other hand, the results from the original EGO shows the original EGO could be solved this problem well when using the SAS, but it cannot get a good solution when using higher number of multiple-additional sampling points. The cross-validation results of the ROS are shown in Table 6. The RBF/Kriging multi-fidelity EGO gives the best solution because the RBF/Kriging multi-fidelity surrogate model obtained the highest accuracy for all cases, where the slope of the regression line can maintain a slope close to 1.0 and R 2 greater than 0.95 for all cases. The Co-Kriging multi-fidelity surrogate model could not obtained the high accuracy for this test problem, but it still have higher accuracy than the Kriging surrogate model. In addition, the accuracy of the Co-Kriging multi-fidelity surrogate model depends on the number of MAs. For this case, the accuracy the Co-Kriging multi-fidelity surrogate model is decreased when using the higher number of MAs. In the same way, The results of the Kriging surrogate model show that the accuracy of the surrogate model is decreased when using the higher number of MAs. This test problem still shows the multi-fidelity technique could be helpful with the use of the MAs technique because it could maintain the accuracy of the surrogate model when using the higher number of MAs. Table 5. Optimum value of the ROS problem.   Figure 10 shows the comparison of additional sampling of the ROS problem across the RBF/Kriging hybrid multi-fidelity EGO, Co-Kriging multi-fidelity EGO and original EGO. The optimum value obtained by RBF/Kriging hybrid multi-fidelity EGO, Co-Kriging multi-fidelity EGO and original EGO with SAS, 2-MAs, 3-MAs and 4-MAs are shown in Table 7. The optimum value obtained by the Co-Kriging multi-fidelity EGO could obtain the best solution when solving this problem with SAS, but it cannot obtain a good solution when solving with 3-MAs and 4-MAs. These results show the Co-Kriging has high impact for multi-additional sampling. The RBF/Kriging hybrid multi-fidelity cannot obtain the best solution when solving with the SAS. However, this method could obtain a similar optimum solution when solving with all methods. The optimal solutions by the original EGO received results lower than the RBF/Kriging hybrid multi-fidelity EGO and the Co-Kriging method except when solving with 4-MAs. The cross-validation results of the COL are shown in Table

Discussion
According to the results from four optimization test functions, the multi-fidelity optimization could reduce the effect of the multi-additional sampling process because the multi-fidelity optimization could obtain higher accuracy of the surrogate model. From the cross-validation results in Tables 2, 4, 6, and 8, it is shown that the accuracy of the multi-fidelity surrogate model could obtain higher accuracy when solving the problem with the higher number of MAs. From some test functions, the Co-Kriging multi-fidelity surrogate model gives the highest accuracy for the SAS case, but it cannot maintain the accuracy of the surrogate model when solving the solution with the higher number MAs. The RBF/Kriging hybrid multi-fidelity surrogate model shows that it is able to maintain the good accuracy when using the higher number of MAs. It can be concluded that the multi-fidelity technique could increase the efficiency of the MAs EGO process because it can maintain the accuracy of the surrogate model when using the higher number of MAs.

Conclusions
This study investigated the effect of multi-fidelity technique to the multi-additional EGO process. The RBF/Kriging hybrid multi-fidelity EGO and CO-Kriging multi-fidelity EGO are selected to test the effect of this framework. The MAs technique could help the EGO process to run the parallel computing that can reduce the computation time. However, the weak point of the MAs technique combined with the EGO results in the accuracy of the surrogate model being reduced, which leads to the use of resources to find the optimum solution. Consequently, the multi-fidelity technique is proposed to combineEGO with MAs to increase the efficiency of the EGO process by increasing the accuracy of the surrogate model.
To examine the multi-fidelity EGO framework, four test functions are solved, and the results are compared with those obtained by the ordinary Kriging EGO. The optimum results for the test functions show that the RBF/Kriging hybrid multi-fidelity EGO has the lowest effect when using the RBF/Kriging hybrid multi-additional sampling framework because the RBF/Kriging hybrid multi-fidelity surrogate model can maintain the accuracy of the surrogate model when it runs with higher number of MAs. For some of the number of MAs test functions the Co-Kriging could receive the best optimal solution when solving with the SAS but it cannot obtain a good solution when solving via multi-additional sampling framework. The ordinary EGO has a high-effect when using the multi-additional sampling framework. The optimal solution of the ordinary EGO gets worse results when solving the optimal solution via higher number of multi-additional sampling points. This results suggest that the hybrid RBF/Kriging multi-fidelity EGO is the best choice for the MAs framework because it can maintain the accuracy of the surrogate model that affects the efficiency of the optimization process.

Conflicts of Interest:
The authors declare no conflict of interest.

Abbreviations
The following abbreviations, nomenclature and symbols are used in this manuscript: