Effects of Exploration Weight and Overtuned Kernel Parameters on Gaussian Process-Based Bayesian Optimization Search Performance

Abstract

Gaussian process-based Bayesian optimization (GPBO) is used to search parameters in machine learning, material design, etc. It is a method for finding optimal solutions in a search space through the following four procedures. (1) Develop a Gaussian process regression (GPR) model using observed data. (2) The GPR model is used to obtain the estimated mean and estimated variance for the search space. (3) The point where the sum of the estimated mean and the weighted estimated variance (upper confidence bound, UCB) is largest is the next search point (in the case of a maximum search). (4) Repeat the above procedures. Thus, the generalization performance of the GPR is directly related to the search performance of the GPBO. In procedure (1), the kernel parameters (KPs) of the GPR are tuned via gradient descent (GD) using the log-likelihood as the objective function. However, if the number of iterations of the GD is too high, there is a risk that the KPs will overfit the observed data. In this case, because the estimated mean and variance output by the GPR model are inappropriate, the next search point cannot be properly determined. Therefore, overtuned KPs degrade the GPBO search performance. However, this negative effect can be mitigated by changing the parameters of the GPBO. We focus on the weight of the estimated variances (exploration weight) of the UCB as one of these parameters. In a GPBO with a large exploration weight, the observed data appear in various regions in the search space. If the KP is tuned using such data, the GPR model can estimate the diverse regions somewhat correctly, even if the KP overfits the observed data, i.e., the negative effect of overtuned KPs on the GPR is mitigated by setting a larger exploration weight for the UCB. This suggests that the negative effect of overtuned KPs on the GPBO search performance may be related to the UCB exploration weight. In the present study, this hypothesis was tested using simple numerical simulations. Specifically, GPBO was applied to a simple black-box function with two optimal solutions. As parameters of GPBO, we set the number of KP iterations of GD in the range of 0–500 and the exploration weight as $\{1,5\}$. The number of KP iterations expresses the degree of overtuning, and the exploration weight expresses the strength of the GPBO search. The results indicate that, in the overtuned KP situation, GPBO with a larger exploration weight has better search performance. This suggests that, when searching for solutions with a small GPBO exploration weight, one must be careful about overtuning KPs. The findings of this study are useful for successful exploration with GPBO in all situations where it is used, e.g., machine learning hyperparameter tuning.

1. Introduction

Gaussian process-based Bayesian optimization (GPBO) optimizes black-box functions and is adopted to save time and/or reduce costs. For example, it is used for concrete design [1,2], material design [3,4,5], and tuning hyperparameters in machine learning (for support vector machines [6,7,8], random forest models [9,10] and neural networks [9,11,12], etc.). Appropriate parameters can be obtained more rapidly using GPBO compared with simple methods such as grid search. For example, Wu et al. [9] reported that the computation time of hyperparameter tuning in machine learning can be reduced significantly using GPBO. Snoek et al. [11] reported that when GPBO was used for tuning, the developed convolutional neural networks had higher generalization scores than networks with parameters tuned by an expert of machine learning.

To perform GPBO, a Gaussian process regression (GPR) model must be developed, which is computationally expensive. The computational cost of the kernel inverse matrix required in GPBO is $\mathcal{O}\left(n^3\right)$ for a data size of n [13]. Due to the fact that n increases with each successive GPBO iteration, the computational cost increases. Therefore, sparse matrix methods [13,14,15,16] and mini-batch methods [17] have been proposed for reducing the computational cost.

In the log-likelihood-based objective function, the kernel inverse matrix is used to tune the hyperparameters of the GPR model [18]. Gradient-based methods [17,19,20], evolutionary algorithms [21,22], and Markov chain Monte Carlo methods [23] have been adopted for hyperparameter tuning of the GPR model. Cross-validation [18] is adopted for tuning the kernel parameters (KPs) to avoid overfitting to the observed data. However, because the size of the observed data gradually increases, performing cross-validation at the beginning of GPBO is difficult. Therefore, we cannot perform cross-validation at the beginning of GPBO to tune the hyperparameters of the GPR model. Moreover, cross-validation increases the computational cost [18].

In the early stages of GPBO, the sample size is insufficient. Additionally, it is difficult to properly tune a GPR model by a small sample size [24,25]. Therefore, consider the situation wherein the KPs are tuned using all samples instead of using a method that reduces the number of data, such as cross-validation. In this case, because there are no data for validation, it is not known to what extent the KPs should be fitted to the observed data. Thus, there is a risk of overtuning the GPR model. Overtuned GPR models can correctly estimate observed regions but cannot properly estimate unobserved regions [26]. GPBO is an algorithm that searches for the optimal solution in unobserved regions. Therefore, if unobserved areas cannot be correctly estimated, a proper search cannot be performed. From this viewpoint, the search performance of Bayesian optimization using an overfitted GPR model is expected to be poor.

The negative effect of overtuned KPs on the search performance depends on other parameters of GPBO. We focus on the exploration weight of the upper confidence bound (UCB) [27,28], which is a GPBO parameter. Generally, when the exploration weight is set to a large value, the next search point tends to be selected from regions with insufficient observations; therefore, the observed samples appear in various areas. In this case, even if the KPs are overfitted to the observed samples, the GPR model can correctly estimate the various input domains. In contrast, with a small exploration weight, because the observed samples appear in only limited regions, the overfitted GPR model only estimates limited regions; therefore, its generalization score is worse.

This implies that the risk of overtuned KPs degrading the search performance depends on the exploration weight. We verified this hypothesis by analyzing the relationships among the overtuned KPs, exploration weight, and GPBO search performance. The results indicated that, for GPBO, more attention must be paid to the overtuning of the KPs in the case of smaller exploration weights. Additionally, it is necessary to pay attention to avoid overtuning KPs when searching for the solution via GPBO with a small exploration weight. These findings are useful for successful exploration with GPBO in all situations where it is used, e.g., hyperparameter tuning in machine learning.

As indicated by previous studies [21,24,25], the likelihood function in GPR may have multiple minimal solutions, making it difficult to find a globally optimal solution. Therefore, we focused on gradient descent (GD), which can rapidly obtain a local minimal solution, as a method for tuning the KPs. This method is widely used for tuning the KPs of GPR models [17,19,20].

2. Gaussian Process-Based Bayesian Optimization

2.1. Surrogate Model

In this study, we use the GPR model as a surrogate model for Bayesian optimization, which we call GPBO. This method outputs the estimated average and variance by assuming a Gaussian distribution from the dataset $\mathcal{D}$ consisting of pairs of observed input and output values.

Here, the output value y is obtained as

$$\begin{array}{c}\hfill y=f\left(\mathit{x}\right)+ϵ,\end{array}$$

(1)

where $\mathit{x}={[x_1\cdots x_D]}^{\top }$ is the D-dimensional input vector, $f\left(\mathit{x}\right)$ is a black-box function, and $ϵ$ denotes observation noise. In addition, we consider a situation in which a dataset

$$\begin{array}{c}\hfill \mathcal{D}=\left\{({\mathit{x}}_n,y_n)\right|n=1,\cdots ,N\}\end{array}$$

(2)

is collected via N observations. In the case of GPR, assuming that the average of y is zero, the average and variance values of the output $y^{\prime }$ for the new input data ${\mathit{x}}^{\prime }$ are given as follows:

$$\begin{array}{cc}\hfill \mathbb{E}\left[y^{\prime }\right|{\mathit{x}}^{\prime },\mathcal{D},\mathit{\theta }]& ={\mathit{k}}^{\prime }{\left(\mathit{\theta }\right)}^{\top }\mathit{K}{\left(\mathit{\theta }\right)}^{-1}\mathit{y},\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill \mathbb{V}\left[y^{\prime }\right|{\mathit{x}}^{\prime },\mathcal{D},\mathit{\theta }]& =k^{\prime \prime }\left(\mathit{\theta }\right)-{\mathit{k}}^{\prime }{\left(\mathit{\theta }\right)}^{\top }\mathit{K}{\left(\mathit{\theta }\right)}^{-1}{\mathit{k}}^{\prime }\left(\mathit{\theta }\right)\hfill \end{array}$$

(4)

where $\mathit{y}={[y_1\cdots y_N]}^{\top }$. $\mathit{K}\left(\mathit{\theta }\right)$ is a kernel matrix defined as follows:

$$\begin{array}{c}\hfill \mathit{K}\left(\mathit{\theta }\right)=\left[\begin{array}{c}k({\mathit{x}}_i,{\mathit{x}}_j;\mathit{\theta })\end{array}\right]\in {\mathbb{R}}_{>0}^{N\times N}\end{array}$$

(5)

where $k({\mathit{x}}_i,{\mathit{x}}_j;\mathit{\theta })$ denotes the kernel function. For Gaussian kernels, the kernel function is defined as

$$\begin{array}{c}\hfill k({\mathit{x}}_i,{\mathit{x}}_j;\mathit{\theta })={\theta }_{1}exp\left(-\frac{\left|\right|{\mathit{x}}_i-{\mathit{x}}_j{\left|\right|}_2^2}{{\theta }_2}\right)+{\theta }_3\delta (i,j),\delta (i,j)=\left\{\begin{array}{cc}0,\hfill & i\ne j\hfill \\ 1,\hfill & i=j\hfill \end{array}\right.,\mathit{\theta }\in {\mathbb{R}}_{>0}^3\end{array}$$

(6)

where $\mathit{\theta }=({\theta }_1,{\theta }_2,{\theta }_3)$ is a vector comprising three KPs. Due to the fact that the estimation accuracy depends on the KPs, appropriate tuning is important. $k^{\prime \prime }\left(\mathit{\theta }\right)$ and ${\mathit{k}}^{\prime }\left(\mathit{\theta }\right)$ are the kernels related to the observed samples ${\mathit{x}}_i$ and new sample ${\mathit{x}}^{\prime }$, respectively, and are defined as follows:

$$\begin{array}{c}\hfill k^{\prime \prime }\left(\mathit{\theta }\right)=k({\mathit{x}}^{\prime },{\mathit{x}}^{\prime };\mathit{\theta }),{\mathit{k}}^{\prime }\left(\mathit{\theta }\right)=\left[\begin{array}{c}k({\mathit{x}}_i,{\mathit{x}}^{\prime };\mathit{\theta })\end{array}\right]\in {\mathbb{R}}_{>0}^N.\end{array}$$

(7)

2.2. Tuning Kernel Parameters

As stated in the Introduction, the KP vector $\mathit{\theta }$ is tuned using GD. Due to the fact that the $\mathit{\theta }$ that maximizes the generation probability of the observed output $\mathit{y}$ is desirable, using the log-likelihood, the objective function L is defined as

$$\begin{array}{cc}\hfill L\left(\mathit{\theta }\right)& =logp\left(\mathit{y}\right|\mathit{\theta })\hfill \\ \hfill & =log\mathcal{N}(\mathbf{0},\mathit{K}(\mathit{\theta }\left)\right)\hfill \\ \hfill & =log\left(\frac{1}{\sqrt{{\left(2\pi \right)}^N\left|\mathit{K}\left(\mathit{\theta }\right)\right|}}exp\left(-\frac{1}{2}{\mathit{y}}^{\top }\mathit{K}{\left(\mathit{\theta }\right)}^{-1}\mathit{y}\right)\right)\hfill \\ \hfill & \propto -log\left|\mathit{K}\left(\mathit{\theta }\right)\right|-{\mathit{y}}^{\top }\mathit{K}{\left(\mathit{\theta }\right)}^{-1}\mathit{y}.\hfill \end{array}$$

(8)

We assume that $p\left(\mathit{y}\right|\mathit{\theta })$ is a Gaussian distribution consisting of the average $\mathbf{0}$ and the covariance matrix $\mathit{K}\left(\mathit{\theta }\right)$. When we adopt the gradient method for the objective function, $\mathit{\theta }$ may be negative. Due to the fact that the Gaussian KP $\mathit{\theta }$ requires a plus (see Equation (6)), $\mathit{\theta }$ is redefined as follows:

$$\begin{array}{cc}\hfill \mathit{\theta }& =g\left({\mathit{\theta }}^{\prime }\right)\hfill \\ \hfill & =exp\left({\mathit{\theta }}^{\prime }\right),{\mathit{\theta }}^{\prime }={\left[{\theta }_1^{\prime }{\theta }_2^{\prime }{\theta }_3^{\prime }\right]}^{\top }\hfill \end{array}$$

(9)

where the map g is

$$\begin{array}{c}\hfill g:{\mathbb{R}}^3\mapsto {\mathbb{R}}_{>0}^3.\end{array}$$

(10)

Therefore, when the gradient method is run as the search target ${\mathit{\theta }}^{\prime }$ and the obtained parameter is transformed into $\mathit{\theta }$ using Equation (9), $\mathit{\theta }$ will certainly be positive. Therefore, we use the update equation for the KPs, as follows:

$$\begin{array}{c}\hfill {\mathit{\theta }}^{\prime (t_{\mathrm{g}}+1)}={\mathit{\theta }}^{\prime \left(t_{\mathrm{g}}\right)}+\gamma \nabla L\left(exp\left({\mathit{\theta }}^{\prime \left(t_{\mathrm{g}}\right)}\right)\right)\end{array}$$

(11)

where $t_{\mathrm{g}}$ represents the iteration count, ${\mathit{\theta }}^{\prime \left(t_{\mathrm{g}}\right)}$ is the $t_{\mathrm{g}}$-th specific value of ${\mathit{\theta }}^{\prime }$, and $\gamma$ represents the learning rate. This partial differentiation is as follows:

$$\begin{array}{cc}\hfill \nabla L\left(exp\left({\mathit{\theta }}^{\prime }\right)\right)=& -\mathrm{tr}\left(\mathit{K}{(exp\left({\mathit{\theta }}^{\prime }\right))}^{-1}\nabla \mathit{K}(exp\left({\mathit{\theta }}^{\prime }\right))\right)\hfill \\ & \hfill +{\left(\mathit{K}{(exp\left({\mathit{\theta }}^{\prime }\right))}^{-1}\mathit{y}\right)}^{\top }\nabla \mathit{K}(exp\left({\mathit{\theta }}^{\prime }\right))\left(\mathit{K}{(exp\left({\mathit{\theta }}^{\prime }\right))}^{-1}\mathit{y}\right).\end{array}$$

(12)

Due to the fact that $\mathit{K}$ is the matrix consisting of kernel functions, i.e., $\nabla \mathit{K}(exp\left({\mathit{\theta }}^{\prime }\right))$, the partial derivative of $k({\mathit{x}}_i,{\mathit{x}}_j;exp\left({\mathit{\theta }}^{\prime }\right))$ with respect to ${\theta }_1^{\prime },{\theta }_2^{\prime },{\theta }_3^{\prime }$ is required, i.e.,

$$\begin{array}{cc}\hfill \frac{\partial k({\mathit{x}}_i,{\mathit{x}}_j;exp\left({\mathit{\theta }}^{\prime }\right))}{\partial {\theta }_1^{\prime }}& =exp\left({\theta }_1^{\prime }\right)exp\left(-\frac{\left|\right|{\mathit{x}}_i-{\mathit{x}}_j{\left|\right|}_2^2}{exp\left({\theta }_2^{\prime }\right)}\right),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill \frac{\partial k({\mathit{x}}_i,{\mathit{x}}_j;exp\left({\mathit{\theta }}^{\prime }\right))}{\partial {\theta }_2^{\prime }}& =\frac{exp\left({\theta }_1^{\prime }\right)}{exp\left({\theta }_2^{\prime }\right)}\left|\right|{\mathit{x}}_i-{\mathit{x}}_j{\left|\right|}_2^{2}exp\left(-\frac{\left|\right|{\mathit{x}}_i-{\mathit{x}}_j{\left|\right|}_2^2}{exp\left({\theta }_2^{\prime }\right)}\right),\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \frac{\partial k({\mathit{x}}_i,{\mathit{x}}_j;exp\left({\mathit{\theta }}^{\prime }\right))}{\partial {\theta }_3^{\prime }}& =exp\left({\theta }_3^{\prime }\right)\delta (i,j).\hfill \end{array}$$

(15)

This calculation technique was described by Mochihashi et al. [29].

After the parameter is updated $T_{\mathrm{g}}$ times, we obtain ${\mathit{\theta }}^{\prime \left(T_{\mathrm{g}}\right)}$. Substituting this into Equation (9) yields the tuned KP ${\mathit{\theta }}^{\left(T_{\mathrm{g}}\right)}$. According to Equation (10), the obtained parameter ${\mathit{\theta }}^{\left(T_{\mathrm{g}}\right)}$ satisfies the condition of ${\mathbb{R}}_{>0}^3$.

2.3. Optimization Algorithm for Experiments

Using Equations (3) and (4), the acquisition function of GPBO is defined as

$$\begin{array}{c}\hfill A({\mathit{x}}^{\prime };\beta ,\mathit{\theta })=\mathbb{E}\left[y^{\prime }\right|{\mathit{x}}^{\prime },\mathcal{D},\mathit{\theta }]+\beta \sqrt{\mathbb{V}\left[y^{\prime }\right|{\mathit{x}}^{\prime },\mathcal{D},\mathit{\theta }]}\end{array}$$

(16)

and it is called the UCB [27,28]. Here, with N observed samples, the $N+1$-th (next) observation sample is determined as follows:

$$\begin{array}{cc}\hfill {\mathit{x}}_{N+1}& =\underset{{\mathit{x}}^{\prime }\in \mathbf{\Psi }}{\mathrm{argmax}}A({\mathit{x}}^{\prime };\beta ,\mathit{\theta }),\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill y_{N+1}& =f\left({\mathit{x}}_{N+1}\right)+ϵ\hfill \end{array}$$

(18)

where $\mathbf{\Psi }$ represents the domain of the input ${\mathit{x}}^{\prime }$. The second term of the acquisition function, which is defined in Equation (16), controls the search weight. When $\beta$ is set to a large value, exploration is emphasized. Subsequently, the observation dataset is updated as

$$\begin{array}{c}\hfill \mathcal{D}=\mathcal{D}\cup \left\{({\mathit{x}}_{N+1},y_{N+1})\right\}.\end{array}$$

(19)

By performing this process $T_{\mathrm{b}}$ times, we obtain the maximum value $y_{\max }$ and the approximate solution ${\mathit{x}}_{\max }$ as follows:

$$\begin{array}{c}\hfill y_{\max }=max{\mathcal{D}}_y,{\mathit{x}}_{\max }=\underset{\mathit{x}\in {\mathcal{D}}_{\mathit{x}}}{\mathrm{argmax}}{\mathcal{D}}_y\end{array}$$

(20)

where ${\mathcal{D}}_y$ and ${\mathcal{D}}_x$ are the sets consisting of the output and input values, respectively, of the set $\mathcal{D}$. This procedure is presented in Algorithm 1.

Algorithm 1 Verification-targeted optimization algorithm

Input:    Initial observation dataset $\mathcal{D}$, maximum number of BO iterations $T_{\mathrm{b}}$,    maximum number of GD iterations $T_{\mathrm{g}}$, GD learning rate $\gamma$, exploration weight $\beta$,    black-box function $f\left(\mathit{x}\right)$, observation noise $ϵ$,    search space $\mathbf{\Psi }$, initial KP ${\mathit{\theta }}^{\left(0\right)}=exp\left({\mathit{\theta }}^{\prime \left(0\right)}\right)$ Output:    Solution ${\mathit{x}}_{\max }$ and its value $y_{\max }$ ................................................................................................................................................ 1: for $t_{\mathrm{b}}=1$ to $T_{\mathrm{b}}$ do 2:    for $t_{\mathrm{g}}=0$ to $T_{\mathrm{g}}-1$ do 3:      ${\mathit{\theta }}^{\prime (t_{\mathrm{g}}+1)}\leftarrow {\mathit{\theta }}^{\prime \left(t_{\mathrm{g}}\right)}+\gamma \nabla L\left(exp\left({\mathit{\theta }}^{\prime \left(t_{\mathrm{g}}\right)}\right)\right)$ 4:    end for 5:    ${\mathit{\theta }}^{\left(T_{\mathrm{g}}\right)}\leftarrow exp\left({\mathit{\theta }}^{\prime \left(T_{\mathrm{g}}\right)}\right)$ 6:    ${\mathit{x}}_{N+1}\leftarrow \underset{{\mathit{x}}^{\prime }\in \mathbf{\Psi }}{\mathrm{argmax}}A({\mathit{x}}^{\prime };\beta ,{\mathit{\theta }}^{\left(T_{\mathrm{g}}\right)})$ 7:    $y_{N+1}\leftarrow f\left({\mathit{x}}_{N+1}\right)+ϵ$ 8:    $\mathcal{D}\leftarrow \mathcal{D}\cup \left\{({\mathit{x}}_{N+1},y_{N+1})\right\}$ 9: end for 10: $y_{\max }\leftarrow max{\mathcal{D}}_y$, ${\mathit{x}}_{\max }\leftarrow \underset{\mathit{x}\in {\mathcal{D}}_{\mathit{x}}}{\mathrm{argmax}}{\mathcal{D}}_y$ 11: return ${\mathit{x}}_{\max }$, $y_{\max }$ ................................................................................................................................................ Notes: · Lines 2–4: Tuning the KPs via GD; · Lines 6–8: Decisions regarding the next search point and observation; · Lines 10–11: Obtain an approximate solution. “GD”: gradient descent, “BO”: Bayesian optimization

2.4. Indices

The GPBO search performance depends on the training and generalization errors of the surrogate model. Therefore, the training and generalization errors are, respectively, defined as

$$\begin{array}{cc}\hfill E_{\mathrm{t}}\left(\mathit{\theta }\right)& =\frac{1}{N}\sum _{n=1}^N{\left(y_n-\mathbb{E}\left[y_n\right|{\mathit{x}}_n,\mathcal{D},\mathit{\theta }]\right)}^2,\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill E_{\mathrm{v}}\left(\mathit{\theta }\right)& =\frac{1}{\left|\mathbf{\Psi }\right|}\sum _{\mathit{x}\in \mathbf{\Psi }}{\left(f\left(\mathit{x}\right)-\mathbb{E}\left[y\right|\mathit{x},\mathcal{D},\mathit{\theta }]\right)}^2.\hfill \end{array}$$

(22)

Moreover, the GPBO search performance depends on whether the next search point ${\mathit{x}}_{N+1}$ and past observed points ${\mathit{x}}_1,\cdots ,{\mathit{x}}_N$ are close, i.e., if the search is performed only near past observation points, the search performance is poor. Therefore, we determine whether they are close to each other as follows:

$$\begin{array}{c}\hfill min\left\{\right||{\mathit{x}}_{N+1}-{\mathit{x}}_n|{|}_2|n=1,\cdots ,N\}<\omega \end{array}$$

(23)

where $\omega$ denotes the threshold value. When the minimum distance on the left side is less than $\omega$, the next search point ${\mathit{x}}_{N+1}$ is close to the previously observed points ${\mathit{x}}_1,\cdots ,{\mathit{x}}_N$.

Furthermore, whether the KP vector ${\mathit{\theta }}^{\prime \left(T_{\mathrm{g}}\right)}$ obtained as $T_{\mathrm{g}}$ times GD has converged is determined as follows:

$$\begin{array}{c}\hfill \frac{\left|\right|{\mathit{\theta }}^{\prime \left(T_{\mathrm{g}}\right)}-{\mathit{\theta }}^{\prime (T_{\mathrm{g}}-1)}{\left|\right|}_2}{\left|\right|{\mathit{\theta }}^{\prime (T_{\mathrm{g}}-1)}{\left|\right|}_2}<\tau \end{array}$$

(24)

where $\tau$ is the threshold for the convergence criterion. The term on the left side is called the “relative change in parameters,” and it is widely used as a convergence criterion [30,31,32].

3. Experiments

3.1. Objective and Outline

From the definition of $A({\mathit{x}}^{\prime };\beta ,\mathit{\theta })$, the search performance of GPBO depends on the KP $\mathit{\theta }$. Due to the fact that the KP is obtained via GD, the maximum number of iterations for updating $T_{\mathrm{g}}$ significantly affects the GPBO search performance. When the number of iterations $T_{\mathrm{g}}$ is too high, the surrogate model is overfitted to the observation samples. In such cases, a relatively poor search performance is expected. When the exploration weight is set to a large value, because the observed samples occur in various regions in the input domain, the generalization error caused by overfitting may be mitigated. In contrast, when a small value is used for the exploration weight, because observation samples only occur in limited regions, the generalization error can be larger. Due to the fact that the generalization performance of the surrogate model affects the GPBO search performance, we consider that the negative effect of the overfitted KPs on the search performance depends on the exploration weight $\beta$. This hypothesis was verified through a simple numerical simulation.

We adopted the black-box function $f\left(\mathit{x}\right)$ and its search space $\mathbf{\Psi }$ as follows:

$$\begin{array}{c}\hfill f\left(\mathit{x}\right)=\frac{1}{3}sin\frac{x_1}{3}sin\frac{x_2}{3}-\frac{{\left(x_1\right)}^2}{300}-\frac{{\left(x_2\right)}^2}{300}+\frac{5}{6},\mathit{x}\in \mathbf{\Psi }:=[-10,10]\times [-10,10].\end{array}$$

(25)

For simplicity, we set the observation noise to $ϵ=0$. The black-box function is shown in Figure 1. The adopted black-box function had two optimal solutions.

We used GPBO based on Algorithm 1, and the values of the input parameters are presented in

Table 1. Input values for Algorithm 1. $\{·\}$ is a set consisting of multiple elements, i.e., multiple patterns were adopted.

Parameters	Value(s)
Maximum number of GD iterations $T_{\mathrm{g}}$	$\{0,50,100,\cdots ,500\}$
GD learning rate $\gamma$	$0.01$
Maximum number of BO iterations $T_{\mathrm{b}}$	50
Exploration weight $\beta$	$\{1,5\}$
Initial KP ${\mathit{\theta }}^{\left(0\right)}$	${\left[111\right]}^{\top }$
Search space $\mathbf{\Psi }$	Equation (25)
Black-box function $f\left(\mathit{x}\right)$	Equation (25)
Observation noise $ϵ$	0
Initial observation dataset $\mathcal{D}$	Four points randomly selected in $\mathbf{\Psi }$

. To verify the aforementioned hypothesis, the effects of the degree of kernel-parameter tuning and the exploration weight on the GPBO search performance were analyzed. Therefore, we adopted multiple values for the maximum number of GD iterations $T_{\mathrm{g}}$ and exploration weight $\beta$. Moreover, the initial observation dataset $\mathcal{D}$ comprised four randomly selected points ${\mathit{x}}_1,\cdots ,{\mathit{x}}_4$ from the search space $\mathbf{\Psi }$. To enhance the reliability of the results, we performed 20 experiments in which the same parameter conditions were used but the random seed identification was changed.

3.2. Results and discussions

The relationship between the maximum number of GD iterations and the kernel-parameter tuning is presented in Figure 2A. The figure shows the rate at which the KPs ${\mathit{\theta }}^{\left(T_{\mathrm{g}}\right)}$ obtained via the GD of each Bayesian optimization step of 50 iterations converged. We investigated whether the convergence was determined by Equation (24) for $\tau =0.01$. The results represent the averages of 20 trials with different seeds. For $\beta =1$ and 5, Figure 2A suggests that the log-likelihood converged when the maximum number of GD iterations $T_{\mathrm{g}}$ was set as more than approximately 300. Therefore, we regarded the KPs as overtuned at >300 GD iterations.

Figure 2B shows the rate of finding the two optimal solutions in 20 trials with different seeds. The maximum number of GD iterations was the degree of kernel-parameter tuning via GD. For $\beta =1$, a larger maximum number of GD iterations corresponded to worse search performance. In contrast, for $\beta =5$, even when the maximum number of GD iterations was large, the rates of finding solutions remained high. Therefore, we consider that even if KPs are overtuned to the observation dataset, in cases of a large exploration weight, GPBO can adequately search for solutions (of course, it is desirable to avoid overfitting). Figure 2C presents the average number of Bayesian optimization steps needed for finding two optimal solutions calculated using only the results where two optimal solutions were successfully found. As shown, for a smaller value of $\beta$ and a larger number of KPs tuned, the optimal solutions were found faster. However, as shown in Figure 2B, the rate of correctly finding the optimal solutions was lower. Therefore, for a small $\beta$, the KPs should not be excessively tuned. In contrast, for a large $\beta$, the negative effect caused by the overtuned KPs was not observed.

Next, we verified the training error $E_{\mathrm{t}}\left(\mathit{\theta }\right)$ and generalization error $E_{\mathrm{v}}\left(\mathit{\theta }\right)$ defined by Equations (21) and (22) for analyzing the effect of the exploration weight $\beta$ on the GPBO search performance. Figure 3A,B show the training errors. Figure 3C,D show the generalization errors. We transformed them into log errors, that is, using $log{E}_{\mathrm{t}}\left(\mathit{\theta }\right)$ instead of $E_{\mathrm{t}}\left(\mathit{\theta }\right)$. As the maximum number of GD iterations $T_{\mathrm{g}}$ and number of Bayesian optimization steps $t_{\mathrm{b}}$ increased, the training errors decreased. In contrast, in the latter half of the Bayesian optimization process, a higher number of tuned KPs (i.e., larger $T_{\mathrm{g}}$) corresponded to larger generalization errors. When the generalization error of the surrogate model was large, because the reliability of the first term of the acquisition function defined by Equation (16) was low, the GPBO search performance was poor. From these results, for $\beta =1$, we attribute the degradation of the GPBO search performance to the overtuning of the KPs, which increased the generalization error. For $\beta =5$, even when the KPs were overtuned, there were many regions with small generalization errors. Therefore, in this case, we consider that the GPBO search performance was not degraded.

When observation samples occur in a limited region, the generalization error is large, because the surrogate model based on these data cannot correctly estimate the output values of various regions in the search space. To verify this hypothesis, we used Equation (23) with $\omega =0.5$ and calculated the rate at which the next search point determined by Equation (17) and the observed points were close. The results are presented in Figure 3E,F. Figure 3E suggests that, when the exploration weight was set as $\beta =1$, the GPBO searched for areas close to previously observed samples. This was particularly true when large numbers of KPs were tuned. For example, in the case of $\beta =1$ and $T_{\mathrm{g}}=500$, when the number of Bayesian optimization steps exceeded approximately 35 ($t_{\mathrm{b}}>35$), the search area of GPBO remained close to the previously observed samples. This trend was weaker for a smaller value of $T_{\mathrm{g}}$. Thus, there was a stronger tendency to search close to the previously observed samples when the KPs were overtuned. Therefore, we considered the GPBO search performance to be poor. Figure 3F suggests that, for $\beta =5$, the GPBO searched various areas even if the KPs were overtuned. In summary, from Figure 3E,F, the degree of kernel-parameter tuning affected whether GPBO searched areas close to previously observed samples. Moreover, a smaller exploration weight $\beta$ corresponded to a higher risk. Thus, for a smaller value of $\beta$, there should be more focus on kernel-parameter tuning.

Next, we present the changes in the surrogate model for each Bayesian optimization step in Figure 4 and Figure 5. These values were $\beta =1,5$, respectively. Due to the fact that we cannot show the results for all the seeds, the results for a specific seed are shown. In the case where no KPs were tuned, that is, $T_{\mathrm{g}}=0$, appropriate searches were not performed regardless of $\beta$. For $T_{\mathrm{g}}=0$, Figure 3A–D indicate that the training and generalization errors were large. We believe that, because the surrogate model was inappropriate, the GPBO search was inappropriate.

With a small number of KPs tuned, i.e., $T_{\mathrm{g}}=50$, the average $\mathbb{E}$ generated from the surrogate model nearly succeeded in reproducing the black-box function $f\left(\mathit{x}\right)$ regardless of $\beta$. Therefore, the rate of correctly finding the optimal solution was high (see Figure 2B).

Figure 4 and Figure 5 suggest that, when the number of iterations of KP tuning was $T_{\mathrm{g}}=500$, the search results depended on the exploration weight $\beta$. For $\beta =1$, because the GPBO searched only the neighborhood of the best point from the initial solutions, even if the search progressed, the output from the surrogate model $\mathbb{E}$ could not reproduce the black-box function $f\left(\mathit{x}\right)$. In contrast, for $\beta =5$, because observation samples appeared in various areas, even if the KPs were overtuned, the output of the surrogate model $\mathbb{E}$ reproduced the black-box function $f\left(\mathit{x}\right)$. The results indicate that the negative effect of overtuning the KPs on the GPBO search performance can be mitigated by increasing the exploration weight $\beta$.

However, in both the $\beta =1$ and 5 cases, when the number of iterations of KP tuning increased ($T_{\mathrm{g}}=500$), the surrogate model output was inappropriate when the number of BO steps $t_{\mathrm{b}}$ was approximately 30–35 (the cases of $T_{\mathrm{g}}=500$ are shown in Figure 4 and Figure 5). In general, overfitting parameters to observed data increases the risk of reduced estimation performance in areas where there are no observed data. Therefore, overfitting can easily occur with high $T_{\mathrm{g}}$ values. Due to the fact that this occurs in both the cases of $\beta =1$ and 5, it is important not to overtune the KPs, regardless of the exploration weight. Although the negative effects of overtuned KPs can be mitigated by increasing the exploration weight, they cannot be completely eliminated.

4. Conclusions

The exploration weight affects the degree of exploration of GPBO. Therefore, the GPBO search performance depends on the exploration weight. In this study, we analyzed the mitigation of the negative effect of overtuning KPs as another effect of the exploration weight. The results indicate that we should pay attention to overtuning the KPs in the case of Bayesian optimization with a small exploration weight. It is preferable to use methods for avoiding the overtuning of KPs, e.g., early stopping of the GD. In contrast, for large exploration weights, the solution discovery rate is high even when overtuning the KPs. These findings are useful in all situations wherein GPBO is used, e.g., hyperparameter tuning in machine learning.

The results and discussions presented in this paper are entirely based on the black-box function defined by Equation (25). The function has two optimal solutions clearly visible in the domain. However, it is not known whether the results of this study would be reproduced if flat functions such as the Beale function [33] or the Goldstein–Price function [34] (the minimum search) were adopted. As shown in Figure 6, the structure of these functions is apparent after log transformation. A similar result can be obtained with a flat function, or a logarithmic transformation may have to be performed. These points are unclear and will be discussed in a future work.