# Exploratory Landscape Validation for Bayesian Optimization Algorithms

^{*}

## Abstract

**:**

## 1. Introduction

- Degree of polynomial regression models;
- Size of the hidden layer for models based on neural networks;
- Kernel type and parameters for models based on Gaussian process (GP) regression;
- Number of trees and tree depth for models based on random forest regression.

## 2. Bayesian Optimization with Hyperparameter Tuning and Prediction

#### 2.1. Bayesian Optimization Algorithm

- Generate the initial sample ${L}^{0}=\left\{\left({X}^{i},{f}^{i}\right),i\in \left(1:{N}_{init}\right)\right\}$ for training the first GP model, where ${N}_{init}<{N}_{max}$ is the initial sample size. Points ${X}^{i}\in {D}_{X}$ are chosen either randomly or according to one of the designs of the experiment algorithm, e.g., the Latin hypercube sampling (LHS) algorithm [18];
- Perform iterations $r\in \left(1:{N}_{iter}\right)$, where ${N}_{iter}={N}_{max}-{N}_{init}$ as follows:
- Build the surrogate model ${\widehat{f}}^{r}\left(X\right)$ using the current training sample ${L}^{r}$ and the vector $P$ of the hyperparameter values;
- Select the next point ${X}^{i+1}$ for the objective function evaluation by optimizing an acquisition function, e.g., by minimizing the lower confidence bound (LCB) [19] function as follows:$${X}^{i+1}=\underset{X\in {D}_{X}\subset {\mathbb{R}}^{\left|X\right|}}{\mathrm{arg}\text{}\mathrm{min}}{\mu}_{\widehat{f}}\left(X\right)-\kappa {\sigma}_{\widehat{f}}^{2}\left(X\right),$$
- Evaluate the objective function for point ${X}^{i+1}$ to obtain the corresponding value ${f}^{i+1}$ and extend the training sample ${L}^{r+1}={L}^{r}{\displaystyle \cup}\left({X}^{i+1},{f}^{i+1}\right)$.

- The point that has the minimal corresponding objective value $\left({X}^{*},{f}^{*}\right)\in {L}^{{N}_{iter}}$ is considered as the solution of the base problem (1).

#### 2.2. Hyperparameter Tuning for a Bayesian Optimization Algorithm

#### 2.3. Hyperparameter Prediction for a Bayesian Optimization Algorithm

- Define a representative set $Q=\left\{{q}_{i},i\in \left(1:{M}_{Q}\right)\right\}$ of test optimization problems ${q}_{i}$, where ${M}_{Q}$ is the number of test problems;
- Generate random training samples $\left\{{L}_{i,j},j\in \left[1:{M}_{L}\right]\right\}$ of different sizes $\left|{L}_{i,j}\right|\le {N}_{max}$, where ${M}_{L}$ is the number of samples for each problem ${q}_{i}$;
- For each sample ${L}_{i,j}$, calculate the vector of ELA features ${C}_{i,j}$ and find the vector ${P}_{i,j}^{*}$ that is the best according to some metric $\varphi \left({L}_{i,j},P\right)$ by solving the problem (3);
- Build a tuning model $\widehat{P}\left(C\right)$ using the set of known pairs $\left\{{C}_{i,j},{P}_{i,j}^{*}\right\}$, the total number of which is ${M}_{Q}\times {M}_{L}$.

- Calculate the vector of features ${C}^{r}$ using the current training sample ${L}^{r}$;
- Predict the best hyperparameter values ${\widehat{P}}^{r}=\widehat{P}\left({C}^{r}\right)$ using the tuning model built at step 4 of the exploration stage;
- Build the surrogate model ${\widehat{f}}^{r}\left(X\right)$ using the current training sample ${L}^{r}$ and the vector ${\widehat{P}}^{r}$ of the predicted hyperparameter values.

## 3. Exploratory Landscape Validation

#### 3.1. Variability Map of an Objective Function

- Select a random point ${X}^{{i}_{2}}$ from sample $L$ and find the closest point ${X}^{{i}_{3}}$:$${i}_{3}=\underset{{i}_{3}\in \left[1:\left|L\right|\right],{i}_{3}\ne {i}_{2}}{\mathrm{arg}\text{}\mathrm{min}}{d}_{{i}_{2},{i}_{3}},$$
- Find all the points $\left\{{X}^{k},k\in \left[1:\left|L\right|\right],k\ne {i}_{2},k\ne {i}_{3}\right\}$ that satisfy the conditions:$${d}_{k,{i}_{2}}<{d}_{k,{i}_{3}},\phantom{\rule{0ex}{0ex}}{d}_{k,{i}_{2}}<{\overline{d}}_{{i}_{2}},\phantom{\rule{0ex}{0ex}}{\alpha}_{k,{i}_{2},{i}_{3}}\ge \pi /2,$$
- For each angle range $\left[{\alpha}^{-},{\alpha}^{+}\right]$ from the set of ranges $\left\{\left[90,120\right],\left[120,150\right],\left[150,180\right]\right\}$, that is formed by splitting the range $\left(90,180\right)$ into three equal ranges, do the following:
- find a point ${X}^{{i}_{1}}\in \left\{{X}^{k}\right\}$ that has the minimal distance ${d}_{{i}_{1},{i}_{2}}$ in that angle range:$${X}^{{i}_{1}}=\underset{{X}^{{i}_{1}}\in \left\{{X}^{k}\right\}}{\mathrm{arg}\text{}\mathrm{min}}\text{}{d}_{{i}_{1},{i}_{2}},\phantom{\rule{0ex}{0ex}}{\alpha}^{-}{\alpha}_{{i}_{1},{i}_{2},{i}_{3}}\le {\alpha}^{+};$$
- extend the set of triples $T$ with a new one $t=\left({i}_{1},{i}_{2},{i}_{3}\right)$;
- increase the distances ${d}_{{i}_{1},{i}_{2}}$ and ${d}_{{i}_{2},{i}_{3}}$ by a factor of 2 so that the other points are considered if ${i}_{2}$ is randomly selected in the next iterations.

- If the maximum number of triples $\left|T\right|$ is not reached, move to step 1.

#### 3.2. Extended Variability Map of an Objective Function

- Collect the set of triples $T=\left\{{t}_{j},j\in \left(1:\left|T\right|\right)\right\}$ as it was described in Section 3.1;
- For each triple $t\in T$, where $t=\left({i}_{1},{i}_{2},{i}_{3}\right)$, perform the following steps:
- split the vector ${X}^{{i}_{1}}{X}^{{i}_{2}}$ into three parts by the points ${X}^{{k}_{1}}$ and ${X}^{{k}_{2}}$, so that the points ${X}^{{i}_{1}},{X}^{{k}_{1}},{X}^{{k}_{2}},{X}^{{i}_{2}}$ are arranged in the given order on the same line in $X$ space, where ${k}_{1},{k}_{2}>\left|L\right|$ for the new points ${X}^{{k}_{1}},{X}^{{k}_{2}}$;
- calculate the approximate objective values ${\tilde{f}}^{{k}_{1}},{\tilde{f}}^{{k}_{2}}$ for the new points ${X}^{{k}_{1}},{X}^{{k}_{2}}$, respectively, using a linear interpolation between the known points $\left\{\left({X}^{{i}_{1}},{f}^{{i}_{1}}\right),\left({X}^{{i}_{2}},{f}^{{i}_{2}}\right)\right\}$;
- update the training sample ${L}_{ext}={L}_{ext}{\displaystyle \cup}\left\{\left({X}^{{k}_{1}},{\tilde{f}}^{{k}_{1}}\right),\left({X}^{{k}_{2}},{\tilde{f}}^{{k}_{2}}\right)\right\}$;
- update the extended set of triples ${T}_{ext}={T}_{ext}{\displaystyle \cup}\left\{\left({i}_{1},{k}_{1},{k}_{2}\right),\left({k}_{1},{k}_{2},{i}_{2}\right)\right\}$;
- repeat steps a-d for the vector ${X}^{{i}_{2}}{X}^{{i}_{3}}$.

#### 3.3. Landscape Validation Metrics

- Build a surrogate model $\widehat{f}\left(X\right)$ using the training sample ${L}^{r}$ and the vector $P$ of hyperparameter values;
- Using the model $\widehat{f}\left(X\right)$, calculate ${\widehat{f}}^{i}$ values for all the points ${X}^{i}$of the extended sample ${L}_{ext}^{r}$, where $i\in \left(1:\left|{L}_{ext}\right|\right)$. The corresponding values form the sample ${\widehat{L}}_{ext}^{r}$;
- Given the samples ${L}_{ext}^{r}$ and ${\widehat{L}}_{ext}^{r}$, that have the common ${X}^{i}$ values, calculate the ranking preservation metric:$${\varphi}_{RP}\left({L}_{ext}^{r},P\right)=\frac{1}{\left(\begin{array}{c}\left|{L}_{ext}^{r}\right|\\ 2\end{array}\right)}{\displaystyle \sum}_{i=1}^{\left|{L}_{ext}^{r}\right|}{\displaystyle \sum}_{j=i+1}^{\left|{L}_{ext}^{r}\right|}\{\begin{array}{l}1,ifcomp\left({f}^{i},{f}^{j}\right)=comp\left({\widehat{f}}^{i},{\widehat{f}}^{j}\right)\hfill \\ 0,otherwise\hfill \end{array},$$

- Build a surrogate model $\widehat{f}\left(X\right)$ using the training sample ${L}^{r}$ and the vector $P$ of hyperparameter values;
- Using the model $\widehat{f}\left(X\right)$, calculate ${\widehat{f}}^{i}$ values for all the points ${X}^{i}$of the extended sample ${L}_{ext}^{r}$, where $i\in \left(1:\left|{L}_{ext}^{r}\right|\right)$. The calculated values form the sample ${\widehat{L}}_{ext}^{r}$;
- For all the triples from ${T}_{ext}^{r}$, calculate the increment values $\left\{\left({\widehat{\delta}}_{1}^{j},{\widehat{\delta}}_{2}^{j}\right),j\in \left(1:\left|{T}_{ext}^{r}\right|\right)\right\}$ using the sample ${\widehat{L}}_{ext}^{r}$;
- Assuming each pair of the increment values $\left({\delta}_{1}^{j},{\delta}_{2}^{j}\right)$ correspond to the vector ${\mathsf{\Delta}}^{j}$ in the ${\delta}_{1}{\delta}_{2}$ space, and each pair of the values $\left({\widehat{\delta}}_{1}^{j},{\widehat{\delta}}_{2}^{j}\right)$—to the vector ${\widehat{\mathsf{\Delta}}}^{j}$, calculate the angular divergence metric based on the cosine similarity of the vectors ${\mathsf{\Delta}}^{j}$ and ${\widehat{\mathsf{\Delta}}}^{j}$:$${\varphi}_{AD}\left({L}_{ext}^{r},P\right)=\frac{1}{\left|{T}_{ext}^{r}\right|}{\displaystyle \sum}_{j=1}^{{T}_{ext}^{r}}\frac{{\mathsf{\Delta}}^{j}\xb7{\widehat{\mathsf{\Delta}}}^{j}}{||{\mathsf{\Delta}}^{j}||||{\widehat{\mathsf{\Delta}}}^{j}\left|\right|},$$

## 4. Computational Experiment

#### 4.1. General Setup

- Fixed vector $P$—no hyperparameter tuning;
- Hyperparameter tuning with the considered metrics:
- -
- metric ${\varphi}_{CV}\left(L,P\right)$ with 5 folds (see Equation (4) in Section 2.2);
- -
- metric ${\varphi}_{RP}\left({L}_{ext},P\right)$ (see Equation (8) in Section 3.3);
- -
- metric ${\varphi}_{AD}\left({L}_{ext},P\right)$ (see Equation (9) in Section 3.3).

- Predicted vector $\widehat{P}$ (see Section 2.3);
- The metrics ${\varphi}_{RP}\left({L}_{ext},P\right)$ and ${\varphi}_{AD}\left({L}_{ext},P\right)$ will be referred to as ${\varphi}_{RP}\left(L,P\right)$ and ${\varphi}_{AD}\left(L,P\right),$ respectively, to simplify the experiment description.

#### 4.2. Bayesian Optimization with Hyperparameter Tuning

- For each problem ${q}_{i}\in Q$ with the objective function ${f}_{i}\left(X\right)$, generate the number of random initial samples $\left\{{L}_{i,j}^{0},j\in \left(1:10\right)\right\}$, each sample of size $\left|{L}_{i,j}^{0}\right|=5\left|X\right|$;
- Using each initial sample ${L}_{i,j}^{0}$, perform iterations $r\in \left(1:10\left|X\right|\right)$ of the Bayesian optimization algorithm, as described in Section 2.1;
- Using each initial sample ${L}_{i,j}^{0}$, perform iterations $r\in \left(1:10\left|X\right|\right)$ of the Bayesian optimization algorithm with hyperparameter tuning, as described in Section 2.2, with each of the following metrics: ${\varphi}_{CV}\left({L}_{i,j}^{r},P\right),{\varphi}_{RP}\left({L}_{i,j}^{r},P\right)$ and ${\varphi}_{AD}\left({L}_{i,j}^{r},P\right)$. The best hyperparameter values are selected from the set ${D}_{\nu}$;
- Estimate the average of the best objective values ${\overline{f}}_{i}^{*}$ found in 10 random runs with fixed hyperparameter values in step 2 and with hyperparameter tuning based on the metrics ${\varphi}_{CV}\left({L}_{i,j}^{r},P\right),{\varphi}_{RP}\left({L}_{i,j}^{r},P\right)$ and ${\varphi}_{AD}\left({L}_{i,j}^{r},P\right)$ in step 3:$${\overline{f}}_{i}^{*}=\frac{1}{10}{\displaystyle \sum}_{j}{f}_{i,j}^{*},$$
- For each problem ${q}_{i}$ select the metric ${\varphi}_{i}\left(L,P\right)$ with which the best objective value was found on average.

#### 4.3. Bayesian Optimization with Hyperparameter Prediction

- Remove problems from the set $Q$ that are based on the same BBOB function as the current problem ${q}_{i}$, including those with different dimensions $\left|X\right|$, so that the remaining problems compose the set ${Q}_{i}=\left\{{q}_{k},k\in \left(1:69\right)\right\}$;
- For each problem ${q}_{k}$ generate the random samples $\left\{{L}_{k,s},s\in \left(1:300\right)\right\}$ with different sizes $\left|{L}_{k,s}\right|\in \left(5\left|X\right|,15\left|X\right|\right)$. Within the given range, 20 sample sizes are chosen and 15 random samples of each size are generated in ${D}_{X}$;
- For each sample ${L}_{k,s}$ calculate the vector of ELA features C
_{k,s}, where $\left|{C}_{k,s}\right|=84$ and find the best hyperparameter values ${P}_{k,s}^{*}$ by solving the problem (3) with the set of allowed values ${D}_{\nu}$. As the hyperparameter efficiency metric use the metric ${\varphi}_{k}\left(L,P\right)$, which showed the best performance for the problem ${q}_{k}$ in Section 4.2; - Use the set of pairs $\left\{{C}_{k,s},{P}_{k,s}^{*}\right\}$ as a training sample to build a tuning model ${\widehat{P}}_{i}\left(C\right)$ by using the random forest classifier implemented in the scikit-learn package [24].

- For the current problem ${q}_{i}$, generate the number of random initial samples $\left\{{L}_{i,j}^{0},j\in \left(1:10\right)\right\}$, each sample of size $\left|{L}_{i,j}^{0}\right|=5\left|X\right|$;
- Using each initial sample ${L}_{i,j}^{0}$ perform iterations $r\in \left(1:10\left|X\right|\right)$ of the Bayesian optimization algorithm with hyperparameter prediction by using the tuning model ${\widehat{P}}_{i}$, as described for the exploitation phase in Section 2.3;
- Estimate the average of the best objective values ${\overline{f}}_{i}^{*}$ found in 10 random runs with hyperparameter prediction.

#### 4.4. Experimental Results

## 5. Discussion

## 6. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Alizadeh, R.; Allen, J.K.; Mistree, F. Managing computational complexity using surrogate models: A critical review. Res. Eng. Des.
**2020**, 31, 275–298. [Google Scholar] [CrossRef] - Palar, P.S.; Liem, R.P.; Zuhal, L.R.; Shimoyama, K. On the use of surrogate models in engineering design optimization and exploration: The key issues. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, Prague, Czech Republic, 13–17 July 2019; Association for Computing Machinery: New York, NY, USA; pp. 1592–1602. [Google Scholar]
- Jariego Perez, L.C.; Garrido Merchan, E.C. Towards Automatic Bayesian Optimization: A first step involving acquisition functions. In Proceedings of the 19th Conference of the Spanish Association for Artificial Intelligence, Advances in Artificial Intelligence, Malaga, Spain, 22–24 September 2021; Springer: Cham, Switzerland; pp. 160–169. [Google Scholar]
- Gan, W.; Ji, Z.; Liang, Y. Acquisition functions in Bayesian optimization. In Proceedings of the 2nd International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE), Zhuhai, China, 24–26 September 2021; IEEE: Piscataway Township, NJ, USA; pp. 129–135. [Google Scholar]
- Palar, P.S.; Parussini, L.; Bregant, L.; Shimoyama, K.; Zuhal, L.R. On kernel functions for bi-fidelity Gaussian process regressions. Struct. Struct. Multidiscip. Multidiscip. Optim. Optim.
**2023**, 66, 37. [Google Scholar] [CrossRef] - Yu, H.; Tan, Y.; Sun, C.; Zeng, J. A comparison of quality measures for model selection in surrogate-assisted evolutionary algorithm. Soft Comput. Comput.
**2019**, 23, 12417–12436. [Google Scholar] [CrossRef] - Bischl, B.; Binder, M.; Lang, M.; Pielok, T.; Richter, J.; Coors, S.; Thomas, J.; Ullmann, T.; Becker, M.; Boulesteix, A.L.; et al. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdiscip. Rev. Data Min. Knowl. Discov.
**2023**, 13, e1484. [Google Scholar] [CrossRef] - Williams, C.K.I.; Rasmussen, C.E. Gaussian Processes for Machine Learning; MIT Press: Cambridge, MA, USA, 2006; Volume 4, pp. 83–89. [Google Scholar]
- Bhosekar, A.; Ierapetritou, M. Advances in surrogate based modeling, feasibility analysis, and optimization: A review. Comput. Comput. Chem. Chem. Eng. Eng.
**2018**, 108, 250–267. [Google Scholar] [CrossRef] - Garbo, A.; German, B.J. Performance assessment of a cross-validation sampling strategy with active surrogate model selection. Struct. Multidiscip. Optim.
**2019**, 59, 2257–2272. [Google Scholar] [CrossRef] - Diaz-Manriquez, A.; Toscano, G.; Coello Coello, C.A. Comparison of metamodeling techniques in evolutionary algorithms. Soft Comput.
**2017**, 21, 5647–5663. [Google Scholar] [CrossRef] - Agasiev, T.A. Characteristic feature analysis of continuous optimization problems based on Variability Map of objective function for optimization algorithm configuration. Open Comput. Sci.
**2020**, 10, 97–111. [Google Scholar] [CrossRef] - Škvorc, U.; Eftimov, T.; Korošec, P. Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis. Appl. Soft Comput.
**2020**, 90, 106138. [Google Scholar] [CrossRef] - Renau, Q.; Doerr, C.; Dreo, J.; Doerr, B. Exploratory landscape analysis is strongly sensitive to the sampling strategy. In Proceedings of the 16th International Conference on Parallel Problem Solving from Nature, Leiden, The Netherlands, 5–9 September 2020; Springer: Cham, Switzerland; pp. 139–153. [Google Scholar]
- Kerschke, P.; Preuss, M. Exploratory landscape analysis. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation, Lisbon, Portugal, 15–19 July 2023; Association for Computing Machinery: New York, NY, USA; pp. 990–1007. [Google Scholar]
- Saini, B.S.; López-Ibáñez, M.; Miettinen, K. Automatic surrogate modelling technique selection based on features of optimization problems. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, Prague, Czech Republic, 13–17 July 2019; Association for Computing Machinery: New York, NY, USA; pp. 1765–1772. [Google Scholar]
- Kerschke, P.; Trautmann, H. Automated algorithm selection on continuous black-box problems by combining exploratory landscape analysis and machine learning. Evol. Comput.
**2019**, 27, 99–127. [Google Scholar] [CrossRef] [PubMed] - Viana, F.A.; Venter, G.; Balabanov, V. An algorithm for fast optimal Latin hypercube design of experiments. Int. J. Numer. Methods Eng.
**2010**, 82, 135–156. [Google Scholar] [CrossRef] - Wang, X.; Jin, Y.; Schmitt, S.; Olhofer, M. Recent advances in Bayesian optimization. ACM Comput. Surv.
**2023**, 55, 1–36. [Google Scholar] [CrossRef] - Varelas, K.; El Hara, O.A.; Brockhoff, D.; Hansen, N.; Nguyen, D.M.; Tušar, T.; Auger, A. Benchmarking large-scale continuous optimizers: The bbob-largescale testbed, a COCO software guide and beyond. Appl. Soft Comput.
**2020**, 97, 106737. [Google Scholar] [CrossRef] - de Nobel, J.; Ye, F.; Vermetten, D.; Wang, H.; Doerr, C.; Bäck, T. Iohexperimenter: Benchmarking platform for iterative optimization heuristics. Evol. Comput.
**2023**, 1–6. [Google Scholar] [CrossRef] [PubMed] - Nogueira, F. Bayesian Optimization: Open Source Constrained Global Optimization Tool for Python. 2014. Available online: https://github.com/bayesian-optimization/BayesianOptimization (accessed on 6 December 2023).
- Prager, R.P.; Trautmann, H. Pflacco: Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems in Python. Evol. Comput.
**2023**, 1–25. [Google Scholar] [CrossRef] [PubMed] - Hao, J.; Ho, T.K. Machine learning made easy: A review of scikit-learn package in python programming language. J. Educ. Behav. Stat.
**2019**, 44, 348–361. [Google Scholar] [CrossRef] - Moré, J.J.; Wild, S.M. Benchmarking derivative-free optimization algorithms. SIAM J. Optim.
**2009**, 20, 172–191. [Google Scholar] [CrossRef]

**Figure 4.**Landscape plots and corresponding VMs for (

**a**) Rosenbrock and (

**b**) Rastrigin test optimization functions. VM’s increment values are represented by blue dots.

**Figure 5.**Triples of points collected (

**a**) based on the max–min distance between the points and (

**b**) with the proposed algorithm based on angular ranges. Black dots represent the training sample points in $X$ space ($\left|X\right|=2$), and multicolor lines connect the points of the collected triples.

**Figure 6.**The points and triples of the (

**a**) original and (

**b**) extended samples. Black dots represent points of the original and the extended training samples in $X$ space ($\left|X\right|=2$), and multicolor lines connect the points of the collected triples.

**Figure 7.**VM plots for the (

**a**) original and (

**b**) extended samples. VM’s increment values are represented by blue dots.

**Figure 8.**Example of models with (

**a**) high and (

**b**) low ranking preservation levels, $\left|X\right|=1$.

**Figure 9.**Example of models with (

**a**) high and (

**b**) low ranking preservation levels for the extended training sample, $\left|X\right|=1$

**Figure 10.**Example of angular divergence of a single triple and the corresponding point on a VM for models with (

**a**) high, (

**b**) medium, and (

**c**) low ranking preservation levels, $\left|X\right|=1$.

**Figure 12.**Performance profile plots for different strategies of Bayesian optimization: the number of test problems solved with better values ${\overline{f}}^{*}$.

**Figure 13.**Performance profile plots for different strategies of Bayesian optimization: the number of runs with better values ${f}^{*}$.

**Figure 14.**Performance profile plots for different Bayesian optimization strategies: the number of test problems solved in less time.

**Figure 15.**Performance profile plots for different Bayesian optimization strategies: the number of runs completed in less time.

**Table 1.**The number of best-solved problems with the fixed hyperparameter value and the hyperparameter tuning approach using different metrics.

Bayesian Optimization Strategy | $\mathbf{Number}\text{}\mathbf{of}\text{}\mathbf{Problems}\text{}\mathbf{with}\text{}\mathbf{the}\text{}\mathbf{Best}\text{}{\overline{\mathit{f}}}_{\mathit{i}}^{*}$ |
---|---|

$\mathrm{Fixed}\text{}\mathrm{vector}\text{}P$ | 12 |

Hyperparameter tuning with: | |

$\mathrm{metric}\text{}{\varphi}_{CV}$ | 24 |

$\mathrm{metric}\text{}{\varphi}_{RP}$ | 24 |

$\mathrm{metric}\text{}{\varphi}_{AD}$ | 21 |

**Table 2.**The experimental results with the fixed hyperparameter value, the hyperparameter tuning approach using different metrics and the hyperparameter prediction approach.

Bayesian Optimization Strategy | $\mathbf{Number}\text{}\mathbf{of}\text{}\mathbf{Problems}\text{}\mathbf{with}\text{}\mathbf{the}\text{}\mathbf{Best}\text{}\mathbf{Value}\text{}{\overline{\mathit{f}}}_{\mathit{i}}^{*}$ | $\mathbf{Average}\text{}\mathbf{of}\text{}\mathbf{Normed}\text{}\mathbf{Best}\text{}\mathbf{Values}\text{}{\overline{\mathit{f}}}_{\mathit{i}}^{*}$ | Average Time for Solving a Problem, s |
---|---|---|---|

$\mathrm{Fixed}\text{}\mathrm{vector}\text{}P$ | 9 | 0.379 | 24 |

Hyperparameter tuning with: | |||

$\mathrm{metric}\text{}{\varphi}_{CV}$ | 19 | 0.303 | 457 |

$\mathrm{metric}\text{}{\varphi}_{RP}$ | 13 | 0.305 | 211 |

$\mathrm{metric}\text{}{\varphi}_{AD}$ | 17 | 0.305 | 173 |

$\mathrm{Predicted}\text{}\mathrm{vector}\text{}\widehat{P}$ | 26 | 0.290 | 144 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Agasiev, T.; Karpenko, A.
Exploratory Landscape Validation for Bayesian Optimization Algorithms. *Mathematics* **2024**, *12*, 426.
https://doi.org/10.3390/math12030426

**AMA Style**

Agasiev T, Karpenko A.
Exploratory Landscape Validation for Bayesian Optimization Algorithms. *Mathematics*. 2024; 12(3):426.
https://doi.org/10.3390/math12030426

**Chicago/Turabian Style**

Agasiev, Taleh, and Anatoly Karpenko.
2024. "Exploratory Landscape Validation for Bayesian Optimization Algorithms" *Mathematics* 12, no. 3: 426.
https://doi.org/10.3390/math12030426