# Gaussian Process Surrogates for Modeling Uncertainties in a Use Case of Forging Superalloys

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- Related work shows that mainly epistemic uncertainty is used for prediction or optimization with GP regression.
- In research using aleatoric and epistemic uncertainties, they are not considered separately when solving inverse problems.

- We present a GP based surrogate that models (a) the mean result, (b) the aleatoric and (c) the epistemic uncertainty of a manufacturing process outcome.
- We utilize aleatoric and epistemic uncertainties in solving inverse problems for robust optimization results.

## 2. GP based Surrogate Model

#### 2.1. Gaussian Process

#### 2.2. Aleatoric Uncertainty GP

#### 2.3. Mean Result GP

#### 2.4. Uncertainty Propagation Analysis

## 3. Active Learning and Solving Inverse Problems

#### 3.1. Active Learning

#### 3.2. Inverse Problem

## 4. Case Study on Forging Superalloys

#### 4.1. Forging Aggregate Characteristic

#### 4.2. FEM Simulation

## 5. Results

#### 5.1. GPs

#### 5.2. Active Learning

#### 5.3. Solving Inverse Problem

- (1)
- Combined (This baseline can be considered as an approximation to the use of heteroskedastic GP in UCB BO.): no distinction of uncertainties in UCB based BO, i.e., simply adding aleatoric and epistemic uncertainty with loss:$$\begin{array}{c}\hfill {\mathcal{L}}_{combined}(d({\mu}_{target},m\left(\mu \right)),m\left({\sigma}_{al}\right),{\sigma}_{ep}(\sigma \left({\sigma}_{al}\right),\sigma \left(\mu \right)))=\\ \hfill d({\mu}_{target},m\left(\mu \right))-[\alpha \xb7m\left({\sigma}_{al}\right)+\beta \xb7{\sigma}_{ep}(\sigma \left({\sigma}_{al}\right),\sigma \left(\mu \right)))]\end{array}$$
- (2)
- Epistemic: neglecting aleatoric uncertainty in UCB based BO with loss:$$\begin{array}{c}\hfill {\mathcal{L}}_{epistemic}(d({\mu}_{target},m\left(\mu \right)),m\left({\sigma}_{al}\right),{\sigma}_{ep}(\sigma \left({\sigma}_{al}\right),\sigma \left(\mu \right)))=\\ \hfill d({\mu}_{target},m\left(\mu \right))-\beta \xb7{\sigma}_{ep}(\sigma \left({\sigma}_{al}\right),\sigma \left(\mu \right))).\end{array}$$

## 6. Discussion

## 7. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Yang, X.S.; Koziel, S.; Leifsson, L. Computational optimization, modelling and simulation: Recent trends and challenges. Procedia Comput. Sci.
**2013**, 18, 855–860. [Google Scholar] [CrossRef][Green Version] - Krige, D.G. A statistical approach to some basic mine valuation problems on the Witwatersrand. J. S. Afr. Inst. Min. Metall.
**1951**, 52, 119–139. [Google Scholar] - Makarova, A.; Usmanova, I.; Bogunovic, I.; Krause, A. Risk-averse Heteroscedastic Bayesian Optimization. In Proceedings of the Thirty-Fifth Conference on Neural Information Processing Systems, Virtual, 6 December 2021. [Google Scholar]
- Binois, M.; Gramacy, R.B. hetGP: Heteroskedastic Gaussian process modeling and sequential design in R. J. Stat. Softw.
**2021**, 98, 1–44. [Google Scholar] [CrossRef] - Tran, T.; Stough, J.V.; Zhang, X.; Haggerty, C.M. Bayesian Optimization of 2D Echocardiography Segmentation. In Proceedings of the 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), Nice, France, 13–16 April 2021; pp. 1007–1011. [Google Scholar]
- Roberts, S.; Kusiak, J.; Liu, Y.; Forcellese, A.; Withers, P. Prediction of damage evolution in forged aluminium metal matrix composites using a neural network approach. J. Mater. Process. Technol.
**1998**, 80, 507–512. [Google Scholar] [CrossRef] - Loghin, A.; Ismonov, S. Augmenting generic fatigue crack growth models using 3D finite element simulations and Gaussian process modeling. In Pressure Vessels and Piping Conference; American Society of Mechanical Engineers: New York, NY, USA, 2019; Volume 58936, p. V002T02A004. [Google Scholar]
- Ming, W.; Zhang, G.; Li, H.; Guo, J.; Zhang, Z.; Huang, Y.; Chen, Z. A hybrid process model for EDM based on finite-element method and Gaussian process regression. Int. J. Adv. Manuf. Technol.
**2014**, 74, 1197–1211. [Google Scholar] [CrossRef] - Su, G.; Peng, L.; Hu, L. A Gaussian process-based dynamic surrogate model for complex engineering structural reliability analysis. Struct. Saf.
**2017**, 68, 97–109. [Google Scholar] [CrossRef] - Guo, M.; Hesthaven, J.S. Reduced order modeling for nonlinear structural analysis using Gaussian process regression. Comput. Methods Appl. Mech. Eng.
**2018**, 341, 807–826. [Google Scholar] [CrossRef] - Hu, X.; Li, Y.; Zhao, Z.; Liu, C.; Salonitis, K. Residual stresses field estimation based on deformation force data using Gaussian Process Latent Variable Model. Procedia Manuf.
**2021**, 54, 279–283. [Google Scholar] [CrossRef] - Yue, X.; Wen, Y.; Hunt, J.H.; Shi, J. Active learning for gaussian process considering uncertainties with application to shape control of composite fuselage. IEEE Trans. Autom. Sci. Eng.
**2020**, 18, 36–46. [Google Scholar] [CrossRef] - Ortali, G.; Demo, N.; Rozza, G. Gaussian process approach within a data-driven POD framework for fluid dynamics engineering problems. arXiv
**2020**, arXiv:2012.01989. [Google Scholar] [CrossRef] - Venkatraman, A.; de Oca Zapiain, D.M.; Lim, H.; Kalidindi, S.R. Texture-sensitive prediction of micro-spring performance using Gaussian process models calibrated to finite element simulations. Mater. Des.
**2021**, 197, 109198. [Google Scholar] [CrossRef] - Lee, T.; Bilionis, I.; Tepole, A.B. Propagation of uncertainty in the mechanical and biological response of growing tissues using multi-fidelity Gaussian process regression. Comput. Methods Appl. Mech. Eng.
**2020**, 359, 112724. [Google Scholar] [CrossRef] [PubMed] - Brevault, L.; Balesdent, M.; Hebbal, A. Overview of Gaussian process based multi-fidelity techniques with variable relationship between fidelities, application to aerospace systems. Aerosp. Sci. Technol.
**2020**, 107, 106339. [Google Scholar] [CrossRef] - Civera, M.; Boscato, G.; Fragonara, L.Z. Treed gaussian process for manufacturing imperfection identification of pultruded GFRP thin-walled profile. Compos. Struct.
**2020**, 254, 112882. [Google Scholar] [CrossRef] - Abdelfatah, K.; Bao, J.; Terejanu, G. Environmental Modeling Framework using Stacked Gaussian Processes. arXiv
**2016**, arXiv:1612.02897. [Google Scholar] - Mao, J.; Wang, H.; Li, J. Bayesian finite element model updating of a long-span suspension bridge utilizing hybrid Monte Carlo simulation and kriging predictor. KSCE J. Civ. Eng.
**2020**, 24, 569–579. [Google Scholar] [CrossRef] - Tapia, G.; Khairallah, S.; Matthews, M.; King, W.E.; Elwany, A. Gaussian process-based surrogate modeling framework for process planning in laser powder-bed fusion additive manufacturing of 316L stainless steel. Int. J. Adv. Manuf. Technol.
**2018**, 94, 3591–3603. [Google Scholar] [CrossRef] - Yu, Z.; Shi, X.; Zhou, J.; Huang, R.; Gou, Y. Advanced prediction of roadway broken rock zone based on a novel hybrid soft computing model using Gaussian process and particle swarm optimization. Appl. Sci.
**2020**, 10, 6031. [Google Scholar] [CrossRef] - Lee, S.H. Optimization of cold metal transfer-based wire arc additive manufacturing processes using gaussian process regression. Metals
**2020**, 10, 461. [Google Scholar] [CrossRef][Green Version] - Saul, A.D.; Hensman, J.; Vehtari, A.; Lawrence, N.D. Chained gaussian processes. In Proceedings of the Artificial Intelligence and Statistics, PMLR, Cadiz, Spain, 9–11 May 2016; pp. 1431–1440. [Google Scholar]
- Binois, M.; Gramacy, R.B.; Ludkovski, M. Practical heteroscedastic gaussian process modeling for large simulation experiments. J. Comput. Graph. Stat.
**2018**, 27, 808–821. [Google Scholar] [CrossRef][Green Version] - Dai Nguyen, T.; Gupta, S.; Rana, S.; Venkatesh, S. Stable bayesian optimization. In Pacific-Asia Conference on Knowledge Discovery and Data Mining; Springer: Cham, Switzerland, 2017; pp. 578–591. [Google Scholar]
- Huang, C.; Ren, Y.; McGuinness, E.K.; Losego, M.D.; Lively, R.P.; Joseph, V.R. Bayesian optimization of functional output in inverse problems. Optim. Eng.
**2021**, 22, 2553–2574. [Google Scholar] [CrossRef] - Uhrenholt, A.K.; Jensen, B.S. Efficient Bayesian optimization for target vector estimation. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, PMLR, Naha, Japan, 16–18 April 2019; pp. 2661–2670. [Google Scholar]
- Snelson, E.; Rasmussen, C.E.; Ghahramani, Z. Warped gaussian processes. Adv. Neural Inf. Process. Syst.
**2004**, 16, 337–344. [Google Scholar] - Plock, M.; Burger, S.; Schneider, P.I. Recent advances in Bayesian optimization with applications to parameter reconstruction in optical nano-metrology. Proc. SPIE
**2021**, 11783, 117830J. [Google Scholar] - Hüllermeier, E.; Waegeman, W. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Mach. Learn.
**2021**, 110, 457–506. [Google Scholar] [CrossRef] - Williams, C.K.; Rasmussen, C.E. Gaussian Processes for Machine Learning; MIT Press: Cambridge, MA, USA, 2006; Volume 2. [Google Scholar]
- Burbidge, R.; Rowland, J.J.; King, R.D. Active learning for regression based on query by committee. In Proceedings of the International Conference on Intelligent Data Engineering and Automated Learning, Birmingham, UK, 16–19 December 2007; Springer: Berlin/Heidelberg, Germany, 2007; pp. 209–218. [Google Scholar]
- Wu, D.; Lin, C.T.; Huang, J. Active learning for regression using greedy sampling. Inf. Sci.
**2019**, 474, 90–105. [Google Scholar] [CrossRef][Green Version] - Meka, R.; Alaeddini, A.; Oyama, S.; Langer, K. An active learning methodology for efficient estimation of expensive noisy black-box functions using Gaussian process regression. IEEE Access
**2020**, 8, 111460–111474. [Google Scholar] [CrossRef]

**Figure 1.**Simulation of Manufacturing Processes with Uncertainties: (

**a**) FEM simulation scenario and (

**b**) GP based surrogate model with manufacturing process-specific parameters ${x}_{m}$, part-specific parameters ${x}_{p}$ and distribution Z that describes a manufacturing process-specific characteristic by mean $m\left(\mathrm{Z}\right)$ and aleatoric manufacturing process uncertainty ${\mathsf{\Sigma}}_{al}\left(\mathrm{Z}\right)$ where ${\overline{\mathsf{\Sigma}}}_{al}\left(\mathrm{Z}\right)$ is an aggregated form of ${\mathsf{\Sigma}}_{al}\left(\mathrm{Z}\right)$. Outputs are the mean of the manufacturing process result $m\left(\mu \right)$ and mean of the aleatoric uncertainty $m\left({\sigma}_{al}\right)$, each with corresponding epistemic uncertainties $\sigma \left(\mu \right)$ and $\sigma \left({\sigma}_{al}\right)$ in the GP based surrogate model.

**Figure 2.**GP takes manufacturing process parameters ${x}_{m}$, part specific parameters ${x}_{p}$ and aleatoric manufacturing process uncertainty ${\overline{\mathsf{\Sigma}}}_{al}\left(\mathrm{Z}\right)$ as input and predicts the mean $m\left({\sigma}_{al}\right)$ and epistemic uncertainty $\sigma \left({\sigma}_{al}\right)$ of the aleatoric uncertainty of the manufacturing process result.

**Figure 3.**GP takes manufacturing process-specific parameters ${x}_{m}$ and part specific parameters ${x}_{p}$ as input and predicts the mean $m\left(\mu \right)$ and epistemic uncertainty $\sigma \left(\mu \right)$ of the manufacturing process result.

**Figure 4.**Uncertainty Propagation Analysis: the characteristic ${\mathrm{Z}}^{\left(j\right)}$ of the manufacturing process is described by a distribution, since deviations occur when the process is repeated with identical ${x}_{m}^{\left(j\right)}$. With M draws of ${\mathrm{Z}}^{\left(i\right)\left(j\right)}$ out of the distribution ${\mathrm{Z}}^{\left(j\right)}$ as manufacturing process characteristic and to be manufactured part parameters ${x}_{p}^{\left(j\right)}$, FEM simulations are executed to obtain targets ${y}^{\left(i\right)\left(j\right)}$ that describe a distribution ${Y}^{\left(j\right)}=({\mu}^{\left(j\right)},{\sigma}_{al}^{\left(j\right)})$ with mean ${\mu}^{\left(j\right)}$ and aleatoric standard deviation, i.e., uncertainty ${\sigma}_{al}^{\left(j\right)}$ for given inputs ${x}_{m}^{\left(j\right)}$ and ${x}_{p}^{\left(j\right)}$.

**Figure 5.**GP takes manufacturing process-specific parameters ${x}_{m}$ and manufacturing process time steps t as input and predicts a manufacturing process-specific characteristic (i.e., velocity profile of the forging die) Z with mean $m\left(\mathrm{Z}\right)$ and uncertainty ${\mathsf{\Sigma}}_{al}\left(\mathrm{Z}\right)$.

**Figure 6.**Exemplary forging press characteristics ${\mathrm{Z}}^{\left(j\right)}$ represented by mean and $95\%$ credibility interval of ${v}_{die}$ over t with (

**a**) low deviation, (

**b**,

**c**) moderate deviation and (

**d**) high deviation.

**Figure 7.**Billet configuration with Diameter $d=220$ mm, Height $h=200$ mm and rounded edges with Radius = 10 mm.

**Figure 8.**Preforming an Inconel 625 superalloy billet: (

**a**) initial billet and randomly drawn velocity profile ${Z}^{\left(i\right)\left(j\right)}$, (

**b**) FEM simulation result with graphical presentation of the horizontal displacement $U,U1$ and selected output variables ${y}^{\left(i\right)\left(j\right)}$, i.e., final diameter of 288 mm and final height of 92.83 mm.

**Figure 9.**R2-Scores of 10-fold cross-validation over number of drawn training data N. In each cross-validation step, models are initially trained on two randomly selected datapoints drawn from the pool dataset. Solid lines depict the mean R2-Score values and shaded areas the upper and lower confidence bounds obtained by adding and subtracting the standard deviations, calculated from the obtained results. (

**a**) $m\left({\mu}_{Diameter}\right)$; (

**b**) $m\left({\sigma}_{al,Diameter}\right)$; (

**c**) $m\left({\mu}_{Height}\right)$; (

**d**) $m\left({\sigma}_{al,Height}\right)$.

**Figure 10.**MSEs of 10-fold cross-validation over number of drawn training data N. In each cross-validation step, models are initially trained on two randomly selected datapoints drawn from the pool dataset. Solid lines depict the mean R2-Score values and shaded areas the upper and lower confidence bounds obtained by adding and subtracting the standard deviations, calculated from the 10 obtained results. (

**a**) $m\left({\mu}_{Diameter}\right)$; (

**b**) $m\left({\sigma}_{al,Diameter}\right)$; (

**c**) $m\left({\mu}_{Height}\right)$; (

**d**) $m\left({\sigma}_{al,Height}\right)$.

**Figure 11.**Representative plots of multi-objective optimization results for different hyperparameter settings $\alpha $ and $\beta $ over number of optimization steps N. Solid lines depict squared error values, and dotted lines represent corresponding mean aleatoric uncertainty $m\left({\sigma}_{al}\right)$. The plots for $\alpha =0$ show only blue lines, because the results of the different methods are the same and the lines are on top of each other.

**Figure 12.**Kernel density estimate plots of squared errors for different hyperparameter settings $\alpha $ and $\beta $, distributions are obtained by 10-fold cross-validation, where in each fold a target vector is randomly selected. The plots for $\alpha =0$ show only blue lines, because the results of the different methods are the same and the lines are on top of each other.

**Figure 13.**Kernel density estimate plots of mean aleatoric uncertainties $m\left({\sigma}_{al}\right)$ for different hyperparameter settings $\alpha $ and $\beta $, distributions are obtained by 10-fold cross-validation, where in each fold a target vector is randomly selected. The plots for $\alpha =0$ show only blue lines, because the results of the different methods are the same and the lines are on top of each other.

Training Data for Forging Press GP | |||||||||
---|---|---|---|---|---|---|---|---|---|

${x}_{1}$ | 12 | 12 | 12 | 16 | 16 | 16 | 20 | 20 | 20 |

${x}_{2}$ | 50 | 60 | 70 | 50 | 60 | 70 | 50 | 60 | 70 |

Evaluation Data for Forging Press Characteristics | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

${x}_{1}$ | 10 | 10 | 10 | 10 | 14 | 14 | 14 | 14 | 18 | 18 | 18 | 18 | 22 | 22 | 22 | 22 |

${x}_{2}$ | 45 | 55 | 65 | 75 | 45 | 55 | 65 | 75 | 45 | 55 | 65 | 75 | 45 | 55 | 65 | 75 |

Configuration Data | |||
---|---|---|---|

Diameter d | 220 | 240 | 260 |

Height h | 200 | 210 | 220 |

Temperature $\theta $ | 900 | 1000 | 1100 |

Abaqus FEM Simulation Settings | |
---|---|

Simulation type | Static, General |

Time period | 1 |

Nlgeom | On |

Max number of increments | 1000 |

Initial increment size | 0.001 |

Min increment size | 1 × 10^{−5} |

Max increment size | 1 |

Equation solver method | Direct |

Solution technique | Full Newton |

Individual GP Evaluations | |||||
---|---|---|---|---|---|

Screwpress | Aleatoric Uncertainty | Mean Result | |||

$m\left(\mathrm{Z}\right)$ | $m\left({\sigma}_{al,{d}_{final}}\right)$ | $m\left({\sigma}_{al,{h}_{final}}\right)$ | $m\left({\mu}_{{d}_{final}}\right)$ | $m\left({\mu}_{{h}_{final}}\right)$ | |

R2-Score | 0.9923 | 0.8146 | 0.8455 | 0.9586 | 0.9555 |

**Table 6.**Mean values of Pearson kurtosis and Fisher-Pearson coefficient of skewness calculated from uncertainty propagation analysis results.

Distribution Properties | ||
---|---|---|

${d}_{final}$ | ${h}_{final}$ | |

$kur{t}_{Pearson}$ | 3.003 | 2.685 |

$ske{w}_{Fisher-Pearson}$ | 0.449 | 0.015 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Hoffer, J.G.; Geiger, B.C.; Kern, R. Gaussian Process Surrogates for Modeling Uncertainties in a Use Case of Forging Superalloys. *Appl. Sci.* **2022**, *12*, 1089.
https://doi.org/10.3390/app12031089

**AMA Style**

Hoffer JG, Geiger BC, Kern R. Gaussian Process Surrogates for Modeling Uncertainties in a Use Case of Forging Superalloys. *Applied Sciences*. 2022; 12(3):1089.
https://doi.org/10.3390/app12031089

**Chicago/Turabian Style**

Hoffer, Johannes G., Bernhard C. Geiger, and Roman Kern. 2022. "Gaussian Process Surrogates for Modeling Uncertainties in a Use Case of Forging Superalloys" *Applied Sciences* 12, no. 3: 1089.
https://doi.org/10.3390/app12031089