# A Bayesian Nonparametric Learning Approach to Ensemble Models Using the Proper Bayesian Bootstrap

^{1}

^{2}

^{3}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Bayesian Nonparametric Learning Using the Dirichlet Process

## 3. Bootstrap Techniques in Nonparametric Learning

**X**refers to the sequence of random variables ${X}_{1},\cdots ,{X}_{n}$. Using bootstrap methods, (3) can be approximated by

#### 3.1. Efron’s Bootstrap

**w**for the observations ${X}_{1},\cdots ,{X}_{n}$ from a Multinomial distribution with parameters $(n,\frac{1}{n}{\mathbb{1}}_{n})$, where ${\mathbb{1}}_{n}$ is the identity matrix of dimension $nxn$. In this way we obtain:

#### 3.2. Rubin’s Bootstrap

#### 3.3. Proper Bayesian Bootstrap

Algorithm 1: |

## 4. Our Proposal: Bayesian Nonparametric Learning Applied to Ensemble Tree Modeling

Algorithm 2: |

## 5. Empirical Analysis

#### 5.1. Simulation Study

#### 5.1.1. Empirical Evaluations Varying Prior on the Covariates

#### 5.1.2. Empirical Evaluations Varying Prior on the Relation among **x** and y

#### 5.1.3. Empirical Evaluations Varying k and Sample Size

#### 5.2. A Real Example: The Boston Housing Dataset

## 6. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Breiman, L.; Friedman, J.; Olshen, R.A.; Stone, C.J. Classification and Regression Trees; Chapman & Hall: New York, NY, USA, 1984. [Google Scholar]
- Breiman, L. Bagging predictors. Mach. Learn.
**1996**, 24, 123–140. [Google Scholar] [CrossRef] [Green Version] - Turney, P. Bias and the quantification of stability. Mach. Learn.
**1995**, 20, 23–33. [Google Scholar] [CrossRef] [Green Version] - Efron, B. Bootstrap methods: another look at the jackknife. In Breakthroughs in Statistics; Springer: Berlin/Heidelberg, Germany, 1992; pp. 569–593. [Google Scholar]
- Ho, T.K. Random decision forests. In Proceedings of the 3rd International Conference on Document Analysis and Recognition, Montreal, QC, Canada, 14–16 August 1995; Volume 1, pp. 278–282. [Google Scholar]
- Breiman, L. Random forests. Mach. Learn.
**2001**, 45, 5–32. [Google Scholar] [CrossRef] [Green Version] - Freund, Y.; Schapire, R.E. Experiments with a new boosting algorithm. In Proceedings of the 13th International Conference on Machine Learning, Bari, Italy, 3–6 July 1996; Volume 96, pp. 148–156. [Google Scholar]
- Friedman, J.H. Greedy function approximation: A gradient boosting machine. Ann. Stat.
**2001**, 29, 1189–1232. [Google Scholar] [CrossRef] - Chipman, H.A.; George, E.I.; McCulloch, R.E. Bayesian CART model search. J. Am. Stat. Assoc.
**1998**, 93, 935–948. [Google Scholar] [CrossRef] - Chipman, H.A.; George, E.I.; McCulloch, R.E. BART: Bayesian additive regression trees. Ann. Appl. Stat.
**2010**, 4, 266–298. [Google Scholar] [CrossRef] - Hernández, B.; Raftery, A.E.; Pennington, S.R.; Parnell, A.C. Bayesian additive regression trees using Bayesian model averaging. Stat. Comput.
**2018**, 28, 869–890. [Google Scholar] [CrossRef] [PubMed] - Lyddon, S.; Walker, S.; Holmes, C.C. Nonparametric learning from Bayesian models with randomized objective functions. In Proceedings of the Advances in Neural Information Processing Systems, Montréal, QC, Canada, 3–8 December 2018; pp. 2071–2081. [Google Scholar]
- Taddy, M.; Chen, C.S.; Yu, J.; Wyle, M. Bayesian and Empirical Bayesian Forests. In Proceedings of the International Conference on Machine Learning, Lille, France, 6–11 July 2015; pp. 967–976. [Google Scholar]
- Rubin, D.B. The Bayesian bootstrap. Ann. Stat.
**1981**, 9, 130–134. [Google Scholar] [CrossRef] - Clyde, M.; Lee, H. Bagging and the Bayesian Bootstrap. In Proceedings of the AISTATS, Key West, FL, USA, 3–6 January 2001. [Google Scholar]
- Fushiki, T. Bayesian bootstrap prediction. J. Stat. Plan. Inference
**2010**, 140, 65–74. [Google Scholar] [CrossRef] - Lo, A.Y. A large sample study of the Bayesian bootstrap. Ann. Stat.
**1987**, 15, 360–375. [Google Scholar] [CrossRef] - Weng, C.S. On a second-order asymptotic property of the Bayesian bootstrap mean. Ann. Stat.
**1989**, 17, 705–710. [Google Scholar] [CrossRef] - Muliere, P.; Secchi, P. Bayesian nonparametric predictive inference and bootstrap techniques. Ann. Inst. Stat. Math.
**1996**, 48, 663–673. [Google Scholar] [CrossRef] - Fong, E.; Lyddon, S.; Holmes, C. Scalable Nonparametric Sampling from Multimodal Posteriors with the Posterior Bootstrap. In Proceedings of the International Conference on Machine Learning, Beach, CA, USA, 10–15 June 2019; pp. 1952–1962. [Google Scholar]
- Ferguson, T.S. A Bayesian analysis of some nonparametric problems. Ann. Stat.
**1973**, 1, 209–230. [Google Scholar] [CrossRef] - Efron, B. Second thoughts on the bootstrap. Stat. Sci.
**2003**, 18, 135–140. [Google Scholar] [CrossRef] - Efron, B. Bayesians, frequentists, and scientists. J. Am. Stat. Assoc.
**2005**, 100, 1–5. [Google Scholar] [CrossRef] - Miller, R.G. The jackknife—A review. Biometrika
**1974**, 61, 1–15. [Google Scholar] [CrossRef] - Muliere, P.; Secchi, P. Weak convergence of a Dirichlet-multinomial process. Georgian Math. J.
**2003**, 10, 319–324. [Google Scholar] - Antoniak, C.E. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. Ann. Stat.
**1974**, 2, 1152–1174. [Google Scholar] [CrossRef] - Friedman, J.H. Multivariate adaptive regression splines. Ann. Stat.
**1991**, 19, 1–67. [Google Scholar] [CrossRef] - Dua, D.; Graff, C. UCI Machine Learning Repository. 2017. Available online: https://archive.ics.uci.edu/ml/index.php (accessed on 2 January 2020).

**Figure 1.**Comparison of nonparametric confidence intervals for MSE, squared bias and model variance related to the validation set for different prior choices on the covariates.

**Figure 2.**Comparison of nonparametric confidence intervals for mean squared error (MSE), squared bias and model variance related to the validation set for different prior choices on the relation among dependent and independent variables.

**Figure 3.**Nonparametric confidence intervals for MSE on the validation set varying N, number of observations in the training set.

**Figure 4.**Nonparametric confidence intervals for the squared bias on the validation set varying N, number of observations in the training set.

**Figure 5.**Nonparametric confidence intervals for the variance on the validation set varying N, number of observations in the training set.

Plot Name | Prior Distribution |
---|---|

normal | $\mathcal{N}({\overline{X}}_{j},{S}_{j}^{2})$ |

uniform01 | $\mathcal{U}(0,1)$ |

lognorm01 | $Lognormal(0,0.5)$ |

uniform02 | $\mathcal{U}(0,2)$ |

uniformmm | $\mathcal{U}(min\left({X}_{j}\right),max\left({X}_{j}\right))$ |

Plot Name | Prior Distribution |
---|---|

knn | K-nearest neighbors with $\widehat{k}$ = 5 |

reglin | Multiple linear regression |

polreg | Polynomial regression with degree = 2 |

spline | Spline regression |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Galvani, M.; Bardelli, C.; Figini, S.; Muliere, P.
A Bayesian Nonparametric Learning Approach to Ensemble Models Using the Proper Bayesian Bootstrap. *Algorithms* **2021**, *14*, 11.
https://doi.org/10.3390/a14010011

**AMA Style**

Galvani M, Bardelli C, Figini S, Muliere P.
A Bayesian Nonparametric Learning Approach to Ensemble Models Using the Proper Bayesian Bootstrap. *Algorithms*. 2021; 14(1):11.
https://doi.org/10.3390/a14010011

**Chicago/Turabian Style**

Galvani, Marta, Chiara Bardelli, Silvia Figini, and Pietro Muliere.
2021. "A Bayesian Nonparametric Learning Approach to Ensemble Models Using the Proper Bayesian Bootstrap" *Algorithms* 14, no. 1: 11.
https://doi.org/10.3390/a14010011