# Bayesian Model Averaging with the Integrated Nested Laplace Approximation

^{1}

^{2}

^{3}

^{*}

^{†}

^{‡}

## Abstract

**:**

`R-INLA`to estimate the posterior marginals in a fraction of the time as typical Markov chain Monte Carlo algorithms. INLA can be extended by means of Bayesian model averaging (BMA) to increase the number of models that it can fit to conditional latent GMRF. In this paper, we review the use of BMA with INLA and propose a new example on spatial econometrics models.

## 1. Introduction

## 2. Spatial Econometrics Models

## 3. The Integrated Nested Laplace Approximation

## 4. Bayesian Model Averaging with INLA

`R-INLA`implements. This was explored in Gómez-Rubio and Palmí-Perales (2019), and the ML estimates of the model parameters could be used to define the CCD points. Note that the dimension of the grid may depend on the model and that it may be difficult to set it beforehand. Grids that are too thin will provide averaged marginals that are too wobbly, and a smaller step may be required.

## 5. Example: Turnout in Italy

`GDPCAP`) in 1997 (in million Lire). The data are comprised of 477 areas, which represent collegi or single-member districts (SDM). Adjacency is defined so that regions whose centroids are 50 km or less away are neighbors to ensure that all regions have neighbors, contiguous regions are neighbors, and Elba is joined to the closest mainland region. In order to assess the impact of the GDP per capita in the estimation of the spatial autocorrelation parameters, two models with and without the covariate are fit. The covariate is included in the log-scale to compare to the results in Ward and Gleditsch (2008).

`spdep`(Bivand et al. 2013), and MCMC estimates are obtained with package

`spatialreg`(Bivand and Piras 2015; Bivand et al. 2013). For MCMC, we used 10,000 burn-in iterations, plus another 90,000 iterations for inference, of which only one in ten was kept to reduce autocorrelation. To speed up convergence, the initial values of $\rho $ and $\lambda $ were set to their maximum likelihood estimates. BMA with INLA estimates was obtained as explained next.

`R-INLA`for the model without the covariate and a grid of $40\times 20$ for the model with

`log(GDPCAP)`. A different grid was used because the model without the covariate required a thinner grid.

`sacsarlm`in the

`spdep`package). Next, $\rho $ and $\lambda $ were fixed at their ML estimates, and the model was fit with INLA. Next, the posterior marginals of the model parameters using BMA with INLA and MCMC are shown. In general, point estimates obtained with the different methods provided very similar values. MCMC and maximum likelihood also provided very similar estimates of the uncertainty of the point estimates (when available for maximum likelihood). BMA with INLA seemed to provide very similar results to MCMC for both models.

`kde2d.weighted`in package

`ggtern`(Hamilton and Ferry 2018). The ML estimate were also added (as a black dot). As with the posterior marginals, the joint distribution was close between MCMC and BMA with INLA. The posterior mode was also close to the ML estimate. The plots showed a negative correlation between the spatial autocorrelation parameters, which may indicate that they struggled to explain spatial correlation in the data (see also Gómez-Rubio and Palmí-Perales 2019). Furthermore, BMA with INLA was a valid approach to make joint posterior inference on a subset of hyperparameters in the model.

`INLABMA`(Bivand et al. 2015) can be used.

## 6. Discussion

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Appendix A. Implementation Details

#### Appendix A.1. Model Fitting

`R-INLA`package using a Gaussian likelihood and a design matrix for the fixed effects ${(\mathit{I}-\rho \mathit{W})}^{-1}\mathit{X}$ and latent random effects with zero mean and precision matrix $\tau \mathsf{\Sigma}$. In particular, the latent random effects can be defined using the effect

`generic0`in

`R-INLA`. Furthermore, the precision of the Gaussian likelihood ${\tau}_{\epsilon}$ is set to $exp\left(15\right)$ to remove the error term. Otherwise, the model would include another error term in the linear predictor in addition to fixed and random effects. Higher values of the precision led to unstable estimates of the model and the marginal likelihood.

`generic0`latent effect is a multivariate Gaussian distribution with zero mean and precision matrix $\tau \mathsf{\Sigma}$. Because $\mathsf{\Sigma}$ is known, its determinant is ignored by

`R-INLA`when computing the log-likelihood. For this reason, we added $+\frac{1}{2}log\left(\right|\mathsf{\Sigma}\left|\right)$ to the computation of the marginal likelihood reported by

`R-INLA`, with $\left|\mathsf{\Sigma}\right|$ the determinant of $\mathsf{\Sigma}$.

`inla.merge`, which took the list of models fit and the vector of weights.

#### Appendix A.2. Grid Definition

#### Appendix A.3. Impacts

#### Appendix A.4. Final Remarks

`mclapply`to fit all the models required by BMA with INLA.

`R`code to run all the models shown in this paper with the different methods is available from GitHub at https://github.com/becarioprecario/SAC_INLABMA.

## References

- Bivand, Roger, and Gianfranco Piras. 2015. Comparing implementations of estimation methods for spatial econometrics. Journal of Statistical Software 63: 1–36. [Google Scholar] [CrossRef] [Green Version]
- Bivand, Roger S., Virgilio Gómez-Rubio, and Håvard Rue. 2014. Approximate Bayesian inference for spatial econometrics models. Spatial Statistics 9: 146–65. [Google Scholar] [CrossRef] [Green Version]
- Bivand, Roger S., Virgilio Gómez-Rubio, and Håvard Rue. 2015. Spatial data analysis with
`R-INLA`with some extensions. Journal of Statistical Software 63: 1–31. [Google Scholar] [CrossRef] [Green Version] - Bivand, Roger S., Edzer Pebesma, and Virgilio Gómez-Rubio. 2013. Applied Spatial Data Analysis with R, 2nd ed. New York: Springer. [Google Scholar]
- Box, Geroge E., and Norman R. Draper. 2007. Response Surfaces, Mixtures, and Ridge Analyses, 2nd ed. Hoboken: John Wiley & Sons, Inc. [Google Scholar]
- Brooks, Steve, Andrew Gelman, Galin L. Jones, and Xiao-Li Meng. 2011. Handbook of Markov Chain Monte Carlo. Boca Raton: Chapman & Hall/CRC Press. [Google Scholar]
- Gilks, Walter R., Sylvia Richardson, and David J. Spiegelhalter. 1996. Markov Chain Monte Carlo in Practice. Boca Raton: Chapman & Hall. [Google Scholar]
- Gómez-Rubio, Virgilio. 2020. Bayesian Inference with INLA. Boca Raton: Chapman & Hall/CRC Press. [Google Scholar]
- Gómez-Rubio, Virgilio, and Håvard Rue. 2018. Markov chain monte carlo with the integrated nested laplace approximation. Statistics and Computing 28: 1033–51. [Google Scholar] [CrossRef] [Green Version]
- Gómez-Rubio, Virgilio, Roger S. Bivand, and Håvard Rue. 2017. Estimating spatial econometrics models with integrated nested Laplace approximation. arXiv arXiv:1703.01273. [Google Scholar]
- Gómez-Rubio, Virgilio, and Francisco Palmí-Perales. 2019. Multivariate posterior inference for spatial models with the integrated nested laplace approximation. Journal of the Royal Statistical Society: Series C (Applied Statistics) 68: 199–215. [Google Scholar] [CrossRef] [Green Version]
- Haining, Robert. 2003. Spatial Data Analysis: Theory and Practice. Cambridge: Cambridge University Press. [Google Scholar]
- Halleck Vega, Solmaria, and J. Paul Elhorst. 2015. The SLX model. Journal of Regional Science 55: 339–63. [Google Scholar] [CrossRef]
- Hamilton, Nicholas E., and Michael Ferry. 2018. ggtern: Ternary diagrams using ggplot2. Journal of Statistical Software, Code Snippets 87: 1–17. [Google Scholar] [CrossRef] [Green Version]
- Hastings, Wilfred K. 1970. Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57: 97–109. [Google Scholar] [CrossRef]
- Hoeting, Jennifer, David Madigan, Adrian Raftery, and Chris Volin Sky. 1999. Bayesian model averaging: A tutorial. Statistical Science 14: 382–401. [Google Scholar]
- Hubin, Aliaksandr, and Geir Storvik. 2016. Estimating the marginal likelihood with Integrated nested Laplace approximation (INLA). arxiv arXiv:1611.01450. [Google Scholar]
- LeSage, James, and Robert Kelley Pace. 2009. Introduction to Spatial Econometrics. London: Chapman and Hall/CRC. [Google Scholar]
- Metropolis, Nicholas, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. 1953. Equations of state calculations by fast computing machine. Journal of Chemical Physics 21: 1087–91. [Google Scholar] [CrossRef] [Green Version]
- Rue, H., and L. Held. 2005. Gaussian Markov Random Fields: Theory and Applications. London: Chapman and Hall/CRC Press. [Google Scholar]
- Rue, Havard, Sara Martino, and Nicolas Chopin. 2009. Approximate Bayesian inference for latent Gaussian models by using integrated nested laplace approximations. Journal of the Royal Statistical Society, Series B 71: 319–92. [Google Scholar] [CrossRef]
- Rue, Håvard, Andrea Riebler, Sigrunn H. Sørbye, Janine B. Illian, Daniel P. Simpson, and Finn K. Lindgren. 2017. Bayesian computing with INLA: A review. Annual Review of Statistics and Its Application 4: 395–421. [Google Scholar] [CrossRef] [Green Version]
- Venables, William N., and Brian D. Ripley. 2002. Modern Applied Statistics with S, 4th ed. New York: Springer, ISBN 0-387-95457-0. [Google Scholar]
- Ward, Michael Don, and Kristian Skrede Gleditsch. 2008. Spatial Regression Models. Thousand Oaks: Sage Publications, Inc. [Google Scholar]

**Figure 1.**Spatial distribution of the turnout in 2011 (left) and GDP per capita in 1997 (right) in Italy at the collegi level.

**Figure 2.**Weights and marginal likelihoods for the model with no covariates (top row) and with covariates (bottom row).

**Figure 3.**Posterior marginals of the spatial autocorrelation parameters for the model with no covariates (top row) and with covariates (bottom row).

**Figure 4.**Joint posterior distribution of the spatial autocorrelation parameters for the model with no covariates (top row) and with covariates (bottom row). The black dot represents the maximum likelihood estimate.

**Figure 5.**Posterior marginal distributions of the coefficients and variances for the model with no covariates (top row) and with covariates (bottom row).

**Figure 6.**Posterior marginals of the average direct (left), indirect (middle) and total (right) impacts of the log-GDP per capita.

**Table 1.**Summary statistics of the model parameters. INLA, integrated nested Laplace approximation; BMA, Bayesian model averaging.

Model | Parameter | Max. lik. | INLA-Max. lik. | BMA | MCMC | ||||
---|---|---|---|---|---|---|---|---|---|

Mean | St.Error | Mean | St. dev. | Mean | St. dev. | Mean | St. Error | ||

No covariates | ${\beta}_{0}$ | 5.86 | 1.54 | 5.88 | 0.10 | 6.39 | 1.59 | 6.56 | 1.95 |

$\rho $ | 0.93 | 0.02 | 0.93 | – | 0.92 | 0.02 | 0.92 | 0.02 | |

$\lambda $ | 0.09 | 0.10 | 0.09 | – | 0.12 | 0.10 | 0.12 | 0.10 | |

${\tau}^{-1}$ | 3.66 | – | 3.68 | 0.24 | 3.71 | 0.25 | 3.75 | 0.28 | |

Covariates | ${\beta}_{0}$ | 5.05 | 2.03 | 5.04 | 1.17 | 5.97 | 2.17 | 5.81 | 2.22 |

${\beta}_{1}$ | 1.60 | 0.51 | 1.60 | 0.34 | 1.81 | 0.58 | 1.77 | 0.59 | |

$\rho $ | 0.87 | 0.04 | 0.87 | – | 0.85 | 0.04 | 0.85 | 0.04 | |

$\lambda $ | 0.18 | 0.12 | 0.18 | – | 0.23 | 0.11 | 0.22 | 0.11 | |

${\tau}^{-1}$ | 3.77 | – | 3.80 | 0.25 | 3.85 | 0.27 | 3.90 | 0.30 |

Impact | Max. lik. | INLA-Max. lik. | BMA | MCMC | ||||
---|---|---|---|---|---|---|---|---|

Mean | St. dev. | Mean | St. dev. | Mean | St. dev. | Mean | St. dev. | |

Direct | 2.25 | – | 2.26 | 0.48 | 2.45 | 0.66 | 2.43 | 0.71 |

Indirect | 9.97 | – | 10.18 | 2.15 | 9.75 | 2.25 | 9.66 | 2.51 |

Total | 12.22 | – | 12.43 | 2.63 | 12.20 | 2.64 | 12.09 | 2.97 |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Gómez-Rubio, V.; Bivand, R.S.; Rue, H.
Bayesian Model Averaging with the Integrated Nested Laplace Approximation. *Econometrics* **2020**, *8*, 23.
https://doi.org/10.3390/econometrics8020023

**AMA Style**

Gómez-Rubio V, Bivand RS, Rue H.
Bayesian Model Averaging with the Integrated Nested Laplace Approximation. *Econometrics*. 2020; 8(2):23.
https://doi.org/10.3390/econometrics8020023

**Chicago/Turabian Style**

Gómez-Rubio, Virgilio, Roger S. Bivand, and Håvard Rue.
2020. "Bayesian Model Averaging with the Integrated Nested Laplace Approximation" *Econometrics* 8, no. 2: 23.
https://doi.org/10.3390/econometrics8020023