Using the Effective Sample Size as the Stopping Criterion in Markov Chain Monte Carlo with the Bayes Module in Mplus
Abstract
:1. Introduction
2. Zitzmann and Hecht’s Method
- Generate iterative samples for each parameter, which constitute one MCMC chain per parameter.
- After every 100th iteration, discard the first half of each chain as burn-in.
- Compute the PSR for each parameter by using the remaining samples. When the PSR values of all parameters fall below the prespecified maximum PSR value, then stop. Otherwise, continue with the estimation. The maximum PSR value can be set manually, and we will make use of this feature later on.
- Run an MCMC chain for each parameter.
- After every 100th iteration, discard the first half of each chain (burn-in phase).
- Compute the ESS for each parameter from the remaining samples. When all ESS values lie above the prespecified minimum ESS value, stop the estimation. Otherwise, continue with it. It is interesting to note that because of the correspondence between the ESS and the PSR, this algorithm also ensures that all PSR values will fall below a certain “implied” maximum PSR value and, thus, that convergence will generally also be reached.
Linking the PSR and the ESS
3. Simulation Study
3.1. Method
3.2. Results
3.2.1. Comparison between the Prespecified ESS and the Empirical ESS
3.2.2. Statistical Properties
4. Discussion
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
Appendix A. Mplus Code
Title: Analysis model
Data: File is filename .dat;
Variable: Names are y_1 y_2 y_3 x_1 x_2 x_3 g; Usevariables are y_1 y_2 y_3 x_1 x_2 x_3 g; Cluster is g;
Analysis: Type is twolevel; Estimator is bayes; Chains = 1; Biterations = 2000000; Point = mean; Bconvergence = 0.0005007513; ! corresponds with minimum ESS = 1000
Model: % within %
y1 by y_1; y1 by y_2( l1yy2 ) *0.0; ! Param 1 y1 by y_3( l1yy3 ) *0.0; ! Param 2
x1 x1 x1 x1( y_y_y_x_x_x_y1 on x1(b1) *0.0; ! Param 11 y1( vare1 ) *0.5; ! Param 12
% between %
y2 by y_1; y2 by y_2( lyy2 ) *0.0; ! Param 1 y2 by y_3( lyy3 ) *0.0; ! Param 2 x2 by x_1; x2 by x_2( lxx2 ) *0.0; ! Param 3 x2 by x_3( lxx3 ) *0.0; ! Param 4 x2( varx2 ) *0.5; ! Param 29
[y_1 ]( myy1 ) *0.0; ! Param 14 [y_2 ]( myy2 ) *0.0; ! Param 15 [y_3 ]( myy3 ) *0.0; ! Param 16
[x_1 ]( mxx1 ) *0.0; ! Param 17 [x_2 ]( mxx2 ) *0.0; ! Param 18 [x_3 ]( mxx3 ) *0.0; ! Param 19
y_1 ( var2yy1 ) *0.5; ! Param 20 y_2 ( var2yy2 ) *0.5; ! Param 21 y_3 ( var2yy3 ) *0.5; ! Param 22 x_1 ( var2xx1 ) *0.5; ! Param 23 x_2 ( var2xx2 ) *0.5; ! Param 24 x_3 ( var2xx3 ) *0.5; ! Param 25
y2 on x2(b2) *0.0; ! Param 27 [y2 ]( my) *0.0; ! Param 26 y2( vare2 ) *0.5; ! Param 28
Model Priors: l1yy2 ~ N( 0, 1000 ); l1yy3 ~ N( 0, 1000 ); l1xx2 ~ N( 0, 1000 ); l1xx3 ~ N( 0, 1000 ); varx1 ~ IG (0.001 , 0.001);
var1yy1 ~ IG (0.001 , 0.001); var1yy2 ~ IG (0.001 , 0.001); var1yy3 ~ IG (0.001 , 0.001); var1xx1 ~ IG (0.001 , 0.001); var1xx2 ~ IG (0.001 , 0.001); var1xx3 ~ IG (0.001 , 0.001);
b1 ~ N( 0, 1000 ); vare1 ~ IG (0.001 , 0.001);
varx2 ~ IG (0.1 , 0.1);
myy1 ~ N( 0, 1000 ); myy2 ~ N( 0, 1000 ); myy3 ~ N( 0, 1000 ); mxx1 ~ N( 0, 1000 ); mxx2 ~ N( 0, 1000 ); mxx3 ~ N( 0, 1000 );
var2yy1 ~ IG (0.1 , 0.1); var2yy2 ~ IG (0.1 , 0.1); var2yy3 ~ IG (0.1 , 0.1); var2xx1 ~ IG (0.1 , 0.1); var2xx2 ~ IG (0.1 , 0.1); var2xx3 ~ IG (0.1 , 0.1);
b2 ~ N( 0, 1000 ); my ~ N( 0, 1000 ); vare2 ~ IG (0.1 , 0.1);
References
- van de Schoot, R.; Winter, S.D.; Ryan, O.; Zondervan-Zwijnenburg, M.; Depaoli, S. A systematic review of Bayesian articles in psychology: The last 25 years. Psychol. Methods 2017, 22, 217–239. [Google Scholar] [CrossRef]
- König, C.; van de Schoot, R. Bayesian statistics in educational research: A look at the current state of affairs. Educ. Rev. 2018, 70, 486–509. [Google Scholar] [CrossRef]
- Muthén, B.O.; Asparouhov, T. Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychol. Methods 2012, 17, 313–335. [Google Scholar] [CrossRef] [PubMed]
- Depaoli, S.; Clifton, J.P. A Bayesian approach to multilevel structural equation modeling With continuous and dichotomous outcomes. Struct. Equ. Model. 2015, 22, 327–351. [Google Scholar] [CrossRef]
- Zitzmann, S.; Lüdtke, O.; Robitzsch, A.; Hecht, M. On the performance of Bayesian approaches in small samples: A comment on Smid, McNeish, Miočević, and van de Schoot (2020). STructural Equ. Model. 2021, 28, 40–50. [Google Scholar] [CrossRef]
- Gelman, A. Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian Anal. 2006, 1, 515–534. [Google Scholar] [CrossRef]
- Geyer, C.J. Practical Markov chain Monte Carlo. Stat. Sci. 1992, 7, 473–511. [Google Scholar] [CrossRef]
- Muthén, L.K.; Muthén, B.O. Mplus User’s Guide, 7th ed.; Muthén & Muthén: Los Angeles, CA, USA, 2012. [Google Scholar]
- Gelman, A.; Rubin, D.B. Inference from iterative simulation using multiple sequences. Stat. Sci. 1992, 7, 457–472. [Google Scholar] [CrossRef]
- Zitzmann, S.; Hecht, M. Going beyond convergence in Bayesian estimation: Why precision matters too and how to assess it. Struct. Equ. Model. 2019, 26, 646–661. [Google Scholar] [CrossRef]
- Link, W.A.; Eaton, M.J. On thinning of chains in MCMC. Methods Ecol. Evol. 2012, 3, 112–115. [Google Scholar] [CrossRef]
- Plummer, M.; Best, N.; Cowles, K.; Vines, K.; Sarkar, D.; Bates, D.; Almond, R.; Magnusson, A. Package ‘Coda’ [Computer Software Manual]. 2016. Available online: https://cran.r-project.org/web/packages/coda/coda.pdf (accessed on 20 July 2021).
- Lüdtke, O.; Marsh, H.W.; Robitzsch, A.; Trautwein, U. A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy-bias trade-offs in full and partial error correction models. Psychol. Methods 2011, 16, 444–467. [Google Scholar] [CrossRef]
- Snijders, T.A.B.; Bosker, R.J. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling, 2nd ed.; Sage: Los Angeles, CA, USA, 2012. [Google Scholar]
- Gulliford, M.C.; Ukoumunne, O.C.; Chinn, S. Components of variance and intraclass correlations for the design of community-based surveys and intervention studies: Data from the Health Survey for England 1994. Am. J. Epidemiol. 1999, 149, 876–883. [Google Scholar] [CrossRef]
- Hox, J.J.; Maas, C.J.M.; Brinkhuis, M.J.S. The effect of estimation method and sample size in multilevel structural equation modeling. Stat. Neerl. 2010, 64, 157–170. [Google Scholar] [CrossRef]
- Stapleton, L.M.; Yang, J.S.; Hancock, G.R. Construct meaning in multilevel settings. J. Educ. Behav. Stat. 2016, 41, 481–520. [Google Scholar] [CrossRef]
- Zitzmann, S.; Lüdtke, O.; Robitzsch, A.; Marsh, H.W. A Bayesian approach for estimating multilevel latent contextual models. Struct. Equ. Model. 2016, 23, 661–679. [Google Scholar] [CrossRef]
- Smid, S.C.; Winter, S.D. Dangers of the defaults: A tutorial on the impact of default priors when using Bayesian SEM with small samples. Front. Psychol. 2020, 11, 3536. [Google Scholar] [CrossRef]
- Jackman, S. Bayesian Analysis for the Social Sciences; Wiley: Chichester, UK, 2009. [Google Scholar]
- Lüdtke, O.; Marsh, H.W.; Robitzsch, A.; Trautwein, U.; Asparouhov, T.; Muthén, B.O. The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychol. Methods 2008, 13, 203–229. [Google Scholar] [CrossRef] [Green Version]
- Muthén, L.K.; Muthén, B.O. How to use a Monte Carlo study to decide on sample size and determine power. Struct. Equ. Model. 2002, 9, 599–620. [Google Scholar] [CrossRef]
- Zitzmann, S.; Helm, C. Multilevel analysis of mediation, moderation, and nonlinear effects in small samples, using expected a posteriori estimates of factor scores. Struct. Equ. Model. 2021, 28, 529–546. [Google Scholar] [CrossRef]
- Vehtari, A.; Gelman, A.; Simpson, D.; Carpenter, B.; Bürkner, P.C. Rank-normalization, folding, and localization: An improved for assessing convergence of MCMC. Bayesian Anal. 2021. [Google Scholar] [CrossRef]
- Hecht, M.; Weirich, S.; Zitzmann, S. Comparing the Effective Sample Size Performance of JAGS and Stan. 2021; submitted. [Google Scholar]
- Hecht, M.; Gische, C.; Vogel, D.; Zitzmann, S. Integrating out nuisance parameters for computationally more efficient Bayesian estimation—An illustration and tutorial. Struct. Equ. Model. 2020, 27, 483–493. [Google Scholar] [CrossRef]
- Hecht, M.; Zitzmann, S. A computationally more efficient Bayesian approach for estimating continuous-time models. Struct. Equ. Model. 2020, 27, 829–840. [Google Scholar] [CrossRef]
- Gabry, J. Shinystan: Interactive Visual and Numerical Diagnostics and Posterior Analysis for Bayesian Models [Computer Software Manual]. 2016. Available online: http://cran.r-project.org/package=shinystan (accessed on 20 July 2021).
- Jak, S.; Jorgensen, T.D.; Rosseel, Y. Evaluating cluster-level factor models with lavaan and Mplus. Psych 2021, 3, 134–152. [Google Scholar] [CrossRef]
- Zitzmann, S.; Lüdtke, O.; Robitzsch, A. A Bayesian approach to more stable estimates of group-level effects in contextual studies. Multivar. Behav. Res. 2015, 50, 688–705. [Google Scholar] [CrossRef]
- Chung, Y.; Rabe-Hesketh, S.; Dorie, V.; Gelman, A.; Liu, J. A nondegenerate penalized likelihood estimator for variance parameters in multilevel models. Psychometrika 2013, 78, 685–709. [Google Scholar] [CrossRef] [Green Version]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zitzmann, S.; Weirich, S.; Hecht, M. Using the Effective Sample Size as the Stopping Criterion in Markov Chain Monte Carlo with the Bayes Module in Mplus. Psych 2021, 3, 336-347. https://doi.org/10.3390/psych3030025
Zitzmann S, Weirich S, Hecht M. Using the Effective Sample Size as the Stopping Criterion in Markov Chain Monte Carlo with the Bayes Module in Mplus. Psych. 2021; 3(3):336-347. https://doi.org/10.3390/psych3030025
Chicago/Turabian StyleZitzmann, Steffen, Sebastian Weirich, and Martin Hecht. 2021. "Using the Effective Sample Size as the Stopping Criterion in Markov Chain Monte Carlo with the Bayes Module in Mplus" Psych 3, no. 3: 336-347. https://doi.org/10.3390/psych3030025
APA StyleZitzmann, S., Weirich, S., & Hecht, M. (2021). Using the Effective Sample Size as the Stopping Criterion in Markov Chain Monte Carlo with the Bayes Module in Mplus. Psych, 3(3), 336-347. https://doi.org/10.3390/psych3030025