Evaluating Cluster-Level Factor Models with lavaan and Mplus

Round 1

Reviewer 1 Report

Review of the Manuscript “Evaluating cluster-level factor models with lavaan and Mplus”

The present article compares software for estimating configural and shared-and-configural two-level factor models. The latter model was proposed by Stapleton, Yang, & Hancock (2016) to separate the “objective” shared factor from nuisance. Simulation studies were conducted to evaluate which software performs best, the R package lavaan or Mplus? The most interesting test case was when strong factorial invariance held across L2 units (i.e., L2 error variances equal to zero) because different software uses different default settings for L2 error variances. Whereas Mplus uses positive constraints (i.e., L2 error variances > 0) when ML estimation is used with Quasi-Newton, lavaan does not use such constraints. This means that with Mplus, negative estimates cannot occur, which can cause convergence issues when strong factorial invariance holds in the data-generating model. Indeed, Mplus showed more convergence issues and inflated test statistics in this case. Therefore, the authors concluded that Mplus holds specific risks that deserve more attention. By contrast, lavaan does not carry such risks.

I find the manuscript to be well-written and perfectly suitable for publication in Psych. However, the manuscript can be polished. In the following, I make some specific recommendations that might be useful to the authors.

• Literature review: Estimation problems with multilevel SEM models are well-known., and many simulation studies have investigated the sources for nonconvergence and inadmissible solutions (e.g., Li & Beretvas, 2013; Lüdtke, Marsh, Robitzsch, & Trautwein, 2011; Zitzmann, Lüdtke, Robitzsch, & Marsh, 2016). The authors should cite this work in their article.
• Alternative methods: The main goal of the article is compare Mplus with lavaan. The authors used frequentist approaches for estimating the two-level factor models. However, it has been argued and repeatedly(!) shown that Bayesian techniques can do much better because they are less prone to estimation problems (Lüdtke, Robitzsch, & Wagner, 2018; Zitzmann et al, 2016). Also, factor score regression can be advantageous (Devlieger, & Rosseel, 2020; Zitzmann & Helm, 2021). I understand that these methods are not the focus of the article. However, because these promising alternatives are also available in lavaan and Mplus, I think that mentioning them would further increase the impact of the article.
• Inadmissible solutions: Inadmissible solutions can occur with lavaan because lavaan allows negative variance estimates. It would thus be very interesting to discuss how such an inadmissible solution is associated with the actual ML estimate. Note that there are different views in statistics on what the ML estimate is when the solution to the estimation problem does not lie within the admissible range. Some statisticians distinguish between inadmissible solutions and ML estimates (e.g., Arnold, 1981; Herbach, 1959; Yuan & Bentler, 2002). For example, Searle, Casella, & McCulloch (1992) showed for a simple two-level model with only one (dependent) variable that the ML estimate of the L2 variance is zero when the solution a negative value. This view is also reflected by the practice of setting negatively estimated variances to zero and then, estimating the model again to obtain the ML estimates of the remaining parameters.

Minor comments

• It is often deemed necessary for multilevel modeling that the intraclass correlations (ICCs) are substantial. Information about the ICCs is lacking. The authors should add this information to the simulation study.

Arnold, S. F. (1981). The theory of linear models and multivariate analysis. New York, NY: Wiley.

Devlieger, I., & Rosseel, Y. (2020). Multilevel factor score regression. Multivariate Behavioral Research, 55, 600–624. doi: 10. 1080/00273171.2019.1661817

Herbach, L. H. (1959). Properties of model i1 type analysis of variance tests, a: optimum nature of the f-test for model i1 in the balanced case. Annals of Mathematical Statistics, 30, 939–959.

Li, X., & Beretvas, S. N. (2013). Sample size limits for estimating upper level mediation models using multilevel SEM. Structural Equation Modeling, 20, 241–264. doi: 10.1080/10705511.2013.769391

Lüdtke, O., Marsh, H. W, Robitzsch, A., & Trautwein, U. (2011). A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy–bias tradeoffs in full and partial error correction models. Psychological Methods, 16, 444–467. doi: 10.1037/a0024376

Lüdtke, O., Robitzsch, A., & Wagner, J. (2018). More stable estimation of the STARTS model: A Bayesian approach using Markov chain Monte Carlo techniques. Psychological Methods, 23, 570–593. doi: 10.1037/met0000155

Searle, S. R., Casella, G. & McCulloch, C. E. (1992). Variance components. New York, NY: Wiley.

Yuan, K.-H. & Bentler, P. M. (2002). On normal theory based inference for multilevel models with distributional violations. Psychometrika, 67, 539–562. doi: 10.1007/BF02295130

Zitzmann, S., & Helm, C. (2021). Multilevel analysis of mediation, moderation, and nonlinear effects in small samples, using expected a posteriori estimates of factor scores. Structural Equation Modeling. Advance online publication. doi: 10.1080/10705511.2020.1855076

Zitzmann, S., Lüdtke, O., Robitzsch, A., & Marsh, H. W. (2016). A Bayesian approach for estimating multilevel latent contextual models. Structural Equation Modeling, 23, 661–679. doi: 10.1080/10705511.2016.1207179

Author Response

Please see attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Please see attached file.

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf