# Estimating the Stability of Psychological Dimensions via Bootstrap Exploratory Graph Analysis: A Monte Carlo Simulation and Tutorial

^{1}

^{2}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Exploratory Graph Analysis

## 3. EGA Compared to Factor Analytic Methods

## 4. Bootstrap Approach

#### 4.1. Descriptive Statistics

#### 4.2. Dimension and Item Stability

## 5. Present Research

## 6. Results

#### 6.1. Dimensionality Accuracy

#### 6.2. Item Stability

## 7. Applied Example: Broad Autism Phenotype Questionnaire

`bapq.dimstab$item.stability$item.stability$mean.loadings`; Table 5).

## 8. Discussion

## 9. Materials and Methods

#### 9.1. Data Generation

#### 9.2. Design

#### 9.3. Network Estimation Methods

#### 9.3.1. GLASSO

#### 9.3.2. TMFG

#### 9.3.3. Walktrap Algorithm

#### 9.3.4. Statistical Analyses

#### 9.4. Broad Autism Phenotype Questionnaire

#### 9.5. Data Analyses

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Abbreviations

EGA | Exploratory Graph Analysis |

bootEGA | Bootstrap Exploratory Graph Analysis |

GLASSO | graphical least absolute shrinkage and selection operator |

LASSO | least absolute shrinkage and selection operator |

GGM | Gaussian Graphical Model |

EBIC | extended Bayesian information criterion |

TMFG | triangulated maximally filtered graph |

PA | parallel analysis |

PAF | principal axis factoring |

PCA | principal component analysis |

ARI | Adjusted Rand Index |

PC | percent correct |

NMI | normalized mutual information |

SSC | Simons Simplex Collection |

BAPQ | Broad Autism Phenotype Questionnaire |

## References

- Golino, H.; Epskamp, S. Exploratory Graph Analysis: A new approach for estimating the number of dimensions in psychological research. PLoS ONE
**2017**, 12, e0174035. [Google Scholar] [CrossRef][Green Version] - Golino, H.; Shi, D.; Christensen, A.P.; Garrido, L.E.; Nieto, M.D.; Sadana, R.; Thiyagarajan, J.A.; Martinez-Molina, A. Investigating the performance of Exploratory Graph Analysis and traditional techniques to identify the number of latent factors: A simulation and tutorial. Psychol. Methods
**2020**, 25, 292–320. [Google Scholar] [CrossRef][Green Version] - Fortunato, S. Community Detection in Graphs. Phys. Rep.
**2010**, 486, 75–174. [Google Scholar] [CrossRef][Green Version] - Borsboom, D.; Robinaugh, D.J.; Group, P.; Rhemtulla, M.; Cramer, A.O.J. Robustness and replicability of psychopathology networks. World Psychiatry
**2018**, 17, 143–144. [Google Scholar] [CrossRef] - Forbes, M.K.; Wright, A.G.C.; Markon, K.E.; Krueger, R.F. Evidence that psychopathology symptom networks have limited replicability. J. Abnorm. Psychol.
**2017**, 126, 969–988. [Google Scholar] [CrossRef] [PubMed] - Forbes, M.K.; Wright, A.G.; Markon, K.E.; Krueger, R.F. Quantifying the reliability and replicability of psychopathology network characteristics. Multivar. Behav. Res.
**2019**. [Google Scholar] [CrossRef][Green Version] - Fried, E.I.; Cramer, A.O.J. Moving forward: Challenges and directions for psychopathological network theory and methodology. Perspect. Psychol. Sci.
**2017**, 12, 999–1020. [Google Scholar] [CrossRef] - Friedman, J.; Hastie, T.; Tibshirani, R. Sparse inverse covariance estimation with the graphical lasso. Biostatistics
**2008**, 9, 432–441. [Google Scholar] [CrossRef] [PubMed][Green Version] - Friedman, J.; Hastie, T.; Tibshirani, R. Glasso: Graphical lasso—Estimation of Gaussian Graphical Models; R Package Version 1.8; The R Project for Statistical Computing: Vienna, Austria, 2014. [Google Scholar]
- Lauritzen, S.L. Graphical Models; Clarendon Press: Oxford, UK, 1996. [Google Scholar]
- Tibshirani, R. Regression Shrinkage and Selection via the lasso. J. R. Stat. Soc. Ser. (Methodol.)
**1996**, 58, 267–288. [Google Scholar] [CrossRef] - Chen, J.; Chen, Z. Extended Bayesian information criteria for model selection with large model spaces. Biometrika
**2008**, 95, 759–771. [Google Scholar] [CrossRef][Green Version] - Epskamp, S.; Borsboom, D.; Fried, E.I. Estimating psychological networks and their accuracy: A tutorial paper. Behav. Res. Methods
**2018**, 50, 195–212. [Google Scholar] [CrossRef][Green Version] - Foygel, R.; Drton, M. Extended Bayesian information criteria for Gaussian graphical models. In Proceedings of the Twenty-Fourth Conference on Neural Information Processing Systems, Hyatt Regency, VC, Canada, 6–11 December 2010; pp. 604–612. [Google Scholar]
- Massara, G.P.; Di Matteo, T.; Aste, T. Network filtering for big data: Triangulated Maximally Filtered Graph. J. Complex Netw.
**2016**, 5, 161–178. [Google Scholar] [CrossRef] - Christensen, A.P.; Cotter, K.N.; Silvia, P.J. Reopening openness to experience: A network analysis of four openness to experience inventories. J. Personal. Assess.
**2019**, 101, 574–588. [Google Scholar] [CrossRef] [PubMed][Green Version] - Song, W.M.; Di Matteo, T.; Aste, T. Hierarchical Information Clustering by Means of Topologically Embedded Graphs. PLoS ONE
**2012**, 7, e31929. [Google Scholar] [CrossRef] [PubMed][Green Version] - Christensen, A.P.; Garrido, L.E.; Golino, H. Comparing community detection algorithms in psychological data: A Monte Carlo simulation. PsyArXiv
**2021**. [Google Scholar] [CrossRef] - Gates, K.M.; Henry, T.; Steinley, D.; Fair, D.A. A Monte Carlo evaluation of weighted community detection algorithms. Front. Neuroinform.
**2016**, 10, 45. [Google Scholar] [CrossRef] [PubMed][Green Version] - Pons, P.; Latapy, M. Computing communities in large networks using random walks. J. Graph Algorithms Appl.
**2006**, 10, 191–218. [Google Scholar] [CrossRef][Green Version] - Newman, M.E.J. Modularity and community structure in networks. Proc. Natl. Acad. Sci. USA
**2006**, 103, 8577–8582. [Google Scholar] [CrossRef][Green Version] - Guttman, L. Some necessary conditions for common-factor analysis. Psychometrika
**1954**, 19, 149–161. [Google Scholar] [CrossRef] - Kaiser, H.F. The application of electronic computers to factor analysis. Educ. Psychol. Meas.
**1960**, 20, 141–151. [Google Scholar] [CrossRef] - Velicer, W.F. Determining the number of components from the matrix of partial correlations. Psychometrika
**1976**, 41, 321–327. [Google Scholar] [CrossRef] - Horn, J.L. A rationale and test for the number of factors in factor analysis. Psychometrika
**1965**, 30, 179–185. [Google Scholar] [CrossRef] - Blondel, V.D.; Guillaume, J.L.; Lambiotte, R.; Lefebvre, E. Fast unfolding of communities in large networks. J. Stat. Mech. Theory Exp.
**2008**, 2008, P10008. [Google Scholar] [CrossRef][Green Version] - Clauset, A.; Newman, M.E.J.; Moore, C. Finding community structure in very large networks. Phys. Rev. E
**2004**, 70, 066111. [Google Scholar] [CrossRef] [PubMed][Green Version] - Efron, B.; Tibshirani, R.J. An Introduction to the Bootstrap; CRC Press: Boca Raton, FL, USA, 1994. [Google Scholar]
- Jung, K.; Lee, J.; Gupta, V.; Cho, G. Comparison of bootstrap confidence interval methods for GSCA using a Monte Carlo simulation. Front. Psychol.
**2019**, 10, 2215. [Google Scholar] [CrossRef][Green Version] - Christensen, A.P.; Golino, H.; Silvia, P.J. A psychometric network perspective on the validity and validation of personality trait questionnaires. Eur. J. Personal.
**2020**, 34, 1095–1108. [Google Scholar] [CrossRef] - Hubert, L.; Arabie, P. Comparing partitions. J. Classif.
**1985**, 2, 193–218. [Google Scholar] [CrossRef] - Christensen, A.P.; Garrido, L.E.; Golino, H. Unique variable analysis: A novel approach for detecting redundant variables in multivariate data. PsyArXiv
**2020**. [Google Scholar] [CrossRef] - Montoya, A.K.; Edwards, M.C. The poor fit of model fit for selecting number of factors in exploratory factor analysis for scale evaluation. Educ. Psychol. Meas.
**2020**. [Google Scholar] [CrossRef] - Auerswald, M.; Moshagen, M. How to determine the number of factors to retain in exploratory factor analysis: A comparison of extraction methods under realistic conditions. Psychol. Methods
**2019**, 24, 468–491. [Google Scholar] [CrossRef] - Hurley, R.S.E.; Losh, M.; Parlier, M.; Reznick, J.S.; Piven, J. The broad autism phenotype questionnaire. J. Autism Dev. Disord.
**2007**, 37, 1679–1690. [Google Scholar] [CrossRef] - Christensen, A.P.; Golino, H. On the equivalency of factor and network loadings. Behav. Res. Methods
**2021**, 53, 1563–1580. [Google Scholar] [CrossRef] - Waller, N.G.; DeYoung, C.G.; Bouchard, T.J. The recaptured scale technique: A method for testing the structural robustness of personality scales. Multivar. Behav. Res.
**2016**, 51, 433–445. [Google Scholar] [CrossRef] [PubMed] - Sasson, N.J.; Lam, K.S.; Childress, D.; Parlier, M.; Daniels, J.L.; Piven, J. The Broad Autism Phenotype Questionnaire: Prevalence and diagnostic classification. Autism Res.
**2013**, 6, 134–143. [Google Scholar] [CrossRef] [PubMed] - Garrido, L.E.; Abad, F.J.; Ponsoda, V. Performance of Velicer’s minimum average partial factor retention method with categorical variables. Educ. Psychol. Meas.
**2011**, 71, 551–570. [Google Scholar] [CrossRef] - Garrido, L.E.; Abad, F.J.; Ponsoda, V. A new look at Horn’s parallel analysis with ordinal variables. Psychol. Methods
**2013**, 18, 454–474. [Google Scholar] [CrossRef][Green Version] - Epskamp, S.; Fried, E.I. A tutorial on regularized partial correlation networks. Psychol. Methods
**2018**, 23, 617–634. [Google Scholar] [CrossRef][Green Version] - Epskamp, S.; Cramer, A.O.J.; Waldorp, L.J.; Schmittmann, V.D.; Borsboom, D. qgraph: Network visualizations of relationships in psychometric data. J. Stat. Softw.
**2012**, 48, 1–18. [Google Scholar] [CrossRef][Green Version] - Christensen, A.P.; Kenett, Y.N.; Aste, T.; Silvia, P.J.; Kwapil, T.R. Network structure of the Wisconsin Schizotypy Scales–Short Forms: Examining psychometric network filtering approaches. Behav. Res. Methods
**2018**, 50, 2531–2550. [Google Scholar] [CrossRef][Green Version] - Tumminello, M.; Aste, T.; Di Matteo, T.; Mantegna, R.N. A Tool for Filtering Information in Complex Systems. Proc. Natl. Acad. Sci. USA
**2005**, 102, 10421–10426. [Google Scholar] [CrossRef][Green Version] - Ward, J.H. Hierarchical clustering to optimise an objective function. J. Am. Stat. Assoc.
**1963**, 58, 238–244. [Google Scholar] - Cohen, J. Statistical Power Analysis for the Behavioural Sciences, 2nd ed.; Routledge: New York, NY, USA, 1988. [Google Scholar]
- R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2020. [Google Scholar]
- Golino, H.; Christensen, A.P. EGAnet: Exploratory Graph Analysis—A Framework for Estimating the Number of Dimensions in Multivariate Data Using Network Psychometrics; R Package Version 0.9.9; R Foundation for Statistical Computing: Vienna, Austria, 2020. [Google Scholar]
- Wickham, H. ggplot2: Elegant Graphics for Data Analysis; Springer: New York, NY, USA, 2016. [Google Scholar]
- Kassambara, A. ggpubr: ‘ggplot2’ Based Publication Ready Plots; R Package Version 0.2; R Foundation for Statistical Computing: Vienna, Austria, 2018. [Google Scholar]
- Christensen, A.P. NetworkToolbox: Methods and Measures for Brain, Cognitive, and Psychometric Network Analysis in R. R J.
**2018**, 10, 422–439. [Google Scholar] [CrossRef] - Csardi, G.; Nepusz, T. The igraph Software Package for Complex Network Research. InterJournal Complex Syst.
**2006**, 1695, 1–9. [Google Scholar]

**Figure 4.**Dimensionality results for EGA (

**left**) and bootEGA (

**right**) for the Broad Autism Phenotype Questionnaire.

**Figure 6.**Dimensionality results for EGA (

**left**) and bootEGA (

**right**) for the Broad Autism Phenotype Questionnaire with unstable items removed.

**Figure 7.**Item stability of the Broad Autism Phenotype Questionnaire after removing the unstable items.

n.Boots | median.dim | SE.dim | CI.dim | Lower.CI | Upper.CI | Lower.Quantile | Upper.Quantile |
---|---|---|---|---|---|---|---|

500 | 4 | 0.458 | 0.9 | 3.1 | 4.9 | 3 | 4 |

# of Factors | Frequency |
---|---|

3 | 0.298 |

4 | 0.702 |

Dimension Stability | |
---|---|

1 | 0.574 |

2 | 1.000 |

3 | 0.930 |

4 | 1.000 |

1 | 2 | 3 | 4 | |
---|---|---|---|---|

P07 | 0.702 | 0.298 | ||

P21 | 0.702 | 0.298 | ||

P34 | 0.702 | 0.298 | ||

A12 | 0.574 | 0.426 | ||

A23 | 0.574 | 0.426 | ||

A25 | 0.574 | 0.426 | ||

A28 | 0.574 | 0.426 | ||

R03 | 1 | |||

R06 | 1 | |||

R08 | 1 | |||

R13 | 1 | |||

R15 | 1 | |||

R19 | 1 | |||

R22 | 1 | |||

R24 | 1 | |||

R26 | 1 | |||

R30 | 1 | |||

R33 | 1 | |||

R35 | 1 | |||

P02 | 1 | |||

P04 | 1 | |||

P14 | 1 | |||

P17 | 1 | |||

P20 | 1 | |||

P32 | 1 | |||

P29 | 0.042 | 0.958 | ||

P10 | 0.07 | 0.93 | ||

P11 | 0.07 | 0.93 | ||

A01 | 1 | |||

A05 | 1 | |||

A09 | 1 | |||

A16 | 1 | |||

A18 | 1 | |||

A27 | 1 | |||

A31 | 1 | |||

A36 | 1 |

1 | 2 | 3 | 4 | |
---|---|---|---|---|

P7 | 0.21 | 0.02 | 0.12 | 0.05 |

P21 | 0.21 | 0.01 | 0.06 | 0.01 |

P34 | 0.30 | 0.01 | 0.13 | 0.02 |

A12 | 0.18 | 0.02 | 0.02 | 0.09 |

A23 | 0.15 | 0.00 | 0.06 | 0.15 |

A25 | 0.23 | 0.01 | 0.06 | 0.15 |

A28 | 0.22 | 0.03 | 0.03 | 0.12 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Christensen, A.P.; Golino, H. Estimating the Stability of Psychological Dimensions via Bootstrap Exploratory Graph Analysis: A Monte Carlo Simulation and Tutorial. *Psych* **2021**, *3*, 479-500.
https://doi.org/10.3390/psych3030032

**AMA Style**

Christensen AP, Golino H. Estimating the Stability of Psychological Dimensions via Bootstrap Exploratory Graph Analysis: A Monte Carlo Simulation and Tutorial. *Psych*. 2021; 3(3):479-500.
https://doi.org/10.3390/psych3030032

**Chicago/Turabian Style**

Christensen, Alexander P., and Hudson Golino. 2021. "Estimating the Stability of Psychological Dimensions via Bootstrap Exploratory Graph Analysis: A Monte Carlo Simulation and Tutorial" *Psych* 3, no. 3: 479-500.
https://doi.org/10.3390/psych3030032