# How to Compare Psychometric Factor and Network Models

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

^{−1}Δ, where Ω contains [significant] partial correlations on the off diagonals, Δ is a weight matrix, and I is an identity matrix; see (Epskamp et al. 2017)). We concur with McFarland’s (2020, p. 2) statement that “in order to have a meaningful comparison of models, it is necessary that they be evaluated using identical data (i.e., correlation matrices)”, but thus emphasize that this is standard procedure.

- Consider a Gaussian graphical model (GGM) and one or more factor models as being competing models of intelligence.
- Fit both type of models (in standard SEM software) on the zero-order correlation matrix.
- ○
- Obtain the fit statistics of the models (as reported by this software).
- ○
- Compare these statistics.

- Fit both types of models (in standard SEM software) on the (pruned) partial correlation matrix derived from an exploratory network analysis.
- ○
- Obtain the fit statistics of the models (as reported by this software).
- ○
- Compare these statistics.

- Transform the zero-order correlation to a partial correlation matrix.
- Use this partial correlation matrix as the input of the R glasso function.
- Take the R glasso output matrix as the input matrix in the standard SEM software (SAS CALIS in (McFarland 2020; SAS 2010)).
- Within this software, specify the relations between the variables according to the network and factor models, respectively.
- Obtain the fit statistics of the models as reported by this software.

_{4}—turns out to be actually general (the loadings on this factor are generally (all but one) significant). Factor g

_{3}turns out to be a Speed factor (only the loadings of the Speed tests are substantial and statistically significant). As a result, the theoretical Speed factor that was also included in the model became redundant (none of the loadings on this factor are substantial or significant). Other factors are also redundant, including factor g

_{1}(none of its loadings are substantial or significant). Furthermore, only one of the subtests has a significant loading on g

_{2}, which makes this factor specific rather than general.

## 2. Method

#### 2.1. Samples

#### 2.2. Analysis

^{2}(52) = 118.59, p < 0.001; CFI = 1; TLI = 0.99; RMSEA = 0.026, CI

_{90}= 0.20–0.33), which was to be expected since the network was derived from that matrix. One may compare this to the extraction of an exploratory factor model and refitting this model back as a confirmatory model on the same sample data. Other than providing a check if a network is specified correctly or to obtain fit statistics of this model, we do not advance such a procedure (see also Kan et al. 2019).

## 3. Results

## 4. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Bartholomew, David J., Ian J. Deary, and Martin Lawn. 2009. A new lease of life for Thomson’s bonds model of intelligence. Psychological Review 116: 567. [Google Scholar] [CrossRef] [PubMed]
- Borsboom, Denny, Eiko I. Fried, Sacha Epskamp, Lourens J. Waldorp, Claudia D. van Borkulo, Han LJ van der Maas, and Angélique OJ Cramer. 2017. False alarm? A comprehensive reanalysis of “Evidence that psychopathology symptom networks have limited replicability” by Forbes, Wright, Markon, and Krueger (2017). Journal of Abnormal Psychology 126: 989–99. [Google Scholar] [CrossRef][Green Version]
- Eid, Michael, Stefan Krumm, Tobias Koch, and Julian Schulze. 2018. Bifactor models for predicting criteria by general and specific factors: Problems of nonidentifiability and alternative solutions. Journal of Intelligence 6: 42. [Google Scholar] [CrossRef][Green Version]
- Epskamp, Sacha. 2019. Lvnet: Latent Variable Network Modeling. R Package Version 0.3.5. Available online: https://cran.r-project.org/web/packages/lvnet/index.html (accessed on 28 September 2020).
- Epskamp, S. 2020a. Psychometric network models from time-series and panel data. Psychometrika 85: 1–26. [Google Scholar] [CrossRef] [PubMed][Green Version]
- Epskamp, Sacha. 2020b. Psychonetrics: Structural Equation Modeling and Confirmatory Network Analysis. R Package Version 0.7.1. Available online: https://cran.r-project.org/web/packages/psychonetrics/index.html (accessed on 28 September 2020).
- Epskamp, Sacha, and Eiko I. Fried. 2018. A tutorial on regularized partial correlation networks. Psychological Methods 23: 617. [Google Scholar] [CrossRef] [PubMed][Green Version]
- Epskamp, Sacha, Mijke Rhemtulla, and Denny Borsboom. 2017. Generalized network psychometrics: Combining network and latent variable models. Psychometrika 82: 904–27. [Google Scholar] [CrossRef] [PubMed][Green Version]
- Forbes, Miriam K., Aidan GC Wright, Kristian E. Markon, and Robert F. Krueger. 2017. Evidence that psychopathology symptom networks have limited replicability. Journal of Abnormal Psychology 126: 969–88. [Google Scholar] [CrossRef] [PubMed]
- Friedman, J., T. Hastie, and R. Tibshirani. 2019. Glasso: Graphical Lasso-Estimation of Gaussian Graphical Models. R Package Version 1. Available online: https://cran.r-project.org/web/packages/glasso/index.html (accessed on 28 September 2020).
- Hood, Steven Brian. 2008. Latent Variable Realism in Psychometrics. Bloomington: Indiana University. [Google Scholar]
- Jensen, Arthur Robert. 1998. The g Factor: The Science of Mental Ability. Westport: Praeger. [Google Scholar]
- Jöreskog, Karl G., and Dag Sörbom. 2019. LISREL 10: User’s Reference Guide. Chicago: Scientific Software International. [Google Scholar]
- Kan, Kees-Jan, Han LJ van der Maas, and Rogier A. Kievit. 2016. Process overlap theory: Strengths, limitations, and challenges. Psychological Inquiry 27: 220–28. [Google Scholar] [CrossRef]
- Kan, Kees-Jan, Han LJ van der Maas, and Stephen Z. Levine. 2019. Extending psychometric network analysis: Empirical evidence against g in favor of mutualism? Intelligence 73: 52–62. [Google Scholar] [CrossRef]
- Kline, Rex B. 2015. Principles and Practice of Structural Equation Modeling. New York: Guilford Publications. [Google Scholar]
- Kovacs, Kristof, and Andrew R.A. Conway. 2016. Process overlap theory: A unified account of the general factor of intelligence. Psychological Inquiry 27: 151–77. [Google Scholar] [CrossRef]
- McFarland, Dennis J. 2017. Evaluation of multidimensional models of WAIS-IV subtest performance. The Clinical Neuropsychologist 31: 1127–40. [Google Scholar] [CrossRef] [PubMed]
- McFarland, Dennis J. 2020. The Effects of Using Partial or Uncorrected Correlation Matrices When Comparing Network and Latent Variable Models. Journal of Intelligence 8: 7. [Google Scholar] [CrossRef] [PubMed][Green Version]
- Neale, Michael C., Michael D. Hunter, Joshua N. Pritikin, Mahsa Zahery, Timothy R. Brick, Robert M. Kirkpatrick, Ryne Estabrook, Timothy C. Bates, Hermine H. Maes, and Steven M. Boker. 2016. OpenMx 2.0: Extended structural equation and statistical modeling. Psychometrika 81: 535–49. [Google Scholar] [CrossRef] [PubMed]
- R Core Team. 2020. R: A Language and Environment for Statistical Computing. Vienna: R Core Team. [Google Scholar]
- Rosseel, Yves. 2012. Lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software 48: 1–36. [Google Scholar] [CrossRef][Green Version]
- SAS. 2010. SAS/STAT 9.22 User’s Guide. Cary: SAS Institute. [Google Scholar]
- Schermelleh-Engel, Karin, Helfried Moosbrugger, and Hans Müller. 2003. Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research 8: 23–74. [Google Scholar]
- Schmank, Christopher J., Sara Anne Goring, Kristof Kovacs, and Andrew RA Conway. 2019. Psychometric Network Analysis of the Hungarian WAIS. Journal of Intelligence 7: 21. [Google Scholar] [CrossRef] [PubMed][Green Version]
- van Bork, R., M. Rhemtulla, L. J. Waldorp, J. Kruis, S. Rezvanifar, and D. Borsboom. 2019. Latent variable models and networks: Statistical equivalence and testability. Multivariate Behavioral Research, 1–24. [Google Scholar] [CrossRef]
- Van Der Maas, Han LJ, Conor V. Dolan, Raoul PPP Grasman, Jelte M. Wicherts, Hilde M. Huizenga, and Maartje EJ Raijmakers. 2006. A dynamical model of general intelligence: the positive manifold of intelligence by mutualism. Psychological Review 113: 842–61. [Google Scholar] [CrossRef] [PubMed]
- Wechsler, David. 2008. Wechsler Adult Intelligence Scale, 4th ed. San Antonio: NCS Pearson. [Google Scholar]

1 | This was also the case in the studies of Kan et al. (2019) and Schmank et al. (2019) that McFarland (2020) discussed. Although McFarland (2020, p. 2) stated that “from the R code provided by Schmank et al. (2019), it is clear that the fit of network models to partial correlations were compared to the fit of latent variable models to uncorrected correlation matrices”, the R code of Schmank et al. (2019, see https://osf.io/j3cvz/), which is an adaption of ours (Kan et al. 2019, see https://github.com/KJKan/nwsem) shows that this in incorrect. Correct is that the variance–covariance matrix implied by the network, ∑ = Δ(I − Ω) ^{−1}Δ, was compared to the observed sample variance–covariance matrix, S. |

2 | In comparing g theory (Jensen 1998) with process overlap theory (Kovacs and Conway 2016), McFarland (2017) argued that (1) a superior fit of the penta-factor model over a hierarchical g model provides evidence against the interpretation of the general factor as a single, unitary variable, as in g theory, while (2) the penta-factor model is (conceptually speaking) in line with process overlap theory. We note here that process overlap theory is a variation of sampling theory (Bartholomew et al. 2009), in which the general factor of intelligence is a composite variable and arises, essentially, as a result of a measurement problem (see also Kan et al. 2016): cognitive tests are not unidimensional, but always sample from multiple cognitive domains. Important measurement assumptions are thus indeed being violated. |

**Figure 1.**Graphical representation of the (nonidentified) penta-factor model. Note: abbreviations of subtests: SI—Similarities; VC—Vocabulary; IN—Information; CO—Comprehension; BD—Block Design; MR—Matrix Reasoning; VP—Visual Puzzles; FW—Figure Weights; PC—Picture Completion; DS—Digit Span; AR—Arithmetic; LN—Letter–Number Sequencing; SS—Symbol Search; CD—Coding; CA—Cancelation. Abbreviations of latent variables: V—Verbal factor; P—Perceptual factor; WM—Working Memory factor; S—Speed factor; g

_{1}to g

_{4}—general factors.

**Figure 2.**Graphical representation of the confirmatory WAIS network model. Note: abbreviations of Verbal subtests: SI—Similarities; VC—Vocabulary; IN—Information; CO—Comprehension; Perceptual subtests: BD—Block Design; MR—Matrix Reasoning; VP—Visual Puzzles; PC—Picture Completion; Working Memory subtests: DS—Digit Span; AR—Arithmetic; LN—Letter–Number Sequencing; FW—Figure Weights; and Speed subtests: SS—Symbol Search; CD—Coding; CA—Cancelation.

Factor | Verbal | Perceptual | Working | Processing | Error | ||||||||||||

Comprehension | Organization | Memory | Speed | Variance | |||||||||||||

Subtest (Abbreviation) | Est. | SE | sig. | Est. | SE | sig. | Est. | SE | sig. | Est. | SE | sig. | Est. | SE | sig. | ||

1 | Similarities | (SI) | −0.3413 | 0.0998 | *** | 0.2928 | 0.0198 | *** | |||||||||

2 | Vocabulary | (VC) | −0.4869 | 0.1317 | *** | 0.1847 | 0.0243 | *** | |||||||||

3 | Information | (IN) | −0.7114 | 0.2994 | * | 0.0001 | ^{1} | ||||||||||

4 | Comprehension | (CO) | −0.3553 | 0.1066 | *** | 0.2635 | 0.0203 | *** | |||||||||

5 | Block Design | (BD) | 0.5063 | 0.0578 | *** | 0.3003 | 0.0480 | *** | |||||||||

6 | Matrix Reasoning | (MR) | 0.0842 | 0.0575 | 0.4388 | 0.0312 | *** | ||||||||||

7 | Visual Puzzles | (VP) | 0.4316 | 0.0584 | *** | 0.3493 | 0.0346 | *** | |||||||||

8 | Figure Weights | (FW) | 0.1238 | 0.0507 | * | 0.3858 | 0.0255 | *** | |||||||||

9 | Picture Completion | (PC) | 0.2638 | 0.0532 | *** | 0.6145 | 0.0331 | *** | |||||||||

10 | Digit Span | (DS) | −0.3849 | 0.2596 | 0.3256 | 0.2446 | |||||||||||

11 | Arithmetic | (AR) | 0.1040 | 0.2586 | 0.0001 | ^{1} | |||||||||||

12 | Letter–Number Sequencing | (LN) | −0.5303 | 0.4294 | 0.2344 | 0.4409 | |||||||||||

13 | Symbol Search | (SS) | −0.6189 | 0.8171 | 0.0001 | ^{1} | |||||||||||

14 | Coding | (CD) | −0.2186 | 0.1188 | 0.5036 | 0.1391 | *** | ||||||||||

15 | Cancelation | (CA) | 0.1866 | 0.5088 | 0.4025 | 0.3731 | |||||||||||

Factor | General Factor | General Factor | General Factor | General Factor | |||||||||||||

g_{1} | g_{2} | g_{3} | g_{4} | ||||||||||||||

Subtest (Abbreviation) | Est. | SE | sig. | Est. | SE | sig. | Est. | SE | sig. | Est. | SE | sig. | |||||

1 | Similarities | (SI) | 0.1584 | 1.9584 | 0.0596 | 2.7983 | −0.2318 | 0.5437 | 0.7130 | 0.1692 | *** | ||||||

2 | Vocabulary | (VC) | 0.0951 | 3.0534 | 0.1458 | 1.7943 | −0.2079 | 0.1715 | 0.7104 | 0.1032 | *** | ||||||

3 | Information | (IN) | −0.1982 | 4.0028 | −0.2636 | 2.9837 | −0.0322 | 0.4997 | 0.6197 | 0.1529 | *** | ||||||

4 | Comprehension | (CO) | 0.1793 | 2.2841 | 0.0725 | 3.1929 | −0.2698 | 0.6062 | 0.7072 | 0.1914 | *** | ||||||

5 | Block Design | (BD) | 0.0198 | 1.8986 | −0.1006 | 0.2779 | 0.0472 | 0.3089 | 0.6562 | 0.0574 | *** | ||||||

6 | Matrix Reasoning | (MR) | 0.0652 | 3.0642 | −0.2035 | 1.0725 | −0.0564 | 0.6956 | 0.7108 | 0.1338 | * | ||||||

7 | Visual Puzzles | (VP) | 0.0474 | 2.5845 | −0.1580 | 0.6918 | 0.0039 | 0.5324 | 0.6612 | 0.0999 | *** | ||||||

8 | Figure Weights | (FW) | −0.0264 | 2.4213 | −0.1714 | 0.3925 | −0.1174 | 0.2968 | 0.7450 | 0.0591 | *** | ||||||

9 | Picture Completion | (PC) | 0.1317 | 2.1678 | −0.0885 | 1.5528 | 0.1853 | 0.7582 | 0.5064 | 0.1370 | *** | ||||||

10 | Digit Span | (DS) | −0.0857 | 1.8393 | 0.1220 | 1.3714 | 0.0011 | 0.6273 | 0.7099 | 0.1222 | *** | ||||||

11 | Arithmetic | (AR) | −0.5003 | 3.5156 | 0.2038 | 7.3568 | −0.1221 | 2.6049 | 0.8260 | 0.5251 | |||||||

12 | Letter–Number Sequencing | (LN) | −0.1229 | 1.0698 | 0.0517 | 1.8421 | −0.0398 | 0.6581 | 0.6819 | 0.1310 | *** | ||||||

13 | Symbol Search | (SS) | 0.1119 | 0.9418 | 0.1328 | 0.5802 | 0.5088 | 0.1748 | ** | 0.5726 | 0.0128 | *** | |||||

14 | Coding | (CD) | 0.0763 | 1.2949 | 0.1579 | 0.5028 | 0.2938 | 0.0066 | *** | 0.5758 | 0.0129 | *** | |||||

15 | Cancelation | (CA) | 0.0828 | 1.1700 | 0.1277 | 0.0029 | *** | 0.5838 | 0.0131 | *** | 0.4458 | 0.0100 | *** |

^{1}not computed; * significant at α = 0.05; ** significant at α = 0.01; *** significant at α = 0.001.

Model | χ^{2} (df) | p-Value | CFI | TLI | RMSEA [CI_{90}] | AIC | BIC |
---|---|---|---|---|---|---|---|

Network | 129.96 (52) | <0.001 | 0.99 | 0.99 | 0.037 [0.029–0.045] | 36,848.56 | 37,189.51 |

Bifactor | 263.41 (75) | <0.001 | 0.98 | 0.97 | 0.048 [0.041–0.054] | 36,966.01 | 37,266.85 |

Measurement | 369.47 (82) | <0.001 | 0.96 | 0.97 | 0.056 [0.050–0.062] | 37,058.07 | 37,323.81 |

Hierarchical g | 376.56 (84) | <0.001 | 0.97 | 0.96 | 0.056 [0.050–0.062] | 37,061.16 | 37,316.87 |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kan, K.-J.; de Jonge, H.; van der Maas, H.L.J.; Levine, S.Z.; Epskamp, S.
How to Compare Psychometric Factor and Network Models. *J. Intell.* **2020**, *8*, 35.
https://doi.org/10.3390/jintelligence8040035

**AMA Style**

Kan K-J, de Jonge H, van der Maas HLJ, Levine SZ, Epskamp S.
How to Compare Psychometric Factor and Network Models. *Journal of Intelligence*. 2020; 8(4):35.
https://doi.org/10.3390/jintelligence8040035

**Chicago/Turabian Style**

Kan, Kees-Jan, Hannelies de Jonge, Han L. J. van der Maas, Stephen Z. Levine, and Sacha Epskamp.
2020. "How to Compare Psychometric Factor and Network Models" *Journal of Intelligence* 8, no. 4: 35.
https://doi.org/10.3390/jintelligence8040035