# Are Fit Indices Biased in Favor of Bi-Factor Models in Cognitive Ability Research?: A Comparison of Fit in Correlated Factors, Higher-Order, and Bi-Factor Models via Monte Carlo Simulations

^{*}

## Abstract

**:**

## 1. Introduction

**Figure 1.**Multiple correlated factor (top panel), higher-order (middle panel), and bi-factor (bottom panel) models.

## 2. Method

#### 2.1. Data Generation

#### 2.1.1. Study Conditions

**Figure 2.**Population models from which random samples were drawn. Solid lines indicate paths that were estimated for all models. Dashed lines indicate paths that were estimated only in the conditions with all four factors being locally just-identified. (

**a**) Correlated Factors Model; (

**b**) Higher-Order Model; (

**c**) Bi-factor Model.

#### 2.2. Evaluation of Results

#### Model Selection Criteria

## 3. Results

#### 3.1. Convergence

#### 3.2. Models with Two Locally Just- and Two Locally Under-Identified Factors

Indicators Per Factor | Sample Size | True Model | Fitted Model | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

Bi-Factor | Correlated Factors | Higher-Order | ||||||||||||||

CFI | TLI | RMSEA | SRMR | CFI | TLI | RMSEA | SRMR | CFI | TLI | RMSEA | SRMR | |||||

3:1; 2:1 | 200 | Bi | 0.997 | 1.000 | 0.015 | 0.021 | 0.990 | 0.986 | 0.040 | 0.038 | 0.982 | 0.975 | 0.055 | 0.127 | ||

CF | 0.991 | 0.988 | 0.027 | 0.033 | 0.995 | 0.999 | 0.016 | 0.016 | 0.974 | 0.964 | 0.049 | 0.089 | ||||

H-O | 0.997 | 0.991 | 0.016 | 0.029 | 0.994 | 0.994 | 0.023 | 0.035 | 0.991 | 0.988 | 0.031 | 0.063 | ||||

800 | Bi | 0.999 | 1.000 | 0.008 | 0.010 | 0.991 | 0.986 | 0.042 | 0.031 | 0.983 | 0.976 | 0.057 | 0.123 | |||

CF | 0.994 | 0.990 | 0.026 | 0.021 | 0.999 | 1.000 | 0.007 | 0.016 | 0.975 | 0.965 | 0.052 | 0.064 | ||||

H-O | 0.999 | 1.000 | 0.007 | 0.014 | 0.997 | 0.996 | 0.018 | 0.022 | 0.993 | 0.990 | 0.032 | 0.055 | ||||

3:1 | 200 | Bi | 0.997 | 0.999 | 0.014 | 0.022 | 0.994 | 0.993 | 0.025 | 0.028 | 0.985 | 0.980 | 0.045 | 0.011 | ||

CF | 0.990 | 0.988 | 0.025 | 0.036 | 0.995 | 0.999 | 0.014 | 0.034 | 0.975 | 0.968 | 0.042 | 0.090 | ||||

H-O | 0.997 | 0.999 | 0.014 | 0.031 | 0.996 | 0.999 | 0.014 | 0.033 | 0.992 | 0.991 | 0.024 | 0.063 | ||||

800 | Bi | 0.999 | 1.000 | 0.007 | 0.011 | 0.996 | 0.994 | 0.026 | 0.018 | 0.986 | 0.981 | 0.047 | 0.108 | |||

CF | 0.993 | 0.989 | 0.025 | 0.024 | 0.999 | 1.000 | 0.007 | 0.017 | 0.976 | 0.970 | 0.047 | 0.083 | ||||

H-O | 0.999 | 1.000 | 0.007 | 0.015 | 0.999 | 1.000 | 0.006 | 0.016 | 0.994 | 0.992 | 0.026 | 0.052 |

**Table 2.**Percentage of solutions selected by each approximate fit index for each cell of the study design (rounded to nearestwhole number).

Indicators Per Factor | Sample Size | True Model | Fitted Model | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

Bi-Factor | Correlated Factors | Higher-Order | ||||||||||||||

CFI | TLI | RMSEA | SRMR | CFI | TLI | RMSEA | SRMR | CFI | TLI | RMSEA | SRMR | |||||

3:1; 2:1 | 200 | Bi | 94 | 91 | 89 | 99 | 14 | 14 | 13 | 1 | 36 | 37 | 35 | 2 | ||

CF | 42 | 38 | 37 | 56 | 86 | 87 | 86 | 51 | 43 | 42 | 41 | 7 | ||||

H-O | 72 | 67 | 67 | 70 | 70 | 71 | 79 | 37 | 65 | 66 | 65 | 6 | ||||

800 | Bi | 100 | 100 | 100 | 100 | 0 | 0 | 0 | 0 | 4 | 4 | 2 | 0 | |||

CF | 9 | 7 | 6 | 12 | 99 | 99 | 99 | 92 | 8 | 8 | 7 | 3 | ||||

H-O | 80 | 71 | 65 | 76 | 79 | 79 | 71 | 36 | 79 | 77 | 67 | 9 | ||||

3:1 | 200 | Bi | 91 | 86 | 85 | 97 | 33 | 33 | 30 | 6 | 30 | 33 | 31 | 0 | ||

CF | 38 | 35 | 34 | 38 | 89 | 90 | 90 | 71 | 33 | 35 | 33 | 0 | ||||

H-O | 72 | 66 | 64 | 71 | 66 | 65 | 62 | 36 | 68 | 71 | 68 | 0 | ||||

800 | Bi | 100 | 100 | 100 | 100 | 3 | 2 | 1 | 0 | 3 | 2 | 1 | 0 | |||

CF | 4 | 4 | 3 | 2 | 99 | 100 | 99 | 99 | 5 | 4 | 3 | 0 | ||||

H-O | 83 | 73 | 66 | 77 | 83 | 79 | 66 | 36 | 80 | 82 | 72 | 0 |

**Table 3.**Percentage of solutions selected by each approximate fit index when only one model fit best for each cell of the study design (rounded to nearest whole number).

Indicators Per Factor | Sample Size | True Modle | Fitted Model | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

Bi-Factor | Correlated Factors | Higher-Order | ||||||||||||||

CFI | TLI | RMSEA | SRMR | CFI | TLI | RMSEA | SRMR | CFI | TLI | RMSEA | SRMR | |||||

3:1; 2:1 | 200 | Bi | 91 | 87 | 85 | 99 | 2 | 3 | 4 | 1 | 6 | 10 | 11 | 0 | ||

(654) | (665) | (702) | (947) | |||||||||||||

CF | 13 | 11 | 11 | 50 | 79 | 80 | 79 | 50 | 8 | 9 | 9 | 0 | ||||

(501) | (512) | (533) | (737) | |||||||||||||

H-O | 46 | 37 | 37 | 67 | 37 | 40 | 39 | 32 | 18 | 23 | 24 | 1 | ||||

(401) | (415) | (443) | (885) | |||||||||||||

800 | Bi | 100 | 100 | 100 | 100 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||

(958) | (964) | (981) | (1000) | |||||||||||||

CF | 1 | 1 | 1 | 7 | 99 | 99 | 99 | 94 | 0 | 0 | 0 | 0 | ||||

(895) | (907) | (920) | (925) | |||||||||||||

H-O | 45 | 35 | 34 | 70 | 36 | 42 | 39 | 30 | 19 | 23 | 27 | 0 | ||||

(229) | (273) | (429) | (814) | |||||||||||||

3:1 | 200 | Bi | 90 | 85 | 83 | 97 | 7 | 8 | 9 | 3 | 3 | 7 | 8 | 0 | ||

(678) | (686) | (717) | (957) | |||||||||||||

CF | 14 | 10 | 11 | 32 | 84 | 85 | 85 | 68 | 2 | 4 | 4 | 0 | ||||

(531) | (538) | (552) | (751) | |||||||||||||

H-O | 54 | 43 | 40 | 69 | 24 | 24 | 25 | 31 | 22 | 32 | 35 | 0 | ||||

(389) | (412) | (475) | (930) | |||||||||||||

800 | Bi | 100) | 100 | 100 | 100 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||

(968) | (973) | (988) | (1000) | |||||||||||||

CF | 1 | 0 | 1 | 1 | 99 | 100 | 99 | 99 | 0 | 0 | 0 | 0 | ||||

(928) | (931) | (942) | (976) | |||||||||||||

H-O | 62 | 48 | 40 | 73 | 26 | 26 | 25 | 27 | 12 | 26 | 34 | 0 | ||||

(195) | (227) | (409) | (874) |

**Table 4.**Percentage of solutions selected by each information criterion (rounded to nearest whole number).

Indicators Per Factor | Sample Size | True Model | Fitted Model | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|

Bi-Factor | Correlated Factors | Higher-Order | |||||||||||

AIC | BIC | aBIC | AIC | BIC | aBIC | AIC | BIC | aBIC | |||||

3:1; 2:1 | 200 | Bi | 55 | 4 | 51 | 7 | 12 | 7 | 38 | 84 | 41 | ||

CF | 2 | 0 | 2 | 81 | 83 | 81 | 16 | 17 | 16 | ||||

H-O | 9 | 0 | 8 | 48 | 50 | 48 | 43 | 50 | 44 | ||||

800 | Bi | 99 | 45 | 90 | 0 | 0 | 0 | 1 | 55 | 10 | |||

CF | 0 | 0 | 0 | 99 | 99 | 99 | 1 | 1 | 1 | ||||

H-O | 7 | 0 | 1 | 50 | 52 | 52 | 44 | 48 | 48 | ||||

3:1 | 200 | Bi | 39 | 0 | 34 | 9 | 1 | 8 | 51 | 99 | 58 | ||

CF | 1 | 0 | 1 | 74 | 24 | 72 | 25 | 76 | 27 | ||||

H-O | 5 | 0 | 4 | 14 | 0 | 12 | 81 | 100 | 84 | ||||

800 | Bi | 98 | 9 | 77 | 1 | 0 | 1 | 1 | 91 | 22 | |||

CF | 0 | 0 | 0 | 100 | 90 | 99 | 0 | 10 | 1 | ||||

H-O | 4 | 0 | 0 | 14 | 0 | 4 | 82 | 100 | 97 |

#### 3.3. Models with Four Locally Just-Identified Factors

#### 3.4. Summary

## 4. Discussion

## Author Contributions

## Conflicts of Interest

## References

- Holzinger, K.J.; Swineford, F. The bi-factor method. Psychometrika
**1937**, 2, 41–54. [Google Scholar] [CrossRef] - Gorsuch, R.L. Factor Analysis; Saunders: Philadelphia, PA, USA, 1974. [Google Scholar]
- Harman, H.H. Modern Factor Analysis; University of Chicago Press: Chicago, IL, USA, 1960. [Google Scholar]
- Thurstone, L.L. Current issues in factor analysis. Psychol. Bull.
**1940**, 37, 189–236. [Google Scholar] [CrossRef] - Thurstone, L.L. Multiple Factor Analysis; University of Chicago Press: Chicago, IL, USA, 1947. [Google Scholar]
- Carroll, J.B. Human Cognitive Abilities: A Survey of Factor-Analytic Studies; Cambridge University Press: New York, NY, USA, 1993. [Google Scholar]
- Cattell, R.B. The Scientific Use of Factor Analysis in Behavioral and Life Sciences; Plenum: New York, NY, USA, 1978. [Google Scholar]
- Eysenck, H.; Easting, G.; Eysenck, S. Personality measurement in children: A dimensional approach. J. Spec. Educ.
**1970**, 4, 261–268. [Google Scholar] [CrossRef] - Horn, J.L. Organization of abilities and the development of intelligence. Psychol. Rev.
**1968**, 75, 242–259. [Google Scholar] [CrossRef] [PubMed] - Jensen, A.R. The g Factor; Praeger: Westport, CT, USA, 1998. [Google Scholar]
- McCrae, R.R.; Costa, P.T. The structure of interpersonal traits: Wiggins’s circumplex and the five-factor model. J. Person. Soc. Psychol.
**1989**, 56, 586–595. [Google Scholar] [CrossRef] - Reeve, C.L.; Blacksmith, N. Identifying g: A review of current factor analytic practices in the science of mental abilities. Intelligence
**2009**, 37, 487–494. [Google Scholar] [CrossRef] - Bentler, P.M.; Weeks, D.G. Linear structural equations with latent variables. Psychometrika
**1980**, 45, 289–308. [Google Scholar] [CrossRef] - Jöreskog, K.G. A general approach to confirmatory maximum likelihood factor analysis. Psychometrika
**1969**, 34, 183–202. [Google Scholar] [CrossRef] - Decker, S.L.; Englund, J.A.; Roberts, A.M. Higher-order factor structures for the WISC-IV: Implications for neuropsychological test interpretation. Appl. Neuropsychol.: Child
**2014**, 3, 135–144. [Google Scholar] [CrossRef] [PubMed] - Watkins, M.W.; Canivez, G.L.; James, T.; James, K.; Good, R. Construct validity of the WISC–IV
^{UK}with a large referred Irish sample. Int. J. School Educ. Psychol.**2013**, 1, 102–111. [Google Scholar] [CrossRef] - Keith, T.Z.; Witta, E.L. Hierarchical and cross-age confirmatory factor analysis of the WISC-III: What does it measure? School Psychol. Q.
**1997**, 12, 89–107. [Google Scholar] [CrossRef] - Marsh, H.W.; Hocevar, D. Application of confirmatory factor analysis to the study of self-concept: First-and higher order factor models and their invariance across groups. Psychol. Bull.
**1985**, 97, 562–582. [Google Scholar] [CrossRef] - Pillow, D.R.; Pelham, W.E., Jr.; Hoza, B.; Molina, B.S.; Stultz, C.H. Confirmatory factor analyses examining attention deficit hyperactivity disorder symptoms and other childhood disruptive behaviors. J. Abnorm. Child Psychol.
**1998**, 26, 293–309. [Google Scholar] [CrossRef] [PubMed] - Plucker, J.A. Exploratory and confirmatory factor analysis in gifted education: examples with self-concept data. J. Educ. Gifted
**2003**, 27, 20–35. [Google Scholar] - Taub, G.E.; McGrew, K.S.; Witta, E.L. A confirmatory analysis of the factor structure and cross-age invariance of the Wechsler Adult Intelligence Scale. Psychol. Assess.
**2004**, 16, 85–89. [Google Scholar] [CrossRef] [PubMed] - Watkins, M.W.; Beaujean, A.A. Bifactor structure of the Wechsler Preschool and Primary Scale of Intelligence–Fourth Edition. School Psychol. Q.
**2014**, 29, 52–63. [Google Scholar] [CrossRef] [PubMed] - Yang, P.; Cheng, C.P.; Chang, C.L.; Liu, T.L.; Hsu, H.Y.; Yen, C.F. Wechsler Intelligence Scale for Children 4th edition-Chinese version index scores in Taiwanese children with attention-deficit/hyperactivity disorder. Psychiatry Clin. Neurosci.
**2013**, 67, 83–91. [Google Scholar] [CrossRef] [PubMed] - Gustafsson, J.; Balke, G. General and specific abilities as predictors of school achievement. J. Person. Soc. Psychol.
**1993**, 28, 407–434. [Google Scholar] [CrossRef] - Reise, S.P. The rediscovery of bifactor measurement models. Multivar. Behav. Res.
**2012**, 47, 667–696. [Google Scholar] [CrossRef] [PubMed] - Reise, S.P.; Morizot, J.; Hays, R.D. The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Qua. Life Res.
**2007**, 16, 19–31. [Google Scholar] [CrossRef] [PubMed] - Chen, F.F.; West, S.G.; Sousa, K.H. A comparison of bifactor and second-order models of quality of life. Multivar. Behav. Res.
**2006**, 41, 189–225. [Google Scholar] [CrossRef] - Thomas, M.L. Rewards of bridging the divide between measurement and clinical theory: Demonstration of a bifactor model for the Brief Symptom Inventory. Psychol. Assess.
**2012**, 24, 101–113. [Google Scholar] [CrossRef] [PubMed] - Betts, J.; Pickart, M.; Heistad, D. Investigating early literacy and numeracy: Exploring the utility of the bifactor model. School Psychol. Q.
**2011**, 26, 97–107. [Google Scholar] [CrossRef] - Chen, F.F.; Hayes, A.; Carver, C.S.; Laurenceau, J.P.; Zhang, Z. Modeling general and specific variance in multifaceted constructs: A comparison of the bifactor model to other approaches. J. Person.
**2012**, 80, 219–251. [Google Scholar] [CrossRef] [PubMed] - Brouwer, D.; Meijer, R.R.; Zevalkink, J. On the factor structure of the Beck Depression Inventory–II: G is the key. Psychol. Assess.
**2013**, 25, 136–145. [Google Scholar] [CrossRef] [PubMed] - DiStefano, C.; Greer, F.W.; Kamphaus, R. Multifactor modeling of emotional and behavioral risk of preschool-age children. Psychol. Assess.
**2013**, 25, 467–476. [Google Scholar] [CrossRef] [PubMed] - Gignac, G.E. Higher-order models versus direct hierarchical models: g as superordinate or breadth factor? Psychol. Sci.
**2008**, 50, 21–43. [Google Scholar] - Yung, Y.F.; Thissen, D.; McLeod, L.D. On the relationship between the higher-order factor model and the hierarchical factor model. Psychometrika
**1999**, 64, 113–128. [Google Scholar] [CrossRef] - Maydeu-Olivares, A.; Coffman, D.L. Random intercept item factor analysis. Psychol. Methods
**2006**, 11, 344–362. [Google Scholar] [CrossRef] [PubMed] - Murray, A.L.; Johnson, W. The limitations of model fit in comparing the bi-factor versus higher-order models of human cognitive ability structure. Intelligence
**2013**, 41, 407–422. [Google Scholar] [CrossRef] - Swineford, F. Some comparisons of the multiple-factor and the bi-factor methods of analysis. Psychometrika
**1941**, 6, 375–382. [Google Scholar] [CrossRef] - Muthén, L.K.; Muthén, B.O. Mplus User’s Guide, 7th ed.; Muthén & Muthén: Los Angeles, CA, USA, 2013. [Google Scholar]
- Paxton, P.; Curran, P.J.; Bollen, K.A.; Kirby, J.; Chen, F. Monte Carlo experiments: Design and implementation. Struct. Eq. Model.
**2001**, 8, 287–312. [Google Scholar] [CrossRef] - Dombrowski, S.C. Investigating the structure of the WJ-III Cognitive at school age. School Psychol. Q.
**2013**, 28, 154–169. [Google Scholar] [CrossRef] [PubMed] - Reynolds, M.R.; Keith, T.Z.; Flanagan, D.P.; Alfonso, V.C. A cross-battery, reference variable, confirmatory factor analytic investigation of the CHC taxonomy. J. School Psychol.
**2013**, 51, 535–555. [Google Scholar] [CrossRef] [PubMed] - Weiss, L.G.; Keith, T.Z.; Zhu, J.; Chen, H. WISC-IV and clinical validation of the four-and five-factor interpretative approaches. J. Psychoeduc. Assess.
**2013**, 31, 114–131. [Google Scholar] [CrossRef] - Millsap, R.E. Structural equation modeling made difficult. Person. Individ. Differ.
**2007**, 42, 875–881. [Google Scholar] [CrossRef] - Wegener, D.T.; Fabrigar, L.R. Analysis and design for nonexperimental data addressing causal and noncausal hypotheses. In Handbook of Research Methods in Social and Personality Psychology; Cambridge University Press: New York, NY, USA, 2000; pp. 412–450. [Google Scholar]
- Hoyle, R.H. Confirmatory factor analysis. In Handbook of Applied Multivariate Statistics and Mathematical Modeling; Academic Press: New York, NY, USA, 2000; pp. 465–497. [Google Scholar]
- Kline, R.B. Principles and Practice of Structural Equation Modeling, 3rd ed.; Guilford: New York, NY, USA, 2011. [Google Scholar]
- Forero, C.G.; Maydeu-Olivares, A.; Gallardo-Pujol, D. Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Struct. Eq. Model.
**2009**, 16, 625–641. [Google Scholar] [CrossRef] - Carroll, J.B. Theoretical and technical issues in identifying a factor of general intelligence. In Intelligence, Genes, and Success: Scientists Respond to the Bell Curve; Spring-Verlag: New York, NY, USA, 1997. [Google Scholar]
- Heng, K. The nature of scientific proof in the age of simulations. Am. Sci.
**2014**, 102, 174–177. [Google Scholar] [CrossRef] - McDonald, R.P. Structural models and the art of approximation. Persp. Psychol. Sci.
**2010**, 5, 675–686. [Google Scholar] [CrossRef] [PubMed] - Canivez, G.L. Bifactor modeling in construct validation of multifactored tests: Implications for understanding multidimensional constructs and test interpretation. In Principles and Methods of Test Construction: Standards And Recent Advancements; Hogrefe Publishing: Gottingen, Germany, in press.

© 2015 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Morgan, G.B.; Hodge, K.J.; Wells, K.E.; Watkins, M.W. Are Fit Indices Biased in Favor of Bi-Factor Models in Cognitive Ability Research?: A Comparison of Fit in Correlated Factors, Higher-Order, and Bi-Factor Models via Monte Carlo Simulations. *J. Intell.* **2015**, *3*, 2-20.
https://doi.org/10.3390/jintelligence3010002

**AMA Style**

Morgan GB, Hodge KJ, Wells KE, Watkins MW. Are Fit Indices Biased in Favor of Bi-Factor Models in Cognitive Ability Research?: A Comparison of Fit in Correlated Factors, Higher-Order, and Bi-Factor Models via Monte Carlo Simulations. *Journal of Intelligence*. 2015; 3(1):2-20.
https://doi.org/10.3390/jintelligence3010002

**Chicago/Turabian Style**

Morgan, Grant B., Kari J. Hodge, Kevin E. Wells, and Marley W. Watkins. 2015. "Are Fit Indices Biased in Favor of Bi-Factor Models in Cognitive Ability Research?: A Comparison of Fit in Correlated Factors, Higher-Order, and Bi-Factor Models via Monte Carlo Simulations" *Journal of Intelligence* 3, no. 1: 2-20.
https://doi.org/10.3390/jintelligence3010002