On the Bias in Confirmatory Factor Analysis When Treating Discrete Variables as Ordinal Instead of Continuous

Abstract

Confirmatory factor analysis is some of the most widely used statistical techniques in the social sciences. Frequently, variables (i.e., items) stemming from questionnaires are analyzed. Two competing approaches for estimating confirmatory factor analysis can be distinguished. First, ordinal variables could be treated as in the case of continuous variables using Pearson correlations, and maximum likelihood estimation method would be applied. Second, an ordinal factor analysis based on polychoric correlations can be fitted. In the majority of the psychometric literature, there is a preference for the ordinal factor analysis based on polychoric correlations because the continuous treatment of variables results in biased factor loadings and biased factor correlations. This article argues that it is not legitimate to speak about bias when comparing the two competing factor analytic approaches because it depends on how true model parameters are defined. This decision can be made individually by a researcher. It is shown in simulation studies and analytical derivations that treating variables ordinally using polychoric correlations instead of continuous using Pearson correlations can also lead to biased estimates of factor loadings and factor correlations. Consequently, it should only be stated that different model parameters are defined in a continuous and an ordinal treatment, and one approach should not generally be preferred over the other.

1. Introduction

Confirmatory factor analysis is frequently applied in social science research. Frequently, variables (i.e., items) stemming from questionnaires are analyzed. The variables are often discrete and have a finite number of ordered categories. Applied researchers are often searching for an adequate approach for factor analysis for ordinal variables. Two competing approaches can be distinguished. First, ordinal variables could be treated as in the case of continuous variables using Pearson correlations, and the same maximum likelihood estimation method would be applied. Second, an ordinal factor analysis based on polychoric correlations can be fitted that is claimed to take the ordinal nature of the variables into account. In the majority of the psychometric literature, there is a preference for the ordinal factor analysis based on polychoric correlations [1,2,3,4,5,6,7]. The main message of these articles is that ordinal variables should be treated as ordinal (i.e., not being treated as continuous variables) if there are only a few categories per item or the marginal distributions are skewed (e.g., [6]). In a previous article, we argued that treating ordinal variables as continuous can always be defended. The argument for doing so does neither depend on the number of categories nor the marginal distribution of ordinal variables [8]. Moreover, we argued that no simulation studies would be required to demonstrate this reasoning [8]. However, we feel that it is valuable to provide further evidence for our reasoning by presenting simulation studies and analytical results. We conclude that it is not legitimate to speak about bias when comparing the two competing factor analytic approaches.

The rest of the article is structured as follows. In Section 2, we discuss the two competing approaches for the factor analysis model. Section 3 presents results from two simulation studies. In Section 4, we discuss analytical findings. Finally, the paper closes with a discussion in Section 5.

2. Factor Analysis for Ordinal Items

2.1. Gaussian Copula Model for Ordinal Items

In the following, we present a multivariate distribution for discrete data. This distribution is referred to as the Gaussian copula [9,10,11]. In psychometrics, it is also known as the underlying latent variable approach [12].

Assume that there is a vector of variables (i.e., items) $\mathit{Y}=(Y_1,,\dots Y_I)$. Each of the variables has integer categories $1,\dots ,K_i$. Now, assume that there are underlying standard normally distributed latent variables $Y_i^{*}$ for the integer-valued variable $Y_i$ for $i=1,\dots ,I$. Denote by ${\mathit{Y}}^{*}$ the vector containing the underlying latent variables. and $Y_i^{*}$ is discretized by means of a vector of thresholds $-\infty ={\mathsf{\tau }}_{Y_i,0}<{\mathsf{\tau }}_{Y_i,1}<\dots <{\mathsf{\tau }}_{Y_i,K_i}<{\mathsf{\tau }}_{Y_i,K_i+1}=\infty$. Then, the discrete variable $Y_i$ takes the value h if $Y_i^{*}$ lies within the interval $[{\mathsf{\tau }}_{Y_i,h-1},{\mathsf{\tau }}_{Y_i,h})$. The multivariate vector ${\mathit{Y}}^{*}$ follows a multivariate normal distribution with a correlation matrix ${\mathbf{\Sigma }}^{*}$. Recall that all variables in ${\mathit{Y}}^{*}$ have zero means and variances of one. By prescribing thresholds $\mathsf{\tau }$ for all variables, the discrete probability distribution is given as

$$\mathrm{P}(Y_i=h,Y_j=l)={\mathsf{\Phi }}_2({\mathsf{\tau }}_{Y_i,h},{\mathsf{\tau }}_{Y_i,l};{\mathsf{\rho }}_{Y_i{Y}_j}^{*})-{\mathsf{\Phi }}_2({\mathsf{\tau }}_{Y_i,h-1},{\mathsf{\tau }}_{Y_i,l};{\mathsf{\rho }}_{Y_i{Y}_j}^{*})-{\mathsf{\Phi }}_2({\mathsf{\tau }}_{Y_i,h},{\mathsf{\tau }}_{Y_i,l-1};{\mathsf{\rho }}_{Y_i{Y}_j}^{*})+{\mathsf{\Phi }}_2({\mathsf{\tau }}_{Y_i,h-1},{\mathsf{\tau }}_{Y_i,l-1};{\mathsf{\rho }}_{Y_i{Y}_j}^{*}),$$

(1)

where ${\mathsf{\rho }}_{Y_i{Y}_j}^{*}$ is the correlation of the underlying latent variables and ${\mathsf{\Phi }}_2(x,y;\mathsf{\rho })$ is the cumulative distribution function of the bivariate standard normal distribution.

Means, variances, and covariances of the components of the discrete variable $\mathit{Y}$ can be computed using the discrete distribution in Equation (1) (see [13]). Hence, one can also determine the manifest correlation ${\mathsf{\rho }}_{Y_i{Y}_j}=\mathrm{Cov}(Y_i,Y_j)/\left(\mathrm{SD}\left(Y_i\right)\mathrm{SD}\left(Y_j\right)\right)$ (i.e., Pearson correlation) for the integer-valued variables $Y_i$ and $Y_j$. While ${\mathsf{\rho }}_{Y_i^{*}{Y}_j^{*}}$ can take values within the interval $[-1,1]$, the Pearson correlation is bounded, where the bounds are determined by the marginal distribution of $Y_i$ and $Y_j$ [14,15,16].

Given an observed discrete distribution of $\mathit{Y}$, one can determine the so-called polychoric correlation ${\mathsf{\rho }}_{Y_i{Y}_j}^{*}$ [12,17]. It can be shown that there is a monotone relation between the polychoric correlation ${\mathsf{\rho }}_{Y_i{Y}_j}^{*}$ and the Pearson correlation ${\mathsf{\rho }}_{Y_i{Y}_j}$ [18]. This property can be used to simulate discrete ordinal data [19,20]. First, the discrete marginal distributions $\mathrm{P}(Y_i=h)$ are fixed. These probabilities directly translate into thresholds ${\mathsf{\tau }}_{Y_i,h}$ by noting that

$$\mathsf{\Phi }({\mathsf{\tau }}_{Y_i,h})=\sum _{l=1}^h\mathrm{P}(Y_i=l),$$

(2)

where $\Phi$ is the distribution function of the standard normal distribution. The specification in Equation (2) implies that a Gaussian marginal distribution is assumed for the underlying latent variables $Y_i^{*}$ (i.e., latent normality; see [8]). Next, fix a correlation matrix $\mathbf{\Sigma }$ for the vector of discrete variables $\mathit{Y}$. Using (1), one solves a nonlinear equation ${\mathsf{\rho }}_{Y_i{Y}_j}=f\left({\mathsf{\rho }}_{Y_i{Y}_j}^{*}\right)$ for every pair of variables $Y_i$ and $Y_j$ for obtaining the polychoric correlation ${\mathsf{\rho }}_{Y_i{Y}_j}^{*}$ [20]. As a result, a matrix ${\mathbf{\Sigma }}^{*}$ involving the polychoric correlations is obtained as a function of all thresholds $\mathsf{\tau }$ and the Pearson correlation matrix $\mathbf{\Sigma }$.

In simulations in psychometrics, one often generates data from ${\mathit{Y}}^{*}$ and a prespecified polychoric correlation matrix ${\mathbf{\Sigma }}^{*}$. As a consequence, a Pearson correlation matrix $\mathbf{\Sigma }$ is obtained for the discrete data $\mathit{Y}$. A frequent reasoning is that by treating variables continuously when Pearson correlations are computed, biased estimates are obtained because the estimates are usually smaller (in absolute values) than the “true” polychoric correlations. However, this reasoning depends on how truth is defined. Indeed, one can also argue that Pearson correlations are the quantity of interest. When polychoric correlations are computed, biased estimations will emerge. These reflections demonstrate that it does not make sense to speak about bias. It is up to the choice of a researcher whether Pearson or polychoric correlations are of interest. Instead of speaking about bias, one can only argue that one statistical parameter is of less interest (i.e., less adequate in a practical application) than another.

2.2. Factor Model for Pearson Correlations and Polychoric Correlations

In factor analysis, the vector of multivariate variables is represented by a vector of factor variables of lower dimension. This means that a covariance (or a correlation) matrix is represented by a common part that is attributed to factors and a residual part that represents item uniqueness [21]. A correlation is represented as

$$\mathbf{\Sigma }\approx \mathbf{\Lambda }\mathbf{\Psi }{\mathbf{\Lambda }}^{\top }+\mathbf{\Theta },$$

where $\mathbf{\Lambda }$ is the matrix of factor loadings, $\mathbf{\Psi }$ is the correlation matrix of factor variables, and $\mathbf{\Theta }$ is a diagonal matrix of residual variances. Frequently, a particular structure in $\mathbf{\Lambda }$ (i.e., a loading structure) is prescribed that determines which variables $Y_i$ load on which factors [21]. Such a specification is labeled as confirmatory factor analysis (CFA).

The most prominent options appearing in the literature for modeling ordinal variables in CFA are treating the variables continuously and ordinally [13]. In the first case, Pearson correlations $\mathbf{\Sigma }$ are used for modeling the factor structure. In the latter case, polychoric correlations ${\mathbf{\Sigma }}^{*}$ are used for modeling the factor structure. Hence, there can be two alternative modeling strategies for the variables $\mathit{Y}$:

$$\mathbf{\Sigma }\approx \mathbf{\Lambda }\mathbf{\Psi }{\mathbf{\Lambda }}^{\top }+\mathbf{\Theta }\mathrm{and}{\mathbf{\Sigma }}^{*}\approx {\mathbf{\Lambda }}^{*}{\mathbf{\Psi }}^{*}{{\mathbf{\Lambda }}^{*}}^{\top }+{\mathbf{\Theta }}^{*}.$$

(3)

In general, factor loadings $\mathbf{\Lambda }$ and ${\mathbf{\Lambda }}^{*}$ obtained from factor models for Pearson correlations and polychoric correlations differ from each other. The same is true for the factor correlation matrices $\mathbf{\Psi }$ and ${\mathbf{\Psi }}^{*}$. Hence, researchers are often unsure which of the two approaches should be chosen. In psychometrics, it is often claimed that factor loadings and factor correlations from Pearson correlations are biased for ordinal data (e.g., [6,13]). As argued in Section 2.1, we think that it is illegitimate to speak about bias because both Pearson correlations $\mathbf{\Sigma }$ and polychoric correlations ${\mathbf{\Sigma }}^{*}$ are well-defined parameters for a vector of discrete variables. Hence, both representations in (3) for modeling the factor structure are well-defined, and it is not justified to speak about a bias when utilizing one of the two alternatives. Alternatively, it is equally true that factor loadings ${\mathbf{\Lambda }}^{*}$ and factor correlations ${\mathbf{\Psi }}^{*}$ obtained from polychoric correlations are biased if a researcher is interested in $\mathbf{\Lambda }$ and $\mathbf{\Psi }$ stemming from the factor model for Pearson correlations. In fact, one can simulate discrete multivariate data with the factor representation in (3) for Pearson correlations and show that applying the CFA for polychoric correlations provides biased estimates (see [8]).

In the rest of the article, we want to show the differences between the two competing approaches. We illustrate that biases can occur in both approaches. If a researcher defines one approach (e.g., ordinal CFA) as the true model, then parameter estimates from the other approach (e.g., CFA based on Pearson correlations for continuous data) will be biased. However, the converse also holds true. We simulate a factor model based on Pearson correlations (as outlined in Section 2.1) and demonstrate that bias for an ordinal CFA will result. In the simulations and the analytical derivations, we are not concerned with sampling errors and compare the different estimators only at the population level (see also [13]).

3. Simulation Studies

In this section, we present two simulation studies with two-dimensional CFA models. In Simulation Study 1 (Section 3.1), we consider equal factor loadings. Simulation Study 2 (Section 3.2) replicates [6,13] and considers unequal factor loadings.

3.1. Simulation Study 1

3.1.1. Method

In Simulation Study 1, we studied a two-dimensional CFA. Each of the two factors is measured by three items (i.e., variables). We employed the same distributions as in [6]. There were discrete variables with 2 to 7 categories and were symmetric (S2, …, S7), moderately asymmetric (MA2, …, MA7), and extremely asymmetric (EA2, …, EA7) distributed. The shapes of the discrete distributions are displayed in Figure 1.

Table A1. Data-generating parameters for marginal distributions of items (i.e., item-category probabilities $\mathrm{P}(Y_i=h)$ for $h=1,\dots ,K_i$).

Dist	Cat1	Cat2	Cat3	Cat4	Cat5	Cat6	Cat7
S2	0.500	0.500
MA2	0.648	0.352
EA2	0.867	0.133
S3	0.206	0.589	0.206
MA3	0.303	0.486	0.211
EA3	0.719	0.151	0.130
S4	0.112	0.388	0.388	0.112
MA4	0.378	0.420	0.154	0.049
EA4	0.620	0.155	0.120	0.106
S5	0.067	0.242	0.383	0.242	0.067
MA5	0.248	0.440	0.213	0.085	0.014
EA5	0.531	0.163	0.116	0.102	0.088
S6	0.064	0.154	0.282	0.282	0.154	0.064
MA6	0.146	0.368	0.271	0.125	0.076	0.014
EA6	0.441	0.151	0.132	0.112	0.092	0.072
S7	0.048	0.108	0.210	0.269	0.210	0.108	0.048
MA7	0.075	0.247	0.301	0.164	0.116	0.082	0.014
EA7	0.388	0.151	0.132	0.112	0.092	0.072	0.053

Note. Dist = distribution type of categorical data; Cat1, …, Cat7 = category 1, …, 7; S2, …, S7 = symmetric distribution with 2 to 7 categories; MA2, …, MA7 = moderate asymmetric distribution with 2 to 7 categories; EA2, …, EA7 = extreme asymmetric distribution with 2 to 7 categories.

in Appendix A contains the numerical values of the marginal distributions. In each of the simulated conditions, all items had the same marginal distribution.

We varied the factor correlations $\mathsf{\psi }$ in three levels (i.e., 0.3, 0.5, and 0.7). We also used three factor levels for factor loadings (i.e., 0.3, 0.5, 0.7). All factor loadings in the two-dimensional CFA were set equal to each other. The residual variances were determined to obtain standardized variables (i.e., variables with a variance of 1).

Two data-generating models were used. Discrete items were simulated in which the factor model involving loadings, factor correlations, and residual variances either holds for the discrete data $\mathit{Y}$ (“Simulated cont”) or the underlying continuous data ${\mathit{Y}}^{*}$ (“Simulated cat”). If the factor model holds for Pearson correlations, the simulation approach in Section 2.2 was used. According to the factor model parameters and the marginal distributions, a Pearson correlation matrix $\mathbf{\Sigma }$ was determined in the first step. In a second step, the corresponding population polychoric correlation matrix ${\mathbf{\Sigma }}^{*}$ was computed.

Moreover, three analysis models were utilized. First, one uses the population Pearson correlation matrix of the original integer-valued variable $\mathit{Y}$ (method “cont”) and maximum likelihood estimation for determining model parameters. Second, the original scores were transformed to normal scores $\tilde{\mathit{Y}}$ [22,23], where score h of item $Y_i$ received the score

$$z_h=\frac{\mathsf{\varphi }({\mathsf{\tau }}_{Y_i,h-1})-\mathsf{\varphi }({\mathsf{\tau }}_{Y_i,h})}{\mathsf{\Phi }({\mathsf{\tau }}_{Y_i,h})-\mathsf{\Phi }({\mathsf{\tau }}_{Y_i,h-1})},$$

where $\mathsf{\varphi }$ is the density function of the standard normal distribution. In the method “cont-adj”, a CFA model of the Pearson correlation matrix of normal scores is estimated with ML. Finally, the CFA based on polychoric correlations (method “cat”) was estimated using unweighted least squares estimation. All models were estimated using the R [24] package lavaan [25]. Note that in all simulation conditions, only population-level data is used. Hence, only asymptotic bias is investigated.

3.1.2. Results

We now show biases for factor loadings and factor correlations in the different simulation conditions.

Table 1. Simulation Study 1: Estimated factor loadings for a factor correlation of 0.5 as a function of data-generating models, estimation methods, and true factor loadings.

	True Loading = 0.3						True Loading = 0.7
	Simulated cont			Simulated cat			Simulated cont			Simulated cat
	Estimation Methods			Estimation Methods			Estimation Methods			Estimation Methods
Dist	cont	cont-adj	cat	cont	cont-adj	cat	cont	cont-adj	cat	cont	cont-adj	cat
S2	0.300	0.300	0.375	0.240	0.240	0.300	0.700	0.700	0.834	0.571	0.571	0.700
S3	0.300	0.300	0.338	0.266	0.266	0.300	0.700	0.700	0.785	0.623	0.623	0.700
S4	0.300	0.300	0.321	0.280	0.280	0.300	0.700	0.700	0.748	0.654	0.654	0.700
S5	0.300	0.300	0.314	0.286	0.287	0.300	0.700	0.700	0.732	0.669	0.669	0.700
S6	0.300	0.300	0.310	0.291	0.291	0.300	0.700	0.700	0.721	0.679	0.679	0.700
S7	0.300	0.300	0.307	0.293	0.293	0.300	0.700	0.700	0.716	0.684	0.684	0.700
MA2	0.300	0.300	0.383	0.234	0.234	0.300	0.700	0.700	0.839	0.564	0.564	0.700
MA3	0.300	0.300	0.334	0.269	0.269	0.300	0.700	0.700	0.772	0.633	0.633	0.700
MA4	0.300	0.301	0.330	0.273	0.273	0.300	0.700	0.698	0.756	0.646	0.645	0.700
MA5	0.300	0.301	0.322	0.280	0.281	0.300	0.700	0.699	0.743	0.659	0.658	0.700
MA6	0.300	0.302	0.316	0.285	0.287	0.300	0.700	0.701	0.731	0.670	0.672	0.700
MA7	0.300	0.303	0.312	0.289	0.291	0.300	0.700	0.703	0.723	0.677	0.681	0.700
EA2	0.300	0.300	0.447	0.195	0.195	0.300	0.700	0.700	0.873	0.511	0.511	0.700
EA3	0.300	0.301	0.379	0.236	0.237	0.300	0.700	0.697	0.808	0.587	0.585	0.700
EA4	0.300	0.302	0.356	0.252	0.254	0.300	0.700	0.699	0.780	0.617	0.618	0.700
EA5	0.300	0.304	0.342	0.262	0.266	0.300	0.700	0.701	0.762	0.636	0.638	0.700
EA6	0.300	0.304	0.331	0.272	0.275	0.300	0.700	0.702	0.747	0.652	0.654	0.700
EA7	0.300	0.304	0.325	0.276	0.280	0.300	0.700	0.702	0.739	0.660	0.663	0.700

Note. Dist = distribution type of categorical data; S2, …, S7 = symmetric distribution with 2 to 7 categories; MA2, …, MA7 = moderate asymmetric distribution with 2 to 7 categories; EA2, …, EA7 = extreme asymmetric distribution with 2 to 7 categories; cont = ML estimation using Pearson correlations of integer scores; cont-adj = ML estimation using Pearson correlations of normal scores; cat = DWLS estimation based on polychoric correlations; Entries with absolute biases larger than 0.025 are printed in bold.

shows the estimated factor loadings for a factor correlation of 0.5. The results for the factor correlations 0.3 and 0.7 coincide with those in Table 1. The general pattern was that biases in estimated factor loadings were more pronounced for a higher true factor loading of 0.7 than for a true loading of 0.3. If discrete items were simulated according to the CFA model for Pearson correlations (“Simulated cont”), it is evident that treating variables as categorical resulted in positively biased factor loadings. In contrast, if the CFA model was simulated for polychoric correlations (“Simulated cat”), continuous treatment of the items resulted in negatively biased factor loadings. Interestingly and similar to [22], CFA based on normal scores provided very similar estimated factor loadings to the CFA model for Pearson correlations based on the original integer scores. Similar to the findings in the literature, it can also be concluded that biases in estimated factor loadings were less pronounced for less asymmetric distributions and an increasing number of item categories.

In

Table 2. Simulation Study 1: Estimated factor correlations as a function of data-generating models, estimation methods, true factor loadings, and true factor correlations.

	True Factor Correlation = 0.3						True Factor Correlation = 0.7
	Simulated cont			Simulated cat			Simulated cont			Simulated cat
	Estimation Methods			Estimation Methods			Estimation Methods			Estimation Methods
Dist	cont	cont-adj	cat	cont	cont-adj	cat	cont	cont-adj	cat	cont	cont-adj	cat
	True loading = 0.3
S2	0.300	0.300	0.301	0.300	0.300	0.300	0.700	0.700	0.701	0.700	0.700	0.700
S3	0.300	0.300	0.300	0.300	0.300	0.300	0.700	0.700	0.700	0.700	0.700	0.700
S4	0.300	0.300	0.300	0.300	0.300	0.300	0.700	0.700	0.700	0.700	0.700	0.700
S5	0.300	0.300	0.300	0.300	0.300	0.300	0.700	0.700	0.700	0.700	0.700	0.700
S6	0.300	0.300	0.300	0.300	0.300	0.300	0.700	0.700	0.700	0.700	0.700	0.700
S7	0.300	0.300	0.300	0.300	0.300	0.300	0.700	0.700	0.700	0.700	0.700	0.700
MA2	0.300	0.300	0.303	0.298	0.298	0.300	0.700	0.700	0.703	0.698	0.698	0.700
MA3	0.300	0.300	0.300	0.300	0.300	0.300	0.700	0.700	0.700	0.700	0.700	0.700
MA4	0.300	0.301	0.302	0.299	0.299	0.300	0.700	0.701	0.702	0.699	0.699	0.700
MA5	0.300	0.301	0.301	0.299	0.300	0.300	0.700	0.701	0.701	0.699	0.700	0.700
MA6	0.300	0.301	0.301	0.299	0.300	0.300	0.700	0.701	0.701	0.699	0.700	0.700
MA7	0.300	0.300	0.300	0.300	0.300	0.300	0.700	0.700	0.700	0.700	0.700	0.700
EA2	0.300	0.300	0.325	0.289	0.289	0.300	0.700	0.700	0.723	0.689	0.689	0.700
EA3	0.300	0.301	0.309	0.294	0.295	0.300	0.700	0.701	0.709	0.694	0.695	0.700
EA4	0.300	0.301	0.306	0.296	0.297	0.300	0.700	0.701	0.706	0.696	0.697	0.700
EA5	0.300	0.301	0.304	0.297	0.298	0.300	0.700	0.701	0.704	0.697	0.698	0.700
EA6	0.300	0.301	0.302	0.298	0.299	0.300	0.700	0.701	0.702	0.698	0.699	0.700
EA7	0.300	0.301	0.302	0.298	0.299	0.300	0.700	0.701	0.702	0.698	0.699	0.700
	True loading = 0.7
S2	0.300	0.300	0.329	0.288	0.288	0.300	0.700	0.700	0.737	0.684	0.684	0.700
S3	0.300	0.300	0.302	0.299	0.299	0.300	0.700	0.700	0.704	0.698	0.698	0.700
S4	0.300	0.300	0.301	0.299	0.299	0.300	0.700	0.700	0.701	0.699	0.699	0.700
S5	0.300	0.300	0.301	0.299	0.300	0.300	0.700	0.700	0.701	0.699	0.700	0.700
S6	0.300	0.300	0.301	0.299	0.300	0.300	0.700	0.700	0.701	0.699	0.699	0.700
S7	0.300	0.300	0.301	0.300	0.300	0.300	0.700	0.700	0.701	0.699	0.700	0.700
MA2	0.300	0.300	0.337	0.283	0.283	0.300	0.700	0.700	0.744	0.679	0.679	0.700
MA3	0.300	0.300	0.306	0.296	0.296	0.300	0.700	0.700	0.707	0.695	0.695	0.700
MA4	0.300	0.303	0.310	0.292	0.294	0.300	0.700	0.702	0.710	0.691	0.694	0.700
MA5	0.300	0.303	0.306	0.295	0.298	0.300	0.700	0.703	0.706	0.695	0.697	0.700
MA6	0.300	0.303	0.305	0.296	0.299	0.300	0.700	0.703	0.705	0.696	0.699	0.700
MA7	0.300	0.303	0.303	0.297	0.300	0.300	0.700	0.703	0.703	0.697	0.699	0.700
EA2	0.300	0.300	0.403	0.246	0.246	0.300	0.700	0.700	0.788	0.642	0.642	0.700
EA3	0.300	0.304	0.350	0.268	0.271	0.300	0.700	0.703	0.748	0.665	0.668	0.700
EA4	0.300	0.305	0.334	0.275	0.279	0.300	0.700	0.704	0.734	0.673	0.677	0.700
EA5	0.300	0.306	0.325	0.281	0.285	0.300	0.700	0.705	0.726	0.679	0.684	0.700
EA6	0.300	0.305	0.317	0.286	0.290	0.300	0.700	0.705	0.718	0.684	0.689	0.700
EA7	0.300	0.305	0.314	0.288	0.292	0.300	0.700	0.705	0.715	0.686	0.691	0.700

Note. Dist = distribution type of categorical data; S2, …, S7 = symmetric distribution with 2 to 7 categories; MA2, …, MA7 = moderate asymmetric distribution with 2 to 7 categories; EA2, …, EA7 = extreme asymmetric distribution with 2 to 7 categories; cont = ML estimation using Pearson correlations of integer scores; cont-adj = ML estimation using Pearson correlations of normal scores; cat = DWLS estimation based on polychoric correlations; Entries with absolute biases larger than 0.025 are printed in bold.

, estimated factor correlations are displayed as a function of true factor loading and true factor correlations. Overall, biases in factor correlations were smaller than in factor loadings. Moreover, the bias in factor correlations was more pronounced for larger true factor loadings. Biases were only of practical relevance for a few item categories (i.e., two categories) or extreme asymmetric distributions (i.e., EA). In addition, estimated factor correlations were positively biased if the CFA model holds for Pearson correlations (“Simulated cont”). At the same time, negative biases existed in a continuous treatment of items when the CFA model holds for polychoric correlations (“Simulated cat”). The latter findings replicate the simulation studies found in the literature. Unfortunately, the psychometric literature conceals the former finding.

3.1.3. Summary

As argued in Section 2.2, it is questionable speaking about bias in the comparison of a continuous or an ordinal CFA approach. Simulation Study 1 shows that can either claim that there is bias when the items are treated ordinally (i.e., using polychoric correlations) or when treated continuously (i.e., using Pearson correlations). It might be more neutral to only speak about parameter differences using different approaches.

3.2. Simulation Study 2: Unequal Loadings

Simulation Study 2 considers the case of unequal loadings in a two-dimensional CFA and is a replication of [6,13].

3.2.1. Method

In this study, we investigate a two-dimensional CFA in which each of the two factors is measured by five items. The factor loadings of the five items for each factor were 0.3, 0.4, 0.5, 0.6, and 0.7. The factor correlation was fixed to 0.3. As in Simulation Study 1, all marginal distributions of the items were equal in each simulation condition. The identical 18 marginal distributions as in Simulation Study 1 were utilized. According to the findings of Simulation Study 1, the CFA model for normal scores was not considered anymore because the results were very similar to those using Pearson correlations. Like in Simulation 1, we either assumed that the CFA model holds for Pearson correlations (“Simulated cont”) or polychoric correlations (“Simulated cat”). Then, we estimated the CFA model based on the other correlation matrix that did not correspond to the data-generating parameters. It is obvious that fitting the corresponding correlation matrix provides unbiased estimates (see also Simulation Study 1).

3.2.2. Results

Table 3. Simulation Study 2: Estimated factor loadings ${\mathsf{\lambda }}_1,\dots ,{\mathsf{\lambda }}_5$ and factor correlations $\mathsf{\psi }$ as a function of data-generating models and estimation methods.

	Simulated cont, Estimated cat						Simulated cat, Estimated cont
Parm True	${\mathsf{\lambda }}_1$ 0.3	${\mathsf{\lambda }}_2$ 0.4	${\mathsf{\lambda }}_3$ 0.5	${\mathsf{\lambda }}_4$ 0.6	${\mathsf{\lambda }}_5$ 0.7	$\mathsf{\psi }$ 0.3	${\mathsf{\lambda }}_1$ 0.3	${\mathsf{\lambda }}_2$ 0.4	${\mathsf{\lambda }}_3$ 0.5	${\mathsf{\lambda }}_4$ 0.6	${\mathsf{\lambda }}_5$ 0.7	$\mathsf{\psi }$ 0.3
Dist
S2	0.379	0.501	0.618	0.730	0.846	0.313	0.238	0.319	0.401	0.485	0.568	0.294
S3	0.338	0.451	0.563	0.674	0.786	0.301	0.266	0.355	0.444	0.533	0.622	0.299
S4	0.322	0.429	0.535	0.642	0.749	0.301	0.280	0.373	0.467	0.561	0.654	0.300
S5	0.314	0.419	0.524	0.628	0.733	0.300	0.286	0.382	0.477	0.573	0.669	0.300
S6	0.310	0.413	0.516	0.619	0.722	0.300	0.291	0.388	0.485	0.582	0.679	0.300
S7	0.307	0.410	0.512	0.614	0.717	0.300	0.293	0.390	0.488	0.586	0.684	0.300
MA2	0.388	0.510	0.626	0.738	0.853	0.318	0.232	0.312	0.394	0.478	0.560	0.290
MA3	0.335	0.445	0.555	0.664	0.774	0.303	0.269	0.359	0.450	0.541	0.645	0.298
MA4	0.331	0.439	0.546	0.652	0.758	0.306	0.272	0.364	0.457	0.551	0.658	0.294
MA5	0.322	0.429	0.534	0.639	0.744	0.304	0.279	0.373	0.468	0.563	0.669	0.297
MA6	0.316	0.421	0.525	0.628	0.732	0.303	0.284	0.380	0.476	0.573	0.677	0.297
MA7	0.312	0.415	0.518	0.621	0.724	0.302	0.288	0.385	0.482	0.580	0.504	0.298
EA2	0.454	0.575	0.686	0.789	0.893	0.367	0.191	0.263	0.341	0.423	0.582	0.262
EA3	0.383	0.498	0.608	0.712	0.819	0.331	0.232	0.316	0.403	0.493	0.613	0.278
EA4	0.359	0.470	0.578	0.682	0.787	0.321	0.249	0.337	0.428	0.521	0.633	0.283
EA5	0.344	0.454	0.560	0.663	0.768	0.315	0.260	0.351	0.444	0.539	0.649	0.287
EA6	0.332	0.440	0.545	0.648	0.752	0.310	0.270	0.363	0.457	0.554	0.658	0.291
EA7	0.327	0.433	0.537	0.640	0.743	0.309	0.275	0.369	0.464	0.561	0.632	0.292

Note. Parm = parameter; True = true parameter value; Dist = distribution type of categorical data; S2, …, S7 = symmetric distribution with 2 to 7 categories; MA2, …, MA7 = moderate asymmetric distribution with 2 to 7 categories; EA2, …, EA7 = extreme asymmetric distribution with 2 to 7 categories; cont = ML estimation using Pearson correlations of integer scores; cont-adj = ML estimation using Pearson correlations of normal scores; cat = DWLS estimation based on polychoric correlations; Entries with absolute biases larger than 0.025 are printed in bold.

presents estimated factor loadings and factor correlations. The factor loadings are positively biased if the factor model holds for Pearson correlations, but it is fitted for polychoric correlations. Only in cases of extreme asymmetry, positive biases in factor correlations were non-negligible. In other cases, the bias in factor correlations can be considered small. In contrast, and as shown in several simulation studies such as [6], estimated factor loadings are negatively biased if the items are treated as continuous variables when the factor model holds for the polychoric correlations. Estimated factor correlations were only substantially biased for extreme asymmetric distributions and a few item categories.

3.2.3. Summary

In the original Rhemtulla et al. [6] study, it was stated that “Normal theory ML [applying CFA to Pearson correlations; added by the author] was found to be more sensitive to asymmetric category thresholds and was especially biased when estimating large factor loadings”. We demonstrated that treating discrete items ordinally by using polychoric correlations is also biased if the true CFA model holds for Pearson correlations. Statements like the one cited above are particularly dangerous in applied research because it seems to imply that there would be a univocal voice under methodologists and statisticians that either one of two approaches must always be preferred for reasons of statistical bias. We argued that these kinds of statements cannot be used to give advice on choosing an adequate estimation approach in a concrete empirical application.

4. Analytical Findings

We now provide an analytical treatment of the two-dimensional factor model for ordinal variables. The basis of our derivations is an empirically obtained relation of the polychoric correlation ${\mathsf{\rho }}^{*}$ and the Pearson correlation $\mathsf{\rho }$. We assume that the bivariate distribution of two discrete variables is represented by a Gaussian copula model (see Section 2.1). It was shown that there exists an injective relationship of ${\mathsf{\rho }}^{*}$ and $\mathsf{\rho }$. By studying the relationship for the distributions presented in Section 3.1.1, it turned out that a quadratic relationship of the square root of the correlations provided a close approximation. That is, we can empirically determine coefficients ${\mathsf{\nu }}_1$ and ${\mathsf{\nu }}_2$ such that

$$\sqrt{{\mathsf{\rho }}^{*}}\simeq {\mathsf{\nu }}_1\sqrt{\mathsf{\rho }}(1-{\mathsf{\nu }}_2\sqrt{\mathsf{\rho }}).$$

(4)

Table A2. Approximation $\sqrt{{\mathsf{\rho }}^{*}}\simeq {\mathsf{\nu }}_1\sqrt{\mathsf{\rho }}(1-{\mathsf{\nu }}_2\sqrt{\mathsf{\rho }})$ of the square root of the polychoric correlation $\sqrt{{\mathsf{\rho }}^{*}}$ as a function of the square root of the Pearson correlation $\sqrt{\mathsf{\rho }}$.

Dist1	Dist2	${\mathsf{\nu }}_1$	${\mathsf{\nu }}_2$	Error
S2	S2	1.339	0.161	0.013
MA2	MA2	1.380	0.191	0.012
EA2	EA2	1.692	0.377	0.006
S2	MA2	1.347	0.144	0.011
S2	EA2	1.331	−0.154	0.041
MA2	EA2	1.455	0.130	0.020
S3	S3	1.137	0.021	0.003
MA3	MA3	1.130	0.035	0.003
EA3	EA3	1.367	0.224	0.006
S3	MA3	1.130	0.021	0.002
S3	EA3	1.224	0.032	0.007
MA3	EA3	1.219	0.049	0.003
S4	S4	1.075	0.007	0.001
MA4	MA4	1.121	0.054	0.002
EA4	EA4	1.258	0.166	0.005
S4	MA4	1.092	0.009	0.001
S4	EA4	1.141	0.014	0.001
MA4	EA4	1.181	0.091	0.002
S5	S5	1.049	0.004	0.000
MA5	MA5	1.085	0.031	0.001
EA5	EA5	1.194	0.127	0.004
S5	MA5	1.063	0.004	0.000
S5	EA5	1.102	0.010	0.001
MA5	EA5	1.131	0.060	0.001
S6	S6	1.034	0.005	0.000
MA6	MA6	1.062	0.025	0.001
EA6	EA6	1.140	0.092	0.003
S6	MA6	1.045	0.004	0.000
S6	EA6	1.074	0.011	0.001
MA6	EA6	1.096	0.047	0.001
S7	S7	1.026	0.004	0.000
MA7	MA7	1.046	0.018	0.001
EA7	EA7	1.116	0.078	0.003
S7	MA7	1.034	0.004	0.000
S7	EA7	1.060	0.009	0.000
MA7	EA7	1.077	0.038	0.001

Note. Dist1 = distribution type of first variable; Dist2 = distribution type of second variable; S2, …, S7 = symmetric distribution with 2 to 7 categories; MA2, …, MA7 = moderate asymmetric distribution with 2 to 7 categories; EA2, …, EA7 = extreme asymmetric distribution with 2 to 7 categories; Error = maximum function approximation error for Pearson correlations ρ smaller than 0.8. Error entries with absolute errors larger than 0.01 are printed in bold.

in Appendix A shows the estimated coefficients of the approximating function of the square root of the polychoric correlation $\sqrt{{\mathsf{\rho }}^{*}}$ as a function of the square root of the Pearson correlation $\sqrt{\mathsf{\rho }}$. The coefficient ${\mathsf{\nu }}_1$ is larger than 1 and is close to 1 for variables with many categories. Moreover, ${\mathsf{\nu }}_1$ is substantially larger than 1 for more asymmetric variables. The coefficient ${\mathsf{\nu }}_2$ is larger than 0 (except for the combination of distributions S2 and EA2; see Table A2). The coefficient ${\mathsf{\nu }}_2$ reaches 0 for variables with many categories and less asymmetrical distributions. The absolute error of the numerical approximation (4) for the range $\mathsf{\rho }\in [0,0.8]$ is also displayed in Table A2. It turned out that the approximation error is negligible (i.e., smaller than 0.01) for variables with at least 3 categories.

Figure 2 illustrates the relationship between the square roots of the polychoric correlation and the Pearson correlation. For two 3-category items with extreme asymmetry (i.e., EA3; see Figure 1), the functional form can be well represented as a quadratic function if the Pearson correlation is smaller than 0.8 (i.e., $|\mathsf{\rho }|<0.8$).

4.1. Case 1: Equal Loadings

Now, we turn to the determination of factor loadings $\mathsf{\lambda }$ and a factor correlation $\mathsf{\psi }$ in confirmatory factor analysis.

We assume that a two-dimensional factor model holds. Each of the two factors, X and Y, is measured by three variables. Furthermore, we assume that there are nonnegative common factor loadings ${\mathsf{\lambda }}_X$ and ${\mathsf{\lambda }}_Y$ for all items of a factor. Assuming three variables, the measurement model is just identified in case of unequal loadings, and there is no model misfit. For equal loadings, all pairs of correlations (i.e., polychoric and Pearson correlations) corresponding to the same factor are equal. Let ${\mathsf{\lambda }}_X$, ${\mathsf{\lambda }}_Y$, and $\mathsf{\psi }$ denote the parameters defined in the factor model by treating the variables continuously (i.e., using Pearson correlations) and ${\mathsf{\lambda }}_X^{*}$, ${\mathsf{\lambda }}_Y^{*}$, and ${\mathsf{\psi }}^{*}$ in the factor model relying on polychoric correlations. The correlation of two items from the same scale X is given as

$${\mathsf{\rho }}_X={\mathsf{\lambda }}_X^2\mathrm{and}{\mathsf{\rho }}_X^{*}={({\mathsf{\lambda }}_X^{*})}^2(\mathrm{similarly}\mathrm{for}Y).$$

(5)

For two items that measure the different factors X and Y, we get

$${\mathsf{\rho }}_{XY}={\mathsf{\lambda }}_X{\mathsf{\lambda }}_Y\mathsf{\psi }\mathrm{and}{\mathsf{\rho }}_{XY}^{*}={\mathsf{\lambda }}_X^{*}{\mathsf{\lambda }}_Y^{*}{\mathsf{\psi }}^{*}.$$

(6)

The Equations (5) and (6) can be used for deriving estimates for ${\mathsf{\lambda }}_h$ and $\mathsf{\varphi }$:

$${\mathsf{\lambda }}_X=\sqrt{{\mathsf{\rho }}_X}\mathrm{and}{\mathsf{\lambda }}_X^{*}=\sqrt{{\mathsf{\rho }}_X^{*}}\mathrm{and}$$

(7)

$$\mathsf{\psi }=\frac{{\mathsf{\rho }}_{XY}}{\sqrt{{\mathsf{\rho }}_X{\mathsf{\rho }}_Y}}\mathrm{and}{\mathsf{\psi }}^{*}=\frac{{\mathsf{\rho }}_{XY}^{*}}{\sqrt{{\mathsf{\rho }}_X^{*}{\mathsf{\rho }}_Y^{*}}}.$$

We now analyze the relations of estimated factor loadings ${\mathsf{\lambda }}_X^{*}$ and ${\mathsf{\lambda }}_X$ and the estimated factor correlation ${\mathsf{\psi }}^{*}$ and $\mathsf{\psi }$. Using (7), we compute by relying on the approximation (4)

$${\mathsf{\lambda }}_X^{*}=\sqrt{{\mathsf{\rho }}_X^{*}}={\mathsf{\nu }}_{1,X}\sqrt{{\mathsf{\rho }}_X}(1-{\mathsf{\nu }}_{2,X}\sqrt{{\mathsf{\rho }}_X})={\mathsf{\nu }}_{1,X}{\mathsf{\lambda }}_X(1-{\mathsf{\nu }}_{2,X}{\mathsf{\lambda }}_X).$$

(8)

For values of ${\mathsf{\nu }}_{2,X}$ near to zero, we get from (8)

$${\mathsf{\lambda }}_X^{*}\simeq {\mathsf{\nu }}_{1,X}{\mathsf{\lambda }}_X$$

(9)

The estimated factor loadings in the two competing approaches are approximately linearly related. Moreover, factor loadings obtained from polychoric correlations will be larger than those from Pearson correlations because ${\mathsf{\nu }}_1$ is typically larger than 1 (see Table A2). Thus, the finding (9) confirms the findings obtained for Simulation Study 1 (Section 3.1).

We now derive the relation for the factor correlations for the two factors. We get by again using (4)

$${\mathsf{\psi }}^{*}=\frac{{\mathsf{\rho }}_{XY}^{*}}{\sqrt{{\mathsf{\rho }}_X^{*}{\mathsf{\rho }}_Y^{*}}}=\frac{{\mathsf{\nu }}_{1,XY}^2{\mathsf{\lambda }}_X{\mathsf{\lambda }}_Y\mathsf{\psi }{\left[1-{\mathsf{\nu }}_{2,XY}\sqrt{{\mathsf{\lambda }}_X{\mathsf{\lambda }}_Y\mathsf{\psi }}\right]}^2}{\sqrt{{\mathsf{\nu }}_{1,X}^2{\mathsf{\lambda }}_1^2{(1-{\mathsf{\nu }}_{2,X}{\mathsf{\lambda }}_X)}^2}\sqrt{{\mathsf{\nu }}_{1,Y}^2{\mathsf{\lambda }}_Y^2{(1-{\mathsf{\nu }}_{2,Y}{\mathsf{\lambda }}_Y)}^2}}$$

(10)

By simplifying (10), we obtain

$${\mathsf{\psi }}^{*}=\mathsf{\psi }{\mathsf{\nu }}_1^{*}\frac{{\left[1-{\mathsf{\nu }}_{2,XY}\sqrt{{\mathsf{\lambda }}_X{\mathsf{\lambda }}_Y\mathsf{\psi }}\right]}^2}{(1-{\mathsf{\nu }}_{2,X}{\mathsf{\lambda }}_X)(1-{\mathsf{\nu }}_{2,Y}{\mathsf{\lambda }}_Y)},$$

(11)

where ${\mathsf{\nu }}_1^{*}=\frac{{\mathsf{\nu }}_{1,XY}^2}{{\mathsf{\nu }}_{1,X}{\mathsf{\nu }}_{1,Y}}$. Note that ${\mathsf{\nu }}_1^{*}$ equals 1 if all variables have the same marginal distribution. By applying the Taylor approximation ${(1-x)}^{-1}\simeq 1+x$ for the terms $(1-{\mathsf{\nu }}_{2,X}{\mathsf{\lambda }}_X)$ and $(1-{\mathsf{\nu }}_{2,Y}{\mathsf{\lambda }}_Y)$, and by ignoring higher-order terms in ${\mathsf{\nu }}_2$, we get by further simplifying (11)

$${\mathsf{\psi }}^{*}\simeq \mathsf{\psi }{\mathsf{\nu }}_1^{*}\left[1+{\mathsf{\nu }}_{2,X}{\mathsf{\lambda }}_1+{\mathsf{\nu }}_{2,Y}{\mathsf{\lambda }}_2-2{\mathsf{\nu }}_{2,XY}\sqrt{{\mathsf{\lambda }}_1{\mathsf{\lambda }}_2\mathsf{\psi }}\right]$$

(12)

For ${\mathsf{\nu }}_2\equiv {\mathsf{\nu }}_{2,X}={\mathsf{\nu }}_{2,Y}={\mathsf{\nu }}_{2,XY}$ and ${\mathsf{\nu }}_1^{*}={\mathsf{\nu }}_{1,X}={\mathsf{\nu }}_{1,Y}={\mathsf{\nu }}_{1,XY}$, we obtain from (12)

$${\mathsf{\psi }}^{*}\simeq \mathsf{\psi }\left[1+{\mathsf{\nu }}_2\left\{{\mathsf{\lambda }}_X+{\mathsf{\lambda }}_Y-\sqrt{{\mathsf{\lambda }}_X{\mathsf{\lambda }}_Y\mathsf{\psi }}\right\}\right]$$

(13)

We now analyze the term in curly brackets in (14). First, assume ${\mathsf{\lambda }}_X\ge {\mathsf{\lambda }}_Y\ge 0$. Then, we can write ${\mathsf{\lambda }}_X={\mathsf{\lambda }}_Y+e$ with $e\ge 0$. We obtain

$${\mathsf{\lambda }}_X+{\mathsf{\lambda }}_Y-\sqrt{{\mathsf{\lambda }}_X{\mathsf{\lambda }}_Y\mathsf{\psi }}=2{\mathsf{\lambda }}_Y+e-\sqrt{{\mathsf{\lambda }}_Y+e}\sqrt{{\mathsf{\lambda }}_Y}\sqrt{\mathsf{\psi }}\ge 2{\mathsf{\lambda }}_Y+e-\sqrt{{\mathsf{\lambda }}_Y+e}\sqrt{{\mathsf{\lambda }}_Y+e}={\mathsf{\lambda }}_Y\ge 0$$

Similarly, we get the same result with ${\mathsf{\lambda }}_Y\ge {\mathsf{\lambda }}_X\ge 0$. Hence, the latent correlation based on polychoric correlations is expected to be larger than based on Pearson correlations because Equation (14) can be written as

$${\mathsf{\psi }}^{*}\simeq \mathsf{\psi }(1+{\mathsf{\nu }}_{2}C),$$

(14)

where C and ${\mathsf{\nu }}_2$ are larger than 0. This finding also confirms the result from the simulation studies. Because the quadratic coefficient ${\mathsf{\nu }}_2$ turned out to be smaller than the deviation of linear coefficient ${\mathsf{\nu }}_1$ from 1 (see Table A2), differences between the approaches using polychoric and Pearson correlations are expected to be larger for factor loadings than for factor correlations.

4.2. Case 2: Unequal Loadings

We now turn to the case of unequal factor loadings in the two-dimensional factor model. This situation was considered in Simulation Study 2 (Section 3.2). For two items $X_i$ and $X_j$ that measure the same factor X, it holds that

$${\mathsf{\rho }}_{X_i{X}_j}={\mathsf{\lambda }}_{X_i}{\mathsf{\lambda }}_{X_j}\mathrm{and}{\mathsf{\rho }}_{X_i{X}_j}^{*}={\mathsf{\lambda }}_{X_i}^{*}{\mathsf{\lambda }}_{X_j}^{*}$$

Hence, we can estimate the factor loadings ${\mathsf{\lambda }}_{X_1}$ and ${\mathsf{\lambda }}_{X_1}^{*}$ of the first item $X_1$ as (see [26])

$${\mathsf{\lambda }}_{X_1}=\sqrt{\frac{{\mathsf{\rho }}_{X_1{X}_2}{\mathsf{\rho }}_{X_1{X}_3}}{{\mathsf{\rho }}_{X_2{X}_3}}}$$

We now derive the estimate based on polychoric correlations. By using the approximation (4), we obtain

$${\mathsf{\lambda }}_{X_1}^{*}=\frac{{\mathsf{\nu }}_{1,X_1{X}_2}\sqrt{{\mathsf{\lambda }}_{X_1}{\mathsf{\lambda }}_{X_2}}(1-{\mathsf{\nu }}_{2,X_1{X}_2}\sqrt{{\mathsf{\lambda }}_{X_1}{\mathsf{\lambda }}_{X_2}}){\mathsf{\nu }}_{1,X_1{X}_3}\sqrt{{\mathsf{\lambda }}_{X_1}{\mathsf{\lambda }}_{X_3}}(1-{\mathsf{\nu }}_{2,X_1{X}_3}\sqrt{{\mathsf{\lambda }}_{X_1}{\mathsf{\lambda }}_{X_3}})}{{\mathsf{\nu }}_{1,X_2{X}_3}\sqrt{{\mathsf{\lambda }}_{X_2}{\mathsf{\lambda }}_{X_3}}(1-{\mathsf{\nu }}_{2,X_2{X}_3}\sqrt{{\mathsf{\lambda }}_{X_2}{\mathsf{\lambda }}_{X_3}})}.$$

(15)

By rearranging terms in (15), we get

$${\mathsf{\lambda }}_{X_1}^{*}={\mathsf{\lambda }}_{X_1}{\mathsf{\nu }}_{X_1}^{*}\frac{(1-{\mathsf{\nu }}_{2,X_1{X}_2}\sqrt{{\mathsf{\lambda }}_{X_1}{\mathsf{\lambda }}_{X_2}})(1-{\mathsf{\nu }}_{2,X_1{X}_3}\sqrt{{\mathsf{\lambda }}_{X_1}{\mathsf{\lambda }}_{X_3}})}{1-{\mathsf{\nu }}_{2,X_2{X}_3}\sqrt{{\mathsf{\lambda }}_{X_2}{\mathsf{\lambda }}_{X_3}}},$$

(16)

where ${\mathsf{\nu }}_{X_1}^{*}=\frac{{\mathsf{\nu }}_{1,X_1{X}_2}{\mathsf{\nu }}_{1,X_1{X}_3}}{{\mathsf{\nu }}_{1,X_2{X}_3}}$. If the quadratic terms involving ${\mathsf{\nu }}_2$ can be neglected, we get from (16) the approximation

$${\mathsf{\lambda }}_{X_1}^{*}\simeq {\mathsf{\lambda }}_{X_1}{\mathsf{\nu }}_{X_1}^{*}$$

Since ${\mathsf{\nu }}_{X_1}^{*}$ will typically be larger than 1, estimated factor loadings based on polychoric correlations will be larger than those obtained with Pearson correlations. This finding is also in agreement with the findings of Simulation Study 2 in Section 3.2. The factor correlation $\mathsf{\psi }$ is no longer be uniquely determined in the case of unequal loadings. However, one can imagine that the finding from the case of equal loadings would still be valid if reasonable assumptions would be used in a derivation. Because our analytical treatment has only illustrative character, we also do not think that a more rigorous treatment would provide more insights.

4.3. Case 3: General CFA with Simplifying Linear Assumptions

We now consider the general CFA model but make simplifying assumptions on the relationship of polychoric and Pearson correlations. In (4), we empirically demonstrated that there is a quadratic relation of $\sqrt{{\mathsf{\rho }}^{*}}$ and $\sqrt{\mathsf{\rho }}$. Now, we further assume that the quadratic coefficient ${\mathsf{\nu }}_2$ can be neglected. A linear relationship translates into a quadratic relationship ${\mathsf{\rho }}^{*}={\mathsf{\nu }}_1^2\mathsf{\rho }$ of ${\mathsf{\rho }}^{*}$ and $\mathsf{\rho }$. We also impose the additional assumption that

$${\mathsf{\nu }}_{1,Y_i{Y}_j}={\mathsf{\kappa }}_{Y_i}{\mathsf{\kappa }}_{Y_j}$$

(17)

for variables $Y_i$ and $Y_j$. This means that the strength of the linear relationship is determined by parameters ${\mathsf{\kappa }}_{Y_i}$ referring to the marginal distributions. This includes the special case that all marginal distributions in $\mathit{Y}=(Y_1,\dots ,Y_I)$ coincide such that ${\mathsf{\nu }}_1={\mathrm{\kappa }}^2$ with a common parameter $\mathrm{\kappa }$. Now, assume that the factor model holds for Pearson correlations:

$$\mathbf{\Sigma }=\mathbf{\Lambda }\mathbf{\Psi }{\mathbf{\Lambda }}^{\top }+\mathbf{\Theta }.$$

Denote by ${\mathsf{\lambda }}_i^{\top }$ the i-th row vector in $\mathbf{\Lambda }$. Because $\mathbf{\Sigma }$ is a correlation matrix, we get the identity ${\mathbf{\lambda }}_i^{\top }\mathbf{\Psi }{\mathbf{\lambda }}_i+{\mathsf{\theta }}_i=1$, where $\mathsf{\theta }$ is the i-th diagonal element in $\mathbf{\Theta }$. Using Equation (17), we have ${\mathsf{\rho }}_{Y_i^{*}{Y}_j^{*}}={\mathsf{\kappa }}_{Y_i}{\mathsf{\kappa }}_{Y_j}{\mathsf{\rho }}_{Y_i{Y}_j}$. Hence, we obtain

$${\mathsf{\rho }}_{Y_i^{*}{Y}_j^{*}}={\mathsf{\kappa }}_{Y_i}{\mathsf{\kappa }}_{Y_j}{\mathsf{\rho }}_{Y_i{Y}_j}={\mathsf{\kappa }}_{Y_i}{\mathsf{\kappa }}_{Y_j}{\mathbf{\lambda }}_i^{\top }\mathbf{\Psi }{\mathbf{\lambda }}_i={{\mathbf{\lambda }}_i^{*}}^{\top }{\mathbf{\Psi }}^{*}{\mathbf{\lambda }}_j^{*},$$

where ${\mathbf{\lambda }}_i^{*}={\mathsf{\kappa }}_{Y_i}{\mathbf{\lambda }}_i$ and ${\mathbf{\Psi }}^{*}=\mathbf{\Psi }$. Moreover, define the i-th diagonal element in ${\mathbf{\Theta }}^{*}$ by ${\mathsf{\theta }}_i^{*}=-{\mathsf{\theta }}_i({\mathsf{\kappa }}_{Y_i}-1){\mathbf{\lambda }}_i^{\top }\mathbf{\Psi }{\mathbf{\lambda }}_i$. Consequently, factor loadings are multiplied by a factor, and factor correlations remain constant. This finding resembles those in simulation studies 1 and 2 in which there is no strong deviation from the linearity assumption of $\sqrt{{\mathsf{\rho }}^{*}}$ and $\sqrt{\mathsf{\rho }}$. Because the parameters from the ordinal CFA model (i.e., ${\mathbf{\Lambda }}^{*}$, ${\mathbf{\Psi }}^{*}$, and ${\mathbf{\Theta }}^{*}$) can be computed from the CFA based on Pearson correlations (i.e., $\mathbf{\Lambda }$, $\mathbf{\Psi }$, and $\mathbf{\Theta }$), deciding among the Pearson or the polychoric CFA model cannot be made based on model fit because either both or none of the two models will fit the data.

5. Discussion and Conclusions

This article argues that the often found recommendation for not treating ordinal variables in factor models and structural equation models as continuous is not justified. The choice for a particular modeling strategy implies that it is assumed whether the factor analysis model holds for Pearson correlations or polychoric correlations. We demonstrated that the choice could not be based on simulation studies, although some psychometric literature argues otherwise.

Our simulation studies and analytical findings illustrate that always treating items as ordinal by employing a polychoric correlation can result in biased factor loadings and factor correlations if the factor model holds for Pearson correlations. However, and in coherence with the literature, one can also show that biases were obtained when incorrectly treating ordinal variables as continuous. Hence, the discussion about potential biases relies on how the true parameters are defined. It is up to a researcher whether the factor model should be imposed on Pearson correlations or polychoric correlations. The widespread statements about the bias of model parameter when treating items as continuous instead of ordinal are therefore unjustified.

Our simulation studies and analytical derivations show that the factor loadings based on polychoric correlations are typically larger than those based on Pearson correlations. As argued in this article, loadings based on Pearson correlations are often characterized as negatively biased in the literature. The loadings are particularly small if the marginal distribution of the ordinal items is (strongly) asymmetrically distributed. However, we tend to argue that loadings based on Pearson correlations tell the researcher the right imprecision that those items provide limited information on the latent variable because the distributions of ordinal items are not well aligned with the center of the distribution of the latent factor. In item response theory, the information function can quantify such a decrease. Consequently, there are well-justified arguments for using factor loadings based on Pearson correlation and item-total correlations instead of loadings based on polychoric correlations and point-biserial correlations.

In applied research, one is frequently confronted with the argument that the normal distribution assumption is violated for ordinal items. Consequently, factor analysis must be performed based on polychoric correlations. However, it must be emphasized that the ordinal treatment presupposes that the underlying latent variables are normally distributed. It has been shown that this latent normality assumption can be tested. However, other distributional assumptions are seldom applied in research practice. In more detail, the probabilities $\mathrm{P}(Y_i\le k)=G_i({{\mathbf{\lambda }}_i^{*}}^{\top }\mathit{F}+{\mathsf{\nu }}_{ik}^{*})$ using a vector of factors $\mathit{F}$ can be modeled using any monotone linking function $G_i$. Using the probit link function $G_i=\mathsf{\Phi }$ when modeling polychoric correlations is mainly justified by computational simplicity and not model fit arguments or substantive reasons. In item response theory, the link functions $G_i$ can be more flexibly estimated using shape parameters that describe asymmetry. As we have argued elsewhere, the distributional modeling assumptions of normality (when using Pearson correlations) and latent normality (when using polychoric correlations) can be both misspecified, and there should be no general preference for one modeling strategy over the other.

Finally, in social science research, we frequently find the argument that discrete items are only “measured” at an ordinal scale level. Because variables would be attributed to an ordinal scale, only particular statistical techniques would be meaningful (i.e., defensible). We generally find such statements flawed because, in our view, the meaningfulness of a statistical operation is not a property of a variable in general but has to be separately evaluated for all statistical analyses involving this variable. Moreover, we think that the concept of ordinal scales in representational measurement theory cannot be connected to empirically testable consequences. Hence, their set of axioms is entirely unrelated to substantive theory that motivates the use of particular items and, hence, the measurement instrument as a whole. It might be wise to abandon the concept of ordinal and interval scales in social science research and from the introductory methodology course because its theory does not have foundations connected to the application of statistical techniques.