Validity Evidence for the Internal Structure of the Maslach Burnout Inventory-Student Survey: A Comparison between Classical CFA Model and the ESEM and the Bifactor Models

Abstract

Academic burnout is a psychological problem characterized by three dimensions: emotional exhaustion, depersonalization, and personal accomplishment. This paper studies the internal structure of the MBI-SS, the most widely used instrument to assess burnout in students. The bifactor model and the ESEM approach have been proposed as alternatives, capable of overcoming the classical techniques of CFA to address this issue. Our study considers the internal structure of the MBI-SS by testing the models most frequently referenced in the literature, along with the bifactor model and the ESEM. After determining which model best fits the data, we calculate the most appropriate reliability index. In addition, we examined the validity evidence using other variables, namely the concurrent relationships with depression, anxiety, neuroticism, and conscientiousness, and the discriminant relationships with the dimensions of engagement, extraversion, and agreeableness. The results obtained indicate that the internal structure of the MBI-SS is well reflected by the three-factor congeneric oblique model, reaching good values of reliability and convergent and discriminant validity. Therefore, when the scale is used in applied contexts, we recommend considering the total scores obtained for each of the dimensions. Finally, we recommend using the omega coefficient and not the alpha coefficient as an estimator of reliability.

1. Introduction

Academic burnout has traditionally been defined as a psychological problem arising from continual exposure to stressors related to the educational institution and to study activities. The syndrome is usually characterized by three dimensions: emotional exhaustion, depersonalization, and low personal accomplishment [1,2,3]. In the academic context, emotional exhaustion (EE) refers to feelings of stress related to the educational center, particularly chronic fatigue. Depersonalization (D) is manifested as an indifferent or distant attitude towards academic tasks and a view of studies as meaningless. Low personal accomplishment (PA) alludes to perceived inefficacy when studying, lack of academic success, and scant benefit obtained from the study course [4].

Academic burnout can have serious physical and psychological consequences for students’ health [5], for example, provoking sleeplessness, depression, low self-esteem, poor academic performance, absenteeism, and dropout. With a prevalence of 2–41% [5,6], the syndrome continues to be a significant social problem, calling for further study and better understanding.

1.1. The Maslach Burnout Inventory: Internal Structure Validity Evidence and Reliability

Various measurement instruments have been developed to assess burnout syndrome among the working population, among which the Maslach Burnout Inventory (MBI) [7] is the most widely used [8,9,10]. Currently, three versions are available: MBI-Human Services Survey (HSS), MBI-Educators Survey (ES), and MBI-General Survey (GS). However, few such instruments specifically target the university population. To our knowledge, only the MBI-Student Survey [4], the Granada Burnout Questionnaire for university students (CBG-US) [5,11], and the Student Burnout Inventory [12] have been developed. The MBI-SS was created as an adaptation of the MBI-GS [13], and measures the three dimensions of burnout established for the MBI (EE, D, and PA). This instrument is the most widely used to assess burnout in students [14,15], and has been adapted for use in numerous other linguistic populations, namely Portuguese, Dutch, Spanish [4], Brazilian [14], Italian [16], French [17], Chinese [18], Iranian [19], Turkish [20], Colombian-Spanish [21], Serbian [22], Hungarian [23], and Sri Lankan [15].

Most studies of the psychometric properties of the MBI-SS have focused on its internal structure and reliability. However, those seeking evidence of internal structure validity have obtained mixed results. In some cases, the three-dimensional structure of the original scale has been replicated [14,20,22], but in others, the solution obtained is unsatisfactory [17] and/or different from the original due to modifications in the specification of the model, for example, changes in the number of factors, the elimination of items with psychometric problems, or the specification of correlations between item error variances in order to improve the overall fit, and specification of orthogonal factors [15,16,18,19].

In addition to the three-dimensional model, other models have been proposed, based on the results of empirical investigations, mainly performed on the MBI-HSS and the MBI-GS. Thus, [10] presented a hierarchical model with three first-order factors (EE, D, and PA) and a second-order general factor (burnout). This model, motivated especially by the strong correlation observed between the three factors, can be integrated into the original theoretical proposal of [2]. In addition, a two-factor model has been proposed, excluding PA, which is viewed as a non-nuclear component of the syndrome [14,24,25]. In another two-factor model, EE and D form a single factor (the burnout core component), with PA as the second factor [26,27,28]. Finally, models with four [29] and even five factors [30] have been proposed for the MBI-HSS, but not for the MBI-GS or the MBI-SS, since these have different numbers of questionnaire items.

As concerns reliability evidence for the MBI-SS, the vast majority of studies use the alpha coefficient [31], which requires compliance with assumptions such as unidimensionality, tau (τ)-equivalence, and normality of the distribution of the items. These assumptions are rarely verified and, when they are, seldom satisfied [32,33,34]. When the alpha coefficient assumptions are not met, this index tends to underestimate the true reliability of the scale. For this reason, to deal with congeneric scales (which do not satisfy the assumption of τ-equivalence), the omega coefficient is usually recommended [35]. Furthermore, the correct coefficient must be calculated since there are different types of omega coefficients (total, hierarchical, and subscale, among others), the choice of which depends on the type of model proposed (e.g., single or multi-factor). In this context, none of the studies previously conducted to estimate the validity of the MBI-SS have tested τ-equivalent models or have used omega as a reliability estimator.

1.2. The Bifactor Model

The bifactor model was proposed by [36] and has been discussed in detail in various papers [37,38]. In this model, each item depends on two or more orthogonal factors: a general factor and one or more group factors that characterize a specific subset of items.

Formally, the factor analysis model (whether exploratory or confirmatory) can be represented with the generic matrix expression [39]:

Y = ΛX + ΨE

(1)

where Y is an n × 1 random vector of observed random variables (responses to items); Λ is an n × r factor-pattern matrix (factor loadings); X is an r × 1 random vector of latent common factors (factor scores); Ψ is an n × n diagonal matrix of unique-factor-pattern loadings (residual variances or uniquenesses); and E is an n × 1 random vector of latent unique-factor variables (residual scores). In confirmatory factor analysis (CFA), not all items are forced to load on all factors; residual variances may be correlated, and restrictions can be made for items, for example, all items may be forced to load equally on the same factor. Equation 1 can also be expressed as follows [40]:

Y = Λ_yη + ε

(2)

where Y is an observed variable, Λy are the coefficients describing the effects of the latent variables on the observed variables, η is a latent factor score, and ε is the measurement error (uniqueness) that can be decomposed into two terms, such as ε = s + e, where s represents the specific variance associated with each variable and e is the remaining random component in Y.

From Equation (2), specific models such as the three oblique factors and the bifactor can be represented. Thus, in the case of the MBI factors (EE, D, and PA), the first of these would be expressed as:

Y = λ_EEη_EE + λ_Dη_{D +} λ_PAη_PA + ε

(3)

in which COV(ηj, ηj) = ψ, and where the bifactor model (with the addition of a general factor, G) is expressed as follows [38]:

Y = λ_Gη_G + λ_EEη_EE + λ_Dη_{D +} λ_PAη_PA + ε

(4)

where COV(ηj, ηj) = 0. For Equations (3) and (4), η are latent factor scores, and λ are standard factor loadings.

With the bifactor model, we can determine whether the responses obtained by a measurement instrument are essentially unidimensional. The term essential unidimensionality refers to structures in which the general factor (i.e., the variance element that is common to all items) dominates in the presence of a certain degree of multidimensionality reflected by the group factors [38]. This is a great advantage in contexts in which it is unlikely to find models that are purely unidimensional or strictly multidimensional, that is, where there are no correlations between the factors.

The bifactor model is a suitable means of representing multidimensionality due to the construct-relevant multidimensionality of instruments that measure general constructs where different content domains coexist [38]. According to [41], there are at least two sources of construct-relevant psychometric multidimensionality: one refers to the hierarchical nature of the construct and the other reflects the fallible nature of the indicators. In consequence, the bifactor model is appropriate for assessing the hierarchical nature of the constructs [41].

In other words, the value of the bifactor model lies in its ability to determine unidimensionality in the presence of multidimensionality and, moreover, to detect relevant (or irrelevant) group factors in the presence of essential unidimensionality. These two potentialities cannot be addressed through classical CFA models, such as unifactorial or correlated factor models [42]. For example, the CFA correlated factor model is subject to significant cross-loads, which reflects the fact that, to a certain extent, multidimensionality is not perfect or unequivocal, and, therefore, cannot be directly evaluated by interpreting the correlation between factors. Although these questions can be addressed via modification indices, these do not allow us to consider essential unidimensionality; furthermore, the danger exists that atheoretical re-specifications may be introduced into the models to improve the fit [43].

1.3. Exploratory Structural Equation Modelling

The exploratory structural equation modelling (ESEM) technique, proposed by [44] as an alternative to classical CFA, specifies that all items load on all factors (unrestricted model), as would be done in an exploratory factor analysis (EFA), but with a confirmatory technique as in CFA. Formally, ESEM can be represented with the following equations [44]:

Y = ν + Λη + ΚX + ε

(5)

η = α + Βη + ΓX + ζ

(6)

in which there are p dependent variables Y = (Y1, …, Yp), q independent variables X = (X1, …, Xq), and m latent variables η = (η1, …, ηm). The standard assumptions of this model are that the ε and ζ residuals are normally distributed with mean 0 and variance covariance matrix θ and ψ, respectively. Equation (5) represents the measurement model where ν is a vector of intercepts, Λ is a factor loading matrix, η is a vector of continuous latent variables, K is a matrix of Y on X regression coefficients, and ε is a vector of residuals for Y. Equation (6) represents the latent variable model where α is a vector of latent intercepts, B is a matrix of η times η regression coefficients, Γ is a matrix of η times X regression coefficients, and ζ is a vector of latent variable residuals.

ESEM was proposed as a means of overcoming the problems encountered with classical CFA models, which often fit the data poorly [44], meaning that models generated with EFA cannot be confirmed using CFA [45]. This problem is partly due to the fact that the classical CFA specification, in which all cross-loadings are set to zero, is unrealistic [44,45]. In general, the measurement instruments used in this context do not have pure items with a single construct [41,45], but present cross-loadings with other constructs or latent variables [44,45]. When zero loadings are misspecified by classical CFA, this can produce distorted factors, and often leads to overestimated factor correlations [44].

ESEM provides a modelling framework that can be considered a generalization of EFA. Both approaches specify unrestricted factor models that can test whether an item loads on the hypothesized factor, using target rotation, and can check the fit of the model to the data, using the chi-square test and fit indices [44]. However, in addition, ESEM has greater modelling flexibility because, among other attributes, it provides local measures of parameter fit, characterizes correlated residuals and enables structural and measurement invariance to be tested. Moreover, it can be incorporated into larger structural models, or into models with method factors, covariates and direct effects, among other features [44,45].

ESEM can also be considered a generalization of CFA in that it specifies an unrestricted model in which all cross-loads are estimated, while the latter specifies a restricted model in which all or most cross-loads are set to zero. In fact, formal tests can be performed to compare the two models [44,45]. Furthermore, despite the loss of parsimony (presenting fewer degrees of freedom and with more parameters to be estimated), ESEM is capable of accurately recovering the factorial structure of population models made up of independent clusters, such as the oblique multifactorial solutions that are typical of classical CFAs [41].

The advantage of ESEM is that it can model one of the two sources of construct-relevant psychometric multidimensionality, namely that which is due to the fallible nature of the items [41], i.e., the fact that the items are rarely pure indicators of the construct to be measured. On the one hand, they contain a degree of measurement error, which is modelled by the error variances in classical CFA models. On the other hand, they present a systematic association with other constructs, which is usually apparent in the form of cross-loadings. ESEM incorporates these cross-loadings, thus making the model constraints more realistic and achieving unbiased factor loadings and factor correlations.

1.4. Limitations of Classical CFA Applied to the MBI and Advantages of Bifactor and ESEM Models

Since the MBI was first presented, numerous studies have examined the internal structure of its different versions, mainly using EFA and CFA to do so. In their systematic review, [10] identified 35 applications of EFA and 28 of CFA. Given the current popularity of structural equation modelling, there are now probably many more applications of CFA than of EFA. Regardless of the technique used, studies have yielded conflicting solutions. According to [10], most EFA applications obtain a three-factor solution, but 25% do not. Regarding CFA applications, 90% replicate the three-factor oblique model. However, of these, 58% introduce some form of re-specification into the model, seeking to improve the global fit indices (for example, by eliminating items or by specifying correlated error variances and cross-loadings). An important consideration is that using modification indices and other ad hoc strategies to respecify the model can produce results that are misleading (for example, confirming a structure that had not previously been hypothesized) or simply incorrect (for example, removing items that are necessary to properly represent the construct) [46]. Furthermore, in some studies, models are retained in accordance with criteria that fail to meet the minimum requirements for deeming the fit to be acceptable [47]; this shortcoming was again observed in a later study focused on the MBI-SS [17]. In short, many studies fail to replicate the original structure of the MBI when classical CFA is applied.

These results are in line with the conclusions drawn in previous reviews of the literature on factor analysis, which have observed that it is fairly common to find factor structures that are not repeated in a subsequent CFA, because the specification of these models is usually unrealistic, especially when multidimensional instruments are involved [41,48]. Furthermore, many studies conclude that the tested model fits the data well (and is therefore retained), despite the fact that its global fit indices do not meet the minimum criteria established for an acceptable fit [45,48].

In view of the debates that have arisen on the structure of the MBI and acknowledging the difficulties encountered with classical CFA models in achieving an acceptable fit, especially with multidimensional instruments, we believe that both the bifactor model and the ESEM can be considered useful methodological tools with which to clarify some of the questions posed regarding the internal structure of the MBI. The authors of [49] were among the first to apply the bifactor model to the MBI-HSS, finding it to obtain the best fit of the options considered. Subsequently, other researchers have tested the bifactor model, either with the MBI-HSS [50,51] or with the MBI-ES [52,53]. However, to our knowledge, none have used the bifactor model to address the MBI-SS. Neither have any such studies used ESEM to study the internal structure of any version of the MBI. To date, the only analysis conducted in this area has been that of Biachi et al., who used ESEM to study the overlap between burnout, depression, and anxiety [54,55,56].

1.5. Objectives

Due to the above-mentioned disparities in empirical results, no consensus has yet been reached on the internal structure of the MBI-SS. The techniques commonly used to address this question, which in many cases is that of classical CFA, are subject to limitations in determining the possible reasons for a repeated failure to obtain a good fit (such as cross-loadings or the importance of a general factor). The bifactor model and the ESEM approach have been proposed as alternatives, capable of overcoming these limitations. For example, the bifactor method has helped clarify the internal structure of both the MBI-HSS and the MBI-ES. However, neither of these models has yet been used to target the MBI-SS. Our study, therefore, considers the internal structure of the MBI-SS in a sample of Spanish undergraduates by testing the models most frequently referenced in the literature, together with the bifactor model and ESEM. After determining which model best fits the data, we then calculate the most appropriate index of reliability. In addition, we examine the evidence of validity using other variables, specifically concurrent relationships with depression, anxiety, neuroticism, and conscientiousness, and discriminant ones with the dimensions of engagement, extraversion, and agreeableness.

2. Method

2.1. Participants

The study sample was composed of 1162 students recruited at various Spanish universities by non-probabilistic sampling. Of these participants, 64.7% were female and, overall, their mean age was 20.9 years (SD = 1.92).

2.2. Procedure

The study data were collected, using the same procedure in every case, during the second quarter of 2018. The process took place in the classroom and during academic hours, with the approval of the university staff involved. All students gave prior informed consent to participate and were assured confidentiality and anonymity.

2.3. Instruments

All participants completed an ad hoc sociodemographic data questionnaire, including their age and sex. The following measuring instruments were administered.

• The Maslach Burnout Inventory-Student Survey (MBI-SS), adapted for Spanish speakers [57]. This questionnaire contains 15 items scored on a seven-point response scale, to measure the three dimensions of the syndrome stipulated in the original proposal by [2]; emotional exhaustion (depletion of psychological and emotional resources); depersonalization (feelings of cynicism and detachment); and scant personal accomplishment (feelings of ineffectiveness and inadequate performance).
• The Utrecht Work Engagement Scale (UWES) [4]: composed of 24 items scored on a seven-point response scale, to measure the three dimensions of engagement: absorption (full concentration and placid immersion in one’s own tasks), dedication (commitment to one’s own tasks, recognition of their importance, and enthusiasm) and vigor (energy and mental resilience).
• Four of the five dimensions in the Spanish version of the NEO Five Factor Inventory (NEO-FFI) [58]: neuroticism, extraversion, conscientiousness, and agreeableness. Each scale consists of 12 items, scored on a five-point Likert response format.
• The depression and anxiety dimensions of the Educational-Clinical Questionnaire: Anxiety and Depression (CECAD) [59]. This questionnaire consists of 50 items with a five-point Likert-type response format. It produces a global evaluation of emotional disorders, based on the scores obtained for six dimensions: depression, anxiety, uselessness, irritability, problematic thoughts, and psychophysiological symptoms.

2.4. Data Analysis

All statistical analyses were performed with R 4.2.1. (R Core Team, Vienna, Austria, 2022), using the lavaan package [60] for factor analyses and based on the unbiased variance-covariance matrix, since this sample statistic has better statistical properties than its biased version. The parameter estimation method used was robust maximum likelihood (MLR), in view of the number of response categories established, the multivariate normality test performed and the asymmetry and kurtosis indices obtained. Target rotation was used in ESEM. Missing values were dealt with by the full information maximum likelihood (FIML) procedure.

The internal structure of the MBI-SS was assessed using the following models: (a) one congeneric factor: a one-dimensional model in which all items freely load on a single general burnout factor; (b) two congeneric factors: a model in which the items load freely on a factor composed of EE and D, where PA is the other factor; (c) two congeneric factors without PA: a model in which the items load freely in the EE and D dimensions, but from which the items of the PA dimension are excluded; (d) three congeneric factors: the model proposed by Maslach et al. [7,13]; (e) a hierarchical model with three first-order factors (EE, D and PA) plus a second-order general factor (burnout); (f) a bifactor model in which the items load freely on each of the three group factors (EE, D and PA) and also on a general factor (burnout); (g) ESEM: a model in which all the items load freely on the three factors, using a Target rotation in accordance with the model proposed by Maslach et al. [7,13].

Additionally, a τ-equivalent specification was tested in the three-factor model, that is, imposing the restriction that the items for each of the factors should have the same factor loading. This requirement was made for two reasons: (a) the three-factor model is the original form and the one most commonly employed; (b) the alpha coefficient is the index that is normally used to estimate reliability, and this measure requires compliance with the τ-equivalence assumption.

The measurement models were assessed using the χ² statistic and the following global fit indices [61,62]: Tucker–Lewis index (TLI), comparative fit index (CFI), root mean square error of approximation (RMSEA) and standardized root mean squared residual (SRMR). For CFI and TLI, values above 0.90 or 0.95 are considered adequate, whereas for SRMR, values below 0.08 are acceptable [63]. RMSEA values below 0.06 are reasonable [63] while those below 0.05 are considered evidence of a satisfactory fit [64]. For all the measurement models that produced adequate fit indices, likelihood-ratio tests were performed to determine whether the difference between the log-likelihoods was statistically significant, and to calculate the difference between these and the Akaike Information Criteria (AIC). When the fit of a more complex model was significantly better than that of a simpler model and, at the same time, the estimated factor loadings were high enough (above 0.30, according to [61]), the more complex model was considered to better represent the internal structure of the MBI-SS. In contrast, when the difference in fit was not statistically significant, the simpler factor structure was retained.

Once it was decided which measurement model best represented the internal structure of the MBI-SS, an appropriate reliability index was chosen and computed. Formulas and references for the reliability indices considered are detailed in Table S2 of the supplementary material. The alpha coefficient was considered for the one-factor τ-equivalent model, and appropriate versions of the omega coefficient for the other measurement models. In the one-factor congeneric model, the total omega, ωt, was used for the whole scale, while each of the corresponding subscales was used with the three-factor model. When a bifactor model was fitted, the hierarchical omega, ωh, was the reliability index that best accounted for the general factor, while the omega subscale, ωs, was used to account for the group factors (EE, D and PA).

In order to assume essential unidimensionality (i.e., the presence of a strong general factor) or essential multidimensionality (the presence of a strong group factor), explained common variance (ECV) values greater than 0.60–0.70 and ωh values greater than 0.70 [65,66,67] are recommended.

A correlation analysis was performed between the mean scores for each factor (and the total), the depression and anxiety scales (CECAD) and the personality factors (NEO-FFI). To estimate the sample size required for this study, an a priori analysis was performed using online software [68]. In this calculation, the minimum expected effect size was r = 0.15, with α = 0.05 and a level of statistical power 1 − β = 0.9, and with a maximum of 4 latent variables and 15 observed variables. The minimum sample size needed to detect the proposed effect was N = 799, which is smaller than the sample size used.

3. Results

3.1. Validity Evidence Based on Internal Structure

Descriptive statistics for the MBI-SS items, together with the inter-item correlations, are shown in Table S1 of the Supplementary Material. Separately,

Table 1. Factor analysis results.

	χ2	df	p	CFI	TLI	RMSEA	90% CI	SRMR	AIC
One-factor	1677.12	90	<0.001	0.640	0.580	0.135	[0.130–0.141]	0.109	61,810.51
Two factors
(EE and D), PA	941.42	89	<0.001	0.809	0.775	0.099	[0.093–0.105]	0.075	60,909.10
EE, D	214.21	26	<0.001	0.928	0.901	0.088	[0.077–0.099]	0.053	38,103.89
Three factors
Congeneric	551.29	87	<0.001	0.896	0.875	0.074	[0.068–0.080]	0.062	60,447.20
τ-equivalent	701.25	99	<0.001	0.866	0.858	0.079	[0.073–0.084]	0.079	60,598.93
Hierarchic	551.29	87	<0.001	0.896	0.875	0.074	[0.068–0.080]	0.062	60,447.20
Bifactor	264.50	75	<0.001	0.958	0.941	0.051	[0.044–0.057]	0.037	60,127.88
ESEM	198.95	63	<0.001	0.969	0.949	0.047	[0.040–0.055]	0.023	60,080.58

df—Degrees of freedom; CFI—Comparative fit index; TLI—Tucker-Lewis index; RMSEA—Root mean square error of approximation; SRMR—Standardised root mean squared residual; AIC—Akaike information criteria; EE—Emotional exhaustion; D—Depersonalisation; PA—Personal accomplishment; ESEM—Exploratory structural equation modelling.

shows the results of the fit obtained for each congeneric model tested. The only models that achieved acceptable fits were the bifactor and the ESEM. Except for the SRMR index, the three-factor and hierarchical models did not produce acceptable fit indices. Neither the one- nor the two-factor models (EE + D and PA) achieved an acceptable fit to the data. However, the two-factor model without PA achieved an acceptable fit for all indices except RMSEA. Although the three-factor congeneric model did not achieve an acceptable fit, that of the τ-equivalent version of the model is presented. This model did not achieve acceptable fit indices (see Table 1) and the comparison with the congeneric model showed that the latter fitted the data better (χ²diff(12) = 151.28, p < 0.001). In the chi-square difference tests, the three-factor model was compared with the bifactor and the latter with the ESEM. In this comparison, the bifactor was better than the three-factor model (χ²diff(12) = 217.26, p < 0.001) while the ESEM outperformed the bifactor (χ²diff(12) = 69.68, p < 0.001).

Table 2. Factor loadings, correlations between factors, and reliability indices.

Item	Three Congeneric Factors			Three τ-Equivalent Factors			Bifactor				ESEM
	EE	D	PA	EE	D	PA	EE	D	PA	General	EE	D	PA
EE1	0.648	0.000	0.00	0.661	0.000	0.000	0.382	0.000	0.00	0.503	0.472	0.248	0.070
EE2	0.593	0.000	0.00	0.613	0.000	0.000	0.480	0.000	0.00	0.373	0.588	0.057	0.058
EE3	0.579	0.000	0.00	0.618	0.000	0.000	0.491	0.000	0.00	0.343	0.580	0.011	0.125
EE4	0.560	0.000	0.00	0.623	0.000	0.000	0.662	0.000	0.00	0.201	0.701	0.069	0.078
EE5	0.770	0.000	0.00	0.676	0.000	0.000	0.503	0.000	0.00	0.550	0.591	0.239	0.096
D1	0.000	0.566	0.00	0.000	0.679	0.000	0.000	0.465	0.00	0.663	0.262	0.402	0.068
D2	0.000	0.551	0.00	0.000	0.653	0.000	0.000	0.083	0.00	0.561	0.157	0.476	0.004
D3	0.000	0.800	0.00	0.000	0.743	0.000	0.000	0.001	0.00	0.783	0.002	0.822	0.022
D4	0.000	0.840	0.00	0.000	0.742	0.000	0.000	0.001	0.00	0.830	0.068	0.759	0.085
PA1	0.000	0.000	0.528	0.000	0.000	0.586	0.000	0.000	0.475	0.247	0.051	0.031	0.547
PA2	0.000	0.000	0.574	0.000	0.000	0.528	0.000	0.000	0.581	0.206	0.032	0.134	0.667
PA3	0.000	0.000	0.687	0.000	0.000	0.656	0.000	0.000	0.640	0.294	0.030	0.035	0.717
PA4	0.000	0.000	0.722	0.000	0.000	0.688	0.000	0.000	0.574	0.401	0.072	0.142	0.637
PA5	0.000	0.000	0.579	0.000	0.000	0.596	0.000	0.000	0.566	0.235	0.097	0.144	0.673
PA6	0.000	0.000	0.578	0.000	0.000	0.622	0.000	0.000	0.319	0.523	0.080	0.419	0.355
α	-	-	-	0.772	0.781	0.778	-	-	-	-	-	-	-
ωh	-	-	-	-	-	-	0.485	0.001	0.579	0.649	-	-	-
ωt	0.768	0.790	0.777	0.772	0.794	0.777	0.632	0.074	0.701	0.767	0.725	0.719	0.775
ECV	-	-	-	-	-	-	0.605	0.138	0.717	0.516	-	-	-
rCE	1	-	-	1	-	-	1	-	-	0.000	1	-	-
rD	0.651	1	-	0.669	1	-	0.000	1	-	0.000	0.376	1	-
rPA	−0.287	−0.528	1	−0.277	−0.538	1	0.000	0.000	1	0.000	−0.094	−0.438	1

Note. ωh refers to ωs when it is applied to the subscales (EE, D and PA). ECV—Explained common variance.

details the results obtained by the three-factor oblique congeneric, the three-factor oblique τ-equivalent, the bifactor, and the ESEM models. The path diagrams for these models are shown in Figure 1, Figure 2, Figure 3 and Figure 4. In the three-factor congeneric, three-factor τ-equivalent and ESEM models, the factor loadings were adequate, i.e., above 0.30 [61]. This finding is especially significant for the ESEM, since the items can load on any factor. In fact, in this model, the factor loading for item RP6 was higher in D than in PA. In the bifactor model, adequate loadings were obtained in all the EE items (although those for EE1 and EE5 were higher on the general factor) and for PA (although the loading of PA6 was higher on the general factor). In D, the loadings of three of the four items were low. In the general factor, low loads were obtained on EE2 and on PA1, PA2, PA3, and PA5.

Evaluation of essential unidimensionality showed that the ECV and the hierarchical omega indices, ωh, were below the recommended values for the general factor. Regarding dimensionality, the EE and PA factors obtained satisfactory indices of ECV but not of ωs. For D, none of the indices obtained a good value. In both the three-factor congeneric and the τ-equivalent models, factor correlations were high between D and EE and between D and PA, and moderate between PA and EE. Correlations were lower (low-moderate), however, in ESEM. In the bifactor model, the correlations were set to zero in the specification.

3.2. Reliability

The reliability estimates for all the factors in the three-factor oblique congeneric, three-factor oblique τ-equivalent, bifactor, and ESEM models are shown in Table 2. Different estimators were considered according to the model selected. For example, the alpha coefficient, α, was used for the three-factor τ-equivalent model, but not for the others, since they are congeneric. Total omega, ωt, was calculated for all models, including the τ-equivalent model, since α and ωt are equivalent if the assumptions of the former are met [32,35]. Hierarchical omega, ωh, was used with the bifactor model as an index to estimate the internal consistency of the total test score that is exclusive of the overall factor; the omega subscale, ωs, was applied to the subscales for the same purpose. The formulas for each estimator are listed in Table S2 of the Supplementary Material.

In general, all of the models, except the bifactor, presented good reliability, above the usual recommended value of 0.70 [69]. For EE and PA, the reliability estimates (whether using alpha or omega) of the τ-equivalent model were practically the same as those of the congeneric model. For D, the reliability estimate of the τ-equivalent model was somewhat lower than that of the congeneric model. The ESEM solution produced slightly lower coefficients. With the bifactor model, only PA and the general factor obtained an ωt value above 0.70. As mentioned above, the values for ωh and ωs were below the recommended minimum.

3.3. Validity Evidence Based on Relations to Other Variables

Table 3. Descriptive statistics and correlations between the study variables.

Item	M	SD	1	2	3	4	5	6	7	8	9	10	11	12
1. EE	13.74	6.25	0.77
2. D	7.19	5.50	0.52 ***	0.79
3. PA	26.98	5.72	−0.21 ***	−0.41 ***	0.78
4. VI	31.95	7.81	−0.24 ***	−0.32 ***	0.64 ***	0.73
5. DE	37.22	9.25	−0.27 ***	−0.63 ***	0.73 ***	0.65 ***	0.92
6. AB	25.59	7.00	−0.12 ***	−0.29 ***	0.65 ***	0.71 ***	0.63 ***	0.74
7. DP	46.39	13.70	0.43 ***	0.26 ***	−0.29 ***	−0.22 ***	−0.19 ***	−0.13 ***	0.94
8. AN	40.74	11.11	0.51 ***	0.30 ***	−0.22 ***	−0.16 ***	−0.18 ***	−0.06 **	0.74 ***	0.89
9. NE	32.30	7.70	0.38 ***	0.20 ***	−0.25 ***	−0.22 ***	−0.15 ***	−0.12 ***	0.72 ***	0.60 ***	0.81
10. CO	43.00	5.52	−0.06 **	−0.24 ***	0.49 ***	0.39 ***	0.40 ***	0.45 ***	−0.25 ***	−0.13 ***	−0.19 ***	0.80
11. AG	43.19	5.87	−0.17 ***	−0.24 ***	0.27 ***	0.25 ***	0.27 ***	0.23 ***	−0.24 ***	−0.16 ***	−0.25 ***	0.21 ***	0.69
12. EX	45.51	7.14	−0.19 ***	−0.27 ***	0.40 ***	0.32 ***	0.36 ***	0.29 ***	−0.40 ***	−0.32 ***	−0.33 ***	0.32 ***	0.29 ***	0.83

Note. EE = Emotional exhaustion; D = Depersonalisation; PA = Personal accomplishment; VI = Vigour; DE = Dedication; AB = Absorption; DP = Depression; AN = Anxiety; NE = Neuroticism; CO = Conscientiousness; AG = Agreeableness; EX = Extraversion. The diagonal contains Omega Total coefficients. *** = p < 0.001; ** = p < 0.01.

shows the descriptive statistics for the total scores of the variables included in the study, together with the Pearson bivariate correlation in each case. In accordance with the results described above and with the theory, we expected to obtain evidence of convergent and discriminant validity with different variables, depending on the burnout dimension considered. Considering the EE and D dimensions, the correlations with depression, anxiety, and neuroticism were positive, moderate, and statistically significant. The correlations with the dimensions of engagement (vigor, dedication, and absorption), conscientiousness, agreeableness, and extraversion were negative and of low or moderate magnitude. The correlations between PA and the dimensions of engagement (vigor, dedication, and absorption), conscientiousness, agreeableness, and extraversion were positive, strong, and statistically significant. In addition, low or moderate negative correlations were obtained with depression, anxiety, and neuroticism.

4. Discussion

The Maslach Burnout Inventory-Student Survey (MBI-SS; [13]) is probably the instrument most widely used to assess academic burnout syndrome [44,45], and has been adapted for use by numerous language groups (e.g., Spanish, Italian, Chinese, and German). According to the original conceptual proposal [7,13], the MBI-SS evaluates three dimensions of burnout: emotional exhaustion (EE), depersonalization (D), and personal accomplishment (PA). The most commonly evaluated psychometric properties of the MBI-SS are its internal structure and reliability. The evidence obtained for its validity, based on the internal structure, has produced mixed results: while some studies have replicated the original structure of three oblique factors [14,20,22], others have obtained different solutions, due to model re-specification [15,16,18,19] or have not achieved an acceptable fit [17]. As concerns the reliability of the instrument, most studies report the alpha coefficient, but this index can underestimate the true reliability when the assumptions of τ-equivalence, unidimensionality and normality in the distribution of the items are not met, as it is often the case with psychological assessment instruments [32,33,34]. A discussion on reliability can be consulted elsewhere [70,71,72,73,74].

To date, evidence of validity has been obtained using EFA or classical CFA (e.g., one factor, three factors or hierarchical), but these present various limitations. Firstly, EFA does not allow the introduction of restrictions in the models, which means that τ-equivalent solutions or those with correlated errors cannot be tested. Moreover, it does not provide local measures of parameter fit, nor can it be incorporated into larger structural models [44]. The classical CFA specification, on the other hand, is often unrealistic in multidimensional instruments since it sets the cross-loadings to zero [41,48]. This type of specification error tends to result in distorted factors, and often produces overestimated factor correlations [44].

Some new approaches to studying the internal structure of measurement instruments have recently been proposed, such as the bifactor model and ESEM. Both of these methods seek to overcome the limitations of classical CFA. The main advantage is that both can explain more sources of variability in the item scores (psychometric multidimensionality), and, therefore, are more realistic, generating less biased results. The authors of [41] highlighted two sources of construct-relevant psychometric multidimensionality: the first concerns the hierarchical nature of the construct, i.e., the expectation that all the items considered will present a significant level of association with their own subscales (for example, each of the items of which EE is composed with the factor as a whole), as well as with hierarchically superior constructs (such as burnout). The second source arises from the fallible nature of the indicators typically used to measure psychological constructs, most of which assess conceptually related and partially overlapping constructs (e.g., burnout and depression).

Despite the advantages offered by the bifactor model and ESEM, to our knowledge, no studies have yet been conducted using either of these approaches to evaluate the internal structure of the MBI-SS. To address this gap, the present study examines the internal structure of the MBI-SS in a sample of Spanish undergraduates by jointly testing the models most frequently cited in the literature, together with the bifactor model and ESEM. Specifically, we analyzed the results obtained by the following models: (a) one congeneric factor: a one-dimensional model in which all items freely load on a single general burnout factor; (b) two congeneric factors: a model in which the items load freely on a factor composed of the EE and D items, with PA being the other factor; (c) two congeneric factors without PA: a model in which the items load freely in the EE and D dimensions, but the items of the PA dimension are excluded from the model specification; (d) three congeneric factors: the model proposed by Maslach et al. [7,13]; (e) hierarchical: a model with three first-order factors (EE, D, and PA) together with a second-order general factor (burnout); (f) bifactor: a model in which the items load freely on each of the three group factors (EE, D, and PA) and on a general factor (burnout); (g) ESEM: a model in which all the items load freely on the three factors, using a Target rotation in accordance with the model presented by Maslach et al. [7,13]. Since the alpha coefficient is the most widely used means of estimating the reliability of the MBI-SS, we also tested the three-factor oblique τ-equivalent model, which requires compliance with the assumption of τ-equivalence.

4.1. Internal Structure of the MBI-SS

According to the results obtained, the only models that achieved an acceptable fit in all of the evaluation indices were the bifactor model and ESEM. In the chi-square difference tests, ESEM was statistically better than the bifactor model. τ-equivalent models do not usually fit the data well unless the congeneric model does so too [61]. However, we tested the fit for the model with three oblique τ-equivalent factors, since this was the only one that satisfied the assumptions for using the alpha coefficient. As expected, the model did not achieve acceptable fit indices, and proved to be statistically worse than the three-factor congeneric model. This result suggests that the alpha coefficient should not be used to estimate the reliability of the MBI-SS.

Regarding the factor loadings of the items, in all models except a few cases with the bifactor model, the recommended level was exceeded. The loadings of the oblique three-factor model were higher than those of the bifactor model and ESEM. This result was expected given that in the three-factor model, the only source of item variability is its factor; however, in the bifactor model, the variability is divided between the general factor and the group factors, while in ESEM, it is divided among the three factors. The fact that acceptable factor loadings are obtained in ESEM indicates that this model adequately explains the variability of cross-loadings while maintaining the structure of three oblique factors. Nevertheless, future studies could further investigate the reasons for the high fluctuations of the loadings in the D. The loadings of the τ-equivalent model were more homogeneous than those of the other models due to the restriction imposed that all the items of the same factor must have the same factor loading. With the bifactor model, the EE and PA items achieved good factor loadings; however, in the general factor and in D, the loadings of some items were less than 0.30 (see Table 2).

The factor loadings obtained with the bifactor model are in line with the results from the evaluation of the essential unidimensionality. Except for PA, none of the factors achieved a satisfactory ωh coefficient. As the ωh index uses factor loadings to estimate reliability (and is interpreted as the percentage of item variance that is explained by the factor, after eliminating the variance explained by the other factors), this result explains the pattern of factor loadings obtained for the different factors. Thus, several items presented significant cross-loadings in the general factor: specifically, four items (of five) in EE, all the items in D, but only two (of six) in PA. Taking the general factor as a reference, 12 items (of 15) showed relevant cross-loadings in one or more of the three group factors. The results for the ECV index are in line with these findings, from which we conclude that the values obtained for EE and, above all, PA are acceptable.

The results obtained with ωh and ECV show that the requirements for the MBI-SS to present essential unidimensionality were not met. These results also show that the most important factors generating multidimensionality (i.e., those which explain a large part of the variance of the items) are EE and PA, while D is incorporated within the general factor.

Regarding the factor correlations, in each of the three-factor models (congeneric and τ-equivalent), the values obtained are in accordance with the theory and with previous research findings [7,13]; specifically, there is a moderate positive correlation between EE and D, while the correlations between PA, EE, and D are weak-moderate and negative. In the bifactor model, the correlations were previously set to zero, that is, it was stipulated that the factors should be orthogonal. In ESEM, the correlations are weaker than with the three-factor oblique models. This result agrees with those obtained for other psychometric evaluations with ESEM and suggests that the correlations obtained with classical CFA models are overestimated [44]. Thus, according to the correlations obtained with this model, EE and D present a positive correlation of intermediate size; EE and PA have a weak negative correlation; and PA and D have a negative correlation of intermediate magnitude. These results for factor correlations may explain the poor fit of the one- and two-factor models (EE + D and PA).

4.2. Reliability of the MBI-SS

Psychometric studies of the MBI-SS have estimated its reliability using the alpha coefficient. In the present research, the alpha value results for the three dimensions of the MBI were above the recommended values [69]. However, this coefficient is not a good estimator of reliability when conditions such as τ-equivalence between the items, the unidimensionality of the scale, and the normal distribution of the item scores are not met [32,33,34]. As discussed above, the tau-equivalent three-factor model did not achieve an acceptable fit, which suggests that the alpha coefficient should not be used for MBI-SS applications. Since this result is common in psychological scales, the omega coefficient is normally recommended as a reliability estimator [35]. An additional advantage of omega over alpha is that it is calculated from the model that achieves a good fit to the data. If alpha was used instead, this measure could suggest a level of model reliability that, ultimately, is not maintained.

The present study is the first to focus on the psychometric properties of the MBI-SS, using omega as a reliability estimator, after fitting by CFA. According to the results obtained, the ESEM model best fitted the data. In this model, the MBI-SS achieved acceptable reliability values for all three factors [69]. The bifactor model also achieved good global indices of fit, but the reliability of the MBI dimensions in this model was not acceptable, except for PA. This result is explained by the low factor loadings of some items on their respective factors and by high ones on the general factor.

Of the oblique three-factor models, neither the congeneric nor the τ-equivalent achieved an acceptable fit to the data. However, since psychometric theory suggests that the reliability of a scale is underestimated by the alpha coefficient when the assumptions are not met, we calculated the reliability of the factors in both models. In this respect, similar values were obtained for EE and PA using the alpha coefficient for the τ-equivalent model and omega for the congeneric model. However, for D, the alpha value was somewhat lower than the omega result. These findings are consistent with the psychometric literature [33]. For PA, the factor loadings in all the models were high, except for item PA6 in the bifactor model and ESEM. As these values are high and similar, the differences between the alpha and omega coefficients are less. This is reflected in the fact that the value of the ECV index for this factor suggested that PA was a relevant group factor. On the other hand, the opposite case was observed for D; in this case, the items obtained very different factor loadings in various models (for example, very low ones in the bifactor model) and the factor obtained a very low ECV index, reflecting the slight relevance of this factor.

4.3. Relation between the MBI-SS and Other Constructs

According to the results obtained, the MBI-SS dimensions showed evidence of convergent and discriminant validity with the constructs examined. Persons who scored more highly in the dimensions of EE and D tended to obtain higher scores, too, for depression, anxiety, and neuroticism. An association was also observed between the MBI-SS and the dimensions of engagement (vigor, dedication, and absorption), conscientiousness, agreeableness, and extraversion. Thus, at higher levels of EE and D, the scores in the latter constructs were lower. As expected, an inverse pattern was obtained between PA and the other variables.

5. Limitations and Future Research

This study presents some limitations. First, the sample selection was not probabilistic, and students were not recruited from all regions of the country. Future studies should confirm the results we present with those from a broader sample. Second, we did not test the τ-equivalent versions for the bifactor and ESEM models. In future consideration of one of these models, it would be interesting to perform these tests, too, to better characterize the differences between the alpha and omega coefficients. Third, the factorial invariance of the model has not been tested. Although the age variable is not relevant in this case because the population considered is very homogeneous, variables such as the sex of the participants should be included in a future analysis. Finally, in this study, the sources of psychometric variability arising from relevant constructs were evaluated separately. On the one hand, we used the bifactor model, which enabled us to determine when it is more appropriate to use a single total score (assuming essential unidimensionality) or when subscale scores should be preferred [42]. On the other hand, we used the ESEM approach to model the cross-loadings that occur due to the fallible nature of the indicators. In future studies, it would be advisable to adjust the ESEM bifactor model [41] to simultaneously explore the two sources of construct-relevant psychometric multidimensionality, that is, the hierarchical nature of the constructs and the fallible nature of the indicators.

6. Conclusions and Practical Recommendations

To our knowledge, the present study is the first to use both the bifactor model and the ESEM approach to study the internal structure of the MBI-SS and to use a reliability estimator consistent with the retained model. Furthermore, it is the first to test tau-equivalent models and compare them with congeneric ones. To sum up, the results obtained indicate that the internal structure of the MBI-SS in Spanish undergraduates is well reflected by the three-factor oblique congeneric model, achieving good values for reliability and convergent and discriminant validity.

Firstly, the results obtained from the bifactor model show that there is insufficient evidence to suggest the essential unidimensionality of the MBI-SS. Therefore, we recommend using the scores for all three dimensions and not a global burnout score. This recommendation is in line with the proposal of [7].

Secondly, according to the ESEM results, the three-factor oblique model fits the data well when cross-loadings are taken into account, that is, accounting for the variability that occurs due to the fallible nature of the indicators. Hence, the results from both models suggest that the measurement model that best represent the data is the three-factor oblique congeneric model.

Therefore, when the scale is used in applied contexts, we recommend considering the total scores obtained for each of the dimensions (emotional exhaustion, depersonalization, and personal accomplishment), and not a global burnout score. Furthermore, when using the MBI-SS for substantive research purposes, such as testing a hypothesis related to the Job Demands-Resources theory [70], the analysis should be based on the ESEM model since this facilitates control of the cross-loadings of the scale and provides unbiased factorial correlations.

Thirdly, all three dimensions of the MBI-SS scored well for reliability. In this respect, similar values were obtained with the alpha coefficient in the three-factor τ-equivalent model and with the omega coefficient in the three-factor congeneric model. However, we recommend reporting the omega values by default since this approach does not need to meet the assumptions of tau-equivalence, item normality, and unidimensionality in order to function correctly.

Finally, the study results we describe provide a new perspective from which to consider the problem of the overlap between the MBI dimensions, especially those of EE and D. Some researchers have proposed a two-factor structure based on the strong correlation between these two dimensions [10]. According to [44], the correlations between the factors obtained with classical CFA tend to be overestimated due to the cross-loading problem. Our study results corroborate this view, and also suggest that the correlation between EE and D is not strong, but rather intermediate. Therefore, combining these two factors, as in the two-factor model proposed by some authors [26,27,28], would not be justified.

In summary, taking all results together, we can conclude that the MBI-SS in Spanish undergraduates is well reflected by the three-factor oblique congeneric model, achieving good values for reliability and convergent and discriminant validity.