Evaluation of Convergent, Discriminant, and Criterion Validity of the Cuestionario Burnout Granada-University Students

Abstract

Burnout is a health problem that affects professionals and students or professionals in training, especially those in health areas. For this reason, it is necessary that it is properly identified to prevent the impact it can have on the work and personal areas of the people who suffer from it. The aim of this work is to study the convergent, discriminant, and criterion validity of the Cuestionario Burnout Granada-University Students. The sample consisted of 463 undergraduate nursing students, selected by non-probabilistic convenience sampling, who participated voluntarily and anonymously in the study. The mean age of the participants was 21.9 (5.12) years, mostly female (74.1%), single (95.8%), and childless (95.6%). Information was collected face-to-face, and the instruments were completed on paper. Comparisons were made in the three dimensions of burnout of the CBG-USS between students with and without burnout, finding statistically significant differences in all three dimensions: Emotional Exhaustion (p < 0.001, d = 0.674), Cynicism (p < 0.001, d = 0.479), and Academic Efficacy (p < 0.001, d = −0.607). The Cuestionario Burnout Granada-University Students presents adequate reliability and validity indices, which demonstrates its usefulness in the identification of burnout. This syndrome has traditionally been measured in professionals, but students also present burnout, so it is necessary to have specific burnout instruments for students, since the pre-work situation and stressors of students are different from those of workers. In order to work on the prevention of university burnout, it is essential to have specific instruments for professionals in training that help in the detection of students with burnout.

1. Introduction

Burnout is a psychological disorder that develops from chronic exposure to stressors. Traditionally, this syndrome has been studied in professions characterized by a strong interaction between professionals and the beneficiaries of their work, especially among workers considered to be at high risk of suffering from the syndrome, such as doctors, nurses, teachers, and police officers. More recently, however, it has also been studied in other population groups, such as informal caregivers, housewives, and university students [1,2,3,4,5,6,7].

University students can be equated with the above workers in terms of their susceptibility to developing burnout syndrome [1,8]. In both cases, there is a relationship with an institution that offers products or services, the incentive for participation being monetary for working professionals and academic and social recognition for students.

The university environment produces many changes in students’ lives, in areas such as relationships and physical and occupational contexts, and can be highly stressful. Students must develop a more independent life while preparing themselves for a professional future. Consequently, some may lack energy to continue, lose interest in their studies, and/or feel unable to meet their commitments and achieve their goals [1,9,10].

The prevalence of burnout in university students is high [11], professionals in training increasingly present symptoms compatible with burnout syndrome. Specifically, nursing students achieve higher average scores in burnout than the general population [12,13,14,15]. In light of this situation, it is crucial to conduct research on the occurrence of burnout among university students, particularly focusing on the variables that could serve as risk or protective factors for the syndrome. Additionally, it is important to develop and validate instruments that can effectively identify burnout in university students. Such endeavors are necessary to propose educational policies aimed at enhancing the conditions under which students pursue their university studies [16].

The most widely accepted conceptualization of burnout syndrome is based on the following three dimensions: Emotional Exhaustion, Depersonalization or Cynicism, and low Personal Accomplishment or Academic Efficacy [17,18,19]. In the university environment, students who perceive a loss of energy, who lack commitment to their studies, who disparage teachers and other students, and who are unable or unwilling to perform the tasks required of them, may be experiencing academic burnout. Indeed, numerous studies have reported high levels of academic burnout in university education [1,20]. This prevalence, together with the serious physical and psychological consequences that may arise from the disorder, underlines the need for an in-depth study of burnout regarding evaluation, prevention, and remedial intervention among university students [2,21,22].

The studies carried out with the aim of estimating the real relevance of burnout among university students indicate that between 9 and 21% of university students are at risk of developing burnout. These data are related to the COVID-19 pandemic, showing that students who have suffered psychological consequences derived from the pandemic present higher scores in Emotional Exhaustion and lower levels of Academic Efficacy than those who have not suffered them [20]. As a result of the COVID-19 pandemic, university students present high levels of burnout [23,24], uncertainty about their professional future [25], and higher levels in psychological variables, such as anxiety or depression [26,27].

In view of the reality of burnout syndrome in students, it is necessary to use instruments that are capable of identifying those students who present burnout in its different levels or intensities. The Maslach Burnout Inventory (MBI) [17,18] was created to measure burnout in workers. Subsequently, the MBI-SS [19] was created to measure burnout syndrome in students. The context and stressors of workers are far from being the same as those of students or workers in training, so the instruments for measuring burnout cannot be the same for both workers and students [1,28,29].

Although the MBI is the most widely used instrument for assessing burnout syndrome in both professional and non-professional samples [30], several studies examining its psychometric properties have identified some problems. For instance, it has been consistently found that the reliability of the Cynicism dimension falls below the recommended cutoff points [31]. Further, and more importantly, the MBI in its Spanish version was adapted back in 1997 [32]. However, the current survey is no longer accessible, making it legally infeasible to use. Additionally, there are no up-to-date scales available for evaluating the Spanish population. It is worth noting that the criteria used for assessment were developed in 1997, which raises concerns about their applicability in the present context. Due to the need to assess burnout syndrome in pre-professional groups such as students, the idea of adapting the CBG for university populations emerged. The theoretical framework of the MBI is well established and widely accepted in the field [31], so the CBG was developed under the same measurement structure.

In this work, we present the Cuestionario Burnout Granada-University Students (CBG-USS, for its Spanish abbreviation) in a sample of university students enrolled in undergraduate nursing. This sample has been chosen because, after the pandemic, students in health areas present high levels of burnout [20,23,24]. Nurses provide care to patients, often in contexts where they are at risk of experiencing physical and verbal aggression [33]. Furthermore, nurses encounter the challenging dichotomy of their daily work. While they are expected to exhibit a compassionate and caring demeanor, which is inherent to their vocation, they frequently find themselves compelled to be emotionally detached in order to make difficult decisions that require an objective approach devoid of emotional responses. For these reasons, nurses consistently face high levels of stress, which is known to contribute to the onset of burnout syndrome [34,35].

It is important to have instruments to measure burnout in workers, but without forgetting that it is also necessary to have specific instruments to measure burnout in students, since the students who present burnout today will be the workers with burnout in the future [36]. Given the need for instruments that adequately measure and discriminate academic burnout in university students, the aim of this study is to examine the reliability and some sources of validity of the Cuestionario Burnout Granada-University Students in a sample of nursing students. Specifically, reliability was analyzed based on internal consistency of the test items, and validity evidence was examined based on its relationship to other variables, namely, convergent, discriminant, and concurrent criterion-related validity.

1.1. Reliability Estimators

Reliability is closely related to measurement precision. In a broad sense, it refers to the degree to which test scores are free from errors of measurement [37]. The importance of measurement reliability is consistently significant, as the demand for precision is more crucial as the significance of decisions and interpretations amplifies.

A wide range of reliability estimators have been developed, depending on the measurement model of the test under study. Probably, the most used estimators are Cronbach’s alpha and McDonald’s total omega [38,39]. The alpha coefficient (α) was originally proposed for τ-equivalent models, and total omega coefficient (ω_t) for congeneric models. A discussion on these models can be consulted in another article of the current Special Issue [16]. Coefficient alpha is a lower bound to reliability, and its calculation is described by the following equation [40]:

$$\alpha =\frac{J}{J-1}\left[1-\frac{{{\displaystyle \sum }}_{1 \le i\ne k\le J}^J\left({\sigma }_{ik}\right)}{{\sigma }_X^2}\right]$$

(1)

where $J$ = number of items; ${\sigma }_{ik}$ = covariance between item I and item k; ${\sigma }_X^2$ = test variance. The calculation of total omega coefficient is described by the following equation [41]:

$${\omega }_t=\frac{{\left({{\displaystyle \sum }}_{i=1}^J{\lambda }_i\right)}^2}{{\left({{\displaystyle \sum }}_{i=1}^J{\lambda }_i\right)}^2+{{\displaystyle \sum }}_{i=1}^J{\sigma }_{\epsilon i}^2}$$

(2)

where ${\lambda }_i$ = factor loading for item i; ${\sigma }_{\epsilon i}^2$ = error variance for item i; ${\lambda }_i^{\left(g\right)}$= factor loading of item i on the general factor g; ${\lambda }_i^{\left(s_1\right)}\dots ,{\lambda }_i^{\left(s_h\right)},\dots {\lambda }_i^{\left(s_p\right)}$= factor loadings of item i on the specific factors $s_1$, …, $s_h$, …, $s_p$, and the specific factors comprise $J_1$, …, $J_h$, …, $J_p$ items.

Alpha coefficient requires compliance with assumptions such as unidimensionality, τ-equivalence, and normality of the distribution of the items [42]. When the alpha coefficient assumptions are not met, it tends to underestimate the true reliability of the scale. For this reason, to deal with congeneric scales (which do not satisfy the assumption of τ-equivalence), the omega coefficient is usually recommended [43]. As for the analysis of reliability of the CBG-USS scores, both alpha and total omega coefficients were calculated and compared because they might give complementary information (a lower bond to reliability based on test scores, and a factor analysis-based estimation of reliability).

1.2. Evidence of Validity

There have been several authors who have developed the current ideas about validity [44,45], many of which have been embraced by the Standards for Educational and Psychological Testing (SEPT) [37]. According to the SEPT, validity refers to the extent to which theory and evidence support the interpretations of test scores for the intended uses of the test. Thus, any validation process must begin by clearly establishing the test’s intended uses. In this line, the proposed uses for the CBG-USS focus on research and applied purposes. As for the research context, it is anticipated to be used in studying the psychometric properties of the test itself and in investigating burnout syndrome (theoretical predictions, prevalence, risk factors, etc.). In applied contexts, the instrument could be used to assess the extent to which university students exhibit burnout symptoms, aiming to detect potential academic difficulties without establishing a diagnosis.

The SEPT framework establishes five sources of validity that are based on the test content, the response processes of participants, the internal structure of the test, the relations of the test to other variables, and the consequences of testing. Being all sources of validity relevant to the validation process, there is no recommendation about prioritizing among them. In the present study, we aimed to analyze validity evidence based on the relationship between CBG-USS scores and other variables that are theoretically related to the three dimensions of the test.

Relation to other variables’ validity provides evidence about the degree to which relationships between two measures are consistent with the common construct underlying the proposed test score interpretations. This type of evidence contributes to a better understanding of the construct and the nomological network in which it is theoretically embedded. Specifically, three strategies of this type of validity have been proposed: convergent, discriminant, and criterion.

To gather convergent validity evidence, a correlational analysis can be conducted, for example, including another test that assesses the same construct. Thus, we also measured burnout syndrome with the MBI that is conceptually equivalent to the CBG. Regarding discriminant validity evidence, the relationship with an external criterion is analyzed. One common strategy is creating (or selecting) groups and comparing the mean scores among these groups.

To gather criterion validity evidence of the CBG-USS, we used the area under the curve (AUC) analysis. AUC, commonly used to evaluate classification accuracies, is a preferred method that relies on predictive models [46]. It eliminates the need for subjective threshold decisions, making it a reliable choice.

AUC expands upon the receiver operating characteristics (ROC) curve, which summarizes the performance of label assignment [47]. It accomplishes this by integrating a confusion matrix (a 2 × 2 table that includes counts for true/false positive and true/false negative) at various threshold levels. These changes can affect classification accuracies. Figure 1 displays the confusion matrix and equations for various commonly calculated statistics derived from it. In the case of a binary classifier, a positive label is assigned when the predicted category of an instance is 1, while a negative label is assigned when the predicted category is 0. Correct predictions are labeled as true, while incorrect predictions are labeled as false. Consequently, each instance corresponds to a specific cell within the confusion matrix, which consolidates the instance counts for each of the four categories.

The ROC curve is plotted on a two-dimensional plane, where the horizontal axis represents the false positive rate and the vertical axis represents the true positive rate. A discrete classifier generates a single confusion matrix, corresponding to a specific point in the ROC space. In the case of probability or scoring classifiers, different thresholds can be used to obtain multiple confusion matrices, and each threshold value results in a distinct point on the ROC curve.

AUC transforms the ROC curve into a numerical measure of performance for a binary classifier. In essence, AUC combines the model’s performance across all possible threshold values. AUC represents the area under the ROC curve and falls between 0 and 1: the highest value signifies a flawless classifier, while zero indicates that all predictions are incorrect. The AUC can be computed for a finite set of instances using the following steps: First, arrange the instances in descending order based on their predicted probabilities of being positive. Then, utilize these predicted probabilities as threshold values and calculate the corresponding true positive rate (TPR) and false positive rate (FPR). This process generates a series of points on the ROC plane, progressing upwards and to the right, thereby forming the ROC curve. Finally, the AUC is obtained by summing the areas of the trapezoids created between each instance point ‘i’ and the subsequent point ‘i + 1’ along with the horizontal axis (false positive rate). The calculation of AUC is described by the following equation [46]:

$$AUC=\frac{1}{2}{{\displaystyle \sum }}_{i=1}^{m-1}\left(FP{R}_{i+1}-FP{R}_i\right)\left(TP{R}_{i+1}+TP{R}_i\right)$$

(3)

where $m$ represents the number of instances.

The manuscript is organized as follows. First, the methodological aspects of the research are presented. Second, the results derived from the conducted analyses are described; specifically, descriptive statistics are presented first. Then, the prevalence of burnout syndrome among nursing students is estimated. Next, the results regarding the reliability of the CBG-USS are presented, followed by the results concerning convergent, discriminant, and concurrent criterion-related validity. In the Discussion section, the scope of the results in terms of psychometrics and the usefulness of the CBG-USS are presented.

2. Method

2.1. General Background

This paper presents an instrumental study to check the proper functioning of the Cuestionario Burnout Granada-University Students (CBG-USS) [48,49], whose objective is to identify burnout in university students. The Cuestionario Burnout Granada-University Students (CBG-USS) has been developed following international guidelines for the construction of measurement instruments [37,50,51]. In creating this instrument, the definition and dimensions of burnout proposed by Maslach and Jackson [17] were taken as a reference. Thus, burnout is understood as a psychological problem arising from continual exposure to stressors related to the educational institution and to study activities. The syndrome is characterized by three dimensions: Emotional Exhaustion, Cynicism or Depersonalization, and low Academic Efficacy or personal accomplishment.

2.2. Participants

A total of 463 undergraduate nursing students formed the sample for this study. The participants were mostly women without a partner or children. To participate in this study, the students had to meet the following inclusion criteria: (a) be enrolled in the Bachelor’s Degree in Nursing at the time of data collection; (b) agree to participate voluntarily in the research and complete the participation documentation, and (c) answer all the items of the CBG-USS and MBI-SS instruments.

2.3. Procedure

This research was conducted as an instrumental study [50,51]. The research project was made known to nursing students in information sessions. The announcement of the realization of the information sessions was made through the students’ electronic learning platform. Those interested in participating were given information about the study, including the expected time for them to complete the questionnaire (15–20 min) and the privacy policy. Information was collected in person, individually, anonymously, and voluntarily. The students who decided to participate in the study completed a paper information collection notebook that included the sociodemographic variables and the burnout instruments used. The sampling technique used was non-probabilistic by convenience [51] reaching a response rate of 82.3%.

2.4. Ethical Considerations

The study was approved by the Ethics Committee of the University of Granada (393/CEIH2017). The participants were informed of the objective of the study. No personal data were collected from the students. Participation in the study was voluntary and anonymous. Written informed consent was requested before starting the study. The collection of information and its subsequent data processing was carried out completely anonymously.

2.5. Study Variables

The following sociodemographic variables were ascertained: sex, age, marital status, and number of children. With respect to the students’ educational background, the following study data were obtained: degree course title, schedule (morning/evening classes), and previous university studies. Two workplace-related variables were included: extra-university work (yes/no) and, if so, the work schedule.

2.6. Instruments

The Cuestionario Burnout Granada-University Students (CBG-USS) is composed of 26 items grouped into three dimensions: Emotional Exhaustion (9 items), Cynicism (7 items), and Academic Efficacy (10 items). The items are presented using a Likert-type response format of five alternatives, where 1 = completely disagree and 5 = completely agree. In scoring the questionnaire items, the responses for 12 (items 1, 4, 10, 12, 13, 19, 20, 21, 23, 24, 25, and 26) were reverse coded. In order to calculate a total score for each dimension, it is necessary to sum all the item scores within that dimension. This provides the following interpretation: the higher the score, the greater the level of burnout experienced by the participant. The CBG-USS items can be consulted in the Supplementary Materials.

The Maslach Burnout Inventory-Student Survey (MBI-SS) [19] is the adaptation of the Maslach Burnout Inventory [17] for use with students. It contains 15 items grouped into three dimensions: Emotional Exhaustion (5 items), Cynicism (4 items), and Academic Efficacy (6 items). Each item is scored on a 7-point Likert scale as follows: Never = 0; A few times a year = 1; Once a month at most = 2; A few times a month = 3; Once a week = 4; A few times a week = 5; Every day = 6. In the present study, the Spanish version of the MBI-SS [52,53] was used.

2.7. Data Analysis

For the description of categorical variables, the relevant percentages and frequencies were calculated for each level of response. For the quantitative variables, descriptive statistics (mean, standard deviation, and minimum and maximum values) were obtained.

To estimate the prevalence of burnout, the Golembiewski, Munzerider, and Stevenson model was used; this classifies participants as high and low for each of the MBI-SS burnout dimensions [54,55].

Alpha and omega reliability coefficients together with their corresponding 95% confidence interval were calculated for each dimension of the CBG-USS and MBI-SS, taking as reference for the evaluation of the reliability coefficients the recommendations made in this regard by George and Mallery [56] and Aguayo et al. [57].

Regarding discriminant validity, differences between groups (students with and without burnout syndrome) on measures of burnout dimensions were detected. The procedure was as follows: First, we measured the dimensions of burnout syndrome using the MBI-SS (not the CBG-USS, which is the instrument we aim to validate). Second, based on the procedure proposed by Golembiewski and Munzenrider [54], we classified the participants into groups of students with high and low levels of burnout symptoms (i.e., students experiencing burnout and students not experiencing burnout). Subsequently, Student’s t-tests for independent groups were conducted to determine if there were statistically significant differences in CBG-USS scores between the two groups. It is important to note that the instrument used to create the groups (MBI-SS) is different from the instrument used as the dependent variable in the hypothesis test (CBG-USS). If, as we expected, the CBG-USS is an instrument that discriminates between different levels of burnout, the results obtained with it would be similar to those obtained with the MBI-SS used as the dependent variable in the hypothesis tests. An effect size index (Cohen’s d) and the corresponding 95% confidence intervals around the point estimation was also performed. The effect size index reports the degree of these differences obtained in the significance tests [58,59].

To gather evidence of convergent validity, a correlational analysis was performed including the dimensions of the MBI and the CBG. Thus, Pearson correlation coefficient and the corresponding 95% confidence interval between burnout dimensions were calculated. Adjusted correlations for reliability were also calculated following the formula by Gulliksen [60].

To study concurrent criterion validity of the CBG-USS, the area under the curve (AUC) and the corresponding 95% confidence interval were calculated for each dimension of burnout. The potential AUC score ranges from 0 (perfect negative prediction) to 1 (perfect positive prediction). An AUC of 0.50 reflects a prediction equal to chance; one of 0.56 to 0.64 represents a small effect; one of 0.64 to 0.71, a medium effect, and one higher than 0.71, a large effect [61]. All statistical analyses were performed using SPSS IBM (v.27).

3. Results

3.1. Sociodemographic Profile

The sample was composed of 463 nursing students, of whom 74.1% were women, aged from 18 to 59 years with a mean age of 21.9 (5.12) years; 95.8% were single and 95.6% had no children.

Regarding the participants’ education variables, 30.9% were in the first year of the degree course, and 57.6% in the second year. Most had classes in the morning; 3.7% had previously obtained a university degree and 8.2% were currently working, as well as studying (of these, 32.4% were working full-time and 67.6% were working part-time) (

Table 1. Categorical variables.

	% (n)		% (n)
Sex		Degree year
Male	25.9 (119)	1st	30.9 (140)
Female	74.1 (341)	2nd	57.6 (261)
		3rd or 4th	11.5 (52)
Marital status		Previous degree
Single	95.8 (435)	Yes	3.7 (17)
Married/Partnership	4.2 (19)	No	96.3 (439)
Children		In employment
0	95.6 (345)	Yes	8.2 (37)
1 or more	4.4 (16)	No	91.8 (413)
Class schedule		Work regime (if any)
Mornings	85.3 (221)	Full-time	32.4 (11)
Afternoons/Evenings	14.7 (38)	Part-time	67.6 (23)

).

3.2. Descriptive Data for the CBG-USS and the MBI-USS

Table 2. Mean, SD, and minimum and maximum values for the CBG-USS and MBI-SS dimensions.

	Mean	SD	Minimum	Maximum	N
CBG-USSEmotional Exhaustion	23.92	5.48	11	42	452
CBG-USSCynicism	13.34	3.86	7	30	456
CBG-USSAcademic Efficacy	40.33	5.71	15	50	453
MBI-SSEmotional Exhaustion	13.23	5.91	0	30	452
MBI-SSCynicism	9.62	5.11	0	29	442
MBI-SSAcademic Efficacy	27.77	5.61	1	36	449

shows the descriptive statistics for burnout scores on the CBG-USS and the MBI-SS. Students obtained a mean score of 23.92 (SD = 5.48) and a range of 11 to 42 on the CBG-USS Emotional Exhaustion scale, 13.34 (SD = 3.86), with a range of 7 to 30, on CBG-USS Cynicism, and 40.33 (SD = 5.71) and a range of 15 to 50 on CBG-USS Academic Efficacy.

On the MBI-SS, the students obtained mean scores of 13.23 (SD = 5.91), ranging from 0 to 30 for Emotional Exhaustion, 9.62 (SD = 5.11), ranging from 0 to 29 for Cynicism, and 27.77 (SD = 5.61), and ranging from 1 to 36 for Academic Efficacy.

3.3. Prevalence of Burnout Syndrome

We used the phase model proposed by Golembiewski and Munzenrider [55] to estimate the prevalence of the burnout syndrome in nursing students, according to the scores obtained in the MBI-SS. These authors propose an 8-phase model in which students are placed in a phase according to their level (Low, High) in each of the Burnout dimensions (Emotional Exhaustion, Cynicism, and Academic Efficacy). Phases 1 and 2 reflect low levels of burnout; phases 3, 4, and 5 correspond to a moderate degree of burnout, and phases 6, 7, and 8 indicate high levels of burnout; 15.4% of the students presented mild levels of burnout (phases 1 and 2), 10.1% presented medium levels of burnout (phases 3 to 5), and 11.7% of the participants recorded high levels of burnout (phases 6 to 8) (

Table 3. Prevalence of burnout according to the phases of the Golembiewski model.

Phase	1	2	3	4	5	6	7	8
Emotional Exhaustion	L	L	L	L	H	H	H	H
Cynicism	L	H	L	H	L	H	L	H
Academic Efficacy	L	L	H	H	L	L	H	H
N	61	10	19	9	19	19	13	22
%	13.2	2.2	4.1	1.9	4.1	4.1	2.8	4.8

Levels (high or low) in each of the burnout dimensions according to the model proposed by Golembiewski. H = High; L = Low.

).

3.4. Sample Reliability of the CBG-USS

We calculated alpha and omega coefficients to study sample reliability of the CBG-USS. The alpha coefficients for the CBG-USS dimensions showed the following values: Emotional Exhaustion = 0.769, 95%CI [0.736, 0.799], Cynicism = 0.809, 95%CI [0.781, 0.835], and Academic Efficacy = 0.820, 95%CI [0.794, 0.843]. The omega coefficients were: 0.771, 95%CI [0.740, 0.803] for Emotional Exhaustion, 0.753, 95%CI [0.719, 0.787] for Cynicism, and 0.823, 95%CI [0.799, 0.847] for Academic Efficacy.

In order to be able to make comparisons, we present here the reliability results concerning the MBI-SS. The alpha coefficients obtained for each of the scales of the MBI-SS for our study sample were: Emotional Exhaustion = 0.735, 95%CI [0.695, 0.771], Cynicism = 0.603, 95%CI [0.544, 0.657], and Academic Efficacy = 0.773, 95%CI [0.740, 0.802]. The omega coefficients were: 0.739, 95%CI [0.694, 0.781] for Emotional Exhaustion, 0.662, 95%CI [0.602, 0.714] for Cynicism, and 0.780, 95%CI [0.730, 0.817] for Academic Efficacy.

3.5. Convergent and Discriminant Validity of the CBG-USS

Evidence of convergent and discriminant validity of the CBG-USS was studied by calculating the correlation coefficients between the MBI-SS dimensions and those of the CBG-USS with the following results: Emotional Exhaustion r = 0.620, 95%CI [0.559, 0.674]; Cynicism, r = 0.281, 95%CI [0.192, 0.365], and Academic Efficacy, r = 0.487, 95%CI [0.413, 0.556] (see

Table 4. Correlation coefficients for the CBG-USS and MBI-SS dimensions.

	MBI-SS Emotional Exhaustion	MBI-SS Cynicism	MBI-SS Academic Efficacy
CBG-USSEmotional Exhaustion	0.620 ** (0.821)	0.314 ** (0.421)	−0.122 * (−0.156)
CBG-USSCynicism	0.166 ** (0.232)	0.281 ** (0.398)	−0.384 ** (−0.520)
CBG-USSAcademic Efficacy	−0.316 ** (−0.407)	−0.380 ** (−0.496)	0.487 ** (0.608)

Values in parentheses are adjusted correlations for reliability. ** p < 0.01; * p < 0.05.

).

We also studied evidence of convergent validity of the CBG-USS by comparing the results of the burnout dimensions in two groups: one that suffers from burnout syndrome, and another one that does not. The 54 students who were classified as corresponding to phases 6–8 of the Eight-Phase Model [54,55] were assumed to present burnout, and the remaining 398 students (corresponding to phases 1–5) were classed as not presenting burnout. With respect to the latter, the students who presented burnout recorded higher scores for Emotional Exhaustion and Cynicism, on both the MBI-SS and the CBG-USS instruments, as follows: [MBI-SS Emotional Exhaustion: 20.13 (3.62); CBG-USS Emotional Exhaustion: 26.98 (5.11)], [MBI-SS Cynicism: 14.90 (5.79); CBG-USS Cynicism: 14.92 (3.70)]. By contrast, the no-burnout group presented higher scores for Academic Efficacy [MBI-SS Academic Efficacy: 28.18 (5.40); CBG-USS Academic Efficacy: 40.77 (5.47)]. There were statistically significant differences between the with/without burnout groups for all the MBI-SS and CBG-USS dimensions. The following effect sizes were measured for each scale: MBI-SS Emotional Exhaustion: 2.01, 95%CI [1.696, 2.323]; MBI-SS Cynicism: 1.062, 95%CI [0.77, 1.355] and MBI-SS Academic Efficacy: −0.557, 95%CI [−0.844, −0.271]; CBG-USS Emotional Exhaustion: 0.674, 95%CI [0.386, 0.962]; CBG-USS Cynicism: 0.479 95%CI [0.193, 0.765]; CBG-USS Academic Efficacy: −0.607 95%CI [−0.895, −0.32] (

Table 5. Mean values, SD, and effect size for students with/without burnout for MBI-SS and CBG-USS.

	Burnout	M (SD)	p	Cohen’s d	95%CI
CBG-USSEmotional Exhaustion	No	23.510 (5.410)	<0.001	0.674	[0.386, 0.962]
	Yes	26.981 (5.119)
CBG-USSCynicism	No	13.141 (3.837)	<0.001	0.479	[0.193, 0.765]
	Yes	14.923 (3.709)
CBG-USSAcademic Efficacy	No	40.778 (5.470)	<0.001	−0.607	[−0.895, −0.32]
	Yes	36.942 (6.415)
MBI-SSEmotional Exhaustion	No	12.30 (5.539)	<0.001	2.01	[1.696, 2.323]
	Yes	20.13 (3629)
MBI-SSCynicism	No	8.894 (4.554)	<0.001	1.062	[0.77, 1.355]
	Yes	14.907 (5.796)
MBI-SSAcademic Efficacy	No	28.189 (5.405)	<0.001	−0.557	[−0.844, −0.271]
	Yes	24.759 (6.252)

Absence of burnout: N = 398; presence of burnout: N = 54.

).

3.6. Concurrent Criterion Vailidity of the CBG-USS

Evidence of validity based on a concurrent criterion was analyzed by using area under the curve analysis. The area under the curve (AUC) was calculated for each dimension (MBI-SS and CBG-USS), obtaining the following results: Emotional Exhaustion, AUC = 0.893, 95%CI [0.862, 0.924] for MBI-SS and 0.687, 95%CI [0.610, 0.764] for CBG-USS; Cynicism, AUC = 0.807, 95%CI [0.733, 0.880] for MBI-SS and 0.654, 95%CI [0.581, 0.726] for CBG-USS, and Academic Efficacy, AUC = 0.338, 95%CI [0.250, 0.426] for MBI-SS and 0.318, 95%CI [0.243, 0.393] for CBG-USS (

Table 6. AUC for each dimension of MBI-SS and CBG-USS.

	AUC	SE	95%CI
CBG-USSEmotional Exhaustion	0.687 *	0.039	[0.610, 0.764]
CBG-USSCynicism	0.654 *	0.037	[0.581, 0.726]
CBG-USSAcademic Efficacy	0.318 *	0.038	[0.243, 0.393]
MBI-SSEmotional Exhaustion	0.893 *	0.016	[0.862, 0.924]
MBI-SSCynicism	0.807 *	0.037	[0.733, 0.880]
MBI-SSAcademic Efficacy	0.338 *	0.045	[0.250, 0.426]

Total sample, N = 463; absence of burnout, N = 398; presence of burnout, N = 54. AUC = area under the curve; SE = standard error; CI = confidence interval; CBG-USS = Cuestionario Burnout Granada-University Students; MBI-SS = Maslach Burnout Inventory-Student Survey; * p < 0.001.

).

4. Discussion

The aim of this study was to obtain evidence of the validity of the Cuestionario Burnout Granada-University Students, in this case, in a sample of nursing students. The CBG-USS is introduced as a university student version of the Cuestionario Burnout Granada (CBG), which has been utilized and validated as a screening tool for burnout syndrome in Nurses and Police Officers [62,63,64]. Since the CBG-USS is a recently created instrument, a reference instrument has been used to demonstrate the correct functioning in the measurement and detection of burnout, in this case, the MBI-SS [19,53], the student version of the MBI [32]. The MBI-SS has been adapted and used in different countries [65,66,67,68,69] and has proven to be an adequate reference measure for the purpose of this work.

The results obtained corroborate the performance of the CBG-USS and provide evidence of its convergent, discriminant, and concurrent criterion-related validity. According to Cohen’s [70] guidelines, high correlation coefficients were obtained between the Emotional Exhaustion and Academic Efficacy dimensions and moderate correlation coefficients for Cynicism between the dimensions of the CBG-USS and the MBI-SS. Given the problems concerning the low reliability values of the Cynicism dimension of the MBI-SS together with good reliability values of the CBG-USS, it was desirable to find a lower correlation coefficient compared with the other two dimensions. As expected by the Classical Test Theory, adjusted correlations for reliability were higher than non-adjusted correlations, leading to high correlations between all burnout dimensions. Both instruments measure burnout according to the three dimensions proposed by Maslach and Jackson [7,17,18,19].

Concerning reliability, the results of the alpha coefficient showed that in all dimensions of burnout, the CBG-USS obtained higher reliability values than the MBI-SS. This is especially relevant in the cynicism dimension, since in the MBI-SS, it was below the recommended cutoff point, a result that has been obtained on several occasions with various versions of the MBI [31]. Regarding the comparison between alpha and omega coefficients, the values were similar for all the dimensions of both measurement instruments. Specifically, while for the CBG-USS omega values were lower than alpha values, for the MBI-USS, the results showed the opposite (higher values for omega than for alpha). These results suggest that the alpha coefficient is not equivalent to the lower limit of reliability when assumptions, such as normality of item distribution or t-equivalence, are not met. Moreover, the alpha coefficient probably does not only underestimate the true reliability but also may overestimate it, as the results of the CBG-USS have shown. Only the Academic Efficacy dimension achieved the same values for alpha and omega coefficients. This was likely because this dimension has better psychometric properties than the other two dimensions, as the values of reliability indicated.

Statistically significant differences were observed between the with/without burnout groups. In general, the participants in the burnout group presented higher scores on the CBG-USS for Emotional Exhaustion and Cynicism, and lower ones for Academic Efficacy. Similar results were obtained with the MBI-SS. Moreover, this is in line with previous research findings concerning students following different degree courses [67]. These results corroborate the validity of the CBG-USS and confirm its ability to discriminate between students with/without burnout. The estimated AUC for the dimensions of the CBG-USS provide evidence of criterion validity of the instrument. The values found indicate adequate performance on all dimensions of the instrument. The results of this study support that the CBG-USS is an instrument that correctly measures and identifies burnout in university students.

Nursing students, professionals in training, have a high probability of developing burnout while pursuing their academic career. During their training, they must perform prolonged periods of internships in hospital centers, experiencing dissatisfaction with such internships or the organization of the same [71,72,73]; the gap between expectations and the reality of clinical practices or the perceived lack of support among peers and/or professors can facilitate the occurrence of burnout in nursing students [74,75].

Academic burnout is related to a higher probability of abandoning school/university and its impact may continue throughout one’s working life. In addition, it can provoke psychological consequences and affect the professional performance of nursing professionals [76,77,78]. Among nursing students in particular, the COVID-19 pandemic aggravated the possibility of burnout, in terms of their learning capacity, their health and well-being, the quality of care provided, and the possible intention to abandon the profession after graduation [20,79].

It has been shown that nursing students are more vulnerable to burnout during the later years of their university education, and this has been especially so since the outbreak of the COVID-19 pandemic [79,80]. Moreover, the impact of burnout is often more intense towards the end of the degree course [81]. According to Hong [82], the level of burnout increases with the level of education and, for nursing students, the most severe conditions are experienced when clinical practice begins, although another study has reported that the level of burnout remains stable over time [83]. Final-year nursing students, moreover, may feel disappointed and believe themselves incapable of assuming the responsibilities of the profession [84].

The use of instruments that are able to measure and correctly identify the level or intensity of burnout syndrome in university students is important for both research and professional practice. Having calibrated instruments helps in the identification of students who should be the target of interventions aimed at reducing the symptomatology and consequences derived from the burnout syndrome they present. From a practical point of view, professionals need to be confident that the instruments they are going to use discriminate correctly between students with and without burnout; screening is the first step to identifying those future professionals who should be the object of interventions aimed at preventing and/or alleviating the effects caused by burnout in people who are affected by it.

Burnout syndrome prevention programs should be focused on the risk and protective factors presented by university students who suffer from it, so it would be advisable to conduct studies with follow-up measures to verify the stability of the identified needs. Research has described some of the factors to which attention should be paid in working with students with burnout; specifically, work should be conducted to increase resilience [80,85,86] and strengthen perceptions of professional usefulness and self-efficacy [24,87], among other aspects. The ultimate goal of intervention regarding students with burnout should be to improve the well-being of future nursing professionals.

Limitations and Future Directions

This research paper presents an approach to the functioning of the CBG-USS in a sample of nursing students. Among the limitations of this work, the following stand out: First, only the results related to the convergent, discriminant, and concurrent criterion validity of the CBG-USS are presented in this work. Second, the reliability of the CBG-USS has been estimated by taking a single measure; no test–retest was performed. Third, the sample used was selected by convenience sampling, and fourth, the sample used included only nursing students. The aspects related to the sample and sampling used derive from the design proposed for this research and the limitations inherent to carrying out a study in a university educational center in which students are not present all year round; there are vacation periods and students carry out exchanges between universities, which is why a test–retest measure was not considered or the sample was not extended to other degree programs. These aspects will be taken into account in future research; among the objectives of the research group is to extend the sample studied, that is, to apply the CBG-USS to university students from other degrees.

This present study addresses just one source of validity evidence, which is the one that is based on the relation to other variables. From the SEPT framework, there are four additional sources of validity. In future studies, we aim to carry out several analyses to gather new validity evidence on the functioning of the CBG-USS, such as analyzing its internal structure and measurement invariance in different samples.

5. Conclusions

The aim of this research was to examine and provide evidence on some psychometric properties of the Cuestionario Burnout Granada-University Students, specifically, the reliability and validity of this instrument. The study sample is formed by students enrolled in the Bachelor’s Degree in Nursing, since health professionals present high rates of burnout. The results found provide evidence of the adequate functioning of the Cuestionario Burnout Granada-University Students, both in reliability and validity. Specifically, given the correlations between the dimensions of MBI and CBG, convergent validity evidence has been obtained, indicating that the burnout measure provided by CBG-USS appears to be appropriately aligned within the nomological network of the construct, given Maslach and Jackson’s three-dimensional theory [17]. Discriminant validity evidence has also been obtained, as there were significant differences in CBG-USS scores between students with and without the syndrome. Thus, it appears that CBG-USS can be appropriately used to identify students who have burnout syndrome. Furthermore, criterion validity evidence was also obtained, considering the estimated values by the AUC analysis. This result, similar to the previous one, suggests that CBG-USS can be used to estimate the degree to which students experience burnout syndrome.

Among the findings of this work is the availability of an instrument that identifies burnout in university students, which can be used both for research purposes and for the identification of future professionals who present different levels of burnout, with the aim of intervening to reduce the levels of burnout and the minimization of the consequences derived from this syndrome.

In future research, we would like to improve some of the limitations of this work. Specifically, to enlarge the sample with students from other degrees, both from other health professions and non-health professions, and to include variables that modulate the intensity and severity of burnout that can help in the intervention carried out with those affected by this syndrome.