Next Article in Journal
Interventions Targeting Bottle and Formula Feeding in the Prevention and Treatment of Early Childhood Caries, Overweight and Obesity: An Integrative Review
Previous Article in Journal
The Prevalence and Correlates of Probable Major Depressive Disorder and Probable Generalized Anxiety Disorder during the COVID-19 Pandemic. Results of a Nationally Representative Survey in Germany
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 18
Issue 23
10.3390/ijerph182312300
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Diagnostic Performances of an Occupational Burnout Detection Method Designed for Healthcare Professionals

by
Agathe Nguyen Huynh
1,
Christine Besse
2,
Zakia Mediouni
1,
Emna El May
1,3,
Yara Shoman
1,*,
Isabelle Hansez
4 and
Irina Guseva Canu
1
1
Center for Primary Care and Public Health (Unisanté), Department of Occupational and Environmental Health, University of Lausanne, 1066 Lausanne, Switzerland
2
Medical Direction of the Department of Psychiatry, CHUV, Les Cèdres (Cery), 1008 Prilly, Switzerland
3
Faculty of Psychology and Educational Sciences, University of Geneva, 1205 Geneva, Switzerland
4
Unit of Promotion of Human Resources, Faculty of Psychology, Speech Therapy and Educational Sciences, University of Liège, 4000 Liège, Belgium
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2021, 18(23), 12300; https://doi.org/10.3390/ijerph182312300
Submission received: 4 October 2021 / Revised: 9 November 2021 / Accepted: 19 November 2021 / Published: 23 November 2021
Versions Notes

Abstract

:
Background: We aimed to assess the validity (criterion and cross-cultural validity) and reliability of the first occupational burnout (OB) detection tool designed for healthcare professionals in Belgium in the context of Swiss medical practice. Methods: First, we assessed the sensitivity and specificity of the Tool. We developed this tool based on the consultation reports of 42 patients and compared its detection to the results of the Oldenburg Burnout Inventory (OLBI), filled-in by patients before a consultation. Second, we performed an inter-rater reliability (IRR) assessment on the OB symptoms and detection reached by the Tool between a psychiatrist, two psychologists, and an occupational physician. Results: The Tool correctly identified over 80% of patients with OB, regardless of the cutoff value used for OLBI scores, reflecting its high sensitivity. Conversely, its specificity strongly varied depending on the OLBI cutoff. There was a slight to fair overall agreement between the four raters on the detection of OB and the number of OB symptoms. Around 41% of symptoms showed a substantial to an almost perfect agreement, and 36% showed a slight to a moderate agreement. Conclusions: The Tool seems useful for identifying OB of moderate and strong severity in both the Belgian and Swiss contexts.

1. Introduction

There are no diagnostic standards for occupational burnout (OB), and none of the currently existing measurement tools could be used as a gold standard or comparator [1,2]. OB detection requires validated measurement tools on which healthcare practitioners and researchers can rely. The measurement tools that are currently available [3,4,5,6,7] are all patient-reported outcome measures (PROMS)—i.e., they rely solely on self-report—and their clinical validity has been questioned [8]. Indeed, PROMs should show vigorous psychometric properties to be utilized validly and reliably in medical research and practice [9]. In a recent systematic review on the validity of five PROMs, only the Copenhagen Burnout Inventory (CBI) and, to a lesser extent, the Oldenburg Burnout Inventory (OLBI) seemed to fulfil this requirement [10]. The insufficient validation of the first burnout PROMs has led to their multiplication and a marked heterogeneity in OB detection [2].
To address these issues, the Belgian Public Service for Employment, Work and Social dialogue (BPSEWS) developed a tool designed for healthcare professionals to use in clinical practice [11]. It aims to guide healthcare professionals through their consultation to establish the presence/absence and the frequency of typical OB symptoms and their relationship to work. The Tool was created in 2010 on the basis of a literature review and focus groups including 32 professionals from occupational medicine, insurance medicine, psychiatry, and psychology [12]. The Tool was tested by 346 general and occupational physicians during three months in 135,131 consultations. This confirmed that the list of symptoms was relevant, and in 2019, the Tool was then updated following nine individual interviews [13]. Its validity and reliability were investigated using the OLBI as a comparator. The Tool was also made available to healthcare professionals in the context of a national program aiming at the early detection of OB in workers exposed to psychosocial risk factors, their treatment, and subsequent two-year follow-up for secondary and tertiary prevention purposes [14].
The conceptualization of occupational burnout has always been problematic. Schaufeli and Taris [15] suggested that burnout should be considered as a work-related phenomenon consisting of at least two dimensions that can be measured generally or specifically. The OLBI conceptualizes occupational burnout as two dimensions: exhaustion and disengagement from work [4]. The exhaustion dimension of the OLBI covers affective, physical, and cognitive aspects, whereas the disengagement dimension covers the relationship between the worker and his/her job (regarding engagement and identification). In Belgium, Hansez et al. consider burnout as a temporal process consisting of four stages of burnout, allowing an underlining of the complexity of its diagnosis [16]. The Tool is the first to assess the clinical judgement of a burnout diagnosis with the help of a structured consultation and detection of the symptoms and patterns of occupational burnout [17].
The importance of PROMs is widely accepted in clinical practice and research [18]. Nevertheless, they are often not enough for diagnostic purposes [19], and the use of hetero-assessment tools is still necessary. In a recent survey of Swiss physicians in primary and occupational healthcare, 88% said that they would find it interesting to integrate an OB diagnostic tool into their practice [20]. Therefore, we aimed at testing the Tool in a Swiss context to establish whether it could be implemented in Swiss medical practice. Our objective was to assess the criterion validity, reliability, and cross-cultural validity of the Tool, when comparing it with OLBI.

2. Materials and Methods

2.1. Study Sample

We used a convenience sample of patients received at the Unisanté “Work and Suffering” Consultation (WSC) between 2010 and 2013. The WSC is a multidisciplinary consultation, specialized in helping people in situations of OB, conflict, stress, violence, or harassment at work [21]. Between 2010 and 2013, the WSC operated in its pilot form, and all patients were asked to complete several PROMs just before the consultation, including the OLBI [4]. When the consultation entered into its regular operation, its duration was shortened, and the use of PROMs was abandoned. We included WSC patients for whom a completed OLBI was available in their medical record along with the WSC detailed report.

2.2. Study Design

We implemented a diagnostic research study design [22]. We assessed the validity (i.e., criterion and cross-cultural validity) and reproducibility (i.e., inter-rater reliability) of the Tool with the use of another measurement tool (i.e., the comparator). We chose the OLBI (the OLBI measures two dimensions of OB: exhaustion and disengagement and it includes 16 items, on which patients have to rate their agreement on a scale from one to four) as a comparator because it was used in the pilot phase of the WSC, and it was available in medical records, but also because the OLBI is the second-most used PROM of OB after the MBI and the second-most validated one after the CBI [10]. Moreover, Hansez et al. (2019) also used the OLBI as a comparator, allowing further comparison of the results [13].

2.3. A New Method for OB Detection by Healthcare Professionals

We used the latest version of the Tool, which was amended to conform to the Swiss medical vocabulary (via the reformulation of the reasons for consultation and alternative diagnoses) and reduced to a more condensed format. This resulted in a two-page document, including questions about the type of medical examination; the reasons for consultation; the socio-demographic characteristics of the patient; a list of the physical, cognitive/affective, and behavioral symptoms typical of OB cases; the relationship to work (i.e., work-related risk factors and missing resources, the importance of work for the patient); the current or past sick leaves; and the detection of OB (including a nota bene indicating that, in order to be diagnosed with OB, the patient needs to report at least two, six, and two physical, cognitive/affective and behavioral symptoms, respectively, and that his/her complaints need to originate primarily from work). These sections all contained multiple choices, allowing the healthcare professional to tick the boxes corresponding to the situation of the patient (See Supplementary Material).
We used the WSC reports instead of real consultations with patients. The WSC reports were written and signed by an occupational physician (usually assisted by a resident in occupational medicine) and co-signed by a psychiatrist who further assessed the presence of psychiatric disorders. The WSC reports were approximately 8 pages long and contained a description of the past and current occupations of the patient (tasks, responsibilities) and the patient’s perception of them, an evaluation of the impact of work factors on the physical and mental health of the patient, the personal history of the patient (family relationships, significant life events), the medical and psychiatric anamnesis of the patient (medical history, health habits), and a summary of the multidisciplinary discussion concerning the patient and his/her treatment.
We distributed a set of the anonymized WSC reports, without the OLBI results, to four raters with different backgrounds and experience: one psychiatrist, one occupational physician, one psychologist, and one master’s student in psychology. Each rater was asked to fill in one Tool on the basis of each WSC report exclusively. The raters were instructed to complete the Tools as exhaustively as possible. For example, if a symptom was not clearly stated in the report but the rater could read its presence between the lines, the symptom was reported. After having fully completed the Tool, the raters were instructed to refer to the cutoff values in the nota bene for their final detection of OB. To ensure the anonymity of the raters in the analyses, a random number was distributed to each of them by the study coordinator.

2.4. Statistical Analysis

2.4.1. Validity of the Tested Method

To assess the validity of the Tool, we tested the criterion and cross-cultural validity. For the criterion validity we used the OLBI as a comparator. We calculated the Tool’s sensitivity, positive predictive value (i.e., the probability that patients who are identified with OB by the Tool truly have OB) and negative predictive value [23]. The OLBI does not identify OB in a dichotomous manner but results in a global score between 16 and 64. Therefore, we considered two possible cutoff values in our analyses: the first cutoff value was 30, corresponding to mild and high severity burnout, while the second cutoff value was 44, corresponding to only high severity burnout. These cutoff values were established by Hansez et al. (2019) based on the distribution of OLBI scores in a large representative sample of Belgian workers [13]. We first conducted the validity analyses considering that patients with an OLBI score above 30 suffered from OB and then considering that patients with an OLBI score above 44 suffered from OB. Finally, we interpreted the diagnostic usefulness of the method (if the sum of sensitivity and specificity was at least 1.5). According to Power et al. [24], the method is considered useful, halfway between 1, which is useless, and 2, which is perfect. The analyses were performed using Stata 16 (StataCorp LLC., College Station, TX, USA) [25].

2.4.2. Reliability of the Tested Method

We performed inter-rater reliability (IRR) analyses to assess the degree of agreement between four raters regarding OB detection and the identification of each of the 22 symptoms of the Tool. Our design being fully crossed, we first calculated Cohen’s kappa [26] for all rater pairs and then used the arithmetic mean of these estimates to provide an overall index of agreement [27]. Following Hallgren’s (2012) [28] recommendations, we investigated the marginal distribution of each variable to apply the appropriate correction to Cohen’s kappa. When a prevalence problem was identified (the marginal distributions of observed ratings fell into one category of ratings at a much higher rate over another), the correction was performed according to Byrt, Bishop, and Carlin [29], resulting in Byrt’s kappa. When a bias problem was identified (the marginal distributions of specific ratings were substantially different between coders), Siegel and Castellan’s correction of Cohen’s kappa was applied [30]. All values of kappa were interpreted according to Landis and Koch [31] (values between 0.0 and 0.2 corresponded to a slight agreement, between 0.21 and 0.40 to a fair agreement, between 0.41 and 0.60 to a moderate agreement, between 0.61 and 0.80 to a substantial agreement, and between 0.81 and 1.00 to an almost perfect/perfect agreement).
Last, in order to assess the degree of agreement in the total number of symptoms identified for each patient (continuous variable), IRR was assessed using a two-way mixed, absolute, single-measure inter-class correlation (ICC) [32]. The value of the ICC was interpreted according to Cicchetti (when the reliability of the coefficient was below 0.40, the level of significance was considered poor, when it was between 0.40 and 0.59, the level of clinical significance was fair; when it was between 0.60 and 0.74, the level of clinical significance was good; and when it was between 0.75 and 1.00, the level of clinical significance was excellent.) (1994) [33]. All IRR analyses were run on R (R Foundation for Statistical Computing. Vienna, Austria) [34] following Hallgren’s (2012) procedure [28].

3. Results

3.1. Characteristics of the Patient Sample

Overall, 65 patients were received at the WSC between 2010 and 2013, and 42 of them completed the OLBI and were included in the study. They were mostly women (74%) and had a median age of 46 (interquartile range (IQR) = 20). A total of 67% were in a relationship, and 33% were single. A total of 69& worked full-time, and 26% worked part-time, while 5% were not employed. Out of the 40 workers, most were employees (72%), 25% were managers, and 3% did not state their status. Most worked in the private sector (64%), and 5% worked in the public sector, while 17% did not provide this information. The patients’ median seniority in the company was six years (IQR = 7.5). More than half of the workers had a permanent contract (67%), 5% had a fixed term contract, 20% were dismissed or in the process of being dismissed, 5% had quit or were in the process of quitting, and 3% did not state the type of their contract. Around 55% of patients were on sick leave at the time of the consultation because of the symptoms they reported during the consultation, and 64% had been on sick leave in the last twelve months for the same reasons.

3.2. Validity of the Tool

Table 1 summarizes the number of patients diagnosed with OB according to the comparator (the OLBI) and the tested method (the Tool). The sensitivity of the Tool was above 0.80 regardless of the OLBI cutoff value considered (Table 2). Conversely, for the specificity assessment, the choice of cutoff value seemed determinant. When the cutoff > 30 was used, the specificity of the Tool was perfect, although there was only one true positive patient. However, when a more stringent cutoff value was applied (>44), the specificity largely dropped to 25%. The sum of sensitivity and specificity was 1.83 when a cutoff > 30 was used, meaning that the test could be considered useful according to Power et al. (2013) [24]. However, this was not the case when a cutoff > 44 was used, i.e., the sum of sensitivity and specificity was 1.17.

3.3. Reliability of the Tool

The raters reported an average of 12 symptoms in total per Tool (SD = 0.23), including three physical symptoms (SD = 0.07), seven cognitive and affective symptoms (SD = 0.17), and two behavioral symptoms (SD = 0.07). They reported on average five work-related risk factors per Tool (SD = 0.15) and five resources that the patient was lacking at work (SD = 0.15). On average, the four raters diagnosed OB in 53% of cases.
The IRR analyses suggested that raters had a slight to a fair agreement in OB detection when using the Tool (Table 3). Similarly, the ICC on the total number of symptoms reported by the raters was in the fair range of clinical significance, ICC = 0.50. When looking at the 22 individual symptoms, 9 had Cohen’s and corrected kappas in the substantial to an almost perfect range (41%), and 8 had Cohen’s and corrected kappas in the slight to moderate range (36%).

4. Discussion

4.1. Summary of the Results

The criterion validity analyses with the OLBI as a comparator revealed that the Tool correctly identified over 80% of patients with OB, regardless of the cutoff value used for OLBI scores, reflecting its high sensitivity. Conversely, its specificity strongly varied depending on the OLBI cutoff, the latter still being undetermined. The IRR varied significantly from symptom to symptom: around 40% of symptoms showed a substantial to almost perfect agreement, while slightly more than one third showed a slight to moderate agreement. There was a slight to fair overall agreement between the four raters on the detection of OB and the number of associated symptoms.

4.2. Cross-Cultural Validity of the Tool

Comparing our results to those of Hansez et al. (2019) [13], who investigated the diagnostic accuracy of the Tool with OLBI as a comparator in Belgium, we had slightly higher sensitivity values than theirs (cutoff > 30, sensitivity = 66%; cutoff > 44, sensitivity = 77%). Interestingly, they also found a perfect specificity when using the cutoff > 30, and it dropped to 55% when using the cutoff > 44. Moreover, we both found that the sum of the sensitivity and specificity exceeded the critical value of 1.5, above which a test can be considered useful, when using the cutoff > 30 but not when using the cutoff > 44 [24]. From these results, Hansez et al. concluded that the Tool could be used in combination with a PROM, and they made it available in their national program with the OLBI. We consider that, in the Swiss context, the use of one validated tool would be more realistic and useful. The Tool is the only non-PROM tool available. After some amendments to increase its specificity and the confirmation of its cross-cultural validity, assessed using a true diagnostic standard (still nonexistent) or a precisely determined cutoff for the OLBI (ongoing research), we could firmly decide whether we should recommend its use.
The results suggest that burnout can be considered as a temporal process consisting of stages even in Swiss context. It was mentioned in the literature that burnout follows certain stages (from exhaustion to reduced professional efficacy) [35,36]. Nevertheless, the Tool is based on the theory that burnout consists of four stages: stage zero being a symptom-free stage, called “Engagement in work with idealistic enthusiasm”. The three further stages of burnout development are “Weakening of the idealism”, “Emergence of protective withdraw”, and “Burnout” [16]. A better understanding and investigation of these stages will allow the enhanced prevention, diagnosis, and treatment of burnout.

4.3. Reliability of the Tool

The results suggesting a slight to fair reliability of the medical detection based on the Tool between four raters deserve careful interpretation. It is interesting to note that the two most experienced raters often did not agree with the detection of OB reached by the Tool (for 29% and 57% of patients, respectively). In the large majority of these cases, the raters recognized that there was a process of OB but that this process had either been exceeded, i.e., the symptoms now reached the threshold for a psychiatric disorder (Code F ICD-10 Diagnosis) [37], or that a pre-existing psychiatric disorder might have contributed to the process of OB and was the predominant feature of the crisis’s symptomatology. The frequent occurrence of psychiatric comorbidities in our sample can be explained by the presence of relatively severe and longstanding cases. Indeed, more than half of patients were on sick leave at the time of the consultation, and 64% had been on sick leave in the prior twelve months. This makes it challenging for a standardized tool to capture the richness and complexity of the specialized multidisciplinary medical expertise contained in the WSC reports. These situations are however less common in other, less specialized consultations, such as those in general medical practice. Recommendations on the Tool’s use in the presence of comorbidities could be helpful and should improve the inter-rater reliability.

4.4. Study Limitations

The OLBI, like any other currently available PROM, cannot be considered as a gold standard for identifying OB. We considered the OLBI as an acceptable comparator, as it was shown to have better content and construct validity as well as internal consistency than other most-used PROMs [10]. Moreover, in accordance with a harmonized definition of OB [1], it measures the exhaustion dimension. Its use allowed us to compare our results with those of Hansez et al., using the same cutoff values. The latter was established based on the distribution of OLBI scores in Belgian workers [13] as a first step towards cutoff definition, but there is still no cutoff value allowing dichotomization of OLBI scores. Its forthcoming establishment should enable a true assessment of the Tool’s validity, both in Belgium and in Switzerland.
For unknown reasons, 19 of the 65 patients who consulted within this period (29.2%) did not complete the OLBI. A comparison of their sociodemographic data, dates, and motives for WSC with those of patients who did complete the OLBI, even incompletely (N = 4), did not reveal any significant difference (results not shown). This allows us to rule out a potential selection bias that might have occurred through the exclusion of patients with no or an incomplete OLBI.
A limitation of this study is that the raters completed the Tool on the basis of consultation reports and not real face-to-face consultations. This is not the way the Tool is meant to be used. However, the reports followed a standardized format and contained detailed and validated information on the patient. Moreover, this method had the advantage of presenting exactly the same material to all raters. This allowed us to truly test the robustness of the Tool. If, alternatively, we had asked the raters to use the Tool four times on the same patient, there could have been variations in the content of the consultation due to the repetitive questioning of the patient, the context of the consultation, or differences in the physician–patient interaction. Moreover, this would have been logistically difficult and would have prolonged the WSC.
Another limitation of this study is that the data collected from patients could not be updated to capture any significant changes in the sample, especially after the pandemic of COVID-19. Unfortunately, in the current form of “Work and Suffering” consultations, patients do not complete PROMs, and only anamnesis and clinical examination reports are available. Additionally, it is necessary to assess the validity and reliability of the Tool when comparing it against additional comparators with a precise and validated cutoff value—for instance, such as the CBI or Burnout Assessment Tool (BAT) [38] in future research.

5. Conclusions

The need for validated OB measurement tools has been stressed by both researchers and healthcare practitioners [2,20,39,40]. This study assessed the performance of the first OB measurement tool to rely on the report of healthcare professionals and not patients. According to our results, the Tool is promising. However, its detection of OB using a better comparator with a precise and validated cutoff value is necessary. The potential lack of specificity and inter-rater reliability could be rectified through an appropriate amendment of the Tool. The adapted version of the Tool could be further tested in practice, during face-to-face consultations, particularly in medical settings, which are not specialized in occupational health.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/ijerph182312300/s1, File S1: Tool for the early detection of occupational burnout.

Author Contributions

Conceptualization, I.H. and I.G.C.; methodology, I.G.C.; software, A.N.H. and E.E.M.; validation, I.G.C.; formal analysis, A.N.H., E.E.M. and I.G.C.; investigation, A.N.H., C.B., Z.M., E.E.M., I.H. and I.G.C.; resources, I.H. and I.G.C.; data curation, A.N.H. and I.G.C.; writing—original draft preparation, A.N.H. and I.G.C. writing—review and editing, A.N.H., C.B., Z.M., E.E.M., Y.S., I.H. and I.G.C.; visualization, A.N.H., C.B., Z.M., E.E.M., Y.S., I.H. and I.G.C.; supervision, project administration, and funding acquisition, I.G.C. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the General Directorate of Health of the Canton of Vaud via the grant of the Commission for Health Promotion and the Fight against Addictions (Grant N° 8273/3636000000-801).

Institutional Review Board Statement

The ethics committee for the canton of Vaud authorized this study and exempted investigators from collecting individual consent forms (CER-VD ID: 2019-02203).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

In accordance with ethics committee’s permission, the raw medical and diagnostic data cannot be provided. All other material used in this study is available as Supplementary Material.

Acknowledgments

We thank Franca Valceschini for her assistance in the patients’ records management.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Guseva Canu, I.; Marca, S.C.; Dell’Oro, F.; Balázs, Á.; Bergamaschi, E.; Besse, C.; Bianchi, R.; Bislimovska, J.; Bjelajac, A.K.; Buggez, M.; et al. Harmonized definition of occupational burnout: A systematic review, semantic analysis, and Delphi consensus in 29 countries. Scand. J. Work. Environ. Health 2021, 47, 95–107. [Google Scholar] [CrossRef] [PubMed]
  2. Rotenstein, L.; Torre, M.; Ramos, M.; Rosales, R.; Guille, C.; Sen, S.; Mata, D. Prevalence of burnout among physicians: A systematic review. J. Am. Med. Assoc. 2018, 320, 1131–1150. [Google Scholar] [CrossRef] [Green Version]
  3. Ackerley, G.; Burnell, J.; Holder, D.; Kurdek, L. Burnout among licensed psychologists. Prof. Psychol. Res. Pract. 1988, 19, 624–631. [Google Scholar] [CrossRef]
  4. Demerouti, E.; Bakker, A.; Nachreiner, F.; Schaufeli, W. The job demands-resources model of burnout. J. Appl. Psychol. 2001, 86, 499–512. [Google Scholar] [CrossRef] [PubMed]
  5. Kristensen, T.; Borritz, M.; Villadsen, E.; Christensen, K. The Copenhagen burnout inventory: A new tool for the assessment of burnout. Work Stress 2005, 19, 192–207. [Google Scholar] [CrossRef]
  6. Maslach, C.; Jackson, S.E.; Leiter, M.P. Maslach Burnout Inventory Manual, 3rd ed.; CPP, Inc: Mountain View, CA, USA, 1996. [Google Scholar]
  7. Pines, A.; Aronson, E. Burnout: From Tedium to Personal Growth; Freedom Press: New York, NY, USA, 1981. [Google Scholar]
  8. Marca, S.C.; Paatz, P.; Györkös, C.; Cuneo, F.; Bugge, M.D.; Godderis, L.; Bianchi, R.; Canu, I.G. Validation of questionnaires and rating scales used in medicine: Protocol for a systematic review of burnout self-reported measures. medRxiv 2020. [Google Scholar] [CrossRef]
  9. Mokkink, L.B.; Terwee, C.B.; Patrick, D.L.; Alonso, J.; Stratford, P.W.; Knol, D.L.; Bouter, L.M.; de Vet, H.C. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J. Clin. Epidemiol. 2010, 63, 737–745. [Google Scholar] [CrossRef] [Green Version]
  10. Shoman, Y.; Marca, S.; Bianchi, R.; Godderis, L.; van der Molen, H.; Guseva Canu, I. Psychometric properties of burnout measures: A systematic review. Epidemiol. Psychiatr. Sci. 2021, 30, E8. [Google Scholar] [CrossRef]
  11. Service Public Fédéral. Détection Précoce du Burnout: Outil Pour le Médecin Généraliste Burnout (Early Detection of Burnout: BPSEWS Tool for the General Physician); 2020; Available online: https://www.souffrance-et-travail.com/magazine/burn-out/outil-de-detection-precoce-du-burn-out/ (accessed on 18 November 2021).
  12. Hansez, I.; Mairiaux, P.; Firket, P.; Braeckman, L. Recherche sur le Burnout au Sein de la Population Active Belge: Rapport Final (Research on Burnout in the Active Belgian Population: Final Report); 2010; Available online: https://orbi.uliege.be/handle/2268/90274 (accessed on 18 November 2021).
  13. Hansez, I.; Rusu, D.; Firket, P.; Braeckman, L. Evolution 2010–2018 du Burnout en Belgique et Intérêt de L’utilisation Conjointe de Deux Outils de Diagnostic (Evolution 2010–2018 of Burnout in Belgium and the Benefit of a Joined Usage of Two Diagnostic BPSEWS Tools); 2019; Available online: https://emploi.belgique.be/sites/default/files/content/documents/Bien-%C3%AAtre%20au%20travail/Projets%20de%20recherche/rapport_final_SPF_burnout.pdf (accessed on 18 November 2021).
  14. Service Public Fédéral. Projets Pilote Prévention Intégrée Contre le Burnout (Pilot Projects for an Integrated Prevention of Burnout). Available online: https://socialsecurity.belgium.be/fr/elaboration-de-la-politique-sociale/projets-pilote-prevention-integree-contre-le-burn-out (accessed on 10 February 2020).
  15. Schaufeli, W.B.; Taris, T.W. The conceptualization and measurement of burnout: Common ground and worlds apart The views expressed in Work & Stress Commentaries are those of the author(s), and do not necessarily represent those of any other person or organization, or of the journal. Work Stress 2005, 19, 256–262. [Google Scholar]
  16. Hansez, I.; Firket, P.; Leclercq, C. Stades du Burnout et le Type de Prévention Approprié Selon le Stade. “Burnout: Les Clefs Pour Agir”. Available online: https://www.health.belgium.be/fr/stades-du-burnout-et-le-type-de-prevention-approprie-selon-le-stade (accessed on 12 December 2019).
  17. Leclercq, C.; Braeckman, L.; Firket, P.; Babic, A.; Hansez, I. Interest of a joint use of two diagnostic tools of burnout: Comparison between the Oldenburg burnout inventory and the early detection tool of burnout completed by physicians. Int. J. Environ. Res. Public Health 2021, 18, 10544. [Google Scholar] [CrossRef]
  18. Meirte, J.; Hellemans, N.; Anthonissen, M.; Denteneer, L.; Maertens, K.; Moortgat, P.; Van Daele, U. Benefits and disadvantages of electronic patient-reported outcome measures: Systematic review. JMIR Perioper. Med. 2020, 3, e15588. [Google Scholar] [CrossRef] [PubMed]
  19. Turner, G.M.; Litchfield, I.; Finnikin, S.; Aiyegbusi, O.L.; Calvert, M. General practitioners’ views on use of patient reported outcome measures in primary care: A cross-sectional survey and qualitative study. BMC Fam. Pract. 2020, 21, 14. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  20. Nguyen Huynh, A.; Béguelin, A.; Krief, P.; Marion-Veyron, R.; Mediouni, Z.; Regamey, F.; Staeger, P.; Guseva Canu, I. Detection and treatment of burnout by physicians of Unisanté. Revue Médicale Suisse 2021, 27, 221–224. [Google Scholar]
  21. Besse, C.; Grolimund Berset, D.; Studer, R.; Quarroz, S.; Praz-Christinaz, S.M.; Rivier, G.; Barlet-Ghaleb, C.; Danuser, B.; Bonsack, C. Consultation pluridisciplinaire “souffrance au travail”: Une exprérience romande (Multidisciplinary consultation “suffering at work”: An french-speaking Swiss experience). Revue Médicale Suisse 2016, 12, 276–279. [Google Scholar] [PubMed]
  22. Bouter, L.; Zielhuis, G.; Zeegers, M. Diagnostic and prognostic research. In Textbook of Epidemiology; Bohn, Stafleu van Loghum: Houten, The Netherlands; Vrije Universiteit Amsterdam: Amsterdam, The Netherlands, 2017; pp. 171–201. [Google Scholar]
  23. Nahavandi, K. Calculating sensitivity, specificity and predictive values for medical diagnostic tests. Gene Cell Tissue 2018, 5, 1–2. [Google Scholar] [CrossRef]
  24. Power, M.; Fell, G.; Wright, M. Principles for high-quality, high-value testing. BMJ Evid. Based Med. 2013, 18, 5–10. [Google Scholar] [CrossRef]
  25. StataCorp. Stata Statistical Software: Release 16; StataCorp LLC: College Station, TX, USA, 2019. [Google Scholar]
  26. Cohen, J. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 1960, 20, 37–46. [Google Scholar] [CrossRef]
  27. Light, R. Measures of response agreement for qualitative data: Some generalizations and alternatives. Psychol. Bull. 1971, 76, 365–377. [Google Scholar] [CrossRef]
  28. Hallgren, K. Computing inter-rater reliability for observational data: An overview and tutorial. Tutor. Quant. Methods Psychol. 2012, 8, 23–34. [Google Scholar] [CrossRef] [Green Version]
  29. Byrt, T.; Bishop, J.; Carlin, J. Bias, prevalence and kappa. J. Clin. Epidemiol. 1993, 46, 423–429. [Google Scholar] [CrossRef]
  30. Siegel, S.; Castellan, N.J. Non Parametric Statistics for the Behavioral Sciences; McGraw-Hill Book Company: New York, NY, USA, 1988. [Google Scholar]
  31. Landis, J.; Koch, G. The measurement of observer agreement for categorical data. Biometrics 1977, 33, 159–174. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  32. McGraw, K.O.; Wong, S.P. Forming inferences about some intraclass correlation coefficients. Psychol. Methods 1996, 1, 30–46. [Google Scholar] [CrossRef]
  33. Cicchetti, D. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol. Assess. 1994, 6, 284–290. [Google Scholar] [CrossRef]
  34. Team, R.D.C. R: A Language Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2019. [Google Scholar]
  35. Leiter, M.P.; Maslach, C. The impact of interpersonal environment on burnout and organizational commitment. J. Organ. Behav. 1988, 9, 297–308. [Google Scholar] [CrossRef]
  36. Mäkikangas, A.; Kinnunen, U. The person-oriented approach to burnout: A systematic review. Burn. Res. 2016, 3, 11–23. [Google Scholar] [CrossRef] [Green Version]
  37. World Health Organization. International Classification of Diseases for Mortality and Morbidity Statistics the Tenth Revision; WHO: Geneva, Switzerland, 2010. [Google Scholar]
  38. Schaufeli, W.B.; Desart, S.; De Witte, H. Burnout assessment tool (BAT)-development, validity, and reliability. Int. J. Environ. Res. Public Health 2020, 17, 9495. [Google Scholar] [CrossRef]
  39. Bianchi, R.; Schonfeld, I.; Laurent, E. Burnout–depression overlap: A review. Clin. Psychol. Rev. 2015, 36, 28–41. [Google Scholar] [CrossRef] [Green Version]
  40. Guseva Canu, I.; Mesot, O.; Györkös, C.; Mediouni, Z.; Mehlum, I.S.; Bugge, M.D. Burnout syndrome in Europe: Towards a harmonized approach in occupational health practice and research. Ind. Health 2019, 57, 745–752. [Google Scholar] [CrossRef] [Green Version]
Table
Table 1. Number of patients diagnosed with occupational burnout according to the reference and the tested methods.
Table 1. Number of patients diagnosed with occupational burnout according to the reference and the tested methods.
MethodDetection+Detection−
Reference method (cutoff > 30, occupational burnout of mild and high severity)41 (98%)1 (2%)
Reference method (cutoff > 44, occupational burnout of high severity only)25 (60%)17 (40%)
Tested method (binary medical detection of occupational burnout)34 (81%)8 (19%)
Table
Table 2. Validity of the Tool.
Table 2. Validity of the Tool.
Cutoff for OLBI 1 ScoreSensitivitySpecificityPPV 2NPV 3
>30, occupational burnout of mild and high severity0.83110.13
>44, occupational burnout of high severity only0.880.290.650.63
1 Oldenburg Burnout Inventory, 2 positive predictive value, 3 negative predictive value.
Table
Table 3. Inter-rater reliability of the symptoms identified and the detection reached by the four raters using the Tool.
Table 3. Inter-rater reliability of the symptoms identified and the detection reached by the four raters using the Tool.
VariableCohen’s KappaProblem(s)Corrected Kappa
Sleep problemsκCo = 0.71 ****PrevalenceκB = 0.90 *****
Lower energyκCo = 0.31 **PrevalenceκB = 0.18 *
Neuro-vegetative/functional complaintsκCo = 0.69 ****PrevalenceκB = 0.62 ****
Profound fatigueκCo = 0.66 ****PrevalenceκB = 0.72 ****
Lower motivation at workκCo = 0.56 ***PrevalenceκB = 0.26 **
BiasκS&C = 0.19 *
FrustrationκCo = 0.55 ***BiasκS&C = 0.21 **
IrritabilityκCo = 0.73 ****PrevalenceκB = 0.83 *****
Depressive moodκCo = 0.49 ***PrevalenceκB = 0.49 ***
BiasκS&C = 0.35 **
RuminationκCo = 0.63 ****PrevalenceκB = 0.83 *****
Ambivalence: leave job or stayκCo = 0.55 ***PrevalenceκB = 0.21 **
BiasκS&C = 0.13 *
AnxietyκCo = 0.41 ***PrevalenceκB = 0.66 ****
Lower self-esteemκCo = 0.78 ****--
Lower focusκCo = 0.86 *****--
Lower feeling of competenceκCo = 0.68 ****PrevalenceκB = 0.47 ***
BiasκS&C = 0.44 ***
Lower feeling of controlκCo = 0.54 *** PrevalenceκB = 0.18 *
BiasκS&C = 0.03 *
Lower memoryκCo = 0.95 *****PrevalenceκB = 0.90 *****
Lower idealismκCo = 0.72 ****PrevalenceκB = 0.45 ***
BiasκS&C = 0.06 *
Change in attitude towards othersκCo = 0.61 ****PrevalenceκB = 0.27 **
BiasκS&C = −0.05
Tendency to isolate oneselfκCo = 0.79 ****--
Absenteeism for illnessκCo = 0.50 ***PrevalenceκB = 0.43 ***
BiasκS&C = 0.44 ***
Lower performanceκCo = 0.73 ****PrevalenceκB = 0.49 ***
BiasκS&C = 0.35 **
Aggressiveness, impulsivityκCo = 0.70 ****BiasκS&C = 0.49 ***
Detection of occupational burnout κCo = 0.38 **PrevalenceκB = 0.21 **
BiasκS&C = 0.10 *
* slight agreement, ** fair agreement; *** moderate agreement, **** substantial agreement; ***** almost perfect/perfect agreement.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Nguyen Huynh, A.; Besse, C.; Mediouni, Z.; El May, E.; Shoman, Y.; Hansez, I.; Guseva Canu, I. Diagnostic Performances of an Occupational Burnout Detection Method Designed for Healthcare Professionals. Int. J. Environ. Res. Public Health 2021, 18, 12300. https://doi.org/10.3390/ijerph182312300

AMA Style

Nguyen Huynh A, Besse C, Mediouni Z, El May E, Shoman Y, Hansez I, Guseva Canu I. Diagnostic Performances of an Occupational Burnout Detection Method Designed for Healthcare Professionals. International Journal of Environmental Research and Public Health. 2021; 18(23):12300. https://doi.org/10.3390/ijerph182312300

Chicago/Turabian Style

Nguyen Huynh, Agathe, Christine Besse, Zakia Mediouni, Emna El May, Yara Shoman, Isabelle Hansez, and Irina Guseva Canu. 2021. "Diagnostic Performances of an Occupational Burnout Detection Method Designed for Healthcare Professionals" International Journal of Environmental Research and Public Health 18, no. 23: 12300. https://doi.org/10.3390/ijerph182312300

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop