Psychometric Properties of the Hospital Anxiety and Depression Scale (HADS) in Previously Hospitalized COVID-19 Patients

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus is associated with psychological/emotional disturbances. This study aimed to assess internal consistency, reliability, and construct validity of the Hospital Anxiety and Depressive Scale (HADS), as a patient-reported outcome measure (PROM) for evaluating emotional consequences of SARS-CoV-2 in hospitalized COVID-19 survivors with long COVID. The LONG-COVID-EXP-CM is a multicenter cohort study including patients hospitalized by COVID-19 during the first wave of the pandemic in five hospitals in Madrid. A total of 1969 (age: 61 ± 16 years, 46.5% women) COVID-19 survivors experiencing post-COVID symptoms a mean of 8.4 ± 1.5 months after hospital discharge completed HADS. Internal consistency (Cronbach α), reliability (item-internal consistency, item-discriminant validity), construct validity (confirmatory factor analysis), and floor effect and ceiling effect were calculated. The mean time for fulfilling HADS was 65 ± 12 s. A ceiling effect ranging from 1.99% to 13.74% and a floor effect ranging from 43.05% to 77.77% was observed. Based on the item-scale correlation coefficients, the Cronbach’s alpha values reflecting the internal consistency reliability were 0.890 for the anxiety scale (HADS-A) and 0.856 for the depressive scale (HADS-D) The correlation coefficient between HADS-A and HADS-D scores was excellent (r: 0.878). The confirmatory factor analysis revealed that five out of the seven fitness indexes were excellent: CFI = 0.969, NNFI = 0.963; TLI = 0.963; AGFI = 0.951; GFI = 0.972), supporting good construct validity. In conclusion, this study indicates that both anxiety and depressive symptoms scales of HADS had overall good psychometric properties to be used for assessing psychological and emotional stress in COVID-19 survivors with long COVID.


Introduction
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus, responsible for causing coronavirus disease 2019 (COVID- 19), mainly affects the respiratory system; however, multisystem effects associated with the development of a plethora of symptoms is also present in most patients [1]. As well as biologically related symptoms, emotional symptoms have been also reported by these patients which supports the observation that biological and behavioral factors interact in COVID-19 context [2]. For instance, the prevalence of anxiety and depressive symptoms has been reported to be 38% in people affected by SARS-CoV-2 virus [3]. Furthermore, anxiety and depressive symptoms are also prevalent after the acute phase of the infection (post-COVID phase) [4]. Interestingly, Bottemanne et al. recently reported that depressive levels after an acute COVID-19 episode are associated with an increased risk of physical post-COVID symptoms such as pain and dyspnea [5].
In addition, the prevalence of anxiety and depressive symptoms has also increased in people not infected by COVID-19. For instance, the prevalence of anxiety/depressive levels in the general population after the COVID-19 outbreak was higher when compared with pre-pandemic data [6]. Similarly, up to 18% of relatives [7] or 38% of healthcare professionals (i.e., nurses) [8] attending to COVID-19 patients also exhibited high levels of anxiety and depression. Therefore, evaluation of anxiety or depressive levels within the COVID era is essential.
Patient-reported outcome measures (PROM) consist of generic or disease-specific self-reported questionnaires evaluating different aspects of a particular condition. Some disease-specific PROMs such as the post-COVID-19 functional status scale (PCFS) have been developed for evaluating the functional capabilities of COVID-19 survivors [9,10]. However, this PROM omits the emotional and psychological aspect of the condition [9,10].
The Hospital Anxiety and Depression Scale (HADS) was originally developed in 1983 to identify the presence of anxiety disorders and depressive symptoms in people in nonpsychiatric hospital clinics [11]. Bjelland et al. found good psychometric properties of both the anxiety and depression scales of HADS for assessing for anxiety disorders and depressive symptoms within the general population and also in psychiatric patients [12]. More recently, studies have confirmed the internal consistency and validity of HADS in a general population aged 65-80 years old [13].
HADS is the PROM most used in COVID-19 research for evaluating anxiety and depressive symptoms [3,4]. However, the psychometric properties in the COVID-19 context has not been properly investigated. Tasnim et al. found that HADS had good psychometric properties for evaluating anxiety and depressive levels in healthcare workers who were in the frontline during the first COVID-19 outbreak [14]. No previous study has evaluated the psychometric properties of HADS in people with long COVID. The current study aimed to describe internal consistency, reliability, and construct validity of HADS in previously hospitalized COVID-19 survivors.

Participants
An instrumental research design evaluating the psychometric data of HADS were used [15]. We used data collected from a multicenter cohort study (LONG-COVID-EXP-CM) including patients hospitalized by COVID-19 during the first wave of the pandemic (from 10 March to 31 May 2020) in five urban hospitals of Madrid (Spain). All diagnoses were conducted with real-time reverse transcription-polymerase chain reaction (RT-PCR) assay of nasopharyngeal/oral swab samples as well as clinical and radiological findings at hospital admission. Patients discharged from the involved hospitals were included in an anonymous database and a random selection of 400 patients from each hospital was performed with randomization software. This multicenter study design was approved by the Local Ethics Committees of all the involved institutions (URJC0907202015920, HCSC20/495E, HUFA 20/126, HUIL/092-20, HUF/EC1517, HSO25112020). Participants provided their informed consent before collecting any data. It should be noted that the participants from the LONG-COVID-EXP-CM study have been referred to in previous letters to the editor or publications [16][17][18][19][20][21] but current data presented here are completely new and not previously published.

COVID-19 and Post-COVID Data Collection
Demographic (e.g., age, gender, height, weight), clinical (e.g., previous medical comorbidities), and hospitalization (COVID-19 associated-onset symptoms experienced at hospital admission, intensive care unit admission, days at hospital) data were collected from hospital records. Participants were scheduled for a telephone interview conducted by healthcare researchers and were systematically asked about the presence of symptoms that they experienced after hospital discharge (post-COVID-19 related symptoms). A predefined list of symptoms including dyspnea, fatigue, throat pain, cough, anosmia, ageusia, hair loss, skin rashes, diarrhea, palpitations, brain fog, concentration loss, or ocular disorders was used, but participants were free to report any symptom not included in the list and experienced at the time of the assessment.

Hospital Anxiety and Depression Scale (HADS)
Both the anxiety (HADS-A) and depression (HADS-D) subscales of HADS were assessed. It has been found that HADS can be properly evaluated by telephone [22]. HADS-A consists of 7 items assessing anxiety symptoms whereas HADS-D consists of 7 items evaluating depressive symptoms. Each item is scored on a 4-point Likert scale (0-3) providing a maximum of 21 points for each subscale [23]. We applied a cut-off score of ≥8 points for each scale since this value has shown good sensitivity and specificity to determine the presence of anxiety or depressive symptoms, respectively [24].

Statistical Analysis
Statistical analysis was performed with STATA 16.1 program (StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX: StataCorp LP. USA). Proportions, means, and standard deviations were used to describe the study population. Since all items from HADS are answered into a Likert scale, we present the percentage of those individuals answering each item. Statistical significance was defined when p-value < 0.05. We tested internal consistency, reliability, and construct-validity properties of HADS according to the consensus-based standards for the selection of health measurement instruments (COSMIN) [25].
The Cronbach's alpha [26] and the Raykow Omega coefficient [27] were calculated to determine internal consistency, where values between 0.7 and 0.95 were considered as good internal consistency. Reliability of HADS was evaluated by calculating item-internal consistency, i.e., testing the correlation between each item and its hypothesized scale with correction for overlap. A correlation coefficient of 0.4 supported item-internal consistency. Furthermore, item-discriminant validity was supported if the correlation between an item and its hypothesized scale was higher than its correlation with all other scales. Construct validity was examined with a confirmatory factor analysis (CFA). The following packages were used: lavaan [28] for CFA analysis and semPlot [29] for visualizing the CFA paths. CFA was first used to assess the fit of the proposed measurement model using a two-factor model (anxiety, and depression). The weighted least square mean and variance adjusted (WLSMV) was used to estimate the model's parameters, whilst the robust standard errors were used. The absolute fit (i.e., root mean square error of approximation [RMSEA], standardized root mean square residual [SRMR], goodness of fit index [GFI]), and the incremental fit (i.e., adjusted goodness of fit index [AGFI], comparative fit index [CFI], Tucker-Lewis index [TLI], normed fit index [NFI]) were observed for the model fit estimation. An excellent model fit is determined when at least two of the four fit indices exceed the following thresholds: RMSEA ≤ 0.05; SRMR ≤ 0.05; CFI ≥ 0.95; NNFI ≥ 0.95 [30]. Hu and Bentler [31] proposed also the thresholds for the following indices: TLI ≥ 0.95, CFI ≥ 0.90, AGFI ≥ 0.90 and GFI ≥ 0.95. We also calculated the percentage of subjects achieving the highest (floor effect) and the lowest (ceiling effect) scores on each question of HADS. Finally, chi-square tests were conducted to determine differences by gender and age (grouped as <45 years, 45-59 years, 60-69 years, and ≥70 years).

General Data
The mean time for fulfilling HADS was 65 (SD 12) seconds. All questions were answered by participants. The mean HADS-A score was 4.9 (SD 5.2), whereas the HADS-D score was 4.7 (SD 4.8). The percentage of data at ceiling (nil dysfunction) ranged from 1.99% to 13.74% and the percentage of data at floor (maximal dysfunction) ranged from 43.05% to 77.77% (Table 2). A greater (p < 0.001) proportion of women exhibited depressive levels (HADS-D ≥ 8 points) as compared to men. No differences (p = 0.637) in the proportion of women and men reporting anxiety levels (HADS-A ≥ 8 points) existed (Figure 1). Significant differences across the age-groups were observed for the proportion of those with anxiety (p < 0.001) and depressive (p = 0.003) levels: individuals aged 60-70 years reported higher anxiety and depressive levels than the other age-groups ( Figure 2).

Reliability and Internal Consistency
The item-internal consistency ranged from 0.520 to 0.888 for HADS-A items and from 0.659 to 0.833 for HADS-D items, suggesting moderate to excellent consistency ( Table 2). The correlation of each of the items with its hypothesized scale was smaller than its correlation with the other scale; accordingly item-discriminant validity was not supported by current data. In fact, item-reliability coefficients were smaller than the correlation coefficient between the HADS-A and HADS-D scales (r: 0.878) The Cronbach's α value of the HADS-A score was 0.890 whereas the Cronbach's α of the HADS-D score was 0.856, supporting good internal consistency (Table 2). Additionally, the Omega coefficient of the HADS-A score was 0.911 whereas the Omega coefficient of the HADS-D score was 0.821 (Table 2). Table 3 details the tested measurement model and associated regression weights of the CFI. Five out of the seven fitness indexes were excellent: CFI = 0.969, NNFI = 0.963; TLI = 0.963; AGFI = 0.951; GFI = 0.972), suggesting that that fit for the measurement model was excellent. However, two fitness indexes did not satisfy their excellent thresholds: RMSEA = 0.217, SRMR = 0.182. Figure 3 presents structural equation modeling (SEM) of HADS in previously hospitalized COVID-19 survivors, and demonstrates that the two subscales (anxiety and depression) were positively and significantly correlated with each other. The factor loading of each item of HADS was also acceptable.

Discussion
Several symptoms perceived after the acute phase of SARS-CoV-2 infection, i.e., post-COVID, are physical; however, psychological or emotional disturbances are also present [3,4]. The heterogeneity manifestations of long COVID supports the need for an assessment covering organic and emotional aspects. HADS is a PROM commonly used for evaluating anxiety and depressive symptoms in different population including COVID-19 survivors [3,4]. The current study evaluated the psychometric properties of HADS in previously hospitalized COVID-19 survivors with long COVID. The results reveal that HADS has appropriate floor and ceiling effects, was internally consistent and reliable, and has overall construct validity for assessing anxiety and depressive symptoms in people with long COVID. These psychometric properties are similar to data published on the general population [12] and in healthcare workers who were in the frontline during the COVID-19 outbreak [14]. Accordingly, HADS could be considered as a comprehensive and an easy PROM to be applied in patients with long COVID since it takes around 60 sec to be fulfilled.
HADS was originally created as two subscales focusing on anxiety and depressive symptoms [11]. Although the current study did not attempt to identify the presence of anxiety and depressive symptoms in our sample of individuals with long COVID, current evidence supported that a cut-off value ≥ 7 points on HADS-D had a sensitivity of 0.82 and specificity of 0.78 whereas a cut-off value ≥ 8 points showed a sensitivity of 0.74 and a specificity of 0.84 for identifying depressive symptoms [32]. We identified that almost 20% of our sample experienced symptoms of depression based on the cut-off value of HADS-D ≥ 8 points. In fact, our study found both scales exhibited overall good psychometric data in people with long COVID. However, we should recognize that, although most goodness of fit indexes were excellent, the RMSEA was high, suggesting that the model contained some sort of misspecification, e.g., redundancy among items. These data are similar to those previously reported by Norton et al. [33]. These authors pooled data from 28 different samples obtained from published studies concerning the latent structure of HADS and conducted several analyses [33]. Current evidence supports that a bifactor structure provides the most acceptable empirical explanation for the HADS correlation structure; however, due to the presence of a strong general factor, HADS does not provide a good separation between symptoms of anxiety and depression, but it is likely that this is a problem for most instruments where symptoms of anxiety and depression are jointly measured [33]. In line with this situation, LoMartire et al. have recently reported that rather than using both subscales of anxiety and depression separately, HADS as a single score is also a valid and reliable PROM of overall emotional distress, at least in people with chronic pain [34].
Women exhibited more depression, but not anxiety, symptoms than men, supporting current assumptions that females exhibit more emotional post-COVID symptoms [19,35]. The lack of gender differences in anxiety in the current study may be related to the fact that our sample included individuals hospitalized during the first wave of the COVID-19 outbreak. Recent studies have reported that depression and anxiety symptoms were more pronounced during the first wave of the COVID-19 outbreak than during the second [36] or later [37] waves. We also observed that individuals aged 60-70 years reported more anxiety and depression than other age-groups. Since patients from our study were hospitalized during the first wave of the COVID-19 outbreak, it is possible that individuals in this age group perceived the first outbreak as more of a threat than other age-groups due to a higher self-perception of frailty against SARS-CoV-2 virus. Nevertheless, this hypothesis should have been more evident in the oldest group. Future studies should test this hypothesis.
The results from this study should be considered according to their strengths and limitations. The main strength consists of the inclusion of a large sample of COVID-19 survivors from different centers and with a long-term follow-up after hospital discharge. However, the main limitation is the inclusion of only hospitalized COVID-19 survivors. We do not know if HADS would exhibit similar psychometric data in non-hospitalized subjects, where emotional aspects could be less relevant than in those hospitalized. Additionally, although we assessed patients at a long-term follow-up period after hospitalization, we did not collect the emotional status of these patients before acute infection or during the first months after hospitalization. Studies investigating the longitudinal evolution of anxiety or depressive symptoms in this population are needed. In fact, longitudinal evolution of depressive and anxiety symptoms during the COVID-19 era is conflicting. Gambin et al. found that the increased anxiety/depressive levels occurring at the early stages of the first COVID-19 lockdown decreased in a small proportion of subjects, being resilient or chronic in most of them [38]. On the contrary, Saunders et al. observed that most individuals with anxiety or depression symptoms evolved positively during the first week after the COVID-19 lockdown [39]. The only study investigating the longitudinal evolution of anxiety and depressive symptoms in subjects with long COVID found that post-COVID anxiety and depressive symptoms tended to slowly recover during the following five years after the SARS-CoV-2 infection [40]. Finally, it is important to consider limitations related to the methodological design of the study. This study only analyzed the psychometric properties of HADS in a specific population such as individuals with long COVID. Although we presented properties including internal consistency, item-internal consistency, itemdiscriminant validity and construct validity, we did not compare HADS with other PROMs and we did not provide test-retest reliability.

Conclusions
This study suggests that HADS exhibits overall good psychometric properties, appropriate floor and ceiling effects, good internal consistency and reliability, and construct validity, to be used as a PROM to evaluate the presence of anxiety and depressive symp-toms in previously hospitalized COVID-19 survivors; nevertheless, values of some indexes suggest that some items can be redundant. Additionally, other PROMs evaluating physical impairments should be used complementarily with HADS for evaluating long COVID patients. The systematic use of HADS in future studies of long COVID could help worldwide comparisons of the presence of mood disorders in this population. Funding: The LONG-COVID-EXP-CM is supported by a grant of Comunidad de Madrid y la Unión Europea, a través del Fondo Europeo de Desarrollo Regional (FEDER), Recursos REACT-UE del Programa Operativo de Madrid 2014-2020, financiado como parte de la respuesta de la Unión a la pandemia de COVID-19. The sponsor had no role in the design, collection, management, analysis, or interpretation of the data, draft, review, or approval of the manuscript or its content. The authors were responsible for the decision to submit the manuscript for publication, and the sponsor did not participate in this decision.