Agreement Between Medico-Administrative Database Algorithms and Survey-Based Diagnoses for Depression and Anxiety in Older Adults

Abstract

Background/Objectives: This study aimed to assess the concordance between depression and anxiety case definitions derived from algorithms based on medico-administrative data and structured interviews aligned with the fourth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) criteria in older adults. Methods: We analyzed data from 1405 primary care older adults (≥65 years) from the Étude sur la Santé des Aînés (ESA)-Services cohort (2011–2013) in Quebec, Canada, who had available survey and medico-administrative data. Cases of depression and anxiety were identified using algorithms incorporating combinations of hospitalization records, physician-visit claims, and medication claims for antidepressants or anxiolytics. The agreement was assessed with the kappa statistics (κ), and the algorithms’ sensitivity, specificity, and positive and negative predictive values were calculated using the case definitions derived from the DSM-IV-aligned ESA-Services interviews as the gold standard. Results: Agreements between the algorithms and the interviews were fair (κ: 0.06–0.22) for depression gooand slight (κ: 0.02–0.09) for anxiety. The algorithms had low sensitivity (2–39.7% for depression and 1.4–39.9% for anxiety) but high specificity (84.5–99.6% for depression and 73–99.2% for anxiety), depending on the algorithm. Conclusions: The agreement between algorithms based on administrative data and DSM-IV-aligned interviews for anxiety or depressive disorders was low. The two methods identified older adults with different characteristics. Despite these discrepancies, algorithms with high specificity provide valuable insights into healthcare utilization patterns associated with these disorders.

1. Introduction

Common mental disorders, such as depression and anxiety, are increasingly prevalent globally due to population ageing [1]. In Canada, one in five individuals experiences mental health problems, with mood and anxiety disorders being the most common [2]. A population-based study reported that 13% of older adults over 65 years experienced depression or anxiety in the past year [3]. Despite the high prevalence rate, population-based prevalence studies on depression and anxiety are still rare, likely due to methodological challenges [4,5,6]. Such studies, however, are essential, as mental disorders adversely affect patients’ abilities to manage their chronic conditions and their overall quality of life [7,8,9,10].

There are many ways to gather information about mental health disorders [11]. Structured diagnostic interviews and clinical assessments based on patient reports are commonly used in clinical settings. These methods are accurate but can be time-consuming, expensive, and dependent on accurate recall, posing challenges for large-scale epidemiological research [12]. Alternatively, routinely collected medico-administrative data offer less costly and more straightforward means of gathering timely information on mental health disorders [13].

Given the substantial burden and associated healthcare costs of depression and anxiety [14], researchers have sought to validate algorithms for identifying these disorders using medico-administrative data [15,16,17]. However, the accuracy of these algorithms significantly depends on data quality, coding procedures, prescription practices, and the selected case definitions. Previous studies have demonstrated poor-to-modest concordance between medico-administrative data and structured interviews like the Composite International Diagnostic Interview (CIDI) or its short form (CIDI-SF) [18,19].

In Canada, medico-administrative data from provincial and territorial chronic disease surveillance systems have informed mental health research and policy through the Canadian Chronic Disease Surveillance System (CCDSS) [20,21]. These databases provide valuable demographic and clinical data on residents covered by public healthcare plans, capturing detailed medical consultations, hospitalizations, and prescription claims. Analysis of these databases enables researchers and policymakers to evaluate illness patterns, healthcare disparities, and variations in treatment delivery, informing decisions on resource allocation and public health planning.

Therefore, this study aimed to assess the agreement between case definitions of depression, anxiety, and anxiety-depressive disorders among older adults in the province of Quebec using medico-administrative databases and the presence of these disorders based on criteria aligned with the fourth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) collected during a structured survey interview.

2. Results

Of the initial 1810 participants in the ESA-S study, 1657 individuals (91.6%) consented and were successfully matched with administrative data from RAMQ. Among these, 1436 (86.7%) had valid public insurance coverage between one year before and one year after the ESA-S interview. Therefore, this analysis included 1405 participants, as shown in Figure 1. Participants were between 65 and 97 years old, with a median age of 73.

Table 1. Distribution of mental health disorder cases according to the ESA-S interview, overall and by sex.

Characteristics	All N (%)	Women N (%)	Men N (%)	p Value
Total	1405	813 (57.9)	592 (42.1)
Depressive disorder *	204 (14.5)	159 (19.6)	45 (7.6)	<0.0001
Major depression	148 (10.5)	113 (13.9)	35 (5.9)	<0.0001
Minor depression	56 (4)	46 (5.7)	10 (1.7)	<0.0001
Anxiety disorder (all) *	213 (15.2)	159 (19.6)	54 (9.1)	<0.0001
GAD	47 (3.4)	38 (4.7)	9 (1.5)	0.0012
PTSS	60 (4.3)	46 (5.6)	14 (2.4)	0.0026
Social phobia	29 (2.1)	15 (1.9)	14 (2.4)	0.4986
Specific phobia	87 (6.2)	64 (7.9)	23 (3.9)	0.0022
Agoraphobia	23 (1.6)	18 (2.2)	5 (0.8)	0.0459
Panic disorder	20 (1.4)	14 (1.7)	6 (1.0)	0.2683
OCD	58 (4.1)	30 (3.7)	28 (4.7)	0.3841

* According to the ESA-S interview, GAD: generalized anxiety disorder, PTSS: post-traumatic stress syndrome, OCD: obsessive-compulsive disorder.

shows the prevalence of mental health disorders among participants overall and by sex, based on ESA-S interviews. Overall, 14.5% (204 participants) had depression (major and minor combined), with women being 2.5 times more likely to have depression than men (19.6% vs. 7.6%). The prevalence of major depression was 10.5% overall (148 participants), again higher in women (13.9%) than men (5.9%).

The prevalence of anxiety disorders was 15.2% overall (213 participants), significantly higher in women compared to men (19.6% vs. 9.1%). This sex difference was consistent for most anxiety disorders except social phobia and panic disorder, where no significant differences were observed.

Using administrative data, the least restrictive depression algorithm (≥1 hospitalization, ≥1 medical consultation, or ≥1 antidepressant claim within 6 months prior to the interview) indicated a prevalence of 18.9% (266 participants), higher in women than men (24.5% vs. 11.3%). The most restrictive algorithm (≥2 physician visits or ≥1 hospitalization within 6 months) showed a lower prevalence of 0.6% (nine participants), for all women. A total of 3.5% (49 participants) had ≥1 physician visit with a diagnostic code for depression in the previous 6 months, more frequently in women (4.4%) than men (2.2%). Only 0.4% (five participants) were hospitalized for depression (all were women). Medication claims for antidepressants were found in 17.7% (249 participants), again higher among women (23.1%) compared to men (10.3%).

The algorithm performance metrics for depression are detailed in

Table 2. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for past 6-month depression case definition algorithms.

Algorithm	Sensitivity [95% CI]	Specificity [95% CI]	PPV [95% CI]	NPV [95% CI]	Kappa [95% CI]
≥1 hospitalization or ≥1 medical consultation OR ≥1 antidepressant claim in 6 months *	39.7 [33–46.4]	84.5 [82.5–86.6]	30.3 [24.8–35.9]	89.2 [87.4–91]	0.22 [0.15–0.28]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 antidepressant claims in 6 months	37.3 [30.6–43.9]	85.4 [83.4–87.4]	30.3 [24.6–36]	88.9 [87.1–90.7]	0.21 [0.14–0.27]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 antidepressant claims in 6 months	33.8 [27.3–40.3]	86.9 [85–88.8]	30.5 [24.5–36.5]	88.6 [86.7–90.4]	0.20 [0.13–0.26]
≥1 hospitalization OR ≥1 medical consultation in 6 months	10.3 [6.1–14.5]	97.3 [96.4–98.3]	39.6 [26.5–52.8]	86.5 [84.6–88.2]	0.11 [0.05–0.17]
≥1 hospitalization OR ≥2 medical consultations in 6 months **	2 [0–3.9]	99.6 [99.2–100]	44 [12–76.9]	85.7 [83.8–87.5]	0.03 [0–0.05]
≥1 hospitalization OR ≥ (1 medical consultations AND ≥1 antidepressant) claims in 6 months	8.3 [4.5–12.1]	98.4 [97.7–99.1]	47.2 [30.9–63.5]	86.3 [84.5–88.2]	0.10 [0.05–0.16]
≥1 hospitalization or ≥1 medical consultation OR ≥1 antidepressant claim in 12 months *	41.2 [34.4–47.9]	83.9 [81.8–85.9]	30.2 [24.8–35.6]	89.4 [87.6–91.2]	0.22 [0.16–0.28]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 antidepressant claims in 12 months	39.7 [33–46.4]	84.5 [82.5–86.6]	30.3 [24.8–35.9]	89.2 [87.4–91.0]	0.21 [0.15–0.28]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 antidepressant claims in 12 months	37.3 [30.6–43.9]	85.4 [83.5–87.4]	30.3 [24.6–36]	88.9 [87.1–90.7]	0.21 [0.14–0.27]
≥1 hospitalization OR ≥1 medical consultation in 12 months	14.2 [9.4–19]	96.7 [95.7–97.7]	42 [30.4–53.7]	86.9 [85.1–88.7]	0.15 [0.08–0.21]
≥1 hospitalization OR ≥2 medical consultations in 12 months **	3.9 [1.3–6.6]	99.6 [99.2–100]	61.5 [35.1–88]	85.9 [84.1–87.8]	0.06 [0.01–0.10]
≥1 hospitalization OR (≥1 medical consultation AND ≥1 antidepressant) claims in 12 months	14.2 [8.6–19.8]	97.7 [96.9–98.5]	42 [28.3–55.7]	90.6 [89.1–92.2]	0.17 [0.09–0.24]
≥1 hospitalization or ≥1 medical consultation OR ≥1 antidepressant claim in 6 months before and after interview	44.1 [37.3–50.9]	82.6 [80.5–84.7]	30.1 [24.9–35.3]	89.7 [87.9–91.5]	0.22 [0.16–0.28]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 antidepressant claims in 6 months before and after interview	43.1 [36.3–49.9]	83.2 [81.1–85.3]	30.4 [25.1–35.6]	89.6 [87.8–91.4]	0.22 [0.16–0.28]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 antidepressant claims in 6 months before and after interview	40.7 [34–47.4]	84.4 [82.4–86.5]	30.7 [25.2–36.2]	89.3 [87.5–91.1]	0.22 [0.16–0.28]
≥1 hospitalization OR ≥1 medical consultation in 6 months before and after interview	12.7 [8.2–17.3]	95.9 [94.8–97]	34.7 [23.9–45.4]	86.6 [84.8–88.5]	0.12 [0.06–0.18]
≥1 hospitalization OR ≥2 medical consultations in 6 months before and after interview **	2.5 [0.3–4.6]	98.5 [97.8–99.2]	21.7 [4.9–38.6]	85.6 [83.8–87.5]	0.01 [−0.02–0.05]
≥1 hospitalization OR ≥ (1 medical consultations AND ≥1 antidepressant) claims in 6 months before and after interview	11.7 [7.3–16.2]	97.6 [96.7–98.5]	45.3 [31.8–58.7]	86.7 [84.8–88.5]	0.13 [0.07–0.20]

PPV: positive predictive value; NPV: negative predictive value; CI: confidence interval, * least restrictive algorithm, ** most restrictive algorithm.

. The least restrictive algorithm within 6 months had a sensitivity of 39.7%, specificity of 84.5%, PPV of 30.3%, and NPV of 89.2%, with fair agreement (κ = 0.22). The most restrictive algorithm within 6 months showed very low sensitivity (2%) but high specificity (99.6%), better PPV (44%), and slight agreement (κ = 0.06).

To determine whether administrative data are better suited for identifying major depression, we stratified our analysis by depression severity.

Table 3. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for past 6-month depression case (major depression) definition algorithms.

Algorithm	Sensitivity [95% CI]	Specificity [95% CI]	PPV [95% CI]	NPV [95% CI]	Kappa [95% CI]
≥1 hospitalization or ≥1 medical consultation OR ≥1 antidepressant claim in 6 months *	41.2 [33.3–49.2]	83 [81.7–85.7]	22.9 [17.8–28]	92.4 [90.8–93.9]	0.18 [0.12–0.25]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 antidepressant claims in 6 months	38.5 [30.7–46.4]	84.3 [82.2–86.3]	22.4 [17.2–27.5]	92.1 [90.5–93.7]	0.17 [0.11–0.23]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 antidepressant claims in 6 months	37.2 [29.4–45]	86.4 [84.5–88.3]	24.3 [18.7–29.9]	92.1 [90.6–93.7]	0.19 [0.13–0.26]
≥1 hospitalization OR ≥1 medical consultation in 6 months	12.2 [6.9–17.4]	97.2 [96.3–98.1]	33.9 [21.2–46.7]	90.4 [88.8–92]	0.11 [0.05–0.17]
≥1 hospitalization OR ≥2 medical consultations in 6 months **	2.7 [0–5.3]	99.6 [99.3–100]	44 [12–76.9]	89.7 [88.1–91.3]	0.04 [0–0.08]
≥1 hospitalization OR (≥1 medical consultations AND ≥1 antidepressant) claims in 6 months	10.1 [5.3–15.1]	98 [97.2–98.8]	37.5 [22.5–52.5]	90.3 [88.7–91.8]	0.12 [0.05–0.19]
≥1 hospitalization or ≥1 medical consultation OR ≥1 antidepressant claim in 12 months *	42.6 [34.6–50.5]	82.9 [80.8–85]	22.7 [17.7–27.6]	92.5 [90.9–94]	0.18 [0.12–0.24]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 antidepressant claims in 12 months	40.5 [32.6–48.5]	85.5 [81.5–85.6]	22.5 [17.5–27.5]	92.4 [90.7–93.8]	0.18 [0.12–0.24]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 antidepressant claims in 12 months	39.2 [31.3–47.1]	84.7 [82.7–86.6]	23.1 [17.9–28.3]	92.2 [90.7–93.8]	0.18 [0.12–0.24]
≥1 hospitalization OR ≥1 medical consultation in 12 months	16.9 [10.9–22.9]	96.5 [95.5–97.5]	36.2 [24.9–47.6]	90.8 [89.2–92.3]	0.17 [0.09–0.25]
≥1 hospitalization OR ≥2 medical consultations in 12 months **	5.4 [1.8–9.1]	99.6 [99.3–100]	61.5 [35.1–88]	89.9 [88.4–91.5]	0.08 [0.02–0.14]
≥1 hospitalization OR (≥1 medical consultations AND ≥1 antidepressant) claims in 12 months	10.1 [5.3–15]	98 [97.2–98.8]	37.5 [22.5–52.5]	90.3 [88.7–91.8]	0.12 [0.05–0.19]
≥1 hospitalization or ≥1 medical consultation OR ≥1 antidepressant claim in 6 months before and after interview *	47.3 [39.2–55.3]	81 [79.7–83.9]	23.4 [18.6–28.2]	93 [91.4–94.5]	0.20 [0.14–0.26]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 antidepressant claims in 6 months before and after interview	46 [37.9–54]	82.3 [80.2–84.5]	23.5 [18.6–28.3]	92.8 [91.3–94.3]	0.19 [0.14–0.26]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 antidepressant claims in 6 months before and after interview	43.9 [35.9–51.9]	83.7 [81.7–85.7]	24.1 [19–29.2]	92.7 [91.2–94.2]	0.20 [0.14–0.26]
≥1 hospitalization OR ≥1 medical consultation in 6 months before and after interview	15.5 [9.7–21.4]	95.9 [94.8–97]	30.7 [20.2–41.1]	90.6 [89–92.2]	0.15 [0.07–0.22]
≥1 hospitalization OR ≥2 medical consultations in 6 months before and after interview **	3.4 [0.5–6.3]	98.6 [97.9–99.2]	21.7 [4.9–38.6]	89.7 [88.1–91.3]	0.03 [−0.01–0.07]
≥1 hospitalization OR (≥1 medical consultations AND ≥1 antidepressant) claims in 6 months before and after interview	11.5 [6.4–16.6]	97.1 [96.2–98.1]	32.1 [19.5–44.6]	90.3 [88.7–91.9]	0.12 [0.05–0.19]

PPV: positive predictive value; NPV: negative predictive value; CI: confidence interval, * least restrictive algorithm, ** most restrictive algorithm.

displays the performance of the algorithms when applied exclusively to cases of major depression.

For anxiety, the least restrictive algorithm (≥1 hospitalization, or ≥1 medical consultation, or ≥1 anxiolytic claim within 6 months prior to the interview) showed a prevalence of 23.5% (331 participants), with women having a higher prevalence (29.8%) than men (14.9%). The most restrictive algorithm (≥2 physician visits or ≥1 hospitalization) showed a lower prevalence of 0.9% (13 participants), also higher in women (1.2%) than men (0.5%). Physician visits with anxiety codes occurred in 8.9% (125 participants), more commonly among women (10.3%) than men (6.9%). Hospitalizations were rare (0.5%, seven participants). Anxiolytic medication claims were common (23.2%, 326 participants), significantly higher among women (29.4%) than men (14.7%). Physician visits with anxiety codes occurred in 8.9% (125 participants), more commonly among women (10.3%) than men (6.9%). Hospitalizations were rare (0.5%, seven participants). Anxiolytic medication claims were common (23.2%, 326 participants), and significantly higher among women (29.4%) than men (14.7%).

Algorithm performance metrics for anxiety are shown in

Table 4. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for past 6-month anxiety case definition algorithms.

Algorithm	Sensitivity [95% CI]	Specificity [95% CI]	PPV [95% CI]	NPV [95% CI]	Kappa [95% CI]
≥1 hospitalization OR ≥1 medical consultation OR ≥1 anxiolytic claim in 6 months *	39.9 [33.3–46.5]	73 [70.5–75.6]	21 [16.9–24.8]	87.2 [85.1–89.3]	0.09 [0.04–0.15]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 anxiolytic claims in 6 months	33.8 [27.5–40.2]	77.2 [74.8–79.6]	20.9 [16.6–25.2]	86.7 [84.7–88.8]	0.09 [0.03–0.14]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 anxiolytic claims in 6 months	28.6 [22.6–34.7]	82.1 [80–84.3]	22.3 [17.3–27.2]	86.6 [84.6–88.6]	0.10 [0.04–0.15]
≥1 hospitalization OR ≥1 medical consultation in 6 months	12.7 [8.2–17.1]	91.2 [89.6–92.8]	20.5 [13.6–27.3]	85.4 [83.5–87.3]	0.05 [−0.01–0.10]
≥1 hospitalization OR ≥2 medical consultations in 6 months **	1.4 [0–3]	99.2 [98.6–99.7]	23.1 [0.2–46]	84.9 [83–86.8]	0.01 [−0.02–0.04]
≥1 hospitalization OR (≥1 medical consultation AND ≥1 anxiolytic) claims in 6 months	6.1 [2.9–9.3]	96.6 [95.6–97.7]	24.5 [12.9–36.1]	85.2 [83.3–87.1]	0.04 [0–0.08]
≥1 hospitalization OR ≥1 medical consultation OR ≥1 anxiolytic claim (without benzodiazepine) in 6 months *	15.5 [10.6–20.4]	89.1 [87.3–90.9]	20.5 [14.1–26.4]	85.5 [83.5–87.5]	0.05 [−0.01–0.11]
≥1 hospitalization OR ≥1 medical consultation OR ≥1 anxiolytic claim in 12 months *	45.5 [38.9–52.2]	68.3 [65.7–70.9]	20.4 [16.8–24.1]	87.5 [85.4–89.7]	0.09 [0.04–0.14]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 anxiolytic claims in 12 months	38.5 [32–45]	72.6 [70–75.1]	20.1 [16.2–23.9]	86.9 [84.8–89]	0.09 [0.04–0.15]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 anxiolytic claims in 12 months	32.4 [26.1–38.7]	76.6 [74.2–79]	19.8 [15.6–24]	86.4 [84.3–88.4]	0.07 [0.02–0.12]
≥1 hospitalization OR ≥1 medical consultation in 12 months	14.5 [9.8–19.3]	88.7 [86.9–90.5]	18.7 [12.8–24.6]	85.3 [83.3–87.3]	0.04 [−0.02–0.09]
≥1 hospitalization OR ≥2 medical consultations in 6 months **	2.4 [0–4.4]	98.8 [98.2–99.4]	26.3 [6.5–46.1]	85 [83.1–86.9]	0.01 [−0.02–0.05]
≥1 hospitalization (OR ≥1 medical consultation AND ≥1 anxiolytic claims) in 12 months	8 [4.3–11.6]	95.2 [94–96.4]	23 [13.4–32.6]	85.3 [83.4–87.2]	0.04 [0–0.10]
≥1 hospitalization OR ≥1 medical consultation OR ≥1 anxiolytic claim (without benzodiazepine) in 12 months *	17.4 [12.7–23]	87 [85.1–88.9]	19.7 [14.1–25.3]	85.6 [83.6–87.5]	0.05 [0–0.10]
≥1 hospitalization OR ≥1 medical consultation OR ≥1 anxiolytic claim in 6 months before and after interview *	44.6 [37.9–51.3]	69.1 [66.5–71.8]	20.5 [16.8–24.2]	87.5 [85.4–89.6]	0.09 [0.04–0.14]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 anxiolytic claims in 6 months before and after interview	40.8 [34.2–47.5]	72.2 [69.6–74.7]	20.8 [16.8–24.7]	87.2 [85.1–89.3]	0.09 [0.04–0.14]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 anxiolytic claims in 6 months before and after interview	36.2 [29.7–42.6]	76.6 [74.2–79]	21.6 [17.4–25.9]	87 [85–89.1]	0.10 [0.05–0.15]
≥1 hospitalization OR ≥1 medical consultation in 6 months before and after interview	16 [11–20.9]	88.2 [86.3–90]	19.4 [13.6–25.3]	85.5 [83.5–87.4]	0.04 [0–0.10]
≥1 hospitalization OR ≥2 medical consultations in 6 months before and after interview **	7 [3.6–10.5]	96.2 [95.1–97.3]	25 [14–36]	85.3 [83.4–87.2]	0.05 [0–0.10]
≥1 hospitalization OR (≥1 medical consultation AND ≥1 anxiolytic claims) in 6 months before and after interview	6.6 [3.2–9.9]	96.6 [95.5–97.6]	25.5 [13.9–37]	85.3 [83.4–87.2]	0.05 [0–0.10]
≥1 hospitalization OR ≥1 medical consultation OR ≥1 anxiolytic claim (without benzodiazepine) in 6 months before and after interview *	18.3 [13.1–23.5]	85.7 [83.8–87.7]	18.7 [13.4–23.9]	85.5 [83.4–87.2]	0.05 [0–0.10]

PPV: positive predictive value; NPV: negative predictive value; CI: confidence interval; * least restrictive algorithm; ** most restrictive algorithm.

. The least restrictive algorithm had a sensitivity of 39.9%, specificity of 73.0%, PPV of 21.0%, NPV of 87.2%, and slight agreement (κ = 0.09). The most restrictive algorithm exhibited high specificity (99.2%) and low sensitivity (1.4%) with minimal agreement (κ = 0.01).

Table A3. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for anxiety-depressive case definition algorithms in 6 months.

Algorithm	Sensitivity [95% CI]	Specificity [95% CI]	PPV [95% CI]	NPV [95% CI]	Kappa [95% CI]
≥1 hospitalization OR ≥1 medical consultation OR any anxiolytic OR antidepressant claims in 6 months *	59 [53.8–64.1]	64.3 [61.4–67.2]	35.1 [31.2–38.9]	82.8 [80.2–85.3]	0.18 [0.14–0.24]
≥1 hospitalization OR ≥1 medical consultation OR ≥2 anxiolytic OR antidepressant claims in 6 months	53.8 [48.5–59]	67.5 [64.7–70.3]	35.1 [31–39.2]	81.7 [79.2–84.3]	0.18 [0.12–0.23]
≥1 hospitalization OR ≥1 medical consultation OR ≥5 anxiolytics OR antidepressant claims in 6 months	49.7 [44.4–55]	71.1 [68.4–73.8]	36 [31.7–40.3]	81.2 [78.7–83.7]	0.18 [0.13–0.24]
≥1 hospitalization OR ≥1 medical consultation in 6 months	22.8 [18.4–27.3]	87.5 [85.6–89.5]	37.4 [30.9–44]	77.6 [75.3–80]	0.11 [−0.06–0.17]
≥1 hospitalization OR ≥2 medical consultations in 6 months **	6.1 [3.6–8.6]	97.8 [97–98.7]	47.7 [33–62.5]	76.12 [73.9–78.4]	0.05 [0.02–0.09]
≥1 hospitalization OR (≥1 medical consultation AND ≥1 anxiolytics OR antidepressant) claims in 6 months	17.1 [13.1–21.0]	92.7 [91.2–94.3]	43.4 [35.1–51.7]	77.4 [75.1–79.7]	0.12 [0.07–0.18]

* least restrictive algorithm; ** most restrictive algorithm.

in Appendix A presents additional metrics for combined anxiety/depressive disorders, showing similarly low sensitivity (6.1–59%) and high specificity (64.3–97.8%).

3. Discussion

In this study, we evaluated the agreement between algorithms derived from medico-administrative data and structured DSM-IV-based interviews for identifying depression, anxiety, and combined anxiety/depression disorders among older adults (≥65 years) in Quebec. This is the first study to assess the performance of Quebec medico-administrative databases, specifically among older adults, and one of the few to examine anxiety disorders in this context. Agreement between data sources was low, with the tested algorithms demonstrating low sensitivity but moderate to high specificity. The findings suggest that administrative data may be effective in identifying true negative cases but less reliable for comprehensive case ascertainment. This pattern suggests that administrative data may be useful for studying cases of depression and anxiety but are less reliable for comprehensive case findings.

Our results align with previous research examining algorithms based on administrative data, which also reported high specificity but low sensitivity and limited agreement with structured diagnostic interviews [17,18]. Two Canadian studies evaluated administrative data algorithms against the Composite International Diagnostic Interview—Short Form (CIDI-SF) as the gold standard. The first study found high specificity (≥96%), low sensitivity (19–32%) and fair agreement (κ: 0.21–0.25) for their best-case definition tested [18]. The second study, conducted among inflammatory bowel disease patients, obtained similar results, with slightly higher sensitivity (48%), high specificity (89%) and fair agreement (κ: 0.35) [17]. Additionally, a study by Hwang et al. [22] using Medicare databases in the United States also reported low sensitivity (42%) and high specificity (88%) against clinical DSM-IV assessments. Collectively, these findings underscore the limitations of claim-based algorithms in accurately identifying anxiety and depressive disorders across various populations and databases.

Several factors could explain the discrepancy between the prevalence of depression and anxiety identified by interviews versus medico-administrative data. Firstly, it is important to note that the cases of anxiety or depressive disorders documented within administrative databases reflect only those individuals who have actively sought medical attention or are currently receiving treatment for their respective conditions. This limitation of the data underscores a significant issue in mental health care: despite the plethora of medical resources available, a considerable number of individuals suffering from anxiety and depressive disorders often refrain from disclosing their symptoms to their primary care providers [3,23,24]. This is particularly true for older patients, as depression may present itself differently and be mistaken for other physical conditions, such as anemia or hypothyroidism [23]. Physicians may also hesitate to record diagnoses of mental disorders due to stigma or social perceptions [25,26]. Furthermore, patients may seek services or non-pharmacological treatments from other providers, such as psychotherapists, that are not covered by public health or drug plans [27], resulting in an underreporting of cases in the medico-administrative databases [28]. Nevertheless, low utilization of psychotherapy among older adults suggests limited influence of this factor [29]. Secondly, incorporating medication claims in the algorithms may overestimate anxiety and depression cases since medications like antidepressants have multiple indications and are frequently prescribed off-label (e.g., migraine prevention) [30,31]. Conversely, relying solely on claims for anxiolytic medications to identify cases might underestimate anxiety cases, as these drugs are seldom recommended for older adults due to their risk profile [32,33], and are substituted by newer antidepressants such as selective serotonin reuptake inhibitors (SSRIs) or serotonin and noradrenaline reuptake inhibitors (SNRIs) [34,35]. Finally, the nature of anxiety/depressive disorders, which occur in episodes and relapse phases, may contribute to the discrepancies between administrative and interview cases. Some individuals may not have shown symptoms during the interview, even if they had experienced depression or anxiety in the past and, therefore, had a diagnosis or treatment recorded in the databases. Individuals who used medication to manage their anxiety or depression may no longer report symptoms during the interview. Furthermore, the potential influence of recall or desirability bias may have contributed to participants’ failure to disclose symptoms of depression or anxiety during the interview process. Conversely, others may have displayed signs of depression or anxiety during the structured clinical interviews but had not yet been diagnosed or treated by a physician for these symptoms.

As mentioned, algorithms relying solely on diagnostic codes from physician visits and hospitalizations for depression or anxiety have shown low sensitivity, potentially resulting from underdiagnosis or underreporting [25,26]. Conversely, including medication claims may increase false positives due to the broad indications and frequent off-label use of anxiolytics and antidepressants [34,35]. Algorithms that combine only hospitalization, ambulatory physician visits, and medication claim codes related to depression and anxiety seem to fail to identify cases accurately. Although the algorithms show low sensitivity, which limits comprehensive case findings, they still provide value for surveillance activities where the main goal may be to monitor trends or service utilization among diagnosed individuals, rather than estimating the true prevalence. In such situations, high-specificity algorithms can offer valid information about treated cases and their trajectories within the healthcare system and may be helpful for evaluating changes over time or the impact of public health interventions. To address these limitations, future algorithm development might consider comorbid physical and psychiatric conditions collectively, using comprehensive data from diagnostic codes and medication claims [36,37,38]. Recent methodological approaches, including machine learning, could enhance algorithm performance and accuracy in identifying depression and anxiety cases, at least among individuals who access medical care [39]. Moreover, although our study focused on cross-sectional validation at the time of the interview, future research could build on this work by adopting a longitudinal perspective to assess the predictive performance, incidence, and persistence of algorithm-identified cases over time.

Strengths and Limitations

The study has notable strengths, including the use of a validation cohort derived from the ESA-S study, which recruited participants from one of Quebec’s largest administrative health regions, providing a representative sample of older adults across both urban and rural areas. The comprehensive in-home structured interviews closely resembled clinical diagnostic evaluations. Additionally, most Quebec residents, especially older adults, have public health and drug insurance coverage, ensuring accurate representation in administrative data for medical consultations, hospitalizations, and medication claims [40]. However, results should be interpreted with caution due to some limitations. First, combining ICD-9 and ICD-10-CA codes in administrative databases could have introduced misclassification, as these classification systems do not fully align. Second, physicians submitting outpatient claims are not mandated to provide diagnoses; when provided, only one diagnosis per consultation is recorded, potentially resulting in further misclassification. Third, the ESA-S structured interviews, our gold standard, could be affected by social desirability bias, leading to underreporting of anxiety and depression. Specifically, for PTSD, the ESA-S relied on the IES-R, a screening tool that captures symptoms but does not encompass all DSM-IV criteria [41,42,43]. Lastly, participants receiving treatment before the interview might have reduced symptoms, contributing to discrepancies and lower agreement between the interview and administrative data-based algorithms.

4. Materials and Methods

4.1. Study Population and Validation Cohort

The study is a secondary analysis of data from the Étude sur la Santé des Aînés Services (ESA-S) Study that recruited adults aged 65 and older awaiting medical services at primary health clinics in Quebec between 2011 and 2013. Detailed methodology of the ESA-S study has been published previously [44]. Briefly, among 744 eligible general practitioners (GPs) initially identified, 409 participated, and 245 recruited an average of 7.3 patients each. Sampling was stratified across four primary care settings: family medicine groups, local community health centers, primary care clinics (PCs) with fewer than three GPs, and PCs with at least three GPs, ensuring representativeness. Participants provided informed consent and underwent in-home interviews conducted by 19 trained health professionals lasting approximately 90 min. Interviews were conducted privately to ensure confidentiality and minimize response bias. Initially, 1811 participants provided informed consent to participate in the ESA-S in-home interview. Before the interview, cognitive status was assessed using the Mini-Mental State Examination (MMSE), and participants with an MMSE score < 22 (n = 46) were excluded due to moderate or severe cognitive impairment. This left 1765 participants who completed the structured in-home interview.

Of these, 1657 agreed to grant access to their medico-administrative data and were successfully identified in the provincial databases. For the present analysis, we included only participants who had continuous public drug insurance coverage during the year before and the year after the ESA-S interview and who had complete data from both the structured interview and the health administrative databases.

4.2. Administrative Data

Administrative data from consenting participants were linked to medico-administrative databases from the Régie de l’Assurance Maladie du Québec (RAMQ), which manages the Quebec Health Insurance Plan and the Public Drug Insurance Plan, covering more than 90% of older adults in Quebec [45]. The RAMQ databases contain demographic and geographic data on people with valid health insurance numbers, including their eligibility for public health and drug insurance, medical and pharmaceutical services received, and dates of services. Hospital discharge data were extracted from MED-ÉCHO (Maintenance et Exploitation des Données pour l’Étude de la Clientèle Hospitalière), which includes detailed patient hospitalization records [45]. Physician visit diagnoses are coded according to the ninth version of the International Classification of Diseases (ICD-9), and hospital discharge diagnoses are coded using the tenth version (ICD-10) [45]. Medication claims include data classified according to the American Hospital Formulary Service (AHFS) system [46], capturing non-proprietary names, quantities, and supply duration.

4.3. ESA-S Case Definitions

Participants were classified as having depression or anxiety based on symptoms reported during structured in-person ESA-S interviews, which included questions based on DSM-IV criteria. The evaluation included the following mental health conditions: generalized anxiety disorder (GAD), specific phobia, social phobia, panic disorder, agoraphobia, obsessive-compulsive disorder (OCD), post-traumatic stress disorder (PTSD)—assessed using the Impact of Events Scale-Revised (IES-R)—major depressive disorder, and minor depression. Participants were classified as having depression if they reported experiencing either depressed mood or loss of interest or pleasure in daily activities for at least two consecutive weeks. participants reporting at least five of the nine DSM-IV depressive symptoms (with one being either depressed mood or loss of interest/pleasure) were classified as having major depression, whereas those reporting two to four symptoms were classified as having minor depression. Participants were classified as having an anxiety disorder if they met the DSM-IV criteria for generalized anxiety disorder, phobias (including social anxiety disorder/social phobia, agoraphobia, or specific phobias), panic disorder, OCD [47] or PTSD. For PTSD, an IES-R score of 30 and above was considered indicative of the disorder.

4.4. Database Case Identification Algorithms

Various case definitions were developed and tested to identify individuals with depression and/or anxiety in the RAMQ databases during the six months and one year prior to the ESA-S interview, as well as in the period from six months before to six months after the interview. These definitions (algorithms) were based on different combinations of the following criteria: (1) physician visit claims with an ICD-9 code for depression and/or anxiety, (2) hospitalization with an ICD-10 code for depression and/or anxiety, and (3) prescription claims for antidepressant and/or anxiolytic medications.

We used the Canadian versions of the ICD-9 and ICD-10 coding systems [48,49]. In Quebec, Canada, ICD-9 diagnostic codes are applied in outpatient settings, while ICD-10 diagnostic codes are employed in inpatient settings. For depression, the ICD-9 codes included 311.x, 300.4, 300.5, 308.0, 309.0, 309.1, 301.1, 296.0, 296.2, 296.3, 296.4, 296.5, 296.6, 296.8, and 298.0. The ICD-10 codes for depression included F32.x, F33.x, F34.1, F34.8, and F34.9. For anxiety, the ICD-9 codes included 300.0, 300.21, 300.23, 300.29, 300.22, 300.3, 309.81, and 308.3. The ICD-10 codes for anxiety included F40, F41, F42, and F43. The study examined medications that were available during the study period, indicated for the treatment of depression or anxiety, and covered by the Quebec public drug insurance plan. All medications categorized under the AHFS code 28:16.04 were considered for the depression case definition, while those under AHFS codes 28:24:04, 28:24:08 and 28:24:92 were considered for the anxiety case definition. Full ICD codes and medication descriptions are presented in Appendix A,

Table A1. International Classification of Diseases (ICD) diagnostic codes considered for anxiety and depressive disorders.

Condition	ICD-9 Code	ICD-10 Code	Description
Depression	311	F32.9	Depressive disorder, not elsewhere classified
	300.4	F34.1	Dysthymic disorder
	300.5		Neurasthenia
	308.0		Acute reaction to stress
	309.0		Adjustment disorder with depressed mood
	309.1		Prolonged depressive reaction
	301.1		Chronic depressive personality disorder
	296.0	F31.0	Bipolar I disorder, single manic episode
	296.2	F32.1	Major depressive disorder, single episode
	296.4	F31.2	Bipolar I disorder, most recent episode manic
	296.5	F31.3	Bipolar I disorder, most recent episode depressed
	296.6	F31.6	Bipolar I disorder, most recent episode mixed
	296.8	F31.8	Bipolar I disorder, most recent episode unspecified
	298.0	F32.3	Depressive type psychosis
Anxiety	300.00	F41.1	Generalized anxiety disorder
	300.3	F42	Obsessive-compulsive disorder
	309.81	F43.1	Posttraumatic stress disorder
	300.01	F41.0	Panic disorder without agoraphobia
	300.21	F40.01	Panic disorder with agoraphobia
	300.22	F40.00	Agoraphobia without history of panic disorder
	300.29	F40.2	Specific phobias
	300.23	F40.1	Social phobia
	308.3	F43.0	Acute stress reaction

and

Table A2. AHFS medication codes considered for anxiety and depressive disorders.

Condition	AHFS Code	Medication Category	Example of Medications
Depression	28:16.04	Antidepressants	“citalopram (bromhydrate de)”, “paroxétine (chlorhydrate de)”, “sertraline (chlorhydrate de)”, “bupropion (chlorhydrate de)”, “duloxétine”, “fluoxétine (chlorhydrate de)”, “fluvoxamine (maléate de)”, “mirtazapine”, “trazodone (chlorhydrate de)”, “venlafaxine (chlorhydrate de)”, Escitalopram (Oxalate d’escitalopram)
Anxiety	28:24:04	Benzodiazepines	“alprazolam”, “bromazépam”, “chlordiazépoxide (chlorhydrate de)”, “clobazam”, “diazépam”, “lorazépam”, “oxazépam”
	28:24:08	Other Anxiolytics, Sedatives, and Hypnotics	“hydroxyzine (chlorhydrate d’)”, “l-tryptophane”
	28:24:92	Miscellaneous Anxiolytics, Sedatives, and Hypnotics

.

4.5. Statistical Analysis

Descriptive analyses were performed, and Chi-square tests were used to identify differences in the prevalence of anxiety and depressive disorders according to sex.

We calculated Cohen’s kappa (κ) coefficient for each algorithm to assess the chance-adjusted agreement and interpreted the results as slight (κ: 0–0.20), fair (κ: 0.21–0.40), moderate (κ: 0.41–0.60), substantial (κ: 0.61–0.80), and very good agreement (κ: 0.81–1.0) [50]. Using conventional two-by-two contingency tables, we also evaluated the algorithms’ performance by calculating sensitivity (Sn), specificity (Sp), positive predictive value (PPV), and negative predictive value (NPV), along with their 95% confidence intervals (CIs), using the interview case definitions as the gold standard.

All the metrics were estimated using the Statistical Analysis System (SAS, 9.4) [51].

Algorithm performance was assessed by constructing conventional two-by-two contingency tables, with interview-based case definitions as the reference standard. We calculated sensitivity (Sn), specificity (Sp), positive predictive value (PPV), and negative predictive value (NPV), each with corresponding 95% confidence intervals (CIs). The calculations were based on the following formulas:

• Sensitivity (Sn) = TP/(TP + FN)
• Specificity (Sp) = TN/(TN + FP)
• Positive Predictive Value (PPV) = TP/(TP + FP)
• Negative Predictive Value (NPV) = TN/(TN + FN)

where TP = true positives, FN = false negatives, TN = true negatives, and FP = false positives, as defined by comparison with the interview-based diagnoses.

All statistical analyses were performed using SAS software, version 9.4 (SAS Institute Inc., Cary, NC, USA) [51].

5. Conclusions

Agreement between medico-administrative database algorithms and the DSM-IV criteria-aligned interviews for identifying depression, anxiety, and anxiety/depressive disorders in older individuals was low. These sources may be identifying different populations and individuals at varying clinical and health service use stages and trajectories. Medico-administrative databases still provide useful information, as algorithms with high specificity can be used to inform healthcare utilization and trajectories related to anxiety/depressive disorders, which are crucial in planning and implementing public health initiatives and resource allocation. Further research and refinement of algorithms are necessary. Integrating other data sources and advanced machine-learning techniques may further enhance the accuracy and reliability of such databases for mental health surveillance and research.