Next Article in Journal
Description of Position Ads for Pharmacy Technicians
Previous Article in Journal
Medication Complexity among Disadvantaged African American Seniors in Los Angeles
Open AccessArticle

Association of a Novel Medication Risk Score with Adverse Drug Events and Other Pertinent Outcomes Among Participants of the Programs of All-Inclusive Care for the Elderly

1
Applied Precision Pharmacotherapy Institute, Tabula Rasa HealthCare, Moorestown, NJ 08057, USA
2
Healthcare Analytics, Tabula Rasa HealthCare, Moorestown, NJ 08057, USA
3
Precision Pharmacotherapy Research and Development Institute, Tabula Rasa HealthCare, Lake Nona, Orlando, FL 32827, USA
4
Chief Executive Officer, Tabula Rasa HealthCare, Moorestown, NJ 08057, USA
*
Author to whom correspondence should be addressed.
Pharmacy 2020, 8(2), 87; https://doi.org/10.3390/pharmacy8020087
Received: 22 April 2020 / Revised: 8 May 2020 / Accepted: 15 May 2020 / Published: 20 May 2020
(This article belongs to the Section Clinical Pharmacy)

Abstract

Preventable adverse drug events (ADEs) represent a significant public health challenge for the older adult population, since they are associated with higher medical expenditures and more hospitalizations and emergency department (ED) visits. This study examines whether a novel medication risk prediction tool, the MedWise Risk Score™ (MRS), is associated with ADEs and other pertinent outcomes in participants of the Programs of All-Inclusive Care for the Elderly (PACE). Unlike other risk predictors, this tool produces actionable information that pharmacists can easily use to reduce ADE risk. This was a retrospective cross-sectional study that analyzed administrative medical claims data of 1965 PACE participants in 2018. To detect ADEs, we identified all claims that had ADE-related International Classification of Diseases and Health Related Problems, 10th revision (ICD-10) codes. Using logistic and linear regression models, we examined the association between the MRS and a variety of outcomes, including the number of PACE participants with an ADE, total medical expenditures, ED visits, hospitalizations, and hospital length of stay. We found significant associations for every outcome. Specifically, every point increase in the MRS corresponded to an 8.6% increase in the odds of having one or more ADEs per year (OR = 1.086, 95% CI: 1.060, 1.113), $1037 USD in additional annual medical spending (adjusted R2 of 0.739; p < 0.001), 3.2 additional ED visits per 100 participants per year (adjusted R2 of 0.568; p < 0.001), and 2.1 additional hospitalizations per 100 participants per year (adjusted R2 of 0.804; p < 0.001). Therefore, the MRS can risk stratify PACE participants and predict a host of important and relevant outcomes pertaining to medication-related morbidity.
Keywords: Programs of All-Inclusive Care for the Elderly; PACE; medical expenditures; medication risk stratification; medicare; adverse drug events; risk score Programs of All-Inclusive Care for the Elderly; PACE; medical expenditures; medication risk stratification; medicare; adverse drug events; risk score

1. Introduction

Among older Americans, adverse drug events (ADEs) represent a significant public health challenge. Estimates indicate that ADEs may occur in up to 35% of older outpatients [1] and cause or contribute to 6%–12% of this cohort’s hospital admissions [2]. Predictably, ADEs carry economic consequences. Medication-related morbidity and mortality may be responsible for 16% of the United States’ annual healthcare costs, representing a $500 billion annual toll [3]. Fortunately, the US healthcare system should be able to mitigate the economic and clinical consequences of ADEs. Consistent data suggest that the majority of ADEs are both predictable and preventable [1,4].
The predictable and preventable nature of ADEs is a call to action. A risk stratification tool that identifies individuals at risk could help policy makers, healthcare organizations, and payers curb the negative clinical and economic consequences of ADEs. While ADE risk prediction tools do exist, a disproportionate amount have been validated in hospital settings and, as recently reported, several of them cannot accurately predict the risk of drug-related problems [2,5,6]. Moreover, many of the existing tools utilize chronic comorbidities or laboratory values, such as renal failure, heart failure, or white blood cell count, to predict ADEs [5,7,8]. While these variables may accurately predict ADE risk, they are not necessarily modifiable or easily actionable. Thus, they are of dubious clinical utility. By contrast, the MedWise Risk Score™ (MRS) [9], described by Cicali et al., uses potentially alterable pharmacological characteristics of patients’ medications for risk stratification, making it potentially easier for healthcare providers to mitigate medication risk. However, until now, associations with ADEs and other important outcomes (e.g., medical expenditures, emergency department visits, hospitalizations) have not been reported in controlled studies.
The US-government funded Program of All-Inclusive Care for the Elderly (PACE) is an ideal setting to examine the MRS. PACE provides supportive services to community-dwelling older adults who are certified by the state in which they reside to require a “nursing home level of care” [10]. Given a capitated payment model, PACE organizations aim to avoid unnecessary medical expenditures, nursing home institutionalization, and frequent hospital visits [10]. PACE organizations are at great risk of unnecessary spending associated with medication-related morbidity given their participants’ significant comorbidities [11] and prevalence of medication-related problems [12]. A tool like the MRS could identify these organizations’ high-risk members and provide guidance to mitigate risk. Therefore, our objective here is to examine whether the MRS correlates with ADEs as well as the outcomes that are important to PACE and other government-funded healthcare programs: total cost, emergency department (ED) visits, hospitalizations, and hospital length of stay.

2. Materials and Methods

2.1. Study Design, Setting, and Context

This was a retrospective cross-sectional study that analyzed administrative medical claims data of PACE participants who received care from a national provider of PACE pharmacy services, CareKinesis. This study was granted a waiver of informed consent by the Biomedical Research Alliance of New York Institutional Review Board (protocol number 19-12-172-427, approved 24 May 2019). Researchers conducted the study in accordance with ethical principles set forth by the Declaration of Helsinki.
The PACE program has been described extensively in previous publications [10,13]. To highlight, each PACE organization receives capitated funding from the Centers for Medicare and Medicaid Services (CMS) and their state’s Medicaid program to provide comprehensive, supportive services to a unique cohort of medically-complex, community-dwelling older adults. The main objective of PACE is to prolong independent living within the community. Given a capitated payment model, PACE organizations have discretion to include any needed service to help promote favorable outcomes. PACE patients, known as “participants,” have their care coordinated by an interdisciplinary team (IDT) of professionals (e.g., primary care physicians, social workers, nursing, physical therapy, etc.). Participants travel to day-care centers to receive medical and supportive services as determined by the IDT. At the time of writing, there are over 250 PACE centers in 31 states in the continental US [11]. Each center often partners with one pharmacy for prescriptions and consultative services [13,14].
As of this writing, CareKinesis PACE Pharmacy provides comprehensive medication services to more than 14,000 PACE participants across the continental US. This census represents nearly 100 individual PACE centers. In addition to providing prescriptions and over-the-counter medications, the pharmacy offers a suite of clinical services delivered by board-certified geriatric pharmacists (i.e., the BCGP credential) who use a novel clinical decision support system (CDSS; EireneRx®). The CDSS helps pharmacists identify and resolve specific medication-related problems [12] as well as optimize medication regimens based on available pharmacogenomic information [15]. Prescribers and pharmacists commonly collaborate to improve medication safety and reduce ADE risk [12,16].

2.2. Study Sample

This study attempted to assess associations between the MRS and outcomes that occur on an annual basis. As a result, the investigators limited the analytical study sample to PACE participants with complete (12 months) 2018 medical claims (physician and facility charges) data and 12 months of MRS (i.e., risk stratification) data during the same period. Therefore, we excluded PACE participants for whom we did not have access to medical claims data (n = 9689), did not have risk stratification data (n = 85), and did not have 12 months of 2018 claims and/or risk stratification data (n = 249).

2.3. Definitions, Terms, and Measures

2.3.1. The Novel MRS

Within the CDSS, MRS is calculated in real time for every participant; therefore, the MRS is updated each time the medication regimen changes. The MRS is generated by processing patient medication data (e.g., National Drug Codes, RxNorm’s Rx Concept Unique Identifiers) within the e-prescribing and pharmacy management platform (EireneRx®). For both prescribers and pharmacists, the MRS is displayed prominently at the top of every participant’s medication profile. Moreover, “sandbox” features help clinicians understand how hypothetical drug regimen alterations (e.g., prescription or over-the-counter medication additions or deletions) can impact the participant’s MRS. This helps the prescriber or pharmacist make informed decisions before making or suggesting changes to the medication regimen. PACE prescribers receive accredited education regarding the MRS.
The MRS is the object of a patent currently under review (WO 2019/089725; pending) and has been previously described in the literature [9]. Briefly, it is derived from aggregated, weighted values for five pharmacokinetic and pharmacodynamic factors shown to be associated with medication-related morbidity and mortality. The total MRS ranges from 0 to 53. Overall, the factors comprising this model have been tested and there is minimal observed multicollinearity. Some collinearity does exist between factors 2 and 3, but the magnitude of collinearity is dependent upon the cohort’s sample size. All five risk factors are further recapped below along with a brief justification for inclusion in the MRS.
By assigning relative odds ratios (RORs) [17] for all single-drug adverse event pairs, factor 1 accounts for the risk of ADEs as reported to the US Food and Drug Administration’s (FDA) Adverse Event Reporting System (FAERS) database. In our analysis, odds ratios (ORs) are calculated for 104 adverse events for each active ingredient. Next, a ROR is calculated by accounting for the entire drug regimen. For example, if a participant has four drugs associated with the same ADE in his/her drug regimen, the calculated ROR accounts for the cumulative frequency of each OR, weighted by the total number of observations in the FAERS for these active ingredients. Factor 2 utilizes the Anticholinergic Cognitive Burden (ACB) scale to calculate the drug regimen’s total anticholinergic burden [18,19]. Cumulative use of anticholinergic medications may contribute to higher total healthcare utilization [20], chronic functional decline, cognitive deficits, weakness, falls, hospitalizations, and all-cause mortality [5,21,22,23,24,25,26]. Factor 3 utilizes the Sedative Load (SL) model [27] of each medication to quantify the sedative burden of the drug regimen. Exposure to highly sedating medications has been associated with poor patient outcomes and adverse drug events, such as cognitive decline, physical impairment, falls, and increased mortality [28,29,30,31]. Factor 4 quantifies the medication regimen’s risk of QT-prolongation because studies have shown a positive correlation between increased QTc length and mortality [32,33]. The QT interval is the time from the start of the Q wave to the end of the T wave on the surface electrocardiogram. The QTc is this interval corrected for heart rate. Lastly, factor 5 utilizes drug metabolizing information and, when available, pharmacogenomic data to quantify the number, severity, and complexity of pharmacokinetic drug interactions. Drug interactions are associated with ADE-related hospitalizations [34,35] and, when available, pharmacogenomic data can help clinicians identify more clinically significant interactions [36].
Since the MRS is calculated in real time for every PACE participant’s drug regimen, the MRS can account for acute-on-chronic medication risk. For example, a pharmacokinetic drug interaction is expected in a patient taking chronic warfarin who is prescribed acute antibiotic therapy with sulfamethoxazole-trimethoprim, resulting in a higher risk of hemorrhaging [37]. Therefore, this will yield a higher MRS compared to baseline from warfarin use alone. To capture acute-on-chronic medication risk and/or worsening risk due to changes in chronic drug therapy, the participant’s highest reported MRS during 2018 was identified to perform all analyses in this study. This value captures the temporal nature of medication risk.

2.3.2. ADEs

To detect ADEs, we utilized a list of ADE-related International Classification of Diseases and Health Related Problems, 10th revision (ICD-10) codes classified by Hohl et al. [38]. In this study, the authors identified 827 ICD-10 codes that have been used in the literature to identify ADEs in claims data. They classified these codes into nine categories, based on the likelihood that each code indicated the presence of an ADE: A1: “induced by medication/drug”; A2: “induced by medication or other causes”; B1: “poisoning by medication”; B2: “poisoning by or harmful use of medication or other causes”; C: “ADE very likely; D: “ADE likely”; E: “ADE possible”; U: “ADE unlikely”; V: “ADE related to vaccine”. We limited our analysis to 325 codes that were classified as A1, A2, B1, and B2 because these codes unambiguously indicate the presence of an ADE. Therefore, our analysis reflects a conservative view of ADEs. To avoid overcounting redundant ADE claims (e.g., multiple similar ADE-related ICD-10 codes recorded over the course of consecutive days or in close proximity), we did not count the total number of ADEs per member. Rather, we considered ADEs to be a binary variable that was either present or absent for a given participant. Specifically, if a participant had evidence of ≥1 ADE-related ICD-10 code in 2018 claims, the participant would be considered “ADE positive”. Conversely, if there was no ADE-related ICD-10 code in the claims, the participant was considered “ADE negative”.

2.3.3. Pertinent Risk Outcomes

In addition to ADEs, we assessed the association between the MRS and several other outcomes that have been associated with ADEs: total annual combined facility and physician expenditures, all-cause emergency department (ED) visits, all-cause hospital admissions, and hospital length of stay. Facility expenditures were defined as the total adjudicated amount (USD) for a participant’s care received from hospitals and/or skilled nursing facilities. Physician expenditures were defined as the total adjudicated amount (USD) from physician charges for participant care. ED visits and hospitalizations were identified by their line-item claims. Hospital length of stay was the number of consecutive days between the admission date and the discharge date, where these two dates were identified based on claims.

2.4. Statistical Procedures

We utilized a binary logistic regression model to determine the probability of having at least 1 ADE (i.e., participant had either ≥1 ADE or no ADE) at any given MRS. For a more intuitive analysis and interpretation, we also utilized a linear regression to evaluate the association between the MRS and the number of members with at least one ADE in 2018. The regression model was weighted by the number of members at each risk score point to diminish any impact of outliers. To determine the MRS’ ability to predict ADE presence, a receiver operator characteristic (ROC) curve was constructed, and the area under the curve (AUC) was calculated. For the other outcomes, relationships were examined using linear regression models, again weighted by the number of members at each risk score point. An alpha level of 0.05 was defined a priori to determine statistical significance. All analyses were processed using R version 3.5.3.

3. Results

During 2018, 11,988 patients were identified in CareKinesis’ total census. After making various exclusions, which are outlined in Figure 1, 1965 PACE participants remained in the analytical sample. PACE participants included in this analysis were 65.2% (n = 1282) female and, on average 76.8 ± 9.9 years old. Participants represented a total of 12 PACE organizations, which are geographically dispersed throughout the United States and vary in overall census size. Table 1 highlights participant demographics. Figure 2 depicts the overall distribution of the MRS among PACE participants. Overall, MRS ranged from 2 to 40 in the total sample and, on average, participants had an MRS of 18.5 ± 7.8.
During 2018, 128 (6.5%) PACE participants experienced at least one ADE. A total of 54 unique ADE-related ICD-10 codes were identified in the 2018 claims. The ICD-10 codes that described ADEs were often related to opioids, extrapyramidal symptoms, anticoagulants, skin eruptions, and psychoactive medications. These symptoms and medication classes accounted for at least 50% (n = 85) of all (n = 170) ADE claims. During our analysis, we found patients with ADE codes for T78.4 (allergy, unspecified), T80.2 (infections following infusion, transfusion, or therapeutic injection), and T40.1 (heroin poisoning). These codes were excluded from our analysis because such ADEs are unrelated to the MRS. The top ADEs and their corresponding ICD-10 codes can be viewed in Table 2. All remaining ADE codes are provided in Appendix A, Table A1.
The logistic regression analysis found that the odds of having one or more ADEs over the course of the year increases by 8.6% per every point increase in the MRS (OR = 1.086, 95% CI: 1.060, 1.113; p < 0.001). A weighted linear regression was also calculated, using MRS to predict the total number of participants with at least one ADE. A significant regression equation was found (F (1, 36) = 31.8; p < 0.001), with an adjusted R2 of 0.454. The number of PACE participants predicted to have at least one annual ADE is equal to –0.0261 + 0.0049x, where x is the MRS. Therefore, every point increase in the MRS corresponds to an additional 4.9 participants per 1000 PACE participants with at least one ADE per year. The ADE vs. MRS weighted regression can be viewed in Figure 3. The area under the ROC curve, which quantifies the ability of the MRS to predict the presence of ADEs, was 0.67, and is depicted in Appendix A, Figure A1.
We also assessed the MRS’ relationship to a variety of other pertinent risk outcomes, which can be viewed in Figure 4, Figure 5, Figure 6 and Figure 7. Notably, a significant association was observed when MRS was regressed against total annual medical expenditures. Figure 4 depicts this weighted linear regression. A significant regression equation was found (F (1, 36) = 105.8; p < 0.001), with an adjusted R2 of 0.739. PACE participants predicted total annual facility and physician costs as equal to 7091.50 + 1036.90x USD, where x is the MRS. Therefore, PACE participants’ annual costs increase $1036.90 USD for every point of MRS. Figure 5, Figure 6 and Figure 7 show all other weighted regressions calculated to predict various risk outcomes based on MRS. In summary, we observed a significant positive correlation between the MRS and various outcomes, including annual ED visits, all-cause hospital admissions, and hospital length of stay.

4. Discussion

This retrospective study of PACE administrative medical claims demonstrates that a novel MRS derived strictly from a medication regimen’s pharmacological properties was significantly associated with ADE occurrence among medically-complex, community-dwelling older adults enrolled in a PACE program. Specifically, we found that every point increase in the MRS corresponded to nearly five additional participants having at least one ADE in the year out of every 1000 participants. However, this is likely a gross underestimation. The literature indicates that ADEs are widely underreported, with several studies citing underreporting rates that exceed 90% [39]. To account for potential underreporting and given established associations with higher healthcare utilization and cost, we anticipated that a higher MRS would be associated with higher costs, hospital admissions, ED visits, and hospital length of stay. Our results supported this idea, as each point increase in the MRS corresponded to over $1000 USD in additional annual medical spending, three additional annual ED visits per 100 participants per year, and two additional hospitalizations per 100 participants per year. Collectively, these results indicate that it is possible to utilize the pharmacological properties of a drug regimen to risk stratify PACE participants and predict a host of important and relevant outcomes pertaining to medication-related morbidity.
The associations identified in this study have important implications for PACE organizations, which are at full financial risk for participant outcomes due to capitated payments from CMS and state Medicaid programs [10]. Projections through 2025 estimate that, in general, costs of medical resources (e.g., hospitalizations and emergency room visits) will increase, while many states could face future annual Medicaid funding restrictions [3,13,40]. This combination will place immense pressure on PACE to engage in cost avoidance and reduction activities [13]. Such activities must not ignore medication-related morbidity, which could cost the healthcare system an additional $2500 USD for each instance of non-optimized drug therapy [3]. Given that resolution of medication-related problems may help avoid unnecessary medical expenditures [3,41], deploying consistent medication risk identification and mitigation strategies in PACE is vital.
While the utilization of such strategies remains inconsistent in this setting [10,13,14], some evidence indicates that PACE healthcare providers (HCPs) desire comprehensive and consistent support to help promote safer pharmacotherapy. Sloane et al. found that PACE physicians frequently reported difficulties in understanding how patients’ complex morbidity can alter risk of adverse health outcomes or wasted spending. This lack of understanding can lead to treatment that is inappropriate, ineffective, or unnecessary [42]. Physicians outside of PACE have echoed these sentiments regarding the pharmaceutical care of the medically-complex older adult. Specifically, polypharmacy resulting from multi-morbidity makes medication management in this cohort especially challenging [43,44,45]. Underscoring these challenges, a recent study found that PACE physicians and nurse practitioners frequently initiated consultation with pharmacists for information and advice when faced with medication safety uncertainties [16]. Notably, their inquiries commonly pertained to opioids and psychoactive drugs [16], which are medication classes responsible for the disproportionate amount of ADE claims identified in the present study.
The MRS can help solve these problems. First, PACE HCPs can now use the MRS to identify participants who are at the greatest risk of medication morbidity and healthcare utilization. Therefore, PACE organizations and pharmacists can properly allocate efforts and resources to the participants who have the greatest need for intervention. For instance, pharmacists and HCPs can focus initial medication reviews on 20% of the population at greatest risk. In this study, this would correspond to participants with an MRS ≥26. Second, point-of-care access to the MRS provides HCPs with actionable information to make geriatric medication management less challenging. The MRS is comprised of five risk factors that highlight aspects of the regimen that need the most intervention or attention. For example, if the MRS identifies that a PACE participant is at high risk due to competitive inhibition, HCPs can act to resolve any present interaction(s).
By identifying tangible ways to mitigate medication-related morbidity among the most “at risk” PACE participants, the MRS can profoundly influence HCP behavior. For example, a survey of PACE physicians found that the overwhelming majority (>85%) of respondents were more likely to deprescribe medications or reconsider medication choice as a result of having access to the MRS [46]. When PACE pharmacists were supported with the MRS in a CDSS, HCPs accepted about 80% of pharmacists’ recommendations to resolve medication-related problems that frequently involved broad medication safety concerns (69%), such as adverse drug reactions (18%) and pharmacokinetic drug interactions (21%) [12]. The interactions frequently involved opioids, anticoagulants, and psychoactive medications [12], which were top contributors to ADEs in the present study. Collectively, this suggests that the MRS helps PACE HCPs and pharmacists identify and mitigate relevant medication risk.
Our analysis is not without limitations, which carry pertinent future research directives. First, retrospectively detecting ADEs with a conservative set of ICD-10 codes may have biased our results and underestimated the true ADE occurrence. As aforementioned, administrative data have a low sensitivity to detect ADEs since it is well-established that ADEs are underreported by HCPs at the point of care [38,39,47]. Since ADEs are a difficult factor to capture through administrative claims alone, this could explain why our area under the ROC curve was fair at 0.67. To our knowledge, other risk scores only examined ADE-induced hospitalizations or inpatient-acquired ADEs, making comparisons difficult. Nevertheless, it is noteworthy that our metric was similar to the area under the ROC curve reported by Parameswaran Nair et al. (0.70), who designed a risk score to detect ADE-related hospitalizations among older outpatients whereby ADEs were clinically validated by expert consensus chart review and patient interviews [5]. Future studies should assess the MRS in a prospective manner, with clinical ADE validation. Nevertheless, associations with ADEs were still statistically significant despite using ICD-10 codes. Therefore, the associations observed between the MRS and the other negative risk outcomes can be explained, at least in part, by medication-related morbidity. Future research will need to validate the MRS against other ADE-specific risk scores [5,6] and various risk indices highly relevant to PACE (e.g., Hierarchical Condition Category scores). Finally, additional studies will be needed in alternative cohorts. Given the uniqueness of the PACE cohort, our results may not be generalizable to inpatient older adults and healthier community-dwelling older adults.

5. Conclusions

We found that a novel MRS that derives a level of risk strictly through aggregation of pharmacological properties of a medication list positively correlates with ADEs, total utilization, hospitalizations, ED visits, and hospital length of stay among PACE participants. Adopting the MRS in PACE offers PACE organizations the opportunity to identify and prospectively manage participants at risk of ADEs and other relevant outcomes in a standardized manner.

Author Contributions

Conceptualization, A.S., C.H.K., V.M., and J.T.; methodology, A.S., S.F., D.L.B., and H.J.; investigation, A.S., S.F., and H.J.; formal analysis, H.J. and S.F.; writing—original draft, D.L.B.; writing—review and editing, all authors; visualization, D.L.B. and H.J.; supervision, A.S. and J.T. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Acknowledgments

We thank Dana Filippoli, MBE, MPH and Orsula Knowlton, PharmD, MBA for thoroughly reviewing the manuscript and for providing authors with background and context; TRHC for financial support of this study; and each PACE organization for their commitment to serving the welfare of vulnerable older adults.

Conflicts of Interest

All authors performed this work as employees (with stock options and shares) of Tabula Rasa HealthCare (d/b/a CareKinesis). The authors have no personal conflicts to disclose.

Appendix A

Table A1. All other adverse drug events (ADEs) identified by ICD-10 codes.
Table A1. All other adverse drug events (ADEs) identified by ICD-10 codes.
ICD CodeDescriptionn (%)
T38.0Glucocorticoids and synthetic analogues4 (2.4)
F15.1Mental and behavioral disorders due to use of other stimulants, including caffeine: harmful use4 (2.4)
T38.3Insulin and oral hypoglycemic (antidiabetic) drugs4 (2.4)
F15.9Mental and behavioral disorders due to use of other stimulants, including caffeine: unspecified mental and behavioral disorder4 (2.4)
I95.2Hypotension due to drugs3 (1.8)
E16.0Drug-induced hypoglycemia without coma3 (1.8)
F15.2Mental and behavioral disorders due to use of other stimulants, including caffeine: dependence syndrome3 (1.8)
T42.6Other antiepileptic and sedative-hypnotic drugs3 (1.8)
T40.4Poisoning by other synthetic narcotics (buprenorphine, pethidine, dextropropoxyphene, etc.)3 (1.8)
T42.8Antiparkinsonism drugs and other central muscle-tone depressants2 (1.2)
G25.1Drug-induced tremor2 (1.2)
F11.1Mental and behavioral disorders due to use of opioids: harmful use2 (1.2)
F19.9Mental and behavioral disorders due to multiple drug use and use of other psychoactive substances: unspecified mental and behavioral disorder2 (1.2)
M87.1Osteonecrosis due to drugs2 (1.2)
R50.2Drug-induced fever2 (1.2)
T45.1Antineoplastic and immunosuppressive drugs2 (1.2)
E03.2Hypothyroidism due to medicaments and other exogenous substances2 (1.2)
T78.3Angioneurotic edema2 (1.2)
T50.3Electrolytic, caloric, and water-balance agents2 (1.2)
T42.0Hydantoin derivatives2 (1.2)
E27.3Drug-induced adrenocortical insufficiency1 (0.6)
T50.2Carbonic-anhydrase inhibitors, benzothiadiazides and other diuretics1 (0.6)
L56.0Drug phototoxic response1 (0.6)
Z03.6Observation for suspected toxic effect from ingested substance1 (0.6)
T43.5Other and unspecified antipsychotics and neuroleptics1 (0.6)
T37.1Antimycobacterial drugs1 (0.6)
D61.1Drug-induced aplastic anemia due to chemotherapy1 (0.6)
T39.3Other nonsteroidal anti-inflammatory drugs (NSAID)1 (0.6)
T43.6Poisoning by psychostimulants with abuse potential1 (0.6)
H40.6Glaucoma secondary to drugs1 (0.6)
T42.4Poisoning by benzodiazepines1 (0.6)
T38.1Thyroid hormones and substitutes1 (0.6)
T37.8Other specified systemic anti-infectives and antiparasitics (poisoning)1 (0.6)
F13.2Mental and behavioral disorders due to use of sedatives or hypnotics: dependence syndrome1 (0.6)
T46.2Other antidysrhythmic drugs, not elsewhere classified1 (0.6)
T46.1Calcium-channel blockers (poisoning)1 (0.6)
T88.7Unspecified adverse event of drug or medicament1 (0.6)
L64.0Drug-induced androgenic alopecia1 (0.6)
T42.3Barbiturates (poisoning)1 (0.6)
G44.4Drug-induced headache, not elsewhere classified1 (0.6)
T44.7Beta-adrenoreceptor antagonists, not elsewhere classified1 (0.6)
N14.2Nephropathy induced by unspecified drug, medicament or biological substance1 (0.6)
T37.0Sulfonamides (poisoning)1 (0.6)
Figure A1. Receiver operator characteristic curve for adverse drug events.
Figure A1. Receiver operator characteristic curve for adverse drug events.
Pharmacy 08 00087 g0a1

References

  1. Hanlon, J.T.; Schmader, K.E.; Koronkowski, M.J.; Weinberger, M.; Landsman, P.B.; Samsa, G.P.; Lewis, I.K. Adverse drug events in high risk older outpatients. J. Am. Geriatr. Soc. 1997, 45, 945–948. [Google Scholar] [CrossRef]
  2. Parameswaran Nair, N.; Chalmers, L.; Peterson, G.M.; Bereznicki, B.J.; Castelino, R.L.; Bereznicki, L.R. Hospitalization in Older Patients Due to Adverse Drug Reactions—The Need for a Prediction Tool. Clin. Interv. Aging 2016, 11, 497–505. [Google Scholar] [CrossRef]
  3. Watanabe, J.H.; McInnis, T.; Hirsch, J.D. Cost of Prescription Drug-Related Morbidity and Mortality. Ann. Pharmacother. 2018, 52, 829–837. [Google Scholar] [CrossRef]
  4. Hakkarainen, K.M.; Hedna, K.; Petzold, M.; Hagg, S. Percentage of Patients with Preventable Adverse Drug Reactions and Preventability of Adverse Drug Reactions—A Meta-Analysis. PLoS ONE 2012, 7, e33236. [Google Scholar] [CrossRef]
  5. Parameswaran Nair, N.; Chalmers, L.; Connolly, M.; Bereznicki, B.J.; Peterson, G.M.; Curtain, C.; Castelino, R.L.; Bereznicki, L.R. Prediction of Hospitalization due to Adverse Drug Reactions in Elderly Community-Dwelling Patients (The PADR-EC Score). PLoS ONE 2016, 11, e0165757. [Google Scholar] [CrossRef]
  6. Brady, A.; Curtis, C.E.; Jalal, Z. Screening Tools Used by Clinical Pharmacists to Identify Elderly Patients at Risk of Drug-Related Problems on Hospital Admission: A Systematic Review. Pharmacy 2020, 8, 64. [Google Scholar] [CrossRef]
  7. Tangiisuran, B.; Scutt, G.; Stevenson, J.; Wright, J.; Onder, G.; Petrovic, M.; van der Cammen, T.J.; Rajkumar, C.; Davies, G. Development and Validation of a Risk Model for Predicting Adverse Drug Reactions in Older People During Hospital Stay: Brighton Adverse Drug Reactions Risk (BADRI) Model. PLoS ONE 2014, 9, e111254. [Google Scholar] [CrossRef]
  8. Onder, G.; Petrovic, M.; Tangiisuran, B.; Meinardi, M.C.; Markito-Notenboom, W.P.; Somers, A.; Rajkumar, C.; Bernabei, R.; van der Cammen, T.J. Development and Validation of a Score to Assess Risk of Adverse Drug Reactions Among In-Hospital Patients 65 Years or Older: The GerontoNet ADR Risk Score. Arch. Intern. Med. 2010, 170, 1142–1148. [Google Scholar] [CrossRef]
  9. Cicali, B.; Michaud, V.; Knowlton, C.H.; Turgeon, J. Application of a Novel Medication-Related Risk Stratification Strategy to a Self-Funded Employer Population. Benef. Q. 2018, 34, 49–55. [Google Scholar]
  10. Bouwmeester, C. The PACE program: Home-Based Long-Term Care. Consult. Pharm. 2012, 27, 24–30. [Google Scholar] [CrossRef]
  11. PACE by the Numbers. Available online: https://www.npaonline.org/sites/default/files/PDFs/pace_infographic_update_final_0719.pdf (accessed on 22 April 2020).
  12. Bankes, D.L.; Amin, N.S.; Bardolia, C.; Awadalla, M.S.; Knowlton, C.H.; Bain, K.T. Medication-Related Problems Encountered in the Program of All-Inclusive Care for the Elderly: An Observational Study. J. Am. Pharm. Assoc. 2020, 60, 319–327. [Google Scholar] [CrossRef] [PubMed]
  13. McCarrell, J. PACE: An Interdisciplinary Community-Based Practice Opportunity for Pharmacists. Sr. Care Pharm. 2019, 34, 439–443. [Google Scholar] [CrossRef] [PubMed]
  14. Vouri, S.M.; Tiemeier, A. The Ins and Outs of Pharmacy Services at a Program of All-Inclusive Care for the Elderly. Consult. Pharm. 2012, 27, 803–807. [Google Scholar] [CrossRef] [PubMed]
  15. Bain, K.T.; Matos, A.; Knowlton, C.H.; McGain, D. Genetic Variants and Interactions from a Pharmacist-Led Pharmacogenomics Service for PACE. Pharmacogenomics 2019, 20, 709–718. [Google Scholar] [CrossRef] [PubMed]
  16. Bankes, D.L.; Schamp, R.O.; Knowlton, C.H.; Bain, K.T. Prescriber-Initiated Engagement of Pharmacists for Information and Intervention in Programs of All-Inclusive Care for the Elderly. Pharmacy 2020, 8, 24. [Google Scholar] [CrossRef]
  17. Sakaeda, T.; Tamon, A.; Kadoyama, K.; Okuno, Y. Data Mining of the Public Version of the FDA Adverse Event Reporting System. Int. J. Med. Sci. 2013, 10, 796–803. [Google Scholar] [CrossRef]
  18. Boustani, M.; Campbell, N.; Munger, S.; Maidment, I.; Fox, C. Impact of Anticholinergics on the Aging Brain: A Review and Practical Application. Aging Health 2008, 4, 311–320. [Google Scholar] [CrossRef]
  19. Salahudeen, M.S.; Duffull, S.B.; Nishtala, P.S. Anticholinergic Burden Quantified by Anticholinergic Risk Scales and Adverse Outcomes in Older People: A Systematic Review. BMC Geriatr. 2015, 15, 31. [Google Scholar] [CrossRef]
  20. Campbell, N.L.; Perkins, A.J.; Bradt, P.; Perk, S.; Wielage, R.C.; Boustani, M.A.; Ng, D.B. Association of Anticholinergic Burden with Cognitive Impairment and Health Care Utilization Among a Diverse Ambulatory Older Adult Population. Pharmacotherapy 2016, 36, 1123–1131. [Google Scholar] [CrossRef]
  21. Paul, K.J.; Walker, R.L.; Dublin, S. Anticholinergic Medications and Risk of Community-Acquired Pneumonia in Elderly Adults: A Population-Based Case-Control Study. J. Am. Geriatr. Soc. 2015, 63, 476–485. [Google Scholar] [CrossRef]
  22. West, T.; Pruchnicki, M.C.; Porter, K.; Emptage, R. Evaluation of Anticholinergic Burden of Medications in Older Adults. J. Am. Pharm. Assoc. 2013, 53, 496–504. [Google Scholar] [CrossRef] [PubMed]
  23. Gerretsen, P.; Pollock, B.G. Rediscovering Adverse Anticholinergic Effects. J. Clin. Psychiatry 2011, 72, 869–870. [Google Scholar] [CrossRef] [PubMed]
  24. Ruxton, K.; Woodman, R.J.; Mangoni, A.A. Drugs with Anticholinergic Effects and Cognitive Impairment, Falls and All-Cause Mortality in Older Adults: A Systematic Review and Meta-Analysis. Br. J. Clin. Pharmacol. 2015, 80, 209–220. [Google Scholar] [CrossRef] [PubMed]
  25. Gray, S.L.; Hanlon, J.T. Anticholinergic Drugs and Dementia in Older Adults. BMJ 2018, 361, k1722. [Google Scholar] [CrossRef]
  26. Szabo, S.M.; Gooch, K.; Schermer, C.; Walker, D.; Lozano-Ortega, G.; Rogula, B.; Deighton, A.; Vonesh, E.; Campbell, N. Association Between Cumulative Anticholinergic Burden and Falls and Fractures in Patients with Overactive Bladder: US-based Retrospective Cohort Study. BMJ Open 2019, 9, e026391. [Google Scholar] [CrossRef]
  27. Linjakumpu, T.; Hartikainen, S.; Klaukka, T.; Koponen, H.; Kivela, S.L.; Isoaho, R. A Model to Classify the Sedative Load of Drugs. Int. J. Geriatr. Psychiatry 2003, 18, 542–544. [Google Scholar] [CrossRef]
  28. Hanlon, J.T.; Boudreau, R.M.; Roumani, Y.F.; Newman, A.B.; Ruby, C.M.; Wright, R.M.; Hilmer, S.N.; Shorr, R.I.; Bauer, D.C.; Simonsick, E.M.; et al. Number and Dosage of Central Nervous System Medications on Recurrent Falls in Community Elders: The Health, Aging and Body Composition Study. J. Gerontol. A Biol. Sci. Med. Sci. 2009, 64, 492–498. [Google Scholar] [CrossRef]
  29. Wright, R.M.; Roumani, Y.F.; Boudreau, R.; Newman, A.B.; Ruby, C.M.; Studenski, S.A.; Shorr, R.I.; Bauer, D.C.; Simonsick, E.M.; Hilmer, S.N.; et al. Effect of Central Nervous System Medication Use on Decline in Cognition in Community-Dwelling Older Adults: Findings from the Health, Aging And Body Composition Study. J. Am. Geriatr. Soc. 2009, 57, 243–250. [Google Scholar] [CrossRef]
  30. Aspinall, S.L.; Springer, S.P.; Zhao, X.; Cunningham, F.E.; Thorpe, C.T.; Semla, T.P.; Shorr, R.I.; Hanlon, J.T. Central Nervous System Medication Burden and Risk of Recurrent Serious Falls and Hip Fractures in Veterans Affairs Nursing Home Residents. J. Am. Geriatr. Soc. 2019, 67, 74–80. [Google Scholar] [CrossRef]
  31. Hart, L.A.; Marcum, Z.A.; Gray, S.L.; Walker, R.L.; Crane, P.K.; Larson, E.B. The Association Between Central Nervous System-Active Medication Use and Fall-Related Injury in Community-Dwelling Older Adults with Dementia. Pharmacotherapy 2019, 39, 530–543. [Google Scholar] [CrossRef]
  32. Haugaa, K.H.; Bos, J.M.; Tarrell, R.F.; Morlan, B.W.; Caraballo, P.J.; Ackerman, M.J. Institution-wide QT Alert System Identifies Patients with a High Risk of Mortality. Mayo Clin. Proc. 2013, 88, 315–325. [Google Scholar] [CrossRef] [PubMed]
  33. Zhang, Y.; Post, W.S.; Blasco-Colmenares, E.; Dalal, D.; Tomaselli, G.F.; Guallar, E. Electrocardiographic QT Interval and Mortality: A Meta-Analysis. Epidemiology 2011, 22, 660–670. [Google Scholar] [CrossRef] [PubMed]
  34. Franceschi, M.; Scarcelli, C.; Niro, V.; Seripa, D.; Pazienza, A.M.; Pepe, G.; Colusso, A.M.; Pacilli, L.; Pilotto, A. Prevalence, Clinical Features and Avoidability of Adverse Drug Reactions as Cause of Admission to a Geriatric Unit: A Prospective Study of 1756 Patients. Drug Saf. 2008, 31, 545–556. [Google Scholar] [CrossRef]
  35. Dechanont, S.; Maphanta, S.; Butthum, B.; Kongkaew, C. Hospital Admissions/Visits Associated with Drug-Drug Interactions: A Systematic Review and Meta-Analysis. Pharmacoepidemiol. Drug Saf. 2014, 23, 489–497. [Google Scholar] [CrossRef] [PubMed]
  36. Verbeurgt, P.; Mamiya, T.; Oesterheld, J. How Common Are Drug and Gene Interactions? Prevalence in a Sample of 1143 Patients with CYP2C9, CYP2C19 and CYP2D6 Genotyping. Pharmacogenomics 2014, 15, 655–665. [Google Scholar] [CrossRef]
  37. Fischer, H.D.; Juurlink, D.N.; Mamdani, M.M.; Kopp, A.; Laupacis, A. Hemorrhage During Warfarin Therapy Associated with Cotrimoxazole and Other Urinary Tract Anti-Infective Agents: A Population-Based Study. Arch. Intern. Med. 2010, 170, 617–621. [Google Scholar] [CrossRef] [PubMed]
  38. Hohl, C.M.; Karpov, A.; Reddekopp, L.; Stausberg, J. ICD-10 Codes Used to Identify Adverse Drug Events in Administrative Data: A Systematic Review. J. Am. Med. Inf. Assoc. 2013, 21, 547–557. [Google Scholar] [CrossRef]
  39. Hazell, L.; Shakir, S.A. Under-reporting of Adverse Drug Reactions: A Systematic Review. Drug Saf. 2006, 29, 385–396. [Google Scholar] [CrossRef]
  40. Keehan, S.P.; Stone, D.A.; Poisal, J.A.; Cuckler, G.A.; Sisko, A.M.; Smith, S.D.; Madison, A.J.; Wolfe, C.J.; Lizonitz, J.M. National Health Expenditure Projections, 2016-25: Price Increases, Aging Push Sector To 20 Percent oOf Economy. Health Aff. (Millwood) 2017, 36, 553–563. [Google Scholar] [CrossRef]
  41. Hough, A.; Vartan, C.M.; Groppi, J.A.; Reyes, S.; Beckey, N.P. Evaluation of Clinical Pharmacy Interventions in a Veterans Affairs Medical Center Primary Care Clinic. Am. J. Health Syst. Pharm. 2013, 70, 1168–1172. [Google Scholar] [CrossRef]
  42. Sloane, P.D.; Oudenhoven, M.D.; Broyles, I.; McNabney, M. Challenges to Cost-Effective Care of Older Adults with Multiple Chronic Conditions: Perspectives of Program of All-Inclusive Care for the Elderly Medical Directors. J. Am. Geriatr. Soc. 2014, 62, 564–565. [Google Scholar] [CrossRef] [PubMed]
  43. Adams, W.L.; McIlvain, H.E.; Lacy, N.L.; Magsi, H.; Crabtree, B.F.; Yenny, S.K.; Sitorius, M.A. Primary Care for Elderly People: Why Do Doctors Find It So Hard? Gerontologist 2002, 42, 835–842. [Google Scholar] [CrossRef] [PubMed]
  44. Riordan, D.O.; Byrne, S.; Fleming, A.; Kearney, P.M.; Galvin, R.; Sinnott, C. GPs’ Perspectives on Prescribing for Older People in Primary Care: A Qualitative Study. Br. J. Clin. Pharmacol. 2017, 83, 1521–1531. [Google Scholar] [CrossRef] [PubMed]
  45. Ailabouni, N.J.; Nishtala, P.S.; Tordoff, D.; Mangin, J.M. Challenges and Enablers of Deprescribing: A General Practitioner Perspective. PLoS ONE 2016, 11, e0151066. [Google Scholar] [CrossRef] [PubMed]
  46. Filippoli, J.; Bottos, K.; Knowlton, O.V. How Does an Evidence-Based Medication Risk Score Affect Clinicians’ Attitudes about De-Prescribing and Medication Changes for Older Adults? In Proceedings of the American Public Health Association Annual Meeting and Expo, Philadelphia, PA, USA, 5 November 2019. [Google Scholar]
  47. Hohl, C.M.; Kuramoto, L.; Yu, E.; Rogula, B.; Stausberg, J.; Sobolev, B. Evaluating Adverse Drug Event Reporting in Administrative Data from Emergency Departments: A Validation Study. BMC Health Serv. Res. 2013, 13, 473. [Google Scholar] [CrossRef]
Figure 1. Data management and workflow diagram.
Figure 1. Data management and workflow diagram.
Pharmacy 08 00087 g001
Figure 2. Distribution of MedWise Risk Score™ among PACE participants. Abbreviations: PACE: Programs of All-Inclusive Care for the Elderly.
Figure 2. Distribution of MedWise Risk Score™ among PACE participants. Abbreviations: PACE: Programs of All-Inclusive Care for the Elderly.
Pharmacy 08 00087 g002
Figure 3. Weighted regression showing relationship between the MedWise Risk Score™ and participants with ≥1 adverse drug event.
Figure 3. Weighted regression showing relationship between the MedWise Risk Score™ and participants with ≥1 adverse drug event.
Pharmacy 08 00087 g003
Figure 4. Weighted regression showing relationship between MedWise Risk Score™ and total annual (2018) medical costs in the total population.
Figure 4. Weighted regression showing relationship between MedWise Risk Score™ and total annual (2018) medical costs in the total population.
Pharmacy 08 00087 g004
Figure 5. Weighted regression showing relationship between MedWise Risk Score™ and emergency department visits in the total population.
Figure 5. Weighted regression showing relationship between MedWise Risk Score™ and emergency department visits in the total population.
Pharmacy 08 00087 g005
Figure 6. Weighted regression showing relationship between MedWise Risk Score™ and hospital admissions in the total population.
Figure 6. Weighted regression showing relationship between MedWise Risk Score™ and hospital admissions in the total population.
Pharmacy 08 00087 g006
Figure 7. Weighted regression showing relationship between MedWise Risk Score™ and hospital length of stay in the total population.
Figure 7. Weighted regression showing relationship between MedWise Risk Score™ and hospital length of stay in the total population.
Pharmacy 08 00087 g007
Table 1. Overall study demographics.
Table 1. Overall study demographics.
CharacteristicTotalParticipants without ADEsParticipants with ADEs
PACE participants, n (%)1965 (100)1837 (93.5)128 (6.5)
Age, mean ± SD (range)76.8 ± 9.9 (56–104)77.1 ± 9.9 (56–104)72.9 ± 9.7 (56–96)
Age, median (IQR)77 (69, 84)77 (69, 84)71 (66, 80)
Female, n (%)1282 (65.2)1206 (65.7)76 (59.4)
Number of profiled medications,
mean ± SD (range) 1
13.4 ± 5.8 (1–42)13.2 ± 5.8 (1–42)16.2 ± 6.1 (4–37)
Number of profiled medications, median (IQR) 113 (9, 17)13 (9, 17)15.5 (12, 20)
MedWise Risk Score™, mean ± SD (range)18.5 ±7.8 (2–40)18.2 ± 7.7 (2–40)23.1 ± 7.8 (3–40)
MedWise Risk Score™, median (IQR)18 (12, 25)18 (12, 24)23 (17.75, 29)
Ethnicity, n (%)
Non-Hispanic White1140 (58.0)1061 (57.8)79 (61.7)
Black454 (23.1)422 (23.0)32 (25.0)
Hispanic142 (7.2)135 (7.3)7 (5.5)
Asian or Pacific Islander59 (3.0)58 (3.2)1 (0.8)
American Indian/Alaskan Native7 (0.4)7 (0.4)0 (0.0)
Unknown or other163 (8.3)154 (8.4)9 (7.0)
Top chronic comorbidities (by diagnosis code), n (%)
Essential (primary) hypertension1233 (62.7)1135 (61.8)98 (76.6)
Type 2 diabetes mellitus without complications790 (40.2)727 (39.6)63 (49.2)
Hyperlipidemia, unspecified622 (31.7)573 (31.2)49 (38.3)
Atherosclerotic heart disease of native coronary artery without angina pectoris428 (21.8)388 (21.1)40 (31.2)
Hypothyroidism, unspecified421 (21.4)385 (21.0)36 (28.1)
Unspecified dementia without behavioral disturbance406 (20.7)374 (20.4)32 (25.0)
Chronic obstructive pulmonary disease, unspecified373 (19.0)335 (18.2)38 (29.7)
Chronic kidney disease, stage 3371 (18.9)335 (18.2)36 (28.1)
Heart failure, unspecified349 (17.8)314 (17.1)35 (27.3)
Pertinent risk outcomes
Total medical expenditures, mean (95% CI), USD$26,299.97
(24,917.54, 27,682.39)
$25,087.07
(23,720.95, 26,453.19)
$43,706.95
(36,149.34, 51,264.56)
Hospital admissions, mean (95% CI),
n per participant per year
0.37 (0.33, 0.41)0.34 (0.3, 0.37)0.84 (0.62, 1.05)
Hospital length of stay, mean (95% CI), days2.1 (1.8, 2.4)1.9 (1.7, 2.2)4.8 (3.3, 6.2)
ED visits, mean (95% CI), n per participant per year0.63 (0.57, 0.69) 0.59 (0.53, 0.65)1.21 (0.88, 1.54)
PACE organizations, n12
Geographic location of PACE, n (%)
Northeast3 (25.0)
South4 (33.3)
Midwest1 (8.3)
West4 (33.3)
PACE census size, n (%)
<120 participants1 (8.3)
120–220 participants7 (58.3)
>220 participants4 (33.3)
Abbreviations: PACE: Programs of All-Inclusive Care for the Elderly; ADE: adverse drug event; ED: emergency department; IQR: interquartile range; CI: confidence interval. 1 Medication counts obtained using December 2018 data. To facilitate reading, subsections are bolded.
Table 2. Most common 1 adverse drug events (ADEs) identified by ICD-10 codes.
Table 2. Most common 1 adverse drug events (ADEs) identified by ICD-10 codes.
ICD CodeDescriptionn (%)
F11.2Mental and behavioral disorders due to use of opioids: dependence syndrome20 (11.8)
F11.9Mental and behavioral disorders due to use of opioids: unspecified mental and behavioral disorder12 (7.1)
D68.3Hemorrhagic disorder due to circulating anticoagulants11 (6.5)
F19.1Mental and behavioral disorders due to multiple drug use and use of other psychoactive substances: harmful use 9 (5.3)
T50.9Poisoning: other and unspecified drugs, medicaments, and biological substances9 (5.3)
G24.0Drug-induced dystonia7 (4.1)
L27.0Generalized skin eruption due to drugs and medicaments6 (3.5)
T40.2Poisoning by other opioids (includes morphine and codeine)5 (2.9)
G21.1Other drug-induced secondary parkinsonism5 (2.9)
T45.5Anticoagulants (poisoning)5 (2.9)
F19.2Mental and behavioral disorders due to multiple drug use and use of other psychoactive substances: dependence syndrome 5 (2.9)
Abbreviations: ICD-10: International Classification of Diseases and Health Related Problems, 10th revision. 1 Accounts for 55.3% (n = 94) of the total ADEs identified by ICD-10 codes. In total, 54 unique ICD-10 codes identified ADEs. These codes appeared 170 times during 2018 claims.
Back to TopTop