Skip to Content
MDPIMDPI

Advanced

JPMJournal of Personalized Medicine
PDF
  • Article
  • Open Access

23 July 2020

13 Pages

An Implementation Science Framework to Develop a Clinical Decision Support Tool for Familial Hypercholesterolemia

,
,
,
,
,
,
,
,
…
1
Department of Cardiovascular Medicine, Mayo Clinic, Rochester, MN 55905, USA
2
Center for Science of HealthCare Delivery, Mayo Clinic, Rochester, MN 55905, USA
3
User Experience Research, Saharafox Creative Agency, Rochester, MN 55905, USA
4
Department of Information Technology, Mayo Clinic, Rochester, MN 55905, USA
J. Pers. Med.2020, 10(3), 67;https://doi.org/10.3390/jpm10030067
Version Notes
Review Reports
Order Reprints

Abstract

Electronic health record (EHR)-based clinical decision support (CDS) can address the low awareness and undertreatment of familial hypercholesterolemia (FH), a disorder associated with a markedly increased risk of coronary heart disease. We aimed to incorporate provider perspectives into the development and implementation of a CDS tool for FH. An implementation science framework and a user-centered design process were used to create a CDS tool for FH. Primary care physicians and specialist physicians participated in qualitative interviews, usability testing and an implementation survey. The CDS was configured in two formats—a best practice alert (BPA) and an in-basket message and subsequently deployed in the EHR in silent mode. The key themes that emerged from the analysis of interview transcripts included understanding and awareness of FH, clinical workflow, physician preferences and value of CDS tools, perspectives on patient needs and values and dissemination and implementation. Recommendations related to usability included preferred CDS format and placement, content, timing and frequency, and level of alert urgency/prioritization. In response to the survey, 84.6% of physicians agreed that the CDS would improve early FH diagnosis and 92.3% agreed that it would help them identify and manage FH patients. Physician feedback led to iterative CDS refinement. In summary, we developed a CDS tool for FH using an implementation science framework and physician feedback. Initial deployment revealed a significant burden of FH and the potential for the CDS tool to have a large impact.

1. Introduction

Familial hypercholesterolemia (FH), is often monogenic in etiology, remains vastly undiagnosed and untreated in the United States and is therefore an ideal use case for the development of a clinical decision support (CDS) tool aimed at improving case detection and treatment as well as promoting cascade testing [1,2,3]. Awareness of FH is low among both patients and providers [4,5]. Providers in both primary care as well as specialty clinics such as cardiology often perceive all cases of hypercholesterolemia similarly, failing to recognize the higher risk of coronary heart disease (CHD) and the need for familial testing when the etiology is genetic. Initiation of appropriate lipid lowering medications is often delayed and goal low-density lipoprotein cholesterol (LDL-C) levels are achieved only in a minority of FH patients [1,6].
In the face of the ever-increasing volume and complexity of medical data, the delivery of CDS to health care providers at the point-of-care is an urgent need. Nowhere is this need more acute than for the practice of genomic medicine in the primary care setting where providers are often uncertain about the interpretation and application of genomic test results to patient care [7,8]. In such settings, CDS tools can provide guidance related to the interpretation of genomic test results and their subsequent application to patient management [9,10]. However, healthcare providers often express concern about alert fatigue and poor usability/functionality of CDS tools, which may partly stem from poor design and a lack of participation by end users during the development phase [11,12,13]. Disappointingly, electronic health record (EHR) systems have added complexity to clinical workflows, reduced provider–patient interactions and increased cognitive burden and burnout [14].
A potential means of making EHR systems more user friendly and ‘intelligent’ is by integrating CDS tools that are designed and implemented based on input from relevant stakeholders such as providers. This could, in turn, reduce provider cognitive burden and increase efficiency and satisfaction [15,16]. Implementation science, defined as the study of methods to facilitate uptake and integration of evidence-based health interventions into routine practice, can be applied to better understand the barriers and facilitators of CDS implementation. The use of an implementation science framework can ensure that CDS tools are optimally designed and integrated in clinical practice to enhance point-of-care management [17,18,19,20,21,22]. We therefore used an implementation science framework to develop a CDS tool for FH based on physician feedback from qualitative interviews, usability testing and an implementation survey.

2. Materials and Methods

This study was considered exempt by the Mayo Clinic Institutional Review Board and was conducted between November 2018 and October 2019.

2.1. Participants and Recruitment

The target sample size for the qualitative interviews and usability testing was ~8–10 participants based on evidence from usability studies, which indicates that up to 80–85% of usability issues can be identified by the first eight participants [23,24]. To achieve the target sample size, we conducted purposive sampling with the following inclusion criteria: (i) staff physicians at the Mayo Clinic campus in Rochester, Minnesota; (ii) from primary care including the departments of family medicine and community internal medicine as well as specialty fields such as cardiology and vascular medicine; and (iii) with previous experience using institutional CDS tools. Physicians were contacted by templated emails that contained invitations to participate in the interviews and to avoid potential coercion; they were not contacted by a supervisor.

2.2. Physician Interviews and Usability Testing

Interviews with physicians were conducted from November 2018 to February 2019. Qualitative methodologies and a user-centered design process were applied within a type 1 effectiveness-implementation hybrid framework [24,25,26,27,28,29]. Semi-structured qualitative interviews and usability testing sessions were conducted to obtain physician perspectives on two EHR-based CDS prototypes. The first prototype was a best practice alert (BPA)—A clinical reminder with evidence-based management guidelines that would passively display to providers (requiring them to click on the alert to open it) at the point-of-care. The second prototype was an asynchronous in-basket alert—A type of inbox message that contained patient laboratory data and would be visible to providers upon logging into the EHR. Both CDS formats (BPA and in-basket) were linked to an automated detection algorithm in the EHR that identified individuals as ‘possible FH’ cases, defined as LDL-C ≥ 190 mg/dL in the absence of secondary causes of hypercholesterolemia. Both CDS formats were presented during each interview and the “Think Aloud” usability technique was applied; physicians were asked to talk out loud and verbalize their thoughts while interacting with the CDS prototypes [30,31,32]. Four rounds of physician interviews and usability testing were conducted—new physicians, without prior exposure to the CDS tool, participated in each round and their feedback informed CDS prototyping and iterative refinements in subsequent rounds.
Interviews and usability testing were conducted by research study team members, including a physician, a qualitative researcher, a user experience/user interface expert and a research program coordinator. Each interview lasted one hour, was audio recorded and transcribed. Participants were given lunch during the interview and received a USD 100 honorarium. A semi-structured interview guide was developed and tested in a ‘mock’ interview to ensure that the questions were comprehensive (see Supplementary file Methods S1). The results from the mock interview were not included in the thematic analysis.

2.3. Implementation Survey

Following each interview, physicians were asked to complete a twenty five-question implementation survey to elicit multi-level contextual factors that could influence CDS implementation in clinical practice (see Supplementary file Methods S2). Modified from Weiner et al., twelve survey questions assessed three implementation outcome measures: Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM) and Feasibility of Intervention Measure (FAM) [33]. The remaining thirteen survey questions were modified from the Consolidated Framework for Implementation Research (CFIR) [34,35].

2.4. Deployment of CDS in Silent Mode

Both the BPA and in-basket CDS formats were deployed in ‘silent mode’ across all Mayo Clinic sites, including the Mayo Clinic Health System (MCHS) for a three-month period (July 2019 to October 2019). The silent mode setting would trigger the CDS alert in the EHR for providers in primary care, general internal medicine and cardiovascular medicine; however the alert was set to not display for providers, triggering only in the EHR background. The goal of deploying in silent mode was to gather initial metrics on the triggering of CDS at different sites and use these metrics as a measure of the burden of ‘possible FH’. Each alert was cued by a different action; the BPA fired when a patient’s EHR was opened, while the in-basket fired when results of a newly ordered lipid were placed in the EHR.

2.5. Data Analysis

The Framework Method was applied to transcript coding [36,37]. An inductive approach was used to generate themes from the transcribed data through open coding of transcripts. An initial codebook was developed by two qualitative researchers (HB and LP) who independently coded two transcripts line-by-line and then read each transcript in a group setting to reach a consensus on applied codes and ensure inter-rater reliability. After the codebook was established, the remaining transcripts were coded independently by two research team members and any inconsistencies in assigned codes were resolved through group discussion at bi-weekly meetings. All transcribed data were imported into ‘Reframer’—A qualitative research software tool (Optimal Workshop 2015, Wellington, New Zealand)—And each code was built into the tool as a ‘tag’ that could be applied to multiple observations. Survey responses of primary care physicians (PCPs) and specialists were compared using Wilcoxon rank-sum test to identify the median score, interquartile range (IQR) and any statistically significant (p < 0.05) differences in the responses between PCPs and specialists. Descriptive data, including frequency of responses, were also noted.

3. Results

Thirteen physicians from primary care and cardiovascular medicine were recruited to the study and gave informed consent prior to participation in the qualitative interviews and usability testing—seven were primary care physicians (PCPs) and six were specialist physicians. Participants included physicians in leadership roles and those serving on institutional committees. Participant demographic characteristics are summarized in Table 1.
Table 1. Physician characteristics (n = 13).

3.1. Physician Interview Findings

Analysis of the physician interview transcripts identified five key themes and eleven subthemes along with representative quotes (Table 2).
Table 2. Themes and representative quotations identified from physician interviews.

3.1.1. Understanding and Awareness of FH

This theme encompassed physicians’ current awareness and knowledge of FH and their perceived role in the detection and management of FH patients.
1.
Subtheme: Physicians awareness of FH
Amongst both PCPs and specialists, there was a lack of FH awareness—physicians did not always distinguish between hypercholesterolemia of a genetic etiology versus hypercholesterolemia due to other causes. In some instances, both were perceived as being similar and requiring similar management.
2.
Subtheme: Physicians perceived scope of work
PCPs and specialists agreed that all physicians were responsible for diagnosing FH; however, they differed in their views on who was responsible for the management of FH patients. Some physicians preferred to manage FH patients themselves, while others preferred referral to a specialist.

3.1.2. Clinical Workflow

Clinical workflow aimed to understand physician workflows in primary care and specialty settings and gain insights into the next steps physicians were likely to take after a patient had been diagnosed with FH.
1.
Subtheme: Diagnosis in workflow
Determining how physicians established a diagnosis in workflow centered on understanding when tasks such as ordering laboratory tests and reviewing results were performed. Physicians identified CDS alert relevance as being based on both encounter type and patient type. PCPs with high patient volumes were likely to have limited time to review alerts.
2.
Subtheme: Next steps for physicians
For a patient with suspected FH, the next steps for physicians included prescribing medication, counseling on lifestyle changes, ordering tests and ruling out secondary causes of hypercholesterolemia. Most physicians were willing to refer patients to the institutional FH Clinic if this was the appropriate next step in management.

3.1.3. Physician Preferences and Value of CDS Tools

This theme encompassed physician perspectives on CDS tools and whether these tools impacted provider cognitive burden and time. Physicians also gave insights into their perceived value of the institutional FH Clinic.
1.
Subtheme: CDS tool preferences
Most physicians had previously used one or more CDS tools and viewed them positively. However, a physician’s personal style or prior knowledge of a patient was likely to influence their decision on whether or not to use a specific CDS tool. With a relatively recent transition to a new EHR system at the time of interviews, some physicians continued to face challenges in interacting with the new EHR and highlighted the need to find more efficient ways to carry out routine tasks.
2.
Subtheme: Cognitive load
A common feedback from all physicians was that the increasing number of CDS alerts and in-basket messages were contributing to rising levels of alert fatigue and information overload, subsequently resulting in CDS alerts being ignored or bypassed.
3.
Subtheme: Value of FH Clinic referral to physicians
Physicians agreed that having the option to refer patients with ‘possible FH’ to the institutional FH Clinic was likely going to reduce their cognitive burden and would be useful for complex cases such as those where: (a) goal LDL-cholesterol was not being attained on maximum statin therapy; (b) a patient had statin intolerance or (c) cases where genetic testing was warranted.

3.1.4. Perspectives on Patient Needs and Values

Physicians highlighted the importance of engaging with patients through digital tools and also shared their insights on the value of the institutional FH Clinic to both patients and their at-risk family members.
1.
Subtheme: Patient engagement through the use of digital tools
Physicians indicated that there was a need for patient education materials that could be printed and attached to messages as well as videos that could be shared directly through the institutional patient portal. Most physicians agreed that there was value in shared decision-making tools such as patient decision aids, especially if these were available on the internet and easily accessible. Physicians also highlighted the importance of being able to share the computer screen displaying the decision aid with patients, so as to engage them during clinical encounters.
2.
Subtheme: Value of FH Clinic referral to patients
Physicians highlighted that the value of a referral to the institutional FH Clinic for patients and their family members centered on: (a) obtaining access to resources available specifically in the FH Clinic; (b) cascade testing of family members; (c) receiving access to new medications such as PCSK9 inhibitors and (d) facilitating appointment scheduling for genetic testing.

3.1.5. Dissemination and Implementation

This theme encompassed multi-level barriers and facilitators that could potentially impact the adoption of CDS in clinical practice.
1.
Subtheme: Facilitators to Dissemination and Implementation
Physicians suggested that departmental meetings, conferences, grand rounds as well as newsletters could be used to promote awareness of the CDS tool and facilitate its uptake in practice. They also highlighted the need to obtain support from institutional champions, opinion leaders, early adopters and others likely to influence the attitudes and behaviors of providers.
2.
Subtheme: Barriers to Dissemination and Implementation
The main barriers identified by physicians that were likely going to impact utilization of the CDS tool in practice included use of pre-existing CDS tools that may prevent adoption of new tools, limited time in patient encounters to view and act upon CDS alerts and lack of institutional guidance and education on new CDS availability.

3.2. Usability Recommendations

The usability testing of both CDS prototypes led to physician feedback that broadly fell into one of four recommendation categories summarized in Table 3.
Table 3. Usability recommendations with corresponding descriptions and representative quotations highlighted from usability testing with providers.

3.3. Implementation Survey Results

The majority of PCPs and specialists responded positively to questions that assessed implementation outcome measures; 84.6% (n = 11) of physicians indicated that they liked the tool and it seemed easy to use (Table 4). Similarly, positive responses were noted for questions pertaining to the CFIR constructs; 84.6% (n = 11) of physicians indicated that the CDS tool would improve early diagnosis of patients with FH, while 92.3% (n = 12) of physicians agreed that the CDS tool would help them identify, refer or manage FH patients. However, in response to the statement ‘this tool will not increase the time needed with a patient’, 69.2% (n = 9) of physicians were either neutral or negative in their response. Overall, there was agreement in responses between both PCPs and specialists (see Supplementary Table S1).
Table 4. Implementation outcome measures by Weiner et al. and Consolidated Framework for Implementation Research (CFIR) constructs were assessed through a post-interview survey conducted with PCPs and specialists.

3.4. Silent Mode Metrics

The rapid prototyping of the BPA and in-basket continued until no new physician feedback was obtained from three consecutive interviews and thematic saturation was attained. The final BPA and in-basket were then deployed in the EHR in ‘silent mode’ for gathering initial metrics (Figure 1). During a three-month period, the BPA triggered 5415 times for 1440 unique patients, while the in-basket triggered 953 times for 924 unique patients at the three major Mayo Clinic sites and the MCHS.
Figure 1. (a) The final best practice alert or BPA deployed in the EHR. (b) The final in-basket message linked to a lipid panel report, deployed in the EHR in silent mode.

4. Discussion

The dearth of usable and relevant digital tools is a major hurdle to the implementation of genomic medicine [7,38]. In this report, we describe the creation of a genomic CDS tool for FH, using an implementation science framework aimed at evaluating CDS effectiveness and gaining early insights into clinical implementation through physician feedback. The relatively high prevalence but yet low awareness and detection of FH motivated the development of a CDS tool for FH. The work described herein could inform the creation of CDS tools for other genomic disorders in diverse settings.
The introduction of EHRs has been associated with increased provider cognitive load, reduced professional satisfaction and increased rates of burnout [39,40]. To address these concerns, the Office of the National Coordinator for Health Information Technology (HIT) proposed aligning EHRs and tools with clinical workflows, improving user interfaces to increase efficiency and implementing health IT based on stakeholder engagement [41]. This includes the creation of well-designed, usable CDS tools that can fill gaps in knowledge and provide guideline-based recommendations at the point-of-care.
We conducted qualitative interviews and usability testing with both PCPs and specialists; PCPs are often the first line of contact for most patients in the community setting, especially for needs such as the evaluation of lipids. They are also more likely to treat multiple family members and thus can play a pivotal role in increasing FH detection and screening within a community. However, physicians identified a number of barriers that were likely to reduce CDS utilization in clinical practice—the most common barrier being the increasing cognitive burden on providers due to EHR complexity and limited time during clinical encounters. These findings are supported by previous studies that examined barriers to CDS implementation [9,42,43].
To promote CDS adoption in clinical practice, physician feedback emphasized the need to deploy passive alerts that would be less disruptive to workflows. They also highlighted that in-basket message alerts should be linked to new lipid panel reports to simplify workflows. Physicians recommended shortening and reorganizing the alert content for clarity and embedding a link to an order set specific for FH so as to reduce the number of clicks. Physicians also recommended including a link to AskMayoExpert, a Mayo Clinic knowledge resource with a topic module dedicated to FH that provides information on clinical presentation, diagnostic criteria and management [44]. AskMayoExpert also has links to additional knowledge modules, thereby enabling providers to access a number of relevant topics without workflow disruption [45].
Another important insight was that concurrent to the development of a CDS tool, an attempt should be made to increase awareness of the tool, provide education on how to use it and obtain support from institutional leadership. This feedback led to efforts to increase FH awareness amongst providers and highlight the CDS tool for FH at the Mayo Clinic Rochester campus. Rapid review pocket cards, which gave an overview of FH, were developed and distributed to PCPs and specialists. Several grand rounds were conducted across the institution by FH physician experts to disseminate knowledge and highlight the available CDS tool.
The implementation survey provided physician perspectives on contextual variables that were likely to influence CDS adoption in clinical practice. Most physicians indicated a receptive attitude towards CDS integration. The only survey item where the majority of the physicians gave either a neutral response or disagreed was regarding the CDS tool ‘not’ increasing time spent with a patient. This response reiterated the need for CDS to be designed to increase efficiency and not add to provider burden.
A relatively large number of both BPA and in-basket alerts fired during silent-mode deployment, indicating a high burden of ‘possible FH’. This suggests a potentially large impact of CDS deployment on health outcomes. Currently, both alerts remain deployed in silent mode and will be further evaluated in a clinical trial to assess the effect of the CDS on outcomes such as changes in patient management post-deployment.
A limitation of our study is that interviews were limited to physicians; the input of nurse practitioners, physician assistants, and residents in training may have relevance for clinical implementation of CDS tools in differing clinical workflows. Additionally, inherent EHR limitations prevented some physician feedback from being implemented in the final CDS tools that were deployed in the EHR. The number of physicians recruited for the study was modest; however, evidence from usability studies indicates that such a number is adequate for developing and establishing the tool and to initiate the application of the tool in clinical practice [23,24]. After initial implementation of the CDS tool, metrics for its use, including the utilization of the FH order set and AskMayoExpert knowledge resource will be evaluated to determine need for refinements and modifications. Such modifications, based on the pilot implementation, could further improve the FH CDS tool for use at the point-of-care.

5. Conclusions

Employing an implementation science framework with qualitative interviews and usability testing, physician feedback was obtained to guide the development of a CDS tool for FH and integrate it with the EHR for providers in primary care and cardiovascular medicine. A relatively large number of alerts fired during CDS deployment in silent mode, indicating a high burden of ‘possible FH’ and the potential for CDS to increase the point-of-care detection of FH. The process described here, along with the physician feedback obtained, can inform the creation of CDS tools for genomic disorders at other institutions and health systems.

Supplementary Materials

The following are available online at https://www.mdpi.com/2075-4426/10/3/67/s1, Supplementary Methods S1: Semi-structured interview guide used to conduct qualitative interviews with providers; Supplementary Methods S2: Post interview implementation survey administered to providers to understand contextual variables that could influence adoption of the clinical decision support (CDS) tool into practice; Table S1: Comparison of implementation survey responses between primary care providers and specialist providers using Wilcoxon Rank Sum test.

Author Contributions

Conceptualization, H.B., J.H.G., L.P., A.M. and I.J.K.; methodology, H.B., J.H.G., L.P., A.M., S.K., R.F., P.J.C. and I.J.K.; software, J.S.; formal analysis, H.B., J.H.G., L.P., L.M., A.M. and O.D.; investigation, H.B., J.H.G. and I.J.K.; resources, I.J.K.; writing—original draft preparation, H.B.; writing—review and editing, H.B. and I.J.K.; visualization, H.B. and I.J.K.; supervision, I.J.K.; project administration, J.H.G.; funding acquisition, I.J.K. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by National Heart, Lung, and Blood Institute (NHLBI) grants HL135879 and K24 HL137010 awarded to Iftikhar Kullo.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Safarova, M.S.; Kullo, I.J. My approach to the patient with familial hypercholesterolemia. Mayo Clin. Proc. 2016, 91, 770–786. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  2. Safarova, M.S.; Liu, H.; Kullo, I.J. Rapid identification of familial hypercholesterolemia from electronic health records: The SEARCH study. J. Clin. Lipidol. 2016, 10, 1230–1239. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  3. Lee, C.; Rivera-Valerio, M.; Bangash, H.; Prokop, L.; Kullo, I.J. New case detection by cascade testing in familial hypercholesterolemia: A systematic review of the literature. Circ. Genom. Precis. Med. 2019, 12, e002723. [Google Scholar] [CrossRef] [Scilit]
  4. Gidding, S.S.; Champagne, M.A.; de Ferranti, S.D.; Defesche, J.; Ito, M.K.; Knowles, J.W.; McCrindle, B.; Raal, F.; Rader, D.; Santos, R.D.; et al. The agenda for familial hypercholesterolemia. Circulation 2015, 132, 2167–2192. [Google Scholar] [CrossRef] [Scilit]
  5. Bangash, H.; Khan, F.; He, B.; Arce, M.; Kullo, I.J. Use of twitter to promote awareness of familial hypercholesterolemia. Circ. Genom. Precis. Med. 2019, 12, e002550. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  6. Degoma, E.M.; Ahmad, Z.S.; O’Brien, E.C.; Kindt, I.; Shrader, P.; Newman, C.B.; Pokharel, Y.; Baum, S.J.; Hemphill, L.C.; Hudgins, L.C.; et al. Treatment gaps in adults with heterozygous familial hypercholesterolemia in the United States: Data from the CASCADE-FH registry. Circ. Cardiovasc. Genet. 2016, 9, 240–249. [Google Scholar] [CrossRef] [Scilit]
  7. Manolio, T.A.; Abramowicz, M.; Al-Mulla, F.; Anderson, W.; Balling, R.; Berger, A.C.; Bleyl, S.; Chakravarti, A.; Chantratita, W.; Chisholm, R.L.; et al. Global implementation of genomic medicine: We are not alone. Sci. Transl. Med. 2015, 7, 290ps13. [Google Scholar] [CrossRef] [Scilit]
  8. Bryan, C.; Boren, S.A. The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: A systematic review of the literature. Inform. Prim. Care 2008, 16, 79–91. [Google Scholar] [CrossRef] [Scilit]
  9. Hasnie, A.A.; Kumbamu, A.; Safarova, M.S.; Caraballo, P.J.; Kullo, I.J. A clinical decision support tool for familial hypercholesterolemia based on physician input. Mayo Clin. Proc. Innov. Qual. Outcomes 2018, 2, 103–112. [Google Scholar] [CrossRef] [Scilit]
  10. Lobach, D.; Sanders, G.D.; Bright, T.J.; Wong, A.; Dhurjati, R.; Bristow, E.; Bastian, L.; Coeytaux, R.; Samsa, G.; Hasselblad, V.; et al. Enabling health care decision making through clinical decision support and knowledge management. Evid. Rep. Technol. Assess. 2012, 203, 1–784. [Google Scholar]
  11. Ash, J.S.; Sittig, D.F.; Campbell, E.M.; Guappone, K.P.; Dykstra, R.H. Some unintended consequences of clinical decision support systems. AMIA Annu. Symp. Proc. 2007, 26–30. [Google Scholar]
  12. Carroll, C.; Marsden, P.; Soden, P.; Naylor, E.; New, J.; Dornan, T. Involving users in the design and usability evaluation of a clinical decision support system. Comput. Methods Programs Biomed. 2002, 69, 123–135. [Google Scholar] [CrossRef] [Scilit]
  13. Ancker, J.S.; Edwards, A.; Nosal, S.; Hauser, D.; Mauer, E.; Kaushal, R.; with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med. Inform. Decis. Mak. 2017, 17, 36. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  14. Linder, J.A.; Schnipper, J.L.; Tsurikova, R.; Melnikas, A.J.; Volk, L.A.; Middleton, B. Barriers to electronic health record use during patient visits. AMIA Annu. Symp. Proc. 2006, 2006, 499–503. [Google Scholar]
  15. Sittig, D.F.; Wright, A.; Osheroff, J.A.; Middleton, B.; Teich, J.M.; Ash, J.S.; Campbell, E.; Bates, D.W. Grand challenges in clinical decision support. J. Biomed. Inform. 2008, 41, 387–392. [Google Scholar] [CrossRef] [Scilit]
  16. Shiffman, R.N. Towards effective implementation of a pediatric asthma guideline: Integration of decision support and clinical workflow support. Proc. Annu. Symp. Comput. Appl. Med. Care 1994, 797–801. [Google Scholar]
  17. Brownson, R.C.; Colditz, G.A.; Proctor, E.K. Dissemination and Implementation Research in Health: Translating Science to Practice; Oxford University Press: Oxford, UK, 2017. [Google Scholar]
  18. Glasgow, R.E.; Vinson, C.; Chambers, D.; Khoury, M.J.; Kaplan, R.M.; Hunter, C. National Institutes of Health approaches to dissemination and implementation science: Current and future directions. Am. J. Public Health 2012, 102, 1274–1281. [Google Scholar] [CrossRef] [Scilit]
  19. Elwyn, G.; Scholl, I.; Tietbohl, C.; Mann, M.; Edwards, A.G.; Clay, C.; Légaré, F.; van der Weijden, T.; Lewis, C.L.; Wexler, R.M.; et al. ‘Many miles to go …’: A systematic review of the implementation of patient decision support interventions into routine clinical practice. BMC Med. Inform. Decis. Mak. 2013, 13 (Suppl. 2), S14. [Google Scholar] [CrossRef] [Scilit]
  20. Hartzler, A.; McCarty, C.A.; Rasmussen, L.V.; Williams, M.S.; Brilliant, M.; Bowton, E.A.; Clayton, E.W.; Faucett, W.A.; Ferryman, K.; Field, J.R.; et al. Stakeholder engagement: A key component of integrating genomic information into electronic health records. Genet. Med. 2013, 15, 792–801. [Google Scholar] [CrossRef] [Scilit]
  21. Downing, G.J.; Boyle, S.N.; Brinner, K.M.; Osheroff, J.A. Information management to enable personalized medicine: Stakeholder roles in building clinical decision support. BMC Med. Inform. Decis. Mak. 2009, 9, 44. [Google Scholar] [CrossRef] [Scilit]
  22. Devaraj, S.; Sharma, S.K.; Fausto, D.J.; Viernes, S.; Kharrazi, H. Barriers and facilitators to clinical decision support systems adoption: A systematic review. JBAR 2014, 3, 36. [Google Scholar] [CrossRef] [Scilit]
  23. Virzi, R.A. Refining the test phase of usability evaluation: How many subjects is enough? Hum. Factors 1992, 34, 457–468. [Google Scholar] [CrossRef] [Scilit]
  24. Kastner, M.; Lottridge, D.; Marquez, C.; Newton, D.; Straus, S.E. Usability evaluation of a clinical decision support tool for osteoporosis disease management. Implement. Sci. 2010, 5, 96. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  25. Grando, A.; Manataki, A.; Furniss, S.K.; Duncan, B.; Solomon, A.; Kaufman, D.; Hirn, S.; Sunday, R.; Bouchereau, J.; Doebbeling, B.; et al. Multi-method study of electronic health records workflows. AMIA Annu. Symp. Proc. 2018, 2018, 498–507. [Google Scholar] [PubMed]
  26. Abouzahra, M.; Tan, J. The multi-level impact of clinical decision support system: A framework and a call for mixed methods evaluation. In Proceedings of the Pacific Asia Conference on Information Systems (PACIS), Chengdu, China, 24–28 June 2014. [Google Scholar]
  27. Curran, G.M.; Bauer, M.; Mittman, B.; Pyne, J.M.; Stetler, C. Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Med. Care 2012, 50, 217–226. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  28. Wolfenden, L.; Williams, C.M.; Wiggers, J.; Nathan, N.; Yoong, S.L. Improving the translation of health promotion interventions using effectiveness-implementation hybrid designs in program evaluations. Health Promot. J. Austr. 2016, 27, 204–207. [Google Scholar] [CrossRef] [Scilit]
  29. Li, A.C.; Kannry, J.L.; Kushniruk, A.; Chrimes, D.; McGinn, T.G.; Edonyabo, D.; Mann, D.M. Integrating usability testing and think-aloud protocol analysis with ‘near-live’ clinical simulations in evaluating clinical decision support. Int. J. Med. Inform. 2012, 81, 761–772. [Google Scholar] [CrossRef] [Scilit]
  30. Fonteyn, M.E.; Kuipers, B.; Grobe, S.J. A description of think aloud method and protocol analysis. Qual. Health Res. 1993, 3, 430–441. [Google Scholar] [CrossRef] [Scilit]
  31. Kushniruk, A.W.; Patel, V.L.; Cimino, J.J. Usability testing in medical informatics: Cognitive approaches to evaluation of information systems and user interfaces. Proc. AMIA Annu. Fall Symp. 1997, 218–222. [Google Scholar]
  32. Richardson, S.; Mishuris, R.; O’Connell, A.; Feldstein, D.; Hess, R.; Smith, P.; McCullagh, L.; McGinn, T.; Mann, D. ‘Think aloud’ and ‘Near live’ usability testing of two complex clinical decision support tools. Int. J. Med. Inform. 2017, 106, 1–8. [Google Scholar] [CrossRef] [Scilit]
  33. Weiner, B.J.; Lewis, C.C.; Stanick, C.; Powell, B.J.; Dorsey, C.N.; Clary, A.S.; Boynton, M.H.; Halko, H. Psychometric assessment of three newly developed implementation outcome measures. Implement. Sci. 2017, 12, 108. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  34. Keith, R.E.; Crosson, J.C.; O’Malley, A.S.; Cromp, D.; Taylor, E.F. Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: A rapid-cycle evaluation approach to improving implementation. Implement. Sci. 2017, 12, 15. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  35. Damschroder, L.; Hall, C.; Gillon, L.; Reardon, C.; Kelley, C.; Sparks, J.; Lowery, J. The Consolidated Framework for Implementation Research (CFIR): Progress to date, tools and resources, and plans for the future. Implement. Sci. 2015, 10, A12. [Google Scholar] [CrossRef] [Scilit]
  36. Gale, N.K.; Heath, G.; Cameron, E.; Rashid, S.; Redwood, S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med. Res. Methodol. 2013, 13, 117. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  37. Smith, J.; Firth, J. Qualitative data analysis: The framework approach. Nurse Res. 2011, 18, 52–62. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  38. Safarova, M.S.; Kullo, I.J. Using the electronic health record for genomics research. Curr. Opin. Lipidol. 2020, 31, 85–93. [Google Scholar] [CrossRef] [Scilit]
  39. Sinsky, C.; Colligan, L.; Li, L.; Prgomet, M.; Reynolds, S.; Goeders, L.; Westbrook, J.; Tutty, M.; Blike, G. Allocation of physician time in ambulatory practice: A time and motion study in 4 specialties. Ann. Intern. Med. 2016, 165, 753–760. [Google Scholar] [CrossRef] [Scilit]
  40. Arndt, B.G.; Beasley, J.W.; Watkinson, M.D.; Temte, J.L.; Tuan, W.J.; Sinsky, C.A.; Gilchrist, V.J. Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations. Ann. Fam. Med. 2017, 15, 419–426. [Google Scholar] [CrossRef] [Scilit]
  41. The Office of the National Coordinator for Health Information Technology. Strategy on Reducing Regulatory and Administrative Burden Relating to the Use of Health IT and EHRs Final Report 2020. Available online: https://www.healthit.gov/sites/default/files/page/2020-02/BurdenReport_0.pdf (accessed on 25 February 2020).
  42. Trivedi, M.H.; Daly, E.J.; Kern, J.K.; Grannemann, B.D.; Sunderajan, P.; Claassen, C.A. Barriers to implementation of a computerized decision support system for depression: An observational report on lessons learned in ‘real world’ clinical settings. BMC Med. Inform. Decis. Mak. 2009, 9, 6. [Google Scholar] [CrossRef] [Scilit]
  43. Lyerla, F. Design and implementation of a nursing clinical decision support system to promote guideline adherence. Comput. Inform. Nurs. 2008, 26, 227–233. [Google Scholar] [CrossRef] [Scilit]
  44. North, F.; Fox, S.; Chaudhry, R. Clinician time used for decision making: A best case workflow study using cardiovascular risk assessments and Ask Mayo Expert algorithmic care process models. BMC Med. Inform. Decis. Mak. 2016, 16, 96. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  45. Hummel, J.; Evans, P. EHR Implementation with Minimal Practice Disruption in Primary Care Settings: The Experience of the Washington & Idaho Regional Extension Center. 2012. Available online: https://www.healthit.gov/sites/default/files/ehr-implementation-wirec.pdf (accessed on 20 March 2020).

Article Metrics

Citations

Article Access Statistics

Journal Statistics
Multiple requests from the same IP address are counted as one view.
Adipose-Derived Stem Cells Reduce Lipopolysaccharide-Induced Myelin Degradation and Neuroinflammatory Responses of Glial Cells in Mice
Previous
Use of Biomarkers in Ongoing Research Protocols on Alzheimer’s Disease
Next