Abstract
The implementation of medication adherence interventions is suboptimal. To guide implementation, this study aimed to tailor an existing implementation determinant framework to support the assessment of the implementability of such interventions in a specific context prior to implementation, and to investigate whether experts can assess in advance which determinants are important for implementing medication adherence interventions. In a Delphi study, experts rated determinants based on constructs of the Consolidated Framework for Implementation Research (CFIR) in terms of their importance for implementing medication adherence interventions. Determinants were then prospectively evaluated in four Dutch living labs implementing medication adherence interventions. The results were compared to assess agreement between expert opinion and real-world practice. Of 40 evaluated CFIR determinants, 16 were important in the majority of the living labs. These determinants concerned the inner setting, characteristics and roles of involved individuals, and implementation process domains of the CFIR. After comparing the prospective evaluation with Delphi results, expert opinions matched living lab observations for 18 out of 40 determinants (45%) regarding (un)importance. The CFIR was tailored to primary care medication adherence interventions based on practice observations, offering a potentially helpful framework to assess implementability of these interventions in specific contexts in advance. Determinant frameworks could benefit from incorporating real-world practice data.
1. Introduction
According to the existing literature, approximately 40% of patients on long-term treatment do not adhere to their medication regimen [1]. Effects of nonadherence are substantial as it culminates in worsened health conditions, more hospital admissions and increased healthcare spending [2]. Subsequently, the World Health Organization states that improving adherence to medical therapy for chronic diseases would result in substantial health and economic benefits [3]. Although effective interventions to improve medication adherence have been developed, these interventions are hardly implemented in routine clinical practice. The extent and speed of the implementation of medication adherence interventions seem to mirror other interventions in healthcare, where only 14% of effective interventions are implemented in daily practice, with an average duration of 17 years [4,5]. The slow adoption in clinical practice may be due to a predominant focus on efficacy rather than on implementation. Moreover, implementation challenges often arise as interventions and their implementation contexts are complex, requiring extensive change in individual roles, behaviors, relationships and processes [6]. Likely, not every intervention can be feasibly and sustainably embedded in a certain context.
To increase the implementation success of medication adherence interventions, insight into the factors hindering and facilitating implementation in clinical practice is essential. A framework specifying these factors, also referred to as implementation determinants, could inform the assessment of whether a medication adherence intervention is implementable in a certain healthcare context in advance. Additionally, such a framework could provide guidance in selecting suitable implementation strategies and inform necessary adaptations to either the intervention or the context in which the intervention is to be implemented [7]. Since implementation is generally a time-consuming process, an implementation framework tailored to medication adherence interventions based on observations in clinical practice may contribute to more efficient and successful implementation, ultimately benefiting clinicians, implementers, adopters, policymakers, and eventually patients.
Despite its potential value, the literature describing which determinants hinder and/or facilitate the implementation of medication adherence interventions is limited. The existing literature focuses on factors influencing medication adherence itself rather than on determinants that influence the implementation of interventions aimed at improving medication adherence [8]. Multiple frameworks aimed at guiding implementation do exist; however, these are not specific to medication adherence interventions [9,10]. An example is the Consolidated Framework for Implementation Research (CFIR), which describes and categorizes determinants that can influence implementation [11]. In addition, a study by Pringle and Coley proposes a framework for implementing medication adherence interventions in community pharmacies. However, this work is theory-based and addresses the implementation of healthcare interventions in general, rather than being informed by real-world data from clinical practice on the implementation of medication adherence interventions [12].
For multiple healthcare interventions, knowledge about implementation determinants is primarily derived from expert opinion rather than observations from clinical practice [13,14]. While expert opinions are valuable, it remains unclear to what extent they correspond to determinants that actually influence implementation in real-world clinical practice. Since implementation is context-dependent, arising from the interaction between an intervention and the context in which it is introduced, the importance of specific implementation determinants can be difficult to predict in advance [11]. This underscores the need to examine whether determinants considered important by experts align with those observed in clinical practice and to identify any differences between expert assessments and real-world observations.
To guide the implementation of medication adherence interventions in clinical practice, this study aims to tailor an existing implementation determinant framework, CFIR, to support the assessment of the implementability of primary care medication adherence interventions in a specific context in advance [9]. In addition, this study aimed to investigate whether experts can assess in advance which determinants are important for implementing medication adherence interventions, by comparing their assessment to real-world observations.
2. Materials and Methods
2.1. Study Design
This study consisted of two phases. First, a Delphi study was conducted between July 2020 and February 2021 to identify which determinants experts consider important for successful implementation of medication adherence interventions. The Delphi design was chosen as a consensus method to ensure that individual opinions remained independent and were not initially influenced by other individuals. Reporting of this first phase adheres to the Conducting and Reporting Delphi Studies (CREDES) [15].
In the second phase, implementation determinants were prospectively evaluated in four Dutch local living labs implementing various evidence-based effective medication adherence interventions between December 2020 and July 2022. Living labs are real-world test environments where healthcare providers, researchers, patients and other relevant stakeholders collaborate to create and refine healthcare innovations [16,17]. These living labs are part of “The Medication Adherence Knowledge and Expertise and Implementation Taskforce” (Make-It) program. This taskforce consists of medication adherence and implementation experts who aim to enhance the implementation of medication adherence interventions in clinical practice, by guiding, monitoring and evaluating their implementation within living labs [17]. A tailored framework for assessing the implementability of medication adherence interventions in a specific context was developed based on observed important implementation determinants. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement was applied for reporting this second phase [18]. To assess whether expert opinions align with real-world observations, the list of determinants deemed important by experts (phase 1) was compared to important implementation determinants observed in real-world practice (phase 2).
2.2. Theoretical Framework
To ensure that all implementation determinants were assessed, the CFIR 2022 version was used as a theoretical base for the study [9]. The CFIR is a comprehensive framework based on a comparison of a broad range of theories on effective implementation of interventions in practice, which consolidates determinants that are potentially relevant to a particular intervention and its context. The CFIR describes five major domains, each consisting of several determinants that can influence the effectiveness of implementation, including (1) the intervention, (2) the inner setting, comprising the features of the organization that implements the intervention, (3) the outer setting, comprising the features of the external context in which an organization resides, (4) the characteristics and roles of the individuals involved and (5) the implementation process.
2.3. Phase 1: Delphi Study
2.3.1. Participants
A three-round Delphi study using online questionnaires was conducted with Dutch experts involved in medication adherence support or implementation, including general practitioners, community pharmacists, medical specialists, hospital pharmacists, nurses, patient representatives and researchers. The sample size was set at 30 to account for potential dropouts across rounds, ensuring that 20 experts participated in each Delphi round, in line with literature recommendations on the number of experts required to reach consensus [19,20]. Participants were identified through purposive sampling from the network of the Make-It consortium based on their profession and expertise, ensuring experts with diverse professional backgrounds and experiences were included [17]. Eligible participants were approached by email and asked to participate. Written informed consent was obtained prior to participation.
2.3.2. Data Collection
In the first round of the Delphi study, participants rated all 40 determinants of the five domains of the CFIR on their perceived importance for implementing medication adherence interventions in general. Participants were provided with a short, translated determinant description. A five-point Likert scale (1 = very unimportant to 5 = very important) was used to rate importance. Determinants were excluded for the second round if there was ≥70% agreement on their unimportance (see Figure 1).
Figure 1.
Flowchart describing three-round Delphi study.
In the second round, participants rated the importance of the remaining determinants for anonymized project proposals describing a specific medication adherence intervention planned to be implemented in a particular living lab. This approach provided participants with a concrete example to facilitate a more accurate assessment. Each living lab had submitted these proposals as part of their grant application to ZonMw, the Netherlands Organization for Health Research and Development, which funded both the living labs and the Make-It consortium. The proposals outlined the context and characteristics of the living lab, along with the selected evidence-based medication adherence intervention. Determinants for which 70% or more experts agreed on their importance, for at least three of the four living labs, were classified as important determinants for the third round. If not, the determinants were classified as unimportant for the third round.
In the third round, participants gave their final vote on the importance of determinants, independent of specific living labs or interventions. Determinants classified as important after the second round were further evaluated with the yes/no question ‘Do you agree this determinant is important for the implementation of medication adherence interventions?’. Determinants classified as unimportant in the second round were assessed with the yes/no question ‘Do you agree this determinant is unimportant for the implementation of medication adherence interventions?’. If there was ≥70% disagreement on the unimportance of the determinant or ≥70% agreement on its importance in the third Delphi round, the determinant was included in the list of determinants that experts find important for implementing medication adherence interventions.
2.3.3. Outcomes
The outcome of this first phase is a set of CFIR determinants considered important by experts for implementing medication adherence interventions. Determinants were included in this list if there was ≥70% disagreement in the third Delphi round on unimportance of the determinant or ≥70% agreement on importance.
2.3.4. Data Analysis
Unimportance, as applied in the first Delphi round, was defined as a rating of 1 or 2 on the Likert scale. In the second Delphi round, importance was defined as a rating of 4 or 5 on the Likert scale. Descriptive statistics were used to check whether the level of (dis)agreement (70%) was obtained. Since the CFIR was updated in 2022, while the Delphi study had already been conducted using the 2009 version, the construct mapping document provided by Damschroder et al. was used to translate the determinants to the 2022 version [9]. For determinants that were not explicitly linked in the mapping document, a linkage was established if a match was identified based on determinant descriptions. This linkage was discussed until consensus was reached between MOL and CB, and was confirmed by other team members (BvdB, LvD). The linkage between the 2009 determinants and the CFIR determinants applied in this study is shown in Supplementary Table S1 [9,11].
2.4. Phase 2: Prospective Evaluation of Living Labs Implementing Medication Adherence Interventions
2.4.1. Living Labs
The four Dutch living labs primarily consisted of community pharmacies and related general practices, in which pharmacy team members were the interventionists. The number of living labs was predetermined by the sponsor, ZonMw, from which they received funding for the implementation period [17]. Living labs were selected by a ZonMw committee based on quality and relevance of the project plan. Project plans included the intervention (chosen from a website presenting evidence-based medication adherence interventions; in some living labs, elements of different interventions were combined), contextual information and an implementation plan [21]. Table 1 describes the medication adherence intervention, setting and patient population for each living lab. The implementation process in the living labs was prospectively evaluated during a period of 1–2 years.
Table 1.
Description of living labs.
2.4.2. Data Collection
Before and during implementation, data regarding the implementation process were collected via field notes, progress reports, and documentation of monthly meetings between the project team and Make-It members. In these monthly meetings, the implementation progress and experienced barriers and facilitators were discussed. Additionally, after 1–2 years of implementation, interviews were conducted by Make-It members with the project leaders to evaluate implementation outcomes, discuss barriers and facilitators encountered during the process, and to obtain additional insights from the project leaders’ perspectives. The interview guide was informed by the RE-AIM evaluation framework (Reach, Effectiveness, Adoption, Implementation and Maintenance) to ensure that all relevant implementation outcomes were addressed. A junior researcher prepared the initial draft of the guide, which was refined based on feedback from senior researchers CvdS, LvD, and MV. See Supplementary Table S2 for the interview guide [29].
2.4.3. Outcomes
The outcome of this phase is the proportion of living labs in which each of the CFIR determinants played a role during the implementation. A determinant was considered to influence implementation if it was described at least once as acting as either a barrier or a facilitator for implementation.
2.4.4. Data Analysis
Data analysis was performed with Atlas.ti version 9 after all data had been collected from the living labs. In the first step, documents were analyzed to identify implementation determinants. In the second step, the previously identified determinants were labeled with open codes and deductively analyzed using content analysis guided by CFIR (version 2022) [9]. The first step was conducted by a team of four senior researchers, one junior researcher and one master’s student in Pharmacy. Transcripts of project leader interviews were analyzed independently by pairs of researchers and then compared to discuss discrepancies. After reaching consensus on when a factor should be labeled as a determinant, determinants in the remaining documentations were identified by the junior researcher. In the second step, the determinants identified from one living lab were labeled with CFIR determinants by a junior researcher and reviewed by a senior researcher. Discrepancies were discussed until consensus was reached on how to place determinants within the CFIR. Subsequently, the junior researcher labeled and analyzed determinants from the remaining living labs, discussing any uncertainties with the senior researcher.
2.5. Developing the Tailored CFIR to Medication Adherence Interventions and Assessing Level of Agreement
2.5.1. Outcomes
The main outcome of this study is the CFIR tailored to medication adherence interventions, which consists of all important implementation determinants observed during phase 2. To provide insights into how these determinants influenced the implementation in the living labs, the context of the determinants is described as well. A secondary outcome is the level of agreement between expert opinions (phase 1) and real-world observations (phase 2) on the importance of determinants for implementing medication adherence interventions. Agreement was presented as an overall percentage of determinants for which expert opinions matched real-world observation, and as a detailed figure showing the similarities and differences for each CFIR determinant at a living lab level.
2.5.2. Data Analysis
Important implementation determinants were defined as determinants that were observed to either hinder or facilitate the implementation in at least three of the four living labs. The level of agreement between expert opinions and real-world observations in the living labs was assessed by dividing the number of determinants for which the expert opinions matched the observations (using the previously described definition for important determinants), by the total number of assessed CFIR determinants.
3. Results
3.1. Phase 1: Delphi Study
Out of the 29 approached Dutch medication adherence and/or implementation experts for the Delphi study, 18 consented to participate, with the following professional backgrounds: researcher (n = 4), (specialized) nurse (n = 4), primary care pharmacist (n = 3), outpatient pharmacist (n = 2), general practitioner (n = 2), medical specialist (n = 2), and hospital pharmacist (n = 1). Of these experts, 13 participated in the second round (5–8 experts per living lab), and 16 in the third round.
Important Implementation Determinants According to Expert Opinions
None of the 40 CFIR determinants were rated as unimportant by ≥70% of all the experts for implementing medication adherence interventions in the first round, resulting in all determinants being rated again in the second round (see Table 2). After reading the project proposals of living labs, experts rated the determinants on their importance for implementing the specific medication adherence intervention in the specific living lab. In this second round, 28 determinants were rated as important for at least three of the four living labs. In the third round, experts ultimately reached consensus that 28 determinants were important for implementing medication adherence interventions.
Table 2.
Responses of experts on (un)importance of CFIR determinants for implementing medication adherence interventions [9].
3.2. Phase 2: Prospective Evaluation of Living Labs Implementing Medication Adherence Interventions
Four living labs were prospectively evaluated to identify CFIR determinants influencing the implementation of medication adherence interventions in real-world practice. In addition to the documentation collected during the implementation process, evaluation interviews were conducted with project leaders with an average duration of 1 h. Results showed that 30 of the 40 assessed CFIR determinants influenced implementation in at least one of the four living labs. Among these, 10 determinants influenced implementation in four living labs, 6 determinants influenced implementation in three living labs, 5 in two living labs and 9 in one living lab.
3.3. CFIR Tailored to Medication Adherence Interventions
In phase 2, 16 determinants were identified as having influenced implementation in at least three of the four living labs. These determinants belong to the innovation, inner setting, roles of individuals, characteristics of individuals and the implementation process domains of the CFIR. Based on their influence observed in real-world practice, these determinants were included in the tailored framework (see Table 3). Determinants that were not included in the tailored framework are detailed in Supplementary Table S3.
Table 3.
CFIR tailored to implementation of medication adherence interventions.
The descriptions below provide insight into how determinants included in the tailored framework influenced implementation in the living labs, organized per CFIR domain.
3.3.1. Innovation
The possibility to adapt the intervention was the only determinant of the innovation domain that influenced the implementation. This increased the willingness of the team to implement the intervention and increased the compatibility of the intervention with the existing working routine.
3.3.2. Inner Setting
Multiple determinants of the inner setting were observed to influence the implementation. First, compatibility was a barrier when the intervention required additional actions outside the daily pharmacy routine. However, integrating the intervention into the pharmacy information system facilitated the implementation. Second, other activities, such as medication reviews, sometimes hindered implementation because they were prioritized due to reimbursement. Another barrier was a lack of available resources. Specifically, a lack of time had a substantial influence in pharmacies and general practices, frequently caused by staff shortages and a high workload.
Differing from expert opinions during the Delphi study, structural characteristics, communications, and culture within pharmacies and general practices was important as well. Structural characteristics include the organization of tasks and responsibilities. Specifically, the reallocation of tasks to pharmacy technicians or managers enabled pharmacists to dedicate more time to pharmaceutical care and project management. Additionally, the location of the pharmacy within a healthcare center alongside a general practice facilitated short communication lines. Lastly, regarding the culture within the living lab, pharmacy technicians mentioned feeling a hierarchical barrier when approaching general practitioners.
3.3.3. Individuals—Roles
In contrast to expert opinions, implementation facilitators (i.e., Make-It guidance) and formally appointed implementation leads influenced the implementation in the living labs. Formally appointed leads and high-level leaders (i.e., project leads and pharmacists) were key facilitators, as they had the ability to influence the team’s motivation. Specifically, pharmacists who actively participated in delivering the intervention boosted overall motivation.
3.3.4. Individuals—Characteristics
Capable and motivated team members were observed as key facilitators during implementation, and were also identified as such by experts. Healthcare providers with intrinsic motivation to care for patients were especially crucial, as they continued to deliver the intervention despite a high workload, whereas others neglected the intervention under these circumstances.
3.3.5. Implementation Process
Aligning with expert opinions, all assessed implementation process determinants were observed to be important during implementation. Different methods were employed to engage healthcare providers to perform the intervention. These activities included a regular newsletter with updates, work meetings that solicited input from the entire team, and sharing success stories. Furthermore, all living labs continuously reflected and evaluated the implementation progress. Benchmarking, through comparison of results across pharmacies and/or general practices within a living lab, provided input for meetings in which successes, barriers, and tips were exchanged.
3.4. Level of Agreement Between Expert Opinions and Living Lab Observations
In the first phase of this study, 28 of the 40 determinants were ultimately considered important by the experts. Of these determinants, 11 were indeed observed to be important in at least three of the four living labs in the second phase. Twelve determinants were not considered important by experts in the first phase. Seven out of twelve were not observed to be important during the prospective evaluation. Ultimately, expert opinions matched real-world observations for 45% of the 40 evaluated determinants. The similarities and differences between expert opinions in phase 1 and real-world observations in phase 2 are presented in Figure 2.
Figure 2.
Comparison between expert opinions (phase 1) and living lab observations (phase 2). Bar lengths are indicative of the proportion of living labs for which the determinant was or was not observed. LL = living lab.
4. Discussion
Our results suggest that the characteristics and roles of individuals, the organization itself and actions undertaken to stimulate the implementation process are at least important for implementing medication adherence interventions in the Dutch primary care setting. These determinants were therefore included in the CFIR tailored to medication adherence interventions. In addition, our results indicate that it is difficult for medication adherence and implementation experts to assess which determinants will influence the implementation of medication adherence interventions in advance. It is therefore essential to include information from real-world clinical practice in implementation frameworks and tools assessing implementability.
Upon comparing expert opinions to observations in the living labs regarding the importance of CFIR determinants for implementing medication adherence interventions, expert opinions aligned with observations regarding the importance of motivated and capable individuals. This is in line with the study by Shoemaker et al., in which characteristics of individuals was the most important CFIR domain for implementing medication management interventions [30]. Weir et al. also found the team members’ views and opinions on delivering the intervention to be important for implementing interventions in community pharmacies [31]. Additionally, expert opinions regarding the inner setting aligned with living lab observations on compatibility, relative priority and available resources. Compatibility of the intervention with pharmacy systems, lack of time and staff shortages have also been found to influence implementation in community pharmacies in the study by Weir et al. [31].
However, discrepancies were identified between experts’ opinions and living lab observations regarding the inner setting. Experts seemed to underestimate the importance of relatively softer characteristics of an organization, such as communications, culture and structural characteristics. This underestimation might be explained by the fact that the project proposals, on which the experts based their opinion regarding the importance of determinants for a specific context in the second Delphi round, did not contain such information. Thus, to facilitate the assessment of implementability in advance, project plans should include more information about the communication structure, organization of tasks and responsibilities, and if possible perceived working culture. This need for detailed context information in project plans has also been emphasized by Proctor et al. [32]. Furthermore, the importance of the intervention itself, as rated by experts, was not reflected in living lab observations. This discrepancy may be attributed to the living labs having the freedom to select which medication adherence intervention to implement, with costs already covered by project funding. Additionally, the living labs made adaptations to the interventions to ensure a better fit with their context. Indeed, innovation adaptability was noted to have influenced implementation in the living labs. Moreover, the outer setting, specifically determinants regarding policies and financing, may have had less influence due to the funding that the living labs already received. In situations where an intervention is not freely chosen and no funding is available, the innovation and outer setting domains might have a greater influence on the implementation of medication adherence interventions than observed in the living labs. More generally, the discrepancies between expert opinions and real-world observations could also be the result of the dynamic interaction between an intervention and its context, as determinants arise from the interaction between the two. This interaction can be difficult to predict in advance [11].
The observed discrepancies demonstrate that assessing implementability, thereby defining which determinants are important for implementing medication adherence interventions, can be challenging for experts. Real-world data should therefore play a more prominent role in the tailoring of determinant frameworks, as these are currently often based on expert opinions [13,14]. The experts’ challenges also highlight the added value of a tailored framework based on real-world data, as it provides guidance on which implementation determinants should be considered when evaluating implementability. To further facilitate this assessment, organizations should incorporate the determinants included in the tailored framework in their project plans and explicitly describe how they may act as barriers or facilitators within their setting. Together, this framework and more informative project plans can provide a more comprehensive assessment of the implementability of medication adherence interventions in specific contexts and thereby improve the chance at successful implementation. In addition to assessing implementability, our findings may also increase implementation success by tailoring strategies to overcome potential barriers, using the CFIR-ERIC matching tool [7]. For further research, it would be interesting to assess the applicability of our results in settings outside the Netherlands, as some determinants may be context-specific, especially those related to the outer setting [11].
A strength of this study is that it focuses on implementation determinants for the implementation of medication adherence interventions based on data obtained from real-world clinical practice. Furthermore, a key strength of the prospective evaluation is the fact that diverse medication adherence interventions were being implemented in four different living labs. Identifying a determinant that influenced implementation in the majority of these varied settings suggests that the determinant is important and may be crucial for the implementation of medication adherence interventions in different contexts. Another strength is the structured approach employed in terms of a Delphi design to obtain experts’ opinions on the importance of CFIR determinants for the implementation of medication adherence interventions. An additional strength is the inclusion of experts in medication adherence support or implementation with diverse backgrounds, thereby making the Delphi ratings more credible. However, several limitations should be acknowledged as well. For the second Delphi round, aimed at rating determinants for implementing a certain medication adherence intervention in a certain context, experts were provided with written information only. The project proposals they evaluated included mainly information on living lab characteristics and the chosen adherence intervention, and were less focused on implementation context. As described earlier, this could possibly explain why relatively softer characteristics of the inner setting domain were underestimated. Additionally, the living labs were early adopters, meaning the living labs and their project leaders had previous experience with implementing interventions in daily practice. Observed determinants could differ for organizations with less experience in implementing interventions. Another reason why our results may be less generalizable is that the living labs were located in primary care settings in the Netherlands. In other contexts, different determinants, particularly of the outer setting, may influence implementation. Furthermore, the design of the living labs, with funding and Make-It guidance, has likely influenced which determinants influenced implementation in the living labs, particularly those related to the outer setting. The fact that these determinants are not included in the tailored framework does not necessarily indicate that they would not impact implementation in other contexts. Consequently, while the tailored framework may provide useful guidance regarding intervention characteristics, inner setting, individual characteristics and roles, and implementation processes, implementers outside the Dutch context or without funding and implementation guidance should additionally evaluate which outer setting determinants may influence implementation in their local context. Lastly, the absence of determinants in the data from living labs obtained during prospective evaluation does not imply that these determinants were not relevant. Some facilitators may have been unconsciously included in the project and therefore not recognized as a determinant. Additionally, certain determinants may have had a strong influence but were not included in the tailored framework as they did not meet the criterion of having influenced implementation in at least three of the four living labs.
In conclusion, this study tailored the Consolidated Framework for Implementation Research to the implementation of medication adherence interventions in primary care based on observations from clinical practice. This tailored framework can be used to guide the assessment of medication adherence interventions’ implementability in specific settings in advance and subsequently improve implementation. Differences between real-world practice data and expert opinions suggest that implementation determinant frameworks could benefit from incorporating real-world practice data.
Supplementary Materials
The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/pharmacy14030088/s1, Table S1: description of how the CFIR 2009 determinants (applied for phase 1—Delphi study) were linked to the CFIR determinants applied for phase 2 of our study (prospective evaluation), based on the CFIR 2022 version; Table S2: interview guide used for interviews with living lab project leaders; Table S3: evaluated CFIR constructs not included in the tailored framework.
Author Contributions
Conceptualization, B.J.F.v.d.B., L.v.D. and C.L.B.; methodology, B.J.F.v.d.B., L.v.D. and C.L.B.; formal analysis, M.A.M.O.L., C.H.P.A.v.d.S.-v.G., M.V. and C.L.B.; investigation, M.A.M.O.L. and C.L.B.; visualization, M.A.M.O.L.; writing—original draft preparation, M.A.M.O.L.; writing—review and editing, B.J.F.v.d.B., C.H.P.A.v.d.S.-v.G., M.V., L.v.D. and C.L.B.; supervision, B.J.F.v.d.B., L.v.D. and C.L.B.; funding acquisition, L.v.D. All authors have read and agreed to the published version of the manuscript.
Funding
This research was funded by ZonMw, the Netherlands Organization for Health Research and Development (project number 80-84800-98-24002).
Institutional Review Board Statement
A WMO-waiver was obtained for he Delphi study performed in the first phase of this study, by the Commission Human Related Research of the Radboudumc, file number 2020-6762. Regarding the prospective evaluation of the living labs conducted in the second phase of this study, ethics approval from a review board was not required by the Dutch Medical Research Involving Human Subject Act. Nonetheless, living labs B and D voluntarily sought approval. Living lab B obtained approval from the Institutional Review Board at the division of Pharmacoepidemiology and Clinical Pharmacology of Utrecht University, file number UPF 2009. Living lab D obtained approval from the review board UPPER with file number UPF2106. All participants signed informed consent and took part voluntarily.
Informed Consent Statement
Informed consent was obtained from all subjects involved in the study.
Data Availability Statement
The original contributions presented in this study are included in the article and Supplementary Material. Further inquiries can be directed to the corresponding author.
Acknowledgments
We would like to thank all the healthcare providers and especially the project leaders of the four living labs for their involvement and their willingness to participate, as well as approving the final version of this manuscript. We would also like to thank all experts who participated in the Delphi study. Finally, we would like to acknowledge all current and former members of the Make-It consortium for their contributions to this work.
Conflicts of Interest
The authors declare no conflicts of interest.
References
- Foley, L.; Larkin, J.; Lombard-Vance, R.; Murphy, A.W.; Hynes, L.; Galvin, E.; Molloy, G.J. Prevalence and Predictors of Medication Non-Adherence among People Living with Multimorbidity: A Systematic Review and Meta-Analysis. BMJ Open 2021, 11, e044987. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Achterbosch, M.; Aksoy, N.; Obeng, G.D.; Ameyaw, D.; Ágh, T.; van Boven, J.F.M. Clinical and Economic Consequences of Medication Nonadherence: A Review of Systematic Reviews. Front. Pharmacol. 2025, 16, 1570359. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- World Health Organization. Adherence to Long-Term Therapies: Evidence for Action; World Health Organization: Geneva, Switzerland, 2003. [Google Scholar]
- Balas, E.A.; Boren, S.A. Managing Clinical Knowledge for Health Care Improvement. Yearb. Med. Inform. 2000, 09, 65–70. [Google Scholar] [CrossRef] [Scilit]
- Green, L.W.; Glasgow, R.E.; Atkins, D.; Stange, K. Making Evidence from Research More Relevant, Useful, and Actionable in Policy, Program Planning, and Practice. Am. J. Prev. Med. 2009, 37, S187–S191. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Greenhalgh, T.; Wherton, J.; Papoutsi, C.; Lynch, J.; Hughes, G.; A’Court, C.; Hinder, S.; Fahy, N.; Procter, R.; Shaw, S. Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies. J. Med. Internet Res. 2017, 19, e367. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Powell, B.J.; Waltz, T.J.; Chinman, M.J.; Damschroder, L.J.; Smith, J.L.; Matthieu, M.M.; Proctor, E.K.; Kirchner, J.E. A Refined Compilation of Implementation Strategies: Results from the Expert Recommendations for Implementing Change (ERIC) Project. Implement. Sci. 2015, 10, 21. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Peh, K.Q.E.; Kwan, Y.H.; Goh, H.; Ramchandani, H.; Phang, J.K.; Lim, Z.Y.; Loh, D.H.F.; Østbye, T.; Blalock, D.V.; Yoon, S.; et al. An Adaptable Framework for Factors Contributing to Medication Adherence: Results from a Systematic Review of 102 Conceptual Frameworks. J. Gen. Intern. Med. 2021, 36, 2784–2795. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Damschroder, L.J.; Reardon, C.M.; Widerquist, M.A.O.; Lowery, J. The Updated Consolidated Framework for Implementation Research Based on User Feedback. Implement. Sci. 2022, 17, 75. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Flottorp, S.A.; Oxman, A.D.; Krause, J.; Musila, N.R.; Wensing, M.; Godycki-Cwirko, M.; Baker, R.; Eccles, M.P. A Checklist for Identifying Determinants of Practice: A Systematic Review and Synthesis of Frameworks and Taxonomies of Factors That Prevent or Enable Improvements in Healthcare Professional Practice. Implement. Sci. 2013, 8, 35. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Damschroder, L.J.; Aron, D.C.; Keith, R.E.; Kirsh, S.R.; Alexander, J.A.; Lowery, J.C. Fostering Implementation of Health Services Research Findings into Practice: A Consolidated Framework for Advancing Implementation Science. Implement. Sci. 2009, 4, 50. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Pringle, J.; Coley, K. Improving Medication Adherence: A Framework for Community Pharmacy-Based Interventions. Integr. Pharm. Res. Pract. 2015, 4, 175–183. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Hanssen, D.J.C.; Ras, A.; Rosmalen, J.G.M. Barriers and Facilitators to the Implementation of Interventions for Medically Unexplained Symptoms in Primary Care: A Modified Delphi Study. J. Psychosom. Res. 2021, 143, 110386. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- van den Houdt, S.C.M.; Widdershoven, J.; Kupper, N. Patient and Healthcare Professionals’ Perceived Barriers and Facilitators to the Implementation of Psychosocial Screening in Cardiac Practice: A Delphi Study. Gen. Hosp. Psychiatry 2023, 85, 104–113. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Jünger, S.; Payne, S.A.; Brine, J.; Radbruch, L.; Brearley, S.G. Guidance on Conducting and REporting DElphi Studies (CREDES) in Palliative Care: Recommendations Based on a Methodological Systematic Review. Palliat. Med. 2017, 31, 684–706. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Hossain, M.; Leminen, S.; Westerlund, M. A Systematic Review of Living Lab Literature. J. Clean. Prod. 2019, 213, 976–988. [Google Scholar] [CrossRef] [Scilit]
- Make-It Consortium. Available online: https://makeitconsortium.nl/ (accessed on 19 December 2024).
- von Elm, E.; Altman, D.G.; Egger, M.; Pocock, S.J.; Gøtzsche, P.C.; Vandenbroucke, J.P. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies. J. Clin. Epidemiol. 2008, 61, 344–349. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Shang, Z. Use of Delphi in Health Sciences Research: A Narrative Review. Medicine 2023, 102, e32829. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Manyara, A.M.; Purvis, A.; Ciani, O.; Collins, G.S.; Taylor, R.S. Sample Size in Multistakeholder Delphi Surveys: At What Minimum Sample Size Do Replicability of Results Stabilize? J. Clin. Epidemiol. 2024, 174, 111485. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Interventienet. Available online: https://interventienet.nl/ (accessed on 24 February 2025).
- Kooij, M.J.; Heerdink, E.R.; van Dijk, L.; van Geffen, E.C.G.; Belitser, S.V.; Bouvy, M.L. Effects of Telephone Counseling Intervention by Pharmacists (TelCIP) on Medication Adherence; Results of a Cluster Randomized Trial. Front. Pharmacol. 2016, 7, 269. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Careanimations Watchyourmeds. Available online: https://careanimations.com/watchyourmeds/ (accessed on 7 August 2024).
- Linn, A.J.; van Dijk, L.; van Weert, J.C.M.; Gebeyehu, B.G.; van Bodegraven, A.A.; Smit, E.G. Creating a Synergy Effect: A Cluster Randomized Controlled Trial Testing the Effect of a Tailored Multimedia Intervention on Patient Outcomes. Patient Educ. Couns. 2018, 101, 1419–1426. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Vervloet, M.; Zwikker, H.E.; Linn, A.J.; Koster, E.S.; Gipmans, S.G.H.; van Aarle, M.C.W.; van Dijk, L. The Development and Proof of Principle Test of TRIAGE: A Practical Question Set to Identify and Discuss Medication-Related Problems in Community Pharmacy. Pharmacy 2020, 8, 178. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- van Boven, J.F.M.; Stuurman-Bieze, A.G.G.; Hiddink, E.G.; Postma, M.J.; Vegter, S. Medication Monitoring and Optimization: A Targeted Pharmacist Program for Effective and Cost-Effective Improvement of Chronic Therapy Adherence. J. Manag. Care Pharm. 2014, 20, 786–792. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Ha Dinh, T.T.; Bonner, A.; Clark, R.; Ramsbotham, J.; Hines, S. The Effectiveness of the Teach-Back Method on Adherence and Self-Management in Health Education for People with Chronic Disease: A Systematic Review. JBI Database Syst. Rev. Implement. Rep. 2016, 14, 210–247. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Maghroudi, E.; van Hooijdonk, C.M.J.; van de Bruinhorst, H.; van Dijk, L.; Rademakers, J.; Borgsteede, S.D. The Impact of Textual Elements on the Comprehensibility of Drug Label Instructions (DLIs): A Systematic Review. PLoS ONE 2021, 16, e0250238. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Glasgow, R.E.; Vogt, T.M.; Boles, S.M. Evaluating the Public Health Impact of Health Promotion Interventions: The RE-AIM Framework. Am. J. Public Health 1999, 89, 1322–1327. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Shoemaker, S.J.; Curran, G.M.; Swan, H.; Teeter, B.S.; Thomas, J. Application of the Consolidated Framework for Implementation Research to Community Pharmacy: A Framework for Implementation Research on Pharmacy Services. Res. Soc. Adm. Pharm. 2017, 13, 905–913. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Weir, N.M.; Newham, R.; Dunlop, E.; Bennie, M. Factors Influencing National Implementation of Innovations within Community Pharmacy: A Systematic Review Applying the Consolidated Framework for Implementation Research. Implement. Sci. 2019, 14, 21. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Proctor, E.K.; Powell, B.J.; Baumann, A.A.; Hamilton, A.M.; Santens, R.L. Writing Implementation Research Grant Proposals: Ten Key Ingredients. Implement. Sci. 2012, 7, 96. [Google Scholar] [CrossRef] [Scilit] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.

