Next Article in Journal
An Overview of Transgenic Mouse Models for the Study of Alzheimer’s Disease
Previous Article in Journal
Neural Stem Cell Therapy for Alzheimer’s Disease: A-State-of-the-Art Review
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Fidelity Assessment Tool for a Dementia Carers’ Group-Psychotherapy Intervention

1
Ontario Shores Centre for Mental Health Sciences, Whitby, ON L1N 5S9, Canada
2
Toronto Dementia Research Alliance, Toronto, ON M4N 3M5, Canada
3
Department of Health Sciences, Ontario Tech University, Oshawa, ON L1G 0C5, Canada
4
Independent Researcher, Toronto ON M5G 2G6, Canada
5
Department of Psychiatry, Sinai Health, Toronto, ON M5G 1X5, Canada
6
Department of Psychiatry, University of Toronto, Toronto, ON M5T 1R8, Canada
7
The Cyril & Dorothy, Joel & Jill Reitman Centre for Alzheimer’s Support and Training, Sinai Health, Toronto, ON M5T 3L9, Canada
*
Author to whom correspondence should be addressed.
J. Dement. Alzheimer's Dis. 2025, 2(1), 1; https://doi.org/10.3390/jdad2010001
Submission received: 18 November 2024 / Revised: 20 December 2024 / Accepted: 24 December 2024 / Published: 30 December 2024

Abstract

:
Context: The systematic evaluation of a practitioner’s adherence to and competence in delivering psychotherapeutic interventions can be complex. This study describes the development of a fidelity assessment tool for the Reitman Centre CARERS Program (RCCP), a carer group-psychotherapy intervention with multiple didactic and clinical components. The tool’s value in informing psychotherapy training and best practices for practitioners from diverse professional settings is examined. Methods: The RCCP Fidelity Assessment Tool (RCCP-FAT) was developed following an iterative process of item writing and checking. Seven components of the RCCP—Group Structure, Dementia Education, Problem-Solving Techniques, Therapeutic Simulation, Vertical Cohesion, Horizontal Cohesion, and Overall Global Rating—were assessed, with three to eight items, and a “global score” assigned to each. Fifteen trained raters were paired up to rate 36 RCCP sessions using the RCCP-FAT. Rater agreement, correlation between itemized and global scores, and correlation between global ratings and RCCP participants’ satisfaction were calculated. Results: A total of 1188 RCCP-FAT items were rated by each of the two rater groups. Rater agreement was calculated to be 54.3% (κ = 0.32; 95% CI, 0.02681–0.3729). A positive correlation was found between the itemized and global scoring for four RCCP components evaluated (R = 0.833 to 0.929; p < 0.01). The global score and the participants’ satisfaction with “Simulation” was also positively correlated (R = 0.626, p < 0.01). Conclusions: The study provided evidence for fair rater agreement for all RCCP-FAT assessment items. More importantly, the process of developing the tool systematically crystallized the clinical elements of the RCCP and helped to standardize the training methods by creating a framework for providing feedback to learners that matches the items on the RCCP-FAT. The use of the RCCP-FAT to guide the training and mentoring of incoming group leaders is essential in the scaling and dissemination of a complex training method like the RCCP to ensure fidelity to the original evidence-based intervention.

1. Introduction

The evaluation of psychotherapeutic interventions in the mental health field is complex. Particularly challenging is the evaluation of fidelity, a process undertaken to determine whether the intervention under study was implemented as intended.
When evaluating fidelity, two elements must be considered: adherence, which refers to the degree to which an intervention protocol is followed [1,2,3,4,5], and competence, which refers to how proficiently the intervention is performed [6]. Fidelity measures have been advocated as a means to measure adherence and support service improvement as measures of service quality, since higher fidelity scores are associated with better services outcomes [7].
The development of a reliable fidelity rating tool for psychotherapeutic interventions is particularly difficult when developers attempt to reduce a complex intervention to a simple scale [8], perhaps explaining why there are few fidelity assessment tools found in the existing literature. One psychometric fidelity assessment tool is the Sharing the Patient’s Illness Representations to Increase Trust (SPIRIT) intervention fidelity assessment tool [9], used to evaluate fidelity in the delivery of a psycho-educational program for patients with end-stage renal disease and their surrogate decision makers. Although this study exhibited an acceptable inter-rater reliability, only one interventionist was evaluated in the trial. Another psychometric fidelity assessment tool, developed by the RAND Corporation [6], was used to assess the fidelity of the Building Recovery by Improving Goals, Habits, and Thoughts (BRIGHT-2) intervention. In this study, the fidelity of five addiction counsellors who were trained to deliver group cognitive behavioural therapy for depression was investigated.
Creating a fidelity scale that captures the complexity of a single provider model, as previously described in the literature, is challenging. The challenge may be compounded by a program-based intervention composed of many parts, often delivered by multiple practitioners [10] and disseminated by training practitioners in diverse professional settings. The effectiveness of a psychotherapeutic intervention relies in large part on the non-specific factors of psychotherapy [11] and the dynamic and individualized nature of the therapeutic relationship [9], both of which are difficult to capture and measure in a fidelity scale. Further, measurement of fidelity may be confounded by variables such as client characteristics and severity of symptoms [12]. Once developed, the verification of the scale’s reliability and the training of raters to use it are resource-intensive [9]. More importantly, even when validated, a tool may not produce useful information. For instance, data collected using the tool may indicate that fidelity to the method is poor, leading to the conclusion that the person delivering the intervention lacks proficiency. However, it is important to recognize that a highly skilled practitioner may appropriately stray from the model, compromising fidelity in order to meet the specific needs of the individual client and to maintain the therapeutic relationship [9,12].
Despite the challenges posed by fidelity measurement, it remains a necessary part of the delivery and evaluation of psychotherapeutic interventions. Knowing whether the intervention is delivered proficiently and as intended is necessary in the determination of treatment efficacy (or the lack thereof), the exact mechanism of any changes that it produces [1,12], and whether the intervention’s success or failure is attributable to the method or the delivery [9,13,14]. The evaluation of a complex psychotherapeutic intervention that does not consider fidelity risks measuring the effects of the program as delivered, rather than as designed, can lead to erroneous conclusions regarding the efficacy of the intervention [10,15,16,17].
The Reitman Centre CARERS—Coaching, Advocacy, Respite, Education, Relationship, Simulation—Program (RCCP) is a group psychotherapeutic intervention developed for carers looking after family members with dementia, with demonstrated efficacy in improving caregiving competence, coping capacity, and mental well-being [18]. It combines therapeutic principles with a targeted approach to education and skills training, along with formal problem-solving techniques (PST) adapted for the needs of family carers [18].
Four to six family carers at a time participate in the 10-week group intervention. The RCCP is delivered by a specially trained group leader and includes a novel therapeutic use of simulation—a validated experiential learning tool used in the education of health and other professionals. A specially trained standardized patient is also present to interact with carers to provide them with the opportunity to practice new skills, e.g., carrying out difficult conversations with the care recipient, while coached by the group leader. Upon completion of the 10-week intervention, participants are asked to complete an anonymous satisfaction survey comprising 20 attitudinal statements relating to the key components of the program: tailored dementia and carer education (5 statements), problem-solving therapy (3 statements); therapeutic simulation (4 statements); group structure (3 statements); general satisfaction (5 statements). Participants were asked to respond to each statement using a 5-item Likert scale (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree).
As the clinical efficacy of the RCCP became known, demands for training in the RCCP methods grew from health professionals, leading to the creation of the Reitman Centre CARERS Training Program for Professionals. Health care professionals from community and major dementia care partners have requested and received the formal training necessary to become RCCP group leaders. Initially delivered by mental health clinicians in the hospital setting, the RCCP has been adapted for broader dissemination in different geographical and cultural settings, locally, nationally, and internationally.
In 2017, the Ontario Ministry of Health in Ontario, Canada, approved and funded a province-wide initiative to address the needs of family care partners in the community providing care to people with dementia at home. This program, called the Enhancing Care for Ontario Care Partners Program (EC program), scaled CARERS to address the needs of both urban and rural family care partners in Canada’s largest province of about 16 million (2024) people. The lead organization, Sinai Health, an academic health sciences centre in Toronto, established and maintains a formal province-wide network of Alzheimer Society and hospital partner agencies to implement CARERS in all the health regions of the province. Mental health practitioners in each of 12 sites were trained to deliver the program. A training program located at Sinai Health is an embedded, funded component of the EC program, designed to maintain the network of practitioners by training new practitioners in response to staff changes and attrition. Maintaining fidelity to the evidence-based, effective model [18] is an ongoing key component of both the program and the training program.
Dissemination of the RCCP and ongoing examination of its effectiveness, as well as of the methods used to train health professionals in its delivery, require a dependable method of assessing their implementation. As no standardized fidelity rating tool existed, the RCCP Fidelity Assessment Tool (RCCP-FAT) was developed to not only monitor and evaluate adherence to the RCCP methods and principles but also the competence of the professionals trained to deliver it. This paper (1) describes the development of the RCCP-FAT; (2) presents exploratory data on the usability and inter-rater reliability; (3) discusses the value and challenges inherent in creating a tool used to measure a complex group psychotherapy intervention.

2. Methods

2.1. Design and Development of the RCCP-FAT

The development of an assessment tool employs multiple research methodologies and designs [6] and two major phases: (1) tool design and development, and (2) tool evaluation. In phase 1, the existing validated tools [6,19,20,21] were identified from the literature and adapted by a panel including Reitman Centre mental health clinicians and external clinical experts. An iterative process was employed in which the panel convened four times to work collectively on item writing and item checking [22]. The process also included considerable deliberation regarding the difference between adherence and competence and the best way to capture each in one tool. External expert review ensured the face validity of the tool, that is, that it contains rating criteria that properly represent the group-psychotherapy constructs they are intended to measure.
The final rendition of the RCCP-FAT includes a preamble that describes the purpose of the tool, each of its components, and instructions for use. The tool assesses trained group leaders’ fidelity in the following seven components of the RCCP: Group Structure, Dementia Education, Problem-Solving Techniques, Therapeutic Coaching of Simulation, Vertical and Horizontal Group Cohesion, and Global Rating of Fidelity. There are three to eight items to which fidelity scores are assigned for each component. Each item is defined, and detailed descriptors are provided to indicate what constitutes a specific score. Scores range from 1 to 5 for each specific item, i.e., 1—“Unsatisfactory”, 2—“Needs Improvement”, 3—“Satisfactory”, 4—“Very Good”, and 5—“Excellent”. A “global score” is also assigned for each of the seven components, allowing the raters to evaluate the overall qualities of the competence of the group leader and how well a specific component was delivered. The contents of the RCCP-FAT and the theoretical foundation of the seven components of the RCCP model are summarized in Table 1. An example of the scoring descriptors and criteria for the RCCP component, Problem-Solving Techniques, is found in Figure 1.

2.2. Development of Rater Training Materials

Two sets of video clips of RCCP sessions were produced to train raters in the use of the RCCP-FAT. Clinically-based, semi–scripted scenarios, written by a media and simulation expert (L.J.N.) and vetted by clinical experts (J.S., and V.W.), were used in the production of the demonstration clips. Simulated scenarios allow for the standardization of scoring and the interpretation of item descriptors on the RCCP-FAT; these are also practical teaching tools.
Both sets of demonstration clips showed a group leader directing a simulated RCCP group. Simulated patients with experience in RCCP methods were trained to portray group participants. Attention was paid to ensuring that the clips showed participants enacting a variety of behaviours and affects that approximate an authentic group experience and effectively demonstrate the items on the RCCP-FAT. The first set of clips provided raters with examples of experienced RCCP group leaders demonstrating the methods and techniques of the intervention, as intended. These clips were used to train raters in the use of the tool and to ensure their understanding of item descriptors and their applications. The second set of clips showed the group leader demonstrating the RCCP methods with different degrees of proficiency and allowed learners to understand the nuances that would distinguish an “Excellent” rating from a “Satisfactory” rating, for example, in regards to a specific item. All demonstration video clips were assessed and rated by an expert panel (J.S., V.W., L.J.N.) using the RCCP-FAT prior to their use in training to ensure usability and to provide baseline scores for comparison with trained raters’ scoring.

2.3. Recruitment and Training Volunteer Raters and Research Ethics Clearance

Fifteen volunteers with relevant backgrounds—experience in education, in the practice of psychotherapy, and/or in facilitating group interventions—were recruited from various postgraduate programs in Toronto, Ontario, and trained (Figure 2). The raters were first oriented to the foundational principles and methods of RCCP by completing a self-directed e-learning program. They then participated in training—8 h in total—which included the following:
  • Discussion and questions regarding the RCCP and self-directed e-learning modules;
  • Systematic review of components of the RCCP-FAT;
  • Practice in use of the RCCP-FAT using standardized video clips;
  • Discussion of scored items, item consensus, and scoring challenges;
  • Practice in use of the RCCP-FAT using standardized video clips; and
  • Discussion of scored items, item consensus (or lack thereof), and rationale for scoring differences.
This research was conducted in accordance with the Declaration of Helsinki (1954) and under the direction of the Sinai Health System Research Ethics Board. All trained volunteer fidelity raters, RCCP group leaders, and carers participating in the RCCP provided informed consent prior to engaging in the study.

2.4. Data Collection

Group leaders from the Reitman Centre, experienced in the delivery of RCCP, were notified of the study and informed that two trained volunteer raters would observe three sessions of an RCCP group they led. Observations of these specific sessions allowed for key components of the RCCP, as described in Table 1, to be assessed. Twelve cycles of RCCP groups, facilitated by different group leaders, were observed and assessed by paired raters using the RCCP-FAT. As paired raters assessed three sessions of each of the 12 cycles, a total of 36 sessions were assessed. This number has been reported in the literature regarding group intervention competence and adherence measurement as the number needed to compute inter-rater reliability for fidelity assessment tools [21]. Given the time-intensive nature of the study, group leaders were rated by different pairs of raters over time [25]. Written user feedback on the RCCP-FAT, provided by the volunteer raters, was also reviewed.

2.5. Data Analysis

2.5.1. Rater Agreement of RCCP-FAT

For the purpose of data analysis, raters were randomly assigned to two groups: Rater Group A and Rater Group B. Raters within each rater group were then randomly paired for each RCCP session observed to calculate the inter-rater reliability of the RCCP-FAT, which is calculated by examining the proportion of observed agreement between raters for each tool item (po = # of observed agreement ÷ total # of observations). Since agreement can be expected by chance alone, the weighted kappa statistic as a measure of inter–rater reliability was examined, with κ values between 0.01 and 0.20 indicating slight agreement, values between 0.21 and 0.40 indicating fair agreement, and those above 0.41 considered to show moderate to substantial agreement [26].

2.5.2. Correlation Between Itemized and Global Scores for Each Fidelity Assessment Component

As previously described, the RCCP-FAT evaluates seven components of the RCCP. Raters also gave a global score for each of the seven components measured. The global scores are intended to allow raters to evaluate the overall qualities of competence of the group leader, as well as how well a specific component was delivered. To investigate the utility of global scoring, a correlational analysis was conducted between the average fidelity scores of all items in each component and the global score for the respective component.

2.5.3. Correlation Between Fidelity Ratings and RCCP Participants’ Satisfaction

The participant satisfaction survey data were collated and analysed to determine whether the carers’ perceptions of RCCP’s impact correlated with the overall treatment fidelity scores of the trained group leaders, as assessed by the RCCP-FAT. Correlation coefficients were calculated for the global scores of four sections in the RCCP-FAT—Dementia Education, Problem-Solving Therapy, Therapeutic Simulation (the three main methods used in the RCCP), and Overall Global Score—and the average carers’ satisfaction survey scores related to these program components. For example, the global score given by raters for the “Dementia Education” section in the RCCP-FAT was correlated with the average satisfaction score of the Dementia Education-related statement on the satisfaction survey completed by the carers, i.e., “The program improved my understanding of the behavioural symptoms associated with dementia”.

3. Results

Descriptive statistics for the observations and ratings using the RCCP-FAT were calculated. In all, 11 trained raters participated in the study and rated 12 RCCP group cycles. In each group cycle, 3 out of the 10 sessions of the RCCP group were observed and rated, for a total of 36 observed sessions. There were 1188 possible items to be rated by each of the two rater groups over the 36 observed sessions. Rater Group A and Rater Group B rated 67.7% and 65.5% of all possible items, respectively. Some items were not rated in each observed session, as not all sections of the tool pertained to every group session. For example, items under Problem-Solving Therapy and Therapeutic Simulation were not rated during Session 1 of the RCCP, when these methods are not employed. The mean scores given by Rater Groups A and B were 4.12 and 4.09, respectively, on a scale of 1 to 5. A frequency count revealed that in both rater groups, a high cumulative percentage of the scores 3, 4, and 5 was awarded (97.1% and 96.6% for Rater Groups A and B, respectively), with score of 5 being awarded notably frequently (38.1% and 40.0% for Rater Groups A and B, respectively).
In terms of rater agreement, there was a 54.3% agreement regarding the overall scoring of the RCCP groups, with a weighted kappa of 0.32 (95% CI, 0.02681–0.3729), which shows fair agreement. The kappa statistic by RCCP-FAT component (Table 2) revealed fair to moderate agreement for all components, except for Horizontal Cohesion, which displayed a slight agreement.
The correlation of the average fidelity scores of all items in each of the seven sections in the RCCP-FAT and the global score of the corresponding section was calculated and is shown in Table 3. There was a positive and statistically significant correlation between these for all components evaluated (R between 0.833 and 0.929; p < 0.01).
The correlation calculation between the global scores of four sections in the RCCP-FAT (Dementia Education, Problem-Solving Therapy, Therapeutic Simulation, and Overall Global Score) and the average carers’ satisfaction survey scores related to these program components can be found in Table 3. The analysis indicated a significant, positive correlation between the global simulation score and carers’ satisfaction with the simulation, with r = 0.626; p < 0.01.

4. Discussion

The RCCP-FAT was developed to monitor and evaluate the intervention integrity of the RCCP, a multi-method intervention for informal dementia carers. The process of developing the tool systematically crystallized the clinical elements of RCCP and helped to standardize training methods by creating a framework for providing feedback to learners that match the items on the RCCP-FAT. It also helped clarify the clinical methods of the RCCP and highlighted its complexity.
The need for a standardized approach to training methods that included a reliable method to measure treatment fidelity was obvious early on in the RCCP training process. From our training experience, a lack of consistency in therapeutic skills was noted among different groups of learners, which included mental health clinicians with a variety of backgrounds and experience. Learners whose current practice did not include therapeutics may exhibit difficulty integrating new methods therapeutically. The development of the RCCP-FAT had an important impact on the standardization of the training methods for incoming RCCP group leaders in that training objectives, goals, and learning activities are now closely linked to RCCP-FAT components and items. Specifically, the RCCP-FAT enhanced the development and inclusion of RCCP-group-leader training materials that could address a broad range of psychotherapeutic skills and experience in learners [14]. The assessment of therapeutic competency within the RCCP-FAT ensures that non-specific group therapy factors—warmth, genuineness, empathy, and cohesion—are also being systematically taught to the new group leaders. Learners are observed during training and receive immediate expert feedback designed to match the items on the RCCP-FAT. This use of the RCCP-FAT to guide training and later on, the mentoring of new group leaders, is essential in ensuring that the evidence-based methods of the RCCP are retained during the scaling and dissemination of the RCCP as new group leaders are trained to deliver the RCCP nationally and internationally.
The development of the RCCP-FAT followed the principle that a psychometric fidelity scale should measure an “intangible collection of abstract concepts and principles” [27] such as warmth/genuineness, empathy, and therapeutic alliance. User feedback from our fidelity raters indicated that it is especially challenging to assess the adherence and competency of a psychotherapeutic intervention due to its dynamic and commonly individualized nature, involving both the clinician and the client [4,9,28]. Competency also includes the ability to flexibly adhere to a given intervention [29], and this concept is similarly challenging to capture and measure using a standardized tool. Thus, as demonstrated by a study by Yeates et al. [30], it is common for more experienced group leaders to make normative rather than criterion-based judgments. In our study, the expectation of flexibility is built into the RCCP methods in that each of the systematized treatment elements—Dementia Education, Problem-Solving Therapy, and Therapeutic Simulation—are designed to flexibly meet the specific learning needs of individual carers. Thus, a clinical or therapeutic decision to skip over a certain element during the group session may be misinterpreted by the volunteer raters as a missed therapeutic opportunity, thus impacting the fidelity scoring.
Rating scales are used with the understanding that “mastery of the parts (i.e., discrete skills on a checklist) does not indicate competency of the whole” [31], and global ratings may capture a more accurate picture of expertise than do binary checklists [32]. In addition, global ratings also allow for the assessment of integrated skills [33]. In this study, the global rating for each component was found to be highly correlated with the average scores of all items within the same component. Data analysis also indicated a significant, positive correlation between the global simulation score and the carers’ satisfaction with the simulation. This suggested that for the simulation component of the RCCP, as fidelity to the methods increased, the satisfaction of the carers also improved, a common phenomenon observed in fidelity measurement [1].

5. Limitations

The challenges in conducting a fidelity measurement study, as described in the literature, were encountered, and we recognize that fair to moderate agreement is a potential limitation or reliability. This was partially due to the use of volunteer raters with inconsistent availability, which may have impacted rater agreement, despite steps taken to ensure standardization in training and scoring. Additionally, interrater agreement can be problematic in the evaluation of psychotherapy interventions. Despite the RCCP researchers’ best efforts to train the raters, there are important differences in the clinical delivery of the RCCP method because it is designed to be adapted to the specific needs of the participants. Hence, as is true of all psychotherapy, elements of the treatment may be modified by some therapists based on their assessment of the needs of patients at a given moment. Trained raters may not have recognized the clinical reasons for the omissions or reemphasizing of certain elements and may have interpreted the alterations as deviations from the method, even though the therapist was adhering to the RCCP’s need for flexibility in the delivery of the interventions during a given group. The rating was also limited to three sessions from each 10-week cycle. This method was chosen primarily to keep the necessary commitment of the volunteer raters to a minimum, while allowing the sufficient assessment of the main methods used in the RCCP. Rating all 10 sessions may have improved the reliability of the findings. Future studies with a larger sample size would allow for statistically sound, reliable conclusions.
Although detailed and specific descriptors and definitions were used to guide the scoring of each component and the individual items within the components, certain non-specific group therapy factors, such as warmth/genuineness, empathy, and therapeutic alliance, remain difficult to measure because it is hard to decontextualize them from the therapeutic process.

6. Conclusions

The RCCP-FAT demonstrates the value of a systematic fidelity tool to inform psychotherapy training and best practices. It functions as a mentoring guide, and as such, has shifted the approach to the design of other educational materials, thereby informing all the health professional training activities delivered according to the Reitman Centre CARERS Program model. It has also improved the clinical integrity of the CARERS Program delivered at the Reitman Centre and its satellite sites, as it continues to provide a common language for clinical discussion of the RCCP methods and for training and mentoring other health professionals.

Author Contributions

Conceptualization, M.C., L.J.N. and J.S.; Methodology, M.C., L.J.N. and A.L.; Validation, M.C.; Formal analysis, A.L. and M.C.; Resources, J.S.; Writing original draft, M.C., L.J.N. and V.W.; Writing review and editing, J.S.; Supervision, J.S.; Project administration, M.C. and J.S.; Funding acquisition, M.C. and J.S. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the Continuing Education Development Fund, Faculty of Medicine, University of Toronto, Toronto M5G 1V7 Canada, No. 8470775.

Institutional Review Board Statement

The study was approved by the Sinai Health System Research Ethics Board, approved 18 June 2014; approval #14-0047E.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The datasets presented in this article are not readily available because of technical limitations.

Acknowledgments

The authors would like to acknowledge Rhonda Feldman, Gita Lakhanpal, and Sarah Gillespie at the Reitman Centre for their involvement in the development of the fidelity assessment tool, as well as Molyn Leszcz, Paula Ravitz, and Robert Maunder for their expert reading of the tool.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Bellg, A.J.; Borrelli, B.; Resnick, B.; Hecht, J.; Minicucci, D.S.; Ory, M.; Ogedegbe, G.; Orwig, D.; Ernst, D.; Czajkowski, S.; et al. Enhancing Treatment Fidelity in Health Behaviour Change Studies: Best Practices and Recommendations for the NIH Behaviour Change Consortium. Health Psychol. 2004, 23, 443–451. [Google Scholar] [CrossRef] [PubMed]
  2. Dumas, J.E.; Lynch, A.M.; Laughlin, J.E.; Phillips, S.E.; Prinz, R.J. Promoting Intervention Fidelity. Conceptual Issues, Methods, and Preliminary Results from the EARLY ALLIANCE Prevention Trial. Am. J. Prev. Med. 2001, 20, 38–47. [Google Scholar] [CrossRef] [PubMed]
  3. Perepletchikova, F.; Kazdin, A.E. Treatment Integrity and Therapeutic Change: Issues and Research Recommendations. Clin. Psychol. 2005, 12, 365–383. [Google Scholar] [CrossRef]
  4. Santacroce, S.J.; Maccarelli, L.M.; Grey, M. Intervention Fidelity. Clin. Nurs. Res. 2004, 53, 63–66. [Google Scholar] [CrossRef] [PubMed]
  5. Stein, K.F.; Sargent, J.T.; Rafaels, N. Intervention Research: Establishing Fidelity of the Independent Variable in Nursing Clinical Trials. Adv. Nurs. Res. 2007, 56, 54–62. [Google Scholar] [CrossRef]
  6. Hepner, K.A.; Howard, S.; Paddock, S.M.; Hunter, S.B.; Osilla, K.C.; Watkins, K.E. A Fidelity Coding Guide for a Group Cognitive Behavioral Therapy for Depression; RAND Corporation: Santa Monica, CA, USA, 2011; Available online: https://www.rand.org/content/dam/rand/pubs/technical_reports/2011/RAND_TR980.pdf (accessed on 12 January 2017).
  7. Breitenstein, S.L.; Robbins, L.J.; Muennich Cowell, J.M. Attention to Fidelity: Why Is It Important. J. Sch. Nurs. 2012, 28, 407–408. [Google Scholar] [CrossRef]
  8. Lenaway, D.; Halverson, P.; Sotnikov, S.; Tilson, H.; Corso, L.; Millington, W. Public Health Systems Research: Setting a National Agenda. Am. J. Public Health 2006, 96, 410–413. [Google Scholar] [CrossRef]
  9. Song, M.K.; Happ, M.B.; Sandelowski, M. Development of a Tool to Assess Fidelity to a Psycho-Educational Intervention. J. Adv. Nurs. 2010, 66, 673–682. [Google Scholar] [CrossRef] [PubMed]
  10. Teague, G.B.; Mueser, K.T.; Rapp, C.A. Advances in Fidelity Measurement for Mental Health Services Research. Psychiat. Serv. 2012, 63, 765–771. [Google Scholar] [CrossRef]
  11. Ardito, R.B.; Rabellino, D. Therapeutic Alliance and Outcome of Psychotherapy: Historical Excursus, Measurements, and Prospects for Research. Front. Psychol. 2011, 18, 270. [Google Scholar] [CrossRef]
  12. Hogue, A.; Henderson, C.E.; Dauber, S.; Barajas, P.C.; Fried, A.; Liddle, H.A. Treatment Adherence, Competence, and Outcome in Individual and Family Therapy for Adolescent Behavior Problems. J. Consult. Clin. Psychol. 2008, 76, 544–555. [Google Scholar] [CrossRef]
  13. Sheridan, S.M.; Swanger-Gagne, M.; Welch, G.W.; Kwon, K.; Garbacz, S.A. Fidelity Measurement in Consultation: Psychometric Issues and Preliminary Examination. Sch. Psych. Rev. 2009, 38, 476–495. [Google Scholar]
  14. Mars, T.; Ellard, D.; Carnes, D.; Homer, K.; Underwood, M.; Taylor, S.J.C. Fidelity in Complex Behaviour Change Interventions: A Standardized Approach to Evaluate Intervention Integrity. BMJ Open 2013, 11, e003555. [Google Scholar] [CrossRef] [PubMed]
  15. Schoenwald, S.K.; Garland, A.F.; Chapman, J.E.; Frazier, S.L.; Sheidow, A.J.; Southam-Gerow, M.A. Toward the Effective and Efficient Measurement of Implementation Fidelity. Adm. Policy Ment. Health 2011, 38, 32–43. [Google Scholar] [CrossRef]
  16. Dobson, D.; Cook, T.J. Avoiding type III Error in Program Evaluation: Results from a Field Experiment. Eval. Program. Plann. 1980, 3, 269–276. [Google Scholar] [CrossRef]
  17. Margison, F.R.; McGrath, G.; Barkham, M.; Clark, J.M.; Audin, K.; Connell, J.; Evans, C. Measurement and Psychotherapy. Evidence-based Practice and Practice-Based Evidence. Br. J. Psychiatry 2000, 177, 123–130. [Google Scholar] [CrossRef] [PubMed]
  18. Sadavoy, J.; Sajedinejad, S.; Chiu, M. A quasi-experimental study of the effectiveness of the Reitman Centre CARERS group intervention on family caregivers of persons with dementia. Int. J. Geriatr. Psychiatry 2021, 36, 811–821. [Google Scholar]
  19. Lu, W.; Yanos, P.T.; Gottlieb, J.D.; Marcello Duva, S.M.; Silverstein, S.M.; Xie, H.; Rosenberg, S.D.; Mueser, K.T. Using Fidelity Assessments to Train Frontline Clinicians in the Delivery of Cognitive-Behavioral Therapy for PTSD in Persons with Serious Mental Illness. Psychiatr. Serv. 2012, 63, 785–792. [Google Scholar] [CrossRef] [PubMed]
  20. North Carolina Evidence Based Practices Centre. Family Psychoeducation Fidelity Scale. Updated 2002. Available online: https://www.ncebpcenter.org/assets/FPE_Protocol.pdf (accessed on 17 November 2024).
  21. Hepner, K.A.; Hunter, S.B.; Paddock, S.M.; Zhou, A.; Watkins, K.E. Training Addiction Counselors to Implement CBT for Depression. Adm. Policy Ment. Health 2011, 38, 313–323. [Google Scholar] [CrossRef]
  22. Richey, R.; Klein, J. Developmental Research Methods: Creating Knowledge from Instructional Design and Development Practice. J. Comput. High. Educ. 2005, 16, 23–38. [Google Scholar] [CrossRef]
  23. Imel, S. Using Adult Learning Principles in Adult Basic and Literacy Education. Updated 1998. Available online: https://sswm.info/sites/default/files/reference_attachments/SUSAN%201998%20Using%20Adult%20Learning%20Principles%20in%20Adult%20Basic%20and%20Literacy%20Education%20Columbus.pdf (accessed on 17 November 2024).
  24. Yalom, I.D.; Leszcz, M. The Theory and Practice of Group Psychotherapy; Basic Books: New York, NY, USA, 2005. [Google Scholar]
  25. Hallgren, K.A. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor. Quant. Methods Psychol. 2012, 8, 23–34. [Google Scholar] [CrossRef]
  26. Viera, A.J.; Garrett, J.M. Understanding Interobserver Agreement: The Kappa Statistic. Fam. Med. 2005, 37, 360–363. [Google Scholar] [PubMed]
  27. Cook, D.A.; Beckman, T.J. Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application. Am. J. Med. 2006, 119, 166.e7–166.e16. [Google Scholar] [CrossRef]
  28. Carroll, K.M.; Nich, C.; Sifry, R.L.; Nuro, K.F.; Frankforter, T.L.; Ball, S.A.; Fenton, L.; Rounsaville, B.J. A General System for Evaluating Therapist Adherence and Competence in Psychotherapy Research in the Addictions. Drug Alcohol Depend. 2000, 57, 225–238. [Google Scholar] [CrossRef] [PubMed]
  29. Breitenstein, S.M.; Gross, D.; Gravey, C.; Hill, C.; Fogg, L.; Resnick, B. Implementation Fidelity in Community-Based Interventions. Res Nurs. Health 2010, 33, 164–173. [Google Scholar] [CrossRef]
  30. Yeates, P.; O’Neill, P.; Mann, K.; Eva, K. ‘You’re Certainly Relatively Competent’: Assessor Biases due to Recent Experiences. Med. Educ. 2013, 47, 910–922. [Google Scholar] [CrossRef] [PubMed]
  31. Ginsburg, L.R.; Tregunno, D.; Norton, P.G.; Smee, S.; de Vries, I.; Sebok, S.S.; VanDenKerkhof, E.G.; Luctkar-Flude, M.; Medves, J. Development and Testing of an Objective Structured Clinical Exam (OSCE) to Assess Socio-Cultural Dimensions of Patient Safety Competency. BMJ Qual. Saf. 2015, 24, 188–194. [Google Scholar] [CrossRef]
  32. Kim, J.; Neilipovitz, D.; Cardinal, P.; Chiu, M. A Comparison of Global Rating Scale and Checklist Scores in the Validation of an Evaluation Tool to Assess Performance in the Resuscitation of Critically Ill Patients During Simulated Emergencies (Abbreviated as “CRM Simulator Study IB”). Simul. Healthc. 2009, 4, 6–16. [Google Scholar] [CrossRef]
  33. Schuwirth, L.W.; van der Vleuten, C.P.M. Programmatic Assessment: From Assessment of Learning to Assessment for Learning. Med. Teach. 2011, 33, 478–485. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Scoring descriptors and criteria for “Problem-Solving Techniques”, a key component of the RCCP; GL = Group Leader.
Figure 1. Scoring descriptors and criteria for “Problem-Solving Techniques”, a key component of the RCCP; GL = Group Leader.
Jdad 02 00001 g001
Figure 2. The study design.
Figure 2. The study design.
Jdad 02 00001 g002
Table 1. The theoretical foundation and description of the seven components of the Reitman Centre CARERS Program model.
Table 1. The theoretical foundation and description of the seven components of the Reitman Centre CARERS Program model.
RCCP ComponentTheoretical FoundationComponent Description
Structured Group Sessions
3 items
Focused Group Therapy
Adult Education
The developed structure of the RCCP model must be followed during a session, and a clear agenda, goals, and expectations must be established for each session.
Dementia Education
3 items
Adult Education [23]Evaluates how effectively group leaders provide basic understanding of dementia based on specific expressed experiences, concerns, and misconceptions.
Problem-Solving Techniques (PST)
5 items
Adapted from CBT and PST
Adult Education [23]
A structured and systematic multi-step technique to identify specific problems and develop targeted strategies to manage them. Participants choose which problem is most pressing to them, and the process is recorded visually.
Therapeutic Simulation
8 items
Simulation Methodology
Adult Education [23]
Re-enactments of communication, relational, or behavioural challenges faced by family care givers for skills building. Includes experiential learning and immediate and specific feedback.
Vertical Cohesion
3 items
Group Therapy [24]Quality of cohesion between the group leader and each carer and between the group leader and the group (includes warmth, genuineness, empathy, and engagement).
Horizontal Cohesion
7 items
Group TherapyAttraction of the group to its members, analogous to therapeutic alliance between the group leader and the group members.
Overall Global Rating
1 item
All (Group Therapy, Adult Education, CBT, and PST)Overall assessment of performance across all major RCCP components.
Table 2. Overall kappa statistic by tool category.
Table 2. Overall kappa statistic by tool category.
Tool ConstructValue95% Confidence
Interval
Asymp. Std. Error Approx. Sig.
Structured Group
Sessions
Measure of Agreement (Kappa)0.3520.2042–0.4990.0770.000
N of Valid Cases102
Dementia EducationMeasure of Agreement (Kappa)0.3150.1687–0.46170.0720.000
N of Valid Cases97
Problem-Solving
Therapy
Measure of Agreement (Kappa)0.3360.1663–0.50650.0870.000
N of Valid Cases66
SimulationMeasure of Agreement (Kappa)0.3700.2157–0.52470.0780.000
N of Valid Cases88
Vertical CohesionMeasure of Agreement (Kappa)0.4530.3241–0.58110.0640.000
N of Valid Cases127
Horizontal CohesionMeasure of Agreement (Kappa)0.1950.1049–0.28530.0460.000
N of Valid Cases258
Table 3. Correlation statistics.
Table 3. Correlation statistics.
NCorrelation CoefficientSig. (2-Tailed)
Correlation between itemized and global scoring of fidelity component measures:
Group Structure 210.833 *0.000
Dementia Education 220.919 *0.000
Problem-Solving Therapy 170.924 *0.000
Therapeutic Simulation 180.929 *0.000
Vertical Cohesion 600.923 *0.000
Horizontal Cohesion 590.892 *0.000
Correlation between RCCP-FAT average global scores and corresponding participant satisfaction survey scores:
Dementia Education 360.1850.280
Problem-Solving Therapy 36−0.1970.250
Simulation 340.626 *0.000
Average Overall Global Score 360.2980.077
* Correlation is significant at the 0.01 level (two-tailed).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Chiu, M.; Nelles, L.J.; Wesson, V.; Lawson, A.; Sadavoy, J. Fidelity Assessment Tool for a Dementia Carers’ Group-Psychotherapy Intervention. J. Dement. Alzheimer's Dis. 2025, 2, 1. https://doi.org/10.3390/jdad2010001

AMA Style

Chiu M, Nelles LJ, Wesson V, Lawson A, Sadavoy J. Fidelity Assessment Tool for a Dementia Carers’ Group-Psychotherapy Intervention. Journal of Dementia and Alzheimer's Disease. 2025; 2(1):1. https://doi.org/10.3390/jdad2010001

Chicago/Turabian Style

Chiu, Mary, Laura J. Nelles, Virginia Wesson, Andrea Lawson, and Joel Sadavoy. 2025. "Fidelity Assessment Tool for a Dementia Carers’ Group-Psychotherapy Intervention" Journal of Dementia and Alzheimer's Disease 2, no. 1: 1. https://doi.org/10.3390/jdad2010001

APA Style

Chiu, M., Nelles, L. J., Wesson, V., Lawson, A., & Sadavoy, J. (2025). Fidelity Assessment Tool for a Dementia Carers’ Group-Psychotherapy Intervention. Journal of Dementia and Alzheimer's Disease, 2(1), 1. https://doi.org/10.3390/jdad2010001

Article Metrics

Back to TopTop