Engaging cancer patients in clinical practice guideline development : a pilot study

Background Patient engagement is a key quality component of cancer guideline development; however, the optimal strategy for engaging patients in guideline development remains unclear. The feasibility and efficacy of two patient engagement models was tested by Cancer Care Ontario’s cancer guideline development program, the Program in Evidence-Based Care (pebc). Methods In model 1, patients participated in the guideline development process as active members of a working group. In model 2, patients formed a separate consultation group to review project plans and recommendations generated by multiple working groups. Training included online resources (model 1) and an in-person orientation (model 2). The pebc’s standard patient engagement process acted as a control. The study was conducted for 1 year. Surveys measured the satisfaction of patients and members of the guideline working groups with the process and the outcome of each model. Results Three guideline projects used model 1 to engage patients, six projects used model 2 to receive feedback, and one project was used as a control group (14 patients total). Most participants, whatever the model, reported satisfaction with their experience. Key challenges to implementation included patient recruitment and long wait times between meetings (model 1), and difficulty focusing on the discussion topic and poor meeting attendance on the part of patients (model 2). Conclusions The pilot study demonstrated that, although both models are feasible and effective for the engagement of patients in cancer guideline development, modifications are required to optimize their continued interest. The pebc will use the study results to inform the implementation of a patient engagement strategy for its program.


BACKGROUND
Practice guidelines (pgs) are evidence-based recommendations designed to facilitate decision-making at the clinical, policy, and health system levels 1 .Guidelines have been shown to improve patient outcomes, care processes, and health systems as a whole 2 .A key quality component of pg development is a consideration of the values and preferences of all relevant stakeholders, which includes patients, caregivers, and family members 3 ; however, the views of the latter three groups are often not considered 4 .Guidance documents for pg development should provide clearer instructions about methods for engaging patients in the guideline development process 5 .Strategies for engaging patients should be tailored to the disease context.In the cancer context, for example, patients are often older and frailer, and compared with patients in other settings, they require more complex care 6 .
From 2014 to 2015, we conducted a research study-the Patients in the Cancer Guidelines Enterprise: How to Optimize Participation and Meet Information Needs (pg-pie) Project-investigating optimal approaches to engaging cancer patients, survivors, and caregivers in the development of cancer-specific pgs 7 .Cancer patients, survivors, and caregivers contributed to the pg-pie Project by attending a workshop, completing a survey, or participating in a telephone interview.The study aimed to gather the thoughts of participants about barriers and facilitators to engaging patients in pg development, to identify patient training needs, and to obtain feedback about two approaches to patient engagement (that is, involving patients as active members of pg groups vs. consulting them at key stages of the development process).Although several participants had experience in cancer pg development, participation in the study did not require knowledge of, or experience with, pg development.Overall, the study results showed that neither patient engagement approach was considered more preferable; however, key components of successful engagement in pg development were identified: n Clear explanation of the roles and responsibilities of patients, caregivers, and family members n Provision of adequate orientation and training for patients, caregivers, and family members n Establishment of positive working group dynamics overall The methods and results of the pg-pie Project have been published 7 .
Based on the results of the pg-pie Project, protocols were developed for two models of patient engagement in cancer pg development (active involvement with a pg working group and participation in a patient-specific consultation group) and were pilot-tested at the Program in Evidence-Based Care (pebc).The objective of the pilot study was to collect preliminary data for a determination of the feasibility and efficacy (for example, participant satisfaction) of the two patient engagement models in the pebc setting, using the pebc's standard engagement practice as a control group for comparison.The results of the 1-year pilot study, conducted from 2016 to 2017, are presented here.
The pebc is an initiative of the Ontario provincial cancer system, Cancer Care Ontario, whose aim is to improve the quality of cancer care in Ontario through the development, dissemination, and evaluation of pgs 8 .At the pebc, multidisciplinary working groups are responsible for developing pgs; the groups usually consist of 4-6 members.With the exception of a health research methodologist (hrm), working group members are most often volunteer clinicians (for example, oncologists, hematologists, family physicians, nurses) with relevant subject matter expertise, who have been invited to work in the group.The understanding of pg development on the part of those clinicians can be variable; an important role of the hrm is therefore to guide the working group members throughout the pg development process 8 .The group begins by defining a topic, determining research questions, and deciding on the scope of the project (which includes developing a project plan).The development of the project plan is followed by a systematic review of evidence, which, together with expert opinion, informs the recommendations for action 8 .Once the completed pg draft is reviewed by a larger panel of clinical experts and the pebc Report Approval Panel, it is published online.A guideline can take up to 2 years to complete and can be more than 100 pages in length.Patient engagement at the pebc has historically been inconsistent, most commonly involving the presence of a single patient representative on each standing pg committee.Given that the pebc's patient engagement process was identified as an area for future improvement, the pebc served as the ideal setting to test the two models created based on the pg-pie Project results.

METHODS
From here forward, the term "patients" encompasses cancer patients, survivors, and their family members and caregivers.Table i describes the recruitment strategy, orientation processes, and evaluative framework for each patient engagement model.All referenced materials were developed by the study team unless otherwise stated.The coordinator of the pilot study was a pebc hrm called the Patient Engagement Coordinator.That individual was responsible for implementing the models, collecting results data, and corresponding with participants.Given that the pilot study was deemed a quality improvement project, the requirement for full research ethics approval was waived by the Hamilton Integrated Research Ethics Board.

Model 1: Active Participation in Working Groups
Patients were active members of a pebc working group and contributed to working group meetings as patient representatives.This model planned to assign at least 2 patients to each participating working group.

Model 2: Consultation Group Members
Patients served as members of a patient-specific consultation group.This model planned to have a minimum of 6 patients participating in the consultation group.The group was to receive, from various pebc working groups, two types of pg documents for review: project plans and draft recommendations.Project plans were to be reviewed by 2-3 randomly chosen consultation group members; draft recommendations were to be reviewed by all consultation group members.It was the responsibility of the Patient Engagement Coordinator to organize and attend all consultation group meetings.The hrms whose working group project plans or recommendations were being reviewed were also expected to attend meetings.Any meeting notes taken by the Patient Engagement Coordinator and the hrms were summarized and forwarded to the respective working group members.

Control Model
The standard pebc patient engagement practice served as a control model.For many years, the pebc has had several standing pg committees with a single patient representative who, with the clinicians, participates in various aspects of the pg development process.These patient representatives did not receive any formal orientation or training when they joined the pg committees.The study team did not intervene in this model, aside from collecting evaluation surveys from the participants.

PEBC Project Selection
For model 1, the Assistant Director of the pebc was consulted by the study team about which ongoing or upcoming pebc projects would be appropriate for inclusion (for example, projects that were in the early stages of pg development, that could benefit from patient input, and so on).The members of the working group for each identified project were contacted and asked to consider including patient representatives in their groups.For model 2, any pebc working groups that had drafted project plans or recommendations during the pilot study period were encouraged to submit them to the consultation group for review.The Assistant Director of the pebc, in consultation with the study team, selected a project with an existing patient representative to serve as the control model.

Patient Recruitment
Cancer Care Ontario's Patient and Family Engagement Specialist distributed calls for participation to Cancer Care Ontario's Patient and Family Advisors by e-mail.Patient and Family Advisors are cancer patients, family members, and caregivers who work alongside Cancer Care Ontario staff to provide feedback about policies and practices that affect patient care.Multiple calls for participation were circulated for model 1; each call described a specific guideline project.Only 1 call for participation was circulated for model 2 because the initial target number of consultation group members was reached with that call.Patients interested in enrolling in either of the two models contacted the Patient Engagement Coordinator to initiate their participation in the study.No patient recruitment was required for the control model.

Patient Orientation
In model 1, patients who provided consent received an electronic orientation package, a terms of reference document, and a sample pg.After reviewing the orientation package, the patients participated in a 1-hour teleconference with the Patient Engagement Coordinator and the hrm from their assigned working group.The teleconference gave the Coordinator and the hrm an opportunity to describe the purpose of the study and the model being implemented, and to discuss the role and responsibilities of the patients.It also gave patients an opportunity to ask questions and express their concerns.
In model 2, interested patients were sent a terms of reference document providing an overview of the pebc and the pilot study, and asking their consent.Patients who provided consent attended an in-person half-day orientation session at the Cancer Care Ontario office in Toronto, Ontario.The orientation session was led by the study's Patient Engagement Coordinator, the Assistant Director of the pebc, and Cancer Care Ontario's Patient and Family Engagement Specialist, who discussed model 2 and guided the patients in analyses of a sample project plan and recommendations to prepare them for their role in the study.The pilot study team felt that the patients participating in model 2 would benefit from an in-person practical session, because they might be required to review fairly complex pg materials as consultation group members.
The control model participant received no formal orientation and training.

HRM and Working Group Member Orientation
The Patient Engagement Coordinator provided hrms with a guidance document about the study that outlined their roles and responsibilities; the Coordinator also conducted brief and informal in-person orientations with them.Participating hrms were asked to send a terms of reference document by e-mail to their fellow working group members to explain the patient engagement model being implemented in their group.Additionally, the study team provided hrms with materials for guiding their communications with patients.For example, hrms using model 1 received talking points for their 1-hour introductory teleconference with patients (supplemental Appendix 1).

Evaluation
A modified version of the Public and Patient Engagement Evaluation Tool (a questionnaire) was used to gather feedback from the patient participants about their level of satisfaction with the process and outcome of each model.This tool is available from McMaster University (Hamilton, ON) upon request 9 .It was developed through an iterative, collaborative process that combined a literature review with input from Canadian patient and public engagement researchers.Contributors focused on usability and comprehensiveness during the tool's development and therefore acknowledge that further testing of its psychometric properties is necessary 10 .
An additional survey was developed by the study team to gather similar data from participating hrms and working group members (supplemental Appendix 2).The Public and Patient Engagement Evaluation Tool and the hrm and working group member survey both consisted of a set of core questions (Likert scale and open-ended) that were altered slightly to reflect each model.All surveys were distributed once, before the conclusion of the study period.In addition to the survey data, regular informal feedback was provided to the study team by the Patient Engagement Coordinator.

RESULTS
Table ii summarizes the level of participation for each model.

Model 1: Working Group Meetings
The pebc's Assistant Director recommended 5 guideline working group projects for inclusion in model 1, but only 3 enrolled.One group chose not to participate because its members felt that the pg topic was too complex to require patient input, and another group had expected to be involved in the study, but was unable to recruit any patients to contribute.As a result, 2 hrms (1 hrm was a member of 2 of the 3 participating working groups) and 6 patients participated in this model.Each working group had 2 patient representatives.
Before participating, 2 patients (on the same working group) indicated that they did not want to attend working group meetings.They were uncomfortable with the thought of talking to a group of experts and did not want to appear uninformed or unhelpful, because they might not have understood the medical terminology and issues being discussed.They therefore met separately with the hrm (each patient representative met with the hrm twice alone, and they met with the hrm together once), rather than with their working group.Another model 1 working group had only 1 meeting during the study period, which their patient representatives attended.Both of the aforementioned working groups were at the stage of project plan completion (one of the earliest stages of pg development) when the pilot study concluded.The participating patient representatives therefore had an opportunity only to review project plans; no documents from any later stages of pg development were evaluated.
One model 1 project was completed during the study period.The document being developed for that project was not a full pg (as in the other 2 model 1 projects), but rather an evidence summary.An evidence summary differs from a pg in that it is a brief document summarizing the best available evidence about a particular topic, and no recommendations are provided.This working group met 2 or 3 times during the study period.Patient representatives attended the working group meetings and reviewed project plans and drafts of the evidence summary document.

Model 2: Consultation Group Meetings
Model 2 attracted the participation of 6 guideline working group projects and 5 hrms (1 hrm was a member of 2 of the participating working groups).One of the participating

Evaluation Participation
Evaluation surveys were collected from study participants only once.

Model 1
All 6 patient representatives from model 1 completed the survey.They agreed that the supports needed to participate were available to them [mean: 4.00 (1 = strongly disagree; 5 = strongly agree)]; however, they felt that their contribution to the pg development process was minimal because there were long periods of waiting between scheduled working group meetings.They suggested more frequent communication with the hrm or the Patient Engagement Coordinator (or both) to clarify the project timeline and to address delays.Follow-up meetings with the hrm to indicate how patient feedback was incorporated during pg development were also suggested.Lastly, although meeting dialogue was sometimes overly technical, patients appreciated the opportunity to participate and enjoyed learning about the evidence-based research process and the pg topic.

Model 2
Of the 7 consultation group members from model 2, 3 completed the survey.They felt that they were able to express their views during meetings (mean: 5.00) and that their feedback was being considered by the working group (mean: 4.67).One participant commented that having a larger consultation group would facilitate a better understanding of the topic (given that some materials are quite technical) and result in better discussions; that participant also suggested the use of an online platform as an avenue for dialogue.Like the patient representatives in model 1, consultation group members in model 2 were appreciative of the opportunity to participate in the guideline development process and to learn more about the various topics.

Control Model
One patient representative from the control model completed the survey.That individual valued learning about guideline development and hearing the perspectives of other working group members.The individual reported feeling unsure about how their feedback would be incorporated into the final guideline and recommended that, in the future, patient input should be sought at the earliest stages of pg development as well as after the final pg is drafted.

Model 1
Of 8 surveys completed, 5 were submitted by working group members, and 3 were submitted by hrms.Two members of the working group in which patient representatives met separately from the working group completed the survey.One individual thought that it was successful to have patient input relayed by the hrm; the other would have preferred that the patient representatives attend working group meetings.Some respondents reported that the members of their working groups were able to successfully engage patient representatives during meetings, because they would simply direct questions to patients any time they were unsure about patient opinion.Two facilitation techniques were suggested for effectively engaging patients during meetings: notifying working group members of the training and resources that patients receive before meetings, and designating breaks for patient input in meeting agendas.In general, survey respondents agreed that the input provided by patient representatives was valuable and relevant (mean: 4.14) and that it ultimately resulted in the development of a higher quality guideline (mean: 4.14).They indicated that they were interested in working with patients on future projects (mean: 4.50).

Model 2
The 3 surveys completed were all submitted by hrms; no working group member completed a survey.All 3 respondents agreed that the consultation group members contributed helpful feedback after reviewing pg materials.They suggested these improvements to model 2: n Involve patients earlier in the pg development process (that is, before project plan review).n Have multiple patients (or 1 patient lead) interact directly with the working group.n Recruit patients who are familiar with the topic.
Overall, the consultation group feedback allowed the working groups to better understand patient values.The hrms are interested in receiving patient input during future projects (mean: 4.67).

Control Model
One hrm from the control model completed the survey and was generally indifferent about the experience; no written feedback was provided.However, the hrm did feel fairly certain that the final pg document encompassed the patient perspective (score of 4/5 provided).

Patient Engagement Coordinator's Feedback
The Patient Engagement Coordinator met with study team members on an informal basis at least once each month to discuss the progress of the pilot study and to make suggestions for future implementation.The following section is a summary of the feedback received and has been validated by the Patient Engagement Coordinator.
The 1-hour orientation teleconference implemented in model 1 was informative for patients.It provided the hrm with an opportunity to describe the purpose of the study and the model being implemented, and to discuss the role and responsibilities of the patients.In addition, hrms appreciated receiving a meeting agenda (that is, talking points) to guide their orientation with the patients, who generally felt comfortable asking questions and voicing concerns.
The in-person half-day orientation session implemented in model 2 was also helpful for patients; they responded well to both the presentation and the practical activities, and they were appreciative of the reimbursement provided for travel-related costs.In regard to model 2 consultation group meetings, the Patient Engagement Coordinator found that a list of talking points and questions (developed by the study team) was useful, because it facilitated structured discussions and kept the meetings focused.
Finally, the guidance documents for hrms and the terms of reference documents for working group members were well received for all models.
Challenges were encountered during implementation.In model 1, patient recruitment was a challenge.Some patients did not respond to certain calls for participation because they assumed that they would not be appropriate patient representatives, given that they did not have experience with the particular cancer addressed in the pg.For other calls, the reason for a lack of response is not known.To recruit patient representatives for one model 1 working group, the Cancer Care Ontario Patient and Family Engagement Specialist contacted a national cancer organization.
With respect to model 2, difficulties were encountered in maintaining consistent attendance at consultation group meetings and keeping patients involved throughout the duration of the study.Another challenge was keeping certain individuals focused on the document review process during meetings, because they frequently strayed off topic.Furthermore, it was evident that some patients were more confident and comfortable than others in contributing during meetings.Lastly, when patients had no questions about pg materials or no suggestions for change, the hrm had no feedback to provide to the working group.That lack of feedback presented a challenge when gathering evaluative data from the working group members about the effectiveness of, and their satisfaction with, the consultation group.A lack of consultation group feedback did, however, imply that the documents had a sufficient person-centred perspective and did not require any changes.

DISCUSSION
This pilot study involved the implementation of two models of patient engagement at Cancer Care Ontario's pebc.In model 1, 6 patients participated as active members of 3 working groups.Of those 6 patients, 4 attended working group meetings, and 2 chose to meet separately with the hrm.In model 2, 7 patients served as consultation group members; they met with the Patient Engagement Coordinator (a hrm at the pebc) and the relevant hrm by teleconference, separately from the working groups, to review pg project plans and recommendations.Six working groups received feedback from the consultation group.The control model consisted of a working group that already had patient representation.The Patient Engagement Coordinator was responsible for coordinating the implementation of the pilot study and provided valuable feedback about the process.
Evaluation survey results suggest that the experience of most participants in all models was generally a positive one.The hrms and working group members strongly believe that engaging patients in pg development is valuable, and they expressed interest in working with patients in the future.Patients from all models appreciated learning both about the pg topic and about the evidence-based research process.

Lessons Learned
Model 1 feedback suggests that, before inviting patients to participate, the working group should decide whether patient input during guideline development is essential and whether patients will be able to meaningfully contribute during meetings.If patient input is valued, but the project topic is quite technical in nature, time must be allocated to ensure that patients understand the materials and are able to provide feedback.The primary challenge in implementing this model is recruiting patients to participate.Emphasizing in the calls for participation that patients are not required to have a particular type of cancer experience in order to participate might help to increase interest.Another solution would be to broaden the recruitment to other patient populations, outside of Cancer Care Ontario's Patient and Family Advisors.Furthermore, in any future implementation of model 1, communication between patients and the Patient Engagement Coordinator or the hrm must be strengthened.To avoid uncertainty or confusion during the early stages of pg development, the hrm should consistently communicate with patients, informing them of any changes to the initial project timeline and acknowledging delays in the process.
Model 2 feedback suggests that an in-person half-day orientation session (involving a short lecture and practical group activities) is an effective method of preparing patients for their role as consultation group members.The primary challenges in executing model 2 were keeping patients on topic and interested in attending meetings, because they often experienced conflicts with work or personal life.Potential solutions might be to assemble a larger initial group of patients (the group consisted of 7 patients at the beginning of this pilot), regularly recruiting and training new consultation group members, and providing members with other avenues for communication, such as Web-based discussion boards.Additionally, all pg materials submitted to the consultation group should be appropriate for patient review.
Control model feedback suggests that the current pebc patient engagement practice is acceptable, but not optimal.

Study Limitations
A limitation of the present study is its lack of a comprehensive literature review of patient engagement models.The study was, however, based on the pg-pie Project, which included an informal review of the patient engagement literature 7 .Another potential limitation is that the patient participants might not be representative of all cancer patients, caregivers, and family members.Evident limitations are the overall low study participation rate and the low survey response rates.A minimal number of working group members and hrms completed the evaluation survey, and we hypothesize that the low response was a result of lack of interest or lack of time.In addition, some working group members might have felt that their evaluations would not be valuable because they did not interact directly with the patients.Another limitation is the short duration of the pilot study, which made it difficult to evaluate model 1, because 2 of the 3 participating projects were still in the early stages of development and very few working group meetings occurred during the study period.Lastly (specific to model 2), only 1 patient was in attendance at 3 of the 6 consultation group meetings.

CONCLUSIONS
Results of our pilot study indicate that both piloted models are feasible and potentially effective for the engagement of patients in cancer pg development at the pebc; however, modifications are required to optimize the continued interest and contribution of patients over time.The pebc will use the results of this pilot study to inform the implementation of a patient engagement strategy in their guideline development program.The findings presented here might also be used to inform future implementation of patient engagement models in other (non-cancer) guideline development settings.

TABLE I
Study recruitment, orientation, and evaluation a In the case of one working group (model 1), the Cancer Care Ontario Patient and Family Engagement Specialist also contacted a national cancer organization to recruit patient representatives.PEBC = Program in Evidence-Based Care; NA = not applicable; HRM = health research methodologist; PEC = Patient Engagement Coordinator.Current Oncology, Vol. 25, No. 4, August 2018 © 2018 Multimed Inc.
Project plans were reviewed at 4 of the meetings, and draft recommendations were reviewed at the other 2. Positive and constructive feedback about the reviewed documents was provided.Most of the criticism received was associated with the technicality of the documents and the need for more consideration of quality of life in the draft recommendations.
a The Assistant Director of the PEBC recommended 5 working groups to participate in model 1, but only 3 enrolled.Current Oncology, Vol. 25, No. 4, August 2018 © 2018 Multimed Inc.
Table iii summarizes the number of completed evaluations for each model.