Implementing changes to a residency program curriculum before competency-based medical education: a survey of Canadian medical oncology program directors

Background Postgraduate medical education is undergoing a paradigm shift in many universities worldwide, transitioning from a time-based model to competency-based medical education ( cbme ). Residency programs might have to alter clinical rotations, educational curricula, assessment methods, and faculty involvement in preparation for cbme , a process not yet characterized in the literature. Methods We surveyed Canadian medical oncology program directors on planned or newly implemented residency program changes in preparation for cbme . Results Prior to implementing cbme , all program directors changed at least 1 clinical rotation, most commonly making hematology/oncology (74%) entirely outpatient and eliminating radiation oncology (64%). Introductory rotations were altered to focus on common tumour sites, and later rotations were changed to increase learner autonomy. Most program directors planned to enhance resident learning with electronic teaching modules (79%), new training experiences (71%), and academic half-day changes (50%). Most program directors (64%) planned to change assessment methods to be entirely based on entrustable professional activities. All programs had developed a competence committee to review learner progress, and most (86%) had integrated academic coaches. Transitioning to cbme led to major structural and curricular changes within medical oncology training programs. Identifying these commonly implemented changes could help other programs transition to cbme .


INTRODUCTION
Competency-based medical education (cbme) is the most recent curricular paradigm adopted in Canada. The Royal College of Physicians and Surgeons of Canada (Royal College) has developed a model for cbme known as Competence by Design. Several years of preparation have gone into cbme, including the University of Toronto's orthopedic surgery residency program, which has piloted the cbme model since July 2009 and researched its merits 1 . Since that time, cbme has been officially implemented in numerous other specialties in Canada, and widespread adoption of cbme by all disciplines is anticipated by 2022 2 . In the traditional medical education model, residents' competence was presumed upon completion of the requisite time in training and demonstration of required knowledge acquisition through end-of-clinical-rotation evaluations. Thus, advancement through stages of training was predominantly time-based rather than based on achievement of competence. The cbme model de-emphasizes time-based training, and instead focuses on learning outcomes and the demonstration of competence 3 . The Royal College has helped residency training programs develop specialty-specific competencies for each stage of training, under the guidance of the previously well-known canmeds framework 4 .
Historically, competence to practice in one's field was assumed by the end of training and passing licensing exams with confirmation of successful completion of training by a university postgraduate office, granting trainees the necessary qualifications for independent practice. This approach addresses only the first level of Miller's pyramid of clinical assessment ( Figure 1) 5 , evaluation of a trainee's knowledge, but fails to address the other crucial elements of this model (competence, performance, and action) 6 . Additionally, knowledge, skills, problem-solving, and attitudes are not consistently linked; thus demonstration of competence in one domain does not predict competence in another 7 . Given that all 4 of these domains are fundamental to competence as a physician, more comprehensive methods of assessment are necessary. Competency-based medical education breaks away from traditional summative assessments and instead focuses on formative assessments to guide learning 3 . A major advantage of cbme is that it allows for early identification of deficiencies in trainees' knowledge, skills, or attitudes so that they can receive supplemental training to address these gaps 8 . Moreover, trainees who demonstrate competence earlier also benefit from potential opportunities for more targeted, higher-level training 8 . Given that physicians are responsible for continuing medical education, which after residency becomes more self-regulated, cbme seeks to equip trainees with the skills needed for lifelong learning following completion of training.
Traditionally, medical oncology programs consisted of 2 years of training in postgraduate years (pgys) 4 and 5 in all provinces. In Quebec, trainees who have completed pgy4 and pgy5 in hematology have the opportunity to pursue a pgy6 year in medical oncology. Throughout medical oncology training, learners would complete various 4-week rotations, and supervision on each rotation was tailored to the learner's level of training. Learners would receive an end-of-rotation evaluation, and provided it was satisfactory, they would be permitted to advance to the next rotation. By the end of pgy5, learners would be granted permission to write the Royal College's medical oncology licensing examination and enter independent clinical practice, provided they successfully completed their postgraduate medical oncology training.
In the cbme model, rotation structure and sequence are designed for graded learning. Training occurs in 4 stages along the competence continuum: transition to discipline, foundations of discipline, core of discipline, and transition to practice 9 . Within each stage, learners must achieve certain entrustable professional activities (epas), which are the essential tasks in a given specialty, in order to accomplish their milestones, which are observed indicators of the learner's abilities. Achieving these epas and milestones is necessary for learners to advance into the subsequent stage of training 10 . In order to be deemed successful in completing the training requirements of the specialty, learners must have achieved all epas and milestones and have completed all 4 stages of training. Though rotations are still time-based, the 4 stages of training and the requirement of successful completion of milestones to advance in training allow for graded learning. The Royal College has also outlined certain required training experiences, which are mandatory academic and clinical experiences that residency training programs must provide to trainees in order for trainees to obtain the necessary competencies of their specialty 11 .
The successful implementation of cbme relies heavily on faculty members to provide direct observation with immediate feedback to trainees and to form a competence committee. The competence committee performs routine reviews of residents' milestones and epas to make informed decisions about whether a resident is ready to progress to the next stage of training 12 . Faculty also serve as academic coaches. An academic coach is a longitudinal advisor who meets regularly with the learner to monitor and revise the learning plan based on the competence committee's recommendations 12 .
Before the implementation of cbme in medical oncology training programs across Canada beginning in July 2018, it was anticipated that program directors would initiate several changes to the overall structure of their programs, including clinical rotations, academic curriculum,  teaching format, and assessment of learners. The extent and scope of changes needed to transition to cbme within Canadian programs was not known. This motivated our study, which sought to determine and measure what specific curriculum and programmatic changes were being implemented by medical oncology program directors in their transition to cbme.

METHODS
We conducted a national cross-sectional survey of all Canadian medical oncology residency program directors. The survey was given to all medical oncology program directors in person, at a national meeting. One program director completed the survey electronically and sent responses back by e-mail. Participants were informed that the survey was for research purposes and that completion and return of the survey provided their implied consent for their responses to be used for the research study. The survey was conducted in May 2018, 2 months prior to the official launch of cbme across all Canadian medical oncology residency training programs. The survey was available only in English.
The survey instrument was developed through literature review and expert consultation. A draft was circulated to 3 experts, including program directors and educators in medical oncology for assessment of content and face validity. The final survey consisted of 22 questions: 17 were yes/no questions with space provided for elaboration, and 5 were short-answer. Survey questions fell into 5 broad categories: structural and curricular rotation changes (including changes to rotation content or sequence), orientation of incoming residents and faculty to cbme, changes to learning resources for residents, changes to methods of teaching and assessment of trainees, and engagement of faculty members. Local ethics approval was attained by the Hamilton Integrated Research Ethics Board, and the project approval number is 5611.
Descriptive statistics were used to summarize yes/ no survey responses. Narrative responses to short-answer questions from respondents were grouped by common themes and subsequently summarized through descriptive statistics.

RESULTS
In total, 14 out of 15 Canadian medical oncology program directors completed our survey (response rate: 93.3%). All institutions had pgy4 trainees participating in Competence by Design, and some residency training programs had pgy5 and pgy6 trainees participating as well (Table i).

Rotation Changes
All program directors responded that they were changing at least one of the following core rotations: medical oncology outpatient clinic rotation, medical oncology inpatient rotation, hematology/oncology rotation, radiation oncology rotation, and palliative care rotation ( Figure 2). Of all rotations, the hematology/oncology rotation was changed most frequently [reported by 71% (10/14) of program directors], with the most commonly reported change being elimination of the inpatient component and a transition to an entirely outpatient, clinic-based rotation. Other reported changes included moving the hematology/oncology rotation from pgy4 to pgy5, eliminating hematology/oncology on-call responsibilities, and eliminating a dedicated hematology/ oncology rotation, with incorporation of the relevant content into pre-existing medical oncology rotations.   Changes to the radiation oncology rotation were implemented in 64% (9/14) of programs. The most commonly reported change was the removal of radiation oncology as a formal rotation and subsequent integration of relevant radiation oncology aspects of care into pre-existing medical oncology rotations as part of multidisciplinary care. The previously mandatory palliative care rotation was altered in 14% (2/14) of programs, where it was removed as a formal rotation, with subsequent integration of oncology-based palliative care clinics into other pre-existing rotations.
Program directors were asked whether changes were planned to the first block of pgy4 medical oncology residency training, corresponding to the "transition to discipline" phase of training. Of responding program directors, 36% (5/14) planned not to make any changes to the first block; 36% (5/14) planned greater clinical exposure to all core disease sites (breast, lung, and gastrointestinal cancers); and the remaining 29% (4/14) planned to change the first block such that all residents would begin on one core disease site, but with more of a focus on orientation topics. Most program directors planned to make changes to the sequence of resident rotations, with some introducing core disease sites earlier (50%, 7/14); others moved the community rotations and electives to later in the pgy4 year (21%, 3/14). A minority of program directors were planning to specifically sequence rotations to fit with epa assessment requirements (21%, 3/14).
With regard to the transition-to-practice final blocks, the majority of program directors reported they were planning to make changes-specifically, more independence in longitudinal clinics (7%, 1/14); more electives or community rotations, or both (29% 4/14); more responsibilities in supervising junior residents (21%, 3/14); and allowing those transition-to-practice blocks to be structured as per a resident's discretion based on their current needs and future career goals (21%, 3/14). No program directors planned on making any changes to residents' medical oncology on-call scheduling or responsibilities.

Orientation of Incoming Residents to the CBME Curriculum
Program directors were asked about how they would provide orientation to incoming pgy4 trainees about cbme ( Table ii). The majority of program directors responded that they planned to utilize presentation slides provided by the Royal College (57%, 8/14) from their Web site and planned to meet directly with their residents as a group for a formal review and discussion (57%, 8/14). Some program directors also planned to have additional cbme orientation sessions led by their local university cbme lead or working group (36%, 5/14). With regard to the content of the orientation material for residents, the majority of program directors planned to incorporate a discussion of epas (57%, 8/14) and an explanation of the key concepts of cbme (50%, 7/14). Other reported content of resident orientation to cbme included descriptions of required training experiences (29%, 4/14), an introduction to the Royal College online assessment software program ePortfolio (29%, 4/14), a review of curriculum maps (7%, 1/14), accreditation requirements (7%, 1/14), and the role of the competence committee (7%, 1/14).

Changes to Learning Resources for Residents
The majority of program directors reported that they were planning to use, or had recently started using, new electronic teaching modules (79%, 11/14). Some of these teaching modules included McMaster University's basic sciences of chemotherapy electronic teaching modules, as well as online e-modules provided through https://www.oncologyeducation.com/ 13 or the American Society of Clinical Oncology University. Half the program directors planned on making changes to the overall structure and content of their academic half-day curriculum, including altering the sequence and timing of various oncology topics (21%, 3/14), providing more teaching of the nonmedical expert intrinsic roles (14%, 2/14), including more flipped classroom teaching (14%, 2/14), and making additional changes specifically to fit the curriculum map or competency milestones (14%, 2/14), and more case-based teaching (7%, 1/14).
Most program directors (71%, 10/14) were planning to implement new training experiences to supplement resident learning and provide further means of assessment. The most commonly reported new educational experiences that program directors were planning to implement included a formal research curriculum, participation in tumourspecific case conferencing rounds starting in the foundations block, and journal clubs (Figure 3). Some programs (21%, 3/14) were already offering some of these new training experiences and thus were not planning to make any further changes, and one program was awaiting further guidance from the Royal College before making any changes.

Changes to Methods of Assessment
The majority of program directors (86%, 12/14) reported that they had already developed curriculum maps for residents to help determine which epas could be evaluated in each clinical rotation. Half of all programs (50%, 7/14) planned to use the Royal College ePortfolio as the primary software for evaluation purposes, while the other programs planned to use other software programs such as Entrada (Elentra Consortium, Queen's University, Kingston, ON), Meditech (Westwood, MA, U.S.A.), or One45 (Vancouver, BC). Most training programs (65%, 9/14) planned to eliminate end-of-rotation evaluations (for example, in-training evaluation reports) and instead intended to replace them with individual epa assessments. The program directors who reported they would continue using rotation evaluations (36%, 5/14) planned to do so at the request of their postgraduate medical education office to continue such assessments until cbme was more established. A minority of program directors (29%, 4/14) planned to use simulation as a means of assessing any of the epas, specifically for assessment of communication skills and chemotherapy suite emergencies.

Responsibilities of Faculty Members
The majority of program directors (86%, 12/14) indicated that they planned to introduce academic coaches or advisors into their training programs. All programs had planned to have a competence committee, and in the majority of programs (79%, 11/14), the competence committee had already been formed and was fully functional for evaluation of resident progression prior to the formal implementation of cbme. Program directors planned to have up to 8 members on their competence committee, with the majority (64%, 9/14) having planned for a committee consisting of 5 or 6 faculty members. All program directors were members of their program's competence committee, and the majority (57%, 8/14) functioned as an active voting member, although some program directors (36%, 5/14) reported they functioned as a nonvoting member. The majority of program directors (86%, 12/14) reported that they were very concerned about the impact of direct observation of residents on clinic flow in busy medical oncology clinics. The strategies they proposed to overcome this challenge included observing residents in their own longitudinal clinics, pre-planning for observed assessment at the start of the clinic, observing only one part of an encounter, observing residents in a different physician's clinic, and providing formal faculty development on how to execute direct observation with immediate feedback in an efficient manner.

DISCUSSION
Our study demonstrates that medical oncology program directors across Canada implemented many changes in curriculum and clinical rotation design as well as overall program structure in preparation for the transition to cbme. We found that the most common changes included re-sequencing of rotations to start with more core rotations early in training, introducing e-modules as a supplementary learning resource for trainees, introducing epas as the primary means of evaluation of competencies, and engagement of faculty members through a competence committee and as academic coaches. The majority of program directors expressed concern for the impact of direct observation on clinic flow in the busy outpatient setting, which has previously been highlighted in the medical education literature surrounding cbme 14,15 . To our knowledge, this is the first study surveying program directors on the changes they have implemented in their residency programs in preparation for cbme. A previous study involved semi-structured interviews of Canadian anesthesia program directors exploring their perceptions   of cbme, including the advantages and potential challenges 16 . Improved goal setting to address weaknesses, accelerating learning for advanced trainees, and a more personalized training program guided by feedback were reported advantages 16 . However, anticipated challenges in implementing cbme largely centred around the need for increased time commitment from administrators, faculty members, and residents 16 . The results from our study build on these findings by defining the upfront investment of time and resources required to implement cbme.
Our results show that the transition to cbme is encouraging medical educators to re-evaluate the current model of education delivery. In the traditional model of medical education, trainees completed clinical rotations for the purpose of being exposed to various clinical environments, each for a required period of time. In cbme, clinical rotations and other training experiences must align with the intended learning outcomes of residency training, and therefore residency training must be more thoughtfully planned and delivered. Our study results suggest that medical oncology program directors do appear to acknowledge the need to align training experiences to learning outcomes given that most are changing the sequence and types of clinical rotations for their training programs to better align with the stages of training on the competence continuum 9 . Additionally, the implementation of cbme is leading educators to develop innovative ways to provide residents with exposure to all of their required training experiences. We found that many program directors eliminated formal radiation oncology and palliative care rotations, but trainees can still achieve these required training experiences through medical oncology rotations as a result of the multidisciplinary nature of medical oncology clinical practice. Although direct observation and the achievement of milestones and epas are key aspects of cbme, the broader and more philosophical change to medical education brought about by this new education paradigm is the alignment of curricular experiences to the attainment of educational outcomes.
The strength of our study includes the high response rate of participants, with 14 of 15 (93%) medical oncology program directors responding to our survey. Another strength of our study is that the survey was carried out just prior to the uniform implementation of cbme in all programs. The implementation and study of any educational intervention across multiple institutions can be challenging due to restrictions on curriculum changes at different sites, differences in evaluation methods, and limitations in the number of available faculty members who can oversee residents' training 17 . Fortunately, cbme was officially implemented by all Canadian medical oncology programs in July 2018, which means that all programs were undergoing changes at the same time, thereby allowing for an accurate cross-sectional assessment.
A limitation of our study is the small sample size. Although our survey was completed by almost all medical oncology program directors in Canada, this still constitutes a small sample size and reflects only one medical specialty. Results of this study might not reflect changes in other specialties, particularly procedural specialties, longer training programs, or the experiences of training programs in other countries. As such, certain findings from our study might be less relevant to other specialties. Further studies are needed to validate our findings and to determine whether similar changes are seen in other contexts. Lastly, our specialty of medical oncology is one of the first and only internal medicine subspecialties to transition to cbme to date. It is likely that when more residency training programs transition to cbme, the downstream effects of alterations to resident rotations will be magnified. Those specialties that are adopters of cbme could potentially experience fewer challenges through their interactions with the early adopters and increased time to prepare for cbme.
Future studies planned include a repeat survey of medical oncology program directors following the implementation of cbme to determine what changes worked well, what changes were not helpful or needed adjustment, and what additional changes are anticipated. Additionally, it might be helpful to survey residents who have trained in the cbme era to determine whether they feel their residency training adequately met their educational needs and whether their residency training program structure allowed them to feasibly complete their epa assessments. Given program directors' concern for the feasibility of direct observation of residents with immediate feedback, a follow-up survey of faculty members' perceptions of the integration of direct observation for cbme into a busy outpatient clinical practice, and successful strategies for making direct observation feasible, would also be valuable.

CONCLUSIONS
Our survey of medical oncology program directors quantified the numerous curricular and structural changes required in residency programs prior to transitioning to cbme. Significant changes were made to the structure and sequence of clinical rotations, to learning resources, and to methods of assessment of residents. Programs have also engaged faculty members through the use of academic coaches and a competence committee. Ultimately, with the implementation of cbme, medical oncology residency training programs have shifted away from time-based education and instead are focusing on aligning residents' training with educational outcomes. The program changes made by medical oncology program directors in anticipation of cbme could be valuable for other programs as cbme becomes widely adopted in the coming years.