Minimum Supervision Levels for Graduation and Practice: Pediatric Gastroenterology Fellowship Program Directors’ Perspective

Abstract

Entrustable professional activities (EPAs) define key tasks expected of physicians. The entrustment levels that US pediatric gastroenterology (GI) fellowship program directors (FPDs) expect of graduating fellows remain unclear. This study aimed to identify the minimum entrustment levels FPDs expect for graduation and for safe and effective practice. We conducted a secondary analysis of GI data from a national multispecialty cross-sectional survey. FPDs reported the minimum entrustment level for fellows expected for graduation, whether they would graduate a fellow not meeting this level, and the minimum level for practice. A minimum entrustment level was defined as a level where at least 80% of FPDs would not allow a lower rating. Most FPDs did not require level 5 (unsupervised practice) for graduation or for practice. Instead, level 4 (indirect supervision with discussion of complex cases) was the most commonly required minimum for GI EPAs. When adjusted for FPD willingness to graduate fellows even if they did not meet the expected minimum, the level was 3 (direct supervision for complex cases) across all GI-specific EPAs. These findings underscore the potential need for ongoing mentorship, professional development, and continuing medical education after GI fellowship, particularly in the early practice period.

1. Introduction

The Accreditation Council for Graduate Medical Education (ACGME) in the United States (US) introduced an initiative in 1998 that reshaped physician education by emphasizing competency development to reliably and accurately assess trainee performance. [1] This initiative, called ‘The ACGME Outcome Project’, required that all US graduate medical education programs foster resident development along competencies and subcompetencies. This system was further revised in 2023 to include milestones to improve the assessment process [2]. Despite the effort to strengthen competency-based assessments, a drawback of this approach is that it can be abstract and may not always provide context for workplace-based assessment. To address this challenge, entrustable professional activities (EPAs) were promoted as measurable units of work that require demonstration of adequate knowledge, skills, and attitudes by physicians deemed competent to practice without supervision [3,4,5,6]. EPAs have been developed and implemented amongst international postgraduate trainees across diverse specialties and global regions [7,8,9]. In the Netherlands, a national modified Delphi study provided content validity for EPAs for pediatric intensive care fellows, achieving high consensus among experts and establishing a foundation for EPA-based assessment in Dutch pediatric subspecialty training [10].

The common pediatric subspecialty EPAs, developed by the American Board of Pediatrics (ABP) in collaboration with multiple pediatric subspecialties, represent broad professional activities that all pediatric subspecialists are expected to perform, such as providing consultations, coordinating longitudinal care, leading interprofessional teams, engaging in quality improvement, and applying evidence-based practice (Table 1). In contrast, subspecialty-specific EPAs capture specialized diagnostic reasoning, procedural skills, and disease management unique to a particular discipline. In North America, pediatric gastroenterology EPAs were first developed by the North American Society of Pediatric Gastroenterology, Hepatology, and Nutrition (NASPGHAN) EPA task force in collaboration with the ABP [2,3,11]. There are five EPAs specific to pediatric GI that encompass all previously described domains of competency and subcompetencies [5,11,12]. Taken together, these five subspecialty-specific EPAs and the seven common pediatric subspecialty EPAs define the skills needed to practice the pediatric gastroenterology, hepatology, and nutrition subspecialty (Table 1).

Table 1. Pediatric gastroenterology, hepatology, nutrition-specific, and common pediatric entrustable professional activities (EPAs).

GI-Specific EPA Title	Short EPA Title	Common EPA Title	Short EPA Title
EPA 1: Care of infants, children, and adolescents with acute and chronic gastrointestinal disorders	Acute and Chronic Gastrointestinal Disorders	EPA 1: Apply public health principles and improvement methodology to improve care for populations, communities, and systems	Quality Improvement (QI)
EPA 2: Care of infants, children, and adolescents with acute and chronic liver diseases, biliary/cholestatic diseases, pancreatic disorders, and those requiring liver transplantation	Acute and Chronic Liver, Biliary, and Pancreatic Disorders	EPA 2: Provide for and obtain consultation from other health care providers caring for children	Consultation
EPA 3: Care for infants, children, and adolescents with common outpatient GI, liver/biliary, pancreatic, and nutritional issues	Common Outpatient	EPA 3: Contribute to the fiscally sound and ethical management of a practice (e.g., through billing, scheduling, coding, and record keeping practices)	Financial and Ethical Management
EPA 4: Care of infants, children, and adolescents with nutritional issues, deficiencies, and obesity	Nutrition	EPA 4: Facilitate handovers to another healthcare provider	Handovers
EPA 5: Perform medical procedures related to gastrointestinal and liver diseases for screening, diagnosis, and intervention	Procedures	EPA 5: Lead and work within interprofessional health care teams	Lead/Work in Teams
		EPA 6: Engage in scholarly activities through the discovery, application, and dissemination of new knowledge (broadly defined)	Scholarly Activity
		EPA 7: Lead within the subspecialty profession	Lead in Specialty

Entrustment Level Scale for EPAs: Level 1—trusted to observe only, Level 2—trusted to execute with direct supervision and coaching, Level 3—trusted to execute with indirect supervision and discussion of information conveyed for most simple and some complex cases, Level 4—trusted to execute with indirect supervision, but may require discussion of information conveyed for a few complex cases, Level 5—trusted to execute without supervision.

In the US and beginning in 2028, EPAs will be part of the certification process to determine eligibility to take the subspecialty certification examination. Previous studies in pediatric subspecialties suggest that fellowship program directors (FPDs) may graduate fellows who may require direct and indirect supervision [13,14,15,16]. Additionally, these studies also show that FPDs feel that practicing physicians may require indirect supervision for limited numbers of complex cases. Collectively, these studies raise important implications for subspecialty certification standards, faculty development efforts, and the design of post-graduation supervision models within pediatric subspecialties. The Association of Pediatric Programs Directors’ Subspecialty Pediatrics Investigator Network (APPD SPIN) surveyed US pediatric gastroenterology FPDs to determine the minimum level of supervision deemed necessary for fellows to graduate, and the minimum level of supervision deemed necessary for physicians to practice independently in a safe and effective manner. We hypothesized that the minimum entrustment level expected by pediatric gastroenterology FPDs for graduation and independent practice would be lower than level 5, such that some supervision of practitioners and graduates would be required across EPAs.

2. Materials and Methods

We performed a secondary analysis of data from a cross-sectional study conducted by the APPD SPIN between April and August 2017 that included an anonymous survey of FPDs from ACGME-accredited fellowships [17,18]. At the time of implementation of the study, the APPD SPIN was a collaborative network composed of 1–2 representatives from each of the 14 pediatric subspecialties, as well as representatives from the Council of Pediatric Subspecialties, the ABP, and the APPD Longitudinal Educational Assessment Research Network (APPD LEARN) [19]. Using a modified Delphi approach, SPIN representatives with graduate medical education (GME) experience designed, reviewed, and revised the survey [17,18]. The survey was piloted among faculty from each subspecialty who previously served in an FPD role and revised based on their feedback. Subsequently, FPDs of ACGME-accredited US fellowship programs were invited to complete a confidential electronic survey. Periodic personalized email reminders, including from SPIN representatives, were sent to non-responders to request survey completion.

For each EPA, FPDs were asked the following questions:

(1)

“For this EPA, what do you believe is the MINIMUM level of supervision a fellow must achieve to successfully complete fellowship?”

(2)

“For this EPA, if a fellow did not achieve at least this minimum level of supervision, would you still allow him/her to graduate?” with choices “yes” or “no”.

(3)

For this EPA, “what is the LOWEST level in which you will consider that a practicing subspecialist (and not necessarily a trainee) should be able to perform most of the activities described above resulting in a safe and effective outcome?”

To ensure a shared mental model, we explained the concept of entrustment in choosing the entrustment level, specifically that the FPD does not necessarily have to see a fellow perform the functions to trust them to do so. In addition, we collected information from FPDs about the number of fellows in the program, duration of program directorship, understanding of EPAs, use of EPAs, and participation in the previous SPIN [20].

2.1. Data Analysis

We summarized responses as numbers (percentages) for categorical survey items and as medians and percentiles, including the minimum entrustment level for graduation, as reported by FPDs to successfully complete fellowship. If an FPD indicated they would allow a fellow to graduate even if they did not achieve their reported minimum entrustment level, we treated their actual minimum as one level below their reported minimum. For example, if a PD indicated that the “minimum level of supervision to successfully complete fellowship” was X, but also reported that they would still allow graduation “even if a fellow did not achieve at least this minimum level of supervision,” we coded the actual minimum as X–1. This approach accounted for the possibility that some PDs may have initially reported their minimum level as X–1; applying a further lower value could therefore introduce additional error. Accordingly, we reconciled these inconsistent responses by using X–1, the option least likely to introduce error. Based on expert consensus and aligning with previously published residency EPA performance data, the minimum entrustment levels for graduation and practice were defined as levels where at least 80% of FPDs would not allow lower ratings for each EPA [13,14,15]. We measured the correlation between key variables with graduation and entrustment levels using Spearman’s ρ. In significance tests, p < 0.05 was considered significant. Data analysis was conducted using R 4.4.1 (R Core Team, Vienna, Austria).

2.2. Ethics Statement

This study was reviewed and deemed exempt by the University of Illinois at Chicago Institutional Review Board.

3. Results

Fifty-four of seventy FPDs (response rate 77.1%) completed the survey. Respondents reported a median duration of 6 years (interquartile range 3–10 years) as an FPD. The number of fellows per program ranged from 1–12 (median 4.5). An equal number of FPDs (46.3%) reported a “basic” and “in-depth” understanding of EPAs while 7.4% rated their understanding as “expert”. No FPD reported being “unfamiliar” with EPAs. There was no correlation (p > 0.05) between FPDs’ expectations of entrustment level with their years as an FPD, the number of fellows in their program, their understanding of EPAs, or whether they were using EPAs in their fellowship.

The minimum entrustment levels for graduation and practice required by FPDs for GI-specific EPAs are shown in Figure 1. Most FPDs did not require a minimum entrustment level of five (perform the EPA without supervision) for graduation or for practice for all GI-specific EPAs. Rather, the majority of FPDs required a minimum entrustment level of four for both graduation and for practice corresponding to performing the EPA with indirect supervision and discussion of a few complex cases (Figure 1). The median minimum level of supervision adjusted for a program director’s willingness to graduate fellows even if they did not meet the expected minimum level was three for all GI-specific EPAs. Similarly, the adjusted median minimum level of supervision that FPDs believed to be required for pediatric gastroenterologists to practice safely and effectively was also three, indicating that indirect supervision and discussion may be required for both simple and complex cases.

Figure 1. Proportion of fellowship directors requiring specific levels of entrustment for each GI-specific EPA for graduation (left) and practice (right). Y-axis represents GI-specific EPAs 1–5. X-axis shows proportion of program directors requiring each EPA level. Abbreviations: GI, gastroenterology hepatology and nutrition, entrustable professional activities; FPDs, fellowship program directors.

The minimum entrustment levels for graduation and practice required by FPDs for common pediatric specialty EPAs are shown in Figure 2. The minimum entrustment level for graduation for Consultation, Handovers, and Leading a Healthcare Team EPAs was three, requiring indirect supervision of graduating fellows for selected simple and complex cases. For the remaining common EPAs, one (QI), three (financial and ethical management), six (scholarly activity) and seven (lead in specialty) FPDs would graduate fellows requiring direct supervision and coaching for safe and effective completion of these EPAs. The minimum level of supervision that FPDs believed to be required by subspecialists to perform safe and effective practice ranged from two (direct supervision and coaching) to four (indirect supervision for few complex cases).

Figure 2. Proportion of fellowship directors requiring specific levels of entrustment for each common pediatric EPA for graduation (left) and practice (right). Y-axis represents common pediatric EPAs 1–7. X-axis shows proportion of program directors requiring each EPA level. Abbreviations: EPAs, entrustable professional activities; FPDs, fellowship program directors.

4. Discussion

We report data that suggest that pediatric gastroenterology FPDs would graduate fellows who require some direct and indirect supervision for at least complex cases for all EPAs. These results are similar to studies in other pediatric subspecialties and general pediatric training [13,14,15,16,17,18,20,21]. Our findings suggest that trainees may need some supervision after graduation. This need reflects the complexity of clinical care and that expertise is developed over years of medical practice.

It should be noted that an entrustment level of four (“trusted to execute the EPA”) includes the expectation that a physician “may still require discussion for complex cases.” As both recent graduates and independent practitioners may benefit from additional support when managing complex clinical situations, this underscores the importance of intentional faculty development structures that promote ongoing consultation and learning. Existing resources, including local and national case conferences, online and email-based forums, subspecialty interest groups, and peer-to-peer consultation, can be leveraged as formal components of faculty development rather than as ad hoc education. Engagement in these collaborative practices through national professional networks, and even across specialties, helps ensure that clinical care is informed by a broad and diverse base of expertise. Institutional leadership and program directors play a critical role in embedding activities that support the transition to practice within faculty development frameworks, both during fellowship and beyond, by actively promoting access to these resources and integrating them into routine practice to foster lifelong learning, professional growth, and patient safety.

For all five GI-specific EPAs, most FPDs required a minimum entrustment level of four for graduation and practice. However, when evaluating Acute and Chronic Liver, Biliary, and Pancreas Disorders, approximately 50% of FPDs chose a minimum entrustment level of four or five, reflecting expectations of minimal to no supervision, while the remaining 50% recommended that some degree of direct and indirect supervision continue beyond graduation into practice. In contrast, Acute and Chronic Gastrointestinal Disorders were viewed as requiring less ongoing oversight, with approximately 75% of FPDs endorsing a minimum entrustment level of four or five. This disparity may be explained by the existence of an advanced fourth-year fellowship in hepatology, which is commonly pursued by fellows intending to care for children with complex liver disease. This additional training likely shapes expectations regarding the level of supervision required to manage patients with acute and chronic liver or cholestatic disease, as well as those requiring liver transplantation—conditions often managed by hepatologists with this specialized training, rather than by general pediatric gastroenterologists.

The minimum entrustment level as defined by consensus, where at least 80% of FPDs would not allow lower ratings for all GI EPAs, was the same for graduation and practice, which suggests that fellows do not graduate at a level lower than expected for practice, but rather FPDs think practicing physicians also need supervision with complex cases. This finding highlights the idea that medical care is complex and underscores the ethical responsibility of training programs to ensure that graduates are competent to provide safe patient care. In 2028, EPAs will be applied to the certification process. Our data suggest that faculty and fellow development is needed, as up to 25% of FPDs reported that fellows are ready to graduate despite requiring indirect supervision. Nonetheless, these EPAs provide a starting point for fellowships as they develop a comprehensive competency-based medical education assessment program.

FPDs selected lower minimum entrustment levels for many of the common pediatric subspecialty EPAs compared to GI-specific EPAs, with the exception of the Consultation and Handover EPAs. These two patient care-focused EPAs were rated among the highest, with over 75% of FPDs endorsing a minimum entrustment level of 4 or 5 for practicing pediatric gastroenterologists. In contrast, common EPAs that are less directly related to clinical care, such as leadership within a specialty and scholarly activity, were assigned lower minimum entrustment levels for practice. These findings are congruent with the most important aspects of training future clinicians with a focus on EPAs that are related to the care of patients. Skills in leadership and scholarship, while essential, may require time to develop through continued practice and experience beyond fellowship training.

Our study had a few limitations. First, although FPDs were surveyed about their expectations for graduation and readiness for unsupervised practice, many were likely not using EPAs as assessment tools, despite being familiar with the framework. Data presented here reflect FPDs’ perceptions and opinions, rather than actual performance-based outcomes. A second limitation was the lack of standardized education on the entrustment scale provided to FPDs prior to the survey. This may have led to variability in how entrustment levels were interpreted. Finally, the use of data collected in 2017 may impact its applicability in current contexts. Although EPAs were developed over a decade ago, the literature suggests that fewer than half of US subspecialty fellowship directors currently utilize EPA-based assessments. This limited uptake may be partly due to the absence of a previous mandate from the ABP to implement EPAs. More recent studies indicate that EPA use among pediatric gastroenterology fellowship program directors remains variable, and that both users and non-users often report limited familiarity or only basic knowledge of EPAs as barriers to using EPAs [22]. Lastly, while GI-specific EPAs may be revised in the near future in preparation for their required use in 2028, until now they have not undergone major revisions [22,23]. As a result, despite the data having been collected in 2017, it is plausible that FPDs’ understanding of EPAs and their use of EPA entrustment scales would be similar if assessed in the current context.

5. Conclusions

EPAs are units of work that collectively define a specialty. Although the data suggests that FPDs consider the level of supervision needed for graduation and practice to be similar, neither reach a level that considers fellows able to practice without supervision. This adds to a growing body of evidence that expertise is not developed in a few years of training, but in a lifetime of medical practice. This data reinforces the need for support for both trainees and graduates alike. Using the EPAs as a guide, timely and formative feedback to fellows on how to improve their performance will ideally lead to improved entrustment for graduates and practicing physicians.