Barriers and Opportunities for Patient-Reported Outcome Implementation: A National Pediatrician Survey in the United States

Purpose: To characterize pediatricians’ perceived barriers and areas of confidence in assessing patient-reported outcomes (PROs) in the U.S., and to test associations of these factors with implementing PRO assessment. Methods: Using a random sample from the members of American Medical Association, we recruited general pediatricians and pediatric subspecialists to complete a survey (July 2011 to December 2013). Perceived barriers and confidence in PRO assessment were compared by age, pediatric specialty (general pediatrics, seven subspecialties), practice settings (academic, private), and region of practice. Multivariable logistic regressions tested associations of demographic factors, barriers, and confidence factors with the implementation of PRO assessment. Findings: The survey was completed by 458 participants (response rate 48.5%); of these, 40.4%, 15.9%, 15.5%, and 8.1% were general pediatricians, cardiology, hematology/oncology, and pulmonary specialists, respectively. PRO assessment was implemented by 29.0% of the pediatricians. The top five barriers for PRO assessment included limited time/manpower (79.0%), limited training (77.4%), lengthy PRO instruments (76%), lack of meaningful cut-offs on PRO scores (75.5%), and unavailable PRO instruments (75.0%). Limited knowledge of PROs (OR 4.10; 95% CI 2.21, 7.60) and unavailability of PRO instruments (OR 1.87; 95% CI 1.01, 3.49) increased the odds of not implementing the assessment, whereas confidence in PRO assessments compatible with norms (OR 0.41; 95% CI 0.23, 0.72) and perceived benefit over clinical judgment alone (OR 0.53; 95% CI 0.31, 0.93) decreased the odds of not implementing the assessment. Interpretation: significant barriers to PRO assessment in pediatric settings suggest the need for providing training, resources, and practical guidance toward implementation. Patient or Public Contribution: healthcare service users contributed to this study by completing a survey and providing feedback about the barriers and areas of confidence in assessing PROs for pediatric populations.


Introduction
Patients' symptoms, quality of life, functional status, and life satisfaction (known as "patient-reported outcomes" or "PROs") are rarely observable but can be measured through self-reports. Various PRO instruments have been developed [1,2], serving as useful tools to track patients' health status and facilitate clinical decision-making [3]. In addition,

Study Design and Participant Identification
A national postal mail survey of general pediatricians and pediatric subspecialists in the U.S. was conducted between July 2011 and December 2013. Using a dataset provided by the American Medical Association (AMA) consisting of a random sample of 1000 general pediatricians and 8488 pediatric subspecialists from four census regions (Midwest, Northeast, South, West), we randomly selected 1000 physicians (400 general pediatricians and 600 subspecialists) to take part in this study. We purposely recruited more subspecialists over the general pediatricians because subspecialists often manage patients who are more severe and require complex interventions compared to general pediatricians. Therefore, it is important to collect sufficient samples of subspecialists to better understand their PRO implementation status.
Eligible study participants were general pediatricians or pediatric subspecialists who were active in practice in the U.S., either as residents, fellows, or attendees. We focused on seven pediatric subspecialty groups: cardiology, endocrinology, gastroenterology, hematology-oncology, nephrology, pulmonology, and rheumatology. We excluded pediatricians with unverified mailing addresses and those who were retired. Pediatricians were required to complete an informed consent form, as a prerequisite to complete the questionnaire. The Institutional Review Board at the University of Florida approved this study.

Survey Questionnaire
This study created a 60-item survey package for assessing PRO implementation issues. The content included the physician's demographic characteristics, practice background, interest in important PRO domains for assessment, and perceived barriers and areas of confidence in assessing PROs. Demographic characteristics included attained age, sex, and race/ethnicity. Practice background included years in pediatric practice since residency, practice setting, and proportion of pediatric patients whose PROs were assessed. PRO domains of interest included global, physical, emotional, social and spiritual well-being, pain, sleep, and fatigue.
In the survey package, 9 items measured demographic information, and 16 items measured issues relevant to PRO implementation with aspects of physicians' knowledge, attitude, confidence, and barriers to implementation. In addition, 1 item captured the current status of implementing PRO assessment. Participants were asked about their interest in particular PRO domains using a binary response category (interested or not interested). A five-point Likert scale (1 = strongly disagree; 5 = strongly agree), was used for items measuring physicians' confidence in using PRO instruments. Another fivepoint Likert scale was used to measure physicians' perceived constraints (1 = extremely; 5 = not at all), as well as the extent of implementation (1 = did not intend to assess PROs; 2 = have thought to assess PROs but unlikely in foreseeable future; 3 = plan to assess PROs; 4 = occasionally assess PROs; 5 = regularly assess PROs). We further categorized the implementation status for PRO assessment as "Yes" if the response scores were 4 or 5, and "No" for otherwise.

Data Collection
We sent a research packet via postal mail to eligible participants, followed by up to five reminders through telephone calls and/or postal mail if they did not respond to the invitation within three months. The research packet comprised the informed consent form, survey questionnaire, and a $20-value gift card as incentive. Participants could choose to return the completed survey along with the signed informed consent forms via postal mail, e-mail (scanned document) or facsimile, or to decline by not returning the survey.
We replaced selected participants who had invalid mailing addresses (per notifications of postal offices) by other samples randomly selected from the general pediatric or subspecialty group they belonged to. We excluded 55 selected participants from the original sampling frame (N = 1000) due to retired (N = 7), deceased (N = 1) or uncertain (N = 48) status, resulting in 945 confirmed eligible participants in the final statistical analyses.

Statistical Analysis
Chi-square tests were performed to examine the crude associations of important PRO domains for assessment and perceived barriers, respectively, with pediatricians' age, specialty, practice setting, and census region in which they practiced. Multivariable logistic regressions were used to examine the associations of pediatricians' perceived barriers and confidence factors, respectively, with the status of PRO assessment through two analytic models. Model 1 tested associations of an individual barrier and confidence factor with the status of implementing PRO assessment, controlling for demographic (age, sex, race), practice (specialty, practice duration and setting), and census region (South, Northeast, Midwest, West) factors. Model 2 used a stepwise backward selection to remove statistically nonsignificant barrier or confidence factors (p-value ≥ 0.2) from the analysis, controlling for aforementioned covariates. All analyses were performed using STATA 15. Statistical significance was set at p-value < 0.05 (two-sided). Table 1 shows the characteristics of the 458 pediatricians who returned the questionnaires (response rate 48.5% among 945 confirmed eligible pediatricians): 72% were non-Hispanic white; 62% were over 40 years old; and 38% practiced in the South region. For clinical background, 40.4% were general pediatricians, 15.9% were cardiologists, 15.5% were hematology/oncology specialists, and 8.1% were pulmonary specialists. In addition, 58.5% had been practicing for at least 10 years, and 55.4% practiced in an academic setting. Approximately 30.0% of pediatricians were currently implementing PRO assessment.

Implementing PRO assessment
Currently not being implemented † 327 (71.0) Currently being implemented ‡ 127 (29.0) † Did not intend to assess PROs, have thought to assess PROs but unlikely in foreseeable future, or plan to assess PROs. ‡ Regularly or occasionally assess PROs. Table 2 shows pediatricians' interest in specific PRO domains for assessment by age, specialty, and practice setting. Over 50% of pediatricians expressed a common interest across nine PRO domains. Of these, the top five domains commonly endorsed by pediatricians were emotional well-being (74.0%), global well-being (73.1%), physical well-being (67.0%), social well-being (59.4%), and pain (57.6%). Younger pediatricians aged 20 to 40 years had more interest in PRO assessment compared to those aged over 40 years, with statistically significant differences in the domains of global well-being (p-value = 0.001), emotional well-being (p-value = 0.001), social well-being (p-value = 0.002), school activities (p-value = 0.004), sleep (p-value = 0.005), pain (p-value = 0.013), and family functioning (p-value = 0.028). Pediatricians' interest in important PRO domains for assessment also varied by specialty. In contrast to other specialties, pediatric hematology/oncology specialties had more interest in assessing fatigue, pain, sleep, and spiritual well-being (all p-values < 0.05), whereas gastroenterology or nephrology specialties had more interest in assessing global well-being, school activities, and family functioning (all p-values < 0.05). General pediatricians reported lower interest in all PRO domains compared to other specialties. Pediatricians in academic settings had more interest in assessing all PRO domains (except emotional and social well-beings) compared to those who worked in private settings (all p-values < 0.05). Table 3 shows knowledge of and logistic barriers to PRO assessment by pediatricians' age, practice setting, and census region. The top five reported constraining factors were limited time and manpower (79.0%), limited training (77.4%), long length of PRO instruments (76.0%), lack of clinically meaningful cut-offs for scoring (75.5%), and lack of appropriate PRO instruments for use (75.0%). By pediatrician characteristics, those aged over 40 years reported more barriers to PRO assessment than aged 20-40 years. The statistically significant barrier factors included skepticism about the validity of PRO instruments (p-value = 0.002), inadequate reimbursement incentives for assessing PROs (p-value = 0.007), lack of evidence that PRO assessment improves care (p-value = 0.008), and unavailability of computerized modes for administering PRO assessment (p-value = 0.015).    Table 3 additionally shows that pediatricians who practiced in the South or Northeast encountered more barriers than those in the Midwest or West. The frequently cited barriers by pediatricians in the South were limited training in PRO instruments (82.0%), lack of clinically meaningful cutoffs (79.7%), and limited skills in scoring PROs (79.5%). There were notable differences in the experience of barriers across census regions, including poor knowledge of PRO concepts, scoring and interpretation, varying capabilities of children, unavailability of computerized modes of administration, lack of evidence that PRO assessment improves care, and skepticism about the validity of PRO instruments (all p-values < 0.05). Multivariable analyses suggest that pediatricians who were older (versus younger) in age and resided in the South and Midwest (versus West) encountered more barriers significantly associated with PRO assessment (Table S1).
Despite these barriers, 44% of pediatricians felt confident that PRO assessment provides more benefits to patients than relying on clinical judgement alone, and 40% of the pediatricians indicated that PRO assessment is compatible with their norms (Figure 1). However, only 26% of the pediatricians were confident in their ability to administer PRO instruments, and 20% expressed concerns about the availability of instruments that could accurately evaluate PROs. Multivariable analyses suggest that pediatricians who practiced in the private (versus academic) setting perceived less benefit and norm compatibility in assessing PROs (Table S2).
Children 2022, 9, x FOR PEER REVIEW 7 of 14 were notable differences in the experience of barriers across census regions, including poor knowledge of PRO concepts, scoring and interpretation, varying capabilities of children, unavailability of computerized modes of administration, lack of evidence that PRO assessment improves care, and skepticism about the validity of PRO instruments (all pvalues < 0.05). Multivariable analyses suggest that pediatricians who were older (versus younger) in age and resided in the South and Midwest (versus West) encountered more barriers significantly associated with PRO assessment (Table S1). Despite these barriers, 44% of pediatricians felt confident that PRO assessment provides more benefits to patients than relying on clinical judgement alone, and 40% of the pediatricians indicated that PRO assessment is compatible with their norms (Figure 1). However, only 26% of the pediatricians were confident in their ability to administer PRO instruments, and 20% expressed concerns about the availability of instruments that could accurately evaluate PROs. Multivariable analyses suggest that pediatricians who practiced in the private (versus academic) setting perceived less benefit and norm compatibility in assessing PROs (Table S2).     2). * p-value < 0.05; ** p-value < 0.01; *** p-value < 0.001. & Status of implementation: currently not implementing PRO assessment (do not intend to assess PROs, have thought to assess PROs but unlikely in foreseeable future, or plan to assess PROs) and currently implementing PRO assessment (regularly or occasionally assess PROs). † Multivariable logistic regression model estimated ORs for PRO assessment currently not being implemented (vs. implemented) associated with individual constraint and confidence variables by adjusting for pediatricians' demographic, practice and census region covariates. ‡ Stepwise backward multivariable logistic regression model estimated ORs for PRO assessment currently not being implemented (vs. implemented) associated with all constraint and confidence variables by adjusting for pediatricians' demographic, practice and census region covariates.
Using a parsimonious method with a backward variable selection, Table 4 shows significant barrier and confidence factors for not implementing PRO assessment, adjusting for covariates mentioned in the Methods section (Model 2). The significant barrier factors included limited knowledge of PRO concepts (OR 4.10; 95% CI 2.21, 7.60) and unavailability of appropriate PRO instruments (OR 1.87; 95% CI 1.01, 3.49), whereas the identified confidence factors included the assessment's compatibility with norms (OR 0.41; 95% CI 0.23, 0.72) and perceived benefit over clinical judgment alone (OR 0.53; 95% CI 0.31, 0.93).

Discussion
This is one of the largest studies to examine pediatricians' perceived barriers and areas of confidence associated with the implementation of PRO assessment. Although it is always a challenge to achieve high response rates from physician surveys, the approximately 50% response rate in our study is comparable to previous physician surveys [23][24][25][26][27][28]. Through a comprehensive survey of general pediatricians and pediatric subspecialists across the U.S., we were able to associate physician-related characteristics with the preferences for various PRO domains. In addition, we identified important knowledge, logistic, and resource barriers that affected the implementation of PRO assessment in routine pediatric practice. This study provides robust evidence for the addressing of PRO implementation issues in pediatrics.
We found that pediatricians who specialized in hematology-oncology, gastroenterology, or nephrology, or who practiced in academic settings, viewed the vast majority of PRO domains as more important versus those who were specialized in general pediatrics, cardiology, pulmonology, and endocrinology, or practiced in private settings. Compared to other specialties, hematologists and oncologists identified pain, fatigue, sleep, and spiritual well-being as important domains because these PRO issues are prevalent in pediatric cancer patients as a result of toxic anticancer therapies and a high stress disease processes [29][30][31][32][33][34]. In addition, hematologists and oncologists were interested in assessing cognitive functioning, which is another side effect resulting from central nervous system-directed therapies, especially prevalent in brain cancer patients [35][36][37][38].
Younger versus older pediatricians reported fewer barriers in PRO assessment. This finding reflects a shift in professional training received in medical education via which the topics of patient centeredness and patient-physician communication have been increasingly emphasized [39]. Younger physicians (including those in training) having fewer barriers against PRO assessment, suggesting that older physicians may have difficulty in instituting new assessment procedures. Expanding educational and organizational efforts (e.g., medical school curriculum, continuous medical education, the development of PRO champions within hospitals) may help to eliminate knowledge constraints and increase health professionals' confidence or perceived benefits of PRO assessment [39,40].
We found significant variation in geographic locations regarding different barrier factors of PRO implementation. Pediatricians who practiced in the South and Northeast regions reported more barriers compared to those in the West and Midwest. It is likely that varied clinical practice styles or culture, available resources for PRO assessment, and patient characteristics and preferences contribute to the regional differences. The significant barriers reported by pediatricians are in line with previous studies regarding the geographic variation of the assessment of PROs in clinical practice for adults (e.g., constraints in physician education and confidence, clinical organization factors, and specificity of PRO assessments) [40][41][42][43].
Independent of age or geographic location, pediatricians who identified constraints such as limited knowledge of PROs and unavailability of appropriate PRO instruments were significantly less likely to assess PROs. Despite the rapid growth of PRO instruments for use, real-world evidence on the implementation status of PRO assessment is still limited, which requires practical implementation guidance (e.g., integrative PRO data collection platforms and actional referral/treatment plans) [44,45]. We found that, although 45% of the pediatricians acknowledged the usefulness of PRO assessment in providing more benefits than clinical judgment alone, only 26% felt confident in their ability to administer PRO assessment. To remedy this issue, professional PRO or medical societies should recommend actionable, interpretable, and clinically meaningful implementation frameworks for pediatrics [46][47][48]. Special attention should be paid to the aspects of PRO instrument selection, score calculation and interpretation per the needs of pediatric subspecialties and patient populations. For example, the International Society for Quality of Life Research has developed a general guideline, which can be tailored by various pediatric societies for use [44]. Pediatric professional societies should provide recommendations regarding implementation strategies. A recent study found that providing financial incentives significantly increased the rate of PRO assessment [49]. Therefore, health insurance agencies may facilitate appropriate payment mechanisms to optimize PRO data collection, which is now considered an important component of value-based performance programs [50,51].
It is also critical to consider an integrative approach to systematically address barriers in assessing PROs by improving clinical workflow and patient accessibility. Currently, technology-based strategies, e.g., electronic PROs (ePROs), to address PRO implementation issues would be particularly useful as more patients and clinicians utilize mobile and electronic devices. The use of ePROs and computer-adaptive tests (CATs) to monitor PROs can increase the feasibility of PRO assessment remotely, reduce administrative burden, and increase the patient engagement in clinical care [52][53][54][55][56]. Moreover, ePRO platforms allow for tailored care delivery to patients and such information can be linked with electronic health records (EHRs) to facilitate "big-data" initiatives [55,57]. In pediatric settings, PROMIS ® and KIDSCREEN have developed free, user-friendly ePROs and CATs. The approach of seamlessly integrating ePROs and CATs with EHRs contributes to real-time clinical decisions.
One of the overlooked implementation issues that can impact PRO integration into clinical workflow is sustainability, especially for longitudinal PRO assessment. This issue is critical for children who are disabled or experience chronic conditions. By establishing meaningful cut-points for PRO domains, developing methods for interpreting score changes over time, and integrating PRO data onto EHRs, we can employ longitudinal data to predict deteriorating health conditions [57]. Developing standardized ePRO platforms will maximize PRO information to be utilized in a variety of care settings and timepoints [47,48,52,53,56].
This study has some weaknesses. First, representativeness of our findings may be limited. Our surveys were collected between 2011 and 2013, and the results may not reflect the contemporary practice of pediatric PRO assessments because modern technology (e.g., mobile technology for PRO data collection, integrating PRO reporting into electronic health record systems) may have been used to overcome some PRO implementation barriers in pediatric oncology. Second, our results may not be generalized to all pediatricians in the U.S. because the participating pediatricians were randomly selected from AMA's members. Future studies are warranted to confirm our findings by using other professional databases (e.g., American Academy of Pediatrics).
In summary, wide variations in the implementation of PRO assessment were found among a national sample of pediatricians. Pediatricians' preferences in PRO domains significantly differed by age, specialty, and practice setting. Barriers faced by pediatricians varied mostly by attained age and census region, rather than the health systems in which they were affiliated. To overcome these barriers, targeted interventions in disseminating education and training in PRO topics, developing useful PRO cut-points, promoting ePROs, professional society advocacy, advancing reimbursement strategies, and integrating ePROs with EHRs to deliver tailored longitudinal care to children are warranted.  Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Data Availability Statement:
Data is available upon request from the corresponding author.