Predicting the Success of International Pharmacists in a Distance-Based US Doctor of Pharmacy (PharmD) Program: Results from a 5-Year Cohort

Background: To establish the predictors of success in an international-trained PharmD (ITPD) program between admission criteria and academic performance. Methods: The primary outcome of this study was the correlation of admission criteria with didactic and experiential grade point averages (GPA) for the first 5 years. Candidates meeting the minimum criteria completed a competency exam or the US-Foreign Pharmacy Graduate Equivalency Exam (US-FPGEE). Tests of English language proficiency (TOEFL(R) and ACTFL’s Oral Proficiency Interview) plus interview with faculty, students, and alumni were also required. Scores were correlated with both didactic and experiential GPAs. Results: The 23 students admitted to the ITPD program had a cumulative GPA of 3.72. There was a significant correlation between total admissions score and the median pharmacy and healthcare course category GPA (ρ 0.53), but not other categories. The composite TOEFL did not predict any performance but TOEFL writing and speaking did correlate with advanced pharmacy practice experience (APPE) performance. The OPI scores were associated with higher GPAs overall, in advanced integrated clinical sciences, and APPEs. The admission interview scores consistently and significantly correlated with preceptor-rated APPE GPA, practitioner skills, and professionalism (ρ > 0.5; p < 0.05). Performance in early courses significantly predicted the performance in advanced courses and experiential performance (ρ 0.48–0.61). Conclusion: The correlations between early and late course performance demonstrated the cohesiveness of this program. Further study is needed between the predictors of success using non-cognitive admission criteria.


Introduction
As the profession of pharmacy continues to evolve, it becomes imperative that pharmacy education considers approaches that diversify the profession, globalize pharmacists' patient care, and supply advanced pharmacy practitioners that meet the needs of local healthcare communities across the world [1,2]. Although the pharmacy profession is undergoing robust global growth, the need for well-trained guides for supporting access to the use of quality medicines worldwide is high [3]. These disparities are magnified where advanced training is necessary in order for pharmacists to participate in roles related to direct patient care. In response, multiple pharmacy organizations are offering guidance to the profession regarding career advancement, workforce development, and educational methods [3,4]. Yet, disparities remain between those institutions that have already developed resources for advanced training for the provision of pharmacists' patient care and those in need of trainers for this type of education [5]. Further study is needed regarding the effective implementation of educational programs that embrace the advancement of pharmacists' patient care on an international scale, and predictors of trainee success in these programs.
The Accreditation Council for Pharmacy Education (ACPE)-accredited International-Trained PharmD (ITPD) program was established in 2014 at the University of Colorado Skaggs School of Pharmacy and Pharmaceutical Sciences (CU SSPPS) to address the need for pharmacists' patient care training internationally. This advanced-standing entry-level Doctor of Pharmacy (PharmD) program provides a combination of didactic and experiential course work designed for pharmacists educated and practicing outside of the United States. The overarching goal of this program is to encourage the advancement of pharmacists' patient care in the student's home country.
In accordance with the 2016 ACPE accreditation standard 17 appendix 3, PharmD programs should conduct correlations between admission variables and academic performance [6]. In addition, prior studies have suggested that admissions processes should be tailored to predict academic performance in both the didactic and the experiential settings [7]. Given the design of the ITPD program, and the global backgrounds of its students, the purpose of this study is to establish predictors of academic performance and student success in the ITPD program by examining the relationship between admission criteria and performance in both didactic and experiential training. By assuring student success, this program is designed to meet healthcare disparities by generating practice leaders and pharmacy educators who can advocate for the healthcare needs of their respective communities.

Curricular Design and Admission Criteria
The ITPD curriculum of the CU SSPPS was designed as an advanced-standing entrylevel PharmD curricular pathway for practicing pharmacists from around the world who aim to advance pharmacists' patient care in their local healthcare communities. While a separate pathway, it is aligned with the school's traditional PharmD pathway by sharing abilitybased outcomes, faculty members, and a majority of the didactic content. The curriculum includes 90 semester credit hours delivered in a hybrid format. This encompasses online didactic classwork and two in-person sessions of interactive didactic classes that correspond with elements of the experiential curriculum. It also includes in-person introductory pharmacy practice experiences (IPPEs), an advanced introductory pharmacy practice experience (aIPPE), and advanced pharmacy practice experiences (APPEs). As the program is designed to be flexible, time from program admission to completion spans 3 to 6 years. Prior publications have detailed success in tailoring educational methodologies to learning outcomes in this program [8,9].
Advanced standing into the program is assured through requiring a baccalaureate degree (or higher) in pharmacy and that candidates have practiced as a pharmacist for at least one year. Applicants also have to successfully pass two foundational competency exams in biomedical and pharmaceutical sciences. These exams align with coursework commonly covered in baccalaureate pharmacy programs and the first and second years of the CU SSPPS entry-level Doctor of Pharmacy Program. They include topics such as: biochemistry, cell biology, mechanisms of disease, pharmaceutics, medicinal chemistry, microbiology, pharmacology/toxicology, pharmacokinetics, drug design and drug action, immunology, and basic anatomy. In place of these exams, applicants may demonstrate competency via the passing of the US-Foreign Pharmacy Graduate Equivalency Exam (US-FPGEE).
The ITPD applicants are required to demonstrate English language proficiency through the Educational Testing Service's Test of English as a Foreign Language (TOEFL ® ) exam and the American Council on the Teaching of Foreign Languages (ACTFL) oral proficiency interview (OPI) exam. Upon meeting these qualifications, applicants are invited to videoconference interviews with multiple faculty members, a current ITPD student, and an ITPD alum. Interviews assess methods of learning and researching, critical thinking and decision making, adhering to principles and values of the profession, commitment to pharmacists' patient care in their home country, and communication and interpretive abilities (assessing the personal and professional domain). These were assessed using a quantitative and qualitative standardized rubric. Scores on each category ranged from 0 to 3 and 0 to 9 through an internally created rubric, with the final interview score calculated as a percentage of possible points, and the multiple interviewer scores averaged. In addition to numeric score, each interviewer provided a qualitative rationale for scoring as well as a qualitative overall impression of each candidate, which were reviewed with candidate scores through our admissions committee.
The admissions score was derived from our admissions process for all our programmatic pathways and was in alignment with ACPE admissions standards regarding the standardized interview process [6]. The admission score was calculated using a sum of the above-mentioned criteria, with TOEFL composite and OPI scores each weighted at 20 points, and the foundational competency exams percentage weighted as 20 points for the biomedical sciences and 30 points for pharmaceutical sciences. If the FPGEE was used in place of the competency exams, the passing percentage was calculated on a 50-point scale. The mean interview score was valued at 100 points. Additional points were offered for preferred criteria of Board of Pharmacy Specialties (BPS) certification, employment at a Joint Commission-accredited institution, being a faculty member at a school/college of pharmacy (all 5 points each), and residency or fellowship training (10 points each).

Outcomes and Statistical Analysis
This study was approved by the Colorado Multi-Institutional Review Board (COMIRB) as a secondary research study with a waiver of informed consent. The primary outcome of this study was the correlation of individualized student quantitative admission criteria with academic performance (both didactic and experiential) completed for the first 5 years of the program's offering with analysis completed at year 5 of the program's offering (Summer 2019). The admission criteria evaluated included a cumulative admissions score, TOEFL scores (composite, reading, listening, speaking, and writing), OPI scores, FPGEE scores/performance on biomedical and pharmaceutical sciences competency exams, and interview scores. The academic performance criteria included mean cumulative programmatic grade point average (cGPA) on a 4.0 scale and median grade point average (GPA) within major didactic course categories (detailed in Table 1). These course categories were created on the basis of the required elements of the didactic curriculum for Doctor of Pharmacy degrees as outlined by ACPE [6] with further delineation of major categories into subcategories for the purposes of analysis. Within the experiential curriculum, IPPE and APPE GPA data were included for correlation, along with preceptor assessment of practitioner skills, communication, and professionalism, which determined grades on APPE rotations using previously validated methods [10] and were used to identify potential areas for improvement for skillsets within the didactic curriculum. The secondary outcomes included correlations between academic performance and years of pharmacy practice experience and the comparison of GPA between students employed or not employed at a joint commission accredited facility or presence of a prior graduate degree. Admission criteria were correlated with individual course performance, and the correlation between foundational and advanced course category performance was assessed to determine if performance in foundational courses predicted performance in advanced didactic and experiential courses.
Since a majority of the data were assumed to be non-parametric in nature, a Spearman's rank-order correlation test was used to assess the strength and significance of each correlation. The strength of correlation was expressed in ρ values, with 0.10-0.39 considered to be a weak correlation, 0.40-0.69 moderate, 0.70-0.89 strong, and >0.90 as very strong [11]. Correlations were considered significant if the p-value was less than 0.05 (95% certainty that correlation did not occur due to chance). Correlations that did not express a monotonic relationship were excluded from the analysis. Other comparisons (joint commission employment, prior graduate degree, and academic performance) were conducted using the non-parametric Wilcoxon rank-sum test using the same threshold for statistical significance. All statistics were conducted using JMP Pro Statistical Software (© SAS Institute, Cary, NC, USA).

Cohort Characteristics and Overall Course Performance
Twenty-three students had been admitted to the ITPD program in its first five years (2014 to 2019). Students earned their original pharmacy degree from 14 unique countries outside of the United States. At the time of the writing of this manuscript, 21 students have graduated and 4 are completing APPE rotations. Two students were unable to continue the program due to non-academic reasons and they were not included in the analysis. At admission, six students had a graduate-level degree and four were employed at Joint Commission-accredited healthcare facilities. The applicants had a mean of 5.5 years (±SD 4.73) of prior pharmacy practice experience.

Correlations between Admissions Criteria and Didactic and Experiential Course Performance
Full results for the assessment of the primary outcome are presented in Table 2 (didactic course performance correlations) and Table 3 (experiential course performance correlations). Within these results, there was a significant correlation between the total admissions score and performance in the pharmacy and healthcare course category (ρ 0.53, p = 0.01) but not cumulative cGPA or other course categories ( Table 2). Although the TOEFL composite scores were not significantly associated with cGPA or any major course category GPA, TOEFL speaking scores were significantly correlated with APPE GPA (ρ 0.46, p = 0.03) and cGPA (ρ 0.48, p = 0.03), and TOEFL writing scores were significantly correlated with fICS GPA (ρ 0.47, p = 0.03) and APPE practitioner scores (ρ 0.55, p = 0.01). Both the TOEFL reading and listening scores, however, were not correlated with any didactic, experiential, or cumulative GPAs. The OPI scores were associated with a higher aICS GPA (ρ 0.48, p = 0.02), APPE GPA (ρ 0.55, p = 0.01), and a higher cGPA (ρ 0.49, p = 0.02). The interview scores were not significantly correlated with any didactic course categories or cGPA but were the strongest predictor for nearly every aspect of APPE, but not IPPE, experiential assessment. The significant correlations included interview scores and APPE GPA (ρ 0.56, p = 0.01), APPE practitioner skills (ρ 0.50, p = 0.02), and APPE professionalism scores (ρ 0.51, p = 0.02) and a non-significant trend toward APPE preceptor rating of communication (ρ 0.39, p = 0.09).

Correlations between Foundational Competency Exam, FPGE, and Academic Performance
When considering assessments of advanced standing, the foundational competency exams showed limited correlation with academic performance. The performance on the biomedical sciences competency exam had no correlation with any didactic or experiential course, or cumulative GPA, while the pharmaceutical sciences competency exam scores had a significant negative correlation with APPE preceptor ratings of a student's practitioner skills (ρ −0.67, p = 0.01) and communication (ρ −0.56, p = 0.03). Of students who took the FPGEE (n = 5), there was a very strong and significant correlation between FPGEE score and foundational and integrated sciences GPA (ρ 0.92, p = 0.03) and cGPA (ρ 0.97, p = 0.01). The interview scores were not significantly correlated with performance in any major didactic course category GPA or cGPA. In addition, performance on the biomedical sciences and pharmaceutical sciences competency exams did not predict cumulative GPA or course category GPAs.

Predictors of Performance within the Program
Assessment of secondary outcomes demonstrated that years of past experience had no correlation with GPA of any major didactic or experiential course categories with the exception of a negative correlation between past experience and the pharmacy and healthcare GPA (ρ −0.43, p = 0.05). The presence of a previous graduate degree (p = 0. 16) or employment at an international joint commission certified health care facility (p = 0.18) was not associated with a significant difference in cGPA. However, students who had prior graduate degrees performed significantly better in the aICS course category compared with those who did not have a prior graduate degree (3.96 median GPA vs. 3.58, p = 0.04). A similar trend was noted among the pharmacy and healthcare course category and prior graduate degree (4.0 median GPA vs. 3.67, p = 0.01). Conversely, students employed at joint commission certified facilities performed significantly lower in advanced ICS courses vs. those who were not employed at such an institution (3.34 median GPA vs. 3.75, p = 0.05). Employment at an international joint commission certified facility or the presence of a prior graduate degree did not significantly predict any experiential grade performance (p > 0.05).

Individual Course Analysis
In terms of individual courses, total admissions score also predicted higher performance in the infectious diseases course (ρ 0.56, p = 0.01), interprofessional ethics (ρ 0. OPI score significantly predicted higher performance in both the pharmacotherapy of endocrine, hematological, pulmonary, urological disorders and women's health course (ρ 0.45, p = 0.04) and the clinical capstone course (ρ 0.47, p = 0.03).

Assessment of ITPD Program Cohesiveness
Once admitted to the program, higher performance in the ITPD foundational course categories significantly predicted success in advanced courses and the experiential curriculum ( Table 4). The performance in foundational integrated sciences significantly predicted performance in integrated clinical sciences courses (ρ 0.73, p < 0.01) and advanced integrated clinical sciences (ρ 0.64, p < 0.01) Similar significant correlations were noted for the pharmacy and healthcare courses and pharmacotherapy courses (ρ 0.63, p < 0.01) as well as the advanced pharmacotherapy courses (ρ 0.53, p = 0.014). Significant correlations were also noted between the communications and informatics courses and pharmacotherapy courses (ρ 0.8, p = ≤ 0.01), and the advanced pharmacotherapy courses (ρ 0.66, p < 0.01). IPPE experiential GPA was significantly correlated with performance in the course categories of foundational integrated sciences (ρ 0.46, p = 0.04) and advanced integrated clinical sciences (ρ 0.46, p = 0.04). The performance in all major didactic course categories correlated well with APPE rotation performance (ρ 0.48-0.61, p < 0.02 for all categories).

Discussion
Establishing effective admission criteria is essential to the success and function of a pharmacy program. Prior pharmacy school admissions studies have shown that traditional cognitive measures (e.g., achievement tests, prior GPA) had strength of correlation of less than 0.30 with didactic performance [7,12,13]. Admissions variables in the cognitive domain are even less predictive for performance in healthcare experiential programs, suggesting the need to bolster measures of non-cognitive criteria such as communication, empathy, citizenship, ethical behavior, commitment to leadership, and diversity [7]. Creation of the multiple mini-interview process has advantages in that it increases the validity of interview scoring in a multicultural context, which overlaps with many aspects of our institution's interview processes. As the diversity of pharmacy programs continues to grow, a multifaceted and multicultural admissions approach that integrates aspects of non-cognitive criteria in addition to cognitive measures will become imperative for predicting student performance and respective areas for targeted improvement within an admitted cohort of pharmacy students.
Our results suggest that the select aspects of our admissions criteria predict performance in specific aspects of the curriculum as opposed to a holistic prediction of cumulative GPA in an international group of pharmacy students. For example, there was a very strong and significant correlation between performance on standardized testing (FPGEE) score and GPAs in the program. Our non-cognitive criteria, such as interview scores, were one of the strongest factors predicting success in the experiential component of our program. This reflects the need for assessment of these "soft" non-cognitive skills as part of the admissions process and substantiates prior literature suggesting these criteria may better predict experiential performance [7]. Importantly, these held true in this global population with varied pharmacy education and professional backgrounds. Prior studies have also demonstrated pre-pharmacy, science-related GPAs are less likely to correlate with PCAT and NAPLEX performance as opposed to didactic GPA in pharmacy school, supporting our findings of a lack of correlation between our basic science admission exams and pharmacy school GPA once our students entered the program [14]. This study also demonstrated that early pharmacy school GPAs predict late pharmacy school performance (PCOA, NAPLEX) which is similar to our findings that demonstrated that strong early program performance predicted strong late program performance [14]. Other studies have demonstrated a stronger correlation between pre-pharmacy GPA and programmatic success, but students were also more likely to be successful with pre-pharmacy BS degrees and advanced biology course work which is similar to our cohort of ITPD students [12] but these results may not strongly predict NAPLEX performance [15]. The results of this study have informed our program of the importance of maintaining a multi-panel interview during the admissions process, has emphasized the importance of maintaining the OPI for candidates that require this interview, and has provided information to our programs regarding establishing TOEFL cutoffs for our admissions process as well as establishing needs for early interventions in order to improve experiential readiness.
The reasoning for the variability we observed in our correlations may be multifaceted. These results may reflect the complexity of the profession, requiring clinicians to operate on high levels of critical thinking, data integration, and communication to successfully practice. Results from a previous study have suggested testing of English language understanding does not predict academic success within a program in non-native English-speaking students. That study is limited in scope as it solely evaluated composite TOEFL and non-native English-speaking students who came from only one geographic region [16]. Our results are from a more diverse cohort and support the importance of English proficiency as students progress to courses that require greater critical thinking, data integration, and interprofessional communication, such as in our aICS and experiential courses.
Our results also suggest that, overall, our weighted admissions score was unable to predict success across all aspects of our program. Yet, our broad criteria are able to identify global pharmacy professional applicants who can succeed in a US-based PharmD program. This may be reflective of not only our admission criteria, but also the common foundational pharmacy education and motivation of the individuals successfully admitted. The correlations between both early and late didactic courses reflect the educational cohesiveness of our program. Some of the weaker correlations between didactic coursework and experiential performance suggest that skillsets in the classroom may not fully translate to a practice-based setting, but predictive ability did remain valid for major course categories. This is consistent with prior research suggesting that targeted assessments and situational judgment testing are more likely to predict experiential readiness than GPA or pre-pharmacy performance (PCAT or pre-pharmacy GPA) [17].
This study is met with several limitations. We could not perform a multivariable regression due to the smaller sample size of this study, prohibiting the number of predictor variables for an effective regression model and as such, our correlations should be consid-ered associative in nature. Secondly, because ITPD students performed exceedingly well in the program, there was a small degree of variance in cumulative GPA. This is mirrored by prior studies showing exceptional performance of non-traditional PharmD pathways [18]. This, combined with limited power, may have prevented statistical significance of many of the correlations seen in this study. In addition, the multiple comparisons conducted for this study increase the likelihood of experiment-wise error and may have confounded some of the significant findings in this study. However, the emphasis of this study was to examine the predictors of programmatic success in an international cohort of learners, and this study was designed to be as comprehensive as possible to delineate potential targets for programmatic improvement. We acknowledge that these results present signals for further potential research and programmatic change but are associative in nature. Although our admissions standards and scoring rubrics have been used successfully at our institution, indicate programmatic success, and are based on prior publications [12][13][14][15][16][17][18], validation of our rubric and admissions process external to our institution would improve the external validity of our results and may explain lack of correlation between our cumulative score and specific academic performance. As previous studies have outlined the profound impact of a PharmD degree on nontraditional graduate careers, we look forward to following this cohort of students and their career achievements as they contribute to patient-centered care worldwide.

Conclusions
This study demonstrated that specific aspects of the TOEFL, OPI, and other admission criteria predicted academic success in didactic coursework for the ITPD program. This study adds to the literature regarding predictors of programmatic success in the didactic and experiential setting in a multicultural student population. Further research is needed regarding the predictors of academic success in healthcare programs with international student populations. Informed Consent Statement: Patient informed consent was waived by IRB due to the retrospective nature of this study.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author. The data are not publicly available due to protection of student privacy.

Conflicts of Interest:
The authors declare no conflict of interest.