Scoping Pharmacy Students’ Learning Outcomes: Where Do We Stand?

Background: The professional abilities of graduate pharmacists have been associated with pharmacy undergraduates’ educational settings and features. This study aimed to perform a scoping review on how students’ learning outcomes are achieved, including learning assessment strategies, focusing on current pharmacy practice education. Methods: Relevant keywords, e.g., “pharmacy practice”, “(students or undergraduates)” and “outcomes” were browsed in Public/Publisher MEDLINE, Scientific Electronic Library Online, Directory of Open Access Journals, and other relevant databases for recently published sources (2018 and 2017). Preferred Reporting Items for Systematic Reviews and Meta-Analyses criteria were followed to assure the scoping quality. All types of students’ learning outcomes were addressed for indexed publications in English, Portuguese or Spanish. Reviews, descriptive studies and commentaries were excluded. Study data are presented in tables comprising objectives, methods, number of participants and main research findings. Results: Overall, 100 studies were identified and 22 were selected. The selected studies were distributed into seven main topics: real practices (n = 9); active-learning strategies (n = 5); comparisons between different teaching pedagogies (n = 3); pharmacy curriculum (n = 2); and other evaluations (n = 3). Conclusions: Studies on pharmacy students’ learning outcomes are limited. Pharmacy undergraduates’ performance was dependent on the learning strategies and extension of syllabus implementation.


Introduction
In recent decades, there have been significant changes in the pharmaceutical market, such as the arrival of improved technologies, new Internet-based apps and information systems, and innovative medicines and medical devices. It is now possible to diagnose, monitor, advise and provide medical information to patients using mobile electronic-based support systems. Pharmacists are facing new societal and technological challenges, similar to other health professions, and this should be reflected in pharmacy education [1,2]. The International Pharmaceutical Federation (FIP) states that pharmacy and pharmaceutical sciences education "develops a competent, professionally educated pharmaceutical workforce (e.g., pharmacists' practitioners, pharmaceutical scientists, pre-service students and pharmacy support workforce) for a diversity of settings (e.g., community, hospital, industry and academia)" [3].
Currently, academies are required to assure that pharmacy students acquire professional and scientific skills that go beyond the traditional knowledge-centered models [4]. In the old-style learning models, teachers were responsible for transmitting information on a certain topic mainly in lectures, while students were required to listen and study materials, usually after the classes (teacher-centered

Materials and Methods
This scoping review was carried out in September 2018 (last online search in 17 September 2018). The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist and flow diagram were applied for quality compliance. These criteria are commonly used to report reviews, comprising an evidence-based minimum set of items that should be followed [24].
All outputs, such as the types of included/excluded studies, as well as qualitative analyses, were double checked by two independent researchers and discrepancies were solved by consensus. Repeated studies were automatically identified with Endnote Web (EndNote™) (Clarivate Analytics, Philadelphia, PA, USA), an online reference manager [25].

Studies Sources
The searched information sources were Public/Publisher MEDLINE (PubMed), Scientific Electronic Library Online (SciELO), Directory of Open Access Journals (DOAJ), ResearchGate, and the Cochrane Library. PubMed is maintained by the National Center for Biotechnology Information as a free resource, with more than 28 million citations [26]. SciELO contained 745,182 papers (at the search date), covering a collection of European and Latin-American scientific journals [27]. DOAJ is an online directory that provides access to more than 3500 articles of open access and peer-reviewed journals [28]. ResearchGate is a professional network for scientists and researchers, comprising over 15 million members [29], retrieved to also access grey literature. Finally, Cochrane Library is an online collection composed of six databases, including more than 7500 systematic reviews [30].
The present scoping review was focused on finding evidence from Latin-speaking countries but not exclusively, i.e., an international overall contextual perspective was kept. Thus, PubMed, ResearchGate and DOAJ were selected based on their extensive coverage, but also from a convenience point-of-view. SciELO was selected to assure the possible inclusion of studies written only in Spanish and Portuguese. Finally, Cochrane Library was screened to identify previous reviews on the present topic.

Searched Keywords
The searched keywords are presented as a string equation ["pharmacy practice" and ("students or undergraduates") and "outcomes"] for titles and abstracts. In addition, the Spanish and Portuguese translations of the searched keywords, ["práctica de farmacia" and ("estudiantes and/or universitarios") and "resultados or desenlaces"] and ["farmácia prática" and ("estudantes ou universitários") and "resultados or desfechos"], respectively, were screened for the possible identification of papers published in those Latin-based languages.
Using the broader terms, "(students or undergraduates) and outcomes", was justified considering that these allow for a wide-ranging search, thus intended to assure the inclusion of the largest possible number of studies in a relatively short period of time. On the other hand, the term "pharmacy practice" was selected to increase the specificity of the present scoping review, since pharmacy practice is one of the most relevant disciplines within the profession and schools of pharmacy, resulting from the fact that community pharmacies are the main source of employment in the pharmaceutical industry around the world [31]. Additionally, the FIP in collaboration with the World Health Organization (WHO) defines a good "pharmacy practice", as the "practice of pharmacy that responds to the needs of the people who use the pharmacists' services to provide optimal, evidence-based care" [32].

Inclusion and Exclusion Criteria
Studies published in English, Portuguese or Spanish, between 17 September 2016 and 17 September 2018, and addressing any type of pharmacy undergraduate students' learning outcomes, were included. Exceptionally, other studies, such as Poirier et al.'s [33], were purposively selected, even though pre-health professional students of pharmacy were included (i.e., not pharmacy undergraduates), since undergraduate students from other health professions were also enrolled [33]. A period of 2 years was conveniently defined to ensure the inclusion of the most recent studies and reports. Commentaries, reviews and descriptive studies, i.e., those without a section presenting results from further analysis of a data set were excluded. Due to the scoping nature of this review, some limitations have been identified in the group of selected studies, with study weaknesses globally appraised in the Discussion.

Results
Overall, 100 studies were identified: 80 from PubMed, 0 from SciELO, 0 from DOAJ, 20 from ResearchGate, and 0 from Cochrane Library. Twenty-two studies were selected as presented in Figure 1.

Selected and Excluded Studies
The number of selected (n = 22) and excluded (n = 75) studies per source is presented in Figure 1, defined through the PRISMA flow diagram [22]. The reasons for excluding these 75 papers were: reviews studies (n = 14), descriptive studies (n = 6), commentaries (n = 2), other language (n = 1), and other evaluations not covering students' outcomes (n = 52). reports. Commentaries, reviews and descriptive studies, i.e., those without a section presenting results from further analysis of a data set were excluded. Due to the scoping nature of this review, some limitations have been identified in the group of selected studies, with study weaknesses globally appraised in the Discussion.

Results
Overall, 100 studies were identified: 80 from PubMed, 0 from SciELO, 0 from DOAJ, 20 from ResearchGate, and 0 from Cochrane Library. Twenty-two studies were selected as presented in Figure  1.

Selected and Excluded Studies
The number of selected (n = 22) and excluded (n = 75) studies per source is presented in Figure  1, defined through the PRISMA flow diagram [22]. The reasons for excluding these 75 papers were: reviews studies (n = 14), descriptive studies (n = 6), commentaries (n = 2), other language (n = 1), and other evaluations not covering students' outcomes (n = 52).

Brief Content Analysis of the Selected Studies
The main study findings, including the year of publication, number of participants/students, methods, results, and conclusions are presented in Table 1 (n = 22 studies).
Seven main research topics were identified in the selected studies, as follows: Records identified through database search

Brief Content Analysis of the Selected Studies
The main study findings, including the year of publication, number of participants/students, methods, results, and conclusions are presented in Table 1 (n = 22 studies).
Seven main research topics were identified in the selected studies, as follows:
Other evaluations (n = 2). Table 1 also offers the geographic origin of the selected studies. To assess the educational impact of engaging second-year student pharmacists in active, direct patient care experiences in health system practice.

28
Setting: A skill-based four-week introductory pharmacy practice experience in health system practice.
Main outcomes: Students were required to complete skills checklists in pre-and post-surveys. Operational and clinical self-efficacy statements were evaluated, e.g., performing proper aseptic technique to compound IV admixtures and gather pertinent patient information from the medical record, respectively. Students also self-identified contributions to patient care. Findings: Significant outcomes were achieved: 81.8% of operational and 100% of clinical self-efficacy statements (p < 0.05), and positive perceptions of the program.
Students' skills were significantly enhanced. It is fundamental to assess data from the experiential education environment to further refine didactic curricula. Main outcomes: Qualitative information collected via in-depth interviews. Quantitative information about student learning and interprofessional interactions also collected. Findings: Strengths and areas needing improvement of APM were identified: some constraints were acknowledged by few students at one site, such as more delineation of expectations, initial support, and initial responsibility.
The APM model is suitable for providing a high-quality learning experience and qualitative results showed precisely the areas needing improvement. APM may be better than the traditional preceptor model for students' integration in practice. Supervisors should evaluate students' adaptation and provide educational programs at APM sites. Setting: Eight sessions on common aging conditions, chronic diseases, and geriatric syndromes (practice), once a month in multiple clinics. Students were placed in interprofessional teams with medicine, pharmacy, nursing and public health students (40% of pharmacy undergraduates); 3-4 students per team, following a minimum of two elders. Main outcomes: After each practice, students and patients were given a five-question Likert-scale survey. Students self-evaluated their communication, knowledge of elder-centered care, understanding of the importance of the clinical topic in delivering elder-centered care, observation of interprofessional collaboration or willingness to participate in future clinics. A similar survey was completed by elders. Findings: Students and patients' self-evaluations were positively rated.
These practices led to increased perceived knowledge, and improved attitudes and perceptions among students. In addition, self-worth, self-care, and enjoyment increased among the elders. Real-world training in geriatrics and interprofessional team-care of older adults is vital.

students and 20 staff
Setting: The IPCP involved the completion of educational modules, addressing patient visits, responding to phone calls, team-based case presentations, medication reconciliation activities, and student-led group diabetes education classes. Staff and students agreed on providing consistent patient education during 1 year in a family practice setting. A mixed methods study design was followed. Main outcomes: Results from several tools were obtained, e.g., Interdisciplinary Education Perception Scale (IEPS), the Entry-level Interprofessional Questionnaire, the Collaborative Practice Assessment Tool, and pre/post module knowledge tests. Diabetic indicators were HgbA1c, glucose, lipid, body mass index, blood pressure, and information on annual dental, foot and eye examinations. Qualitative data from focus groups with staff and students were also gathered. Findings: Students and staff significantly improved their knowledge on Team Dynamics and Tips for Behavioral Changes knowledge. Only HgbA1c and glucose levels showed a significant decrease. Interprofessional perceptions were higher at the beginning and did not change throughout the study.
Patients' outcomes improved in this family practice setting. In addition, students and staff benefited from this program.  Setting: Third-year students from 2010 and 2012 were enrolled. All activities were carried out in a fictitious health system, using virtual patients. Twenty-two cases were used in the fall semester and 54 in the spring semester. Main outcomes: Data from a survey applied to evaluate students' perceived confidence in clinical skills. Findings: Students' confidence in their clinical skills was improved. Students' knowledge on therapeutic principles and pharmacotherapeutic recommendations was reinforced.
Students were able to apply knowledge in simulated clinical settings, adding to an increase in their confidence in some of the topics.

Rivkin [47] USA 2016
To describe a student-centered teaching method used to introduce a pharmacist patient care process (PPCP).

85
Setting: A sample of first-year pharmacy students was selected to receive the PPCP. The PPCP was aimed at preparing students for taking medication history, learning to write Subjective Objective Assessment & Plan notes, and patients' information and drug-related problems assessment. Examples were the "deconstruction" of a patient case or reorganization of patients' story, as well as identifying a drug-related problem from the medication history. Main outcomes: Data from students' evaluations comprising multiple-choice examinations, online course evaluations, and the assessment of students' SOAP notes submissions. Findings: Mean exam question marks ranged from 3.7% to 18.8%, with sampled students' performance significantly better than the comparative cohort.
Teaching methods were effective. Consistent and systematic delivery of the PPCP may improve students' skills and confidence, offering a safety environment for introducing patient care into the pharmacy curriculum.   To design and implement an undergraduate course for pre-health professional students of pharmacy and senior undergraduate students from a variety of majors including pre-medical, pre-dental, nursing, exercise science, and the physical and biological sciences, using a variety of resources from the humanities.

22
Setting: Undergraduate course for pre-health professional students that used literature, films, and podcasts to promote students' discussion. Focused topics were public health, stigmatization, portrayals of health care providers, patient experiences, health care ethics, aging, and death and dying. A quasi-experimental design was followed. Main outcomes: Data from tasks of reflective writings, a formal written and oral presentation on a selected health-related book, and data from pre-and post-course surveys. Findings: Students' interpersonal skills improved, as well as their critical thinking, concerning different health care issues.
Humanities could excel in supporting indispensable patient care skill enhancement in students. Impact of resident instructors on student course performance was not educationally significant. Students' performance varied. Pharmacy residents should be taught to create quality learning objectives that help students to focus on learning the most important course content. To evaluate students' self-efficacy to provide cessation counseling for cigarette and hookah tobacco.

169
Setting: Training session in the College of Pharmacy on cigarette (82%) and hookah smoking cessation (16%). Main outcomes: Data from an email survey comprising the confidence in counseling and perception of knowledge, based on the Ask-Advise-Assess-Assist-Arrange follow-up (5A's) model. Findings: Students' self-confidence in counseling and perception of knowledge was higher for smoking cessation than hookah tobacco.
Training in tobacco cessation is still desirable. Pharmacy students need further training to provide counseling on alternative tobacco products.

Gillette et al. [52] USA 2016
To characterize how independent variables predicted students' performance on the Pharmacy Curriculum Outcomes Assessment (PCOA) during 1st to 3rd professional years.

Not described
Setting: All students at Marshall University School of Pharmacy participated in surveys, including the PCOA, a national examination used to measure the academic progress of pharmacy students. The PCOA is composed of 4 domains: basic biomedical sciences (16%); pharmaceutical sciences (30%); social, behavioral, and administrative sciences (22%); and clinical sciences (32%). Main outcomes: Data obtained from the Pharmacy College Admissions Test (PCAT), the Health Science Reasoning Test (HSRT), and the PCOA (as a target variable). Findings: PCAT, HSRT, and cumulative pharmacy grade point average were significant predictors of a higher PCOA.
Admission criteria and performance while studying pharmacy were associated with a higher score in PCOA.

Discussion
Few studies on students' learning outcomes were identified and selected for this scoping review, which may reflect the limited global research on the present topic. Moreover, no pharmacy higher education institutions, based in countries speaking Spanish or Portuguese, have recently reported their pedagogical innovations concerning pharmacy practice and learning outcomes, although, in Brazil, only 529 pharmacy degrees are officially registered (2016 data) [53].
Most of the selected studies were non-controlled, non-longitudinal, and non-multicentric. None of the selected studies was representative of a certain national population, and the number of enrolled students was very heterogeneous in the included studies. Some studies presented methodological flaws (e.g., no description of sample size) that are critical when interpreting study findings, probably excluding the study from systematic reviews. Additionally, different instruments were applied to collect information on students' outcomes, such as skills checklists, in-depth interviews, agreement level on Likert scales for different constructs (i.e., tools on self-perceived knowledge, attitudes, communication, interprofessional practice, interdisciplinary education or confidence), or multiple-choice examinations. Contrary to what would be desirable, only some studies have applied pre-and post-surveys, and the number of studies enrolling multiple evaluations was also limited [5,[34][35][36][37][38][39][40][41][42][43][44][45][46][47][48][49][50][51][52]. On the one hand, all of these limitations compromise the accuracy and representativeness of the present study findings and conclusions; on the other hand, this clearly shows the need to improve the robustness of learning outcome evaluations in this area.

Real Practice
The studies that evaluated undergraduates' outcomes in real practice were the most frequently identified, which may reflect the relevance of evaluating students' performance in actual practice settings [34][35][36][37][38][39][40][41][42], moving from competence acquisition to suitable professional activities (EPAs) [54]. The studies were implemented in health systems, such as medicine services or pharmacies. Around half of these studies did not include the simultaneous evaluation of patients' outcomes, which may have contributed to an incomplete analysis of students' learning superiority and performance [34][35][36][37].
Undergraduates' skills, or perception of interprofessional experiences, attitudes and knowledge seemed to improve in real practice [34,35,38,39], although qualitative and quantitative data on students' performance should be collected during these programs to monitor their progression and adaptation. Particularly, adaptation problems, such as the need for more initial support or delineation of expectations were detected. In addition, students' intervention seemed to improve patients' outcomes, such as (i) self-worth, self-care and enjoyment in elder-centered care; (ii) hospital readmissions; or (iii) the health of diabetic patients [38][39][40][41][42]. These findings support the movement towards core EPAs being implemented for new pharmacy graduates in the USA today [54].

Active-Learning Strategies
Students from different course years benefited from participating in active-learning training sessions, such as case scenarios, simulated clinical settings, or role-plays. In some studies, a training/video session or a lecture was provided before the initial laboratory sessions, thus improving students' prior knowledge. Active-learning strategies successively enhanced students' self-confidence, identification of drug-related problems, patient management or communication skills [43,[45][46][47]. The use of virtual reality and avatars is advancing [55] and has been tested in Portuguese pharmacy undergraduate programs with promising results [56].

Comparisons between Different Pedagogies and Teaching Models
As expected, flipped models have increased students' performance and knowledge (on pain management), but no advantages seem to be gained by employing team-based learning in comparison with lectures, at least in one study [5,48], although this is not certain [57]. Interestingly, another study has demonstrated that it is possible to increase external dissemination, i.e., the publication of more posters or papers, while developing and implementing optimized practice models and prescribing patterns. This new co-curricular model was composed of diverse courses on research/didactics, research catalogues or practical and experimental projects [49].

Pharmacy Curriculum: Design of New Courses and Topics
Concerning the pharmacy syllabus, only one innovative topic paper and one methodology study were presented within the selected studies. One study focused on the necessity of introducing Medicinal Chemistry into the pharmacy syllabus, as a way of enhancing students' critical thinking and therapeutic decision-making skills; the other study highlighted the importance of integrating humanistic methodologies in a course for pre-health care students to enhance students' critical thinking, thus supporting patient care skills [33,50]. There are international guidelines to design quality pharmacy syllabi, including the introduction of useful changes in existing ones, thus inspiring the progress (and reporting) of excellent pharmacy education [58].

Pharmacy Residents as Tutors, and Other Evaluations
The impact of pharmacy residents as instructors seemed to be less educationally relevant, suggesting previous training, namely, on the construction of quality learning objectives [12]. It was also detected limited training in tobacco cessation programs. In addition, possible predictors of students' performance on the national pharmacy examination were also discussed [51]. Nevertheless, near-peer and peer-led education in different settings has produced good outcomes [59,60]. The last study in Table 1 was the only one applying multivariate regression models, and it concluded that admission criteria and academic performance were the only predictors of learning outcomes [52], suggesting a conservative approach to education attainment.

Limitations of the Scoping Review
As an initial scoping review, the number of included databases and keywords was narrow, which may have limited the present findings. It is not possible to assure the representativeness of all global research published regarding this topic in the considered time interval.

Practical Implications
Although knowing all the limitations when interpreting data from the selected papers, present findings point to benefits emerging from a re-evaluation of curricular programs, at least in the USA. Integrating more real practices and active-learning strategies in pharmacy schools may bring significant benefits, since they may have a direct impact on patients' health. The provision of educational programs, integrated in real or active-learning practices should be considered; these programs are probably more beneficial if applied from the time of admission, and not towards the end [35][36][37]43,44,48]. The application of flipped and team-based models may be more useful than the traditional expositive models, such as lectures, despite the fact that implementation and outcomes require monitoring [5,48]. If needed, it is possible to increase external dissemination, i.e., students' scientific outputs, through specific curricular models [49]. Innovative humanistic methodologies may also be systematically considered to increase students' multicultural background, arguing capacity, and critical thinking [33]. This would be particularly interesting to see in place, knowing that most programs provide clinical training but without necessarily preparing graduates to deal with, e.g., human suffering [61]. Alternative tutors, such as pharmacy residents, should be trained before teaching pharmacy students [12] and could benefit from medical humanities education [51]. The inclusion of a national examination in pharmacy schools to measure the academic curriculum outcomes may be relevant to monitor students' knowledge [52], especially in other countries besides the USA.

Limitations of the Selected Studies, Future Research and Final Remarks
Additional research on the present topic is recommended, given the social and cultural discrepancies between different educational settings, including anthropological studies and the construction of predictive models. Consequently, the development and validation of standardized tools to evaluate students' outcomes would be beneficial to a global workforce. The impact of students' intervention on patients' outcomes should be evaluated in groups of chronic patients in prevalent diseases, but also those enrolled in lifestyle and behavioral change programs in real practice. It is especially relevant to develop and conduct representative, longitudinal, controlled and multicentric studies, specifically, to design and implement political decisions in pharmacy education.
Considering that some schools of pharmacy are implementing innovative interventions within their curricula, these schools should be encouraged to publish their pedagogies. Additionally, the publication of updated national and international guidelines on teaching/learning strategies is recommended. The findings presented here have limited value to an update of pharmaceutical education guides. The selected studies in this scoping review showed several critical limitations, including a reduced number of participants per study; insufficiently described study designs and/or study protocols; lack of study outcomes monitoring; or not describing the professional experience of the enrolled professors (e.g., number of years). In fact, professors' performance was not rated or evaluated by the researcher (e.g., other professor). In this sense, governing bodies, such as Center for the Advancement of Pharmacy Education (CAPE) or Accreditation Council for Pharmacy Education (ACPE) may plan or carry out larger studies or reviews to generate updated guidelines and regulations on teaching/learning strategies [14,23]. Besides monitoring and registering undergraduates' progress, schools of pharmacy should keep track of the professional outcomes and/or knowledge of their graduate pharmacists [17,18].
The teaching/learning strategies identified here seem to have improved students' learning outcomes. Overall, a positive correlation between a certain teaching strategy and a certain educational outcome seems to exist. However, additional studies are required to check this assumption, knowing, for instance, that perception of interprofessional experiences, attitudes and knowledge of undergraduates seemed to improve in real practice [34,35,38,39].

Conclusions
Studies evaluating undergraduates´learning outcomes are limited, with only a few specific themes and settings being described. As expected, pharmacy students' performance and quality assessment were dependent on the teaching strategies and syllabus implementation. Further studies are recommended.