Tips for Developing an Integrated Online and Simulation Course Based on 6-Years of Experience

Technological advancements, changes in pharmacy students’ learning preferences, and increased educational costs have necessitated the development and implementation of innovative teaching modalities. The University of Pittsburgh, School of Pharmacy has been using simulation-based learning throughout the curriculum for several years. To further advance this practice, a novel course was designed to teach students new concepts through online video lectures, slide sets and quizzes, and knowledge application during weekly practica time involving multiple patient cases taught with high fidelity simulation. While this course has been well received by students, it does require resources, organization, and time for development. In this article, we describe our experience developing, modifying, and sustaining this blended course with the hope that sharing our experiences over the past six years will lead to expediting successes at other institutions. Tips for success such as keeping online segments short, holding students accountable, thinking of simulation approaches beyond the mannequin, and developing standardized assessment tools are discussed. Overall the blended course of online learning and simulation is a unique educational experience akin to real-world pharmacy practice and is worth the effort with a goal of optimizing learning.


Introduction
Today's pharmacy students benefit from multiple learning modalities to enhance engagement in their education and to prepare them adequately to enter the workforce as pharmaceutical care providers (standard 25.7) [1].Known as Generation Y or Millennials, most current students are deemed "digital natives" because technology has been an integral part of their daily lives from a young age.As a result, some have theorized that their learning preferences are vastly different than previous generations who were taught by traditional means, such as the didactic lecture [2].While there are opponents to this theory, it is generally agreed that education must evolve to meet the needs of current and future students [3].
Online learning has become popular among institutions of higher education as a way to provide quality education, while attempting to utilize faculty resources efficiently and reduce cost increases [4].Within pharmacy education, online learning has proven to be an effective educational tool for a variety of topics, such as substance abuse and diversion, basic statistics, as well as proper inhaler technique [5][6][7].Recently, a school of pharmacy employed a hybrid program when opening their satellite campus, which combined online learning with traditional face-to-face instruction.Student performance at the new satellite campus was compared to that of students at the main campus, who received only traditional education techniques.The results showed no differences between groups when comparing overall course scores, grade point average, and preceptor evaluations [8].Simulation-based learning experiences have provided similar results.When compared with lecture style education and other interactive learning techniques, simulation-based learning produced better quiz results and students reported increased satisfaction [9][10][11].
In order for students to become life-long learners, a requirement of any healthcare professional, the 2011 Accreditation Council for Pharmacy Education (ACPE) standards recommend fostering adult learning concepts from an early stage (standard 11.1) [1].Adult learning places the burden of assessing current knowledge and learning needs, as well as developing a plan to meet those needs, solely on the student [1].Quality pharmacy education is not simply providing students with a solid foundation on which to enter the workforce, but teaching students how to advance their knowledge base post-graduation.Introducing these adult learning concepts earlier into the pharmacy curriculum better prepares students for advanced pharmacy practice experiences, post-graduate training, and to be successful practicing professionals.This is evident in the 2009 report by the American Association of Colleges of Pharmacy (AACP), which states classroom time should no longer be dominated by didactic transfer of information from instructor to pupil [12].Additionally, ACPE increasingly recommends the use of practice-based exercises in pharmacy curriculum, which includes computer and simulation-based instructional tools (standard 11.2) [1].The application of technologies meets the adapting needs of adult learners.
The University of Pittsburgh School of Pharmacy introduced a dual-immersive course combining online education with high-fidelity patient simulators in 2008.The course was designed to improve students' abilities in the following curricular outcomes: Critical thinking, development of knowledge and skills, communication skills, patient assessment, pharmaceutical care plan development, medication therapy management, pharmacodynamic decision making, and pharmaceutical product preparation, dispensing, and administration.Although this course has been beneficial to students in terms of knowledge gained and overall satisfaction, implementation into the existing curriculum did not come without challenges [13].We would like to provide insight in developing similar courses, with the hope that sharing our experiences over the past six years will lead to expedited successes at other institutions.

Course Description
Acute Care Pharmacotherapy Simulation, as the course was titled, was designed as a three-credit elective focusing on the complex pharmaceutical care provided to patients within intensive care units.The 15-week course was offered to third year pharmacy students and met once per week for three hours.Due to the highly interactive nature of the course, enrollment was capped at 12-16 motivated students interested in critical care medicine.At the completion of the course, students were expected to be capable of optimizing the outcomes of patients with cardiovascular and/or critical illness through the provision of evidence-based medication therapy as an integral part of an interprofessional team.Each week, students were responsible for viewing online video lectures prior to class that introduced a new concept of critical care medicine prior to the live, faculty-led activity.These video lectures and slide sets provided background information about various topics, which were subsequently applied during the scheduled practica sessions involving high-fidelity patient simulators.The term practica is used to describe a live, face-to-face interaction between faculty and students that utilizes active learning techniques rather than traditional didactic lecture.During simulation sessions (practica), students worked in groups to complete three patient cases.This included gathering information about the patient's demographics, past medical history, allergies, current symptoms, and laboratory results.Based on the simulator observations and interactions, the student group then collaboratively developed and communicated a complete pharmacotherapy plan to the facilitator, which included drug therapy (drug, dose, route, frequency, and duration), monitoring, prophylaxis, and non-pharmacologic recommendations.Following the conclusion of each patient case, faculty facilitators led a debriefing session where students were provided feedback on their choices.Debriefing sessions are conversations led by faculty facilitators that allow individuals to reflect on simulated patient cases [14].The goals are to reinforce the simulation objectives and improve future performance.
Student assessment was performed in a variety of ways throughout the course.Prior to each simulation practica, students completed a pre-test to assess their knowledge on the topic gained solely from the online video lectures.Post-tests were administered after the simulation concluded to assess if the student's understanding of the topic improved.In addition, students were individually assessed using the clinical skills assessment (CSA) format twice during the semester, once at the course mid-point and again at the conclusion of the course.Clinical Skills Assessments are intended to assess students' preparedness for entering practice as a pharmacist based on various clinical scenarios [15].Finally, students were responsible for documenting one of the patient cases presented each week via a subjective, objective, assessment, and plan (SOAP) note, which was submitted electronically and graded.The complete course description has been previously described [13].

Keep Video Segments Short
The weekly video lectures for the course consisted of three to four videos, which were each 10 to 20 min in length.We feel breaking the lectures into shorter segments serves various purposes.Shorter videos allow students flexibility in viewing the material in a way that fits their schedule.While it may be conducive to watch all the video segments at once during the week, the viewing could be spaced out over several days, and this format provides both options to students.Additionally, online lectures allow students to go back and review content they were not comfortable with during the first review.Although this could be achieved if the lecture were one large video instead of multiple short segments, we believe reviewing several short segments was more user friendly.
We also felt that multiple short videos assisted with students' attention span.As was previously stated, students were responsible for learning this information on their own outside of the classroom.Many have speculated the average student's ability to maintain sustained attention is between 10 to 15 min [16].Although well-conducted studies in this area are lacking, it is safe to assume most students are unable to maintain focus for 50 min continuously.
The final reason we chose to format the lecture content in short video segments was to make updating the material easier for faculty.Instead of requiring that an hour video lecture be completely re-filmed when new evidence is published, our model allows the video segment which is relevant to the update be changed alone.A perfect example of this was the lecture on anticoagulation.Since the introduction of this course many new oral anticoagulation agents were introduced into practice.It was important to cover these agents and explore their use in critically ill patients, however it did not necessitate changing all the online lecture content.As a result, this material was presented in its own segment without changing other segments.

Hold Students Accountable for Lecture Material Prior to Simulation Practica
In order for simulation-based learning to be successful, students must come prepared with baseline knowledge of the topics to be covered.Without this foundation, simulation practica are less like a real-world patient interaction and more like a traditional lecture led by the faculty facilitator.To ensure students were properly prepared, we used multiple tools to maintain accountability.
We employed weekly pre-tests which were administered online and were to be completed with a satisfactory score prior to each simulation practica.These pre-tests served two purposes.First, to gauge each student's baseline understanding of the topic presented allowing a comparison with a post-test given after the simulation.Second, to incentivize students to devote proper time to learning the material prior to class.Both the pre-tests and the post-tests affected the student's overall grade in the course.
We also utilized video tracking tools offered as part of the Panopto Viewer software (Panopto, Inc., Pittsburgh, PA, USA).This tool allowed us to monitor which students viewed each video, when the video was viewed, and how long the video played.With this information, faculty are able to reach out to individuals who are not viewing the lectures to assess barriers.In addition, tracking these data can help highlight trends among the viewers and can lead to future course improvements.

Format Videos Based on Student Preferences
The Panopto Viewer software allows students to view online video lectures in a variety of ways.Options include viewing only video of the presenter, only the PowerPoint slides with the presenter audio, or both the presenter video and the lecture slides simultaneously.Based on student feedback, we have found that students prefer actual video of the lecturer presenting rather than just the audio component.We believe tailoring online learning to students' preferences increases their willingness to devote adequate time to learning the material and increases overall satisfaction in the course.

Simulation Requires More Than a Patient Simulator
Simulation-based learning cases utilized SimMan ® patient simulators (Laerdal Corporation, Stavanger, Norway) to represent human patients.While these devices demonstrate a wide array of physiologic conditions, facilitators must continually search for new ways to engage students.For us, that meant utilizing the full capabilities of the patient simulators as well including "props" that the students used during the practica.For example, when discussing medication safety, an IV bag was prepared using two incompatible salts, which formed a visible precipitate.This was one of the errors students had to identify which could have resulted in patient harm.When discussing phenytoin-related adverse drug events, a purple mitten was placed on the simulator to emphasize purple glove syndrome.Facilitators and students also used a mobile application developed by BUMP Technologies, Inc (Mountain View, CA, USA) to transfer images such as chest radiographs, laboratory values, and other components of the electronic medical record to students' personal devices when requested.This provides students with immediate feedback when the appropriate questions are asked during the simulation.The use of BUMP Technologies' application in this course has been previously reported [17].
Students can benefit from actually administering medications (or simulated medications) rather than just recommending them.While this is not feasible for all medications administered in an ICU setting, it is especially beneficial when discussing appropriate administration rates.An example of this is illustrating the benefits of fosphenytoin administration in place of phenytoin to a patient who requires a rapid loading dose.Rather than just reading the maximum administration rates for each drug in the package insert, students can calculate the appropriate dose, draw up the correct volume, and administer the medication per the recommendations.This is a more concrete way to see the benefit of one agent over another when the difference may be less immediately evident otherwise.So, these extra nuances may not be absolutely necessary to complete the objectives of each patient case, they provide more opportunity for student engagement and contribute to the real-world nature of the simulation practica.

Develop Cases Based on Instructor Notes and Progress in the Level of Complexity of Cases
The patient cases, including patient characteristics, symptoms, laboratory values, and interventions, should follow notes presented by the instructor, unless students have been specifically instructed to consult other resources.This is necessary for a variety of reasons.Simulation based learning practica should elaborate on and reinforce the introductory knowledge students gained during the video lectures.Additionally, students may become frustrated if the knowledge gained from the video lectures is not applicable to simulation cases.This may, in turn, decrease their willingness to participate in both activities.
Initially, patient cases should be straightforward.This allows students to adjust to the new learning environment and gain a better understanding of what is expected of them.As the course progresses, so too should the level of complexity of each case.This can be accomplished by combining topics from previous simulation practica with the current simulation topic.Students are expected to make therapeutic decisions based on information previously presented to them, whether that is via the online lectures or previous simulation practica.

Include a Debriefing Session Following Each Simulation Practica
Following each simulated patient case with a debriefing session in which students and facilitators discuss aspects of the case, provides a variety of benefits to both parties.Students have the opportunity to ask more questions about the case and have concepts further clarified.Facilitators have the opportunity to see areas of instruction that may not have been covered in enough detail.Additionally, debriefing sessions provide a platform for facilitators to discuss clinical pearls and other information that may be difficult to incorporate into the actual simulation.Facilitators also have the opportunity to conduct impromptu mini-lectures on topics not covered normally in the course, but where students are struggling.One example from this past year was the need to discuss general antimicrobial coverage between classes of antibiotics.
Just as the patient cases progress throughout the semester, so too must the debriefing sessions.Initially, we allow the students to ask questions during the simulated patient experience, which provides a blended simulation-debriefing experience.As the course progresses, students are encouraged to complete each simulation without relying on facilitators to answer questions.The separated simulation and debriefing provides a more real-world environment and fosters critical thinking to work through problems encountered along the way.However, it still provides the same learning opportunity for students to discuss unanswered questions post-simulation.Overall, these sessions are invaluable for both students and facilitators and provide a low-stress learning environment.

Evaluate Students on Knowledge, Skill, and Communication
All pharmacists must be knowledgeable about the drugs they are recommending and dispensing, however this is no longer their only responsibility.Therefore, students today must be evaluated on more than just their ability to recall drug information on demand.A key component of this course is the ability of students to work-up a patient and formulate an appropriate pharmacotherapy plan.Communication and documentation were also important when developing this course because it is reflective of real-world practice.We utilized various assessment tools to evaluate students' progress throughout the course based on these goals.The unique nature of this course requires the multiple assessment modalities utilized to properly assess students' progress.When developing a course at your institution, be sure each of your stated objectives can be assessed objectively throughout the course.

Utilize a Standardized Assessment Tool
Multiple-choice exams, like the ones we used for pre-simulation and post-simulation assessment, are inherently objective.However, students' skill in more complex areas such as patient work-up, plan development, communication, and documentation require a more structured approach to ensure students are fairly assessed.This ensures each individual is given an equal opportunity to succeed.To objectify an ordinarily subjective process, standardized assessment rubrics were developed for both the patient simulation practica and the post-simulation SOAP note.The rubric designed for patient simulation practica evaluated each student individually on all aspects of patient care, and was intended to be completed by the facilitator or another observer while the simulated interaction was taking place.The rubric used for exams and blood pressure monitoring is published elsewhere [18,19].The SOAP note rubric focused more on specifics of the pharmacotherapy plan including ranking of the patient problems, appropriate monitoring parameters, and nonpharmacological therapy.The formatting, clarity, and conciseness of each note were also evaluated.
When developing standardized rubrics, there are several things to consider.Evaluation criteria should be based on best practices, as this is the level at which students should be expected to perform at this stage in their educational career.After development and prior to any actual assessment, all rubrics should be made available to students.This allows each individual to formulate their plan for conducting the simulation based on the evaluation criteria.Finally, assessment rubrics should have high inter-rater reliability so as to standardize the assessment process.This may require testing the rubric initially by having multiple evaluators assess the same student until it has been proven reliable.Although student scores should be assigned using a standardized rubric, each person should also be provided with specific comments on their performance by the evaluator.This should include both areas of strength and where improvement is necessary.

Continually Seek Feedback from Students
Student feedback should be at the center of all future course improvements.Due to the highly interactive nature of this course, it is essential to ensure students' needs are met in order to maintain future involvement.This is especially important during the early iterations of the course.We suggest offering the course to a small number of highly motivated students the first few years.These students will provide the feedback which will shape the course for future years, so it is important to include students whose opinions will be relevant and meaningful.
Additionally, the format in which feedback is solicited may be important.For most courses, students complete an online survey at the conclusion of the semester.These surveys generally rely on a Likert Scale to assess student satisfaction with various aspects of the course.Although this is beneficial for well-established courses, it may be insufficient for a new innovative course such as this.We suggest allowing for an individualized discussion near the end of the term with each student to gauge overall course impressions as well as possible areas for improvement.

Always Look to the Future
Faculty who have successfully implemented a dual-immersive online-simulation course such as this into their curriculum must then turn their attention to future innovations.With the help of students' feedback, the course coordinators at our institution have identified several new areas of interest to make this course more beneficial.Development of a mobile application or podcast has been suggested to make viewing online lecture content easier and more convenient for students.As mobile devices and tablet