Next Article in Journal
A Proposed VR Platform for Supporting Blended Learning Post COVID-19
Previous Article in Journal
Samantha Green vs. Peter Stevens: Reversed Gender Stereotype Threat in Online Chess
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Education Sciences
Volume 12
Issue 7
10.3390/educsci12070434
education-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The Role of Game-Based Learning in Experiential Education: Tool Validation, Motivation Assessment, and Outcomes Evaluation among a Sample of Pharmacy Students

by
Mariam Dabbous
1,2,*,
Anwar Kawtharani
2,
Iqbal Fahs
1,
Zahraa Hallal
1,
Dina Shouman
2,
Marwan Akel
1,3,4,
Mohamad Rahal
1 and
Fouad Sakr
1,5,6,*
1
School of Pharmacy, Lebanese International University, Beirut 1105, Lebanon
2
School of Education, Lebanese International University, Beirut 1105, Lebanon
3
INSPECT-LB (Institut National de Santé Publique, d’Épidémiologie Clinique et de Toxicologie-Liban), Beirut 1103, Lebanon
4
International Pharmaceutical Federation (FIP), 2517 The Hague, The Netherlands
5
École Doctorale Sciences de la Vie et de la Santé, Université Paris-Est Créteil, 94010 Créteil, France
6
UMR U955 INSERM, Institut Mondor de Recherche Biomédicale, Université Paris-Est Créteil, 94010 Créteil, France
*
Authors to whom correspondence should be addressed.
Educ. Sci. 2022, 12(7), 434; https://doi.org/10.3390/educsci12070434
Submission received: 28 May 2022 / Revised: 20 June 2022 / Accepted: 21 June 2022 / Published: 24 June 2022
(This article belongs to the Special Issue Game-Based Learning: Evaluation of Integrating Pedagogical Content and Assessment)
Browse Figure
Versions Notes

Abstract

:
(1) Background: There is a scarcity of data regarding game-based learning (GBL) in experiential pharmacy education; in addition, the impact of educational games on the attainment of intended learning outcomes and students’ motivation to actively learn and practice through non-traditional pedagogical tools are yet to be explored. (2) Methods: This was a prospective quasi-experimental study that introduced GBL into the Pharmacy Practice Experience course of the Bachelor of Pharmacy (BPharm) program at the Lebanese International University in Lebanon. Data collection took place between July and September 2021. The study objectives were to assess the impact of GBL on attaining intended learning outcomes, determine students’ motivation to engage in GBL, and assess the impact of this motivation on the attainment of learning outcomes. (3) Results: GBL was associated with a significantly higher exam average (mean difference = 7.152, p < 0.001). Moreover, an active learning motivation assessment scale (ALMAS) was constructed and validated; it was found to have good reliability as well as high sensitivity and specificity, and it determined a good level of motivation to engage in GBL. Game-based learners with higher motivation scores had significantly higher exam averages compared to those with lower scores (Beta = 0.296, 95% CI 0.110–0.545, p = 0.004). (4) Conclusion: GBL was associated with better attainment of intended learning outcomes. Students appear to be motivated to learn by this style of active learning, and motivation is prognostic of the attainment of learning outcomes.

1. Introduction

Game-based learning (GBL) is the addition of certain game principles and elements to conventional pedagogies in order to engage learners in a joyful and dynamic way [1]. GBL is not just creating games for students to play; it is designing learning activities that incrementally introduce concepts and guide learners towards an end goal [2]. Game mechanics are the building blocks of games that invoke a meaningful response from the learner [3]. They are utilized to improve interactivity, rewards, and motivation through a distinctive combination of strategies and game features [4,5]. Game mechanics can be subclassified into game attributes and game elements. Game attributes, such as rules, goals, and competition, are features inherent in the game structure that initiate interest in gaming activities [6]. On the other hand, game elements, such as badges, levels, and rewards, are tools shared by games but not necessarily critical conditions of games [1].
GBL provides a fun, interactive, personalized, and challenging learning environment while enabling learners to experience and apply knowledge [7,8,9]. Despite the popularity of educational games, there is little consensus about whether they improve academic performance [10]. Several studies suggested that GBL enhances learning performance and outcomes through affective (e.g., enjoyment), cognitive (e.g., cognitive load), and behavioral (e.g., intention to participate in learning) mechanisms [11,12,13,14]. However, other studies reported no effect [15] or even negative effects of educational games on students’ exam scores [16]. Lack of integration between gaming and teaching, as well as a poor balance between enjoyment and education, are important drawbacks of educational games. Games can be too attractive but fail to reflect intended learning outcomes, or they can be too educational but unable to trigger interest among learners [17].
GBL improves students’ self-efficacy in that educational games can provide more diverse learning methods that are effective in increasing students’ interest and learning efficiency [18,19]. In addition, such games deepen the students’ understanding of the educational content so they can solve more complex problems and develop creative and critical thinking [20,21]. Several studies have highlighted the positive impact of GBL on learners’ motivation and attitude [22,23]. Motivation is crucial to learning, and educational games can help motivate learners [24]. Utilizing game mechanics and aesthetics, such as levels, points, leaderboards, and a competitive environment, can enhance the students’ external and internal motivations to engage in these games [25]. Appropriate use of competition, challenge, scoring, and ranking can motivate learners to win, help learners gain a sense of accomplishment and satisfaction, and make learners highly motivated to learn [26]. GBL aims to improve motivation and, consequently, the teaching–learning process [27]. Encouraging interaction between the teacher and the student through educational games increases motivation, leading to an improvement in the capacity to assimilate knowledge and acquire skills [28].
The limited use of innovative teaching techniques in pharmacy education, with the widespread use of teacher-centered strategies, led to the decreased retention of knowledge, lower student engagement, decreased motivation, and an increased classroom absence rate [29]. This highlights the need for the refinement of teaching methods and the implementation of new pedagogical methods in the pharmacy curriculum to enhance learner motivation, engagement, and knowledge retention. GBL is a current teaching and learning trend spanning a range of academic studies [2]. Many universities have begun to employ a variety of game-based technologies in an effort to add an extra level of motivation and incentive to many higher education activities. The Accreditation Council for Pharmacy Education (ACPE) has included active learning as a required component of the pharmacy curriculum [30]. Pharmacy educators are trying to meet ACPE standards through the utilization of GBL activities. Escape rooms, software and real-life simulations, and quiz-based review games are examples of GBL activities applied in pharmacy education [2]. GBL was utilized in pharmaceutical and non-pharmaceutical modules, including immunology, opioid safety, community pharmacy, and pharmacy management [31]. Examples of educational games for pharmacy students include an advanced software program that allows players to manage a city that produces multiple herbal products [32], a 3D simulation of working at a community pharmacy [33,34], an innovative escape room utilizing pre-set locks, clues, and boxes to teach clinical concepts of toxicology [35], and group-based quizzing followed by a Kahoot!® competition to prepare students for immunology assessments [36]. Several advantages of GBL experienced by pharmacy students were demonstrated, such as practicing their clinical skills without the risk of endangering the patients’ health, implementing collaborative learning, receiving immediate feedback, and inducing behavioral and attitude changes [37].
Gaming activities can be an especially effective method to teach students when content is application-based; this allows students to practice with a more hands-on approach or review previous content covered by other traditional teaching methods. Experiential education is an essential cornerstone of pharmacy education and training, according to several international organizations [38]. It is a practice-based experience that provides pharmacy students with real-life training opportunities to apply taught knowledge and develop personal attributes and professional skills [39,40]. Community pharmacy practice experience courses offer pharmacy students rich learning opportunities to become competent pharmacists and worthwhile members of healthcare teams [41]. Although GBL was introduced to higher education in Lebanon by an English teacher in 2015 [42], limited literature exists with respect to pharmacy education. Moreover, there is a scarcity of data regarding GBL in pharmacy practice experience (PPE) courses; in addition, the impact of educational games on the attainment of intended learning outcomes and students’ motivation to actively learn and practice through non-traditional pedagogical tools are yet to be explored. This study aimed to assess the impact of GBL on attaining the intended learning outcomes of a PPE course. It also aimed to determine students’ motivation to engage in these active learning activities and assess the role of this motivation on the attainment of learning outcomes.

2. Materials and Methods

2.1. Study Design and Setting

This was a prospective quasi-experimental study that introduced GBL into the PPE II course of the Bachelor of Pharmacy (BPharm) program at the Lebanese International University (LIU) in Lebanon. Data collection took place between July and September 2021. PPE II is a twelve-week supervised experiential education course in a community pharmacy setting that takes place at the end of the second professional year of the program. PPE II explores advanced non-communicable and communicable diseases topics related to patient care. This practice experience provides students with adequate advanced community knowledge and communication skills to engage in patient care and interact with healthcare professionals. A manual with several modules tackling the required topics has to be reviewed and discussed by students. The manual aims to guide students to acquire and develop the necessary knowledge, skills, and values in order to be effective and competent community pharmacists. The learning outcomes of the course are to equip students with competencies related to 4 domains, including (1) foundational knowledge, (2) pharmaceutical care, (3) essentials for practice and care, and (4) approach to practice and care.
With the suspension of community pharmacy onsite trainings due to COVID-19 restrictions, the PPE II delivery method was switched to a virtual training rotation. Various activities were performed to acquire the needed competencies and to compensate for the impact of the loss of the onsite community training. Thus, students met their preceptor, who was the faculty member supervising their experiential education, on a weekly basis via Google Meet. The meetings of students with their preceptors included open discussions on the different module topics included in the course manual. The discussions also included simulated patient cases, simulated patient prescriptions, and online topic presentations.

2.2. Educational Setting and Procedures

The study protocol was approved by the Research and Ethics Committee of the School of Pharmacy at the Lebanese International University (2020RC-055-LIUSOP). Students were invited to voluntarily participate in GBL activities by an email invitation that was sent to all students registered in the PPE II course. The PPE II students were then divided into two groups. The first group contained students who engaged in GBL, while the second group contained students who only took part in the traditional learning activities of the course. All students were examined at the beginning of the course in order to determine the baseline characteristics and comparability of the research groups. Students in both groups were then evaluated at the end of the course by a standardized common exam that is routinely used to evaluate the course outcomes. This final examination was used to assess the overall attainment of the intended learning outcomes and the subsequent competency-based domains relating to foundational knowledge, pharmaceutical care, essentials for practice and care, and approach to practice and care. In addition, the GBL group responded to a questionnaire after 12 weeks of engagement in educational gamification to assess students’ motivation to engage in GBL.

2.3. Game-Based Learning Activities

The PPE II course coordinator prepared several games using different gaming platforms, such as Gamilab, Wisc-Online, and Quizizz, and encouraged students to utilize these gaming tools at their own pace in order to memorize medications’ brand names and advance their counseling skills in a fun way. In addition, the course coordinator created a Google Classroom and met with the students every other week over a 12-week course period. Students were divided into teams and challenged to answer different open-ended and multiple-choice questions related to different case scenarios. The scenarios aimed to improve students’ skills, and all questions were mapped to the intended learning outcomes of the course distributed over the four previously described competency-based domains. Teams who answered first earned points, and each time the winner was the team with the highest score.

2.4. Variables and Outcomes

The sociodemographic and academic characteristics relating to age, gender, grade point average (GPA), area of residence, and examination scores were collected for all students through the university management system. Students’ variables were linked at the beginning and at the end of the study by their student university identification numbers, which were changed at the end of the study into sequential numbering to de-identify students prior to analysis. The final exam scores were used to evaluate the learning outcomes. The exam total average and the subsequent averages of the four competency-based domains were used to assess the attainment of the intended learning outcomes. Motivation to engage in GBL was assessed for the participating group through a questionnaire that was inspired by another study [43]. Permission was granted to utilize the questionnaire from the corresponding author. The questionnaire in the current study included 14 items that were positively phrased to evaluate motivation by a 5-point Likert scale ranging from “strongly disagree” to “strongly agree”. An active learning motivation assessment scale (ALMAS) was constructed from the 14 items on the questionnaire to assess the impact of motivation to engage in GBL on the attainment of learning outcomes. The objectives of this study were to (1) assess the impact of GBL on attaining the intended learning outcomes, (2) determine students’ motivation to engage in GBL, and (3) assess the impact of this motivation on the attainment of learning outcomes.

2.5. Statistical Analysis

Data were analyzed using the IBM Statistical Package for Social Sciences (IBM SPSS) version 26.0. The baseline characteristics of the students were evaluated by descriptive statistics and compared by bivariate analysis. Continuous variables were reported by their means (±standard deviation) or medians (interquartile range, IQR), and categorical variables were reported by their frequencies and percentages. The normal distribution of variables was assessed by histogram and the Shapiro–Wilk test. Bivariate analyses utilized the independent sample t-test and the Pearson correlation (r) for continuous variables and the chi-square for categorical variables. Factor analysis with principal component analysis was conducted to validate the ALMAS for GBL with a rotated matrix. Factors retained in the final scale had Eigen values greater than one. Adequate values for the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy and Bartlett’s test of sphericity were ensured. The correlation of each scale component with the whole scale was determined by the Pearson correlation (r). Cronbach’s alpha was used to measure the internal consistency and reliability of the total scale and the subscales. The scale sensitivity and specificity were determined by receiver operating characteristic (ROC) curve analysis, with an optimal cutoff point determined by the Youden index, defined as J = maximum (sensitivity + specificity − 1). A multivariable linear regression was conducted to determine the impact of GBL motivation on the attainment of learning outcomes, using the ALMAS as the independent variable and using the end-of-class total average of the GBL group as the dependent variable, while adjusting for the pre-game-based-learning average as well as students’ age, gender, and GPA as potential confounders. Results were reported as an unadjusted beta with a 95% confidence interval (CI). p values < 0.05 were considered statistically significant with an acceptable margin of error = 5%.

3. Results

3.1. Baseline Characteristics of Students

A total of 233 students were included in the study, with 69 students (29.6%) in the GBL group and 164 students (70.4%) in the traditional learning group. The baseline characteristics of the study groups were comparable, except for the average score in essentials for practice and care, which was higher in the GBL group. The total sample included 70% female students and 36.1% students from the capital city of Beirut, and the mean age of the students was 23.16 ± 1.82. The pre-game-based-learning total average was 65.59 ± 18.86, and the averages of the competency-specific domains were 13.68 ± 4.15 for foundational knowledge, 13.79 ± 4.15 for pharmaceutical care, 17.53 ± 7.07 for essentials for practice and care, and 20.59 ± 6.62 for approach to practice and care. The complete baseline characteristics of the students are shown in Table 1.

3.2. Learning Outcomes

The impact of GBL on the attainment of the course’s intended learning outcomes was determined by comparing the class averages for game-based learners to those of traditional learners at the completion of the course. The total average was significantly higher among the game-based learners (mean difference = 7.152, p < 0.001). The game-based learners also had significantly higher averages for foundational knowledge (mean difference = 1.369, p < 0.001), pharmaceutical care (mean difference = 1.431, p = 0.002), essentials for practice and care (mean difference = 1.850, p = 0.03), and approach to practice and care (mean difference = 2.502, p < 0.001). The comparison of class averages between game-based learners and traditional learners is shown in Table 2.

3.3. Active Learning Motivation Assessment Scale (ALMAS)

3.3.1. Factor Analysis

The ALMAS was used to determine students’ motivation to engage in GBL, a form of experiential education. Factor analysis was run to validate the ALMAS among the game-based learners’ group. All 14 items of the initial scale were extracted and, thus, constituted the final ALMAS for GBL. No variables over-correlated with each other (r > 0.9), had low communality (<0.3), or low factor loading (<0.3). The KMO measure of sampling adequacy was 0.889, with a significant Bartlett’s test of sphericity (p < 0.001). The scale items converged over two factors with Eigen values greater than 1, explaining 68.81% of the total variance. Table 3 shows the Promax rotated matrix of the ALMAS components.

3.3.2. Reliability and Validity Measures

The ALMAS had a high Cronbach’s alpha of 0.968, and the subscales loading over factor 1 and factor 2 also had high Cronbach’s alphas (0.965 and 0.904, respectively). All items of the scale significantly correlated with the total scale. The correlation coefficients ranged between 0.653 and 0.914. Table 4 reports the correlation of each item of the ALMAS with the whole scale.
The ALMAS had a median of 40 (IQR 32–45), with higher values indicating higher motivation. The ROC curve analysis revealed an optimal cutoff point for better motivation at 33.50, with 91.3% sensitivity and 76.9% specificity. The area under the curve was 0.878 (95% CI 0.801–0.956, p < 0.001). Figure 1 presents the ROC curve of the ALMAS comparing game-based learners to traditional learners.

3.4. Motivation and Learning Outcomes

The bivariate analysis of the ALMAS and the total class average determined a significant correlation between the two (r = 0.350, p = 0.001) among game-based learners. To preclude potential confounders, a multivariable linear regression was conducted, taking the end-of-class total average as the dependent variable and the ALMAS as the independent variable, while adjusting for the pre-game-based-learning total average, GPA, age, and gender. A significant positive association between motivation and post-game-based attainment of learning outcomes was found. Students with higher ALMAS scores had significantly higher total averages compared to those with lower scores (Beta = 0.296, 95% CI 0.110–0.545, p = 0.004). Table 5 reports the multivariable linear regression of the post-game-based-learning total average.

4. Discussion

The current study evaluated the impact of GBL on the attainment of intended learning outcomes of experiential education in a professional BPharm program. It also constructed and validated the ALMAS and determined the impact of motivation to learn actively utilizing GBL on the attainment of course learning outcomes. GBL was associated with better acquisition of all intended learning outcomes in terms of foundational knowledge, pharmaceutical care, essentials for practice and care, and approach to practice and care. The ALMAS was validated as a reliable tool to assess students’ motivation to engage in GBL and to determine the impact of this motivation on the attainment of learning outcomes. Attainment of the intended learning outcomes was predicted by motivation to participate in GBL, with higher ALMAS scores being associated with better achievement with respect to learning outcomes as determined by the higher average of the game-based learners.
We found a significant association between GBL and attaining the intended learning outcomes. Students who participated in educational games had significantly higher average scores on the PPE course assessment exam compared to students who were in the traditional learning group. Therefore, GBL appears to play an important role in active learning, information retention, and attainment of competencies in pharmacy experiential education. Our findings are comparable with those of other research that found that students who used educational gamification as an additional pedagogical tool in an introductory PPE course obtained higher scores on quizzes [44]. Nonetheless, that study only looked at low-order thinking since it contained material and gamification elements requiring routine memorization. The current study, on the other hand, also assessed higher-order thinking by incorporating challenging questions that simulated real clinical case scenarios on the platforms of the games. The higher-level thinking related to dealing with real patient case scenarios that required students to analyze patient data, evaluate and select available pharmacotherapeutic options, and create an individualized care plan for the patient. The required levels of thinking, ranging from lower- to higher-order, are mapped to the four competency-based domains of the PPE course. GBL apparently triggers active learning that boosts information retention and overall performance [45]. Our findings add to the literature evidence that GBL is not only important for knowledge retention but also has a significant role in the acquisition of skills via practical experiences of pharmacy education.
The intended learning outcomes of introductory experiential education in a pharmacy curriculum are to equip students with four competency-specific domains relating to foundational knowledge, pharmaceutical care, essentials for practice and care, and approach to practice and care [46]. Foundational knowledge is the first basic domain that allows students to make a clinical decision by applying knowledge learned in didactic courses. Pharmaceutical care is the second domain, and it empowers students to select correct drug dosage forms, compound and dispense medications, and provide patient counseling. In the third domain, students acquire essentials for practice and care to offer and optimize patient-centered care and promote health. The fourth domain is approach to practice and care, which provide competencies that enable students to collaborate, advocate, and become problem solvers by developing the necessary behaviors and skills [30,47,48].
Educational gamification can improve learner performance in several cognitive areas, including memory, comprehension, and the application of concepts [49]. To further understand the role of educational games as an active learning activity and determine their outcomes in depth, this study assessed the impact of GBL on the competency-specific domains of the intended learning outcomes. We found that GBL had an important positive impact on all of the four competency-specific domains by significantly improving the outcomes for foundational knowledge, pharmaceutical care, essentials for practice and care, and approach to practice and care. The GBL educational activities supported students in obtaining foundational knowledge and memorizing the brand names of medications. Furthermore, students practiced pharmaceutical mathematics to calculate the proper dosages of medications. Students also practiced presenting effective health and drug information to patients or caregivers through case scenarios that focused on vital counseling, side effects, and drug interactions. Students worked together in a team collaboration environment to offer patient counseling services. To the best of our knowledge, no studies have looked at the effect of GBL on competency-specific domains of courses. However, our findings are coherent with previous research that revealed that GBL could enhance knowledge, build long-term memory, encourage teamwork, and improve communication skills [50,51,52].
Motivation to learn using various active learning activities can substantially influence students’ engagement in educational activities and, therefore, can have an important impact on the attainment of learning outcomes [28]. Higher motivation may be associated with active engagement in game-based learning; thus, it is suggested that motivation enhances the attainment of intended learning outcomes [27]. To assess motivation and determine its impact on achieving learning outcomes, the present study constructed and validated the ALMAS as a reliable tool to assess motivation to engage in GBL and determine the prognostic role of motivation on the outcomes of active learning activities. Our findings provide evidence that the ALMAS is valid and reliable in determining motivation to engage in GBL. The two factors of the scale have a very good internal consistency [53], and the results suggest excellent reproducibility because all components correlated highly significantly with the total scale. To the best of our knowledge, no valid tools are available for determining students’ motivation to engage in game-based or other forms of active learning. The present study provides an important tool for motivation assessment in this context. The construct validity was confirmed by computing the sensitivity and specificity of the scale. The ALMAS had high sensitivity and specificity, supporting its use for GBL.
We found that gamification is motivational for students and encourages them to engage in GBL activities, as the ALMAS median score among our sample was higher than the ROC curve cutoff point for higher motivation. We also found a significant positive association between motivation to engage in GBL and the attainment of learning outcomes. Students who were more motivated to engage in GBL had better results with respect to learning outcomes, as determined by a higher mean of the course final exam scores. Consequently, this study verifies that educational gamification is motivational for students, encouraging them to engage and obtain the required knowledge and skills through active learning involving game-based activities. Students are motivated to engage in these activities, which may be more motivational for learning compared to traditional class activities. Moreover, higher motivation to engage in GBL may be associated with higher rates of class attendance, more active participation, better focus, and greater interest in learning the subject; all of these factors, in turn, can improve the attainment of intended learning outcomes and the students’ grades in the course. Although no other studies have measured the prognostic role of motivation to engage in active learning activities on the attainment of learning outcomes, our findings are consistent with previous literature that reported engagement and satisfaction of students who participated in GBL activities, in addition to confidence in the acquired information and skills [54,55,56]. Educational games improve student participation in the educational journey and offer learning in a joyful and interactive environment [57]. Game mechanics incorporating immediate feedback, scores, and leader boards are principal features of educational games, and it is suggested that they have a vital role in encouraging engagement and motivation [58,59,60]. These features were incorporated into each of the gaming activities in this research study in order to promote competition between students by rewarding winning teams with points, which could be an additional reason for the observed better attainment of the intended learning outcomes.

Strengths and Limitations

The quasi-experimental design of the study endorses the impact of GBL on the attainment of intended learning outcomes and supports the internal validity of our results. Our findings are valid and reliable because the assessment tool used in this research was validated and had excellent internal consistency. Moreover, the assessment of learning outcomes employed valid measurements by utilizing the standard assessment examination of the studied course. This valid and objective assessment minimized any possible risk of selection or information bias. On the other hand, several limitations cannot be avoided. Traditional examination may not be the best technique to assess success with respect to the outcomes of all areas of the competency-based domains. Moreover, the sizes of the study groups were not equal; this was because of student self-selection into each group. This may indicate a possible risk of selection bias; however, this bias was minimized because the two groups were academically and demographically comparable, and there was no significant difference in the baseline characteristics of the students. Our findings also do not have a high external validity because this research involved only students from a practice course in the second professional year of the professional BPharm program. Additional research is recommended in this area to assess the impact of GBL during different years of professional pharmacy programs on the attainment of intended learning outcomes for both experiential and didactic courses to confirm the generalizability of the results.

5. Conclusions

GBL was associated with better attainment of the intended learning outcomes and resulted in better acquisition of foundational knowledge, pharmaceutical care, essentials for practice and care, and approach to practice and care. Students appear to be motivated to learn by this style of active learning, and this motivation is prognostic of the attainment of learning outcomes. GBL offers an added pedagogical method for faculty members to encourage the greater retention of knowledge and development of skills, and it is recommended for implementation in PPE courses and for future research in didactic pharmacy courses in Lebanon and all over the world. The ALMAS is a novel, valid, and reliable tool for assessing students’ motivation to engage in GBL. It can also be suggested as a generic tool for assessing motivation to engage in different non-traditional learning activities.

Author Contributions

Conceptualization, M.D. and F.S.; methodology, M.D. and F.S.; software, F.S.; validation, F.S.; formal analysis, F.S.; investigation, M.D.; resources, M.D.; data curation, M.D.; writing—original draft preparation, M.D., F.S., I.F. and Z.H.; writing—review & editing, A.K., D.S., M.A. and M.R.; supervision, A.K. and M.R.; project administration, M.D.; funding acquisition, M.D. All authors have read and agreed to the published version of the manuscript.

Funding

The APC was funded by the Lebanese International University.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Research and Ethics Committee of the School of Pharmacy at the Lebanese International University (protocol code: 2020RC-055-LIUSOP).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available from the corresponding author on reasonable request.

Acknowledgments

The authors would like to thank all students who participated in this research.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Deterding, S.; Dixon, D.; Khaled, R.; Nacke, L. From Game Design Elements to Gamefulness: Defining “Gamification”. In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, Tampere, Finland, 28–30 September 2011; Association for Computing Machinery: New York, NY, USA, 2011; pp. 9–15. [Google Scholar]
  2. Sera, L.; Wheeler, E. Game on: The Gamification of the Pharmacy Classroom. Curr. Pharm. Teach. Learn. 2017, 9, 155–159. [Google Scholar] [CrossRef] [PubMed]
  3. Zichermann, G.; Cunningham, C. Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps; O’Reilly Media, Inc.: Sebastopol, CA, USA, 2011; ISBN 978-1-4493-9767-8. [Google Scholar]
  4. Abdul Jabbar, A.I.; Felicia, P. Gameplay Engagement and Learning in Game-Based Learning: A Systematic Review. Rev. Educ. Res. 2015, 85, 740–779. [Google Scholar] [CrossRef]
  5. Fabricatore, C. Gameplay and game mechanics design: A key to quality in videogames. In Proceedings of the ENLACES (MINEDUC Chile)—OECD Expert Meeting on Videogames and Education, Santiago de Chile, Chile, 29–31 October 2007. [Google Scholar]
  6. Hull, D.C.; Williams, G.A.; Griffiths, M.D. Video Game Characteristics, Happiness and Flow as Predictors of Addiction among Video Game Players: A Pilot Study. J. Behav. Addict. 2013, 2, 145–152. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Brangier, E.; Marache-Francisco, C. Measure of the Lived and Functional Effects of Gamification: An Experimental Study in a Professional Context. In Proceedings of the Advances in Ergonomics in Design, San Diego, CA, USA, 16–20 July 2020; Springer: Cham, Switzerland, 2020; pp. 242–253. [Google Scholar]
  8. Chen, C.-H.; Liu, J.-H.; Shou, W.-C. How Competition in a Game-Based Science Learning Environment Influences Students’ Learning Achievement, Flow Experience, and Learning Behavioral Patterns. J. Educ. Technol. Soc. 2018, 21, 164–176. [Google Scholar]
  9. Taguchi, N. The Routledge Handbook of Second Language Acquisition and Pragmatics; Routledge: London, UK, 2019; ISBN 978-1-351-16406-1. [Google Scholar]
  10. Bai, S.; Hew, K.F.; Huang, B. Does Gamification Improve Student Learning Outcome? Evidence from a Meta-Analysis and Synthesis of Qualitative Data in Educational Contexts. Educ. Res. Rev. 2020, 30, 100322. [Google Scholar] [CrossRef]
  11. Eltahir, M.E.; Alsalhi, N.R.; Al-Qatawneh, S.; AlQudah, H.A.; Jaradat, M. The Impact of Game-Based Learning (GBL) on Students’ Motivation, Engagement and Academic Performance on an Arabic Language Grammar Course in Higher Education. Educ. Inf. Technol. 2021, 26, 3251–3278. [Google Scholar] [CrossRef]
  12. Koivisto, J.; Hamari, J. The Rise of Motivational Information Systems: A Review of Gamification Research. Int. J. Inf. Manag. 2019, 45, 191–210. [Google Scholar] [CrossRef]
  13. Marín, B.; Frez, J.; Cruz-Lemus, J.; Genero, M. An Empirical Investigation on the Benefits of Gamification in Programming Courses. ACM Trans. Comput. Educ. 2018, 19, 4. [Google Scholar] [CrossRef]
  14. Sailer, M.; Homner, L. The Gamification of Learning: A Meta-Analysis. Educ. Psychol. Rev. 2020, 32, 77–112. [Google Scholar] [CrossRef] [Green Version]
  15. Rachels, J.R.; Rockinson-Szapkiw, A.J. The Effects of a Mobile Gamification App on Elementary Students’ Spanish Achievement and Self-Efficacy. Comput. Assist. Lang. Learn. 2018, 31, 72–89. [Google Scholar] [CrossRef]
  16. De-Marcos, L.; Domínguez, A.; Saenz-de-Navarrete, J.; Pagés, C. An Empirical Study Comparing Gamification and Social Networking on E-Learning. Comput. Educ. 2014, 75, 82–91. [Google Scholar] [CrossRef]
  17. Pan, L.; Tlili, A.; Li, J.; Jiang, F.; Shi, G.; Yu, H.; Yang, J. How to Implement Game-Based Learning in a Smart Classroom? A Model Based on a Systematic Literature Review and Delphi Method. Front. Psychol. 2021, 12, 749837. [Google Scholar] [CrossRef]
  18. Wang, M.; Zheng, X. Using Game-Based Learning to Support Learning Science: A Study with Middle School Students. Asia-Pac. Educ. Res. 2021, 30, 167–176. [Google Scholar] [CrossRef]
  19. Yang, K.-H.; Lu, B.-C. Towards the Successful Game-Based Learning: Detection and Feedback to Misconceptions Is the Key. Comput. Educ. 2021, 160, 104033. [Google Scholar] [CrossRef]
  20. Nadolny, L.; Valai, A.; Cherrez, N.J.; Elrick, D.; Lovett, A.; Nowatzke, M. Examining the Characteristics of Game-Based Learning: A Content Analysis and Design Framework. Comput. Educ. 2020, 156, 103936. [Google Scholar] [CrossRef]
  21. Perini, S.; Luglietti, R.; Margoudi, M.; Oliveira, M.; Taisch, M. Learning and Motivational Effects of Digital Game-Based Learning (DGBL) for Manufacturing Education—The Life Cycle Assessment (LCA) Game. Comput. Ind. 2018, 102, 40–49. [Google Scholar] [CrossRef]
  22. Taub, M.; Sawyer, R.; Smith, A.; Rowe, J.; Azevedo, R.; Lester, J. The Agency Effect: The Impact of Student Agency on Learning, Emotions, and Problem-Solving Behaviors in a Game-Based Learning Environment. Comput. Educ. 2020, 147, 103781. [Google Scholar] [CrossRef]
  23. Yadav, A.K.; Oyelere, S.S. Contextualized Mobile Game-Based Learning Application for Computing Education. Educ. Inf. Technol. 2021, 26, 2539–2562. [Google Scholar] [CrossRef]
  24. Batson, L.; Feinberg, S. Game Designs That Enhance Motivation and Learning for Teenagers. Electron. J. Integr. Technol. Educ. 2006, 5, 34–43. [Google Scholar]
  25. Kalogiannakis, M.; Papadakis, S.; Zourmpakis, A.-I. Gamification in Science Education. A Systematic Review of the Literature. Educ. Sci. 2021, 11, 22. [Google Scholar] [CrossRef]
  26. Jagušt, T.; Botički, I.; So, H.-J. Examining Competitive, Collaborative and Adaptive Gamification in Young Learners’ Math Learning. Comput. Educ. 2018, 125, 444–457. [Google Scholar] [CrossRef]
  27. De-Marcos, L.; García-Cabot, A.; García, E. Towards the Social Gamification of E-Learning: A Practical Experiment. Int. J. Eng. Educ. 2017, 33, 66–73. [Google Scholar]
  28. Fuster-Guilló, A.; Pertegal-Felices, M.L.; Jimeno-Morenilla, A.; Azorín-López, J.; Rico-Soliveres, M.L.; Restrepo-Calle, F. Evaluating Impact on Motivation and Academic Performance of a Game-Based Learning Experience Using Kahoot. Front. Psychol. 2019, 10, 2843. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  29. Unimed. Digihealth, Innovative Digital Skills & Teaching Methods 4 Effective Health Education in Lebanon & Syria. Available online: https://www.uni-med.net/progetti/digihealth/ (accessed on 6 June 2022).
  30. ACPE Standards. Available online: https://www.ahrq.gov/health-literacy/professional-training/pharmacy/app-2.html (accessed on 25 May 2022).
  31. Oestreich, J.H.; Guy, J.W. Game-Based Learning in Pharmacy Education. Pharm. J. Pharm. Educ. Pract. 2022, 10, 11. [Google Scholar] [CrossRef]
  32. Ee, R.W.X.; Yap, K.Z.; Yap, K.Y.-L. Herbopolis—A Mobile Serious Game to Educate Players on Herbal Medicines. Complement. Ther. Med. 2018, 39, 68–79. [Google Scholar] [CrossRef]
  33. Berger, J.; Bawab, N.; De Mooij, J.; Sutter Widmer, D.; Szilas, N.; De Vriese, C.; Bugnon, O. An Open Randomized Controlled Study Comparing an Online Text-Based Scenario and a Serious Game by Belgian and Swiss Pharmacy Students. Curr. Pharm. Teach. Learn. 2018, 10, 267–276. [Google Scholar] [CrossRef] [Green Version]
  34. Bindoff, I.; Ling, T.; Bereznicki, L.; Westbury, J.; Chalmers, L.; Peterson, G.; Ollington, R. A Computer Simulation of Community Pharmacy Practice for Educational Use. Am. J. Pharm. Educ. 2014, 78, 168. [Google Scholar] [CrossRef] [Green Version]
  35. Korenoski, A.S.; Ginn, T.R.; Seybert, A.L. Use of an Immersive, Simulated Learning Game to Teach Pharmacy Students Clinical Concepts of Toxicology. Curr. Pharm. Teach. Learn. 2021, 13, 556–559. [Google Scholar] [CrossRef]
  36. Khalafalla, F.G.; Alqaysi, R. Blending Team-Based Learning and Game-Based Learning in Pharmacy Education. Curr. Pharm. Teach. Learn. 2021, 13, 992–997. [Google Scholar] [CrossRef]
  37. Cain, J.; Piascik, P. Are Serious Games a Good Strategy for Pharmacy Education? Am. J. Pharm. Educ. 2015, 79, 47. [Google Scholar] [CrossRef] [Green Version]
  38. Thompson, M.; Kanmaz, T.; Cheryl, C.; Tanzer, K. Recommendations for Participation in Experiential Education Activities during a National, Regional or Local Crisis 2020. Available online: https://www.acpe-accredit.org/wp-content/uploads/Experiential-Education-ACPE-Proposal-Response-Final.pdf (accessed on 28 May 2022).
  39. Lucas, C.; Williams, K.; Tudball, J.; Walpola, R.L. Community, Hospital and Industry Preceptor Perceptions of Their Role in Experiential Placements- the Need for Standardization of Preceptor Responsibilities and Evaluations on Students. Curr. Pharm. Teach. Learn. 2018, 10, 1447–1455. [Google Scholar] [CrossRef] [PubMed]
  40. Wilbur, K.; Wilby, K.J.; Pawluk, S. Pharmacy Preceptor Judgments of Student Performance and Behavior During Experiential Training. Am. J. Pharm. Educ. 2018, 82, 6451. [Google Scholar] [CrossRef] [PubMed]
  41. Mort, J.R.; Johnson, T.J.; Hedge, D.D. Impact of an Introductory Pharmacy Practice Experience on Students’ Performance in an Advanced Practice Experience. Am. J. Pharm. Educ. 2010, 74, 11. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  42. Al-Azar, M. AUB Study: Gamification Improves Active Learning and Class Engagement in Higher Education 2015. Available online: https://www.aub.edu.lb/communications/media/Documents/feb-15/gamification-highered-EN.pdf (accessed on 28 May 2022).
  43. Bangalee, V.; Oosthuizen, F.; Perumal-Pillay, V.A.; Suleman, F.; Walters, F. Pharmacy Students Experience with PharmacyPhlash—A Pilot Educational Board Game. Curr. Pharm. Teach. Learn. 2021, 13, 292–301. [Google Scholar] [CrossRef]
  44. Whitman, A.C.; Tanzer, K.; Nemec, E.C. Gamifying the Memorization of Brand/Generic Drug Names. Curr. Pharm. Teach. Learn. 2019, 11, 287–291. [Google Scholar] [CrossRef]
  45. Legaki, N.-Z.; Xi, N.; Hamari, J.; Karpouzis, K.; Assimakopoulos, V. The Effect of Challenge-Based Gamification on Learning: An Experiment in the Context of Statistics Education. Int. J. Hum.-Comput. Stud. 2020, 144, 102496. [Google Scholar] [CrossRef]
  46. Akel, M.E.; Rahal, M.; Dabbous, M.; Mourad, N.; Dimassi, A.; Sakr, F. Experiential Education in Pharmacy Curriculum: The Lebanese International University Model. Pharmacy 2020, 9, 5. [Google Scholar] [CrossRef]
  47. Devine, P.S.; Darbishire, P.L. National Trends in IPPE Programs at US Schools of Pharmacy from 2008–2013. Am. J. Pharm. Educ. 2015, 79, 39. [Google Scholar] [CrossRef] [Green Version]
  48. Parinyarux, P.; Dhippayom, T.; Wongpoowarak, P.; Kitikannakorn, N. Development of Community Pharmacy Competencies. J. Pharm. Technol. 2022, 38, 875512252210813. [Google Scholar] [CrossRef]
  49. Von Wangenheim, C.G.; Savi, R.; Borgatto, A.F. DELIVER!—An Educational Game for Teaching Earned Value Management in Computing Courses. Inf. Softw. Technol. 2012, 54, 286–298. [Google Scholar] [CrossRef]
  50. Jones, J.S.; Tincher, L.; Odeng-Otu, E.; Herdman, M. An Educational Board Game to Assist PharmD Students in Learning Autonomic Nervous System Pharmacology. Am. J. Pharm. Educ. 2015, 79, 114. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  51. Lam, J.T.; Gutierrez, M.A.; Goad, J.A.; Odessky, L.; Bock, J. Use of Virtual Games for Interactive Learning in a Pharmacy Curriculum. Curr. Pharm. Teach. Learn. 2019, 11, 51–57. [Google Scholar] [CrossRef] [PubMed]
  52. Lee, C.Y.; White, P.J.; Malone, D.T. Online Educational Games Improve the Learning of Cardiac Pharmacology in Undergraduate Pharmacy Teaching. Pharm. Educ. 2018, 18, 298–302. [Google Scholar]
  53. Tavakol, M.; Dennick, R. Making Sense of Cronbach’s Alpha. Int. J. Med. Educ. 2011, 2, 53–55. [Google Scholar] [CrossRef]
  54. Patel, J. Using Game Format in Small Group Classes for Pharmacotherapeutics Case Studies. Am. J. Pharm. Educ. 2008, 72, 21. [Google Scholar] [CrossRef] [Green Version]
  55. Roche, V.; Alsharif, N.; Ogunbadeniyi, A. Reinforcing the Relevance of Chemistry to the Practice of Pharmacy Through the Who Wants to Be a Med Chem Millionaire? Learning Game. Am. J. Pharm. Educ. 2004, 68, 116. [Google Scholar] [CrossRef]
  56. Shah, S.; Lynch, L.M.J.; Macias-Moriarity, L.Z. Crossword Puzzles as a Tool to Enhance Learning about Anti-Ulcer Agents. Am. J. Pharm. Educ. 2010, 74, 117. [Google Scholar] [CrossRef] [Green Version]
  57. Neef, N.A.; Perrin, C.J.; Haberlin, A.T.; Rodrigues, L.C. Studying as Fun and Games: Effects on College Students’ Quiz Performance. J. Appl. Behav. Anal. 2011, 44, 897–901. [Google Scholar] [CrossRef] [Green Version]
  58. da Rocha Seixas, L.; Gomes, A.S.; de Melo Filho, I.J. Effectiveness of Gamification in the Engagement of Students. Comput. Hum. Behav. 2016, 58, 48–63. [Google Scholar] [CrossRef]
  59. Kapp, K. The Gamification of Learning and Instruction: Game-Based Methods and Strategies for Training and Education; Pfeiffer: San Francisco, CA, USA, 2012; ISBN 978-1-118-09634-5. [Google Scholar]
  60. O’Donovan, S.; Gain, J.; Marais, P. A Case study in the gamification of a university-level games development course. In Proceedings of the SAICSIT ’13 South African Institute for Computer Scientists and Information Technologists Conference, East London, South Africa, 7–9 October 2013; p. 251. [Google Scholar]
Education 12 00434 g001 550
Figure 1. ROC curve of ALMAS for game-based learning.
Figure 1. ROC curve of ALMAS for game-based learning.
Education 12 00434 g001
Table
Table 1. Baseline characteristics of the students.
Table 1. Baseline characteristics of the students.
VariableTotal Sample
N = 233
Mean (SD) or
N (%)		Game-Based Learning Group
N = 69 (29.6%)
Mean (SD) or
N (%)		Traditional Learning Group
N = 164 (70.4%)
Mean (SD) or
N (%)		p Value
Age23.16 (1.82)23.49 (2.05)23.02 (1.7)0.075
Gender
Male
Female
70 (30.0)
163 (70.0)
17 (24.6)
52 (75.4)
53 (32.3)
111 (67.7)		0.243
GPA3.14 (0.35)3.12 (0.40)3.14 (0.34)0.707
Area of residence
Beirut
Bekaa
Mount Lebanon
North of Lebanon
South of Lebanon
84 (36.1)
50 (21.5)
32 (13.7)
26 (11.2)
41 (17.6)
31 (44.9)
13 (18.8)
8 (11.6)
7 (10.1)
10 (14.5)
53 (32.3)
37 (22.6)
24 (14.6)
19 (11.6)
31 (18.9)		0.492
Pre-game-based-learning averages
Total average
Averages of domains
Foundational knowledge
Pharmaceutical care
Essentials for practice and care
Approach to practice and care
65.59 (18.86)

13.68 (4.15)
13.79 (4.15)
20.59 (6.62)
17.53 (7.07)
66.86 (20.10)

13.27 (4.51)
13.73 (4.27)
19.23 (7.18)
20.64 (6.93)
65.04 (18.34)

13.85 (4.0)
13.82 (4.11)
16.79 (6.91)
20.58 (6.50)
0.513

0.347
0.875
0.019
0.950
Table
Table 2. Comparison of class averages between game-based learners and traditional learners.
Table 2. Comparison of class averages between game-based learners and traditional learners.
Class AverageGame-Based LearnersTraditional LearnersMean
Difference		95% Confidence Intervalp Value
Mean (SD)Mean (SD)LowerUpper
Total81.77 (9.81)74.62 (9.64)7.1524.4149.891<0.001
Foundational knowledge17.41 (2.49)16.04 (2.60)1.3690.6442.095<0.001
Pharmaceutical care15.22 (3.15)13.79 (3.26)1.4310.5192.3430.002
Essentials for practice and care23.90 (4.29)22.05 (4.24)1.850.6473.0520.03
Approach to practice and care25.25 (4.50)22.74 (4.63)2.5021.2043.801<0.001
Table
Table 3. Promax rotated matrix of the active learning motivation assessment scale (ALMAS).
Table 3. Promax rotated matrix of the active learning motivation assessment scale (ALMAS).
FactorFactor 1 LoadingFactor 2 LoadingCommunalities
Beneficial for acquiring the desired knowledge more easily0.911 0.685
Beneficial for acquiring the intended learning outcomes0.886 0.790
Helped to learn course content0.886 0.762
Helped to connect ideas in new ways0.876 0.681
Helped to apply the course content0.831 0.797
Helped to develop confidence0.803 0.648
Enhanced learning experience and made it easier0.791 0.592
Helped to participate in course activities that enhanced learning0.765 0.756
Beneficial for incorporating additional information0.530 0.495
Greater attention to tasks 0.9630.671
Motivated learning the course material more than traditional 0.8780.708
Recommended to be part of the course 0.6410.584
Provided opportunity to practice and improve competencies 0.6040.743
Important supplement to this class 0.5340.723
Percentage of variances explained60.665%8.142%
Percentage of the total variance explained: 68.81%. Kaiser–Meyer–Olkin (KMO) = 0.889. Bartlett’s test of sphericity: p < 0.001.
Table
Table 4. Correlation of each item of the ALMAS with the whole scale.
Table 4. Correlation of each item of the ALMAS with the whole scale.
Itemr *p Value
Beneficial for acquiring the desired knowledge more easily0.833<0.001
Beneficial for acquiring the intended learning outcomes0.908<0.001
Helped to learn course content0.893<0.001
Helped to connect ideas in new ways0.859<0.001
Helped to apply the course content0.908<0.001
Helped to develop confidence0.845<0.001
Enhanced learning experience and made it easier0.829<0.001
Helped to participate in course activities that enhanced learning0.914<0.001
Beneficial for incorporating additional information0.812<0.001
Greater attention to tasks0.653<0.001
Motivated learning the course material more than traditional0.775<0.001
Recommended to be part of the course0.813<0.001
Provided opportunity to practice and improve competencies0.876<0.001
Important supplement to this class0.886<0.001
* Pearson correlation coefficient.
Table
Table 5. Multivariable linear regression taking the post-game-based-learning total average as the dependent variable.
Table 5. Multivariable linear regression taking the post-game-based-learning total average as the dependent variable.
VariableBeta95% Confidence Intervalp Value
LowerUpper
ALMAS for game-based learning0.2960.1100.5450.004
Pre-game-based-learning total average0.029−0.0870.1180.767
GPA0.3323.14715.3220.003
Age−0.116−1.6140.4820.286
Gender−0.253−10.150−1.3390.011
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Dabbous, M.; Kawtharani, A.; Fahs, I.; Hallal, Z.; Shouman, D.; Akel, M.; Rahal, M.; Sakr, F. The Role of Game-Based Learning in Experiential Education: Tool Validation, Motivation Assessment, and Outcomes Evaluation among a Sample of Pharmacy Students. Educ. Sci. 2022, 12, 434. https://doi.org/10.3390/educsci12070434

AMA Style

Dabbous M, Kawtharani A, Fahs I, Hallal Z, Shouman D, Akel M, Rahal M, Sakr F. The Role of Game-Based Learning in Experiential Education: Tool Validation, Motivation Assessment, and Outcomes Evaluation among a Sample of Pharmacy Students. Education Sciences. 2022; 12(7):434. https://doi.org/10.3390/educsci12070434

Chicago/Turabian Style

Dabbous, Mariam, Anwar Kawtharani, Iqbal Fahs, Zahraa Hallal, Dina Shouman, Marwan Akel, Mohamad Rahal, and Fouad Sakr. 2022. "The Role of Game-Based Learning in Experiential Education: Tool Validation, Motivation Assessment, and Outcomes Evaluation among a Sample of Pharmacy Students" Education Sciences 12, no. 7: 434. https://doi.org/10.3390/educsci12070434

APA Style

Dabbous, M., Kawtharani, A., Fahs, I., Hallal, Z., Shouman, D., Akel, M., Rahal, M., & Sakr, F. (2022). The Role of Game-Based Learning in Experiential Education: Tool Validation, Motivation Assessment, and Outcomes Evaluation among a Sample of Pharmacy Students. Education Sciences, 12(7), 434. https://doi.org/10.3390/educsci12070434

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop