Meta-Analysis of Flipped Learning Effects in Nursing Education

Background: This study is a meta-analysis confirming the effect size of clinical competence, critical thinking ability, self-directedness, and learning satisfaction, the outcome variables of flipped learning applied to nursing education. Methods: We selected 18 related studies that analyzed data using CMA (Comprehensive Meta-Analysis 2.2). Results: The effect size of the entire study was Hedges’ g = 0.68 (95% CI = 0.43~0.92). The heterogeneity of the overall effect size was I2 = 90.7% (Q = 246.67, p < 0.001); critical thinking ability had an effect size of Hedges’ g = 0.87, learning satisfaction of Hedges’ g = 0.79, clinical competence of Hedges’ g = 0.53, and self-directedness of Hedges’ g = 0.37. The differences were statistically significant. Conclusion: Flipped learning can effectively improve nursing students’ clinical competence, critical thinking ability, self-direction, and learning satisfaction.


Introduction
Recently, flipped learning education has been applied to increase the efficiency of education and learning processes due to rapid changes in the educational environment and to convert students into active learners [1,2].
Flipped learning is a learner-centered teaching method in which learners self-learn online before class, asking questions to the instructor and discussing the content with peers during class [3]. This provides a dynamic mutual learning environment for professors and allows students to focus creatively on their learning needs without time and space constraints [4]. In flipped learning, students manage the learning process through learning content before class, so students' self-discipline and self-regulation can be improved [5]. Additionally, during this class, students can participate in various active learning activities, such as question-and-answer, discussion, group activities, problem-solving, and receiving immediate feedback [6]. A positive change in academic achievement levels has been observed in medical education with an increasing interest in flipped learning classes and reducing boredom [7].
The first study that applied flip learning in the field of nursing education was published in 2013 [8]. A flipped learning model has been applied in nursing education to reinforce nursing competence according to changes in the clinical environment. It is difficult to acquire sufficient knowledge within limited class hours and clinical practice with the existing teaching methods. This is due to the lack of both direct nursing experience in observation-oriented practice and a connection between the theory and clinical practice. Upon employment in a hospital, a gap is noticed in performing practical tasks in clinical nursing [9]. Thus, nursing educators applied flipped learning to improve the quality of education and prepare for complex nursing practice [1,6]. Flipped learning applied to nursing education has a large amount of learning and professional content, and A total of 18 studies were included in the systematic review per the data selection criteria (illustrated in Figure 1). In the first stage, 681 articles were searched using the search strategy for each database. The overlapping 314 episodes were excluded using the bibliographic management program. In the second step, 135 papers were selected by reviewing the titles and abstracts of 232 papers. In the third stage, the original texts of the 135 selected papers were reviewed, and the criteria for exclusion of the research subject and research design were applied. Of these, four non-original papers, one non-English paper, 27 non-conforming research results, six non-nursing students, and 59 studies where the research design did not match the criteria were excluded. Twenty studies were selected from 115 studies. In the fourth stage, the full texts of the 20 studies were reviewed. Two of them were excluded because the result value could not be extracted due to insufficient statistical data, and the final 18 studies were confirmed for meta-analysis.

Characteristics of documents subject to systematic literature review
The research design of the 18 papers selected for the analysis of this study wa 88.9% and RCT 11.1%, and the publication type was journal 88.9% and thesis 1 for the subject type applied to the intervention, practical subjects accounted for at 50%, and video (video content, video learning, lecture video) as the pre-ed method was 72.2%. The sample size range of the experimental group was 22 to control group was 21 to 198, and the number of interventions was 4 to 14. The p application time was 2-3 h per session, but there were studies that did not desc intervention method in detail. The proportion of outcome variables was 38.9% fo performance, 27.8% for critical thinking ability, 27.8% for self-direction, and 3 learning satisfaction (Table 1).

Evaluation of Literature Quality
Considering the results of the 18 methodological quality assessments in th 50% of the studies had a low RoB in randomization order in two RCT papers concealment items in the order of assignment, the low RoB was 50%, and the i blindfolding of researchers had 100% uncertain RoB. In the case of blindfolding an tive reporting of results, the low RoB was 100%, and the high RoB in insufficie data was 100%. In the 16 NRCT papers, the low bias was 100% in the subject gro parability, exposure measurement, result evaluation, and selective result reportin lecting a target group, the low RoB was 50%, the high RoB was 37.5%, and the unc of the RoB was 12.5%. Regarding the confounding variables, the low RoB was 81.2 the high RoB was 18.75%. In the evaluator's blindfold, the low RoB was 25%, and certainty RoB was 75%. The unsafe result data showed that the low RoB was 25% high RoB was 68.75%; the uncertainty of the RoB was 6.25%. A single group was e from the literature selection criteria, and papers with clear pre-and post-evaluat those planned were included in the literature containing all expected results. The of the papers was found to be higher than average because of the overall evaluatio

Characteristics of Documents Subject to Systematic Literature Review
The research design of the 18 papers selected for the analysis of this study was NRCT 88.9% and RCT 11.1%, and the publication type was journal 88.9% and thesis 11.1%. As for the subject type applied to the intervention, practical subjects accounted for the most at 50%, and video (video content, video learning, lecture video) as the pre-education method was 72.2%. The sample size range of the experimental group was 22 to 287, the control group was 21 to 198, and the number of interventions was 4 to 14. The program application time was 2-3 h per session, but there were studies that did not describe the intervention method in detail. The proportion of outcome variables was 38.9% for clinical performance, 27.8% for critical thinking ability, 27.8% for self-direction, and 33.3% for learning satisfaction (Table 1).

Evaluation of Literature Quality
Considering the results of the 18 methodological quality assessments in this study, 50% of the studies had a low RoB in randomization order in two RCT papers. For the concealment items in the order of assignment, the low RoB was 50%, and the items for blindfolding of researchers had 100% uncertain RoB. In the case of blindfolding and selective reporting of results, the low RoB was 100%, and the high RoB in insufficient result data was 100%. In the 16 NRCT papers, the low bias was 100% in the subject group comparability, exposure measurement, result evaluation, and selective result reporting. In selecting a target group, the low RoB was 50%, the high RoB was 37.5%, and the uncertainty of the RoB was 12.5%. Regarding the confounding variables, the low RoB was 81.25%, and the high RoB was 18.75%. In the evaluator's blindfold, the low RoB was 25%, and the uncertainty RoB was 75%. The unsafe result data showed that the low RoB was 25%, and the high RoB was 68.75%; the uncertainty of the RoB was 6.25%. A single group was excluded from the literature selection criteria, and papers with clear pre-and postevaluations and those planned were included in the literature containing all expected results. The quality of the papers was found to be higher than average because of the overall evaluation (Table 1) Kim OS. (2020) [31] Journal TAU 22 26 10 Basic Nursing Practice Jo YS. (2020) [32] Thesis TAU 27 26 4 Emergency Nursing Journal TAU 22 26 10 Basic Nursing Practice Jo YS. (2020) [32] Thesis TAU 27 26 4 Emergency Nursing Journal TAU 22 26 10 Basic Nursing Practice Jo YS. (2020) [32] Thesis TAU 27 26 4 Emergency Nursing Kim OS. (2020) [31] Journal TAU 22 26 10 Basic Nursing Practice Jo YS. (2020) [32] Thesis TAU 27 26 4 Emergency Nursing Kim OS.
(2020) [31] Journal TAU 22 26 10 Basic Nursing Practice Kim OS. (2020) [31] Journal TAU 22 26 10 Basic Nursing Practice                     Table 2 and the forest plots present the effect size of flipped learning applied to nursing education on clinical competence, critical thinking ability, learning satisfaction, and self-directedness ( Figure 2). The effect size of the entire study was Hedges' g = 0.68 (95% CI = 0.43~0.92), showing a statistically significant median effect size. The heterogeneity of the total effect size was I 2 = 90.7% (Q = 246.67, p < 0.001), appearing to fall within a large range [18]. The effect size of each effect variable of the flipped learning intervention was statistically significant because the 95% CI excluded 0, and the critical thinking ability was Hedges' g = 0.87, showing the largest effect size. Next, learning satisfaction (Hedges' g = 0.79), clinical competence (Hedges' g = 0.53), and self-directedness (Hedges' g = 0.37) are shown in Table 2.

Effect Size of Flipped Learning
By verifying the intervention effect, the subject type and the prior learning method type were used as modulators. In the subject method type, the effect size value is calculated by the practical subjects, the practical and theoretical integrated subjects, and the theoretical subjects. In the practical subjects had Hedges' g = 0.95, the practical and theoretical integrated subjects had Hedges' g = 0.36, and the theoretical subjects had Hedges' g = 0.30. According to the subject type, there was a significant difference in the flipped learning effect (Q = 13.59, p = < 0.001). In the prior learning method type, as the effect size value is calculated by classifying the video method and the method where the video and lecture book were given simultaneously, the Hedges' g was 0.71 for the video method and 0.45 for the video and lecture book; however, there was no statistically significant difference (Q = 1. The heterogeneity of the total effect size was I 2 = 90.7% (Q = 246.67, p < 0.001), appearing to fall within a large range [18]. The effect size of each effect variable of the flipped learning intervention was statistically significant because the 95% CI excluded 0, and the

Publication Bias
The publication bias of this study was analyzed by Egger's regression test. As a result of statistically checking the significance of the degree of asymmetry, it was found that there was no publication bias (t = 0.82, p = 0.42) (Figure 3). g = 0.30. According to the subject type, there was a significant difference in the flipped learning effect (Q = 13.59, p = < 0.001). In the prior learning method type, as the effect size value is calculated by classifying the video method and the method where the video and lecture book were given simultaneously, the Hedges' g was 0.71 for the video method and 0.45 for the video and lecture book; however, there was no statistically significant difference (Q = 1.22, p = 0.269).

Publication Bias
The publication bias of this study was analyzed by Egger's regression test. As a result of statistically checking the significance of the degree of asymmetry, it was found that there was no publication bias (t = 0.82, p = 0.42) (Figure 3).

Discussion
This study systematically examined and meta-analyzed the effects of flipped learning on clinical competence, critical thinking ability, self-directedness, and learning satisfaction applied to nursing education to present the effects in an integrated and objective manner. Therefore, the main findings are as follows:

Discussion
This study systematically examined and meta-analyzed the effects of flipped learning on clinical competence, critical thinking ability, self-directedness, and learning satisfaction applied to nursing education to present the effects in an integrated and objective manner. Therefore, the main findings are as follows: The selected 18 studies considered flipped learning as the intervention group and traditional lecture-style learning as the control group. Of these, the two (11.1%) randomized controlled pre-and post-design studies using an RCT design were conducted overseas. Sixteen pieces (88.9%) were pre-post designs for the non-equivalent control group using the NRCT design, where six were conducted overseas and ten were conducted domestically.
Considering the items with high RoB or uncertain RoB in the methodological quality evaluation of theses, seven papers (38%) in selection bias were not randomly assigned. Part 2 (12.5%) did not present a specific method for generating the randomization order, and in Part 3 (16.6%), the order of allocation was not concealed. Thus, the researcher could predict the intervention group. Additionally, in terms of performance bias, no specific method for blinding research participants and researchers was presented in 14 papers (77.7%). In flipped learning, as instructor intervention is required, there are limitations to blindfolding participants and researchers, influencing the meta-analysis results. Despite the limitations in RCT research methods in flipped learning, various approaches to verify the quality of learning methods must be sought.
Regarding the subject types that applied flipped learning, practical subjects were the highest, with nine studies (50%), followed by seven theoretical subjects (38.9%) and two theory and practical subjects (11.1%). The theoretical subjects were critical thinking, nursing process, pathophysiology, health assessment, and patient safety, and the practical subjects were simulation practice, basic nursing practice, and clinical nursing practice.
Among the analysis papers, when applying flipped learning in practical subjects, nursing techniques cannot be directly performed. Thus, the methods and procedures of nursing techniques are repeatedly learned through audio-visual data [11]. Video content was used [19] or developed and applied by the instructor [31]. As the learning experience provided by the medium is different, the instructors set a clear goal through flipped learning, and the curiosity and motivation of learners were met.
As a result of the meta-analysis of flipped learning studies applied to 18 studies, the overall average effect size was 0.68, corresponding to the median effect size. This is similar to the study showing the median effect size of 0.58 as the overall average effect size of the learning effects of domestic flipped learning [40] and the study showing a median effect size of 0.59 as the effect size of flipped learning on domestic college students [41]. It was confirmed that flipped learning was more effective than traditional lecture-style classes when applied to nursing education.
As a result of comparing the effect sizes of the flipped learning outcome variables in this study, they were found in this order of critical thinking ability, learning satisfaction, clinical competence, and self-direction.
The critical thinking ability had a Hedges' g of 0.87, showing a large effect size and a high level of heterogeneity. In five studies, the experimental group to which flipped learning was applied showed improved critical thinking ability compared to the control group. It can be predicted that flipped learning, which allows students to ask questions and solve problems on their own through prior learning and to improve critical thinking skills through discussion with other students during class in the classroom [42], is effective in nursing education. Additionally, a previous study [19] showed that although it is possible to acquire knowledge and skills, there is a limit to critical thinking. Instructors must grasp the learner's disposition, attitude, and learning strategy and find and search for information rather than memorize knowledge; based on this, it is necessary to develop a running program.
As a result of analyzing the effect of flipped learning on learning satisfaction, the intermediate effect size was found to be Hedges' g = 0.79. When learning flipped learning, information was provided to students, such as videos, animations, and images of clinical cases, inducing interest and resulting in positive responses. Satisfaction was high, as this allowed flexibility to learn at a desired speed and time [43,44]. However, there have also been studies with low learning satisfaction due to tasks assigned from prior learning during flipped learning, online environmental problems for learning, and adaptation to new teaching methods [8]. The instructor should consider the learner's learning environment when developing flipped learning activities. It is necessary to consider avoiding a learning gap, such as understanding the infrastructure for watching videos and prior learning, and it is necessary to analyze the learning conditions in detail per the learning goal. In other words, it is necessary to promote understanding of learning by providing students with a plan for the learning content and learning method of flipped learning in advance so that they can familiarize themselves with the content to be learned. Additionally, the learner's learning process was constantly monitored. A strategy for linking the content of prior learning and offline activities is needed by providing feedback on prior learning in the classroom.
As a result of analyzing whether flipped learning is more effective than traditional methods in improving nursing students' clinical nursing skills, the effect size of clinical competence had a Hedges' g of 0.53, showing a medium-sized effect. This confirmed that learning the procedure and techniques through videos was effective [45], and it was also confirmed by another study [46] that the clinical performance was improved by flipped learning in clinical practice. Nursing is a practical study. Direct nursing skills and repetitive practice are required to acquire and improve such skills. However, it is difficult to obtain qualitative improvement because of the inability to perform direct skills during flipped learning. When flipped learning is applied in practical classes, nursing skills and nursing theories are learned based on evidence, and in classroom sessions, the contents of prior learning can be directly performed. Additionally, it effectively improves students' nursing skills through immediate feedback from instructors [47]. Factors for achieving clinical performance include interpersonal relationships and communication, nursing process, critical thinking, nursing intervention, creative thinking, basic nursing, and problem-solving ability [48]. These are improved by applying flipped learning to the subject in the corresponding domain, helping them to effectively perform the clinical nursing skills corresponding to the detailed elements.
In this study, the effect size of self-directedness was small (Hedges' g = 0.37). The most important aspect of flipped learning is to increase the degree of self-directedness, and in many research results, self-directedness was improved through flipped learning [49,50]. Learners watched video lectures on pre-learning to familiarize themselves with the learning content, and the activities in the classroom were focused on learners. Notably, the self-directedness is improved due to the increased sense of responsibility and activeness in all learning processes [51]. Self-directedness is an important variable affecting learning outcomes, such as learning motivation, academic achievement, and satisfaction [52]. Through flipped learning, the instructors foster the learner's learning ability and act as facilitators such that learners can have flexibility and develop lifelong learning habits for self-directedness [53]. Accordingly, when the learner's characteristics and teaching methods are matched, self-directedness is achieved successfully. When designing a flipped learning curriculum, it is necessary to develop educational methods to improve self-directedness ability by focusing on learner-centered education.
Other variables related to the subject type and pre-learning method were the modulating variables that explained the difference in effect size. As a result of meta-ANOVA using the subject type as the moderator variable, the following were practical subjects, theory and practice mixed design subjects, and theory subjects.These results showed a significant difference in the flipped learning effect according to the subject type (Q = 13.59, p ≤ 0.001), and it was confirmed that the effect of a flipped learning class was higher in practical subjects. In a practical subject, flipped learning helps one experience knowledge integrated with observation and experience [12]. The interaction between the instructor and students increases, and the nursing skills gradually improve [8]. Studies have confirmed this effect. Additionally, the effect size value was calculated by classifying the learning methods into video method (15 episodes) and the method where video and lecture books were given (five episodes) simultaneously, indicating that the learning by the video method was higher but not statistically significant (Q = 1.22, p = 0.269). This could be because when the video and lecture book were given simultaneously, the number of assignments to be learned increased, consequently increasing the learning burden, and the lack of time and grade were not reflected. It is necessary for the instructor to explain in advance how the pre-learning videos are related to the lecture contents in order for the students to plan an appropriate amount of pre-study.
It was difficult to measure various variables because there were differences in the specific research methods such as the flip learning education method applied to each study, the number of applications, and the time. In future research, it is necessary to improve the quality of research by describing specific research methods for teaching and learning designs applied to flipped learning classes, such as the characteristics of instructors and learners and the learning process.
Critical thinking, clinical reasoning, and clinical performance are required for nursing students to adapt to the rapidly changing nursing environment [14]. Applying flipped learning to equip self-directedness ability, students learn how to acquire abilities and competencies by applying various information rather than simply accumulating knowledge and facing problems in a medical nursing environment.
The limitations of this study are as follows. An extensive search on flipped learning was performed using flipped learning keywords and a series of other relevant keywords.
However, the possibility of publication bias cannot be excluded because the publication languages are limited to English and Korean.
This study selected RCT and NRCT research among the studies related to flipped learning applied to nursing education to increase the reliability of the research results. The academic significance included confirming the magnitude of the effect on clinical Competence, critical thinking ability, self-directedness, and learning satisfaction as outcome variables, an effect that was determined to exist.

Conclusions
Flipped learning applied to nursing education is a learning method that effectively improves nursing students' critical thinking ability, learning satisfaction, clinical competence, and self-directedness. In particular, it helps increase critical thinking ability and clinical competence necessary in nursing education, which occupies a large part of clinical practice education. Several studies have confirmed that flipped learning is being applied to practical education. However, the research results on students' prior learning methods, the time required, and educator preparation are insufficient. Therefore, detailed methods for future learning types should be developed, and research to prove their effectiveness should be conducted.
Therefore, it is recommended that flipped learning education be actively used as an effective nursing education teaching method. It is expected that continuous research on the effects and learning effects on students will be conducted.