Mapping of Scientific Production on Blended Learning in Higher Education

Blended learning is a set of pedagogical strategies and practices that are closely related to the emergence of new technologies in society and in higher education. They are characterized by a combination of face-to-face and online learning that integrates a variety of materials in the teaching and learning process. The aim of our research was to analyze scientific production on this topic using two prestigious international databases: Scopus and Web of Science (WOS). A bibliometric study was conducted on 508 papers, and content analysis was performed on 119 openly accessible publications. Results show that blended learning is being applied in all scientific and professional spheres. This learning model is praised for its positive impact on motivation and learning effectiveness, as well as for promoting student autonomy. With respect to content, it is worth noting that many authors observe that methodological success is related to cultural context and access to devices and materials. Furthermore, it is suggested that greater methodological diversity is preferable to a single hegemonic approach. In short, the present paper contributes to the educational community by shedding light on how blended learning is being implemented and its impact on higher education.


Introduction
The challenges of higher education are conditioned by historical circumstances, the evolution of research and innovation, the demands of an ever-changing labor market, and the economic situation at a particular time. All of these circumstances determine not only goals and objectives, but also the epistemological and methodological approaches guiding university study programs. Knowing how to teach and how higher education students learn are basic elements that need to be debated [1]. Several research projects have shown the need to connect teaching methodology to students' actual learning styles [2], since this can have an enormous impact on their motivation towards learning [3]. Previous research has highlighted the importance of integrating proposals based on active methodologies in order to favour the acquisition of instrumental, systemic, and personal competences [4], promote autonomous learning [5], foster classroom participation and use of technological tools [6], better apply gamification [7], and connect the teaching-learning process to real future professional work prospects as in the case of service-learning [8].
Research carried out over recent decades has revealed a need for combining methodologies. Higher education students demand active and oftentimes hybrid methodologies [9] tailored to their social and professional environment. A variety of teaching-learning scenarios and experiences must be offered, using diverse resources in which technology plays a relevant but not overriding role. In this context, the scientific literature points out that blended learning (BL) stands as an innovative pedagogical proposal for all educational stages, especially higher education [10]. BL in higher education is here understood to mean combining face-to-face and online learning, together with learning practices mediated by technology and other types of material.
BL is rooted in trends promoting active pedagogies, such as constructivism. It has become more visible in the global scientific community in recent years mainly due to the implementation of ICTs and LKTs (learning and knowledge technologies) at different educational stages. Current research has analyzed the contributions of various strategies following BL models. Nonetheless, some authors consider that implementing face-to-face practices in the classroom facilitates interactions and allows greater depth in the approach to certain contents compared to online proposals in BL [11].
Previous bibliographical reviews have highlighted the need to further explore BL, since scientific production has mostly focused on specific aspects, with the exception of research providing an overview of common BL implementation aspects in various contexts [12]. A study comprising 827 papers [13] concluded that there are two views regarding the role of technology in innovation. Some research has indicated that BL contributes to improving teaching-learning processes, while other research has found no significant differences with respect to alternative models. A study analyzing 45 publications [14] identified research trends between the year 2000 and 2016 [15], focusing on digital tools or platforms to foster interaction between people and electronic devices. Likewise, and to contextualize our analysis, other reviews have been analyzed including one involving books and chapters [16] and another based on 205 academic theses and final master's projects [17]. The research under consideration ranged from an older review identifying evaluation techniques such as self-test quiz tools, discussion forums and e-portfolios [18] to more recent studies which, despite involving smaller samples, have been of great help in broadening our analysis to consider student satisfaction, teacher commitment and the benefits of cooperation, resource exchange, the strengthening of communication between the actors of the formal and informal teaching-learning process, and the development of the cognitive process [19][20][21].
In light of the impact of higher education internationally, here we present a bibliographic and bibliometric analysis of the main research databases. The research objectives were as follows: (a) to analyze scientific impact, collaboration, scattering, and productivity, (b) to describe research evaluating both procedures and results of higher education teaching-learning based on blended learning, and (c) to identify the limitations of BL studies insofar as creating science-based guidelines. The first objective was addressed from a bibliometrics perspective, while the other two were studied through content analysis.

Methodology
Our work consisted of two studies: a bibliometric analysis performed from a scientometrics perspective, and a content analysis revealing the limitations of research on the subject.
The search ran up to 30 June 2020. Studies focusing on the analysis of BL in higher education were selected after screening for inclusion and exclusion criteria (Table 1) adhering to common standards listed in previous studies [22]. Institutions with more than 5 papers were analysed. Journals Journals with more than 10 papers were analysed. Languages Keywords in English, Spanish, Portuguese, and French.

Citations
Papers with more than 150 citations were analysed. Authors Authors with more than 10 articles were identified.

Bibliometric map
Including keywords with more than 10 occurrences.
The WOS and Scopus databases were chosen due to their inclusiveness and recognized international prestige [23]: they incorporate journals of proven quality and scientific rigor, have a wider representation in terms of linguistic diversity and countries of origin, and have an extensive collection of open access publications.
The process, separated into four stages following the PRISMA Statement [24], is shown in Figure 1.
Educ. Sci. 2021, 11, x FOR PEER REVIEW Institutions with more than 5 papers were analysed Journals Journals with more than 10 papers were analysed. Languages Keywords in English, Spanish, Portuguese, and Frenc Citations Papers with more than 150 citations were analysed. Authors Authors with more than 10 articles were identified.

Bibliometric map
Including keywords with more than 10 occurrences The process, separated into four stages following the PRISMA Statement shown in Figure 1. The corpus of identified papers was reviewed by two researchers, yielding degree of agreement and a good Cohen's kappa coefficient (k = 0.82). The analysi 508 papers (465 from Scopus and 43 from WOS) was based on a series of indicat The corpus of identified papers was reviewed by two researchers, yielding a high degree of agreement and a good Cohen's kappa coefficient (k = 0.82). The analysis of the 508 papers (465 from Scopus and 43 from WOS) was based on a series of indicators described in previous research [25], namely productivity, impact, collaboration, and scattering.
Content analysis was performed on the 119 openly available publications (111 from Scopus and 8 from WOS). After comprehensively reading all the papers for the bibliometric study and examining the various topics addressed in this field of research, the data were fragmented into minimal meaningful units and classified into four categories based on meaning similarities and differences for each of the categories that emerged [26]. Based on these criteria, three researchers discussed and interpreted the categorized data and reached a consensus, thus increasing internal validity and verification control to ensure data quality.
To provide a comprehensive understanding of the content, an ad hoc record sheet was used [27]. It considered four categories: analysis of models, designs, and development; model comparison; implications for the participants; and agents' evaluation.
The graphic representation of the results was created using VOSviewer 1.6.11 [28].

Bibliometric Analysis
Firstly, scientific production was analyzed to obtain relevant information regarding impact, collaboration, scattering, and productivity in studies on the use of BL models in higher education.
In terms of citation impact, 21.7% of the analyzed papers had no citations, while four had over 150 citations [29][30][31][32]. Only two papers had been referenced more than 200 times [33,34], accounting for 2% of total citations. In addition, it was possible to identify key researchers in the field such as Graham, Moskal, and Garrison.
Regarding impact factor, we should point out that 1.7% of the papers had values greater than 10. Among these, several studies [32,35] had factors greater than 16, which is particularly noteworthy in the case of research only recently published. Table 2 shows the journals that published 10 or more papers on the topic and their impact factor. Regarding collaboration, the importance of co-authorship networks is evidenced by the number of authors signing each paper. On most occasions, these networks were made up of researchers from the same institution or territory. Most documents were written by more than one author (78.3%), and two studies had more than ten authors [36,37].
In terms of scattering analysis, we identified three groups among the 508 papers published in 190 journals, and these followed Bradford's law. This law relates the number of articles and journals, establishing that most of the articles published on a subject tend to be concentrated in a small number of journals. As seen in Figure 2, the core of the sample consisted of half of the papers analyzed (n = 255), which were published in a small number of journals (n = 36), thus confirming the law. Finally, scientific productivity was analyzed by considering area o and place of publication, institution, type of producer, and language.
Over the last decade, papers about BL in higher education have be very different areas, including social science (59%), computational scienc gineering (8%).
In terms of number, more than 20 papers have been published per ye most productive year with 98 publications, representing 19.3% of the tot with Price's Law indicating that scientific production doubles every 10 2010 documents represent only 14.21% of the corpus, and there has b growth in publications on BL in higher education over the last decade.
Regarding the country of origin, it is certainly noteworthy that this su of study in almost a hundred countries. In terms of bibliometric size, coun USA, the United Kingdom, Australia, Spain, China, and Malaysia stand account each country's size, population, number of higher education ce the case of Malaysia, Australia, and Spain are especially noteworthy.
In relation to the number of publications per institution, five univer more than five papers each. The first of these is Vrije Universiteit Brus followed by Universiti Teknologi MARA, in Malaysia. Brigham Young U and Griffith University (Australia) share the third position, and Kazan Fe (Russia) also stands out as a prominent institution on the subject.
A total of 1187 authors signed the 508 manuscripts in the sample. L productivity law states that a small number of authors specializing in a Finally, scientific productivity was analyzed by considering area of research, date and place of publication, institution, type of producer, and language.
Over the last decade, papers about BL in higher education have been published in very different areas, including social science (59%), computational science (16%), and engineering (8%).
In terms of number, more than 20 papers have been published per year. 2019 was the most productive year with 98 publications, representing 19.3% of the total. This is in line with Price's Law indicating that scientific production doubles every 10 to 15 years: pre-2010 documents represent only 14.21% of the corpus, and there has been exponential growth in publications on BL in higher education over the last decade.
Regarding the country of origin, it is certainly noteworthy that this subject is the focus of study in almost a hundred countries. In terms of bibliometric size, countries such as the USA, the United Kingdom, Australia, Spain, China, and Malaysia stand out. Taking into account each country's size, population, number of higher education centers, and GDP, the case of Malaysia, Australia, and Spain are especially noteworthy.
In relation to the number of publications per institution, five universities published more than five papers each. The first of these is Vrije Universiteit Brussel, in Belgium, followed by Universiti Teknologi MARA, in Malaysia. Brigham Young University (USA) and Griffith University (Australia) share the third position, and Kazan Federal University (Russia) also stands out as a prominent institution on the subject.
A total of 1187 authors signed the 508 manuscripts in the sample. Lotka's personal productivity law states that a small number of authors specializing in a subject tend to have the highest number of publications.

Content Analysis
The second study aim was to analyze the content of open access papers. Four categories were established after reading the complete papers, taking into account their area of research and results.
To begin with, a fractional counting keywords co-occurrence map was developed in VOSviewer ( Figure 3). As can be observed, the lines indicate the trend of joint appearance of keywords in the analyzed papers. The 21 keywords are grouped into 4 large clusters. Some terms such as education, teaching, learning, curriculum, students, and motivation represent key elements in school organization and didactics.

Content Analysis
The second study aim was to analyze the content of open access papers. Four categories were established after reading the complete papers, taking into account their area of research and results.
To begin with, a fractional counting keywords co-occurrence map was developed in VOSviewer ( Figure 3). As can be observed, the lines indicate the trend of joint appearance of keywords in the analyzed papers. The 21 keywords are grouped into 4 large clusters. Some terms such as education, teaching, learning, curriculum, students, and motivation represent key elements in school organization and didactics. Two lines of inquiry were identified within this category. The first focused on providing frameworks for helping to develop hybrid courses (face-to-face and online); the second analyzed course designs and developments from various universities and provided recommendations for facilitating the implementation of blended learning.
One of the standout studies on methods, standards, and algorithms for improving the online learning experience [38] described the results of implementing the open-source software Pinvox, which is used to check students' attention level during virtual lectures or lessons and improve educational evaluation. A similar study concerning evaluation Two lines of inquiry were identified within this category. The first focused on providing frameworks for helping to develop hybrid courses (face-to-face and online); the second analyzed course designs and developments from various universities and provided recommendations for facilitating the implementation of blended learning.
One of the standout studies on methods, standards, and algorithms for improving the online learning experience [38] described the results of implementing the open-source software Pinvox, which is used to check students' attention level during virtual lectures or lessons and improve educational evaluation. A similar study concerning evaluation [39] presented a blended learning framework to help higher education faculty self-assess their digital literacy competence before implementing BL.
Another study proposed using the SLAM framework (science learning activities model) [40] as a tool for helping to design science courses that motivate students and take into account the use of mobile phones, augmented reality, and virtual reality. Another interesting approach involved the construction of a coding scheme to analyze peer feedback in the Moodle forums of a blended university course [41]. Finally, an effectiveness analysis of the SBPBL model (sharing blended project based learning) [42] concluded that its application increased the use of e-learning resources and improved the motivation of both students and teachers.
In general, the authors of this line of research considered that the availability of tools to facilitate the design of blended learning courses was still limited.
The second line of identified research included papers focusing on the application of blended learning models in higher education environments that could serve as examples for institutions and faculty members interested in implementing BL. Some case studies praised the benefits of participatory design and persuasive technology for the development of blended learning courses [43], while others focused on analyzing the limitations, obstacles, and challenges of pilot programs and courses [44][45][46][47]. Some of the obstacles mentioned include the limited technological skills of older faculty members, the time required to develop online materials, the fear of failure, the lack of complete tutorials for each platform, problems related to registration and access, and institutional culture.
There were also papers presenting results from the implementation of blended learning using technologies such as Edmodo [48], Blackboard [49], and Facebook [29]. These studies tended to present a positive view of using these methods.

Model Comparison
Another identified category included studies comparing traditional learning models with blended learning. In this group, research examining blended and traditional environments for teaching English as a second language were common; they often compared data from an experimental group and a control group. One of these studies [50] concluded that university students had more positive attitudes and improved English learning performance with blended learning. Another study involving higher education English language students [51] concluded that students who learned with a blended learning method had significantly higher levels of intrinsic motivation for learning English, a better attitude towards the subject, and greater satisfaction with the learning environment than students who used a traditional face-to-face method. Similar results were reported by other research [52][53][54], including a study revealing that German language students who used blended learning environments were more successful and motivated than the control group which attended German lessons in a traditional learning environment.
Regardless of course content, most of these research projects indicated that blended learning models led to better student success compared to traditional teaching [55][56][57][58][59]. In a publication comparing learning and information literacy achievements among Thai student groups [56], it was found that students using blended learning understood the content better and enjoyed their increased autonomy. Other authors also reported students' appreciation of autonomy and intrinsic motivation [58]. In a pilot study with physical therapy and medical students [57], the blended learning model was more effective than the face-to-face method for teaching practices related to chronic patient care.
Also notable was the fact that student evaluation of teaching design and general satisfaction was reported to be higher in courses with fully online lectures [60].

Implications for Participants
The next line of research with relevant presence in the literature involved benefits of BL in the learning experiences of higher education students, which are reported worldwide. The significant relationship between the application of the blended learning model and an increase in HOTS (higher-order thinking skills) for students of vocational higher education (V-HOTS) was particularly noteworthy [61]. Critical thinking and creative thinking were examples of skills that benefited from this approach [62].
Generally speaking, various investigations described important benefits that could contribute to student training in scientific fields such as modern engineering [63] and business [64].
Moreover, the contributions of BL had an impact on student satisfaction as well as their level of interest and motivation [65,66], in addition to enriching the exchange of knowledge, team efficiency, and individual learning [67]. In this sense, postgraduate dentistry students recognized that the satisfaction obtained from blended learning experiences derived from a better use of their time and greater time savings compared to face-to-face activities, which in turn contributed positively to achieving course goals [68].
BL was found to transform the training process for students in the second year of Social Medicine at the University of Los Andes (Colombia) [69]. This experience confirms the trend perceived in research that BL improves the effectiveness of teaching-learning processes and increases student training quality [67]. Students recognized the benefits of blended learning for transforming course design through flexibility and autonomy in the learning process.
One factor that might contribute to this improvement is the potential for feedback among all participants [70]. Similar results were reported by an investigation carried out in Turkey [71]. Interviewed medical students responded that the value of these strategies was to promote autonomy and thus increase student academic success. The improvement was likely to be closely related to the flexibility and independence that BL brings to the educational process [72], which fosters a better adaptation to the diverse needs and potential of higher education students.
All of these results confirm the impact of BL on outcomes and academic performance [62]. A very compelling study involving the opinions of 427 university students in Jordan [73] concluded that the positive and significant impact on learning with BL correlated with greater academic success and self-esteem, better quality of life, less dependence on books for learning, and increased competence for autonomous work. Another study [74] also detected internal relationships between learning motivation and strategies in English as a second language. It was found that self-regulation strategies promoted better effectiveness and student motivation in the teaching-learning process. In this sense, BL can contribute to promoting greater student involvement and motivation for learning.
Finally, and within the same area, we found research [75] contributing ideas and recommendations for improving BL processes insofar as user participation and e-tutorial adequacy. Students are thus provided with different means for acquiring essential literacy skills and interacting with content. Similarly, some authors [76] praised blended learning models based on the Moodle platform, reporting their usefulness for furthering teaching effects and potential as a reference for the innovation of models and methodologies. In this line, a study analyzing students' perception of the quality of two BL educational courses, one using Moodle and another not [77], found that the students preferred to participate mainly in collaborative activities.

Agents' Evaluation
A significant and numerous group of publications aimed to identify the perception of their study subjects (faculty and students) regarding aspects of BL such as device evaluation and adequacy, accessibility, and contributions to the teaching-learning process.
Some of these papers analyzed the characteristics and influence of devices used in BL [78]. They evaluated the suitability of devices for the learning process, highlighted the need to adapt learning device design and development, and offered guidelines regarding their implementation. Implementation success can be attributed to pre-planning and planning, as well as to the learning environment and choice of learning devices. However, the success of is conditioned by a variety of contextual aspects. For example, one study [47] showed that the greatest obstacle to the development and maintenance of BL was existing institutional culture. Other important factors contributing to success were found to be device suitability, accessibility, and usability [75].
A qualitative research project [79] was carried out with second year engineering students who participated in training experiences in a context rich in technological resources and devices. Students were found to appreciate the BL since it provided immediate and asynchronous access to learning resources and materials, thus facilitating adaptation to diverse learning situations. These results are in line with a study conducted a few years earlier [80], which found positive evaluations regarding how the BL accounted for each student's rhythm and fostered autonomy and self-esteem.
It has been reported [81] that the contribution of BL to the process of learning a foreign language (English, in this particular instance) lay in the combination of techniques, which had an impact on improving discursive knowledge in the target language and positively affected the ability to integrate information.
Nevertheless, it is also worth highlighting that some studies failed to show significant differences between students' learning styles and their perception of blended learning [82].
The use of BL was positively perceived not only by students, but also by university faculty, who perceived it as beneficial for their own training [83].
However, some authors [84] pointed out that there is still a long way to go before students become involved and take greater advantage of online learning. These and other researchers mention examples of how students recognize the pedagogical value of face-to-face training for the acquisition of particular contents or skills.
To sum up, we would like to point out that BL-related research has evolved over the years. A particularly noteworthy fact is that early research seemed to focus on questions involving types of resources and tools, the comparison of BL-based models and other designs, determining the main uses of e-learning, the opinions of students and managers, and the teaching models developed. Likewise, many initial studies focused on analyzing the design of e-learning platforms. On the other hand, most recent research projects seem to focus on the study of implementation processes and their follow-up from a methodological point of view. We should also note that many studies have followed a quantitative approach.

Identifying Limitations
Our final aim was to identify the limitations of the analyzed studies. These have been grouped into four broad categories.
The first category refers to the concept of BL itself, because important differences exist in the conception of this pedagogical strategy and its main characteristics. Approaches range from constructivism [85], whereby practices and materials complement each other for the acquisition of objectives, to technocratic models, which attribute a leading role to resources [86].
The second limitation is related to space and time variables. As is evidenced by the bibliometric analysis, publications increasingly originate from around the world, yet papers written in Africa [87] or Central and South America [68] are still very scarce. The digital and economic gap between countries and continents explains why many research teams lack the necessary resources for carrying out studies and disseminating their findings in languages such as English, which monopolize scientific literature and may be an obstacle for some researchers.
Thirdly, we must mention that few evaluation tools exist for identifying significant differences between participants in terms of social background or potential digital divide. In other words, there is little research integrating socio-demographic data that might be useful for determining the usefulness of this approach insofar as aspects such as gender, age group, and socio-cultural background.
Finally, the fourth limitation has to do with sample characteristics. Some studies included small samples [81] or lacked a control group [88], thus making their results difficult to generalize. Furthermore, and despite the abundant number of publications in recent years, no publications were found that evaluated the impact of BL on the learning processes of differently-abled university students, reflecting a deficiency in inclusiveness.

Conclusions
Our results are in line with previous meta-analyses [89,90]. With respect to earlier research, the contribution of the present study stems from the need for an updated analysis after the introduction of new devices and teaching modalities in recent years. For example, the influence of mobile learning in higher education could have produced differences with respect to earlier conclusions.
In addition, this study makes it possible to compare the uses and implications of pre-COVID-19 BL with the educational changes imposed by the worldwide health emergency, as well as providing a basis for conducting post-COVID-19 analyses in the future.
Finally, the performance of bibliometric studies allowed for a much wider sample than other types of study, such as bibliographic analysis or meta-analysis. We have complemented this approach with a content analysis from a qualitative standpoint.
These analyses provide insight into the state of BL research throughout the world, based on publications identified in two prestigious databases.
On the one hand, the bibliometric study allowed us to look at the data from a quantitative perspective. The results corroborated Lotka's laws on scientific productivity, as well as Price's law on exponential growth and Bradford's law on the dispersion of literature. It also allowed us to identify leading authors, institutions, and countries regarding this subject matter, highlighting the diverse fields of knowledge in which BL is being utilized to train future professionals.
The research results in the analyzed papers hardly provided any conclusive data regarding the socio-demographic variables of participating students. A recent bibliographic review of the research that included results relative to gender and age in the use of BL [91] concluded that there were no significant differences in participants' perceptions according to gender. On the other hand, age did present significant differences, as older people considered the use of BL to be more positive.
Generally speaking, both students and teachers value BL positively, although they are aware that its full potential has yet to be explored. On a technological level, there is potential for improving online platforms and resources to allow for much more personalized learning, but better training in digital skills is considered necessary. These needs have become more urgent due to the COVID-19 pandemic, which has forced universities all over the world to use technology as the main vehicle for teaching and learning. This "new normal" is being combined with face-to-face education.
The combination of face-to-face and online education is a useful tool that can contribute to improving the quality of higher education teaching and learning processes by creating a motivating environment that can promote the acquisition of tailored and innovative knowledge and skills in diverse scientific fields. There is a degree of consensus among the faculty and especially students insofar as being satisfied with their participation in BL-based learning processes, largely due to its potential for autonomous, flexible, and fast learning. Together with face-to-face methods, better adaptation to specific contextual learning needs can be facilitated and, thus, the gaps left by monolithic methodologies can be filled.
It is evident that knowledge is being constructed mainly with the results of very restricted and local research. It is difficult to generalize since findings are based on reduced normo-typical samples involving classroom experiences for a particular subject that rarely go beyond a single academic year. Furthermore, purely quantitative or qualitative methodologies have mostly been applied, rather than taking advantage of mixed designs.
The results obtained from this bibliographic review are interesting for a variety of reasons. For example, it becomes evident that educators can improve results by evaluating materials from a pedagogical perspective. Clearly, practical activities must be proposed in the classroom so that students become more involved in the assessment of their own experience. It is also necessary to address differently-abled students and different societal agents. Administrators would do well to promote digital competence in all members of the educational community.
Regarding study limitations, the most notable is the use of only two databases, as well as the limited number of open access publications included in the content analysis.
It is vital to remember that mere implementation of practices and materials does not guarantee the success of an instructional initiative, but rather contextual conditioning factors should also be taken into consideration. These include the underlying professional culture, the provision of digital skills training for educators, the availability of both technological devices and fast Internet connections for students and, finally, the characteristics of the subject and student group for which it is being implemented.
In light of comments made by the authors of the analyzed studies, an evident need exists for further research combining both qualitative and quantitative methodologies in order to learn what is really happening in "classrooms" with BL. An analysis of necessary student competences and the competences being developed through the use of BL would also be quite relevant. Therefore, future research will require better planning and designing of intervention proposals, after an adequate selection of devices and materials as well as consideration of teacher training. In addition, it would be necessary to carry out studies with larger and more diverse samples to complement existing results and facilitate the development of a shared theoretical framework integrating other variables into the research (e.g., socio-demographic variables and different intervention strategies). Finally, it would be advisable to combine bibliometric indicators with altmetrics from networks such as Facebook, Twitter, ORCID, Mendeley, and ResearchGate, as was conducted by the prolific author Chang Zhu.