Next Article in Journal
Supporting Undergraduate STEMM Education: Perspectives from Faculty Mentors and Learning Assistants in Calculus II
Next Article in Special Issue
Development of Game-Based M-Learning Apps for Preschoolers
Previous Article in Journal
Understanding and Addressing the Deficiencies in UK Mathematics Education: Taking an International Perspective
Previous Article in Special Issue
Mobile App for Science Education: Designing the Learning Approach
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Education Sciences
Volume 11
Issue 3
10.3390/educsci11030142
education-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Application of Virtual Reality in Computer Science Education: A Systemic Review Based on Bibliometric and Content Analysis Methods

by
Friday Joseph Agbo
1,*,
Ismaila Temitayo Sanusi
1,
Solomon Sunday Oyelere
2,* and
Jarkko Suhonen
1
1
School of Computing, University of Eastern Finland, FIN-80101 Joensuu, Finland
2
Department of Computer Science, Electrical and Space Engineering, Luleå University of Technology, 971 87 Luleå, Sweden
*
Authors to whom correspondence should be addressed.
Educ. Sci. 2021, 11(3), 142; https://doi.org/10.3390/educsci11030142
Submission received: 22 February 2021 / Revised: 15 March 2021 / Accepted: 16 March 2021 / Published: 23 March 2021
(This article belongs to the Special Issue Current Trends in Game-Based Learning)
Browse Figures
Versions Notes

Abstract

:
This study investigated the role of virtual reality (VR) in computer science (CS) education over the last 10 years by conducting a bibliometric and content analysis of articles related to the use of VR in CS education. A total of 971 articles published in peer-reviewed journals and conferences were collected from Web of Science and Scopus databases to conduct the bibliometric analysis. Furthermore, content analysis was conducted on 39 articles that met the inclusion criteria. This study demonstrates that VR research for CS education was faring well around 2011 but witnessed low production output between the years 2013 and 2016. However, scholars have increased their contribution in this field recently, starting from the year 2017. This study also revealed prolific scholars contributing to the field. It provides insightful information regarding research hotspots in VR that have emerged recently, which can be further explored to enhance CS education. In addition, the quantitative method remains the most preferred research method, while the questionnaire was the most used data collection technique. Moreover, descriptive analysis was primarily used in studies on VR in CS education. The study concludes that even though scholars are leveraging VR to advance CS education, more effort needs to be made by stakeholders across countries and institutions. In addition, a more rigorous methodological approach needs to be employed in future studies to provide more evidence-based research output. Our future study would investigate the pedagogy, content, and context of studies on VR in CS education.

1. Background of the Study

Virtual reality (VR) has recently become a popular technology in different contexts such as entertainment, military, and education [1]. VR combines technologies to provide an immersive presence through highly interactive objects in a virtual environment but stimulates users’ sensory awareness to perceive being in an almost natural environment. The use of VR in education to support training, teaching, and learning through 3D simulation and visualization of learning content in a virtual presence has grown recently [2]. This increasing VR application growth in the educational field is evident, as revealed by the literature, including a recent VR study in computer science education [3]. VR technology provides an opportunity to develop a state-of-the-art smart learning environment with a high level of interaction, engagement, and motivation for an enhanced learning experience [1,2,3,4,5,6,7,8]. This study refers to computer science (CS) education as the art and science involved in learning and teaching computer science, including computing, algorithmic and computational thinking [9]. For example, the science behind curriculum design, pedagogical approach, and instructional tools and techniques educators adopt to support computer science teaching and learning.
This study investigated the role of VR in CS education by conducting a comprehensive content and bibliometric analysis of relevant articles published between 2011 and 2020 in journals and conferences. Bibliometric and content analysis of articles focused on VR in CS education would provide a deeper understanding of the evolution of research conducted in this field and how VR applications have advanced CS education over the years [4,10,11]. From the standpoint of bibliometric mapping analysis, this study investigates the publication growth of studies on VR in CS education within the last 10 years, reveals the most active authors and affiliations contributing to the development of VR in CS education, and anticipates the future direction on the basis of the co-occurrence pattern analysis of current studies. In addition, this study explicates the role of VR in CS education from the perspective of methodological approaches used in studies related to VR in CS education [7], the kind of data collected for such studies, the sample size, and the types of data analysis conducted.
Research on VR in education has claimed several benefits, such as positively affecting users’ attitude [12,13], presenting an effective and efficient learning and training environment [14,15], and increasing students’ motivation to learn within a virtual environment [14,15,16,17]. Furthermore, many systematic review studies related to VR in education have been published in recent years. However, there have been only a limited number of such studies focused on computer science education. For example, Pirker et al. [3] conducted a systematic literature review of VR in CS education, focusing on the technology used to deploy VR applications for CS education, the learning objectives, and challenges recorded in studies related to VR in CS education. Pirker and colleagues revealed that VR desktop applications using Oculus Rift and HTC Vive dominate the technology currently used to deploy VR in CS education. On the other hand, the majority of studies on VR in CS education focused on cognitive learning with topics such as fundamental components of algorithms and object-oriented programming [3].
Similarly, Oyelere et al. [1] studied VR games in CS education, focusing on developmental features such as the technology, pedagogy, and gaming elements used in such studies. In terms of technology, Oyelere et al. [1] finding was in congruence with that of Pirker et al. [3], where Oculus Rift, HTC Vive, and PC-based applications dominate the technology aspect. Both studies show that mobile-based VR applications for CS education are still growing, with less than 15% of deployment of VR applications on mobile devices.
We could find only a few studies regarding recent studies that focused on content and bibliometric analysis of articles related to VR in education. For example, Arici et al. [11] conducted content and bibliometric mapping analysis of augmented reality (AR) in science education. Lorenzo et al. [17] investigated VR articles’ scientific production for inclusive learning of people with autism spectrum disorder (ASD). Sobral and Pestana [18] studied a bibliometric analysis of articles related to VR application to learn about dementia from 1998 until 2018 by focusing on articles’ intellectual structure and emerging trends. Lai et al. [19] conducted a bibliometric analysis of VR research in engineering education published and indexed in the Scopus database that spans over 26 years. Thus, Lai et al. [19] provided valuable insights in terms of article production, trends, and co-occurrence network of VR studies within the field of engineering. Another bibliometric study related to VR in CS field-specific was recently conducted by Enebechi and Duffy [20]. This study [21] focused on bibliometric analysis of VR and artificial intelligence (AI) articles in mobile computing and applied ergonomics.
While all these related studies highlighted above are relevant and provided essential knowledge about the field, our current research would expand on the existing research rather than re-inventing the wheel. For example, while the work of Pirker et al. [3] mainly focused on the technology used to deploy VR application for CS education, the learning objectives, and challenges recorded in studies related to VR in CS education, our research would address the aspect of methodological approach used in studies on VR in CS education, kind of data collected for such studies, the sample size, and types of data analysis conducted. The majority of these related studies analyzed a small sample size, limiting the study, and cannot justify the generalization of their findings. For example, Pirker et al. [3] analyzed 13 pieces of data, Lorenzo et al. [17] revealed 18 articles, Lai et al. [19] conducted bibliometric analysis on 274 articles, and Enebechi and Duffy [20] presented a content analysis of 8 papers. Our study took a different approach by analyzing more extensive data to discover more profound knowledge in the field. It is worth mentioning that our study drew motivation from [11] by focusing content analysis of variables such as materials and method trends, sample sizes, and method of an investigation conducted by articles on VR in CS education in the last 10 years. The authors hope that the approach used in this study would contribute to the existing knowledge in terms of unveiling how VR has supported CS education and what scientific achievement have been made in this field.
As a result of this comprehensive content and bibliometric analysis of studies on VR in CS education, we hoped that our findings would contribute to the existing knowledge by providing a deeper understanding of VR applications’ role in honing CS education over the last decade. In addition, the authors believe that this study will unveil information regarding what scholars have made a scientific achievement in this field in terms of advancing teaching and learning of CS topics in the different contexts, which will serve as a boost for active researchers. In contrast, new scholars would derive motivation and valuable resources for future studies. To achieve objectives, this study set out to answer the following research questions:
RQ1
How is the growth of research publication and citation of articles on VR in computer science education?
RQ2
Who are the most active authors, institutions, and countries publishing articles on the use of VR in computer science education?
RQ3
What co-occurrence patterns exist in studies on the use of VR in computer science education?
RQ4
What is the trend of the research methodology employed in articles on VR in computer science education?
RQ5
What were the most preferred data collection tools and sampling methods in articles on the use of VR in computer science education?
RQ6
What were the sample sizes in articles on the use of VR in computer science education?
RQ7
What were the most preferred data analysis methods in articles on the use of VR in computer science education?

2. Methods

The method explored in this study was centered on content and bibliometric mapping analysis. This study followed the recommended workflow for science mapping provided by Aria and Coccurullo [21] to conduct our bibliometric mapping analysis. In contrast, the approach shown by [11] was followed to present the content analysis, respectively.
Article selection process
The article selection process for this study includes 3 phases similar to the one presented by [4], namely, (i) literature search and data collection; (ii) data extraction, loading, and conversion; and (iii) data synthesis. A graphical representation of the data collection process is presented in Figure 1, showing detailed actions in each phase.
(i)
Literature search and data collection
This study obtained data from 2 databases, i.e., the Web of Science (WoS) and the Scopus databases. These 2 databases have been acclaimed to contain comprehensive data of scientific outputs relevant to this study [14]. To conduct an extensive data collection needed for this study, we define the search keywords to include “virtual reality” “VR”, “computer science”, and “computing education”. A number of common protocols for data collection were applied to both databases. They include the same search keywords used in combination with the binary operators such as “OR” and “AND” across the 2 databases, limited time span to the period from 2011 to 2020, and language selected as “English”. Table 1 presents details of the search protocol, how they were applied in each database, and the result obtained.
(ii)
Data extraction, loading, and conversion
After data from the independent databases were collected and downloaded in BibTex format, we conducted data extraction and conversion into a comma-separated values CSV file to merge the 2 datasets from WoS and Scopus. The process of merging the data is presented in Table 2, followed by executing command line instructions (CLI) shown in Figure 2. R-studio is an integrated development environment for R programming language (https://rstudio.com, accessed on 18 January 2018) software was used to combine the data into a single CSV file before uploading it to biblioshiny (Biblioshiny is a web interface for bibliometrix r-package (https://www.bibliometrix.org/Biblioshiny.html, accessed on 18 January 2018) for bibliometrix R-package [17].
After completing the steps in Table 2, we executed the line of commands (lines 6 to 15) in Figure 2 to complete the remaining process of data conversion and merging. This merging of the two converted points of data by running the command in line 11 of Figure 2A triggered the R- Function that identified 49 similar articles from WoS and Scopus databases. The identified similar articles were removed to avoid having duplicate data. Removing duplicate articles left the remaining data at 971, which was uploaded to biblioshiny for bibliometric mapping analysis. The search was conducted on 2 January 2021.
(iii)
Data Synthesis
In Table 3, we present the synthesized data used for the bibliometric analysis. However, for the content analysis, 3 researchers screened the entire data by reading each paper’s abstract to decide whether it was relevant or not. Further criteria for selecting relevant papers suitable for the content analysis included:
(i)
the paper must focus on virtual reality for education in computer science education;
(ii)
the paper designed a study or developed a solution to facilitate CS education in a VR environment;
(iii)
the study reported any outcome by evaluating with users (students, educators, or experts);
(iv)
the paper is open access and could be downloaded for detailed review.
After applying the criteria, we arrived at 39 papers that met the content analysis requirements presented in Section 3.2.

3. Results

3.1. Findings from Bibliometric Mapping Analysis

This section presents our findings from the bibliometric analysis on the basis of the data generated from WoS and Scopus databases. This bibliometric analysis intends to provide insight into how studies on the use of VR for CS education have grown in the last 10 years. In addition, the result reveals authors, institutions, and countries who have been contributing to the field by actively publishing research related to VR in CS education. Furthermore, the result presents how studies on VR in CS education have had an impact in terms of their citations and authors co-occurrence pattern analysis. The section delineates the analysis of common keywords used in articles on VR for CS education, thereby presenting the thematic area of the current research landscape and topic hotspots.

3.1.1. Research Publication Growth of Articles on the Use of VR in Computer Science Education

Figure 3 shows the articles’ distribution in terms of the publication year regarding the article production and development across 10 years. The overall publication trend of articles related to VR in CS education shows that 2011 witnessed the highest production year, reaching 148 articles, followed closely by 135 articles in 2018.
The publication volume decreased from 2012 to 2015 and from 2019 to 2020. There was an increase in article production from 2016 to 2018 before the slight decline until 2020. This trend occurred probably because the selected articles were limited to only education, leaving out other domains, such as health, business, entertainment, and media.

3.1.2. Most Active Authors, Institutions, and Countries Publishing Articles on the Use of VR in Computer Science Education

Regarding authors’ production over time, we investigated the top 20 authors. Our findings showed that most of those top authors were already publishing articles on VR in CS education by 2011. However, about half of those authors were not active from 2019. As shown in Figure 4, many articles related to VR in CS education were published between 2011 and 2020.
As we can see in Figure 4, the author Li Y. had the highest publication over time, having had several articles published yearly for 7 years from 2011 to 2020, except in 2013, 2014, and 2016. With the least productivity over time was the author is Dengel A., with publications only in 2019 and 2020.
We analyzed the top 20 authors’ number citations across the production years (m-index) regarding their impact. M-index is calculated by dividing the total number of citations by the total number of years of production. In order words, this study measures the authors’ impact by dividing the H-index by the total number of years of production. Note that the total years of production varied for different authors. Although the total number of years investigated in this study remained at 10, some authors did not start publishing from 2011; therefore, such an author’s total number of years of production would count from the year the author published his/her first paper. For example, Dengel A. started publishing articles on VR in CS education in 2019; hence, the total number of years remained at two. Therefore, the m-index would be the total number of citations in 2019 and 2020, divided by 2.
As shown in Figure 5, the authors’ m-index was highest at 1.0 (to a single decimal). Therefore, the result indicates that Dengel A., with the highest m-index, remained the most impactful author at the end of 2020. This finding suggests that Dengel A. had had an unbroken research activity in the area of VR in CS education since the first publication and had received a significant number of citations.
Our analysis revealed some top universities regarding institutions (authors’ affiliations) and countries fronting VR in CS education. As shown in Figure 6, some of these universities, to name a few, were the University of Southern California, USA; Aalborg University, Denmark; and University of Rennes, France.
Figure 6 shows the USA as the most productive country in terms of publishing articles related to VR in CS education in countries. From the European continent, France, Denmark, Italy, the UK, Germany, and Spain made significant contributions. Only China and Japan made contributions regarding VR in CS education from the Asian continent.

3.1.3. Keywords Co-Occurrence Patterns of Studies on the Use of VR in Computer Science Education

A keywords co-occurrence pattern (KCP) focuses on understanding the knowledge components and knowledge structure of a scientific field by examining the links between keywords in the published articles within the same area [4].
Figure 7 focuses on keyword co-occurrence patterns of studies on the use of VR in computer science education. As observed in Figure 7, the root keyword in the field remains “virtual reality”. Other keywords that are frequently used by articles on VR in CS education are shown in red color. For instance, we notice keywords such as gamification, simulation, higher education, mixed reality, serious games, and more. In addition, as expected, keywords that define the characteristics of virtual reality technology were seen to be strongly connected to the root keyword. For example, we observe a thick line connecting keywords such as immersion, interaction, and presence, to the root keyword “virtual reality”. Moreover, virtualization, cloud computing, and virtual machine are keywords that show a strong connection. Other keywords that show a close relationship to virtual reality include augmented reality and computer science.
Furthermore, Figure 8 presents a visualization of frequently used keywords in VR for CS education. It is clear from the size of the nodes that other related terms used for virtual reality, for example, “virtualization” and “virtual environment” were found to be highly connected to “computer science” and “education”. In addition, some pedagogical concepts for teaching and learning, such as games, gamification, collaborative learning, and immersive learning, are visible in the network. Figure 8 also shows clustering of concepts where terms such as virtualization, virtual environment, computer science, and education form clusters depicted with different colors.
One way to examine how VR application has influenced CS education is to analyze trending topics over the period considered in this study. Figure 9 presents the trending topics or approaches scholars have explored to provide VR intervention for CS education.
This study analyzed the authors’ keywords to determine what research hotspot in terms of topics and approaches have been explored by VR applications in CS education in the last decade. This analysis was conducted through the word cloud of authors’ keywords, which gives a pointer to what has been the scholars’ interest. This analysis also provides insight regarding the future outlook of VR interventions in CS education. Figure 9 delineates that virtualization, cloud computing, the virtual world, and virtual machine dominate VR studies in CS education between the years 2011 and 2015. In addition, slightly different changes were observed where keywords such as computer science education, serious games, and higher education emerged among the trending topics between 2015 and 2017.
Furthermore, it was observed that between the years 2018 and 2020, new keywords such as augmented reality, immersion, presence, gamification, game-based learning, and human–computer interaction were added to the trending topics. Therefore, topics such as immersion, presence, human–computer interaction, gamification, and game-based learning dominate the list of research hotspots in recent times. This finding suggests that one of the most appreciated learning and teaching approaches used by studies on VR application in CS education is game-based learning.

3.2. Findings from Content Analysis

This section presents the content analysis findings to address some of the research questions (RQ4 to RQ7). Moreover, an overview of the data analyzed in this section is presented as an Appendix A. In the Appendix A, information regarding the study focus and outcome are highlighted to showcase how the selected articles have employed VR in CS education.

3.2.1. Trends of the Research Methodology Employed in Articles on the Use of VR in Computer Science Education

According to Figure 10, 47% of the articles used a quantitative design approach, 16% used a qualitative design, 3% used mixed design, and 12% utilized a design and development research approach. In comparison, others may include review/meta-analysis research accounts for 5%.
Figure 11 revealed the research method trends related to VR in CS education in the past 10 years. The use of quantitative methods increased in 2018 and declined from 2019 to 2020. The next prominent method utilized is the design research method used in 2011 and in 2014, and witnessed an increase in 2020. While mixed methods are almost inexistent, qualitative and other methods showed no significant distribution variations over time. Review and meta-analysis began to be used in 2019 as the quantitative design was found to be the most used research method over the years.

3.2.2. The Most Preferred Data Collection Tools and Sampling Methods in Articles on the Use of VR in Computer Science Education

Data collection tools and sampling methods in research conducted on VR in CS education show that the questionnaire (46%) remains the most used tool. However, quite a number of studies (23%) either did not conduct evaluation or did not specify what method of data collection was used.
As shown in Figure 12, the use of interviews (13%) is still growing as fewer studies have been seen to use the method.

3.2.3. Sample Populations and Sample Sizes in Articles on the Use of VR in Computer Science Education

According to Figure 13, the most commonly used sample size in articles published between 2011 and 2020 fell between 11–20 participants. Closely followed were 1–10 persons and 51–100 people. Although other studies utilized samples between 21–50 and 101–200 respondents, a few studies did not specify the sample size they used.

3.2.4. Most Preferred Data Analysis Methods in Articles on the Use of VR in Computer Science Education

The findings show that most studies were performed using descriptive analysis regarding the most preferred data analysis conducted in studies focused on VR in CS education.
Other preferred analysis methods, as shown in Figure 14, are meta-analysis and content analysis. Moreover, some studies adopted a theoretical approach while some other studies did not conduct any form of research, and therefore we categorized these types of studies as “others/not specified”.

4. Discussion

The bibliometric method’s potential is seen by earlier research [4]. It was opined that bibliometric study advances complement meta-analysis and qualitative research for the scientific evaluation of literature. This study delved into VR’s role in CS education to provide a deeper understanding of the evolution of research conducted in this field and anticipate the future direction on the basis of the analysis of the co-occurrence pattern of keywords used in studies conducted in the last 10 years. The study contributes to knowledge by presenting valuable findings that can boost the morale of prolific scholars who have been contributing to this field and researchers and practicing managers who may be starting to research into VR for CS education. This current study obtained its bibliometric and content analysis data from the Web of Science and Scopus databases.
The bibliometric analysis of articles related to the use of VR in CS education, together with the methodological research trends over the last 10 years, was revealed. Bibliometric analysis results showed that the year 2011 was the highest in article production (148 articles). This result was closely followed by the year 2018 with 135 articles. This finding implies that between 2012 and 2017, articles related to VR in CS education dwindled. Regarding the authors’ production over time, Li Y. had the highest number of articles produced in the field, which is not surprising as the author consistently published in 2011–2012, 2015, and 2017–2020.
Moreover, we analyzed studies’ impact by investigating the number of citations obtained by authors within 10 years. The analysis was focused on the m-index of each author. Considering the 10 years duration in this study, we calculated the m-index by dividing the total number of citations by the total number of years authors have been publishing. For example, Dengel A. emerged as the most impactful author because this author had produced one paper per year for only two years. This means that Dengel’s impact analysis was computed on the basis of the output of these two years. However, it was surprising to discover that Li Y., who had the highest number of articles produced over the years, was not as impactful as Dengel A., who had a limited number of articles published within just two years. Earlier studies have examined intrinsic factors affecting the number of citations of articles [22,23]; however, some indicators are not directly related to the quality or content of articles’ extrinsic factors [24]. The previous finding reveals that price index, number of references, keywords, and length of studies are essential explanatory factors [24]. It can be concluded that it is likely that Li’s articles are easily accessible to researchers via open access medium. The relevancy of their topic or even the quality of their paper in terms of content and presentation may account for the citations and rapid impact.
Regarding the institutions and countries contributing to VR in CS education, the results further showed that the University of Southern California, USA; Aalborg University, Denmark; and the University of Rennes, France, remain the top universities in terms of publishing VR in CS education articles. On the other hand, the USA emerged as the most productive country. However, other countries from Europe (France, Denmark, Italy, the UK, and Germany) and Asia (China) are making a significant contribution towards advancing CS education using VR technology. The co-occurrence pattern of authors’ keywords revealed that VR characteristics are leveraged for CS education. For example, immersion, presence, interaction, and gamification are being explored in advancing CS education [1,16,18]. Moreover, these keywords also form the research hotspots in VR, primarily to support learning. Therefore, this study anticipates that VR in CS education would continue to be researched within the scope of these keywords [14].
The content analysis results showed that quantitative studies (47%) dominate the studies in terms of research methodology. The reason for quantitative method preference may be due to the simplified way of presenting quantitative research, as well as less time and effort required to conduct and analyze quantitative data [25]. It might also be the case that the generalization and replicability that the quantitative approach provides accounts for its dominance in the studies. The percentage for the use of mixed methods studies was meager, reflecting that the use of mixed approach studies presents methodological difficulties and challenges [12]. It is safe to conclude that only a few studies consider the potential of mixed-method research, which adds rigor and validity to research through triangulation and convergence of multiple and different sources of information [26,27]. Moreover, few qualitative studies have been conducted in the last 10 years. This may have been due to the rigor and non-use of numbers, making it difficult to simplify findings and observations [25]. On the contrary, Johnson and Christensen [28] assert that reliance on collecting non-numerical primary data such as words and pictures makes qualitative research well-suited for providing factual and descriptive information.
Regarding the frequency of the sampling size utilized over the years, the most used sample sizes were 11–20. We were surprised to find out that most published articles on VR in CS education were evaluated with about 11 to 20 participants. Since the research method’s preference was quantitative research, we expected that many studies would have used more participants to arrive at a generalized outcome. Although studies that used 51–100 sample sizes were also seen in the result, one could have thought that 20 participants may be too small for a quantitative study. According to Faber and Fonseca [29], very small samples undermine the internal and external validity, while huge samples tend to transform minor differences into statistically significant differences.
Our findings revealed that the questionnaire is the most used data collection tool, while descriptive analysis remains the preferred data analysis method. One way to reflect on this result is that the questionnaire seems more straightforward, quicker, and cost-effective to collect data from participants. Moreover, the preference for descriptive analysis may be used to simplify data efficiently [30]. The researcher may have adopted this data analysis method to reduce the time and effort required to format and present beneficial, easily interpretable results to practitioners, policymakers, and other researchers to understand a phenomenon better.

5. Conclusions

This study provides a comprehensive view of scientific papers on VR in CS education published in peer-reviewed journals and conferences between 2011 and 2020. Two main approaches were explored to answer the research questions presented in this study. First, the bibliometric analysis answered the questions regarding the article production growth in the field within a decade, prolific scholars and their affiliations publishing to advance VR in CS education, and research hotspots in the field may guide scholar’s future research focus. Second, content analysis of articles that met the inclusion criteria for this study was analyzed to provide a methodological overview of studies conducted on VR in CS education. Several findings were presented in this study. These findings show that VR research for CS education has fared well; however, some of the years (between 2013 and 2016) witnessed low article production. The study also revealed the prolific scholars and authors’ impact analysis in this field and provided insightful information regarding research hotspots by analyzing the authors’ keywords co-occurrence.
Regarding the scientific methodology and data sampling technique used by studies on VR in CS education, the most preferred is the quantitative method. At the same time, the questionnaire was the most used data collection technique. Moreover, descriptive analysis was mainly used to analyze data in studies on VR in CS education.
This study witnessed a limitation regarding the content analysis. It would be interesting to see the educational context where VR technology is being used and the learning contents deployed in the VR application for CS education. Nonetheless, this study contributes to knowledge in significant ways. The study revealed that pedagogical approaches such as game-based learning and gamification were explored for VR education in CS education. The findings from this study can provide insight into how VR technology research has progressed in a decade. Moreover, the result can be generalized since this study could obtain relevant data from two databases (WoS and Scopus) to conduct its analysis. The process for merging these data is another contribution as scholars interested in running a similar study would find this helpful study. Our future study would address the limitations by providing answers regarding the pedagogy, content, and context of studies on VR in CS education.
By implication, we conclude that findings from this study suggest that even though scholars are leveraging VR to advance teaching and learning in the field of CS, more effort needs to be made, especially from continents, countries, and institutions that were not reported among the top-20 list revealed in this study. In addition, a more rigorous methodological approach needs to be employed in a future study to provide more evident-based research output. For example, our study revealed only a few studies that used a mixed-methods approach, which has been more rigorous in terms of quality of scientific research.

Author Contributions

Conceptualization, F.J.A., I.T.S.; Data curation, F.J.A., I.T.S.; Formal analysis, F.J.A., I.T.S., S.S.O. and J.S.; Investigation, F.J.A., I.T.S. and S.S.O.; Methodology, F.J.A., I.T.S., S.S.O. and J.S.; Project administration, F.J.A.; Resources, F.J.A., I.T.S.; Software, F.J.A., I.T.S., S.S.O. and J.S.; Supervision, S.S.O. and J.S.; Validation, S.S.O. and J.S.; Writing—original draft, F.J.A. and I.T.S.; Writing—review & editing, F.J.A., I.T.S., S.S.O. and J.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data sharing is not applicable to this article.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table
Table A1. Published articles contained in the content analysis of VR for CS education (2011–2020).
Table A1. Published articles contained in the content analysis of VR for CS education (2011–2020).
AuthorsAim of the StudyResults of the Study
Nguyen et al. [31] Virtual reality (VR) programming environment called VRASP was developed allow students to produce an avatar (agent) in a virtual world that is able to answer questions in spoken natural language.Findings from the study show that students were able to communicate with the environment intuitively with an accuracy of 78%.
Srimadhaven et al. [32]The study focused on conducting an experiment with the virtual reality mobile app in order to assess the cognitive level of the students in a Python course.The authors anticipated that findings can be useful to higher education students and enhance the performance of all levels of learners.
Bouali et al. [33]This study presented a VR-based learning game to support the teaching and learning of object-oriented programming (OOP) concepts in computing education.The authors envisaged that the designed game would spark interest for learning CS programming concepts such as IF condition, Arrays, and Loops.
Dengel [34]This study demonstratred how metaphorical representations in VR can enhance the understanding of theoretical computer science concepts by using the Treasure Hunt game.The study anticipated measuring students’ cognition, presence, usability, and satisfaction in their future study.
Bolivar et al. [35]This study presented an immersion 3D environment in the form of a video game. The environment offers the player the opportunity to explore basic CS concepts without removing any of the entertaining aspects of games.The authors anticipated a positive impact of the framework when their future research is completed.
Parmar et al. [36]This authors developed a virtual reality tool—VEnvI—to support CS students in learning about the fundamental of CS.The study presented several cases and sample projects developed to assist teachers in their classes.
Kerdvibulvech [37]This study proposed a virtual environment framework for human–computer interaction.The author envisaged that this approach could provide significant ducational values.
Rodger et al. [38]The authors have developed curriculum materials for several disciplines both for student and teacher use. The curriculum materials include tutorials, sample projects, and challenges for teaching CS topics.Demonstration and evaluation of the tool was expected to produce useful outcome.
Vallance [39] This study aimed to set a medium of collaboration within a 3D virtual world.This study was still a work in progress, and hence a concrete result was not presented.
Arrington et al. [40]This study designed and implemented Dr. Chestr, a virtual human in a virtual environment game aimed at supporting the understanding and retention of introductory programming cources.The study measured students’ cognition, presence, usability, and satisfaction and found that students enjoyed the experience and were successfully engaged the virtual world.
Vanderdonckt and Vatavu [41] This study present a VR application where the user, a psychologist, controls a virtual puppet (a cartoon-like character in VR).The study found that when receiving lectures in a virtual environment by a teacher, the child was calm, focused, and capable of working on his assignments without showing any disruptive behaviors.
Parmar et al. [42] The authors developed a VR tool—VEnvI—to support CS students in learning about the fundamental CS concepts such as sequences, loops, variables, conditionals, and functions.Participants who tested the VR tool agreed that the visual aspect improved the overall learning experience.
Adjorlu and Serafin [43] This study investigated the feasibility of using VR to reduce disruptive classroom behavior of a child diagnosed with autism spectrum disorder (ASD).The study provided guidelines to educators and instructional designers who wish to offer interactive and engaging learning activities to their students.
Berns et al. [44]A VR educational platform MYR was built to spark student interest in computer science by allowing them to write code that generates three-dimensional, animated scenes in virtual reality environment. The goal of the project was to gain insight into computing students’ success, motivation, and confidence in learning computing.Evaluation with CS students shows that MYR is hard for CS students to provide clear 3D representations for programming concepts; however, the study was able to derive some common figures.
Christopoulos et al. [45]Authors investigared what effect instructional design decisions have on motivation and engagement of students learning in virtual and physical world.Evaluation of this tool suggests that users’ experience is enhanced through the 3D animation.
Ortega et al. [46]The study developed a 3D virtual programming language to provide an interactive tool for beginners and intermediate students to learn programming concepts.The study reported that the method creates fun and effective means of interdisciplinary study.
Sanna et al. [47] This study proposed a virtual 3D tool (touchless interface) to support people without any prior knowledge in code writing to promote user friendliness and usability experience.Feedback from the workshop participants generally shows that they had a good experience.
Cleary et al. [48]This study explored a style of teaching youths how to write computer program using reactive programming in a 3D virtual environment.The study tested educational virtual environments (EVEs) with pre- and post-test and found to be significantly effective.
Domik et al. [49]The authors created “Move the World” workshop in a summer camp to increase high school juniors’ interest in computer science by leveraging math and virtual worlds.Overall comments from participants of the workshop revealed that learning in the virtual world is appealing and inspiring.
Dengel [50]The study modeled three computer scienc topics- asymmetric encryption/decryption, and finite state machines in a 3 D immersive VR to teach these topics.The study discusses students’ preconceptions towards the inclusion of 3D virtual learning environments in the context of their studies and further elicit their thoughts related to the impact of the “hybrid” interactions
Koltai et al. [51]This study used a VR game (Mazes) to teache CS concepts.The study reported positive impact on computer science education by increasing engagement, knowledge acquisition, and self-directed learning.
Christopoulos et al. [52]This authors developed a tool—FunPlogs application—to deply microlearning.The study generally indicated that participants perceived a high joy of use while playing FunPlogs, which indicated that despite the simple game concept, complex matters as the while-loop could be transported to programming laymen.
Banic and Gamboa [53] The study explored a summer course that uses visual design problem-based learning pedagogy with virtual environments as a strategy to teach computer science.The study concluded that interactions in VR plays a crucial role in learner engagement.
Horst et al. [54]This study introduced a VR puzzle mini-game for learning fundamental programming principles.The study outcome shows that the proposed module helps students learn stacks and queues while being satisfactorily usable.
Christopoulos and Conrad [55]Authors examined the impact that the virtual reality learning process has on university students who study CS and have almost no experience in the use of virtual worlds.Results show that the self-overlapping maze is experienced as freely walkable while the map is mostly understandable.
Stigall and Sharma [56]This study designed a game theme-based instructional (GTI) module to teach undergraduate CS majors about stacks and queues.The analysis of SEQ usability test shows good acceptance.
Serubugo et al. [57]This study investigated how working with VR setups can be walkable in small physical spaces or included in non-HMD participants using self-overlapping maze Analysis of the usability and likeability of the survey shows that students felt motivated and engaged in learning programming concepts.
Pilatásig et al. [58]This study designed a VR tool to assist in training and rehabilitation of hands and wristThe study reported that students gained cognitive thinking process and had a greater range of expressing sufficiently alternative to self-explanatory solutions.
Segura et al. [59] This study designed a VR application (VR-OCKS) to teach basic programming concepts such as flow statements and conditional selections.The initial evaluation of this tool shows that it enhanced creative thinking of young children.
Pellas and Vosinakis [60] The authors explored a 3D simulation game to teach computational problem-solving. Evaluation results demonstrated positive student perceptions about the use of gaming instructional modules to advance student learning and understanding of the concepts.
Stigall and Sharma [61] This study designed and developed two gaming modules for teaching CS students object-oriented programming (OOP) and binary search.Result analysis suggests that participants showed similar connectedness in affiliative tour and competitive design.
Sharma and Ossuetta [62]The authors developed virtual reality instructional (VRI) modules for teaching loops and arrays that can provide a better understanding of the concept.The study measured participants’ intentions toward majoring in a computing discipline, attitudes toward computing, and overall satisfaction with the camp, and showed positive indication.
Ijaz et al. [63] This study proposed a VR exergaming platform that combines a recumbent tricycle and real-world panoramic images where the player can navigate real locations in a safe virtual environmentThis study argued that comparative studies are a useful method for analyzing benefits of different approaches to controlling virtual agents.
Hulsey et al. [64]This study reported the experience of a summer camp that introduced computing concepts to middle school girls in the context of an online, multiplayer, virtual world.This study demonstrated that familiarity may reduce working memory load and increase children’s spatial memory capacity for acquiring sequential temporal–spatial information from virtual displays.
Gemrot et al. [65] This study presents results of comparing the usability of an academic technique designed for programming intelligent agents’ behavior with the usability of an unaltered classical programming language.Outcome of the experiment with CodeSpells shows that students were able to understand and write basic Java code after only 8 h of playing the game.
Korallo et al. [66]This study examined the potential use of virtual environment in general computer knowledge in virtual environment.Outcome of the study provide overview of the two reviewed approaches for implementing VR gestures, which may guide experts.

References

  1. Oyelere, S.S.; Bouali, N.; Kaliisa, R.; Obaido, G.; Yunusa, A.A.; Jimoh, E.R. Exploring the trends of educational virtual reality games: A systematic review of empirical studies. Smart Learn. Environ. 2020, 7, 1–22. [Google Scholar] [CrossRef]
  2. Carruth, D.W. Virtual reality for education and workforce training. In Proceedings of the 2017 15th International Conference on Emerging eLearning Technologies and Applications (ICETA), Stary Smokovec, Slovakia, 26–27 October 2017; pp. 1–6. [Google Scholar]
  3. Pirker, J.; Dengel, A.; Holly, M.; Safikhani, S. Virtual Reality in Computer Science Education: A Systematic Review. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology; Association for Computing Machinery (ACM), Tokyo, Japan, 1–4 November 2020; pp. 1–8. [Google Scholar]
  4. Agbo, F.J.; Oyelere, S.S.; Suhonen, J.; Tukiainen, M. Scientific production and thematic breakthroughs in smart learning environments: A bibliometric analysis. Smart Learn. Environ. 2021, 8, 1–25. [Google Scholar] [CrossRef]
  5. Agbo, F.J.; Oyelere, S.S.; Bouali, N. A UML approach for designing a VR-based smart learning environment for programming education. In Proceedings of the 2020 IEEE Frontiers in Education Conference (FIE), Uppsala, Sweden, 21–24 October 2020; pp. 1–5. [Google Scholar]
  6. Agbo, F.J.; Oyelere, S.S. Smart mobile learning environment for programming education in Nigeria: Adaptivity and context-aware features. In Intelligent Computing-Proceedings of the Computing Conference; Springer: Berlin/Heidelberg, Germany, 2019; pp. 1061–1077. [Google Scholar]
  7. Agbo, F.J.; Oyelere, S.S.; Suhonen, J.; Tukiainen, M. Identifying potential design features of a smart learning environment for programming education in Nigeria. Int. J. Learn. Technol. 2019, 14, 331–354. [Google Scholar] [CrossRef]
  8. Agbo, F.J.; Oyelere, S.S.; Suhonen, J.; Tukiainen, M. Smart learning environment for computing education: Readiness for implementation in Nigeria. In Proceedings of the EdMedia+ Innovate Learning, Association for the Advancement of Computing in Education (AACE), Amsterdam, The Netherlands, 24–28 June 2019; pp. 1382–1391. [Google Scholar]
  9. Agbo, F.J.; Oyelere, S.S.; Suhonen, J.; Adewumi, S. A systematic review of computational thinking approach for programming education in higher education institutions. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research, Koli, Finland, 21–24 November 2019; pp. 1–10. [Google Scholar]
  10. Zupic, I.; Čater, T. Bibliometric methods in management and organisation. Organ. Res. Methods 2015, 18, 429–472. [Google Scholar] [CrossRef]
  11. Arici, F.; Yildirim, P.; Caliklar, Ş.; Yilmaz, R.M. Research trends in the use of augmented reality in science education: Content and bibliometric mapping analysis. Comput. Educ. 2019, 142, 103647. [Google Scholar] [CrossRef]
  12. Buchholtz, N. Planning and conducting mixed methods studies in mathematics educational research. In Compendium for Early Career Researchers in Mathematics Education ICME-13 Monographs; Kaiser, G., Presmeg, N., Eds.; Springer: Cham, Switzerland, 2019; pp. 131–152. [Google Scholar]
  13. Tüysüz, C. The Effect of the Virtual Laboratory on Students’Achievement and Attitude in Chemistry. Int. Online J. Educ. Sci. 2010, 2, 37–53. [Google Scholar]
  14. Cheng, K.; Tsai, C. Students’ motivational beliefs and strategies, perceived immersion and attitudes towards science learning with immersive virtual reality: A partial least squares analysis. Br. J. Educ. Technol. 2020, 51, 2140–2159. [Google Scholar] [CrossRef]
  15. Bogusevschi, D.; Muntean, C.; Muntean, G.M. Teaching and Learning Physics using 3D Virtual Learning Environment: A Case Study of Combined Virtual Reality and Virtual Laboratory in Secondary School. J. Comput. Math. Sci. Teach. 2020, 39, 5–18. [Google Scholar]
  16. Huang, H.-M.; Liaw, S.-S. An Analysis of Learners’ Intentions Toward Virtual Reality Learning Based on Constructivist and Technology Acceptance Approaches. Int. Rev. Res. Open Distrib. Learn. 2018, 19. [Google Scholar] [CrossRef] [Green Version]
  17. Lorenzo, G.; Lorenzo-Lledó, A.; Lledó Carreres, A.; Pérez-Vázquez, E. Approach from a Bibliometric Perspective of the Educational Application of Virtual Reality in People with Autism Spectrum Disorder. Educ. Knowl. Soc. 2020, 21, 4–14. [Google Scholar]
  18. Sobral, M.; Pestana, M.H. Virtual reality and dementia: A bibliometric analysis. Eur. J. Psychiatry 2020, 34, 120–131. [Google Scholar] [CrossRef]
  19. Lai, N.Y.G.; Wong, K.H.; Yu, L.J.; Kang, H.S. Virtual Reality (VR) in Engineering Education and Training: A Bibliometric Analysis. In Proceedings of the 2020 2nd World Symposium on Software Engineering, San Francisco, CA, USA, 18–20 May 2020; pp. 161–165. [Google Scholar]
  20. Enebechi, C.N.; Duffy, V.G. Virtual Reality and Artificial Intelligence in Mobile Computing and Applied Ergonomics: A Bibliometric and Content Analysis. In Proceedings of the Constructive Side-Channel Analysis and Secure Design, Lugano, Switzerland, 1–3 April 2020; pp. 334–345. [Google Scholar]
  21. Aria, M.; Cuccurullo, C. Bibliometrix: An R-tool for comprehensive science mapping analysis. J. Inf. 2017, 11, 959–975. [Google Scholar] [CrossRef]
  22. Chen, C. Predictive effects of structural variation on citation counts. J. Am. Soc. Inf. Sci. Technol. 2011, 63, 431–449. [Google Scholar] [CrossRef] [Green Version]
  23. Didegah, F.; Thelwall, M. Determinants of research citation impact in nanoscience and nanotechnology. J. Am. Soc. Inf. Sci. Technol. 2013, 64, 1055–1064. [Google Scholar] [CrossRef]
  24. Onodera, N.; Yoshikane, F. Factors affecting citation rates of research articles. J. Assoc. Inf. Sci. Technol. 2015, 66, 739–764. [Google Scholar] [CrossRef] [Green Version]
  25. Eyisi, D. The Usefulness of Qualitative and Quantitative Approaches and Methods in Researching Problem-Solving Ability in Science Education Curriculum. J. Educ. Pract. 2016, 7, 91–100. [Google Scholar]
  26. Jokonya, O. The significance of mixed methods research in information systems research. In Proceedings of the Midwest Association for Information Systems Conference (MWAIS), Springfield, IL, USA, 19–20 May 2016. [Google Scholar]
  27. Regnault, A.; Willgoss, T.; Barbic, S. Towards the use of mixed methods inquiry as best practice in health outcomes research. J. Patient-Rep. Outcomes 2018, 2, 19. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  28. Johnson, R.B.; Christensen, L.B. Educational Research: Quantitative, Qualitative, and Mixed Approaches, 6th ed.; SAGE Publications: Thousand Oaks, CA, USA, 2017. [Google Scholar]
  29. Faber, J.; Fonseca, L.M. How sample size influences research outcomes. Dent. Press J. Orthod. 2014, 19, 27–29. [Google Scholar] [CrossRef]
  30. Loeb, S.; Dynarski, S.; McFarland, D.; Morris, P.; Reardon, S.; Reber, S. Descriptive Analysis in Education: A Guide for Researchers. NCEE 2017-4023 Natl. Cent. Educ. Eval. Reg. Assist. 2017, 1–40. [Google Scholar]
  31. Nguyen, V.T.; Zhang, Y.; Jung, K.; Xing, W.; Dang, T. VRASP: A Virtual Reality Environment for Learning Answer Set Programming. In International Symposium on Practical Aspects of Declarative Languages; Springer: Berlin/Heidelberg, Germany, 2020; pp. 82–91. [Google Scholar]
  32. Srimadhaven, T.; AV, C.J.; Harshith, N.; Priyaadharshini, M. Learning analytics: Virtual reality for programming course in higher education. Procedia Comput. Sci. 2020, 172, 433–437. [Google Scholar]
  33. Bouali, N.; Nygren, E.; Oyelere, S.S.; Suhonen, J.; Cavalli-Sforza, V. Imikode: A VR Game to Introduce OOP Concepts. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research 2019, Koli, Finland, 21–24 November 2019; pp. 1–2. [Google Scholar]
  34. Dengel, A. Seeking the treasures of theoretical computer science education: Towards educational virtual reality for the visualization of finite state machines. In Proceedings of the 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), Wollong, Australia, 4–7 December 2018; pp. 1107–1112. [Google Scholar]
  35. Bolivar, S.; Perez, D.; Carrasquillo, A.; Williams, A.S.; Rishe, N.D.; Ortega, F.R. 3D Interaction for Computer Science Educational VR Game. In International Conference on Human-Computer Interaction; Springer: Berlin/Heidelberg, Germany, 2019; pp. 408–419. [Google Scholar]
  36. Parmar, D.; Babu, S.V.; Lin, L.; Jörg, S.; D’Souza, N.; Leonard, A.E.; Daily, S.B. Can embodied interaction and virtual peer customization in a virtual programming environment enhance computational thinking? In Proceedings of the 2016 Research on Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT), Atlanta, GA, USA, 11–13 August 2016; pp. 1–2. [Google Scholar]
  37. Kerdvibulvech, C. Vision and virtual-based human computer interaction applications for a new digital media visualization. In Proceedings of the WSCG 2015 Conference on Computer Graphics, Visualization and Computer Vision, Pilsen near Prague, Czech Republic, 8–12 June 2015; pp. 247–254. [Google Scholar]
  38. Rodger, S.H.; Brown, D.; Hoyle, M.; MacDonald, D.; Marion, M.; Onstwedder, E.; Ward, E. Weaving computing into all middle school disciplines. In Proceedings of the 2014 Conference on Innovation Technology in Computer Science Education 2014, Uppsala, Sweden, 21–25 June 2014; pp. 207–212. [Google Scholar]
  39. Vallance, M. The affect of collaboratively programming robots in a 3D virtual simulation. In Proceedings of the 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), Tokyo, Japan, 3–6 March 2013; pp. 245–246. [Google Scholar]
  40. Arrington, C., Jr.; Wilson, D.M.; Lehmann, L. Improving performance and retention in computer science courses using a virtual game show. In Proceedings of the 49th Annual Southeast Regional Conference 2011, Kennesaw, GA, USA, 24–26 March 2011; pp. 320–321. [Google Scholar]
  41. Vanderdonckt, J.; Vatavu, R.D. A pen user interface for controlling a virtual puppet. In Proceedings of the 12th ACM SIGCHI Symposium on Engineering Interactive Computing Systems 2020, Sophia Antipolis, France, 23–26 June 2020; pp. 1–6. [Google Scholar]
  42. Parmar, D.; Isaac, J.; Babu, S.V.; D’Souza, N.; Leonard, A.E.; Jörg, S.; Daily, S.B. Programming moves: Design and evaluation of applying embodied interaction in virtual environments to enhance computational thinking in middle school students. In Proceedings of the 2016 IEEE Virtual Reality (VR), Greenville, SC, USA, 19–23 March 2016; pp. 131–140. [Google Scholar]
  43. Adjorlu, A.; Serafin, S. Head-mounted display-based virtual reality as a tool to reduce disruptive behavior in a student diagnosed with autism spectrum disorder. In Interactivity, Game Creation, Design, Learning, and Innovation; Springer: Berlin/Heidelberg, Germany, 2019; pp. 739–748. [Google Scholar]
  44. Berns, C.; Chin, G.; Savitz, J.; Kiesling, J.; Martin, F. Myr: A web-based platform for teaching coding using VR. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education 2019, Minneapolis, MN, USA, 27 February–2 March 2019; pp. 77–83. [Google Scholar]
  45. Christopoulos, A.; Conrad, M.; Shukla, M. Learner experience in hybrid virtual worlds: Interacting with pedagogical agents. In Proceedings of the 11th International Conference on Computer Supported Education (CSEDU 2019), Heraklion, Crete, Greece, 2–4 May 2019; pp. 488–495. [Google Scholar]
  46. Ortega, F.R.; Bolivar, S.; Bernal, J.; Galvan, A.; Tarre, K.; Rishe, N.; Barreto, A. Towards a 3D virtual programming language to increase the number of women in computer science education. In Proceedings of the 2017 IEEE Virtual Reality Workshop on K-12 Embodied Learning through Virtual Augmented Reality (KELVAR), Los Angeles, CA, USA, 19 March 2017; pp. 1–6. [Google Scholar]
  47. Sanna, A.; Lamberti, F.; Bazzano, F.; Maggio, L. Developing touch-less interfaces to interact with 3D contents in public exhibitions. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics; Springer: Berlin/Heidelberg, Germany, 2016; pp. 293–303. [Google Scholar]
  48. Cleary, A.; Vandenbergh, L.; Peterson, J. Reactive game engine programming for stem outreach. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education 2015, Kansas City, MO, USA, 4–7 March 2015; pp. 628–632. [Google Scholar]
  49. Domik, G.; Arens, S.; Stilow, P.; Friedrich, H. Helping High Schoolers Move the (Virtual) World. IEEE Comput. Graph. Appl. 2013, 33, 70–74. [Google Scholar] [CrossRef]
  50. Dengel, A. How Important is Immersion for Learning in Computer Science Replugged Games? In Proceedings of the 51st ACM Technical Symposium on Computer Science Education 2020, New York, NY, USA, 11–14 March 2014; pp. 1165–1171. [Google Scholar]
  51. Koltai, B.G.; Husted, J.E.; Vangsted, R.; Mikkelsen, T.N.; Kraus, M. Procedurally Generated Self Overlapping Mazes in Virtual Reality. In Interactivity, Game Creation, Design, Learning, and Innovation; Springer International Publishing: Berlin/Heidelberg, Germany, 2020; pp. 229–243. [Google Scholar]
  52. Christopoulos, A.; Conrad, M.; Shukla, M. What Does the Pedagogical Agent Say? In Proceedings of the 2019 10th International Conference on Information, Intelligence, Systems and Applications (IISA), Piraeus, Greece, 15–17 July 2019; pp. 1–7. [Google Scholar]
  53. Banic, A.; Gamboa, R. Visual Design Problem-based Learning in a Virtual Environment Improves Computational Thinking and Programming Knowledge. In Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 23–27 March 2019; pp. 1588–1593. [Google Scholar]
  54. Horst, R.; Naraghi-Taghi-Off, R.; Diez, S.; Uhmann, T.; Müller, A.; Dörner, R. FunPlogs–A Serious Puzzle Mini-game for Learning Fundamental Programming Principles Using Visual Scripting. In International Symposium on Visual Computing; Springer International Publishing: Berlin/Heidelberg, Germany, 2019; pp. 494–504. [Google Scholar]
  55. Christopoulos, A.; Conrad, M.; Shukla, M. Increasing student engagement through virtual interactions: How? Virtual Real. 2018, 22, 353–369. [Google Scholar] [CrossRef] [Green Version]
  56. Stigall, J.; Sharma, S. Usability and Learning Effectiveness of Game-Themed Instructional (GTI) Module for Teaching Stacks and Queues. SoutheastCon 2018, 1–6. [Google Scholar] [CrossRef]
  57. Serubugo, S.; Škantárová, D.; Evers, N.; Kraus, M. Self-overlapping Maze and Map Design for Asymmetric Collaboration in Room-Scale Virtual Reality for Public Spaces. In Interactivity, Game Creation, Design, Learning, and Innovation; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; pp. 194–203. [Google Scholar]
  58. Pilatasig, M.; Tigse, J.; Chuquitarco, A.; Pilatasig, P.; Pruna, E.; Acurio, A.; Buele, J.; Escobar, I. Interactive System for Hands and Wrist Rehabilitation. In Advances in Intelligent Systems and Computing; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; pp. 593–601. [Google Scholar]
  59. Segura, R.J.; Del Pino, F.J.; Ogáyar, C.J.; Rueda, A.J. VR-OCKS: A virtual reality game for learning the basic concepts of programming. Comput. Appl. Eng. Educ. 2020, 28, 31–41. [Google Scholar] [CrossRef]
  60. Pellas, N.; Vosinakis, S. Learning to Think and Practice Computationally via a 3D Simulation Game. In Interactive Mobile Communication, Technologies and Learning; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; pp. 550–562. [Google Scholar]
  61. Stigall, J.; Sharma, S. Virtual reality instructional modules for introductory programming courses. In Proceedings of the 2017 IEEE Integrated STEM Education Conference (ISEC), Princeton, NJ, USA, 11 March 2017; pp. 34–42. [Google Scholar]
  62. Sharma, S.; Ossuetta, E. Virtual Reality Instructional Modules in Education Based on Gaming Metaphor. Electron. Imaging 2017, 2017, 11–18. [Google Scholar] [CrossRef] [Green Version]
  63. Ijaz, K.; Wang, Y.; Milne, D.; Calvo, R.A. VR-Rides: Interactive VR Games for Health. In Joint International Conference on Serious Games; Springer International Publishing: Berlin/Heidelberg, Germany, 2016; Volume 9894, pp. 289–292. [Google Scholar]
  64. Hulsey, C.; Pence, T.B.; Hodges, L.F. Camp CyberGirls: Using a virtual world to introduce computing concepts to middle school girls. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education, Atlanta, GA, USA, 5 March 2014; pp. 331–336. [Google Scholar]
  65. Gemrot, J.; Hlavka, Z.; Brom, C. Does High-Level Behavior Specification Tool Make Production of Virtual Agent Behaviors Better? In International AAMAS Workshop on Cognitive Agents for Virtual Environments; Springer: Berlin/Heidelberg, Germany, 2013; Volume 7764, pp. 161–183. [Google Scholar]
  66. Korallo, L.; Foreman, N.; Boyd-Davis, S.; Moar, M.; Coulson, M. Do challenge, task experience or computer familiarity influence the learning of historical chronology from virtual environments in 8–9 year old children? Comput. Educ. 2012, 58, 1106–1116. [Google Scholar] [CrossRef]
Education 11 00142 g001 550
Figure 1. The procedure followed to obtain data used for bibliometric and content analysis.
Figure 1. The procedure followed to obtain data used for bibliometric and content analysis.
Education 11 00142 g001
Education 11 00142 g002 550
Figure 2. (A) shows the set of commands to be executed in R-Studio command line instructions (CLI) to implement the conversion and to merge of data downloaded from Web of Science (WoS) and Scopus databases; (B) shows the output of the executed commands; (C) depicts the console for the CLI where the line execution returns a value including line errors.
Figure 2. (A) shows the set of commands to be executed in R-Studio command line instructions (CLI) to implement the conversion and to merge of data downloaded from Web of Science (WoS) and Scopus databases; (B) shows the output of the executed commands; (C) depicts the console for the CLI where the line execution returns a value including line errors.
Education 11 00142 g002
Education 11 00142 g003 550
Figure 3. Annual scientific production of articles on virtual reality (VR) in computer science (CS) education.
Figure 3. Annual scientific production of articles on virtual reality (VR) in computer science (CS) education.
Education 11 00142 g003
Education 11 00142 g004 550
Figure 4. Top 20 authors publishing articles on VR in CS education between 2011 and 2020: the size of each circle indicates the number of articles. The amount of boldness of the circles shows the number of citations in that year.
Figure 4. Top 20 authors publishing articles on VR in CS education between 2011 and 2020: the size of each circle indicates the number of articles. The amount of boldness of the circles shows the number of citations in that year.
Education 11 00142 g004
Education 11 00142 g005 550
Figure 5. Top-20 authors’ impact analysis within 10 years.
Figure 5. Top-20 authors’ impact analysis within 10 years.
Education 11 00142 g005
Education 11 00142 g006 550
Figure 6. Three-field plot of active institutions and countries publishing articles related to VR in CS education between 2011 and 2020.
Figure 6. Three-field plot of active institutions and countries publishing articles related to VR in CS education between 2011 and 2020.
Education 11 00142 g006
Education 11 00142 g007 550
Figure 7. Co-occurrence patterns of authors’ keywords in articles on VR in CS education between 2011 and 2020.
Figure 7. Co-occurrence patterns of authors’ keywords in articles on VR in CS education between 2011 and 2020.
Education 11 00142 g007
Education 11 00142 g008 550
Figure 8. Visualized authors’ keywords co-occurrence analysis of articles on VR in CS education: these are among the highest number of repetitive keywords within the field.
Figure 8. Visualized authors’ keywords co-occurrence analysis of articles on VR in CS education: these are among the highest number of repetitive keywords within the field.
Education 11 00142 g008
Education 11 00142 g009 550
Figure 9. (a–c) Word cloud showing the trending topics of VR in CS education in terms of authors keywords. While (a) shows the trending topics between 2011 and 2015, (b) presents the topics between 2015 and 2017, and (c) depicts the trending topics on VR in CS education between 2018 and 2020.
Figure 9. (a–c) Word cloud showing the trending topics of VR in CS education in terms of authors keywords. While (a) shows the trending topics between 2011 and 2015, (b) presents the topics between 2015 and 2017, and (c) depicts the trending topics on VR in CS education between 2018 and 2020.
Education 11 00142 g009
Education 11 00142 g010 550
Figure 10. Frequency of research methods in articles on VR in CS education between 2011 and 2020.
Figure 10. Frequency of research methods in articles on VR in CS education between 2011 and 2020.
Education 11 00142 g010
Education 11 00142 g011 550
Figure 11. Trends of research methods in articles on VR in CS in the past 10 years.
Figure 11. Trends of research methods in articles on VR in CS in the past 10 years.
Education 11 00142 g011
Education 11 00142 g012 550
Figure 12. Data collection tools and sampling methods of articles on the use of VR in CS education.
Figure 12. Data collection tools and sampling methods of articles on the use of VR in CS education.
Education 11 00142 g012
Education 11 00142 g013 550
Figure 13. Frequency of use of sample sizes in articles.
Figure 13. Frequency of use of sample sizes in articles.
Education 11 00142 g013
Education 11 00142 g014 550
Figure 14. Most preferred data analysis method between 2011 and 2020.
Figure 14. Most preferred data analysis method between 2011 and 2020.
Education 11 00142 g014
Table
Table 1. Data search procedures and obtained amount of data.
Table 1. Data search procedures and obtained amount of data.
DatabaseDescription of the ProtocolCombination of Search String Based on Database AlgorithmSearch Outcome
WoSApplying the search keywords in quotation to the WoS TOPIC field with binary operators.TOPIC: (“virtual reality” OR “VR”) AND TOPIC: (“computer science” OR “computing education”).80
Additional conditions were applied by limiting the results to only articles and proceedings papers, with time span set to 2011–2020.TOPIC: (“virtual reality” OR “VR”) AND TOPIC: (“computer science”).Refined by: DOCUMENT TYPES: (ARTICLE OR PROCEEDINGS PAPER) AND PUBLICATION YEARS: (2020 OR 2014 OR 2019 OR 2013 OR 2018 OR 2012 OR 2017 OR 2011 OR 2016 OR 2015)Timespan: All years. Indexes: SCI-EXPANDED, SSCI, A&HCI, ESCI.58
ScopusApplying the search keywords in quotation to Scopus title, abstract, and keywords field with binary operators and limiting the time span to 2011–2020.(TITLE-ABS-KEY (“virtual reality” OR “VR”) AND TITLE-ABS-KEY (“computer science” OR “computing education”)) AND PUBYEAR > 2010 AND PUBYEAR < 2021.1058
Applying additional conditions by limiting to only articles and conference papers.(TITLE-ABS-KEY (“virtual reality” OR “VR”) AND TITLE-ABS-KEY (“computer science” OR “computing education”)) AND PUBYEAR > 2010 AND PUBYEAR < 2021 AND (LIMIT-TO (DOCTYPE, “cp”) OR LIMIT-TO (DOCTYPE, “ar”)) AND (LIMIT-TO (LANGUAGE, “English”)).962
After merging both files, we removed 49 duplicated documents.971
Total
Table
Table 2. Data conversion and merging steps.
Table 2. Data conversion and merging steps.
StepsInstructions on How to Merge Two Points of Data from WoS and Scopus Databases
1Download in BibTex format independently from databases (in this case, WoS and Scopus).
2Save data in a directory with a name that says “rawData”.
3Open RStudio and import the bibliometrix library by running the script < library(“bibliometrix”) > in the command-line interface (CLI).
4In Rstudio CLI, run the script < setwd (“C:/../…/… /rawData”) > to open the directory where data would be imported from and saved. Not that the ellipsis (…) indicates the paths to the directory and should be correctly inserted.
5Download in BibTex format independently from databases (in this case, WoS and Scopus).
6Save data in a directory with a name that says “rawData”.
Table
Table 3. Data synthesis indicating the primary information about the data and document type.
Table 3. Data synthesis indicating the primary information about the data and document type.
DescriptionResults
Main information about data
Timespan2011–2020
Sources (journals, books, etc.)378
Documents971
Average years from publication4.53
Average citations per documents3.754
Average citations per year per doc0.7841
References21,021
Document types
Article157
Conference paper814
Document contents
Keywords plus (ID)6281
Author’s keywords (DE)2848
Authors
Authors2738
Author appearances3308
Authors of single-authored documents98
Authors of multi-authored documents2640
Author collaboration
Single-authored documents102
Documents per author0.355
Authors per document2.82
Co-authors per documents3.41
Collaboration index3.04
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Agbo, F.J.; Sanusi, I.T.; Oyelere, S.S.; Suhonen, J. Application of Virtual Reality in Computer Science Education: A Systemic Review Based on Bibliometric and Content Analysis Methods. Educ. Sci. 2021, 11, 142. https://doi.org/10.3390/educsci11030142

AMA Style

Agbo FJ, Sanusi IT, Oyelere SS, Suhonen J. Application of Virtual Reality in Computer Science Education: A Systemic Review Based on Bibliometric and Content Analysis Methods. Education Sciences. 2021; 11(3):142. https://doi.org/10.3390/educsci11030142

Chicago/Turabian Style

Agbo, Friday Joseph, Ismaila Temitayo Sanusi, Solomon Sunday Oyelere, and Jarkko Suhonen. 2021. "Application of Virtual Reality in Computer Science Education: A Systemic Review Based on Bibliometric and Content Analysis Methods" Education Sciences 11, no. 3: 142. https://doi.org/10.3390/educsci11030142

APA Style

Agbo, F. J., Sanusi, I. T., Oyelere, S. S., & Suhonen, J. (2021). Application of Virtual Reality in Computer Science Education: A Systemic Review Based on Bibliometric and Content Analysis Methods. Education Sciences, 11(3), 142. https://doi.org/10.3390/educsci11030142

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop