Student Response Systems: A Multidisciplinary Analysis Using Visual Analytics

Abstract

In recent years, several innovations have emerged in the field of education, including Blended-Learning, Massive Open Online Courses, Flipped Classroom and Gamification. In particular, several investigations have highlighted the effectiveness of student response systems, or clickers, in different subjects and disciplines. Although some literature reviews have been published on this subject, none of them offer a review of a large volume of publications from a multidisciplinary approach. Similarly, in the literature there are no studies that have analyzed scientific collaborations on this subject. To respond to these concerns, we proposed the use of a bot to retrieve information from a large number of papers (1696 documents co-authored by a total of 4091 researchers) included in the Scopus database. The disciplines covered include natural sciences, engineering and technology, medical and health sciences, agricultural and veterinary sciences, social sciences and humanities, and the arts. The review of the literature reveals that student response systems are generally well-perceived by teachers and students in all the disciplines. Another interesting result achieved from visual data obtained using network visualization software and word clouds is that student response systems are mainly used in some disciplines, such as physics, chemistry, medicine, and nursing. It is clearly observed that the relationship between researchers from the same country is stronger than between researchers from different countries. Finally, some reflections are included on the role of student response systems in online teaching, especially regarding the changes experienced after the COVID-19 pandemic.

1. Introduction

Several investigations have shown that the traditional format of expository sessions in which students passively receive information from the instructor does not motivate students or benefit the communication between teachers and students. In recent years, several instructional methodologies have been proposed to promote student-centered learning, including cooperative learning [1], problem-based learning [2], project-based learning [3], or flipped classroom [4], among others.

Educational institutions continually incorporate digital technologies to improve the learning process of students so that they achieve good academic results, develop skills for their future work, and become responsible digital citizens [5]. In fact, the appropriate use of educational technologies favors interactive learning and increases the students’ motivation [6], but always having in mind that the effectiveness of interactive systems can be affected by cognitive and affective factors [7]. Furthermore, digital technologies are often used for assessment for learning implementation [8]. Educational technologies include a host of software packages and electronic devices such as computers, mobile devices or interactive whiteboards [9]. For example, nowadays, almost all students in developed countries have access to virtual platforms, such as Moodle or Blackboard [10], to obtain information about the courses and to communicate with the lecturer and/or their classmates. In addition, lecturers in colleges and universities are increasingly using Student Response Systems (SRSs), also known as audience response systems, classroom response systems, or simply clickers, in order to improve the level of engagement, commitment, and learning of students [11].

SRS hardware/software devices allow teachers to create interactive learning environments in a simple way. It facilitates to evaluate the participants’ knowledge, as well as verify attendance and participation levels in face-to-face sessions. From a technical point of view, SRSs allow teachers to ask students specific questions, which are answered in real time using specific clicker devices or generic devices such as desktops, laptops or smartphones [12]. Although there are specifically designed devices created to be used as SRSs, their use has not been generalized due to their high cost, so alternative systems based on the use of generic electronic devices have been sought, including personal digital assistants (PDAs) and mobile phones [13]. Mobile devices are used as SRSs through online services that provide the collection and evaluation of responses in real time [14]. In practice, SRSs offer teachers the opportunity to pose multiple-choice questions to their students, so that they respond individually using a handheld wireless transmitter, while these responses can be displayed immediately using charts. Additionally, the students can include questions, comments, doubts, suggestions or incidences occurred in classroom that are later analyzed by the teacher [15]. Kahoot [16], NetClick [17], Socrative [18], VotApedia [19], Poll Everywhere [20], i2Vote [21], iClickers [22], i-SIDRA [23], Mentimeter [24], and Quizizz [25] are some examples of software tools that implement SRSs in a simply and fast way. Some studies even suggest that SRSs can be conveniently implemented using Twitter and other social networks [26]. From a pedagogical point of view, SRSs promote interaction between students and teachers with the aim of increasing student attention levels [27]. For example, it is suggested that students’ learning processes are significantly improved by combining the use of dynamic concept maps with SRSs in the classroom, especially among those with low initial self-efficacy [28]. Some authors remark that clicker assessment and feedback may be more effective in increasing students’ commitment to the subject if teachers carry out proper planning of activities [29]. Despite the advantages of using clickers, some authors have noticed that preparing activities with clickers is a time-consuming task [30,31] that requires a significant effort from the teachers [32] and does not always lead to an overall improvement in student performance [33].

In spite of the large number of publications about SRS found in the literature, including experiences in secondary education [15] and higher education [34], to our knowledge a multidisciplinary review on the subject has not been undertaken. This paper analyses the publications related to SRS that are included in the Scopus database using a word clouds, network visualization software and clustering techniques based on community detection.

2. Materials and Methods

A thorough literature review provides a solid foundation for any research endeavor and becomes an essential task for its evolution [35]. Some authors have presented reviews about SRS [36,37,38,39,40,41,42], but some researchers have indicated that there is a lack of reviews analyzing the impact of SRSs on improving learning outcomes [43].

This paper presents a multidisciplinary literature review about SRSs based on the information contained in the papers retrieved from the Scopus database, such as author affiliation, keywords and citations from other publications [44]. The main reasons to select the Scopus database are that it is the largest database of scientific literature including research topics from all scientific disciplines [45] and it allows to use scripts to retrieve data about publications. More specifically, it is used the bot ResNetBot [46] to collect data about all the papers published in the Scopus database up to 2019 that include any of the following terms in the title, abstract or keywords: “ student response system* ”, “ audience response system* ”, “ classroom response system* ”, or “ clickers ”.

Furthermore, literature review helps us to understand the role that existing literature plays within a knowledge creation community [35]. This is the reason why the paper also provides an analysis of the main research topics and scientific collaborations in this topic using visual analytics. Visual analytics is a young field with potential for many disciplines [47] that can be defined as the science of analytical reasoning facilitated by interactive visual interfaces [48].

Some studies have used visual analytics, including clustering methods in networks, to facilitate performance of cognitive activities involving big data [49]. Despite some alternatives have been proposed in the past, including database-oriented solutions, visual analytics tools are needed in many investigations [50]. An important challenge of visual analytics is to establish standards for evaluating results of research and development [51], which is why some authors demand theoretically founded evaluation frameworks for visual analytics [52].

The data retrieved from the Scopus database is analyzed using two strategies. On the one hand, the keywords included in the documents retrieved from Scopus database are analyzed using word clouds. The word cloud [53] is a visualization technique that provides the reader a bird’s eye view of a document by representing the word frequency of a given text, such that the more times a word appears in the text, the bigger the size this word is in the word cloud. On the other hand, the structure of the citations among documents and the scientific collaborations among researchers is analyzed using ForceAtlas2 [54] and Fruchterman-Reingold [55] algorithms included in Gephi [56], an open-source and free visualization and exploration software for any type networks modelled by graphs.

3. Multidisciplinary Literature Review

The use of SRSs in different disciplines revised below. Due to space limitations, it is not possible to include in this document information on all the 1696 SRS-related documents retrieved from the Scopus database. However, this section presents a multidisciplinary literature review that aims to show that student response systems have been used in many different disciplines. The documents included in this section have been selected according to two criteria: they are articles published in peer-reviewed journals; they cover disciplines of the six top-level categories included in the science and technology classification proposed in Frascati’s manual [57]: Natural sciences, Engineering and technology, Medical and health sciences, Agricultural and veterinary sciences, Social sciences and Humanities and the arts. The authors of these publications are affiliated with academic and research institutions around the world.

3.1. Natural Sciences

Numerous studies show the advantages of using active learning strategies in science, technology, engineering and mathematics (STEM) disciplines [58,59]. For example, the use of clickers and active learning methods have revealed the improvement in the academic performance of students in science and mathematics [60]. It was presented an interesting literature review that examined the methods used to teach mathematics to nursing students, while highlighting the procedures and skills followed by them in solving problems [61]. An investigation has showed that integrating the use of SRSs with Massive Open Online Courses (MOOCs) allows physics undergraduate students to improve their learning both for small and large size lectures [62]. Other studies involving chemistry students conclude that remote control technologies integrated with peer-to-peer instruction is a time-saving method of voting and peer-to-peer teaching [63]. antos et al. [64] examined the impact of SRSs to identify low levels of student knowledge in chemistry lab classes and concluded that students consider that SRSs improve their learning process, while teachers observed an improvement in academic performance, as well as in terms of personal relations between students and teacher.

Montealegre-Ortiz et al. [6] presented a study in immunology and molecular biology courses supported by WebCT, while also using SRSs to motivate the student participation in classroom. The results obtained highlighted that SRSs promote active participation and useful feedback for both the students and the teacher. Some researchers have focused their interest in the analysis of the perceptions about active learning in introductory biology classrooms [65]. These authors observed that the use of active methodologies in large classes is a great incentive and challenge for students, as it allows them to carry out formative assessments among peer groups. Another study [59] described how students in an introductory biology course improved their level of engagement and interrelationship with their peers by making use of clickers, while this was reflected in an overall improvement in test scores, especially among higher-performing students.

3.2. Engineering and Technology

The application of an SRS has also been very significant in the different engineering and technological disciplines. It was presented a study in higher education that concluded that clickers allow students to answer a wide variety of questions interactively, then promoting a constructive discussion and interaction amongst the engineering students [66]. Other authors analysed how oral presentations of engineering students can be peer assessed by classmates using SRSs and how displaying the results in charts allowed to discuss the differences in the assessment of these oral presentations [30]. The improvement in the performance and commitment levels in engineering ethics courses using SRS is studied by Dabbour [67], while the impact of SRS for civil engineering students is analyzed by Cantero-Chinchilla et al. [68]. These authors showed an impact on performance through rapid feedback linked to formative evaluation in class, where conceptual reminders at the beginning of practical lessons increase motivation. Other studies have demonstrated that SRS was effective for students involved in a degree in mechanical engineering, as they indicated that feedback from questionnaires helps them identify possible deficiencies in their knowledge [69]. The use of SRS for peer instruction in a chemical engineering thermodynamics course in which students were asked to give written explanations according to their responses to multiple choice questions is discussed [70]. Other studies have highlighted that SRSs improve student learning and increases levels of involvement and active participation among university students enrolled in engineering instrumentation and control subjects [33]. Studies involving the use of SRSs in other engineering disciplines such as metrology [71], mechatronics [72], geotechnical engineering [73], engineering dynamics [74] and transportation engineering [75] have also been published.

3.3. Medical and Health Sciences

In some disciplines, such as medicine and nursing, the number of students in the classrooms is very high, making it difficult for them to actively participate and interact in those class sessions. Some literature reviews related to clicker use in health-related disciplines, including medicine, nursing, and allied health have been presented [76]. This is the case of the investigation carried out by Tornwall et al. [77], who studied whether or not the frequency of participation in activities guided by SRSs influenced the time required by the students to obtain their final course grades. Another review paper analysed the potential benefits of SRSs in a nutrition and dietetics course [78]. According to [79], the possibility of using SRSs for anonymous voting enables conference attendees from different disciplines (family medicine, pediatrics and obstetrics and gynecology residents) to increase their retention rates. The results obtained by Pradhan et al. [80] demonstrated how obstetrics and gynecology residents who received interactive lectures using SRSs improved their knowledge retention in comparison with those who received the standard lecture. SRSs were used to survey patients with central nervous system hemorrhage about how they were managed by clinicians in their treatment with anticoagulation therapy [81]. It was analysed the level of agreement of a group of vascular trainees with respect to the use of SRSs using smartphones, and responses revealed that they favor collaborative learning thanks to the instant written feedback [82]. It was observed that the use of an SRS (called i-SIDRA) by a large number of medical students enrolled in an anatomy of the locomotor system course improved medical degree students’ performance [23]. Other researchers have revealed emergence of using SRSs and designing of wide range of customized tools according to learner needs assessment in radiology education [21,83,84]. An interesting study was designed by Johansson et al. [85], in which several speech-language pathologists listen a playback of recordings of a set of speakers, then using an SRS to distinguish speakers with cervical spinal cord injuries from non-injured speakers.

Several papers have analysed how to incorporate SRS technology into nursing contexts [86]. Other researchers have indicated that nurse educators using SRSs promote critical thinking skills and clinical decision-making in non-threatening environments [87]. Results obtained by Jones et al. [88] showed that clickers improve assessment, increase attendance and interaction, and provide a media-enriched environment to nursing students. SRS has also been used in an adult medical-surgical to provide the students the possibility of responding anonymously, facilitate the validation of answers while providing immediate feedback, and increase the student engagement [89]. Clickers have also been successfully applied in classrooms to facilitate the understanding of complex bioscience concepts [90]. Several additional investigations have analysed the improvement provided by SRSs in terms of engagement and critical thinking of undergraduate nursing students during classroom sessions and online discussion forums [91].

Some studies have also empirically demonstrated the advantages of SRSs to improve student motivation and feedback of Pharmacy students and practitioners [92]. It was described how SRS was used to collect anonymous and voluntary responses in a 20-question survey of 306 pharmacy students [93]. Another investigation concluded that students using SRSs in a first-year pharmacy practice course increased their positive attitudes and improved their perceptions as compared with other students who did not used clickers [94]. Fabbro et al. [95] collected the responses of pre- and post-test survey questions through an SRS to pharmacists attending in a symposium to collect their perceptions of dermatology. Another investigation [18] studied the features of Socrative, an SRS application that can be installed on personal portable devices for free, in the context of pharmacy education. From the responses collected, it can be concluded that students consider that Socrative generated a suitable environment for raising and responding to questions raised in class, being also a very positive factor to actively participate in class.

3.4. Agricultural and Veterinary Sciences

Despite the use of clickers in agricultural sciences is not so intensive as in other disciplines, several contributions can be found in the literature. One of the first studies in which an SRS was applied to agricultural sciences was presented by Gupta [96]. In this paper we proposed the use of tablet computers and clickers in the delivery of an agricultural mathematics course and observed that clickers were useful tools for collaborative assessment during the examination. Sciarappa et al. [97] demonstrated a wide audience acceptance and suitability of SRSs in agricultural and horticultural programming, including farmers and master gardeners. In other cases, peer discussion and SRS questions are incorporated into the presentations of farming learning science at higher education [98]. The results of that experience suggested that these strategies favor the learning process of the least formally educated students. SRSs have also been used for collecting stakeholder opinions about policy impacts for agriculture and land conservation [99]. Crowther et al. [31] used case-based learning and SRS as active methodologies in the part of a preclinical veterinary curriculum and observed that it clearly improved the dynamics of large and small group sessions.

3.5. Social Sciences

SRSs have successfully been applied in the field of social sciences, although some authors consider that predominant pedagogical models for SRS have mainly been developed in the natural sciences, and therefore they often do not promote a way of learning appropriate to social science disciplines [100]. Several investigations have highlighted how sociology instructors often use SRSs to support conceptual application in college courses [101]. It is argued that the sociology classroom, aided by an SRS, is an effective forum for discussing and preventing misperceptions of demographics and improving understanding among students [102]. An interesting online SRS (named GoSoapBox) was described by Carroll et al. [43]. GoSoapBox was used in a subject dealing with sociology and public health that become a useful mechanism for promoting discussions and debates on topics that usually generate controversy, such as gender, religion and politics.

Researchers and academics have often used SRSs in education and psychology. Nissen et al. [103] studied how students interact to solve tasks in secondary schools’ social-science classrooms using a visual analytics interface including interactive maps and graphs. Wan et al. [104] studied the use and acceptance of clickers by learning approaches. Interesting conclusions were also obtained by Stowell et al. [105], where it was analysed how a group of psychology students answer to controversial questions and observed that students who often experience anxiety are more comfortable using SRSs than raising their hand in class. It has also been determined the existence of positive effects derived from the use of SRS-based technologies on cognitive and non-cognitive learning outcomes [106]. Conclusions obtained by Mayer et al. [107] indicated that higher education students in educational psychology who used an SRS to answer questions were more cognitively engaged during learning and scored better on course exams than those who did not use SRS technology. Another interesting experience proposed to the participants in a course in organizational psychology was to analyze in small groups the questions posed by the teacher before answering them using clickers [108]. According to these authors, this strategy substantially improved the answers compared to those participants who had not discussed the questions in groups. An investigation carried out by Fortner-Wood et al. [109] revealed that SRSs promoted engagement and reduced absenteeism among students attending psychology courses. Twyman et al. [110] analysed how to improve student learning by using different strategies, such as clickers, response cards, and hand-raising. Findings obtained in an empirical investigation involving students attending an introductory psychology course concluded that students preferred to use SRSs over handheld response cards, although SRSs did not seem to improve the academic performance [111]. The validity and reliability for collecting data and associated psycho-social determinants for upper-elementary schoolchildren using SRSs is proved by Gray et al. [112]. Some authors suggest that clickers should be used by following cognitive principles that will help students to understand and memorize the material. A recent study includes a review about the use of SRSs from a pedagogical perspective [42].

Academic and researchers of economics, accounting and marketing disciplines have successfully used SRSs. The results presented by Creese [113] describe an experience carried out in a university classroom with students of economics in which the SRS was used to ask them about their knowledge on different topics. The benefits of clickers for students attending a subject on principles of economics are shown by Salemi [114]. Some researchers suggest that using interactive teaching tools such as Kahoot can be useful to teach economics to non-economists [115]. Interactive tools have also been analysed in terms of academic performance in a subject about global economic environment and deduced that students with a better global mindset do not take advantage of SRSs as weaker students do [116]. Carnaghan et al. [12] presented a review about the use of SRSs with accounting students that provided evidence of student satisfaction and engagement using clickers and also improvements in learning and behaviors. The conclusions obtained by Frick et al. [117] demonstrated that it is possible to enhance the attitudes of students attending traditional accounting lectures by applying a blended learning model that combines online pre-reading material and an SRS to answer multiple-choice questions. The effectiveness of an SRS (named Quizizz) on enhancing students’ learning experiences in an accounting classroom was demonstrated by Zhao [25]. Recently, some investigations have analysed the advantages of mobile technology-based SRSs in comparison with students in classes using traditional clickers [118]. Results show that students using SRSs obtained better qualifications and reported a more positive experience in financial accounting courses. A practical experience provided insights about the advantages of using Socrative software to improve the student motivation and knowledge acquisition when the students respond to questions in class via mobile devices in a marketing research course [119]. The investigation described by Rana et al. [120] analysed the opinions of a group of students who used SRS for the learning digital marketing interactivity and the results revealed that the interactivity offered by clickers improved levels of satisfaction, promoted positive attitudes and improved the level of academic performance compared to another group of students who did not use clickers. Graduate students attending a course in entrepreneurship management were surveyed using SRSs [121] and results indicated that it is a useful and effective tool for enhancing students’ interest about entrepreneurial knowledge acquisition. A literature review on student perceptions and outcomes of the use of clickers in business administration and management disciplines is presented by Rana et al. [122].

SRSs have also been proven to be an effective tool in large courses of political science [123]. In other cases, a set of clicker-based techniques and activities are designed for political science teachers to adapt their courses to innovative educational practices [124]. It was presented a new SRS in the context of political science that promotes a new teaching paradigm in which students play a central role in classroom activities [24]. Another innovative design is described by Thompson [125], where it is proposed to improve students’ levels of understanding of certain concepts related to political science through SRSs within an electoral system in order to analyze different voting behaviors. SRSs have been compared with social networks (Twitter) in international relations courses, and results revealed that SRSs outperformed Twitter in terms of students’ experiences and performance [126].

3.6. Humanities and the Arts

SRSs are applicable to humanities disciplines which require divergent questioning, and supported complex interactions between engagement, attendance, and preparation. Some investigations have analyzed how SRSs can enhance the participation and engagement in law courses, which is particularly interesting if it is considered that the study of law crosses the boundaries between the social sciences and humanities. Habel et al. [19] presented the results of a research-action project in the classroom carried out in order to know the influence of an SRS based on mobile phone voting (using VotApedia) among law students and concluded that the technological intervention increases student engagement and formative assessment.

On other hand, Slominski [127] presents an analysis about how discussions can be prompted and enhanced using SRSs within religious studies classrooms at public colleges and universities and concluded that clickers provide an opportunity to integrate students’ assumptions and stances about religion without putting individuals on the spot for their beliefs. Other authors indicate that SRSs are effective technological teaching resources for use in religious education and concluded that it creates safe learning environments and an open atmosphere for discussing religious subject matter [128].

SRSs have also been widely applied in language education. This is the case of the investigation presented by Langman et al. [129], which authors presented a case study examining the effects of using SRSs on patterns of discourse in an English high school and concluded that it encourages students to become active participants. The impact of SRSs in the process of learning new languages is discussed in detail by Cook et al. [130], which conclusions are that students who had used an SRS in class acknowledged that they had developed critical thinking about the activities, while scoring significantly better compared to those students who did not use this methodology. It has also been observed that integrating SRSs for formative assessment within the English as a foreign language classroom allows teachers to detect the students’ knowledge gaps and stimulate their attention and engagement [131]. Some researchers have proposed the use of SRSs with flipped-classroom methodology [132], such that their application enhances the learning achievement and motivation of students attending courses of English as foreign language [133]. Other empirical studies have also demonstrated that the use of SRSs helps to re-engage students after their attention fades during a philosophy lecture course, improves the attitudes and stimulates discussion among students [134].

4. Analysis of the Network of Publications

This section analyses the most investigated topics related to SRSs and which are the most important papers according not only to the citation count, but also to their position in the network (graph) structure. It is therefore not a simple task, as there are many articles on different topics. One way to analyze this information is to consider the keywords in the manuscripts. Authors usually include as keywords those words or phrases relevant to the topic addressed in the articles so that the reader can know quickly and easily if the document may be of interest. Using the proposed methodology, it is possible to process and refine in a quick way the scientific production indexed in Scopus up to 2019.

Figure 1 shows the network of scientific publications related to SRSs indexed by Scopus in the period indicated. Each node represents a publication and the edges between two nodes denote that one of the documents cites the other. The network has been built using the ForceAtlas2 plugin [54] included in Gephi. ForceAtlas2 is a force-directed algorithm that produce visual clusters to denote structural densities. This network contains a total of 1686 publications (nodes), with 6730 edges (citations between publications). As can be seen in the inner part of the figure, there is a group of connected publications (1108 documents), while there is another group of publications (578 documents) represented on the outer part that have no relation to internal documents. In other words, the nodes located on the outside of the network do not cite any documents from the inside, nor are they cited by the latter. Since the most frequently cited SRS documents are included in the internal part, documents on the outside part may include SRS-related terms in the title, abstract or keywords but without going into detail about these technological teaching resources in the rest of the article.

By using the “Giant Component” filter available in Gephi, it is possible to analyze the inner (connected)-group of publications, which contains 1108 documents, with 6604 edges. Figure 2 shows these publications colored by type of document, while the node size is set in relation to the number of relationships it has with other nodes. The largest node corresponds to the highly cited Caldwell’s article [135]. As it can be seen, most documents related to SRS are published in journal papers (67%, red), and conference proceedings (25%, yellow), while there are a few documents published in books (8%, green). It is also observed that nodes having a higher degree often correspond to journal papers, that is, journal papers often receive a larger number of citations than the papers published in books or conference proceedings.

By applying a filter by type of publication to the Figure 2, it is possible to obtain the network of publications corresponding to peer-reviewed journals, as Figure 3 shows. The 10 journals with the highest number of SRS-related articles have been highlighted. By applying the statistical tools of Gephi to this filtered network, it is possible to conclude that top 4 journals publishing papers related to SRSs cover about 7.6% of the total of publications about this topic (including journals, conference proceedings and books) and about 11.4% of the journal papers: Journal of Chemical Education (3.79% of the documents); Computers and Education (2.57%); CBE Life Sciences Education (2.57%); Journal of Science Education and Technology (2.44%). Figure 4 shows the articles published in these four journals, where it can be clearly seen that published articles tend to include citations from other articles published in the same journal. In fact, 25 of 28 papers (89.3%) related to SRSs that have been published in the Journal of Chemical Education cite other papers of the same journal. This percentage is even higher (94.5%) in the other three journals shown in Figure 4. In any case, journal self-citation is a common practice in the academic community [136].

On the other hand, the proposed methodology intends to go one step further in order to apply community detection methods [137] to the network of publications shown in Figure 2. The reason for this is that different previous studies have shown that community detection is a useful tool for extracting information from the structure of complex networks. Figure 5 shows the five communities detected using the Louvain method implemented in the statistical package provided by Gephi. As it can be seen, the largest community (yellow) includes two large nodes [37,135]. At the bottom (red) and at the top left (green) there are two communities that contain numerous nodes, some of them of medium size in their central part. Finally, on the right side we can see the blue community, which consists of a reduced set of scattered nodes. The analysis of the keywords of each community using word clouds will be useful to establish a preliminary classification of a great number of documents without the need to read and analyze all of them.

Figure 6 shows the word clouds in the keywords of the four communities containing a larger number of nodes. It is important to notice that generic words have been removed from the word clouds to focus the study on the methods, tools and the areas of application of SRSs. The four largest communities are (from left to right and up to down): a community of papers focused to pedagogical aspects of the use of SRSs; another community the technology and applications related to SRSs; a community closely related to medical and nursing disciplines; and a community mainly related to physics-chemistry. The fact that some of these communities correspond to science disciplines reinforces the conclusions of previous studies that stressed that SRSs are effective in applied hard sciences [138].

5. Analysis of the Scientific Collaborations

The previous section analysed the main topics in which academics and researchers have applied SRSs, as well as the main sources in which the results of these studies have been published. Now, they are analysed the main characteristics of the network of scientific collaborations between the authors that have co-authored the papers commented above. According to the data retrieved from Scopus, there is a total of 4091 authors (nodes) and 11,668 collaborations (edges).

Figure 7 shows the network of authors (nodes), where edges indicate that two authors have co-authored at least one paper of those analysed in previous Section (including internal and external papers of Figure 1). In this case, some different layouts will be built based on a force-directed layout procedure named Fruchterman-Reingold algorithm [44]. As Figure 7 shows, the authors affiliated to academic or research institutions of the top 8 countries have been highlighted in different colors. In quantitative terms, scientific production related to SRSs is led by academics and researchers affiliated with institutions from the United States of America (52.4 %), United Kingdom (5.2%), Germany (4.3%), Australia (3.7%), Canada (3.2%), Spain (2.7%), India (1.8%) and China (1.6%). As expected, there is a greater relationship between different researchers from the same country, although there are also collaborations between co-authors from different countries. It is noticed that the larger network of collaborations located at the center of the network correspond to a large group of authors that have published some papers related to SRSs [139,140].

In addition, using the network visualization software it is also possible to focus the study in authors of a given country. For example, Figure 8 shows these authors affiliated to institutions from the United States of America. The ten universities having a larger number of researchers that have published at least one paper related to SRSs are highlighted in a different color. It is possible to observe significant number of contributions from authors affiliated to San Francisco State University (2.1%), Mayo Clinic (1.6%), University of Colorado at Boulder (1.6%), University of California, San Diego (1.2%), and University of Washington (1.2%). On the other hand, in line with conclusions obtained from Figure 6, it is important to remark that there are several medical institutions among the 50 institutions having a larger number of authors involved in SRS issues: Mayo Clinic, University of Arkansas for Medical Sciences, The Johns Hopkins School of Medicine, Medical University of South Carolina, Mayo Medical School, Carolinas Medical Center, Henry Ford Hospital, Columbia University Medical Center, etc. A similar analysis can be applied to other countries, as it is done in Figure 9 for these authors affiliated to British academic and research institutions.

6. Conclusions

This paper fills a gap in the educational technology literature since it provides new information about the use of SRSs in different disciplines using visual analytics. In particular, we used a bot that retrieves information from a large number of papers included in the Scopus database. These documents have been analysed to present a multidisciplinary review of the use of SRSs that includes visual representation of the network of publications in order to detect the main disciplines of application of SRSs. Furthermore, the network of worldwide collaborations between researchers investigating in this topic is also studied. The analysis of the literature in this field provides some interesting conclusions. On the one hand, most investigations included in this review conclude that the use of clickers is a pedagogical resource that is well-perceived by lecturers and students. Some studies indicate that SRSs allow students’ attention to wander and provide a fast feedback for the lecturer to evaluate the students’ knowledge about a given subject, although the use of SRSs does not always produce a significant improvement in the academic performance of the students. Some investigations indicate that clickers may facilitate students to organize and understand material presented in the classroom, while other educational technology researchers highlight that the benefits of using SRSs outweigh the costs and time required to prepare the sessions. The analysis of the network of publications reveals that most of the publications related to SRSs are published in peer-reviewed journals with physics, chemistry, medical, and nursing disciplines being the most common areas of application. On the other hand, the network of scientific collaborations indicates that the relationship between researchers from the same country is stronger than that between researchers from different countries. More active researchers in the study of SRSs are those affiliated to academic and research institutions from the United States of America, United Kingdom, Germany, Australia, Canada, Spain, India and China. A methodological limitation of this study comes from the fact that the results are based on the data retrieved from a unique database, because it is not possible to identify research that has been disregarded. Besides, the networks of publications and researchers do not display all the data retrieved from the Scopus database. As future work, it is planned to apply this methodology to the study of other educational technology issues. Furthermore, according to the report "Education during COVID-19 and beyond" published by the United Nations, the SARS-CoV-2 coronavirus pandemic has created the largest disruption of education systems in history, such that 94% of the of the world’s student population have been affected by closures of learning spaces, requiring in many cases teachers to move to online delivery of lessons. Therefore, as future work, we plan to analyze the use of SRSs in online learning environments in the post-COVID19 era.