Gamifying Massive Online Courses: Effects on the Social Networks and Course Completion Rates

Abstract

This paper analyzes the effects of gamification in the social network of a massive online course. An educational social-networking platform gathered information about the contributions of participants and about the social networks that were formed during the course. A gamification layer with three game elements (points, badges, and leaderboard) was then implemented in the online learning platform. Social network analysis (SNA) and principal component analysis (PCA) were used to analyze the differences between a treatment and a comparison group (N = 591 and N = 427), using a set of 20 variables for each participant which quantified contributions to the learning platform as well as position and influence in the social network. The results of SNA show that gamification influences the structure of the social network of the course. The results also suggest that the variables cluster similarly for each group and that the linear combination of variables called the first component (F1) is a good descriptor of students’ work and position in the network. F1 can be used to build predictive models of course completion. The models show that the probability of passing the course increases more rapidly in the treatment (gamified) group.

1. Introduction

Gamification is the use of game elements in non-game contexts to promote participation and motivate action [1,2] Framing activities as a game through game elements, such as points, badges, and leaderboards, holds as much psychological power as the full game mechanics [3]. Gamification increases users’ performance of simple repetitive tasks [4], but findings of its motivational effects are contradictory [5]. Effectiveness is also in question, pointing to the necessity to align gamification with the goal of the activity and to address the psychological needs of the expected users at design time [6].

Educators are trying to harness the potential of gamification to design motivational learning experiences. Not surprisingly, education is the most common context in which gamification is implemented and reported [7,8,9]. Gamification can be used in traditional learning as well as blended and online learning. A recent literature review suggests that empirical studies equally addressed both online and regular courses, although online learning is considered a more promising field to apply gamification because it usually requires stronger motivation [10]. Since it intends to promote participation, gamification is particularly promising in massive online courses such as massive open online courses (MOOCs) in which the participation of instructors is limited, and feedback depends on the contributions of learners. Social networking presents a unique opportunity to develop peer support, thereby filling this instructional void [11]. E-learning can also take advantage of personalized and adaptive methods such as testing for measuring student performance [12], which can be used to tailor content and feedback. However, the effects of gamification in the MOOC context have not been thoroughly explored yet because it is being implemented in similar ways to those in small-scale contexts [13]. Since gamification and social networking serve different but complementary purposes, approaches combining both can result in motivationally-enhanced-social-learning experiences. Gamification can increase participation, as measured by contributions within the social network, resulting in increased social interaction and communication, which, in turn, would also result in motivational gains that fuel further participation.

Social network analysis (SNA) provides a set of measures for a social network and its participants. In their literature review, Cela et al. [14] pointed out that the application of SNA to e-learning environments is at a very early stage, although the number of studies is increasing. Network measures can be used to analyze the social interactions and the structure of the network in courses and the influence that different elements have on the network [15]. If gamification motivates participation in a social networked setting, it should be reflected in the contributions and in the structure of the social network, and SNA can be used to find such effects. Measures of each individual participant represent his/her position and influence in the network. Such measures can reflect the effects of gamification in the social network, but they also provide the means to analyze learning performance in relation to the contribution, position and influence in the network. Previous studies suggested that network metrics can be used to predict user behavior and characteristics [16].

This work sets out to study the effect of game mechanics (points, achievements, and leaderboards) in the social network of massive online courses in which the participation of facilitators is limited, and students have to take an active role in the learning platform. Since positioning in the network can also impact course completion, this study also aims to analyze the effects of gamification on course completion by building predictive models of it.

2. Related Work

Game elements such as achievements and badges serve several functions, for example, giving instructions, setting goals, and providing reputation, affirmation, and group identification [17]. Existing research presents mixed results regarding the impact of gamified elements such as badges, points, and leaderboards in learning outcomes. For instance, initial studies suggested a positive effect on the practical assignments but a negative effect on conceptual learning [18] of undergraduate students. Similarly, Hew et al. reported greater student contribution in the discussion forums but no significant difference in students’ recall of factual knowledge in an undergraduate course. They also found that the quality of the learning artifacts produced by students who used gamification mechanics was higher and that game mechanics also had a positive effect on motivating students to engage in more difficult tasks when compared to a control group [19]. Landers and Landers pointed to the necessity to study the effect of each game element on learning. They reported that students using the leaderboard spend more time on a task, and that time-on-task positively correlated with learning outcomes when compared with students who did not use the leaderboard [20]. Boticki et al. studied the effects of a gamified mobile application using badges in primary education. They found that the use of the app had a significant effect on learning, as measured by the quantity and quality of students’ contributions [21]. Contrastingly, other studies reported no effects or even adverse effects. For instance, Attali and Arieli-Attali studied the effect of points in computerized assessment. They found that points did not affect the accuracy of responses, although the speed of responses increased [22]. Hanus and Fox experimented with the effects of gamification in motivation using game elements aligned explicitly with learning objectives. They found that students who used gamification were less motivated in class and got lower grades than a control group [5]. When it comes to MOOCs, Qu et al. reported on student achievement based on temporal learning behavior in the learning platform [23].

Gamification addresses the psychological needs for competence, meaningfulness, and relatedness by providing timely feedback on learning progress and social experiences [24]. It also partially addresses the need for relatedness by providing recognition and status with elements such as badges. Social networks further support relatedness by facilitating communication and collaboration among participants. However, a significant limitation is that most of the existing research on the utility and effectiveness of social media in higher education is limited to self-reported data (e.g., surveys and questionnaires) and content analyses [25]. For instance, Hamid et al. analyzed students’ perceptions of social networks for learning and reported the following benefits: engaging with content, peer learning, promotion of critical thinking, self-directed learning, self-monitoring of learning progress, and interaction with lecturers [26]. Educational social-networking sites provide data for analysis, like the contributions of participants and the connections between them. SNA can then be used to mitigate the limitations of self-reported data by providing a set of measures that represents the student’s position and influence in the network. SNA also facilitates finding tendencies and patterns among participants in learning environments that can guide further analysis and even prompt intervention [15]. As participants create new bonds, the structure of the network and the positions of individual participants change over time [27]. The density and centrality of groups of students also increase over time when lecturers participate [28]. Cho et al. found that two centrality measures (degree and closeness centrality) were associated with students’ final grades [29]. Paredes and Chung also found that social learning is influenced by the relations, position and structure of the social network [30]. Shen et al. used SNA to describe interaction, explaining how it influences the sense of community of students in online learning environments [31]. Students in a structured network act as bridges, and they also trigger learning interactions, as compared with non-structured settings in which students mostly take passive roles and facilitators become the center of activity [32].

Still, the number of studies that bring together gamification and social networking is limited. De Lima and Zorrilla suggested that network metrics can be used as elements of gamification because they quantify the relevance of participants [33]. Simoes et al. presented a social gamification framework to assist teachers in creating motivational learning experiences fitted to learners’ needs [34], but the authors did not analyze the effects of it. Thom et al. reported the effects of removing gamification from an enterprise-social-networking site, finding that the number of contributions decreased significantly [35]. de-Marcos et al. compared gamification and social networking, concluding that both yielded similar results regarding learning performance for undergraduate courses; however, both approaches are best suited for practical assignments but not appropriate to convey conceptual knowledge [36,37]. The structure of gamification-driven educational social networks is a small world that resembles the features of similar networks analyzed in other domains. However, predictive models of learning success returned mixed results and were not very representative [38].

Existing research analyzed the structure of educational social-networks, changes over time, and the influence of other factors such as the intervention of teachers. It also informed about the effects of the social network on a variety of learning and other outcomes. This study contributes to the current state of the art (1) by analyzing how gamification influences the structure of the social network, and (2) by providing new models that describe learning success in terms of the contribution and positioning of individuals in the network. To the best of our knowledge, there are no previous works that study the former (number 1 above). Studies reporting the latter (number 2 above) offered mixed results that do not relate activity in the network and learning performance. Furthermore, most of the reported results focus on traditional or blended settings with a few participants, but our study reports the effects in a massive educational setting.

3. Materials and Methods

3.1. Research Questions

This study explores the following research questions related to gamification and social networking in a large MOOC-like educational setting:

RQ1. What is the influence of gamification on the structure of the social network in a large-scale online course?

RQ2. How does gamification relate to course completion in a large-scale online course?

To address the first research question, we explored the similarities and differences between basic educational networking and gamification-driven social networking, which may reflect the effect that gamification has on the social network. A gamification layer that includes achievements, points and a leaderboard is deployed on the top of a social networking site to explore such effect. We then studied the structure of the social network by examining the graph of connections between participants at the end of the course. SNA then provides a set of network metrics that can be used to compare the courses. Furthermore, SNA also provides a set of useful metrics to characterize each participant (student in this study). Metrics can be grouped using statistical methods to analyze the differences among groups of students that use different learning instruments.

The second objective of this study is to analyze the impact of gamification on learning success in a large-scale autonomous learning environment. Network measures of each participant can be used to build predictive models about learning success based on the student’s work in the course and on the student’s positioning in the network. Such models can also inform about differences among groups, providing further insights into the effect that gamification may have in the social network.

This study focuses on centrality measures to analyze the position and influence of participants in the network. The preeminent use in this study of centrality is based on previous evidence of the effect of such metrics in the learning and performance of students, as reported in Section 2. Alternatives may include segmentation measures and, particularly, community detection algorithms. This study also considers one of these measures (clustering coefficient).

3.2. Setting

The educational setting was the undergraduate online course, “Digital Skills for Teachers”, which is offered by an open university. The course uses a MOOC approach that requires autonomous and independent learning with a strong emphasis on the social and collaborative dimension. The level of involvement of each student depends on his/her interests and needs. Students have to work independently, exploring and finding relevant material, reflecting on their learning experience, and producing artifacts that show evidence of their competence and understanding of the topics. Students should also take an active role in the ongoing class dialog, contributing to the collaborative building of knowledge. The learning activities of the course are just a guide to support learning and facilitate collaboration and discussion of the course topics. The course is offered free of charge in Portuguese to teachers and teaching students worldwide.

The syllabus includes three learning modules: (1) searching and sharing online resources; (2) using digital tools in the classroom, and (3) promoting collaborative learning using digital tools. The course has a duration of 6 weeks. Students have to complete two assignments. The first consists of designing and building a learning experience based on the use of learning technologies (modules 1 and 2). The second assignment is an essay describing a cooperative/collaborative learning activity that includes the pedagogical use or design of new digital tools to support and facilitate learning (module 3). Assignments are anonymously peer-reviewed using a rubric by three other students. Students who get a positive assessment (pass) in both assignments obtain a digital certificate that acknowledges course completion.

3.3. Instruments

A social-networking educational online environment provides support for the pedagogical needs of the course. It uses the Moodle learning management system to centralize all course information and the Elgg social networking engine. Both platforms are integrated using a single sign-on service, offering a transparent learning environment which delivers the following tools and resources to participants: news, learning guide, dashboard, blogs, bookmarks, and an internal tweeting system. The basic version of the course measures the contributions of each participant including the number of blogs, tweets, “likes”, messages, comments, followers, and logins. A plugin deployed on the Elgg social networking engine provides the gamification features. It implements a set of thirteen achievements, points, and a leaderboard. Students get achievements for their participation and for interacting with the learning environment and with other participants. Achievements have a visual badge and a short description of the requirements to get them (Figure 1). For instance, the achievement, “iPopular”, is given to participants who are followed by at least 25 other participants. Points are given for completing the learning activities and for getting achievements. Points are then used to implement a leaderboard that students can use to compare their performance with other participants. Besides the measures provided by the educational social networking site, the gamification layer provided measures of the number of achievements and points for each participant.

Self-determination theory [39] provided the theoretical background that guided the inclusion and design of the gamified social elements. These are designed to provide means to afford the basic needs of competence, autonomy, and relatedness in alignment with learning objectives. By providing gamification elements to address these affordances, we aim to motivate participation and interaction in the learning platform that may eventually result in students producing more and better assignments that increase course completion rates. Competence is the feeling of adequacy and possession of the skill, knowledge, or qualification required to complete the current task. Competence was acknowledged by the gamification system by providing points for completing the assignment. Autonomy is the feeling of freedom in one’s actions. By providing a set of thirteen different achievements, the gamified layer offers students different choices. Relatedness is the feeling of association and connection with other participants, and of being part of a community with common interests or goals. The social component of the system provided different opportunities for connection, such as following other participants. The social dimension was also emphasized by the peer-review, the social-based achievements, and competition through the leaderboard.

3.4. Participants

Experimentation included two instances of the course. A control group of 427 students used the social-networking online environment to take the course. The social-networking environment that included the gamification layer was used by the experimental group (591 students). Students were assigned to the group based on the date in which they enrolled, so a quasi-experimental design was used. Course contents and teachers were the same. Evaluation items and criteria were used consistently in both editions. Course contents and teachers were the same in both instances of the course. Evaluation items and criteria were used consistently in both editions.

Table 1. Geographic location, gender, educational background and occupation of participants as a percentage of the sample.

	Comparison (n = 427)	Treatment (n = 591)		Comparison (n = 427)	Treatment (n = 591)
Geography			Gender
Portugal	46.84	40.95	Female	69.32	65.48
Brazil	24.12	27.24	Male	30.68	34.52
Spain	6.56	5.41
Other (<5%)	14.52	16.24	Occupation
n/a	7.96	10.15	Teacher (primary)	5.15	5.92
			Teacher (secondary)	29.04	22.67
Education			Lecturer (tertiary)	12.88	10.15
Secondary	7.96	8.12	Teacher (other)	13.11	15.91
Graduate	35.60	38.58	Education (other)	7.73	6.26
Postgraduate	51.52	47.38	Student	14.52	17.60
n/a	4.92	5.92	Other	9.60	11.68
			n/a	7.96	9.81

presents the socio-demographic data of participants. It suggests that both groups are similar regarding the characteristics of learners. As registration was freely open in the two instances of the course, twenty-nine students participated in both editions, and three of them passed on their second try. Although the starting date was different in both editions, it was a short 6-week course starting in the middle of the term in both cases.

3.5. Measures

The interaction of participants in the social-networking online learning environment results in a set of measures that quantifies their participation. The social-networking online environment facilitates the gathering of the following measures: blogs, tweets, “likes”, messages (to other participants), comments (to any publication), followers, following, logins, and total interactions. All measures refer to the number of contributions made. The Elgg social networking engine implements a system of followers, similar to Twitter. For each participant, it is possible to know the number of followers that he/she has and the number that the participant is following. Total interactions return the number of actions (requests sent to the web server) that participants completed in the online environment. Most browsing actions in the learning system are not reflected in any of the previous measures. For instance, clicking on a particular link may be meaningful for the learner, but it is not recorded as a contribution because such action does not result in new information stored in the web server’s database. So, the total number of interactions is an estimate of the total amount of actions that participants perform in the learning platform. For the experimental group, the gamification layer gathers two additional measures about the interaction with the game elements: the number of achievements and the number of points that each participant earned.

SNA can be used to mine the social graph of participants, returning a set of network measures for each participant. Network measures quantify relevance in the network and the quality of connections. The definitions of measures used in this study are as follows: Degree is the total number of connections of a participant. Closeness centrality is the average of the shortest distance to all other participants in the network. Eccentricity is the largest distance between a participant and any other participant. Betweenness centrality reflects the number of shortest paths in the network in which a participant takes part. Eigenvector centrality is a measure of the influence of a participant in the network, based on relative scores to all other participants. Clustering coefficient returns the level of embeddedness of a participant with her neighbors. It is a measure of community formation (network segmentation) that can help to identify the level of connection of students within their local group. Clustering coefficient contrasts with the other centrality metrics that represent the position in the network.

Link analysis algorithms offer quantitative measures of the importance of a node, based not only on the quantity but also on the quality of edges. There are three of these metrics also used in this study; namely, PageRank, authority, and hub. PageRank [40] measures the importance of participants by considering the quantity and quality of links. The more important participants are likely to have more followers. Authority and hub also measure the importance of participants in the network. Both are computed using the Hyperlink-Induced Topic Search (HITS) algorithm [41]. Authority estimates the value of a participant as a function of his/her connections. A good authority is followed by many hubs. Hub estimates the value of the links to other participants. A good hub follows many participants.

Students credited or passed the course, getting a certificate; or they failed the course, based on learning performance as measured by the reviews of the learning artifacts. A binary variable called “Certificate” was included in the dataset. All measures were made when the instances of the course finished and reflected the activities and the structure of the social network of participants at the end of the course.

3.6. Analytical Tools

The primary instrument used to analyze the instances of the course was the principal component analysis (PCA) [42]. PCA determines the orthogonal axis of maximum variance by finding the eigenvalues and the eigenvector of the correlation matrix. Variables are projected onto those axes, calling these “factor loadings”. Observations, when projected onto the axis, are called “factor scores”. Factor loadings show how variables group together. The dimension of a dataset can be reduced when factor loadings of different variables are somewhat close in their biplot, indicating that the variables are very closely correlated. Furthermore, if the factors resulting from PCA are adequate and interpretable, then the factor scores can also be used to analyze correlations and build predictive models of desired outcomes. Since groups can be different, independent PCAs were carried out for each dataset (group).

The data gathered satisfy PCA assumptions. Firstly, all variables were measured at the continuous level. Second, we can assume a linear relation between variables since they all represent contributions to the learning platform, or they are measures of position in the social network. Third, Bartlett’s test of sphericity returned that data for both groups were suitable for data reduction (p < 0.001 in both cases). Fourth, the Kaiser–Meyer–Olkin (KMO) test returned 0.880 for the experimental group and 0.788 for the control group suggesting that data had sampling adequacy. Finally, no significant outliers were found. A Grubb’s test returned two possible outliers with a significantly high number of messages and interactions. Subsequent analysis showed that this was due to a mistake in data collection that added messages from other courses. Records were fixed and included in the dataset. We also checked that this error did not occur in other records.

The Gephi visualization-and-exploration software [43] was used to create the graphs that represent the social networks and to compute their global metrics. Gephi was also used to compute the individual metrics of each participant.

4. Results

4.1. Social Networks and Descriptive Statistics

The final structure of the social network of participants in each group is represented as a directed graph in which nodes are participants and edges are connections (followers). In massive online courses, many participants do not do any work and do not follow other participants. Two hundred and eighty-seven participants in the experimental group and 269 in the control group did not make any connection. Only nodes with edges are included to improve the readability of the graphs. A connected component is a subgraph in which all nodes are connected. Both instances of the course present only one central connected component. Figure 2 presents the central connected component of the experimental group (313 nodes), and Figure 3 shows the central connected component of the control group (167 nodes).

For the experimental group, we see that there is a central cluster with many nodes and many connections between them. There are also several hubs (nodes with many connections) and only a few nodes in the periphery with one or two connections. The density of connections makes it difficult to see if two particular nodes are connected. For the control group, we see only a few hubs with many connections and many nodes with only one or two connections. We observe three nodes with more than 100 connections; those correspond to the teachers. Contrastingly, in the experimental group, central nodes are both students and teachers. The five nodes with more than 130 connections comprise three students and two teachers, and the node with the most connections in the experimental group is a student. Although the differences in Figure 2 and Figure 3 could be attributed to the differences in the size of both groups (591 vs. 427), we can observe that the number of connections is 4.48 times larger for the treatment group, suggesting that something besides the size of the group shapes the final structure of the social network.

Table 2. Comparison of network metrics for the different instances of an online course.

	Experimental Group	Control Group
Nodes (including teachers)	600	436
Edges	3200	715
Avg. degree	5.33	1.64
Graph density	0.009	0.004
Avg. path length	2.29	2.59
Nodes with degree <> 0 (at least 1 connection)	313 (53%)	167 (39%)
Students who get certificate	31 (5.25%)	14 (3.28%)
The minimum degree to get a certificate	3	1

presents descriptive statistics and network metrics—the average degree increases (5.33 for the experimental group and 1.64 for the control group). Graph density represents the ratio between the number of edges and the number of possible edges. A graph is complete (density = 1) if all possible edges between each pair of nodes are present. The experimental group graph has twice as much density as the control group. The number of participants with connections, the number of participants who passed, and the minimum number of connections to pass are also higher for the experimental group. The average path length (average shortest distance from one node to all others) decreases for the experimental group, suggesting that additional connections reduce the distance between participants.

4.2. Reduction in Dimensionality

The results of the PCA for the treatment group show that the first component (F1), which represents 68.44% of the variability, is associated with all variables except the Messages and the clustering coefficient (

Table 3. Results of the PCA: squared cosines of the variables, eigenvalues, and variability of the factors. Values in bold correspond to the factor for which the squared cosine is the largest.

	Experimental Group			Control Group
Variable	F1	F2	F3	F1	F2	F3
Blogs	0.773	0.011	0.017	0.496	0.244	0.004
Tweets	0.743	0.110	0.030	0.382	0.530	0.000
Likes	0.618	0.224	0.003	0.520	0.003	0.048
Messages	0.021	0.000	0.864	0.265	0.020	0.058
Comments	0.742	0.108	0.001	0.462	0.350	0.000
Followers	0.897	0.031	0.003	0.825	0.013	0.112
Following	0.787	0.046	0.002	0.441	0.171	0.114
Logins	0.589	0.001	0.002	0.486	0.191	0.001
Interactions	0.758	0.135	0.008	0.604	0.338	0.008
Achievements	0.831	0.014	0.008	-	-	-
Points	0.822	0.002	0.019	-	-	-
Degree	0.926	0.004	0.003	0.757	0.112	0.006
Eccentricity	0.477	0.340	0.000	0.587	0.113	0.028
Closeness centrality	0.451	0.346	0.001	0.573	0.111	0.026
Betweenness centrality	0.648	0.100	0.014	0.518	0.071	0.073
Clustering coefficient	0.090	0.326	0.039	0.754	0.025	0.001
Eigenvector centrality	0.884	0.050	0.002	0.085	0.009	0.705
Authority	0.894	0.037	0.003	0.699	0.021	0.235
Hub	0.879	0.023	0.009	0.759	0.015	0.158
PageRank	0.855	0.015	0.014	0.752	0.124	0.030
EigenValue	13.687	1.922	1.041	9.965	2.459	1.608
Variability (%)	68.44	9.61	5.20	55.36	13.67	8.93
Cumulative %	68.44	78.05	83.25	55.36	69.03	77.96

), suggesting that F1 is a measure of the students’ global work. Indeed, F1 is a weighted sum of different variables. The weights are proportional to the cosines between the variables and the principal component axis, that is, Pearson’s correlation between the variable and a principal component value. Table 3 also suggests that several variables can measure the same thing; i.e., the variables are associated with the same principal component with a similar square cosine. As the square root of 0.5 is 0.707, any square cosine greater than 0.5 indicates a strong correlation between the variable and the principal component. In Figure 4, we present the correlation plot of factor loadings for the treatment group. We can see that there are at least five clusters of variables, potentially meaning that the twenty variables described previously can be reduced to five or six if we select one representative variable per cluster. Each group represents closely correlated variables. For example, closeness centrality and eccentricity measure similar things.

For the comparison group, the first component (55.36% of variability) includes all variables except Tweets, Messages, and eigenvector centrality (Table 3). Therefore, for both groups, F1 is associated with the most course activities and measures, suggesting that it is a functional characterization of the students’ work. The correlation plot of the comparison group (Figure 5) shows that variables are even more clustered, showing only 2 or 3 groups, which implies that for this group, most of the measurements taken measure similar constructs. In statistics, this is called collinearity, and it is a severe problem as it tends to make regression models biased to the current sample, meaning that the model seems to represent very well the current sample but will do so poorly if tried on additional data. In standard regression, collinear variables should be eliminated from the analysis. Going even further, we could also argue that those activities should be eliminated from the course as they achieve the same result over and over, but we will not go that far at this point. The use of the results of PCA to create predictive models, as shown in the next section, significantly reduces the effect of collinearity in variable prediction.

If we compare the results of both groups, we can see that they are similar in terms of how variables group together and that, also in both cases, the first component (F1) is a good descriptor of students’ work since it includes most variables. We can use this information to build predictive models of course completion based on the activity completed by students as represented by F1. This can also provide valuable information to instructors monitoring future courses since we can focus only on a subset of 2–3 instead of all variables, or on F1 if they can perform a PCA, to analyze the results of the learning action. For instance, we could study the scores of students for F1 during the course to determine the levels of participation of students and decide whether to take further action to motivate them or even provide automated individual feedback based on the results of this score.

4.3. Predictive Models

Because for both groups, the first component (F1) includes most variables, we analyzed whether F1 (factor scores) can be used as a predictor of the probability of learning success. Remember that F1 is a weighted sum of variables. As the variable "Certificate" can have only two values (“1” for Certificate and “0” for course fail), testing the relationship is difficult, as shown in Figure 6A that plots Certificate vs. F1. Even though the relationship exists with R² = 0.33, the binary nature of the Certificate makes the appreciation of the relationships hard to see.

For a better characterization of the relationships, we introduced a new measure called the success probability, or in this case, the probability of getting the certificate. To estimate this probability, we ordered the dataset by F1 and sampled the certificate values above and below a given value of the variable F1, taking n samples that created a sliding window of samples. In essence, we considered each result (certificate or not) in the sample as a Bernoulli experiment and used the sample mean of the Certificate variable as an estimator for the probability, which is a well-known-maximum-likelihood estimator [44]. This estimator is similar to the Krichevsky–Trofimov estimator, which is used as a conditional estimator of the outcome of the next Bernoulli experiment [45]. However, since our dataset is not a time series and our objective is to create a global model for the success probability as a function of the variable F1, we used the sample mean of the current window, as shown in [46,47]. In general, n data points from the plots are lost (n/2 points at the beginning of the plot and n/2 at the end). Initially, we used n = 11, which is 11 samples from the ordered list, as the window size. The samples are taken, and then the number of successes (or certificates) is counted.

Assume $A\left(i\right)$ is a measure which indicates that horizontal point $x_i$ is a success ($A\left(i\right)=1$) or not ($A\left(i\right)=0$). When the horizontal coordinate $x_i$ is ordered, the probability of success $P_s\left(i\right)$ at the horizontal point $i$ is estimated using Equation (1).

$$P_s\left(i\right)=\frac{1}{n}{\displaystyle \sum }_{j=i-\frac{\left(n-1\right)}{2}}^{j=i+\frac{\left(n-1\right)}{2}}A\left(j\right)$$

(1)

Equation (1): Probability of success (getting the certificate) at point $i$.

Figure 6B plots the probability of Certificate vs. F1 with n = 11. Notice how the R² statistic improved significantly (R² = 0.77). The plot and the fit provide mathematical models to describe the student’s performance. Figure 6B shows that the probability of getting a certificate improves significantly when participation in the course increases, as measured by the variables that are part of F1. It is noteworthy that the plot of this figure has R² of only 0.334 (Figure 6A) and that by using the certificate-probability estimate, the R² increases to 0.774 with n = 11 (Figure 6B), which is a significant increase in model prediction capability. The sample size used to estimate the success probability was small, so tests were made with different values of n (Figure 6B–D). It was found that the behavior of the probability estimate was about the same, showing stability in the estimate. Naturally, the bigger the sample, the more precise the estimate will be. In Figure 7, we show the success probability estimator against F1 for the control group. We can appreciate that F1 can be used as a predictor of learning success for both groups. We also see that the steepness of the slope is higher in the experimental group (0.05 for experimental and 0.03 for control). This means that the probability of getting the certificate increases more rapidly in the experimental groups for students who engage in getting higher scores in the variables that are part of F1.

5. Discussion and Conclusions

Regarding the influence of gamification in the structure of the social network (RQ1), we observe substantially more connections in the experimental group and many nodes with a large number of connections. This suggests a redistribution in the flow of communication among students that widens and changes the patterns of participation as previously reported [32], but that in this study is found in a massive online course. We also see that the social network of the gamified group presents the structural characteristics such as low density and short average paths that are also common in networks that involve human social interactions [48]. Descriptive statistics of the structural properties of the networks then suggest that there are differences between the treatment and comparison groups. However, we can also observe that the proportion of nodes with at least one connection is still small for each group (39% for comparison and 59% for treatment) and that the number of students that complete the course is a very small proportion, as is common in MOOCs. This suggests that a substantial number of students do not participate.

The results of the PCA show that F1 includes most variables for both groups, suggesting that F1 is a good representation of students’ work on the platform and students’ position and influence in the network. Personal activity and structural centrality in the educational social-network are correlated, as also shown in previous studies about social networking sites [49]. We also note that the variability explained by the first component (F1) is higher for the experimental group. This suggests that the gamification features and the two variables included to measure it (points and achievements) contribute to explaining the students’ work, providing a better statistical description of the gamified group.

To analyze the relationship between gamification and course completion (RQ2), we investigated the difference in overall success probability (the probability of getting a certificate) in both groups and found that probability increased from 3.28% in the comparison group to 5.25% in the treatment group. We further investigated the relationship between gamification and course completion by building a mathematical model to describe completion as a function of the amount of work, characterized as a function of the factor scores of F1 obtained from the PCA. The results suggest that F1 is a good predictor of the probability of completing the course for both groups. A closer examination of the models highlights the differences among groups. For students who engaged heavily, the probability of getting a certificate increased to almost 70% in the experimental group, while it was around 35% in the control group. This suggests that, besides the effect of gamification in participation and the structure of the network, gamification may also impact on learning success. Previous studies point to correlations between social networking and learning success [29] and between gamification-driven social networking and learning success [38,50], which our study confirms in a large-scale course. We also show that F1, in addition to being a good representation of a student’s work and position in the network, is a reliable estimator of the probability of success. Future work can community detection to further analyze the relationships between learning communities and their impact on course completion [51].

Our study has several limitations. First, it is essential to note that the causal relationship between the effect of social networking and gamification in learning performance is not proven. It may be that if students carry out the activities that increase the scores in the variables mentioned above, they will improve their chances of getting a certificate. However, it may also be that students who are responsible and motivated will get their certificates even if they do little work, and they do all the activities just because they are asked to do them. Our experiment is quasi-experimental because students were preassigned to experimental conditions based on the date in which they enrolled. Randomized-control trials need to be set up to check the causality of the relationships. Nevertheless, our analysis does point out the questions to address in those experiments. Another limitation of our study relates to generalization. Since the setting is a course on Digital Skills for Teachers, participants may be more motivated to learn and apply any digital skill just by their choice. Gamification is the very subject of the study, but it is also another digital skill in which participants are just interested and participate actively. In addition, only two cohorts of students and two social networks are studied. We suggest performing similar experiments to analyze whether and under which conditions the results reported here about the effect on the structure of the social network and on completion rates are found in other educational settings.