Could MOOC-Takers’ Behavior Discuss the Meaning of Success-Dropout Rate? Players, Auditors, and Spectators in a Geographical Analysis Course about Natural Risks

Abstract

Research interest in massive online and open courses (MOOCs) is rapidly growing, questioning who enrolls, why and how to conceive engagement, and success rates. This study is focused on MOOC-takers behavior obtained from a seven-week MOOC experience on natural risks. Data scraping principles have been used to collect data. Demographics, success-dropout rates, engagement periods, achievement and scoring, and behavior were analyzed through descriptive statistics, non-parametric correlation analysis, and statistical hypothesis testing. The results show that students who start earlier and those who finish earlier the course obtain better grades in some of the modules (motivation and background on natural risks could be the explanation). However, for ‘last moment students’, speed in passing the modules is either related to greater motivation, although in this case it is not related to better grades. Furthermore, students who complete tasks during the weekend take less time to complete the modules and obtain a better grade. In addition, a learning strategy is promoted by reconsidering who is learning: players (those who complete the course and earning a certificate), auditors (those who have completed a thematic unit or the whole module, earning partial knowledge), and spectators (those enrolled until the end of the course, who intend earning experience in e-learning).

1. Introduction

Massive online and open courses (hereinafter, MOOCs) are an example of the gradual transformation in university education [1]. MOOCs are generally constructed and coordinated by universities and provided through commercial platforms. According to the Class Central portal—an aggregator of the MOOCs available on the main global platforms—the number of students enrolled in MOOCs continues to increase, exceeding more than 110 million students in 2019, and more than 900 universities have at least 1 of the 13,500 courses that are counted worldwide. About 30 platforms are offering MOOCs, of which Coursera and EdX stand out as the main platforms on content diversity and number of courses offered in English. MiriadaX is the main MOOC platform in Spanish, and Spain is, since 2013, the first European country producing MOOCs, with about 30% of the total offered courses [2].

The rise of MOOCs has also been considered an example of the democratization of access to education in two ways [3]. On the one hand, a debate about the validity of the traditional educational model and its ability to attract a greater number of students with diverse academic and professional interests has started [4]. On the other hand, the design, development, and delivery of a MOOC are an additional effort for teachers, changing their role [5]. According to this last statement, the teacher loses part of his function as a face-to-face transmitter of information to become a deferred actor focused on providing student orientation and support more than content, tools, and evaluation [6]. This affects the instructional quality of the MOOCs, which tend to be scored highly on the organization and presentation of the course material, while they fail at designing supported interaction and feedback, a key principle of effective instructional design [7]. That is, teachers are experts in a specific discipline or skills area rather than in pedagogy [8], so the role of instructional designers is shortening the distance between the student and the instructor, ensuring that students remain active throughout the course and self-regulation is running [9]. MOOCs therefore need teachers who are both instructors and assistants, able to automate the content and optimize resources, without forgetting that the student is the one who must filter, interpret, and enrich their learning process [10]. Furthermore, students need to self-regulate their learning more than in other modes of education, so MOOCs provide an excellent venue to promote autonomous learning [11]. Lastly, and taking into account situations of confinement or social distancing arising from pandemics such as Covid-19, MOOCs become a safe tool for advancing education [12].

Acceptance could be considered as a combination of technological (such as digitalization and e-learning) and pedagogical and social (such as students’ motivation and teachers’ use) recognition [13]. Several factors explain the success and acceptance of MOOCs: Its adaptation to different formative degrees of the audience; its academic and professional profile but also its ability to generate public interest; its ease and flexible use; and its gratuity [14,15]. However, the spread of MOOCs is not without limitations and criticisms. For students, the lack of interaction with teachers, the intensity of learning—concentrated in a few weeks and according to certain delivery times—or the required background are latent barriers [16,17,18]. For teachers, preparing course material for diverse and unknown audiences, promoting peer-to-peer evaluation, maintaining a minimum degree of interaction with students through forums, as well as evaluating non-face-to-face students’ dedication and managing the student retention or dropout rate—around 90%—are the main challenges to face [19]. The last one—dropout rate—is caused by the ineffectiveness of MOOCs in responding to learners needs [20] and this calls for a new perspective in which completing courses is not the only way of benefitting from participation in a MOOC [21].

What do we (un)know about success and dropout rates? Despite its popularity, the number of MOOC-takers who actually completed the course after enrolment was reported to be very low with high dropout rates (dropout equals not receiving a certificate). Success rate (considering only those students who earned a certificate for completion) in MOOC courses is less than 10% on an average (i.e., 5% on according to Breslow et al. [22]; 7% fixed by Rai and Chunrao [23]), although some more positive results—doubling this average—are fixed by Jordan [24]. However, it is worth noting that failure to complete the course does not mean that MOOCs are ineffective, because interest and motivation to learn could remain intact if students do not drop the course [25]. Students can simply audit the course or to have some feedback from high-quality universities and reputable instructors [26]. In fact, one of the main reasons for this kind of low completion rate is that many students consider that course completion is not necessary at all, because most students just want to access to specific knowledge from the whole course content [27]. Another reason that the course is not completed is the “minimum learning by doing” issue, by which online lessons are effective but they are only fit to provide theoretical education [28]. The lack of personal support and human intervention in favor of a complete machine-supported or auto-graded system of learning also explains the low completion rate [29]. In this case, students tend to be more motivated to learn when the instructor or facilitator is present within their proximity to reach a closed and systematic support system [30].

Each person experiences learning and motivation differently [31]. What if we changed the way in which we understand both success and dropout rates? Although research interest in MOOCs is rapidly growing, questioning who enrolls in and why [32], and how to analyze engagement and success rates is still discussed [33]. Although engagement is commonly described and measured as the investment of time, effort, and resources to achieve a goal [34,35], definitions about MOOC achievement or success vary. According to Liu et al. [36], success should not be limited to obtaining a certificate after completing all course lectures and exercises because this belittles the benefit of MOOCs as the opportunity for students to follow their own learning paths. This would allow learners to adopt their own strategy, such as following the course synchronously with the provided support, completing at their own rate, joining late and catching up, or just accessing materials that they were interested in [37]. Therefore, failure to finish a MOOC course is not a complete failure at all if students can learn a new topic or extend current knowledge, they present curiosity about MOOCs, and they face a personal challenge [38].

2. Background of the Research

Education is a critical driver in the context of disaster risk reduction because it conveys the essential fundamentals for risk-conscious and risk-mitigating actions among the population [39]. Preparing for disasters is a top priority for many educational and government institutions, who needs to increase students’ and societal motivation to learn about being less vulnerable to the impact of natural disasters [40]. Why an MOOC on natural risk analysis? MOOCs’ characteristics facilitate learning processes on dynamic issues across space and time, such as natural risks and hazards [41]. Increasing empirical evidence of climate change related to abnormal weather events, floods, and droughts could trigger an increase in risk perception of climate change and motivate changes in decision-making processes [42]. According to the latest annual report of the Swiss Re reinsurance multinational, in 2019, economic losses due to natural disasters were US 155 billion dollars, with more than 20 million people affected and 11,000 victims or missing persons. In this context, assessing the risk of potential hazards from climate change drivers on a community is vital to understanding how to be adequately prepared and adapted to climate change [43] as fundamental actions in climate change adaptation plans. Furthermore, teaching on natural risks, and educating and empowering local communities and stakeholders by increasing their sensitivity and adaptive capacity to natural risks is one of the main challenges to be addressed according to the Sustainable Development Goals 2030 Agenda, especially Goals n° 4 and 11 [44,45,46].

The aim of this paper is to deepen knowledge on how MOOC-takers’ behavior affects learning and success-dropout rates by improving understanding of how MOOC-takers manage their learnings and the relationship this has with achievement, success-dropout rates, and behavioral patterns. The study was designed to be exploratory (inductive research) with the purpose to deepen understanding of MOOC-takers’ behavior in order to develop some specific hypotheses or predictions about students’ patterns. However, we consider that a draft hypothesis of this research is that analyzing student behavior can help to refresh and discuss the concept of success and dropout rates, and this could provide new tools for learning designers when addressing students’ feedback and motivation. By addressing descriptive statistics, non-parametric correlation analysis, and statistical hypothesis testing, main patterns of how those enrolled in the course organize their time and learning experiences can be used to discuss the success-dropout rate meaning and the need to take into account different levels of learning that identify different types of MOOC-takers.

3. Materials and Methods

3.1. Sample Data

The empirical research presented here is supported by data obtained from a MOOC titled Geographical analysis of natural risk: Perceive, plan, and manage uncertainty. This course, with an estimated workload of 30 h, has been conceived as an opportunity to introduce to the off-campus audience the main content of the Master in Planning and Management of Natural Risks. Promoted in 2010 by the Interuniversity Institute of Geography in collaboration with the Faculty of Arts at the University of Alicante, this master is the first specific master in Spain on natural risks. The course has a double objective: (1) To conceptualize, from the geographical discipline, the physical, social, and territorial dimension of natural risk, and (2) to offer tools for analyzing natural risk and its management. The ultimate purpose of the course is to promote a global analysis of the conceptual, methodological, and perceptual aspects of natural risk to reduce vulnerability and promote the resilience of society and territory.

The number of students initially enrolled to the course was 1015, of which 988 (97.3%) remained until the end of the course (this difference in the number of students is due to the fact the registration system of the course was open until the end of the course, so once registered, students could decide to unsubscribe). In total, 215 (21.2%) completed the course (

Table 1. General information on massive online and open courses (MOOC) geographical analysis of natural risk: Perceive, plan, and manage uncertainty.

Variable	Content
Title of the course	Geographical analysis of natural risk: Perceive, plan, and manage
E-learning platform and social network profile	MiriadaX (www.miriadax.net); @MoocRiesgosUA
Institution	University of Alicante, Spain
Organization and production	Interuniversity Institute of Geography and Faculty of Arts
Date (1st edition)	9 September–27 October 2019
Length	7 weeks
Structure	6 content modules (6 quizzes, 30 thematic units in total) + presentation module + evaluation module (final quiz)
Estimated workload	4–5 h per week (30 h in total)
Scientific area	Geography (Social Sciences), Environmental Sciences
Level	Introductory
Prerequisites	None
Teachers	10 (2–3 for module)
Language of exposition, video subtitles and transcriptions	Spanish
Supplementary material	Spanish, English
Assessment	Module quizzes and forum discussions

). The course was divided into an introductory module and 6 thematic weekly modules. Each weekly module was divided into five units, each of them with a video lecture (of no more than 25), optional sources (text reading, case studies and examples, links, and databases), and a module quiz. Students wishing to receive a statement of accomplishment must have passed the module and final quizzes by scoring 50% or better on each quiz. Students could take module quizzes twice and they had three attempts to overcome the final quiz. The score of the record was the best score on each quiz. There was only one deadline, the final day of the course, to complete all quizzes.

3.2. Data Collection

Investigating students’ interaction with a course by way of learning analytics can provide useful information about their learning experience and behavior [47]. Data to explore students’ learning were collected from the MiriadaX platform data analytics services, and web scraping and data extraction from Google Chrome^© Data Miner extension were employed to analyze MOOC-takers’ effort, time-stamped logs of student activities such as lecture video views, and quiz pass [48].

3.3. Data Analysis

Overall results related to demographics, success–dropout rates, engagement periods, achievement and scoring, and behavior were analyzed through descriptive statistics, non-parametric correlation analysis (Spearman), and statistical hypothesis testing (Mann Whitney U test). Although the main relationships with completion rates investigated are related to educational background, gender, and geographic location [49], the available information in our case study only includes the score obtained in the questionnaires and the time taken to pass each module. All the same, there is not much evidence about issues related to student behavior [50]. Consequently, in order to deepen understanding of learner’s behavior, the sample of students was divided into several groups.

Firstly, data were analyzed from those students who completed at least one module quiz by gender (GENDER). Secondly, these data were grouped according to the types of learners, differentiating between students who completed the course and students who partially completed learning, or did not complete the course but had at least completed the first module (TYPE OF LEARNING). Thirdly, we wanted to check whether the day of the week in which the student completed each module influences the student’s performance, so the sample was divided between those who finished a module during midweek and those who finished at the weekend (WEEKDAY). In third and fourth place, the sample was divided according to the course start period (STARTING PERIOD) and the completion period (ENDING PERIOD). In the first case, differentiating between students who finished the first test during the first week and the rest of the students and, in the second case, between students who finished the last module before the last week and the rest of the students. Finally, the sample was divided between those students who completed more than half of the modules during the last week and the rest of the students (LAST MOMENT).

The characteristics of these groups are summarized in

Table 2. Descriptive statistics of the students who have completed at least one module quiz and learners’ groups.

Learners’ Groups		Module 1					Module 2					Module 3					Module 4					Module 5					Module 6					Module 7 1
		N	SA	SSD	TA	TSD	N	SA	SSD	TA	TSD	N	SA	SSD	TA	TSD	N	SA	SSD	TA	TSD	N	SA	SSD	TA	TSD	N	SA	SSD	TA	TSD	N	SA	SSD	TA	TSD
COURSE		346	77.3	16	5.1	8.4	306	79.4	15.9	2.6	5.1	281	76.1	16.4	2.3	4.5	260	77.1	16.8	2.8	5.3	247	81.9	15.8	1.1	2.7	226	82.2	17.4	1.4	3.7	221	80.8	15.4	23.3	15.9
GENDER	Men	215	78	15.1	4.6	7.6	194	79.4	15.4	2.3	4.8	181	76.3	16.1	2.3	4.5	172	77.3	16.2	2.9	5	163	81.5	15.8	1.3	3.1	148	82.9	17.2	1.5	4.2	145	81	14.3	21.8	15.8
	Women	120	76.1	16.8	5.9	9.4	104	79.5	16.5	3.2	5.6	95	75.4	16.5	2.2	4	84	76.4	17.7	2.7	6	80	82	16	0.9	1.8	76	81.3	17.7	1.2	2.5	74	80.2	17.4	25.8	16.1
TYPE OF LEARNING	Total	221	77.8	15.7	4.8	7.9	221	80.4	16.0	2.5	5.2	221	77.1	16.0	2.0	4.0	221	77.8	16.3	2.9	5.5	221	81.9	15.9	1.1	2.7	221	82.0	17.6	1.4	3.7	221	80.8	15.4	23.3	15.9
	Partial	125	76.5	16.5	5.8	9.4	85	77.1	15.3	3.0	5.0	60	72.5	17.3	3.4	6.0	39	72.3	19.5	2.7	4.0	26	81.2	15.6	1.2	2.8	6	88.3	11.7	1.8	3.6	-	-	-	-	-
WEEKDAY	Midweek	268	77.4	15.8	4.7	7.8	222	78.2	15.9	2.9	5.4	206	75.8	15.8	2.6	4.9	174	76.8	16.9	3.1	5.5	151	82.1	15.3	1.5	3.3	137	80.7	17.7	1.8	4.3	121	80.0	14.8	21.6	14.3
	Weekend	78	77.3	16.7	6.7	10.3	84	82.6	15.4	2.1	4.5	75	77.2	17.9	1.6	3.0	86	77.7	16.5	2.4	5.0	96	81.5	16.7	0.7	1.4	89	84.6	16.9	0.9	2.4	100	81.7	16.1	25.4	17.5
STARTING PERIOD	1st week	114	80.0	15.5	1.2	1.5	105	81.9	15.8	2.4	4.7	99	76.6	16.9	2.4	5.0	89	79.4	15.3	3.3	6.8	87	83.2	16.1	1.4	3.6	78	83.5	16.9	2.3	5.3	76	82.7	13.5	25.1	16.6
	>First week	232	76.0	16.1	7.0	9.7	201	78.2	15.8	2.8	5.4	182	75.9	16.1	2.2	4.2	171	75.8	17.4	2.6	4.3	160	81.1	15.7	1.0	2.1	148	81.6	17.8	1.0	2.4	145	79.8	16.3	22.3	15.5
ENDING PERIOD	<Last week	109	77.7	16.6	2.6	5.0	109	81.7	16.1	1.3	3.4	109	78.0	16.0	1.3	3.4	109	80.6	15.3	1.5	2.9	109	83.8	15.7	1.0	2.1	109	82.2	17.6	1.1	2.4	109	82.6	14.8	15.7	12.4
	Last week	112	77.9	14.9	6.9	9.5	112	79.0	15.8	3.7	6.3	112	76.3	16.0	2.6	4.4	112	75.1	16.8	4.2	7.0	112	80.2	15.9	1.3	3.3	112	81.9	17.7	1.8	4.6	112	79.1	15.6	30.7	15.5
LAST MOMENT	≥ 4 modules	151	77.5	15.5	3.5	5.3	151	80.9	15.6	1.8	3.8	151	77.7	15.4	1.9	3.6	151	77.4	15.9	2.7	4.7	151	82.6	15.7	1.4	3.2	151	81.7	17.4	1.9	4.4	151	81.4	15.4	22.1	15.5
	Rest	70	78.6	16.2	7.6	11.2	70	79.3	16.9	4.0	7.2	70	76.0	17.2	2.2	4.7	70	78.6	17.1	3.3	7.0	70	80.6	16.4	0.6	0.9	70	82.7	18.0	0.4	0.7	70	79.6	15.4	25.8	16.7

¹ Module 7 specifies the total time spent to complete the course.

; specifically, the number of learners (N) and the average value and the standard deviation of the score obtained in the questionnaires (S_A and S_SD) and the time length to complete each module and its standard deviation (T_A and T_SD), measured in days. The non-parametric Mann Whitney U test was applied to check if there were statistically significant differences between groups in the score obtained in the questionnaires or the time spent in each module. Due to the fact that the sample size of some groups was very small in some modules and thus differences were more difficult to detect, Mann Whitney U test results were analyzed at a confidence level of 90%, without losing sight that p-value <0.05 indicate more robust results. The dataset was analyzed with the R software.

4. Results

4.1. Demographics

Information at aggregate level was available on nationality for 97.4% of enrollees, on gender for 94.2%, and for the rest of sociodemographic variables (age and education level) less than one-third of those enrolled. Regarding the origin of the students, the course has had representation from 40 countries, especially from Latin America (62.4%), which includes 20 countries from South America, Central America, and the Caribbean, and, secondly, Europe (34%), which was the country with most enrollees, Spain (30.4%), followed by Peru (18.8%), Colombia (9.4%), Mexico (8.6%), and Ecuador (6%).

Data related to gender indicate that 64% of those enrolled were men and 36% women (far from results obtained by Ruiz-Palmero et al. [51], in which the number of men and women was practically identical). These results are in line with those of other studies that indicate the masculinization of MOOC courses [52]. Regarding age, the most represented age groups were, firstly, students between 25 and 34 years old (30.2% of those who indicated age), followed by the group of 35–44 years old (25.3%), and thirdly, the group of students between 18 and 24 years old (21%). As in other studies, the most representative age groups for MOOC courses were those between 25 and 44 years old [53]. With regard to education level, 96% of those who responded had some type of university’ experience, including those who completed their university studies (50.7%), those who were pursuing their university studies (27.8%), and those who were university teachers (17.5%).

4.2. Success-Dropout Rates

Just over half of the total enrolled, 55.6%, started the course, while less than half of those finished it (21.2%), doubling the average success rates usually obtained in MOOC courses (Figure 1).

The proportion of each type of student fluctuates because (1) the registration system is open during the course and (2) students’ behavior is changing weekly. The figure shows how while the number of students registered (the sum of the three categories) increased, the number and proportion of spectators decreased as the weeks passed because students achieve the goals of each module and moved from the spectators category to the auditors and players categories. In addition, the lack of registration deadlines stimulates news enrolments during the passage of the course. After an increase in the absolute number of enrolled of 9.3% in the first week, the increase stabilized going from 4% in the second week to 2.6% in the third and between 1.5% and 1% for the rest. In the last week, however, there was a decrease of 2.2% for the enrolled students, who decided to leave the course. Regarding the students who started the course, the greatest increases occurred in the first three weeks and were subsequently reduced to increments of 9.7% in the fourth week and between 6% and 4.8% in the following weeks. It is worth noting that, in relation to the number of students who finished the course, a rebound occurred during the last week, in which half of the total students completed the course.

Despite the fact that the course was started by 55.6% of the enrolled, only 34.7% (352 students) performed the first module quiz. That is to say, it was during the first module when there was a higher dropout rate, understood as the proportion of students who start a module but do not finish it. This proportion decreased sharply in the second module and remained more or less stable as the course progressed (Figure 2).

In total, 21.4% of enrollees dropped out of the course after having started a module, with two-thirds of those dropping out during the first module. It is also necessary to take into account the dropouts that occurred from students after passing a module quiz, which in total represent another 4.4% of those enrolled, with more than half occurring before the third module.

4.3. Engagement, Achievement, and Scoring

4.3.1. Engagement Periods

Concerning the engagement temporal distribution by weeks and days, if we attend to the moment in which the participant begins the activity and passes at least one module, engagement was concentrated during the first three weeks. Regarding the moment in which the course ends, only 2.8% of those who completed the course did so during the first week. It is in the last week of the course when the greatest increase in the course completion rate occurred (115% compared to Week 6), since, as said, 50.7% of the total number of students completed the course. The temporal distribution of the visualization of the videos in each unit as well as the performance of the module quizzes differed between the first two modules, where the week of greatest activity was the first compared to that of the rest of the modules, since module 3, the week of greatest activity, was the last (Figure 3).

This indicates that most of the students do not meet the recommendation to follow a weekly calendar to pass each module. In this sense, it is worth noting that only one of the 215 students who passed the course completed each module in its corresponding week. This low-rate pattern confirms that students generally self-managed their involvement in the course. However, the number of students who followed a weekly calendar was a little higher in the rest of the modules, especially in the first module (116 students, 32.9%). Likewise, it was observed that during the first modules (modules 1–3), there was a greater activity during midweek, while this activity was gradually transferred to the weekend as the course progressed, especially in the last three modules. For the two first modules, the days with the highest involvement were Monday and Wednesday, whereas for the rest of the course was Sunday, both in the visualization of the videos of the units of each module and in the quiz performance (Figure 4).

4.3.2. Achievement by Unit Interest and Time Length

The introductory videos of each unit were viewed by 98.3% of the students who started each unit. However, in the first module, this rate was reduced to 92.8%, especially in the video of the first unit, which was the one with the highest dropout rate of the whole course (14.4% of the students who started the video of this unit watched less than half of the video). In the rest of the modules, practically 100% of those who started a thematic unit watched at least 50% of the video. In this regard, it should be borne in mind that the video display for each thematic unit can be completed in several phases, which is reflected in the average number of accesses. The average number of accesses for the whole course was 2.03 times per video, although this decreased from the first module, which had the highest average with 2.8 attempts, to 1.7 in modules 5 and 6. In this regard, it has been verified that there is no correlation between the length of the videos and the average number of views, but there is between the percentage of dropouts and the average number of views (Spearman rho = 0.55, p-value = 0.001).

Regarding the time length to complete each module, measured in days from the start of the first unit until the completion of the module quiz, there were significant differences for a 95% confidence interval in the time spent to pass each module in aggregate terms after performing the Mann Whitney U test (

Table 3. Mann Whitney U test results between time lengths to complete each module.

	M2	M3	M4	M5	M6	Average Time
M1	W = 67,234 p-value = 1 × 10−8	W = 60,912 p-value = 4 × 10−7	W = 53,138, p-value = 0.0005	W = 59,663, p-value = 2 × 10−15	W = 53,674 p-value = 1 × 10−12	5.1
M2	-	W = 41,300 p-value = 0.36	W = 35,560 p-value = 0.02	W = 41,614 p-value = 0.04	W = 37,023 p-value = 0.19	2.62
M3	-	-	W = 33,829 p-value = 0.10	W = 40,031 p-value = 0.003	W=35,859 p-value= 0.01	2.27
M4	-	-	-	W = 39,578 p-value = 8 × 10−6	W = 35,668 p-value = 0.0001	2.82
M5	-	-	-	-	W = 27,281 p-value = 0.61	1.15
M6	-	-	-	-	-	1.43

Note: ‘M’ refers to ‘module’. Because module 7 has no content, since it only consists of the final quiz, it has not been included.

). Specifically, these differences were established between module 1, which, on average, took the longest time to complete (5.1 days), and the rest of the modules, which ranged from 1.1 to 2.8 days. Likewise, statistically significant differences were established between the time length to complete module 2 and the modules 4 and 5, and between modules 3–4 and 5–6, which were those that were overcome in a shorter time interval. In summary, it is verified that the period elapsed to complete the modules decreases as the course progresses. However, the duration of the videos in modules 1 and 2 was less than in the rest of the modules (33^′ and 31^′ of duration for modules 1 and 2, respectively; and 66^′, 100^′ 37^′ and 53^′ of duration for modules 3, 4, 5, and 6, respectively).

4.3.3. Module Quizzes Score

The average score ranged from 76.1/100 points (module 3) to 82.2/100 points (module 6). Mann Whitney U tests were carried out to assess whether there were statistically significant differences between the score of the module’s quizzes (

Table 4. Mann Whitney U test results between modules quiz scores.

	M2	M3	M4	M5	M6	M7	Average Score
M1	W = 49,445 p-value = 0.10	W = 51,028 p-value = 0.34	W = 45,739 p-value = 0.82	W = 35,929 p-value = 0.0004	W = 32,241 p-value = 0.0001	W = 32,558 p-value = 0.001	77.3
M2	-	W = 48,113 p-value = 0.01	W = 43,286 p-value = 0.09	W = 34,618 p-value = 0.06	W = 30,896 p-value = 0.02	W = 31,408 p-value = 0.1206	79.4
M3	-	-	W = 35,729 p-value = 0.55	W = 27,963 p-value = 5 × 10−5	W=25,278 p-value = 3 × 10−5	W = 25,357 p-value= 0.0002	76.1
M4	-	-	-	W = 27,074 p-value = 0.001	W = 24,351 p-value = 0.0005	W = 24,696 p-value=0.004	77.1
M5	-	-	-	-	W=27,131 p-value=0.48	W = 27,881 p-value = 0.80	81.9
M6	-	-	-	-	-	W = 26,482 p-value = 0.34	82.2
M7	-	-	-	-	-	-	80.8

Note: ‘M’ refers to ‘module’.

).

It was found that there are significant differences (p < 0.05) between two groups of modules. On the one hand, the initial modules up to the fourth presented significantly lower average marks (77.3, 79.4 and 76.1 and 77.1, respectively) and on the other, the modules 5, 6, and 7, whose mean scores were 81.9, 82.2, and 80.8, respectively.

Finally, it was checked whether there exists a relationship between the time length to complete each module and the whole course, and the score obtained in each module quiz. The results indicated that there was a slight negative correlation between the score obtained and the time length in module 4 (Spearman rho = −0.19; p-value = 0.002) and module 2 (Spearman rho = −0.15; p-value = 0.006). Although the correlation was not very intense, the sign of the correlation was striking; that is, the less time it took to overcome these modules, the better the score obtained was.

4.4. Behaviour

Non-parametric Mann Whitney U tests were applied to each group to determine whether there were significant differences in the quiz score and the time spent to pass each module (

Table 5. Mann Whitney U test results between learner’s groups and modules (p-value).

Learners’ Groups		M1	M2	M3	M4	M5	M6	M7
GENDER	Quiz score	0.24	0.91	0.53	0.71	0.77	0.57	0.99
	Time length	0.47	0.46	0.9	0.52	0.99	0.76	0.08 *
TYPE OF LEARNING	Quiz score	0.45	0.08 *	0.05 *	0.12	0.74	0.51	-
	Time length	0.79	0.09 *	0.29	0.64	0.32	0.65	-
WEEKDAY	Quiz score	0.93	0.03 **	0.47	0.7	0.92	0.07*	0.3
	Time length	0.42	0.09 *	0.02 **	0.03 **	0.02 **	<0.01 ***	0.06 *
BEGINNING PERIOD	Quiz score	0.02 **	0.04 **	0.89	0.11	0.26	0.5	0.27
	Time length	<0.01 ***	0.59	0.55	0.5	0.39	<0.01 ***	0.16
ENDING PERIOD	Quiz score	0.97	0.18	0.45	0.01 **	0.07 *	0.9	0.09 *
	Time length	<0.01 ***	<0.01 ***	0.05 *	<0.01 ***	0.1	0.44	<0.01 ***
LAST MOMENT	Quiz score	0.64	0.55	0.56	0.61	0.42	0.58	0.39
	Time length	0.06 *	0.29	0.44	0.54	0.17	<0.01 ***	0.09 *

Note: * p < 0.1 ** p < 0.05 *** p < 0.01.

). Regarding GENDER, there were no significant differences in module quizzes scores, but in relation to time spent completing each module, there were gender differences in the time length to complete the whole course at a 90% confidence level. Women who completed the course took 25.8 days compared to 21.8 days for men. This may be explained because 63.5% of the women who finished the course did so in the last week, while in men this proportion was 44.1%.

In the TYPE OF LEARNING group, significant differences were found at a 90% confidence level in the quiz score of module 2 (p-value = 0.08) and module 3 (p-value = 0.05). In both cases, average scores were higher in students who completed the course, which occurred in all the modules except in module 6, where the sample of partial learners was very small (n = 6). During the three first modules, where the dropout rates were concentrated, it stood out that those students that obtained partial learning took longer to complete modules. In this respect, there were statistically significant differences in the time length to complete module 2 (p-value = 0.09), which was higher in students who received partial learning.

With regard to the work schedule (WEEKDAY), it was identified that those who had the habit of working during the weekends obtained a better score than those students who had the habit of working during the week. This hypothesis was confirmed in module 2 at a 95% confidence level (p-value = 0.03), where students who performed the quiz during the weekend obtained an average score of 82, while midweek students obtained 78.2. Likewise, significant differences were identified in module 6 at a 90% confidence level (p-value = 0.07), where the midweek students obtained an average score of 80.7, while the weekend students obtained 84.6. However, the greatest differences were established in relation to the time elapsed between the beginning and the end of each module. These differences were identified in all modules except the first, indicating that weekend students took less time to complete the modules than midweek students do. This premise was not fulfilled in module 7, since many midweek students performed the final quiz on the last days of the course, which were at the weekend.

Significant differences have also been identified in the BEGINNING PERIOD group. With respect to module quiz scores, there were differences in module 1 (p-value = 0.02) and module 2 (p-value = 0.04), since students who began during the first week obtained a higher score. Likewise, there existed differences in the time length in module 1 (p-value < 0.01) and module 6 (p-value < 0.01) but with a different meaning. For those students who started and complete module 1 in the first week, the length of time was much less than for the others (1.2 days versus 7 days). However, at the end of the course, the opposite happened, as less time elapsed in finishing module 6 for students who started later, at 1 day versus 2.4 days for students who started the first week. Even though the average values for total time length for students who started earlier was higher than for the rest of students (25.1 days versus 22.3 days), at the beginning of the seventh week, 72.4% of those who started early had finished the course, while in the other group, only 37.3% of the students had concluded the course.

The two last groups only include students who completed the course. As could have been deduced beforehand in the ENDING PERIOD learner’s group, there existed significant differences in the time length of the first four modules and in the whole course. For this group, there were more remarkable differences established in the score of module 4 (p-value = 0.01), module 5 (p-value = 0.07), and module 7 (p-value = 0.09). In all these cases, students who finished the last module before the last week showed better scores in modules quizzes compared with the rest of the students, which may be related to a greater interest in the course content. Finally, the results indicate that there only existed differences in the time length of module 1, 6, and the whole course for the LAST MOMENT learner’s group. Results show that students who completed more than half of the modules during the last week completed the course content quicker, as expected. However, it was also expected that this quickness could have an impact on the test scores, but this hypothesis was not confirmed.

5. Discussion and Conclusions

Success rate could comprise any individual goal achieved during the course; that is, MOOC-takers could intend to complete only some specific lessons or modules, following the whole course without doing exercises or taking the quizzes, or getting a certificate [54]. According to this approach, any specific skill and useful knowledge achieved during the course should be considered a way of success [55]. Although our obtained results highlighted a success-dropout rate of 21.2% (doubling the average success rates usually obtained in MOOC courses), learning rate could be increased to 34.7% if considering learners as those students who passed the quiz of module 1, for example, and not only those who complete the whole course. The consideration of who are and who are not considered as learner and which type of analysis could be applied to deepen on this big question is a challenge for e-learning. This study extends previous work only focused on demographics and success–dropout rates and contributes to the field by interrogating the behavior of students from engagement, achievement, and scoring, while considering different types of learning processes and MOOC-takers’ attitudes and patterns that go beyond simply looking at those who completed or failed a course. This leaves a door open to further research on learners’ experiences and how this experience can motivate a reinterpretation of the success–dropout rates. Accordingly, discussion about the success–dropout rate mining could be understood, compared to the traditional learning system, like measuring the student’ knowledge achieved in a partial exam or during an exercise. In fact, the educational research community offers a diversity of interpretations of what constitutes a learning strategy. According to Jovanovic et al. [56], a learning strategy includes any thoughts, behaviors, beliefs, or emotions that facilitate the acquisition, understanding, or later transfer of new knowledge and skills that have to be mined/detected using appropriate analytical methods and techniques. Essentially, e-learning analytics leverages digital trace data from online systems by which feedback is provided when a lecture video is watched, a quiz is completed, or a task is posted on the forum discussion board [57].

This study lays the groundwork for future research into behavioral modelling and mapping within MOOC learning environment [58]. Despite that this study is exploratory and advantaged by access to detailed data for those students who have completed the course, some limitations on data collection have to be considered. For example, only partial information has been obtained from those students who have been registered in the course but have not been able to achieve the course goals. In addition, data on forum participation or student’s motivation to register for the course have been unavailable. This lack of detailed and full information has not allowed establishing a greater degree of accuracy on students’ behavior patterns [59]. Furthermore, available data come from a single edition of the course, so additional research incorporating multiple editions of the same course would enhance the generalizability of our conclusions. Notwithstanding these limitations, obtained results highlighted some issues regarding different levels of learning and, accordingly, the diversity of learner’ profiles. In this sense, a learning strategy could be promoted by reconsidering how and to what extent MOOC-takers are learning. For example, players could be those students who complete the course (earning a certificate), auditors could be those who completed a thematic unit, the whole module, or some of them (earning partial and infrequently knowledge), and spectators could be those enrolled until the end of the course without taking any action (although they could intend on earning experience in e-learning). Undoubtedly, the auditors’ category was the most diversified because it included different students’ profiles, but also the most interesting because it allows evaluating and reinforcing the interest of the student in a specific issue [60]. For example, a detailed analysis of how success is achieved in this MOOC confirms how the ability to build an argument-based approach focused on natural risks conceptual analysis during the first week of the course was key in reducing the dropout rate and increasing auditors and players [61]. However, the difficulty of getting the student involved in this first module is reflected in a greater number of attempts to watch the videos of each thematic unit and may be the reason that a higher dropout rate is experienced.

Furthermore, self-regulated learning (that is, the weekly period in which thematic units, modules, and quizzes are achieved) could affect the success-dropout rate but, above all, the learning rate [62]. According to the obtained results, only during the first week (in which the first two modules are passed by most of those enrolled) did the activity occur mostly during the week, while the rest of the modules are carried out largely during the last week of the course or during the weekend. To address this gap, some authors rely on promoting self-regulated learning frames strategies that students can use to enhance motivation and persistence, while teachers can overcome the limitation of not being able to provide personalized course delivery and individual feedback [34,63,64]. Accordingly, more attention to the instructional design process and the role of instructors and designers is needed. For example, and according to our experience, weekly email delivery to refresh tasks and contents or forums’ comments reply have been found insufficient to ensure students’ self-regulation. This requires a rethinking by learning designers about which strategies could be promoted taking into account that not only learners’ motivation and confidence influence the iteration of the success-dropout rates, but also the structure of the course, the delivery environment, and the perceived value of learning itself [65]. According to the five principles of instruction for learning activities designed two decades ago by Merrill [66], learning is promoted when students are engaged in solving real-world problems; prior knowledge is activated; new knowledge is demonstrated to the student; the student applies new knowledge; and new knowledge is integrated into the student’s perceptions and experiences. Recently, Margaryan [7] has added a set of principles among which collaboration and feedback must be highlighted. For example, by including a starting quiz in which students could share the reason that they enrolled into the course; by promoting students’ group tasks through the forum; or by moving from current self-assessment (quizzes) to peer-assessment, enhancing the critical thinking, comprehension, and writing capabilities of students [67].

Regarding the analysis of students’ behavior, a further in-depth analysis requires that e-learning platforms facilitate access to data on the characteristics of the students (country of origin, age, and educational level). Furthermore, detailed information on students’ involvement and risk of abandonment should be provided to deepen on MOOC-takers’ behavior (by considering participation in the forum, number of attempts to view the videos, and the supplementary material) [68]. However, with the available data, some key issues related to student behavior have been identified:

• Students who obtain partial learning (auditors or learners) take longer to complete the modules and obtain worse grades. To address this gap, it should be useful to send questionnaires to those who do not finish the course, asking why (lack of interest in the course content, does not meet expectations, need for specific knowledge already satisfied, lack of time, the difficulty of the course, etc.).
• Students who complete tasks during the weekend take less time to complete the modules and obtain a better grade. This could be related to many factors, but it would be interesting to focus the communication strategies with the students to be promoted also during the weekend (beyond reminders or communication at the beginning of the modules that are provided each Monday).
• Students who start earlier and those who finish earlier obtain better grades in some of the modules (motivation could be the explanation, but also students’ background in the subject: Natural risks). However, ‘last moment students’ (those students who complete the course last week) demonstrated that speed in passing the modules is either related to greater motivation, although in this case it is not related to better grades.

Learning designers could check the behavioral rules to deepen on which learning strategy is more useful to optimize the performance of online courses. To this end, the meaning of success-dropout rates and the level of motivation of those enrolled in the course should be reconsidered [69]. Furthermore, in line with other recent studies focused on geography education [70], this study is also an opportunity to highlight how the study of natural risks from a geographical perspective can be benefited from e-learning by contributing to the Sendai Framework for Disaster Risk Reduction 2015-2030 [71].