Determining Factors in MOOCs Completion Rates: Application Test in Energy Sustainability Courses

Abstract

Massive Online Open Courses (MOOCs) are open educational activities that allow for distance learning and professional updating, although the academic community has questioned their effectiveness due to their low completion rates. This research analyzes which factors (personal, family, social, labor, and instructional design) are involved in the value expectations and engagement of the MOOCs and to what degree these affect the decision to enroll and the completion of the MOOC. To this end, in the context of 12 MOOCs on energy sustainability carried out between 2017 and 2018, 8737 participants were surveyed using two instruments designed according to theoretical constructs and expert judgment. The main results show that all the factors reviewed influence the decision to take a MOOC, although the “professional development” aspect has the most significant impact on participants who have graduated from technical and engineering careers. Additionally, this study emphasizes that the “instructional design” factor is decisive in the engagement of younger participants, showing that the conventional design of xMOOCs (Stanford Model) may be one of the most important reasons for the low completion rates of this type of course.

1. Introduction

Technological developments and the evolution of telecommunications have brought with them new possibilities for training, as is the case with the Massive Online Open Courses (MOOCs). In the scope given by open education, MOOCs are an alternative global practice that have made the way in which it is studied fairer [1]. However, there are also difficulties for terminal efficiency, with aspects related to several types of engagements, among which the emotional, social, and cognitive commitment stand out [2]. There are many factors involved in the engagement and completion rates of MOOCs, and it is certainly interesting to explore some of them in this framework. How can personal, family, social, instructional design, and labor factors influence expectations-values, engagement and completion rates?

Personal factors are certain fundamental elements in studying MOOCs. Several studies have identified determining characteristics for engagement, e.g., attitude takes a dominant role in understanding student development [3]. Additionally, language, culture and social engagement are motivators for learning through MOOCs [4], where courses are required to be dynamic and flexible to respond to participants’ motivations and goals [5,6], as well as to find value in the course content and the opportunities that can be provided to provide a relevant experience [7].

Familiar factors also have a relation to the commitment of MOOC participants. Several studies have developed instruments that consider the background and demographic data as substantial [8] and literature reviews that highlight the importance of building academic commitment and self-regulation to be effective [9,10]. Self-efficacy takes into account student characteristics to understand why some students do not complete these courses [11], where the theory of self-determination touches on issues and sections such as participants’ autonomy, competence, and affinity, and based on this, studies and analyzes the background of MOOC participants [12]. In this way, students who participate in MOOC programs usually have considerable autonomy in their training process, with self-regulated learning [13].

Social factors are another aspect that have been highlighted in the MOOCs. Discussion forums in MOOCs find innovative ways to convey safe and accurate information, which allows participants to maintain trust and interaction [14]. Analyzing social interactions has also allowed us to find supporting aspects for terminal efficiency by studying trajectories [15], as well as analyzing the patterns of involvement for MOOCs that are influences on decisions involving pedagogy, depending on the audience to be reached [16].

Instructional design factors are critical to the commitment and completion rates. Skills such as motivation for achievement and self-esteem, self-efficacy, effectiveness, design, and development are crucial elements to consider in the instructional design [17] and, since they are new learning models, studies should be conducted to learn in detail about the changes that students make [18]. Research has learned about the effects of instructional designs that can be useful for MOOCs’ providers and students in their efforts to develop strategies to increase completion rates of MOOCs [19], as well as to determine the duration and analyze group behaviors [20] in order to implement different ways to increase the percentage of effectiveness. In this regard, the use of “learning analytics” allows the collection, measurement, analysis, and reporting of various MOOCs’ learner data [21], and provides design data to have an attractive and interactive interface, problem/situation-based learning and complexity to promote self-efficacy and flexibility. Design is a critical factor in countering dropout [22], as well as integrating teaching strategies such as game-playing, social networking, and collaborative learning which become attractive to users [23,24,25].

Labor considerations are also essential elements for engagement in MOOCs. Research on Massive Online Open Courses finds that one way to keep participants involved in training activities is to offer a certificate of completion for a fee, increasing student engagement by 10–12% [26,27], teacher reputations [28], and the development of new skills that will be useful in their work or studies [29].

While the study of predictors of MOOC engagement has been extensively reviewed in the academic literature separately, the present study seeks to analyze the correlations between personal, occupational, social, family, and instructional design factors with completion rates. This will allow for a broader view of the degree to which each of these factors is affected, which would serve to better understand the nature of participants’ motivations, which must be taken into account for academic offerings and instructional designs.

The study of the factors that are determinants for identifying decisions to take and complete MOOCs is of great value for these training initiatives to promote commitment and to have better results in the learning they want to promote. This article presents the materials and methods in a set of MOOCs on the subject of energy sustainability, the results from the application of validated instruments, the discussion in the framework of providing data for the commitment and improvement of terminal efficiency and conclusions that are intended to be of value to designers, decision-makers, stakeholders and the community in general, who are interested in these new open-access training modalities.

2. Materials and Methods

The objective of this research is to identify the factors that influence the decision to take a MOOC (personal, family, social, instructional design and labor) and to analyze to what extent each of them is relevant in the completion rates, in order to understand which of them has the most significant impact on engagement. For this purpose, a quantitative design and exploratory-correlational research is carried out on two instruments (pre-test and post-test) that are applied to the participants of 12 MOOCs on energy sustainability, delivered by Tecnológico de Monterrey (Mexico) through the platforms MexicoX and edX between 2018 and 2019, with a total of 123,124 participants and a global completion rate of 13.715% (Table 1), a higher than average completion rate—between 5%–8% of these type of training activities [30,31,32,33,34].

Table 1. Participants in the Massive Open Online Courses (MOOCs) on Sustainable Energy 2018–2019.

MOOC	n (e)	n (f)	CR
Energy conservation	12,929	2019	15.616%
Distribution of electrical energy	5549	639	11.515%
Smart Grid: Electrical networks of the future	6608	821	12.424%
Smart Grid: Technical fundamentals	5498	743	13.514%
Electric power transmission	5961	1074	18.017%
Conventional, clean energy and its technology	18,693	2770	14.818%
Electric power: Concepts and principles	15,978	1807	11.309%
Energy: Past, present, and future	13,224	2106	15.925%
Carbon markets	6710	910	13.561%
Energy markets	10,255	846	8.249%
The new electricity industry in Mexico	8975	1224	13.637%
Energy reform and its opportunities	12,744	1928	15.128%
TOTAL	123,124	16,887	13.715%

n (e) = Number of enrollments; n (f) = Number of finished; C_R = Completion Rate.

Despite the fact that these courses were not part of free configuration recognition credits, that is, participants did not receive ECTS (European Credit Transfer System) credits for studying this program, Tecnológico de Monterrey carried out an important communication campaign to promote the study of these MOOCs. In addition, such high completion rates—as opposed to the average referred to ut supra—may be due to the fact that Mexico was at a time of energy reform, so many technical career graduates could be interested in professional updating.

The courses were prepared by a group of experts and professors from various disciplines (such as Economics, Ecology, Electrical and Electronic Engineering, Marketing, Education and Humanities), gathered at the “Binational Laboratory for the Intelligent Management of Energy Sustainability and Technological Training” an initiative created by the Tecnológico de Monterrey in conjunction with the National Commission for Science and Technology of Mexico (CONACYT) and the Ministry of Energy (SENER), whose purpose was to propose the development of renewable and clean energies in the face of a market, such as the Mexican one, in which the consumption of fossil fuels (oil, coal, natural gas, and liquid gas) prevails, with a population of more than 130 million inhabitants.

In this sense, this study takes on special relevance since it involves initiating an educational and awareness-raising campaign in e-learning formats to instruct the population on alternative energies, while at the same time, training technicians and engineers in new sustainable energy technologies, all in a country that generates around 250 billion kWh per year, with consumption of 9,249,746 PJ/year, of which only 52,000 MW come from renewable sources (geothermal, hydroelectric and nuclear) [35]. Thus, not only was the pedagogical design of the MOOCs and their success in completion rates vital to the object of these awareness processes (as can be seen in Table 1) but also to analyze the participants in terms of their interest and engagement factors for possible future applications of these strategies.

These courses followed the classic instructional design of the xMOOCs [36,37,38], as the content was pre-configured in tables of contents with explanatory (asynchronous) videos, discussion forums, self-assessment tests and spaces for submission of papers and co-evaluation. Likewise, the courses had a start and end date, among which the different topics that formed the syllabus of each training activity were opened.

2.1. Sample Preparation

The MOOCs were made available to the public between January 2018 and April 2019 and, despite being on global platforms such as MexicoX and edX and the content being in Spanish, 94.5% of the universe of participants came from Mexico. Of the total 123,124 people registered in the 12 MOOCs, 16,887 (13.715%) completed all educational activities. In this sense, the number of people who completed the courses [n (f)] will be taken as the universe of the present study, since it is understood as those who were able to access both the pre-test (before the beginning of the activities) and the post-test (sent automatically upon completion of the last activity of each course). However, due to privacy and data protection regulations in Mexico, none of the tests were mandatory, being completed by 9075 participants (pre-test) (53.739% of the universe) and 6029 (post-test) (35.702% of the universe).

Taking into account that some participants had completed more than one MOOC, a purge was done in the database based on the individualization of the sample by a single participant. This individualization was done through the e-mails registered in the system to access the course. In summary, the Effective Sample (ES) of the present study will be made up only of those individuals who have completed both tests, which gives a total of ES = 8737.

These data allow us to obtain a confidence value equal to 95% and a margin of error of ±5% for data analysis, understanding that students participated randomly in the tests. Concerning gender, 5814 participants were male, and 2923 were female. The ages of the Effective Sample (ES) ranged from 18–24 years (7.8%), 25–40 years (68.43%), 40–65 years (19.42%), and ≥ 65 years (4.35%), while the majority of participants had technical or university training (79.34%).

2.2. Instruments

The pre-test and post-test instruments were designed according to theoretical constructs and expert judgment, obtaining in all dimensions of analysis an average kappa cohen of k = 0.8, thus denoting a high level of agreement. Those dimensions that did not exceed the agreement of k = 0.75, and a Likert average of +3.0 (0–5) were not included (Table 2).

Table 2. Expert judgment on factors influencing the completion of MOOCs.

Factors	Likert Average (0–4)	k
Personal	3.5	0.80
Family	4.25	0.75
Academic and Training	4.85	0.90
Professional Improvements	4.5	0.90
Entrepreneurship	2.85	0.35
Work Opportunities	3	0.65
Labor Improvements	4.0	0.80
Social Concerns	3.75	0.80
Instructional Design	4.25	0.85

The pre-test instrument consisted of 36 questions, of which 13 were related to independent and identification variables, while 4 questions were related to the factors that drive the participant to enroll in the MOOC. The answers were simple selection on a Likert scale ranging from 4 (strongly agree) to 0 (strongly disagree). The questions were as follows:

• IC1-R16: I think this course will help to satisfy the training needs that led me to enroll in it. [Personal/Academic and Training].
• IC1-R17: I believe that this course will help improve my professional development (current or future). [Professional Improvements].
• IC1-R20: I think this course will improve my academic training. [Academic and Training].

An Exploratory Factorial Analysis (EFA) was performed to generate theoretical constructs that measure on a scale of 1 to 4 (4 being the maximum), the level according to a series of statements. The three dimensions of the first test (pre-test) were conclusive: variance = 66.83%, KMO = 0.930 and Bartlett’s sphericity test: [X2(190) = 63,854.763, p ≤ 0.0001]. Cronbach’s alpha was all good, above 0.84.

The post-test instrument, which was completed only by those participants who completed each MOOC, included 32 questions, of which 4 were for independent variables and identification, and another 4 were for the dimensions under study. Of these, 3 were answered on a Likert scale (4 = very much in agreement/0 = very much in disagreement), and the last one was answered by simple selection from a series of options that will be explained later. The questions were the following:

• FC1-R05: This course fulfilled the training needs that led me to enroll in it. [Personal/Academic and Training].
• FC1-R06: After having taken the course, I am convinced that this course will help improve my professional development. [Professional Improvements].
• FC1-R09: I think this course improved my academic background. [Academic and Training].
• FC1-R12: To which factors do I ascribe the successful completion of this course (in order of highest to lowest importance, with 1 being the most critical factor and 5 the least important factor)? [All factors].

  ○

  Response options: Personal factors, family factors, social factors, instructional design factors, and labor factors.

3. Results

Personal, Academic and Professional Expectations-Values

In relation to the dimensions reviewed in the pre-test [IC1-R16, IC1-R17 and IC1-R20], regarding personal, academic, and professional predictors, there were almost no significant differences between the dimensions. In this sense, the expectation-value of the factor “training needs” [IC1-R16], “professional development” [IC1-R17], and “academic training” [IC1-R20] obtain together about 99.686% of the participants’ agreement (Figure 1).

Figure 1. Predictive factors by expectation-value of personal, academic and professional interests for enrolling in a MOOC.

Even though there were no significant differences or findings regarding gender and age, in the independent variable “educational level” (maximum level of studies completed) the factor “professional development” [IC1-R17] was identified as having a higher preponderance in graduates of technical careers and bachelor or engineering degrees than in lower (secondary) or higher (master and doctorate) levels. This means that the most influential factor in the decision to pursue a MOOC for technical and engineering graduates is to enhance professional development (current or future).

Concerning FC1-R05, corresponding to whether the course fulfilled the training needs that led him to enroll in it (Personal/Academic and Training factors), closely linked to IC1-R16 (expectation-value), 91.322% of the participants agreed between very much (4) and agree (3) that the course met their expectations. At this point, it is essential to note that there is no significant difference by gender or age. However, there is by the variable “educational level”, since again, graduates of technical careers and engineering degrees were those who concentrated the most extremely positive opinions (4) very much in agreement at 98.471%. This data also coincides very closely with that of question FC1-R09 (I believe that this course improved my academic training), in which 93.09% were between very much in agreement (4) and in agreement (3).

Regarding the factor of improvement of professional qualities [FC1-R06], although the data were positive, they did not reach the level of satisfaction of the previous dimensions, especially if we discriminate by the co-variable “occupation” (profession), relativizing the data by the statistical differences on each cluster. This dimension obtained a global score of 82.441%, adding the opinions of agreement (3) and very much agreement (4), but it obtained very different results between the different occupations (Figure 2).

Figure 2. Participants’ opinions about the usefulness they believe the content learned at the MOOC have in improving their professional skills [FC1-R06].

As for the preponderant completion factors [FC1-R12], it was observed that personal values (M = 20.9; SD = 1.29) were the main ones, followed by labor factors (M = 2.72; SD = 1.43). Social factors were considered the least relevant (M = 3.74; WD = 1.29). It is important to note that in these variables, the lower values mean greater importance of the factor in question. The Student T-test for independent samples showed that gender did not play a significant role in any of these factors (Table 3).

Table 3. Factors for the completion of the course, discriminated by gender.

Factor	Gender	n	Mean	SD	SEM
Personal	M	2167	2.09	1.308	0.028
	F	1067	2.07	1.258	0.039
Familiar	M	2221	3.32	1.259	0.027
	F	1101	3.34	1.270	0.038
Social	M	2369	3.71	1.298	0.027
	F	1162	3.80	1.279	0.038
Instructional Design	M	2334	2.87	1.242	0.026
	F	1152	2.84	1.220	0.036
Labor	M	2631	2.74	1.436	0.028
	F	1289	2.69	1.430	0.040

S_D = Std. Deviation; S_EM = Std. Error Mean.

A Levene’s Test for Equality of Variances and a t-test for Equality of Means were also applied to the factors in order to assess the equality of variances for a variable calculated for two or more groups (Table 4).

Table 4. Statistical tests for equality of variances and means.

		Levene’s Test for Equality of Variances		T-Test for Equality of Means
		F	Sig.	t	df	Sig. (2-tailed)
Personal	EVA	4.403	0.036	0.387	3232	0.699
	EVNA			0.393	2197.224	0.695
Familiar	EVA	0.073	0.787	−0.285	3320	0.776
	EVNA			−0.284	2177.355	0.777
Social	EVA	3.154	0.076	−1.936	3529	0.053
	EVNA			−1.945	2337.126	0.052
Instructional Design	EVA	1.578	0.209	0.624	3484	0.533
	EVNA			0.628	2328.886	0.530
Labor	EVA	0.044	0.834	1.018	3918	0.309
	EVNA			1.019	2567.240	0.308

EV_A = Equal Variances Assumed; EV_NA = Equal Variances Not Assumed.

Age, on the other hand, was not significant for personal factors (A) [r(3079) = −0.017, p > 0.05], but it was significant for all other factors, although with generally low effects. The correlation is direct for family factors (B) [r(3160) = 0.075, p < 0.001] and social factors (C) [r(3358) = 0.081, p < 0.001]; and indirect for instructional design factors (D) [r(3323) = −0.151, p < 0.001] and labor factors (E) [r(3729) = −0.050, p < 0.01] (Figure 3).

Figure 3. Correlation of participant age with (A) personal factors, (B) family factors, (C) social factors, (D) instructional design factors, and (E) occupational factors. X axis = age; Y axis = significance of the factor.

4. Discussion

In response to the general objective of this research, it was found that personal factors [3,4], family factors [8,9,10], social factors [14], instructional design factors [17,18] and labor factors [29] influence both the decision to take a MOOC (expectation-value) and, to a certain extent, engagement in the training program—measured by completion rates—as noted in the review of the state of the art. However, not all factors do so in the same way.

Firstly, with “training needs“ [IC1-R16], “professional development“ [IC1-R17], and “academic training“ [IC1-R20] obtained in the pre-test, together, about 99.686% of the participants agreed (see Figure 1). This coincides with the findings of Loizzo et al. (2017) [5], Littlejohn et al. (2016) [6], Sun et al. (2019) [12], and Tang and Chaw (2019) [19], although it is the “professional development“ aspect [IC1-R17] that was more prevalent among technical and engineering graduates than at lower (secondary) or higher (master’s and doctoral) levels. These results may well be due to the eminently technical nature of the MOOCs offered, but it is also an indicator that the entry profile of technical career students are eager for continuing education, especially in sustainability and renewable energy issues.

As for the personal/academic factor [FC1-R05 and FC1-R09], it is precisely the same participants from technical careers (engineers, industrial technologists, etc.), who concentrated more extreme agreement (4) (very much agreement) that the course met their educational needs and expectations. However, although 82.441% of the sample responded that the course provided them with improved professional skills [FC1-R06] (Figure 2), the level of agreement is much lower, which may indicate that, while MOOCs play a role in professional development, maybe they did not effectively provide participants with new skills or competencies that can be applied in the workplace. These results may be in line with those explained by Deng, Benckendorff and Gannaway (2020) [2] and with Romero-Rodríguez, Ramírez-Montoya and Valenzuela González (2019) [32], in the sense that MOOCs present difficulties for terminal efficiency, especially with aspects related to various types of commitments, including emotional, social and cognitive.

Finally, despite the fact that age as an independent variable was not significant in any of the factors in studies, there is a trend directly proportional to age and family and social factors, while indirectly proportional to instructional design and labor factors. Although the former may be due to the age-related nature of the priorities (for example, the older the age, the less need for professional improvement), the latter does represent the importance of instructional design to achieve engagement, in relation to the digital competencies of the new generations (Romero-Rodríguez, Ramírez-Montoya and Valenzuela-González, 2020) [34], since the conventional design of xMOOCs may go against the need for interactivity and educational resources—such as gamification—that make training activities more friendly [32].

Although the instructional design factor does not respond directly to the independent or demographic variables of the participants, but to the pedagogical construction of the course not belonging to the same category, it is included in the present study as a modifying co-variable. In previous studies [32,34], it has been shown that instructional design (xMOOCs, cMOOCs, tMOOCs) and the inclusion of pedagogical innovations (gamification, simulations, laboratories) [23,32] in MOOCs can increase engagement and, consequently, course completion rates.

As the main limitation, it is explained that the MOOCs in which this research is developed are of an eminently technical nature, mainly linked to the generation and transmission of renewable energies (see Table 1), so the results may be biased to some extent due to the fact that the profile of the students in these courses was mainly technical personnel or engineering graduates. On the other hand, another limitation that occurred in the development of this research is that most of the universe of participants in the MOOCs (94.5%) were from Mexico. Even though these two limitations do not imply a scientific demerit of the findings, they do open a set of research prospects to expand the results and discussions of this study.

ICTs have undoubtedly revolutionized communication, training and work processes [39,40], so today’s universities need to establish appropriate conditions to encourage more student-centered learning, using innovative teaching methods, critical training and active citizens who are willing to provide their knowledge for social service [41]. There is also a need to implement innovative methodologies using ICTs and teaching strategies that expand training scenarios and interaction between students and teachers [42], facilitating access to content from multiple perspectives and encouraging flexible and enriched learning [43].