Measuring the Post-Impact of Programming MOOCs: Development and Validation of an Instrument

Abstract

Massive open online courses (MOOCs) can potentially affect the lives of millions of people, but there is little research on how a programming MOOC could affect participants’ lives after participation. In Estonia, we have organised several programming MOOCs over the years, attended by over 14,000 people. This inspired us to develop and validate a scale for measuring the impact of programming MOOC on participants’ lives. We analysed data from 1261 MOOC participants who had completed at least one of our programming MOOCs. Confirmatory factor analysis (CFA) was used to validate the developed scale. The final model fitted quite well to the data. We found that the factors that influence learners’ lives after a MOOC include acquired learning skills, interest in continuing computer science related studies, learning experience gained from the MOOC, the MOOC’s impact on work and personal life, and new contacts that are established with other participants and organisers. We also compared the means of the factors. The highest-rated factor was related to the learning experience from the MOOC and the lowest-rated was related to finding new contacts through the MOOC.

1. Introduction

Over the years MOOCs have attracted millions of learners worldwide to help them learn something new or complement their existing knowledge. Consequently, MOOCs have the potential to impact the lives of millions of people and, thereby, society as a whole.

In Estonia, organisation of MOOCs is also becoming more and more popular at universities. Over the years, Estonian universities have offered a progressively growing range of different courses, including programming MOOCs. Notably, there is a great shortage of labour in the IT sector in Estonia as well as elsewhere in Europe [1] and universities can help to increase interest in IT through MOOCs, as the need for IT skills increases also in other fields. For this very purpose, our university has been offering programming MOOCs since 2014. Over the years, we have organised three different courses, which have been attended by more than 14,000 people (approximately 1.2% of the population of Estonia) with a completion rate of 30–60% [2,3]. The increasing need for IT skills can affect the Estonian labour market and have an impact on the lives of the people who participated. This motivated us to conduct a study that would provide an overview of the impact of programming MOOCs on participants’ lives, which requires first an instrument to help to achieve this goal.

2. Previous Studies on Measuring the Impact of MOOCs

The effects of MOOCs have been studied in different ways and from different time perspectives after the course. In this study we use both long- and short-term studies to describe the impact of MOOCs because there is a lack of long-term studies that have addressed this topic. A long-term study in this paper is a research that is carried out at least three months after the end of the course.

A characteristic feature of MOOCs is a large number of participants with different goals and intentions to participate. With every participant being a unique individual, there is a wide range of knowledge, attitudes, beliefs and experience. Several studies have been conducted on the impact of MOOCs on knowledge and skills in long-term (more than three months after the course) using tests, e.g., in medicine [4] and in programming [5]. There are also studies on the impact of medicine-related MOOCs on participants’ attitudes and beliefs in long-term [4] and shortly after the course [6]. Less is known about how participation in a MOOC affects the different characteristics and future intentions of participants and how the knowledge and skills gained from a MOOC are put to use. Nevertheless, there are some studies which have addressed the issue using qualitative, quantitative or mixed methods approaches regardless of the course topic.

MOOC learners are highly diverse individuals with multiple reasons for pursuing their learning in open online courses. In a MOOC, the participant is responsible for his/her learning. Previous studies have investigated the self-regulation of learning in the context of completion of a MOOC [7,8]. Also, it appears that self-regulated learning skills are important for online learning environments that offer high levels of learner autonomy and low level of teacher presence [9]. Time management skills can be seen as a part of self-regulated learning [10] and, therefore, these skills can play a decisive role in the completion of a MOOC [11]. It follows that participation in MOOCs can also affect the development of such skills [12].

In a previous study of university students, Jung and Lee [13] identified the order of characteristics which affect MOOC learners’ perceptions right after the course. They claim that the participants value the most the growth in knowledge, followed by attitudes, aspirations, and skills. The participants’ learning activities after participation in a MOOC have been seen as useful features to improve our understanding of the participants’ behaviour [14]. Thus, the learning experience that a MOOC offers can have an effect on the learner’s satisfaction with the course, leading to higher self-esteem after completing the MOOC and being valuable in future life choices in long-term [15]. The extent to which learners apply the knowledge they acquired from a MOOC in practice has been investigated in short-term with the result that a small number of participants continued using the learnt programming language [16] after a programming MOOC, while also starting to participate in the community of practice [14,15]. In addition to measuring the usage of acquired knowledge, another interesting aspect to investigate is whether learners continue to study the MOOC subject or related topics after the end of the course [15,17,18].

The first question to ask before measuring the post-MOOC development of a participant is what do participants themselves intend to achieve after the completion of a MOOC. In addition to personal developmental goals or knowledge gains [19], the reasons for enrolling in and completing a MOOC can also be related to an intention of changing one’s job [20] or pursuing career advancement [21,22,23]. Participating in MOOCs could also retain employment [24].

Participants also evaluate usefulness of the knowledge obtained from MOOCs in the context of personal life [15]. An intention to continue using massive open online courses has been seen as a reason for participating and completing a MOOC. Studies have indicated that participating in a MOOC can be helpful for choosing future educational paths [25] or provide inspiration for taking paid-for university courses [19].

Although programming MOOCs are no longer a novelty in online learning, some aspects of MOOCs are still under-studied, such as the impact on participants’ lives after the course. There has been recent attention to whether a MOOC has a longitudinal impact after the end of the course [26,27], but little is known about the impact of programming courses.

An analysis of the existing literature reveals that there are instruments for measuring the impact of a MOOC in short-term (less than three months) and even some studies on long term impact (at least three months) regardless of the field, but an instrument to measure the influence of programming MOOCs on participants’ lives in long-term was not found. Overall, there are very few studies investigating the long-term effects of MOOCs on participants’ life after the course. There are some well-known models to evaluate the results of a training program in a business context [28] that can be tested on MOOCs, but they lack some aspects that are important in MOOCs, e.g., career and study choices, which have been found important in previous studies [15,17,18]. In addition, these models are too general and not specific enough for programming MOOCs. Also, they need more modernisation to fit better with today’s rapidly changing world. Therefore, the aim of this study is to develop and validate a scale that would measure the impact of a programming MOOC on the lives of completers after the course, and thus also contribute to the understanding of the effects of MOOCs. According to the purpose of this study, two research questions were posed:

1.

Is the developed scale for measuring the impact of programming MOOCs on the lives of completers a valid and reliable instrument?

2.

What impact factors are rated higher and what factors are rated lower by MOOC completers?

3. Method

3.1. Context of the Study

This study focuses on three different programming MOOCs that are organised at our university. The courses are held in Estonian and teach the programming language Python primarily to adults, but participation is open to anyone.

First course is a 4-week course for people who have no previous experience in programming. It gives an overview of basic programming structures and concepts. This MOOC was organised from 2014 to 2019 and has had over 10,000 participants with an overall completion rate of 60%.

The second course is also a course for novice programmers with no experience in programming, but it is an 8-week course that covers more programming topics in depth than the previously described course. This 8-week course covers topics like Expressions, Conditional Execution, Functions, Iterations, Strings, Files, Lists and Graphics. From 2015 to 2019, more than 6000 people have participated and, overall, almost 50% of the participants complete this course.

The third course is intended for people who have previous experience in programming. This course aims to deepen participants’ knowledge in programming and covers more complex topics than the previous courses. This course has been held from 2017 to 2019 and has had over 2000 participants, with a completion rate of 30%.

The high completion rate in these courses was accomplished mainly through different support mechanisms and assessment tools [2]. All these courses include automatic assessment of programming exercises, self-assessment questions, troubleshooters for programming exercises to help solve common problems, weekly-quizzes, learning videos, additional learning materials, belletristic stories related to programming, and forums to support learners. In addition, a helpdesk was used in the first and second courses, where participants could write 24/7 if they had any problems with programming exercises. In all of these courses, the learner has to work through the learning materials and solve the weekly quiz and programming exercises. Over the years, many people have participated in these courses multiple times.

3.2. Sample

The study included participants who had completed the course and received a certificate that may have affected their job and career opportunities more than non-graduates. Data were collected via an online questionnaire that was sent to 8418 participants with valid email addresses, who had completed at least one of our MOOCs and no less than one year has passed since his/her last participation in our course. The one-year time gap between the MOOC completion and our study was set as a criterion because we tried to find an influence on career and study choices, and these changes take time. The questionnaire was answered by 1261 completers. The sample included 1261 programming MOOC participants, corresponding to 15% of the total MOOC alumni. There were 52.49% female and 47.51% male participants in the sample. The average age was 39.52 years (SD = 11.66), with a minimum age of 12 and a maximum age of 85 years. The completers had participated in courses an average of 1.64 (SD = 0.79) times, with a minimum of 1 and a maximum of 8 times. The average number of years that had passed since last participation in our programming MOOC was 2.43 years (SD = 1.08), ranging from 1 to 4 years.

3.3. Instrument and Procedure

The questionnaire was compiled with experts from the university in the field of computer science and educational theory. We wanted to create an instrument that would address various aspects that may affect MOOC participants’ life after the course. Therefore, we took into account the recurring themes that emerged from previous research. We designed six constructs of the scale.

At first, based on the conclusion of Sablina et al. [15] that learners’ future choices can be influenced by their satisfaction with the course and higher self-esteem after completing a MOOC, we created the first construct for learning experiences from MOOC, including statement items both related to general experiences like Completing the MOOC gave me satisfaction and experiences related to programming like My interest in programming grew after passing the MOOC.

As several studies [4,5,13] have reported MOOCs’ impact on knowledge and skills, we developed the second construct with items measuring knowledge and skills obtained from MOOCs (e.g., Participation in the MOOC complemented my knowledge of programming) and learning skills (e.g., Participation in the MOOC helped me develop self-management skills).

However, some learners enrol in MOOCs because they need these skills in their personal life, for instance, to educate their children [23], and the knowledge and skills gained after participating in a MOOC can be estimated in context of personal life [15]. We, therefore, created the third construct to measure the usage of knowledge and skills in personal life (e.g., I have applied the [programming] knowledge gained from the MOOC in my private life).

Since many learners are keen to continue studying the MOOC topic after the course, as has been found in several studies [15,16,18,19,25], we developed the fourth construct to represent interest in future programming studies, with items like This course gave me the prerequisite knowledge to study a computer science-related speciality.

As completing a MOOC can lead to a job change [20] or career advancement [21,22], and participants in MOOCs have applied the obtained knowledge in their professional lives [15], we created the fifth construct, measuring the impact on working life (e.g., I have applied the [programming] knowledge gained from MOOC in my work).

The last construct, measuring the acquisition of new contacts (e.g., Participation in the MOOC helped me establish new contacts with course participants), was based on the previous studies [14,15], which indicated that participants started taking part in the community of practice after MOOC.

The composed scale consisted of 22 items (see

Table A1. The model of the designed six-factor scale.

Factor	No.	Item
Knowledge and Skills Obtained from MOOC	1	Participation in the MOOC helped me develop self-management skills.
	2	Participating in MOOC helped me develop my time management skills.
	3	Participating in MOOC helped me develop my ability to learn independently.
	4	Participation in the MOOC complemented my knowledge of programming.
	5	This course gave me the basic knowledge to continue programming by myself.
Impact on Working Life	6	I have applied the [programming] knowledge gained from MOOC in my work.
	7	Participation in MOOC helped me to better compete in the labour market.
	8	Participation in MOOC helped me to get a promotion or a new job.
Usage of Knowledge and Skills in Personal Life	9	I have applied the [programming] knowledge gained from the MOOC in my private life.
	10	I have used the knowledge I gained from MOOC to teach others.
Interest in Future Studies	11	This course gave me the prerequisite knowledge to study a computer science-related speciality.
	12	This course motivated me to apply for a computer science-related specialty.
	13	Participating in MOOC helped me make decisions in my choice of specialty.
Learning Experience from MOOC	14	Completing the MOOC gave me satisfaction.
	15	Participation in MOOC was a good learning experience for me.
	16	Graduating from MOOC raised my self-esteem.
	17	Attending the course helped me better understand programmers.
	18	The course certificate was the most useful thing I got from the course.
	19	My interest in programming grew after passing the MOOC.
	20	Participation in MOOC gave me courage to participate in other e-courses as well.
New Contacts	21	Participation in MOOC helped to make new contacts with the course organizers.
	22	Participation in the MOOC helped me establish new contacts with course participants.

). The participants were asked to evaluate the extent to which they agreed with each item on a scale from 1 (strongly disagree) to 7 (completely agree). Questions about background data were also included in the questionnaire.

A pilot study was carried out a few weeks before the main study in February 2020. The questionnaire was sent to eight volunteering MOOC completers with different background indicators (e.g., number of courses taken, sex, age). The participants were asked to give feedback about the clarity of the statement items and questions. The results showed that all items were easily understood and that one question about background needed rewording. The participation in the main study was voluntary and no benefits were offered (e.g., payments, credits, prizes) for completing the questionnaire.

3.4. Data Analysis

Prior to the analysis, the data were screened and two participants were excluded from the sample due to anomaly in their answers. At first, exploratory factor analysis (EFA), using the Maximum Likelihood method with varimax rotation, was performed to identify the underlying structure. Then, confirmatory factor analysis (CFA) was carried out to verify the factor structure. The analyses were carried out using SPSS version 26 and AMOS version 26. The following five indicators for the goodness-of-fit of the models were evaluated: (a) chi-square test; (b) comparative fit index (CFI); (c) Tucker-Lewis index (TLI); (d) root mean square error of approximation (RMSEA); and (e) standardised root mean square residual (SRMR). The following cut-off points were used for the goodness-of-fit of the models: CFI ≥ 0.95 [29], TLI ≥ 0.90 [30], RMSEA ≤ 0.08 [29] and SRMR ≤ 0.08 [29]. Repeated measures ANOVA with the Bonferroni adjustment for multiple comparisons was used to identify differences between the factors.

4. Results

4.1. EFA

The Kaiser-Meyer Olkin measure of sampling adequacy (KMO = 0.886) and Bartlett’s test of sphericity (${\chi }^2$ = 12,836.289, df = 231, p < 0.001) showed that the data were factorable. The Kaiser-Guttman rule indicated that the structure of the five-factor model was suitable. One item (see Table A1 item no. 18) was excluded due to low commonality, which was less than 0.20 [31]. The factor structure with five factors and 21 items was used in further analysis (see

Table A2. The items of the used scale.

Factor	No.	Item
Learning Skills Acquired from MOOC (LSAM)	LSAM 1	Participation in the MOOC helped me develop self-management skills.
	LSAM 2	Participating in MOOC helped me develop my time management skills.
	LSAM 3	Participating in MOOC helped me develop my ability to learn independently.
Impact on Work and Personal Life (IWPL)	IWPL 1	I have applied the [programming] knowledge gained from MOOC in my work.
	IWPL 2	Participation in MOOC helped me to better compete in the labour market.
	IWPL 3	Participation in MOOC helped me to get a promotion or a new job.
	IWPL 4	I have applied the [programming] knowledge gained from the MOOC in my private life.
	IWPL 5	I have used the knowledge I gained from MOOC to teach others.
Interest in Future Studies (IFS)	IFS 1	This course gave me the prerequisite knowledge to study a computer science-related speciality.
	IFS 2	This course motivated me to apply for a computer science-related specialty.
	IFS 3	Participating in MOOC helped me make decisions in my choice of specialty.
	IFS 4	This course gave me the basic knowledge to continue programming by myself.
Learning Experience from MOOC (LEM)	LEM 1	Completing the MOOC gave me satisfaction.
	LEM 2	Participation in MOOC was a good learning experience for me.
	LEM 3	Graduating from MOOC raised my self-esteem.
	LEM 4	Attending the course helped me better understand programmers.
	LEM 5	Participation in the MOOC complemented my knowledge of programming.
	LEM 6	My interest in programming grew after passing the MOOC.
	LEM 7	Participation in MOOC gave me courage to participate in other e-courses as well.
New Contacts (NC)	NC 1	Participation in MOOC helped to make new contacts with the course organizers.
	NC 2	Participation in the MOOC helped me establish new contacts with course participants.

). The total variance explained by the five factors was 53.98%. All items related well to their factor (all loadings were above 0.4). All factors were labelled and internal consistency reliability of each factor was calculated (see

Table 1. Internal consistency reliability of the factors detected.

Factor	$\mathit{\alpha }$
Learning Skills Acquired from MOOC (LSAM)	0.93
Interest in Future Studies (IFS)	0.82
Learning Experience from MOOC (LEM)	0.78
New Contacts (NC)	0.76
Impact on Work and Personal Life (IWPL)	0.80

).

4.2. CFA

Confirmatory factor analysis was performed with the statements of scale to examine the factor structure. At first, the five factor model showed an unacceptable fit: ${\chi }^2$ = 1984.140, df = 182, CFI = 0.86, TLI = 0.83, RMSEA = 0.09 and SRMR = 0.07. We took into account modification indices proposed. After this, the model had acceptable fit indices: ${\chi }^2$ = 789.098, df = 158, CFI = 0.95, TLI = 0.93, RMSEA = 0.06 and SRMR = 0.05. Also, standardised factor loadings range from 0.39 to 0.95 (Figure 1), indicating that all items correlate fair (over 0.30) with their factor [32]. Item reliabilities range from 0.15 to 0.91 (Figure 1). Three items reliabilities were less than 0.20, suggesting that these items should be removed, but we retained them because they were conceptually relevant to their factors and their factor loadings were statistically significant (p < 0.001).

Convergent validity was evaluated by examining the average variance extracted (AVE), which is recommended to be greater than 0.5 [33]. However, it has been emphasised that AVE is a stricter measure than composite reliability (CR), and that the researcher may conclude the convergent validity of the construct is adequate even if AVE is less than 0.5 but composite reliability is higher than 0.6 [34,35]. Considering that CRs of all factors are above 0.7, the authors agreed that all AVEs are acceptable and the convergent validity was achieved.

Discriminant validity was evaluated using the heterotrait-monotrait (HTMT) criterion [36,37,38,39].

Table 2. Convergent and discriminant validity.

Factor	CR	AVE	HTMT
			LSAM	IWPL	IFS	LEM	NC
LSAM	0.94	0.84
IWPL	0.87	0.57	0.38
IFS	0.89	0.67	0.38	0.67
LEM	0.76	0.32	0.62	0.56	0.59
NC	0.77	0.62	0.39	0.52	0.38	0.30

illustrates the values of the heterotrait-monotrait criterion, which are all less than 0.85 [40] and therefore, discriminant validity was established.

The final model is presented in Figure 1.

Mauchly’s test of sphericity indicated that the assumption of sphericity has been violated, ${\chi }^2$(9) = 509.462, p < 0.001, and degrees of freedom were corrected using Greenhouse-Geisser estimates of sphericity. There were statistically significant differences between factors (F(3.349, 4219.575) = 2028.258, p < 0.001). Pairwise comparison with Bonferroni correction revealed that all factors were significantly different from each other (p < 0.0001). Means and standard deviations of the constructs are presented in

Table 3. Descriptive statistics of the factors detected.

Factor	Mean	SD	Min	Max
Learning Experience from MOOC	5.78	0.99	1.00	7.00
Learning Skills Acquired from MOOC	4.41	1.74	1.00	7.00
Interest in Future Studies	3.89	1.76	1.00	7.00
Impact on Work and Personal Life	3.38	1.44	1.00	7.00
New Contacts	1.81	1.14	1.00	7.00

. The data indicated that, after completing a MOOC, the participants valued the ‘Learning Experience form MOOC’ the highest and ‘New Contacts’ the lowest.

5. Discussion

The general aim of this study was to develop and validate an instrument for measuring the impact of programming MOOCs on the lives of participants after the course and also for understanding the effects of MOOCs.

Firstly, based on previous research, we developed a scale for measuring the impact of MOOCs on the lives of completers. The five-factor model proved to be applicable, taking into account the modification indices. The factors conformed to the developed constructs. The constructs ‘Impact on Working Life’ and ‘Usage of Knowledge and Skills in personal life’ were merged into one factor, which we named ‘Impact on Work and Personal Life’. Also, all items measuring knowledge and skills related to programming did not remain in the knowledge and skills factor. The item Participating in MOOC complemented my knowledge of programming correlated with ‘Learning Experience from MOOC’ and the item This course gave me the basic knowledge to continue programming by myself allied with ‘Interest in Future Studies’. These items related well with their factor and the factors did not need to be renamed. Therefore, the factor describing the construct ‘Knowledge and skills obtained from MOOC’ was renamed as ‘Learning Skills Acquired from MOOC’. The convergent and discriminant validity of the scale were established as well. The results indicated that all the detected factors demonstrated reliability coefficients of 0.70 or above, indicating good reliability. As the answer to the first research question, we could conclude that the developed scale for measuring the impact of MOOCs on the lives of completers is a valid and reliable instrument and covers different aspects of influencing factors from qualitative [11,15,19,21], quantitative e.g., [4,5,13,22] and mixed method [7,25,26] studies.

Secondly, we compared the factors in terms of how highly they were rated by MOOC completers. The result showed that ‘Learning Experience from MOOC’ was the highest-rated factor in our study, and that could be explained by the sample characteristics. Only 15% of completers filled in the questionnaire and it is possible that the people who had better memories about these MOOCs were more willingly to participate in our study, they were more satisfied and, therefore, their attitudes towards MOOC learning experiences were more positive. The other potential explanation for that result is that our sample consisted of learners who had completed at least one MOOC and were, therefore, more likely to have had a positive learning experiences. In addition, the support mechanisms (e.g., helpdesk) in our courses could also have positive effect on the learning experience.

As self-regulated learning skills are important for online learning environments [9], it might be that completers can sense the growth of this kind of learning skills and, therefore, the factor ‘Learning Skills Acquired from MOOC’ was also among the two highest rated factors. In our case, this factor described the learning skills that are related to self-regulated learning, such as time management, independent learning and self-management, and the tools in our MOOCs, like troubleshooters, automated assessment for programming exercises and self-assessments questions, support self-regulated learning skills.

The reason why learning experience and skills are the top-rated factors may be explained by the fact that the satisfaction with the course influences the learning skills. It has been found that when learner satisfaction increases, their self-regulated learning skills improve [12].

The factor ‘Interest in Future Studies’ was moderately rated (an average of 3.89 on the seven-point scale) in our study. The items in this factor only referred to studying computer science related specialities because the topic of our MOOCs was computer programming. However, studies [25] have indicated that participating in a MOOC can help participants choose their future educational paths and, therefore, it is possible that participation in a computer programming MOOC helped some participants decide that this was not the right choice for their future studies.

The factor ‘New Contacts’ was the lowest rated factor, while ’Impact on Work and Personal Life’ was penultimate in the ranking order of factors. The means of these two factors were below the neutral point (3.5) of the scale. The factor ‘Impact on Work and Personal Life’ being among the two lowest rated factors was an unexpected result because many studies e.g., [20,21,22] claim that people enrol in MOOCs with the intention to change a job or advance their career. The reason why the ‘New Contacts’ factor is one of the lowest rated factors may be caused to a lack of student to student interaction in our programming courses. The participants could use forums to communicate with each other but the course organizers did not initiate it.

6. Conclusions

This study sought to fill a significant gap in previous studies by developing a scale for measuring the post-impact of MOOCs, especially programming MOOCs, on their completers. The developed scale consists of five factors and is a valid and reliable instrument. The instrument can be used to measure not only the impact of programming MOOCs but also adapted in other studies in computer science. This study has shown that learning experience and gained learning skills from MOOC are important factors that have a long-term impact on course completers.

However, the study has some limitations which have to be considered before generalising the findings. First, only 15% of completers filled in the questionnaire and, therefore, the results may be more representative of people who had more positive experiences. Second, despite the fact that we tried to take into account all aspects, the limited number of previous studies means that there could be other factors that might have an impact on MOOC completers’. A possibility for future studies is to investigate in-depth the differences between participants’ characteristics in relation to their background data. Also, it might be useful to use this scale for studying the impact of MOOCs that are dedicated to topics other than programming.