Next Article in Journal
Anti-Asian Racism during COVID-19: Emotional Challenges, Coping, and Implications for Asian American History Teaching
Previous Article in Journal
Educational System Resilience during the COVID-19 Pandemic—Review and Perspective
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Assessing Massive Open Online Courses for Developing Digital Competences among Higher Education Teachers

by
Effrosyni Bitakou
*,
Maria Ntaliani
,
Konstantinos Demestichas
* and
Constantina Costopoulou
Informatics Laboratory, Department of Agricultural Economics and Rural Development, Agricultural University of Athens, 11855 Athens, Greece
*
Authors to whom correspondence should be addressed.
Educ. Sci. 2023, 13(9), 900; https://doi.org/10.3390/educsci13090900
Submission received: 29 July 2023 / Revised: 30 August 2023 / Accepted: 1 September 2023 / Published: 6 September 2023

Abstract

:
The outstanding growth in digital technologies has raised higher education teachers’ attention towards developing digital competences for effectively fulfilling students’ learning requirements and keeping up with the rapidly evolving global education system. Nowadays, higher education teachers can find a plethora of online courses provided by educational organizations, universities, and businesses for reskilling and upskilling. The objective of this paper is to provide results from the assessment of existing massive open online courses regarding their potential to cover a large range of digital competences that higher education teachers should possess. A total of one hundred and sixty-two (162) online courses from three prominent international digital platforms, offering massive open online courses, were subjected to evaluation using a model incorporating six categories of attributes aligned with pertinent digital competences. The results indicate that the majority of the existing online courses lack sufficient coverage of key digital competences required by higher education teachers. Notably, these inadequately addressed competences regard evaluating students’ performance in the digital environment and ensuring positive online student experiences through collaborative engagement and self-reflection. The findings of this study offer valuable insights not only to higher education teachers, but also to online course developers and education policy-makers. The identified gaps in digital competences underscore the pressing need for substantial improvements in existing online courses or the development of new courses to effectively bridge these competence gaps. By addressing these shortcomings, higher education institutions can better equip their teachers to navigate the digital realm and elevate the overall quality of education in the digital era.

1. Introduction

The COVID-19 outbreak had a significant impact on educational systems across the world [1,2] by disrupting educational activities [3,4] and creating severe challenges for learners and educators [5]. With the support of Information and Communication Technology (ICT), education has experienced an unprecedented shift towards digital learning platforms, electronic learning, and distance education [6], used as countermeasures during this destabilizing learning crisis [7].
In higher education, this digital transformation was, in many cases, abrupt, but serving as the only alternative for continuing academic activities [8]. Digital tools, such as teleconferencing and online lessons, were used to replace face-to-face practices for teaching, exams, communication, and collaboration among students and academics [9]. Given that twenty-first-century-generation students have experienced the rapid development of digital technology (e.g., Internet of Things, virtual reality, and artificial intelligence), the emphasis on digital competence in higher education continues to grow in popularity [10]. Higher education teachers need a comprehensive set of competences in a range of areas in order to make digital education for students equal and effective. Under these circumstances, current studies make evident that higher education teachers’ digital competence cannot longer be considered only as a tool, but also as an essential pedagogical element [11], and a key point in new learning scenarios [12]. It is noteworthy that there is a positive relation between the development of teachers’ digital competence with students’ digital competence quality [13].
Different terms, such as digital literacy, ICT literacy, and digital skills, have been used to discuss digital competence in education. The competences and skills necessary to comprehend and apply digital technology in educational practices have more recently been referred as professional digital competence. There are various theoretical views and conceptual frameworks that describe how digital technologies are used in educational activities and that define the attributes of instructors who are digitally competent [14]. The state-of-the-art of digital competence among university teachers, and the training needed for enhancing teachers’ digital competence are discussed in recent review articles. Nevertheless, there is a lack of understanding regarding the conceptualization of teachers’ digital competence inside the research literature. More specifically, teachers’ digital competence has frequently been defined and articulated in terms of the practical user abilities that are regarded as necessary to operate a digital classroom. This is why the term “skills” is more often mentioned explicitly by research works that deal with teachers’ digital competence, while “knowledge” and “attitude” come second and third in popularity. Nonetheless, there has been a small increase in publications referring to “attitude” during COVID-19, which shows how the increased usage of technology during COVID-19 appears to have impacted researchers’ interest in instructors’ attitudes about technology. Teachers’ attitudes may have changed as a result of being required to utilize technology, yet for some, this may have had a motivating effect [15].
The European Commission (EC) defines the term digital competence as “the confident, critical and responsible use of the technologies from the society of information for work, entertainment and education”. Also, the EC has included digital and technology-based competences in the eight key competences of lifelong learning, along with literacy; multilingualism; numerical, scientific, and engineering skills; interpersonal skills and the ability to adopt new competences; active citizenship; entrepreneurship; and cultural awareness and expression [16]. Moreover, the EC has developed the European Framework for the Digital Competence of Educators (DigCompEdu). The framework proposes 22 competences categorized in six areas, namely professional engagement, digital resources, teaching and learning, assessment, empowering learners, and facilitating learners’ digital competence. It aims at educators at all levels of formal education, as well as vocational education and training, special needs education, and non-formal learning contexts, to support them in developing educator-specific digital competences. It acknowledges that the innovation of education requires educators to master a set of digital competences specific to their profession [17].
Based on all of the above, we can deduce that the current streams of research in the area of digital competences have shed light on numerous aspects, including the following: (a) Definitions and frameworks: digital competences extend beyond mere technological proficiency, encompassing a range of abilities from using digital tools to critically evaluating digital information and understanding online ethics. Renowned frameworks, such as the DigCompEdu mentioned above, offer structured categorizations, breaking down digital competence into key components like information literacy, safety, and problem-solving. (b) Importance in modern society: research has consistently shown the central role of digital competences in various sectors, from education and employment to civic participation. Those lacking these competences are at a distinct disadvantage, enhancing the digital divide. (c) Assessment and Metrics: efforts have been made to measure digital competence, with several tools and rubrics developed. These assessments typically consider a blend of theoretical understanding and practical skills.
In parallel, according to the cited literature, certain topics in this area are still insufficiently explored: (a) Long-term evolution: as technology continuously evolves, it is challenging to predict the future requirements concerning digital competences. Existing frameworks might be less suited to address future digital landscapes, particularly with advancements like quantum computing or augmented reality on the horizon. (b) Universal and equal access: while the importance of digital competences is recognized, research has yet to offer comprehensive solutions for global disparities in digital access and education. The true depth of the digital divide, particularly in under-researched areas and demographics, remains a puzzle. (c) Integration challenges in education: while there is a consensus on the need for digital competences in curricula, the most effective ways to integrate them across diverse educational contexts, age groups, and disciplines remain underexplored.
In light of the above, our present study investigates the adequacy of existing massive open online courses (MOOCs) in equipping higher education teachers with sufficient teaching digital competences, thus contributing to the investigation of topics related to universal and equal access as well as to integration in education, as defined above. In particular, in recent years, there has been a significant increase in the availability of online courses covering a wide range of subjects, providing educators with ample opportunities to develop their digital competences. There are various online platforms, like Coursera [18], edX [19] and Udemy [20], providing online courses. Early in 2012, Coursera emerged as an independent for-profit technology. Several independent non-profit initiatives like Udacity and Udemy were created in the same year. Following that, edX adopted the MITx platform created by MIT and Harvard [21]. At present, with either free or paid access, the Coursera platform provides more than 7.000 online courses, edX provides over 4.000, and Udemy provides over 210.000. These platforms cover different subjects, such as health, science, and information technology. Typically, individuals enrolling in an online course are often required to pay for additional benefits.
Although there are available online courses and resources for the development of digital competences, there is preliminary, yet still limited, evidence (for instance, refer to our study from Greece [22]) suggesting that these are not provided in a structured and integrated way to support higher education teachers in identifying specific digital competence gaps and achieve efficient upskilling. The objective of this paper is to present the outcomes of an assessment study conducted on existing massive open online courses to evaluate their potential in addressing the diverse digital competences essential for higher education teachers. More specifically, the focus of this paper is to provide answers to the following questions regarding the digital competences of higher education teachers:
RQ1: Which digital competences of higher education teachers are better covered by the available online courses?
RQ2: To what extent do existing online courses cover the development of the broad spectrum of digital competences required by higher education teachers?
To the best of our knowledge, this is the first assessment study reported in the literature that investigates the potential and relevance of massive open online courses in covering the needs of higher education teachers in terms of digital competences. The structure of this paper is as follows: Section 2 analytically presents the methodology followed, as well as the model used for the assessment of online courses. Section 3 presents the results of the assessment of online courses. Finally, concluding remarks and future work are provided in Section 4.

2. Materials and Methods

2.1. Overview of the Methodology

In this section, the steps followed, as well as the model used for the assessment of online courses, are presented in detail. More specifically, as depicted in Figure 1, the design of this evaluation study comprised the following steps:
  • Online course collection: A group of three (3) specialized scientists was created that would act as the reviewers of the online courses. Three digital platforms, namely Coursera, edX, and Udemy were selected to find available online courses. Udemy is often categorized as an MOOC platform, since it contains a very large number of online courses, so we chose to include it for the purposes of this study. These platforms were selected for our assessment because they are among the most popular worldwide. The search was conducted manually and was focused using particular keywords: “teaching” or “education” or “competences” or “pedagogy” or “instruction” or “online teaching” or “digital” or “course development” or “online tools” or “online classes” or “core competences” or “usage competences” or “online teaching strategies” or “ensuring positive online student experiences” or “assessment strategies” or “physical health” or “data protection” or “identity management” or “law” or “storage” or “search” or “active participation” or “collaboration” or “production and sharing” or “creation and change” or “understand” or “design and structure” or “achieve” or “digital delivery” or “structured learning” or “self-regulated learning” or “top-down” or “collaborative” or “inspire” or “reflection” or “knowledge gaps” or “formative” or “summative” or “scaffolding” or “group working” or “peer assessment” or “data analysis” or “identity material” or “modify material” or “communicate value” or “interactivity” or “engaging static content”.
  • Online course assessment: A dedicated model (described in Section 2.1) was used to assess the collected online courses regarding their potential in covering the development of a range of digital competences of higher education teachers.
  • Peer review scheme: Once an expert performed an assessment, two additional experts were asked to perform their own independent assessment of the same online courses in a blind manner (i.e., without knowing the initially assigned scores). In cases of differences in the assessment of a particular course, the median values were calculated and assigned as final scores. This review scheme was randomized so that the three experts had equal chances of serving in the roles of first, second, and third reviewer for each course.
  • Final filtering: In cases where the consolidated experts’ opinion pointed out that an online course was relevant to computer science field and the teaching of at least one digital competence of higher education teachers, then the online course was considered for the final step of results analysis. Otherwise, if the course did not cover any of the digital competences, then it was excluded from the last and final step.
  • Results analysis: The results of the assessment were subjected to statistical analysis so as to provide insights into the strengths and weaknesses of the available online courses for enhancing higher educators’ digital skills.

2.2. Online Course Assessment Model

The model employed in this study was based on the DigCompEdu framework and was developed in the context of the “Advancing Digital Competence in Higher Education-ADVICE” Erasmus + project for supporting higher education teachers in tackling their digital competence gaps and mismatches [23]. The aim of the model is to assess available online courses in terms of their ability to develop a range of digital competences required by higher education teachers for making digital education more accessible and effective for their students.
More specifically, the model involves fifty-five (55) attributes for the description and assessment of each online course. These attributes are categorized in fifteen (15) categories, namely nine (9) descriptive categories and six (6) assessment categories. In essence, the attributes of the descriptive categories represent specific key meta-data of an online course, while the attributes of the assessment categories represent key digital competences that a higher education teacher might need to develop.
The descriptive categories are the following: (1) content description, (2) subject, (3) intellectual property, (4) identifiers, (5) resource, (6) language, (7) audience, (8) sources and relations, and (9) rating. These descriptive categories include twenty-two (22) attributes in total; in each one, the experts are asked to record specific meta-data regarding the online course they are viewing.
The assessment categories are the following (for ease of reference, we use consecutive numbering): (10) core competences, (11) usage core competences, (12) online teaching strategies, (13) ensuring positive online student experiences, (14) assessment strategies, and (15) creating digital content. These assessment categories include thirty-three (33) attributes in total. In each of these attributes and for each online course under investigation, the experts are asked to provide an assessment, using a scale from 0 to 3, whereby 0 represents no coverage of the development of the corresponding digital skill, 1 represents low coverage, 2 represents medium coverage, and 3 represents high coverage.
Consequently, by summing up the individual scores of its attributes, each of the assessment categories can be characterized as low, medium, or high coverage. Low coverage means that a category of digital competences is not satisfactorily covered or not covered at all in the online course, whereas high coverage means that a category of digital competences is very satisfactorily or fully covered in the online course. Since the assessment categories have different numbers of attributes, the levels of coverage are defined as follows:
(i)
Categories with five attributes (“Core Competences”, “Usage Core Competences”, “Ensuring Positive Online Student Experiences”, “Creating Digital Content”) can reach a maximum total score of 5 × 3 = 15, so the levels are defined as follows: 0 ≤ low < 5; 5 ≤ medium < 10; and 10 ≤ high ≤ 15.
(ii)
The “Online Teaching Strategies” category, which has six attributes, can reach a maximum total score of 6 × 3 = 18, so the levels are defined as follows: 0 ≤ low < 6; 6 ≤ medium < 12; and 12 ≤ high ≤ 18.
(iii)
The “Assessment Strategies” category, which has seven attributes, can reach a maximum total score of 7 × 3 = 21, so the levels are defined as follows: 0 ≤ low < 7; 7 ≤ medium < 14; and 14 ≤ high ≤ 21.
Finally, since the assessment categories include 33 attributes in total, the total score that an online course can gain ranges from 0 to 99. Thus, an online course can be characterized as low, medium, or high coverage, based on the sum of the individual scores of these attributes (0 ≤ low < 33; 33 ≤ medium < 66; 66 ≤ high ≤ 99).

2.2.1. Descriptive Categories and Their Attributes

In the following, the attributes of each descriptive category are analyzed in detail. The selection of the attributes included in our model was based upon existing meta-data standards, such as the Dublin Core Metadata Element Set and the IEEE Learning Object Metadata (LOM) [24].
(1) Content Description
Title: the descriptive name given to an online course (e.g., course/unit/lecture, certificate) to identify or describe it.
Abstract: a summary providing a general description and the most important information about the course.
(2) Subject
Broad: a wide area of knowledge that the course belongs to (e.g., Law, Art).
Narrow: the specific subject that the course belongs to (e.g., Data Protection Legislation).
(3) Intellectual Property
Creator: the person that has created the course.
Contributor: someone who has contributed to the course (e.g., in the form of additional content, articles, reviews, etc.).
Publisher: person or organization who is responsible for publishing the course online.
License type: the type of license of the course (e.g., commercial, public domain).
License cost: each course can be characterized as “free”, “paid”, or “mixed”. The “mixed” attribute refers to a combination of “free” and “paid” courses.
(4) Identifiers
Identifier: a Uniform Resource Locator (URL), commonly known as Web address, is the unique identifier used to locate a course on the Internet.
Alternative identifier: an additional URL or other type of unique code or number which can be used to uniquely refer to the course.
(5) Resource
Type: the types of learning resources that are included in an online course. These types are article/reference, assessment (e.g., exam, quiz), course/unit/lecture, image/visuals/PowerPoint, software, textbook, or other.
Format: the format of learning resources that are included in an online course. These are document, image, audio, video, multimedia, or other.
Length: the duration of the online course in various metrics, e.g., teaching hours and video projections, the expected duration in weeks and months, and the number of sections, lectures, and questions.
(6) Language
Language: the language in which the course is provided (e.g., English, French, etc.).
(7) Audience
Audience: the groups of people that the course targets. In the present study, there are two types of audiences: “higher level educators” and “general public”, excluding students at schools.
Education level: the level of difficulty of the course, namely beginner, intermediate, advanced, or other.
(8) Sources and Relations
Requires: specific background knowledge/skills that participants should have to successfully attend the course.
References: any information necessary for identifying and retrieving relevant works.
(9) Rating
User rating: the average evaluation of the course by people who have already attended it. This attribute typically uses a five-star rating system.
Official rating: a rating provided by an official authority, e.g., a quality assurance organization, a peer reviewer, a university, etc.
Other rating: any other type of rating provided for this course that does not belong to the two aforementioned types.

2.2.2. Assessment Category Attributes

In the following, the attributes of each assessment category are analyzed in detail. In principle, each attribute represents a different digital skill that higher education teachers need to learn and that an online course might teach (to some extent, on a scale from 0 to 3). As noted in the beginning of Section 2.2, the model used was developed by the ERASMUS+ Project “ADVICE” [23], as an extension and adaptation of the DigCompEdu framework of the European Commission.
(10) Core Competences: digital skills related to safe digital environment navigation.
Physical health: does the course teach how to maintain physical health (e.g., protecting eyesight) when participating in digital learning?
Data protection: does the course teach how to ensure the protection of third-party data?
Identity management: does the course teach how to ensure the participant’s online identity?
Law: does the course teach how to ensure data protection and privacy?
Storage: does the course teach how to store data and information securely?
(11) Usage Core Competences: digital skills for retrieving and adapting information and collaborating with others online.
Search: does the course teach how to find and identify required information online?
Active participation: does the course teach how to engage and participate in online communities?
Collaboration: does the course teach how to work with colleagues in collaborative online platforms (e.g., Slack, MS Teams, etc.)?
Production and sharing: does the course teach how to create digital resources to present ideas online?
Creation and change: does the course teach how to adapt self-created digital resources to different audience needs (e.g., improve their accessibility)?
(12) Online Teaching Strategies: digital skills related to teaching strategies in online digital spaces.
Understand: does the course teach how to introduce new online teaching methods? (In particular, this refers to the adaptation of teaching methods according to the specific objectives of the course.)
Design and structure: does the course teach how to structure lessons or teaching processes in general?
Achieve: does the course teach how to deliver hybrid (online and offline) lessons?
Digital delivery: does the course teach how to adapt digital lessons to different types of students?
Structured learning: does the course teach how to structure online lessons?
Self-regulated learning: does the course teach how to plan lessons, monitor, and reflect on their delivery, and use this reflection to inform the planning of the next lesson?
(13) Ensuring Positive Online Student Experiences: digital skills for improving the experience of online students.
Top-down: does the course teach how to create interactive and interesting lectures online for students?
Collaborative: does the course teach how to collaborate with students in an online space?
Inspire: does the course teach how to build safe online spaces for students?
Reflection: does the course teach how to encourage positive behaviors in students in academic spaces (such as self-refection, self-critique, and discipline)?
Knowledge gaps: does the course teach how to identify gaps in students’ knowledge using online tools?
(14) Assessment Strategies: digital skills that help participants to assess their students in the digital environment.
Understand: does the course teach how to assess students’ work/comprehension of the topics online?
Formative: does the course teach how to monitor students’ work and offer meaningful feedback in the digital environment?
Summative: does the course teach how to test student skills and knowledge?
Scaffolding: does the course teach how to provide the appropriate amount of online support to students to enable them to independently complete tasks beyond their initial capacity?
Group working: does the course teach how to evaluate student group work online?
Peer assessment: does the course teach how to encourage student peer review and assessment?
Data analysis: does the course teach how to analyze and assess different students’ learning behaviors?
(15) Creating Digital Content: digital skills that help participants to develop digital content for online lessons.
Identify material: does the course teach how to identify useful online and offline digital content?
Modify material: does the course teach how to modify pre-existing digital content?
Communicate value: does the course teach how to communicate the value of content to students in the online environment?
Interactivity: does the course teach how to develop interactive online content for students (e.g., interactive models)?
Engaging static content: does the course teach how to transform static content into interactive content (e.g., book excerpts)?

3. Results

3.1. Online Course Collection

The evaluation study was conducted from August 2021 to December 2022. From the search of online courses, in total, one hundred and sixty-two (162) online courses were found to be relevant. Specifically, twenty-six (26) courses were found inside the edX platform, one hundred (100) inside Udemy, and thirty-six (36) inside Coursera. Three reviewers from Greece participated in the evaluation of the online courses and were asked to use and apply the aforementioned assessment model (described in Section 2.2). The experts used Excel files and Google sheets to fill in the values of the attributes of the descriptive and assessment categories. In the following, the results regarding each attribute of the descriptive and assessment categories are presented in detail.

3.2. Assessment of Online Courses

3.2.1. Descriptive Category Attributes

(1) Content Description
Title: courses are related to strategies for online teaching and learning (50), data protection (32), privacy (18), identity management (10), cyber security (6), and the rest (46) to other relevant topics.
Abstract: all online courses provide an abstract.
(2) Subject
Broad: broad subjects include education and teacher training courses (46); information technology (IT) and software (27); social sciences (20); law (10); business strategy (9); computer science (9); network and security (9); data analysis and statistics (8); management (6); database design, app and web development (5); operating systems and servers (3); personal productivity (2); software engineering (2); e-commerce (1); communication courses (1); programming languages (1); career development (1); general health (1); and social media marketing (1).
Narrow: most of the courses (98) have indicated specific narrow subjects. These subjects refer to several different topics, with the majority focusing on online teaching (30) and data protection (15).
(3) Intellectual Property
Creator: all courses provide the name of the creator (professor, university, or company).
Contributor: all courses provide the name of the contributor (professor, university, or company).
Publisher: the name of the publisher is typically the platform name (edX, Udemy, Coursera).
License type: only two courses explicitly provide this type of information (commercial license).
License cost: courses are either “paid” (84), “free” (38), or “mixed” (40).
(4) Identifiers
Identifier: all courses have a Web address (URL).
Alternative identifier: no alternative identifiers were found for the courses under investigation.
(5) Resource
Type: the types of learning resources included in courses are courses/units/lectures (158) and assessments (e.g., exam, quiz) (4).
Format: the formats are multimedia (138), videos (20), documents (3), and other (1).
Length: the majority of courses (138) provided this information in number of hours, while some courses (82) provided this information in number of modules, as well.
(6) Language
Language: courses are provided mainly in English (161), and, additionally, in French (11), German (10), Italian (11), Portuguese (11), Spanish (10), Polish (10), Russian (9), Arabic (8), Vietnamese (8), Indonesian (2), Romanian (2), Greek (1), Japanese (1), and Thai (1).
(7) Audience
Audience: courses are mainly addressed to the general public (149), while a few (13) are specifically targeted to higher-level educators.
Education level: most courses (120) target the beginners’ level, while fewer target the intermediate level (37) or the advanced level (3).
(8) Sources and Relations
Requires: in total, 120 courses require basic knowledge of information technology; the rest do not have any prerequisites.
References: no course explicitly provides this information.
(9) Rating
User rating: only courses provided by the Udemy (99) and Coursera (34) platforms provide a five-star rating system. The max rating noted is 4.8 and the min rating is 4.1.
Official rating: no course provides this value.
Other rating: no course provides this value.

3.2.2. Assessment Category Attributes

As already mentioned, in each attribute of the assessment categories, an online course was evaluated using a scale from 0 to 3. A value of 0 suggests that the online course is not relevant to the teaching of the corresponding digital skill, while a value of 1, 2, or 3 suggests that the online course is relevant to the teaching of that digital skill to a low (1), medium (2), or high (3) degree, respectively.
The key results of this study are presented through the following bar charts. In particular, Figure 2 presents the assessment of the attributes of the core competences category. It can be noted that, with the exception of the “data protection” attribute or digital skill, the rest of the attributes of the core competences category are not adequately covered by the offered courses. In particular, approximately half of the investigated courses cover the teaching of data protection to either a high (38.9%) or medium (11.7%) degree. The skill of “identity management” is covered by 9.9% of the courses to a high degree and by 13% of the courses to a medium degree. The remaining attributes (physical health, storage, and law) are covered by only a small part of the courses (for instance, 0.6% in the case of physical health).
Figure 3 presents the assessment of the attributes of the usage core competences category. It shows that the majority of the courses does not sufficiently cover the skills in this category. This is especially true for the “collaboration” skill, which is covered by less than 1 out of 10 courses. On the other hand, “search” and “active participation” are the best-covered skills in this category, with approximately 3 out of 10 courses being relevant (to a medium or high degree) to their teaching.
Figure 4 presents the assessment of the attributes in the online teaching strategies category. It can be noted that, in particular, the “achieve” skill is not sufficiently covered, in comparison with the others. On the other hand, “digital delivery”, “understand”, and “structured learning” are among the best-covered skills in this category, with approximately one out of two courses being relevant (to a medium or high degree) to their teaching.
Figure 5 presents the assessment of the attributes related to ensuring positive online student experiences. It shows that the majority of the courses does not sufficiently cover the skills in this category. This is especially true for the “collaborative”, “inspire”, “reflection”, and “knowledge gaps” skills, which are covered by less than 10–15% of the investigated courses. On the other hand, the “top-down” skill is the most widely covered in this category, with approximately 4 out of 10 courses being relevant (mostly to a medium degree) to its teaching.
Figure 6 presents the assessment of the attributes in the assessment strategies category. It shows that the majority of the courses does not sufficiently cover the skills in this category. This is especially true for the “understand”, “formative”, “summative”, “scaffolding”, “group working”, and “peer assessment” skills. On the other hand, the “data analysis” skill is the most widely covered in this category, with one out of two courses being relevant (to a medium or high degree) to its teaching.
Figure 7 presents the assessment of the attributes related to the creating digital content category. It shows that the majority of the courses does not sufficiently cover the skills in this category. This is especially true for the “engaging static content” skill. On the other hand, the “identify material” skill is the best-covered in this category, with almost 3 out of 10 courses being relevant (to a medium or high degree) to its teaching.
For each digital skill, Table 1 presents the number of online courses that are relevant to the teaching of the particular digital skill, which means that they have achieved a score from one to three (i.e., above zero).
As may be observed, the digital skill that is covered by most of the courses is “data analysis” (59.87%). Moreover, about half of the courses cover the development of the following digital skills: “data protection” (54.47%); “understand” (51.85%) and “design and structure” (45.68%) from the online teaching strategies category; “digital delivery” (51.23%); and “structured learning” (48.15%). The least-addressed digital competences, namely those that are covered by less than 10% of courses, are as follows: “physical health” (0.62%); “storage” (1.23%); “peer assessment” (7.41%); “collaboration (9.26%); and ”inspire” (9.87%).
Table 2 shows, for each of the assessment categories, the number and percentage of online courses, split in terms of coverage level (low/medium/high). As mentioned, a category is characterized to provide a low, medium, or high level of coverage, based on the sum of the individual scores of its attributes.
According to Table 2, all of the assessment categories are covered by the courses we investigated, at least to a low degree. With the exception of the online teaching strategies, all the other assessment categories are characterized by a low level of coverage. Specifically, the category of online teaching strategies is the one addressed in the most sufficient manner compared with the rest, and presents 46.30% low coverage, 40.12% medium coverage, and 13.58% high coverage. Notably, the category of core competences does not present high coverage at all.
Subsequently, for each online course, we calculated the total score of the course as the sum of the attribute scores. As explained in Section 2.2, based on this total score, a course can be considered to provide a low, medium, or high coverage of digital competences needed by higher education teachers (0 ≤ low < 33; 33 ≤ medium < 66; 66 ≤ high ≤ 99). Table 3 depicts the mean, maximum, and minimum total scores of all courses by platform.
According to the mean score in Table 3, the courses provided by the three platforms present, on average, a low overall degree of coverage of digital competences. The maximum total score, which is 56, is presented by only one course offered through the Coursera platform. The minimum total score, which is two, is presented by two courses, one offered through the edX platform and one through Udemy.
Table 4 shows, for each of the three platforms, the number and percentage of courses by coverage level.
In total, there are no courses presenting a high degree of coverage of digital competences. Moreover, 87.65% of the courses are characterized by low coverage, whereas the remaining 12.35% are characterized by medium coverage.
Subsequently, a correlation analysis was performed, employing the Pearson correlation coefficient, to investigate whether there is a relationship among the assessment categories. According to Table 5, ensuring positive online student experiences is not related with assessment strategies. All the other categories are related, either in a weak (i.e., where the Pearson coefficient ranges from 0.25 to 0.50) or moderate manner (i.e., where the Pearson coefficient ranges from 0.50 to 0.75). For instance, there is a moderate positive relation between the online teaching strategies and the usage core competences, suggesting that courses that perform well in the development of online teaching strategies also do well in the development of usage core competences, and vice versa. Notably, there is a negative relation (of weak or moderate strength) among core competences and all the other categories.

4. Discussion

Higher education teachers are faced with a complex, dynamic, and uncertain educational panorama, in which digital competences are considered as key competences that they have to master to further evolve their teaching [25,26].
This work has presented the results from assessing existing massive open online courses provided by popular international platforms, so as to investigate whether they cover a range of digital competences for higher education teachers. In total, one hundred and sixty-two (162) courses were assessed. The majority of them are courses/units/lectures, supported with learning resources in multimedia formats, and they are targeted to the general public. Also, most of the courses are mainly targeted for beginners’ and require basic knowledge of information technology. More than half require payment for enrollment and/or accessing more modules.
According to the results, the majority of the online courses (87.65%) is characterized by a low degree of coverage of key digital competences, and the rest (12.35%) by a medium degree of coverage, meaning that they do not sufficiently cover the broad range of digital competences required by higher education teachers. Notably, we identified no courses presenting a high degree of coverage of digital competences. Although all of the assessment categories are covered by the courses, at least to a low degree, the core competences, usage core competences, ensuring positive online experiences, assessment strategies, and creating digital content categories are characterized by low coverage, which denotes that these categories are not satisfactorily covered or not covered at all in the courses under study. The least-covered categories are core competences and assessment strategies. On the other hand, the category of digital competences that is most covered is online teaching strategies, presenting a high coverage of 13.58%.
As far as the specific digital skills (attributes) are concerned, none of them are covered by more than 60% of the courses. The attribute that is covered by most of the courses is “data analysis” (59.87%), which belongs to the assessment strategies category. The least-covered attribute is “physical health” (0.62%), which is part of the core competences category.
Moreover, the correlation analysis has revealed that there is either a weak or moderate positive relation between most of the assessment categories. For instance, there is a moderate positive relation between the online teaching strategies and the usage core competences, suggesting that the performance of a course in the former assessment category is an indicator of its performance in the latter, and vice versa. Notably, there is a negative relation (of weak or moderate strength) among core competences and all the other categories.
As far as the limitations of this study are concerned, since the assessment of online courses was realized by three reviewers, it also entails an element of subjectivity. We made an effort to address this through the peer review scheme included in our methodology. We acknowledge that, if we employed more reviewers with different backgrounds, we could reduce the risk of bias even more. Another limitation is the number of MOOC platforms that we investigated; although we selected the most popular ones, Coursera, and edX, as well as, the Udemy platform. It is worthy noted that there are additional MOOC platforms present in the market that we have not investigated in this study. Moreover, we have to acknowledge that, although the reviewers performed their best to evaluate each course accurately and reliably, they relied on the description and metadata (curriculum) provided by the courses themselves, in conjunction with a quick cross-check of their content. Thus, since there is the possibility that the content of some courses is not well-reflected in their description, this could have introduced some degree of inaccuracy in our study. Nonetheless, through the clear and robust evaluation design specified in Section 2.1, we have strived to reduce the effects of these limitations so that we can draw useful conclusions for the stakeholders involved. This type of inter-rater reliability check is consistent with best practices in both quantitative and qualitative research to reduce individual biases [27].

5. Conclusions

This work provides insights not only for higher education teachers, but also for developers of online courses and decision-makers in education. The added value of this evaluation study is that it provides higher education teachers with a structured way when searching for online courses to identify digital competence attributes that are suitable for them. Also, this study has revealed gaps in existing online courses regarding digital competences that developers of online courses should bear in mind for improving existing courses or developing new ones so as to cover higher education teachers’ needs.
In a nutshell, according to the results of this work, the following are suggested:
  • For policy-makers:
    o
    There is a noticeable lack of online courses that adequately align with current digital competence frameworks. Existing online courses fail to cover the wide range of necessary digital skills.
    o
    There is a lack of online courses explicitly targeted for the needs of higher educators (rather than general audiences).
  • For higher education teachers and their institutes:
    o
    The top five digital competences addressed by existing online courses are ”data analysis”, “data protection”, “understand”, “digital delivery”, and “structured learning”, meaning that there are good chances of finding relevant online courses to cover them.
    o
    The bottom five digital competences addressed by existing online courses are: “physical health”, “storage”, “peer assessment”, “collaboration”, and “inspire”, meaning that there are small chances of finding relevant online courses to cover them.
  • For developers of online courses and platforms offering massive open online courses:
    o
    The average degree of coverage of digital competences offered by existing online courses is low (ranging from 15 to 24, depending on the platform, on a scale from 0 to 99).
    o
    There is need for better coverage of both individual digital competences (see the bottom five competences identified above) as well as of entire categories (refer to Table 2), by developing new courses or expanding existing ones.
    o
    Digital skills falling into the categories of core competences, ensuring positive online student experiences, and assessment strategies are covered to a mostly low degree by existing online courses. Digital skills related to usage core competences and creating digital content are better-covered, although still to a low to medium degree. Digital skills belonging to the category of online teaching strategies are, overall, covered to a more satisfactory degree at the moment.
    o
    Innovative courses need to be meticulously crafted from the outset, specifically tailored for higher educators, and be based on comprehensive digital competence frameworks. Simultaneously, to address the fragmented nature of current courses, it is essential to establish learning pathways. These pathways, should combine different course collections for more effective skill enhancement.
According to recent research [25], there has been a notable exponential increase in the occurrence of issues and gaps concerning the competences of higher education teachers. This requires continuous training and effort. However, since most of the existing courses require payment, work needs to be undertaken by decision-makers towards the provision of high-quality and certified online courses that are cost-free for higher educators. This could be achieved through the establishment of collaboration networks among European universities and research institutions.
In the future, the established methodology can be used for the assessment and comparison of additional online courses related to the development of digital competences. Thus, we aim to maintain and update our dataset containing the assessment of online courses, which can prove to be useful for third-party researchers and course developers alike. Moreover, a study has to be realized in order to investigate the reasons why there are not many online courses dedicated to higher-level educators (i.e., not to the general public), taking into consideration their particular needs and characteristics. Lastly, future studies can be focused on evaluating in more detail the quality of the content and learning methodology of the existing courses, thus further deepening the results of this study.

Author Contributions

Conceptualization, K.D. and C.C.; Methodology, M.N., K.D. and C.C.; Validation, M.N., K.D. and C.C.; Formal analysis, E.B. and M.N.; Investigation, E.B.; Resources, E.B.; Data curation, E.B.; Writing—original draft, E.B., M.N., K.D. and C.C.; Writing—review & editing, E.B., M.N., K.D. and C.C.; Visualization, E.B.; Supervision, C.C.; Project administration, K.D. and C.C.; Funding acquisition, K.D. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the project “Advancing Digital Competence in Higher Edu-cation-ADVICE”, an ERASMUS+Strategic Partnership (2020-1-PL01-KA226-HE-096098), Key Action 2, funded by the European Union.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Publicly available datasets were created and analyzed in this study. These data are openly available here: https://informatics.aua.gr/wp-content/uploads/2023/09/MOOC-TDC-v1.zip.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

References

  1. Banović, A.; Mladenović, M.; Boljat, I. Informatics teacher’s perspective on distance learning during COVID-19 lockdown in Croatia. In Proceedings of the 14th Annual International Conference of Education, Research and Innovation, Online Conference, 8–9 November 2021. [Google Scholar]
  2. Cui, Y.; Ma, Z.; Wang, L.; Yang, A.; Liu, Q.; Kong, S.; Wang, H. A survey on big data-enabled innovative online education systems during the COVID-19 pandemic. J. Innov. Knowl. 2023, 8, 100295. [Google Scholar] [CrossRef]
  3. El Masri, A.; Sabzalieva, E. Dealing with disruption, rethinking recovery: Policy responses to the COVID-19 pandemic in higher education. Policy Des. Pract. 2020, 3, 312–333. [Google Scholar] [CrossRef]
  4. Karalis, T. Planning and evaluation during educational disruption: Lessons learned from COVID-19 pandemic for treatment of emergencies in education. Eur. J. Educ. Stud. 2020, 7, 125–142. [Google Scholar]
  5. Daniel, S.J. Education and the COVID-19 pandemic. Prospects 2020, 49, 91–96. [Google Scholar] [CrossRef] [PubMed]
  6. Aldulaimi, S.H.; Abdeldayem, M.M.; Keir, M.A.; Al-Sanjary, O. E-learning in higher education and COVID-19 outbreak: Challenges and opportunities. Psychol. Educ. J. 2021, 58, 38–43. [Google Scholar]
  7. Krishnamoorthy, R.; Keating, K. Education crisis, workforce preparedness, and COVID-19: Reflections and recommendations. Am. J. Econ. Sociol. 2021, 80, 253–274. [Google Scholar] [CrossRef] [PubMed]
  8. Edem Adzovie, D.; Jibril, A.B. Assessment of the effects of COVID-19 pandemic on the prospects of e-learning in higher learning institutions: The mediating role of academic innovativeness and technological growth. Cogent Educ. 2022, 9, 2041222. [Google Scholar] [CrossRef]
  9. Alsoud, A.R.; Harasis, A.A. The impact of COVID-19 pandemic on student’s e-learning experience in Jordan. J. Theor. Appl. Electron. Commer. Res. 2021, 16, 1404–1414. [Google Scholar] [CrossRef]
  10. Zhao, Y.; Llorente, A.M.P.; Gómez, M.C.S. Digital competence in higher education research: A systematic literature review. Comput. Educ. 2021, 168, 104212. [Google Scholar] [CrossRef] [PubMed]
  11. Núñez-Canal, M.; de Obesso, M.D.L.M.; Pérez-Rivero, C.A. New challenges in higher education: A study of the digital competence of educators in Covid times. Technol. Forecast. Soc. Chang. 2022, 174, 121270. [Google Scholar] [CrossRef]
  12. Tejedor, S.; Cervi, L.; Pérez-Escoda, A.; Jumbo, F.T. Digital literacy and higher education during COVID-19 lockdown: Spain, Italy, and Ecuador. Publications 2020, 8, 48. [Google Scholar] [CrossRef]
  13. Tondeur, J.; Howard, S.; Van Zanten, M.; Gorissen, P.; Van der Neut, I.; Uerz, D.; Kral, M. The HeDiCom framework: Higher Education teachers’ digital competencies for the future. Educ. Technol. Res. Dev. 2023, 71, 33–53. [Google Scholar] [CrossRef]
  14. Masoumi, D.; Noroozi, O. Developing early career teachers’ professional digital competence: A systematic literature review. Eur. J. Teach. Educ. 2023. [Google Scholar] [CrossRef]
  15. Smestad, B.; Hatlevik, O.E.; Johannesen, M.; Øgrim, L. Examining dimensions of teachers’ digital competence: A systematic review pre-and during COVID-19. Heliyon 2023, 9, e16677. [Google Scholar] [CrossRef] [PubMed]
  16. European Commission. Directorate-General for Education, Youth, Sport and Culture, Key competences for lifelong learning. Publications Office, 2019. Available online: https://data.europa.eu/doi/10.2766/569540 (accessed on 20 June 2023).
  17. Redecker Christine. European Framework for the Digital Competence of Educators: DigCompEdu. Available online: https://policycommons.net/artifacts/2163302/european-framework-for-the-digital-competence-of-educators/2918998/ (accessed on 20 June 2023).
  18. Coursera|Degrees, Certificates, & Free Online Courses. Available online: https://www.coursera.org (accessed on 28 July 2023).
  19. edX. edX|Build New Skills. Advance Your Career. Available online: https://www.edx.org (accessed on 28 July 2023).
  20. Udemy. Online Courses-Learn Anything. On Your Schedule|Udemy. Available online: https://www.udemy.com (accessed on 28 July 2023).
  21. Baturay, M.H. An overview of the world of MOOCs. Procedia-Soc. Behav. Sci. 2015, 174, 427–433. [Google Scholar] [CrossRef]
  22. Bitakou, E.; Demestichas, K.; Ntaliani, M.; Costopoulou, C. An assessment of online courses for supporting Greek higher education teachers’ digital competence. In Proceedings of the 15th International Conference on Education and New Learning Technologies (EDULEARN23), Palma de Mallorca, Spain, 3–5 July 2023. [Google Scholar]
  23. ADVICE Project–Advancing Digital Competence in Higher Education. Available online: https://advice.civitas.edu.pl (accessed on 27 August 2023).
  24. McClelland, M. Metadata standards for educational resources. Computer 2003, 36, 107–109. [Google Scholar] [CrossRef]
  25. Basilotta-Gómez-Pablos, V.; Matarranz, M.; Casado-Aranda, L.A.; Otto, A. Teachers’ digital competencies in higher education: A systematic literature review. Int. J. Educ. Technol. High. Educ. 2022, 19, 8. [Google Scholar] [CrossRef]
  26. Guillén-Gámez, F.D.; Cabero-Almenara, J.; Llorente-Cejudo, C.; Palacios-Rodríguez, A. Differential analysis of the years of experience of higher education teachers, their digital competence and use of digital resources: Comparative research methods. Technol. Knowl. Learn. 2022, 27, 1193–1213. [Google Scholar] [CrossRef]
  27. Krippendorff, K. Content Analysis: An Introduction to Its Methodology; Sage Publications: Los Angeles, CA, USA, 2019. [Google Scholar]
Figure 1. Overview of the methodology followed in this assessment study.
Figure 1. Overview of the methodology followed in this assessment study.
Education 13 00900 g001
Figure 2. Assessment of the attributes of the core competences category.
Figure 2. Assessment of the attributes of the core competences category.
Education 13 00900 g002
Figure 3. Assessment of the attributes of the usage core competences category.
Figure 3. Assessment of the attributes of the usage core competences category.
Education 13 00900 g003
Figure 4. Assessment of the attributes of the online teaching strategies category.
Figure 4. Assessment of the attributes of the online teaching strategies category.
Education 13 00900 g004
Figure 5. Assessment of the attributes in the ensuring positive online student experiences category.
Figure 5. Assessment of the attributes in the ensuring positive online student experiences category.
Education 13 00900 g005
Figure 6. Evaluation of the attributes in the assessment strategies category.
Figure 6. Evaluation of the attributes in the assessment strategies category.
Education 13 00900 g006
Figure 7. Assessment of the attributes in the creating digital content category.
Figure 7. Assessment of the attributes in the creating digital content category.
Education 13 00900 g007
Table 1. Provision of assessment category attributes.
Table 1. Provision of assessment category attributes.
Assessment CategoryDigital SkillOnline Courses That Teach This Digital Skill
No.%
Core CompetencesPhysical health10.62
Data protection8554.47
Identity management1811.11
Law2012.34
Storage21.23
Usage Core CompetencesSearch4628.39
Active participation5232.10
Collaboration159.26
Production and sharing4024.69
Creation and change3219.75
Online Teaching StrategiesUnderstand8451.85
Design and structure7445.68
Achieve2112.96
Digital delivery8351.23
Structured learning7848.15
Self-regulated learning5735.18
Ensuring Positive Online Student ExperiencesTop-down6439.50
Collaborative1811.11
Inspire169.87
Reflection1710.49
Knowledge gaps2314.20
Assessment StrategiesUnderstand1911.73
Formative1811.11
Summative2917.90
Scaffolding2414.81
Group working2012.34
Peer assessment127.41
Data analysis9759.87
Creating Digital ContentIdentify material4729.01
Modify material3521.60
Communicate value3722.84
Interactivity3521.60
Engaging static content2515.43
Table 2. Courses by coverage level for each of the assessment categories.
Table 2. Courses by coverage level for each of the assessment categories.
Assessment CategoriesLevel of Coverage by Courses
LowMediumHigh
No.%No.%No.%
Core Competences13985.802314.2000.00
Usage Core Competences11671.603521.60127.41
Online Teaching Strategies7546.306540.122213.58
Ensuring Positive Online Student
Experiences
14388.27169.8831.85
Assessment Strategies14690.12159.2610.62
Creating Digital Content11470.374427.1642.47
Table 3. Mean, maximum, and minimum of total scores of courses by platform.
Table 3. Mean, maximum, and minimum of total scores of courses by platform.
PlatformMean ScoreMax ScoreMin Score
Coursera24.025612
edX15.61482
Udemy16.83492
Table 4. Number of courses by coverage level for each platform.
Table 4. Number of courses by coverage level for each platform.
PlatformsLevel of Coverage by Courses
LowMediumHigh
No.%No.%No.%
Coursera2817.2884.9400.00
edX2414.8121.2300.00
Udemy9055.56106.1700.00
Total14287.652012.3500.00
Table 5. Correlation of assessment categories.
Table 5. Correlation of assessment categories.
Correlations
Core
Competences
Usage
Core
Competences
Online
Teaching
Strategies
Ensuring Positive Online Student
Experiences
Assessment StrategiesCreating Digital Content
Core
Competences
Pearson Correlation1−0.580 **−0.528 **−0.234 **−0.258 **−0.470 **
Sig. (2-tailed) 0.0000.0000.0030.0010.000
N162162162162162162
Usage Core CompetencesPearson Correlation−0.580 **10.591 **0.571 **0.293 **0.553 **
Sig. (2-tailed)0.000 0.0000.0000.0000.000
N162162162162162162
Online
Teaching Strategies
Pearson Correlation−0.528 **0.591 **10.572 **0.255 **0.530 **
Sig. (2-tailed)0.0000.000 0.0000.0010.000
N162162162162162162
Ensuring
Positive Online
Student
Experiences
Pearson Correlation−0.234 **0.571 **0.572 **10.1090.278 **
Sig. (2-tailed)0.0030.0000.000 0.1650.000
N162162162162162162
Assessment StrategiesPearson Correlation−0.258 **0.293 **0.255 **0.10910.266 **
Sig. (2-tailed)0.0010.0000.0010.165 0.001
N162162162162162162
Creating
Digital
Content
Pearson Correlation−0.470 **0.553 **0.530 **0.278 **0.266 **1
Sig. (2-tailed)0.0000.0000.0000.0000.001
N162162162162162162
** Correlation is significant at the 0.01 level (2-tailed).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Bitakou, E.; Ntaliani, M.; Demestichas, K.; Costopoulou, C. Assessing Massive Open Online Courses for Developing Digital Competences among Higher Education Teachers. Educ. Sci. 2023, 13, 900. https://doi.org/10.3390/educsci13090900

AMA Style

Bitakou E, Ntaliani M, Demestichas K, Costopoulou C. Assessing Massive Open Online Courses for Developing Digital Competences among Higher Education Teachers. Education Sciences. 2023; 13(9):900. https://doi.org/10.3390/educsci13090900

Chicago/Turabian Style

Bitakou, Effrosyni, Maria Ntaliani, Konstantinos Demestichas, and Constantina Costopoulou. 2023. "Assessing Massive Open Online Courses for Developing Digital Competences among Higher Education Teachers" Education Sciences 13, no. 9: 900. https://doi.org/10.3390/educsci13090900

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop