Accessibility Challenges in OER and MOOC: MLR Analysis Considering the Pandemic Years

Abstract

The review of state of the art on creating and managing learning resources and accessible Open Educational Resources (OER) and Massive Open Online Courses (MOOC) is a topic that cannot only consider formal literature. The evidence and lack of a measurement consensus require the inclusion of contextual information, corroborating scientific results with practical experiences. For this reason, this article presents a review of accessibility models, OER and MOOC, considering the gray literature to capture experiences and trying to establish a shared understanding of the terminology commonly used in research on virtual accessibility and its impact on higher education. The bibliographic review relies on analyzing articles and scientific publications related to the topic following the Multivocal Literature Review (MLR) format. The results of this review establish that it is possible to apply accessibility review methodologies with transversal actions in the creation and management of learning resources and MOOCs. The research is related to one of the seventeen sustainable development goals defined by the United Nations to ensure inclusive and equitable quality education and promote lifelong learning opportunities for all.

1. Introduction

The exponential growth of e-learning and virtual education has led to the accelerated development of teaching and learning resources in which we find a huge diversity in how they are generated and managed as well as student interaction and learning.

One of the seventeen sustainable development goals (SDG 4) defined by the United Nations is to “ensure inclusive and equitable quality education and promote lifelong learning opportunities for all” [1]. To achieve this goal, education must be of quality and accessible, in the sense of accessibility described above, because accessible education gives people with disabilities the opportunity to learn under equal conditions [2].

Establishing the relationship between the concepts expressed by terms associated with learning resources, learning objects, digital multimedia resources, is critical nowadays given the importance of their use in virtual learning environments.

Relevant research from Europe and Latin America provides guidelines for the creation and management of accessible learning resources. For the present study, experiences of Latin American and European institutions are considered. The review of articles reveals that several accessibility indicators involved in learning resources and Massive Open Online Course (MOOC) are related to Web Content Accessibility Guidelines (WCAG) compliance, usability, user experience, learning design, and quality standards.

The results of the research have shown that accessibility is transversal to the evaluation of educational resources and is considered within the tools and models to evaluate the methodology of a course and its technical aspects.

This study analyzes concepts and relationships of learning resources, virtual courses, and MOOCs from the accessibility approach, then it reviews related literature and research that answer the research questions, considering academic literature and gray literature. The objectives of this study are as follows:

• RO1: Provide an overview of the current status of initiatives in accessible learning resources and MOOCs.
• RO2: Identify good accessibility practices for the creation and accessible management of learning resources and MOOCs.

The article is organized as follows. Section 2 presents the background. Section 3 describes the research methodology. Section 4 provides an analysis of the results, while Section 5 contains a discussion and recommendations. Finally, Section 6 presents the main conclusion of this study.

2. Background

The concept of learning object was first introduced in 1994 by Hodgins, who defined it based on the intersection of three basic design principles: discoverability, reusability, and interoperability [3]. This premise of being easily reusable received wide acceptance by the scientific community [4]. Technological progress and the increased use of digital resources in the mediation of learning, caused the concept to evolve quickly. The definition of learning object by [5] as a “…entidad digital, autónoma y reutilizable, con una clara finalidad educativa, constituida por al menos tres componentes internos editables…” (…digital entity, self-contained and reusable, with a clear educational purpose, constituted by at least three editable internal components…), and the constant coincidence in the characteristics of identification, recovery, detectability, reusability, and interoperability; helped researchers to delimit the concept even further. However, it has also allowed to understand the variability and cultural evolution of its practice in virtual learning environments. As the concept evolved, the consideration of the legal aspects involved became paramount and reuse licenses were established. At the turn of the millennium and ever since the concept merged with the principles of open education, which gave rise to the notion of OER (Open Education Resource) (The origin goes back to a UNESCO Forum held in 2002. http://web.archive.org/web/20021019010259/http:/www.unesco.org/education/news_en/080702_free_edu_ress.shtml accessed on 14 February 2022). The term “open” represents a cultural change in the design philosophy, which can be best summarized in five major additional features established by Wiley and Hilton [6] as the 5Rs. Users of OER should be able to retain, reuse, revise, remix, and redistribute the content. Figure 1 points out the main characteristics of these 5Rs:

In the Paris Declaration of 2012 [7], UNESCO recommends member states to “promote quality assurance and peer review of OER. Encourage the development of mechanisms for the assessment and certification of learning outcomes achieved through OER”. The World Education Forum 2015 [8] in its Education 2030 Declaration states that “Information and communication technologies (ICTs) need to be harnessed to strengthen education systems, knowledge dissemination, access to information, effective and quality learning, and more efficient service delivery”. With this, the desirable characteristics of an OER are open access and author acknowledgement.

2.1. Massive Open Online Course—MOOC

The acronym MOOC was coined by Dave Cormier in 2008, to refer to the course “Connectivity and Connective Knowledge” offered by Stephen Downes, senior researcher at the National Research Council of Canada, and George Siemens associate director of research and development at the University of Manitoba [9,10]. The main feature of a MOOC responds to an open online course with massive student participation. The materials of a MOOC could be protected by copyright—xMOOC or use and create OERs under Creative Commons license—cMOOC [4]. Several researches seek to classify or evaluate MOOCs. Based on learning characteristics, [11] notes that an iMOOC could highlight its focus on individual responsibility, interaction, interpersonal relationships, innovation, and inclusion, or provide a learning experience marked by social interactions and participation considered as sMOOC [12,13]. The study [14] also adds the bMOOC as a recent combination of the advantages of the online MOOC and the need for face-to-face interaction. See Figure 2.

The relationship between OER and MOOCs converges in that the identification of their content is treated individually as a learning object or within a course [15]. Courses can be complete, open but with recognized copyright, usually from a university institution OCW; or open, massive, and online courses, such as MOOCs.

Accessibility is transversal in any of the defined components such as LMS, digital resources, learning objects, virtual learning environments, and/or virtual courses and everything that converges in MOOCs and OER.

2.2. Accessibility

Accessibility is related to several concepts that seek to facilitate the development or use of something in particular, among them: flexibility, customization, universality, usability, interoperability, reusability, and navigability. The standard ISO/IEC 24751-2, defines accessibility as “Usability of a product, service, environment or facility by individuals with the widest possible range of abilities” [16].

There are several considerations to take into account to make viable or prevent the access of a person with a disability, such as:

• Hearing impairment, oral, and/or sonorous expression generates problems in understanding the environment. Written comprehension is usually limited, mainly in those who lost their hearing ability before learning to speak. Captioning, use of pictures and diagrams, sign language are required.
• Physical disability: Those in this group are considered those who have problems with transfer, movement, or coordination in handling objects. Mouse movement and clicking may be complex or limited features.
• Visual impairment: Graphic information makes comprehension difficult if there is no textual-aural alternative. Small font sizes and inadequate contrast management affect navigation. Interaction is usually with a keyboard.
• Intellectual disability: Difficulties in understanding, assimilating, or retaining information. Interpretation of symbolic language and guidance may be complex. A simple vocabulary, simple syntax, and the use of headings, standardized pictograms, and lists of categories are fundamental elements in the understanding of users.

It is important to consider that there are people who, even if they do not have a permanent or temporary disability, have difficulties in accessing information, such as those who do not know about technology or do not have the optimal technology, temporary accidents, as well as older adults who are losing skills in interacting with a computer.

2.3. Related Works

The standards development establishes rules and requirements that must be met, making it possible for resources to be platform-independent, strengthening their interoperability, reusability, durability, updateability, scalability, among others. This generates standards for various areas related to learning resources and MOOCs.

Accessibility in e-learning is not only framed in technology and its interaction, it also requires feedback from the design of learning experiences for all, considering not only technology and pedagogy but also ethics [17].

Methodological proposals focused on the quality of virtual educational resources are based on ISO standards, establishing guidelines for applying ICTs in teaching. However, some studies [18,19] identify the lack of an accessibility methodology with a holistic and adaptive approach.

Standards such as [20,21,22] establish guidelines that are related to accessibility; however, the applicability and dissemination are still limited. In several Latin American countries, the use of standards of private organizations is not possible until they are considered official standards such as ISO, which is why the WCAG could not be adopted until 2012 when the standard [23] Information Technology—W3C Web Content Accessibility Guidelines (WCAG) 2.0 was created.

Some projects have been developed to favor accessibility in virtual environments. The shared experiences EU4ALL [4], ESVIAL [18], TILE, AEGIS, ACCESSIBLE [24], and OBBA in Brazil [25], to mention a few, highlight research and implementation efforts to favor educational inclusion. In parallel, the evaluation of learning resources and MOOCs generates proposals for models and standards to be applied, for which the accessibility criterion is considered relevant, but has not yet achieved a consensus of information.

Currently, universities face the challenge of providing quality education by strengthening the inclusion approach and addressing the high rates of exclusion, discrimination, and educational inequality. The creation of conditions for the development of education for all, which guarantees quality with equity, implies transformations in the educational system of HEIs, in their cultures, policies, and practices, involving an active and participatory manner evaluative processes that validate the efforts made. In [26], the Convention on Human Rights and its Optional Protocol states in Article 24: “States Parties recognize the right of persons with disabilities to education. With a view to realizing this right without discrimination and on the basis of equal opportunity, States Parties shall ensure an inclusive education system at all levels as well as lifelong learning”.

Regarding the main MOOCs platforms identified, it is established that out of a total of 12 (EDX; CUORSERA; UDEMY; FUTURE LEARN; UDACITY; MIRIADAX; NOVOED; UNED; KHAN ACADEMY; TUTELLUS; CREHANA; LYNDA), only four have accessibility policies. See

Table 1. Accessibility policies in MOOCs.

MOOC	Accessibility Policies
EDX	https://www.edx.org/es/accessibility (accessed on 20 December 2021)
COURSERA	https://learner.coursera.help/hc/es/articles/209818883-Pol%C3%ADtica-de-accesibilidad-de-Courserahttps://learner.coursera.help/hc/es/articles/208280056-Adaptaciones-para-estudiantes-con-discapacidades (accessed on 20 December 2021)
FUTURE LEARN	https://about.futurelearn.com/terms/accessibility-policy (accessed on 20 December 2021)
UNED	https://blogs.uned.es/unedabierta/canal-fundacion-once/ (accessed on 20 December 2021)

.

3. Materials and Methods

The review of the state of the art on the creation and management of learning resources and accessible MOOCs is a topic that cannot only consider formal literature; the evidence and lack of a measurement consensus require the inclusion of contextual information, corroborating scientific results with practical experiences. With this, the incorporation of Grey Literature (GL) within the structure of the review protocol with the Multivocal Literature Review (MLR) methodology based on Systematic Literature Review (SLR) [27].

3.1. Conducting the Review

The following research questions (RQ) were formulated based on the research objectives (RO) outlined above:

RQ 1:

How are accessible learning resources created and managed?

This question is posed by the diversity of existing accessibility practices in accessible resources.

To answer this RQ, relevant previous studies on accessible learning resources and the educational institutions that influenced their development were analyzed.

RQ 2:

Can accessibility be mainstreamed in MOOCs?

The question is posed to establish the phase in which accessibility is considered within the creation of a MOOC.

To answer this RQ, comparative studies of MOOC accessibility and the establishment of policies or standards applicable in MOOCs were analyzed.

RQ 3:

What models have been used to promote accessibility in MOOCs and learning resources?

The question is asked to establish models frequently employed when approaching accessibility in MOOCs and learning resources.

To answer this RQ, this study investigated the models employed on various platforms by identifying best practices, learning outcomes, and satisfaction.

RQ 4:

What are the challenges and opportunities that have been addressed in the creation and management of accessible educational resources and MOOCs with experience in the pandemic?

The question seeks to establish current and future research trends on the subject of accessibility in educational resources and MOOCs.

To answer this RQ, this study investigates the limitations of existing tools and systems related to accessibility in educational resources. It also summarizes and provides recommendations used in the pandemic.

The search string identified in this study is

(ACCESSIBILITY) AND (MOOC) AND (“OPEN EDUCATIONAL RESOURCES” OR “OPEN EDUCATION RESOURCES” OR “LEARNING OBJECT”).

For academic literature, the search engines used are:

ACM, SCOPUS, IEEE, ERIC, AND GOOGLE SCHOLAR.

For GL literature, GOOGLE and pages identified by their reports on the subject of accessibility and virtual education were used.

3.2. Study Selection Criteria

After completing the search string, the following criteria were considered as inclusion criteria in the academic literature (see Figure 3):

• Must have been published in the period from January 2013 to December 2021.
• It must be written in English or Spanish.
• The article must be related to accessibility strategies in educational resources and MOOCs.
• The article must respond to relevant research in the partner countries or of high connotation in accessibility issues.

As exclusion criteria in academic literature are considered:

• Papers that are not related to accessibility and experiences in virtual education.
• In Google Scholar, the 200 most cited (citation index >40) are considered.

In gray literature and Google, after fulfilling the search string, the following are considered as inclusion criteria:

• The first 100 searches.

As exclusion criteria in gray literature are considered:

• Commercial advertisements and images.
• Documents not related to accessibility and experiences in virtual education.
• Broken links or with access to purchase books.

Assessment Criteria for Study Quality and Result

Table 2. Quality question list.

Item	Assessment Criteria	Description of Checklist
QA1	Is the author’s recognition clearly identified or associated with a recognized organization based on the experience of the subject?	Yes. Authorship and expertise are clearly identified. No. The authorship data is not identified. Partially. Presents data, but does not specify evidence to support the experience.
QA2	Is the research methodology clearly identified?	Yes. It presents clear objectives and a methodological process supported by limited reliable references in a particular population or situation. No. It lacks identification of a research methodology. Partially. Presents a description of the proposed approach, but lacks reliable references or little delimitation of the topic.
QA3	Is the target’s support adequately described in an unbiased way?	Yes. The content of the source is discussed and supported by data. No. Lack of supporting data. Partially. It alludes to a discussion, however, the opinion is not impartial or is not supported by real data.
QA4	Does the source present a clear date of its elaboration?	Yes. The date is present. No. Does not present an elaboration date. Partially. It refers to a period but does not set a specific date.
QA5	Is there an unprecedented and significant contribution to the research?	Yes. The source provides innovation and reinforces or refutes current ideas on the subject. No. The font does not provide innovation or novelty. Partially. It reinforces current ideas but does not contribute something unique to the research.

establishes the questions to evaluate the quality of each study. In (QA1) the prestige of the author is evaluated, where it is detected that all of them comply with the identification of the author and experience in the area. The second criterion (QA2) evaluates the data collection and procedures that respond to a research methodology and it is concluded that 31 articles present clear objectives and methodological process supported by reliable references limited to a particular population or situation, 20 present a description of the proposed approach, but lack reliable references or scarce delimitation of the topic. The third criterion (QA3) examines the objectivity presented, it is concluded that out of 41 articles, the content of the source is discussed and supported by data, while ten allude to a discussion; however, the opinion is not impartial or is not supported by real data. The fourth criterion (QA4) evaluates whether the source presents a clear date of elaboration, to which 43 specify a clear date, seven refer to a period but do not establish a specific date, and one does not present the date of elaboration. The fifth criterion (QA5) evaluates whether there is an unpublished and significant contribution to the research, concluding that 34 articles contribute innovation, 14 articles reinforce current ideas but do not contribute something unique to the research, and three articles do not contribute innovation or novelty.

In order to determine how the publications in the academic literature studied are related, a statistical analysis was performed with four pairs of quality questions. As can be seen in Figure 4, part (a) compares the results obtained in questions QA5 (Novel contribution) and QA3 (Data support). In this sense, we can see that 27 and 24 publications have a maximum evaluation (1) for each question, respectively. Meanwhile, seven papers have a mean value (0.5) and three publications with a minimum value (0) for each question. On the other hand, part (b) compares the evaluations of the articles with respect to questions QA5 (Novel contribution) and QA2 (Methodology). In this case, it can be seen that 21 articles receive a maximum score in the two questions, having seven with the maximum combination for QA2—mean QA5 and vice versa and three articles with the minimum combination. This in general terms gives us a clear guideline of the quality of the publications that have been studied, and how they respond to the quality questions that have been previously established.

In the case of GL, the criterion of type is added for its quality evaluation and it is concluded that 12 articles are of the first level because they correspond to books and journals of scientific dissemination and specialized foundations and one is considered of the second level because it is a presentation. The same analysis was performed as in the case of academic literature. Figure 5 compares the results obtained in questions QA5 (Novel contribution) and QA3 (Data support). Here it can be seen how the publications have a lower score in the different combinations. For example, most of the articles (7) score high for QA3 and medium for QA5. This result is perfectly aligned with aspects of scientific rigor that occur more frequently in the case of academic literature. Meanwhile, it also compares the evaluations of the articles with respect to questions QA5 (Novel contribution) and QA2 (Methodology). In this case, it can be seen that the same phenomenon is present where the most important evaluations are the averages (7). However, it is important to note that in the case of the gray literature there are no zero evaluations, while in the academic literature there are such values.

4. Results and Analysis

The identification of relevant studies that were selected for this study are included in the analysis for subsequent discussion with respect to RQs.

4.1. Conducting the Review

The following research questions (RQ) were formulated based on the research objectives (RO) outlined above:

Out of all the studies, 43% of the selected articles were published in scientific journals, while 26% belong to high impact conferences, 29% are articles and papers on the internet, which constitute the majority of the grey literature. The results of the 51 studies analyzed in this research are presented in Appendix A. Each result is presented according to the corresponding research question. Figure 6 shows the distribution of articles according to the year of publication.

The volume of publications increased in 2016, a period in which local legislation establishes compliance with accessibility regulations so that virtual courses and research on the subject take a greater interest.

Table 3. Studies grouped by the answers to the research questions.

Research Question	Total Responses	Study ID
RQ1	19	S02, S03, S04, S06, S07, S08, S10, S11, S12, S14, S15, S16, S37, S38, S40, S42, S43, S46, S47.
RQ2	9	S02,S04, S05, S09, S17, S28, S29, S41, S45, S50.
RQ3	27	S01, S02, S04, S06, S08, S09, S10, S11, S13, S14, S15, S16, S17, S20, S21, S22, S23, S24, S25, S26, S27, S28, S29, S32, S36, S41, S43, S48.
RQ4	36	S01, S02, S03, S04, S6, S7, S10, S11, S12, S14, S15, S17, S18, S19, S21, S22, S24, S29, S30, S31, S33, S34, S35, S36, S37, S38, S39, S41, S44, S45, S46, S47, S48, S49, S50, S51.

shows the studies grouped by the answers to the research questions.

4.2. RQ1: How to Create and Manage Accessible Learning Resources?

Since their emergence, OER have proposed several categorizations based on usage rights, applicability to learning design, and their benefits, which has generated a changing conception [28]. The digital educational material will have the purpose of integrating a sharable collection in an educational environment, so they must comply with characteristics that facilitate their reuse, interoperability, and accessibility.

The creation and management of accessible learning resources cover several components depending on their typology. We can have documents, PDFs, presentations, spreadsheets, videos, audios, glossaries, images, forms, and other interaction elements. As there is a range of possibilities of accessible learning resources, some authors [25,29] suggest the integration of metadata according to profiles, adding accessibility features. The establishment of profiles feeds back to the learner’s needs and preferences [30]. However, determining a profile does not always respond to developing adaptations to a particular material, as stated by [4], when determining that students made more mistakes when requesting pre-established adaptations. It would be more ideal to provide accessibility information to serve all students.

The evaluation of the level of accessibility and usability according to [4] is proposed from the quantification of aspects of efficiency, effectiveness, and satisfaction, for which it proposes a methodology based on a joint evaluation of WCAG and ISO 9241-11. Some authors [31,32,33] argue the need for accessibility evaluation in resource management and the generation of new resources from them. Ref. [17] points out the importance of considering the level of understanding perceived by different user profiles according to the pedagogical objective, thus establishing that the level of accessibility should be measured as a package. The evaluation process requires scenario validation to illustrate and explore accessibility issues [34]. Literature states that the creation and management of accessible resources are composed of phases each in a constant process of continuous improvement [35]. The establishment of techniques, models, and methodologies in many cases responds to local, national, or country realities and their pedagogical context, so the subject of accessibility and inclusion covers various cultural contexts and awareness of the problem [36]. Literature also points out the importance of the organization and its technological infrastructure to ensure proper interoperability with assistive technology, as well as support for teachers in creating, publishing, and searching for accessible learning resources [37,38,39]. The design of inclusive resources requires the application of principles or strategies that support the teacher in the process of generating accessible material for their students. Some works [40,41] propose the application of Universal Design Learning (UDL). Several existing investigations on OER tend to focus on the evaluation of accessibility and usability, little attention has been paid to support the participation of students with disabilities in OER that according to the principles of UDL consider the participation of the student through of various representations [42].

As an answer to this question, it is concluded that the creation and management of accessible learning resources go beyond the technological use of tools. It is required to understand the synergy that must exist between technology and methodological design to establish a coherent creation and management that allows sustainability and contemplates standards and constant updating scenarios such as ubiquitous computing and its relationship with many devices and simultaneous systems.

4.3. RQ2: Can Accessibility Be Mainstreamed in MOOCs?

A study [10] established the life cycle of a virtual educational project requires constant feedback from successful experiences, so there is no common formula to follow [18]. MOOCs facilitate open education, so they are considered an innovative way to change education [28]. LMS virtual environments claim to comply with accessibility standards at the interface level; however, the evolution towards graphical and interactive webs increases complexity and interaction [29]. It is necessary to consider that the authoring tools must facilitate accessibility as part of the design and development of educational resources, considering accessibility checklists to evaluate the resources [43].

Another study [10] considers several challenges in the implementation of accessibility in MOOCs, among them the use of guidelines, specifications, and standards [4], the definition of architectures for MOOCs that take advantage of accessible content, providing educational material in various alternative formats, the definition of design patterns, testing compliance with accessibility requirements at the platform and content level, real user testing, sharing accessibility reviews at the public level, among others. Ref. [40] established a scenario-based approach to identify challenges and current practices in discussing possible solutions, considering that accessibility in MOOCs and learning resources, mainly depend on an institutional agenda of inclusion. Involving educators, content generators, and students feeds back into the development and evaluation of a course. Accessibility is not only legal compliance of guidelines, it requires an inclusive design analysis that allows to positively promote the needs and preferences of students [29]. MOOCs have the challenge of offering agile, fluid, and rhizomatic learning opportunities that favor inclusion and the common good [44].

Work [45] establishes that accessibility is not a priority parameter when establishing a MOOC, which is proven by performing 288 accessibility tests in eight recognized MOOC platforms, for which they use automatic tools and expert evaluation, establishing that more than 50% do not meet success criteria of levels A and AA. eDX and FutureLearn are considered as the best scored. From this research it is concluded that nine level A and three level AA criteria are less fulfilled by MOOCs and these are

Table 4. Accessibility criteria at level A and AA frequently breached in MOOCs.

Level	Criterion	Description
A	1.3.1.	Information and relationships: The information, structure, and relationships communicated through the presentation can be determined by software or are available as text.
A	4.1.2.	Name, role, value: For all UI components (including, but not limited to: form elements, links, and script-generated components), the name and role can be determined by software; the states, properties, and values that can be assigned by the user can be specified by software; and changes to these elements are available for consultation by user agents, including technical aids.
A	2.4.1.	Avoid blocks: There is a mechanism to avoid content blocks that are repeated on multiple web pages.
A	3.3.2.	Labels or instructions: Labels or instructions are provided when content requires user input.
A	4.1.1.	Processing: In content implemented using markup languages, elements have full opening and closing tags; the elements are nested according to your specifications; the elements do not contain duplicate attributes and the IDs are unique, except where the specifications allow these characteristics.
A	1.1.1.	Non-textual content: All non-textual content that is presented to the user has a textual alternative that serves the same purpose.
A	2.1.1.	Keyboard: All content functionality is operable through a keyboard interface without requiring a certain speed for each individual keystroke, except when the internal function requires input that depends on the path of user movements and not just the start and end points.
A	1.2.1.	Audio-only and video-only (recorded): For recorded audio-only content, an alternative is provided for time-dependent media that presents equivalent information for recorded audio-only content. For recorded video-only content an alternative to time-dependent media is provided or soundtrack is provided that presents information equivalent to the content of the recorded video-only medium.
A	2.4.4.	Purpose of the links (in context): The purpose of each link can be determined with only the text of the link or through the text of the link added to the context of the link determined by software, except when the purpose of the link would be ambiguous for users in general.
AA	1.4.3.	Contrast (minimum): The display of text and text images has a contrast ratio of at least 4.5: 1.
AA	1.2.5.	Audio description (Recorded): An audio description is provided for all recorded video content within synced media.
AA	2.4.6.	Headings and tags: Headings and tags describe the topic or purpose.

It is concluded that it is possible to mainstream the accessibility of MOOCs, for which it is necessary the involvement of several instances, both from the organizational and institutional part, as well as creators, designers, academics, and constant feedback from students. Constant practices of application and evaluation of accessibility contribute significantly to a culture of inclusion.

:

4.4. RQ3: What Models Have Been Used to Promote Accessibility in MOOCs and Learning Resources?

OERs and MOOCs need to be seen from the need to evaluate their quality, focused on their own dimensions of an educational training in its process and result. Ref. [28] states that, from an innovation perspective, MOOCs go beyond OER, since they facilitate processes and experiences.

Establishing models, regulations, standards on the subject of accessibility and virtual education, places us in a similar context and with it, a starting point that goes from the general to the specific. At a general level, we find WCAG with guidelines and recommendations on the subject of web accessibility [46], a requirement to be considered both in MOOCs and learning resources. It is in the interaction of the student with a virtual learning environment and its resources, which leads to seek solutions that effectively respond to various issues. Research and their proposals of models and techniques converge on similar points such as: decrease of barriers [34,47,48,49], quality assessment [4,50], user experience feedback [39,49,51,52,53], personalization and recommendation [11,38,54,55], and effective publication of accessibility information [24,56].

Europe’s experience in the field of MOOCs research and accessible resources is latent, as evidenced by the number of MOOCs offered and especially Spain, whose legislation regulates the mandatory compliance in public institutions [25]. With this, the legal structure is a fundamental requirement when guaranteeing accessible and quality virtual education for all [10,40,57].

Socialization, awareness, and the development of specific competences [41] in all actors [18,29] who contribute in an accessible virtual course, is an indispensable requirement to generate accessibility culture. The identification of roles and responsibilities converge for the sustainable implementation over time of a model that promotes accessibility in MOOCs and learning resources.

In response to the research question, it is established that the various models to promote accessibility in MOOCs and learning resources are based on regulations and standards that seek to ensure the didactic and technological effectiveness of resources, which contributes to quality processes considering accessibility as an evaluative parameter.

4.5. RQ4: What Are the Challenges and Opportunities That Have Been Addressed in the Creation and Management of Accessible Educational Resources and MOOCs with Experience in the Pandemic?

We are still going through the pandemic situation, so several educational experiences in e-learning have been generated on the fly, remembering that all students, whether they have disabilities or not, have the right to access an online educational environment designed to help them reach their maximum potential. Although the year 2012 was called the year of MOOCs, the pandemic marked another milestone, also the UNESCO OER recommendation for the implementation of its five areas encourages strengthening its use in education [44]. In the results report on the effects and consequences of the COVID crisis, it indicates that 65% of those consulted have dedicated time to training compared to previous reports whose percentage was reduced to 23%, pointing out that it is necessary to improve accessibility and diversity of topics of the training platforms, avoiding specific courses for people with disabilities [58].

At a general level, the research highlights the lack of dominant studies on the production of MOOCs and OER in developing countries, thus the experiences of educators and students overlap with global trends that do not reflect a significant contribution in access and training in online learning environments within structural constraints [33,59]. Even more so in a pandemic situation where not everyone was a user of a virtual environment and the demand for tutorials to access or register independently was imminent [60].

International policies and legal and organizational regulations [36] must be taken into account for the management of those involved [34,44,61]. The wide range of disabilities, the variability in learning, and their modes of interaction [34] suggest performing deeper analyses of the wide variety of assistive technology and their technical issues [24,48] fed back to the user experience by developing a holistic approach [51] and measuring the scope and subsequent benefits of those involved [30]. OER-based pedagogy is proposed as future research, for which, [42] points out the importance of collecting information on related instructional practices, where learning experiences are deepened.

There is a need to support models with more robust analyses of access, interaction, and feedback of students with disabilities [29,50,62]. However, under a virtual educational world in a pandemic, the contributions of accessibility, usability, UDL, present interesting aspects of analysis such as: providing different forms for involvement, consistent and regular feedback, and establishing spaces to analyze the self-efficacy of the student [49] could positively influence collective learning.

The collaborative approach from which OER and MOOCs are born leads to the joint pursuit of pedagogical and technological challenges to achieve quality-enhanced reconstruction [47,63,64]. It is necessary to evidence defined metrics that endorse methodologies [18] and reference international guidelines or instructions related to design for all [65]. The “open” spirit of MOOCs and OER should evidence accessibility from their creation and management [58,66] guaranteeing didactic and technological effectiveness within a continuous evaluative process.

The incorporation of intelligent systems could contribute to the evaluation of accessible resources and in the feedback of profiles and personalization from the user experience [37,56]. It is necessary to measure the impact and implement processes to identify accessibility barriers, with mechanisms that involve the participation of students in co-design and co-evaluation [43].

The efficient publication of accessibility information would facilitate an optimal search for resources according to learner needs and preferences [32,67].

In response to RQ4, we can say that the field of accessibility is extensive, so the associated techniques and standards need to be homogenized and socialized to generate a multiplier effect on the developers of e-learning resources. The pandemic crisis in many cases shows the lack of time for sufficient training in the adaptation of material and consideration of the most vulnerable groups. However, the opportunities generated to create new ways of teaching and learning through a virtual environment invite the creation of a new model in which people feel valued and welcomed. It is necessary to share experiences and educational material generated to extend its use through the appropriate use of metadata, so that the learning curve achieves a common language in the development of accessible digital resources that are easy to implement and search on the web.

4.6. Limitations

The present study presented limitations during the process and in its quest to answer the research questions. The selection of research keywords and exclusion criteria were mostly subjective. However, by obtaining a coefficient of 0.743 in Krippendorff’s alpha, an optimal level of inter-observer reliability was determined, thus guaranteeing a reduction of bias. The systematic review is based on databases of scientific literature and gray literature and is therefore not completely exhaustive. The omission of articles may also respond to the selected time period 2013–2019 and the consideration of Google as a search engine in the gray literature; however, the bias was reduced by choosing a set of databases covering the main disciplinary fields in which accessibility in MOOCs and OER can be addressed. This study, in its academic literature, focused on journal articles and conferences due to the fact that the topic is addressed in several scientific congresses covering developments in e-learning and accessibility. In the gray literature, it is oriented to search positioning.

Another limitation is the selection of documents written in English and Spanish, so trends in other countries and their current state of research may have been lost. However, it is considered that the systematic review process offers a good overview of the state of research on MOOCs and accessible OER, considering that several topics have been investigated in previous reviews, identifying various relationships in the creation and management of virtual learning environments and accessible resources.

5. Discussion and Recommendations

The present review of the state of the art aimed to provide an overview of the current state of research on the creation and management of accessible OER and MOOCs. The results showed a lack of applicability and data to support the current situation in Latin America; however, the experiences of European projects and regulations that endorse their sustainability, establish guidelines that could guide implementation processes in higher education institutions.

The application of processes that guide accessibility in virtual education still responds to subjective criteria that depend on local or institutional models of evaluation in virtual education and general guidelines. The studies also showed a lack of measurement of the impact on the applicability of accessibility in MOOCS and OERs from the experience of a significant sample of students with disabilities, as well as the satisfactory or unsatisfactory results of their teaching–learning process. Case reports are established with a limited number of subjects. The studies focus essentially on local experiences.

Future Directions

Through the findings, research perspectives, and the challenges posed to improve the field of accessibility in the creation and management of MOOCs and OER, it is relevant to explore the efforts generated to establish models that promote accessibility.

Although some authors [4,50] propose guidelines to mainstream accessibility, it is still complex to answer questions about models and good practices that cover the entirety of a virtual educational process and the inclusion of new features that improve learning design.

The advantages of an adequate implementation of accessibility in MOOCs and OER courses is still not a general domain knowledge, so concluded several authors [19,32]. The publication of accessibility information could favor the correct adoption of practices that generate a future direction that can focus on the effective search for courses and educational resources that respond to the needs and preferences of a student with a disability, considering that the efforts generated to create accessible educational material enriches the universality of education.

The creation and management of accessible educational resources, supported by the degree of satisfaction of a student when interacting, achieve synchrony with resources that can be reused. Undoubtedly, education generates valuable educational material that could favor repositories and enrich the educational process in a virtual environment as it knows no borders and has open availability.

The systematic review places us in the impact of accessible MOOCs and OER. The publications analyzed show an interest in strengthening their implementation. An important need is established in the generation of guides, tools, and techniques that promote their development and strengthen their evaluation and impact in developing countries.

Based on the literature review, it is established that there is no accessibility evaluation model for OER and MOOCs. It is considered that it is necessary to establish accessibility guidelines to assist in the elimination of barriers, so we propose as future work, to explore and investigate the strengths and weaknesses of accessible educational resources, compatibility with assistive technology, and the implementation of guidelines that favor the training of accessible OER to generate a culture of inclusive design.

Although the year 2012 was called the Year of MOOCs [68], now with the pandemic, 2020 marked another milestone in that MOOCs have drastically changed the way people learn and how to access knowledge [44], which entails constant feedback from the community and diversity in learning.

It is necessary to elaborate and implement legal regulations, especially in developing countries. The importance of demonstrating the social and institutional benefits of implementing accessible MOOC platforms and content contributes to optimal quality assessment. More research is needed on the needs not only of specific disabilities, but in the context of the learning experience, platform design, maintenance, and inclusion of new features.

6. Conclusions

Our study was developed in the framework of establishing technical guidelines for the creation and management of accessible learning resources, OER, and MOOCs in the context of Latin American HEIs, with special emphasis on the partner universities of the EduTech project in Ecuador and Mexico.

Despite the limitations of this research, since it is based on data from scientific literature and gray literature in a specific timeframe, we consider that the potential bias was reduced by covering the disciplinary fields of informatics, education, and information search and applicability experiences in developed countries whose legislation supports accessibility in virtual education. We also consider the use of a rigorous methodology in the development of the systematic review, in addition to the analysis of agreement reached among the reviewers, which achieved an optimal level of reliability.

Three main conclusions can be drawn. First, there is a scarcity of information and indicators regarding the evaluation of accessibility of virtual educational resources and courses, which is endorsed by people with disabilities. Moreover, the available studies tend to focus more on design recommendations than on evaluating the effectiveness of their implementation and improvement process. In second place, the use of accessibility standards is subjective, in several cases, it responds to evaluative models that, although they consider accessibility as an evaluative metric, it is inconsistent to reach a common implementation process, especially with courses that go beyond the legislation of a country. Lastly, there is a lack of references that establish an important sample of students with disabilities, their follow-up, monitoring, and process and improvement in learning design, which requires more time to obtain reliable data. The different responses that emerged during the review process allowed us to identify promising directions for future research on the subject.

In conclusion, this study presents the various models, standards, and tools used for the application of accessible learning resources and MOOCs, information that may help other researchers to consider incorporating accessibility issues in the creation of resources with “open” features. The publication of accessibility information in educational resources, learning objects, and MOOCs bears great influence on the effective response of personalized search engines according to user needs and preferences.

Finally, it is concluded that, although the contributions throughout history have generated norms and regulations that have motivated research on the subject, there is a lack of adequate implementation and frequent use for its application, especially in developing countries. The information from quantitative, qualitative, or mixed studies is insufficient to determine the impact on students with disabilities at a general level, and much more post-pandemic teaching is expected, so the generation of policies and audit processes will be necessary to develop a culture of continuous improvement that involves and commits all the actors within an educational project that considers diversity.