An Evaluation of Open Source Adaptive Learning Solutions

Abstract

The education sector has never been so shaken up as much as this past year. COVID-19 has imposed new rules. Several countries were forced to switch overnight from a traditional educational model to a full eLearning one. Like most other countries, the Moroccan government decided to promote distance learning by implementing several initiatives, though they remained at an embryonic stage. To contribute to the movement of transforming the national educational landscape, we aimed to develop a solution that will leverage the technological advances in this field and influence the ways students learn. This will be possible by providing learners with the latest features enabling online and adaptive learning modes. Hence, the purpose of this first study is to provide an empirical evaluation of the existing open source Ed-tech projects, which will serve as the basis for the development of our global adaptive eLearning solution. Unlike existing work, which is based on literature reviews to compare the existing adaptive eLearning platforms, we have used the OpenBRR assessment methodology as a comparison methodology due to its flexibility and ease of use. This work will help us to understand the concepts of adaptivity in education. It will also describe the most popular open source Maturity Models as well as provide a clear idea about the differences between these Ed-tech open source solutions.

1. Introduction

The last year has been a turning point in the field of education. In fact, the COVID-19 pandemic has pushed several governments to adopt remote learning as the new method for students’ education. Schools were obliged to turn to eLearning solutions and to adapt to this new teaching paradigm to remain in the race [1].

Among the multitude of educational solutions, the ones that stand out from the crowd are those encompassing the adaptive learning functionalities. These represent an ambitious objective of innovative pedagogical systems, as they describe the ability of a solution to propose the most appropriates learning paths and assessment based on learners specificities, namely, initial knowledge, learning objectives, learning styles, personal traits, and assessment results.

Gartner Inc. defines adaptive learning as the new methodology that transforms the traditional approach to teaching students, by recommending contents based on their produced data.

Exploring eLearning scenarios in national educational institutions has never been a priority. It is only after the Moroccan government had required eLearning during lockdown that we started seeing some shy initiatives in the form of live streaming courses, but nothing similar to courses specifically designed for eLearning. Once adopted, a new dilemma has arisen among the different stakeholders [2].

Conversely, in Europe and the US, educative institutions have been using adaptive eLearning solutions for more than 10 years. Deloitte confirmed this in a recent study in 2019 [3], and added that AI-based education products have been widely used, and accepted by all categories of western countries’ users. In the same study, Deloitte emphasized the increasing role that adaptive learning solutions will play in response to changing learners’ requirements. In the same context, Kamal Kakish and Lissa Pollacia [4] have led a study consisting of the analysis of the impact of the implementation of adaptive eLearning solutions on learners’ outcomes. The study spanned three years (from 2014 to 2017), and showed that students’ exam scores and pass rates improved by almost 10%.

Moreover, the developments in the field of big data and the proliferation of new artificial intelligence (AI) methods, especially machine learning, have tremendously accelerated the adoption of these intelligent educative solutions by several companies and schools [5,6].

Gartner defined adaptive learning as the process of dynamically adjusting the way instructional content is presented to students based on their responses or preferences. According to the research firm, by 2025, more than 60% of K-12 organizations will be experimenting with or fully deploying adaptive learning features in their offerings [7]. The adaptive solution can include features such as [8]:

• Adaptive learning path proposition;
• The ability to address knowledge gaps as needed;
• The creation of better learning design, and the leverage of research to improve learners’ retention;
• Assessments for measuring growth.

In order to fit the Moroccan educative ecosystem, we have identified additional fundamental capabilities that should be included in the educative solution, in order to better support the learning process from its starting point to the end. We can quote:

Content creation: The solution should enable the creation of different kinds of content;

Students’ performance tracking: The solution should be able to provide teachers with information related to students’ characteristics, behaviors and learning progress. This can be possible through the combination of a Learning Management System (LMS) and an adaptive learning engine;

Real time adaptivity: The solution should be able to process learners’ data in real time and act consequently on students’ recommendations;

Assessments feedbacks: The solution should enable different kinds of learners’ assessments and provide appropriate feedback on a regular basis;

AI-based techniques: The solution should leverage the latest techniques in AI to analyse students’ data and make the decision to change the sequence of tasks and lessons accordingly;

Languages: The solution will support Arabic and French languages, as it targets the Moroccan students as a first step;

Design and architecture: The solution must be scalable, modular and open, exposing standard APIs, enabling integration with third-party providers;

Offline capabilities: The solution has to enable users to continue their learning activities even in non-connected areas.

2. Motivation

In 2016, Morocco fourth graders ranked 47th out of 50 participating countries in the Progress in International Reading Literacy Study [9]. Children are struggling to obtain an adequate education that will enable them to acquire the necessary knowledge and skills needed to support lifelong learning and employment. The major challenges encountered by the Moroccan children in order to receive a quality education can be summarized as follows:

• Overcrowded classes with more than 41 student per class;
• High teacher absenteeism;
• Multi-lingual environment at school;
• Multi-grade classrooms due to the lack of qualified teachers and educational structures;
• Scarcity of educational resources in Arabic, French and Berber, which are the most spoken languages in the country.

The consequences of the above factors are, first, the increase in students’ early school dropout and second, the weakness of the learners in the fundamental elements of reading and problem-solving. Indeed, the Organization for Economic Co-operation and Development Programme for International Student Assessment (PISA) has undertaken a study in over 79 countries in 2018 [10], consisting of the assessment of students’ knowledge level in several disciplines: mathematics, science and reading. The results confirmed the low performance of Moroccan students over all the assessment axes.

In order to remedy the situation, medium and high income families resort to personal tutors, but it turns out that this is not a viable solution due to the high cost of this kind of service. Parents also tried the option of enroling their children in some eLearning curricula, but they faced several difficulties when seeking appropriate courses, given that the majority of training is delivered in the English language, which is not taught until high school.

The Moroccan government, with the help of the United Nations, has undertaken several initiatives in order to redress the educational sector. This has been carried out through the improvement of schools’ personnel capacity, by providing training related to the best teaching practices, and by delivering some educational resources as learning materials. However, exploring eLearning scenarios in the national educational institutions has never been a priority. It is only after the government required eLearning during lockdown that we started seeing some shy initiatives in the form of live streaming courses, but nothing similar to courses specifically designed for eLearning. Once adopted, a new dilemma has been raised among the different stakeholders [2]. We quote:

• The efficiency of this eLearning mode versus the lack of adequate tools and the untrained teachers;
• The takeover of the educators’ role by parents when the latter are not prepared;
• Unavailability of continuous internet connectivity for a large number of students;
• The usage or not of blended learning which is an educational technique that combines the usage of various academic sources and modes to provide an optimal learning experience [11].

Our study will be the first part of an ambitious project that consists of developing an educational solution that will tackle all the previously quoted limitations. This solution will provide Moroccan learners with the appropriate environment, coherent with their culture and languages, and using the most recent AI-based techniques, to facilitate the acquisition and the anchorage of knowledge. This project will target all the categories of K-12 students.

This study will allow readers to have a macroscopic view of what adaptive learning solutions are. It also provides an empirical evaluation of the existing open source Ed-tech projects, which will serve as the basis for the development of our global adaptive eLearning solution.

To choose a robust, scalable and flexible open source solution (OSS), we need to select good quality, free components with an active community. To do this, there is a variety of assessment methodologies, namely, QSOS, which is sponsored by Atos Origin; OpenBRR, created by Intel and O’Reilly [12]; open source Maturity Model created by Cap Gemini [13]; and the qualitative weight and sum approach (QWS) [14]. For our study, we opted for the OpenBRR methodology, which offers a basic framework for OSS comparison and offers the desired degree of flexibility, allowing us to integrate the specificities of the Moroccan educational context during the process of evaluation.

The remainder of this article is organized as follows. Section 2 presents a description of an adaptive eLearning solution. Section 3 shows the results obtained through the comparison of the different OSS. Finally, Section 4 concludes the article and suggests future work.

3. Adaptive Learning at Glance

At the end of the last century, several educational institutions and businesses embraced adaptive eLearning solutions. NASA introduced the use of adaptive eLearning technologies for multiple purposes. In 1988, [15] led a study aimed at developing an expert system that can perform several diagnostic chores.

These systems have been fed by a set of data as input, and produced diagnostics and recommendations as output. During the same year, the University of Montreal organized the first international conference on Intelligent Tutoring Systems. Five years later, in 1993, the first world conference on Artificial Intelligence in Education was held in Edinburgh, UK.

At the beginning of this century, there was a rise of adaptive educative solutions such as Knewton that fostered the adoption of this technology in the educational industry. In 2019, the U.S. Air force embraced adaptive technologies to train and educate their pilots (accessed on 10 October 2022) (https://www.af.mil/News/Article-Display/Article/2056415/air-force-beta-tests-adaptiveLearning-platform-in-basic-military-training/).

Today, more than 100 Million users have experienced working with an adaptive eLearning solution [3]. In addition, the COVID-19 pandemic context has exponentially accelerated the rate of adoption. Many Learning Management System (LMS) and Content Management Systems (CMS) providers started integrating adaptive learning modules in their educational solutions, to provide their customers with the most complete and suitable learning experience. Several types of adaptation have been developed, but the most satisfying ones were based on artificial intelligence methods.

3.1. Adaptive Learning Framework

Based on relevant studies in the literature, we found that adaptive eLearning systems are different when it comes to the level of detail and design. Nevertheless, the majority of these systems are built around three main components [16]:

Content Model: Includes the topics that the student needs to learn. We can assimilate it to the objectives the learners need to reach during the learning experience. It refers to the skills or competencies that the student should acquire once the training is complete [17];

Learner Model: The learner profile. It contains information about the learners’ characteristics: learning objectives, learning interests, learning preferences, prior knowledge and background knowledge, as well as several other bits of useful information needed to adapt the learning experience [18]. The model can also encompass learners’ personal traits, such as learning style and cognitive characteristics (see Figure 1). Reference [19] asserted that the personal traits are the criteria most frequently used during the implementation of adaptive eLearning solutions.

Instructional or adaptation Model: Refers to the pedagogical aspects. It manages the choice of learning materials that are proposed to a learner during his learning journey. This model collects the information from the content and learner models in order to produce unique feedback that helps students to progress (see Figure 2).

In a few research papers, the researchers mentioned a fourth component belonging to adaptive eLearning systems, which is the ‘User Interface’. The latter plays the role of the intermediary between the learners and the other modules. The learner has access only to the interface, while the other modules run in the background [20].

Building an end-to-end adaptive Ed-tech solution requires, in many cases, complementing the above-described framework with additional modules to be able to provide the users with all the features needed to build, manage, use and reuse learning contents. We can quote:

• Authoring module, which is a piece of software that enables educators to create interactive courses designed for students to engage with using a computer;
• Assessment module, which is a tool used generally for measuring or determining a student’s academic abilities, skills, and proficiency in a given topic area. It serves three main purposes: the classroom, guidance and administrative;
• Collaborative module, which is a tool enabling learners and teachers to exchange with each other through interactive discussions and online collaboration activities, as well as to share electronic resources;
• Tracking and reporting module, which constitutes of a set of features that enables the tracking of the learners’ progress so that teachers can ensure learning outcomes are being met.

Nowadays, all those complementary features are packaged and provided as convergent solutions called Learning Management Solutions (LMS) [21]. This latter are software that enables the administration, tracking, reporting and delivering of courses and lessons using one simple web-based interface. The appearance of LMS dates from the beginning of the 1990s with the creation of the first ever LMS by SoftArc in 1990. At that time, LMS were designed to cover only a very limited scope of functionalities such as uploading and downloading files, reading news bulletins, sending messages on chat or public message boards and playing text-based games. Today, many LMS suppliers have embraced the digital transformation shift and have enriched the scope and coverage of their solutions by including many new features and capabilities such as gamification, learners’ segmentation, multi-device compliancy, multi-media content support, feedback from trainees and live collaborative tools. These convergent solutions have considerably simplified the implementation of eLearning projects by reducing the number of complex integrations between heterogeneous modules.

It is important to mention that an important number of Ed-tech’s solution providers have launched cloud versions of their products in order to overcome the high price of implementation. This is possible by leveraging the cloud advantages in terms of providing virtualized resources that are dynamically scalable, secure, fault tolerant and cost effective [22].

3.2. Adaptive Learning Approaches

Adaptive learning solutions vary greatly in features and adaptation strategies; the most suitable adaptive learning solution is directly tied to the user’s requirements. Sometimes, a simple personalization with a low cost can generate the value that the customer is looking for. In other cases, the personalization should be implemented at high scale in order to cater to the learner’s requirements.

Below, we will outline the different types of adaptive eLearning solutions found on the market nowadays, which are linear systems, macro-adaptive systems and micro-adaptive systems.

Linear systems represent the classical solution that proposes the same sequencing of content whatever the learner’s category. The disadvantage of this type of solution is that when a student has already mastered a part of the training, he or she is forced to go, a second time, through all the training’s stages, which can affect the student’s motivation.

Macro-adaptive approach. According to F. Mödritscher et al. [23] the macro-adaptive approach consists of producing personalized propositions based on the analysis of a few variables, such as learners’ goals and learners’ initial knowledge. In this approach, the learning recommendations are made by the system based on an analysis of the user’s data and before the start of the course [24].

Micro-adaptive approach. This approach enables the diagnostic of students’ needs on a micro-level by analysing their specific learning needs in real-time and then providing precise recommendations. Unlike the macro-adaptive approach, this approach is dynamic due to the continuous use of updated users’ data, namely, reactions towards courses as well as behaviors such as response errors, time to answer quizzes, and personal traits. The micro-adaptive systems can, in turn, be broken down into three subcategories: rule-based, preference-based, and algorithm-based. Rule-based systems consist of programming a set of rules, which will dictate the behavior of the solution with respect to each user’s behavior. The preference-based approach is related to the analysis of the learner’s preferences as he or she progresses on the learning journey, in order to propose precise instructions accordingly. The algorithm-based approach is related to the use of AI-based algorithms to determine the optimal modules’ sequencing for each learner, at any point in time. The data generated by the learners are analysed in real-time to deliver content accordingly. This last approach can be broken down into three other sub-categories, which are repetition or memory management, adaptive assessment, and integrated learning networks.

• Memory management consists of reviewing the topics to be anchored at increasingly large time intervals. This is particularly the case with the Leitner system [25]. In this approach, we seek to optimize anchoring according to the forgetting curve. The more the learner responds to reminders, the more solid his memory is and the more the reminders can be spaced out over time.
• Adaptive assessment [26] consists of efficiently adjusting the difficulty degree of test items, depending on the answers of the learners to a specific bank of questions. The assessment systems provide insights about a student’s acquisition of knowledge and skills, as well as the learning trajectory of a student during their learning journey.
• Integrated learning networks consist of gathering, in the network, all the learners’ data (profile, personal traits, learning objectives, assessments results) and building relationships between them. Algorithms are running continuously to analyse data in real-time, and providing the student with the best next content. Through this approach, the user receives the most appropriate customization at any point of time during their learning journey. This approach guarantees the highest rate of personalization with the highest user satisfaction (see Figure 3).

4. Literature Review

In recent years, a great need for adaptive learning has arisen and has attracted the attention of many researchers. Therefore, a great deal of effort has been devoted to reviewing adaptive eLearning systems. Here is an overview:

The paper of [27] provided a detailed comparative user experience (UX) assessment of two platforms used to implement large-scale open online courses: Open edX and Moodle. The results show that Open edX outperforms Moodle in almost all UX aspects ( e.i; task success, task time ). However, the choice of platform used by an institution is influenced by several factors of a non-technical nature, such as student proficiency with eLearning platforms, available financial and human resources, as well as delivery deadlines.

The study of [28] intended to examine factors that predict the use of LMS by higher education students at King Saud University during the COVID-19 pandemic in addition to performing a literature review of previous studies on the topic from the time of the pandemic. The results demonstrated that LMS was an effective and interesting source of learning during the COVID-19 pandemic. According to students, it is a committed and fruitful technique for learning with sustained interaction that supports quick administration and the use of remote learning. No matter the constraints of time, students may easily access the educational materials.

Ref. [29] described and compared the existing plugins used to import questions into Moodle, classifying them according to the necessary computing resources. They also outlined their developed open source plugin FastTest PlugIn, recently approved by Moodle, which is a promising alternative to mitigating the detected limitations in analyzed plugins.

Ref. [30] examined a snapshot of written LMS policies from twenty universities in four countries and identified seventeen elements that could be included in a policy design template. These elements were classified into six policy categories: accounts, courses, ownership, usage, support, and protection. In addition, three other qualities of LMS policy statements were also established: standalone comprehensibility, platform-neutral statements, and contemporary relevance. After an Overview of the Common Elements of Learning Management System Policies in Higher Education Institutions, the results showed that Canvas was the most mentioned LMS platform in the examined policies. The next most featured LMS was Moodle.

In order to develop a new adaptive eLearning system, the authors of [31] compared the five most popular LMS tools, namely: Moodle, EduBrite, TalentLMS, Edomodo, and Sakai. They found that Moodle is one of the best alternatives for implementing the conceptual model proposed because it is open source and has all the important features for implementing an adaptive learning environment.

The objective of [32] was to propose the implementation of an adaptive and personalized eLearning system that is based on open source software and technologies. The authors first reviewed the traditional Learning Management Systems and existing adaptive eLearning systems (AES). They concluded that a combination of the advantages of modern AES, such as adaptability and personalization, with the key features of traditional LMS, which are integration and re-use, is necessary. This is important for developing an efficient and open learning platform.

The authors of [33] evaluated the use of the three LMSs—Schoology, Moodle, and Atutor—and the three CMSs—Drupal, Joomla, and WordPress—in higher education regarding the technical and educational features provided. They concluded that, compared to CMS, LMS platforms provide all the necessary modules for supporting the learning process, but due to their predefined modules, they provide certain capabilities and limited configuration options.

The study of [34] compared the identified and significant aspects of many widely-used LMS, including Moodle, Blackboard, and Canva. They discovered that Moodle is far superior to the other LMS and is the best option for use as an eLearning LMS.

5. eLearning Open Source Solutions Evaluation

In this section, we will describe the results of the conducted comparative study of a list of digital eLearning solutions selected to choose the open source product that will be the core of our future solution. However, before starting the OSS assessment, we undertook an adaption work to tailor the OpenBRR template by including or removing criteria in order to fit perfectly to our context.

The implementation of OpenBRR Methodology is composed of four phases:

• Preliminary filtering step: The user starts by performing a first rough evaluation of the OSS candidates for selection. He eliminates those that diverge clearly from their target, and keeps only those that seem to correspond to the user’s needs;
• Target Usage Assessment: The OpenBRR methodology proposes a template containing 12 categories, each one of those is in turn constituted of set of metrics. The assessor must assign a percentage to each category so that the sum of the percentages of all categories is equal to 100%. Then, he has to go down to the metrics level and assign percentages to each measure within each category, so that the sum of the percentages of the metrics within the same category is equal to 100%;
• Categories’ rating: This is the phase where the assessor gathers the necessary data to evaluate each metric. The scoring of the latter is based on a ’1’ to ’5’ scale (’1’ means unacceptable and ’5’ excellent). Weighted scores are computed afterward;
• Final score computation: The OpenBRR score is computed based on the previously computed categories’ ratings.

Figure 4 describes the different steps taken to evaluate the Open Source Adaptive Learning Solutions studied with OpenBRR.

The result of the adaptation work is shown in

Table 1. OpenBRR tailored assessment template.

Category	Sub-Catgory	Metric Description
Functionalities	File imports	Ability to upload files such as: PDFs, JPEGs, presentation files, word documents.
	Multimedia content	Support of multimedia content, such as video, audio, animation.
	Lesson library	Availability of pre-made lessons that students or teachers can select.
	Administrative dashboards	Availability of visual dashboards enabling insights on students’ performances.
		Ability to customize the reporting.
		Ability to export the dashboards into reports.
	Assessments and Quizzes	Availability of pre-made assessments.
		Ability to assign assessments to students to measure their understanding of specific courses.
	Device control	Ability to control the student devices during the training session.
	Audience feedback	Ability to set up a live poll to have realtime feedbacks from students.
	Gamification	Availability of gamified elements or content such as friendly competitions or the granting of badges at courses’ completion.
	Blended learning	Ability to provide a mixture between online courses and traditional learning.
	e-Commerce functionality	Enabling the integration with e-commerce solutions to be able to sell courses.
	Social and collaborative	Availability of collaboration tools: chat, forum, e-mails, files exchange, wiki, glossary …etc.
	Offline learning	Ability to capture offline assessment results. Ability to provide offline training.
	Authoring tool	Enabling content creation and delivery.
Adaptive features and capabilities	Adaptive eLearning	Adaptive functions based on expert rules.
		AI based adaptive eLearning.
		Availability of adaptive assessment features.
Operational Software Characteristics	Usability	Well-designed and intuitive user interface.
		Required time for preparing and installing the open source software.
	Security	Number of security vulnerabilities (moderate, critical) during the last 3 months.
		Availability of data related to security subjects (web page, wiki).
	Performance	Availability of Performance Testing and Benchmark Reports.
		Ease of performance tuning and configuration.
	Scalability	Availability of reference architecture and deployment procedures.
		Solution designed to be highly scalable.
	Portability(Device supported)	The solution supports multiple devices: Laptops, mobiles, tablets.
Service and support	Community support	Availability of an efficient and free community support.
	Paid support	Quality of professional support.
Software Technology Attributes	Architecture	Availability of 3rd party Plug-ins.
		Capability to integrate with external Service through public API.
	Quality	# of minor releases in past 12 months
		# of major releases in past 12 months
Documentation		Availability and accessibility of various product documentation.
Adoption and Community	Adoption	The volume of real world product deployment. Average volume of general mailing list used to get free help.
	Community	Number of unique code contributors in the last year.
	Development team integration	Difficulty or ease to enter the core developer team.
Languages	Morocco spoken languages	Support Arabic and French
Compliancy	Educational standards	Compliancy with eLearning standards: SCORM, AICC, xAPI, cmi5, or IMS.
License	Type of licencing	Strong/Weak/Non Copyleft License

.

5.1. OSS Candidates Panel Description

Our initial objective was to identify open source solutions offering adaptation features in order to model them according to the Moroccan context, as well as to develop a complete adaptive eLearning solution in accordance with the reference model described in this article’s introduction. However, after a first literature review, it turned out that the majority of the solutions designed especially for ”adaptive eLearning” were not global, and require several integrations with complementary modules, notably LMS and CMS, to be able to cover all teaching aspects, namely, ‘Authoring Tools’, ‘Quizzes and Assessments’, ‘Gamification’, and ‘Blended learning’.

Therefore, we decided to broaden the research spectrum in order to integrate into the panel OSSs eLearning solutions that are mature on the market even if they do not encompass any aspect of adaptivity. The goal is to be able to enrich them subsequently with the personalization and adaptation functionalities, thus enabling end-to-end eLearning experiences. The list of identified open source software for our comparative study is composed of Moodle, Canvas, Open Edx, Sakai, Claroline, Totaralearn, Atutor, Chamilo, Scale, Alosi, Grapple, Concerto, COFALE, DotLRN, Lon-Capa, ILIAS, Opingo, Kolibri, PointSquare, Education Algorithms and Format.LMS. Since we have opted for the OpenBRR maturity model, a first level of product filtering was carried out in order to keep only, for the assessment, the most relevant solutions to our context. The result of this activity was the exclusion of several candidates from the initial panel (

Table 2. Discarded solutions.

Name	Comments
Decals	Scarcity of the documentation. Last update in 2016.Very complex installation procedure.
PointSquare	Very small ecosystem. Documentation scarcity. Institutional website out of service. No clear evidence related to the success of the concept.
Education Algorithms	Last update performed in 2014.
Grapple	Deprecated solution.
Cofale	Unmaintained solution. No documentation available. Inaccessible Websites.
Claroline	No support of Arabic content.
Atutor	Unsupported solution.
SCALE	New tool: 2 years old, with small community. Unknown rules of the adaptive engine.
Alosi	Very small ecosystem. New solution created in 2018. Still under testing.
Opingo	Arabic language not supported. Assessment features not covered.
Format.LMS	Not designed for educational system. Arabic language not supported. Authoring features not covered.
DotLRN	Very limited community contributions.Arabic language not supported.
Totaralearn	Not designed for educational ecosystem.
Concerto	Restricted only to assessments. Lacks of many other important educational components, such as recommender engine, authoring tools.
LONCAPA	Very slow deployment cycle. Last version is 2.11.2 released in June 12 2017. Arabic language not supported.
Kolibri	Limited number of functionalities. Needs to be integrated with a multitude of complementary tools to provide an end-to-end learning experience.

and

Table 3. Shortlisted solutions.

Name	Comments
Moodle	Benefits: Ease of use. engaging content, communicating and collaborating with peers, dashboard, self-reflection and gamification. Inconvenient: Old style UI.
Canvas	Intuitive and ergonomic UI. No differentiation features included, only standard ones.
Open Edx	Modern LMS worldwide - easy to use, easy to manage.
Sakai	Flexible and lack of intuitiveness.
Chamilo	Easy to use and to manage. Ergonomics and effective. Default configuration is not optimal, and the process to modify it is not straightforward. Needs to develop the mobile interface.
ILIAS	Complex software. Hard to use for new users. The user interface style quite outdated and needs to be reviewed.

).

5.2. Shortlisted Solutions

After the filtering step, we kept only six eLearning solutions, namely Moodle, Canvas, Open Edx, Sakai, Chamilo, ILIAS.

The Modular Object-Oriented Dynamic Learning Environment (Moodle) [35] is the most popular learning management system that caters to the needs of the majority of schools and universities. Its modular design based on different plugins provides a high degree of flexibility when conceiving a learning experience. Accessible through different devices, Moodle provides users with a multitude of features ranging from course design to student assessments. This includes sharing learning materials, managing access to learning materials, updating course content, automated assessment, managing work submissions, communicating with learners and learner tracking.

Canvas LMS is an open source LMS developed by the company ‘Instructure’ under AGPLv3 license. The “standard” version is widely used in the USA and targets higher educational institutions. The K-12 version was built for primary and secondary schools. We can distinguish also the Canvas Network, which offers free MOOCs (Massive Open Online Courses) [36] on various subjects (science, history, ecology … etc.). Finally, Canvas Bridge is the Corporate version of the LMS.

Created by MIT and Harvard University, Open Edx is an open source platform for creating, delivering, and analyzing online courses. It is the platform that powers Edx courses. Open Edx was initially created to meet the educational requirements of several institutions and businesses, enabling them to offer engaging, end to end, educational experiences. The platform is built around four main components: (1) Open Edx LMS, responsible for managing all the aspects related to courses publishing, teams management, and grades edition; (2) Authoring tool, responsible for content creation; (3) Insights module, responsible for tracking learners’ performances; (4) E-commerce module, whose role is to enable users’ content monetization.

Sakai is an open source LMS for schools that handles assignments, students’ grades, polls, quizzes, calendars and courses. The open source LMS was designed to cater to the education industry’s needs by offering a flexible solution and rich functionalities, enabling an optimized and fluid learning experience. The product development strategy is continuous over time, in order to always be at the cutting edge of learning technology.

Created in Belgium further to a Claroline fork, Chamilo LMS is a complete and intuitive educational solution for content production and distribution. It is easy to use and has a very active community. Its ergonomics is designed to limit the number of clicks for each action. With a large ecosystem, this LMS contributed strongly to the development of French-speaking resources. Built on an open and scalable architecture, Chamilo promotes the values of standardization in order to facilitate and improve the accessibility of knowledge worldwide.

ILIAS is a web-based learning platform created in 1998. Since then, the solution continued growing by including several new capabilities, such as tools for collaboration and communication, an assessment engine, and content management features. In parallel, the community has become greater, fostering innovations and providing high quality support. ILIAS has a particularity in comparison with the other LMSs, which is the Personal Repository that contains only specific content related to what the learner has selected from the repository, such as viewed courses, calendars, blogs.

Table 3 shows the reasons for choosing these solutions.

6. Results and Discussion

We applied our open source evaluation approach based on the OpenBRR model to a set of six eLearning solutions, which we tested and compared. Below, the comparison’s results (

Table 4. Performance of six open source eLearning platforms assessed according to the modified OpenBRR maturity model.

	Moodle	Canvas	Open Edx	Sakai	Chamilo	ILIAS
Functionalities	0.85	0.65	0.69	0.6	0.71	0.72
Adaptive features and capabilities	0.35	0.15	0.35	0.1	0.1	0.2
Operational Software Characteristics	0.7	0.63	0.65	0.46	0.6	0.41
Service and support	0.7	0.7	0.61	0.4	0.45	0.5
Software Technology Attributes	0.48	0.46	0.48	0.34	0.5	0.44
Documentation	0.45	0.45	0.45	0.4	0.3	0.4
Adoption and Community	0.6	0.46	0.4	0.36	0.37	0.45
Languages	0.25	0.2	0.15	0.1	0.25	0.25
Compliancy	0.25	0.20	0.1	0.15	0.25	0.25
License	0.2	0.2	0.2	0.2	0.2	0.2
Total	4.83	4.1	4.08	3.11	3.73	3.82

) present the Business Readiness Rating of each solution. A software component’s Business Readiness Rating is scored from 1–5, with one being “Unacceptable”, and 5 being “Excellent. After that, we present the overall evaluation results by describing the particularities of each OSS according to the main categories of our adapted OpenBRR model.

6.1. Functionalities

Supported by a large number of investors, and widely deployed, Moodle succeeded in keeping improving its core product, by encompassing the majority of the popular educational features, such us “Video/Web conferencing “, “Personalized Reporting”, “ Collaborative Tools”, “Gamification”, “Intuitive Authoring tool”, “ Assessment module” and “Mobile app”. Likewise, its great community provides the Moodle team with high flexibility, enabling them to quickly develop new features in a short period of time, which keep it, consequently, at the forefront of educational technologies [37].

Sparked in 2008, Canvas has become a major player in the educational technology market by disrupting the LMS landscape through its diversified features’ portfolio designed to better engage users in the learning processes. We can quote as an example of some differentiating features the “Outcomes Feature” which enables a prediction of the learners’ abilities, “Speedgrader,” which enables the management of students’ submissions, and “Canvas Parent” which enables parents to follow up the learning progression of their children. However, despite its functional richness, Canvas still does not cover all the learning domains, such as “Content Creation” as well as “Gamification” which is enabled only through integration with third party tools. Finally, Canvas is designed only for schools, whereas Moodle is for Schools and Business.

Because of its design as a platform for Massive Open Online Courses (MOOCs), Open Edx enables greater interactivity in participation compared to a traditional online classroom. Similarly to its predecessors Moodle and Canvas, Open Edx allows educators to configure features such as quizzes, animations, exams and videos. However, this OSS exceeds its peers in the quality aspects due to the XBlocks module that has been made to enable a much easier design. Open Edx is considered a modern educational platform, offering an easy and intuitive experience, and enabling fast scaling, yet it misses some important capabilities, such as asynchronous learning and learner results tracking.

Similar to Canvas, Sakai has been designed solely for schools. It provides a limited set of features compared to what is offered by its peers, namely asynchronous learning, collaborative learning, e-commerce capabilities and assessment modules. It is very simple to submit assignments and take assessments. However, some features that are considered very beneficial to the user’s learning experience are still missing, such as gamification, video conferencing, blended learning and also skills management.

Agnostic to users’ types, the Chamilo LMS is especially appreciated by the majority of its users. Its friendly interface and intuitiveness made it successful among the French spoken community. It offers the ability to use and recycle lessons, and makes it easy to create and manage classes. It also allows blended learning, as well as the ability to set audio quizzes. However, similar to Open Edx and Sakai, the gamification aspects have not yet been addressed correctly. Unlike Sakai, Chamilo does not provide any “Learner Portal” which enables the learners to access all the resources that facilitate their learning.

The vocation of Ilias LMS is to provide training to public institutions and big firms. It offers a global assessment tool enabling complete exams and learner self-evaluations. In addition to that, it encompasses an evolved authoring tool as well as a rich collaboration platform. Nonetheless, we notice the absence of coverage of the features related to “e-Commerce”, “Gamification” and “Synchronous Learning”.

6.2. Adaptive Learning

In 2018, Moodle community members came out with a new plugin managing access to lessons based on learners’ assessment results [38]. This has been made possible by setting up a specific configuration through the “Restrict Access” function that conditions the access to resources based on the scores obtained in the evaluations. This adaptive experience remains very basic since it is based on predefined rules that are set and configured by teachers. Artificial intelligence algorithms have not been considered in the Moodle adaptive landscape, which allows space to enrich this OSS with more elaborated adaptive techniques.

Similar to Moodle, Canvas enables teachers to tailor the learning experience based on performance on assessments. This has been made possible via the functionality ‘Mastery Paths’, which pushes different learning paths (e.i. Content, Assignments) to each learner depending on their profile. As with Moodle, the personalization provided by Canvas remains a rule-based adaptive learning approach. The latter is similar to the approach of an expert system, without any embedded intelligence.

Several adaptive learning initiatives have been initiated to enable students’ personalized learning experience on the Open Edx platform. Educational institutions were the main contributors to that. The first project that we quote is the MS/Harvard VPAL based on TutorGen’s SCALE, which provides an adaptive assessment feature consisting of delivering specific assessments to each learner. The second main project that we quote is ALOSI, which is the result of a collaboration between Harvard and Microsoft. The aim of the latter project is setting a framework that enables the measurement of students’ learning gains and consequently proposing either ‘Remediation Actions’ or ‘Continuity Actions’.

The personalization in Chamilo is mainly visible on the assessments pane. Indeed, the students are provided customized questions depending on their level of mastery of specific subjects. The personalization of learning paths is still not covered in the current live version.

For the case of Ilias, the contributors built an expert rules-based engine offering an adaptive experience to the learners. Indeed, the personalization scenarios are built upon the logic of precondition, which consists of setting several checkpoints that the learner should satisfy to gain access to a given resource.

Sakai does not address any learning experience’s customizations in its functional scope.

6.3. Operational Software Characteristics

The look of the user interface gives the impression that the Moodle solution is a bit dated. However, the technologies behind the development of this interface are among the greatest in the field. We quote Bootstrap 4, which is the latest stable version of Bootstrap. The latter is the most advanced HTML, JavaScript and CSS framework for enabling website responsiveness. Regarding the security aspects, the number of security vulnerabilities has increased slightly this two last years, going from 17 vulnerabilities in 2017 to 20 in 2020. However, the community members demonstrated reactivity by intervening quickly to patch up the issues and secure solutions. Lastly, the solution performances are good. The solution is widely deployed in different contexts, of different sizes, and always proved to be very efficient.

Largely deployed, Canvas offers a feature-rich environment for teaching and provides several alternatives and means to engage learners. Its mobile application is very convenient and well designed. However, some users face difficulties during the first hours of usage due to the initial setup which can be confusing, especially for ’non-techy’ users. These difficulties are accentuated by the existence of a multitude of rarely-used subheadings within courses which are omnipresent on the user interface. Compared to Moodle, the latter permits more flexibility in styling and CSS theming which is not possible with Canvas. Regarding the security aspects, Canvas is implementing the most popular industry standards which are ISO 27001, NIST’s Cyber Security Framework, and AICPA’s and SANS’ CIS Critical Security Controls. The strategy of vulnerability detection is implemented on different levels, namely, preventative and detective.

Already used by more than 19 million people, the Open Edx platform is well-organised and easy to navigate. It enables the learners to find courses in an intuitive and straightforward way. Users have the option to “audit” the courses in order to have a first idea about the contents, which is very helpful when deciding which course to engage with. Regarding the performance aspect, many users have attested that the performances delivered by the platform were satisfying mainly due to detailed design work, taking into consideration the characteristics and particularities of each use case.

Even if it is among the first twentieth century LMS, Sakai did not succeed in evolving at the same pace as its peers. The users continue to have some difficult experiences, mainly due to performance issues affecting the speed of task execution, complex manual tasks when it comes to importing data from other tools and the non-intuitiveness of the user interface which is a source of frustration for several learners. Finally, the unoptimized accessibility from a mobile device certainly needs some improvement.

Simple to use and very ergonomic, Chamilo succeeded in gaining the heart of its users due to its intuitiveness, ease of use and its well-optimized performances due to the low memory and CPU consumption. The installation procedure is clear and simple, and there is no configuration needed to start using the platform. The solution flexibility enables the users to easily find better ways to work according to their contexts. Chamilo is also mutli-device and mobile friendly. However, the documentation will need to be enriched and frequently updated. The data analytics also need to evolve to enable better student tracking. Regarding the templates, their number remains low compared to similar LMS, notably Moodle and Canvas.

Scalable, with reliable foundations and diversified features, Ilias offers an adapted learning experience to each user, while making it easy to navigate and administer the platform. Its personal workspace makes the trainings portfolio creation straightforward and it provides an adapted learning journey. However, the look and feel of the user interface remains to be improved to enable leveraging the full capabilities of the platform.

6.4. Service and Support

The use of OSS is not effortless or painless; coherent support strategies have to be proposed to users to handle the difficulties that they may encounter during the product’s usage. We can identify three main strategies: (1) hiring skilled developers that will join the community ecosystems in order to be aware of the discussions, novelties and encountered issues; (2) donating to groups of developers; (3) contracting for commercial support.

Regarding the studied OSS, all of them can cover the different cited support strategies. However, choosing to go with one of the two first strategies can be easier for Moodle, Canvas and Open Edx users due to the important community supporting their products, and also due to the availability of independent experts, which is not the case for the other OSS. Likewise, for helping their users to contract for a paid support option, the OSS founders provide on their official website a list of companies that are certified on their products who can offer high quality support to users.

6.5. Software Technology Attributes

Based on a modular design, Moodle allows trainers to build their own courses using plugins for different activities and contents. Several connectors are developed enabling data transfer safely from Moodle to third party applications such as CRMs, e-Commerce solutions, databases or messaging tools. Regarding the product roadmap, the Moodle team delivers four yearly releases; two of them are major, and the two others are minor.

Canvas was built with the objective of having an open architecture solution, exposing standard interfaces and enabling easy integrations with third parties. Indeed, Canvas supports Learning Tools Interoperability, which is a standard built specifically to make possible the integration of Canvas with other vendors’ tools. This allows course enhancement based on external materials from the Internet or other applications. Regarding the product roadmap, the Canvas team assure a continuous deployment of new features. Every third Saturday of each month, a new release note is issued, indicating the upcoming new features and updates.

Modularity is the main principle behind the design of Open EdX’s platform. Indeed, Open Edx is structured as a set of independent web services called Independently Deployed Applications (IDAs). The latter allows the developers to bring several modifications to the solution without modifying the code. The Open Edx design and architecture enables a better management of the platform’s code complexity and encourages the community members to contribute more to the project. Regarding the releases, Open Edx has no fixed date for deliveries. Its delivery strategy is aligned with the principle of modularity. Indeed, Open Edx proceeds to releases by functional module, and does not consider one release for the whole solution. We can have several deliveries related to a single module during a given period versus no delivery for others.

Based on a Service Oriented Architecture, Sakai provides a list of extensive APIs which enable easy and standard integrations with third party applications. The connection with external modules and solutions is hence simplified, as well as interactions between heterogeneous components. Nonetheless, changes and evolution in Sakai are not easy to undertake. The users may spend a great amount of time trying to implement new configurations that can be quickly applied to other solutions. Regarding releases, Sakai delivery frequency is one major release yearly and periodic minor ones.

Built as a web based learning platform, Chamilo development guidelines are to carry out an easy-to-use platform in varied contexts that enables simple and quick integrations and which is open in the case of needed extensions. Chamilo is a flexible tool adapting to both educational institutes as well as business. Its architecture is designed, relying on three principals: (1) flexibility by making the solution adaptable to a variety of environments; (2) modularity by enabling the users to work only with the modules that are needed for them; (3) extensibility by making it possible and easy to add new features. Regarding the releases, Chamilo delivers up to three releases yearly.

Among the first contemporary LMS, Ilias exposes several plugins that allow easy and standard integrations with external software. In addition, different types of content can also be integrated through the LTI interfaces. However, this solution is suffering from performance problems. Several users experience hampering slowness due to inadequate solution configuration. Indeed, skilled developers are needed to implement open source and to maintain it. During the initial installation, developers need to modify the back-end coding to ensure acceptable performances.

6.6. Adoption and Community

Moodle grew out of its base of K-12 education which is composed of developers, teachers, administrators and students. It enjoys a greater community in terms of volume in comparison with its peers. Therefore, the number of contributions is among the highest, whether in terms of new features or financial contributions. Regarding its adoption, Moodle can be found on more than 63,000 sites, with more than 78 million users from 222 countries.

Canvas LMS also enjoys a large community which exceeds 1.2 million members. Additionally, the number of users surpassed 30 million global users, which makes the Canvas community among the most active in the field of education software. Partnerships development is also one of the pillars of Canvas’ expansion strategy. Indeed, several partners from diversified fields are contributing to product development. The most recent ones are AWS Educate, Microsoft and Nexus Edge.

The Open EdX community came mainly from higher education with the purpose of serving students with advanced information via MOOCs. More than 50 prestigious institutions and businesses have adopted this platform as a centerpiece of their eLearning strategy. We quote Amazon Web Services, Google, IBM, MIT, Harvard and also Stanford.

Chamilo LMS community emerged mainly from the educational and human resources sectors. It counts more than 12 million members of which the majority are passive. The active members that have contributed to product development or documentation or even forum discussions remain few.

Sakai community is composed mainly of educational players. We can find on the Sakai official website 250 universities from different parts of the world that are product users and network members. However, Sakai development suffers from a lack of financial contributions which directly affect its evolution.

Ilias users opted to form different groups related to each geographical region to exchange information about the evolution of the platform. The community members belong mainly to European countries, many of which are German institutions and businesses. English is designated as the official forum language. However, we notice that there is a large number of discussions that are written in German, which does not benefit international users.

6.7. Synthesis

The comparison of the six shortlisted open source softwares shows that Moodle dominates the evaluation by being ranked first in all the evaluation criteria. Functionally, it covers almost all the main features provided by its peers and which are depicted in the introduction of this article, namely, content creation, student assessments and learner tracking. Moreover, the offline learning feature is a crucial differentiating point due to the lack of Internet accessibility to a great number of Moroccan youths. Another important point to mention is the availability of the documentation which makes the tool appropriation easy for developers. Likewise, the large Moodle community and ecosystem enables the Moodle team to be, on one hand, among the trailblazers in terms of innovation, and on the other hand, to be highly reactive when it comes to catching up to competitors’ developed new features (

Table 5. Comparison of six open source eLearning platforms.

	Documentation	Language	Compliancy	License
Moodle	Available on several languages ( official web site, github)	More than 119 language. Arabic and French included.	SCORM, AICC, xAPI, cmi5, and IMS.	GNU General Public License.
Canvas LMS	Available on the official web site and github.	Thirty-seven languages supported including Arabic and French.	Canvas support SCORM, 1.2 and 2004 editions 2, 3, 4. AICC not supported. xAPI through BLTI dispatch in SCORM Cloud.	GNU General Public License.
Open Edx	Available on the official web site.	English is the official language of Open Edx. Other languages can be supported through Transifex.	Does not support SCORM natively, but it is possible via integration with SCORM Cloud. xAPI, AICC, xAPI are not supported natively. These content formats can be supported via SCORM Cloud and the LTI Consumer XBlock.	AGPL and Apache license.
Chamilo	Available on the official web site and github.	Twenty-six languages supported including Arabic and French.	SCORM, AICC and xAPI compliant.	GPL v3.
Sakai	Available on the official web site and github.	Nineteen languages supported including Arabic and French.	SCORM and xAPI compliant. AICC not supported	Educational Community License.
Ilias	Available on the official web site and github.	Fifty-one languages supported including Arabic and French.	SCORM 2004, SCORM 1.2, AICC and xAPI compliant.	GNU General Public License

).

Regarding the “adaptability aspects”, Moodle still lacks advanced adaptability features. Indeed, artificial intelligence techniques have not been used to predict the learners’ optimal learning paths. Hence, our project of building an “adaptive module” that will enrich the functional spectrum of Moodle by providing a personalized learning experience, will positively affect the educational landscape of the region, since Moodle is the most used tool in North Africa, and precisely in Morocco. This large deployment will also allow better support of the solution due to the availability of competencies in the region. However, some drawbacks remain to be tackled, namely, the simplification of technical customizations that can be complicated to implement, and which may require time to optimize. On the other hand, the user interface needs to be reviewed in order to make it more attractive and easier to use.

Regarding blended learning, Canvas can be an excellent choice due to its flexibility and its functional richness. Its user interface makes it intuitive and easy to navigate. The learner objectives definition is accurately made based on evidence gathered from quizzes and assignment results. Canvas is also great at integration with third party tools through its support of Learning Tools Interoperability. Lastly, it is important to mention that the Canvas cloud version is an engineered solution, built natively for the cloud and hosted by Amazon Web Services. However, Canvas lacks natively some crucial features which negatively affect its rank and makes the learning experience incomplete, namely the authoring tools as well as the gamification features.

Regarding flexibility, Open Edx is the one that differentiates the most. It provides its users with a highly flexible platform that enables the scale of education. Likewise, its richness in terms of content creation tools helps the creation of effective courses by using a variety of supports such as photos, animations, and videos. However, some limitations may not favor Open Edx against its competitors, such as the organization of learning materials that can be confusing for some users, as well as the non-compliance with some standards such as SCORM, xAPI, AICC and xAPI.

Chamilo and Ilias are almost ex aequo. They provide a large functional spectrum, and are keen on their products’ development and improvement. However, the documentation aspect needs to be addressed in order to make available the needed documents for product installation and usage.

Sakai LMS is ranked last. The reasons for this low ranking are, firstly, the lack of some critical features, secondly, the slowness of the solution and lastly, the non-intuitive user interface, which makes the usage of the solution complex and confusing.

The obtained results are similar to those of the study by [31,39] who recommend Moodle as the best open source LMS platform. This is because of its many features that cater to students’ demands, its greater community ( more than 80 million users) and all the important features that support an adaptive learning environment. On the contrary, the results of [27] show that Open edX outperforms Moodle in almost all user experience aspects. On the other hand, according to the examined policies of each LMS platform in Higher Education Institutions, ref. [30] showed that Canvas was the most mentioned LMS platform. The next most featured LMS was Moodle.

To conclude, each platform has its advantages. Open source learning management systems are free, however upkeep and improvement costs will apply. The characteristics described in this paper may influence a university’s choice of LMS. These properties are of great importance for administrators in correctly selecting a platform that meets student needs, teaching and learning goals, and enables a social learning environment for students. In fact, a major factor influencing student satisfaction is whether the features available in the LMS meet their needs and are easy to use. Each LMS has different criteria that institutions have to evaluate the LMS platforms according to the specifications and needs of the users.

The necessity of feedback that takes the learner’s social and emotional conditions into account is one of the shortcomings of current platforms. The incorporation of content-based recommendation systems and mechanisms for sentiment analysis throughout the learning process is required for these platforms.

Moodle is certainly not perfect and still needs improvement, but in our case, and based on the results of the evaluation, it turns out to be the best open source option that can meet our needs, while having a strong impact on a large community of learners.

7. Conclusions and Future Work

We live in an era where client customization has become a necessity, and where children’s future welfare is determined mainly by the quality of their current education.

Our daily interactions with the world are more and more customized, whether through physical or virtual channels. However, in the education field, transformations are progressing at a very slow pace. A large number of students are still following the traditional way of education, which has demonstrated several gaps in comparison to the modern eLearning approaches. For this purpose, we undertook this study, consisting of analyzing and evaluating a list of open source solutions in order to choose the most suitable one. The result will serve as the basis for our new eLearning solution that will provide high customization capabilities.

In this article, we first presented the mechanism of adaptive eLearning systems; afterwards, twenty-two solutions among the most popular educational OSS in the market were assessed. The evaluation has been conducted following the OpenBRR maturity model that consists of evaluating the OSS candidates based on a list of qualitative and quantitative weighted criteria. After a pre-evaluation phase, six OSS solutions have been identified and were further analyzed. Moodle has been ranked first in all evaluation categories, because it provides a convergent solution that includes the most-used educational features, which are required for an end-to-end learning experience. In parallel, it enables smooth integrations due to its open and modular architecture, as well as its compliance with the majority of eLearning communication standards, such as SCORM, AICC, xAPI, cmi5, and IMS.

In our future work, we intend in the short term to build a prototype by extending Moodle in a way in which the courses adapt to the learner’s initial knowledge, learning styles, and learning objectives, as well as personal traits. In the mid-term, we aim to provide an advanced adaptive solution that can be implemented as an add-on, which can be leveraged by Moodle users in order to offer a personalized, effective and optimal learning experience.