Next Article in Journal
A Decision-Making Method for Machinery Abnormalities Based on Neural Network Prediction and Bayesian Hypothesis Testing
Next Article in Special Issue
Early Dropout Prediction in MOOCs through Supervised Learning and Hyperparameter Optimization
Previous Article in Journal
An Efficient Far-Field Wireless Power Transfer via Field Intensity Shaping Techniques
Previous Article in Special Issue
Futuristic Trends and Innovations for Examining the Performance of Course Learning Outcomes Using the Rasch Analytical Model
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 10
Issue 14
10.3390/electronics10141611
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Recommendation Systems for Education: Systematic Review

by
María Cora Urdaneta-Ponte
1,2,
Amaia Mendez-Zorrilla
1,* and
Ibon Oleagordia-Ruiz
1
1
eVIDA Research Group, Faculty of Engineering, University of Deusto, 48007 Bilbao, Spain
2
Faculty of Engineering, Andres Bello Catholic University (UCAB), Ciudad Guayana 08050, Venezuela
*
Author to whom correspondence should be addressed.
Electronics 2021, 10(14), 1611; https://doi.org/10.3390/electronics10141611
Submission received: 10 May 2021 / Revised: 1 July 2021 / Accepted: 2 July 2021 / Published: 6 July 2021
(This article belongs to the Special Issue Machine Learning in Educational Data Mining)
Browse Figures
Versions Notes

Abstract

:
Recommendation systems have emerged as a response to overload in terms of increased amounts of information online, which has become a problem for users regarding the time spent on their search and the amount of information retrieved by it. In the field of recommendation systems in education, the relevance of recommended educational resources will improve the student’s learning process, and hence the importance of being able to suitably and reliably ensure relevant, useful information. The purpose of this systematic review is to analyze the work undertaken on recommendation systems that support educational practices with a view to acquiring information related to the type of education and areas dealt with, the developmental approach used, and the elements recommended, as well as being able to detect any gaps in this area for future research work. A systematic review was carried out that included 98 articles from a total of 2937 found in main databases (IEEE, ACM, Scopus and WoS), about which it was able to be established that most are geared towards recommending educational resources for users of formal education, in which the main approaches used in recommendation systems are the collaborative approach, the content-based approach, and the hybrid approach, with a tendency to use machine learning in the last two years. Finally, possible future areas of research and development in this field are presented.

1. Introduction

The penetration of information and communications technologies (ICTs) has resulted in major social changes, marking the onset of an era that has been characterized by globalization, the information society and knowledge. Likewise, it has meant a challenge in different spheres of activity, one of which is education, in which they are incorporated into teaching and learning processes. The incorporation of ICTs has entailed a growth in Internet services, which in turn has been reflected in a major increase both in the amount and in complexity of the information available online. A problem for users has emerged as a consequence of the information overload in terms of the time they spend on their search and the amount of information retrieved by it. Being able to suitably and reliably ensure relevant, useful information is a determining factor when taking decisions.
Recommendation systems (RSs) have emerged in order to deal with this problem, with the purpose of helping users find what is genuinely relevant to their needs. According to previous study [1], the RSs are software tools to help users in the decision-making process by applying information filtering, data mining, and prediction algorithms. This offers each user a variety of choices and options according to his or her interests and preferences [2].
A classic way of categorizing the different types of RS was provided by Burke [3], who distinguished between six different classes of recommendation approaches: Collaborative Filtering (CF), Content-Based Filtering (CBF), knowledge-based filtering, context-based filtering, demographic filtering, and hybrid filtering.
For Herlocker [4], collaborative filtering systems are the simplest, they calculate the similarity between users, and they predict product ratings for the active user according to ratings provided by other users who have very similar preferences to the current users. Content-based RSs are based on features provided previously by the user, which will then be used to filter all the elements in the system. Articles that have the highest similarity score will be recommended to users [5]. According to [6], the knowledge-based approach suggests elements to the user according to the knowledge it has about them and their relations, so as to meet the user’s specific needs. The context-aware recommendation systems are able to recognize the user’s contexts and daily activities in real time such as location, environment information, etc., for suggesting contextually relevant items [7]. In demographic filtering, recommendations are established on a demographic profile of the user [8], and the recommendations are suggested from similarities in terms of demographic data in user profiles such as nationality, age and gender, etc. [9].
Hybrid recommendation systems that combine two or more techniques from among the approaches described previously to improve recommendation performance have emerged as a means to overcome any problems that may emerge via the use of the different techniques, such as the cold-star problem [3]. The cold-start problem refers to situations where there are only a few ratings on which to base recommendations [4], which usually happens when new items are registered in the system and normally have no rate from the users [5].
In recent years, artificial intelligence techniques have been introduced for RSs. The study [6] includes the computational intelligence-cased recommendation in the classification, which include Bayesian techniques, artificial neural networks, Machine Learning (ML) techniques, genetic algorithms, and fuzzy set techniques. According to [1], the use of these techniques has been a promising solution when designing RSs in the era of Big Data.
RSs have become a very commonly used tool in different domains such as e-commerce, social networks, digital media, and books [10] and in the field of education, as well as in teaching and academic advisory services [11], the latter being the subject of interest in this systematic review.
Recommendation systems depend to a great extent on the domain in order to operate, and taking the recommendation given by a system and transferring it to another system is no easy task. Therefore, the challenge facing educational recommendation systems is how to better understand the user’s interest and the purpose of the domain [12].
The purpose of the domain is conditioned by the different levels and type of education, which may belong to one of three major groups: Formal education, non-formal education, and informal education, the latter being what is acquired in day-to-day life through interaction with individuals and their relationship with the environment.
Formal education is understood to refer to what is included within the education system, which follows an established school curriculum and includes nursery, primary, compulsory secondary and baccalaureate stages, as well as vocational training and university teaching. As for non-formal education, this does not follow any type of regulation, and is what enables individuals to pursue lifelong learning. It is devised as a means for achieving stable, evolutionary training in competences, knowledge, and skills [13]. For Belando-Montoro [14], the purpose of any learning activity pursued throughout one’s life is to improve knowledge, competences, and skills from a personal, civic, social, or work-related standpoint.
The number of educational resources is growing nowadays, making it increasingly difficult for traditional search engines to meet requirements related to online searches for information about educational products and services by students during the learning process [15].
A significant number of recommendation systems have been proposed in the field of education, as well as in teaching and academic advisory services. Within the domain of education, target users are students, teachers, and academic advisors, and the recommended elements are educational materials, learning objects, papers, universities, and information such as that about courses, student performance, and the field of study [11].
Applying RSs to the field of education requires taking into account a broad set of variables that may include, among others, level of knowledge, competences, and learning styles on the part of students. Given the rapid evolution of these systems, it is necessary to be aware of the trend in the techniques used for development.
The aim of this systematic review is to obtain an overview of RSs in education, its fields of work, recommendation elements, and the techniques used to identify any gaps, while at the same time providing a suitable framework guideline for future research activities. The search for articles was conducted between the years 2015 and 2020, in the course of which 98 works were analyzed after setting out relevant search criteria.
The article is structured as follows: Section 2 explains how the method used in the systematic review was developed; Section 3 provides an analysis of records; and lastly, Section 4 contains the conclusions and discussion about future work.

2. Materials and Methods

The phases suggested by [16] were taken as reference in order to conduct the systematic review of literature (SRL), and these are shown in the following diagram (Figure 1).
The purpose of this SRL is to determine any gaps in recommendation systems in education, especially in higher education, in its different fields of study and areas of knowledge, aspects related to the development of recommender systems, such as the filtering algorithms, and other techniques used and their validation, to obtain accurate information from the review that may contribute towards the objective that has been set out. Thus, the following search questions were defined, in which Table 1 specifies the purpose of each of them.

Review Protocol

Once the objectives had been defined and the search questions set out, the following repositories were used to search for articles: IEEE and ACM as the two main digital libraries of scientific content in the area of informatics and computing, and to complement the search, Web of Science (WoS) and Scopus were taken into account as they are two databases that have access to an important and wide number of applications in the different areas of knowledge.
The keywords were then selected by taking into account the initial mapping literature and the key words found in abstracts pertaining to them. The terms used were as follows: Recommender system, recommendation system, and education. The advanced repository option was used to fine-tune the results, which enables search chains to be defined via the use of logical operators, using the following for the search: “recommender system” OR “recommendation system” AND education; the asterisk (*) was also used to conduct searches, and this symbol was used as a catch-all, and helps to represent one or more characters in a given term. In the recommend* search, this might imply terms such as recommend or recommendation. The details in the different sources are reflected in Table 2. The search was confined to records between the years 2015 and 2020.
The following PRISMA guidelines [17] were followed to select articles from the systematic review, and these suggested applying 4 steps: Identification, filtering, eligibility, and inclusion.
Figure 2 represents the PRISMA-adapted flow chart [17], which shows details about the number of records in each selection phase of the articles identified that were included and excluded, and the criteria used to determine which primary studies would be included in the review.
In total, 2937 records were obtained when conducting the search of the sources selected, the distribution of which can be seen in Table 2. Once these initial records had been identified, a first filter was then applied in which any repeated studies were disregarded, ending up with 2537, to which the following inclusion and exclusion criteria were applied until obtaining 802 (see Table 3).
Abstracts of the articles that had been kept were then read, excluding those in which the subject of study failed to cover any recommendation systems in education. In total, 587 articles were excluded from this process, with 225 remaining for in-depth review. These were read in their entirety and 127 were ruled out, leaving 98 articles included for the systematic review.
A systematic review involves the selection of significant material according to the objective established in the review [18]. Prisma [17] provides guidance for this review, including a checklist that can be used to assess the quality of the review. Based on this checklist, we have defined 10 indicators (Table 4), which permit us to measure the quality of the articles, the first 7 of which are focused on the content of the article and others related to publication quality metrics.
For the assessment of metrics related to bibliographic references, we have based the criteria on [18] to establish the criteria in terms of the number of references and the percentage of references that are up to date (last five years). For the number of citations, we have used the metrics obtained from Google Scholar.
The total score will have a value between 0 and 16, classifying them as deficient (0–2), sufficient (3–5), good (6–10), very good (11–13), and excellent (14–16).

3. Results

Once the articles included in the systematic review had been included [19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116], data mining then got underway, for which purpose those elements that provided a response to the research questions were identified when the documents were read (Table 1). How the articles were distributed according to the categories found in each approach were identified and quantitatively shown in the analysis conducted.
Table A1 shows the information obtained from each article, in accordance with the questions asked in the course of the systematic review.
From the information in Table A1 and Table A2 in Appendix A, according to the geographical distribution of the articles included in the systematic review, the regions with the highest number of publications are Asia, followed by North America and Europe (Figure 3). Figure 4 highlights the trend towards an increase in the number of RSs, where growth in the last two years can be observed, representing 37% of the total. It also shows the number of articles by type of publication.

3.1. Types of Education Covered by the Recommendation Systems

After reading the articles in their entirety, it was then possible to identify which RSs are geared towards formal education and which to non-formal education. Each article can be identified in Table A1 together with the type of education covered, while at the same time it could be seen that of the total, 58 RSs referred to formal education—in particular, university studies—33 of the works focused on non-formal education, and 7 both on formal and non-formal education. The percentage of articles according to type of education can be observed in Table 5. In most cases, the area of knowledge to which the recommendations are geared is not specified, while the works [32,53,54,109,113] indicate that studies in their case recommend resources pertaining to the field of architecture and engineering.
In formal education, there are some articles such as [21,30,31,45] that base their recommendations on learning style. The RSs of [30,35] use the Felder–Silverman learning styles for this purpose. In non-formal education, we highlight the articles [27,113], which recommending lifelong learning, with the latter using job competences to recommend courses.
When analyzing the data obtained to determine what types of elements are subject to recommendation, it was noted that 96% of works focus on offering the service to post-graduate students and universities such as learning resources, courses, post-graduate studies, and universities. Of these articles, 78 are oriented specifically for students (Table 6), most of which make suggestions for courses and educational resources, where only [26] focuses on teachers by recommending resources for teaching practice in the view of the increasing availability of such resources online and the resulting difficulty in locating them, and [50] focuses on both students and teachers, where the student receives suggestions for support material to study while teachers receive student profiles for review.
In terms of the type of element recommended, it was noted that 37 works stand out in which RSs offer learning resources, and these recommendations are mostly given in accordance with user preferences; 133 RSs suggest courses; 5 recommend a sequence of courses/syllabuses; 5 recommend elective degrees courses; and the remainder focus on recommending papers, postgraduate courses, academic advice, and professions, among others. The articles [85,106,108] that give academic advice stand out. Likewise, articles [31,33,112] that recommend study sequence or syllabuses. The work proposed by [41] recommends jobs for students and young professionals based on job descriptions and user profiles. Table A1 shows the elements subject to recommendation in detail while Table 7 provides the percentage distribution according to recommendation element.

3.2. Developmental Approach of RSs in the Field of Education

When we asked about the approach used in developing RSs in the field of education, we found that they were mainly collaborative, content-based, knowledge-based, and hybrid, highlighting the use of ML techniques in the last two years. Table 8 shows the number of articles according to developmental approach.
As for distribution, the following was observed: 32 RSs use the collaborative approach, and the following most-used approaches are the hybrid approach with 20 RSs, ML with 19 RSs, and the knowledge-based approach with 18 RSs. Finally, two RSs use the content-based approach, highlighting the work [69], which takes some features from LinkedIn to create the user profile, such as the user’s geographical location and activities of interest, and represents the information using an ontology.
Of the articles that used ML, 79% used supervised ML and 17% used unsupervised ML. The techniques most commonly used include: [74,75,79,87,94,99,100,109,116] use Neural Network, representing 47%, of which [74,79,87,116] use Deep Learning; [73,111] use Support Vector Machine (10.5%); [86,88] use K-nearest neighbor (10.5%); [85,96] K-means (10.5%); and finally [77,82,89,93,103] use multiple techniques (21%).

3.3. Type of Platform Implemented in Recommendation Systems

Of the studies reviewed, 21 adopted an online platform for implementation purposes (Table 9), of which [19,20,33,41,55,73,75,77,80,82,83,85,87,88,103,116] used universities/university platforms as a source of data, and [38,44,60,72,92,94,109,117] used learning platforms.
Moodle was used in the case of [24,66], while four of the recommendation systems sought information from online resources, namely [47,62,84,95].
Ref. [52] proposes a MOOC-FRS SR based on user behavior on the MOOC platform for personalized course recommendation. The article [64] proposes to recommend courses in MOOCs, taking into account the existence of more than one provider in the recommendation process. It will also use the knowledge in these MOOCs ecosystems to improve the recommendation of courses through these platforms. The article [81] uses a chatbot to give the recommendations, while the article [104] uses a mobile application to give the recommendations.
The remaining articles analyzed offered proposals or system prototypes in which recommendation algorithms were implemented. Some used test data as a way in and contrasted their results by comparing them with other algorithms used.

3.4. Quality Metrics

To ascertain the assessment related to the quality metrics of each of the articles, the corresponding measurement was carried out according to the defined indicators (Table 4). Each of the articles was classified on a scale from deficient to excellent, according to the significant contributions to our systematic review, evaluating them in terms of the degree of innovation, details of the proposal, validation, results, and analysis, as well as the references and number of citations to it. Table A2 shows the score obtained by each of the articles.
The articles classified as excellent and very good are selected and analyzed again to obtain information related to users, educational level, type of ML, and system validation. This result is shown in Table 10.
The studies were conducted with students or learners (seven articles), academic users (one article), and one article with programmers, of which 67% are oriented to higher education and only one article is oriented to lifelong learning, whereby [27] indicates that the proposed system facilitates personalized lifelong learning for members of a community and promotes interactions between groups of learners.
Of these selected articles, six of which use ML, representing 67%, three use supervised ML, two unsupervised ML and one booster ML.
In relation to the metrics used to measure the performance of SRs, precision, which corresponds to the proportion of the recommendation that is relevant to the user, is used by 67%, followed by recall and F1.
For SR’s validation, 89% of the articles use real data, and only article [59] uses artificial data. Articles [70,75,90] validate off-line, while [72] validate off-line and on-line. For the verification of SR performance, the most used metric is accuracy (67%) followed by F1, the harmonic mean between precision and recall.
Finally, analyzing the purpose of these articles, we observe that they have very varied objectives. Regarding the innovation appreciated, we can highlight [27], which recommends courses based on learning communities, [69] recommends courses based on their relevance to relate course profiles, learner profiles, and jobs, and [69] recommends courses based on their relevance to relate course profiles, learner profiles, and jobs. Ref. [72] recommends exercises adapted to each student according to the knowledge and results of previous exercises. Ref. [84] recommends courses for teachers, research supervision based on publications, research interests, and educational training.

4. Discussion and Conclusions

The incorporation of ICTs into education has marked changes in its processes, whether in distance learning or in support for the different processes through educational resources available online. These are growing rapidly, whereby an increase in RSs has been noted in this sphere of activity, especially as support in formal education.
From the analysis, it can then be observed that RSs take into account user preferences when making suggestions based on recommendations from similar users, while [21,30,35,45,53] make recommendations based on learning style, and [24,31,46,50,61,64] based on diagnosis/student progress and the knowledge group. Likewise, another element they take into account are user skills and/or competences [40,42,56,69] and competences related to work associated with their profile within Internet job search portals.
RSs also base their suggestions on learning style [25,35,96]. The proposal of [30] use a survey based on the Felder and Silverman Learning Styles Model (FSLSM) to determine this, and these learning styles are classified according to the following categories: Sensory, intuitive, visual, verbal, reflexive, sequential, and global.
In terms of RSs that use the collaborative approach, they are thought to evidence the cold-start problem, with works [36,41,56] suggesting a greater volume of data to improve performance, and [22,48] adding more parameters to the user profile, such as learning styles or reading tastes—in general, it is suggested that this be combined with other approaches in order to improve performance. Ref. [84] indicates that the use of deep learning techniques with collaborative filtering deals with the cold-start problem in recommender systems and [115] solves the cold-start problem using collaborative filtering system by adding classification information. Furthermore, [69,86,110,113] indicates that the use of ontology for the representation of user information helps to solve the cold-start problem.
Of the articles that use the hybrid approach, we can draw attention to [69], which states that the context should be incorporated in the user profile in order to improve performance, while [27,64] suggests incorporating social networks in the future. As for those that use the knowledge-based approach, the systems work with ontologies and semantics, recommending that information be gathered from a range of sources and be represented via ontologies.
In the systematic review, we have analyzed 98 articles related to the use of SRs in education, most of them in a formal education context, so we can suggest further study in non-formal education. Table 10 shows the trend in the use of ML over the last two years. These techniques are combined with different filtering approaches to improve recommendations and cold-start-related problems.
A feature found in the analysis of the articles is the heterogeneity of the data in the domain of education, where [69] indicates that the integration of data from multiple heterogeneous sources helps the system to improve recommendations. There is also a need to study algorithms based on a semantic approach in more detail, the idea of which would be to use ontological knowledge to describe the elements in order to obtain a detailed representation of their content. This may in turn contribute towards improving the results obtained from the recommendation in terms of relevance of the educational material suggested and, therefore, would improve the student’s learning process.
In the validation of RSs, the source and size of the dataset must be considered. Different strategies are used to determine the quality of the recommendation, such as offline and online validation, with possibilities for expert assessment up to the use of multiple metrics.
The RSs subject to study mainly sought elements to be recommended in a single place, although the search for information should also be explored in a range of sources so as to be able to ensure a wide variety of elements and offer a better recommendation service. Social networks also constitute a potential means for searching for information via recommendation systems. Ref. [27] indicates that the use of social media could improve the problem of data sparsity. According to the report on prominent indicators from the information society issued by the Spanish National Observatory for Telecommunications and Information Society, the percentage use of social networks in Spain is 67.9% and in the European Union 64.9% [118].
To conclude, out of the 98 articles included in the systematic review, the questions posed could be answered to a large extent, where Q1, Q2, and Q3 were covered in their entirety. From the analysis of the articles, we can highlight the following:
  • According to the type of education, the SRs cover mostly formal education, especially oriented towards students.
  • As for the elements subject to recommendation, they are very varied, highlighting educational resources and courses.
  • The most commonly used development techniques are collaborative filtering, followed by RSs that combine different techniques. Similar systematic reviews, such as the one presented by [119], agree with this result, finding a gap in the use of intelligent techniques. It can be seen in our review how this potential area has been covered since 2019, where proposals for RSs with machine learning are presented.
  • The incipient use of ontology for the representation of information and the construction of user profiles can be observed.
  • In the analysis of question Q4, in which we asked about the type of platform used by the RSs, not much information was obtained, as 76% of the articles analyzed did not specify the type of platform on which they were to be implemented. However, from the work [119], we can observe 50% of RSs using a web platform.
In addition to the limitation found in question Q4, we can list the following:
  • When selecting the articles for analysis, we selected those where the focus was on higher education, because the interest of the researchers is adult education. In order to obtain a broader view, it is suggested to take into account all levels of education.
  • The evaluation metrics of the SRs were only analyzed in the selected studies according to the quality metrics. For further study, it is suggested to extend this analysis to all articles.
Some gaps were identified in the systematic review, which allows us to suggest that future work should focus on the following:
  • The development of hybrid systems, in particular, the use of intelligent techniques and the use of ontology and evaluate the performance of RSs.
  • User information should be explored with information from different sources, such as social media, to build a more complete profile beyond user preference and similar users.

Author Contributions

This review was conceptualized by M.C.U.-P.; supervision, A.M.-Z. and I.O.-R.; methodology, writing—review and editing, M.C.U.-P., A.M.-Z. and I.O.-R. All authors have read and agreed to the published version of the manuscript.

Funding

This document has the support of the Unesco Chair in Human Resources and the eVida group recognised by the Basque Government from the University of Deusto, Bilbao—Spain.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table
Table A1. Articles include in the systematic review.
Table A1. Articles include in the systematic review.
REF.YearCountryType of EducationType of UserRecommendation ElementDevelopmental ApproachType of Platform
[19]2015SpainFormalStudentsLearning resourcesHybridOnline
[20]2015IndiaFormalStudentsUniversitiesCollaborativeOnline
[21]2015BrazilFormalStudentsLearning resourcesKnowledge-basedN/S
[22]2015New ZealandFormalStudentsCoursesCollaborativeOnline
[23]2015BrazilFormalStudentsLearning resourcesHybridN/S
[24]2015ItalyNon-formalStudentsCoursesKnowledge-basedOnline
[25]2015ColombiaFormal/InformalStudentsLearning resourcesHybridN/S
[26]2015TaiwanFormalTeachersTeaching practice resourcesKnowledge-basedN/S
[27]2015USANon-formalStudentsCoursesCollaborativeN/S
[28]2016USAFormalStudentsCoursesCollaborativeN/S
[29]2016TunisiaFormal/InformalStudentsLearning resourcesKnowledge-basedN/S
[30]2016LithuaniaFormalStudentsLearning resourcesKnowledge-basedN/S
[31]2016MoroccoFormalStudentsStudy sequence/syllabusesHybridN/S
[32]2016EcuadorFormalStudentsLearning resourcesKnowledge-basedN/S
[33]2016USAFormalStudentsStudy sequence/syllabusesHybridOnline
[34]2016BangladeshFormalStudentsPostgraduate coursesCollaborativeN/S
[35]2016MoroccoFormalStudentsLearning resourcesCollaborativeOnline
[36]2016USANon-formalStudentsCoursesHybridN/S
[37]2016CanadaFormalStudentsCoursesHybridOnline
[38]2016USANon-formalStudentsLearning resourcesKnowledge-basedOnline
[39]2017TaiwanFormalStudentsProfessionsCollaborativeN/S
[40]2017CanadaFormalStudentsCoursesCollaborativeN/S
[41]2017USAFormalStudentsStudy sequence/ProfessionsCollaborativeOnline
[42]2017SpainFormal/InformalStudentsCoursesKnowledge-basedN/S
[43]2017BrazilFormalStudentsLearning resourcesHybridN/S
[44]2017FranceNon-formalStudentsLearning resourcesKnowledge-basedOnline
[45]2017TaiwanFormalStudentsLearning resourcesCollaborativeN/S
[46]2017FranceNon-formalStudentsLearning resourcesCollaborativeN/S
[47]2017SpainNon-formalStudentsLearning resourcesKnowledge-basedOnline
[48]2017ThailandFormalStudentsElective degree coursesCollaborativeN/S
[49]2017MoroccoNon-formalStudentsLearning resourcesHybridN/S
[50]2017USANon-formalTeachers/StudentsLearning resourcesKnowledge-basedN/S
[51]2017IndiaFormalStudentsElective degree coursesKnowledge-basedN/S
[52]2017ChinaNon-formalStudentsCoursesCollaborativeOnline
[53]2017TaiwanFormalStudentsCoursesKnowledge-basedN/S
[54]2018IndiaNon-formalStudentsLearning resourcesHybridN/S
[55]2018USAFormalStudentsPostgraduate coursesHybridOnline
[56]2018TurkeyFormalStudentsElective degree coursesCollaborativeN/S
[57]2018TaiwanFormalStudentsLearning resourcesKnowledge-basedN/S
[58]2018MoroccoNon-formalStudentsLearning resourcesCollaborativeN/S
[59]2018TaiwanFormalStudentsStudy sequence/syllabusesCollaborativeN/S
[60]2018TaiwanNon-formalStudentsLearning resourcesCollaborativeOnline
[61]2018USAFormalStudentsLearning resourcesKnowledge-basedN/S
[62]2018BrazilFormal/InformalStudentsCoursesContent-basedOnline
[63]2018USAFormalStudentsLearning resourcesCollaborativeN/S
[64]2018TaiwanNon-formalLearnersPapersHybridN/S
[65]2018BrazilNon-formalStudentsLearning resourcesKnowledge-basedN/S
[66]2018PortugalNon-formalStudentsLearning resourcesCollaborativeOnline
[67]2018ChinaNon-formalStudentsCoursesHybridOnline
[68]2018SingaporeFormalStudentsLearning resourcesContent-basedN/S
[69]2019UKFormalStudentsCoursesHybridN/S
[70]2019USAFormalLearnersPedagogical resourcesMachine LearningN/S
[71]2019BrasilFormalLearnersLearning resourcesContext-awareN/S
[72]2019USAFormalN/SMulti-Objective ExamsMachine LearningN/S
[73]2019IndiaFormalStudentsElective degree coursesMachine LearningN/S
[74]2019USAFormal/InformalStudentsCoursesMachine LearningN/S
[75]2019USAFormalStudentsCoursesMachine LearningN/S
[76]2019ChinaFormalStudentsLearning resourcesCollaborativeN/S
[77]2019IndonesiaFormalStudentsElective degree coursesMachine LearningN/S
[78]2019RusiaFormalStudentsCoursesProbabilistic ModelN/S
[79]2019USAFormalN/SCoursesMachine LearningN/S
[80]2019USAFormalStudentsCoursesMachine LearningN/S
[81]2019USAFormalStudentsProgramming problemsN/SChatbot
[82]2019USAFormalStudentsCoursesMachine LearningN/S
[83]2019PolandFormalStudentsLearning resourcesSemanticN/S
[84]2019USAFormalAcademicsAcademic advisingMachine LearningOnline
[85]2019USAFormalStudentsAcademic advisingMachine LearningN/S
[86]2019IndiaFormalStudentsEducational programsCollaborativeN/S
[87]2019PortugalNon-formalStudentsCoursesMachine LearningN/S
[88]2020IndiaNon-formalStudentsCoursesCollaborativeN/S
[89]2020SerbiaFormalAcademicLearning resourcesCollaborativeN/S
[90]2020ChinaFormal/InformalStudentsLearning resourcesHybridN/S
[91]2020UKFormalN/SLearning resourcesHybridN/S
[92]2020IndiaNon-formalN/SLearning resourcesFuzzy logicN/S
[93]2020CanadaFormalStudentsEducational programsMachine LearningN/S
[94]2020ChinaNon-formalN/SCoursesGraphsN/S
[95]2020SwitzerlandNon-formalLearnersCoursesHybridN/S
[96]2020ChinaNon-formalStudentsLearning resourcesCollaborativeN/S
[97]2020USAFormalStudentsLearning resourcesMachine LearningN/S
[98]2020MalaysiaNon-formalN/SOnline learningCollaborativeN/S
[99]2020ChinaNon-formalStudentsCoursesMachine LearningN/S
[100]2020ThailandNon-formalStudentsCoursesCollaborativeN/S
[101]2020UKFormalStudentsLearning resourcesCollaborativeweb
[102]2020USANon-formalN/SOnline learningCollaborativeN/S
[103]2020NigeriaFormalStudentsCoursesMachine LearningN/S
[104]2020ChinaFormal/InformalLearnersLearning resourcesCollaborativeMobile
[105]2020ChinaNon-formalStudentsLearning resourcesHybridN/S
[106]2020USAFormalStudentsAcademic advisingCollaborativeN/S
[107]2020AustraliaNon-formalLearnersCoursesClick-through rate predictionN/S
[108]2020ChinaFormalStudentsAcademic advisingN/SOnline
[109]2020ItalyNon-formalStudentsCoursesMachine LearningN/S
[110]2020USANon-formalN/SCoursesHybridOnline
[111]2020NetherlandsFormalN/SOnline learningMachine LearningN/S
[112]2020ChinaFormalStudentsStudy sequence/syllabusesContext-awareN/S
[113]2020USANon-formalN/SCoursesHybridN/S
[114]2020USANon-formalEmployeeCoursesCollaborativeN/S
[115]2020UKFormalN/SCoursesCollaborativeN/S
[116]2020ChinaFormalStudentsCoursesMachine LearningN/S
N/S: NOT SPECIFIED.
Table
Table A2. Assessment of articles using quality metrics.
Table A2. Assessment of articles using quality metrics.
REF.M1M2M3M4M5M6M7M8M9M10Total
[19]00100000113.5
[20]11111000117.5
[21]11111100118.5
[22]11111100219.5
[23]11111000117.5
[24]01111001118
[25]11100001117
[26]11001000104
[27]111111011313
[28]110111001310.5
[29]01000000203
[30]11100010105
[31]11111000117.5
[32]11000000204
[33]11111000129
[34]11011000116.5
[35]11111001119
[36]11111100118.5
[37]11011100117.5
[38]11011000116.5
[39]10000000113.5
[40]11001000115.5
[41]11111100219.5
[42]11110000217.5
[43]11100000205
[44]11100000115.5
[45]11010000104
[46]01100000114.5
[47]01000000113.5
[48]11111000129
[49]111110012110
[50]11000000114.5
[51]10010000126
[52]11100000115.5
[53]10111100117.5
[54]01000000136.5
[55]11111100219.5
[56]11010000115.5
[57]11111000218.5
[58]11100001128.5
[59]111111112112
[60]111111011211.5
[61]111110002210
[62]01111100117.5
[63]11111100219.5
[64]10011101219
[65]01100000215.5
[66]10111000105
[67]11111100118.5
[68]10011100116.5
[69]111111012212.5
[70]101111111211.5
[71]10000100114.5
[72]111111102212
[73]11000100104
[74]10010000114.5
[75]111111011211.5
[76]01100100205
[77]11001000104
[78]11111100219.5
[79]11110000105
[80]11111100219.5
[81]10100000103
[82]11111010017.5
[83]11100000104
[84]111111012111
[85]11101100117.5
[86]01101100105
[87]11111100208
[88]11100001117
[89]10101001206.5
[90]01100100104
[91]11010000115.5
[92]11111110108
[93]11111101209.5
[94]111111102110.5
[95]111111011110
[96]011111012110
[97]011111112111
[98]11010000104
[99]01111010106
[100]01100110116.5
[101]01111101107.5
[102]10000000102
[103]11011100106
[104]01101100105
[105]10111100106
[106]10000000102
[107]11100100105
[108]10000001103.5
[109]11111100208
[110]11000000103
[111]10111000105
[112]11101100207
[113]11011010218.5
[114]11111000218.5
[115]10011100105
[116]01100100115.5

References

  1. Karakaya, M.Ö.; Aytekin, T. Effective methods for increasing aggregate diversity in recommender systems. Knowl. Inf. Syst. 2018, 56, 355–372. [Google Scholar] [CrossRef]
  2. Venkatraman, S. A proposed Business Intelligent Framework for Recommender System. Informatics 2017, 4, 40. [Google Scholar] [CrossRef] [Green Version]
  3. Burke, R.; Brusilovsky, P.; Kobsa, A.; Nejdl, W. Hybrid Web Recommender Systems. In The Adaptive Web: Methods and Strategies of Web Personalization; Springer: Berlin/Heidelberg, Germany, 2007; Volume 4321 in Computer Science; pp. 377–408. [Google Scholar]
  4. Herlocker, J.L.; Konstan, J.A.; Terveen, L.G.; Riedl, J.T. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 2004, 22, 5–53. [Google Scholar] [CrossRef]
  5. Adam, N.L.; Zulkafli, M.A.; Soh, S.C.; Kamal, N.A.M. Preliminary study on educational recommender system. In Proceedings of the 2017 IEEE Conference on e-Learning, e-Management and e-Services (IC3e), Kuching, Malaysia, 15–17 November 2017; pp. 97–101. [Google Scholar]
  6. Anusha, U.A.; Biradar, S. Recommender Systems: A Survey. IJLTEMAS 2016, V, 2278–2540. [Google Scholar]
  7. Otebolaku, A.M.; Andrade, M.T. Context-Aware Media Recommendations. In Proceedings of the 2014 28th International Conference on Advanced Information Networking and Applications Workshops, Victoria, BC, Canada, 13–16 May 2014; pp. 191–196. [Google Scholar] [CrossRef]
  8. Thorat, P.B.; Goudar, R.M.; Barve, S. Survey on Collaborative Filtering and Content-Based Recommending. Int. J. Comput. Appl. 2015, 110, 31–36. [Google Scholar]
  9. Ortega, F.; Sanchez, J.L.; Bobadilla, J.; Gutierrez, A. Improving Collaborative Filtering Based Recommender Systems Using Pareto Dominance. Diss. E Inform. 2013, 239, 50–61. [Google Scholar] [CrossRef]
  10. Nadine, U.; Cao, H.; Deng, J. Competitive recommendation algorithm for E-commerce. In Proceedings of the 2016 12th International Conference on Natural Computation; Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Changsha, China, 13–15 August 2016; pp. 1539–1542. [Google Scholar]
  11. Deschênes, M. Recommender systems to support learners’ Agency in a Learning Context: A systematic review. Int. J. Educ. Technol. High Educ. 2020, 17, 1–23. [Google Scholar] [CrossRef]
  12. Tang, J.; Wu, S.; Sun, J.; Su, H. Cross-domain collaboration recommendation. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD), Beijing, China, 12–16 August 2012; pp. 1285–1293. [Google Scholar]
  13. Bernal Castañeda, S. Lifelong learning and limiting factors in second language acquisition for adult students in post-obligatory education. Cogent Psychol. 2017, 4. [Google Scholar] [CrossRef]
  14. Belando-Montoro, M.R. Aprendizaje a lo largo de la vida. Concepto y componentes. Rev. Iberoam. De Educ. 2017, 75, 219–234. [Google Scholar] [CrossRef] [Green Version]
  15. Cui, L.-Z.; Guo, F.-L.; Liang, Y.-J. Research Overview of Educational Recommender Systems. In Proceedings of the 2nd International Conference on Computer Science and Application Engineering CSAE 18, Hohhot, China, 22–24 October 2018. [Google Scholar]
  16. Kitchenham, B.A.; Charters, S. Guidelines for performing Systematic Literature Reviews in Software Engineering. Version 2.3 (EBSE-2007-01). 2007. Available online: www.elsevier.com/_data/promis_misc/52444systematicreviewsguide.pdf (accessed on 4 November 2020).
  17. Rethlefsen, M.L.; Kirtley, S.; Waffenschmidt, S.; Ayala, A.P.; Moher, D.; Page, M.J.; Koffel, J.B. PRISMA-S: An extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst. Rev. 2021, 10, 39. [Google Scholar] [CrossRef]
  18. Martin, S.G.; Lafuente, V. References: Indicators for Evaluation in Scientific Papers. Investigación Bibliotecológica. Archivonomía; bibliotecología e información. 2017, Volume 31, pp. 151–180. Available online: http://rev-ib.unam.mx/ib/index.php/ib/article/view/57814 (accessed on 18 March 2021).
  19. Tejeda-Lorente, Á.; Moreno, J.B.; Porcel, C.; Galindo-Moreno, P.; Herrera-Viedma, E. A Dynamic Recommender System as Reinforcement for Personalized Education by a Fuzzly Linguistic Web System. In Proceedings of the 3rd International Conference on Information Technology and Quantitative Management ITQM, Rio De Janeiro, Brazil, 21–24 July 2015. [Google Scholar]
  20. Bokde, D.K.; Girase, S.; Mukhopadhyay, D. An Approach to a University Recommendation by Multi-criteria Collaborative Filtering and Dimensionality Reduction Techniques. In Proceedings of the 2015 IEEE International Symposium on Nanoelectronic and Information Systems, Indore, India, 21–23 December 2015; pp. 231–236. [Google Scholar]
  21. Borges, G.; Stiubiener, I. Recommending learning objects based on utility and learning style. In Proceedings of the 2014 IEEE Frontiers in Education Conference (FIE), Madrid, Spain, 22–25 October 2014; pp. 1–9. [Google Scholar]
  22. Ganeshan, K.; Li, X. An intelligent student advising system using collaborative filtering. In Proceedings of the 2015 IEEE Frontiers in Education Conference (FIE), El Paso, TX, USA, 21–24 October 2015; pp. 1–8. [Google Scholar]
  23. Behar, P.; da Silva, K.; Schneider, D.; Cazella, S.; Torrezzan, C.; Heis, E. Development and System Assessment of Learning Object Recommendation based on Competency—RecOAComp. In Proceedings of the 7th International Joint Conference on Knowledge Discovery; Knowledge Engineering and Knowledge Management, Lisbon, Portugal, 12–14 November 2015; SciTePress: Setubal, Portugal, 2015; Volume 3, pp. 274–280. [Google Scholar]
  24. Montuschi, P.; Lamberti, F.; Gatteschi, V.; Demartini, C. A Semantic Recommender System for Adaptive Learning. IT Prof. 2015, 17, 50–58. [Google Scholar] [CrossRef]
  25. Rodríguez, P.; Ovalle, D.; Duque, N. A Student-Centered Hybrid Recommender System to Provide Relevant Learning Objects from Repositories; Learning and Collaboration Technologies; Springer International Publishing: New York, NY, USA, 2015; pp. 291–300. [Google Scholar]
  26. Sergis, S.; Sampson, D.G. Enhancing Learning Object Recommendations for Teachers Using Adaptive Neighbor Selection. In Proceedings of the 2015 IEEE 15th International Conference on Advanced Learning Technologies, Hualien, Taiwan, 6–9 July 2015; pp. 391–393. [Google Scholar]
  27. Zheng, X.; Chen, C.; Hung, J.; He, W.; Hong, F.; Lin, Z. A Hybrid Trust-Based Recommender System for Online Communities of Practice. In Proceedings of the 2015 IEEE 15th International Conference on Advanced Learning Technologies, Hualien, Taiwan, 6–9 July 2015; Volume 8, pp. 345–356. [Google Scholar]
  28. Elbadrawy, A.; Karypis, G. Domain-Aware Grade Prediction and Top-n Course Recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ‘16), Boston, MA, USA, 15–19 September 2016; ACM: New York, NY, USA, 2016; pp. 183–190. [Google Scholar]
  29. Khacheb, A.; Cheniti Belcadhi, L. Open assessment resources recommendation for competence assessment. In Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM ‘16), Salamanca, Spain, 2–4 November 2016; García-Peñalvo, F.J., Ed.; ACM: New York, NY, USA, 2016; pp. 395–399. [Google Scholar]
  30. Kurilovas, E.; Kurilova, J.; Kurilova Melesko, J. Personalised learning system based on estudents ´learning styles and application intelligent technologies. In Proceedings of the 9th International Conference of Education, Research and Innovation (ICERI 2016), Seville, Spain, 14–16 November 2016; pp. 6976–6986. [Google Scholar]
  31. Meryem, G.; Douzi, K.; Chantit, S. Toward an E-orientation platform: Using hybrid recommendation systems. In Proceedings of the 2016 11th International Conference on Intelligent Systems: Theories and Applications (SITA), Mohammedia, Morocco, 19–20 October 2016; pp. 1–6. [Google Scholar]
  32. Aguilar, J.; Valdiviezo-Díaz, P.; Riofrio, G. A Fuzzy Cognitive Map like Recommender System of Learning Resources. In Proceedings of the 2016 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Vancouver, BC, Canada, 25–29 July 2016; pp. 1539–1546. [Google Scholar]
  33. Xu, J.; Xing, T.; van der Schaar, M. Personalized Course Sequence Recommendations. IEEE Trans. Signal Process. 2016, 64, 5340–5352. [Google Scholar] [CrossRef] [Green Version]
  34. Hasan, M.; Ahmed, S.; Abdullah, D.M.; Rahman, M.S. Graduate school recommender system: Assisting admission seekers to apply for graduate studies in appropriate graduate schools. In Proceedings of the 2016 5th International Conference on Informatics; Electronics and Vision (ICIEV), Dhaka, Bangladesh, 13–14 May 2016; pp. 502–507. [Google Scholar]
  35. Bourkoukou, O.; El Bachari, E. E-learning personalization based on collaborative filtering and learner’s preference. J. Eng. Sci. Technol. 2016, 11, 1565–1581. [Google Scholar]
  36. Fazeli, S.; Rajabi, E.; Lezcano, L.; Drachsler, H.; Sloep, P. Supporting Users of Open Online Courses with Recommendations: An Algorithmic Study. In Proceedings of the 2016 IEEE 16th International Conference on Advanced Learning Technologies (ICALT), Austin, TX, USA, 25–28 July 2016; pp. 423–427. [Google Scholar]
  37. Kuznetsov, S.; Kordík, P.; Řehořek, T.; Dvořák, J.; Kroha, P. Reducing cold start problems in educational recommender systems. In Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada, 25–29 July 2016; pp. 3143–3149. [Google Scholar]
  38. Symeonidis, P.; Malakoudis, D. MoocRec.com: Massive Open Online Courses Recommender System. In RecSys Posters, Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA, 15–19 September 2016; ACM: New York, NY, USA, 2016. [Google Scholar]
  39. Ochirbat, A.; Shih, T.K. Occupation Recommendation with Major Programs for Adolescents. In Proceedings of the International Symposium on Grids and Clouds 2017, Taipei, Taiwan, 28 June–1 July 2017. [Google Scholar]
  40. Bakhshinategh, B.; Spanakis, G.; Zaïane, O.; ElAtia, S. A Course Recommender System based on Graduating Attributes. In Proceedings of the 9th International Conference on Computer Supported Education, Barcelona, Spain, 20–22 December 2017. [Google Scholar]
  41. Patel, B.; Kakuste, V.; Eirinaki, M. CaPaR: A Career Path Recommendation Framework. In Proceedings of the 2017 IEEE Third International Conference on Big Data Computing Service and Applications (BigDataService), San Francisco, CA, USA, 6–9 April 2017; pp. 23–30. [Google Scholar]
  42. Bañeres, D.; Conesa, J. eOrient@—A Recommender System to Address Life-Long Learning and Promote Employability. In Proceedings of the 2016 International Conference on Intelligent Networking and Collaborative Systems (INCoS), Ostrava, Czech Republic, 7–9 September 2016; pp. 351–356. [Google Scholar]
  43. De Borba, E.; Gasparini, I.; Lichtnow, D. The Use of Time Dimension in Recommender Systems for Learning. In Proceedings of the 19th International Conference on Enterprise Information Systems (ICEIS 2017), Porto, Portugal, 26–29 April 2017; Volume 2, pp. 600–609. [Google Scholar]
  44. Hajri, H.; Bourda, Y.; Popineau, F. MORS: A System for Recommending OERs in a MOOC. In Proceedings of the 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT), Timisoara, Romania, 3–7 July 2017; pp. 50–52. [Google Scholar]
  45. Li, H.; Shi, J.; Zhang, S.; Yun, H. Implementation of intelligent recommendation system for learning resources. In Proceedings of the 2017 12th International Conference on Computer Science and Education (ICCSE), Houston, TX, USA, 22–25 August 2017; pp. 139–144. [Google Scholar]
  46. Brigui-Chtioui, I.; Caillou, P.; Negre, E. Intelligent Digital Learning: Agent-Based Recommender System. In Proceedings of the ICMLC 2017 9th International Conference on Machine Learning and Computing, Singapore, 24–29 February 2017. [Google Scholar]
  47. Chicaiza, J.; Piedra, N.; Lopez-Varga, J.; Tovar-Caro, S. Recommendation of open educational resources. An approach based on linked open data. In Proceedings of the 2017 IEEE Global Engineering, E. Education Conference (EDUCON), Athens, Greece, 25–28 April 2017; pp. 1316–1321. [Google Scholar]
  48. Bhumichitr, K.; Channarukul, S.; Saejiem, N.; Jiamthapthaksin, R.; Nongpong, K. Recommender Systems for university elective course recommendation. In Proceedings of the 2017 14th International Joint Conference on Computer Science and Software Engineering (JCSSE), Nakhon Si Thammarat, Thailand, 12–14 July 2017; pp. 1–5. [Google Scholar]
  49. El Mabrouk, M.; Gaou, S.; Rtili, M.K. Towards an Intelligent Hybrid Recommendation System for E-Learning Platforms Using Data Mining. Int. J. Emerg. Technol. Learn. 2017, 12, 52–76. [Google Scholar] [CrossRef] [Green Version]
  50. Dang, Q.V. Implementing an individualized recommendation system using latent semantic analysis. In Proceedings of the 6th International Conference on Information and Education Technology (ICIET 2018), Osaka, Japan, 6–8 January 2018; ACM: New York, NY, USA, 2018; pp. 239–243. [Google Scholar]
  51. Dwivedi, S.; Roshni, V.S.K. Recommender system for big data in education. In Proceedings of the 2017 5th National Conference on E-Learning and E-Learning Technologies (ELELTECH), Hyderabad, India, 3–4 August 2017; pp. 1–4. [Google Scholar]
  52. Li, Y.; Li, H. MOOC-FRS: A new fusion recommender system for MOOCs. In Proceedings of the 2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chongqing, China, 25–26 March 2017; pp. 1481–1488. [Google Scholar]
  53. Yan, Y.; Hara, K.; Kazuma, T.; He, A. A Method for Personalized C Programming Learning Contents Recommendation to Enhance Traditional Instruction. In Proceedings of the 2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA), Taipei, Taiwan, 27–29 March 2017; pp. 320–327. [Google Scholar]
  54. Gulzar, Z.; Leema, A.; Deepak, G. PCRS: Personalized Course Recommender System Based on Hybrid Approach. Procedia Comput. Sci. 2018, 125, 518–524. [Google Scholar] [CrossRef]
  55. Baskota, A.; Ng, Y.K. A Graduate School Recommendation System Using the Multi-Class Support Vector Machine and KNN Approaches. In Proceedings of the 2018 IEEE International Conference on Information Reuse and Integration, Salt Lake City, UT, USA, 7–9 July 2018. [Google Scholar]
  56. Bozyiğit, A.; Bozyiğit, F.; Kilinç, D.; Nasiboğlu, E. Collaborative Filtering based Course Recommender using OWA operators. In Proceedings of the 2018 International Symposium on Computers in Education (SIIE), Jerez, Spain, 19–21 September 2018; pp. 1–5. [Google Scholar]
  57. Cheng, B.; Zhang, Y.; Shi, D. Ontology-Based Personalized Learning Path Recommendation for Course Learning. In Proceedings of the 9th International Conference on Information Technology in Medicine and Education (ITME), Hangzhou, China, 19–21 October 2018; pp. 531–535. [Google Scholar]
  58. El Fazazi, H.; Qbadou, M.; Sahi, I.; Mansouri, K. Personalized recommender system for e-Learning environment based on student’s preferences. Int. J. Comput. Sci. Netw. Secur. 2018, 18, 173–178. [Google Scholar]
  59. Zhu, H.; Liu, Y.; Tian, F.; Ni, Y.; Wu, K.; Chen, Y.; Zheng, Q. A Cross-Curriculum Video Recommendation Algorithm Based on a Video-Associated Knowledge Map. IEEE Access 2018, 6, 57562–57571. [Google Scholar] [CrossRef]
  60. Li, H.; Li, H.; Zhang, S.; Zhong, Z.; Cheng, J. Intelligent learning system based on personalized recommendation technology. Neural Comput. Z Appl. 2018, 31, 1–8. [Google Scholar] [CrossRef]
  61. Bauman, K.; Tuzhilin, A. Recommending Remedial Learning Materials to the Students by Filling their Knowledge Gaps. In Proceedings of the International Workshop on Educational Recommender Systems (EdRecSys), Omaha, NE, USA, 13 October 2016. [Google Scholar]
  62. Lessa, L.F.; Brandao, W.C. Filtering Graduate Courses based on LinkedIn Profiles. In Proceedings of the WebMedia 2018, Salvador, Brazil, 16–19 October 2018. [Google Scholar]
  63. Nadeem, M.; Dustin, S.; Shane, M. Neural Educational Recommendation Engine (NERE). In Proceedings of the 2018 IEEE International Conference on Data Mining Workshops (ICDMW), Singapore, 17–19 November 2018. [Google Scholar]
  64. Zhao, Q.; Wang, C.; Wang, P.; Zhou, M.; Jiang, C. A Novel Method on Information Recommendation via Hybrid Similarity. IEEE Trans. Syst. Man. Cybern. Syst. 2018, 48, 448–459. [Google Scholar] [CrossRef]
  65. Campos, R.; dos Santos, P.; Oliveira, J. Web-Based Recommendation System Architecture for Knowledge Reuse in MOOCs Ecosystems. In Proceedings of the 2018 IEEE R. International Conference on Information Reuse and Integration (IRI), Salt Lake City, UT, USA, 7–9 July 2018; pp. 193–200. [Google Scholar]
  66. Hinz, V.T.; Pimenta, M.S. Integrating Reputation to Recommendation Techniques in an e-learning Environment. In Proceedings of the 2018 17th International Conference on Information Technology Based Higher Education and Training (ITHET), Olhao, Portugal, 26–28 April 2018; pp. 1–6. [Google Scholar]
  67. Huang, X.; Tang, Y.; Qu, R.; Li, C.; Yuan, C.; Sun, S.; Xu, B. Course Recommendation Model in Academic Social Networks Based on Association Rules and Multi–similarity. In Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design (CSCWD), Nanjing, China, 9–11 May 2018; pp. 277–282. [Google Scholar]
  68. Huo, Y.; Xiao, J.; Ni, L.M. Towards Personalized Learning Through Class Contextual Factors-Based Exercise Recommendation. In Proceedings of the 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS), Singapore, 11–13 December 2018; pp. 85–92. [Google Scholar]
  69. Ibrahim, M.E.; Yang, Y.; Ndzi, D.; Yang, G.; Almaliki, M. Ontology-based personalised course recommendation framework. IEEE Access 2018, 7, 5180–5199. [Google Scholar] [CrossRef]
  70. Dahdouh, K.; Dakkak, A.; Oughdir, L.; Ibriz, A. Large-scale e-learning recommender system based on Spark and Hadoop. J. Big Data 2019, 6, 2. [Google Scholar] [CrossRef]
  71. Gianotti, R.C.; Cazella, S.C.; Behar, P.A. A Model for Integrating Personality Traits into an Educational Recommender System. In Proceedings of the 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), Maceio, Brazil, 15–19 July 2019; pp. 383–385. [Google Scholar] [CrossRef]
  72. Huang, Z.; Liu, Q.Z.; Yin, Y.; Chen, E.; Gao, W.; Hu, G. Exploring Multi-Objective Exercise Recommendations in Online Education Systems. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM ‘19), Beijing, China, 3–7 November 2019; Association for Computing Machinery: New York, NY, USA, 2019; pp. 1261–1270. [Google Scholar] [CrossRef] [Green Version]
  73. Britto, J.; Prabhu, S.; Gawali, A.; Jadhav, Y. A Machine Learning Based Approach for Recommending Courses at Graduate Level. In Proceedings of the 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli, India, 27–29 November 2019; pp. 117–121. [Google Scholar] [CrossRef]
  74. Jiang, W.; Pardos, Z.A. Time slice imputation for personalized goal-based recommendation in higher education. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys’19), Copenhagen, Denmark, 16–20 September 2019; Association for Computing Machinery: New York, NY, USA, 2019; pp. 506–510. [Google Scholar] [CrossRef]
  75. Jiang, W.; Pardos, Z.A.; Wei, Q. Goal-based Course Recommendation. In Proceedings of the 9th International Conference on Learning Analytics and Knowledge, Tempe, AZ, USA, 4–8 March 2019; Association for Computing Machinery: New York, NY, USA, 2019; pp. 36–45. [Google Scholar] [CrossRef] [Green Version]
  76. Shi, K.; Wen, B.; Wang, J.; Ouyang, Z. Research for Personalized Learning Reasource Recommendation Model based on Academic Emotions. In Proceedings of the 2019 IEEE International Conference on Computer Science and Educational Informatization (CSEI), Kunming, China, 16–19 August 2019; pp. 255–258. [Google Scholar] [CrossRef]
  77. Kamila, V.Z.; Subastian, E. KNN and Naive Bayes for Optional Advanced Courses Recommendation. In Proceedings of the 2019 International Conference on Electrical; Electronics and Information Engineering (ICEEIE), Denpasar, Indonesia, 3–4 October 2019; pp. 306–309. [Google Scholar] [CrossRef]
  78. Morsomme, R.; Alferez, S.V. Content-based Course Recommender System for Liberal Arts Education. Presented at the 12th International Conference on Educational Data Mining (EDM), Montreal, QC, Canada, 2–5 July 2019. [Google Scholar]
  79. Mrhar, K.; Abik, M. Toward a Deep Recommender System for MOOCs Platforms. In Proceedings of the 2019 3rd International Conference on Advances in Artificial Intelligence (ICAAI 2019), Istanbul, Turkey, 26–28 October 2019; Association for Computing Machinery: New York, NY, USA, 2019; pp. 173–177. [Google Scholar] [CrossRef]
  80. Pardos, Z.A.; Jiang, W. Designing for serendipity in a university course recommendation system. In Proceedings of the Tenth International Conference on Learning Analytics and Knowledge (LAK ‘20), Frankfurt, Germany, 23–27 March 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 350–359. [Google Scholar] [CrossRef] [Green Version]
  81. Prisco, A.; dos Santos, R.; Bez, J.L.; Tonin, N.; Neves, M.; Teixeira, D.; Botelho, S. A Facebook chat bot as recommendation system for programming problems. In Proceedings of the 2019 IEEE Frontiers in Education Conference (FIE), Covington, KY, USA, 16–18 October 2019; pp. 1–5. [Google Scholar] [CrossRef]
  82. Ren, Z.; Ning, X.; Lan, A.S.; Rangwala, H. Grade Prediction with Neural Collaborative Filtering. In Proceedings of the 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Washington, DC, USA, 5–8 October 2019; pp. 1–10. [Google Scholar] [CrossRef]
  83. Rizun, M. Concept of Recommender System for Building an Individual Educational Profile. In Proceedings of the 18th International Conference on Perspectives in Business Informatics Research (BIR 2019), Katowice, Poland, 23–25 September 2019; Volume Workshops and Doctoral Consortium, p. 2443. [Google Scholar]
  84. Samin, H.; Azim, T. Knowledge Based Recommender System for Academia Using Machine Learning: A Case Study on Higher Education Landscape of Pakistan. IEEE Access 2019, 7, 67081–67093. [Google Scholar] [CrossRef]
  85. Slim, A.; Hush, D.; Ojha, T.; Abdallah, C.; Heileman, G.; El-Howayek, G. An Automated Framework to Recommend A Suitable Academic Program; Course and Instructor. In Proceedings of the 2019 IEEE Fifth International Conference on Big Data Computing Service and Applications (BigDataService), Newark, CA, USA, 4–9 April 2019; pp. 145–150. [Google Scholar] [CrossRef]
  86. Vaidhehi, V.; Suchithra, R. An Enhanced Approach Using Collaborative Filtering For Generating Under Graduate Program Recommendations. In Proceedings of the 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP), Gangtok, India, 25–28 February 2019; pp. 1–6. [Google Scholar] [CrossRef]
  87. Zhang, M.; Liu, S.; Wang, Y. STR-SA: Session-based Thread Recommendation for Online Course Forum with Self-Attention. In Proceedings of the 2020 IEEE Global Engineering Education Conference (EDUCON), Porto, Portugal, 27–30 April 2020; pp. 374–381. [Google Scholar] [CrossRef]
  88. Mondal, B.; Patra, O.; Mishra, S.; Patra, P. A course recommendation system based on grades. In Proceedings of the 2020 International Conference on Computer Science; Engineering and Applications (ICCSEA), Gunupur, India, 13–14 March 2020; pp. 1–5. [Google Scholar] [CrossRef]
  89. Brik, M.; Touahria, M. Contextual Information Retrieval within Recommender System: Case Study, E-learning System. TEM J. 2020, 9, 1150–1162. [Google Scholar] [CrossRef]
  90. Na, C.; Xiuyuan, Z.; Na, L. Research on Network Personalized Learning Recommendation Algorithm Based on Knowledge Representation. In Proceedings of the 2020 International Conference on Information Science, Parallel and Distributed Systems (ISPDS), Xi’an, China, 14–16 August 2020; pp. 348–351. [Google Scholar] [CrossRef]
  91. Cheng, Y.; Bu, X. Research on Key Technologies of Personalized Education Resource Recommendation System Based on Big Data Environment. J. Phys. Conf. Ser. 2020, 1437, 012024. [Google Scholar] [CrossRef]
  92. Das, A.; Akour, M.A. Intelligent Recommendation System for E-Learning using Membership Optimized Fuzzy Logic Classifier. In Proceedings of the 2020 IEEE Pune Section International Conference (PuneCon), Pune, India, 16–18 December 2020; pp. 1–10. [Google Scholar] [CrossRef]
  93. Dhar, J.; Jodder, A. An Effective Recommendation System to Forecast the Best Educational Program Using Machine Learning Classification Algorithms. Ingénierie Systèmes Inf. 2020, 25, 559–568. [Google Scholar] [CrossRef]
  94. Gong, J.; Wang, S.; Wang, J.; Feng, W.; Peng, H.; Tang, J.; Yu, P. Attentional Graph Convolutional Networks for Knowledge Concept Recommendation in MOOCs in a Heterogeneous View. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘20), Virtual Event, Xi’an, China, 25–30 July 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 79–88. [Google Scholar] [CrossRef]
  95. Gordillo, A.; López-Fernández, D.; Verbert, K. Examining the Usefulness of Quality Scores for Generating Learning Object Recommendations in Repositories of Open Educational Resources. Appl. Sci. 2020, 10, 4638. [Google Scholar] [CrossRef]
  96. Chen, H.; Yin, C.; Li, R.; Rong, W.; Xiong, Z.; David, B. Enhanced learning resource recommendation based on online learning style model. Tsinghua Sci. Technol. 2020, 25, 348–356. [Google Scholar] [CrossRef]
  97. Liu, H.; Ding, J.; Yang, L.T.; Guo, Y.; Wang, X.; Deng, A. Multi-Dimensional Correlative Recommendation and Adaptive Clustering via Incremental Tensor Decomposition for Sustainable Smart Education. IEEE Trans. Sustain. Comput. 2020, 5, 389–402. [Google Scholar] [CrossRef]
  98. Hidayat, A.F.; Suwawi, D.D.J.; Laksitowening, K.A. Learning Content Recommendations on Personalized Learning Environment Using Collaborative Filtering Method. In Proceedings of the 2020 8th International Conference on Information and Communication Technology (ICoICT), Yogyakarta, Indonesia, 24–26 June 2020; pp. 1–6. [Google Scholar] [CrossRef]
  99. Tan, J.; Chang, L.; Liu, T.; Zhao, X. Attentional Autoencoder for Course Recommendation in MOOC with Course Relevance. In Proceedings of the 2020 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), Chongqing, China, 29–30 October 2020; pp. 190–196. [Google Scholar] [CrossRef]
  100. Wang, J.; Xie, H.; Au, O.T.S.; Zou, D.; Wang, F.L. Attention-Based CNN for Personalized Course Recommendations for MOOC Learners. In Proceedings of the 2020 International Symposium on Educational Technology (ISET), Bangkok, Thailand, 24–27 August 2020; pp. 180–184. [Google Scholar] [CrossRef]
  101. Li, J.; Ye, Z. Course Recommendations in Online Education Based on Collaborative Filtering Recommendation Algorithm. Complexity 2020, 2020, 6619249. [Google Scholar] [CrossRef]
  102. Liao, T.X.; Feng, X.H.; Sun, Y.L.; Wang, H.T.; Liao, C.; Li, J.B. Online Teaching Platform Based on Big Data Recommendation System. In Proceedings of the 5th International Conference on Information and Education Innovations (ICIEI 2020), London, UK, 26–28 July 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 35–39. [Google Scholar] [CrossRef]
  103. Isma’il, M.; Haruna, U.; Aliyu, G.; Abdulmumin, I.; Adamu, S. An Autonomous Courses Recommender System for Undergraduate Using Machine Learning Techniques. In Proceedings of the 2020 International Conference in Mathematics, Computer Engineering and Computer Science (ICMCECS), Ayobo, Nigeria, 18–21 March 2020; pp. 1–6. [Google Scholar] [CrossRef]
  104. Wan, N.; Wu, X.; Guo, S.; Yang, L.; Han, Q.; Yin, R. Research on Personalized Recommendation of learning Resource Based on Big Data of Education. In Proceedings of the 2020 IEEE 2nd International Conference on Computer Science and Educational Informatization (CSEI), Xinxiang, China, 12–14 June 2020; pp. 306–311. [Google Scholar] [CrossRef]
  105. Niyigena, J.P.; Jiang, Q. A Hybrid Model for E-Learning Resources Recommendations in the Developing Countries. In Proceedings of the 2020 4th International Conference on Deep Learning Technologies (ICDLT 2020), Beijing, China, 10–12 July 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 21–25. [Google Scholar] [CrossRef]
  106. Park, Y. Supporting Student Success in Computing Courses through Personalized Peer Advice. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (SIGCSE’20), Portland, OR, USA, 11–14 March 2020; Association for Computing Machinery: New York, NY, USA, 2020; p. 1329. [Google Scholar] [CrossRef] [Green Version]
  107. Chen, Q.; Yu, X.; Liu, N.; Yuan, X.; Wang, Z. Personalized Course Recommendation Based on Eye-Tracking Technology and Deep Learning. In Proceedings of the 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), Sydney, NSW, Australia, 6–9 October 2020; pp. 692–968. [Google Scholar] [CrossRef]
  108. Rismanto, R.; Rachmad, S.A.; Agusta, B.P.C. Research Supervisor Recommendation System Based on Topic Conformity. Int. J. Mod. Educ. Comput. Sci. 2020, 12, 26–34. [Google Scholar] [CrossRef]
  109. Pandey, S.; Lan, A.; Karypis, G.; Srivastava, J. Learning Student Interest Trajectory for MOOC Thread Recommendation. In Proceedings of the 2020 International Conference on Data Mining Workshops (ICDMW), Sorrento, Italy, 17–20 November 2020; pp. 400–407. [Google Scholar] [CrossRef]
  110. Sebbaq, H.; el Faddouli, N.; Bennani, S. Recommender System to Support MOOCs Teachers: Framework based on Ontology and Linked Data. In Proceedings of the 13th International Conference on Intelligent Systems: Theories and Applications, Rabat, Morocco, 23–24 September 2020; Association for Computing Machinery: New York, NY, USA, 2020. Article 18. pp. 1–7. [Google Scholar] [CrossRef]
  111. Suganya, G.; Mariappan, P.; Dubey, P.; Drolia, A.; Srihari, S. Subjective Areas of Improvement: A Personalized Recommendation. Procedia Comput. Sci. 2020, 172, 235–239. [Google Scholar]
  112. Morrow, T.; Hurson, A.R.; Sedigh Sarvestani, S. Algorithmic Support for Personalized Course Selection and Scheduling. In Proceedings of the 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC), Madrid, Spain, 13–17 July 2020; pp. 143–152. [Google Scholar] [CrossRef]
  113. Tavakoli, M.; Mol, S.T.; Kismihók, G. Labour Market Information Driven; Personalized; OER Recommendation System for Lifelong Learners. In Proceedings of the 12th International Conference on Computer Supported Education, Online, 2–4 May 2020. [Google Scholar]
  114. Wang, C.; Zhu, H.; Zhu, C.; Zhang, X.; Chen, E.; Xiong, H. Personalized Employee Training Course Recommendation with Career Development Awareness. In Proceedings of the WWW ‘20: The Web Conference 2020, Taipei, Taiwan, 20–24 April 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 1648–1659. [Google Scholar] [CrossRef]
  115. Wu, W.; Wang, B.; Liu, Y.; Yin, L.; Zheng, W. Higher Education Online Courses Personalized Recommendation Algorithm Based on Score and Attributes. J. Phys. Conf. Ser. 2020, 1673, 012025. [Google Scholar] [CrossRef]
  116. Pan, X.; Li, X.; Lu, M. A MultiView Courses Recommendation System Based on Deep Learning. In Proceedings of the 2020 International Conference on Big Data and Informatization Education (ICBDIE), Zhangjiajie, China, 23–25 April 2020; pp. 502–506. [Google Scholar] [CrossRef]
  117. Yin, S.; Yang, K.; Wang, H. A MOOC Courses Recommendation System Based on Learning Behaviours. In Proceedings of the ACM Turing Celebration Conference—China (ACM TURC’20), Hefei, China, 22–24 May 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 133–137. [Google Scholar] [CrossRef]
  118. ONTSI. Indicadores Destacados de la Sociedad de la Información; Ministerio de Economía y Empresas: Madrid, Spain, 2019. [Google Scholar]
  119. Rivera, A.; Tapia-Leon, M.; Luján-Mora, S. Recommendation Systems in Education: A Systematic Mapping Study 2018. In Proceedings of the International Conference on Information Technology and Systems (ICITS 2018), Libertad, Ecuador, 10–12 January 2020; Springer: Cham, Switzerland; Volume 721. [Google Scholar] [CrossRef]
Electronics 10 01611 g001 550
Figure 1. Phases of systematic review of literature.
Figure 1. Phases of systematic review of literature.
Electronics 10 01611 g001
Electronics 10 01611 g002 550
Figure 2. Flow chart showing study selection.
Figure 2. Flow chart showing study selection.
Electronics 10 01611 g002
Electronics 10 01611 g003 550
Figure 3. Geographical distribution of the articles included in the systematic review.
Figure 3. Geographical distribution of the articles included in the systematic review.
Electronics 10 01611 g003
Electronics 10 01611 g004 550
Figure 4. Number of articles included in the systematic review per year.
Figure 4. Number of articles included in the systematic review per year.
Electronics 10 01611 g004
Table
Table 1. Purpose of the search questions.
Table 1. Purpose of the search questions.
QuestionPurpose
Q1At what type of education are recommendation systems aimed, and which areas do they cover?To learn about fields of education and types of elements that are recommended, so as to determine which areas can be explored.
Q2What type of user is the RS aimed at?To determine the users and which of their characteristics are taken into account in the RSs.
Q3What is the developmental approach used in RSs in the field of education?To provide guidance about the most commonly used techniques in developing recommendation systems in education.
Q4On what type of platform is the RS developed: web or mobile?To determine the trend in the types of recommendation platform for education.
Table
Table 2. Details of the search.
Table 2. Details of the search.
SourceSearch ChainRecords
ACM JournalTitle = recommend* system*
Full Text = education
Author keyword = (“recommend* system*”) + “education“		808
IEEE Xplore((“recommend* system*”) AND education)1006
Scopus(TITLE-ABS-KEY ((“recommend* system*”))AND KEY((“recommend* AND system*”) AND (education)))808
WoSTopic (“recommend* system*”) AND “education“315
Total records 2937
Table
Table 3. Inclusion and exclusion criteria.
Table 3. Inclusion and exclusion criteria.
Criteria
InclusionExclusion
Complete documentsPublications in languages other than English.
Publications in journals or congressesPublications such as: letters to the editor, reviews, etc.
Title and abstract with words used in the searchPublications in areas other than education
Unavailable publications
Table
Table 4. Quality metrics.
Table 4. Quality metrics.
MetricsValueWeight
About the text of the article itselfM1: The abstract provides information and a balanced summary of what was done and what was found0/11
M2: Give the eligibility criteria and the sources and methods of selection of participants0/11
M3: Provides system architecture and details of components0/11
M4: Provides details on the validation of the system0/11
M5: Provides results according to objectives, constraints, and analysis0/11
M6: Provides an accurate and unbiased discussion0/11
M7: References0/11.5
Other Quality MetricsM8: Publication type (Conference/Journal)0/11
M9: Innovation1/2/31.5
M10: Number of references0/1/2/31
Table
Table 5. Percentage number of articles according to type of education.
Table 5. Percentage number of articles according to type of education.
Type of EducationNumber of Articles% of Articles
Formal5859.18
Non-formal3333.67
Formal/Non-formal77.14
Table
Table 6. Percentage number of articles according to type of user.
Table 6. Percentage number of articles according to type of user.
Type of UserNumber of Articles% of Articles
Students7678%
Learners67%
User in academic22%
Teachers11%
Students/Teachers11%
N/S1111%
Table
Table 7. Percentage number of articles according to recommendations elements.
Table 7. Percentage number of articles according to recommendations elements.
Recommendation’s ElementsNumber of Articles% of Articles
Academic advice44.08
Courses3333.67
Educational program22.04
Elective degree courses55.1
Learning resources3737.76
Multi-objective exam11.02
Online learning33.06
Papers11.02
Pedagogical resources11.02
Postgraduate courses22.04
Professions11.02
Programming problems11.02
Study sequence/syllabuses55.1
Teaching practice resources11.02
Universities11.02
Table
Table 8. Percentage number of articles according to developmental approach.
Table 8. Percentage number of articles according to developmental approach.
Developmental ApproachNumber of Articles% of Articles
Collaborative3232.99
Content-based22.06
Context-aware22.06
Knowledge-based1616.49
Hybrid2020.62
Machine Learning1919.59
Semantic11.03
Probabilistic Model11.03
Graphs11.03
Fuzzy logic11.03
Click-through rate prediction11.03
Table
Table 9. Percentage number of articles according to type of platform.
Table 9. Percentage number of articles according to type of platform.
Type of PlatformNumber of Articles% of Articles
Online2121.43
Chatbot11.02
Mobile11.02
Not Specified7576.53
Table
Table 10. Highlights of selected articles according to quality metrics.
Table 10. Highlights of selected articles according to quality metrics.
REF.UserEducational LevelMachine LearningMetrics
[27]Programming professionalsLifelong learningSupervisedAccuracy/Recall
[59]StudentsHigher education-Accuracy/Recall
[60]LearnerGeneral-Accuracy/MAE *
[69]StudentsHigher educationSupervisedAccuracy/Recovery
[70]LearnerHigher education-Relevance score
[72]StudentsHigher educationUnsupervisedNDCG **, MAE and F1 ***
[75]StudentsHigher educationReinforcementAccuracy/F1
[84]Academic’s userHigher educationSupervisedNDCG
[97]LearnerGeneralUnsupervisedAccuracy/Recall/F1
MAE *: Mean absolute error; NDCG **: Normalized Discounted Cumulative Gain; F1 ***: Harmonic mean between precision and recall.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Urdaneta-Ponte, M.C.; Mendez-Zorrilla, A.; Oleagordia-Ruiz, I. Recommendation Systems for Education: Systematic Review. Electronics 2021, 10, 1611. https://doi.org/10.3390/electronics10141611

AMA Style

Urdaneta-Ponte MC, Mendez-Zorrilla A, Oleagordia-Ruiz I. Recommendation Systems for Education: Systematic Review. Electronics. 2021; 10(14):1611. https://doi.org/10.3390/electronics10141611

Chicago/Turabian Style

Urdaneta-Ponte, María Cora, Amaia Mendez-Zorrilla, and Ibon Oleagordia-Ruiz. 2021. "Recommendation Systems for Education: Systematic Review" Electronics 10, no. 14: 1611. https://doi.org/10.3390/electronics10141611

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop