Recommendation Systems for Education: Systematic Review

: Recommendation systems have emerged as a response to overload in terms of increased amounts of information online, which has become a problem for users regarding the time spent on their search and the amount of information retrieved by it. In the ﬁeld of recommendation systems in education, the relevance of recommended educational resources will improve the student’s learning process, and hence the importance of being able to suitably and reliably ensure relevant, useful information. The purpose of this systematic review is to analyze the work undertaken on recommendation systems that support educational practices with a view to acquiring information related to the type of education and areas dealt with, the developmental approach used, and the elements recommended, as well as being able to detect any gaps in this area for future research work. A systematic review was carried out that included 98 articles from a total of 2937 found in main databases (IEEE, ACM, Scopus and WoS), about which it was able to be established that most are geared towards recommending educational resources for users of formal education, in which the main approaches used in recommendation systems are the collaborative approach, the content-based approach, and the hybrid approach, with a tendency to use machine learning in the last two years. Finally, possible future areas of research and development in this ﬁeld are presented.


Introduction
The penetration of information and communications technologies (ICTs) has resulted in major social changes, marking the onset of an era that has been characterized by globalization, the information society and knowledge. Likewise, it has meant a challenge in different spheres of activity, one of which is education, in which they are incorporated into teaching and learning processes. The incorporation of ICTs has entailed a growth in Internet services, which in turn has been reflected in a major increase both in the amount and in complexity of the information available online. A problem for users has emerged as a consequence of the information overload in terms of the time they spend on their search and the amount of information retrieved by it. Being able to suitably and reliably ensure relevant, useful information is a determining factor when taking decisions.
Recommendation systems (RSs) have emerged in order to deal with this problem, with the purpose of helping users find what is genuinely relevant to their needs. According to previous study [1], the RSs are software tools to help users in the decision-making process by applying information filtering, data mining, and prediction algorithms. This offers each user a variety of choices and options according to his or her interests and preferences [2].
A classic way of categorizing the different types of RS was provided by Burke [3], who distinguished between six different classes of recommendation approaches: Collaborative Filtering (CF), Content-Based Filtering (CBF), knowledge-based filtering, context-based filtering, demographic filtering, and hybrid filtering.
For Herlocker [4], collaborative filtering systems are the simplest, they calculate the similarity between users, and they predict product ratings for the active user according education, target users are students, teachers, and academic advisors, and the recommended elements are educational materials, learning objects, papers, universities, and information such as that about courses, student performance, and the field of study [11].
Applying RSs to the field of education requires taking into account a broad set of variables that may include, among others, level of knowledge, competences, and learning styles on the part of students. Given the rapid evolution of these systems, it is necessary to be aware of the trend in the techniques used for development.
The aim of this systematic review is to obtain an overview of RSs in education, its fields of work, recommendation elements, and the techniques used to identify any gaps, while at the same time providing a suitable framework guideline for future research activities. The search for articles was conducted between the years 2015 and 2020, in the course of which 98 works were analyzed after setting out relevant search criteria.
The article is structured as follows: Section II explains how the method used in the systematic review was developed; Section III provides an analysis of records; and lastly, Section IV contains the conclusions and discussion about future work.

Materials and Methods
The phases suggested by [16] were taken as reference in order to conduct the systematic review of literature (SRL), and these are shown in the following diagram ( Figure 1). The purpose of this SRL is to determine any gaps in recommendation systems in education, especially in higher education, in its different fields of study and areas of knowledge, aspects related to the development of recommender systems, such as the filtering algorithms, and other techniques used and their validation, to obtain accurate information from the review that may contribute towards the objective that has been set out. Thus, the following search questions were defined, in which Table 1 specifies the purpose of each of them.

Q1
At what type of education are recommendation systems aimed, and which areas do they cover?
To learn about fields of education and types of elements that are recommended, so as to determine which areas can be explored.

Q2
What type of user is the RS aimed at?
To determine the users and which of their characteristics are taken into account in the RSs.

Q3
What is the developmental approach used in RSs in the field of education?
To provide guidance about the most commonly used techniques in developing recommendation systems in education.

Q4
On what type of platform is the RS developed: web or mobile?
To determine the trend in the types of recommendation platform for education. The purpose of this SRL is to determine any gaps in recommendation systems in education, especially in higher education, in its different fields of study and areas of knowledge, aspects related to the development of recommender systems, such as the filtering algorithms, and other techniques used and their validation, to obtain accurate information from the review that may contribute towards the objective that has been set out. Thus, the following search questions were defined, in which Table 1 specifies the purpose of each of them.

Review Protocol
Once the objectives had been defined and the search questions set out, the following repositories were used to search for articles: IEEE and ACM as the two main digital libraries of scientific content in the area of informatics and computing, and to complement the search, Web of Science (WoS) and Scopus were taken into account as they are two databases that have access to an important and wide number of applications in the different areas of knowledge.
The keywords were then selected by taking into account the initial mapping literature and the key words found in abstracts pertaining to them. The terms used were as follows: Recommender system, recommendation system, and education. The advanced repository option was used to fine-tune the results, which enables search chains to be defined via the use of logical operators, using the following for the search: "recommender system" OR "recommendation system" AND education; the asterisk (*) was also used to conduct searches, and this symbol was used as a catch-all, and helps to represent one or more characters in a given term. In the recommend* search, this might imply terms such as  Table 2. The search was confined to records between the years 2015 and 2020.

Q1
At what type of education are recommendation systems aimed, and which areas do they cover?
To learn about fields of education and types of elements that are recommended, so as to determine which areas can be explored.

Q2
What type of user is the RS aimed at?
To determine the users and which of their characteristics are taken into account in the RSs.

Q3
What is the developmental approach used in RSs in the field of education?
To provide guidance about the most commonly used techniques in developing recommendation systems in education.

Q4
On what type of platform is the RS developed: web or mobile?
To determine the trend in the types of recommendation platform for education. Table 2. Details of the search.

ACM Journal
Title = recommend* system* Full Text = education Author keyword = ("recommend* system*") + "education" 808 IEEE Xplore (("recommend* system*") AND education) 1006 Scopus (TITLE-ABS-KEY (("recommend* system*"))AND KEY(("recommend* AND system*") AND (education))) 808 WoS Topic ("recommend* system*") AND "education" 315 Total records 2937 The following PRISMA guidelines [17] were followed to select articles from the systematic review, and these suggested applying 4 steps: Identification, filtering, eligibility, and inclusion. Figure 2 represents the PRISMA-adapted flow chart [17], which shows details about the number of records in each selection phase of the articles identified that were included and excluded, and the criteria used to determine which primary studies would be included in the review.
In total, 2937 records were obtained when conducting the search of the sources selected, the distribution of which can be seen in Table 2. Once these initial records had been identified, a first filter was then applied in which any repeated studies were disregarded, ending up with 2537, to which the following inclusion and exclusion criteria were applied until obtaining 802 (see Table 3).
Abstracts of the articles that had been kept were then read, excluding those in which the subject of study failed to cover any recommendation systems in education. In total, 587 articles were excluded from this process, with 225 remaining for in-depth review. These were read in their entirety and 127 were ruled out, leaving 98 articles included for the systematic review.
A systematic review involves the selection of significant material according to the objective established in the review [18]. Prisma [17] provides guidance for this review, including a checklist that can be used to assess the quality of the review. Based on this checklist, we have defined 10 indicators (Table 4), which permit us to measure the quality of the articles, the first 7 of which are focused on the content of the article and others related to publication quality metrics.
Abstracts of the articles that had been kept were then read, excluding those in which the subject of study failed to cover any recommendation systems in education. In total, 587 articles were excluded from this process, with 225 remaining for in-depth review. These were read in their entirety and 127 were ruled out, leaving 98 articles included for the systematic review.  For the assessment of metrics related to bibliographic references, we have based the criteria on [18] to establish the criteria in terms of the number of references and the percentage of references that are up to date (last five years). For the number of citations, we have used the metrics obtained from Google Scholar.

Results
Once the articles included in the systematic review had been included , data mining then got underway, for which purpose those elements that provided a response to the research questions were identified when the documents were read (Table 1). How the articles were distributed according to the categories found in each approach were identified and quantitatively shown in the analysis conducted. Table A1 shows the information obtained from each article, in accordance with the questions asked in the course of the systematic review.
From the information in Tables A1 and A2 in Appendix A, according to the geographical distribution of the articles included in the systematic review, the regions with the highest number of publications are Asia, followed by North America and Europe ( Figure 3). Figure 4 highlights the trend towards an increase in the number of RSs, where growth in the last two years can be observed, representing 37% of the total. It also shows the number of articles by type of publication.

Types of Education Covered by the Recommendation Systems
After reading the articles in their entirety, it was then possible to identify which RSs are geared towards formal education and which to non-formal education. Each article can be identified in Table A1 together with the type of education covered, while at the same time it could be seen that of the total, 58 RSs referred to formal education-in particular, university studies-33 of the works focused on non-formal education, and 7 both on formal and non-formal education. The percentage of articles according to type of education can be observed in Table 5. In most cases, the area of knowledge to which the recommendations are geared is not specified, while the works [32,53,54,109,113] indicate that studies in their case recommend resources pertaining to the field of architecture and engineering.

Types of Education Covered by the Recommendation Systems
After reading the articles in their entirety, it was then possible to identify which RSs are geared towards formal education and which to non-formal education. Each article can be identified in Table A1 together with the type of education covered, while at the same time it could be seen that of the total, 58 RSs referred to formal education-in particular, university studies-33 of the works focused on non-formal education, and 7 both on formal and non-formal education. The percentage of articles according to type of education can be observed in Table 5. In most cases, the area of knowledge to which the recommendations are geared is not specified, while the works [32,53,54,109,113] indicate that studies in their case recommend resources pertaining to the field of architecture and engineering.
In formal education, there are some articles such as [21,30,31,45] that base their recommendations on learning style. The RSs of [30,35] use the Felder-Silverman learning styles for this purpose. In non-formal education, we highlight the articles [27,113], which recommending lifelong learning, with the latter using job competences to recommend courses.

Types of Education Covered by the Recommendation Systems
After reading the articles in their entirety, it was then possible to identify which RSs are geared towards formal education and which to non-formal education. Each article can be identified in Table A1 together with the type of education covered, while at the same time it could be seen that of the total, 58 RSs referred to formal education-in particular, university studies-33 of the works focused on non-formal education, and 7 both on formal and non-formal education. The percentage of articles according to type of education can be observed in Table 5. In most cases, the area of knowledge to which the recommendations are geared is not specified, while the works [32,53,54,109,113] indicate that studies in their case recommend resources pertaining to the field of architecture and engineering.
In formal education, there are some articles such as [21,30,31,45] that base their recommendations on learning style. The RSs of [30,35] use the Felder-Silverman learning styles for this purpose. In non-formal education, we highlight the articles [27,113], which recommending lifelong learning, with the latter using job competences to recommend courses.  In formal education, there are some articles such as [21,30,31,45] that base their recommendations on learning style. The RSs of [30,35] use the Felder-Silverman learning styles for this purpose. In non-formal education, we highlight the articles [27,113], which recommending lifelong learning, with the latter using job competences to recommend courses.
When analyzing the data obtained to determine what types of elements are subject to recommendation, it was noted that 96% of works focus on offering the service to postgraduate students and universities such as learning resources, courses, post-graduate studies, and universities. Of these articles, 78 are oriented specifically for students ( Table 6), most of which make suggestions for courses and educational resources, where only [26] focuses on teachers by recommending resources for teaching practice in the view of the increasing availability of such resources online and the resulting difficulty in locating them, and [50] focuses on both students and teachers, where the student receives suggestions for support material to study while teachers receive student profiles for review. In terms of the type of element recommended, it was noted that 37 works stand out in which RSs offer learning resources, and these recommendations are mostly given in accordance with user preferences; 133 RSs suggest courses; 5 recommend a sequence of courses/syllabuses; 5 recommend elective degrees courses; and the remainder focus on recommending papers, postgraduate courses, academic advice, and professions, among others. The articles [85,106,108] that give academic advice stand out. Likewise, articles [31,33,112] that recommend study sequence or syllabuses. The work proposed by [41] recommends jobs for students and young professionals based on job descriptions and user profiles. Table A1 shows the elements subject to recommendation in detail while Table 7 provides the percentage distribution according to recommendation element.

Developmental Approach of RSs in the Field of Education
When we asked about the approach used in developing RSs in the field of education, we found that they were mainly collaborative, content-based, knowledge-based, and hybrid, highlighting the use of ML techniques in the last two years. Table 8 shows the number of articles according to developmental approach.
As for distribution, the following was observed: 32 RSs use the collaborative approach, and the following most-used approaches are the hybrid approach with 20 RSs, ML with 19 RSs, and the knowledge-based approach with 18 RSs. Finally, two RSs use the contentbased approach, highlighting the work [69], which takes some features from LinkedIn to create the user profile, such as the user's geographical location and activities of interest, and represents the information using an ontology.
Ref. [52] proposes a MOOC-FRS SR based on user behavior on the MOOC platform for personalized course recommendation. The article [64] proposes to recommend courses in MOOCs, taking into account the existence of more than one provider in the recommendation process. It will also use the knowledge in these MOOCs ecosystems to improve the recommendation of courses through these platforms. The article [81] uses a chatbot to give the recommendations, while the article [104] uses a mobile application to give the recommendations.
The remaining articles analyzed offered proposals or system prototypes in which recommendation algorithms were implemented. Some used test data as a way in and contrasted their results by comparing them with other algorithms used.

Quality Metrics
To ascertain the assessment related to the quality metrics of each of the articles, the corresponding measurement was carried out according to the defined indicators ( Table 4). Each of the articles was classified on a scale from deficient to excellent, according to the significant contributions to our systematic review, evaluating them in terms of the degree of innovation, details of the proposal, validation, results, and analysis, as well as the references and number of citations to it. Table A2 shows the score obtained by each of the articles.
The articles classified as excellent and very good are selected and analyzed again to obtain information related to users, educational level, type of ML, and system validation. This result is shown in Table 10. The studies were conducted with students or learners (seven articles), academic users (one article), and one article with programmers, of which 67% are oriented to higher education and only one article is oriented to lifelong learning, whereby [27] indicates that the proposed system facilitates personalized lifelong learning for members of a community and promotes interactions between groups of learners.
Of these selected articles, six of which use ML, representing 67%, three use supervised ML, two unsupervised ML and one booster ML.
In relation to the metrics used to measure the performance of SRs, precision, which corresponds to the proportion of the recommendation that is relevant to the user, is used by 67%, followed by recall and F1.
For SR's validation, 89% of the articles use real data, and only article [59] uses artificial data. Articles [70,75,90] validate off-line, while [72] validate off-line and on-line. For the verification of SR performance, the most used metric is accuracy (67%) followed by F1, the harmonic mean between precision and recall.
Finally, analyzing the purpose of these articles, we observe that they have very varied objectives. Regarding the innovation appreciated, we can highlight [27], which recommends courses based on learning communities, [69] recommends courses based on their relevance to relate course profiles, learner profiles, and jobs, and [69] recommends courses based on their relevance to relate course profiles, learner profiles, and jobs. Ref. [72] recommends exercises adapted to each student according to the knowledge and results of previous exercises. Ref. [84] recommends courses for teachers, research supervision based on publications, research interests, and educational training.

Discussion and Conclusions
The incorporation of ICTs into education has marked changes in its processes, whether in distance learning or in support for the different processes through educational resources available online. These are growing rapidly, whereby an increase in RSs has been noted in this sphere of activity, especially as support in formal education.
From the analysis, it can then be observed that RSs take into account user preferences when making suggestions based on recommendations from similar users, while [21,30,35,45,53] make recommendations based on learning style, and [24,31,46,50,61,64] based on diagnosis/student progress and the knowledge group. Likewise, another element they take into account are user skills and/or competences [40,42,56,69] and competences related to work associated with their profile within Internet job search portals.
RSs also base their suggestions on learning style [25,35,96]. The proposal of [30] use a survey based on the Felder and Silverman Learning Styles Model (FSLSM) to determine this, and these learning styles are classified according to the following categories: Sensory, intuitive, visual, verbal, reflexive, sequential, and global.
In terms of RSs that use the collaborative approach, they are thought to evidence the cold-start problem, with works [36,41,56] suggesting a greater volume of data to improve performance, and [22,48] adding more parameters to the user profile, such as learning styles or reading tastes-in general, it is suggested that this be combined with other approaches in order to improve performance. Ref. [84] indicates that the use of deep learning techniques with collaborative filtering deals with the cold-start problem in recommender systems and [115] solves the cold-start problem using collaborative filtering system by adding classification information. Furthermore, [69,86,110,113] indicates that the use of ontology for the representation of user information helps to solve the cold-start problem.
Of the articles that use the hybrid approach, we can draw attention to [69], which states that the context should be incorporated in the user profile in order to improve performance, while [27,64] suggests incorporating social networks in the future. As for those that use the knowledge-based approach, the systems work with ontologies and semantics, recommending that information be gathered from a range of sources and be represented via ontologies.
In the systematic review, we have analyzed 98 articles related to the use of SRs in education, most of them in a formal education context, so we can suggest further study in non-formal education. Table 10 shows the trend in the use of ML over the last two years. These techniques are combined with different filtering approaches to improve recommendations and cold-start-related problems.
A feature found in the analysis of the articles is the heterogeneity of the data in the domain of education, where [69] indicates that the integration of data from multiple heterogeneous sources helps the system to improve recommendations. There is also a need to study algorithms based on a semantic approach in more detail, the idea of which would be to use ontological knowledge to describe the elements in order to obtain a detailed representation of their content. This may in turn contribute towards improving the results obtained from the recommendation in terms of relevance of the educational material suggested and, therefore, would improve the student's learning process.
In the validation of RSs, the source and size of the dataset must be considered. Different strategies are used to determine the quality of the recommendation, such as offline and online validation, with possibilities for expert assessment up to the use of multiple metrics.
The RSs subject to study mainly sought elements to be recommended in a single place, although the search for information should also be explored in a range of sources so as to be able to ensure a wide variety of elements and offer a better recommendation service. Social networks also constitute a potential means for searching for information via recommendation systems. Ref. [27] indicates that the use of social media could improve the problem of data sparsity. According to the report on prominent indicators from the information society issued by the Spanish National Observatory for Telecommunications and Information Society, the percentage use of social networks in Spain is 67.9% and in the European Union 64.9% [118].
To conclude, out of the 98 articles included in the systematic review, the questions posed could be answered to a large extent, where Q1, Q2, and Q3 were covered in their entirety. From the analysis of the articles, we can highlight the following:

•
According to the type of education, the SRs cover mostly formal education, especially oriented towards students.

•
As for the elements subject to recommendation, they are very varied, highlighting educational resources and courses.

•
The most commonly used development techniques are collaborative filtering, followed by RSs that combine different techniques. Similar systematic reviews, such as the one presented by [119], agree with this result, finding a gap in the use of intelligent techniques. It can be seen in our review how this potential area has been covered since 2019, where proposals for RSs with machine learning are presented.

•
The incipient use of ontology for the representation of information and the construction of user profiles can be observed.

•
In the analysis of question Q4, in which we asked about the type of platform used by the RSs, not much information was obtained, as 76% of the articles analyzed did not specify the type of platform on which they were to be implemented. However, from the work [119], we can observe 50% of RSs using a web platform.
In addition to the limitation found in question Q4, we can list the following: • When selecting the articles for analysis, we selected those where the focus was on higher education, because the interest of the researchers is adult education. In order to obtain a broader view, it is suggested to take into account all levels of education.

•
The evaluation metrics of the SRs were only analyzed in the selected studies according to the quality metrics. For further study, it is suggested to extend this analysis to all articles.
Some gaps were identified in the systematic review, which allows us to suggest that future work should focus on the following:

•
The development of hybrid systems, in particular, the use of intelligent techniques and the use of ontology and evaluate the performance of RSs. • User information should be explored with information from different sources, such as social media, to build a more complete profile beyond user preference and similar users.

Conflicts of Interest:
The authors declare no conflict of interest.