- freely available
Algorithms 2011, 4(3), 131-154; https://doi.org/10.3390/a4030131
Abstract: This paper analyzes how recommender systems can be applied to current e-learning systems to guide learners in personalized inclusive e-learning scenarios. Recommendations can be used to overcome current limitations of learning management systems in providing personalization and accessibility features. Recommenders can take advantage of standards-based solutions to provide inclusive support. To this end we have identified the need for developing semantic educational recommender systems, which are able to extend existing learning management systems with adaptive navigation support. In this paper we present three requirements to be considered in developing these semantic educational recommender systems, which are in line with the service-oriented approach of the third generation of learning management systems, namely: (i) a recommendation model; (ii) an open standards-based service-oriented architecture; and (iii) a usable and accessible graphical user interface to deliver the recommendations.
Recommender systems enable people to share their opinions and benefit from each other's experience . They can be defined as “any system that produces individualized recommendations as output or has the effect of guiding the user in a personalized way to interesting or useful objects in a large space of possible options” . Recommender systems were initially developed to support web users in their decision-making in daily life situations in terms of pre-selecting information that might be of interest to them, where they confronted situations without sufficient experience in the available alternatives [3,4].
Recommender technology has traditionally focused on e-commerce activities to select and suggest extra potential purchase to consumers, trying to ease the information search and the decision process . Most developed systems recommend entertainment products (e.g., movies, books, songs, etc.), usually from a commercial perspective, aimed to help users decide which products to buy (or consume). Successful applications of e-commerce recommenders are reported elsewhere .
Aforesaid success has motivated the implementation of recommender systems in the educational domain [7,8]. In this domain, the ultimate goal is that learners acquire knowledge and educators support the learning process. In other words, there are clear differences that impinge on how to design recommender systems for each domain .
In this paper we introduce the need to provide inclusive personalization in current e-learning settings and discuss how recommender systems can contribute to improve this support. In particular, we propose a semantic educational-oriented approach for building the so called “Semantic Educational Recommender Systems” (SERS), which entails identification of the following key components as requirements for their design: (i) a recommendation model; (ii) an open standards-based service-oriented architecture; and (iii) a graphical user interface. The paper ends with some conclusions.
2. Inclusive Personalization Support in Formal E-Learning Scenarios
Each individual has particular needs and requirements, and in the case of learning scenarios, other factors such as background, learning goals and their evolution over time need also to be considered. Educational institutions worldwide are currently pushed to support user-centered scenarios mediated by technology and have to respond appropriately in terms of personalized end-user services, which are intangible facilities provided to address the specific needs of learners, including the support to disabilities in higher and further education . However, there is a lack of research on generic solutions applicable at large scale to support the development of inclusive and personalized end-user services which meet learners' evolving needs; even though this need for personalization and inclusive support in the learning process is widely requested in the literature [11–15].
Most universities use learning management systems (LMS) to support their teaching and learning process online [16,17]. Many LMS are available in the e-learning context (the reader interested can consult available comparative websites, such as Edutools ), both proprietary (e.g., Blackboard/WebCT) and open source (e.g., Moodle, dotLRN, Sakai, ATutor). Each of them has their own particularities, but they all serve the same general purpose, i.e., to provide web based functionalities to support the learning process and some management facilities for both the educator and the educational institution to control it. LMS share some basic functionality (e.g., forums, file storage, calendar) and can provide diverse support for deploying standards-based contents, which cover not just the course contents but the learner and learning processes as well (see Table 1 below). However, current LMS settings do not properly cover personalization  and accessibility issues . According to Dagger et al. , the next generation of LMS is focused on service-oriented architectures where external educational web based services can interoperate with the LMS , but the same issues (personalization and inclusiveness) still remain in these new settings.
Moreover, in line with advances on the Semantic Web , a wide set of specifications exist to support the semantic description and reusability of learning items in several platforms. There are standards that impinge on personalization and accessibility issues, which cover user models, learning scenarios, interaction preferences, device capabilities and metadata for specifying the delivery of resources to meet users' needs . A summary of previous compilations on these standards is reported elsewhere . Table 1 lists them in alphabetical order and compiles the acronym, the full name and the main objective of the standard to offer an inclusive personalized support in the learning process.
Thus, efforts are being made towards building the third generation of LMS following a fully service-oriented approach. Moreover, the offer of a wide set of specifications contributes to lead this efforts towards the semantic web. Therefore, it is expected that the required semantic support for information exchange among components will be available in the near future. In this context, solutions that allow offering personalized inclusive support in service-oriented and standards-based LMS are of value. Bearing this in mind, and from our previous experience in providing standards-based personalized scenarios on top of LMS , we came up with a proposal focused on developing Semantic Educational Recommender Systems (SERS), which are described in the next sections.
3. Semantic Educational Recommender Systems
In this section we present the concept or SERS. First, we comment on related work on recommender systems in formal educational settings. Next, we introduce the SERS approach, which is compared with existing educational recommender systems. Finally, we outline how the design of the recommendations is managed in the SERS approach, which combines user centered design methods and web mining analysis.
3.1. Related Works
Recommender systems have shown their success in many domains where information overload exists . This has motivated their application in e-learning settings. Two perspectives can be considered when developing recommender systems in the educational domain: (i) a top-down approach suitable for formal e-learning, where the structure, learning materials and learning plans are maintained by domain professionals; and (ii) a bottom-up approach suitable for non-formal e-learning as it takes place in a self directed way within learning networks, where learners interact with information sources shared in the network . Moreover, formal and non-formal e-learning scenarios offer a rather different context that has to be taken into account by recommender systems in order to offer personalized information to individual learners .
To apply recommender systems for the educational domain, and specially, for formal e-learning scenarios, an important issue is to determine whether both domains, i.e., e-commerce and education, require a different design of the recommender. Drachsler et al. state that both domains share the need to manage design activities before the runtime and also during the maintenance of the system . However, recommender systems in education are quite different from recommender systems in e-commerce, as they have to consider not only the learners or the educator's preferences for a certain material, but also how this material may help them to achieve their goals .
A critical issue that both domains share is the need of building rich datasets to facilitate their development and benchmarking. Dataset-driven research can be followed to build recommender systems for educational scenarios. It has not been until very recently that datasets with information related to educational content have started to be compiled. In particular, it has been explored in the literature datasets that capture learner interactions with tools and resources , although they cannot be considered fully educational data sets, as the datasets reported deal with (i) scientific papers; (ii) resources (documents, videos, links) from a work-integrated learning system; (iii) contributions from a community of knowledge sharing that aggregates Web 2.0 contributions from a range of remote web based services such as Delicious, Youtube, Flickr, Slideshare, and Twitter; (iv) contents from a learning portal for organic agriculture educators; (v) advanced graphical metadata-based access to learning resources in architecture; and (vi) open educational resources. Only the last three focus on educational contents and are relevant from a formal e-learning scenario point of view.
Thus, although educational recommender systems (ERS) share the same key objectives as e-commerce recommenders (i.e., helping users to select the most appropriate item from a large information pool), there are some particularities that make not possible to directly apply existing solutions from those systems to recommenders for the educational domain. For instance, Manouselis et al. discussed in  the particular educational users' tasks and showed that they are different from the typical recommendation tasks that Herlocker et al.  have described for generic recommender systems.
In fact, recommender systems are strongly domain dependent , so the characteristics of the educational domain should be taken into account. However, up to now, and as reported in existing reviews of the state of the art (e.g., ), ERS are not taking advantage of the semantic descriptions provided by the specifications and standards currently used in the educational domain, such as those compiled in Table 1. Moreover, they mainly focus on recommending learning objects from educational repositories  and do not consider the wide range of recommendation opportunities available in LMS. In turn, semantic recommender systems are being developed, whose common characteristic is the use of profiles to represent the users' long-term information needs and interests in terms of semantic descriptions .
3.2. The SERS Approach
As introduced in the previous section, in our research we focus on deploying semantic recommender systems in the educational domain. From the literature, semantic recommender systems are those whose performance are based on a knowledge base usually defined as a concept diagram (like a taxonomy or thesaurus), or an ontology . In our approach, the idea is to extend the functionality of standards-based LMS with adaptive navigation support in order to provide learners with personalized and inclusive scenarios. To this end, the approach considers standards and specifications to support the semantic modeling of the knowledge required for the recommendation process . Thus, we coined the concept of Semantic Educational Recommender Systems (SERS) . This concept entitles the following terms:
Recommender system, which as previously introduced, is a tool to help users in decision making processes in environments with information overload by offering recommendations as links, thus providing adaptive navigation support. Recommenders apply recommendation algorithms, which facilitate the automation of the recommendation process.
Semantic, as it allows recommendation designers to provide high-level descriptions of the recommendations, which are also interpretable by the recommender system, both in the process of selecting and delivering the most appropriate recommendations and in the process of automatically generating new recommendations. Moreover, these descriptions allow explaining users their rationale, thus supporting trust in the system. And no less important, they facilitate the information exchange among the software components involved in the recommendation process that is orchestrated in the context of a service-oriented architecture.
Educational, since it is applied in the educational domain, which has specific particularities (e.g., recommendations should be guided by educational criteria and not just by the learners' preferences) and potentially, they can cover any action (e.g., read or contribute) to any object in the LMS, so the involvement of the education is needed to identify the recommendations that are appropriate for their formal e-learning scenarios.
Hence, based on educational criteria, SERS are characterized by guiding learners in their interaction within e-learning platforms. To this end, they provide personalized and inclusive recommendations that could target any possible action within an LMS. These recommendations are semantically described and can be obtained thanks to the information exchange among the different components involved in the process of generating and delivering the recommendations. In order to clarify the differences between ERS and SERS, a comparison regarding eight key features is provided in Table 2.
Bearing the features compiled in Table 2 in mind, three key issues are required in the SERS approach:
A recommendation model that supports the recommendation of actions on LMS objects (F3) and semantically characterizes the recommendations (F4) in order to bridge the gap between their description by the educator and the recommender logic when delivering recommendations in the running course.
An open standards-based service-oriented architecture, which serves to support the interoperability of the SERS with existing LMS through web services (F5). To this, SERS can take advantage of standards and specifications to describe the information exchange (F6). Further, it is advisable that SERS become a service integrated into the e-learning platform.
A graphical user interface to deliver the recommendations to the learners in a usable and accessible way (F7). This graphical user interface is to be developed within the corresponding LMS presentation layer and allows describing the recommendation and its features as informed by the recommendation model (F8).
The above three elements are essential to properly support a semantically oriented recommendation process in education, which covers the management of recommendations by the educator  and complements the current functionality of LMS. Hence, SERS provide the infrastructure for the delivery of recommendations, but they are not involved in understanding and identifying recommendation opportunities in formal e-learning scenarios, which lie on educational issues that have to be identified by educators. For this, the TORMES methodology has been defined  and it is presented in the next subsection.
3.3. Designing the Recommendations with TORMES Methodology
When it comes to involve educators in the recommendations design process, there are two key issues to consider: (1) the elicitation of educational oriented recommendations; and (2) the evaluation of the recommender (and the recommendations designed) for its educational purpose. To this end, a methodology has been proposed named TORMES, which stands for Tutor-Oriented Recommendation modeling for Educational Systems . TORMES is based on the ISO standard 9241-210 and aims to involve educators in the process of designing educationally oriented recommendations through user centered design methods, which are complemented with data mining analysis. In order to elicit educational oriented recommendations, and following the ISO standard 9241-210, the following four activities are to be considered:
description of the context of use through individual interviews with educators, which is complemented with data mining analysis from past courses;
requirements specification through the scenario based approach that is used to extract knowledge from the educators and allows identifying recommendation opportunities that can be provided in the scenario outlined to improve the learning experience;
creation of design solutions in terms of semantically described recommendations, where educators are involved in validating and refining the initial set of recommendations elicited from the scenarios in the previous activity through focus groups; and
evaluation of the recommendations designed against the requirements by rating their relevance and classifying them in terms of their scope.
Through this step-wise process, the SERS approach is able to deal with the particularities of the educational domain. Moreover, the semantic descriptions of the recommendations can provide explanations to the learners about the recommendations offered. This provision of explanations has already been acknowledged in the literature as beneficial for user experience with the recommender .
To evaluate the recommender system and the recommendations designed for it, four dimensions have been proposed. These dimensions are: (i) integration of the SERS into the e-learning system; (ii) impact of the recommendations on the users (both learners and educators); (iii) value of recommender systems' quality properties such as utility, serendipity, coverage, etc.; and (iv) impact of the recommendations delivery on learners' interactions.
A large scale experience has been carried out with 377 learners to design educational oriented recommendations and evaluate them with the aforementioned dimensions in an e-learning system, which results from integrating the SERS into Willow. Willow is an adaptive computer assisted assessment system that is able to automatically process short learner's answers or essays written in natural language and provide feedback on the responses given . Some results have already been reported elsewhere. For instance, in  we have reported the evaluation of the impact of this integration in the learning process, showing that a positive statistical impact was detected on indicators related to the engagement in the course, the learning effectiveness and efficiency, and the knowledge acquisition. We also analyzed the impact on the user experience in terms of the consistency usability principle, showing that the recommendations kept the high perception of the users regarding usability and satisfaction. However, it should be investigated if the recommendations do have an impact on users' perception when the usability of the original system is low. In this experience, the results showed high degree of usability due to the dialogue metaphor in which the interaction is based in Willow. Hence, the participants benefited from the additional support provided by the recommendations, since an improvement is perceived on several types of indicators (i.e., engagement, learning efficiency, learning effectiveness and knowledge acquisition).
As previously introduced, data mining analysis can be used to support the design of the recommendations produced by the educators with the user centered design process. The detailed descriptions of the recommendations done by the educators (see Table 5 for an example of a recommendation described in terms of the recommendation model) can be used as input data for the mining process. In particular, the following analysis can be carried out:
Identify troublesome or promising situations, which helps the educator to think of appropriate recommendation needs . For instance, interaction data from previous courses can be analyzed to: (i) identify learners with some shared features that are not performing well in the course; (ii) course activities that have become a hindrance for a subset of learners, or (iii) LMS functionalities that are being misused in certain activities.
Tune the design of the recommendations proposed by educators while applying the user centered design methods . In particular, as it is discussed later in Section 4.1, recommendations are described in terms of some applicability conditions. That is, the applicability conditions describe when a recommendation should be delivered to the learner (i.e., what learner features and context characteristics should take place for a recommendation to be offered to a particular learner). For this, web usage mining can be used over past interaction data to tune the educator's design work with specific values for these conditions. The following adjustments can be done: (i) learn specific values for the attributes used as applicability conditions; (ii) find new conditions (attributes) that emerge from the interaction data; and (iii) remove conditions (attributes) suggested by the educator that seem not relevant from the interaction data analyzed.
Adjust the recommendations design after the course experience. Once the recommendations are designed and delivered to the learners, their performance can be analyzed to see if they have covered the educational criteria required. If that is not the case, the applicability conditions can be modified to try to improve the recommendations impact on the learning experience.
4. SERS Requirements
Three requirements have been proposed for developing suitable SERS: (i) a recommendation model to characterize the recommendations; (ii) an open standards-based service-oriented architecture to support the required interoperability among software components; and (iii) a graphical user interface to deliver the recommendations in a usable and accessible way. Each of them is described in this section.
4.1. Recommendation Model
A recommendation model has been proposed to cover the following objectives:
Support the educator in describing educational oriented recommendations;
Facilitate the delivery of the recommendations in the course following a rule-based approach;
Present information to the learner about the rationale behind the recommendations offered;
Provide semantic information that can be used to reason about which recommendations are appropriate for a given situation;
Facilitate the automation of the recommendations generation process, which draws on user modeling based on machine learning techniques.
With the above features the model is intended to bridge the gap between the recommendations description produced by the educator and the recommendations delivery within the running course. When educators describe the recommendations, they identify, among others, the conditions that have to be fulfilled at runtime (for example, the learner to be recommended has to have a sequential learning style, be using a device with audio capabilities and just downloaded the contents from lesson 2). A rule-based approach can be used to select those recommendations whose conditions match the current learner and her context at runtime. Learners receive the appropriate recommendations according to their individual features and current context as well as an explanation about the educational rationale for the recommendation offered.
Moreover, through the provided semantic information, the system may determine appropriate recommendations for a given situation. The interaction can be mined to infer which recommendation types are the most appropriate. For instance, if the given situation is at the beginning of the course, a common learners' behavior might be detected in the mining process. Similarly, there might be common situations in a collaborative situation.
The proposed recommendation model facilitates the management of the recommendations as it allows describing the following aspects:
What should be recommended: actions on objects available in the platform (i.e., answering a message in the forum);
How the recommendation should be communicated: proper usage of the language (e.g., formal vs. informal language);
When the recommendation should be offered: the user features as well as the user and course context have to be taken into account;
Why the recommendation is delivered: provides a justification of the recommendation;
Which are the recommendation characteristics: describes the recommendations in terms of meaningful attributes.
An element has been considered in the model for each of the above aspects. Moreover, each element involves several attributes. Details of the model are reported elsewhere . In order to have an overview, they are compiled and summarized in Table 3. In Table 5 an example of a recommendation described in terms of the model is provided. Following Table 3, it should be noticed that the proposed recommendation model considers the possible actions that may occur on any LMS object. As in any web-based system, an object is any item in the LMS that is linkable, that is, can be accessed through an HTML link. Since an LMS consist of items which are accessed through links, recommendations in SERS provide links to LMS items. Moreover, since some content goes with the link recommended, that content can specify what action is to be done on the linked object. Thus, when the learner receives a recommendation with a link to a particular object (e.g., a calendar event, a message in the forum, etc.), the recommendation content points to a specific action on a particular LMS object. Two kinds of actions can be recorded. On the one hand, passive actions, such as to read a specific event so the learner takes it into account in her planning or to read a message posted in a forum by a classmate that is related to the activity the user is working on, where information is given only from the system to the user. On the other hand, it is also possible to recommend active actions, in which users are suggested to produce a change on the information stored by the system, such as to change the contents of an event that the user has previously created or to post a reply to a message in the forum.
In order to cope with reusability and interoperability issues, the information managed by the above attributes can be supported by the standards and specifications compiled in Table 1, as explained in Table 4.
Table 4 shows that several standards and specifications exist that provide descriptions for the type and runtime information attributes. However, this mapping is not straight forward, as has already been acknowledged in literature; there are still some aspects for which different specifications overlap, while there are other aspects for which there are as yet no defined specifications . The content attributes should follow the W3C WGAG to make the information accessible to the users. For the justification element, both the educational view (i.e., IMS-RDCEO) and the accessibility view (W3C WCAG) are to be considered. Finally, for the semantic information attributes (i.e., category, stage, origin and relevance) there is currently no standard or specification available that can support their description, as they are very much dependant on the recommendation nature.
With all this in mind, an example is provided in Table 5 of a recommendation described in terms of the recommendation model and making use of existing standards and specifications. It corresponds to the recommendation 2 used as example in Section 4.3.
The above examples shows the description of a recommendation in terms of the elements and attributes defined in the recommendation model, and when appropriate, points to elements of the existing educational standards and specifications, such as the <interest>, <competency> elements in IMS-LIP, <environment> in IMS-LD, <rdceo> in IMS-RDCEO and <ScreenSize> in W3C CC/PP. Since there is no specification that deals with the interaction data, it is suggested to add an extension in the <extension> element in IMS-LIP. Moreover, the contents to be shown to the learner aimed to be readable. The link to the recommended object is built by adding the object identifier to the corresponding platform URL, for instance: http://www.platformhost.com/message-view?message_id=5637.
4.2. Open Standards-Based Service-Oriented Architecture
To facilitate the interaction between SERS and existing LMS a service-oriented architecture (SOA) is suitable. For instance, in the EU4ALL project (IST-2006-034478) a SOA interoperates with a SERS to provide personalized and inclusive support in formal e-learning scenarios . To clarify involved issues, in this section the EU4ALL SOA components that are required to support the recommendation functionality to be provided by the SERS are described.
Figure1 presents the operation flow among the components involved in the process of delivering the recommendations. It shows the web services (WSDL based) exchanged that relate to the recommendation process. In order to produce the required functionality, SERS can interact with other components such as the user model (UM), the device model (DM) and the LMS. The UM is in charge of collecting the information about the learner to support the personalization process, which can be described in terms of IMS-LIP, IMS-AfA (in particular the subpart that deals with the Accessibility for LIP–AccLIP) and ISO/IEC 24751 (in particular, the subpart that deals with the Personal Needs and Preferences–PNP). The DM stores the information about the device that the learner is using to select the recommendations. For this, the W3C CC/PP specification can be considered.
Moreover, a tool is required to manage the recommendations, either those manually generated by the educator with the TORMES methodology or automatically derived from the algorithms . This tool requires information from the LMS to be aware of the available objects, which are candidates for a recommendation. This means, for instance, that in order to define a recommendation for a forum message object in a specific course, the systems should provide the list of existing messages in the forums of that specific course. Thus, following the SOA approach, the LMS has to offer the corresponding web services to inform about the structural composition of the virtual spaces created. In this way, the recommendation designed can point to the appropriate object identifier in the LMS (i.e., pointer or object _id).
The process to deliver a recommendation starts when the SERS server receives a request from the LMS. After that, the SERS requests information from the UM, the DM and the LMS tracking component. With this collected information, the SERS identifies the available recommendations and selects the recommendations that match the current applicability conditions and restrictions. If the number of recommendations exceeds the maximum number of recommendations that can be shown in the device screen, then the top with higher relevance are selected and delivered to the LMS. As a result, the recommendations are offered to the learner through the LMS presentation layer as described in the next section (or e-mailed to the learner). Moreover, the LMS can provide feedback to the SERS about the interactions done by the learners on the recommendations offered, that is, if the recommendations have been followed or not by the learners. Moreover, learners have the possibility of providing some feedback on the perceived utility for each recommendation, by selecting one of the following three options: (i) the recommendation was useful; (ii) the recommendation may be useful, but not in the moment offered; and (iii) the recommendation is not useful at all. In case the learner provides this feedback through the LMS, it is sent to the SERS to be processed.
Next, a high-level description of the information needs that is required to be offered by each of the aforementioned components is done, following the web services presented in Figure 1. The head of arrows point to the component that offers the required web service. Table 6 compiles the identified web services. Regarding the LMS, two web services are required to provide access to both the structural and the interaction information. The UM should allow storing and retrieving the learner data. Since the device information is not affected by the recommendation process, the DM should only provide access to information about the devices. Finally, the SERS service should provide web services both to the SERS admin graphical user interface (GUI), to manage the creation of the recommendations, and to the LMS, to allow the exchange of information during the process of requesting a recommendation.
4.3. Graphical User Interface
As in any other recommender systems, a critical issue is how to present the recommendations to the user, in this case, to the learner. A common approach in ERS is to offer a list with the most relevant recommendations for the learner , which she has the freedom to follow or not. The information shown as the content of the recommendation and the way it is presented in the GUI influence the attitude of the learner towards the recommendations . For instance, providing explanations on how the recommendation has been produced increases the user trust of the system . Moreover, usability and accessibility criteria have to be followed when designing the GUI to be included in the LMS to deliver the recommendations. In particular, although the technologies to implement this GUI are very diverse and dependant on the front-end technology used in the LMS where the SERS is integrated, to offer recommendations in an inclusive way, the GUI should comply with the W3C WCAG (assuming that the LMS complies with them, too).
Thus, the GUI is to be integrated into the LMS presentation layer to show the recommendations and describe their features, as informed by the recommendation model. An example of a recommendation list to be shown in the LMS GUI is presented in Figure 2, where the second recommendation is the one used as example in Table 5.
Taking into account the recommendation model, a proposal to present a recommendation list in the LMS GUI was done in  and is reproduced in Figure 3. In the figure, the following elements can be identified: (i) the greetings; (ii) the recommendations (in this case, there are three recommendations in the list); (iii) the action suggested on the LMS object (in this case, it is highlighted the recommendation 2, which suggests the learner to read a message posted in the forum by Mary); (iv) the origin of the recommendation, where each origin is identified by a different icon at the start of the recommendation sentence; and (v) the justification and semantic information which is linked to another page that shows this information in detail.
Figure 4 shows the additional page that includes the detailed information of the recommendation in terms of the model, in particular, the semantic information (i.e., category, stage, origin, relevance), and the justification (i.e., rationale and explanation). In this way, learners are allowed to scrutinize the recommendation model in detail. Supporting scrutability of the user model seems to offer potential benefits in the learning process .
Although current LMS do not properly cover personalization and accessibility issues, and are still struggling to support the reusability requirements coming from the need of a pervasive usage of standards and specifications, efforts in the LMS field are being done towards building the third generation of LMS following a fully service-oriented approach. Moreover, as compiled in Table 1, a wide set of standards and specifications exist that can be used to describe characteristics of the recommendations, although in some cases there is overlapping of information and in others lack of required descriptions. These issues are being addressed in the research arena [17,20]. Thus, there is a need to provide the required semantic support for information exchange among components. In this context, solutions that allow offering personalized inclusive support in service-oriented and standards-based LMS are of value, especially if they support modularization of the required functionalities.
Standards and specifications can be used to represent long-term information needs for the recommendation process, and thus, provide the foundations for the semantic modeling of the knowledge required for the recommendation process. In this way, the following nine issues mentioned in  are addressed in the SERS approach to a greater or lesser extent: (i) inter-operability of system resources and homogeneity of the information representation; (ii) dynamic contextualization of user preferences; (iii) performance in social networks and collaborative filtering; (iv) communication processes between agents and between agents and users; (v) limitation of the cold start problem through inferences; (vi) semantically extended descriptions of user contextual factors; (vii) representation and description of different system elements; (viii) description of system logic by admitting the inclusion of a set of rules; (ix) descriptions enriched by web services to facilitate their discovery by software agents.
Bearing the above issues in mind, the SERS approach defines three requirements, namely a recommendation model, an open service-oriented architecture and a graphical user interface. These requirements are strongly interrelated due to the semantic descriptions shared among them, which rely on existing standards and specifications. However, there is also room to new standards and specifications that provide better descriptions of the educational functionalities provided within LMS, which support the interrelations when describing restrictions and applicability conditions, as well as the descriptions of the semantic information elements, that is, category, stage, origin and relevance. The recommendation model characterizes the recommendations in a way that specifies when they have to be delivered and how they are presented to the user, and thus, facilitates the interpretation of the semantics of the data exchanged. The service-oriented architecture supports the information exchange among the different components involved in the recommendation process. The graphical user interface is the communication channel with the learner and presents the information used by the components in a human readable way.
In this way, the learners' needs during the learning process are supported by providing educational oriented recommendations that take into account their interaction needs in the course to provide and effective, efficient and satisfactory learning experience. This is meant to attend the needs of every learner, including their disabilities, access preferences and context by considering web content accessibility guidelines (WCAG), along with learners' accessibility preferences in ISO PNP and IMS AccLIP and the characteristics of the devices used (CC/PP).
In this paper we have explored the requirements for the inclusive and personalization support required in formal e-learning scenarios. To cope with the distinctive issues that characterize the educational domain, we propose the SERS, which includes new conceptual and developmental approaches that extend the features provided by the current application of recommender systems in education. This proposal has been coined to describe those ERS that are developed to extend the adaptive capabilities of LMS (with adaptive navigation support) in an interoperable manner to support learners in a personalized and inclusive way. SERS consider all the potential actions in an LMS, beyond recommending learning resources or courses. For instance, active actions can be considered, such as contributing to a forum to promote self-reflection, which may be relevant from an educational point of view.
SERS provide the infrastructure for the delivery of recommendations, but it is not involved in understanding and identifying recommendation opportunities in formal e-learning scenarios, which are educational issues that can be identified by educators. For this, the TORMES methodology has been defined. TORMES supports the elicitation and design of suitable educational oriented recommendations, which are delivered by the SERS when appropriate learner's needs and context are met. Thus, SERS are characterized by guiding, based on educational criteria, learners in their interaction within LMS. The recommendations are semantically described and can be obtained thanks to the information exchange among the different components involved in the process of generating and delivering the recommendations. SERS development draws on the following three elements: (1) a recommendation model to semantically characterize the recommendations; (2) an open standards-based SOA to guide the integration of the SERS with existing LMS and additional components such as the UM and DM; and (3) a GUI to be integrated in the LMS presentation layer to deliver the recommendations to the learners.
The main difference of SERS with respect to existing ERS is the semantic interoperability among the different data available, such as user data (demographic, interaction preferences, learning styles, ability profile, learning needs, previous knowledge, interests, background, course outcomes, etc.), context data (device, environment) and course data (metadata of contents and instructional design) in terms of existing specifications and standards which allow for the interpretation of the semantics of the data. Interoperability is achieved during the information exchange among the different software components involved.
In summary, SERS extend the adaptive capabilities of existing LMS with adaptive navigation support to offer a personalized and inclusive guidance to learners in formal e-learning scenarios and have the following features:
Guide learners through the information overload and inexperience of alternatives in the LMS;
Reuse existing infrastructure at educational institutions since it offers a new component (i.e., the SERS) to be integrated with existing LMS in an interoperable way;
Manage the rich contextual information available in formal e-learning scenarios (i.e., diversity of actions on LMS objects).
SERS are to be exploited by the emerging generation of LMS, which are increasingly focused on service-oriented approaches and on incorporating semantic descriptions to achieve the required semantic support with information exchange among components. Moreover, they can also serve to cope with those personalized guiding needs that characterize the development of Personal Learning Environments (PLE) , where users apply Web 2.0-style services to create their own learning management tools .
|ADL SCORM||Sharable Content Object Reference Model||Defines the interrelationship of content objects, data models and protocols such that objects are sharable across systems that conform to the same model.|
|IEEE-LOM||IEEE Learning Object Metadata||Specifies which aspects of a learning object should be described and what vocabulary can be used in that description.|
|IMS-AfA||IMS Access for All||Defines and describes resource accessibility to match individual's requirements.|
|IMS-CP||IMS Content Packaging||Describes data structures that can be used to exchange data between systems that wish to import, export, aggregate, and disaggregate packages of content.|
|IMS-LD||IMS Learning Design||Describes the structure of tasks and activities assigning them to roles and the flow of units of learning; supports the use of a wide range of pedagogies in on-line learning.|
|IMS-LIP||IMS Learner Information Package||Collects information on learners, individually or in groups.|
|IMS-MD||IMS Metadata||Same as IEEE-LOM, since it adapts it.|
|IMS-QTI||IMS Question and Test Interoperability||Describes questions and tests to allow the interoperability of content with assessment systems.|
|IMS-RDCEO||IMS Reusable Definition of Competency or Educational Objective||Provides a data model for the definition of competences to create a common understanding of the skills that are presented as part of a learning system or a career plan.|
|ISO/IEC 24751||Individualized adaptability and accessibility in e-learning, education and training||Describes the needs and preferences of learners and provides a description of the relevant digital learning resources so that individual learning preferences and requirements of the learner can be met through user interface tools and appropriate digital learning resources.|
|ISO/IEC 24751||Individualized adaptability and accessibility in e-learning, education and training||Describes the needs and preferences of learners and provides a description of the relevant digital learning resources so that individual learning preferences and requirements of the learner can be met through user interface tools and appropriate digital learning resources.|
|W3C CC/PP||W3C Composite Capabilities/Preference Profiles||Describes the device capabilities and user preferences to guide the adaptation of content presented to a given device.|
|W3C WCAG||W3C Web Content Accessibility Guidelines||Defines and explains the requirements for making web-based information and applications accessible to a wide range of people with disabilities.|
|F1. Considers education criteria||yes||yes||A recommender system in the educational domain is bound to consider educational criteria directly or indirectly (i.e., in terms of context).|
|F2. Recommends access to learning objects or courses||yes||yes||State of the art shows that recommender systems in the educational domain mainly recommend users to access learning objects or to enroll in courses. SERS extend the scope of the recommendations to other elements beyond learning objects and courses.|
|F3. Recommends actions on any item in the LMS||no||yes||Since an LMS consists of items accessed through links, a recommendation in a SERS is a link to any LMS item. Moreover, some content goes with the link recommended, thus it can specify what action is to be done on the object linked.|
|F4. Recommendations are semantically described through an explicit recommendation model that can be filled in by educators||no||yes||ERS apply algorithms to select the object to recommend but educators cannot provide knowledge to the recommendation process. In the SERS approach, the recommendation model allows educators to elicit knowledge, which can be used by the recommendation mechanism.|
|F5. The web services that can be defined for the information exchange among the components follow the recommendation model||no||yes||Since there is no recommendation model in ERS, web services descriptions (defined to support software components interoperability) can only consider the technical information required for the communication process. In SERS, web services descriptions map the recommendation model.|
|F6. Standards and specifications are considered when describing the recommendations||no||yes||ERS do not take advantage of the semantic descriptions provided by educational standards and specifications. In SERS, this is a distinctive feature.|
|F7. Usability and accessibility principles are considered when presenting the recommendation in the user interface||no||yes||The recommendation information that can be shown by ERS is more limited than that shown by SERS. So, usability and accessibility issues are crucial in SERS, as recommendation related information needs to be displayed.|
|F8. When recommendations are shown to the user, their description is also provided||no||yes||There is no explicit description of the recommendation in ERS so it cannot be shown to users. In SERS, this is a distinctive feature.|
|type (what)||object||Any item from the learning platform (e.g., a file of the documents area, a forum message, a calendar event, a learning object of the course…).|
|action||What the learner is told to do with the object. Two different types of actions are identified: (i) passive actions (e.g., reading, visiting) or (ii) active actions (e.g., selecting, posting, commenting, filling in, changing…).|
|content (how)||text||Explanation to the learner of the action recommended on the object (to be shown to her in the e-learning platform graphical user interface or delivered by e-mail).|
|link||Part of the text which contains an HTML link that points to the object recommended.|
|title||Explanation of the page where the learner will go if she clicks in the link of the recommendation.|
|runtime information (when)||restrictions||Information about the validity of the recommendation to facilitate ruling out the checking of recommendations that are out of date or not applicable.|
|applicability conditions||Definition of the values that should take place for the learner's attributes and her context at runtime to be offered the corresponding recommendation.|
|justification (why)||rationale||Educational foundations of the recommendation, that is, the educational goal that is expected to be achieved by the learner is she follows the recommendation.|
|explanation||Reason for the learner to whom the recommendation is to be delivered, aimed to motivate her and support trust in the system.|
|semantic information (which)||category||Criteria in which the recommendation is focused, such as (i) active participation; (ii) technical support; (iii) communication; (iv) relevant information; (v) accessibility; (vi) motivation, (vii) evaluation activities; (viii) course materials; (ix) progress in knowledge; and (x) profile.|
|stage||Classification of the course situation, e.g., if getting used to the platform or if doing course activities.|
|origin||Source that originated the recommendation, such as (i) defined in the course design; (ii) explicitly stated as preferred by the learner; (iii) popular among most similar learners; or (iv) produced by the educator.|
|relevance||Prioritization of the recommendations to be offered (in the case that there are too many recommendations that match the current learner and context).|
|Element||Attribute||Standards/Specs.||Usage of the Standard/Specification by Attribute|
|type||object||IMS-LD, IMS-QTI, SCORM, IMS-CP||The object attribute refers to the resource element in IMS-LD or SCORM. In IMS-LD the resource element refers not only to content but also to other facilities such as the conference functionality. IMS-QTI describes a particular type of test item, and thus, describes a specific object type. Content resources can be packed with their metadata with IMS-CP.|
|action||IMS-LD, SCORM||The activity element in IMS-LD and SCORM provide the instructions of what to do on the resource (i.e., the object).|
|content||text||W3C WCAG||The text should consider the WCAG 2.0 guideline “3.1 Readable”, which points to making text content readable and understandable.|
|link||The link and the and its title relates to WCAG 2.0 guideline “2.4 Navigable”, which points to providing ways to help users navigate, find content, and determine where they are.|
|runtime information||restrictions applicability conditions||IEEE-LOM, IMS-AfA, IMS-LD, IMS-LIP, IMS-MD, ISO/IEC 24751, SCORM, W3C CC/PP||For both the restrictions and the applicability conditions (i.e., the runtime information element in the recommendation model) metadata that characterizes the context of the recommendation is needed. This information can involve the description of user features (in terms of IMS-LIP, IMS AfA or ISO/IEC 24751), resources (IEEE-LOM, IMS-AfA, IMS-MD, ISO/IED 24751), conditions on properties among the resources (IMS-LD, SCORM) or device capabilities (W3C CC/PP).|
|justification||rationale||IMS-RDCEO||IMS-RDCEO can be used to facilitate the description of the competence addressed by the recommendation as it defines and structure of these competences, including a human readable description of the competency.|
|explanation||W3C WCAG||The explanation should consider WCAG 2.0 guideline “3.1 Readable”, which points to making text content readable and understandable.|
|semantic information||category||not available||Currently there is no standard or specification available that can support the description of these attributes, as they are very much dependant on the recommendation nature.|
|Recommendation 2: Read a Contribution of a Classmate Related to Your Interests.|
|Object: forum message||Action: read||Link: 5637 (object id to be used to build the URL)|
|Text: “Read Mary's contributions in the ‘Module 1 forum’ of the course”|
|Title: “Access the contents of the message posted by Mary in the Module 1 forum”|
|Category: interest||Stage: doing activities||Origin: popular||Relevance: 82%|
|Rationale: point to relevant information regarding the learner interest (defined in <rdceo> element in IMS-RDCEO)|
|Explanation: “Mary has contributed to the Module 1 forum–which is related to the objective Polymorphism of the course when doing some activities related to the objective Polymorphism. According to your model, the objective Polymorphism of the course has a high interest level for you”.|
|LMS||getInfoAboutLMSObjects||Offers structural information about the LMS objects that can be used to generate a recommendation.|
|getTrackingInfo||Provides information about the interactions carried out in the LMS by the learners.|
|UM||storeUMInfo||Allows the components to store information about a given learner in terms of IMS-LIP, IMS-AccLIP and ISO PNP attributes.|
|getUMInfo||Allows the components to retrieve information about a given learner in terms of IMS-LIP, IMS-AccLIP and ISO PNP attributes.|
|DM||getDMInfo||Allows the components to get information about the capabilities of a given device in terms of W3C CC/PP attributes.|
|SERS||getRecsForUser||Allows the components (basically the LMS) to ask for the appropriate recommendations for a given learner in the current context, which should consider the learning design specified in IMS-LD or SCORM, if available as well as the learner (in terms of IMS-LIP, IMS-AccLIP and ISO PNP) and device (in terms of W3C CC/PP) features.|
|setRecClicked||Allows the LMS to inform the SERS service that the learner has followed a specific recommendation.|
|setRecFeedback||Allows the LMS to inform the SERS service about the feedback given by a learner to a specific recommendation.|
|getRecsBehaviourData||Returns the behavior indicators of a recommendation after its delivery in a course.|
|newRecInstance||Obtains the description of a new recommendation and stores it.|
This work is framed in the context of the projects carried out by the aDeNu group. In particular, EU4ALL (IST-2006-034478), [email protected] (TIN2008-06862-C04-01/TSI), CISVI (TSI-020301-2008-21) and ALTER-NATIVA (DCI-ALA/2010/88).
- Terveen, L.G.; Hill, W. Beyond Recommender Systems: Helping People Help Each Other. In HCI in the New Millennium; Carroll, J., Ed.; Addison Wesley: Boston, MA, USA, 2001. [Google Scholar]
- Burke, R. Hybrid recommender systems: Survey and experiments. User-Model. User-Adapt. Interact. 2002, 12, 331–370. [Google Scholar]
- Resnick, P.; Varian, H.R. Recommender Systems. Commun. ACM 1997, 40, 56–58. [Google Scholar]
- Adomavicius, G.; Tuzhilin, A. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 2005, 17, 734–749. [Google Scholar]
- Schafer, J.B.; Konstan, J.A.; Riedl, J. E-commerce recommendation applications. Data Min. Knowl. Discov. 2001, 5, 115–152. [Google Scholar]
- Sarwar, B.; Karypis, G.; Konstan, J.; Riedl, J. Analysis of Recommendation Algorithms for Ecommerce. Proceedings of the ACM Conference on Electronic Commerce, Mineapolis, MN, USA, 17–20 October 2000; pp. 158–167.
- Santos, O.C.; Boticario, J.G. Educational Recommender Systems and Techniques: Practices and Challenges; IGI Publisher: Hershey, PA, USA, 2011; in press. [Google Scholar]
- Manouselis, N.; Drachsler, H.; Vuorikari, R.; Hummel, H.; Koper, R. Recommender Systems in Technology Enhanced Learning. In Recommender Systems Handbook: A Complete Guide for Research Scientists and Practitioners; Kantor, P., Ricci, F., Rokach, L., Shapira, B., Eds.; Springer: Berlin, Germany, 2010. [Google Scholar]
- Drachsler, D.; Hummel, H.G.K.; Koper, R. Identifying the goal, user model and conditions of recommender systems for formal and informal learning. J. Digital Inf. 2009, 10, 1–17. [Google Scholar]
- Moreno, G.; Martinez-Normand, L.; Boticario, J.G.; Fabregat, R. Research on standards supporting [email protected]: Adaptation and accessibility for All in higher education. CEUR Workshop Proc. 2009, 495, 1–10. [Google Scholar]
- Iorio, A.D.; Feliziani, A.A.; Mirri, S.; Salomoni, P.; Vitali, F. Automatically producing accessible learning objects. Educ. Technol. Soc. 2006, 9, 3–16. [Google Scholar]
- Kelly, B.; Sloan, D.; Brown, S.; Seale, J.; Petrie, H.; Lauke, P.; Ball, S. Accessibility 2.0: People, Policies and Processes. Proceedings of the 2007 International Cross-disciplinary Conference on Web Accessibility (W4A), Banff, Canada, 7–8 May 2007.
- Lanzilotti, R.; Ardito, C.; Costabile, M.F.; Angeli, A.D. eLSE Methodology: A systematic approach to the e-learning systems evaluation. Educ. Technol. Soc. 2006, 9, 42–53. [Google Scholar]
- Seale, J.L.; Draffan, E.A.; Wald, M. Exploring Disabled Learners' Experiences of E-learning: LEXDIS Project Report; University of Southampton: Southampton, UK, 2008. [Google Scholar]
- Sawyer, R.K. Optimizing Learning: Implications of Learning Sciences Research. In Innovating to Learn, Learning to Innovate; OECD: Paris, France, 2008. [Google Scholar]
- Barajas, M.; Gannaway, G. Implementing e-learning in the traditional higher education institution. Higher Educ. Eur. 2007, 32, 111–119. [Google Scholar]
- Muñoz-Merino, P.J.; Delgado-Kloos, C.; Fernández-Naranjo, J. Enabling interoperability for LMS educational services. Comput. Stand. Interfaces 2009, 31, 484–498. [Google Scholar]
- Edutools. Avalable online: http://www.edutools.info/index.jsp?pj=1 (accessed on 8 July 2011).
- Hauger, D.; Kock, M. State of the Art of Adaptivity in E-Learning Platforms. Proceedings of the Workshop Adaptivity and User Modeling in Interactive Systems (ABIS), Halle/Salle, Germany, 24-26 September 2007.
- Dagger, D.; O'Connor, A.; Lawless, S.; Walsh, E.; Wade, V.P. Service-oriented e-learning platforms. From Monolithic systems to flexible services. IEEE Internet Comput. 2007, 3, 28–35. [Google Scholar]
- Berners-Lee, T.; Hendler, J.; Lassila, O. The semantic web. Sci. Am. Mag. 2001, 5. [Google Scholar]
- Santos, O.C.; Boticario, J.G.; Raffenne, E.; Granado, J.; Rodríguez-Ascaso, A.; Gutiérrez y Restrepo, E. A Standards-Based Framework to Support Personalisation, Adaptation and Interoperability in Inclusive Learning Scenarios. In Handbook of Research on E-Learning Standards and Interoperability: Frameworks and Issues; Lazarinis, F., Green, S., Person, E., Eds.; IGI Publisher: Hershey, PA, USA, 2010; pp. 126–169. [Google Scholar]
- Boticario, J.G.; Santos, O.C. An open IMS-based user modelling approach for developing adaptive LMS. J. Interact. Media Educ. 2007, 2, 1–19. [Google Scholar]
- Drachsler, H. Navigation Support for Learners in Informal Learning Networks; Open Universiteit Nederland: Heerlen, The Netherlands, 2009. [Google Scholar]
- Drachsler, H.; Bogers, T.; Vuorikari, R.; Verbert, K.; Duval, E.; Manouselis, N.; Beham, G.; Lindstaedt, S.; Stern, H.; Friedrich, M.; Wolpers, M. Issues and Considerations Regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning. Proceedings of the 1st Workshop on Recommender Systems for Technology Enhanced Learning (RecSysTEL 2010), Barcelona, Spain, 29–30 September 2010; pp. 2849–2858.
- Bozo, J.; Alarcón, R.; Iribarra, S. Recommending Learning Objects According to a Teachers' Contex Model. In Sustaining TEL: From Innovation to Learning and Practice; Lecture Notes in Computer Science; Springer: Berlin, Germany, 2010; Volume 6383, pp. 470–475. [Google Scholar]
- Verbert, K.; Drachsler, H.; Manouselis, N.; Wolpers, M.; Vuorikari, R.; Duval, E. Dataset-driven Research for Improving Recommender Systems for Learning. Proceedings of the 1st International Conference Learning Analytics & Knowledge, Banff, Canada, 27 February–1 March 2011.
- Herlocker, J.; Konstan, J.; Terveen, L.; Riedl, J. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 2004, 22, 5–53. [Google Scholar]
- Manouselis, N.; Vuorikari, R.; Van Assche, F. Collaborative recommendation of e-learning resources: An experimental investigation. J. Comput. Assist. Learn. 2010, 26, 227–242. [Google Scholar]
- Peis, E.; Morales-del-Castillo, J.M.; Delgado-López, J.A. Analysis of the state of the topic. Hipertext.net 2008, 6. [Google Scholar]
- Santos, O.C.; Boticario, J.G. Modeling Recommendations for the Educational Domain. Proceedings of the 1st Workshop Recommender Systems for Technology Enhanced Learning (RecSysTEL 2010), Barcelona, Spain, 29–30 September 2010; pp. 2793–2800.
- Santos, O.C.; Boticario, J.G. Usability methods to elicit recommendations for semantic educational recommender systems. IEEE Learn. Technol. Newsl. 2010, 12, 11–12. [Google Scholar]
- Santos, O.C.; Mazzone, E.; Aguilar, M.J.; Boticario, J.G. Designing a user interface to managing recommendations for virtual learning communities. Int. J. Web Based Commun. 2011. in press. [Google Scholar]
- Santos, O.C.; Boticario, J.G. TORMES methodology to elicit educational oriented recommendations. Lect. Notes Artif. Intell. 2011, 6738, 541–543. [Google Scholar]
- Tintarev, N.; Masthoff, J. Designing and Evaluating Explanations for Recommender Systems. In Recommender Systems Handbook; Ricci, F., Rokach, L., Shapira, B., Kantor, P.B., Eds.; Springer: Berlin, Germany, 2011; pp. 479–510. [Google Scholar]
- Pérez-Marín, D.; Alfonseca, E.; Rodríguez, P.; Pascual-Nieto, I. Willow: Automatic and adaptive assessment of students free-text answers. Span. Soc. Nat. Lang. Proc. J. 2006, 37, 367–368. [Google Scholar]
- Pascual-Nieto, I.; Santos, O.C.; Perez-Marin, D.; Boticario, J.G. Extending Computer Assisted Assessment systems with Natural Language Processing, User Modeling, and Recommendations based on Human Computer Interaction and Data Mining. Proceedings of the International Joint Conference on Artificial Intelligence, Barcelona, Spain, 19–22 July 2010; pp. 2519–2524.
- Valdiviezo, P.M.; Santos, O.C.; Boticario, J.G. Aplicación de métodos de diseño centrado en el usuario y minería de datos para definir recomendaciones que promuevan el uso del foro en una experiencia virtual de aprendizaje. Revista Iberoamericana de Educación a Distancia 2010, 13, 237–264. [Google Scholar]
- Romero, C.; Ventura, S.; Delgado, J.A.; de Bra, P. Personalised Links Recommendation Based on Data Mining in Adaptive Educational Hypermedia Systems. Proceedings of Second European Conference on Technology Enhanced Learning, ECTEL 2007, Crete, Greece, 17–20 September 2007; pp. 292–306.
- Swearingen, K.; Sinha, R. Beyond Algorithms: An HCI Perspective on Recommender Systems. Proceedings of the SIGIR 2001 Workshop on Recommender Systems, New Orleans, LA, USA, 13 September 2001; Volume 13, Number 5–6. pp. 393–408.
- Santos, O.C.; Boticario, J.G. Users' Experience with a Recommender System in an Open Source Standards-Based LMS. Proceedings of 4th Symposium of the WG HCI and UE of the Austrian Computer Society—Usability and HCI for Education and Work (USAB 2008), Graz, Austria, 20–21 November 2008; pp. 185–204.
- Kay, J. Stereotypes, Student Models and Scrutability. Lect. Notes Comp. Sci. 2000, 1839, 19–30. [Google Scholar]
- Mödritscher, F. Towards a recommender strategy for personal learning environments. Proceedings of the 1st Workshop ‘Recommender Systems for Technology Enhanced Learning’ (RecSysTEL 2010), Barcelona, Spain, 19–22 July 2010; pp. 2775–2782.
- Wilson, S.; Liber, O.; Johnson, M.; Beauvoir, P.; Sharples, P.; Milligan, C. Personal learning environments: Challenging the dominant design of educational systems. J. E-Learn. Knowl. Soc. 2007, 3, 27–38. [Google Scholar]
© 2011 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).