Next Article in Journal
Transformational Training Programs and Quality Orientation of Employees: Does Employees’ Loyalty Matter?
Next Article in Special Issue
Soil Biodiversity in Urban Forests as a Consequence of Litterfall Management: Implications for São Paulo’s Ecosystem Services
Previous Article in Journal
Population Agglomeration and Residential Energy Consumption: Evidence from Japan
Previous Article in Special Issue
Formation Mechanism and Coping Strategy of Public Emergency for Urban Sustainability: A Perspective of Risk Propagation in the Sociotechnical System
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Enhancing the Efficiency of Massive Online Learning by Integrating Intelligent Analysis into MOOCs with an Application to Education of Sustainability

School of Economics and Management, Beihang University, Beijing 100191, China
Author to whom correspondence should be addressed.
Sustainability 2018, 10(2), 468;
Submission received: 14 December 2017 / Revised: 18 January 2018 / Accepted: 8 February 2018 / Published: 9 February 2018


Massive Open Online Courses (MOOCs) is an innovative method in modern education, especially important for autonomous study and the sharing of global excellent education resources. However, it is not easy to implement the teaching process according to the specific characters of students by MOOCs because the number of participants is huge and the teacher cannot identify the characters of students through a face to face interaction. As a new subject combined with different areas, such as economics, sociology, environment, and even engineering, the education of sustainability-related courses requires elaborate consideration of individualized teaching for students from diverse backgrounds and with different learning styles. Although the major MOOC platforms or learning management systems (LMSs) have tried lots of efforts in the design of course system and the contents of the courses for sustainability education, the achievements are still unsatisfied, at least the issue of how to effectively take into account the individual characteristics of participants remains unsolved. A hybrid Neural Network (NN) model is proposed in this paper which integrates a Convolutional Neural Networks (CNN) and with a Gated Recurrent Unit (GRU) based Recurrent Neural Networks (RNN) in an effort to detect individual learning style dynamically. The model was trained by learners’ behavior data and applied to predicting their learning styles. With identified learning style for each learner, the power of MOOC platform can be greatly enhanced by being able to offer the capabilities of recommending specific learning path and the relevant contents individually according to their characters. The efficiency of learning can thus be significantly improved. The proposed model was applied to the online study of sustainability-related course based on a MOOC platform with more than 9,400,000 learners. The results revealed that the learners could effectively increase their learning efficiency and quality for the courses when the learning styles are identified, and proper recommendations are made by using our method.

1. Introduction

Due to the increasing impact of population growth, environmental degradation, and resource depletion, sustainable development has gradually become a worldwide focus and received great concerns from most countries. Promoting educational equity is the one of 17 key goals in the Sustainable Development Goals (SDGs), officially known as “Transforming our world: the 2030 Agenda for Sustainable Development” issued by United Nations in 2015 [1], i.e., improving people’s overall quality, ensure inclusive and equitable quality education for boys and girls, man and woman, eliminate gender disparities and promote lifelong learning opportunities for all. Plenty of efforts have been dedicated to realizing these goals, including various innovative ideas, advanced teaching methods and facilities, and indispensable infrastructure. Among those, the open education service platform equipped with massive learning resources, personalized learning plan and instructions, and efficient measurement and evaluation technologies, is one of the most critical strategic elements for effective learning, especially in higher education and continuous education. The importance of this platform not only lies in promoting and integrating the global education resources, but offers the equal chances to all learners to access the most excellent subjects and instructors.
Various schemes and their technical supports have been designed since the idea of open education was concerned, among which the Massive Open Online Courses (MOOCs), emerging in 2012, has attracted wide attention and been adopted by a lot of educational organizations [2]. MOOCs start a new era of open education which makes lifelong learning and widely sharing of high-quality education resources being feasible and acceptable via the power of the internet. Different from to Open Courseware (OCW) [3], initiated by MIT in 2001 which provides global learners and instructors free access to its undergraduate and postgraduate course materials through the internet, MOOCs not only offer learning resources and instructional services for mass learners, but also provides user forums to support community interactions among students, professors, and teaching assistants, making it easy to communicate with course instructors and discuss learning issues with fellow learners. Aiming at providing low-cost and equal learning opportunities to more people, MOOC and its service providers obtain widespread support from governments, different social organizations, and even venture capitals [4]. According to an incomplete statistic from Class Central, 2600 new courses were announced in 2016 (up to 1800 in 2015), and the total number of MOOCs reaches 6850 contributed from over 700 universities. The total number of students who registered at least one subject of MOOC is 58 million, while the number for 2015 is only 35 million [5]. Traditionally, Coursera, edX, and Udacity were top three MOOC providers in the world, but in 2016, from China has exceeded Udacity in user base and courses hosted. The fast growth of participants and courses shows that MOOC has gradually become an essential component of the modern education system which can effectively facilitate the people to accept high-class education in a more efficient and equitable manner.
Yet, on the other hand, there are still some problems with MOOCs which affect the expected performance of it. Although nowadays MOOCs are well designed and operated by outstanding teaching teams, even the completion rate for most courses is below 13%, typically range from 2% to 10%, in view of thousands of participants enrolling in these courses [6,7]. Investigations have been conducted to explore the possible resolutions. Some authors argued that student intentions should be taken into account [6], but problems about learning experience and styles have been less discussed. For example, many of the MOOC courses are still simply organized in static mode, and teach all the learners in the same way, lack of consideration about their different needs and characteristics, such as prior knowledge and learning patterns. In fact, many courses, especially those from interdisciplinary subjects, require higher individualization in teaching and learning to enhance the learning performance. Sustainability-related subjects typically fall into this category.
Sustainability education involves the integration of social, environmental, and technological elements with economic considerations, it is a broad concept and difficult to precisely define. The concept of “sustainability education” first emerged in “Our Common Future”, also known as the Brundtland Report, issued by the United Nations World Commission on Environment and Development (WCED) in 1987 [8]. UNESCO (United Nations Educational, Scientific, and Cultural Organization) also launched Education for Sustainable Development (ESD) as a guide to teaching and learning that promotes sustainable development growing from an idea into a global movement [9,10]. Nowadays, there are two main research fields in the education of sustainable development: (1) learning contents: what kind of topics and contents should be taught in courses? How to properly represent these broad-scale and cross-disciplinary contents in sustainability? (2) Pedagogical methods: how should we teach students the sustainability-related courses, especially in an open and distant learning environment, facing the large-scale population of learners? Lots of efforts have been exerted on this issue, and researches on pedagogical methods are diverse from nationwide experiments [11,12] to learning specific courses with MOOCs. Quite some investigations indicate that individualized teaching and learning under the massive online learning environment is a significant way to increase the performance of sustainability education. This conclusion has greatly boosted the online education of Sustainability. For example, there is none of Chinese university providing sustainability-related online courses in 2015 [13], while there are nine courses on in 2017.
In recent years, increasing attention has been paid to the characteristics of learners such as learning styles [14,15,16,17,18,19], including the impact on learning performance and how such individual characteristics should be supported by adaptive systems. Students' learning styles can be determined in many different ways [20].But other scientists state that there is no adequate evidence base to justify incorporating learning styles assessments into general educational practice [21], thus this mismatch between practice and evidence has provoked controversy, and some have labeled Learning Styles a ‘myth’ [22]. However, all these studies or discussion were based on experiments and evidence in traditional classroom, which means, learning style measurement or tailoring teaching to a student's preferred style are low cost-effective. By contrast, with the rapid development of artificial intelligence and internet technology, online massive individualized education becomes possible, educational interactions can be controlled by system with an acceptable cost. Researches on providing adaptive learning materials with different learning styles in online learning environment [23] and employing fuzzy logic to determine users’ quality of interaction [24] indicates the key success factors could be different between online and traditional offline classroom, demonstrating the potential of new technology.
The first and crucial step to implementing individualized learning is to identify the personal characteristics of the learners. Graf and Kappel introduced the learning styles to learning management systems in an effort to increase the adaptivity of the system, in which learning styles were defined by adopting the Silverman learning style model [25]. With their approach, however, a survey including 44 questions which takes more than 10 minutes has to be completed in order to induce the individual learning style, this is quite boring and inconvenient to most of the learners and even lead to erroneous or unfaithful responses. Moreover, the learning style will change in case of learning pressure or situation changes in the learning process, therefore the static model cannot accurately reflect the individual characters in a changing environment.
The primary purpose of this work is to design a real-time intelligent interaction mechanism which can automatically and dynamically identify the learning style of the learners, and then to provide proper instruction methods and materials in accordance to the personal characteristics of different users for improving their learning efficiency in massive online learning environment. Sustainability subject is chosen as the case to demonstrate the effectiveness of the method in this paper because it is a typical interdisciplinary subject and more suitable for individualized learning as mentioned before. In addition, sustainability-related courses have been adopted by more and more educational organizations, how to teach the courses in a more effective and efficient manner is with particular significance.
The contents of the paper are organized as follows. The problem of individualized learning is briefly introduced in the next section. And the method of how to dynamically identify the learning styles is presented in Section 3. To demonstrate the performance of the proposed method, it is applied to the online study of sustainability-related courses based on a MOOC platform with more than 9,400,000 learners, and the results are analyzed in Section 4. Finally, some conclusions are summarized and possible future works are recommended.

2. Individualized Learning Problem

In traditional education, instructors need to prepare the course contents and design the teaching process. Besides, they have to realize the students’ background and pay close attention to students’ behavior to understand their characters in learning and provide special arrangements after class. These interactions are quite essential for achieving better performance in study. Nowadays in massive learning era, online education becomes one of the most important components of modern education system. With the power of computers and internet, the system can provide 7 × 24 h personalized service and treat learners with great patience. And with the great progress in artificial intelligence individualized learning also becomes possible. One of the important ways to implement the individualization is to integrate the adaptivity to learning management systems. The following three fundamental problems have to be addressed for this design.

2.1. Learning Style Modeling

Various learning style models were proposed in the last 50 years, with concerns on different aspects, such as student characters, emotional situations, cognitive styles, and even environments. Among the existing studies, Felder-Silverman Learning Style Model (FSLSM) is the one which has been most wildly adopted, especially in adaptive learning systems [26]. FSLSM took many advantages of previous work, and the design of each of the four dimensions was greatly inspired by other learning style models, e.g., the learning style model by Kolb [27], Pask [28], as well as the Myers-Briggs Type Indicator [29]. With FSLSM, learners are characterized from four dimensions. These dimensions are based on the main concerns in the field of learning styles and can be viewed independently from each other, indicates how learners prefer to processing (Active/Reflective), perception (Sensory/Intuitive), input (Verbal/Visual), and understanding (Sequential/Global) information [26]. Learning styles and their main characteristics are summarized in Table 1.
To identify the learning styles by FSLSM, Felder and Soloman developed the Index of Learning Styles (ILS), which includes a 44-item questionnaire [30]. The learners express their preferences by using values between +11 to −11 per dimension with step of 2, in completing the questionnaire. The values can well classify the learners into proper categories regarding learning styles, and can even identify the extent to certain category.

2.2. The Challenge of Initializing Learning Style

FSLSM considers learning styles as “flexible stable”, arguing that the learning styles of students come from their previous learning experiences together with other environmental factors [15]. That is to say, learning styles tend to be more or less stable but can change over time, so we have to keep on the identification process. The first step of study with LMSs is to identify the learning style which is the start of enhancing the learning process. As mentioned in Section 2.1, the traditional measuring method for ISL is to finish a questionnaire, which is convenient to carry out in a closed environment (e.g., a classroom) because the education institutes can easily arrange the students who are using the system to fill it out. However, it is almost impossible to ask every user to finish such a survey in massive online learning environment for time and cost concern. Hence the system has to face the challenge of initializing learning style, or so-called “Cold Start”.

2.3. Dynamic Detecting

It is very critical to introduce the adaptivity to the LMSs by developing the mechanism which is capable of dynamically identifying the learning styles to competently meet the requirement of FSLSM. However, it is impossible to trace the changing of styles by repeatedly asking the learners to complete the questionnaire. Therefore, an automated dynamic learning style tracking mechanism is the essential part of an efficient massive online learning environment.
Currently, there are two different approaches adopted for learning styles dynamic detecting based on FSLSM: data-driven approach and the literature-based approach. The data-driven approach aims at building a model that imitates the ILS questionnaire. It uses sample data to construct a model and derives the proper patterns for identifying learning styles from the behavior of learners. Then Decision Trees and Hidden Markov Models [31] or Bayesian Networks [32] are employed to obtain the parameters of the model. This approach only uses the fixed and specific data relevant to the particular course. Hence it strongly depends on the available data and the course characteristic, and is less flexible for the variety of subjects or when the student’s behavior style changes.
The literature-based approach is to use the typical behavior of students to get hints about their learning style preferences, then apply a simple rule-based method to deduce their learning styles based on the number of patterns matched. This approach is more generalized and applicable to the data gathered from any course. However, the approach might have problems in accuracy, because it is hard to estimate the importance of different hints used for calculating the learning styles.
The process of learning styles detection can be demonstrated in Figure 1.
Although the above two approaches can detect and predict learning styles to some extent with certain accuracy, there are obvious limitations when apply. For example, learning behavior could change in different courses due to the knowledge backgrounds and learning habits of learners; learning style could even change in different parts of the same course, for the difficulty level or knowledge type changes. This problem can be easily solved in traditional classrooms because the teacher can identify the changes and make the relevant adjustment timely. In a machine-based learning environment, however, the current system is not capable to detect such changes in learning styles of learners and hence might make the improper recommendations. Therefore, dynamic detecting is an essential requirement for modern massive online learning facilities.

3. Dynamic Identification of Learning Styles

About thirty years ago, Learning Management Systems (LMSs), also referred to as Virtual Learning Environments, was developed in the UK, comprising a collection of software and web applications that enable the online delivery of course materials as well as the tracking and reporting of student participation. Recently, Next-generation LMS has been proposed [33], also called Next-Generation Digital Learning Environments (NGDLE), referring to the development of more flexible systems that support personalization and follow the universal design standards. Although both systems have been widely used in technology enforced education they provide only a little, or even none in most cases, the capability of adaptivity for the learning process. To enhance the capability of LMS, an adaptivity mechanism is designed by introducing the dynamic identification for learning styles of learners based on FSLSM in this section.

3.1. Framework for LMS with Adaptivity

The framework of the enhanced adaptive LMS is proposed as Figure 2.
Learning style identification is initialized in User Modeling Process, and a basic user model is set up by user profiling based on demographic and user tracking data. The identified learning styles in terms of FSLSM can be dynamically updated by Behavior Analyzing module when users are interacting with the system, and new characteristics have been captured. On another side, learning materials are no longer simple static videos and texts. In Content Analyzing Process, various kinds of videos, texts, and subtitles are highly organized so that concepts can be extracted to become the nodes of knowledge graph by Concept Extraction module, and Concept Relation Mapping module builds a prerequisite relationship matrix between segments and courses. Finally, the Interacting Process provides adaptive learning path routing to learners based on their learning styles kept in the knowledge base.
One of the core functions of above system is to detect the learning styles dynamically. This can be realized by adopting intelligent analysis technologies. Recently, a large volume of methods about intelligent analysis has been developed, among which some, e.g., Neural Network and Deep Learning, have achieved great success and applied to various fields such as image and speech recognition. The main advantage of these methods lies in the strong capability of processing unstructured data, which also makes it possible to detect learning styles dynamically via the behavior sequence of learners in LMS.
By the identified learning styles, the system can provide individualized learning contents, learning path, and suitable method, which will lead to higher efficiency and more satisfying performance in learning.

3.2. User Modeling

According to FSLSM, learners can be categorized using Active/Reflective, Sensing/Intuitive, Visual/Verbal, Sequential/Global dimensions, ranked in different levels, and the adaptive learning system then provides appropriate contents and learning path correspondingly. For example, the system will provide Reflective users the method based on Problem-Based Learning Approach which employs a series of questions to lead the students finding the solution to the problems. And for the Active users who tend to get information aggressively, it will continuously push the latest ideas on Course Interactive Forum to them so as to inspire a more in-depth and efficient study.
Firstly, we set up a basic leaner property model for user i at time t as expression (1):
U i , t = U ( D i , L i , t )
where D i denotes demographical distribution of user i, and L i , t represents the learning style based on FSLSM with four dimensions defined as expression (2).
  L i , t = L ( ( A R , W A R ) , ( S I , W S I ) , ( V V , W v v ) , ( S G , W S G ) )
The initial assignment of learning style for user i L i , 0 can be determined either by the final state of his immediately previous course, or based on the demographical distribution if there is not a predecessor. Having initiated, the system will continuously observe and analyze the user behavior to detect the changes in learning style, as depicted in Figure 3.

3.3. Behavior Features Extraction

Based on lecture study on FSLSM, dimension values and rankings of learning style can be determined by the means of using user behavior sequences, which can be measured with ILS questionnaire. However, different knowledge structure and study pressure for different courses might lead to varying results in ILS, hence the static ILS cannot precisely reflect the behavior characteristics of learners. A more appropriate candidate is to differentiate the learners in terms of learning style by proper classification scheme based on their behavior features, instead of using the exact index values.
Graf proposed a kind of approach to automatically detect the learning styles with LMS [34], by which threshold-based detecting behavior patterns was calculated relatively in percentage after learning process. The thresholds model can indicate the trend of learning style by combining the behavior patterns and making the relevant analysis. However, it needs the data after the learning process and hence is not applicable to dynamic detection.
The main idea of the method in this paper is to take the advantages of both thresholds-based approach and FSLSM by setting up thresholds for user behavior patterns and mapping user behavior logs to four learning style dimensions of FSLSM as indicated in Table 2.
Learning style detection module will continuously track the user behavior records and take time frame into consideration as well. Each log(channel), is mapped into a time(by day)-content(by chapter) based 2-dimensions array, the nodes of each feature events are associated with a timestamp and normalized to the same level. For example, effective learning point density log L E i , j including absolute learning time and relative learning progress can be mapped into day based time serials and content as expressed in Formula (3)
L E i , j = L E ( ( t v i T , t i ) · C j )
where t v i is the playing time of video clip for learning action i, and T is total video content time, t i is absolute time of learning, and C j is index of the jth segment of course content. For the non-content related logs: channel 2 and 3, the node was filled with 0 for the lines does not contains data.
As an example, logs from “Data Structure” (Part 1, 2017), one of most popular MOOC courses on with more than 23,000 students enrolled is examined. The course includes six chapters and lasts 73 days. Two learners are randomly selected and the logs for their effective learning points density are visualized as Figure 4.
Based on the aforementioned FSLSM summarized in Table 1 we can infer the difference in learning styles of the two users: (a) User #4441580 learning one by one chapter in sequence (Sequential progress) and got a higher point in learning point density than average (Think more detail, Detail Oriented), evidently expressed the tendency of Reflective and Sequential type; (b) User #6565569 learning more than one chapter in one day four times (Try something out, Overall picture, Non-sequential progress), thus tend to behave in a more Active and Global manner.

3.4. Intelligent Analysis Process

Convolutional Neural Network (CNN) is a class of deep learning, feed-forward neural networks that have successfully applied to analyzing visual imagery. The network can be trained to classify images in high recognizing efficiency with low error rate.
Recurrent Neural Network (RNN) is another popular NN model which has the strong capability of processing variable-length sequence data and is often applied to the field of Natural Language Processing. Some research employs a unified CNN-RNN framework learns a joint image-label embedding to characterize the semantic label dependency as well as the image-label relevance, and it can be trained end-to-end from scratch to integrate both information in one framework, the model achieves better performance than the state-of-the-art multi-label classification models [35]. The model for intelligent analysis in this paper is mainly composed of two parts, a CNN for learning styles detection and a RNN combined with gated recurrent unit for learning styles prediction. A hidden internal state was added in the conventional feed-forward deep models network [36]. Standard RNNs updates their hidden state h t using function as following
h t = g ( W x t + U h t 1 )
where g is a smooth and bounded function such as the logistic sigmoid function, x t is the input of the unit at time t. The RNN outputs a probability distribution over the next element of the sequence, given its current state h t .
Gated Recurrent Unit (GRU) aims at dealing with vanishing gradient problem for RNN, the unit gates learn when and by how much to update the hidden state of the unit [37]. The activation of the GRU is a linear interpolation between the previous activation and candidate activation
h t = ( 1 z t ) h t 1 + z t h ˜ t
And the update gate z t is given by
z t = σ ( W z · [ h t 1 , x t ] )
The candidate activation function is computed as
h ˜ t = tan h ( W · [ r t h t 1 , x t ] )
The reset gate is given by
r t = σ ( W r · [ h t 1 , x t ] )
The proposed hybrid architecture of two parts can be illustrated in Figure 5.
In the first part, a serial of log sequences are sliced into snapshot segments by pieces marked with t, t-1, and t-2 time frame, which are composed of different user learning styles. The data was processed by a CNN with a simple multi-layered network architecture with fully connected layers, named Observer. The input layer has 8 × 73 units, 8 channels, based on the snapshot vector, followed by two hidden layers of 128 units with the ReLU activation function. The output layer has four output neuron for different learning styles in four dimensions, L i , t , which uses softmax activation function and outputs the probability of learning styles in each dimension for the given state.
In the second part, the output of the Observer is connected with a GRU-based RNN, named Inference Engine where the received learning style L i , t detected in the first part is used to predict the next style L i , t + 1 . With the hidden state in multiple GRU layers, the input can be optionally connected deeper in network which may reinforce the memory effect, hence the performance of prediction can be improved.

4. Model Training and Test

4.1. Datasets

User behavior logs from between 1 September 2016 and 31 January 2017 are adopted, which contains 162,650 independent users’ behavior actions in 1115 courses, including 7,530,648 videos seek records, 348,942 forum visits/post logs, and 36,604,330 navigation records. Activate sessions of length less than three are filtered out by preprocessing, and 153,426 users are left in the dataset. The dataset is split into a training set of 122,740 users (80%) and a test set of 30,686 users (20%), and the data are shuffled by course group. The training dataset is used for training the neural network, while the test dataset is used to evaluate the trained network. A lecture based activity analyzing model proposed by Graf was used to labeling the learning styles in terms of the dimensions of FSLSM [25], as indicated in Section 3.3, for the users in the training dataset. The distribution of learning styles of the learners in training dataset is visualized as Figure 6, indicates that most users are intended to be reflective style in processing dimension, which means they are preferred impersonal and fewer activities in community, same in Chinese university classrooms. In perceive dimensions the majority users are sensing styles, almost two times than intuitive styles, most users preferred existing ways. Visual content is primary receiving style by users. Sequential style got 20% more than global style users, not as much as expected for Chinese users, this could be come from the actions taken by many learners jump in online learning environment, which increases the number of global style users.

4.2. Training and Test

The model is implemented and run with Google Tensorflow 1.2.1 trained with the above training dataset, the learning rate is 0.0001 with the Adam optimization algorithm [38] and batch size for 10. In the first part of the intelligent analysis process, the Observer tries to identify learning style with the given behavior snapshot, acting as to solve a classification problem. In this case, identical data examples in our dataset can have a different dimension of learning styles. Then in the second part, the Inference Engine ranks possible Learning Style depends on the previous Learning styles in a learning session provided by Observer, learning to rank approaches generally better than other approaches [39]. Ranking can be pointwise, pairwise, or list wise. Pairwise ranking outperforms the other two, it compares the score or the rank of pairs of a positive and a negative item and the loss enforces that the rank of the positive item should be lower than that of the negative one.
Several architectures are also examined, and the single layer of GRU units is found to be the best performer while adding additional layers might result in the performance loss in training.
The effectiveness of the model is evaluated by processing the test dataset with trained Observer and Inference Engine. The best test result reaches an average error rate of 9.235% in four dimensions, which means that the model can properly detect the learning styles most of the time and the accuracy can meet the requirement of the system. Through interaction between users and LMS, the present and previous behavior patterns of users are dynamically obtained to predict their next learning styles. Evidence shows that 2869 times of the event for learning style change are detected on the testing dataset in the intelligent analysis process, with 1703 times (59.33%) in science courses, as listed in Table 3.

4.3. Discussion

By contrast to high cost of determining student learning style in face to face classroom, the suggested model can detect learning style dynamically in an acceptable cost by taking the advantage of LMSs and intelligent analysis method, makes that is possible make interventions by individual learning style. The result reveals that learning style may change in the learning process, especially when learning tougher or more quantitative contents such as science courses. Changes in Visual/Verbal dimension are relatively less than in other dimensions. The possible reason is that most online contents are in video or graphic format and fewer verbal contents provided. It is noteworthy that quite some users change from Reflective to Active. It is found by further investigations on the logs that the change is usually caused by forum related behaviors, which implies that it is possible to enhance engagement of this kind of students by sending them messages about the latest discussions on course forums.

5. Results from Empirical Research on Sustainability Education

The proposed model for dynamic learning style identification has been applied on, and the effectiveness is evaluated in this section based on the empirical research on the online study of sustainability-related courses.
Some sustainability-related MOOCs on are listed in Table 4, offered by Tsinghua University, Jilin University, Beijing Normal University, and Hubei University, respectively. Some courses have been operated since 2015, and some are just online in 2017. The total enrollment of the courses is 37,520, with an average of 4168 for each. Two courses of running more than two rounds and enrollments more than 6000 are selected as the target courses to conduct the empirical research. The primary goal of the research is to examine whether the learning performance can be improved when the learning styles of the students are identified by the proposed model, and thus the corresponding pedagogical models suitable for the specific learners are provided.
A simple pedagogical model was designed, in which proper actions would be taken while the event for learning style change was detected, as listed in in Table 5. The investigation was limited to providing only different course contents at this time, therefore content-related events “Verbal to Visual” and “Visual to Verbal” were ignored.
The two selected MOOCs were “Green buildings and Sustainable development” and “China's perspective on climate change”. The traditional LMS was enhanced with the functions of learning style identification and recommendation message feedback (actions in Table 5). The message was sent by email, notifications, and hints on courseware. Learning behaviors were collected daily, and processed by the proposed intelligent analysis model on 2:00 AM every day. The messages driven by events were sent on 10:00 AM. During the study period of the courses, 918 messages were sent among which 210 were read by learners; the average read rate is 22.8% with details shown in Table 6.
The enhanced LMS with the intelligent analysis model was implemented in early 2017. A model with five categories of direct effect and moderations: information quality, system quality, use, user satisfaction and net benefits were used to evaluate the success of LMSs from a student’s point of view [40]. Five indexes in two categories: Net Benefits and Use can be benchmarked to evaluate the learning performance and engagement of the users, and the results of two rounds of course learning are listed in Table 7, where the learning process for Round-2016#2 was completed with traditional LMS (Interventions triggered by Teaching Assistant manually) and that for Round-2017#1 was with enhanced LMS (Interventions triggered by learning style detecting model to specific learners).
From the results of above experiments on the learning of sustainability-related subjects with MOOCs are easy to find evidence about improvement on the learning performance due to the introduction of the intelligence analysis mechanism. The users can significantly benefit from the individualized pedagogical model in accord with their learning styles, especially for the study of complex and cross-disciplinary subjects. Another valuable finding is the remarkable improvement on the interactions among the users in the forums, this could positively help users learn from each other rather than only from courses or instructors which is very important for sustainability education. Of course, the more general conclusion still depend on the further investigation with more rigorous control on the experiments.

6. Summary

The learning style identification is an essential problem for individualized teaching or learning which can significantly improve the learning performance and participation, especially in an online or hybrid education environment. In this paper, a kind of intelligent analysis mechanism was proposed based on the hybrid neural network model to dynamically detect and predict the learning styles of learners through their behavior in an interactive learning system. A convolutional neural network was designed to observe and identify the state of user behavior, and then connected with a modified GRU-based recurrent neural network to predict the learning style of next state. The testing process showed that the trained model in this paper can identify the learning styles of users efficiently and effectively, and could be further extended for the intelligent analysis of an AI enforced adaptive learning management system. This is valuable for the next considerations. The future work could also include the exploration of new methods to further increase the accuracy of learning style detection and prediction, as well as more sophisticated pedagogical methods suitable for online learning.
Massive online learning is one of the most important ways to make full use of the limited global education resources and increase the equity of education, which are the primary concerns of education sustainability. With great potential to offer stronger support and better service to learners in order to increase their study performance in online learning environment by adopting the advanced technologies (such as the intelligent analysis mechanism proposed in this paper) can manage the learning process effectively. We are planning to enhance the adaptive learning system of by providing more diverse contents and sophisticated learning paths for users according to much accurate identification of their profiles and learning styles, which will better serve over 9,000,000 learners across the world. We still have a long way to go.


The authors would like to express their appreciations to the National Development and Reform Commission and Ministry of Education of the People’s Republic of China Programme National Engineering Laboratory of Education Big Data Application for their support to the research (under grant number NDRC-2017-149 and MOE-2017-41).

Author Contributions

Chao Li and Hong Zhou conceived and designed the experiments; Chao Li performed the experiments; Chao Li and Hong Zhou analyzed the data; Chao Li contributed analysis tools; Chao Li and Hong Zhou wrote the paper.

Conflicts of Interest

The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.


  1. United Nations (Ed.) Transforming Our World: The 2030 Agenda for Sustainable Development; United Nations: New York, NY, USA, 2015. [Google Scholar]
  2. Pappano, L. The year of the MOOC. N. Y. Times 2012, 2, 2012. [Google Scholar]
  3. The Massachusetts Institute of Technology (MIT). About Opencourseware. Available online: (accessed on 18 August 2017).
  4. Hawtin, N. Major Players in the MOOC Universe. Available online: (accessed on 1 May 2017).
  5. Central, C. Monetization over Massiveness: A Review of MOOC Stats and Trends in 2016. Available online: (accessed on 19 August 2017).
  6. Sinclair, D.O.J. Dropout rates of massive open online courses: Behavioural patterns. In Proceedings of the 6th International Conference on Education and New Learning Technologies (EDULEARN14), Barcelona, Spain, 7–9 July 2014. [Google Scholar]
  7. Reich, J. MOOC Completion and Retention in the Context of Student Intent. Available online: (accessed on 1 May 2017).
  8. World Commission on Environment and Development (WCED). Our Common Future; Oxford University Press: Oxford, UK, 1978. [Google Scholar]
  9. Hopkins, C. Twenty years of education for sustainable development. J. Educ. Sustain. Dev. 2012, 6, 1–4. [Google Scholar] [CrossRef]
  10. Kaplan, A.M.; Haenlein, M. Higher education and the digital revolution: About MOOCs, SPOCs, social media, and the cookie monster. Bus. Horiz. 2016, 59, 441–450. [Google Scholar] [CrossRef]
  11. Carm, E. Rethinking education for all. Sustainability 2013, 5, 3447–3472. [Google Scholar] [CrossRef] [Green Version]
  12. Rowe, D. Sustainability. Education for a sustainable future. Science 2007, 317, 323–324. [Google Scholar] [PubMed]
  13. Zhan, Z.; Fong, P.; Mei, H.; Chang, X.; Liang, T.; Ma, Z. Sustainability education in massive open online courses: A content analysis approach. Sustainability 2015, 7, 2274–2300. [Google Scholar] [CrossRef]
  14. Viola, S.R.; Graf, S.; Leo, T. Analysis of felder-silverman index of learning styles by a data-driven statistical approach. In Proceedings of the 8th IEEE International Symposium on Multimedia, San Diego, CA, USA, 11–13 December 2006; pp. 959–964. [Google Scholar]
  15. Felder, R.M.; Spurlin, J.E. Applications, reliability and validity of the index of learning styles. Int. J. Contin. Eng. Educ. Life-Long Learn. 2005, 21, 103–112. [Google Scholar]
  16. Dissanayake, D.; Perera, T.; Elladeniya, C.; Dissanayake, K.; Herath, S.; Perera, I.; Dissanayake, D.; Perera, T.; Elladeniya, C.; Dissanayake, K. Identifying the learning style of students in MOOCs using video interactions. In Proceedings of the 4th International Conference on Society, Education and Psychology (ICSEP 2017), Macau, China, 6–8 May 2017. [Google Scholar]
  17. Bicans, J.; Grundspenkis, J. Student learning style extraction from on-campus learning context data. Procedia Comput. Sci. 2017, 104, 272–278. [Google Scholar] [CrossRef]
  18. Gesa, R.F.; Lázaro, J.L.M.; Carrillo, C.I.P.D. Teaching Support Units; Springer: Dordrecht, The Netherlands, 2000; pp. 163–174. [Google Scholar]
  19. Hughes, G.; Dobbins, C. The utilization of data analysis techniques in predicting student performance in massive open online courses (MOOCs). Res. Pract. Technol. Enhanc. Learn. 2015, 10, 10. [Google Scholar] [CrossRef]
  20. Coffield, F.; Ecclestone, K.; Moseley, D.; Hall, E. Learning Styles and Pedagogy in Post 16 Education: A Critical and Systematic Review; Learning & Skills Research Centre: London, UK, 2004. [Google Scholar]
  21. Pashler, H.; Mcdaniel, M.; Rohrer, D.; Bjork, R. Learning styles: Concepts and evidence. Psychol. Sci. Public Interest 2008, 9, 105–119. [Google Scholar] [CrossRef] [PubMed]
  22. Newton, P.M.; Miah, M. Evidence-based higher education—Is the learning styles ‘myth’ important? Front. Psychol. 2017, 8, 444. [Google Scholar] [CrossRef] [PubMed]
  23. Waes, L.V.; Weijen, D.V.; Leijten, M. Learning to write in an online writing center: The effect of learning styles on the writing process. Comput. Educ. 2014, 73, 60–71. [Google Scholar] [CrossRef]
  24. Dias, S.B.; José, A. FuzzyQoi model: A fuzzy logic-based modelling of users’ quality of interaction with a learning management system under blended learning. Comput. Educ. 2013, 69, 38–59. [Google Scholar] [CrossRef]
  25. Graf, S. Adaptivity in Learning Management Systems Focussing on Learning Styles. Ph.D. Thesis, Vienna University of Technology, Vienna, Austria, 2007. [Google Scholar]
  26. Felder, R.M.; Silverman, L.K. Learning and teaching styles in engineering education. Eng. Educ. 1988, 78, 674–681. [Google Scholar]
  27. Kolb, D. Experiential Learning: Experience as the Source of Learning and Development; Prentice Hall: Upper Saddle River, NJ, USA, 1984; Volume 1, pp. 16–17. [Google Scholar]
  28. Pask, G. Styles and strategies of learning. Br. J. Educ. Psychol. 1976, 46, 128–148. [Google Scholar] [CrossRef]
  29. Myers, I.B.; Mccaulley, M.H.; Most, R. Manual: A Guide to the Development and Use of the Myers-Briggs Type Indicator; Consulting Psychologists Press: Sunnyvale, CA, USA, 1985. [Google Scholar]
  30. Felder, R.M.; Soloman, B.A. Index of Learning Styles Questionnaire. Available online: (accessed on 1 May 2017).
  31. Cha, H.J.; Yong, S.K.; Park, S.H.; Yoon, T.B.; Jung, Y.M.; Lee, J.H. Learning styles diagnosis based on user interface behaviors for the customization of learning interfaces in an intelligent tutoring system. Lect. Notes Comput. Sci. 2006, 4053, 513–524. [Google Scholar]
  32. García, P.; Amandi, A.; Schiaffino, S.; Campo, M. Evaluating bayesian networks’ precision for detecting students’ learning styles. Comput. Educ. 2007, 49, 794–808. [Google Scholar] [CrossRef]
  33. Brown, M.D.J.; Millichap, N. The Next Generation Digital Learning Environment: A Report on Research; EDUCASE Learning Initiative Paper; EDUCAUSE: Louisville, CO, USA, 2015; Volume 2015, p. 11. [Google Scholar]
  34. Graf, S.; Kinshuk, P. An approach for detecting learning styles in learning management systems. In Proceedings of the IEEE International Conference on Advanced Learning Technologies, Kerkrade, The Netherlands, 5–7 July 2006; pp. 161–163. [Google Scholar]
  35. Wang, J.; Yang, Y.; Mao, J.; Huang, Z.; Huang, C.; Xu, W. Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 27–30 June 2016; pp. 2285–2294. [Google Scholar]
  36. Hidasi, B.; Karatzoglou, A.; Baltrunas, L.; Tikk, D. Session-based recommendations with recurrent neural networks. In Proceedings of the International Conference on Learning Representations (ICLR) 2016, San Juan, Puerto Rico, 2–4 May 2016; Available online: (accessed on 9 February 2018).
  37. Cho, K.; Merrienboer, B.V.; Bahdanau, D.; Bengio, Y. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of the Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, Empirical Methods in Natural Language Processing, Doha, Qatar, 25 October 2014; pp. 103–111. [Google Scholar]
  38. Kingma, D.; Ba, J. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR) 2015, San Diego, CA, USA, 7 May 2015; Available online: (accessed on 9 February 2018).
  39. Steck, H. Gaussian ranking by matrix factorization. In Proceedings of the ACM Conference on Recommender Systems, Vienna, Austria, 16–20 September 2015; pp. 115–122. [Google Scholar]
  40. Alfaro-Perez, J.L. Moderating effect of learning styles on a learning management system’s success. Telemat. Inform. 2017, 34, 272–286. [Google Scholar]
Figure 1. The process of learning styles detection.
Figure 1. The process of learning styles detection.
Sustainability 10 00468 g001
Figure 2. Framework of enhanced adaptive LMS.
Figure 2. Framework of enhanced adaptive LMS.
Sustainability 10 00468 g002
Figure 3. Logic of dynamic User Learning Style detecting and learning path routing.
Figure 3. Logic of dynamic User Learning Style detecting and learning path routing.
Sustainability 10 00468 g003
Figure 4. Log for effective learning points density of two users.
Figure 4. Log for effective learning points density of two users.
Sustainability 10 00468 g004
Figure 5. Hybrid architecture for dynamic learning style detection and prediction.
Figure 5. Hybrid architecture for dynamic learning style detection and prediction.
Sustainability 10 00468 g005
Figure 6. Distribution of each learning style dimensions of users in training dataset labeled by Graf’s model.
Figure 6. Distribution of each learning style dimensions of users in training dataset labeled by Graf’s model.
Sustainability 10 00468 g006
Table 1. FSLSM Learning style dimensions and keywords group of learners’ behavior.
Table 1. FSLSM Learning style dimensions and keywords group of learners’ behavior.
DimensionsStyleKeywords Groups
ProcessingActiveTry something out
Social preferred
ReflectiveThinking more detail
Impersonal preferred
PerceiveSensingExisting ways
Concrete material
Careful with details
IntuitiveNew ways
Abstract material
Less attention to details
VerbalOral and written words
Difficulty with visual style
UnderstandingSequentialDetailed oriented
Sequential progress
From parts to the whole
GlobalOverall picture
Non-sequential progress
Table 2. Behavior logs for learning style dimensions of FSLSM.
Table 2. Behavior logs for learning style dimensions of FSLSM.
ChannelLearning StyleFeatured Behavior Logs
1Active/ReflectiveEffective learning points density log
2 Forum visit density log
3 Forum post density log
4Sensing/IntuitiveContent web page scrolling log
5 Video seeking log (Tracking forward/rewind clicks and timeframe)
6Visual/VerbalPicture stays timing log
7Sequential/GlobalNavigation Log
8 Effective learning points density log
Table 3. Statistics of the events for learning style change.
Table 3. Statistics of the events for learning style change.
from Reflective to Active103235.97%
from Active to Reflective37313.00%
from Intuitive to Sensing67323.46%
from Sensing to Intuitive42314.74%
from Verbal to Visual863.00%
from Visual to Verbal1214.22%
from Global to Sequential321.12%
from Sequential to Global1294.50%
Table 4. Sustainability-related MOOCs on xuetangX.
Table 4. Sustainability-related MOOCs on xuetangX.
Ecology in the humanistic perspectiveJilin University12,000 1
Green buildings and Sustainable developmentTsinghua University10,000 1
China’s perspective on climate changeTsinghua University6313
Air Pollution Control EngineeringTsinghua University5755
Hydraulic StructuresTsinghua University1993
Environmental pollution incidents and emergency responseBeijing Normal University753
Corporate Social Responsibility IntroductionTsinghua University348
Cyclic economy and sustainable development enterprisesBeijing Normal University215
Ecology and Environmental SciencesHuBei University143
1 Numbers more than 10,000 are rounded to the nearest thousand. Enrollments do not include SPOC (Small Private Online Course) platform.
Table 5. Action list of learning style change event detected.
Table 5. Action list of learning style change event detected.
EventAction 1Trigger
Reflective to ActivePush latest threads from forums>2
Active to ReflectiveSend message with quiz at last stops=1
Intuitive to SensingSend message with suggested reading materials=1
Sensing to IntuitiveSend message with extra relative reading materials=1
Verbal to VisualIgnore-
Visual to VerbalIgnore-
Global to SequentialSend message about reminder information for next chapter >2
Sequential to GlobalSend message for outline of course>2
1 Actions triggered by event or learning style does not change.
Table 6. Statistics for actions against learning style change.
Table 6. Statistics for actions against learning style change.
ActionTimesMessage Read Rate 1
Push latest threads from forums10325.24%
Send message with quiz at last stops42521.88%
Send message with suggested reading materials13517.77%
Send message with extra relative reading materials8313.25%
Send message about reminder information for next chapter 14036.42%
Send message for outline of course3215.6%
1 Message read rate is the rate of learners receiving the message and opening the link.
Table 7. Benchmark index of user interaction and course performance.
Table 7. Benchmark index of user interaction and course performance.
Index CategoriesIndexRound-2016#2Round-2017#1Improvement (%)
Net BenefitsCompletion Rate6.56%8.67%2.11% (32.16%)
Net BenefitsAverage quiz score71798 (11.2%)
UseDiscussion threads13615519 (13.97%)
UseDiscussion Replies30635448 (15.68%)
UseDiscussion Views11241349225 (20.02%)

Share and Cite

MDPI and ACS Style

Li, C.; Zhou, H. Enhancing the Efficiency of Massive Online Learning by Integrating Intelligent Analysis into MOOCs with an Application to Education of Sustainability. Sustainability 2018, 10, 468.

AMA Style

Li C, Zhou H. Enhancing the Efficiency of Massive Online Learning by Integrating Intelligent Analysis into MOOCs with an Application to Education of Sustainability. Sustainability. 2018; 10(2):468.

Chicago/Turabian Style

Li, Chao, and Hong Zhou. 2018. "Enhancing the Efficiency of Massive Online Learning by Integrating Intelligent Analysis into MOOCs with an Application to Education of Sustainability" Sustainability 10, no. 2: 468.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop