Transfer Learning from Deep Neural Networks for Predicting Student Performance
Abstract
:1. Introduction
2. The Transfer Learning Approach
- : the feature space of the source and target domain are different. For example, the courses have different structure and context;
- : the marginal probability distribution of the source and target domain are different. For example, the same courses offered by different departments or the same course offered in different years by the same department, thus consisting of different students;
- : the label spaces of the source and target task are different (this setting usually occurs with setting four). For example, the source domain has two classes (e.g., {pass, fail}) and the target domain has six classes (e.g., {A, B, C, D, E, F});
- : the conditional probability distributions of the source and target tasks are different. For example, the source and target courses are very unbalanced in relation to the defined classes.
3. Related Work
4. Research Methodology
4.1. Research Goal
4.2. Data Analysis
4.3. The Proposed Transfer Learning Approach
Algorithm 1: Transfer learning through paired course using deep neural networks. |
Input: c1, c2, scores = [[] # c1 is the source course dataset, and c2 is the target Output: scores # accuracy scores of the target course for every set of epochs 1: (c1′, c2′) ← commonRepresentation (c1, c2) # construct a common representation 2: model1 ← createAndCompileModel () # configure the deep learning process 3: weights ← fitModel (model1, c1′, epochs=150) # train the model, and save its weights 4: model2 ← createAndCompileModel(weights) # load the weights of pre-trained model1 5: for each in [0, 10, 20, 30, 40, 50, 100, 150] do 6: model2′ ← fitModel (model2, c2′, epochs=) # further tune the pre-trained model2 7: score ← evaluate (model2′, c2′, folds=10) # evaluate model2′ using the accuracy metric 8: add (score, scores) # save score for the output 9: end for each 10: 11: # construct a common feature representation for the two courses 12: function commonRepresentation (dataset1, dataset2) 13: dataset1′= [[], dataset2′= [[] # init empty datasets 14: # match the features of dataset1 with the features of dataset2, 15 # create new features when necessary 16: for each t in [‘forum’, ‘page’, ’recourse’, ‘folder’, ‘url’, ‘assign views’, ‘assign’] do 17: features1 ← getFeaturesOfType (dataset1, t) # get all features for this type 18: features2 ← getFeaturesOfType (dataset2, t) 19: size ← min (features1.size, features2.size) 20: diff ← absoluteDifference (features1.size, features2.size) 21: for i = 0 to size-1 do 22: add (features1[i], dataset1′) 23: add (features2[i], dataset2′) 24: end for 25: for j=0 to diff-1 do 26: if f1 ← getFeatureAt(features1, features1.size + j) do # if f1 exists 27: add (f1, dataset1′) 28: add (createEmptyFeature(), dataset2′) 29: else f2 ← getFeatureAt(features2, features2.size + j) # else f2 exists 30: add (f2, dataset2′) 31: add (createEmptyFeature(), dataset1′) 32: end if 33: end for 34: end for each 35:return dataset1′, dataset2′ # return the new datasets |
5. Results
6. Discussion
7. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Pan, S.J.; Yang, Q. A survey on transfer learning. Ieee Trans. Knowl. Data Eng. 2009, 22, 1345–1359. [Google Scholar] [CrossRef]
- Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016. [Google Scholar]
- Brownlee, J. Better Deep Learning: Train Faster, Reduce Overfitting, and Make Better Predictions; Machine Learning Mastery: Vermont Victoria, Australia, 2019. [Google Scholar]
- Brownlee, J. Deep Learning with Python: Develop Deep Learning Models on Theano and Tensorflow Using Keras; Machine Learning Mastery: Vermont Victoria, Australia, 2016. [Google Scholar]
- Ng, A. Nuts and bolts of building AI applications using Deep Learning. Nips Keynote Talk. In Proceedings of the Thirtieth Conference on Neural Information Processing Systems. 2016 NIPS’16, Barcelona, Spain, 5–10 December 2016. [Google Scholar]
- Yosinski, J.; Clune, J.; Bengio, Y.; Lipson, H. How transferable are features in deep neural networks? In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014. [Google Scholar]
- Liz-Domínguez, M.; Caeiro-Rodríguez, M.; Llamas-Nistal, M.; Mikic-Fonte, F.A. Systematic literature review of predictive analysis tools in higher education. Appl. Sci. 2019, 9, 5569. [Google Scholar] [CrossRef] [Green Version]
- Boyer, S.; Veeramachaneni, K. Transfer learning for predictive models in massive open online courses. In International Conference on Artificial Intelligence in Education; Springer: Berlin/Heidelberg, Germany, 2015. [Google Scholar]
- Ding, M.; Wang, Y.; Hemberg, E.; O’Reilly, U.-M. Transfer Learning using Representation Learning in Massive Open Online Courses. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, Tempe, AZ, USA, 4–8 March 2019. [Google Scholar]
- Boyer, S.A. Transfer Learning for Predictive Models in MOOCs. Ph.D. Thesis, Massachusetts Institute of Technology, Cambridge, MA, USA, 2016. [Google Scholar]
- Guo, B.; Zhang, R.; Xu, G.; Shi, C.; Yang, L. Predicting students performance in educational data mining. In Proceedings of the 2015 International Symposium on Educational Technology (ISET), Wuhan, China, 27–29 July 2015. [Google Scholar]
- Okubo, F.; Yamashita, T.; Shimada, A.; Ogata, H. A neural network approach for students’ performance prediction. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference, Vancouver, BC, Canada, 13–17 March 2017. [Google Scholar]
- Kim, B.-H.; Vizitei, E.; Ganapathi, V. GritNet: Student performance prediction with deep learning. arXiv 2018, arXiv:1804.07405. [Google Scholar]
- Kostopoulos, G.; Tsiakmaki, M.; Kotsiantis, S.; Ragos, O. Deep Dense Neural Network for Early Predic-tion of Failure-Prone Students. In Machine Learning Paradigms-Advances in Theory and Applications of Deep Learning; Springer: Berlin/Heidelberg, Germany, 2019. [Google Scholar]
- Wang, W.; Yu, H.; Miao, C. Deep model for dropout prediction in MOOCs. In Proceedings of the ACM International Conference Proceeding Series, Beijing, China, 6–9 July 2017. [Google Scholar]
- Whitehill, J.; Mohan, K.; Seaton, D.; Rosen, Y.; Tingley, D. Delving Deeper into MOOC Student Dropout Prediction. arXiv 2017, arXiv:1702.06404. [Google Scholar]
- Xing, W.; Du, D. Dropout prediction in MOOCs: Using deep learning for personalized intervention. J. Educ. Comput. Res. 2018, 57, 547–570. [Google Scholar] [CrossRef]
- Bosch, N.; Paquette, L. Unsupervised Deep Autoencoders for Feature Extraction with Educational Data. In Proceedings of the Deep Learning with Educational Data Workshop at the 10th International Conference on Educational Data Mining, Wuhan, Hubei, 25–28 June 2017. [Google Scholar]
- Ruder, S.; Peters, M.E.; Swayamdipta, S.; Wolf, T. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, Minneapolis, MN, USA, 2 June 2019. [Google Scholar]
- Weiss, K.; Khoshgoftaar, T.M.; Wang, D.D. A survey of transfer learning. J. Big Data 2016, 3, 1345–1459. [Google Scholar] [CrossRef] [Green Version]
- Arnold, A.; Nallapati, R.; Cohen, W.W. A Comparative Study of Methods for Transductive Transfer Learning. In Proceedings of the Seventh IEEE International Conference on Data Mining Workshops, Omaha, NE, USA, 28–31 October 2007. [Google Scholar]
- Romero, C.; Ventura, S. Educational data mining and learning analytics: An updated survey. Wiley Interdiscip. Rev. 2020, e1355. [Google Scholar] [CrossRef]
- Moreno-Marcos, P.M.; Alario-Hoyos, C.; Muñoz-Merino, P.J.; Kloos, C.D. Prediction in MOOCs: A review and future research directions. IEEE Trans. Learn. Technol. 2018, 12, 384–401. [Google Scholar] [CrossRef]
- Costa, E.B.; Fonseca, B.; Santana, M.A.; de Araújo, F.F.; Rego, J. Evaluating the effectiveness of educational data mining techniques for early prediction of students’ academic failure in introductory programming courses. Comput. Hum. Behav. 2017, 73, 247–256. [Google Scholar] [CrossRef]
- Vitiello, M.; Walk, S.; Chang, V.; Hernandez, R.; Helic, D.; Guetl, C. MOOC dropouts: A multi-system classifier. In Proceedings of the European Conference on Technology Enhanced Learning, Tallinn, Estonia, 12–15 September 2017. [Google Scholar]
- Hunt, X.J.; Kabul, I.K.; Silva, J. Transfer Learning for Education Data. In Proceedings of the KDD Workshop, Halifax, NS, Canada, 13–17 August 2017. [Google Scholar]
- Tri, P.T.; Chau, V.T.N.; Phung, N.H. Combining transfer learning and case-based reasoning for an educational decision making support model. In Proceedings of the Multi-disciplinary Trends in Artificial Intelligence: 11th International Workshop, MIWAI 2017, Gadong, Brunei, 20–22 November 2017. [Google Scholar]
- Zeng, Z.; Chaturvedi, S.; Bhat, S.; Roth, D. DiAd: Domain adaptation for learning at scale. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, Tempe, Arizona, 4–8 March 2019. [Google Scholar]
- López-Zambrano, J.; Lara, J.A.; Romero, C. Towards Portability of Models for Predicting Students’ Final Performance in University Courses Starting from Moodle Logs. Appl. Sci. 2020, 10, 354. [Google Scholar] [CrossRef] [Green Version]
- Tsiakmaki, M.; Kostopoulos, G.; Kotsiantis, S.; Ragos, O. Implementing AutoML in Educational Data Mining for Prediction Tasks. Appl. Sci. 2019, 10, 90. [Google Scholar] [CrossRef] [Green Version]
- Chollet, F. Keras. Available online: https://keras.io (accessed on 1 January 2020).
- Romero, C.; Ventura, S. Data mining in education. Wiley Interdiscip. Rev. 2013, 3, 12–27. [Google Scholar] [CrossRef]
- Wang, X.; Huang, T.-K.; Schneider, J. Active transfer learning under model shift. In Proceedings of the International Conference on Machine Learning, Beijing, China, 21–26 June 2014. [Google Scholar]
Course | Female | Male | Pass | Fail | ||||
---|---|---|---|---|---|---|---|---|
C1: Physical Chemistry I (Spring 2018) | 122 | 43.3% | 160 | 56.8% | 134 | 47.5% | 148 | 52.5% |
C2: Physics III (Spring 2018) | 90 | 50.0% | 90 | 50.0% | 74 | 41.1% | 106 | 56.9% |
C3: Analytical Chemistry Lab (Spring 2018) | 57 | 48.2% | 72 | 55.8% | 105 | 81.4% | 24 | 18.6% |
C4: Physics III (Spring 2019) | 80 | 50.6% | 78 | 49.4% | 68 | 43.0% | 90 | 57.0% |
C5: Analytical Chemistry Lab (Spring 2019) | 61 | 52.1% | 56 | 47.9% | 100 | 85.5% | 17 | 14.5% |
Learning Resources | Description | Possible Values | |||||
---|---|---|---|---|---|---|---|
Forum | 1 | 1 | 1 | 1 | 1 | Total number of times a student accessed the resource | 0 or positive integer |
Page | 7 | 6 | 2 | 2 | 0 | ||
Recourse | 17 | 15 | 4 | 12 | 10 | ||
Folder | 2 | 0 | 17 | 12 | 0 | ||
Url | 0 | 0 | 1 | 1 | 0 | ||
Assignments | 8 | 9 | 8 | 8 | 9 | ||
Submitted Assignments | 3 | 9 | 8 | 8 | 9 | Student Grades | [0, 10] decimal |
Forum views | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Page views | 7 | 7 | 8 | 7 | 0 | 0 | 2 | 0 | 2 | 2 |
Recourse views | 17 | 17 | 17 | 17 | 11 | 11 | 11 | 10 | 12 | 12 |
Folder views | 2 | 2 | 0 | 2 | 12 | 0 | 12 | 13 | 12 | 12 |
URL views | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 1 | 1 |
Assignments views | 13 | 8 | 9 | 8 | 9 | 9 | 9 | 9 | 8 | 9 |
Submitted Assignments | 8 | 8 | 9 | 8 | 8 | 9 | 8 | 9 | 8 | 9 |
Total views | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Total activity | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Gender | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Total Number of Features | 51 | 46 | 47 | 47 | 44 | 33 | 47 | 46 | 47 | 49 |
{C1,C2} | {C1,C3} | {C1,C4} | {C1,C5} | {C2,C3} | ||||||
C1 | C2 | C1 | C3 | C1 | C4 | C1 | C5 | C2 | C3 | |
Baseline | 0.7627 | 0.6094 | 0.7667 | 0.7047 | 0.7563 | 0.6333 | 0.7424 | 0.5591 | 0.6011 | 0.7144 |
Epochs | C2,1 | C1,2 | C3,1 | C1,3 | C4,1 | C1,4 | C5,1 | C1,5 | C3,2 | C2,3 |
0 | 0.6106 | 0.6172 | 0.7524 | 0.8227 | 0.5675 | 0.6371 | 0.5889 | 0.8644 | 0.6306 | 0.6538 |
10 | 0.7701 | 0.6414 | 0.7988 | 0.8382 | 0.8128 | 0.6387 | 0.8087 | 0.8561 | 0.5975 | 0.8234 |
20 | 0.7842 | 0.6299 | 0.7917 | 0.8537 | 0.8093 | 0.5833 | 0.8055 | 0.8553 | 0.6132 | 0.8382 |
30 | 0.7877 | 0.6132 | 0.8198 | 0.8537 | 0.8126 | 0.6008 | 0.8092 | 0.8644 | 0.6290 | 0.8394 |
40 | 0.7732 | 0.6234 | 0.8022 | 0.8608 | 0.8236 | 0.6075 | 0.8019 | 0.8553 | 0.6241 | 0.8453 |
50 | 0.7766 | 0.6076 | 0.8062 | 0.8465 | 0.8166 | 0.6325 | 0.7987 | 0.8386 | 0.6179 | 0.8537 |
100 | 0.7768 | 0.5968 | 0.7847 | 0.8465 | 0.8061 | 0.6762 | 0.7916 | 0.8114 | 0.6077 | 0.8394 |
150 | 0.7552 | 0.6185 | 0.7882 | 0.8620 | 0.7738 | 0.6325 | 0.7845 | 0.8371 | 0.6064 | 0.8472 |
{C2,C4} | {C2,C5} | {C3,C4} | {C3,C5} | {C4,C5} | ||||||
C2 | C4 | C2 | C5 | C3 | C4 | C3 | C5 | C4 | C5 | |
Baseline | 0.5650 | 0.6263 | 0.5498 | 0.6735 | 0.7096 | 0.6196 | 0.7715 | 0.7955 | 0.6811 | 0.6646 |
Epochs | C4,2 | C2,4 | C5,2 | C2,5 | C4,3 | C3,4 | C5,3 | C3,5 | C5,4 | C4,5 |
0 | 0.6207 | 0.6008 | 0.4112 | 0.8379 | 0.4641 | 0.4988 | 0.8156 | 0.8644 | 0.5375 | 0.6902 |
10 | 0.5941 | 0.5817 | 0.5916 | 0.8644 | 0.8310 | 0.5892 | 0.8156 | 0.8303 | 0.5758 | 0.7621 |
20 | 0.6154 | 0.6133 | 0.6031 | 0.8644 | 0.8394 | 0.6067 | 0.8239 | 0.8470 | 0.5888 | 0.8197 |
30 | 0.6120 | 0.5946 | 0.6188 | 0.8561 | 0.8465 | 0.5563 | 0.8322 | 0.8561 | 0.6013 | 0.8114 |
40 | 0.6074 | 0.6192 | 0.6130 | 0.8644 | 0.8406 | 0.5883 | 0.8310 | 0.8561 | 0.6017 | 0.8121 |
50 | 0.6330 | 0.6258 | 0.5938 | 0.8644 | 0.8549 | 0.6263 | 0.8251 | 0.8561 | 0.6529 | 0.8023 |
100 | 0.5885 | 0.6388 | 0.6272 | 0.8553 | 0.8322 | 0.6458 | 0.8329 | 0.8553 | 0.6538 | 0.7947 |
150 | 0.5947 | 0.6133 | 0.6105 | 0.7947 | 0.8251 | 0.6392 | 0.8299 | 0.8470 | 0.6346 | 0.8121 |
Epochs | p-Value |
---|---|
0 | 0.2449 |
10 | 0.0051 |
20 | 0.0023 |
30 | 0.0022 |
40 | 0.0013 |
50 | 0.0003 |
100 | 0.0002 |
150 | 0.0012 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tsiakmaki, M.; Kostopoulos, G.; Kotsiantis, S.; Ragos, O. Transfer Learning from Deep Neural Networks for Predicting Student Performance. Appl. Sci. 2020, 10, 2145. https://doi.org/10.3390/app10062145
Tsiakmaki M, Kostopoulos G, Kotsiantis S, Ragos O. Transfer Learning from Deep Neural Networks for Predicting Student Performance. Applied Sciences. 2020; 10(6):2145. https://doi.org/10.3390/app10062145
Chicago/Turabian StyleTsiakmaki, Maria, Georgios Kostopoulos, Sotiris Kotsiantis, and Omiros Ragos. 2020. "Transfer Learning from Deep Neural Networks for Predicting Student Performance" Applied Sciences 10, no. 6: 2145. https://doi.org/10.3390/app10062145
APA StyleTsiakmaki, M., Kostopoulos, G., Kotsiantis, S., & Ragos, O. (2020). Transfer Learning from Deep Neural Networks for Predicting Student Performance. Applied Sciences, 10(6), 2145. https://doi.org/10.3390/app10062145