# An Introduction to Bayesian Knowledge Tracing with pyBKT

^{1}

^{2}

^{3}

^{4}

^{5}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Bayesian Knowledge Tracing

## 3. Variants of Bayesian Knowledge Tracing

#### 3.1. IRT-BKT Model

#### 3.2. KT-IDEM Model

## 4. Estimating BKT Models with pyBKT

#### 4.1. Data

#### 4.2. The pyBKT Library

#### 4.3. Case Study 1: Estimating the Standard BKT Model

- Learner ID
- Question ID
- Skill name
- Learners’ dichotomous responses to questions

Listing 1. Installing necessary Python libraries. |

Listing 2. Importing the Cognitive Tutor dataset. |

Listing 3. Training BKT for a specific set of skills. |

Listing 4. Training BKT for each unique skill. |

Listing 5. Mapping the existing column names to expected pyBKT columns. |

Listing 6. Parameters for unknown states. |

Listing 7. Model evaluation. |

Listing 8. Custom evaluation metrics. |

Listing 9. Three-fold cross-validation. |

Listing 10. Training BKT variants. |

#### 4.4. Case Study 2: Comparing IRT and BKT in Modeling Response Accuracy

Listing 11. Knowledge prediction with BKT and IRT. |

#### 4.5. Case Study 3: Associations between IRT and BKT Parameters

Listing 12. Python code example linking BKT and IRT parameters. |

## 5. Discussion

#### Limitations and Future Research

## 6. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## Abbreviations

AUC | Area under the curve |

BKT | Bayesian knowledge tracing |

DKT | Deep Knowledge Tracing |

HMM | Hidden Markov Model |

IRT | Item Response Theory |

IRT-BKT | Knowledge Tracing Model on Item Response Theory |

ITS | Intelligent tutoring systems |

KT-IDEM | Knowledge Tracing Item Difficulty Effect model |

MAE | Mean absolute error |

RMSE | Root-mean-square error |

1PL | One-parameter logistic model |

3PL | Three-parameter logistic model |

## Appendix A

Listing A1. Python example. |

## References

- Pelánek, R. Bayesian knowledge tracing, logistic models, and beyond: An overview of learner modeling techniques. User Model. User-Adapt. Interact.
**2017**, 27, 313–350. [Google Scholar] [CrossRef] - Aleven, V.; Koedinger, K.R. Knowledge component (KC) approaches to learner modeling. Des. Recomm. Intell. Tutoring Syst.
**2013**, 1, 165–182. [Google Scholar] - Chrysafiadi, K.; Virvou, M. Student modeling approaches: A literature review for the last decade. Expert Syst. Appl.
**2013**, 40, 4715–4729. [Google Scholar] [CrossRef] - Woolf, B.P. Student Modeling. In Advances in Intelligent Tutoring Systems; Nkambou, R., Bourdeau, J., Mizoguchi, R., Eds.; Springer: Berlin/Heidelberg, Germany, 2010; pp. 267–279. [Google Scholar]
- Harrison, B.; Roberts, D. A review of student modeling techniques in intelligent tutoring systems. In Proceedings of the 8th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Stanford, CA, USA, 8–12 October 2012; Swanson, R., Vaish, R., Orkin, J., Niehaus, J., Godwin, J.A., Guarino, S., Youngblood, G.M., Eds.; The AAAI Press: Washington, DC, USA, 2012; pp. 61–66. [Google Scholar] [CrossRef]
- Am, E.H.; Hidayah, I.; Kusumawardani, S.S. A Literature Review of Knowledge Tracing for Student Modeling: Research Trends, Models, Datasets, and Challenges. J. Inf. Technol. Comput. Sci.
**2021**, 6, 344. [Google Scholar] [CrossRef] - Xiong, X.; Zhao, S.; Van Inwegen, E.G.; Beck, J.E. Going deeper with deep knowledge tracing. In Proceedings of the International Conference on Educational Data Mining (EDM), Raleigh, NC, USA, 29 June–2 July 2016. [Google Scholar]
- Minn, S.; Yu, Y.; Desmarais, M.C.; Zhu, F.; Vie, J.J. Deep knowledge tracing and dynamic student classification for knowledge tracing. In Proceedings of the 2018 IEEE International Conference on data Mining (ICDM), Singapore, 17–20 November 2018; pp. 1182–1187. [Google Scholar]
- Sapountzi, A.; Bhulai, S.; Cornelisz, I.; van Klaveren, C. Dynamic models for knowledge tracing & prediction of future performance. In Proceedings of the Seventh International Conference on Data Analytics, Athens, Greece, 18–22 November 2019. [Google Scholar]
- Carlon, M.K.J.; Cross, J.S. Knowledge tracing for adaptive learning in a metacognitive tutor. Open Educ. Stud.
**2022**, 4, 206–224. [Google Scholar] [CrossRef] - Corbett, A.T.; Anderson, J.R. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Model. User-Adapt. Interact.
**1994**, 4, 253–278. [Google Scholar] [CrossRef] - Pardos, Z.; Bergner, Y.; Seaton, D.; Pritchard, D. Adapting bayesian knowledge tracing to a massive open online course in edx. In Proceedings of the Educational Data Mining 2013, Memphis, TN, USA, 6–9 July 2013; Citeseer: Princeton, NJ, USA, 2013. [Google Scholar]
- Hawkins, W.J.; Heffernan, N.T.; Baker, R.S. Learning Bayesian knowledge tracing parameters with a knowledge heuristic and empirical probabilities. In Proceedings of the Intelligent Tutoring Systems: 12th International Conference, ITS 2014, Honolulu, HI, USA, 5–9 June 2014; Proceedings 12. Springer: Berlin/Heidelberg, Germany, 2014; pp. 150–155. [Google Scholar]
- Vaisakh, K.; Ravi, A.; Aakash, K.; Sai, A.; Bhaskar, J. SkillCrest: A Skill Assessment System Using Deep Knowledge Tracing and User Feedback Sentiment Analysis. In Proceedings of the 2021 2nd Global Conference for Advancement in Technology (GCAT), Bangalore, India, 1–3 October 2021; pp. 1–7. [Google Scholar]
- Ritter, S.; Anderson, J.R.; Koedinger, K.R.; Corbett, A. Cognitive Tutor: Applied research in mathematics education. Psychon. Bull. Rev.
**2007**, 14, 249–255. [Google Scholar] [CrossRef] [PubMed] - Sao Pedro, M.; Baker, R.; Gobert, J. Incorporating scaffolding and tutor context into bayesian knowledge tracing to predict inquiry skill acquisition. In Proceedings of the Educational Data Mining 2013, Memphis, TN, USA, 6–9 July 2013; Citeseer: Princeton, NJ, USA, 2013. [Google Scholar]
- Mao, Y. Deep Learning vs. Bayesian Knowledge Tracing: Student Models for Interventions. J. Educ. Data Min.
**2018**, 10, 28–54. [Google Scholar] - David, Y.B.; Segal, A.; Gal, Y. Sequencing educational content in classrooms using Bayesian knowledge tracing. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, Edinburgh, UK, 25–29 April 2016; pp. 354–363. [Google Scholar]
- Baker, R.S.D.; Corbett, A.T.; Gowda, S.M.; Wagner, A.Z.; MacLaren, B.A.; Kauffman, L.R.; Mitchell, A.P.; Giguere, S. Contextual slip and prediction of student performance after use of an intelligent tutor. In Proceedings of the User Modeling, Adaptation, and Personalization: 18th International Conference, UMAP 2010, Big Island, HI, USA, 20–24 June 2010; Proceedings 18. Springer: Berlin/Heidelberg, Germany, 2010; pp. 52–63. [Google Scholar]
- Naeem, M.; Tidswell, A.; Magdy, Y. Bayesian Knowledge Tracing for Assessment Results Analysis. In Proceedings of the 2021 17th International Computer Engineering Conference (ICENCO), Cairo, Egypt, 29–30 December 2021; pp. 30–34. [Google Scholar]
- Qiu, Y.; Qi, Y.; Lu, H.; Pardos, Z.A.; Heffernan, N.T. Does Time Matter? Modeling the Effect of Time with Bayesian Knowledge Tracing. In Proceedings of the EDM, Eindhoven, The Netherlands, 6–8 July 2011; pp. 139–148. [Google Scholar]
- Badrinath, A.; Wang, F.; Pardos, Z. pyBKT: An accessible Python library of Bayesian Knowledge Tracing Models. In Proceedings of the 14th International Conference on Educational Data Mining, Virtual Event, 29 June–2 July 2021; pp. 468–474. [Google Scholar]
- Piech, C.; Bassen, J.; Huang, J.; Ganguli, S.; Sahami, M.; Guibas, L.J.; Sohl-Dickstein, J. Deep Knowledge Tracing. In Proceedings of the Advances in Neural Information Processing Systems, NIPS 2015, Montreal, QC, Canada, 7–12 December 2015; Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R., Eds.; Curran Associates, Inc.: New York, NY, USA, 2015; Volume 28. [Google Scholar]
- Corbett, A. Cognitive computer tutors: Solving the two-sigma problem. In Proceedings of the User Modeling 2001: 8th International Conference, UM 2001, Sonthofen, Germany, 13–17 July 2001; Springer: Berlin/Heidelberg, Germany, 2001; pp. 137–147. [Google Scholar]
- Käser, T.; Klingler, S.; Schwing, A.G.; Gross, M. Dynamic Bayesian networks for student modeling. IEEE Trans. Learn. Technol.
**2017**, 10, 450–462. [Google Scholar] [CrossRef] - Pardos, Z.A.; Heffernan, N.T. KT-IDEM: Introducing item difficulty to the knowledge tracing model. In Proceedings of the User Modeling, Adaption and Personalization: 19th International Conference, UMAP 2011, Girona, Spain, 11–15 July 2011; Proceedings 19. Springer: Berlin/Heidelberg, Germany, 2011; pp. 243–254. [Google Scholar]
- Yudelson, M.V.; Koedinger, K.R.; Gordon, G.J. Individualized bayesian knowledge tracing models. In Proceedings of the Artificial Intelligence in Education: 16th International Conference, AIED 2013, Memphis, TN, USA, 9–13 July 2013; Proceedings 16. Springer: Berlin/Heidelberg, Germany, 2013; pp. 171–180. [Google Scholar]
- Baker, R.S.d.; Corbett, A.T.; Aleven, V. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In Proceedings of the Intelligent Tutoring Systems: 9th International Conference, ITS 2008, Montreal, QC, Canada, 23–27 June 2008; Proceedings 9. Springer: Berlin/Heidelberg, Germany, 2008; pp. 406–415. [Google Scholar]
- Deonovic, B.; Yudelson, M.; Bolsinova, M.; Attali, M.; Maris, G. Learning meets assessment: On the relation between item response theory and Bayesian knowledge tracing. Behaviormetrika
**2018**, 45, 457–474. [Google Scholar] [CrossRef] [Green Version] - Wang, F.; Huang, Z.; Liu, Q.; Chen, E.; Yin, Y.; Ma, J.; Wang, S. Dynamic Cognitive Diagnosis: An Educational Priors-Enhanced Deep Knowledge Tracing Perspective. IEEE Trans. Learn. Technol.
**2023**, 16, 306–323. [Google Scholar] [CrossRef] - Kang, T.; Cohen, A.S.; Sung, H.J. IRT model selection methods for polytomous items. In Proceedings of the Annual Meeting of the National Council on Measurement in Education, Montreal, QC, Canada, 11–15 April 2005. [Google Scholar]
- Khajah, M.M.; Huang, Y.; González-Brenes, J.P.; Mozer, M.C.; Brusilovsky, P. Integrating knowledge tracing and item response theory: A tale of two frameworks. In Proceedings of the CEUR Workshop Proceedings, Crete, Greece, 27 May 2014; University of Pittsburgh: Pittsburg, PA, USA, 2014; Volume 1181, pp. 7–15. [Google Scholar]
- Wang, S.; Han, Y.; Wu, W.; Hu, Z. Modeling student learning outcomes in studying programming language course. In Proceedings of the 2017 Seventh International Conference on Information Science and Technology (ICIST), Da Nang, Vietnam, 16–19 April 2017; pp. 263–270. [Google Scholar]
- Xu, Y.; Mostow, J. Using item response theory to refine knowledge tracing. In Proceedings of the Educational Data Mining 2013, Memphis, TN, USA, 6–9 July 2013. [Google Scholar]
- Deonovic, B.; Chopade, P.; Yudelson, M.; de la Torre, J.; von Davier, A.A. Application of cognitive diagnostic models to learning and assessment systems. Handbook of Diagnostic Classification Models: Models and Model Extensions, Applications, Software Packages; Springer: Berlin/Heidelberg, Germany, 2019; pp. 437–460. [Google Scholar]
- Gorgun, G.; Bulut, O. Considering Disengaged Responses in Bayesian and Deep Knowledge Tracing. In Proceedings of the Artificial Intelligence in Education, Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium, Durham, UK, 27–31 July 2022; Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V., Eds.; Springer: Cham, Switzerland, 2022; pp. 591–594. [Google Scholar]
- Liu, Z.; Liu, Q.; Chen, J.; Huang, S.; Tang, J.; Luo, W. pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models. arXiv
**2023**, arXiv:cs.LG/2206.11460. [Google Scholar] - Wang, Y.; Heffernan, N. Extending knowledge tracing to allow partial credit: Using continuous versus binary nodes. In Proceedings of the Artificial Intelligence in Education: 16th International Conference, AIED 2013, Memphis, TN, USA, 9–13 July 2013; Proceedings 16. Springer: Berlin/Heidelberg, Germany, 2013; pp. 181–188. [Google Scholar]
- Ghosh, A.; Raspat, J.; Lan, A. Option Tracing: Beyond Correctness Analysis in Knowledge Tracing. arXiv
**2021**, arXiv:2104.09043. [Google Scholar] [CrossRef]

Knowledge Components | Number of Items | Number of Students |
---|---|---|

Calculate part in proportion with fractions | 112 | 453 |

Calculate the total in proportion with fractions | 88 | 451 |

Calculate unit rate | 200 | 470 |

Finding the intersection, GLF | 22 | 72 |

Finding the intersection, Mixed | 16 | 74 |

Finding the intersection, SIF | 14 | 71 |

Plot decimal - thousandths | 14 | 251 |

Plot imperfect radical | 21 | 253 |

Plot non-terminating improper fraction | 13 | 255 |

Plot pi | 10 | 253 |

Plot terminating proper fraction | 31 | 256 |

Plot whole number | 9 | 256 |

Knowledge Components | Number of Problems | Correlation |
---|---|---|

Calculate part in proportion with fractions | 112 | 0.809 |

Calculate total in proportion with fractions | 88 | 0.821 |

Calculate unit rate | 200 | 0.808 |

Finding the intersection, GLF | 22 | 0.550 |

Finding the intersection, Mixed | 16 | 0.937 |

Finding the intersection, SIF | 14 | 0.879 |

Plot decimal - thousandths | 14 | 0.911 |

Plot imperfect radical | 21 | 0.904 |

Plot pi | 10 | 0.919 |

Plot terminating proper fraction | 31 | 0.931 |

Plot whole number | 9 | 0.797 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Bulut, O.; Shin, J.; Yildirim-Erbasli, S.N.; Gorgun, G.; Pardos, Z.A.
An Introduction to Bayesian Knowledge Tracing with pyBKT. *Psych* **2023**, *5*, 770-786.
https://doi.org/10.3390/psych5030050

**AMA Style**

Bulut O, Shin J, Yildirim-Erbasli SN, Gorgun G, Pardos ZA.
An Introduction to Bayesian Knowledge Tracing with pyBKT. *Psych*. 2023; 5(3):770-786.
https://doi.org/10.3390/psych5030050

**Chicago/Turabian Style**

Bulut, Okan, Jinnie Shin, Seyma N. Yildirim-Erbasli, Guher Gorgun, and Zachary A. Pardos.
2023. "An Introduction to Bayesian Knowledge Tracing with pyBKT" *Psych* 5, no. 3: 770-786.
https://doi.org/10.3390/psych5030050