Advancing Pancreatic Cancer Prediction with a Next Visit Token Prediction Head on Top of Med-BERT
1
McWilliams School of Biomedical Informatics, UTHealth at Houston, Houston, TX 77030, USA
2
Department of Artificial Intelligence and Informatics, Mayo Clinic, Jacksonville, FL 32224, USA
*
Authors to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Cancers 2025, 17(3), 516; https://doi.org/10.3390/cancers17030516
Submission received: 6 January 2025
/
Revised: 27 January 2025
/
Accepted: 30 January 2025
/
Published: 4 February 2025
(This article belongs to the Special Issue Novel Computational and Artificial Intelligence Models in Cancer Research)
Simple Summary
Pancreatic cancer (PaCa) is estimated to be the fourth leading cause of cancer death in men, following lung, colon, and prostate cancers, and the third leading cause in women, following lung and breast cancers. PaCa is often referred to as a “silent killer” because symptoms typically manifest only in the late stages of the disease. Consequently, early detection is crucial for improving patient outcomes. This study explores the use of electronic health records (EHRs) to enhance the prediction of PaCa onset. Our research leverages Med-BERT, a foundation model designed for EHR data, to improve early detection using deep learning techniques. By aligning the prediction task with Med-BERT’s pretraining task, we aimed to enhance its accuracy, especially in scenarios with limited data. This approach can facilitate the earlier detection of PaCa in patients, thereby improving their prognosis.
Abstract
Background: Electronic Health Records (EHRs) encompass valuable data essential for disease prediction. The application of artificial intelligence (AI), particularly deep learning, significantly enhances disease prediction by analyzing extensive EHR datasets to identify hidden patterns, facilitating early detection. Recently, numerous foundation models pretrained on extensive data have demonstrated efficacy in disease prediction using EHRs. However, there remains some unanswered questions on how to best utilize such models, especially with very small fine-tuning cohorts. Methods: We utilized Med-BERT, an EHR-specific foundation model, and reformulated the disease binary prediction task into a token prediction task and a next visit mask token prediction task to align with Med-BERT’s pretraining task format in order to improve the accuracy of pancreatic cancer (PaCa) prediction in both few-shot and fully supervised settings. Results: The reformulation of the task into a token prediction task, referred to as Med-BERT-Sum, demonstrated slightly superior performance in both few-shot scenarios and larger data samples. Furthermore, reformulating the prediction task as a Next Visit Mask Token Prediction task (Med-BERT-Mask) significantly outperformed the conventional Binary Classification (BC) prediction task (Med-BERT-BC) by 3% to 7% in few-shot scenarios with data sizes ranging from 10 to 500 samples. These findings highlight that aligning the downstream task with Med-BERT’s pretraining objectives substantially enhances the model’s predictive capabilities, thereby improving its effectiveness in predicting both rare and common diseases. Conclusions: Reformatting disease prediction tasks to align with the pretraining of foundation models enhances prediction accuracy, leading to earlier detection and timely intervention. This approach improves treatment effectiveness, survival rates, and overall patient outcomes for PaCa and potentially other cancers.
Keywords:
foundation model; pancreatic cancer; masked language model
