Combating the Problems of the Necessity of Continuous Learning in Medical Datasets of Type 2 Diabetes ^†

Abstract

There are two major problems that researchers must contend with when dealing with machine learning in the medical field. The first being the ever-changing nature of what is considered good practice, and the lack of available data to train. The change in opinion on what is considered good practice requires an ongoing effort to update the machine learning models. This requires a concept called continual learning, which requires that researchers must combat against the problem of a model forgetting previously learned information and the balancing of bigger classes and newer smaller classes. Usually, when new information is introduced, a model must be retrained, which threatens the previously gained knowledge. The training is then difficult because of the lack of data. However, when dealing with medications they can become irrelevant to use. When such things happen when dealing with the standard machine learning models, the entire model needs to be retrained in order to remove the specific medication. This causes even more difficulties, because patient data are heavily protected and there is the chance that the dataset will not be available for training again in the future. While most papers focus on medical imaging and diagnosis, medicine does not end with diagnosis. We have an elderly population that is growing and not enough doctors are are available. To have everyone be able to see specialists, or even a doctor, is becoming even harder. To combat these issues, we need to have models in use that we can update continuously to help bridge the gap of care. We propose a method that can be trained continuously in order to easily remove outdated medications.

1. Introduction

If machine learning seeks to be of assistance to medical practitioners, it should first seek to be understandable to them. All the risk and responsibility fall to practitioners to ensure that patients receive the best care possible. In that case, machine learning needs to give a valuable second opinion that they can trust to be useful.

Machine learning is currently viewed as being limited to areas that have a large dataset. The trade-off is that the greater the accuracy of prediction, the more the model becomes an non-understandable black box, which plays into the legal and ethical questions the medical field has about the role machine learning can play.d A paper by Klincewicz and Frank highlights two moral foundations that they view as vital for machine learning to meet in order to be viewed as trustworthy or useful. Those are “the position of the trusted doctor and respect for patient autonomy” [1].

We show in this research paper that it is possible to have high accuracy and a clearly understandable model for the use of practitioners that can be trained on an ongoing basis for the future.

Medicine is a fast-moving area of research. New discoveries are made, guidelines are revised, and different technologies become available. We must continuously change to keep in line with updated guidelines, new findings, and new medication. This means that prescription classification is a never-ending continuous learning problem. We can never consider a model to be completely trained. This is a pressing challenge to take on due to several upcoming circumstances. Even more challenging is that as new medications become available, the number of samples for them is incredibly small compared to the already known medications that have much larger sample numbers.

There is an expanding elderly generation throughout the world and a lack of practitioners. Most people see general practitioners when they need to see specialists. The United States of America alone is forecasting a shortage of at least 122,000 physicians by 2032 [2].

The number of specialists available is simply not enough. This is especially true for diseases, including for type 2 diabetes. Instead of seeing a specialist, patients are forced to seek out generalists to receive care.

To help with this problem, machine learning can be used to bridge that gap between the number of people seeing generalists and their need to see specialists. By being trained on a dataset made from the decision making of specialists, we can give generalists a second opinion on medication prescriptions for type 2 diabetics. We cannot assume that this model can be trained just once. As medical opinion evolves, so should the model. This is where we can use continual learning.

Continual learning is common in certain areas of medical machine learning. For instance, medical image analysis [3]. Rapid development of deep-learning algorithms has advanced the research area such as tumor diagnosis. A problem that developed in this advancement is the following: when the inference dataset slightly differs from what the model was trained on, the model’s performance becomes compromised, requiring the retraining of the model from scratch. This is inefficient for multiple reasons, the least of which is the time it takes to retrain the model.

Instead, continual learning has to be used to help the models adapt and change over time. However, unlike image analysis, which is continuously updating to achieve better accuracy at a single focus, we can assume that medical prescription classification will at some point want to remove certain options from the pool of medications.

It is for this purpose that we use a series of binary classifications instead of multi-classification. With multi-classification, the more classes are added to a model, the more data are needed to train that model. Medical data are scarce due to lack of organized records or patient privacy. Such specialized datasets are hard to come by. Training such a large model is difficult. The lack of data can be overcome by the use of techniques like SMOTE (artificial data generation) when used in moderation, but these techniques cannot overcome such a huge gap that is needed to train a model with nine classes and an unbalanced dataset.

We propose that instead of trying to overcome this gap in multi-classification the class be split into binary classification models for each pair of combinations. Also, to overcome the lack of data, that the dataset be preprocessed by using an imputation algorithm, stratified k-fold, and SMOTE.

2. Materials and Methods

2.1. Models

To demonstrate the difference between multi- and binary classification on a small dataset, six different machine learning models are compared on multi-classification: XGBoost, Support Vector Machine (SVM), Multilayer Perceptron (MLP), Random Forest (RF), KNN Neighbors, and Logistic Regression (LR).

There are a mix of different types of machine learning models. XGBoost and Random Forest are decision trees. Both are also supervised learning algorithms. A decision tree is a branch structure made up of nodes and paths. It starts with a single node, where each node has a value determination that decides which of two paths it follows. From therem, it branches out into multiple possible outcomes and each branch represents one of the possible outcomes until the final node is reached and a decision is made. An example of a decision tree is shown in Figure 1. The paths branch according to the condition of each node to continue on that path.

SVM, MLP, KNN, and LR are all supervised learning algorithms. However, they all have different levels of interpretability. Decision trees are easy to read. There is no black box. SVM is also easy to understand as it is always trying to find the best divider between groups. MLP is the hardest to understand and is the origin of what is called the black box. It is a series of weights and biases that are manipulated, which is unreadable and must simply be trusted. KNN is simply classified according to proximity, and LR is based on the probability of an event according to a set of independent variables.

They may seem simple-to-understand models, but as opposed to the decision trees, they are not readable. A practitioner is able to look through a decision tree and see at each node why and from what feature it is making its decisions.

2.1.1. Multi-Classification

XGBoost is a decision tree that uses a One-vs-Rest (OvR) multiclass classifier. OvR is where the model trains in a binary way, where there is a target class that is trained against all other classes that are grouped together and considered as a single class. XGBoost is also a boosting algorithm, where the model is trained sequentially by adding simple models to correct for the errors made by the errors earlier in training. That is what differentiates it from other decision trees like Random Forest [4,5].

Support Vector Machine is a supervised machine learning algorithm that uses hyperplanes to separate different classes and classify them. Hyperplanes are lines that act as a separation between classes. If classes have too much overlap, a clean separation cannot be made. Multi-classification is not native to SVM, but just as XGBoost uses OvR to turn a multi-classification problem into a binary, so does SVM [6,7].

Multi-layer Perceptron is a neural network. Neural networks are powerful algorithms, but have a difficult time with tabular data such as medical data. Unlike the other models, this algorithm is based on updating weights and biases to minimize differences between predicted classes and the actual classes. This creates a seemingly black box where numbers are input, undergo transformation, and then the model returns a prediction of classification [8,9].

K-Nearest Neighbors is a supervised learning algorithm that classifies according to what is closest to the given sample. Given the distance between a given sample and the ones closest to it, based on the class of the closest samples, the model decides that a sample must belong to that class. Over time, it groups those data points together [10].

Random Forest is a decision tree, the same type of algorithm as XGBoost, but the difference is that it combines the output of multiple decision trees into a single result that is the prediction of class. Unlike XGBoost which has each tree in a series built off of the last one and only uses the most recent, Random Forest uses each individual tree that is trained [11,12].

Logistic Regression is a statistical model that is commonly used in the medical field, which is why it was included in this experiment. Using multi-classification with logistic regression is an expansion upon binary logistic regression by simply isolating each class and building a regression for it that calculates the probability that an observation belongs to a given class. When given a sample, it selects the class with the highest probability for the given sample [13].

2.1.2. Binary Classification

For binary classification, we train each possible combination of medications. We do this for two reasons: binary classification does not require as much data to obtain a better result, and it easily separates out the medications so that every time we want to get rid of a medication it does not take retraining the whole model itself. Therefore, each combination becomes its own model, thus making it a binary classification.

2.2. Experiment

Experimental Data

The data were extracted on patients from December 2009 to March 2015 by software (CoDiC) developed by the Japan Diabetes Clinical Data Management Study Group (JDDM) to promote clinical research on diabetes. JDDM is a large network of diabetes specialists in Japan within 98 facilities. Study participants were individuals aged 20 years or older who regularly visit hospitals for at least 3 months in outpatient clinics. Data were analyzed on 16,691 patients who were prescribed antidiabetic medications as the initial medical treatment (n = 16,691).

For each patient, the data include 11 features: age, body mass index (BMI), duration of diabetes, systolic blood pressure (SBP), diastolic blood pressure (DBP), HbA1c, renal function (CRE), total cholesterol (TCHO), high-density lipoprotein cholesterol (HDLC), aspartate aminotransferase (AST), and platelet count (PLT). In addition, the kind of treatment provided to the patient is recorded in the data. The types of treatments are divided into nine categories: insulin (INS), biguanides (BG), sulfonylureas (SU), dipeptidyl peptidase-4 inhibitors (DPP-4I), alpha-glucosidase inhibitors(GI), and glinides (Grinide), (TZD), (SGLT), and (GLP). The numbers of patients for SU, BG, DPP, INS, GI, Grinide, TZD, SGLT, and GLP are 3739, 3558, 2957, 2215, 1849, 1426, 763, 116, and 68 respectively. JDDM is a collection of over 100 medical facilities. The data were collected from patients that come from a variety of doctors who have differing views on medication decisions.

This dataset has a lot of NAN (not a number) values. This means that for a feature, that sample has no value. Many machine learning models cannot be trained when there are NAN values. Other than XGBoost, none of the models that were used in this research are able to train with NAN values. This creates a problem, but there are at least three possible ways of dealing with the issue.

One is to drop any sample that has a NAN value. For medical datasets, this is unadvised. Multiple classes have less than 1000 samples. GLP has less than 100. Any sample lost is too high of a cost. Another is to change from a NAN value into a different value. It is common to replace the NAN with the mean of the column. Again, this is not advised for a medical dataset.

Lastly, we can use an imputer method. Imputer methods are used when a model is trained on the dataset itself on what the missing values should be depending on the class it belongs to. This is the method that we chose to use for this experiment. Figure 2 shows an overview of the pipelined methods.

MissForest was the imputer that was used. Unlike the other methods, this one uses machine learning itself to figure out what the best input would be based on other samples and their feature values. One of the popular imputation algorithms is KNN itself, but it has many weaknesses such as being sensitive to outliers and noise. On the other hand, MissForest is based off of the Random Forest algorithm. In 2011, MissForest was compared to other imputation methods on datasets with randomly missing values, which MissForest was able to outperform all others on, including the aforementioned KNN imputation [14].

To summarize the method, the NAN values are first filled in by using the median of the feature column. The missing values are what we try to predict and the others are viewed as training rows. These are the data that are given to the Random Forest model for prediction. Using what MissForest predicts, the missing information is inserted in the NAN’s place. We repeat the process several times, each time a new iteration improving upon the previous attempt.

Drawbacks of MissForest are that it is computationally expensive, and as a model, it must be run every time data are imputed [14]. So if an increasing amount of data with missing values is received, when the model is trained in a continuous way, the imputer method needs to be run on the data. This can be easily overcome by requiring that new data have no NAN values. Medical data have a lot of restrictions, but if certain features are simply too hard to always have values for or are simply rare, then they can be removed from the dataset. Certain features are guaranteed to be included, specifically for type 2 diabetes such as HBA1c, or for any given disease that has its own dataset to train from. The curation of the data has to be made on a case-by-case basis for any given disease.

It is important to note that these new values are not random. They are picked based on the predictive value given by the other samples that are similar to it and the classification they ascribed to. We can now use this dataset to train our models.

Lastly, we need to address the extreme imbalance in the dataset. It is easy to simply remove classes that are too small, but that is not true to the reality of medical data. Unless we wait years to build up to what is considered a sufficient number of trainable samples, we must try to overcome this limitation. To address the unbalanced aspect of the dataset, we attempt two things: Stratified K-Fold and using SMOTE.

Stratified K-Fold is a cross-validation technique. We use cross-validation techniques to split the data in order to ensure better training. Stratified builds upon this by making sure that in every group that is trained, each fold has the same proportion of the classes that the entire dataset does, thereby ensuring that the smaller classes are always trained in each fold nstead of skipping them for multiple folds.

SMOTE stands for Synthetic Minority Over-Sampling Technique. We want to prevent higher count classes from overwhelming the model and causing bias towards them, while K-Fold ensures that every class is there in each fold. When we have our fold start, we deploy SMOTE to help the model. This is achieved by SMOTE synthesising new minority instances of the smaller classes. At the onset of each fold, we have a proportionally representative amount of each class. SMOTE then takes the minority classes and artificially creates more that are based on the class samples.

This model uses a split of n = 10.

3. Results

3.1. Multi-Classification

For this model, we combined all the medications into a single dataset, used MissForest to fill in the NAN values, and then used the Stratified K-Fold to train.

In

Table 1. Binary classification of medication combinations.

Model	Accuracy
XGBoost	25.4%
Support Vector Machine	14.8%
Multi-layer Perceptron	18.5%
Random Forest	23.3%
Logistic Regression	16.1%
KNN	14.7%

, we show the accuracy of the model after using Stratified K-Fold in combination with SMOTE within each split to train the model.

As shown in Table 1, XGBoost has the highest accuracy among all of the trained models. For these reasons, we can split up all the medication combinations into their own models. Once we see this split in action, we can observe an increase in accuracy compared to putting them all into one model. The model has a hard time generalizing to make good enough predictions.

KNN is the worst performing model, which could express that there is a large amount of overlap between classes. Random Forest nearly did as well as XGBoost and as the only other decision tree did far better than all of the other models. SVM is equally as bad a KNN, which adds credence to the hypothesis of a high level of overlap between the samples.

The accuracy of the models overall is extremely low and as such is of no use. More data are needed to train a model to generalize better, but that route is not available to us. It would take a lot of time in order to collect such specialized data. We cannot be assured that we would be able to ever collect enough data over time, or that the field of medicine for type 2 diabetes would stay the same while we did so. Instead, we can leverage the advantage of binary classification.

3.2. Binary Classification

Here, we can show a method that helps increase the accuracy of our model, but also now integrate our continual learning. Instead of using multi-classification, which would be easier to train, we split every possible combination of medications into their own models.

As we can see from

Table 2. Binary Classification of Medication Combinations.

VS	BG	SU	DPP	INS	GI	GR	TZD	SGLT	GLP
BG		63%	61%	65%	64%	68%	78%	96%	97%
SU			64%	63%	64%	67%	80%	96%	97%
DPP				63%	60%	68%	77%	95%	97%
INS					59%	63%	72%	94%	96%
GI						59%	64%	92%	94%
GR							62%	91%	94%
TZD								84%	89%
SGLT									72%
GLP

, there is a massive increase in accuracy. It has effectively tripled. By limiting the model to only two classes at a time, we can leverage the power of binary classification when dealing with smaller datasets and increase our prediction ability. This does require more effort upfront to train so many models, but the time and effort is saved once the models are trained and do not have to all be updated any time something is changed.

This is also useful in our continuous learning efforts. By having every combination separate, if we need to eliminate one medication from the rotation, we simply need to eliminate that medication’s models. Additionally when we continue to train these models, we do not have to train all the models or the one singular model at once. We are able to train the models that can be updated with new samples while leaving the others alone.

Of all the classes, the best performing overall ones are SGLT and GLP, which are the two smallest classes of the entire dataset with a count of 116 and 68, respectively. The fact that the smallest classes were able to perform the best even though they are multiple times smaller than the larger classes of SU or BG shows that binary classification is better used than multi-classification.

The worst performing classes were still above an accuracy of 60 percent. A single match of Grinide vs. GI performed the worst at 59 percent. This is a much better performing average than the multi-classification model that was only able to reach 25 percent accuracy.

4. Discussion

Once a general model is created, we need to accommodate for future improvements. We assume that the patient requires medication. The vast number of medications available for type 2 diabetes is oral medication. It is a list that is constantly changing, with new medications becoming available and/or other medications being faded out or replaced. As such, the samples available for each medication vary, and dealing with an unbalanced dataset is another challenge.

Once a model is trained, what we are most concerned with in continual learning is catastrophic forgetting. However, medicine is a unique mixture of learning new things and forgetting things that were not working or are considered outdated. Sometimes it is desirable to completely get rid of certain medications but not have to retrain an entire model from scratch to update it with new medications or get rid of others.

What is usually performed is called static learning. This is when a model is trained once and considered ready for use. The model no longer needs to be updated on an ongoing basis. If changes are needed or the model deteriorates, then the current model is discarded and a new model is trained to take its place. The new model does not keep track of what the old model learned, so previously learned information is not kept track of.

As new samples become available, we want to keep training the model so that it improves over time. Medical data are sensitive and slow to be collected. Instead of retraining a model every time new samples become available, it would be easier to slowly train the model over time. This is continual learning. The model can be updated as fast as new samples become available. This is called stateful training, because we are retaining the previous state of the model and building upon it. There exists no overlap between the new and old datasets. We are not creating a new model; it is an updated version of the same one.

Most research conducted with continual learning is carried out by assuming the model in question has a neural network and that updating the weights of the network is the intended goal. Neural networks, however, do not perform as well on tabular data as other networks do. Another problem is the issue of the black box of neural networks. Since the ultimate responsibility of the patients falls upon the practitioner, we cannot ignore the fact that practitioners need to be able to understand the reasoning behind why a model makes a decision.

Now, we have a combined problem of needing to update models and for those models to be understandable. Some problems require the use of heavily complicated models. Imaging, as well as other data such as audio, require neural networks to obtain the best classification possible. However, for this experiment’s data, we use tabular pre-classified samples. As such, in this research, we choose to use a decision tree as shown in the results.

For this purpose, XGBoost is especially useful because it already has continual learning available by being able to pass the old model as an attribute of the new model that will be trained with the newly available data. For medical data, there is the possibility that it is in fact wanted for the model to forget certain medications. Over time, medications may become outdated and thus, to remove them, complete retraining of the model may be needed. This could lead to forgetting or a worse model overall.

Other drawbacks of keeping to only a multi-classification is that as new medications are included, more data are needed for training. New medications will not have datasets big enough to compare to the older medications that have been in training for longer. Also, with more data to train, it is not only the new medication data that are needed. More data overall from all classes are needed to match the computer’s need for more data. Unbalanced data can be managed easier by using a binary model than multi-classification.

There are limitations and problems for continuous learning: the time commitment to keep training, the potential financial costs to keep an ongoing number of samples to train coming in, the information overload, or the catastrophic forgetting of older learned information. Such limitations are beyond the scope of this paper as the model we use is XGBoost and not a neural network which is the model that continuous learning is usually used for. Also, by limiting ourselves to binary classification, we can limit the drawbacks for ourselves.

5. Conclusions

In this research paper, we show a method to accommodate for the continual learning of small medical datasets that can be understandable to practitioners. By using a combination of MissForest to replace NAN values, Stratified K-Fold to train the models by cross-validation, SMOTE to overcome the challenge of the smaller classes, and then binary classification instead of multi-classification, we can greatly increase the accuracy of our predictions.

Currently, machine learning is not widely utilized in the ongoing care of patients. Most machine learning advancements in medicine pertain to the training of models for the diagnosis of diseases or of images such as tumor identification. In order for machine learning to become a more widely trusted second opinion for practitioners, it must be understandable and demonstrate an ability to improve with training.

Decision trees are easily understandable to practitioners who can read through each node and know why the decision tree reached any given decision that it made. While decision trees can be extremely long and in depth, they are never impossible to be understood but simply time consuming.

As data become more and more available and the models can be trained over time, we can expect them to continue to increase in accuracy and therefore increase in usability. As that accuracy increases, more trust will be afforded to these models as a valuable second opinion for practitioners. It will be necessary in the future for the health care of patients to overcome the lack of personal care they might receive due to the gap of the number of patients compared to the number of available practitioners.