LSTM DSS Automatism and Dataset Optimization for Diabetes Prediction ^†

Featured Application

Implementation of DSS of patient management system based on LSTM for homecare assistance.

Abstract

The paper is focused on the application of Long Short-Term Memory (LSTM) neural network enabling patient health status prediction focusing the attention on diabetes. The proposed topic is an upgrade of a Multi-Layer Perceptron (MLP) algorithm that can be fully embedded into an Enterprise Resource Planning (ERP) platform. The LSTM approach is applied for multi-attribute data processing and it is integrated into an information system based on patient management. To validate the proposed model, we have adopted a typical dataset used in the literature for data mining model testing. The study is focused on the procedure to follow for a correct LSTM data analysis by using artificial records (LSTM-AR-), improving the training dataset stability and test accuracy if compared with traditional MLP and LSTM approaches. The increase of the artificial data is important for all cases where only a few data of the training dataset are available, as for more practical cases. The paper represents a practical application about the LSTM approach into the decision support systems (DSSs) suitable for homecare assistance and for de-hospitalization processes. The paper goal is mainly to provide guidelines for the application of LSTM neural network in type I and II diabetes prediction adopting automatic procedures. A percentage improvement of test set accuracy of 6.5% has been observed by applying the LSTM-AR- approach, comparing results with up-to-date MLP works. The LSTM-AR- neural network can be applied as an alternative approach for all homecare platforms where not enough training sequential dataset is available.

1. Introduction

A research topic in telemedicine is the predictive diagnostic improved by artificial intelligence (AI). Different open source tools [1,2,3,4] such as RapidMiner Studio, Weka, Konstanz Information Miner (KNIME), Orange Canvas, Keras, TensorFlow, and Theano can be applied for this purpose, implementing generic artificial neural networks (ANN) predicting patient health status. These tools are suitable for decision support systems (DSS) based on artificial intelligence algorithms [5,6,7,8,9,10,11,12,13] predicting diagnosis [14,15,16]. Specifically in references [5,6,10,11,12,13] are discussed how data mining could support hospital and assistance processes, while references [7,8,9,14,15,16] provide different healthcare applications where artificial intelligence plays an important role in decision making processes enabled by health status prediction. Accordingly, with homecare assistance facilities for de-hospitalization processes, the use of certified smart sensors transmitting data in a cloud network could remotely control the patients at home [17]. The sensor enabling homecare assistance can be implemented into a more complex information hospital system embedding automatic alerting conditions based on different risk levels [18]. In this direction, KNIME workflows can be easily interfaced as a Graphical User Interface (GUI) to the control room information system, thus allowing the connectivity with big data systems and timing data process by cron job run managing the multilayer perceptron (MLP) ANN analyses [19]. In Figure 1a is illustrated an example information system basic architecture of the MLP ANN network linked with the control room and big data system for homecare assistance [19], and in Figure 1b schematized the related KNIME workflow by distinguishing the data process phases such as time delay for the workflow execution, python node enabling data input access, data pre-processing, data model processing, and reporting [19].

The ANN model implemented by workflows with objects are user friendly but cannot be easily implemented into Enterprise Resource Planning (ERP) software. For this purpose it is preferable to embed ANN scripts directly into the ERP framework, thus facilitating the DSS platform implementation and execution. For this purpose, it is preferable to adopt the python language, which can be easily embedded in different ERP frameworks. In previous literature the Long Short-Term Memory (LSTM) neural network has been adopted for predictive diagnostics, assuring good performance results [20,21,22]. Following this direction, the traditional ANN MLP prediction network, applied in the work [19] using a single attribute labeling, has been substituted by an LSTM neural network based on a multi-attribute analysis. The passage from the workflow implementation to the python script is necessary in order to properly design a neural network embedded into an ERP platform, potentially enabling data processing automatisms. In order to check the performance of the upgraded network has been processedt the experimental dataset of [23,24], representing a good dataset for testing LSTM neural network. The experimental dataset [24] has been adopted in the literature for different data mining testing [24,25,26,27,28,29]. Specifically in reference [25], the K-means algorithm has been applied for predicting diabetes, in reference [26] some authors applied synthetic data in order to balance a machine learning dataset model, while references [27,28,29] have analyzed different machine learning algorithms for diabetes prediction.

Concerning data mining algorithms, some researchers focused their attention on the formulation of decision tree models for Type 2 Diabetes Mellitus (T2DM) [30]. Other studies analyzed the sensitivity of Machine Learning Algorithms about self-monitored blood glucose (SMBG) readings [31], thus enhancing the importance to construct a good learning model. The Deep Learning Approach has also been adopted for the prediction of blood glucose levels [32]. Furthermore, data mining algorithms can be applied for prediction and prevention of complications associated with diabetes [33,34]. According to the World health Organization, the number of people with diabetes has risen from 108 million in 1980 to 422 million in 2014, respectively. For this reason, a good DSS could support diagnostic prediction, thus facilitating diabetes care.

This paper developed an LSTM neural network suitable for DSS platforms, upgrading the architecture of Figure 1 by adding the following specifications:

-

LSTM python script enabling software verticalization and integration in ERP platforms oriented on patient management;

-

Integration of LSTM neural network into the information system collecting patient information and patient data;

-

Creation of different data models allowing data pre-processing and new facilities oriented on patient management;

-

Creation of a prediction model based on the simultaneous analysis of multiple attributes;

-

Adoption of artificial data in order to improve the training dataset;

-

Possibility to choose the best prediction models by reading different model outputs.

2. Materials and Methods

Based on several studies, we found that a commonly used dataset for health data mining was the Pima Indians Diabetes Dataset from the University of California, Irvine (UCI) Machine Learning Database [24,25,26,27,28,29]. The datasets consist of several medical predictor (independent) variables and one target (dependent) variable, Outcome. Independent variables include the number of pregnancies the patient has had, their BMI, insulin level, age, and so on:

• PregnanciesNumber (PN): Pregnant number;
• GlucosePlasma (GP): Glucose concentration (after 2 h of oral glucose tolerance test);
• BloodPressureDiastolic (BPD): Blood pressure (mm Hg);
• SkinThicknessTriceps (STT): Skin fold thickness (mm);
• Insulin2-Hour (I): Serum insulin (mu U/mL);
• BMIBody (BMI): Mass index (weight in kg/(height in m)²);
• DiabetesPedigreeFunctionDiabetes (DPFD): Pedigree function;
• AgeAge (AA): Years old;
• OutcomeClass (OC): Binary variable (0 indicates the no-diabetes status of 268 samples, and 1 indicates the diabetes status of the remaining 500 cases of the training dataset).

In Figure 2 is illustrated the statistic distribution of the above listed attributes plotted by RapidMiner tool.

In general, before processing data by data mining algorithms, it is important to analyze the correlation between attributes in order to choose the less correlated variables: By processing strong correlated variables, which can be introduced into the system redundancies and calculus sensitivities, which can alter the results and increase the data process error or the prediction error. These considerations are valid also for LSTM processing. A method to estimate the correlation between variables generating a weights vector based on these correlations is Pearson’s correlation coefficient evaluation. The algorithm calculates this coefficient, which is the covariance of the two variables divided by the product of their standard deviations [35,36]:

$${\rho }_{X,Y}=\frac{cov(X,Y)}{{\sigma }_X{\sigma }_Y}$$

(1)

being Cov (X,Y), the covariance of the variables X and Y (σ_XY), and σ_X and σ_Y the standard deviation of variable X and Y, respectively.

By observing the calculated correction matrix of

Table 1. Correlation matrix between experimental dataset attributes.

	PN	GP	BPD	STT	I	BMI	DPFD	AA	OC
PN	1	0.13	0.14	−0.08	−0.07	0.02	−0.03	0.54	0.22
GP	0.13	1	0.15	0.06	0.03	0.22	0.14	0.26	0.47
BPD	0.14	0.15	1	0.21	0.09	0.28	0.04	0.24	0.07
STT	−0.08	0.06	0.21	1	0.04	0.39	0.18	−0.11	0.07
I	−0.07	0.33	0.09	0.44	1	0.2	0.19	−0.04	0.13
BMI	0.02	0.22	0.28	0.39	0.02	1	0.14	0.04	0.29
DPFD	−0.03	0.14	0.04	0.18	0.19	0.14	1	0.03	0.17
AA	0.54	0.26	0.24	−0.11	−0.04	0.04	0.03	1	0.24
OC	0.22	0.47	0.07	0.07	0.1	0.29	0.17	0.24	1

(data processing of the experimental dataset) it is clear that all the attributes are not strongly correlated.

A first check of correlation can also be performed by directly observing the 2D plots between a couple of variables. By focusing the attention on the OutcomeClass variable indicating diabetic status, it was evident from Figure 3, Figure 4 and Figure 5 that generally the classes 1 and 0 were not distinguished in the function of the other variables (data overlapping). This confirmed that the results found in the correlation matrix and provided information about samples dispersion.

The prediction results about the outcome variable (labeled variable) were performed by the LSTM neural network. The LSTM basic architecture [37] was composed of a cell, an input gate, an output gate, and a forget gate. Each cell recalled values over arbitrary time intervals, besides the 3 gates regulated the flow of information into and out of the cell. In Figure 6 it has ditched a scheme of the LSTM neural network cell where the input (input activation at the time step t i_t), output (output activation at the time step t o_t), and forget (forget activation at the time step t f_t) gates behaved as neuron computing in a feed-forward or multi-layer neural network: The gates calculated their activations at time step t by considering the activation of the memory cell C at time step t-1. More details about the LSTM neural network models are in the script comments of Appendix A.

The output parameters indicating the LSTM performance are the model accuracy, the model loss and the Receiver Operating Characteristic (ROC) curve indicating the Area under the ROC Curve—AUC—(performance indicator). Loss value defines how well the LSTM neural network model behaves after each iteration of optimization (ideally, one would expect the reduction of loss after each, or several, iterations). The accuracy parameter is defined as:

$$Accuracy=\frac{Numberofcorrectpredictions}{Totalnumberofpredictions}=\frac{TP+TN}{TP+TN+FP+FN}$$

(2)

being TP = True Positives, TN = True Negatives, FP = False Positives, and FN = False Negatives.

The loss function is a binary cross-entropy used for the problems involving yes/no (binary) decisions. For instance, in multi-label problems, where an example can belong simultaneously to multiple classes, the model tries to decide for each class whether the example belongs to that class or not. This performance indicator is estimated as:

$$Loss\left(y,y_p\right)=-\frac{1}{N}{\displaystyle \sum _{i=0}^N\left(y\cdot \log \left(y_p\right)+\left(1-y\right)\cdot \log \left(1-y_p\right)\right)}$$

(3)

where y_p is the predicted value.

As calculation tools have been adopted Keras API and TensorFlow library: Keras is a high-level API suitable for building and training deep learning models (LSTM), and TensorFlow is a free and open-source software library for dataflow and differentiable programming across a range of tasks.

3. Results

In this section are shown the LSTM neural network results by enhancing some aspects of model consistency in function of the training and testing dataset percentage used for the calculation.

Training and Testing Dataset

The training and the testing dataset were randomly extracted from the whole dataset made by 768 records. This allows a decrease in the error calculation of the LSTM network by limiting data redundancy and consecutively data correlation and sensitivity.

Table 2. Example of predicted outcomes (diabetes prediction): OC is the labeled class.

PN	GP	BPD	STT	I	BMI	DPFD	AA	OC (Predicted)
6	148	72	35	0	33.6	0.627	50	1
1	85	66	29	0	26.6	0.351	31	0
8	183	64	0	0	23.3	0.672	32	1
1	89	66	23	94	28.1	0.167	21	0
0	137	40	35	168	43.1	2.288	33	1
5	116	74	0	0	25.6	0.201	30	0
3	78	50	32	88	31.0	0.248	26	1
10	115	0	0	0	35.3	0.134	29	0
2	197	70	45	543	30.5	0.158	53	1
8	125	96	0	0	0	0.232	54	1

illustrates a table extracted from output results indicating the diabetic outcome prediction, where the predicted OC is the output and the other listed variables are the input testing attributes.

In order to estimate, the outcome prediction has been applied to the LSTM neural network by changing the partitioning between experimental and training dataset. Different calculations have been performed by changing the testing dataset percentage. In particular, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11 illustrate the accuracy the losses and the ROC curve of the case of testing dataset percentage of 5%, 10%, 15%, 20%, and 25%, respectively.

The best convergence was observed for the model accuracy of Figure 10, thus confirming that a good balancing between test and train model was achieved (case of testing dataset of 20%). Evident overfitting was observed in the model accuracy of Figure 9 related to 15% of testing dataset (no good balancing of parameters).

From the ROC curves can be calculated the AUC values.

Summarized in

Table 3. LSTM neural network model and decision support systems (DSS) reading automatism: area under the curve (AUC), accuracy, and loss results.

Testing Samples	5%	10%	15%	20%	25%
AUC %	87.7	87	83.9	82	86.7
Accuracy %	75	73	70	75	76
Loss %	100	100	70	55	65

are the results of the AUC, accuracy, and loss of the adopted models, where the green color indicates a better result.

The red and the green colors refer to values above or below thresholds considered valid for LSTM outputs. Specifically, the following thresholds have been considered: 86% for AUC %, 75% for the accuracy, and 60% for loss. The thresholds could be integrated into an automatic procedure able to select the best model to apply.

In Appendix A is listed the python script used for the testing.

The LSTM approach has been improved by implementing a new approach to the training dataset construction based on artificial data creation (LSTM artificial records—AR—). In the case of 20% of the testing dataset characterized by the best compromise between accuracy and loss parameter has created a new training dataset following these criteria:

-

Choose the attributes characterized by a higher correlation if compared with other attributes (in the case of study are insulin correlated with glucose, and skin thickness correlated with BMI);

-

Split the dataset for patients having diabetes or not (first partition);

-

The first partition has been furthermore split by considering the age (second partition);

-

The second partition is then split into a third one representing pregnant women (third partition);

-

Change of the correlated attributes by a low quantity of the values couple glucose and insulin (by increasing insulin is decreased the glucose of the same entity in order to balance the parameter variation), and skin thickness and BMI of the same person belonging to the same partition.

The goal of the proposed criteria is to generate artificial records improving the training dataset stability and test accuracy. The increase in artificial data is important for all cases where only few data of the training dataset are available, as for more practical cases.

In the case of this study, a training dataset has been created of 10,000 records, where only 768 are real.

The cross validation has been performed on MLP traditional methods, and on LSTM using artificial records—AR—(LSTM-AR-). In

Table 4. Cross validation of results.

Method	Test Set Accuracy %
MLP	77.5 [38]
LSTM	75
LSTM-AR-	84

a benchmarking performed by the comparison of the test set accuracy parameter is provided between traditional MLP [38], LSTM traditional algorithm, and the innovative LSTM-AR-approach.

Observing the comparison, it is evident an efficiency increase of the LSTM-AR- of 9% if compared with the LSTM traditional approach, and of 6.5% if compared with the MLP method optimized for diabetes prediction model [38]. Figure 12, Figure 13 and Figure 14 illustrate the accuracy, the loss, and the ROC curve of the LSTM-AR- outputs.

In particular, the model accuracy of Figure 12 proves that a good parameter balancing is achieved in terms of convergence, and no overfitting is observed.

Table 5. LSTM/LSTM-AR models and DSS reading automatism: AUC, accuracy and loss results.

Testing Samples	LSTM (20%)	LSTM-AR (20%)
AUC %	82	89
Accuracy %	75	84
Loss %	55	50

For the other testing dataset (5%, 10%, 15%, 25%) the same increase/decrease percentage has been observed as in the case of Table 5.

reports the comparison between the traditional LSTM approach and LSTM-AR- one, where it is observed that there is an efficiency improvement performed by LSTM-AR-.

4. Discussion

The proposed results allow us to define guidelines to adopt for LSTM data processing in general for data analysis in health applications using a generic dataset. The main steps are summarized as follows:

-

Calculation of correlation matrix (analysis of correlation and weights between variables);

-

Check of 2D variable functions (check of samples dispersion);

-

Calculation of LSTM prediction of diabetic outcomes by changing the partitioning between the testing and the training dataset;

-

Choice procedures of the best LSTM model.

In order to apply correctly the LSTM, one approach is to balance both the indicators loss and accuracy. By observing Table 3, the case of the training dataset of 20% represents a good case of this balancing but allows a relative low AUC value if compared with the other cases. For this purpose, it is important to compare the outputs of the model with the case of good AUC performance related to the cases of testing samples of 5%, 10%, and 25%. This “cross comparison” will facilitate a better understanding of which samples can be classified in false positive or false negative classes. Observing correlation matrix results of Table 1, we note that GlucosePlasma (GP) and OutcomeClass (OC) are correlated by a factor of 0.47, and PregnanciesNumber (PN) and AgeAge (AA) are correlated by a factor 0.57. For this purpose, these attributes could contribute negatively to the model convergence and for AUC values. In other dataset cases, the correlations between attributes can be stronger by adding further complexity to the LSTM output analysis. For this reason, it is important to compare the results of different models in order to find the best reading procedure involving:

-

The extraction of outliers related to wrong measurements and to neglect from the training and testing dataset;

-

The combined analysis of the therapeutic plan of the monitored patient;

-

The analysis of possible failures of the adopted sensors;

-

A dynamical update of the training model by changing anomalous data records;

-

The digital traceability of the assistance pattern in order to choose a patient more suitable to construct the training model;

-

A pre-clustering of patients (data pre-processing performed by combining different attributes such as age, pathology, therapeutic plan, etc.).

We note that in medical and clinical analysis the AUC is considered as a classifier able to discriminate the capacity of a test (see

Table 6. AUC values [39].

AUC	AUC < 0.5 (50%)	AUC = 0.5	0.5 (50%) < AUC ≤ 0.7 (70%)	0.7 (70%) < AUC ≤ 0.9 (90%)	0.9 (90%) < AUC ≤ 1 (100%)	AUC = 1
Classification of the discriminating capacity of a test	No sense test	Non-informative test	Inaccurate test	Moderately accurate test	Highly accurate test	Perfect test

) [39]. All the AUC values found during the test are classified as “moderately accurate test”. In addition, for this reason, it is important to focus the attention on the convergence between Loss and Accuracy parameters.

The sensitivity of the LSTM neural network is then correlated with the specific used model and with the choosen dataset. The possibility to find common data patterns is then important to formulate a correct training dataset.

The goal is to perform a preliminary cross-analysis by considering all the patient information, which are collected into a database system (see Appendix B representing the adopted experimental database). The cross-analysis will contribute to creating the best LSTM training model. A good procedure to follow is:

-

Phase 1: Collecting patient data (by means of a well-structured database system allowing different data mining processing);

-

Phase 2: Pre-clustering and filtering of patient data (construction of a stable training dataset);

-

Phase 3: Pre-analysis of correlations between attributes and analysis of data dispersions;

-

Phase 4: Execution of the LSTM neural network algorithm by processing simultaneously different attributes (multi-attribute data processing);

-

Phase 5: Comparison of results by changing the testing dataset;

-

Phase 6: Choice of the best model to adopt following the analysis of phase 5.

We observe that by repeating the calculation of the random testing datasets, same range values are obtained of the plots of Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15, thus confirming the validity of the result discussion.

The limitations and advantages of the proposed study are summarized in the following

Table 7. Limitations and advantages of the proposed study.

Advantages	Limitations
DSS tool for diabetes prediction ready to use	Accurate training dataset
Multi attribute analysis	Redundancy of data processing (correlated attributes)
Reading procedure of outputs results	Presence of positive false and negative false due to wrong measurements
Choose of the best model according to simultaneous analyses (accuracy, loss, and AUC)	Finding a true compromise of efficiency parameter values
Network having a memory used for the data processing	It is necessary to acquire a correct temporal data sequence
Powerful approach if compared with ANN MLP method	High computational cost

:

Concerning dataset optimization has increased the LSTM performances by adding artificial data into the training dataset by defining the DSS automatism represented by the flow chart of Figure 15: The LSTM neural network model is applied automatically when the training dataset is constructed with enough data, otherwise a new training dataset will be formulated by artificial data (LSTM-AR- model) following the criteria discussed in Section 3. The flowchart of Figure 15 summarizes all the concepts discussed in this paper.

In order to test the LSTM-AR- algorithm on a time series dataset has been considered the sequential dataset of reference [40] (9086 time series data generated by 70 patients). This dataset is suitable for many architectures related to homecare smart assistance platforms.

By observing the results of

Table 8. LSTM, LSTM-AR, and MLP models: AUC, accuracy and loss results by considering the dataset found in reference [40].

Testing Samples	LSTM (20%)	LSTM-AR (20%)	MLP (20%)
AUC %	91	91	94
Accuracy %	86	86	82
Loss %	10	10	14

, it is clear that LSTM and LSTM-AR- approaches are characterized by the same performances. In particular LSTM and the LSTM-AR- exhibit a percentage improvement of Accuracy and Loss of 4% if compared with MLP results.

In this case, the artificial records (454,300 artificial records) have been created by considering the sequential dataset by extracting sub- data set sequences with traditional sliding window approach. The MLP network is optimized for the new performed test (1 hidden layer enabling 30 neurons). Appendix C indicates the adopted MLP network. The adopted LSTM is the recurrent neural network—RNN—described in Appendix A (where sequential datasets will not be considered in structure reshaping).

In this last case, the selected epochs number is 200 because over 200 there was no performance improvement. Figure 16 and Figure 17 illustrate two graphs proving this cross validation method [41]. For all the other cases, the choice of the epochs number followed the same criterion.

The illustrated outputs are the average result of 25 trainings.

As observed in Table 6, the LSTM-AR- approach is characterized by the same performance of the LSTM method by confirming that it is suitable for all homecare platforms where not enough in the training sequential dataset is available.

5. Conclusions

The proposed paper shows how important the data sensitivity analysis in LSTM diabetes is, and predictions also considered patient attributes characterized by low correlations. The high sensitivity is mainly due to the creation of the training and testing dataset. The research is focused on the sensitivity analysis versus the testing dataset partitioning, by means of a stable experimental dataset tested in the literature. Following the performed analysis, a useful guideline to execute correct data processing and analysis by means of the LSTM neural network algorithm, processing different patient attributes, has been formulated. The discussion is mainly focused on the simultaneous analysis and comparison of the LSTM performance indicators such as accuracy, loss, and AUC. The study is completed by presenting the python code used for the calculation and database design of an information system providing more information suitable for the data pre-processing and for data processing. The structured database is integrated into the DSS information system oriented on homecare assistance, providing prediction results and different analysis models, and predicted health risks. The choice to use different test set sizes is dictated by the fact that many datasets are not available with a perfect sequential structure (missing values, not periodical measurements, human measurement errors, records exchanged, etc.), and are characterized by different dimensions. For these reasons, a criterion has been formulated for a generic dataset by changing the testing size where all the proposed results are the average results of 25 trainings. The work also proposes an innovative approach based on the construction of an efficient training artificial dataset based on the weak variation of correlated attributes. The approach, named LSTM-AR-, can be applied to other applications and dataset different from the diabetes prediction following the same logic improved for the proposed DSS automatism. The LSTM-AR- approach can be adopted for all the platforms characterized by a poor training dataset.