# Prediction of Blood Risk Score in Diabetes Using Deep Neural Networks

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

## 3. Results

## 4. Discussion

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Appendix A

**Figure A1.**(

**a**) Imputed data percentage per individual. (

**b**) Mean number of consecutive imputed values per individual. The error bars correspond to the standard deviation. (

**c**) Number of measurements per individual. In all plots, the x-axis is the ranked imputed data percentage from smallest to largest.

## References

- Saeedi, P.; Petersohn, I.; Salpea, P.; Malanda, B.; Karuranga, S.; Unwin, N.; Colagiuri, S.; Guariguata, L.; Motala, A.A.; Ogurtsova, K.; et al. Global and regional diabetes prevalence estimates for 2019 and projections for 2030 and 2045: Results from the International Diabetes Federation Diabetes Atlas, 9th edition. Diabetes Res. Clin. Pract.
**2019**, 157, 107843. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Brownlee, M. Biochemistry and molecular cell biology of diabetic complications. Nature
**2001**, 414, 813–820. [Google Scholar] [CrossRef] [PubMed] - Cobelli, C.; Renard, E.; Kovatchev, B. Artificial pancreas: Past, present, future. Diabetes
**2011**, 60, 2672–2682. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Cobelli, C.; Dalla Man, C.; Sparacino, G.; Magni, L.; De Nicolao, G.; Kovatchev, B.P. Diabetes: Models, signals, and control. IEEE Rev. Biomed. Eng.
**2009**, 2, 54–96. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Magni, L.; Raimondo, D.M.; Dalla Man, C.; De Nicolao, G.; Kovatchev, B.; Cobelli, C. Model predictive control of glucose concentration in type I diabetic patients: An in silico trial. Biomed. Signal Process. Control
**2009**, 4, 338–346. [Google Scholar] [CrossRef] - Hovorka, R.; Canonico, V.; Chassin, L.J.; Haueter, U.; Massi-Benedetti, M.; Federici, M.O.; Pieber, T.R.; Schaller, H.C.; Schaupp, L.; Vering, T.; et al. Nonlinear model predictive control of glucose concentration in subjects with type 1 diabetes. Physiol. Meas.
**2004**, 25, 905. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Mirshekarian, S.; Bunescu, R.; Marling, C.; Schwartz, F. Using LSTMs to learn physiological models of blood glucose behavior. In Proceedings of the 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jeju, Republic of Korea, 11–15 July 2017; pp. 2887–2891. [Google Scholar]
- Li, K.; Daniels, J.; Liu, C.; Herrero, P.; Georgiou, P. Convolutional recurrent neural networks for glucose prediction. IEEE J. Biomed. Health Inform.
**2019**, 24, 603–613. [Google Scholar] [CrossRef] [Green Version] - Martinsson, J.; Schliep, A.; Eliasson, B.; Mogren, O. Blood glucose prediction with variance estimation using recurrent neural networks. J. Healthc. Inform. Res.
**2020**, 4, 1–18. [Google Scholar] [CrossRef] [Green Version] - Deng, Y.; Lu, L.; Aponte, L.; Angelidi, A.M.; Novak, V.; Karniadakis, G.E.; Mantzoros, C.S. Deep transfer learning and data augmentation improve glucose levels prediction in type 2 diabetes patients. NPJ Digit. Med.
**2021**, 4, 109. [Google Scholar] [CrossRef] - De Bois, M.; El Yacoubi, M.A.; Ammi, M. Adversarial multi-source transfer learning in healthcare: Application to glucose prediction for diabetic people. Comput. Methods Programs Biomed.
**2021**, 199, 105874. [Google Scholar] [CrossRef] - Yang, T.; Yu, X.; Ma, N.; Wu, R.; Li, H. An autonomous channel deep learning framework for blood glucose prediction. Appl. Soft Comput.
**2022**, 120, 108636. [Google Scholar] [CrossRef] - Felizardo, V.; Garcia, N.M.; Pombo, N.; Megdiche, I. Data-based algorithms and models using diabetics real data for blood glucose and hypoglycaemia prediction–a systematic literature review. Artif. Intell. Med.
**2021**, 118, 102120. [Google Scholar] [CrossRef] - The Blood Glucose Level Prediction Challenge. Available online: http://smarthealth.cs.ohio.edu/bglp/bglp-rules.html (accessed on 19 January 2023).
- Rahaghi, F.N.; Gough, D.A. Blood glucose dynamics. Diabetes Technol. Ther.
**2008**, 10, 81–94. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Clements, M.A.; Schwandt, A.; Donaghue, K.C.; Miller, K.; Lück, U.; Couper, J.J.; Foster, N.; Schröder, C.; Phelan, H.; Maahs, D.; et al. Five heterogeneous HbA1c trajectories from childhood to adulthood in youth with type 1 diabetes from three different continents: A group-based modeling approach. Pediatr. Diabetes
**2019**, 20, 920–931. [Google Scholar] [CrossRef] [PubMed] - Kovatchev, B.P.; Cox, D.J.; Gonder-Frederick, L.A.; Clarke, W. Symmetrization of the blood glucose measurement scale and its applications. Diabetes Care
**1997**, 20, 1655–1658. [Google Scholar] [CrossRef] - Shahid, A.; Lewis, D.M. Large-Scale Data Analysis for Glucose Variability Outcomes with Open-Source Automated Insulin Delivery Systems. Nutrients
**2022**, 14, 1906. [Google Scholar] [CrossRef] - OpenAPS. About the OpenAPS Data Commons on Open Humans. 2022. Available online: https://openaps.org/outcomes/data-commons/ (accessed on 13 February 2023).
- Lewis, D. Research Guidelines for Working with the OpenAPS Data Commons in Open Humans. 2022. Available online: https://docs.google.com/document/d/1_ukUNdnVHZm5piu_RI6OBG2PL_9Rd1xcT5Z1sFsrTeA/edit (accessed on 13 February 2023).
- Koutny, T. Glucose-level interpolation for determining glucose distribution delay. In Proceedings of the XIII Mediterranean Conference on Medical and Biological Engineering and Computing 2013: MEDICON 2013, Seville, Spain, 25–28 September 2013; Springer: Berlin/Heidelberg, Germany, 2014; pp. 1229–1232. [Google Scholar]
- Koutny, T.; Ubl, M. Parallel software architecture for the next generation of glucose monitoring. Procedia Comput. Sci.
**2018**, 141, 279–286. [Google Scholar] [CrossRef] - Koutny, T.; Ubl, M. Smartcgms as a testbed for a blood-glucose level prediction and/or control challenge with (an fda-accepted) diabetic patient simulation. Procedia Comput. Sci.
**2020**, 177, 354–362. [Google Scholar] [CrossRef] - Clarke, W.; Kovatchev, B. Statistical tools to analyze continuous glucose monitor data. Diabetes Technol. Ther.
**2009**, 11, S-45–S-54. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Mayo, M.; Koutny, T. Neural multi-class classification approach to blood glucose level forecasting with prediction uncertainty visualisation. In Proceedings of the 5th International Workshop on Knowledge Discovery in Healthcare Data (KDH 2020), Santiago de Compostela, Spain, 29–30 August 2020; Volume 2675, pp. 80–84. [Google Scholar]
- Innes, M. Flux: Elegant machine learning with Julia. J. Open Source Softw.
**2018**, 3, 602. [Google Scholar] [CrossRef] [Green Version] - Toledo-Marin, J.Q. Diabetes Risk Score Prediction Using Deep Learning Repository. 2022. Available online: https://github.com/jquetzalcoatl/DiabetesRiskPrediction (accessed on 13 February 2023).
- Clarke, W.L. The original Clarke error grid analysis (EGA). Diabetes Technol. Ther.
**2005**, 7, 776–779. [Google Scholar] [CrossRef] [PubMed] - Fleischer, J.P.; von Laszewski, G.; Theran, C.; Parra Bautista, Y.J. Time Series Analysis of Cryptocurrency Prices Using Long Short-Term Memory. Algorithms
**2022**, 15, 230. [Google Scholar] [CrossRef] - Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv
**2022**, arXiv:2204.06125. [Google Scholar] - Laubenbacher, R.; Niarakis, A.; Helikar, T.; An, G.; Shapiro, B.; Malik-Sheriff, R.; Sego, T.; Knapp, A.; Macklin, P.; Glazier, J. Building digital twins of the human immune system: Toward a roadmap. NPJ Digit. Med.
**2022**, 5, 64. [Google Scholar] [CrossRef] [PubMed]

**Figure 1.**CGM normalized histograms before and after data imputation. Each panel corresponds to a different individual in the OpenAPS Data Commons. Each panel shows the CGM data before data imputation in blue, and after data imputation in red. The overlap between the two histograms in each panel suggests that the linear interpolation (proposed in Equation (2)) has little to no effect in changing the data structure. Notice the skewness in the histograms typical of CGM data. Panels (

**a**), (

**b**) and (

**c**) correspond to 154,378, 85,346 and 46,611 glucose measurements, respectively.

**Figure 3.**BG-risk standardized variable, $\xi $, RMSE for (

**a**) 15, (

**b**) 30, and (

**c**) 60 min prediction horizon. Each panels shows the RMSE for different architectures specified in the x-axis and for the last-measurement prediction (LM). Each box corresponds to the model tested on $N-10$ individuals.

**Figure 4.**CGM prediction root mean squared error for (

**a**) 15, (

**b**) 30, and (

**c**) 60 min horizons, and for different architectures. Each panels shows the RMSE for different architectures, specified in the x-axis and for the last-measurement prediction (LM). Each box corresponds to the model tested on $N-10$ individuals.

**Figure 5.**CGM prediction root mean squared error weighed with the normalized BG risk score for (

**a**) 15, (

**b**) 30, and (

**c**) 60 min horizons, and for different architectures. Each panels shows the RMSE for different architectures, specified in the x-axis and for the last-measurement prediction (LM). Each box corresponds to the model tested on $N-10$ individuals.

**Figure 6.**Clarke error grid for CGM prediction using the CNN10 and a sample in the test set for (

**a**) 15, (

**b**) 30, and (

**c**) 60 min horizons. Panel (

**d**) shows the ratio of the number of points between CNN10 and LM that fall in each zone in the Clarke error grid for the three different prediction horizons. For each zone in the x-axis, values above (below) 1 imply a larger (smaller) number of points from the CNN10 than the LM. In the case of the CNN10, the fraction of points in each zone are: (

**a**) ${\rho}_{A}^{\left(15\right)}=0.908$, ${\rho}_{B}^{\left(15\right)}=0.068$, ${\rho}_{C}^{\left(15\right)}=0.003$, ${\rho}_{D}^{\left(15\right)}=0.021$, ${\rho}_{E}^{\left(15\right)}=0$; (

**b**) ${\rho}_{A}^{\left(30\right)}=0.819$, ${\rho}_{B}^{\left(30\right)}=0.160$, ${\rho}_{C}^{\left(30\right)}=0.004$, ${\rho}_{D}^{\left(30\right)}=0.016$, ${\rho}_{E}^{\left(30\right)}=0.001$; and (

**c**) ${\rho}_{A}^{\left(60\right)}=0.566$, ${\rho}_{B}^{\left(60\right)}=0.345$, ${\rho}_{C}^{\left(60\right)}=0.003$, ${\rho}_{D}^{\left(60\right)}=0.085$, ${\rho}_{E}^{\left(60\right)}=0.001$.

**Figure 7.**CGM prediction root mean squared error reduced by the LM prediction for (

**a**) 15, (

**b**) 30, and (

**c**) 60 min horizons, and for different architectures. Each panels shows the RMSE for different architectures specified in the x-axis and for the last-measurement prediction (LM). The RMSE of the given model is divided by the RMSE of the LM. Models below (above) 1 perform better (worse) than the LM approach.

**Table 1.**Model prefix and the corresponding description. Several replicas per model per prediction horizon (PH) were trained. Models were named as Model prefix_number. The model prefix defines the architecture, the training set, and the training scale used.

Model Prefix | Replicas | PH | Description |
---|---|---|---|

RNN0 | 5 | 15/30/60 | One individual’s data in glucose scale |

RNN | 5 | 15/30/60 | One individual’s data in risk-score scale |

GRU | 5 | 15/30/60 | One individual’s data in risk-score scale |

LSTM | 5 | 30/60 | One individual’s data in risk-score scale |

CNN | 5 | 15/30/60 | One individual’s data in risk-score scale |

CNN10 | 5 | 15/30/60 | Ten individuals’ data in risk-score scale |

**Table 2.**CNN architecture. The input consisted of the 16 previous measurements arranged in a 4 × 4 lattice with one channel. Regularizers were not required due to the kernel sizes and the NN size overall.

Type of Layer | Kernel | Padding | Stride | Output Channels | Activation Function |
---|---|---|---|---|---|

Convolution | 2 × 2 | 0 | 1 | 4 | $tanh$ |

Convolution | 2 × 2 | 0 | 1 | 8 | $tanh$ |

Convolution | 2 × 2 | 0 | 1 | 16 | $tanh$ |

Fully Connected | - | - | - | 1 | $tanh$ |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Toledo-Marín, J.Q.; Ali, T.; van Rooij, T.; Görges, M.; Wasserman, W.W.
Prediction of Blood Risk Score in Diabetes Using Deep Neural Networks. *J. Clin. Med.* **2023**, *12*, 1695.
https://doi.org/10.3390/jcm12041695

**AMA Style**

Toledo-Marín JQ, Ali T, van Rooij T, Görges M, Wasserman WW.
Prediction of Blood Risk Score in Diabetes Using Deep Neural Networks. *Journal of Clinical Medicine*. 2023; 12(4):1695.
https://doi.org/10.3390/jcm12041695

**Chicago/Turabian Style**

Toledo-Marín, J. Quetzalcóatl, Taqdir Ali, Tibor van Rooij, Matthias Görges, and Wyeth W. Wasserman.
2023. "Prediction of Blood Risk Score in Diabetes Using Deep Neural Networks" *Journal of Clinical Medicine* 12, no. 4: 1695.
https://doi.org/10.3390/jcm12041695