Explainable Machine Learning-Based Approach to Identify People at Risk of Diabetes Using Physical Activity Monitoring

Abstract

Objective: This study aimed to investigate the utilization of patterns derived from physical activity monitoring (PAM) for the identification of individuals at risk of type 2 diabetes mellitus (T2DM) through an at-home screening approach employing machine learning techniques. Methods: Data from the 2011–2014 National Health and Nutrition Examination Survey (NHANES) were scrutinized, focusing on the PAM component. The primary objective involved the identification of diabetes, characterized by an HbA1c ≥ 6.5% (48 mmol/mol), while the secondary objective included individuals with prediabetes, defined by an HbA1c ≥ 5.7% (39 mmol/mol). Features derived from PAM, along with age, were utilized as inputs for an XGBoost classification model. SHapley Additive exPlanations (SHAP) was employed to enhance the interpretability of the models. Results: The study included 7532 subjects with both PAM and HbA1c data. The model, which solely included PAM features, had a test dataset ROC-AUC of 0.74 (95% CI = 0.72–0.76). When integrating the PAM features with age, the model’s ROC-AUC increased to 0.79 (95% CI = 0.78–0.80) in the test dataset. When addressing the secondary target of prediabetes, the XGBoost model exhibited a test dataset ROC-AUC of 0.80 [95% CI; 0.79–0.81]. Conclusions: The objective quantification of physical activity through PAM yields valuable information that can be employed in the identification of individuals with undiagnosed diabetes and prediabetes.

1. Introduction

Type 2 diabetes mellitus (T2DM) represents one of the most significant and increasing health problems worldwide, affecting hundreds of millions of individuals [1]. Its long-term complications, such as cardiovascular disease, neuropathy, retinopathy, and nephropathy, place a tremendous burden on both individuals and healthcare systems [2,3,4].

Early diagnosis and intervention are crucial for reducing these complications and improving the overall prognosis of individuals with T2DM [5,6]. While traditional risk assessment and diagnostic methods have been valuable, there is a growing imperative for the development of innovative approaches that enable earlier detection and intervention [7]. In this regard, the application of machine learning and physical activity monitoring represents a promising avenue for identifying individuals at risk of undiagnosed T2DM [8,9,10,11].

Early diagnosis is critical in patients with T2DM due to the nature of the disease in its initial stages. Many individuals remain undiagnosed for several years [12]. The consequence is that by the time T2DM is diagnosed, individuals may already develop complications [13]. The prevention of these complications and the management of T2DM primarily rely on timely diagnosis and the initiation of lifestyle modifications and pharmacological interventions [14].

Machine learning, particularly in the context of healthcare, has emerged as a transformative tool with the potential to revolutionize disease diagnosis and risk assessment [10,11,15]. The integration of machine learning algorithms with vast datasets that include clinical, genetic, and lifestyle information has enabled the development of models that can predict the risk of T2DM and complications more accurately than traditional risk factors alone [16,17,18,19,20,21]. Furthermore, recent advances in wearable technology and the widespread adoption of activity-tracking devices, such as smartphones and watches, have made it possible to monitor an individual’s physical activity patterns continuously [22].

Physical activity (PA), an essential component of lifestyle, has a direct and significant impact on T2DM risk. It has been established that sedentary behavior and inadequate physical activity are closely associated with the development of insulin resistance and subsequent T2DM [23]. The challenge lies in harnessing the data generated by these wearable devices to identify patterns that are indicative of undiagnosed T2DM. The potential advantages of this approach are twofold: first, it can serve as a noninvasive, low-cost, and easily accessible means of identifying individuals at risk, and second, it may enable the detection of T2DM at an earlier stage, thus potentially reducing the long-term complications associated with the disease.

The aim of this study was to explore the notion that patterns in physical activity monitoring, when analyzed with machine learning techniques, could represent a novel approach to identifying individuals at risk of T2DM in a home-based screening approach.

Statement of Significance
Problem	Undiagnosed type 2 diabetes mellitus (T2DM) remains a significant public health issue, often leading to delayed intervention and increased complications.
What is Already Known	Physical activity is closely linked to T2DM risk, and wearable devices enable continuous activity monitoring. Machine learning has shown potential in leveraging such data for early disease detection, but current models primarily rely on clinical or demographic data and often overlook physical activity patterns.
What This Paper Adds	This study introduces a novel approach by utilizing physical activity monitoring (PAM) data, combined with machine learning techniques, to identify individuals at risk of undiagnosed T2DM and prediabetes. By using features derived from PAM data in combination with age, the study demonstrates improved model performance and offers a noninvasive, low-cost method for diabetes risk screening at home.

2. Methods

2.1. Data Sources

The purpose of this study was to explore the possibility of identifying undiagnosed diabetes using physical activity patterns in an at-home based screening approach. We analyzed data from the 2011–2014 National Health and Nutrition Examination Survey (NHANES) [24], which included the physical activity monitoring (PAM) component. The data are available at https://wwwn.cdc.gov/nchs/nhanes/ (accessed on 10 April 2024). The NHANES is a large-scale program administered by the National Center for Health Statistics (NCHS), a part of the Centers for Disease Control and Prevention (CDC), aimed at evaluating the health and nutritional status of the U.S. population. The NHANES stands out for its methodology, which integrates interviews, physical exams, and lab tests to collect comprehensive health data. It consists of two main parts: a household interview and an extensive physical exam conducted in a mobile examination center (MEC). The NHANES dataset was selected for its ability to represent the general population due to its sampling design [24], making it well-suited for evaluating the real-world applicability of this screening method.

This study analyzed NHANES participants aged 18 and older who had both physical activity monitor (PAM) data and HbA1c measurements available from the mobile examination center (MEC). Age, a well-established independent risk factor for type 2 diabetes (T2DM) [25], was included as a predictor due to its relevance and ease of availability in home-based screening settings. To align with the study’s focus on an at-home screening approach, additional predictors were excluded from the analysis. Potential predictors such as blood pressure, BMI, waist circumference, diet, and genetics were omitted because they either require specialized measuring equipment or are subject to both measurement uncertainty and/or daily inter-individual variations.

2.2. Physical Activity Monitoring (PAM)

All NHANES participants aged 2 years and older were instructed to wear a physical activity monitor (PAM) for seven consecutive days. The PAM was to be worn starting from the day of their examination at the NHANES mobile examination center (MEC). Participants were asked to wear the device continuously, from midnight on the first day through to midnight on the seventh day, and to remove it on the morning of the ninth day.

To preprocess the data, the device recorded acceleration data along the xyz axes, which was summarized into Monitor-Independent Movement Summary (MIMS) units in hourly intervals [26]. In this analysis, only individuals with a complete seven-day monitoring period were included. Additionally, only hours with a data quality rate exceeding 80% of valid minutes were considered for analysis. The modeling approach used in this study is illustrated in Figure 1.

2.3. Targets

The primary target was the identification of diabetes, defined as an HbA1c level ≥ 6.5% (48 mmol/mol) [27]. The secondary target also included people with prediabetes, defined as those with an HbA1c level ≥ 5.7% (39 mmol/mol).

2.4. Feature Engineering

To comprehensively analyze the PA data obtained from the PAM, we evaluated a range of potential predictors of diabetes. The selection of potential features was guided by an exploratory approach, aiming to capture various aspects of the PAM signal. These predictors encompassed time-domain [28], statistical [29], and waveform shape [30] features. We computed these features using the PAM data from each individual and assessed a total of 22 features for relevance. Specific thresholds for the features were found empirically using the training data and 5-fold cross-validation. The overview of the features with mathematical definition is presented in

Table 1. The PAM-derived features for input to the models. These predictors encompassed time-domain [28], statistical [29], and waveform shape [30] features.

Feature	Explanation	Definition
Statistical
Mean	Mean value of the PAM signal	μ
Variance	Spread or dispersion of the signal	σ2
Skewness	Asymmetry of the signal distribution	${\displaystyle \frac{\frac{1}{N}{\sum }_{i=1}^N{\left(x_i-\mu \right)}^3}{{\left(\frac{1}{N}{\sum }_{i=1}^N{\left(x_i-\mu \right)}^2\right)}^{3/2}}}$
Kurtosis	Peakedness of the signal distribution	${\displaystyle \frac{\frac{1}{N}{\sum }_{i=1}^N{\left(x_i-\mu \right)}^4}{{\left(\frac{1}{N}{\sum }_{i=1}^N{\left(x_i-\mu \right)}^2\right)}^2}}$
Root Mean Square (RMS)	Square root of the mean of the squared values of the signal	$\sqrt{{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{x}_i^2}$
Time Domain
Zero-Crossing Rate	Number of times the signal crosses a given threshold	${\displaystyle \frac{{\sum }_{i=1}^{N-1}\left(x_i\ge lim\right)\bigwedge \left(x_{i+1}<lim\right)}{N-1}}$
Time Above	Fraction of time spent above a given threshold	$100·{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\left\{\begin{array}{c}1 \\ 0\end{array}\right.}_{otherwise}^{if x_i \ge lim}$
Signal Energy	Sum of squared values of the signal	${\displaystyle \frac{1}{N}}{\sum }_{t=1}^N{x}_i^2$
Autocorrelation	Similarity between the signal and its delayed versions	$R\left(k\right)={\displaystyle \frac{1}{n}}{\sum }_{t=1}^{N-k}\left(x\left(t\right)-\overline{x}\right)\left(x\left(t+k\right)-\overline{x}\right)$
Entropy	Signal’s complexity or uncertainty	$-{\sum }_{i=1}^n{p}_i·log\left(p_i\right)$
Waveform Shape
Peak-to-Peak Amplitude	Range (maximum and minimum values of the signal)	$max\left(x\right)-min\left(x\right)$
Crest Factor	Ratio of the peak value to the RMS value of the signal	${\displaystyle \frac{max\left(x\right)}{rms\left(x\right)}}$
Slope Sign Change	Number of directional changes in the signal	${\sum }_{t=1}^{N-2}{sign\left(diff\left(x\right)\right)}_i\ne {sign\left(diff\left(x\right)\right)}_{i+1}$
Waveform Length	The total of the absolute differences between each pair of consecutive samples in the signal	${\sum }_{t=1}^{N-1}\left|x_{i+1}-x_i\right|$

.

2.5. Model Development

In this analysis, we evaluated a binary outcome scenario involving the classification of individuals as either having diabetes or not having diabetes. We developed two predictive models using the XGBoost classification algorithm [31] and logistic regression (LR). XGBoost is a machine learning technique recognized for its capability to manage datasets with non-linear dependencies. XGBoost functions by integrating multiple simple predictive models, known as decision trees, to form a powerful ensemble model. It performs well, capturing non-linear relationships, managing imbalanced or missing data, and preventing overfitting, resulting in good predictive accuracy. Its effectiveness in clinical prediction has been demonstrated across various medical domains and applications [32,33,34,35]. LR is a statistical method used for binary classification problems, predicting the probability of an event occurring by modeling the relationship between input features and a target variable through a logistic function. The model assumes a linear relationship between the independent variables and the log-odds of the dependent variable. Studies has shown that the predictive capabilities are often comparable to non-linear models in the medical domains [36].

In search of the best model performance, hyperparameter adjustment of the XGBoost model is necessary. However, this process can be challenging due to the multitude of hyperparameters involved. We used a grid search strategy for hyperparameter optimization [37]. To manage the complexity of the parameter space, we selected a subset of hyperparameter combinations (max_depth:1, 2, 3, 4, 5; gamma: 0.1, 0.2, 0.3, 0.4; learning_rate: 0.01, 0.1, 0.2; n_estimators; 50, 100, 200, 300). Model selection was strictly assessed through a five-fold cross validation approach.

All analyses were performed using MATLAB (R2021b), Python (v3), the scikit-learn library (v0.23.2) for machine learning tasks, and the XGBoost package (v1.7.5). The final model was then evaluated on the test dataset. The essential code and details of all versions can be found in the Supplementary Materials.

2.6. Model Assessment

The receiver operating characteristic (ROC) curve plots the relationship between the True-Positive Rate (TPR) and the False-Positive Rate (FPR) at various thresholds. The Area Under the Curve (AUC) quantifies the overall ability of the model to discriminate between the positive and negative classes. It is calculated as the integral of the ROC curve. The model’s performance was evaluated by examining its discriminative ability, measured by the area under the receiver operating characteristic curve (ROC-AUC), using the test dataset [38].

ROC-AUC confidence intervals (CIs) were calculated using 1000 bootstrap replicates. In this approach, the test dataset was resampled with replacements 1000 times to create multiple bootstrapped datasets, each of the same size as the original dataset. For each bootstrap replicate, the ROC-AUC was calculated, resulting in a distribution of 1000 ROC-AUC values.

From this distribution, the 2.5th and 97.5th percentiles were determined to define the 95% confidence interval for the ROC-AUC. This method provides a robust, non-parametric estimate of uncertainty around the ROC-AUC metric, accommodating potential variability in the sample data.

To visualize the distribution of predicted probabilities for both classes, a normalized frequency plot was generated. To assess the clinical relevance of the model, the relative risk (RR) was calculated across different decision thresholds, ranging from 0% to 100% of patients, with a particular focus on those identified as at-risk (positive predictions). A given proportion, p, of the population was selected based on predicted risk. The predicted group includes the top p % of the population based on risk scores. The non-predicted group includes the remaining (1 − p) % of the population.

$${Risk}_{pred}={\displaystyle \frac{Number of events in top p\%}{Total number of individuals in top p\%}}$$

$${Risk}_{non-pred}={\displaystyle \frac{Number of events in bottom (1-p)\%}{Total number of individuals in bottom (1-p)\%}}$$

$$RR={\displaystyle \frac{{Risk}_{pred}}{{Risk}_{non-pred}}}$$

Since the cutoff between diabetes and non-diabetes is not strictly deterministic and may include individuals with prediabetes, and to gain deeper insight into the model’s performance and its clinical implications, we analyzed HbA1c levels among both false-positive and true-negative predictions. If physical activity data, such as that collected from smartwatches or smartphones, were used for preliminary diabetes screening, subsequent confirmation would be necessary at a clinical facility. Notably, false-positive individuals with elevated blood glucose levels may be at increased risk of developing diabetes and related comorbidities, making them important candidates for targeted preventive interventions.

2.7. Analyzing Feature Importance and Model Interpretability

Machine learning models like XGBoost pose challenges in terms of interpretability, as their predictions can be difficult to understand. To address this, SHapley Additive exPlanations (SHAP) values [39] offer a valuable method for elucidating the model’s predictions. SHAP values provide a robust framework for explaining the outputs of machine learning models, and are based on Shapley values from cooperative game theory. These values assign an importance score to each feature in a prediction, revealing how much each feature contributes to the final outcome. The SHAP framework was used to evaluate the significance of features in the proposed model, and a box plot was generated to further explore the contribution of individual features.

3. Results

We included 7532 (of 19,931) patients with available PAM and HbA1c data from the NHANES 2011–2014. The median age was 39 years [interquartile range (IQR), 22–57]; the BMI was 26.6 kg/m² [interquartile range (IQR), 22.8–31.3]; the WC was 93.2 cm [interquartile range (IQR), 81.7–105]; and 51.9% were women. A total of 10.4% had diabetes, defined by the HbA1c criteria. The box plot for all included potential predictors is shown in Figure 2.

3.1. Model Performance

The included subjects were divided into a dataset for training (60%) and a dataset for testing (40%), and the model performance was reported based on the independent test dataset. Including only PAM features, the XGBoost model had an ROC-AUC of 0.74 [95% CI; 0.72–0.76] in the test dataset. Including the PAM features with age, the XGBoost model had an ROC-AUC of 0.79 [95% CI; 0.78–0.80] in the test dataset. The hyperparameter tuning using the training set and cross-validation provided the final hyperparameters (max_depth: 2; gamma: 0.1; learning_rate: 0.1; n_estimators: 50). Including only PAM features, the LR model had an ROC-AUC of 0.72 [95% CI; 0.69–0.74] in the test dataset. Including the PAM features with age, the LR model had an ROC-AUC of 0.77 [95% CI; 0.75–0.78] in the test dataset.

Figure 3 presents the normalized predicted probabilities for diabetes versus no diabetes (upper left). The distributions reveal a discernible difference between the two groups. The analysis (lower right) further demonstrates that false-positive patients exhibited higher HbA1c levels compared to true-negative patients, regardless of the selected decision boundary or the percentage of patients identified as at risk. This points toward false-positive cases that could also be a target of clinical interest. The RRs associated with the patients at risk are illustrated in Figure 3 (lower left). This analysis shows that the RRs of having undiagnosed diabetes in the model-predicted group is substantially increased for up to 20% of the samples’ predicted values.

The results for the secondary targets, including prediabetes, are presented in the Supplementary Material Figures S1 and S2.

Including only PAM features, the XGBoost model had an ROC-AUC of 0.71 [95% CI; 0.69–0.72] in the test dataset. Including the PAM features with age, the XGBoost model had an ROC-AUC of 0.80 [95% CI; 0.79–0.81] in the test dataset. Including only PAM features, the LR model had an ROC-AUC of 0.69 [95% CI; 0.68–0.71] in the test dataset. Including the PAM features with age, the LR model had an ROC-AUC of 0.79 [95% CI; 0.78–0.80] in the test dataset.

3.2. Feature Importance

SHAP feature importance analysis revealed that age was the single feature with the highest importance for identifying patients with and without diabetes. Moreover, several PAM features also had a significant impact on the model’s discriminative performance (Figure 4). These findings highlight that the combination with PA patterns provides information on the risk of diabetes. The mean PA measured in the MIMS (Figure 2 and Figure 4) was not a good discriminator in itself; however, PAM features related to PA dynamics and high-intensity activity were shown to contain better discriminative information. Entropy was shown to be the best PAM feature. This analysis suggests that general measures of PA alone may not be sufficient to reliably distinguish between individuals with and without undiagnosed diabetes. Incorporating multiple aspects of PA, such as entropy, variance, mean levels, and peak-to-peak amplitude, could enhance the model’s discriminative ability and improve its effectiveness in identifying individuals at risk.

4. Discussion

Using a cohort representative of the U.S. population, we developed and internally validated machine learning models to predict the risk of dysglycemia. These models leverage objectively measured physical activity data and readily available age information, with an emphasis on investigating a potential at-home screening. The final best model using PAM data was shown to have acceptable discriminative ability (ROC-AUC, 0.74), and the model utilizing both the PAM and age was shown to have good discriminative ability (ROC-AUC, 0.79). The main driver for the prediction models was, not surprisingly, age; however, several PAM features also had a substantial impact on the prediction. The mean physical activity measured via the MIMS did not have good discriminative ability; this finding indicates that the dynamics of PA combined have better discriminative ability between people with and without diabetes.

In comparison between the XGBoost and LR models, the non-linear approach using XGBoost might have a slightly better performance than the linear LR approach. However, further studies are needed to validate these findings.

Our data further demonstrated that, in addition to the binary discriminative capability of the models, false positives within the at-risk groups exhibited higher HbA1c levels than true negatives. This highlights the model’s ability to identify not only individuals with clinically defined diabetes but also those with prediabetes who are at risk of developing diabetes.

4.1. Comparison to Related Work

Eskelund and colleagues [40] demonstrated that insufficient physical activity is linked to insulin resistance and the development of type 2 diabetes mellitus. Notably, the amount of time spent in sedentary or light-intensity activities was not significantly associated with insulin resistance, whereas engaging in physical activities of moderate and vigorous intensity was correlated with insulin resistance [40]. This finding is in line with our findings, indicating that the mean objectively measured PA is not a good discriminator. A review [41] on PA in T2DM patients reported that, irrespective of the method of measurement, the data reported, or the study’s geographical location, adults with T2DM exhibit low levels of physical activity and a high degree of sedentary behavior. Individuals with T2DM are notably less active and more sedentary than individuals without T2DM. Lam et al. [42] investigated the hypothesis that accelerometer trace measures of physical activity can be used as a predictor of T2DM. Lam and colleagues [42] showed that PA patterns measured by an accelerometer improved the model’s performance compared to a model that included sociodemographic, lifestyle, and anthropometric predictors. These findings are in line with the findings of the present research.

4.2. Strengths and Limitations

The NHANES comprises a representative sample of the United States population, which can enhance the applicability of these findings to real-world screening scenarios. However, despite the dataset available from the NHANES 2011–2014, in which a cohort of individuals with T2DM can be identified, discerning differences between those with T2DM and the control group remains a challenge owing to the relatively low prevalence of this disease in the population. This is evident in our study, where the dataset available for training machine learning models was relatively small.

The study defined diabetes using the HbA1c criterion and the absence of a previously reported diabetes diagnosis. Relying solely on a single HbA1c measurement may not be the optimal approach for diagnosing diabetes, as variations can arise from various factors. Our study’s results offer intriguing insights into the potential use of physical activity monitoring (PAM) in diabetes screening or prescreening. Nevertheless, in future studies it is essential to validate these findings in individuals beyond the NHANES 2011–2014 sample, including various regions, cohorts, and other cultures, to ensure their broader applicability. Future studies should incorporate a standardized definition of diabetes, such as requiring two abnormal glucose test results, including HbA1c, to reduce the likelihood of false-positive diagnoses. Additionally, the analysis relies on data from wearable activity monitors, which may be influenced by user compliance challenges and potential measurement inaccuracies, particularly for certain types of physical activities [43].

5. Conclusions

We developed and internally validated machine learning models for predicting undiagnosed diabetes. The use of objective physical activity measurements offers valuable insights that can aid in identifying individuals with dysglycemia, including diabetes and prediabetes, using an in-home screening approach. However, further research is necessary to validate these findings across diverse populations, encompassing different regions, cohorts, and cultural contexts, to ensure their generalizability and broader applicability.