Performance Analysis and Assessment of Type 2 Diabetes Screening Scores in Patients with Non-Alcoholic Fatty Liver Disease

Abstract

Type 2 diabetes (T2D) and non-alcoholic fatty liver disease (NAFLD) are worldwide chronic diseases that have strong relationships with one another and commonly exist together. Type 2 diabetes is considered one of the risk factors for NAFLD, so its occurrence in people with NAFLD is highly likely. As the high and increasing number of T2D and NAFLD, which potentially followed by existing together number, an analysis and assessment of T2D screening scores in people with NAFLD is necessary to be done. To prevent this potential case, an effective early prediction model is also required to be developed, which could help the patients avoid the dangers of both existing diseases. Therefore, in this study, analysis and assessment of T2D screening scores in people with NAFLD and the early prediction model utilizing a forward logistic regression-based feature selection method and multi-layer perceptrons are proposed. Our analysis and assessment results showed that the prevalence of T2D among patients with NAFLD was 8.13% (for prediabetes) and 37.19% (for diabetes) in two population-based NAFLD datasets. The variables related to clinical tests, such as alanine aminotransferase (ALT), aspartate aminotransferase (AST), alkaline phosphatase (ALP), gamma-glutamyl transferase (GGT), and systolic blood pressure (SBP), were found to be statistically significant predictors (p-values < 0.001) that indicate a strong association with T2D among patients with NAFLD in both the prediabetes and diabetes NAFLD datasets. Finally, our proposed model showed the best performance in terms of all performance evaluation metrics compared to existing various machine learning models and also the models using variables recommended by WHO/CDC/ADA, with achieved accuracy as much as 92.11% and 83.05% and its improvement scores after feature selection of 1.35% and 5.35%, for the first and second dataset, respectively.

1. Introduction

Type 2 diabetes (T2D) has emerged as one of the chronic diseases to which people are most vulnerable. People with this disease are becoming increasingly susceptible to infections and complicated illnesses as a result of aspects of unhealthy lifestyles, especially poor eating habits and a lack of exercise [1,2]. When it happens, the blood glucose becomes improperly managed, which causes a number of difficulties, such as blurred vision, hearing difficulty, oral health problems, liver problems, etc. Non-alcoholic fatty liver disease (NAFLD) is highly likely to occur in people with T2D, affecting approximately 24–26% of people in the US and leading to a high percentage of them developing T2D or impaired glucose tolerance [3]. When they drink little to no alcohol, this effect happens and could lead to fat accumulation, which leads to liver damage and inflammation [4]. Type 2 diabetes is one of the risk factors for NAFLD, but it may also be the other way around. Non-alcoholic fatty liver disease may potentially be a factor in T2D (prediabetes and diabetes), as the liver is crucial in controlling blood sugar levels in the body, and fasting glucose levels are more difficult to regulate as a result of the critical organ’s accumulation of fat. Additionally, it increases the body’s resistance to insulin, putting the pancreas and its beta cells under stress and hastening the onset of T2D [5]. Therefore, T2D and NAFLD are correlated with one another, and they commonly exist together [6]. When this happens, it has the potential to cause chronic conditions (e.g., hepatic and extrahepatic), complications, such as the development of cardiovascular disease (CVD), chronic kidney disease (CKD), and stroke, which could result in a higher risk of mortality [7].

As reported by the International Diabetes Federation, the prevalence of T2D in 2021 was 536.5 million adult people and was going to rise to 783.2 million in 2045 around the world [8,9]. This was followed by the prevalence of NAFLD, which is estimated at around 24% of the total global population [10]. Given the significant global incidence of T2D and NAFLD and its hazards, an appropriate solution is needed to prevent the development of T2D in patients with NAFLD.

Analysis and assessment of disease screening scores are one of the alternatives that can be utilized to early detect T2D [11]. Analysis and assessment of disease screening scores is an important stage in order to identify the issues related to the patient, summarize the main predictors of the disease [12], as well as correctly detect problems to enable appropriate care for the patient [13]. Kianpour et al. [14] evaluated the performance of diabetes screening scores in the Iranian population. The evaluation is proposed to find out the association between the risk factors and T2D, the prevalence of T2D based on the results of capillary blood glucose (CBG), venus plasma glucose (VPG), and glycated hemoglobin (HbA1c) tests, and measure the effectiveness of those tests by measuring their sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the curve (AUC). By conducting screening score analysis, they found that the risk factors have a strong relationship with T2D in the Iranian population, and using CBG, VPG, and HbA1c, the prevalence of T2D was higher compared to WHO or ADA recommendation tests, while the performance of those tests was better than that of WHO or ADA recommendation tests and also previous studies. Meng et al. in 2021 [12] assessed the performance of diabetes and kidney disease screening scores in the United States and Korean populations. Statistical analyses are conducted to summarize the nationally representative sample, find out the characteristics and general status among the subject’s populations, measure the association between predictors and the diseases, and assess the performance of the diagnostic tests using various cut points based on cut-point determination methods. The screening score assessment showed that 30% to 60% of participants were categorized as being at high risk, and some of the measured predictors were found to be statistically significant and associated with the diseases. They could reveal that age equal and older than 50 years old has a very high odds ratio (OR), where the incidence of kidney disease is 5 to 73 times more likely to be exposed to people that age than those without kidney disease. The results of the test performance also showed better when using Youden Index-based cut points compared to the cut points recommended by WHO/ADA. Using the same population as [12], Lee et al. [14] used the screening score for diabetes. The assessment is conducted to reveal the association between the risk factors and diabetes, measure the prevalence of high-risk groups, and evaluate the performance of the tests. Using cut points of 5 for the screening score range, the results showed that 47% of adults were classified as being at high risk for diabetes. The performance of the test also performed better compared to the WHO/ADA recommendation cut point, achieving a sensitivity of 81%, a specificity of 54%, a PPV of 6%, and an LR+ of 1.8 with an AUC of 0.73. In 2020, Mao et al. [15] measured the efficacy of the new Chinese diabetes risk score (NCDRS) in screening for undiagnosed T2D and prediabetes using a community-based cross-sectional study in eastern China. The researchers measured the characteristic scores of the participant and the performance of the proposed new Chinese diabetes risk score, which ranged from 25 to 32. Using a cut point of 25 for the NCDRS, the prevalence of high risk increased, as did the performance of the test compared with optimal Youden Index cut point utilization, with sensitivity and specificity of 84.8% and 50.1%, respectively. Finally, in 2022, Fitriyani et al. [16] analyzed the diabetes scores using nationally representative demographic data, consisting of Chinese, Japanese, Korean, Trinidadians, and PIMA Indians. The objectives of the analysis are to find out the relationship between the risk factors and diabetes, determine the optimal cut points for classification, and determine the best model for prediction. As the results show, by using diabetes assessment scores, all the objectives are achieved and concluded by various findings across all nationally representative populations.

Aside from the analysis and assessment of disease screening scores using statistical analysis, another method that is widely used for assessing the screening scores is machine learning-based analysis which has been widely and successfully used in medical informatics [17,18,19,20,21,22,23,24]. Several recent studies have analyzed disease screening and diagnosis, especially for diabetes, utilizing machine learning methods, while most of them also proposed prediction models utilizing machine learning methods [25,26,27]. One of the well-known and commonly used machine learning models for disease prediction is a multilayer perceptron (MLP). Al Bataineh and Manacek [28] proposed a heart disease prediction model using MLP combined with particle-based optimization (PSO). The proposed model is applied to the Cleveland heart disease dataset. The performance of the model showed that the MLP + PSO model outperforms all the existing models and previous studies, with the accuracy of the model being 84.60% and an AUC is 0.848. In 2022, Pal et al. [29] used MLP for predicting cardiovascular disease. The MLP algorithm was applied after feature selection in the CVD dataset. It achieved the best accuracy and AUC compared to other machine learning algorithms, with accuracy and AUC of 82.47% and 0.854, respectively. Bikku [30] proposed a health risk prediction model that utilized MLP. The proposed research focuses on supervised learning techniques and how well they may uncover hidden patterns in actual historical medical data. The goal is to anticipate future risks using the multi-layer perceptron approach with a specified degree of probability. The results of comparing the suggested approach to conventional classification methods demonstrate that the proposed method is superior to Long Short-Term Memory (LSTM) and Recurrent Neural Network (RNN), with an average accuracy of more than 95%. Sivaranjani and Yuvaraj [31] developed a model using MLP for early predicting cardiac functionalities. The proposed prediction model could achieve an accuracy as high as 97.67%. Aside from utilizing a proper machine learning model for classification and prediction, previous studies also considered utilizing a method for selecting the most important features, thus improving the performance of the prediction model. In medical research, Zhang [32] analyzes the performance of the feature selection methods, such as stepwise and best subset regression. The stepwise methods used in the analysis consisted of forward and backward methods. The results showed that the forward method effectively selected the feature with the smallest number of features. Sanchez-Pinto et al. [33] evaluated the performance of the various feature selection models when applied to any machine learning models for predicting clinical deterioration and acute kidney injury in children. Among the eight (8) feature selections used in the experiment, stepwise logistic regression-based feature selection, which selects the features with a p-value < 0.05, raised the AUC score of the model from 0.78 to 0.837. Soroush et al. developed a hybrid customer prediction system utilizing a forward stepwise logistic regression model in order to predict mobile home policy purchasers [34]. 47 out of 86 features were selected and then used for prediction. Utilizing logistic regression, the model could predict 58.3% of total purchasers and 17.4% of predicted customers. The proposed model performs better compared to other existing models. We found that several previous studies have used forward logistic regression as an effective feature selection method, which could improve the performance of the prediction model. According to this literature review results, MLP combined with a forward logistic regression-based feature selection method has not been applied by previous studies, especially for predicting diabetes. MLP was found to be a good and widely used model in the biomedical field for disease prediction due to its ability in solving complex computational problems from large sets of data with multiple variables. The capability of MLP fits the medical data and thus could be used for risk analysis and accurate classification.

Therefore, in this study, we proposed a T2D prediction model utilizing an MLP-based machine learning model combined with a forward logistic regression-based feature selection method and also aimed to analyze and assess the T2D screening scores among patients with NAFLD.

2. Materials and Methods

2.1. Data Sources and Study Population

In this study, we utilized and analyzed two clinical publicly available datasets which contain all subjects with positive NAFLD in mild and severe types, called NAFL and NASH. The analyses are conducted by measuring the T2D scores, especially in prediabetes and diabetes stages. The detailed data sources and study populations are as follows:

• The first dataset is NAFLD in the Gifu area longitudinal analysis dataset (Gifu NAFLD). The Gifu NAFLD dataset is an investigation of medical examination program data from Murakami Memorial Hospital in Gifu, Japan, using a population-based longitudinal approach [35]. The participants consist of 15,464 in all, ranging in age from 22 to 70. In this dataset, none of the subjects have fasting blood glucose (FBG) greater than 125 mg/dL or glycated hemoglobin (HbA1c) greater than 6.5%, which is categorized as diabetes. However, the T2D cases are in the range of 100 to 125 mg/dL for FBG and 5.7 to 6.4% for HbA1c, which means that the subjects who were diagnosed with T2D in this dataset were subjects with prediabetes. In order to analyze the T2D scores among patients with NAFLD, we modified the dataset by selecting data where the patient is diagnosed with NAFLD; thus, the final data used is from 2741 patients, consisting of 2255 men and 486 women. We used all available data for the analysis, with 30 features with some classes included.
• The second dataset is NAFLD [36] dataset which is a population-based longitudinal data collection gathered from a program for patients with NAFLD who were undergoing medical examinations. This dataset has a total of 605 participants, including 321 males and 284 women. The patients who took part in this medical examination program ranged in age from 18 to 71, and the majority of them had the severe form of NAFLD known as NASH, which affected 537 individuals, or 88.76% of the total. In this dataset, according to FBG and HbA1c data, the T2D cases are in the stage of prediabetes and diabetes stages. 62 features with some classes make up the whole dataset.

2.2. Study Design and Implementation

Our proposed study has four main objectives, shown in Figure 1. To achieve the objectives, we first collected clinical data related to T2D and NAFLD. Once T2D data are collected, we further analyze the patient characteristics to summarize representative study participants. The patient characteristics measurement is accomplished by descriptive statistics, with measurement of the mean, standard deviation (STD), and the number of samples and their percentage, n (%). After analyzing the patient characteristics, furthermore, we measured the associations between the predictors, as well as the risk factors and T2D in each stage among patients with NAFLD, in NAFL or NASH stages. The association measurement is calculated by utilizing logistic regression, to achieve odds ratio (OR), 95% confidence index (95% CI), and significance p-value with an applied chi-square test [12,14,37]. We accomplished the association analysis by following the guidelines of WHO [38,39], the CDC [40], and ADA [41], thus the predictors and risk factors could be revealed. In the next step, we analyze the performance of the T2D prediction model based on the cut-point determination methods and logistic regression using clinical variables (FBG and HbA1c) recommended by WHO, CDC, and ADA. We measure the performance of the models in terms of various performance evaluation metrics, such as sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), likelihood ratio positive (LR+), likelihood ratio negative (LR-), accuracy, and area under the curve (AUC) along with its receiver operating characteristic (ROC) curve. Our diagnosis and classification are determined based on the optimal cut points, such as WHO/CDC/ADA recommendations, Youden Index, and product of sensitivity and specificity. For the last objective, we developed an efficient T2D prediction model for the early detection of T2D among patients with NAFLD. Our proposed T2D model utilized a forward logistic regression-based feature selection method and multi-layer perceptron. The performance of the proposed model was then evaluated by compared with other existing models, as well as with the models using clinical variables recommended by WHO/CDC/ADA.

Figure 2 shows the proposed T2D prediction model among people with NAFLD. Using collected publicly clinical diabetes-NAFLD datasets, we first preprocessed the dataset by selecting the data where the patients are diagnosed with NAFLD and then removing the missing values that existed. After data preprocessing, forward logistic regression-based feature selection is applied to select important features which have good performance, thus they could improve the accuracy of the model. After preprocessing and feature extraction, new data are created and ready to be used for learning and classification. In this stage, we employed the k-fold cross-validation technique, with k = 10, applied to all machine learning prediction models, such as logistic regression (LR), k-nearest neighbor (KNN), decision tree (DT), extreme gradient boosting (XGB), support vector machine (SVM), and our proposed model, multi-layer perceptron (MLP). The dataset is partitioned into 10 equal sections, and each classifier undergoes a 10-fold cross-validation. Nine of the sections are used for training while the remaining one is for testing. We went through that procedure ten times, switching the test set each time. The performance metrics (precision, recall, f1, and accuracy) were then selected in the final stage based on the results across iterations. The effectiveness of the prediction models used in this study was next evaluated across all of them.

2.3. Forward Logistic Regression-based Feature Selection Method

Feature selection is an important process before developing a prediction model. It is a process of reducing the number of input variables, considered as variables that have the strongest relationship with the target variable, thus significantly impacting the prediction of the model output [42]. The forward method based on logistic regression is one of the proper statistical-based feature selection methods which has been widely used, which could effectively gain the performance of the prediction model. In this study, we applied the forward method to the logistic regression model. In general, the forward feature selection method works by putting all the variables or the features into the n models and then checking the performance individually. It will then select the variable that produces the best performance, repeat the procedure while adding one variable at a time, keep the variable that produces the largest improvement, and repeat this process until there is no longer a noticeable improvement in the model’s performance. The full procedure of the forward logistic regression-based feature selection method is described in Algorithm 1.

Algorithm 1: A forward logistic regression-based feature selection method

Select dataset Let Mo be the null model, y = ß0 Start with a model with no variable For k = 0, …, p − 1 Consider all p-k models that augmented the variable in Mk with one additional variable Add the most significant variable, thus a model with one variable is created Choose the best model and call it Mk+1 based on the smallest significant p-value and best likelihood ratio of variable Select the best of the best from M0, M1, …., Mp, where Number of model = 1 + ${\sum }_{1=0}^{p-1}(p-k)$ = 1 + $\frac{p(p+1)}{2}$ End loop of k Return the the Mk+1

2.4. Multi-Layer Perceptron

A multilayer perceptron (MLP) model with numerous input features, a single hidden layer of numerous hidden neurons, and one output layer is developed for T2D classification among patients with NAFLD. Multilayer perceptron is a feed-forward neural network complement [30,43]. It has three different kinds of layers: an input layer, an output layer, and a hidden layer. The input layer is where the input signal for processing is received [44]. The output layer completes the necessary task, such as classification and prediction. The real computational engine of the MLP consists of an arbitrary number of hidden layers that are sandwiched between the input and output layers. Data travels from the input to the output layer of an MLP in the forward direction, much like a feed-forward network. With the help of the backpropagation learning method, the MLP’s neurons are taught. An MLP architecture is visualized in Figure 3.

The following calculations are carried out at each neuron in the hidden and output layers:

$$o\left(x\right)=G(b\left(2\right)+W\left(2\right)h\left(x\right))$$

(1)

$$h\left(x\right)=\phi \left(x\right)=s(b\left(1\right)+W\left(1\right)x)$$

(2)

Given the activation functions $G$ and $s$, the weight matrices $W\left(1\right)$ and $W\left(2\right)$, and the bias vectors $b\left(1\right)$ and $b\left(2\right)$. The $W\left(1\right),b\left(1\right),W\left(2\right),\mathrm{and} b\left(2\right)$ are the set of parameters that need to be learned. A common option for $s$ is the logistic sigmoid function, with sigmoid(a) = 1/(1 + e^−a), or the tanh function, with tanh(a) = (e^a − e^−a)/(e^a + e^−a).

2.5. Evaluation of the Model’s Performance

To evaluate the performance of the T2D prediction models, the 2 × 2 confusion matrix is utilized. The classification of disease status was defined into two categories, they are “diabetes” and “non-diabetes”, considered as a positive event and negative event. Using two categorical classes applied to the confusion matrix, the true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) values could be generated.

In order to evaluate the performance of the models using clinical variables recommended by WHO/CDC/ADA, we used sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the curve (AUC) along with the receiver operating characteristic (ROC) curve [45]. Using a ROC graph, the performance of the classifier may be organized, arranged, and chosen. The probability that the classifier would give a positively chosen instance a higher score than a negatively chosen instance at random is expressed as the area under the ROC curve (AUC). Furthermore, for machine learning-based prediction models and proposed model evaluation, we measure the precision or PPV, recall or sensitivity, f1, and accuracy. With regard to the models using clinical variables recommended by WHO/CDC/ADA, the performance measurements were carried out using SPSS version 25.0, while the machine learning-based prediction models and proposed model are implemented in Scikit-learn 1.1.2 and Python 3.9.7, respectively. The following equations can be used to determine sensitivity or recall, specificity, PPV or precision, NPV, f1, accuracy, and AUC [46]:

$$\mathit{Sensitivity}\mathit{or}\mathit{Recall}=\frac{TP}{TP+FN},$$

(3)

$$\mathit{Specificity}=\frac{TN}{FP+TN},$$

(4)

$$\mathit{PPV}or\mathit{Precision}=\frac{TP}{TP+FP},$$

(5)

$$\mathit{NPV}=\frac{TN}{FN+TN},$$

(6)

$$F1=\frac{2\times precision\times recall}{precision+recall},$$

(7)

$$Accuracy=\frac{TP+TN}{TP+FN+FP+TN}.$$

(8)

3. Results

3.1. Characteristics of Patient Population

A total of 2741 and 605 patients with NAFLD from the Gifu and NAFLD datasets are enrolled in the study population.

Table 1. Characteristics of the patient population.

Characteristics	Gifu NAFLD Dataset (n = 2741)	NAFLD Dataset (n = 605)
Sex, Men/Women, n (%)	2255 (82.24)/486 (17.72)	321 (53.06)/284 (46.94)
Age, year	44.00 ± 13.00	47.00 ± 16.00
Weight, kg	71.40 ± 13.90	85.00 ± 16.00
BMI, kg/m2	25.08 ± 3.78	31.21 ± 5.91
Normal (18.5 ≤ MI < 23), n (%)	544 (19.84)	3 (0.50)
Overweight (23 ≤ BMI < 25), n (%)	789 (28.77)	36 (5.95)
Obese I (25 ≤ BMI < 30), n (%)	1185 (43.22)	188 (31.07)
Obese II (BM ≥ 30), n (%)	219 (7.99)	377 (62.31)
WC, Men/Women, cm	86.00 ± 9.00/82.00 ± 12.00	106.00 ± 10.00/112.00 ± 16.00
HC, Men/Women, cm	-	104.00 ± 12.00/104.00 ± 15.75
ALT, U/L	27.00 ± 19.00	66.00 ± 54.00
AST, U/L	21.00 ± 9.00	42.00 ± 25.00
ALP, U/L	-	89.50 ± 46.00
GGT, U/L	23.00 ± 18.00	49.00 ± 43.75
LDH, U/L	-	215.00 ± 101.00
HDL, mg/dL	44.00 ± 13.50	44.00 ± 13.00
LDL, mg/dL	-	134.00 ± 53.00
Total Cholesterol, mg/dL	210.00 ± 44.00	211.00 ± 56.00
Triglyceride, mg/dL	111.00 ± 83.00	164.50 ± 114.25
HbA1c in general, %	5.30 ± 0.40	5.80 ± 1.00
FBG, mg/dL	97.00 ± 9.00	100.00 ± 25.00
SBP, mmHg	122.50 ± 19.00	121.00 ± 15.00
DBP, mmHg	77.50 ± 15.50	80.00 ± 10.00
Ethanol, mg/dL	50.89 ± 86.474	-
Alcohol, n (%)
Level 1	2088 (76.15)	-
Level 2	286 (10.43)	-
Level 3	250 (9.12)	-
Level 4	117 (4.27)	-
Smoking, n (%)
Level 1 (Never Smoking)	1226 (44.71)	267 (44.13)
Level 2 (Left Smoking)	726 (26.48)	206 (34.05)
Level 3 (Smoking)	789 (28.77)	104 (17.19)
Physical Inactive, n (%)	2340 (85.38)	-
Hypertension, n (%)	-	214 (35.37)
Hyperlipidemia, n (%)	-	351 (58.02)
Metabolic Syndrom, n (%)	-	392 (64.79)
NASH, n (%)	-	537 (88.76)
Diabetes and NASH, n (%)	-	207 (34.21)
Diabetes, n (%)	223 (8.13)	225 (37.19)

Note: Values are presented as mean ± standard deviation or number (%). BMI, body mass index; WC, waist circumference; HC, hip circumference; ALT, alanine aminotransferase; AST, aspartate aminotransferase, ALP, alkaline phosphatase; GGT, gamma-glutamyl transferase; LDH, lactate dehydrogenase; HDL, high-density lipoprotein; LDL, low-density lipoprotein HbA1c, glycated hemoglobin; FBG, fasting blood glucose; SBP, systolic blood pressure; DBP, diastolic blood pressure, NASH, non-alcoholic steatohepatitis.

shows the characteristics of the study participant.

3.2. Measure of Associations

Table 2. Association scores of predictor and T2D in NAFLD patient.

Predictor (Assigned Score)	Gifu NAFLD Dataset (T2D: Prediabetes Stage) (T2D Cases = 223, n = 2741)		NAFLD Dataset (T2D: Prediabetes + Diabetes Stages) (T2D Cases = 225, n = 605)
	OR (95% CI)	p-Value	OR (95% CI)	p-Value
Sex, Women	1.05 (0.74–1.50)	0.79	2.19 (1.57–3.07)	<0.001
Age, year
35 ≤ Age < 45	1.68 (0.80–3.53)	0.17	3.75 (1.84–7.62)	<0.001
45 ≤ Age < 60	2.03 (0.97–4.23)	0.06	5.58 (2.85–10.91)	<0.001
Age ≥ 60	2.13 (0.83–5.48)	0.11	14.18 (6.23–32.26)	<0.001
BMI, kg/m2
Overweight (23 ≤ BMI < 25)	0.91 (0.57–1.44)	0.68	1.15 (0.11–12.44)	0.91
Obese I (25 ≤ BMI < 30)	1.53 (1.03–2.29)	0.03	1.18 (0.12–11.58)	0.89
Obese II (BM ≥ 30)	2.76 (1.66–4.56)	<0.001	2.24 (0.23–21.70)	0.48
WC ≥ 90 (Men), cm	2.27 (1.67–3.09)	<0.001	1.63 (0.18–14.76)	0.66
ALT, U/L	1.91 (1.31–2.78)	<0.001	0.50 (0.36–0.70)	<0.001
AST, U/L	1.50 (0.68–3.35)	<0.001	0.99 (0.36–1.40)	<0.001
ALP, U/L	-		0.77 (0.51–1.17)	<0.001
GGT, U/L	1.52 (0.96–2.40)	<0.001	1.39 (0.99–1.96)	<0.001
HDL, mg/dL
Elevated in Men (40 ≤ HDL < 59)	2.80 (1.01–7.72)	<0.001	0.78 (0.26–2.40)	0.67
Low in Men (HDL < 40)	4.55 (1.64–12.60)	<0.001	0.81 (0.26–2.55)	0.72
Elevated in Women (50 ≤ HDL < 59)	4.92 (1.39–17.41)	<0.001	0.88 (0.41–1.90)	0.75
Low in women (HDL < 50)	5.54 (1.60–18.52)	<0.001	0.71 (0.36–1.42)	0.34
Total Cholesterol ≥ 240 (High), mg/dL	1.65 (1.13–2.40)	0.01	0.87 (0.57–1.32)	0.51
Triglyceride ≥ 240 (High), mg/dL	2.19 (1.55–3.08)	<0.001	1.12 (0.77–1.64)	0.55
SBP ≥ 240 (High), mmHg	4.53 (3.10–6.63)	<0.001	2.74 (1.63–4.61)	<0.001
DBP ≥ 90 (High), mmHg	1.41 (0.94–2.12)	<0.001	1.30 (0.81–2.06)	0.26
Alcohol Intake
Level 2	1.11 (0.72–1.72)	0.62	-	-
Level 3	0.76 (0.45–1.29)	0.31	-	-
Level 4	0.93 (0.46–1.87)	0.04	-	-
Smoking
Level 2	1.07 (0.74–1.55)	0.70	0.97 (0.67–1.41)	0.87
Level 3	1.94 (1.44–2.69)	<0.001	0.92 (0.57–1.47)	0.72
Physical Inactive	1.62 (1.03–2.55)	0.04	-	-
Hypertension, n (%)	-	-	2.62 (1.85–3.70)	<0.001
Hyperlipidemia, n (%)	-	-	2.08 (1.47–2.94)	<0.001
Metabolic Syndrom, n (%)	-	-	4.38 (2.52–6.58)	<0.001
Type of Disease = Severe Illness, n (%)	-	-	1.83 (1.00–3.38)	0.05
NASH, n (%)	-	-	1.74 (0.99–3.07)	0.05

Notes: Values are presented as an odd ratio, 95% confidence index, and p-value with the chi-square test. BMI, body mass index; WC, waist circumference; ALT, alanine aminotransferase; AST, aspartate aminotransferase, ALP, alkaline phosphatase; GGT, gamma-glutamyl transferase; HDL, high-density lipoprotein; TC, total cholesterol; SBP, systolic blood pressure; DBP, diastolic blood pressure; NASH, non-alcoholic steatohepatitis.

displays the predictor and T2D association scores in NAFLD patients. According to the characteristics of the patient’s results, the Gifu T2D patients are considered to have prediabetes, while in the NAFLD dataset, the positive class of T2D patients is the patients who are considered to have diabetes.

3.3. Measure of the Performance of Diabetes Diagnostic Tests

The performance of T2D diagnostic tests in patients with NAFLD is shown in

Table 3. Performance of T2D diagnostic tests in patients with NAFLD.

Model Population	Gifu NAFLD Dataset (T2D: Prediabetes Stage) (T2D Cases = 223, n = 2741)		NAFLD Dataset (T2D: Prediabetes + Diabetes Stages) (T2D Cases = 225, n = 605)
Diagnostic Test	FPG	HbA1c	FPG	HbA1c
Cut point as recommended by WHO or ADA (a)	100	5.7	100	5.7
Youden Index-based optimal cut point (b)	99.50	5.426	111.50	5.960
Sensitivity and Specificity-based optimal cut point (c)	100.50	5.426	103.50	5.885
Percentage of high risk based on a/b/c	5.91/5.91/5.47	3.14/5.36/5.36	51.75/30.62/42.67	56.91/40.60/46.10
Sensitivity, based on a/b/c (%)	72.65/72.65/67.26	38.57/65.92/65.92	83.33/67.12/77.03	82.95/73.27/75.12
Specificity, based on a/b/c (%)	63.66/63.66/68.39	88.80/68.51/68.51	66.75/90.77/77.31	59.37/79.83/72.05
PPV, based on a/b/c (%)	15.04/15.04/15.86	23.37/15.64/15.64	59.49/80.98/66.54	56.07/69.43/62.69
NPV, based on a/b/c (%)	96.33/96.33/95.93	94.23/95.78/95.78	87.24/82.49/85.17	84.77/82.69/82.24
LR+, based on a/b/c (%)	2.00/2.00/2.13	3.44/2.09/2.09	2.51/7.27/3.39	2.04/3.63/2.69
LR−, based on a/b/c (%)	0.43/0.43/0.48	0.69/0.50/0.50	0.25/0.36/0.30	0.29/0.33/0.35
Accuracy, based on a/b/c (%)	64.39/64.39/68.30	84.71/68.30/68.30	72.88/82.03/77.20	68.44/53.10/73.23
Youden Index, based on a/b/c (%)	36.31/36.31/35.65	27.37/34.43/34.43	50.09/57.88/54.34	42.32/53.10/47.16
Standard error	0.018	0.018	0.018	0.020
p-value (CST, HLT)	<0.001, 0.514	<0.001, 0.214	<0.001, 0.000	<0.001, 0.000
95% CI of AUC	0.69–0.76	0.69–0.76	0.81–0.89	0.78–0.87
AUC	0.73	0.73	0.85	0.82
RMSE	0.998	0.999	0.856	0.849
	0.990		0.806
GM ± SD	1.89 ± 1.273		1.53 ± 0.487

Notes: WHO, world health organization; ADA, American diabetes association; PPV, positive predictive value; NPV, negative predictive value; LR+, likelihood ratio positive; LR−, likelihood ratio negative; 95% CI, 95% confidence index; CST, chi-square test; HST, Hosmer, and Lemeshow test; AUC, the area under the curve; RMSE, root mean standard error; GM, geometric mean; SD, standard deviation; ^a, cut point as recommended by WHO or ADA; ^b, Youden index-based optimal cut point; ^c, sensitivity and specificity-based optimal cut point.

. The cut-off points for FPG and HbA1c based on the sensitivity and specificity of the Gifu and NAFLD datasets are shown in Figure 4 and Figure 5. The ROC curves of FBG and HbA1c applied to the Gifu and NAFLD datasets are shown in Figure 6 and Figure 7.

3.4. Feature Selection Results Based on Forward Logistic Regression Method

Before developing the T2D prediction model, we conducted a feature selection applied to all NAFLD datasets used in this study. Feature selection based on the forward logistic regression method is employed in the proposed model, as well as other existing machine learning models. The characteristics of the dataset can be seen in

Table 4. Characteristics of the dataset used in the study.

Dataset	Characteristics
	Size	Number of Original Features	Number of Class	Number of Selected Features
Gifu NAFLD dataset	2741	30	2	7
NAFLD dataset	605	62	11	13

. Before applying the forward logistic regression-based feature selection method, the original datasets consisted of 30 and 62 features for the Gifu NAFLD dataset and NAFLD dataset, respectively. Figure 8 shows the results of selected features based on the forward logistic regression method in the Gifu NAFLD dataset (see Figure 8a) and NAFLD dataset (see Figure 8b).

3.5. Performance of Proposed T2D Prediction Model

The last analysis conducted in this study is, we measure the performance of the T2D prediction models utilizing numerous machine learning techniques, measured by precision, recall, F1, and accuracy (see Figure 9 and Figure 10). Along with that, the 95% confidence intervals of all evaluation metrics are also measured in our study (see

Table 5. The 95% confidence intervals (95% CI) of performance evaluation metrics on the Gifu NAFLD dataset before and after feature selection (FS).

Model	95% CI of Precision (%)		95% CI of Recall (%)		95% CI of F1 (%)		95% CI of Accuracy (%)
	Before FS	After FS	Before FS	After FS	Before FS	After FS	Before FS	After FS
LR	49.81–59.89	54.38–65.62	48.16–57.91	46.25–55.81	45.40–58.92	43.35–56.54	83.12–100.00	83.44–100.00
KNN	52.22–62.79	61.91–74.71	46.93–56.43	48.37–58.37	44.51–58.05	46.75–60.98	82.29–99.37	82.15–99.55
DT	47.48–57.09	47.81–57.48	47.21–56.77	48.37–58.37	45.40–58.92	44.80–58.44	72.84–87.95	72.78–88.20
XGB	46.26–55.63	47.38–47.17	43.60–52.55	45–53–54.95	42.20–54.77	43.07–56.19	75.97–91.73	80.05–97.00
SVM	41.89–50.36	41.81–50.45	45.60–54.83	47.15–56.67	42.03–54.54	47.13–56.67	83.59–100.00	83.44–100.00
MLP	47.38–61.48	73.45–88.64	49.02–58.94	48.96–59.10	48.14–62.47	48.14–62.47	82.32–99.40	83.68–100.00

and

Table 6. The 95% confidence intervals (95% CI) of performance evaluation metrics on the NAFLD dataset before and after feature selection (FS).

Model	95% CI of Precision (%)		95% CI of Recall (%)		95% CI of F1 (%)		95% CI of Accuracy (%)
	Before FS	After FS	Before FS	After FS	Before FS	After FS	Before FS	After FS
LR	73.29–85.43	78.02–90.56	70.10–81.71	79.10–83.73	67.80–84.21	72.72–89.78	70.64–84.96	75.70–90.71
KNN	61.76–71.99	72.68–69.83	59.91–69.83	68.60–79.63	58.00–71.61	67.02–82.75	60.72–73.03	71.95–85.01
DT	66.29–77.27	68.74–80.13	65.25–76.06	67.99–78.93	62.64–77.80	65.90–81.37	64.92–78.07	68.49–82.05
XGB	74.77–87.16	76.34–88.61	67.19–78.32	75.15–87.23	71.52–88.83	73.09–90.02	73.75–88.69	75.34–90.28
SVM	71.58–83.43	76.22–88.47	67.22–88.69	71.30–83.11	65.43–81.26	69.87–86.27	68.67–82.59	73.40–87.95
MLP	68.37–84.92	78.08–90.63	70.61–82.30	74.94–86.99	68.37–84.92	72.99–90.65	70.86–85.22	75.87–90.91

).

Table 7. The improvement score after forward logistic regression-based feature selection method application on the MLP model.

Dataset	Improvement Score (%)
	Precision	Recall	F1	Accuracy
Gifu NAFLD dataset	10.40	0.04	0.89	1.65
NAFLD dataset	4.94	4.51	5.17	5.35

shows the improvement score after the forward logistic regression-based feature selection method application on the MLP model.

4. Discussion

As shown in Table 2, in the Gifu NAFLD patient population, there were no associations between sex, age, overweight, or obesity level I with T2D where the p-values were larger than 0.05 (p-value > 0.05). In contrast, the sex and age associations with T2D are statistically significant (p-value < 0.001). The significant scores of sex and age were followed by odds ratio (OR) scores, particularly in patients aged 45 to less than 60 years or older than 60 years, with OR scores of 5.58 and 14.18, respectively. The OR scores indicate that T2D is 5.58 and 14.18 times more likely to be present in NAFLD patients aged 45 to 59 and older than 60, respectively than in those without T2D. Obesity level II and a waist circumference greater than 90 cm in men were found to have significant associations with diabetes (p-value < 0.001) in the Gifu NAFLD patient population. The association scores regarding the medical test results, such as ALT, AST, ALP, and GGT, were found to be statistically significant, with p-values of < 0.001 in both the Gifu and NAFLD patient populations. The ALT, AST, ALP, and GGT are considered the important predictors for T2D among patients with NAFLD, followed by HDL, triglyceride, SBP, DBP, and smoking level 3, whereas all are significant in the Gifu population among men and women. The high OR scores in HDL and SBP are shown in NAFLD patients in Gifu. T2D in NAFLD patients is 4.55 and 5.54 times more likely to be present in patients with HDL of lower than 40 mg/dL in men and lower than 50 mg/dL in women, respectively than in those without diabetes. The SBP was also shown high, where the OR scores of elevated and high are 3.83 and 4.53. In the NAFLD population, high levels of SBP, hypertension, hyperlipidemia, and metabolic syndrome are significantly associated with T2D in patients with NAFLD. T2D was recorded as 4.38 times more likely to be present in patients with metabolic syndrome than those without T2D. According to the results using the Gifu dataset (with prediabetes), among patients with NAFLD, the risk factors for T2D based on the WHO/CDC/ADA [38,39,40,41] guidelines that have significant scores are BMI with obesity level II, tests related to hypertension (SBP and DBP), high total cholesterol, high triglyceride, smoking level 3, alcohol intake level 4, and physical inactivity. These risk factors were found to be similar to WHO/CDC/ADA guidelines and previous studies’ findings [12,13,14,15,17]. Only sex and age of ≥45 are not significant. While using the NAFLD dataset, where the case of T2D consists of prediabetes and diabetes levels, the risk factors for T2D were found to be women, the age for all ranges (starting from ≥35 years old), SBP, and hypertension. Other risk factors such as BMI, cholesterol, triglycerides, and smoking did not show to be significant due to the fact that the majority of the patients were at high levels for those risk factors, which means low variation, and thus they were not found to be statistically significant, while alcohol intake and physical inactivity values were due to unavailability in the dataset.

According to the FBG and HbA1c data, the optimal cut-off points measured by Youden Index were recorded to be optimal at 99.50 mg/dL and 5.426%, respectively in the Gifu NAFLD dataset. While in the NAFLD dataset, the Youden Index-based optimal cut-off points are 111.50 mg/dL and 5.96% for FBG and HbA1c, respectively. The optimal cut-off points produced by sensitivity and specificity were found different, where the optimal cut-off points for FBG are 100.5 mg/dL and 103.50 mg/dl, and for HbA1c are 5.426% and 5.885%, shown in Figure 4 and Figure 5 respectively. Using those various cut-off points, the results show the best performance in terms of sensitivity majority produced by cut-off points as recommended by WHO, 100 mg/dL for FBG, with sensitivity scores of 72.65%, 83.33%, and 82.95%. In terms of specificity, the best performance is recorded by the product of sensitivity and specificity cut-off point and WHO cut-off point recommendation with the specificity of 68.39% and 88.80%. Youden Index-based cut-off point produces the best sensitivity in both FBG and HbA1c in the NAFLD dataset, with a sensitivity of 83.33% and 82.95%. In terms of accuracy, the FBG test applied to the Gifu NAFLD dataset can accurately classify diabetes as much as 68.30% using a cut-off point based on sensitivity and specificity products. While the HbA1c test is capable to classify T2D as much as 84.71% based on the WHO cut-off point recommendation. In the NAFLD dataset, the FBG test is a maximum of 82.03% capable to classify diabetes using the Youden Index-based cut-off point, and the HbA1c test is 73.23% capable to classify diabetes using the product of sensitivity and specificity cut-off point. In terms of AUC, the ability of the FBG and HbA1c tests applied to the Gifu NAFLD dataset is similar (73%) and both are worthless [47], while in the NAFLD dataset, the performance is much better, whereas 82% and 83% were able to classify T2D. Furthermore, the goodness of fit test results based on the Hosmer and Lemeshow tests show a good fit for both the Gifu and NAFLD datasets, with p-values of 0.684 and 0.082, respectively. The results indicate the model adequately fits the data while utilizing FPG and HbA1c, with p-values larger than 0.05. Finally, the optimal scaling applied to glucose (FPG) and HbA1c levels for both the Gifu and NAFLD datasets using elastic-net regularization with 10-fold cross-validation displayed good scores in terms of the root mean square error (RMSE). The RMSE scores of the models utilizing glucose (FPG) and HbA1c in the Gifu NAFLD dataset are 0.998 and 0.999, respectively. In the NAFLD dataset, the RMSE scores of the models utilizing glucose (FPG) and HbA1c are 0.856 and 0.849, respectively. The RMSE scores of the models in both datasets were low, which indicates that the predictions are close to the actual values and also display small errors between the predicted and actual values [48]. In addition, we also measured the geometric means following Gerstein et al. [49]. The geometric mean of the glucose (FPG) and HbA1c variables in the Gifu NAFLD dataset are 1.89, with a standard deviation of 0.273. In the NAFLD dataset, the geometric mean for the glucose and HbA1c variables is 1.53, with a standard deviation of 0.487. Both of the geometric means showed not much difference compared to [49], which utilized the same variables. The results in Table 3 showed, using a cut point recommended by WHO, CDC, and ADA, that the accuracy of the FPG test used to diagnose T2D is 64.39% in the Gifu NAFLD dataset. In the NAFLD dataset, the accuracy of the test using HbA1c is 68.44%. Our results show that utilization of a cut point recommended by WHO/CDC/ADA is not always the best solution. The validity and reliability of these recommended tests still need to be evaluated [14]. The inconsistency in the validity of the test results could be due to several factors, such as disease progress that may change the blood glucose level of the patient over time [50] and also due to technical problems that affect the validity of the test results run by the test tool, such as a glucometer [51]. In this study, the results showed the best accuracy scores achieved by utilizing other cut points, such as those based on the Youden index and the product of sensitivity and specificity. For example, utilizing cut points based on the product of sensitivity and specificity achieved the best accuracy scores for FBG and HbA1c tests in the Gifu and NAFLD datasets, with accuracy scores of 68.30% and 73.23%, respectively. While another result found the best accuracy achieved by utilizing a cut point based on the Youden index in the NAFLD dataset, with an accuracy of 82.03% using the FPG test. Therefore, analysis of the performance of T2D diagnostic tests is suggested when conducting an assessment of T2D screening scores [12,13,14,15,16,17].

According to the results of feature selection shown in Figure 8a, the selected features after the application of the forward logistic regression-based feature selection method are WC (a value that represents a patient’s abdomen measurement in centimeters, cm), ALT (a value representing the amount of the alanine transaminase enzyme in the liver), Weight (the amount of the patient’s body weight measured in kilograms, kg), HDL-mgdl (the amount of high-density multiprotein complex particles that move all lipid molecules through the body’s water and outside of cells, measured in milligrams per deciliter, mgdL), HbA1c-mmol (the amount of blood sugar or glucose attached to a patient’s hemoglobin, measured in millimoles, mmol), Smoking (a patient’s smoking status measured in levels 1 to 3), and glucose-mgdl (a patient’s fasting blood sugar (FBS) level, measured in milligram per deciliter, mgdL) for the Gifu NAFLD dataset. For the NAFLD dataset (see Figure 8b), the selected features are hip circumference (a value that represents a patient’s hip measurement in centimeters, cm), hypertension (the high level of blood pressure measured in millimeters of mercury, mmHg), hyperlipidemia (the presence of a high amount of lipids in the patient’s blood), body mass index (an indicator of total body fat in a patient, which is measured by the patient’s weight in kilograms divided by the square of their height in meters), systolic blood pressure (the amount of pressure experienced by the arteries while the heart is beating, measured in millimeters of mercury, mmHg), metabolic syndrome (the patient’s status according to the presence of a cluster of risk factors for diabetes, hypertension, and heart disease), albumin (the amount of albumin, which is a protein in blood plasma made by the liver, measured in gram per deciliter, g/dL), age (a period of a patient’s life, measured in years), HDL (the amount of good multiprotein complex particles that move all lipid molecules through the body’s water and outside of cells, measured in milligram per deciliter, mgdL), LDL (the amount of low-density multiprotein complex particles that move all lipid molecules through the body’s water and outside of cells (measured in milligrams per deciliter, mgdL), glucose (a patient’s fasting blood sugar (FBS) level, measured in milligrams per deciliter, mgdL), Hemoglobin-A1C (the amount of blood sugar or glucose attached to a patient’s hemoglobin, measured in millimoles, mmol), and fibrosis (the estimated amount of scarring in the liver, measured in scores 0 to 4).

As shown in Figure 9 and Figure 10, before applying feature selection based on the forward logistic regression method, the performance of the models was considered to be low in both Gifu and NAFLD datasets compared to after the feature selection method application. Our proposed model outperforms in terms of all performance evaluation metrics among all the existing models, such as LR, KNN, DT, XGB, and SVM in both datasets. Using the Gifu NAFLD dataset (see Figure 9), our proposed T2D prediction model achieves a precision of 80.69%, a recall of 53.78%, an F1 score of 54.84%, and an accuracy of 92.11%. The improvement scores after the forward logistic regression-based feature selection method application recorded as much as 10.40%, 0.04%, 0.89%, and 1.65%, respectively, for precision, recall, F1, and accuracy (see Table 7). Furthermore, using the NAFLD dataset (see Figure 10), our proposed T2D prediction model achieves precision, recall, F1, and accuracy as high as 84.12%, 80.74%, 81.37%, and 83.05%, respectively, with improvement scores of 4.94%, 4.51%, 5.17%, and 5.35% (see Table 7).

Finally, we also compared our proposed T2D prediction model to the models using clinical variables (diabetes diagnostic tests) recommended by WHO and ADA (see Table 3). Based on the results, our proposed T2D prediction model produces better performance compared to the models using clinical variables (diabetes diagnostic tests) recommended by WHO and ADA when applied in both Gifu and NAFLD datasets. Utilizing the Gifu NAFLD dataset, our proposed T2D prediction model achieves better performance in terms of recall or PPV and accuracy, where the maximum score of recall or PPV of the model using clinical variables is 23.37% and accuracy is 84.71%, our proposed model achieves 80.69% and 92.11%. Followed by the results in the NAFLD dataset (see Figure 9), our proposed model also outperforms the models using clinical variables (diabetes diagnostic tests) recommended by WHO and ADA in terms of recall or PPV and accuracy, with the maximum score of the recall or PPV is 80.98% and accuracy is 82.03%, where our proposed model achieves 84.12% and 83.05% for recall or PPV and accuracy, respectively.

Based on our findings in this study, there are three clinical relevances. First, using the chi-square test, the ALT, AST, ALP, GGT, and SBP were significant predictors (p-values < 0.001) in both the Gifu and NAFLD patient populations, which means the important predictors or risk factors of T2D among patients with NAFLD. Thus, these predictors could be used or assigned as the risk factors of T2D in patients with NAFLD by clinical practitioners or organizations. The second clinical relevance is, we found that the optimal cut points with the best sensitivity to diagnose prediabetes in patients with NAFLD based on FPG and HbA1c are 99.50 mg/dL and 5.426%, respectively. The FPG cut point of 99.50 mg/dL and HbA1c cut point of 5.426% could be used to diagnose prediabetes in patients with NAFLD by clinical practitioners or organizations. The last clinical relevance is, as the best accuracy shown by our proposed T2D prediction model, it also could be used by clinical practitioners or organizations as the non-clinical alternative to diagnose T2D (prediabetes and diabetes) in patients with NAFLD.

Along with the significant improvement in performance made by our proposed T2D model, the limitations of the model should be noted. First, our proposed model could not handle the imbalance of class distribution or ratio. As shown in Figure 8, the recall or sensitivity in the Gifu NAFLD dataset showed low scores, which are around 50–53%. These low scores of recall or sensitivity are caused by the low rate of diabetes cases (called the “minority class”) compared with non-diabetes cases (called the “majority class”) [14]. Once the minority class and majority class are much different, it will produce an imbalanced class distribution or ratio, which could affect the classification results [52,53,54,55]. Utilization of the data balancing method has been proven [52,53,54,55] to increase the performance of the model. On the other side, the low score of sensitivity could also be caused by the presence of unmatched data with the cutoff point of the test criteria. Therefore, it affects the performance of the model in terms of recall or sensitivity. Based on these low recall or sensitivity scores, which reflect the limitations of our proposed model, we suggest utilizing the data balancing method for solving the imbalanced class distribution or ratio, thus improving the performance of the model in correctly classifying the positive cases. The second limitation is, due to the limitations of the T2D dataset in patients with NAFLD, our proposed T2D prediction model has not been validated by external validation datasets to test its validity in a set of new patient populations. Therefore, the utilization of more T2D-NAFLD datasets is still needed to improve the validity of the model. By utilizing more T2D-NAFLD datasets, it could also be used to reveal the variables that are important to predict T2D among patients with NAFLD. Revealing the variables for prediction could also be considered predictors or risk factors of T2D in NAFLD. Thus, we suggest the utilization of more T2D-NAFLD datasets and their investigation.

5. Conclusions

The utilization of the forward logistic regression-based feature selection method on the MLP model significantly improves the performance of the proposed T2D prediction model in terms of precision, PPV, recall, sensitivity, f1, and accuracy. Our proposed model also outperforms the existing machine learning models, such as LR, KNN, DT, XGB, and SVM. Moreover, our proposed T2D prediction model is expected to aid NAFLD patients in both preventing T2D incidents and taking proactive measures once T2D is diagnosed.