Deep Neural Network for Predicting Diabetic Retinopathy from Risk Factors

Abstract

Extracting information from individual risk factors provides an effective way to identify diabetes risk and associated complications, such as retinopathy, at an early stage. Deep learning and machine learning algorithms are being utilized to extract information from individual risk factors to improve early-stage diagnosis. This study proposes a deep neural network (DNN) combined with recursive feature elimination (RFE) to provide early prediction of diabetic retinopathy (DR) based on individual risk factors. The proposed model uses RFE to remove irrelevant features and DNN to classify the diseases. A publicly available dataset was utilized to predict DR during initial stages, for the proposed and several current best-practice models. The proposed model achieved 82.033% prediction accuracy, which was a significantly better performance than the current models. Thus, important risk factors for retinopathy can be successfully extracted using RFE. In addition, to evaluate the proposed prediction model robustness and generalization, we compared it with other machine learning models and datasets (nephropathy and hypertension–diabetes). The proposed prediction model will help improve early-stage retinopathy diagnosis based on individual risk factors.

1. Introduction

Diabetes is a chronic disease associated with abnormal blood glucose (BG) levels. Patients with type 1 diabetes (T1D) cannot control their BG naturally due to lacking insulin secretion, while for type 2 diabetes (T2D), the body cannot utilize its produced insulin [1,2]. Thus, T1D patients must administer insulin via injection or an insulin pump to achieve a near-normal glucose metabolism [3]. For T2D patients, a healthy diet, physical exercise, and drug administration are suggested to control BG levels and prevent many complications. Diabetes patients commonly develop acute complications, such as hypoglycemia (BG < 70 mg/dL) and hyperglycemia (BG > 180 mg/dL) if they fail to carefully self-manage BG levels [4]. Excessive BG (hyperglycemia) can result in long-term complications, e.g., retinopathy [5,6], nephropathy or kidney disease [7,8], and cardiovascular disease [9,10]. Diabetic retinopathy (DR) is the most common ocular complication from diabetes and the leading cause of visual impairment among patients [11]. Thus, it is important to accurately predict related complications to help prevent their progression [12], and individual significant risk factors can be utilized as input features to improve prediction model performance.

Considerable research effort has recently focused on developing of retinopathy prediction models using machine learning based on individual risk factors, aiming for high accuracy and good generalization. Pre-diagnosis models have been used for different populations and have shown significant performance predicting diabetic retinopathy [13,14,15,16,17]. Deep neural networks (DNNs) are machine learning systems that incorporate many layers to learn more complex patterns, achieving exceptional prediction accuracy for many applications. In particular, DNNs have shown excellent performance by improving classification accuracy compared with conventional models [18,19,20,21,22,23]. DNNs have also been trained to successfully predict diabetic retinopathy based on retinal fundus images [24,25,26,27,28,29]. Furthermore, previous studies have established positive impacts from support vector machine–recursive feature elimination (SVM-RFE) as the feature selection algorithm on improving classification accuracy [30,31,32,33,34,35,36,37], especially for DNNs [38,39].

However, none of these previous studies included DNNs with RFE for retinopathy prediction based on individual risk factors. Therefore, the current study integrated DNN and SVM-RFE for retinopathy to improve prediction accuracy. We employed SVM-RFE to extract significant risk factors and DNN to subsequently generate higher accuracy prediction models. Utilizing the identified significant retinopathy predictors (risk factors) provides optimized prediction models. Thus, this study will help diabetic patients to reduce diabetic retinopathy risk, the major cause of blindness in such patients.

The remainder of this paper is organized as follows. Section 2 discusses related works for retinopathy prediction, including related DNN and RFE applications for healthcare. Section 3 details the proposed retinopathy prediction model, and Section 4 discusses experimental results for the proposed model. Section 5 summarizes and concludes the paper, including discussing study limitations and future research directions.

2. Literature Review

This section discusses previously proposed machine learning models for retinopathy pre-diagnosis, with particular attention regarding DNN and RFE for health-related datasets.

2.1. Diabetic Retinopathy Prediction

Prediction models based on machine learning and employing individuals’ risk factors as input features helped improve retinopathy pre-diagnosis for Iran, Korea, US, and Taiwan populations. Hosseini et al. [13] used a logistic regression (LR) model to predict retinopathy for 3734 T2D patients from the Isfahan Endocrinology and Metabolism Research Center, Iran. The model achieved AUC = 0.704, sensitivity = 60%, and specificity = 69%. Oh et al. [14] proposed retinopathy risk prediction based on Lasso. They used a dataset from the Korea National Health and Nutrition Examination Surveys (KNHANES) V-1, with 327 patients randomly selected as training data and 163 as validation. The proposed model achieved AUC = 0.81, accuracy = 73.6%, sensitivity = 77.4%, and specificity = 72.7%.

Retinopathy prediction has also been applied for US patients. Ogunyemi and Kermah [15] investigated machine learning models for predicting retinopathy from six health centers in South Los Angeles, comprising 513 T2D patients. The dataset was split into approximately 80% for training and the remainder for testing. They compared RUSBoost ensemble and AdaBoost model predictions and showed that RUSBoost ensemble was superior to AdaBoost, achieving accuracy = 73.5%, AUC = 0.72, sensitivity = 69.2%, and specificity = 55.9%. LR, support vector machine (SVM), and artificial neural network (ANN) models also helped improve retinopathy prediction performance for the overall US population [16] using an updated dataset comprising 27,116 T1D and T2D patients from the Los Angeles County Department of Health Services. Combining ANN and the synthetic minority oversampling technique (SMOTE) achieved superior results with AUC = 0.754, sensitivity = 58%, and specificity = 80%. Tsao et al. [17] proposed SVM-, decision tree (DT)-, ANN-, and LR-based models to predict DR based on several risk factors for T2D patients. The dataset was gathered from a private hospital in Northern Taiwan, incorporating 430 normal and 106 DR patients. The SVM based model was superior to the other algorithms considered, achieving accuracy = 79.5% and AUC = 0.839.

2.2. Deep Neural Network

Deep neural network models have been used in many previous studies to the improve prediction accuracy compared with other models. For diabetes prediction-related issues, DNNs have only been applied to predict T2D. Kim et al. [22] used phenotype and genotype data from the Nurses’ Health Study (NHS) with data from the Health Professionals Follow-up Study (HPFS) to evaluate the prediction model performance. The proposed DNN outperformed LR, with AUC = 0.931 and 0.928 for male and female patients. Stacked autoencoders in DNN were also applied for T2D diabetes classification [23]. Their model was applied to a Pima Indian dataset and achieved classification accuracy = 86.26%.

Diabetic retinopathy is a complication of diabetes that causes damage to the blood vessels in the retina and leading to vision impairment. Therefore, an accurate prediction model for pre-diagnosis retinopathy would be very useful to improve patient health outcomes. DNN have shown good performance diagnosing retinopathy from retinal fundus images, including datasets from Otago and Messidor [24], and three clinical departments in Sichuan Provincial People’s Hospital [25]. Parmar et al. [26] employed a convolutional neural network to detect DR from retinal images and their model outperformed others considered. Furthermore, the ResNet architecture model was utilized to detect DR from fundus images achieving an excellent classification accuracy [27,28]. Finally, Gadekallu et al. [29] employed a DNN with grey wolf optimization (GWO) and principle components analysis (PCA) to optimize the parameters and reduce dimensionality, respectively, to predict DR based on extracted features from retinal imaging. However, these studies only diagnosed retinopathy from retinal fundus images, and to our best knowledge, no previous study considered DNNs for retinopathy based on risk factors.

2.3. Recursive Feature Elimination

Many previous studies employed feature selection to improve the model prediction accuracy. Guyon et al. [30] introduced RFE to select the most significant gene(s) for cancer classification and, hence, improve classification model accuracy. The algorithm calculates a rank score and eliminates the lowest-ranking features. Previous studies showed significant performance improvements by employing RFE, including predicting mental states (brain activity) [31,32], Parkinson [33], skin disease [34], autism [35], Alzheimer [36], and T2D [37]. They showed that SVM-RFE achieved superior performance than several comparison methods. In addition, previous studies demonstrated DNN accuracy improvement by integrating RFE as the feature selection algorithm [38,39]. The experimental results showed that integration of SVM-RFE to DNN algorithms achieved best prediction accuracy as compared to other methods.

To our best knowledge, only Kumar et al. [37] considered RFE for diabetes prediction. Kumar et al. used SVM-RFE to identify the most discriminatory gene target for T2D. These identified significant genes could then be focused on as potential drug targets. However, although SVM-RFE was employed to extract significant features for T2D, this was not applied to the DR dataset. Similarly, previous DR studies used DNNs to classify disease from retinal fundus images only, and risk factors were not utilized as DNN input features. Therefore, the current study proposed SVM-RFE and DNN to improve DR prediction accuracy from individual risk factors. To our best knowledge, this is the first time SVM-RFE and DNN using individual risk factors were employed to improve DR prediction accuracy.

3. Methodology

3.1. Datasets

Previous studies established that T1D or T2D patients tend to develop complications, such as retinopathy, nephropathy, cardiovascular disease (CVD), etc. Therefore, we proposed a DNN-based model to predict whether T1D or T2D patients will later develop DR. The dataset was collected by Khodadadi et al. [40] and related with diabetes complications in Lur and Lak populations of Iran. Informed consent was obtained from patients, and the dataset was made publicly available by previous authors (https://data.mendeley.com/datasets/k62fdsnwkg/1). The dataset was gathered from 133 diabetic patients covering known risk factors for neuropathy, nephropathy, diabetic retinopathy (DR), peripheral vessel disease (PVD), CVD, food ulcer history, and dawn effect. Originally, the dataset consisted of 24 information gathered from diabetic patients (T1D and T2D). We removed irrelevant features, leaving 14 potentially DR-relevant risk factors, as shown in

Table 1. Diabetic retinopathy dataset.

No	Attribute	Description	Type	Range
1	BMI	Subject’s body mass index	Numeric	18–41
2	DM duration	Subject’s diabetes duration (y)	Numeric	0–30
3	A1c	Subject’s average blood glucose level over the past 3 months (mg/dL)	Numeric	6.5–13.3
4	Age	Subject’s age (y)	Numeric	16–79
5	FBS	Subject’s fasting blood sugar level (mg/dL)	Numeric	80–510
6	LDL	Subject’s low-density lipoprotein level (mg/dL)	Numeric	36–267
7	HDL	Subject’s high-density lipoprotein level (mg/dL)	Numeric	20–62
8	TG	Subject’s triglyceride level (mg/dL)	Numeric	74–756
9	Sys BP	Subject’s systolic blood pressure (mmHg)	Numeric	105–180
10	Dias BP	Subject’s diastolic blood pressure (mmHg)	Numeric	60–120
11	Sex	Subject’s sex	Categorical	0 = Female 1 = Male
12	DM type	Subject’s diabetes type (T1D or T2D)	Categorical	0 = T1D 1 = T2D
13	DM treat	Subject’s diabetes treatment	Categorical	0 = Both (Insulin and oral agent) 1 = Insulin 2 = Oral agent
14	Statin	Subject’s statin status (frequently used as part of diabetes care)	Categorical	0 = Ator (atorvastatin) 1 = No statin 2 = ROS (rosuvastatin)
15	Retinopathy (class)	Subject’s retinopathy status	Categorical	0 = No (91) 1 = Yes (42)

. The class label (retinopathy) was assigned when the subject had symptomatic cases with a history of laser or surgical therapy. The objective of our study was to classify whether a diabetic patient will develop diabetic retinopathy (DR) in the future.

3.2. Design of Proposed Model

Figure 1 shows the proposed DNN model to predict DR diagnosis from several risk factors, based on a public DR dataset. Data pre-processing removed inappropriate and inconsistent data. During the pre-processing stage, data normalization was applied by rescaling real valued numeric attributes into [0, 1]. Missing values in the numeric and nominal attributes were replaced by mean and mode, respectively. Then RFE removed irrelevant features, and a DNN-based prediction was developed using the grid search algorithm to optimize the model hyperparameter and, hence, maximize DNN performance. Performance was evaluated by comparing the proposed and other best-practice machine learning models from previous studies.

We used stratified 10-fold cross-validation (CV), a variation of k-fold CV for the proposed and comparison machine learning models. In k-fold CV, the dataset is split into k subsets of equal size and the instances for each subset or fold are randomly selected. Each subset, in turn, is used for testing and the remainder for the training set. The model is evaluated k times such that each subset is used once as the test set. However, in stratified k-fold cross-validation, each subset is stratified so that they contain approximately the same proportion of class labels as the original dataset. By this procedure, the variance among the estimates are reduced and the average error estimate is reliable. Furthermore, our dataset is imbalanced, with 32% of subjects classified as DR patients. A previous study demonstrated that stratified k-fold CV is generally considered superior to regular CV, particularly for unbalanced datasets [41]. Figure 2 shows the overview of the model validation based on CV and stratified CV applied to the two-class dataset.

3.3. Recursive Feature Elimination (RFE)

Feature selection removes redundant and irrelevant features to improve machine learning quality and efficiency. This study applied RFE with SVM kernels, i.e., linear function, to evaluate feature significances for DR dataset [30]. SVM-RFE works first by training the dataset with the SVM classifier. Next, the ranking weights for all features are computed. Finally, the feature with smallest weight is deleted. This process is repeated until no features are left, with later eliminated features having higher ranks. The bottom ranked ones are the least informative and removed in the first iteration. Thus, irrelevant features are gradually eliminated and important features retained for classification. The process is summarized in Algorithm 1.

Algorithm 1. SVM-RFE pseudocode
Input:	$X_0={\left(x_1,x_2,\dots ,x_m\right)}^{\mathrm{T}}$
	$y={\left(y_1,y_2,\dots ,y_m\right)}^{\mathrm{T}}$
	$s=\left[1,2,\dots ,n\right]$
	$r=[]$
Output:r
whiles is not empty do
	$X=X_0\left(:,s\right)$
	$\alpha =SVM\_train\left(X,y\right)$
	$w={\displaystyle {\displaystyle \sum }_k}{\alpha }_k{y}_k{x}_k$
	$c_i={\left(w_i\right)}^2$
	$f=argmin\left(c\right)$
	$r=\left[s\left(f\right),r\right]$
	$s=\mathrm{s}\left(1:f-1,f+1:\mathrm{length}\left(s\right)\right)$
end while
returnr

The SVM-RFE algorithm can be divided into three steps, which are input, calculation of the weight of each feature, and removing the lowest ranked feature. In Algorithm 1, during the input stage, $X_0$ is defined as the training sample, $y$ is class labels, $s$ is subset of remaining features, and $r$ is feature sorted list. In the next process, the weight calculation of each feature is conducted where the algorithm repeats the process until the list of $s$ is empty. The new training sample $X$ is defined according to the remaining features $s$. The set of paired inputs and outputs is used by training the classifier $\alpha$. The calculation of the weight vector $w$ and ranking criteria $c_i$ is conducted at this stage. When the lowest ranking feature $f$ was obtained, the feature sorted list $r$ can be updated. At the last stage, the feature with the smallest ranking criterion is removed and $s$ is updated. The users can stop the iteration when the list of s is not empty, so that desired number of features can be obtained.

In Algorithm 1, SVM_train (linear SVM) is utilized to learn from the set of paired inputs $X$ and outputs $y.$ The linear SVM classifies training data by mapping the original data onto a high dimensional feature space and finding the linear optimal hyperplane to separate instances of each class from the others [42]; for the case of separating training vectors belonging to two linearly separable classes:

$$\left(x_i,y_i\right),x_i\in R^n,y_i\in \left\{+1,-1\right\},i=1,\dots ,n$$

(1)

where $x_i$ is a real valued n-dimensional input vector and $y_i$ is the class label associated with the training vector. The separating hyperplane is determined by an orthogonal vector $w$ and bias b, which identify points that satisfy

$$w.x+b=0$$

(2)

Thus, the classification mechanism for linear SVM can be expressed as

$$ma{x}_{\alpha }\left[{\sum }_{i=1}^n{\alpha }_i-\frac{1}{2}{\sum }_{i,j=1}^n{\alpha }_i{\alpha }_j{y}_i{y}_j\left(x_i.x_j\right)\right]$$

(3)

with constraints

$${\sum }_{i=1}^n{\alpha }_i{y}_j=0,0\le {\mathsf{\alpha }}_i\le C,i=1,2,\dots .,n$$

(4)

where $\alpha$ is the parameter vector for the classifier hyperplane, and C is a penalty parameter to control the number of misclassifications.

Figure 3 shows the attribute ranking for the DR dataset. We investigated the impact of the top k features for DNN accuracy. The SVM-RFE was executed to remove irrelevant features, and important features (k) are utilized as the input for the DNN. This process is repeated for all possible k features. Finally, we found that the first (top) 13 DR features is the optimal number of features that can maximize DNN accuracy. Hence only systolic blood pressure (Sys BP) was removed from the DR dataset. Section 4.2 discusses optimizing the number of features for the dataset.

We also compared the performance of SVM-RFE with other feature selection methods on improving DNN accuracy, such as chi-squared, ANOVA, and extra trees. The result of feature selection execution for the dataset as shown in

Table 2. Feature selection results for the diabetes retinopathy (DR) dataset.

No	Attribute	Feature Selection Model
		RFE (Rank)	Chi-Squared (Score)	ANOVA (F-Value)	Extra Trees (Gini Importance)
1	BMI	10	0.225	2.352	0.056
2	DM duration	1	5.474	49.028	0.161
3	A1c	5	1.054	4.780	0.081
4	Age	4	1.352	24.473	0.098
5	FBS	2	0.970	12.349	0.088
6	LDL	9	0.520	6.207	0.059
7	HDL	3	0.571	8.726	0.077
8	TG	12	0.643	6.419	0.077
9	Sys BP	14	1.992	18.519	0.083
10	Dias BP	7	1.734	18.738	0.070
11	Sex	8	2.669	4.641	0.045
12	DM type	13	0.127	1.870	0.009
13	DM treat	11	0.779	2.889	0.061
14	Statin	6	0.064	0.176	0.031

. The feature selection methods showed different results in terms of extracting the most important features. A higher value of score, f-value, and gini importance indicate the importance of the features. The impact of a different feature selection on DNN accuracy is presented in Section 4.2.

The proposed SVM-RFE generated the top five features, i.e., DM duration, FBS, HDL, Age, and A1c as significant risk factors for retinopathy. Other feature selection methods, such as chi-squared, ANOVA, and extra trees, generated a different subset of important features. In chi-squared, the attributes DM duration, Sex, Sys BP, Dias BP, and Age were identified as the top five features. Furthermore, the attributes DM duration, Age, Dias BP, Sys BP, and FBS were identified as the top five features in ANOVA, while in extra trees, the result is similar to ANOVA except that A1C was included in the top five features instead of the attribute Dias BP. These top five features generated by the feature selection methods can be utilized as the input features for the deep neural network to improve classification accuracy.

Finally, the SVM-RFE identified the attribute Sys BP as an irrelevant feature, while the attribute Statin was recognized by chi-squared and ANOVA as a less important feature. The attribute DM type was discovered by extra trees as an unimportant feature; therefore, these irrelevant features generated by feature selection must be removed as input for the classifier. Excluding these irrelevant features is expected to improve DNN performance. A more detailed discussion regarding our significant risk factors for retinopathy and a comparison with the results from previous studies are presented in Section 4.3.

3.4. Proposed Deep Neural Network

We employed a DNN model to predict DR from the risk factors dataset. A DNN is an ANN class with multiple hidden layers between input and output layers and has recently become highly successful for classification. A DNN is fully connected; hence, each unit receives connections from all units in the previous layer. Thus, each unit has its own bias and a weight for every pair of units in two consecutive layers. The net input was calculated by multiplying each input with its corresponding weight and then summing. Each unit in the hidden layer took the net input and applied an activation function. Thus, network computation with three hidden layers can be expressed as

$$h_i^{\left(1\right)}={\phi }^{\left(1\right)}\left({\sum }_j{w}_{ij}^{\left(1\right)}{x}_j+b_i^{\left(1\right)}\right)$$

(5)

$$h_i^{\left(2\right)}={\phi }^{\left(2\right)}\left({\sum }_j{w}_{ij}^{\left(2\right)}{h}_j^{\left(1\right)}+b_i^{\left(2\right)}\right)$$

(6)

$$h_i^{\left(3\right)}={\phi }^{\left(3\right)}\left({\sum }_j{w}_{ij}^{\left(3\right)}{h}_j^{\left(2\right)}+b_i^{\left(3\right)}\right)$$

(7)

and

$$y_i={\phi }^{\left(4\right)}\left({\sum }_j{w}_{ij}^{\left(4\right)}{h}_j^{\left(3\right)}+b_i^{\left(4\right)}\right)$$

(8)

where $x_j$ is the input units, w is weight, b is bias, y is output units, $h_i^{\left(l\right)}$ is units in the lth hidden layer, and $\phi$ is the activation function. In our study, we used and evaluated several activation functions, such as sigmoid, hyperbolic tangent, and rectified linear unit (ReLU), which are presented in detail in Equations (9)–(11), respectively.

$$\phi \left(v\right)=\frac{1}{1+e^{-v}}$$

(9)

$$\phi \left(v\right)=\frac{e^{2v}-1}{e^{2v}+1}$$

(10)

$$\phi \left(v\right)=max\left\{v,0\right\}$$

(11)

The DNNs were trained using the back propagation (BP) algorithm [43], which compares the prediction result with the target value and modifies each training tuple’s weights to minimize error between prediction and target values. In measuring a good prediction model to predict the expected outcome, a loss function is required. Our study focused on binary classification, where the number of classes is two. The cross-entropy loss function can be calculated as

$$Loss=-{\sum }_i\left(y_i^{\prime }log\left(y_i\right)+\left(1-y_i^{\prime }\right)log\left(1-y_i\right)\right)$$

(12)

where $y_i^{\prime }$ is true probability and $y_i$ is predicted probability value. This process was iterated to produce optimal weights, providing optimal predictions for the test data. Figure 4 shows the proposed DNN model to predict DR from individuals’ risk factors.

We utilized the grid search algorithm [44] to automatically select the best parameters for the proposed deep neural network (DNN) model. We then applied a grid search for the training set and measured cross-validation to obtain the best prediction model, as shown in Figure 5. The objective for this method was to select the best hyperparameter for the proposed DNN model to achieve the highest accuracy for DR. We found a five-hidden-layers network with different neurons each, ReLU as activation function, and SGD as weight optimization were the best parameters for the DNN.

Table 3. Optimized hyperparameter using grid search.

Hyperparameter	Optimized Value
Hidden layer size	100, 64, 128, 64, 32
Activation function	ReLU
Alpha	0.0001
Initial learning rate	0.01
Maximum iteration	500
Optimization algorithm	SGD

Notes: ReLU = rectified linear unit, SGD = stochastic gradient descent.

shows optimized hyperparameter details for the proposed DNN for DR. Finally, we applied these parameters to the proposed DNN model.

3.5. Experimental Setup

Machine-learning models were applied to distinguish DR from the public dataset. Classification and feature selection methods were implemented in Python V3.7.3, utilizing the Scikit-learn V0.22.2 library [45]. We used Scikit-learn default parameters to simplify implementing other models. We performed the experiments on an Intel Core i5-4590 computer with 8 GB RAM running Windows 7 64 bit. Average performance metrics, such as accuracy (%), precision (%), sensitivity or recall (%), specificity (%), F1 score (%), and AUC were obtained by conducting 10 runs of stratified 10-fold CV.

4. Results and Discussion

This section considers the proposed DNN model performance and feature selection model impacts and discusses DR risk factors. We also verified a good generalization capability by applying our results to other public datasets (nephropathy and hypertension–diabetes).

4.1. Prediction Model Performances

We applied the proposed DNN with RFE model to predict DR using known risk factors and compared the outcomes with several current best-practice data-driven models that have wide acceptance and a proven track record for accuracy and efficiency: k-nearest neighbor (KNN), C4.5 decision tree (DT), support vector machine (SVM), naïve Bayes (NB), and random forest (RF).

Table 4. Performance metrics for diabetes retinopathy prediction.

Method	Accuracy	Precision	Sensitivity (Recall)	Specificity	F1	AUC
KNN	77.418	79.714	49.500	69.806	56.492	0.698
DT	75.989	63.095	65.000	73.222	59.558	0.732
SVM	78.846	75.214	51.500	71.361	56.333	0.714
NB	73.022	54.970	66.000	70.889	56.939	0.709
RF	78.352	62.500	39.000	67.889	45.944	0.679
Proposed model (DNN + RFE)	82.033	72.937	76.000	80.389	71.820	0.804

compares model performance metrics, averaged from 10 runs of stratified 10-fold CV. The proposed DNN with RFE model was superior to the traditional models with respect to accuracy, sensitivity, specificity, F1 score, and AUC, achieving 82.033%, 76.000%, 80.389%, 71.820%, and 0.804, respectively. In terms of detecting the positive cases, the KNN model achieved the highest precision (79.714%); however, our proposed model generated the highest recall (76.000%). The precision is the ratio of correctly predicted positive cases to the total predicted positive cases. On other hand, the recall indicates the accuracy of a model in predicting the positive cases for the cases where the actual condition is positive. Therefore, to identify prediction of positive cases (retinopathy) accurately, the model should focus more on recall rather than precision. In the medical area, it is common to focus more on sensitivity (recall) and specificity to evaluate medical tests [46]. Furthermore, it is also important to detect positive cases accurately, since when the model fails to detect the retinopathy, it will lead to blindness in such patients. Finally, the proposed model achieved 5.308% accuracy improvement compared with the current best-practice DR models.

Accuracy rate is the most common metric for classifier performance. However, class distribution must be considered for unbalanced datasets using specific classifier metrics. ROC is a useful tool to provide evaluation criteria for unbalanced datasets [47]. The ROC curve contrasts false positive and false negative outcomes, as shown in Figure 6, where AUC indicates overall classification performance [48], with the best model having AUC ≈ 1. Figure 6 shows ROC curves analysis for the proposed and other considered classification models for DR dataset. The proposed model achieved the highest AUC = 0.80.

Thus, the proposed model achieved significantly improved metrics compared with current best practice classification methods. Specific impacts for RFE and other feature selection methods on performance accuracy are presented in Section 4.2.

4.2. Feature Selection Impacts

The optimal number of features is required to implement RFE. Therefore, we investigated the impact of top k features on DNN accuracy. The full dataset included 14 features, and RFE expected to remove irrelevant features. Figure 7a shows the impact of the best k features as defined by RFE on DNN model accuracy. Optimal, k = 13 for DR dataset, and maximum DNN model accuracy are achieved when including only these defined optimal features. The result showed that removing high number of features leads to the reduction of accuracy; therefore, the highest accuracy can be achieved by removing a small number of features.

Based on the strategy of searching, the feature selection can be categorized into three methods, such as wrapper, filter, and embedded methods [49,50]. Filter methods measure the relevance of features by their correlation with the target variable, while wrapper methods utilize a learning machine to measure the usefulness of a subset of features according to their predictive power. Embedded methods perform feature selection in the process of training based on specific learning machines. In our study, we used SVM-RFE as an application of wrapper methods, while for filter methods, ANOVA and chi-squared were utilized. For embedded methods, we utilized the extra trees algorithm to extract relevant features [51]. Figure 7b shows the impact of the feature selection method on DNN model accuracy. The feature selection based on extra trees automatically selected the optimal number of features = 7. We also investigated and found that the optimal number of features for ANOVA and chi-squared are same (13 features). RFE generated superior accuracy, up to 9.07% for DR dataset compared to outcomes without feature selection. However, the other considered feature selection methods performed poorly, with only slight accuracy improvements for the DR dataset using chi-squared, ANOVA, and extra trees. Thus, RFE was the best choice for the DNN, providing the maximum DR prediction accuracy.

4.3. Risk Factors and Previous Studies

Table 5. Comparison with previous studies for predicting diabetes retinopathy (DR).

Dataset	Population	Study	Method	Number of Features	Model Validation	Accuracy	AUC
DR	Iran	[13]	LR	9	-	-	0.704
	South Korea	[14]	Lasso	19	Holdout (67:33)	0.736	0.810
	United States	[15]	Ensemble RUSBoost	11	Holdout (80:20)	0.735	0.720
	Taiwan	[17]	SVM	10	Holdout (80:20)	0.795	0.839
	United States	[16]	ANN + SMOTE	8	Holdout (66:34)	-	0.754
	Iran	Current	DNN + RFE	13	Stratified 10-fold CV	0.820	0.804

Notes: DNN = deep neural network; RFE = recursive feature elimination; LR = logistic regression; SVM = support vector machine; ANN = artificial neural network; SMOTE = synthetic minority oversampling technique.

compares the proposed and previous models for DR. Previous prediction models with different input variables were applied to various populations, including Iran, South Korea, US, and Taiwan. To our best knowledge, no previous study considered DNN with RFE for DR prediction. The proposed DNN and RFE model achieved superior performance compared with all previous models in terms of model accuracy. However, SVM [17] and Lasso [14] achieved superior AUC for DR. Most of the previous studies used holdout for model validation, whereas we used stratified 10-fold CV to avoid overfitting.

Risk factors and their relative importance vary across the world and the previous studies showed that important predictors could be retrieved for DR. Hosseini et al. [13] used age, BMI, sex, diabetes duration, and HbA1c; whereas Oh et al. [14] identified fasting BG, triglyceride, low BMI, and insulin therapy as strong predictors. Other studies also identified insulin use [17] and A1C [16] as important risk factors. Our proposed study used RFE to select the best variables and identified the top five risk as diabetes duration, fasting BG, HDL, Age, and A1c; which are largely consistent with selected risk factors from previous studies.

Directly comparing these results is inappropriate, since they were derived from different datasets, pre-processing methods, and validation methods. Therefore, Table 5 should not be considered as strong evidence regarding model performance, but it provides a general comparison and allows discussions regarding the proposed model and previous approaches. We used a public dataset for the current study, which was limited to small populations in Iran. Benchmarking machine learning models will become somewhat fairer as other datasets become publicly available.

4.4. Another Diabetes Dataset

To evaluate the proposed prediction model robustness and generalization, we compared it with other machine learning models and datasets (diabetic nephropathy and hypertension–diabetes). Diabetic nephropathy (DN) is a serious kidney-related complication (kidney disease) relatively commonly developed by T1D and T2D patients. The DN dataset we employed was related to DR (Table 1), provided by Khodadadi et al. [40]. The dataset was gathered from 133 diabetic patients with 73 among them developing DN. Thus, we used the same 14 input features as in Table 1 but with a different output class, i.e., DN. The description of DN dataset can be seen in Appendix A (see

Table A1. Diabetic nephropathy (DN) dataset.

No	Attribute	Description	Type	Range
1	BMI	Subject’s body mass index	Numeric	18–41
2	DM duration	Subject’s diabetes duration (y)	Numeric	0–30
3	A1c	Subject’s average blood glucose level over the past 3 months (mg/dL)	Numeric	6.5–13.3
4	Age	Subject’s age (y)	Numeric	16–79
5	FBS	Subject’s fasting blood sugar level (mg/dL)	Numeric	80–510
6	LDL	Subject’s low-density lipoprotein level (mg/dL)	Numeric	36–267
7	HDL	Subject’s high-density lipoprotein level (mg/dL)	Numeric	20–62
8	TG	Subject’s triglyceride level (mg/dL)	Numeric	74–756
9	Sys BP	Subject’s systolic blood pressure (mmHg)	Numeric	105–180
10	Dias BP	Subject’s diastolic blood pressure (mmHg)	Numeric	60–120
11	Sex	Subject’s sex	Categorical	0 = Female 1 = Male
12	DM type	Subject’s diabetes type	Categorical	0 = T1D 1 = T2D
13	DM treat	Subject’s diabetes treatment	Categorical	0 = Both (Insulin and oral agent) 1 = Insulin 2 = Oral agent
14	Statin	Subject’s statin status (frequently used as part of diabetes care)	Categorical	0 = Ator (atorvastatin) 1 = No statin 2 = ROS (rosuvastatin)
15	Nephropathy (class)	Subject’s nephropathy status	Categorical	0 = No (60) 1 = Yes (73)

). The objective for this dataset was to classify whether the diabetic patient would develop DN.

We also gathered a dataset from the National Health Insurance Sharing Service (NHISS) Korea comprising applicant’s general health data 2013–2014 [52]. The original input variables were age group (BTH_G), systolic blood pressure (BP), diastolic BP, fasting BG, BMI, and sex. The dataset included four classes, where the subject was diagnosed to have hypertension, diabetes (T1D or T2D), hypertension and diabetes, or healthy (no diabetes or hypertension history). We converted this multiclass into a binary classification problem, transforming it into healthy or disease (hypertension, diabetes), and removed fasting BG, since this variable is closely related with diabetes diagnosis. We randomly selected 1000 individuals from approximately 1 million, and hence, the final dataset comprised 761 healthy and 239 disease (hypertension, diabetes) patients. The description of NHISS dataset can be seen in Appendix A (see

Table A2. National Health Insurance Sharing Service (NHISS) Korea dataset.

No	Attribute	Description	Type	Range
1	BTH_G	Age group of a subject	Categorical	0 = 20–24 1 = 25–26 2 = 27–28 … 26 = greater than 75
2	SBP	Subject’s systolic blood pressure (mmHg)	Numeric	84–190
3	DBP	Subject’s diastolic blood pressure (mmHg)	Numeric	50–120
4	BMI	Subject’s body mass index	Numeric	15.6–39.9
5	SEX	Subject’s sex	Categorical	0 = Male 1 = Female
6	DIS (class)	Subject’s disease (hypertension, diabetes) status	Categorical	0 = No (761) 1 = Yes (239)

). The objective for this dataset was to classify whether the subject would develop disease (i.e., hypertension or diabetes).

Table 6. Proposed prediction model performance compared with other models for public datasets.

Method	DN		NHISS
	Accuracy	AUC	Accuracy	AUC
KNN	81.813	0.814	80.600	0.690
DT	81.978	0.817	80.900	0.679
SVM	83.297	0.834	80.700	0.665
NB	67.527	0.649	79.600	0.701
RF	82.747	0.825	80.775	0.664
Proposed DNN + RFE	84.121	0.839	81.600	0.702

Notes: DN = diabetes nephropathy dataset, NHISS = hypertension–diabetes dataset from NHISS, Korea. RF = random forest; DNN = deep neural network; RFE = recursive feature elimination; KNN = k-nearest neighbor; SVM = support vector machine; DT = decision tree; NB = naïve Bayes.

compares classification performance for the proposed DNN + RFE model with other machine learning models (KNN, DT, SVM, NB, and RF). Average accuracy and AUC were calculated from 10 runs of stratified 10-fold CV. The proposed model achieved superior accuracy and AUC for both datasets: 84.121% and 0.839 for the DN and 81.600% and 0.702 for the NHISS datasets, respectively.

Applying RFE, we selected 13 features for the DN dataset and 3 for the NHISS dataset (BTH_G, Systolic BP, and BMI), and we used grid search to optimize the DNN hyperparameter. Thus, we found the optimal DNN design for the DN dataset to be five hidden layers (100, 64, 128, 64, 32), ReLU activation function, and a maximum of 100 iterations. For NHISS dataset, DNN with five hidden layers (100, 64, 128, 64, 32) and tanh activation function has achieved highest accuracy. The best optimization algorithms were identified as LBFGS (limited-memory Broyden–Fletcher–Goldfarb–Shanno) and Adam for the DN and NHISS datasets, respectively. These additional experiments confirmed the proposed model robustness toward different healthcare domains.

5. Conclusions and Future Work

The current study combined DNN with RFE to predict DR. The model is expected to help individuals foresee DR danger based on risk factors during initial disease phases. A public dataset incorporating DR risk factors was utilized, and the proposed model performance was compared with previous best-practice KNN, DT, SVM, NB, and RF models. The proposed model outperformed conventional classification models and most other previous models, achieving accuracy = 82.033%.

We applied RFE to identify significant features from all datasets, i.e., extracting the highest DR risk factors. Combining RFE and DNN improved the prediction accuracy as compared with all other considered feature selection methods (chi-squared, ANOVA, and extra trees). The proposed DNN + RFE model improved the accuracy (9.07%) compared with DNN without feature selection. Thus, machine learning combined with feature selection can effectively detect DR. This offers increased cost-effectiveness for health care systems, where decision support based on the proposed prediction model could provide decision opinions. We hope this study will help reduce the DR risk for diabetic patients, which is the major cause of blindness.

The dataset used here was from a relatively small and quite specific population; hence, the prediction model outcomes cannot not be simply generalized for broader application. Similarly, the identified important risk factors might not be appropriate for other populations. Thus, as it stands, the proposed model would be unsuitable for clinical trials due to dataset limitations. Therefore, the proposed approach should be extended to other clinical datasets and compared widely with other prediction and feature selection models. Once model validation is extended to broader datasets, other risk factors affecting DR could be identified.