Blended Multi-Modal Deep ConvNet Features for Diabetic Retinopathy Severity Prediction

Abstract

Diabetic Retinopathy (DR) is one of the major causes of visual impairment and blindness across the world. It is usually found in patients who suffer from diabetes for a long period. The major focus of this work is to derive optimal representation of retinal images that further helps to improve the performance of DR recognition models. To extract optimal representation, features extracted from multiple pre-trained ConvNet models are blended using proposed multi-modal fusion module. These final representations are used to train a Deep Neural Network (DNN) used for DR identification and severity level prediction. As each ConvNet extracts different features, fusing them using 1D pooling and cross pooling leads to better representation than using features extracted from a single ConvNet. Experimental studies on benchmark Kaggle APTOS 2019 contest dataset reveals that the model trained on proposed blended feature representations is superior to the existing methods. In addition, we notice that cross average pooling based fusion of features from Xception and VGG16 is the most appropriate for DR recognition. With the proposed model, we achieve an accuracy of 97.41%, and a kappa statistic of 94.82 for DR identification and an accuracy of 81.7% and a kappa statistic of 71.1% for severity level prediction. Another interesting observation is that DNN with dropout at input layer converges more quickly when trained using blended features, compared to the same model trained using uni-modal deep features.

1. Introduction

Diabetic Retinopathy (DR) is an adverse effect of Diabetes Mellitus (DM) [1] that leads to permanent blindness in humans. It is usually caused by the damage to blood vessels that provide nourishment to light-sensitive tissue called the retina. As per statistics [2], DR is the fifth leading cause for blindness across the globe. According to the World Health Organization (WHO), in 2013, around 382 million people were suffering from DR, and this number may rise to 592 million by 2025. It is possible to save many people from going blind if DR is identified in the early stages. Small lesions are formed in the eyes of DR-affected people and the type of lesions formed decides the level of severity of DR. Figure 1a shows types of lesions that include Micro Aneurysms (MA), exudates, hemorrhages, cotton wool spots, and improperly grown blood vessels on the retina.

DR can be categorized into five different stages [3]: no DR (Class 0), mild DR (Class 1), moderate DR (Class 2), severe DR (Class 3), and proliferative DR (Class 4). Sample retinal images with different severity levels of DR are shown in Figure 1b. Mild DR is the early stage during which the formation of Micro Aneurysms (MA) can be observed. As the disease progresses to moderate stage, swelling of blood vessels can be found, which leads to blurred vision. During the later non-proliferative DR (NPDR) stage, abnormal growth of blood vessels can be noticed. This stage is severe due to the blockage of a large number of blood vessels. Proliferative DR (PDR) is the advanced stage of DR; during this stage, retinal detachment along with large retinal break can be observed that leads to complete vision loss [4].

In traditional DR diagnosis approaches, manual grading of the retinal scan is required to identify the presence or absence of retinopathy. If DR is confirmed as positive, further diagnosis is recommended to identify severity level of the disease. This kind of diagnosis is quite expensive and time consuming as it demands human expertise. If DR identification is automated, then diagnosis of the disease becomes affordable to many people. In the recent past, several machine learning tools have been introduced to address this problem.

Early approaches to DR identification, where the presence or absence of DR is revealed, focuses on spotting the Hard Exudates (HEs). A dynamic threshold based Support Vector Machine (SVM) is used to segment HE in the retinal images [5]. Fuzzy C-means is used to detect HE and SVM is used to identify severity level of the disease to make the system more sophisticated [6]. SVM-based classifiers are adapted to find cotton wool spots in the retinal images.

With the introduction of deep learning, the focus of researchers has shifted from spotting HEs to MAs. A two-step CNN has been introduced to segment MAs in the given retinal scans [7]. Another CNN architecture that is trained using selective sampling approach is proposed to detect hemorrhages [8]. A max-out activation is introduced to improve the performance of a DNN model, for which DR is used as an application to find MA [9]. Recently, a bounding box based approach has been introduced to identify the region of interest in the retinal images [10]. Although many methods are available in the literature, they are either sub-optimal or complex. Hence, there is a need for a solution that is simple and robust.

The objective of this work is to design a simple and robust deep learning-based approach to recognize DR from the given retinal images. The major focus of this work is to obtain a better feature representation of the retinal images, which ultimately leads to a better model. To accomplish this, we propose uni- and multi-modal approaches. Initially, for the given retinal images, deep features are extracted from different pre-trained ConvNets such as VGG16, NASNet, Xception, and Inception ResNetV2. In the uni-modal approach, features extracted from a single pre-trained ConvNet gives the final feature representation. In the multi-modal approach, our idea is to blend the deep features extracted from multiple ConvNets to get the final feature representation. We propose different pooling-based approaches to blend multiple deep features. To check the efficiency of our feature representation, a Deep Neural Network (DNN) architecture is proposed for identification of DR (Task 1) and to recognize severity level of DR (Task 2). We observe that, in the multi-modal approach, blending deep features from Xception and Inception ResNet V2 outperforms others in both tasks. Another interesting observation is that there is a drop in the number of false positives, which is most desirable. Experimental studies on the benchmark APTOS 2019 dataset revealed that our blended feature representations trained using DNN model give superior performance compared to the existing methods.

The following are the major contributions of the proposed work:

• Effectiveness of the uni-modal feature representation is verified.
• A blended multi-modal feature representation approach is introduced
• Different pool-based approaches are proposed to blend deep features.
• A DNN architecture with dropout at the input layer is proposed to test the efficiency of the proposed uni-modal and blended multi-modal feature representations.
• APTOS 2019 benchmark dataset was used to compare the performance of the proposed approach with existing models.

2. Related Work

Recently, machine learning models are very popular to solve various problems such as image classification [11], text processing [12], real-time fault diagnosis [13], and healthcare tasks [14,15]. It is very common to use ML algorithms to address disease prediction [16,17,18].

In this section, we report various conventional models available in the literature for the task of DR recognition. In [19], an easy to remember scientific approach is introduced for DR severity identification. In [20], the authors presented a hybrid classifier by using both GMM and SVM as an ensemble model to improve the accuracy of the model. The same approach has been modified by augmenting the feature set with shape, intensity, and statistics of the affected region [21]. A random forest-based approach is proposed in [22,23] and segmentation-based approaches are proposed in [24]. In [25], a genetic algorithm-based feature extraction method is introduced. Different shallow classifiers such as GMM, KNN, SVM, and AdaBoost have been analyzed [26] to differentiate lesions from non-lesions. A hybrid feature extraction based approach is used in [27].

In the next few lines, deep learning models available in the literature for the task of DR severity identification are introduced. A large dataset consisting of 1,28,175 retinal images is used and trained using deep CNN. In [28], data augmentation method is used to generate the data on CNN architecture. Fuzzy models are used in [29]; a hybrid model that is designed based on fuzzy logic, Hough Transform, and numerous extraction methods are implemented as part of their system. A combination of fuzzy C-means and deep CNN architectures are used in [30]. A Siamese convolutional neural network is used in [31] to detect diabetic retinopathy.

With the introduction of deep learning models, focus has shifted to deep feature-based models. In [32], used features extracted from different layers of pre-trained ConvNets such as VGG19 and further applied PCA and SVD on those features for dimension reduction [33] to avoid overfitting. In the case of former models, the model is not robust, and, in the latter case, the models are robust, but large datasets are needed to train them. A PCA based fire-fly model [34] along with deep neural network is used for DR detection in [35], and the UCI repository is used for the experiments.

The performance of any ML algorithm is subject to the features extracted from the given data. Conventional ML models need a separate algorithm (GIST, HOG, and SIFT) for feature learning and give a global or local representation of the images and the features. Features extracted in this process are known as handcrafted features. Until the entry of deep learning models, these handcrafted features were dominant and being widely used for feature extraction.

Deep ConvNets for Feature Extraction and Transfer Learning

Deep learning models [36,37,38] learn the essential characteristics of the input images. This exceptional capability of the deep models make them representation models, as they can represent the data efficiently and reduce the use of the additional feature extraction phase where features are handcrafted. Deeper layers of the CNN models can represent the entire given input more efficiently than the early layers.

The downside of the deep learning models is that they need enormous amounts of data for training, which is usually scarce for most real-time applications. This problem can be addressed by the introduction of transfer learning, where the knowledge gained by a deep learning model can be transferred to other models. To achieve this, deep pre-trained CNN models such as VGG16 and ResNet152 are available for transfer learning. Pre-trained models are the models that are trained on large amounts of data, and the weights updated during the training of the complex model can be applied to similar kinds of tasks.

There are different types of pre-trained models which are trained on large scale datasets, such as ImageNet that consists of more than a million images. Popular pre-trained deep CNN models such as VGG16, VGG19, ResNet152, InceptionV3, Xception, NASNet, Inception ResNet V2 and DarkNet are briefly described below:

• Visual Geometric Group (VGG 16): VGG16 is a deep ConvNet trained on 14 million images belonging to 1000 different classes and topped the leader board in ILSVR (ImageNet) challenge. In this architecture, 2 × 2 filters are used with stride 1 for convolution operation, and 2 × 2 filters with stride two and the same padding are used for max-pooling operation across the network. At the end of architecture, two fully connected dense layers of 4096 neurons are connected followed by soft-max layer.
• Neural Architecture Search Network (NASNet): This is a special kind of deep CNN which searches for a better architectural building block on small datasets such as CIFAR10 and transfers it to larger datasets such as ImageNet. It has a better regularization mechanism called scheduled drop path, which significantly improves generalization.
• Xception: Xception is another deep ConvNet architecture that supports depth-wise separable convolution operations and outperformed ResNet and InceptionV3 in the ILSVR challenge.
• Inception ResNetV2: This is popularly known as InceptionV4, as it combines two different architectures: InceptionV3 and ResNet152. It has both inception and residual connections, which boost the performance of the model.

Deep neural networks give excellent performance only when trained with extensive data. If the data used to train are not sufficient, then the DNN models tend to overfit. Deep, convolutional neural networks are introduced in [39] for the task of scalable image recognition. Xception, a deep CNN, is developed using depth-wise separable convolutions to improve the performance [40]. A flexible architecture is defined in [41], which can search for a better convolutional cell with better regularization mechanism. All these models are trained on ImageNet Dataset for ILSVR challenge.

Our objective is to create a robust and efficient model to recognize DR with limited datasets and with limited computational resources. To achieve our objective of creating a robust model with small datasets, we seek the help of transfer learning and use various pre-trained ConvNets to extract deep features. We use the knowledge of these models to extract the most prominent features of color fundus images. A deep neural network with dropout introduced at early layers is trained to detect and classify the severity levels of diabetic retinopathy. As we introduced dropout at the input layer, deep neural network is immune to overfitting.

3. Proposed Methodology

In this work, our objective is to develop a robust and efficient model to automate DR diagnosis. We focus on the extraction of deep features that are most descriptive and discriminate, which ultimately improves the performance of DR recognition. To get an optimal representation, features are extracted from multiple pre-trained CNN architectures and are blended using pooling-based approaches. These final representations are used to train a deep neural network with a dropout at the input layer. The proposed model has three different modules: feature extraction, model training, and evaluation module.

3.1. Feature Extraction

Performance of any machine learning model is highly influenced by the feature representations and the same is applicable to models used for DR recognition. With this motivation, we propose two different approaches (uni-modal and multi-modal) to extract optimal features from the given retinal images.

In the proposed work, initial representations of the retinal images are obtained from the pre-trained VGG16, NASNet, Xception Net, and Inception ResNetV2. As each of the pre-trained models expects input images of varying sizes, the given retinal images are reshaped according to the input dimensions accepted by these models. For example, when VGG16 is used, images are reshaped to 224 × 224 × 3. These reshaped retinal images are fed to the pre-trained models after removing the soft-max layer and freezing the rest of the layers. Activation outputs from the penultimate layers form the basis for the proposed feature extraction module. For each retinal image, deep features are extracted from the pre-trained ConvNets and the details are as follows:

• Each of the first (fc1) and second (fc2) fully connected layers of VGG16 produces a feature vector of 4096 dimensions.
• The final global average pooling layer of NASNet, Xception, and InceptionResNetV2 gives feature vectors of size 4032, 2048, and 1536, respectively.

Figure 2 gives the architectural details of the pre-trained VGG16, NASNet, Xception, and InceptionResNetV2 and pointers are marked at the feature extraction layers. These features form the input to the proposed uni-modal and blended multi-modal approaches to obtain the optimal feature representations of the retinal images.

3.2. Uni-Modal Deep Feature Extraction

In this approach, deep features are extracted from the final layers of one of the pre-trained ConvNets (VGG16, NASNet, Xception, or ResNet V2) to get the global representation of the retinal images. These deep features are fed to classification models for DR identification and recognition. We propose to use DNN architecture with a dropout at the input layer for DR identification and classification. Figure 3 gives the details of different stages involved in DR recognition process that uses uni-modal deep ConvNet features.

3.3. Blended (Multi-Modal) Deep Feature Extraction

Unlike uni-modal approaches, multi-modal approaches use deep features extracted from multiple ConvNets and are blended using fusion techniques. The features obtained from different pre-trained models provide a different representation of the retinal images as they follow different architectures and are trained on different datasets. A stronger representation can be obtained by blending features from multiple ConvNets, as features of one ConvNet complement the features from other ConvNets involved in the process.

We propose various pooling approaches to fuse the deep features extracted from multiple pre-trained ConvNets. The final blended deep features provide better descriptive and discriminate representation of the retinal images. These blended features are fed to the classification models for DR identification or severity recognition. Figure 4 gives the details of different stages involved in DR recognition process that uses blended multi-modal deep ConvNet features. The proposed blended multi-modal feature extraction module uses features from both the fully connected layers of VGG16 (fc1 and fc2) and global average poling layer of Xception as input. The rationale behind choosing features of VGG16 and Xception over the others is two fold. In VGG16, each feature map of the final convolution block learns the presence of different lesions from the retinal images. Xception Net learns correlations across the 2D space; as a result, each feature map provides the comprehensive representation of the entire retinal scan. Figure 5 visualizes the feature maps obtained from the final convolution blocks of VGG16 and Xception models when a retinal image is passed to these models.

Approaches to Blend Deep Features from Multiple ConvNets

In this work, two different pooling-based approaches (1D pooling and cross pooling) are proposed to fuse multi-modal deep features that are extracted from VGG16 (fc1 and fc2) and Xception. 1D pooling is used to select prominent local features from each region of VGG16, whereas cross pooling allows aggregating the prominent features obtained by 1D pooling with global representation of Xception.

1D pooling-based fusion takes one feature vector U as input and produces another feature vector $\hat{U}$, where $U\in R^{d1}$, $\hat{U}\in R^{d2}$, and $d_2\le d_1$. $\hat{U}$ is a reduced representation of U, where $U=\{u_1,u_2\dots u_{d1}\}$ and $\hat{U}=\{{\hat{u}}_1,{\hat{u}}_2\dots {\hat{u}}_{d2}\}$. Each feature element ${\hat{u}}_i$, of the output vector $\hat{U}$, is computed using one of the following three approaches:

$$1\mathrm{D}\mathrm{Max}\mathrm{pooling}:{\hat{u}}_i=max(u_{i\ast 2},u_{i\ast 2+1})\forall i\in \{1,2\dots d_2\}$$

(1)

$$1\mathrm{D}\mathrm{Min}\mathrm{pooling}:{\hat{u}}_i=min(u_{i\ast 2},u_{i\ast 2+1});\forall i\in \{1,2\dots d_2\}$$

(2)

$$1\mathrm{D}\mathrm{Average}\mathrm{pooling}:{\hat{u}}_i=mean(u_{i\ast 2},u_{i\ast 2+1});\forall i\in \{1,2\dots d_2\}$$

(3)

$$1\mathrm{D}\mathrm{Sum}\mathrm{pooling}:{\hat{u}}_i=u_{i\ast 2}+u_{i\ast 2+1};\forall i\in \{1,2\dots d_2\}$$

(4)

In cross pooling-based feature fusion, two different feature vectors X and Y are passed as input and another feature vector Z is produced, where $X,Y,Z\in R^d$. Each feature element $z_i$, of the output vector Z, is computed using one of the following three approaches:

$$\mathrm{Cross}\mathrm{Max}\mathrm{pooling}:z_i=max(x_i,y_i)\forall i\in \{1,2\dots d\}$$

(5)

$$\mathrm{Cross}\mathrm{Min}\mathrm{pooling}:z_i=min(x_i,y_i)\forall i\in \{1,2\dots d\}$$

(6)

$$\mathrm{Cross}\mathrm{Average}\mathrm{pooling}:z_i=mean(x_i,y_{i+1})\forall i\in \{1,2\dots d\}$$

(7)

$$\mathrm{Cross}\mathrm{Sum}\mathrm{pooling}:y_i=x_i+y_i\forall i\in \{1,2\dots d\}$$

(8)

1D pooling is applied independently on features extracted from fc1 and fc2 layers of VGG16. Then, the cross pooling approach is applied on the resultant pooled features. This feature vector is merged with the features extracted from the Xception using cross pooling. Fusion module produces deep blended features, which are used to train the proposed DNN model. Figure 6 shows the proposed architecture of the deep feature fusion approach used to blend features from different ConvNets. As the final feature vector is a blended version of the local and global representations of the retinal images, it provides strong features. Algorithm 1 gives the sequence of steps involved in the blended multi-modal feature fusion-based DR recognition.

Algorithm 1: Blended multi-modal deep feature fusion based DR recognition task.

Input: Let $D^{Tr}$ and $D^{Tst}$ be the train and test datasets of fundus images, respectively, where     $D^{Tr}=\{{(x_i,y_i)}_{i=1}^{N_{Tr}}\}$ and $D^{Tst}=\{{(x_i)}_{i=1}^{N_{Tst}}\}$. $x_i$ represents ith color fundus image in the     dataset and $y_i$ is the severity level of DR associated with $x_i$. In the case of DR     identification task, $y_i\in \{0,1\}$, whereas, in the case of DR severity classification, task     $y_i\in \{0,1,2,3\}$.  Output: $y_i$ for each $x_i\in D^{Tst}$  Step 1: Preprocess each image $x_i$ in the dataset.  Step 2: Feature Extraction   For each preprocessed image $x_i$ three different features ($V_i,U_i,W_i$) are extracted.    $V_i\leftarrow$ Features extracted from fc1 layer of VGG16    $U_i\leftarrow$ Features extracted from fc2 layer of VGG16    $W_i\leftarrow$ Features extracted from global avg pool layer of Xception     Where $V_i\in d1,U-i\in d2$ and $W_i\in d3$  Step 3: Deep feature fusion     Apply feature feature fusion on the deep features extracted from each image     ${\hat{V}}_i\leftarrow max(V_{i\ast 2},V_{i\ast 2+1)};\forall i\in \{1,2\dots d_1\}$ (1D max pooling)     ${\hat{U}}_i\leftarrow max(U_{i\ast 2},U_{i\ast 2+1)};\forall i\in \{1,2\dots d_2\}$ (1D max pooling)     ${\widehat{UV}}_i\leftarrow \frac{({\hat{V}}_i+{\hat{U}}_i)}{2};\forall i\in \{1,2\dots d_2\}$ (Average Cross pooling)     ${\hat{x}}_i\leftarrow \frac{({\widehat{\hat{UV}}}_i+W_i)}{2};\forall i\in \{1,2\dots d_3\}$ (Average Cross pooling)     ${\hat{x}}_i:$ blended feature vector corresponding to $x_i$  Step 4: Model Training   Training dataset is prepared using the blended features $D^{Tr}={\left\{({\hat{x}}_i,y_i)\right\}}_{i=1}^{N_{Tr}}$   Train a deep neural network (DNN) using $D^{Tr}$  Step 5: Model evaluation   Test dataset is prepared using the blended features $D^{Tst}={\left\{\left({\hat{x}}_i\right)\right\}}_{i=1}^{N_{Tst}}$   Evaluate the performance of $D^{Tst}$ using the DNN trained in Step 4

3.4. Model Training and Evaluation

During this phase, we trained the ML model with deep blended pre-trained features. We preferred to use Deep Neural Network (DNN) model for training. For DR identification task, as it is a simple binary classification task, a DNN with two hidden layers with 256 and 128 units, respectively, with ReLU activation was used.

For DR severity classification task, a DNN with three hidden layers with 512, 256, and 128 units, respectively, using ReLU activation was used. For both the DNNs with the input layer, we applied 0.2 dropout to avoid the model from overfitting. This helped the model to become robust. Figure 7 represents the architecture of proposed approach for model training and evaluation.

4. Experimental Results

In this section, we provide details of experimental studies that were carried out to understand the efficiency of the proposed blended multi-modal deep features representation.

4.1. Dataset Summary

For the experimental studies, the APTOS 2019 kaggle benchmark dataset available as part of the blindness detection challenge was used [42]. This is a large dataset of retinal images taken using fundus photography under a variety of imaging conditions. The images are graded manually on a scale of 0 to 4 (0, no DR; 1, mild; 2, moderate; 3, severe; and 4, proliferative DR) to indicate different severity levels.

Table 1. Dataset summary of APTOS 2019 dataset.

Severity Level	# Samples
Class 0 (Normal)	1805
Class 1 (Mild Stage)	370
Class 2 (Moderate Stage)	999
Class 3 (Severe Stage)	193
Class 4 (Proliferative Stage)	295
Total	3662

gives the number of retinal images available in the dataset under each level of severity. We can observe that the dataset has an imbalance with more normal images and very few images in Class 3. In all experiments, 80% of the data were used for training and the remaining 20% were used for validation.

4.2. Performance Measures

For the evaluation of the proposed model, we report different measures: accuracy, precision, recall, and F1 score. In addition, we used an additional metric called lappa statistic to compare an observed accuracy with an expected accuracy. Kappa Statistic is calculated as

$$\mathrm{Kappa}\mathrm{Score}=\frac{(\mathrm{Observed}\mathrm{Accuracy}-\mathrm{Expected}\mathrm{Accuracy})}{(1-\mathrm{Expected}\mathrm{Accuracy})}$$

Observed accuracy is defined as the number of samples that are correctly classified. Expected accuracy is defined as the accuracy that a classifier would be expected to achieve, which is directly related to the number of examples of each class, along with the number of examples that the predicted value satisfied with the correct label.

4.3. DR Identification and Severity Level Prediction

The experiments carried out in this work were divided into two different tasks. In Task 1, presence or absence of DR was identified, whereas, in Task 2, the severity level was predicted for the given retinal image.

4.3.1. Task 1—DR Identification

In this task, given the DR image of a diabetic patient, we need to check whether the person is affected by retinopathy or not. DR identification is a binary classification task, thus binary cross entropy loss was used to measure the loss, and Adam optimizer was used to optimize the objective function. The dataset contains images belonging to five different classes, as shown in Table 1, and is not suitable for binary classification task. Merging all the DR-affected images into a single class gives 1857 positively labeled images and the remaining 1805 normal images are labeled as negative.

4.3.2. Task 2—Severity Level Prediction

The objective of Task 1 is to identify the presence or absence of DR, given a retinal image. While treating the DR-affected patients, mere identification of DR would not be sufficient and understanding the level of severity would be helpful for better treatment. Hence, we treat severity level identification as a separate task that categorizes the given retinal image into one of the five severity levels. Categorical cross entropy loss was used to represent loss and Adam optimizer was used to optimize the objective function.

4.4. Experimental Studies to Show the Representative Nature of Uni-Modal Features for Task 1

This experiment was carried out to understand how efficiently retinal images are represented using uni-modal features that are directly obtained from single pre-trained ConvNet. Models such as VGG16, Xception, NASNET, and ResNetV2 were considered to extract uni-modal features. For classification, models such as Naïve Bayes classifier, logistic regression, decision tree, k-Nearest Neighborhood (KNN) classifier, Multi Layered Perceptron (MLP), Support Vector Machine (SVM), and Deep Neural Network (DNN) were used.

Table 2. Performance of ML algorithms on Task 1 using features from fc2 layer of VGG16.

Model	Accuracy	Precision	Recall	F1 Score	Kappa Statistic
Logistic Regression	97.13	97	97	97	94.27
KNN	95.36	96	95	95	90.73
Naive Bayes	77.08	82	77	76	54.45
Decision Tree	91.27	91	91	91	82.52
MLP	96.45	97	96	96	92.91
SVM (linear)	96.58	97	97	97	93.17
SVM (RBF)	96.86	97	97	97	93.73
DNN	97.32	98	98	98	94.63

and

Table 3. Performance of ML algorithms on Task 1 using features from Xception.

Model	Accuracy	Precision	Recall	F1 Score	Kappa Statistic
Logistic Regression	96.45	96	96	96	93
KNN	95.5	96	95	95	91
Naive Bayes	82.95	84	83	83	65.9
Decision Tree	87.59	88	88	88	75.17
MLP	96	96	96	96	91.89
SVM (linear)	96.18	96	96	96	92.36
SVM (RBF)	97.4	97	97	97	94.82
DNN	97.41	97	97	97	94.82

show the performance of DR identification task using different ML models when the retinal images are represented with the features extracted from the first fully connected layer (fc2) of VGG16 and Xception, respectively. Based on these results, we concluded that DNN outperforms the rest of the ML models irrespective of the models. Hence, we decided to use DNN model alone in the rest of the experiments.

Table 4. Task 1 performance using DNN trained on different uni-modal features.

Model	Accuracy	Precision	Recall	F1 Score	Kappa Statistic
VGG16-fc1	97.27	97	98	97	95.12
VGG16-fc2	97.32	98	98	98	94.63
NASNet	97.14	97	97	97	94.27
Xception	97.41	97	97	97	94.82
Inception ResNetV2	97.34	97	97	97	94.54

shows the representative power of uni-modal features that are extracted from different pre-trained models. It is clear from the results that the performance of the DNN model varies depending on the uni-modal features used. This experiment gives a clue that each pre-trained model extracts a different set of features from retinal images. The features extracted from Xception yield better performance in terms of accuracy for the diabetic retinopathy identification task. A nominal difference in terms of accuracy and kappa score can be observed between the models trained using different uni-modal features.

To better understand the representative nature of different uni-modal features, loss and number of epochs taken to converge by the DNN models are reported in

Table 5. Task 1: Comparison of DNN model (trained on uni-modal features) in terms of loss and number of epochs when trained on different uni-modal features.

Model	# Epochs	Loss	Accuracy
VGG16-fc1	65	0.0024	97.27
VGG16-fc2	67	0.0139	97.32
NASNet	37	0.0310	97.14
Xception	16	0.0213	97.41
Inception ResNet V2	19	0.0815	97.34

. We can observe that the model trained using VGG16-Fc1 reaches minimum loss compared to the rest of the models. In terms of convergence, Xception takes only 16 epochs, whereas the Inception ResNetV2 outperforms the other models.

To summarize the experiments on DR identification task, features extracted from Xception, VGG16-fc2, and Inception ResnetV2 yields the same accuracy with nominal differences. However, models trained on the VGG16-fc1 features give better kappa scores compared to others. We can also observe that models trained on the VGG16-fc2 features give better performance in terms of precision, recall, and F1 scores. Regardless of the type of uni-modal features used, DNN consistently outperforms the rest of the models, especially in terms of kappa scores. The reason for the superior performance of the models trained using VGG16 and Xception features is that these models are good at extracting the lesion information that is useful to discriminate the DR-affected images from those that are not affected.

4.5. Experimental Studies to Show the Representative Nature of Uni-Modal Features for Task 2

We ran a set of experiments to understand the nature of uni-modal features for severity prediction of DR. Task 2 is more challenging compared to Task 1 as it involves multiple classes. DNN model with dropout at the input layer was used with different uni-modal features.

Based on the results reported in

Table 6. Task 2 performance using DNN trained on different uni-modal features.

Type of Uni-Modal Features	Accuracy	Precision	Recall	F1 Score	Kappa Statistic
VGG16-fc1	80.06	80	81	80	70.02
VGG16-fc2	79.81	79	80	79	68.88
NASNET	76.4	75	76	75	63.87
Xception	78.99	78	79	78	67.67
Inception ResNetV2	79.73	78	78	78	67.67

, we can observe the same trend that was observed in Task 1. The scores obtained for Task 2 show the complexity of severity prediction. The model trained on VGG-16+fc1 features shows superior performance to rest of the models. The same can be observed in terms of all the metrics.

In

Table 7. Task 2: Comparison of DNN model (trained on uni-modal features) in terms of loss and number of epochs when trained on different uni-modal features.

Model	# Epochs	Loss	Accuracy
VGG16-fc1	76	0.3623	80.06
VGG16-fc2	79	0.3986	79.81
NASNet	37	0.5612	76.39
Xception	23	0.4175	78.99
Inception ResNet V2	89	0.382	79.73

, it is clear that, among all the pre-trained features, VGG16-fc1 yields superior performance with minimum loss. However, Xception converges in fewer epochs compared to other models.

4.6. Performance Evaluation of the Proposed Blended Multi-Modal Features

A clue from the experiments on uni-modal features is that different uni-modal features extract different sets of features from the retinal images. If we can use multiple deep features extracted from different models, they complement each other and help to improve the scores. To benefit from more than one set of uni-modal features, we propose a blended multi-modal feature representation. This section is dedicated to show the representative power of the proposed feature representation with an application to DR identification and severity level prediction.

In addition, we applied the proposed pooling methods to blend the features from multiple pre-trained models. Initially, we blended features from the first and second fully connected layers of VGG16. Then, we extended this to the fusion of three different features from fc1 and fc2 layers of VGG16 and Xception.

4.6.1. Blended Multi-Modal Deep Features for Task 1

We experimented on the effect of blending deep features extracted from multiple pre-trained models on DR identification task. In addition, we verified the proposed maximum, sum, and average pooling approaches to blend multiple deep features.

In

Table 8. DNN with blended multi-modal features with different fusions for Task 1.

Modalities	Pooling	Accuracy	Kappa Statistic	Epochs	Loss
VGG16-fc1 and VGG16-fc2	Max-pooling	96.12	91.89	68	0.0352
	Avg-pooling	97.39	94.61	51	0.0293
	Sum-pooling	95.5	91	64	0.0419
VGG16-fc1, VGG16-fc2 and Xception	Max-pooling	96.85	92.6	69	0.0314
	Avg-pooling	97.92	94.93	43	0.0201
	Sum-pooling	96.1	92.31	56	0.0396

, we can observe that average pooling based fusion works better for DR Detection compared to other models. Using average fusion the models trained on multi-modal features leads to superior performance in terms of accuracy and kappa static. In addition, the model converges more quickly, in less than 50 epochs, and attains minimum loss. The accuracy obtained by model trained using multi-modal features is significantly better compared with to those trained on uni-modal features.

4.6.2. Blended Multi-Modal Deep Features for Task 2

From the previous experiments, we understand that the models trained on multi-modal features give better performance compared to those trained on uni-modal features in the context of DR identification which is simple binary task. To understand that the proposed blended performs efficiently for more complex multi-class classification task, we applied the proposed feature representation for the severity prediction task shown in

Table 9. DNN with blended multi-modal features with different fusions for Task 2.

Modalities	Pooling	Accuracy	Kappa Statistic	Epochs	Loss
VGG16-fc1 and VGG16-fc2	Maximum	78.06	66.87	72	0.4176
	Average	80.34	69.21	62	0.2987
	Sum	76.8	65.64	68	0.5693
VGG16-fc1, VGG16-fc2 and Xception	Maximum	79.25	67.29	74	0.3986
	Average	80.96	70.9	54	0.2619
	Sum	77.12	66.42	61	0.4782

.

In

Table 10. Comparison of Proposed method using with existing methods.

Model	Accuracy
DR detection using Deep Learning [43]	57.2%
DR Classification Using Xception [44]	79.59
DR Classification Using InceptionV3 [44]	78.72
DR Classification Using MobileNet [44]	79.01
DR Classification Using ResNet50 [44]	74.64
Blended features + DNN (proposed)	80.96

, we can see that average pooling based fusion of multiple deep features works better for diabetic severity prediction. Compared to the blended features from VGG16-fc1 and VGG16-fc2, the blended features from VGG16-fc1, VGG16-fc2, and Xception gives better representation. For severity prediction as well, the model that uses average pooling approach for fusion converges more quickly with better accuracy and kappa score when compared with the other approaches for fusion.

4.7. Comparison of Proposed Blended Feature Extraction with Existing Methods

In this experiment, we showed the effectiveness of the proposed DNN with dropout at the input layer trained using the proposed blended multi-modal deep feature representation and with the existing models in the literature for DR prediction. We compared the proposed model with the performances of the models used in [43,44]. In Table 10, we can see that the proposed method gives an accuracy of 80.96%, which is significantly better than existing models in the literature. When compared to the existing models, the proposed DNN model is simple with only three hidden layers with 512, 256 and 128 units each hidden layer. The confusion matrix in Figure 8 shows the mis-classifications produced by the proposed model when applied for DR severity prediction task. In the figure, we can see that most of the proliferate DR type images are predicted as moderate.

As the final feature vector is a blended version of the local and global representations of the retinal images the final representation provides strong features. The reason for improvement in the performance of the proposed model is that each feature map of the final convolution block of VGG16 learns the presence of different lesions from the retinal images and Xception Net comprehensive representation of the entire retinal scan. When we combine the deep features from VGG16 and Xception gives a compact representation that gives the wholistic representation of DR images.

5. Conclusions

The major objective of this work is to acquire a compact and comprehensive representation of retinal images as the feature representations extracted from retinal images significantly influence the performance of DR prediction. Initially, we extract features from deep pre-trained VGG16-fc1, CGG16-fc2 and Xception models. VGG16 model learns the lesions and Xception learns the global representation of the images. Then, the features from multiple ConvNets are blended to get final prominent representation of color fundus images. The final representation is obtained by pooling the representations from VGG16 and Xception features. A DNN model was trained using these blended features for the task of diabetic retinopathy severity level prediction. The proposed DNN model with dropout at the input avoids overfitting and converges more quickly. Our experiments on benchmark APTOS 2019 dataset showed the superiority of the proposed model when compared to the existing models. Among the proposed pooling approaches, average pooling used to fuse the features extracted from the penultimate layers of multiple pre-trained ConvNets gives better performance with minimum loss in fewer epochs compared to others.