A Hybrid Deep Learning Approach for COVID-19 Diagnosis via CT and X-ray Medical Images ^†

Abstract

The COVID-19 pandemic has been a global health problem since December 2019. To date, the total number of confirmed cases, recoveries, and deaths has exponentially increased on a daily basis worldwide. In this paper, a hybrid deep learning approach is proposed to directly classify the COVID-19 disease from both chest X-ray (CXR) and CT images. Two AI-based deep learning models, namely ResNet50 and EfficientNetB0, are adopted and trained using both chest X-ray and CT images. The public datasets, consisting of 7863 and 2613 chest X-ray and CT images, are respectively used to deploy, train, and evaluate the proposed deep learning models. The deep learning model of EfficientNetB0 consistently performed a better classification result, achieving overall diagnosis accuracies of 99.36% and 99.23% using CXR and CT images, respectively. For the hybrid AI-based model, the overall classification accuracy of 99.58% is achieved. The proposed hybrid deep learning system seems to be trustworthy and reliable for assisting health care systems, patients, and physicians.

1. Introduction

The outbreak of COVID-19 is considered an epidemic and pandemic, affecting people around the world in a short period. It is rapidly transmitted among people in different local and global communities due to travel issues [1]. To date, the number of confirmed cases and deaths has reached 226 million and 4 million worldwide, respectively. COVID-19 is a novel coronavirus coined as Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) and it targets the human respiratory system. The confirmed biological symptoms of COVID-19 are fever, shortness of breath, dizziness, cough, headache, sore throat, fatigue, and muscle pain. Accurate and rapid classification techniques have become necessary to automatically diagnose COVID-19, especially in a pandemic situation. Recently, AI techniques (deep learning and machine learning) were employed to build a robust decision-making system against COVID-19 [2,3,4]. Traditionally, COVID-19 screening involves RT-PCR (reverse transcription polymerase chain reaction) carried out at the pathogen laboratory. Due to its higher time consumption and lower sensitivity, medical imaging techniques such as computed tomography (CT) as well as Chest X-ray (Radiological Image) images are being used to fight and classify the COVID-19 respiratory disease [5,6,7]. The lungs are the major target of the COVID-19 virus. RT-PCR is useful for the diagnosis of disease, while CT and CXR images are useful to assess the damage caused to the lungs due to COVID-19 at various stages of the disease. Inflammation of lung tissues can be identified based on the size and shape of attacked tissues with the help of X-ray and CT images [3,6]. Deep convolutional networks are extensively utilized in the fields of hyperspectral images, microscopic images, and medical image analysis; current trending coronavirus-related diagnostic studies also used deep learning-based architectures, namely COVID-SDNet, DL-CRC, and EDL-COVID [1,6,8]. Machine learning-based techniques such as SOM-LWL, PB-OCSVM, and one-shot cluster-based approaches for COVID CXR images have also been introduced to the diagnosis of COVID detection and classification [9,10,11]. In addition, other techniques such as transfer and learning methods have been implemented using MobileNet, VGG, ResNet, Alexnet, and DensNet architectures as a base module for training for the task of COVID image classification [12,13]. Computer-aided diagnosis systems have been proposed for several medical image analysis tasks such as breast cancer, brain tumors, and kidney and lung disorders using deep learning methods [5,12,14,15].

In this proposed work, a hybrid deep learning system is deployed to perform the classification task of COVID-19 using two CXR and CT datasets. Deep convolutional networks have a promising feature extraction method to automatically identify a large quantity of deep features directly from the input images, thus improving overall classification accuracy. The objective of this study is to provide a unified deep learning model using both medical CXR and CT images. The main contributions in this hybrid system are summarized as follows: First, the building of a novel hybrid deep learning model in unified architecture to automatically and rapidly classify COVID-19 disease using both CXR and CT images. Second, deep learning regularizations of data balancing, transfer learning-based approaches, and data augmentation are used in order to improve overall diagnostic performance. Such experiments will help to improve understanding of COVID-19 disease and to diagnose it using different medical imaging modalities [16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36].

The objective of this work is to provide a robust and feasible AI-based system for medical institutions, health care service providers, physicians, and patients by providing practical solutions for COVID-19 diagnosis.

The rest of this paper is organized as follows: A review of the relevant literature is presented in Section 2; The technical aspects of the deep learning methods for classification systems are detailed in Section 3; The results of the experiment with COVID-19 are reported and discussed in Section 4 and Section 5; Finally, the most important findings of this work are summarized in the conclusion in Section 6.

2. Related Work

In early 2020, while the world was under pandemic conditions due to the COVID-19 outbreak, some computer-aided diagnosis systems, proposed based on deep learning, were introduced to predict COVID-19 on digital X-ray and CT images. In [18], Yang et al. presented diagnosis of COVID-19 with the help of CT images, and proposed an AI-based diagnosis system based on DensNet and ResNet pre-trained models with transfer and learning techniques to classify, and reported an accuracy of 89%, AUC of 98%, and F1 score of 90%; the dataset was made open-source. Madallah Alruwaili et al. in [19] used an improved Inception-ResNetV2 to diagnose COVID-19 in X-ray images, which had high accuracy in the radiography dataset at detecting COVID-19. Mundher et al. in [13] designed a model to detect COVID-19 from X-ray images using convolutional neural networks with transfer learning-based techniques with VGG16, and MobileNet modules reported the highest accuracy of 98.28% with VGG16 as a base model. In [19], Madallah Alruwaili et al. used an improved Inception-ResNetV2 to diagnose COVID-19 in X-ray images. Xception, VGG16, InceptionV3, ResNet50V2, MobileNetV2, ResNet101V2, and DenseNet121 models were used for experimentation and with CXR images, with the Inception-ResNetV2 model achieving 99.8%. In [20], Fareed Ahmad et al. designed a deep learning model for detecting COVID-19 using chest X-ray images, using different CNN models such as MobileNet, InceptionV3, and ResNet50. The best model was InceptionV3, which reported 95.75% and 91.47% accuracy and F-score, respectively. In [5], Boran Sekeroglu et al. used the CNN model to detect COVID-19 from chest X-ray images using the available dataset. They used CNN without preprocessing and with a decreasing number of layers, and were capable of detecting COVID-19 in a limited number of data and imbalanced chest X-ray images with an accuracy of 98.50%. In [21] Pramit Brata Chanda et al. implemented a new model to diagnose COVID-19 using chest X-rays. They used the CNN-based transfer learning framework for the classification task and reported an accuracy of 96.13%. In [22], Mubashir Rehman et al. designed a platform-monitoring system to detect and diagnose of COVID-19 using the breathing rate measurement. In [8], S. Tabik et al. contributed a new open-source dataset, called COVIDGR-1.0. In their experiment, they designed a new model to detect COVID-19 using X-ray images, and also helped to measure severity. They reported the classification as moderate and severe—86.90% and 97.72%, respectively, on the basis of the CXR database. In [12] Wentao et al. presented a new model based on deep learning for the diagnosis of COVID-19 using CT images. The transfer learning technique achieved a good accuracy of 98%. In [6], Sadman et al. proposed a deep learning-based chest radiograph classification (DL-CRC) framework to distinguish COVID-19 cases with high accuracy from two classes, abnormal and normal. They presented a deep learning model called the DL-CRC framework, with two parts: the DARI algorithm and generic data augmentation, with an accuracy of 93.94%. In [23] Khalid M. Hosny et al. designed a hybrid model to detect COVID-19 using two types of CT scans and chest X-ray images. Their work combined two types of images to fit memory and computational time. They proposed the framework for CXR and CT images with 99.3% and 93.2%, respectively. In [7], transfer learning was presented to detect COVID-19 using X-ray and CT-scan images. This was because in COVID-19, initial screening of chest X-rays (CXR) may provide significant information in the detection of suspected COVID-19 cases. In [24], Ravi et al. presented a model to detect COVID-19 using both CT and CXR datasets. In [25], Elmehdi Benmalek et al. aimed to make a comparison for the performances of CT-scan and chest X-ray images to detect COVID-19 utilizing CNN, achieving an accuracy equal to 98.5% and 98.6%, respectively. In [16], Muhammad E. H. et al. presented a strong model to detect COVID-19 pneumonia using chest X-ray images, utilizing the pre-trained deep learning technique. They created a database by merging data that had been created by previous work. They obtained a classification accuracy of 99.77%.

3. Methods and Materials

The proposed hybrid deep learning system for COVID-19 diagnosis is demonstrated in Figure 1. Two different deep learning models, namely ResNet50 and EfficientNetB0, were used for CXR and CT images, respectively. Both deep learning models were trained using 100 epochs. The final layers from both deep learning models were concatenated together to merge the derived deep features and generate the single most robust deep-feature set. This set carries promising features generated from both CXR and CT images at the same time; this is key to improve the overall accuracy performance of the proposed deep learning system. The concatenated deep features were then scaled in 1D form using a global average pooling (GAP); this is to make the derived feature maps suitable for the following two fully connected layers. Finally, Softmax layer is used to make the final decision of whether the output is a positive COVID-19 case or a normal negative case. To reduce overfitting that may occur during the training phase, the 0.5 dropout strategy is used. For pre-training, the transfer leering strategy is used with the ImageNet database.

3.1. Preprocessing

The preprocessing technique is the most significant step of the model. Here, we considered raw data and transformed it into a specific input data format and dimension [3]. We inculcated the data augmentation and class-balancing strategy to reduce overfitting, and this acted as a catalyst for the training process [15,31]. Later, we divided both data into 70% for training, 20% for testing, 10% for validation. For each class, the dataset was selected in a randomized manner. For hyperparameter initialization, the transfer-earning strategy was applied using the dataset of ImageNet [3,15,31].

3.2. Feature Extraction

Deep CNN has shown improved performance regardless of domain, particularly in medical imaging, and generalization of the model has been observed. Transfer learning is being explored to provide an efficient solution [6]. In our experimental analysis, we employed ResNet50 [27] and EfficientNet0 [24,26] models for the task of feature generation; deep features were later passed to custom user-specific layers. In our work we pushed to global average pooling, followed by a fully connected layer [24]. We used two FC layers, improving efficiency; and to generalize learning, we also introduced dropout in the middle of the FC layers. These extracted features were passed to the classification layer to assign the appropriate class label of the given input data instance.

3.3. Classification

The pipeline of extracted deep features through feed-forward models ResNet50 and EfficientNet0 was passed to the SoftMax layer for classification and feature extraction. The results were generated for both CT and CXR databases separately and results were discussed in the below tables. The results were promising in contrast to existing research.

4. Experimental Analysis

4.1. Dataset

To quantify our work, two datasets of chest X-rays and CTs were used. These datasets are publicly available at Kaggle databases [16,17,18]. The datasets are described as shown in

Table 1. Chest X-ray and CT dataset distribution per class.

Type of Images	COVID	Normal
CT	1323	1290
X-ray	3923	3960

.

4.2. Implementation Environment

To perform all experiments in this study, we used a PC with the following specifications: Intel R © Core(TM) i7-6850 K processor with 32 GB RAM, 3.360 GHz frequency GPUs NVIDIA GeForce GTX1050Ti. Deep learning algorithms were implemented herein using Python 3.8.0 programming with Anaconda [Jupyter notebook]. The Python-based ML libraries such as Torch, TensorFlow, OpenCV, pandas, and Scikitlearn were utilized to investigate the performance metrics by the proposed methods; at the same time TensorFlow and Keras in Colab were used to implement transfer learning. The results and discussions concerning various techniques incorporated are highlighted in the subsequent sections. The source codes are available at GitHub (https://github.com/IIITK-AI-LAB/Hybrid-covid-model (accessed on 25 September 2021)).

4.3. Evaluation Metrics

To assess our proposed system, we used the evaluation metrics of recall/sensitivity (Re), specificity (Sp), F1-measure (F-M), and overall accuracy (Az). The mathematical formula for these evaluation metrics is defined as follows:

$$\mathrm{Recall}/\mathrm{Sensitivity} \left(\mathrm{Re}\right)=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} ,$$

(1)

$$\mathrm{Specificity} \left(\mathrm{Sp}\right)=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}} ,$$

(2)

$$\mathrm{F}1\text{-}\mathrm{score} \left(\mathrm{F}\text{-}\mathrm{M}\right)=\frac{2·\mathrm{TP}}{2·\mathrm{TP}+\mathrm{FP}+\mathrm{FN}} ,$$

(3)

$$\mathrm{Overall} \mathrm{accuracy} \left(\mathrm{Az}\right)=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FN}+\mathrm{TN}+\mathrm{FP}} ,$$

(4)

where TP, TN, FP, and FN are defined to represent the number of true positive, true negative, false positive, and false negative detections, respectively. The confusion matrix is used to derive all of these parameters.

5. Results and Discussion

This section shows our experimental results in the following two different scenarios: a single straightforward scenario and a hybrid scenario. The former scenario means both deep learning models (i.e., ResNet50 and EfficientNetB0) are separately used and tested to investigate which model could provide the best overall classification accuracy for a single dataset—chest X-ray or CT images. The later scenario means both deep learning models are concatenated to produce the proposed hybridization model, as shown in Figure 1. This is to check which hybridization combination could achieve the best performance when both medical chest images are used.

5.1. Single Straightforward Scenario

For each medical chest dataset (i.e., chest X-ray or CT images), two different experiments were performed. One experiment was carried out using the deep convolutional ResNet50 model, and the other was performed using the EfficientNetB0 deep learning model. In other words, the single deep learning model (i.e., ResNet50 or EfficientNetB0) was trained twice: once for the chest X-ray and again for the CT images. In both training styles, the same deep learning architecture, as well as training/testing settings, were used.

5.1.1. COVID-19 Classification Based on Chest X-ray Images

In this case, the input dataset consists only of X-ray images for the ResNet50 or EfficientNetB0 deep learning models. The overall classification evaluation results are summarized in

Table 2. Classification evaluation results (%) using chest X-ray images.

Model	Az	Sp	Re	F-M
ResNet50	98.47	99.0	100	99.0
EfficientNetB0	99.36	98.0	99.0	99.0

. Although it is obviously shown that both deep learning models achieve almost the same results, the EfficientNetB0 deep learning model achieves a slightly better overall accuracy of 99.36%.

5.1.2. COVID-19 Classification Based on Chest CT Images

The chest CT dataset is only used to separately train the deep learning models of ResNet50 and EfficientNetB0. The overall classification evaluation results are reported in

Table 3. Classification evaluation results using chest CT images.

Model	Az	Sp	Re	F-M
ResNet50	98.85	99.0	98.0	99.0
EfficientNetB0	99.23	99.0	99.0	99.0

. The EfficientNetB0 deep learning model achieves a slightly better overall accuracy of 99.23%, while other evaluation metrics show a consistent and stable performance.

5.2. Hybrid Scenario: COVID-19 Classification Using Chest and X-ray Images

In the proposed hybrid deep learning model, both chest X-ray and CT datasets are used as input, as shown in Figure 1. The evaluation classification results for the best combination hybrid model are demonstrated in

Table 4. Classification evaluation results (%) for the proposed hybrid deep learning model using both chest X-ray and CT medical images.

Model	Az	Sp	Re	F-M
ResNet50	98.01	99.0	99.0	99.0
EfficientNetB0	99.58	99.0	99.0	99.0

. Each row in Table 4 presents the classification assessment results by using a single deep learning model in a hybrid style for both chest X-ray and CT images.

6. Conclusions

A hybrid deep learning model is proposed to automatically detect COVID-19 respiratory disease from both chest X-ray and CT images. The proposed hybrid model uses two deep convolutional networks, namely ResNet and EfficientNet, to generate promising deep hierarchical features. The proposed hybrid deep learning approach could achieve classification accuracies of 99.58% using chest X-ray and CT images. Further improvements could be achieved by including ultrasound images as well. This can help to construct and build a much more robust and reliable diagnosis system to fight COVID-19 in the early stages. The promising results could help to provide a better real-time diagnosis system for health care service providers, physicians, and patients.