Pneumonia Transfer Learning Deep Learning Model from Segmented X-rays

Pneumonia is a common disease that occurs in many countries, more specifically, in poor countries. This disease is an obstructive pneumonia which has the same impression on pulmonary radiographs as other pulmonary diseases, which makes it hard to distinguish even for medical radiologists. Lately, image processing and deep learning models are established to rapidly and precisely diagnose pneumonia disease. In this research, we have predicted pneumonia diseases dependably from the X-ray images, employing image segmentation and machine learning models. A public labelled database is utilized with 4000 pneumonia disease X-rays and 4000 healthy X-rays. ImgNet and SqueezeNet are utilized for transfer learning from their previous computed weights. The proposed deep learning models are trained for classifying pneumonia and non-pneumonia cases. The following processes are presented in this paper: X-ray segmentation utilizing BoxENet architecture, X-ray classification utilizing the segmented chest images. We propose the improved BoxENet model by incorporating transfer learning from both ImgNet and SqueezeNet using a majority fusion model. Performance metrics such as accuracy, specificity, sensitivity and Dice are evaluated. The proposed Improved BoxENet model outperforms the other models in binary and multi-classification models. Additionally, the Improved BoxENet has higher speed compared to other models in both training and classification.


Introduction
Pneumonia is a pulmonary disease that is considered as the top cause of death from pulmonary diseases [1]. Providentially, this disease can be treated at early diagnosis and subsequent management of medication [2]. X-rays are usually utilized for revealing pulmonary pneumonia [3,4]. In medical practice, radiographs are inspected by medical experts for the detection of this disease. Nevertheless, it is a time-intense and biased procedure. Pneumonia is misdiagnosed as other pulmonary diseases of comparable radiologic forms [5][6][7][8]. This can lead to incorrect prognosis and deterioration of the patient case. Moreover, radiologists in poor countries and rural regions are rare. Therefore, automated models can perform significant mass screening by investigating X-rays. Large-size labelled databases and deep learning models yield correct X-ray diagnoses. Deep learning permits hierarchical feature extraction from sufficient training inputs [8][9][10][11]. The medical sector is also completely unlike other arenas as it has not satisfied the ambitions of humanity, while it engages a large proportion of countries' budgets [12]. The health experts' investigative methods with subjective variances lead to X-ray misinterpretation. Therefore, machine learning in the medical sector is of great consideration in current times. Machine learning solutions have been presented for medical field including pneumonia, brain cancer detection and health monitoring [12][13][14][15][16][17][18][19]. Deep and machine learning have presented great potential in image segmentation and classification and are widely approved by the research municipal [18][19][20][21][22]. Image radiography is a highly utilized imaging method. Moreover, Table 1. Recent research in pneumonia prediction deep learning models.

Ref.
Method Model Database Average Accuracy [13] Binary classification Spatial similarity matrix Contrastweighed X-rays database 90.23% [14] Pneumonia identification Region CNN 4064 X-rays of 1200 patients 89.76% [15] Classification of pneumonia and healthy cases Capsule CNN 3770 X-rays 93.7% [16] Classification of pneumonia into three stages (preliminary, moderate, severe cases) Deep CNN Architecture 2044 X-rays of 2000 patients (average 2 X-rays per patient) 96.4% [17] Pneumonia and tuberculosis classifications CNN and discrete wavelet transform 2054 X-rays 94.5-97.5% [18] Pneumonia classification Transfer learning deep CNN 5942 X-rays 96.5% with less CPU time [19] Pneumonia classification Deep learning Region-based CNN method 2115 chest X-ray 97.67% [20] Pneumonia grading Textural based feature extraction 640 lung X-ray 97.2% [21] Pneumonia classification Texture and hue feature extraction 670 cases with 1580 X-rays 95.8% [22] X-ray pneumonia grading Genetic algorithms with deep learning Unknown achieves better performance than solo CNN [23] Prediction of pneumonia with high speed High-speed region CNN 320 chest X-rays 97.8%, with a small-size database advantage

Materials and Methods
The method of this research includes two diverse datasets. Segmented/Complete X-ray images for pneumonia classification are utilized. Three experiments are performed in this research. Two BoxENet models are considered to find the appropriate model for segmenting pulmonary areas of the X-ray. At the next stage, X-ray images are utilized for pneumonia classification employing five pre-trained CNNs. The validation of the classification accuracy is performed using class activation mapping (CAM) metric [15]. Segmented X-rays are utilized for pneumonia classification utilizing the five models and tested their accuracy utilizing the CAM metric. At the last phase, the t-SNE method is realized via Python programming language platform. The parameters are adjusted to check the accuracy of the best model. Table 2 depicts the architecture of the BoxENet and its parameters. Additionally, Figure 1 depicts the BoxENet dense structure.

Data Set
In this research, the Kaggle X-ray dataset and pulmonary mask database [29] are utilized for training the pulmonary segmentation method. Eight hundred X-rays and their analogous ground truth pulmonary masks are presented. All masks were annotated by expert radiologists and labelled by medical experts; X-ray instances and their analogous masks are depicted in Figure 2. The database includes 460 healthy X-ray images and 340 diseased pulmonary X-ray images. Consequently, BoxENet model are trained with both X-ray images.  Healthy.

Data Set
In this research, the Kaggle X-ray dataset and pulmonary mask database [29] are utilized for training the pulmonary segmentation method. Eight hundred X-rays and their analogous ground truth pulmonary masks are presented. All masks were annotated by expert radiologists and labelled by medical experts; X-ray instances and their analogous masks are depicted in Figure 2. The database includes 460 healthy X-ray images and 340 diseased pulmonary X-ray images. Consequently, BoxENet model are trained with both X-ray images.

Data Set
In this research, the Kaggle X-ray dataset and pulmonary mask database [29] are utilized for training the pulmonary segmentation method. Eight hundred X-rays and their analogous ground truth pulmonary masks are presented. All masks were annotated by expert radiologists and labelled by medical experts; X-ray instances and their analogous masks are depicted in Figure 2. The database includes 460 healthy X-ray images and 340 diseased pulmonary X-ray images. Consequently, BoxENet model are trained with both X-ray images.  Four public datasets are utilized for pneumonia classification. The datasets are PND Pneumonia, BelPnem dataset, NIAPn Pneumonia dataset and RPNA dataset: the PND database is found in the US public library of medicine [25], and it has two pulmonary X-ray datasets containing 230 and 767 posterior-anterior lung X-rays. The X-ray image resolution is 2048 × 2048 pixels. In the first dataset (DS1) of 230 X-rays, 100 images are captured from diverse pneumonia cases and 130 images are from healthy cases. In the second dataset (DS2), out of 767 X-rays, 467 X-rays are captured from pneumonia cases and 300 X-rays are normal cases. Thus, in this PND dataset, there are 430 normal and 567 Pneumonia cases.
The BelPnem database [7] was composed for drug research started by the Institute of Infections, USA. The database has 400 X-rays of 160 cases. The X-rays were captured utilizing the Kodak-370 system and the X-ray resolution is 3200 × 3200 pixels. All the X-rays of this dataset are of pneumonia cases.
NIAPn Pneumonia database: The NIAPn Pneumonia portal dataset [17] features 2500 pneumonia cases from a total of 3500 X-rays. All X-rays are stored in PNG format. In this research, we utilized 3200 pneumonia cases out of the total X-rays. From this dataset, 300 X-rays of unsatisfactory quality were excluded.
RPNA database: The RPNA pneumonia detection database [12] has more than 3000 lung X-rays, of which 1000 X-rays are healthy, and the rest of the cases are pneumonia or pulmonary opaque. All X-rays are in Digital Communications Medical Imaging format (DCMI). To generate a standard database of 4000 normal healthy X-rays for this research, 2900 normal X-rays were obtained from this dataset and the remaining of the 1100 healthy X-rays were obtained from the PND dataset. Nevertheless, the number of Pneumonia X-rays are collected as 300 X-rays from PND, 400 images from BelPnem and 3000 X-rays from NIAPn pneumonia dataset. In total, 700 pneumonia and 4000 healthy images are utilized in this research. Instances of the collected X-rays are depicted in Figure 3.
Four public datasets are utilized for pneumonia classification. The datasets are PND Pneumonia, BelPnem dataset, NIAPn Pneumonia dataset and RPNA dataset: the PND database is found in the US public library of medicine [25], and it has two pulmonary X-ray datasets containing 230 and 767 posterior-anterior lung X-rays. The X-ray image resolution is 2048 × 2048 pixels. In the first dataset (DS1) of 230 X-rays, 100 images are captured from diverse pneumonia cases and 130 images are from healthy cases. In the second dataset (DS2), out of 767 X-rays, 467 X-rays are captured from pneumonia cases and 300 X-rays are normal cases. Thus, in this PND dataset, there are 430 normal and 567 Pneumonia cases.
The BelPnem database [7] was composed for drug research started by the Institute of Infections, USA. The database has 400 X-rays of 160 cases. The X-rays were captured utilizing the Kodak-370 system and the X-ray resolution is 3200 × 3200 pixels. All the X-rays of this dataset are of pneumonia cases.
NIAPn Pneumonia database: The NIAPn Pneumonia portal dataset [17] features 2500 pneumonia cases from a total of 3500 X-rays. All X-rays are stored in PNG format. In this research, we utilized 3200 pneumonia cases out of the total X-rays. From this dataset, 300 X-rays of unsatisfactory quality were excluded.
RPNA database: The RPNA pneumonia detection database [12] has more than 3000 lung X-rays, of which 1000 X-rays are healthy, and the rest of the cases are pneumonia or pulmonary opaque. All X-rays are in Digital Communications Medical Imaging format (DCMI). To generate a standard database of 4000 normal healthy X-rays for this research, 2900 normal X-rays were obtained from this dataset and the remaining of the 1100 healthy X-rays were obtained from the PND dataset. Nevertheless, the number of Pneumonia Xrays are collected as 300 X-rays from PND, 400 images from BelPnem and 3000 X-rays from NIAPn pneumonia dataset. In total, 700 pneumonia and 4000 healthy images are utilized in this research. Instances of the collected X-rays are depicted in Figure 3. The X-ray image resolutions for various CNNs are different and hence the preprocessing phase is required to resize the input images. In the segmentation phase, the X-rays are downsized to 412 × 412 pixels for both the BoxENet and the improved BoxENet. In the classification phase, X-rays are downsized to 256 × 256 pixels for BoxENet and SqueezeNet. All X-rays are normalized utilizing Z-metric process utilizing mean and variance.

Methodology
The improved BoxENet is described as follows: We classified pneumonia from segmented X-rays through transfer learning from two CNNs (ImgNet and Squeeznet) followed by deep dense learning from BoxENet. As depicted in Figure 4, the improved BoxENet is composed of: 1. An input feeding stage; The X-ray image resolutions for various CNNs are different and hence the preprocessing phase is required to resize the input images. In the segmentation phase, the X-rays are downsized to 412 × 412 pixels for both the BoxENet and the improved BoxENet. In the classification phase, X-rays are downsized to 256 × 256 pixels for BoxENet and SqueezeNet. All X-rays are normalized utilizing Z-metric process utilizing mean and variance.

Methodology
The improved BoxENet is described as follows: We classified pneumonia from segmented X-rays through transfer learning from two CNNs (ImgNet and Squeeznet) followed by deep dense learning from BoxENet. As depicted in Figure 4, the improved BoxENet is composed of: 1.
Parallel ImgNet and Squeezenet for transfer learning to produce the parameters; 3.
Fusion layer to fuse the best parameters; 4.
BoxENet for final prediction (it is also fed with the segmented X-ray lung images).
The BoxENet structure has three dense blocks each followed by convolution and pooling. A Softmax classifier produces the predicted output. Dense blocks tackle the fading gradient problem utilizing preceding feature maps as the input to the next layer. pooling. A Softmax classifier produces the predicted output. Dense blocks tackle the fad-ing gradient problem utilizing preceding feature maps as the input to the next layer.
The improved BoxENet with transfer learning is utilized on Kaggle X-ray. Pulmonary mask images are utilized for pulmonary X-ray segmentation. Of all X-rays and labelled masks, 70% are utilized for training, 15% for the testing phase and 15% for the validation phase, as depicted in Table 3. A ten-fold cross-validation technique is utilized for the validation of the whole database.   The improved BoxENet with transfer learning is utilized on Kaggle X-ray. Pulmonary mask images are utilized for pulmonary X-ray segmentation. Of all X-rays and labelled masks, 70% are utilized for training, 15% for the testing phase and 15% for the validation phase, as depicted in Table 3. A ten-fold cross-validation technique is utilized for the validation of the whole database. The models are executed utilizing PyTorch Python 3.7 on Xeon CPU v4 (3.3 GHz) and 128 GB RAM, with a 32 GB GTX GPU. Both BoxENet architectures are trained utilizing a Gradient Descent model with a learning rate of 10-2 and a dropout rate of 0.3 with batch size of 64 X-rays with 80 epochs.
Five different deep learning architectures are trained and tested disjointedly utilizing complete and segmented X-rays for the detection of pneumonia and non-pneumonia images. The image set is partitioned into 70% training set and 30% for validation and testing. Data used in the ten-fold validation is 10% of training subset for validation. For instance, 70% (2800) of 4000 healthy X-rays are utilized for training and 15% (600) images are utilized for validation and 600 X-rays for testing. Table 4 depicts the training, validation and testing X-rays utilized in experiment settings of whole and segmented X-rays. All five CNNs are realized utilizing PyTorch Python 3.7 on Xeon CPU v4 (3.3 GHz) and 128 GB RAM, with a 32 GB GTX GPU. Two shallow CNNs, namely, MobNetv2 and ResNet, and three deep CNNs, namely, Inceptionv3, BoxENet and DensNet, were tested in our research to study whether shallow CNNs or deep architectures are appropriate for this medical application. Performance variance of pretrained models on images of other lung X-ray were compared with BoxENet CNN. BoxENet is a 130-layer DensNet and is only pretrained using lung X-rays. The five models are pertained utilizing similar parameters and discontinuing criteria. Thirty-five epochs were utilized for prediction. Ten-fold validation values were averaged to generate the resultant accuracy, and other evaluation confusion matrices. Image amplification and overriding aid in evading overfitting problem [13]. Table 5 depicts the comparison of both BoxENet networks in terms of Loss Function and Batch segmentation. Other parameters (epochs count and learning rate) were automated to be updated in case that no enhancement of accuracy was detected.

Experimental Results
The accuracy of various CNNs was compared at the end of the training stage and it utilized several metrics as follows: model accuracy (ACC), sensitivity (SEN), specificity (SPEC) and Dice coefficient (D). The formulas utilized to compute these metrics are presented in the following equations: where TP is the count of true positives, TN is the count of true negatives, FP is the count of false positives and FN is the count of false negatives. The sensitivity is defined as the rate of true correctly predicted as positives, and the specificity is defined as the rate of true correctly predicted as of negatives. The comparison of the accuracies of various CNNs networks, for the testing subset, is performed at the end of the training stage. The accuracy is measured utilizing six metrics: model accuracy (Equation (1)), model sensitivity (Equation (2)) or recall, model specificity (Equation (3)), model precision (Equation (5)), area under curve (AUC) and F1 (Dice) metric (Equation (4)). TP, TN, FP and FN are defined in the following table (Table 6) where t C1 and t C0 define the start time and the end time for a CNN to predict the diagnosis of an X-ray, while t T1 and t T0 are the start time and the end time of the training of a single epoch by a CNN. The experimental results are depicted below.

X-ray Segmentation
Both BoxENet and the improved BoxENet CNNs were trained and tested on the testing subset for the X-rays segmentation process. The highest accurate parameters for X-ray segmentation are depicted in Table 7. In Figure 5, testing X-rays radiography and associated ground truth are displayed. Segmented pulmonary radiography was produced by both BoxENet CNNs for the Kaggle database. It is depicted that the improved BoxENet outperformed the original BoxENet in the segmentation process of the pulmonary areas from the X-ray radiography.

X-ray Segmentation
Both BoxENet and the improved BoxENet CNNs were trained and tested on the testing subset for the X-rays segmentation process. The highest accurate parameters for X-ray segmentation are depicted in Table 7. In Figure 5, testing X-rays radiography and associated ground truth are displayed. Segmented pulmonary radiography was produced by both BoxENet CNNs for the Kaggle database. It is depicted that the improved BoxENet outperformed the original BoxENet in the segmentation process of the pulmonary areas from the X-ray radiography.  The most accurate BoxENet model is utilized to segment the dataset (4000 healthy and 3700 pneumonia X-rays) and used for the classification process. It is imperative to evaluate a whole untested X-rays set with pneumonia and healthy X-rays to prove the correctness and performance of the segmentation process. It is shown that the original BoxENet network that is trained on Kaggle database will segment the pulmonary areas of the X-rays in a robust way.

Pneumonia Classification
In our research, we performed two experiments (utilizing non-segmented X-rays and complete pulmonary images) for the prediction of pneumonia and healthy (non-pneumonia) cases. The comparison between the various models for the binary prediction is depicted in Table 8. It is obvious that the pretrained transfer models outperform other models in binary classification of pneumonia and healthy cases. BoxENet is one of the models that is trained with unsegmented X-rays and still achieves high performance for predicting pneumonia from the X-rays. BoxENet with initial pretraining offers more benefits in predicting pneumonia from X-rays and is presenting higher accuracy than other models. Deeper CNNs usually outperform other shallow CNNs. In our research, we found that BoxENet is a worthy transfer learning model and it achieves better accuracy than other CNNs. Comparable performance has been attained by researchers in the COVID-19 prediction model [14]. BoxENet is also characterized by high speed, as seen from the CPU The most accurate BoxENet model is utilized to segment the dataset (4000 healthy and 3700 pneumonia X-rays) and used for the classification process. It is imperative to evaluate a whole untested X-rays set with pneumonia and healthy X-rays to prove the correctness and performance of the segmentation process. It is shown that the original BoxENet network that is trained on Kaggle database will segment the pulmonary areas of the X-rays in a robust way.

Pneumonia Classification
In our research, we performed two experiments (utilizing non-segmented X-rays and complete pulmonary images) for the prediction of pneumonia and healthy (nonpneumonia) cases. The comparison between the various models for the binary prediction is depicted in Table 8. It is obvious that the pretrained transfer models outperform other models in binary classification of pneumonia and healthy cases. BoxENet is one of the models that is trained with unsegmented X-rays and still achieves high performance for predicting pneumonia from the X-rays. BoxENet with initial pretraining offers more benefits in predicting pneumonia from X-rays and is presenting higher accuracy than other models. Deeper CNNs usually outperform other shallow CNNs. In our research, we found that BoxENet is a worthy transfer learning model and it achieves better accuracy than other CNNs. Comparable performance has been attained by researchers in the COVID-19 prediction model [14]. BoxENet is also characterized by high speed, as seen from the CPU time needed for average testing (∆t T ) being the shortest among all other compared models. The training processing time used per each single epoch (∆t C ) is also equated to other CNNs, which is credited to the count of convolutional layers in this CNN being reduced than other deeper CNNs such as InceptV5 and DSNet3. It also indicates that the accuracy for pretrained BoxENet is the highest with non-segmented X-rays. The segmented pulmonary X-rays yield a higher performance in terms of all accuracy metrics for pretrained CNNs. This echoes the point that pretrained CNNs can differentiate between pneumonia and healthy cases with high accuracy. Even though all the networks deliver better accuracy in classification, DSeNet3 exhibited the best performance in predicting pneumonia and healthy cases for segment X-rays. Even though accuracy-wise DSNet3 is the most accurate, it exhibits relatively low speed in classifying and training. DSNet3 also requires a longer time to process an X-ray because of its extremely deep structure. It is also noted that ∆t C is greater and ∆t T is less for all the CNN when processing segmented pulmonary X-rays in comparison with nonsegmented X-rays.
In summary, BoxENet and DSNet3 are generating the greatest prediction accuracies of 97.4% and 96.68% for whole and segmented X-rays, respectively. DSNet3 is outperforming all other models for the segmented X-rays, which reveals that deeper CNN can perform with higher accuracy for segmented pulmonary X-rays. It is obvious that the segmentation process enhances the accuracy of classification considerably. Nevertheless, as this model solves a binary classification, all the tested CNNs performed in a better manner. Therefore, we tested the models to perform multiclassification by classifying four stages of pneumonia, namely: Congestion, Red Hepatization, Grey Hepatization and Resolution [26][27][28][29]. In Table 9, comparison of various CNNs for pneumonia multiclassification with whole and segmented X-rays are presented.
Confusion matrices for multiclassification for segmented and non-segmented X-rays for the proposed Improved BoxENet are depicted in Figures 6 and 7.
Training accuracy and loss are depicted in Figures 8 and 9 for the proposed improved BoxENet model. Confusion matrices for multiclassification for segmented and non-segmented X-rays for the proposed Improved BoxENet are depicted in Figures 6 and 7.     Training accuracy and loss are depicted in Figures 8 and 9 for the proposed improved BoxENet model.

Conclusions
Pneumonia is a common disease in many countries, especially in poor countries. This disease is defined as obstructive pneumonia which has the same impression on pulmonary radiographs as other lung diseases, which makes it hard to distinguish it from other pulmonary diseases, even for medical radiologists. Lately, image processing and deep learning models have been established to rapidly and precisely diagnose pneumonia. In this research, we predict pneumonia diseases from X-ray images, employing image segmentation and machine learning models. This work proposes a transfer learning pre-

Conclusions
Pneumonia is a common disease in many countries, especially in poor countries. This disease is defined as obstructive pneumonia which has the same impression on pulmonary radiographs as other lung diseases, which makes it hard to distinguish it from other pulmonary diseases, even for medical radiologists. Lately, image processing and deep learning models have been established to rapidly and precisely diagnose pneumonia. In this research, we predict pneumonia diseases from X-ray images, employing image segmentation and machine learning models. This work proposes a transfer learning pre-training classification model with a deep learning model for the binary and multiclassification of pneumonia from X-rays. The performance metrics of five CNN models were tested for the prediction of pneumonia. The proposed Improved BoxENet model with two transfer learning models with majority voting method outperforms other deep learning models for the databases for both whole and segmented X-rays, whereas DSNet3 is comparable for the segmented X-rays but consumes much more CPU time. The prediction accuracy, sensitivity and specificity for the multiclassification of pneumonia were measured to be 98.60%, 98.66% and 98.46% with X-ray segmentation and 97.6%, 99.66%, and 97.46% without segmentation respectively for multiclassification. It is also depicted that X-ray segmentation can considerably enhance multiclassification precision.