# Visualization and Interpretation of Convolutional Neural Network Predictions in Detecting Pneumonia in Pediatric Chest Radiographs

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Related Work

## 3. Materials and Methods

#### 3.1. Data Collection and Preprocessing

#### 3.2. Configuring CNNs for Pneumonia Detection

#### 3.2.1. Sequential CNN

^{−7}1 × 10

^{−1}], [0.7 0.99], and [1 × 10

^{−10}1 × 10

^{−2}] for the network depth, learning rate, momentum, and L2-regularization respectively. The objective function takes these variables as input, trains, validates and saves the optimal network that gives the minimum classification error on the test data. Figure 3 illustrates the steps involved in optimization.

#### 3.2.2. Residual CNN

#### 3.2.3. Inception CNN

#### 3.2.4. Customized VGG16

^{−6}1 × 10

^{−1}], [0.7 0.99], and [1 × 10

^{−10}1 × 10

^{−1}] for the learning rate, momentum, and L2-regularization respectively. Callbacks are used to view the internal states during training and retain the best performing model for analysis. We performed hold-out testing with the test data after every step. The performance of customized CNNs are evaluated in terms of the following performance metrics: (i) accuracy; (ii) AUC; (iii) precision; (iv) recall; (v) specificity; (vi) F-Score; and, (vii) Matthews Correlation Coefficient (MCC). We used the NIH Biowulf Linux cluster (https://hpc.nih.gov/) and the high performance computing facility at the National Library of Medicine (NLM) for computational analyses. Software frameworks included with Matlab R2017b are used to configure and evaluate the sequential CNN along with Keras and Tensorflow backend for other customized models used in this study.

#### 3.3. Visualization Studies

#### 3.3.1. Visual Explanation through Discriminative Localization

^{m}denote the GAP that spatially averages the m-th feature map from the deepest convolutional layer, and ${w}_{m}^{p}$ denote the weights connecting the m-th feature map to the output neuron corresponding to the expected class p. A prediction score S

_{p}at the output neuron is expressed as a weighted sum of GAP as shown in Equation (1).

_{m}(x, y) denotes the m-th feature map activation in the spatial location (x, y). The CAM for the class p denoted by CAM

_{p}is expressed as the weighted sum of the activations from all the feature maps with respect to the expected class p at the spatial location (x, y) as shown in Equation (2).

_{p}with respect to the m-th feature map as shown in Equation (5).

#### 3.3.2. Model-Agnostic Visual Explanations

^{d}be the explained instance, and k′ $\in $ {0, 1}

^{d}, the binary vector that denotes the presence/absence of a superpixel. Let g $\in $ G denote the explanation where G is a class of interpretable linear models. Let ℽ(g) denote the complexity measure associated with the explanation g $\in $ G. The value ℽ(g) denotes the number of non-zero coefficients for the linear model. Let m: ℝ

^{d}→ ℝ denote the explained model and m(k), the probability that k belongs to a given class. Let Π

_{k}(x) denote the measure of proximity between the instance x to k and P(m, g, Π

_{k}) denote the loss of g toward approximating m in the neighborhood defined by Π

_{k}. The value P(m, g, Π

_{k}) is minimized and the value of ℽ(g) remains low enough for interpretability. Equation (7) gives the explanations produced by LIME.

_{k}) is approximated by drawing samples weighted by Π

_{k}. Equation (8) shows an exponential kernel defined on the L2-distance function (J) with width €. For a given input perturbed sample b′ $\in $ {0, 1}

^{d′}containing a fraction of non-zero elements, the label for the explanation model m(b) is obtained by recovering the sample in the original representation b ∈ ℝ

^{d}as shown in Equation (9).

## 4. Results and Discussion

#### 4.1. Performance Evaluation of Customized CNNs

^{−3}, 0.9, and 1 × 10

^{−6}for the network depth, learning rate, momentum, and L2-regularization parameters respectively. The number of convolutional layer filters is increased by a factor of 2 each time a max-pooling layer is used, in order to ensure roughly the same number of computations in the network layers. Rectified Linear Unit (ReLU) layers are added to introduce non-linearity and prevent vanishing gradients during backpropagation [7].

#### 4.2. Visualization through Discriminative Localization

#### 4.3. Visual Explanations with LIME

## 5. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Le Roux, D.M.; Myer, L.; Nicol, M.P. Incidence and severity of childhood pneumonia in the first year of life in a South African birth cohort: The Drakenstein Child Health Study. Lancet Glob. Health
**2015**, 3, e95–e103. [Google Scholar] [CrossRef] - Mcluckie, A. Respiratory Disease and Its Management, 1st ed.; Springer: London, UK, 2009; pp. 51–59. ISBN 978-1-84882-094-4. [Google Scholar]
- Cherian, T.; Mulholland, E.K.; Carlin, J.B.; Ostensen, H.; Amin, R.; De Campo, M.; Greenberg, D.; Lagos, R.; Lucero, M.; Madhi, S.A.; et al. Standardized interpretation of paediatric chest radiographs for the diagnosis of pneumonia in epidemiological studies. Bull. World Health Organ.
**2005**, 83, 353–359. [Google Scholar] [PubMed] - Wang, X.; Peng, Y.; Lu, L.; Lu, Z.; Bagheri, M.; Summers, R.M. ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 3462–3471. [Google Scholar]
- Karargyris, A.; Siegelman, J.; Tzortzis, D.; Jaeger, S.; Candemir, S.; Xue, Z.; KC, S.; Vajda, S.; Antani, S.K.; Folio, L.; et al. Combination of texture and shape features to detect pulmonary abnormalities in digital chest X-rays. Int. J. Comput. Assist. Radiol. Surg.
**2016**, 11, 99–106. [Google Scholar] [CrossRef] [PubMed] - Neuman, M.I.; Lee, E.Y.; Bixby, S.; Diperna, S.; Hellinger, J.; Markowitz, R.; Servaes, S.; Monuteaux, M.C.; Shah, S.S. Variability in the Interpretation of Chest Radiographs for the Diagnosis of Pneumonia in Children. J. Hosp. Med.
**2012**, 7, 294–298. [Google Scholar] [CrossRef] [PubMed] - Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), Lake Tahoe, NV, USA, 3–6 December 2012; pp. 1097–1105. [Google Scholar]
- Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Li, F.F. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]
- Zeiler, M.D.; Fergus, R. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland, 6–12 September 2014; Springer: Cham, Switzerland, 2014; pp. 818–833. [Google Scholar]
- Bar, Y.; Diamant, I.; Wolf, L.; Lieberman, S.; Konen, E.; Greenspan, H. Chest Pathology Detection Using Deep Learning with Non-Medical Training. In Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI), Brooklyn, NY, USA, 16–19 April 2015; pp. 294–297. [Google Scholar]
- Oliveira, L.L.G.; Silva, S.A.E.; Ribeiro, L.H.V.; De Oliveira, R.M.; Coelho, C.J.; Ana Lúcia, A.L.S. Computer-Aided Diagnosis in Chest Radiography for Detection of Childhood Pneumonia. Int. J. Med. Inform.
**2008**, 77, 555–564. [Google Scholar] [CrossRef] [PubMed] - Abe, H.; Macmahon, H.; Shiraishi, J.; Li, Q.; Engelmann, R.; Doi, K. Computer-aided diagnosis in chest radiology. Semin. Ultrasound CT MR
**2004**, 25, 432–437. [Google Scholar] [CrossRef] [PubMed] - Giger, M.; MacMahon, H. Image processing and computer-aided diagnosis. Radiol. Clin. N. Am.
**1996**, 34, 565–596. [Google Scholar] [PubMed] - Monnier-Cholley, L.; MacMahon, H.; Katsuragawa, S.; Morishita, J.; Ishida, T.; Doi, K. Computer-aided diagnosis for detection of interstitial opacities on chest radiographs. AJR Am. J. Roentgenol.
**1998**, 171, 1651–1656. [Google Scholar] [CrossRef] [PubMed] - Kermany, D.S.; Goldbaum, M.; Cai, W.; Valentim, C.C.S.; Liang, H.; Baxter, S.L.; McKeown, A.; Yang, G.; Wu, X.; Yan, F.; et al. Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning. Cell
**2018**, 172, 1122–1131. [Google Scholar] [CrossRef] [PubMed] - Rajpurkar, P.; Irvin, J.; Zhu, K.; Yang, B.; Mehta, H.; Duan, T.; Ding, D.; Bagul, A.; Langlotz, C.; Shpanskaya, K.; et al. CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning. arXiv. 2018. Available online: https://arxiv.org/abs/1711.05225 (accessed on 23 January 2018).
- Guan, Q.; Huang, Y.; Zhong, Z.; Zheng, Z.; Zheng, L.; Yang, Y. Diagnose like a Radiologist: Attention Guided Convolutional Neural Network for Thorax Disease Classification. arXiv. 2018. Available online: https://arxiv.org/abs/1801.09927v1 (accessed on 17 June 2018).
- Candemir, S.; Jaeger, S.; Palaniappan, K.; Musco, J.P.; Singh, R.K.; Xue, Z.; Karargyris, A.; Antani, S.; Thoma, G.; McDonald, C.J. Lung Segmentation in Chest Radiographs Using Anatomical Atlases with Nonrigid Registration. IEEE Trans. Med. Imaging
**2014**, 33, 577–590. [Google Scholar] [CrossRef] [PubMed] - Candemir, S.; Antani, S.; Jaeger, S.; Browning, R.; Thoma, G. Lung Boundary Detection in Pediatric Chest X-Rays. In Proceedings of the SPIE Medical Imaging, Orlando, FL, USA, 21–26 February 2015; Volume 9418, p. 94180Q. [Google Scholar]
- Liu, C.; Yuen, J.; Torralba, A. SIFT Flow: Dense Correspondence across Scenes and Its Applications. IEEE Trans. Pattern Anal. Mach. Intell.
**2011**, 33, 978–994. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Snoek, J.; Rippel, O.; Adams, R.P. Scalable Bayesian Optimization Using Deep Neural Networks. In Proceedings of the International Conference on Machine Learning (ICML), Lille, France, 6–11 July 2015; pp. 2171–2180. [Google Scholar]
- Deep Learning Using Bayesian Optimization. Available online: https://www.mathworks.com/help/nnet/examples/deep-learning-using-bayesian-optimization.html (accessed on 14 January 2018).
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the Inception Architecture for Computer Vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2818–2826. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA, 7–9 May 2015; pp. 1–32. [Google Scholar]
- Bergstra, J.; Bengio, Y. Random Search for Hyper-Parameter Optimization. JMLR
**2012**, 13, 281–305. [Google Scholar] - Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning Deep Features for Discriminative Localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2921–2929. [Google Scholar]
- Selvaraju, R.R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In Proceedings of the IEEE International Conference of Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 618–626. [Google Scholar]
- Ribeiro, M.T.; Singh, S.; Guestrin, C. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), San Francisco, CA, USA, 13–17 August 2016; pp. 1135–1144. [Google Scholar]
- Sharma, S.; Maycher, B.; Eschun, G. Radiological imaging in pneumonia: Recent innovations. Curr. Opin. Pulm. Med.
**2007**, 13, 159–169. [Google Scholar] [CrossRef] [PubMed]

**Figure 1.**Pediatric CXRs: (

**a**) Normal CXR showing clear lungs with no abnormal opacification; (

**b**) Bacterial pneumonia exhibiting focal lobar consolidation in the right upper lobe; (

**c**) Viral pneumonia manifesting with diffuse interstitial patterns in both lungs.

**Figure 4.**The architecture of customized residual CNN: (

**a**) Residual block; (

**b**) Customized residual CNN stacked with six residual blocks.

**Figure 5.**The architecture of customized InceptionV3 CNN: (

**a**) InceptionV3 module; (

**b**) Customized Inception CNN stacked with six InceptionV3 modules.

**Figure 6.**VGG16 model truncated at the deepest convolutional layer and added with a GAP and dense layer.

**Figure 8.**Confusion matrices for the performance of the customized VGG16 model: (

**a**) Normal v. Pneumonia; (

**b**) Bacterial v. Viral Pneumonia.

**Figure 9.**ROC curves demonstrating the performance of the customized VGG16 model: (

**a**) Normal v. Pneumonia; (

**b**) Bacterial v. Viral Pneumonia.

**Figure 10.**Performance of customized VGG16 model in multiclass classification: (

**a**) Confusion matrix; (

**b**) ROC curves.

**Figure 11.**Visual explanations through gradient-based localization using CAM: (

**a**) Input CXRs; (

**b**) Bounding boxes localizing regions of activations; (

**c**) CAM showing heat maps superimposed on the original CXRs; (

**d**) Automatically segmented lung masks; (

**e**) CAM showing heat maps superimposed on the cropped lungs.

**Figure 12.**Visual explanations through gradient-based localization using grad-CAM: (

**a**) Input CXRs; (

**b**) Bounding boxes localizing regions of activations; (

**c**) Grad-CAM showing heat maps superimposed on the original CXRs; (

**d**) Automatically segmented lung masks; (

**e**) Grad-CAM showing heat maps superimposed on the cropped lungs.

**Figure 13.**Visual explanations through average-CAM: (

**a**) Bacterial and viral CXR (top and bottom); (

**b**) Average-CAM localizing class-specific ROI with bounding boxes highlighting the regions of maximum activation; (

**c**) Automatically segmented lung masks; (

**d**) Average-CAM localizing class-specific ROI with the extracted lung regions.

**Figure 14.**Visual explanations through average-grad-CAM: (

**a**) Bacterial and viral CXR (top and bottom); (

**b**) Average-grad-CAM localizing class-specific ROI with bounding boxes highlighting the regions of maximum activation; (

**c**) Automatically segmented lung masks; (

**d**) Average-grad-CAM localizing class-specific ROI with the extracted lung regions.

**Figure 15.**Visual explanations through LIME: (

**a**) Input CXRs; (

**b**) Automatically segmented lung masks; (

**c**) Copped lung regions; (

**d**) Superpixels with the highest positive weights with the others greyed out; (

**e**) Superpixels with the highest positive weights are superimposed on the cropped lungs.

Category | Training Samples | Test Samples | File Type |
---|---|---|---|

Normal | 1349 | 234 | JPG |

Bacterial | 2538 | 242 | JPG |

Viral | 1345 | 148 | JPG |

**Table 2.**Optimal values for the hyperparameters of the customized residual and inception CNNs obtained through a randomized grid search.

Model | Learning Rate | Momentum | L2 Regularization |
---|---|---|---|

Residual CNN | 1 × 10^{−3} | 0.9 | 1 × 10^{−6} |

Inception CNN | 1 × 10^{−2} | 0.95 | 1 × 10^{−4} |

Customized VGG16 | 1 × 10^{−4} | 0.99 | 1 × 10^{−6} |

Task | Data | Models | Accuracy | AUC | Precision | Recall | Specificity | F-Score | MCC |
---|---|---|---|---|---|---|---|---|---|

Normal vs. Pneumonia | Baseline | Customized VGG16 | 0.957 | 0.990 | 0.951 | 0.983 | 0.915 | 0.967 | 0.908 |

Sequential | 0.943 | 0.983 | 0.920 | 0.980 | 0.855 | 0.957 | 0.878 | ||

Residual | 0.910 | 0.967 | 0.908 | 0.954 | 0.838 | 0.931 | 0.806 | ||

Inception | 0.886 | 0.922 | 0.887 | 0.939 | 0.800 | 0.913 | 0.755 | ||

Cropped ROI | Customized VGG16 | 0.962 | 0.993 | 0.977 | 0.962 | 0.962 | 0.970 | 0.918 | |

Sequential | 0.941 | 0.984 | 0.930 | 0.995 | 0.877 | 0.955 | 0.873 | ||

Residual | 0.917 | 0.971 | 0.913 | 0.959 | 0.847 | 0.936 | 0.820 | ||

Inception | 0.897 | 0.932 | 0.896 | 0.947 | 0.817 | 0.921 | 0.778 | ||

Bacterial vs. Viral Pneumonia | Baseline | Customized VGG16 | 0.936 | 0.962 | 0.920 | 0.984 | 0.860 | 0.951 | 0.862 |

Sequential | 0.928 | 0.954 | 0.909 | 0.984 | 0.838 | 0.946 | 0.848 | ||

Residual | 0.897 | 0.921 | 0.880 | 0.967 | 0.784 | 0.922 | 0.780 | ||

Inception | 0.854 | 0.901 | 0.841 | 0.934 | 0.714 | 0.886 | 0.675 | ||

Cropped ROI | Customized VGG16 | 0.936 | 0.962 | 0.920 | 0.984 | 0.860 | 0.951 | 0.862 | |

Sequential | 0.928 | 0.956 | 0.909 | 0.984 | 0.838 | 0.946 | 0.848 | ||

Residual | 0.908 | 0.933 | 0.888 | 0.976 | 0.798 | 0.930 | 0.802 | ||

Inception | 0.872 | 0.919 | 0.853 | 0.959 | 0.730 | 0.903 | 0.725 | ||

Normal vs. Bacterial vs. Viral Pneumonia | Baseline | Customized VGG16 | 0.917 | 0.938 | 0.917 | 0.905 | 0.958 | 0.911 | 0.873 |

Sequential | 0.896 | 0.922 | 0.888 | 0.885 | 0.948 | 0.887 | 0.841 | ||

Residual | 0.861 | 0.887 | 0.868 | 0.882 | 0.933 | 0.875 | 0.809 | ||

Inception | 0.809 | 0.846 | 0.753 | 0.848 | 0.861 | 0.798 | 0.688 | ||

Cropped ROI | Customized VGG16 | 0.918 | 0.939 | 0.920 | 0.900 | 0.960 | 0.910 | 0.876 | |

Sequential | 0.897 | 0.923 | 0.898 | 0.898 | 0.949 | 0.898 | 0.844 | ||

Residual | 0.879 | 0.909 | 0.883 | 0.890 | 0.941 | 0.887 | 0.825 | ||

Inception | 0.821 | 0.865 | 0.778 | 0.855 | 0.878 | 0.815 | 0.714 |

Task | Model | Accuracy | AUC | Precision | Recall | Specificity | F-Score | MCC |
---|---|---|---|---|---|---|---|---|

Normal v. Pneumonia | Customized VGG16 | 0.962 | 0.993 | 0.977 | 0.962 | 0.962 | 0.970 | 0.918 |

Kermany et al. | 0.928 | 0.968 | - | 0.932 | 0.901 | - | - | |

Bacterial v. Viral Pneumonia | Customized VGG16 | 0.936 | 0.962 | 0.920 | 0.984 | 0.860 | 0.951 | 0.862 |

Kermany et al. | 0.907 | 0.940 | - | 0.886 | 0.909 | - | - | |

Normal v. Bacterial v. Viral Pneumonia | Customized VGG16 | 0.918 | 0.939 | 0.920 | 0.900 | 0.960 | 0.910 | 0.876 |

Kermany et al. | - | - | - | - | - | - | - |

*****Bold numbers indicate superior performance.

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Rajaraman, S.; Candemir, S.; Kim, I.; Thoma, G.; Antani, S.
Visualization and Interpretation of Convolutional Neural Network Predictions in Detecting Pneumonia in Pediatric Chest Radiographs. *Appl. Sci.* **2018**, *8*, 1715.
https://doi.org/10.3390/app8101715

**AMA Style**

Rajaraman S, Candemir S, Kim I, Thoma G, Antani S.
Visualization and Interpretation of Convolutional Neural Network Predictions in Detecting Pneumonia in Pediatric Chest Radiographs. *Applied Sciences*. 2018; 8(10):1715.
https://doi.org/10.3390/app8101715

**Chicago/Turabian Style**

Rajaraman, Sivaramakrishnan, Sema Candemir, Incheol Kim, George Thoma, and Sameer Antani.
2018. "Visualization and Interpretation of Convolutional Neural Network Predictions in Detecting Pneumonia in Pediatric Chest Radiographs" *Applied Sciences* 8, no. 10: 1715.
https://doi.org/10.3390/app8101715