Diagnosing Malaria Patients with Plasmodium falciparum and vivax Using Deep Learning for Thick Smear Images
Abstract
:1. Introduction
1.1. Literature Review
1.2. Contribution
2. Materials and Methods
2.1. Data Set
2.2. Methodology
2.2.1. Adapting Mask R-CNN for Parasite Detection
2.2.2. Patch-Level Two-Class Classification
2.2.3. Proposed PlasmodiumVF-Net Framework
- Read an image out of N images per patient.
- Detect in parallel two sets of candidate patches using Mask R-CNN for both P. falciparum and P. vivax using the two two-class detectors. We apply here the two detectors because we have no prior knowledge about the parasite species causing the infection or whether the patient is uninfected.
- Filter out false positives using two binary classifiers named PV_U_ResNet50 and PF_U_ResNet50.
- Set two flags, PV and PF, to indicate whether the framework detects more than one parasite for P. vivax and P. falciparum, respectively.
- Based on PV and PF, there are four possibilities:
- (a)
- If both flags are zero, our proposed PlasmodiumVF-Net reports the image as uninfected and increases the counter, Sum_U, of the number of uninfected images by one.
- (b)
- When PV = 0 and PF = 1, then PlasmodiumVF-Net reports that the image contains only P. falciparum parasites.
- (c)
- When PV = 1 and PF = 0, then PlasmodiumVF-Net reports that the image contains only P. vivax parasites.
- (d)
- If both flags are one, this means that there are candidate patches for both P. falciparum and P. vivax. In this case, all of the candidates need to be tested by the VF_ResNet50 classifier. After testing all the patches, the prediction probabilities are aggregated. The averages, represented by Avg_PV and Avg_PF, are computed by dividing the aggregated probabilities by the number of patches. VF_ResNet50 classifies patches as P. vivax if they have probabilities of less than 0.5 and as P. falciparum if their probabilities are higher than 0.5. Consequently, if Avg_PV is less than Avg_PF, then the image is considered to contain P. vivax; otherwise, P. falciparum.
- At this point, we have an image-level decision, and PlasmodiumVF-Net needs to check some parameters and conditions to produce a patient-level decision. TotalPV and TotalPF accumulate the total number of patches when PlasmodiumVF-Net found that the image is infected by P. vivax or P. falciparum, respectively.
- If all N images are processed, go to Step 8, otherwise return to Step 1 to process a new image from the current patient.
- If the PlasmodiumVF-Net found that more than half of the images of the current patient are uninfected based on U_patients_score, which is calculated by dividing the total number of uninfected images by N, then it considers the patient as uninfected; otherwise, it proceeds to the final step.
- Calculate the PF_patient_score and PV_patient_score by dividing the total number of detected patches, represented by TotalPF and TotalPV, by N. The PlasmodiumVF-Net decides that the patient is infected by P. falciparum parasites if the PF_patient_score is higher; otherwise, the patient is considered to be infected by P. vivax parasites.
3. Results and Discussion
3.1. Experimental Network Settings
3.2. Quantitative Performance Evaluation and Discussion
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- WHO. World Malaria Report 2020; World Health Organization: Geneva, Switzerland, 2020. [Google Scholar]
- Tangpukdee, N.; Duangdee, C.; Wilairatana, P.; Krudsood, S. Malaria diagnosis: A brief review. Korean J. Parasitol. 2009, 47, 93. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- WHO. Guidelines for the Treatment of Malaria; World Health Organization: Geneva, Switzerland, 2015. [Google Scholar]
- Bloland, P.B.; World Health Organization. Drug Resistance in Malaria; Technical Report; World Health Organization: Geneva, Switzerland, 2001. [Google Scholar]
- Davenport, T.; Kalakota, R. The potential for artificial intelligence in healthcare. Future Healthc. J. 2019, 6, 94. [Google Scholar] [CrossRef] [Green Version]
- Zemouri, R.; Zerhouni, N.; Racoceanu, D. Deep learning in the biomedical applications: Recent and future status. Appl. Sci. 2019, 9, 1526. [Google Scholar] [CrossRef] [Green Version]
- Rong, G.; Mendez, A.; Assi, E.B.; Zhao, B.; Sawan, M. Artificial intelligence in healthcare: Review and prediction case studies. Engineering 2020, 6, 291–301. [Google Scholar] [CrossRef]
- Poostchi, M.; Silamut, K.; Maude, R.J.; Jaeger, S.; Thoma, G. Image analysis and machine learning for detecting malaria. Transl. Res. 2018, 194, 36–55. [Google Scholar] [CrossRef] [Green Version]
- Das, D.K.; Mukherjee, R.; Chakraborty, C. Computational microscopic imaging for malaria parasite detection: A systematic review. J. Microsc. 2015, 260, 1–19. [Google Scholar] [CrossRef]
- Rajaraman, S.; Antani, S.K.; Poostchi, M.; Silamut, K.; Hossain, M.A.; Maude, R.J.; Jaeger, S.; Thoma, G.R. Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images. PeerJ 2018, 6, e4568. [Google Scholar] [CrossRef] [PubMed]
- Kassim, Y.M.; Jaeger, S. A Cell Augmentation Tool for Blood Smear Analysis. In Proceedings of the IEEE Applied Imagery Pattern Recognition Workshop, Washington, DC, USA, 13–15 October 2020; pp. 1–6. [Google Scholar]
- Kassim, Y.M.; Palaniappan, K.; Yang, F.; Poostchi, M.; Palaniappan, N.; Maude, R.J.; Antani, S.; Jaeger, S. Clustering-Based Dual Deep Learning Architecture for Detecting Red Blood Cells in Malaria Diagnostic Smears. IEEE J. Biomed. Health Inform. 2021, 25, 1735–1746. [Google Scholar] [CrossRef]
- Tsai, M.; Yu, S.; Chan, Y.; Jen, C. Blood smear image based malaria parasite and infected-erythrocyte detection and segmentation. J. Med. Syst. 2015, 39, 118. [Google Scholar] [CrossRef]
- May, Z.; Aziz, S.S.A.M.; Salamat, R. Automated quantification and classification of malaria parasites in thin blood smears. In Proceedings of the IEEE International Conference on Signal and Image Processing Applications, Melaka, Malaysia, 8–10 October 2013; pp. 369–373. [Google Scholar]
- Vijayalakshmi, A.; Rajesh Kanna, B. Deep learning approach to detect malaria from microscopic images. Multimed. Tools Appl. 2020, 79, 15297–15317. [Google Scholar] [CrossRef]
- Mathison, B.A.; Pritt, B.S. Update on malaria diagnostics and test utilization. J. Clin. Microbiol. 2017, 55, 2009–2017. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kaewkamnerd, S.; Intarapanich, A.; Pannarat, M.; Chaotheing, S.; Uthaipibull, C.; Tongsima, S. Detection and classification device for malaria parasites in thick-blood films. IEEE Int. Conf. Intell. Data Acquis. Adv. Comput. Syst. 2011, 1, 435–438. [Google Scholar]
- Elter, M.; Haßlmeyer, E.; Zerfaß, T. Detection of malaria parasites in thick blood films. In Proceedings of the International conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA, USA, 30 August–3 September 2011; pp. 5140–5144. [Google Scholar]
- Purnama, I.K.E.; Rahmanti, F.Z.; Purnomo, M.H. Malaria parasite identification on thick blood film using genetic programming. In Proceedings of the International Conference on Instrumentation, Communications, Information Technology and Biomedical Engineering, Bandung, Indonesia, 7–8 November 2013; pp. 194–198. [Google Scholar]
- Quinn, J.A.; Andama, A.; Munabi, I.; Kiwanuka, F. Automated blood smear analysis for mobile malaria diagnosis. Mob. Point Care Monit. Diagn. Device Des. 2014, 31, 115. [Google Scholar]
- Chakrabortya, K.; Chattopadhyayb, A.; Chakrabarti, A.; Acharyad, T.; Dasguptae, A.K. A combined algorithm for malaria detection from thick smear blood slides. J. Health Med. Inform. 2015, 6, 645–652. [Google Scholar] [CrossRef]
- Delahunt, C.B.; Mehanian, C.; Hu, L.; McGuire, S.K.; Champlin, C.R.; Horning, M.P.; Wilson, B.K.; Thompon, C.M. Automated microscopy and machine learning for expert-level malaria field diagnosis. In Proceedings of the IEEE Global Humanitarian Technology Conference, Seattle, WA, USA, 8–11 October 2015; pp. 393–399. [Google Scholar]
- Rosado, L.; Da Costa, J.; Elias, D.; Cardoso, J.S. Automated Detection of Malaria Parasites on Thick Blood Smears via Mobile Devices. Procedia Comput. Sci. 2016, 90, 138–144. [Google Scholar] [CrossRef]
- Dave, I.R.; Upla, K.P. Computer aided diagnosis of malaria disease for thin and thick blood smear microscopic images. In Proceedings of the International Conference on Signal Processing and Integrated Networks, Noida, India, 2–3 February 2017; pp. 561–565. [Google Scholar]
- Mehanian, C.; Jaiswal, M.; Delahunt, C.; Thompson, C.; Horning, M.; Hu, L.; McGuire, S.; Ostbye, T.; Mehanian, M.; Wilson, B.; et al. Computer-Automated Malaria Diagnosis and Quantitation Using Convolutional Neural Networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy, 22–29 October 2017; pp. 116–125. [Google Scholar]
- Yang, F.; Poostchi, M.; Yu, H.; Zhou, Z.; Silamut, K.; Yu, J.; Maude, R.J.; Jaeger, S.; Antani, S. Deep Learning for Smartphone-based Malaria Parasite Detection in Thick Blood Smears. IEEE J. Biomed. Health Inform. 2019, 24, 1427–1438. [Google Scholar] [CrossRef] [PubMed]
- Chibuta, S.; Acar, A.C. Real-time malaria parasite screening in thick blood smears for low-resource setting. J. Digit. Imaging 2020, 33, 763–775. [Google Scholar] [CrossRef] [PubMed]
- Abdurahman, F.; Fante, K.A.; Aliy, M. Malaria parasite detection in thick blood smear microscopic images using modified YOLOV3 and YOLOV4 models. BMC Bioinform. 2021, 22, 112. [Google Scholar] [CrossRef]
- Horning, M.P.; Delahunt, C.B.; Bachman, C.M.; Luchavez, J.; Luna, C.; Hu, L.; Jaiswal, M.S.; Thompson, C.M.; Kulhare, S.; Janko, S.; et al. Performance of a fully-automated system on a WHO malaria microscopy evaluation slide set. Malar. J. 2021, 20, 110. [Google Scholar] [CrossRef]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 2015, 28, 91–99. [Google Scholar] [CrossRef] [Green Version]
- Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 580–587. [Google Scholar]
- Girshick, R. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 11–18 December 2015; pp. 1440–1448. [Google Scholar]
- Redmon, J.; Farhadi, A. YOLO9000: Better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 7263–7271. [Google Scholar]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
- Vuola, A.O.; Akram, S.U.; Kannala, J. Mask R-CNN and U-Net ensembled for nuclei segmentation. In Proceedings of the IEEE International Symposium on Biomedical Imaging, Venice, Italy, 8–11 April 2019; pp. 208–212. [Google Scholar]
- Couteaux, V.; Si-Mohamed, S.; Nempont, O.; Lefevre, T.; Popoff, A.; Pizaine, G.; Villain, N.; Bloch, I.; Cotten, A.; Boussel, L. Automatic knee meniscus tear detection and orientation classification with Mask R-CNN. Diagn. Interv. Imaging 2019, 100, 235–242. [Google Scholar] [CrossRef] [PubMed]
- Johnson, J.W. Automatic nucleus segmentation with Mask R-CNN. In Science and Information Conference; Springer: Cham, Switzerland, 2019; pp. 399–407. [Google Scholar]
- Kopelowitz, E.; Engelhard, G. Lung Nodules Detection and Segmentation Using 3D Mask R-CNN. arXiv 2019, arXiv:1907.07676. [Google Scholar]
- Almubarak, H.; Bazi, Y.; Alajlan, N. Two-stage Mask R-CNN approach for detecting and segmenting the optic nerve head, optic disc, and optic cup in fundus images. Appl. Sci. 2020, 10, 3833. [Google Scholar] [CrossRef]
- Kassim, Y.M.; Byrne, M.E.; Burch, C.; Mote, K.; Hardin, J.; Larsen, D.R.; Palaniappan, K. Small object bird detection in infrared drone videos using Mask R-CNN deep learning. Electron. Imaging 2020, 2020, 85-1–85-8. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar]
- Iandola, F.N.; Han, S.; Moskewicz, M.W.; Ashraf, K.; Dally, W.J.; Keutzer, K. SqueezeNet: AlexNet-level accuracy with 50× fewer parameters and <0.5 MB model size. arXiv 2016, arXiv:1602.07360. [Google Scholar]
- Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 2818–2826. [Google Scholar]
- Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4700–4708. [Google Scholar]
- Szegedy, C.; Ioffe, S.; Vanhoucke, V.; Alemi, A. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017. [Google Scholar]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 3–6 December 2012; pp. 1097–1105. [Google Scholar]
- Yu, H.; Yang, F.; Rajaraman, S.; Ersoy, I.; Moallem, G.; Poostchi, M.; Palaniappan, K.; Antani, S.; Maude, R.; Jaeger, S. Malaria Screener: A smartphone application for automated malaria screening. BMC Infect. Dis. 2020, 20, 1471–2334. [Google Scholar] [CrossRef]
P. vivax | P. falciparum | |
---|---|---|
Number of patients | 150 | 150 |
Number of images | 3013 | 1818 |
Number of parasites | 43,042 | 84,961 |
Parasite radius range | 6–144 | 2–96 |
Average parasite radius | 42 | 22 |
Number of parasites per image | 1–98 | 1–341 |
Average number of parasites per image | 14 | 47 |
Number of images per patient | 15–30 | 3–22 |
Average number of images per patient | 20 | 12 |
Number of parasites per patient | 24–1345 | 8–3130 |
Average number of parasites per patient | 287 | 522 |
Variable | Definition |
---|---|
PV_U_ResNet50 | ResNet50 classifier is trained to classify patches as either P. vivax or uninfected |
PF_U_ResNet50 | ResNet50 classifier is trained to classify patches as either P. falciparum or uninfected |
PV | This flag is set if more than one P. vivax parasite is still detected after all false positives are filtered out by the PV_U_ResNet50 classifier |
PF | This flag is set if more than one P. falciparum parasite is still detected after all false positives are filtered out by the PF_U_ResNet50 classifier |
VF_ResNet50 | ResNet50 classifier is trained to classify patches as either P. falciparum or P. vivax |
Avg_PV | Sum of all probabilities for detected P. vivax patches divided by the number of patches detected for a single image |
Avg_PF | Sum of all probabilities for detected P. falciparum patches divided by the number of patches detected for a single image |
TotalPV | Total number of detected P. vivax patches for all images of a single patient |
TotalPF | Total number of detected P. falciparum patches for all images of a single patient |
Sum_U | Total number of uninfected images |
U_patients_score | Total number of uninfected images divided by the total number of images for a single patient |
PV_patient_score | TotalPV/number of images for a single patient |
PF_patient_score | TotalPF/number of images for a single patient |
P. vivax | P. falciparum | |||
---|---|---|---|---|
Detection Rate Using a Three-Class Classifier | Detection Rate Using a Two-Class Classifier | Detection Rate Using a Three-Class Classifier | Detection Rate Using a Two-Class Classifier | |
Fold1 | 85.58 | 92.94 | 61.34 | 83.76 |
Fold2 | 82.04 | 90.60 | 60.77 | 86.83 |
Fold3 | 88.41 | 96.70 | 67.41 | 87.45 |
Fold4 | 89.81 | 96.22 | 73.15 | 90.02 |
Fold5 | 88.93 | 93.68 | 69.89 | 90.87 |
Avg. | 86.95 | 94.03 | 66.51 | 87.79 |
(a) GoogleNet Classification Experiments with Average Accuracy Equal to 99.15% [42]. | ||
---|---|---|
P. falciparum | P. vivax | |
P. falciparum | 84,961 | 1087 |
P. vivax | 0 | 41,955 |
(b) SqueezeNet Classification Experiments with Average Accuracy Equal to 99.28% [43]. | ||
P. falciparum | P. vivax | |
P. falciparum | 84,961 | 912 |
P. vivax | 0 | 42,130 |
(c) ResNet50 Classification Experiments with Average Accuracy Equal to 99.98% [41]. | ||
P. falciparum | P. vivax | |
P. falciparum | 84,961 | 19 |
P. vivax | 0 | 43,023 |
(d) InceptionV3 Classification Experiments with Average Accuracy Equal to 96.76% [44]. | ||
P. falciparum | P. vivax | |
P. falciparum | 84,961 | 4141 |
P. vivax | 0 | 38,901 |
(a) Pipeline 1, Image-Level Identification Results with Accuracy = 68.4% | ||
---|---|---|
P. falciparum | P. vivax | |
P. falciparum | 1245 | 952 |
P. vivax | 573 | 2061 |
Sum of images | 1818 | 3013 |
(b) Pipeline 1, Patient-Level Identification Results with Accuracy = 78.7% | ||
P. falciparum | P. vivax | |
P. falciparum | 131 | 45 |
P. vivax | 19 | 105 |
Sum of patients | 150 | 150 |
(c) Pipeline 2, Image-Level Identification Results with Accuracy = 77.8% | ||
P. falciparum | P. vivax | |
P. falciparum | 1700 | 955 |
P. vivax | 118 | 2058 |
Sum of images | 1818 | 3013 |
(d) Pipeline 2, Patient-Level Identification Results with Accuracy = 83% | ||
P. falciparum | P. vivax | |
P. falciparum | 141 | 42 |
P. vivax | 9 | 108 |
Sum of patients | 150 | 150 |
(e) Pipeline 3, Image-Level Identification Results with Accuracy = 83.5% | ||
P. falciparum | P. vivax | |
P. falciparum | 1675 | 653 |
P. vivax | 143 | 2360 |
Sum of images | 1818 | 3013 |
(f) Pipeline 3, Patient-Level Identification Results with Accuracy = 91% | ||
P. falciparum | P. vivax | |
P. falciparum | 148 | 25 |
P. vivax | 2 | 125 |
Sum of patients | 150 | 150 |
(g) PlasmodiumVF-Net, Image-Level Identification Results | ||
with Accuracy = 90.8% | ||
P. falciparum | P. vivax | |
P. falciparum | 1756 | 375 |
P. vivax | 52 | 2630 |
Sum of images | 1808 | 3005 |
(h) PlasmodiumVF-Net, Patient-Level Identification Results | ||
with Accuracy = 96.7% | ||
P. falciparum | P. vivax | |
P. falciparum | 148 | 8 |
P. vivax | 2 | 142 |
Sum of patients | 150 | 150 |
(a) Summation of Confusion Matrices for Five-Fold Cross-Validation of | |||
---|---|---|---|
PlasmodiumVF-Net with Average Accuracy Equal to 83.9% on Image Level | |||
P. falciparum | P. vivax | Uninfected | |
P. falciparum | 1714 | 337 | 317 |
P. vivax | 36 | 2537 | 66 |
Uninfected | 68 | 139 | 758 |
Sum of Images | 1818 | 3013 | 1141 |
(b) Summation of Confusion Matrices for Five-Fold Cross-Validation of | |||
PlasmodiumVF-Net with Average Accuracy Equal to 92.3% on Patient Level | |||
P. falciparum | P. vivax | Uninfected | |
P. falciparum | 145 | 8 | 12 |
P. vivax | 2 | 141 | 1 |
Uninfected | 3 | 1 | 37 |
Sum of Patients | 150 | 150 | 50 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kassim, Y.M.; Yang, F.; Yu, H.; Maude, R.J.; Jaeger, S. Diagnosing Malaria Patients with Plasmodium falciparum and vivax Using Deep Learning for Thick Smear Images. Diagnostics 2021, 11, 1994. https://doi.org/10.3390/diagnostics11111994
Kassim YM, Yang F, Yu H, Maude RJ, Jaeger S. Diagnosing Malaria Patients with Plasmodium falciparum and vivax Using Deep Learning for Thick Smear Images. Diagnostics. 2021; 11(11):1994. https://doi.org/10.3390/diagnostics11111994
Chicago/Turabian StyleKassim, Yasmin M., Feng Yang, Hang Yu, Richard J. Maude, and Stefan Jaeger. 2021. "Diagnosing Malaria Patients with Plasmodium falciparum and vivax Using Deep Learning for Thick Smear Images" Diagnostics 11, no. 11: 1994. https://doi.org/10.3390/diagnostics11111994