Neonatal Jaundice Diagnosis Using a Smartphone Camera Based on Eye, Skin, and Fused Features with Transfer Learning
Abstract
:1. Introduction
2. Related Works
3. Material and Methods
3.1. Dataset
3.2. Preprocessing
3.3. ROI Detection and Segmentation
3.4. Transfer Learning
3.4.1. Feature Extraction
3.4.2. Classification
3.5. Traditional Machine Learning
3.5.1. Feature Extraction
3.5.2. Classification
3.6. Evaluation
4. Results and Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Pagliari, C.; Sloan, D.; Gregor, P.; Sullivan, F.; Detmer, D.; Kahan, J.P.; Oortwijn, W.; MacGillivray, S. What is eHealth (4): A scoping exercise to map the field. J. Med. Internet Res. 2005, 7, e391. [Google Scholar] [CrossRef] [PubMed]
- Hsu, C.-M.; Hsu, C.-C.; Hsu, Z.-M.; Shih, F.-Y.; Chang, M.-L.; Chen, T.-H. Colorectal Polyp Image Detection and Classification through Grayscale Images and Deep Learning. Sensors 2021, 21, 5995. [Google Scholar] [CrossRef] [PubMed]
- Khasawneh, N.; Fraiwan, M.; Fraiwan, L.; Khassawneh, B.; Ibnian, A. Detection of COVID-19 from Chest X-ray Images Using Deep Convolutional Neural Networks. Sensors 2021, 21, 5940. [Google Scholar] [CrossRef] [PubMed]
- Umair, M.; Khan, M.S.; Ahmed, F.; Baothman, F.; Alqahtani, F.; Alian, M.; Ahmad, J. Detection of COVID-19 Using Transfer Learning and Grad-CAM Visualization on Indigenously Collected X-ray Dataset. Sensors 2021, 21, 5813. [Google Scholar] [CrossRef] [PubMed]
- Ahmedt-Aristizabal, D.; Armin, M.A.; Denman, S.; Fookes, C.; Petersson, L. Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future. Sensors 2021, 21, 4758. [Google Scholar] [CrossRef] [PubMed]
- Maisels, M.J. Neonatal jaundice. Pediatr. Rev. 2006, 27, 443. [Google Scholar] [CrossRef] [PubMed]
- Mansor, M.N.; Hariharan, M.; Basah, S.N.; Yaacob, S. New newborn jaundice monitoring scheme based on combination of pre-processing and color detection method. Neurocomputing 2013, 120, 258–261. [Google Scholar] [CrossRef]
- Munkholm, S.B.; Krøgholt, T.; Ebbesen, F.; Szecsi, P.B.; Kristensen, S.R. The smartphone camera as a potential method for transcutaneous bilirubin measurement. PLoS ONE 2018, 13, e0197938. [Google Scholar] [CrossRef] [PubMed]
- De Greef, L.; Goel, M.; Seo, M.J.; Larson, E.C.; Stout, J.W.; Taylor, J.A.; Patel, S.N. Bilicam: Using mobile phones to monitor newborn jaundice. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Seattle, WA, USA, 13–17 September 2014; pp. 331–342. [Google Scholar]
- Saini, N.; Kumar, A.; Khera, P. Non-Invasive Bilirubin Detection Technique for Jaundice Prediction Using Smartphones. Int. J. Comput. Sci. Inf. Secur. 2016, 14, 1060. [Google Scholar]
- Padidar, P.; Shaker, M.; Amoozgar, H.; Khorraminejad-Shirazi, M.; Hemmati, F.; Najib, K.S.; Pourarian, S. Detection of neonatal jaundice by using an android OS-based smartphone application. Iran. J. Pediatr. 2019, 29, e84397. [Google Scholar] [CrossRef] [Green Version]
- Taylor, J.A.; Stout, J.W.; de Greef, L.; Goel, M.; Patel, S.; Chung, E.K.; Koduri, A.; McMahon, S.; Dickerson, J.; Simpson, E.A.; et al. Use of a smartphone app to assess neonatal jaundice. Pediatrics 2017, 140, e20170312. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Swarna, S.; Pasupathy, S.; Chinnasami, B. The smart phone study: Assessing the reliability and accuracy of neonatal jaundice measurement using smart phone application. Int. J. Contemp. Pediatr. 2018, 5, 285. [Google Scholar] [CrossRef] [Green Version]
- Aydın, M.; Hardalaç, F.; Ural, B.; Karap, S. Neonatal Jaundice Detection System. J. Med. Syst. 2016, 40, 166. [Google Scholar] [CrossRef]
- Castro-Ramos, J.; Toxqui-Quitl, C.; Manriquez, F.V.; Orozco-Guillen, E.; Padilla-Vivanco, A.; Sánchez-Escobar, J.J. Detecting jaundice by using digital image processing. In Three-Dimensional and Multidimensional Microscopy: Image Acquisition and Processing XXI; International Society for Optics and Photonics: Bellingham, WA, USA, 2014; Volume 8949. [Google Scholar]
- HARDALAÇ, F.; Aydin, M.; Kutbay, U.Ğ.; Ayturan, K.; AKYEL, A.; Çağlar, A.; HAi, B.; Mert, F. Classification of neonatal jaundice in mobile application with noninvasive image processing methods. Turk. J. Electr. Eng. Comput. Sci. 2021, 29, 2116–2126. [Google Scholar] [CrossRef]
- Leung, T.S.; Kapur, K.; Guilliam, A.; Okell, J.; Lim, B.; MacDonald, L.W.; Meek, J. Screening neonatal jaundice based on the sclera color of the eye using digital photography. Biomed. Opt. Express 2015, 6, 4529–4538. [Google Scholar] [CrossRef] [PubMed]
- Mariakakis, A.; Banks, M.A.; Phillipi, L.; Yu, L.; Taylor, J.; Patel, S.N. Biliscreen: Smartphone-based scleral jaundice monitoring for liver and pancreatic disorders. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2017, 1, 20. [Google Scholar] [CrossRef] [Green Version]
- Outlaw, F.; Meek, J.; MacDonald, L.W.; Leung, T.S. Screening for Neonatal Jaundice with a Smartphone. In Proceedings of the 2017 International Conference on Digital Health, London, UK, 2–5 July 2017; pp. 241–242. [Google Scholar]
- Rizvi, M.R.; Alaskar, F.M.; Albaradie, R.S.; Rizvi, N.F.; Al-Abdulwahab, K. A Novel Non-invasive Technique of Measuring Bilirubin Levels Using BiliCapture. Oman Med. J. 2019, 34, 26. [Google Scholar] [CrossRef] [PubMed]
- Laddi, A.; Kumar, S.; Sharma, S.; Kumar, A. Non-invasive jaundice detection using machine vision. IETE J. Res. 2013, 59, 591–596. [Google Scholar] [CrossRef]
- Outlaw, F.; Nixon, M.; Odeyemi, O.; MacDonald, L.W.; Meek, J.; Leung, T.S. Smartphone screening for neonatal jaundice via ambient-subtracted sclera chromaticity. PLoS ONE 2020, 15, e0216970. [Google Scholar] [CrossRef] [Green Version]
- Parinyanut, P.; Bandisak, T.; Chiengkriwate, P.; Tanthanuch, S.; Sangkhathat, S. Digital camera image analysis of faeces in detection of cholestatic jaundice in infants. African J. Paediatr. Surg. 2016, 13, 131. [Google Scholar]
- Ho, E.Y.W.; Lee, S.Y.R.; Chow, C.B.; Chung, J.W.Y. BiliCheck transcutaneous bilirubinometer: A screening tool for neonatal jaundice in the Chinese population. Hong Kong Med. J. 2006, 12, 99. [Google Scholar] [PubMed]
- Boo, N.; Ishak, S. Prediction of severe hyperbilirubinaemia using the Bilicheck transcutaneous bilirubinometer. J. Paediatr. Child Health 2007, 43, 297–302. [Google Scholar] [CrossRef] [PubMed]
- Hemmati, F.; Rad, N.A.K. The value of Bilicheck® as a screening tool for neonatal jaundice in the South of Iran. Iran. J. Med. Sci. 2013, 38, 122. [Google Scholar] [PubMed]
- Engle, W.D.; Jackson, G.L.; Stehel, E.K.; Sendelbach, D.M.; Manning, M.D. Evaluation of a transcutaneous jaundice meter following hospital discharge in term and near-term neonates. J. Perinatol. 2005, 25, 486. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Gomez, M.; Bielza, C.; Fernandez del Pozo, J.A.; Rios-Insua, S. A graphical decision-theoretic model for neonatal jaundice. Med. Decis. Mak. 2007, 27, 250–265. [Google Scholar] [CrossRef] [PubMed]
- American Academy of Pediatrics. Subcommittee on Hyperbilirubinemia, Management of hyperbilirubinemia in the newborn infant 35 or more weeks of gestation. Pediatrics 2004, 114, 297. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Moyer, V.A.; Ahn, C.; Sneed, S. Accuracy of clinical judgment in neonatal jaundice. Arch. Pediatr. Adolesc. Med. 2000, 154, 391–394. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- De Luca, D.; Zecca, E.; Zuppa, A.A.; Romagnoli, C. The joint use of human and electronic eye: Visual assessment of jaundice and transcutaneous bilirubinometry. Turk. J. Pediatr. 2008, 50, 456. [Google Scholar]
- Riskin, A.; Abend-Weinger, M.; Bader, D. How accurate are neonatologists in identifying clinical jaundice in newborns? Clin. Pediatr. 2003, 42, 153–158. [Google Scholar] [CrossRef]
- Gupta, A.; Kumar, A.; Khera, P. Jaundice prediction through non-invasive techniques: Issues and challenges. In Proceedings of the 2015 Annual IEEE India Conference (INDICON), New Delhi, India, 17–20 December 2015; pp. 1–5. [Google Scholar]
- Chou, J.H. Predictive Models for Neonatal Follow-Up Serum Bilirubin: Model Development and Validation. JMIR Med. Inform. 2020, 8, e21222. [Google Scholar] [CrossRef]
- Azar, A.T.; Inbarani, H.H.; Kumar, S.U.; Own, H.S. Hybrid system based on bijective soft and neural network for Egyptian neonatal jaundice diagnosis. Int. J. Intell. Eng. Inform. 2016, 4, 71–90. [Google Scholar] [CrossRef] [Green Version]
- Ferreira, D.; Oliveira, A.; Freitas, A. Applying data mining techniques to improve diagnosis in neonatal jaundice. BMC Med. Inform. Decis. Mak. 2012, 12, 143. [Google Scholar] [CrossRef] [Green Version]
- Sajana, B.; Chandana, T. A compration of unsupervised learning techniques in jaundice diagnosis. Int. J. Pure Appl. Math. 2018, 116, 139–144. [Google Scholar]
- Singla, R.; Singh, S. A framework for detection of jaundice in new born babies using homomorphic filtering based image processing. In Proceedings of the 2016 International Conference on Inventive Computation Technologies (ICICT), Coimbatore, India, 26–27 August 2016; Volume 3, pp. 1–5. [Google Scholar]
- Hsu, W.Y.; Cheng, H.C. A Fast and Effective System for Detection of Neonatal Jaundice with a Dynamic Threshold White Balance Algorithm. Healthcare 2021, 9, 1052. [Google Scholar] [CrossRef]
- Robertson, A.; Kazmierczak, S.; Vos, P. Improved transcutaneous bilirubinometry: Comparison of SpectR x BiliCheck and Minolta jaundice meter JM-102 for estimating total serum bilirubin in a normal newborn population. J. Perinatol. 2002, 22, 12. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Szabo, P.; Wolf, M.; Bucher, H.U.; Fauchere, J.-C.; Haensse, D.; Arlettaz, R. Detection of hyperbilirubinaemia in jaundiced full-term neonates by eye or by bilirubinometer? Eur. J. Pediatr. 2004, 163, 722–727. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Johnson, S.M.; Vasu, V.; Marseille, C.; Hill, C.; Janvier, L.; Toussaint, P.; Battersby, C. Validation of transcutaneous bilirubinometry during phototherapy for detection and monitoring of neonatal jaundice in a low-income setting. Pediatr. Int. Child Health 2020, 40, 25–29. [Google Scholar] [CrossRef] [PubMed]
- Gasparini, F.; Schettini, R. Color balancing of digital photos using simple image statistics. Pattern Recognit. 2004, 37, 1201–1217. [Google Scholar] [CrossRef]
- Sharma, S.; Shanmugasundaram, K.; Ramasamy, S.K. FAREC—CNN based efficient face recognition technique using Dlib. In Proceedings of the 2016 International Conference on Advanced Communication Control and Computing Technologies (ICACCCT), Ramanathapuram, India, 25–27 May 2016; pp. 192–195. [Google Scholar]
- Amos, B.; Ludwiczuk, B.; Satyanarayanan, M. Openface: A general-purpose face recognition library with mobile applications. CMU Sch. Comput. Sci. 2016, 6, 18. [Google Scholar]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. Commun. ACM 2017, 60, 84–90. [Google Scholar] [CrossRef]
- Dietterich, T.G. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput. 1998, 10, 1895–1923. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Ref. | Feature Extraction | Method | Dataset | Result |
---|---|---|---|---|
[7] | Face skin (mean, standard deviation, skewness, kurtosis, energy, entropy) | K-Nearest Neighbours (KNN) | 120 random images from Google infant monitoring | Accuracy = 90–96% |
[8] | Forehead skin (RGB) | Linear regression model | 64 images at Aalborg University Hospital in Denmark | Green sensitivity = 100%, specificity = 62% Blue sensitivity = 90%, specificity = 60% |
[9] | Sternum and forehead skin (YCbCr and lab color spaces) | Ensemble of five regression algorithms (KNN, Least angle regression (LARS), LARS-Lasso Elastic Net, Support vector regression (SVR), Random forest (RF)) | 100 images collected from University of Washington Medical Center (UWMC) and the Roosevelt Pediatric Care Center | A linear correlation of 0.84 with TSB, with a mean error of 2.0 mg/dL |
[10] | Forehead and sternum skin (Lab color spaces) | Matching | Standard set of serum bilirubin coloration on detection strips | Correlation = 0.93 |
[11] | Forehead skin (RGB and Hue, Saturation, Intensity (HIS) values) | Regression | 113 images at Hafez and Shoushtari hospitals in Shiraz, Iran using a Samsung phone | Sensitivity = 68% Specificity = 92.3% |
[12] | Sternum skin (YCbCr and lab color spaces) | Regression | 530 images of different races in US including African American, Hispanic, and Asian American | Sensitivity = 84.6% Specificity = 75.1% |
[13] | Sternum and abdomen skin (Hue and Saturation values) | Regression | 35 images in Chennai, India | Sternum correlation = 0.6 Abdomen correlation = 0.55 |
[14] | Abdomen skin (YCbCr, RGB, and lab color spaces) | KNN SVR | 80 image from Fırat University Faculty of Medicine, Neonatal Department in Turkey | KNN accuracy = 85% SVR accuracy = 75% |
[15] | Soles, palm, forehead, and arm skin (RGB + diffuse reflectance spectra) | SVM | 20 images of Mexican infants | Sensitivity = 71.8% Specificity = 78.8% |
[16] | Face, arms, feet and middle body skin (RGB) | Linear regression | 196 images at Firat university, Faculty of Medicine using an Android mobile phone or tablet | Accuracy = 92.5% |
[17] | Eye (RGB) | Linear regression | 110 images at University College London Hospital captured using a Nikon D3200 camera | Correlation = 0.75 |
[18] | Eye (sclera blue pixels) | Random forest regression | 70 images of adults eyes at University of Washington using an iPhone SE | Sensitivity = 89.7% Specificity = 96.8% |
[19] | Eye (RGB) | Regression | 86 images at the UCH Neonatal Unit in London using a Nikon Dh3200 camera | Correlation = 0.71 |
[20] | Eye | Diazo method with dichloroaniline (DCA) | 100 images at King Khalid Hospital at Al-Majma’ah, Saudi Arabia and Alpine Hospital, Gurgaon, India using a Samsung 10 | Sensitivity = 92.0% specificity = 75.6% |
[21] | Eye (PCA to extract L, a, and b values per CIE lab color) | Artificial neuro-fuzzy inference system (ANFIS) | 420 images of adults’ eyes captured in fixed conditions using a 3CDD digital camera in aphotic housing made up of acrylic sheet | Accuracy = 90% |
[22] | Eye (RGB) | Jaundice Eye Color Index Scleral-Conjunctival Bilirubin ((JECI-SCB) model and SCBxy model | 51 images from the UCL Hospital using an LG Nexus 5X smartphone | Correlation = 0.75 |
[38] | Bilirubin sample strips (homomorphic filter and blue color intensity) | Correlation between actual and predicted bilirubin level | 8 images of bilirubin sample strips | Correlation coefficient increased from magnitude 0.5261 to magnitude 0.6974 after filtering |
Characteristic | Value |
---|---|
Dataset size (images) | 68 |
Gender (images) | |
Male | 44 |
Female | 24 |
Gestation age (weeks) | |
Max. | 42 |
Avg. | 38 |
Min. | 35 |
Age (days) | |
Max. | 5 |
Avg. | 1 |
Min. | 1 |
TCB level (Mmol/M) | |
Max. | 280 |
Avg. | 135 |
Min. | 0 |
Weight (kg) | |
Max. | 4 |
Avg. | 3 |
Min. | 2 |
Class (images) | |
Healthy | 44 |
Jaundiced | 24 |
Classification Model | Parameters Values |
---|---|
MLP | Loss = binary_crossentropy |
optimizer = Adam | |
epochs = 50 | |
batch_size = 32 | |
layers = 2 | |
Hidden layers = 200 | |
Relu | |
softmax | |
Dropout = 0.5 | |
SVM | Kernel = RBF |
C = 1000 | |
Gamma = 0.7 | |
DT | Criterion = gini |
Splitter = best | |
Max_depth = None | |
min_samples_split = 2 | |
min_samples_leaf = 1 | |
min_weight_fraction_leaf = 0.0 | |
max_features = None | |
random_state = None | |
max_leaf_nodes = None | |
min_impurity_decrease = 0.0 | |
min_impurity_split = 0 | |
class_weight = none | |
ccp_alpha = 0.0 | |
RF | n_estimators = 100 |
criterion = gini | |
max_depth = none | |
min_samples_split = 2 | |
min_samples_leaf = 1 | |
min_weight_fraction_leaf = 0.0 | |
max_features = “auto” | |
max_leaf_nodes = None | |
min_impurity_decrease = 0.0 | |
min_impurity_split = None | |
bootstrap = True | |
oob_score = False | |
n_jobs = None | |
random_state = None | |
verbose = 0 | |
warm_start = False | |
class_weight = None | |
ccp_alpha = 0.0 | |
max_samples = None | |
CNN | batch_size = 100 |
epochs = 500 | |
momentum = 0.8 | |
SGD Optimizer |
Features | Accuracy | Precision | Recall | F1 Score | AUC |
---|---|---|---|---|---|
Skin | 86.83% | 84.49% | 81.05% | 82.12% | 81.05% |
Eye | 79.03% | 75.28% | 69.67% | 70.73% | 69.67% |
Fusion | 79.95% | 79.76% | 71.25% | 72.12% | 71.25% |
Features | Classifier | Accuracy | Precision | Recall | F1 Score | AUC |
---|---|---|---|---|---|---|
Skin | MLP | 66.02% | 66.26% | 65.64% | 64.47% | 65.64% |
SVM | 65.95% | 69.42% | 65.95% | 64.60% | 67.50% | |
DT | 62.35% | 61.35% | 61.18% | 61.40% | 60.89% | |
RF | 64.77% | 72.54% | 64.77% | 61.50% | 60.04% | |
Avg. | 64.77% | 67.39% | 64.39% | 62.99% | 63.52% | |
Eye | MLP | 79.61% | 80.62% | 79.04% | 78.84% | 79.04% |
SVM | 74.97% | 75.97% | 74.97% | 74.70% | 75.96% | |
DT | 62.35% | 64.37% | 62.22% | 59.70% | 60.25% | |
RF | 77.19% | 77.58% | 77.19% | 77.10% | 81.06% | |
Avg. | 73.53% | 74.64% | 73.36% | 72.59% | 74.08% | |
Fusion | MLP | 77.62% | 78.66% | 77.62% | 77.71% | 77.41% |
SVM | 76.41% | 76.44% | 76.41% | 75.80% | 82.01% | |
DT | 67.19% | 70.65% | 67.19% | 69.8% | 70.17% | |
RF | 72.75% | 73.89% | 72.75% | 72.10% | 78.86% | |
Avg. | 73.49% | 74.91% | 73.49% | 73.85% | 77.11% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Althnian, A.; Almanea, N.; Aloboud, N. Neonatal Jaundice Diagnosis Using a Smartphone Camera Based on Eye, Skin, and Fused Features with Transfer Learning. Sensors 2021, 21, 7038. https://doi.org/10.3390/s21217038
Althnian A, Almanea N, Aloboud N. Neonatal Jaundice Diagnosis Using a Smartphone Camera Based on Eye, Skin, and Fused Features with Transfer Learning. Sensors. 2021; 21(21):7038. https://doi.org/10.3390/s21217038
Chicago/Turabian StyleAlthnian, Alhanoof, Nada Almanea, and Nourah Aloboud. 2021. "Neonatal Jaundice Diagnosis Using a Smartphone Camera Based on Eye, Skin, and Fused Features with Transfer Learning" Sensors 21, no. 21: 7038. https://doi.org/10.3390/s21217038
APA StyleAlthnian, A., Almanea, N., & Aloboud, N. (2021). Neonatal Jaundice Diagnosis Using a Smartphone Camera Based on Eye, Skin, and Fused Features with Transfer Learning. Sensors, 21(21), 7038. https://doi.org/10.3390/s21217038