Conditional Generative Adversarial Networks for Data Augmentation of a Neonatal Image Dataset
Abstract
:1. Introduction
1.1. Motivation
1.2. Problem Statement
1.3. Approach
2. Related Work
3. Materials and Methods
3.1. Datasets
3.2. Data Preprocessing
3.3. Spectral Normalization
3.4. Training
3.5. Validation
4. Results
4.1. Qualitative Results
4.1.1. Edge Detector
4.1.2. Training Epochs
4.1.3. Generator and Discriminator
4.2. Quantitative Results
5. Discussion
6. Conclusions and Outlook
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Kumar, N.; Akangire, G.; Sullivan, B.; Fairchild, K.; Sampath, V. Continuous vital sign analysis for predicting and preventing neonatal diseases in the twenty-first century: Big data to the forefront. Pediatr. Res. 2020, 87, 210–220. [Google Scholar] [CrossRef] [PubMed]
- Allen, K. Neonatal thermal care: A discussion of two incubator modes for optimising thermoregulation. A care study. J. Neonatal Nurs. 2011, 17, 43–48. [Google Scholar] [CrossRef]
- Lund, C. Medical adhesives in the NICU. Newborn Infant Nurs. Rev. 2014, 14, 160–165. [Google Scholar] [CrossRef]
- Antink, C.H.; Lyra, S.; Paul, M.; Yu, X.; Leonhardt, S. A broader look: Camera-based vital sign estimation across the spectrum. Yearb. Med Inform. 2019, 28, 102–114. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Esteva, A.; Chou, K.; Yeung, S.; Naik, N.; Madani, A.; Mottaghi, A.; Liu, Y.; Topol, E.; Dean, J.; Socher, R. Deep learning-enabled medical computer vision. NPJ Digit. Med. 2021, 4, 5. [Google Scholar] [CrossRef]
- Althnian, A.; AlSaeed, D.; Al-Baity, H.; Samha, A.; Dris, A.B.; Alzakari, N.; Abou Elwafa, A.; Kurdi, H. Impact of dataset size on classification performance: An empirical evaluation in the medical domain. Appl. Sci. 2021, 11, 796. [Google Scholar] [CrossRef]
- Ramanan, M.; Singh, L.; Kumar, A.S.; Suresh, A.; Sampathkumar, A.; Jain, V.; Bacanin, N. Secure blockchain enabled Cyber-Physical health systems using ensemble convolution neural network classification. Comput. Electr. Eng. 2022, 101, 108058. [Google Scholar] [CrossRef]
- Buolamwini, J.; Gebru, T. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, New York, NY, USA, 23–24 February 2018; Volume 81, Machine Learning Research. Friedler, S.A., Wilson, C., Eds.; 2018; pp. 77–91. [Google Scholar]
- Obermeyer, Z.; Powers, B.; Vogeli, C.; Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019, 366, 447–453. [Google Scholar] [CrossRef] [Green Version]
- Goodfellow Ian, J.; Jean, P.A.; Mehdi, M.; Bing, X.; David, W.F.; Sherjil, O.; Courville Aaron, C. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; Volume 2, pp. 2672–2680. [Google Scholar]
- Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1125–1134. [Google Scholar]
- Wang, T.C.; Liu, M.Y.; Zhu, J.Y.; Tao, A.; Kautz, J.; Catanzaro, B. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 8798–8807. [Google Scholar]
- Zhang, H.; Xu, T.; Li, H.; Zhang, S.; Wang, X.; Huang, X.; Metaxas, D.N. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 5907–5915. [Google Scholar]
- Regmi, K.; Borji, A. Cross-view image synthesis using conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3501–3510. [Google Scholar]
- Pang, Y.; Lin, J.; Qin, T.; Chen, Z. Image-to-image translation: Methods and applications. IEEE Trans. Multimed. 2021, 24, 3859–3881. [Google Scholar] [CrossRef]
- Mirza, M.; Osindero, S. Conditional generative adversarial nets. arXiv 2014, arXiv:1411.1784. [Google Scholar]
- Alqahtani, H.; Kavakli-Thorne, M.; Kumar, G. Applications of generative adversarial networks (gans): An updated review. Arch. Comput. Methods Eng. 2021, 28, 525–552. [Google Scholar] [CrossRef]
- Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2223–2232. [Google Scholar]
- Karras, T.; Laine, S.; Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–17 June 2019; pp. 4401–4410. [Google Scholar]
- Chen, Y.; Yang, X.H.; Wei, Z.; Heidari, A.A.; Zheng, N.; Li, Z.C.; Chen, H.; Hu, H.; Zhou, Q.; Guan, Q. Generative Adversarial Networks in Medical Image augmentation: A review. Comput. Biol. Med. 2022, 144, 105382. [Google Scholar] [CrossRef]
- Zhou, T.; Li, Q.; Lu, H.; Cheng, Q.; Zhang, X. GAN review: Models and medical image fusion applications. Inf. Fusion 2023, 91, 134–148. [Google Scholar] [CrossRef]
- Khalili, N.; Turk, E.; Zreik, M.; Viergever, M.A.; Benders, M.J.; Išgum, I. Generative adversarial network for segmentation of motion affected neonatal brain MRI. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China, 13–17 October 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 320–328. [Google Scholar]
- Delannoy, Q.; Pham, C.H.; Cazorla, C.; Tor-Díez, C.; Dollé, G.; Meunier, H.; Bednarek, N.; Fablet, R.; Passat, N.; Rousseau, F. SegSRGAN: Super-resolution and segmentation using generative adversarial networks—Application to neonatal brain MRI. Comput. Biol. Med. 2020, 120, 103755. [Google Scholar] [CrossRef]
- Alam, S.B.; Kobashi, S. Comprehensive modeling of neonatal brain image generation for disorder development onset prediction based on generative adversarial networks. In Multidisciplinary Computational Anatomy; Springer: Berlin/Heidelberg, Germany, 2022; pp. 269–273. [Google Scholar]
- Karthik, K.; Nithin, K.; Dhanush, B.; Praveen, K.; Sarath, S. Data Augmentation of Neonatal Thermal Images Using Deep Learning. In Proceedings of the IEEE 2021 12th International Conference on Computing Communication and Networking Technologies (ICCCNT), Kharagpur, India, 6–8 July 2021; pp. 1–6. [Google Scholar]
- Lyra, S.; Rixen, J.; Heimann, K.; Karthik, S.; Joseph, J.; Jayaraman, K.; Orlikowsky, T.; Sivaprakasam, M.; Leonhardt, S.; Hoog Antink, C. Camera fusion for real-time temperature monitoring of neonates using deep learning. Med. Biol. Eng. Comput. 2022, 60, 1787–1800. [Google Scholar] [CrossRef] [PubMed]
- Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 1986, PAMI-8, 679–698. [Google Scholar] [CrossRef]
- Xie, S.; Tu, Z. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1395–1403. [Google Scholar]
- Su, Z.; Liu, W.; Yu, Z.; Hu, D.; Liao, Q.; Tian, Q.; Pietikäinen, M.; Liu, L. Pixel difference networks for efficient edge detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 5117–5127. [Google Scholar]
- Wibisono, J.K.; Hang, H.M. Traditional Method Inspired Deep Neural Network For Edge Detection. In Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Nanjing, China, 3–5 July 2020; pp. 678–682. [Google Scholar]
- Poma, X.S.; Riba, E.; Sappa, A. Dense extreme inception network: Towards a robust cnn model for edge detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA, 1–5 March 2020; pp. 1923–1932. [Google Scholar]
- Alotaibi, A. Deep generative adversarial networks for image-to-image translation: A review. Symmetry 2020, 12, 1705. [Google Scholar] [CrossRef]
- Arjovsky, M.; Bottou, L. Towards Principled Methods for Training Generative Adversarial Networks. In Proceedings of the International Conference on Learning Representations, Toulon, France, 24–26 April 2017. [Google Scholar]
- Miyato, T.; Kataoka, T.; Koyama, M.; Yoshida, Y. Spectral normalization for generative adversarial networks. arXiv 2018, arXiv:1802.05957. [Google Scholar]
- Biewald, L. Experiment Tracking with Weights and Biases, 2020. Software. Available online: wandb.com (accessed on 1 November 2022).
- Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; Chen, X. Improved techniques for training gans. Adv. Neural Inf. Process. Syst. 2016, 29. [Google Scholar] [CrossRef]
- Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017; Volume 30. [Google Scholar]
- Shmelkov, K.; Schmid, C.; Alahari, K. How good is my GAN? In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 213–229. [Google Scholar]
- Seitzer, M. PyTorch-Fid: FID Score for PyTorch. Version 0.3.0. 2020. Available online: https://github.com/mseitzer/pytorch-fid (accessed on 8 November 2022).
- Pizer, S.M.; Amburn, E.P.; Austin, J.D.; Cromartie, R.; Geselowitz, A.; Greer, T.; ter Haar Romeny, B.; Zimmerman, J.B.; Zuiderveld, K. Adaptive histogram equalization and its variations. Comput. Vision Graph. Image Process. 1987, 39, 355–368. [Google Scholar] [CrossRef]
- Human Benchmark. Available online: https://humanbenchmark.com/tests/reactiontime/statistics (accessed on 22 November 2022).
- Ghosh, A.; Zhang, R.; Dokania, P.; Wang, O.; Efros, A.; Torr, P.; Shechtman, E. Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 1171–1180. [Google Scholar]
- Chen, W.; Hays, J. SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018; pp. 9416–9425. [Google Scholar]
- Liu, B.; Zhu, Y.; Song, K.; Elgammal, A. Self-supervised sketch-to-image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, (Held Virtually), 2–9 February 2021; Volume 35, pp. 2073–2081. [Google Scholar]
- Zhang, H.; Yuan, J.; Tian, X.; Ma, J. GAN-FM: Infrared and visible image fusion using GAN with full-scale skip connection and dual Markovian discriminators. IEEE Trans. Comput. Imaging 2021, 7, 1134–1147. [Google Scholar] [CrossRef]
- Antink, C.H.; Ferreira, J.C.M.; Paul, M.; Lyra, S.; Heimann, K.; Karthik, S.; Joseph, J.; Jayaraman, K.; Orlikowsky, T.; Sivaprakasam, M.; et al. Fast body part segmentation and tracking of neonatal video data using deep learning. Med Biol. Eng. Comput. 2020, 58, 3049–3061. [Google Scholar] [CrossRef] [PubMed]
Hyperparameter | Parameter Sweep |
---|---|
# filters (generator) | 48, 64, 96 |
# residual blocks (generator) | 7, 9, 12 |
# training epochs [decays] | 200 [100], 400 [200], 600 [200, 300, 400] |
# discriminators | 2, 3 |
# filters (discriminator) | 48, 64, 96 |
Edge Detector | |||||
---|---|---|---|---|---|
Feature | Canny | HED | PiDiNet | TIN | DexiNed |
Colorization | + | − | − | − | + |
Distinction | − | − | + | + | + |
Facial features | − | − | − | + | + |
Prevention of textured noise | − | − | + | − | + |
Stage | |||||
---|---|---|---|---|---|
Choices [%] | 1 | 2 | 3 | 4 | 5 |
Real | 23 | 66 | 72 | 81 | 63 |
Fake | 77 | 34 | 28 | 19 | 37 |
Dataset Comparison | ||||
---|---|---|---|---|
(a) | (b) | (c) | (d) | |
FID | 233.81 | 103.82 | 254.50 | 142.25 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Lyra, S.; Mustafa, A.; Rixen, J.; Borik, S.; Lueken, M.; Leonhardt, S. Conditional Generative Adversarial Networks for Data Augmentation of a Neonatal Image Dataset. Sensors 2023, 23, 999. https://doi.org/10.3390/s23020999
Lyra S, Mustafa A, Rixen J, Borik S, Lueken M, Leonhardt S. Conditional Generative Adversarial Networks for Data Augmentation of a Neonatal Image Dataset. Sensors. 2023; 23(2):999. https://doi.org/10.3390/s23020999
Chicago/Turabian StyleLyra, Simon, Arian Mustafa, Jöran Rixen, Stefan Borik, Markus Lueken, and Steffen Leonhardt. 2023. "Conditional Generative Adversarial Networks for Data Augmentation of a Neonatal Image Dataset" Sensors 23, no. 2: 999. https://doi.org/10.3390/s23020999