Image Translation by Ad CycleGAN for COVID-19 X-Ray Images: A New Approach for Controllable GAN
Abstract
:1. Introduction
2. Materials and Methods
2.1. Cycle-Consistent Adversarial Network and Its Restriction
2.2. External Criterion in Cycle GAN Optimization
2.3. Ad CycleGAN Architecture
- Adversarial loss:
- Cycle consistency loss:
- Identity loss:
- Criterion Loss:
- Thus, the total generator loss in Equation (5) is revised as:
Algorithm 1. Ad CycleGAN Optimization | |
1: | for number of epochs do |
2: | for number of batches do |
3: | Sample minibatch |
4: | Sample minibatch |
5: | Generate m synthetic samples of |
6: | |
7: | |
8: | Compute the Adversarial loss |
9: | |
10: | |
11: | Generate m cycle sample of |
12: | |
13: | |
14: | Compute the Cycle loss |
15: | |
16: | Generate m identical sample of |
17: | |
18: | |
19: | Compute the identity loss |
20: | |
21: | Compute the criterion loss for cycle sample: |
22: | Compute the criterion loss for identical sample: |
23: | Compute the total generator loss |
24: | |
25 | Update the Discriminator |
26: | |
27: | |
28: | Update the Generators |
29: | |
30: | end do |
31: | end do |
2.4. Evaluation Metrics for Translated Images
- The performance evaluation of GAN networks is usually subjective and remains as an open problem [24]. Our objective is to generate synthetic medical images with good fidelity and diversity. We need to measure both the quality of the images and ensure the generated images belonging to the correct category, i.e., carrying the diagnostically significant patterns. The latter task can be measured by the classification accuracy of the synthetic images. There are generally two types of methods to measure the quality of the synthetic images: subjective evaluation and objective evaluation. Subjective evaluation requires human expertise. It is time consuming and difficult to replication. Therefore, we apply the objective metrics to compare the synthetic images and the generated images with the assumption that the high-quality synthetic images have higher degrees of the similarity to the real images. The quantitative evaluation metrics for our experiments include Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Peak Signal-to-Noise Ratio (PSNR), Universal Image Quality Index (UIQI), and Visual Information Fidelity (VIF).
- MSE, RMSE and PSNR are metrics to measure the pixel difference between the synthetic images and the real images. MSE is the accumulated mean squared error of two images, and RMSE is the accumulated root mean square error of the two images. PSNR is a measure for image quality [25] based on the pixel difference between the synthetic image and the real image. UIQI summarizes the attributes of human vision [26], where synthetic images and the real images are compared in three aspects: luminance, contrast, and structure. VIF is another measure based on human visual perception. VIF quantifies the image fidelity by the difference of the information extracted from the real image and the information loss to the synthetic image by human brain is quantified as the VIF score using visual natural scene statistics (NSS), human visual system (HVS) and an image distortion model. For comparison, the synthetic images with low MSE and RMSE, and with high scores in PSNR, UIQI and VIF are considered to be of better quality. In addition, we also use the Frechet Inception Distance (FID) which is a commonly accepted metric to compare the quality of the images synthesized by different generative models. FID was proposed by Heusel, M. et al. in 2017 to calculate the distance between feature vectors calculated for real and generated images [27]. It reflects how similar the two image groups are in terms of statistics on computer vision features of the raw images calculated using a pretrained classifier. Low FID score indicates the two groups of images are similar or have more similar statistics.
- In the next section, we will present the experiments of respectively using Cycle GAN and Ad CycleGAN to perform image translation between normal CXR images and COVID-19 positive CXR images, and the comparisons of the quality of the synthetic with the above quantitative metrics.
3. Experiments
3.1. Material and Methods
3.2. Results and Interpretation
4. Discussion
5. Conclusions
Author Contributions
Funding
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
Ad CycleGAN | adaptive cycle-consistent generative adversarial network |
AI | artificial intelligence |
ARDS | acute respiratory distress syndrome |
CGAN | conditional GAN |
COVID-19 | Coronavirus disease 2019 |
Cycle GAN | Cycle-consistent generative adversarial network |
CT | computed tomography |
CXR | chest X-ray |
DNN | deep neural network |
FID | Frechet inception distance |
GAN | generative adversarial network |
HVS | human visual system |
LUS | lung ultrasound |
MR | magnetic resonance |
MSE | mean squared error |
NSS | natural scene statistics |
PET | positron emission tomography |
PSNR | peak signal-to-noise ratio |
RMSE | root mean squared error |
RT-PCR | real-time reverse-transcriptase polymerase chain reaction |
UIQI | universal image quality index |
VIF | visual information fidelity |
WGAN | Wasserstein GAN |
References
- Chen, N.; Zhou, M.; Dong, X.; Qu, J.; Gong, F.; Han, Y.; Qiu, Y.; Wang, J.; Liu, Y.; Wei, Y.; et al. Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: A descriptive study. Lancet 2020, 395, 507–513. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Deeks, J.J.; Dinnes, J.; Takwoingi, Y.; Davenport, C.; Spijker, R.; Taylor-Phillips, S.; Adriano, A.; Beese, S.; Dretzke, J.; di Ruffano, L.F.; et al. Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane Database Syst. Rev. 2020, 6, CD013652. [Google Scholar] [CrossRef]
- Shen, D.; Gao, Y.; Munoz-Barrutia, A.; Debuc, D.C.; Percannella, G. Special Issue on Imaging-Based Diagnosis of COVID-19. IEEE Trans. Med. Imaging 2020, 39, 2569–2571. [Google Scholar] [CrossRef]
- Sufian, A.; Ghosh, A.; Sadiq, A.S.; Smarandache, F. A Survey on Deep Transfer Learning and Edge Computing for Mitigating the COVID-19 Pandemic. J. Syst. Archit. 2020, 108, 101830. [Google Scholar] [CrossRef]
- Minaee, S.; Kafieh, R.; Sonka, M.; Yazdani, S.; Jamalipour Soufi, G. Deep-COVID: Predicting COVID-19 from chest X-ray images using deep transfer learning. Med. Image. Anal. 2020, 65, 101794. [Google Scholar] [CrossRef]
- Hirano, H.; Koga, K.; Takemoto, K. Vulnerability of deep neural networks for detecting COVID-19 cases from chest X-ray images to universal adversarial attacks. PLoS ONE 2020, 15, e0243963. [Google Scholar] [CrossRef]
- Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 139–144. [Google Scholar] [CrossRef]
- He, D.; Xia, Y.; Qin, T.; Wang, L.; Yu, N.; Liu, T.; Ma, W.T. Dual learning for machine translation. In Proceedings of the Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016; pp. 820–828. [Google Scholar]
- Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 1125–1134. [Google Scholar] [CrossRef] [Green Version]
- Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 2223–2232. [Google Scholar] [CrossRef] [Green Version]
- Cai, J.; Zhang, Z.; Cui, L.; Zheng, Y.; Yang, L. Towards cross-modal organ translation and segmentation: A cycle- and shape-consistent generative adversarial network. Med. Image Anal. 2019, 52, 174–184. [Google Scholar] [CrossRef] [PubMed]
- Li, Y.; Tang, S.; Zhang, R.; Zhang, Y.; Li, J.; Yan, S. Asymmetric GAN for Unpaired Image-to-Image Translation. IEEE Trans. Image Process. 2019, 28, 5881–5896. [Google Scholar] [CrossRef] [Green Version]
- Siddiquee, M.M.R.; Zhou, Z.; Tajbakhsh, N.; Feng, R.; Gotway, M.B.; Bengio, Y.; Liang, J. Learning Fixed Points in Generative Adversarial Networks: From Image-to-Image Translation to Disease Detection and Localization. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019; pp. 191–200. [Google Scholar] [CrossRef] [Green Version]
- Fu, J.; Singhrao, K.; Cao, M.; Yu, V.; Santhanam, A.P.; Yang, Y.; Guo, M.; Raldow, A.C.; Ruan, D.; Lewis, J.H. Generation of abdominal synthetic CTs from 0.35T MR images using generative adversarial networks for MR-only liver radiotherapy. Biomed. Phys. Eng. Express 2020, 6, 015033. [Google Scholar] [CrossRef] [Green Version]
- Lee, J.H.; Han, I.H.; Kim, D.H.; Yu, S.; Lee, I.S.; Song, Y.S.; Joo, S.; Jin, C.; Kim, H. Spine Computed Tomography to Magnetic Resonance Image Synthesis Using Generative Adversarial Networks: A Preliminary Study. J. Korean Neurosurg. Soc. 2020, 63, 386–396. [Google Scholar] [CrossRef] [Green Version]
- Hu, N.; Zhang, T.; Wu, Y.; Tang, B.; Li, M.; Song, B.; Gong, Q.; Wu, M.; Gu, S.; Lui, S. Detecting brain lesions in suspected acute ischemic stroke with CT-based synthetic MRI using generative adversarial networks. Ann. Transl. Med. 2022, 10, 35. [Google Scholar] [CrossRef]
- Nie, D.; Trullo, R.; Lian, J.; Petitjean, C.; Ruan, S.; Wang, Q.; Shen, D. Medical Image Synthesis with Context-Aware Generative Adversarial Networks. Int. Conf. Med. Image Comput. Comput.-Assist. Interv. 2017, 10435, 417–425. [Google Scholar] [CrossRef]
- Emami, H.; Dong, M.; Nejad-Davarani, S.P.; Glide-Hurst, C.K. Generating synthetic CTs from magnetic resonance images using generative adversarial networks. Med. Phys. 2018, 45, 3627–3636. [Google Scholar] [CrossRef] [PubMed]
- Hu, Z.; Li, Y.; Zou, S.; Xue, H.; Sang, Z.; Liu, X.; Yang, Y.; Zhu, X.; Liang, D.; Zheng, H. Obtaining PET/CT images from non-attenuation corrected PET images in a single PET system using Wasserstein generative adversarial networks. Phys. Med. Biol. 2020, 65, 215010. [Google Scholar] [CrossRef]
- Bazangani, F.; Richard, F.J.P.; Ghattas, B.; Guedj, E. Alzheimer’s Disease Neuroimaging I. FDG-PET to T1 Weighted MRI Translation with 3D Elicit Generative Adversarial Network (E-GAN). Sensors 2022, 22, 4640. [Google Scholar] [CrossRef] [PubMed]
- Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia, 6–11 August 2017; pp. 214–223. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. Int. Conf. Med. Image Comput. Comput.-Assist. Interv. 2015, 234–241. [Google Scholar] [CrossRef]
- Li, M.; Huang, H.; Ma, L.; Liu, W.; Zhang, T.; Jiang, Y. Unsupervised image-to-image translation with stacked cycle-consistent adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 184–199. [Google Scholar] [CrossRef]
- Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; Chen, X. Improved techniques for training gans. Adv. Neural Inf. Process. Syst. 2016, 29, 2234–2242. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [Green Version]
- Wang, Z.; Bovik, A.C. A universal image quality index. IEEE Signal Process. Lett. 2002, 9, 81–84. [Google Scholar] [CrossRef]
- Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017. [Google Scholar] [CrossRef]
- Motamed, S.; Rogalla, P.; Khalvati, F. Data augmentation using Generative Adversarial Networks (GANs) for GAN-based detection of Pneumonia and COVID-19 in chest X-ray images. Inf. Med Unlocked 2021, 27, 100779. [Google Scholar] [CrossRef]
- Morís, D.I.; de Moura Ramos, J.J.; Buján, J.N.; Hortas, M.O. Data augmentation approaches using cycle-consistent adversarial networks for improving COVID-19 screening in portable chest X-ray images. Expert Syst. Appl. 2021, 15, 115681. [Google Scholar] [CrossRef] [PubMed]
- Khaldi, Y.; Benzaoui, A. A new framework for grayscale ear images recognition using generative adversarial networks under unconstrained conditions. Evol. Syst. 2021, 12, 923–934. [Google Scholar] [CrossRef]
- Creswell, A.; Bharath, A.A. Inverting the generator of a generative adversarial network. IEEE Trans. Neural Netw. Learn. Syst. 2018, 30, 1967–1974. [Google Scholar] [CrossRef] [PubMed]
Model (Translation Direction) | MSE | RMSE | PSNR | UIQI | VIF |
---|---|---|---|---|---|
Cycle GAN () | 3608.97 (1398.37) | 58.77 (12.42) | 12.97 (2.10) | 0.81 (0.08) | 0.12 (0.05) |
Cycle GAN () | 409.00 (495.043) | 17.55 (10.035) | 24.43 (4.398) | 0.97 (0.029) | 0.55 (0.050) |
Ad CycleGAN () | 3750.71 (1789.51) | 59.54 (14.30) | 12.88 (2.12) | 0.80 (0.13) | 0.10 (0.03) |
Ad CycleGAN () | 435.84 (461.59) | 18.73 (9.21) | 23.59 (3.86) | 0.97 (0.03) | 0.52 (0.05) |
Model (Translation Direction) | FID | Accuracy |
---|---|---|
Cycle GAN | 0.9375 | |
Cycle GAN | 1.0 | |
Ad CycleGAN | 0.9843 | |
Ad CycleGAN | 1.0 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liang, Z.; Huang, J.X.; Antani, S. Image Translation by Ad CycleGAN for COVID-19 X-Ray Images: A New Approach for Controllable GAN. Sensors 2022, 22, 9628. https://doi.org/10.3390/s22249628
Liang Z, Huang JX, Antani S. Image Translation by Ad CycleGAN for COVID-19 X-Ray Images: A New Approach for Controllable GAN. Sensors. 2022; 22(24):9628. https://doi.org/10.3390/s22249628
Chicago/Turabian StyleLiang, Zhaohui, Jimmy Xiangji Huang, and Sameer Antani. 2022. "Image Translation by Ad CycleGAN for COVID-19 X-Ray Images: A New Approach for Controllable GAN" Sensors 22, no. 24: 9628. https://doi.org/10.3390/s22249628
APA StyleLiang, Z., Huang, J. X., & Antani, S. (2022). Image Translation by Ad CycleGAN for COVID-19 X-Ray Images: A New Approach for Controllable GAN. Sensors, 22(24), 9628. https://doi.org/10.3390/s22249628