Conditional Generative Adversarial Networks with Total Variation and Color Correction for Generating Indonesian Face Photo from Sketch
Abstract
:Featured Application
Abstract
1. Introduction
2. Materials and Methods
2.1. Generative Adversarial Networks (GAN)
2.2. Conditional GAN (cGAN)
2.3. Proposed Method
2.3.1. Dataset
- Splitting. In the splitting step, the image was separated into two different images, i.e., the original image and the sketch.
- Converting to tensors. Because the GAN model can only accept tensor input, both images were converted into a tensor with the float data type.
- Resize and augmentation. The resize step was carried out to ensure that the input tensor was in the appropriate size. In addition, we added noise to the training data, such as mirroring and cropping them arbitrarily in this step. This was to increase the variety of the training data.
- Normalization. At this stage, each pixel in the image was normalized and converted to a corresponding value in a range of 0–1. This was to ensure that each pixel had the same distribution, and to speed up the GAN’s convergence during training.
2.3.2. cGAN-TV Loss
2.3.3. Color Correction
2.3.4. Test Scenario
2.3.5. Evaluation Parameters
3. Results
3.1. Testing Results
3.2. Color Correction Evaluation
4. Discussion
4.1. Comparison with Other Methods
4.2. Results of Hand-Drawn Sketch Input
4.3. Loss Behavior
4.4. Future Works
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Kurniawan, M.H. Penggunaan Metode Sketsa Wajah Dalam Menemukan Pelaku Tindak Pidana—Ums Etd-Db; Universitas Muhammadiyah Surakarta: Surakarta, Indonesia, 2009. [Google Scholar]
- “Begini Rumitnya Membuat Sketsa Pelaku Penyiraman Novel Baswedan.” n.d. Available online: https://news.detik.com/berita/d-3583949/begini-rumitnya-membuat-sketsa-pelaku-penyiraman-novel-baswedan (accessed on 10 September 2021).
- Zhang, L.; Lin, L.; Wu, X.; Ding, S.; Zhang, L. End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghai, China, 23–26 June 2015; pp. 627–634. [Google Scholar] [CrossRef] [Green Version]
- Chen, W.; Hays, J. SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 9416–9425. [Google Scholar] [CrossRef] [Green Version]
- Li, Y.; Wu, F.; Chen, X.; Zha, Z.J. Linestofacephoto: Face Photo Generation from Lines with Conditional Self-Attention Generative Adversarial Network. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21–25 October 2019; pp. 2323–2331. [Google Scholar] [CrossRef]
- Sangkloy, P.; Lu, J.; Fang, C.; Yu, F.; Hays, J. Scribbler: Controlling Deep Image Synthesis with Sketch and Color. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6836–6845. [Google Scholar] [CrossRef] [Green Version]
- Li, L.; Tang, J.; Shao, Z.; Tan, X.; Ma, L. Sketch-to-Photo Face Generation Based on Semantic Consistency Preserving and Similar Connected Component Refinement. Vis. Comput. 2021, 1–18. [Google Scholar] [CrossRef]
- Isola, P.; Zhu, J.-Y.; Zhou, T.; Efros, A.A. Image-to-Image Translation with Conditional Adversarial Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Lei, Y.; Du, W.; Hu, Q. Face Sketch-to-Photo Transformation with Multi-Scale Self-Attention GAN. Neurocomputing 2020, 396, 13–23. [Google Scholar] [CrossRef]
- Zhu, M.; Liang, C.; Wang, N.; Wang, X.; Li, Z.; Gao, X. A Sketch-Transformer Network for Face Photo-Sketch Synthesis. IJCAI Int. Jt. Conf. Artif. Intell. 2021, 2, 1352–1358. [Google Scholar] [CrossRef]
- Hu, M.; Guo, J. Facial Attribute-Controlled Sketch-to-Image Translation with Generative Adversarial Networks. EURASIP J. Image Video Process. 2020, 2020, 1–13. [Google Scholar] [CrossRef]
- Zhang, F.; Zhao, H.; Ying, W.; Liu, Q.; Raj, A.N.J.; Fu, B. Human Face Sketch to Rgb Image with Edge Optimization and Generative Adversarial Networks. Intell. Autom. Soft Comput. 2020, 26, 1391–1401. [Google Scholar] [CrossRef]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Nets. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QU, Canada, 8–13 December 2014; Volume 3, pp. 2672–2680. [Google Scholar]
- Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015. [Google Scholar]
- Demir, U.; Unal, G. Patch-Based Image Inpainting with Generative Adversarial Networks. arXiv 2018, arXiv:1803.07422. [Google Scholar]
- Chang, Y.-L.; Liu, Z.Y.; Lee, K.-Y.; Hsu, W. Free-Form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 27–28 October 2019; pp. 9066–9075. [Google Scholar]
- Image Datasets for Machine Learning|Generated Photos. Available online: https://generated.photos/datasets (accessed on 10 September 2021).
- Karkkainen, K.; Joo, J. FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation. In Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, 3–8 January 2021; pp. 1547–1557. [Google Scholar]
- Liliana, D.Y.; Basaruddin, T.; Oriza, I.I.D. The Indonesian Mixed Emotion Dataset (IMED): A Facial Expression Dataset for Mixed Emotion Recognition. In Proceedings of the ACM/International Conference on Artificial Intelligence and Virtual Reality, Nagoya, Japan, 23–25 November 2018; pp. 56–60. [Google Scholar]
- Wang, X.; Tang, X. Face Photo-Sketch Synthesis and Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2009, 31, 1955–1967. [Google Scholar] [CrossRef] [PubMed]
- Chen, S.Y.; Su, W.; Gao, L.; Xia, S.; Fu, H. DeepFaceDrawing: Deep Generation of Face Images from Sketches. ACM Trans. Graph. 2020, 39, 72. [Google Scholar] [CrossRef]
- Abu-Srhan, A.; Abushariah, M.A.M.; Al-Kadi, O.S. The Effect of Loss Function on Conditional Generative Adversarial Networks. J. King Saud Univ. Comput. Inf. Sci. 2022, 34, 6977–6988. [Google Scholar] [CrossRef]
- Strong, D.; Chan, T. Edge-Preserving and Scale-Dependent Properties of Total Variation Regularization. Inverse Probl. 2003, 19, S165–S187. [Google Scholar] [CrossRef]
- Minaee, S.; Minaei, M.; Abdolrashidi, A. Palm-GAN: Generating Realistic Palmprint Images Using Total-Variation Regularized GAN. arXiv 2020, arXiv:2003.10834. [Google Scholar]
- Chadha, A.; Britto, J.; Roja, M.M. ISeeBetter: Spatio-Temporal Video Super-Resolution Using Recurrent Generative Back-Projection Networks. Comput. Vis. Media 2020, 6, 307–317. [Google Scholar] [CrossRef]
- Wang, Z.; Chen, J.; Hoi, S.C.H. Deep Learning for Image Super-Resolution: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 43, 3365–3387. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Reinhard, E.; Adhikhmin, M.; Gooch, B.; Shirley, P. Color Transfer between Images. IEEE Comput. Graph. Appl. 2001, 21, 34–41. [Google Scholar] [CrossRef]
- Yi, R.; Liu, Y.-J.; Lai, Y.-K.; Rosin, P.L. APDrawingGAN: Generating Artistic Portrait Drawings from Face Photos with Hierarchical GANs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 10743–10752. [Google Scholar]
Scenario | Photo for Color Transfer | Gamma |
---|---|---|
Pale White | Not applied | 0.8 |
Fair | 0.85 | |
Tan | 1.2 |
Scenario | a | b | c | d | e | f | g * | h | i | j | k |
---|---|---|---|---|---|---|---|---|---|---|---|
0.01 | 0.1 | 1 | 10 | 100 | 150 | 100 | 150 | 100 | |||
0.00001 | 0 | 0.0001 | 0.001 | 0.0001 | 0.00005 |
Scenario | a | b | c | d | e | f | g * | h | i | j | k |
---|---|---|---|---|---|---|---|---|---|---|---|
0.01 | 0.1 | 1 | 10 | 100 | 150 | 100 | 150 | 100 | |||
0.00001 | 0 | 1 × 10−4 | 1 × 10−3 | 1 × 10−4 | 5 × 10−5 | ||||||
SSIM | 0.695 | 0.756 | 0.725 | 0.763 | 0.834 | 0.809 | 0.815 | 0.823 | 0.832 | 0.837 | 0.813 |
FID | 220.976 | 235.060 | 241.096 | 162.398 | 94.705 | 97.285 | 97.078 | 101.960 | 101.114 | 97.050 | 115.742 |
Dataset | SSIM | FID |
---|---|---|
Validation | 0.83 | 94.705 |
Testing | 0.73 | 93.019 |
Scenario | SSIM | FID |
---|---|---|
Without Color Correction | 0.83 | 94.705 |
With Color Correction | 0.76 | 78.944 |
Method | Validation | Testing | ||
---|---|---|---|---|
SSIM | FID | SSIM | FID | |
Autoencoder | 0.69 | 114.096 | 0.64 | 105.638 |
Pix2pix | 0.815 | 97.078 | 0.71 | 101.376 |
cGAN-TV | 0.83 | 94.705 | 0.73 | 93.019 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Rizkinia, M.; Faustine, N.; Okuda, M. Conditional Generative Adversarial Networks with Total Variation and Color Correction for Generating Indonesian Face Photo from Sketch. Appl. Sci. 2022, 12, 10006. https://doi.org/10.3390/app121910006
Rizkinia M, Faustine N, Okuda M. Conditional Generative Adversarial Networks with Total Variation and Color Correction for Generating Indonesian Face Photo from Sketch. Applied Sciences. 2022; 12(19):10006. https://doi.org/10.3390/app121910006
Chicago/Turabian StyleRizkinia, Mia, Nathaniel Faustine, and Masahiro Okuda. 2022. "Conditional Generative Adversarial Networks with Total Variation and Color Correction for Generating Indonesian Face Photo from Sketch" Applied Sciences 12, no. 19: 10006. https://doi.org/10.3390/app121910006
APA StyleRizkinia, M., Faustine, N., & Okuda, M. (2022). Conditional Generative Adversarial Networks with Total Variation and Color Correction for Generating Indonesian Face Photo from Sketch. Applied Sciences, 12(19), 10006. https://doi.org/10.3390/app121910006