A New Method for Training CycleGAN to Enhance Images of Cold Seeps in the Qiongdongnan Sea
Abstract
:1. Introduction
- The effect of cold seep image enhancements using MSRCR on different imaging devices in different detectors was tested, and it was shown that a single enhancement coefficient or a fixed table could not meet the requirements of different scenes.
- CycleGAN was trained using the standard dataset and applied to the image enhancement of cold seep images. It was observed that the model worked well in some conditions and failed in other conditions.
- We found an effective way to build datasets to train CycleGAN with the help of MSRCR for cases in which a clear image dataset is difficult to obtain.
- Finally, an active underwater image enhancement CycleGAN that can be applied to practical applications rather than standard data models was trained. Compared with previous studies, the training ideas proposed in this paper may be applied to any underwater scene, with good universal applicability.
2. Materials and Methods
2.1. Principle of MSRCR
2.2. Principle of CycleGAN
2.2.1. Net Structure
2.2.2. Cycle Consistency Loss
3. Tests and Results
3.1. Dataset Preparation Using the MSRCR Method
3.2. CycleGAN Training and Underwater Image Enhancement
3.3. Versatility Test of the CycleGAN
3.4. EnlightenGAN Test
4. Discussion
5. Conclusions and Future Steps
- A better conventional underwater image enhancement method can be found compared to MSRCR, and a new dataset for training can be built.
- Improved datasets can be obtained by implementing better enhancement evaluation methods.
- Our method is generic, and in future work, we will apply more updated models to explore the path of underwater image enhancement and explore more dataset-building schemes.
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Yang, L.; Liu, B.; Xu, M.J.; Liu, S.X.; Guan, Y.X.; Gu, Y. Characteristics of active cold seeps in Qiongdongnan Sea Area of the northern South China Sea. Chin. J. Geophys. 2018, 61, 2905–2914. (In Chinese) [Google Scholar]
- Zhao, J.; Liang, Q.Y.; Wei, J.G.; Tao, J.; Yang, S.X.; Liang, J.Q.; Lu, J.A.; Wang, J.J.; Fang, Y.X.; Gong, Y.H.; et al. Seafloor geology and geochemistry characteristic of methane seepage of the ”Haima” cold seep, northwestern slope of the South China Sea. Geochimica 2020, 49, 108–118. (In Chinese) [Google Scholar]
- Feng, J.; Li, N.; Luo, M.; Liang, J.; Yang, S.; Wang, H.; Chen, D. A quantitative assessment of methane-derived carbon cycling at the cold seeps in the northwestern South China Sea. Minerals 2020, 10, 256. [Google Scholar] [CrossRef]
- Di, P.; Feng, D.; Tao, J.; Chen, D. Using time-series videos to quantify methane bubbles flux from natural cold seeps in the South China Sea. Minerals 2020, 10, 216. [Google Scholar] [CrossRef]
- Raveendran, S.; Patil, M.; Birajdar, G.K. Underwater image enhancement: A comprehensive review, recent trends, challenges and applications. Artif. Intell. Rev. 2021, 54, 5413–5467. [Google Scholar] [CrossRef]
- Liu, R.; Fan, X.; Zhu, M.; Hou, M.; Lu, Z. Real-world underwater enhancement: Challenges, benchmarks, and solutions under natural light. IEEE Trans. Circuits Syst. Video Technol. 2020, 30, 4861–4875. [Google Scholar] [CrossRef]
- Buchsbaum, G. A spatial processor model for object color perception. J. Frankl. Inst. 1980, 310, 337–350. [Google Scholar] [CrossRef]
- Forsyth, D.A. A novel algorithm for color constancy. Int. J. Comput. Vis. 1990, 5, 5–35. [Google Scholar] [CrossRef]
- Liu, Y.C.; Chan, W.H.; Chen, Y.Q. Automatic white balance for digital still camera. IEEE Trans. Consum. Electron. 2004, 41, 460–466. [Google Scholar]
- Bae, Y.; Jang, J.H.; Ra, J.B. Gamut-adaptive correction in color image processing. IEEE Int. Conf. Image Process. 2010, 3597–3660. [Google Scholar]
- Land, E.H.; McCann, J.J. Lightness and retinex theory. J. Opt. Soc. Am. 1971, 61, 1–11. [Google Scholar] [CrossRef]
- Land, E.H. The retinex theory of color vision. Sci. Am. 1981, 237, 108–128. [Google Scholar] [CrossRef]
- Land, E.H. An alternative technique for the computation of the designator in the retinex theory of color vision. Proc. Natl. Acad. Sci. USA 1986, 83, 3078–3080. [Google Scholar] [CrossRef]
- Joshi, K.R.; Kamathe, R.S. Quantification of retinex in enhancement of weather degraded images. In Proceedings of the 2008 International Conference on Audio, Language and Image Processing, Shanghai, China, 7–9 July 2008; pp. 1229–1233. [Google Scholar]
- Rahman, Z.; Jobson, D.J.; Woodell, G.A. Multi-scale retinex for color image enhancement. In Proceedings of the 3rd IEEE International Conference on Image Processing, Lausanne, Switzerland, 19 September 1996; Volume 3, pp. 1003–1006. [Google Scholar]
- Jobson, D.J.; Rahman, Z.; Woodell, G.A. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Process. 1997, 6, 965–976. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Ke, L.; Xujian, L. De-hazing and enhancement methods for underwater and low-light images. Acta Optica Sin. 2020, 40, 73–85. [Google Scholar]
- Hu, K.; Zhang, Y.; Lu, F.; Deng, Z.; Liu, Y. An underwater image enhancement algorithm based on MSR parameter optimization. J. Mar. Sci. Eng. 2020, 8, 741. [Google Scholar] [CrossRef]
- Anwar, S.; Li, C. Diving deeper into underwater image enhancement: A survey. Singal Process Image Commun. 2020, 89, 115978. [Google Scholar] [CrossRef]
- Perez, J.; Attanasio, A.C.; Nechyporenko, N.; Sanz, P.J. A deep learning approach for underwater image enhancement. In Biomedical Applications Based on Natural and Artificial Computing, Proceedings of the International Work-Conference on the Interplay Between Natural and Artificial Computation, IWINAC 2017, Corunna, Spain, 19–23 June 2017; Ferrández, V.J., Álvarez-Sánchez, J., Paz, L.F., Toledo, M.J., Adeli, H., Eds.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2017; Volume 10338, pp. 183–192. [Google Scholar]
- Anwar, S.; Li, C.; Porikli, F. Deep underwater image enhancement. arXiv 2018, arXiv:1807.03528. [Google Scholar]
- Li, J.; Skinner, K.A.; Eustice, R.M.; Johnson, R.M. WaterGAN: Unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Robot. Autom. Lett. 2018, 3, 387–394. [Google Scholar] [CrossRef]
- Han, Y.; Huang, L.; Hong, Z.; Cao, S.; Zhang, Y.; Wang, J. Deep supervised residual dense network for underwater image enhancement. Sensors 2021, 21, 3289. [Google Scholar] [CrossRef]
- Hu, K.; Zhang, Y.; Weng, C.; Wang, P.; Deng, Z.; Liu, Y. An underwater image enhancement algorithm based on generative adversarial network and natural image quality evaluation index. J. Mar. Sci. Eng. 2021, 9, 691. [Google Scholar] [CrossRef]
- Zhang, X.; Fang, X.; Pan, M.; Yuan, L.; Zhang, Y.; Yuan, M.; Lv, S.; Yu, H. A marine organism detection framework based on the joint optimization of image enhancement and object detection. Sensors 2021, 21, 7205. [Google Scholar] [CrossRef] [PubMed]
- Goodfellow, I.J.; Abadie, J.P.; Mirza, M.; Xu, B.; Farley, D.W.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. arXiv 2014, arXiv:1406.2661. [Google Scholar]
- Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2242–2251. [Google Scholar]
- Fabbri, C.; Jahidul, I.; Sattar, J. Enhancing underwater imagery using generative adversarial networks. In Proceedings of the IEEE International Conference on Robotics and Automation, Brisbane, Australia, 22–25 May 2018; pp. 7159–7165. [Google Scholar]
- Ahn, J.; Yasukawa, S.; Sonoda, T.; Ura, T.; Ishii, K. Erratum to: Enhancement of deep-sea floor images obtained by an underwater vehicle and its evaluation by crab recognition. J. Mar. Sci. Technol. 2017, 22, 758–770. [Google Scholar] [CrossRef]
- Lu, J.; Li, N.; Zhang, S.; Yu, Z.; Zheng, H.; Zheng, B. Multi-scale adversarial network for underwater image restoration. Opt. Laser Technol. 2019, 110, 105–113. [Google Scholar] [CrossRef]
- Du, R.; Li, W.; Chen, S.; Li, C.; Zhang, Y. Unpaired underwater image enhancement based on CycleGAN. Information 2022, 13, 1. [Google Scholar] [CrossRef]
- Jiang, Y.; Gong, X.; Liu, D.; Cheng, Y.; Fang, C.; Shen, X.; Yang, J.; Zhou, P.; Wang, Z. EnlightenGAN: Deep light enhancement without paired supervision. IEEE Trans. Image Process. 2021, 30, 2340–2349. [Google Scholar] [CrossRef]
- Ma, J.; Fan, X.; Ni, J.; Zhu, X.; Xiong, C. Multi-scale retinex with color restoration image enhancement based on Gaussian filtering and guided filtering. Int. J. Mod. Phys. B 2017, 31, 1744077. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. arXiv 2015, arXiv:1409.1556. [Google Scholar]
- Guo, Y.; Li, H.; Zhuang, P. Underwater image enhancement using a multiscale dense generative adversarial network. IEEE J. Ocean. Eng. 2019, 45, 862–870. [Google Scholar] [CrossRef]
- Neumann, S. Drawing and layer modes. In GIMP Pocket Reference; Neumann, S., Ed.; O’Reilly & Associates: Sebastopol, CA, USA, 2008; pp. 93–95. [Google Scholar]
- Panetta, K.; Gao, C.; Agaian, S.S. Human-visual-system-inspired underwater image quality measures. IEEE J. Ocean. Eng. 2016, 41, 541–551. [Google Scholar] [CrossRef]
- Chen, L. Unsupervised learning: Deep generation model. In Introduction to Deep Learning with MindSpore; TsingHua University Press: Beijing, China, 2020; pp. 158–185. [Google Scholar]
- Deimling, J.S.; Linke, P.; Schmidt, M.; Rehder, G. Ongoing methane discharge at well site 22/4b (North Sea) and discovery of a spiral vortex bubble plume motion. Mar. Pet. Geol. 2015, 68, 718–730. [Google Scholar] [CrossRef]
MSRCR | Qiongdongnan Cold Seeps | North Sea Cold Seeps | |||
Msrcr Dyn values | Mean | Variance | Mean | Variance | |
Original images | 2.180 | 0.204 | 2.454 | 0.382 | |
Msrcr-0.4 | 1.034 | 0.047 | 1.488 | 0.110 | |
Msrcr-1.2 | 2.054 | 0.1632 | 2.143 | 0.233 | |
Msrcr-2.0 | 2.045 | 0.219 | 1.813 | 0.247 | |
Msrcr-2.8 | 1.923 | 0.234 | 1.567 | 0.244 | |
Msrcr-3.6 | 1.752 | 0.236 | 1.520 | 0.200 | |
Msrcr-4.4 | 1.586 | 0.228 | 1.337 | 0.164 | |
Msrcr-5.2 | 1.440 | 0.213 | 1.186 | 0.135 | |
Msrcr-6.0 | 1.318 | 0.197 | 1.063 | 0.110 | |
Msrcr best value chosen | 2.444 | 0.189 | 2.572 | 0.197 | |
CycleGAN | CycleGAN Training times | Mean | Variance | Mean | Variance |
20 | 2.344 | 0.202 | 2.572 | 0.126 | |
40 | 2.398 | 0.181 | 2.606 | 0.140 | |
60 | 2.478 | 0.157 | 2.590 | 0.171 | |
80 | 2.421 | 0.158 | 2.572 | 0.208 | |
100 | 2.343 | 0.171 | 2.544 | 0.218 | |
120 | 2.489 | 0.174 | 2.637 | 0.215 | |
140 | 2.422 | 0.166 | 2.628 | 0.187 | |
160 | 2.499 | 0.154 | 2.839 | 0.112 | |
180 | 2.465 | 0.173 | 2.635 | 0.234 | |
200 | 2.426 | 0.177 | 2.647 | 0.260 |
EnlightenGAN Training Times | Mean | Variance |
---|---|---|
20 | 2.355 | 0.190 |
40 | 2.384 | 0.165 |
60 | 2.352 | 0.174 |
80 | 2.199 | 0.177 |
100 | 2.231 | 0.178 |
120 | 2.500 | 0.174 |
140 | 2.213 | 0.187 |
160 | 2.189 | 0.192 |
180 | 2.186 | 0.193 |
200 | 2.264 | 0.176 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Li, Y.; Yang, S.; Gong, Y.; Cao, J.; Hu, G.; Deng, Y.; Tian, D.; Zhou, J. A New Method for Training CycleGAN to Enhance Images of Cold Seeps in the Qiongdongnan Sea. Sensors 2023, 23, 1741. https://doi.org/10.3390/s23031741
Li Y, Yang S, Gong Y, Cao J, Hu G, Deng Y, Tian D, Zhou J. A New Method for Training CycleGAN to Enhance Images of Cold Seeps in the Qiongdongnan Sea. Sensors. 2023; 23(3):1741. https://doi.org/10.3390/s23031741
Chicago/Turabian StyleLi, Yuanheng, Shengxiong Yang, Yuehua Gong, Jingya Cao, Guang Hu, Yutian Deng, Dongmei Tian, and Junming Zhou. 2023. "A New Method for Training CycleGAN to Enhance Images of Cold Seeps in the Qiongdongnan Sea" Sensors 23, no. 3: 1741. https://doi.org/10.3390/s23031741