A Two-Stage Network for Zero-Shot Low-Illumination Image Restoration
Abstract
:1. Introduction
- We propose a two-stage low-illumination image restoration network, in which a pre-decomposition submodule is incorporated to divide the original image into illumination, reflectance, and feature. Moreover, the whole network is optimized in a zero-shot way instead of using supervised learning.
- To guide the decomposition network focusing on the dark area of the image, a new loss function is proposed for our network. The loss function can also obtain relatively clearer texture features in the dark area, and avoid the problem of overexposure or underexposure in the others areas.
- Experiments show that our method achieves better performance on the benchmark datasets. Compared with recent methods based on decomposition theory, the proposed method can visually better retain the detailed features of images and avoid the problem of overexposure. There is a significant improvement in PSNR and SSIM (reference evaluation indices), and NIQE and LOE (no-reference evaluation indices).
2. Related Work
2.1. Retinex-Model-Based Approach
2.2. Learning-Based Approach
3. Methodology
3.1. Decom-Net
3.2. Enhance-Net
4. Loss Function
4.1. Reconstruction Loss
4.2. Smoothness Loss
4.3. Feature Estimation Loss
5. Experimental Results and Analysis
5.1. Subjective Evaluation
5.2. Objective Evaluation
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Fang, M.; Li, H.; Lei, L. A review on low light video image enhancement algorithms. J. Changchun Univ. Sci. Technol. 2016, 39, 56–64. [Google Scholar]
- Ackar, H.; Abd Almisreb, A.; Saleh, M.A. A review on image enhancement techniques. Southeast Eur. J. Soft Comput. 2019, 8, 42–48. [Google Scholar] [CrossRef]
- Wang, Y.-F.; Liu, H.-M.; Fu, Z.-W. Low-light image enhancement via the absorption light scattering model. IEEE Trans. Image Process. 2019, 28, 5679–5690. [Google Scholar] [CrossRef]
- Yan, X.; Liu, T.; Fu, M.; Ye, M.; Jia, M. Bearing Fault Feature Extraction Method Based on Enhanced Differential Product Weighted Morphological Filtering. Sensors 2022, 22, 6184. [Google Scholar] [CrossRef]
- Han, Q.; Liu, Y.; Li, Z.; Zhang, Y.; Chen, Y. Investigation of energy accommodation coefficient at gas-solid interface of a hypersonic flying vehicle. Aerosp. Sci. Technol. 2022, 126, 107585. [Google Scholar] [CrossRef]
- Abdullah-Al-Wadud, M.; Kabir, M.H.; Dewan, M.A.A.; Chae, O. A dynamic histogram equalization for image contrast enhancement. IEEE Trans. Consum. Electron. 2007, 53, 593–600. [Google Scholar] [CrossRef]
- Zhu, Y.; Feng, K.; Hua, C.; Wang, X.; Hu, Z.; Wang, H.; Su, H. Model Analysis and Experimental Investigation of Soft Pneumatic Manipulator for Fruit Grasping. Sensors 2022, 22, 4532. [Google Scholar] [CrossRef]
- Li, C.; Guo, C.; Han, L.; Jiang, J.; Cheng, M.-M.; Gu, J.; Loy, C.C. Lighting the darkness in the deep learning era. arXiv 2021, arXiv:2104.10729. [Google Scholar]
- Jin, X.; Bagavathiannan, M.; McCullough, P.E.; Chen, Y.; Yu, J. A deep learning-based method for classification, detection, and localization of weeds in turfgrass. Pest Manag. Sci. 2022, 78, 4809–4821. [Google Scholar] [CrossRef] [PubMed]
- Stark, J.A. Adaptive image contrast enhancement using generalizations of histogram equalization. IEEE Trans. Image Process. 2000, 9, 889–896. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Jobson, D.J.; Rahman, Z.-u.; Woodell, G.A. Properties and performance of a center/surround retinex. IEEE Trans. Image Process. 1997, 6, 451–462. [Google Scholar] [CrossRef] [PubMed]
- Zhu, H.; Chan, F.H.; Lam, F.K. Image contrast enhancement by constrained local histogram equalization. Comput. Vis. Image Underst. 1999, 73, 281–290. [Google Scholar] [CrossRef]
- Rahman, Z.-u.; Jobson, D.J.; Woodell, G.A. Multi-scale retinex for color image enhancement. In Proceedings of the 3rd IEEE International Conference on Image Processing, Lausanne, Switzerland, 19 September 1996; pp. 1003–1006. [Google Scholar]
- Barnard, K.; Funt, B. Investigations into Multi-Scale Retinex. In Colour Imaging: Vision and Technology; Wiley: Hoboken, NJ, USA, 1998. [Google Scholar]
- Parthasarathy, S.; Sankaran, P. An automated multi scale retinex with color restoration for image enhancement. In Proceedings of the 2012 National Conference on Communications (NCC), Kharagpur, India, 3–5 February 2012; pp. 1–5. [Google Scholar]
- Lore, K.G.; Akintayo, A.; Sarkar, S. LLNet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognit. 2017, 61, 650–662. [Google Scholar] [CrossRef] [Green Version]
- Wei, C.; Wang, W.; Yang, W.; Liu, J. Deep retinex decomposition for low-light enhancement. arXiv 2018, arXiv:1808.04560. [Google Scholar]
- Wang, W.; Wu, X.; Yuan, X.; Gao, Z. An experiment-based review of low-light image enhancement methods. IEEE Access 2020, 8, 87884–87917. [Google Scholar] [CrossRef]
- Jobson, D.J.; Rahman, Z.-u.; Woodell, G.A. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Process. 1997, 6, 965–976. [Google Scholar] [CrossRef] [Green Version]
- Zotin, A. Fast algorithm of image enhancement based on multi-scale retinex. Procedia Comput. Sci. 2018, 131, 6–14. [Google Scholar] [CrossRef]
- Shen, L.; Yue, Z.; Feng, F.; Chen, Q.; Liu, S.; Ma, J. Msr-net: Low-light image enhancement using deep convolutional network. arXiv 2017, arXiv:1711.02488. [Google Scholar]
- Wang, W.; Wei, C.; Yang, W.; Liu, J. Gladnet: Low-light enhancement network with global awareness. In Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China, 15–19 May 2018; pp. 751–755. [Google Scholar]
- Jiang, Y.; Gong, X.; Liu, D.; Cheng, Y.; Fang, C.; Shen, X.; Yang, J.; Zhou, P.; Wang, Z. Enlightengan: Deep light enhancement without paired supervision. IEEE Trans. Image Process. 2021, 30, 2340–2349. [Google Scholar] [CrossRef]
- Fu, Y.; Hong, Y.; Chen, L.; You, S. LE-GAN: Unsupervised low-light image enhancement network using attention module and identity invariant loss. Knowl. Based Syst. 2022, 240, 108010. [Google Scholar] [CrossRef]
- Zhang, L.; Zhang, L.; Liu, X.; Shen, Y.; Zhang, S.; Zhao, S. Zero-shot restoration of back-lit images using deep internal learning. In Proceedings of the 27th ACM International Conference on Multimedia, Nice, France, 21–25 October 2019; pp. 1623–1631. [Google Scholar]
- Wang, R.; Zhang, Q.; Fu, C.-W.; Shen, X.; Zheng, W.-S.; Jia, J. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 6849–6857. [Google Scholar]
- Zhu, A.; Zhang, L.; Shen, Y.; Ma, Y.; Zhao, S.; Zhou, Y. Zero-shot restoration of underexposed images via robust retinex decomposition. In Proceedings of the 2020 IEEE International Conference on Multimedia and Expo (ICME), London, UK, 6–10 July 2020; pp. 1–6. [Google Scholar]
- Li, C.; Guo, J.; Porikli, F.; Pang, Y. LightenNet: A convolutional neural network for weakly illuminated image enhancement. Pattern Recognit. Lett. 2018, 104, 15–22. [Google Scholar] [CrossRef]
- Guo, C.; Li, C.; Guo, J.; Loy, C.C.; Hou, J.; Kwong, S.; Cong, R. Zero-reference deep curve estimation for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 1780–1789. [Google Scholar]
- Lim, S.; Kim, W. DSLR: Deep stacked Laplacian restorer for low-light image enhancement. IEEE Trans. Multimed. 2020, 23, 4272–4284. [Google Scholar] [CrossRef]
- Yu, C.; Xu, X.; Lin, H.; Xinyan, Y. Low-illumination image enhancement method based on a fog-degraded model. J. Image Graph. 2017, 22, 1194–1205. [Google Scholar]
- Ignatov, A.; Kobyshev, N.; Timofte, R.; Vanhoey, K.; Van Gool, L. Wespe: Weakly supervised photo enhancer for digital cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–22 June 2018; pp. 691–700. [Google Scholar]
- Mittal, A.; Soundararajan, R.; Bovik, A.C. Making a “completely blind” image quality analyzer. IEEE Signal Process. Lett. 2012, 20, 209–212. [Google Scholar] [CrossRef]
- Fu, X.; Zeng, D.; Huang, Y.; Zhang, X.-P.; Ding, X. A weighted variational model for simultaneous reflectance and illumination estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 2782–2790. [Google Scholar]
Method | PSNR↑ | SSIM↑ |
---|---|---|
Input | 5.10 | 0.19 |
HE [10] | 15.23 | 0.59 |
Retinex [19] | 10.92 | 0.37 |
ExCNet [25] | 14.32 | 0.75 |
RRDNet [27] | 8.72 | 0.60 |
LightenNet [28] | 7.87 | 0.52 |
Zero-DCE [29] | 11.98 | 0.76 |
DSLR [30] | 10.78 | 0.67 |
LLNet [16] | 14.00 | 0.78 |
Ours | 17.84 | 0.74 |
Method | NIQE↓ | LOE↓ |
---|---|---|
Input | 28.12 | 0 |
HE [10] | 30.76 | 254.87 |
Retinex [19] | 23.33 | 291.14 |
ExCNet [25] | 17.96 | 316.85 |
RRDNet [27] | 18.47 | 251.37 |
LightenNet [28] | 20.97 | 305.50 |
Zero-DCE [29] | 21.50 | 351.37 |
DSLR [30] | 18.40 | 272.58 |
LLNet [16] | 26.35 | 302.76 |
Ours | 18.02 | 249.25 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tang, H.; Fei, L.; Zhu, H.; Tao, H.; Xie, C. A Two-Stage Network for Zero-Shot Low-Illumination Image Restoration. Sensors 2023, 23, 792. https://doi.org/10.3390/s23020792
Tang H, Fei L, Zhu H, Tao H, Xie C. A Two-Stage Network for Zero-Shot Low-Illumination Image Restoration. Sensors. 2023; 23(2):792. https://doi.org/10.3390/s23020792
Chicago/Turabian StyleTang, Hao, Linfeng Fei, Hongyu Zhu, Huanjie Tao, and Chao Xie. 2023. "A Two-Stage Network for Zero-Shot Low-Illumination Image Restoration" Sensors 23, no. 2: 792. https://doi.org/10.3390/s23020792
APA StyleTang, H., Fei, L., Zhu, H., Tao, H., & Xie, C. (2023). A Two-Stage Network for Zero-Shot Low-Illumination Image Restoration. Sensors, 23(2), 792. https://doi.org/10.3390/s23020792