Water-Air Interface Imaging: Recovering the Images Distorted by Surface Waves via an Efficient Registration Algorithm
Abstract
:1. Introduction
2. Related Work
3. Methods
3.1. Principles
3.2. Lucky-Patch Search with Initial Registration
3.3. Image Registration Based on the JADE & LBFGS Optimization Strategy
3.3.1. Optimization Strategy
3.4. Post-Processing
Algorithm 1: The correct format is:An efficient method for recovering the images distorted by surface waves |
Input: Distorted Video sequence |
Output: Distortion-free sequence and high-quality frame |
While do |
if |
else |
end |
for |
end |
end |
Computer the frame |
for each patch sequence |
end |
4. Results
4.1. Test with the Data Set from the Air-to-Water Imaging Scenario
4.1.1. Analysis of Restoration Results
4.1.2. Analysis of Computational Efficiency
4.2. Test with the Data Set from the Water-to-Air Imaging Scenario
5. Discussion
5.1. Analysis of the Lucky-Path Search Strategy
5.1.1. Selection the Image Quality Metric
5.1.2. Analysis of the Evolution of Reference Frame
5.2. Analysis on the Choice of Reference Frame Deblurring or Frame Blurring
5.3. Comparison of the Optimization Strategy
5.4. Compared with Deep Learning Method
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Data Availability Statement
Conflicts of Interest
References
- Molkov, A.A.; Dolin, L.S. The Snell’s window image for remote sensing of the upper sea layer: Results of practical application. J. Mar. Sci. Eng. 2019, 7, 70. [Google Scholar] [CrossRef] [Green Version]
- Gardashov, R.H.; Gardashov, E.R.; Gardashova, T.H. Recovering the instantaneous images of underwater objects distorted by surface waves. J. Mod. Opt. 2021, 68, 19–28. [Google Scholar] [CrossRef]
- Li, N.; Thapa, S.; Whyte, C.; Reed, A.W.; Jayasuriya, S.; Ye, J. Unsupervised non-rigid image distortion removal via grid deformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, 10–17 October 2021; pp. 2522–2532. [Google Scholar]
- Zhang, Z.; Tang, Y.G.; Yang, K. A two-stage restoration of distorted underwater images using compressive sensing and image registration. Adv. Manuf. 2021, 9, 273–285. [Google Scholar] [CrossRef]
- Tahtali, M. Imaging Techniques Through Unfavorable Media. In Propagation Through and Characterization of Atmospheric and Oceanic Phenomena; Optica Publishing Group: Vancouver, BC, Canada, 2022; p. PTh5F. 3. [Google Scholar]
- Jian, B.; Ma, C.; Zhu, D.; Sun, Y.; Ao, J. Seeing through Wavy Water–Air Interface: A Restoration Model for Instantaneous Images Distorted by Surface Waves. Future Internet 2022, 14, 236. [Google Scholar] [CrossRef]
- Treibitz, T.; Schechner, Y.; Kunz, C.; Singh, H. Flat refractive geometry. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 34, 51–65. [Google Scholar] [CrossRef] [Green Version]
- Diamant, Y.; Schechner, Y.Y. Overcoming visual reverberations. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 23–28 June 2008; IEEE: Piscataway, NJ, USA, 2008; pp. 1–8. [Google Scholar]
- Schechner, Y.Y.; Karpel, N. Attenuating natural flicker patterns. In Proceedings of the Oceans’ 04 MTS/IEEE Techno-Ocean’04 (IEEE Cat. No. 04CH37600), Kobe, Japan, 9–12 November 2004; IEEE: Piscataway, NJ, USA, 2004; Volume 3, pp. 1262–1268. [Google Scholar]
- Suiter, H.; Flacco, N.; Carter, P.; Tong, K.; Ries, R.; Gershenson, M. Optics near the snell angle in a water-to-air change of medium. In Proceedings of the OCEANS 2007, Vancouver, BC, Canada, 29 September–4 October 2007; IEEE: Piscataway, NJ, USA, 2007; pp. 1–12. [Google Scholar]
- Lynch, D.K. Snell’s window in wavy water. Appl. Opt. 2015, 54, B8–B11. [Google Scholar] [CrossRef] [Green Version]
- Holst, G.C.; Krapels, K.A. Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXIV. In Proceedings of SPIE; SPIE: Bellingham, DC, USA, 2013; Volume 8706, p. 870601-1. [Google Scholar]
- Efros, A.; Isler, V.; Shi, J.; Visontai, M. Seeing through water. Adv. Neural Inf. Process. Syst. 2005, 7, 393–400. [Google Scholar]
- Donate, A.; Dahme, G.; Ribeiro, E. Classification of textures distorted by water waves. In Proceedings of the 18th International Conference on Pattern Recognition (ICPR’06), Hong Kong, China, 20–24 August 2006; IEEE: Piscataway, NJ, USA, 2006; Volume 2, pp. 421–424. [Google Scholar]
- Wen, Z.; Lambert, A.; Fraser, D.; Li, H. Bispectral analysis and recovery of images distorted by a moving water surface. Appl. Opt. 2010, 49, 6376–6384. [Google Scholar] [CrossRef]
- Kanaev, A.V.; Hou, W.; Woods, S. Multi-frame underwater image restoration. In Electro-Optical and Infrared Systems: Technology and Applications VIII; SPIE: Bellingham, DC, USA, 2011; Volume 8185, pp. 208–215. [Google Scholar]
- Kanaev, A.V.; Hou, W.; Restaino, S.R.; Matt, S.; Gładysz, S. Restoration of images degraded by underwater turbulence using structure tensor oriented image quality (STOIQ) metric. Opt. Express 2015, 23, 17077–17090. [Google Scholar] [CrossRef]
- Zhang, R.; He, D.; Li, Y.; Huang, L.; Bao, X. Synthetic imaging through wavy water surface with centroid evolution. Opt. Express 2018, 26, 26009–26019. [Google Scholar] [CrossRef]
- Tian, Y.; Narasimhan, S.G. Seeing through water: Image restoration using model-based tracking. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan, 29 September–2 October 2009; IEEE: Piscataway, NJ, USA, 2009; pp. 2303–2310. [Google Scholar]
- Tian, Y.; Narasimhan, S.G. Globally optimal estimation of nonrigid image distortion. Int. J. Comput. Vis. 2012, 98, 279–302. [Google Scholar] [CrossRef]
- Seemakurthy, K.; Rajagopalan, A.N. Deskewing of underwater images. IEEE Trans. Image Process. 2015, 24, 1046–1059. [Google Scholar] [CrossRef]
- Li, Z.; Murez, Z.; Kriegman, D.; Ramamoorthi, R.; Chandraker, M. Learning to see through turbulent water. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 512–520. [Google Scholar]
- James, J.G.; Agrawal, P.; Rajwade, A. Restoration of Non-rigidly Distorted Underwater Images using a Combination of Compressive Sensing and Local Polynomial Image Representations. In Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019. [Google Scholar]
- Thapa, S.; Li, N.; Ye, J. Learning to Remove Refractive Distortions from Underwater Images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 5007–5016. [Google Scholar]
- Oreifej, O.; Shu, G.; Pace, T.; Shah, M. A two-stage reconstruction approach for seeing through water. In Proceedings of the CVPR 2011, Colorado Springs, CO, USA, 20–25 June 2011; IEEE: Piscataway, NJ, USA, 2011; pp. 1153–1160. [Google Scholar]
- Hu, W.; Yuan, X.; Zhang, W.; Tan, Y. Removing Water Fluctuation via Motion Field-based Kernel Regression. J. Inf. Comput. Sci. 2014, 11, 5289–5296. [Google Scholar] [CrossRef]
- Halder, K.K.; Paul, M.; Tahtali, M.; Anavatti, S.G.; Murshed, M. Correction of geometrically distorted underwater images using shift map analysis. JOSA A 2017, 34, 666–673. [Google Scholar] [CrossRef]
- Zhang, Z.; Yang, X. Reconstruction of distorted underwater images using robust registration. Opt. Express 2019, 27, 9996–10008. [Google Scholar] [CrossRef]
- Sun, T.; Tang, Y.; Zhang, Z. Structural information reconstruction of distorted underwater images using image registration. Appl. Sci. 2020, 10, 5670. [Google Scholar] [CrossRef]
- Yang, L.; Ji, H. A variational EM framework with adaptive edge selection for blind motion deblurring [C]//. 2019: 10167-10176. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 2019, 10167–10176. [Google Scholar]
- Wen, F.; Ying, R.; Liu, Y.; Liu, P.; Truong, T.K. A simple local minimal intensity prior and an improved algorithm for blind image deblurring. IEEE Trans. Circuits Syst. Video Technol. 2020, 31, 2923–2937. [Google Scholar]
- Xu, Z.; Chen, H.; Li, Z. Fast blind deconvolution using a deeper sparse patch-wise maximum gradient prior. Signal Process. Image Commun. 2021, 90, 116050. [Google Scholar] [CrossRef]
- Aubailly, M.; Vorontsov, M.A.; Carhart, G.W.; Valley, M.T. Automated video enhancement from a stream of atmospherically-distorted images: The lucky-region fusion approach. In Atmospheric Optics: Models, Measurements, and Target-in-the-Loop Propagation III; SPIE: Bellingham, DC, USA, 2009; Volume 7463, pp. 104–113. [Google Scholar]
- Cox, C.; Munk, W. Slopes of the sea surface deduced from photographs of sun glitter. 1956. Available online: https://escholarship.org/uc/item/1p202179 (accessed on 28 November 2022).
- Cai, C.; Meng, H.; Qiao, R.; Wang, Y. Water–air imaging: Distorted image reconstruction based on a twice registration algorithm. Mach. Vis. Appl. 2021, 32, 64. [Google Scholar] [CrossRef]
- Alterman, M.; Swirski, Y.; Schechner, Y.Y. STELLA MARIS: Stellar marine refractive imaging sensor. In Proceedings of the 2014 IEEE International Conference on Computational Photography (ICCP), Santa Clara, CA, USA, 2–4 May 2014. [Google Scholar]
- Alterman, M.; Schechner, Y.Y. 3D in Natural Random Refractive Distortions; Javidi, B., Son, J.-Y., Eds.; International Society for Optics and Photonics: Bellingham, DC, USA, 2016; Volume 9867. [Google Scholar]
- Rauf, H.T.; Bangyal WH, K.; Lali, M.I. An adaptive hybrid differential evolution algorithm for continuous optimization and classification problems. Neural Comput. Appl. 2021, 33, 10841–10867. [Google Scholar] [CrossRef]
- Chang, D.; Sun, S.; Zhang, C. An accelerated linearly convergent stochastic L-BFGS algorithm. IEEE Trans. Neural Netw. Learn. Syst. 2019, 30, 3338–3346. [Google Scholar] [CrossRef]
- Rueckert, D.; Sonoda, L.; Hayes, C.; Hill, D.; Leach, M.; Hawkes, D. Nonrigid registration using free-form deformations: Application to breast MR images. IEEE Trans. Med. Imaging 1999, 18, 712–721. [Google Scholar] [CrossRef] [PubMed]
- Kaufman, L.; Rousseeuw, P.J. Finding Groups in Data: An Introduction to Cluster Analysis; John Wiley & Sons: Hoboken, NJ, USA, 2009. [Google Scholar]
- Zhang, X.; Feng, X.; Wang, W.; Xue, W. Edge strength similarity for image quality assessment. IEEE Signal Process. Lett. 2013, 20, 319–322. [Google Scholar] [CrossRef]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhang, L.; Zhang, L.; Mou, X.; Zhang, D. FSIM: A feature similarity index for image quality assessment. IEEE Trans. Image Process. 2011, 20, 2378–2386. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Ryu, E.K.; Yin, W. Large-Scale Convex Optimization: Algorithms & Analyses via Monotone Operators; Cambridge University Press: Cambridge, UK, 2022. [Google Scholar]
- Dai, Y.H. A perfect example for the BFGS method. Math. Program. 2013, 138, 501–530. [Google Scholar] [CrossRef]
- Gower, R.M.; Richtárik, P. Randomized quasi-Newton updates are linearly convergent matrix inversion algorithms. SIAM J. Matrix Anal. Appl. 2017, 38, 1380–1409. [Google Scholar] [CrossRef] [Green Version]
- Lin, Z.; Chen, M.; Ma, Y. The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. arXiv preprint 2010, arXiv:1009.5055. [Google Scholar]
- Jian, B. Observing Through Wavy Water Surface: Reconstruction the Distortion-Free Image from a Distorted Video Sequence. 2022. (accessed on 28 November 2022). [CrossRef]
- Panetta, K.; Gao, C.; Agaian, S. Human-visual-system-inspired underwater image quality measures. IEEE J. Ocean. Eng. 2015, 41, 541–551. [Google Scholar] [CrossRef]
- Mittal, A.; Soundararajan, R.; Bovik, A.C. Making a “completely blind” image quality analyzer. IEEE Signal Process. Lett. 2012, 20, 209–212. [Google Scholar] [CrossRef]
- He, K.; Sun, J.; Tang, X. Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 35, 1397–1409. [Google Scholar] [CrossRef]
- Rahaman, M.M.; Li, C.; Yao, Y.; Kulwa, F.; Rahman, M.A.; Wang, Q.; Qi, S.; Kong, F.; Zhu, X.; Zhao, X. Identification of COVID-19 samples from chest X-Ray images using deep learning: A comparison of transfer learning approaches. J. X-Ray Sci. Technol. 2020, 28, 821–839. [Google Scholar] [CrossRef]
- Chen, H.; Li, C.; Wang, G.; Li, X.; Rahaman, M.; Sun, H.; Hu, W.; Li, Y.; Liu, W.; Sun, C.; et al. Gashis-transformer: A multi-scale visual transformer approach for gastric histopathology image classification. arXiv e-prints 2021, arXiv:2104.14528. [Google Scholar]
- Ma, J.; Jiang, X.; Fan, A.; Jiang, J.; Yan, J. Image matching from handcrafted to deep features: A survey. Int. J. Comput. Vis. 2021, 129, 23–79. [Google Scholar] [CrossRef]
- Zhang, J.; Li, C.; Kosov, S.; Grzegorzek, M.; Shirahama, K.; Jiang, T.; Sun, C.; Li, Z.; Li, H. LCU-Net: A novel low-cost U-Net for environmental microorganism image segmentation. Pattern Recognit. 2021, 115, 107885. [Google Scholar] [CrossRef]
- Chen, H.; Li, C.; Li, X.; Rahaman, M.M.; Hu, W.; Li, Y.; Liu, W.; Sun, C.; Sun, H.; Huang, X. IL-MCAM: An interactive learning and multi-channel attention mechanism-based weakly supervised colorectal histopathology image classification approach. Comput. Biol. Med. 2022, 143, 105265. [Google Scholar] [CrossRef]
- Li, X.; Li, C.; Rahaman, M.M.; Sun, H.; Li, X.; Wu, J.; Yao, Y.; Grzegorzek, M. A comprehensive review of computer-aided whole-slide image analysis: From datasets to feature extraction, segmentation, classification and detection approaches. Artif. Intell. Rev. 2022, 55, 4809–4878. [Google Scholar] [CrossRef]
Brick | Checkerboard | Large Fonts | Middle Fonts | Small Fonts | |
---|---|---|---|---|---|
frame size | |||||
patch size |
Data Sets | Methods | MSE(L) | PSNR(H) | SSIM(H) | UIQM(H) |
---|---|---|---|---|---|
Middle fonts | Tian [19] | 0.0079 | 21.0275 | 0.6598 | 2.5215 |
Oreifej [25] | 0.0073 | 21.3829 | 0.7412 | 2.6182 | |
Z. Zhang [28] | 0.0164 | 17.8407 | 0.5773 | 2.5773 | |
T. Sun [29] | 0.0119 | 19.2361 | 0.6137 | 2.6277 | |
Our method | 0.0053 | 22.7591 | 0.7891 | 2.7056 | |
Small fonts | Tian [19] | 0.0043 | 23.6916 | 0.6475 | 2.5852 |
Oreifej [25] | 0.0038 | 24.1490 | 0.7098 | 2.8036 | |
Z. Zhang [28] | 0.0061 | 22.1768 | 0.5689 | 2.7409 | |
T. Sun [29] | 0.0068 | 21.7069 | 0.5424 | 2.7853 | |
Our method | 0.0036 | 24.4752 | 0.7243 | 2.8409 | |
Bricks | Tian [19] | 0.0108 | 19.6800 | 0.6017 | 2.4071 |
Oreifej [25] | 0.0051 | 22.9278 | 0.6361 | 2.5631 | |
Z. Zhang [28] | 0.0182 | 17.3951 | 0.4112 | 2.4807 | |
T. Sun [29] | 0.0118 | 19.2700 | 0.4373 | 2.5846 | |
Our method | 0.0051 | 22.9278 | 0.6497 | 2.7126 | |
Large fonts | Tian [19] | 2.2026 | |||
Oreifej [25] | 2.0533 | ||||
Z. Zhang [28] | 2.1072 | ||||
T. Sun [29] | 2.1218 | ||||
Our method | 2.2851 | ||||
Checkerboard | Tian [19] | 2.4701 | |||
Oreifej [25] | 2.4900 | ||||
Z. Zhang [28] | 2.5504 | ||||
T. Sun [29] | 2.5240 | ||||
Our method | 2.5928 |
Data Sets | UIQM(H) Oreifej/Z. Zhang/T. Sun/Our method |
---|---|
ripple | 1.9231/1.9544/2.0128/2.1573 |
gravity | 1.3119/1.3218/1.3266/1.3347 |
Iteration | Methods | MSE(L) | PSNR(H) | SSIM(H) |
---|---|---|---|---|
1th | Oreifej | 0.0120 | 19.1934 | 0.4847 |
Z. Zhang | 0.0149 | 18.2634 | 0.3940 | |
T. Sun | 0.0117 | 19.3080 | 0.4907 | |
Our method | 0.0120 | 19.1934 | 0.4847 | |
2th | Oreifej | 0.0107 | 19.7244 | 0.5576 |
Z. Zhang | 0.0171 | 17.6689 | 0.4293 | |
T. Sun | 0.0122 | 19.1431 | 0.5904 | |
Our method | 0.0072 | 21.4257 | 0.6491 | |
3th | Oreifej | 0.0073 | 21.3932 | 0.7123 |
Z. Zhang | 0.0225 | 16.4742 | 0.4364 | |
T. Sun’s | 0.0130 | 18.8677 | 0.6054 | |
Our method | 0.0057 | 22.4323 | 0.7475 | |
4th | Oreifej | 0.0072 | 21.4247 | 0.7340 |
Z. Zhang | 0.0121 | 19.1718 | 0.6115 | |
T. Sun | 0.0132 | 18.7993 | 0.6086 | |
Our method | 0.0056 | 22.5441 | 0.7700 | |
5th | Oreifej | 0.0072 | 21.4247 | 0.7407 |
Z. Zhang | 0.0172 | 17.6536 | 0.5737 | |
T. Sun | 0.0133 | 18.7726 | 0.6090 | |
Our method | 0.0055 | 22.5986 | 0.7781 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Jian, B.; Ma, C.; Zhu, D.; Huang, Q.; Ao, J. Water-Air Interface Imaging: Recovering the Images Distorted by Surface Waves via an Efficient Registration Algorithm. Entropy 2022, 24, 1765. https://doi.org/10.3390/e24121765
Jian B, Ma C, Zhu D, Huang Q, Ao J. Water-Air Interface Imaging: Recovering the Images Distorted by Surface Waves via an Efficient Registration Algorithm. Entropy. 2022; 24(12):1765. https://doi.org/10.3390/e24121765
Chicago/Turabian StyleJian, Bijian, Chunbo Ma, Dejian Zhu, Qihong Huang, and Jun Ao. 2022. "Water-Air Interface Imaging: Recovering the Images Distorted by Surface Waves via an Efficient Registration Algorithm" Entropy 24, no. 12: 1765. https://doi.org/10.3390/e24121765
APA StyleJian, B., Ma, C., Zhu, D., Huang, Q., & Ao, J. (2022). Water-Air Interface Imaging: Recovering the Images Distorted by Surface Waves via an Efficient Registration Algorithm. Entropy, 24(12), 1765. https://doi.org/10.3390/e24121765