Blind Deblurring of Saturated Images Based on Optimization and Deep Learning for Dynamic Visual Inspection on the Assembly Line
Abstract
:1. Introduction
- Dynamic visual inspection inherently poses a blind image deblurring problem that should include blur kernel estimation and deconvolution modeling. Still, linear motion of inspected objects in structured environments (e.g., assembly lines) allows to parameterize the blur kernel.
- Conventional model optimization relies on a handcrafted prior, which is often sophisticated and non-convex. Moreover, the tradeoff between computation time and recovered visual quality should be considered, as image deblurring for dynamic visual inspection requires fast optimization.
- In the case considered for this study, the sheet metal of objects for dynamic visual inspection might intensely reflect light, resulting in a blurred image with saturated pixels, which affect the goodness of fit in data fidelity. Consequently, deblurring based on linear degradation models might fail due to the appearance of severe ringing artifacts.
2. Related Work
2.1. Blur Kernel Estimation
2.2. Deconvolution Modeling
2.3. Outlier Handling
3. Overview of Deblurring for Dynamic Visual Inspection
3.1. Dynamic Visual Inspection System
3.2. Blur Mitigation
4. Proposed Method
4.1. Blur Kernel Estimation
4.1.1. Sharp Edge Selection
4.1.2. Blur Length Estimation
4.2. Deconvolution Modeling
4.2.1. Deconvolution Model Splitting
4.2.2. Outlier Handling
4.2.3. FFDNet Denoising
5. Experimental Evaluation
5.1. Parameter Settings
5.2. Blur Kernel Estimation
5.3. Deconvolution Modeling
6. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Si, L.; Wang, Z.; Xu, R.; Tan, C.; Liu, X.; Xu, J. Image enhancement for surveillance video of coal mining face based on single-scale retinex algorithm combined with bilateral filtering. Symmetry 2017, 9, 93. [Google Scholar] [CrossRef]
- Kang, J.S.; Kim, C.S.; Lee, Y.W.; Cho, S.W.; Park, K.R. Age estimation robust to optical and motion blurring by deep residual CNN. Symmetry 2018, 10, 108. [Google Scholar] [CrossRef]
- Pandey, A.; Gregory, J.W. Iterative blind deconvolution algorithm for deblurring a single PSP/TSP image of rotating surfaces. Sensors 2018, 18, 3075. [Google Scholar] [CrossRef] [PubMed]
- Wang, R.; Ma, G.; Qin, Q.; Shi, Q.; Huang, J. Blind UAV images deblurring based on discriminative networks. Sensors 2018, 18, 2874. [Google Scholar] [CrossRef] [PubMed]
- Fergus, R.; Singh, B.; Hertzmann, A.; Roweis, S.T.; Freeman, W.T. Removing camera shake from a single photograph. ACM Trans. Graph. 2006, 25, 787–794. [Google Scholar] [CrossRef]
- Lu, Q.; Zhou, W.; Fang, L.; Li, H. Robust blur kernel estimation for license plate images from fast moving vehicles. IEEE Trans. Image Process. 2016, 25, 2311–2323. [Google Scholar] [CrossRef]
- Xu, X.; Pan, J.; Zhang, Y.J.; Yang, M.H. Motion blur kernel estimation via deep learning. IEEE Trans. Image Process. 2018, 27, 194–205. [Google Scholar] [CrossRef] [PubMed]
- Moghaddam, M.E.; Jamzad, M. Motion blur identification in noisy images using fuzzy sets. In Proceedings of the IEEE International Symposium on Signal Processing and Information Technology, Athens, Greece, 18–21 December 2005; pp. 862–866. [Google Scholar]
- Cho, S.; Wang, J.; Lee, S. Handling outliers in non-blind image deconvolution. In Proceedings of the IEEE International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 495–502. [Google Scholar]
- Zhang, K.; Zuo, W.; Zhang, L. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 2018, 27, 4608–4622. [Google Scholar] [CrossRef] [PubMed]
- Shan, Q.; Jia, J.; Agarwala, A. High-quality motion deblurring from a single image. ACM Trans. Graph. 2008, 27, 73. [Google Scholar] [CrossRef]
- Xu, L.; Zheng, S.; Jia, J. Unnatural L0 sparse representation for natural image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 1107–1114. [Google Scholar]
- Pan, J.; Hu, Z.; Su, Z.; Yang, M.H. Deblurring text images via L0-regularized intensity and gradient prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 2901–2908. [Google Scholar]
- Pan, J.; Sun, D.; Pfister, H.; Yang, M.H. Blind image deblurring using dark channel prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 27–30 June 2016; pp. 1628–1636. [Google Scholar]
- Yan, Y.; Ren, W.; Guo, Y.; Wang, R.; Cao, X. Image deblurring via extreme channels prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6978–6986. [Google Scholar]
- Oliveira, J.P.; Figueiredo, M.A.T.; Bioucas-Dias, J.M. Parametric blur estimation for blind restoration of natural images: Linear motion and out-of-focus. IEEE Trans. Image Process. 2014, 23, 466–477. [Google Scholar] [CrossRef] [PubMed]
- Moghaddam, M.E.; Jamzad, M. Motion blur identification in noisy images using mathematical models and statistical measures. Pattern Recognit. 2007, 40, 1946–1957. [Google Scholar] [CrossRef]
- Dash, R.; Majhi, B. Motion blur parameters estimation for image restoration. Optik 2014, 125, 1634–1640. [Google Scholar] [CrossRef]
- Yan, R.; Shao, L. Blind image blur estimation via deep learning. IEEE Trans. Image Process. 2016, 25, 1910–1921. [Google Scholar]
- Lu, Y.; Xie, F.; Jiang, Z. Kernel Estimation for Motion Blur Removal Using Deep Convolutional Neural Network. In Proceedings of the IEEE International Conference on Computational Photography, Beijing, China, 17–20 September 2017; pp. 3755–3759. [Google Scholar]
- Moghaddam, M.E.; Jamzad, M. Linear motion blur parameter estimation in noisy images using fuzzy sets and power spectrum. Eur. J. Adv. Signal Process. 2006, 2007, 068985. [Google Scholar] [CrossRef]
- Richardson, W.H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 1972, 62, 55–59. [Google Scholar] [CrossRef]
- Wiener, N. Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications; MIT Press: Cambridge, MA, USA, 1964; Volume 113, pp. 1043–1054. [Google Scholar]
- Oliveira, J.P.; Bioucas-Dias, J.M.; Figueiredo, M.A.T. Adaptive total variation image deblurring: A majorization-minimization approach. Signal Process. 2009, 89, 1683–1693. [Google Scholar] [CrossRef]
- Zuo, W.; Lin, Z. A generalized accelerated proximal gradient approach for total-variation-based image restoration. IEEE Trans. Image Process. 2011, 20, 2748–2759. [Google Scholar] [CrossRef]
- Krishnan, D.; Fergus, R. Fast image deconvolution using hyper-laplacian priors. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 7–10 December 2009; pp. 1033–1041. [Google Scholar]
- Zuo, W.; Meng, D.; Zhang, L.; Feng, X.; Zhang, D. A generalized iterated shrinkage algorithm for non-convex sparse coding. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 217–224. [Google Scholar]
- Danielyan, A.; Katkovnik, V.; Egiazarian, K. BM3D frames and variational image deblurring. IEEE Trans. Image Process. 2012, 21, 1715–1728. [Google Scholar] [CrossRef] [PubMed]
- Dong, W.; Zhang, L.; Shi, G.; Li, X. Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 2013, 22, 1618–1628. [Google Scholar] [CrossRef]
- Yair, N.; Michaeli, T. Multi-scale weighted nuclear norm image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3165–3174. [Google Scholar]
- Libin, S.; Sunghyun, C.; Jue, W.; Hays, J. Edge-based blur kernel estimation using patch priors. In Proceedings of the IEEE International Conference on Computational Photography, Cambridge, MA, USA, 19–21 April 2013; pp. 1–8. [Google Scholar]
- Eigen, D.; Krishnan, D.; Fergus, R. Restoring an image taken through a window covered with dirt or rain. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 633–640. [Google Scholar]
- Schuler, C.J.; Burger, H.C.; Harmeling, S.; Schoelkopf, B. A machine learning approach for non-blind image deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 1067–1074. [Google Scholar]
- Kim, J.; Lee, J.K.; Lee, K.M. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 27–30 June 2016; pp. 1646–1654. [Google Scholar]
- Xu, L.; Ren, J.S.J.; Liu, C.; Jia, J. Deep convolutional neural network for image deconvolution. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 1790–1798. [Google Scholar]
- Ren, W.; Zhang, J.; Ma, L.; Pan, J.; Cao, X.; Zuo, W.; Liu, W.; Yang, M.H. Deep non-blind deconvolution via generalized low-rank approximation. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 2–8 December 2018; pp. 295–305. [Google Scholar]
- Zhang, J.; Pan, J.; Lai, W.S.; Lau, R.W.H.; Yang, M.H. Learning fully convolutional networks for iterative non-blind deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6969–6977. [Google Scholar]
- Zhang, K.; Zuo, W.; Gu, S.; Zhang, L. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2808–2817. [Google Scholar]
- Liu, Y.; Wang, J.; Cho, S.; Finkelstein, A.; Rusinkiewicz, S. A no-reference metric for evaluating the quality of motion deblurring. ACM Trans. Graph. 2013, 32, 175. [Google Scholar] [CrossRef]
- Zhang, L.; Zuo, W. Image restoration: From sparse and low-rank priors to deep priors. IEEE Signal Process. Mag. 2017, 34, 172–179. [Google Scholar] [CrossRef]
- Dong, J.; Pan, J.; Su, Z.; Yang, M.H. Blind image deblurring with outlier handling. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2497–2505. [Google Scholar]
- Whyte, O.; Sivic, J.; Zisserman, A. Deblurring shaken and partially saturated images. Int. J. Comput. Vis. 2014, 110, 185–201. [Google Scholar] [CrossRef]
- Hu, Z.; Cho, S.; Wang, J.; Yang, M.H. Deblurring low-light images with light streaks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 3382–3389. [Google Scholar]
- Geman, D.; Yang, C. Nonlinear image recovery with half-quadratic regularization. IEEE Trans. Image Process. 1995, 4, 932–946. [Google Scholar] [CrossRef]
- Boyd, S.; Parikh, N.; Chu, E.; Peleato, B.; Eckstein, J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 2010, 3, 1–122. [Google Scholar] [CrossRef]
- Chambolle, A.; Pock, T. A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 2011, 40, 120–145. [Google Scholar] [CrossRef]
- Xiao, L.; Heide, F.; Heidrich, W.; Scholkopf, B.; Hirsch, M. Discriminative transfer learning for general image restoration. IEEE Trans. Image Process. 2018, 27, 4091–4104. [Google Scholar] [CrossRef]
- Lai, W.S.; Huang, J.B.; Hu, Z.; Ahuja, N.; Yang, M.H. A comparative study for single image blind deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 27–30 June 2016; pp. 1701–1709. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed]
- Zhang, L.; Zhang, L.; Mou, X.; Zhang, D. FSIM: A feature similarity index for image quality assessment. IEEE Trans. Image Process. 2011, 20, 2378–2386. [Google Scholar] [CrossRef]
Grayscale Image | Color Image |
---|---|
15 convolutional layers | 12 convolutional layers |
Downsampling + noise-level map | |
(Conv 3 × 3 + ReLU) × 1 layer | (Conv 3 × 3 + ReLU) × 1 layer |
(Conv 3 × 3 + BN + ReLU) × 13 layers | (Conv 3 × 3 +BN + ReLU) × 10 layers |
(Conv 3 × 3) × 1 layer | (Conv 3 × 3) × 1 layer |
Upscaling |
Figure 8 | (a) | (b) | (c) | (d) | (a) | (b) | (c) | (d) |
---|---|---|---|---|---|---|---|---|
Measures | SSIM | FSIM | ||||||
RL [22] | 0.6932 | 0.6467 | 0.6126 | 0.7337 | 0.7586 | 0.7675 | 0.7384 | 0.8140 |
IRCNN [38] | 0.7208 | 0.7178 | 0.7300 | 0.7030 | 0.7587 | 0.7685 | 0.7685 | 0.7390 |
VEM [9] | 0.7094 | 0.7279 | 0.7198 | 0.6911 | 0.7384 | 0.7970 | 0.7864 | 0.7063 |
Our method | 0.7263 | 0.7417 | 0.7328 | 0.7055 | 0.7407 | 0.7977 | 0.7889 | 0.7093 |
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, B.; Liu, G.; Wu, J. Blind Deblurring of Saturated Images Based on Optimization and Deep Learning for Dynamic Visual Inspection on the Assembly Line. Symmetry 2019, 11, 678. https://doi.org/10.3390/sym11050678
Wang B, Liu G, Wu J. Blind Deblurring of Saturated Images Based on Optimization and Deep Learning for Dynamic Visual Inspection on the Assembly Line. Symmetry. 2019; 11(5):678. https://doi.org/10.3390/sym11050678
Chicago/Turabian StyleWang, Bodi, Guixiong Liu, and Junfang Wu. 2019. "Blind Deblurring of Saturated Images Based on Optimization and Deep Learning for Dynamic Visual Inspection on the Assembly Line" Symmetry 11, no. 5: 678. https://doi.org/10.3390/sym11050678