# Blind Deblurring of Saturated Images Based on Optimization and Deep Learning for Dynamic Visual Inspection on the Assembly Line

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- Dynamic visual inspection inherently poses a blind image deblurring problem that should include blur kernel estimation and deconvolution modeling. Still, linear motion of inspected objects in structured environments (e.g., assembly lines) allows to parameterize the blur kernel.
- Conventional model optimization relies on a handcrafted prior, which is often sophisticated and non-convex. Moreover, the tradeoff between computation time and recovered visual quality should be considered, as image deblurring for dynamic visual inspection requires fast optimization.
- In the case considered for this study, the sheet metal of objects for dynamic visual inspection might intensely reflect light, resulting in a blurred image with saturated pixels, which affect the goodness of fit in data fidelity. Consequently, deblurring based on linear degradation models might fail due to the appearance of severe ringing artifacts.

## 2. Related Work

#### 2.1. Blur Kernel Estimation

_{0}regularization [12,13] or channel prior [14,15]. These methods model the blur kernel as a projection transform and estimate it in a coarse-to-fine process. This process, however, is not suitable for the linear motion blur appearing during dynamic visual inspection [16]. In fact, linear motion blur in the transform domain is identifiable without an image prior and determined by the blur angle and blur length. Thus, blur kernel estimation in linear motion relies on specific features from the transform domain [8,17].

#### 2.2. Deconvolution Modeling

#### 2.3. Outlier Handling

## 3. Overview of Deblurring for Dynamic Visual Inspection

#### 3.1. Dynamic Visual Inspection System

#### 3.2. Blur Mitigation

## 4. Proposed Method

#### 4.1. Blur Kernel Estimation

#### 4.1.1. Sharp Edge Selection

#### 4.1.2. Blur Length Estimation

^{−1}denote the Fourier transform and its inverse, respectively, and $\Phi $ is the autocorrelation function with the same size as the blurred image. To visualize function $\Phi $, we plotted its columns according to the row in a plane, where the x-axis scale ranged from 1 to the number of rows in $\Phi $. This visualization results in conjugate pairs of valleys that retrieve sinc-shaped curves in the autocorrelation function, where the lowest pair corresponds to the blur length.

#### 4.2. Deconvolution Modeling

#### 4.2.1. Deconvolution Model Splitting

#### 4.2.2. Outlier Handling

#### 4.2.3. FFDNet Denoising

## 5. Experimental Evaluation

#### 5.1. Parameter Settings

#### 5.2. Blur Kernel Estimation

#### 5.3. Deconvolution Modeling

## 6. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Si, L.; Wang, Z.; Xu, R.; Tan, C.; Liu, X.; Xu, J. Image enhancement for surveillance video of coal mining face based on single-scale retinex algorithm combined with bilateral filtering. Symmetry
**2017**, 9, 93. [Google Scholar] [CrossRef] - Kang, J.S.; Kim, C.S.; Lee, Y.W.; Cho, S.W.; Park, K.R. Age estimation robust to optical and motion blurring by deep residual CNN. Symmetry
**2018**, 10, 108. [Google Scholar] [CrossRef] - Pandey, A.; Gregory, J.W. Iterative blind deconvolution algorithm for deblurring a single PSP/TSP image of rotating surfaces. Sensors
**2018**, 18, 3075. [Google Scholar] [CrossRef] [PubMed] - Wang, R.; Ma, G.; Qin, Q.; Shi, Q.; Huang, J. Blind UAV images deblurring based on discriminative networks. Sensors
**2018**, 18, 2874. [Google Scholar] [CrossRef] [PubMed] - Fergus, R.; Singh, B.; Hertzmann, A.; Roweis, S.T.; Freeman, W.T. Removing camera shake from a single photograph. ACM Trans. Graph.
**2006**, 25, 787–794. [Google Scholar] [CrossRef] - Lu, Q.; Zhou, W.; Fang, L.; Li, H. Robust blur kernel estimation for license plate images from fast moving vehicles. IEEE Trans. Image Process.
**2016**, 25, 2311–2323. [Google Scholar] [CrossRef] - Xu, X.; Pan, J.; Zhang, Y.J.; Yang, M.H. Motion blur kernel estimation via deep learning. IEEE Trans. Image Process.
**2018**, 27, 194–205. [Google Scholar] [CrossRef] [PubMed] - Moghaddam, M.E.; Jamzad, M. Motion blur identification in noisy images using fuzzy sets. In Proceedings of the IEEE International Symposium on Signal Processing and Information Technology, Athens, Greece, 18–21 December 2005; pp. 862–866. [Google Scholar]
- Cho, S.; Wang, J.; Lee, S. Handling outliers in non-blind image deconvolution. In Proceedings of the IEEE International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 495–502. [Google Scholar]
- Zhang, K.; Zuo, W.; Zhang, L. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process.
**2018**, 27, 4608–4622. [Google Scholar] [CrossRef] [PubMed] - Shan, Q.; Jia, J.; Agarwala, A. High-quality motion deblurring from a single image. ACM Trans. Graph.
**2008**, 27, 73. [Google Scholar] [CrossRef] - Xu, L.; Zheng, S.; Jia, J. Unnatural L
_{0}sparse representation for natural image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 1107–1114. [Google Scholar] - Pan, J.; Hu, Z.; Su, Z.; Yang, M.H. Deblurring text images via L
_{0}-regularized intensity and gradient prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 2901–2908. [Google Scholar] - Pan, J.; Sun, D.; Pfister, H.; Yang, M.H. Blind image deblurring using dark channel prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 27–30 June 2016; pp. 1628–1636. [Google Scholar]
- Yan, Y.; Ren, W.; Guo, Y.; Wang, R.; Cao, X. Image deblurring via extreme channels prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6978–6986. [Google Scholar]
- Oliveira, J.P.; Figueiredo, M.A.T.; Bioucas-Dias, J.M. Parametric blur estimation for blind restoration of natural images: Linear motion and out-of-focus. IEEE Trans. Image Process.
**2014**, 23, 466–477. [Google Scholar] [CrossRef] [PubMed] - Moghaddam, M.E.; Jamzad, M. Motion blur identification in noisy images using mathematical models and statistical measures. Pattern Recognit.
**2007**, 40, 1946–1957. [Google Scholar] [CrossRef] - Dash, R.; Majhi, B. Motion blur parameters estimation for image restoration. Optik
**2014**, 125, 1634–1640. [Google Scholar] [CrossRef] - Yan, R.; Shao, L. Blind image blur estimation via deep learning. IEEE Trans. Image Process.
**2016**, 25, 1910–1921. [Google Scholar] - Lu, Y.; Xie, F.; Jiang, Z. Kernel Estimation for Motion Blur Removal Using Deep Convolutional Neural Network. In Proceedings of the IEEE International Conference on Computational Photography, Beijing, China, 17–20 September 2017; pp. 3755–3759. [Google Scholar]
- Moghaddam, M.E.; Jamzad, M. Linear motion blur parameter estimation in noisy images using fuzzy sets and power spectrum. Eur. J. Adv. Signal Process.
**2006**, 2007, 068985. [Google Scholar] [CrossRef] - Richardson, W.H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am.
**1972**, 62, 55–59. [Google Scholar] [CrossRef] - Wiener, N. Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications; MIT Press: Cambridge, MA, USA, 1964; Volume 113, pp. 1043–1054. [Google Scholar]
- Oliveira, J.P.; Bioucas-Dias, J.M.; Figueiredo, M.A.T. Adaptive total variation image deblurring: A majorization-minimization approach. Signal Process.
**2009**, 89, 1683–1693. [Google Scholar] [CrossRef] - Zuo, W.; Lin, Z. A generalized accelerated proximal gradient approach for total-variation-based image restoration. IEEE Trans. Image Process.
**2011**, 20, 2748–2759. [Google Scholar] [CrossRef] - Krishnan, D.; Fergus, R. Fast image deconvolution using hyper-laplacian priors. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 7–10 December 2009; pp. 1033–1041. [Google Scholar]
- Zuo, W.; Meng, D.; Zhang, L.; Feng, X.; Zhang, D. A generalized iterated shrinkage algorithm for non-convex sparse coding. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 217–224. [Google Scholar]
- Danielyan, A.; Katkovnik, V.; Egiazarian, K. BM3D frames and variational image deblurring. IEEE Trans. Image Process.
**2012**, 21, 1715–1728. [Google Scholar] [CrossRef] [PubMed] - Dong, W.; Zhang, L.; Shi, G.; Li, X. Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process.
**2013**, 22, 1618–1628. [Google Scholar] [CrossRef] - Yair, N.; Michaeli, T. Multi-scale weighted nuclear norm image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3165–3174. [Google Scholar]
- Libin, S.; Sunghyun, C.; Jue, W.; Hays, J. Edge-based blur kernel estimation using patch priors. In Proceedings of the IEEE International Conference on Computational Photography, Cambridge, MA, USA, 19–21 April 2013; pp. 1–8. [Google Scholar]
- Eigen, D.; Krishnan, D.; Fergus, R. Restoring an image taken through a window covered with dirt or rain. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 633–640. [Google Scholar]
- Schuler, C.J.; Burger, H.C.; Harmeling, S.; Schoelkopf, B. A machine learning approach for non-blind image deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 1067–1074. [Google Scholar]
- Kim, J.; Lee, J.K.; Lee, K.M. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 27–30 June 2016; pp. 1646–1654. [Google Scholar]
- Xu, L.; Ren, J.S.J.; Liu, C.; Jia, J. Deep convolutional neural network for image deconvolution. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 1790–1798. [Google Scholar]
- Ren, W.; Zhang, J.; Ma, L.; Pan, J.; Cao, X.; Zuo, W.; Liu, W.; Yang, M.H. Deep non-blind deconvolution via generalized low-rank approximation. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 2–8 December 2018; pp. 295–305. [Google Scholar]
- Zhang, J.; Pan, J.; Lai, W.S.; Lau, R.W.H.; Yang, M.H. Learning fully convolutional networks for iterative non-blind deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 6969–6977. [Google Scholar]
- Zhang, K.; Zuo, W.; Gu, S.; Zhang, L. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2808–2817. [Google Scholar]
- Liu, Y.; Wang, J.; Cho, S.; Finkelstein, A.; Rusinkiewicz, S. A no-reference metric for evaluating the quality of motion deblurring. ACM Trans. Graph.
**2013**, 32, 175. [Google Scholar] [CrossRef] - Zhang, L.; Zuo, W. Image restoration: From sparse and low-rank priors to deep priors. IEEE Signal Process. Mag.
**2017**, 34, 172–179. [Google Scholar] [CrossRef] - Dong, J.; Pan, J.; Su, Z.; Yang, M.H. Blind image deblurring with outlier handling. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2497–2505. [Google Scholar]
- Whyte, O.; Sivic, J.; Zisserman, A. Deblurring shaken and partially saturated images. Int. J. Comput. Vis.
**2014**, 110, 185–201. [Google Scholar] [CrossRef] - Hu, Z.; Cho, S.; Wang, J.; Yang, M.H. Deblurring low-light images with light streaks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 3382–3389. [Google Scholar]
- Geman, D.; Yang, C. Nonlinear image recovery with half-quadratic regularization. IEEE Trans. Image Process.
**1995**, 4, 932–946. [Google Scholar] [CrossRef] - Boyd, S.; Parikh, N.; Chu, E.; Peleato, B.; Eckstein, J. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn.
**2010**, 3, 1–122. [Google Scholar] [CrossRef] - Chambolle, A.; Pock, T. A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis.
**2011**, 40, 120–145. [Google Scholar] [CrossRef] - Xiao, L.; Heide, F.; Heidrich, W.; Scholkopf, B.; Hirsch, M. Discriminative transfer learning for general image restoration. IEEE Trans. Image Process.
**2018**, 27, 4091–4104. [Google Scholar] [CrossRef] - Lai, W.S.; Huang, J.B.; Hu, Z.; Ahuja, N.; Yang, M.H. A comparative study for single image blind deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 27–30 June 2016; pp. 1701–1709. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process.
**2004**, 13, 600–612. [Google Scholar] [CrossRef] [PubMed] - Zhang, L.; Zhang, L.; Mou, X.; Zhang, D. FSIM: A feature similarity index for image quality assessment. IEEE Trans. Image Process.
**2011**, 20, 2378–2386. [Google Scholar] [CrossRef]

**Figure 2.**Examples of preprocessing for dynamic visual inspection. The examples in (

**a**) and (

**b**) show the blurred image (left) and the result of preprocessing (right).

**Figure 4.**Deblurring with resulting ringing artifacts. (

**a**) Blurred image from dynamic visual inspection system and (

**b**) its restoration via the Richardson–Lucy (RL) deconvolution.

**Figure 5.**Diagram of proposed blind deblurring of saturated images for dynamic visual inspection. Deblurring handles saturated pixels, and denoising proceeds using image prior obtained from FFDNet. Conv, convolution; ReLU, rectified linear unit; BN, batch normalization.

**Figure 6.**Blind image deblurring for dynamic visual inspection based on estimated blur kernel. The estimated blur lengths are (

**a**) 28 px, (

**b**) 20 px, (

**c**) 22 px, and (

**d**) 27 px. The blurred images and their restored versions are shown side by side.

**Figure 7.**Magnified views of examples in Figure 6. The blurred images and their restored versions are shown side by side.

**Figure 8.**Restorations including magnified views of the blurred images in Figure 6 for dynamic visual inspection using different methods where the estimated blur lengths are (

**a**) 28 px, (

**b**) 20 px, (

**c**) 22 px, (

**d**) 27 px. In the left to right and top-down order, deblurring was performed by applying RL [22], iterative decoupled deblurring-block matching and 3D filtering (IDD-BM3D) [28], nonlocally centralized sparse representation (NCSR) [29], hyper-Laplacian (HL) [26], fully convolutional neural network (FCNN) [37], image restoration convolutional neural network (IRCNN) [38], variational expectation–maximization (VEM) [9], and our proposed method.

Grayscale Image | Color Image |
---|---|

15 convolutional layers | 12 convolutional layers |

Downsampling + noise-level map | |

(Conv 3 × 3 + ReLU) × 1 layer | (Conv 3 × 3 + ReLU) × 1 layer |

(Conv 3 × 3 + BN + ReLU) × 13 layers | (Conv 3 × 3 +BN + ReLU) × 10 layers |

(Conv 3 × 3) × 1 layer | (Conv 3 × 3) × 1 layer |

Upscaling |

**Table 2.**Score comparisons of the structural similarity index (SSIM) and feature similarity index (FSIM) on the restorations of Figure 8.

Figure 8 | (a) | (b) | (c) | (d) | (a) | (b) | (c) | (d) |
---|---|---|---|---|---|---|---|---|

Measures | SSIM | FSIM | ||||||

RL [22] | 0.6932 | 0.6467 | 0.6126 | 0.7337 | 0.7586 | 0.7675 | 0.7384 | 0.8140 |

IRCNN [38] | 0.7208 | 0.7178 | 0.7300 | 0.7030 | 0.7587 | 0.7685 | 0.7685 | 0.7390 |

VEM [9] | 0.7094 | 0.7279 | 0.7198 | 0.6911 | 0.7384 | 0.7970 | 0.7864 | 0.7063 |

Our method | 0.7263 | 0.7417 | 0.7328 | 0.7055 | 0.7407 | 0.7977 | 0.7889 | 0.7093 |

**Table 3.**Average run time (in seconds) on the restorations of Figure 8.

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Wang, B.; Liu, G.; Wu, J.
Blind Deblurring of Saturated Images Based on Optimization and Deep Learning for Dynamic Visual Inspection on the Assembly Line. *Symmetry* **2019**, *11*, 678.
https://doi.org/10.3390/sym11050678

**AMA Style**

Wang B, Liu G, Wu J.
Blind Deblurring of Saturated Images Based on Optimization and Deep Learning for Dynamic Visual Inspection on the Assembly Line. *Symmetry*. 2019; 11(5):678.
https://doi.org/10.3390/sym11050678

**Chicago/Turabian Style**

Wang, Bodi, Guixiong Liu, and Junfang Wu.
2019. "Blind Deblurring of Saturated Images Based on Optimization and Deep Learning for Dynamic Visual Inspection on the Assembly Line" *Symmetry* 11, no. 5: 678.
https://doi.org/10.3390/sym11050678