Beyond Staircasing Effect: Robust Image Smoothing via ℓ_{0} Gradient Minimization and Novel Gradient Constraints^{ †}
Abstract
:1. Introduction
 Image smoothing while maintaining gradient characteristics of reference image: Existing smoothing methods based on the ${\mathit{\ell}}_{0}$ gradient and TV do not explicitly consider any constraints in the gradient domain. In the proposed method, the local smoothness properties of a reference image can be explicitly considered constraints in the gradient domain. Therefore, we can suppress artifacts, including the staircasing effect, through image smoothing.
 Strict or flexible gradient constraints on the sign of gradients: Since the boxtype gradient constraint is strict with respect to the sign of gradients, gradient reversals are well suppressed. In contrast, the balltype constraint is flexible with respect to the sign of gradients, allowing robust image smoothing, even when a reference image is degraded by noise and has different shading characteristics, including gradient reversals.
2. Preliminaries
2.1. ${\mathit{\ell}}_{0}$ Gradient
2.2. Alternating Direction Method of Multipliers
2.3. Proximal Tools
3. Proposed Methods
3.1. Gradient Constraints
3.1.1. BoxType Gradient Constraint
3.1.2. BallType Gradient Constraint
3.2. ${\mathit{\ell}}_{0}$Smoothing Based on BoxType Gradient Constraint
3.2.1. Minimization Problem
3.2.2. Optimization
Algorithm 1 Proposed algorithm for (18). 

3.3. ${\mathit{\ell}}_{0}$Smoothing Based on BallType Gradient Constraint
3.4. ${\mathit{\ell}}_{0}$ Gradient Projection with Gradient Constraint
4. Experiments
4.1. BoxType vs. BallType Gradient Constraint
4.2. Detail Enhancement
4.3. Tone Mapping
4.4. JPEG Artifact Removal in ClipArt Images
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
 Rudin, L.I.; Osher, S.; Fatemi, E. Nonlinear Total Variation Based Noise Removal Algorithms. Physics D 1992, 60, 259–268. [Google Scholar] [CrossRef]
 Afonso, M.V.; BioucasDias, J.M.; Figueiredo, M.A. An augmented Lagrangian approach to the constrained optimization formulation of imaging inverse problems. IEEE Trans. Image Process. 2011, 20, 681–695. [Google Scholar] [CrossRef] [PubMed]
 Xu, L.; Zheng, S.; Jia, J. Unnatural L_{0} Sparse Representation for Natural Image Deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA, 23–28 June 2013; pp. 1107–1114. [Google Scholar] [CrossRef]
 Pan, J.; Hu, Z.; Su, Z.; Yang, M. Deblurring Text Images via L_{0}Regularized Intensity and Gradient Prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014; pp. 2901–2908. [Google Scholar] [CrossRef]
 Farbman, Z.; Fattal, R.; Lischinski, D.; Szeliski, R. Edgepreserving Decompositions for Multiscale Tone and Detail Manipulation. ACM Trans. Graph. 2008, 27, 67. [Google Scholar] [CrossRef]
 Gastal, E.S.L.; Oliveira, M.M. Domain Transform for Edgeaware Image and Video Processing. ACM Trans. Graph. 2011, 30, 69. [Google Scholar] [CrossRef]
 Durand, F.; Dorsey, J. Fast Bilateral Filtering for the Display of Highdynamicrange Images. ACM Trans. Graph. 2002, 21, 257–266. [Google Scholar] [CrossRef]
 Fattal, R. Edgeavoiding Wavelets and Their Applications. ACM Trans. Graph. 2009, 28, 22. [Google Scholar] [CrossRef]
 Liang, Z.; Xu, J.; Zhang, D.; Cao, Z.; Zhang, L. A Hybrid L_{1}L_{0} Layer Decomposition Model for Tone Mapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018; pp. 4758–4766. [Google Scholar] [CrossRef]
 Figueiredo, M.A.T.; BioucasDias, J.M. Restoration of Poissonian Images Using Alternating Direction Optimization. IEEE Trans. Image Process. 2010, 19, 3133–3145. [Google Scholar] [CrossRef]
 Xu, Q.; Yu, H.; Mou, X.; Zhang, L.; Hsieh, J.; Wang, G. LowDose Xray CT Reconstruction via Dictionary Learning. IEEE Trans. Med. Imaging 2012, 31, 1682–1697. [Google Scholar] [CrossRef]
 Ramani, S.; Fessler, J.A. A SplittingBased Iterative Algorithm for Accelerated Statistical XRay CT Reconstruction. IEEE Trans. Med. Imaging 2012, 31, 677–688. [Google Scholar] [CrossRef]
 Graber, G.; Pock, T.; Bischof, H. Online 3D reconstruction using convex optimization. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, 6–13 November 2011; pp. 708–711. [Google Scholar] [CrossRef]
 Ferstl, D.; Reinbacher, C.; Ranftl, R.; Ruether, M.; Bischof, H. Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV), Sydney, Australia, 1–8 December 2013; pp. 993–1000. [Google Scholar] [CrossRef]
 Wang, Q.; Tao, Y.; Lin, H. EdgeAware Volume Smoothing Using L_{0} Gradient Minimization. Comput. Graph. Forum 2015, 34, 131–140. [Google Scholar] [CrossRef]
 Yuan, Q.; Zhang, L.; Shen, H. Hyperspectral Image Denoising Employing a SpectralSpatial Adaptive Total Variation Model. IEEE Trans. Geosci. Remote Sens. 2012, 50, 3660–3677. [Google Scholar] [CrossRef]
 Yuan, Q.; Zhang, L.; Shen, H. Hyperspectral Image Denoising with a SpatialSpectral View Fusion Strategy. IEEE Trans. Geosci. Remote Sens. 2014, 52, 2314–2325. [Google Scholar] [CrossRef]
 Yuan, Q.; Zhang, Q.; Li, J.; Shen, H.; Zhang, L. Hyperspectral Image Denoising Employing a SpatialSpectral Deep Residual Convolutional Neural Network. IEEE Trans. Geosci. Remote Sens. 2019, 57, 1205–1218. [Google Scholar] [CrossRef]
 LikformanSulem, L.; Darbon, J.; Smith, E.H.B. Enhancement of historical printed document images by combining Total Variation regularization and Nonlocal Means filtering. Image Vis. Comput. 2011, 29, 351–363. [Google Scholar] [CrossRef]
 Ge, F.; He, L. A denoising method based on L_{0} gradient minimization and guided filter for ancient Chinese calligraphy works on steles. EURASIP J. Image Vid. Process. 2019, 2019, 32. [Google Scholar] [CrossRef]
 Tomasi, C.; Manduchi, R. Bilateral filtering for gray and color images. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV), Bombay, India, 7 January 1998; pp. 839–846. [Google Scholar] [CrossRef]
 He, K.; Sun, J.; Tang, X. Guided Image Filtering. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1397–1409. [Google Scholar] [CrossRef] [PubMed]
 Seo, H.J.; Milanfar, P. Robust flash denoising/deblurring by iterative guided filtering. EURASIP J. Adv. Signal Process. 2012, 2012, 3. [Google Scholar] [CrossRef]
 Baba, T.; Matsuoka, R.; Shirai, K.; Okuda, M. Misaligned Image Integration With Local Linear Model. IEEE Trans. Image Process. 2016, 25, 2035–2044. [Google Scholar] [CrossRef]
 Matsuoka, R.; Shirai, K.; Okuda, M. Referencebased local color distribution transformation method and its application to image integration. Signal Process. Image Comm. 2019, 76, 231–242. [Google Scholar] [CrossRef]
 Buades, A.; Coll, B.; Morel, J.M. A NonLocal Algorithm for Image Denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA, 20–26 June 2005; pp. 60–65. [Google Scholar] [CrossRef]
 Zhan, Y.; Ding, M.; Xiao, F.; Zhang, X. An Improved Nonlocal Means Filter for Image Denoising. In Proceedings of the International Conference on Intelligent Computation and BioMedical Instrumentation (ICBMI), Wuhan, China, 14–17 December 2011; pp. 31–34. [Google Scholar] [CrossRef]
 BioucasDias, J.M.; Figueiredo, M.A.T. A New TwIST: TwoStep Iterative Shrinkage/Thresholding Algorithms for Image Restoration. IEEE Trans. Image Process. 2007, 16, 2992–3004. [Google Scholar] [CrossRef]
 Figueiredo, M.A.T.; Nowak, R.D.; Wright, S.J. Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems. IEEE J. Sel. Top. Signal Process. 2007, 1, 586–597. [Google Scholar] [CrossRef]
 Wright, S.J.; Nowak, R.D.; Figueiredo, M.A.T. Sparse Reconstruction by Separable Approximation. IEEE Trans. Signal Process. 2009, 57, 2479–2493. [Google Scholar] [CrossRef]
 Xu, L.; Lu, C.; Xu, Y.; Jia, J. Image Smoothing via L_{0} Gradient Minimization. ACM Trans. Graph. 2011, 30, 174. [Google Scholar] [CrossRef]
 Miyata, T.; Sakai, Y. Vectorized total variation defined by weighted L infinity norm for utilizing inter channel dependency. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Orlando, FL, USA, 30 September–3 October 2012; pp. 3057–3060. [Google Scholar] [CrossRef]
 Xu, L.; Yan, Q.; Xia, Y.; Jia, J. Structure Extraction from Texture via Relative Total Variation. ACM Trans. Graph. 2012, 31, 139. [Google Scholar] [CrossRef]
 Condat, L. A Generic Proximal Algorithm for Convex Optimization—Application to Total Variation Minimization. IEEE SPS Lett. 2014, 21, 985–989. [Google Scholar] [CrossRef]
 Ono, S.; Yamada, I. Decorrelated Vectorial Total Variation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014; pp. 4090–4097. [Google Scholar] [CrossRef]
 Bi, S.; Han, X.; Yu, Y. An L_{1} Image Transform for Edgepreserving Smoothing and Scenelevel Intrinsic Decomposition. ACM Trans. Graph. 2015, 34, 78. [Google Scholar] [CrossRef]
 Ono, S. L_{0} Gradient Projection. IEEE Trans. Image Process. 2017, 26, 1554–1564. [Google Scholar] [CrossRef] [PubMed]
 Kobayashi, I.; Matsuoka, R. A Study on JPEG Artifact Removal by FourDirectional Differencebased ℓ_{0,1} norm Regularization. In Proceedings of the IEEE Global Conference on Consumer Electronics (GCCE), Osaka, Japan, 18–21 October 2022; pp. 724–725. [Google Scholar] [CrossRef]
 Ma, X.; Li, X.; Zhou, Y.; Zhang, C. Image smoothing based on global sparsity decomposition and a variable parameter. Comput. Vis. Media 2021, 7, 483–497. [Google Scholar] [CrossRef]
 Liu, W.; Zhang, P.; Lei, Y.; Huang, X.; Yang, J.; Ng, M. A Generalized Framework for EdgePreserving and StructurePreserving Image Smoothing. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 44, 6631–6648. [Google Scholar] [CrossRef]
 Huang, J.; Wang, H.; Wang, X.; Ruzhansky, M. SemiSparsity for Smoothing Filters. IEEE Trans. Image Process. 2023, 32, 1627–1639. [Google Scholar] [CrossRef]
 Li, Y.; Brown, M.S. Single Image Layer Separation Using Relative Smoothness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014; pp. 2752–2759. [Google Scholar] [CrossRef]
 Shibata, T.; Akai, Y.; Matsuoka, R. Reflection Removal Using RGBD Images. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 1862–1866. [Google Scholar] [CrossRef]
 Li, Y.; Tan, R.T.; Guo, X.; Lu, J.; Brown, M.S. Single Image Rain Streak Decomposition Using Layer Priors. IEEE Trans. Image Process. 2017, 26, 3874–3885. [Google Scholar] [CrossRef] [PubMed]
 Jiang, T.; Huang, T.; Zhao, X.; Deng, L.; Wang, Y. A Novel TensorBased Video Rain Streaks Removal Approach via Utilizing Discriminatively Intrinsic Priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 2818–2827. [Google Scholar] [CrossRef]
 Jiang, T.; Huang, T.; Zhao, X.; Deng, L.; Wang, Y. FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors. IEEE Trans. Image Process. 2019, 28, 2089–2102. [Google Scholar] [CrossRef] [PubMed]
 Jeon, J.; Cho, S.; Tong, X.; Lee, S. Intrinsic Image Decomposition Using StructureTexture Separation and Surface Normals. In European Conference Computer Vision (ECCV); Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T., Eds.; Springer: Cham, Switzerland, 2014; pp. 218–233. [Google Scholar]
 Matsuoka, R.; Baba, T.; Rizkinia, M.; Okuda, M. White Balancing by Using Multiple Images via Intrinsic Image Decomposition. IEICE Trans. Inf. Syst. 2015, 98, 1562–1570. [Google Scholar] [CrossRef]
 Matsuoka, R.; Ono, S.; Okuda, M. TransformedDomain Robust MultipleExposure Blending With Huber Loss. IEEE Access 2019, 7, 162282–162296. [Google Scholar] [CrossRef]
 Palsson, F.; Sveinsson, J.R.; Ulfarsson, M.O. A New Pansharpening Algorithm Based on Total Variation. IEEE Geosci. Remote Sens. Lett. 2014, 11, 318–322. [Google Scholar] [CrossRef]
 He, X.; Condat, L.; BioucasDias, J.M.; Chanussot, J.; Xia, J. A New Pansharpening Method Based on Spatial and Spectral Sparsity Priors. IEEE Trans. Image Process. 2014, 23, 4160–4174. [Google Scholar] [CrossRef] [PubMed]
 Takeyama, S.; Ono, S.; Kumazawa, I. Robust and Effective Hyperspectral Pansharpening Using SpatioSpectral Total Variation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 15–20 April 2018; pp. 1603–1607. [Google Scholar] [CrossRef]
 Takeyama, S.; Ono, S.; Kumazawa, I. Hyperspectral Pansharpening Using Noisy Panchromatic Image. In Proceedings of the APSIPA Annual Summit and Conference (APSIPA ASC), Honolulu, HI, USA, 12–15 November 2018; pp. 880–885. [Google Scholar] [CrossRef]
 Zach, C.; Pock, T.; Bischof, H. A Globally Optimal Algorithm for Robust TVL_{1} Range Image Integration. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV), Venice, Italy, 22–29 October 2017; pp. 1–8. [Google Scholar] [CrossRef]
 Sénchez Pèrez, J.; MeinhardtLlopis, E.; Facciolo, G. TVL_{1} Optical Flow Estimation. Image Process. Line 2013, 3, 137–150. [Google Scholar] [CrossRef]
 Huber, P.J. Robust estimation of a location parameter. Ann. Math. Stat. 1964, 35, 73–101. [Google Scholar] [CrossRef]
 Ono, S.; Yamada, I. Signal Recovery With Certain Involved Convex DataFidelity Constraints. IEEE Trans. Signal Process. 2015, 63, 6149–6163. [Google Scholar] [CrossRef]
 Nguyen, R.M.H.; Brown, M.S. Fast and Effective L_{0} Gradient Minimization by Region Fusion. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV), Santiago, Chile, 7–13 December 2015; pp. 208–216. [Google Scholar] [CrossRef]
 Arvanitopoulos, N.; Achanta, R.; Süsstrunk, S. Single Image Reflection Suppression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 1752–1760. [Google Scholar] [CrossRef]
 Gabay, D.; Mercier, B. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 1976, 2, 17–40. [Google Scholar] [CrossRef]
 Matsuoka, R.; Kyochi, S.; Ono, S.; Okuda, M. Joint Sparsity and Order Optimization Based on ADMM With NonUniform Group Hard Thresholding. IEEE Trans. Circuits Syst. I Regul. Pap. 2018, 65, 1602–1613. [Google Scholar] [CrossRef]
 Akai, Y.; Shibata, T.; Matsuoka, R.; Okuda, M. L_{0} Smoothing Based on Gradient Constraints. In Proceedings of the IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 3943–3947. [Google Scholar] [CrossRef]
 Moreau, J.J. Fonctions convexes duales et points proximaux dans un espace hilbertien. C. R. Acad. Sci. 1962, 255, 2897–2899. [Google Scholar]
 Reinhard, E.; Stark, M.; Shirley, P.; Ferwerda, J. Photographic Tone Reproduction for Digital Images. ACM Trans. Graph. 2002, 21, 267–276. [Google Scholar] [CrossRef]
 Wang, Z.; Bovik, A.; Sheikh, H.; Simoncelli, E. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. 
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Matsuoka, R.; Okuda, M. Beyond Staircasing Effect: Robust Image Smoothing via ℓ_{0} Gradient Minimization and Novel Gradient Constraints. Signals 2023, 4, 669686. https://doi.org/10.3390/signals4040037
Matsuoka R, Okuda M. Beyond Staircasing Effect: Robust Image Smoothing via ℓ_{0} Gradient Minimization and Novel Gradient Constraints. Signals. 2023; 4(4):669686. https://doi.org/10.3390/signals4040037
Chicago/Turabian StyleMatsuoka, Ryo, and Masahiro Okuda. 2023. "Beyond Staircasing Effect: Robust Image Smoothing via ℓ_{0} Gradient Minimization and Novel Gradient Constraints" Signals 4, no. 4: 669686. https://doi.org/10.3390/signals4040037