Local and Nonlocal Steering Kernel Weighted Total Variation Model for Image Denoising

To eliminate heavy noise and retain more scene details, we propose a structure-oriented total variation (TV) model based on data dependent kernel function and TV criterion for image denoising application. The innovative model introduces the weights produced from the local and nonlocal symmetry features involved in the image itself to pick more precise solutions in the TV denoising process. As a result, the proposed local and nonlocal steering kernel weighted TV model yields excellent noise suppression and structure-preserving performance. The experimental results verify the validity of the proposed model in objective quantitative indices and subjective visual appearance.

Many traditional image processing methods are exploited on the basis of the local structural regularity assumption present in natural images.The rationale of denoising algorithms is to make use of the structural patterns to regularize the ill-posed restoration problem and make the texture region less blurry and the flat region smoother [13][14][15].The gradient based total variation (TV) is a state-of-the-art method that has been proven to restore real scenes from noisy images effectively [16].However, the TV model tends to introduce staircase effect and texture loss.To surmount the inherent defects of TV regularization, some improved TV models with structure preserving performance are presented.By combining intensity into the definition of the distance between pixels, bilateral filtering [17] clearly relieves the blurring effect of the Gaussian filter and provides detail preserving performance.In view of this, the bilateral total variation (BTV) model [18] and non-local total variation (NLTV) model [19] are successively exploited to more precisely restore the details by fusing the idea of bilateral filtering and non-local means filtering [20] with TV criterion.However, the BTV model only considers the spatial distance but ignores the neighborhood similarity in obtaining the gradient of a pixel, which leads to the derogation of structure information in the recovered image.Moreover, the nonsymmetrical structure preserving ability of the NLTV model tends to be weakened with growing noise strength, which may result from neglecting the robust local structure constraints.
Existing investigations have shown that feature descriptors using local steering kernel (LSK) are robust to noise interruption [21].The reason lies in that LSK is exploited to solve image noise and uncertainty by estimating the local structure.Moreover, a patch containing a flat region, textural clutter, and structural part have significant differences for LSK-based descriptors.In view of this, we encode Symmetry 2019, 11, 329 2 of 16 the noise corrupting image patches using the LSK method [22] to robustly recover the original image structure and remove the noise disturbance.
Inspired by the fact that the disconnected nonlocal components with spatial support provide more useful information in image restoration [23], we further combine the nonlocal self-similarity [24,25] with the local constraints of LSK to weight the respective measurements of TV, which enhances the structure preserving capability of the TV model in denoising application.In this way, our proposed local and nonlocal steering kernel weighted TV model for image denoising can robustly estimate the local structure of the image, as well as effectively remove the annoying noise.Obviously, it utilizes the redundancy of symmetrically similar patches in the corrupted image and the sensitivity of local feature description to implement the denoising task.
The rest of this paper is structured as follows.Section 2 briefly reviews the related works of this study, Section 3 introduces the presented algorithm and discusses its mechanism, Section 4 states experimental results and analysis, and Section 5 gives the conclusion of the paper.

Related Works
where Y denotes the observation, V indicates the zero-mean additive white noise perturbation that is uncorrelated to the true image X.

Regularization Based Denoising Framework
For the image denoising problem, we need to solve the minimization problem as [26] X = Argmin where X is the denoised image, Y − X p p is the data fidelity item used to retain the original image characteristics and reduce the image distortion, the scalar µ is used to properly balance the fidelity term Y − X p p with the regularizer Υ X , a smaller µ increases the impact of fidelity item and provides stable convergence, while a larger µ tends to enhance the influence of regularization term and induces fast tracking; • p p denotes the L p (1 ≤ p ≤ 2) norm of residual.The well-known Tikhonov regularizer [27,28] is defined as where Γ represents the high-pass operator.It is clear that the Tikhonov regularization method tends to constrain the total energy of the image or implements spatial smoothing.
Since the noise and image textures both contain abundant high frequency components, the regularization procedure easily remove both of them indiscriminately, and the corresponding result is that the denoised images lose most of their sharp edges and details.
The popular edge preserving regularization strategy for image restoration applications is the TV method [16].The TV criterion penalizes the L 1 norm of gradient amplitude that measures the total change of image and is expressed as where ∇ is the gradient operator.The advantage of TV criterion lies in that it can retain edges during refactoring without seriously penalizing the steep gradients [29,30].On the basis of TV criterion and bilateral filter, a spatially adaptive regularizer called bilateral total variation (BTV) that attempts to eliminate the staircase effect and detail the loss problem of the TV prior model is presented in [18].The BTV regularizer is mathematically formulated as x S m y X 1 (5) where operators S l x and S m y , respectively, implement l pixel horizontal translation and m pixel vertical translation, so as to present multi-scale derivatives.P indicates the radius of the search window.The scalar weight α(0 < α < 1) is utilized to produce spatial decay for the regularization terms.Larger α provides a larger impact of neighbor pixels, but an exceeded α generally blots out the scene details, as is the case with the TV prior, and results in over-smooth effects.On the contrary, a tiny α sharply attenuates the spread of weight with the increase of spatial distance, which decreases the noise suppression capability and leads to slower convergence.A proper selection of α plays an important role in balancing noise suppression and detail preservation.
Local image structures are often repeated themselves within the image and across the image sequences.The redundant information contained in similar patches has a vital significance for solving most ill-posed image restoration problems.This is because similar patches are generally considered to be various observations of the same real scene.
To overcome the performance decline of local derivative-based prior models in noise suppression, a self-similarity-based nonlocal prior model is employed in the regularized framework, which gives the nonlocal total variation (NLTV) regularizer as [19] where i indicates any of the pixel in the image X : Ω → R , j represents the pixels located in the R sized search window around i and denoted as N i , the weight formula is represented as where N i (•) and N j (•) denote the r sized square similarity patch surrounding pixel i and j, respectively; σ represents the filtering parameter that controls the smoothness.

Local Steering Kernel
Steering kernel regression (SKR) [21] depends on not only the position and intensity, but also the intrinsic local structure of the samples.Therefore, the size and shape of the regression kernel will significantly affect its spread and feature extraction characteristics [31,32].The core of the SKR method is the local steering kernel (LSK) function, which estimates the local structures accurately, even in strong noise.

Let coordinate vector
T represents the position of a certain pixel, and X(x i ) denotes the intensity of the x i pixel.The structural representation capability of LSK mainly relies on the so-called gradient covariance matrix or steering matrix [21].Assume there exists a p pixels involved patch N(x i ) = x 1 , ..., x i , ...x p centered at x i , the structure adaptive steering kernel representation can be modeled as where x j ∈ N(x i ), the symmetric covariance matrix C j is evaluated by the spatial gradient vectors surrounding x j [33].A good choice of C j is vital for estimating the LSK and will expand the kernel weight along the local edges; h(2 ≤ h ≤ 3) is a global smoothing parameter. Let the positions of the M adjacent pixels surrounding x j .Natively, C j can be directly estimated by G T j G j , in which G j is expressed as where Xh (•) and Xv (•), respectively, indicate the first-order gradients in the horizontal and vertical direction.
In order to enhance robustness and promote stability, the proposed method estimates the covariance matrix using a regularized parametric method.Based on the singular value decomposition (SVD) formula of G j , we can obtain the singular values (s 1 , s 2 ) and the singular vectors (u 1 , u 2 ), which are further used to calculate the stable covariance matrix where the amplification factor φ and the constant eps are set to 0.5 and 10 −7 , respectively.According to the intrinsic structure, we can regulate f and g to make the induced kernel be isotropic at the smooth area and uniform along the image contour (g > f ) as where γ is a tuning parameter for controlling the kernel spread and is set to 1, which suppresses the noise, as well as decreases the detail loss; β and ϕ are the eigen values of the structure tensor [34] that reflects the gradient strength along the direction of each eigenvector.

Local and Nonlocal Steering Kernel Weighted Total Variation Model
The local derivative-based models (such as TV and BTV) are sensitive to noise in the homogenous region, while the patch similarity-based models (such as NLTV) are not fit to deal with the noise in the cluttered texture region.In view of this, we incorporate the steering kernel [35]-based structure descriptor into the TV regularization framework and present an innovative regularizer.This local structure based regularizer can smooth out noise while preserving the details, even in very noisy circumstances.The weight function based on the local steering kernel (LSK) can be defined as Symmetry 2019, 11, 329 5 of 16 Furthermore, we consider both the nonlocal similarity and local structure properties and propose a joint local and nonlocal structural weight, which is then normalized in its neighborhood to prevent the nonuniform weight in various patches, and is characterized by where i is a pixel located in the patch that shifts l and m pixels from x i in the horizontal and vertical direction.For x i ∈ Ω, either of the W l,m LSK (i) calculated by (14) or W l,m NLSK (i) calculated by ( 15) can be used to form the weight matrix uniformly represented by W l,m .
In order to compare the spread characterization of different kernels, Figure 1 shows the subjective visual representation of the steering kernel for different local structures (texture, strong edge) in the "House" image on the noisy and noiseless cases.Seeing the weight map of LSK, the shape and orientation of its footprints elongate along the edge to realize edge preservation and noise smoothing.The weight map of non-local kernel (NLK) is strewn according to the nonlocal similarity but neglects the influence of local structure.In contrast, the weight map of non-local steering kernel (NLSK) contains both neighborhood similarity and local spatial support, and assigns large weights to the locally and structurally similar pixels along with the central pixels of nonlocal similar patches.Specifically speaking, the weight map of NLSK spread closer than other kernels in the direction perpendicular to edges, which adaptively reduces the blurriness with respect to the local feature of the image.Moreover, we can easily find that all of the kernels show a rapid decline for spread characterization in the noisy condition and NLSK is still the most precise structure descriptor.
even in very noisy circumstances.The weight function based on the local steering kernel (LSK) can be defined as Furthermore, we consider both the nonlocal similarity and local structure properties and propose a joint local and nonlocal structural weight, which is then normalized in its neighborhood to prevent the nonuniform weight in various patches, and is characterized by , , where is a pixel located in the patch that shifts l and m pixels from i x in the horizontal and vertical direction.In order to compare the spread characterization of different kernels, Figure 1 shows the subjective visual representation of the steering kernel for different local structures (texture, strong edge) in the "House" image on the noisy and noiseless cases.Seeing the weight map of LSK, the shape and orientation of its footprints elongate along the edge to realize edge preservation and noise smoothing.The weight map of non-local kernel (NLK) is strewn according to the nonlocal similarity but neglects the influence of local structure.In contrast, the weight map of non-local steering kernel (NLSK )contains both neighborhood similarity and local spatial support, and assigns large weights to the locally and structurally similar pixels along with the central pixels of nonlocal similar patches.Specifically speaking, the weight map of NLSK spread closer than other kernels in the direction perpendicular to edges, which adaptively reduces the blurriness with respect to the local feature of the image.Moreover, we can easily find that all of the kernels show a rapid decline for spread characterization in the noisy condition and NLSK is still the most precise structure descriptor.Rather than defining the weight by local intensity or nonlocal similarity, we, respectively, introduce the LSK and NLSK prior to weight for the neighboring gradient according to the nonlocal and intrinsic structure of the image itself.On the basis of TV criterion and LSK or NLSK prior, we uniformly define the noise tolerant LSKTV and NLSKTV regularizer as Combining the above presented idea, we propose an innovative structure feature-guided cost function for the denoising problem as ˆˆÂ rgMin 2 where , ˆˆ∇ = − l m l m x y X X S S X .
Considering the equivalent constrained problem to (17), such that get the unconstrained version and solve it by using the split Bregman algorithm [36,37] given by μ λ where 0 λ > is a constant.
Following (18), we solve the equations using the Gauss-Seidel iteration written as ( ) At the boundaries of the domain, one-sided finite differences are used instead of the centered finite differences.We get Rather than defining the weight by local intensity or nonlocal similarity, we, respectively, introduce the LSK and NLSK prior to weight for the neighboring gradient according to the nonlocal and intrinsic structure of the image itself.On the basis of TV criterion and LSK or NLSK prior, we uniformly define the noise tolerant LSKTV and NLSKTV regularizer as Combining the above presented idea, we propose an innovative structure feature-guided cost function for the denoising problem as where ∇ l,m X = X − S l x S m y X. Considering the equivalent constrained problem to (17), such that W l,m • ∇ l,m X = d l,m , we can get the unconstrained version and solve it by using the split Bregman algorithm [36,37] given by where λ > 0 is a constant.Following (18), we solve the equations using the Gauss-Seidel iteration written as Symmetry 2019, 11, 329 7 of 16 At the boundaries of the domain, one-sided finite differences are used instead of the centered finite differences.We get The whole algorithm procedure is presented in Algorithm 1.

Algorithm 1: Proposed image denoising algorithm
Input: noisy observation Y.

Experimental Results and Analysis
In this section, we will contrast the performance of the presented method with the previous variational denoising methods on artificially degraded samples that are generated by adding Gaussian noise with zero mean and standard deviation of 10, 25, and 40 to the 512 × 512 sized standard bitmap (BMP) format test images.In the following experiments, the parameter sensitivity will be firstly discussed for finding a tune strategy to obtain balanced and higher performance.It is worthy to note that the parameters selected according to the advice of original documents are employed to pursue the best performance.In the following experiments, the stopping criteria ε of the proposed local steering kernel total variation (LSKTV) and non-local steering kernel total variation (NLSKTV) algorithm is set to 1 × 10 −3 to guarantee stable convergence.
In order to facilitate quantitative comparison, Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity index (SSIM) [38] are employed to objectively assess the performance of various denoising methods.The PSNR is defined as where H and W indicate the height and width of the image, respectively.In addition, the SSIM index can be calculated by where µ x and µ y stand for the mean value of x and y, σ x and σ y indicate the variance of x and y, σ xy denotes the covariance of x and y.The constants c 1 and c 2 are, respectively, set to 6.5025 and 58.6225 to stabilize the division with the weak denominator.

Parameter Sensitivity Analysis
As for the proposed NLSK weight, appropriate window size is crucial for precisely measuring the local and nonlocal structure, which will directly affect the final restoration effect and subsequent parameter adjustment.In view of this, we select different sizes of similar windows and search windows, and then implement the proposed denoising method upon the same simulated image on three noise strengths.For ease of comparison, we uniformly symbolize the search window size of NLK, LSK, and NLSK weight with R = 2l + 1 = 2m + 1.The PSNR curves of the iterative denoising process for different parameter configurations are shown in Figure 2. Through comparison, we can find that the size of similar window r and search window R should be aptly raised with the increase of noise level to effectively utilize redundant information in the neighborhood for high precision and stable convergence.However, extremely large similar windows and search windows may remove the image details and cause image over-smoothness effect, which instead results in performance degradation.

Parameter Sensitivity Analysis
As for the proposed NLSK weight, appropriate window size is crucial for precisely measuring the local and nonlocal structure, which will directly affect the final restoration effect and subsequent parameter adjustment.In view of this, we select different sizes of similar windows and search windows, and then implement the proposed denoising method upon the same simulated image on three noise strengths.For ease of comparison, we uniformly symbolize the search window size of NLK, LSK, and NLSK weight with . The PSNR curves of the iterative denoising process for different parameter configurations are shown in Figure 2. Through comparison, we can find that the size of similar window r and search window R should be aptly raised with the increase of noise level to effectively utilize redundant information in the neighborhood for high precision and stable convergence.However, extremely large similar windows and search windows may remove the image details and cause image over-smoothness effect, which instead results in performance degradation.The learning rate µ is the key impact factor to control the convergence property of the proposed method [39,40].As can be seen from the Figure 3, the increase of µ promotes the convergence rate, but if µ is too large it will interrupt the convergence process.The reason lies in that a larger µ is beneficial for accelerating the process to reach maximum PSNR, but excessive µ will produce non-convergence issues, which instead depresses the PSNR.As for higher noise level, the learning rate µ should be set larger to ensure the algorithm can effectively remove noise and improve convergence speed.The learning rate μ is the key impact factor to control the convergence property of the proposed method [39,40].As can be seen from the Figure 3, the increase of μ promotes the convergence rate, but if μ is too large it will interrupt the convergence process.The reason lies in that a larger μ is beneficial for accelerating the process to reach maximum PSNR, but excessive μ will produce non-convergence issues, which instead depresses the PSNR.As for higher noise level, the learning rate μ should be set larger to ensure the algorithm can effectively remove noise and improve convergence speed.Considering the smoothing parameter h in the steering kernel, we will further find the optimal selection of h is beneficial to achieve higher restoration precision.As can be seen from Figure 4, h should be raised with the increase of noise strength on the premise of stable convergence.A larger h is helpful to smooth out noise and promote PSNR, but if h is too large it will lead to over-smoothness effect and destroy the intrinsic structure of images, which instead depresses the PSNR.Considering the smoothing parameter h in the steering kernel, we will further find the optimal selection of h is beneficial to achieve higher restoration precision.As can be seen from Figure 4, h should be raised with the increase of noise strength on the premise of stable convergence.A larger h is helpful to smooth out noise and promote PSNR, but if h is too large it will lead to over-smoothness effect and destroy the intrinsic structure of images, which instead depresses the PSNR.

Performance Comparisons
Figure 5 shows the PSNR for the denoised "Zebra" image of TV, BTV, TGV [41], NLTV, and the proposed algorithms on variant noise strength.Obviously, the proposed NLSKTV and LSKTV method achieve an expanded leading advantage in PSNR with the increase of noise strength σ

Performance Comparisons
Figure 5 shows the PSNR for the denoised "Zebra" image of TV, BTV, TGV [41], NLTV, and the proposed algorithms on variant noise strength.Obviously, the proposed NLSKTV and LSKTV method achieve an expanded leading advantage in PSNR with the increase of noise strength σ when compared to TV, BTV, TGV, and NLTV method.As for the highest noise strength σ = 40, NLSKTV can still reach 27.56 dB (the input PSNR is only 16.08 dB at that time), which is 2.56 dB, 1.64 dB, 1.72 dB, 1.29 dB, and 0.09 dB higher than TV, BTV, TGV, NLTV, and LSKTV, respectively.This phenomenon indicates that the proposed NLSKTV and LSKTV deal with the strong noise more effectively.In order to intuitively observe the visual effect, we implement the denoising with different algorithms.We can see clearly from Figure 6 that TV and TGV methods generate over-smoothed details and produce staircase effect in varying degrees.Furthermore, there are partial noise residuals and texture distortions in the results of BTV and NLTV method.Obviously, the noise suppression ability of the proposed LSKTV and NLSKTV are superior to others.It is worth noting that the presented NLSKTV method preserves more details for its special local structure and nonlocal similarity balanced weight.
To quantitatively evaluate the precision of the abovementioned denoising methods, we carry out comparisons of PSNR and SSIM on different test images respectively corrupted by additive Gaussian noise with 1 0, 2 5, 4 0 σ = . The results summarized in Table 1 indicate that the PSNR and SSIM of the proposed LSKTV is better than that of TV, BTV, total generalized variation (TGV), and NLTV in most cases; furthermore, its superiority becomes increasingly significant with the increasing noise strength.Note that NLSKTV constantly achieves the best performance by fusing additional nonlocal self-similarity to describe the image structure on the basis of LSKTV.In summary, the proposed method effectively preserves the image details and removes the noise simultaneously.In order to intuitively observe the visual effect, we implement the denoising with different algorithms.We can see clearly from Figure 6 that TV and TGV methods generate over-smoothed details and produce staircase effect in varying degrees.Furthermore, there are partial noise residuals and texture distortions in the results of BTV and NLTV method.Obviously, the noise suppression ability of the proposed LSKTV and NLSKTV are superior to others.It is worth noting that the presented NLSKTV method preserves more details for its special local structure and nonlocal similarity balanced weight.

PSNR/dB
To quantitatively evaluate the precision of the abovementioned denoising methods, we carry out comparisons of PSNR and SSIM on different test images respectively corrupted by additive Gaussian noise with σ = 10, 25, 40.The results summarized in Table 1 indicate that the PSNR and SSIM of the proposed LSKTV is better than that of TV, BTV, total generalized variation (TGV), and NLTV in most cases; furthermore, its superiority becomes increasingly significant with the increasing noise strength.Note that NLSKTV constantly achieves the best performance by fusing additional nonlocal self-similarity to describe the image structure on the basis of LSKTV.In summary, the proposed method effectively preserves the image details and removes the noise simultaneously.

Image Deblurring Application
In this section, we will validate the deblurring performance of the proposed method with the existing TV, BTV, TGV, and NLTV methods on standard test image "cameraman".Following the blurring degradation model in [42], the test image is blurred by a 9 × 9 Gaussian kernel with standard deviation 5 and 10 and corrupted with Gaussian noise with standard deviation 10.The PSNR and SSIM results of different methods are shown in Table 2.It is apparent that the LSKTV and NLSKTV achieve higher PSNR and SSIM than TV, BTV, TGV, and NLTV.
Figure 7 shows the visual effects of different deblurring methods.It is obvious that other competitive methods, such as TV, BTV, TGV, and NLTV, tend to generate staircase effects in flat regions and zigzag effect on edges, which reduces the structural similarity significantly.In contrast, the proposed LSKTV and NLSKTV preserve sharp edges and recover high frequency details, as well as yield visually pleasant results.In summary, the quantitative and qualitative results indicate that the proposed local and nonlocal steering kernel-weighted total variation regularizer is a good candidate for image deblurring application.
For i x ∈Ω , either of the , ( ) ) can be used to form the weight matrix uniformly represented by , l m W .

Figure 1 .
Figure 1.Comparison of local weight map produced by LSK, NLK, and NLSK in the edge area and the texture area.(a) The original image.(b) The noisy image with Gaussian noise (σ = 25).

Figure 1 .
Figure 1.Comparison of local weight map produced by LSK, NLK, and NLSK in the edge area and the texture area.(a) The original image.(b) The noisy image with Gaussian noise (σ = 25).

Figure 2 .
Figure 2. Comparisons of PSNR for different similar window radius (r) and search window radius (R) at different noise levels.

Figure 2 .
Figure 2. Comparisons of PSNR for different similar window radius (r) and search window radius (R) at different noise levels.

Figure 3 .Figure 3 .
Figure 3. Comparisons of PSNR for different learning rate ( μ ) at different noise levels.

Figure 4 .
Figure 4. Comparisons of PSNR for different smoothing parameters ( h ) at different noise levels.

Figure 4 .
Figure 4. Comparisons of PSNR for different smoothing parameters (h) at different noise levels.

Symmetry 2019 ,
11, x FOR PEER REVIEW 11 of 16when compared to TV, BTV, TGV, and NLTV method.As for the highest noise strength 40 σ = , NLSKTV can still reach 27.56 dB (the input PSNR is only 16.08 dB at that time), which is 2.56 dB, 1.64 dB, 1.72 dB, 1.29 dB, and 0.09 dB higher than TV, BTV, TGV, NLTV, and LSKTV, respectively.This phenomenon indicates that the proposed NLSKTV and LSKTV deal with the strong noise more effectively.

Figure 5 .
Figure 5.Comparison of PSNR for denoising result of the "Zebra" image at different noise strengths.

Figure 5 .
Figure 5.Comparison of PSNR for denoising result of the "Zebra" image at different noise strengths.

Symmetry 2019 , 16 Figure 6 .Figure 6 .
Figure 6.Comparison of visual effects for various denoising methods.Images from top to bottom raw are standard test images, noisy images (σ = 25), results of TV, BTV, TGV, NLTV, LSKTV, and Figure 6.Comparison of visual effects for various denoising methods.Images from top to bottom raw are standard test images, noisy images (σ = 25), results of TV, BTV, TGV, NLTV, LSKTV, and NLSKTV method.The first four columns are the dollar and its partial enlargement and zebra and its partial enlargement, respectively.The last column is pixel test images.

Table 2 .
PSNR and SSIM of the deblurring results for different methods.