 Next Article in Journal
Roadmap on Recent Progress in FINCH Technology
Previous Article in Journal
Editorial for Special Issue “Fine Art Pattern Extraction and Recognition” Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

# Variational Anisotropic Gradient-Domain Image Processing

Department of Computer Science, Norwegian University of Science and Technology (NTNU), 2802 Gjøvik, Norway
J. Imaging 2021, 7(10), 196; https://doi.org/10.3390/jimaging7100196
Received: 27 August 2021 / Revised: 23 September 2021 / Accepted: 26 September 2021 / Published: 29 September 2021

:

## 1. Introduction and Background

In 2002, Fattal et al.  introduced the method of gradient-domain high-dynamic-range compression. The technique consisted of first computing the gradient field $∇ u 0$ of a high-dynamic-range image $u 0 : Ω → C$, where $Ω ⊂ R 2$ is the image domain and $C ⊂ R 3$ is the colour space. The gradient was then rescaled nonlinearly as $G = f ( ∇ u 0 )$. The resulting tensor field $G$, which is no longer necessarily a gradient field, was then reintegrated by solving the Poisson equation
$∇ 2 u = ∇ · G$
to obtain the compressed image. This is often solved by a gradient descent,
$∂ u ∂ t = ∇ 2 u − ∇ · G$
The Poisson equations can be obtained through a variational approach by minimising the functional
$E ( u ) = 1 2 ∫ Ω | | ∇ u − G | | F 2 d Ω = ∫ Ω L ( u , ∇ u ) d Ω$
where $Ω ⊂ R 2$ is the image domain and the index F indicates the Frobenius norm taken over both image and colour coordinates. This is done by solving the corresponding Euler–Lagrange equations,
$∇ · ∂ L ∂ ∇ u − ∂ L ∂ u = 0$
Perez et al.  soon generalised this into the technique called Poisson Image Editing, and showed that it allowed for a broad range of image processing applications such as image inpainting, seamless cloning, texture transfer, feature exchange, insertion of transparent objects through gradient mixing, texture flattening, local illumination correction, and seamless tiling. All this is obtained by various processing of the gradient or gradients of the original images to obtain the tensor field $G$ used in Equation (1).
One significant challenge of Poisson Image Editing is that it tends to produce visual haloing or blurring artefacts in the reconstructed images. The case of $G = 0$ in Equation (1) is often used for image denoising. For greyscale images, in order to stop the diffusion at edges, and thus reduce the blurring, Perona and Malik  introduced an image-dependent, local, nonlinear diffusion method described by the equation
$∂ u ∂ t = ∇ · ( D ( s ) ∇ u )$
where $s = | ∇ u | 2$ describing the image ‘structure’ has been introduced. (Despite the fact that the method is isotropic, but nonlinear and local, the authors termed it ‘anisotropic diffusion’ in the original publication . This has caused considerable confusion in the terminology in the following literature.) They proposed two different diffusion coefficients with different properties,
$D ( s ) = exp − s K 2$
$D ( s ) = 1 1 + s / K 2$
Instead of designing partial differential equations (PDEs) directly, Rudin et al.  used a variational approach like Equation (3) and introduced the concept of total variation. They showed that the minimisation of the functional
$E = ∫ Ω | ∇ u | d Ω$
$∂ u ∂ t = ∇ · ∇ u | ∇ u |$
Comparing with the Perona–Malik diffusion, Equation (5), we see that it can be written in the same form choosing
$D ( s ) = 1 s$
For total variation and Perona–Malik diffusion, the extension to colour images is not that straight forward. Blomgren and Chan  introduced the concept of colour total variation by using a functional as an $ℓ 2$ norm of the functionals in Equation (8) for each of the colour channels resulting in
$E = ∑ i E i 2$
where $E i$ is the total variation for each colour channel according to Equation (8).
Later approaches have been based on the structure tensor by Di Zenzo  and Bigun and Granlund , with components
$S = ∇ u · ∇ u$
where the dot product is taken over the colour coordinates. Sapiro and Ringach  proposed to use the eigenvalues
$λ ± = 1 2 S 11 + S 22 ± ( S 11 − S 22 ) 2 + 4 S 12 2$
and the corresponding eigenvectors $θ ±$ of the structure tensor as a basis for constructing the diffusion equations. In terms of these eigenvalues, an alternative to the colour total variation by Blomgren and Chan  can be obtained by the functional
$E = ∫ Ω λ + + λ − d Ω = ∫ Ω s d Ω$
For greyscale images, where $λ + = | ∇ u | 2$ and $λ − = 0$, this reduces to total variation. For colour images, the corresponding Euler–Lagrange equations become
$∂ u ∂ t = ∇ · ∇ u | | ∇ u | | F$
which again is on the form of Equation (5) with $D ( s ) = 1 / s = 1 / λ + + λ − = 1 / S 11 + S 22 = 1 / | | ∇ u | | F$. Notice the coupling between the colour channels introduced by the sum in the denominator of Equation (15).
Tschumperlé and Deriche  extended this approach to a anisotropic diffusion by introducing the general Lagrangian density $ψ ( λ + , λ − )$ in the functional
$E = ∫ Ω ψ ( λ + , λ − ) d Ω$
The corresponding Euler–Lagrange equations are
$∂ u ∂ t = ∇ · ( D · ∇ u )$
where $D$ is the diffusion tensor
$D = 2 ∂ ψ ∂ λ + θ + θ + T + ∂ ψ ∂ λ − θ − θ − T$
where $θ ±$ are the eigenvectors of the structure sensor $S$. It should be noted that this actually encompasses all previously presented diffusion methods as follows:
$ψ ( λ + , λ − ) = λ + + λ −$
gives the solution of Sapiro and Ringach  for colour images and total variation of Rudin et al.  for greyscale images,
$ψ ( λ + , λ − ) = − K 2 exp ( − s / K 2 ) and$
$ψ ( λ + , λ − ) = K 2 ln ( 1 + s / K 2 )$
give the two equations of Perona and Malik , and
$ψ ( λ + , λ − ) = s / 2$
gives the classical linear diffusion, Equation (2).
Choosing
$ψ ( λ + , λ − ) = ϕ ( λ + + λ − ) = ϕ ( s )$
in general leads to isotropic equations, since then $∂ ψ / ∂ λ + = ∂ ψ / ∂ λ − = ϕ ′ ( s )$ and the diffusion tensor in Equation (18) reduces to the scalar diffusion coefficient $D = 2 ϕ ′ ( s )$. With
$ψ ( λ + , λ − ) = ϕ + ( λ + ) + ϕ − ( λ − ) ,$
the diffusion in the mutually orthogonal directions of maximal and minimal change can be controlled independently, like used by, e.g., Farup .
For other application of Poission image editing, i.e., applications of Equation (2) where $G ≠ 0$, extensions to anisotropic and edge-preserving methods have been obtained by ad hoc modifications of Equation (2). This has been done for e.g., colour gamut mapping , colour image demosaicing , colour-to-greyscale conversion , and colour image Daltonisation . Common to these is that they solve an equation on the form
$∂ u ∂ t = ∇ · D · ( ∇ u − G )$
where $D$ is a diffusion tensor constructed from the structure tensor like, e.g., Equation (18).
However, a unifying variational formulation of anisotropic gradient-domain image processing has not yet been given. In this paper, we provide this by introducing the difference structure tensor based on the difference of the original image gradient $∇ u$ and the tensor field $G$ and follow the process of Tschumperlé and Deriche . The resulting PDE is similar, but not identical, to the ones obtained by the ad hoc modification of Poisson Image editing, Equation (25). For the applications to linear and nonlinear local contrast enhancement and colour image Daltonisation, we show that the two approaches gives very similar results.

## 2. Variational Anisotropic Gradient-Domain Formulation

The variational formulation of anisotropic diffusion by Tschumperlé and Deriche  was based on the structure tensor $S$, Equation (18). In analogy, we introduce the difference structure tensor,
$S ′ = ( ∇ u − G ) · ( ∇ u − G )$
with eigenvalues
$λ ± ′ = 1 2 S 11 ′ + S 22 ′ ± ( S 11 ′ − S 22 ′ ) 2 + 4 S 12 ′ 2$
and corresponding eigenvectors $θ ± ′$.
In analogy with Tschumperlé and Deriche , we define the functional in terms of the eigenvalues of this difference structure tensor,
$E = ∫ Ω ψ ( λ + ′ , λ − ′ ) d Ω$
The corresponding Euler–Lagrange equations are
$∇ · ∂ ψ ∂ ∇ u − ∂ ψ ∂ u = 0$
It is clear that $∂ ψ / ∂ u = 0$. To derive the explicit form of the Euler–Lagrange equations, we will need to look at the individual components. Using Greek indices for the colour coordinates, Latin indices for the spatial coordinates, letting a comma denote partial differentiation with respect to the following coordinates, and using Einstein’s convention of summing over repeated indices, the expression within the parenthesis in the first term can be written
$∂ ψ ∂ u , i ρ = ∂ ψ ∂ λ p ′ ∂ λ p ′ ∂ S k l ′ ∂ S k l ′ ∂ u , i ρ$
where the index p is used for summing over the two eigenvalues of the difference structure tensor. The first factor, $∂ ψ / ∂ λ p ′$, can be computed directly since the Lagrangian $ψ$ is defined explicitly in terms of the eigenvalues of the difference structure tensor, and will thus depend on the design of the Lagrangian density.
The second factor can be computed implicitly following the method of Tschumperlé and Deriche  as follows:
$δ i k δ j l = ∂ S k l ′ ∂ S i j ′ = ∂ λ p ′ ∂ S i j ′ θ p k ′ θ p l ′ + λ p ′ ∂ θ p k ′ ∂ S i j ′ θ p l ′ + λ p ′ θ p k ′ ∂ θ p l ′ ∂ S i j ′$
Multiplying with $θ m k ′$ and $θ m l ′$, summing over the k and l indexes, using that $θ m k ′ θ p k ′ = δ p m$, and $θ p k ′ ( ∂ θ p k ′ / ∂ S i j ′ ) = 0$, the latter due to the orthonormality of the $θ ′$s gives
$∂ λ p ∂ S k l ′ = θ p k ′ θ p l ′$
(no sum).
The last term, $∂ S k l ′ / ∂ u , i ρ$ deviates from the derivation of Tschumperlé and Deriche due to the definition of the difference structure tensor, Equation (26),
$∂ S k l ′ ∂ u , i ρ = δ i k ( u , l ρ − v l ρ ) + δ i l ( u , k ρ − v k ρ )$
Inserted into Equation (30) and exploiting the symmetry of Equation (32) gives
$∂ ψ ∂ u , k ρ = 2 ∂ ψ ∂ λ p ′ θ p k ′ θ p l ′ ( u , k ρ − v k ρ ) = D k l ′ ( u , k ρ − v k ρ )$
where the diffusion tensor
$D k l ′ = 2 ∂ ψ ∂ λ p ′ θ p k ′ θ p l ′$
has been defined.
Inserting this into Equation (29) and solving by gradient descent gives the variational anisotropic gradient-domain image processing PDE (switching back to the regular vector notation):
$∂ u ∂ t = ∇ · D ′ · ( ∇ u − G ) , where D ′ = 2 ∂ ψ ∂ λ + ′ θ + ′ θ + ′ T + ∂ ψ ∂ λ − ′ θ − ′ θ − ′ T$
The form of this equation is similar to the one based on the ad hoc approach, Equation (25). The only difference is that, in this case, the diffusion tensor is computed from the difference structure tensor, and not the common structure tensor.

## 3. Results and Discussion

In order to demonstrate the usefulness of the proposed method, we apply it to three example problems: linear local contrast enhancement, nonlinear local contrast enhancement, and colour image Daltonisation.

#### 3.1. Implementation

The algorithm represented by Equation (36) is implemented using the finite difference method (FDM). For the time derivative, the explicit (forward) Euler method is used. For the spatial derivatives, forward differences are used for the $∇ u$ and $G$ terms, and backward difference is used for the divergence in order to balance the overall resulting numerical scheme. The resulting code is available online. https://github.com/ifarup/variational-anisotropic-gradient-domain (accessed on 28 September 2021).
A selection of colourful and detailed test images was downloaded from Pixnio. https://pixnio.com (accessed on 28 September 2021). The images are shown in Figure 1. All images are available under the CC0 licence, https://creativecommons.org/share-your-work/public-domain/cc0/ (accessed on 28 September 2021).

#### 3.2. The Diffusion Tensor

The purpose of the diffusion tensor is to direct the diffusion of the image information. The idea is that the diffusion should be avoided across the edges of the image, but allowed to some degree along the edges. The expression for the actual diffusion tensor is given in Equation (36). A choice has to be made for the function $ψ$. Here, we will use the Perona–Malik motivated functional
$ψ ( λ + ′ , λ − ′ ) = K 2 ln ( 1 + λ + ′ / K 2 ) + K 2 ln ( 1 + λ − ′ / K 2 )$
The resulting diffusion coefficients along and across the edges, represented by $2 ∂ ψ ′ / ∂ λ + ′$ and $2 ∂ ψ ′ / ∂ λ − ′$, respectively, are shown for one of the test images in Figure 2, where it is easily seen how the edges influence the directional diffusion.

#### 3.3. Linear Local Contrast Enhancement

Local contrast enhancement in the gradient domain is a technique that is known to be particularly prone to haloing problems. First, we follow a simple approach and define the contrast enhancement as a scalar multiplication of the original image gradient
$G = a ∇ u$
where $a > 1$.
We compare this with the standard Poisson method, Equation (2), and the ad hoc addition with the diffusion tensor derived from the structure tensor instead of the difference structure tensor, Equation (18). We use $a = 2$ in order to obtain a quite extreme contrast enhancement, and use with $K = 10 − 3$ for the variational approach based on the difference structure tensor, and $K = 3 × 10 − 4$ for the ad hoc solution based on the standard structure tensor. Different constants are needed since the scaling of the structure tensor will be different in the two cases. Example results are shown in Figure 3. We can see that, with this choice of K, the results of the variational and the ad hoc approaches are indistinguishable, whereas the Poisson solutions exhibit severe haloing artefacts, as expected.

#### 3.4. Nonlinear Local Contrast Enhancement

A somewhat more sophisticated contrast enhancement can be achieved by using the nonlinear gamma compression in the gradient domain,
$G = sign ( ∇ u ) | ∇ u | γ with 0 < γ < 1$
computed element-wise. Solving as for the linear case with the same choice of parameters as above gives the results shown in Figure 4. We can observe the indistinguishable behaviour between the variational approach and the ad hoc solution, and that the haloing problem of the Poisson solution is solved.

#### 3.5. Colour Image Daltonisation

Colour image Daltonisation denotes the recolouring of colour images to increase detail visibility for colour-deficient observers. It is a problem that is known to be particularly prone to the loss of finer image details and textures. It has previously been shown that this process can be performed in the gradient-domain  by constructing a gradient field based on the original image as
$G = ∇ u 0 + ( ∇ u 0 · e d ) e c$
where $u 0$ is the original colour image, $e d$ is the first principal component of the difference between the original image and a colour-vision-deficiency simulation the same image, and $e c$ is a unit vector of maximum chromatic visibility for the colour-vision-deficient observer (in practice, a vector orthogonal to both the lightness axis and $e d$). Figure 5 shows the resulting daltonised images (left) and the corresponding colour-vision-deficient simulations (right). We observe that the variational approach solves the same problem of lost detail visibility as the ad hoc anisotropic solution, and again gives results that are indistinguishable.

## 4. Conclusions

In this paper, a variational formulation of anisotropic gradient-domain image processing has been introduced. It generalises previous formulations of anisotropic processing and also introduces the difference structure tensor. The resulting PDE deviates somewhat from the one usually found when anisotropic diffusion is added in and ad hoc manner to do gradient-domain image processing. The difference is that the diffusion tensor is based on the difference structure tensor rather than the conventional structure tensor. The example applications of linear and nonlinear local contrast enhancement and colour image Daltonisation illustrate that the behaviour is very similar to what is obtained using the conventional approach. This indicates that the proposed variational formulation is well suited for rigorous derivations of anisotropic gradient-domain image processing for a broad range of applications.

## Funding

This research was funded by the Research Council of Norway over the project ‘Individualised Color Vision-based Image Optimisation’, Grant No. 287209.

Not applicable.

Not applicable.

## Data Availability Statement

The code is available at https://github.com/ifarup/variational-aniso-tropic-gradient-domain (accessed on 28 September 2021).

## Conflicts of Interest

The authors declare no conflict of interest.

## References

1. Fattal, R.; Lischinski, D.; Werman, M. Gradient domain high dynamic range compression. ACM Trans. Graph. 2002, 21, 249–256. [Google Scholar] [CrossRef][Green Version]
2. Pérez, P.; Gangnet, M.; Blake, A. Poisson image editing. ACM Trans. Graph. 2003, 22, 313–318. [Google Scholar] [CrossRef]
3. Perona, P.; Malik, J. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 1990, 12, 629–639. [Google Scholar] [CrossRef][Green Version]
4. Rudin, L.I.; Osher, S.; Fatemi, E. Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 1992, 60, 259–268. [Google Scholar] [CrossRef]
5. Blomgren, P.; Chan, T.F. Color TV: Total variation methods for restoration of vector-valued images. IEEE T. Image Process. 1998, 7, 304–309. [Google Scholar] [CrossRef] [PubMed][Green Version]
6. Di Zenzo, S. A note on the gradient of a multi-image. Comput. Vis. Graph. Image Process. 1986, 33, 116–125. [Google Scholar] [CrossRef]
7. Bigun, J.; Granlund, G. Optimal orientation detection of linear symmetry. In Proceedings of the First International Conference on Computer Vision, ICCV, London, UK, 8–11 June 1987; pp. 433–438. [Google Scholar]
8. Sapiro, G.; Ringach, D.L. Anisotropic Diffusion of Multivalued Images with Applications to Color Filtering. IEEE Trans. Image Process. 1996, 5, 1582–1586. [Google Scholar] [CrossRef] [PubMed]
9. Tschumperlé, D.; Deriche, R. Vector-valued image regularization with PDEs: A common framework for different applications. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27, 506–517. [Google Scholar] [CrossRef] [PubMed][Green Version]
10. Farup, I. Individualised Halo-Free Gradient-Domain Colour Image Daltonisation. J. Imaging 2020, 6, 116. [Google Scholar] [CrossRef] [PubMed]
11. Alsam, A.; Farup, I. Spatial Colour Gamut Mapping by Means of Anisotropic Diffusion. In Proceedings of the Computational Colour Imaging Workshop (CCIW), Milan, Italy, 20–21 April 2011; Lecture Notes in Computer Science. Springer: Berlin, Germany, 2011; Volume 6626, pp. 113–124. [Google Scholar]
12. Thomas, J.B.; Farup, I. Demosaicing of Periodic and Random Colour Filter Arrays by Linear Anisotropic Diffusion. J. Imaging Sci. Technol. 2018, 62, 050401-1–050401-8. [Google Scholar] [CrossRef][Green Version]
13. Farup, I.; Pedersen, M.; Alsam, A. Colour-to-Greyscale Image Conversion by Linear Anisotropic Diffusion of Perceptual Colour Metrics. In Proceedings of the 2018 Colour and Visual Computing Symposium (CVCS), Gjovik, Norway, 19–20 September 2018. [Google Scholar]
Figure 1. Test images from Pixnio. All images are available under the CC0 licence.
Figure 1. Test images from Pixnio. All images are available under the CC0 licence.
Figure 2. The diffusion coefficients $2 ∂ ψ ′ / ∂ λ + ′$ (left) and $2 ∂ ψ ′ / ∂ λ − ′$ (right) for one of the test images.
Figure 2. The diffusion coefficients $2 ∂ ψ ′ / ∂ λ + ′$ (left) and $2 ∂ ψ ′ / ∂ λ − ′$ (right) for one of the test images.
Figure 3. Example results for linear contrast enhancement with $a = 2$. Original images in the first column, Poisson solution in the second, ad hoc anisotropic diffusion with $K = 10 − 3$ in the third, and the proposed variational with $K = 3 × 10 − 4$ in the fourth. All images are available under the CC0 licence.
Figure 3. Example results for linear contrast enhancement with $a = 2$. Original images in the first column, Poisson solution in the second, ad hoc anisotropic diffusion with $K = 10 − 3$ in the third, and the proposed variational with $K = 3 × 10 − 4$ in the fourth. All images are available under the CC0 licence.
Figure 4. Example results for gamma contrast enhancement with $γ = 0.7$. Original images in the first column, Poisson solution in the second, ad hoc anisotropic diffusion with $K = 10 − 3$ in the third, and the proposed variational with $K = 0.3 × 10 − 4$ in the fourth. All images are available under the CC0 licence.
Figure 4. Example results for gamma contrast enhancement with $γ = 0.7$. Original images in the first column, Poisson solution in the second, ad hoc anisotropic diffusion with $K = 10 − 3$ in the third, and the proposed variational with $K = 0.3 × 10 − 4$ in the fourth. All images are available under the CC0 licence.
Figure 5. Example of colour image Daltonisation by the proposed method. Colour images in the left column, and corresponding colour–vision–deficiency simulations in the right. Top to bottom: original image, simple global Daltonisation, ad hoc anisotropic gradient domain Daltonisation, and the proposed variational gradient-domain solution.
Figure 5. Example of colour image Daltonisation by the proposed method. Colour images in the left column, and corresponding colour–vision–deficiency simulations in the right. Top to bottom: original image, simple global Daltonisation, ad hoc anisotropic gradient domain Daltonisation, and the proposed variational gradient-domain solution.
 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Share and Cite

MDPI and ACS Style

Farup, I. Variational Anisotropic Gradient-Domain Image Processing. J. Imaging 2021, 7, 196. https://doi.org/10.3390/jimaging7100196

AMA Style

Farup I. Variational Anisotropic Gradient-Domain Image Processing. Journal of Imaging. 2021; 7(10):196. https://doi.org/10.3390/jimaging7100196

Chicago/Turabian Style

Farup, Ivar. 2021. "Variational Anisotropic Gradient-Domain Image Processing" Journal of Imaging 7, no. 10: 196. https://doi.org/10.3390/jimaging7100196

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.