Morphological PDEs on Graphs for Image Processing on Surfaces and Point Clouds

Abstract

Partial Differential Equations (PDEs)-based morphology offers a wide range of continuous operators to address various image processing problems. Most of these operators are formulated as Hamilton–Jacobi equations or curve evolution level set and morphological flows. In our previous works, we have proposed a simple method to solve PDEs on point clouds using the framework of PdEs (Partial difference Equations) on graphs. In this paper, we propose to apply a large class of morphological-based operators on graphs for processing raw 3D point clouds and extend their applications for the processing of colored point clouds of geo-informatics 3D data. Through illustrations, we show that this simple framework can be used in the resolution of many applications for geo-informatics purposes.

1. Introduction

Partial Differential Equations (PDEs) are widely used for modeling and solving many inverse problems in image processing and computer vision. PDEs of the Laplacian or p-Laplacian type have been used successfully in many applications, such as image denoising, restoration, segmentation and inpainting; see [1,2,3,4] and the references therein. PDEs of the Hamilton–Jacobi type play a central role in continuous mathematical morphology. They have been used to model many continuous morphological operators, such as dilation, erosion, leveling or watershed transforms [5,6,7,8,9]. However, many of these models focus on images defined on the Euclidean domain where discretization, based on finite differences, finite elements, finite volumes, etc, is well investigated and traditionally used.

On the other hand, with the development of 3D scanning technology, it is now easy to generate 3D models from real objects, together with standard images painted on them. LIDAR (Light Detection and Ranging) scanners allow one to generate point clouds by estimating the distance between sensors and ground objects and have found interest in geoscience; see [10,11,12,13]. These scanners generally provide raw data in the form of (noisy) unorganized point clouds representing surface samples with images on point clouds. With the increasing popularity and the very broad applications in geoscience, art, medicine and security of manufacturing, there is a growing interest to transpose PDEs-based tools for signal/image processing to image processing on point clouds. The discretization of differential operators becomes more difficult compared to the previously mentioned approaches.

There are a variety of methods for solving PDEs on surfaces numerically [14,15,16]. However, these methods do not process directly the raw 3D point clouds and require point clouds to be in a suitable representation. One can quote several methods that can be classified into four categories according to the representation:

• Methods using explicit representations of surfaces represented by a parametric function: This corresponds to attaching a two-dimensional parametric domain $(u,v)$ to the 3D object. By relying on a specific parametrization of the given surface, differential operators can be defined and computed analytically [17,18]. However, the computation of the parametrization is a difficult task for arbitrary given surfaces, and topological changes can hardly be handled.
• Methods using implicit representations of surfaces represented by a zero level-set function of a signed distance function in Euclidean domains. The differential operators are then approximated by combining the Euclidean differential operators with a projection along the normal direction [1,19]. For instance, in [20], the coordinates of the closest point for each point of the surface is used, and fast algorithms can be obtained in Euclidean domains. Implicit representations can deal easily with topological changes, but all of the data have to be extended to the definition domain of the implicit function.
• Methods using intrinsic geometry to study variational problems directly on the surface represented as a triangular mesh. Lai and Chan have recently proposed [14] a framework for intrinsic image processing on surfaces. They approximate surface differential operators, such as surface gradient and divergence, by specific intrinsic differential geometry definitions. Intrinsic methods do not need any pre-processing, but they require a specific discretization scheme on triangles.
• Methods solving PDEs directly on point clouds [15,16] by using intermediate representations to approximate differential operators on point clouds. The authors in [16] use a local triangulation that requires pre-processing. This pre-processing is needed to estimate differential operators on triangular meshes. This method can then further be categorized as an intrinsic method. The authors of [15] compute a local approximation of the manifold using moving least squares from the k-nearest neighbors. From this local coordinate system, a local metric tensor is computed at each point such that the differentiation on the manifold is simplified. This method can therefore be categorized as an explicit method.

In our previous works [21,22], we have proposed a different approach that can cope with both 3D meshes and point clouds of arbitrary connectivity. To achieve this goal, we proposed the Partial difference Equation (PdE) framework. This method provides a new approach that can be used to overcome all of the difficulties and drawbacks of the methods mentioned above. Let us recall briefly the method. A local or non-local graph is first constructed from the geometry of surfaces or point clouds. Then, the transcription of PDEs on the graph is done and resolved by using the framework of PdEs. Our approach allows one to transpose and extend many processes in classical image processing to surfaces and point clouds. Contrary to other approaches, our method provides the following advantages: (1) no parametrization, nor pre-processing, nor assumption is required; and (2) local and non-local processing are unified. Since 3D point clouds and triangular meshes can have very different topologies, we proposed to rely on graph-based methods that directly work in any discrete domain.

In this paper, we adopt the PdE framework, and we focus on some PDEs-based continuous morphological operators in the Euclidean domain: dilation/erosion, mean curvature flows and the Eikonal equation. One strong benefit of our approach is that it enables the processing of both geometry and color associated with raw 3D point clouds. Our motivation is to extend their applications for the processing of 3D surfaces and 3D point clouds. We apply this approach for geo-informatics purposes on examples, such as restoration, denoising, inpainting, object extraction or estimation of the minimal path.

Mathematical Morphology (MM) is a popular nonlinear approach for image processing that has found numerous applications including shape and texture analysis, biomedical image processing, document recognition or multiresolution techniques [23,24]. MM offers a wide range of operators to address various image processing problems. These operators can be defined in terms of algebraic (discrete) tools or as PDEs. We now briefly review some of these operators formulated as Hamilton–Jacobi equations, curve evolution level sets and morphological flows. For a detailed review and numerical algorithms, see [2,25].

Hamilton–Jacobi equation for morphological dilation/erosion: Most of the morphological continuous operators can be modeled by the Hamilton–Jacobi equation. Let us assume an image as a Lipschitz continuous function $f:\mathrm{\Omega }\subset {\mathbb{R}}^n\to \mathbb{R}$. Consider the general following initial-value PDEs:

$$\{\begin{array}{ll}\frac{\partial f}{\partial t}(x,t)& =\pm H(x,\mathrm{\nabla }f(x,t)),\mathrm{in}\mathrm{\Omega }\times (0,+\infty ),\\ f(x,0)& =g(x),\mathrm{in}\mathrm{\Omega }\end{array}$$

(1)

A solution of this Hamiltonian is called a viscosity solution [26]. For a particular Hamiltonian H, a kind of canonical morphological PDE is formulated:

$$\{\begin{array}{ll}\frac{\partial f}{\partial t}(x,t)& =\pm \frac{1}{2}{||\mathrm{\nabla }f(x,t)||}^2,x\in {\mathbb{R}}^n\\ f(x,0)& =g(x),x\in {\mathbb{R}}^n\end{array}$$

(2)

The corresponding viscosity solutions correspond to a dilation ${\delta }_{B}f(x)$ and an erosion ${ϵ}_{B}f(x)$ of the function $f(x)$ defined as:

$$\begin{array}{cc}\hfill {\delta }_{B}f(x)& =\underset{y\in {\mathbb{R}}^n}{sup}\{g(y)+B(y-x)\},\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {ϵ}_{B}f(x)& =\underset{y\in {\mathbb{R}}^n}{inf}\{g(y)-B(y-x)\},\hfill \end{array}$$

(4)

using as the structuring function $B(x)$ the so-called multiscale quadratic (or parabolic) structuring function:

$$p_t(x)=-\frac{{||x||}^2}{2t}.$$

(5)

Due to its properties of being a semigroup (dimension separability and invariance to transform domain [27]), the structuring function $p_t(x)$ can be considered as the canonical one in morphology, playing a similar role as the Gaussian kernel for linear filtering. Other particular forms of the Hamilton–Jacobi model Equation (1) cover the flat morphology by disks [7] (i.e., $f_t=\pm ||\mathrm{\nabla }f||$), as well as operators with more general P-power concave structuring functions (i.e., $f_t=\pm {||\mathrm{\nabla }f||}^p$). For the application of the latter model to adaptive morphology, see [25] and the references therein.

These versions of dilation and erosion are also the basis of other continuous morphological operators, such as leveling or continuous watershed with the Eikonal equation [28]. To be applicable to images, PDEs are discretized, and specific numerical schemes are used: for instance, see [29,30,31] and, more recently, a variant of the flux corrected transport technique [32,33] used for tensor images and matrix morphological processing [34]. Other morphological PDEs, based on curve evolution, the level set or the Eikonal equation, are used for image segmentation, generalized distance computation or shape analysis.

Curve evolution, level sets and morphological flows: The level set formulation to describe the curve evolution has been introduced by Osher and Sethian [2]. It provides well-known advantages, such as dealing with self-intersections or topological changes, and can be easily extended to ${\mathbb{R}}^d$ with $d\ge 1$. Given a parametrized curve ${\mathrm{\Gamma }}_0:[0,1]\to \mathrm{\Omega }$, evolving on a domain Ω due to the effect of a velocity $\mathcal{F}$:

$$\frac{\partial \mathrm{\Gamma }(x,t)}{\partial t}=\mathcal{F}·\mathcal{N},$$

(6)

with $\mathcal{F}:\mathrm{\Omega }\to \mathbb{R}$ a function on Ω that depends on the curvature and the normal $\mathcal{N}$. The level set method aims to find a function $\varphi (x,y)$, such that at each time t, the evolving curve ${\mathrm{\Gamma }}_t$ can be provided by the zero-level set of ϕ. In other words ${\mathrm{\Gamma }}_t=\{x|\varphi (x,y)=0\}$ and the curve evolution can be processed by solving:

$$\{\begin{array}{ll}\frac{\partial \varphi (x,t)}{\partial t}& =\mathcal{F}|\mathrm{\nabla }\varphi |,\\ \varphi (x,0)& ={\varphi }_0(x)\end{array}$$

(7)

where ${\varphi }_0(x)$ is the initial embedding of Γ or an initial image. The velocity function $\mathcal{F}$ can depend on the gradient or mean curvature. For $\mathcal{F}=\pm 1$, we get the dilation and erosion:

$$\frac{\partial \varphi }{\partial t}=\pm |\mathrm{\nabla }\varphi |,$$

(8)

For $\mathcal{F}=div\left(\frac{\mathrm{\nabla }\varphi }{|\mathrm{\nabla }\varphi |}\right)$, we get the mean curvature flow:

$$\frac{\partial \varphi }{\partial t}=div\left(\frac{\mathrm{\nabla }\varphi }{|\mathrm{\nabla }\varphi |}\right)|\mathrm{\nabla }\varphi |.$$

(9)

which found application in image denoising, segmentation, reconstruction and active contour models. In this paper, we consider the following general level set equation on surfaces or point clouds:

$$\frac{\partial \varphi (x,t)}{\partial t}=\mathcal{H}\left(t,x,{\varphi }_S,\mathrm{\nabla }{\varphi }_S,div\left(\frac{\mathrm{\nabla }{\varphi }_S}{|\mathrm{\nabla }{\varphi }_S|}\right)\right).$$

(10)

To solve this PDE on a surface S, we replace the differential operators by their discrete analogous provided by the PdE framework on an adapted graph $G(V,E,w)$. Thus, the transcription of the PDE Equation (10) on the graph is defined as:

$$\{\begin{array}{l}\frac{\partial \varphi (u,t)}{\partial t}=\widehat{\mathcal{H}}\left(t,u,\varphi ,{\mathrm{\nabla }}_w^{+}\varphi ,{\mathrm{\nabla }}_w^{-}\varphi ,{\mathcal{K}}_w\varphi \right)\\ \varphi (u,0)={\varphi }_0(u)\end{array}$$

(11)

where $\widehat{\mathcal{H}}$ is an approximation of the $\mathcal{H}$ function, and ${\mathrm{\nabla }}_w^{+},{\mathrm{\nabla }}_w^{-},{\mathcal{K}}_w$ are the external gradient, the internal gradient and the mean curvature defined on a weighted graph. Respectively, these operators will be defined in the following section.

The rest of this paper is organized as follows. In Section 2, we present partial difference operators on graphs. We present morphological operators on graphs in Section 3. In Section 4, we present the construction of a patch-based weighted graph from a point cloud and examples of filtering, inpainting and segmentation of images on point clouds. The last section concludes.

2. Partial Difference Operators on Graphs

In this section, we recall definitions and operators on graphs. This constitutes the basis of the framework of PdEs [35] that enables transposing PDEs on graphs. All of these definitions are borrowed from [6,9,36].

2.1. Notations and Preliminaries

A weighted graph $G=(V,E,w)$ consists of a finite set $V=\{v_1,\dots ,v_N\}$ of N vertices and a finite set $E\subset V\times V$ of weighted edges. We assume G to be undirected, with no self-loops and no multiple edges. Let $(u,v)$ be the edge of E that connects two vertices u and v of V. Its weight, denoted by $w(u,v)$, represents the similarity between its vertices. Similarities are usually computed by using a positive symmetric function $w:V\times V\to \mathrm{I}{\mathrm{R}}^{+}$ satisfying $w(u,v)=0$ if $(u,v)\notin E$. The notation $u\sim v$ is also used to denote two adjacent vertices. The degree of a vertex u is defined as ${\delta }_w(u)={\sum }_{v\sim u}w(u,v)$. Let $H(V)$ be the Hilbert space of real-valued functions defined on the vertices of a graph. A function $f:V\to \mathrm{I}\mathrm{R}$ of $H(V)$ assigns a real value $f(u)$ to each vertex $u\in V$. The function space $H(V)$ is endowed with the usual inner product ${\langle f,h\rangle }_{H(V)}={\sum }_{v\in V}f(v)h(v)$, where $f,h:V\to \mathbb{R}$.

2.2. Difference Operators on Weighted Graphs

Let $G=(V,E,w)$ be a weighted graph, $f:V\to \mathrm{I}\mathrm{R}$ be a function of $H(V)$ and $w:V\times V\to {\mathbb{R}}^{+}$ a weight function defined as a similarity measure between two vertices.

Definition 1.

The directional derivative (or edge derivative) of f, at a vertex $u\in V$, along an edge $e=(u,v)$, is defined as:

$${\partial }_{v}f(u)=\sqrt{w(u,v)}(f(v)-f(u)).$$

(12)

The external and internal morphological directional partial derivative operators are respectively defined as [9]:

$$\begin{array}{ccc}\hfill {\partial }_v^{+}f(u)& =& {\left({\partial }_{v}f(u)\right)}^{+},\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill {\partial }_v^{-}f(u)& =& {\left({\partial }_{v}f(u)\right)}^{-}.\hfill \end{array}$$

(14)

where ${(x)}^{+}=max(x,0)$ and ${(x)}^{-}=-min(x,0)$.

Definition 2.

Discrete upwind non-local weighted gradients are defined as:

$$({\mathrm{\nabla }}_w^{\pm }f)(u)={\left(({\partial }_v^{\pm }f)(u)\right)}_{v\in V}^T,$$

(15)

where w corresponds to the weight function defined on graphs.

The ${\mathcal{L}}_p$ and ${\mathcal{L}}_{\infty }$ norms of these gradients are defined by:

$$\begin{array}{cc}\hfill ||({\mathrm{\nabla }}_w^{\pm }f)(u){||}_p& ={\left[\sum _{v\sim u}w{(u,v)}^{\frac{p}{2}}{\left[{(f(v)-f(u))}^{\pm }\right]}^p\right]}^{\frac{1}{p}},\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill ||({\mathrm{\nabla }}_w^{\pm }f)(u){||}_{\infty }& =\underset{v\sim u}{max}\left({w(u,v)|(f(v)-f(u))}^{\pm }|\right).\hfill \end{array}$$

(17)

Definition 3.

The normalized two-Laplacian on the graph is defined as:

$$({\Delta }_{w,2}^{N}f)(u)=\frac{{\sum }_{v\sim u}w(u,v)f(v)}{{\delta }_w(u)}-f(u)$$

(18)

Definition 4.

The ∞-Laplacian on the graph is defined as:

$$({\Delta }_{w,\infty }f)(u)=\frac{1}{2}\left[||({\mathrm{\nabla }}_w^{+}f)(u){||}_{\infty }-||({\mathrm{\nabla }}_w^{-}f)(u){||}_{\infty }||\right].$$

(19)

Definition 5.

The anisotropic p-Laplacian on the graph of a function $f:V\to \mathbb{R}$ for $1\le p<\infty$ is defined as:

$${\Delta }_{w,p}f(u)=\sum _{v\in V}w{(u,v)}^{\frac{p}{2}}{|f(v)-f(u)|}^{p-2}(f(v)-f(u)).$$

(20)

Definition 6.

The mean curvature ${\mathcal{K}}_w$ of a function f at $u\in V$ on a graph is defined as:

$${\mathcal{K}}_w(u,f)=\frac{{\sum }_{f(v)-f(u)\ge 0}w(u,v)-{\sum }_{f(v)-f(u)<0}w(u,v)}{{\delta }_w(u)}.$$

(21)

This corresponds to:

$${\mathcal{K}}_w(u,f)=\frac{{\sum }_{u\in V}w(u,v)sign(f(v)-f(u))}{{\delta }_w(u)},$$

(22)

with $sign(r)=\left\{\begin{array}{cc}1\hfill & ifr\ge 0\hfill \\ -1\hfill & otherwise.\hfill \end{array}\right.$

3. Morphological Operators on Graphs

Based on the discrete gradient on weighted graphs, we now present a class of discrete equations that mimic PDEs-based definitions of erosion, dilation, ∞-Laplacian and mean curvature flows, the time level-set and the Eikonal equation.

3.1. Dilation and Erosion on Graphs for Filtering

The dilation and erosion of an initial function $f^0:V\to \mathbb{R}$ is defined by [6]:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \frac{\partial f}{\partial t}(u,t)& =+||({\mathrm{\nabla }}_w^{+}f)(u){||}_p\hfill \\ \hfill \frac{\partial f}{\partial t}(u,t)& =-||({\mathrm{\nabla }}_w^{-}f)(u){||}_p,\hfill \end{array}\end{array}$$

(23)

for $1\le p\le \infty$, with initial condition $f(u,0)=f^0(u)$. The discrete expression of the internal and external gradient constitutes a direct spectral numerical scheme, with the usual notation $f^n(u)=f(u,n\Delta t)$. The iterative scheme for dilation and erosion can be defined as:

$$f^{n+1}(u)=f^n(u)\pm \Delta t||{\mathrm{\nabla }}_w^{\pm }{f}^n(u){||}_p\mathrm{with}{f}^{(0)}=f^0(u)$$

(24)

considering the ${\mathcal{L}}_p$ norm, the equation becomes:

$$\begin{array}{cc}\hfill f^{n+1}(u)& =f^{(n)}(u)\pm \Delta t{\left[\sum _{v\in V}{(w(u,v))}^{\frac{p}{2}}{|{\partial }_v^{\pm }{f}^n(u)|}^p\right]}^{\frac{1}{p}}\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill f^{(0)}& =f^0(u)\hfill \end{array}$$

(26)

Note that, under this form, the coefficients of this algebraic equation can depend on the data.

For $w=1$ (unweighted graph) and for a grid graph, this corresponds to the first order discretization scheme of Osher and Sethian.

Considering the ${\mathcal{L}}_{\infty }$ norm, this equation becomes:

$$f^{(n+1)}(u)=f^{(n)}(u)\pm \Delta t\underset{v\sim u}{max}(\sqrt{w(u,v)}|{\partial }_v^{\pm }{f}^{(n)}(u)|).$$

(27)

With $\Delta t=1$ and $w=1$, the dilation and erosion equations become:

$$\begin{array}{cc}\hfill f^{(n+1)}& =\underset{v\sim u}{max}(f^{(n)}(v),f^{(n)}(u))\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill f^{(n+1)}& =\underset{v\sim u}{min}(f^{(n)}(v),f^{(n)}(u))\hfill \end{array}$$

(29)

In the special case where $\Delta t=1$, the dilation and the erosion can be interpreted as iterative Non-Local Dilation (NLD) and a Non-Local Erosion (NLE) process, respectively. These processes can be expressed as:

$$f^{n+1}(u)=NLD(f^n)(u),$$

(30)

where:

$$NLD(f)(u)=f(u)+||({\mathrm{\nabla }}_w^{+}f)(u){||}_{\infty }$$

(31)

for the dilation and:

$$f^{n+1}(u)=NLE(f^n)(u),$$

(32)

where:

$$NLE(f)(u)=f(u)-||({\mathrm{\nabla }}_w^{-}f)(u){||}_{\infty }$$

(33)

for the erosion.

This approach can be used to define other morphological operators based on erosion ϵ or dilation δ operators, such as openings $\gamma =(\delta ϵ)$, closings $\varphi =(ϵ\delta )$ or morphological gradients $(\delta -ϵ)$. For instance, we propose a formulation of the Non-Local Closing (NLC) operation that can be defined as:

$$\begin{array}{cc}\hfill \frac{\partial f}{\partial t}(u,t)=& -sig{n}^{+}(t-s+1)||({\mathrm{\nabla }}_w^{-}f)(u){||}_{\infty }\hfill \\ & +sig{n}^{+}(s-t)||({\mathrm{\nabla }}_w^{+}f)(u){||}_{\infty },\hfill \end{array}$$

(34)

with $t\in [0,2s[$, $s\in {\mathbb{R}}^{+}$ and:

$$sig{n}^{+}(x)=\{\begin{array}{l}1\mathrm{if}x>0,\\ 0\mathrm{otherwise}.\end{array}$$

(35)

This PdE has a time-dependent switching coefficient that makes it act as a dilation δ for $t\in [0,s]$ and an erosion ϵ for $t\in [s,2s]$. This formulation is different from the classical PDEs one [5] and does not produce discontinuities at the switching time. This can be interpreted as the following non-local iterative process:

$$\begin{array}{cc}\hfill f^{(n+1)}(u)=& NLC(f^{(n)})(u)\hfill \\ \hfill =& sig{n}^{+}(t-s+1)NLE(f^{(n)})(u)\hfill \\ & +sig{n}^{+}(s-t)NLD(f^{(n)})(u).\hfill \end{array}$$

(36)

Similarly, one can express the Non-Local Opening (NLO) as an iterative process.

3.2. Infinity Laplacian and Mean Curvature Flows on Graphs

In this subsection, we present a numerical scheme for the ∞-Laplacian and the mean curvature flows on graphs. We will show that these flows can be formulated using non-local dilation or non-local erosion defined above. If we consider the following parabolic PDE on the graph involving the ∞-Laplacian:

$$\{\begin{array}{ll}\frac{\partial f}{\partial t}(u,t)& ={\Delta }_{w,\infty }f(u,t)u\in V\\ f(u,0)& =f^0(u),\end{array}$$

(37)

using the notation $f^n(u)=f(u,n\Delta t)$, one can we replace the ∞-Laplacian by its definition (19):

$$\{\begin{array}{ll}{f}^{n+1}(u)& =f^n(u)+\Delta t\frac{1}{2}\left[||{\mathrm{\nabla }}_w^{+}{f(u)||}_{\infty }-||{\mathrm{\nabla }}_w^{-}f(u){||}_{\infty }\right]\\ f^{(0)}& =f^0(u).\end{array}$$

(38)

If $\Delta t=1$, the ∞-Laplacian can be interpreted as a symmetric morphological filter that depends on non-local dilation and non-local erosion. Indeed, Equation (38) can be rewritten as:

$$\{\begin{array}{ll}{f}^{n+1}(u)& =f^n(u)+NLA(f^n)(u)\\ f^{(0)}& =f^0(u),\end{array}$$

(39)

where:

$$NLA(f)(u)=\frac{1}{2}\left[NLD(f)(u)+NLE(f)(u)\right]$$

(40)

For the discretization of the continuous mean curvature flow equation:

$$\frac{\partial \varphi }{\partial t}=div\left(\frac{\mathrm{\nabla }\varphi }{|\mathrm{\nabla }\varphi |}\right)|\mathrm{\nabla }\varphi |,$$

(41)

we propose to replace the continuous curvature by the curvature on the graph and to use the following morphological scheme:

$$\left\{\begin{array}{c}\frac{\partial f}{\partial t}(u)={({\mathcal{K}}_w(f(u)))}^{+}\parallel ({\mathrm{\nabla }}_w^{+}f)(u){\parallel }_{\infty }-{({\mathcal{K}}_w(f(u)))}^{-}{\parallel ({\mathrm{\nabla }}_w^{-}f)(u)\parallel }_{\infty }\hfill \\ f(u,0)=f_0(u),\hfill \end{array}\right.$$

(42)

The time discretization leads to the following equation:

$$f^{n+1}(u)=[1-\Delta t|{\mathcal{K}}_w(f^n(u))|]f^n(u)+\Delta t{({\mathcal{K}}_w(f^n(u)))}^{+}·NLD(f^n)(u)+\Delta t{({\mathcal{K}}_w(f^n(u)))}^{-}·NLE(f^n)(u).$$

(43)

One can note that for $|{\mathcal{K}}_w|\ne 0$ and $\Delta t<\frac{1}{|{\mathcal{K}}_w(f^n(u))|}$, the previous equation corresponds to the following non-local average filtering:

$$f^{n+1}(u)=NL{A}_0(f^n)(u),$$

(44)

where:

$$NL{A}_0(f)(u)=(1-\Delta t|{\mathcal{K}}_w(f(u))|·f(u)+\Delta t{({\mathcal{K}}_w(f(u)))}^{+}·NLD(f)(u)+\Delta t{({\mathcal{K}}_w(f(u)))}^{-}·NLE(f)(u).$$

(45)

One can remark that this filter alternates between the non-local dilation of the image or non-local erosion of the image according to the sign of the curvature.

For $|{\mathcal{K}}_w(f^n(u))|\ne 0$ and $\Delta t=\frac{1}{|{\mathcal{K}}_w(f^n(u))|}$, the discrete mean curvature flow corresponds to the following non-local average filter:

$$f^{n+1}(u)=NL{A}_1(f^n)(u),$$

(46)

where:

$$NL{A}_1(f)(u)=\frac{{({\mathcal{K}}_w(f(u)))}^{+}}{|{\mathcal{K}}_w(f(u))|}·NLD(f)(u)+\frac{{({\mathcal{K}}_w(f(u)))}^{-}}{|{\mathcal{K}}_w(f(u))|}·NLE(f)(u).$$

(47)

$NL{A}_1$ is a non-local morphological filter alternating between non-local dilation and non-local erosion according to the curvature sign.

3.3. Level Set Equations on Graphs

Now, we present our transcription of PDEs level set equations on weighted graphs of arbitrary topology and an algorithm for solving the Eikonal equation on graphs.

Time-dependent level set: Let $G(V,E,w)$ be a weighted graph. A front evolving on G is defined as a subset ${\mathrm{\Omega }}_0\subset V$ and is implicitly represented at the initial time by a level set function ${\varphi }_0={\mathcal{U}}_0={\chi }_{{\mathrm{\Omega }}_0}-\chi {\mathrm{\Omega }}_0^c$, where $\chi :V\to \{0,1\}$ is the indicator function and ${\mathrm{\Omega }}_0^c$ is the complement of ${\mathrm{\Omega }}_0$. In other words, ${\varphi }_0$ equals one in ${\mathrm{\Omega }}_0$ and $-1$ on its complementary.

Then, from the general Equation (7) transposed on graphs, the front propagation can be described by:

$$\{\begin{array}{ll}\frac{\partial \varphi }{\partial t}(u)& =\mathcal{F}(u)||({\mathrm{\nabla }}_w\varphi )(u){||}_{\infty }\\ {\varphi }_0(u)& ={\mathcal{U}}_0\end{array}$$

(48)

with $\mathcal{F}\in \mathcal{H}(V)$, and $w:V\times V\to {\mathbb{R}}^{+}$ is the weight function. Based on the previous definition of discrete dilation and erosion on graphs [9,37], the front propagation can be expressed as a morphological process with the following sum of dilation and erosion:

$$\{\begin{array}{ll}\frac{\partial \varphi }{\partial t}(u)& ={\mathcal{F}}^{+}||({\mathrm{\nabla }}_w^{+}\varphi )(u){||}_{\infty }-{\mathcal{F}}^{-}||({\mathrm{\nabla }}_w^{-}\varphi )(u){||}_{\infty }\\ {\varphi }_0(u)& ={\mathcal{U}}_0\end{array}$$

(49)

To solve this dilation and erosion process, on the contrary to the PDEs case, no spatial discretization is required thanks to derivatives directly expressed in a discrete form. Then, the general iterative scheme to compute ϕ at time $t+1$ for all $u\in V$ is given by:

$$\begin{array}{cc}\hfill {\varphi }^{t+1}(u)={\varphi }^t(u)+\Delta t[& {(\mathcal{F}(u))}^{+}||({\mathrm{\nabla }}_w^{+}{\varphi }^t)(u){||}_{\infty }-\hfill \\ & {(\mathcal{F}(u))}^{-}||({\mathrm{\nabla }}_w^{-}{\varphi }^t)(u){||}_{\infty }]\hfill \end{array}$$

(50)

At each time $t+1$, the new value at a vertex u only depends on its value at time t and the existing values in its neighborhood. This equation can be split into two independent equations, depending on the sign of $\mathcal{F}$:

$${\varphi }^{t+1}(u)=\{\begin{array}{c}{\varphi }^t(u)+\Delta t\mathcal{F}(u)||({\mathrm{\nabla }}_w^{+}{\varphi }^t)(u){||}_{\infty },\mathcal{F}(u)>0\hfill \\ {\varphi }^t(u)+\Delta t\mathcal{F}(u)||({\mathrm{\nabla }}_w^{-}{\varphi }^t)(u){||}_{\infty },\mathcal{F}(u)<0.\hfill \end{array}$$

(51)

Stationary level set: Eikonal equation: When $\mathcal{F}\ge 0$ on Ω, the relation between the level set formulation (48) and the Eikonal equation $(\mathcal{F}(x)||\mathrm{\nabla }T(x)||=1)$ stems from the following change of variable: $\varphi (x,y)=t-T(x)$ (where $T(x)$ is the arrival time of the curve at a point x).

Using previous definitions of morphological evolution equations, one can formulate the same relation and obtain a PdEs-based version of the Eikonal equation, defined on weighted graphs of arbitrary topology. Let $G(V,E,w)$ be a weighted graph, with functions T and $\mathcal{F}$ defined on $\mathcal{H}(V)$. Because $\mathcal{F}\ge 0$, the evolution process described by Equation (48) can be seen as a dilation process, and the evolution equation can be rewritten as:

$$\frac{\partial \varphi }{\partial t}(u)=\mathcal{F}||({\mathrm{\nabla }}_w^{+}\varphi )(u)||.$$

(52)

With a similar change of variable $\varphi (u)=t-T(u)$, one has:

$$\frac{\partial \varphi }{\partial t}(u)=\mathcal{F}||({\mathrm{\nabla }}_w^{+}(t-T)(u)||=\mathcal{F}||({\mathrm{\nabla }}_w^{-}T)(u)||,$$

(53)

and then,

$$\mathcal{F}||({\mathrm{\nabla }}_w^{-}T)(u)||=1.$$

(54)

Finally, with $P=1/\mathcal{F}$, we obtain a discrete adaptation of the Eikonal equation on weighted graph, which describes a morphological erosion process, and defined as:

$$\{\begin{array}{ll}||({\mathrm{\nabla }}_w^{-}f)(u){||}_p=P(u)& \forall u\in V\\ f(u)=0& \forall u\in V_0\end{array}$$

(55)

From Equation (55) and using norms defined in Equations (16) and (17) with the property $min(x,0)=-max(-x,0)$, we obtain the following equations for the ${\mathcal{L}}_p$ and ${\mathcal{L}}_{\infty }$ norms:

• When $p\in \{1,2\}$:

  $${\left(\sum _{v\sim u}{w}_{uv}^{p/2}max{\left(0,f(u)-f(v)\right)}^p\right)}^{1/p}=P(u)$$

  (56)

  With a simple transformation of variables and some conventional notations, Equation (56) can be rewritten as:

  $$\sum _{i=1}^n{\left(\frac{{(x-a_i)}^{+}}{h_i}\right)}^p={\mathcal{C}}^p,$$

  (57)

  where $x=f(u),h_i=\sqrt{1/w_{uv}},n=\mathrm{card}(N(u)),a=\{f(v_i)|v_i\in N(u)\mathrm{with}i=1,...,n\},\mathcal{C}=P(u)$.
• When $p=\infty$:

  $$\underset{v\sim u}{max}(\sqrt{w_{uv}}max(0,f(u)-f(v)))=P(u)$$

  (58)

  Using the same transformation of variables, we obtain:

  $$\underset{i}{max}\left(\frac{{(x-a_i)}^{+}}{h_i}\right)=\mathcal{C}.$$

  (59)

  Local solutions (i.e., solution for a vertex, assuming the others are held fixed) of Equation (55) over a weighted graph are given by Equations (57) and (59). Both equations are clearly independent of the graph formulation and can be applied to weighted graphs of arbitrary topology.

Algorithm 1: $\overline{x}$ computation (Local solution).

For the case $p=\{1,2\}$, local solution $\overline{x}$ at a particular vertex can be easily obtained with an iterative algorithm as described in Algorithm 1. It is based on the knowledge that there exists k ($1\le k\le n$), such that $a_k\le \overline{x}\le a_{k+1}$, and $\overline{x}$ is the unique solution of the equation. Then, the algorithm consists of sorting increasingly the values $a_i$ and computing temporary solution ${\widehat{x}}_m$ until the condition ${\widehat{x}}_m\le a_{m+1}$ is satisfied. To compute the solution of Eikonal equation at each vertex, we used Fast Marching’s updating scheme [37].

Label propagation: We propose a simple way to propagate an initial set of labels (from a set of seed vertices $V_0$) through the graph, following the evolution of the propagating fronts. Then, the propagating front that reaches a vertex is necessarily the front coming from the nearest seed (according to the weight function). Therefore, each time a distance is updated on a vertex u, we find the neighbor v of u, which is the closest to both u and a seed $v\in V_0$, and extend the label of v to the current vertex u. The labeling process can be summarized by the following formula: each time $f(u)$ is updated, the label $L(u)$ is given by:

$$L(u)=L(v)|v\sim u,f(v)<f(u)\mathrm{and}\frac{f(v)}{w(u,v)}=\underset{z\sim u}{min}\left(\frac{f(z)}{w(u,z)}\right)$$

(60)

4. Graph Construction and Examples of Image Processing on Point Clouds

4.1. Graph Construction from Images on 3D Point Clouds

We consider a surface or a point cloud composed by a set S of vertices, such as $S=\{x_1,x_2,...\}$, with $x_i\in {\mathbb{R}}^3$. To each raw point $x_i\in S$, one associates a vertex of a graph G to define a set of vertices V. Data on the image or point cloud can be defined as a function $f:V\to \mathbb{R}$. The construction of such a graph consists of modeling the neighborhood relationships between the data through the definition of a set of edges E and using a pairwise similarity measure $\mu :V\times V\to \mathrm{I}{\mathrm{R}}^{+}$. In the particular case of images, edges based on geometric neighborhoods are particularly well-adapted to represent the geometry.

From a point cloud, a k-nn-graph is built. k controls the number of incident edges at each vertex and depends on the application. Generally, a low value (e.g., $k=5$) is set for a local processing and a higher value for a non-local processing.

From a mesh, the set E is deduced from the implicit graph in the triangular mesh. The similarity between two vertices is computed by a similarity measure $s:E\to \mathrm{I}{\mathrm{R}}^{+}$, which satisfies:

$$w(u,v)=\left\{\begin{array}{cc}s(u,v),\hfill & \mathrm{if}(u,v)\in E\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

Common similarity functions are the following:

$$\begin{array}{cc}\hfill s_0(u,v)\equiv & 1,\hfill \\ \hfill s_1(u,v)=& exp(-\mu (f(u),f(v)\left)/{\sigma }^2\right),\hfill \end{array}$$

for which the variance parameter $\sigma >0$ usually depends on the variation of the function μ.

The function f used to describe the data at a vertex u is considered as the feature vector. Several choices can be considered for the expression of the feature vectors, depending on the nature of the features to be used for graph processing. In the context of image processing, one can use a simple gray scale or color feature vector $F_u$ or a patch feature vector $F_u^{\tau }={\bigcup }_{v\in {\mathcal{W}}^{\tau }(u)}{F}_v$ (i.e., the set of values $F_v$ for which v is in a square window ${\mathcal{W}}^{\tau }(u)$ of size $(2\tau +1)\times (2\tau +1)$ centered at pixel u). Note that the latter vector allows one to incorporate nonlocal features for $\tau \ge 1$. We now present our method for the construction of patches on surfaces and point clouds.

Patch construction for 3D point clouds: Extending the notion of a patch to three-dimensional point cloud data is not an easy task. In [21], we have proposed a novel definition of patches that can be used for any graph representation associated with meshes or point clouds. In particular, around each vertex, we build a two-dimensional grid (the patch) describing the close neighborhood. This grid is defined on the tangent plane of the point (i.e., the vertex). Then, the patch is oriented accordingly, and finally, the patch is filled in with a weighted average of the graph-signal values in the local neighborhood. The first step consists of estimating the orientation of each patch. The second step consists of the actual patch construction. The set of values inside the patch of the vertex u is denoted as $\overrightarrow{\mathcal{P}}(u)$. Let $C_k(u)$ denote the k-th cell of the constructed patch around u with $k\in \{1,\cdots ,n^2\}$. With the proposed patch construction process, one can define the set $V_k(u)=\{v|{\overrightarrow{p}}_v^{\prime }\in C_k(u)\}$ as the set of vertices v that are assigned to the k-th patch cell of u. Then, the patch vector is defined as $\overrightarrow{\mathcal{P}}(u)={\left(\frac{\sum _{v\in V_k(u)}w({\overrightarrow{c}}_k,{\overrightarrow{p}}_v)f(v)}{\sum _{v\in V_k(u)}w({\overrightarrow{c}}_k,{\overrightarrow{p}}_v)}\right)}_{k\in \{1,\cdots ,n^2\}}^T$ with $w({\overrightarrow{c}}_k,{\overrightarrow{p}}_u)=exp(-\frac{||{\overrightarrow{c}}_k-{\overrightarrow{p}}_u^{\prime }{||}_2^2}{{\sigma }^2})$, for which $\overrightarrow{c_k}$ is the vector of coordinates of the k-th patch cell center. This weighting enables us to take into account the point distribution within the patch cells in order to compute their mean feature vectors. Figure 1a summarizes our patch construction method.

4.2. Local, Non-Local Inpainting and Filtering of Images on Point Clouds

In this section, we illustrate through some examples the application of morphological operators on graphs for inpainting, filtering of images on point clouds. All processing have been implemented in C++ and take almost several seconds on a current laptop running on a GNU/Linux operating system.

Inpainting: Image inpainting using a patch-based approach has been proposed by [38]. Recent works on inpainting tend to unify local and non-local approaches under a discrete or continuous variational formulation (for more details, see [39,40] and the references therein). Here, we consider that images on point clouds are defined on a general domain represented by an adaptive graph $G=(V,E,w)$. We consider both local and non-local inpainting as an interpolation problem. Let $f^0:V\to \mathrm{I}\mathrm{R}$ be an initial function. Let $A\subset V$ be the subset of vertices with unknown values and $\partial A$ the subset of vertices with known values. The purpose of inpainting is to find a function f extending $f^0$ in V by solving the ∞-Laplacian equation:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}({\Delta }_{w,\infty }f)(u)=0\forall u\in A,\hfill \\ f(u)=f^0(u)\forall u\in \partial A.\hfill \end{array}\right.\end{array}$$

(61)

The solution f is said to be infinity-harmonic [40].

The solution is obtained with this simple iterative algorithm based on the NLA operator introduced in Equation (40). At each iteration, only the internal boundary ${\partial }^{-}A=\{u\in A|\exists v\sim u,v\in \partial A\}$ is interpolated:

$$\{\begin{array}{ll}NLA(f)(u)=\frac{1}{2}[NLD(f)(u)+NLE(f)(u)]& \forall u\in {\partial }^{-}A,\\ NLA(f)(u)=f^0(u)& \forall u\in \partial A.\end{array}$$

(62)

At the end of each iteration, the set $\partial A$ is updated by $\partial A^{(n+1)}=\partial A^{(n)}\cup {\partial }^{-}{A}^{(n)}$, and ${\partial }^{-}{A}^{(n+1)}$ is updated from $\partial A^{(n+1)}$. The algorithm stops when the set of vertices to inpaint is empty.

$$\{\begin{array}{l}{f}^{(0)}(u)=f^0(u),\\ f^{(n+1)}(u)=NLA(f^{(n)})(u).\end{array}$$

(63)

Figure 2 shows the inpainting results of a synthetic textured point cloud with a constant ($w=1$) weight function defined locally (Figure 2b) and with a patch-based similarity metric defined on a non-local neighborhood (Figure 2c). The method presented in Section 4.1 has been used to construct the graph from the point cloud. Better results are obtained with non-local processing with the patch-based similarity measure. This weighting scheme captures texture information and is then able to interpolate repetitive patterns. Figure 3 shows an inpainting result of a colored point cloud acquired from a real object.

Filtering: We give some examples of filtering by local and non-local morphological operators: NLD (Non-Local Dilation), NLE (Non-Local Erosion), NLA (∞-Laplacian) and $NL{A}_1$ (mean curvature).

Figure 4 shows the filtering of a colored point cloud with the $NLD$, $NLE$ and $NLA$ operators. The weight functions w are computed from the colors of the point cloud. Results are shown with $w\ne 1$ and $w=1$. When $w\ne 1$, denoising results are better since edges are preserved. Figure 5 shows the filtering of a colored point cloud with several operators. The $NL{A}_1$ operator defined in Equation (47) corresponds to a filtering operator by alternating the process of non-local dilation and non-local erosion according to the sign of the curvature ${\mathcal{K}}_w$ on the graph Equation (21). When $w=1$, the filtering is unable to restore the colors on the point clouds. Figure 6 shows a comparison of the result obtained with the $NLA$ and $NL{A}_1$ operators on a scanned room [41]. The $NL{A}_1$ operator produces better results by smoothing the colors while preserving details and contours, while the $NLA$ operator smooths by removing many details.

4.2.1. Generalized Weighted Distance, Shortest Path and Eikonal Equation for Segmentation of the Image on Point Clouds

The Eikonal equation defined in Equation (55) on a weighted graph of arbitrary topology allows one to compute generalized distances on graphs. We use the Eikonal equation with label propagation for computing the shortest path and segmentation on meshes or images painted on point clouds.

Generalized weighted distance: We compute the generalized distance on several point clouds by solving the Eikonal equation; see Figure 7. We built the graph as a k-nn with $k=5$, in the coordinate space of the point cloud. Since the spatial discretization step is regular enough, we use a constant weight function ($w(u,v)=1$). The superimposed red line on the figure is the shortest path between the source point (the point from which the distance is computed) and another point in the point cloud. This path was deformed using the computed distance function. Figure 8 shows the evolution of the distance estimation on a mesh from a vertex, with or without colorimetric constraints.

Figure 9 shows the shortest path between two points of a 3D city. This latter 3D point cloud has been generated from the online web interface to estimate colorized point clouds obtained from stereo satellite images [42]. The shortest path was obtained by processing the raw point cloud. Note that, even if the point cloud estimated is highly noisy and not uniform, the shortest path takes roads by nicely avoiding buildings. The result has been obtained with $k=10$.

Segmentation: Figure 10 shows the segmentation of a point cloud from colorimetric values. In Figure 8 and Figure 10, the similarity function w is computed from the color distance with $w(u,v)=e^{-||f(u)-{f(v)||}^2/{\sigma }^2}$ and $f:V\to \mathbb{R}$ the colorimetric function. Figure 11 shows the segmentation of meshes from the estimation of the geometric curvature; the similarity function is computed with $w(u,v)=e^{-{(K(u)-K(v))}^2/{\sigma }^2}$; and $K:V\to \mathbb{R}$ the function that associates an estimation of the square of major principal curvature at each vertex $u\in V$. The estimation of the principal curvature is obtained by computing the eigenvalues as proposed in [43,44].

5. Conclusions

In this paper, we have adopted the PdE framework, and we have focused on some PDEs-based continuous morphological operators on Euclidean domains: dilation/erosion, mean curvature flows and the Eikonal equation. We have extended their applications for the processing of colored point clouds of geo-informatics 3D data. We briefly presented the construction of a graph from mesh or point cloud and show several examples, such as restoration, denoising, inpainting, object extraction or estimation of the minimal path. We have shown results on geographic data, such as the estimation of the shortest path on a raw point cloud of a 3D city or the segmentation of a LIDAR 3D point cloud based on the colorimetric values. The results demonstrated many potentials of the point cloud approach for the processing of geo-informatics 3D data.