# Morphological PDEs on Graphs for Image Processing on Surfaces and Point Clouds

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- Methods using explicit representations of surfaces represented by a parametric function: This corresponds to attaching a two-dimensional parametric domain $(u,v)$ to the 3D object. By relying on a specific parametrization of the given surface, differential operators can be defined and computed analytically [17,18]. However, the computation of the parametrization is a difficult task for arbitrary given surfaces, and topological changes can hardly be handled.
- Methods using implicit representations of surfaces represented by a zero level-set function of a signed distance function in Euclidean domains. The differential operators are then approximated by combining the Euclidean differential operators with a projection along the normal direction [1,19]. For instance, in [20], the coordinates of the closest point for each point of the surface is used, and fast algorithms can be obtained in Euclidean domains. Implicit representations can deal easily with topological changes, but all of the data have to be extended to the definition domain of the implicit function.
- Methods using intrinsic geometry to study variational problems directly on the surface represented as a triangular mesh. Lai and Chan have recently proposed [14] a framework for intrinsic image processing on surfaces. They approximate surface differential operators, such as surface gradient and divergence, by specific intrinsic differential geometry definitions. Intrinsic methods do not need any pre-processing, but they require a specific discretization scheme on triangles.
- Methods solving PDEs directly on point clouds [15,16] by using intermediate representations to approximate differential operators on point clouds. The authors in [16] use a local triangulation that requires pre-processing. This pre-processing is needed to estimate differential operators on triangular meshes. This method can then further be categorized as an intrinsic method. The authors of [15] compute a local approximation of the manifold using moving least squares from the k-nearest neighbors. From this local coordinate system, a local metric tensor is computed at each point such that the differentiation on the manifold is simplified. This method can therefore be categorized as an explicit method.

## 2. Partial Difference Operators on Graphs

#### 2.1. Notations and Preliminaries

#### 2.2. Difference Operators on Weighted Graphs

**Definition**

**1.**

**Definition**

**2.**

**Definition**

**3.**

**Definition**

**4.**

**Definition**

**5.**

**Definition**

**6.**

## 3. Morphological Operators on Graphs

#### 3.1. Dilation and Erosion on Graphs for Filtering

#### 3.2. Infinity Laplacian and Mean Curvature Flows on Graphs

#### 3.3. Level Set Equations on Graphs

- When $p\in \{1,2\}$:$${\left(\sum _{v\sim u}{w}_{uv}^{p/2}max{\left(0,f(u)-f(v)\right)}^{p}\right)}^{1/p}=P(u)$$$$\sum _{i=1}^{n}{\left(\frac{{(x-{a}_{i})}^{+}}{{h}_{i}}\right)}^{p}={\mathcal{C}}^{p},$$
- When $p=\infty $:$$\underset{v\sim u}{max}(\sqrt{{w}_{uv}}max(0,f(u)-f(v)))=P(u)$$$$\underset{i}{max}\left(\frac{{(x-{a}_{i})}^{+}}{{h}_{i}}\right)=\mathcal{C}.$$

Algorithm 1: $\overline{x}$ computation (Local solution). |

## 4. Graph Construction and Examples of Image Processing on Point Clouds

#### 4.1. Graph Construction from Images on 3D Point Clouds

#### 4.2. Local, Non-Local Inpainting and Filtering of Images on Point Clouds

#### 4.2.1. Generalized Weighted Distance, Shortest Path and Eikonal Equation for Segmentation of the Image on Point Clouds

## 5. Conclusions

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## References

- Osher, S.; Fedkiw, R. Level Set Methods and Dynamic Implicit Surfaces; Springer: New York, NY, USA, 2003; Volume 153. [Google Scholar]
- Sethian, J.A. Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science; Cambridge University Press: Cambridge, UK, 1999; Volume 3. [Google Scholar]
- Elmoataz, A.; Toutain, M.; Tenbrinck, D. On the p-Laplacian and ∞-Laplacian on Graphs with Applications in Image and Data Processing. SIAM J. Imaging Sci.
**2015**, 8, 2412–2451. [Google Scholar] [CrossRef] - Elmoataz, A.; Lozes, F.; Toutain, M. Nonlocal PDEs on Graphs: From Tug-of-War Games to Unified Interpolation on Images and Point Clouds. J. Math. Imaging Vis.
**2016**. [Google Scholar] [CrossRef] - Alvarez, L.; Guichard, F.; Lions, P.L.; Morel, J.M. Axioms and fundamental equations of image processing. Arch. Ration. Mech. Anal.
**1993**, 123, 199–257. [Google Scholar] [CrossRef] - Elmoataz, A.; Desquesnes, X.; Lézoray, O. Non-Local Morphological PDEs and-Laplacian Equation on Graphs With Applications in Image Processing and Machine Learning. IEEE J. Sel. Top. Signal Process.
**2012**, 6, 764–779. [Google Scholar] [CrossRef] - Maragos, P. Differential morphology and image processing. IEEE Trans. Image Process.
**1996**, 5, 922–937. [Google Scholar] [CrossRef] [PubMed] - Meyer, F.; Maragos, P. Multiscale Morphological Segmentations Based on Watershed, Flooding, and Eikonal PDE. In Proceedings of the International Conference on Scale-Space Theories in Computer Vision, Corfu, Greece, 26–27 September 1999; pp. 351–362.
- Ta, V.T.; Elmoataz, A.; Lézoray, O. Nonlocal PDEs-based Morphology on Weighted Graphs for Image and Data Processing. IEEE Trans. Image Process.
**2011**, 26, 1504–1516. [Google Scholar] - Dorninger, P.; Pfeifer, N. A Comprehensive Automated 3D Approach for Building Extraction, Reconstruction, and Regularization from Airborne Laser Scanning Point Clouds. Sensors
**2008**, 8, 7323–7343. [Google Scholar] [CrossRef] - Wallace, L.; Lucieer, A.; Watson, C.; Turner, D. Development of a UAV-LiDAR System with Application to Forest Inventory. Remote Sens.
**2012**, 4, 1519–1543. [Google Scholar] [CrossRef] - Jochem, A.; Höfle, B.; Rutzinger, M.; Pfeifer, N. Automatic Roof Plane Detection and Analysis in Airborne Lidar Point Clouds for Solar Potential Assessment. Sensors
**2009**, 9, 5241–5262. [Google Scholar] [CrossRef] [PubMed] - Dandois, J.P.; Ellis, E.C. Remote Sensing of Vegetation Structure Using Computer Vision. Remote Sens.
**2010**, 2, 1157–1176. [Google Scholar] [CrossRef] - Lai, R.; Chan, T.F. A framework for intrinsic image processing on surfaces. Comput. Vis. Image Underst.
**2011**, 115, 1647–1661. [Google Scholar] [CrossRef] - Liang, J.; Lai, R.; Wong, T.W.; Zhao, H. Geometric Understanding of Point Clouds Using Laplace-Beltrami Operator. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA, 16–21 June 2012; pp. 214–221.
- Lai, R.; Liang, J.; Zhao, H. A local mesh method for solving PDEs on point clouds. Inverse Probl. Imaging
**2013**, 7, 737–755. [Google Scholar] [CrossRef] - Stam, J. Flows on surfaces of arbitrary topology. ACM Trans. Graph.
**2003**, 22, 724–731. [Google Scholar] [CrossRef] - Spira, A.; Kimmel, R. Geometric curve flows on parametric manifolds. J. Comput. Phys.
**2007**, 223, 235–249. [Google Scholar] [CrossRef] - Bertalmio, M.; Cheng, L.; Osher, S.; Sapiro, G. Variational Problems and Partial Differential Equations on Implicit Surfaces. J. Comput. Phys.
**2001**, 174, 759–780. [Google Scholar] [CrossRef] - Ruuth, S.J.; Merriman, B. A simple embedding method for solving partial differential equations on surfaces. J. Comput. Phys.
**2008**, 227, 1943–1961. [Google Scholar] [CrossRef] - Lozes, F.; Elmoataz, A.; Lezoray, O. Partial Difference Operators on Weighted Graphs for Image Processing on Surfaces and Point Clouds. IEEE Trans. Image Process.
**2014**, 23, 3896–3909. [Google Scholar] [CrossRef] [PubMed] - Lozes, F.; Elmoataz, A.; Lezoray, O. PDE-Based Graph Signal Processing for 3-D Color Point Clouds: Opportunities for cultural heritage. IEEE Signal Process. Mag.
**2015**, 32, 103–111. [Google Scholar] [CrossRef] - Haralick, R.M.; Sternberg, S.R.; Zhuang, X. Image analysis using mathematical morphology. IEEE Trans. Pattern Anal. Mach. Intell.
**1987**, 9, 532–550. [Google Scholar] [CrossRef] [PubMed] - Serra, J.; Soille, P. Mathematical Morphology and Its Applications to Image Processing; Springer Science & Business Media: Berlin, Germany, 2012; Volume 2. [Google Scholar]
- Angulo, J. Morphological PDE and Dilation/Erosion Semigroups on Length Spaces. In International Symposium on Mathematical Morphology and Its Applications to Signal and Image Processing; Springer: Cham, Switzerland, 2015; pp. 509–521. [Google Scholar]
- Crandall, M.G.; Lions, P.L. Viscosity solutions of Hamilton-Jacobi equations. Trans. Am. Math. Soc.
**1983**, 277, 1–42. [Google Scholar] [CrossRef] - Maragos, P. Slope transforms: Theory and application to nonlinear signal processing. IEEE Trans. Signal Process.
**1995**, 43, 864–877. [Google Scholar] [CrossRef] - Li, H.; Elmoataz, A.; Fadili, J.M.; Ruan, S. An improved image segmentation approach based on level set and mathematical morphology. In Proceedings of the Third International Symposium on Multispectral Image Processing and Pattern Recognition, Beijing, China, 20–22 October 2003; pp. 851–854.
- Osher, S.; Sethian, J.A. Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations. J. Comput. Phys.
**1988**, 79, 12–49. [Google Scholar] [CrossRef] - Rouy, E.; Tourin, A. A viscosity solutions approach to shape-from-shading. SIAM J. Numer. Anal.
**1992**, 29, 867–884. [Google Scholar] [CrossRef] - Pizarro, L.; Burgeth, B.; Breuß, M.; Weickert, J. A Directional Rouy-Tourin Scheme for Adaptive Matrix-Valued Morphology. In International Symposium on Mathematical Morphology and Its Applications to Signal and Image Processing; Springer: Berlin/Heidelberg, Germany, 2009; pp. 250–260. [Google Scholar]
- Breuß, M.; Weickert, J. A shock-capturing algorithm for the differential equations of dilation and erosion. J. Math. Imaging Vis.
**2006**, 25, 187–201. [Google Scholar] [CrossRef] - Breuß, M.; Weickert, J. Highly Accurate PDE-Based Morphology for General Structuring Elements. In Proceedings of the International Conference on Scale Space and Variational Methods in Computer Vision, Voss, Norway, 1–5 June 2009; Springer: Berlin/Heidelberg, Germany, 2009; pp. 758–769. [Google Scholar]
- Burgeth, B.; Breuß, M.; Pizarro, L.; Weickert, J. PDE-Driven Adaptive Morphology for Matrix Fields. In Proceedings of the International Conference on Scale Space and Variational Methods in Computer Vision, Voss, Norway, 1–5 June 2009; pp. 247–258.
- Elmoataz, A.; Lezoray, O.; Bougleux, S. Nonlocal Discrete Regularization on Weighted Graphs: A Framework for Image and Manifold Processing. IEEE Trans. Image Process.
**2008**, 17, 1047–1060. [Google Scholar] [CrossRef] [PubMed] - Bougleux, S.; Elmoataz, A.; Melkemi, M. Local and nonlocal discrete regularization on weighted graphs for image and mesh processing. Int. J. Comput. Vis.
**2009**, 84, 220–236. [Google Scholar] [CrossRef] - Desquesnes, X.; Elmoataz, A.; Lézoray, O. Eikonal equation adaptation on weighted graphs: fast geometric diffusion process for local and non-local image and data processing. J. Math. Imaging Vis.
**2013**, 46, 238–257. [Google Scholar] [CrossRef] - Efros, A.A.; Leung, T.K. Texture Synthesis by Non-Parametric Sampling. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–27 September 1999; Volume 2, pp. 1033–1038.
- Arias, P.; Facciolo, G.; Caselles, V.; Sapiro, G. A Variational Framework for Exemplar-Based Image Inpainting. Int. J. Comput. Vis.
**2011**, 93, 319–347. [Google Scholar] [CrossRef] - Ghoniem, M.; Elmoataz, A.; Lézoray, O. Discrete Infinity Harmonic Functions: Towards a Unified Interpolation Framework on Graphs. In Proceedings of the 2011 18th IEEE International Conference on Image Processing (ICIP), Brussels, Belgium, 11–14 September 2011; pp. 1361–1364.
- Lai, K.; Bo, L.; Fox, D. Unsupervised Feature Learning for 3D Scene Labeling. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 31 May–7 June 2014.
- De Franchis, C.; Meinhardt-Llopis, E.; Michel, J.; Morel, J.M.; Facciolo, G. An automatic and modular stereo pipeline for pushbroom images. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.
**2014**, 2, 49–56. [Google Scholar] [CrossRef] - Berkmann, J.; Caelli, T. Computation of surface geometry and segmentation using covariance techniques. IEEE Trans. Patttern Anal. Mach.
**1994**, 16, 1114–1116. [Google Scholar] [CrossRef] - Digne, J.; Morel, J.M. Numerical analysis of differential operators on raw point clouds. Numer. Math.
**2014**, 127, 255–289. [Google Scholar] [CrossRef]

**Figure 1.**Interpolation of the content of the patch with a patch length of $l>0$. ${\overrightarrow{o}}_{0}(u)$ and ${\overrightarrow{o}}_{1}(u)$ are the orientations of the patch $\overrightarrow{\mathcal{P}}(u)$ at a point ${\overrightarrow{p}}_{u}$. Elements marked by a “×” symbol correspond to the projected neighbors ${\overrightarrow{p}}_{u}^{\prime}$ of the point ${\overrightarrow{p}}_{u}$ on the patch. These projections are used to deduce the values of each patch cell (a “o” symbol) by a weighted average of the associated graph signal values.

**Figure 2.**Inpainting of the color on a point cloud. (

**a**) Point cloud to inpaint; (

**b**) result of inpainting with $w=1$; (

**c**) result of inpainting with w based on the patch defined on point clouds (see Section 4.2).

**Figure 3.**Inpainting of the color on a point cloud. (

**a**) Original point cloud; (

**b**) point cloud to inpaint, where the red color represents points to inpaint; (

**c**) result of inpainting with $w=1$; (

**d**) result of inpainting with the patch-based similarity metric (see Section 4.2).

**Figure 4.**Filtering of a colored point cloud with the $NLD$, $NLD$ and $NLA$ operators with $k=8$. (

**a**) Original; (

**b**) $NLA\phantom{\rule{4pt}{0ex}}w=1$; (

**c**) $NLE\phantom{\rule{4pt}{0ex}}w=1$; (

**d**) $NLD\phantom{\rule{4pt}{0ex}}w=1$; (

**e**) $NL{A}_{0}\phantom{\rule{4pt}{0ex}}w\ne 1$; (

**f**) $NLA\phantom{\rule{4pt}{0ex}}w\ne 1$; (

**g**) $NLE\phantom{\rule{4pt}{0ex}}w\ne 1$; (

**h**) $NLD\phantom{\rule{4pt}{0ex}}w\ne 1$. (

**b**–

**d**,

**g**,

**h**) The results after 20 iterations. (

**e**,

**f**) The results after 200 iterations (see Section 4.2).

**Figure 5.**Filtering of a colored point cloud with several operators. Parameters are $k=8,\sigma =15$ with 20 iterations for (

**a**–

**e**) (see Section 4.2). (

**a**) Original; (

**b**) $NLD$; (

**c**) $NLE$; (

**d**) $NLA$; (

**e**) $NL{A}_{1}$.

**Figure 7.**Geodesic distances on 3D point clouds with the Eikonal equation on the graph. (

**a**–

**c**) show the original point clouds; (

**d**–

**f**) show the evolution of the distances with the minimal path linking two points of the point cloud. Blue and red color show low and high distances, respectively. White lines display isolines.

**Figure 8.**Generalized distance on the mesh with the Eikonal equation on the graph. From left to right: (

**a**) original mesh; (

**b**) result with $w=1$; (

**c**) result with w that depends on the vertices’ colors (see Section 4.2.1). Blue and red color show low and high distances, respectively. White lines display isolines.

**Figure 9.**Estimation of a shortest path on raw point clouds of a 3D city. The point cloud has been estimated from satellite images using [42]. From left to right: (

**a**) one of the satellite images used to estimate the 3D point cloud; (

**b**) 3D point cloud of a city, with the shortest path depicted in red.

**Figure 10.**Segmentation of 3D point clouds based on the colorimetric values by solving the Eikonal equation with $P(u)=1$ (see Section 4.2.1). (

**a**–

**c**) contain initial labels on point clouds. (

**d**–

**f**) show the results of the colorimetric segmentation, where red color represents the border between the blue and the green area.

**Figure 11.**Segmentation of 3D meshes based on the geometric principal curvatures by solving the Eikonal equation with $P(u)=1$ (see Section 4.2.1). (

**a**,

**b**) show the square of major principal curvature (blue color for low values, red for high values); (

**c**,

**d**) show the initial labels on the 3D mesh; (

**e**,

**f**) show the results of the geometric segmentation.

© 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Elmoataz, A.; Lozes, F.; Talbot, H.
Morphological PDEs on Graphs for Image Processing on Surfaces and Point Clouds. *ISPRS Int. J. Geo-Inf.* **2016**, *5*, 213.
https://doi.org/10.3390/ijgi5110213

**AMA Style**

Elmoataz A, Lozes F, Talbot H.
Morphological PDEs on Graphs for Image Processing on Surfaces and Point Clouds. *ISPRS International Journal of Geo-Information*. 2016; 5(11):213.
https://doi.org/10.3390/ijgi5110213

**Chicago/Turabian Style**

Elmoataz, Abderrahim, François Lozes, and Hugues Talbot.
2016. "Morphological PDEs on Graphs for Image Processing on Surfaces and Point Clouds" *ISPRS International Journal of Geo-Information* 5, no. 11: 213.
https://doi.org/10.3390/ijgi5110213