#
A PDE-Free Variational Method for Multi-Phase Image Segmentation Based on Multiscale Sparse Representations^{ †}

^{*}

^{†}

^{‡}

## Abstract

**:**

## 1. Introduction

**About our method.**As was remarked in [19], the multi-phase image segmentation problem is different from any binary segmentation task. Our method, just as the above mentioned level set segmentation, by construction avoids the issues of ‘vacuum’ and ‘overlap’; moreover, m binary functions are enough to encode ${2}^{m}$-valued piecewise constant approximation. The case when $m=2$ is considered especially interesting, since, formally, the partition of any piecewise smooth image near any boundary junction can be encoded by only two binary functions, per the four color theorem. Therefore, in our problem setup, we focus on the four-phase image segmentation as a typical case of a problem of approximating a given image by a piecewise constant one, attaining at most four intensity values, with connected components of the output satisfying certain regularity conditions. Those conditions are imposed by the properties of the energy functional we introduce in Section 2.1.

**Novelty and contributions.**Preliminary findings about this approach appeared in the SPIE Conference Proceedings [36]. In this paper, we do an in-depth investigation of this segmentation model. We provide a detailed description of the numerical scheme and conditions that make it gradient stable; we include the formal proof of the minimizer existence, a detailed description of the fidelity term choice and parameter choice; we also describe generalizations and adaptations of this method for particular applications: using it with post-processing for specific needs of medical imaging (blood vessel detection) and generalization of the method to process color (vector-valued) images, both illustrated with examples of numerical simulations. Moreover, following up on the requests of this journal’s reviewers, we include the quantitative area-segmentation comparisons to other methods (using benchmarks described in [31] and other works associated with the Berkeley segmentation database(s); please, see Section 4.1 and Appendix B.1).

#### 1.1. PDE-Free Variational Models Based on Sparse Representations: Motivation

#### 1.2. Wavelets in Variational Models for Image Analysis and Recovery, Wavelet Ginzburg–Landau Energy

## 2. The Proposed Model for the Four-Phase Image Segmentation

#### 2.1. The Problem Setup

#### 2.2. Motivation for the Energy Design

#### 2.3. Solving the Minimization Problem

## 3. Numerical Implementation

#### 3.1. Numerical Scheme

#### 3.2. Initialization

#### 3.3. Choosing the Edge-Forcing Term

## 4. Results

#### 4.1. Four-Phase Segmentation of Grayscale Images

#### 4.2. Blood Vessel Detection in Medical Images

#### 4.3. Color Image Segmentation

## 5. Conclusions

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Apendix A. Some Additional Comments

#### Appendix A.1. Existence of Minimizers

**Theorem**

**A1.**

**Remark**

**A1.**

**Proof.**

#### Appendix A.2. Remarks about Determining the Initial Values of $\overrightarrow{c}$ from the Histogram of the Image Intensity Values

## Appendix B. Additional Details Regarding Numerical Simulations

#### Appendix B.1. More Examples

**Figure A2.**(

**a**) Original ‘Horse’ image, (

**b**) ground truth boundary information (from [59]), (

**c**) images (

**a**,

**b**) overlaid, (

**d**) our algorithm output, (

**e**) boundaries of the segmented regions, (

**f**) boundaries for the segmented regions overlaid on the original image.

**Figure A3.**(

**a**) Original ‘Plane’ image (from Berkeley Segmentation Data Set 300 [33], (

**b**) ground truth boundary information (also from BSDS300), (

**c**) images (

**a**,

**b**) overlaid, (

**d**) our algorithm output, (

**e**) boundaries of the segmented regions, (

**f**) boundaries for the segmented regions overlaid on the original image.

#### Appendix B.2. Edge Information Used in the Fidelity Terms for the Images from Sections and Appendix B.1

**Figure A4.**Edge information used in the fidelity term for (

**a**) the ‘Peppers’ image with added noise, SNR = 15 dB (

**b**) ‘Leaf’, (

**c**) the ‘Sectors’ image with added noise, SNR = 10 dB, (

**d**) retinal image, (

**e**) ‘Elephants’, (

**f**) ‘Plane’, (

**g**) ‘Horse’.

## References

- Lindeberg, T.; Li, M.X. Segmentation and classification of edges using minimum description length approximation and complementary junction cues. Comput. Vis. Image Underst.
**1997**, 1, 144–151. [Google Scholar] [CrossRef] - Boykov, Y.; Kolmogorov, V. An Overview of Image and Video Segmentation in the last 40 years. In Proceedings of the 6th International Symposium on Signal Processing and Its Applications, Kuala Lumpur, Malaysia, 13–16 August 2001; pp. 144–151. [Google Scholar]
- Pal, S.K.; Ghosh, A.; Shankar, B.U. Segmentation of remotely sensed images with fuzzy thresholding, and quantitative evaluation. Int. J. Remote Sens.
**2000**, 21, 2269–2300. [Google Scholar] [CrossRef] - Roerdink, J.; Meijster, A. The Watershed Transform: Definitions, Algorithms and Parallization Strategies. Fundam. Inform.
**2001**, 41, 187–228. [Google Scholar] - Beucher, S.; Meyer, F. The Morphological Approach to Segmentation: The Watershed Transformation. In Mathematical Morphology in Image Processing; Marcel Dekker Inc.: New York, NY, USA, 1993; pp. 433–481. [Google Scholar]
- Cheng, J.; Rajapakse, J.C. Segmentation of Clustered Nuclei With Shape Markers and Marking Function. IEEE Trans. Biomed. Eng.
**2009**, 56, 741–748. [Google Scholar] [CrossRef] [PubMed] - Zhang, C.; Sun, C.; Su, R.; Pham, T.D. Segmentation of clustered nuclei based on curvature Weighting. In Proceedings of the 27th Conference on Image and Vision Computing New Zealand, Dunedin, New Zealand, 26–28 November 2012; McCane, B., Mills, S., Deng, J.D., Eds.; ACM: New York, NY, USA, 2012; pp. 49–54. [Google Scholar]
- Atta-Fosu, T.; Guo, W.; Jeter, D.; Mizutani, C.; Stopczynski, N.; Sousa-Neves, R. 3D Clumped Cell Segmentation Using Curvature Based Seeded Watershed. J. Imaging
**2016**, 2. [Google Scholar] [CrossRef] [PubMed] - Potts, R.B. Some Generalized Order-Disorder Transformations. Math. Proc.
**1952**, 48, 106–109. [Google Scholar] [CrossRef] - Boykov, Y.; Veksler, O.; Zabih, R. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell.
**2001**, 23, 1222–1239. [Google Scholar] [CrossRef] - Boykov, Y.; Kolmogorov, V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell.
**2004**, 26, 1124–1137. [Google Scholar] [CrossRef] [PubMed] - Appleton, B.; Talbot, H. Globally Minimal Surfaces by Continuous Maximal Flows. IEEE Trans. Pattern Anal. Mach. Intell.
**2006**, 28, 106–118. [Google Scholar] [CrossRef] [PubMed] - Yuan, J.; Bae, E.; Tai, X.; Boykov, Y. A Continuous Max-Flow Approach to Potts Model. In Proceedings of the 11th European Conference on Computer Vision, Heraklion, Greece, 5–11 September 2010; pp. 379–392. [Google Scholar]
- Bae, E.; Yuan, J.; Tai, X.C. Global minimization for continuous multiphase partitioning problems using a dual approach. Int. J. Comput. Vis.
**2011**, 92, 112–129. [Google Scholar] [CrossRef] - Yuan, J.; Bae, E.; Tai, X.; Boykov, Y. A spatially continuous max-flow and min-cut framework for binary labeling problems. Numer. Math.
**2014**, 126, 559–587. [Google Scholar] [CrossRef] - Mumford, D.; Shah, J. Optimal approximation by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math.
**1989**, 42, 557–685. [Google Scholar] [CrossRef] - Caselles, V.; Kimmel, R.; Sapiro, G. Nonlinear total variation based noise removal algorithms. Int. J. Comput. Vis.
**1997**, 22, 61–79. [Google Scholar] [CrossRef] - Vese, L.; Chan, T. Active contours without edges. IEEE Trans. Image Process.
**2001**, 10, 266–277. [Google Scholar] - Vese, L.; Chan, T. A Multiphase Level Set Framework for Image Segmentation Using the Mumford and Shah Model. Int. J. Comput. Vis.
**2002**, 50, 271–293. [Google Scholar] [CrossRef] - Kang, S.; March, R. Multiphase Image Segmentation via Equally Distanced Multiple Well Potential. J. Vis. Commun. Image Represent.
**2014**, 25, 1446–1459. [Google Scholar] [CrossRef] - Bresson, X.; Esedoglu, S.; Vandergheynst, P.; Osher, S. Fast global minimization of the active contour/snake model. J. Math. Imaging Vis.
**2007**, 28, 151–167. [Google Scholar] [CrossRef] - Pock, T.; Schoenemann, T.; Graber, G.; Bischof, H.; Cremers, D. A convex formulation of continuous multi-label problems. In Proceedings of the 10th European Conference on Computer Vision: Part III, Marseille, France, 12–18 October 2008; Springer: Berlin/Heidelberg, Germany, 2008; pp. 792–805. [Google Scholar]
- Pock, T.; Chambolle, A.; Cremers, D.; Bischof, H. A convex relaxation approach for computing minimal partitions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA, 20–25 June 2009; pp. 810–817. [Google Scholar]
- Lellmann, J.; Kappes, J.; Yuan, J.; Becker, F.; Schnörr, C. Convex multi-class image labeling by simplex-constrained total variation. In Proceedings of the International Conference on Scale Space and Variational Methods in Computer Vision, Voss, Norway, 1–5 June 2009; Springer: Berlin, Germany, 2009; Volume 5567, pp. 150–162. [Google Scholar]
- Lellmann, J.; Schnörr, C. Continuous multiclass labeling approaches and algorithms. SIAM J. Imaging Sci.
**2011**, 4, 1049–1096. [Google Scholar] [CrossRef] - Brown, E.S.; Chan, T.F.; Bresson, X. Completely convex formulation of the Chan-Vese image segmentation model. Int. J. Comput. Vis.
**2012**, 98, 103–121. [Google Scholar] [CrossRef] - Papadakis, N.; Aujol, J.-F.; Caselles, V.; Yildizoǧlu, R. High-dimension multi-label problems: Convex or non convex relaxation. SIAM J. Imaging Sci.
**2013**, 6, 2603–2639. [Google Scholar] [CrossRef] - Li, B.; Chui, C.; Chang, S.; Ong, S. Integrating spatial fuzzy clustering with level set methods for automated medical image segmentation. Comput. Biol. Med.
**2011**, 41, 1–10. [Google Scholar] [CrossRef] [PubMed] - Garcia-Cardona, C.; Merkurjev, E.; Bertozzi, A.; Flenner, A.; Percus, A.G. Multiclass Data Segmentation Using Diffuse Interface Methods on Graphs. IEEE Trans. Pattern Anal. Mach. Int.
**2014**, 36, 1600–1613. [Google Scholar] [CrossRef] [PubMed] - Merkurjev, E.; Garcia, C.; Bertozzi, A.L.; Flenner, A.; Percus, A. Diffuse interface methods for multiclass segmentation of high-dimensional data. Appl. Math. Lett.
**2014**, 63, 29–34. [Google Scholar] [CrossRef] - Arbelaez, P.; Maire, M.; Fowlkes, C.; Malik, J. Contour Detection and Hierarchical Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell.
**2011**, 33, 898–916. [Google Scholar] [CrossRef] [PubMed] - Unnikrishnan, R.; Pantofaru, C.; Hebert, M. Toward Objective Evaluation of Image Segmentation Algorithms. IEEE Trans. Pattern Anal. Mach. Intell.
**2007**, 29, 929–944. [Google Scholar] [CrossRef] [PubMed] - Martin, D.; Fowlkes, C.; Tal, D.; Malik, J. A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. In Proceedings of the Eighth IEEE International Conference on Computer Vision (ICCV), Vancouver, BC, Canada, 7–14 July 2001. [Google Scholar]
- Shi, R.; Ngan, K.; Li, S.; Paramesran, R.; Li, H. Visual Quality Evaluation of Image Object Segmentation: Subjective Assessment and Objective Measure. IEEE Trans. Image Process.
**2015**, 24, 5033–5045. [Google Scholar] [CrossRef] [PubMed] - Shi, R.; Ngan, K.; Li, S. The Objective Evaluation of Image Object Segmentation Quality. In Proceedings of the International Conference on Advanced Concepts for Intelligent Vision Systems (ACIVS), Poznan, Poland, 28–31 Octorber 2013. [Google Scholar]
- Dobrosotskaya, J.; Guo, W. A PDE-free variational model for multiphase image segmentation. In Proceedings of the Conference on Wavelets and Sparsity XVI, San Diego, CA, USA, 10–12 August 2015; Volume 9597, p. 959706. [Google Scholar]
- Emmerich, H. Diffuse Interface Approach in Materials Science Thermodynamic Concepts and Applications of Phase-Field Models; Springer: Berlin/Heidelberg, Germany, 2003. [Google Scholar]
- Jung, Y.; Kang, S.H.; Shen, J. Multiphase image segmentation via Modica-Mortola phase transition. SIAM J. Appl. Math.
**2007**, 67, 1213–1232. [Google Scholar] [CrossRef] - Modica, L.; Mortola, S. Un esempio di gamma-convergenza. Boll. Unione Mat. Ital.
**1977**, 5, 285–299. [Google Scholar] - Bertozzi, A.; Esedoglu, S.; Gillette, A. Analysis of a two-scale Cahn-Hilliard model for image inpainting. Multiscale Model. Simul.
**2007**, 6, 913–936. [Google Scholar] [CrossRef] - Esedoglu, S. Blind Deconvolution of Bar Code Signals. Inverse Probl.
**2004**, 20, 121–135. [Google Scholar] [CrossRef] - Wang, M.Y.; Zhou, S.; Ding, H. Non-linear diffusions in topology optimization. Struct. Multidiscip. Optim.
**2004**, 28, 262–276. [Google Scholar] [CrossRef] - Favaro, P.; Osher, S.; Soatto, S.; Vese, L.A. 3D Shape from Anisotropic Diffusion. In Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Madison, WI, USA, 18–20 June 2003; Volmue 1, pp. 179–186. [Google Scholar]
- Perona, P.; Malik, J. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell.
**1990**, I, 629–639. [Google Scholar] [CrossRef] - Dobrosotskaya, J.; Bertozzi, A. Analysis of the wavelet Ginzburg–Landau energy in image applications with edges. SIAM J. Imaging Sci.
**2013**, 6, 698–729. [Google Scholar] [CrossRef] - Dobrosotskaya, J.; Bertozzi, A. Wavelet analogue of the Ginzburg–Landau energy and its Γ-convergence. Interfaces Free Bound.
**2010**, 12, 497–525. [Google Scholar] [CrossRef] - Chan, T.F.; Shen, J.; Zhou, H.M. Total variation wavelet inpainting. J. Math. Imaging Vis.
**2006**, 25, 107–125. [Google Scholar] [CrossRef] - Elad, M.; Starck, J.L.; Querre, P.; Donoho, D.L. Simultaneous cartoon texture image inpaitning using morphological component analysis. Appl. Comput. Harmon. Anal.
**2005**, 19, 340–358. [Google Scholar] [CrossRef] - Chambolle, A.; DeVore, R.A.; Lee, N.; Lucier, B.J. Nonlinear Wavelet Image Processing: Variational Problems, Compression, and Noise Removal through Wavelet Shrinkage. IEEE Trans. Image Process.
**1998**, 7, 319–333. [Google Scholar] [CrossRef] [PubMed] - Cai, J.F.; Dong, B.; Osher, S.; Shen, Z. Image Restoration: Total Variation, Wavelet Frames, and Beyond. J. Am. Math. Soc.
**2012**, 25, 1033–1089. [Google Scholar] [CrossRef] - Chambolle, A.; Lions, P.L. Image recovery via total variation minimization and related problems. Numer. Math.
**1997**, 76, 167–188. [Google Scholar] [CrossRef] - Rudin, L.I.; Osher, S.; Fatemi, E. Nonlinear Total Variation based noise removal algorithms. Physics D
**1992**, 60, 259–268. [Google Scholar] [CrossRef] - Maso, G.D. An Introduction to Gamma Convergence. Progress in nonlinear Differential Equations and Their Applications; Birkhauser Boston, Inc.: Boston, MA, USA, 1993. [Google Scholar]
- Eyre, D.J. An Unconditionally Stable One-Step Scheme for Gradient Systems. 1997. Available online: https://pdfs.semanticscholar.org/4f01/6a98fe25bfc06b9bcab3d85eeaa47d3ad3ca.pdf (accessed on 4 July 2017).
- Mallat, S. Wavelet Tour of Signal Processing; Academic Press: London, UK, 1999. [Google Scholar]
- Staal, J.; Abramoff, M.; Niemeijer, M.; Viergever, M.; van Ginneken, B. Ridge based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging
**2004**, 23, 501–509. [Google Scholar] [CrossRef] [PubMed] - Dobrosotskaya, J.; Ehler, M.; King, E.; Bonner, R.; Czaja, W. Modeling of the rhodopsin bleaching with variational analysis of retinal images. SPIE Med. Imaging
**2011**, 7962. [Google Scholar] [CrossRef] - Czaja, W.; Dobrosotskaya, J.; Manning, B. Composite wavelet representations for reconstruction of missing data. Proc. SPIE
**2013**, 8750. [Google Scholar] [CrossRef] - Ren, X.; Fowlkes, C.C.; Malik, J. Cue Integration for Figure/Ground Labeling. In Proceedings of the 18th International Conference on Neural Information Processing Systems (NIPS’05), Vancouver, BC, Canada, 5–8 December 2005; MIT Press: Cambridge, MA, USA, 2005; pp. 1121–1128. [Google Scholar]

**Figure 2.**More details about the above MRI image segmentation and the gradient descent setup: (

**a**) the edges to be preserved, (

**b**) initial guess for ${\varphi}_{1}$, (

**c**) initial guess for ${\varphi}_{2}$ (percentiles used for computing the initial guess: $15\%,\phantom{\rule{3.33333pt}{0ex}}38\%,\phantom{\rule{3.33333pt}{0ex}}61\%,\phantom{\rule{3.33333pt}{0ex}}85\%$).

**Figure 3.**(

**a**) the overall segmentation output of the proposed method, (

**b**) output ${\varphi}_{1}$, (

**c**) output ${\varphi}_{2}$.

**Figure 5.**Segmentation of the ‘Leaf’ image: (

**a**) the original image; the segmented outputs of (

**b**) the proposed method, (

**c**) graph cut method, (

**d**) Vese–Chan and (

**e**) fuzzy segmentation methods.

**Figure 6.**Compare the segmentation results of (

**a**) the peppers image with added Gaussian noise (SNR = 15 dB) using (

**b**) the proposed method, (

**c**) graph cut, (

**d**) Vese–Chan and (

**e**) fuzzy segmentation methods.

**Figure 8.**Each row ((

**a**) to (

**c**)) shows the input image, the minimization output and the thresholded (rounded) output (from left to right) for different levels of additive noise: (

**a**) noise = 15 dB, >99% classified correctly, (

**b**) noise = 10 dB, >98% classified correctly, (

**c**) noise = 5 dB, >95% classified correctly.

**Figure 9.**Comparison to other methods: (

**a**) the input image (noisy ‘Sectors’ images); segmentation output of (

**b**) the proposed method, (

**c**) graph cut method, (

**d**) Vese–Chan method, (

**e**) fuzzy segmentation method, (

**f**) k-means.

**Figure 10.**(

**a**) Original image (from BSDS500), (

**b**) ground truth boundaries obtained by manual segmentation (from BSDS500), (

**c**) ground truth boundaries shown on the original, (

**d**) the result of the proposed four-class segmentation method, (

**e**) boundaries of the regions shown in (

**d**,

**f**) boundaries of the segmented regions shown on the original image.

**Figure 11.**(

**a**) Original image, (

**b**) image after background removal (

**c**) segmented output, (

**d**) segmented output with post-processing.

**Figure 12.**(

**a**) Original image, (

**b**) k-means segmentation that is used to initialize ${\varphi}_{1}$ and ${\varphi}_{2}$ (

**c**) segmentation output.

SNR/Benchmark | Rand Index | Variation of Information | Segmentation Covering |
---|---|---|---|

15 db | 0.9881 | 0.1903 | 0.9763 |

10 db | 0.9846 | 0.2382 | 0.9716 |

7 db | 0.9807 | 0.2991 | 0.9618 |

5 db | 0.8901 | 0.5099 | 0.9533 |

3 db | 0.9377 | 0.7165 | 0.8759 |

**Table 2.**Segmentation evaluation for several methods applied to the ‘Sectors’ image with 10 dB noise.

Method | Rand Index | Variation of Information | Segmentation Covering |
---|---|---|---|

Proposed method | 0.9846 | 0.2382 | 0.9716 |

Fuzzy | 0.7253 | 2.3699 | 0.4622 |

Vese–Chan | 0.8901 | 1.1218 | 0.774 |

Graph Cuts | 0.7523 | 1.1897 | 0.2711 |

k-means | 0.7081 | 2.6804 | 0.4137 |

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Dobrosotskaya, J.; Guo, W.
A PDE-Free Variational Method for Multi-Phase Image Segmentation Based on Multiscale Sparse Representations. *J. Imaging* **2017**, *3*, 26.
https://doi.org/10.3390/jimaging3030026

**AMA Style**

Dobrosotskaya J, Guo W.
A PDE-Free Variational Method for Multi-Phase Image Segmentation Based on Multiscale Sparse Representations. *Journal of Imaging*. 2017; 3(3):26.
https://doi.org/10.3390/jimaging3030026

**Chicago/Turabian Style**

Dobrosotskaya, Julia, and Weihong Guo.
2017. "A PDE-Free Variational Method for Multi-Phase Image Segmentation Based on Multiscale Sparse Representations" *Journal of Imaging* 3, no. 3: 26.
https://doi.org/10.3390/jimaging3030026