Image Segmentation with a Priori Conditions: Applications to Medical and Geophysical Imaging

Abstract

In this paper, we propose a method for semi-supervised image segmentation based on geometric active contours. The main novelty of the proposed method is the initialization of the segmentation process, which is performed with a polynomial approximation of a user defined initialization (for instance, a set of points or a curve to be interpolated). This work is related to many potential applications: the geometric conditions can be useful to improve the quality the segmentation process in medicine and geophysics when it is required (weak contrast of the image, missing parts in the image, non-continuous contour…). We compare our method to other segmentation algorithms, and we give experimental results related to several medical and geophysical applications.

1. Introduction

The problem of segmenting an image into its significant components has been studied for over 30 years in computers science, applied mathematics and more generally in computer vision.

Recently, from convolutional neural networks (see Fukushima [1], Waibel et al. [2], and LeCun et al. [3], among others…) to recurrent neural networks (Hochreiter and Schmidhuber [4]), encoder–decoders (Badrinarayanan et al. [5]), or generative adversarial networks (Goodfellow et al. [6]), deep learning based techniques have achieved huge successes in the field of artificial intelligence and image segmentation (see [7] for a recent survey). These approaches leads to excellent results, including in medical applications (see for example Fantazzini et al. [8]). However, in general, the performance heavily depends on labeled data, and this point is a main difficulty on several applications when lacking labeled data such as in many medical and geophysical applications. The data augmentation is possible [9], but it is often complicated, and requires more CPU/GPU time than other segmentation techniques based on energy minimization or variational approaches (Kass et al. [10], Chand and Vese [11], Mumford and Shah [12], Vese and Le Guyader [13]…) or geometrical ones (fast marching methods, see Sethian [14] or Forcadel et al. [15]).

In this work, we focus on a modeling based on a variational approach: it basically consists of evolving an initial contour subject to constraints towards the boundary of the object to be detected. This deformation is done by minimizing a functional depending on the curve and defined so that a local minimum is obtained at the boundary of the object. This energy-like functional minimization problem has led to many research works in the last 25 years. Caselles, Kimmel and Sapiro [16] have shown that by setting one of the regularization parameters to zero in the classical active contour model, one gets a problem equivalent to finding a geodesic curve in a Riemann space whose metric depends on the image contents. The issue was then no longer seen as an energy-like minimization problem but as a curve evolution one. Let us note that this approach is very suitable to our problems, and it could be possible to use deep learning (DL) based algorithm (as used by Le Guyader et al. [17]), but the a priori conditions are complicated to integrate in such DL models (and such DL models require a lot of hardware, much more that our proposed algorithm).

An edge can be viewed as the locus of connected points for which the image gradient varies abruptly. However, when data acquisition cannot be performed in an optimal manner (e.g., liver in medical imaging), this criterion can no longer be applied. In many applications, image data are missing or of poor quality, and occultation phenomena can often occur: two objects that are adjacent to one another can own intrinsic homogeneous textures so that it is hard to clearly identify the interface between them. It is then relevant to introduce geometric constraints in the modeling to help the segmentation process. In [18,19,20,21], the authors introduce the concept of a finite set of points to be interpolated during the segmentation process (like in Gout and Le Guyader [20]) or a variational approach to integrate higher-level shape priors into level set based segmentation method (see Cremers et al. [18]). However, in some applications (in geophysics or in several medical applications), this is not sufficient and adding new a priori conditions to be satisfied is required both to improve the segmentation process and also to simplify the initial condition (in [20], the method requires a priori conditions plus an initial guess to start the segmentation process, which constitutes a drawback of their algorithm).

In this work, we propose to consider a curve (or a set of curves) belonging to the searched contour of interest, integrated as an initial condition to be satisfied by the final segmentation contour obtained at the end of the segmentation process. This is an added value considering previous models (like [19]), where it was necessary to both give an initial condition and geometric conditions. In order to define this curve, it is possible to use $D^m$-spline functions (see Gout et al. [22]) obtained by minimizing an energy functional on a suitable Hilbert space (and so a set of points given by the user is sufficient to create such geometric conditions).

In this paper, we generalize and improve the approach of Gout and Le Guyader [20]: we combine the curve evolution approach developed by Caselles, Kimmel and Sapiro [16], with a geometrical approach consisting in a curve given by the user (a set of points can also be considered like in Gout and Le Guyader [20]), and the initial condition is then automatically generated (from the geometric conditions).

In Section 2, we give the modeling of our problem, based on a level set method (LSM) approach (Osher and Sethian [23], see also [24]). Using Euler-Lagrange variational principle, we obtain the PDE satisfied by the 3D LSM function $\mathsf{\Phi }$ and the corresponding evolution problem is then given. The existence and uniqueness of the viscosity solution of the associated parabolic problem is also established: we give the main steps of the proof (based on Caselles et al. [16] and Gout and Le Guyader [20]). The discretization of the problem is tackled in Section 3, and is solved using an Additive Operator Splitting scheme (Weickert and Kühne [25]). Comparisons and experimental results with the proposed approach are given for several medical and geophysical imaging applications.

2. Mathematical Modelling

The well-known level set approach (see Osher and Sethian [23]) consists in considering the evolving active contour $\mathsf{\Gamma }=\mathsf{\Gamma }\left(t\right)$ as the zero level set of a function $\mathsf{\Phi }$, which is a Lipschitz continuous function defined by:

$$\left\{\begin{array}{cccc}\hfill \mathsf{\Phi }:& \mathsf{\Omega }\times [0,+\infty [& ⟶& \mathrm{I}\mathrm{R},\\ \hfill & (x,t)& ⟼& \mathsf{\Phi }(x,t),\end{array}\right.$$

(1)

such that:

$$\mathsf{\Gamma }\left(t\right)=\{x\in \mathsf{\Omega }\mid \mathsf{\Phi }(x,t)=0\},$$

(2)

and $\mathsf{\Phi }(·,t)$ takes opposite signs on each side of $\mathsf{\Gamma }\left(t\right)$. This level set approach enables us to re-write the energy in terms of $\mathsf{\Phi }$:

$$E\left(\mathsf{\Phi }\right)={\int }_{\mathsf{\Omega }}g\left(\right|\nabla I\left(x\right)\left|\right)|\nabla H\left(\mathsf{\Phi }\left(x\right)\right)|dx,$$

(3)

where H is the one-dimensional Heaviside function and where $g\left(s\right)={\displaystyle \frac{1}{1+s^2}}$. We formulate the shape optimization problem as follows: “Find a shape such that the energy E is minimized”:

$$\left\{\begin{array}{c}\mathrm{Search}\mathrm{for}\mathsf{\Phi }\mathrm{such}\mathrm{that}:\hfill \\ E\left(\mathsf{\Phi }\right)=\underset{\xi }{min}E\left(\xi \right).\hfill \end{array}\right.$$

(4)

The functional E being not Gâteaux-differentiable, we regularize the problem by considering slightly regularized versions ($C^1$ or $C^2$ regularization) of the functions H and $\delta$ (the one dimensional Dirac measure) denoted, respectively, by $H_{ϵ}$ and ${\delta }_{ϵ}$, and considering that ${\delta }_{ϵ}=H_{ϵ}^{\prime }$, the associated regularized functional $E_{ϵ}$ becomes:

$$E_{ϵ}\left(\mathsf{\Phi }\right)={\int }_{\mathsf{\Omega }}g\left(\right|\nabla I\left(x\right)\left|\right){\delta }_{ϵ}\left(\mathsf{\Phi }\left(x\right)\right)|\nabla \mathsf{\Phi }\left(x\right)|dx,$$

(5)

where:

$${\int }_{\mathsf{\Omega }}{\delta }_{ϵ}\left(\mathsf{\Phi }\left(x\right)\right)|\nabla \mathsf{\Phi }\left(x\right)|dx$$

(6)

is an approximation of the length of the zero level set of $\mathsf{\Phi }$.

2.1. A Priori Conditions

We now propose to include a priori condition in the process. We consider the a priori condition as a smooth arc curve $\mathcal{C}$ given by the user. The curve $\mathcal{C}$ is parametrised by $c\in C^1([0,1],\mathsf{\Omega })$. By construction, we impose that the curve $\mathcal{C}$ belongs to the level set $\{{\mathsf{\Phi }}_0=0\}$. The first step consists in constructing the initial condition from the geometric conditions given by the user. We have chosen this option, because it permits to avoid the stage of giving an initial condition plus the a priori conditions.

The first step consists in closing the arc to get a closed curve. We introduce a point $a\in \mathsf{\Omega }$ and a vector $v=\left(v_1,v_2\right)\in \mathrm{I}\mathrm{R}$${}^2.$ It is easy to construct two (unique) polynomials curves ${\mathcal{C}}_1$ and ${\mathcal{C}}_2\subset \mathsf{\Omega }$ that are parameterized using $c_1$ and $c_2\in P_3\left(\left[0,1\right],\mathsf{\Omega }\right)$ such that:

$$\begin{array}{ccc}c\left(1\right)=c_1\left(0\right),& c_1\left(1\right)=c_2\left(0\right)=a,& c_2\left(1\right)=c\left(0\right),\\ -c^{\prime }\left(1\right)=c_1^{\prime }\left(0\right),& c_1^{\prime }\left(1\right)=-c_2^{\prime }\left(0\right)=v,& c_2^{\prime }\left(1\right)=-c^{\prime }\left(0\right).\end{array}$$

(7)

We set ${\mathsf{\Gamma }}_0=\mathcal{C}\cup {\mathcal{C}}_1\cup {\mathcal{C}}_2,$ and we consider ${\mathsf{\Phi }}_0$ as the signed distance to ${\mathsf{\Gamma }}_0:$

$${\mathsf{\Phi }}_0\left(x\right)=\left\{\begin{array}{cc}\hfill d(x,{\mathsf{\Gamma }}_0)& \mathrm{if}x\in Int\left({\mathsf{\Gamma }}_0\right),\hfill \\ \hfill -d(x,{\mathsf{\Gamma }}_0)& \mathrm{otherwise}.\hfill \end{array}\right.$$

(8)

To define the energy $E_C,$ we also introduce the mask $\eta$: $\mathsf{\Omega }\to$IR:

$$\eta \left(x\right)=exp\left(-{\displaystyle \frac{d{(x,C)}^2}{\sigma }}\right):\forall x\in \mathsf{\Omega },$$

(9)

where $\sigma \in$IR${}^{+},$ is a positive parameter which controls the width of the mask. Then, we can define the following energy functional:

$$E_C\left(\mathsf{\Phi }\right)={\int }_{\mathsf{\Omega }}\eta \left(x\right){\left(\mathsf{\Phi }\left(x\right)-{\mathsf{\Phi }}_0\left(x\right)\right)}^{2}dx.$$

(10)

The goal is to minimize the $L^2$ norm between $\mathsf{\Phi }$ and ${\mathsf{\Phi }}_0$: we impose the contour to be close to ${\mathsf{\Gamma }}_0.$ The mask permits to impose such influence in (and only in) a local neighborhood of $\mathcal{C}$.

2.2. Minimization Problem and Evolution Equation

The modeling of our problem consists in minimizing the energy:

$$E\left(\mathsf{\Phi }\right)=E_{ϵ}\left(\mathsf{\Phi }\right)+\frac{\alpha }{2}{E}_C\left(\mathsf{\Phi }\right)$$

(11)

with $\alpha >0.$

We now give the evolution problem. We minimize $E_{ϵ}$ with respect to $\mathsf{\Phi }$ and deduce the associated Euler–Lagrange equation for $\mathsf{\Phi }$:

$$E^{\prime }\left(\mathsf{\Phi }\right)={\delta }_{ϵ}\left(\mathsf{\Phi }\right)div\left(g\left(\right|\nabla I\left|\right)\frac{\nabla \mathsf{\Phi }}{|\nabla \mathsf{\Phi }|}\right)+\alpha \eta \left(\mathsf{\Phi }-{\mathsf{\Phi }}_0\right)$$

(12)

with the boundary conditions (homogeneous Neumann):

$${\displaystyle \frac{{\delta }_{ϵ}\left(\mathsf{\Phi }\right)}{\left|\nabla \mathsf{\Phi }\right|}}{{\displaystyle \frac{\partial \mathsf{\Phi }}{\partial \nu }}}_{\mid \partial \mathsf{\Omega }}=0,$$

(13)

$\nu$ denoting the exterior normal to the boundary of $\mathsf{\Omega }.$

When a local minimum is reached, then the quantity ${\displaystyle \frac{\partial \mathsf{\Phi }}{\partial t}}$ tends to 0, which means that the steady state is reached. A rescaling can be made so that the motion is applied to all level sets by replacing ${\delta }_{ϵ}$ by $|\nabla \mathsf{\Phi }|$ (It makes the flow independent of the scaling of $\mathsf{\Phi }$).

Let us also note that the speed of convergence of the model can be increased by adding the component $\beta g\left(\left|\nabla I\right|\right)|\nabla \mathsf{\Phi }|$ in the evolution equation, $\beta$ being a constant. This component can be seen as an area constraint. An analogy with the Balloon model developed by Cohen [26] can be made: this constant motion term prevents the curve from stopping on a non-significative local minimum and is also of importance when initializing the process with a curve inside the object to be detected.

Thus, the proposed parabolic problem with the associated boundary conditions ${\displaystyle \frac{\partial \mathsf{\Phi }}{\partial \nu }}=0$ can be written:

$$\left\{\begin{array}{cc}\mathsf{\Phi }(x,0)\hfill & ={\mathsf{\Phi }}_0\left(x\right),\hfill \\ {\displaystyle \frac{\partial \mathsf{\Phi }}{\partial t}}=\hfill & |\nabla \mathsf{\Phi }|\left[div\left(g\right(|\nabla I|){\displaystyle \frac{\nabla \mathsf{\Phi }}{|\nabla \mathsf{\Phi }|}})\right]+\alpha \eta \left(\mathsf{\Phi }-{\mathsf{\Phi }}_0\right)\hfill \\ \hfill & +\beta g\left(\left|\nabla I\right|\right)|\nabla \mathsf{\Phi }|,\hfill \\ {\displaystyle \frac{\partial \mathsf{\Phi }}{\partial \nu }}=\hfill & 0\mathrm{on}\partial \mathsf{\Omega }.\hfill \end{array}\right.$$

(14)

2.3. Existence, Uniqueness of the Solution

This section is a theoretical part showing the existence and uniqueness of the solution of our segmentation model. We can show the existence and uniqueness of a viscosity solution to the evolution problem (14). We rewrite our problem as:

$$\left\{\begin{array}{cccc}\hfill u_t+\tilde{F}(t,x,u,Du,D^{2}u)& =& 0\hfill & in]0,T[\times \mathsf{\Omega }\hfill \\ \hfill B(x,Du)& =& 0\hfill & in]0,T[\times \partial \mathsf{\Omega },\hfill \end{array}\right.$$

(15)

where, for $(t,x,u,p,X)\in [0,T]\times \overline{\mathsf{\Omega }}\times \mathrm{I}{\mathrm{R}}^2\times {\mathcal{S}}^2$:

$$\tilde{F}(t,x,u,p,X)=F(t,x,p,X)+g(x,u),B(x,p)=p·\overrightarrow{n},$$

(16)

and:

$$\begin{array}{ccc}\hfill F(t,x,p,X)& =& -trace\left(g\left(\right|\nabla I\left(x\right)\left|\right)\left(I-\frac{p\otimes p}{{\left|p\right|}^2}\right)X\right)\hfill \\ \hfill & & -\langle \nabla \left(g\right(|\nabla I(x\left)\right|\left)\right),p\rangle \hfill \\ \hfill g(x,u)& =& \alpha \eta \left(x\right)(u-{\mathsf{\Phi }}_0\left(x\right)).\hfill \end{array}$$

(17)

We assume that $\mathsf{\Omega }$ is a bounded domain in $\mathrm{I}{\mathrm{R}}^{\mathrm{n}}$ with a $C^1$ boundary. Under the following conditions, the Equation (15) has a unique viscosity solution.

• $\tilde{F}\in C([0,T]\times \overline{\mathsf{\Omega }}\times \mathrm{I}\mathrm{R}\times (\mathrm{I}{\mathrm{R}}^{\mathrm{n}}-\left\{0\right\})\times {\mathrm{S}}^{\mathrm{n}}),$ where $S^n$ denotes the space of $n\times n$ symmetric matrices equipped with the usual ordering.
• There exists a constant $\gamma \in \mathrm{I}\mathrm{R}$ such that for each:

  $$(t,x,p,X)\in [0,T]\times \overline{\mathsf{\Omega }}\times (\mathrm{I}{\mathrm{R}}^{\mathrm{n}}-\left\{0\right\})\times {\mathrm{S}}^{\mathrm{n}},$$

  the function:

  $$u\mapsto \tilde{F}(t,x,u,p,X)-\gamma u$$

  (18)

  is non-decreasing on $\mathrm{I}\mathrm{R}$.
• For each $R>0$, there exists a continuous function $w_R:[0,\infty [⟶[0,\infty [$ satisfying $w_R\left(0\right)=0$ such that if $X,Y\in S^n$ and ${\mu }_1,{\mu }_2\in [0,\infty [$ satisfy:

  $$\left(\begin{array}{cc}X& 0\\ 0& Y\end{array}\right)\le {\mu }_1\left(\begin{array}{cc}I& -I\\ -I& I\end{array}\right)+{\mu }_2\left(\begin{array}{cc}I& 0\\ 0& I\end{array}\right)$$

  (19)

  then:

  $$\begin{array}{c}\tilde{F}(t,x,u,p,X)-\tilde{F}(t,y,u,q,-Y)\hfill \\ \ge -w_R({\mu }_1{\left(\right|x-y|}^2+\rho {(p,q)}^2)\hfill \\ +{\mu }_2+|p-q|+|x-y|(1+max(\left|p\right|,\left|q\right|\left)\right)),\hfill \end{array}$$

  (20)

  for all $t\in \left[0,T\right],x,y\in \overline{\mathsf{\Omega }},u\in \mathrm{I}\mathrm{R}$ with $\left|u\right|\le R$ and $p,q\in \mathrm{I}{\mathrm{R}}^{\mathrm{n}}\setminus \left\{0\right\}$.
• $B\in C(\mathrm{I}{\mathrm{R}}^{\mathrm{n}}\times \mathrm{I}{\mathrm{R}}^{\mathrm{n}})\bigcap {\mathrm{C}}^{1,1}(\mathrm{I}{\mathrm{R}}^{\mathrm{n}}\times (\mathrm{I}{\mathrm{R}}^{\mathrm{n}}\setminus \left\{0\right\})).$
• For each $x\in \mathrm{I}{\mathrm{R}}^{\mathrm{n}}$, the function $p\mapsto B(x,p)$ is positively homogeneous of degree one in p, i.e., $B(x,\lambda p)=\lambda B(x,p)$, $\forall \lambda \ge 0,p\in \mathrm{I}{\mathrm{R}}^{\mathrm{n}}\setminus \left\{0\right\}$.
• There exists a positive constant $\theta$ such that $<\nu \left(z\right),D_{p}B(z,p)>\ge \theta$ for all $z\in \partial \mathsf{\Omega }$ and $p\in \mathrm{I}{\mathrm{R}}^{\mathrm{n}}-\left\{0\right\}$. Here, $\nu \left(z\right)$ denotes the unit outer normal vector of $\mathsf{\Omega }$ at $z\in \partial \mathsf{\Omega }.$

All those conditions are clearly satisfied for $\tilde{F}=F$, and B (see for instance [20]). We here have to extend the proof to $\tilde{F}=F+g$.

• Thanks to [20], we already know that $F\in C([0,T]\times \overline{\mathsf{\Omega }}\times (\mathrm{I}{\mathrm{R}}^{\mathrm{n}}\setminus \left\{0\right\})\times {\mathcal{S}}^{\mathrm{n}})$. Since, $\eta ,{\mathsf{\Phi }}_0\in C\left(\overline{\mathsf{\Omega }}\right)$, it is clear that $g\in C(\overline{\mathsf{\Omega }}\times \mathrm{I}\mathrm{R})$ and therefore $\tilde{F}\in C([0,T]\times \overline{\mathsf{\Omega }}\times \mathrm{I}\mathrm{R}\times (\mathrm{I}{\mathrm{R}}^{\mathrm{n}}\setminus \left\{0\right\})\times {\mathcal{S}}^{\mathrm{n}}$.
• Note that since F does not depend on u. It suffices to show that $u\to g(x,u)-\gamma u$ is non-decreasing on $\mathrm{I}\mathrm{R}$. On can easily see that the condition is satisfied for all $x\in \overline{\mathsf{\Omega }}$ as soon as $\gamma <0$.
• Since g does not depend on X, it suffices to check the condition on F. We refer the reader to Gout and Le Guyader [20] for the proof.

The proof for conditions 4–6 are basic, and can be found for instance in [20]. Therefore, all the conditions are satisfied and the evolution problem (14) has a unique viscosity solution.

3. Experimental Results

For the discretization scheme, we use the additive operator splitting scheme (AOS) proposed by Weickert and Kuhne ([25], see also [20]). This corresponding algorithm requires a computational cost linear in $M\times N$ (size of the unknown vector $\mathsf{\Phi }$) at each step. This scheme is well-suited to our problem that differs only in the introduction of the function $\eta \left(x\right)$.

We use the method proposed in this paper knowing that:

–

Following the considered application, the initial condition can be a set of points, or a curve (then can be constructed from a set points using a basic spline function (see Gout et al. [22])).

–

The stop criterion can be either a preset number of iterations or a check that the solution is stationary.

–

The distance is normalized in order to have the same weight between a priori information of the image and geometrical constraints.

–

The discretization is made using finite differences as done in Chan and Vese [11].

–

In the numerical examples, we take $\delta t=0.1$, the regularization term is equal to 0.8.

To illustrate the interest of our approach, we start with a simple test case, using a three-band image. Then, we run the method on blood vessel image. We obtain the following results. Without a priori condition, the segmentation process will of course give the entire white band as a result. This is the interest of adding such a priori condition to enforce the contour to arbitrary stop following the choice of the user.

We can see that the final contour correctly takes into account the curve given by the user imposing a constraint independent of the pixel values of the images, and correctly segment the white zone.

Of course, our approach allows us (to help) the segmentation process on “complicated” images, when ground truth informations are required like in Figure 1 or in many other cases, especially in geophysical or medical imaging. In Section 3.1, we illustrate the added value brought by our approach considering the initial guess choice improvement. Two examples are given to show that, compared to Gout et al. method [19], the approach given in this paper is much faster to get the same result since the initial guess is (strongly) optimized.

In Section 3.2, we propose to compare our approach with well-known algorithms (Chan-Vese [11], U-Net [27]).

In Section 3.3 and Section 3.4, we give experimental results in medicine and geophysics.

3.1. Impact of the Initial Guess on the Segmentation Process

In this part, we focus on the added value brought by the automatic initial guess we propose, obtained from the geometrical condition given by the user. We compare our approach to the one introduced by Gout, Le Guyader and Vese [19], we take the same geometrical conditions, and then compare the number iteration to get the same final contour. The comparison is relevant because the segmentation algorithm is analogous in the two considered methods. In [19], note that we have to add a closed contour to the geometric conditions to start the segmentation process, while our proposed approach leads to an automatic initial guess (obtained from the geometric conditions given by the user). In the following two numerical examples, we take exactly the same interpolation conditions (3 points).

In Figure 2, Figure 3 and Figure 4, we can see that our approach requires 5 times less iterations.

We repeat the same process in the following example (Figure 5, Figure 6 and Figure 7). The initial contour for Gout et al. [19] algorithm is here chosen outside the region of interest. Of course, we take exactly the same interpolation conditions (3 points) in both cases.

In the example given on Figure 5, Figure 6 and Figure 7, we can see that our approach requires 30 times less iterations.

In this subsection, we illustrate the gain of the automatic generated initial guess we propose from the geometric conditions (see

Table 1. Gain of the automatic generated initial guess.

Method	Example 1	Example 2
Gout et al. [19]	420 iterations	260 iterations
Our method	80 iterations	8 iterations

).

To get the same final contour, our method clearly requires much less iterations. This result is analogous on all the tests we have done, especially when a closed contour is chosen outside the region of interest (in [19]), in this case, the total number of iterations is reduced by a factor of 5 to 60. If the initial contour in [19] is chosen inside the region of interest, the total number of iterations is reduced by a factor of 4 to 20. The only cases where the number of iterations is equivalent between the two different methods are when we choose as initial guess in [19] a closed curve very close to the final result and interpolating at least 2 given interpolation conditions, this is of course a major constraint.

3.2. Quantitative Performance

We use the BraTS Dataset [28], which was collected and shared by the MICCAI Brain Tumor Segmentation Challenge (see [28] for more details) several years ago. We use 270 MR scans, each with four MIR sequences: T1-weighted (native image, sagittal or axial 2D acquisitions with 1–6 mm slice thickness), T1-weighted with contrast-enhanced (Gadolinium) image, T2-weighted image (axial 2D acquisition, with 2–6 mm slice thickness), and FLAIR (T2-weighted FLAIR image, axial, coronal, or sagittal 2D acquisitions, 2–6 mm slice thickness) (see [28] for more details). The training data have the size $240\times 240\times 155$ pixels, and have the manual segmentation labels for many types of brain tumors. We trained the network to generate segmentation masks corresponding to the center slice of the input. We used the well-known Adam optimization method [29]. We trained the deep network using 238 training data, we set the initial learning rate as ${10}^{-4}$, and multiplied by 0.5 after every 20 epochs. Using a single GPU, we trained the models with batch size 4 for 40 epochs.

In order to evaluate the performance of our method, we give comparisons between several algorithms. We here recall several metrics:

–

The Jaccard index [30], or Intersection over Union (IoU), is a commonly used metric in segmentation. It is defined as the area of intersection between the predicted segmentation map and the ground truth, divided by the area of union between the predicted segmentation map and the ground truth:

$$\mathrm{IoU}=J(G,P)={\displaystyle \frac{\left|G\cap P\right|}{\left|G\cup P\right|}},$$

(21)

where G is the ground truth and P denotes the predicted segmentation maps. It ranges between 0 and 1. Mean-IoU (mIoU) is defined as the average IoU over all classes. It is widely used in reporting the performance of segmentation algorithms.

–

The Dice coefficient (Dice) is a popular metric for image segmentation, especially in medical imaging. This coefficient can be defined as twice the overlap area of predicted and ground-truth maps, divided by the total number of pixels in both images:

$$\mathrm{Dice}={\displaystyle \frac{2\left|G\cap P\right|}{\left|G\right|+\left|P\right|}}$$

(22)

When applied to binary segmentation maps, and referring to the foreground as a positive class, the Dice coefficient is essentially identical to the F1 score:

$$\mathrm{Dice}={\displaystyle \frac{2TP}{2TP+FP+FN}}=\mathrm{F}1,$$

(23)

(The F1 score, which is defined as the harmonic mean of precision (Prec) and recall (Rec): $\mathrm{F}1={\displaystyle \frac{2\mathrm{Prec}\mathrm{Rec}}{\mathrm{Prec}+\mathrm{Rec}}},$ where Prec = ${\displaystyle \frac{TP}{TP+FP}}$ and Rec = ${\displaystyle \frac{TP}{TP+FN}}.$), where TP refers to the true positive fraction, FP refers to the false positive fraction, and FN refers to the false negative fraction.

–

The Hausdorff distance (Hd) evaluates the quality of the segmentation boundaries by computing the maximum distance between the prediction and its ground truth:

$$\mathrm{Hd}=max\left\{\underset{p\in \partial P}{sup}\underset{z\in \partial G}{inf}{∥p-z∥}_2,\underset{z\in \partial G}{sup}\underset{p\in \partial P}{inf}{∥p-z∥}_2\right\}$$

(24)

where $\partial P$ and $\partial G$ are the surface point sets of P (segmentation prediction) and G (ground truth).

Of course, a large Dice coefficient or a small HD means an accurate segmentation result. Figure 8 shows several numerical example (here, we ramdomly sampled 33% of BRATS labeled dataset). In Figure 8, we give the dice score obtained for U-Net and our proposed algorithm. We can see that our approach clearly gives better results.

We also compare with the U-net algorithm ([27], see also [31]), tables show the evaluation results, please note that we have suppressed the best and worst 10% of the results, and that for our proposed approach, we have considered only 2 to 4 points as geometric conditions (the initial guess -curve- is automatically constructed from these points), all that leads to make the global results quite homogeneous. We also compare with the Chan-Vese algorithm [11]: in this case, the initial condition is a closed contour located inside the region of interest.

If we use 100% of the BraTS dataset, we obtain the

Table 2. Accuracies of segmentations models on a third of labeled data from BraTS Dataset [28]. We use a Nvidia GeForce RTX 2080 Ti 11G Turbo Edition (Boost frequence: 1545 MHz, GPU memory: 11 GB).

Method	mIoU	Dice	Hd	GPU Time
U-Net [27]	78.3	87.7	43.5	4.45
Chan-Vese [11]	77.6	88.1	41.5	2.02
Our method	79.6	89.1	39.5	2.12

.

We see that the GPU time is clearly better with our algorithm (and the Chan-Vese one too). The quality is globally equivalent: a little better with our algorithm when not considering the entire database, and equivalent if not. This is an important point because it validates our algorithm when we compare with MR scans segmented with deep learning algorithms with few labeled data.

3.3. Applications to Medical Imaging

Here, we apply our method to 2D medical images, the goal being to outline the cross-sectional area of a great thoracic vessel, namely the main pulmonary artery, in order to non-invasively assess pulmonary arterial hypertension. Figure 9 refers to a slice set perpendicularly to the MPA axis (arrow) of a 78 years old patient, who was suspected of pulmonary arterial hypertension, suffering from breathlessness. The presented magnitude image was acquired from a velocity encoded sequence (Venc = 1.5 m/s), during a 5 min time duration, with a 30 ms temporal resolution (1T Magnetom Expert imager; Siemens; Germany). The image quality is moderate, and in particular the vessel wall is irregular around the arrow, that is, not anatomically possible, and therefore an interpolation in the outlining by the physician is needed.

Since the lack of annotated data is one of the most common limitations encountered in many deep learning approaches like the ones we will study in this subsection, deep leaning approaches are not suitable (at least for now!) for our considered applications.

An example of the segmentation of a pulmonary artery with a curve to interpolate is given in Figure 10. A part of this artery is difficult to segment since its bottom left is always located in a blurred zone.

Let us note that the MPA CSA variations versus time, throughout a cardiac cycle, obtained from the present automatic segmentation method are in agreement with previous results obtained from a manual outlining (Hôpital of Bordeaux, France). Moreover, the present method enables us to get a reliable assessment of the systolic-diastolic difference in the vessel CSA, which should lead to a further improvement in the accuracy of a non-invasive blood pressure estimation (using well-known methods, see [32,33]). In several medical applications, let us note that a registration process must be added to the segmentation process (see [34,35,36,37] for more details).

3.4. Applications to Geophysical Imaging

We now apply our algorithm to the segmentation of a non-continuous layer on a geophysical dataset (Figure 11 (right)) extracted from a 3D dataset (Figure 11 (left)). This is a complicated image to segment since there is a vertical fault (see Figure 12). As a priori condition, we here consider a set of point to interpolate (instead of a curve) in order to help the segmentation process (Figure 12). The algorithm starts with a polygonal line interpolating the set of points (to interpolate) as an initial condition, and then segment the image until the edge(s) of the image. Here, the final contour is a segment and not a closed curve, this is of great interest when one wants to track faults or cracks in segmentation processes.

The obtained result (Figure 13) has been validated by geophysicians, but the next step requires an algorithm for 3D applications in order to segment 3D geophysical datasets. We are also in the process of creating a free labeled geophysical dataset, in order to test deep learning techniques on these images.

4. Conclusions

We have proposed a method to segment medical or geophysical images with a priori conditions. The efficiency of the method has been shown on 2D images, knowing that such images are difficult to segment with usual approaches since the searched contour is not continuous (geophysical dataset) or when the zone to segment is blurred (medical images).

Compared to Gout et al. [19], the added value brought by our proposed method (automatic initial guess) is materialized by a total number of iterations reduced by a factor of 4 to 60.

The GPU time is also a key point to underline about our algorithm, much lower than deep learning based approaches. About the sensitivity of the model, especially for geophysical data, it is not possible to segment correctly without a precise choice of the interpolation conditions: the choice of such condition strongly impacts the final result, this point is less obvious in medical applications where the choice of the initial condition improves the result only on very noisy areas and/or with a blurred edge, otherwise, the result (whatever the initial condition) is very close to classic algorithms (Chan Vese algorithm for example). The implementation of a 3D code is a work in progress, the main problem for a 3D segmentation process consists in choosing the initial (geometric) condition (a surface, a curve and/or points), which is a complicated task on complex 3D images (mainly because there are 3D data visualization difficulties “to see inside” the 3D dataset, especially in geophysics to give the initial condition—see Figure 14)—the mathematical results remain valid in 3D.