An Improved Level Set Method on the Multiscale Edges

: The level set method can segment symmetrical or asymmetrical objects in real images according to image features. However, the segmentation performance varies with feature scale. In order to improve the segmentation e ﬀ ect, we propose an improved level set method on the multiscale edges, which combines the level set method with image multi-scale decomposition to form a uniﬁed model. In this model, the segmentation relies on multiscale edges, and the multiscale edges depend on multiscale decomposition. A novel total variation regularization is proposed in multiscale decomposition to preserve edges. The multiscale edges obtained by the multiscale decomposition are integrated into the segmentation process, and the object can be easily extracted from a proper scale. Experimental results indicate that this method has superior performance in precision, recall and F-measure, compared with relative edge-based segmentation methods, and is insensitive to noise and inhomogeneous sub-regions.


Introduction
Image segmentation is a technique of partitioning an image into disjoint regions [1]. It is one of the most critical steps toward image understanding. Many image segmentation techniques are widely applied in various fields of computer vision such as moving object detection [2,3], object recognition [4,5] and vehicle tracking [6]. These segmentation techniques use low-/mid-level features, such as the intensity/color information [2], edges [4], or shape [5]. However, it is still difficult to segment real images since the symmetrical or asymmetrical objects of real images are diverse and the background is complex, and random.
According to features for segmentation, image segmentation methods can be divided into two categories, which are based on the learned features and based on the artificial features. The learned features are generally obtained by the deep learning method from the training, and the artificial features generally are low-level features of an image. The segmentation methods based on learned features can extract the objects by using the features obtained from deep learning [7] on the training set. These segmentation methods usually require a large amounts of training data consisting of objects that may appear in an image. In practice, the training data are limited in many specific tasks. In this case, they cannot replace the methods based on the artificial features. Fully convolutional networks (FCNs) can auto-implement the image segmentation with multi-level information [8], which is obtained through deep learning. Compared with the traditional segmentation algorithms, FCNs can achieve out-performance for the objects with massive samples, but not for the objects with small training

Proposed Method
The image segmentation performance based on the level set method depends not only on the algorithm itself, but also on edges. The image edges are sensitive to scale. For example, the fine information of the image edges is lost at large scales, which causes the edges to deviate from the actual edges; at small scales, the image edges are sensitive to noise, leading to false edge detection. Therefore, it is necessary to select the proper scale in image segmentation. In this paper, we propose an improved level set method which combines the segmentation and the multiscale decomposition to extract the object from a proper scale. The overall structure of the proposed method is shown in Figure 1. The proposed model can be described as the following formula: where u 0 : Ω → [0, 1] is an image with N pixels, φ is the level set function, a set (x, y) φ(x, y) = 0 denotes the segmentation curve, (x, y) φ(x, y) > 0 denotes the background, and (x, y) φ(x, y) < 0 denotes the object. The first term D(u 0 , u) is a multiscale decomposition term, where u is a smoothed image. The second term S(∇u, φ) is a segmentation term, where ∇u is the gradient of the smoothed image.

Proposed Method
The image segmentation performance based on the level set method depends not only on the algorithm itself, but also on edges. The image edges are sensitive to scale. For example, the fine information of the image edges is lost at large scales, which causes the edges to deviate from the actual edges; at small scales, the image edges are sensitive to noise, leading to false edge detection. Therefore, it is necessary to select the proper scale in image segmentation. In this paper, we propose an improved level set method which combines the segmentation and the multiscale decomposition to extract the object from a proper scale. The overall structure of the proposed method is shown in Figure 1. The proposed model can be described as the following formula: where 0 : x y x y φ = denotes the segmentation curve, {( , ) | ( , ) 0} x y x y φ > denotes the background, and {( , ) | ( , ) 0} x y x y φ < denotes the object. The first term

The Multiscale Edges
The segmentation result, that is an optimal solution of (1), depends on the edges which vary with the scale of u . For an image with the homogeneous regions, since the color distributions within each region contrast highly among regions, the object extraction could achieve a good performance even in a fine scale, as with the image 0 u . In practice, the smoothed image in a fine scale often contains inhomogeneity within a region and leads to pseudo edges. To improve the smoothing effect to preserve strong edges and enhance weak edges, a novel edge-preserved smoothing model is used to analyze the image, which is formulated as based on the following minimization problem: Suppose that 0 ( , ) D u u has a minimum point. Then, it formally satisfies the Euler-Lagrange equation:

The Multiscale Edges
The segmentation result, that is an optimal solution of (1), depends on the edges which vary with the scale of u. For an image with the homogeneous regions, since the color distributions within each region contrast highly among regions, the object extraction could achieve a good performance even in a fine scale, as with the image u 0 . In practice, the smoothed image in a fine scale often contains inhomogeneity within a region and leads to pseudo edges. To improve the smoothing effect to preserve strong edges and enhance weak edges, a novel edge-preserved smoothing model is used to analyze the image, which is formulated as based on the following minimization problem: Suppose that D(u 0 , u) has a minimum point. Then, it formally satisfies the Euler-Lagrange equation: The simple finite difference scheme and lagged diffusivity fixed-point iteration method are exploited to linearize the above equation at each iteration u (n−1) → u (n) . If p is a member of the four adjacent pixel set Λ on the target pixel p 0 (Figure 2), u (n−1) (p 0 ) can be updated by: where ω (n) (p) is the weight coefficient of the pixel p: when ∇u (n) (p) → 0 , ω (n) (p) → 1 , the inhomogeneous sub-region of an image is smoothed. When ∇u (n) (p) → ∞ , ω (n) (p) → 0 and u (n) (p 0 ) ≈ u (0) (p 0 ), the edges are preserved.
The simple finite difference scheme and lagged diffusivity fixed-point iteration method are exploited to linearize the above equation at each iteration ( -1)    approximates the weighted-mean of adjacent pixels, the edges are blurred and these blurred edges will cause the segmentation curve to converge excessively.
The fixed ( ) n τ cannot adaptively harmonize edge-preserving and sub-region smoothing according to the local information of the image [23,24]. In order to solve this problem, the 2-clustering of the target pixel and its adjacent pixels are analyzed based on their spatial relationship.
• If the target pixel is inside or outside the object, the adjacent pixels should be the object or not due to region connectivity ( Figure 2a).

•
Suppose the target pixel is located at the object boundary, the adjacent pixels would belong to one of the following cases.
 If one of the four adjacent pixels is located in the background, there are four kinds of situations ( Figure 2b). The parameter τ (n) in Equation (4) trades off edge preservation and region-smoothing. If τ (n) >> ω (n) (p), then u (n) (p 0 ) ≈ u 0 (p 0 ) which cannot optimize the edges. If τ (n) << ω (n) (p), the u (n) (p 0 ) approximates the weighted-mean of adjacent pixels, the edges are blurred and these blurred edges will cause the segmentation curve to converge excessively.
The fixed τ (n) cannot adaptively harmonize edge-preserving and sub-region smoothing according to the local information of the image [23,24]. In order to solve this problem, the 2-clustering of the target pixel and its adjacent pixels are analyzed based on their spatial relationship.

•
If the target pixel is inside or outside the object, the adjacent pixels should be the object or not due to region connectivity ( Figure 2a).

•
Suppose the target pixel is located at the object boundary, the adjacent pixels would belong to one of the following cases. If one of the four adjacent pixels is located in the background, there are four kinds of situations ( Figure 2b). If two of the four adjacent pixels are located in the background, there are six kinds of situations ( Figure 2c).
As shown in Figure 2, the target pixel should belong to the sub-class that contains more pixels, and the parameter τ (n) should be set as: The multiscale edges of an image can be described as: The presence of u 0 at each step constantly ensures that the raw image is not forgotten during the smoothing, which contains the crucial edges of an image. At step k, the smoothed image u (k) depends on u (k−1) . This causes the small-scale edges ∇u (k−1) to be smeared step by step. As the number of iterations increases, the strong edges are preserved, and the weak edges are enhanced.

The Level Set Method
In this paper, the object is extracted using the edges in the smoothed image. Assuming that each region in the smoothed image is a constant or an approximate constant, the edge-based level set methods can get accurate results effectively. The main idea is to represent the contour as the zero level set of a high-dimensional level set function, and make it evolve to the object edges driven by the energy function. The segmentation model using the level set method for an image u is formulated through the following minimization problem [16]: In Equation (7), the first two terms are the external energy which drive the zero level set toward the object boundaries, while the last term is the internal energy which controls the penalization effect. µ > 0 is a parameter controlling the penalization effect. λ > 0 and ν are the weight of the circumference and area of the segmentation curve, respectively. H(φ) is the Heaviside function, and δ(φ) is the Dirac measure. H(φ) and δ(φ) are defined as follows: g(∇u) is the edge indicator function. In our proposed model, the edge indicator function of the gradient |∇u| is defined as: Suppose that S(|∇u|, φ) has a minimum point, by introducing an artificial temporal variable t, the steepest descent process could be used to obtain the minimization of the function S(|∇u|, φ). Its gradient flow is: where ∆(•) is the Laplacian operator. The central difference is use to approximate ∇φ and introduce an artificial temporal variable ∆t, while φ (n−1) can be updated by:

Segmentation on the Multi Scale Edges
The proposed image segmentation method consists of the multiscale decomposition and segmentation. In the process of the discrete computation on (1), this model can be simplified as follows: A series of smoothed images with different scale edges are obtained using the multiscale decomposition model to decompose an image. Then the level set method is used to extract the object from the smoothed image with different scales one by one. At step k, the segmentation curve φ k is obtained from the evolution of smoothed image u k by level set algorithm with initial curve φ k−1 . The steps of multiscale edge segmentation are shown in Figure 1.
In the segmentation process, since the level set algorithm uses smoothed images with different scale edges each time, the results of each segmentation are inconsistent. In order to evaluate the performance of the segmentation using the adjacent scale-edges, a significance level of segmentation region is established, which is defined as follows: where A k and A k−1 represent the objects using the smoothed images with different scale edges at number of iterations k and k − 1, in which A = i φ(i) ≤ 0, i = 1, 2, · · · , N . In (13), with the number of iterations k gradually increases, the strong edges are preserved, and the weak edges are enhanced. The segmentation curve gradually evolves to the contour of the object and Sl(k) gradually decrease. When the segmentation curve reaches the contour of the object, A k and A k−1 are the closest and Sl(k) = 0, that is, Sl(k) is the minimum. When the number of iterations k continues to increase, the edges of the image become blurred, the segmentation curve continue to shrink, and Sl(k) gradually increase until the segmentation curve disappears. In summary, the automatic iteration termination condition for segmentation should be constructed as Sl(k) = 0.

Experimental Results and Analysis
The experiments were conducted using VC 6.0 on a PC with Intel-Core i5 CPU @ 3.40 GHz and 4GB of RAM without any particular code optimization. During the implementation of the proposed model, the parameter λ = 5.0, µ = 0.04, ν = 3.0, ∆t = 5.0 were used for all experiments. In addition, the proposed model is implemented to monochannel images.
In order to test the segmentation performance using the proposed method on real images with strong or weak edges, experiments were carried out to compare this with relative level set models and other edge-based segmentation models, such as Li's model [16], SDREL model [19], DCLSM model [25] and IMST model [26]. In these models, Li's model is an edge-based level set model that uses the Gaussian filter to smooth images. SDREL model is an improved level set model that considers the saliency map into account and combines region information with edge information into the level set model. Recently, many level set methods combined with deep learning have been proposed [25,27,28]. These existing methods mainly use deep learning to perform coarse segmentation first to obtain the initial mask for level set evolution. Then, the level set function uses this mask as a prior shape to evolve to get the contours of the objects. However, these methods would not get good segmentation Symmetry 2020, 12, 1650 7 of 14 results when the initial mask is not accurate enough or covers the object. To prove the performance of the proposed model, DCLSM model which is based on deep learning was considered for comparison.
The different sizes of images that were obtained from the International web and the Berkeley segmentation database were segmented. These partial results are shown in Figure 3. The effects of these three algorithms on the image with strong edges are almost the same (Figure 3a). For images with weak edges (Figure 3b,c), the segmentation result using this method outperformed the other two methods. Compared with other methods, this method has the following three characteristics: (1) integrating smoothing, image segmentation, and the number of smoothing iterations depends on the significant level of segmentation; (2) the weight of the target pixel is set according to local information, and adaptively harmonizes edge-preservation and region-smoothing; (3) the weak edges are enhanced. In Li's model, the segmentation accuracy declines since the edges are blurred by Gaussian smoothing (Figure 3c). The segmentation effects are poor when using the IMST method for images with weak edges.
In order to evaluate the segmentation performance using this method on real images with inhomogeneous regions, the different sizes of images were segmented, and the partial results are presented in Figure 4. In Li's model, the segmentation curve was far away from the ground truth due to the edge blurred by Gaussian smoothing. The effect of this method was better than the other two models. However, there were non-semantic sub-regions when using this model in Figure 4c. The reason is that the weak edges were over-enhanced. More experimental results are shown in Figure 5 to prove the segmentation performance.
Symmetry 2020, 12, x FOR PEER REVIEW 7 of 14 segmentation first to obtain the initial mask for level set evolution. Then, the level set function uses this mask as a prior shape to evolve to get the contours of the objects. However, these methods would not get good segmentation results when the initial mask is not accurate enough or covers the object.
To prove the performance of the proposed model, DCLSM model which is based on deep learning was considered for comparison. The different sizes of images that were obtained from the International web and the Berkeley segmentation database were segmented. These partial results are shown in Figure 3. The effects of these three algorithms on the image with strong edges are almost the same (Figure 3a). For images with weak edges (Figure 3b,c), the segmentation result using this method outperformed the other two methods. Compared with other methods, this method has the following three characteristics: (1) integrating smoothing, image segmentation, and the number of smoothing iterations depends on the significant level of segmentation; (2) the weight of the target pixel is set according to local information, and adaptively harmonizes edge-preservation and region-smoothing; (3) the weak edges are enhanced. In Li's model, the segmentation accuracy declines since the edges are blurred by Gaussian smoothing (Figure 3c). The segmentation effects are poor when using the IMST method for images with weak edges.
In order to evaluate the segmentation performance using this method on real images with inhomogeneous regions, the different sizes of images were segmented, and the partial results are presented in Figure 4. In Li's model, the segmentation curve was far away from the ground truth due to the edge blurred by Gaussian smoothing. The effect of this method was better than the other two models. However, there were non-semantic sub-regions when using this model in Figure 4c. The reason is that the weak edges were over-enhanced. More experimental results are shown in Figure 5 to prove the segmentation performance.          Table  1. This method tolerates a higher computational cost to obtain better segmentation results. The computational cost using this model mainly depends on the degree of region inhomogeneity, such as those presented in Figures 3a and 4b, and the CPU times were 3.715 and 7.098 s, respectively.             Table 1. This method tolerates a higher computational cost to obtain better segmentation results. The computational cost using this model mainly depends on the degree of region inhomogeneity, such as those presented in Figures 3a and 4b, and the CPU times were 3.715 and 7.098 s, respectively.
Compared with the Li's model, the computational cost using this model was higher. The reason is that this method uses iteration smoothing while Li's model uses Gaussian smoothing just once.
Noise is a negative effect on segmentation. In order to test for robustness against noise, 480 × 345 degraded images with additive Gaussian noise were segmented and compared with the IMST model [26] and Li's model [16] using the same initial curve. Partial results are shown in the Figure 6. For the original images, the effect of the proposed method was similar as those of the IMST model, and this performed better than that of Li's model. With the degrading image quality, the segmentation accuracy of this method was higher than that of other models. The segmentation scores are shown in Table 2. The variance of F-measure using this method, IMST model [26] and Li's model [16] were respectively 0.9%, 4% and 2.3%. The variance of F-measure was smaller to those of the other two models. The mean of the F-measure was 0.970, 0.933 and 0.906, respectively. The mean of F-measure using the proposed method was higher than those of the other two models. From the statistical mean and variance, this proposed method was not sensitive to noise.
The CPU time of segmentation on the noisy image is shown in Table 3. Li's model uses Gaussian smoothing only once, while the proposed method uses iteration smoothing to deal with the inhomogeneous sub-region. Thus, the computational cost is higher than that of Li's model.    Compared with the Li's model, the computational cost using this model was higher. The reason is that this method uses iteration smoothing while Li's model uses Gaussian smoothing just once.
Noise is a negative effect on segmentation. In order to test for robustness against noise, 480 × 345 degraded images with additive Gaussian noise were segmented and compared with the IMST model [26] and Li's model [16] using the same initial curve. Partial results are shown in the Figure 6. For the original images, the effect of the proposed method was similar as those of the IMST model, and this performed better than that of Li's model. With the degrading image quality, the segmentation accuracy of this method was higher than that of other models. The segmentation scores are shown in Table 2. The variance of F-measure using this method, IMST model [26] and Li's model [16] were respectively 0.9%, 4% and 2.3%. The variance of F-measure was smaller to those of the other two models. The mean of the F-measure was 0.970, 0.933 and 0.906, respectively. The mean of F-measure using the proposed method was higher than those of the other two models. From the statistical mean and variance, this proposed method was not sensitive to noise.  The CPU time of segmentation on the noisy image is shown in Table 3. Li's model uses Gaussian smoothing only once, while the proposed method uses iteration smoothing to deal with the inhomogeneous sub-region. Thus, the computational cost is higher than that of Li's model.

Conclusions
In this work, we propose an improved level set method for segmentation on the multiscale edges. Compared with relative edge-based segmentation methods, this model improves the segmentation performance in terms of the following aspects: (1) the multiscale edges of an image are incorporated into the segmentation model; (2) the strong edges for segmentation are preserved and the weak edges are enhanced; (3) the optimal scale edge for segmentation is selected. However, the proposed method needs to set the initial curve artificially. Furthermore, the computational cost is high. In the future, our work will be focused on designing an algorithm to adaptively label the initial curve adjacent to the object boundaries.