Fuzzy Superpixel Segmentation with Anisotropic Total Variation Regularization

Abstract

This paper presents a superpixel segmentation algorithm that integrates anisotropic total variation regularization within a fuzzy clustering framework. While isotropic total variation is well-known for its edge-preserving properties, its non-adaptive nature often leads to over-regularization. In contrast, the anisotropic model formulates superpixel regularity in relation to image contours, thereby preventing the loss of image details in areas of high contour density during optimization. Compared to classical segmentation algorithms that employ non-adaptive regularization, the proposed content-adaptive approach enhances superpixel regularity while maintaining boundary adherence to image contours. Furthermore, to optimize the functional effectively, an alternating direction method of multipliers along with the enhanced Chambolle’s fast duality projection algorithm are employed. Competitive experiments against existing regular segmentation algorithms demonstrate that our proposed methodology achieves superior performance in terms of boundary recall, compactness, and shape regularity criteria, outperforming these methods by an average of at least 3%, 5%, and 3%, respectively. Furthermore, when compared with irregular segmentation algorithms, our approach achieves the best results in terms of compactness, contour density, and shape regularity criteria, with average improvements of at least 56%, 22%, and 45%, respectively.

1. Introduction

Superpixel segmentation is a crucial pre-processing step in image processing, with applications spanning image classification [1] and 3D reconstruction [2]. It works by over-segmenting an image into atomic regions that ideally correspond to individual objects. By grouping pixels into superpixels, this method reduces computational complexity—transforming millions of pixels into a few hundred meaningful regions, regardless of resolution. This compact representation eliminates redundant pixel-level processing, enabling more efficient algorithms compared to traditional pixel-based approaches.

Unlike pixel representation, which is essentially a regular grid, superpixels often exhibit irregular shapes and varying sizes. These irregularities are unavoidable as the algorithms strive to classify pixels into superpixels while aligning boundaries with object contours as precisely as possible. Consequently, subsequent processes that utilize superpixels must be specifically designed to accommodate their diverse shapes. These procedures typically involve analyzing the statistics of each superpixel and establishing relationships between neighboring superpixels. Irregular superpixel segmentation can significantly impede these processes; for example, tiny superpixels may result in poor feature estimations, while snake-shaped superpixels may incorrectly connect with spatially distinct neighbors. Therefore, regularity is a key metric of superpixel segmentation quality [3,4] and strongly influences their utility as pre-processing tools. However, a survey [5] shows that some superpixel algorithms perform poorly at maintaining the regularity of the generated superpixels. Furthermore, the benchmark in [6] indicates that many algorithms fail to produce regular superpixels on noisy images, limiting their applicability to noise-free, controlled environments. Nonetheless, many methods incorporate specific mechanisms to improve superpixel regularity. For instance, ERS [7] incorporates a balance term in its energy functional to promote superpixels of similar size. The SEEDS algorithm [8] introduces a boundary term that discourages image patches from containing multiple superpixel labels, thereby enhancing regularity by reducing the number of pixels near superpixel boundaries. Similarly, SLIC [9] and CAS [10] include a regularization term that favors compact superpixels. Additionally, ETPS [11] incorporates energy terms that encourage compactness, shorter boundary lengths, and enforce both connectedness and a minimum size for the superpixels.

The regularities mentioned above are non-adaptive, meaning that they do not take into account the actual content of the images or the locations of the superpixels. Most existing algorithms combine these regularity measures with fidelity terms to capture object boundaries while enforcing superpixel regularity. While this approach yields good overall performance in terms of generating regular superpixels, it inevitably results in the loss of finer image details, such as sharp corners, non-convex regions, and elongated segments. The accuracy loss stems from the fact that these non-adaptive regularities are evaluated solely based on the shapes of the superpixels. Without prior knowledge of the image, it becomes challenging to differentiate between irregular image objects and artifacts introduced by the fidelity terms, leading to both being suppressed by the regularization process. In many cases, the necessary over-smoothing of irregular geometries is required to achieve an overall smoothness of the superpixels under these non-adaptive regularity measures.

While the issue of non-adaptive superpixel regularity is seldom addressed, the related challenge of non-adaptive image regularity has been extensively studied in the field of image denoising. Recognizing that non-adaptive denoising filters often lead to over-smoothing, many image denoising techniques have been developed with various strategies to adapt to the local content of images. One such approach involves modifying the weight coefficients of mean filters adaptively for each pixel. For instance, bilateral filtering computes weights based on both spatial distances and color similarities between pixels, whereas the weights in non-local means filtering depend on the similarities between image patches. Another common strategy is to restrict regularization isotropically in areas where edges and other details are likely to be present. An example of this is the Perona–Malik diffusion [12], which employs a diffusion coefficient that is inversely related to the magnitude of the gradient.

Motivation and Contribution

To overcome the limitations of non-adaptive regularities employed by current algorithms and the inadequate performance of existing superpixel segmentation methods in capturing objects with irregular geometries mentioned above, an anisotropic approach is proposed to address these challenges. The contributions of this paper are summarized as follows:

• A content-adaptive superpixel regularity measure, based on an anisotropic total variation model, is introduced to enhance the preservation of image details and is integrated into a fuzzy clustering-based segmentation framework.
• An anisotropic variational fuzzy superpixel segmentation algorithm is developed based on the alternating direction method of multipliers and an enhanced version of Chambolle’s fast duality projection algorithm.

This paper is organized as follows. In the next section, existing superpixel segmentation algorithms are reviewed. Section 3 presents our proposed anisotropic variational fuzzy superpixel segmentation model with content-adaptive regularity. The optimization procedure and implementation details of our proposed algorithm are discussed in Section 4. Finally, experiments and competitive results are reported in Section 5 and Section 6, concluding the paper.

2. Related Works

Since the introduction of superpixels in [13], which demonstrated their effectiveness in image segmentation, numerous superpixel algorithms have been developed over the past few decades. Most of these algorithms focus on enhancing performance in capturing object boundaries, while only a few address the regularity of superpixels. Existing superpixel algorithms can be classified into three main categories: clustering-based, boundary-based, and graph-based algorithms. Additionally, two special types of algorithms deserve a separate mention: hierarchical algorithms and deep-learning-based algorithms.

2.1. Clustering-Based Algorithms

Clustering-based algorithms classify pixels into superpixels using techniques that involve alternating updates of centroids and pixel reassignment. Notable examples include SLIC, LSC [14], and CAS, which utilize k-means clustering based on various feature representations. By incorporating spatial features like xy-coordinates, these algorithms create compact superpixels at low computational costs and can be accelerated with GPU computation. However, the Euclidean distances used in standard k-means may not effectively capture the differences between individual pixels and centroids, leading to the poor adherence of superpixel boundaries to image contours. Alternatively, some algorithms adopt non-Euclidean distances to enhance performance. For instance, SCALP [15] employs a robust distance metric that considers color features and contour intensity along linear paths between pixels and superpixel barycenters, resulting in superpixels that align well with image contours when advanced contour detectors are used. Additionally, a multi-scale version of SLIC is proposed in [16], which employs a “Difference Image” to determine local scales for finer segmentation in areas with significant changes. Recently, another modification of SLIC is presented in [17], which utilizes non-uniform seed initialization and a constrained distance measure to improve the connectivity and regularity of the superpixels.

The hard labeling nature of k-means can yield unsatisfactory results, particularly in degenerated images like radar or medical images. This can negatively affect sequential processing and the classification accuracy. To address this, the authors in [18] identified uncertain pixel labels using fuzzy clustering. The fuzzy-SLIC algorithm presented in [19] allows multiple superpixels to share a single pixel’s membership and utilizes local spatial fuzzy c-means (FCM) clustering to enhance noise robustness, outperforming traditional hard labeling methods. Additionally, a superpixel algorithm based on a Gaussian mixture model (GMMSP) [20] employs the expectation–maximization method to estimate parameters and determine pixel label probabilities, allowing for explicit control over superpixel compactness.

2.2. Boundary-Based Algorithms

Boundary-based algorithms create superpixels by iteratively adjusting their boundaries, allowing for the enforcement of the smoothness and regularity criteria. These algorithms can be divided into two main types.

The first type starts with an initial segmentation, often in the form of a regular grid, and iteratively reassigns boundary pixels to neighboring superpixels. A notable example is ETPS, which, like the TPS algorithm [21], swaps boundary pixels to minimize an energy function related to color and spatial dissimilarity, as well as boundary smoothness. However, ETPS prevents swaps that would disrupt superpixel connectivity and adopts a coarse-to-fine approach similar to SEEDS, enabling real-time optimization and better results. Comparative experiments show that ETPS excels at generating regular superpixels that closely adhere to image contours. Notable examples in this category include VCells [22] and CCS [23].

The second type employs region-growing strategies, where superpixels expand from seed locations until all pixels are classified or a termination condition is met. Many algorithms in this category are based on the watershed transform (WT), a classic region-growing segmentation method. While WT can directly generate superpixels, its original formulation lacks regularity. The compact watershed algorithm [24] addresses this by adding a compactness term to the distance function. The waterpixel [25] regularizes the gradient map of the watershed by considering the spatial distances between pixels and markers. Additionally, the method in [26] enhances image quality through morphological reconstruction before applying the watershed transform. Other notable region-growing algorithms include TurboPixel [27], which uses boundary curvature and local contour intensity to guide flow velocity, and NICE [28], a non-iterative clustering framework with efficient priority queue implementation that can ensure superpixel connectivity when evolving the centroids.

2.3. Graph-Based Algorithms

Graph-based algorithms treat the image as a graph, where each pixel is a vertex and neighboring pixels are connected by edges. Various techniques are then used to partition the graph and create superpixel segmentations. While many graph-based algorithms [29,30] achieve high accuracy in capturing image details, they often produce irregular superpixels due to the lack of spatial features in their representations. One notable exception is the graph-cut-based approach introduced in [31], which partitions the pixel graph based on normalized cut criteria. This method can generate highly regular superpixels with good boundary adherence, but its high computational complexity limits its application to large images. Another exception is the superpixel lattice algorithm [32], which recursively partitions the image using minimum-cost paths between image borders.By controlling the tortuosity of these paths, it regulates how much the superpixels deviate from regular grid structures.

2.4. Hierarchical Algorithms

A special category of algorithms creates a hierarchy of segmentations, consisting of layers of superpixels at different scales, with aligned boundaries across layers. This feature is crucial for many computer vision applications [33] that require multi-scale superpixels to accurately represent image structures. One example of a hierarchical algorithm is presented in [34], which is a hierarchical version of VCells. Another example is a graph-based approach [35], where the hierarchy is derived from the Borůvka’s algorithm. Additionally, some classical graph-based algorithms [29] can produce hierarchical segmentations through top-down or bottom-up processes. However, these hierarchies tend to be unbalanced, as traditional algorithms do not account for this aspect.

2.5. Deep-Learning-Based Algorithms

Recent advancements in deep learning have led to the development of algorithms that integrate superpixels with neural networks. These algorithms can be divided into three categories.

The first category applies traditional methods to image features extracted by neural networks. For example, a deep affinity learning network is used in [36] to aggregate multi-scale features, and then a graph-based algorithm is employed to compute superpixels from the learned affinities. Another example [37] combines a convolutional network with a mean-shift clustering module to compute pixel embeddings, which are then used in a non-iterative clustering framework to generate superpixels. In [38], superpixels with a lattice topology are generated by the coarse-to-fine refinement of superpixels’ boundaries under the guidance of a pretrained network. While these algorithms allow for some control over superpixel regularity, they are not fully end-to-end trainable since traditional methods are not integrated into the neural network architecture.

The second category directly predicts superpixel labels for pixels. These algorithms typically position superpixels on a regular grid and predict pixel associations with neighboring superpixels. The earliest example is the fully convolutional network, which computes predictions using a series of convolutional layers trained with a reconstruction cost function. This inspired further models, such as AINet [39], which includes an association implantation module to compute associations based on pixel and superpixel embeddings. On the other hand, the biological network in [40] supports the computation of an association map with enhanced screening modules, and the network is trained with boundary-aware labels to emulate human visual mechanisms. In a two-stage approach called ESNet [41], an image is first fed into a CNN-based feature extractor, and then a pyramid–gradient network is used to compute the pixel associations from the extracted features. An unsupervised method called LSN-Net [42] uses a non-iterative clustering module for soft pixel assignments to superpixels, along with a gradient rescaling module to prevent overfitting. More recently, a superpixel-based visual transformer [43] has predicted associations through cross-attention updates. While these algorithms effectively utilize neural networks to distinguish between pixel classes and are end-to-end trainable, they often produce superpixels with irregular shapes.

The third category features algorithms that incorporate differentiable modifications of classical superpixel methods, allowing for end-to-end training while maintaining better control over superpixel regularity. A notable example is the superpixel sampling network (SSN) [44], which makes SLIC differentiable and uses a deep network to learn pixel features. The SSN is trained with a combination of reconstruction and compactness loss functions to enhance regularity. An improvement, BP-Net [45], incorporates a boundary detection network to handle depth information in RGB-D images, and is trained with a block regularity loss function, resulting in highly regular superpixels.

3. Anisotropic Variational Fuzzy Superpixel Segmentation

The superpixel segmentation can be regarded as an unsupervised learning problem that divides an image with N pixels to K superpixels. In this paper, the following anisotropic variational fuzzy superpixel segmentation model is studied:

$$E(U,C)=E_{\mathrm{FCM}}(U,C)+\alpha E_{\mathrm{ATV}}\left(U\right),$$

(1)

subject to

$$\sum _k{u}_{i,k}=1\forall i,U\ge 0,$$

(2)

where $U=\left[u_{i,k}\right]$ is the partition matrix in which $u_{i,k}$ represents the association degree of membership of the ith pixel to the kth superpixel, $C=[c_1,\dots ,c_K]$ is the collection of the centroids of superpixels, and $\alpha$ is the parameter that controls the regularization. In (1), the first term, $E_{\mathrm{FCM}}$, represents the objective function of the FCM clustering model, which classifies pixels into fuzzy superpixels. The second term, $E_{\mathrm{ATV}}$, measures the total variation energy of the fuzzy membership function, enhancing the regularity of the superpixels. Notably, $E_{\mathrm{ATV}}$ is an anisotropic version of the total variation energy, where anisotropy is derived from image contours. Unlike isotropic total variation, $E_{\mathrm{ATV}}$ reduces the penalties for irregularities near contours, allowing the superpixels to better align with image boundaries. It is important to remark that anisotropic total variation regularization is being introduced for the first time in superpixel segmentation to enhance superpixel regularity. In what follows, the two energy terms are discussed in detail.

3.1. Dissimilarity Measurement and Feature Representation

The first energy term is the cost function of the FCM clustering model with the weighted Euclidean distance, defined as follows:

$$E_{\mathrm{FCM}}(U,C)=\sum _{i,k}{u}_{i,k}^2{d}^2(f_i,c_k),$$

(3)

where $f_i$ is a feature vector of the ith pixel and $d(f_i,c_k)$ is a dissimilarity measure between $f_i$ and $c_k$. Since computational complexity is one of the most important issues in the application of superpixel segmentation, the spatial feature and CIELAB color are adopted for feature representation, i.e., $f_i=(f_i^x,f_i^y,f_i^l,f_i^a,f_i^b)$. The element $(f_i^x,f_i^y)$ represents the spatial coordinates of the ith pixel while the other components correspond to the CIELAB color values of the same pixel. It is important to note that a spatial feature is utilized to enhance superpixel compactness, while the CIELAB color space is chosen for its perceptual uniformity in relation to human color vision, making it effective across various applications. Here, the dissimilarity measure is defined as a weighted Euclidean distance, where the weights assigned to the spatial feature are determined by a compactness parameter M > 0. Furthermore, to expedite the segmentation process, the computation of the dissimilarity measure is restricted to only those pixels that lie within the $(2L_k+1)\times (2L_k+1)$ window $W_k$ centered at the centroid $c_k$. This ensures that the radius of each superpixel does not exceed the predefined maximum radius $L_k$. In other words, the dissimilarity measure is defined as follows:

$$\begin{array}{c}\hfill d^2(f_i,c_k)=M{d}_s^2(f_i,c_k)+d_c^2(f_i,c_k)+d_b(f_i,c_k),\end{array}$$

where the first two terms are

$$d_s^2(f_i,c_k)=\sum _{\varphi \in \{x,y\}}{(f_i^{\varphi }-c_k^{\varphi })}^2,$$

$$d_c^2(f_i,c_k)=\sum _{\varphi \in \{l,a,b\}}{(f_i^{\varphi }-c_k^{\varphi })}^2,$$

while the last term,

$$d_b(f_i,c_k)=\left\{\begin{array}{cc}0& \mathrm{if}(f_i^x,f_i^y)\in W_k,\hfill \\ \infty & \mathrm{otherwise}\hfill \end{array}\right.$$

penalizes the distance by ∞ if the pixel is too far away from the centroid.

3.2. Anisotropic Total Variation Regularization

The second energy term measures the regularity of the superpixels by considering the anisotropic total variation (ATV) of each superpixel. Here, the kth superpixel is treated as a monochromatic image $u_k=u_{·,k}$. Mathematically, this energy term can be expressed as follows:

$$\begin{array}{cc}\hfill E_{\mathrm{ATV}}\left(U\right)& ={\displaystyle \sum _k}ATV\left(u_k\right)\hfill \\ \hfill & ={\displaystyle \sum _k}{\displaystyle \sum _i}\sqrt{{\displaystyle \sum _{j\in {\mathcal{N}}_i}}{\displaystyle \frac{a_{i,j}^2{(u_{i,k}-u_{j,k})}^2}{{(f_i^x-f_j^x)}^2+{(f_i^y-f_j^y)}^2}}},\hfill \end{array}$$

(4)

where ${\mathcal{N}}_i$ denotes the eight neighbors of pixel i and $a_{i,j}$ represents the anisotropy at pixel i in the direction toward pixel $j\in {\mathcal{N}}_i$. Additionally, the denominator in (4) is simply the spatial distance between pixel i and pixel j, which is used for the forward difference approximation of the directional derivatives.

The $ATV$ above is an anisotropic generalization of the well-known Rudin, Osher, and Fatem (ROF) functional [46]:

$$T{V}_2\left(u\right)={\int }_{\mathrm{\Omega }}{∥\nabla u∥}_2={\int }_{\mathrm{\Omega }}\sqrt{\nabla u^T\nabla u},$$

where $\mathrm{\Omega }$ is the image domain and $\nabla u$ is the gradient of u. In fact, our $ATV$ can be written as follows:

$$ATV\left(u\right)={\int }_{\mathrm{\Omega }}\sqrt{\nabla u^T{A}^2\nabla u},$$

(5)

where A is the anisotropy tensor. Here, the anisotropy refers to the assignment of varying weights to different components of the gradient operator ∇, based on the specific image locations and the directions of these components.

In addition to anisotropy, another distinction between our $ATV$ and the original ROF functional is noteworthy. In the ROF framework and its discretization [47], the gradient operator ∇ comprises only two components, corresponding to the partial derivatives along the x- and y-axes. To achieve faster convergence and a more effective representation of anisotropy, an eight-directional $ATV$ is employed in this paper. In our formulation, the gradient ${(\nabla u)}_i$ at pixel i is calculated in eight directions corresponding to the eight neighboring pixels. For each direction $\theta$, the $\theta$th component of $\nabla u$ represents the directional derivative of u along that direction, which is discretized as follows:

$${(\nabla u)}_i^{\theta }={\displaystyle \frac{u_{\theta \left(i\right)}-u_i}{\sqrt{{(f_i^x-f_{\theta \left(i\right)}^x)}^2+{(f_i^y-f_{\theta \left(i\right)}^y)}^2}}},$$

(6)

where $\theta \left(i\right)$ is the neighbor of the pixel i in the direction of $\theta$.

It is essential to emphasize that anisotropy should not be calculated from the regularized outputs of the previous iteration, as this would add unnecessary overhead to each iteration of the optimization process. Additionally, image details such as edges and corners may be compromised due to over-smoothing during optimization, meaning that anisotropy derived from intermediate outputs may not adequately capture these crucial features. In rare cases, the coupling between anisotropy and intermediate outputs could even result in instabilities. This issue can be circumvented by precomputing the anisotropy from the input. This strategy aligns with our goal of regularizing the fuzzy membership function to enhance edge directions, rather than regularizing the input image itself.

For each neighboring pixel pair $(i,j)$, the probability $p_{i,j}$ is first calculated so that an edge separates the two pixels by applying an edge detector to the input image.The anisotropy is then set as

$$a_{i,j}=1-p_{i,j}$$

(7)

to reduce the regularization of the fuzzy membership function across the edges. It is important to note that the anisotropy only needs to be computed once at initialization and is independent of the fuzzy membership function.

4. Optimization Procedure

The minimization of (1) with the energies (3) and (4) and the constraints (2) forms a class of constrained nonlinear optimization problems whose solutions are unknown. To efficiently optimize the energy functional subject to a set of constraints, the alternating direction method of multipliers (ADMM) is applied to decouple the two energy terms in (1), which yields the following augmented Lagrangian:

$$L_{\rho }(U,V,C,Y)=E_{\mathrm{FCM}}(U,C)+\alpha E_{\mathrm{ATV}}\left(V\right)+{\displaystyle \frac{\rho }{2}}{∥U-V+Y∥}_F^2,$$

where Y is the multiplier of the new constraint $U=V$ under the ADMM scheme, $\rho$ is the penalty parameter, and ${∥·∥}_F$ is the Frobenius norm. From $L_{\rho }$, the ADMM scheme is used to iteratively update the following variables:

$$\begin{array}{cc}\hfill \hat{V}& =\underset{V}{argmin}\left\{\alpha E_{\mathrm{ATV}}\left(V\right)+{\displaystyle \frac{\rho }{2}}{∥U-V+Y∥}_F^2\right\},\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill \widehat{Y}& =Y+U-V,\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill \widehat{U}& =\underset{U}{argmin}\left\{E_{\mathrm{FCM}}(U,C)+{\displaystyle \frac{\rho }{2}}{∥U-V+Y∥}_F^2\right\},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill \widehat{C}& =\underset{C}{argmin}\left\{E_{\mathrm{FCM}}(U,C)\right\}.\hfill \end{array}$$

(11)

4.1. Fuzzy Membership Function

The solution to the sub-problem (10) under the constraints (2) can be obtained by solving the following problem for each pixel i:

$${\widehat{u}}_i=\underset{u_i}{argmin}\left\{\sum _k{u}_{i,k}^2{d}^2(f_i,c_k)+{\displaystyle \frac{\rho }{2}}{(u_{i,k}-z_{i,k})}^2\right\},$$

(12)

where $z_{i,k}=v_{i,k}-y_{i,k}$. The problem outlined above can be addressed using a similar technique as described in [48]. For each i, let ${\left\{z_{i,\left(k\right)}\right\}}_k$ be the reordering of the sequence ${\left\{z_{i,k}\right\}}_k$ in descending order. Define

$${\lambda }_{i,\left(k\right)}={\displaystyle \frac{1-{\sum }_{h=1}^k\left(\rho z_{i,\left(h\right)}\right)/(\rho +2d^2(f_i,c_{\left(h\right)}))}{{\sum }_{h=1}^k{(\rho +2d^2(f_i,c_{\left(h\right)}))}^{-1}}},$$

$$k_i^{\ast }=max\left\{k:\rho z_{i,\left(k\right)+{\lambda }_{i,\left(k\right)}}>0\right\},$$

and ${\lambda }_i^{\ast }={\lambda }_{i,\left(k_i^{\ast }\right)}$; then, the solution to (12) is given by

$${\widehat{u}}_{i,k}=max\left\{{\displaystyle \frac{\rho z_{i,k}+{\lambda }_i^{\ast }}{\rho +2d^2(f_i,c_k)}},0\right\}.$$

(13)

4.2. Superpixel Centroid Adjustment

The sub-problem (11) can be derived by setting the corresponding partial derivatives of $E_{\mathrm{FCM}}$ to 0. For each k, the solution is given by

$${\widehat{c}}_k={\displaystyle \frac{{\sum }_i{u}_{i,k}^2{f}_i}{{\sum }_i{u}_{i,k}^2}}.$$

(14)

4.3. Anisotropic Total Variation Minimization

Solving the sub-problem (8) is equivalent to addressing the following anisotropic total variation regularization problem for each k:

$${\widehat{v}}_k=\underset{v_k}{argmin}\left\{\alpha ATV\left(v_k\right)+{\displaystyle \frac{\rho }{2}}{∥v_k-g_k∥}_2^2\right\},$$

where $g_k=u_k+y_k$. This problem can be solved based on the Chambolle’s fast duality projection algorithm [47]. However, to adapt it for our $ATV$ model, it is necessary to replace the two-directional gradient operator in [47] with our eight-directional anisotropic operator, denoted as ${\nabla }_A$. The component in the direction of $\theta$ at pixel i is defined as follows:

$${\left({\nabla }_{A}v\right)}_i^{\theta }=a_{i,\theta \left(i\right)}{(\nabla v)}_i^{\theta }.$$

In the formula above, ∇ represents the isotropic gradient operator as defined in (6). The term $\theta \left(i\right)$ denotes the neighbor of pixel i in the direction of $\theta$, while $a_{i,\theta \left(i\right)}$ refers to the anisotropy defined in (7). When applying Chambolle’s algorithm, the sub-problem (8) is solved as follows: Let ${\mathrm{div}}_A$ be the adjoint of ${\nabla }_A$, and for each k, let the sequence $\{p_k^{\left(0\right)},p_k^{\left(1\right)},\dots \}$ be defined such that $p_k^{\left(0\right)}\equiv 0$ and

$$p_k^{(n+1)}={\displaystyle \frac{p_k^{\left(n\right)}+\tau {\nabla }_A({\mathrm{div}}_A{p}_k^{\left(n\right)}-\rho g_k/\alpha )}{1+\tau |{\nabla }_A({\mathrm{div}}_A{p}_k^{\left(n\right)}-\rho g_k/\alpha )|}},$$

where $0<\tau \le 1/32$; then, the solution to (8) is given by the limit

$${\widehat{v}}_k=\underset{n\to \infty }{lim}\left(g_k-(\alpha /\rho ){\mathrm{div}}_A{p}_k^{\left(n\right)}\right).$$

(15)

It is important to note that the condition for convergence requires $\tau \le 1/32$. This can be proven using techniques similar to those in [49], along with the fact that $a_{i,j}\le 1$ for all $i,j$.

4.4. The Overall Implementation

While a larger number of ADMM iterations is usually necessary for algorithm convergence, only a few iterations are often sufficient for effective superpixel segmentation in practice. The required iterations can be significantly reduced if the variables are initialized close to the optimal solution. However, the careful selection of the initialization method is crucial, as it can greatly influence the final outcome. Using regular grids to initialize U may seem straightforward, but it can waste computational resources on unnecessary regularization, especially since centroids may shift significantly in the initial phase. A more effective approach is the coarse-to-fine initialization used by ETPS, which is efficient and typically yields solutions close to the optimum. The ETPS algorithm represents an image with a pyramid structure, approximating it with blocks of decreasing sizes at each level. It optimizes superpixel segmentation by minimizing the following objective function:

$$\begin{array}{cc}\hfill E_{\mathrm{ETPS}}(U,C)& ={\lambda }_{pos}\sum _{i,k}{u}_{i,k}{d}_s(f_i,c_k)+\sum _{i,k}{u}_{i,k}{d}_c(f_i,c_k)+{\lambda }_b\sum _{i,k}\sum _{j\in {\mathcal{N}}_i}\left|u_{i,k}-u_{j,k}\right|\hfill \\ \hfill & +E_{\mathrm{topo}}\left(U\right)+E_{\mathrm{size}}\left(U\right),\hfill \end{array}$$

where the first two terms measure spatial and color dissimilarities between pixels and their assigned superpixels, the third term accounts for boundary lengths, and the last two terms impose penalties for disconnected and tiny superpixels, respectively. At each level, the algorithm iteratively exchanges boundary blocks between superpixels until a local optimum is reached, and then transitions to the next level to repeat the process with smaller blocks. This continues until the final level is reached, resulting in a refined initialization.

It is important to remark that the original ETPS initialization uses seed locations on a rectangular grid, which can maintain grid structure in homogeneous image regions. However, this approach can lead to instability with TV regularization due to four-corner singularities. To address this, it is proposed that the algorithm be initialized with seed locations on a hex grid, effectively eliminating these singularities. In our implementation, the parameters for the ETPS initialization are set to ${\lambda }_{pos}=M/2$ and ${\lambda }_b=10$, where M is the compactness parameter.

A pseudocode of the proposed algorithm is summarized below:

Algorithm 1 Anisotropic Variational Fuzzy Superpixel Segmentation (AVFS) Algorithm

Input:  Input image I Output:  Fuzzy membership function U   1: Compute the anisotropy $a_{i,j}$   2: Initialize U by ETPS with hex-grid   3: Initialize $Y^0\leftarrow 0$.   4: Set $t\leftarrow 0$.   5: repeat   6:       Update C by (14)   7:       Update V by (15)   8:       Update Y by (9)   9:       Update U by (13) 10:       Set $t\leftarrow t+1$. 11: until  $t>MaxIter$ 12: return  $U^{t+1}$

The parameter MaxIter mentioned in Algorithm 1 indicates the number of iterations in the experiment. To visualize the segmentation results and compare the proposed method with the existing approaches, as reported in Section 5, the defuzzification process is performed by assigning the ith pixel to the $\widehat{k}$th superpixel that maximizes the optimal fuzzy membership function, where $\widehat{k}=\arg {\max }_k\left\{u_{i,k}\right\}$.

5. Experiments

5.1. Experimental Settings

The proposed method is evaluated using the well-known Berkeley Segmentation Dataset 500 (BSDS500) [50], which comprises 200 test images (of size 481 × 321 or 321 × 481) depicting a variety of scenes, including humans, animals, buildings, and landscapes, along with multiple human-labeled ground truths. Unless otherwise specified, the parameters across all our experiments are fixed, as follows: $100\le K\le 1200$, $M=1$, $\alpha =400$, $\tau =1/32$, $\rho =10,000$, and $L_k=max(\sqrt{N/K},L_0^k)$, where $L_0^k$ represents the initial radius of the superpixel $s_k$, and $\mathrm{MaxIter}=5$. To assess the effectiveness of our methodology, six evaluation metrics are employed: under-segmentation error (UE) [51], achievable segmentation accuracy (ASA) [52], boundary recall (BR) [53], contour density (CD) [25], compactness (CO) [3], and shape regularity criteria (SRC) [4]. The first two metrics evaluate how accurately the superpixels capture image objects in relation to the ground truths, the third metric assesses the alignment of superpixel boundaries with object boundaries, while the latter three metrics measure the regularity of the superpixels according to different criteria.

• Under-segmentation Error (UE)

The UE concerns how each superpixel $S=\left\{s_k\right\}$ is divided by a ground truth segmentation $G=\left\{g_j\right\}$. It is defined as as follows:

$$UE={\displaystyle \frac{1}{N}}\sum _{g_j\in G}\sum _{s_k\in S}min\left\{|s_k\cap g_j|,|s_k\setminus g_j|\right\}.$$

• Achievable Segmentation Accuracy (ASA)

The ASA calculates the average of the largest overlaps between each superpixel and the corresponding ground truth segment, as follows:

$$ASA={\displaystyle \frac{1}{N}}\sum _{s_k\in S}\underset{g_j\in G}{max}\left\{|s_k\cap g_j|\right\}.$$

• Boundary Recall (BR)

The BR measures the proportion of ground truth boundaries that are captured by superpixel boundaries. A ground truth boundary pixel p is classified as a true positive (TP) if any superpixel boundary falls within a window of size ${(2r+1)}^2$ centered at p. Conversely, if no superpixel boundary is found within this window, the pixel p is classified as a false negative (FN) boundary pixel. For all experiments, $r=1$ is used. The calculation of BR is performed as follows:

$$BR={\displaystyle \frac{TP}{TP+FN}},$$

where $TP$ and $FN$ represent the number of true positive and false negative ground truth boundary pixels, respectively.

• Contour Density (CD)

The BR metric might overestimate the performance of irregular superpixel algorithms due to the increased number of boundary pixels in such segmentations. To assess the credibility of the BR measure, the CD is also reported, defined as the ratio of the total number of superpixel boundary pixels B to the total number of image pixels N:

$$CD=B/N.$$

• Compactness (CO)

The CO calculates the average isoperimetric quotients of the superpixels throughout the image, as shown below:

$$CO={\displaystyle \frac{1}{N}}\sum _{k=1}^K\left|s_k\right|{\displaystyle \frac{4\pi A\left(s_k\right)}{P^2\left(s_k\right)}},$$

where $A\left(s\right)$ and $P\left(s\right)$ represent the area and the perimeter of s, respectively. The CO favors superpixels that have circular shapes and smooth boundaries. Note that the perimeter $P\left(s\right)$ is computed using the $L_2$ distance, meaning that the distance between two diagonally adjacent pixels is $\sqrt{2}$ instead of 2, which is more favorable to superpixels with diagonal boundaries.

• Shape Regularity Criteria (SRC)

The study in [4] revealed that the CO measure is overly sensitive to boundary smoothness, making it difficult to differentiate regular shapes (like circles) with noisy boundaries from less regular shapes (such as ellipses). This limitation is addressed by the SRC metric, which compares a superpixel s to its convex hull $H\left(s\right)$. The SRC is computed, as follows:

$$SRC=\sum _k{\displaystyle \frac{|s_k|}{N}}{\displaystyle \frac{CC\left(H\left(s_k\right)\right)}{CC\left(s_k\right)}}V\left(s_k\right),$$

where $CC\left(s\right)=P\left(s\right)/A\left(s\right)$ and $V\left(s\right)=\sqrt{{\lambda }_1/{\lambda }_2}$, with ${\lambda }_1\le {\lambda }_2$ being the two eigenvalues of the covariance matrix of the x, y-coordinates in s. Note that the ratio $V\left(s\right)$ is computed differently from [4] to ensure rotation invariance.

5.2. Effectiveness of Anisotropy

The effectiveness of anisotropy is first examined on the performance of AVFS. For comparison, four contour detectors are selected: (i) the Prewitt detector, (ii) the Canny edge detector, (iii) the ultrametric contour map (UCM), and (iv) the structured edge (SE) detector [54]. Additionally, two special anisotropy cases are used as the baselines. The first is the trivial anisotropic (i.e., isotropic), where $a_{i,j}=1$ for any neighboring pixels i and j. The second is “perfect” anisotropy, where $a_{i,j}=0$ if and only if there is a ground truth (GT) edge between pixels i and j.

To illustrate the trade-off between boundary adherence and regularity, multiple superpixel segmentations are generated by varying the compactness parameter with $0.01\le M\le 100$ and setting $K=1200$. Figure 1 displays the results, plotting the regularity metrics against boundary recall. Notably, the perfect anisotropy (AVFS-GT) significantly outperforms the isotropic model (IVFS), achieving better regularity at the same BR values. This suggests that the proposed AVFS algorithm can effectively balance boundary adherence and regularity when appropriate anisotropy is applied. Additionally, even basic contour detectors like Prewitt and Canny improve results compared to IVFS, while more advanced detectors like UCM and SE yield even better performance. This indicates that although the performance of AVFS is influenced by the quality of contour detectors, simple detectors are sufficient for improvement over the IVFS model, highlighting the value of anisotropy in our approach.

Figure 2, Figure 3 and Figure 4 present zoomed-in comparisons of IVFS and AVFS. In Figure 2, AVFS successfully captures small image objects, while IVFS misses parts of these objects due to over-regularization. Figure 3 demonstrates that AVFS effectively captures irregular shapes, such as the face in the second row, whereas IVFS struggles with over-smoothed boundaries. Additionally, Figure 4 highlights that AVFS adheres better to image edges, particularly at superpixel corners, which are overly smoothed in IVFS. Overall, AVFS preserves image details more effectively than IVFS without compromising regularity, underscoring the effectiveness of ATV in our model.

5.3. Comparative Results

In this subsection, the proposed AVFS algorithm is compared with the state-of-the-art regular superpixel methods, including hard-clustering-based SLIC and SCALP, soft-clustering-based GMMSP and fuzzy-SLICNC [19]. Boundary-based ETPS and SCAC [55] are also included. Note that the SE detector is used to calculate anisotropy.

Table 1. Quantitative comparison of the proposed AVFS and regular superpixel algorithms.

$\mathit{K}=400$	UE	ASA	BR	CO	CD	SRC	Time(s)
SLIC	0.088	0.955	0.765	0.505	0.212	0.666	0.179
ETPS	0.086	0.957	0.787	0.519	0.226	0.682	0.721
Fuzzy SLICNC	0.086	0.957	0.766	0.500	0.213	0.642	1.036
SCALP	0.084	0.958	0.767	0.517	0.204	0.653	1.271
SCAC	0.082	0.959	0.782	0.528	0.214	0.680	0.186
GMMSP	0.089	0.955	0.765	0.489	0.228	0.627	0.299
AVFS	0.083	0.958	0.791	0.544	0.212	0.689	0.416
$K=1200$	UE	ASA	BR	CO	CD	SRC	Time(s)
SLIC	0.059	0.970	0.894	0.527	0.358	0.669	0.220
ETPS	0.056	0.972	0.904	0.541	0.370	0.689	0.974
Fuzzy SLICNC	0.056	0.972	0.900	0.539	0.354	0.662	1.093
SCALP	0.060	0.970	0.880	0.508	0.327	0.633	0.845
SCAC	0.054	0.973	0.904	0.563	0.348	0.696	0.309
GMMSP	0.055	0.972	0.900	0.528	0.375	0.655	0.280
AVFS	0.055	0.972	0.906	0.577	0.345	0.700	0.655

presents the average performance metrics of all algorithms for $K=400$ and $K=1200$. The proposed AVFS outperforms all other algorithms in SRC and CO, and ranks second in CD, highlighting the effectiveness of ATV in enhancing superpixel regularity. Additionally, AVFS achieves the highest BR, demonstrating its ability to maintain boundary adherence to image edges while improving superpixel regularity. In terms of segmentation accuracy, AVFS performs well on UE and ASA metrics, being only slightly behind the recently developed SCAC. Overall, these results indicate that our method effectively distinguishes between different image components, with ATV causing only a minor reduction in the segmentation accuracy.

To further investigate the performance of regular superpixel algorithms in balancing boundary adherence and regularity, the parameters for each algorithm are fine-tuned and only the best results are reported. These findings are displayed in Figure 5 for $K=400$ and Figure 6 for $K=1200$. The figures clearly demonstrate that our AVFS significantly outperforms other algorithms in both CO and CD as the BR–CO and BR–CD curves for AVFS consistently surpass those of the competing algorithms. Additionally, AVFS achieves the highest SRC across a broad range of BR ($0.6\le \mathrm{BR}\le 0.8$ for $K=400$ and $0.75\le \mathrm{BR}\le 0.9$ for $K=1200$), though SCALP and SCAC perform slightly better at lower and higher BR values. Figure 7 illustrates the average performance metrics across varying numbers of superpixels K, from 100 to 1200, revealing that AVFS excels in BR and all regularity metrics while maintaining competitive in accuracy metrics. In

Table 2. Quantitative comparison of the proposed AVFS and irregular superpixel algorithms.

$\mathit{K}=400$	UE	ASA	BR	CO	CD	SRC	Time(s)
SH	0.093	0.953	0.883	0.161	0.358	0.259	0.095
DISF	0.074	0.963	0.903	0.163	0.334	0.268	1.157
DAL-HERS	0.096	0.952	0.900	0.111	0.414	0.202	0.496
AVFS	0.080	0.959	0.877	0.370	0.274	0.485	0.549
$K=1200$	UE	ASA	BR	CO	CD	SRC	Time(s)
SH	0.062	0.969	0.948	0.216	0.503	0.317	0.070
DISF	0.052	0.974	0.961	0.200	0.472	0.305	1.398
DAL-HERS	0.063	0.968	0.959	0.145	0.562	0.242	0.516
AVFS	0.054	0.973	0.935	0.475	0.389	0.580	0.786

, AVFS is compared with other superpixel algorithms that have limited regularity control, including graph-based SH [35], DISF [30], and the deep-learning-based DAL-HERS [36]. Although the irregular algorithms may yield slightly better accuracy metrics than AVFS, the regularity metrics clearly indicate that AVFS produces significantly more regular superpixels. Finally, the visual examples in Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 demonstrate that AVFS effectively generates regular superpixels that adhere well to object boundaries.

To further validate our proposed algorithm, experiments were conducted using the Stanford Background Dataset (SBD dataset) [56], which consists of 715 images (approximately 320 × 240 pixels) depicting urban and rural scenes from various public image collections. The results, presented in

Table 3. Quantitative comparison of the proposed AVFS and regular superpixel algorithms (SBD dataset).

$\mathit{K}=400$	UE	ASA	BR	CO	CD	SRC	Time(s)
SLIC	0.058	0.942	0.765	0.408	0.277	0.546	0.153
ETPS	0.061	0.939	0.725	0.503	0.324	0.659	0.699
Fuzzy SLICNC	0.056	0.944	0.810	0.464	0.305	0.586	1.240
SCALP	0.059	0.941	0.792	0.437	0.284	0.559	1.361
SCAC	0.056	0.944	0.727	0.515	0.315	0.646	0.176
GMMSP	0.055	0.945	0.796	0.491	0.335	0.619	0.215
AVFS	0.054	0.946	0.856	0.557	0.297	0.687	0.508
$K=1200$	UE	ASA	BR	CO	CD	SRC	Time(s)
SLIC	0.040	0.960	0.896	0.507	0.435	0.631	0.271
ETPS	0.040	0.960	0.908	0.531	0.507	0.671	0.843
Fuzzy SLICNC	0.038	0.962	0.937	0.574	0.459	0.681	1.295
SCALP	0.041	0.959	0.890	0.490	0.417	0.611	0.906
SCAC	0.038	0.962	0.901	0.573	0.478	0.681	0.341
GMMSP	0.039	0.961	0.911	0.574	0.478	0.697	0.197
AVFS	0.037	0.963	0.937	0.598	0.471	0.708	0.755

, demonstrate the superior performance of our algorithm across multiple evaluation metrics, including UE, ASA, BR, CO, and SRC. These findings highlight our algorithm’s exceptional ability to adhere to image edges while improving superpixel regularity. In terms of accuracy metrics, AVFS also outperforms other algorithms. Additionally, Figure 14 shows the average performance metrics for varying numbers of superpixels (from 100 to 1200). Overall, these results consistently illustrate AVFS’s superior segmentation accuracy, boundary adherence, and superpixel regularity compared to competing algorithms.

5.4. Computational Cost

All experiments were conducted on a laptop equipped with an Intel Core i7-6700HQ (Intel Corporation, Santa Clara, CA, USA) and an NVIDIA GeForce GTX 960 M (NVIDIA Corporation, Santa Clara, CA, USA). The proposed algorithm has a computational complexity of $O(NlogN)$, with the $logN$ factor arising from sorting N queues of pixels when calculating the fuzzy membership function. Table 1, Table 2 and Table 3 present the average computation times for all algorithms at $K=400$ and $K=1200$. As can be seen, SH and SLIC are the fastest, as the former uses a region-merging technique that grows regions in a Borůvka fashion, while the latter employs a standard k-means clustering with simple image features. Other algorithms typically generate superpixels with computation times of around or less than 1.0 s per image. The computational cost of our proposed method is primarily influenced by the optimization process, resulting in a total superpixel generation time of approximately 0.6 s, which is deemed satisfactory.

6. Conclusions

This paper presents a fuzzy superpixel segmentation algorithm based on content-adaptive anisotropic total variation regularization. By aligning superpixels with image contours, the model enhances regularity while preserving fine details in regions with high contour density. Unlike conventional methods that use non-adaptive regularization, our approach maintains strong adherence to image edges without sacrificing regularity. As a result, the geometry of the generated superpixels can serve as informative features for downstream tasks. In contrast, many existing models produce superpixels that are either overly irregular or oversmoothed, limiting the usefulness of their geometry. Consequently, our algorithm provides a more informative image representation that can improve the performance of a wide range of superpixel-based applications.

To optimize the proposed functional effectively, an efficient alternating direction method of multipliers in conjunction with an enhanced Chambolle’s fast duality projection algorithm are adopted. The experimental results show that our method outperforms state-of-the-art approaches, producing more regular superpixels with better boundary adherence and higher segmentation accuracy.

Nevertheless, our algorithm has several limitations. First, it must solve the total variation regularization sub-problem (8), which can be computationally expensive. The fuzzy membership sub-problem (13) requires the local sorting of the sequence ${\left\{z_{i,k}\right\}}_k$, which can become a bottleneck in practice. In addition, the anisotropy term depends on the chosen edge detector and can be degraded by image noise. To improve computational efficiency, fast approximate solvers for each sub-problem are worth exploring. Moreover, an improved anisotropy design via coupled optimization could enhance the adaptability to the image contents. Finally, integrating the algorithm into a deep learning framework may enable richer feature representations and more sophisticated superpixel modeling.