Interactive Segmentation for Medical Images Using Spatial Modeling Mamba

Abstract

Interactive segmentation methods utilize user-provided positive and negative clicks to guide the model in accurately segmenting target objects. Compared to fully automatic medical image segmentation, these methods can achieve higher segmentation accuracy with limited image data, demonstrating significant potential in clinical applications. Typically, for each new click provided by the user, conventional interactive segmentation methods reprocess the entire network by re-inputting the click into the segmentation model, which greatly increases the user’s interaction burden and deviates from the intended goal of interactive segmentation tasks. To address this issue, we propose an efficient segmentation network, ESM-Net, for interactive medical image segmentation. It obtains high-quality segmentation masks based on the user’s initial clicks, reducing the complexity of subsequent refinement steps. Recent studies have demonstrated the strong performance of the Mamba model in various vision tasks; however, its application in interactive segmentation remains unexplored. In our study, we incorporate the Mamba module into our framework for the first time and enhance its spatial representation capabilities by developing a Spatial Augmented Convolution (SAC) module. These components are combined as the fundamental building blocks of our network. Furthermore, we designed a novel and efficient segmentation head to fuse multi-scale features extracted from the encoder, optimizing the generation of the predicted segmentation masks. Through comprehensive experiments, our method achieved state-of-the-art performance on three medical image datasets. Specifically, we achieved 1.43 NoC@90 on the Kvasir-SEG dataset, 1.57 NoC@90 on the CVC-ClinicDB polyp segmentation dataset, and 1.03 NoC@90 on the ADAM retinal disk segmentation dataset. The assessments on these three medical image datasets highlight the effectiveness of our approach in interactive medical image segmentation.

1. Introduction

Precise and reliable segmentation of organs or lesions from medical imaging data (such as CT and MRI) is essential for a wide range of clinical applications. Despite significant advancements in deep learning-based automatic segmentation methods [1,2] over the years, including the success of U-Net [3,4] and Transformer-based approaches [5,6], achieving consistent and accurate segmentation in complex pathological conditions remains challenging for clinical applications. These challenges primarily arise from the inherent characteristics of medical images, including poor image quality, variations in imaging modalities, and inter-patient differences, which make segmentation more difficult. In contrast, interactive segmentation methods [7], which leverage users’ expertise and experience to obtain more accurate segmentation results, are more practical in clinical applications. Interactive segmentation allows additional user prompts (such as bounding boxes [8,9,10], scribbles [11,12,13], and clicks [14,15,16,17]) to effectively segment target objects in images. Among these, click-based methods are the most widely used due to their simplicity and well-established training and evaluation protocols. In this study, we use widely adopted click-based interactions to guide the network segmentation process, with positive clicks in the foreground region and negative clicks in the background region.

Although interactive segmentation typically results in better segmentation quality, the optimal interactive segmentation method should satisfy the following criteria: (1) achieve precise segmentation with minimal user interaction to reduce the user’s burden; (2) be highly efficient to provide real-time responses when processing data; and (3) adapt well to various objects and imaging techniques. However, current interactive segmentation methods struggle to simultaneously meet all these competing requirements.

Recent advances in click-based interactive methods focus on two orthogonal directions: (1) developing more efficient backbone networks and (2) exploring refinement modules built on top of the backbones. For the former, various hierarchical backbones have been developed. For instance, FocusCut [18] and RITM [17] use ConvNets as the foundational segmentation network; iSegformer [19] proposes a segmentation network combining Swin Transformer and a lightweight multilayer perceptron (MLP); and SimpleClick [20] uses ViT with a simple feature pyramid as the backbone for interactive segmentation. Compared to traditional methods, these approaches often rely on large backbones to achieve satisfactory segmentation results, increasing the training complexity of the model parameters. To further enhance segmentation performance, a number of refinement modules have been proposed, such as click refinement [18,21] and click imitation [22]. Addressing the efficiency of refinement modules, EMC-Click [23] proposes an effective method using a lightweight mask correction network, but it still relies on a large backbone network, failing to consider the model’s overall performance. In this research, we explore the first direction, designing an efficient backbone network based on a general pipeline for interactive segmentation methods to better meet practical application needs.

Transformers are excellent at capturing long-term relationships, but they have drawbacks for high-resolution biomedical images due to the high computational cost associated with the quadratic scaling of the self-attention mechanism with input size. Conversely, Mamba [24], derived from State Space Models (SSMs) [25], scales with sequence length in a linear or nearly linear manner while maintaining the ability to represent long-range dependencies. They improve the efficiency of training and inference by using hardware-aware algorithms and selection mechanisms, providing a promising solution for efficiently handling long sequences in medical image segmentation. Recent studies have explored the application of Mamba in computer vision (CV). For instance, Vision Mamba [26] proposed a general-purpose visual backbone with bidirectional Mamba blocks, which performed well across various vision tasks. In contrast, VMamba [27] developed a Mamba-based visual backbone with hierarchical representations and introduced a cross-scan module to address direction sensitivity issues arising from the differences between 1D sequences and 2D images. Task-specific architectures using Mamba blocks based on nnUNet [28] and Swin-UNETR [29] are proposed by U-Mamba [30] and SegMamba [31], respectively. These models have achieved notable success in vision tasks, further supporting Mamba’s superior performance as a viable alternative to Transformers in visual modeling. However, the potential application of Mamba-based backbone networks for interactive segmentation has not yet been explored. Therefore, our work investigates the segmentation performance of Mamba models with feature inputs fused from clicks and images.

Our main contributions can be summarized as follows:

• We propose ESM-Click, the first interactive medical image segmentation method using the Mamba-based backbone network ESM-Net. Our method reduces the computational cost of the segmentation network while maintaining high-quality segmentation results.
• We design a Spatial Feature Enhancement Convolution (SAC) module to enhance the spatial expression ability of fused features from 2D image information and interaction information, using a gating mechanism to learn dynamic feature selection for each channel and spatial position.
• We construct a multi-scale feature-fusion module as part of the segmentation head, combining the advantages of KAN in nonlinear modeling ability and interpretability to improve the model’s segmentation accuracy.
• Comprehensive evaluations on three medical image datasets demonstrate that our ESM-Click exhibits good robustness, paving the way for Mamba’s application in interactive segmentation.

2. Related Work

2.1. Interactive Image Segmentation

Interactive segmentation (IS) is a highly active research area focusing on the dynamic interaction between humans and machines. Traditional interactive segmentation methods [10,12,32,33] employ graphs defined on image pixels to address segmentation challenges. However, these approaches rely solely on low-level features, rendering them inadequate for handling complex environments. Given the success of ConvNets in extracting robust image features, several methods have adopted successful backbone networks to enhance segmentation outcomes. The first deep learning approach to interactive segmentation was presented by DIOS [7], which proposed a classic sampling strategy to simulate positive and negative clicks for model training. Based on the existing erroneous prediction areas, IT IS [34] proposed a novel online iterative sampling technique that was refined in RITM [17] using less computational resources. iCMFormer [35] proposed a cross-modal Transformer that effectively utilizes click information to guide model segmentation, producing more robust results. FDRN [36] introduced a decoupled recovery strategy from three perspectives, avoiding redundant computation by preventing feature extraction from the source image in each interaction. ClickAttention [37] addressed the sparse nature of clicks by proposing a click attention mechanism to enhance the propagation of click information across feature maps. MST [38] introduced a multi-scale token adaptive algorithm to improve the segmentation of objects of varying sizes and incorporated contrastive loss to enhance robustness. In addition to global segmentation, f-BRS [16] minimizes the difference between the original image and the predicted mask for optimalization, achieving high-quality segmentation results through further refinement. FocalClick [21] and FocusCut [18] effectively improved segmentation results from the perspective of local refinement. GPCS [39] formulated IS as a Gaussian process classification model for each individual image. EMC-Click [23] designed an efficient mask-refinement network based on the backpropagation process, reducing the computational burden of multiple user interactions and thus improving the efficiency of the model. However, these methods typically rely on a large backbone segmentation network, increasing the difficulty of applying interactive segmentation in clinical applications for medical images. Therefore, we explored efficient backbone network design methods equipped with the Mamba module, which has fewer parameters and significantly speeds up the interactive process.

2.2. Visual Applications of Mamba

To address the challenges associated with long-sequence modeling, Mamba [24], which originated from state-space models (SSM) [25], was developed to handle long-range dependencies. Inspired by the Mamba model, VMamba [27] designed a CSM module to connect 1-D array scanning with 2-D plane traversal, maintaining model fitting capability at linear complexity and demonstrating excellent performance in visual tasks. U-Mamba [30] extended this approach by integrating Mamba layers into the nnUNet [28] encoder, enhancing general medical image segmentation. However, due to U-Mamba’s combination of CNN and SSM as fundamental modules and reliance on nnUNet’s pre- and post-processing capabilities, the model consumes substantial computational resources during training, making it unsuitable as a backbone network for interactive segmentation. Additionally, Vision Mamba [26] combines bidirectional SSM for data-dependent global visual context modeling and positional embedding for position-aware visual understanding. However, Vision Mamba’s design, inspired by ViT [40], segments images into patch sequences, leading to a loss of click information and a reduction in the positional awareness of image features. Our comparative experiments indicated that such structures perform poorly as backbone networks in interactive segmentation models, making them equally unsuitable for interactive segmentation methods. Building on previous explorations of the Mamba model in visual tasks, we investigated its performance in interactive image segmentation, focusing on backbone network structure and model efficiency. Additionally, we developed an efficient interactive segmentation backbone network that incorporates a spatially enhanced Mamba module.

3. Method

We propose ESM-Click to address the efficiency gap between existing methods and practical clinical applications. First, we present an overview of our general pipeline and model structure, followed by a detailed description of the proposed efficient backbone network, ESM-Net.

3.1. Overview Architecture

Our pipeline, depicted in Figure 1, includes a lightweight mask-refinement network and a primary segmentation network called ESM-Net. The overall process is as follows: the original image and user clicks are input into the primary network to generate an initial mask, which is then refined through multiple iterations of backpropagation in the refinement network to achieve satisfactory segmentation results.

Specifically, similar to FCA-Net [15], the initial clicks are used to locate the target object, and the initial segmentation result is crucial for optimizing the subsequent network. Based on the user’s first click, we input the click information into the segmentation network to generate an initial mask and extract relevant features of the target. From the second click onwards, the mask-refinement network is iteratively executed, gradually optimizing the initial mask, and ultimately producing a satisfactory prediction with a minimal number of clicks.

Unlike traditional automatic segmentation methods, interactive segmentation networks integrate the image, click map, and predicted mask to perform segmentation. The click map is typically represented as a binary disc or distance map to encode positive and negative clicks. The encoded click map is input into the segmentation network along with the image, and the user determines the next click position based on the current segmentation result.

ESM-Net handles the segmentation task for the first click, utilizing a standard encoder-decoder architecture that effectively captures both local features and global contextual information. The encoder integrates spatially enhanced convolution (SAC) modules and Mamba modules, leveraging the lightweight MBConv modules from EfficientNetv2 [41] for downsampling. During upsampling, we designed a segmentation head based on KAN [42], which fuses features from four scales to generate the segmentation result. Experiments have validated the effectiveness of this segmentation head.

3.2. Proposed ESM-Net

As illustrated in Figure 2, the overall architecture of ESM-Net is introduced. Specifically, ESM-Net includes an input stem, four downsampling stages for extracting multi-scale features, an upsampling decoder, and a segmentation head for outputting the segmentation results. The detailed design and implementation of each module are described below.

3.2.1. Mamba Block

Global and multi-scale feature modeling play a decisive role in medical image segmentation. Typically, Transformers are employed for global feature extraction; however, handling excessively long feature sequences often results in computational overhead. In contrast, VMamba [27] effectively combines the advantages of CNNs and ViTs, utilizing cross-scanning modules to traverse the spatial domain, addressing the issue of image-orientation sensitivity. Additionally, Mamba’s block-sequence traversal enhances the flow of information during feature extraction, thereby improving the quality of visual representations, which results in outstanding performance in image segmentation tasks. Given that medical images inherently suffer from noise, distortion, and quality issues, simply converting images into block sequences using methods like ViT often results in further loss of detail. The superior performance of Mamba over Transformers in medical image segmentation raises expectations for its application in interactive medical image segmentation. Moreover, Transformer-based networks rely on attention mechanisms to achieve better contextual understanding. However, the downside is the large computational cost, which increases quadratically with the output size. In contrast, Mamba offers faster inference speed, enabling computational costs to scale linearly or near-linearly with sequence length. This characteristic makes it well-suited for building efficient segmentation networks based on Mamba, addressing the prediction time limitations of interactive segmentation methods. Additionally, the Mamba model employs a hardware-aware algorithm that fully leverages the memory hierarchy of GPUs, enhancing computation speed. This algorithm combines the recursive computational efficiency of RNNs with the parallel processing advantages of CNNs, resulting in significantly higher efficiency when handling long sequence data. While CNNs offer faster processing speeds, they have limitations when dealing with complex structures and patterns, where Mamba excels.

Leveraging the aforementioned advantages, we employ the Mamba module for global feature extraction, ensuring high efficiency in both training and inference phases. Our core Mamba module is derived from the VMamba, as illustrated in Figure 2b. Initially, the input undergoes dimensional transformation, converting 2D features into feature sequences. Following layer normalization, the sequences are split into two paths: one path passes through a linear embedding layer followed by SiLU activation, while the other path passes through linear layers, separable convolutions, and activation functions. The 2D-Selective-Scan module is then used to further extract features, and the results of the two branches are multiplied element-wise to obtain the merged features. Finally, residual connections are used to add the input and output, yielding the output of the Mamba module. The computational process at each stage can be defined as:

$${{\hat{F}}_n}^l=SAC({F_n}^l), {{\tilde{F}}_n}^l=Mamba(LN({{\hat{F}}_n}^l))+{{\hat{F}}_n}^l,$$

(1)

where SAC denotes the proposed Spatial Augmented Convolution module, which will be discussed in the following sections. LN represents layer normalization.

3.2.2. Spatial Augmented Convolution (SAC) Module

The Mamba module compresses 2D features into a feature sequence to model feature dependencies, which can lead to deficiencies in spatial relationship representation. To address this issue and enhance the spatial representation of features prior to the Mamba layer, we designed a Spatially Augmented Convolution module (SAC), which incorporates a gating-like mechanism and uses depth-wise separable convolution blocks with varying kernel depths as its core components.

Gating mechanisms are typically used in Recurrent Neural Network [43] (RNN) units to process sequential data, which aligns with the feature sequences in our Mamba module. By introducing a gating mechanism to control the flow of information, it enhances the memory and representation capabilities of the model. During training, the network adjusts the parameters of its gating mechanism based on the relationship between the input data and target labels. This allows the model to selectively retain and transmit important information while filtering out irrelevant or redundant data. We implement the gating mechanism using clicks and nonlinear activation functions, enabling the network to flexibly filter and process information, thereby improving the overall task performance.

As illustrated in Figure 2a, the input 2D features are fed into two composite convolution modules with kernel sizes of 1 × 1 and 3 × 3. These are followed by standard BatchNormalization layers and hswish activation functions. The features from the two paths are then multiplied pixel-wise to control information transmission akin to a gating mechanism. The collaborative operation of these two paths effectively regulates feature updates, thereby capturing the spatial dependencies of the data. A final composite convolution block further integrates the features, leveraging residual features to reuse input features and enhance the overall feature representation. The entire computational process can be formulated as:

$$SAC(y)=y+C^{3\times 3}(C^{3\times 3}\cdot C^{1\times 1}(y)),$$

(2)

where y denotes the input 2D features, C represents the depth-wise separable convolution, and the superscripts indicate the respective convolution kernel parameters. The activation functions and normalization layers have been omitted for brevity.

3.2.3. KANSegHead

Kolmogorov–Arnold Networks [42] (KAN) represent an innovative shift in neural network design, challenging traditional concepts such as the Multilayer Perceptron (MLP). The core of KAN is rooted in an elegant and abstract mathematical theorem by Kolmogorov and Arnold. This theorem states that any multivariate continuous function can be represented as a nested superposition of univariate continuous functions. By adapting the Kolmogorov–Arnold theorem, KAN employs learnable univariate functions and additive operations instead of adjusting the weights and biases of linear combinations for each node input, thus transforming this theorem into a neural network architecture. In KANs, each weight is essentially a small function. Unlike traditional nodes that apply fixed nonlinear activation functions, each learnable activation function in KAN edges processes inputs and produces outputs.

KAN convolutions are similar to traditional convolutions, but instead of applying dot products between corresponding pixels in the kernel and image, they apply learnable nonlinear activation functions to each element before summing them. The KAN convolution kernel is equivalent to a KAN linear layer with four inputs and one output neuron. Compared to traditional convolution methods, KAN convolution layers reduce the number of parameters by almost half while maintaining comparable accuracy, making them suitable for our network.

In our work, we implement a lightweight segmentation head, KANSegHead, using efficient and expressive convolution layers based on KAN theory, as illustrated in Figure 2c. Initially, our segmentation head receives image features at four different scales, modifies the channel numbers and dimensions through convolution operations, and concatenates all feature maps for integration. After a dropout process to prevent over-reliance on local features, the final segmentation result is produced using KAN convolution. The computational process of the entire segmentation head can be described as:

$$F=Conv(Concat(Up(Xi,(H_0,W_0)))),i=1,2,3,4,$$

(3)

$$S=ConvKAN(F,W_{kan},b_{kan}),$$

(4)

where $Up$ represents the upsampling process, $H_0$ and $W_0$ denotes the features at unified spatial dimensions, and $W_{kan}$ and $b_{kan}$ denotes the convolution kernel and bias parameters. In this formula, we describe the process of integrating the four received scales, followed by our ConvKAN layer to output the segmentation results. Our lightweight segmentation head effectively utilizes multi-scale features from the backbone network and completes an efficient segmentation process.

3.3. Loss Function

In the overall model architecture, we supervise the coarse segmentation network using Normalized Focal Loss [44] $L_{nfl}$ and Sigmoid Binary Cross-Entropy loss $L_{sBCE}$. Previous studies have demonstrated that NFL converges faster and outperforms the widely used binary cross-entropy in interactive segmentation tasks. Similar training pipelines have been proposed by RITM [17] and subsequent works [21]. For the mask-refinement part, we follow the FocalClick [21] method by adding a boundary weight of 1.5 to the NFL loss, denoted as $L_{bnfl}$. The total training loss can be expressed by the following Equation (5):

$$L=L_{nfl}+L_{sBCE}+L_{bnfl},$$

(5)

where $L_{nfl}$, $L_{sBCE}$, and $L_{bnfl}$ represent the Normalized Focal Loss, Binary Cross-Entropy loss, and Boundary Normalized Focal Loss, respectively. The entire model, encompassing both stages, is trained in a unified end-to-end framework.

4. Experiments

4.1. Datasets

We evaluated our model on three challenging medical image segmentation datasets: Kvasir-SEG [45], CVC-ClinicDB [46], and ADAM [47].

• Kvasir-SEG [45] is a dataset for pixel-level colon polyp segmentation, containing 1000 gastrointestinal polyp images and their corresponding masks, all annotated and verified by experienced gastroenterologists. Following the official dataset split, we selected 800 images for the training set, 100 images for the validation set, and 100 images for the test set.
• CVC-ClinicDB [46] is the official dataset for the MICCAI 2015 sub-challenge on automatic polyp detection in colonoscopy videos, consisting of 612 static images extracted from colonoscopy video sequences from 29 different series. Out of these, 489 images were used for training, 61 images for validation, and 61 images for testing.
• ADAM [47] was introduced at the ISBI 2020 satellite event, aimed at improving diagnostic capabilities for Age-related Macular Degeneration (AMD). The primary task of ADAM-Task2 is the detection and segmentation of the optic disc in retinal images. The ADAM dataset includes 381 retinal images and their corresponding masks. Consistent with the splits of the previous datasets, 304 images were used for training, 38 for validation, and 38 for testing.

For all three medical image segmentation datasets, we followed an 8:1:1 split ratio for the training, validation, and test sets. All images were resized to 448 × 448 pixels for input into the model.

4.2. Evaluation Metrics

We employed standard metrics commonly used in interactive segmentation to ensure fair comparison. During evaluation, clicks are simulated based on the discrepancies between the previous predicted mask and the ground truth. The first click is positioned at the center of the target, while subsequent clicks are placed at the center of the largest error region. We used the Number of Clicks (NoC) as an evaluation metric to determine the number of clicks needed to reach the target Intersection over Union (IoU). We established two target IoU thresholds: 85% and 90%, denoted as NoC%85 and NoC%90, respectively.

$$I\mathrm{oU}={\displaystyle \frac{TP}{TP+FP+FN}},$$

(6)

$$Dice={\displaystyle \frac{2\times TP}{2\times TP+FP+FN}},$$

(7)

In the above formulas, TP (True Positive) represents the number of pixels predicted as positive that are also labeled as positive, FP (False Positive) refers to the number of pixels predicted as positive but labeled as negative, and FN (False Negative) indicates the number of pixels predicted as negative but labeled as positive. The Dice coefficient is used to measure the similarity between images, while IoU (Intersection over Union) represents the ratio of the overlap between the prediction and the target label. Both metrics range from 0 to 1, with values closer to 1 indicating a higher degree of similarity between the predicted results and the ground truth labels.

Additionally, to validate the effectiveness of our model, we used the average IoU and average Dice corresponding to the first click as evaluation metrics to consistently measure the segmentation quality of a single segmentation. Our goal is to achieve higher Dice and IoU scores with the first-click-guided segmentation to ensure better subsequent click-segmentation performance, ultimately achieving better segmentation quality with fewer clicks.

4.3. Implementation Details

For all datasets, we trained the model for 200 epochs with an initial learning rate of 5 × 10⁻⁴, which was decayed tenfold at the 170th and 190th epochs. The network was trained using images of size 448 × 448 with a batch size of 16. Similar to EMC-Click, we employed image-augmentation techniques such as random resizing within the range of [0.75, 1.40], horizontal flipping, and random adjustments to brightness, contrast, and RGB values. Our models were developed using Python 3.8 and the PyTorch 1.10 [48] framework.

4.4. Comparison with State-of-Art Methods

4.4.1. Computational Analysis

The objective of ESM-Click is to propose a lightweight backbone segmentation network. For the backbone network, maintaining segmentation quality while ensuring efficiency is a critical factor.

Table 1. Comparison of the computational cost across different backbones.

Backbone	FLOPs (G)	Params (M)	SPC/ms
hr18s+ocr	8.2	4.21	95
hr32+ocr	39.27	30.94	146
Segformer-B3	28.9	45.66	119
ViT-Base448	67.17	87.02	127
ESM-Net	4.51	2.46	87

presents a detailed analysis and comparison of model parameters and FLOPs. We categorized the models into four prototypes based on their backbones following the methods of previous works. For a fair comparison, we used the same benchmarks and the same computing environment (NVIDIA RTX 3090, Intel Xeon E5-2678 v3). All models were evaluated using images with a resolution of 448 × 448 pixels to calculate the aforementioned metrics.

In Table 1, we observe that most existing works employ larger base models as the backbone network for interactive segmentation, resulting in high parameter counts and prolonged training times. Compared to these prevalent networks, including purely CNN-based networks such as HRNet, our approach demonstrates significantly lower FLOPs and parameter counts. Additionally, due to our lightweight backbone and mask-refinement networks, our entire network exhibits substantial computational cost advantages. We also calculated the inference time per click, SPC (seconds per click), for different networks during the inference process. Since the number of images in the test set is fixed and the maximum number of clicks for each network is the same, we used the total inference time of the model to compute the time consumed per click, reflecting the real-time performance of the models during testing. As shown in Table 1, ESM-Net requires less inference time, making it suitable for practical applications. Additionally, thanks to the lightweight backbone network and mask-refinement network, our entire network exhibits significant computational cost advantages. The results demonstrate that methods based on the Mamba model outperform Transformer and CNN-based models in terms of computational efficiency and inference time.

4.4.2. Performance Comparison

• Comparison of the Performance of Different Interactive Segmentation Models

Table 2. Comparisons between six benchmarks. NoC@85 and NoC@90, respectively, denote the number of clicks required to reach the IoU of 85% and 90%, and lower is better for the value.

Model	Backbone	Kvasir-SEG		CVC-ClinicDB		ADAM
		NoC@85	NoC@90	NoC@85	NoC@90	NoC@85	NoC@90
CDNet [49]	Resnet34	1.4	2.05	3.7	4.93	1.24	1.95
RITM [17]	HRNet18s	1.59	2.12	1.69	2.75	1.03	1.42
	HRNet32	1.5	1.9	1.67	2.28	1.8	1.26
SimpleClick [20]	ViT-B	2.26	3.16	6.97	9.69	6.18	7.26
AdaptiveClick [50]	ViT-B	1.57	2.13	5.93	8.66	1.21	2.82
FocalClick [21]	HRNet18s-S2	1.49	1.84	1.34	2.07	1.82	2.74
	SegformerB3-S2	1.46	1.87	2.08	3.15	2	2.55
EMC-Click [23]	SegformerB3-S2	1.41	1.81	1.33	1.85	1.21	1.95
	HRNet18s-S2	1.42	1.83	1.3	1.59	1.05	1.08
ESM-Click (Ours)	ESM-Net	1.17	1.43	1.15	1.57	1.03	1.03

presents a comparison of our method with existing interactive segmentation approaches. To ensure a fair comparison, all methods were trained under identical settings. In contrast to state-of-the-art methods such as CDNet [49], RITM [17], FocalClick [21], SimpleClick [20], AdaptiveClick [50], and EMC-Click [23], our method demonstrates superior performance and higher segmentation quality. Notably, our approach requires fewer user clicks to achieve accurate segmentation.

In Figure 3, we display the refinement results of different methods after each click, evaluating the segmentation quality using mIoU and mDice scores. It is evident that ESM-Click consistently achieves higher segmentation accuracy during the refinement process after each click. By leveraging the ESM-Net backbone network, our method accurately identifies the segmentation target with the first click and progressively improves precision with subsequent clicks. In contrast, models based on ViT backbones, such as SimpleClick and AdaptiveClick, perform less effectively on these datasets. Interactive segmentation methods based on HRNet [51] and Segformer [52], such as RITM, FocalClick, and EMC-Click, also perform less effectively compared with our approach. Our method not only ensures precise target segmentation but also minimizes the number of clicks required to reach a certain segmentation quality, aligning with the objectives of our interactive segmentation methodology.

2.

Improvement with a Single Interaction Click

To further evaluate the effectiveness of our method, we compared the average IoU and Dice scores achieved after the first click, as shown in

Table 3. The mIoU and mDice values of the segmentation results after the first click for different methods and their respective backbone networks.

Model	Backbone	Kvasir-SEG		CVC-ClinicDB		ADAM
		IoU (%)	Dice (%)	IoU (%)	Dice (%)	IoU (%)	Dice (%)
FocalClick [21]	HRNet18s-S2	86.17	91.93	89.7	94.3	93.41	96.54
	SegformerB3-S2	86.6	92.19	79.96	87.8	93.64	96.68
EMC-Click [23]	SegformerB3-S2	88.31	93.42	89.77	94.45	88.61	93.89
	HRNet18s-S2	89.01	93.33	93.83	96.46	94.37	97.09
ESM-Click (Ours)	ESM-Net	92.7	95.87	94.96	97.36	95.21	97.84

. The first click, primarily processed by the backbone network, is crucial for locating the segmentation target. Compared to other methods, our model produces higher-quality masks after the first click. This initial high-quality mask ensures that subsequent mask refinements yield better results.

4.5. Qualitative Analysis

Figure 4 illustrates examples of interactive segmentation performed by the ESM-Click model on the Kvasir-SEG, CVC-ClinicDB, and ADAM datasets. We chose challenging cases from each dataset during testing as segmentation examples, displaying segmentation results with different numbers of clicks for comparative analysis. From the segmentation results, ESM-Click consistently achieves higher segmentation accuracy throughout the refinement process. Our model achieves good segmentation performance with just one user-guided click and progressively refines the mask with subsequent clicks. This leads to a final prediction closely approximating the ground truth. The high-quality segmentation achieved with the first click allows subsequent clicks to focus on refining edges and correcting specific areas. This is particularly beneficial for segmenting objects with regular sizes, such as the optic disc images in the ADAM dataset. From the results of the challenging cases illustrated, our model shows significant advantages in segmenting irregular, uneven, and multi-target objects.

4.6. Ablation Studies

We conducted ablation experiments on the new modules of the backbone network to assess the effectiveness of each new module in the backbone network, with the results displayed in

Table 4. Ablation study of the proposed ESM-Net backbone components, using NoC@85, NoC@90, and mIoU and mDice scores for the first click as evaluation metrics. Our ablation experiments revealed that using the same number of Mamba layers as the downsampling convolution modules at each stage significantly enhances model performance. To ensure model efficiency, the spatial augmentation module is applied before all Mamba modules. Additionally, the experiments indicate that Mamba modules require effective residual connections to maintain accuracy.

Model	Kvasir-SEG				CVC-ClinicDB
	NoC@85	NoC@90	IoU(%)	Dice (%)	NoC@85	NoC@90	IoU (%)	Dice (%)
MBConv	1.96	2.59	89.49	93.99	1.36	1.97	88.55	93.69
mamba*1	1.66	2.22	91.01	94.85	1.33	1.82	89.47	94.25
mamba*depth-4	1.2	1.58	91.81	95.31	1.2	1.67	91.36	95.38
Mamba*depth-4+SAC	1.21	1.44	92.24	95.62	1.16	1.74	92.62	96.1
Mamba+SAC+KANSegHead	1.17	1.43	92.7	95.87	1.15	1.57	94.96	97.36

. These experiments focused on the challenging Kvasir-SEG and CVC-ClinicDB datasets.

Ablation of Mamba and SAC Modules To comprehensively evaluate the SAC and Mamba modules, we used a baseline network that consisted of the MBConv module for downsampling, along with simple upsampling and segmentation heads. Starting with this baseline, we sequentially added each module and evaluated segmentation performance across datasets. As shown in the table, the Mamba module significantly improves global information modeling, enabling users to achieve the specified IoU with fewer clicks and accurately enhancing target segmentation performance with a single click. Through ablation of the number of Mamba layers, we determined that maintaining the same number of Mamba layers as MBConv layers resulted in optimal performance. Consequently, we ensured that each downsampling module of the network contained an equal number of Mamba modules and MBConv layers.

Ablation of KANSegHead Module To assess the effectiveness of the KANSegHead, we compared it with a simple segmentation head that used a single 2D convolution layer to output the mask from the fused features. Compared to this simple segmentation head, our KANSegHead significantly enhances the relationships, resulting in higher-quality mask outputs. Specifically, we calculated the Dice and IoU scores for the first segmentation result and the ground truth, as shown in Table 4. The results indicate that our segmentation head provides superior segmentation performance.

5. Conclusions

In this paper, we propose an efficient segmentation method, ESM-Click, based on click interaction, which has been successfully applied to interactive medical image segmentation tasks. The experimental results demonstrate that this method exhibits strong segmentation performance on two polyp segmentation datasets and one optic disc segmentation dataset. Compared to traditional interactive segmentation models, the proposed ESM-Net backbone significantly reduces computational complexity, enhancing feasibility for real-time applications. However, the model remains dependent on high-quality input data, and its performance tends to degrade in the presence of substantial noise or low-contrast regions. To further enhance the practical utility of this method, future research will focus on incorporating more adaptive mechanisms and extending the approach to multimodal or 3D data processing.

Moreover, we believe that the proposed method in this study is not only applicable to polyp and optic disc segmentation but also holds potential for other medical image processing tasks, such as tumor detection and organ segmentation. Future studies could evaluate the method across a wider range of medical scenarios to further validate its broader applicability.