Prototype-Guided Promptable Retinal Lesion Segmentation from Coarse Annotations

Abstract

Accurate segmentation of retinal lesions is critical for the diagnosis and management of ophthalmic diseases, but pixel-level annotation is labor-intensive and demanding in clinical scenarios. To address this, we introduce a promptable segmentation approach based on prototype learning that enables precise retinal lesion segmentation from low-cost, coarse annotations. Our framework treats clinician-provided coarse masks (such as ellipses) as prompts to guide the extraction and refinement of lesion and background feature prototypes. A lightweight U-Net backbone fuses image content with spatial priors, while a superpixel-guided prototype weighting module is employed to mitigate background interference within coarse prompts. We simulate coarse prompts from fine-grained masks to train the model, and extensively validate our method across three datasets (IDRiD, DDR, and a private clinical set) with a range of annotation coarseness levels. Experimental results demonstrate that our prototype-based model significantly outperforms fully supervised and non-prototypical promptable baselines, achieving more accurate and robust segmentation, particularly for challenging and variable lesions. The approach exhibits excellent adaptability to unseen data distributions and lesion types, maintaining stable performance even under highly coarse prompts. This work highlights the potential of prompt-driven, prototype-based solutions for efficient and reliable medical image segmentation in practical clinical settings.

1. Introduction

Accurate segmentation of retinal lesions in fundus images is crucial for the diagnosis and management of ocular diseases such as diabetic retinopathy (DR). In recent years, deep learning models, particularly those based on U-Net [1] and its variants [2,3,4], have achieved remarkable, even expert-level, performance in this task. However, the success of these fully supervised approaches critically depends on the availability of large-scale, meticulously annotated datasets in which every lesion pixel is precisely delineated. Constructing such datasets remains a well-recognized bottleneck in medical imaging; it demands substantial manual effort from expert ophthalmologists, resulting in high costs and inefficiencies. This significantly hinders the widespread clinical adoption of these advanced deep learning models. A pragmatic alternative to labor-intensive, pixel-wise annotation is the use of coarse annotations, where clinicians provide simple geometric shapes—such as circles or ellipses—to approximate lesion locations. This approach is over six times more efficient than detailed pixel-level labeling [5]. While coarse prompts effectively capture the core location and approximate scale of lesions, they inevitably lack boundary precision. This limitation raises a critical research question: how can we harness the efficiency of coarse annotations to achieve the segmentation accuracy required for clinical practice?

To address this challenge, we propose a novel promptable segmentation paradigm based on prototype learning for accurate retinal lesion segmentation with low-cost, coarse annotations. As illustrated in Figure 1, in this framework, the provided coarse mask (such as an elliptical contour) is utilized as a prompt to guide the model to segment the actual lesion region. Specifically, our method accepts both a retinal image patch and its corresponding coarse prompt as inputs. By leveraging a lightweight, U-Net-inspired feature extraction backbone, the network effectively integrates image content with the spatial prior embedded in the prompt. Through prototype learning, the model initially computes feature prototypes for both foreground (lesion) and background directly from the prompted region. Final lesion segmentation is then dynamically determined at the pixel level by evaluating the learned similarity between each pixel’s feature and the respective foreground/background prototypes. Because our prototypical approach generates image-specific prototypes that adaptively characterize each image, it exhibits enhanced robustness to intra-class variance and large distribution shifts across different datasets or previously unseen classes.

However, the inherent imprecision of coarse annotations presents a significant challenge for prototype extraction. In the case of small or irregularly shaped lesions, coarse prompts often encompass substantial background areas. When the foreground prototype is generated by simply aggregating features (e.g., through average pooling) within the entire prompted region, a considerable amount of background information is incorporated, resulting in “prototype dilution”. This significantly undermines the accuracy and discriminative power of the lesion prototype. Thus, we introduce an innovative superpixel-guided prototype weighting module. Within the coarse prompt region, this module first employs superpixel segmentation to divide the area into multiple subregions based on local feature similarity, computing a representative sub-prototype for each subregion. Subsequently, the distinctiveness of each sub-prototype from the global background prototype is evaluated, and an adaptive weight is assigned—subregions that are more dissimilar to the background (and therefore more likely to correspond to true lesion) receive higher weights. Finally, all sub-prototypes are fused in a weighted manner to form a foreground prototype that is more robust to background interference and more accurately captures the true characteristics of the lesion.

Our main contributions are as follows:

• We propose a novel prototype-guided promptable segmentation framework that effectively refines coarse retinal lesion annotations.
• We introduce a superpixel-based prototype weighting mechanism to counteract prototype dilution and enhance feature discrimination.
• We conduct comprehensive experiments on three fundus lesion datasets (IDRiD [6], DDR [7], and a private dataset), substantially improving upon the initial coarse annotations and achieving state-of-the-art performance compared to both non-prototypical mask refinement baselines and existing general medical promptable segmentation models.

2. Related Work

2.1. Retinal Lesion Segmentation

Retinal image analysis is crucial for ophthalmic disease diagnosis, monitoring, and treatment planning. Precise segmentation of retinal lesions (e.g., microaneurysms, hemorrhages, exudates, drusen) provides objective evidence for early screening and assessment of blinding diseases such as diabetic retinopathy and age-related macular degeneration [8,9]. Early approaches relied on handcrafted feature engineering and classical image processing methods such as vessel topology extraction and texture analysis, combined with thresholding or region growing. For example, Niemeijer et al. [10] leveraged Gaussian-based matched filtering for microaneurysm detection. However, these methods often fail to generalize robustly due to large variations in lesion scale, blurred boundaries, and morphological diversity.

With advances in deep learning, U-Net and its derivatives have become the main solutions for retinal lesion segmentation. The U-Net [1] encoder–decoder structure with skip connections enables multi-scale feature extraction. For complex lesions, attention mechanisms and feature fusion have been widely explored. Wang et al. [11] introduced a dual-branch attention network, while Zhou et al. [12] utilized hierarchical guidance via attention heat maps. Nonetheless, bottom-up global fusion can suffer from locational inaccuracies for tiny lesions. Liu et al. [2] enhanced small lesion detection via local feature modules, but at the cost of slower convergence for large lesions. Ding et al. [4] combined high-res local cropping and global context, achieving accurate segmentation across diverse scales. Despite these advances, reliance on large expert-annotated datasets remains a bottleneck.

2.2. Prototype Learning

Prototype learning aims to extract discriminative prototypes to represent each class, enabling classification or segmentation via similarity matching [13,14]. This paradigm, intuitively aligned with human reasoning and offering good interpretability, is especially attractive for few-shot and cross-domain medical segmentation [15,16,17]. Zhang et al. [15] proposed self-aware cross-sample prototype learning for semi-supervised segmentation, leveraging limited annotations through dynamic prototype construction. Zhou et al. [14] embedded prototypes within hybrid networks for improved breast tumor segmentation, while Liu et al. [17] used spatial–semantic prototype affinity for pseudo-label optimization in weakly supervised learning, boosting the model’s reliability in ambiguous boundary regions. Inspired by these successes, our work innovatively adapts prototype learning to a prompt-driven framework for refining coarse annotations. We specifically introduce a superpixel-guided mechanism to tackle the unique challenge of prototype dilution caused by imprecise clinical prompts, which has not been addressed by prior work.

2.3. Promptable Segmentation

Promptable segmentation leverages interactive prompts (points, boxes, scribbles, etc.) to dynamically guide the segmentation process. The Segment Anything Model (SAM) [18] established a generic, prompt-driven segmentation framework using a massive pretraining dataset, demonstrating strong zero-shot generalization. In medical imaging, prompt-based segmentation has become an active focus.

Dedicated models have emerged to address medical needs: SAM-Med2D [19] adapts the original SAM for multimodal 2D medical images, while MedSAM [20] scales training with 1.5 million medical segmentations for improved performance. ScribblePrompt [21] introduces flexible 2D prompting modes (point/box/scribble) for various medical tasks. For 3D imaging, SAM-Med3D [22] and NVIDIA VISTA [23] extend promptable segmentation to volumetric data. SegVol [24] integrates 3D spatial prompts and textual descriptions for broad multimodal applications. These developments collectively highlight the evolving landscape of prompt-driven medical image segmentation.

3. Methods

This section details our prototype-based promptable segmentation approach for retinal lesion detection, as illustrated in Figure 2. The core objective is to efficiently transform the coarse mask prompt (e.g., an elliptical annotation) into an accurate pixel-wise segmentation.

To achieve this, our method comprises three main steps. First, a coarse mask prompt simulation procedure (Section 3.1) generates training pairs with simulated coarse prompts from finely annotated data. Second, we construct a prototype learning-based refinement network (Section 3.2), which integrates image features and prompt information to learn discriminative foreground and background prototypes for initial segmentation. Finally, to mitigate background interference within coarse prompts, a superpixel-guided prototype weighting module (Section 3.3) is introduced to enhance foreground prototype discrimination and overall segmentation accuracy.

3.1. Coarse Mask Prompt Simulation

To effectively train the promptable segmentation network, a large volume of retinal image and corresponding coarse mask prompt pairs are required. Direct manual annotation at scale is costly; therefore, we simulate realistic coarse prompts from existing pixel-level precise annotations. This simulation mimics the common clinical practice of marking lesions quickly with simple geometric shapes (e.g., ellipses). The main steps are as follows (illustrated in Figure 3):

1.

Preprocessing and Region Dilation: Each binary lesion mask is first dilated using a morphological operation, simulating the looser coverage typical of coarse annotation and merging adjacent small lesions to reduce the number of separate regions.

2.

Connected Component Analysis and Centroid Extraction: We perform connected component analysis on the dilated mask to identify all $N_c$ foreground regions. For each region, the centroid coordinate is calculated as an input for clustering.

3.

Centroid-based Clustering: Since lesions often cluster spatially, we apply KMeans clustering to the centroids to merge nearby components, reflecting how ophthalmologists may annotate spatially adjacent lesions as a single region. The cluster number is dynamically determined as

$$N_{\mathrm{cluster}}=min(\lceil N_c/r\rceil ,N_{\max -\mathrm{cluster}})$$

(1)

where r is a reduction factor and $N_{\max -\mathrm{cluster}}$ is the maximal cluster number.

4.

Ellipse Fitting: For each cluster, we fit a minimum-area enclosing ellipse, then randomly scale its axes by 10% to 50% to further mimic the imprecision of coarse clinical annotation.

5.

Training Pair Construction: Based on each fitted ellipse, a fixed-size image patch is cropped from the original fundus image with a matching binary mask, yielding the final “image patch/coarse mask prompt” pairs for training.

This automated pipeline efficiently converts fine-grained pixel-level datasets into abundant, realistic training pairs with simulated coarse prompts, providing a reliable foundation for prompt-based segmentation research. The generated prompts closely align with the morphological features of rapid human annotation while incurring minimal additional cost.

3.2. Prototype-Based Coarse Annotation Refinement

3.2.1. Feature Fusion

Given paired retinal image patches $I\in {\mathbb{R}}^{H\times W\times 3}$ and the corresponding coarse lesion mask $M\in {\mathbb{R}}^{H\times W\times 1}$, we concatenate them along the channel dimension to obtain an $H\times W\times 4$ input. A lightweight U-Net [1] backbone (following [25]) extracts features, with reduced upsampling layers to lower computation. The output deep feature $F\in {\mathbb{R}}^{H_0\times W_0\times C}$ ($H_0=H/4,W_0=W/4$) is concatenated with a resized mask $M^{\prime }\in {\mathbb{R}}^{H_0\times W_0\times 1}$ to form an intermediate feature of size $H_0\times W_0\times (C+1)$. This is followed by a lightweight fusion module: a $1\times 1$ convolution (reducing channels to $C_0$), batch normalization, and ReLU activation, resulting in a fused feature $F^{\prime }\in {\mathbb{R}}^{H_0\times W_0\times C_0}$. This design jointly encodes both local detail (e.g., fine vessel/lesion structure) and global guidance (prompt region), facilitating downstream prototype extraction.

3.2.2. Prototype Extraction

Prototype learning is adopted to capture discriminative features for foreground (lesion) and background. Firstly, foreground prototype ${\mathit{\mu }}_1$ is computed as the mean of fused features within the mask:

$${\mathit{\mu }}_1={\displaystyle \frac{{\sum }_{(x,y)}{F}^{\prime }(x,y)\mathbb{I}[M^{\prime }(x,y)=1]}{{\sum }_{(x,y)}\mathbb{I}[M^{\prime }(x,y)=1]}}$$

(2)

Background prototype ${\mathit{\mu }}_0$ is similarly calculated over $M^{\prime }(x,y)=0$:

$${\mathit{\mu }}_0={\displaystyle \frac{{\sum }_{(x,y)}{F}^{\prime }(x,y)\mathbb{I}[M^{\prime }(x,y)=0]}{{\sum }_{(x,y)}\mathbb{I}[M^{\prime }(x,y)=0]}}$$

(3)

3.2.3. Prototype-Based Segmentation

For each position $(x,y)$ in $F^{\prime }$, we compute its distance to both prototypes and convert it into a probability map $P_c$ ($c\in \{0,1\}$, background/foreground) via softmax normalization:

$$P_c(x,y)={\displaystyle \frac{exp(-\alpha ·d(F^{\prime }(x,y),{\mathit{\mu }}_c))}{{\sum }_{c^{\prime }\in \{0,1\}}exp(-\alpha ·d(F^{\prime }(x,y),{\mathit{\mu }}_{c^{\prime }}))}}$$

(4)

where $d(·,·)$ is a distance metric (e.g., Euclidean), and $\alpha$ is a scaling factor (fixed to 20) to sharpen the probability distribution.

3.2.4. Training and Inference

The model is trained end-to-end with a composite loss:

$${\mathcal{L}}_{\mathrm{loss}}={\mathcal{L}}_{\mathrm{dice}}+{\mathcal{L}}_{\mathrm{bce}}$$

(5)

where ${\mathcal{L}}_{\mathrm{dice}}$ and ${\mathcal{L}}_{\mathrm{bce}}$ denote the Dice and binary cross-entropy losses between predicted $P_1$ and ground truth $M_{gt}$.

At inference, each input patch is predicted as a foreground probability map. For patches from overlapping regions in the original image, the final probability is taken as the maximum across all overlapping predictions. Applying a fixed threshold yields the final binary lesion mask.

3.3. Superpixel-Guided Prototype Weighting

As shown in Figure 4, when the true lesion area is much smaller than the coarse mask, the foreground prototype computed via Equation (2) is easily contaminated by background pixels. To address this, we introduce a superpixel-guided foreground prototype weighting strategy to alleviate background interference within the prompt region.

Inspired by the maskSLIC algorithm [26], we first partition the prompt region into multiple superpixels based on feature similarity. For each superpixel $S_i$, a sub-prototype vector ${\mathit{g}}_i$ is extracted as in Equation (2). All sub-prototypes form the set $G=\left\{{\mathit{g}}_i\right\}$, where $i=1,2,\dots ,N_{sp}$, and $N_{sp}$ is the number of superpixels. To adaptively determine $N_{sp}$ based on the prompt’s scale, we set

$$N_{sp}=min\left(⌊{\displaystyle \frac{{\sum }_{(x,y)}{M}^{\prime }(x,y)}{S_{sp}}}⌋,N_{\max -\mathrm{sp}}\right)$$

(6)

where ${\sum }_{(x,y)}{M}^{\prime }(x,y)$ is the pixel count in the prompt mask, $S_{sp}$ is the expected area per superpixel (empirically set to 100), and $N_{\max -\mathrm{sp}}$ is capped at 20 to keep computation tractable.

To assess the distinctiveness of each sub-prototype from the background, we compute the cosine distance between each ${\mathit{g}}_i$ and the background prototype ${\mathit{\mu }}_0$:

$$w_i=1-{\displaystyle \frac{{\mathit{g}}_i·{\mathit{\mu }}_0}{\parallel {\mathit{g}}_i\parallel ·\parallel {\mathit{\mu }}_0\parallel }}$$

(7)

where $\parallel ·\parallel$ denotes L2 norm. The weights are then normalized via softmax:

$$w_i^{\prime }={\displaystyle \frac{exp(\beta ·w_i)}{{\sum }_{{\mathit{g}}_{i^{\prime }}\in G}exp(\beta ·w_{i^{\prime }})}}$$

(8)

Here, we follow [27,28] to set the scaling factor $\beta =10$ to sharpen the weighting effect.

The final weighted foreground prototype is computed as

$${\mathit{\mu }}_1^{\prime }=\sum _{{\mathit{g}}_i\in G}{w}_i^{\prime }·{\mathit{g}}_i$$

(9)

In subsequent segmentation computations, ${\mathit{\mu }}_1^{\prime }$ replaces the original ${\mathit{\mu }}_1$ in Equation (4). Experimental results demonstrate that this superpixel-guided weighted prototype more accurately captures discriminative lesion characteristics, especially in cases where the coarse prompt severely overestimates lesion size, thus significantly improving segmentation performance.

4. Experiments and Results

4.1. Datasets and Evaluation Metrics

4.1.1. Datasets

We conducted comprehensive evaluations on three datasets, including two public retinal lesion datasets (IDRiD [6] and DDR [7]) and one private clinical dataset:

• IDRiD: Comprises 81 color fundus images (54 for training, 27 for testing) with a resolution of 4288 × 2848 pixels. Pixel-level annotations cover four lesion types associated with diabetic retinopathy: hard exudates (EXs), hemorrhages (HEs), microaneurysms (MAs), and soft exudates (SEs).
• DDR: Consists of 757 fundus images (383 for training, 149 for validation, 225 for testing), with resolutions ranging from 1088 × 1920 to 3456 × 5184 pixels. Lesion classes are the same as in IDRiD.
• Private Clinical Dataset: Includes 211 color fundus images with pixel-level annotations of preretinal hemorrhage (Prh) and drusen, labeled by two experienced ophthalmologists.

The overall model training is conducted using the training split of the IDRiD dataset, which contains 54 fundus images. Applying the coarse mask prompt simulation algorithm described in Section 3.1, a total of 3320 valid training image–mask pairs are generated from these images. These pairs comprehensively cover all four key diabetic retinopathy lesion types: hard exudates (EXs, 30.2% of samples), hemorrhages (HEs, 23.5%), microaneurysms (MAs, 43.6%), and soft exudates (SEs, 2.7%). To address the scarcity of SE lesions, an oversampling strategy is employed to increase their effective representation in the training set, thereby enhancing the model’s sensitivity and learning capacity for this rare category.

For evaluation, the 27 IDRiD test images, 225 DDR test images, and all images from the private dataset are preprocessed using the same mask prompt simulation pipeline as in training. This standardized preprocessing ensures consistency between training and testing data, enhances comparability across datasets, and enables a fair assessment of the model’s capability in refining coarse fundus lesion annotations.

4.1.2. Evaluation Metrics

Following previous work [2,3,4], the Intersection over Union (IoU) is adopted as the primary evaluation metric for lesion segmentation performance in fundus images.

4.2. Implementation Details

All experiments were conducted using the Adam optimizer with a batch size of 64, an initial learning rate of $1\times {10}^{-4}$, and a weight decay of $1\times {10}^{-4}$. The model was trained for 120 epochs. Learning rate scheduling was handled by the ReduceLROnPlateau strategy to adaptively adjust the learning rate for better convergence and performance.

To enhance generalization, various data augmentation techniques were applied during training, including random translation, scaling, rotation, blur, and random adjustments of brightness and contrast. The maximum number of superpixels $N_{max-sp}$ was fixed at 20. All input image–prompt pairs were resized to $256\times 256$ before being fed into the model.

4.3. Comparison Methods

To comprehensively evaluate the proposed method for coarse lesion annotation refinement, we conducted extensive comparative experiments covering both conventional fully supervised approaches and the latest promptable segmentation techniques.

Firstly, we benchmarked our model against several state-of-the-art fully supervised fundus lesion segmentation methods [2,4], which do not utilize any mask prompt information. These serve as baselines to assess the achievable segmentation performance without auxiliary prompt guidance.

Secondly, we implemented multiple non-prototypical segmentation models as baseline promptable methods. These models take both the image patch and its corresponding mask prompt as input, extract features via a prompt fusion module (identical to Section 3.2 except for the backbone choice), and employ a $1\times 1$ convolution as the segmentation head to generate refined probability maps. For a thorough evaluation, three popular backbone networks were explored: ResNet-18 [29], HRNet-18 [30], and U-Net [1]. This diversity allows us to investigate the impact of backbone architectures on segmentation performance.

Lastly, we compared our approach to 2D medical promptable segmentation models, specifically SAM-Med2D [19] and MedSAM [20]. Since these models do not support elliptical prompt masks, we used the lesion’s bounding box as the input prompt for a fair comparison and evaluation of segmentation performance.

4.4. Results on Each Dataset

4.4.1. Results on IDRiD Dataset

We first train all segmentation models on the IDRiD training set, covering all four major lesion types (MA, SE, EX, HE), and evaluate their coarse prompt refinement performance on the IDRiD test set. The quantitative results are shown in Table 1. Several key observations are summarized as follows:

• Both our prototype-based promptable segmentation method and other prompt-based baselines significantly outperform the initial coarse mask prompt across all lesion categories, demonstrating substantial improvement in segmentation accuracy.
• Without prompt guidance, traditional fully supervised approaches (e.g., HRDecoder) achieve only 32.6% IoU on the highly challenging MA class, much lower than for other lesions. This highlights the difficulty of detecting tiny lesions and the effectiveness of spatial prompts in enhancing detection.
• Our method outperforms non-prototypical baselines for all four lesion types. For example, our method achieves 84.3% IoU for MA, surpassing HRNet18 (79.1%) and U-Net (77.6%). On average, the prototype-based approach yields an improvement of over 5.2% for all lesions.
• General-purpose medical promptable models such as MedSAM and SAM-Med2D perform less effectively on fundus lesion segmentation. For instance, on EX, MedSAM achieves only 56.0% IoU, whereas our approach attains 66.9%. This suggests that such models are not specifically optimized for the unique challenges of fundus lesion segmentation, notably the fine-grained, tiny, and diffusive lesion patterns with low contrast and blurred boundaries.
• The superpixel-guided prototype weighting module delivers significant performance gains for block-like, large-area lesions (e.g., HEs), but has a limited effect for spot-like micro-lesions (e.g., MAs). For small lesions, the prompt mask is typically small and may default to a single prototype, while larger lesions benefit from multiple superpixels and effective weighting, resulting in more accurate prototypes and improved segmentation.

Table 1. Comparisons with other methods on the IDRiD dataset. For overall comparison, the best results are highlighted in bold, the second-best are underlined, and our results are displayed in gray.

Methods	IDRiD (IoU (%) ↑)
	MA	SE	EX	HE	Mean
Coarse Mask Prompt	9.6 ± 2.6	49.3 ± 10.4	15.0 ± 5.2	33.2 ± 8.6	26.8 ± 6.7
M2MRF [2]	31.4 ± 8.3	52.3 ± 12.1	64.6 ± 9.6	48.5 ± 11.2	49.2 ± 10.3
HRDecoder [4]	32.6 ± 7.9	56.1 ± 14.5	65.0 ± 10.1	49.8 ± 10.9	50.9 ± 10.9
ResNet18 [29]	73.9 ± 7.3	68.4 ± 14.2	54.1 ± 9.4	62.6 ± 10.9	64.7 ± 10.5
HRNet18 [30]	79.1 ± 7.5	78.1 ± 5.3	56.8 ± 9.1	64.8 ± 11.8	69.7 ± 8.4
U-Net [1]	77.6 ± 6.8	75.6 ± 12.6	58.9 ± 8.9	67.9 ± 11.4	70.0 ± 9.9
MedSAM [20]	60.7 ± 7.1	59.6 ± 11.3	56.0 ± 13.5	62.2 ± 9.7	59.6 ± 10.4
SAM-Med2D [22]	57.9 ± 7.6	61.1 ± 13.2	51.2 ± 14.7	65.9 ± 11.0	59.0 ± 11.6
Ours w/o superpixel	84.2 ± 6.2	80.7 ± 9.2	65.3 ± 8.1	69.9 ± 13.9	75.0 ± 9.4
Ours	84.3 ± 6.1	81.4 ± 9.6	66.7 ± 8.4	72.0 ± 11.3	76.1 ± 9.4

Table 1. Comparisons with other methods on the IDRiD dataset. For overall comparison, the best results are highlighted in bold, the second-best are underlined, and our results are displayed in gray.

Methods	IDRiD (IoU (%) ↑)
	MA	SE	EX	HE	Mean
Coarse Mask Prompt	9.6 ± 2.6	49.3 ± 10.4	15.0 ± 5.2	33.2 ± 8.6	26.8 ± 6.7
M2MRF [2]	31.4 ± 8.3	52.3 ± 12.1	64.6 ± 9.6	48.5 ± 11.2	49.2 ± 10.3
HRDecoder [4]	32.6 ± 7.9	56.1 ± 14.5	65.0 ± 10.1	49.8 ± 10.9	50.9 ± 10.9
ResNet18 [29]	73.9 ± 7.3	68.4 ± 14.2	54.1 ± 9.4	62.6 ± 10.9	64.7 ± 10.5
HRNet18 [30]	79.1 ± 7.5	78.1 ± 5.3	56.8 ± 9.1	64.8 ± 11.8	69.7 ± 8.4
U-Net [1]	77.6 ± 6.8	75.6 ± 12.6	58.9 ± 8.9	67.9 ± 11.4	70.0 ± 9.9
MedSAM [20]	60.7 ± 7.1	59.6 ± 11.3	56.0 ± 13.5	62.2 ± 9.7	59.6 ± 10.4
SAM-Med2D [22]	57.9 ± 7.6	61.1 ± 13.2	51.2 ± 14.7	65.9 ± 11.0	59.0 ± 11.6
Ours w/o superpixel	84.2 ± 6.2	80.7 ± 9.2	65.3 ± 8.1	69.9 ± 13.9	75.0 ± 9.4
Ours	84.3 ± 6.1	81.4 ± 9.6	66.7 ± 8.4	72.0 ± 11.3	76.1 ± 9.4

4.4.2. Cross-Dataset and Cross-Class Results

To further verify model generalization, we directly evaluate the model trained on the IDRiD dataset on the DDR test set and our private clinical dataset (containing two novel lesion types: Drusen and Prh) without any fine-tuning. Results are shown in

Table 2. Comparisons with other methods on the DDR dataset. For overall comparison, the best results are highlighted in bold, the second-best are underlined, and our results are displayed in gray.

Methods	DDR (IoU (%) ↑)
	MA	SE	EX	HE	Mean
Coarse Mask Prompt	6.9 ± 3.4	32.3 ± 10.8	14.8 ± 8.8	23.1 ± 11.4	19.3 ± 8.6
M2MRF [2]	12.7 ± 6.2	30.8 ± 8.7	40.8 ± 9.7	31.7 ± 10.3	29.0 ± 8.7
HRDecoder [4]	13.1 ± 6.9	29.9 ± 9.3	41.6 ± 10.1	33.8 ± 8.5	29.6 ± 8.7
ResNet18 [29]	56.3 ± 12.4	67.6 ± 13.6	52.3 ± 13.2	54.4 ± 13.3	57.6 ± 13.1
HRNet18 [30]	65.1 ± 13.5	68.2 ± 12.6	54.2 ± 13.9	58.5 ± 12.9	61.5 ± 13.3
U-Net [1]	60.9 ± 13.2	70.2 ± 15.0	55.4 ± 12.8	59.8 ± 12.7	61.6 ± 13.4
MedSAM [20]	51.2 ± 15.1	60.3 ± 19.2	43.9 ± 17.3	50.1 ± 14.8	51.4 ± 16.6
SAM-Med2D [22]	48.3 ± 12.4	62.5 ± 17.6	45.7 ± 15.0	52.3 ± 15.7	52.2 ± 15.2
Ours w/o superpixel	68.8 ± 12.0	71.2 ± 16.9	58.4 ± 12.7	61.3 ± 15.7	64.9 ± 14.3
Ours	69.6 ± 11.8	72.5 ± 17.3	59.9 ± 12.5	63.2 ± 14.5	66.3 ± 14.0

and

Table 3. Comparisons with other methods on the private dataset. For overall comparison, the best results are highlighted in bold, the second-best are underlined, and our results are displayed in gray.

Methods	Private (IoU (%) ↑)
	Drusen	Prh	Mean
Coarse Mask Prompt	33.6 ± 19.1	49.7 ± 13.0	41.6 ± 16.0
ResNet18 [29]	43.9 ± 16.6	52.9 ± 15.2	48.4 ± 15.9
HRNet18 [30]	44.2 ± 21.1	57.5 ± 16.4	50.9 ± 18.8
U-Net [1]	45.2 ± 20.9	57.9 ± 16.4	51.6 ± 18.6
MedSAM [20]	40.1 ± 17.2	50.7 ± 17.8	45.4 ± 17.5
SAM-Med2D [22]	39.2 ± 20.4	53.5 ± 16.2	46.4 ± 18.3
Ours w/o superpixel	47.9 ± 23.4	56.3 ± 20.7	52.1 ± 22.1
Ours	49.6 ± 22.0	62.7 ± 16.7	56.2 ± 19.4

, with key findings summarized below.

On the DDR dataset, cross-dataset evaluation reveals that fully supervised baselines (M2MRF, HRDecoder) experience a sharp drop in mean IoU (to 29.0% and 29.6%, respectively, over 20% lower than on IDRiD), indicating strong sensitivity to data distribution shift. Our method achieves a mean IoU of 66.3%, outperforming HRNet18 (61.5%) and U-Net (61.6%) by 4.8% and 4.7%, with only a 10.2% drop from IDRiD. This demonstrates much greater stability under domain shift, attributable to per-instance prototype modeling of lesion semantics.

On the private dataset, cross-class evaluation shows our method achieves 49.6% IoU for Drusen and 62.7% for Prh, significantly surpassing MedSAM (40.1%, 50.7%) and U-Net (45.2%, 57.9%). The superpixel-guided prototype weighting module yields a notable improvement for the irregular, large-area Prh class, with a 6.4% performance gain over the global prototype baseline, highlighting its effectiveness.

Overall, the results demonstrate that our promptable segmentation network not only achieves excellent performance on in-distribution data, but also maintains robust accuracy when applied to new datasets and previously unseen lesion types, confirming its strong generalization capabilities.

4.4.3. Qualitative Analysis

Figure 5 presents qualitative comparisons that intuitively illustrate the differences and characteristics of various segmentation methods across lesion types. In the first two rows, the non-prototypical U-Net baseline exhibits prominent boundary discontinuities and fragmented mask predictions for Prh. Models using only global prototypes achieve more contiguous results, but tend to over-segment, failing to accurately fit real lesion contours. In contrast, our superpixel-guided prototype weighting method produces more precise boundaries, demonstrating superiority for complex lesion shapes and blurred edges. It is also worth noting that for some large, diffuse lesions like the Prh example (third row), our method, while more contiguous than the U-Net baseline, still exhibits some over-segmentation compared to the ground truth. This may occur because the true boundaries of such lesions are inherently ambiguous, even for human experts.

The last row of Figure 5 shows a challenging case from the private dataset, where the input prompt covers both a familiar lesion (EX) seen in training, and an unseen lesion (Drusen). All compared methods, including ours, predominantly segment the familiar EX region, while often ignoring or misclassifying the previously unseen Drusen area. This highlights the inherent bias of models toward frequent lesion types in the training data and points to areas for future improvement in handling rare or novel lesions.

4.4.4. Sensitivity Analysis of Maximum Number of Superpixels

To evaluate the impact of the maximum number of superpixels, $N_{max-sp}$, on our model’s performance, we conducted a sensitivity analysis on the IDRiD dataset. The results are presented in

Table 4. Sensitivity analysis of the impact of the maximum number of superpixels $N_{max-sp}$. For overall comparison, the best results are highlighted in bold, the second-best are underlined, and our results are displayed in gray.

${\mathit{N}}_{\mathit{max}-\mathit{sp}}$	IDRiD (IoU (%) ↑)
	MA	SE	EX	HE	Mean
1	84.2 ± 6.2	80.7 ± 9.2	65.3 ± 8.1	69.9 ± 13.9	75.0 ± 9.4
10	84.3 ± 6.2	81.0 ± 9.1	65.9 ± 8.5	71.8 ± 12.9	75.8 ± 9.1
20	84.3 ± 6.1	81.4 ± 9.6	66.7 ± 8.4	72.0 ± 11.3	76.1 ± 9.4
30	84.3 ± 6.1	81.5 ± 9.5	66.4 ± 8.9	72.4 ± 12.6	76.2 ± 9.2

. The analysis reveals that the superpixel-guided weighting module consistently enhances segmentation accuracy. When $N_{max-sp}$ is set to 1, the entire prompt region is treated as a single superpixel, which is equivalent to our model without the weighting mechanism. As $N_{max-sp}$ increases from 1 to 20, the mean IoU steadily improves from 75.0% to 76.1%, demonstrating the benefit of partitioning the coarse prompt region to refine the foreground prototype. However, the performance gains begin to saturate at higher values. Increasing $N_{max-sp}$ from 20 to 30 yields only a marginal improvement of 0.1% in mean IoU (from 76.1% to 76.2%). This suggests that subdividing the prompt into more than 20 superpixels provides diminishing returns. Therefore, we chose $N_{max-sp}=20$ for our main experiments, as it offers an excellent trade-off between segmentation performance and computational efficiency. This analysis confirms that our model is robust to the choice of this hyperparameter within a reasonable range.

4.4.5. Generalization to Box Prompt

Despite our focus on elliptical prompts, the model’s architecture is inherently flexible. Technically, our model processes any geometric prompt by first converting it into a binary mask for feature fusion. To explicitly validate this, we conducted an experiment using box prompts. We modified our coarse mask simulation pipeline by replacing the “Ellipse Fitting” step with a “Bounding Box Fitting” step and retrained the model. The results, presented in

Table 5. Segmentation performance with box prompts on different datasets.

Methods	IDRiD (IoU (%) ↑)	DDR (IoU (%) ↑)	Private (IoU (%) ↑)
	Mean	Mean	Mean
Coarse Mask Prompt (Ellipse)	26.8 ± 6.7	19.3 ± 8.6	41.6 ± 16.0
Coarse Mask Prompt (Box)	23.4 ± 7.8	16.7 ± 9.0	36.2 ± 15.8
Ours (Elliptical Prompt)	76.1 ± 9.4	66.3 ± 14.0	56.2 ± 19.4
Ours (Box Prompt)	74.0 ± 10.0	64.5 ± 13.8	53.3 ± 20.8

, demonstrate two key findings. First, the initial IoU of the coarse elliptical prompts is consistently higher than that of box prompts across all datasets. This quantitatively supports our observation that ellipses offer a better initial fit for the contours of lesion clusters compared to bounding boxes. Second, while our model achieves peak performance with elliptical prompts, it remains highly effective with box prompts, showing only a minor performance decrease of approximately 2–3%. For instance, on the IDRiD dataset, the model still improves the IoU from a coarse 23.4% to a refined 74.0%. This confirms that our framework is robust and can generalize effectively to different prompt shapes, even if they are less optimal than ellipses.

4.4.6. Effect of Prompt Coarseness on Segmentation Performance

In clinical scenarios, ophthalmologists often employ coarse elliptic annotations to quickly cover multiple, disconnected lesion regions, introducing varying degrees of annotation coarseness. To systematically study how this affects segmentation performance, we control the level of prompt coarseness in our simulation pipeline by adjusting the K-Means reduction factor r. As r increases, the number of generated elliptical mask prompts per image decreases, and each prompt tends to cover a larger area—potentially including more lesions and greater background, thus increasing prompt coarseness. As shown in Figure 6, larger r values result in fewer, broader prompts and reduced prompt precision.

We evaluate the proposed method (with and without superpixel guidance) and three non-prototypical baselines (U-Net, HRNet18, ResNet18) on IDRiD, DDR, and the private dataset across a range of r values (from 1.0 to 2.0, step 0.1). Results in Figure 7 indicate that, while all methods experience some performance drop as prompt coarseness increases, our prototype learning framework exhibits remarkable robustness. Even under highly coarse prompts ($r=2.0$), our method maintains high IoU scores (over 50% on IDRiD and private, and around 50% on DDR), consistently outperforming the U-Net baseline by 8–11 percentage points. These findings demonstrate the strong adaptability and superior segmentation performance of our framework—particularly with superpixel-guided prototype weighting—under varying levels of prompt quality.

5. Discussion and Conclusions

In this study, we proposed a prototype-guided promptable segmentation framework for refining coarse retinal lesion annotations into accurate pixel-level masks. By leveraging prototype learning and superpixel-guided weighting, our method can adaptively model the complex feature distributions of retinal lesions using only coarse spatial prompts such as ellipses—substantially reducing the annotation burden in clinical workflows. Through systematic experiments on three datasets (IDRiD, DDR, and a private clinical set), we demonstrated that the proposed approach consistently outperforms fully supervised networks and non-prototypical prompt-based baselines, with superior robustness to prompt coarseness, cross-dataset domain shift, and previously unseen lesion categories. Notably, the superpixel-guided prototype weighting module proved especially valuable for segmenting large and irregular lesion regions, further highlighting the flexibility and adaptability of the prototype-based framework.

For practical deployment, our framework could be integrated into existing Picture Archiving and Communication Systems (PACSs) or ophthalmology imaging software as a segmentation-assistance tool. The clinical workflow would involve an ophthalmologist drawing a quick, coarse ellipse around a region of interest, which our model would then process in near real-time to generate a refined, pixel-level mask for quantitative analysis or disease monitoring. However, the integration of such AI tools necessitates a discussion of potential risks. It is essential to mitigate the risk of automation bias, where clinicians might over-rely on the model’s output. Therefore, the proposed tool should be implemented as a decision-support system that assists, rather than replaces, expert judgment, and it must include intuitive interfaces for clinicians to easily review and edit the AI-generated segmentations.

While this study demonstrates strong technical performance, we acknowledge its limitations, which present avenues for future research. First, our claim of reducing annotation costs is based on established literature rather than a direct time-saving analysis conducted in a clinical setting. Future work should include a formal usability study to quantify the precise efficiency gains our method provides to ophthalmologists. Second, although the model performed well on rare classes like soft exudates (SEs), performance on underrepresented classes can always be improved. Future iterations of this work will explore advanced strategies beyond oversampling, such as implementing class-weighted loss functions or using generative models (GANs) to synthesize realistic training data for rare lesions, thereby further enhancing model robustness.

In summary, this work underscores the potential of prototype-based promptable segmentation to bridge the gap between annotation efficiency and clinical accuracy. Our findings highlight a promising direction for developing scalable, adaptable, and reliable medical image analysis tools that can be effectively integrated into modern clinical workflows.