PRefiner: Enhancing Overlapped Cervical Cell Segmentation Through Progressive Refinement

Abstract

Cervical cancer is one of the most prevalent and easily contracted diseases among women, significantly impacting their daily lives. Computer vision-based cervical cell morphology diagnosis technology can offer robust support for cervical cell analysis at a lower cost. However, the presence of a substantial number of overlapping cells in cervical images renders existing cell segmentation methods less accurate, thereby complicating the guidance of medical diagnosis. In this paper, we introduce a tristage Progressive Refinement method (PRefiner) for overlapping cell segmentation that decouples the traditional end-to-end pipeline, with the final stage specifically correcting anomalous results to enhance precision. We achieve separable overlapping cervical cell segmentation results through a cell nucleus locator, a single-cell segmenter, and a Segmentation Result Mask Refiner. Specifically, we employ a hybrid U-Net as the primary network for the cell nucleus locator and single-cell segmenter, which determines the position of the cell nucleus and procures the initial coarse segmentation result. In the mask refiner, we incorporate a conditional generation framework to address the perception decision problem and design a local–global dual-scale discriminator to ensure that the segmentation result aligns with the prior of a single-cell mask. Experimental results on CCEDD and ISBI2015 demonstrate that PRefiner achieves optimal performance by effectively resolving abnormal segmentations. Notably, our method improves the Dice coefficient of abnormal results from five different models by an average of 2.62% (ranging from 1.0% to 5.1%).

1. Introduction

Cervical cancer ranks as one of the most prevalent diseases among women, with an estimated 560,000 women diagnosed annually worldwide. The impact is particularly severe in developing countries such as China and India, where both the number of affected patients and the mortality rate are significantly higher. Computer vision-based cervical cell morphology diagnostic technology offers effective support for automated cervical cell analysis at a comparatively low cost [1]. The accurate segmentation of cervical cells, serving as a prerequisite for its automatic morphological analysis, is of paramount importance and has garnered considerable attention from researchers in recent years. Methods for cervical cell image segmentation can be broadly divided into two categories: traditional methods based on machine learning and those based on deep learning. The former primarily encompasses techniques such as threshold segmentation, edge detection [2], watershed [3] algorithms, etc. Despite the relatively lower requirement for annotated data, traditional methods exhibit subpar segmentation performance in the context of complex backgrounds. The segmentation of different samples necessitates the tuning of multiple hyperparameters by experts, thereby posing challenges to both accuracy and automation. On the other hand, deep learning segmentation algorithms, being data-driven, demonstrate superior segmentation precision and enhanced cross-scenario generalization [4,5,6,7,8,9]. However, in the realm of cervical cell image segmentation, the segmentation of overlapping cervical cells remains a significant challenge, despite the success of numerous methods in enhancing cell edge segmentation accuracy, suppressing background noise, and augmenting annotated data. This challenge persists even with the advent of medical foundation models such as MedSAM [10] and UniverSeg [11]. As shown in Figure 1, both (a) SAM and (b) MedSAM struggle with overlapping cell segmentation, producing noticeable errors.

Figure 1. Overlapping cervical cell segmentation results from SAM and MedSAM (lines of different colors to represent the segmented outlines of different cells).

Overlapping cervical cells pose a significant challenge to instance segmentation due to their overlapping cellular regions and generally blurred cell boundaries, resulting in a visual form where one cell appears embedded within another. The common failure cases in segmentation are primarily characterized as follows:

1.

Loss of discernible edge features: Low contrast and blurred boundaries lead to edge localization errors (Figure 2a).

Figure 2. Failures and their manifestations in the segmentation of overlapping cervical cell images (lines of colors to represent the segmented outlines of different cells).

2.

Erroneous aggregation: The network mistakenly merges overlapping cells into one and inaccurately localizes the edges (Figure 2b).

3.

Combination of multiple factors: Multiple issues jointly cause segmentation failure (Figure 2c).

This challenge is not unique to cervical cell segmentation; Zhao et al. [12] observed that class-agnostic models in partially supervised instance segmentation also struggle with overlapped objects due to the ambiguity of distinguishing foreground from distractors, further highlighting the inherent difficulty of handling overlapping instances. Overlapping cervical cells compromise segmentation accuracy. Inspired by image inpainting techniques [13], we present PRefiner, a progressive refinement framework that decomposes this task into nucleus localization, initial single-cell segmentation, and result refinement. This strategy counters overlap-induced errors, producing high-quality instance segmentation. The main contributions of the paper can be encapsulated as follows:

1.

Our proposed three-stage progressive refinement method decouples traditional end-to-end segmentation approaches. By incorporating a refinement stage after single-cell instance segmentation, it identifies and rectifies aberrant outcomes, thereby improving overall accuracy. Notably, this model-agnostic refiner can also optimize segmentation results produced by other cell segmentation methods.

2.

Within the refiner, we incorporate a conditional generation framework to address the perceptual decision-making problem and design a dual-scale discriminator to enforce consistency with the single-cell mask prior at multiple levels. This refinement process is underpinned by a hybrid U-Net, which integrates Transformer and CNN architectures and serves as the core component of both the nucleus localizer and the single-cell segmenter, ensuring precise nucleus localization and accurate initial cell segmentation.

3.

Extensive experiments on CCEDD [14] and ISBI2015 [15] validate that our method resolves overlapping cell segmentation by accurately rectifying abnormal results, achieving optimal performance across multiple metrics. Crucially, it improves the Dice coefficient for five distinct models by an average of 2.62%, confirming its model-agnostic nature and broad applicability.

2. Related Work

2.1. Medical Image Segmentation

Medical image segmentation has emerged as a primary technology assisting doctors in diagnosing, which can be divided into traditional methods and deep learning based methods. Traditional methods mainly include threshold segmentation, edge detection, watershed algorithm, wavelet transform, etc. However, these methods struggle to address the issues caused by imaging sensor noise and individual patient variations.

With the development of deep learning, medical image segmentation based on deep learning methods has been widely applied in medical image processing. Ronneberger et al. [16] designed the U-Net network based on the Fully Convolutional Network (FCN), which integrates low-resolution and high-resolution semantic features, thereby improving the quality of image segmentation. Subsequently, many researchers improved the U-Net for better performance. For instance, Ibtehaz et al. [17] proposed MultiResUNet to enhance U-Net’s multi-scale capability and mitigate the semantic differences between encoder and decoder features. Gu et al. [18] proposed a context encoding network, CE-Net, which retains 3D spatial information while capturing high-level information. However, U-Net and its improved models [19,20] still cannot establish large- scale contextual information for the whole image. Cao et al. [21] introduced the concept of shift window into U-Net and proposed Swin-Unet, using Swin Transformer as the network backbone to learn the global features. Recently, Li et al. [22] further advanced this direction by proposing APU-Net, which integrates adaptive weight fusion and pyramid dual cross-attention to enhance multi-scale feature representation and boundary delineation for medical image segmentation.

In this paper, we design a hybrid architecture with the attention mechanism and convolutional blocks. By introducing the attention mechanism, we can enrich the original U-Net’s contextual understanding capability for cervical cell segmentation.

2.2. Generative Adversarial Networks

The Generative Adversarial Network (GAN) [23] consists of a generator and a discriminator. Through adversarial training, the generator learns the distribution of real data, making it difficult for the discriminator to accurately determine the source of the input data. The Conditional GAN (CGAN) [24] is an extension of GAN that uses image, text, and other constraint information as additional input conditions for controlled generation. It has a wide range of applications in image editing, image super-resolution, style transfer, and image restoration.

Karras et al. [25] proposed Style GAN v1, a generative adversarial network capable of style editing. Ledig et al. [26] introduced the Super-Resolution GAN (SRGAN), which added a deep residual network with skip connections to the traditional generative adversarial structure to achieve image super-resolution. Isola et al. [24] proposed the Pix2PixGAN based on the idea of CGAN. This algorithm replaces random noise with style images as the conditional input to the generator, allowing the network to capture more complete structural feature information and achieve clear image style transfer. Zhu et al. [27] proposed the Cycle GAN (Cycle consistency GAN), which introduces two pairs of generators and discriminators to solve the problem that style transfer requires paired images for training.

In this paper, we introduce the concept of image restoration into the correction of erroneously segmented cells. We construct a segmentation result refiner using the adversarial learning mechanism of the conditional generative adversarial network, and use generation architecture to solve the perception and decision problems of cervical cell segmentation. In the refiner, we design a dual-scale discriminator to improve the global consistency of the refined part and obtain smoother cell contours for better segmentation results.

3. Materials and Methods

3.1. Overview

This paper presents a novel method for precise segmentation of overlapping cervical cells through a progressive refinement approach. As illustrated in Figure 3, the framework consists of three main components: a nucleus locator, a single-cell segmenter, and a segmentation result refiner.

Figure 3. The overall network structure of PRefiner (lines of colors to represent the segmented outlines of different cells).

Given an input image $I\in {\mathbb{R}}^{H\times W\times 3}$, the nucleus locator first identifies the nuclei (center points) of overlapping cells. Based on dataset statistics, regions of interest around these points are extracted and fed into the single-cell segmenter to obtain initial coarse segmentations for individual cells. Although cell sizes vary within the same dataset, they are on the same order of magnitude, allowing a fixed ROI to achieve reasonably accurate segmentation. However, overlapping cells may be inaccurately segmented at this stage—as shown in Figure 3, one of the two cells exhibits a distinct defect. The coarse results, together with the original image, are then input into the Segmentation Result Refiner, which effectively rectifies such defects, yielding precise instance-level segmentations.

3.2. Preliminary

Both the nucleus locator and the single-cell segmenter rely on accurate segmentation. To address this, we develop an Attention–Convolution Hybrid U-Net that enriches the original U-Net’s contextual understanding by incorporating an attention mechanism and convolution, which we refer to as Attention Convolution Hybrid U-Net. Specifically, the original U-Net consists of an encoder, a decoder, and skip connections. The encoder uses two consecutive $3\times 3$ convolutional layers and a max pooling layer for progressive feature extraction. The decoder reconstructs the segmentation map layer by layer through the deconvolution with an upsampling rate of 2, and skip connections link blocks at the same level in the encoder and decoder.

The attention–convolution hybrid U-Net introduces several improvements. Firstly, the max pooling downsampling is replaced with a convolutional layer with a stride of 2 and a kernel size of 3 to reduce information loss. A single-layer SwinTransformer Block is inserted after each encoder–decoder block to establish global dependencies of features and enrich the contextual information of the features. The upsampling operator in the decoder is replaced with PixelShuffle to minimize the checkerboard effect. The overall network structure is illustrated in Figure 4.

Figure 4. The overall network structure of attention–convolution hybrid U-Net.

3.3. Nucleus Locator

Our first step involves extracting the nuclei from the cell slices, as these can provide precise location guidance for each cell’s segmentation. We model each nucleus as a “point” neighborhood response, making it feasible to extract the nuclei using the hybrid U-Net. The extracted nuclei are then processed into regions of interest: based on the statistical characteristics of the dataset, we design regions of interest centered on the nucleus and with the maximum width and height of the cells as determined statistically. This is a very lightweight and convenient method for generating regions of interest. Each ROI contains a nucleus at its center, and the cell corresponding to this nucleus is defined as the central cell of that ROI. To achieve instance-level results, we only need to fully segment these central cells, which greatly simplifies the complexity of overlapping cell segmentation. The segmentation results in Figure 5b and the visualization on the original image in Figure 5c demonstrate that our nucleus locator accurately segments nuclei at cell centers while ignoring background distractions. Specifically, all 31 nuclei in this image are correctly localized with no false positives or false negatives, and a region of interest (ROI) is generated for each true cell, as shown in Figure 5d. Sampling statistics further show that 97.2% of nuclei are accurately detected and localized, guiding correct ROI generation. Missed detections (2.83%) and false positives (0.35%) rarely occur, and only in cases of highly overlapping cells or severely deformed nuclei. Although the impact on subsequent algorithm performance is currently not significant, further improvement in nucleus segmentation accuracy is needed.

Figure 5. The results from the nucleus locator and the generated ROIs. (a) Cervical cell image. (b) Cell nucleus. (c) Visualization of results (red dots) on the cell image. (d) Regions of interest (red boxes).

3.4. Single-Cell Segmenter

Once the regions of interest are obtained, we can use the hybrid U-Net trained on individual cells for single-cell segmentation. The detailed segmentation process is illustrated in Figure 6. As shown in Figure 6c, due to cell overlap, the composition of each extracted ROI is complex. Specifically, an ROI may contain not only the target central cell and the image background, but also adjacent non-central cells, including both their fragments and the regions where they overlap with the central cell.

Figure 6. The process for the single-cell segmentation. (a) Nucleus localization. (b) ROI delineation. (c) ROI delineation. (d) Segmentation result.

To ensure accurate segmentation under such complex conditions, we define the segment task as a binary problem where the central cell is the sole target. In single-cell segmentation, we approach central cell segmentation as a binary problem. Pixels belonging to the central cell are treated as foreground—regardless of their overlap status—while all other pixels are assigned to the background. This design forces the model to focus solely on the central cell, minimizing interference from overlapping neighbors. As a result, each ROI produces a segmentation mask for a single cell, effectively preventing the merging of multiple cells caused by overlap. However, edge blurring and unclear boundaries caused by overlapping areas may still lead to inaccurate segmentation edges.

3.5. Mask Refiner

In this section, we introduce how to utilize data priors to reduce the issue of imprecise segmentation edges. We employ a generative adversarial framework to fully incorporate data distribution priors. The main structure of the generator is the hybrid U-Net, which outputs refined results G(x) which is close to the ground truth to cheat the discriminator. The discriminator is used to judge whether the generated refined results satisfy the data priors. The generator and discriminator are then trained alternately, achieving balance through adversarial learning. The network structure is illustrated in Figure 7.

Figure 7. The architecture of the mask refiner (The red blocks pointed to by the arrows indicate that there are abnormal segmentations in the corresponding areas of the original image).

Generator: The generator in this paper reuses the proposed hybrid U-Net. To address the specific generation problem—learning the mapping from cell images with abnormal segmentation to those with correct segmentation through adversarial training with the discriminator—the following modifications have been made: Dropout layers are added to the first two encoder blocks of the hybrid U-Net to introduce noise, thereby enhancing the generator’s robustness.

Discriminator: The discriminator plays a critical role in generative adversarial networks. To enhance both the overall integrity and local precision of segmentation results, we propose a dual-level local–global discriminator that incorporates cell shape priors. Its architecture consists of five 3 × 3 convolutional blocks (Conv-BN-ReLU) followed by a Transformer encoding layer to capture global dependencies. The encoder takes the coarse segmentation results and their corresponding generated results as input, and produces a 30 × 30 probability map where each entry indicates the authenticity of a local region. Regions exhibiting abnormal segmentation tend to have lower confidence in this map. Unlike traditional methods such as CGAN that rely on a single scalar output, our patch-based evaluation attends to local details while preserving global consistency. Moreover, the dual-scale discriminator outputs feature matrices with enhanced sensitivity to high-frequency details across different image regions. The network architecture is illustrated in Figure 8.

Figure 8. The detailed architecture of the discriminator (Arrows in different colors represent features at two scales).

Training Process: The generator takes the coarse segmentation results x as input, and aims to generate a refined result G(x) close to the label to deceive the discriminator. The discriminator needs to accurately distinguish between the uncorrected refined result G(x) and the labeled image (ground truth y). During training, the discriminator concatenates x with the refined results G(x) as a fake date, and concatenates x with the ground truth y as the real date. The loss function of CGAN is:

$$L_{\mathrm{CGAN}}(G,D)={\mathbb{E}}_{x,y}[logD(x,y)]+{\mathbb{E}}_x[log(1-D(x,G\left(x\right)))]$$

(1)

In the formula, the function D(x,y) is the probability of manual labelling for input image, and the function D(x, G(x)) is the probability of the input image being the correction result of the generator for x. In the training of the generator, the loss function enables the correction result of the generator to get close to label y as much as possible, namely maximizing D(x, G(x)); when training the discriminator, the loss function maximizes the probability of correct judgment, namely maximizing D(x,y), and meanwhile it also minimizes the probability of the correction result G(x) being distinguished, namely minimizing D(x, G(x)). To make the corrected parts clear in the segmentation result, a loss function L1 is included in the network loss function to find the low-frequency information of the image. The loss function L1 is shown as follows:

$$L_{L_1}\left(G\right)=E_{x,y}[\parallel y-G\left(x\right)\parallel ]$$

(2)

The complete loss function of optimization network is shown in Equation (3), in which λ is the hyper-parameter to balance the loss of CGAN and L1. The parameter $\lambda$ is empirically set to 80 in this paper.

$$G^{\ast }=arg\underset{G}{min}\underset{D}{max}{L}_{\mathrm{CGAN}}(G,D)+\lambda L_{L_1}\left(G\right)$$

(3)

After adversarial training, our generator acquires the capability to rectify abnormal cell segmentation results. When the input segmentation result is abnormal, the network corrects the anomalous parts (by supplementing or removing). Conversely, when the input segmentation result is normal, the network preserves the image appearance, ensuring consistency between the input and output.

4. Experiments and Analyses

4.1. Experimental Dataset and Settings

Experiments were conducted on the ISBI2015 and CCEDD datasets in this study. The ISBI2015 dataset, publicly curated by the IEEE International Symposium on Biomedical Imaging in 2015, serves as a valuable resource for cervical cell segmentation. CCEDD was proposed by the Liaoning Cancer Hospital and the Institute of Automation, Chinese Academy of Sciences. The image size is 2048 × 1536, and each image contains dozens or even hundreds of cells, as shown in Figure 9. As the original dataset only provides annotations for the boundaries of the cytoplasm and nucleus, we re-annotated the individual cells within the nuclei and regions of interest based on these labels to validate our proposed method. The annotation of the nuclei is shown in Figure 10.

Figure 9. Samples from the CCEDD dataset.

Figure 10. The annotations of the nuclei.

Based on the nuclei, we cut out 4500 regions of interest (ROIs) from images, each ROI containing only one complete cell instance centered on the nucleus, as shown in Figure 11a. The segmentation masks of the central cells are obtained based on the ROI annotations. It is worth noting that apart from the central cell, the background and other cells are marked as background. The annotation of a pair of overlapping cells is shown in Figure 11b.

Figure 11. Overlapping cervical cell segmentation results from SAM and MedSAM. (a) Region of interest. (b) One complete cell instance centered on the nucleus.

A total of 96 coarse result images of abnormal segmentation are screened out from 1000 ROIs. The result of abnormal segmentation and related GT makes paired data. After data enhancement by flipping horizontally and vertically, 288 image pairs are fed into the segmentation result optimization network for training. Twenty percent of training sets are treated as verification sets at random by the program and an update is made every 10 epochs. The set training parameters of network are as follows: learning rate 0.0002, epoch 200 and batch size 8.

4.2. Evaluation Protocol and Metrics

Both pixel-level metrics (e.g., Dice, TPRp, FPRp) and object-level metrics (e.g., FNRo, detection accuracy) are employed to comprehensively evaluate segmentation performance. The Dice coefficient stands out as the most commonly employed metric in medical image segmentation. It quantifies the similarity between two samples, typically the predicted segmentation mask (A) and the ground truth (B), as expressed in Equation (4):

$$\mathit{Dice}={\displaystyle \frac{2|A\cap B|}{\left|A\right|+\left|B\right|}}$$

(4)

TPRp and FPRp are shown in Equations (5) and (6) as follows:

$$\begin{array}{cc}\hfill \mathit{TPRp}& ={\displaystyle \frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}}\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill \mathit{FPRp}& ={\displaystyle \frac{\mathit{FP}}{\mathit{TN}+\mathit{FP}}}\hfill \end{array}$$

(6)

where TP, FP, TN and FN denote number of true positive, false positive, true negative, and false negative pixels, respectively.

A cell segmentation result is considered valid if its Dice coefficient exceeds a threshold DC. The FNRo (object-level false negative rate) represents the proportion of cells whose Dice coefficient falls below DC, as defined in Equation (7), where FNo denotes the number of cells below the threshold and TPo denotes the number of validly segmented cells. In our experiments, the segmentation threshold DC is set to 0.7 by default.

$$\begin{array}{cc}\hfill \mathit{FNRo}& ={\displaystyle \frac{\mathit{FNo}}{\mathit{TPo}+\mathit{FNo}}}\hfill \end{array}$$

(7)

Specifically, the nucleus segmentation accuracy Dice is calculated based on the semantic segmentation results of nuclei after Stage 1, while the cell segmentation accuracy is computed based on the instance segmentation results within each ROI after Stage 3.

4.3. Implementation Details

In our study, we conducted experiments utilizing an NVIDIA GeForce GTX 2060 (12 GB) graphics card (NVIDIA Corporation, Santa Clara, CA, USA). The codebase was developed using PyTorch 2.0.1 (CUDA 12.1). Below, we provide detailed training specifications for each component of our neural network architecture.

Nucleus Extraction Network: We split the dataset into an 8:2 ratio, with 192 images allocated for training and 48 images for testing. We employed the AdamW optimizer with a learning rate of 0.001, a batch size of 3, and 200 epochs.

Coarse Segmentation Network: Similar to the nucleus extraction network, we maintained an 8:2 split. Specifically, 2800 region of interest (ROI) images were used for training, while 700 ROI images served as the test set. We continued to use the AdamW optimizer with a learning rate of 0.001, 100 epochs, and a batch size of 16.

Refinement Network: To enhance the ROI images and coarse segmentation results, we applied data augmentation techniques such as horizontal flipping and vertical flipping. We randomly selected 20% of the training samples as the validation set. The refiner was optimized using the AdamW optimizer with a learning rate of 0.0002, 200 epochs, and a batch size of 8.

4.4. Ablation Study

The improved hybrid U-Net network obtained more feature information, increasing the Dice coefficient for nuclei segmentation on the CCEDD dataset from 0.715 with the original network to 0.736, and for cell segmentation from 0.889 to 0.928. Although this accuracy has not reached an extremely high level at this stage, it is sufficient for providing reliable coarse segmentation results to the subsequent refinement network.

The primary innovation of this study resides in the development of a refinement network for single cells based on generative adversarial networks (GANs). Figure 12 shows the refined masks of single cells at different epochs. As can be seen from the figure, as the number of training iterations increases, the refinement ability of the generator improves, and the contours of the cells gradually become smoother, until the abnormal lumps disappear.

Figure 12. Correction of abnormal segmentation.

To verify how effective the dual-scale discriminator is for the improvement of the segmentation result refine network, the correction performance of the PatchGAN discriminator [24] is compared with that of the dual-scale discriminator. Not all the segmentation results are abnormal and it is very critical for the refinement of segmentation results to precisely correct abnormal parts. To verify if abnormal results were effectively corrected by the network and if there was a false correction causing a decline in overall precision, the initial segmentation of all cell samples obtained in the stage 2 and the segmentation results after optimization of abnormal segmentation images were evaluated in terms of performance in ablation experiment according to the 4 evaluation indexes, like Dice. The result of the experiment is shown in Table 1. It can be seen from the experiment that the refiner network for the segmentation result trained by the double-scale discriminator is significantly better than the Dice coefficient and FPR index of the original discriminator, and there are segmentation results of abnormality. The method proposed in this paper significantly improves precision. The comparison of correction performance is shown in Figure 13. It can be seen that the cell outline transition looks more natural and cellular morphology is better in the correction results of the dual-scale discriminator.

Table 1. Comparative experiment for PatchGAN and abnormal segmentation correction of dual-scale discriminator.

		Dice	TPRp	FPRp	FNRo
All cell segmentation correction	Before refiner	0.928	0.927	0.013	0.019
	After refiner (PatchGAN)	0.929	0.927	0.011	0.019
	After refiner (dual-scale)	0.931	0.929	0.009	0.019
abnormal segmentation correction	Before refiner	0.835	0.859	0.029	0.057
	After refiner (PatchGAN)	0.850	0.862	0.020	0.057
	After refiner (dual-scale)	0.863	0.864	0.017	0.057

Figure 13. Comparison of correction performance. (a) Abnormal segmentation. (b) PatchGAN. (c) Dual-scale discriminator.

We conducted experiments on several coarse segmentation networks, including U-Net++, U2-Net, Deeplab v3, and the MedSAM using the CCEDD dataset. Table 2 presents the performance of these networks before and after applying the refiner. Notably, the refiner significantly enhances abnormal segmentation outcomes across various coarse segmentation networks, leading to an overall improvement in segmentation accuracy. On average, the Dice coefficient increases by 2.6% across the five models.

Table 2. Comparison of segmentation accuracy before and after refinement.

	Improved U-Net		U-Net++		U2-Net		Deeplab v3		MedSAM
	Before refiner	After refiner	Before refiner	After refiner	Before refiner	After refiner	Before refiner	After refiner	Before refiner	After refiner
Dice	0.835	0.863	0.818	0.869	0.854	0.875	0.846	0.856	0.901	0.922
TPRp	0.859	0.862	0.916	0.934	0.933	0.944	0.797	0.812	0.828	0.835
FPRp	0.029	0.017	0.038	0.025	0.044	0.039	0.015	0.014	0.021	0.017
FNRo	0.057	0.057	0.119	0	0.1	0	0	0	0.033	0.033

4.5. Comparison with the State of the Art

The ISBI dataset refers to a series of benchmark datasets released in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) challenges. These datasets typically contain microscopy images of cells with ground-truth annotations for tasks such as segmentation and tracking. In the context of cervical cells, the most representative are the ISBI 2014 and ISBI 2015 challenge datasets. Specifically, the ISBI 2014 dataset includes 16 Extended Depth of Field (EDF) real images and 945 synthetic images, while the ISBI 2015 dataset contains 17 multi-layer cervical cell volume images (approximately 680 2D slices in total). We selected the ISBI 2015 dataset for our comparative experiments.

We compared the proposed PRefiner with other segmentation methods, including EMSEG [28], U2 -Net [29], DCNN [30], Mask RCNN [31], and Cell GAN [32] on the ISBI2015 dataset. Since the image sizes in this dataset differ from the model’s training input size, we resized the ROI images of each cell. The comparison results are shown in Table 3. As can be seen from the table, PRefiner has achieved the best overall performance, with a Dice coefficient of 0.917, and TPR of 0.955. The observations demonstrate the unique advantage of PRefiner in the segmentation of overlapping cervical cells.

Table 3. Comparative Results on the ISBI2015 Dataset.

Method	Dice	TPRp	FPRp	FNRo
EMSEG	0.810	0.723	0.006	0.434
U2-Net	0.893	0.937	0.014	0.130
DCNN	0.920	0.910	0.004	0.280
Mask RCNN	0.892	0.925	0.001	0.360
Cell GAN	0.899	0.954	0.031	0.064
Ours	0.917	0.955	0.014	0.092

Figure 14 presents a visual comparison of PRefiner with other methods on images from the ISBI2015 dataset, showing not only the overall segmentation results but also detailed comparisons of segmentation on selected cell clusters. It can be observed that for cells with large overlapping areas, the Cell GAN method tends to over-segment. The EMSEG method shows deficiencies in preserving cell shapes. While Mask RCNN and U2-net methods can segment contours relatively accurately, they may miss some predictions when dealing with complex cell clusters. For the segmentation of complex overlapping cells under low-contrast conditions, the proposed PRefiner achieves more complete segmentation results that are closest to the ground truth.

Figure 14. Segmentation results of our method and other methods on images from the ISBI2015 dataset. (a) EMSEG; (b) U2-Net; (c) Mask RCNN; (d) Cell GAN; (e) Ours result; (f) GT.

As a refinement module, PRefiner plays a crucial role in our proposed method. Figure 15 shows the coarse segmentation results of some overlapping cells and the refined segmentation results after using the refiner. The correction of abnormal results is the main reason for the improved accuracy of our method.

Figure 15. Examples of abnormal segmentation correction for images in ISBI2015 dataset (Figures (a–d) show the visualization results of four sets of abnormal segmentation corrections).

The comparison results of segmentation accuracy are shown in Table 3. As can be seen from the table, the proposed PRefiner achieves a Dice coefficient of 0.917 and a TPR of 0.955. The hybrid U-net architecture is adopted as a balanced trade-off between computational complexity and segmentation accuracy in our method. Although its individual cell segmentation results are not optimal, after refinement using the proposed method, the overall segmentation results achieve the best performance. These observations demonstrate the unique advantage of PRefiner in the segmentation of overlapping cervical cells.

Significantly, the proposed refinement module is architecture-agnostic and can be seamlessly integrated as a plug-in to enhance the segmentation outputs of existing methods, consistently boosting their accuracy.

5. Conclusions

In order to refine the abnormal segmentation caused by overlapping of cervical cells, this paper has introduced the concept of image inpainting to the correction of abnormal segmentation. The adversarial learning mechanism from a conditional generative adversarial network was employed as a refiner to optimize the segmentation result. During adversarial learning, a dual-scale discriminator was designed to compete against the generator. This aims to ensure the overall consistency of the generated images, preserve more details in corrected regions, and produce more natural outlines. The result of comparative experiments of different models based on CCEDD datasets has verified the feasibility and accuracy of the proposed method. On the ISBI 2015 dataset, the proposed method demonstrates superior performance for severely overlapped cervical cells. It effectively refines initial segmentations to yield clear cell contours and notably enhances segmentation accuracy.