Generative Adversarial Networks in Histological Image Segmentation: A Systematic Literature Review

Abstract

Histological image analysis plays a crucial role in understanding and diagnosing various diseases, but manually segmenting these images is often complex, time-consuming, and heavily reliant on expert knowledge. Generative adversarial networks (GANs) have emerged as promising tools to assist in this task, enhancing the accuracy and efficiency of segmentation in histological images. This systematic literature review aims to explore how GANs have been utilized for segmentation in this field, highlighting the latest trends, key challenges, and opportunities for future research. The review was conducted across multiple digital libraries, including IEEE, Springer, Scopus, MDPI, and PubMed, with combinations of the keywords “generative adversarial network” or “GAN”, “segmentation” or “image segmentation” or “semantic segmentation”, and “histology” or “histological” or “histopathology” or “histopathological”. We reviewed 41 GAN-based histological image segmentation articles published between December 2014 and February 2025. We summarized and analyzed these papers based on the segmentation regions, datasets, GAN tasks, segmentation tasks, and commonly used metrics. Additionally, we discussed advantages, challenges, and future research directions. The analyzed studies demonstrated the versatility of GANs in handling challenges like stain variability, multi-task segmentation, and data scarcity—all crucial challenges in the analysis of histopathological images. Nevertheless, the field still faces important challenges, such as the need for standardized datasets, robust evaluation metrics, and better generalization across diverse tissues and conditions.

1. Introduction

The analysis of histological images has become increasingly prominent as a critical component in supporting clinical decision-making [1]. This growing relevance is largely driven by the integration of machine learning algorithms, which have shown great potential in automating complex visual tasks; for example, its identification enables the precise identification of cellular and tissue structures, such as nuclei, cytoplasm, glands, or tumor regions [2].

During histopathological evaluation, hematoxylin and eosin (H&E) staining is widely used to highlight cellular and tissue structures. Hematoxylin stains cell nuclei blue or purple, while eosin colors the cytoplasm and extracellular matrix in shades of pink [3]. Although effective, the staining process is highly sensitive to variations in protocols, reagent concentrations, and incubation times, which can result in considerable differences in color and contrast between samples. In addition, image acquisition may be affected by differences in slide scanners, lighting conditions, and resolution settings [4]. These sources of variability can lead to inconsistencies in visual appearance, making automated analysis, especially tasks like segmentation, more challenging and less generalizable across datasets.

In response to the challenges posed by variability in staining and image acquisition, generative neural networks, such as generative adversarial networks (GANs), have emerged as promising tools in the field. These models are capable of learning complex data distributions and producing realistic synthetic representations, making them particularly valuable for enhancing model generalization across heterogeneous datasets [5]. Building upon the growing interest in the use of GANs for histological image segmentation, recent research has explored their integration not only as primary segmentation models but also in auxiliary roles to address specific limitations in medical imaging workflows. Generative networks have been employed for tasks such as data augmentation [6], domain adaptation [7], and label refinement [8]. For example, Hou et al. [6] leveraged GANs to synthesize realistic histopathological patches, enabling improved segmentation performance in datasets with limited annotations. In the context of domain adaptation, Vo and Khan [7] proposed an edge-preserving GAN to bridge domain gaps caused by differences in staining or scanning protocols, thus ensuring consistent segmentation across diverse datasets. Furthermore, Qu et al. [8] demonstrated how GANs can refine weak annotations by learning the underlying structural distribution of histological components. Their ability to simulate consistent, high-quality training data and adapt models to new domains directly addresses key limitations associated with manual staining and imaging inconsistencies, contributing to more robust and accurate segmentation in histological image analysis.

However, despite their potential, GANs in particular present significant challenges that may limit their practical adoption in histological imaging workflows. These include the need for large amounts of training data, sensitivity to hyperparameters, and the lack of standardized evaluation metrics to assess the quality of generated outputs [9]. In order to gain a comprehensive understanding of GANs and their applications in image segmentation, a systematic review is necessary to consolidate current knowledge, evaluate the methodologies employed, and identify trends, limitations, and gaps in the literature.

This review summarizes published articles on GAN-based architectures for histological image segmentation. The reviewed studies are categorized according to the specific tasks addressed by the GANs, such as data augmentation, stain normalization, virtual stain, and tasks addressed by the segmentation in organs, including breast, kidney, liver, stomach, prostate, and other regions.

Therefore, this review study is expected to make valuable contributions by guiding researchers in selecting relevant research topics within this broad domain and by supporting the development of more effective methodological approaches. Although the use of Generative Adversarial Networks (GANs) is not new and has been explored in several review studies from different perspective [10,11,12], existing reviews do not comprehensively address the application of GANs to histological images. This gap can pose challenges for researchers entering this field. Thus, this systematic literature review aims to fill a significant research gap by providing a focused and thorough analysis of GAN-based approaches in the context of histological imaging.

The main contributions of this article can be summarized as follows:

• A thorough synthesis of the current state of the art regarding the use of generative models, particularly GANs, in histological image segmentation;
• Important perspectives on the use of GANs across different aspects, including yearly trends in GAN applications, the specific tasks addressed, the organs involved, the datasets utilized, and the evaluation criteria employed in the studies analyzed;
• A practical guidance for future research, helping to shape the direction of generative model development in histological imaging.

The remainder of this article is organized as follows: Section 2 discusses previous literature reviews in related research domains, aiming to identify existing gaps and trace the field’s evolution. Section 3 introduces the fundamental principles of histological image segmentation and explains the operational concepts of GANs. The review methodology, including the research questions, search strategy, and study selection criteria, is detailed in Section 4. Section 5 presents the review’s main findings, such as standard evaluation metrics for segmentation models and the specific contributions of GANs in medical image segmentation. Section 6 discusses the implications of the results, offering a critical analysis of the reviewed literature. Finally, Section 7 presents the study’s findings and the references used throughout the article.

2. Related Review Works

While systematic literature reviews are widely recognized as essential for understanding a research domain, it is equally important to identify prior studies with similar objectives. Doing so helps uncover existing gaps in the literature and provides insight into how the field has evolved. This section presents previous literature reviews in related areas, serving as a foundation to support the discussions throughout this article.

In recent years, medical image segmentation has significantly transformed from traditional rule-based algorithms to more advanced deep learning-based approaches. In particular, histological image segmentation, a domain characterized by complex tissue structures, staining variability, and subtle morphological patterns, has greatly benefited from data-driven methods. These modern techniques address challenges such as low contrast, staining artifacts, and overlapping cellular structures, enabling more accurate and robust segmentation of tissue components and cellular features [13,14,15].

Among the most promising approaches are GANs, which have demonstrated significant potential in histopathological image segmentation. GANs operate through adversarial training, allowing them to model complex distributions and generate realistic segmentation outputs even when working with limited or imbalanced datasets. One of the key advantages of GANs in this domain is their ability to handle domain adaptation and data bias by learning the underlying data distribution in a supervised, semi-supervised, or unsupervised fashion [16,17].

Several studies have comprehensively reviewed applying GANs in medical imaging, highlighting their promise and limitations. For instance, Sultan et al. [16], Alhumaid et al. [18] focused on the role of GANs in medical image segmentation, especially emphasizing their potential to improve image quality and structure delineation. The review covers various medical imaging modalities without specializing in histological images. The article is helpful as a general foundation but does not delve into critical aspects such as histopathology-specific challenges, staining heterogeneity, or cellular-level image characteristics. Segmentation is conceptualized, and the authors do not conduct a systematic analysis of datasets or performance metrics. In contrast, Islam et al. [19] delivers a comprehensive review of GAN applications across the medical imaging domain, covering tasks such as synthesis, augmentation, enhancement, and segmentation. However, segmentation is only treated as one of several applications, and histological images are not a central concern.

From a theoretical perspective, Gui et al. [17] offers an in-depth review of GAN algorithms, models, and training strategies. While this work is essential for understanding foundational and architectural aspects of GANs, including advanced variants such as PGGAN and WGAN, it does not focus on medical imaging tasks and completely omits segmentation-specific discussions. Therefore, it serves as a firm technical reference but does not bridge the gap toward domain application.

Specific to histopathology, Jose et al. [20] explored using GANs in digital pathology, focusing on whole-slide imaging and nuclear segmentation. This work highlights the variability of histological data and the potential of GANs to normalize and adapt images across domains. Nonetheless, it provides limited quantitative analysis and no comparison of GAN variants. Other works like Chen et al. [5] and Zhao et al. [21] centered on data augmentation and attention mechanisms, respectively. While they broaden the understanding of GAN applications, they generally do not address regulatory or ethical implications.

A more recent and comprehensive contribution by Hussain et al. [22] approached the use of GANs from the perspective of medical image reconstruction while also considering segmentation as part of the broader landscape of image enhancement tasks. The review distinguishes itself by offering a well-structured analysis of application domains, model architectures, and performance outcomes, providing valuable insights into the evolution of GAN-based methods.

Focusing specifically on image segmentation, Xun et al. [23] presented an in-depth review of GAN-based methods and their architectures tailored for medical segmentation tasks. While the review covers many techniques, it does not systematically evaluate the datasets or compare performance across benchmarks, limiting its reproducibility utility.

Unlike previous reviews that broadly address the use of GANs in general medical imaging or focus on specific applications such as data augmentation or stain normalization, this review provides a focused and structured analysis of GAN-based architectures, specifically within the context of histological image segmentation. It distinguishes itself by not only examining segmentation models but also exploring the auxiliary roles GANs play, such as domain adaptation and label refinement, within segmentation pipelines. Furthermore, the review categorizes studies according to both the technical tasks performed and the anatomical regions analyzed, offering a multidimensional perspective that is currently lacking in the literature. This level of granularity aims to support researchers and practitioners in identifying methodological gaps, understanding current trends, and selecting appropriate GAN approaches tailored to specific histological challenges.

Table 1. Comparative analysis of review articles focused on GANs in the medical area.

Study	Year	Image Type	Datasets	Task Investigated	GAN Types	Metrics	Objective
Jose et al. [20]	2021	Histopathology (WSI)	TCGA, Camelyon	Histological image processing	CycleGAN, Pix2Pix, CGAN	Qualitative	GANs in digital pathology and histology
Chen et al. [5]	2022	Radiology, microscopy	BRATS, LUNA, ISIC	Augmentation	CycleGAN, StyleGAN, DCGAN	Limited	GANs for data augmentation
Xun et al. [23]	2022	General medical imaging	ACDC, DRIVE	Segmentation	CGAN, U-GAN, SegAN	Limited	Review of segmentation using GANs
Zhao et al. [21]	2022	Radiology, pathology	Not detailed	Attention-enhanced segmentation	AttentionGAN, TransGAN	Limited	Emerging GAN architectures with attention
Gui et al. [17]	2023	General (incl. medical)	Not detailed	General overview	DCGAN, WGAN, InfoGAN, BigGAN	Theoretical	Broad review of GANs including healthcare
Islam et al. [19]	2024	Radiology, Pathology	Public/private datasets	Augmentation, Synthesis, Segmentation	DCGAN, CGAN, CycleGAN, StyleGAN	Limited	Comprehensive review on GANs in medical imaging
Sultan et al. [16]	2024	MRI, CT, Ultrasound	Not specified	Segmentation	Vanilla GAN, CGAN, CycleGAN	Limited	Conceptual overview of GANs in medical image segmentation
Hussain et al. [22]	2025	CT, MRI	Multiple (e.g., BraTS)	Reconstruction (incl. segmentation)	Progressive GANs, CycleGAN	Performance evaluation	Systematic review on GANs in image reconstruction
Ibrahim et al. [26]	2025	Multi-modal (MRI, histology, EHR)	Multimodal datasets	Synthetic data generation	Diffusion, GANs, VAEs	Limited	Synthetic data generation in medicine
This review	2025	Histology and histopathology	Multiple datasets	Segmentation	GAN, CycleGAN, PGGAN, Pix2Pix, StyleGAN, StarGAN, cGAN, WGAN	Qualitative and quantitive	Systematic review focused on GANs in histological image segmentation

presents a comparative analysis of recent and relevant studies in the literature focused on using generative adversarial networks for histological image segmentation. The study includes key aspects such as types of images addressed (including histopathological slides and whole slide images), datasets, investigated tasks, GAN models discussed, evaluation metrics, and objectives, which follows well-recognized standards to compare scientific works in SLR studies [24,25]. Moreover, the selected metrics align with those commonly used in AI-driven imaging reviews across multiple domains.

Overall, these studies underscore the increasing interest in GAN-based methods for enhancing analytical accuracy in complex tissue structures. They address critical challenges such as stain variability, scarcity of annotated data, and domain shifts across heterogeneous datasets. Based on the performed analyses, we could identify that while GANs have shown promising results in histological segmentation tasks, the current literature remains sparse and fragmented. Most reviews still concentrate on broader medical imaging domains (e.g., MRI, CT) and provide limited discussion on histology-specific applications. This reveals a clear gap in domain-focused synthesis, particularly concerning validating GANs. As presented in this study, a focused, systematic review is thus essential for understanding the capabilities, limitations, and emerging directions of GANs in histological image segmentation.

This review article focuses exclusively on using GANs for the “image segmentation task” in image histology. It examines papers published from GAN’s introduction in 2014 to February 2025. Although the analysis may not have been exhaustive, these studies are expected to provide sufficient knowledge to aid both novice and experienced researchers in selecting research topics and refining methodological approaches within the studied subject.

3. Fundaments and Basic Definitions

3.1. Histological Image Segmentation

Histology is the science dedicated to the study of biological tissues at a microscopic level. Histological images are digital representations of tissue slides, usually obtained by optical microscopy after application of specific dyes, such as H&E. These images capture a complex cellular and tissue architecture, and are fundamental for understanding tissue morphology and diagnosing diseases, especially in the field of pathology. In biomedical practice, histological images allow the identification of cellular patterns, anatomical structures and alterations that may indicate pathological processes, such as inflammation, dysplasia or cancer [27,28].

The role of histological images in medical diagnosis is irreplaceable. In diagnostic pathology, they are a basis for the diagnosis of several diseases, guiding therapeutic and prognostic decisions. Furthermore, they are essential in biomedical research, for example, to understand molecular mechanisms of diseases or evaluate responses to experimental treatments [29].

However, manual analysis of these images, traditionally performed by experienced pathologists, faces significant challenges. This process is time-consuming, subject to human fatigue, and often presents inter- and intra-observer variability, i.e., different specialists may interpret the same image differently [30]. Furthermore, the complexity of histological images, with their color variations, morphological patterns, and heterogeneous textures, makes the task even more challenging. The increase in the volume of exams due to the growing demand for health services also overloads professionals, highlighting the need for automated methods to assist in the analysis [28].

In this context, the segmentation of histological images emerges as a fundamental step in automated analysis. Segmentation aims to identify and delimit specific structures within images, such as cell nuclei, glands, collagen fibers, or tumor regions. This step is essential to extract quantitative information that complements the qualitative analysis performed by experts [27]. Practical applications of segmentation include tumor detection in histological sections [31,32], quantification of cell nuclei for tumor health studies [33], assessment of degrees of inflammation or fibrosis [30], and prognostic analysis based on morphological characteristics [34].

Despite its potential, segmentation of histological images presents significant technical difficulties. Color and intensity variations introduced by different staining protocols, texture patterns, and overlapping cellular structures make the problem challenging for traditional algorithms. In addition, artists generated during slide preparation, such as folds or bubbles, can interfere with the accuracy of segmentations [29]. These difficulties have motivated the development of more robust approaches, such as the use of deep neural networks and, more recently, models based on generative adversarial networks (GANs).

3.2. Introduction to Generative Adversarial Networks

Introduced in [35], GANs are a class of deep learning models designed to generate realistic synthetic data by learning the underlying distribution of a given dataset. The architecture of a generative adversarial network is composed of two main components: the generator $\left(G\right)$ and the discriminator $\left(D\right)$, which are trained simultaneously in an adversarial process. Figure 1 illustrates the basic architecture of a GAN model.

During the training process of a GAN, the architecture is composed of two key components: a generator G and a discriminator D. The GAN is trained using two distributions: a real data distribution $P_{\mathrm{data}}\left(X\right)$, from which a dataset X is sampled, and a noise distribution $P_z\left(Z\right)$, from which random latent variables Z are drawn.

Initially, the generator G receives as input M noise vectors $z_1,z_2,\dots ,z_m\sim P_z\left(Z\right)$ and transforms them into synthetic samples ${\widehat{x}}_1,{\widehat{x}}_2,\dots ,{\widehat{x}}_m$, denoted collectively as $\widehat{X}$. Simultaneously, M real samples $x_1,x_2,\dots ,x_m\sim P_{\mathrm{data}}\left(X\right)$ are used to train the discriminator.

The discriminator D is trained to distinguish real samples from fake ones using a gradient ascent approach. It acts as a binary classifier, aiming to output $D\left(x\right)=1$ for real data and $D\left(\widehat{x}\right)=0$ for generated data. Once D is trained for several iterations, the generator is updated via gradient descent, typically with a lower learning rate to ensure training stability.

The ultimate goal is for the generator G to produce samples that the discriminator D cannot distinguish from real ones, i.e., $D\left(\widehat{x}\right)\approx 0.5$ for $\widehat{x}=G\left(z\right)$. Mathematically, the value function $V(G,D)$ can be used to represent the adversarial game process in GAN, which is formulated as a minimax optimization problem [36], as shown in Equation (1):

$$\underset{G}{min}\underset{D}{max}V(D,G)={\mathbb{E}}_{x\sim P_{\mathrm{data}}\left(x\right)}[logD\left(x\right)]+{\mathbb{E}}_{z\sim P_z\left(z\right)}[log(1-D\left(G\left(z\right)\right))]$$

(1)

Here, the generator seeks to minimize $log(1-D(G\left(z\right)\left)\right)$, effectively attempting to fool the discriminator, while the discriminator is trained to maximize its ability to distinguish between real and fake samples by minimizing the binary cross-entropy (BCE) loss.

Despite their theoretical soundness and success in various applications, generative adversarial networks (GANs) face several practical challenges during training. Among the most frequently encountered issues are convergence difficulties, where the adversarial training dynamic between the generator G and discriminator D fails to reach a stable equilibrium [37,38]. Another key problem is GANs is highly sensitive to hyperparameters and network architectures; minor adjustments can lead to vastly different outcomes, making the training process brittle and difficult to tune [35,39]. Additionally, overfitting can occur when the discriminator becomes overly strong, memorizing the training data and failing to provide useful feedback to the generator [40].

Addressing these challenges has motivated a rich body of research proposing extensions, regularization techniques, alternative architectures, and novel training algorithms aimed at stabilizing GAN training and improving output quality.

To mitigate these issues, numerous variants and extensions of GANs have been proposed, which will be explored in subsequent sections.

3.3. Types of GANs

Since their introduction by Goodfellow et al. [35], generative adversarial networks have transformed the field of data generation and image synthesis. The original GAN architecture, despite its conceptual simplicity, faced challenges related to training instability, convergence issues, and mode collapse. In response, numerous GAN variants have been proposed to address these limitations and adapt the architecture to specific applications, such as medical image segmentation, and more recently, histological image analysis. In this subsection, we will introduce the representative GAN variants.

3.3.1. cGAN

Conditional GAN (cGAN), proposed by Mirza and Osindero [41], are an extension of the traditional GAN architecture, where both the generator and the discriminator are conditioned on auxiliary information, such as class labels or more detailed tags associated with the data. This conditional input, denoted as c, is provided to both networks as an additional input to guide the data generation process. As a result, the generator learns to produce samples that not only resemble real data but also conform to the specified condition, rather than generating generic samples from an unknown noise distribution. Equation (2) formally represents the loss function of a cGAN:

$$\underset{G}{min}\underset{D}{max}V(D,G)={\mathbb{E}}_{x\sim p_{data}\left(x\right)}[logD\left(x\right|c)]+{\mathbb{E}}_{z\sim p_z\left(z\right)}[log(1-D\left(G\left(z\right|c)\right))]$$

(2)

where $D(x\mid c)$ and $G(z\mid c)$ represent the discriminator and generator conditioned on the auxiliary information c, respectively; x denotes the real data sample, and $p_z\left(z\right)$ refers to the prior noise distribution used as input to the generator [20].

3.3.2. CycleGAN

For scenarios where paired data is unavailable, CycleGAN offers a powerful alternative. Proposed by Zhu et al. [42], it enables designed to perform image-to-image translation between two domains. Traditionally, image translation tasks rely on supervised learning with paired datasets, where each input image in one domain has a corresponding target image in the other. However, in many real-world scenarios, such paired training data are not available. CycleGAN addresses this limitation by enabling unpaired image translation—learning to map the underlying structure and style from one domain to another without the need for aligned image pairs.

The CycleGAN architecture consists of two generators, $G:X\to Y$ and $F:Y\to X$, and two discriminators, $D_X$ and $D_Y$. The discriminator $D_X$ aims to distinguish between real images from domain X and synthetic images generated by $F\left(Y\right)$, while $D_Y$ differentiates between real images in domain Y and generated images $G\left(X\right)$. To ensure consistency and preserve key features during translation, the model enforces a cycle consistency loss, encouraging $F\left(G\right(X\left)\right)\approx X$ and $G\left(F\right(Y\left)\right)\approx Y$ [36].

3.3.3. StyleGAN

StyleGAN [43,44] is a generative adversarial network architecture specifically designed to improve the control and quality of image synthesis. Unlike traditional GANs, which map a latent vector z directly to an image, StyleGAN introduces an intermediate latent space $\mathcal{W}$, allowing for greater disentanglement of image features. The latent vector is first passed through a mapping network to produce $w\in \mathcal{W}$, which is then used to modulate the layers of the generator via adaptive instance normalization (AdaIN). This mechanism enables fine-grained control over various visual attributes, such as texture, pose, and structure, at different levels of the network.

The generator in StyleGAN progressively synthesizes high-resolution images through a coarse-to-fine architecture, while the discriminator learns to distinguish real images from those generated. The result is the production of highly realistic images, especially in human face synthesis, with improved variability and control over style compared to earlier GAN models.

3.3.4. PGGAN

Progressive Growing of GANs (PGGAN), proposed by Karras et al. [45], introduced a novel training strategy for generative adversarial networks that significantly enhances the quality, stability, and variation of high-resolution image synthesis. Unlike traditional GANs, which are trained with fixed-size networks on full-resolution images, PGGAN starts with low-resolution images (e.g., $4\times 4$) and progressively increases the resolution by adding layers to both the generator and the discriminator during training. This incremental growth allows the network to first learn global structures before focusing on fine-grained details.

The standard adversarial objective remains the same (Equation (3)). However, PGGAN introduces two core innovations to stabilize training and improve output quality:

Progressive architecture growth: new layers are gradually added to the generator and discriminator. During transitions, a linear interpolation (fade-in) smoothly blends the outputs from the lower and higher-resolution paths:

$$x_{\mathrm{output}}=(1-\alpha )·x_{\mathrm{low}}+\alpha ·x_{\mathrm{high}}$$

(3)

where $\alpha \in [0,1]$ increases over time as the network adapts to the new resolution.

Improved normalization: instead of traditional batch normalization, PGGAN employs pixelwise feature vector normalization in the generator and minibatch standard deviation in the discriminator to reduce mode collapse and stabilize training.

Furthermore, Karras et al. [43,45] evaluate the performance using the Fréchet Inception Distance (FID), a perceptual quality metric that measures both the fidelity and diversity of generated images. PGGAN showed significant improvements over prior GAN variants, especially for complex datasets.

In medical imaging, particularly in histopathology, the ability to synthesize realistic, high-resolution tissue structures is highly valuable. PGGAN has been adopted for synthetic data generation and augmentation in tasks such as tumor detection, gland segmentation, and classification of histological patterns, helping to improve the robustness of machine learning models where real annotated data is scarce [46,47,48].

3.3.5. WGAN

The Wasserstein GAN (WGAN), introduced by Arjovsky et al. [49], addresses key training instabilities and mode collapse issues observed in traditional GANs by replacing the Jensen–Shannon (JS) divergence with the Earth Mover (EM) distance, also known as the Wasserstein-1 distance.

Given two probability distributions $P_r$ (real data) and $P_g$ (generated data), the Wasserstein-1 distance is defined as:

$$W(P_r,P_g)=\underset{\gamma \in \mathsf{\Pi }(P_r,P_g)}{inf}{\mathbb{E}}_{(x,y)\sim \gamma }[\parallel x-y\parallel ]$$

(4)

where $\mathsf{\Pi }(P_r,P_g)$ denotes the set of all joint distributions $\gamma (x,y)$ whose marginals are $P_r$ and $P_g$.

Using the Kantorovich-Rubinstein duality, the WGAN objective becomes:

$$\underset{D\in \mathcal{D}}{max}{\mathbb{E}}_{x\sim P_r}\left[D\left(x\right)\right]-{\mathbb{E}}_{z\sim P_z}\left[D\left(G\left(z\right)\right)\right]$$

(5)

Here, D (often referred to as the critic instead of a discriminator) is constrained to be 1-Lipschitz, originally enforced via weight clipping. The generator G is trained to minimize this objective.

The use of the Wasserstein distance provides a smoother and more meaningful loss metric that correlates better with the quality of generated samples. As a result, WGAN improves convergence and offers more stable training, even in cases of poor discriminator performance.

In practice, the WGAN critic loss is:

$$L_D=-{\mathbb{E}}_{x\sim P_r}\left[D\left(x\right)\right]+{\mathbb{E}}_{z\sim P_z}\left[D\left(G\left(z\right)\right)\right]$$

(6)

and the generator loss is:

$$L_G=-{\mathbb{E}}_{z\sim P_z}\left[D\left(G\left(z\right)\right)\right]$$

(7)

In summary, WGAN and its variants have become valuable tools in medical image processing, enabling artifact correction, anomaly detection, and synthetic data generation, ultimately contributing to improved diagnostic performance and model reliability [50,51]. These models address key challenges in the field such as the scarcity of labeled data, noise suppression, and the synthesis of realistic medical images.

3.3.6. Pix2Pix

Pix2Pix is a conditional generative adversarial network (cGAN) architecture proposed by Isola et al. [52], specifically designed for image-to-image translation tasks. Unlike traditional GANs that learn to generate images from random noise, Pix2Pix learns a mapping from an input image to an output image, making it suitable for tasks such as image colorization, edge-to-photo translation, and semantic segmentation.

The core idea of Pix2Pix is to train a generator G that produces an output image $G(x,z)$, given an input image x and optional noise z, such that the output image resembles a target image y. A discriminator D is simultaneously trained to distinguish between real image pairs $(x,y)$ and fake pairs $(x,G(x,z\left)\right)$. The conditional adversarial loss that guides this process is defined as:

$${\mathcal{L}}_{cGAN}(G,D)={\mathbb{E}}_{x,y}[logD(x,y)]+{\mathbb{E}}_{x,z}[log(1-D(x,G(x,z)))]$$

(8)

To further encourage the generator to produce outputs close to the ground truth, a traditional $L_1$ loss is added:

$${\mathcal{L}}_{L1}\left(G\right)={\mathbb{E}}_{x,y,z}{[\parallel y-G(x,z)\parallel }_1]$$

(9)

The final objective function for the Pix2Pix model is a weighted combination of the adversarial loss and the $L_1$ loss:

$$G^{*}=arg\underset{G}{min}\underset{D}{max}{\mathcal{L}}_{cGAN}(G,D)+\lambda {\mathcal{L}}_{L1}\left(G\right)$$

(10)

where $\lambda$ is a hyperparameter that balances the two loss terms.

4. Review Methodology

This systematic literature review was conducted following a structured methodology based on the PRISMA guidelines [25,53] (see Supplementary File PRISMA checklist). The review process included the definition of research questions, the construction of a comprehensive search strategy, the selection of relevant studies through predefined inclusion and exclusion criteria, and the extraction of data. The quality of the included studies was assessed using a standardized checklist to ensure methodological rigor and transparency. The overall process was documented using a PRISMA flow diagram to provide a clear overview of the study selection process.

To ensure the novelty of our systematic review, we consulted the PROSPERO database [54], which serves as a registry for systematic review protocols. This platform is crucial in promoting research transparency and minimizing redundancy by publicly making information on ongoing and planned reviews available. Our search did not identify any registered reviews that addressed the same research question, indicating a gap in the existing literature.

4.1. Research Questions

This review was structured around a set of research questions formulated to examine the application of generative adversarial networks in histological image segmentation. The objective was to analyze the breadth of applications, methodological approaches, and evaluation strategies reported in the existing body of literature. To guide this investigation, the following Research Questions (RQs) were established:

• (RQ1) Which types of histological tissues are used in studies applying generative adversarial networks (GANs) for image segmentation tasks?
• (RQ2) Which datasets are used in studies?
• (RQ3) What are the main goal of applying GANs to histological segmentation tasks?
• (RQ4) Which GAN architectures are used in the studies?
• (RQ5) What metrics are used to evaluate the performance of GANs in the proposed tasks?
• (RQ6) What is the specific goal of the segmentation performed in the developed studies?
• (RQ7) Which segmentation architecture is used in the studies?
• (RQ8) What metrics are used to evaluate the segmentation quality?

4.2. Search Strategy

A systematic search for primary studies was conducted on 4 February 2025, using the following digital libraries: Springer Link, IEEE Xplore (Institute of Electrical and Electronics Engineers), Scopus, Multidisciplinary Digital Publishing Institute (MDPI), and PubMed. The search process was carried out by one researcher and validated by two additional researchers to ensure consistency and reliability. The search terms were defined based on the main components of the research, technology, task, and domain, and included the following combination of keywords and their synonyms: (“generative adversarial network” OR “GAN”) AND (“segmentation” OR “image segmentation” OR “semantic segmentation”) AND (“histology” OR “histological” OR “histopathology” OR “histopathological”).

4.3. Selection Criteria

The study selection process was conducted in three main stages, following predefined inclusion and exclusion criteria, as outlined in

Table 2. Inclusion and exclusion criteria for studies.

Inclusion Criteria (IC)	Exclusion Criteria (EC)
(IC1) Studies using GANs for segmentation of histological images.	(EC1) Studies using GANs only to synthesize images, without a focus on segmentation.
(IC2) Studies with experimental validation (quantitative or qualitative).	(EC2) Works that do not involve histological or microscopic images.
(IC3) Published between 2014 and 2025.	(EC3) Studies using GANs only for classification or detection, not segmentation.
(IC4) Articles in English.	(EC4) Articles in a language other than English.
(IC5) Full articles.	(EC5) Review, surveys, Case studies, reports, short papers, conference, abstracts, communications, theses, and dissertations.

. In the first stage, duplicate records retrieved from different databases were removed. In the second stage, during the title and abstract screening, independent reviewers evaluated whether the studies met the inclusion criteria. In the third stage, the full texts of the remaining articles were assessed in detail to confirm their eligibility. In cases of disagreement, a third reviewer was consulted or the issue was resolved through discussion.

4.4. Data Extraction and Synthesis

To ensure consistency, reproducibility, and reliability in the data extraction process, a structured form was developed using Google Forms. This instrument was carefully designed to encompass all elements necessary to answer the research questions, comprehensively characterize the selected studies, and identify gaps and promising trends for future investigations. Each article was read in full by the reviewers, and the relevant data were systematically and consistently extracted. Subsequently, Microsoft Excel (version 2504, Build 16327.20264) was employed as a supporting tool for organizing and systematizing the extracted data. The information collected from each article included in this systematic review covered key aspects essential for a detailed comparative analysis of GAN-based approaches applied to histological image segmentation. Among the extracted data were the year of publication, the organ analyzed in the study, the specific segmentation domain, the nature of the dataset used (public or private), the dataset name, the GAN technique employed, the metrics used to evaluate GAN performance along with their respective results, the specific task assigned to the GAN, the segmentation technique applied, the evaluation metrics for segmentation and their results, the experimental setting or environment, the availability of source code or algorithm URLs, the type of learning employed (supervised, unsupervised, or semi-supervised), and the overall objective of the study as reported by the authors. The synthesized results are presented in Supplementary File S1.

5. Results

5.1. Search Results

The study selection process was conducted in three phases, as illustrated by the PRISMA in Figure 2. Initially, after searching five databases, we retrieved a total of 1227 articles. Initially, after searching five databases, we retrieved a total of 1227 articles. In the first phase, identification, 41 duplicate articles were excluded. In the next phase, screening, we performed title and abstract reviews, excluding 965 articles that did not meet the initial criteria. Subsequently, 221 reports were sought for retrieval, of which 4 could not be accessed in full text and were therefore excluded. In the final phase, eligibility, 217 articles were assessed through full-text reading. After detailed analysis, 175 articles were discarded, resulting in a final total of 41 articles included in the review.

5.2. Applications and Trends of GANs in Histopathological Image Segmentation

The application of generative adversarial networks in medical image analysis has gained significant momentum in recent years due to their ability to synthesize high-quality, realistic data and enhance image characteristics across various clinical tasks. In histopathological image processing, GANs have been particularly valuable for tasks such as data augmentation, stain normalization, and image-to-image translation, offering robust solutions to data scarcity and variability challenges [51,55]. Furthermore, segmentation tasks have also benefited from GAN-based approaches, which often incorporate adversarial losses to refine boundary accuracy and structural consistency [50]. The following figures provide a quantitative overview of how GANs have been utilized across different segmentation tasks and target regions in histological imaging.

Figure 3 and Figure 4 provide a detailed quantitative overview of the distribution of GAN-related research in histological image segmentation. As illustrated in Figure 3, the most frequently addressed task using GANs was data augmentation, appearing in 20 studies, followed by stain normalization with 12 studies. Other relevant applications included virtual staining, color normalization, and synthetic image generation, each represented in 4 studies. Less frequently explored were domain adaptation and reconstruction. This distribution highlights a clear research emphasis on preprocessing and data enrichment tasks, particularly those aimed at improving dataset variability and quality in histopathological imaging.

Figure 4 categorizes the segmented regions targeted by these studies. The majority focused on nuclei segmentation, accounting for 20 studies, indicating a strong interest in cellular-level analysis. Segmentation of combined structures such as tissue and cells (8 studies), tissue alone (6 studies), and tissue and nuclei (2 studies) was also present, reflecting the complexity and layered structure of histological data. Fewer works addressed cell-only segmentation (1 study), cell and nuclei combined (2 studies), and the segmentation of area, count, and distance between cells (1 study). These findings suggest that nuclei-focused segmentation remains a priority, while more integrative or structural analyses involving multiple tissue components are less commonly pursued.

Together, these results reinforce the role of GANs as versatile tools in histological image segmentation, particularly in tasks that support dataset augmentation and normalization, and in anatomically detailed applications such as nuclei delineation.

Yearly Distribution

The analysis presented in Figure 5 highlights the yearly progression of GAN-based research focused on histological image analysis. From the first identified study in 2018 through early 2025, a total of 41 publications met the selection criteria. The data indicate a growing interest in leveraging GANs for histopathology applications, starting with a single publication in 2018, gradually increasing to 3 in 2019 and 2 in 2020. A more pronounced growth can be observed from 2021 onward, with 6 publications in 2021, 9 in 2022, and peaking at 10 in 2023. Although there was a slight decrease in 2024 (7 publications) and 2025 (3 publications as of the analysis date), the trend overall reflects the expanding role of GANs in supporting histological image processing tasks such as stain normalization, synthetic image generation, and segmentation. This upward trajectory underscores the increasing adoption of GANs as powerful tools to enhance diagnostic workflows, improve image quality, and support data augmentation in histopathological research.

5.3. Datasets

The studies included in this review used a variety of datasets to train and evaluate GAN-based models applied to histological image segmentation. The

Table 3. Overview of the datasets used in the included studies.

Dataset	Paper	Type of Tissue	Size	Magnification	Labeled	Availability
ANHIR	[56]	Kidney, breast, colon, lung and gastric mucosa	231 images (WSI) (100,000 × 200,000)	20× and 40×	Yes	Public
BBBC-data	[57,58]	Liver, Brain, Colon, Lung, Ovary, Kidney, Blood, Immune System, Embryo, Heart and Nervous System	700 images	20×	Yes	Public
BMCD-FGCD	[59]	Bone marrow	29 images (WSI) (1500 × 800)	—	Yes	Public
CAMELYON16	[60]	Breast (lymph nodes)	400 images (WSI) (100,000 × 200,000)	20× and 40×	Yes	Public
CARS	[61]	Thyroid	200 images (256 × 256)	40×	Yes	Private
CoNiC	[62]	Colon	4981 patches (256 × 256)	—	Yes	Public
CoNSep	[58,63,64]	Adrenal, Larynx, Lymph node, Mediastinum.	30 images (512 × 512)	40×	Yes	Public
CPM17	[58,63,64,65]	Colon	64 images (500 × 500)	20× and 40×	Yes	Public
Crag	[66,67]	Colorectal carcinome	213 images (1512 × 1512)	20×	Yes	Public
CryoNuSeg	[63]	Adrenal gland, larynx, lymph node, mediastinum, pancreas, pleura, testis, thymus and Thyroid gland	30 images (512 × 512)	20×	Yes	Public
Digestpath	[66]	Colorectal carcinoma	46 images (5000 × 5000)	20×	Yes	Public
Glomeruli	[56]	Kidney	200 images (WSI) (1000 × 1000)	40×	Yes	Public
HN Cancer	[68]	Head and Neck	1562 images (1024 × 1280)	10×	Yes	Private
HuBMAP	[69]	Kidney	9 images (WSI)	40×	Yes	Public
ICIAR-BACH	[70]	Breast	400 images (2048 × 1536)	20×	Yes	Public
In-House	[67]	Colorectal	1093 images (WSI) (512 × 512)	40×	Yes	Public
In-House-MIL	[67]	Kidney, Lung, Breast, Prostate, Endometrium	260 images (WSI) (10,000 × 10,000)	40×	Yes	Public
ICPR14 e 12	[59,71]	Breast	163 images (WSI) (2000 × 2000)	40×	Yes	Public
Kather	[67]	Large intestine (colon and rectum)	25,058 images (224 × 224)	20×	Yes	Public
KidneyArtPathology	[72]	Kidney	5000 images (512 × 512)	20× and 40×	Yes	Public
KIRC	[58,63,64,73]	Kidney	>190,000 images (WSI) (100,000 × 100,000)	20×	No	Public
KPMP	[69]	Kidney	85 images (WSI)	40×	Yes	Public
Kumar	[58,64,65,67,74]	Breast, Liver, Kidney, Prostate, Bladder, Colon and Stomach	30 images (1000 × 1000)	40×	Yes	Public
Lizard	[67]	Large intestine (colon and rectum)	133 images (1016 × 917)	20×	Yes	Public
Lung dataset	[56]	Lung	23,744 patchs	40×	-	Private
LYON19 (IHC)	[58]	Breast, colon, prostate	871 images (256 × 256)	40×	Yes	Public
MICCAI’16 GlaS	[68,70,75,76,77]	Large intestine (colon)	165 images (775 × 522)	20×	Yes	Public
MITOS-ATYPIA 14	[70,76]	Breast	1200 images (1539 × 1376)	10×, 20× and 40×	Yes	Public
MoNuSAC	[63,78]	Lung, prostate, kidney and breast	294 images (113 × 81 and 1398 × 1956)	40×	Yes	Public
MoNuSeg	[63,68,79,80]	Prostate, Lung, Kidney, Colon, Breast, Pancreas, Oral cavity, Stomach, Liver and Bladder	44 images (1000 × 1000)	40×	Yes	Public
MThH	[58]	Colon	36,000 images (256 × 256)	20× and 40×	Yes	Public
MUC1	[81]	Breast	13,000 images (256 × 266)	40×	No	Private
N7, E17, N5	[82]	Kidney (mouse)	1917 patchs (128 × 128)	—		Private
NSCLC	[83]	Lung	269,000 pachs (128 × 128)	10×	Yes	Public
Onco-SG	[84]	Lung	57 images (1792 × 768)	20×	Yes	Public
PanNuke	[62,67,80]	Breast, Lung, Prostate, Kidney, Brain, Colon and Liver	7901 images (256 × 256)	20× and 40×	Yes	Public
Roche	[85]	Liver (colorectal)	50 images (WSI) (512 × 512)	10x		Private
RWTH Aachen	[69]	Kidney	1009 images (WSI)	40×	Yes	Private
TCGA	[73,77,79,81,86,87,88,89,90,91]	Breast, Liver, Kidney, Prostate, Bladder, Brain, Colon and Stomach	>20,000 WSIs (100,000 × 100,000)	20× and 40×	No	Public
TNBC	[58,63,64,74,79]	Breast	50 images (512 × 512)	20× and 40×	Yes	Public
TUPAC16	[70,71,77]	Breast	821 images (WSI)	40×	Yes	Public
VALIGA	[69]	Kidney	648 images (WSI)	40×	Yes	Private
-	[92]	Liver	64, 128, 256, 512 e 1024 patchs	40×	Yes	Private
-	[93]	Colon and Kidney	100 images (WSI)	—	Yes	Private
-	[94]	Kidney	10 images	40×	Yes	Private
-	[95]	Prostate and Colon	102 images (WSI) (1024 × 1024)	10×	Yes	Public

provides a comprehensive overview of the datasets employed in histopathological image studies. It offers a structured analysis based on tissue type, image size, magnification, annotation, and availability criteria.

First, various tissue types are represented across the datasets, supporting the development of generalizable models. Dataset sizes range from fewer than 100 to hundreds of thousands of images, with resolutions spanning small patches to full whole-slide images (WSIs). Most images were acquired at 20× or 40× magnification, critical for capturing histological detail, though some datasets lack magnification metadata, which may hinder reuse. High-quality annotations are available in many datasets, enabling precise model training, though some (e.g., TCGA, KIRC) lack detailed labels, restricting their use to unsupervised or semi-supervised approaches. Public availability of most datasets promotes reproducibility and algorithm benchmarking, but access to some remains restricted, limiting transparency and broader validation in clinical settings.

5.4. GANs in Histological Image

The application of GANs in digital pathology has expanded rapidly, serving crucial roles in tasks such as data augmentation, stain normalization, virtual staining, synthetic image generation, domain transformation and adaptation; and High-resolution image reconstruction. An extensive review of recent studies reveals the most commonly used models, the specific tasks they address, settings, and their quantitative performance metrics.The Table 4 presents a wide range of applications of GANs in the field of digital histology image processing.

5.4.1. Data Augmentation

Employing GANs for data augmentation enables the generation of histological images with significant morphological diversity, thereby enhancing the reliability and adaptability of segmentation, classification, and detection models [96]. For example, in a study by Kweon et al. [86], PGGAN was utilized on an RTX 3090 GPU to produce synthetic images, achieving FID = 34.96 and IS = 2.91, indicating satisfactory visual fidelity and diversity. Conversely, Hossain et al. [73] utilized CycleGAN but with lower performance metrics (MSE = 5.664, SSIM = 0.204), showcasing the variation in image quality depending on the GAN architecture and implementation specifics.

Noteworthy is the study by Baykal et al. [76] which combined CycleGAN and RRAGAN, achieving superior results with SSIM = 0.933 and PSNR = 24.109, utilizing Google Colab for training. These outcomes underscore how the selection of architecture and computational resources impacts the effectiveness of data augmentation tasks. Apart from enhancing model performance, data augmentation with GANs aids in preventing overfitting, enhancing class balance, and facilitating deep neural network training in resource-constrained settings, thereby serving as both a technological advancement and a solution to systemic challenges in medical data availability. This positions data augmentation as a fundamental application of GANs in the realms of digital pathology and biomedical imaging.

5.4.2. Stain Normalization

Stain normalization has emerged as one of the most critical and frequently addressed applications of generative GANs in digital pathology. To overcome this challenge, numerous studies have adopted GAN-based approaches that aim to harmonize the visual characteristics of histological slides while preserving the morphological integrity of the tissues [67,68,97]. Among the various architectures, CycleGAN has become one of the most commonly used models due to its ability to perform unpaired image-to-image translation, a crucial advantage when paired samples across different staining domains are unavailable. For instance, Puma et al. [68] applied CycleGAN to normalize slides from diverse datasets, achieving strong visual fidelity as indicated by Fréchet Inception Distance (FID) scores of 15.32 on MoNuSeg and 6.80 on head and neck cancer datasets.

Other notable implementations of CycleGAN include its use by Song et al. [67], who reported an AJI of 0.6570 and PQ of 0.625 in downstream nucleus segmentation, as well as Gasermayr et al. [97], who explored its utility in unsupervised stain adaptation. Additionally, Taieb and Hamarneh [75] pioneered early stain normalization with GANs, achieving Inception Scores (IS) of 0.700 for MITOSIS, 0.950 for COLON, and 0.260 for OVARY datasets using adversarial training on an NVIDIA Titan X GPU.

Table 4. Comparative analysis of GAN models for histological image processing tasks.

Work	Task	Model	Setting	Performance
Azam et al. (2024) [84]	Virtual Staining	Pix2Pix-GAN CUT	-	$Pix2Pix:FID=11.16,CUT:FID=38.52$
Bouteldja et al. (2022) [69]	Data augmentation, Stain Normalization	CycleGAN	GPU NVIDIA A100, 10 GB VRAM	-
De Bel et al. (2021) [93]	Data augmentation, Stain normalization	CycleGAN	-	-
Deshpande et al. (2022) [66]	Synthetic image generation	GAN	GPU NVIDIA GTX TITAN X single, 12 GB RAM	$FID=142.84$
Deshpande et al. (2024) [62]	Data augmentation	SynCLay	-	CoNiC-$FID=60,15\pm 1.07,$ $SSIM=0.86\pm 0.002$, PanNuke-$FID=100.80\pm 1.83,$ $SSIM=0.79\pm 0.02$
Du et al. (2025) [70]	Stain normalization	P2P-GAN, DSTGAN	GPU NVIDIA RTX 4090	TUPAC: $SSIM=0.985,PCC=0.992,PSNR=39.130\pm 0.844$; MITOS: $SSIM=0.984,PCC=0.991,PSNR=37.698\pm 2.635$; ICIAR: $SSIM=0.984,PCC=0.992,PSNR=38.399\pm 1.750$; MICCAI: $SSIM=0.975,PCC=0.990,PSNR=34.174\pm 0.892$.
Falahkheirkhah et al. (2023) [95]	Synthetic image generation	Pix2Pix-GAN	GPU NVIDIA 2080	$Mean=4.9379\mathrm{and}SD=0.7177$
Fan et al. (2024) [65]	Color normalization	CycleGAN	-	-
Gadermayr et al. (2019) [97]	Stain normalization	CycleGAN	GPU NVIDIA GTX 1080 Ti	-
Guan et al. (2024) [56]	Stain Normalization and Virtual staining	GramGAN	NVIDIA RTX 3090 GPU, Intel (TM) i5-10400 CPU, 16 GB RAM	CSS = 0.714, FID = 52.086, KID = 5.815
He et al. (2022) [61]	Synthetic Image generation	cGAN	GPU NVIDIA TITAN RTX, 24 GB VRAM	$SSIM=0.9066\pm 0.065,$ $MSE=0.0076\pm 0.003$
Hou et al. (2020) [87]	Data augmentation	GAN	-	-
Hossain et al. (2024) [73]	Data augmentation	CycleGAN	-	$MSE=5.664,PSNR=10.610,$ $SSIM=0.204$
Hossain et al. (2023) [91]	Data augmentation	CycleGAN	-	$SSIM=0.335,MSE=5.094,PSNR=11.111$
Hu et al. (2018) [59]	Data augmentation and Stain normalization	WGAN-GP	NVIDIA Tesla V100 GPU with 32 GB of memory, CPU Intel Xeon, 128 GB of RAM	-
Juhong et al. (2022) [81]	Reconstruction high-resolution image	SRGAN- ResNeXt	GPU NVIDIA RTX 2060, CPU Intel Core i7-9750H, 16 GB RAM	$PSNR=32.13,$ $SSIM=0.93,$ $MSE=2.75$
Kablan (2023) [76]	Data augmentation, Stain Normalization	CycleGAN RRAGAN	Google Colabority, GPU Tesla K80, 12 GB RAM	$FSIM=0.921,PSNR=24.109,SSIM=0.933,$ $MSE=2.88\times {10}^2,MSSSIM=0.965,RMSE=8.3576,$ $ERGAS=4.29\times {10}^3,UQI=0.987,RASE=6.18\times {10}^2$
Kapil et al. (2021) [83]	Data augmentation and Domain transformation	CycleGAN DASGAN	GPU NVIDIA V100, 32 GB RAM and GPU NVIDIA K80	-
Kweon et al. (2022) [86]	Data augmentation	PGGAN	Intel Core i7-10700, NVIDIA RTX 3090 GPU 24 GB RAM	$FID=34.9635,$ $IS=2.91609\pm 0.0387$
Lafarge et al. (2019) [77]	Stain normalization and Data augmentation	DANN	-	-
Lahiani et al. (2020) [85]	Virtual staining	CycleGAN	GPU NVIDIA Testa v100	$CWSSIM=0.880\pm 0.170$
Li, Hu and Kak (2023) [63]	Data augmentation	G-SAN	AMD Ryzen 7 5800X, 32 GB of RAM, NVIDIA RTX 3090 GPU, 24 GB of memory	-
Liu, Wagner and Peng (2022) [57]	Data augmentation	GAN	GPU Tesla V100, CPU Intel Xeon 6230, 128 GB RAM	-
Lou Wei et al. (2022) [79]	Data augmentation	CSinGAN	NVIDIA Tesla V100	-
Mahapatra and Maji (2023) [90]	Color normalization	LSGAN TredMiL	-	$NMI=0.800,$ $BiCC=0.750,WsCC=0.590$
Mahmood et al. (2019) [88]	Stain normalization	cGAN	Intel Xeon E5-2699 v4, 256 GB of RAM, NVIDIA Tesla V100 GPU	-
Naglah et al. (2022) [92]	Data augmentation, Color normalization, Domain transformation	cGAN CycleGAN	-	cGAN: $MMI=0.26\pm 0.07,NMI=0.10\pm 0.03,$ $HC=0.75\pm 0.13,BCD=0.31\pm 0.06$; CycleGAN: $MMI=0.24\pm 0.08,NMI=0.09\pm 0.03,$ $HC=0.61\pm 0.26,BCD=0.35\pm 0.15$.
Purma et al. (2024) [68]	Stain normalization	CycleGAN	GPU NVIDIA Tesla V100, 16 GB de memória, Intel Xeon CPU E5-2680 v4	FID: ClaS = 82.44, FID: MoNuSeg = 15.32, FID: HN cancer = 6.80, FID: Combined = 7.63
Razavi et al. (2022) [71]	Data augmentation and Stain Normalization	cGAN MiNuGAN	GPU NVIDIA 2080 Ti, 11 GB RAM	-
Rong et al. (2023) [89]	Color Normalization	Restore-GAN	GPU NVIDIA P100, 16 GB RAM	$SSIM=0.949\pm 0.020,$ $PSNM=30.678\pm 1.624$
Shahini et al. (2025) [78]	Synthetic image generation	ViT-P2P	-	-
Song et al. (2023) [67]	Stain normalization	CycleGAN	Nvidia RTX 3090 GPU, 24 GB of GPU memory, Intel Xeon, 64 GB of RAM	-
Taieb and Hamarneh (2017) [75]	Stain Normalization	GAN	NVIDIA Titan X GPU	$MITOSI\mathrm{S-}IS=0.700,$ $COLO\mathrm{N-}IS=0.950,$ $OVAR\mathrm{Y-}IS=0.260$
Vasiljevic et al. (2021) [94]	Data augmentation	CycleGAN, StarGAN	-	$CycleGAN:FID=5.359,$ $StarGAN:FID=11.572$
Vasiljevic et al. (2023) [72]	Data augmentation and Virtual staining	CycleGAN HistoStarGAN	-	$FID=56.00,SSIM=0.81$
Wang et al. (2021) [82]	Stain normalization	ACCP-GAN	GPU NVIDIA RTX, CPU Intel Xeon Silver 4110, 128 GB RAM	$FSIM=0.9657,M\mathrm{S-}SSIM=0.9694,$ $PSNR=27.5058,VIF=0.9031,$ $FID=27.1231$
Wang et al. (2022) [80]	Data augmentation	GAN	NVIDIA GTX 1070, 16 GB RAM	-
Ye et al. (2025) [60]	Stain normalization	SASSL	-	-
Yoon et al. (2024) [74]	Virtual staining	CUT, E-CUT CycleGAN E-CycleGAN	GPU Nvidia RTX 3090 AMD EPYC 7302 CPU, 346 GB of RAM	FID-CycleGAN: 67.76, E-CycleGAN: 61.91, CUT: 54.87, E-CUT: 50.91 KID-CycleGAN: 2.290, E-CycleGAN: 1.631, CUT: 0.642, E-CUT: 0.245
Zhang et al. (2021) [64]	Data augmentation	MASG-GAN	GPU NVIDIA RTX 2080 Ti, CPU Intel (TM) i9-10850 K, 32 GB RAM	-
Zhang et al. (2022) [58]	Color normalization and Virtual staining	CSTN	NVIDIA RTX 3090 GPU	$FID=101.37,$ $KID=6.129,AHR=1.98$

Table 4. Comparative analysis of GAN models for histological image processing tasks.

Work	Task	Model	Setting	Performance
Azam et al. (2024) [84]	Virtual Staining	Pix2Pix-GAN CUT	-	$Pix2Pix:FID=11.16,CUT:FID=38.52$
Bouteldja et al. (2022) [69]	Data augmentation, Stain Normalization	CycleGAN	GPU NVIDIA A100, 10 GB VRAM	-
De Bel et al. (2021) [93]	Data augmentation, Stain normalization	CycleGAN	-	-
Deshpande et al. (2022) [66]	Synthetic image generation	GAN	GPU NVIDIA GTX TITAN X single, 12 GB RAM	$FID=142.84$
Deshpande et al. (2024) [62]	Data augmentation	SynCLay	-	CoNiC-$FID=60,15\pm 1.07,$ $SSIM=0.86\pm 0.002$, PanNuke-$FID=100.80\pm 1.83,$ $SSIM=0.79\pm 0.02$
Du et al. (2025) [70]	Stain normalization	P2P-GAN, DSTGAN	GPU NVIDIA RTX 4090	TUPAC: $SSIM=0.985,PCC=0.992,PSNR=39.130\pm 0.844$; MITOS: $SSIM=0.984,PCC=0.991,PSNR=37.698\pm 2.635$; ICIAR: $SSIM=0.984,PCC=0.992,PSNR=38.399\pm 1.750$; MICCAI: $SSIM=0.975,PCC=0.990,PSNR=34.174\pm 0.892$.
Falahkheirkhah et al. (2023) [95]	Synthetic image generation	Pix2Pix-GAN	GPU NVIDIA 2080	$Mean=4.9379\mathrm{and}SD=0.7177$
Fan et al. (2024) [65]	Color normalization	CycleGAN	-	-
Gadermayr et al. (2019) [97]	Stain normalization	CycleGAN	GPU NVIDIA GTX 1080 Ti	-
Guan et al. (2024) [56]	Stain Normalization and Virtual staining	GramGAN	NVIDIA RTX 3090 GPU, Intel (TM) i5-10400 CPU, 16 GB RAM	CSS = 0.714, FID = 52.086, KID = 5.815
He et al. (2022) [61]	Synthetic Image generation	cGAN	GPU NVIDIA TITAN RTX, 24 GB VRAM	$SSIM=0.9066\pm 0.065,$ $MSE=0.0076\pm 0.003$
Hou et al. (2020) [87]	Data augmentation	GAN	-	-
Hossain et al. (2024) [73]	Data augmentation	CycleGAN	-	$MSE=5.664,PSNR=10.610,$ $SSIM=0.204$
Hossain et al. (2023) [91]	Data augmentation	CycleGAN	-	$SSIM=0.335,MSE=5.094,PSNR=11.111$
Hu et al. (2018) [59]	Data augmentation and Stain normalization	WGAN-GP	NVIDIA Tesla V100 GPU with 32 GB of memory, CPU Intel Xeon, 128 GB of RAM	-
Juhong et al. (2022) [81]	Reconstruction high-resolution image	SRGAN- ResNeXt	GPU NVIDIA RTX 2060, CPU Intel Core i7-9750H, 16 GB RAM	$PSNR=32.13,$ $SSIM=0.93,$ $MSE=2.75$
Kablan (2023) [76]	Data augmentation, Stain Normalization	CycleGAN RRAGAN	Google Colabority, GPU Tesla K80, 12 GB RAM	$FSIM=0.921,PSNR=24.109,SSIM=0.933,$ $MSE=2.88\times {10}^2,MSSSIM=0.965,RMSE=8.3576,$ $ERGAS=4.29\times {10}^3,UQI=0.987,RASE=6.18\times {10}^2$
Kapil et al. (2021) [83]	Data augmentation and Domain transformation	CycleGAN DASGAN	GPU NVIDIA V100, 32 GB RAM and GPU NVIDIA K80	-
Kweon et al. (2022) [86]	Data augmentation	PGGAN	Intel Core i7-10700, NVIDIA RTX 3090 GPU 24 GB RAM	$FID=34.9635,$ $IS=2.91609\pm 0.0387$
Lafarge et al. (2019) [77]	Stain normalization and Data augmentation	DANN	-	-
Lahiani et al. (2020) [85]	Virtual staining	CycleGAN	GPU NVIDIA Testa v100	$CWSSIM=0.880\pm 0.170$
Li, Hu and Kak (2023) [63]	Data augmentation	G-SAN	AMD Ryzen 7 5800X, 32 GB of RAM, NVIDIA RTX 3090 GPU, 24 GB of memory	-
Liu, Wagner and Peng (2022) [57]	Data augmentation	GAN	GPU Tesla V100, CPU Intel Xeon 6230, 128 GB RAM	-
Lou Wei et al. (2022) [79]	Data augmentation	CSinGAN	NVIDIA Tesla V100	-
Mahapatra and Maji (2023) [90]	Color normalization	LSGAN TredMiL	-	$NMI=0.800,$ $BiCC=0.750,WsCC=0.590$
Mahmood et al. (2019) [88]	Stain normalization	cGAN	Intel Xeon E5-2699 v4, 256 GB of RAM, NVIDIA Tesla V100 GPU	-
Naglah et al. (2022) [92]	Data augmentation, Color normalization, Domain transformation	cGAN CycleGAN	-	cGAN: $MMI=0.26\pm 0.07,NMI=0.10\pm 0.03,$ $HC=0.75\pm 0.13,BCD=0.31\pm 0.06$; CycleGAN: $MMI=0.24\pm 0.08,NMI=0.09\pm 0.03,$ $HC=0.61\pm 0.26,BCD=0.35\pm 0.15$.
Purma et al. (2024) [68]	Stain normalization	CycleGAN	GPU NVIDIA Tesla V100, 16 GB de memória, Intel Xeon CPU E5-2680 v4	FID: ClaS = 82.44, FID: MoNuSeg = 15.32, FID: HN cancer = 6.80, FID: Combined = 7.63
Razavi et al. (2022) [71]	Data augmentation and Stain Normalization	cGAN MiNuGAN	GPU NVIDIA 2080 Ti, 11 GB RAM	-
Rong et al. (2023) [89]	Color Normalization	Restore-GAN	GPU NVIDIA P100, 16 GB RAM	$SSIM=0.949\pm 0.020,$ $PSNM=30.678\pm 1.624$
Shahini et al. (2025) [78]	Synthetic image generation	ViT-P2P	-	-
Song et al. (2023) [67]	Stain normalization	CycleGAN	Nvidia RTX 3090 GPU, 24 GB of GPU memory, Intel Xeon, 64 GB of RAM	-
Taieb and Hamarneh (2017) [75]	Stain Normalization	GAN	NVIDIA Titan X GPU	$MITOSI\mathrm{S-}IS=0.700,$ $COLO\mathrm{N-}IS=0.950,$ $OVAR\mathrm{Y-}IS=0.260$
Vasiljevic et al. (2021) [94]	Data augmentation	CycleGAN, StarGAN	-	$CycleGAN:FID=5.359,$ $StarGAN:FID=11.572$
Vasiljevic et al. (2023) [72]	Data augmentation and Virtual staining	CycleGAN HistoStarGAN	-	$FID=56.00,SSIM=0.81$
Wang et al. (2021) [82]	Stain normalization	ACCP-GAN	GPU NVIDIA RTX, CPU Intel Xeon Silver 4110, 128 GB RAM	$FSIM=0.9657,M\mathrm{S-}SSIM=0.9694,$ $PSNR=27.5058,VIF=0.9031,$ $FID=27.1231$
Wang et al. (2022) [80]	Data augmentation	GAN	NVIDIA GTX 1070, 16 GB RAM	-
Ye et al. (2025) [60]	Stain normalization	SASSL	-	-
Yoon et al. (2024) [74]	Virtual staining	CUT, E-CUT CycleGAN E-CycleGAN	GPU Nvidia RTX 3090 AMD EPYC 7302 CPU, 346 GB of RAM	FID-CycleGAN: 67.76, E-CycleGAN: 61.91, CUT: 54.87, E-CUT: 50.91 KID-CycleGAN: 2.290, E-CycleGAN: 1.631, CUT: 0.642, E-CUT: 0.245
Zhang et al. (2021) [64]	Data augmentation	MASG-GAN	GPU NVIDIA RTX 2080 Ti, CPU Intel (TM) i9-10850 K, 32 GB RAM	-
Zhang et al. (2022) [58]	Color normalization and Virtual staining	CSTN	NVIDIA RTX 3090 GPU	$FID=101.37,$ $KID=6.129,AHR=1.98$

Beyond CycleGAN, more advanced architectures have demonstrated improvements in both chromatic coherence and structural fidelity. Du et al. [70] employed Pix2Pix GAN (P2P-GAN) and DSTGAN across datasets such as TUPAC, MITOS, and MICCAI, attaining high performance with SSIM > 0.97 and PSNR values exceeding 34. Similarly, Restore-GAN introduced by Rong et al. (2023) [89] reached SSIM = 0.949 and PSNR = 30.678, validating the efficacy of more customized GAN variants for stain normalization tasks.

Complementary approaches such as WGAN-GP (Hu et al. [59]) and cGAN (Mahmood et al. [88], Razavi et al. [71]) have further diversified the GAN landscape, offering benefits in both stain transfer and data augmentation. For example, Hu et al. leveraged WGAN-GP on high-memory V100 GPUs for combined stain normalization and augmentation, while Mahmood et al. used cGANs with Tesla V100 to handle large-scale stain harmonization pipelines.

Other innovative models such as GramGAN (Guan, Li and Zhang [56]) and ACCP-GAN (Wang et al. [82]) have been proposed to improve visual quality metrics like FSIM, MS-SSIM, and VIF. Meanwhile, hybrid frameworks like RRAGAN and CycleGAN as explored by Kablan [76], and self-supervised methods such as SASSL [60], underscore the evolution of stain normalization into a field marked by methodological diversity and increasing performance benchmarks.

Foundational contributions by Lafarge et al. [77], Bel et al. [93], and Bouteldja et al. [69] further demonstrate how stain normalization is often coupled with data augmentation to boost training generalizability.

5.4.3. Virtual Staining

Virtual staining is an innovative and rapidly advancing task within computational pathology. GANs are employed to simulate histological staining on unstained tissue images, such as those acquired via fluorescence or autofluorescence microscopy [98]. This approach aims to replace traditional, labor-intensive, and costly staining protocols with automated computational transformations, enabling faster and more cost-effective tissue analysis [99].

A range of GAN architectures has been proposed for this task, each showing varying degrees of success. For example, Yoon et al. [74] used a combination of CycleGAN and E-CUT architectures, reporting a FID of 50.91 and KID of 0.2451, demonstrating a moderately effective simulation of staining. Similarly, Guan, Li and Zhang (2024) introduced GramGAN, obtaining FID = 52.086, KID = 5.815, and CSS = 0.714, suggesting a reasonable approximation of stain distribution with somewhat higher KID values.

A noteworthy contribution is from Zhang et al. [58], who proposed the CSTN model for both stain normalization and virtual staining, but reported a high FID of 101.37, indicating room for improvement in image fidelity and realism. Vasiljevic et al. [72] introduced HistoStarGAN, a domain-adaptive method that generalizes across multiple stain types. Their model reached FID = 56.0 and SSIM = 0.81 across seven domains, indicating solid performance and robustness, although with slightly reduced structural similarity compared to state-of-the-art models.

On the other hand, Lahiani et al. [85] explored the application of CycleGAN for virtual staining on an NVIDIA Tesla V100 GPU. They reported CWSSIM = 0.880 ± 0.170 and SNR = 1.132, indicating strong structural consistency and acceptable signal quality. In a more advanced approach, Azam et al. [84] developed a method combining Pix2Pix and CUT, achieving FID = 11.16 on colorectal datasets and FID = 38.52 on breast cancer data.

5.4.4. Synthetic Image Generation

Synthetic image generation is a strategy that is particularly valuable in pathology, where obtaining annotated images is resource-intensive [100]. Building on this concept, He et al. [61] employed a Conditional GAN (cGAN) model for both image generation and transformation tasks using semantic label maps. Implemented on an NVIDIA TITAN RTX GPU with 24 GB VRAM, their approach achieved strong quantitative results: Structural Similarity Index (SSIM) of 0.9066 ± 0.065 and Mean Squared Error (MSE) of 0.0076 ± 0.003. Shahini et al. [78] introduced a novel Vision Transformer-based model, ViT-P2P, specifically for semantic-to-image generation. The study evaluated image realism using alternative perceptual quality metrics: Natural Image Quality Evaluator (NIQE) = 8.94 ± 1.62, Multi-scale Aesthetic Image Quality Assessment (MANIQA) = 0.51 ± 0.03, and Perceptual Aesthetics Quality Predictor (PAQ2PIQ) = 70.22 ± 1.76.

Deshpande et al. [66] proposed a GAN-based model trained on an NVIDIA GTX TITAN X GPU with 12 GB RAM. Their model, designed for synthetic data augmentation, reported a Fréchet Inception Distance (FID) of 142.84. In Falahkheirkhah et al. [95] developed a high-definition image-to-image translation model (Pix2Pix-GAN) for synthetic histological image generation. Running on an NVIDIA 2080 GPU, the model achieved a mean perceptual quality score of 4.9379 with a standard deviation of 0.2692.

5.4.5. Color Normalization

Color normalization standardizes image appearance and significantly enhances downstream analytical outcomes, as shown in Fan et al. [65]. In the paper, CycleGAN was applied for color normalization and reported performance metrics directly linked to segmentation accuracy, PQ = 0.5527 and AJI = 0.5797.

In a broader application, Naglah et al. [92] compared cGAN and CycleGAN for color normalization, domain transformation, and data augmentation. The results showed that cGAN achieved Mutual Information (MI) = 0.26 ± 0.07, Normalized MI (NMI) = 0.10 ± 0.03, Hierarchical Clustering (HC) = 0.75 ± 0.13, and Bhattacharyya Distance (BCD) = 0.31 ± 0.06. In contrast, CycleGAN yielded MI = 0.24 ± 0.08, NMI = 0.09 ± 0.03, HC = 0.61 ± 0.26, and BCD = 0.35 ± 0.15, suggesting cGAN slightly outperformed CycleGAN in this context for preserving domain-specific features.

Moving toward more advanced models, Zhang et al. [58] developed a Cross-Stain Transfer Network (CSTN) for both color normalization and virtual staining. Implemented on an NVIDIA RTX 3090 GPU, CSTN produced an FID of 101.37, Kernel Inception Distance (KID) of 6.129, and Average Hue Ratio (AHR) of 1.98, reflecting moderate visual alignment between synthetic and reference staining styles.

Likewise, Rong et al. [89] introduced Restore-GAN, which achieved outstanding quantitative performance with SSIM = 0.949 ± 0.020 and PSNR = 30.678 ± 1.624, confirming its effectiveness in preserving morphological structure while correcting stain variability.

Further contributing to this field, Mahapatra and Maji [90] proposed a combined LSGAN and TredMiL approach, obtaining NMI = 0.800, BCC = 0.750, and WsCC = 0.590, reinforcing the feasibility of integrating adversarial learning with structural constraints to ensure consistent chromatic representation.

5.4.6. Domain Transformation and Reconstruction of High-Resolution Image

Domain adaptation and high-resolution image reconstruction are essential for improving cross-domain generalization, as they align heterogeneous data distributions and recover fine structural details. Kapil et al. [83] explored domain adaptation using the CycleGAN and DASGAN architectures. Supported by powerful hardware, including NVIDIA V100 (32 GB) and K80 GPUs, their approach demonstrated the feasibility of unsupervised domain adaptation for harmonizing histopathological data from diverse sources. However, specific quantitative metrics were not reported, limiting the objective assessment of the model’s performance.

Complementing this, Naglah et al. [92] performed a comprehensive comparison between cGAN and CycleGAN for domain transformation alongside color normalization and data augmentation. Their results indicated that cGAN marginally outperformed CycleGAN concerning mutual information (MI = 0.26 ± 0.07 vs. 0.24 ± 0.08), normalized mutual information (NMI = 0.10 ± 0.03 vs. 0.09 ± 0.03), hierarchical clustering (HC = 0.75 ± 0.13 vs. 0.61 ± 0.26), and Bhattacharyya distance (BCD = 0.31 ± 0.06 vs. 0.35 ± 0.15).

In high-resolution image reconstruction, Juhong et al. [81] proposed a super-resolution approach combining SRGAN with a ResNeXt backbone to enhance histopathological image quality. Executed on a system with NVIDIA RTX 2060 GPU and Intel Core i7-9750H CPU, their model achieved a Peak Signal-to-Noise Ratio (PSNR) of 32.13, Structural Similarity Index Measure (SSIM) of 0.93, and Mean Squared Error (MSE) of 2.75.

5.5. Segmentation Task in Histological Image Analysis

Segmentation in histopathological images is a crucial step for both quantitative and qualitative analysis of biological tissues, extensively applied in detecting cell nuclei, tissue structures, and morphometric analysis [101]. Numerous studies have proposed deep learning architectures and approaches to perform these tasks, utilizing different models and learning strategies, including supervised, unsupervised, semi-supervised, and self-supervised learning. In the following subsections, GAN-based segmentation approaches are systematically categorized according to the specific segmentation tasks, such as nuclei, cells, and tissues. A comparative analysis of segmentation tasks in histological image analysis is summarized in

Table 5. Comparative analysis of segmentation tasks in histological image analysis.

Paper	Task	Model	Learning	Data Available *	Performance
Azam et al. (2024) [84]	Nuclei	U-Net	Supervised	http://github.com/abubakrazam/Pix2Pix_TIL_H-E.git	$Pearsoncorrelation=0.95,$ $Acc=0.982$
Bouteldja et al. (2022) [69]	Cell and Tissue	U-Net	Unsupervised	https://github.com/NBouteldja/KidneyStainAugmentation	$IDSC=94.6$
De Bel et al. (2021) [93]	Tissue	U-Net	Unsupervised	-	$DSC=0.850$
Deshpande et al. (2022) [66]	Cell and Tissue	U-Net	Supervised	http://warwick.ac.uk/TIALab/SAFRON	$DSC=0.97\pm 0.03$
Deshpande et al. (2024) [62]	Cell	HoVer-Net	Supervised	https://github.com/Srijay/SynCLay-Framework	-
Du et al. (2025) [70]	Tissue and Nuclei	U-net	Semi-supervised	-	MICCAI: $DSC=0.860\pm 0.009,$ $IoU=0.760\pm 0.012,$ $PA=0.864\pm 0.008$
Falahkheirkhah et al. (2023) [95]	Tissue	U-Net, ResNet	Supervised	https://github.com/kiakh93/Synthesizing-histological-images	Real images + Synthesized—$PA=0.9267,mPA=0.9522,$ $IoU=0.7265$
Fan et al. (2024) [65]	Nuclei	Mask R-CNN	Unsupervised	-	Kumar: $DSC=0.7930\pm 0.0446,AJI=0.5797\pm 0.0740,$ $SQ=0.7373\pm 0.0307,PQ=0.5527$; CPM17: $DSC=0.8237\pm 0.0471,AJI=0.6090\pm 0.0867,$ $SQ=0.7624\pm 0.0337,PQ=0.6274\pm 0.0867$
Gadermayr (2019) [97]	Cells and Nuclei	U-Net	Supervised and Unsupervised	-	Supervised—$F1-score=0.900,Precision=0.890,Recall=0.920$
Guan, Li and Zhang (2024) [56]	Cell	Mask R-CNN	Unsupervised	https://github.com/xianchaoguan/GramGAN	$mAP@[0.50:0.95]=0.617$
He et al. (2022) [61]	Tessue	U-Net+	Supervised	-	$DSC=0.8751\pm 0.079$
Hou et al. (2020) [87]	Nuclei	Mask R-CNN	-	-	$DSC=0.797$
Hossain et al. (2023) [91]	Nuclei	U-Net	Supervised	-	$ACC:1.0,DSC:0.999,JI:0.999$
Hossain et al. (2024) [73]	Nuclei	Mask R-CNN, CNN, U-Net	-	-	Real images: $Acc=0.979,DSC=0.875,AJI=0.791$; Synthetic image: $Acc=1.0,DSC=0.999,AJI=0.999$.
Hu et al. (2018) [59]	Cell	-	Unsupervised	https://github.com/bohu615/nu_gan	$IoU=0.560,F\mathrm{1-}score=0.700$
Juhong et al. (2022) [81]	Nuclei	Interception U-Net	Supervised	-	$IoU=0.869,DSC=0.893$
Kablan (2023) [76]	Cell and Tissue	U-Net	Unsupervised	https://github.com/junyanz/pytorch-CycleGAN-andpix2pix	$DSC=0.881,Acc=0.887,Pre=0.910,Recall=0.867$
Kapil et al. (2021) [83]	Cell and Tissue	SegNet	Supervised	-	Two class-$F\mathrm{1-}score=0.916$, Three class $F\mathrm{1-}score=0.899$
Kweon et al. (2022) [86]	Cell and Tissue	U-Net	-	-	-
Lafarge et al. (2019) [77]	Nuclei	CNN	Supervised	-	$F\mathrm{1-}Score=0.851\pm 0.011$
Lahiani et al. (2020) [85]	Tissue	ResNet	Unsupervised	-	Tumor: $F\mathrm{1-}Score=0.81$, No Tumor: $F\mathrm{1-}Score=0.940$
Li, Hu and Kak (2023) [63]	Nuclei	CNN	Unsupervised	https://github.com/lifangda01/GSAN-Demo	$PQ=0.4885,AP=0.5035,AJI=0.4693$
Liu, Wagner and Peng (2022) [57]	Nuclei	Mask R-CNN	Supervised	http://www.kaggle.com/c/data-science-bowl-2018/submit	$IoU=0.570$
Lou Wei et al. (2022) [79]	Nuclei	Mask R-CNN	Semi-supervised and Supervised	-	$AJI=0.5374,DSC=0.7536$
Mahapatra and Maji (2023) [90]	Nuclei	Watershed	Unsupervised	-	$DSC=0.7042$
Mahmood et al. (2019) [88]	Nuclei	U-Net	Supervised	https://github.com/mahmoodlab/NucleiSegmentation	$aHD=4.291,F\mathrm{1-}Score=0.866,AJI=0.721$
Naglah et al. (2022) [92]	Cell and Tissue	U-Net	Unsupervised	-	$Acc=0.86\pm 0.08,DSC=0.75\pm 0.09$
Purma et al. (2024) [68]	Tissue and Nuclei	U-Net	Supervised and Self-supervised	https://github.com/suhas-srinath/GenSelfDiff-HIS	Glas: $AJI=0.8470,IoU=0.8470,HD=6.3517,F\mathrm{1-}score=0.9026$; MoNuSeg—$AJI=0.6545,IoU=0.6545,HD=7.3261,F\mathrm{1-}score=0.7895$; HC Cancer—$AJI=0.7913,IoU=0.8105,HD=4.6855,F\mathrm{1-}score=08413$
Razavi et al. (2022) [71]	Cell and Nuclei	-	Supervised	-	Mitosis—$DSC=721$, Nuclei—$DSC=0.784$, $F\mathrm{1-}Score=0.854$
Rong et al. (2023) [89]	Nuclei	Mask R-CNN	Unsupervised	http://pytorch.org/get-started/previous-versions/#v1101	$Acc=0.8204$
Shahini et al. (2025) [78]	Nuclei	-	Usupervised	-	$DSC=84.86$
Song et al. (2023) [67]	Nuclei	Mask R-CNN	Self-supervied	https://github.com/zhiyuns/UNITPathSSL	$Kumar:F\mathrm{1-}Score=0.9560,DSC=0.853,AJI=0.657,PQ=0.625$
Taieb and Hamarneh (2017) [75]	Nuclei	AlexNet	Unsupervised	-	MITOSIS: $ACC=0.900\pm 0.010$; COLON: $ACC=0.820\pm 0.090$; OVARY: $ACC=0.617\pm 0.130$.
Vasiljevic et al. (2021) [94]	Cell and Tissue	U-Net	Unsupervised	-	$Precision=0.838,Recall=0.919,F\mathrm{1-}score=0.875$
Vasiljevic et al. (2023) [72]	Cell and Tissue	U-Net	Supervised	-	$Precision=0.864,Recall=0.877,F\mathrm{1-}score=0.870$
Wang et al. (2021) [82]	Tissue	U-Net	Unsupervised	-	N5: $Acc=0.9995,Rcall=0.9996,Precision=0.9994,$ $DSC=0.9927,IoU=0.9990$
Wang et al. (2022) [80]	Nuclei	RCSAU-Net	Supervised	https://github.com/antifen/Nuclei-Segmentation	MoNuSeg: $ACC=0.919,F\mathrm{1-}score=0.867,DSC=0.820,$ $AJI=0.619,HD=4.334,Recall=0.893$; PanNuke: $Precision=0.9203,Recall=0.9101,F\mathrm{1-}score=0.8661,$ $DSC=0.8482,AJI=0.6550,HD=4.3768$.
Ye et al. (2025) [60]	Tissue	PRANet	Self-supervised	http://github.com/YeahHighly/SASSL_PR_2024	$PA=0.948,DSC=0.946,IoU=0.898$
Yoon et al. (2024) [74]	Area, count and distance cells	U-Net	-	https://github.com/YoonChiHo/DL-based-frameworkfor-automated-HIA-of-label-free-PAH-images	-
Zhang et al. (2021) [64]	Nuclei	U2-Net	Supervised	-	TNBC: $PA=0.9152,Recall=0.8539,specificity=0.8620,$ $F\mathrm{1-}score=0.8556,IoU=0.7908,DSC=0.9301,PRI=0.9889$.
Zhang et al. (2022) [58]	Nuclei	DSCN	Supervised and Unsupervised	http://github.com/wangpengyu0829	$PA=0.8735,DSC=0.7256,AJI=0.5997,MAE=0.0882$

* Accessed on 19 June 2025.

, emphasizing the used model, diversity in learning strategies and performance. Moreover, to promote transparency and reproducibility, we include information on the availability of code that was provided in the original studies. For each reviewed article, we identified whether the authors had shared their resources through public platforms. This information is summarized in a dedicated column (Data available) in Table 5, enabling readers to quickly assess the reproducibility potential of each work.

5.5.1. Nuclei Segmentation

Nuclei segmentation is the most common task among the reviewed studies, as evidenced by the predominance of models specifically applied to this cellular structure. Hou et al. [87] developed an analysis pipeline that segments nuclei in whole-slide tissue images from multiple cancer types, utilizing a multi-level quality control process (WSI level and image patch level) to evaluate the quality of segmentation results. In Hossain et al. [73], an application based on deep learning networks has been developed. The system performs image preprocessing and generates synthetic images to address the scarcity of pathological data through a CycleGAN network. The generated images are fed into CNN, Mask R-CNN, and modified U-Net networks to segment nuclear regions and separate overlapping kidney nuclei.

The study by Zhang et al. [40] addresses the challenging task of nuclei segmentation in histopathological images by proposing a Cross-Boosted Multi-Target Domain Adaptation (CB-MTDA) framework. This method tackles the significant issue of domain variability, which often obstructs the generalization of segmentation models across different staining protocols and imaging modalities.

Unsupervised domain adaptation (UDA) methods have been proposed to mitigate the distributional gap between different imaging modalities for unsupervised nuclei segmentation in histopathology images (Fan et al.) [65]. Moreover, a dual-branch nucleus shape and structure-preserving module has been proposed to prevent nucleus over-generation and deformation in the synthesized images of liver cancer, kidney cancer, and colon cancer. Li, Hu and Kak [63] introduced a Generative Stain Augmentation Network (G-SAN) for stain augmentation of H&E-stained histological images. G-SAN can augment an input cell image with selected and realistic stains by disentangling the morphological and stain-related representations. Through the downstream tasks of patch classification and nucleus segmentation, it was quantitatively demonstrated that the quality of G-SAN-augmented images surpasses that of images produced by existing stain augmentation approaches.

GANs are also widely employed as supportive tools in histopathological image analysis. For instance, BenTaieb and Hamarneh [75] leveraged GANs to perform stain normalization via adversarial stain transfer. This pre-processing step mitigates variability in staining protocols and improves the consistency of nuclei segmentation algorithms. Similarly, Mahmood et al. [88] introduced a deep adversarial training framework specifically designed for multi-organ nuclei segmentation. Their architecture employs a segmentation network guided by a discriminator that evaluates the morphological realism of the segmented masks. This adversarial learning paradigm improved generalization across different tissue types and staining variations.

Song et al. [67] proposed a self-supervised pretraining method using CycleGANs to translate histological images between unpaired domains. The framework aims to capture nuclear morphology and information distribution by translating histopathological images and pseudo-masked images. While the approaches above rely on supervised learning, other studies explore unsupervised and semi-supervised paradigms. Hu et al. [59] demonstrated the use of unified GANs with a novel loss formulation to perform robust cell-level visual representation learning in an unsupervised setting. The model is not only label-free and easily trained but also capable of unsupervised segmentation of bone marrow cellular components. Lou Wei et al. [79] explored full-nuclei segmentation by choosing only a few image patches to annotate, augmenting the training set of selected samples, and achieving nuclei segmentation in a semi-supervised manner.

Zhang et al. [64] presented MASG-GAN, a novel architecture that combines multi-view attention and superpixel-guided mechanisms. This framework not only performs nuclei segmentation but also classifies tissue types, leveraging shared features for both tasks. The model capitalizes on the semantic interdependencies between segmentation and classification, enhancing overall performance in complex tissue environments. Building upon this, Wang et al. [80] proposed a multi-task GAN architecture that simultaneously learns nuclei segmentation and image reconstruction. Their model enhances spatial coherence and focuses on the most informative regions by integrating dual attention mechanisms and recurrent convolutional modules.

Data augmentation and synthetic data generation represent other important applications of GANs. Hossain et al. [91] proposed generating annotated synthetic data from small labeled samples. Synthetic backgrounds are created using base image patches, with GANs preserving texture characteristics. Nuclear shapes are extracted from original images using ground truth masks and then transformed with image processing techniques. These shapes are placed on filtered backgrounds, maintaining nuclear morphology and distribution. The synthetic image patches, along with their corresponding masks, are used to train a modified U-net for nuclear region segmentation. Lafarge et al. [77] conducted a comparative analysis involving staining normalization and data augmentation across two distinct tasks: generalization to images obtained in invisible pathology laboratories for mitosis detection and generalization to invisible organs for nuclei segmentation.

GANs are also widely employed as supportive tools in histopathological image analysis. Rong et al. [89] took this further by developing Restore-GAN, which improves image quality by restoring blurred regions, enhancing low resolution, and normalizing colors. When the Mask R-CNN segmentation model was applied to these restored images, its performance improved significantly. Similarly, Juhong et al. [81] demonstrated a custom CNN for both super-resolution image enhancement from low-resolution images and characterization of both cells and nuclei from hematoxylin and eosin (H&E) stained breast cancer histopathological images by using a combination of generator and discriminator networks, so-called super-resolution generative adversarial network-based on aggregated residual transformation (SRGAN-ResNeXt), to facilitate cancer diagnosis in low-resource settings.

Shahini et al. [78] proposed SENSE (SEmantic Nuclear Synthesis Emulator), a novel framework for synthesizing realistic histological images with precise control over cellular distributions. The approach introduces three main innovations: a statistical modeling system that captures class-specific nuclear features from specialized annotations, a hybrid ViT-Pix2Pix GAN architecture that effectively translates semantic maps into high-fidelity histological images, and a modular design that allows independent control of cellular properties, including type, count, and spatial distribution.

Mahapatra and Maji [90] presented a new deep generative model (LSGAN) to reduce the color variation found among histological images. The proposed model assumes that the latent color appearance information, extracted through a color appearance encoder, and the constraint information, extracted via a patch density encoder, are independent of one another. The research demonstrated that the performance of nuclei segmentation improves on the image patch after color normalization compared to the image patch before normalization.

Style augmentation using multi-modal microscopy data is the focus of [57]. The authors developed a technique that generates diverse training samples by simulating different imaging modalities. The concept of style augmentation is strongly connected to image-to-image translation models, such as the use of a GAN, which can simulate various staining conditions. These augmentations assist nuclei segmentation models in generalizing across datasets.

Finally, Azam et al. [84] evaluated the impact of deriving cell labels on the performance of H&E-based models applied to lung cancer tissue. Specifically, two virtual staining models were compared based on the Pix2Pix (P2P-GAN) architecture: one trained with cell labels obtained from the same tissue section and another trained using cell labels from an adjacent tissue section. The H&E image was then converted into a nuclear mask through nuclear segmentation using the H&E-trained StarDist model. The cells identified in the mIHC image were matched to the nearest nuclei in the generated H&E image.

5.5.2. Cell Segmentation

Cell segmentation is essential for quantitatively analyzing cell populations and tumor characteristics in histopathology. Studies such as Hu et al. [59] take an essential early step in using GANs for segmenting cells in histopathology images. Their approach is particularly noteworthy for being unsupervised; by leveraging GANs, they generate realistic-looking images that assist the model in learning cell-level features without relying on extensive manual annotations. This represents a significant contribution, as annotating histological data can be time-consuming and costly. However, while their work concentrates on learning visual representations and extracting cell-level features, the segmentation accuracy is limited since the model primarily aims to capture cellular structures rather than produce precise segmentation masks. Similarly, Guan et al. [56] introduced a more advanced unsupervised framework that emphasizes stain style transfer using a multi-domain progressive GAN architecture. While their primary goal is stain normalization and harmonization across datasets, they also implicitly address segmentation challenges by enhancing the consistency in the visual appearance of histological images, which is crucial for robust cell segmentation.

On the other hand, supervised approaches, such as the one by Deshpande et al. [62], introduced SynCLay. This novel interactive synthesis framework enables users to create synthetic histology images from customized cellular layouts. Although their focus is on image generation rather than direct segmentation, the ability to generate realistic cellular structures proves highly valuable for training segmentation models. By simulating diverse and complex cellular arrangements, SynCLay generates augmented datasets that help segmentation models learn to identify and delineate cells in challenging situations, such as overlapping or morphologically varied nuclei.

5.5.3. Tissue Segmentation

The use of GANs for segmenting and processing histological tissue images has encompassed diverse approaches, each addressing unique challenges in digital pathology. Wang et al. [82] introduce a novel end-to-end framework named automatic consecutive context perceived transformer GAN (ACCP-GAN) for fully automatic serial sectioning image blind inpainting. The first stage network (auto-detection module) is designed to detect and repair broken areas roughly. It then guides the second stage network (refined inpainting module) to generate expected patches precisely; therefore, the segmentation part integrates with the restoring part. Based on a self-attention mechanism, the transformer module (SPTransformer) is introduced to enable the refined inpainting module to focus on features from neighboring images, assisting in correcting inpainting results.

De Bel et al. [93] propose a Residual CycleGAN for domain adaptation across different histology stains. This work is particularly valuable for segmentation tasks as it tackles the common issue of stain variability, ensuring that segmentation models trained on one stain generalize more effectively across datasets. By transforming tissue images between domains while preserving underlying structures, their method enables robust segmentation across diverse staining protocols.

Ye et al. [60] introduced a novel stain-adaptive self-supervised (SASSL) learning method framework for histopathological image analysis. The method focuses on harmonizing stain variations, which, like de Bel et al. [93], indirectly strengthens segmentation pipelines by improving data consistency without requiring manual labels.

Lahiani et al. [85] take a creative angle with a perceptual embedding consistency-driven GAN for generating seamless virtual whole-slide images (WSIs). This approach enables the synthesis of large-scale, realistic tissue images while maintaining fine structural details. Although their focus is image synthesis, the ability to generate high-fidelity tissue representations provides valuable data augmentation for segmentation tasks, particularly in rare or complex tissue structures.

Supervised learning is key for segmentation in histology, and several studies use GANs to enhance data quality. He et al. [61] present a UNet+/seg-cGAN model for translating label-free images into histology-like images, facilitating visual reviews by pathologists. Meanwhile, Falahkheirkhah et al. [95] explore the generation of deepfake histological images using a Pix2Pix-GAN model. The work highlights the potential of synthetic data to augment and diversify training datasets for segmentation models in prostate cancer histological images using simple semantic labeling.

5.5.4. Multiclass Segmentation: Tissues + Nuclei or Cells

Some studies have sought to integrate the segmentation of multiple structures simultaneously, aiming for a more comprehensive understanding of the tissue microenvironment. Kweon et al. [86] propose a supervised GAN model with an integrated segmentation module for generating oligodendroglioma images, effectively incorporating segmentation into the synthesis pipeline. Yoon et al. [74] present a multi-task deep learning framework for virtual staining, segmentation, and classification of label-free photoacoustic images, demonstrating the potential of multimodal learning. Gadermayr et al. [97] introduce a GAN-based approach for stain-independent segmentation, facilitating generalization across domains. Purma et al. [68] leverage generative diffusion models with self-supervision for histology segmentation, highlighting progress in self-supervised learning for segmentation tasks.

Other studies focus on specific challenges: Naglah et al. [92] use cGANs for fibrosis detection and quantification. Baykal Kablan [76] proposes a region-aware GAN architecture for stain normalization, addressing key issues for segmentation consistency. Vasiljevic et al. [72] introduce HistoStarGAN, a unified framework for stain normalization, stain transfer, and stain-invariant segmentation in renal histopathology, while Vasiljevic et al. [94] focus on domain generalization through unsupervised domain augmentation with CycleGAN.

Du et al. [70] present an intensely supervised two-stage GAN for stain normalization, while Bouteldja et al. [69] use CycleGAN-based augmentation to tackle stain variability. Razavi et al. [71] developed MiNuGAN, a conditional GAN model for simultaneous mitosis and nuclei segmentation on multi-center breast H&E images, demonstrating GANs’ versatility for specific segmentation tasks. Deshpande et al. [66] introduce SAFRON, a synthesis framework for colorectal cancer histology images, which also supports segmentation models. Lastly, Kapil et al. [83] apply domain adaptation techniques for automated tumor cell scoring and survival analysis on PD-L1 stained tissue images, linking segmentation with clinical outcome analysis.

The field of histopathological image segmentation is advancing towards increasingly generalizable, efficient, and robust solutions, with potential direct impact on clinical applications and biomedical research.

5.6. Overview of Evaluation Metrics

Table 6. Summary of common evaluation metrics of GAN and segmentation model.

GAN		Segmentation
Name	Paper	Name	Paper
Fréchet Inception Score (FID)	[56,58,62,66,68,74,82,86,94]	Accuracy (Acc)	[73,75,76,80,82,84,89,91,92]
Structural Similarity Index (SSIM)	[61,62,70,73,76,81,82,89,91]	Dice Similarity Coefficient (DSC)	[58,60,61,64,65,66,67,69,70,71,76,78,79,80,81,82,87,90,91,92,93]
Peak Signal-to-Noise Ratio (PSNR)	[70,73,76,81,82,89,91]	F1-Score	[59,64,67,68,71,72,77,80,83,85,88,90,94,97]
Meane Square Error (MSE)	[61,73,76,81,91]	Intersection Over Union (IOU)	[57,59,60,64,68,70,81,82,95]
Average Human Rank (AHR)	[58]	Aggregated Jaccard Index (AJI)	[58,63,65,68,73,79,80,88]
Pearson correlation coefficient (PCC)	[70]	Jaccard Index (JI)	[91]
Root Meane Square Error (RMSE)	[76]	Pixel Accuracy (PA)	[58,60,64,70,95]
Kernel Inception Distance (KID)	[56,58,74]	Precision (Pre)	[63,72,73,76,82,94,97]
Interception Score (IS)	[75,86]	Recall (Rec)/Sensitivity (Sen)	[64,72,76,82,94,97]
Contrast-Structure Simirarity (CSS)	[56]	Average Pompeiu–Hausdorff distance (aHD)	[88]
Normalized Mutual Information (NMI)	[90,92]	Hausdorff Distance (HD)	[68,80]
Complex Wavelet Structural Similarity (CWSSIM)	[85]	Mean Absolute Error (MAE)	[58]
Bhattacharyya Distance (BCD)	[92]	Panoptic Quality (PQ)	[63,65]
Histogram Correlation (HC)	[92]	Probabilistic Rand Index (PRI)	[64]
Feature Similarity Index (FSIM)	[76,82]	Mean Average Precision (mAP)	[56,95]
Multi-scale Structural Similarity Index (MSSSIM)	[76,82]	Pearson Correlation Coefficient (PCC)	[84]
Erreur Relative Globale Adimensionnelle de Synthese (ERGAS)	[76]	Score Segmentation (SQ)	[65]
Universal Quality Index (UQI)	[76]	Average Precision (AP)	[63]
Erreur Relative Globale Adimension (Misep)	[76]
Erreur Relative Globale Adimension (Miss)	[76]
Name	Paper	Name	Paper
Average Spectral Error (RASE)	[76]
Between-image Color Constancy (BiCC)	[90]
Within-set Color Constancy (WsCC)	[90]
Measuring Mutual Information (MMI)	[92]
Visual Information Fidelity (VIF)	[82]
Standard Deviation (SD)	[95]

provides a comprehensive overview of the most commonly used evaluation metrics for GANs and segmentation models, particularly in the context of histopathological image analysis. What immediately stands out is that these metrics are not applied generically; they are directly aligned with the distinct goals of each model type. For GANs, the focus lies in the visual fidelity of generated images, aiming to make them as realistic as possible. In contrast, segmentation models are evaluated based on their precision in identifying and delineating specific structures, such as cell nuclei, which are essential for accurate pathological assessment.

Among the metrics used for evaluating GANs, the Fréchet Inception Distance (FID) emerges as one of the most prevalent, cited in numerous studies [56,58,62,66,68,74,82,86,94]. Another widely adopted metric is the Structural Similarity Index (SSIM), found in works such as [73,76,81], which assesses structural fidelity, an important factor for ensuring that fine-grained histological details are preserved.

Other metrics, such as Peak Signal-to-Noise Ratio (PSNR) and Mean Squared Error (MSE), continue to be used due to their simplicity, though they are less effective in capturing perceptual quality. More specialized metrics, including Between-image Color Constancy (BiCC) and Within-set Color Constancy (WsCC), appear in studies like [76,90] and address color consistency, a crucial consideration in histological images, where subtle color differences can significantly impact interpretation.

On the segmentation side, the Dice Similarity Coefficient (DSC) stands out as the leading metric. It is referenced in over twenty papers [58,65,67,81,87], underscoring its importance for quantifying overlap between predicted and annotated segmentations. This metric is particularly vital in nucleus segmentation tasks, where even minor pixel-level inaccuracies can affect diagnostic outcomes. Complementary metrics such as the F1-score, precision, and recall/sensitivity are also common [82,90,97], offering a deeper understanding of a model’s ability to handle false positives and false negatives, critical aspects in clinical applications.

The Intersection over Union (IoU), used in works like [57,68,81], provides a strong measure of segmentation accuracy by comparing the predicted and ground-truth masks. Metrics such as Pixel Accuracy (PA) and Jaccard Index (JI) are also employed to offer broader performance insights. Additionally, spatial metrics like the Hausdorff Distance (HD), which calculates the largest distance between corresponding boundary points, are important when the shape and positioning of segmented nuclei have clinical implications.

5.7. Study Summary

Table 7. Summary of publications for GAN in histological image segmentation.

Paper	Model GAN	Model Segmentation	Datasets	Metrics GAN	Metrics Segmentation
Azam et al. (2024) [84]	Pix2Pix-GAN CUT	U-Net	Onco-SG	FID	Pearson correlation, Acc
Bouteldja et al. (2022) [69]	CycleGAN	U-Net	RWTH Aachen VALIGA	-	IDSC
De Bel et al. (2021) [93]	CycleGAN	U-Net	-	-	DSC
Deshpande et al. (2022) [66]	GAN	U-Net	-	FID	DSC
Deshpande et al. (2024) [62]	SynCLay	HoVer-Net	Crag Digestpath	FID, SSIM	-
Du et al. (2025) [70]	P2P-GAN, DSTGAN	U-Net	ICIAR-BACH MICCAI’16 GlaS MITOS-ATYPIA 14 TUPAC16	SSIM, PCC, PSNR	DSC, IoU, PA
Falahkheirkhah et al. (2023) [95]	Pix2Pix-GAN	U-Net, ResNet	-	Mean, SD	PA, mPA, IoU
Fan et al. (2024) [65]	CycleGAN	Mask R-CNN	CPM17 Kumar	-	DSC, AJI, SQ, PQ
Gadermayr et al. (2019) [97]	CycleGAN	U-Net	-	-	F1-Score, Precision, Recall
Guan, Li and Zhang (2024) [56]	GramGAN	Mask R-CNN	ANHIR Glomeruli Lung dataset	CSS, FID, KID	mAP
He et al. (2022) [61]	cGAN	U-Net+	CARS	SSIM, MSE	DSC
Hou et al. (2020) [87]	GAN	Mask R-CNN	TCGA	-	DSC
Hossain et al. (2023) [91]	CycleGAN	U-Net	TCGA	SSIM, MSE, PSNR	Acc, DSC, JI
Hossain et al. (2024) [73]	CycleGAN	Mask R-CNN, CNN, U-Net	KIRC TCGA	SSIM, MSE, PSNR	Acc, DSC, AJI
Hu et al. (2018) [59]	WGAN-GP	-	BMCD-FGCD Dataset A, B, C and D	-	IoU, F1-Score
Juhong et al. (2022) [81]	SRGAN- ResNeXt	Interception U-Net	MUC1 TCGA	SSNR, SSIM, MSE	IoU, DSC
Kablan (2023) [76]	CycleGAN RRAGAN	U-Net	MICCAI’16 GlaS MITOS-ATYPIA 14	FSIM, PSNR, SSIM, MSE, MSSSIM, RMSE, ERGAS, UQI, RASE	DSC, Acc, Precision, Recall
Kapil et al. (2021) [83]	CycleGAN DASGAN	SegNet	NSCLC	-	F1-Score
Kweon et al. (2022) [86]	PGGAN	U-Net	TCGA	FID, IS	-
Lafarge et al. (2019) [77]	DANN	CNN	MICCAI’16 GlaS TCGA TUPAC16	-	F1-Score
Lahiani et al. (2020) [85]	CycleGAN	ResNet	Roche	CWSSIM	F1-Score
Li, Hu and Kak (2023) [63]	G-SAN	CNN	CPM17 CoNSep CryoNuSeg KIRC MoNuSAC MoNuSeg TNBC	-	PQ, AP, AJI
Liu, Wagner and Peng (2022) [57]	GAN	Mask R-CNN	BBBC-data	-	IoU
Lou Wei et al. (2022) [79]	CSinGAN	Mask R-CNN	MoNuSeg TCGA TNBC	-	AJI, DSC
Mahapatra and Maji (2023) [90]	LSGAN TredMiL	Watershed	TCGA	NMI, BiCC, WsCC	DSC
Mahmood et al. (2019) [88]	cGAN	U-Net	TCGA	-	aHD, F1-Score, AJI
Naglah et al. (2022) [92]	cGAN CycleGAN	U-Net	-	MMI, NMI, HC, BCD	Acc, DSC
Purma et al. (2024) [68]	CycleGAN	U-Net	HN Cancer MICCAI’16 GlaS MoNuSeg	FID	AJI, IoU, HD, F1-score
Razavi et al. (2022) [71]	ICPR14 e 12 TUPAC16	-	cGAN MiNuGAN	-	DSC, F1-score
Rong et al. (2023) [89]	Restore-GAN	Mask R-CNN	TCGA	SSIM, PSNM	Acc
Shahini et al. (2025) [78]	ViT-P2P	-	MoNuSAC	-	DSC
Song et al. (2023) [67]	CycleGAN	Mask R-CNN	KatherKumar Lizard In-House In-House-MIL PanNuke	-	F1-Score, DSC, AJI, PQ
Taieb and Hamarneh (2017) [75]	GAN	AlexNet	MICCAI’16 GlaS	IS	Acc
Vasiljevic et al. (2021) [94]	CycleGAN, StarGAN	U-Net	-	FID	Precision, Recall, F1-score
Vasiljevic et al. (2023) [72]	CycleGAN HistoStarGAN	U-Net	KidneyArtPathology	FID, SSIM	Precision, Recall, F1-score
Wang et al. (2021) [82]	ACCP-GAN	U-Net	N7, E17, N5	FSIM, MS-SSIM, PSNR, VIF, FID	Acc, Rcall, Precision, DSC, IoU
Wang et al. (2022) [80]	GAN	RCSAU-Net	MoNuSeg PanNuke	-	Acc, F1-score, DSC, AJI, HD, Recall
Ye et al. (2025) [60]	SASSL	PRANet	CAMELYON16	-	PA, DSC, IoU
Yoon et al. (2024) [74]	CUT, E-CUT CycleGAN E-CycleGAN	U-Net	CPM17 Kumar TNBC	FID, KID	-
Zhang et al. (2021) [64]	MASG-GAN	U2-Net	CoNSep CPM17 KIRC Kumar TNBC	-	PA, Recall, specificity, F1-score, IoU, DSC, PRI
Zhang et al. (2022) [58]	CSTN	U-Net	BBBC-data CoNSep CPM17 KIRC Kumar LYON19 (IHC) MThH TNBC	FID, KID, AHR	PA, DSC, AJI7, MAE

presents a summary that compares key details of each study.

6. Discussion

This systematic review investigated the growing application of GANs in histological image segmentation, aiming to answer a set of research questions ranging from the types of tissues and datasets used to the model architectures and evaluation metrics employed. The findings underscore the versatility of GANs in addressing the inherent challenges of histopathological image analysis, such as staining variability, data scarcity, and the need for accurate segmentation of complex structures.

In addressing the research questions (RQ1 and RQ2), concerning histological tissues and datasets, the analysis reveals a wide diversity in the types of histological tissues used, reflecting the medical research focus on pathologies related to vital organ systems. Organs such as the kidney, breast, colon, lung, and prostate are frequently represented, highlighting their high clinical relevance. In addition, less commonly studied tissues—including the thymus, testis, pleura, mediastinum, and adrenal gland—have also been investigated, indicating an effort to develop more generalizable models. Datasets such as BBBC-data, PanNuke, and MoNuSeg, which contain samples from multiple organs, are particularly valuable for training models with broader generalization capabilities.

Regarding the datasets, a significant variation in size was observed, ranging from small collections of 30 to 100 images (e.g., CryoNuSeg or in-house datasets) to large-scale repositories containing tens or even hundreds of thousands of images (e.g., TCGA, KIRC, Kather). Image dimensions vary widely, from small patches (e.g., 256 × 256 pixels) to whole slide images (WSIs), which can exceed 100,000 × 100,000 pixels. Image magnification, most commonly 20× or 40×, is a critical factor that directly affects resolution and the level of detail available, thereby impacting segmentation outcomes. Most datasets provide detailed annotations, which are essential for supervised learning tasks. However, some (e.g., KIRC, TCGA, MUC1) are released without comprehensive labeling, limiting their applicability to unsupervised or semi-supervised approaches or requiring additional, resource-intensive annotation efforts. The public availability of most datasets is a significant strength, fostering scientific reproducibility and advancing AI research in pathology, although access to specific datasets remains restricted.

Regarding the research questions on the objectives of applying GANs and the architectures used (RQ3 and RQ4), the studies analyzed reveal various purposes and technical approaches, highlighting the versatility of these networks. The most prevalent application of GANs identified in the review is data augmentation, the most common use case reported in 20 studies. In this context, GAN architectures such as CycleGAN, PGGAN, cGAN, and WGAN-GP synthesize realistic histological images, enriching training datasets and enhancing the robustness and adaptability of segmentation models.

The second most frequent application is stain normalization, essential for mitigating the bias introduced by staining variability. Models like CycleGAN, Pix2Pix GAN (P2P-GAN), DSTGAN, Restore-GAN, WGAN-GP, cGAN, and GramGAN have been widely used to harmonize the visual appearance of images while preserving their underlying morphological structure. Following this, color normalization, although overlapping with stain normalization, aims explicitly to adjust color variations caused by different scanners and acquisition settings, directly influencing analytical consistency and segmentation accuracy [102,103]. Another important application is virtual staining, which simulates histological staining on unlabeled images, enabling faster and more cost-effective analyses while preserving the original tissue samples [104]. Architectures such as CycleGAN, E-CUT, GramGAN, CSTN, HistoStarGAN, Pix2Pix, and CUT have been employed for this purpose.

Synthetic image generation, which involves creating artificial histological images from semantic maps or structural sketches, is another valuable application, especially in contexts where annotation is resource-intensive [105]. cGAN and ViT-P2P are examples of models applied to this task. Lastly, domain transformation and high-resolution image reconstruction play a key role in overcoming dataset heterogeneity and improving image quality, facilitating cross-domain generalization and restoring fine structural details [106]. Notable models in these areas include CycleGAN and SRGAN-ResNeXt, respectively.

The most frequently used GAN architectures in histological image processing include CycleGAN, which emerged as the most prominent model, especially for stain normalization and data augmentation, due to its ability to perform image-to-image translation without requiring paired data. Other prominent architectures include cGAN, Pix2Pix, PGGAN, WGAN, and their variants, each tailored to address specific domain challenges. More recent and specialized models like DST-GAN, HistoStarGAN, and ViT-P2P indicate a trend toward customized GAN architectures designed for specific histopathological tasks.

In addressing the research questions regarding the objectives of segmentation and the segmentation architectures employed (RQ6 and RQ7), this review shows that the primary goal of segmentation in histopathological images is to identify and delineate specific structures—such as cell nuclei, glands, collagen fibers, or tumor regions—in order to extract quantitative information that complements qualitative analysis. The most common segmentation task is nuclear segmentation, reported in 20 studies, due to the clinical relevance of nuclear morphology in cancer diagnosis and prognosis [107]. Models such as U-Net, Mask R-CNN, and their variants are frequently used, often enhanced by GAN-based techniques for data augmentation, stain normalization, or domain adaptation.

Another relevant task is cell segmentation, essential for quantitatively analyzing cell populations and tumor features. Both supervised and unsupervised approaches are explored, often leveraging GANs for synthetic data generation or style enhancement. Tissue segmentation, which targets larger and more complex regions, addresses challenges such as staining variability and enhances model generalization. Common architectures used include U-Net, ResNet, and SegNet.

In multiclass segmentation (e.g., tissues + nuclei or cells), some studies simultaneously segment multiple structures to provide a more comprehensive understanding of the tissue microenvironment, often using U-Net-based models and integrating GANs for data generation or enhancement. Due to their effectiveness in medical image segmentation, the dominant segmentation architectures remain U-Net and Mask R-CNN, particularly under data-limited scenarios. Other architectures such as CNN, ResNet, HoVer-Net, and SegNet are also mentioned and frequently integrated with or supported by GANs.

In the research questions addressing GAN performance evaluation and segmentation quality assessment (RQ5 and RQ8), a distinct set of metrics is employed, each aligned with the specific goals of the respective model types. For GANs, the primary focus lies in visual fidelity and the statistical similarity between generated images and real data. The Fréchet Inception Distance (FID) is one of the most commonly used metrics, as it measures the distance between feature distributions extracted from real and synthetic images, indicating how convincingly realistic the generated images are. The Structural Similarity Index (SSIM) evaluates structural fidelity, which is essential for preserving fine histological details. Additional metrics such as Peak Signal-to-Noise Ratio (PSNR) and Mean Squared Error (MSE) are also employed, along with more specialized measures for color consistency, such as Between-image Color Constancy (BiCC) and Within-set Color Constancy (WsCC).

For segmentation models, the evaluation prioritizes the accuracy in identifying and delineating structures. The Dice Similarity Coefficient (DSC) is the primary metric, quantifying the overlap between predicted and annotated segmentations and is particularly critical in tasks involving nuclear segmentation. Complementary metrics such as F1-score, precision, and recall are commonly used to assess the model’s ability to handle false positives and false negatives. The Intersection over Union (IoU) offers a robust measure of segmentation precision by comparing predicted and ground truth masks. Additionally, spatial metrics like the Hausdorff Distance (HD), which calculates the most significant distance between corresponding boundary points, are essential when the shape and positioning of segmented nuclei have clinical implications. While the metrics used for GANs and segmentation focus on different aspects, there is an increasing need to consider them in a complementary fashion, particularly in hybrid approaches where synthetic images are used to improve the training of segmentation models or where segmentation maps guide image generation.

The growing interest in GAN-based approaches is primarily driven by their ability to synthesize realistic images, address data scarcity issues, and facilitate tasks such as stain normalization, data augmentation, and domain adaptation. Several studies have demonstrated the potential of GANs in improving segmentation accuracy, especially in challenging scenarios involving small datasets, class imbalance, and variations in staining protocols. These contributions have enabled progress in key segmentation tasks, including nuclei, cellular, and tissue segmentation, ultimately supporting downstream applications such as tumor detection, quantification of cellular structures, and prognostic assessments.

Current research trends reveal a strong focus on developing hybrid models that integrate GANs with other architectures, such as U-Net, attention mechanisms, and transformers, to enhance the accuracy and generalization of segmentation models. Moreover, there is an increasing interest in exploring self-supervised and weakly supervised learning strategies, aiming to reduce the dependency on costly pixel-level annotations. The integration of multi-modal data, such as combining histological images with clinical or genomic information, is also emerging as a promising avenue to support holistic analysis and improve clinical decision-making.

Besides exploring technical and methodological aspects, it is important to consider the ethical and clinical implications of using GAN-based approaches in histopathology. Creating synthetic data, while helpful for training models and improving generalization, also risks introducing artifacts that may not match real biological structures, which could lead to misinterpretations or diagnostic mistakes. Therefore, thorough validation of these models in clinical settings is essential to guarantee reliability and patient safety. Additionally, incorporating GAN-based tools into clinical workflows involves navigating complex regulatory frameworks that often lag behind technological progress. Addressing these issues requires not only technological innovation but also collaboration across disciplines with pathologists, ethicists, and regulatory agencies to ensure AI solutions meet healthcare standards and ethical responsibilities.

7. Conclusions

This review offers an original perspective by systematically categorizing the use of GANs in histological image segmentation according to their specific applications. The studies analyzed demonstrate the versatility of GANs in handling challenges such as stain variability, multi-task segmentation, and data scarcity, which are critical factors in histopathological analysis. While promising, the field still faces substantial challenges, including the need for standardized datasets, robust evaluation metrics, and improved generalization across diverse tissues and conditions. Continued research and collaboration across disciplines will be essential to fully harness the potential of GANs and drive impactful advancements in digital pathology. By bridging the gap between deep learning innovation and clinical application, GANs promise to transform how we analyze and interpret histological data, ultimately contributing to more accurate, efficient, and accessible diagnostics in the future.