From Pixels to Volumes: Generative AI in 3D Medical Imaging

Abstract

With the advent of generative AI, medical imaging has been revolutionized, allowing for unprecedented capabilities in data generation, volumetric reconstruction, and clinical decision support. Although significant advances have been achieved in two-dimensional modalities, extending them to three-dimensional medical imaging, such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT), and Positron Emission Tomography (PET), remains a developing frontier with new technical and clinical challenges. This survey offers a thorough, systematic exploration of generative AI approaches uniquely applicable to 3D medical imaging, including voxel-based generative models, implicit neural representations, and latent diffusion models. The literature is organized in three orthogonal axes: imaging modality (MRI, CT, and PET), model architecture (GAN, VAE, diffusion, and NeRF), and clinical application (augmentation, reconstruction, surgical planning, and anomaly detection). We provide detailed taxonomy tables for each axis, including landmark papers, strengths, limitations, key techniques, and benchmark performance. We also address evaluation protocols, ethical issues related to synthetic data, and open research challenges. We analyzed more than 50 representative works and found that latent diffusion models have firmly established themselves as the standard for high-fidelity 3D synthesis and that implicit neural 3D representations are best for reconstructing 3D scenes from sparse views with limited memory. It does not, however, mean that they perform better across all modalities and tasks, as they have significantly greater requirements in terms of computational and memory load, sampling time, and training data compared to alternatives like diffusion-based methods (which account for about 44% of the surveyed landmark architectures. Finally, we propose a clinical, ethical, and technically sound blueprint for the use of generative AI in volumetric medical imaging.

1. Introduction

Over the last decade, the advent and development of generative artificial intelligence (GAI) has opened new possibilities in almost every field of computational science, with medicine at the forefront of potential impact. Medical imaging, a large family of volumetric imaging modalities, produces tremendous amounts of high-dimensional data with high information content that is sparsely labeled and annotated. Generative models, including Generative Adversarial Networks (GANs) [1], Variational Autoencoders (VAEs) [2], score-based diffusion models [3], and implicit neural representations (INRs) [4], can generate natural images, reconstruct undersampled acquisitions, and enrich sparse clinical datasets. However, extending two-dimensional slice capabilities into the 3D space of a volume presents a qualitatively different set of challenges that the literature has not adequately surveyed.

Three main volumetric imaging modalities are pillars of modern clinical diagnostic radiology and interventional planning. Magnetic Resonance Imaging (MRI) acquires data using the nuclear magnetic relaxation of hydrogen protons. It provides high-contrast soft-tissue images, which are extremely valuable for neuroimaging, musculoskeletal evaluation, and oncology [5]. Computed Tomography (CT) images volumetric density maps from multiple X-ray projections acquired at various angles. It is useful in trauma, pulmonary, and cardiovascular imaging because it acquires images quickly and provides excellent bone-to-tissue contrast [6]. Positron Emission Tomography (PET) records functional and metabolic activity by measuring coincidence photons from the injected tracer, providing molecular-level information that cannot be obtained elsewhere in oncological staging and neurological evaluation [7]. Different modalities have different acquisition physics, reconstruction pipelines, and noise properties, which impose different constraints on generative models that must stay within physical bounds to be clinically acceptable.

In medicine, the equivalent danger is creating synthetic volumes that are anatomically or physiologically impossible and can lead to false diagnoses or training; in false news, the danger is the same as with synthetic content, which can spread unchecked and lead to misinformation. However, with responsible generation, synthetically created 3D medical data can (i) fill gaps in information for rare pathologies; (ii) facilitate privacy-preserving data sharing and federated learning; (iii) reconstruct high-quality images from low-dose or sparse acquisitions and thus reduce patient exposure; (iv) serve as the backbone of next-generation clinical training platforms; and (v) enable preoperative surgical simulation. This is important and timely because a comprehensive, well-structured examination of the state of the art is required.

Review Methodology and Search Strategy

We conducted a literature search to ensure methodological transparency and reproducibility in accordance with the framework. See Figure 1 for overview of studies selected for the survey. We considered six major electronic databases and scholarly sources: PubMed/MEDLINE, IEEE Xplore, Scopus, Web of Science, Google Scholar, and arXiv. We searched through December 2024, focusing on research in medical imaging and generative AI. The search strategy comprised combinations of terms that included the following: generative models, medical imaging applications, such as generative models for Computed Tomography (CT), Positron Emission Tomography (PET), and Magnetic Resonance Imaging (MRI), and other terms related to image synthesis, reconstruction, segmentation, data augmentation, and harmonization. We created Boolean search strings for each database and adapted them as needed to match each database’s syntax and indexing requirements.

We included studies that analyzed generative artificial intelligence (AI) techniques in medical imaging, including their development, application, or evaluation. We also considered studies that addressed the development, application, and evaluation of generative artificial intelligence (AI) in medical imaging. The review included studies using generative architectures such as GANs, VAEs, diffusion models, score-based generative models, implicit neural representations, and other generative approaches. We considered original studies reporting research, methodological developments, quantitative and/or qualitative experimental results, and applications to medical imaging tasks. Studies that were not on medical imaging, not on generative AI techniques, reviews or editorial/commentary, were not original publications, and/or were poorly described in terms of method were excluded. We also removed duplicate records before screening.

The authors’ survey addresses the field in four main respects. First, it offers a taxonomy of 3D generative model architectures tailored to the constraints of volumetric medical imaging, including voxel-based, implicit neural, and latent diffusion architectures. Second, it surveys advances in MRI, CT, and PET, organized by modality in summary tables similar to those used in recent benchmark surveys [8,9,10]. Third, it systematically examines clinical applications such as data augmentation, surgical planning, digital twins, and education. Fourth, it critically reviews evaluation protocols, ethical issues, and open challenges and ends with a research roadmap for the future development of clinically deployable 3D generative systems. The rest of this paper is organized as follows: Section 2 reviews the background in medical imaging physics and generative model fundamentals, Section 4 and Section 5 describe the advances in medical imaging respective to each modality and cross-modally, Section 6 discusses the clinical applications, Section 7 discusses the evaluation, Section 8 discusses ethics and limitations, and Section 9 concludes with future directions.

2. Background and Preliminaries

2.1. Medical Imaging Modalities

Volumetric data acquisition systems underpin modern clinical imaging, and their physical properties determine the format of the acquired datasets. MRI acquires tissue-specific proton relaxation rates (T1, T2, and T2*) in the k-space frequency domain. Traditionally, reconstruction methods rely on inverse Fourier transforms, but undersampling for accelerated protocols creates aliasing artifacts that reconstruction methods must address [5]. The signal-to-noise ratio (SNR) is proportional to the magnetic field strength, the volume of voxels captured, and the time spent capturing the data to produce the image, and there is an inherent trade-off between scan speed and image quality that generative models hope to overcome by generating high-SNR volumes from low-SNR inputs.

During acquisition, CT generates maps of the X-ray attenuation coefficient in units known as Hounsfield Units (HU): air (−1000 HU), water (0 HU), and cortical bone (~+1000 HU). The angular reconstruction algorithm is filtered backprojection (FBP) or iterative reconstruction, and the number of projections and radiation dose intensity are proportional, respectively [6]. Dose reduction through sparse-view or low-current acquisition can degrade reconstruction quality, and a generative model trained on paired data can reconstruct diagnostic-quality volumes from these inputs. PET acquisitions generate count maps of annihilation photon pairs, which must be corrected for attenuation and reconstructed to localize radiotracer uptake. Weakly absorbed PET reduces injected tracer activity, increasing statistical noise; generative enhancement is therefore a high-impact target [7].

2.2. Generative Model Fundamentals

Four lineages of foundational generative architectures have been used in 3D medical imaging. However, GANs [1] use an adversarial training paradigm with a generator G and a discriminator D and can generate high-fidelity outputs but are prone to training instability and mode collapse. VAEs [2] train a variational lower bound for the likelihood of the data. At the same time, the latent representations learned are smooth and can be interpolated and conditioned, but the outputs can be fuzzy. Normalizing flows [11] learn an invertible transformation to approximate the likelihood but become very slow in high dimensions. Score-based diffusion models [3] add Gaussian noise to data via a forward stochastic differential equation (SDE) and train a neural network to reverse it, yielding state-of-the-art sample quality and flexible conditioning. Implicit neural representations (INRs) are functions that describe volumetric structure as a continuous function in space, parameterized by a neural network and flexible enough to generate output images at arbitrary resolutions, without the need for a fixed voxel grid, such as NeRF [4] and Signed Distance Functions (SDFs).

These models are difficult to extend from 2D images to 3D medical volumes. A single MRI brain volume can consist of 256 × 256 × 180 voxels, or ~12 M values per sample, compared to ~65 K for a 256 × 256 image. Spatial resolution scales cubically with memory, requiring architectural innovations such as patch-based training, hierarchical generation, and latent-space compression. Furthermore, medical volumes have modality-specific physical constraints that must be respected, such as realistic HU distributions for CT, appropriate k-space properties for MRI, and physically plausible radiotracer distributions for PET [12].

2.3. Generative Model Families and Unified Notation

Table 1. Unified mathematical notation used throughout this survey.

Symbol	Definition	Used in
$x_0$	Original (clean) image or volume, $x_0\in {\mathbb{R}}^{H\times W\times D}$	Section 2.4 and Section 3.3
$x_t$	Noised/latent version of $x_0$ at diffusion timestep t	Section 2.4, Section 3.3 and Section 9.2
t, T	Diffusion timestep (t ∈ {1, …, T} discrete, or t ∈ [0, 1] continuous SDE); T = total number of denoising steps	Section 2.4, Section 3.3 and Section 9.1
ε, εθ	Ground-truth Gaussian noise; noise predicted by the network with parameters θ	Section 2.4 and Section 3.3
z	Latent variable in a compressed (VAE) representation	Section 2.4, Section 3.1 and Section 3.3
θ	Learnable parameters of the generative network (generator, denoiser, or decoder)	Throughout
sθ(x,t)	Score function (gradient of the log-density) estimated by a neural network	Section 2.4 and Section 3.1
G, D	Generator and discriminator networks in adversarial (GAN) training	Section 2.4 and Section 3.1
q(xt|xt−1)	Forward (noising) transition distribution	Section 2.4 and Section 3.3
pθ(xt−1|xt)	Reverse (denoising) transition distribution parameterized by θ	Section 2.4 and Section 3.3
p(x), pdata(x)	Model distribution/true data distribution over images or volumes	Section 2.4 and Section 7
${\overline{\alpha }}_t$	Cumulative noise schedule (product of per-step retained-signal ratios) used in the closed-form forward diffusion process	Section 2.4 and Section 3.3
σ(t)	Noise scale/schedule as a function of diffusion time (SDE formulation)	Section 2.4 and Section 3.3
f(x,t), g(t)	Drift and diffusion coefficients of the forward SDE, dx = f(x,t)dt + g(t)dw	Section 2.4

lists the principal symbols and variables used throughout the mathematical formulations of this survey and the generative model families.

2.4. Key Mathematical Formulations

We collect the theoretical formulations referenced qualitatively in Section 2.2 here as explicit, numbered display equations, using the notation defined in Table 1.

$$\underset{G}{\min }\underset{\mathrm{D}}{\max }V\left(D,G\right)={\mathbb{E}}_{x~p_{data}\left(x\right)}\left[\log D\left(x\right)\right]+{\mathbb{E}}_{z~p_z\left(z\right)}\left[\log \left(1-D\left(G\left(z\right)\right)\right)\right]$$

(1)

The standard adversarial (GAN) objective (1) is an objective with which both the generator $G$ and the discriminator $D$ (Table 1) are optimized: The discriminator is optimizing to increase the probability of distinguishing real data $x$ from samples generated by the generator $G$, and the generator is optimizing to decrease the probability of the discriminator being able to distinguish between real data $x$ and the samples generated by the generator. Here, $p_{data}\left(x\right)$ is the true data distribution, and $p\left(z\right)$ is the prior over the latent input $z$, both defined in Table 1.

$$L\left(\theta ,\phi ;x\right)={\mathbb{E}}_{q_{\varphi }(z|x)}\left[\log p_{\theta }(x|z)\right]-D_{KL}\left(q_{\varphi }(z|x)\parallel p_{\theta }\left(z\right)\right)$$

(2)

The Variational Autoencoder (VAE) evidence lower bound (ELBO) is given in Equation (2) and is optimized by the encoder $q_{\phi }$ and decoder $p_{\theta }$. The first term is the expected reconstruction term of $x$ with the learned latent $z$ (Table 1), and the second term is the minimization of Kullback–Leibler divergence between the learned posterior $q_{\varphi }(z|x)$ and the prior $p\left(z\right)$ to keep the learned space close to a known distribution.

$$dx=f(x,t)dt+g\left(t\right)dw$$

(3)

Score-based diffusion models add noise to data sequentially, using the forward stochastic differential equation (SDE) shown in Equation (3). A sequence of infinitesimal increments of a standard Wiener process $dw$ is called a Brownian motion, and its deterministic and stochastic parts are governed by the drift coefficient $f(x,t)$ and diffusion coefficient $g\left(t\right)$ (Table 1), respectively.

$$dx=\left[f\left(x,t\right)-g{\left(t\right)}^2{\nabla }_x\mathrm{l}\mathrm{o}\mathrm{g}{p}_t\left(x\right)\right]dt+g\left(t\right)\hspace{0.17em}d\bar{w}$$

(4)

The following SDE, Equation (4), is the inverse or “backward” version and is used to create samples using the noising process from Equation (3) in reverse. A neural network $s_{\theta }(x,t)$ (Table 1) estimates its score term ${\nabla }_x\mathrm{l}\mathrm{o}\mathrm{g}{p}_t\left(x\right)$, the gradient of the log-density of the noised data at time $t$, and $d\overline{w}$ represents the increment of a reverse-time Wiener process.

$$x_t=\sqrt{{\bar{\alpha }}_t}\hspace{0.17em}{x}_0+\sqrt{1-{\bar{\alpha }}_t}\hspace{0.17em}\epsilon$$

(5)

The closed-form forward (noising) process of DDPM-style diffusion models is given by Equation (5), which shows that the noised sample $x_t$ is explicitly a weighted sum of the clean data $x_0$ and Gaussian noise $ϵ$ (Table 1), without iterating through all intermediate timesteps. The weighting term written ${\overline{\alpha }}_t$ (“alpha-bar-t”) is the product of the noise schedules, that is, the product of the noise schedule at each timestep up to timestep $t$.

$$L\left(\theta \right)={\mathbb{E}}_{x_0,\epsilon ,t}\left[{\Vert \epsilon -{\epsilon }_{\theta }\left(x_t,t\right)\Vert }^2\right]$$

(6)

Equation (6) shows the denoising score-matching training objective, which the network ${ϵ}_{\theta }$ described in Table 1 minimizes. It is trained to estimate the noise $ϵ$ that was added to $x_0$ to produce $x_t$ at timestep $t$, with the loss being the mean-squared error averaged over data samples, noise samples, and timesteps.

$$f_{\theta }:\left(x,y,z\right)\to \left(\sigma ,c\right)$$

(7)

An implicit neural representation (INR) is a coordinate-based network $f_{\theta }$, parameterized by $\theta$ (Table 1), that maps a continuous spatial location $\left(x,y,z\right)$ to a density or intensity value $\sigma$ and, if applicable, color or feature channels $c$.

$$C\left(r\right)={\int }_{t_n}^{t_f}T\left(t\right)\hspace{0.17em}\sigma \left(r\left(t\right)\right)\hspace{0.17em}c\left(r\left(t\right),d\right)\hspace{0.17em}dt$$

(8a)

$$T\left(t\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\int }_{t_n}^t\sigma \left(r\left(s\right)\right)\hspace{0.17em}ds\right)$$

(8b)

The volume-rendering integral Equation (8) is used to render an INR (e.g., NeRF) along a ray $r\left(t\right)$. The rendered color $C\left(r\right)$ is the integral of the density $\sigma \left(r\right(t\left)\right)$ and color/feature term $c\left(r\right(t),d)$ over the range of $t$ from $t_n$ to $t_f$, weighted by the transmittance $T\left(t\right)$, which is the probability that the ray will pass through a point on the ray from $t_n$ to $t$ without being absorbed or scattered.

3. Taxonomy of 3D Generative Architectures

In Figure 2, we categorize 3D generative models for medical imaging into three main classes, corresponding to three different computational philosophies for encoding, synthesizing, and optimizing volumetric structures.

Table 2. Taxonomy of 3D generative models in medical imaging.

Ref. No.	Model Type	Key Techniques Used	Strengths	Limitations	Target Task	Code	Dataset
[13]	Voxel-Based GAN	3D convolutional GAN; volumetric occupancy grids; probabilistic latent space; ShapeNet training	First 3D-GAN for shape synthesis; learns shape distributions; enables sampling and interpolation	Memory-intensive; mode collapse risk; cubic scaling of resolution cost	Shape/volume synthesis	https://github.com/zck119/3dgan-release (accessed on 1 August 2026)	Public (ShapeNet)
[14]	Voxel-Based GAN	GAN-based augmentation; learned spatial transforms; one-shot registration; synthesized training pairs	Significantly improves segmentation with minimal labels; adaptable to rare anatomy	Relies on atlas quality; limited to deformable structures; requires aligned input pairs	Data augmentation (one-shot seg.)	Not reported	Not reported
[15]	Voxel-Based VAE	Probabilistic generative model; anatomical shape priors; unsupervised latent segmentation	No manual labels needed; incorporates domain knowledge; robust across scanners	Limited baseline comparison; requires pre-registration steps; slow convergence	Unsupervised segmentation	Not reported	Public (ADNI/brain MRI)
[16]	Voxel-Based EBM	Deep 3D energy-based model; MCMC sampling; analysis by synthesis via maximum likelihood	No auxiliary networks (unlike GANs/VAEs); avoids mode collapse; realistic 3D shape generation	MCMC sampling computationally slow; voxel representation memory-heavy at high resolution	Shape synthesis	Not reported	Not reported
[17]	Voxel-Based VAE (Anomaly)	Comparative VAE study; reconstruction-error anomaly scoring; unsupervised lesion detection	No labeled lesion data required; surveys multiple VAE variants; strong generalization	Performance depends on healthy data quality; sensitive to noise in voxel-wise detection	Unsupervised anomaly detection	Not publicly reported	Public (brain MRI)
[18]	Voxel-Based Score Model	Score-based diffusion (denoising score matching) + Markov chain Monte Carlo (MCMC/Langevin) posterior sampling; k-space data-consistency term incorporated into the reverse diffusion at every step	Works from highly undersampled k-space (tested at 4×–10× and up to 8× on FLAIR); no hand-crafted sparsity prior needed (learned generative prior replaces wavelet/ℓ1 regularization); also yields pixel-wise uncertainty (variance) maps as a by-product	Computationally intensive (~10 min per reconstruction vs. ~5 s for ℓ1-ESPIRiT with BART); hallucinations/artifacts can appear at high acceleration	MRI reconstruction (undersampled k-space), brain imaging (T1w, T2w, FLAIR, T2*w)	https://github.com/mrirecon/spreco (accessed on 1 August 2026)	Public (fastMRI)
[19]	Voxel-Based Diffusion	DDPM; 3D volume generation; noise scheduling adapted for clinical MRI/CT volumes	Realistic anatomically correct 3D images; addresses data privacy and scarcity	Computationally intensive training and inference; complex implementation for clinical deployment	3D volumetric synthesis (MRI/CT)	https://github.com/FirasGit/medicaldiffusion (accessed on 1 August 2026)	Public (BraTS, LIDC-IDRI)
[20]	Voxel-Based CNN	Residual encoder–decoder; perceptual loss; 3D volumetric convolutions; paired low/full-dose CT	Clinically indistinguishable in 91% of cases; real-time inference	Requires paired training data; may over-smooth fine textures; site-specific training needed	Low-dose CT denoising	Not reported	Public (AAPM Low-Dose CT)
[21]	Voxel-Based GAN (Cross-modal)	CycleGAN (3D); unpaired MRI-to-CT synthesis; Hounsfield Unit prediction; dose-volume evaluation	Eliminates separate CT scan; low MAE; DSC 0.91 on bone structures; enables MRI-only workflows	High memory footprint; requires pre-registration; limited generalizability across scanners	Cross-modal translation (MRI→CT)	Not reported	Not reported (institutional)
[22]	Implicit (NeRF)	NeRF coordinate-based rendering; multi-view X-ray/CT synthesis; volume rendering integral; parametric motion	Memory-efficient 3D representation; continuous resolution; strong few-shot view synthesis	Long inference time per point evaluation; difficulty on non-rigid anatomy; noisy clinical data challenges	Sparse-view/dynamic CT reconstruction	https://github.com/awreed/DynamicCTReconstruction (accessed on 1 August 2026)	Public (LIDC-IDRI)
[23]	Untrained implicit neural representation	Fourier-feature MLP with separate spatial/temporal embeddings; per-scan fitting via k-space data-consistency loss only; no explicit regularizer	No training data/ground truth or ECG needed; on par with/slightly beats SOTA untrained CNN (t-DIP); clearly beats other implicit-representation baselines (NIK, KFMLP); easily extends to 3D	High computational cost; tested on only one healthy volunteer (limited generalizability); no real-data ground truth (relies on SER + visual comparison, with a phantom used for true SSIM/VIF)	Free-breathing, ungated, real-time cardiac cine MRI reconstruction	Public—https://github.com/MLI-lab/cinemri (accessed on 1 August 2026)	Public in-house free-breathing cardiac MRI dataset (DOI 10.21227/f057-dw29) + MRXCAT synthetic phantom
[24]	Latent Diffusion (LDM)	Score-based SDE framework; DDPM generalization; classifier-free guidance; stochastic generation	High sample quality; stable training dynamics; unifies DDPM and NCSN frameworks	Slow sequential sampling; computationally expensive at inference; large model size	Generative framework (theory/benchmark)	https://github.com/yang-song/score_sde (accessed on 1 August 2026)	Public (CIFAR-10, CelebA)
[25]	Latent Diffusion (3D)	VAE latent space diffusion; 3D U-Net backbone; multi-modal conditioning; brain MRI synthesis	Anatomically plausible MRI/CT synthesis; multi-modal capability; scalable compression	Requires large datasets; complex latent space tuning; resolution limited by VAE bottleneck	3D brain MRI synthesis	https://github.com/Project-MONAI/GenerativeModels (accessed on 1 August 2026)	Public (BraTS-derived)
[26]	Latent Diffusion (Medical)	Task-specific conditioning; anatomy-guided diffusion; segmentation ensemble generation	Generates rare pathologies; preserves structural integrity; interpretable conditioning	Limited to isotropic resolutions; needs paired data for conditioning; slow inference	Segmentation-ensemble/rare-pathology synthesis	Not reported	Public (segmentation-conditioned)
[27]	Latent Diffusion (PET)	DDPM-based PET denoising; MRI-guided prior integration; Poisson noise-aware iterative refinement	Outperforms non-local mean and U-Net baselines; flexible use of MRI prior; handles multiple tracers	Slow inference time; relies on MRI prior availability; limited to 2D slice-wise processing	PET denoising	Not reported	Not reported (clinical PET)
[28]	Latent Diffusion (MRI)	Forward model integration; physics-guided DDPM sampling; k-space consistency enforcement	Accelerated MRI reconstruction; uncertainty quantification; physical measurement consistency	Sensitive to forward model errors; computationally intensive; limited multi-coil support	General inverse-problem solving (MRI/CT recon.)	https://github.com/DPS2022/diffusion-posterior-sampling (accessed on 1 August 2026)	Not reported

provides a comparative overview of landmark papers in each category.

Whether the training/evaluation dataset, source code, pretrained model weights, prompts/conditioning configurations, preprocessing pipeline, inference settings, computational resources used, and external validation results are publicly available is reported for each method in Table 2 as of the time of writing this survey, based on the original publication and the corresponding repository. Entries are listed as public, partial (some but not all artifacts released or released with restrictions), or not reported (where the publication does not state that artifacts are available). The 16 studies reported details important for reproducing the results inconsistently. For unconditional models, only the full specification is given for prompting or conditioning in [19,25] and only a partial specification in [14,26]. Preprocessing pipelines were fully reported only in a few studies. Inference settings were reported in a few papers only, partially in [13,15,21,25,27,28]. Fewer than half of the studies reported compute resources (in [13,18,19,24,25] and partially in [20,22]). At the same time, external validation was the least developed attribute overall, performed only in [20] and partially in [17,19] and absent in the remaining twelve studies, indicating a broader shortage of transparency and independent evaluation in the reviewed generative modeling literature.

Overall, code is released publicly for approximately half of the architectures; in many cases, when code is made available, pretrained weights are not released; and preprocessing/inference configuration dimensions are the least consistently reported dimension, a gap which significantly hinders independent replication and is seen throughout the benchmarking discussion of Section 7 and in the standardization challenge of Section 9.4.

3.1. Voxel-Based Generative Models

The 2D generative architectures (as shown in Figure 3) are expanded to 3D volumes of data, represented as an explicit grid with 3D convolutions. They have been extensively applied to brain MRI synthesis, tumor segmentation, and organ shape modeling: GAN-Based Models. Early 3D GANs used volumetric operations rather than 2D convolutions to create realistic 3D shapes from latent vectors [13]. Researchers have applied GANs in medical imaging for data augmentation and cross-modal synthesis. Zhao et al. [14] introduced learned spatial transforms to GAN-based augmentation to improve one-shot segmentation, and Maspero et al. [21] proposed a 3D CycleGAN for MRI-to-CT synthesis with MRI-only radiotherapy planning, achieving a Dice score of 0.91.

Postgraduate training in Segmentation and Anomaly Detection in the context of VAEs: The VAEs learn smooth latent representations by solving probabilistic reconstruction tasks. Dalca et al. [15] combined anatomical prior information with a convolutional VAE for unsupervised segmentation in the context of biomedical images. Baur et al. [17] also used VAE variants to detect lesions in brain MRI images unsupervised using reconstruction error: energy-based models (EBMs). Xie et al. [16] developed deep 3D EBMs for shape synthesis based on maximum likelihood and MCMC sampling. These models overcome the mode collapse problem in GANs but require high memory consumption and costly sampling: score-based and diffusion models. Luo et al. [18] proposed score-based stochastic differential equations (SDEs) for fidelity in MRI reconstruction under high undersampling. To address privacy and data scarcity, Khader et al. [19] modified DPMs to generate anatomically realistic synthetic 3D MRI and CT images. Residual CNNs for CT denoising: For low-dose CT denoising, Chen et al. [20] proposed a residual encoder–decoder structure for denoising called RED-CNN. The approach produced clinically acceptable image quality and enabled real-time inference. One limitation of voxel-based methods is that the number of days for 3D training scales cubically with resolution, which can make full-resolution training expensive.

Table 3. Modality-specific characteristics, challenges, and evaluation considerations in generative medical imaging.

Modality	Key Data Characteristics	Modality-Specific Challenges	Important Preprocessing	Generative Applications	Recommended Evaluation Considerations
MRI	Multi-sequence (T1, T2, FLAIR, DWI, etc.); scanner-, vendor-, and site-dependent intensity distributions; variable contrast	Scanner-, vendor-, field-strength-, and sequence-dependent intensity variation; motion and noise sensitivity; intensities not directly quantitative	Registration, resampling, bias-field correction, and intensity normalization	Reconstruction, synthesis, cross-sequence translation, and augmentation	PSNR, SSIM, MAE, and NMSE; downstream segmentation/task performance
CT	Quantitative attenuation expressed in Hounsfield Units (HU); relatively standardized intensity scale across scanners	Preservation of HU values; dose–noise trade-off (low dose vs. full dose); anatomical fidelity at reduced dose	HU clipping/windowing, normalization, and resampling	Low-dose reconstruction, denoising, and modality synthesis (e.g., MR-to-CT)	HU-MAE/HU-RMSE, PSNR, SSIM, noise, and structural fidelity
PET	Tracer-dependent uptake distribution (e.g., FDG and PSMA); quantitative Standardized Uptake Value (SUV); dose- and count-dependent statistics	Tracer distribution and kinetics; low-count/low-dose statistical noise; limited spatial resolution; SUV preservation	SUV normalization, registration, and resampling	Low-dose PET reconstruction, PET synthesis, and denoising	SUV error/bias, MAE/RMSE, PSNR, SSIM, and lesion-level detection metrics

shows considerations specific to each modality. While all three modalities, MRI, CT, and PET, are depicted as “three-dimensional files,” their physical acquisition methods and quantitative characteristics are quite different and thus pose different requirements to the generative models. Intensity can vary significantly across scanners, sites, and sequences (acquisition-domain variability), so intensity normalization is an important factor for synthesis and reconstruction. CT provides quantitative attenuation data in Hounsfield Units; therefore, generative models must retain HU fidelity and anatomical realism, especially when reconstructing and using images for quantitative applications. PET also has special challenges with tracer distribution, acquisition dose and count statistics, spatial resolution, and maintaining Standardized Uptake Values (SUVs). Accordingly, the taxonomy and evaluation framework proposed in this survey separate modality-specific requirements and treat all volumetric medical images as different inputs to the evaluation process. Thus, the new taxonomy accounts for both the generative architecture and the properties of the imaging modality used, and it employs modality-specific quantitative and perceptual measures to assess generated/reconstructed images.

3.2. Implicit Neural Representations

Implicit neural representations (INRs) represent volumetric structures as continuous coordinate-based neural functions (as depicted in Figure 4). This provides a high-resolution, memory-efficient representation: NeRF for CT reconstruction. Reed et al. [22] adopted the neural radiance fields (NeRF) approach to render a scene’s 3D geometry from various 2D projections and model scene motion to reconstruct the CT image from sparse views. This framework enables volume reconstruction from limited projections, even with few projections, but it also requires expensive per-coordinate inference. Dynamic cardiac MRI reconstruction: The authors in [23] used INR frameworks to reconstruct cardiac MRI images continuously and dynamically, jointly modeling spatial and temporal information. They achieved this by building up sparse k-space with reduced artifacts and improved temporal continuity. While INRs are memory-efficient and resolution-independent, they infer slowly because they repeatedly evaluate neural networks and cannot generate volume data for other subjects.

While the near-instant per-scan optimization of a pretrained voxel- or latent-space model is desirable, it should not be confused with the efficiency of their compact, resolution-independent representation: a few minutes to hours of training time is required to optimize per scan before a volume can be rendered, as opposed to the near-instant inference of a pretrained model; per-scan rendering latency also depends on the number of coordinate evaluations needed to reach the desired resolution. INR-based reconstruction is also highly dependent on the geometry used for the optimization (e.g., the number and the angular distribution of the sparse views). It has been less validated on pathological or highly heterogeneous anatomy (e.g., big lesions and post-surgical changes) than on relatively regular anatomy as used in most reported benchmarks.

3.3. Latent Diffusion Models

Today, the state-of-the-art generative medical imaging method is called a latent diffusion model (LDM). These methods reduce the complexity of the denoising process by using VAEs to compress images and conducting diffusion-based generation in the compressed space. Song et al. [24] combined DDPMs and score-based models with stochastic differential equations (SDEs) to achieve stable, high-quality image synthesis. Pinaya et al. [25] adopted latent diffusion to generate 3D brain MRI and CT images using a 3D U-Net denoising backbone. They achieved multi-modal, scalable synthesis of anatomically realistic outputs. To create segmentation ensembles and synthesize rare pathologies, Wolleb et al. [26] introduced anatomy-guided diffusion models and showed that they could condition models with a task-specific prompt. PET denoising and MRI reconstruction: Gong et al. [27] proposed diffusion models for PET image denoising with MRI-guided priors, and Chung et al. [28] proposed diffusion posterior sampling for accelerated MRI reconstruction with uncertainty estimation and k-space consistency. LDMs outperform pixel-space diffusion models in image fidelity, conditioning, and computational efficiency. But for clinical use, they still require massive datasets, multiple denoising iterations, and substantial computing resources.

Figure 5 below provides a dedicated architectural schematic of a latent diffusion model (LDM) for 3D medical image synthesis, illustrating how the components introduced qualitatively above (VAE encoder/decoder, latent representation, forward noising process, denoising network, and conditioning) fit together end to end.

The components discussed in the previous section are integrated into a 3D medical image synthesis latent diffusion model (LDM), as shown in Figure 5 below. A forward diffusion process (training only) is used to add Gaussian noise to the latent, which is then used as input to a denoising 3D U-Net optionally conditioned on a mask, a text prompt, or a reference modality; the denoised latent is then fed into the VAE decoder to reconstruct a full-resolution 3D volume.

“Standard” is the most commonly used architecture for high-fidelity volumetric generation in the reviewed works, but it is not necessarily best for all modalities and applications. The five architectures summarized in Table 2 that are latent-space diffusion (31%) and seven that use diffusion broadly (44%) make up a plurality, but not a near-universal share, of the sixteen summarized architectures; several of those listed as “latent diffusion” in Table 2 (e.g., [24,27,28]) work in pixel/voxel space, not a compressed latent space. The picture also changes by task: for sparse-view CT reconstruction, NeRF-based methods remain competitive in fidelity (SSIM 0.91,

Table 4. Modality-specific advances in 3D generative medical imaging.

Key Model	Modality	Anatomical Region/Population	Conditioning Input	Evaluation Protocol	Main Challenge	Approach	Key Metric/Result	Dataset/Code Link If Available	Comparability Flag and Rationale	Target Task	Ref.
MRI—Magnetic Resonance Imaging
Bayesian Diffusion-MCMC Reconstruction (score-based generative model)	MRI	Brain; healthy volunteers (in-house, n = 13) + fastMRI brain subset (T1w post-contrast, T2w, FLAIR)—not knee	Undersampled multi-coil k-space; coil sensitivity maps from ESPIRiT; no paired ground-truth needed at inference	Retrospective, simulated undersampling (single-coil unfolding, multicoil, 4×/8×/10×); compared against ℓ1-wavelet-regularized reconstruction	Ill-posedness of reconstruction from undersampled k-space; quantifying reconstruction uncertainty; heavy computational cost of MCMC sampling	Score-based (denoising score matching) generative diffusion prior + Bayesian posterior sampling via MCMC/Langevin dynamics	PSNR ≈ 34–37 dB, SSIM ≈ 0.90–0.94 depending on setting; recovers finer detail than ℓ1-wavelet regularization	In-house brain dataset (1300 images, 13 healthy volunteers) + public fastMRI (brain subset). Code: https://github.com/mrirecon/spreco (accessed on 1 August 2026)	✗ Not directly comparable to other rows—uses a private in-house dataset in addition to fastMRI, and MCMC-based posterior sampling	MRI reconstruction (undersampled k-space) with joint uncertainty estimation	[18]
Medical Diffusion (DDPM)	MRI/CT	Brain tumor (BraTS); lung nodule (LIDC-IDRI)	Unconditional generation	FID vs. real-data distribution	Data scarcity; privacy concerns	Denoising diffusion probabilistic model (DDPM) for 3D volumetric generation	Realistic 3D brain MRI and lung CT synthesis; FID scores comparable to real data	BraTS, LIDC-IDRI Link—https://github.com/FirasGit/medicaldiffusion (accessed on 1 August 2026)	△ Shares BraTS w/DeepHarmony, LIDC-IDRI w/NeRF-CT—different task/metric each time	3D volumetric synthesis (MRI/CT)	[19]
Fourier-feature MLP implicit neural representation	MRI (cardiac cine, real-time)	• Heart • 1 healthy volunteer (30 y, male), 3T Philips Elition X—not fastMRI, not a population study	• Continuously acquired, undersampled multi-coil k-space • Free-breathing, ungated, partial-Fourier Cartesian sampling	• Retrospective; untrained/self-supervised (no training data) • Baselines: t-DIP (CNN), NIK, KFMLP	Continuous cardiac + respiratory motion during acquisition • No ground-truth data available for real-time free-breathing MRI	Untrained implicit neural network (MLP) with separate spatial and temporal Fourier-feature embeddings	SER ≈ 9–17 dB across datasets—on par with/slightly better than t-DIP, clearly better than NIK/KFMLP	• Public dataset: DOI 10.21227/f057-dw29 • Public code: https://github.com/MLI-lab/cinemri (accessed on 1 August 2026)	✗ Not directly comparable to fastMRI-benchmarked rows—own single-subject dataset, no PSNR/SSIM ground truth (uses SER instead)	• Free-breathing, ungated real-time cardiac cine MRI reconstruction (untrained method)	[23]
DeepHarmony	MRI	Multi-site; mixed healthy + BraTS tumor cohorts	Paired T1→T2 style transfer	Downstream cross-scanner segmentation consistency (task-based)	Domain shift across scanner manufacturers/vendors	GAN-based contrast harmonization (T1→T2 style transfer)	Improved cross-scanner segmentation consistency; preserves anatomical fidelity	IXI, ABIDE, and BraTS	△ Shares BraTS with Medical Diffusion below—different task/metric, not comparable	Scanner/contrast harmonization (style transfer)	[29]
CT—Computed Tomography
RED-CNN	CT	Abdomen (AAPM cohort)	Paired low-dose/full-dose CT	Reader study (clinical indistinguishability) + PSNR	Ionizing radiation dose; low-dose noise	Residual encoder–decoder CNN with perceptual loss and adversarial training	PSNR improvement at 25% dose; preserves high-frequency details; clinically indistinguishable in 91% of cases	AAPM Low-Dose CT Challenge	✗ Not comparable to NeRF-CT below—reader-study result vs. pure SSIM value	Low-dose CT denoising	[20]
NeRF for Sparse-View CT	CT	Lung nodule cohort (public LIDC-IDRI)	Sparse projections (≤10 views) vs. FBP	SSIM vs. filtered backprojection baseline only	Radiation exposure; limited projections	Neural radiance fields (NeRF) with volume rendering from sparse views (≤10 projections)	SSIM 0.91 from 10 views vs. 0.78 for FBP; reduced streak artifacts	LIDC-IDRI	△ Shares LIDC-IDRI with Medical Diffusion—different task, not comparable	Sparse-view CT reconstruction	[30]
Penalized-likelihood statistical image reconstruction with ordered-subsets algorithm for polyenergetic X-ray CT	CT	Simulated numerical phantom containing bone and soft tissue	Known energy-dependent mass attenuation coefficients per material; assumed polyenergetic X-ray source spectrum	Simulated X-ray CT transmission measurements of a two-material (bone/soft-tissue) phantom;	Conventional (monoenergetic) reconstruction methods ignore the polyenergetic nature of the X-ray source, causing severe beam-hardening artifacts.	Physical model accounting for polyenergetic spectrum and energy-dependent attenuation; penalized-likelihood cost function optimized via an ordered-subsets	Substantially reduced beam-hardening artifacts compared to conventional/monoenergetic reconstruction; improved voxel-density accuracy for bone/soft-tissue mixtures	No public dataset or code release; simulation-only data generated by the authors	✗ Not comparable	Beam-hardening artifact reduction and material-density estimation in CT image reconstruction	[31]
PET—Positron Emission Tomography
Denoising Diffusion Probabilistic Model	PET	Human brain imaging; 120 18F-FDG datasets and 140 18F-MK-6240 (tau) datasets	Low-count/noisy PET image and/or MR prior image, supplied either as direct network input	Regional and surface-based quantification on FDG and MK-6240 datasets;	PET image quality is degraded by physical degradation factors and limited photon counts, especially in low-dose/low-count acquisitions s	DDPM iteratively transforms a normal distribution into the target PET data distribution; evaluates variants where PET/MR prior images are supplied as network input versus as a refinement-step data-consistency constraint	DDPM-based methods incorporating PET information outperform nonlocal-mean and U-Net-based denoising; best performance achieved using MR prior as network input combined with PET as a data-consistency constraint during inference	Clinical datasets not publicly released; no code	✗ Comparable within the PET deep-generative denoising literature and to other DDPM-based PET works using similar tracers	PET image denoising/noise reduction leveraging prior anatomical or PET information	[27]
Attention-Based cGAN for PET Synthesis	PET	Alzheimer’s cohort (ADNI)	Paired MRI→PET (conditional)	MAE/PSNR/SSIM vs. CycleGAN baseline; SUV-preservation check	Missing PET data in multi-modal studies	Conditional GAN (cGAN) with attention mechanism for MRI→PET synthesis	Superior MAE/PSNR/SSIM compared to CycleGAN; SUV preservation	ADNI	✗ Not comparable to Low-Dose PET Diffusion row—different dataset/task	Cross-modal synthesis (MRI→PET)	[30]
General/Multi-Modality (Foundational Methods)
Score-Based Generative Modeling (SDE)	General	N/A—natural images, not patients	Unconditional	Standard generative-modeling benchmark (non-clinical)	Theoretical foundation for diffusion models	Stochastic differential equations (SDE) unifying DDPM and NCSN frameworks	Unified framework; high-quality generation; stable training	CIFAR-10, CelebA Link—https://github.com/yang-song/score_sde (accessed on 1 August 2026)	✗ NOT COMPARABLE to any medical imaging row—non-medical benchmark (CIFAR-10/CelebA)	General generative framework (non-medical benchmark)	[24]
Diffusion Posterior Sampling (DPS)	General	Not specified	Physics-guided forward-model consistency	Aggregated across multiple inverse problems; no single dataset	Noisy inverse problems (MRI, CT, etc.)	Physics-guided DDPM sampling with forward model consistency	Uncertainty quantification; outperforms supervised methods on several inverse problems	Various (MRI, CT, and deblurring) Link—https://github.com/DPS2022/diffusion-posterior-sampling (accessed on 1 August 2026)	✗ Not comparable—dataset unspecified/aggregated across tasks	General inverse-problem solving (MRI/CT/deblurring)	[28]
Adversarial Diffusion Model for Unsupervised Translation	General	Not specified	Unpaired image to image	Qualitative structural-integrity preservation	Unpaired image-to-image translation (any modality)	Adversarial diffusion model combining DDPM with adversarial loss	Unsupervised translation without paired data; preserves structural integrity	MRI, CT, and fundus images	✗ Not comparable—multi-modality aggregate, unpaired, no quantitative value	Unpaired cross-modality image translation	[32]
Dual-Task Consistency for Semi-Supervised Segmentation	General	Not specified	Shared representation (segmentation + reconstruction)	Segmentation accuracy with few labels	Limited labeled data for segmentation	Dual-task consistency (segmentation + reconstruction) with shared representation	Improved segmentation with few labels; not a generative model per se	Various medical images	✗ Not comparable AND a scope concern—table notes this is “not a generative model per se” (see Comment 4)	Semi-supervised segmentation (reconstruction-assisted)	[33]

✗ Not comparable; △ conditionally comparable.

), while conditional GANs are competitive for MRI-to-PET synthesis (Table 4). Diffusion-based methods clearly outperform in fine-grained conditioning or uncertainty quantification applications. The gains have non-trivial costs for clinical translation: sampling often involves 10–1000 sequential denoising passes per volume (Section 9.1); fewer than 30% of the diffusion-based papers reviewed in this survey publish public code, so it is not easy to verify the reported computational cost; and several papers explicitly mention the need for large datasets and long training times as limiting factors. Finally, Figure 6 shows a schematic, illustrative comparison of the output properties obtained from generative 3D models based on GANs, INRs, and diffusion.

Figure 6 shows schematic, non-reproduced examples of qualitative output tendencies reported across the three architectural families listed in Table 2: (a) GAN-based synthesis (as in cross-modal translation approaches such as [21], (b) implicit neural representations (as in sparse-view reconstruction approaches such as [22], and (c) diffusion-based synthesis (as in volumetric generation approaches such as [19,25]). The panel annotation summarizes the qualitative differences reported in the corresponding original papers but does not show the actual images published in the papers.

4. Modality-Specific Advances

The performance and optimal design of 3D generative models will differ significantly across imaging modalities, each with distinct acquisition principles, resolution limits, and clinical needs. Beyond these common challenges, each of the three technological modalities associated with generative architectures has its own set of problems, such as long acquisition times for MRI, managing radiation doses for CT, and low signal-to-noise reconstruction for PET. No single solution or blueprint meets these modality-specific challenges; instead, the field has developed a robust set of focused solutions. This section briefly summarizes the most important recent developments in each of these three modalities, as shown in Table 4, and shows how acquisition physics, deep generative design decisions, target task, code/weights availability, and clinical translation are interlinked.

4.1. Magnetic Resonance Imaging (MRI)

Two fundamental issues with MRI are slow acquisition and the need to reconstruct high-fidelity images from undersampled k-space measurements, as well as the need to harmonize contrast across different and heterogeneous scanners. Therefore, DeepHarmony [29] showed that an adversarial image-to-image translation method could be used to separate the scanner-induced contrast variation from the ground truth anatomical structure between three datasets: IXI, ABIDE, and BraTS for scanner harmonization. Medical Diffusion [19] created realistic 3D brain MRIs with FID scores similar to those of standard MRIs. The authors in [23] developed a generative modeling framework for dynamic cardiac MRI using implicit neural representations that encode spatial and temporal dimensions together, allowing them to reconstruct MRI images from sparse k-space sample points even when they are unevenly spaced in time.

4.2. Computed Tomography (CT)

The main research goal of CT is to produce diagnostic images, even when the dose or number of projections is reduced to lower radiation exposure. The RED-CNN paradigm [20] became a landmark, using a residual encoder–decoder and adversarial perceptual loss to produce clinically comparable results to full-dose CT in 91% of radiologist readings at 25% of the standard dose. Jiao et al. [30] modified neural radiance fields for 3D reconstruction from CT, achieving an SSIM of 0.91 with only 10 projection images, compared with 0.78 for filtered backprojection. In addition, the authors in [31] rather than assuming a single X-ray energy, we introduce a statistical (penalized-likelihood) image reconstruction method for X-ray CT, which explicitly models the polyenergetic nature of the X-ray source and its energy-dependent attenuation.

4.3. Positron Emission Tomography (PET)

Two research directions have been motivated by PET’s fundamental compromise between image quality and radiotracer dose: the first focuses on new scanning methods that improve image quality, and the second involves developing novel methods to use existing scanners for enhanced image quality. The purpose of this paper in [27] is to apply diffusion probabilistic models (DDPM) to PET image denoising, exploring different ways to incorporate the noisy PET image and/or an MR anatomical prior (as network input versus as a data-consistency constraint during refinement When direct PET acquisition is not possible, Wang et al. [34] proposed a mechanism that uses multi-modality (PET + MRI) features rather than fixed fusion rules to make the fusion locality-adaptive, learned jointly within the 3D conditional GAN.

4.4. Foundational Cross-Modality Methods

Several algorithmic frameworks support developments across modalities. Song et al. [24] brought these together in a single, continuous SDE framework for principled conditional generation and likelihood evaluation, which is the most fundamental reference in contemporary 3D medical generative imaging. Chung et al. [28] further extended this to general inverse problems (covering MRI undersampling, CT reconstruction, and deblurring), using a single model across these three tasks without task-specific retraining. For cross-modality translation, Özbey et al. [32] applied DDPMs with adversarial loss to achieve state-of-the-art results on tasks such as MRI to CT and fundus to MRI. In contrast, the dual-task consistency framework of Luo et al. [33] uses reconstruction as a self-supervisory signal to improve segmentation when labels are scarce, highlighting the growing cross-pollination between generative and discriminative clinical goals.

4.5. Summary and Cross-Modality Observations

Across all three modalities, diffusion-based architectures have emerged as the preferred generative paradigm over GANs because of their training stability, uncertainty quantification, and posterior sampling. In the MRI case, k-space consistency constraints are always crucial. In the CT case, forward models are always crucial, with the polyenergetic beam model in the latter case being a special case of the former for CT. In each case, a purely data-driven model could hallucinate anatomical structures or lose quantitative signal fidelity, making physics-informed constraints important for clinical translation. In particular, methodologies are increasingly merging: NeRF-based representations from CT are tested in MRI recovery; INR-based frameworks are extended to dynamic PET; and DPS is the first to offer a common reconstruction algorithm for all three modalities. This convergence points toward modality-agnostic architectures based on acquisition-specific physics models, rather than separate model families for each imaging domain.

5. Cross-Modal and Multi-Modal Synthesis

Multi-modal clinical imaging workflows often involve multiple modalities acquired in series or on combined scanners (e.g., PET/CT or PET/MRI), creating the need and opportunity for cross-modal synthesis, where one imaging modality is derived from another in a computational sense. Major challenges in cross-modal synthesis that remain in the clinic include eliminating redundant acquisitions, reducing patient radiation dose, avoiding the need for costly radiotracers, and overcoming registration errors associated with sequential multi-modal scans. Generative modeling methods for cross-modal synthesis range from unpaired cycle-consistent GANs to conditional diffusion models and physics-aware methods.

Table 5. Cross-modal and multi-modal 3D synthesis.

Ref	Dataset	Model/Method	Source Modality	Target Modality	Architecture	Key Metric/Result
[35]	Head & Neck (Jiangxi Cancer Hospital)	cycleSimulationGAN	MRI (T1/T2)	CT (HU map)	CycleGAN + channel-wise attention + structural similarity loss	MAE 52.3 HU; DSC 0.91 on bone; outperforms standard CycleGAN
[36]	Brain & Prostate (Emory University, institutional)	MC-IDDPM (3D Transformer Diffusion)	MRI (T1)	CT (HU map)	Swin-Vnet denoising diffusion probabilistic model (3D)	Brain: MAE 48.8 HU, SSIM 0.947, and NCC 0.976; Prostate: MAE 55.1 HU and SSIM 0.878
[32]	IXI, BRATS (multi-contrast MRI); in-house MRI–CT	SynDiff	MRI (multi-contrast)	CT/PET (target)	Adversarial diffusion model with cycle-consistent architecture (unpaired)	Superior PSNR/SSIM/FID vs. CycleGAN and DDPM baselines on multi-contrast MRI and MRI–CT translation
[37]	ADNI (Alzheimer’s Disease Neuroimaging Initiative)	MRI-to-PET Diffusion with Pathology Awareness	MRI (T1)	FDG-PET	Conditional diffusion model with pathology-aware attention	Improved SUV correlation; better lesion-region synthesis vs. attention cGAN; evaluated for AD staging
[38]	Pelvic CT–MRI paired (institutional)	MD-DGA (Multi-Dimensional Diffusion Generation Architecture)	CT	MRI (T2)	2D scalable diffusion model (2D-SDM) + 3D scalable latent diffusion model (3D-SLDM) with 3D-VQVAE	High-fidelity 3D MRI synthesis from paired CT volumes; preserves volumetric consistency
[39]	5000 scans across 6 modalities (MRI T1/T2/FLAIR/PD, CT); multiple public datasets	SynthSeg	MRI (any contrast/resolution)	Synthetic segmentation label map	CNN trained on domain-randomized synthetic data from generative model conditioned on segmentations	DSC comparable to supervised CNNs across 5000 scans; 6 modalities; 10 resolutions; no retraining needed
[40]	Clinical PET (brain/whole-body)	PET Consistency Model (PET-CM)	Low-dose PET (1/8 or 1/4 dose)	Full-dose PET	Denoising diffusion with PET Shifted-window Vision Transformer (PET-VIT); consistency model	1/8-dose: PSNR 33.9 dB, SSIM 0.964, and NCC 0.968; 12× faster inference than DDPM

summarizes typical cross-modal 3D generative work, which includes MRI-to-CT, MRI-to-PET, CT-to-MRI, and universal multi-modal synthesis.

5.1. MRI-to-CT and MRI-to-PET Synthesis

The most clinically established cross-modal task is MRI-to-CT synthesis, motivated by the requirement of radiotherapy planning, where MRI provides the highest soft-tissue contrast. Still, CT is essential for dose calculation. Gong et al. [35] used an attention-enhanced CycleGAN to get an MAE of 52.3 HU, and Pan et al. [36] used a transformer-based diffusion model (MC-IDDPM) to further reduce this to 48.8 HU (which is clinically acceptable for MRI-only radiotherapy workflows with a dose coverage difference within ±0.34%). SynDiff [32] was proposed by Özbey et al. for unpaired bidirectional translation between MRI and PET and achieved better results than the GAN and the DDPM baseline training models. A conditional diffusion model proposed by Lee et al. [37] trained with pathologically relevant regions of the brain led to an improvement in SUV correlation in diagnostically relevant areas of the cortex for staging for Alzheimer’s disease.

5.2. CT-to-MRI Synthesis

One application of CT-to-MRI synthesis is when MRI is unavailable but soft-tissue contrast is desirable for tumor delineation and planning surgical procedures. One of the difficulties is to maintain volumetric consistency from slice to slice, which 2D translation models cannot do. This was addressed by Huang et al. [38] by using MD-DGA, which integrated a 2D scalable diffusion model for detail within the slices with a 3D latent diffusion model with a vector-quantized VAE for coherence between the slices to achieve high-fidelity T2-weighted MRI volumes from a pelvic CT input, concluding that hybrid 2D–3D diffusion architectures are required for clinically useful volumetric synthesis.

5.3. Universal and Contrast-Agnostic Synthesis

Rather than learning fixed pairwise modality mappings, domain-randomized approaches learn on synthetically generated images across the full space of plausible imaging appearances. Billot et al. [39] showed that this is possible with SynthSeg, which only needed to be segmented once and achieved comparable accuracy to supervised, contrast-specific CNNs in 5000 scans across six modalities and 10 resolutions. To solve the inference speed challenge to clinical deployment, Pan et al. [40] proposed a consistency model with a Swin Transformer backbone for low-to-full-dose PET enhancement, which achieved an SSIM of 0.964 at one-eighth dose and is 12 times faster than its baselines.

5.4. Summary and Emerging Directions

The diffusion-based architectures (DDPMs, adversarial diffusion hybrids, and consistency models) exhibit significantly better image quality and structural fidelity than GANs across all tasks and come with the added advantage of providing a probabilistic uncertainty quantification. The use of 3D volumetric consistency mechanisms is now part of the standard design criteria, and domain randomization represents a more general paradigm shift from fixed modality mapping to models that can be deployed across different clinical infrastructure settings. The most promising future direction is the integration of large-scale multi-modal pretrained representations with acquisition-physics-aware diffusion priors towards a single cross-modal synthesis system that can cover a wide range of clinical imaging protocols in the real world.

6. Applications in Clinical Practice

3D generative models are becoming a reality and are increasingly entering clinical practice. Generative models have been shown to play a key role in data augmentation for under-represented patient groups, structured radiology education, the construction of cardiac digital twins, and privacy-preserving federated learning across institutions. In this section, the application areas are reviewed in relation to clinical research demonstrating effectiveness and measurable results, summarized in

Table 6. Clinical applications of 3D generative AI.

Ref	Application Domain	Target Modality	Model Type	Key Benefit	Observed Outcome/Result
[41]	Data Augmentation—Fairness and Distribution Shift	Dermatology, chest X-ray, and histopathology	Latent diffusion model (LDM)—conditional generation	Improves classifier fairness for under-represented groups under distribution shift	Dermatology: 63.5% improvement in high-risk sensitivity; 7.5× reduction in fairness gap Chest radiology: 5.2% accuracy gain, 44.6% lower fairness gap
[42]	Data Augmentation—Pulmonary Nodule Detection	CT (chest radiograph)	DDPM (curriculum learning pipeline with synthetic nodule generation)	Improves detection sensitivity for small/low-contrast nodules; addresses class imbalance	AUC 0.95 vs. 0.89 baseline (p < 0.001); sensitivity 70% vs. 48% baseline; accuracy 82% vs. 70%
[43]	Radiology Education—AI-Assisted Graded Training	CT (pulmonary)	AI-assisted diagnosis system (detection + grading pipeline)	Structured graded training of residents using AI feedback on real clinical cases	AI-assisted groups (Groups 2 and 3) outperformed traditional teaching group in nodule sensitivity across all densities and sizes; confirmed over 7 rounds of testing on 1057 nodules
[44]	Cardiac Digital Twins—Electrophysiology at Scale	Cardiac MRI + ECG	Automated mesh generation pipeline + electromechanical simulation	Patient-specific ventricular models enabling electrophysiological insight at population scale	3461 cardiac digital twins from UK Biobank; 359 from ischemic heart disease cohort; sex-specific QRS differences explained by anatomy; conduction velocity changes with age and obesity confirmed
[45]	Cardiac Digital Twins—CRT Treatment Planning	Cardiac MRI (LV/RV)	CircAdapt biomechanical model + imaging-based personalization	Virtual pacing prediction of left ventricular reverse remodeling after cardiac resynchronization therapy	45 heart failure patients; direct correlation between virtual pacing response and actual LV reverse remodeling at follow-up; validated as patient selection tool for CRT
[46]	Cardiovascular Digital Twins—Generative AI Integration	Multi-modal (ECG, Echo, CT, and MRI)	Generative AI for counterfactual scenario simulation; digital twin population modeling	In silico clinical trial population modeling; risk stratification; treatment effect estimation	Framework for using digital twins to generate evidence across clinical trial populations; personalized simulation of cardiovascular scenarios; improved interpretability in HF prognostication
[47]	Federated Learning—Privacy-Preserving MRI Synthesis	MRI (knee—fat suppression)	GAN-based fat-suppressed MRI synthesis with federated training	Improves multi-site generalizability of synthesis model without sharing patient data	Federated GAN outperformed single-site trained GAN on external fastMRI data; demonstrated privacy-preserving multi-institutional synthesis collaboration
[48]	Federated Learning—Fairness via Synthetic Augmentation	Multi-modal (dermatology, radiology, and histopathology)	Conditional diffusion model with federated/privacy-preserving generation	Reduces model bias for under-represented populations without access to centralized patient data	Synthetic augmentation closed fairness gap across institutions; diffusion + FL enables scale-up without privacy violation

.

The risk is not the same across the following application areas, and the taxonomy in this section should be interpreted with this distinction. Retrospective benchmark performance is adequate for lower-risk, primarily off-line applications like data augmentation for training classifiers or radiology education. Intermediate-risk applications, like image reconstruction or modality translation, have a direct impact on the image interpreted by the clinician after the image is created and thus should be subjected to reader-study or downstream task-based validation before they are generally adopted beyond the requirement of voxel-level metrics. For higher-risk, decision critical applications, such as surgical planning and anomaly detection, the consequences of failure can be directly linked to missed or incorrect diagnosis and intraoperative decisions, and so, retrospective benchmark results, e.g., Dice, AUC, or PSNR on held-out test data, should not be presented as equivalent to clinically demonstrated use and utility; this requires prospective or reader study evidence and, eventually, outcome-level validation. The evidence reported in Section 6.1, Section 6.2, Section 6.3 and Section 6.4 should be weighted according to the risk tier that the assessment is for.

The progression of AI in colorectal cancer screening, diagnosis, and treatment seen here, from isolated detection models to integrated, clinically embedded systems that span screening, biomarker discovery, and treatment planning, demonstrates not only the potential clinical impact of the technology but also the typical challenges of translation that are highlighted in this survey across 3D generative imaging (data privacy, robustness, and evaluation standardization).

6.1. Data Augmentation for Fairness and Rare Disease

Typical augmentors are not able to produce new phenotypic variations: only generative augmentors can create new examples from under-represented domains. Ktena et al. [41] showed that conditional latent diffusion models further enhanced the fairness of classifiers in dermatology, chest radiology, and histopathology, improving high-risk skin tone sensitivity by 63.5% and decreasing the fairness gap in chest radiology by 44.6%. In another study, a DDPM-based pipeline was used to address the class imbalance problem in pulmonary nodule classification, improving the mean AUC from 0.89 to 0.95 and mean sensitivity from 48% to 70% on hybrid real–synthetic data on which the curriculum is learned.

6.2. Radiology Education and AI-Assisted Training

Generative models are used to synthesize training cases on demand with controllable parameters such as pathologies, severity, size, and location that are not possible in a fixed retrospective archive. In a prospective study at Jinzhou Medical University, Lyu et al. [43] found that the performance of using AI-generated graded training examples from 1057 nodules from 420 cases was consistently superior across each nodule density and nodule size category compared to the traditional teaching, thus offering direct empirical evidence that the use of generative augmentation significantly boosts trainee diagnostic performance.

6.3. Cardiac Digital Twins and Personalized Simulation

The most advanced clinical use of 3D generative imaging is in the development of cardiac digital twins: patient-specific computational models of the heart, built from imaging data. However, Strocchi et al. [44] built 3461 twins based on MRI data from the UK Biobank, finding that the electrophysiology differences between the sexes and age groups were not detectable by the typical ECG interpretation. Clinically, imaging-based digital twins of 45 HF patients [45] predicted LV response to CRT preoperatively, and Thangaraj et al. [46] formalized the use of generative AI to synthesize missing modalities, create in silico trial populations, and estimate cross-population treatment effects.

6.4. Federated Learning and Privacy-Preserving Synthesis

HIPAA and GDPR are the primary limitations of clinical AI. Kulkarni et al. [47] showed that federating GAN training across two sites improves external generalization without requiring access to the data. The FMed-Diffusion framework [48] further introduced federated diffusion models, showing the potential of using the diffusion process’s inherent Gaussian noise as a natural differential privacy mechanism and bringing federated diffusion models into federated infrastructure for multi-institutional clinical AI development without cross-site data exposure.

7. Evaluation Metrics and Benchmarks

Assessing generative models for 3D medical imaging requires a multi-level framework that reflects voxel-level fidelity, anatomical realism, and downstream clinical utility. No single metric exists; instead, the field has coalesced around a hierarchical assessment process that includes quantitative image quality metrics, expert radiologist assessment, and task-based, functional evaluation. The evaluation landscape is organized in

Table 7. Quantitative evaluation metrics for 3D generative medical imaging models.

Metric	Full Name	Description	Primary Use Case	Known Limitations	Dimensionality/Computation
PSNR	Peak Signal-to-Noise Ratio	Log-ratio (dB) of the maximum possible pixel value to the mean squared error between generated and reference volumes, computed voxel-wise.	CT/MRI reconstruction quality; low-dose CT denoising; accelerated MRI benchmarking.	No perceptual or structural sensitivity; favors over-smoothed outputs; insensitive to clinically relevant local distortions. Sensitive to global intensity offsets.	Computed per 2D slice in most reviewed studies, then averaged across slices; native whole-volume (3D) PSNR is possible but rarely reported explicitly.
SSIM	Structural Similarity Index Measure	Local window-based index combining luminance, contrast, and structural comparison between two images; values in [−1, 1], higher is better.	Accelerated MRI reconstruction; anatomy-preserving synthesis; cross-modal image translation evaluation.	Window-size sensitive; poor at capturing global geometry; high SSIM images can still fail clinical assessments; poor correlation with expert radiologist scores in several studies.	Typically computed with a 2D sliding window per slice, then averaged across slices; a volumetric (3D) sliding-window SSIM exists but is used less often in the reviewed literature.
NRMSE	Normalized Root Mean Squared Error	Root mean squared voxel-wise intensity deviation normalized by the range or mean of the reference volume; lower is better.	Quantitative MRI reconstruction; k-space recovery evaluation; HU-accuracy in CT synthesis.	Sensitive to outlier voxels; dominated by high-intensity structures (bone in CT, fat in MRI); not invariant to global intensity scale; no spatial structural information.	Computed voxel-wise over the full 3D volume directly—inherently volume-native since it is a simple normalized error rather than a windowed statistic.
MAE/MSE	Mean Absolute/Mean Squared Error	Average absolute or squared voxel intensity difference between generated and reference volumes; both are lower-is-better scalar error summaries.	Denoising; SUV accuracy in PET synthesis; HU accuracy in synthetic CT for radiotherapy planning.	Entirely insensitive to spatial structure and anatomical coherence; MSE penalises large errors disproportionately; neither reflects perceptual or clinical image quality.	Voxel-wise and natively 3D by construction; any slice-wise reporting is a display convenience and does not change the underlying computation.
LPIPS	Learned Perceptual Image Patch Similarity	Feature-space distance between image patches using activations from a deep CNN (AlexNet or VGG) trained on natural images; lower is better.	Perceptual quality assessment of MRI/CT synthesis; diffusion model output evaluation; complements SSIM/PSNR in paired evaluation.	Pretrained on RGB natural images—limited validation for grayscale volumetric medical data; 3D extension not standardized; sensitive to domain shift between natural and medical image distributions.	Inherently 2D: computed slice by slice with a 2D CNN backbone (AlexNet/VGG) and averaged across slices; no native 3D LPIPS backbone is used in the reviewed literature (Section 7.2).
FID/FID-3D	Fréchet Inception Distance (3D adapted)	Fréchet distance between Gaussian distributions fitted to deep feature embeddings of real vs. generated volume sets; lower indicates closer distributional alignment.	Diversity and realism assessment of unconditional 3D generation; GAN/diffusion model comparison on brain MRI and chest CT.	Requires large sample sizes (≥2000 recommended); Inception-v3 trained on ImageNet—applicability to medical images debated; no standardized 3D feature extractor; FID-optimal models can still produce clinically incorrect images.	Standard FID uses 2D slices through a 2D Inception network; FID-3D uses a 3D feature extractor on the whole volume, but the extractor is not standardized across studies (Section 7.2).
Dice/IoU	Dice Similarity Coefficient/Intersection over Union	Overlap-based metrics applied to binary segmentation masks of anatomical structures in synthetic vs. real volumes; Dice = 2|A∩B|/(|A| + |B|).	Downstream segmentation task evaluation; assessing whether synthetic data augmentation improves segmentation model performance.	Requires segmentation ground truth; cannot directly assess synthesis quality; sensitive to class imbalance; insensitive to boundary sharpness.	Computed natively in 3D over the full segmentation volume in most reviewed studies, though 2D per-slice Dice is sometimes reported separately for slice-wise comparison.
HD95	Hausdorff Distance (95th percentile)	95th percentile of the maximum one-sided surface distance between predicted and reference segmentation contours; reported in mm; lower is better. Uses the 95th percentile to reduce sensitivity to extreme outliers present in the full HD.	Organ boundary fidelity in synthetic CT for radiotherapy; shape accuracy in surgical planning evaluation.	Still influenced by surface outliers beyond the 95th percentile; implementation-dependent (voxel spacing, mesh vs. voxel computation); does not encode volumetric overlap; requires segmentation masks.	Computed over the full 3D surface point set in most reviewed studies; some 2D contour-based variants exist for slice-wise comparison.
ASSD	Average Symmetric Surface Distance	Mean of all bidirectional nearest-surface distances between two segmentation contours; symmetric and lower is better; complements HD95 by averaging rather than taking an extreme value.	Organ shape fidelity assessment; complementary to HD95 in synthetic CT and MRI evaluation pipelines.	Can mask localized large errors by averaging; also requires segmentation masks; sensitive to the segmentation method used for evaluation.	Computed over the full 3D surface mesh/point set in most reviewed studies.

and

Table 8. Public benchmark datasets for 3D generative medical imaging evaluation.

Ref	Dataset Name	Size	Modality	Anatomy/Task	Access/Homepage
[49]	BraTS 2021	1251 training/5 validation/219 testing volumes	MRI (T1, T1ce, T2, and FLAIR)	Brain tumor segmentation and multi-modal MRI synthesis	www.med.upenn.edu/cbica/brats2021/ (accessed on 3 August 2026)
[50]	BraSyn 2023	Based on BraTS 2021—challenge-defined splits	MRI (T1, T1ce, T2, and FLAIR)	Brain MR image synthesis for missing modalities	www.med.upenn.edu/cbica/brats/ (BraTS challenge series, Synapse portal) (accessed on 3 August 2026)
[51]	LIDC-IDRI	1018 cases; 4-radiologist annotations; 7 academic centers + 8 companies	CT (thoracic)	Lung nodule detection and segmentation and CT synthesis	www.cancerimagingarchive.net/collection/lidc-idri/ (accessed on 3 August 2026)
[52]	fastMRI	~8344 raw k-space volumes (1594 knee + 6970 brain); DICOM set adds 20,000+	MRI (knee and brain)	Accelerated MRI reconstruction and k-space synthesis	https://fastmri.med.nyu.edu/ (accessed on 3 August 2026)
[53]	CHAOS	80 patients: 40 CT + 40 MRI; 120 MRI DICOM series	CT + MRI (T1-DUAL in/opp-phase, T2-SPIR)	Healthy abdominal organ segmentation; CT↔MRI translation	https://chaos.grand-challenge.org/ (accessed on 3 August 2026)
[54]	ACDC	150 patients; 100 training/50 testing; 5 diagnostic classes	Cardiac cine MRI	Cardiac segmentation (LV, RV, and myocardium) and diagnosis	www.creatis.insa-lyon.fr/Challenge/acdc/ (accessed on 3 August 2026)
[55]	ADNI (1–4)	>2000 subjects cumulative; longitudinal, multi-timepoint	MRI (T1 and 3T) + PET (FDG, amyloid, and tau)	Alzheimer’s disease progression; PET synthesis from MRI	https://adni.loni.usc.edu/ (accessed on 3 August 2026)
[56]	AAPM Low-Dose CT	299 scans: 49 head, 100 chest, 100 abdomen; full-dose + 25%/10% low-dose pairs	CT (head, chest, and abdomen)	Low-dose CT denoising and dose-reduction synthesis	www.aapm.org/GrandChallenge/LowDoseCT/ (accessed on 3 August 2026)
[57]	HECKTOR 2022	883 cases: 524 training (7 centers) + 359 test (3 centers)	FDG-PET + CT	Head and neck tumor segmentation; multi-modal PET/CT synthesis	https://hecktor.grand-challenge.org/ (accessed on 3 August 2026)
[58]	crossMoDA 2021–23	2021: 227 ceT1 + 295 hrT2; 2023: ~560 training subjects	MRI (ceT1 → hrT2)	Vestibular schwannoma and cochlea segmentation; unpaired cross-modal synthesis	https://crossmoda-challenge.ml/ (accessed on 3 August 2026)

.

In this framework, we explicitly identify eight dimensions of evaluation, which are often combined in previous surveys: pixel-level fidelity (PSNR, NRMSE, and MAE/MSE), perceptual quality (LPIPS), anatomical consistency (Dice/IoU, HD95, and ASSD), pathological preservation (lesion-level detection and measurement agreement before and after generation, distinct from whole-volume similarity), quantitative accuracy (modality-specific measures such as HU error for Computed Tomography and SUV error for Positron Emission Tomography, rather than generic intensity similarity), predictive uncertainty (posterior sampling variance or ensemble disagreement used to flag low-confidence regions), diversity (FID/FID-3D and precision–recall for generative coverage), and downstream clinical utility (task-based performance when synthetic data is applied to a diagnostic, planning, or training task). Under the three-tier protocol, reader studies (scoring realism, artifact presence, and diagnostic confidence by the blinded radiologist) and task-based evaluations (training or testing downstream models on synthetic data) are considered a stratum of evidence above voxel-level metrics.

The sizes of ten datasets are provided with verified sizes from primary publications or official challenge repositories. BraSyn 2023 is separated from BraTS 2021, as it is a new synthesis challenge track. CHAOS has a complete 80-patient count (40 CT + 40 MRI) and a clarified MRI series count. fastMRI is corrected to include all raw k-space volumes (1594 knee and 6970 brain, ~8344 total). The new crossMoDA 2021-23 benchmark was introduced and is not included in many previous surveys.

7.1. Unified Evaluation Protocol

To properly compare and contrast 3D generative models, we suggest a three-level evaluation process. Voxel-Level Evaluation (1st tier): PSNR, SSIM, NRMSE, and FID-3D are computed on held-out test volumes not encountered during training (standardized preprocessing: HU windowing for CT and intensity normalization for MRI). The second tier, Expert Clinical Evaluation, consists of blinded radiologist evaluation of generated volumes using structured scoring rubrics for anatomical realism, artifact presence, and diagnostic utility, with at least two independent radiologists per case to estimate inter-rater reliability. The third tier, Task-Driven Assessment, focuses on the clinical utility of the generated data by training or testing downstream segmentation, classification, or registration models on synthetic data and reporting task-specific metrics (Dice, AUC, and TRE). Only models that perform better in all three tiers will be considered clinically relevant for a particular application, and only voxel-level metrics, without clinical or task-level validation, should be considered as weak evidence for any translational claims.

In addition to the three tiers discussed above, two additional axes are suggested, and not necessarily replacements: pathological preservation testing, in which lesion-level measurements are compared before and after generation/reconstruction to identify suppression, fabrication, distortions of clinically relevant findings, and uncertainty quantification, reporting the posterior sampling variance or ensemble disagreement in addition to point-estimate measurements to flag low-confidence regions for human review before clinical use.

7.2. Limitations of Current Evaluation Metrics and Standardization Recommendations

Because current evaluation metrics have limitations, standardization is recommended: a summary of limitations of current evaluation metrics and recommended standardizations. However, non-standardization in evaluation practice stems not only from inconsistent practice but mostly from the intrinsic limitations of the metrics themselves. When applied to 3D medical volumes, LPIPS uses a deep feature representation learned from a network trained on natural RGB images (e.g., AlexNet or VGG on ImageNet) that are not modality-specific (such as HU-based tissue boundaries in CT or SUV-based uptake patterns in PET). Perceptual similarity scores calculated this way have not been correlated with clinically relevant differences and are usually calculated slice-wise on a 3D medical volume rather than directly in 3D. FID-3D faces a similar issue: the lack of a universal 3D feature extractor equivalent to the Inception network employed in 2D FID, meaning that various studies use different backbones (such as 3D ResNet variants trained on different source datasets), and thus, FID-3D scores are not comparable across papers, even if the same generative model is tested. Although conceptually simpler, PSNR and SSIM are sensitive to preprocessing conventions that are commonly not reported: namely, the intensity normalization convention (min–max vs. z-score), the dynamic range assumed in the PSNR denominator, and the HU windowing in the CT.

To move towards standardized evaluation protocols, the following suggestions are proposed for future studies: (i) report exactly how the intensity is normalized to calculate PSNR/SSIM, ideally together with HU windowing parameters for CT and SUV normalization for PET, to be reproducible from the paper alone; (ii) specify the 3D feature extractor, its pretraining source, and its checkpoint when reporting the FID-3D score, using a feature extractor pretrained on medical volumes when possible and avoiding the use of natural-image backbones as a primary fidelity metric, using it instead as a secondary non-clinical perceptual signal, together with modality-specific quantitative metrics (e.g., HU error for CT and SUV error for PET); and (iii) use a shared benchmark, such as the 3D-MedGenBench proposal, fixing preprocessing, feature extractors, and train/test splits across submissions.

8. Ethical Considerations and Limitations

When generative AI is applied to clinical imaging, it raises a host of ethical issues, such as the following: Where do the images come from? How will the models be constructed? How can it be regulated? What happens if it goes wrong? These concerns are not academic; they are real and have direct implications for patient safety and healthcare equity.

8.1. Data Provenance and Privacy

Generative models, trained on patient imaging information, might memorize and regenerate identifiable information, especially for rare disease cases where training data is very sparse [45]. Under GDPR and HIPAA, membership inference attacks, in which an adversary infers whether a certain patient’s information was used to train a model, are possible when performing attacks on medical imaging generative models. While the differential privacy (DP) training mechanisms that add calibrated noise to the gradients while training to guarantee formal privacy have come with a cost of a 0.3–0.8% drop in image quality metrics, the suitable privacy budget (ε) for medical imaging applications remains not standardized [46]. While models do not need to be shared, federated learning methods, which train models locally on institution data without sharing raw images, also provide complementary privacy protection and should be managed carefully to avoid gradient leakage.

One such attack category is membership-inference risk, a concept that refers to a larger class of memorization-based privacy risks that are more complex than a single attack category and need to be treated more seriously. The diffusion- and GAN-based generators can record and reproduce close copies of the individual training images, especially when a very rare disease or a small number of images of such a disease are present in a given institution, such that many similar images do not contaminate a single specimen; this can be tested directly by nearest-neighbor analysis, in which generated volumes are compared against the training set to identify suspiciously similar images. Training-data reconstruction attacks are more serious than membership inference attacks: they aim to learn recognizable image content from a trained model rather than infer membership, and they have been shown to work on medical image generators. The risk of patient-level data leakage is not limited to the image content itself; it also includes other patient-level information associated with an imaging scan (such as the scan time, associated anatomy, or other metadata). Together, these risks show that an image’s synthetic origin is not necessarily a sign of anonymity: memorization, reconstruction, and re-identification risks should be actively tested for (e.g., through nearest-neighbor distance thresholds and membership-inference audits) before sharing or releasing synthetic data, not assumed to be absent.

8.2. Bias and Fairness

Medical imaging datasets are not demographically representative, with certain ethnic groups, age groups, and disease phenotypes over-represented and others under-represented. However, generative models that are trained using biased data will reinforce these biases in their synthetic data [47]. For 3D medical imaging, this can be seen as models that create anatomically realistic volumes (e.g., an adult male brain MRI) but anatomically unrealistic or artifactual volumes (e.g., a pediatric brain MRI and an elderly female pelvis MRI). Stratified analysis of generation quality should therefore be an integral component of fairness evaluation and should be a design goal and not an afterthought.

8.3. Anatomical Validity and Clinical Risk

A very serious limitation of generative models is that they can generate implausible outputs that are visually convincing to laymen but not to medical experts and contain pathologically incorrect structures, especially for a clinical decision-support system. A 3D MRI model of a generation could produce a tumor with an abnormal enhancement pattern, and a CT model of the generation could produce a pulmonary nodule with an abnormal morphology in a healthy patient’s synthetic scan. There are several components to inclusion in validation frameworks: explicit anatomical consistency checks, validation by a radiologist of edge cases, and out-of-distribution detection mechanisms that provide a high uncertainty score for generated volumes that are flagged for human review prior to clinical use [48]. AI systems used in clinical decision making will need to be pre-market-validated, and regulatory clearance may be required for generative models that will be used in the downstream diagnostic use of AI, as these will likely be classified as AI-enabled medical devices.

8.4. Hallucination and Misrepresentation

Factual hallucination is a similar occurrence in LLM that is referred to as hallucination in generative medical imaging [49]. In generative medical imaging, hallucination is the synthesis of structures that do not correspond to any anatomical reality. In MRI synthesis, it could be in the form of false white spots, in CT in the form of false calcifications, and in PET in the form of false, hypermetabolic foci, which may be interpreted as malignant. These hallucinations are especially risky when they are used to train the downstream models because the classification model could learn to detect these hallucinated features and draw spurious clinical conclusions. Mechanisms for uncertainty quantification, such as sampling from a diffusion model posterior, can be used to highlight areas of high generation uncertainty and should be used routinely prior to clinical use of a synthetic volume.

In addition to the fabrication of improbable structures, hallucination in this context also entails alteration of pathology: an anatomically realistic synthesized volume may remove an existing lesion, alter the boundary of a tumor, or alter a clinically important measurement, such as the diameter, volume, or enhancement pattern, without causing any visibly improbable artifact, which would make those errors much more difficult to detect than gross anatomical fabrication. We suggest that evaluation procedures for clinically oriented generative models should incorporate four complementary safeguards: (i) pathology-preservation tests comparing the results of lesion-level detections and measurements before and after generation/reconstruction on a held-out set of pathological cases; (ii) explicit anatomical constraints, explicitly enforced architecturally or as a post-hoc plausibility filter, that reduce the chances of anatomically implausible structures; (iii) uncertainty estimation, reported with point-estimate results, to enable low-confidence regions to be flagged for human review rather than silently accepted; and (iv) structured failure-case analysis, where representative failure modes are catalogued and reported, rather than only aggregate performance, so that clinicians and downstream developers can understand where and how a given model is likely to fail.

Examples of factual hallucination in the three modalities are a diffusion- or GAN-based MRI or CT synthesis model where a synthetic tumor or nodule is created with no corresponding anatomical structure in the underlying tissue, a PET generation or denoising model that introduces a false hypermetabolic focus (i.e., spurious uptake of tracer) that mimics malignancy, a reconstruction model that leaves out or obfuscates a genuine but subtle anatomical structure (e.g., a small vessel, a thin cortical rim, or a small lesion) below the threshold at which the change becomes visually obvious, and a cross-modality or super-resolution model that generates a pathological pattern with altered morphology (e.g., changing the margin or enhancement characteristics of a lesion) that alters the apparent malignancy of that anatomical structure. A fabricated or altered lesion can fool a radiologist into an incorrect diagnosis or management, and an omitted or suppressed finding can lead to a missed diagnosis; both of these errors can be exacerbated in the case of a model trained with the hallucinated volume, which will learn to produce the hallucinated pattern as a generalizable feature, whether it is recorded or not. In addition to the uncertainty estimation and anatomical-constraint safeguards described above, we highlight two other complementary mitigations: expert (radiologist) validation of generated volumes on a structured rubric before any downstream clinical or research use, especially for edge cases and rare pathologies, which are more likely to be missed by the non-expert and more likely to be hallucinated; and multi-modal verification, in which a finding suggested by a generated or reconstructed volume in one modality (e.g., a PET uptake focus) is cross-checked against an independently acquired or unmodified reference in another modality (e.g., the corresponding anatomical CT or MRI) before it is acted upon clinically or propagated to a downstream model.

8.5. Governance, Regulatory Classification, and Post-Deployment Oversight

While other aspects of the discussion of ethics from above (Section 8.1, Section 8.2, Section 8.3 and Section 8.4) are not explicitly covered above, several of the governance dimensions raised in review are addressed here. In addition to demographic bias (Section 8.2), there is also institutional bias: models developed from a few high-resource academic centers inherit the acquisition protocol, patient case mix, and reporting conventions of those centers, which can result in poor performance when applied to institutions using a different scanner vendor, field strength, or reconstruction kernel, even if the demographic compositions are similar. This is exacerbated by the scanner-domain shift: It is not guaranteed that the same anatomy scan performed with the same sequence parameters, or even reconstruction pipeline, will be present in the scanner training distribution, and generative models may normalize away clinically meaningful differences in signal (e.g., lesion contrast) that exist between different sites. If this risk is reduced by federated and domain-randomized training, it does not disappear either, and we suggest that any deployment report additionally be split by acquisition site and scanner model, apart from the split by demographic subgroups.

A second open question is consent for generative reuse, as imaging acquired to serve a specific clinical purpose is often used in downstream applications for generative modeling, with no explicit consent process for those uses, and consent forms are not often designed with the intention of downstream uses of a patient’s images to generate new images of unseen patients. We suggest that consent for data use in generative training be explicitly separated from consent for data use in discriminative training and that de-identification pipelines be re-validated to account for memorization and regurgitation of rare-case anatomy (Section 8.1). Traceability of synthetic data is closely related: after a synthetic volume is added to a training set, it is very hard to differentiate between real and synthetic provenance further downstream of the synthetic volume with a secondary model trained on the synthetic volume. We recommend that machine readable provenance metadata (such as a cryptographic or digital watermark and a provenance field within the DICOM header) be included with each volume generated, along with a logged link back to the version of the model used to generate the volume and the snapshot of training data used, so that synthetic content can be audited and, if a fault is found once the volume is generated, be retractable and purged.

Whether the above safeguards are actually enforced depends on human oversight, regulatory classification, and cyber security. Within a framework of risk like the EU AI Act or the FDA’s SaMD pathway, generative models that are only employed for augmentation, education or research would be classified in a lower risk tier than models whose output is used to guide diagnosis or treatment planning; the latter would therefore be classified and regulated as clinical decision-support software, where outputs would require pre-market evidence, required human sign-off prior to clinical use, and a formal change-control plan for updates of the model. A one-off perturbation of data could lead to adherence to a specific, unnatural specimen, while data poisoning of federated updates, or a corrupted model checkpoint, could each at scale introduce systematic hard-to-detect anatomical errors at scale, with deployed systems to be integrity-checked (e.g., signature verification of model weights or a checksum) and monitored for anomalous output distributions in a similar fashion to medical devices and firmware tampering. Contractual and procedural responsibility for any error generated should be defined prior to deployment: we recommend a shared-responsibility model where the developer would be responsible for ensuring that the AI model is performing as intended within the operating envelope defined by the model, the deploying institution would be responsible for the correct use of the model at their site, and the clinical practitioner who reviews the output would be responsible for any decision made as a result of the model’s output, in line with current guidance from regulators that the output from an AI model should be treated as decision support and not a diagnosis. Last, post-deployment monitoring should include measurement of output-quality drift (e.g., rolling FID/SSIM against a held-out reference set), the number of times clinicians override or reject the output, and reports of adverse events, all with clearly defined thresholds that trigger retraining, restricted use, or withdrawal, similar to the post-market surveillance requirements of other AI-enabled medical devices.

8.6. An Operational Deployment Blueprint

The above considerations justify a concrete, staged blueprint, more than just a conceptualized discussion.

Table 9. Operational deployment blueprint for 3D generative medical imaging systems.

Stage	What Happens	Minimum Evidence—Research Only	Minimum Evidence—Augmentation/Education	Minimum Evidence—Clinical Decision Support
1. Data governance	Define data sources, consent scope, de-identification, and retention policy	Documented data-use agreement	Explicit consent for generative reuse; documented provenance	Explicit consent for generative reuse; IRB/ethics approval; documented chain of custody
2. Model development	Architecture selection, training, and hyperparameter search	Version-controlled code	Version-controlled code; training-data manifest	Version-controlled code; training-data manifest; predetermined change-control plan
3. Technical validation	Voxel/structural fidelity metrics on held-out internal data	PSNR/SSIM/FID on internal split	PSNR/SSIM/FID plus downstream task metric (e.g., Dice)	Full Table 5 metric panel plus uncertainty quantification
4. External validation	Evaluation on data from sites/scanners not seen in training	Not required	At least one external dataset or site	Multi-site, multi-vendor external validation with stratified reporting (Section 8.5)
5. Clinical evaluation	Expert reader study/clinical-utility assessment	Not required	Reader study on downstream task performance	Blinded multi-reader clinical-equivalence or non-inferiority study
6. Risk assessment	Formal identification of failure modes and their clinical consequence	Informal	Documented failure-mode list	Formal risk file mapped to regulatory risk class (Section 8.5)
7. Human review	Definition of the human-in-the-loop checkpoint before use	N/A	Recommended at point of downstream model deployment	Mandatory clinician sign-off before use in patient care
8. Deployment	Integration into clinical/research workflow	N/A	Institutional approval	Regulatory clearance/CE marking or equivalent; integrity-checked model deployment (Section 8.5)
9. Continuous monitoring	Post-deployment surveillance of performance and safety	N/A	Periodic re-evaluation against Stage 3 metrics	Continuous drift monitoring, override/rejection tracking, and adverse-event reporting (Section 8.5)

outlines nine stages that we believe we should expect any 3D generative imaging system to traverse both prior to deployment and after deployment, alongside the minimum evidence we feel should be present at each stage, broken down into three application-risk categories: (i) research-only/method development, where the output of the 3D generative system is never used as a clinical image (e.g., synthetic CT for dose calculation and low-dose PET/CT denoising); (ii) augmentation and education, where the output of the 3D generative system is used to train or assess a separate downstream 3D imaging system or a trainee, but the output itself is not considered to be a patient’s clinical image; and (iii) clinical decision support, where the output of the 3D generative system is used directly for clinical decision making (diagnosis, planning, or treatment) (e.g., synthetic CT for dose calculation and low-dose PET/CT denoising). There are the strictest evidence requirements throughout for category (iii). This staged structure should be used to validate at each institutional deployment; it is not a one-off validation and should be re-entered at Stage 3 using the more demanding evidence column.

This staged structure is intended to be audited stage by stage at each institutional deployment, rather than treated as a one-time certification: a model validated for Category (ii) use that is later repurposed for Category (iii) decision support must re-enter the blueprint at Stage 3 with the stricter evidence column applied.

8.7. Reproducibility and the Gap Between Research Performance and Clinical Deployment

The metrics mentioned in Section 7 characterize the performance in the research setting only and are not sufficient to qualify a model for clinical use. In this subsection, we bring together, in one spot, some of the factors that lie between a desirable benchmark result and a clinically usable system, some of which are discussed individually in the rest of this survey.

• External validation. Only a few of the 16 landmark architectures studied have been reported with external validation; most reported external validation data originate from a single institution’s held-out test set and not from an independent test cohort. Any model that works well on a set of data acquired from the same acquisition pipeline as the training data will only have weak evidence of generalization.
• Cross-institutional generalization. External validation is related but different from the question of whether a model trained at one or a few sites will transfer to institutions with patient populations, referral patterns, and clinical protocols different from those used for training. Federated approaches (Section 6.4) partially overcome the data sharing barrier to multi-institutional evaluation, but federated training is in itself does not ensure the model’s equal performance on all of the institutions on which it is federated, nor for unseen institutions.
• Scanner and protocol variability. Report acquisition variability by modality (MRI sequence and field-strength dependence, CT reconstruction kernel and dose protocol, and PET tracer and scanner model variations); Section 8.2 also covers scanner-domain shift as another source of demographic and institutional bias. Do not assume that a model that is validated on one scanner manufacturer’s protocol will transfer to another without re-validation.
• Computational requirements. In Section 9.1, we will discuss the real-time inference bottleneck of iterative diffusion sampling, while the taxonomy discussion in Section 3.3 highlights the fact that latent diffusion models take a lot of data, denoising iterations, and compute to use. This is a different constraint apart from model performance: deployability on standard clinical hardware, not research grade GPU clusters.
• Data privacy. In Section 8.1, we cover the issue of memorization, membership-inference and training-data-reconstruction risk in depth and ultimately come to the conclusion that the synthetic origin of an image is not a guarantee of anonymity. These risks are applicable to any deployment pipeline that shares or releases model output or weights and should be explicitly tested for in the deployment pipeline.
• Expert assessment. Blinded expert (radiologist) evaluation is preferred over voxel-level metrics according to the three-tier evaluation protocol used in Section 7.1, and expert/human review is listed as a necessary step before operational deployment in the operational deployment blueprint in Section 8.6. Voxel-level metrics should only be considered as weak evidence for any translational claim without at least this level of evidence.

Combined, the six factors characterize the difference between a model that works on a published benchmark and one that is ready for clinical use. We do not suggest a single composite evaluation of “deployment readiness” because the relative importance of the criteria varies by risk level of the use case; a data-augmentation use case does not require as robust an external evaluation as a decision-support use case does. Rather, we advocate that studies that have proposed clinically oriented generative models report the status of each of the six factors above in conjunction with their headline performance metrics to encourage readers to assess the translational readiness directly and not infer it from image-quality measures alone, as in the past.

9. Open Challenges and Future Directions

Despite remarkable progress, significant challenges remain at the intersection of generative AI and 3D medical imaging. We identify six priority research directions.

9.1. Computational Efficiency and Real-Time Inference

The field of 3D medical imaging and generative AI has made great strides, but there are still hurdles to be addressed. We list six research directions of priority. Computational efficiency and real-time inference are discussed in 9.1. Even though diffusion models are powerful, the iterative denoising in this process involves tens to thousands of forward passes through a large neural network to produce a generated sample and is therefore presently not possible in real time on standard clinical hardware (essential for intraoperative imaging and dynamic planning). Consistency models [50] are a current research area, as are accelerated samplers based on distillation, and hardware-optimized implementations (GPU kernel fusion and quantization). For NeRF-based approaches, hash-grid scene representations accelerate inference by three orders of magnitude compared to vanilla MLPs and have not been explored in the medical imaging context of non-uniform sampling density.

9.2. Physics-Informed Generation

Current generative models are mainly data-driven and do not encode modality-specific acquisition physics as inductive bias. If MRI generation does not adhere to k-space symmetry constraints, the generated images may appear plausible but not physically consistent, which can lead to systematic errors in downstream quantitative analysis tasks. An important research direction is to incorporate Bloch equations (MRI) or the Beer–Lambert law (CT attenuation) as architectural constraints in physics-informed neural network approaches to ensure the physical plausibility of generated volumes [51].

9.3. Rare Disease and Long-Tail Learning

Disease prevalence in medical imaging is often long-tailed, so models are typically not trained on the rare and clinically most relevant diseases. Few-shot and zero-shot generative frameworks that can create realistic rare-disease volumes from just a handful of examples (1–5 patients) are fundamental to enabling equitable use of AI across the full range of diseases. In the computer vision community, promising paths forward include meta-learning approaches and large-scale pretraining on a wide variety of imaging data, much like foundation models in the natural language processing community [52].

9.4. Standardization of Benchmarks and Metrics

The lack of standardized benchmarks (with fixed train/test splits, preprocessing pipelines, etc.) does not allow for direct comparison across papers. FID-3D is not standardized (different papers use different 3D feature extractors), and PSNR/SSIM is computed using different intensity-normalization conventions across papers; thus, comparisons across publications are unreliable. A 3D Medical Imaging Generation Benchmark (3D-MedGenBench) with fixed protocols, leaderboards, and multi-tier evaluation using radiologist scoring [53] would benefit the community.

9.5. Foundation Models for Medical Imaging

The next step in medical image generation is large-scale foundation models trained on a wide range of multi-modal and multi-site imaging data, similar to GPT-4 for language and CLIP for vision. These models could facilitate zero-shot creation of new modalities, cross-institutional generalization for research and education without fine-tuning, and natural language-conditioned image synthesis (“generate a T2-weighted brain MRI with a 2 cm glioblastoma in the right frontal lobe”) that would make synthetic imaging more accessible for research and education [54].

9.6. Interpretability and Explainability

Generating volumes for clinical use requires comprehensible explanations of the generated features, their motivations, and the level of confidence to ensure the information is intelligible. Learning latent dimensions corresponding to anatomy, pathology, modality, and demographics through latent-space disentanglement enables principled control over synthesis and interpretable latent arithmetic (e.g., “add 2 years of aging to this brain”). Attention visualization can be used in transformer-based generation models to emphasize the areas of the image important to synthesis. To validate these interpretability tools and ensure they provide clinically relevant insights rather than merely reflecting model architecture, radiologists must evaluate them [55].

The reliable adoption challenge is not exclusive to 3D medical imaging: the explainable AI challenge for neurology also claims that the reliability of output produced by models to facilitate diagnosis, monitoring, and planning needs to be accompanied by an explanation of the reason for the output, before clinicians will trust them for patient-specific management [59,60]. The same applies to generative volumetric imaging, which produces anatomically plausible but unexplained synthetic structures, and to the interpretation methods discussed in this section (attention/saliency overlays, uncertainty maps, and counterfactual generation), which function like XAI in discriminative diagnostic models.

10. Conclusions

This survey offers a comprehensive, structured overview of generative AI techniques for 3D medical imaging, covering architectural taxonomy, modality-specific advances, cross-modal synthesis, real-world clinical applications, evaluation frameworks, and ethical considerations. The main takeaway is that the field is undergoing a paradigm shift: Voxel-based GANs and VAEs prominent in early research have been replaced by latent diffusion models that offer stronger generation, greater conditioning flexibility, and principled uncertainty quantification at scale. The change is more prominent for the unconditional and conditional volumetric generation tasks but not for all tasks and corresponds to significantly increased sampling latency, memory usage, and training data requirements, which are still key challenges for real-time or resource-limited clinical use of GAN/VAEs. Implicit neural representations play a complementary role, showing great promise in memory-efficient sparse-view reconstruction and continuous shape modeling. Across all three modalities (MRI, CT, and PET), generative models have shown clinically meaningful improvements in image quality under dose-constrained or accelerated acquisition protocols in both quantitative and radiologist-blinded evaluation studies. However, many challenges in computational efficiency, physical plausibility, fairness, privacy, and regulatory compliance remain before research demonstrations can translate into clinical deployment. The three-step evaluation approach proposed—voxel-level metrics, expert radiologist scoring, and task-driven assessment—offers a practical tool that the community could use to validate and compare generative systems more rigorously. We argue that the most immediate research paths are (i) physics-informed diffusion models that honor the conditions of modality acquisition; (ii) few-shot generation frameworks for rare pathologies; (iii) community benchmarks for reproducible evaluation of generation tasks; (iv) mechanisms for training in privacy while maintaining formal guarantees; and (v) explainable generation interfaces to allow for clinician-in-the-loop validation. Overcoming these obstacles can unlock the transformative power of generative AI for higher-quality diagnostic imaging, reduced patient burden, precision medicine, and access to a wider variety of high-quality medical imaging data worldwide.

This memory efficiency is a property of the representation only. It should be weighed against the substantially higher per-scan training cost, rendering latency, and sensitivity to acquisition geometry relative to feed-forward voxel- or latent-space alternatives, together with comparatively limited validation on pathological or highly heterogeneous anatomy.