Deep Learning for Brain MRI Tissue and Structure Segmentation: A Comprehensive Review

Abstract

Brain MRI segmentation plays a crucial role in neuroimaging studies and clinical trials by enabling the precise localization and quantification of brain tissues and structures. The advent of deep learning has transformed the field, offering accurate and fast tools for MRI segmentation. Nevertheless, several challenges limit the widespread applicability of these methods in practice. In this systematic review, we provide a comprehensive analysis of developments in deep learning-based segmentation of brain MRI in adults, segmenting the brain into tissues, structures, and regions of interest. We explore the key model factors influencing segmentation performance, including architectural design, choice of input size and model dimensionality, and generalization strategies. Furthermore, we address validation practices, which are particularly important given the scarcity of manual annotations, and identify the limitations of current methodologies. We present an extensive compilation of existing segmentation works and highlight the emerging trends and key results. Finally, we discuss the challenges and potential future directions in the field.

1. Introduction

Magnetic Resonance Imaging (MRI) is a non-invasive imaging modality that enables the detailed visualization of anatomical structures, pathological changes, and functional properties of tissues. MRI has become a principal biomarker in the diagnosis and treatment of neurological disorders. Accurate segmentation of brain MRI images is a critical step in many clinical and research applications, including tumor detection and the study and treatment of neurodegenerative diseases such as Alzheimer’s disease (AD) and multiple sclerosis (MS). Brain segmentation involves partitioning brain MRI into different tissues such as gray matter (GM), white matter (WM), cerebrospinal fluid (CSF), anatomical structures or regions of interest, and pathological regions like lesions. The outputs are used for diagnosis, identification of pathology, and morphometric estimation of brain structures, and are provided as input to downstream neuroimaging steps such as volume estimation and reconstruction of cortical geometry.

While manual segmentation by medical experts is typically considered the gold-standard benchmark for brain segmentation, it too can be error-prone and operator-dependent, and is tedious and time-consuming [1]. Numerous automatic tools have therefore been developed for the task. Traditional approaches are typically based on a series of image transformation steps, such as registration, intensity thresholding, edge detection, clustering, and region growing. Simpler traditional approaches typically struggle with segmenting complex and heterogeneous structures of varying contrasts that can be found in the brain. The more sophisticated approaches, such as atlas-based methods, may lack robustness and suffer from long runtimes, especially when including non-rigid registration. This limits their scalability to large studies and clinical trials, which include thousands of scans or require immediate results.

In recent years, the advent of deep learning techniques has revolutionized the field of medical image analysis [2]. Convolutional Neural Networks (CNNs), a class of deep neural networks designed for image processing, demonstrated remarkable success and achieved state-of-the-art performance in medical image segmentation. The application of deep learning methods holds the potential to improve the accuracy, robustness, and automation of brain MRI segmentation. This reduces the burden on medical experts and enhances the reliability and reproducibility of diagnostic procedures. By providing accurate and scalable segmentation, deep learning models can facilitate large-scale neurological studies and support improved diagnosis, treatment planning, and monitoring of neurological diseases, ultimately contributing to improved patient outcomes. Many methods, however, face important challenges that limit their practical deployment.

Several reviews have addressed segmentation in specific contexts such as tumors [3], lesions [4], or fetal populations [5], but none have focused on brain segmentation in adults. The most closely related recent review by Wu et al. [6] surveyed segmentation of brain tissues (GM, WM, and CSF) across different age groups, including 35 studies on adults. However, several important gaps remain unaddressed. Our review builds on the existing literature in four key ways: (1) we cover segmentation into anatomical structures and regions of interest, which is crucial for many neuroimaging applications and presents distinct challenges from tissue segmentation, and we include 66 publications on this task; (2) we include an additional 54 publications on tissue segmentation, extending the scope of Wu et al.’s work; (3) we systematically analyze architectural choices, model dimensionalities (2D, 2.5D, 3D) and input strategies (patch-based vs. full slice/volume), and generalization techniques; and (4) we assess commonly utilized approaches for validation and identify key methodological limitations. Together, these contributions provide a more comprehensive and deeper overview of the current landscape of deep learning-based brain segmentation in adult populations.

2. Methods

This review is registered on the Open Science Framework (OSF; https://osf.io/k879r, accessed on 8 October 2025) and follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines where applicable (see Supplementary Materials S1 for the PRISMA checklist).

We outline our systematic literature search strategy in Section 2.1. We describe brain MRI segmentation and its applications in Section 2.2 and the traditional approaches in Section 2.3 in order to highlight the context of deep learning methods. We discuss deep learning approaches in Section 2.4 and focus on key aspects regarding model architecture in Section 2.5, patch-based and whole-image-based models in Section 2.6, model dimensionality in Section 2.7, and generalization in Section 2.8.

2.1. Literature Search

We conducted a search on PubMed (https://pubmed.ncbi.nlm.nih.gov/, accessed on 23 May 2025) to identify publications on brain segmentation of MRI using deep learning models. The query used was

(brain[MeSH Terms]) AND (magnetic resonance imaging[MeSH Terms]) AND (segmentation[Title/Abstract] OR segmentations[Title/Abstract]) AND (“neural network”[Title/Abstract] OR “neural networks”[Title/Abstract] OR “deep learning”[Title/Abstract] OR transformer[Title/Abstract])

The search was limited to publications from 1 January 2015 onward to exclude studies preceding the influential U-Net [7] segmentation model, and was last conducted on 23 May 2025. The query returned 660 works, from which we excluded exclusively pediatric, ex vivo, and non-human studies, structure segmentation with a low (<7) number of labels, and tumor and lesion segmentation; only publications written in English were considered. This resulted in 131 publications. A subsequent manual review to ensure the eligibility criteria and direct relevance to deep learning brain MRI segmentation yielded 88 publications. For comprehensive coverage, we supplemented these results with forward and backward citation tracking, giving a total of 120 publications. Figure 1 illustrates the steps followed during the screening process to identify eligible publications.

2.2. Definition and Applications

Brain segmentation partitions the brain into brain tissues, anatomical structures, and regions of interest. Specifically, tissue segmentation classifies each brain voxel as GM, WM, or CSF, and may identify additional elements such as the scalp and skull [8], while structure segmentation assigns voxels to one of several structures or regions. In the literature, whole brain segmentation refers to structure segmentation of the entire brain. Tissue and structure segmentation differ in applications and computational requirements, with whole brain segmentation typically being the most challenging task due to a large number of output labels. Some models segment the brain into both tissues and structures simultaneously [9]. Although structure segmentation outputs can often be combined to produce tissue segmentation, this approach may yield inferior results compared to dedicated tissue segmentation [10] and requires greater computational resources.

Brain segmentation is a critical step in neuroimaging pipelines, providing detailed anatomical delineation necessary for structure visualization, morphometric or volumetric quantification, and downstream computational analyses. It supports large-scale studies examining brain development, structural variability among populations, and pathology associated with neurodegenerative, psychiatric, and neurodevelopmental disorders. As neuroimaging datasets increasingly comprise thousands of scans [11], the demand for automated, scalable, and robust segmentation methods has intensified. This requirement is also present in clinical trials, where high-throughput and accurate volume estimation from MRI is essential for reproducibility and statistical power. In clinical settings, brain segmentation further facilitates surgical planning and image-guided interventions. Segmentation results are also used as inputs to further steps in neuroimaging workflows and for aligning PET and CT images for better anatomical localization.

Segmentation-derived volume and thickness measurements play a crucial role in the assessment and follow-up of neurodegenerative disorders. Healthy aging is characterized by slow GM and WM atrophy, cortical thinning, and expansion of the ventricular system and subarachnoid CSF. Estimates of greater and localized atrophy, especially in the hippocampus, amygdala, entorhinal cortex, and medial temporal lobe [12], are associated with mild cognitive impairment. Such patterns have prominently been studied in AD, which is further distinguished by atrophy of the posterior cingulate and the parietal lobes. Tracking AD progression and treatment effects is additionally motivated by increased atrophy and ventricular enlargement found in recent trials of anti-amyloid drugs [13]. In MS, deep GM atrophy and damage show high clinical relevance in predicting physical disability and cognitive impairment [14]. WM lesions, visible as T1 hypointensities and more prominently as T2 and FLAIR hyperintensities, also correlate with disability in MS. Some whole brain segmentation methods therefore include segmentation of WM lesions, while WM tissue masks can improve separate lesion segmentation [15]. In Huntington’s disease, the primary image marker of progression is atrophy of the caudate and putamen [16]. Volumetry also aids in distinguishing Parkinson’s disease (PD) from Parkinson-plus syndromes, which manifest with additional clinical and structural features. For example, midbrain atrophy is seen in progressive supranuclear palsy (PSP), but is typically absent in PD, while atrophy of the pons, cerebellum, and putamen is more indicative of multiple system atrophy (MSA) [17].

Brain segmentation is also utilized in psychological and neurological conditions, where subcortical volumes differentiate major depressive, bipolar, and remitted major depressive disorder [18], while subcortical enlargement has been observed in schizophrenia [19]. Structural brain changes have also been investigated in relation to conditions such as atrial fibrillation [20], migraine [21], and even hospitalization [22]. Segmentation supports magnetic resonance spectroscopy by enabling metabolite quantification by tissue type [23]. In transcranial electrical stimulation, accurate segmentation of sulci and gyri is essential for precise electric field modeling [24]. Furthermore, segmentation aids in defining functional region boundaries for functional MRI (fMRI), offering applications even when structural MRI reveals no abnormalities, such as in mild traumatic brain injury [25]. Delineation of the cortex, WM, and the meninges is also performed as the first step in cortical reconstruction pipelines [26]. The applications to various domains and settings increase the need for accurate, robust, generalizable, and efficient brain MRI segmentation.

2.3. Traditional Segmentation Approaches

Traditional brain segmentation algorithms are typically multi-stage pipelines that consist of several processing steps [27]. Intensity-based approaches use thresholding, region growing, clustering, and classification, and are suitable for the simpler task of tissue segmentation. Whole brain segmentation is typically performed by the more intensive atlas-based methods. These methods begin with preprocessing steps, such as intensity normalization, bias field correction, and skull stripping, followed by non-rigid registration to an atlas in a standard space. Anatomical labels of the atlas are then propagated onto the subject image. This approach underlies established and publicly available neuroimaging toolkits such as FreeSurfer [28], SPM [29], and CAT [30], which are frequently utilized in studies and clinical trials. However, such methods have very long runtimes due to non-rigid registration: for example, CAT12 and FreeSurfer may require approximately 1 h and 8 h, respectively, to segment a single scan. Furthermore, they are sensitive to registration quality, with high intersubject variability of brain anatomy further limiting accurate atlas matching. Multi-atlas label fusion methods therefore extend the paradigm by using labels from multiple annotated scans to produce the final predictions on the input volume. While such methods produce very accurate segmentations, they require multiple non-linear registrations and are exceptionally computationally intensive, requiring even up to 147 h for a single segmentation [31]. Long runtimes limit the scalability of atlas-based methods for large-scale studies and trials, where throughput is essential to processing thousands of scans. The methods also leave room for improvements in accuracy and reliability. In contrast, deep learning models are a scalable alternative, achieving state-of-the-art accuracy while drastically reducing inference time.

2.4. Deep Learning Segmentation

The success of deep learning models in segmentation tasks has been driven by hardware advances, the availability of annotated datasets, and improvements in deep learning architectures. Since their adaptation to brain MRI segmentation, the models have achieved state-of-the-art accuracy while significantly accelerating segmentation; accurate methods perform segmentation in seconds on GPUs. Such models typically avoid the complex pipelines of traditional methods and do not rely on registration or hand-crafted features; instead, they learn directly from annotated data in a supervised manner. Their performance, therefore, highly depends on the quality and diversity of training data, as well as the model architecture and training strategy.

Deep learning has been extensively applied to tumor [3] and lesion [4] segmentation. The progress has been somewhat slower in brain segmentation, partly due to the longstanding availability of established traditional methods, which have remained widely used in both research and clinical contexts. More critically, there is a notable scarcity of comprehensive, manually labeled ground truth datasets for the entire brain, hindering supervised training and robust evaluation. In comparison, there is a larger and more diverse volume of manually annotated data available for tumor and lesion segmentation. Additionally, brain segmentation encompasses a broad anatomical scope, requiring models to generalize across a wide range of brain structures with varying size, shape, and intensity profiles. Whole brain segmentation, in particular, demands great computational resources due to the large numbers of target labels.

A central design consideration in deep learning-based segmentation is the choice of model architecture. Most brain segmentation models have adapted architectures originally developed for other domains, evolving from early convolutional classification networks applied in a sliding-window manner [32] to fully convolutional networks [33], U-Net variants [7], and, more recently, transformer-based [34] and state space models [35]. Due to the large GPU memory requirements of training on 3D images, initial approaches were commonly patch-based and subdivided the input MRI or slice into smaller regions to reduce memory demands and increase training efficiency. Although the increase in computational resources allowed for many current models to be trained on full volumes, several studies have reported improved results when training on large patches instead [36]. Another important factor is model dimensionality: 2D methods operate on individual slices or thin blocks of slices, 3D methods process full volumes or sub-volumes, while 2.5D ensemble models trained on orthogonal 2D slices. The choice of the segmentation input (patches vs. full slices or volumes) and model dimensionality reflects a trade-off between the amount of utilized spatial context and the computational cost of training and inference. We discuss the segmentation input in Section 2.6 and model dimensionality in Section 2.7. Crucially for practical adoption, segmentation methods must generalize across highly diverse imaging parameters and populations. We discuss generalization strategies in Section 2.8.

We list all structure and tissue segmentation methods in Table 1 and Table 2, respectively, summarizing key properties such as input modalities, model dimensionality, input size, and model architecture. For consistency, we report the highest Dice similarity coefficient (DSC)—the most commonly available performance measure—as an indicator of segmentation accuracy. Additionally, structure segmentation results on the Mindboggle101 dataset [37] are presented in Table 4.

2.5. Segmentation Architectures

Deep learning brain MRI segmentation has closely followed advancements in deep learning for general medical image segmentation. There, early efforts processed images patch-by-patch and classified the center voxel of each patch using a convolutional classification model in a sliding-window fashion [32]. Such approaches struggle to incorporate global context, as each patch is processed in isolation and is computationally expensive due to redundant computations. The introduction of Fully Convolutional Networks (FCNs) [33] addressed these issues by processing images of arbitrary size and generating correspondingly-sized output, enabling end-to-end learning. FCNs replaced the fully-connected layers in classification models with convolutional ones to produce dense, pixel-wise predictions. FCNs utilize transposed convolutions to restore resolution after downsampling and combine coarser predictions from deep layers with fine-grained predictions from shallow layers using skip connections. Compared to sliding-window models, FCNs improved accuracy and efficiency by utilizing global context and eliminating redundant computations [33].

Building on FCNs, “U-shaped” models such as the U-Net [7] introduced a symmetric encoder–decoder architecture with skip connections that combine activations from deep and shallow layers. The encoder and decoder are composed of multiple blocks, each block consisting of a series of convolutional layers, non-linearities such as the ReLU, and normalization layers. The latter include batch, group [36], instance [38,39,40], and weight [41] normalization layers in brain segmentation. The encoder progressively downsamples activations via max pooling or strided convolutions, while the decoder upsamples activations towards the original resolution using transposed convolutions [7] or pooling indices taken from the encoder [42]. Compared to FCNs, the U-Net architecture has greater representational power and greater ability to combine high-level features and fine details.

U-Net achieved notable success in various segmentation tasks and inspired numerous modifications which continue to achieve state-of-the-art results in medical image segmentation. In brain segmentation, it remains the backbone architecture of the majority of models, with notable architectural refinements incorporated over time. These include residual blocks in the encoder and decoder [24,36,43,44], dense [8,45,46] or competitive dense [26,47,48] blocks, the UNet++ architecture [49] with nested dense skip pathways, multiscale convolutional features [38,50], and multiple paths capturing details at different scales [51,52]. Multiple U-shaped models utilize pooling indices instead of or in conjunction with skip connections [26,42,47,48,53,54,55], with the aim of greater preservation of fine details and smaller structures. Works have also focused on reducing model sizes [10,44,50,56]. Some architectures have utilized dilated convolutions as additions or alternatives to U-Net [57,58,59,60,61,62].

A special class of architectural refinements to U-Net models are attention-gating mechanisms, which have been increasingly adopted to improve discrimination and localization capabilities. In brain segmentation, such mechanisms include Squeeze-and-Excitation blocks [63], which weigh feature channels according to their global importance, 3D spatial attention mechanisms [64], global attention modules [54,55], which integrate local features with their respective global dependencies, cross-attention for capturing dependencies between dual-modality features [65], and split-attention for weighing groups of channels [49]. Works have also utilized scale attention mechanisms [38,46]. While models incorporating attention modules demonstrate performance gains compared to baseline, other works report no improvement when incorporating attention [36,48,66,67]. These differences highlight scenario dependence and a need for more extensive validation.

Initially developed for sequence-to-sequence language processing, transformers have demonstrated success in vision tasks and have been widely adopted in segmentation [34]. Transformers utilize self-attention mechanisms, which allow for each input element to directly interact with and aggregate information from all other input elements based on learned pairwise relationships. This enables better capturing of long-range dependencies and global context than in conventional CNNs, which have limited receptive fields in shallow layers due to fixed-sized kernels, yielding longer chains between distant inputs. However, transformer-based models have higher computational requirements—quadratic with regards to input size—and require more training data, hindering straightforward application to brain segmentation. There, CNN-Transformer hybrid models have utilized self-attention in bottleneck blocks between the encoders and decoders of U-Net backbones, including full-volume models for tissue segmentation [68] and, partly due to large GPU memory requirements, patch-based models for subcortical [69] and whole brain [70] segmentation. Yu et al. [71] designed a hierarchical transformer encoder block and utilized it for whole brain segmentation combined with intracranial and posterior fossa segmentation. Transformers have also been combined with generative adversarial networks in order to more easily segment MS lesions alongside healthy tissues [72]. Interestingly, Rao et al. [68] obtained marginal improvements over a residual U-Net on datasets used in training, but noticeably better generalization to unseen sites.

Most recently, Mamba-based models have been adopted to medical image segmentation. Mamba utilizes a state space model with a selective scan mechanism and a hardware-aware algorithm which allow for the model to scale linearly with input size. Wei et al. [73] found that the U-Mamba architecture outperforms nnU-Net and the transformer-based SwinUNETR model in segmenting 122 GM structures. Cao et al. [74] used a hybrid CNN-Mamba U-Net with state space model blocks as the bottleneck for subcortical segmentation and reported improvements compared to state-of-the-art models, including the transformer-based TABSurfer [69].

Despite architectural improvements compared to the U-Net baseline, the influential nnU-Net works [75,76] have questioned the significance of the reported results in 2D and 3D medical image segmentation. In various settings, they reported minimal or non-existent improvements over optimized U-Net models and hence suggested an ongoing innovation bias. The works emphasized the need for stricter validation standards and presented the self-adapting nnU-Net framework designed for the rigorous validation of new methods.

We report the model architecture for each of the brain segmentation models in Table 1 and Table 2. We show the progression of architectures over time in Figure 2.

2.6. Patch-Based and Whole-Image Based Models

An important distinction in brain MRI segmentation is between patch-based models and those operating on whole volumes or slices. Patch-based models approaches subdivide the MRI data into smaller, manageable regions, which can simplify learning by emphasizing localized features and substantially reduce memory requirements. The reduction is particularly advantageous for models processing high-resolution (HiRes) scans, multi-modal inputs, or performing structural segmentation involving a large number of output labels. Patch-based models have demonstrated robustness in handling HiRes data and in mitigating the effects of noise and variability in smaller regions. Some works have shown that the overlapping of patches is crucial for good prediction accuracy at border voxels [36,77], as it allows for the aggregation of predictions across wider spatial context. Moreover, overlapping patches also increase the number of effective training samples for certain models, which can improve generalization. On the other hand, Yamanakkanavar, Lee et al. [78,79] report improved performance when using non-overlapping patches in a 2D setting. Despite some of the advantages, patch-based models nevertheless face limitations in capturing global context and integrating information across larger brain structures, with the use of overlapping also introducing redundant computation. Furthermore, artifacts on patch borders may be visible, even when overlapping is used [80].

With the growth of computational resources, patch sizes progressed from small local ones—e.g., 32 × 32 [81]—to those encompassing large portions of the brain, e.g., 128 × 128 × 128 or single-hemisphere volumes [82]. Several early works have utilized patches of different sizes [51,83,84,85] either across separate network pathways or by combining small local patches with more global downsampled ones. In structural segmentation, adaptive patch sizing based on the target region of interest has also been explored for enabling more efficient training and testing [86].

In contrast, many models segment the full MRI volume or slice in a single forward pass. By leveraging the full spatial context of the brain, these models can capture long-range dependencies and improve segmentation consistency across large or spatially distant structures. Such segmentation also avoids the computational redundancy associated with overlapping patches and generally results in faster inference times. However, full-volume models in particular require substantially more training memory, which can limit their feasibility for large models or HiRes inputs, especially in structural segmentation tasks. Even with the large memory capacities of modern GPUs, such models are typically limited to training on small batch sizes, which may compromise training stability; group normalization layers [36,87] can be used to mitigate this issue. Despite these challenges, many recent methods adopt full-volume inputs specifically to benefit from global image context. A direct structural 3D segmentation comparison by Roy et al. [36], however, demonstrated better performance when using large overlapping subvolumes, suggesting that full-volume inputs do not universally guarantee improved accuracy. The input configuration used in each brain segmentation model reviewed in this work is reported in Table 1 and Table 2.

Table 1. Summary of brain MRI structure segmentation methods, including input modalities, number of anatomical labels, dimensionality, input type (patch, full slice/volume), backbone architecture, implementation availability (Avail.), and highest reported mean Dice similarity coefficient (DSC) across the labels.

Method	Modalities	Labels	Dim.	Input	Arch.	Avail.	DSC (%)
Brébisson et al. [83]	T1	133	2.5D and 3D	Patch	CNN	No	72.5
Shakeri et al. [88]	T1	8 subcortical	2D	Full	FCN	Yes	82.4
Moeskops et al. [84]	T1, T2	8	2D	Patch	CNN	No	89.8
Milletari et al. [89]	QSM	26 subcortical	2D, 2.5D, or 3D	Patch	CNN	No	77
Bao et al. [90]	T1	7 subcortical	2D	Patch	CNN	No	82.22
Moeskops et al. [58]	T1	7	2D	Patch	FCN	No	92
Dolz et al. [91]	T1	8 subcortical	3D	Patch	FCN	Yes	89
Kushibar et al. [81]	T1	14 subcortical	2.5D	Patch	CNN	Yes	86.9
Mehta et al. [85]	T1	32-134	2D and 3D	Patch	CNN	No	84.4
Mehta et al. [92]	T1	7 subcortical	2D and 3D	Patch	U-Net	No	83
Wachinger et al. [93]	T1	25	3D	Patch	CNN	Yes	92
Li et al. [57]	T1	155	3D	Patch	Dilated	Yes	84.3
Karani et al. [94]	T1, T2	7 subcortical	/	/	U-Net	No	89.3
Roy et al. [47]	T1	27	2.5D	Full	U-Net	Yes	90.1
Roy et al. [63]	T1	27	/	/	U-Net + Attention	No	86.2
Li et al. [59]	T1, T1-IR, FLAIR	10	2D	Full	U-Net	No	80.9
Kaku et al. [95]	T1	102	2D	Full	U-Net	Yes	81.9
Huo et al. [96]	T1	133	3D	Patch	U-Net	Yes	77.6
Jog et al. [97]	T1, T2	9, 12	3D	Patch	U-Net	Yes	94
Novosad et al. [98]	T1	8 and 12 subcortical	3D	Patch	FCN	Yes	89.5
Novosad et al. [99]	T1	12 subcortical	3D	Patch	CNN	Yes	80.7
Sun et al. [64]	T1, T1-IR, FLAIR	Tissue, 25	3D	Patch	U-Net + Attention	No	84.8
Dai et al. [100]	T1	15-138	3D	Patch	U-Net	No	87.9
Roy et al. [101]	T1	33	2.5D	Full	U-Net	Yes	88
Luna et al. [102]	T1, T1-IR, FLAIR	8	3D	Patch	U-Net	No	85.5
McClure et al. [61]	T1	50	3D	Patch	Dilated	Yes	83.7
Dalca et al. [103]	T1, PD	12	3D	Full	U-Net	Yes	83.5
Coupe et al. [104]	T1	133	3D	Patch	U-Net	Yes	79
Ramzan et al. [62]	T1, T1-IR, FLAIR	Tissues, 8	3D	Patch	Dilated	No	91.4
Henschel et al. [26]	T1	95	2.5D	Full	U-Net	Yes	89
Bontempi et al. [80]	T1	8	3D	Full	U-Net	Yes	91.3
Liu et al. [105]	T1	14 subcortical	3D	Patch	U-Net + LSTM	No	88.7
Lee et al. [49]	T1	33, 100+	3D	Patch	U-Net + Attention	No	89.7
Zopes et al. [106]	T1, T2, DWI, CT	27	3D	Patch	U-Net	No	85.3
Li et al. [16]	T1	8 subcortical	3D	Patch	C-LSTM	No	97.6
Li et al. [107]	T1	8 subcortical	3D	Patch	U-Net	No	96.8
Svanera et al. [108]	T1	8	3D	Full	U-Net	Yes	97.8
Li et al. [109]	T1	133	2D	Full	U-Net + Attention	Yes	89.7
Greve et al. [110]	T1	12 subcortical	3D	Full	U-Net	Yes	77.8
Li et al. [40]	T1	54	3D	Full	FCN	Yes	83.1
Meyer et al. [41]	T1	8	3D	Patch	U-Net	Yes	93.2
Wu et al. [86]	T1	14, 54	3D	Patch	M-FCN	No	92.2
Nejad et al. [111]	T1	12	2D	Patch	U-Net	Yes	89.3
Liu et al. [112]	T1	5, 7	3D	Full	CLMorph	No	76.3
Ghazi et al. [113]	T1	133	2.5D	Full	U-Net	Yes	81
Henschel et al. [48]	T1	95	2.5D	Full	U-Net	Yes	89.9
Laiton-Bonadiez et al. [70]	T1	37	3D	Patch	Transformer	No	90
Wei et al. [114]	T1	136	2D	Full	U-Net + Attention	Yes	86
Yee et al. [82]	T1	102	3D	Patch	U-Net	No	84
Baniasadi et al. [115]	T1	30 subcortical	3D	Patch	U-Net	Yes	89
Billot et al. [116]	T1, T2, PD, DBS	110	3D	Full	U-Net	Yes	88
Billot et al. [117]	T1, T2, DBS, FLAIR, PD, CT	33	3D	Full	U-Net	Yes	88
Cao et al. [69]	T1	31 subcortical	3D	Patch	Transformer	No	87.2
Li et al. [46]	T1	28, 139	2D	Full	U-Net + Attention	Yes	87.7
Moon et al. [118]	T1	109	3D	/	U-Net	No	/
Cao et al. [74]	T1	31 subcortical	3D	Patch	Mamba	Yes	88.4
Diaz et al. [119]	T1, T2, FLAIR	7	3D	Patch	U-Net	Yes	88
Kujawa et al. [120]	T1	108	3D	Patch	U-Net	No	87.5
Lorzel et al. [25]	T1	58	3D	Patch	U-Net	No	81
Svanera et al. [52]	T1	7	3D	Full	LOD-brain	Yes	93
Goto et al. [121]	T1	107	3D	Full	U-Net	No	/
Le Bot et al. [122]	FLAIR	133	3D	Patch	U-Net	No	91
Li et al. [65]	T1, PET	45	3D	Full	Transformer	No	85.3
Li et al. [17]	T1	12	3D	Patch	U-Net	Yes	90
Puzio et al. [123]	T1, T2	38	3D	Patch	U-Net	No	87
Wei et al. [73]	T1	122 GM	3D	Patch	Mamba	No	91.1

Table 1. Summary of brain MRI structure segmentation methods, including input modalities, number of anatomical labels, dimensionality, input type (patch, full slice/volume), backbone architecture, implementation availability (Avail.), and highest reported mean Dice similarity coefficient (DSC) across the labels.

Method	Modalities	Labels	Dim.	Input	Arch.	Avail.	DSC (%)
Brébisson et al. [83]	T1	133	2.5D and 3D	Patch	CNN	No	72.5
Shakeri et al. [88]	T1	8 subcortical	2D	Full	FCN	Yes	82.4
Moeskops et al. [84]	T1, T2	8	2D	Patch	CNN	No	89.8
Milletari et al. [89]	QSM	26 subcortical	2D, 2.5D, or 3D	Patch	CNN	No	77
Bao et al. [90]	T1	7 subcortical	2D	Patch	CNN	No	82.22
Moeskops et al. [58]	T1	7	2D	Patch	FCN	No	92
Dolz et al. [91]	T1	8 subcortical	3D	Patch	FCN	Yes	89
Kushibar et al. [81]	T1	14 subcortical	2.5D	Patch	CNN	Yes	86.9
Mehta et al. [85]	T1	32-134	2D and 3D	Patch	CNN	No	84.4
Mehta et al. [92]	T1	7 subcortical	2D and 3D	Patch	U-Net	No	83
Wachinger et al. [93]	T1	25	3D	Patch	CNN	Yes	92
Li et al. [57]	T1	155	3D	Patch	Dilated	Yes	84.3
Karani et al. [94]	T1, T2	7 subcortical	/	/	U-Net	No	89.3
Roy et al. [47]	T1	27	2.5D	Full	U-Net	Yes	90.1
Roy et al. [63]	T1	27	/	/	U-Net + Attention	No	86.2
Li et al. [59]	T1, T1-IR, FLAIR	10	2D	Full	U-Net	No	80.9
Kaku et al. [95]	T1	102	2D	Full	U-Net	Yes	81.9
Huo et al. [96]	T1	133	3D	Patch	U-Net	Yes	77.6
Jog et al. [97]	T1, T2	9, 12	3D	Patch	U-Net	Yes	94
Novosad et al. [98]	T1	8 and 12 subcortical	3D	Patch	FCN	Yes	89.5
Novosad et al. [99]	T1	12 subcortical	3D	Patch	CNN	Yes	80.7
Sun et al. [64]	T1, T1-IR, FLAIR	Tissue, 25	3D	Patch	U-Net + Attention	No	84.8
Dai et al. [100]	T1	15-138	3D	Patch	U-Net	No	87.9
Roy et al. [101]	T1	33	2.5D	Full	U-Net	Yes	88
Luna et al. [102]	T1, T1-IR, FLAIR	8	3D	Patch	U-Net	No	85.5
McClure et al. [61]	T1	50	3D	Patch	Dilated	Yes	83.7
Dalca et al. [103]	T1, PD	12	3D	Full	U-Net	Yes	83.5
Coupe et al. [104]	T1	133	3D	Patch	U-Net	Yes	79
Ramzan et al. [62]	T1, T1-IR, FLAIR	Tissues, 8	3D	Patch	Dilated	No	91.4
Henschel et al. [26]	T1	95	2.5D	Full	U-Net	Yes	89
Bontempi et al. [80]	T1	8	3D	Full	U-Net	Yes	91.3
Liu et al. [105]	T1	14 subcortical	3D	Patch	U-Net + LSTM	No	88.7
Lee et al. [49]	T1	33, 100+	3D	Patch	U-Net + Attention	No	89.7
Zopes et al. [106]	T1, T2, DWI, CT	27	3D	Patch	U-Net	No	85.3
Li et al. [16]	T1	8 subcortical	3D	Patch	C-LSTM	No	97.6
Li et al. [107]	T1	8 subcortical	3D	Patch	U-Net	No	96.8
Svanera et al. [108]	T1	8	3D	Full	U-Net	Yes	97.8
Li et al. [109]	T1	133	2D	Full	U-Net + Attention	Yes	89.7
Greve et al. [110]	T1	12 subcortical	3D	Full	U-Net	Yes	77.8
Li et al. [40]	T1	54	3D	Full	FCN	Yes	83.1
Meyer et al. [41]	T1	8	3D	Patch	U-Net	Yes	93.2
Wu et al. [86]	T1	14, 54	3D	Patch	M-FCN	No	92.2
Nejad et al. [111]	T1	12	2D	Patch	U-Net	Yes	89.3
Liu et al. [112]	T1	5, 7	3D	Full	CLMorph	No	76.3
Ghazi et al. [113]	T1	133	2.5D	Full	U-Net	Yes	81
Henschel et al. [48]	T1	95	2.5D	Full	U-Net	Yes	89.9
Laiton-Bonadiez et al. [70]	T1	37	3D	Patch	Transformer	No	90
Wei et al. [114]	T1	136	2D	Full	U-Net + Attention	Yes	86
Yee et al. [82]	T1	102	3D	Patch	U-Net	No	84
Baniasadi et al. [115]	T1	30 subcortical	3D	Patch	U-Net	Yes	89
Billot et al. [116]	T1, T2, PD, DBS	110	3D	Full	U-Net	Yes	88
Billot et al. [117]	T1, T2, DBS, FLAIR, PD, CT	33	3D	Full	U-Net	Yes	88
Cao et al. [69]	T1	31 subcortical	3D	Patch	Transformer	No	87.2
Li et al. [46]	T1	28, 139	2D	Full	U-Net + Attention	Yes	87.7
Moon et al. [118]	T1	109	3D	/	U-Net	No	/
Cao et al. [74]	T1	31 subcortical	3D	Patch	Mamba	Yes	88.4
Diaz et al. [119]	T1, T2, FLAIR	7	3D	Patch	U-Net	Yes	88
Kujawa et al. [120]	T1	108	3D	Patch	U-Net	No	87.5
Lorzel et al. [25]	T1	58	3D	Patch	U-Net	No	81
Svanera et al. [52]	T1	7	3D	Full	LOD-brain	Yes	93
Goto et al. [121]	T1	107	3D	Full	U-Net	No	/
Le Bot et al. [122]	FLAIR	133	3D	Patch	U-Net	No	91
Li et al. [65]	T1, PET	45	3D	Full	Transformer	No	85.3
Li et al. [17]	T1	12	3D	Patch	U-Net	Yes	90
Puzio et al. [123]	T1, T2	38	3D	Patch	U-Net	No	87
Wei et al. [73]	T1	122 GM	3D	Patch	Mamba	No	91.1

2.7. Model Dimensionality

Brain segmentation models can be categorized into 2D, 2.5D, and 3D, depending on the dimensionality of their input and processing. Early deep segmentation models were developed for 2D images, and thus splitting MRI volumes into 2D slices along the axial, coronal, or sagittal plane allowed for easy application of the models onto brain MRI [84,88,90]. Compared to 3D approaches, this strategy reduces memory requirements and, for some models, increases the number of effective training samples per volume. However, 2D models lack access to inter-slice context and volumetric spatial dependencies, which are critical for accurate segmentation. Although most of the recent slice-based models operate on thin blocks of adjacent slices [46,114,124] in order to capture the important local 3D information, they remain limited in their ability to leverage broader volumetric context.

Three-dimensional segmentation models directly segment subvolumes or entire image volumes, and are equivalent to standard two-dimensional networks with an added spatial dimension. These approaches are able to utilize a wide, three-dimensional image context. However, a significant drawback of 3D models is the memory requirements due to large data volumes, which scale rapidly with input resolution, network depth, and the number of output labels. Large parameter counts and potentially smaller numbers of training samples per volume additionally lead to generalization problems. Computational requirements grow especially large in structural segmentation involving numerous target regions; for example, a 3D U-Net segmenting a full 256 × 256 × 256 volume into 79 structures may exceed 100 GB of memory with computation sharded across multiple GPUs [36]. Techniques have therefore been developed to reduce the requirements by grouping and merging labels before inference and recovering the labels at inference time [120]. This reduces the practical feasibility of training large 3D networks, especially if operating on full volume inputs, and restricts batch sizes. In our review, we found that most recent works utilize 3D models with full volumes or large patches as inputs.

In contrast, 2.5D models represent a hybrid solution that ensembles models segmenting orthogonal 2D slices or blocks of adjacent slices, thus combining some of the advantages of 2D and 3D segmentation methods. Typically, 2.5D models first generate slice segmentations for the axial, coronal, and sagittal planes separately. The final classification for each voxel is then derived by combining the predictions from the three slices encompassing the voxel. This approach provides a compromise between computational efficiency and use of spatial context by requiring significantly less computation compared to 3D approaches while still incorporating the orthogonal 3D context. Crucially, 2.5D models often deliver the same levels of segmentation accuracy as 3D models [26,36,47,48].

Early studies comparing 2D, 2.5D, and 3D models reported no consistent ranking in terms of performance metrics [77,89]. However, qualitative differences were noted—for example, Bernal et al. [77] observed that 3D models produced smoother predictions across slices than 2D ones, as the latter processed slices individually. However, this property also reduced the sensitivity of 2D models to variations in voxel spacing. More recent comparisons have consistently shown that 3D models outperform 2D models in terms of accuracy [125,126,127]. The relationship between 2.5D and 3D models remains less clear. A direct comparison in [36] reported that 2.5D models outperformed full-volume 3D models, but underperformed relative to 3D models trained on large overlapping patches. We note that the 2.5D models in the comparison by Avesta et al. [126] on three subcortical structures were not based on orthogonal slices, but instead used stacks of five adjacent slices from a single plane.

Models are not limited to one-dimensionality and can combine inputs of different spatial properties [83,85,92]. We note that manual segmentation by medical experts may follow different strategies, analyzing slices from a single views, using orthogonal slices, or utilizing 3D context, which are important, as manual segmentation is often used as a ground truth benchmark. In Table 1 and Table 2, we specify the dimensionality of each model.

Higher-dimensional models capture richer spatial information, but require greater computational resources and, in some cases, can still be more efficient by reducing redundant context aggregation. Efficiency is also shaped by factors such as model size, architectural complexity, and hyperparameter choices, making comparisons across studies difficult. A major limitation is the inconsistent reporting of computational complexity, which highlights the need for a more systematic evaluation in future work.

Table 2. Summary of brain MRI tissue segmentation methods, including input modalities, dimensionality, input type (patch, full slice/volume), backbone architecture, implementation availability (Avail.), and highest reported mean Dice similarity coefficient (DSC) across the tissues.

Method	Modality	Dim.	Input	Arch.	Avail.	DSC (%)
Stollenga et al. [128]	T1, T1-IR, T2-FLAIR	3D	Patch	PyraMiD-LSTM	No	85.7
Nguyen et al. [129]	T1	2.5D	Patch	CNN	No	86
Fedorov et al. [130]	T1	3D	Patch	Dilated	No	86.5
Chen et al. [43]	T1, T1-IR, T2-FLAIR	3D	Patch	FCN	No	86.6
Khagi et al. [131]	T1	2D	Full	SegNet	No	76.2
Rajchl et al. [132]	T1	3D	Patch	FCN	Yes	93
Kumar et al. [53]	T1	2D	Patch	SegNet	No	80
Gottapu et al. [133]	T1	2D	Patch	CNN	No	67.3
Mahbod et al. [134]	T1, T1-IR, FLAIR	/	/	ANN	No	85.3
Chen et al. [135]	T1, T1-IR, T2-FLAIR	2D	Patch	OctopusNet	No	82.9
Kong et al. [136]	T1	2D	Patch	CNN	No	/
Bernal et al. [77]	T1, and T1 and T2	2D, 3D	Patch	FCN, U-Net	Yes	92.9
Dolz et al. [45]	T1, T1-IR, T2-FLAIR	3D	Patch	FCN	Yes	87.2
Gabr et al. [137]	T1, T2, T2-FLAIR, PD	2D	Full	U-Net	Yes	93
Ito et al. [138]	T1	3D	Patch	FCN	No	86
Yogananda et al. [139]	T1	3D	Patch	U-Net	No	86.6
Mujica-Vargas et al. [140]	T1, T2, T2-FLAIR	2D	Full	U-Net	No	93.1
Wang et al. [141]	T1	3D	Patch	U-Net	No	90.4
Xie et al. [142]	T1, T2, PD	2D	Full	LSTM	No	98.7
Yan et al. [143]	T1	3D	Full	GCN	No	91.6
Li et al. [60]	T1, T1-IR, T2-FLAIR	2.5D	Full	Dilated	Yes	87
Wei et al. [51]	T1	2.5D	Patch	U-Net	No	96.3
Sun et al. [64]	T1, T1-IR and T2-FLAIR	3D	Patch	U-Net + Attention	No	87
Ramzan et al. [62]	T1, T1-IR, T2-FLAIR	3D	Patch	Dilated	No	88
Lee et al. [79]	T1	2D	Patch	U-Net	Yes	93.6
Mostapha et al. [39]	T1	3D	Full	U-Net	No	90.3
Narayana et al. [144]	T1, T2, T2-FLAIR, PD	2D	Full	U-Net	Yes	92.5
Yamanakkanavar et al. [78]	T1	2D	Patch	U-Net	No	95.2
Sendra-Balcells et al. [24]	T1	2D	Full	U-Net	No	88.5
Dayananda et al. [54]	T1	2D	Patch	Squeeze U-Net	No	95.3
Basnet et al. [10]	T1, T2	3D	Patch	U-Net	Yes	93.1
Long et al. [145]	T1, T1-IR, T2-FLAIR	3D	Patch	MSCD-UNet	No	88.5
Woo et al. [125]	T1	3D	Full	U-Net	No	94.9
Yamanakkanavar et al. [50]	T1	2D	Patch	U-Net	No	95.7
Zhang et al. [146]	Diffusion, T1, T2	2.5D	Full	U-Net	No	85
Zhang et al. [147]	T1	3D	Full	GCN	No	92.3
Wei et al. [66]	T1	3D	Patch	Nes-Net	No	88.5
Niu et al. [67]	T1	2D	Full	U-Net	Yes	91.1
Goyal et al. [148]	T1	2D	Patch	SegNet	No	83
Prajapati et al. [124]	T1	2D	Full	U-Net	No	95.7
Rao et al. [68]	T1	3D	Full	Transformer	Yes	95.6
Yamanakkanavar et al. [50]	T1	2D	Patch	Squeeze U-Net	No	96
Yamanakkanavar et al. [149]	T1	2D	Patch	MF2-Net	No	95.3
Dayananda et al. [44]	T1	2D	Patch	Squeeze U-Net	No	95.7
Clerigues et al. [150]	T1, T2-FLAIR	3D	Patch	U-Net	Yes	94.6
Guven et al. [151]	T1	2D	Full	GAN	No	/
Gi et al. [8]	T2	2.5D, 3D	Patch, full	U-Net	No	91.3
Oh et al. [152]	T1	3D	Full	U-Net	Yes	88.5
Simarro et al. [9]	T1	3D	Patch	U-Net	Yes	/
Hossain et al. [127]	T1	3D	Patch	U-Net	Yes	93.7
Liu et al. [38]	T1	3D	Patch	U-Net + Attention	Yes	98.9
Mohammadi et al. [72]	T1, T2	2D	Full	Transformer	No	84.4

Table 2. Summary of brain MRI tissue segmentation methods, including input modalities, dimensionality, input type (patch, full slice/volume), backbone architecture, implementation availability (Avail.), and highest reported mean Dice similarity coefficient (DSC) across the tissues.

Method	Modality	Dim.	Input	Arch.	Avail.	DSC (%)
Stollenga et al. [128]	T1, T1-IR, T2-FLAIR	3D	Patch	PyraMiD-LSTM	No	85.7
Nguyen et al. [129]	T1	2.5D	Patch	CNN	No	86
Fedorov et al. [130]	T1	3D	Patch	Dilated	No	86.5
Chen et al. [43]	T1, T1-IR, T2-FLAIR	3D	Patch	FCN	No	86.6
Khagi et al. [131]	T1	2D	Full	SegNet	No	76.2
Rajchl et al. [132]	T1	3D	Patch	FCN	Yes	93
Kumar et al. [53]	T1	2D	Patch	SegNet	No	80
Gottapu et al. [133]	T1	2D	Patch	CNN	No	67.3
Mahbod et al. [134]	T1, T1-IR, FLAIR	/	/	ANN	No	85.3
Chen et al. [135]	T1, T1-IR, T2-FLAIR	2D	Patch	OctopusNet	No	82.9
Kong et al. [136]	T1	2D	Patch	CNN	No	/
Bernal et al. [77]	T1, and T1 and T2	2D, 3D	Patch	FCN, U-Net	Yes	92.9
Dolz et al. [45]	T1, T1-IR, T2-FLAIR	3D	Patch	FCN	Yes	87.2
Gabr et al. [137]	T1, T2, T2-FLAIR, PD	2D	Full	U-Net	Yes	93
Ito et al. [138]	T1	3D	Patch	FCN	No	86
Yogananda et al. [139]	T1	3D	Patch	U-Net	No	86.6
Mujica-Vargas et al. [140]	T1, T2, T2-FLAIR	2D	Full	U-Net	No	93.1
Wang et al. [141]	T1	3D	Patch	U-Net	No	90.4
Xie et al. [142]	T1, T2, PD	2D	Full	LSTM	No	98.7
Yan et al. [143]	T1	3D	Full	GCN	No	91.6
Li et al. [60]	T1, T1-IR, T2-FLAIR	2.5D	Full	Dilated	Yes	87
Wei et al. [51]	T1	2.5D	Patch	U-Net	No	96.3
Sun et al. [64]	T1, T1-IR and T2-FLAIR	3D	Patch	U-Net + Attention	No	87
Ramzan et al. [62]	T1, T1-IR, T2-FLAIR	3D	Patch	Dilated	No	88
Lee et al. [79]	T1	2D	Patch	U-Net	Yes	93.6
Mostapha et al. [39]	T1	3D	Full	U-Net	No	90.3
Narayana et al. [144]	T1, T2, T2-FLAIR, PD	2D	Full	U-Net	Yes	92.5
Yamanakkanavar et al. [78]	T1	2D	Patch	U-Net	No	95.2
Sendra-Balcells et al. [24]	T1	2D	Full	U-Net	No	88.5
Dayananda et al. [54]	T1	2D	Patch	Squeeze U-Net	No	95.3
Basnet et al. [10]	T1, T2	3D	Patch	U-Net	Yes	93.1
Long et al. [145]	T1, T1-IR, T2-FLAIR	3D	Patch	MSCD-UNet	No	88.5
Woo et al. [125]	T1	3D	Full	U-Net	No	94.9
Yamanakkanavar et al. [50]	T1	2D	Patch	U-Net	No	95.7
Zhang et al. [146]	Diffusion, T1, T2	2.5D	Full	U-Net	No	85
Zhang et al. [147]	T1	3D	Full	GCN	No	92.3
Wei et al. [66]	T1	3D	Patch	Nes-Net	No	88.5
Niu et al. [67]	T1	2D	Full	U-Net	Yes	91.1
Goyal et al. [148]	T1	2D	Patch	SegNet	No	83
Prajapati et al. [124]	T1	2D	Full	U-Net	No	95.7
Rao et al. [68]	T1	3D	Full	Transformer	Yes	95.6
Yamanakkanavar et al. [50]	T1	2D	Patch	Squeeze U-Net	No	96
Yamanakkanavar et al. [149]	T1	2D	Patch	MF2-Net	No	95.3
Dayananda et al. [44]	T1	2D	Patch	Squeeze U-Net	No	95.7
Clerigues et al. [150]	T1, T2-FLAIR	3D	Patch	U-Net	Yes	94.6
Guven et al. [151]	T1	2D	Full	GAN	No	/
Gi et al. [8]	T2	2.5D, 3D	Patch, full	U-Net	No	91.3
Oh et al. [152]	T1	3D	Full	U-Net	Yes	88.5
Simarro et al. [9]	T1	3D	Patch	U-Net	Yes	/
Hossain et al. [127]	T1	3D	Patch	U-Net	Yes	93.7
Liu et al. [38]	T1	3D	Patch	U-Net + Attention	Yes	98.9
Mohammadi et al. [72]	T1, T2	2D	Full	Transformer	No	84.4

2.8. Generalization Strategies

A crucial consideration in deep learning is generalization: the ability to perform reliably on domains beyond those used in training. Brain MRI data are highly heterogeneous in imaging parameters (scanner vendor, modality, sequence type, field strength, contrast, resolution) and populations (age, condition, demographics). A segmentation model must generalize effectively in order to handle diverse datasets, multi-center studies, and longitudinal trials where brain structure changes due to age or disease; conventional methods have shown significant robustness in such settings. Furthermore, in clinical scenarios physicians may prefer sparse sets of 2D slices instead of the isotropic 3D volumes common in research studies. Clinical adoption, therefore, requires adaptation to variability in in-plane resolution, slice orientation, and slice spacing, and to larger partial voluming. As deep learning models are highly sensitive to training data characteristics, many domain adaptation strategies have been proposed for brain MRI, especially in structure segmentation. These include data augmentation, domain randomization, and the use of large and diverse training sets.

Data augmentation is widely used to improve generalization. Svanera et al. [52] systematically evaluate geometrical transformations, six noise distortions, and ghosting and MR field in homogeneity artifacts as augmentations. They found that flips and grid transformations, noise distortions, and artifact introductions improve robustness, while rotations and translations do not. Meyer et al. [41] propose a Gaussian Mixture Model-based data augmentation that modifies tissue intensities to vary contrast while preserving anatomy. The method improves generalization to unseen scanners, even with multi-scanner training data, with further work needed to assess performance without bias-field correction and to adapt it to MS lesions without available masks.

SynthSeg [117] trains entirely on synthetic data sampled from a generative model conditioned on segmentation maps. By fully randomizing contrast, resolution, orientation, and slice spacing, the model learns domain-independent features. SynthSeg achieves remarkable generalizability across diverse sequences and modalities (including FLAIR and CT), anisotropic low-resolution (up to 7.0 mm) MRI, and atrophy [116], without any retraining or fine-tuning, and outperforms supervised and domain-adaptation methods in experiments. Its ability to handle sparse 2D acquisitions makes it particularly relevant for clinical workflows. On FLAIR, the specialized FLAIRBrainSeg [122] supervised learning method was shown to outperform such modality-agnostic approaches.

FastSurferVINN [48] achieves resolution-independence by integrating resolution normalization directly into the network. Leveraging diverse training data, the model combines the abundance of 1.0 mm scans with the fine detail of HiRes data (0.7–0.9 mm). This not only enables generalization to resolutions within the training range, but also to those outside it (from 0.7 mm to 1.6 mm), while outperforming fixed-resolution models. The authors further demonstrated that scaling the training set from 120 to ∼1300 scans significantly improves performance across all resolutions, even under severe resolution imbalance. Similarly, Simarro et al. [9] showed that training on diverse age groups (2–81 years) yields a single model, with performance on par with age-specific models, facilitating consistent segmentation methodology in developmental studies.

A powerful generalization strategy is to leverage large and diverse MRI datasets. In the simpler task of brain extraction, Fletcher et al. [153] systematically showed that training data diversity is the most critical factor, followed by dataset size, with network complexity playing a lesser role. Their model achieved “production-level” performance on diverse data when trained on 8 cohorts and thousands of scans. For brain segmentation, Svanera et al. [52] similarly demonstrated the power of diverse training data, while observing a performance plateau across sites with 32–70 training datasets and a slight performance drop on individual sites. More volumes per dataset improved accuracy for both seen and unseen sites, with diminishing returns after ∼1000 scans (further extensively augmented). Their model showed excellent generalizability and outperformed QuickNat, FastSurfer, and SynthSeg on motion artifacts, and several established traditional methods on single-hemisphere subjects.

Several tissue segmentation studies report good performance on internal datasets using scarce training data. Narayana et al. [144] systematically showed good segmentation with just 10 training scans, although segmenting T2 MS lesions required hundreds. Importantly, domain-adaptation may benefit from fine-tuning: Roy et al. [47] fine-tuned on 28 manually segmented scans, Karani et al. [94] on 4 scans using domain-specific batch normalization, and Sun et al. [64] bridged labeling differences using a single image.

3. Results

We present the validation techniques found in brain MRI segmentation studies and discuss their limitations in Section 3.1. We discuss the results obtained by specific studies in Section 3.2.

3.1. Validation Strategies

Robust validation is essential for evaluating model accuracy, generalizability, and potential for clinical application. Nevertheless, validation remains a fundamental challenge in brain MRI segmentation, in large part due to the scarcity of manually annotated datasets.

The most common approach to validation in the works in our review is the direct comparison to ground truth using the DSC and Hausdorff distance [154]; the 95th percentile Hausdorff distance and the average Hausdorff distance [154] are frequently utilized to reduce sensitivity to outliers. Related metrics include the volumetric similarity coefficient [154], absolute volumetric difference [127], average surface distance [48], and average symmetric surface distance [69]. While the DSC was reported in all ground truth comparisons, we note that it is highly sensitive to small voxel-level differences, limiting its relevance for small subcortical structures, and does not account for the spatial distribution of errors; Taha et al. provide a discussion of the metrics [154].

The golden ground truth in direct comparison involves segmentations produced manually by medical experts. For tissue segmentation, the most commonly utilized publicly available datasets are MRBrainS13 [155] (20 healthy subjects), [155] (23 subjects with pathologies), and IBSR 18 [156] (18 healthy subjects), which also include segmentations into 8, 9, and 32 structures, respectively. In structure segmentation, high-quality manual datasets additionally include Mindboggle101 [37] (101 healthy subjects, 95 structures) and MICCAI 2012 (MALC) [157] (30 healthy subjects), 20 repeat (20 healthy subjects, two scans each), ADNI30 (15 healthy subjects and 14 AD patients), and OASIS30 (30 healthy subjects), which includes 132 structures [158].

Despite their importance, manual labels are in general susceptible to inter- and intra-rater variability issues and errors, and leave out many pathological populations. Notably, it is known that IBSR 18 tissue labels contain sulcal CSF voxels misclassified as GM [159], while Svanera et al. [52] completely excluded the manual structure annotations of MALC and IBSR due to quality concerns and replaced them with FreeSurfer outputs, retaining only Mindboggle-101 labels. Similarly, Kaku et al. [95] report artifacts of MALC labels in saggital and axial views, as the manual segmentation was performed in the coronal plane, and use Mindboggle101 instead. On the other hand, Cao et al. [69] report rough contours in Mindboggle-101. Some works utilize their own proprietary manual annotations, often to tailor validation to specific populations [9], resolutions [48], and field strengths [66,108]. However, such approaches limit reproducibility and comparability across studies. Therefore, a critical need remains for diverse, high-quality manual labels, particularly for diverse populations and acquisition parameters.

Numerous studies include ’silver’ ground truth for validation. These are segmentations obtained by using established tools such as FSL FAST [38,62,132,150,160], MRIAP [144,161], SPM [132,139], and FreeSurfer [24,28,125] for tissue segmentation and, most commonly, FreeSurfer for structure and ROI segmentation; tools such as ANTS [73,104,152,162], GIF [57,163], NLSS [96,164], icobrain [9,41,165], and SPM [121] are also used to produce structure annotations. Rajchl et al. [132] use outputs from multiple tools for training and validation. However, the ground truth obtained from established methods includes systematic errors and shows variability across software versions and even platforms, significantly limiting their reliability [117] and reducing the significance of direct comparisons [130]. Billot et al. [117], therefore, only considers the DSC w.r.t FreeSurfer segmentations valid only up to a specific threshold. Notably, many deep learning methods outperform these well-established tools in terms of accuracy against manual labels, test–retest reliability, sensitivity to clinical effects, and qualitative analysis [26,47,61,66,69,80], making silver-standard-based validation increasingly insufficient for meaningful comparison. Worryingly, numerous works completely rely on silver ground truth and do not include comparisons to manual data or perform additional tests.

To supplement direct comparison to ground truth labels, several works perform additional statistical tests. Test–retest reliability is estimated by computing the similarity of segmentations across two or more scans taken in a short time-frame. It is commonly utilized to assess stability in a scenario where anatomy is expected to remain stable. Certain studies quantify reliability directly on segmentations using metrics such as the DSC and Hausdorff distance [98,104], with Kondrateva et al. [166] stressing the importance of using multiple metrics to capture different aspects of segmentation quality. On the other hand, other works perform volume estimation instead and utilize the intra-class correlation coefficient, absolute volume difference, and percent relative difference metrics [26,110,121]. Puzio et al. [123] motivate the latter approach by the high sensitivity of the DSC to minor variations along structure boundaries, which may lack clinical relevance. Furthermore, Worth et al. [158] have shown the test–retest DSC of expert manual segmentations to be limited; Novosad et al. [98] achieve a highly similar coefficient (86.6%) to that reported in Worth et al. on several subcortical structures.

Several works perform sensitivity analyses to estimate the capability to detect known variations in brain morphology between diagnostic or treatment groups. For example, group differences between controls, individuals with mild cognitive impairment, and AD patients are estimated utilizing GM atrophy, volume loss in subcortical structures (namely the hippocampus, amygdala, and thalamus), ventricular enlargement, and thinning of cortical regions of interest. Sensitivity is evaluated by computing effect sizes [47] or fitting linear models and reporting p-values [26]. In MS, gray matter volumes have been used to estimate the sample sizes needed to detect treatment effects [167].

An important property of models is their generalizability to different imaging sites, populations, and acquisition protocols, especially in ensuring their practical utility in real-world clinical scenarios and large-scale multi-center studies. Many works therefore evaluate their models on diverse datasets, incorporating scans from multiple sites, scanners, and subject groups. Importantly, to assess robustness to domain shifts, models are trained on data with limited diversity and tested on that with characteristics not encountered during training, such as unseen scanner types [41,115], acquisition protocols [38,50], resolutions [48], demographics, or clinical conditions [26,47,107]. Robustness is also evaluated on simulated artifacts [52], unseen MRI modalities or CT [117], and single-hemisphere brains [52]. Furthermore, test–retest analysis on scans acquired by different scanners can be used to evaluate a model’s applicability across various scanning conditions [123].

Quantitative analyses are criticized by human experts, such as some physicians and neuroscientists, as they do not necessarily account for the severity of each segmentation error [154]. Kaku et al. [95] therefore measure segmentation agreement between their model and a group of medical experts, while Bontempi, Svanera et al. [80,108] include a “Turing test” in which they survey a group of experts to choose the best segmentation results among the competing methods.

3.2. Results

Validation practices in brain MRI segmentation exhibit substantial variability in terms of methods, datasets, and metrics, with comparison across works further complicated by the use of different training approaches and validation datasets.

The highest DSC reported by structure and tissue segmentation studies are summarized in Table 1 and Table 2, respectively. For each segmentation type, box plots of the DSC by input size and by dimensionality are shown in Figure 3, and the mean differences and p-values (Mann–Whitney U test) between methods are summarized in

Table 3. Comparison of segmentation methods across input size and dimensionality for structure and tissue segmentation. $\Delta$ Mean indicates the difference between the mean DSC of method groups 1 and 2.

Segmentation Type	Comparison	Method 1	Method 2	$\mathbf{\Delta }$ Mean	p-Value
Structure	Input Size	Patch	Full	0.3	0.41
		2D	2.5D	−1.67	0.56
	Dimensionality	2D	3D	−1.1	0.58
		2.5D	3D	0.56	0.83
Tissue	Input Size	Patch	Full	−0.02	0.97
		2D	2.5D	6.1	0.7
	Dimensionality	2D	3D	2.7	0.36
		2.5D	3D	−3.4	0.34

. The boxplots provide a descriptive overview of performance trends, with the lack of statistical significance (all p-values > 0.34) expected due to the limited number of studies.

In tissue segmentation, most studies utilized the manually annotated IBSR18, MRBrainS13, and MRBrainS18 datasets and the simulated BrainWeb dataset [168] for ground truth reference. Structural segmentation studies demonstrated a much greater diversity in both validation strategies and test datasets, with considerable variation in the number and definitions of anatomical labels. To provide a limited but meaningful comparison, we report the performance results for the methods evaluated on the Mindboggle101 dataset [37], the manually segmented dataset with the largest number of scans and with few reported quality concerns. The dataset consists of 101 scans with manually corrected labels based on the DKT cortical labeling protocol [37], including 33 subcortical and 62 cortical labels. T1 scans of healthy adults (age: 19–61) were collected specifically for the dataset or from a diverse set of publicly available studies, with field strengths ranging from 1.5 T to 7 T. In

Table 4. Summary of studies employing Mindboggle101 for validation, including evaluated methods, use of Mindboggle101 in training, number of anatomical labels, and reported Dice similarity coefficient (DSC).

Reference	Method	Training	Labels	DSC (%)
Kaku et al. [95]	DenseUNet	No	102	74.31
		Yes	102	81.9
	U-Net	No	102	73.29
		Yes	102	80
Henschel et al. [26]	FastSurfer	No	33 subcortical	80.19
			62 cortical	80.65
	3D U-Net		33 subcortical	78.65
			62 cortical	79
Li et al. [109]	AceNet QuickNat v2	Yes	62 cortical	82.5 77.7
Laiton-bonaidez et al. [70]	Proposed	Yes	37	75
Liu et al. [112]	CLMorph	No	5 combined	64.6
Henschel et al. [48]	FastSurferVINN	No	33 subcortical	80.06
			62 cortical	81.89
	FastSurfer		33 subcortical	80.06
			62 cortical	81.23
Cao et al. [69]	TABSurfer	Yes	31 subcortical	79.2
	FastSurfer	No		75.8
	FreeSurfer	\		74
Kujawa et al. [120]	Proposed	No	108	74
Lorzel et al. [25]	AutoParch	Yes	58 combined	77.8
	FreeSurfer			85.7
Svanera et al. [52]	LOD-Brain	No	7 combined	95.5
	FastSurfer			96.5
Diaz et al. [119]	e3nn	Yes	7 subcortical	0.88
	nnUnet			0.89

, we show the DSC reported by all nine structure segmentation publications that validate on Mindboggle101. For each work, we report the models evaluated on the dataset, whether the dataset was utilized in training, the number of output structure labels, and the DSC, which is the most commonly used metric among these works.

The models in Table 4 segment the brain into different anatomical labels. The highest reported DSC, 96.5%, was achieved by FastSurfer, as reported by Svanera et al. [52], who grouped the finer-grained DKT labels into seven broader classes. The authors of FastSurfer [26] obtained DSCs of ∼80% for both cortical and subcortical segmentation, with 95 labels in total, while the FastSurferVINN model obtains higher performance [48]. Cao et al. report a noticeably lower score of 75.8% for FastSurferVINN on the same subcortical labels, but with the largest structure (cortical white matter) excluded; FreeSurfer, the most frequently utilized traditional method, achieves a DSC of 74%. Some studies in Table 4 incorporated Mindboggle101 scans into training, generally reporting higher DSC, with Kaku et al. [95] demonstrating a large improvement of approximately 7% following fine-tuning.

The results in Table 4 are strongly influenced by the choice of output classes and training strategies. We note that, apart from the DSC, the set of reported evaluation metrics varies considerably across the studies. Together, these factors highlight the difficulty of comparisons across works, even when the models are validated on the same dataset.

4. Discussion

Deep learning methods have emerged as powerful tools for brain MRI segmentation, offering advantages in computational efficiency, robustness, and accuracy compared to conventional approaches. Nevertheless, deep learning models remain underutilized in neurological studies and clinical trials, which still feature traditional approaches such as atlas-based methods. This is in part due to a much larger volume of studies validating conventional methods, and is also due to greater robustness across diverse image conditions and populations and the easier interpretability of such methods. With the growing availability of publicly available MRI datasets and the use of conventional methods to obtain silver-truth training data, deep learning models have started to show very high consistency across sites [52].

In our review, we observed a steady increase in publications on deep learning-based brain MRI segmentation, reflecting a growing interest in the field. Early models were constrained by limited GPU memory, often relying on 2D architectures that processed individual slices or small blocks of slices. While computationally feasible at the time, such methods are unable to capture full 3D anatomical context. In structure segmentation, we found that models have primarily shifted toward 3D processing. Furthermore, 2.5D architectures are also utilized and offer a compromise between computational feasibility and context utilization, with studies reporting state-of-the-art results from such models in whole brain segmentation [48]. Our survey also revealed a clear progression from small patch-based inputs to very large patches or entire volumes. Some studies reported improved accuracy when using very large patches instead of full volumes [36], suggesting that the strategy may confer benefits beyond computational feasibility.

Architectural developments in brain MRI segmentation have closely paralleled progress in the broader medical image segmentation. Early approaches were based on sliding-window classification and fully convolutional networks. The U-Net architecture has since become the dominant backbone and remains central to most modern brain MRI segmentation methods. Numerous additions and alternatives to the architecture have been proposed, with the most recent being the transformer and state space models. However, the benefit of many modifications over optimized U-Net models remains uncertain, with performance gains often dependent on specific datasets and evaluation conditions. As in the broader medical segmentation landscape [76], there is a need for greater validation and comparisons of different models.

Several works have focused on improving the generalization capabilities of brain MRI segmentation models. Strategies such as data augmentation, domain randomization, training on large and diverse datasets, and fine-tuning on limited data have produced robust performance across sites, protocols, and populations. A consistent set of findings emerge across studies: diverse training cohorts are essential for broad generalization, do not reduce performance on individual datasets, and may even enhance it; increasing the number of training scans improves performance, but with diminishing returns; and dataset size imbalances do not necessarily disadvantage smaller cohorts. Moreover, targeted fine-tuning may be effective with a small number of scans. Together, these insights highlight the power of dataset diversity and scale, complemented when needed by small-scale fine-tuning.

Validation strategies varied widely across studies, but showed important limitations. Manually labeled datasets remain small, biased toward healthy populations, and often contain segmentation errors, underscoring the need for high-quality annotations across diverse cohorts and increasingly common MRI modalities such as submillimeter imaging. Furthermore, many structure segmentation works rely on imperfect silver-standard references rather than manual labels. Statistical measures such as test–retest reliability, group differences, and longitudinal sensitivity are sometimes used to supplement evaluation, while qualitative assessment by independent experts remains rare. Standardized frameworks like nnU-Net [75,76], which provide rigorous validation, have been used in some studies but remain underutilized. Their adoption is strongly recommended, as they improve reproducibility and comparisons across works, helping mitigate evaluation bias.

The diversity of validation techniques complicates cross-study comparisons. As shown in Table 4, comparison across structure segmentation studies remains challenging even when the same test sets are utilized, owing to differences in training strategies, output labels, and evaluation metrics. There is therefore a clear need to unify validation approaches for method comparison.

Despite substantial progress in accuracy and efficiency, the practical adoption of deep learning-based brain MRI segmentation remains limited. Many methods are not trained or evaluated for robustness across diverse acquisition parameters and populations, a key concern for widespread applicability. Only a few works explicitly address the increasingly used submillimeter acquisitions [48] or the anisotropic low-resolution scans [116] widely used by clinicians. Models are typically tied to a fixed set of labels, offering less flexibility than established methods such as SPM and CAT, which can register to multiple atlases. Independent large-scale validation studies remain scarce, and many methods lack thorough documentation or even public availability. Like conventional tools that have been refined for decades, deep learning models would benefit from continuous development to keep pace with evolving user needs and MRI advances. Together, these issues highlight the gap between promising research results and routine, reliable deployment. Tools like FastSurfer [48], SynthSeg+ [116], and LOD-Brain [52] have taken strong steps in this direction, providing extensive documentation and validation across diverse datasets. FastSurfer has been continuously updated, introducing resolution-independence and subfield segmentation. SynthSeg+ offers high performance on both GPUs and CPUs, automatic quality control, and support for clinical scans, and has been incorporated into the FreeSurfer toolbox.

Future work in brain MRI segmentation should leverage advancements in MRI. The increasing availability of ultra-high-field 7T MRI, offering superior tissue contrast for subcortical delineation, and submillimeter imaging, which can capture fine cortical folding patterns, remains underexploited, despite its potential to enhance segmentation accuracy. In addition, current efforts toward improving generalization should be extended into robust pipelines capable of handling scans acquired under diverse conditions, spanning different scanners, populations, age groups, and imaging modalities. Longitudinal brain segmentation, which utilizes multiple scans of the same patient, is used by conventional methods to reduce sensitivity to noise and better monitor changes due to age, disease, and treatment. This strategy, however, remains very underexplored in deep learning literature. Critically, efforts to strengthen validation are essential. Larger manually annotated datasets, derived from diverse cohorts and imaging settings, are needed alongside the unification of validation protocols. Expanded benchmarking and comparative analyses will be crucial to rigorously assess model performance, disentangle the influence of architecture from training configurations, and ultimately support the reliable deployment of deep learning-based segmentation in clinical trials, neuroimaging studies, and clinical practice. Broader adoption in practice will also benefit from more comprehensive documentation, compatibility with existing neuroimaging pipelines and downstream tasks, and the public availability of implementations.

5. Conclusions

Deep learning has markedly advanced brain MRI segmentation, enabling fast, accurate, and scalable analysis. Models have evolved from slice- and patch-based CNNs to 3D and 2.5D U-Nets and emerging transformer-based architectures. Recent generalization techniques have enabled remarkable robustness across diverse data. Despite state-of-the-art accuracy, deep learning methods remain underutilized in large studies and clinical practice. Key challenges include the limited application of generalization strategies, lack of independent validation, difficult cross-study comparability, and a scarcity of high-quality manual segmentations. Moreover, most methods lack thorough documentation, regular updates, or integration into broader neuroimaging tools. Future work should focus on expanding diverse, high-quality datasets, unifying validation protocols, leveraging MRI advancements, and developing robust models that generalize across populations and scanners. Addressing these issues will facilitate broader adoption in research and clinical practice, supporting more reliable neuroimaging studies and improved patient outcomes.