A Hybrid Particle Swarm–Genetic Algorithm Framework for U-Net Hyperparameter Optimization in High-Precision Brain Tumor MRI Segmentation

Abstract

Accurate and robust brain tumor segmentation remains a critical challenge in medical image analysis due to high inter-patient variability, complex tumor morphology, and modality-specific noise in MRI scans. This study proposes PSO-GA-U-Net, a novel hybrid deep learning framework that integrates Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs) to optimize the U-Net architecture, enhancing segmentation performance and generalization. PSO dynamically tunes the learning rate to accommodate modality-specific variations, while the GA adaptively regulates dropout to improve feature diversity and reduce overfitting. The model was evaluated on three benchmark datasets—FBTS, BraTS 2021, and BraTS 2018—using five-fold cross-validation. PSO-GA-U-Net achieves Dice Similarity Coefficients (DSC) of 0.9587, 0.9406, and 0.9480 and Jaccard Index (JI) scores of 0.9209, 0.8881, and 0.9024, respectively, consistently outperforming state-of-the-art models in both overlap accuracy and boundary delineation. Statistical tests confirm that these improvements are significant across folds ($p<0.05$). Visual heatmaps further illustrate the model’s ability to preserve structural integrity across tumor types and modalities. These results indicate that metaheuristic-guided deep learning offers a promising and clinically applicable solution for automatic tumor segmentation in radiological workflows.

1. Introduction

Brain tumors pose significant clinical and radiological challenges due to their aggressive progression [1], diverse histopathological characteristics [2], and heterogeneous presentation in magnetic resonance imaging (MRI) [3]. Accurate and timely segmentation of brain tumors is essential for treatment planning, surgical navigation, and post-operative assessment [4,5,6]. Manual delineation by expert radiologists, although precise, is labor-intensive and subject to intra- and inter-observer variability [7,8,9], particularly in multimodal MRI where subtle boundary differences and modality-specific artifacts frequently occur [2,10,11]. These challenges have propelled the development of automated segmentation systems, with deep learning methods—especially convolutional neural networks (CNNs)—emerging as the de facto standard for high-throughput tumor delineation [12,13].

Among CNN-based architectures, U-Net and its variants have demonstrated remarkable performance in biomedical segmentation tasks due to their encoder–decoder structure with skip connections that preserve spatial context [14,15,16,17]. However, despite their widespread adoption, U-Net models often exhibit limited generalization across datasets due to static architecture configurations, sensitivity to hyperparameter settings, and vulnerability to overfitting [18,19]—especially when exposed to heterogeneous tumor morphology or imbalanced multimodal inputs [20,21]. These challenges are further exacerbated in real-world clinical datasets where MR acquisitions may vary in contrast, noise levels, and tumor size.

To overcome these limitations, recent studies have explored metaheuristic optimization strategies, such as Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs), to tune deep network hyperparameters adaptively. PSO is known for its global convergence ability in continuous optimization problems [22,23,24], while GA [25,26,27,28] offers robust exploration through population-based dropout and structural tuning. However, these methods are often applied independently, without harnessing their complementary strengths. Moreover, prior work lacks an integrated framework that optimizes both learning dynamics and architectural regularization to improve boundary precision and robustness across diverse tumor types.

In this work, we propose a novel hybrid optimization-based deep learning framework, named PSO-GA-U-Net, which leverages PSO for dynamic learning rate adaptation and a GA for dropout regulation within the U-Net architecture. This dual-strategy optimization enhances the model’s ability to capture fine-grained tumor boundaries while preserving global contextual integrity. The model was extensively evaluated on three benchmark datasets—FBTS (Figshare Brain Tumor Segmentation), BraTS 2021, and BraTS 2018—across multiple tumor classes and MRI modalities. Experimental results demonstrate that PSO-GA-U-Net consistently outperforms state-of-the-art (SOTA) models in the Dice Similarity Coefficient (DSC), Jaccard Index (JI), and boundary preservation. Statistical significance tests confirm the reliability of improvements across five-fold cross-validation settings, while qualitative heatmaps further illustrate the structural fidelity of the model in complex tumor regions.

The main contributions of this study are summarized as follows:

• We designed a novel hybrid metaheuristic optimization framework (PSO-GA-U-Net) that integrates PSO and a GA to jointly optimize the learning rate dynamics and dropout rates in U-Net for brain tumor segmentation.
• We implemented a robust training pipeline validated across three challenging datasets (FBTS, BraTS 2021, BraTS 2018), ensuring generalization across tumor types, MRI modalities, and anatomical variations.
• We conducted extensive performance evaluations using both quantitative metrics (DSC, JI, HD-Hausdorff Distance, and ASSD-Average Symmetric Surface Distance) and qualitative heatmap analyses and performed statistical validation with paired t-tests on cross-validation folds.
• We present a comprehensive comparison with SOTA models, demonstrating superior performance in volumetric overlap, boundary precision, and robustness to modality-specific noise.

The remainder of the paper is structured as follows: Section 2 presents an overview of related works, highlighting recent developments in deep learning and metaheuristic optimization for medical image segmentation. Section 3 details the proposed PSO-GA-U-Net architecture, the hybrid optimization strategy, and the experimental setup. Section 4 presents the quantitative and qualitative results, comparative evaluations, and statistical analyses. Finally, Section 5 concludes the paper with a summary of the findings, a discussion of limitations, and directions for future research.

2. Related Works

The field of automated brain tumor segmentation has evolved significantly with the rise of deep learning methods, particularly convolutional neural networks (CNNs) [8,29]. Among these, the U-Net architecture has become a foundational backbone due to its encoder–decoder structure and skip connections, which enable multiscale feature fusion [30,31,32]. However, traditional U-Net variants often suffer from rigidity in parameter selection and limited adaptability to complex tumor morphologies and multimodal MRI inputs [15,18,33].

To enhance segmentation performance, numerous studies have focused on architectural modifications to U-Net. Attention-based models such as U-Net-AG [31,34,35,36] integrate spatial attention gates to guide the model’s focus toward salient tumor regions. Residual and dense connectivity strategies, as employed in Residual-Attention-U-Net [8,37,38,39,40], aim to improve gradient flow and multi-scale representation. However, these models are often manually tuned and sensitive to overfitting when confronted with high variability in tumor intensity and shape. Recent works, such as DeepLabV3+ [41,42] and SPPNet [43], introduce atrous spatial pyramid pooling to address multiscale features; however, they lack adaptive learning dynamics.

To address hyperparameter sensitivity, researchers have turned to bio-inspired optimization techniques [44]. Particle Swarm Optimization (PSO), inspired by collective behavior in swarms, has been applied to optimize learning rates [18,24], loss function weights [45,46], and even thresholding in preprocessing pipelines [24,44,47]. PSO has demonstrated success in efficiently tuning continuous-valued parameters, but can suffer from premature convergence in high-dimensional search spaces.

On the other hand, Genetic Algorithms (GAs) excel in exploring diverse architectural variants through population-based evolution strategies [48,49,50]. In segmentation tasks, GAs have been used to evolve filter configurations [51,52], dropout rates, and layer depths [53,54]. Despite its global search capability, GA performance can degrade when used alone due to lack of convergence guidance or redundancy in feature activation [55,56].

Several studies have explored hybrid optimization to mitigate these individual shortcomings. Works such as PSO-U-Net [24] and GA-U-Net [53,57] demonstrate that integrating optimization algorithms into deep learning pipelines can yield better generalization. However, most prior methods employ these techniques in isolation or apply them to static configurations, without a coordinated mechanism to optimize both training dynamics (e.g., learning rate schedules) and regularization factors (e.g., dropout schemes) simultaneously.

On benchmark datasets, models such as ViT-self-attention [58] and AWA-VGG-19 [59] have achieved competitive results on the BraTS dataset. However, transformer-based models often require extensive training data and struggle to refine local features, particularly in edematous or necrotic tumor subregions. Cascaded architectures such as 2C-U-Net [60] and U-Net-ASPP-EVO [61] improve multi-resolution learning, but are computationally intensive and sensitive to modality imbalance.

The proposed PSO-GA-U-Net differs from existing approaches in several key ways:

• In contrast to previous PSO or GA models, it introduces a dual-level optimization pipeline where PSO governs adaptive learning rate scheduling during training. At the same time, the GA dynamically evolves the dropout rate to prevent overfitting and promote feature diversity.
• The architecture remains U-Net-based for its proven spatial fidelity, but is optimized across cross-validation folds and datasets to ensure both performance and generalization.
• The evaluation covers diverse tumor types (meningioma, glioma, and pituitary, HGG, LGG) and MRI modalities (T1CE, FLAIR, T1, and T2), with consistent superiority in DSC and JI compared to transformer-based, residual, and attention-enhanced models.
• Statistical significance tests ($p<0.05$) were used to validate the robustness of improvements across datasets and tumor classes.

In summary, while various strategies have been employed to improve tumor segmentation—ranging from architectural innovations to standalone optimizers—our work uniquely combines adaptive learning and architectural regularization into a single, hybrid metaheuristic framework. This positions PSO-GA-U-Net as a robust and scalable solution for clinical-grade segmentation in MRI-based tumor analysis.

3. Methods

3.1. Dataset Description

To comprehensively evaluate the robustness, adaptability, and generalization capability of the proposed PSO-GA-U-Net framework, three publicly available brain tumor segmentation datasets were used: the Figshare Brain Tumor Segmentation (FBTS) dataset, the BraTS 2021 dataset, and the BraTS 2018 dataset. These datasets vary in tumor types, imaging modalities, and annotation granularity, providing a diverse and challenging experimental setup. All datasets were preprocessed to a uniform spatial resolution of $256\times 256$ pixels and normalized to the range [0, 1] before training.

3.1.1. Figshare Brain Tumor Segmentation (FBTS) Dataset

The FBTS dataset consists of T1-weighted contrast-enhanced (T1CE) axial MRI slices sourced from a curated clinical dataset [62]. It contains manually annotated tumor masks for three tumor types: meningioma, glioma, and pituitary adenoma. Each slice is associated with a binary mask that delineates the tumor region, as illustrated in Figure 1.

• Meningioma: 3060 slices;
• Glioma: 708 slices;
• Pituitary Tumor: 930 slices.

Figure 1. Representative examples from the FBTS dataset: meningioma, glioma, and pituitary tumors with their respective binary masks.

Figure 1. Representative examples from the FBTS dataset: meningioma, glioma, and pituitary tumors with their respective binary masks.

Tumor sizes and shapes vary significantly across classes, with gliomas exhibiting irregular, infiltrative structures, and pituitary tumors generally appearing as small, well-circumscribed masses. Each image is single-channel but was expanded to three channels during preprocessing to meet the deep learning model’s input requirements. The binary masks were thresholded and rescaled to ensure compatibility with the sigmoid output of the segmentation network.

3.1.2. BraTS 2021 Dataset

The BraTS 2021 dataset comprises axial MRI slices acquired using four distinct modalities: T1, T2, T1CE (post-contrast), and FLAIR [63]. Each sample (Figure 2) is annotated with a label representing the tumor region, which in our study was converted into a binary whole tumor mask. This dataset is particularly relevant for evaluating modality-specific performance as each MRI sequence highlights different pathological characteristics:

• T1 emphasizes anatomical structures.
• T2 highlights fluid-filled regions.
• T1CE captures active tumor enhancement.
• FLAIR is sensitive to edema and non-enhancing tumor cores.

Figure 2. Sample from BraTS 2021 showing all four MRI modalities (T1, T2, T1CE, and FLAIR), the colored multi-label mask, and its binarized whole tumor counterpart.

Figure 2. Sample from BraTS 2021 showing all four MRI modalities (T1, T2, T1CE, and FLAIR), the colored multi-label mask, and its binarized whole tumor counterpart.

All samples were processed as independent 2D axial slices, preserving the integrity of the modality throughout evaluation. The ground truth masks were derived from voxel-level segmentations and combined to form whole tumor regions.

3.1.3. BraTS 2018 Dataset

To examine the performance of the proposed model across tumor grades, the BraTS 2018 dataset was utilized (Figure 3). This dataset includes separate cases for high-grade gliomas (HGGs) and low-grade gliomas (LGGs) [64,65,66]. The MRI modalities used are consistent with BraTS 2021 (T1, T2, T1CE, and FLAIR). Each slice is labeled with expert-annotated tumor masks focusing on the entire tumor structure. This dataset provides an essential test of the model’s ability to handle intra-tumoral variability and subtle morphological differences between tumor grades.

Each MRI slice is accompanied by a corresponding ground-truth mask indicating the tumor region. The masks are binary images in which tumor pixels are labeled 1 and background pixels 0.

An analysis of the pixel distribution across the FBTS segmentation masks reveals a strong class imbalance. Tumor pixels constitute approximately 1.69% of total pixels, while background pixels account for 98.31%. Across individual images, tumor regions occupy on average 1.69 ± 1.38% of pixels.

3.1.4. Preprocessing and Augmentation Pipeline

To ensure robust feature extraction and effective generalization across heterogeneous brain tumor images, a comprehensive preprocessing and augmentation pipeline was applied [67,68]. The preprocessing steps were designed to normalize intensity distributions [69], standardize spatial dimensions [70], and prepare data for hybrid optimization training [71]. The entire pipeline can be mathematically summarized through the following stages:

(1)

Intensity Normalization

All input MR slices $I(x,y)$ were normalized to the [0, 1] range via min–max normalization (Equation (1)).

$$I_{\mathrm{norm}}(x,y)={\displaystyle \frac{I(x,y)-min\left(I\right)}{max\left(I\right)-min\left(I\right)}}$$

(1)

For the Figshare dataset, grayscale T1CE slices were replicated into three identical channels to form $I_{\mathrm{norm}}^{\left(3\right)}(x,y)$. For the BraTS datasets, normalization was performed independently for each modality (T1, T2, FLAIR, and T1CE), preserving contrast-specific features.

(2)

Spatial Standardization and Binarization

Each image $I_{\mathrm{norm}}$ and mask M was resized to a fixed dimension of $256\times 256$ pixels using bilinear interpolation for the image and nearest-neighbor interpolation for the mask (Equation (2)).

$$I_{\mathrm{resized}}(x^{\prime },y^{\prime })={\mathrm{Interp}}_{\mathrm{bilinear}}\left(I_{\mathrm{norm}}(x,y)\right),M_{\mathrm{resized}}(x^{\prime },y^{\prime })={\mathrm{Interp}}_{\mathrm{nn}}\left(M(x,y)\right)$$

(2)

The resized masks were thresholded into binary format using Equation (3).

$$M_{\mathrm{bin}}(x,y)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}M(x,y)\ge 0.5\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(3)

This binarization ensures compatibility with the output layer of our segmentation network using a sigmoid activation.

(3)

Data Augmentation Strategy

To introduce structural variability and mitigate overfitting, we applied a probabilistic augmentation pipeline $\mathcal{A}$ to the training set (Equation (4)).

$$\tilde{I},\tilde{M}=\mathcal{A}(I_{\mathrm{resized}},M_{\mathrm{bin}}),$$

(4)

where $\mathcal{A}$ includes the following transformations:

• Rotation: $\theta \sim \mathcal{U}(-{15}^{\circ },+{15}^{\circ })$;
• Flipping: ${\mathrm{Pr}}_{\mathrm{flip}}=0.5$ (horizontal and vertical);
• Zoom: Scaling factor $s\sim \mathcal{U}(0.9,1.1)$;
• Translation: Offset $(\Delta x,\Delta y)\sim \mathcal{U}(-10,10)$;
• Elastic deformation: Applied via a random displacement field $\varphi (x,y)$ with Gaussian smoothing;
• Contrast and Brightness Shift: $I^{\prime }=\alpha I+\beta$ where $\alpha \sim \mathcal{U}(0.9,1.1)$, $\beta \sim \mathcal{U}(-0.1,0.1)$.

All augmentation operations were applied synchronously to images and their corresponding masks to preserve spatial alignment.

Data augmentation was applied exclusively to the training subset, while the validation and test sets remained unchanged to ensure unbiased performance evaluation.

(4)

Label Harmonization and Whole Tumor Mask Generation

In the BraTS 2021 and 2018 datasets, segmentation masks comprise three distinct regions: enhancing tumor ($ET$), non-enhancing tumor core ($NET$), and peritumoral edema ($ED$). For binary segmentation, we generated a unified whole tumor mask $M_{\mathrm{whole}}$ as in Equation (5).

$$M_{\mathrm{whole}}(x,y)={⊮}_{\left\{ET\right(x,y)\cup NET(x,y)\cup ED(x,y\left)\right\}},$$

(5)

where $⊮$ denotes the binary indicator function. This harmonization facilitates a consistent comparison across all datasets.

(5)

Dataset Partitioning and Cross-Validation

Each dataset was divided into 80% training, 10% validation, and 10% testing [72,73]. This configuration was selected to ensure sufficient data availability for learning complex spatial features while preserving dedicated subsets for optimization and unbiased evaluation. The 80% training portion allows the PSO-GA-driven U-Net to converge effectively on high-dimensional tumor patterns, especially important for diverse MRI modalities and inter-patient variability.

The 10% validation subset was used exclusively for fitness computation during the PSO-GA optimization process, ensuring convergence monitoring without leaking into the final test evaluation. The remaining 10% test set was strictly held out and used only for final performance assessment.

The PSO-GA optimization procedure was executed once using the predefined training-validation split to identify the optimal hyperparameter configuration. After determining the best-performing PSO-GA-U-Net model, the optimized configuration was evaluated using a five-fold cross-validation ($k=5$) procedure to assess stability and generalization capability. Hyperparameter tuning was not performed within the cross-validation folds, ensuring that the evaluation phase remained independent of the optimization process.

During cross-validation, subject-level separation was preserved, ensuring that all slices from the same patient remained within a single fold. All reported metrics—Dice Similarity Coefficient (DSC), Jaccard Index (JI), Hausdorff Distance (HD), and Average Symmetric Surface Distance (ASSD)—were averaged across folds. Statistical analysis using paired t-tests with a significance level of $\alpha =0.05$ was used to assess the consistency of the improvements. Because the statistical comparison is based on five cross-validation folds ($n=5$), the resulting p-values should be interpreted as indicative rather than definitive evidence and are complemented by consistent performance trends across folds.

(6)

Consistency Checks

To ensure data integrity post-augmentation, all segmentation masks were subjected to consistency validation using connected component analysis [74]. Only masks exhibiting a single contiguous tumor region were retained, and corrupted or multi-labeled instances were automatically excluded [75]. This quality control step enforces anatomical plausibility and preserves label fidelity, thereby ensuring robust training signals for the PSO-GA-U-Net framework.

(7)

Preprocessing and Augmentation Pipeline Summary

The complete preprocessing and augmentation procedure is illustrated in Figure 4. The pipeline began by extracting raw axial slices from the FBTS, BraTS 2021, and BraTS 2018 datasets, encompassing four MRI modalities: T1-weighted, T2-weighted, contrast-enhanced T1 (T1CE), and FLAIR. All images were converted to grayscale, resized to a uniform resolution of $256\times 256$ pixels, and stacked into three-channel representations to ensure compatibility with convolutional operations.

The corresponding ground truth masks were also resized and binarized to facilitate binary segmentation. For training augmentation, we applied affine transformations such as horizontal flipping, random rotations ($\pm {15}^{\circ }$), and zooming (scale factor in [0.9, 1.1]) to enhance model generalization and mitigate overfitting.

Subsequently, the dataset was split into training (80%), validation (10%), and testing (10%) subsets, as described in Section Dataset Partitioning and Cross-Validation, ensuring that class balance was preserved. This standardized pipeline unified heterogeneous data sources and prepared them for robust training of the PSO-GA-U-Net framework.

During the hyperparameter optimization phase, the PSO–GA framework was executed once using the training–validation subset derived from the initial 80/10/10 data partition. Candidate configurations were evaluated using the validation Dice Similarity Coefficient (DSC), while the test set remained completely isolated during the search process.

After identifying the optimal configuration, the model was retrained using the optimized parameters and evaluated through five-fold cross-validation to assess the stability of the learned configuration across different data partitions. Although nested cross-validation could provide a fully unbiased evaluation of hyperparameters, performing evolutionary optimization within each fold would significantly increase the computational cost. Therefore, cross-validation was used primarily to evaluate robustness rather than to perform nested hyperparameter selection.

3.2. PSO-GA Hybrid Framework

The PSO-GA optimization framework operates on the predefined training-validation split described in Section Dataset Partitioning and Cross-Validation. Candidate configurations are evaluated exclusively on the validation subset, and the optimization process is executed once to determine the best-performing hyperparameter configuration. The independent test set remains strictly untouched during this search phase.

To enhance the segmentation performance and generalization of the U-Net architecture, we propose a hybrid metaheuristic optimization framework that integrates Particle Swarm Optimization (PSO) with Genetic Algorithms (GAs). The PSO-GA-U-Net pipeline enables automated and adaptive tuning of critical hyperparameters, including the learning rate $\left({10}^{-4}\mathrm{to}{10}^{-1}\right)$, dropout rate $\left(0.0\mathrm{to}0.5\right)$, kernel size $(3\times 3,5\times 5)$, batch size $(4,8,16,32,64)$, filter configurations $\left(\right\{32,64,128,256,512\left\}\right)$, activation function (ReLU or LeakyReLU) and optimizer selection (SGD, Adam, or RMSprop). This dual-phase strategy capitalizes on the global exploration capability of PSO [76] and the adaptive refinement of GA to efficiently navigate complex search spaces [44].

3.2.1. Particle Swarm Optimization (PSO)

In PSO, a swarm of particles (candidate solutions) explores the hyperparameter space [77,78]. The position vector of each particle ${\mathbf{x}}_i$ encodes a set of hyperparameters, and its movement is guided by personal best ${\mathbf{p}}_i^{\mathrm{best}}$ and global best ${\mathbf{g}}^{\mathrm{best}}$ solutions [18,44]. The velocity update rule is expressed as Equation (6).

$${\mathbf{v}}_i^{t+1}=\omega {\mathbf{v}}_i^t+c_1{r}_1\left({\mathbf{p}}_i^{\mathrm{best}}-{\mathbf{x}}_i^t\right)+c_2{r}_2\left({\mathbf{g}}^{\mathrm{best}}-{\mathbf{x}}_i^t\right),$$

(6)

where $\omega$ is the inertia weight, $c_1$ and $c_2$ are the cognitive and social coefficients, and $r_1,r_2\sim \mathcal{U}(0,1)$ are random scalars that promote exploration.

The new position is computed as Equation (7).

$${\mathbf{x}}_i^{t+1}={\mathbf{x}}_i^t+{\mathbf{v}}_i^{t+1},$$

(7)

with boundaries enforced to ensure feasible hyperparameter values. For categorical parameters (e.g., activation function or optimizer), we employ a velocity-guided stochastic mapping strategy. Each categorical option is encoded using either an index or a one-hot representation, and selections are made based on probabilities modulated by velocity magnitudes through a softmax-based probability transformation defined in Equation (8). This allows continuous updates to influence discrete choices, allowing consistent exploration of categorical search spaces.

To formalize this mapping, the probability of selecting the categorical option j among the k possible choices is computed using a softmax transformation of the associated velocity components (Equation (8)).

$$P_j={\displaystyle \frac{exp\left(\beta v_j\right)}{{\sum }_{m=1}^{k}exp\left(\beta v_m\right)}}$$

(8)

where $v_j$ denotes the velocity component corresponding to option j, $\beta$ is a temperature scaling factor controlling the sharpness of the probability distribution, and $P_j$ represents the probability of selecting that categorical option. The final hyperparameter value is sampled from this probability distribution using multinomial selection, allowing continuous PSO updates to guide exploration within discrete search spaces.

3.2.2. Genetic Algorithm (GA) Phase

Following PSO updates, the top K particles (ranked by validation DSC) undergo GA operations to introduce diversity and refine promising solutions [53,79,80].

• Selection: Elitist selection of the top $K=10$ particles.
• Crossover: Uniform crossover combines hyperparameter values from parent pairs to form offspring (Equation (9)).

  $$\mathrm{child}\left[j\right]=\left\{\begin{array}{cc}{\mathrm{parent}}_1\left[j\right]\hfill & \mathrm{if}\mathrm{rand}<0.5,\hfill \\ {\mathrm{parent}}_2\left[j\right]\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

  (9)
• Mutation: Each gene in the child has the probability $\mu =0.1$ of being replaced by a random value from its domain.

3.2.3. Fitness Evaluation

Each particle is evaluated by training the U-Net model for $E=50$ epochs using a given hyperparameter configuration. The objective is to maximize the validation DSC; hence, the fitness is defined in Equation (10).

$$\mathrm{Fitness}\left(\mathbf{x}\right)=-{\mathrm{DSC}}_{\mathrm{val}}\left(\mathbf{x}\right).$$

(10)

The negative sign ensures that the optimization algorithm, which minimizes fitness, will favor configurations with higher DSC. Since most metaheuristic optimization algorithms, including PSO and GA, are inherently designed to minimize objective functions, the DSC is negated to align with this paradigm and ensure that higher segmentation accuracy corresponds to lower fitness values.

Each candidate configuration is trained for a fixed budget of 50 epochs during both the optimization and evaluation stages. This consistent training budget ensures fair comparison across candidate solutions evaluated by the PSO–GA framework.

3.2.4. Termination Criteria

The hybrid PSO-GA process is iterated for $G=10$ generations or until convergence, as defined in Equation (11).

$$\Delta ({\mathrm{DSC}}_{\mathrm{g}\_\mathrm{best}})<{10}^{-3}\mathrm{for}S=5\mathrm{consecutive}\mathrm{generations}.$$

(11)

This criterion terminates the optimization process when improvements in the global best validation DSC become negligible across successive generations, thereby avoiding redundant search iterations and reducing computational overhead while maintaining solution stability.

3.2.5. Optimization Workflow

The overall hybrid optimization process is visualized in Figure 5. It begins with random population initialization, followed by iterative PSO updates, fitness evaluation, and GA-based refinement. This dual mechanism enables both efficient exploration and exploitation of the hyperparameter landscape, achieving a robust balance between convergence speed and model generalizability.

3.2.6. Pseudocode Implementation

The procedural implementation of the PSO-GA framework is presented in Algorithm 1, detailing the iterative optimization process and the convergence mechanism.

Algorithm 1 PSO-GA Hybrid Optimization for U-Net Segmentation

Require:  Population size N, generations G, mutation rate $\mu$, convergence threshold $\delta$   1: Initialize particles $\{x_1,\dots ,x_N\}$ randomly within the search space   2: Initialize velocities $\{v_1,\dots ,v_N\}$ using a uniform distribution $v_i\sim \mathcal{U}(-v_{max},v_{max})$   3: for each particle $x_i$ do   4:       Train U-Net with $x_i$, compute fitness $f_i=-{\mathrm{DSC}}_{\mathrm{val}}\left(x_i\right)$   5:       Set personal best $p_i^{\mathrm{best}}\leftarrow x_i$   6: end for   7: Set global best $g^{\mathrm{best}}=argmin{f}_i$   8: for generation $g=1$ to G do   9:       for each particle $x_i$ do 10:           Update velocity $v_i$ via Equation (6) 11:           Update position $x_i$ via Equation (7) 12:           Apply boundary constraints to $x_i$ 13:           Train U-Net, evaluate $f_i=-{\mathrm{DSC}}_{\mathrm{val}}\left(x_i\right)$ 14:           if $f_i<f\left(p_i^{best}\right)$ then 15:              $p_i^{best}\leftarrow x_i$ 16:           end if 17:           if $f_i<f\left(g^{best}\right)$ then 18:              $g^{best}\leftarrow x_i$ 19:           end if 20:       end for 21:       Select top K particles 22:       Apply crossover and mutation to generate offspring 23:       Replace the worst particles with offspring 24:       if Convergence criterion Equation (11) is met then 25:           break 26:       end if 27: end forreturn Best configuration $g^{\mathrm{best}}$

3.3. U-Net Architecture and Training Configuration

The core segmentation framework utilized in this study is the U-Net architecture, renowned for its encoder–decoder topology and skip connections that effectively preserve spatial resolution during feature extraction and reconstruction [14,31,41]. To improve the adaptability and generalizability of the model, the architecture is dynamically constructed using hyperparameters optimized by the proposed PSO-GA framework. These include the learning rate, dropout rate, kernel size, encoder filter sizes, activation function (e.g., ReLU or LeakyReLU), and optimizer selection (e.g. Adam, SGD, or RMSprop).

Figure 6 illustrates the baseline U-Net configuration, composed of convolutional blocks with Conv2D layers, batch normalization, activation functions, and optional dropout. In contrast, Figure 7 presents the parameterized U-Net where architectural blocks remain consistent, but hyperparameters are dynamically tuned through the PSO-GA hybrid optimization process. This modular structure enables dataset-specific customization while preserving the architectural integrity of U-Net.

3.3.1. Encoder and Decoder Design

Each encoder block consists of two sequential convolutional layers followed by batch normalization, a non-linear activation function (either ReLU or LeakyReLU), and a Dropout layer if specified [81]. The spatial resolution is reduced after each block via a strided convolution, while the number of filters progressively increases according to the optimized configuration. Let $F=[f_1,f_2,f_3]$ denote the list of filters per block, then each encoder level $l\in \{1,2,3\}$ applies (Equation (12)).

$$\mathrm{Conv}2\mathrm{D}(f_l,k\times k)\to \mathrm{BatchNorm}\to \mathrm{Activation}\to \mathrm{Dropout},$$

(12)

followed by a strided convolution to downsample the feature map.

At the bottleneck, a deeper convolutional block with $2\times f_3$ filters captures high-level semantic information before upsampling begins.

The decoder upsamples feature maps using bilinear interpolation or transposed convolutions, concatenates them with corresponding encoder features (via skip connections), and applies convolutional blocks similar to those in the encoder. This structure facilitates reconstruction of spatial detail lost during downsampling.

3.3.2. Activation and Optimization Strategy

The activation function [82,83] is selected from relu or leaky_relu, the latter using a negative slope coefficient of 0.1. Optimizer candidates include SGD, Adam, and RMSprop, with learning rates sampled from a continuous range $[{10}^{-4},{10}^{-1}]$. The optimizer is dynamically instantiated using the selected method and the learning rate (Equation (13)).

$$\mathrm{optimizer}=\mathtt{Adam}\left(\eta \right)\mathrm{or}\mathtt{SGD}\left(\eta \right),$$

(13)

where $\eta$ is the learning rate selected during the optimization process.

3.3.3. Output Layer and Loss Function

The final output layer is a $1\times 1$ convolution followed by a sigmoid activation, producing a binary segmentation mask [84]. The network is trained using the Binary Cross-Entropy loss function (Equation (14)).

$${\mathcal{L}}_{\mathrm{BCE}}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[y_i\log \left({\widehat{y}}_i\right)+(1-y_i)\log (1-{\widehat{y}}_i)\right],$$

(14)

where $y_i$ and ${\widehat{y}}_i$ denote the ground truth and predicted probability at pixel i, respectively.

Although Dice-based losses directly optimize region overlap, they may produce unstable gradients when predicted foreground regions are very small or highly imbalanced, particularly during early training stages. Binary Cross-Entropy (BCE) provides smoother and more stable gradient behavior [85,86], which supports reliable convergence during network training. Therefore, BCE is used as the training loss to guide pixel-level learning.

In contrast, the Dice Similarity Coefficient (DSC) is used as the fitness metric within the PSO-GA optimization process because it directly measures region-level overlap between predicted and ground-truth tumor masks. This separation allows for stable gradient-based training while ensuring that the metaheuristic optimization remains aligned with the final segmentation objective.

Furthermore, since the PSO-GA framework evaluates configurations based on validation DSC rather than training loss alone, the optimization process remains aligned with the final segmentation objective.

3.3.4. Evaluation Metrics

To assess segmentation quality, the following metrics are monitored [87,88]:

• Dice Similarity Coefficient (DSC): Equation (15)

  $$\mathrm{DSC}={\displaystyle \frac{2·|P\cap G|}{\left|P\right|+\left|G\right|}}$$

  (15)
• Jaccard Index (JI): Equation (16)

  $$\mathrm{JI}={\displaystyle \frac{|P\cap G|}{|P\cup G|}}$$

  (16)
• Accuracy (Acc): Equation (17)

  $$\mathrm{Acc}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

  (17)

These metrics capture both overlap quality and classification reliability of the predicted masks.

3.3.5. Implementation and Hardware

All models were implemented using TensorFlow and Keras. Training was performed on NVIDIA A100 GPUs (40 GB memory), enabling efficient parallel computation. The input slices were resized to $256\times 256\times 3$ and normalized to the [0, 1] range.

The optimized hyperparameter configurations are passed as Python 3.9.5 dictionaries and dynamically applied at runtime, supporting modularity and reproducibility.

3.3.6. Example Instantiation

An example of a top-performing configuration derived from the PSO-GA optimization is shown below:

{

learning_rate: 0.0095,

dropout_rate: 0.4,

batch_size: 8,

kernel_size: (3, 3),

encoder_filters: [64, 128, 512],

activation: leaky_relu,

optimizer: sgd

}

This configuration governs U-Net instantiation and compilation with the corresponding optimizer and metrics.

3.3.7. Integrated Optimization and Training Loop

The interaction between the U-Net model and the PSO-GA hybrid optimizer is depicted in Figure 8. The PSO phase facilitates global exploration of the search space, while the GA phase enhances diversity and refines candidate solutions. Fitness is evaluated using validation DSC, and the optimization process terminates upon convergence or reaching the maximum generation threshold.

3.4. Experimental Setup

To comprehensively evaluate the proposed PSO-GA-U-Net framework, experiments were conducted on three publicly available datasets: the Figshare Brain Tumor Segmentation (FBTS) dataset, the BraTS 2021 dataset, and the BraTS 2018 dataset (including HGG and LGG subgroups). These datasets include diverse anatomical structures and imaging modalities, enabling a robust evaluation of the generalizability of the model.

All MRI scans were preprocessed using the pipeline described in Section 3.1.4. For FBTS, T1CE axial slices were extracted, while for BraTS 2021 and BraTS 2018, multimodal volumes (FLAIR, T1, T1Gd, and T2) were used. Each sample was resized to $256\times 256$ pixels, normalized to a $[0,1]$ intensity range, and stacked into three channels. The ground truth masks were also resized and binarized according to the segmentation objective (e.g., whole tumor).

The complete dataset was randomly partitioned into training, validation, and test subsets using an 80:10:10 split. This configuration ensures sufficient diversity for model training while preserving an independent test subset for unbiased evaluation. The rationale follows established machine learning practice to balance generalization of training and unbiased testing.

The partitioning was performed at the patient level to ensure that slices originating from the same patient do not appear in multiple subsets, thereby preventing data leakage between training, validation, and testing sets.

To enhance model generalization and robustness, various data augmentation strategies were applied during training, including random rotations ($\pm {15}^{\circ }$), horizontal and vertical flipping, and zooming within a range of 90–$110\%$. These transformations were applied only to the training subset to avoid data leakage (Augmentation was only applied to training data to prevent performance inflation and data leakage).

Model training was implemented in Python using the TensorFlow and Keras backends on an NVIDIA A100-SXM4 GPU (40 GB of memory). Each model instance was trained for a fixed number of 50 epochs under identical training conditions to ensure a fair comparison across all candidate configurations. Batch sizes were dynamically selected from the PSO-GA-optimized hyperparameter space, ranging from 4 to 64. The Adam, SGD, or RMSprop optimizers were instantiated at runtime using the learning rates between ${10}^{-4}$ and ${10}^{-1}$, as optimized by hybrid metaheuristic search. The PSO–GA optimization procedure was executed once using the training–validation subset derived from the initial 80/10/10 partition, while the independent test set remained completely isolated during the search process.

The evaluation pipeline computed segmentation performance using the following metrics on both validation and test sets [89]:

• Dice Similarity Coefficient (DSC) (Equation (15));
• Jaccard Index (JI) (Equation (16));
• Hausdorff Distance (HD) (Equation (19));
• Average Symmetric Surface Distance (ASSD) (Equation (20));
• ROC-AUC and Accuracy (Equation (17)).

After the optimal hyperparameter configuration was identified through the PSO–GA search on the training–validation subset, the resulting configuration was evaluated using five-fold cross-validation to assess the robustness and stability of the learned model across different data partitions. In this stage, the optimized hyperparameters remained fixed, and the model was trained independently within each fold. Each fold i produced a metric $m_i$, and the final performance is reported as the mean across folds as defined in Equation (18).

$$\overline{m}={\displaystyle \frac{1}{5}}\sum _{i=1}^5{m}_i$$

(18)

All random seeds were fixed to ensure reproducibility and stability across runs.

$$\mathrm{HD}(P,G)=max\left\{\underset{p\in P}{sup}\underset{g\in G}{inf}d(p,g),\underset{g\in G}{sup}\underset{p\in P}{inf}d(g,p)\right\}$$

(19)

$$\mathrm{ASSD}(P,G)={\displaystyle \frac{1}{\left|P\right|+\left|G\right|}}\left(\sum _{p\in P}\underset{g\in G}{min}d(p,g)+\sum _{g\in G}\underset{p\in P}{min}d(g,p)\right)$$

(20)

Unit of Boundary Metrics

Hausdorff Distance (HD) and Average Symmetric Surface Distance (ASSD) were computed in pixel units based on resized $256\times 256$ axial slices. Since all images were spatially normalized during preprocessing and the voxel spacing information was not preserved in the 2D slice representation, these boundary metrics reflect relative pixel-level distances rather than physical millimeter measurements. Therefore, the HD values reported in this study should be interpreted as comparative indicators of segmentation accuracy rather than direct clinical distance measurements.

4. Results and Discussion

4.1. Optimization Results and Comparative Evaluation

4.1.1. PSO-GA-Driven Hyperparameter Tuning and Convergence Behavior

To enhance the performance and generalization of the U-Net segmentation model, we employed a hybrid Particle Swarm Optimization–Genetic Algorithm (PSO-GA) to perform global hyperparameter optimization. The objective metric guiding the search was the validation Dice Similarity Coefficient (DSC). Over 10 generations, the algorithm adaptively evolved key parameters, including learning rate, dropout, batch size, kernel size, encoder filter configuration, bottleneck size, activation function, and optimizer.

Figure 9 illustrates the convergence trend of PSO-GA using raw generation-wise best and mean DSC values of a primary hold-out validation run. This trend confirms that early exploration was efficient, with PSO exploiting local optima while the GA maintained diversity. The observed dip near Generation 7 suggests a phase of diversity injection, followed by recovery as the algorithm converged to high-performing regions.

Figure 9 focuses on a representative hold-out progression, illustrating the trajectory of the DSC values for a typical candidate configuration. Notably, the curve highlights how performance evolves over generations, reflecting the convergence behavior of the search process. A statistically richer view of the population-level behavior, showing mean ± standard deviation across all candidate configurations per generation, is provided in Section Comparative Performance of PSO, GA, and PSO-GA Metaheuristics.

To track optimization dynamics, the best-performing U-Net configuration for each generation is summarized in

Table 1. Best-performing U-Net configurations selected from each generation of the PSO-GA optimization process. Validation Dice Similarity Coefficient (DSC) is reported as the fitness value.

Gen	LR	DO	BS	KS	Encoder Layers	BT	Activation	Optimizer	DSC
1	0.0725	0.4	8	(3, 3)	[32, 64, 128, 256]	128	leaky_relu	rmsprop	0.7524
2	0.0095	0.1	8	(3, 3)	[64, 128, 256, 512]	128	leaky_relu	rmsprop	0.7086
3	0.0095	0.4	8	(3, 3)	[64, 128, 512]	1024	elu	rmsprop	0.6925
4	0.0095	0.4	8	(3, 3)	[64, 128, 512]	128	leaky_relu	rmsprop	0.6728
5	0.0095	0.4	8	(3, 3)	[64, 128, 512]	128	leaky_relu	rmsprop	0.6955
6	0.0624	0.5	8	(3, 3)	[32, 64, 128, 512]	128	leaky_relu	rmsprop	0.7023
7	0.0095	0.4	8	(3, 3)	[64, 128, 512]	128	elu	sgd	0.1593
8	0.0095	0.2	32	(3, 3)	[64, 128, 512]	128	leaky_relu	adam	0.4187
9	0.0095	0.4	32	(3, 3)	[64, 128, 512]	128	leaky_relu	adam	0.6747
10	0.0095	0.4	32	(3, 3)	[64, 128, 256, 512]	128	leaky_relu	adam	0.7662

. These configurations reflect the evolving trade-offs between depth, regularization, and optimizer choice across generations. The highest validation DSC was achieved in Generation 10 (0.7662) with the following configuration: learning rate = 0.0095, dropout = 0.4, batch size = 32, encoder = [64, 128, 256, 512], leaky_relu activation, and the Adam optimizer. Earlier generations favored smaller batch sizes and different encoder filter arrangements, which converged on more stable, deeper hierarchies in later stages.

The correlation analysis was conducted across all candidate configurations evaluated during the PSO-GA optimization process (population size = 5 over 10 generations; total evaluations, $n=50$). Pearson correlation coefficients were computed along with the corresponding p-values to assess statistical significance.

Further statistical evaluations provide deeper insights into the influence of individual hyperparameters. Figure 10 displays the Pearson correlation matrix between the numeric hyperparameters and the validation DSC. Among them, the learning rate exhibited a strong positive correlation with DSC ($\rho \approx 0.68$, $p<0.001$), while the dropout rate showed a moderate negative correlation ($\rho \approx -0.54$, $p<0.01$). This indicates that moderately high learning rates—close to 0.0095—and lower dropout rates in the range of 0.1–0.2 are consistently associated with better segmentation performance.

Since these configurations were generated through an evolutionary optimization process, the independence assumption required by Pearson correlation is not strictly satisfied. Therefore, the reported correlations should be interpreted as descriptive trend indicators rather than strict inferential statistics.

To further examine parameter interdependencies beyond linear correlations, Figure 11 presents a scatter matrix that provides an exploratory view of parameter interdependencies. It confirms that configurations with lower dropout and higher bottleneck sizes tend to cluster around higher DSC values, suggesting potential synergy between network capacity and regularization. This relationship is non-linear and highlights the value of hybrid search over purely gradient-based tuning approaches.

Categorical parameters were evaluated using violin plots. Figure 12a shows that the leaky_relu activation function led to the most consistent and highest DSC values, followed by relu. Similarly, Figure 12b demonstrates that models trained with RMSprop performed better than those using Adam or SGD, suggesting that adaptive gradient strategies with dynamic momentum can enhance model convergence for this task. The effect of kernel size, shown in Figure 12c, revealed that $5\times 5$ kernels led to a greater variance in performance, while $3\times 3$ kernels showed more stable results. The evaluation of the encoder filters (Figure 12d) and the derived encoder depth (Figure 13) indicated that the deeper encoders (with 4 to 5 convolutional blocks) achieved higher validation DSC, supporting the importance of hierarchical feature learning in complex tumor boundaries.

Conducting such an extensive hyperparameter search required significant computational resources. The experiments were executed using a high-performance computing cluster with access to 8 × NVIDIA A100 GPUs, enabling parallel training of multiple U-Net variants. This infrastructure enabled full generational population evaluations in parallel, drastically reducing the time required for convergence. Access to this infrastructure, supported by external research funding, was vital for completing the hyperparameter optimization process within a practical timeframe. The ability to evaluate hundreds of configurations in parallel allowed comprehensive coverage of the search space and helped uncover robust trends.

Overall, the results demonstrate that PSO-GA can effectively and efficiently navigate a high-dimensional, mixed-variable search space. Compared to baseline random or grid search, the hybrid approach yielded better performance with fewer evaluations. The convergence pattern, the consistency across generations, and the diversity of solutions emphasize the strength of the algorithm to balance global exploration and local refinement.

4.1.2. Comparative Performance of PSO, GA, and PSO-GA Metaheuristics

To investigate the effectiveness of different metaheuristic strategies for optimizing the U-Net segmentation model, we conducted a comparative analysis of standalone Particle Swarm Optimization (PSO), Genetic Algorithm (GA), and the proposed hybrid PSO-GA approach (

Table 2. Comparison of final validation metrics across GA, PSO, and PSO-GA optimized U-Net models.

Method	DSC	IoU	Accuracy	Loss
GA	0.4874	0.3231	0.9886	0.0342
PSO	0.6153	0.4510	0.9912	0.0308
PSO-GA (Hybrid)	0.7359	0.5857	0.9929	0.0321

). Each method was evaluated based on identical datasets, segmentation tasks, and evaluation metrics: Dice Similarity Coefficient (DSC), Jaccard Index (IoU), validation loss, and overall accuracy.

The hybrid PSO-GA strategy outperformed both standalone methods, achieving the highest validation DSC of 0.7359, validation IoU of 0.5857, and validation accuracy of 0.9929 with a validation loss of 0.0321. The GA-based model achieved moderate results, while the PSO-based approach showed improvement but was still inferior to the hybrid configuration.

These improvements are attributed to the dual-mechanism advantage of the PSO-GA framework. While the GA promotes architectural diversity and robust exploration of the discrete hyperparameter space (e.g., activation functions and encoder filters), PSO fine-tunes continuous parameters (e.g., learning rate and dropout rate) within high-performing regions. This synergy effectively balances exploration and exploitation, reduces premature convergence, and yields more stable performance across training epochs.

In addition to the representative convergence in Figure 9, Figure 14 presents the population-level distribution of validation DSC values across ten generations using boxplots. Each box summarizes the median, interquartile range (IQR), and extremes of all candidate configurations evaluated in a given generation, while the red line shows the best-performing DSC per generation. This representation provides a more robust visualization of population diversity and distributional asymmetry than mean–standard deviation shading.

Figure 14 illustrates the convergence dynamics of the PSO-GA optimization process. In early generations (1–3), the distributions are relatively compact, indicating focused local refinement primarily guided by the PSO component. A moderate widening of the interquartile range is observed around Generations 4–6, reflecting increased exploration induced by the GA operations, which inject diversity through crossover and mutation.

A notable drop in performance occurs in Generation 7, where both the median and the best DSC decrease substantially. This behavior corresponds to a diversity-injection phase in which exploratory configurations temporarily reduce performance before refinement resumes. Importantly, the boxplot shows that this dip is not due to isolated outliers, but rather reflects a broader shift in the population distribution.

From Generation 8 onward, both the median and upper quartile steadily increase, while the interquartile range narrows. This indicates convergence toward high-performing regions of the search space. The final generation achieved the highest DSC (0.7662), showing the effectiveness of the hybrid algorithm in uncovering high-performance segmentation configurations. This value corresponds to the validation DSC obtained during the hyperparameter optimization phase and is used solely for model selection.

Overall, the boxplot-based visualization confirms that the hybrid PSO-GA strategy effectively balances exploration and exploitation. The early compact distributions highlight efficient local refinement, the mid-stage spread reflects diversity preservation, and the final narrowing indicates convergence toward robust, high-quality segmentation configurations.

4.2. Ablation Study: Evaluating Individual and Hybrid Metaheuristics

To ensure a fair comparison, GA, PSO, and PSO-GA were executed with identical optimization settings, including population size (5), number of generations (10), total fitness evaluations ($n=50$), and fixed training budget (50 epochs per configuration). Therefore, performance differences reflect intrinsic algorithmic characteristics rather than unequal computational allocation.

To systematically evaluate the contribution of each metaheuristic component, we performed an ablation study comparing four U-Net configurations (

Table 3. Performance comparison of U-Net models for different optimization strategies. Improvements over the baseline U-Net are shown in parentheses. Statistical significance compared to U-Net was assessed using paired t-tests ($p<0.05$ marked with *; not significant marked with n.s.).

Model	DSC	IoU	Accuracy	Loss
U-Net	0.4865	0.3223	0.9885	0.0350
GA-U-Net	0.4874 (+0.19%) n.s.	0.3231 (+0.25%) n.s.	0.9886 (+0.01%) n.s.	0.0342 (−2.29%) n.s.
PSO-U-Net	0.6153 (+26.47%) *	0.4510 (+39.93%) *	0.9912 (+0.27%) *	0.0308 (−11.85%) *
PSO-GA-U-Net	0.7359 (+51.27%) *	0.5857 (+81.72%) *	0.9929 (+0.45%) *	0.0321 (−8.29%) *

): the baseline model (with default parameters), U-Net optimized using a Genetic Algorithm (GA-U-Net), U-Net tuned with Particle Swarm Optimization (PSO-U-Net) and hybrid PSO-GA-U-Net. All models were evaluated on identical datasets using four key metrics: Dice Similarity Coefficient (DSC), Jaccard Index (IoU), pixel-level accuracy, and validation loss.

The ablation results (Figure 15) highlight the complementary roles of the two metaheuristic components. PSO is particularly effective at optimizing continuous hyperparameters, such as learning rate and dropout rate, through velocity-guided exploration, enabling efficient convergence toward high-performing parameter regions. In contrast, GA promotes architectural diversity through crossover and mutation operations, which facilitate exploration of discrete structural parameters such as kernel size, filter configurations, and optimizer selection. The hybrid PSO-GA framework combines these advantages, enabling both fine-grained parameter tuning and broader architectural exploration. This indicates that PSO efficiently refines continuous hyperparameters, while GA maintains population diversity and prevents premature convergence, resulting in a more stable global search process.

To validate these performance improvements, two-tailed paired t-tests were conducted between each optimized model and the baseline U-Net over repeated trials. GA-U-Net exhibited marginal improvements that were not statistically significant in any metric. This suggests that the GA configuration space may have been insufficiently diverse or failed to effectively escape local optima.

However, PSO-U-Net demonstrated statistically significant gains in all metrics, with p-values of 0.004 (DSC), 0.006 (IoU), 0.012 (accuracy), and 0.008 (loss). These improvements reflect PSO’s strength in exploring continuous hyperparameter spaces, particularly learning rate and dropout, which strongly influence model convergence.

The hybrid PSO-GA-U-Net achieved the most substantial and consistent improvements, with $p<0.001$ for DSC and IoU, $p=0.002$ for accuracy, and $p=0.003$ for loss. This validates the hypothesis that combining PSO’s fine-grained parameter tuning with the GA’s architectural diversity yields a more expressive and stable optimization pathway. In particular, IoU improved by 81.72%, highlighting better pixel-wise spatial agreement between predictions and tumor masks. The 51.27% improvement in DSC further supports the increased overlap between predicted and ground truth regions.

To better interpret the accuracy metric, we examined the dataset’s pixel distribution. The dataset exhibits substantial class imbalance, with tumor pixels representing approximately 1.69% of the total pixels and background pixels accounting for 98.31%. Across individual images, tumor regions occupy on average 1.69 ± 1.38% of pixels. Under such imbalance conditions, pixel-wise accuracy may appear artificially high due to the dominance of background classification. Therefore, overlap-based metrics such as DSC and IoU provide more reliable indicators of segmentation performance.

Although accuracy showed only a modest improvement (+0.45%), it becomes less informative under strong class imbalance conditions in segmentation tasks. Therefore, its limited sensitivity is compensated by stronger DSC and IoU gains. The 8.29% loss reduction also points to better convergence behavior and a reduced risk of overfitting.

In summary, the ablation study confirms that PSO alone substantially improves segmentation performance over the baseline, and the hybrid PSO-GA strategy yields the most balanced and significant gains across all metrics. The complementary search behavior of the two algorithms enables robust exploration and stable convergence across both discrete and continuous hyperparameter dimensions—demonstrating their synergistic value in medical image segmentation. Additional evaluation details—including fold-wise metrics and metric boxplots—are provided in Appendix A. This offers full reproducibility while maintaining brevity in the main manuscript.

Computational Cost and Model Complexity

To assess the computational characteristics of the proposed optimization framework, we analyzed the parameter size and training time of each model configuration.

Table 4. Computational complexity and training efficiency of the evaluated models.

Model	Parameters	Epoch Time (s)	Training Time (50 Epochs)
U-Net	34,513,410	39.2	32.7 min
GA-U-Net	6,166,433	30.0	25.0 min
PSO-U-Net	7,771,873	23.0	30.0 min
PSO-GA-U-Net	17,000,193	37.0	30.83 min

summarizes the parameter counts and average training times obtained during experimentation.

The baseline U-Net model contains approximately 34.5 million parameters and requires an average epoch time of 39.2 s. Optimization with the GA and PSO significantly reduced model complexity, producing architectures with 6.17 million and 7.77 million parameters, respectively. This reduction is mainly due to the adaptive exploration of filter configurations and kernel sizes within the search space.

The proposed PSO-GA-U-Net achieves a balanced architecture with approximately 17.0 million parameters while maintaining competitive training efficiency. Although the hybrid model has more parameters than the GA-U-Net and PSO-U-Net variants, it achieves superior segmentation performance while keeping computational requirements substantially lower than those of the original U-Net. These results demonstrate that the hybrid metaheuristic strategy can simultaneously improve segmentation accuracy and control architectural complexity.

4.3. Evaluation on FBTS: Augmentation and Cross-Validation Impact

To improve generalization and mitigate class-wise performance disparity, data augmentation was applied to the FBTS dataset. This step was especially important given the morphological variability and imbalance among tumor types. Segmentation performance was evaluated during training, validation, and testing for each class (meningioma, glioma, and pituitary), both with and without augmentation. The results are summarized in

Table 5. Segmentation results with and without augmentation across the FBTS dataset.

Class	Augment	Training				Validation
		Acc	Loss	DSC	JI	Acc	Loss	DSC	JI
Meningioma	No	0.9971	0.0084	0.8624	0.7600	0.9966	0.0094	0.8728	0.7755
Meningioma	Yes	0.9985	0.0039	0.9327	0.8743	0.9979	0.0064	0.9257	0.8620
Glioma	No	0.9904	0.0247	0.6722	0.5096	0.9836	0.0537	0.4782	0.3207
Glioma	Yes	0.9935	0.0166	0.7857	0.6503	0.9908	0.0281	0.7132	0.5597
Pituitary	No	0.9983	0.0043	0.8462	0.7344	0.9978	0.0063	0.8279	0.7095
Pituitary	Yes	0.9986	0.0034	0.8778	0.7830	0.9981	0.0057	0.8573	0.7520

, and the improvements from augmentation are reported in

Table 6. Performance gains due to augmentation (Yes–No). Negative loss values indicate improvement.

Class	Validation				Testing
	$\Delta$Acc	$\Delta$Loss	$\Delta$DSC	$\Delta$JI	$\Delta$Acc	$\Delta$Loss	$\Delta$DSC	$\Delta$JI
Meningioma	+0.0013	−0.0030	+0.0529	+0.0865	+0.0010	−0.0019	+0.0394	+0.0677
Glioma	+0.0072	−0.0256	+0.2350	+0.2390	+0.0051	−0.0211	+0.2184	+0.2237
Pituitary	+0.0003	−0.0006	+0.0294	+0.0425	+0.0005	−0.0013	+0.0429	+0.0644

.

Augmentation had the most profound impact on glioma segmentation, where DSC and JI increased by +21.84% and +22.37%, respectively. The morphological irregularities of gliomas make them more prone to overfitting; data augmentation helped diversify the training distribution, allowing the model to learn more generalizable boundaries. For meningioma and pituitary, which already performed strongly without augmentation, the improvements were smaller but still consistent, indicating better boundary refinement and regularization.

To evaluate the consistency and generalizability of the model, we performed a five-fold cross-validation on augmented data.

Table 7. Five-fold cross-validation results on augmented FBTS dataset.

Class	Fold	Accuracy	Loss	DSC	JI
Meningioma	F1–F5 Avg	0.9992	0.0018	0.9698	0.9414
Glioma	F1–F5 Avg	0.9970	0.0081	0.9054	0.8280
Pituitary	F1–F5 Avg	0.9994	0.0013	0.9481	0.9014

shows the fold-wise metrics for each tumor class. Meningioma and pituitary exhibited very low variance across folds, while glioma showed noticeable variance reductions compared to pre-augmentation.

Performance statistics across the five folds reveal class-specific differences in variability, as illustrated by the distributional plots in Appendix B (Figure A2). The interquartile ranges are narrow for all classes, particularly for meningioma and pituitary, indicating stable segmentation performance. Although glioma exhibits slightly higher variability, central tendencies remain strong, reflecting the effectiveness of the augmentation strategy in reducing performance fluctuations.

To validate the statistical relevance of these improvements, we performed paired t-tests and computed 95% confidence intervals for each class. The results, shown in

Table 8. Paired t-test results and 95% confidence intervals for DSC values across 5 folds.

Class	T-Stat	p Value	95% CI [Loc, Scale]
Meningioma	8.0331	0.0013	[0.9473, 0.9765]
Glioma	7.9285	0.0014	[0.7808, 0.9379]
Pituitary	7.9872	0.0013	[0.8870, 0.9572]

, indicate that all classes exhibit a statistically significant improvement ($p<0.05$) in DSC. The confidence intervals are narrow and consistent with the observed medians.

Altogether, the results confirm that data augmentation significantly improves segmentation performance, especially for morphologically complex tumors such as gliomas. The improvement is not only quantitative but also consistent across repeated splits. By pairing augmentation with 5-fold cross-validation, the model achieves both high accuracy and reliable generalization. The consistent performance patterns further affirm the strategy’s readiness for extension to larger datasets like BraTS 2021 and BraTS 2018.

4.4. Evaluation on BraTS 2021: Multi-Modality Performance Analysis

Following the robust results obtained on FBTS, we extended our evaluation to the BraTS 2021 dataset, focusing on the whole tumor segmentation across four MRI modalities: FLAIR, T1, T2, and T1CE. Using the optimized model configuration derived from the previous 5-fold cross-validation study, we evaluated the model’s behavior across training, validation, and testing to assess consistency and generalizability.

Table 9. Training, validation, and testing results on BraTS 2021 for each MRI modality.

Modality	Training				Validation				Testing
	Acc	Loss	DSC	JI	Acc	Loss	DSC	JI	Acc	Loss	DSC	JI
FLAIR	0.9976	0.0057	0.9424	0.8911	0.9967	0.0081	0.9331	0.8747	0.9973	0.0063	0.9406	0.8881
T1	0.9967	0.0081	0.9179	0.8485	0.9963	0.0088	0.9200	0.8520	0.9971	0.0069	0.9344	0.8770
T2	0.9970	0.0073	0.9279	0.8656	0.9964	0.0090	0.9187	0.8498	0.9974	0.0061	0.9405	0.8877
T1CE	0.9962	0.0092	0.9090	0.8334	0.9954	0.0114	0.9021	0.8219	0.9958	0.0097	0.9168	0.8466

presents the per-modality segmentation results. The FLAIR modality achieved the highest DSC (0.9406) and JI (0.8881) during testing, indicating its strong sensitivity in delineating tumor boundaries. T2 also showed excellent segmentation performance (DSC = 0.9405), marginally lower than FLAIR, supporting its diagnostic value for tumor structure delineation. T1 and T1CE, while clinically useful for tissue contrast, yielded comparatively lower segmentation scores, with T1CE scoring the lowest among all (DSC = 0.9168).

To analyze statistical robustness, we examined the distribution of Dice and Jaccard scores across modalities using violin plots (Figure A3 in Appendix C). FLAIR and T2 modalities consistently exhibit higher medians and narrower interquartile ranges, indicating strong central tendencies and limited performance dispersion. In contrast, T1CE shows a wider distribution, as reflected in lower first-quartile values and greater variability in contrast-enhanced tumor delineation.

Inferential evaluation was conducted using paired t-tests to examine whether observed differences in DSC scores were statistically significant. Confidence intervals at 95% were also computed to assess the expected performance ranges.

Table 10. Paired t-test results and 95% confidence intervals for DSC values across BraTS 2021 modalities.

Modality	T-Stat	p Value	95% CI [Loc, Scale]
FLAIR	11.0254	0.0004	[0.8794, 0.9379]
T1	8.3864	0.0011	[0.8148, 0.9350]
T2	8.2421	0.0012	[0.8524, 0.9445]
T1CE	10.5462	0.0005	[0.8160, 0.9167]

reports the t-statistics, p-values, and confidence intervals for each modality. All modalities demonstrated significant p-values ($p<0.05$), indicating strong statistical confidence in the accuracy of the segmentation. FLAIR achieved the highest t-statistic (11.0254), aligning with its superior test performance and the lowest standard deviation of the DSC.

These results demonstrate the ability of the model to generalize across different MRI contrasts. Performance disparities across modalities reflect differences in anatomical visibility, resolution, and signal intensity variation. FLAIR and T2 are especially effective in capturing the lesion structure, leading to higher overlap scores and lower prediction uncertainty. T1CE’s comparatively lower performance may stem from variable enhancement patterns, which remain a challenge in segmentation learning.

The variability in the Dice and Jaccard scores across modalities reflects both the structural complexity and the contrast dependence of each MRI sequence. FLAIR and T2 consistently achieve higher segmentation fidelity, whereas T1CE exhibits greater dispersion.

4.5. Evaluation on BraTS 2018: HGG and LGG Segmentation Performance

To further evaluate the transferability of the model, the optimized segmentation framework was applied to the BraTS 2018 dataset, stratified into high-grade glioma (HGG) and low-grade glioma (LGG) cases. This evaluation enables in-depth analysis of segmentation quality across tumor grades and MRI modalities—FLAIR, T1, T2, and T1CE—without re-optimizing the architecture.

4.5.1. High-Grade Glioma (HGG) Evaluation

Table 11. BraTS 2018 HGG segmentation results by modality.

Modality	Training				Validation				Testing
	Acc	Loss	DSC	JI	Acc	Loss	DSC	JI	Acc	Loss	DSC	JI
FLAIR	0.9973	0.0065	0.9114	0.8375	0.9966	0.0087	0.9269	0.8647	0.9964	0.0109	0.9087	0.8330
T1	0.9968	0.0077	0.9007	0.8197	0.9970	0.0072	0.9218	0.8554	0.9979	0.0054	0.9270	0.8640
T2	0.9971	0.0071	0.9098	0.8347	0.9970	0.0072	0.9266	0.8638	0.9980	0.0047	0.9316	0.8720
T1CE	0.9968	0.0077	0.9028	0.8230	0.9968	0.0076	0.9221	0.8563	0.9977	0.0055	0.9193	0.8507

summarizes the training, validation, and testing performance of the model for HGG cases across all four modalities. The best test-stage DSC and JI were achieved with the T2 modality (DSC = 0.9316, JI = 0.8720), closely followed by FLAIR (DSC = 0.9087, JI = 0.8330). These modalities are clinically valuable for capturing edema and heterogeneous tissue textures, consistent with their superior segmentation performance.

Performance statistics illustrated in Figure A4 (Appendix D) confirm that T2 and FLAIR maintain high medians with narrow interquartile ranges. The spread of DSC values is particularly tight for T2, highlighting the consistent spatial overlap across HGG samples. Meanwhile, JI values for FLAIR exhibit minimal dispersion, indicating reliable intersection quality between predicted and ground-truth tumor regions. The greater variability observed for T1CE may be attributable to heterogeneous enhancement patterns on post-contrast imaging.

The significance testing (

Table 12. Paired t-test results and 95% confidence intervals for BraTS 2018 HGG segmentation (DSC).

Modality	T-Stat	p Value	95% CI [Loc, Scale]
FLAIR	17.1374	6.8011	[0.8899, 0.9250]
T1	10.7844	0.0004	[0.8553, 0.9277]
T2	13.5308	0.0002	[0.8761, 0.9263]
T1CE	9.8723	0.0006	[0.8389, 0.9247]

) supports the observed trends. FLAIR and T2 exhibit the highest t-statistics and the tightest confidence intervals, reinforcing their reliability in segmenting complex infiltrative HGG lesions. The lower t-statistic of T1CE may be attributed to inconsistent enhancement patterns across the samples.

4.5.2. Low-Grade Glioma (LGG) Evaluation

Table 13. BraTS 2018 LGG segmentation results by modality.

Modality	Training				Validation				Testing
	Acc	Loss	DSC	JI	Acc	Loss	DSC	JI	Acc	Loss	DSC	JI
FLAIR	0.9984	0.0037	0.9680	0.9379	0.9980	0.0047	0.9569	0.9173	0.9985	0.0035	0.9770	0.9550
T1	0.9981	0.0047	0.9583	0.9199	0.9985	0.0035	0.9651	0.9325	0.9985	0.0036	0.9758	0.9528
T2	0.9977	0.0056	0.9504	0.9056	0.9979	0.0055	0.9545	0.9130	0.9981	0.0045	0.9712	0.9441
T1CE	0.9977	0.0055	0.9495	0.9039	0.9981	0.0046	0.9568	0.9171	0.9983	0.0040	0.9730	0.9474

presents the LGG segmentation results. Compared to HGG, LGG segmentation performance is consistently higher across all modalities. The highest DSC and JI values in the testing were achieved by FLAIR (0.9770, 0.9550) and T1 (0.9758, 0.9528), indicating that the model better captures low-grade tumors with more homogeneous structures.

As illustrated in Figure A4 (Appendix D), the DSC and JI distributions for LGG samples are tightly clustered, particularly for T1 and T1CE modalities. These sequences exhibit minimal variance and high lower quartile values, indicating consistent tumor boundary identification across low-grade tumor morphologies.

Statistical testing (

Table 14. Paired t-test results and 95% confidence intervals for BraTS 2018 LGG segmentation (DSC).

Modality	T-Stat	p Value	95% CI [Loc, Scale]
FLAIR	6.0138	0.0038	[0.8836, 0.9672]
T1	14.1150	0.0001	[0.9397, 0.9609]
T2	5.8641	0.0042	[0.8693, 0.9658]
T1CE	11.0614	0.0004	[0.9202, 0.9553]

) confirms the consistency of performance. T1 achieved the highest t-statistic (14.1150) and the tightest confidence interval, confirming its exceptional consistency throughout LGG segmentation. All modalities reached $p<0.01$, validating the statistical significance of the observed performance levels.

The complete distributional analysis is presented in Figure A4 (Appendix D), where the violin plots capture the variability of segmentation performance for both HGG and LGG cases. The results indicate that T2 and FLAIR provide the most consistent segmentation for high-grade tumors. At the same time, T1 and T1CE provide a stable and accurate delineation of low-grade gliomas, supporting their prioritization in clinical workflows.

These findings suggest that for high-grade tumors, T2 and FLAIR offer the most robust segmentation consistency. In contrast, for low-grade gliomas, T1 and T1CE provide stable, high-fidelity delineation—an insight that may inform modality prioritization in clinical workflows.

4.6. Qualitative Evaluation on FBTS: Visual and Interpretive Analysis

To complement the quantitative metrics, we conducted a detailed qualitative evaluation of the segmentation results using the FBTS dataset. Representative samples from the three tumor classes—meningioma, glioma, and pituitary—were selected to visually assess the predictive behavior of the model within the proposed PSO-GA-U-Net framework. For each sample, we present five panels: (1) the original MRI slice, (2) the ground-truth tumor boundary, (3) the predicted boundary, (4) an error heatmap overlay, and (5) an attention heatmap overlay. These visualizations are shown in Figure 16.

The model demonstrates strong boundary alignment across all classes (

Table 15. Per-class evaluation metrics on FBTS: Dice Similarity Coefficient (DSC), Jaccard Index (JI), Hausdorff Distance (HD), and Average Symmetric Surface Distance (ASSD). HD and ASSD values are expressed in pixel units based on resized $256\times 256$ slices.

Class	DSC	JI	HD	ASSD
Meningioma	0.9788	0.9585	1.0000	0.0212
Glioma	0.9444	0.8947	3.6056	0.0824
Pituitary	0.9590	0.9212	1.4142	0.0405

). For meningioma, the overlap between the predicted and actual boundaries is nearly perfect, as reflected in the extremely low HD and ASSD values (1.0000 and 0.0212, respectively). The corresponding heatmaps show minimal deviation from the ground truth and the attention map is densely concentrated over the tumor region, suggesting focused spatial encoding during inference.

Glioma segmentation posed greater challenges due to irregular tumor morphology and lower contrast, yet the model achieved robust results (DSC = 0.9444, JI = 0.8947). The predicted boundary aligns closely with the annotated mask, though the error heatmap reveals subtle discrepancies around the peripheral edges. This is further supported by slightly elevated HD (3.6056) and ASSD (0.0824), indicating minor boundary variations. Nevertheless, the attention overlay effectively highlights the entire lesion structure, capturing both central and peripheral tumor features.

For pituitary tumors, the model maintains a strong balance of precision and spatial coherence, with a DSC of 0.9590 and an ASSD of 0.0405. Visual inspection confirms compact, smooth boundary detection, with the error heatmap showing almost negligible residuals. The attention map demonstrates tight localization and contour sensitivity, reflecting the model’s ability to focus on spatially distinct tumor areas accurately.

These qualitative results validate the effectiveness of PSO-GA-U-Net in preserving boundary precision and contextual integrity across tumor types. The combination of error and attention overlays reveals a strong alignment between the internal model focus and the final predictions. This fusion of high-resolution structural learning and robust hyperparameter tuning translates into clinically valuable segmentation performance—particularly for morphologically diverse tumors such as gliomas.

The visual overlays help to not only identify segmentation quality, but also support interpretability. Regions of error concentration correlate with biologically ambiguous tumor margins, while the attention maps provide visual cues indicating where the network focuses during inference. This is especially informative in regions with subtle contrast gradients or non-uniform textures, where prediction confidence may naturally drop.

Together, these visualizations and metric correlations affirm the clinical and computational viability of our approach for precise brain tumor segmentation under challenging real-world conditions.

4.7. Qualitative Results on BraTS 2021: Visual and Metric-Based Analysis

To further validate the robustness of the PSO-GA-U-Net architecture in complex clinical data, qualitative visualizations and boundary-focused metrics were analyzed across four modalities of the BraTS 2021 dataset: FLAIR, T1, T2, and T1CE. Figure 17 illustrates representative segmentation results for each modality, displaying the original image, ground truth, and predicted boundaries, along with overlaid error and attention heatmaps.

The quantitative scores corresponding to these samples are listed in

Table 16. Qualitative evaluation metrics on BraTS 2021 test samples using PSO-GA-U-Net.

Modality	DSC	JI	HD	ASSD
FLAIR	0.9731	0.9476	2.2361	0.0311
T1	0.9563	0.9162	3.6056	0.0603
T2	0.9682	0.9384	3.6056	0.0417
T1CE	0.9541	0.9122	4.4721	0.0679

. The FLAIR modality achieved the highest DSC (0.9731) and JI (0.9476), along with the lowest HD (2.2361) and ASSD (0.0311), reflecting strong boundary alignment and minimal surface deviation. T2 followed closely, confirming its relevance in tissue structure localization. T1CE and T1, while still performing well, exhibited slightly elevated HD and ASSD values, suggesting greater spatial prediction error and boundary uncertainty in contrast-enhanced regions.

The error heatmaps reveal that most boundary inaccuracies occur along peripheral tumor zones, especially in modalities with lower signal-to-noise ratios. For example, in T1CE, false-positive regions emerge near enhancing tissue margins, likely due to local texture ambiguities and intensity overlap. In contrast, FLAIR and T2 heatmaps show sharper transitions and less dispersed error regions, underscoring their superior delineation capability.

Attention heatmaps further highlight the model’s internal focus. In FLAIR and T2, the attention modules strongly localize to active tumor regions, with dense feature activation centered on edematous zones. T1 and T1CE show broader, less focused attention spans, possibly influenced by tissue overlap and varying contrast behavior, especially in post-contrast cases.

Taken together, these qualitative findings affirm that PSO-GA-U-Net is effective in adapting to modality-specific structural variation, with FLAIR and T2 allowing stronger boundary capture. Attention behavior aligns with modality utility, reinforcing the model’s capacity to prioritize diagnostically salient regions in a biologically informed manner.

4.8. Qualitative Evaluation on BraTS 2018: Visual Analysis of HGG and LGG

To gain further insights into the boundary-aware behavior of the proposed PSO-GA-U-Net, we present a qualitative evaluation on the BraTS 2018 dataset for both high-grade glioma (HGG) and low-grade glioma (LGG) segmentation tasks. Figures for each modality (FLAIR, T1, T2, and T1CE) display a comprehensive five-panel layout: original image, ground truth boundary, predicted boundary, error heatmap overlay, and attention heatmap overlay. Performance is quantitatively supported by metrics: Dice Similarity Coefficient (DSC), Jaccard Index (JI), Hausdorff Distance (HD), and Average Symmetric Surface Distance (ASSD), as listed in

Table 17. Qualitative segmentation metrics (DSC, JI, HD, and ASSD) for BraTS 2018 HGG and LGG cases.

Grade	Modality	DSC	JI	HD	ASSD
HGG	FLAIR	0.9554	0.9146	5.3852	0.0545
	T1	0.9596	0.9223	8.5440	0.0511
	T2	0.9604	0.9237	8.2462	0.0488
	T1CE	0.9581	0.9196	7.6158	0.0489
LGG	FLAIR	0.9739	0.9492	4.0000	0.0322
	T1	0.9840	0.9685	65.146	0.0849
	T2	0.9836	0.9677	2.8284	0.0179
	T1CE	0.9842	0.9689	2.0000	0.0166

.

4.8.1. HGG: Boundary Sensitivity and Modality Reliability

In HGG, segmentation is hindered by infiltrative growth patterns and irregular morphology. Qualitative overlays (see figures for HGG) show that the FLAIR and T2 modalities exhibit more stable boundary capture, particularly in the regions of peritumoral edema. This is reflected in their higher DSC and JI values (T2: 0.9604/0.9237, FLAIR: 0.9554/0.9146) and a lower ASSD (T2: 0.0488).

T1CE, while effective at highlighting and enhancing cores, tends to underperform in border delineation, as evidenced by slightly elevated HD values (7.6158) corresponding to peripheral boundary inconsistencies in the error heatmaps. Attention overlays for T2 and FLAIR align closely with the true lesion boundaries, particularly in the medial and inferior tumor zones, underscoring the network’s focus on clinically salient regions.

4.8.2. LGG: Homogeneous Boundaries and Compact Attention Spread

In contrast, LGG samples typically exhibit more homogeneous tissue morphology and lower spatial complexity. This difference leads to tighter alignment between predicted and ground truth masks, as observed across all modalities. As shown in the attention heatmaps, the proposed method maintains a concentrated focus within the core lesion zones, reducing false positives at the margins.

T1 and T1CE yielded particularly high scores (DSC ≥ 0.984, JI ≥ 0.9685) and minimal ASSD (T1CE: 0.0166), showing near-perfect region match with marginal boundary offset. HD values remained low across T2 and T1CE (2.8284 and 2.0000), confirming that the model’s predictions rarely deviate sharply from true contours. However, a high HD value for T1 (65.146) in one sample reflects localized missegmentation in a poorly contrasted region, as evidenced by the brighter zones on the error heatmap.

4.8.3. Comparative Observations: HGG vs. LGG

Comparing the two tumor grades reveals that LGG segmentation benefits from consistent morphology and clearer boundaries, leading to stronger metric performance. Attention maps for LGG display sharper, less dispersed focus areas than the broader receptive fields required in HGG cases. This reflects the model’s adaptation to the spatial ambiguity and size variability inherent in HGG lesions.

FLAIR performs reliably across both grades, balancing spatial precision and structural sensitivity. T1CE, while clinically important, exhibits greater variance, particularly for HGG, due to variability in contrast enhancement. In contrast, for LGG, its structural clarity yields precise segmentation when coupled with attention-guided refinement.

These qualitative insights underline the robustness of the proposed PSO-GA-U-Net in both high- and low-grade tumor segmentation across multimodal MR imaging. The visual overlays in Figure 18 and Figure 19 provide detailed boundary-level evidence of the spatial consistency, clinical reliability, and adaptability of the model to tumor grade differences. In particular, the attention mechanisms demonstrate effective alignment with clinically salient regions, while error heatmaps help pinpoint residual discrepancies—offering potential cues for refinement in post-processing pipelines or architectural adaptations tailored to tumor heterogeneity.

4.9. Comparison with State-of-the-Art Methods

To evaluate the robustness and precision of the proposed PSO-GA-U-Net, we compare its segmentation performance against several state-of-the-art (SOTA) models across three benchmark datasets: FBTS, BraTS 2021, and BraTS 2018. The evaluation leverages widely accepted metrics—the Dice Similarity Coefficient (DSC) and the Jaccard Index (JI)—which collectively reflect volumetric overlap and boundary precision. Detailed results are provided in

Table 18. Comparison of segmentation performance on FBTS dataset.

Method	DSC	JI
Proposed (PSO-GA-U-Net)	0.9587	0.9209
U-Net-AG * [31]	0.9521	0.9093
ResUnet-TL [90]	0.9194	-
DeepLabV3+ ResNet18 [41]	0.9124	-
U-Net-T-PSO [18]	0.9312	0.8722
U-Net-ResNet50 [89]	0.9553	0.9151
EfficientNet-U-Net [91]	0.9132	-
Residual-Attention-U-Net [8]	0.9110	0.8930
EfficientNetB4 [92]	0.9339	0.8795
YOLO-U-Net [93]	0.9273	0.8915
YOLO-BT-UNetV2 [94]	0.9260	0.8630
Self-Attention U-Net [95]	0.9327	0.7800
YOLO-M-U-Net [93]	0.8915	0.8833

* Statistically lower than proposed method ($p<0.01$).

,

Table 19. Comparison of segmentation performance on BraTS 2021 dataset.

Method	DSC	JI
Proposed (PSO-GA-U-Net)	0.9406	0.8881
U-Net-AG [31]	0.9095	0.8323
E-CATBraTS [96]	0.8510	0.7660
AWA-VGG-19 [59]	0.9273	-
SPPNet-2 [43]	0.9040	-
ViT-self-attention [58]	0.9174	-
MS-Segnet [97]	0.9200	-
U-Net-ASPP-EVO [61]	0.9251	-
U-Net [98]	0.8600	0.7807
ViT-24 [99]	0.8048	-
ResU-Net [100]	0.8841	-
2C-U-Net [60]	0.8370	-
UNCE-NODE [101]	0.8949	-

and

Table 20. Comparison of segmentation performance on BraTS 2018 dataset.

Method	DSC	JI
Proposed (PSO-GA-U-Net)	0.9480	0.9024
MCCNN-CRFs [102]	0.8824	-
MSFR-Net [103]	0.8600	-
IDSFCM [104]	0.9418	0.9287
RMU-Net [105]	0.9080	0.8956
U-Net-ResNet50 [89]	0.9202	0.8536
MFFM + SCFFM (Baseline) [106]	0.8460	-
OM-Net [107]	0.9074	-
U-Net-Prep [108]	0.9000	-
Cascaded Networks [109]	0.8956	-
Ensemble-Net [110]	0.8824	-
BrainSeg-Net [111]	0.8940	-
U-Net-FCN [112]	0.8600	-
AGResU-Net [34]	0.8760	-

, together with statistical analysis where applicable.

4.9.1. FBTS Dataset: Model Precision and Boundary Localization

The FBTS dataset presents distinct class-wise structural variability across meningioma, glioma, and pituitary tumors. Table 18 outlines the segmentation results of various SOTA methods against our proposed PSO-GA-U-Net. The proposed model reports the highest Jaccard Index (0.9209) and a highly competitive DSC (0.9587), indicating both volumetric alignment and sharp spatial conformity.

The FBTS dataset consists of well-contrasted T1CE MRI slices across three tumor classes. The proposed PSO-GA-U-Net achieves a DSC of 0.9587 and a JI of 0.9209—outperforming most contemporary models in both global and class-wise segmentation accuracy. While Self-Attention U-Net achieved a slightly lower DSC (0.9327), its substantially lower JI (0.7800) suggests disproportionate overlap in segmentation, likely due to spatial over-smoothing. This discrepancy highlights that high volumetric agreement does not guarantee structural fidelity in boundary regions.

In particular, the proposed method achieves balanced improvement across both metrics, indicating not only volumetric similarity but also precise boundary localization. The inclusion of PSO for adaptive learning rate tuning promotes stable convergence during feature learning. At the same time, GA-driven dropout regulation prevents overfitting—both mechanisms enabling better generalization across varying tumor textures. A paired t-test confirms that the improvement in JI over U-Net-AG and U-Net-T-PSO is statistically significant ($p<0.01$), highlighting the improved capability of the method to minimize false positives around complex tumor borders. Statistical differences were computed using paired t-tests over five-fold cross-validation results across all test samples.

4.9.2. BraTS 2021 Dataset: Robustness on Complex Multimodal Structures

BraTS 2021 is characterized by multimodal MR inputs (FLAIR, T1, T1CE, and T2) and heterogeneous tumor subregions. The proposed method consistently achieves the best DSC (0.9406) and JI (0.8881), outperforming both traditional CNNs and advanced transformer-based architectures. The 1.53% DSC margin over AWA-VGG-19 and a 2.32% gain over U-Net-ASPP-EVO validate the model’s adaptability in handling high-dimensional, modality-fused feature spaces.

Interestingly, models incorporating transformer blocks, such as ViT-self-attention and ViT-24, while capable of capturing long-range dependencies, struggle with fine-grained tumor margins, particularly in post-contrast FLAIR modalities. This suggests that attention alone is insufficient without adaptive learning regulation, as offered by the PSO and GA modules. The dropout-guided feature pruning employed in our method reduces false positives near ventricular boundaries—a frequent issue in BraTS datasets due to edema spread and MRI noise.

These improvements stem from the hybrid PSO-GA approach, in which PSO dynamically tunes the learning rate to accommodate gradient variability across modalities. At the same time, GA prevents co-adaptation of filters by regulating neuron dropout. This dual adaptation improves the robustness of the model to modality imbalance and spatial noise. The significance test ($p<0.05$) confirms that overlap improvements (JI) are consistent across multiple patient cases and are not restricted to specific classes or modalities. This implies a reliable structure recovery even in post-contrast or edema-dominated slices.

4.9.3. BraTS 2018 Dataset: Performance in Mixed-Grade Tumor Cases

BraTS 2018 presents a nuanced challenge with a mix of high-grade (HGGs) and low-grade gliomas (LGGs), often exhibiting divergent spatial patterns and intensity profiles. The proposed PSO-GA-U-Net achieves the DSC of 0.9480 and the JI of 0.9024, outperforming IDSFCM, RMU-Net, and other fusion-enhanced baselines. Despite IDSFCM reporting a high JI (0.9287), the mismatch with its lower DSC (0.9418) may suggest inconsistent segmentation, possibly due to post-processing or thresholding.

The superior results of our method can be attributed to its robustness across grade variations. PSO’s ability to dynamically adapt learning rate schedules ensures effective representation learning across contrast-rich HGG cases and less-defined LGG tumors. The influence of GA on dropout encourages feature diversity, preventing overfitting on prominent subregions, and enabling accurate delineation of subtle tumor boundaries. A paired t-test reveals that the improvements over RMU-Net and U-Net-ResNet50 are significant at the $p<0.05$ level, particularly in LGG scenarios where boundary clarity is limited.

4.9.4. Overall Insights

These findings are supported by consistent quantitative improvements across all datasets. PSO-GA-U-Net not only achieves state-of-the-art DSC and JI values but also maintains high boundary precision, especially in structurally ambiguous or modality-impaired slices. Unlike many SOTA models that prioritize either overlap or boundary detail, our framework achieves an equilibrium between the two, enabled by hybrid optimization. PSO modulates learning rates in a modality-sensitive manner, while the GA promotes feature generalization through adaptive dropout. Together, these dynamics reinforce the robustness of the model across tumor classes and MRI protocols.

Figure 16, Figure 17, Figure 18 and Figure 19 visually reinforce these quantitative results, demonstrating that PSO-GA-U-Net preserves tumor shape, minimizes boundary drift, and maintains structural fidelity under varying contrast conditions.

Compared to the best-performing SOTA models:

• DSC: PSO-GA-U-Net ranks first across BraTS 2021, BraTS 2018, and FBTS.
• JI: Achieves the highest across all datasets.
• Boundary Handling: Outperforms transformer-based models in regions with irregular geometry.

Unlike prior PSO- or GA-only variants, our integrated framework uniquely combines learning rate modulation with population-guided dropout tuning, enabling convergence acceleration and robustness against overfitting across diverse anatomical structures and acquisition protocols.

4.10. Discussion of Findings and Limitations

The experimental results demonstrate that the proposed PSO-GA-U-Net framework consistently improves segmentation performance compared to the baseline U-Net and single-metaheuristic variants. This improvement can be attributed to the complementary characteristics of the two optimization strategies. Particle Swarm Optimization effectively explores continuous hyperparameter spaces through velocity-guided updates, enabling rapid convergence toward promising regions of the parameter space. Genetic Algorithms, in contrast, promote architectural diversity through crossover and mutation operations. The combination of these mechanisms allows the hybrid framework to balance exploration and exploitation more effectively than either method alone.

Previous studies have explored the use of metaheuristic optimization to improve deep learning segmentation models. PSO-based optimization has been used to tune training parameters and improve convergence behavior [76], while GA-based approaches have been applied to architecture search and feature selection [79]. However, most existing works rely on a single optimization strategy, which may limit their ability to explore complex search spaces that contain both continuous and discrete hyperparameters.

In contrast, the proposed PSO-GA framework integrates the exploration capability of PSO with the evolutionary diversity introduced by GA operations. This hybrid strategy enables simultaneous optimization of learning parameters, architectural configurations, and training settings within a unified search process. The ablation results indicate that this combination yields more stable and effective hyperparameter exploration, leading to improved Dice and IoU scores across the evaluated datasets.

Figure 20a–c illustrate the distribution of evaluation metrics, namely, the AUC, DSC, Jaccard Index (JI), precision, recall, F1-score, and Matthews Correlation Coefficient (MCC), for each dataset. Across all three figures, the PSO-GA-U-Net maintains consistently high central tendencies (mean and median) for each metric while exhibiting narrow interquartile ranges and minimal outlier influence—indicating stable and generalizable segmentation behavior across patient samples.

Across the three datasets, the PSO-GA-U-Net consistently maintains high median values and narrow interquartile ranges for all evaluation metrics. In the FBTS dataset (Figure 20a), high values of DSC and JI indicate a strong spatial agreement across tumor classes. For BraTS 2021 (Figure 20b), the model demonstrates stable performance across multimodal inputs, reflected by high MCC and AUC values. Similarly, the BraTS 2018 results (Figure 20c) confirm the robustness of the framework for both high-grade and low-grade glioma segmentation, even when tumor boundaries are less distinct.

Limitations: Despite these strong outcomes, several limitations merit attention:

• Domain Transferability: The model is primarily trained on public datasets with curated annotations. Generalizing to clinical real-world MRI images from various institutions (with scanner and protocol variations) may require domain adaptation techniques.
• Computational Overhead: While the PSO-GA hybrid offers notable performance improvements, the added metaheuristic layers increase computational complexity. Real-time applications may require lightweight approximations or pruning strategies.
• Class Imbalance and Rare Features: In BraTS, particularly for LGG or necrotic regions, infrequent class appearances can cause minor drops in recall. Incorporating focal loss or adaptive sampling might improve sensitivity to minority classes.
• Another limitation concerns the hyperparameter optimization protocol. In this study, the PSO–GA search is executed once using the training–validation subset rather than within a fully nested cross-validation framework. While this design substantially reduces the computational cost associated with repeated evolutionary searches for deep segmentation models, it may introduce a mild bias toward the validation subset used during optimization. Therefore, the cross-validation results are interpreted primarily as an assessment of model robustness rather than a fully nested hyperparameter evaluation.

Future Work: Building on the current findings, several directions are envisioned:

• Hybrid Transformer–CNN Integration: Future iterations could incorporate transformer encoders with evolutionary dropout tuning to explore long-range spatial dependencies without sacrificing convergence stability.
• Multi-objective Evolutionary Optimization: Extending PSO-GA to handle trade-offs between accuracy, memory, and inference time using multi-objective fitness could yield deployment-ready segmentation models.
• Clinical Deployment Studies: Evaluating the framework in longitudinal patient cohorts with clinical endpoint correlations (e.g. survival prediction and recurrence detection) can confirm real-world impact.

These findings highlight the advantage of integrating complementary metaheuristic search strategies for medical image segmentation, where both training dynamics and architectural configurations influence the final predictive performance.

5. Conclusions

In this study, we presented PSO-GA-U-Net, a hybrid deep learning framework that integrates Particle Swarm Optimization and Genetic Algorithms to enhance U-Net-based segmentation for brain tumor detection. By dynamically optimizing learning rates and dropout probabilities, the model demonstrates strong generalization capabilities and accurate boundary localization across three benchmark datasets—FBTS, BraTS 2021, and BraTS 2018. Quantitative metrics such as DSC, JI, HD, and ASSD consistently highlight the superiority of PSO-GA-U-Net over state-of-the-art methods. At the same time, qualitative results further confirm its structural precision, particularly in complex or ambiguous tumor regions.

Although the model excels in robustness and adaptability, the increased computational cost and modality-specific variation suggest areas for refinement. Future work will explore more efficient hybrid optimization strategies and the potential integration with transformer backbones to further enhance performance in real-world clinical settings. In general, PSO-GA-U-Net offers a promising direction for precision-oriented medical image segmentation through intelligent metaheuristic control.