Cross-Architecture Assessment of Hyperparameter Optimization Techniques in Convolutional Neural Networks

Abstract

Hyperparameter optimization is a critical determinant of the performance and generalizability of convolutional neural networks (CNNs), yet selecting optimal configurations remains a complex challenge due to the large, non-linear search space. While numerous optimization techniques have been explored in the literature, most comparative studies have relied on benchmark datasets, limiting insights into their practical effectiveness under real-world constraints. This study evaluates four widely used hyperparameter optimization methods (i.e., Grid Search, Optuna, Hyperopt, and DEAP) across five CNN architectures (i.e., AlexNet, GoogLeNet, LeNet, MobileNet, and ResNet18) under two distinct data regimes: the Modified National Institute of Standards and Technology (MNIST) benchmark and a custom SkinCancer image dataset. On MNIST, all configurations achieved strong classification performance, with test accuracies ranging from 95.45% to 97.61%, led by ResNet18 optimized via Grid Search (97.61%). In this simpler setting, DEAP provided the shortest optimization runtimes, while Optuna and Hyperopt offered an effective balance between performance and computational cost. In contrast, the SkinCancer dataset exhibited substantially greater variability in performance, highlighting the stronger influence of both model architecture and optimization strategy. ResNet18 achieved the highest overall test accuracy (81.50%) when optimized with Hyperopt. Although no single optimization method consistently outperformed all others, adaptive approaches frequently achieved competitive or superior predictive performance while requiring considerably less computation than exhaustive search. These findings demonstrate that hyperparameter optimization dynamics depend heavily on dataset complexity, where computational efficiency is the primary differentiator for simpler classification tasks, but optimization architecture selection becomes critical for navigating challenging medical imaging applications.

1. Introduction

Deep learning models, particularly convolutional neural networks (CNNs), have become the cornerstone of modern real-world applications including handwriting recognition [1,2,3] and medical image analysis [4,5,6]. Despite their remarkable success, the performance of CNNs is highly dependent on the appropriate selection of hyperparameters such as learning rate, batch size, optimizer type, and network-specific configurations [7]. This task, commonly referred to as the hyperparameter optimization problem, poses a considerable challenge due to the vast search space and the non-linear interactions between hyperparameters. Effective hyperparameter tuning is therefore crucial, as suboptimal configurations can significantly degrade model performance, limit generalizability, and hinder deployment in real-world applications [8]. Effective hyperparameter optimization involves several key elements: the choice of hyperparameters to tune, the definition of the search space, the optimization strategy, the evaluation criteria, and computational considerations. Understanding these components is essential for selecting appropriate optimization methods and interpreting their performance across different datasets and architectures [9,10,11].

A variety of strategies have been proposed in the literature to address this problem. Traditional approaches such as Grid Search [12,13] and Random Search [13] remain popular due to their simplicity and ease of implementation, though they are often computationally inefficient. To overcome the limitations of Grid Search and traditional exploration, hybrid and evolutionary variants have emerged to navigate discrete and continuous search spaces more efficiently. For instance, researchers have combined evolutionary search mechanics with numerical optimization to handle strict distortion constraints in medical imaging [14], while others have integrated Deep-Differential-Evolutionary algorithms with unsupervised transfer learning to optimize clustering in high-dimensional genomics [15].

More advanced methods, including Bayesian optimization frameworks like Hyperopt [16], adaptive search strategies such as Optuna [17], and evolutionary algorithms like DEAP [18], have emerged as more efficient alternatives. These methods leverage probabilistic modeling, heuristic search, or evolutionary principles to explore the hyperparameter space more effectively, striking a balance between accuracy and computational cost. Within Automated Machine Learning (AutoML) approaches, these principles have led to self-tuning architectures that eliminate manual trial-and-error altogether. This is evident in advanced multi-population architectures like the Co-Evolutionary Genetic Algorithm (CEGA), which was paired with pre-trained networks (EfficientNetB3 and DenseNet121) to automate hyperparameter discovery for brain tumor classification [19]. Similarly, meta-heuristic deep tuning techniques, such as integrating the Chimp Optimizer Algorithm (COA) for automated feature extraction [20] and employing Mutual Learning-based Artificial Bee Colony (ML-ABC) optimization within semi-supervised GAN architectures [21], have shown remarkable success in stabilizing network weights and maximizing diagnostic accuracy without heavy data augmentation.

Adaptive frameworks like Optuna and multi-fidelity pruning frameworks focus heavily on accelerating search speeds by combining intelligent sampling with early-stopping mechanisms. Demonstrating the efficacy of this paradigm, a recent two-stage multi-fidelity framework (DE-MFO-BO) utilized rapid, low-fidelity evolutionary exploration to isolate high-performing hyperparameter regions before executing deep refinement [22]. This approach yielded drastic enhancements in precision and recall for complex object detection tasks without requiring any modifications to the core deep learning architecture [22].

Meanwhile, sequential model-based optimization methods, particularly Hyperopt and Bayesian Optimization, have become the gold standard for tuning highly sensitive downstream tasks. In medical segmentation, a Gaussian-process-driven Bayesian framework (BO-UNet) successfully automated encoder–decoder structural tuning to achieve state-of-the-art segmentation maps [23], while a separate dual-optimization pipeline implemented a Bayesian Optimization Algorithm (BOA) to refine ensemble networks for hemorrhage detection [20]. Furthermore, addressing the severe dataset-specific volatility of massive biological foundation models, the adoption of a TPE-based Bayesian framework for fine-tuning scGPT consistently stabilized validation loss and preserved vital biological metrics across highly heterogeneous datasets [24].

Deep learning architectures that benefit from such optimization include standard CNNs and more complex networks such as super-resolution generative adversarial network (SRGAN) [25] and GridFormer for high-resolution image generation [26]. These architectures achieve state-of-the-art performance but are highly sensitive to hyperparameter settings, demonstrating the practical importance of systematic hyperparameter optimization studies.

While prior studies have extensively compared these optimization methods, the majority of research relies heavily on benchmark datasets such as Modified National Institute of Standards and Technology (MNIST), CIFAR-10, or ImageNet [3,27,28,29]. Aszemi and Dominic [27] addressed the challenges of hyperparameter optimization in CNNs, emphasizing that manual tuning and Grid Search are often impractical for large search spaces. Initial experiments employed Random Search, achieving moderate performance (~80% accuracy on CIFAR-10), while proposing the future use of hybrid genetic algorithms combined with local search to optimize both network architecture and training parameters. Additionally, alternative methods such as Bayesian and evolutionary algorithms have been discussed as efficient strategies for hyperparameter selection. Chowdhury et al. [28] evaluated the performance and efficiency of multiple hyperparameter optimization algorithms for deep learning models, including Grid Search, Random Search, Bayesian optimization [16,17], Genetic Algorithms (GA) [30], Hyperband [31], and Particle Swarm Optimization [32]. Experiments on VGG16 [33] and ResNet50 [34] architectures using CIFAR-10 and Intel Image Classification datasets showed that while all algorithms achieve similar optimization times, their performance varied across datasets and architectures, and no single method consistently outperformed the others. The study concluded that selecting an appropriate optimization strategy depends on the specific dataset and model. Ying [29] proposed a novel space-adjusted Bayesian optimization algorithm that selected the top percent points at each iteration to focus the search on promising regions of the hyperparameter space. When applied to test functions and datasets including MNIST and CIFAR-10, the algorithm achieved 36% and 18% improvement over canonical Efficient Global Optimization (EGO) [35], respectively. The approach adapted to variable importance and allowed sensitivity analysis for its hyperparameter, though challenges remained with local optima and theoretical convergence guarantees. The methodology is flexible and could potentially enhance other Bayesian optimization frameworks in future research. Meissner et al. [36] presented the Optimized Particle Swarm Optimization (OPSO) method, which meta-optimizes the free parameters of PSO by employing swarms within a swarm. The approach was evaluated on five artificial fitness functions and compared to two standard PSO implementations, showing improved optimization speed and quality. When applied to neural network training for predicting blood–brain barrier permeation of small molecules, OPSO reduced training time by factors of four and two relative to the other methods, producing a predictive model with good correlation across training, test, and validation data. The study concluded that meta-optimizing PSO parameters can significantly enhance overall performance and is straightforward to implement. Eberhart and Kennedy [37] described the use of PSO for non-linear function optimization and compared different paradigms, including the globally informed GBEST and locally informed LBEST models. Benchmark testing and applications, such as neural network training and robot task learning, demonstrated that GBEST converged fastest on average, while LBEST with a small neighborhood was more resistant to local minima. PSO was highlighted as a simple, robust, and memory-enabled algorithm bridging evolutionary computation and artificial life, with stochastic search and momentum effects allowing thorough exploration of solution spaces. The study concluded that PSO’s conceptual simplicity and effectiveness make it a versatile optimization method requiring minimal parameter specification. Nematzadeh et al. [38] proposed a method for hyperparameter tuning of machine learning algorithms using Gray Wolf Optimization (GWO) [39] and GA metaheuristics. Eleven algorithms, including classical models and deep neural networks with multiple architectures, were evaluated on 33 benchmark functions. The results showed that hyperparameter optimization improved training performance in all trials, with GWO demonstrating significantly better performance and faster convergence than Exhaustive Grid Search. The method automatically recommended the best algorithm and parameter configuration for a given dataset, making it suitable for datasets with unknown distributions, complex algorithm behavior, or users lacking deep expertise in data science.

Although benchmark datasets offer a controlled and consistent environment, they fail to capture the complexities of real-world datasets, which are often small, noisy, imbalanced, and heterogeneous. This reliance on benchmarks may limit the practical insights of hyperparameter optimization research, as models tuned on ideal datasets may not generalize effectively in applied domains such as medical imaging, environmental monitoring, and embedded AI systems. In these contexts, optimization strategies must balance accuracy, stability, and computational efficiency, particularly when operating under constraints of limited data, energy, or hardware resources.

Several studies have investigated hyperparameter optimization for deep learning; however, prior work often evaluates a limited set of optimization methods or focuses on single architectures and datasets. Although custom datasets are used in the literature, they are commonly employed to demonstrate newly proposed optimization techniques and are compared against only a small number of baselines. In contrast, this study provides a systematic, multidimensional evaluation by comprehensively comparing multiple optimization strategies across a diverse matrix of CNN architectures and heterogeneous datasets. The research question for the study is “How do different hyperparameter optimization techniques compare in terms of performance, stability, and computational efficiency across multiple CNN architectures under both benchmark and real-world data conditions?”

In this context, this study undertakes a comparative analysis of four hyperparameter optimization techniques (i.e., Grid Search, Optuna, Hyperopt, DEAP) across five CNN architectures (i.e., AlexNet [40], GoogLeNet [41], LeNet [42], MobileNet [43], ResNet18 [34]). CNNs were selected as the baseline architecture to ensure fair and reproducible comparison across optimization algorithms, given their well-established training dynamics and standardized evaluation protocols. This choice isolates the impact of hyperparameter optimization from architectural variability, enabling clearer attribution of performance differences to the optimization methods themselves. The evaluation is structured into two complementary experiments: one using the MNIST benchmark dataset [3] to examine optimization performance under ideal conditions, and the other using a custom SkinCancer dataset [44] to reflect the constraints of real-world medical imaging, including challenging conditions, limited data samples, and data diversity. This dual-experiment design enables a holistic assessment of the suitability of different optimization strategies across both controlled and practical contexts. Moreover, we provide statistical pairwise validation of optimization outcomes, offering rigorous evidence of method effectiveness. In addition to systematically comparing hyperparameter optimization strategies, this study provides recommendations helping practitioners choose effective optimization approaches for both benchmark and real-world applications.

The key contributions of this study are threefold. First, it provides a systematic comparison of common hyperparameter optimization methods across a diverse set of CNN models, thereby offering insights into their relative strengths and weaknesses. Second, by incorporating both a benchmark dataset and a real-world medical dataset, it highlights how optimization strategies behave under contrasting data regimes. Third, statistical pairwise tests were conducted on both datasets to complement conventional performance metrics and to better understand the impact of different hyperparameter optimization methods.

2. Materials and Methods

2.1. Datasets

(1)

Modified National Institute of Standards and Technology (MNIST) Dataset

The MNIST dataset is a benchmark dataset widely used for evaluating image classification and handwritten digit recognition algorithms [3]. It comprises a total of 70,000 labeled grayscale images of handwritten digits ranging from 0 to 9, with 60,000 images allocated for training and 10,000 for testing. Each image is a $28\times 28$ pixel square. The digits were originally collected from a broader set in the MNIST dataset, then centered and normalized to improve uniformity. The dataset is balanced across classes, with approximately 7000 examples per digit. Due to its simplicity, consistency, and relatively small size, MNIST is commonly used for prototyping and benchmarking machine learning algorithms, particularly in the early stages of deep learning research. Figure 1 illustrates one representative image from each of the ten digit classes (0–9), highlighting the variation in handwriting styles captured within the dataset. Despite being considered a “solved” problem in recent years, MNIST continues to serve as a fundamental entry point for experimentation in image classification and neural network design.

Figure 1. A sample from each class of the MNIST dataset [3].

(2)

SkinCancer Dataset

The SkinCancer dataset utilized in this study represents a real-world medical imaging resource sampled directly from the International Skin Imaging Collaboration (ISIC) Archive via a curated public repository [44,45]. The dataset comprises 3600 balanced dermoscopic images, with exactly 1800 benign and 1800 malignant cases, all standardized to a resolution of $224\times 224$ pixels [44]. Per the established ISIC Archive data provenance standards [45,46], the ground truth annotations follow a strict clinical validation hierarchy. Malignant cohorts were verified strictly through histopathological examination (biopsy) to ensure absolute diagnostic accuracy. Benign cohorts were verified either via histopathology, in vivo confocal microscopy, or the formal clinical consensus of multiple expert dermatologists. This rigorous tracking guarantees the integrity and reliability of the classification labels used throughout our benchmarking pipeline. Unlike synthetic or simulated datasets, it reflects the variability and complexity of actual clinical cases, making it highly relevant for training and evaluating machine learning models intended for medical application. Its balanced design helps mitigate class imbalance issues frequently encountered in clinical datasets, enabling robust model development and fair performance assessment. Figure 2 shows some sample images from each class of the dataset. This dataset is particularly valuable because it incorporates the nuances of real-world data, such as variations in lighting, skin tones, and mole characteristics, which are critical for developing robust models that can generalize well to unseen clinical scenarios. Therefore, leveraging this dataset allows for a more reliable evaluation of model performance in a context that closely resembles actual clinical practice.

Figure 2. A few samples from each class of the SkinCancer dataset [44].

2.2. CNN Architectures

(1)

LeNet

LeNet, developed by Yann LeCun [42] in the late 1980s and widely known as LeNet-5, is one of the earliest CNN architectures and was originally designed for handwritten digit recognition, such as classifying digits from the MNIST dataset. It introduced the foundational components of modern CNNs, including convolutional layers for feature extraction, subsampling (pooling) layers for dimensionality reduction, and fully connected layers for classification. LeNet is relatively shallow and consists of only seven layers, making it computationally efficient and suitable for small-scale image tasks. Its simplicity and modular design laid the groundwork for deeper and more complex CNNs, and while it lacks advanced features like Rectifying Linear Unit (ReLU) activations or batch normalization, it remains an essential historical milestone in the evolution of deep learning.

(2)

AlexNet

AlexNet, proposed by Alex Krizhevsky et al. [40] in 2017, represented a breakthrough in CNN architecture that won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) by a significant margin, reigniting global interest in deep learning. It consists of eight learned layers (five convolutional and three fully connected), and introduced several key innovations: the use of ReLU for faster training, dropout layers to prevent overfitting, and overlapping max-pooling layers. Notably, AlexNet leverages GPU acceleration to train on large-scale datasets, a technique that became standard practice thereafter. Compared to LeNet, AlexNet is significantly deeper and more computationally intensive, and is designed to handle complex, high-resolution images from large datasets like ImageNet rather than small, simple inputs.

(3)

GoogLeNet

GoogLeNet, introduced by Szegedy et al. [41] in 2015 as the winner of ILSVRC 2014, is notable for its deep architecture (22 layers) and the introduction of the Inception module, a novel approach to improving efficiency and depth. The Inception module applies multiple convolution filters (1 $\times$ 1, 3 $\times$ 3, 5 $\times$ 5) and a pooling operation in parallel, concatenating their outputs to capture features at multiple scales without excessively increasing computational cost. This design reduces the number of parameters compared to fully connected layers while maintaining high accuracy. Unlike traditional sequential CNNs like AlexNet, GoogLeNet emphasizes computational efficiency and structural diversity within each layer, setting the stage for future modular and scalable architectures.

(4)

ResNet18

ResNet18 is a variant of the Residual Network (ResNet) architecture introduced by He et al. [34] in 2016, designed to address the degradation problem in deep neural networks, where increasing depth leads to worse performance due to vanishing gradients. The key innovation in ResNet is the use of residual connections, which are shortcut paths that allow the network to learn residual functions instead of direct mappings. These identity shortcuts enable efficient gradient flow during backpropagation, allowing networks to be trained with hundreds of layers without performance degradation. ResNet18, with 18 layers, serves as a lightweight yet powerful model that balances depth and computational efficiency. Unlike earlier models like AlexNet or GoogLeNet, which focus on architectural width or multi-scale filtering, ResNet emphasizes depth with structural support to ensure trainability and performance stability.

(5)

MobileNet

MobileNet, developed by Howard et al. [43] in 2017, is a lightweight CNN architecture optimized for mobile and embedded vision applications where computational resources are limited. Its core innovation is the use of depthwise separable convolutions, which decompose standard convolutions into a depthwise convolution (per-channel spatial filtering) followed by a pointwise convolution (1 $\times$ 1 convolution across channels). This reduces the model size and computational cost dramatically, while maintaining competitive accuracy on tasks like image classification and object detection. MobileNet also allows customization via width and resolution multipliers, enabling trade-offs between accuracy and efficiency. Compared to heavier models like AlexNet or GoogLeNet, MobileNet is specifically designed to run efficiently on low-power devices with limited memory.

2.3. Hyperparameter Optimization Techniques

(1)

Grid Search

Grid Search is a brute force hyperparameter optimization technique that systematically evaluates all possible combinations of a predefined set of hyperparameter values [12,13]. For each hyperparameter, the user specifies a discrete set of values, and the model is trained and evaluated for every combination across the Cartesian product of these values. While simple and exhaustive, Grid Search becomes computationally expensive as the number of hyperparameters or their candidate values increases, making it less suitable for high-dimensional search spaces. Its main advantage is reproducibility and thoroughness, but its rigid grid structure can lead to wasted resources on uninformative regions of the hyperparameter space, especially when only a few hyperparameters have a strong influence on performance.

(2)

Distributed Evolutionary Algorithms in Python (DEAP)

DEAP is a flexible evolutionary computation framework that supports a wide range of bio-inspired optimization algorithms, including genetic algorithms (GAs) [47] for hyperparameter tuning [48]. In this approach, a population of candidate solutions (i.e., sets of hyperparameters) evolves over multiple generations using operators such as selection, crossover, and mutation. DEAP is particularly powerful for navigating complex, non-differentiable, or highly non-convex search spaces where gradient-based or probabilistic methods may struggle [18,49]. Its flexibility and extensibility allow researchers to implement custom fitness functions and hybrid strategies. Unlike Bayesian approaches like Hyperopt or Optuna, DEAP is population-based and does not rely on probabilistic models, making it well-suited for global optimization at the cost of higher computational demands.

(3)

Optuna

Optuna is a modern, open-source hyperparameter optimization framework that employs automated, adaptive search strategies using techniques like Tree-structured Parzen Estimators (TPE) [50] and pruning to efficiently explore hyperparameter spaces [17]. Unlike Grid Search, Optuna is designed to be dynamic and feedback-driven as it constructs a search space programmatically, performs intelligent sampling based on past trial outcomes, and prunes unpromising trials early to reduce computation time. Optuna also supports parallel execution and visualization tools, making it scalable and user-friendly. Its performance-oriented and model-agnostic design makes it highly suitable for tuning complex models like deep neural networks.

(4)

Hyperopt

Hyperopt is a Python library for serial and parallel hyperparameter optimization using Bayesian optimization algorithms, most notably the TPE [16]. It allows users to define search spaces that include real, integer, categorical, and conditional parameters, and it explores these using probabilistic models to predict which configurations are likely to perform well. Compared to Grid Search, Hyperopt uses prior results to inform future trials, significantly reducing the number of evaluations needed. While similar in principle to Optuna, Hyperopt is slightly older and is often used in research environments requiring custom search space definitions. However, it lacks some of the dynamic features and user interface improvements found in newer frameworks like Optuna.

2.4. Experimental Design

This study is designed to investigate the effectiveness of various hyperparameter optimization techniques when applied to CNNs spanning different generations of deep learning architectures. The primary objective is to perform a comparative analysis of how these optimization approaches perform across both ideal benchmark datasets and more practical, limited-data scenarios. This dual focus enables a comprehensive evaluation of tuning strategies, offering insights into their suitability depending on model complexity, data availability, and application context. To achieve this, we have structured our investigation into two experiments, each targeting a different data context while maintaining consistent model and optimization technique configurations:

• Experiment A: This experiment evaluated four hyperparameter optimization methods (i.e., Grid Search, Optuna, Hyperopt, DEAP) across five CNN models (i.e., AlexNet, GoogLeNet, LeNet, MobileNet, ResNet18). The experiments were conducted using the MNIST dataset, a widely used benchmark consisting of grayscale images of handwritten digits. This dataset was chosen due to its clean, balanced, and structured nature, which allows for controlled, large-scale experimentation in an ideal environment where data is plentiful and consistent.
• Experiment B: This experiment mirrors the setup of Experiment A but was conducted using a small-scale, custom SkinCancer image dataset. Unlike MNIST, this dataset reflects a more realistic medical imaging scenario characterized by limited data and increased variability. The goal here was to observe how each optimization method copes with practical constraints such as data scarcity and class heterogeneity, which are common challenges in real-world deep learning applications.

The selected models represent key milestones in CNN development. LeNet reflects the early stages of CNN design with a simple, shallow architecture; AlexNet introduced deeper layers, ReLU activation, and Graphics Processing Unit (GPU) acceleration, catalyzing the deep learning revolution; GoogLeNet contributed the Inception module, allowing multi-scale feature extraction with reduced computational cost; ResNet-18 demonstrated the power of residual connections to facilitate the training of deeper networks; and MobileNet offers a modern, efficient design optimized for mobile and embedded applications using depthwise separable convolutions. By choosing architectures from different generations, we ensure that our evaluation spans a range of design principles, network depths, and computational footprints.

Similarly, the four hyperparameter optimization techniques selected for this study represent a broad methodological spectrum. Grid Search serves as a baseline through exhaustive combinatorial exploration. Hyperopt and Optuna both leverage Bayesian optimization via TPE, allowing for adaptive and guided exploration of the hyperparameter space based on prior trial performance. DEAP, by contrast, adopts an evolutionary algorithm approach using population-based optimization through genetic operations such as mutation and crossover. This diversity in optimization strategies allows us to assess each method not only in terms of performance but also in efficiency, adaptability, and suitability for different model and data complexities.

2.5. Experimental Protocols and Evaluation Measures

All experiments in this study were implemented using Python 3.12.11 and PyTorch 2.8.0. Model development and training were conducted on Google Colab Pro+ with an A100 GPU (80 GB VRAM). The CNN architectures were trained from scratch for 10 epochs with a batch size of 64, using the Adam optimizer and cross-entropy loss. Single-channel images were up-sampled to $64\times 64$ pixels, normalized, and processed without spatial data augmentation. Accordingly, the first layer of each network was modified to accept a single input channel, while the final layer was mapped to match the target classes of each dataset using the optimized dropout rate.

To ensure strict test-set isolation and prevent data leakage during hyperparameter optimization (HPO) for both Experiment A and Experiment B, the datasets were partitioned into an 80:10:10 train-validation-test split. The HPO algorithms explored the search space using validation set accuracy as the sole performance metric. Following the completion of the 64-trial HPO budget, the optimal hyperparameter configuration was selected. Using this optimal configuration, a brand-new model instance was initialized and completely retrained from scratch to ensure no parameter states or weight histories carried over from the tuning trials. During this final retraining stage, the train subset was utilized within the epoch loop to update the network weights, while the validation subset was evaluated at the end of each epoch to continuously monitor validation loss and accuracy.

Once the fixed training duration was complete, the live model weights from the very last epoch (Epoch 10) were evaluated exactly once on the isolated test set to record the final test accuracy. Due to the high computational overhead of the multi-architecture, multi-seed evaluation matrix, nested cross-validation and early stopping were not utilized; all models were trained for a fixed epoch duration. To maintain the integrity of the experiments, the same data split was used for all hyperparameter optimization methods and model runs within each dataset. Data preprocessing steps such as normalization and resizing (where required) were uniformly applied across all samples to ensure comparability.

Each model was trained using four different hyperparameter optimization techniques: Grid Search, Optuna, Hyperopt, and DEAP. For the Bayesian optimization frameworks, both toolkits were evaluated over a total budget of 64 trials without pruning mechanisms. Hyperparameter sampling was driven by the Tree-structured Parzen Estimator algorithm, configured to maximize classification accuracy. The DEAP framework utilized a genetic algorithm with a population size of 16 over 3 generations. Optimization employed tournament selection (size = 3), uniform crossover with a crossover probability of 0.5, and a discrete random resetting mutation with an individual-level rate of 0.3 and a per-gene mutation probability of 0.3. The optimal hyperparameters were extracted from the best individual across all evaluated trials.

These optimizers were tasked with tuning key hyperparameters including learning rate, dropout rate and weight decay. The hyperparameter search space was set as follows: learning rate $\in \{1\times {10}^{-1}, 1\times {10}^{-2}, 1\times {10}^{-3}, 1\times {10}^{-4}\};$ weight decay $\in \{0, 1\times {10}^{-8}, 1\times {10}^{-4}, 1\times {10}^{-2}\}$; and dropout rate $\in \{0.1, 0.2, 0.3, 0.4\}.$ The optimization process was constrained by a fixed budget of evaluations (64 trials) per technique to ensure a fair comparison in terms of computational cost and time. Bayesian methods are inherently sample-efficient and are commonly applied under limited evaluation budgets when training deep networks [16,17,51]. Although evolutionary approaches may benefit from larger populations or longer runs, constrained budgets are frequently used to assess their relative performance and stability rather than asymptotic convergence [18]. This design isolates optimizer behavior under identical and realistic computational constraints.

To assess model performance, we employed a combination of standard classification metrics and computational efficiency measures. Specifically, the following evaluation metrics were used:

• Accuracy: Training accuracy measures how well the model fits the training data, providing insight into its learning capacity. Testing accuracy evaluates the model’s generalization performance on unseen data, which is critical for assessing real-world applicability. Equation (1) presents the mathematical expression for accuracy as a function of True Positives ($TP$), False Positives ($FP$), True Negatives ($TN$), and False Negatives ($FN$).

  $$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

  (1)
• Precision: Calculates the proportion of correctly predicted positive samples among all samples predicted as positive. This is especially relevant in datasets with class imbalance. The mathematical expression used to calculate precision is given in Equation (2).

  $$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

  (2)
• Recall: Measures the proportion of correctly predicted positive samples among all actual positive samples, indicating the model’s sensitivity. The mathematical expression used to calculate recall is given in Equation (3).

  $$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

  (3)
• F1 Score: The harmonic mean of precision and recall, offering a balanced view of the model’s predictive power when both false positives and false negatives are of concern. Equation (4) shows the mathematical expression for calculating F1 score.

  $$\mathrm{F}1=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

  (4)
• Time Taken: Captures the total runtime required for each optimization technique to complete its search and training process, serving as a practical measure of computational efficiency.
• Receiver Operating Characteristic (ROC) curve: a graphical representation of a classifier’s ability to distinguish between classes, plotting the true positive rate against the false positive rate at various threshold settings. The Area Under the Curve (AUC) quantifies this performance as a single scalar value, where an AUC of 1.0 indicates perfect classification and 0.5 represents random guessing.

Rather than relying on a standard computer-vision classification threshold of 0.5, we established a clinical operational threshold policy for the skin cancer dataset, calibrated to prioritize sensitivity over specificity and approach the 95% target for malignancies, aligning our evaluation framework with safety-first benchmarks in recent literature [52,53]. To determine this threshold, we iteratively reduced the decision threshold below 0.5 on the validation set and evaluated multiple discrete values until obtaining a classification recall reasonably close to the 95% target. Crucially, this threshold-selection procedure serves strictly as a computational HPO benchmarking framework for algorithmic comparison, and the interpretation of these results remains limited to computational performance rather than clinical validation.

Algorithmic variance and stochastic initialization bias were strictly controlled by executing 20 independent runs with 20 distinct random seeds for each of the four optimizers across all five architectures and two datasets. To satisfy the structural assumptions of paired statistical analysis, a shared, fixed pool of 20 random initialization seeds was consistently applied across all optimization conditions within each comparison. Consequently, identical seeds governed the data splits and network initializations, directly pairing the performance samples across the HPO methods and generating $N=20$ independent performance samples per configuration. A two-step non-parametric statistical framework was subsequently employed at a $5\%$ significance level ($\alpha =0.05$) to assess the performance differences among the competing optimization algorithms for each dataset using the obtained test accuracy values as the metric for all statistical evaluations. First, a global Friedman test was conducted for each dataset and CNN architecture configuration to evaluate the overall statistical significance of performance differences across the optimization methods. Following a significant Friedman test result, post hoc pairwise comparisons were executed using the Wilcoxon signed-rank test to isolate the specific variations between the four optimization methods (comprising six distinct pairs per architecture within each dataset). To quantify precision and stability, final classification accuracy values are reported as the mean performance with its corresponding 95% Confidence Interval (CI) via a Student’s $t-$distribution $(df=19)$ [54]. Furthermore, to evaluate the practical magnitude of improvement beyond binary p-values, the Matched-Pairs Rank-Biserial Correlation $(r_{rb}={\displaystyle \frac{Z}{\sqrt{N}}})$ was computed as the native non-parametric effect size for statistically significant pairwise Wilcoxon signed-rank tests to directly quantify optimization superiority [55]. To strictly control the family-wise error rate and mitigate the risk of Type I errors (false positives) arising from these multiple comparisons, the Holm-Bonferroni correction was applied to all post hoc $p$-values.

The final classification results are reported as the mean accuracy $\pm$ standard deviation across the 20 runs. The learning curves presented correspond to the mean values across the 20 independent runs.

3. Results

3.1. Experiment A Results

The experiments on the MNIST dataset demonstrated strong classification performance across all CNN architectures and hyperparameter optimization methods, reflecting the relative simplicity of the benchmark. As summarized in Table 1, test accuracies ranged from 95.45% to 97.61%, with the highest performance achieved by ResNet18 using Grid Search (0.9761 ± 0.0019), followed closely by GoogLeNet with Grid Search (0.9747 ± 0.0017). Among the shallower networks, LeNet attained a maximum test accuracy of 0.9658 ± 0.0008 using Optuna, while AlexNet achieved 0.9717 ± 0.0008 under Optuna. MobileNet exhibited the lowest overall performance, with test accuracies varying between 0.9545 and 0.9643 across optimization approaches. The exceptionally narrow 95% confidence intervals across all evaluations (with margins of error strictly bounded within $\pm 0.0020$) underscore the high precision and statistical stability of these classification metrics. Precision, recall, F1-score, and specificity closely mirrored test accuracy for all models, indicating balanced classification performance without notable class bias. Furthermore, all architectures achieved exceptionally high AUC-ROC and AUC-PR values (>0.993), confirming robust discriminative capability despite differences in classification accuracy.

Table 1. Summary of Quantitative Results for the MNIST Dataset. Test accuracies are reported alongside their 95% confidence intervals [95% CI], and algorithmic search time is reported in seconds (s).

Method	Train Accuracy	Test Accuracy	Val Accuracy	Precision	Recall	F1 Score	Specificity	AUC ROC	AUC PR	Time Taken
LeNet
Grid Search	0.9615 ± 0.0026	0.9638 ± 0.0019 [0.9629, 0.9647]	0.9636 ± 0.0020	0.9637 ± 0.0019	0.9639 ± 0.0020	0.9638 ± 0.0019	0.9637 ± 0.0020	0.9950 ± 0.0011	0.9947 ± 0.0012	8229.5
Optuna	0.9673 ± 0.0010	0.9658 ± 0.0008 [0.9654, 0.9662]	0.9659 ± 0.0009	0.9657 ± 0.0008	0.9659 ± 0.0009	0.9658 ± 0.0008	0.9657 ± 0.0009	0.9954 ± 0.0005	0.9951 ± 0.0006	6401.9
Hyperopt	0.9564 ± 0.0015	0.9614 ± 0.0013 [0.9608, 0.9620]	0.9609 ± 0.0014	0.9615 ± 0.0013	0.9613 ± 0.0014	0.9614 ± 0.0013	0.9615 ± 0.0014	0.9941 ± 0.0008	0.9938 ± 0.0009	6448.6
DEAP	0.9653 ± 0.0013	0.9619 ± 0.0011 [0.9614, 0.9624]	0.9622 ± 0.0012	0.9620 ± 0.0011	0.9618 ± 0.0012	0.9619 ± 0.0011	0.9620 ± 0.0012	0.9944 ± 0.0007	0.9941 ± 0.0008	5673.1
AlexNet
Grid Search	0.9763 ± 0.0020	0.9707 ± 0.0018 [0.9699, 0.9715]	0.9712 ± 0.0019	0.9707 ± 0.0018	0.9708 ± 0.0019	0.9708 ± 0.0018	0.9707 ± 0.0019	0.9954 ± 0.0010	0.9951 ± 0.0011	16,213.5
Optuna	0.9781 ± 0.0009	0.9717 ± 0.0008 [0.9713, 0.9721]	0.9723 ± 0.0008	0.9717 ± 0.0008	0.9717 ± 0.0009	0.9717 ± 0.0008	0.9717 ± 0.0009	0.9957 ± 0.0005	0.9955 ± 0.0005	12,728.9
Hyperopt	0.9749 ± 0.0013	0.9697 ± 0.0012 [0.9691, 0.9703]	0.9702 ± 0.0011	0.9698 ± 0.0012	0.9697 ± 0.0013	0.9697 ± 0.0012	0.9698 ± 0.0012	0.9951 ± 0.0007	0.9948 ± 0.0008	12,809.4
DEAP	0.9719 ± 0.0012	0.9707 ± 0.0010 [0.9702, 0.9712]	0.9708 ± 0.0011	0.9708 ± 0.0010	0.9707 ± 0.0011	0.9707 ± 0.0010	0.9707 ± 0.0011	0.9954 ± 0.0006	0.9951 ± 0.0007	11,036.8
GoogLeNet
Grid Search	0.9790 ± 0.0019	0.9747 ± 0.0017 [0.9739, 0.9755]	0.9751 ± 0.0018	0.9746 ± 0.0017	0.9748 ± 0.0018	0.9747 ± 0.0017	0.9746 ± 0.0018	0.9966 ± 0.0010	0.9963 ± 0.0011	16,620.3
Optuna	0.9641 ± 0.0011	0.9732 ± 0.0008 [0.9728, 0.9736]	0.9723 ± 0.0009	0.9732 ± 0.0008	0.9732 ± 0.0009	0.9732 ± 0.0008	0.9732 ± 0.0009	0.9962 ± 0.0006	0.9960 ± 0.0006	12,903.2
Hyperopt	0.9777 ± 0.0014	0.9722 ± 0.0012 [0.9716, 0.9728]	0.9727 ± 0.0013	0.9723 ± 0.0012	0.9721 ± 0.0013	0.9722 ± 0.0012	0.9723 ± 0.0013	0.9958 ± 0.0008	0.9955 ± 0.0009	13,041.4
DEAP	0.9759 ± 0.0013	0.9727 ± 0.0010 [0.9722, 0.9732]	0.9730 ± 0.0011	0.9728 ± 0.0010	0.9726 ± 0.0011	0.9727 ± 0.0010	0.9728 ± 0.0011	0.9960 ± 0.0007	0.9957 ± 0.0007	11,241.6
ResNet18
Grid Search	0.9765 ± 0.0021	0.9761 ± 0.0019 [0.9752, 0.9770]	0.9762 ± 0.0020	0.9760 ± 0.0019	0.9762 ± 0.0020	0.9761 ± 0.0019	0.9760 ± 0.0020	0.9970 ± 0.0011	0.9968 ± 0.0012	15,882.5
Optuna	0.9850 ± 0.0009	0.9712 ± 0.0008 [0.9708, 0.9716]	0.9726 ± 0.0009	0.9711 ± 0.0008	0.9713 ± 0.0009	0.9712 ± 0.0008	0.9711 ± 0.0009	0.9964 ± 0.0006	0.9962 ± 0.0006	12,615
Hyperopt	0.9850 ± 0.0011	0.9707 ± 0.0012 [0.9701, 0.9713]	0.9721 ± 0.0013	0.9708 ± 0.0012	0.9706 ± 0.0013	0.9707 ± 0.0012	0.9708 ± 0.0013	0.9961 ± 0.0008	0.9958 ± 0.0009	12,742.6
DEAP	0.9850 ± 0.0010	0.9707 ± 0.0010 [0.9702, 0.9712]	0.9721 ± 0.0011	0.9706 ± 0.0010	0.9708 ± 0.0011	0.9707 ± 0.0010	0.9706 ± 0.0011	0.9960 ± 0.0007	0.9957 ± 0.0008	10,858
MobileNet
Grid Search	0.9741 ± 0.0022	0.9643 ± 0.0020 [0.9634, 0.9652]	0.9653 ± 0.0021	0.9642 ± 0.0020	0.9644 ± 0.0021	0.9643 ± 0.0020	0.9642 ± 0.0021	0.9952 ± 0.0011	0.9949 ± 0.0012	14,883.1
Optuna	0.9653 ± 0.0012	0.9545 ± 0.0011 [0.9540, 0.9550]	0.9555 ± 0.0010	0.9544 ± 0.0011	0.9546 ± 0.0012	0.9545 ± 0.0011	0.9544 ± 0.0012	0.9939 ± 0.0007	0.9936 ± 0.0008	11,565
Hyperopt	0.9621 ± 0.0014	0.9594 ± 0.0012 [0.9588, 0.9600]	0.9597 ± 0.0013	0.9595 ± 0.0012	0.9593 ± 0.0013	0.9594 ± 0.0012	0.9595 ± 0.0013	0.9944 ± 0.0008	0.9941 ± 0.0009	11,666.2
DEAP	0.9677 ± 0.0013	0.9609 ± 0.0010 [0.9604, 0.9614]	0.9615 ± 0.0011	0.9610 ± 0.0010	0.9608 ± 0.0011	0.9609 ± 0.0010	0.9610 ± 0.0011	0.9947 ± 0.0007	0.9944 ± 0.0008	10,057.6

With regard to the optimization frameworks, no single method consistently achieved the best results across all architectures. Optuna produced the highest test accuracy for both LeNet (0.9658) and AlexNet (0.9717), whereas Grid Search yielded the strongest performance for GoogLeNet (0.9747), ResNet18 (0.9761), and MobileNet (0.9643). Hyperopt generally achieved competitive results but rarely outperformed the leading methods, while DEAP consistently produced accuracies comparable to those of Hyperopt and Optuna. The optimized hyperparameters shown in Table 2 indicate that learning rates of either 10⁻³ or 10⁻⁴ were most frequently selected, whereas optimal dropout values varied considerably across architectures and optimization techniques. For example, Optuna favored a low dropout rate of 0.1 for AlexNet and MobileNet but selected higher values for GoogLeNet and ResNet18, highlighting the architecture-dependent nature of hyperparameter selection.

Table 2. Summary of Optimized Hyperparameters for MNIST Dataset.

Method	Learning Rate	Dropout Rate	Weight Decay
LeNet
Grid Search	$1.00\times {10}^{-3}$	0.4	$1.00\times {10}^{-4}$
Optuna	$1.00\times {10}^{-3}$	0.2	$1.00\times {10}^{-4}$
Hyperopt	$1.00\times {10}^{-3}$	0.4	$1.00\times {10}^{-2}$
DEAP	$1.00\times {10}^{-3}$	0.3	$1.00\times {10}^{-2}$
AlexNet
Grid Search	$1.00\times {10}^{-4}$	0.4	$1.00\times {10}^{-8}$
Optuna	$1.00\times {10}^{-4}$	0.1	$1.00\times {10}^{-4}$
Hyperopt	$1.00\times {10}^{-4}$	0.3	$1.00\times {10}^{-4}$
DEAP	$1.00\times {10}^{-3}$	0.1	$1.00\times {10}^{-4}$
GoogLeNet
Grid Search	$1.00\times {10}^{-4}$	0.2	$1.00\times {10}^{-8}$
Optuna	$1.00\times {10}^{-3}$	0.3	$1.00\times {10}^{-4}$
Hyperopt	$1.00\times {10}^{-4}$	0.2	$1.00\times {10}^{-4}$
DEAP	$1.00\times {10}^{-4}$	0.1	$1.00\times {10}^{-4}$
ResNet18
Grid Search	$1.00\times {10}^{-3}$	0.4	$1.00\times {10}^{-8}$
Optuna	$1.00\times {10}^{-4}$	0.2	$1.00\times {10}^{-4}$
Hyperopt	$1.00\times {10}^{-4}$	0.3	$1.00\times {10}^{-2}$
DEAP	$1.00\times {10}^{-4}$	0.3	$1.00\times {10}^{-2}$
MobileNet
Grid Search	$1.00\times {10}^{-3}$	0.3	—
Optuna	$1.00\times {10}^{-3}$	0.1	$1.00\times {10}^{-2}$
Hyperopt	$1.00\times {10}^{-3}$	0.1	$1.00\times {10}^{-4}$
DEAP	$1.00\times {10}^{-3}$	0.3	$1.00\times {10}^{-2}$

Note: The em-dash (—) denotes a weight decay value of 0 (no regularization applied).

A more pronounced distinction among the optimization methods was observed in computational efficiency. Across all architectures, Grid Search required the greatest runtime, ranging from approximately 8230 s for LeNet to over 16,620 s for GoogLeNet. In contrast, DEAP consistently achieved the shortest execution times, reducing optimization time by approximately 32% relative to Grid Search while maintaining similar predictive performance. Optuna and Hyperopt offered an intermediate balance between runtime and accuracy, generally requiring substantially less computation than Grid Search while achieving comparable or superior classification results. Notably, for the deep architectures GoogLeNet and ResNet18, DEAP reduced runtime to approximately 11,242 s and 10,858 s, respectively, compared with 16,620 s and 15,883 s obtained through Grid Search. These findings suggest that evolutionary optimization approaches can provide considerable computational savings without substantial degradation in predictive performance.

The learning curves presented in Figure 3 and Figure 4 indicate stable training behavior across all models and optimization techniques. Training and validation accuracies increased rapidly during the early epochs and subsequently plateaued, while training and validation losses consistently declined, demonstrating effective convergence. Although deeper architectures generally attained higher final accuracies, the gaps between training and validation performance remained small, suggesting limited overfitting.

Figure 3. Training and validation accuracy curves for the CNN models trained using multiple hyperparameter optimization techniques on the MNIST dataset.

Figure 4. Training and validation loss curves for the CNN models trained using multiple hyperparameter optimization techniques on MNIST dataset.

The ROC curves shown in Figure 5 further support these observations, with all models achieving AUC values exceeding 0.99 regardless of the optimization strategy employed. Overall, for the MNIST dataset, the choice of optimization method primarily influenced training efficiency rather than final accuracy, with Optuna and Hyperopt offering a favorable balance between performance and runtime, and DEAP excelling in computational speed.

Figure 5. Cumulative ROC curves for the CNN models trained using multiple hyperparameter optimization techniques on MNIST dataset.

The Holm Corrected Wilcoxon Test and Friedman Test from Table 3 and Table 4, respectively, confirmed that the choice of hyperparameter optimization method significantly influenced performance across all architectures, as evidenced by the Friedman test results for LeNet ($p=0.0019$), AlexNet ($p=0.0008$), GoogLeNet $(p=0.0004$), ResNet18 ($p=0.0007$), and MobileNet ($p=0.0003$), all indicating significant differences among optimization strategies ($p<0.01$). Pairwise comparisons showed that these differences were not uniformly distributed across methods. For LeNet, Grid Search, Hyperopt, and Optuna did not differ significantly from one another, whereas DEAP exhibited highly significant differences relative to all three competing methods with strong effect sizes against Grid Search $(p=0.0048, r_{rb}=0.6306),$ Hyperopt $(p=0.0048, r_{rb}=0.6306),$ and Optuna $(p=0.0035, r_{rb}=0.6529).$ A similar pattern was observed for AlexNet, GoogLeNet, and ResNet18, where Grid Search, Optuna, and Hyperopt were often statistically comparable ($p>0.05$), while DEAP frequently showed significant differences from the other approaches, yielding rank-biserial effect sizes ranging from $r_{rb}=0.5500$ to $r_{rb}=0.7242$ across these three architectures. MobileNet displayed the greatest degree of statistical separation, with multiple significant pairwise differences involving Hyperopt, DEAP, and Optuna. However, despite the statistical significance identified by both the Friedman and pairwise tests, the practical performance differences remained relatively small, with test accuracies typically differing by less than 2% across optimization methods. These findings suggest that the observed significance arises primarily from consistent performance variations across repeated trials rather than substantial disparities in predictive capability, indicating that optimization strategy has a measurable but limited impact on MNIST classification performance.

Table 3. Summary of Holm Corrected Wilcoxon Test Results and Rank-Biserial Effect Sizes for the MNIST Dataset. Active cells report post hoc pairwise p-values along with their corresponding non-parametric effect sizes in parentheses.

	Grid Search	Hyperopt	DEAP
LeNet
Hyperopt	0.1062
DEAP	0.0048 ** ($r_{rb}=0.6306)$	0.0048 ** ($r_{rb}=0.6306)$
Optuna	0.1062	0.1062	0.0035 **$(r_{rb}=0.6529)$
AlexNet
Hyperopt	0.0921
DEAP	0.0046 **$(r_{rb}=0.6337)$	0.0139 * $(r_{rb}=0.5500)$
Optuna	0.1108	0.1108	0.0057 **${ (r}_{rb}=0.6182)$
GoogLeNet
Hyperopt	0.1691
DEAP	0.0019 **$(r_{rb}=0.6944)$	0.0054 ** $(r_{rb}=0.6221)$
Optuna	0.1816	0.1816	0.0012 **${ (r}_{rb}=0.7242)$
ResNet18
Hyperopt	0.0952
DEAP	0.0019 ** $(r_{rb}=0.6944)$	0.0077 ** $(r_{rb}=0.5959)$
Optuna	0.1000	0.1000	0.0012 ** $(r_{rb}=0.7242)$
MobileNet
Hyperopt	0.0739
DEAP	0.0192 * $(r_{rb}=0.5236)$	0.0023 ** $(r_{rb}=0.6817)$
Optuna	0.0833	0.0192 * $(r_{rb}=0.5236)$	0.0023 ** $(r_{rb}=0.6817)$

* p < 0.05, ** p < 0.01. The gray cells indicate redundant comparisons.

Table 4. Friedman Test Results for the MNIST Dataset.

Model	p-Value
LeNet	0.0019 **
AlexNet	0.0008 **
GoogLeNet	0.0004 **
ResNet18	0.0007 **
MobileNet	0.0003 **

** p < 0.01.

The violin plots in Figure 6 further illustrate the distribution of model performance across optimization strategies. Optuna generally exhibited narrow distributions and consistently strong median performance, particularly for LeNet and AlexNet, whereas Grid Search displayed mixed behavior on deeper models, showing a wide spread and lower median on ResNet18 but remaining top-heavy on MobileNet. DEAP displayed competitive but slightly lower median accuracies in several cases, accompanied by relatively stable distributions. Hyperopt demonstrated balanced behavior, typically producing performance distributions similar to those of Optuna and Grid Search. Collectively, the visual and statistical evidence indicates that although optimization strategy exerts a measurable influence on performance, the resulting accuracy differences are comparatively small relative to the substantial differences in computational cost. Therefore, for the MNIST dataset, the choice of optimization technique is better guided by efficiency considerations than by absolute performance gains. Optuna appears to offer the most favorable balance between accuracy and runtime, whereas DEAP provides the greatest computational efficiency, particularly for deeper architectures. Overall, these findings suggest that on relatively simple image-classification tasks such as MNIST, most modern hyperparameter optimization methods are capable of identifying near-optimal solutions, making runtime efficiency the primary distinguishing factor among competing approaches.

Figure 6. Violin plots for the CNN models trained using multiple hyperparameter optimization techniques on the MNIST dataset.

3.2. Experiment B Results

The SkinCancer dataset, unlike MNIST, exhibited substantially greater variability in both training dynamics and classification performance, reflecting the increased complexity and heterogeneity of dermoscopic images. Quantitative results presented in Table 5 show that classification performance varied considerably across architectures and optimization methods. Overall, the highest test accuracy was achieved by ResNet18 using Hyperopt (0.8150), closely followed by DEAP (0.8125) and Optuna (0.8075). Among the remaining architectures, GoogLeNet demonstrated strong and consistent performance, achieving test accuracies between 0.7910 and 0.8120, with Grid Search producing the best result. AlexNet achieved moderate performance, with test accuracies ranging from 0.7685 to 0.7954, while MobileNet exhibited mixed outcomes, attaining accuracies between 0.7395 and 0.7955 depending on the optimization strategy. LeNet produced the weakest classification results overall, although Hyperopt improved its test accuracy to 0.7650, noticeably outperforming the other optimization approaches. Crucially, the narrow 95% confidence intervals across all evaluations, with margins of error consistently restricted to $\pm 0.0024$ or less, demonstrate high precision and stability in the reported test accuracies. Across precision, recall, F1-score, specificity, AUC-ROC, and AUC-PR metrics, ResNet18 and GoogLeNet consistently delivered the most balanced performance, whereas LeNet generally exhibited weaker generalization despite maintaining relatively high recall values. Table 6 summarizes the corresponding optimized hyperparameter configurations, including learning rate, dropout rate, and weight decay selected by each optimization method.

Table 5. Summary of Quantitative Results for the SkinCancer Dataset. Test accuracies are reported alongside their 95% confidence intervals [95% CI], and algorithmic search time is reported in seconds (s).

Method	Train Accuracy	Test Accuracy	Val Accuracy	Precision	Recall	F1 Score	Specificity	AUC ROC	AUC PR	Time Taken
LeNet
Grid Search	0.8238 ± 0.0026	0.7060 ± 0.0024 [0.7049, 0.7071]	0.7178 ± 0.0025	0.6414 ± 0.0025	0.9340 ± 0.0022	0.7605 ± 0.0024	0.4780 ± 0.0027	0.8210 ± 0.0021	0.8135 ± 0.0022	12,075.7
Optuna	0.7732 ± 0.0015	0.7165 ± 0.0013 [0.7159, 0.7171]	0.7222 ± 0.0014	0.6467 ± 0.0014	0.9540 ± 0.0010	0.7709 ± 0.0013	0.4790 ± 0.0016	0.8335 ± 0.0011	0.8260 ± 0.0012	9168.1
Hyperopt	0.8219 ± 0.0011	0.7650 ± 0.0009 [0.7646, 0.7654]	0.7707 ± 0.0010	0.6939 ± 0.0010	0.9480 ± 0.0008	0.8013 ± 0.0009	0.5820 ± 0.0012	0.8735 ± 0.0008	0.8668 ± 0.0009	9181.4
DEAP	0.8705 ± 0.0013	0.7505 ± 0.0011 [0.7500, 0.7510]	0.7625 ± 0.0012	0.6813 ± 0.0012	0.9410 ± 0.0009	0.7904 ± 0.0011	0.5600 ± 0.0014	0.8610 ± 0.0010	0.8547 ± 0.0011	7929.7
AlexNet
Grid Search	0.8958 ± 0.0024	0.7930 ± 0.0022 [0.7920, 0.7940]	0.8033 ± 0.0023	0.7278 ± 0.0023	0.9360 ± 0.0021	0.8189 ± 0.0022	0.6500 ± 0.0024	0.8805 ± 0.0020	0.8676 ± 0.0021	13,554.6
Optuna	0.8581 ± 0.0012	0.7954 ± 0.0010 [0.7949, 0.7959]	0.8017 ± 0.0011	0.7267 ± 0.0012	0.9470 ± 0.0009	0.8223 ± 0.0010	0.6438 ± 0.0013	0.8832 ± 0.0008	0.8708 ± 0.0009	10,728.7
Hyperopt	0.8804 ± 0.0015	0.7942 ± 0.0013 [0.7936, 0.7948]	0.8028 ± 0.0014	0.7271 ± 0.0014	0.9420 ± 0.0011	0.8207 ± 0.0012	0.6464 ± 0.0015	0.8819 ± 0.0010	0.8691 ± 0.0011	10,792.7
DEAP	0.8613 ± 0.0017	0.7685 ± 0.0015 [0.7678, 0.7692]	0.7778 ± 0.0016	0.7036 ± 0.0016	0.9280 ± 0.0013	0.8003 ± 0.0014	0.6090 ± 0.0018	0.8514 ± 0.0012	0.8352 ± 0.0013	9236.2
GoogLeNet
Grid Search	0.9469 ± 0.0025	0.8120 ± 0.0023 [0.8109, 0.8131]	0.8255 ± 0.0024	0.7468 ± 0.0024	0.9440 ± 0.0021	0.8339 ± 0.0023	0.6800 ± 0.0025	0.9092 ± 0.0018	0.9048 ± 0.0019	15,587.3
Optuna	0.8249 ± 0.0012	0.8005 ± 0.0010 [0.8000, 0.8010]	0.8029 ± 0.0011	0.7310 ± 0.0012	0.9510 ± 0.0009	0.8266 ± 0.0011	0.6500 ± 0.0013	0.9008 ± 0.0009	0.8957 ± 0.0010	12,224.4
Hyperopt	0.9075 ± 0.0015	0.8035 ± 0.0013 [0.8029, 0.8041]	0.8139 ± 0.0014	0.7358 ± 0.0014	0.9470 ± 0.0011	0.8282 ± 0.0013	0.6600 ± 0.0015	0.9026 ± 0.0011	0.8979 ± 0.0012	12,133.1
DEAP	0.8132 ± 0.0017	0.7910 ± 0.0015 [0.7903, 0.7917]	0.7932 ± 0.0016	0.7270 ± 0.0016	0.9320 ± 0.0013	0.8168 ± 0.0015	0.6500 ± 0.0017	0.8895 ± 0.0013	0.8840 ± 0.0014	10,170.4
ResNet18
Grid Search	0.8317 ± 0.0024	0.8050 ± 0.0023 [0.8039, 0.8061]	0.8077 ± 0.0024	0.7402 ± 0.0024	0.9400 ± 0.0021	0.8282 ± 0.0023	0.6700 ± 0.0025	0.9030 ± 0.0018	0.8975 ± 0.0019	13,057.2
Optuna	0.8015 ± 0.0015	0.8075 ± 0.0013 [0.8069, 0.8081]	0.8069 ± 0.0014	0.7393 ± 0.0014	0.9500 ± 0.0011	0.8315 ± 0.0013	0.6650 ± 0.0015	0.9050 ± 0.0010	0.8997 ± 0.0011	10,029.3
Hyperopt	0.8410 ± 0.0011	0.8150 ± 0.0009 [0.8146, 0.8154]	0.8176 ± 0.0010	0.7480 ± 0.0010	0.9500 ± 0.0008	0.8370 ± 0.0009	0.6800 ± 0.0011	0.9150 ± 0.0007	0.9098 ± 0.0008	10,039
DEAP	0.8052 ± 0.0013	0.8125 ± 0.0011 [0.8120, 0.8130]	0.8118 ± 0.0012	0.7490 ± 0.0012	0.9400 ± 0.0009	0.8337 ± 0.0011	0.6850 ± 0.0013	0.9125 ± 0.0009	0.9073 ± 0.0010	8676.2
MobileNet
Grid Search	0.7738 ± 0.0025	0.7910 ± 0.0023 [0.7899, 0.7921]	0.7893 ± 0.0024	0.7235 ± 0.0024	0.9420 ± 0.0021	0.8184 ± 0.0023	0.6400 ± 0.0025	0.9035 ± 0.0019	0.8978 ± 0.0020	13,861
Optuna	0.7935 ± 0.0012	0.7955 ± 0.0010 [0.7950, 0.7960]	0.7953 ± 0.0011	0.7254 ± 0.0012	0.9510 ± 0.0009	0.8230 ± 0.0011	0.6400 ± 0.0013	0.9060 ± 0.0009	0.9002 ± 0.0010	10,795.4
Hyperopt	0.9420 ± 0.0016	0.7395 ± 0.0014 [0.7388, 0.7402]	0.7597 ± 0.0015	0.6737 ± 0.0015	0.9290 ± 0.0012	0.7810 ± 0.0014	0.5500 ± 0.0017	0.8505 ± 0.0012	0.8425 ± 0.0013	10,884
DEAP	0.9272 ± 0.0014	0.7490 ± 0.0012 [0.7484, 0.7496]	0.7668 ± 0.0013	0.6807 ± 0.0013	0.9380 ± 0.0010	0.7889 ± 0.0012	0.5600 ± 0.0015	0.8620 ± 0.0010	0.8549 ± 0.0011	9439.2

Table 6. Summary of Optimized Hyperparameters for SkinCancer Dataset.

Method	Learning Rate	Dropout Rate	Weight Decay
LeNet
Grid Search	$1.00\times {10}^{-2}$	0.3	—
Optuna	$1.00\times {10}^{-2}$	0.3	$1.00\times {10}^{-2}$
Hyperopt	$1.00\times {10}^{-2}$	0.4	$1.00\times {10}^{-2}$
DEAP	$1.00\times {10}^{-3}$	0.3	$1.00\times {10}^{-4}$
AlexNet
Grid Search	$1.00\times {10}^{-4}$	0.4	$1.00\times {10}^{-4}$
Optuna	$1.00\times {10}^{-3}$	0.4	$1.00\times {10}^{-2}$
Hyperopt	$1.00\times {10}^{-4}$	0.2	$1.00\times {10}^{-4}$
DEAP	$1.00\times {10}^{-4}$	0.3	$1.00\times {10}^{-2}$
GoogLeNet
Grid Search	$1.00\times {10}^{-4}$	0.4	$1.00\times {10}^{-2}$
Optuna	$1.00\times {10}^{-4}$	0.3	$1.00\times {10}^{-2}$
Hyperopt	$1.00\times {10}^{-4}$	0.2	$1.00\times {10}^{-4}$
DEAP	$1.00\times {10}^{-2}$	0.1	$1.00\times {10}^{-4}$
ResNet18
Grid Search	$1.00\times {10}^{-2}$	0.2	$1.00\times {10}^{-4}$
Optuna	$1.00\times {10}^{-2}$	0.3	$1.00\times {10}^{-2}$
Hyperopt	$1.00\times {10}^{-3}$	0.3	$1.00\times {10}^{-2}$
DEAP	$1.00\times {10}^{-2}$	0.1	$1.00\times {10}^{-2}$
MobileNet
Grid Search	$1.00\times {10}^{-1}$	0.4	$1.00\times {10}^{-4}$
Optuna	$1.00\times {10}^{-1}$	0.2	$1.00\times {10}^{-4}$
Hyperopt	$1.00\times {10}^{-3}$	0.3	$1.00\times {10}^{-2}$
DEAP	$1.00\times {10}^{-4}$	0.3	$1.00\times {10}^{-2}$

Note: The em-dash (—) denotes a weight decay value of 0 (no regularization applied).

The choice of hyperparameter optimization strategy had a pronounced effect on model performance. For several architectures, adaptive search techniques outperformed exhaustive approaches. Hyperopt delivered the highest test accuracy for both LeNet (0.7650) and ResNet18 (0.8150), while Optuna achieved the best result for AlexNet (0.7954) and MobileNet (0.7955). In contrast, Grid Search produced the strongest performance for GoogLeNet (0.8120) but required substantially longer training times. Bayesian optimization approaches, particularly Optuna and Hyperopt, frequently achieved competitive or superior performance while maintaining lower computational costs than Grid Search. For example, ResNet18 trained with Hyperopt required approximately 10,039 s compared with 13,057 s for Grid Search, while simultaneously improving test accuracy from 0.8050 to 0.8150. Similarly, LeNet optimized by Hyperopt achieved notable gains in accuracy and AUC metrics relative to Grid Search and Optuna. DEAP also demonstrated strong computational efficiency, consistently requiring the shortest training time among the methods for most architectures while maintaining competitive predictive performance. For example, DEAP reduced training time for ResNet18 to 8676 s while achieving a test accuracy of 0.8125, only marginally below the best-performing Hyperopt configuration.

The training accuracy and loss curves shown in Figure 7 and Figure 8 further support these observations. Simpler architectures such as LeNet frequently displayed larger gaps between training and validation performance, indicating increased susceptibility to overfitting. AlexNet also exhibited greater performance fluctuations across optimization techniques. In contrast, deeper architectures, particularly ResNet18 and GoogLeNet, demonstrated more stable convergence behavior and superior generalization across validation and test sets. The performance advantages of these deeper networks are also reflected in their higher AUC-ROC values, which exceeded 0.90 for the best-performing configurations. Specifically, ResNet18 achieved an AUC-ROC of 0.9150 under Hyperopt, while GoogLeNet reached 0.9092 under Grid Search. These results indicate strong discriminative capability despite the challenges associated with the SkinCancer dataset. Figure 9 similarly illustrates clear separation among optimization methods, with AUC-ROC values ranging from approximately 0.82 for LeNet to over 0.91 for ResNet18.

Figure 7. Training and validation accuracy curves for the CNN models trained using multiple hyperparameter optimization techniques on the SkinCancer dataset.

Figure 8. Training and validation loss curves for the CNN models trained using multiple hyperparameter optimization techniques on the SkinCancer dataset.

Figure 9. Cumulative ROC curves for the CNN models trained using multiple hyperparameter optimization techniques on the SkinCancer dataset.

The Holm Corrected Wilcoxon Test and Friedman Test results presented in Table 7 and Table 8, respectively, provide further evidence of the influence of optimization strategy on classification performance. The Friedman test revealed statistically significant differences among hyperparameter optimization methods for all evaluated architectures. The strongest evidence was observed for AlexNet ($p=0.0009$) and GoogLeNet ($p=0.0006$), while significant differences were also identified for LeNet ($p=0.0054$), ResNet18 ($p=0.0025$), and MobileNet ($p=0.0361$). These findings indicate that optimization method selection significantly affects model performance across the entire set of CNN architectures. Pairwise post hoc comparisons revealed highly architecture-dependent statistical differences between optimization approaches. For LeNet, DEAP significantly differed from Grid Search and Optuna, yielding substantial rank-biserial effect sizes $(r_{rb}=0.6529$ and $0$.$5791$, respectively), but matched Hyperopt, while no other methods diverged. On AlexNet and ResNet18, DEAP consistently demonstrated significant performance deviations from all three alternative frameworks, marked by prominent effect sizes peaking at $r_{rb}=0.6817$. Conversely, GoogLeNet showed unique variations where Optuna differed significantly from both Grid Search and DEAP, though the most pronounced effect size in this architecture occurred between DEAP and Grid Search $(p=0.0012,$ $r_{rb}=0.7242$). Finally, MobileNet exhibited the least separation, with a significant difference emerging uniquely between DEAP and Hyperopt $(p=0.0058$, $r_{rb}=0.6169)$, demonstrating that optimization impacts vary greatly by network architecture.

Table 7. Summary of Holm Corrected Wilcoxon Test Results and Rank-Biserial Effect Sizes for the SkinCancer Dataset. Active cells report post hoc pairwise p-values along with their corresponding non-parametric effect sizes in parentheses.

	Grid Search	Hyperopt	DEAP
LeNet
Hyperopt	0.1270
DEAP	0.0035 ** $(r_{rb}=0.6529)$	0.1270
Optuna	0.1270	0.1270	0.0096 ** $(r_{rb}=0.5791)$
AlexNet
Hyperopt	0.0923
DEAP	0.0023 ** $(r_{rb}=0.6817)$	0.0092 ** $(r_{rb}=0.5824)$
Optuna	0.1231	0.1231	0.0023 ** $(r_{rb}=0.6817)$
GoogLeNet
Hyperopt	0.0708
DEAP	0.0012 ** $(r_{rb}=0.7242)$	0.0058 ** $(r_{rb}=0.6169)$
Optuna	0.0237 * $(r_{rb}=0.5058)$	0.0708	0.0108 * $(r_{rb}=0.5700)$
ResNet18
Hyperopt	0.0952
DEAP	0.0023 ** $(r_{rb}=0.6817)$	0.0183 * $(r_{rb}=0.5276)$
Optuna	0.0985	0.0952	0.0185 * $(r_{rb}=0.5267)$
MobileNet
Hyperopt	0.1275
DEAP	0.1046	0.0058 ** $(r_{rb}=0.6169)$
Optuna	0.1275	0.1275	0.0635

* p < 0.05, ** p < 0.01. The gray cells indicate redundant comparisons.

Table 8. Friedman Test Results for the SkinCancer Dataset.

Model	p-Value
LeNet	0.0054 **
AlexNet	0.0009 **
GoogLeNet	0.0006 **
ResNet18	0.0025 **
MobileNet	0.0361 *

* p < 0.05, ** p < 0.01.

The violin plots in Figure 10 summarize performance distributions on the SkinCancer dataset. High overall volatility is evident across almost all architectures and optimization approaches. On LeNet, Optuna stands out with a highly concentrated, low-variability distribution. Conversely, on GoogLeNet and ResNet18, adaptive methods display high dispersion with long lower tails. MobileNet reveals a reversal, where Grid Search achieves a concentrated, top-heavy profile. On this same model, DEAP and Hyperopt show heavily downward-skewed performance. Visually, no single optimizer consistently guarantees tight stability across all network types. Overall, the combination of quantitative metrics, statistical significance testing, and visual analysis suggests that adaptive optimization approaches, particularly Hyperopt, Optuna, and DEAP, offer a more effective balance between predictive accuracy, robustness, and computational efficiency than traditional exhaustive search methods for the SkinCancer dataset.

Figure 10. Violin plots for the CNN models trained using multiple hyperparameter optimization techniques on the SkinCancer dataset.

3.3. Discussion on Results

The experimental results demonstrate that the effectiveness of hyperparameter optimization is strongly influenced by dataset complexity, affecting both predictive performance and computational efficiency. Across both experiments, all optimization methods were capable of identifying competitive hyperparameter configurations; however, their relative advantages varied substantially between the simple MNIST benchmark and the more challenging SkinCancer dataset. These findings emphasize that the selection of a hyperparameter optimization strategy should be informed not only by predictive performance but also by the complexity of the learning task and available computational resources.

For the MNIST dataset, all CNN architectures achieved high classification performance regardless of the optimization technique employed. Test accuracies ranged from 95.45% to 97.61%, with ResNet18 optimized using Grid Search producing the highest accuracy (97.61%), closely followed by GoogLeNet (97.47%). Similarly, precision, recall, F1-score, specificity, AUC-ROC, and AUC-PR values remained consistently high across all models, with AUC values generally exceeding 0.99. These results indicate that MNIST represents a relatively straightforward classification task in which the underlying class distributions are well separated and can be effectively learned by a wide range of network architectures. Consequently, the influence of hyperparameter optimization on final predictive performance was limited, with differences in test accuracy typically below 2% across optimization methods.

Although performance differences were relatively small, noticeable differences emerged in computational efficiency. Grid Search consistently required the longest optimization time, particularly for deeper architectures such as GoogLeNet and ResNet18, where runtimes exceeded 15,000 s. In contrast, DEAP achieved the shortest execution times across all models, reducing optimization time by approximately 25–35% relative to Grid Search while maintaining comparable predictive performance. Optuna and Hyperopt provided an effective compromise between runtime and classification accuracy, often requiring substantially less computation than Grid Search while achieving similar or superior results. The training and validation curves further confirmed stable convergence behavior, with only minor gaps between training and validation performance, suggesting limited overfitting. Statistical testing revealed significant differences among optimization methods for all architectures according to the Friedman test; however, the practical magnitude of these differences remained small. Therefore, for a relatively simple benchmark such as MNIST, computational efficiency appears to be a more important consideration than marginal improvements in predictive performance.

A markedly different pattern emerged for the SkinCancer dataset. Compared with MNIST, classification performance was substantially lower and more variable across architectures and optimization methods, reflecting the increased complexity, heterogeneity, and visual variability of dermoscopic images. In this setting, deeper architectures consistently outperformed shallower models. ResNet18 achieved the highest overall test accuracy (81.50%) when optimized using Hyperopt, followed closely by DEAP (81.25%) and Optuna (80.75%). GoogLeNet also demonstrated strong performance, achieving test accuracies between 79.10% and 81.20%. In contrast, LeNet and MobileNet generally produced lower classification accuracy and weaker overall generalization performance. These findings suggest that deeper architectures were better able to capture the complex hierarchical features required for accurate skin lesion classification.

Unlike MNIST, the choice of hyperparameter optimization method exerted a more substantial influence on predictive performance for the SkinCancer dataset. Hyperopt achieved the highest accuracy for both LeNet and ResNet18, while Optuna produced the strongest results for AlexNet and MobileNet. Grid Search yielded the best performance for GoogLeNet but required considerably longer computational time. Overall, adaptive optimization approaches, particularly Hyperopt and Optuna, frequently matched or exceeded the performance of Grid Search while substantially reducing optimization costs. For example, Hyperopt improved ResNet18 test accuracy from 80.50% under Grid Search to 81.50% while simultaneously reducing runtime by approximately 23%. These findings suggest that Bayesian optimization techniques are particularly effective when navigating the more complex hyperparameter landscapes associated with high-dimensional medical imaging problems.

DEAP exhibited a different but equally important advantage. Although it did not consistently achieve the highest classification accuracy, it was the most computationally efficient optimization strategy across nearly all architectures. In several cases, DEAP produced predictive performance comparable to the best-performing methods while requiring substantially less optimization time. For example, DEAP optimized ResNet18 in 8676 s while achieving a test accuracy of 81.25%, only marginally below the best Hyperopt configuration. Similar trends were observed for GoogLeNet, AlexNet, and MobileNet. These results demonstrate that evolutionary optimization approaches can provide attractive trade-offs between predictive performance and computational cost, particularly when rapid model development or limited computational resources are important considerations.

The variations in total optimization runtime across methods—despite an identical budget of 64 trials—stem from the specific hyperparameter configurations selected for evaluation. Suboptimal parameter combinations can lead to significantly longer training times per trial, which impacts the aggregate runtime of exhaustive methods like Grid Search.

The statistical analyses further support these observations. For both datasets, Friedman tests identified significant differences among optimization strategies across all evaluated architectures. However, the interpretation of these results differs between datasets. On MNIST, statistically significant differences were associated with very small absolute performance variations, suggesting limited practical impact despite consistent ranking differences among methods. On the SkinCancer dataset, significant differences were accompanied by more substantial performance variations, indicating that optimization strategy plays a more meaningful role when learning complex image representations. Pairwise Holm Corrected Wilcoxon test revealed that DEAP often differed significantly from the other methods, while Grid Search, Optuna, and Hyperopt were frequently statistically comparable. These findings suggest that although all methods can produce effective models, their behavior and relative advantages vary according to the characteristics of the underlying dataset.

The results indicate that dataset complexity is a key factor governing the value of hyperparameter optimization. For relatively simple image-classification tasks such as MNIST, most modern optimization methods are capable of identifying near-optimal solutions, making computational efficiency the primary criterion for method selection. In contrast, for more complex medical imaging tasks, adaptive optimization strategies become increasingly important, as they can improve classification performance while maintaining reasonable computational requirements. Among the evaluated approaches, Optuna and Hyperopt provided the most favorable overall balance between predictive accuracy and computational efficiency, whereas DEAP offered the greatest reductions in optimization time. Consequently, the choice of optimization strategy should be guided by the desired balance between performance and computational cost rather than by predictive accuracy alone.

4. Study Limitations

While this study provides valuable insights into the comparative performance of hyperparameter optimization methods, several limitations should be acknowledged. First, the experiments were restricted to two datasets, which may not fully capture the diversity and complexity of real-world image classification problems. Although the MNIST and SkinCancer datasets provide useful representations of low-complexity and medically relevant classification tasks, respectively, future investigations should include a broader range of datasets and extend the evaluation to more challenging domains such as multi-class and three-dimensional (3D) medical imaging. Second, the analysis primarily focused on classification performance metrics and training time, without a detailed assessment of additional computational factors such as energy consumption, memory utilization, hardware efficiency, batch-size sensitivity, or optimization scalability. Third, the study considered only four optimization methods and did not investigate hybrid or ensemble optimization frameworks, which may potentially combine the strengths of Bayesian and evolutionary search strategies to achieve further improvements in performance and robustness.

An additional limitation relates to the metadata constraints of the SkinCancer dataset. Because the dataset was obtained from a public repository and did not provide patient-level or lesion-level identifiers, the implementation of grouped data-partitioning strategies was not possible. Consequently, the possibility of data leakage arising from multiple lesions belonging to the same patient being distributed across training, validation, and testing sets cannot be completely excluded. To mitigate this limitation, an automated visual audit was conducted using the imagededup and OpenCV libraries to identify potential redundancies within the dataset. This analysis confirmed the absence of exact and near-duplicate images, ensuring that the benchmark was not affected by explicit image replication artifacts. Furthermore, as the primary objective of this work was to provide a computational benchmark for evaluating hyperparameter optimization strategies rather than a clinical diagnostic study, advanced medical imaging validation procedures, including external clinical validation, model calibration, diagnostic error analysis, and acquisition-source harmonization, were beyond the scope of the current investigation. Future studies should utilize datasets containing comprehensive patient metadata to enable stricter partition isolation and more rigorous clinical evaluation protocols.

5. Conclusions

This study systematically compared four hyperparameter optimization strategies (i.e., Grid Search, Optuna, Hyperopt, and DEAP) across five CNN architectures spanning multiple generations of deep learning. By conducting experiments on both the MNIST benchmark and a custom SkinCancer dataset, we assessed the methods under contrasting data regimes: an idealized, abundant dataset and a realistic, data-constrained medical imaging scenario. The results consistently highlight the dataset-dependent impact and sensitivity of optimization methods. On MNIST dataset, all architectures achieved near-saturated accuracy. Optuna and Hyperopt matched or exceeded Grid Search’s predictive performance while significantly reducing runtime. DEAP consistently proved to be the fastest optimization method across all architectures without suffering from increased variance. Conversely, the high-dimensional SkinCancer dataset revealed greater optimization sensitivity. Hyperopt and Optuna offered strong trade-offs between accuracy and time across most architectures, though Grid Search retained a slight accuracy advantage on deeper models like GoogLeNet. DEAP maintained its position as the most computationally efficient method while remaining highly competitive, particularly on ResNet18.

The findings suggest that modern hyperparameter optimization approaches are capable of identifying near-optimal solutions for relatively simple classification tasks, where computational efficiency becomes the primary deciding factor. For more challenging applications such as medical image analysis, adaptive optimization methods offer greater benefits by improving predictive performance while maintaining reasonable computational requirements. Among the evaluated approaches, Optuna and Hyperopt provided the most favorable balance between accuracy and efficiency, whereas DEAP offered the strongest computational advantages.

Future work should extend these analyses to more diverse and large-scale datasets, including 3D medical imaging, while incorporating additional evaluation metrics such as energy efficiency, batch size sensitivity, and optimization runtime. Moreover, exploring hybrid or ensemble optimization frameworks could further enhance robustness and adaptability across varying data complexities and deployment environments.