Optimizing Brain Tumor Classification: Integrating Deep Learning and Machine Learning with Hyperparameter Tuning

Abstract

Brain tumors significantly impact global health and pose serious challenges for accurate diagnosis due to their diverse nature and complex characteristics. Effective diagnosis and classification are essential for selecting the best treatment strategies and forecasting patient outcomes. Currently, histopathological examination of biopsy samples is the standard method for brain tumor identification and classification. However, this method is invasive, time-consuming, and prone to human error. To address these limitations, a fully automated approach is proposed for brain tumor classification. Recent advancements in deep learning, particularly convolutional neural networks (CNNs), have shown promise in improving the accuracy and efficiency of tumor detection from magnetic resonance imaging (MRI) scans. In response, a model was developed that integrates machine learning (ML) and deep learning (DL) techniques. The process began by splitting the data into training, testing, and validation sets. Images were then resized and cropped to enhance model quality and efficiency. Relevant texture features were extracted using a modified Visual Geometry Group (VGG) architecture. These features were fed into various supervised ML models, including support vector machine (SVM), k-nearest neighbors (KNN), logistic regression (LR), stochastic gradient descent (SGD), random forest (RF), and AdaBoost, with GridSearchCV used for hyperparameter tuning. The model’s performance was evaluated using key metrics such as accuracy, precision, recall, F1-score, and specificity. Experimental results demonstrate that the proposed approach offers a robust and automated solution for brain tumor classification, achieving the highest accuracy of 94.02% with VGG19 and 96.30% with VGG16. This model can significantly assist healthcare professionals in early tumor detection and in improving diagnostic accuracy.

1. Introduction

Brain cancer is a life-threatening disease that severely impacts patients’ health and quality of life. Primary detection is essential for improving treatment outcomes and survival rates, yet diagnosing brain tumors in their initial stages remains a significant challenge. The complexity of brain tumors, combined with the intricate structure of the human brain, poses significant challenges for diagnosis. As the brain controls essential functions such as motor activities, vision, emotions, memory, breathing, and responses, the formation of a tumor in this vital organ can severely disrupt these critical processes [1,2].

Brain tumors are categorized into two types: primary tumors, which originate within the brain tissue itself, and metastatic tumors, which develop in other parts of the body and spread to the brain. Diagnosing these tumors presents unique challenges due to the presence of the blood–brain barrier (BBB), a protective shield that prevents standard radioactive markers from detecting abnormal tumor cell activity [3]. Consequently, MRI scans are the preferred diagnostic tool for identifying BBB disruptions.

Brain tumors affect approximately 7 to 11 individuals per 100,000 individuals annually in several age groups. This condition is responsible for about 227,000 deaths globally each year, and around 7.7 million people live with related disabilities [4]. Early diagnosis is crucial for improving survival rates and reducing the risk of long-term disability. Detecting brain tumors in their early stages allows for less invasive treatment and reduced surgical intervention. Typically, radiologists obtain brain images that are then analyzed by specialists to formulate a treatment strategy [5]. However, research reveals discrepancies in manual diagnostic outcomes among experts. Although general brain tumor diagnoses show an agreement rate between 90% and 95%, this percentage decreases to 77% and 58% for more complex tumor types such as mixed gliomas and medulloblastomas [6].

Advancements in digital image processing and medical imaging have paved the way for the increased use of computer-aided diagnosis (CAD) systems. MRI remains the preferred imaging technique due to its non-ionizing nature and its ability to clearly detect blood flow in veins. The availability of extensive brain MRI datasets has facilitated the combination of machine learning (ML) and deep learning (DL) methods for tumor detection. Creating accurate ML and DL models requires training in large datasets to provide reliable predictions and valuable information, which are essential for informed clinical decision-making [7]. This study investigates the application of the DL technique for detecting brain tumors and enhancing diagnostic efficiency.

The primary contribution of this work is the development of a hybrid brain tumor classification framework. A modified VGG architecture was used for more effective feature extraction. Additional dense and dropout layers were added to the VGG model to better capture tumor characteristics and reduce overfitting. Unlike traditional methods, we did not rely solely on CNN’s final layers for classification. Instead, deep features extracted from the modified VGG model were fed into various supervised machine learning classifiers, including SVM, KNN, LR, SGD, RF, and AdaBoost. GridSearchCV-based hyperparameter tuning was applied to optimize each classifier’s performance. This two-stage approach improves accuracy and model robustness compared to standard end-to-end CNN models.

2. Related Work

Given the substantial impact of deep learning models [8] in this field, several relevant studies have been thoroughly reviewed in this section.

2.1. CNN-Based Classification Models

Irmak et al. [9] proposed a deep CNN model for classifying brain MRI scans using a grid search optimizer for hyperparameter tuning. Although multiple models were developed, the one classifying tumors into five categories achieved a relatively lower accuracy (92.66%) compared with others. Similarly, Badža et al. [10] classified glioma, meningioma, and pituitary tumors using a CNN model trained on merged datasets with multiple validation methods, showing promising but dataset-dependent results.

Ayadi et al. [11] also employed a CNN model using multiple public datasets and achieved reliable classification among three tumor types. However, none of these works explored deeper integration with machine learning techniques for improving classification robustness. Other models focused on binary classification.

Irsheidat and Duwairi [12] used grayscale MRIs and data augmentation to train a CNN model that achieved 96.7% validation accuracy but only 88.25% on the test set, showing overfitting and limited generalization due to a small dataset. Cinar and Yildirim [13] fine-tuned ResNet-50 for binary classification but did not address specific tumor types, limiting clinical relevance.

2.2. Segmentation and GAN-Based Approaches

Segmentation-based approaches have also been widely studied. Naser et al. [14] applied a U-Net based CNN for tumor segmentation, but did not extend to classification. Elazab et al. [15] used a GP-GAN with 3D U-Net architecture to model early stage glioma growth. However, the method assumed continuous tumor growth and did not account for treatment effects, such as shrinkage. Hamghalam et al. [16] introduced a multistage attention-GAN for segmenting core, enhancing, and whole tumors. Although this method produced high-contrast images, it incurred a significant computational cost and complexity.

2.3. Hybrid and Ensemble Approaches

Hybrid approaches have shown potential by combining deep learning with classical techniques. Saba et al. [17] combined deep features from VGG19 with handcrafted features like HOG and LBP for glioma classification. Despite improved accuracy, the approach was limited to distinguishing between glioma and healthy tissues. Noreen et al. [18] proposed a fusion of Inception-v3 and DenseNet201 features, classified using a softmax layer. Although effective, their method required considerable computation and did not explore classifier diversity. Aurna et al. [19] proposed an ensemble-based classification framework, but detailed architectural improvements were not discussed.

2.4. Miscellaneous Techniques and Limitations

Some works introduced alternative feature extraction and classification pipelines. Özyurt et al. [20] used a fuzzy c-means-based segmentation with SqueezeNet and extreme learning machines. Siar et al. [21] used ImageNet-based transfer learning for binary tumor detection. However, both approaches lacked granularity in tumor-type classification. Ghassemi [22] utilized a GAN with a pre-trained CNN, but the input resolution was limited to 64 × 64, restricting the method’s real-world applicability.

Although many studies have achieved high accuracy, several limitations persist. A large portion of the existing work focuses on binary classification or limited tumor types, reducing clinical usefulness. Few studies integrate deep CNN-based feature extraction with classical machine learning classifiers to leverage the strengths of both paradigms. Additionally, challenges like interpretability, computational efficiency, model generalization across datasets, and handling multi-class classification remain open. These gaps underscore the need for hybrid, optimized, and scalable frameworks—motivating the approach proposed in this study.

3. Materials and Methods

3.1. Details of Dataset

To evaluate the performance of the implemented model, we collected samples from three different datasets: figshare, SARTAJ, and Br35H. The resulting dataset consists of 7023 human brain MRI images, categorized into four classes: glioma (2131 images), meningioma (1645 images), pituitary tumors (1757 images), and normal (2000 images). Despite variations in image resolutions, all images were resized to a consistent dimension of 224 × 224 × 3 before being input into the model. The dataset is publicly available and can be downloaded at https://www.kaggle.com/datasets/masoudnickparvar/brain-tumor-mri-dataset, accessed on 2 November 2024.

3.2. Methodology

The proposed methodology integrates advanced machine learning (ML) and deep learning (DL) techniques to overcome the limitations of traditional brain tumor diagnosis methods, such as invasiveness, time consumption, and susceptibility to human error. Figure 1 presents the proposed model’s workflow for classifying brain tumors in MRI images.

The methodology is discussed step-by-step as follows:

3.2.1. Data Preprocessing

The initial step involves preparing the dataset for training, validation, and testing in a ratio of 7:2:1, respectively. MRI images are resized to a uniform size and focused on the region of interest (ROI) while discarding irrelevant areas. Proper preprocessing ensures consistency in input data, reduces noise, and improves the efficiency of the model by providing high-quality, standardized input. This step is crucial for minimizing computational overhead and avoiding bias in training.

3.2.2. Feature Extraction by Modified VGG Architecture

The base VGG model is used as a feature extractor with frozen layers to retain pre-trained knowledge while reducing computational cost. A Flatten layer is introduced to convert the extracted feature maps into a one-dimensional vector, followed by a Dropout (0.5) layer to prevent overfitting. A fully connected Dense (128 units) layer is added to learn complex patterns, followed by another Dropout (0.5) layer to further enhance generalization. Finally, a Dense output layer with softmax activation is used for classification. This architecture leverages deep feature extraction from VGG while utilizing additional layers to refine the classification performance.

Table 1. Training parameters of the proposed model.

Parameter	Value
Loss Function	Categorical Crossentropy
Optimizer	RMSprop
Batch Size (Train)	32
Total Training Samples	4925
Steps per Epoch	154
Epochs	100
Validation Samples	1406
Validation Batch Size	16
Learning rate	1 $\times {10}^{-4}$

provides the training parameters of the proposed model.

Texture features are extracted from MRI scans using a modified VGG architecture. This design has been refined to capture complex patterns unique to the classification of brain tumors. In order to differentiate between various kinds of brain tumors, the model may concentrate on the most pertinent features of the tumors, such as texture, shape, and intensity fluctuations, thanks to feature extraction. The changes made to VGG improve its capacity to adjust to jobs involving medical imaging.

3.2.3. Classification of Brain Tumor Types

The extracted features are input into various supervised ML classifiers, including SVM, KNN, LR, SGD, RF, and AdaBoost. Each classifier processes the features and assigns a label to the tumor type. GridSearchCV is used for hyperparameter tuning to ensure an optimal performance. The use of multiple classifiers allows for a comprehensive evaluation of the extracted features. GridSearchCV ensures that the model parameters are optimized, improving the classification process’s accuracy, efficiency, and robustness. This step is essential for identifying the best-performing model. GridSearchCV is a hyperparameter tuning technique provided by the scikit-learn library in Python-3.7. It is used to systematically search for the best combination of hyperparameters for a machine learning model by performing an exhaustive search over a specified parameter grid. It evaluates different parameter combinations using cross-validation to select the set of hyperparameters that results in the best performance.

To enhance classification performance, we employed GridSearchCV for systematic hyperparameter tuning of each supervised machine learning model. This technique evaluates multiple combinations of hyperparameters using 10-fold cross-validation and selects the combination that yields the best validation performance. Below are the parameter grids used for each classifier:

• Stochastic Gradient Descent (SGD): Parameters included various loss functions (hinge, log, squared_hinge, etc.), regularization penalties (l2, l1, elasticnet), and alpha values ranging from 0.0001 to 0.1.
• Support Vector Machine (SVM): Explored different kernels (linear, rbf, sigmoid, poly), regularization strengths C, kernel coefficients gamma, and decision function strategies (ovo, ovr).
• k-Nearest Neighbors (KNN): Parameter grid included number of neighbors, weighting schemes (uniform, distance), distance metrics (p), and algorithm selection (auto, ball_tree, etc.).
• Logistic Regression (LR): Tuned over solvers (lbfgs, liblinear, etc.), tolerance levels, and random states to optimize convergence and stability.
• Random Forest (RF): Tuned parameters included tree depth, number of estimators, number of features considered at each split, and split criteria.
• AdaBoost Classifier: Used various base estimators (shallow decision trees), learning rates, and numbers of estimators. The SAMME algorithm was used for multi-class boosting.

These parameter grids were selected based on prior research and experimentation to ensure thorough exploration of the model capabilities.

Table 2. Summary of classifiers and GridSearchCV hyperparameters.

Classifier	Key Parameters Tuned	Parameter Values/Ranges
SGD	loss, alpha, penalty	[‘hinge’, ‘log’, ‘squared_hinge’, ‘modified_huber’, ‘perceptron’], [0.0001–0.1], [‘l2’, ‘l1’, ‘elasticnet’, ‘none’]
SVM	C, gamma, kernel, probability, decision_function_shape	[1–1000], [1, 0.1, 0.001, 0.0001, ‘scale’, ‘auto’], [‘linear’, ‘rbf’, ‘sigmoid’, ‘poly’], [‘ovo’, ‘ovr’]
KNN	n_neighbors, weights, algorithm, p	[3–11], [‘uniform’, ‘distance’], [‘auto’, ‘ball_tree’, ‘kd_tree’, ‘brute’], [1–3]
Logistic Regression	tol, solver, random_state	[0.0001–0.1], [‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’], [None, 0, 1, 2, 4, 5, 10]
Random Forest	max_depth, max_features, min_samples_leaf, min_samples_split, n_estimators	[80–110], [2, 3], [3–5], [8–12], [50–1000]
AdaBoost	base_estimator, n_estimators, learning_rate, algorithm, random_state	DecisionTreeClassifier (max_depth=1–4), [50, 100, 200], [0.1–1.0], [‘SAMME’], [None, 5, 10]

presents a detailed overview of the classifiers used in this study along with the specific hyperparameter grids defined for tuning via GridSearchCV. Each classifier was fine-tuned by exhaustively searching over a predefined set of hyperparameters using 10-fold cross-validation. This tuning process helped in identifying the optimal configuration for each model, thereby enhancing classification accuracy and model stability. The table outlines the key hyperparameters considered for each classifier, along with the corresponding ranges or values explored during the search. This systematic optimization strategy contributed to the superior performance of our proposed framework.

3.2.4. Peformance Evaluation

The performance of the model is assessed using key metrics such as accuracy, precision, recall, F1-score, and specificity. Evaluation metrics provide quantitative measures of the model’s performance, offering insights into its strengths and weaknesses. These metrics validate the model’s ability to correctly identify true positives (tumors) while minimizing false positives and false negatives, ensuring reliability in real-world applications.

4. Results

The performance of the proposed brain tumor classification model was evaluated using multiple deep learning and machine learning techniques. This section presents the experimental results obtained from different model configurations, highlighting their effectiveness in classifying brain tumors from MRI images. The confusion matrices of various models provide insights into their classification effectiveness, while comparative analysis with state-of-the-art methods further establishes the robustness of the proposed approach.

Figure 2 presents the confusion matrices for VGG19, VGG16, and their SVM-based variations in brain tumor classification. The CNN models (Figure 2a,c) perform well but show misclassifications, especially between glioma and meningioma. The SVM-based models (Figure 2b,d) improve accuracy, significantly reducing misclassification, particularly for glioma and meningioma, while achieving near-perfect classification for normal and pituitary tumors. Overall, VGG16_SVM delivers the best performance, highlighting the effectiveness of combining CNN-based feature extraction with SVM classifiers for enhanced brain tumor detection.

Table 3. Overall performance metrics of VGG19 and its combination with ML classifiers in brain tumor classification.

Model	Precision (%)	Sensitivity (or) Recall (%)	F1 Score (%)	Specificity (%)	Accuracy (%)
VGG19	91.65	91.28	91.46	97.25	91.74
VGG19_AB	91.20	91.03	91.11	97.15	91.45
VGG19_KNN	92.52	92.11	92.31	97.53	92.59
VGG19_LR	91.09	91.02	91.05	97.15	91.45
VGG19_RF	90.66	90.30	90.48	96.91	90.74
VGG19_SGD	91.84	91.64	91.74	97.34	92.02
VGG19_SVM	93.77	93.69	93.73	98.01	94.02

presents the classification performance of VGG19 as a feature extractor combined with various machine learning classifiers. Among all combinations, VGG19_SVM achieves the highest accuracy of 94.02%, demonstrating its superior ability to classify brain tumors effectively. Additionally, VGG19_SVM exhibits the highest precision (93.77%), recall (93.69%), F1-score (93.73%), and specificity (98.01%), making it the most robust classifier for this architecture.

Table 4. Overall performance metrics of VGG16 and its combination with ML classifiers in brain tumor classification.

Model	Precision (%)	Sensitivity (or) Recall (%)	F1 Score (%)	Specificity (%)	Accuracy (%)
VGG16	91.18	90.2	90.69	96.87	90.6
VGG16_AB	94.05	93.84	93.94	98.01	94.02
VGG16_KNN	95.04	95.06	95.05	98.39	95.16
VGG16_LR	92.91	92.78	92.85	97.67	93.02
VGG16_RF	93.23	93.06	93.15	97.77	93.3
VGG16_SGD	93.34	93.41	93.38	97.86	93.59
VGG16_SVM	96.16	96.15	96.16	98.77	96.30

presents the performance of VGG16 in combination with the same machine learning classifiers. Similar to VGG19, VGG16 extracts features from MRI images, which are then classified using different ML models. The results indicate that VGG16_SVM achieves the highest accuracy of 96.30%, outperforming all other configurations. It also exhibits the highest precision (96.16%), recall (96.15%), F1-score (96.16%), and specificity (98.77%), making it the most effective classifier for this architecture.

5. Conclusions

The proposed method presents an automated and robust approach for brain tumor classification by integrating CNN-based feature extraction with advanced machine learning classifiers. Using a modified VGG architecture, the model achieved high accuracy, with VGG19_SVM reaching 94.02% and VGG16_SVM achieving 96.30%, outperforming many existing methods. Performance evaluation using accuracy, precision, recall, F1-score, and specificity confirms the model’s effectiveness in handling diverse tumor datasets. By minimizing reliance on invasive diagnostic techniques and improving classification accuracy, this approach can assist healthcare professionals in early detection and treatment planning, ultimately enhancing patient outcomes. Future work may focus on optimizing the model further and exploring real-time clinical deployment.

For future work, we aim to further optimize the model and explore challenges related to real-time clinical deployment, including reducing computational complexity and latency. Additionally, we plan to investigate model interpretability to enhance trust in automated predictions and extend the framework through domain adaptation techniques to improve generalization across different MRI scanners, imaging protocols, and patient populations.