Multiclass Classification of Brain Tumors with Various Deep Learning Models ^†

Abstract

Brain cancer is one of the most dangerous cancer types in the world, and thousands of people are suffering from malignant brain tumors. Depending on the level of cancer, early diagnosis can be a lifesaver. However, thousands of scans must be studied in order to classify tumor types with high accuracy. Deep learning models can handle that amount of data, and they can present results with high accuracy. It is already known that deep learning models can give different results depending on the dataset. In this paper, the effectiveness of some of the deep learning models on two different publicly available MRI (Magnetic Resonance Imaging) brain tumor datasets is examined. The reason for choosing this topic is that we are trying to find the best solution to classify tumors in the datasets. Different deep learning models are used separately on preprocessed datasets with the Contrast Limited Adaptive Histogram Equalization (CLAHE) preprocessing variable to extract features from images and classify them. Datasets are shuffled randomly for 80% training, 10% validation, and 10% testing. For fine-tuning, models are modified so that the output channel of the classifier is equal to the number of classes in the datasets. The results show that pre-trained and fine-tuned ResNet, RegNet, and Vision Transformer (ViT) deep learning models can achieve accuracies higher than 90% and that they can be used as classifiers when diagnosis is required.

1. Introduction

The brain is the most complex organ in vertebrates, and it is located in the center of the nervous system [1]. Tumor types in the brain can be mainly classified as benign and malignant tumors. Additionally, brain tumors can be classified as primary and secondary. Tumors that start to grow in the tissue of the brain are named primary brain tumors, and if neoplasm has grown in another organ and then affected the brain, the corresponding type of tumor is called a secondary brain tumor [2]. The most common primary brain tumors are meningiomas (referred as meningioma tumor), pituitary adenomas (referred as pituitary tumor), and astroglial neoplasms (including glioblastoma and referred to as glioma tumor) [3]. Treatments are dependent on the patient, but common treatment techniques for primary brain tumors are multimodality treatments, radiation, and chemotherapy [4].

Although there are many types of benign and malignant tumors, the most common ones are meningioma, glioma, and pituitary ones. Meningioma tumors form in the thin layers of tissue that cover the spinal cord and brain [5]. Gliomas are tumors that are thought to derive from neuroglial stem or progenitor cells [6]. They comprise 80% of all malignant brain tumors [7]. Pituitary adenomas are tumors of the anterior pituitary, and most of them are benign and slow-growing [8].

In this study, a classification of MR images into four different tumor classes, one normal and three different abnormal brain tumor classes, was carried out. Meningioma, glioma, and pituitary are the abnormal classes.

2. Related Works

There are many studies that have been conducted on brain tumor images with deep learning models in the literature. Rajat et al. have obtained a 99.04% binary classification accuracy with their pretrained AlexNet model on a public dataset obtained from The Cancer Imaging Archive (TCIA). The F1-scores of their model for benign and malignant tumors are 0.985075 and 0.992958, respectively [9]. Jianfeng et al. have obtained a 94.82% accuracy and 89.52% precision on a multiclass malignant tumor classification of a randomly divided CE-MRI dataset with the VGG19 (Visual Geometry Group) model [10]. Javed et al. used the Inceptionresnet V2 model and acquired a 98.91% accuracy and 98.28% precision. Their study on a publicly available Kaggle dataset consists of a malignant tumor classification [11]. Arshia et al. studied a publicly available Figshare dataset that consists of meningioma, pituitary, and glioma tumor classes. They obtained a 98.69% test accuracy with a fine-tune VGG16 model, data augmentation, and SGDM (stochastic gradient descent with momentum) optimizer [12]. In another study, Mohamed et al. used a custom dataset that has 155 tumor and 98 non-tumor brain images. They augmented the dataset to 1516 images and acquired the best accuracy of 98.24% with MobileNetV2 [13].

In the literature, one can see that to classify MR brain tumor images obtained from different hospitals, ResNet50, VGG16, and Inception v3 deep learning models are mainly used. In this study, classification processes were done with three different deep learning models and a preprocessing variable on open-access randomly distributed train, validation, and test datasets, which are different from the literature.

3. Materials and Methods

Two different datasets, which are available in open access on the Kaggle platform, are used for the multiclass classification of MR brain images [14,15]. The classes, percentages, and quantities of datasets can be seen in the

Table 1. Information about DS-1 (Dataset-1) [14].

Classes	Train Split	Validation Split	Test Split	Total
Normal	328	28	40	396
Meningioma	733	98	106	937
Glioma	752	95	79	926
Pituitary	715	95	91	901
Total	2528 (80%)	316 (10%)	316 (10%)	3160

and

Table 2. Information about DS-2 (Dataset-2) [15].

Classes	Train Split	Validation Split	Test Split	Total
Normal	1587	215	198	2000
Meningioma	1297	174	174	1645
Glioma	1334	143	144	1621
Pituitary	1401	170	186	1757
Total	5619 (80%)	702 (10%)	702 (10%)	7023

below.

In this study, ResNet50, RegNetY_16GF, and VisionTransformer_L_16 deep learning based models have been used for the classification process. All information about models and customizations is given below.

ResNet50 was used as the first model in this study. Residual Networks can be used as image classifiers. The architecture consists of sequential layers, and these layers contain bottleneck blocks [16]. In the Torchvision Library, the bottleneck blocks assigned the downsampling strides to the second 3 × 3 convolution, whereas the original paper assigned it to the first 1 × 1 convolution [17]. The last fully connected (FC) layer originally worked to classify images into 1000 categories, but datasets have 4 categories (normal, meningioma, glioma, and pituitary). Therefore, the last FC layer’s output features are customized to the number of classes.

RegNetY_16GF was used as the second model in this study. RegNet is a product of design spaces [18]. All RegNet models have stem, layer, and head blocks. These blocks can be customized with parameters. The stem layer is a Convolution + Batch Normalization + ReLU block. For this layer, the stride and filter size are 2 and 3, respectively. The layer block consists of chains of residual blocks. Residual blocks contain bottleneck blocks as in ResNet, but the RegNetY model has a squeeze and excitation attention module. Finally, the head block contains an AveragePool2D and FC layer. Similarly, output features are customized to the number of classes.

VisionTransformer_L_16 (ViT) was used as the third and last model. ViT uses a different deep learning method called transformer [19]. Encoders are the main blocks, and they have multiple layers. Each block consists of three elements: Layer Norm, Multi-head Attention, and Multi-Layer Perceptrons. Like the other two models, the head of the model was customized to the output features, equal to the number of classes.

In the training part, datasets are fed into models, where preprocessing is variable. Figure 1 shows the major processes of the training part.

For training and testing, Pytorch implementations of models are used. Training is partially done by HPC sources. Information about hardware can be seen in

Table 3. Information about hardware.

CPU	GPU	Memory	OS
Intel Xeon Scalable Gold 6148 (20 cores used)	2 X Nvidia Tesla V100 16 GB	170 GB	CentOS 7.3

.

The Contrast Limited Adaptive Histogram Equalization (CLAHE) preprocess has been applied to RGB images by converting the color format from BGR (blue green red) to LAB and then applying CLAHE on the L channel with a custom clip limit and tile grid size. An example of the CLAHE process can be seen in Figure 2.

4. Results

The obtained results are presented in the tables below. In

Table 4. Results for various models on DS-1 with CLAHE preprocess.

Model	Accuracy	Precision	Recall	F1 Score
ResNet50	0.94937	0.94	0.94	0.94
RegNetY_16GF	0.96519	0.96	0.96	0.96
VisionTransformer_L_16	0.9557	0.95	0.95	0.95

, one can see that the highest accuracy on DS-1 has been acquired with the RegNet model with preprocessing. In

Table 5. Results for various models on DS-1 without CLAHE preprocess.

Model	Accuracy	Precision	Recall	F1 Score
ResNet50	0.95253	0.95	0.94	0.95
RegNetY_16GF	0.93354	0.93	0.93	0.93
VisionTransformer_L_16	0.95253	0.95	0.94	0.95

, one can see that both VisionTransformer (ViT) and ResNet models have acquired the same accuracy, and preprocessing has not been applied to DS-1. Similarly,

Table 6. Results for various models on DS-2 with CLAHE preprocess.

Model	Accuracy	Precision	Recall	F1 Score
ResNet50	0.99288	0.99	0.99	0.99
RegNetY_16GF	0.99288	0.99	0.99	0.99
VisionTransformer_L_16	0.9886	0.99	0.99	0.99

shows that the best accuracy has been acquired with ResNet and RegNet models on DS-2 with preprocessing. Lastly,

Table 7. Results for various models on DS-2 without CLAHE preprocess.

Model	Accuracy	Precision	Recall	F1 Score
ResNet50	0.9943	0.99	0.99	0.99
RegNetY_16GF	0.99145	0.99	0.99	0.99
VisionTransformer_L_16	0.99003	0.99	0.99	0.99

shows that ResNet 50 has the best accuracy on DS-2 without preprocessing.

5. Conclusions and Future Work

Within the scope of this work, MR brain images are classified with various deep learning models, and it is observed that the Contrast Limited Adaptive Histogram Equalization (CLAHE) preprocess has positive effects on some of the models and datasets. Classification results are highly dependent on the used dataset and deep learning model. As a result of the multiclass classification study, the highest accuracy and recall on DS-1 have been 96.519% and 96%, respectively, and these results have been achieved with the RegNetY_16GF model. For DS-2, the best model has been ResNet50. Furthermore, the accuracy and recall have been 99.43% and 99%, respectively. The best results on DS-1 have been achieved with the CLAHE preprocess. In contrast, the CLAHE did not improve results on DS-2.

In future work, a hybrid system can be developed to assist physicists who are working in this field. Machine learning (ML) algorithms can be an addition to deep learning models in this system.