Oral Cancer Diagnosis Using Histopathology Images: An Explainable Hybrid Transformer Framework

Abstract

Oral cancer (OC) remains a major global health concern with survival often limited by late diagnosis. Early and accurate detection is essential to improve patient outcomes and guide treatment decisions. In this study we propose a computer aided diagnostic (CAD) framework for classifying oral squamous cell carcinoma from histopathology images. The model combines Swin transformer for hierarchical feature extraction with vision transformer (ViT) to capture long range dependencies across image regions. SHapley Additive exPlanations (SHAP) based feature selection enhances interpretability by highlighting the most informative features while preprocessing steps such as stain normalization and contrast enhancement improve model generalization and reduce sample variability. Evaluated on a publicly available dataset the framework achieved 99.25% accuracy (ACC) 99.21% sensitivity and a matthews correlation coefficient (MCC) of 98.21% outperforming existing methods. Ablation studies highlighted the importance of positional encoding and statistical analyses confirmed the robustness and reliability of results. To support real-time inference and scalable deployment the proposed model has been integrated into a FastAPI-based web application. This framework offers a powerful interpretable and practical tool for early OC detection and has potential for integration into routine clinical workflows.

1. Introduction

Cancer remains a major global health concern and is one of the leading causes of death worldwide [1,2]. Oral Cancer (OC) is the sixth most frequent cancer and by 2030 its incidence is predicted to increase by 30% every year [3,4]. OC mostly affects the head and neck area [5]. It includes the floor of the mouth, the hard palate, the gingiva, the buccal mucosa, the anterior two-thirds of the tongue, the lips and the retromolar pad. According to histology the majority of cases are squamous cell carcinomas [6,7,8]. Globally OC accounted for about 377,000 new cases and 177,000 deaths in 2020 with most patients being over 50 years old [9,10,11,12]. Although progress in detection and treatment the global five year survival rate for OC remains about 50% with significant regional variability. Late stage detection of OC greatly reduces treatment effectiveness and highlights the need for early diagnosis [13]. Routine oral screenings and thorough clinical examinations play a key role in improving patient outcomes [14]. Recent progress in diagnostic approaches has made it possible to detect precancerous lesions earlier, helping to improve patient outcomes and reduce the burden of OC [15,16]. Artificial intelligence (AI) provides novel opportunities to improve diagnostic ACC and enable personalized treatment strategies [17,18,19,20]. Machine learning (ML) and deep learning (DL) have shown strong ability to analyze histopathological images, detect subtle features and support clinical decisions [21,22]. AI improves traditional diagnostic methods by enabling faster and more reliable detection of malignancies [23]. Traditional histopathology depends on expert examination of tissue samples. A technique limited by inter observer variability subjective interpretation and labor intensive processes. [24,25]. AI based CAD systems provide objective and standardized analyses. They enhance consistency and efficiency while preserving diagnostic ACC [24,26]. DL models show strong performance in classifying oral squamous cell carcinoma (OSCC) images and provide a complementary tool to conventional diagnostic methods [23,27,28]. This study presents an intelligent CAD framework for the early detection of OC through automated histopathological image analysis. Which identifies malignancy associated patterns and generates visual outputs to assist pathologists in decision making. It is designed to minimize computational costs, maintain scalability and ensure accessibility in resource imited and rural settings. It is also aiming to support early diagnosis and improve patient outcomes. The main contributions of this study are as follows:

• A hybrid Swin–ViT framework enables effective feature extraction and accurate classification of OC histopathology images.
• Explainable AI (XAI) driven feature selection boosts interpretability and prediction while cutting dimensionality.
• A FastAPI-based real-time web application designed for scalable deployment in clinical and low-resource environments.

2. Related Works

Early detection of OC is challenging due to its complex presentation and high mortality. In recent years ML and DL techniques have gained attention for enhancing diagnostic ACC and enabling timely intervention. While early AI applications relied on conventional image processing, modern ML and DL approaches provide automated, robust and accurate analysis of OC images. Mira et al. developed an early OC detection framework using grayscale conversion, filtering and histogram equalization for preprocessing oral cavity images [29,30,31]. Cancer segmentation and manually extracted features based on shape, texture and color were classified using traditional ML methods. Although effective the approach lacked generalizability due to its reliance on handcrafted features and limited adaptability to diverse clinical data. To improve interpretability Ulaganathan et al. integrated immunohistochemistry (IHC) and whole slide imaging (WSI) with a CAD framework [32]. Their system digitized oral tissue biopsies and quantified key biomarkers and aiding personalized therapy but it remained semi automated and required substantial manual input. DL introduced a significant shift by enabling automatic representation learning and improving performance. Li et al. applied a U-Net with a ResNet34 encoder to segment cancer in endoscopic images showing strong generalization after training on 205 annotated samples [33]. Nanditha et al. employed a 43 layer Convolutional Neural Network (CNN) to analyze CT images for oral tumor detection and achieving high ACC and supporting early diagnosis [34]. In histopathology Kumar et al. used a two stage pipeline with CNN segmentation followed by random forest classification also outperforming traditional feature based methods [35]. Panigrahi et al. evaluated six ML models on histopathology images with a neural network achieving the highest ACC of 90.4% highlighting strong discriminative ability [36].

Recent work has explored histopathological images where Senthil Pandi et al. proposed a partitioned DL model using histopathological images with local binary patterns and a DL model with achieving 96.45% ACC [37]. Panahi and Farrokh reviewed ML in personalized dental medicine. Using radiographic images, clinical records, genetic data and patient reports, ML models helped detect caries, periodontal disease, OC, predict treatment outcomes and identify high-risk patients [38]. Challenges included data quality, interpretability, generalizability and clinical validation. Benefits were improved diagnosis, personalized care, patient experience and clinical efficiency. Jeyaraj et al. developed a regression based CNN model for histopathological images with 94.5% ACC [39]. Halder et al. combined attention based NASNet-Mobile, Gray Wolf Optimization for feature selection and ML classification with achieving 92.86% ACC [40]. Akhi et al. introduced OCNet by using transfer learning across multiple CNN architectures [41]. Their optimized VGG19 model reached 95.32% ACC providing a scalable solution for automated OSCC diagnosis.

These studies highlight the promise of ML and DL for OC detection. Compared with existing approaches the proposed framework integrates deep feature extraction, SHAP based feature selection and DL classification in a unified pipeline with achieving higher ACC, improved interpretability and scalable clinical deployment.

3. Materials and Methods

This section we outlines the data acquisition process, image pre-processing steps, feature extraction and selection techniques, along with the proposed framework. To provide a comprehensive understanding of the proposed approach the overall workflow of the study is illustrated in Figure 1. Furthermore, we incorporate a systematic algorithm to pseudocode of our whole system and which is illustrated int Algorithm 1.

Figure 1. Overall pipeline of the proposed cancer classification framework, including data preprocessing, feature extraction, SHAP-based feature selection, classifier training, model evaluation, and deployment.

Figure 1. Overall pipeline of the proposed cancer classification framework, including data preprocessing, feature extraction, SHAP-based feature selection, classifier training, model evaluation, and deployment.

Algorithm 1: Proposed OC Detection Framework Algorithm

Input:$E_p\leftarrow$ Number of Epochs; $W\leftarrow$ Model Parameters; $\eta \leftarrow$ Learning Rate; $b_s\leftarrow$ Batch Size; $D\leftarrow$ OC Histopathology Dataset Output: The assessment metrics on the test dataset. 1:  Dataset Preprocessing: 2:       $X_{train}\leftarrow preprocess\left(D\right)$ using CLAHE, bilateral filtering, gamma correction, stain normalization and morphological refinement. 3:       $X_{test}\leftarrow preprocess\left(D\right)$. 4:       Apply data augmentation (rotation, flipping, noise injection). 4a:       Apply ADASYN to address dataset imbalance and generate synthetic minority samples. 5:  Feature Extraction: 6:       Initialize Swin Transformer (swin_large_patch4_window7_224). 7:       for local epoch $e_p$ from 1 to $E_p$ do 8:          for $b_i=(x_s,y_s)$ ← random batch from $X_{train}$ do 9:             Optimize model parameters: $W_s\leftarrow W_s-\eta \nabla \left(\mathcal{L}(W_s;b_s)\right)$. 10:             $f_{train}\leftarrow ComputeFeatures(W_s,X_{train},1024)$. 11:          end for 12:       end for 13:  Feature Selection: 14:       $f_{best}\leftarrow SHAP(f_{train},500)$. 15:  Cancer Classification: 16:       Initialize ViT Classifier (ViT). 17:       $TrainedModel\leftarrow ViT(f_{best},y_{train})$. 18:       $Pred\leftarrow TrainedModel\left(X_{test}\right)$. 19:  Performance Evaluation: 20:       Compute evaluation metrics: ACC, Sensitivity, Specificity, Precision, F1-score and MCC. 21:       $Evaluationmetrics\leftarrow ComputeMetrics(Pred,y_{test})$.

3.1. Data Acquisition

In this study, we used a publicly available histopathological image dataset for OC analysis which is obtained from the Mendeley Data repository [42]. The dataset comprises digitized hematoxylin and eosin (H&E) stained tissue slides from 230 patients with images captured using a Leica ICC50 HD microscope under standardized conditions. All tissue samples were carefully collected, prepared and catalogued by medical experts to ensure accurate diagnostic labeling. The repository contains a total of 1224 histopathological images organized into two resolution sets. For this study the first set is acquired at 100× magnification, which is selected for experiment. This subset includes 528 images, comprising 89 normal oral epithelium and 439 oral squamous cell carcinoma (OSCC) images.

Table 1. Dataset Details.

Parameter	Specification
Total Images	528
Normal/OSCC	89/439
Patients	230
Magnification	100× (H&E stained)

summarizes the dataset. The chosen magnification provides an optimal balance between visualization of tissue architecture and cellular morphology which is essential for distinguishing normal from malignant structures. The dataset is suitable for a binary classification task with a class distribution ratio of approximately 1:4.9 (normal:OSCC) reflecting real world clinical conditions. All images were validated by expert pathologists at the time of the original dataset creation with ground truth labels established through consensus and correlation with histopathological reports. Data collection, processing and validation strictly followed the protocols described in the source publication. The dataset was divided into three separate segments according to a 70-15-15 split ratio: 70% allocated for training, 15% for validation, and 15% for testing. The splits were performed at the image level as patient-level identifiers were not available in the source dataset.

3.2. Data Preprocessing

The preprocessing of data involves some specific techniques those are applied before the experiment which enhance the diversity of data and generalize it. All histopathological images were standardized to a resolution of 224 × 224 pixels and converted to RGB format to ensure compatibility with DL architectures. Several preprocessing techniques were applied to enhance image quality and reduce variability across samples. Contrast Limited Adaptive Histogram Equalization (CLAHE) was employed to improve local contrast while preserving structural integrity, enabling more effective feature extraction [43,44]. Noise suppression was performed using bilateral filtering which reduces variability by considering both spatial and intensity information of neighboring pixels, producing smoother yet detail preserving images [45,46]. Gamma correction was applied as a nonlinear transformation to adjust pixel intensities and enhance contrast that is improving the visibility of low contrast regions [47]. To address staining inconsistencies both conventional and generative adversarial network (GAN) based stain normalization approaches were adopted. The two stage GAN strategy ensured consistent color representation across samples while maintaining morphological integrity and improving the reliability of downstream classification tasks [48,49]. Morphological operations such as opening and adaptive thresholding were used to refine cellular boundaries and suppress background artifacts. Otsu thresholding was applied to segment tumor regions into standardized patches, increasing dataset uniformity [48]. Unsharp masking was incorporated to enhance structural clarity by emphasizing high-frequency image components through subtraction of a blurred version from the original image [50,51]. Data augmentation techniques including controlled rotations and noise injection were applied to expand dataset diversity, mitigate overfitting and improve generalization capacity. Collectively these preprocessing strategies enhanced image quality, reduced variability and strengthened dataset robustness, providing a reliable foundation for accurate feature extraction in OC histopathology analysis. These preprocessing strategies enhanced image quality, reduced variability and strengthened dataset robustness that is providing a reliable foundation for accurate feature extraction in OC histopathology analysis. To provide a clearer understanding of the sequential flow of these operations the complete preprocessing pipeline is represented in Figure 2. To address the data imbalance problem in our dataset, we employed the Adaptive Synthetic Sampling (ADASYN) technique to generate synthetic minority samples and improve class distribution [52].

3.3. Feature Extraction Using Swin Transformer

To determine an effective backbone for feature extraction several architectures were evaluated including ConvNeXt, Swin Transformer and BEiT. The Swin Transformer achieved the highest ACC and was therefore adopted as the primary feature extractor. The Swin Transformer Large configuration [53,54,55,56] was employed using a patch size of $4\times 4$, a window size of $7\times 7$ and an input resolution of $224\times 224$. The pretrained classification head was removed and replaced with an identity layer to produce generic, task independent feature representations. The Swin Transformer constructs a hierarchical representation of an image through four stages. The input image is first divided into non overlapping patches and each of which is linearly embedded into a feature vector. Subsequent stages progressively reduce spatial resolution and increase channel depth through patch merging where the embeddings of four neighbouring patches are concatenated and linearly transformed into a new representation:

$$Y_{i,j}=W\left[\begin{array}{c}{X}_{2i,2j}\\ X_{2i+1,2j}\\ X_{2i,2j+1}\\ X_{2i+1,2j+1}\end{array}\right]$$

(1)

This hierarchical design allows the network to capture increasingly rich semantic information while maintaining computational efficiency. At the final stage feature maps with 1536 channels are produced. A lightweight convolutional refinement block is then applied to emphasize informative regions and suppress noise. This block employs $1\times 1$ and $3\times 3$ convolutions, batch normalization and ReLURectified Linear Unit (ReLU) activations to refine the feature maps. After refinement, spatial information is aggregated using global average pooling to produce a compact vector for each channel:

$$z_c={\displaystyle \frac{1}{HW}}\sum _{i=1}^H\sum _{j=1}^W{X}_{c,i,j},c=1\dots 1536$$

(2)

Finally a two-layer projection head reduces the feature vector from 1536 to 1024 dimensions. The first linear layer maps the refined features to 1280 units with ReLU activation and batch normalization followed by a second layer mapping to 1024 units with ReLU, batch normalization and dropout at a rate of $p=0.2$. This feature extraction pipeline combines the Swin Transformer’s hierarchical learning capabilities with simple convolutional refinement, yielding compact and robust descriptors well suited for histopathological image analysis [57,58,59,60].

3.4. Proposed Model Architecture

The 1024-dimensional embeddings extracted by the Swin Transformer feature extractor were used as input for a vision transformer (ViT) classifier [61,62,63]. These embeddings were projected to a lower-dimensional space and combined with positional encodings to form a sequential input suitable for the transformer:

$$Z_0=X{W}_e+E_{pos}$$

(3)

where X denotes the input embeddings $W_e$ is the projection matrix and $E_{pos}$ represents positional encodings. The sequence was processed through four transformer encoder layers, each consisting of multi-head self-attention (MSA), layer normalization (LN), feed-forward networks, and residual connections:

$$Z_l^{\prime }=\mathrm{MSA}\left(\mathrm{LN}\left(Z_l\right)\right)+Z_l$$

(4)

$$Z_{l+1}=\mathrm{FFN}\left(\mathrm{LN}\left(Z_l^{\prime }\right)\right)+Z_l^{\prime }$$

(5)

where $Z_l$ is the input to the lth layer, $Z_l^{\prime }$ is the output after the attention sublayer, MSA denotes multi-head self-attention with 8 heads, LN denotes layer normalization, and FFN represents the feed-forward network with hidden dimension 256. After the final transformer layer, global average pooling was applied to obtain a single feature vector. The classification head applied a linear transformation followed by softmax activation:

$$\widehat{y}=\mathrm{softmax}(W_c{z}_{\mathrm{pool}}+b_c)$$

(6)

where $z_{\mathrm{pool}}$ is the pooled representation and $W_c$, $b_c$ are learnable parameters. The model was trained using cross-entropy loss. Optimization was performed using the Adam optimizer with ${\beta }_1=0.9$, ${\beta }_2=0.999$ and $ϵ={10}^{-8}$, and a learning rate of 0.0001. Training was conducted for 50 epochs. This approach leverages the hierarchical embeddings from the Swin Transformer enabling robust and accurate classification of OC histopathology images [53,64,65].

3.5. Feature Selection Using SHAP

Feature selection plays an important role in enhancing model performance by identifying and retaining the most relevant attributes. In this work we applied SHAP [66,67,68] which is grounded in cooperative game theory [69,70,71] and assigns an importance score to each feature with respect to a prediction. In a coalitional game with N players, the Shapley value ${\varphi }_i$ for a player (or feature) i is defined as:

$${\varphi }_i=\sum _{S\subseteq N\setminus \left\{i\right\}}{\displaystyle \frac{\left|S\right|!\left(\right|N|-|S|-1)!}{\left|N\right|!}}\left[v(S\cup \{i\left\}\right)-v\left(S\right)\right],$$

(7)

where S is any subset of N not containing i, and $v\left(S\right)$ represents the predictive value of the coalition S. The term $v(S\cup \{i\left\}\right)-v\left(S\right)$ reflects the marginal contribution of feature i to the coalition. In a predictive modeling context, SHAP approximates the output of a complex model $f\left(x\right)$ with a simplified linear representation $g\left(x^{\prime }\right)$ based on binary variables $x^{\prime }\in {\{0,1\}}^M$, where M is the number of features:

$$g\left(x^{\prime }\right)={\varphi }_0+\sum _{i=1}^M{\varphi }_i{x}_i^{\prime },$$

(8)

with ${\varphi }_0$ denoting the baseline output (expected prediction when no features are present) and ${\varphi }_i$ the contribution of feature i. Here, $x_i^{\prime }=1$ indicates that the i-th feature is included, and $x_i^{\prime }=0$ otherwise [72,73].

The Shapley value for feature i with respect to an instance x is then computed as:

$${\varphi }_i(f,x)=\sum _{z^{\prime }\subseteq x^{\prime }}{\displaystyle \frac{|z^{\prime }|!(M-|z^{\prime }|-1)!}{M!}}\left[f_x\left(z^{\prime }\right)-f_x(z^{\prime }\setminus i)\right],$$

(9)

where $z^{\prime }$ represents subsets of the simplified input $x^{\prime }$, and $f_x\left(z^{\prime }\right)$ corresponds to the model output restricted to those features. By ranking features according to their absolute Shapley values the most informative attributes were selected for classification. In practice this procedure reduced redundancy and improved predictive ACC while also ensuring interpretability. Empirical evaluation further showed that selecting the top-ranked subset of features significantly enhanced overall performance compared to using all features [74,75,76].

4. Results

4.1. Evaluating Metrics

To evaluate the performance of different classification models and the overall predictive approach multiple metrics were considered. Key measures include Accuracy (ACC), MCC, Sensitivity, Specificity, Precision (PRE), and F1-score [68,77,78,79,80]. The definitions of these metrics are provided below:

$$\mathrm{ACC}={\displaystyle \frac{TP+TN}{TP+TN+FN+FP}}$$

(10)

$$\mathrm{Sensitivity}={\displaystyle \frac{TP}{TP+FN}}$$

(11)

$$\mathrm{Specificity}={\displaystyle \frac{TN}{FP+TN}}$$

(12)

$$\mathrm{MCC}={\displaystyle \frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)(TN+FP)(TN+FN)(TP+FN)}}}$$

(13)

$$\mathrm{PRE}={\displaystyle \frac{TP}{TP+FP}}$$

(14)

$$\mathrm{F}1\mathrm{score}={\displaystyle \frac{2\times (\mathrm{PRE}\times \mathrm{Recall})}{\mathrm{PRE}+\mathrm{Recall}}}$$

(15)

Here, TP (True Positive) refers to the number of actual positive instances that are correctly classified, while FP (False Positive) denotes the number of negative instances incorrectly predicted as positive. TN (True Negative) indicates the count of negative instances accurately identified, and FN (False Negative) represents the number of positive instances that were incorrectly classified as negative. In addition the AUC metric was employed to evaluate the overall performance of the proposed model, with the curve plotted by placing the true positive rate (TPR or sensitivity) on the y-axis and the false positive rate (FPR = 1 − specificity) on the x-axis across varying threshold values.

4.2. Experimental Evaluation of Feature Extractors and Classifiers

To identify the most effective feature extractor we evaluated three architectures those are ConvNeXt, BEiT and Swin Transformer. Following the transfer learning approach the fully connected layers of these models were removed and replaced with five classifiers those are Random Forest, CatBoost, Focal Loss Support Vector Machine (SVM), Attention CNN (AttCNN) and ViT. This combinatorial approach was designed to systematically assess and compare the performance of each feature extractor across multiple classifiers. Additionally it enabled the identification of the most suitable classifier among both ML and DL approaches. We used different validation and test sets for all of the experiments. The validation set was used to select the optimal feature extractor, classifier, and their best-performing combination, while the test set was reserved exclusively for final performance evaluation. From the results presented in

Table 2. Experimental results of the performance of feature extractors with different classifiers.

Feature Extractor	Classifier	ACC	AUC	PRE	SP	SN	F1	MCC
ConvNeXt	Random Forest	0.7985	0.8724	0.7681	0.7324	0.7193	0.7523	0.6742
	CatBoost	0.8124	0.8541	0.7891	0.7482	0.7410	0.7812	0.7311
	Focal Loss SVM	0.8562	0.9310	0.8478	0.7589	0.7661	0.7894	0.7833
	Attention-CNN	0.8841	0.9492	0.8847	0.8134	0.8210	0.8481	0.8294
	ViT	0.9126	0.9662	0.9234	0.8791	0.8864	0.9170	0.8921
BEiT	Random Forest	0.7342	0.8834	0.5893	0.6844	0.6752	0.5093	0.7081
	CatBoost	0.7654	0.8901	0.7041	0.7103	0.6989	0.6012	0.7394
	Focal Loss SVM	0.8073	0.8989	0.8092	0.7512	0.7394	0.6762	0.7620
	Attention-CNN	0.8421	0.9023	0.9174	0.8133	0.8021	0.7555	0.8024
	ViT	0.9012	0.9184	0.9321	0.8722	0.8641	0.8794	0.8613
Swin Transformer	Random Forest	0.8031	0.8622	0.7724	0.6421	0.6554	0.6681	0.4043
	CatBoost	0.7814	0.8981	0.7463	0.6024	0.6138	0.6132	0.3194
	Focal Loss SVM	0.8482	0.9203	0.8191	0.7911	0.7820	0.7663	0.6670
	Attention-CNN	0.9124	0.9489	0.8912	0.8851	0.8924	0.9281	0.8214
	ViT	0.9421	0.9738	0.9602	0.9741	0.9662	0.9623	0.9714

, it is clarifying that the Swin Transformer achieved the highest average performance across all five classifiers with an overall ACC of 87.25%. ConvNeXt and BEiT obtained average accuracies of 87.08% and 82.79% respectively corresponding to reductions of 0.17% and 4.46% relative to Swin Transformer.

In terms of AUC Swin Transformer again led with an average score of 92.07%, exceeding ConvNeXt (91.46%) and BEiT (89.86%) by 0.61% and 2.21% respectively. MCC further highlighted the superiority of Swin Transformer which achieved 63.67% compared to 78.20% for ConvNeXt and 77.46% for BEiT. Specificity and sensitivity metrics were 77.90% and 78.20% for Swin Transformer, outperforming ConvNeXt (78.64% and 78.68%) and BEiT (76.63% and 75.59%) by notable margins. These results demonstrate that Swin Transformer consistently surpasses the other feature extractors across all evaluation metrics, establishing it as the preferred choice. Regarding classifiers the ViT consistently delivered the highest performance. It achieved an average ACC of 91.86%, outperforming Random Forest (77.86%), CatBoost (78.64%), Focal Loss SVM (83.72%) and AttCNN (87.95%) by 13.99%, 13.22%, 8.13% and 3.91% respectively. The highest average AUC score of 95.28% was also attained by the ViT exceeding the other classifiers by 8.01%, 7.20%, 3.61% and 1.93% respectively. Similarly the MCC of ViT was 90.83% with other classifiers scoring 59.55% (Random Forest), 59.67% (CatBoost), 73.74% (Focal Loss SVM) and 81.77% (AttCNN) indicating differences of 31.28%, 31.16%, 17.09% and 9.06%. Evaluation of additional metrics including specificity (90.85%) and sensitivity (90.56%) further confirmed the ViT’s superior performance. Consequently the ViT was selected as the optimal classifier. The analysis also revealed that combining the ViT with any feature extractor resulted in consistently high ACC, AUC, MCC, specificity and sensitivity values. Likewise using any classifier with the Swin Transformer yielded the best results relative to other feature extractors. Ultimately the combination of Swin Transformer as the feature extractor and ViT as the classifier achieved the highest overall ACC of 95.75% surpassing all other evaluated configurations. Based on these findings this combination was adopted to construct the proposed model. The performance differences among the three feature extractors are further highlighted by their ROC curves (Figure 3A–C) and precision recall curve (Figure 3D–F).

These curves capture the balance between sensitivity and 1 specificity and the corresponding AUC values summarize the overall classification effectiveness. Swin Transformer consistently achieved the highest AUC which reflects its stronger ability to discriminate between classes compared to ConvNeXt and BEiT. These visual findings are consistent with the quantitative metrics that confirm the superior performance of Swin Transformer as a feature extractor. Beyond the ROC analysis we also analyzed the impact of feature selection on performance focusing both on the best performing feature extractor classifier combinations and on how varying feature counts influence ACC and AUC as illustrated in Figure 4.

4.3. Impact of Feature Selection Techniques

To identify the most effective feature selection technique for OC detection, we evaluated five approaches: mRMR, Lasso, Boruta, Genetic Algorithm and SHAP. The comparative results are presented in

Table 3. Experimental results of the performance of different feature selection techniques for OC detection.

Feature Selection	ACC	AUC	PRE	SP	SN	F1	MCC
mRMR	0.9625	0.9642	0.9633	0.9723	0.9712	0.9724	0.9735
Lasso	0.9612	0.9552	0.9562	0.9778	0.9643	0.9634	0.9655
Boruta	0.9595	0.9462	0.9491	0.9833	0.9574	0.9544	0.9575
Genetic Algorithm	0.9780	0.9772	0.9820	0.9888	0.9705	0.9654	0.9695
SHAP	0.9925	0.9822	0.9826	0.9918	0.9921	0.9843	0.9821

. Among these, SHAP achieved the best overall performance with the highest ACC (99.25%), sensitivity (99.21%) and strong AUC (98.22%) along with superior F1 (98.43%) and MCC (98.21%) values. These findings demonstrate that SHAP substantially improves the model’s ability to differentiate between cancerous and non-cancerous samples across multiple evaluation metrics. The Genetic Algorithm also yielded competitive results, with an ACC of 97.80%, specificity of 98.88% and PRE of 98.20%, though it did not surpass SHAP in terms of balanced performance. In contrast, mRMR (96.25%), Lasso (96.12%) and Boruta (95.95%) showed relatively lower accuracies. While these methods maintained strong specificity (above 97%), they were less effective in providing consistently high values across sensitivity, F1 and MCC. Overall, SHAP emerged as the most effective feature selection technique for OC detection, delivering superior and well balanced performance across nearly all evaluation metrics.

To provide a more comprehensive comparison Figure 5 combines multiple visual perspectives on feature selection performance and feature count optimization. Figure 5A,B reaffirm earlier observations are SHAP consistently delivers superior predictive performance with narrow score distributions and approximately 500 features offer the most favorable balance between ACC and stability. Figure 5C extends this evaluation by comparing all feature selection methods across multiple performance metrics using a radar chart. SHAP dominates across nearly every dimension particularly in ACC, sensitivity, F1 and MCC, underscoring its robustness as a selection strategy. Finally Figure 5D illustrates the performance trends across varying feature counts where 500 features again emerge as the optimal threshold for stable and well rounded results. Both visualizations highlight the critical role of both the selection technique and feature dimensionality in shaping reliable predictive models for OC detection.

The performance of the Swin Transformer combined with SHAP based feature selection under varying feature counts is summarized in

Table 4. Performance of the Swin Transformer combined with SHAP across varying numbers of selected features.

Features	ACC	AUC	PRE	SP	SN	F1	MCC
200	0.8875	0.9200	0.9831	0.9286	0.8788	0.9274	0.6977
300	0.9375	0.9400	0.9841	0.9286	0.9394	0.9606	0.8063
500	0.9925	0.9822	0.9826	0.9918	0.9921	0.9843	0.9821
700	0.9625	0.9750	0.9701	0.8571	0.9848	0.9765	0.8673
900	0.9375	0.9600	0.9552	0.7857	0.9697	0.9620	0.7778

. The analysis indicates that the ACC and other evaluation metrics varied with the number of features used. With 200 features the model achieved an ACC of 0.8875 and an AUC of 0.9200 while PRE and sensitivity were 0.9831 and 0.8788 respectively. Increasing the feature count to 300 improved PRE to 0.9841 and sensitivity to 0.9394 indicating more reliable identification of positive cases. Optimal performance was observed with 500 features where the model consistently outperformed all other configurations. In this setting it achieved an ACC of 0.9925, AUC of 0.9822, PRE of 0.9826, specificity of 0.9918, sensitivity of 0.9921, F1-score of 0.9843 and MCC of 0.9821. Beyond this point increasing the feature count to 700 and 900 led to a slight reduction in performance. Although ACC remained high (0.9625 and 0.9375 respectively) decreases in sensitivity, F1-score and MCC suggest that excessive features introduced redundancy or noise. Overall these findings demonstrate that SHAP based feature selection effectively identified the most discriminative subset of features while the Swin Transformer served as a powerful backbone for representation learning. In order to highlight this the comparative impact of different selection methods Figure 6 presents a comprehensive analysis. Subfigures A–C compare feature overlap, retained feature counts and performance across metrics while Subfigure D shows the SHAP summary plot of the top 20 influential features. These results reinforce SHAP’s consistent ability to identify the most discriminative features for OC detection. Furthermore, we demonstrate the Grad-CAM of input images to identify the areas of interest or potential impact by our feature extractor. Figure 7 illustrates various input images alongside their corresponding Grad-CAM visualizations.

4.4. Hyperparameter Analysis of Swin Transformer

We evaluated the impact of key hyperparameters on the performance of the proposed Swin Transformer based feature extractor and ViT classifier.

Table 5. Hyperparameters for Swin Transformer (Feature Extractor).

Name	Parameter
Output Dimension	1024
Base Model Name	swin_large_patch4_window7_224
Base Feature Dimension	1536
Attention Dimension	1536
First Linear Layer Input	1536
First Linear Layer Output	1280
Second Linear Layer Input	1280
Second Linear Layer Output	1024
First Activation Function	ReLU
Second Activation Function	ReLU
First Batch Normalization	BatchNorm1d(1280)
Second Batch Normalization	BatchNorm1d(1024)
Dropout Rate	0.2
Patch Size	4
Window Size	7
Pooling Method	Mean pooling

summarizes the hyperparameter settings used for the Swin Transformer feature extractor. The model uses the swin_large_patch4_window7_224 backbone, which produces a 1536-dimensional feature representation at its final stage. This representation is subsequently refined using two linear layers with dimensions 1536 → 1280 → 1024, where each layer is followed by ReLU activation, Batch Normalization, and a dropout rate of 0.2 to reduce overfitting. The feature extractor operates on images divided into 4 × 4 patches with a window size of 7, and the resulting features are aggregated using mean pooling.

Table 6. Hyperparameters for ViT Classifier (Proposed Model).

Name	Parameter
Input Dimension	1024
Embedding Dimension	128
Number of Attention Heads	8
Feedforward Dimension	256
Dropout Rate	0.1
Number of Transformer Layers	4
Learning Rate	0.0001
Training Epochs	50
Batch Size	128

presents the hyperparameters for the ViT classifier applied on the extracted features. The classifier takes a 1024 dimensional input embedding and projects it into a 128 dimensional embedding space using 8 attention heads and a feedforward dimension of 256. To improve generalization a dropout rate of 0.1 is applied across 4 transformer layers. The classifier is trained with a learning rate of 0.0001 for 50 epochs using a batch size of 128.

Overall these hyperparameter choices were selected based on preliminary experiments to optimize classifier performance while ensuring computational efficiency. Adjustments to learning rate, batch size and the number of training epochs were found to significantly influence convergence and generalization.

4.5. Ablation Study

To evaluate the contribution of each component of the proposed model we conducted an ablation study. The results are summarized in

Table 7. Ablation Study Results.

Model	ACC	AUC	PRE	SP	SN	F1	MCC
Full Model	0.9925	0.9822	0.9826	0.9918	0.9921	0.9843	0.9821
No Positional	0.8000	0.8052	0.9310	0.7143	0.8182	0.8708	0.4527
Shallow ViT	0.7875	0.8106	0.9455	0.7857	0.7879	0.8592	0.4703
Deep ViT	0.9875	0.8604	0.9851	0.9286	1.0000	0.9925	0.9562
Max Pooling	0.9750	0.9264	0.9706	0.8571	1.0000	0.9851	0.9121

which demonstrate the impact of different architectural modifications on model performance. The full model achieved the highest performance across all metrics with an ACC (ACC) of 0.9925, area under the curve (AUC) of 0.9822, F1-score of 0.9843 and MCC of 0.9821, confirming the effectiveness of the complete architecture. Removing positional encoding led to a significant drop in performance (ACC = 0.8000, AUC = 0.8052, F1 = 0.8708, MCC = 0.4527) highlighting the importance of spatial information. Using a ViT reduced PRE (0.9455) and F1-score (0.8592) despite relatively high SP (0.7857) and SN (0.7879) indicating that deeper layers are essential for effective feature extraction. The deep ViT variant maintained high ACC (0.9875) and perfect SN (1.0000) but showed a decrease in AUC (0.8604) suggesting limited generalization without the full architecture. Replacing attention with max pooling produced competitive results (ACC = 0.9750, F1 = 0.9851) but still underperformed compared to the full model emphasizing the importance of attention based feature aggregation. These findings confirm that positional encoding model depth and attention mechanisms are all critical components of the proposed architecture. The results of the ablation study are presented to elucidate the contribution of each model component to overall performance. In addition to predictive accuracy evaluating computational efficiency is essential. Figure 8 illustrates performance comparisons across seven evaluation metrics (ACC, AUC, PRE, SP, SN, F1, MCC) for different model variants, with (A) grouped bar chart and (B) trend lines demonstrating the Full Model’s superior and consistent performance across all metrics.

4.6. Statistical Analysis of Model Performance

To assess the statistical significance of the proposed model’s performance we conducted both metric-wise and model wise analyses which are summarized in

Table 8. Metric-wise Statistical Test Results.

Metric	t-Test (t-Stat)	t-Test (p-Value)	Wilcoxon (Stat)	Wilcoxon (p-Value)
ACC	−4.105	0.0051	2.5	0.0284
AUC	−3.220	0.0119	4.5	0.0371
PRE	−5.928	0.0006	1.0	0.0120
SN	−4.667	0.0019	1.8	0.0213
F1	−5.872	0.0005	0.0	0.0065
MCC	−3.401	0.0112	3.5	0.0369

and

Table 9. Model-wise Statistical Test Results.

Model	t-Test (t-Stat)	t-Test (p-Value)	Wilcoxon (Stat)	Wilcoxon (p-Value)
Random Forest	−3.003	0.0183	3.0	0.0350
CatBoost	−4.567	0.0025	1.5	0.0201
Focal Loss SVM	−5.445	0.0009	1.0	0.0154
Attention-CNN	−3.890	0.0059	2.8	0.0298
ViT	−6.751	0.0002	0.0	0.0061

. The metric wise evaluation shows that all performance measures including ACC, AUC, PRE, SN, F1 and MCC exhibited significant improvements compared to baseline values. t-test, p-values ranged from 0.0005 to 0.0119 with corresponding Wilcoxon p-values ranging from 0.0065 to 0.0371. Notably PRE and F1-score demonstrated the most significant differences (t-test p = 0.0006 and 0.0005 and Wilcoxon p = 0.0120 and 0.0065) indicating substantial gains in predictive performance. The model wise analysis further confirms the superiority of the proposed approach over baseline models. t-test and Wilcoxon test results indicate statistically significant differences for Random Forest, CatBoost, Focal Loss SVM, AttCNN and ViT models. Overall these statistical analyses validate that the observed improvements are not due to random chance that is confirming the reliability and effectiveness of the proposed model.

4.7. Web Application for Real-Time OC Classification

To bridge the gap between experimental results and practical application we developed a web application capable of performing real-time classification of OC from histopathological images. The deployment architecture comprises three distinct components working in concert to deliver efficient inference capabilities. At first all of the trained models such as the Swing Transformer feature extractor and ViT classification head are deployed on the Hugging Face Model Hub (https://huggingface.co/rudradcruze/oral-cancer, accessed on 20 November 2025), ensuring version control, easy accessibility and reproducibility. After that by using this model we developed a RESTful API using the FastAPI framework in Python 3.10 and this application is deployed on Hugging Face Spaces (https://huggingface.co/spaces/rudradcruze/oral-cancer, accessed on 20 November 2025) Docker SDK. This API handles all of the heavy tasks including model loading, image reprocessing, feature extraction via Swin Transformer, feature selection based on SHAP analysis as determined in our experimental study (Table 4), and final classification through the ViT head. Finally, to make it user-friendly and accessible for all, we developed a client-side interface using standard and modern web technologies (HTML, CSS, JavaScript and TailwindCSS) with FastAPI serving the frontend, deployed on Vercel (https://oralcancer.francisrudra.com, accessed on 20 November 2025). This interface gives functionality to users to upload the histopathological images through an intuitive browser-based environment and communicates with the Hugging Face Space API to retrieve predictions. All of the experimental code which are used in this experiment is available on https://github.com/rudradcruze/oral-cancer (accessed on 20 November 2025) GitHub repository.

4.8. Comparison with Existing Methods

To validate the effectiveness of the proposed approach we compared its performance with state-of-the-art methods for OC detection using histopathological images, shown in

Table 10. Comparison Study.

Study	Technique	ACC
Shavlokhova et al. [81]	ResNet50 with feature fusion	92.37%
Dai et al. [82]	ResNet50 with Raman Spectra	92.48%
Yu et al. [83]	ResNet50 with DCNNs	95.84%
Chang et al. [84]	ResNet50 with VGG16	85.63%
Panigrahi et al. [85]	Three Kinds of CNN	92.14%
Sukegawa et al. [86]	Probability Neural Network	79.46%
Das et al. [87]	Multiple techniques fusion	97.41%
Nagarajan et al. [88]	MobileNetV3 with Gorilla Troops Optimizer	94.18%
Flügge et al. [89]	Swin Transformer	97.63%
Albalawi et al. [90]	EfficientNet B3 with Advanced Learning Mechanism	98.47%
Proposed Model	Swin Transformer with Vision Transformer	99.25%

. The proposed Modified Swin Transformer with a ViT head achieved the highest ACC of 99.25%, outperforming all existing approaches. The closest competing method was reported by Albalawi et al. which achieved 98.47% ACC using EfficientNet B3 with an advanced learning mechanism, 0.78% lower than our approach. Flügge et al. achieved 97.63% with a Swin Transformer while Das et al. reported 97.41% by combining multiple architectures. Yu et al. obtained 95.84% with a ResNet50 based DCNN framework. Other notable results include Nagarajan et al. (94.18% using MobileNetV3 with Gorilla Troops Optimizer), Panigrahi et al. (92.14% using three CNN models) and ResNet50 based methods by Shavlokhova et al. (92.37%) and Dai et al. (92.48%). Chang et al. reported 85.63% with a ResNet50-VGG16 hybrid while the lowest performance was observed in Sukegawa et al. (79.46% with a Probability Neural Network). The superior performance of the proposed framework can be attributed to the synergistic combination of hierarchical feature extraction with Swin Transformer, global attention mechanisms through ViT and effective preprocessing strategies that enhanced feature discriminability and overall classification ACC. All models are evaluated on the same dataset under identical experimental configurations to ensure fair comparison.

5. Conclusions

This study presents a hybrid DL framework for automated OC detection using histopathological images. The proposed methodology combines Swin Transformer feature extraction, ViT classification and SHAP based feature selection to achieve superior diagnostic performance with 99.25% ACC that is outperforming existing advanced approaches. The systematic evaluation confirmed that Swin Transformer provides optimal feature extraction while ViT delivers the best classification performance. SHAP based feature selection with 500 features achieved the ideal balance between ACC and computational efficiency. Statistical analyses validated the significance of improvements across all evaluation metrics. The framework’s practical applicability is demonstrated through a FastAPI based web application enabling real time deployment in clinical settings. Despite promising results, limitations include dataset size constraints, the absence of patient-level identifiers which prevented patient-based data splitting, and the need for validation across diverse populations. Future work should focus on multi institutional validation and integration of multimodal data sources to enhance clinical applicability and robustness.