HistoMoCo: Momentum Contrastive Learning Pre-Training on Unlabeled Histopathological Images for Oral Squamous Cell Carcinoma Detection

Abstract

The early detection and intervention of oral squamous cell carcinoma (OSCC) using histopathological images are crucial for improving patient outcomes. The current literature for identifying OSCC predominantly relies on models pre-trained on ImageNet to minimize the need for manual data annotations in model fine-tuning. However, a significant data divergence exists between visual domains of natural images in ImageNet and histopathological images, potentially limiting the representation and transferability of these models. Inspired by recent self-supervised research, in this work, we propose HistoMoCo, an adaptation of Momentum Contrastive Learning (MoCo), designed to generate models with enhanced image representations and initializations for OSCC detection in histopathological images. Specifically, HistoMoCo aggregates 102,228 histopathological images and leverages the structure and features unique to histological data, allowing for more robust feature extraction and subsequent downstream fine-tuning. We perform OSCC detection tasks to evaluate HistoMoCo on two real-world histopathological image datasets, including NDB-UFES and Oral Histopathology datasets. Experimental results demonstrate that HistoMoCo consistently outperforms traditional ImageNet-based pre-training, yielding more stable and accurate performance in OSCC detection, achieving AUROC results up to 99.4% on the NDB-UFES dataset and 94.8% on the Oral Histopathology dataset. Furthermore, on the NDB-UFES dataset, the ImageNet-based pre-training solution achieves an AUROC of 89.32% using 40% of the training data, whereas HistoMoCo reaches an AUROC of 89.58% using only 10% of the training data. HistoMoCo addresses the issue of domain divergence between natural images and histopathological images, achieving state-of-the-art performance in two OSCC detection datasets. More importantly, HistoMoCo significantly reduces the reliance on manual annotations in the training dataset. We release our code and pre-trained parameters for further research in histopathology or OSCC detection tasks.

1. Introduction

The increasing incidence of cancer globally underscores the urgent need for effective screening and diagnostic strategies to enable timely detection and treatment. Among various types of cancer, oral squamous cell carcinoma (OSCC) has emerged as a significant health concern [1], particularly affecting regions with high prevalence rates, such as Asia. According to recent reports, OSCC ranks 16th in terms of global incidence and mortality, with Asia accounting for approximately 65.8% of cases and a striking death rate of 74.0% [1]. Early detection and intervention are crucial for improving outcomes, as they substantially enhance the chances of complete remission. Despite advancements in awareness programs and clinical protocols, the effective screening of OSCC remains challenging, emphasizing the importance of developing robust diagnostic tools. These tools are particularly needed to address the significant variations in clinical presentations and the impact of risk factors like tobacco use [2]. Therefore, enhancing OSCC detection methodologies is vital to reduce the disease burden and improve patient survival rates.

In recent years, deep learning-based solutions [3,4] have exhibited state-of-the-art performance in extracting valuable insights from histopathological images [5,6], offering promising avenues to address these challenges. An AI-assisted workflow could potentially serve as a “second reader”, pre-screening normal samples to reduce pathologists’ workload, automatically highlighting suspicious regions to guide attention, and providing remote preliminary screening for underserved areas. Such systems could significantly reduce analysis time while improving diagnostic consistency and accessibility. However, developing effective AI systems requires an extremely large number of images with fine annotations to train deep neural networks (DNNs) and deliver decent performance in a supervised learning manner [7,8,9]. Unfortunately, large-scale annotated OSCC histopathological datasets remain scarce due to multiple factors: (1) stringent patient privacy regulations and ethical constraints limiting data sharing, (2) the prohibitive cost of expert annotation, with each image requiring meticulous labeling by specialists, and (3) heterogeneity in sample processing and imaging protocols across institutions, resulting in technical variability that complicates dataset standardization.

Most state-of-the-art models [10,11,12] used for histological image analysis are pre-trained on ImageNet due to the lack of large-scale annotated histopathological datasets. Although this approach has proven effective, the domain divergence between natural images in ImageNet and histopathological images may hinder the model’s ability to generalize effectively to medical image tasks. As shown in Figure 1, there is significant distributional divergence in Contrast, Homogeneity, Energy, and Correlation, between natural and histopathological images, posing challenges for model fine-tuning. Contrast measures local variations in pixel intensity within an image, mathematically represented as intensity differences between adjacent pixels in the Gray Level Co-occurrence Matrix (GLCM). High contrast values indicate significant intensity variations in tissue structures, typically reflecting abnormal cell density and tissue structure disruption in OSCC. Homogeneity measures the uniformity of image texture by evaluating the similarity of pixel pairs. Normal tissues typically exhibit higher homogeneity, while tumor tissues show lower homogeneity due to cellular atypia and irregular arrangements. Energy measures the orderliness and uniformity of texture, calculated as the sum of squared GLCM elements. High energy values indicate uniform texture or numerous repetitive patterns, typically higher in normal tissues than in tumor tissues. Correlation is a measure of linear dependencies in the GLCM, representing the degree of correlation between adjacent pixels. Correlation can help distinguish between normal organized structures (high correlation) and the disordered arrangement of tumor tissues (low correlation). As a result, there is a growing demand for pre-training strategies that are specifically designed to address the unique characteristics of histopathological images.

To reduce the required sample size and enable end-to-end adaptation to the target domain, recent approaches have proposed the self-supervised learning (SSL) paradigm, which enhances the performance of DNN models by directly learning visual features from target domain images, without the need for labels. Among the different self-supervised learning methods, contrastive learning algorithms [13,14,15] utilize a similarity-based metric to measure the distance between embeddings derived from different views of the same image. These views are generated through data augmentation techniques such as rotation, cropping, and shifting, while the embeddings are extracted from a DNN with learnable parameters. Specifically, in computer vision tasks, the contrastive loss is specifically computed using feature representations from the encoder network, which aids in clustering similar samples while separating dissimilar ones. Recent methods such as SwAV [16], SimCLR [13], MoCo [17], and PILR [18] have shown superior performance over traditional supervised learning methods on natural images.

Contrastive learning was initially proposed for natural images and has demonstrated promising results. In recent years, it has been gradually applied to various modalities of medical images, such as X-rays [19,20,21], MRI [22,23,24], and cellular images [25,26,27], proving its effectiveness.

In recent years, self-supervised learning methods have made significant progress in histopathological image analysis, with numerous innovative studies emerging [28,29,30,31,32,33]. These methods encompass various paradigms including contrastive learning, masked image modeling, and self-distillation, providing effective approaches to address the scarcity of annotated data. Our study aims to explore the applicability and optimization strategies of the MoCo framework in the specific domain of oral squamous cell carcinoma, complementing existing research.

In this work, we aim to extract more domain-specific features from histopathological images in an unsupervised manner, enhancing the model’s performance on downstream OSCC detection tasks. Our method involves pre-training a deep learning model using a self-supervised objective on a large set of histopathological images before fine-tuning on a smaller, labeled OSCC dataset.

The main contributions of this work are threefold:

1.

To address the distributional divergence between histopathological images and natural images, we developed HistoMoCo, a custom pre-training framework tailored for histopathological images. HistoMoCo aims to generate enhanced image representations and initialize models for OSCC detection in histopathological images.

2.

We provide comprehensive evaluations and analyses, highlighting the benefits of self-supervised learning for histopathological image analysis; demonstrating the proposed HistoMoCo provides high-quality representations and transferable initializations for histopathological images interpretation and OSCC detection.

3.

We release our code and pre-trained parameters for further research in histopathology or OSCC detection tasks.

The remainder of this manuscript is organized as follows: Section 2 reviews related work on self-supervised learning and the application of MoCo in medical imaging. Section 3 provides the necessary background on MoCo. Section 4 presents detailed specifications of our method. Section 5 outlines our experimental methodology and detailed experimental results, while Section 6 concludes the work.

2. Related Work

2.1. Detection of Oral Squamous Cell Carcinoma

Oral squamous cell carcinoma (OSCC) detection increasingly utilizes deep learning, particularly for pre-trained models based on large datasets like ImageNet. Several studies have demonstrated the effectiveness of convolutional neural networks (CNNs) in analyzing histopathology images for OSCC diagnosis. Redie et al. [12] evaluated ten pre-trained CNN models, finding that VGG-19 achieved a classification accuracy of 96.26% when combined with data augmentation. Kavyashree et al. [11] compared DenseNet architectures, reporting that DenseNet-201 reached an accuracy of 85.00%, while DenseNet-169 achieved a training accuracy of 98.96%. Mohan et al. [1] proposed OralNet, which utilized a four-stage approach for OSCC detection, achieving over 99.50% accuracy with histopathology images. These findings highlight the potential of pre-trained models for feature extraction and classification. However, most existing work [6,11,12] relies on models pre-trained on ImageNet, which may not fully capture the unique characteristics of histopathological images. Thus, there is a need for specialized pre-training methods tailored to histopathology to improve OSCC detection accuracy and enhance clinical outcomes.

2.2. Application of MoCo in Medical Image Analysis

Contrastive learning is one of the mainstream paradigms of self-supervised learning (SSL), aiming to learn consistent representations through the comparison of positive and negative sample pairs without requiring additional annotations. Given the tremendous potential of label-efficient learning [17,34], SSL has garnered significant attention in the field of medical image analysis [19,20,21,22,23,25,26]. Specifically, Liao et al. [20,21] integrated multi-task learning into the self-supervised paradigm of MoCo to enhance the model’s representational ability on X-ray images. Chaitanya et al. [22] proposed a strategy to extend the contrastive learning framework for segmenting 3D MRI images in semi-supervised settings by leveraging domain-specific and problem-specific prompts. Cao et al. [35] applied MoCo to enhance the performance of deep learning models in detecting rib fractures on CT images.

Recent advances have seen the emergence of foundation models for computational pathology. Chen et al. [30] introduced UNI, a general-purpose self-supervised model trained on over 100 million images from 100,000+ whole-slide images across 20 tissue types, demonstrating superior performance on 34 computational pathology tasks. Vorontsov et al. [32] presented Virchow, a 632-million parameter vision transformer model trained using DINOv2 on 1.5 million H&E stained slides, achieving state-of-the-art performance on pan-cancer detection and various benchmarks. Lu et al. [31] developed CONCH, a visual-language foundation model trained on diverse histopathology images and over 1.17 million image-caption pairs, excelling in tasks including classification, segmentation, and retrieval. Xiang et al. [33] introduced MUSK, a vision-language foundation model pretrained on 50 million pathology images and one billion text tokens using unified masked modeling, demonstrating strong performance across 23 benchmarks and various outcome prediction tasks. Our study aims to explore the applicability and optimization strategies of the MoCo framework in the specific domain of oral squamous cell carcinoma, complementing existing research.

3. Preliminaries

3.1. Contrastive Learning

Contrastive learning enables the self-supervised learning of image representations. Given a dataset $\mathcal{D}$, the objective is to find a mapping function $F:{\mathbb{R}}^d\mapsto {\mathbb{R}}^a$ where $a\ll d$, that satisfies the following condition:

$$s\left(F\left(x\right),F\left(x^{+}\right)\right)\gg s\left(F\left(x\right),F\left(x^{-}\right)\right)$$

(1)

where $s(·,·)$ measures image similarity, while F is responsible for both representation learning and dimensionality reduction. In this context, positive and negative samples are denoted as $x^{+}$ and $x^{-}$, respectively, where $x^{+}$ is similar to x and $x^{-}$ is dissimilar. The model learns representations by maximizing the agreement between different augmented views $x_q$ and $x_k$ of the same example x, utilizing a contrastive loss in the latent space. These augmented views $x_q$ and $x_k$ are generated through data augmentation techniques denoted as $\mathcal{DA}$.

3.2. Momentum Contrastive (MoCo) Learning

Momentum Contrastive (MoCo) is a state-of-the-art contrastive learning method that surpasses supervised pre-training on various downstream tasks. Unlike other contrastive learning approaches, MoCo does not require a large batch size or a memory bank. Instead, it utilizes a memory queue to store a set of previously computed representations. Furthermore, MoCo introduces a momentum encoder, which will be elaborated on shortly.

3.2.1. Dictionary as a Queue

MoCo trains an encoder to perform a dictionary lookup task, where a query q and encoded samples $x_1,\dots ,x_N$ act as the keys in the dictionary. A match occurs when the query q is similar to the positive sample $x^{+}$, while there is no match for the negative samples $x^{-}$. Specifically, MoCo utilizes two visual encoders, labeled $f_q$ and $f_k$, to learn query representations $q=f_q\left(x_q\right)$ and key representations $k=f_k\left(x_k\right)$. Here, $x_q$ denotes the query sample, and $x_k$ refers to the key sample. To allow the encoder to reuse previously encoded samples, MoCo employs the dictionary as a queue. The pre-trained model is trained using the following loss function:

$$\mathcal{L}=-log{\displaystyle \frac{exp\left(q,k^{+}\right)/\tau }{exp\left(q,k^{+}\right)/\tau +{\sum }_{k^{-}}exp\left(q,k^{-}\right)/\tau }}$$

(2)

where $\tau$ is a temperature hyperparameter, and $k^{+}$ and $k^{-}$ denote representations of positive and negative samples, respectively.

3.2.2. Momentum Update

MoCo employs a momentum update strategy for the parameters of the visual encoders. Let ${\theta }_q$ represent the parameters of $f_q$ and ${\theta }_k$ those of $f_k$. The parameters ${\theta }_q$ are updated through back-propagation using the contrastive loss $\mathcal{L}$ from Equation (2), and the parameters ${\theta }_k$ are updated using the following equation:

$${\theta }_k\leftarrow m{\theta }_k+(1-m){\theta }_q$$

(3)

where $m\in [0,1)$ is a momentum coefficient. This updating strategy ensures a smoother evolution of ${\theta }_k$ compared to ${\theta }_q$ [17].

4. Methods

4.1. Histopathological Datasets and OSCC Detection Tasks

In this study, we utilize four publicly available histopathological image datasets, including NCT-CRC-HE-100K [36], EBHI-Seg [37], NDB-UFES [38], and Oral Histopathology [39].

1.

NCT-CRC-HE-100K [36] is a large pathology dataset comprising 100,000 H&E-stained histological images of human colorectal cancer and healthy tissues, extracted from 86 patients. It covers nine tissue types, with each image sized at 224 × 224 pixels.

2.

EBHI-Seg [37] is a dataset for segmentation tasks, containing 2228 original H&E images and corresponding ground truth annotations, with each original H&E image also having a resolution of 224 × 224 pixels.

3.

NDB-UFES [38] dataset (OSCC detection dataset) presents a total of 237 samples with histopathological images and sociodemographic and clinical data. Its subset comprises 3763 image patches for downstream classification tasks, with each patch classified as either OSCC or Normal, and sized at 512 × 512 pixels.

4.

Oral Histopathology [39] (OSCC detection dataset) is composed of histopathological images of the normal epithelium of the oral cavity and images of OSCC. It includes two different magnifications, 100× and 400×, with a total of 290 samples classified as normal and 934 samples classified as OSCC, all with image sizes of 2048 × 1536.

4.2. HistoMoCo Pre-Training for Histopathological Interpretation

We applied MoCo pre-training to histopathological images. Figure 2a shows how data augmentation is used to generate views of a specific image, followed by contrastive learning to learn embeddings in an unsupervised manner.

We chose to use MoCo due to the limitations of large-scale computational resource costs in medical imaging AI. In comparison to other self-supervised frameworks such as SimCLR [13], MoCo requires significantly smaller batch sizes during pre-training [40]. The MoCo implementation used a batch size of 256 and achieved performance similar to SimCLR on ImageNet, while SimCLR struggled with a batch size of 256 [40]. MoCo mitigates the reliance on large batch sizes by utilizing a momentum-updated queue of previous samples to generate contrastive pair encodings.

We used 80% of the images from NCT-CRC-HE-100K [36] and the entire EBHI-Seg [37] datasets for MoCo pre-training. We chose to apply MoCo pre-training on models with ImageNet to leverage potential convergence advantages [41]. Due to the widespread availability of ImageNet pre-trained weights, initializing models with ImageNet weights before MoCo pre-training incurs no additional cost.

We update MoCo to be customized for histopathological images. Specifically, the data augmentation techniques commonly used in self-supervised learning for natural images may not be suitable for histopathological images. For example, color jittering and random grayscale can significantly alter the staining variations in histopathological images. Therefore, we disable these augmentations. Instead, we introduce multi-scale cropping to simulate tissue views at different magnification levels. In addition, the queue length was set to 2¹⁶ (larger than the standard MoCo setting), with a momentum update coefficient of 0.999 to better maintain representation consistency across different batches, which is particularly important for subtle feature differences in histopathological images. The detailed experimental parameter settings for HistoMoCo include a batch size of 256, a learning rate of 0.03, a feature dimension of 128, a queue size of $2^{16}$, and a MoCo momentum of 0.999.

We observed that using higher-resolution images significantly increases GPU memory consumption during model training. For example, when using ResNet-50 as the backbone with a batch size of 256, even when fine-tuning only the linear layer, images with a resolution of 512 × 512 require 6040 MB of memory, while those with a resolution of 1024 × 1024 demand 18,904 MB. This imposes a substantial computational burden [42]. Therefore, we downsampled the images to a resolution of 224 × 224 in both the pre-training and fine-tuning stages to account for computational resource constraints and ensure compatibility with the standard MoCo implementation.

Two encoder networks, ResNet-18 and ResNet-50 [43], were used to evaluate our consistency across model architectures. In downstream tasks, we evaluated the model performance in both Linear Tuning and End-to-End Tuning settings.

4.3. HistoMoCo Fine-Tuning

We perform fine-tuning on NCT-CRC-HE-100K [36] and two OSCC detection datasets, cluding NDB-UFES [38] and Oral Histopathology [39]. Specifically, we partition these three datasets, with the detailed division ratios presented in Figure 3. Notably, the 80% of NCT-CRC-HE-100K [36] used for fine-tuning is identical to the 80% previously used for pre-training. This setup ensures that the validation and test sets of NCT-CRC-HE-100K [36] remain unseen during the training process. Figure 3 shows the overall training process using HistoMoCo pre-training and subsequent fine-tuning.

We retained the model checkpoint with the best performance over 100 training epochs based on the AUROC metric (using macro-AUROC for NCT-CRC-HE-100K, trained for 10 epochs) on the validation set for evaluation on the test set. To evaluate the transferability of the representations, as shown in Figure 2b, we froze the encoder model and trained a linear classifier on top of it using labeled data (Linear Tuning). Additionally, we unfroze all layers and fine-tuned the entire model end-to-end using labeled data to assess the overall transferability of the performance (End-to-End Tuning). For Linear Tuning, a learning rate of 30 and a batch size of 256 were used, whereas for End-to-End Tuning, the learning rate was decreased to ${10}^{-2}$.

5. Experiments

5.1. Statistics of the Datasets

We use the NCT-CRC-HE-100K [36], EBHI-Seg [37], NDB-UFES [38], and Oral Histopathology [39] datasets for the pre-training, fine-tuning, and evaluation of HistoMoCo. The statistical information of these datasets is shown in

Table 1. Statistics of the histopathological datasets used in HistoMoCo.

Dataset	Resolution	Task	Patients	Images	Targets
Used for pre-training of HistoMoCo
NCT-CRC-HE-100K [36]	224 × 224	Classifaction	86	100,000	9
EBHI-Seg [37]	224 × 224	Segmentation	N/A	2228	6
Used for fine-tuning of HistoMoCo
NDB-UFES [38]	512 × 512	Classifaction	237	3763	2
Oral Histopathology [39]	2048 × 1536	Classifaction	N/A	1224	2

, along with their classification/segmentation targets as follows:

• NCT-CRC-HE-100K [36]: Nine tissue types: adipose, background, debris, lymphocytes, mucus, smooth muscle, normal colon mucosa, cancer-associated stroma, colorectal adenocarcinoma epithelium.
• EBHI-Seg [37]: Six tumor differentiation stages: normal, polyp, low-grade intraepithelial neoplasia, high-grade intraepithelial neoplasia, serrated adenoma, and adenocarcinoma.
• NDB-UFES [38]: Two types: oral squamous cell carcinoma, normal epithelium.
• Oral Histopathology [39]: Two types: oral squamous cell carcinoma, normal epithelium.

5.2. Experimental Setups

5.2.1. Baseline Models

We propose the following baselines for comparison:

• Scratch: models are initialized using Kaiming’s random initialization [44] and then fine-tuned on the target datasets.
• ImageNet: models are initialized with the officially released weights pre-trained on the ImageNet dataset and fine-tuned on the target datasets.
• MoCo: models are initialized with the officially released weights (https://github.com/facebookresearch/moco (accessed on 12 March 2025)) from [17] and fine-tuned on the target datasets.

5.2.2. Evaluation Metrics

To evaluate these algorithms, we conduct a comparison of their performance by assessing and contrasting Macro-AUROC (MaROC.), Micro-AUROC (MiROC.), Macro-AUPRC (MaPRC.), Micro-AUPRC (MiPRC.), and Accuracy (Acc.) for multi-class classification task (colorectal tissue classification on NCT-CRC-HE-100K [36]), AUROC (ROC.), AUPRC (PRC.), Accuracy (Acc.), Precision (Pre.), Sensitivity (Sen.), Specificity (Spe.) and F1 Score (F1) for binary classification task (OSCC detection on NDB-UFES [38] and Oral Histopathology [39]).

5.3. Experimental Results

5.3.1. Transfer Performance of HistoMoCo Representations

We investigated whether representations acquired through HistoMoCo pre-training are of higher quality than those transferred from Scratch, ImageNet, and MoCo [40]. To evaluate these representations, we used the Linear Tuning protocol [45,46], where a linear classifier is trained on a frozen base model, and test performance is used as a proxy for representation quality. We also explored whether HistoMoCo pre-training translates to higher model performance by conducting End-to-End Tuning. The results are shown in

Table 2. Model performance on colorectal tissue classification task of NCT-CRC-HE-100K dataset. In each experimental setup, the best-performing results are shown in bold, and the second-best results are underlined with underline. When using ResNet-50 as the backbone with End-to-End Tuning, HistoMoCo achieves a 100% Macro-AUROC and Micro-AUROC. This is due to rounding to three decimal places. In fact, under this setting, HistoMoCo achieves a Macro-AUROC of 0.999746 and a Micro-AUROC of 0.999557.

Encoder	Fine-Tuning	Pre-Training	MaROC.	MiROC.	MaPRC.	MiPRC.	Acc.
ResNet-18	Linear	Scratch	0.888	0.887	0.508	0.453	0.562
		ImageNet	0.986	0.982	0.920	0.895	0.919
		HistoMoCo	0.982	0.984	0.890	0.898	0.985
ResNet-18	End-to-End	Scratch	0.953	0.961	0.733	0.755	0.887
		ImageNet	0.997	0.998	0.978	0.986	0.987
		HistoMoCo	0.998	0.998	0.989	0.986	0.991
ResNet-50	Linear	Scratch	0.794	0.572	0.379	0.142	0.105
		ImageNet	0.992	0.987	0.951	0.926	0.912
		MoCo	0.994	0.995	0.963	0.968	0.965
		HistoMoCo	0.998	0.999	0.990	0.990	0.992
ResNet-50	End-to-End	Scratch	0.970	0.967	0.824	0.817	0.879
		ImageNet	0.999	0.999	0.993	0.995	0.991
		MoCo	0.998	0.999	0.991	0.992	0.986
		HistoMoCo	1.000	1.000	0.998	0.997	0.994

. It can be observed that with ResNet-50 as the architecture, although the performance of ResNet-50 models on the NCT-CRC-HE-100K dataset is close to saturation, HistoMoCo still shows significant improvements compared to ImageNet and MoCo, especially in terms of AUPRC and accuracy metrics. This indirectly demonstrates the stronger robustness of HistoMoCo pre-training parameters for imbalanced positive and negative samples. ResNet-18 also reflects similar patterns.

5.3.2. Transfer Benefit of HistoMoCo on OSCC Detection Task

We conducted experiments to test whether the HistoMoCo pre-trained histopathological image representations obtained from the colorectal dataset could be transferred to two OSCC detection datasets (NDB-UFES and Oral Histopathology datasets). Consistent with the previous section, the experiments were based on both Linear Tuning protocol and End-to-End Tuning settings, with results shown in

Table 3. Model performance on OSCC detection task of NDB-UFES dataset. In each experimental setup, the highest-performing results are highlighted in bold, while the second-best results are marked with an underline.

Encoder	Fine-Tuning	Pre-Training	ROC.	PRC.	Acc.	Pre.	Sen.	Spe.	F1
ResNet-18	Linear	Scratch	0.737	0.869	0.298	0.000	0.000	1.000	0.000
		ImageNet	0.917	0.962	0.855	0.874	0.927	0.684	0.900
		HistoMoCo	0.949	0.978	0.852	0.937	0.845	0.866	0.889
ResNet-18	End-to-End	Scratch	0.864	0.936	0.812	0.843	0.900	0.604	0.870
		ImageNet	0.983	0.993	0.928	0.938	0.961	0.850	0.949
		HistoMoCo	0.991	0.996	0.947	0.953	0.973	0.888	0.963
ResNet-50	Linear	Scratch	0.713	0.840	0.298	0.000	0.000	1.000	0.000
		ImageNet	0.951	0.974	0.887	0.928	0.909	0.834	0.918
		MoCo	0.952	0.979	0.893	0.919	0.930	0.807	0.924
		HistoMoCo	0.979	0.990	0.935	0.935	0.948	0.904	0.953
ResNet-50	End-to-End	Scratch	0.870	0.941	0.802	0.850	0.873	0.636	0.861
		ImageNet	0.993	0.997	0.957	0.966	0.973	0.920	0.969
		MoCo	0.991	0.996	0.951	0.957	0.973	0.898	0.965
		HistoMoCo	0.994	0.997	0.957	0.964	0.975	0.914	0.969

and

Table 4. Model performance on OSCC detection task of oral histopathology dataset. In each experimental setup, the best-performing results are highlighted in bold, while the second-best results are underlined with an underline.

Encoder	Fine-Tuning	Pre-Training	ROC.	PRC.	Acc.	Pre.	Sen.	Spe.	F1
ResNet-18	Linear	Scratch	0.728	0.888	0.738	0.869	0.768	0.645	0.816
		ImageNet	0.834	0.939	0.770	0.811	0.905	0.355	0.856
		HistoMoCo	0.855	0.954	0.794	0.805	0.958	0.290	0.875
ResNet-18	End-to-End	Scratch	0.862	0.956	0.802	0.812	0.958	0.323	0.879
		ImageNet	0.924	0.977	0.857	0.889	0.926	0.645	0.907
		HistoMoCo	0.948	0.984	0.865	0.906	0.916	0.710	0.911
ResNet-50	Linear	Scratch	0.721	0.876	0.246	0.000	0.000	1.000	0.000
		ImageNet	0.848	0.945	0.770	0.875	0.811	0.645	0.842
		MoCo	0.921	0.973	0.778	0.772	1.000	0.097	0.872
		HistoMoCo	0.915	0.969	0.810	0.832	0.937	0.419	0.881
ResNet-50	End-to-End	Scratch	0.859	0.956	0.786	0.827	0.905	0.419	0.864
		ImageNet	0.882	0.958	0.825	0.876	0.895	0.613	0.885
		MoCo	0.939	0.980	0.881	0.892	0.958	0.645	0.924
		HistoMoCo	0.941	0.982	0.873	0.883	0.958	0.613	0.919

. It can be observed that HistoMoCo achieved the overall best performance on both external datasets. Notably, the Linear Tuning showed significant improvements in key metrics such as AUROC and AUPRC compared to ImageNet pre-training. Under End-to-End Tuning, even ImageNet pre-training can achieve over 99% ROC and PRC on the OSCC detection task of the NDB-UFES dataset when using ResNet-50. This suggests that the simplicity of this dataset makes it easy to attain high performance, making it challenging to HistoMoCo. However, HistoMoCo still outperformed other baselines overall. Given the small data volume, we believe that the occasional deviation in individual metrics is acceptable. The results on external datasets confirm the generalizability of HistoMoCo to out-of-domain data from different tissue sites, suggesting that it can serve as a more robust pre-training foundation for subsequent histopathological image research. Figure 4 presents the ROC curves of ResNet-50 using various pre-training algorithms on two OSCC detection tasks, 95% confidence intervals were estimated using bootstrapping with 100 independent trials on the testing sets.

Notably, initializing the backbone model from scratch and performing Linear Tuning resulted in highly biased outcomes, with both precision and sensitivity being 0. This indicates that, under this setting, the model exhibited an extreme tendency to classify all images as normal. The primary reason for this is that the scratch initialization did not provide the backbone model with any features relevant to histopathological images. Consequently, the model failed to generate meaningful feature representations for the subsequent linear classification layer, leading to such extreme results.

5.3.3. Performance Comparison with Similar Literature

We further introduce UNI [30], CONCH [31], DINO [47], and SwAV [16] as baseline models to evaluate the effectiveness of HistoMoCo. Specifically, we utilized UNI [30] and CONCH [31] as feature extractors and performed Linear Tuning experiments on the OSCC detection task of the NDB-UFES dataset (the larger dataset among the two OSCC detection datasets we used to ensure the reliability of the results). The experimental results are shown in

Table 5. Model performance on OSCC detection task of NDB-UFES dataset using HistoMoCo and SOTA models. In each experimental setup, the highest-performing results are highlighted in bold, while the second-best results are marked with an underline.

Methods	Encoder	Param.	ROC.	PRC.	Acc.	Pre.	Sen.	Spe.	F1
Linear Tuning
UNI2-h [30]	ViT-h/14-reg8	630.76M	0.982	0.992	0.936	0.926	0.972	0.902	0.948
CONCH [31]	ViT-b/16	86.57M	0.976	0.984	0.930	0.925	0.963	0.902	0.944
DINO [47]	ResNet-50	25.56M	0.970	0.985	0.917	0.960	0.920	0.909	0.940
SwAV [16]	ResNet-50	25.56M	0.958	0.981	0.896	0.896	0.964	0.738	0.929
HistoMoCo	ResNet-50	25.56M	0.979	0.990	0.935	0.935	0.948	0.904	0.953
End-to-End Tuning
DINO [47]	ResNet-50	25.56M	0.975	0.988	0.925	0.970	0.930	0.920	0.940
SwAV [16]	ResNet-50	25.56M	0.962	0.986	0.905	0.940	0.925	0.850	0.930
HistoMoCo	ResNet-50	25.56M	0.994	0.997	0.957	0.964	0.975	0.914	0.969

. The experimental results demonstrate that both UNI2-h and CONCH achieved outstanding performance after extensive pre-training on large-scale histopathological image datasets. This further validates our hypothesis that pre-training should be conducted on data distributions consistent with the downstream task rather than on distributionally mismatched data, which could introduce biases. In comparison with UNI2-h, we openly acknowledge the performance limitations of HistoMoCo. UNI2-h achieved the best results in the majority of cases, including ROC, PRC, accuracy, and sensitivity. However, HistoMoCo remains noteworthy, as it achieved the best F1-score and maintained the second-best performance in most cases. More importantly, both UNI2-h and CONCH adopt the Vision Transformer [48] as model backbone, which, while effective, introduces a significant number of trainable parameters (630.76M for UNI2-h and 86.57M for CONCH). In contrast, HistoMoCo employs ResNet-50 [43] with only 25.56M parameters. Notably, when compared to CONCH, HistoMoCo achieved superior performance with fewer parameters. We believe that HistoMoCo strikes a balance between performance and computational efficiency, addressing an existing gap in current research.

We implemented DINO [47] and SwAV [16] methods and compared them under the same experimental settings as MoCo, performing Linear Tuning and End-to-End Tuning experiments on the OSCC detection task of the NDB-UFES dataset. The experimental results are shown in Table 5. The experimental results indicate that HistoMoCo outperforms these state-of-the-art (SOTA) models, which is consistent with our hypothesis. Since both DINO and SwAV use natural images for pre-training, the divergence between natural images and histopathological images hinders the generalization of these models to downstream tasks.

5.3.4. Ablation Study

We note that the pre-training dataset, NCT-CRC-HE-100K, consists of histopathological images of colorectal cancer, whereas our target task focuses on OSCC detection. However, OSCC and colorectal cancer exhibit significant anatomical and morphological differences. Therefore, we conduct an ablation study on NCT-CRC-HE-100K to evaluate its effectiveness during the pretraining stage. The experimental results, as presented in

Table 6. Ablation study of NCT-CRC-HE-100K in HistoMoCo. In each experimental setup, the better-performing results are highlighted in bold.

Methods	ROC.	PRC.	Acc.	Pre.	Sen.	Spe.	F1
Linear Tuning
HistoMoCow/o NCT-CRC-HE-100K	0.946	0.973	0.851	0.937	0.845	0.866	0.889
HistoMoCo	0.979	0.990	0.935	0.935	0.948	0.904	0.953
End-to-End Tuning
HistoMoCow/o NCT-CRC-HE-100K	0.983	0.990	0.945	0.953	0.973	0.888	0.963
HistoMoCo	0.994	0.997	0.957	0.964	0.975	0.914	0.969

, indicate that the removal of the NCT-CRC-HE-100K dataset leads to a significant performance drop in both Linear Tuning and End-to-End Tuning. We attribute this to HistoMoCo learning relevant histopathological representations from the NCT-CRC-HE-100K dataset, which are subsequently utilized for OSCC detection. Ideally, we believe that using only OSCC images for pre-training would yield the best results. However, due to the scarcity of such data, we resorted to colorectal cancer histopathological images as a related alternative to minimize the distribution gap between pre-training and fine-tuning datasets.

5.3.5. Sensitivity Analysis

We further analyze several hyperparameters in HistoMoCo, including the momentum update rate (moco-m), the queue size (moco-k), and projection head size (moco-dim), with the experimental results presented in

Table 7. Sensitivity analysis of the momentum update rate (moco-m) in HistoMoCo.

	ROC.	PRC.	Acc.	Pre.	Sen.	Spe.	F1
Linear Tuning
moco-m = 0.999	0.979	0.990	0.935	0.935	0.948	0.904	0.953
moco-m = 0.9	0.971	0.987	0.928	0.948	0.950	0.877	0.949
moco-m = 0.5	0.599	0.734	0.703	0.703	1.000	0.005	0.826
moco-m = 0.1	0.646	0.785	0.705	0.716	0.961	0.102	0.821
End-to-End Tuning
moco-m = 0.999	0.994	0.997	0.957	0.964	0.975	0.914	0.969
moco-m = 0.9	0.996	0.998	0.968	0.973	0.982	0.936	0.977
moco-m = 0.5	0.745	0.875	0.742	0.777	0.886	0.401	0.828
moco-m = 0.1	0.764	0.889	0.751	0.778	0.902	0.396	0.836

,

Table 8. Sensitivity analysis of the queue size (moco-k) in HistoMoCo.

	ROC.	PRC.	Acc.	Pre.	Sen.	Spe.	F1
Linear Tuning
moco-k = 216	0.979	0.990	0.935	0.935	0.948	0.904	0.953
moco-k = 214	0.983	0.993	0.951	0.979	0.950	0.952	0.964
moco-k = 212	0.977	0.989	0.930	0.956	0.943	0.898	0.950
moco-k = 210	0.982	0.992	0.933	0.959	0.945	0.904	0.952
End-to-End Tuning
moco-k = 216	0.994	0.997	0.957	0.964	0.975	0.914	0.969
moco-k = 214	0.994	0.998	0.962	0.964	0.982	0.914	0.973
moco-k = 212	0.994	0.998	0.96	0.973	0.97	0.936	0.972
moco-k = 210	0.994	0.998	0.965	0.973	0.977	0.936	0.975

and

Table 9. Sensitivity analysis of the projection head size (moco-dim) in HistoMoCo.

	ROC.	PRC.	Acc.	Pre.	Sen.	Spe.	F1
Linear Tuning
moco-dim = 64	0.981	0.992	0.936	0.957	0.952	0.898	0.954
moco-dim = 128	0.979	0.990	0.935	0.935	0.948	0.904	0.953
moco-dim = 256	0.981	0.992	0.936	0.959	0.950	0.904	0.954
End-to-End Tuning
moco-dim = 64	0.995	0.998	0.963	0.964	0.984	0.914	0.974
moco-dim = 128	0.994	0.997	0.957	0.964	0.975	0.914	0.969
moco-dim = 256	0.994	0.998	0.963	0.979	0.968	0.952	0.974

. In this experiment, we adhere to the controlled variable method, analyzing only a single hyperparameter in each experiment. When varying a specific hyperparameter, all other hyperparameters are set to their default values as specified in the methodology, i.e., momo-m = 0.999, moco-k = $2^{16}$, and moco-dim = 128 (see Section 4.2). The results indicate that the hyperparameters queue size and projection head size have minimal impact on the final model performance, particularly in the case of End-to-End Tuning. However, the momentum update rate proves to be a crucial parameter. Our experiments suggest that selecting a relatively large momentum update rate (e.g., 0.999) is essential for maintaining a robust MoCo pre-training process.

5.3.6. Robustness of HistoMoCo Against Data Insufficiency

We evaluate the robustness of HistoMoCo against insufficient training samples using the NDB-UFES Dataset with ResNet-18. To simulate data insufficiency, we reduce the image size of the training set from 80% to 50%/40%/30%/20%/10% of the entire dataset, while keeping the test set fixed for a fair comparison. Experiments are conducted across these different settings, and the performances of AUROC and AUPRC are plotted in Figure 5. As shown in Figure 5, HistoMoCo consistently outperforms the ImageNet pre-training solution in all settings. More importantly, with Linear Tuning, ImageNet achieved an AUROC of 89.32% using 40% of the training data, whereas HistoMoCo achieved an AUROC of 89.58% using only 10% of the training data. Similarly, with End-to-End Tuning, ImageNet reached an AUROC of 97.08% using 50% of the training data, while HistoMoCo attained an AUROC of 97.16% using only 40% of the training data. This further demonstrates that HistoMoCo enables deep learning models to achieve higher performance with less training data, reducing reliance on manually labeled data.

6. Conclusions

We introduce HistoMoCo, a method that provides high-quality representations and transferable initializations for interpreting histopathological images, tackling the challenge of distributional divergence between histopathological and natural images. Despite the substantial differences in data and task properties between natural image classification and histopathological interpretation, HistoMoCo successfully adapts the MoCo pre-training approach to histopathological images. This highlights the potential for self-supervised methods to extend beyond natural image classification contexts.

We validated the effectiveness of HistoMoCo across various histopathological images, using the pre-training dataset NCT-CRC-HE-100K and two oral OSCC datasets: Oral Histopathology and NDB-UFES. Extensive experiments demonstrated the model’s generalization capability. Notably, the images in the external oral OSCC datasets were not included in the unsupervised pre-training of HistoMoCo and differed in anatomical sites, as the pre-training images were sourced from the rectum and colon.