Whole-Slide Image Classification via Deep Feature Fusion and Unsupervised Conditional Domain Adaptation

Abstract

Deep learning models have received widespread attention in pathological image classification and recognition tasks. However, their performance relies on large amounts of annotated data, which are difficult to obtain for pathological images, severely limiting model generalization. Moreover, whole-slide images (WSIs) are extremely large and must be divided into patches for processing, which often leads to the loss of global information and degrades recognition performance. To address these issues, this paper proposes a cross-domain WSI classification and recognition method based on deep feature fusion and conditional domain alignment (CTCA). The method targets unsupervised domain adaptation scenarios. It constructs a parallel architecture of transfer-pretrained networks, BreNet and Swin Transformer, to jointly extract local detail and global contextual features, achieving multi-scale and multi-perspective deep feature fusion. Subsequently, labels are introduced as conditional variables in the latent space to perform conditional domain alignment, preserving category correlations while reducing distribution discrepancies between domains. Finally, lesion regions in WSIs are visually annotated using predicted probability heatmaps. Experiments show that, under an unsupervised setting, the method effectively leverages small-scale labeled data to guide lesion recognition in unlabeled WSIs, outperforming existing unsupervised domain adaptation methods in accuracy and stability and enabling visualization of regions of interest to support clinical diagnosis.

1. Introduction

According to global cancer statistics, cancer is one of the most serious health threats worldwide [1,2]. Pathological examination is a commonly used clinical diagnostic method and serves as the “gold standard” among all diagnostic tools. With advances in digital technology, digital pathological images have become an essential part of pathology research [3]. Digital pathological image processing enables various cellular image analysis tasks, such as nucleus and gland segmentation [4,5,6], lesion detection for lymph node metastasis [7,8,9], and mitosis counting [10]. The rich phenotypic information contained in pathological images is crucial for early clinical diagnosis, intervention, and patient survival prediction. With the wide use of WSI scanners in clinical practice, computer-aided pathological diagnosis (CAPD) has become an effective way to assist pathologists in reducing diagnostic errors and saving time [11,12].

With the application of deep learning (DL), significant progress has been made in medical image processing [13]. Among many classical deep learning models, convolutional neural networks (CNNs) can capture local structural features of images and automatically learn feature representations [14,15]. A two-stage framework combining deep learning and machine learning is proposed for multi-classification of breast digital pathology images [16]. In addition, self-attention mechanisms can capture global dependencies in large-scale image data. Particularly, Vision Transformers (ViTs) can learn more unified semantic information from images, which is critical for accurate cancer classification and segmentation [17,18,19]. A Swin Transformer-based multiple instance learning network (MIST) is applied to achieve an accurate classification of colorectal tumor WSIs using only WSI-level labels.

Vision Transformers are good at extracting cross-slide semantic information, while CNNs mainly focus on local edge features formed by neighboring pixels. Hybrid architectures combining CNNs and Transformers can leverage the strengths of both sides. It enables further feature exploration and multi-scale feature fusion for rich cellular edge and inter-tissue semantic information in histopathological images. This naturally becomes a key research direction for further optimization in CAPD applications.

Due to data privacy concerns, public pathological image and WSI datasets are rare. Moreover, small-sized pathological images are relatively easy to annotate precisely, while detailed annotation of WSIs is difficult. Limited annotated WSI datasets cannot meet the data demands of deep neural networks, leading to overfitting and poor generalization when trained directly.

To address these issues, transfer learning (TL) has gained increasing attention in pathological image analysis [20]. Unsupervised domain adaptation (UDA), a branch of transfer learning, aims to discover domain-invariant and task-specific representations from labeled source domain samples and unlabeled target domain samples [21,22,23]. It has been applied in pathology to reduce the annotation burden and support various downstream medical tasks. Multiple domain-specific feature extractors with MMD loss have been constructed for cross-domain pathological image classification [24]. An ensemble autoencoder composed of a marginal autoencoder and a convolutional autoencoder is designed to learn diverse cross-domain feature representations in an unsupervised manner [25]. However, data differences and domain shifts between source and target pathological images may limit the classification performance of the models if UDA is applied directly.

To address these challenges, this paper proposes an unsupervised cross-domain pathological image classification method based on deep feature fusion and conditional domain alignment. Considering the specificity of pathological images, the method first pretrains an improved deep convolutional network, BreNet [26], on ImageNet and then uses it in parallel with a Swin Transformer (CNN-SwinT) module to extract and fuse local edge and global semantic features. Subsequently, label information is used as a conditional variable, and a conditional covariance operator is introduced to measure cross-domain feature distribution differences, achieving domain alignment. This enables the transfer of useful information from small-sized labeled pathological images to unlabeled WSIs, facilitating the identification of cancerous regions in WSIs.

The main contributions of this paper are as follows:

(1) A parallel architecture combining BreNet and Swin Transformer is designed to extract and fuse global and local features, representing fine-grained edge and coarse-grained semantic information in pathological images.

(2) By minimizing both the conditional covariance discrepancy and maximum mean discrepancy (MMD) under conditional constraints between source and target domains, the method further discovers cancer-specific, domain-invariant representations to guide lesion identification in unlabeled target-domain WSIs.

(3) Cross-species and cross-domain experiments are conducted on WSI datasets. Results demonstrate that the proposed method is effective and superior compared to other state-of-the-art approaches.

2. Notations

This paper uses certain symbols to represent the cross-domain pathological image classification process. For an unsupervised image classification task with c classes, ${\left\{\right({\mathbf{x}}_j^s,y_i^s\left)\right\}}_{i=1}^{n_s}$ denotes the sample set from the source domain and its labels, and ${\left\{{\mathbf{x}}_j^t\right\}}_{j=1}^{n_t}$ denotes the unlabeled sample set from the target domain. These data are sampled from the source and target domains, respectively, where ${\mathbf{X}}_s=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{n_s}\}\in {\mathbb{R}}^{d\times n_s}$ and ${\mathrm{Y}}_s=\{{\mathrm{Y}}_1,\dots ,Y_{n_s}\}\in {\mathbb{R}}^{d\times n_s}$ represent the input samples and their slide-level labels from the source domain dataset; ${\mathbf{X}}_t=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_{n_t}\}\in {\mathbb{R}}^{d\times n_t}$ represents unlabeled samples from the target domain dataset. $d$ denotes the data dimension, and $n_s$ and $n_t$ denote the number of labeled source domain samples and unlabeled target domain samples, respectively. Given an unlabeled target domain sample ${\mathbf{x}}_j^t$, its output probability after the classification layer is defined as the generated pseudo-label ${\hat{y}}_j^t$. Then, each target domain sample is paired one-to-one with its corresponding pseudo-label to form $({\mathbf{x}}_j^t,{\hat{y}}_j^t)$.

3. Method

As shown in Figure 1, the model consists of four submodules: (a) Input: Pathological images from the source domain are cropped into 224 × 224 patches; WSIs from the target domain are segmented and randomly sampled into 224 × 224 patches containing only valid pathological tissue and cells. (b) CNN and Swin Transformer Feature Extraction and Fusion (CTFF) module: Images from the source and target domains are fed into a deep convolutional network, BreNet, and a Swin Transformer network, respectively, to extract features that capture classification-relevant information from different dimensions and perspectives. The features extracted by the two models are then fused with learned weights. (c) Conditional-Sample Domain Alignment (CSDA) module: Domain alignment is achieved in the reproducing kernel Hilbert space (RKHS) by aligning conditional covariance operators and distributions of features from the two domains, enabling alignment of high-dimensional complex features between source and target samples. (d) Cancerous Region Identification: The patch-level classification probabilities obtained by the classifier are mapped back to their original positions on the WSI, generating a final heatmap for lesion identification across the entire WSI.

Figure 1. Framework diagram of the unsupervised domain adaptation network with CNN-SwinT feature fusion and conditional domain alignment for cancer detection in pathological images.

3.1. CNN–SwinT Feature Extraction and Fusion

Pathological images are characterized by extremely high resolution and large size. Therefore, a complete pathological image (WSI) contains abundant intracellular and inter-tissue information from lesion regions. Due to limitations on input dimensions in deep learning models, pathological images must be preprocessed and split into smaller patches before being fed into the model. However, conventional feature extraction networks have limited receptive fields and focus more on local details, inevitably leading to the loss of some inter-tissue information during feature extraction.

To fully extract cancer-specific features from images and capture both fine-grained cellular information and coarse-grained inter-tissue information, the model uses not only the convolutional network BreNet for fine-grained feature extraction but also the Swin Transformer in parallel to capture coarse-grained features.

The BreNet is pre-trained on the ImageNet public dataset as the training set. An input sample $x_i$ with dimension 3 × 224 × 224 is processed through multiple convolutional layers to extract a high-dimensional feature of size 512 × 7 × 7. Then, to match the feature dimension from Swin Transformer, $F_{SwinT}$, global average pooling (GAP) is applied to this high-dimensional feature to obtain the final extracted feature $F_{BN}$. Thus, the process of extracting feature $F_{BN}$ from sample $x_i$ using the deep convolutional neural network can be described as follows:

$$F_{BN}=\mathrm{GAP}\left(s\left(x_i\right)\right)$$

(1)

where $s\left(x_i\right)$ denotes the operation of BreNet extracting features from sample $x_i$, and $\mathrm{GAP}\left(\cdot \right)$ denotes the global average pooling operation.

To capture more global and multi-scale features from the input sample, a Swin Transformer is introduced to process the input in parallel with the convolutional network. The network first divides the input sample into 4 × 4 patches and then maps them into corresponding feature sequence tokens, which are fed into the basic Swin Transformer block to finally extract the feature $F_{SwinT}$.

The output of Swin Transformer is processed with global average pooling (GAP), resulting in a final output feature $F_{BN}$ with the same channel dimension as $F_{SwinT}$, expressed as

$$F_{SwinT}=\mathrm{GAP}\left(\mathrm{SwinT}\left(x_i\right)\right)$$

(2)

Features $F_{BN}$ and $F_{SwinT}$ are extracted by the deep convolutional neural network BreNet and the Swin Transformer, respectively. Through weighted feature fusion, the final feature map $F_{CT}$ extracted from the original patch by this module is obtained:

$$F_{CT}={\alpha }_1{F}_{BN}+{\alpha }_2{F}_{SwinT}$$

(3)

where ${\alpha }_1>0$ and ${\alpha }_2>0$ are the weight parameters for features $F_{BN}$ and $F_{SwinT}$, respectively, which can be optimized during training via grid search.

3.2. Conditional-Sample Domain Alignment

Unsupervised domain adaptation methods assume that the labeled source domain and unlabeled target domain share the same feature space but have different feature distributions. Most UDA methods mainly align the two domains by minimizing the marginal distribution discrepancy between domain features or mitigating domain shift. Yet, they often ignore the potential loss of label information during alignment. Therefore, to preserve the distribution differences among samples of different classes within each domain, this module introduces the conditional covariance operator to measure how the relationships between samples change across domains under given conditions.

In the RKHS space, the conditional covariance operator can be viewed as a linear operator. By using a kernel function, the label $Y$ is embedded as a conditional variable into the high-dimensional RKHS space, allowing the model to capture complex nonlinear dependencies among samples while preserving label information. The Gaussian kernel function $\kappa \left(\cdot \right)$ is used as the explicit kernel function, with the following expression:

$$\kappa \left(x,y\right)=\exp \left(- {\displaystyle \frac{{‖x-y‖}^2}{2{\sigma }^2}}\right)$$

(4)

The features ${F_{CT}}^{s/t}$ extracted from the source and target domain samples, along with the pseudo labels $Y^{s/t}$ of the source and target domain datasets, are mapped into the RKHS. The corresponding implicit mapping matrices are defined as ${\Phi }_{s/\mathrm{t}}$ and ${\Psi }_{s/\mathrm{t}}$. Then, their covariance operator can be written as follows:

$$\left\{\begin{array}{l}{R}_{XY}^s={\displaystyle \frac{1}{n_s-1}}{\Phi }_s{H}_{n_s}{\Phi }_s^{\mathrm{T}}\\ R_{XY}^t={\displaystyle \frac{1}{n_t-1}}{\Psi }_t{H}_{n_t}{\Psi }_t^{\mathrm{T}}\end{array}\right.$$

(5)

where $H_n=I_n-1/n\cdot 1_n{1}_n^{\mathrm{T}}$ is the centering matrix of $n\times n$, which centers the $\Phi$ and $\Psi$ matrices to zero mean, eliminating the influence of mean values; $n_s$ and $n_t$ denote the number of labeled source domain features and unlabeled target domain features, respectively.

The conditional covariance operator $R_{\left.XX\right|Y}$ based on the conditional variable $Y$ is calculated according to the following equation:

$$R_{\left.XX\right|Y}=R_{XX}-R_{XY}{\left(R_{YY}+\epsilon I\right)}^{-1}{R}_{YX}$$

(6)

where $\epsilon$ is a regularization parameter to prevent $R_{YY}$ from being singular. The conditional covariance operators corresponding to the feature mappings of the source and target domains are computed as $R_{\left.XX\right|Y}^s$ and $R_{\left.XX\right|Y}^t$, respectively.

To measure the discrepancy in conditional covariance between the two domains in the high-dimensional RKHS space, the Wasserstein Distance [27,28] between the two conditional covariance operators, $R_{\left.XX\right|Y}^s$ and $R_{\left.XX\right|Y}^t$, is used as the loss ${\mathcal{L}}_{CDA}$. The second-order Wasserstein distance is suitable for measuring similarities in “shape” and “location” between two probability distributions. In RKHS, the geometric sensitivity of the Wasserstein distance is enhanced, enabling a more accurate reflection of the local geometric structure of the distributions. When two probability distributions are identical, their second-order Wasserstein distance is zero. Therefore, by minimizing the conditional domain alignment (CDA) loss function ${\mathcal{L}}_{CDA}$, feature alignment between the two domains can be achieved. The calculation is expressed as follows:

$$\begin{array}{cc}{\mathcal{L}}_{CDA}& ={\mathrm{W}}_2^2\left(R_{\left.XX\right|Y}^s,R_{\left.XX\right|Y}^t\right)\hfill \\ & =\mathrm{tr}\left(R_{\left.XX\right|Y}^s+R_{\left.XX\right|Y}^t-2R_{\left.XX\right|Y}^{st}\right)\hfill \end{array}$$

(7)

where ${\mathrm{W}}_2^2$ denotes the second-order Wasserstein distance between two variables, $R_{\left.XX\right|Y}^{st}=\sqrt{\sqrt{R_{\left.XX\right|Y}^s}{R}_{\left.XX\right|Y}^t\sqrt{R_{\left.XX\right|Y}^s}}$.

To further reduce the distribution discrepancy between source and target domain features in the RKHS space, the maximum mean discrepancy (MMD)-based loss function is used to align feature distributions across domains. The corresponding sample domain alignment (SDA) loss ${\mathcal{L}}_{SDA}$ is calculated as follows:

$$\begin{array}{cc}{\mathcal{L}}_{SDA}& ={\mathrm{MMD}}^2\left({\widehat{F}}^S,{\widehat{F}}^T\right)\hfill \\ & ={\displaystyle \frac{1}{n_s^2}}{\displaystyle \sum _{i=1}^{n_s}{\displaystyle \sum _{j=1}^{n_s}\kappa \left({\widehat{F}}_i^s,{\widehat{F}}_j^s\right)}}+{\displaystyle \frac{1}{n_t^2}}{\displaystyle \sum _{i=1}^{n_t}{\displaystyle \sum _{j=1}^{n_t}\kappa \left({\widehat{F}}_i^t,{\widehat{F}}_j^t\right)}}-{\displaystyle \frac{2}{n_s{n}_t}}{\displaystyle \sum _{i=1}^{n_s}{\displaystyle \sum _{j=1}^{n_t}\kappa \left({\widehat{F}}_i^s,{\widehat{F}}_j^t\right)}} \hfill \end{array}$$

(8)

Finally, the classifier output is normalized by a sigmoid function to obtain the binary classification prediction score ${\sigma }_i=\left\{{\sigma }_i^0,{\sigma }_i^1\right\}$ for the input patch sample $x_i$. For the source domain sample $x_i^s$, given the true label $y_i^s$ and the corresponding predicted score ${\sigma }_i^{y_i^s}$, the cross-entropy loss is used as the classifier loss, which can be expressed as follows:

$${\mathcal{L}}_{CLS}=-{\displaystyle \frac{1}{n_s}}{\displaystyle \sum _{i=1}^{n_s}[y_i^s\log {\sigma }_i^{y_i^s}+(} 1-{\mathrm{y}}_i\left)\log \right(1-{\sigma }_i)] y_i^s\in \left\{0,1,\dots ,C-1\right\}$$

(9)

For the target domain sample $x_j^t$, the maximum prediction score is taken as the predicted probability ${\sigma }_j^{{\widehat{y}}_j^t}$, and the predicted label ${\widehat{y}}_j^t$ is the model’s classification label of the image patch, expressed as

$$\left\{\begin{array}{l}{\sigma }_j^{{\widehat{y}}_j^t}=\max \left({\sigma }_j^0,{\sigma }_j^1\right)\\ {\widehat{y}}_j^t=\underset{c\in \left\{0,1\right\}}{\arg \max }\left({\sigma }_j^0,{\sigma }_j^1\right)\end{array}\right.$$

(10)

In summary, the overall loss function of the CTCA algorithm can be defined as follows:

$${\mathcal{L}}_{CTCA}={\mathcal{L}}_{CLS}+{\beta }_1{\mathcal{L}}_{CDA}+{\beta }_2{\mathcal{L}}_{SDA}$$

(11)

where ${\beta }_1$ and ${\beta }_2$ are the weighting parameters for Equation (7) and Equation (8), respectively.

The pseudocode of the CTCA algorithm is shown in Algorithm 1:

Algorithm 1: CTCA Algorithm

Input: Labeled source domain samples $\left\{X_s,Y_s\right\}$, Unlabeled target domain samples $\left\{X_t\right\}$

Output: F1-Score, Accuracy, AUC, Sensitivity, Specificity

Divide the source domain samples and target domain samples into training sets and testing datasets respectively

1. For training epochs do:

2. Extract features using BreNet network and perform max mean pooling to ob-tain features $F_{BN}$

3. Extract features using Swin Transformer network and perform max mean pooling to obtain features $F_{SwinT}$

4. Calculate the fused features $F_{CT}$ according to Equation (3)

5. Map the source domain and target domain features to the RKHS space according to kernel function (4)

6. Calculate the covariance operators of the source $R_{\left.XX\right|Y}^s$ and target $R_{\left.XX\right|Y}^t$ domain feature mappings according to Equation (6)

7. Calculate the second-order Wasserstein distance between $R_{\left.XX\right|Y}^s$ and $R_{\left.XX\right|Y}^t$ according to Equation (7), obtaining ${\mathcal{L}}_{CDA}$

8. Calculate the maximum mean discrepancy of cross-domain feature distribu-tions according to Equation (8), obtaining ${\mathcal{L}}_{SDA}$

9. Calculate ${\mathcal{L}}_{CLS}$ according to Equation (9)

10. Calculate the total model loss ${\mathcal{L}}_{CTCA}$ based on the weight ${\beta }_1$ and ${\beta }_2$ of each module’s loss function

11. Backpropagate and update model parameters

12. Calculate the slice prediction probability of target samples according to Equation (10)

13. End for

4. Experiments

4.1. Datasets and Evaluation Metrics

In this study, experiments were conducted using the small-sized breast cancer pathological image dataset BreaKHis_100× and the public breast cancer WSI dataset CAMELYON16. Additionally, a private small-sized mouse colon cancer pathological image dataset (denoted as Mice) and the public human colon cancer WSI dataset PAIP2020 were used as source and target domains, respectively, to evaluate the proposed model’s performance in identifying cancerous regions in WSI. Experiments on both cancer types were also carried out to validate the model’s generalization ability in the field of pathological diagnosis.

The BreaKHis dataset [29] is a public breast cancer pathological dataset. It contains 7909 breast pathological images, all stained with hematoxylin and eosin (H&E). In the experiments, images with 100× magnification (abbreviated as BreaKHis_100×) were selected to match the scale of the other datasets. The BreaKHis_100× subset contains 644 normal and 1437 cancerous images.

The CAMELYON16 dataset is a breast cancer pathological whole-slide image (WSI) dataset from the “Grand Challenge CAMELYON16” [30]. It contains H&E-stained slides, including 170 breast lymph node WSIs as the training set and 100 WSIs as the test set, used for breast cancer detection and classification. The dataset provides detailed manual annotations of cancerous regions in each WSI made by pathologists at the pixel level, serving as the ground truth (GT) or “gold standard” for breast cancer detection. To visually show the location of cancerous regions in WSIs, binary mask images are used in this study to visualize the GT. Figure 2 displays sample WSI thumbnails and their corresponding binary mask GT images. In the masks, white regions indicate cancerous areas in the original WSI, while black regions represent blank background and normal tissue areas.

Figure 2. Some breast cancer WSIs from the Camelyon16 dataset (top: WSI thumbnails; bottom: ground truth).

The mice were provided by Southwest Hospital. It contains a total of 9042 pathological images, all H&E-stained, with a resolution of 1024 × 768 and a magnification of 400×.

The public colon cancer pathological WSI dataset PAIP2020 comes from the “Grand Challenge PAIP2020” [31]. All images are randomly assigned to training, validation, and test sets. All WSIs are H&E-stained with a scanning magnification of 40×. The dataset includes 118 WSIs. Binary mask images are generated based on the GT from this dataset to visualize cancerous region detection. As shown in Figure 3, the top row displays sample WSI thumbnails from the PAIP2020 dataset, and the bottom row shows their corresponding GT binary mask images.

Figure 3. Some colorectal cancer WSIs from the PAIP2020 dataset (top: WSI thumbnails; bottom: ground truth).

Figure 4 shows several pathological slices from the above datasets after preprocessing.

Figure 4. Representative pathological patches in the datasets.

The links to the public datasets used in this paper are as follows:

BreaKHis: https://web.inf.ufpr.br/vri/databases/breast-cancer-histopathological-database-breakhis/ (accessed on 6 June 2022);

CAMELYON16: https://camelyon16.grand-challenge.org/ (accessed on 10 May 2023);

PAIP2020: https://paip2020.grand-challenge.org/ (accessed on 5 June 2022);

In quantitative evaluation, this study adopts widely used metrics, including accuracy (ACC), F1 score, sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve (AUC), to evaluate model performance. These metrics are defined as follows:

$$\left\{\begin{array}{l}ACC={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}\\ F1-Score={\displaystyle \frac{2\cdot TP}{2\cdot TP+FP+FN}}\\ Sensitivity={\displaystyle \frac{TP}{TP+FN}}\\ Specificity={\displaystyle \frac{TN}{TN+FP}}\end{array}\right.$$

(12)

where $TP$, $FP$, $TN$, and $FN$ are the numbers of samples predicted as true positive, false positive, true negative, and false negative, respectively.

4.2. Experimental Design and Implementation

We designed two sets of experiments to evaluate the performance of the proposed model and its component modules.

In the two experiments, we conducted domain adaptation from small-sized pathological images to WSI images on breast and colon cancer datasets. For breast cancer WSI detection, the BreaKHis dataset was used as the source domain, and the CAMELYON16 dataset was used as the target domain for cross-domain experiments. For colon cancer prediction, the Mice dataset and the PAIP2020 dataset were used as source and target domains, respectively, for cross-species experiments.

WSIs need to be cropped into patches before training. The image patch extraction process is as follows:

(1) Tissue region detection is performed using the HSV color space threshold method combined with Otsu’s automatic threshold segmentation, and only regions with saturation >0.1 and brightness <0.9 are retained as valid tissue.

(2) The filtering criterion is as follows: the proportion of tissue regions in a single 224 × 224 image patch must be ≥60%; otherwise, the patch is excluded.

(3) Full-coverage, non-overlapping sampling (step size = 224) is applied to small-sized source domain images. For target domain WSIs, 3000 valid patches are randomly sampled per WSI during training to balance computational efficiency and tissue coverage. During inference, sliding-window sampling with a step size of 224 is applied to all valid tissue regions in each WSI, ensuring complete spatial coverage for whole-slide heatmap generation.

(4) To enhance the generalization ability of the model, target domain WSIs are resampled in each training epoch. A fixed sampling sequence is used for the same WSI in the inference phase to ensure result consistency.

The proposed algorithm is compared with several state-of-the-art unsupervised domain adaptation (UDA) methods to evaluate its ability to predict cancerous regions in WSI patches. The compared methods include the following:

(1) CDICL [32]: constructs an information-aware space via instance contrastive learning for unsupervised domain adaptation;

(2) BACG [33]: uses a dual-discriminator adversarial network with category gradient alignment for unsupervised domain adaptation;

(3) RSDA [34]: proposes a reweighted adversarial training strategy to effectively reduce intra-domain discrepancies in the target domain, improving unsupervised domain adaptation;

(4) RCE [35]: introduces a novel risk-consistent training method for unsupervised domain adaptation.

This experiment quantitatively evaluates the patch-level classification performance of each method using evaluation metrics. Additionally, the predicted probability scores for each WSI patch are mapped back to their corresponding locations on the WSI to generate probability heatmaps for cancerous region prediction. Then, they are compared with those produced by the state-of-the-art methods.

The second experiment evaluates the effectiveness of the model’s component modules through ablation studies. The model is decomposed to separately validate the impact of the CNN-SwinT feature extraction and fusion module and the conditional-sample domain alignment module on model performance.

During training, the Adam optimizer was used. Hyperparameters were automatically optimized using the grid search method. The maximum number of training epochs and batch size were set to 50 and 24, respectively. The initial learning rate was 0.001 and was adaptively adjusted during training.

The CTCA algorithm and all comparison methods were implemented in Python 3.8 with PyTorch 2.0.1 on a system equipped with an Intel^® Xeon(R) CPU E5-2620 v4 @ 2.10 GHz×32 and a GeForce GTX 2080Ti GPU. The random seeds of Python, NumPy, and PyTorch were fixed in the experiments. Data shuffling, network weight initialization, and pseudo-label generation were all controlled by this unified random seed.

Hyperparameter Search Space:

Feature fusion weights ${\alpha }_1$, ${\alpha }_2$: grid search over [0, 1] (subject to ${\alpha }_1+{\alpha }_2=1$);

Loss weighting coefficients: ${\beta }_1$ (classification loss) ∈ [0.1, 1.5], ${\beta }_2$ (domain alignment loss) ∈ [0.1, 1.5];

Kernel bandwidth parameter $\sigma$ ∈ {0.1, 0.5, 1.0, 2.0};

Regularization coefficient $\lambda$ ∈ {1 × 10⁻⁴, 5 × 10⁻⁴, 1 × 10⁻³}.

All hyperparameters were tuned only on the source domain validation set, without using any target domain label information. Independent tuning was performed for different dataset pairs to adapt to cross-domain differences.

4.3. Experiment Results and Analysis

To compare the proposed method with recent state-of-the-art UDA methods in predicting cancerous regions in breast cancer pathological images, all methods were evaluated under the same dataset configuration over five independent runs. The mean and standard deviation of each evaluation metric across the five runs are recorded in Table 1 and Table 2.

Table 1. Detailed results of different compared methods for classification (BreaKHis100×-Camelyon16) (unit: %).

Algorithm	F1 Score (%)	Accuracy (%)	AUC (%)	Sensitivity (%)	Specificity (%)
CDICL	72.18 ± 5.67	70.21 ± 3.9	71.19 ± 1.19	73.69 ± 9.06	65.89 ± 1.79
BACG	78.39 ± 0.24	74.59 ± 0.03	71.53 ± 0.68	84.35 ± 1.39	63.11 ± 1.48
RSDA	79.01 ± 0.54	75.46 ± 0.25	73.85 ± 0.73	86.52 ± 1.39	64.57 ± 0.66
RCE	79.28 ± 0.09	75.34 ± 0.04	74.89 ± 0.19	89.45 ± 0.32	59.14 ± 0.15
CTCA	92.41 ± 0.39	89.87 ± 0.67	95.25 ± 1.20	92.30 ± 2.28	84.40 ± 4.30

Table 2. Detailed results of different compared methods for classification (Mice-PAIP2020) (unit: %).

Algorithm	F1 Score (%)	Accuracy (%)	AUC (%)	Sensitivity (%)	Specificity (%)
CDICL	69.89 ± 0.86	64.92 ± 0.69	75.65 ± 0.57	59.24 ± 1.18	78.52 ± 0.55
BACG	75.88 ± 0.05	72.02 ± 0.05	79.32 ± 0.19	68.11 ± 0.06	78.15 ± 0.21
RSDA	76.62 ± 0.75	71.68 ± 0.81	78.75 ± 0.64	69.67 ± 1.27	75.56 ± 0.03
RCE	74.21 ± 1.04	69.49 ± 0.88	76.27 ± 0.35	66.15 ± 1.89	76.15 ± 1.36
CTCA	90.36 ± 0.33	87.45 ± 0.63	92.28 ± 0.56	94.05 ± 1.56	76.28 ± 3.36

As shown in the tables, the proposed method significantly outperforms all other compared methods in cancer prediction on both breast cancer (BreaKHis100×-CAMELYON16) and colon cancer (Mice-PAIP2020) WSI. For the breast cancer detection experiment, Table 1 shows that the proposed method improves F1 score and accuracy by approximately 13% and AUC by approximately 20% compared to the second-best method, RCE. This demonstrates its superior overall performance. In addition, the specificity values of other methods are relatively low, indicating a weaker ability to identify normal samples, which would lead to higher false-positive rates in clinical applications.

For the colon cancer detection experiment, Table 2 shows that the proposed method outperforms all other methods, especially in F1 score, accuracy, and AUC, indicating stronger overall performance. The low sensitivity values of the other methods suggest weaker detection capability for cancerous samples, increasing the risk of missed detection of cancerous regions in clinical practice.

Overall, compared to other UDA methods, the proposed CTCA algorithm demonstrates stable performance across two different types of cancer detection tasks, showing stronger generalization ability and proving its potential for practical clinical applications. This indicates that the CTCA algorithm is not only suitable for detecting specific cancer types but can also be widely applied to the analysis of various pathological images, providing clinicians with more reliable diagnostic assistance.

Comparing the probability heatmaps with the GT binary mask images allows for a qualitative assessment of the model’s ability to identify cancerous regions in WSIs. As shown in Figure 5, for the task of identifying cancerous regions in CAMELYON16 WSIs, the proposed method detects lesion regions that are closest in shape and size to the GT with more accurate and detailed boundaries compared to other methods. This suggests that the method can assist clinicians in precisely identifying diseased areas and providing targeted diagnostic support, effectively improving diagnostic efficiency. In contrast, other methods tend to classify all tissue regions as cancerous, indicating that they fail to learn discriminative features for cancer classification during training and are therefore unsuitable for assisting diagnosis.

Figure 5. The comparison between the thumbnail, ground-truth binary maps, and probability heatmaps of different compared methods for representative WSIs from the Camelyon16 dataset.

It should be noted that the probability heatmaps presented here are generated by mapping patch-level (224 × 224) classification probabilities back to their original WSI coordinates. Although pixel-level ground-truth masks are available in the datasets, the heatmaps represent aggregated decisions at the patch level, rather than pixel-accurate segmentation. Consequently, the visual comparison with ground-truth masks serves primarily as qualitative evidence of spatial reasoning for image-level classification, rather than as a validated segmentation output. This design aligns with the core objective of our study—WSI-level diagnostic classification—with heatmaps functioning as interpretability aids to assist pathologists in understanding model decisions.

The cancer identification results for the colon cancer dataset PAIP2020 are shown in Figure 6. The probability heatmap generated by the proposed CTCA algorithm matches the cancerous regions in the GT most closely. For the first and fifth WSIs, where other methods show large prediction errors, CTCA successfully identifies the cancerous regions. In addition, the boundary of the labeled regions in the probability heatmap obtained by the proposed method aligns more closely with the GT mask boundary, and the method detects more accurate cancerous regions, indicating higher recognition accuracy and better compliance with the complex and high-standard requirements of clinical application.

Figure 6. The comparison between thumbnail, ground-truth binary maps, and probability heatmaps of different compared methods for representative WSIs from the PAIP2020 dataset.

4.4. Ablation Experiment Results and Analysis

This section evaluates the effectiveness of the model’s feature extraction and unsupervised domain adaptation components through ablation studies.

The ablation experiments aim to assess the contributions of the CNN-SwinT feature extraction and fusion module and the conditional-sample domain alignment module. The proposed CTCA framework is compared with the following variants:

(1) CTCA-Varient1: This variant removes the BreNet convolutional network in feature extraction and uses only the Swin Transformer to extract image features;

(2) CTCA-Varient2: This variant retains BreNet for feature extraction but removes the Swin Transformer component;

(3) CTCA-Varient3: This variant removes the conditional domain alignment operation but keeps the sample-level domain alignment;

(4) CTCA-Varient4: This variant retains the conditional domain alignment operation but removes the sample-level domain alignment.

Table 3 presents the submodule differences between the CTCA algorithm and the ablated variants. The classification results of the ablation study are shown in Table 4 and Table 5.

Table 3. Ablation algorithm settings for the estimation of each module.

Algorithm	BreNet	Swin Transformer	Conditional Domain Alignment	Sample-Level Domain Alignment
CTCA-Variant1	×	√	√	√
CTCA-Variant2	√	×	√	√
CTCA-Variant3	√	√	×	√
CTCA-Variant4	√	√	√	×
CTCA	√	√	√	√

“√” represents the inclusion of the component in the ablation algorithm, whereas “×” represents its removal.

Table 4. Ablation variant performance analysis (BreaKHis100×-Camelyon16) (unit: %).

Algorithm	F1 Score (%)	Accuracy (%)	AUC (%)	Sensitivity (%)	Specificity (%)
CTCA-Variant1	66.64 ± 0.03	49.97 ± 0.03	45.35 ± 7.36	100 ± 0.00	0.07 ± 0.07
CTCA-Variant2	83.11 ± 0.59	80.18 ± 1.13	93.96 ± 1.36	73.15 ± 1.05	94.25 ± 5.50
CTCA-Variant3	89.48 ± 3.85	85.75 ± 7.57	91.74 ± 2.98	90.87 ± 6.87	75.50 ± 2.15
CTCA-Variant4	86.89 ± 1.48	81.67 ± 3.15	91.95 ± 1.57	91.15 ± 3.17	62.70 ± 14.30
CTCA	92.41 ± 0.39	89.87 ± 0.67	95.25 ± 1.20	92.30 ± 2.28	84.40 ± 4.30

Table 5. Ablation variant performance analysis (Mice-PAIP2020) (unit: %).

Algorithm	F1 Score (%)	Accuracy (%)	AUC (%)	Sensitivity (%)	Specificity (%)
CTCA-Variant1	59.95 ± 16.97	43.44 ± 19.06	20.00 ± 9.37	67.71 ± 32.29	2.97 ± 2.97
CTCA-Variant2	89.45 ± 0.10	86.01 ± 0.24	84.19 ± 0.38	93.37 ± 2.28	80.85 ± 4.30
CTCA-Variant3	89.98 ± 0.95	86.76 ± 1.63	91.87 ± 0.24	95.17 ± 1.39	72.75 ± 0.34
CTCA-Variant4	83.34 ± 1.58	76.79 ± 1.34	74.30 ± 7.72	92.85 ± 4.85	50.00 ± 4.50
CTCA	90.36 ± 0.33	87.45 ± 0.63	92.28 ± 0.56	94.05 ± 1.56	76.28 ± 3.36

4.4.1. Analysis of Contributions of the Feature Extraction Module

Ablation of the feature extraction module focuses on the complementarity of BreNet and Swin Transformer. As shown in Table 4 and Table 5, Variant 1 (Swin Transformer only) achieves an F1 score of merely 66.64% in the breast cancer transfer task and performs significantly worse than other configurations in the colon cancer task. The model also tends to classify almost all samples as positive. This indicates that relying solely on the global attention mechanism makes it difficult to capture key local texture and cellular morphological features in pathological images. In contrast, Variant 2 (BreNet only) demonstrates robust performance in both tasks, with an F1 score of 83.11% for breast cancer and high accuracy maintained for colon cancer. It performs particularly well in terms of specificity, verifying the advantage of convolutional networks in extracting local fine-grained features. However, Variant 2 has relatively low sensitivity, suggesting limitations in its ability to identify complex tumor regions.

By performing weighted fusion of the features of BreNet and Swin Transformer, the complete CTCA framework further increases the F1 score of the breast cancer task to 92.41% (+9.30%) on the basis of Variant 2 and significantly improves the sensitivity–specificity balance (92.30% vs. 84.40%). This gain stems from the complementarity of the two networks: BreNet excels at capturing cell-level local abnormalities, while Swin Transformer models the tissue structure context of a larger receptive field through the hierarchical window attention mechanism. The fusion of the two effectively enhances the model’s ability to represent multi-scale pathological features.

4.4.2. Analysis of Contributions of the Domain Alignment Module

Ablation of the domain alignment module aims to verify the synergy of conditional alignment and sample alignment. Variant 3 (sample-level alignment only) achieves an F1 score of 89.48% in the breast cancer task. Though superior to the single feature extractor configurations, it has significantly higher sensitivity (90.87%) than specificity (75.50%). This indicates that aligning only the marginal distribution is prone to causing a shift in decision boundaries, and a similar imbalance is observed in cross-species transfer (Table 5). Variant 4 (conditional alignment only) shows the opposite trend: excessively high sensitivity with severely insufficient specificity (62.70%). This reflects that conditional distribution alignment, without global sample-level constraints, is likely to misclassify normal tissues as lesions and increase the risk of clinical misdiagnosis.

The complete CTCA integrates both conditional alignment and sample alignment, achieving optimal and balanced performance in both tasks (F1 = 92.41% for breast cancer, sensitivity = 92.30%/specificity = 84.40%). This result demonstrates that sample-level alignment provides a stable foundation for domain-invariant representations for the model by matching the overall feature distributions of the source and target domains; conditional alignment further refines the distribution alignment of samples from the same class and alleviates class conditional shift. The two form complementary constraints, jointly improving the robustness and clinical applicability of cross-domain classification.

4.4.3. Summary of Component Contributions

The comprehensive ablation results can be summarized as follows: (1) BreNet, as the base feature extractor, exerts a decisive effect on model performance, and its absence will lead to model failure. (2) The introduction of Swin Transformer leads to an F1 gain of approximately 9.3% through multi-scale feature complementarity. (3) Sample-level and conditional-level domain alignment have orthogonal contributions. Using either strategy alone will cause an imbalance between sensitivity and specificity, while their synergy can further boost the F1 score by about 2.9% and optimize the balance of clinical metrics. The above analysis shows that each component in the CTCA framework not only has independent value but also realizes cumulative performance improvement through synergy.

5. Discussion

In domain adaptation experiments from small-sized pathological images to WSI of breast and colon cancer, the proposed algorithm achieves classification accuracies of 89.87% and 87.45%, respectively. It significantly outperforms other unsupervised domain adaptation and ablation methods. The experimental results of all methods are shown in Figure 7 and Figure 8, demonstrating the algorithm’s strong performance and good generalization ability. In ablation studies, the CTCA algorithm performs better than other ablated variants in breast and colon cancer lesion prediction. This indicates that combining the BreNet and Swin Transformer effectively captures cancer-specific features at both local and global scales, enhancing the classifier’s ability to identify and classify image patches. The combination of conditional domain alignment and sample-level domain alignment effectively reduces domain shift and feature distribution differences between source and target domains, enabling the model to perform accurate unsupervised classification on target domain images. By mapping the predicted probabilities of the patches back to the original WSI, visual probability heatmaps are generated, showing the model’s ability to predict cancerous regions in WSIs and highlighting the potential of this proposed method for clinical application.

Figure 7. Accuracy comparison across algorithms (BreaKHis_100 × Camelyon16).

Figure 8. Accuracy comparison across algorithms (Mice-PAIP2020).

The p-value comparisons between CTCA and the benchmark and ablation methods, based on classification accuracy, are shown in Table 6 (BreaKHis100×-Camelyon16) and Table 7 (Mice-PAIP2020). These results show that CTCA achieves statistically significant performance improvements over the compared methods (p < 0.05), confirming its effectiveness in pathological image classification tasks.

Table 6. Statistical analysis of p-values (BreaKHis_100×Camelyon16).

Method	p-Value
CDICL	1.07 × 10−5
BACG	2.50 × 10−11
RSDA	7.98 × 10−11
RCE	3.63 × 10−11
CTCA-Variant1	1.14 × 10−14
CTCA-Variant2	1.84 × 10−7
CTCA-Variant3	4.48 × 10−2
CTCA-Variant4	4.57 × 10−4

Table 7. Statistical analysis of p-values (Mice-PAIP2020).

Method	p-Value
CDICL	1.89 × 10−11
BACG	9.92 × 10−12
RSDA	6.19 × 10−10
RCE	1.82 × 10−10
CTCA-Variant1	8.63 × 10−4
CTCA-Variant2	1.39 × 10−3
CTCA-Variant3	1.09 × 10−4
CTCA-Variant4	2.22 × 10−7

6. Conclusions

This paper proposes an unsupervised domain adaptation method for cancerous region identification in WSIs, named the CTCA algorithm. The CTCA algorithm uses small-sized pathological images as the source domain to extract cancer-specific information, which guides the model in performing unsupervised cancer prediction on target WSI patches. First, the model employs a parallel feature extraction and fusion module combining BreNet and Swin Transformer to extract sample features. This design integrates the CNN’s high sensitivity for fine details and the Transformer’s effective capture of global semantic information, enabling multi-scale and multi-perspective extraction of high-dimensional features followed by weighted fusion. Then, a conditional-sample domain alignment module maps the fused features into the RKHS space. By using label information as a conditioning variable, the method minimizes the second-order Wasserstein distance between conditional covariance operators across domains while also considering the maximum mean discrepancy between cross-domain features. This aligns the marginal distributions of the source and target domains while preserving intra-domain class distribution differences. The model then performs unsupervised cancer classification on both breast and colon cancer pathological images across domains or species. Compared with other UDA and ablation methods, CTCA achieves the best performance in detecting cancerous regions in WSI images, demonstrating its ability to effectively extract pathological features and transfer cancer-specific information from the source domain to enable cancer detection on the target domain WSI. Visualization of the predictions can assist pathologists in detecting cancerous regions, significantly improving diagnostic efficiency. Moreover, the model performs well in both breast and colon cancer prediction tasks across domains or species, further demonstrating its generalization ability and clinical application potential.

In an upcoming study, we aim to investigate a wider range of network architectures for feature extraction and integration. We also intend to apply our approach to additional datasets and evaluate its performance across diverse medical imaging modalities.