Breast Histopathological Image Classification Based on Auto-Encoder Reconstructed Domain Adaptation

Abstract

As an effective computer-aided diagnostic tool, deep learning has been successfully applied to the classification of breast histopathological images. However, the performance of the deep model is data-driven, and it is difficult to obtain satisfied results when the number of histopathological images is small and labelling histopathological images is difficult. Moreover, in traditional deep learning methods, the representation of features is monotonous, which leads to the limitation of the classification performance of the model. This study proposes an auto-encoder reconstructed semi-supervised domain adaptation for a breast histopathological image classification algorithm. First, the model was pre-trained and transferred to extract high-level features of the sample images. Then, the encoding and decoding parts of the auto-encoder were used to reconstruct the feature representation learning and the sample feature reconstruction learning, respectively. This ensured that the useful information for the classification was purified and retained. At the same time, the domain discriminator was used to confuse the source and target domain features to enhance the learning ability of the model. Finally, the distribution difference of features at different depths of the auto-encoder was measured to minimize the discrepancy of feature distribution between domains, so as to complete the classification of histopathological images. Compared to the results of the comparative and ablation algorithms from the BreakHis to SNL datasets, the proposed method achieved the best results in terms of F1 score (93.40%), accuracy (95.24%), sensitivity (94.66%), and specificity (95.56%). The experimental results demonstrate that the proposed method achieves a remarkable classification performance.

1. Introduction

Breast cancer is one of the most serious diseases threatening women’s health. Accurate diagnosis is crucial for patient survival. Among various diagnostic methods, clinical histopathological diagnosis serves as the gold standard for breast cancer detection. However, manual detection is not only time-consuming and labor-intensive, but also requires the experience of examiners. As an effective computer-aided diagnostic (CAD) tool, deep learning has been extensively applied to breast histopathological image classification tasks [1,2,3]. The key to address the contradiction between high detection demand and limited examiners lies in establishing efficient deep learning models.

Developing stable deep learning diagnostic algorithms faces two challenges. Firstly, deep models require a large amount of labeled data; otherwise, they may suffer from insufficient generalization ability and overfitting [4,5]. Labeling histopathological images requires an experienced pathologist, and the labeling process is extremely time-consuming. Developing deep models based on limited labeled data poses a significant challenge. Additionally, low-level features such as texture and shape are very useful for histopathological image classification. Hence, efficiently fusing multi-level features represents another challenge.

Our motivation is to address the two challenges faced by histopathological image classification models. A semi-supervised domain adaptation (DA) method is proposed to address the labelled data shortage issue, and on the other hand, an auto-encoder is applied to reconstruct image features, refining and preserving useful information for classification. We aim to enhance the performance of domain adaptation algorithms by reconstructing sample features.

This paper introduces an auto-encoder reconstructed semi-supervised domain adaptation algorithm for breast histopathological image classification. This algorithm employs a deep transfer network to obtain multi-level features of histopathological images. The samples from two domains are confused using similarity loss and domain discriminators in order to blur the boundaries between domains. A lightweight auto-encoder is created for feature reconstruction. The encoder not only compresses the extracted features and mines representation embedding, but also learns domain-invariant information. Additionally, the decoder uses reconstruction loss as a constraint for feature reconstruction, prompting the model to learn representations for classification tasks. During the feature extraction and reconstruction process, the difference in feature distribution between the source and target domains is measured multiple times to enhance the discriminative accuracy of the sample classifier.

The rest of this paper is structured as follows: In Section 2, we will specifically introduce previous related research works and the differences from the proposed method. In Section 3, the proposed method is presented in detail. In Section 4, the comparative experiments and ablation experiments are presented and analyzed. Finally, we conclude the paper in Section 5 and offer outlooks for future work.

2. Related Work

Deep learning, especially Convolutional Neural Network (CNN)-based CAD algorithms, has been extensively applied to histopathological image classification tasks. We categorize the related research works into three major classes. The first class focuses on lightweight feature extraction networks. The second class aims to extract more representative image features by modifying network structures. The third class proposes various semi-supervised and unsupervised domain adaptation methods to address the scarcity of data labels.

For the lightweight feature extraction networks, Alom et al. [6] proposed a IRRCNN model for breast histopathological image classification. By combining the advantages of an Inception Network, Residual Network, and Recurrent Convolutional Neural Network, the model achieved robust learning performance. Gandomkar et al. [7] introduced a deep learning framework called MuDeRN for breast histopathological image classification. This model integrates multi-layer feature information of a deep residual network, which helps the model capture more contextual information of histopathological images. Despite the powerful image feature extraction capability of CNNs, these large networks require large amounts of data and high computing power to support their performance. Therefore, improved lightweight CNNs have been proposed for histopathological image classification [8,9]. Li et al. [10] proposed a novel Deep Second-order Pooling Network (DSoPN) for breast cancer histopathological image classification, which improves the performance by introducing second-order statistical features. Kumar et al. [11] presented a lightweight CNN model named MobiHisNet, aiming to reduce the time required for histopathological diagnosis through mobile edge computing.

Apart from methods based on network structures, numerous domain adaptation algorithms have been applied to histopathological image classification [12,13,14]. Semi-supervised and unsupervised domain adaptations are more suitable for the CAD, and have attracted more attention from researchers. Since unsupervised DA methods lack the guidance of target domain labels, their experimental performance needs to be further improved compared with that of semi-supervised DA methods with a few labeled target samples [15,16]. Wang et al. [17] studied a multiple weighted semi-supervised DA model for whole slide image classification. Medela et al. [18] used a few-shot approach to reduce the need for labeled data for lung and breast histopathological image classification. Li et al. [19] proposed a covariance matching method for semi-supervised domain adaptation, which combines multiple kernel versions and shows significant performance in both isomorphic and heterogeneous case experiments.

Reconstructed domain adaptation can improve performance by reconstructing samples or features, which focus on common representations of features between domains and independent representations within a single domain [20]. The reconstruction of source and target domain samples or features is used as a secondary task. Different classes of auto-encoders are applied to learn the shared encoded representation between domains [21,22,23]. Moreover, Generative Adversarial Networks (GANs) are used to obtain circular maps, comparing the differences between the reconstructed image and the original image within the domain as the reconstruction loss [24,25,26]. GANs possess strong generative capabilities, but require specialized designs to effectively perform reconstruction tasks. While auto-encoders are particularly suited for reconstruction tasks as they directly restore input data by minimizing reconstruction errors [27], self-supervised learning methods [28] primarily focus on learning feature representations and typically do not directly optimize for data reconstruction performance. Moreover, the existing domain adaptation algorithms based on feature reconstruction are mostly unsupervised methods which lack the supervision of target domain labels, and their performance needs to be further improved. In addition, most of the DA methods based on auto-encoder reconstruction use image reconstruction as an auxiliary task of DA [29,30]. Due to the different sizes of input images, the capacity of the auto-encoder is difficult to control. If the capacity of the auto-encoder is too large, the auto-encoder will be biased towards learning to perform the replication task without extracting useful information about the data distributions. The training will then be incomplete, and it will be difficult to extract representative information for the recognition of histopathological images.

Regarding the aforementioned research work, a table has been compiled to summarize the models, datasets, and domain adaptation methods (

Table 1. Summary of related research work.

Author	Dataset	Evaluation Indicators	Model	Domain Adaptation Method
Alom et al. [6]	BreakHis	Accuracy	IRRCNN	None
Gandomkar et al. [7]	Mitosis-Atypia database	Accuracy	Based on ResNet	None
Li et al. [10]	BreakHis	IRR, PRR	DSoPN	None
Kumar et al. [11]	BreakHis	Accuracy	MobiHisNet	None
Wang et al. [17]	DigestPath 2019	Accuracy, sensitivity, specificity,	HisNet	Semi-supervised domain adaptation based on manifold regularization
Medela et al. [18]	UMCM	Accuracy, sensitivity, specificity	Siamese Neural Network	Semi-supervised domain adaptation based on class distance metric

).

The main contributions of this paper are: (1) a novel network is pre-trained and transferred to extract high-level features from histopathological images; (2) an auto-encoder is designed to obtain the reconstructed representation of extracted features to purify and retain useful information for classification; (3) similarity loss and domain discriminators are proposed to confuse source and target domain features to enhance the learning ability of the model.

3. Method

3.1. Notations

In this paper, subscripts S and T represent the source and target domain, respectively. The source domain training dataset and its label are represented as $X_S=[x_1,\dots ,x_{n_S}]\in {\Re }^{e\times n_S}$ and $Y_S=[y_1,\dots ,y_{n_S}]\in {\{0,1\}}^{e\times n_S}$, respectively. $X_T=[x_1,\dots ,x_{n_T}]\in {\Re }^{e\times n_T}$ denotes the training dataset of the target domain, which is divided into labeled part and unlabeled part, represented as $X_{Tl}=[x_1,\dots ,x_{n_{Tl}}]\in {\Re }^{e\times n_{Tl}}$ and $X_{Tu}=[x_1,\dots ,x_{n_{Tu}}]\in {\Re }^{e\times n_{Tu}}$, respectively. The corresponding label of $X_{Tl}$ is $Y_{Tl}=[y_1,\dots ,y_{Tl}]\in {\{0,1\}}^{e\times n_{Tl}}$. $\left\{x_S,x_T\right\}$, $\left\{x_{CS},x_{CT}\right\}$, and $\left\{{\widehat{x}}_S,{\widehat{x}}_T\right\}$ denote the high-level extracted features of the source and target domains, the latent features of the auto-encoder, and the reconstructed features of the auto-encoder. $e$ is the dimension of the features, while $n_S$, $n_{Tl}$, and $n_{Tu}$ are the number of labeled source samples, labeled target samples, and the unlabeled target samples, respectively. $\theta$ represents the parameter matrix of the model.

For ease of understanding, we have compiled a table listing the symbols and their meanings that appear in the article for reference (

Table 2. Symbol description.

Symbol	Meaning
$(X_S,Y_S)$	Annotated source domain dataset
$(X_{Tl},Y_{Tl})$	Annotated target domain dataset
$(X_{Tu},Y_{Tu})$	Unannotated target domain dataset
$X_i^n$	Intermediate layer output of neural networks
$f\left(•\right)$	Activation function of neural networks
${\mathcal{L}}_m$	Total loss of the algorithm
${\mathcal{L}}_{task}$	Loss for classification task
${\mathcal{L}}_{recon}$	Loss for reconstruction task
${\mathcal{L}}_{similarity}$	Domain similarity loss
${\mathcal{L}}_{MMD}$	Domain adaptive feature alignment loss
$\alpha ,\beta ,\gamma$	Hyperparameters of the network

).

3.2. Framework

Figure 1 shows that the proposed deep learning model consists of two stages: (1) a deep Convolutional Neural Network (CNN) is pre-trained and transferred to obtain high-level features of the samples, and (2) an auto-encoder reconstructed semi-supervised DA is applied for histopathological image classification. In the feature extraction part, as previously described, the CNN network is pre-trained and transferred to the domain adaptation model for high-level feature extraction. In the auto-encoder reconstructed semi-supervised DA part, the source domain and the target domain share an auto-encoder for domain adaptation training based on feature reconstruction. Firstly, the extracted features are fed into the auto-encoders. In encoding, the encoders perform representative learning and feature reconstruction for the high-level features to obtain a domain-invariant feature representation. In decoding, the decoder is used for feature recovery and reconstruction. The reconstruction is guided by the reconstructed loss to learn feature representation that is instructive for sample prediction. The domain adaptation learning and patch-level sample classification are completed after aligning high-level features, latent features, and reconstructed features. A domain discriminator is used to predict the domain labels using latent features of the source and target domains. It can improve the network’s differentiation capability by blurring the domain edge based on similarity. The image-level prediction probabilities are obtained by aggregating the patch-level classification result.

3.2.1. Deep Feature Extraction

The network contains five convolution structures to transfer knowledge from the pre-trained dataset. In addition, there is a classifier containing three fully connected layers to complete the pre-trained classification task. Each convolution structure contains two 3 $\times$ 3 convolutional kernels, two ReLU activating layers, and a pooling layer. The classifier consists of fully connected layers with 2048, 1024, and 1000 neurons and a Softmax classification layer. After the sample is fed into the network, the extracted features in the $n\mathrm{th}$ layer obtained can be described as:

$${\mathbf{x}}_i^n=f\left(W^n\ast {\mathbf{x}}_i^{n-1}+{\mathbf{b}}^n\right)$$

(1)

where ${\mathbf{x}}_i^n$ and ${\mathbf{x}}_i^{n-1}$ denote the output and input of the $n\mathrm{th}$ convolutional layer. $f\left(·\right)$ denotes the nonlinear activation function, which is used to increase the nonlinear properties of the model. $W^n$ and ${\mathbf{b}}^n$ denote the weight matrix and bias matrix of the $n\mathrm{th}$ convolutional layer, respectively.

3.2.2. High-Level Feature Reconstruction and Alignment

A fully connected layer-based auto-encoder is designed to reconstruct high-level features. It contains encoder and decoder parts. The encoder is used to compress and learn the feature embedding of high-level features of the histopathological images. The latent feature of the encoder is reconstructed by the decoder, so that the feature vector contains the meaningful representation of the breast histopathological images. It improves the learning efficiency of the auto-encoder for the high-level features. Since the proposed auto-encoder reconstructs the extracted features rather than the whole image, the auto-encoder is lighter, more controllable in capacity, and substantially improved in convergence. It reduces the model complexity and training difficulty. The encoder consists of three fully connected layers of 512, 256, 128 neurons. The decoder is built with three fully connected layers of 128, 256, 512 neurons.

High-level features are extracted by a deep learning model and fed into a fully connected auto-encoder to learn the mapping from the original space to the latent space. The distributions of the extracted features, latent features, and reconstructed features are aligned and the differences are minimized, which enables adaptive classification of breast histopathological images. The objective of the training is to minimize the loss as:

$${\mathcal{L}}_m={\mathcal{L}}_{task}+\alpha {\mathcal{L}}_{recon}+\beta {\mathcal{L}}_{similarity}+\gamma {\mathcal{L}}_{MMD}$$

(2)

where $\alpha$, $\beta$, and $\gamma$ are weights that control the interaction of the loss function, ${\mathcal{L}}_{task}$ is the cross-entropy loss, ${\mathcal{L}}_{recon}$ is the reconstruct loss, ${\mathcal{L}}_{similarity}$ is the similarity loss, and ${\mathcal{L}}_{MMD}$ is the domain alignment loss.

The cross-entropy loss ${\mathcal{L}}_{task}$ not only learns the histopathological image classification information carried by the labels adequately, but also ensures that the domain-distinguishing features carried by the entire data from the source domain and the labeled data from the target domain are learned effectively. The cross-entropy loss function can be expressed as:

$${\mathcal{L}}_{task}={\mathcal{L}}_{task\_s}+{\mathcal{L}}_{task\_t}$$

(3)

$${\mathcal{L}}_{task\_s}=-{\displaystyle \sum _{i=1}^{n_S}{\displaystyle \sum _{m=1}^c{y}_{S_i}^m\log (p(y_{S_i}|x_{S_i}^m))}}$$

(4)

$${\mathcal{L}}_{task\_t}=-{\displaystyle \sum _{i=1}^{n_l}{\displaystyle \sum _{m=1}^c{y}_{l_i}^m\log (p(y_{l_i}|x_{l_i}^m))}}$$

(5)

where $n_S$ and $n_l$ denote the number of labeled samples in the source and target domains, $c$ denotes the number of the patch categories, $y_{S_i}^m$ and $y_{l_i}^m$ denote the true labels of the samples $i$ in both domains, $x_{S_i}^m$ and $x_{l_i}^m$ denote the model inputs for the labeled samples $i$ in the source and target domains, and $p(y_{S_i}|x_{S_i}^m)$ and $p(y_{l_i}|x_{l_i}^m)$ denote the probability labels obtained at the last layer of the sample classifier.

The auto-encoder uses a nonlinear function to map high-level features from the input layer to the hidden layer to perform the coding operation:

$$x_C=f(W_{au}\ast x+b_{au})$$

(6)

where $x_C$ denotes the latent feature output (which is also used as the input of both domain discriminator and sample classifier), $W_{au}$ denotes the encoding weight matrix, and $b_{au}$ denotes the encoding bias vector. Subsequently, the latent features are reconstructed into high-level reconstructed features by the decoding operation:

$$\widehat{x}=g(x_C)=g({W^{\prime }}_{au}\ast x+{b^{\prime }}_{au})$$

(7)

where $\widehat{x}$ denotes the high-level reconstructed features, and ${W^{\prime }}_{au}$ and ${b^{\prime }}_{au}$ denote the decoding weight matrix and bias vector. The mean square error (MSE) is used as a loss function to calculate the reconstruction loss as a penalty to the auto-encoder. The reconstruction loss is described as:

$${\mathcal{L}}_{recon}={\displaystyle \sum _{i=1}^{n_S}{\mathcal{L}}_{mse}(x_{S_i},{\widehat{x}}_{S_i}})+{\displaystyle \sum _{i=1}^{n_T}{\mathcal{L}}_{mse}(x_{T_i},{\widehat{x}}_{T_i}})$$

(8)

$${\mathcal{L}}_{mse}(x,\widehat{x})={\displaystyle \frac{1}{k}}{‖x-\widehat{x}‖}_2^2$$

(9)

where $\left\{x_{S_i},x_{T_i}\right\}$ denotes the extracted features of the samples in the source and target domain, $\left\{{\widehat{x}}_{S_i},{\widehat{x}}_{T_i}\right\}$ denotes the high-level reconstructed features of the samples in the source and target domain, and k denotes the extracted feature parameter quantities.

The domain adaptation feature space is then constructed based on the Maximum Mean Discrepancy (MMD) criterion. MMD is a non-parametric method used to map features to a Reproducing Kernel Hilbert Space (RKHS) to enhance the inter-domain transferability of features. Then mean-embedding matching is performed, which is able to minimize the difference of feature distributions between the source and target domain and preserve discriminative information to construct the feature space. The alignment of extracted features with the MMD criterion can be expressed as:

$$\begin{array}{l}MM{D}^2(x_S,x_T)={\displaystyle \frac{1}{n_S^2}}{\displaystyle \sum _{i=1}^{n_S}{\displaystyle \sum _{j=1}^{n_S}\kappa (x_{S_i},x_{S_j})}}-2n_S{n}_T{\displaystyle \sum _{i=1}^{n_S}{\displaystyle \sum _{j=1}^{n_T}\kappa (x_{S_i},x_{T_j})}}\\ +{\displaystyle \frac{1}{n_T^2}}{\displaystyle \sum _{i=1}^{n_T}{\displaystyle \sum _{j=1}^{n_T}\kappa (x_{T_i},x_{T_j})}}\\ s.t.\hspace{1em}\kappa (x,y)=\exp \left(-{\displaystyle \frac{{‖x-y‖}^2}{\sigma }}\right)\end{array}$$

(10)

where $\left\{x_S,x_T\right\}$ denotes the high-level extracted features sets of the source and target domain, $\left\{x_{S_i},x_{S_j}\right\}$ and $\left\{x_{T_i},x_{T_j}\right\}$ denote the extracted features in the source and target domains, respectively, and $\kappa$ is the Gaussian kernel function. The decoding reconstructed features $\left\{{\widehat{x}}_S,{\widehat{x}}_T\right\}$ and the encoding latent features $\left\{x_{SC},x_{TC}\right\}$ are carried out for feature distribution alignment and for measuring differences. So, the domain adaptive feature alignment loss function can be expressed as:

$${\mathcal{L}}_{MMD}=MM{D}^2(x_S,x_T)+MM{D}^2(x_{SC},x_{TC})+MM{D}^2({\widehat{x}}_S,{\widehat{x}}_T)$$

(11)

3.2.3. Similarity Loss

This study predicts the domain labels of the latent features generated by the encoder after feeding the extracted features into the auto-encoder. In addition, the model confuses the subsequent decoder and sample classifier to discriminate the features, in order to improve the domain similarity of the features between domains and enhance the model’s ability to learn different classification information between domains. Specifically, a Gradient Reversal Layer (GRL) and a domain discriminator are used to maximize this confusion and improve the overall adversarial learning ability of the model. In particular, the GRL is able to maintain an identical transformation in the forward propagation, but automatically inverts the gradient in the backward propagation. Formally, for a function $f(\mathbf{u})$, the GRL can be defined as $Q(f(\mathbf{u}))=f(\mathbf{u})$, and its gradient is defined as:

$${\displaystyle \frac{d}{d\mathbf{u}}}Q(f(\mathbf{u}))=-{\displaystyle \frac{d}{d\mathbf{u}}}f(\mathbf{u})$$

(12)

Essentially, this study maximizes the binary cross-entropy for the label prediction task in the source and target domain, denoted as:

$${\mathcal{L}}_{similarity}={\displaystyle \sum _{i=0}^{n_S+n_T}\{d_i\log {\widehat{d}}_i+(1-d_i)\log (1-{\widehat{d}}_i)\}}$$

(13)

where $n_T$ denotes the number of samples in target domain and $d_i\in \left\{0,1\right\}$ denotes the true label of the domain category, while label 0 represents the source domain and 1 represents the target domain. ${\widehat{d}}_i\in \left\{0,1\right\}$ denotes the domain label of the samples predicted by the domain classifier.

The pseudo code of the proposed auto-encoder reconstructed semi-supervised DA algorithm is shown in Algorithm 1.

Algorithm 1: AER-SSDA Algorithm

Input: BreakHis dataset, SNL dataset

1. Initialize loss function weights: $\alpha$, $\beta$ and $\gamma$.

2. Split the samples and divide into training and test sets.

3. Build feature extractor, auto-encoder and classifier.

4. Pre-train the feature extractor with ImageNet dataset.

5. Transfer the feature extractor and corresponding parameters.

6. Reconstruct the model and fine-tune it with the training dataset.

7. WHILE training epochs do:

8.   Feed the extracted features into encoder to obtain latent features.

9.   Feed the latent features into decoder to reconstruct high-level reconstructed features.

10.    $\mathrm{Compute} \mathrm{the} \mathrm{domain} \mathrm{alignment} \mathrm{loss} {\mathcal{L}}_{recon}$ according to Equations (8) and (9).

11.    $\mathrm{Compute} \mathrm{the} \mathrm{reconstruction} \mathrm{loss} {\mathcal{L}}_{\mathrm{MMD}}$ according to Equations (10) and (11).

12.    Feed the latent features into domain discriminator.

13.    $\mathrm{Compute} \mathrm{the} \mathrm{similarity} \mathrm{loss} {\mathcal{L}}_{similarity}$ according to Equation (13).

14.    Feed the latent features into sample classifier.

15.    $\mathrm{Compute} \mathrm{the} \mathrm{cross}-\mathrm{entropy} \mathrm{loss} {\mathcal{L}}_{task}$ according to Equations (3)–(5).

16.    Backpropagate with the objective function and update the parameters of the network.

17.  END WHILE.

Output: F1 score, accuracy, sensitivity and specificity of the test set

4. Experiments

4.1. Dataset and Preprocessing

The datasets used in the AER-SSDA experiment contained a publicly available breast histopathological image dataset (BreakHis) and a private breast cancer dataset (SNL). The BreakHis dataset consisted of 7909 breast histopathological images from 82 patients [31]. The H&E-stained slides were collected at four magnification scales (40×, 100×, 200×, 400×) and the 200× images were used for this study. The private dataset was obtained from normal, uninvolved (cytopathological normal cells from malignant patients), and malignant breast tissue stained by H&E. The dataset contained 134 breast histopathological images with a resolution of 512 × 512, including 87 of normal cells, 23 of malignant cells, and 24 of cytopathological normal cells. For the experiment, the three types of histopathological samples were combined into two categories, namely 89 normal images and 47 malignant images. The representative images of the dataset samples are shown in Figure 2.

All of the BreakHis samples were cut into non-overlapping samples with a resolution of 230 $\times$ 230 during the dataset preprocessing. SNL dataset samples were divided into overlapping samples with a resolution of 230 $\times$ 230 at a step of 40. The number of samples before and after cutting is shown in

Table 3. The number of images of datasets.

Dataset	Class	Total Number	Patches	Train	Test
BreakHis	Malignant	1390	8340	5838	2502
	Benign	623	3738	2616	1122
SNL	Malignant	47	3008	2048	960
	Benign	87	5568	3840	1728

. Before input to the network for training, all samples were normalized as well as scaled and cut into 224 $\times$ 224 pixels. To meet the requirements of semi-supervised domain adaption training, all target domain samples were divided into training and test sets by 7 to 3. Most of the target domain samples without labels and a small portion of samples with labels were fed into the network for training with a labeled ratio of $\eta$. For the target domain dataset, 70% of the histopathological images were randomly selected as the training set and the remaining 30% as the test set. After obtaining the classification results of the patches, the decision fusion was performed to obtain the image-level classification prediction labels.

4.2. Experiment Environment and Parameter Setting

This experiment was operated in Ubuntu 18.06, and implemented on a workstation equipped with GeForce GTX 3080ti 32GB GPU and Intel(R) Core (TM) i7-8700 @3.2 GHz CPU. The program was implemented using Python 3.6 and PyTorch 1.7.0. Following previous work [17], the batch size was 32, the training epoch was set to 30, and the learning rate was set to 0.0001. After parameter tuning by Bayesian optimization, the corresponding parameters $\alpha$, $\beta$, and $\gamma$ were set to 0.01, 0.25, and 0.001, respectively. The labeled sample ratio of the target domain $\eta$ was set to 0.2. A 10-fold cross validation was used to verify the validity of the model.

4.3. Evaluation Metrics

The proposed AER-SSDA algorithm used binary classification tasks to distinguish benign from malignant breast histopathological images, with evaluation metrics including F1 score, accuracy, sensitivity, and specificity [17].

4.4. Result

4.4.1. Feature Distribution

The features extracted from each part of the network were compared visually in order to intuitively reflect the effect of the AER-SSDA algorithm on the sample feature distribution. In this study, the network was frozen to save the parameters after the model training. The t-distributed stochastic neighbor embedding (t-SNE) [32] was used to visualize the target domain feature distribution.

Figure 3 presents the original feature distribution, high-level feature distribution, auto-encoder latent feature distribution, and classifier feature distribution of the target domain dataset. In this figure, red scatter points represent benign breast tissue pathology image samples, while blue scatter points represent malignant breast tissue pathology image samples for each feature distribution plot. The first column of the t-SNE feature distribution plots shows the feature distribution when the SNL dataset is the target domain, and the second column shows the feature distribution when the BreakHis dataset is the target domain. The first row of two feature distribution plots (a) and (e) displays the feature distribution of original images without network processing. It can be observed that there is significant overlap between benign and malignant samples, resulting in the worst separability. The second row of two feature distribution plots (b) and (f) exhibits the high-level feature distribution of histopathological images after passing through a pre-trained deep network. Samples of the same class exhibit a large radial distribution, and there is a certain separation between different classes. The separability between samples has been improved to a certain degree. The third row of two feature distribution plots (c) and (g) demonstrates the latent feature distribution after feature learning and compression through auto-encoder encoding. Compared to (b) and (f), the aggregation degree of the same class sample points becomes higher, the distance between the centers of the two classes increases, and the classification boundary becomes clearer. The images in the last row (d) and (h) show the latent feature distribution after processing by the label discriminator. Compared to the images in the first three rows, the scatter plots of the two different classes have the clearest classification boundary, and the separability between the two classes reaches the highest level. The distribution characteristics and differences in the features among the different samples indicate that the proposed AER-SSDA algorithm can effectively mine information beneficial for classification during compressed representation learning and feature reconstruction.

4.4.2. Comparative Experiment

To validate the effectiveness of our proposed algorithm (AER-SSDA), we compared it with several other domain adaptation algorithms. The experiments utilized the BreakHis dataset and the SNL dataset to conduct bidirectional domain adaptation classification experiments, verifying the effectiveness of the method through these bidirectional experiments.

The comparison algorithms included two unsupervised domain adaptation algorithms (DSN [33] and MCD_DA [34]), as well as two semi-supervised domain adaptation algorithms (FADA [35] and CCSA [36]). Among them, DSN and FADA are also reconstruction-based domain adaptation algorithms.

The experimental results from the BreakHis dataset to the SNL dataset are shown in

Table 4. The experimental results of different comparative algorithms (BreakHis to SNL)

Method	F1 Score (%)	Accuracy (%)	Sensitivity (%)	Specificity (%)
DSN [33]	71.58 ± 0.78	64.87 ± 1.24	61.60 ± 2.31	73.22 ± 1.72
MCD_DA [34]	73.34 ± 1.11	75.03 ± 1.49	90.20 ± 1.76	66.14 ± 2.94
FADA [35]	85.86 ± 0.88	80.43 ± 0.60	82.65 ± 0.92	74.78 ± 0.89
CCSA [36]	78.63 ± 0.69	78.07 ± 0.73	80.12 ± 1.08	76.35 ± 1.43
AER-SSDA	93.40 ± 0.48	95.24 ± 0.32	94.66 ± 0.87	95.56 ± 1.27

The bold numbers in the table represent the best comparison results.

. The proposed AER-SSDA algorithm exhibited the best experimental results (F1 score: 93.40%, accuracy: 95.24%, sensitivity: 94.66%, specificity: 95.56%), fully demonstrating the effectiveness of this method. Due to the lack of target sample labels that can provide classification guidance during training, the unsupervised domain adaptation methods, DSN (F1 score: 71.58%, accuracy: 64.87%, sensitivity: 61.60%, specificity: 73.22%) and MCD_DA (F1 score: 73.34%, accuracy: 75.03%, sensitivity: 90.20%, specificity: 66.14%), performed significantly worse than the proposed AER-SSDA algorithm. Compared with the other two semi-supervised domain adaptation methods, CCSA (F1 score: 78.63%, accuracy: 78.07%, sensitivity: 80.12%, specificity: 76.35%) and FADA (F1 score: 85.86%, accuracy: 80.43%, sensitivity: 82.65%, specificity: 74.78%), the proposed AER-SSDA algorithm demonstrated a stronger learning ability in semi-supervised domain adaptation tasks. This is attributed to the proposed auto-encoder, which learns more useful information for classification within a single domain during the feature extraction and reconstruction task, enhancing the representation ability of intra-domain features. When aligning multiple feature representations during the encoding–decoding process, it measures and reduces the differences in feature distributions between the source and target domains. This facilitates the transfer of labeled feature representations from the source domain to the target domain, thereby improving the classification performance of the model.

Table 5. The experimental results of different comparative algorithms (SNL to BreakHis)

Method	F1 Score (%)	Accuracy (%)	Sensitivity (%)	Specificity (%)
DSN [33]	69.64 ± 2.39	77.78 ± 2.35	71.11 ± 3.17	81.48 ± 2.83
MCD_DA [34]	72.95 ± 1.68	74.74 ± 1.46	95.17 ± 1.27	63.38 ± 4.58
FADA [35]	82.42 ± 1.23	78.25 ± 1.07	79.88 ± 1.64	73.46 ± 2.04
CCSA [36]	74.91 ± 0.98	74.65 ± 0.85	74.58 ± 1.67	75.45 ± 1.52
AER-SSDA	92.08 ± 0.59	89.47 ± 0.47	88.87 ± 1.19	89.14 ± 1.22

The bold numbers in the table represent the best comparison results.

presents the experimental results of the comparison algorithms from SNL to BreakHis. Due to the decrease in the number of source domain samples, leading to a reduction in the knowledge transferable to the target domain, the model’s classification performance declined. However, compared with other comparison methods, the proposed AER-SSDA algorithm still demonstrated better performance. This further validates the effectiveness of the model and algorithm. The experiments showed that the proposed AER-SSDA achieved the best performance in breast histopathological image classification tasks.

4.4.3. Ablation Experiment

To demonstrate the effectiveness of each key component in the algorithm, we designed three ablation variants: AER-Variant1, AER-Variant2, and AER-Variant3, for comparison with the proposed algorithm. Compared to the proposed algorithm, AER-Variant1 removes the domain similarity loss generated during the domain discrimination process; in other words, it removes loss term ${\mathcal{L}}_{similarity}$. AER-Variant2 explores the impact of the auto-encoder structure on experimental results by conducting domain adaptation training without reconstruction loss; in other words, it removes loss term ${\mathcal{L}}_{recon}$. AER-Variant3 removes both the domain discriminator, the auto-encoder and the corresponding loss function; in other words, it removes loss term ${\mathcal{L}}_{MMD}$.

Table 6. The experimental results of different ablation algorithms (BreakHis to SNL).

Method	Loss			F1 Score (%)	Accuracy (%)	Sensitivity (%)	Specificity (%)
	${\mathcal{L}}_{\mathit{s}\mathit{i}\mathit{m}\mathit{i}\mathit{l}\mathit{a}\mathit{r}\mathit{i}\mathit{t}\mathit{y}}$	${\mathcal{L}}_{\mathit{r}\mathit{e}\mathit{c}\mathit{o}\mathit{n}}$	${\mathcal{L}}_{\mathit{M}\mathit{M}\mathit{D}}$
AER-Variant1		✓	✓	88.17 ± 0.71	90.96 ± 0.67	93.33 ± 1.21	89.63 ± 1.37
AER-Variant2	✓		✓	84.70 ± 1.25	88.57 ± 1.01	87.98 ± 2.12	88.89 ± 1.46
AER-Variant3	✓	✓		83.32 ± 1.63	87.62 ± 2.09	85.33 ± 2.75	88.86 ± 1.94
AER-SSDA	✓	✓	✓	93.40 ± 0.48	95.24 ± 0.32	94.66 ± 0.87	95.56 ± 1.27

In the table, a ✓ below the loss item indicates that this loss exists, and the absence of a ✓ represents that this loss item is discarded in the total loss function. The bold numbers in the table represent the best comparison results.

presents the experimental results of the ablation variants from BreakHis to SNL. On the BreakHis -to-SNL dataset, the experimental results of the three ablation variants, AER-Variant1 (F1 score: 88.17%, accuracy: 90.96%, sensitivity: 93.33%, specificity: 89.63%), AER-Variant2 (F1 score: 84.70%, accuracy: 88.57%, sensitivity: 87.98%, specificity: 88.89%), and AER-Variant3 (F1 score: 83.32%, accuracy: 87.62%, sensitivity: 85.33%, specificity: 88.86%), demonstrated the necessity and importance of each part of the model. The results of AER-Variant1 showed that the similarity loss generated by the domain discriminator could blur the domain discriminative information in the encoder’s feature representations and enhance the model’s overall adversarial learning capability. AER-Variant2 replaced the auto-encoder embedding with a fully connected network with the same network parameters. The experimental results indicate that the proposed auto-encoder reconstruction domain adaptation model is not merely a simple compressed representation for feature extraction. In a series of feature transformations during encoding and decoding, the auto-encoder learns abstract information about histopathological images under the constraint of feature reconstruction loss, which can more effectively mine and learn the classification information of the samples. The experimental results of AER-Variant3 demonstrated that the domain similarity loss and high-level feature reconstruction loss could cooperate with each other. The analysis of the above ablation variants proves the effectiveness of the overall algorithm and its components in the domain adaptation experiment from BreakHis to SNL.

Table 7. The experimental results of different ablation algorithms (SNL to BreakHis).

Method	Loss			F1 Score (%)	Accuracy (%)	Sensitivity (%)	Specificity (%)
	${\mathcal{L}}_{\mathit{s}\mathit{i}\mathit{m}\mathit{i}\mathit{l}\mathit{a}\mathit{r}\mathit{i}\mathit{t}\mathit{y}}$	${\mathcal{L}}_{\mathit{r}\mathit{e}\mathit{c}\mathit{o}\mathit{n}}$	${\mathcal{L}}_{\mathit{M}\mathit{M}\mathit{D}}$
AER-Variant1		✓	✓	87.02 ± 0.86	84.17 ± 0.95	82.40 ± 1.43	88.13 ± 1.37
AER-Variant2	✓		✓	83.89 ± 1.21	81.23 ± 1.41	75.53 ± 2.38	87.80 ± 1.89
AER-Variant3	✓	✓		81.17 ± 1.76	77.26 ± 1.83	68.71 ± 3.41	88.75 ± 1.53
AER-SSDA	✓	✓	✓	92.08 ± 0.59	89.47 ± 0.47	88.87 ± 1.19	89.14 ± 1.22

In the table, a ✓ below the loss item indicates that this loss exists, and the absence of a ✓ represents that this loss item is discarded in the total loss function. The bold numbers in the table represent the best comparison results.

presents the experimental results of the ablation variants from SNL to BreakHis. Due to the reduction in the number of source domain samples, the model’s classification performance declined. However, compared to other ablation variants, the proposed AER-SSDA algorithm still exhibited a better performance. This further demonstrates the effectiveness of each part of the model and algorithm. Every component in the model and loss terms is important and indispensable, and the absence of any one of them can lead to a decrease in model performance.

4.4.4. Discussion of Experimental Results

The experimental results reveal that the proposed AER-SSDA method exhibited the best classification performance. Compared to unsupervised domain adaptation methods DSN and MCD_DA, it achieved significant improvements in the results, indicating that using a small number of labeled target samples can enhance domain adaptation performance. When compared to semi-supervised domain adaptation methods FADA and CCSA, the proposed AER-SSDA method demonstrated stronger capabilities in addressing overfitting and under-training issues. This may be attributed to the fact that the domain adaptation with auto-encoder reconstruction can more effectively learn classification-beneficial information from the extracted features. Compared to AER-Variant1, AER-Variant2, and AER-Variant3, feature reconstruction enriches the useful information for classification, while the similarity loss enhances the network’s ability to extract information and minimizes cross-domain differences. The F1 score results of the comparative experiments and ablation experiments are presented in Figure 4 (BreakHis to SNL) and Figure 5 (SNL to BreakHis), which clearly demonstrate the performance advantages of the proposed AER-SSDA algorithm.

The statistical p-value tests based on accuracy for the AER-SSDA algorithm compared to the competitive and ablation algorithms are shown in

Table 8. The statistical result of the p-value (BreakHis to SNL).

Methods	p-Value (%)
DSN [33]	$1.77\times {10}^{-11}$
MCD_DA [34]	$1.81\times {10}^{-9}$
FADA [35]	$3.5\times {10}^{-11}$
CCSA [36]	$3.82\times {10}^{-11}$
AER-Variant 1	$1.24\times {10}^{-6}$
AER-Variant 2	$6.3\times {10}^{-7}$
AER-Variant 3	$4.14\times {10}^{-5}$

(BreakHis to SNL) and

Table 9. The statistical result of the p-value (SNL to BreakHis).

Methods	p-Value (%)
DSN [33]	$4.42\times {10}^{-6}$
MCD_DA [34]	$2.33\times {10}^{-8}$
FADA [35]	$2.33\times {10}^{-8}$
CCSA [36]	$5.95\times {10}^{-10}$
AER-Variant 1	$3.67\times {10}^{-6}$
AER-Variant 2	$1.67\times {10}^{-6}$
AER-Variant 3	$5.14\times {10}^{-7}$

(SNL to BreakHis). This indicates that the experimental results of the AER-SSDA algorithm are statistically significant compared to the comparative and ablation experiments, further proving that the AER-SSDA algorithm has more advantages than other algorithms in the feature reconstruction-based domain adaptation classification tasks.

5. Conclusions

In this study, an auto-encoder reconstructed semi-supervised domain adaption for a histopathological image classification algorithm (AER-SSDA) was proposed. The feature extractor, auto-encoder, domain discriminator, and sample classifier were built and performed reconstructed domain adaptation learning for the classification task of patch-level samples. First, the convolutional feature extractor pre-trained by thr ImageNet dataset performed high-level feature extraction. Subsequently, the extracted features were reconstructed by the auto-encoder. The encoder and decoder completed the information mining, representation learning, and feature reconstruction, and in the process, the original features, latent features, and reconstructed features were minimized for inter-domain distribution differences. Then, the latent features from the auto-encoder were fed to the domain discriminator and sample classifier for domain confusion and patch sample classification, respectively. Finally, the predicted labels of the whole test image were obtained by decision fusion. The experimental results of the proposed AER-SSDA method demonstrated better classification performance compared with other methods, and it has potential for clinical applications.

Despite achieving promising results on the pathological image datasets, our work still has certain limitations. We designed an auto-encoder for sample reconstruction and a feature extraction network specifically for pathological images, but we did not explore the impact of diversified network structures on image classification outcomes. In future work, we will further explore different network structures for feature extraction and sample reconstruction. Although we tested our method on two pathological image datasets and obtained good results, we have not yet explored its application on larger datasets. In the future, we plan to extend our method to large datasets and further investigate the effectiveness of the model on other types of medical images, such as CT images.