Anti-Forgetting Adaptive Teacher-Driven Knowledge Distillation for Medical Image Classification

Abstract

Deep neural networks (DNNs) have achieved remarkable success in medical image classification, yet their performance remains sensitive to dataset size. Knowledge distillation (KD) alleviates this issue by transferring knowledge from a high-capacity teacher to a lightweight student. However, conventional KD relies on a static teacher, while adaptive teacher updating may improve performance on the student-learning data while reducing retention of knowledge acquired during teacher pretraining. To address these limitations, we propose an Anti-forgetting Adaptive Teacher-driven Knowledge Distillation framework (A²T-KD), which aims to balance teacher adaptation and pretraining-knowledge retention. The proposed framework integrates three modules: MITR for cross-epoch representation consistency, DSDO for prediction-space decoupling and class discriminability, and SGKD for feature- and logit-level knowledge transfer. Across nine medical imaging datasets, A²T-KD achieved higher mean values than the fixed-teacher Vanilla KD baseline in 30 of 36 dataset–metric comparisons. It also exhibited the lowest pretraining-set ACC degradation among the evaluated teacher-update baselines on all nine datasets, supporting the intended balance between teacher adaptation and pretraining-knowledge retention under the evaluated settings.

1. Introduction

Recent advances in deep neural networks (DNNs) have expanded the application of learning-based methods in medical image analysis [1,2,3,4,5]. Nevertheless, achieving stable predictive performance across medical imaging tasks with different dataset sizes and acquisition characteristics remains challenging. Medical imaging datasets often vary substantially in scale, annotation quality, and acquisition protocols, and the limited availability of large, high-quality labeled datasets remains a major challenge for medical image classification [6]. While large-capacity models typically benefit from abundant data, they are prone to overfitting in low-data settings, where distillation-based learning has been investigated to improve generalization [7]. Therefore, developing learning frameworks that enable reliable knowledge transfer across medical imaging tasks and dataset scales remains an important challenge.

Knowledge distillation (KD) [8] transfers knowledge from a high-capacity teacher to a lightweight student [9,10]. However, most existing KD methods rely on a fixed teacher throughout training. For student-training samples on which the teacher produces inaccurate predictions, the distillation objective may disagree with ground-truth supervision and weaken knowledge transfer. Consequently, inaccurate teacher predictions may induce gradients that conflict with those generated by the ground-truth supervision (Figure 1a), leading to a conflict between the distillation and ground-truth objectives and degraded knowledge transfer. To alleviate optimization conflicts caused by unreliable teacher supervision, recent studies have explored adaptive teacher-based knowledge distillation frameworks that dynamically update the teacher during training [11,12]. By updating the teacher during student training, these methods can reduce the mismatch between teacher predictions and the current learning objective. However, continual teacher adaptation may introduce catastrophic forgetting, where the teacher gradually loses previously acquired knowledge and may exhibit reduced predictive performance on the teacher-pretraining subset.

Therefore, an effective adaptive KD framework should balance teacher adaptation with the retention of knowledge acquired during teacher pretraining. Motivated by this observation, we propose Anti-forgetting Adaptive Teacher-driven Knowledge Distillation (A²T-KD), a unified framework that dynamically updates the teacher while constraining cross-epoch changes in both feature and prediction spaces. As illustrated in Figure 1c, A²T-KD is designed to balance teacher adaptation and the retention of knowledge acquired during teacher pretraining by constraining cross-epoch changes in both feature and prediction spaces. The framework consists of three modules: (i) Mutual Information Temporal Regularization (MITR), which enforces cross-epoch consistency to reduce representational drift; (ii) Discriminative Subspace Decoupling Optimization (DSDO), which enhances class discriminability while preserving relational information contained in non-target predictions; and (iii) Structure-Guided Knowledge Distillation (SGKD), which performs feature- and logit-level knowledge transfer. The contributions of this work are summarized as follows:

• We propose A²T-KD, an adaptive knowledge distillation framework that updates the teacher during student training while constraining the loss of knowledge acquired during teacher pretraining.
• We introduce MITR and DSDO to constrain cross-epoch changes in teacher representations and predictions, thereby supporting the retention of pretraining knowledge during teacher adaptation.
• We evaluate A²T-KD on nine medical imaging datasets and multiple teacher–student configurations. A²T-KD achieved competitive predictive performance relative to the evaluated teacher-update baselines.

2. Related Work

Knowledge Distillation for Medical Image Analysis: Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a lightweight student while preserving predictive performance. Existing methods can be broadly categorized into feature-based, relation-based, and logit-based approaches [13,14,15]. KD has been successfully applied to diverse medical imaging tasks, including physiological measurement, radiology report generation, whole-slide image classification, brain tumor segmentation, and dental image detection and classification [16,17,18,19,20]. However, most medical KD methods rely on a fixed teacher that cannot correct erroneous predictions encountered during student training.

Adaptive Teacher-based KD and Catastrophic Forgetting: To address the limitations of static teachers, adaptive KD dynamically updates the teacher during training to provide more reliable supervision [11,12]. While such strategies alleviate the mismatch between teacher predictions and learning objectives, continual adaptation may introduce catastrophic forgetting, causing the teacher to lose previously acquired knowledge. Existing forgetting mitigation methods [21,22], such as parameter regularization, replay, and self-distillation, have shown promise in continual learning but are not specifically designed for adaptive KD, where forgetting can further propagate unreliable supervision to the student.

Optimization Conflicts in Knowledge Distillation: Optimization conflicts between teacher supervision and ground-truth labels represent another key challenge in KD [23]. Inaccurate teacher predictions may induce gradients that conflict with the classification objective, hindering optimization and weakening knowledge transfer. Although gradient conflicts have been extensively studied in multi-task learning [24,25], they remain largely underexplored in medical KD. Existing methods typically address either optimization conflicts or forgetting independently, overlooking their interaction. Motivated by these two challenges, we propose A²T-KD, which dynamically updates the teacher while preserving information acquired during teacher pretraining and from preceding optimization epochs in both the feature and prediction spaces.

3. Method

3.1. Notations

Given a labeled medical image dataset $\mathcal{D}={\left\{({\mathbf{x}}_i,{\mathbf{y}}_i)\right\}}_{i=1}^N$, where ${\mathbf{x}}_i\in {\mathbb{R}}^{H\times W\times D}$ and ${\mathbf{y}}_i\in \{1, \dots , C\}$, the teacher and student at epoch e are denoted by ${\mathrm{\Theta }}_t^e={\varphi }_t^e\circ f_t^e$ and ${\mathrm{\Theta }}_s^e={\varphi }_s^e\circ f_s^e$, respectively. The corresponding feature extractors produce representations ${\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e\in {\mathbb{R}}^{K_t}$ and ${\mathbf{z}}_{\mathrm{s},\mathrm{i}}^e\in {\mathbb{R}}^{K_s}$, where $K_t$ and $K_s$ depend on the teacher and student backbones, respectively. When $K_t\ne K_s$, a learnable feature projection is used to map the teacher and student representations to a common distillation dimension $K_d$ before feature-level alignment. The framework is shown in Figure 2.

3.2. Key Modules in A²T-KD

3.2.1. Mutual Information Temporal Regularization (MITR)

Adaptive distillation continuously updates the teacher, which may lead to representational drift and catastrophic forgetting. To address this issue, MITR characterizes local cross-epoch variations in teacher representations by jointly considering the embeddings from two consecutive epochs.

Let ${\mathbf{z}}_{\mathrm{t},\mathrm{i}}^{e-1}$ and ${\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e$ denote the feature embeddings of the i-th sample extracted by the teacher at epochs $e-1$ and e, respectively. MITR treats the preceding- and current-epoch teacher representations as a paired two-token input to a lightweight Transformer encoder. Both representations are first mapped to the 128-dimensional embedding space using the same input projection $P_{\mathrm{in}}(·)$:

$${\mathcal{Z}}_{\mathrm{t},\mathrm{i}}^e=\left\{P_{\mathrm{in}}\left({\mathbf{z}}_{\mathrm{t},\mathrm{i}}^{e-1}\right),P_{\mathrm{in}}\left({\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e\right)\right\}.$$

(1)

Figure 2. Overview of the proposed A²T-KD framework. The pipeline comprises three core modules: (1) MITR promotes cross-epoch consistency by jointly processing representations from adjacent epochs with stability and mutual-information regularization; (2) DSDO decomposes predictions into target and non-target subspaces to support target-class discrimination while preserving inter-class relational structure across training epochs; and (3) SGKD performs feature- and prediction-level alignment between teacher and student.

Figure 2. Overview of the proposed A²T-KD framework. The pipeline comprises three core modules: (1) MITR promotes cross-epoch consistency by jointly processing representations from adjacent epochs with stability and mutual-information regularization; (2) DSDO decomposes predictions into target and non-target subspaces to support target-class discrimination while preserving inter-class relational structure across training epochs; and (3) SGKD performs feature- and prediction-level alignment between teacher and student.

The Transformer encoder integrates the two projected adjacent-epoch representations and produces a context-aware representation ${\mathbf{h}}_{\mathrm{t},\mathrm{i}}^e$:

$${\mathbf{h}}_{\mathrm{t},\mathrm{i}}^e=\mathrm{Transformer}\left({\mathcal{Z}}_{\mathrm{t},\mathrm{i}}^e\right).$$

(2)

Here, ${\mathbf{h}}_{\mathrm{t},\mathrm{i}}^e$ denotes the context-aware embedding that captures the temporal dynamics of the teacher representations across consecutive epochs via self-attention. The Transformer preserves the two-token sequence length, and the output corresponding to the current-epoch token is used as ${\mathbf{h}}_{\mathrm{t},\mathrm{i}}^e$. To enforce smooth evolution across epochs, we further define ${\widehat{\mathbf{z}}}_{\mathrm{t},\mathrm{i}}^{e-1}$ as a non-trainable 128-dimensional reference obtained by applying the same input projection to the preceding-epoch teacher embedding and then stopping the gradient: ${\widehat{\mathbf{z}}}_{\mathrm{t},\mathrm{i}}^{e-1}=StopGrad\left(P_{\mathrm{in}}\left({\mathbf{z}}_{\mathrm{t},\mathrm{i}}^{e-1}\right)\right)$. Formally, the stability loss is then written as

$${\mathcal{L}}_{\mathrm{stab}}^e={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{∥{\mathbf{h}}_{\mathrm{t},\mathrm{i}}^e-{\widehat{\mathbf{z}}}_{\mathrm{t},\mathrm{i}}^{e-1}∥}_2^2,$$

(3)

where ${\parallel ·\parallel }_2$ is the ${\ell }_2$-norm. This stability loss encourages the integrated representation $\mathbf{h}{t,i}^e$ to stay consistent with the preceding-epoch representation, thereby reducing abrupt representational drift and integrating information from both epochs. We further introduce a mutual-information regularizer to preserve information shared between the representations from consecutive epochs. We apply non-linear projection heads $l_{\mathrm{t}}^e(·)$ and $g_{\mathrm{t}}^e(·)$ to ${\mathbf{z}}_{\mathrm{t},\mathrm{i}}^{e-1}$ and ${\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e$, respectively.

$${\mathbf{l}}_{\mathrm{t},\mathrm{i}}^e=l_{\mathrm{t}}^e\left({\mathbf{z}}_{\mathrm{t},\mathrm{i}}^{e-1}\right),{\mathbf{g}}_{\mathrm{t},\mathrm{i}}^{\mathrm{e}}=g_{\mathrm{t}}^e\left({\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e\right),$$

(4)

where ${\mathbf{l}}_{\mathrm{t},\mathrm{i}}^e\in {\mathbb{R}}^d$ and ${\mathbf{g}}_{\mathrm{t},\mathrm{i}}^e\in {\mathbb{R}}^d$ denote the projected features used for the subsequent estimation of mutual information. The projection heads $l_{\mathrm{t}}^e(·)$ and $g_{\mathrm{t}}^e(·)$ are implemented as independent one-layer MLPs with $K_t\to 128$ dimensions and a ReLU activation between the two linear layers. Accordingly, the projected feature dimension is 128. The mutual-information estimation objective is defined as follows:

$${\mathcal{L}}_{\mathrm{MI}}^e=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}log{\displaystyle \frac{exp\left(〈{\mathbf{l}}_{\mathrm{t},\mathrm{i}}^e,{\mathbf{g}}_{\mathrm{t},\mathrm{i}}^e〉/{\tau }_1\right)}{{\sum }_{j=1}^{N}exp\left(〈{\mathbf{l}}_{\mathrm{t},\mathrm{i}}^e,{\mathbf{g}}_{\mathrm{t},\mathrm{j}}^e〉/{\tau }_1\right)}},$$

(5)

where ${\tau }_1$ is a temperature coefficient, and $\langle ·,·\rangle$ denotes the inner product. The overall MITR loss at epoch e is defined as follows:

$${\mathcal{L}}_{\mathrm{MITR}}^e={\mathcal{L}}_{\mathrm{stab}}^e+{\mathcal{L}}_{\mathrm{MI}}^e.$$

(6)

For implementation, MITR employs a Transformer encoder with an embedding dimension of 128, 4 encoder layers, 4 attention heads, and a feed-forward dimension of 512. Before entering the Transformer, the teacher representations are mapped from their native dimension $K_t$ to the 128-dimensional embedding space by an input projection layer. MITR therefore combines pairwise cross-epoch modeling, an explicit stability anchor, and mutual-information regularization to preserve consistent teacher representations between consecutive epochs.

3.2.2. Discriminative Subspace Decoupling Optimization (DSDO)

To mitigate prediction-level forgetting, the Discriminative Subspace Decoupling Optimization (DSDO) module decomposes the classifier output into target and non-target subspaces. A target-class classification loss enhances class discriminability, while an angular regularizer preserves inter-class relational knowledge in the non-target subspace. These constraints provide cross-epoch-consistent and discriminative supervision during adaptive distillation.

Let ${\mathbf{W}}_{\mathrm{t}}^e=\left[{\mathbf{w}}_{\mathrm{t},1}^e,\dots ,{\mathbf{w}}_{\mathrm{t},\mathrm{C}}^e\right]\in {\mathbb{R}}^{K\times C}$ denote the teacher classifier at epoch e, where ${\mathbf{w}}_{\mathrm{t},\mathrm{c}}^e$ is the classifier weight associated with class c. Given a sample with ground-truth label $\widehat{c}$, we decompose ${\mathbf{W}}_{\mathrm{t}}^e$ into a target subspace ${\mathbf{W}}_{\mathrm{t},\mathrm{tar}}^e\in {\mathbb{R}}^{K\times 1}$ and a non-target subspace ${\mathbf{W}}_{\mathrm{t},\mathrm{nt}}^e\in {\mathbb{R}}^{K\times (C-1)}$. The former corresponds to the classifier weight of class $\widehat{c}$, whereas the latter consists of all remaining class weights and preserves inter-class relational information. Furthermore, we project the teacher embedding ${\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e$ onto these two complementary subspaces to promote logit-level knowledge preservation, which is formally defined as follows:

$${\mathbf{p}}_{\mathrm{t},\mathrm{i}}^{\mathrm{tar},\mathrm{e}}={\left({\mathbf{W}}_{\mathrm{t},\mathrm{tar}}^e\right)}^{\top }{\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e,{\mathbf{p}}_{\mathrm{t},\mathrm{i}}^{\mathrm{nt},\mathrm{e}}={\left({\mathbf{W}}_{\mathrm{t},\mathrm{nt}}^e\right)}^{\top }{\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e,$$

(7)

where ${(·)}^{\top }$ is the transpose operator. Here, ${\mathbf{p}}_{\mathrm{t},\mathrm{i}}^{\mathrm{tar},e}$ denotes the target-class logit, whereas ${\mathbf{p}}_{\mathrm{t},\mathrm{i}}^{\mathrm{nt},e}$ denotes the vector of non-target logits.

To reinforce class discriminability, we optimize the scalar target-class logit relative to the non-target logits using a target-relative cross-entropy objective. The non-target logits are treated as stop-gradient references when computing this loss. The target-class classification loss is defined as

$${\mathcal{L}}_{\mathrm{clc}}^e=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}log{\displaystyle \frac{exp\left({\mathrm{p}}_{\mathrm{t},\mathrm{i}}^{\mathrm{tar},e}\right)}{exp\left({\mathrm{p}}_{\mathrm{t},\mathrm{i}}^{\mathrm{tar},e}\right)+{\sum }_{j=1}^{C-1}exp\left(\mathrm{sg}\left({\mathrm{p}}_{\mathrm{t},\mathrm{i},\mathrm{j}}^{\mathrm{nt},e}\right)\right)}},$$

(8)

where $\mathrm{sg}(·)$ denotes the stop-gradient operator. The forward value of this objective is identical to that of the standard multi-class cross-entropy loss under the target/non-target decomposition. However, the stop-gradient operator prevents the non-target logits from being updated by ${\mathcal{L}}_{\mathrm{clc}}^e$. Consequently, the objective increases the target-class logit relative to fixed non-target references without imposing an arbitrary regression target on the logit magnitude. The non-target logits additionally encode relational information among the remaining classes. To preserve this inter-class structure during teacher adaptation, we project the current teacher representation ${\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e$ onto the non-target classifier weights from the preceding epoch, ${\tilde{\mathbf{p}}}_{\mathrm{t},\mathrm{i}}^{\mathrm{nt},e-1}={\left({\mathbf{W}}_{\mathrm{t},\mathrm{nt}}^{e-1}\right)}^{\top }{\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e$, where ${\tilde{\mathbf{p}}}_{\mathrm{t},\mathrm{i}}^{\mathbf{nt},e-1}\in {\mathbb{R}}^{C-1}$ denotes the non-target logit vector induced by the classifier weights from the preceding epoch under the current feature representation.

$${\mathcal{L}}_{\mathrm{angle}}^e={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\left(1-cos\left({\mathbf{p}}_{\mathrm{t},\mathrm{i}}^{\mathrm{nt},e},{\tilde{\mathbf{p}}}_{\mathrm{t},\mathrm{i}}^{\mathrm{nt},e-1}\right)\right).$$

(9)

Here, $cos(·,·)$ denotes the cosine similarity between the current non-target logit vector ${\mathbf{p}}_{\mathrm{t},\mathrm{i}}^{\mathrm{nt},e}$ and the prediction ${\tilde{\mathbf{p}}}_{\mathrm{t},\mathrm{i}}^{\mathrm{nt},e-1}$ induced by the preceding-epoch classifier weights. The angular consistency loss encourages the current classifier to preserve the relational structure encoded by the preceding-epoch classifier, thereby reducing prediction-level forgetting during teacher adaptation.

The overall DSDO loss is defined as follows:

$${\mathcal{L}}_{\mathrm{DSDO}}^e={\mathcal{L}}_{\mathrm{clc}}^e+{\mathcal{L}}_{\mathrm{angle}}^e.$$

(10)

Therefore, DSDO jointly enhances class discriminability and preserves the inter-class geometry encoded by the preceding-epoch classifier.

3.2.3. Structure-Guided Knowledge Distillation (SGKD)

Our framework incorporates the Structure-Guided Knowledge Distillation (SGKD) module, comprising feature- and logit-level distillation branches. The feature branch aligns teacher and student representations, whereas the logit branch aligns their prediction distributions to preserve output-level consistency. The details are described below.

We first perform feature alignment by minimizing the discrepancy between the student embedding and its corresponding teacher counterpart in the latent space:

$${\mathcal{L}}_{\mathrm{feat}}^e={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{∥P_t\left({\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e\right)-P_s\left({\mathbf{z}}_{\mathrm{s},\mathrm{i}}^e\right)∥}_2^2.$$

(11)

where ${\mathbf{z}}_{\mathrm{t},\mathrm{i}}^e$ and ${\mathbf{z}}_{\mathrm{s},\mathrm{i}}^e$ denote the teacher and student feature embeddings of the i-th sample at epoch e, respectively. For heterogeneous teacher–student pairs, the native backbone representations may have different dimensions, and we therefore map them to a common space using learnable projection layers $P_t(·)$ and $P_s(·)$. This objective enforces instance-level feature consistency and reduces the representational gap between teacher and student. The feature extraction locations, native feature dimensions, and projection configurations for all evaluated teacher–student architectures are provided in Supplementary Table S1.

In addition, logit-level knowledge transfer is implemented by minimizing the Kullback–Leibler divergence between student and teacher prediction distributions, which is formulated as follows:

$${\mathcal{L}}_{\mathrm{logit}}^e={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\left({\tau }_2\right)}^2\mathrm{KL}\left(\sigma \left({\displaystyle \frac{{\mathbf{p}}_{\mathrm{t},\mathrm{i}}^e}{{\tau }_2}}\right)\parallel \sigma \left({\displaystyle \frac{{\mathbf{p}}_{\mathrm{s},\mathrm{i}}^e}{{\tau }_2}}\right)\right),$$

(12)

where $\sigma (·)$ denotes the softmax function and ${\tau }_2$ is the temperature parameter. The distillation loss is formulated as follows:

$${\mathcal{L}}_{\mathrm{SGKD}}^e={\mathcal{L}}_{\mathrm{feat}}^e+{\mathcal{L}}_{\mathrm{logit}}^e.$$

(13)

The dual-branch transfer strategy enables the student to learn representation-level structure through feature alignment and class-level predictive information through logit supervision.

3.3. Model Optimization

The proposed A²T-KD framework integrates MITR, DSDO, and SGKD to preserve teacher knowledge, enhance class discriminability, and improve knowledge transfer. At each epoch, the teacher is first optimized with anti-forgetting objectives, followed by student distillation using the updated teacher.

• Updating the teacher model while freezing the student model:

  $${\mathcal{L}}_{\mathrm{teacher}}^e={\mathcal{L}}_{\mathrm{MITR}}^e+\lambda {\mathcal{L}}_{\mathrm{DSDO}}^e$$

  (14)
• Updating the student model while freezing the teacher model:

  $${\mathcal{L}}_{\mathrm{student}}^e={\mathcal{L}}_{\mathrm{KD}}^e.$$

  (15)

In Equation (14), $\lambda$ is a hyperparameter. In our experiments, we empirically set $\lambda =1$. This alternating scheme is designed to limit abrupt changes across consecutive teacher updates while allowing the student to learn from the updated teacher.We evaluate A²T-KD on nine medical imaging benchmarks under the experimental protocol described below.

4. Experiments

4.1. Experiment Setup

4.1.1. Datasets

To evaluate the classification performance of the proposed model in classification tasks, we conducted experiments on multiple medical imaging datasets. These datasets are divided into three groups based on the number of images: (i) large-scale datasets containing more than 5000 images, (ii) medium-scale datasets containing between 1000 and 5000 images, and (iii) small-scale datasets containing fewer than 1000 images. A detailed summary of these datasets is provided in

Table 1. Summary of the datasets. All websites in this table accessed on 23 August 2026.

Dataset	#C	#N	#S	Modality	Split Unit	Disease
LC25000 https://www.kaggle.com/datasets/javaidahmadwani/lc25000	5	25,000	–	Histopathology	Original Source Image	Lung Cancer
HAM10000 https://www.kaggle.com/datasets/vrindaat/ham10000-dataset	7	10,015	7470	Dermoscopy	Lesion	Skin Lesions
Chaoyang https://github.com/bupt-ai-cz/HSA-NRL/tree/main	4	6160	–	Histopathology	Image	Colorectal Cancer
Multiple Myeloma	4	1816	63	MRI	Patient	Multiple Myeloma
Brain Tumor https://www.kaggle.com/datasets/masoudnickparvar/brain-tumor-mri-dataset	4	3264	–	MRI	Image	Brain Tumor
Kidney https://github.com/Ritesh18117/Detection-and-Classification-of-Kidney-Diseases-Using-CT-Scanned-Image/tree/master	4	3956	–	CT	Image	Kidney Disease
Breast Tumor https://www.kaggle.com/datasets/sabahesaraki/breast-ultrasound-images-dataset	3	780	600	Ultrasound	Patient	Breast Cancer
Cataract https://www.kaggle.com/datasets/jr2ngb/cataractdataset	4	601	–	Retinal photography	Image	Cataract
PAPILA https://figshare.com/articles/dataset/PAPILA/14798004	3	488	244	Fundus photography	Patient	Glaucoma

Note: #C denotes the number of classes, #N the number of images used in this study, and #S the number of patients, lesions, or other source entities when explicitly available; “–” indicates that a reliable source-entity count was not provided in the released data. For LC25000, “Original Source Image” indicates that each original image and all of its augmentation-derived images were assigned to the same split. For Brain Tumor and Kidney, subsets of the corresponding public releases were used, comprising 3264 and 3956 images, respectively. For Breast Tumor, the accompanying segmentation masks were excluded and only the original ultrasound images were retained. Version/access information: LC25000, 25,000-image release; HAM10000, 10,015-image release; Chaoyang, official release (20 November 2021); Brain Tumor, Version 2; Breast Tumor (BUSI), 780-image release; Cataract, 601-image release; and PAPILA, Version 2 (6 May 2022). License/access conditions: Brain Tumor, CC BY 4.0; PAPILA, GPL 3.0+; Chaoyang, non-commercial research use under the repository terms; LC25000, HAM10000, Kidney, Breast Tumor, and Cataract, no explicit license specified in the cited repositories. Notably, all these public datasets were accessed and downloaded in November 2024.

.

4.1.2. Evaluation Metrics

We employed the following metrics for model evaluation: accuracy (ACC), macro-averaged F1 score (Macro-F1), one-vs-one area under the receiver operating characteristic curve (AUC-OVO), and one-vs-rest area under the receiver operating characteristic curve (AUC-OVR). Macro-F1 was computed as the unweighted mean of the class-wise F1 scores, assigning equal weight to each class regardless of class size. For multi-class AUC, AUC-OVR was computed by treating each class in turn as the positive class against all remaining classes, whereas AUC-OVO was computed over all pairwise class combinations. Both AUC-OVR and AUC-OVO were macro-averaged across classes or class pairs, respectively. For all these metrics, higher values indicate better predictive performance.

4.1.3. Experimental Protocol and Data Partitioning

For all knowledge distillation methods included in the comparison, ResNet101 served as the teacher backbone and was pretrained on 40% of each dataset. The pretrained teacher was then used to guide student optimization of each KD method. With ResNet18 as the student backbone, the remaining 60% of each dataset was further divided into training, validation, and test subsets at a ratio of 70:10:20, corresponding to 42%, 6%, and 12% of the complete dataset, respectively. Notably, the partitioning strategy accounted for related images within individual datasets. HAM10000 was split by lesion, while PAPILA and Breast Tumor were split by patient, ensuring that images from the same patient remained within the same split. For our institutional Multiple Myeloma dataset, where each patient may have multiple MRI images or examinations, all data from the same patient were assigned to the same split. For LC25000, each original image and all of its augmentation-derived images were grouped and assigned to the same split. For the remaining public datasets, data splitting followed the available dataset structure and metadata. These splitting strategies are summarized in Table 1. The teacher-pretraining subset was disjoint from the student-training, validation, and test subsets. To ensure fair comparisons, all baseline models and KD methods used identical student training, validation, and test partitions. All classification experiments were repeated five times using different random seeds, and the results are reported as the mean ± standard deviation. Specifically, all computational-complexity measurements were obtained on a single NVIDIA RTX A6000 GPU using PyTorch 1.11.0. Training time was measured over the complete KD training stage and averaged over three independent runs. No separate warm-up phase was used, and data-loading time was included in the reported wall-clock training time. Class-stratified partitioning was applied while preserving patient-, lesion-, or source-image grouping where available. The class distributions across the teacher-pretraining, student-training, validation, and test sets are reported in Supplementary Table S2.

4.1.4. Implementation Details

All input images were resized to $224\times 224$ pixels, converted to tensors, and normalized using the ImageNet mean $(0.485, 0.456, 0.406)$ and standard deviation $(0.229, 0.224, 0.225)$. No additional data augmentation was applied. Models were optimized using Adam with an initial learning rate of $1\times {10}^{-4}$. A fixed learning rate was used without a learning-rate scheduler, and no weight decay was applied. Training was performed for 100 epochs with a batch size of 128 and without early stopping. The matched random seeds were 42, 1234, 666, 2026, and 9999. The dataset-specific values of ${\tau }_1$ and ${\tau }_2$ were selected based exclusively on validation performance and are reported in Supplementary Table S3. For the evaluated teacher–student architectures, backbone features were projected into a shared 128-dimensional space, with the feature extraction locations, native dimensions, and projection settings provided in Supplementary Table S1. For the cross-epoch constraints, MITR uses the preceding-epoch teacher representations as detached references, whereas DSDO uses the preceding-epoch teacher classifier weights as fixed references. Gradients are propagated only through the current-epoch branches.

4.1.5. Comparison Methods

We compared the proposed A²T-KD framework with two scratch-trained baselines and several representative knowledge distillation methods. Baseline 1 and Baseline 2 employed randomly initialized ResNet18 [26] and ResNet101 [26], respectively, and were trained on the student-training set using only the ground-truth cross-entropy loss, without teacher supervision or knowledge distillation. Vanilla KD [13] adopted a fixed ResNet101 teacher pretrained on the pretraining set and a ResNet18 student. The other static teacher-based KD methods included MDR [27], USKD [28], LSKD [15], OFAKD [29], and UniDistill [30], while the adaptive teacher-based KD methods included AFFAKT [31] and FG-KD [32]. All comparison methods used identical student-training, validation, and test partitions and followed the same general experimental protocol. Their implementation sources and method-specific configurations are provided in Supplementary Table S4.

4.2. Experimental Results and Analysis

4.2.1. Performance Comparison

Table 2. Comparison with various methods on large datasets (%). Bold and underlined values indicate the best result for each metric on each dataset.

Method	LC25000				HAM10000				Chaoyang
	ACC	Macro-F1	AUC-OVO	AUC-OVR	ACC	Macro-F1	AUC-OVO	AUC-OVR	ACC	Macro-F1	AUC-OVO	AUC-OVR
Baseline 1	78.6 ± 3.0	77.7 ± 1.6	91.3 ± 3.2	90.5 ± 0.6	71.4 ± 0.9	24.7 ± 7.3	60.9 ± 3.0	69.9 ± 2.2	60.4 ± 0.5	37.1 ± 0.3	79.0 ± 0.8	82.0 ± 0.5
Baseline 2	95.1 ± 0.2	94.9 ± 0.6	98.8 ± 0.0	98.7 ± 0.1	72.7 ± 0.8	30.1 ± 1.2	66.1 ± 1.0	77.0 ± 2.7	73.9 ± 2.2	68.3 ± 1.5	87.1 ± 0.9	87.3 ± 1.0
Vanilla KD	99.1 ± 0.5	99.6 ± 0.5	93.8 ± 3.1	99.7 ± 0.3	80.2 ± 1.1	68.1 ± 1.5	90.3 ± 0.7	91.0 ± 0.7	80.6 ± 1.1	76.4 ± 1.6	89.3 ± 0.7	92.8 ± 0.6
MDR	97.0 ± 0.4	98.5 ± 0.5	99.2 ± 0.2	99.3 ± 0.3	81.9 ± 0.9	43.0 ± 1.7	69.3 ± 1.4	68.1 ± 1.6	77.6 ± 0.9	74.9 ± 1.1	92.0 ± 0.6	92.8 ± 0.6
USKD	82.2 ± 1.2	84.5 ± 1.1	94.8 ± 0.5	95.7 ± 1.0	75.2 ± 1.4	41.2 ± 2.1	73.6 ± 1.3	82.4 ± 1.1	72.4 ± 1.0	67.0 ± 1.7	87.6 ± 0.7	90.9 ± 0.8
LSKD	65.8 ± 2.1	61.7 ± 2.3	92.0 ± 0.7	95.9 ± 0.5	78.8 ± 1.1	43.5 ± 3.1	82.2 ± 0.9	84.9 ± 0.6	73.9 ± 1.4	60.2 ± 2.2	82.9 ± 1.2	88.3 ± 0.6
OFAKD	85.0 ± 0.9	87.1 ± 0.9	97.2 ± 0.3	99.0 ± 0.4	78.0 ± 1.4	51.9 ± 2.7	83.2 ± 0.9	86.3 ± 0.6	72.4 ± 2.4	69.2 ± 2.8	84.6 ± 0.9	87.6 ± 0.6
UniDistill	92.9 ± 0.7	95.8 ± 0.6	98.7 ± 0.5	99.6 ± 0.2	79.2 ± 1.4	28.8 ± 3.3	66.0 ± 1.9	60.2 ± 2.3	76.6 ± 1.3	74.3 ± 1.4	87.6 ± 0.7	91.9 ± 0.5
AFFAKT	98.9 ± 0.4	99.1 ± 0.3	99.7 ± 0.1	99.2 ± 0.8	83.0 ± 0.7	65.2 ± 2.5	87.6 ± 0.5	92.7 ± 0.4	78.5 ± 1.2	73.3 ± 1.5	89.9 ± 0.9	91.9 ± 0.7
FG-KD	97.1 ± 0.5	97.8 ± 0.5	98.4 ± 0.4	98.7 ± 0.2	80.9 ± 0.9	63.8 ± 2.3	86.6 ± 1.0	92.0 ± 0.6	80.3 ± 0.9	74.2 ± 2.1	91.5 ± 0.5	92.9 ± 0.4
A2T-KD	99.5 ± 0.2	99.3 ± 0.6	99.8 ± 0.1	99.9 ± 0.1	84.8 ± 0.6	70.4 ± 1.2	93.0 ± 0.5	96.5 ± 0.3	83.6 ± 1.2	77.9 ± 1.1	92.2 ± 0.4	93.4 ± 0.4

,

Table 3. Comparison with various methods on medium datasets (%). Bold and underlined values indicate the best result for each metric on each dataset.

Method	Multiple Myeloma				Brain Tumor				Kidney
	ACC	Macro-F1	AUC-OVO	AUC-OVR	ACC	Macro-F1	AUC-OVO	AUC-OVR	ACC	Macro-F1	AUC-OVO	AUC-OVR
Baseline 1	52.2 ± 3.0	52.3 ± 3.0	71.7 ± 1.7	74.4 ± 1.8	79.1 ± 1.4	81.0 ± 1.6	88.6 ± 0.9	90.2 ± 0.7	93.5 ± 0.6	90.8 ± 1.1	97.5 ± 0.3	98.5 ± 0.2
Baseline 2	56.6 ± 2.2	57.8 ± 2.8	78.1 ± 1.4	79.6 ± 1.3	82.2 ± 1.0	80.8 ± 1.2	91.6 ± 0.7	92.3 ± 0.5	96.2 ± 0.3	97.6 ± 0.5	98.8 ± 0.2	99.2 ± 0.1
Vanilla KD	60.7 ± 2.3	56.8 ± 2.6	80.4 ± 1.8	80.1 ± 2.1	79.9 ± 1.6	78.0 ± 1.8	91.9 ± 1.3	92.0 ± 0.7	96.6 ± 0.6	95.5 ± 0.7	98.9 ± 0.3	99.2 ± 0.2
MDR	47.7 ± 2.6	30.4 ± 2.4	70.1 ± 1.9	71.9 ± 2.2	81.4 ± 1.4	81.2 ± 2.8	90.8 ± 1.1	91.5 ± 0.7	97.9 ± 0.5	96.8 ± 0.6	98.4 ± 0.4	99.0 ± 0.3
USKD	55.0 ± 2.3	53.3 ± 2.2	80.4 ± 1.1	77.8 ± 1.5	80.2 ± 1.7	75.9 ± 2.6	91.9 ± 0.6	88.0 ± 4.8	86.8 ± 2.2	68.6 ± 2.3	95.5 ± 0.5	97.9 ± 0.3
LSKD	55.2 ± 2.3	42.8 ± 2.3	74.0 ± 2.2	73.1 ± 2.3	79.1 ± 1.5	80.9 ± 1.5	93.8 ± 0.7	94.0 ± 0.6	91.6 ± 0.9	91.8 ± 1.1	98.2 ± 0.4	99.7 ± 0.2
OFAKD	56.2 ± 3.0	56.7 ± 2.8	83.3 ± 1.2	82.3 ± 1.4	73.7 ± 2.4	75.3 ± 2.1	92.8 ± 0.6	93.1 ± 0.6	89.8 ± 1.2	88.9 ± 2.0	99.8 ± 0.2	98.5 ± 0.4
UniDistill	50.6 ± 3.4	39.2 ± 3.3	60.3 ± 3.3	58.8 ± 3.0	82.1 ± 1.2	80.7 ± 1.5	88.1 ± 2.1	89.7 ± 1.4	98.4 ± 0.3	97.5 ± 0.6	97.7 ± 0.4	99.5 ± 0.3
AFFAKT	61.2 ± 2.4	56.4 ± 2.6	76.8 ± 2.0	76.9 ± 1.9	81.3 ± 1.4	79.6 ± 1.7	89.7 ± 1.5	90.5 ± 1.1	95.3 ± 0.6	93.7 ± 1.0	97.6 ± 0.5	99.1 ± 0.8
FG-KD	58.8 ± 2.2	57.4 ± 2.7	77.7 ± 1.9	78.7 ± 2.2	78.5 ± 1.6	77.7 ± 1.9	91.1 ± 1.1	91.2 ± 0.8	94.8 ± 0.9	93.6 ± 1.0	97.9 ± 0.4	99.7 ± 0.1
A2T-KD	64.9 ± 2.8	61.2 ± 2.6	80.3 ± 1.9	82.7 ± 1.3	85.6 ± 1.2	83.6 ± 1.5	94.7 ± 0.7	93.7 ± 0.7	97.2 ± 0.4	99.0 ± 0.3	99.1 ± 0.1	98.8 ± 0.2

and

Table 4. Comparison with various methods on small datasets (%). Bold and underlined values indicate the best result for each metric on each dataset.

Method	Breast Tumor				Cataract				PAPILA
	ACC	Macro-F1	AUC-OVO	AUC-OVR	ACC	Macro-F1	AUC-OVO	AUC-OVR	ACC	Macro-F1	AUC-OVO	AUC-OVR
Baseline 1	76.7 ± 2.4	69.1 ± 2.2	86.2 ± 1.1	85.8 ± 0.8	69.1 ± 2.6	59.5 ± 3.3	80.7 ± 1.5	79.7 ± 1.3	79.6 ± 1.8	65.0 ± 3.5	80.9 ± 1.7	78.7 ± 1.6
Baseline 2	68.0 ± 2.6	53.2 ± 3.1	84.7 ± 1.3	86.1 ± 1.2	56.8 ± 3.2	32.0 ± 2.5	61.7 ± 2.3	66.6 ± 2.7	62.9 ± 2.5	48.9 ± 2.4	80.7 ± 1.6	77.2 ± 1.7
Vanilla KD	83.8 ± 2.0	83.1 ± 2.3	93.5 ± 0.7	95.9 ± 0.6	71.6 ± 2.1	58.0 ± 2.2	81.8 ± 1.1	83.4 ± 1.0	83.1 ± 1.1	75.4 ± 1.9	85.1 ± 0.9	88.9 ± 0.8
MDR	81.3 ± 2.0	77.7 ± 2.3	89.9 ± 0.8	87.6 ± 1.1	70.7 ± 2.3	60.5 ± 2.4	81.0 ± 1.6	77.5 ± 2.1	69.9 ± 2.0	32.1 ± 3.1	63.6 ± 2.9	64.1 ± 2.1
USKD	79.9 ± 2.2	73.7 ± 2.9	89.2 ± 1.1	88.0 ± 1.1	61.9 ± 2.9	49.3 ± 4.0	68.2 ± 1.9	70.1 ± 1.9	75.8 ± 2.0	66.2 ± 2.1	83.8 ± 1.3	83.2 ± 1.4
LSKD	66.8 ± 2.1	45.3 ± 2.6	89.2 ± 0.9	87.8 ± 1.0	63.4 ± 2.9	42.2 ± 2.5	74.2 ± 2.5	74.3 ± 2.0	73.5 ± 1.8	43.4 ± 2.7	80.2 ± 2.1	82.9 ± 1.6
OFAKD	75.4 ± 2.0	65.9 ± 2.3	83.2 ± 1.3	83.4 ± 1.4	71.6 ± 2.7	65.3 ± 2.2	84.2 ± 1.1	82.1 ± 1.6	73.3 ± 2.1	42.8 ± 2.7	76.9 ± 2.5	78.4 ± 2.1
UniDistill	73.2 ± 2.0	68.1 ± 3.2	89.6 ± 1.1	86.8 ± 1.3	65.2 ± 2.4	39.3 ± 2.6	57.5 ± 2.9	58.6 ± 2.7	67.9 ± 2.7	25.4 ± 2.5	50.3 ± 2.5	52.6 ± 2.4
AFFAKT	81.3 ± 1.8	75.6 ± 1.8	91.5 ± 0.6	92.4 ± 0.7	64.0 ± 2.4	43.2 ± 2.6	79.9 ± 1.5	79.7 ± 1.4	71.4 ± 2.2	40.7 ± 2.8	79.4 ± 1.6	81.7 ± 1.4
FG-KD	77.6 ± 2.2	77.0 ± 2.3	93.6 ± 0.5	90.5 ± 0.7	68.7 ± 2.1	53.9 ± 2.4	79.2 ± 1.6	77.3 ± 2.3	79.0 ± 2.0	66.4 ± 2.3	86.1 ± 1.2	87.0 ± 1.0
A2T-KD	87.2 ± 1.7	84.1 ± 2.0	96.2 ± 1.0	95.8 ± 0.4	72.2 ± 1.9	58.0 ± 2.0	80.9 ± 1.1	84.4 ± 1.1	85.8 ± 0.9	76.9 ± 2.1	87.6 ± 1.0	89.6 ± 0.8

summarize the performance of the competing methods across nine medical imaging datasets. Baseline 2 generally achieved higher results than Baseline 1 on the large- and medium-scale datasets, whereas Baseline 1 performed better on most metrics of the small-scale datasets. A²T-KD generally showed favorable mean performance relative to the fixed-teacher Vanilla KD baseline across the evaluated datasets. To formally evaluate the consistency of these differences, we conducted paired statistical comparisons using the five matched seed-level results for each dataset–metric combination. Several comparisons significantly favored A²T-KD at the unadjusted p < 0.05 level, whereas none significantly favored Vanilla KD. The complete per-seed classification results are reported in Supplementary Table S5, while the paired mean differences, 95% confidence intervals, unadjusted p-values, and Holm-adjusted p-values are provided in Supplementary Table S6. Compared with the other adaptive teacher-based methods, A²T-KD generally showed stronger performance across datasets of different scales. For example, on HAM10000, A²T-KD achieved 84.8% ACC and 96.5% AUC-OVR, compared with 83.0% and 92.7% for AFFAKT, respectively. On Brain Tumor, it reached 85.6% ACC and 94.7% AUC-OVO, exceeding the corresponding results of the other adaptive teacher-based baselines. Similar trends were observed on Breast Tumor and PAPILA, where A²T-KD achieved 87.2% and 85.8% ACC, respectively.

Table 5. Computational complexity during the KD stage on the Chaoyang dataset. Training time, peak GPU memory, and training FLOPs are reported as relative multipliers (×) with respect to Vanilla KD, which is normalized to 1.00. The parameter count excludes the teacher backbone but includes all additional trainable modules introduced by each method. Training time, peak memory, and FLOPs are measured for the complete KD procedure and therefore include teacher computation and all auxiliary modules.

Method	Training Time (×)	Peak Memory (×)	Parameters (M)	Training FLOPs (×)
Vanilla KD	1.00	1.00	11.18	1.00
LSKD	1.01	1.00	11.18	1.00
UniDistill	1.18	1.15	11.85	1.11
OFAKD	1.30	1.23	12.30	1.20
MDR	5.20	2.35	15.50	4.70
A2T-KD	2.65	1.95	13.99	2.85

compares the computational complexity of A²T-KD with representative static-teacher KD methods on the Chaoyang dataset. LSKD introduces negligible computational overhead, as it relies primarily on logit standardization, whereas UniDistill and OFAKD require moderately greater resources for multi-level feature alignment and auxiliary exit branches, respectively. MDR incurs the highest computational cost due to its multi-stage relational modeling and self-supervised training components. A²T-KD requires 2.65× the training time, 1.95× the peak memory, and 2.85× the training FLOPs of Vanilla KD. Excluding the teacher backbone, the student and auxiliary trainable modules comprise 13.99 million parameters. This additional training cost mainly arises from the dynamic teacher updates and cross-epoch representation modeling in MITR. Under the evaluated Chaoyang/RTX A6000 setting, A²T-KD requires lower training time, memory consumption, and FLOPs than MDR. Notably, the teacher adaptation and knowledge-preservation mechanisms are used exclusively during training. Once training is completed, only the lightweight ResNet18 student is retained for inference. Therefore, the improved classification performance of A²T-KD is achieved without increasing the parameter count or computational cost of the deployed model.

4.2.2. Teacher Knowledge Preservation, Teacher Updating, and Student Performance

We compare A²T-KD with representative anti-forgetting baselines, including Baseline-T, EMA-T, EWC-T, Replay-T, and SelfDistill-T. Baseline-T updates the teacher without forgetting mitigation, whereas EMA-T, EWC-T, Replay-T, and SelfDistill-T alleviate forgetting through exponential moving averages, parameter regularization, experience replay, and logit-level self-distillation, respectively. Lower ACC degradation indicates better retention of predictive performance on the teacher-pretraining subset.

Figure 3 evaluates teacher knowledge preservation, teacher test performance changes, and the resulting student performance. The initial and final teachers refer to the pretrained teacher before distillation and the teacher after training, respectively. In Figure 3a, pretraining ACC degradation is calculated as the initial teacher ACC minus the final teacher ACC on the teacher-pretraining subset, with lower values indicating better knowledge retention. Baseline-T exhibits the largest degradation across the evaluated datasets, whereas A²T-KD consistently achieves the lowest degradation, ranging from 1.3% to 2.1%. In Figure 3b, test ACC change is calculated as the final teacher ACC minus the initial teacher ACC on the held-out student test subset. Baseline-T shows ACC declines ranging from 1.7% to 3.6%, whereas A²T-KD achieves positive changes of 1.6%–5.2% across all nine datasets. Figure 3c further shows that A²T-KD achieves the highest mean final student ACC on all evaluated datasets, exceeding the strongest competing teacher-update baseline by 0.5–4.6 percentage points. The differences are particularly evident on Multiple Myeloma (64.9% vs. 61.8%) and Cataract (72.2% vs. 67.6%). The complete per-seed numerical results underlying Figure 3a–c are provided in Supplementary Tables S7, S8, and S9, respectively. The reduced teacher degradation, positive teacher test ACC changes, and improved student performance suggest that jointly preserving previously acquired teacher knowledge while updating the teacher can provide more effective supervision under the evaluated setting.

Figure 3. Comparison of teacher knowledge preservation, teacher test performance changes, and final student performance across nine medical imaging datasets. (a) Teacher ACC degradation on the teacher-pretraining subset. (b) Teacher test ACC change after updating with student-training data. (c) Final student ACC. Error bars indicate the standard deviation across five matched seeds, and the numerical labels indicate the corresponding mean values (%). In panel (c), bar heights should be compared only within the same dataset because dataset-specific y-axis ranges are used. The light-blue background shading is used only to visually distinguish the different datasets.

Figure 3. Comparison of teacher knowledge preservation, teacher test performance changes, and final student performance across nine medical imaging datasets. (a) Teacher ACC degradation on the teacher-pretraining subset. (b) Teacher test ACC change after updating with student-training data. (c) Final student ACC. Error bars indicate the standard deviation across five matched seeds, and the numerical labels indicate the corresponding mean values (%). In panel (c), bar heights should be compared only within the same dataset because dataset-specific y-axis ranges are used. The light-blue background shading is used only to visually distinguish the different datasets.

4.3. Ablation Study

4.3.1. Effectiveness of MITR

As shown in

Table 6. Ablation results of MITR subcomponents across diverse datasets using ACC (%). Bold and underlined values indicate the best result for each metric on each dataset.

Model Variant	Chaoyang	Kidney	Breast Tumor
Direct ${\ell }_2$ Smoothing	$78.8\pm 1.4$	$92.9\pm 1.6$	$78.7\pm 2.0$
w/o MI	$81.9\pm 2.8$	$96.0\pm 0.6$	$81.1\pm 1.5$
MLP-Enc	$77.1\pm 2.6$	$90.3\pm 1.0$	$76.8\pm 2.4$
w/o Stab	$78.4\pm 3.1$	$96.2\pm 0.6$	$79.4\pm 1.9$
MITR (Ours)	83.6 ± 1.2	97.2 ± 0.4	87.2 ± 1.7

, direct ${\ell }_2$ smoothing achieved ACC values of 78.8%, 92.9%, and 78.7% on Chaoyang, Kidney, and Breast Tumor, respectively, whereas the full MITR achieved 83.6%, 97.2%, and 87.2%. Replacing the Transformer with an MLP reduced ACC to 77.1%, 90.3%, and 76.8%, respectively. Removing the mutual-information term reduced ACC to 81.9%, 96.0%, and 81.1%, while removing the stability regularizer resulted in 78.4%, 96.2%, and 79.4%.

4.3.2. Effectiveness of DSDO

We evaluate DSDO through three ablations: removing the target-class classification loss (w/o CLC), removing the angular alignment loss (w/o Ang), and removing the subspace decoupling mechanism (w/o Dec). As shown in

Table 7. Ablation results of DSDO subcomponents across diverse datasets using ACC (%). Bold and underlined values indicate the best result for each metric on each dataset.

Model Variant	LC25000	Brain Tumor	Cataract
w/o CLC	$98.8\pm 0.5$	$81.4\pm 1.1$	$68.4\pm 2.0$
w/o Ang	$94.2\pm 0.6$	$77.4\pm 2.1$	$67.6\pm 3.2$
w/o Dec	$96.9\pm 0.6$	$79.1\pm 1.6$	$70.3\pm 2.3$
DSDO (Ours)	99.5 ± 0.2	85.6 ± 1.2	72.2 ± 1.9

, all ablation variants yield lower performance across the evaluated datasets. Specifically, w/o CLC reduced ACC from 99.5% to 98.8% on LC25000. Removing angular alignment (w/o Ang) decreased ACC from 72.2% to 67.6% on Cataract. Among the ablations, w/o Dec reduced Brain Tumor ACC from 85.6% to 79.1%, supporting the contribution of explicit target/non-target subspace decoupling to the observed performance.

4.3.3. Effectiveness of SGKD

To evaluate the contributions of feature- and logit-level distillation in SGKD, we construct two ablation variants: Feature Only and Logit Only. As shown in

Table 8. Ablation results of SGKD subcomponents across various datasets using ACC (%). Bold and underlined values indicate the best result for each metric on each dataset.

Method Variant	LC25000	Kidney	Breast Tumor
Feature Only	$98.2\pm 0.5$	$97.2\pm 0.7$	$81.7\pm 2.1$
Logit Only	$95.5\pm 0.8$	$95.3\pm 1.7$	$85.9\pm 1.6$
Full SGKD (Ours)	99.5 ± 0.2	97.2 ± 0.4	87.2 ± 1.7

, removing either branch generally reduced performance across the evaluated datasets. Full SGKD achieved the highest ACC on LC25000 and Breast Tumor. On Kidney, Feature Only and Full SGKD achieved the same mean ACC of 97.2%, while Full SGKD showed lower variability across runs. These results suggest that feature- and logit-level supervision provide generally complementary but dataset-dependent contributions to knowledge transfer.

4.4. Backbone Configurations

To examine compatibility with different teacher–student architectures, we evaluated A²T-KD using CNN-to-CNN, CNN-to-mobile, and Transformer-to-CNN configurations. The results in

Table 9. Performance of A²T-KD under diverse teacher–student configurations across various datasets (AUC-OVO, %). Bold and underlined values indicate the best result for each metric on each dataset.

Teacher → Student	LC25000	Multiple Myeloma	Breast Tumor
ResNet101 → ResNet18	99.8 ± 0.1	80.3 ± 1.9	96.2 ± 1.0
ResNet50 → ResNet18	$97.6\pm 0.7$	$80.2\pm 2.1$	$92.5\pm 1.5$
ResNet101 → MobileNetV2	$99.0\pm 0.3$	$79.8\pm 2.6$	$94.1\pm 1.1$
ResNet101 → MobileNetV3	$99.0\pm 0.3$	$84.4\pm 1.6$	$93.1\pm 1.2$
Swin-Tiny → ResNet18	$98.9\pm 0.4$	$78.7\pm 2.2$	$94.8\pm 1.0$
ViT-S → ResNet18	$98.0\pm 0.5$	$75.8\pm 2.5$	$95.5\pm 0.9$
ResNet50 → ShuffleNetV2	$98.8\pm 0.7$	$81.4\pm 1.5$	$92.7\pm 1.2$
DenseNet121 → EfficientNet-B0	$98.4\pm 0.4$	$82.8\pm 1.6$	$94.5\pm 0.7$

show that A²T-KD can be implemented with the evaluated architecture pairs, although its performance varies across datasets and configurations. For example, ResNet101→ResNet18 achieved AUC-OVO values of 99.8%, 80.3%, and 96.2% on LC25000, Multiple Myeloma, and Breast Tumor, respectively. These results indicate compatibility with the evaluated architecture pairs but do not establish architecture-level generalization.

4.5. Hyperparameter Analysis (${\tau }_1$ and ${\tau }_2$)

We analyze two temperature parameters: ${\tau }_1$ for mutual-information estimation in Equation (5) and ${\tau }_2$ for logit-level distillation in Equation (12). The parameter ${\tau }_1$ controls the sharpness of similarities in the latent space, whereas ${\tau }_2$ controls the softness of the teacher and student prediction distributions. As shown in

Table 10. Effect of ${\tau }_1$ (mutual-information temperature) on performance on the HAM10000 dataset. Bold and underlined values indicate the best result for each metric on each dataset.

${\mathit{\tau }}_1$	ACC	Macro-F1	AUC-OVO	AUC-OVR
0.1	$80.7\pm 1.6$	$63.2\pm 3.2$	$88.7\pm 1.3$	$92.5\pm 0.9$
0.5	$83.1\pm 1.3$	$64.6\pm 2.7$	$91.4\pm 1.5$	$93.1\pm 0.8$
1	84.8 ± 0.6	70.4 ± 1.2	93.0 ± 0.5	96.5 ± 0.3
2	$82.8\pm 1.2$	$67.3\pm 1.7$	$90.8\pm 1.3$	$92.9\pm 0.8$
4	$80.8\pm 1.1$	$68.2\pm 1.9$	$90.3\pm 0.9$	$95.1\pm 0.8$

and

Table 11. Effect of ${\tau }_2$ (logit temperature) on performance on the Chaoyang dataset. Bold and underlined values indicate the best result for each metric on each dataset.

${\mathit{\tau }}_2$	ACC	Macro-F1	AUC-OVO	AUC-OVR
1	$80.7\pm 1.5$	$76.6\pm 1.8$	$\mathbf{94}.\mathbf{5}\pm \mathbf{0}.\mathbf{6}$	$\mathbf{95}.\mathbf{6}\pm \mathbf{0}.\mathbf{5}$
2	83.6 ± 1.2	77.9 ± 1.1	92.2 ± 0.4	93.4 ± 0.4
4	$\mathbf{85}.\mathbf{3}\pm \mathbf{1}.\mathbf{1}$	$\mathbf{81}.\mathbf{5}\pm \mathbf{1}.\mathbf{3}$	$92.9\pm 0.9$	$93.9\pm 0.7$
8	$84.9\pm 1.6$	$80.5\pm 1.8$	$94.0\pm 1.1$	$94.2\pm 1.7$

, the best overall performance for ${\tau }_1$ is obtained at ${\tau }_1=1$, with an ACC of 84.8%, Macro-F1 of 70.4%, AUC-OVO of 93.0%, and AUC-OVR of 96.5%. For ${\tau }_2$, a value of 4 produced the highest ACC and Macro-F1, whereas a value of 1 produced the highest AUC-OVO and AUC-OVR.

5. Discussion

Across the nine evaluated medical imaging datasets, which varied in size, imaging modality, and disease category, A²T-KD achieved competitive performance relative to the evaluated baselines. However, these results do not establish cross-dataset or cross-domain generalization. Furthermore, the ablation studies suggest generally complementary but dataset-dependent contributions from the individual components.

Static-teacher knowledge distillation provides stable supervision and remains effective when the teacher-pretraining and student-learning data have similar characteristics. Adaptive-teacher approaches provide greater flexibility by updating the teacher during student training, although this adaptation may affect the preservation of previously acquired knowledge. Within this context, A²T-KD combines teacher adaptation with knowledge preservation. Its competitive classification performance and reduced teacher degradation suggest that retaining knowledge acquired during teacher pretraining may complement teacher adaptation under the current same-task setting.

From a practical perspective, the additional teacher-adaptation and knowledge-preservation components are used only during training, whereas inference requires only the lightweight student model. This design may facilitate deployment in resource-constrained medical imaging settings. Nevertheless, the present results demonstrate methodological performance on retrospective benchmark datasets and should not be interpreted as evidence of clinical effectiveness. External validation, model calibration, interpretability analysis, and prospective evaluation are required before clinical deployment.

However, an important limitation of this study is that the teacher-pretraining and student-learning subsets, although mutually exclusive, were randomly sampled from the same datasets. Therefore, the current protocol evaluates adaptation within the same task distribution rather than under an explicit temporal, cross-center, or cross-device distribution shift. In addition, the present study did not evaluate model calibration, interpretability, or the influence of model outputs on clinical decision-making. Future work will evaluate A²T-KD using independent multi-center cohorts and will further investigate model calibration, interpretability, and potential clinical utility in prospective settings.

6. Conclusions

In this work, we proposed A²T-KD, an adaptive teacher-driven knowledge distillation framework that combines teacher updating with cross-epoch representation and prediction constraints. Across nine medical imaging datasets evaluated using within-dataset random partitions, A²T-KD achieved higher mean values than the fixed-teacher Vanilla KD baseline in 30 of 36 dataset–metric comparisons. It also produced lower pretraining-set ACC degradation than the evaluated teacher-update baselines, suggesting an improved balance between teacher adaptation and retention of pretraining knowledge under the current setting. However, this study did not evaluate sequential clinical cohorts or explicit temporal, cross-center, or cross-device distribution shifts. Future work will evaluate the framework using independent multi-center cohorts and sequential adaptation settings.