Annotation-Efficient Vulnerable Carotid Plaque Identification in 3D MRI: A Multimodal Knowledge-Transferred Framework

Abstract

Multimodal learning has gained attention in recent years due to its ability to effectively utilize data features from various modalities. Diagnosing the vulnerability of atherosclerotic plaques directly from carotid 3D MRI images is challenging for both radiologists and conventional 3D vision networks. In clinical practice, radiologists assess patients using a multimodal approach that incorporates various imaging modalities and domain-specific expertise, paving the way for the creation of multimodal diagnostic networks. In this study, we proposed an effective framework to leverage radiologists’ domain knowledge to improve the automated diagnosis of carotid plaque vulnerability through variational inference and multimodal knowledge distillation (VMD). This framework excels in harnessing cross-modality prior knowledge from limited image annotations and radiology reports within training data, thereby enhancing the diagnostic network’s accuracy for unannotated 3D MRI images. We validated the proposed VMD framework on our in-house dataset, demonstrating its effectiveness.

1. Introduction

Patients with carotid stenosis often have associated carotid atherosclerotic plaques, among which vulnerable plaques carry the risk of rupture, potentially leading to ischemic stroke events with high disability and mortality rates [1,2,3,4,5].

High-resolution magnetic resonance imaging sequences can effectively display the degree of carotid stenosis and detailed plaque composition, such as thin-cap fibroatheroma (TCFA), inflammatory cell infiltration, lipid-rich necrotic core (LRC), calcification (CA), and intraplaque hemorrhage (IPH) [6,7,8,9]. Precise analysis of plaque risk level and vulnerability typically requires years of diagnostic experience by radiologists to identify dangerous components such as LRC and IPH within mixed compositions, guiding the choice of subsequent treatment methods [10,11].

U-Net-like visual networks [12,13,14] have been employed for tasks such as vessel wall segmentation and component detection to address the challenges of carotid vessel analysis [11,15,16]. However, these strongly supervised segmentation strategies often entail significant labor costs due to intensive medical image annotation, especially on 3D medical images, and the subjective nature of these annotations can introduce errors that complicate clinical applications [15]. One solution is to adopt limited annotations without detailed plaque component information to reduce labor intensity, as shown in Figure 1, but this may limit the model’s ability to directly assess plaque vulnerability from the segmentation results. However, due to the specificity of plaques and their components, detailed annotations require labeling of fine components with a grayscale coverage exceeding 30%. Moreover, the grayscale differences between vulnerable components such as LRC and IPH, which have different signal expressions in the images, and relatively stable components like CA, are quite ambiguous, as shown in Figure 2.

Figure 1. Example of the difference between full annotation and limited annotation. Full annotation images include multiple complex annotations of the carotid vessel wall, lumen, lipid-rich necrotic core (LRC), calcification (CA), and intraplaque hemorrhage (IPH), requiring detailed pixel-level annotations. Limited annotation only involves annotating the vessel region at the carotid bifurcation, without the need to involve the lumen and other components.

Figure 2. Example of grayscale overlaps for different components and limited annotation with vessel wall. The left image shows the grayscale distribution of the vessel wall, lumen, and various complex components within the plaque in the MRI image. The grayscale coverage is obtained by calculating the ratio of the lumen and component parts to the entire vessel area. The right image displays the grayscale distribution histograms of CA, LRC, and IPH. Despite the differences, the boundaries of the grayscale values remain quite ambiguous, posing challenges for annotation and segmentation.

Considering that the additional anatomical information provided by limited annotations, such as the location of carotid plaques, is crucial in the diagnostic process, the method of knowledge distillation (KD) [17,18] inspired us to treat these limited annotations as an additional modality. Through the distillation process, the learnable features related to anatomical structures can be transferred to the diagnostic target model. Therefore, it is crucial to maximize the informational richness derived from limited annotations during the training phase.

It is noteworthy that multimodal information produced during the diagnostic process, such as radiology reports, contains rich domain knowledge, providing a natural environment for constructing multimodal learning [19,20,21], yet it is often overlooked in the automatic analysis of carotid plaques. However, the modal differences between language and vision may introduce noise that is detrimental to the construction of diagnostic models [22,23]. Therefore, it becomes highly appealing to enable neural networks to learn expert information from cross-modal information that is beneficial for precise classification in unannotated carotid 3D MRI images while eliminating noise interference.

In this study, we proposed a novel three-level distillation framework based on Variational inference and Multimodal knowledge Distillation (VMD) to transfer comprehensive understanding of radiology reports and limited annotations to student network for plaque vulnerability diagnosis by a student–teacher–expert structure, which enabled us to accomplish the complex task at a lower cost.

Contrastive learning-based mutual information (MI) optimization excels in effective cross-modal feature transfer and alignment, similar to KD [17,18,24,25,26]. Limited annotated data, focusing on the vessel area, provide noise-reduced anatomical information for diagnosing plaque vulnerability and could be considered an additional modality for multimodal learning. Exploiting the similarity between annotated and unannotated data, we speculated that samples of the same category, whether they are vulnerable or stable plaques, from these two modalities should exhibit a higher degree of similarity [25,26]. Therefore, we proposed a contrastive learning-based KD network aimed at improving the diagnostic accuracy for unannotated images. This is achieved by maximizing MI between the student model, which lacks annotations, and the teacher model, which has access to limited annotations. This approach is designed to leverage the inherent similarities within the data to enhance the learning process, enabling the student model to better identify and diagnose plaque vulnerability without the need for extensive annotated data, as shown in Figure 3b.

Figure 3. Schematic illustration of the VMD distillation mechanism integrating limited annotations and radiology reports. (a) The student network directly diagnoses plaque vulnerability from unannotated 3D MRI. (b) Limited annotations serve as the teacher network, providing anatomical prior knowledge. (c) Radiology reports serve as the expert network, supplying domain-specific diagnostic knowledge. Both the teacher and expert networks guide the student through contrastive learning-based mutual information maximization and variational inference, significantly enhancing its diagnostic performance without additional inference cost.

Expertise derived from radiology reports can be integrated into the aforementioned networks, offering expert-level guidance to both the teacher and student models [22,23]. Therefore, we designed an additional expert network to improve the previous distillation framework for both the teacher and student networks. The KD method based on variational inference does not require additional costs at the inference stage [21,27], making it suitable for direct diagnosis from unannotated images. This approach, which minimizes a specific Kullback–Leibler divergence (KLD) to estimate the evidence lower bound (ELBO) of the optimization objective [27], facilitates the transfer of knowledge across different modalities and can be considered as another way of maximizing MI, as shown in Figure 3c.

2. Related Works

2.1. Student–Teacher Distillation Structure

The concept of KD was first proposed in [17] and is considered highly effective in model compression and feature transfer. The student–teacher distillation structure used therein has been applied in many complex tasks. VID [28] maximizes the MI between the student and the teacher network by minimizing the entropy of the teacher to effectively transfer knowledge to the student. CRD [18] introduces contrastive learning into the student–teacher structure. CCL [25] separates image and audio in video classification tasks and optimizes the performance through feature mixing contrast learning. This idea is further explained in AMID [26] from the perspective of MI among different modalities. However, our method, i.e., VMD, utilizes manual annotations to acquire anatomical knowledge from the unannotated images and feed it back to the student by maximizing inter-class MI.

2.2. Image-Text Medical Knowledge Transfer

Pre-trained biomedical semantic understanding models have demonstrated commendable proficiency in comprehending medical texts in recent years [22,23,29]. Some studies [19,21,30] guided the neural network in diagnosing diseases in 2D image modality by extracting features closely associated with the diagnosis from text by a pre-trained language model. These methods employ either knowledge distillation [17,31] or contrastive learning [20] to align features in order to address the problem. By integrating contrastive learning with conditional variational inference, we expanded the strategy of VKD [21] into a student–teacher–expert structure to provide a tighter constraint to transfer the knowledge from the expert to teacher networks because of the substantial disparity between 3D and 2D images.

2.3. Three-Dimensional Carotid MRI Automatic Analysis

Rapid and precise intelligent analysis is highly appealing for the clinical diagnosis of carotid stenosis and atherosclerotic plaque MRI images. Some studies have been designed to address this issue. Wu et al. [16] proposed an end-to-end DeepMAD network that performed excellently in the task of carotid vessel wall segmentation in black-blood MRI sequences, achieving high-precision segmentation of the vessel lumen and outer wall. Zhu et al. [32] designed a carotid segmentation model for multi-parameter MRI based on a 3D residual U-net and a multi-level joint feature optimization framework. Chen et al. [11] proposed a carotid vessel segmentation strategy based on HR-MRI, which can effectively segment the lumen and vessel wall at the carotid bifurcation from 2D MRI images, achieving classification of normal vessels, early lesions, and advanced lesions, but did not involve the division of specific plaque components. Wang et al. [15] designed an end-to-end multi-parameter black-blood MRI carotid vessel segmentation and plaque component recognition network with good performance in component recognition, which had its application range somewhat limited by the need for multi-parameter MRI images. Li et al. [33] combined multimodal data input from CTA and MR with SAM adaptation for medical images, designing a method to extend sparse annotations and segment the carotid vessel wall and lumen. The aforementioned analytical methods are largely based on labor-intensive manual annotations. The fatigue caused by prolonged detailed annotations may introduce inconsistencies in annotation quality, and they overlook the potential role of radiology report information that contains rich guiding prior knowledge and is easily accessible.

3. Methods

The overall architecture of VMD is illustrated in Figure 4. We employed pre-trained models with frozen weights as feature encoders for the Expert network E (dark orange), teacher network T (green), and the weaker-performing student network S (blue), and fine-tuned them using different fully connected layers. The primary objective was to reduce the cost of training the network in the early stages of learning 3D images and report modalities. This strategy has been repeatedly proven effective in medical imaging tasks. Then we applied amortization techniques and reparameterization [27] to get the prior $\left(x^T\right)$ and prior $\left(x^S\right)$ for both T and S. We used a two-layer MLP whose latent feature size is 512 to play the role of z. Using techniques akin to those for image input, we computed the posterior $\left(x^E\right)$.

Figure 4. The overview of VMD. (a) is the transverse section of the original 3D MRI image. The carotid artery region (b,d) for the student network is obtained through cropping. The input (e) for the teacher network is the product of (d) and the limited annotations (c). The radiology report (f) corresponding to the image is for the expert network. When training VMD, we maximize the MI and minimize the KL divergence between the latent spaces z of different networks. During the inference phase, there is no guidance for the student to diagnose the plaque vulnerability.

In order to efficiently transfer cross-modal prior knowledge, we needed to maximize the MI between different student–teacher pairs [18]. During the optimization process, we also minimized the cross-entropy $H_{cls}$ of all networks as much as possible to make the prediction approach the true value. The global optimization objective $\mathcal{O}$ is shown in (1), ${\alpha }_1$, ${\alpha }_2$, ${\alpha }_3$, ${\alpha }_4$ are weights for the corresponding terms.

$$\mathrm{argmax}\left(\mathcal{O}\right)={\alpha }_1\mathcal{I}\left(S,T\right)+{\alpha }_2\mathcal{I}\left(S,E\right)+{\alpha }_3\mathcal{I}\left(T,E\right)-{\alpha }_4{H}_{cls}$$

(1)

where $\mathcal{I}$ means MI. We will sequentially solve for each term of the equation in the following sections.

3.1. Cross-Modal Knowledge Transfer from Reports: $\mathcal{I}\left(S,E\right)$

Conditional variational inference [31] is highly suitable for structured prediction tasks, and VKD [21] has validated its effectiveness in understanding medical image analysis tasks. As shown in Figure 5, the student, teacher, and expert share similar inference structure modules, each consisting of a two-layer MLP to compute the mean and standard deviation, followed by the reparameterization trick. We construct a variational probability model and formulate the problem of predicting the target y from the unannotated image $x^S$ as solving the posterior probability $p\left(y|x_I^S\right)$. In order to obtain the following conditional likelihood formula, we introduced the latent variable $z^S$:

$$\begin{array}{c}\hfill \log p\left(y|x_I^S\right)=\log \int p\left(y|x_I^S,z_I^S\right)p\left(z_I^S|x_I^S\right)d{z}_I^S\end{array}$$

(2)

Figure 5. Example of the modified variational inference module. The student, teacher, and expert share similar inference structure modules, fine-tuning feature parameters from different pre-trained networks through a two-layer MLP, calculating the mean and standard deviation, respectively, and then ensuring gradient backpropagation through the reparameterization trick.

The $p\left(z_I^S|x_I^S\right)$ is the conditional prior for $x^S$, which is easy to obtain by the neural network, as shown in Figure 4 denoted as prior $\left(x^S\right)$. The prediction problem can be translated into maximizing this integral w.r.t. z. Thus, we needed to find the maximized posterior probability $p\left(z^S|x^S,y\right)$, which is relatively difficult to solve. Inspired by [21], we introduced the representation $z^E$ from the expert network and used the variational posterior distribution $q\left(z^E|x^E\right)$ to approximate $p\left(z^S|x^S,y\right)$. Given that the KLD is well-behaved in the distance metrics [27], we can obtain $kl{d}_E^S$ in Figure 4 by applying Bayesian formulae given the weight $\theta$ of the networks, and translating the original problem into minimizing the KLD between $q\left(z^E|x^E\right)$ and $p\left(z^S|x^S\right)$:

$$\begin{array}{cc}\hfill kl{d}_E^S& =D_{KL}\left[q_{\theta }\left(z^E|x^E\right)|\left|p\left(z^S|x^S,y\right)\right]\right.\hfill \\ & =\int q_{\theta }\left(z^E|x^E\right)\log {\displaystyle \frac{q_{\theta }\left(z^E|x^E\right)}{p\left(z^S|x^S,y\right)}}d\theta \hfill \\ & =\log p\left(y|x^S\right)+\int q_{\theta }\left(z^E|x^E\right)\log p\left(z^S|x^S,y\right)d\theta \hfill \\ & +\int q_{\theta }\left(z^E|x^E\right)\log {\displaystyle \frac{p\left(y|x^S\right)}{q_{\theta }\left(z^E|x^E\right)}}d\theta \hfill \\ & =\log p\left(y|x^S\right)-{\mathbb{E}}_{q_{\theta }\left(z^E|x^E\right)}\left[\log p\left(y|x^S,z^S\right)\right]\hfill \\ & +D_{KL}\left[q_{\theta }\left(z^E|x^E\right)\left|\right|p\left(z^S|x^S\right)\right]\hfill \end{array}$$

(3)

Since the KL divergence is non-negative ($D_{KL}\ge 0$), we can derive the evidence lower bound (ELBO) on $\log p\left(y\right|x^S)$ as shown in (4). Maximizing this ELBO minimizes the KL divergence between the variational posterior $q\left(z^E\right|x^E)$ and the true posterior $p\left(z^S\right|x^S,y)$, thereby transferring knowledge from the expert network to the student network.

$$\begin{array}{cc}\hfill \log p\left(y|x^S\right)& \ge {\mathbb{E}}_{q_{\theta }\left(z^E|x^E\right)}\left[\log p\left(y|x^S,z^S\right)\right]\hfill \\ & -D_{KL}\left[q_{\theta }\left(z^E|x^E\right)\left|\right|p\left(z^S|x^S\right)\right]\hfill \end{array}$$

(4)

From an information-theoretic perspective, maximizing the ELBO serves as a surrogate for maximizing the mutual information $\mathcal{I}(S,E)$ between the student and expert networks. Specifically, the ELBO provides a tractable lower bound for the mutual information, and we optimize this bound as follows:

$$\begin{array}{cc}\hfill \mathcal{I}\left(S,E\right)& =ELB{O}_{SE}={\mathbb{E}}_{q\left(z^E|x^E\right)}\left[\log p\left(y|x^S,z^S\right)\right]\hfill \\ & -D_{KL}\left[q\left(z^E|x^E\right)\left|\right|p\left(z^S|x^S\right)\right]\hfill \end{array}$$

(5)

3.2. Maximizing Mutual Information Based on Contrastive Learning: $\mathcal{I}\left(S,T\right)$

For the homologous annotated $x^T$ and unannotated $x^S$, suppose a pair of predictions ${\widehat{y}}_j^T$ and ${\widehat{y}}_i^S$ derived from T and S, we aim to minimize the distance between predictions with the same class label and maximize the gap between predictions with different classes for all N samples. To maximize the MI between S and T, we consider applying a similarity-based model $M\left({\widehat{y}}_j^T,{\widehat{y}}_i^S\right)$ inspired by [26] to quantify the difference between ${\widehat{y}}_j^T$ and ${\widehat{y}}_i^S$, shown in (6).

$$M\left({\widehat{y}}_j^T,{\widehat{y}}_i^S\right)={\displaystyle \frac{\exp \left(\varphi \frac{{\widehat{y}}_j^T,{\widehat{y}}_i^S}{\tau }\right)}{{\displaystyle \sum _{k=1}^N}\exp \left(\varphi \frac{\left({\widehat{y}}_j^T,{\widehat{y}}_k^S\right)}{\tau }\right)}}$$

(6)

where $\varphi$ is a cosine similarity scoring function computed on ${\mathsf{\ell }}_2$-normalized feature vectors, and temperature $\tau$ is set empirically to $0.5$. As illustrated in Figure 6, this similarity-based optimization aligns the student’s predicted distribution with the teacher’s better-fitted feature distribution, while the same principle applies to the expert’s constraint on the teacher. To maximize the mutual information between the teacher and student networks, we employ a supervised contrastive loss inspired by the InfoNCE objective [24]. Unlike the standard InfoNCE, which constructs positive pairs from different augmented views of the same sample, our formulation leverages class labels as the supervisory signal. Specifically, for a batch of N samples, B denotes the number of samples sharing the same class label as the anchor sample j (positive samples), while $N-B$ denotes the number of samples with different class labels (negative samples). The batch is constructed using a class-balanced sampling strategy, where each class contributes approximately half of the samples, ensuring that the contrastive loss is well-defined and stable during training. The optimization objective is formulated as shown in (7),

$$\begin{array}{cc}\hfill \mathcal{I}\left(S,T\right)& ={\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=1}^N}{\displaystyle \sum _{i=1}^B}\log M\left({\widehat{y}}_j^T,{\widehat{y}}_i^S\right)\hfill \\ & +{\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=1}^N}{\displaystyle \sum _{k=1}^{N-B}}\log \left(1-M\left({\widehat{y}}_j^T,{\widehat{y}}_k^S\right)\right)\hfill \end{array}$$

(7)

Figure 6. Example of the contrastive-based inter-class mutual information optimization strategy. The student’s predicted class-level output for the input sample is further constrained by the teacher, which can be viewed as aligning the underfitted distribution curve with the better-fitted feature curve through similarity optimization. The expert’s constraint on the Teacher follows the same principle.

3.3. The Further Constraint on the Teacher: $\mathcal{I}\left(T,E\right)$

The further constraint on the teacher stems from both variational inference and contrastive learning. Therefore, we first computed the KLD between the latent spaces z based on (5) and then computed the similarity between the predictions based on (7). It is noteworthy that T and S share the same parameters in the classification head.

Equation (8) shows the summary of $\mathcal{I}\left(T,E\right)$.

$$\begin{array}{cc}\hfill \mathcal{I}\left(T,E\right)& ={\mathbb{E}}_{q\left(z^E|x^E\right)}\left[\log p\left(y|x^T,z^T\right)\right]\hfill \\ & -D_{KL}\left[q\left(z^E|x^E\right)\left|\right|p\left(z^E|x^T\right)\right]\hfill \\ & +{\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=1}^N}{\displaystyle \sum _{i=1}^B}\log M\left({\widehat{y}}_j^T,{\widehat{y}}_i^E\right)\hfill \\ & +{\displaystyle \frac{1}{N}}{\displaystyle \sum _{j=1}^N}{\displaystyle \sum _{k=1}^{N-B}}\log \left(1-M\left({\widehat{y}}_j^T,{\widehat{y}}_k^E\right)\right)\hfill \end{array}$$

(8)

At last, we used the minimized cross-entropy loss function ${\mathcal{L}}_{ce}$ to further obtain more accurate classifications for all three parts of VMD, as shown in (9), ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$ are different weights for these terms.

$$H_{cls}={\lambda }_1{\mathcal{L}}_{ce}\left({\widehat{y}}_i^T,y_i\right)+{\lambda }_2{\mathcal{L}}_{ce}\left({\widehat{y}}_i^S,y_i\right)+{\lambda }_3{\mathcal{L}}_{ce}\left({\widehat{y}}_i^E,y_i\right)$$

(9)

Through our reasoning, modified variational inference and contrastive-based on similarity optimization have been well integrated from the perspective of MI optimization.

Substituting the above reasoning results into (1) yields the final optimization objective of VMD. The training procedure of VMD is summarized in Algorithm 1, and the inference procedure for the student network is described in Algorithm 2.

Algorithm 1: Training of VMD

Algorithm 2: Inference of VMD’s student

4. Experiment

4.1. Dataset

Patients diagnosed with carotid stenosis according to the 2022 guidelines of the European Society for Vascular Surgery [34], who were hospitalized in the Department of Neurosurgery at Capital Medical University Xuanwu Hospital from September 2021 to September 2023, were retrospectively included. The experiment incorporated 502 carotid artery samples from 303 patients, including 350 vulnerable and 152 stable samples. Inclusion criteria were (1) age 45–85 years; (2) presence of plaque in unilateral or bilateral carotid artery determined by ultrasonography (wall thickening > 1.5 mm) [11]; (3) no contraindications to MR or contrast agent injection. Exclusion criteria were (1) history of carotid endarterectomy, carotid stenting, or neck radiotherapy; (2) history of tumor or chemotherapy; (3) recent acute or chronic infection, autoimmune disease, or ongoing anti-inflammatory treatment; (4) poor image quality. This study adhered to the Declaration of Helsinki and was approved by the Ethics Committee of Xuanwu Hospital of Capital Medical University, exempting subjects from informed consent, with the approval number: [2022]023.

The selected high-quality images are categorized into average quality and good quality based on the presence of issues such as the lumen and the external contour of the blood vessel being locally blurred, with few artifacts that do not affect the diagnosis. There are 153 images of average quality and 349 images of good quality. Images with severe motion artifacts, poor contrast-to-noise ratio, or incomplete coverage of the carotid bifurcation that compromised diagnostic interpretability were excluded prior to the final analysis. The preprocessing pipeline was applied uniformly to all 502 images. First, each 3D MRI volume was normalized using Z-score standardization computed from the voxel intensities within the carotid artery region to ensure consistent intensity scaling across different scans. All images were then cropped to a volume of (128, 128, 60) based on the scanning center, which was automatically determined from the geometric center of the field of view, ensuring fully deterministic and reproducible cropping. The images were resampled to a standardized spacing of 0.60 × 0.60 × 0.60 mm³ using trilinear interpolation, consistent with the original acquisition parameters. During training, online data augmentation was applied to improve generalization, including random rotation (±10 degrees), random translation (±5 voxels), random scaling (±10%), random flipping (horizontal and vertical), and random elastic deformation ($\mathsf{\sigma }$ = 4 pixels), with all augmentations applied to the 3D volumes and their corresponding annotations. The imaging was performed using a 3.0T PET/MR (United Imaging uPMR790) with an 8-channel carotid-specific surface coil. Scanning was centered on the carotid bifurcation. Sequence parameters were TR = 800 ms, TE = 14.52 ms, FOV = 180 mm × 180 mm, resolution = 0.60 mm × 0.60 mm × 0.60 mm, number of layers = 390.

Two experienced radiologists (>10 years of experience in neuroradiology) collaboratively classified the images of each vessel as either “vulnerable” or “stable”, in which plaques containing LRC, IPH, thrombus, and ulceration are marked as vulnerable, as indicated in Table 1. It is important to clarify that the radiology reports used in our expert network are descriptive clinical narratives that document imaging findings (e.g., vessel morphology, signal intensity characteristics, stenosis measurements) rather than structured diagnostic conclusions. To further mitigate any potential risk of label leakage, all explicit diagnostic statements (e.g., “vulnerable plaque”, “high-risk lesion”) were removed from the reports during preprocessing. The vulnerability labels (vulnerable vs. stable) were independently assigned by two experienced radiologists specifically for this research study, based solely on established MRI criteria for vulnerable plaques (presence of LRC, IPH, thrombus, or ulceration) and were not derived from the clinical reports. This design ensures that the expert network provides descriptive imaging features to guide the student network, without directly revealing the diagnostic label. Importantly, to prevent data leakage, all dataset splitting (training/validation/test) was performed strictly at the patient level. Specifically, the 303 patients were first partitioned into training, validation, and test sets, and then the corresponding carotid artery samples (left and right) from each patient were assigned to the same set. This ensures that bilateral arteries from the same patient never appear in different sets simultaneously. For patients with bilateral carotid artery samples, both arteries were treated as independent analytical units for model training and evaluation. And the training and test sets are separated using a ratio of 5:1, and 5-fold cross-validation was employed for model training and validation, ensuring that the independent test set was not visible during the training phase.

Table 1. Detail descriptions of train, validation, and test set. ¹: Age in years; ²: CB = carotid bifurcation, ICA = internal carotid artery, CCA = common carotid artery; ³: mild (<50%), moderate (50–69%), severe (≥70%); ⁴: LRC = lipid-rich necrotic core, CA = calcification, IPH = intraplaque hemorrhage, UC = ulceration, TB = thrombus.

Dataset	Number	Age 1	Gender		Location 2			Stenosis Degree 3			Components 4
			Male	Female	CB	ICA	CCA	Mild	Moderate	Severe	LRC	CA	IPH	UC	TB
train	338	65.76 ± 6.96	294	44	305	29	4	131	152	55	166	178	63	36	0
val	82	65.11 ± 7.02	70	12	77	5	0	31	45	6	42	44	21	22	2
test	82	62.8 ± 7.54	78	4	77	3	2	28	38	16	45	40	16	11	3

4.2. Implementation

We first employed MedicalNet [35], pre-trained on 23 CT and MRI public datasets, as the visual encoder backbone network and used a two-layer MLP to fine-tune it. For the expert network E, we applied Mc-BERT [29], a Chinese medical semantic understanding BERT-base network pre-trained in a considerably large Chinese biomedical text corpus [36], as the expert encoder and used one single transformer layer to fine-tune it. All experiments were performed in two NVIDIA A40 GPUs, using the Adam optimizer for 400 epochs. The learning rate is initialized to $5\times {10}^{-4}$ and weight decay is $1\times {10}^{-4}$. Five random seeds were used to calculate the average and standard deviation of the evaluation metrics. The $\beta$ of Fbeta score is 0.5. It should be noted that the SOTA comparison (Table 2) and the ablation studies (Table 3 and Table 4) were conducted with independent random seed sets to ensure robustness. Specifically, the “w/o both” baseline in Table 4 refers to the student network trained without any distillation guidance, while the teacher-related ablation results are presented in Table 3. These configurations are inherently different, and the minor numerical variations between the two tables are within the expected stochastic variance of neural network training.

Table 2. Comparison of classification performance with SOTA methods, bold represents the best.

Setting	Accuracy	Precision	Recall	Fbeta ($\mathit{\beta }=0.5$)	AUC
Baseline [35]	0.6588 ± 0.0007	0.7802 ± 0.0055	0.7213 ± 0.0063	0.7487 ± 0.003	0.7550 ± 0.008
VPT [37]	0.6195 ± 0.0055	0.7044 ± 0.0074	0.7966 ± 0.0077	0.7476 ± 0.0009	0.7150 ± 0.010
VID [28]	0.7171 ± 0.0055	0.7959 ± 0.0016	0.8069 ± 0.0077	0.8014 ± 0.0046	0.8020 ± 0.009
VKD [21]	0.678 ± 0.0185	0.8065 ± 0.0203	0.7172 ± 0.0094	0.7592 ± 0.0123	0.7650 ± 0.014
CCL [25]	0.6361 ± 0.0044	0.778 ± 0.0045	0.6779 ± 0.0123	0.7238 ± 0.0035	0.7450 ± 0.008
VMD	0.7244 ± 0.0204	0.8197 ± 0.0192	0.7828 ± 0.0094	0.8008 ± 0.0135	0.8120 ± 0.015

4.3. Evaluations

We compared the diagnostic performance of plaque vulnerability between VMD and other methods by comparing the AUCs. When comparing with junior doctors (<2 years experience in neuroradiology), we used the professional statistical analysis software SPSS 25.0 for analysis, conducting T-tests on the diagnostic accuracy and time consumption between VMD and the doctors, with a p-value less than 0.05 considered statistically significant.

5. Results

We first compare VMD with the baseline method, three state-of-the-art knowledge distillation methods, and a network fine-tuning method, namely VID [28], VKD [21], CCL [25], and VPT [37], as shown in Table 2. We employed the fully fine-tuned 3DResNet-50 network as our classification baseline, a model derived from MedicalNet. Additionally, we used the grad-CAM method [38] to visualize the feature distribution of the terminal 3D convolutional layer in the student network of VMD during the inference diagnosis process, as shown in Figure 7. Secondly, we compare our method with the diagnostic results of junior doctors, demonstrating the effectiveness of VMD in terms of diagnostic efficiency and accuracy. Finally, we conducted ablation studies to validate the significance of the teacher–expert module in VMD.

Figure 7. Grad-CAM [38] visualization of the features from the final convolutional layer of the student network. (A–D) Four representative test samples. For each sample, 6 consecutive 2D slices from the middle of the 3D volume are shown. In each sample, the first row shows the heatmaps (red: high attention, blue: low attention), the second row shows the original images, and the third row shows the overlay of both. The visualized feature distributions are predominantly concentrated within the carotid vessel regions and their immediate surroundings.

5.1. Diagnosis Feature Visualization

In the prediction phase, the feature distribution of the last bottleneck layer in the image encoder of the VMD student network was visualized using the Grad-CAM [38], as shown in Figure 7. We selected four representative images and displayed the middle six layers of each 3D image. The first row in each of the four subplots is the heatmap, with red representing areas of high network attention and blue representing areas of low attention; the second row is the original image; the third row is the overlay of the above two images. The visualized feature distributions indicate that the student network’s attention is predominantly concentrated within the carotid vessel regions and their immediate surroundings, which correspond to the clinically relevant areas for plaque vulnerability assessment.

5.2. Comparison with Junior Radiologists

In terms of deep learning-based automated methods, we deployed our VMD network on an Nvidia A40 graphics card and obtained the trained student network as the inference model for comparative experiments. On the other hand, we included two junior radiologists to perform the same diagnostic task for comparison. We selected 54 cases from the independent test set for the experiment, including 36 vulnerable and 18 stable artery samples, with the distribution ratio of the two types of images being the same as the training set. The two junior radiologists were asked to diagnose 27 randomly selected samples from the 54 cases. By recording the diagnostic efficiency and accuracy of both VMD and the radiologists, we demonstrated the effectiveness of our method. Statistical methods were used for significance analysis.

The diagnosis process for 27 test images took an average of 2622.5 ± 591.8484 s for junior doctors, with each image diagnosis taking 81.6296 ± 19.4188 s and 112.6296 ± 71.374 s; the average diagnostic accuracy of the two junior doctors was 0.6111 ± 0.0786. Based on the diagnostic results of five random seeds, VMD’s student model took 20.2628 ± 0.2057 s to read the 54 images, 0.7915 ± 0.0826 s to load the model, and 17.9684 ± 0.4708 s to diagnose, with a total time of 39.0227 ± 0.3771 s to finish the diagnosis of 54 cases. The average time to diagnose an image was 0.2659 ± 0.0991 s, and the accuracy rate was 0.7481 ± 0.0101, as shown in Figure 8.

Figure 8. Comparative evaluation of diagnostic performance between the VMD student model and junior radiologists on the independent carotid 3D MRI test set. VMD achieved superior performance in both time efficiency (a) and diagnostic accuracy (b), with all differences being statistically significant. ** $p<0.01$, **** $p<0.0001$.

5.3. Ablation Study

In VMD, the multi-level distillation knowledge transfer to the student comes from both the teacher and the expert, and can be divided into two parts based on direct and indirect effects. The direct effects include the feature distribution constraints between the teacher and the student, and between the expert and the student, which are marked as $\mathcal{I}\left(S,E\right)$ and $\mathcal{I}\left(S,T\right)$ in the methods section, corresponding to (5) and (7). The indirect effects involve the feature distribution constraints through the expert’s limitation on the teacher, which is denoted as $\mathcal{I}\left(T,E\right)$ from (8). The teacher learns richer prior knowledge from the expert, enhancing its diagnostic efficiency and indirectly improving the knowledge transfer level to the downstream student, achieving stricter feature distribution migration.

To validate the roles of the teacher network and expert network in VMD, we designed two ablation experiments for the teacher network and the student network respectively.

Firstly, in Table 3, we aim to verify that the knowledge transfer from the expert can effectively improve the teacher model’s performance in diagnosing plaque vulnerability in MRI images with limited annotations. We ablated (8), and as shown in Table 3, after the expert network from radiology reports participated in the knowledge transfer during the training phase, the teacher achieved substantial performance improvements under the dual optimization of variational inference distribution constraints and similarity-based inter-class mutual information optimization. Specifically, the teacher model improved its accuracy from 0.6732 to 0.7829 (a relative improvement of approximately 16.3%), recall from 0.7130 to 0.7930 (approximately 11.2%), Fbeta from 0.7505 to 0.8079 (approximately 7.6%), and AUC from 0.7400 to 0.8300 (approximately 12.2%). This result indicates that our distillation strategy effectively realizes our hypothesis that the text information in radiology reports, often overlooked in MRI diagnostic intelligent networks, can provide significant assistance to multimodal intelligent diagnostic networks.

Table 3. Ablation study for teacher w/o expert (8) to diagnose annotated images. ✓ indicates with, × indicates without. Bold indicates the best performance.

Limited Annotations	Report	Setting	Accuracy	Precision	Recall	Fbeta	AUC
✓	×	w/o expert	0.6732 ± 0.0134	0.8151 ± 0.0101	0.7130 ± 0.0150	0.7505 ± 0.018	0.7400 ± 0.012
✓	✓	teacher	0.7829 ± 0.0181	0.8222 ± 0.0487	0.7930 ± 0.0250	0.8079 ± 0.0344	0.8300 ± 0.020

Table 3. Ablation study for teacher w/o expert (8) to diagnose annotated images. ✓ indicates with, × indicates without. Bold indicates the best performance.

Limited Annotations	Report	Setting	Accuracy	Precision	Recall	Fbeta	AUC
✓	×	w/o expert	0.6732 ± 0.0134	0.8151 ± 0.0101	0.7130 ± 0.0150	0.7505 ± 0.018	0.7400 ± 0.012
✓	✓	teacher	0.7829 ± 0.0181	0.8222 ± 0.0487	0.7930 ± 0.0250	0.8079 ± 0.0344	0.8300 ± 0.020

Subsequently, we conducted ablation experiments on the diagnostic performance of the student on unannotated 3D carotid MRI images in the VMD training strategy to demonstrate the positive role of knowledge transfer from the teacher and expert networks, as shown in Table 4. Equation (5) represents the expert’s individual guidance to the student; (7) represents the teacher’s individual guidance to the student.

Table 4. Ablation study for student w/o teacher (7), (8) and expert (5), (8) to diagnose unannotated images. ✓ indicates with, × indicates without. Bold indicates the best performance.

(5)	(7)	(8)	Setting	Accuracy	Precision	Recall	Fbeta	AUC
×	×	×	w/o both	0.6537 ± 0.0109	0.7859 ± 0.0181	0.7034 ± 0.0463	0.7412 ± 0.0197	0.7200 ± 0.015
✓	×	×	w/o teacher	0.678 ± 0.0185	0.8065 ± 0.0203	0.7172 ± 0.0094	0.7592 ± 0.0123	0.7450 ± 0.014
×	✓	×	w/o expert	0.6512 ± 0.0164	0.76 ± 0.0175	0.7421 ± 0.0016	0.7505 ± 0.0087	0.7300 ± 0.012
✓	✓	✓	VMD	0.722 ± 0.0181	0.8191 ± 0.0186	0.7793 ± 0.0077	0.7987 ± 0.0115	0.8120 ± 0.015

Table 4. Ablation study for student w/o teacher (7), (8) and expert (5), (8) to diagnose unannotated images. ✓ indicates with, × indicates without. Bold indicates the best performance.

(5)	(7)	(8)	Setting	Accuracy	Precision	Recall	Fbeta	AUC
×	×	×	w/o both	0.6537 ± 0.0109	0.7859 ± 0.0181	0.7034 ± 0.0463	0.7412 ± 0.0197	0.7200 ± 0.015
✓	×	×	w/o teacher	0.678 ± 0.0185	0.8065 ± 0.0203	0.7172 ± 0.0094	0.7592 ± 0.0123	0.7450 ± 0.014
×	✓	×	w/o expert	0.6512 ± 0.0164	0.76 ± 0.0175	0.7421 ± 0.0016	0.7505 ± 0.0087	0.7300 ± 0.012
✓	✓	✓	VMD	0.722 ± 0.0181	0.8191 ± 0.0186	0.7793 ± 0.0077	0.7987 ± 0.0115	0.8120 ± 0.015

On one hand, compared to the baseline method without guidance, i.e., the 3D convolutional diagnostic network simply fine-tuned, the ablation results of both individual guidances showed significant improvements and outperformed the baseline method across all reported metrics. This indicates that both guidances provided effective knowledge transfer for improving diagnostic efficiency in the unannotated diagnostic task.

On the other hand, the ablation results of both individual guidances were inferior to the fully guided training strategy of VMD. Specifically, compared with the baseline (w/o both), VMD achieved consistent performance improvements across all five metrics: Accuracy improved by 10.5% (from 0.6537 to 0.7220), Precision by 4.2% (from 0.7859 to 0.8191), Recall by 10.8% (from 0.7034 to 0.7793), Fbeta by 7.8% (from 0.7412 to 0.7987), and AUC by 12.8% (from 0.7200 to 0.8120). This suggests that the feature transfer from the teacher and expert did not result in feature conflicts in improving diagnostic performance, proving the correctness of the inference that both optimization approaches mentioned in the methods section can be summarized as mutual information optimization strategies.

6. Discussion

In this study, we proposed a novel multi-level cascaded multimodal knowledge distillation network VMD, which is based on MI maximization, providing an intelligent method for the diagnosis of plaque vulnerability in carotid 3D MRI, which relies less on manual annotations by radiologists. This structure allowed for better extraction of feature information beneficial for disease diagnosis from both text and image modalities without introducing additional inference costs. Thanks to the diagnostic knowledge derived from cross-modal variational inference and the anatomical prior gained by maximizing inter-class MI, our method achieved the best in most evaluation metrics and is only slightly different from the best results in Recall and Fbeta. Also, while ensuring diagnostic accuracy, it also achieved superior diagnostic efficiency. With further validation, this approach has the potential to assist junior doctors in the early diagnosis of plaques in patients with carotid stenosis, though prospective clinical evaluation is required before deployment.

In recent years, some research has also provided solutions to plaque detection and component recognition-related problems based on deep learning methods [11,15,16]. For example, Chen et al. [11] constructed a domain adaptive quality assessment and lesion classification model for 3D-MERGE sequence images, but limited the classification target of plaques to the more vague early lesions and late lesions, rather than the clinically more concerned diagnosis of plaque vulnerability; Wang et al. [15] constructed an end-to-end plaque component segmentation model that can be used in the joint diagnosis process of multiple sequence MRIs, achieving the segmentation and recognition of IPH, LRC, and CA in carotid plaques, but in the inference phase, it requires strict joint images of multiple sequences as inputs. Compared to the above research, VMD focuses more on the full use of more easily obtained multimodal information from reports and images, combined with advanced variational inference optimization ideas, redefines the problem of plaque vulnerability diagnosis as an optimization problem of feature distance, thus using the rich domain knowledge contained in clinically very easy to obtain radiology reports to improve the student model’s understanding of plaque 3D MRI, and at the same time does not need any additional annotations in the inference phase, only need to input the original 3D MRI into the model to get the diagnosis result, which may offer advantages in future clinical deployment if further validated on larger, multi-center datasets.

The essence of distillation learning can be understood as the learning of feature distribution between models [17,25,26]. Some previous student–teacher-based distillation networks focus on learning the mapping across modalities [19,20,25,26]. VMD takes into account the stability of anatomical structures in MRI images and the precise mapping of vascular trends. The same type of plaques in the same part have more similar image representations. It uses limited annotations of the vessel wall and lumen to provide more local auxiliary information, and combines the similarity optimization method of contrastive learning to narrow the distance of corresponding features in the network of the same type of plaque images, thus achieving further mining of the potential ability of the encoder to process image features. In the clinical comparison results, VMD demonstrates superior diagnostic efficiency compared with junior doctors, achieving a 15% higher diagnostic accuracy rate. In the early screening stage, this approach may potentially provide decision support for identifying high-risk patients, although this remains speculative until prospective clinical validation is performed.

Our research also has some limitations. 1: Our research included patient data from a single center and the scanning modality was also limited to single-sequence images. In the future, more in-depth research needs to be conducted in multi-center data to eliminate the error impact brought by scanning modalities, parameters, and population. 2: At present, VMD only directly classifies images into vulnerable and stable categories. In the future, the diagnostic target needs to be further expanded, such as diagnosing the risk coefficient by combining more levels of carotid feature information. 3: VMD still includes manual annotation of limited vessel areas during the training phase. In the future, a lower-cost diagnostic model should be constructed by combining multi-sequence complementary information that is beneficial for plaque component identification. 4: The patient data we included were all patients with carotid stenosis. In the future, the proportion of healthy subjects should be expanded to further extend the boundaries of the VMD’s ability to automate disease diagnosis. 5: The test set contains only 82 images with an inherent class imbalance, which may limit the generalizability of our findings. Furthermore, the five random seeds used in our experiments reflect variability due to different initializations and training dynamics rather than independent test replications; the small standard deviations indicate stable convergence behavior of VMD, but future work would benefit from cross-site external validation with larger, more balanced cohorts. Despite these limitations, the statistical significance of our results is supported by the comparison with junior radiologists (p < 0.05), where VMD demonstrated both superior accuracy and efficiency. Future work will involve multi-center data collection and prospective studies to further validate the clinical utility of our framework. 6: We acknowledge that the current accuracy level (approximately 0.72–0.75) has not been prospectively validated in clinical settings. The performance reported in this study reflects methodological proof-of-concept rather than readiness for clinical implementation. Future work involving prospective clinical trials and external validation on independent, multi-center datasets is essential to rigorously assess the added value of VMD in real-world clinical workflows.

7. Conclusions

The task of automated diagnosis of vulnerable plaques in carotid 3D MRI is challenging. In this study, we proposed a student–teacher–expert distillation framework named VMD for the automated diagnosis of carotid 3D MRI, which achieves a significant improvement in the diagnosis of atherosclerotic plaque vulnerability. By combining variational inference and contrastive learning-based MI optimization theory, we efficiently transfer the cross-modal information from the teacher network and the expert network to the student network, without introducing additional inference costs. However, we acknowledge that the current results are based on a single-center dataset, which may limit the generalizability of our findings. Future work will involve external validation on multi-center, multi-scanner datasets to further assess the robustness and clinical applicability of our proposed framework.