ADCNet: Anomaly-Driven Cross-Modal Contrastive Network for Medical Report Generation

Abstract

Medical report generation has made significant progress in recent years. However, generated reports still suffer from issues such as poor readability, incomplete and inaccurate descriptions of lesions, and challenges in capturing fine-grained abnormalities. The primary obstacles include low image resolution, poor contrast, and substantial cross-modal discrepancies between visual and textual features. To address these challenges, we propose an Anomaly-Driven Cross-Modal Contrastive Network (ADCNet), which aims to enhance the quality and accuracy of medical report generation through effective cross-modal feature fusion and alignment. First, we design an anomaly-aware cross-modal feature fusion (ACFF) module that introduces an anomaly embedding vector to guide the extraction and generation of anomaly-related features from visual representations. This process enhances the capability of visual features to capture lesion-related abnormalities and improves the performance of feature fusion. Second, we propose a fine-grained regional feature alignment (FRFA) module, which dynamically filters visual and textual features to suppress irrelevant information and background noise. This module computes cross-modal relevance to align fine-grained regional features, ensuring improved semantic consistency between images and generated reports. The experimental results from the IU X-Ray and MIMIC-CXR datasets demonstrate that the proposed ADCNet method significantly outperforms existing approaches. Specifically, ADCNet achieves notable improvements in natural language generation metrics, as well as the accuracy, completeness, and fluency of medical report generation.

1. Introduction

Medical report generation (MRG) utilizes advanced algorithms to automatically extract key information from medical images and generate the corresponding reports. These advances not only help clinicians to efficiently review and diagnose medical conditions, but also contribute to the broader automation of the medical imaging workflow. Similarly to the image captioning task, the main approaches in MRG are typically based on the encoder–decoder architecture, using Convolutional Neural Networks (CNNs) to extract visual features and Recurrent Neural Networks (RNNs), or their variants, to generate reports [1,2,3]. Although this method successfully captures the coarse-grained features and basic semantic structures of the images, the generated reports tend to be brief, with limited information coverage, often failing to describe subtle or complex abnormalities present in the images. In recent years, the Transformer model [4], which has gained widespread use in Natural Language Processing (NLP), has been increasingly integrated into MRG models [5,6,7,8]. The Transformers’ ability to capture long-distance dependencies and model rich contextual relationships within data has led to significant improvements in the quality of generated reports. Reports produced by Transformer-based models are typically more detailed and comprehensive, enhancing the clinical value of the generated content. The integration of attention mechanisms, particularly self-attention and cross-attention [3], allows the model to dynamically adjust its focus across different features, enabling a more effective extraction of key information from medical images and improving the accuracy of visual feature representation [6,7]. Furthermore, cross-modal alignment techniques such as cross-attention and gating mechanisms have been developed to promote the alignment of visual and textual modalities, enhancing the semantic consistency between the two. These techniques allow the model to better understand the relationships between the image content and the corresponding text, thereby improving the model’s ability to generate accurate, contextually relevant and coherent medical reports. As a result, with the advancement of these methods, substantial progress has been made in improving the accuracy, completeness, and readability of the reports generated in MRG. These contributions not only help clinicians in the efficient review and diagnosis of medical conditions but also aid in the general automation of the medical imaging workflow, streamlining the healthcare process, and reducing human error.

However, several challenges remain in this field. Due to the inherent nature of medical images and reports, cross-modal feature interaction and alignment in this visual-language task are particularly difficult. First, the intrinsic properties of medical images complicate the task of distinguishing visual features. Zhou et al. [9] pointed out that medical images often have low contrast and blurry boundaries, while Wang et al. [10] highlighted that radiographic images exhibit both intra-class variation, where the same lesion appears differently across individuals, and inter-class similarity, where different types of lesions appear visually similar. As shown in Figure 1a, images of two diseases are displayed. It is evident that these two types of images share certain similarities, making them difficult to distinguish and visually vague. Second, the proportion of lesion areas in medical images and their corresponding textual descriptions is relatively small, making it challenging to extract and learn the features necessary for identifying abnormal lesions. Additionally, in commonly used datasets such as IU X-Ray [11] and MIMIC-CXR [12], the distribution of diseases is highly imbalanced, as illustrated in Figure 1b. This imbalance can cause deep neural network models to prioritize descriptions of normal conditions and overemphasize certain diseases while neglecting others. In clinical practice, missing the diagnosis of diseases or abnormalities can have severe consequences.

Earlier research typically modeled cross-modal interaction using self-attention mechanisms applied to the visual and textual features extracted from the encoder–decoder architecture. However, this approach struggles to fully capture complex cross-modal patterns. Recent studies have shifted focus to contrastive learning strategies, which bring the representations of matching image–text pairs closer together while pushing non-matching pairs further apart. This enables the model to learn stronger cross-modal alignment in a unified feature space, thereby improving the accuracy and consistency of report generation. However, in medical imaging, a single instance may contain multiple types of lesions or abnormalities, making coarse-grained contrastive learning less effective at capturing subtle variations between samples within the same category. Traditional contrastive learning methods typically compare entire images or full text segments as instances, overlooking the importance of local features such as specific lesions or tissue regions. As shown in Figure 1a, the feature heatmap is not clearly distinguishable, and the extracted image features fail to properly attend to different regions. Similarly, the text features exhibit the same issue, with words of little significance, such as prepositions, receiving disproportionate attention. This misalignment in feature representation leads to inaccuracies in graphic similarity. For example, while image 1 and image 3 may appear highly similar, they correspond to different diseases. Therefore, there is a need to develop fine-grained feature matching methods that leverage information from local lesion regions, enhancing the model’s ability to focus on relevant areas and improving the overall quality of report generation.

To address the challenges, we propose an Anomaly-Driven Cross-Modal Contrastive Network (ADCNet), designed to improve the quality and accuracy of medical report generation. This method incorporates abnormal lesion information and uses a cross-modal feature fusion and alignment mechanism to reduce semantic discrepancies between medical images and reports, thereby enhancing the semantic consistency between the two modalities. Additionally, contrastive learning is employed to refine the model’s image understanding. Overall, the proposed method increases the sensitivity of visual features to abnormal regions and applies fine-grained alignment across modal features to improve feature representation. First, we introduce the anomaly-aware cross-modal feature fusion (ACFF) module, which introduces a new anomaly embedding vector. This vector dynamically represents lesion information and is updated throughout training. Combined with visual features, it enhances the model’s ability to capture lesion abnormalities, thus improving the feature fusion process. Second, we propose the fine-grained regional feature alignment (FRFA) module. This module employs an adaptive filtering mechanism to dynamically select relevant visual and textual features, effectively eliminating irrelevant information and background noise. After filtering, the features undergo cross-modal similarity calculations to derive weighted values at both the image region and text token levels. Furthermore, through residual fusion and contrastive learning, the fine-grained feature alignment is strengthened. Experimental results demonstrate that ADCNet outperforms both baseline models and existing state-of-the-art (SOTA) methods on the IU X-Ray and MIMIC-CXR standard datasets, showcasing significant improvements in performance. In summary, the contributions of this paper are as follows:

• We propose an anomaly embedding vector that acts as a bridge between the two modalities, effectively capturing and enhancing the complex semantic interactions between visual and textual features and thereby improving the representation of lesion areas in the visual features.
• We design a feature-adaptive filtering method that aligns fine-grained local visual regions with more precise textual descriptions, enhancing the semantic consistency between the cross-modal features.
• Through extensive experiments, we show that the proposed method outperforms existing approaches in terms of accuracy, completeness, and readability of medical reports.

2. Related Works

2.1. Medical Report Generation

As a visual-language task, medical report generation (MRG) widely adopts the encoder–decoder framework. To address the challenge of extracting abnormal features from lesions in medical images, researchers have enhanced feature capture capabilities from multiple perspectives, such as intra-modal fine-grained feature extraction and cross-modal fusion. The introduction of external knowledge and intermediate variables, such as knowledge graphs [13,14] and structured high-level semantics [15], can assist in aligning and correcting visual features. However, this approach often requires additional prior knowledge and is heavily dependent on external data and disease labels, which are typically scarce in medical report data. To address the semantic gap between medical images and reports, existing work has focused on optimizing cross-modal interactions. Some researchers have designed memory modules to integrate visual and textual information, such as the memory-driven Transformer model [16] and cross-modal memory networks [17], which record the alignment between visual regions and text tokens to facilitate cross-modal fusion. Additionally, methods like CGFTrans [18] improve feature alignment by fusing local and global visual features with medical report embeddings, while HDGAN [19] introduces a multi-level text generator to capture word- and sentence-level features, enhancing the accuracy and fluency of generated reports. While these approaches effectively fuse features from both modalities, they still tend to extract single-modal features independently before fusion, leading to insufficient alignment depth and failing to fully address the fine-grained feature alignment between modalities.

2.2. Contrastive Learning

Contrastive learning enables the model to distinguish between similar and different visual features by extracting visual and semantic information from matching medical image–text pairs. Abnormal areas are highlighted using attention mechanisms, contrastive loss optimization, and other techniques. Methods [10,20] enhance the representation of visual features by applying various contrastive strategies. However, due to the inherent complexity and ambiguity of medical images, fully capturing the semantic content of the image is challenging. Additionally, medical images often come from different perspectives and are multi-labeled, meaning the same instance may contain multiple lesions. This makes measuring similarity difficult and complicates the accurate detection of complex lesion abnormalities, as simply pulling similar features together and pushing different ones apart is insufficient.

Researchers have taken several approaches to optimize this problem. CLIP [21] has shown strong performance in aligning visual and textual representations by leveraging a large-scale contrastive learning approach on diverse datasets. But in MRG task, its general-purpose nature and lack of domain-specific knowledge often limit its ability to accurately capture the intricate details and abnormalities in medical images. XProNet [22] introduces a cross-modal prototype as an intermediate representation between visual and textual features, using a memory matrix to store prototypes. A multi-label contrastive loss is applied, where pairs sharing at least one label are positive, and those with no common label are negative. However, this approach is vulnerable to data imbalance, as it is influenced by label distribution. Additionally, the coarse-grained classification of positive and negative pairs often misclassifies samples with multiple distinct lesions, introducing noise. CECL [23] uses an adaptive clustering network to group image and text features and applies contrastive loss. However, when feature representations are insufficient, the generated cluster labels may be inaccurate, negatively affecting performance. Moreover, an over-reliance on category information can neglect fine-grained features. DCL [13] proposes a dynamic graph construction method that incorporates external knowledge to optimize visual features during training. It applies image–text contrastive loss, bidirectional contrast, and momentum queue optimization to improve cross-modal alignment. By introducing auxiliary information, this method enhances the extraction of fine-grained, discriminative image features, improving image–text contrast. Based on the aforementioned research, this paper leverages the semantic information within the data and uses a high-dimensional anomaly embedding vector to guide cross-modal feature fusion, enhancing the model’s attention to abnormal features. Additionally, before decoding, feature filtering and fine-grained feature alignment are performed, followed by cross-modal interaction to further refine feature representation. These steps collectively improve the alignment between visual and textual modalities, leading to more accurate and consistent medical report generation.

3. Methods

In the MRG task, let $I\in {\mathbb{R}}^{3\times 224\times 224}$ represents the medical image, where I has a resolution of $224\times 224$ pixels and 3 color channels. The corresponding medical report is represented as $R=\{r_1,r_2,\dots ,r_{N_t}\}\in {\mathbb{R}}^{N_t}$, where each $r_i\in \mathbb{V}$ is the word index from the vocabulary $\mathbb{V}$, with a vocabulary size of $N_v$. The goal of the MRG task is to generate a report $\widehat{R}=\{{\widehat{r}}_1,{\widehat{r}}_2,\dots ,{\widehat{r}}_{N_g}\}\in {\mathbb{R}}^{N_g}$ based on the provided image, such that it is similar to the ground-truth report R. Here, $N_t$ and $N_g$ denote the lengths of the real report and the generated report, respectively.

CheXpert [24] can generate diagnostic labels for chest X-ray radiology reports. We use this tool to annotate the diseases in the training set, obtaining 14 disease labels, represented as $l=\{c_1,c_2,\dots ,c_{N_c}\mid c_i=0,1\}\in {\mathbb{R}}^{N_c}$, where $N_c$ represents the number of disease labels.

The overall structure of the model is based on a Transformer-based encoder–decoder architecture, as shown in Figure 2. It mainly consists of four components: (a) feature extraction; (b) anomaly-aware cross-modal feature fusion; (c) fine-grained regional feature alignment; (d) text generation.

3.1. Feature Extraction

3.1.1. Visual Feature Extraction

The image I is passed through a pre-trained ResNet-101 model to extract patch-level features and obtain local visual feature $V_p\in {\mathbb{R}}^{N_P\times d_v}$, with $N_P$ representing the number of patches and $d_v$ representing the visual feature dimension.

3.1.2. Text Feature Extraction

For text sequences, the word embedding module in the Transformer maps discrete word indices into a $d_m$-dimensional continuous vector space, producing an embedding vector T. To stabilize the training process, T is scaled and normalized as follows:

$$T={\displaystyle \frac{W_{emb}R}{\sqrt{d_m}}},$$

(1)

where $T\in {\mathbb{R}}^{N_t\times d_m}$, and $W_{emb}\in {\mathbb{R}}^{N_v\times d_m}$ is the word embedding matrix, with $N_v$ as the vocabulary size and $d_m$ as the embedding dimension.

Next, positional encoding is added to incorporate word order into the embedding. The positional encoding is computed as follows:

$$p_{i,2k}=sin\left({\displaystyle \frac{i}{{\mathrm{10,000}}^{2k/d_m}}}\right),p_{i,2k+1}=cos\left({\displaystyle \frac{i}{{\mathrm{10,000}}^{2k/d_m}}}\right),$$

(2)

where $i\in \{1,2,\dots ,N_t\}$ is the position index of a word in the sequence, and $k\in \{1,2,\dots ,d_m/2\}$ corresponds to the embedding dimension indices. The positional encoding matrix P is added to the word embedding.

The final text feature $E\in {\mathbb{R}}^{N_t\times d_m}$ is computed as the element-wise sum of the normalized word embedding T and the positional encoding P:

$$E=T+P.$$

(3)

This entire process can be summarized as follows:

$$E=\mathrm{Embedding}\left(R\right).$$

(4)

3.1.3. Feature Enhancement

The standard Transformer encoder is used to encode the features of the visual modality to obtain enhanced local visual feature $E_v$. Similarly, a standard self-attention module is applied to encode the text embeddings, producing the text embedding features $E_t$:

$$E_v=\mathrm{Encoder}\left(V_p\right),E_t=W_Q\left(T^{\prime }\right)·\mathrm{Softmax}(W_K{\left(T^{\prime }\right)}^T·W_V\left(T^{\prime }\right))+T^{\prime }.$$

(5)

where $E_v\in {\mathbb{R}}^{N_p\times d_m}$, $E_t\in {\mathbb{R}}^{N_t\times d_m}$, and Encoder denotes the standard Transformer encoder.

Next, average pooling is performed to obtain the global visual feature $E_{v_g}\in {\mathbb{R}}^{d_m}$.

Finally, the global features are passed through a classification layer to obtain the disease label prediction probability $\widehat{L}$:

$$\widehat{L}=\mathsf{\sigma }(W_l·E_{{\mathrm{v}}_g}),$$

(6)

where $\widehat{L}\in {\mathbb{R}}^{N_c}$, $W_l\in {\mathbb{R}}^{d_m\times N_c}$, and $\mathsf{\sigma }$ represents the sigmoid activation function.

3.2. Anomaly-Aware Cross-Modal Feature Fusion

The anomaly-aware cross-modal feature fusion (ACFF) module is designed to address the challenge of capturing and aligning clinically significant abnormalities across different modalities. By utilizing predicted disease labels to enhance the fusion process, the ACFF module ensures that visual features are dynamically guided by anomaly-specific semantic information. This innovative approach enhances the model’s ability to focus on critical diagnostic regions, leading to more accurate and clinically relevant cross-modal feature alignment. The design details of the module are illustrated in Figure 3.

3.2.1. Anomaly Information Generation

The generation of anomaly-specific information is a key component of the ACFF module. Predicted disease label probabilities $\widehat{L}$ are mapped into a specialized embedding space to produce an anomaly embedding vector $L_e$. This vector encodes the semantic meaning and abnormal characteristics associated with the predicted diseases, providing a rich representation of clinically relevant information.

We begin by transforming the predicted disease label $\widehat{L}\in {\mathbb{R}}^{N_c}$ into a corresponding anomaly embedding vector $L_e$ within the text feature space. Before mapping, $\widehat{L}$ is converted into a discrete multi-hot vector by applying a thresholding operation, which can be formulated as follows:

$${\tilde{L}}_i=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\widehat{L}}_i\ge \tau ,\hfill \\ 0,\hfill & \mathrm{if}{\widehat{L}}_i<\tau ,\hfill \end{array}\right.$$

(7)

where $i\in \{1,2,\dots ,N_c\}$, and $\tau$ is a predefined threshold parameter and its value is given in Section 4.3. Through this operation, the resulting multi-hot vector $\tilde{L}\in \{0,1\}$ indicates the potential disease categories associated with the input image.

However, this vector represents a discrete classification result, and directly computing it with continuous feature vectors may lead to information loss. To address this, the classification results are mapped into a continuous feature space using a trainable embedding process, producing the anomaly embedding vector $L_e$. This embedding introduces both semantic information from the text space and anomaly-specific guidance, improving alignment with clinically relevant features.

Next, we integrate the visual features $E_v$ with the generated anomaly embeddings. To achieve this, we project the visual features $E_v$ into the anomaly embedding space by applying a trainable transformation matrix $W_l$. This step produces a new visual-label vector $V_l$, which represents a cross-modal fusion of visual features and anomaly-aware information.

$$\begin{array}{c}{L}_e=\mathrm{Embedding}\left(\tilde{L}\right),\\ V_l=W_l·E_v.\end{array}$$

(8)

Here, $L_e\in {\mathbb{R}}^{N_c\times d_m}$, and the embedding function $\mathrm{Embedding}(·)$ is the same as the embedding module, as shown in Equation (6), which generates a $d_m$-dimensional vector for each disease label $\tilde{L}$. $L_e$ serves as a guiding signal for the model, directing its attention towards regions of interest in the medical image that correspond to the abnormalities. Additionally, $V_l\in {\mathbb{R}}^{N_c\times d_m}$, and $W_l\in {\mathbb{R}}^{N_c\times N_p}$ is the transformation matrix that projects the visual features $E_v$ into the anomaly embedding space.

By mapping discrete disease labels to $L_e$, we incorporate semantic guidance that directs the model’s focus to clinically relevant regions associated with the diagnosed diseases. This process generates a representation of the image enriched with both visual content and anomaly-specific information, forming a solid foundation for feature fusion and alignment.

3.2.2. Anomaly Information Injection

To fuse visual and anomaly-aware features, we employ a cross-attention mechanism that aligns visual features with anomaly-specific information, focusing on diagnostically critical regions.

The first cross-attention operation aligns the visual-label vector $V_l$ with the anomaly embedding vector $L_e$, generating the visual-anomaly vector $L_{va}$. This step ensures that the anomaly embedding vector provides semantic and anomaly-specific guidance, enabling the model to focus on regions in the visual modality that correspond to specific disease categories. Next, the second cross-attention layer integrates this anomaly-aware information into the visual feature map $E_v$, producing the fused anomaly-aware visual feature $E_{vf}$. This step enhances the model’s focus on critical diagnostic areas and improves cross-modal feature alignment.

$$\begin{array}{c}{L}_{va}=\mathrm{Softmax}(W_Q\left(V_l\right)·W_K{\left(L_e\right)}^T)·W_V\left(L_e\right)+V_l,\\ E_{vl}^{\prime }=\mathrm{Softmax}(W_Q\left(E_v\right)·W_K{\left(L_{va}\right)}^T)·W_V\left(L_{va}\right)+E_v,\\ E_{vl}=E_v+E_{vl}^{\prime },\end{array}$$

(9)

where $L_{va}\in {\mathbb{R}}^{N_c\times d_m}$, and $W_Q$, $W_K$, and $W_V$ are the query, key, and value matrices for the attention mechanisms $E_{vl}^{\prime }\in {\mathbb{R}}^{N_p\times d_m}$ and $E_{vl}\in {\mathbb{R}}^{N_p\times d_m}$.

The first attention interaction aligns the visual features with disease-related anomaly embeddings, while the second refines the original visual features with this information. Together, these steps enhance the model’s ability to mine diagnostically significant information and improve the accuracy and clinical relevance of the generated reports.

3.2.3. Visual-Anomaly Vector Loss Calculation

To optimize this fusion process, we calculate a loss function based on the cross-entropy between the abnormal embedding vector $L_e$ and the mapped visual features $V_l$:

$$L_a=-\sum _{i=1}^{N_c}\left[V_{li}log\left(\sigma \left(L_{ei}\right)\right)+(1-V_{li})log(1-\sigma \left(L_{ei}\right))\right],$$

(10)

where $\sigma$ represents the sigmoid activation function, and the loss function encourages the model to focus on the correct abnormal regions.

By integrating visual and textual features with a focus on anomalies, this method not only enhances the accuracy of the reports but also ensures that critical diagnostic information, often embedded within abnormalities, is effectively captured and highlighted.

3.3. Fine-Grained Regional Feature Alignment

For paired medical image and report data, visual and textual features typically exhibit semantic consistency, but challenges arise due to data imbalance and the inherent complexity of medical images and reports. Images often suffer from high intra-class variability and low inter-class discriminability, while reports contain complex language structures. These issues hinder effective feature correlation. To address these challenges, we propose fine-grained regional feature alignment, as shown in Figure 4, a novel method that uses dynamic feature weighting and contrastive loss to enhance the alignment of image and text features. This approach selectively emphasizes important visual regions and text tokens, ensuring more precise and discriminative cross-modal feature alignment.

3.3.1. Adaptive Feature Filtering

To emphasize key image regions and text tokens, we propose a feature filtering module that adaptively generates importance weights for visual and textual features using a multilayer perceptron (MLP):

$$w_{vs}=\mathrm{MLP}\left(E_{vl}\right),w_{ts}=\mathrm{MLP}\left(E_t\right),\mathrm{MLP}\left(x\right)=\mathrm{Softmax}(W_2·\mathrm{ReLU}(W_1·x+b_1)+b_2).$$

(11)

Here, $E_{vl}\in {\mathbb{R}}^{N_p\times d_m}$ and $E_t\in {\mathbb{R}}^{N_t\times d_m}$ represent the input visual and textual features. The MLP is a single-layer feedforward network applied independently to each feature vector (i.e., each row of $E_{vl}$ or $E_t$). The first layer transforms the input feature vector of size $d_m$ to a hidden representation of size $h_d$. This transformation is performed using $W_1\in {\mathbb{R}}^{h_d\times d_m}$ and bias $b_1\in {\mathbb{R}}^{h_d}$, followed by a ReLU activation. The second layer projects the hidden representation to a scalar with $W_2\in {\mathbb{R}}^{1\times h_d}$ and bias $b_2\in \mathbb{R}$. Finally, a Softmax activation is applied to the output, ensuring that the resulting weights $w_{vs}\in {\mathbb{R}}^{N_p}$ and $w_{ts}\in {\mathbb{R}}^{N_t}$ form a probability distribution (i.e., the weights sum to 1).

These weights dynamically adjust to the input context, highlighting relevant features while suppressing irrelevant ones.

In comparison, self-attention in Transformers considers global relationships across the entire sequence. It learns attention scores that determine how each token or region relates to all others. In contrast, our adaptive weighting is more focused. It generates weights specifically tailored to the importance of features within the context of their respective modality. This design allows for a more refined emphasis on key features, which is particularly beneficial when dealing with complex and imbalanced medical data, where the relevant information may be highly localized in both visual and textual domains.

Next, both visual and textual features are weighted according to the generated weights to obtain the filtered features $E_{vs}$ and $E_{ts}$:

$$E_{vs}=w_{vs}\odot E_{vl},E_{ts}=w_{ts}\odot E_t,$$

(12)

where ⊙ denotes element-wise multiplication, and $E_{vs}\in {\mathbb{R}}^{N_p\times d_m}$, $E_{ts}\in {\mathbb{R}}^{N_t\times d_m}$.

3.3.2. Fine-Grained Feature Alignment

Based on the filtered features, the cross-modal similarity between image regions and text tokens is calculated through matrix multiplication, resulting in the similarity matrix S:

$$S=E_{vs}·E_{ts}^T,$$

(13)

where $S\in {\mathbb{R}}^{N_p\times N_t}$.

Next, fine-grained feature correlation is computed based on S. It is normalized and averaged across both the text and visual dimensions to obtain the cross-modal global attention weights for each image region and each token, denoted as $S_v$ and $S_t$, respectively:

$$\begin{array}{c}{A}_v=\mathrm{Softmax}(S,\dim =-1),A_t=\mathrm{Softmax}(S,\dim =0),\\ S_v={\displaystyle \frac{1}{N_t}}\sum _{i=1}^{N_t}{A}_{v_{:,i}},S_t={\displaystyle \frac{1}{N_p}}\sum _{j=1}^{N_p}{A}_{t_{j,:}},\end{array}$$

(14)

where $A_v\in {\mathbb{R}}^{N_p\times N_t}$ represents the attention weight of the image region with respect to the text token, and $A_t\in {\mathbb{R}}^{N_p\times N_t}$ represents the attention weight of the text token with respect to the image region. $S_v\in {\mathbb{R}}^{N_p\times 1}$ and $S_t\in {\mathbb{R}}^{N_t\times 1}$. The image and text features are then weighted element-wise to obtain $E_{va}$ and $E_{ta}$:

$$E_{va}=S_v·E_{vs},E_{ta}=S_t·E_{ts},$$

(15)

where $E_{va}\in {\mathbb{R}}^{N_p\times d_m}$, $E_{ta}\in {\mathbb{R}}^{N_t\times d_m}$.

Finally, soft attention weights are used to perform dynamic residual fusion of the attended feature and aligned feature to obtain the regional aligned features $E_{vf}$ and $E_{tf}$:

$$E_{vf}=\alpha ·E_{va}+(1-\alpha )·E_{vf},E_{tf}=\beta ·E_{ta}+(1-\beta )·E_t,$$

(16)

where $\alpha$ and $\beta$ are hyperparameters that controls the fusion of the attended feature and the original feature.

This method reduces interference from redundant image regions and irrelevant text tokens by combining local region filtering, feature alignment, and a dynamic residual mechanism. Meanwhile, it enhances the alignment of image and text features through contrastive loss, improving the semantic consistency between cross-modal features.

3.3.3. Fine-Grained Cross-Modal Contrastive Loss Calculation

After obtaining the fine-grained features $E_{vf}$ and $E_{tf}$, the final step is to compute the contrastive loss. The features are first L2-normalized:

$${\widehat{E}}_{vf}={\displaystyle \frac{E_{vf}}{\parallel E_{vf}{\parallel }_2}},{\widehat{E}}_{tf}={\displaystyle \frac{E_{tf}}{\parallel E_{tf}{\parallel }_2}}.$$

(17)

Then, the similarity for both image-to-text and text-to-image is computed using the temperature coefficient T:

$$S_{it}\left(i\right)={\displaystyle \frac{exp\left({\widehat{E}}_{v{f}_i}·{\widehat{E}}_{tf}^T/T\right)}{{\sum }_{k=1}^{N_t}exp\left({\widehat{E}}_{v{f}_i}·{\widehat{E}}_{t{f}_k}^T/T\right)}},S_{ti}\left(i\right)={\displaystyle \frac{exp\left({\widehat{E}}_{t{f}_i}·{\widehat{E}}_{vf}^T/T\right)}{{\sum }_{k=1}^{N_p}exp\left({\widehat{E}}_{t{f}_i}·{\widehat{E}}_{v{f}_k}^T/T\right)}},$$

(18)

here, $S_{it}\in {\mathbb{R}}^{N_p\times N_t}$, $S_{ti}\in {\mathbb{R}}^{N_p\times N_t}$ and the value of T is given in Section 4.3.

Finally, the contrastive loss is computed by summing both the image-to-text and text-to-image losses:

$$L_{\mathrm{c}}=-\sum _{i=1}^{N_p}log\left(S_{it}\left(i\right)\right)-\sum _{i=1}^{N_t}log\left(S_{ti}\left(i\right)\right).$$

(19)

3.4. Text Generation

This paper uses the standard Transformer decoder as a text generator. In the training phase, medical images, real reports, and real labels are input, while in the validation and testing phases, only the model generates the predicted report text in an autoregressive manner based on the hidden vector. The decoding process is optimized using cross-entropy loss.

The decoder generates each token ${\widehat{r}}_t$ conditioned on the previously generated tokens ${\widehat{r}}_1,\dots ,{\widehat{r}}_{t-1}$, the fused visual features $E_{vf}$, and text features $E_t$. The fusion of these features allows the decoder to incorporate both image and text information when generating the report. The model is trained to maximize the likelihood of generating the target sequence $\widehat{R}$ given the input image I, formulated as follows:

$$p\left(\widehat{R}\right|I)=\prod _{t=1}^{N_g}p\left({\widehat{r}}_t\right|{\widehat{r}}_1,\dots ,{\widehat{r}}_{t-1},E_{vf},E_t),$$

(20)

where $\widehat{R}=\{{\widehat{r}}_1,{\widehat{r}}_2,\dots ,{\widehat{r}}_{N_g}\}$ is the generated report sequence.

At each time step, the decoder generates the next token ${\widehat{r}}_t$ based on the context provided by the previously generated tokens and the cross-modal features from both the image and the text. This autoregressive process continues until the entire report is generated.

The decoding process is optimized using cross-entropy loss:

$$L_d=-{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\displaystyle \frac{1}{N_g}}\sum _{j=1}^{N_g}logp\left(r_{ij}\right|{\widehat{r}}_{ij}).$$

(21)

3.5. Loss Function

In order to effectively integrate the three loss functions and calculate the complete overall loss function, an automatically adjustable weight factor is introduced to help balance the contribution of various loss components to model training. The total loss is calculated as follows:

$$L_t={\lambda }_a{L}_a+{\lambda }_c{L}_c+{\lambda }_d{L}_d,$$

(22)

where the values of the weight parameters ${\lambda }_a$, ${\lambda }_c$, and ${\lambda }_d$ are given in Section 4.3.

4. Experiment Setup

4.1. Dataset

The experiments in this paper are conducted on two datasets: IU X-Ray [11] and MIMIC-CXR [12]. Fourteen categories of common chest diseases are annotated using the CheXpert [24] dataset. All reports are standardized to lowercase, and rare words with a frequency of less than 3 are filtered out, resulting in 760 and 7863 unique lexicons for the IU X-Ray and MIMIC-CXR datasets, respectively. Data segmentation is performed following the R2Gen benchmark [16]. The IU X-Ray dataset was split in a 7:1:2 ratio, while MIMIC-CXR adhered to the official division standard.

4.2. Evaluation Indicators

We use a variety of natural language generation (NLG) metrics (BLEU-n [25], METEOR [26], and ROUGE-L [27]) to evaluate the performance of our model. These metrics measure the similarity between generated reports and reference reports in terms of n-gram overlap and semantic similarity. For these evaluation metrics, the higher the score, the better the model performance.

In addition, we evaluate clinical efficacy (CE) metrics, including precision, recall, and F1-score. CE metrics are introduced by Chen et al. [28] as a way to assess the clinical relevance of generated radiology reports. As shown in this paper, we employ CheXpert [24] to extract disease labels from both the reference reports and the generated reports. CE metrics calculate how well the generated reports capture clinically important findings by comparing the predicted labels with ground-truth labels across 14 thoracic disease categories. The CE scores are computed as the average precision (P), recall (R), and F1-score ($F_1$) across all disease categories. Higher CE scores indicate a better alignment of the generated reports with clinically relevant information.

4.3. Experimental Parameter Settings

We resize the images to 224 × 224 pixels, apply center cropping, and scale the pixel values to [−1, 1]. ResNet-101 pre-trained on ImageNet [29] is used as the visual feature encoder, which divides each radiology image into 2048-dimensional patch features. A baseline model is constructed using a standard Transformer decoder as the text generator. The number of patches is 98 for the IU X-Ray dataset and 49 for the MIMIC-CXR dataset, respectively. The Transformer architecture uses eight attention heads in the encoder and decoder with a hidden dimension of 512. The model is trained over 50 epochs with a batch size of 16 using the Adam optimizer and cross-entropy loss, implemented in Python 3.7 using PyTorch 1.7.1. The learning rate for the visual extractor is $5\times {10}^{-5}$, and the learning rate for other parameters is $1\times {10}^{-4}$, with a decay of 0.8 per round. The dropout rate is uniformly set to 0.1 in this study. The initial values for the weights of the loss function $L_a$, $L_c$, and $L_d$ are set to 0.4, 0.4, and 0.2, respectively, and are dynamically updated during training. The temperature coefficient T for $L_c$ is set to 0.07. The threshold value $\tau$ for the labels is set to 0.8. We conduct a parameter sensitivity experiment to determine the optimal value of T, which is detailed in Section 4. The beam search size used during report generation is set to 3. The distribution of loss values and evaluation indicators are shown in Figure 5 and Figure 6, respectively, showing a continuous decline and steady progress.

Our baseline model is built on top of a standard Transformer architecture and comprises two main components: (1) a visual feature extractor that uses a pre-trained ResNet-101 network to extract patch-level image features and (2) a standard Transformer-based encoder–decoder module for cross-modal feature encoding and report generation. This design was inspired by the work of [16], where a memory-driven Transformer was proposed for radiology report generation. However, for the baseline model, we removed the innovative memory-driven module and other specific enhancements proposed in their method, retaining only the core Transformer-based structure for feature encoding and decoding.

5. Experiment Results

5.1. Comparative Experimental Analysis

Table 1. Comparison of NLG experimental results of different methods on two datasets.

Dataset	Model	B-1	B-2	B-3	B-4	METEOR	ROUGE-L
IU X-Ray	R2Gen (2020) [16]	0.470	0.304	0.219	0.165	0.187	0.371
	CA (2021) [6]	0.492	0.314	0.222	0.169	0.193	0.381
	R2GenCMN (2021) [17]	0.475	0.309	0.222	0.170	0.191	0.375
	DCL (2023) [13]	—	—	—	0.163	0.193	0.383
	MvCo-DoT (2023) [30]	0.453	0.318	0.223	0.157	0.196	0.374
	CGFTrans (2024) [18]	0.503	0.318	0.224	0.165	0.206	0.381
	HDGAN (2024) [19]	0.477	0.318	0.226	0.167	0.190	0.417
	CL-FD (2024) [31]	0.479	0.319	0.235	0.179	—	0.380
	S3-Net (2024) [32]	0.499	0.334	0.246	0.172	0.206	0.401
	Ours (ADCNet)	0.493	0.317	0.249	0.179	0.194	0.388
MIMIC-CXR	R2Gen (2020) [16]	0.353	0.218	0.145	0.103	0.142	0.227
	CA (2021) [6]	0.350	0.219	0.152	0.109	0.151	0.283
	R2GenCMN (2021) [17]	0.371	0.233	0.153	0.106	0.143	0.278
	DCL (2023) [13]	—	—	—	0.109	0.150	0.284
	CGFTrans (2024) [18]	0.382	0.235	0.155	0.110	0.154	0.285
	HDGAN (2024) [19]	0.352	0.195	0.122	0.084	0.115	0.290
	AERMNet (2024) [33]	0.273	0.169	0.120	0.090	0.157	0.232
	MCSAM (2024) [34]	0.379	0.230	0.153	0.109	0.149	0.284
	AdaMatch-Cyclic (2024) [35]	0.379	0.234	0.154	0.106	0.163	0.286
	Ours (ADCNet)	0.406	0.246	0.161	0.111	0.156	0.263

The bold values in the table represent the best experimental results for each metric.

presents a comparative analysis of the performance of ADCNet compared to existing SOTA models on two datasets. The results show that ADCNet outperforms other methods across many evaluation metrics, demonstrating its effectiveness in generating high-quality medical reports. On the IU X-Ray dataset, ADCNet performs better in BLEU-3 and BLEU-4. For the MIMIC-CXR dataset, ADCNet achieves a notable improvement, with an average increase of 3.94% in BLEU scores (from BLEU-1 to BLEU-4). However, the performance of the METEOR and ROUGE-L metrics is lower for both datasets, suggesting that the reports generated by ADCNet exhibit lower surface-level overlap with reference reports. This observation can be explained by the highly subjective nature of medical reports. Unlike more formal or structured domains, medical reports vary significantly in language style, terminology, and sentence structure due to the preferences of the individual physician and the clinical context and practice. Such variability presents a challenge for traditional NLG models, which often aim for high shallow overlap similarity to reference texts. In contrast, the ADCNet approach generates more structured and standardized output, which may not always align closely with the phrasing and sequence patterns seen in the reference reports. This leads to a decrease in the surface-level overlap as measured by ROUGE-L and METEOR. Additionally, while traditional NLG models may focus on achieving high lexical overlap with reference texts, a more robust approach for medical report generation should also consider semantic accuracy and clinical relevance.

Table 2. Comparison of CE experimental results of different methods on MIMIC-CXR datasets.

Model	Precision	Recall	F1
R2Gen (2020) [16]	0.333	0.273	0.276
R2GenCMN (2021) [17]	0.334	0.275	0.278
CA (2021) [6]	0.352	0.298	0.303
R2GENRL (2022) [36]	0.342	0.294	0.292
RAMIT (2023) [37]	0.380	0.342	0.335
METransformers (2023) [38]	0.364	0.309	0.331
Ours (ADCNet)	0.347	0.369	0.337

The bold values in the table represent the best experimental results for each metric.

demonstrates that our method shows improvements in CE metrics for medical report generation. ADCNet achieves a recall of 0.369, outperforming other models. It also performs well in terms of $F_1$, with a score of 0.337, indicating a good balance between precision and recall. As mentioned above, in the context of medical image report generation, recall is particularly important, as the missed detection of lesions can result in significant diagnostic errors. A high recall rate helps to mitigate such risks, confirming the reliability of our method in detecting multiple lesions.

5.2. Ablation Experiment Analysis

As illustrated in

Table 3. The ablation experiments of the method in this paper are performed on two datasets.

Dataset	Base	ACFF	FRFA	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L
IU X-Ray	✓			0.441	0.265	0.189	0.146	0.170	0.341
	✓	✓		0.488	0.317	0.220	0.170	0.193	0.363
	✓		✓	0.494	0.310	0.222	0.172	0.189	0.389
	✓	✓	✓	0.493	0.317	0.249	0.179	0.194	0.388
MIMIC-CXR	✓			0.299	0.187	0.127	0.092	0.126	0.271
	✓	✓		0.357	0.232	0.156	0.106	0.144	0.261
	✓		✓	0.362	0.238	0.157	0.108	0.146	0.261
	✓	✓	✓	0.406	0.246	0.161	0.111	0.156	0.263

The bold values in the table represent the best experimental results for each metric.

, in the ablation study, we systematically investigated the contributions of the ACFF and FRFA modules to the performance of MRG. In both the IU X-Ray and MIMIC-CXR datasets, the baseline model, devoid of these specialized modules, exhibited relatively lower performance across a range of evaluation metrics. The introduction of the ACFF module marked a significant improvement, particularly in terms of n-gram matching (BLEU scores) and fluency (METEOR score), indicating its effectiveness in enhancing the cross-modal feature fusion process and facilitating the generation of more coherent and accurate medical reports.

Furthermore, the subsequent addition of the FRFA module built upon the improvements achieved with the ACFF module, leading to further enhancements in performance across most metrics. Notably, on the MIMIC-CXR dataset, the FRFA module contributed to reversing the slight decline in ROUGE-L score observed with the ACFF module alone, suggesting its role in improving recall-oriented aspects of the generated reports. The FRFA module’s fine-grained alignment of regional features likely helps in capturing more nuanced differences between images and their corresponding reports, thereby enhancing the semantic consistency and completeness of the generated medical reports. Overall, the combined effect of the ACFF and FRFA modules results in a robust and effective framework for medical report generation that outperforms the baseline model and demonstrates the importance of specialized modules in addressing the unique challenges posed by this task.

5.3. Parameter Analysis

As shown in Figure 7, we analyze the impact of the temperature coefficient T, which influences the sharpness of the similarity distribution during the cross-modal contrastive loss calculation in the FRFA module. The temperature coefficient T plays a critical role in balancing the image–text alignment during training. By varying T across different values, we examine its effect on model performance.

As observed, the model’s performance improves across several evaluation metrics as T increases from 0.01 to 0.07. This trend aligns with the settings used in pre-trained CLIP models, where $T=0.07$ has been shown to provide a good balance between precision and recall. However, when T is further increased beyond 0.07, the performance begins to degrade. This decline can be attributed to the model’s tendency to over-relax the similarity criteria, resulting in less accurate alignment of fine-grained image–text features. Specifically, overly large values of T cause the model to match a broader set of image–text pairs, including irrelevant ones, which reduces the overall precision. Based on these results, we conclude that $T=0.07$ is the optimal value for maintaining effective cross-modal feature alignment while ensuring robust performance in the medical report generation task.

5.4. Qualitative Experiments

To qualitatively assess the effectiveness of our ACFF module, we visualized a representative example from the test set of the MIMIC-CXR dataset, as seen in Figure 8. This figure illustrates the heatmap showing the alignment between the visual features encoded by the ACFF module and the words in the generated report. The first column presents the data pair, consisting of the image and the ground-truth report. The upper part of the second column contains the heatmap generated by the encoder of the BASE model, along with the report generated by the BASE model, while the lower part displays the heatmap of the feature map output by the ACFF module and the report generated by the ADCNet method. In the heatmap, the color range from blue to red represents the attention weights, with blue indicating low attention and red indicating high attention. This highlights the degree of correspondence between the visual regions in the image and the words in the report.

By comparing the two generated reports with the original report, we observe that the ADCNet method correctly identifies the presence of “atelectasis” in the image and accurately determines that “pneumothorax” is absent. In contrast, the BASE model misidentifies the lesion as “pleural effusion”. Moreover, the heatmap of the BASE model fails to align correctly with the key visual regions. For instance, the area corresponding to the word “lung” is noticeably misaligned, with most of the highlighted regions appearing at the lower part of the image, suggesting that the BASE model struggles to distinguish between relevant visual features. On the other hand, the heatmap generated by the ACFF module provides a more accurate alignment. For example, the locations corresponding to words like “right lung” and “pleural effusion” are correctly identified, demonstrating that the ACFF module improves visual-semantic alignment.

Figure 9 compares the real medical reports with those generated by the baseline model and our proposed method, ADCNet. As shown, the reports generated by ADCNet exhibit greater similarity to the real reports in terms of length, structure, and fluency. For example, in the first case, the sentences generated by the baseline model contain incomprehensible semantics, such as ’xxxx,’ which makes them difficult to understand. In contrast, ADCNet describes the image content in a more coherent and fluid manner. Moreover, ADCNet provides more comprehensive descriptions of lesions. For instance, in the second row, ADCNet accurately identifies “atelectasis”, which the baseline model fails to detect. Additionally, ADCNet demonstrates higher semantic consistency with the real reports. It not only detects the presence of diseases but also offers more precise and complete descriptions of normal conditions and anatomical structures. In the third row, the real report mentions “hemidiaphragm” and “cardiomegaly”. ADCNet successfully identifies both abnormalities and accurately describes their location and severity, correctly noting “right” and “mild”. In contrast, the baseline model fails to detect these conditions and incorrectly identifies “pleural effusion” on the right side. These results show that ADCNet significantly outperforms the baseline model in both accuracy and detail, improving the overall quality and reliability of the medical reports generated.

6. Conclusions

6.1. Summary of Contributions

In this paper, we propose the anomaly-driven cross-modulal contrast network (ADCNet) for the generation of medical reports. ADCNet improves the fusion of image and text modalities, improving the accuracy of feature representation and generating complete, accurate, and fluent reports. The Anomaly-Aware Cross-Modal Fusion (ACFF) module predicts disease labels from image features and maps them to the text embedding space, helping the model to focus on lesion abnormalities. The fine-grained regional feature alignment (FRFA) module filters image and text features to suppress irrelevant information and aligns them for contrastive loss optimization. The experimental results show the superiority of ADCNet in generating medically accurate and semantically consistent reports.

6.2. Future Work and Challenges

Future work will firstly focus on improving ADCNet’s performance in cases with limited input data, such as the MIMIC-CXR dataset, which consists of single-view X-ray images. Optimization is needed to handle such inputs more effectively. Additionally, while the model shows strong results, there is room for improvement in detecting subtle lesions, as indicated by the clinical efficacy (CE) metrics. Another challenge is the lower performance on ROUGE-L and METEOR metrics, which reflects the model’s focus on clinical relevance and semantic accuracy rather than surface-level overlap. Future research will aim to balance this trade-off, enhancing lexical overlap without compromising clinical precision. Finally, while the model is effective for CT and MRI data of lung diseases, it will be extended to handle a broader range of imaging modalities and disease types, improving its generalizability.