MDFormer: Transformer-Based Multimodal Fusion for Robust Chest Disease Diagnosis

Abstract

With the increasing richness of medical images and clinical data, abundant data support is provided for multimodal chest disease diagnosis methods. However, traditional multimodal fusion methods are often relatively simple, leading to insufficient exploitation of crossmodal complementary advantages. At the same time, existing multimodal chest disease diagnosis methods usually focus on two modalities, and their scalability is poor when extended to three or more modalities. Moreover, in practical clinical scenarios, missing modality problems often arise due to equipment limitations or incomplete data acquisition. To address these issues, this paper proposes a novel multimodal chest disease classification model, MDFormer. This model designs a crossmodal attention fusion mechanism, MFAttention, and combines it with the Transformer architecture to construct a multimodal fusion module, MFTrans, which effectively integrates medical imaging, clinical text, and vital signs data. When extended to multiple modalities, MFTrans significantly reduces model parameters. At the same time, this paper also proposes a two-stage masked enhancement classification and contrastive learning training framework, MECCL, which significantly improves the model’s robustness and transferability. Experimental results show that MDFormer achieves a classification precision of 0.8 on the MIMIC dataset, and when 50% of the modality data are missing, the AUC can reach 85% of that of the complete data, outperforming models that did not use two-stage training.

1. Introduction

Chest diseases, including pneumonia, lung cancer, and tuberculosis, are among the types of diseases with high incidence and mortality rates worldwide. According to the World Health Organization (WHO), six out of the top ten global causes of death in 2021 were chest diseases [1], posing significant challenges to global public health. Due to the subtle early symptoms of these chest diseases, many patients are diagnosed at advanced stages, missing the optimal treatment window. Therefore, early and accurate diagnosis is crucial for improving patient survival rates and quality of life. Traditional diagnostic methods for chest diseases primarily rely on medical imaging (such as X-rays, CT scans, etc.) or clinical data (such as medical history, symptoms, and signs). However, information from a single modality is often insufficient to provide a comprehensive diagnostic basis. Numerous medical studies have repeatedly shown that the lack of access to clinical and laboratory data during image interpretation can lead to a decline in diagnostic performance by doctors [2,3]. Furthermore, in a survey of radiologists, the majority (87%) of doctors stated that clinical information significantly impacts their interpretation [4]. As a result, the fusion and analysis of multimodal medical data have garnered extensive attention and research in the current medical field [5,6].

However, existing multimodal fusion methods still face many challenges in practical applications. First, data from different modalities are heterogeneous; for example, medical images are high-dimensional spatial data, clinical texts are discrete symbolic data, and vital sign time-series are continuous time data. Effectively aligning and merging these heterogeneous data is a problem that needs to be addressed. Traditional methods often use simple weighted averaging or concatenation techniques, which fail to capture the complex relationships between modalities and may lead to the loss of fine-grained semantic information. Second, some Transformer-based multimodal fusion methods have poor scalability, and when expanded to multiple modalities, the model complexity increases significantly. Additionally, in practical clinical settings, due to equipment limitations, incomplete data collection, and other reasons, complete modal data are often unavailable. This requires chest disease diagnosis models to be robust enough to handle incomplete data. However, existing research has paid little attention to the missing modality problem and lacks effective solutions.

To address the challenges of multimodal fusion difficulties, poor model scalability, and insufficient adaptability to the missing modality problem, this study proposes a Transformer-based multimodal fusion module, MFTrans, and builds a multimodal model, MDFormer, for chest disease classification on this basis. Additionally, we design an efficient two-stage training framework, MECCL. This approach not only improves classification accuracy but also effectively reduces the model’s parameter count, enhances scalability, and allows for flexible handling of various modality combinations. More importantly, the model demonstrates excellent robustness in the presence of missing modality. The main contributions of this paper are summarized as follows:

• We extend the self-attention mechanism to use different modalities as queries, simultaneously learning both intra-modality and inter-modality correlations. By combining group convolutions to extract local information, we design a lightweight fusion module MFTrans based on a multi-head attention mechanism. This approach effectively enhances fusion performance while maintaining low computational overhead.
• By integrating multimodal data such as time-series vital signs, chest X-ray images, and patient diagnostic reports, we propose a Transformer-based multimodal chest disease classification model, MDFormer. This model processes data from different modalities through modality-specific encoders, implements multimodal feature interaction and fusion in MFTrans, and ultimately achieves accurate chest disease diagnosis.
• This study combines modality masking and contrastive learning to propose a two-stage Mask-Enhanced Classification and Contrastive Learning training framework, MECCL. To further balance the optimization process of different task objectives in the two-stage training, we introduce a Sigmoid dynamic balance loss weight, enabling adaptive adjustment between loss terms. This framework allows the model to maintain effective diagnostic and classification performance even when some modality data are missing, thereby enhancing the model’s robustness and training stability.

The structure of the rest of this paper is arranged as follows: Section 2 provides a review of related work, including deep learning-based chest disease diagnosis, medical multimodal disease diagnosis, and Transformer-based multimodal data fusion methods. Section 3 details the construction process and preprocessing methods of the dataset used, the proposed improved Transformer-based multimodal fusion model, and the two-stage training framework employed. Section 4 presents the experimental setup, demonstrates comparative experimental results with current state-of-the-art methods, and provides an in-depth analysis and discussion of the results. Finally, Section 5 summarizes the research work presented in this paper and discusses future research directions.

2. Related Works

2.1. Deep Learning-Based Chest Disease Diagnosis

Accurate diagnosis and classification of chest diseases are crucial for early detection, personalized treatment, and improving prognosis. In recent years, deep learning technologies have made significant progress in medical image analysis. For example, Asuntha et al. improved the accuracy of lung cancer detection and classification by combining various image feature extraction methods with a fuzzy particle swarm optimization algorithm (FPSO) [7]. Schroeder et al. successfully predicted COPD by combining X-ray images and pulmonary function data using a CNN model, achieving an AUC of 0.814 [8]. Pham et al. improved the efficiency of pneumonia diagnosis by combining CNN and ViT models, outperforming traditional models [9]. Hussein et al. proposed a hybrid architecture that significantly improved the classification accuracy of lung diseases by combining SVM and CNN [10]. Mann et al. evaluated the performance of various deep learning models and found that DenseNet121 and ResNet50 performed exceptionally well in diagnosing diseases such as emphysema [11]. Chandrashekar et al. evaluated the application of transfer learning in COVID-19 diagnosis, with the MobileNet model performing best, achieving an accuracy of 98% [12]. Hayat et al. proposed a hybrid deep learning model that combines EfficientNetV2 and Vision Transformer (ViT) for breast cancer histopathological image classification. Experiments conducted on the BreakHis dataset showed that the model achieved an accuracy of 99.83% in binary classification and 98.10% in multi-class classification, demonstrating its superior performance in breast cancer detection [13].

In addition, lung sounds are also an important basis for diagnosing chest diseases. Tariq et al. achieved approximately 97% accuracy in lung sound classification by analyzing lung sound spectrogram features using a CNN model [14]. Pham et al. proposed a CNN-MoE-based framework that outperformed existing methods in lung disease detection [15]. Lal et al. proposed an algorithm combining VGGish with BiGRU, improving the accuracy of lung sound recognition [16].

Currently, in the diagnosis of chest diseases, despite the availability of various modal data (such as images, physiological signals, clinical texts, etc.), the utilization of these data is often limited to single-modal or simple combinatorial analyses, failing to fully explore the potential correlations between multimodal information. This single-modal or shallow multimodal analysis approach limits the comprehensiveness and accuracy of chest disease diagnosis.

2.2. Medical Multimodal Disease Diagnosis

Deep neural networks, with their powerful ability to learn hierarchical representations, are particularly well suited for multimodal learning problems. However, how to effectively combine the marginal and joint representations of heterogeneous modalities remains a core challenge in multimodal fusion [6]. Multimodal fusion can occur at different stages, primarily including data-level fusion, feature-level fusion, and decision-level fusion [6]. Data-level fusion involves concatenating the raw data from different modalities into a single vector, which is then treated as a single-modal input for processing by deep learning models. Feature-level fusion is performed after learning the marginal representations of each modality, followed by their integration. Decision-level fusion involves independently modeling multiple sub-models and combining them with ensemble strategies to arrive at a final decision.

Data-level fusion has the advantage of ease of implementation, as it does not require the construction of complex unimodal feature extraction frameworks [17]. The EHR-KnowGen model proposed by Niu et al., by integrating multimodal data from electronic health records, improves the quality of disease diagnosis [18]. Glicksberg et al. used the pre-trained language model BlueBERT to analyze emergency department patient data, achieving AUCs of 0.9014 and 0.6475, although this approach may lose some accuracy when converting numerical data into text [19]. When input features have structural information, such as genomic data or time-series data, RNNs [20] or CNNs can effectively process the concatenated inputs. For example, the multitask learning method proposed by Bichindaritz et al., combining Cox loss and ordinal loss tasks, uses genomic data to predict patient survival risk and improves model performance [20]. Despite the simplicity and frequent use of data-level fusion in multimodal learning, its limitations are also evident. Directly modeling joint representations makes it difficult to extract useful marginal representations of modalities, especially when the features between modalities only become apparent in high-level abstractions. Furthermore, data-level fusion is suitable for cases where the modality distributions are similar, but when there are significant modality differences, such as between images and molecular data, it may not perform well.

In feature-level fusion, different types of deep learning networks are used to learn the marginal representations of each modality, and these features are then integrated through a fusion layer to learn a joint representation. This method is suitable for tasks that require the preservation of modality-specific features [17]. Yan et al. integrated pathological images and electronic medical record data, achieving an accuracy of 92.9%. Additionally [21], Lee et al. combined fundus images and clinical risk factors to predict cardiovascular diseases, with an AUC of 0.781 [22]. The MMD framework proposed by Cui et al., by handling missing data, significantly improved the accuracy and robustness of brain cancer survival prediction [23]. Liu et al. proposed the self-attention-based SFusion module, which can automatically learn the correlations between multimodal data and generate shared representations, thereby enhancing model performance [24]. In 2024, Al et al. introduced an AI-based multimodal breast cancer diagnostic framework, achieving a classification accuracy of 97.73% [25].

In decision-level fusion, the key is to synthesize the decisions of unimodal models rather than directly merging the data or features [26]. The most straightforward aggregation method is to take the average of the results from different models, assuming that each sub-model contributes equally. The LVH-fusion model developed by Soto et al. surpassed traditional methods in predicting left ventricular hypertrophy [27]. El et al. evaluated deep convolutional neural network models, and decision-level fusion outperformed unimodal methods in classifying eye diseases [28]. Since different modalities contribute unequally to prediction performance, using a weighted aggregation method can better adjust the contribution of each sub-model. Liu et al. integrated genomic and imaging data to predict breast cancer molecular subtypes, using weighted fusion to improve prediction accuracy [29]. The 3D convolutional neural network framework proposed by Saikia et al., combining CT and PET images, significantly improved cancer prediction accuracy [30]. Meta-learning methods can learn the complex relationships between different sub-models and integrate prediction results through classifiers, improving the final accuracy. For example, Reda et al. combined clinical biomarkers and imaging data and used a meta-learning approach to enhance early prostate cancer diagnosis accuracy, achieving a diagnostic accuracy of 94.4% [31]. Decision-level fusion is suitable for situations where signals are not complementary or when data are missing, as it can still provide effective predictive power. In contrast, data-level and feature-level fusion may not be adaptable when data is missing.

2.3. Transformer-Based Multimodal Data Fusion Methods

In 2017, Vaswani et al. proposed the Transformer model, which is based on the multi-head attention mechanism and constructs a sequence-to-sequence encoder–decoder structure, providing a new solution for multimodal data fusion [32]. Compared to traditional convolutional neural networks and recurrent neural networks, the Transformer reduces inductive bias [33], allowing it to learn higher-quality feature representations from multimodal inputs. The consistency of the self-attention mechanism across different modalities enables the Transformer to flexibly represent multimodal clinical data. Common Transformer-based multimodal data fusion methods are shown in the Figure 1.

Simply concatenating the data or features of two modalities is referred to as SelfAttention. This method enables modality interaction by directly merging information from two modalities. Its advantages lie in efficiency and simplicity, avoiding complex modality interaction models, and it has been proven effective in multimodal pretraining and some downstream tasks [34,35,36,37].

CrossAttention is the most commonly used multimodal fusion method today, which calculates attention across the representations of different modalities. This method has been widely applied in multimodal medical data fusion [38]. However, CrossAttention faces the issue of parameter explosion when extended to multiple modalities [39], and it cannot fully capture the self-attention within each modality, potentially leading to information loss. In addition, combining CrossAttention with SelfAttention forms the Cross-Self-Attention mechanism [40]. This method not only preserves the internal information of each modality but also promotes effective interaction between modalities, though it still faces scalability issues. To mitigate the parameter explosion problem, some methods treat one modality as the primary modality and reduce the influence of the other modalities [41]. In addition, Flamingo and its medical adaptation Med-Flamingo adopt decoder-based architectures with CrossAttention for vision-language fusion, achieving strong few-shot performance [42,43].

Furthermore, LearnableQAttention captures the complex relationships between modalities by introducing trainable queries, increasing the model’s adaptability and flexibility when it comes to multimodal inputs [44,45]. However, similar to CrossAttention, this method also overlooks the correlations within each modality.

3. Data and Methodology

3.1. Dataset Construction

In this study, we constructed a multimodal, multi-label dataset for chest disease classification by combining MIMIC-CXR-JPG [46] and MIMIC-IV-ED [47], as illustrated in Figure 2. MIMIC-CXR-JPG contains chest X-ray images (in JPEG format), corresponding radiology reports, and disease label information. MIMIC-IV-ED provides time-series vital sign data of emergency department patients. First, we retained one chest X-ray image for each radiology report from the MIMIC-CXR-JPG dataset. At this stage, the data contained 65,379 patients and 227,827 reports. Then, for each chest X-ray image, we retrieved the corresponding emergency department stay record for the patient. In the edstays table, the intime and outtime columns record the time span of hospitalization, while the StudyDate and StudyTime columns in the metadata table indicate the time the chest X-ray image was taken. If the image’s capture time falls within the span of an emergency department stay, the image is associated with the corresponding stay-id; otherwise, we continue searching other hospitalization records to ensure the integrity and accuracy of the data. During the integration process, only 61,854 images matched with a stay-id, while the remaining images might have been taken after the patient left the emergency department. The final MIMIC multimodal dataset contains 61,854 chest X-ray images, corresponding radiology reports, 7 time-series vital signs, and 14 structured labels. The final dataset exhibits class imbalance in its label distribution. To alleviate this issue, we incorporated class weights during training to reduce the impact of label imbalance on model performance.

3.2. Data Preprocessing

In text preprocessing, we first used regular expressions to remove punctuation and special characters to retain the core content of the text. In addition, in the MIMIC-CXR-JPG dataset, radiology reports mainly consist of two sections: ‘Findings’ and ‘Impression’. The ‘Findings’ section mainly includes descriptive statements about the patient’s images, while the ’Impression’ section mainly contains brief diagnostic information. Considering that the labels in the MIMIC-CXR-JPG dataset are automatically extracted from the reports using the CheXpert method, we remove the ‘Impression’ part of the radiology reports during preprocessing (82.4% of reports in the original dataset contain the ’Impression’ section). Directly using the ’Impression’ information may lead to label leakage. Additionally, to ensure the completeness of the text expression, we removed short phrases consisting of fewer than three words.

For image preprocessing, we implemented several measures to enhance the model’s generalization ability and robustness. First, we resized the images to the specified dimensions through random cropping, ensuring image diversity, and used cubic interpolation to preserve image quality. Second, random horizontal flipping was applied to increase data diversity, allowing the model to learn richer information from different perspectives. Finally, normalization was performed to ensure pixel values fell within a reasonable range, improving the stability of training.

For time-series data, we conducted feature selection and outlier handling to improve data quality. First, we removed the feature “rhythm” due to its high missing rate. Next, we transformed the “pain” feature by converting non-numeric data into numeric values and standardizing different recording methods. Additionally, outliers in temperature data were corrected, specifically fixing the misrecording between Celsius and Fahrenheit. When handling missing values, we applied a neighboring value imputation method within the same stay-id to ensure data integrity and usability.

3.3. Multimodal Chest Disease Diagnosis Model

3.3.1. Overview

In the diagnosis of chest diseases, a patient’s health condition often involves multiple interrelated diseases. For instance, certain chest diseases (such as Chronic Obstructive Pulmonary Disease, COPD) may coexist with other diseases (such as lung infections or cardiovascular diseases). Therefore, multi-class classification methods cannot effectively capture the complexity of medical scenarios, and multimodal multi-label classification becomes a more clinically relevant approach for modeling. Specifically, let the set of modalities be denoted as $\mathcal{M}=1,2,\dots ,m$, where m is the total number of modalities. Then, each patient’s data can be represented as a multimodal sample $X_i=x_i^{\left(1\right)},x_i^{\left(2\right)},\dots ,x_i^{\left(m\right)}m\in \mathcal{M}$, where $x_i^{\left(m\right)}\in {\mathcal{X}}^{\left(m\right)}$ represents the feature of the m-th modality for the i-th sample, and ${\mathcal{X}}^{\left(m\right)}$ is the feature space of the m-th modality. For example, this could include imaging data, laboratory test data, gene expression data, and clinical text. Data from different modalities each carry important medical information and have strong complementarity. The final prediction result is a set of associated disease labels $Y_i=(y_1,y_2,\dots ,y_C)\in {0,1}^C$, where C is the total number of potential labels, which may include categories like pneumonia, tuberculosis, pleural effusion, heart failure, and others. Our goal is to learn a mapping function $f:{\mathcal{X}}^{\left(1\right)}\times {\mathcal{X}}^{\left(2\right)}\times \dots \times {\mathcal{X}}^{\left(M\right)}\to {[0,1]}^C$ from the given dataset $\mathcal{D}=(X_i,Y_i){i=1}^N$ by optimizing the binary cross-entropy loss (BCE), such that ${\widehat{y}}_i=f\left(X_i\right)$ is as close as possible to the true label $y_i$.

In related work, we emphasize the importance of multimodal fusion and the challenges of extending models to multiple modalities. To address these two challenges, we have developed a multimodal multi-label chest disease classification model, MDFormer, as shown in Figure 3. The model consists of three parts: a modality-specific encoder ${\Phi }_{encoder}^{\left(m\right)}$, a fusion module ${\Phi }_{MFTrans}$, and a classifier ${\Phi }_{cls}$. First, each modality is processed through its own encoder to learn the marginal representation of each modality, all of which are then jointly input into the fusion module to obtain a multimodal fused representation. Finally, the fused representation is subjected to multi-label classification to generate the multi-label disease classification results. MDFormer can be configured with different modality-specific encoders according to the use case, enabling plug-and-play adaptation to different modalities.

3.3.2. Modality-Specific Encoders

In the implementation of MDFormer, we use BioViL to initialize our image and text encoders, which are responsible for extracting the input features from images and texts. BioViL is a self-supervised vision-language model proposed by Microsoft Health in 2022 [48], specifically adapted to the chest X-ray domain. The inputs to the image and text encoders include an image $x_i^{\left(img\right)}\in {\mathbb{R}}^{C\times H\times W}$ (in this study, the image size is uniformly set to $224\times 224$, with $C=3$ representing the number of channels), and a text $x_i^{\left(txt\right)}\in {\mathbb{R}}^{L_t\times d}$, where $L_t=128$ is the text sequence length, and $d=768$ is the word embedding dimension. The specific encoding process is as follows:

$$\begin{array}{cc}\hfill e_i^{\left(img\right)}& =Pro{j}_{img}\left(BioVi{L}_{img}\left(x_i^{\left(img\right)}\right)\right)\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill e_i^{\left(txt\right)}& =BioVi{L}_{txt}\left(x_i^{\left(txt\right)}\right)\hfill \end{array}$$

(2)

Here, $Pro{j}_{img}(·)$ is the mapping operation applied to the image features to ensure they align with the dimensions of the text features. The image features $e_i^{\left(img\right)}\in {\mathbb{R}}^{L_i\times d}$ are generated by the image encoder $BioVi{L}_{img}(·)$, and the text features $e_i^{\left(txt\right)}\in {\mathbb{R}}^{L_t\times d}$ are generated by the text encoder $BioVi{L}_{txt}(·)$, where $L_i=106$ represents the length of the image features. These modality representations are then input into the multimodal fusion module to further learn both global and local crossmodal associations.

We use LightTS as the time-series encoder to model the time-series features. LightTS is a lightweight multivariate time-series model based on the MLP architecture, which improves efficiency and modeling capabilities through sampling strategies [49]. Specifically, for the vital sign time-series data $x_i^{\left(ts\right)}\in {\mathbb{R}}^{T\times C_{ts}}$, where $T=10$ is the length of the time-series and $C_{ts}=7$ represents the number of variables, the vital sign time-series data are processed by the time-series encoder and mapping module to obtain the time-series representation $e_i^{\left(ts\right)}\in {\mathbb{R}}^{L_{ts}\times d}$. The encoding process is as follows:

$$e_i^{ts}=Pro{j}_{ts}\left(LightTS\left(x_i^{\left(ts\right)}\right)\right)$$

(3)

Here, $Pro{j}_{ts}(·)$ is the projection layer, which ensures the time-series representation’s dimension is compatible with the subsequent modalities. Through this design, MDFormer is able to efficiently learn from the vital sign data and provide rich time-series features for the subsequent multimodal fusion module.

3.3.3. MFTrans

Before performing multimodal fusion, we first apply each modality-specific encoder independently to extract high-dimensional feature representations for each modality. Specifically, for the j-th modality data of the i-th patient, after being processed by the encoder, we obtain the feature representation $e_i^{\left(j\right)}$, where $j\in \{img,txt,ts\}$. To further enhance the distinguishability of features from different modalities, we introduce modality type embeddings, denoted as $t_j$, to the representation of each modality. Modality type embeddings assign a learnable vector to each modality, helping the model understand the specific attributes of that modality while strengthening the interactions between different modalities. Then, we perform an addition operation on $e_i^{\left(j\right)}$ and $t_j$, followed by normalization via LayerNorm to stabilize the training process, resulting in ${\widehat{e}}_i^{\left(j\right)}\in {\mathbb{R}}^{L_j\times d}$. The specific process is as follows:

$${\widehat{e}}_i^{\left(j\right)}=LayerNorm(e_i^{\left(j\right)}+t_j)$$

(4)

Based on this, we design a Multimodal Fusion Attention (MFAttention) mechanism for crossmodal information fusion, as shown in Figure 4. In MFAttention, there are three independent query mapping layers $W_q^{\left(j\right)}$, where the features ${\widehat{e}}_i^{\left(j\right)}$ of each modality are passed through their respective query mappings $W_q^{\left(j\right)}$ to obtain the corresponding query matrices $Q_i^{\left(j\right)}$. Additionally, MFAttention includes a key mapping layer $W_k$ and a value mapping layer $W_v$. The concatenated representation of the three modalities $[{\widehat{e}}_i^{\left(img\right)};{\widehat{e}}_i^{\left(txt\right)};{\widehat{e}}_i^{\left(ts\right)}]$ is passed through the key mapping layer $W_k$ and the value mapping layer $W_v$ to obtain the key matrix $K_i$ and the value matrix $V_i$. The specific computation process is as follows:

$$\begin{array}{cc}\hfill Q_i^{\left(j\right)}& =W_q^{\left(j\right)}{\widehat{e}}_i^{\left(j\right)}\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill K_i& =W_k[{\widehat{e}}_{img};{\widehat{e}}_{txt};{\widehat{e}}_{ts}]\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill V_i& =W_v[{\widehat{e}}_{img};{\widehat{e}}_{txt};{\widehat{e}}_{ts}]\hfill \end{array}$$

(7)

Subsequently, based on scaled dot-product attention, the attention output for each modality $A_i^{\left(j\right)}$ is calculated, where $A_i^{\left(j\right)}$ represents the features after multimodal interaction. To further enhance the expression of local information, we introduce group convolution [50] on each modality feature, obtaining the convolution output $Con{v}_i^{\left(j\right)}$. Group convolution is an efficient feature extraction method that divides the features into several groups and performs convolution operations independently on each group, reducing computational cost and enhancing the model’s ability to perceive local features. In this study, we set the number of groups to 12, with a kernel size of 3, stride of 1, and padding of 1, ensuring the input and output have the same shape. After that, we add the outputs from MFAttention $A_i^{\left(j\right)}$, the group convolution $Con{v}_i^{\left(j\right)}$, and the residual connection $e_i^{\left(j\right)}$ to obtain $AttCon{v}_i^{\left(j\right)}$. Next, we input the three fused modality features into a shared Feedforward Neural Network (FFN) module to obtain the representation $F_i^{\left(j\right)}$. We then stack 6 layers of the fusion layer described above to form the fusion module MFTrans, as shown on the right side of Figure 3. Finally, the outputs of the three modalities are concatenated and aggregated through max-pooling to obtain the final output of MFTrans, $M_i\in {\mathbb{R}}^d$:

$$\begin{array}{cc}\hfill A_i^{\left(j\right)}& =Attention(Q_i^{\left(j\right)},K_i,V_i)\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill Con{v}_i^{\left(j\right)}& =GroupConv\left({\widehat{e}}_i^{\left(j\right)}\right)\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill AttCon{v}_i^{\left(j\right)}& =A_i^{\left(j\right)}+Con{v}_i^{\left(j\right)}+e_i^{\left(j\right)}\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill F_i^{\left(j\right)}& =FFN\left(AttCon{v}_i^{\left(j\right)}\right)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill M_i& =Maxpooling\left([F_i^{\left(img\right)};F_i^{\left(txt\right)};F_i^{\left(ts\right)}]\right)\hfill \end{array}$$

(12)

In summary, MFAttention primarily models global modality interactions and is suitable for long-term dependency modeling across modalities, while group convolution enhances the perception of local features, helping to capture fine-grained modality-specific features. Together, they complement each other. This combination allows MDFormer to effectively fuse multimodal information at different scales, improving its ability to understand clinical data.

3.3.4. Classification Head

In MDFormer, our classification head is used for multi-label chest disease classification tasks. Given the fused multimodal feature vector $M_i$, we first perform feature mapping through two fully connected layers, where the first layer uses a ReLU activation function to introduce non-linear transformations. Finally, we use a Sigmoid activation function to generate the class probability distribution, obtaining the final prediction vector ${\widehat{y}}_i\in {\mathbb{R}}^C$, where C is the number of classes. We optimize the model using binary cross-entropy loss, as shown below:

$$\begin{array}{cc}\hfill {\widehat{y}}_i& =Sigmoid(W_2·ReLU(W_1·M_i+b_1)+b_2)\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{cls}& =-{\displaystyle \frac{1}{N\times C}}\sum _{i=1}^N\sum _{j=1}^C[y_{ij}log\left({\widehat{y}}_{ij}\right)+(1-y_{ij})log(1-{\widehat{y}}_{ij})]\hfill \end{array}$$

(14)

Here, N is the batch size, C is the total number of classes, $y_{ij}\in [0,1]$ represents the true label of sample i for class j, and ${\widehat{y}}_{ij}$ is the predicted probability output by the Sigmoid function.

3.4. Two-Stage Training Framework

3.4.1. Overview

In the process of actual clinical data collection, missing modality is an unavoidable problem. We aim to build a multimodal multi-label classification model that can handle missing modality. Let the set of modalities for a sample be $\mathcal{M}=1,2,\dots ,m$, and the actual available subset of modalities be $\Omega \in \mathcal{M}$. Then, the input data for the i-th patient becomes $X_i^{\Omega }=x_i^{\left(m\right)}m\in \Omega$, meaning the input data includes only the modalities observed in $\Omega$. At this point, the mapping that the model needs to learn becomes $f\Omega :{\prod }_{m\in \Omega }{\mathcal{X}}^{\left(m\right)}\to {[0,1]}^C$.

To address the above problem, we design a two-stage Masked Enhanced Classification and Contrastive Learning training framework, MECCL, to improve the model’s classification performance in the case of missing modality while maintaining the stability of its performance when all modalities are available. The core idea of this training framework is to simulate missing modality during training so that the model learns how to classify effectively with incomplete data. The overall structure of the framework is shown in Figure 5. To ensure the efficiency of model training and fully utilize the representations learned in the first stage, we only adjust the parameters of the model fusion module MFTrans and the classification head in the second stage, while freezing the modality-specific encoders. The parameters adjusted in the second stage account for only 20% of the original model size. Four losses are jointly optimized: the classification loss for complete modalities ${\mathcal{L}}_{cls}$, the classification loss for modality masks ${\mathcal{L}}_{clsm}$, the modality mask contrastive learning loss ${\mathcal{L}}_{mmc}$, and the KL divergence loss ${\mathcal{L}}_{KL}$.

3.4.2. Modality Masked Classification

Based on the first stage, we introduce a random dynamic masking enhancement strategy in the second stage to improve the model’s performance on missing modalities while maintaining stability for complete modalities. Specifically, in each training iteration, we apply a random mask to the input sample $X_i$, meaning that we randomly mask one or two modalities to generate a masked sample $X_i^{\Omega }$. This process simulates the incomplete data scenarios often encountered in real clinical applications. The dynamic masking strategy forces the model to learn in the presence of missing modalities, thereby enhancing its adaptability to missing modalities.

Subsequently, for the complete modality input $X_i$ and the masked modality input $X_i^{\Omega }$, two forward passes are performed, yielding prediction results based on the complete dataset $\widehat{y}i=f\theta \left(X_i\right)$ and prediction results based on the masked dataset $\widehat{y}{i}^{\Omega }=f\theta \left(X_i^{\Omega }\right)$. Then, the binary cross-entropy classification losses are computed for each:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{cls}& =-{\displaystyle \frac{1}{N\times C}}\sum _{i=1}^N\sum _{j=1}^C[y_{ij}log\left({\widehat{y}}_{ij}\right)+(1-y_{ij})log(1-{\widehat{y}}_{ij})]\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{clsm}& =-{\displaystyle \frac{1}{N\times C}}\sum _{i=1}^N\sum _{j=1}^C[y_{ij}log\left({\widehat{y}}_{ij}^{\Omega }\right)+(1-y_{ij})log(1-{\widehat{y}}_{ij}^{\Omega })]\hfill \end{array}$$

(16)

Through $\mathcal{L}cls$, we constrain the classification ability on the complete modality input, ensuring the model maintains high classification accuracy when modalities are intact. By introducing the random dynamic masking enhancement strategy, $\mathcal{L}clsm$ directly addresses the classification task on masked modalities, allowing the model to make accurate predictions even with missing modalities. Since the mask is random, the model must learn to predict under different modality combinations, enhancing the information complementarity between modalities and preventing the model from over-relying on any specific modality.

3.4.3. Modality Masked Contrastive Learning

In order to enable the model to effectively represent disease information even in the case of missing modality, we introduce Multimodal Masked Contrastive Learning, as shown in the Figure 6. This method enhances the consistency of representations between modalities through contrastive learning, thereby alleviating the issue of incomplete information caused by modality missing. Contrastive Learning is a self-supervised learning method that aims to bring the representations of positive samples closer together while separating those of negative samples. A commonly used loss function is the InfoNCE loss:

$${\mathcal{L}}_{InfoNCE}(q,k)=-{\displaystyle \frac{1}{N}}\sum _{i=1}^{N}log{\displaystyle \frac{exp(s(q_i,k_i)/\tau )}{{\sum }_{j=1}^{N}exp(s(q_i,k_j)/\tau )}}$$

(17)

where N is the batch size, $s(q_i,k_j)$ is the similarity between vectors $q_i$ and $k_j$, and $\tau$ is the temperature coefficient, used to control the smoothness of the Softmax distribution.

In Multimodal Masked Contrastive Learning, we treat the complete modality data $X_i$ and the corresponding masked modality data $X_i^{\Omega }$ as positive sample pairs, while other samples in a batch are considered negative samples. The goal of the model is to maximize the similarity between these positive samples while minimizing their similarity to negative samples. Specifically, for complete modality input sample $X_i$ and masked modality input sample $X_i^{\Omega }$, we first pass them through modality-specific encoders for feature extraction to obtain preliminary modality features. These preliminary representations are then processed by the fusion module MFTrans to produce the fused representations $M_i$ and $M_i^{\Omega }$. To further optimize the contrastive learning of features, we add a linear transformation layer and a normalization layer for feature transformation. After this processing, we obtain two normalized representations $Z_i$ and $Z_i^{\Omega }$. The specific formulas are as follows:

$$\begin{array}{cc}\hfill Z_i& =Norm\left(Linear\left(M_i\right)\right)\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill Z_i^{\Omega }& =Norm\left(Linear\left(M_i^{\Omega }\right)\right)\hfill \end{array}$$

(19)

where $Linear(·)$ represents a fully connected layer used to map the fused representations to the feature space of contrastive learning, and $Norm(·)$ refers to L2 normalization to enhance the stability of similarity calculation. We then perform contrastive learning on these two normalized representations and compute the Multimodal Masked Contrastive Learning loss $\mathcal{L}mmc$:

$${\mathcal{L}}_{mmc}={\displaystyle \frac{1}{2}}[{\mathcal{L}}_{InfoNCE}(Z_i,Z_i^{\Omega })+{\mathcal{L}}_{InfoNCE}(Z_i^{\Omega },Z_i)]$$

(20)

This objective encourages the model to learn crossmodal relationships and maximize the similarity between representations of complete data and modality-masked data. Furthermore, in contrastive learning, the choice of negative samples is crucial to the model’s training. However, in real-world applications, due to the uncertainty of the missing modality problem, negative samples may contain examples semantically similar to positive samples, leading to false negatives, where the model misjudges the sample as negative. This problem is referred to as the false negative problem. In the process of multimodal masked contrastive learning, a batch may still contain relevant positive samples, but the one-hot labels used in contrastive learning will penalize all negative samples indiscriminately, introducing noise during training. To alleviate this issue, we introduce pseudo-targets generated by a momentum model for learning. The momentum model $f_{{\theta }_m}$ is a continuously evolving teacher model, obtained by the Exponential Moving Average (EMA) of MDFormer.

$${\theta }_m\leftarrow \alpha {\theta }_m+(1-\alpha )\theta$$

(21)

where $\alpha =0.995$ is the momentum update coefficient. We then pass the complete modality input sample $X_i$ and modality-masked input sample $X_i^{\Omega }$ into the momentum model to obtain the corresponding momentum representations $M_i^{mom}$ and $M_i^{mom,\Omega }$.

$$\begin{array}{cc}\hfill Z_i^{mom}& =Norm\left(Linear\left(M_i^{mom}\right)\right)\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill Z_i^{mom,\Omega }& =Norm\left(Linear\left(M_i^{mom,\Omega }\right)\right)\hfill \end{array}$$

(23)

where $Z_i^{mom}$ is the momentum representation of the complete modality sample, and $Z_i^{mom,\Omega }$ is the momentum representation of the modality-masked sample. We then generate additional predictions through the momentum model and introduce the KL divergence loss $\mathcal{L}KL$ to constrain the distribution consistency between MDFormer and the momentum model:

$$\begin{array}{cc}\hfill p_i^{f2m}\left(Z_i\right)& ={\displaystyle \frac{exp(s(Z_i,Z_i^{\Omega })/\tau )}{\sum {j=1}^{N}exp(s(Z_i,Z_j^{\Omega })/\tau )}}\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill p_i^{m2f}\left(Z_i^{\Omega }\right)& ={\displaystyle \frac{exp(s(Z_i^{\Omega },Z_i)/\tau )}{{\sum }_{j=1}^{N}exp(s(Z_i^{\Omega },Z_j)/\tau )}}\hfill \end{array}$$

(25)

$${\mathcal{L}}_{KL}={\displaystyle \frac{1}{2}}\left[KL\right(q^{f2m}\left(Z\right)\left|\right|p^{f2m}\left(Z\right))+KL(q^{m2f}\left(Z^{\Omega }\right)\left|\right|p^{m2f}\left(Z^{\Omega }\right)\left)\right]$$

(26)

where $p^{f2m}\left(Z_i\right)$ and $p_i^{f2m}\left(Z_i^{\Omega }\right)$ represent the probability distributions of the similarity between complete modality samples and modality-masked samples after Softmax normalization, generated by MDFormer. $q^{f2m}\left(Z_i\right)$ and $q_i^{f2m}\left(Z_i^{\Omega }\right)$ are the probability distributions generated by the momentum model. $KL(·||·)$ calculates the KL divergence between two probability distributions, encouraging the model’s prediction distribution to match the momentum model’s distribution as closely as possible.

3.4.4. Dynamic Balancing of Loss Weights

To better balance different loss functions in the second-stage training, we leverage the non-linear adjustment properties of the Sigmoid function to dynamically adjust the loss weights based on the number of training epochs. This allows for the allocation of the most appropriate optimization objectives at different stages of training. Specifically, in the first 5 epochs of training, we use the Sigmoid function to gradually increase the loss weight $w_{clsm}$ for $\mathcal{L}clsm$ and decrease the loss weight $wmmc$ for $\mathcal{L}mmc$. Meanwhile, the weights $wcls$ and $w_{kl}$ for $\mathcal{L}cls$ and $\mathcal{L}KL$ are kept constant at 1. This ensures training stability, allowing the model to more stably adapt to the modality mask early in the second-stage training and establish a more robust representational foundation. The specific weight adjustment formulas are as follows:

$$\begin{array}{cc}\hfill w_{clsm}& =1+4·{\displaystyle \frac{1}{1+e^{-k·\frac{epoch-mid}{adjust\_epochs}}}}\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill w_{mmc}& =9-4·{\displaystyle \frac{1}{1+e^{-k·\frac{epoch-mid}{adjust\_epochs}}}}\hfill \end{array}$$

(28)

Here, $epoch$ represents the current training epoch, $mid$ is the midpoint of the adjustment, i.e., half of the total adjustment epochs, and $adjust\_epochs$ is the total number of adjustment epochs, set to 5 epochs. This means that the loss weights are dynamically adjusted in the first 5 epochs, after which the loss weights remain constant. k controls the steepness of the Sigmoid curve, governing the rate of weight adjustment; it is set to 10, where a larger k results in a steeper adjustment process, and a smaller k results in a gentler one. Finally, combining the four loss terms, the second-stage loss function adopts the following weighted form:

$${\mathcal{L}}_{stage2}=w_{cls}·{\mathcal{L}}_{cls}+w_{clsm}·{\mathcal{L}}_{clsm}+w_{mmc}·{\mathcal{L}}_{mmc}+w_{kl}·{\mathcal{L}}_{kl}$$

(29)

This method allows the model to better focus on modality mask contrastive learning during the early stages of training, thereby learning the effective relationship between full modality samples and modality mask samples. As training progresses, the model gradually shifts more attention to the modality mask classification task, ensuring that its classification capability in the case of missing modalities is fully improved.

4. Results and Discussion

4.1. Experimental Setup

This paper compares the performance of different types of models on the MIMIC dataset, divided into three categories for analysis. The first category consists of unimodal models, including DenseNet121 [51], BioLinkBert [52], and Informer [53]; the second category includes bimodal models such as MedFuse [54], ALBEF [55], and MedCLIP [56]; and the third category includes trimodal models, mainly ConcantateFusion and MedFusePlus.

Among these, ConcantateFusion is a simple multimodal fusion model. Its encoder structure is identical to that of MDFormer, but in the fusion stage, the representations of the three modalities are directly concatenated to generate a unified feature representation, which is then used for classification tasks. MedFusePlus is an extended version of MedFuse, incorporating BioLinkBert as the text encoder, and further extends multimodal learning from clinical time-series data and chest X-ray images into a trimodal model. Additionally, we compared various Transformer-based multimodal fusion methods, including Self-Attention, Cross-Attention, LearnableQAttn, and Cross-Self-Attention. The experiments use precision, recall, F1 score, ROC curve, and AUC value as evaluation metrics.

The replication of all baseline models follows the configuration from the original papers or official code, with a consistent classification head design across all models. In the experiments, 10% of the samples were kept for evaluation. In the second-stage masked test set, 50% of the data were randomly masked, with one or two modalities masked at a time to simulate missing modality scenarios. Our method used the AdamW optimizer for training, with the following parameter settings: ${\beta }_1=0.9$, ${\beta }_2=0.98$, and $ϵ=1\times {10}^{-6}$, a maximum learning rate of $5\times {10}^{-5}$, and cosine learning rate decay. The batch size was set to 256, and for models exceeding memory capacity, it was adjusted to 128. The training process was stopped early after 15 epochs, with a weight decay coefficient set to 0.05 and a learning rate warmup ratio of 0.03.

4.2. Comparative Experiment

The comparison of the experiments mainly includes two aspects: first, the classification performance comparison of different models on the same dataset; second, exploring the impact of different multimodal data fusion strategies on classification performance under the same encoder. The best and second-best values of the experimental results are marked in bold and underlined, respectively, for an intuitive comparison of the model’s performance.

“I” represents chest X-ray image data, “T” represents radiology report data, “TS” represents vital signs time-series data, and “✓” indicates the modality used during training.

Table 1. Results of MDFormer and baseline models.

Modalities			Model	Precision	Recall	F1	AUC	Params	Inference Time
I	T	TS
✓			Densenet121	0.6012	0.5956	0.5984	0.7162	8M	23.75 ms
	✓		BioLinkBert	0.7166	0.6863	0.7011	0.8164	108M	40.84 ms
		✓	Informer	0.5397	0.5271	0.5333	0.6793	27M	23.63 ms
✓		✓	MedFuse	0.6627	0.6494	0.6560	0.7647	23M	31.71 ms
✓	✓		ALBEF	0.7699	0.7776	0.7737	0.8890	209M	26.11 ms
✓	✓		MedCLIP	0.7556	0.7169	0.7357	0.8585	137M	29.70 ms
✓	✓	✓	ConcatenateFusion	0.7688	0.7231	0.7437	0.8689	138M	23.89 ms
✓	✓	✓	MedFusionPlus	0.7601	0.7488	0.7632	0.8731	132M	24.60 ms
✓	✓	✓	MDFormer	0.8043	0.7891	0.7966	0.9028	173M	26.63 ms

Note: The best results are highlighted in bold, and the second-best results are underlined.

presents the classification performance of different models on the MIMIC dataset. It can be seen that MDFormer achieved the best results across all metrics. Specifically, MDFormer reached an F1 score of 0.7966, which is a 3% improvement over the second-best model, ALBEF, confirming its effectiveness in multimodal fusion. Notably, among all the unimodal models, BioLinkBert achieved an AUC of 0.8164, significantly higher than DenseNet121 and Informer. This indicates that the text modality performs better than chest X-ray images and vital signs time-series data in the chest disease diagnosis task. This may be because radiology reports provide a more intuitive summary of the condition, not only highlighting key points from the image data but also incorporating the doctor’s professional knowledge. Additionally, we observed that multimodal fusion methods generally outperform unimodal models. For example, MedFuse and ConcatenateFusion significantly surpass the unimodal baselines. Furthermore, ALBEF, which uses both radiology reports and chest X-ray images, outperforms ConcatenateFusion and MedFusePlus, indicating that simple concatenation or LSTM-based fusion strategies cannot fully leverage multimodal information, while more advanced crossmodal interaction mechanisms can further improve classification performance.

To further assess the practicality of each model in clinical settings, we compare their average inference time on the test set. As shown in Table 1, although MDFormer achieves the best overall performance across all evaluation metrics, its inference time (26.63 ms) remains within a reasonable range. In fact, it is comparable to or even faster than some less accurate multimodal models, such as MedFuse (31.71 ms) and MedCLIP (29.70 ms). This demonstrates that MDFormer not only provides high accuracy but also maintains efficient inference speed, making it suitable for real-time or time-sensitive medical applications. In addition, we performed 100 bootstrap resamplings on the test set. MDFormer achieved an average F1 score of 0.7920 (95% CI: 0.7796–0.8044) and an AUC of 0.9011 (95% CI: 0.8948–0.9074), while ALBEF achieved an average F1 score of 0.7720 (95% CI: 0.7596–0.7844) and an AUC of 0.8860 (95% CI: 0.8798–0.8922). Furthermore, using the same data splits, we conducted paired Wilcoxon signed-rank tests to examine the differences in F1 score and AUC between the two methods. The results showed statistically significant improvements (F1: p = 0.0047; AUC: p = 0.0031), further validating the stability and reliability of our approach. MDFormer achieves the best results across all metrics mainly due to its use of the MFAttention and group convolution feature extraction, which effectively capture the correlations between different modalities and integrate both local and global features.

From the above experimental data, the following conclusions can be drawn: Unimodal models generally perform poorly due to the lack of crossmodal complementary information, limiting the model’s discriminative ability. Simple multimodal fusion methods fail to fully utilize the complementary advantages of modalities and do not adequately explore the synergistic information between modalities. MDFormer achieves optimal results on various datasets thanks to its multimodal attention mechanism and the adaptive fusion module, MFTrans, which can flexibly handle different multimodal data.

Figure 7 shows the ROC curves of MDFormer and baseline models on the MIMIC dataset. The x-axis represents the false positive rate (FPR), and the y-axis represents the true positive rate (TPR). The values in parentheses in the legend represent the model’s AUC score. It is clearly visible that the ROC curve of MDFormer is above all the baseline models, further confirming its advantage in the chest disease classification task. Especially compared to MedFusionPlus and ALBEF, MDFormer shows a significant improvement in TPR in the low-FPR region, indicating stronger discriminatory ability.

Additionally, we used different Transformer-based multimodal fusion methods in combination with a fixed encoder for comparative analysis, and the experimental results are shown in

Table 2. Results of different fusion methods.

Fusion Method	Precision	Recall	F1	AUC	Params
SelfAttention	0.7703	0.7696	0.7699	0.8756	18M
CrossAttention	0.7905	0.7872	0.7888	0.8912	114M
Cross-Self-Attention	0.7923	0.7774	0.7848	0.8876	170M
LearnableQAttn	0.7744	0.7619	0.7682	0.8669	18M
MFTrans	0.8043	0.7891	0.7966	0.9028	36M

Note: The best results are highlighted in bold, and the second-best results are underlined.

. To visually demonstrate the relationship between performance and parameter count for different fusion methods on the MIMIC dataset, we use a bubble chart for visualization, as shown in Figure 8. In this chart, the x-axis represents AUC, the y-axis represents F1, and the size of the bubbles corresponds to the model’s parameter count. First, MFTrans achieved the best performance on all key metrics. At the same time, MFTrans uses only 36M parameters, while CrossAttention and Cross-Self-Attention require 3.1 times and 4.6 times the number of parameters as MFTrans, respectively. This indicates that, although CrossAttention-related methods have certain performance advantages, they come with a large parameter overhead and poor scalability, whereas MFTrans achieves optimal performance while maintaining a lower parameter count. Furthermore, SelfAttention and LearnableQAttn have smaller parameter counts but significantly lag behind MFTrans in all performance metrics.

Finally, we present the hyperparameter tuning experiments for the fusion module MFTrans, focusing on evaluating the impact of different numbers of fusion layers and grouped convolution groups on model performance. The experimental results are shown in the Figure 9. As illustrated in the left panel, the model’s F1 score steadily improves as the number of fusion layers increases from two to six, with a particularly notable gain between two and four layers. This suggests that increasing the number of fusion layers helps to more effectively model multimodal information. However, when the number of layers increases to eight, the performance gain saturates and even slightly declines, possibly due to redundant information or overfitting introduced by the overly deep structure. In the right panel, the number of grouped convolutions has a limited effect on performance when it exceeds two. Specifically, from 4 to 12 groups, the F1 score remains around 0.796, but performance slightly drops when the group number exceeds 16. These results indicate that moderately increasing the number of convolution groups can reduce the number of parameters, but an excessive number may harm model performance. Considering both performance and computational cost, the optimal configuration for the fusion module is six fusion layers and 12 convolution groups.

4.3. Ablation Study

MDFormer uses MFTrans to jointly interpret multimodal information, effectively capturing both global and local features. To evaluate the actual contribution of each component of MFTrans to the model’s performance, we conducted ablation experiments on the MIMIC dataset, comparing the following MDFormer variants:

• w/o MFAttention: Removes the MFAttention, relying only on the self-attention mechanisms of individual modalities for information extraction in the fusion module.
• w/o Group Convolution: Removes the group convolution bypass.
• w/o MFTrans: Completely removes the MFTrans fusion module.

Table 3. The results of the ablation experiments.

Model	Precision	Recall	F1	AUC
w/o MFAttention	0.7712	0.7613	0.7662	0.8786
w/o Group Convolution	0.7904	0.7702	0.7821	0.8931
w/o MFTrans	0.7688	0.7231	0.7437	0.8539
MDFormer	0.8043	0.7891	0.7966	0.9028

Note: The best results are highlighted in bold.

present the results of the ablation experiments. It can be observed that, on both datasets, the classification performance of MDFormer slightly decreases when the group convolution bypass is removed, indicating that group convolution effectively extracts local information from each modality and contributes to the model’s classification performance. When MFAttention is removed, the model’s metrics are significantly lower than those of the full model, suggesting that MFAttention, compared to the self-attention of individual modalities, better captures the correlations between modalities, thus enhancing the feature fusion capability. Additionally, combining MFAttention with group convolution further improves the model’s performance, indicating that both components complement each other in the multimodal information fusion process. Among all the ablation experiments, removing MFTrans had the most significant impact, with a noticeable drop in model performance, further confirming the crucial role of the fusion module in enhancing the model’s predictive ability.

4.4. Two-Stage Training Experiment

To comprehensively evaluate the effectiveness of the MECCL two-stage training method, we compared the classification performance differences of the MDFormer model before and after the two-stage training on both the full modality dataset and the modality-masked dataset. The experimental results are shown in

Table 4. MDFormer two-stage training results.

Model	Dataset	Precision	Recall	F1	AUC
one-stage model	original test set	0.8043	0.7891	0.7966	0.9028
	masked test set	0.4566	0.4271	0.4441	0.5943
two-stage model	original test set	0.8006	0.7842	0.7923	0.9024
	masked test set	0.6770	0.6505	0.6675	0.7702

. The changes in ROC before and after the two-stage training are shown in Figure 10.

As can be seen from the table, on the MIMIC dataset, the F1 score of the one-stage model on the original test set is 0.7966, while it drops to only 0.4441 on the masked test set. In contrast, after the MECCL two-stage training, although the F1 score on the original test set remains nearly unchanged at 0.7923, its performance on the masked test set significantly improves to 0.6675, which is about a 50% increase compared to the one-stage model, with an AUC improvement of 29.6%. These results indicate that the two-stage training effectively enhanced the model’s classification ability in the presence of modality missing data.

Meanwhile, the two-stage model’s classification performance on the full modality test set remains almost unchanged, indicating that the MECCL training strategy does not negatively affect the model’s performance with complete modalities. Additionally, the two-stage training further optimized the model’s performance on the modality-masked dataset while freezing modality-specific encoders. This suggests that the model structure of MDFormer has already learned to represent multimodal data in the first-stage training and has extracted robust feature representations. Through modality-masked classification and modality-masked contrastive learning, the model can better identify the important features of existing modalities, and by comparing the representations of modality-missing and complete data, it further aids the model in recovering complete labels from missing data. This training strategy not only improved the model’s classification performance on modality-missing data but also enhanced its generalization ability, allowing it to more flexibly handle data with different modality combinations.

4.5. Visualization Analysis

We performed PCA dimensionality reduction and visualization on the final representations obtained by MDFormer for all cases in the test set, as shown in Figure 11. The reduced-dimensional results reveal a clear clustering of samples from different categories in the two-dimensional space, indicating that MDFormer effectively integrates multimodal medical data.

5. Discussion

Beyond technical advancements, we also deeply recognize the significant importance of applying MDFormer in real clinical environments. In emergency departments or resource-limited medical settings, the ability to make accurate decisions quickly based on incomplete multimodal data is critical for clinical decision-making. Our experimental results show that MDFormer demonstrates strong robustness even under conditions of modality missingness, making it an ideal choice for such application scenarios. Especially in the context of telemedicine or remote regions with scarce medical resources, MDFormer shows great potential for practical deployment. To further enhance MDFormer’s adaptability to real-world clinical settings, we plan to expand our experiments to more comprehensive multimodal datasets (such as CheXpert) and incorporate textual perturbation analysis involving low-quality or non-standard radiology reports. This will allow us to systematically assess the model’s stability and practicality when faced with cross-institutional and heterogeneous data. Future research will also focus on the seamless integration of MDFormer into existing clinical workflows, such as Electronic Health Record (EHR) systems, targeting key tasks like early disease detection and critical case triage. This includes addressing the challenges posed by asynchronous or real-time multimodal data input. Specifically, MDFormer is expected to serve as an intelligent decision support tool within EHR systems, enabling real-time analysis of patient data from various modalities to assist clinicians in identifying early disease signals more accurately, providing diagnostic recommendations, or automatically assessing the urgency of clinical cases. Furthermore, we will explore the potential of MDFormer in disease progression prediction and personalized treatment planning. To meet the practical needs of telemedicine under edge-computing resource constraints, we also plan to conduct research on model lightweighting and deployment optimization. In summary, these efforts will lay a solid foundation for advancing MDFormer from research into clinical application. Future work will primarily focus on the following directions:

• Future work may consider integrating more data modalities, such as genomic data, pathology data, and laboratory test results, to build a more comprehensive multimodal fusion framework for more accurate diagnosis.
• It could also validate the model’s generalization ability on a broader range of multimodal datasets and systematically evaluate its robustness under low-quality or non-standard input conditions.
• Additionally, self-supervised or semi-supervised learning methods can be adopted to train multimodal medical encoders with strong representation abilities, better adapting to the needs of downstream classification tasks.
• Such work can explore lightweight deployment techniques such as model quantization, pruning, and knowledge distillation to enable real-time, resource-efficient inference on edge devices, which is essential for mobile healthcare and remote diagnostic scenarios.

6. Conclusions

This paper first combines the attention mechanism and grouped convolutions to propose a multimodal fusion attention mechanism called MFAttention. MFAttention can not only mine valuable features within each individual modality but also capture crossmodal global semantic information, further improving the effectiveness of multimodal data fusion. Based on this, the paper further constructs a multimodal fusion module, MFTrans, which effectively integrates heterogeneous data from medical images, clinical texts, and vital sign time-series. This process not only enhances the model’s expressive power but also significantly improves the model’s accuracy and stability in disease diagnosis. Experimental comparisons with baseline models on the MIMIC dataset demonstrate that MDFormer performs well in chest disease classification tasks. Additionally, ablation experiments validate the effectiveness of the various modules in MDFormer.

Next, considering the issue of missing modality in practical applications, this paper proposes an innovative two-stage Mask-enhanced Classification and Contrastive Learning training framework, MECCL. Specifically, the second-stage training simultaneously optimizes the classification loss of complete modality data and the classification loss of masked data. By combining contrastive learning, it maximizes the representation consistency between the two, thereby improving the model’s robustness and generalization ability. Furthermore, to balance various losses, the paper introduces a Sigmoid function to construct a dynamic balancing loss weight. The experimental results show that the proposed model not only achieves excellent classification performance when multiple modalities are complete but also maintains good stability and diagnostic effectiveness when facing missing modalities. We will release our code at https://github.com/thulxl/MFTrans upon publication to support reproducibility (accessed on 6 May 2025).