MMAC-Net: A Multi-Modal Multi-Label Attention-Based Deep Learning Approach for Automated ICD-9 Coding of Rare Disease Admissions from Electronic Health Records

Abstract

Automating the identification of International Classification of Diseases (ICD) codes from electronic health records (EHRs) presents a critical challenge, particularly for rare diseases where existing computational methods severely underperform due to extreme long-tail label distributions. To address this, we propose a multi-modal deep learning framework known as MMAC-Net, designed to enhance the retrospective assignment of ICD-9 codes to admissions involving rare pathologies. The model integrates unstructured clinical narratives with structured auxiliary data, specifically pharmacological prescriptions and microbiology events, using a convolutional attention-based architecture. Through a late fusion mechanism, it synthesizes attention-weighted textual representations with dense embeddings of the structured data types. Validation on the MIMIC-III dataset shows consistent improvements over a matched text-only baseline evaluated under an identical protocol. On the full dataset of 8930 ICD codes, the framework achieved a Micro-AUC of 0.997 and Precision@8 of 0.875. On the subset of admissions carrying at least 1 of 568 rare codes, adding the two structured modalities to the text encoder raises Macro-F1 from 0.011 to 0.084 and Micro-F1 from 0.368 to 0.513 relative to the text-only baseline, corresponding to relative increases of 6.69 and 0.39, respectively, while Precision@8 rises from 0.092 to 0.159 and Micro-AUC from 0.966 to 0.985. While extreme class imbalance remains a formidable obstacle, these findings underscore that incorporating structured clinical context partially mitigates the limitations of purely natural language processing approaches. Practically, the framework is intended as a decision-support component that presents a ranked shortlist of candidate codes to a human coder or clinician; by recovering rare codes that text-only systems miss, it targets the under-coding of low-prevalence conditions that degrades registry completeness and downstream epidemiological estimates.

1. Introduction

The domain of medical informatics has witnessed a paradigm shift with the advent of high-performance computing and the availability of large-scale electronic health records (EHRs). However, despite these advancements, the identification and diagnosis of rare diseases remains a formidable challenge in modern healthcare. While individual rare diseases affect a small percentage of the population, collectively, they represent a significant global health burden. This study explores the application of advanced deep learning methodologies to address the complexities of automated ICD coding of admissions involving rare diseases, utilizing the rich, multi-modal data available in the MIMIC-III dataset.

1.1. Problem Statement

The core problem addressed in this research is the computational difficulty of accurately identifying rare diseases within sparse and noisy clinical data. By definition, a rare disease affects fewer than 200,000 individuals in the United States [1], or fewer than 1 in 2000 people globally [2]. This statistical rarity creates a long-tail distribution of diseases in medical datasets [3,4], where the vast majority of training examples correspond to common conditions (e.g., hypertension, diabetes), while rare diseases are represented by remarkably few instances.

In the context of automated International Classification of Diseases (ICD) coding, this manifests itself as an extreme multi-label classification problem. Standard machine learning models tend to bias towards majority classes, frequently misclassifying or failing to detect rare conditions entirely. Furthermore, the lack of a universally accepted nomenclature is a predominant challenge as observed by the differences between ORPHAcode, ICD-9, ICD-10, and ICD-11 classifications. This complicates the mapping of clinical phenotypes to specific diagnostic codes [2,5]. Consequently, significant diagnostic delays occur in clinical settings, contributing to adverse patient outcomes that could be mitigated through earlier detection.

Throughout this article, we distinguish ICD coding from clinical diagnosis. ICD coding is the retrospective assignment of administrative classification codes to a completed hospital admission on the basis of the documentation produced during that admission. We accordingly use ICD coding or code assignment for the task, diagnosis code and procedure code for the two label types, and reserve diagnosis for the clinical act.

1.2. Motivation

The motivation for this study stems from the staggering global impact of these orphan or rare diseases. Researchers have identified over 7000 distinct rare diseases [6]. While individually rare, they impact approximately 25 million people in the United States and over 300 million people worldwide, a demographic often composed of children [6].

Current clinical workflows are often ill-equipped to handle the nuances of these conditions. Patients frequently face a diagnostic odyssey [7,8], characterized by repeated misdiagnoses and a lack of methodical treatment plans due to the scarcity of specialized medical knowledge, particularly in underdeveloped regions [9].

There is a compelling need to leverage artificial intelligence (AI) to bridge this gap. The Medical Information Mart for Intensive Care (MIMIC-III) [10] dataset offers a unique opportunity to train such models. Containing over 53,000 distinct hospital admissions and multi-modal data, including vital signs, laboratory measurements, and unstructured clinical notes, MIMIC-III provides the granular data necessary to detect subtle patterns indicative of rare pathologies. Developing a computational framework that can interpret this unstructured text and map it to rare disease codes is not merely a technical exercise but it is a vital step toward improving the completeness of rare disease coding, which in turn supports registry quality, prevalence estimation and the earlier recognition of low-prevalence conditions.

1.3. Rare Disease Classification

Rare diseases are classified by several major organizations. The World Health Organization (WHO) maintains the International Classification of Diseases (ICD), which is the standard for health recording and death certification globally [11]. However, the granularity required for rare diseases is often better captured by Orphanet’s ORPHAcode classification system, which specifically targets these conditions and aligns with other ontologies like OMIM and SNOMED-CT [5].

1.4. State-of-the-Art in Automated Coding

Recent literature has demonstrated the efficacy of deep learning in automated ICD coding, though challenges remain regarding rare subsets.

• Transformer Models: Aden et al. [12] achieved state-of-the-art results using pre-trained ClinicalBERT [13] combined with RNNs and LSTMs. However, their evaluation was primarily focused on the top-10 and top-50 most common codes, achieving high precision (0.87) for common diseases but leaving the performance on rare codes less explored.
• Ensemble Methods: Merchant et al. [14] proposed ensemble neural models leveraging BiLSTMs for unstructured text [15], demonstrating strong performance on the full dataset with a micro-AUC of 98.8%.
• Explainability: Douglas et al. [16] emphasized the importance of explainability in clinical models, utilizing RoBERTa encoders to extract clinical entities [17] from discharge summaries, aiming to make black box predictions more transparent to clinicians.

Despite these successes, a gap remains in effectively handling the extreme class imbalance presented by rare diseases [18], motivating the exploration of attention-based convolutional neural networks (CNNs) and potential graph-based extensions to capture the complex inter-dependencies between clinical entities.

1.5. Research Gap and Questions

Two specific gaps motivate this work. First, existing multi-modal ICD coding systems fuse imaging or laboratory time series with clinical text, whereas medication and microbiology events, which are the modalities most directly reflecting clinical action and therefore the most informative when the narrative is terse, have not been systematically evaluated for rare-code assignment. Second, prior studies report aggregate metrics without stratifying by label frequency, so whether the reported multi-modal gains reach the long tail at all is unknown. Accordingly this article addresses three research questions:

• RQ1. Does adding prescription and microbiology modalities to a per-label attention text encoder improve ICD-9 code assignment for rare codes, relative to a matched text-only model evaluated under an identical protocol?
• RQ2. Is any such improvement concentrated in the long tail, that is, does it persist when results are stratified by training frequency, including few-shot codes?
• RQ3. Can this be achieved at a parameter and inference cost below that of transformer-based coders of comparable accuracy?

1.6. Organization of the Article

The remainder of this article is organized as follows:

• Section 2 (Related Works) describes the recent advances in identification of rare diseases using various technology and methods, especially with machine learning and deep learning-based approaches, and closes with a structured comparison of prior systems.
• Section 3 (Methodology) details the proposed pipeline, including the dataset and rare-subset construction, data pre-processing, feature fusion of structured and unstructured MIMIC-III data, and the architecture of the attention-based CNN model used for multi-label classification.
• Section 4 (Evaluation) defines the evaluation framework utilized during the experiments to measure the efficiency of the model. This section discusses the loss function and evaluation metrics used in details.
• Section 5 (Experimental Design) describes the experimental setup, including hyperparameter tuning, the computational environment, and the specific metrics used to evaluate performance on imbalanced data (Precision@k, Macro-F1 vs. Micro-F1).
• Section 6 (Experimental Results and Comparisons) presents the quantitative results of the model, analyzing the discrepancy between Micro-AUC and Macro-F1 scores, and discussing the model’s behavior regarding over-fitting and validation loss, together with validity checks and a stratification of performance by label frequency.
• Section 7 (Results Analysis) discusses about the actual impact of the model components and multi-modal efficiency of the model, and presents attention-based explanations and two contrasting case studies.
• Section 8 (Conclusions and Future Work) summarizes the contributions, states the limitations and ethical considerations, and outlines future research directions, such as the integration of graph neural networks and ablation studies for enhanced explainability.

2. Related Works

2.1. Literature Review

The automated identification of rare diseases from EHRs represents a significant computational challenge, primarily due to the long-tail distribution of disease prevalence. While common conditions provide abundant training examples, rare diseases, which by definition affect fewer than 1 in 2000 individuals, suffer from extreme data scarcity. This literature review surveys the evolution of computational methods addressing this disparity, organized by the technology stack employed, ranging from traditional machine learning to contemporary deep learning and graph-based architectures.

2.2. Traditional Machine Learning Approaches

The majority of research in this domain utilizes supervised learning algorithms to classify patients based on structured and unstructured EHR data. Among these, Support Vector Machines (SVMs) and tree-based ensemble methods are the most frequently validated.

SVMs are particularly favored for their ability to handle the high dimensionality and skewed class distributions typical of rare disease datasets. Rigg et al. demonstrated that SVMs outperformed logistic regression in detecting undiagnosed rare oncology cases, achieving higher sensitivity at fixed positive predictive values [19]. Similarly, in the context of Stiff Person Syndrome, Park et al. utilized an SVM model integrated with Shapley Additive Explanation (SHAP) values to identify key discriminatory clinical features such as depression and hypothyroidism, achieving an AUC of 0.808 [20]. For Acute Hepatic Porphyria, Cohen et al. found that an SVM with a radial basis function kernel yielded the best cross-validation performance among tested models, successfully ranking potential undiagnosed patients for manual review [21]. In a comparative study on acromegaly, SVMs were also employed, though they were slightly outperformed by random forest algorithms in that specific instance [22].

Tree-based ensemble methods, including random forest (RF) and Gradient Boosting, are widely cited for their robustness and interpretability. Crisafulli et al. reported that the random forest algorithm achieved the highest performance (AUC 0.83) in identifying acromegaly predictors from claims databases, outperforming SVM and decision trees [22]. For Gaucher disease (GD), multiple studies have validated ensemble approaches. Revel-Vilk et al. trained decision tree and random forest models using the Gaucher Earlier Diagnosis Consensus (GED-C) score features, achieving high specificity (0.99) and effectively distinguishing GD type 1 patients from controls [23]. Tenenbaum et al. further demonstrated that ML models trained on longitudinal data could detect GD approximately three years earlier than standard clinical diagnosis [24]. Additionally, Garg et al. applied a bootstrap ensemble classifier to identify cardiac amyloidosis, achieving a high F1 score of 0.98 by leveraging features such as carpal tunnel syndrome and heart failure [25].

Naive Bayes classifiers have also been explored for their probabilistic interpretability. Ehsani-Moghaddam et al. developed a naive Bayes model to calculate the posterior probability of mucopolysaccharidosis type II (Hunter syndrome) in male patients, successfully filtering a population of over 500,000 down to 125 high-likelihood candidates [26].

While traditional ML remains prevalent, deep learning (DL) architectures are increasingly applied to capture complex temporal and non-linear patterns in EHRs. Rolando et al. and Kim et al. [27] benchmarked various models on the MIMIC-III database for general rare disease detection. Their study included long short-term memory (LSTM) networks, convolutional neural networks (CNNs), and transformer-based models (BERT). Interestingly, while DL models showed promise, their SVM implementation achieved the highest F-measure (92.7%) in that specific experimental setup [28]. Yang et al., in a methodological review, noted that while supervised deep learning is the most popular paradigm, it often offers only marginal improvements over traditional ML for many phenotypes, though it excels in handling unstructured clinical notes [29].

Given the scarcity of labeled training data for RDs, several approaches pivot from pure classification to similarity-based ranking and collaborative filtering. Jia et al. developed the Rare Disease Auxiliary Diagnosis (RDAD) system, which employs both phenotypic similarity methods and machine learning to map clinical phenotypes to diseases. Their evaluation showed that ML-enhanced models yielded higher recall (up to 95%) than similarity methods alone [30]. Faviez et al. proposed a pipeline for detecting NPHP1-related renal ciliopathies that incorporates a semantic similarity module to address phenotype dependence, combined with supervised classification, achieving a sensitivity of 86% [31].

Shen et al. explored the use of collaborative filtering, a technique common in recommendation systems, to stratify patients. By integrating EHR phenotypes with biomedical literature (mining abstracts from Semantic MEDLINE), they demonstrated that collaborative filtering could effectively recommend rare disease diagnoses by identifying patients with similar phenotypic profiles, even across heterogeneous data sources [32].

The literature highlights critical methodological considerations for ML in this domain. A primary bottleneck is the lack of gold-standard confirmed diagnoses. Colbaugh et al. addressed this by utilizing silver-standard labels derived from provisional evidence [33]. Similarly, Wilson et al. developed age-based and prevalence-based algorithms to capture the heterogeneity of Gaucher disease presentations, noting that combining these strategies significantly improved the identification of undiagnosed patients compared to standard clinical diagnostic algorithms [34]. García-García et al. emphasized the utility of treating hospitalization episodes as data vectors to manage high dimensionality in the absence of large-scale labeled data for cardiac amyloidosis [35].

The application of machine learning to EHRs for rare disease identification has evolved from simple rule-based algorithms to complex ensemble and deep learning architectures. While deep learning models like BERT and LSTMs are being explored [28], robust supervised methods such as SVMs [19,20] and random forests [22] currently dominate the landscape due to their performance on small, imbalanced datasets. Future progress relies on improving phenotype extraction through semantic similarity [31] and integrating diverse data sources to overcome the inherent limitations of rare disease research [36].

2.3. Deep Learning

The identification of rare diseases (RDs) within EHRs presents a unique diagnostic odyssey due to low disease prevalence, heterogeneous clinical presentations, and the high dimensionality of longitudinal data [28,37,38]. Deep learning (DL) has emerged as a transformative paradigm to overcome these barriers by integrating multimodal data sources and modeling complex temporal patterns [39,40,41].

While EHRs are often associated with structured codes, convolutional neural networks (CNNs) are frequently employed to analyze the high-fidelity imaging data stored within these records [39]. The Convolutional Attention for Multi-Label classification (CAML) framework established a strong baseline by applying CNNs to discharge summaries, allowing the model to focus on specific text segments relevant to each ICD code [42]. While effective for common codes, standard CNNs often struggle with the class imbalance inherent to rare diseases. Recent studies demonstrate that multimodal fusion of CNN-processed radiographic images with tabular EHR data significantly improves diagnostic sensitivity for rare conditions like atypical femur fractures, increasing the area under the ROC curve (AUC) from 0.966 to 0.987 [43]. CNN models have also been applied to text-based features to detect Noonan syndrome, achieving high specificity and outperforming previous rule-based methods in prioritizing patients for genetic referrals [44,45]. Lee et al. in a scoping review of 332 articles found that CNNs were the most utilized architecture (307/332), primarily due to the availability of medical imaging for rare neoplastic and genetic diseases [39].

Recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) and gated recurrent units (GRUs), are the primary choice for modeling the longitudinal and temporal nature of patient histories in EHRs [46,47]. RNN-based models utilize historical diagnosis codes and clinical events to predict future diagnoses, often incorporating medical context attention (MCA) to account for patient demographics and phenotypic differences [46]. In the context of specific rare disease, like, sickle cell disorder, various RNN architectures like Jordan and Elman networks have been compared, with Jordan networks demonstrating superior performance in classifying medication requirements and clinical data sets [48]. RNNs excel in handling the patient sequence data typical of 7-year medical claims studies, outperforming standard benchmarks in detecting conditions like exocrine pancreatic insufficiency (EPI) [49,50]. To address sequential dependencies, models like CRNNet [51] introduced copy recurrent neural network structures. This approach specifically targets complicating diseases, often rare or secondary conditions, by employing a copy mechanism that mimics how clinicians infer complications from primary diagnoses, achieving a 57.30% improvement in detecting these subtle signals compared to standard RNNs.

To combat the extreme data imbalance inherent in rare disease research, researchers utilize hybrid frameworks that incorporate generative adversarial networks (GANs) and advanced optimization [49,52,53]. Models like CONAN use GANs to generate synthetic positive samples from uncertain patients, employing a max-margin classifier to improve detection in low-prevalence tasks like idiopathic pulmonary fibrosis [52]. The MLGAN framework acts as a meta-learning adapter that can enhance any existing diagnosis prediction method by generating synthetic embeddings to mitigate label noise and low prevalence [53,54]. Advanced architectures now amalgamate CNNs (for images), RNNs (for sequential data), and dense neural networks (for feature integration) to create cohesive precision medicine frameworks [55]. Recent advancements have introduced privacy-preserving federated learning to allow multi-institutional diagnosis of rare diseases while keeping heterogeneous EHR data secure [56].

While neural networks often provide superior accuracy, some studies noted that conventional models like random forest can occasionally outperform RNNs in specific clinical dosage classification tasks depending on the dataset characteristics [47].

Recent advancements have been dominated by transformer architectures, which leverage self-attention mechanisms to model long-range dependencies in EHRs. The shift toward natural language processing (NLP) in EHR analysis has popularized transformer-based models for extracting rare disease phenotypes from unstructured clinical notes [57,58]. Large-scale frameworks integrate Med-BERT and Transformer-XL to capture semantic and long-term temporal relations in longitudinal narratives, outperforming state-of-the-art multimodal baselines [41]. Researchers have combined Bidirectional Encoder Representations from Transformers (BERT) with medical ontologies (e.g., Orphanet, UMLS) to identify RDs from discharge summaries and radiology reports without requiring extensive manual annotation [58,59]. BioBERT, a domain-specific variant, has achieved an F1 score of 85.2% in recognizing rare diseases from medical texts, though challenges remain in identifying complex clinical manifestations [57].

The integration of large language models (LLMs) into the diagnostic workflow for rare diseases represents a burgeoning frontier in clinical informatics. Given that rare diseases affect over 300 million people worldwide and often involve a diagnostic odyssey lasting years, LLMs offer a promising mechanism to synthesize complex, heterogeneous clinical data from electronic health records (EHRs) [60,61,62]. Models such as ClinicalBERT [13] and BioBERT [63] have become standard encoders for medical text. Aden et al. [12] demonstrated that pre-trained ClinicalBERT, when combined with LSTM layers, achieves state-of-the-art performance on top-50 ICD codes. However, the performance drops significantly for rare codes, a phenomenon attributed to the lack of specialized vocabulary in the pre-training corpus. The Foresight model [64] utilizes a GPT-based architecture to forecast patient timelines. Unlike masked language models (e.g., BERT), Foresight generates future medical concepts (disorders, medications) sequentially, offering a probabilistic view of disease progression that is particularly valuable for anticipating rare complications. The RareAgents [65] framework proposed a multi-disciplinary team approach using autonomous LLM agents. By simulating a patient agent and an attending physician agent, this system iteratively refines diagnoses. The authors notably released the MIMIC-IV-Ext-Rare dataset, a subset specifically curated to benchmark performance on 4760 rare disease patients, highlighting the specific failure modes of general-purpose LLMs in this domain.

Recent research has transitioned from using general-purpose LLMs to developing specialized agentic systems that incorporate domain-specific knowledge and multi-step reasoning. DeepRare [61] is a notable agentic system that utilizes a central host with long-term memory and over 40 specialized tools to achieve high diagnostic accuracy across 2919 diseases. Similarly, RDguru [66] employs a DQN-based multi-source fusion model to integrate GPT-4 with phenotype matching and specialized recommendation strategies. RareSeek R1 [60] uses staged instruction tuning and graph-grounded retrieval to outperform traditional medical LLMs, particularly by identifying decisive non-phenotypic evidence such as imaging and functional tests. Systems like LA-MARRVEL [67] use LLMs as a language-aware re-ranking layer on top of traditional bioinformatics pipelines (like AI-MARRVEL), improving precision by leveraging expert-engineered context and consensus-based ranked voting.

To mitigate the limitations of zero-shot performance, researchers have developed sophisticated prompting techniques. The Models-Vote Prompting (MVP) [68] approach improves identification by prompting multiple LLMs to perform the same task and conducting a majority vote, which has been shown to outperform any single model in the ensemble. The integration of retrieval-augmented generation (RAG) with chain-of-thought (CoT) reasoning is a critical trend. RAG-driven CoT is effective for high-quality clinical notes, while CoT-driven RAG assists in processing noisy or lengthy narratives [69]. These frameworks often retrieve evidence from authoritative sources like Orphanet and OMIM [69,70]. Dynamic few-shot prompting, leveraging comprehensive rare disease knowledge graphs, has been shown to significantly enhance GPT-4’s diagnostic capabilities [62].

Evaluating LLM performance in rare disease contexts requires standardized benchmarks and comparison against traditional tools. Studies indicate that models like Claude 3.5 Sonnet can achieve diagnostic accuracies as high as 78.9%, significantly surpassing human physicians in specific case analyses [71]. The PhenoBrain [72] pipeline also demonstrates superior recall compared to both specialist physicians and general LLMs like ChatGPT. Despite progress, some benchmarks suggest that generalized LLMs have not yet reached the diagnostic accuracy of traditional tools like Exomiser when evaluated on large-scale phenopacket datasets [73]. The MIMIC-RD [74] benchmark highlights the gap between LLM performance on idealized case studies versus real-world clinical settings where ICD codes often undercount rare diseases.

Current research emphasizes the shift toward privacy-preserving and interpretable systems. Frameworks utilizing local models (e.g., Ollama Llama2) allow for diagnostic support while maintaining patient privacy through local inference [75]. Combining dictionary-based NLP tools (like SemEHR) with LLMs helps refine rare disease phenotyping from unstructured clinical reports, uncovering cases not documented in structured records [70]. Systematic reviews identify persistent hurdles, including hallucinations (e.g., fabricating patient data), potential biases, and the need for standardized evaluation metrics across the 22 different clinical purposes identified in the current literature [76,77].

2.4. Graph-Based Deep Learning

Given the hierarchical nature of medical ontologies (e.g., ICD-9, ICD-10, ORDO), graph neural networks (GNNs) have emerged as a powerful tool for rare disease detection. GNNs effectively model the relationships between diseases, allowing information from common conditions to propagate to rare ones via shared graph structures. The LabGraph framework [78] reformulates ICD coding as a graph generation problem rather than simple multi-label classification. By constructing a graph where nodes represent ICD codes and edges represent medical correlations, LabGraph utilizes adversarial domain adaptation to better localize target labels in the large search space, significantly outperforming baselines on Micro-F1 scores for rare codes. A recent work introduced a knowledge-guided multi-modal transformer that integrates Swin Transformers for imaging, Med-BERT for text, and GNNs for genomic data [41]. By grounding the learning process in the Orphanet Rare Disease Ontology (ORDO), this graph-based approach ensures that model predictions are medically consistent, even when training data is sparse. The ICDBigBird model [79] integrates graph convolutional networks (GCNs) with the BigBird transformer. This architecture addresses the limitation of standard transformers in handling long clinical documents while simultaneously using the GCN to create enriched code representations based on the ICD hierarchy, proving essential for classifying rare codes that rarely appear in the training data.

2.5. Addressing Data Scarcity: Few-Shot and Synthetic Learning

To directly tackle the lack of training samples for rare diseases, recent research has focused on data augmentation and synthetic data generation. The RareSyn framework [80] employs large language models to synthesize high-fidelity EHRs for rare diseases. By using seed EHRs and a retrieval-augmented generation approach, RareSyn creates diverse, privacy-preserving synthetic records that augment the training set, demonstrating clear improvements in downstream diagnostic models. To rigorously evaluate these methods, the MIMIC-RD benchmark [74] was developed by mapping clinical text entities directly to Orphanet codes. This study revealed that while state-of-the-art LLMs are capable, they still perform poorly on differential diagnosis for rare diseases in real-world settings compared to specialized graph-based methods.

In summary, the literature indicates a decisive shift towards neuro-symbolic and graph-based approaches. While transformers provide powerful text encoding, the integration of structured medical knowledge via GNNs appears critical for bridging the gap between common and rare disease detection.

2.6. Critical Comparison of Prior Work

Table 1. Comparison of representative prior work on rare-disease and long-tail ICD coding. “Tail reported” indicates whether performance on rare or low-frequency codes is reported separately from the aggregate.

Study	Dataset	Modalities	Architecture	Tail Reported	Limitation Addressed Here
CAML [42]	MIMIC-III full	Text only	CNN with per-label attention	Aggregate only	No structured clinical context
Aden et al. [12]	MIMIC-III top-10/50	Text only	ClinicalBERT + LSTM	No	Rare codes excluded by design
Merchant et al. [14]	MIMIC-III full	Text + structured	BiLSTM ensemble	Aggregate only	No frequency stratification
ICDBigBird [79]	MIMIC-III	Text + ICD hierarchy	BigBird + GCN	Partial	No structured EHR events
LabGraph [78]	MIMIC-III	Text + label graph	Graph generation	Yes	Text-only inputs
Abugabah et al. [41]	Institutional + ORDO	Imaging + text + genomic	Swin + Med-BERT + GNN	Yes	Not ICD coding; corpus not public
Schilcher et al. [43]	Radiographs + EHR	Imaging + tabular	CNN late fusion	Single disease	Single-label, one condition
RareAgents [65]	MIMIC-IV-Ext-Rare	Text only	LLM agent ensemble	Yes	Diagnosis, not code assignment
MMAC-Net (this work)	MIMIC-III full + rare subset	Text + prescriptions + microbiology	DWConv + SE + per-label attention, late fusion	Yes	—

summarizes the principal prior systems along the axes of dataset, modalities, architecture, whether long-tail or rare codes are explicitly evaluated, and the metrics reported. Read together, the literature exhibits four recurring weaknesses. First, most systems report only aggregate metrics such as Micro-F1 or Micro-AUC, which are dominated by high-frequency codes and therefore conceal performance on the tail that motivates the work. Second, label-space sizes and splits differ across publications, so reported numbers are frequently not comparable even when the corpus is nominally the same. Third, variance estimates are almost universally absent, so small differences between systems cannot be distinguished from noise. Fourth, multi-modal work in this area has concentrated on imaging and laboratory time series; medication and microbiology events, which record what clinicians actually did in response to a suspected condition, remain largely unexploited for code assignment.

3. Methodology

This section details the methodological framework employed in this study, encompassing the data acquisition from the MIMIC archives, the multi-modal pre-processing pipelines, the implementation of the multi-modal and multi-label attention-based convolutional network (MMAC-Net) classification architecture, and the training and evaluation protocols.

3.1. Dataset

The Medical Information Mart for Intensive Care (MIMIC) databases serve as the foundation for this research. A large, publicly accessible database called MIMIC-III [10] contains deidentified health information about more than 40,000 patients who were admitted to the Beth Israel Deaconess Medical Center’s critical care units between 2001 and 2012. Demographics, laboratory test results, procedures, prescriptions, caregiver notes, radiology reports, vital sign measures, and death (including post-hospital discharge) are all included in the database. While the preprocessing pipelines are designed to be adaptable, the current implementation primarily targets the MIMIC-III structure.

For the main experimentation, a rare disease subset from the original MIMIC-III dataset has been adopted utilizing the OrphaData [5]. Only disease codes that are rare, defined by OrphaCode and any associated codes in the admissions database, have been kept in the final dataset. For validation purposes, the original dataset split has been used to ensure proper comparison with state-of-the-art.

Table 2. Dataset description for the subset of admissions carrying at least one rare code and the whole dataset.

Metric	All	Rare
Admissions	58,976	39,304
Unique patients	46,520	31,515
Distinct ICD-9 codes	8930	568
Mean age	53.5	60.5
Median age	60.5	64.5
90th percentile age	81.9	82.7
Unique ICD-9 codes per admission (avg)	11.04	4.72

represents the comparative descriptors for the rare subset and the full dataset.

Construction of the rare-code subset. The construction proceeds as follows, and is released as a reproducible script so that the subset can be reconstructed exactly. The Orphanet nomenclature pack of release December 2025 supplies the ORPHAcode to ICD-10 alignment together with a relation qualifier for each mapping (exact mapping, narrower-term-to-broader-term, broader-term-to-narrower-term, or undetermined). Because the Orphanet pack does not distribute ICD-9 alignments, ICD-10 codes are mapped to ICD-9 through the General Equivalence Mappings (GEMs) crosswalk [81], and codes with no match are discarded. The complete list, with ORPHAcode, ICD-10 code, ICD-9 code and mapping relations, is provided in Supplementary Materials.

The rare subset comprises 39,304 of 58,976 admissions, and this proportion should not be read as a prevalence estimate. An admission is retained if at least one of its assigned codes maps to an ORPHAcode. Since an intensive-care admission carries 11.04 codes on average (Table 2), the probability that at least one of them falls within a set of 568 rare codes is far greater than the prevalence of any individual rare condition, and rare conditions in an intensive-care population almost always co-occur with common comorbidities rather than appearing in isolation. To avoid the confusion, the subset will be referred to throughout as the subset of admissions carrying at least one rare code rather than as a rare-disease population.

The split is admission-level and follows the benchmark split used by the systems in  [14,27,42,82], which is why it was adopted. A consequence is that a patient with several admissions may contribute to more than one split.

3.2. Dataset Modalities

To capture the complexity of rare disease code assignment, this study integrates multiple data modalities beyond unstructured text. These include the discharge summary notes, the pharmacological prescriptions and the microbiology events. As the data types are quite different, the proposed system is designed to adopt a multi-modal data model.

3.3. ICD Coding

The dataset utilizes the International Classification of Diseases, Ninth Revision (ICD-9), for coding diagnoses and procedures. The preprocessing pipeline extracts these codes from the diagnosis data table and procedures data table files. These codes serve as the ground-truth labels for the multi-label classification task. The system is configured to filter and identify rare ICD-9 codes, creating a subset of data, as shown in Figure 1, specifically focused on low-prevalence pathologies.

3.4. Modalities and Features

The methodology incorporates the following modalities, extracted and merged based on Hospital Admission Identifiers (HADM_ID):

• Clinical Notes: Extracted from NOTEEVENTS.csv, specifically filtering for discharge summaries. These free-text narratives provide the primary unstructured data source for the model.
• Prescriptions: Structured data loaded from PRESCRIPTIONS.csv. Key features extracted include drug_type, drug name, prod_strength (product strength), dose_val_rx (dosage value), and route of administration.
• Microbiology Events: Extracted from MICROBIOLOGYEVENTS.csv, capturing infectious disease data. Features include org_itemid (organism), ad_itemid (antibiotic), dilution_value, and interpretation (susceptibility).

3.5. Pre-Processing

The data pre-processing pipeline executes several critical steps, which prepare the raw CSV data for modeling. First, clinical notes undergo text cleaning via a clinical note processor, and notes associated with the same admission are concatenated to form a single document per admission ID from the dataset. Concurrently, structured data from prescriptions and microbiology events are cleaned, with missing values dropped, and categorical variables encoded using a label encoder to transform string identifiers into numerical indices. The pipeline shown in Figure 2 merges these modalities with the filtered ICD codes, ensuring that only admissions containing relevant rare codes are retained if configured. Finally, the dataset is tokenized using a configurable tokenizer and split into training, validation, and testing sets.

• Section-header handling. Section headers are detected using the header lexicon. Headers are retained by default.
• Normalization. All text is lower-cased.
• Noise reduction. Punctuation and numeric characters are removed.
• Stopword filtering. The NLTK English stopword list is applied.
• Lemmatization. The nltk.WordNetLemmatizer reduces words to their base form.
• Tokenization and truncation. A SpaceTokenizer is applied and documents are truncated or padded to 1500 tokens, truncating from the end of the document.
• Vocabulary. The vocabulary is constructed with a minimum token frequency of 3 on the training split only.

An admission may carry many prescription and microbiology records. All records belonging to an admission are aggregated into a single fixed-width feature vector, so that one admission remains one sample with one multi-hot label vector: categorical fields are represented as indicator vectors over the training-fitted vocabulary, and numerical fields are summarized by their per-admission statistics.

All label vocabularies, categorical encoders and numerical scalers are fitted on the training split alone and applied unchanged to the validation and test splits; categories unseen during training are mapped to a reserved index.

3.6. Feature Conversion

The processed data has been converted into tensor formats suitable for deep learning.

• Text Embeddings: Clinical notes are truncated to a maximum sequence length and converted into indices based on a vocabulary loaded from pre-trained Word2Vec embeddings.
• Categorical Features: Auxiliary categorical features (e.g., drug types, organism) are mapped to integer indices, handling unknown values by assigning them to a specific index.
• Numerical Features: Numerical attributes are normalized using Z-score standardization (subtracting the mean and dividing by the standard deviation) to ensure numerical stability during training. Two sample numerical attributes are the prescription dosage value and the microbiology dilution value, both of which are heavy-tailed and contain implausible extreme values arising from unit inconsistencies and free-text entry. Z-score standardization was preferred to min-max scaling for this reason: a min-max range is determined entirely by the two extreme order statistics, so a single erroneous dosage compresses the whole bulk of the distribution towards zero, whereas standardization is driven by the mean and standard deviation and degrades more gracefully.

We use Word2Vec rather than a contextual transformer encoder because it is a deliberate design decision rather than an oversight. The binding constraint in this setting is the ratio of parameters to positive training examples: a ClinicalBERT-class encoder introduces on the order of ${10}^8$ parameters, while many rare codes have fewer than five positive examples, and its 512-token limit would require segmenting discharge summaries that the present encoder processes at 1500 tokens in a single pass. Retaining the embedding dimension and sequence length of the CAML lineage [42] also keeps the text branch comparable with the baselines rather than advantaging it with a larger encoder.

3.7. Design Rationale for Multimodal Fusion

Traditional deep learning models (standard CNNs and RNNs) and large language models (LLMs) are highly prone to overfitting when applied to the extreme class imbalance and data scarcity inherent to rare diseases. To counter this, we propose MMAC-Net by decoupling spatial filtering from channel-wise mixing. Depth-wise convolution drastically reduces the parameter count and computational overhead compared to the standard CNNs or heavy transformer models. This is intended to make MMAC-Net parameter-efficient and less susceptible to overfitting on small rare disease dataset fragments; the parameter and latency measurements supporting the efficiency claim are reported in Section 5.5.

While LLMs (like BioBERT) and RNNs capture sequential data, they can be overwhelmed by long, heterogeneous clinical documents, often missing the “needle in the haystack” signals of a rare disease. In MMAC-Net, this is countered by using Per-Label Attention Mechanism, combined with the squeeze-and-excitation (SE) blocks, which acts as a highly targeted filter. Instead of creating a single, diluted document vector, the model dynamically recalibrates channel weights via SE and computes a unique document vector for every single label. This allows the model to strategically extract exactly which n-gram patterns correlate to a rare disease code, ignoring the surrounding clinical noise.

The literature points out that relying solely on text (NLP) or solely on structured data (traditional ML-like SVMs) leaves performance on the table. In MMAC-Net, the dedicated Tabular Fusion Branch explicitly addresses this by running in parallel to the convolutional text encoder. By embedding categorical features and normalizing numerical EHR data, and then fusing them via an MLP with the text representations, MMAC-Net captures a holistic patient phenotype. It combines the structured precision favored by traditional ML with the deep, unstructured pattern recognition of CNNs.

3.8. Model Architecture

The core architecture employed is the multi-modal and multi-label attention-based convolutional neural network (MMAC-Net) classification model. This architecture depicted in Figure 3 is designed to handle long clinical documents by learning to focus on specific parts of the text relevant to each label.

The model consists of several integrated components:

• Embeddings: A learnable embedding layer initialized with pre-trained Word2Vec weights converts token indices into dense vectors. A dropout layer is applied immediately after embedding to prevent overfitting.
• Convolutional Encoder: A 1D convolutional layer iterates over the document to capture local n-gram patterns. The architecture supports variable kernel sizes and depths, using Xavier uniform initialization for weights.
• Per-Label Attention Mechanism: The model employs a specialized attention mechanism where a matrix U projects the convolutional outputs to the output space of size equal to the number of classes. A softmax function is applied to generate attention weights $\alpha$, which are used to compute a weighted sum of the document representation, creating a unique document vector for each label.
• Depthwise Convolution Layers: The transition from standard 1D convolutions to depthwise separable convolutions (DWConv) in the deeper blocks is a design choice intended for balancing high-capacity feature extraction with parameter efficiency. By decoupling the sequential spatial filtering from the channel-wise feature mixing, depthwise convolutions drastically reduce both the computational overhead and the risk of overfitting.
• Squeeze and Excitation: The squeeze-and-excitation (SE) blocks integrated into the convolutional pipeline serve as a dynamic channel-wise attention mechanism, fundamentally enhancing the representational power of the MMACNet architecture. Rather than treating all extracted feature maps equally, the SE block explicitly models the inter-dependencies between channels to perform adaptive feature re-calibration.
• Tabular Fusion Branch: A distinct branch processes structured data. Categorical features are passed through separate embedding layers, while numerical features undergo batch normalization. These representations are concatenated and processed by a multi-layer perceptron (MLP) before being fused with the text-based representations.
• Output Classifier: The final classification is performed by a linear layer that maps the fused document-label representations to logits, followed by a sigmoid activation depending on the loss configuration.

Forward pass. Let B denote the batch size, L the sequence length, $d_e$ the embedding dimension, C the channel width and $\left|\mathcal{Y}\right|$ the number of labels. The token indices $x\in {\mathbb{N}}^{B\times L}$ are embedded to ${\mathbb{R}}^{B\times L\times d_e}$ and passed through the stem convolution to ${\mathbf{h}}_0\in {\mathbb{R}}^{B\times C\times L}$. Six residual depthwise blocks follow, each of the form given in Equation (2), where the squeeze-and-excitation operator is defined in Equation (1). Per-label attention then projects ${\mathbf{h}}_6$ through the matrix $U\in {\mathbb{R}}^{C\times \left|\mathcal{Y}\right|}$ to obtain attention weights $\alpha \in {\mathbb{R}}^{B\times L\times \left|\mathcal{Y}\right|}$ and the label-specific document representation $M\in {\mathbb{R}}^{B\times C\times \left|\mathcal{Y}\right|}$. In parallel, the structured features are embedded and passed through the tabular MLP to a vector in ${\mathbb{R}}^{B\times d_t}$, which is broadcast across the label axis and concatenated with M to give ${\mathbb{R}}^{B\times (C+d_t)\times \left|\mathcal{Y}\right|}$. Two fully connected layers then reduce this to the logit vector $\widehat{z}\in {\mathbb{R}}^{B\times \left|\mathcal{Y}\right|}$. Each symbol of Figure 3 legend is defined in

Table 3. Operators of the MMAC-Net computational graph shown in Figure 3.

Symbol	Operation	Shape (in → out)
Conv1D	Stem convolution over embedded tokens	$B\times L\times d_e\to B\times C\times L$
DWConv	Depthwise separable convolution	$B\times C\times L\to B\times C\times L$
BN	Batch normalization over channels	unchanged
SE	Channel gate, Equation (1)	$B\times C\times L\to B\times C\times L$
⊕	Residual addition, Equation (2)	unchanged
⊗	Channel-wise or attention product	unchanged
U, $\alpha$	Per-label attention projection and softmax	$B\times C\times L\to B\times L\times \left|\mathcal{Y}\right|$
M	Label-specific document representation	$\to B\times C\times \left|\mathcal{Y}\right|$
Tabular MLP	Structured-feature encoder	$B\times d_s\to B\times d_t$
Concat	Fusion across the label axis	$\to B\times (C+d_t)\times \left|\mathcal{Y}\right|$
FC 0, FC 1	Output classifier	$\to B\times \left|\mathcal{Y}\right|$

, and the corresponding pseudocode is given in Appendix A.

$$\mathrm{SE}\left(\mathbf{h}\right)=\sigma \left(W_2\mathrm{ReLU}\left(W_1\mathrm{GAP}\left(\mathbf{h}\right)\right)\right)\odot \mathbf{h}$$

(1)

$${\mathbf{h}}_{k+1}=\mathrm{ReLU}\left({\mathbf{h}}_k+\mathrm{SE}\left(\mathrm{BN}\left(\mathrm{DWConv}\left({\mathbf{h}}_k\right)\right)\right)\right)$$

(2)

Here, $\mathrm{GAP}$ denotes global average pooling over the sequence axis, which produces the per-channel descriptor that the two projections $W_1$ and $W_2$ turn into a channel gate; ⊙ is the channel-wise product drawn as ⊗ in Figure 3; and the addition in Equation (2) is the residual connection drawn as ⊕. The purpose of the residual path is to keep the six-block stack trainable, and the purpose of the SE gate is to let the network suppress channels that carry no evidence for the label under consideration.

3.9. Model Training

The training process is governed by the a config YAML file. The pipeline utilizes a standard iterative approach:

1.

Initialization: The model optimizer deployed was Adam, and loss function are initialized based on the provided configuration.

2.

Training Loop: The model iterates through the training dataset in batches. For each batch, gradients are computed via backpropagation, and the optimizer updates model parameters.

3.

Checkpointing: The checkpointing scheme monitors performance and saves the model state at regular intervals or when a best metric ($precision@8$) value is achieved.

4.

Regularization: A label-description regularization loss is added to the objective function, enforcing similarity between the learned attention vectors and the embeddings of the ICD code descriptions.

Data partitioning is handled during the preprocessing stage, creating distinct JSON files for training, validation, and testing. The data split is kept as the originally provided dataset as found in the MIMIC III original repository to ensure comparable results with the state-of-the-art works. The trainer loads these splits into PyTorch DataLoader instances. The training loader is shuffled to ensure stochasticity, while the validation and test loaders are sequential to ensure consistent evaluation.

4. Evaluation

4.1. Loss Function

The task is multi-label, each admission carries a set of ICD codes, represented as a multi-hot vector $y\in {\{0,1\}}^{\left|\mathcal{Y}\right|}$, and the objective used for the reported experiments is therefore the binary cross-entropy of Equation (4), applied independently to each label with a sigmoid activation.

The total loss ${\mathcal{L}}_{total}$ is defined in Equation (3):

$${\mathcal{L}}_{total}={\mathcal{L}}_{BCE}+\lambda {\mathcal{L}}_{reg}$$

(3)

The total binary cross-entropy component, ${\mathcal{L}}_{BCE}$, sums the loss over all N samples and L labels, as shown in Equation (4):

$${\mathcal{L}}_{BCE}=-{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{l=1}^L}\left[y_{i,l}log\left(\sigma \left({\widehat{z}}_{i,l}\right)\right)+(1-y_{i,l})log(1-\sigma \left({\widehat{z}}_{i,l}\right))\right]$$

(4)

and the regularization term ${\mathcal{L}}_{reg}$, is detailed in Equation (5):

$${\mathcal{L}}_{reg}={\displaystyle \frac{1}{L_{batch}}}{\displaystyle \sum _{l=1}^{L_{batch}}}\left|\right|{\mathbf{w}}_l-{\mathbf{v}}_{desc,l}{\left|\right|}_2^2$$

(5)

where

• N is the batch size.
• L is the total number of ICD classes.
• $y_{i,l}\in \{0,1\}$ is the ground truth binary label for class l and patient i.
• ${\widehat{z}}_{i,l}$ is the raw output logit from the model.
• $\sigma (·)$ is the sigmoid activation function.
• $\lambda$ is the regularization coefficient (config parameter lmbda).
• ${\mathbf{w}}_l$ is the learned embedding vector for label l.
• ${\mathbf{v}}_{desc,l}$ is the fixed description embedding of label l, obtained by averaging the Word2Vec vectors of the words of its ICD-9 description.

Two mechanisms in the present design address the long tail, and it is worth stating them explicitly because the objective itself applies no reweighting. The per-label attention mechanism computes a separate document vector for every code, so a rare code is scored from the evidence that attends to it rather than from a single pooled representation dominated by common conditions; and the description regulariser ${\mathcal{L}}_{reg}$ ties each label vector to the embedding of its ICD description, transferring prior semantic knowledge to codes with few positive examples. These embeddings are generated by averaging the Word2Vec vectors of the words constituting the ICD-9 code description, ensuring that the label attention mechanism aligns with the semantic meaning of the diagnosis.

4.2. Evaluation Metrics

To rigorously assess model performance, particularly on the imbalanced rare disease subset, the following metrics are computed:

• Precision@k (P@k): The proportion of relevant labels in the top-k predictions, that is, the number of correctly ranked labels within the top k divided by k. This is crucial for clinical decision support, where a physician typically reviews only the top few suggestions.

  $$P@k={\displaystyle \frac{1}{k}}{\displaystyle \sum _{i=1}^k}\mathbb{I}({\widehat{y}}_i\in Y)$$

  (6)
• Micro-F1 Score: The harmonic mean of precision and recall calculated globally by counting the total true positives, false negatives, and false positives. This metric biases towards common disease classes.

  $$\text{Micro-}\mathrm{P}={\displaystyle \frac{{\sum }_{c=1}^{C}T{P}_c}{{\sum }_{c=1}^C(T{P}_c+F{P}_c)}}$$

  (7)

  $$\text{Micro-}\mathrm{R}={\displaystyle \frac{{\sum }_{c=1}^{C}T{P}_c}{{\sum }_{c=1}^C(T{P}_c+F{N}_c)}}$$

  (8)

  $$\text{Micro-}\mathrm{F}1=2\times {\displaystyle \frac{\text{Micro-}\mathrm{P}\times \text{Micro-}\mathrm{R}}{\text{Micro-}\mathrm{P}+\text{Micro-}\mathrm{R}}}$$

  (9)
• Macro-F1 Score: The unweighted mean of the F1 scores calculated for each label individually. This metric is particularly important for this study as it treats rare diseases equally to common ones, highlighting the model’s performance on the long tail of the distribution.

  $$\text{Macro-}\mathrm{F}1={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}F{1}_c$$

  (10)

  where $F{1}_c$ is the F1 score for class c.
• Macro-AUC is the arithmetic mean of the area under the ROC curve (AUC) calculated for each class individually.

  $$\text{Macro-AUC}={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C}AU{C}_c$$

  (11)
• Micro-AUC is the area under the receiver operating characteristic curve calculated using the global true positive rate ($TP{R}_{micro}$) and false positive rate ($FP{R}_{micro}$), defined as

  $$TP{R}_{micro}={\displaystyle \frac{{\sum }_{c=1}^{C}T{P}_c}{{\sum }_{c=1}^C(T{P}_c+F{N}_c)}}$$

  (12)

  $$FP{R}_{micro}={\displaystyle \frac{{\sum }_{c=1}^{C}F{P}_c}{{\sum }_{c=1}^C(F{P}_c+T{N}_c)}}$$

  (13)

Micro-AUC pools all label-instance pairs. With 8930 labels and 11.04 positive labels per admission, the positive rate is approximately 0.12%, so the statistic is dominated by the ease of ranking overwhelmingly many trivial negatives and values above 0.98 are routine on this benchmark.

The values $k=8$ and $k=15$ were fixed before experimentation and follow the convention established for the full-label MIMIC-III setting by Mullenbach et al. [42] and adopted by the systems compared.

The choice is also grounded in the data, since the full dataset averages 11.04 codes per admission, so that $k=8$ falls just below the mean label cardinality and $k=15$ just above it.

Decision thresholds for the F1 metrics are selected on the validation split only and applied unchanged to the test split. Because macro-averages in a label space this sparse are sensitive to the treatment of labels for which no positive prediction is made, we state the convention used: such labels are assigned an F1 of zero and are included in the average.

5. Experimental Design

To evaluate the proposed multi-modal rare disease detection system, a robust experimental framework has been established. The design is grounded in a rigorous configuration of data preprocessing pipelines, model hyper-parameters tuning, and training protocols, ensuring reproducibility and statistical validity. The experiments utilize the MIMIC-III critical care database, specifically configured to handle the challenges of high-dimensional multi-label classification.

5.1. Data Preparation and Preprocessing

The efficacy of deep learning models in the clinical domain is heavily contingent upon the quality of input data. As defined in the preprocessing configuration, a comprehensive pipeline was implemented to transform raw clinical logs into structured feature vectors.

The experimental dataset consists of five primary data modalities extracted from the MIMIC-III database:

• Clinical Narratives: Discharge summaries extracted from NOTEEVENTS.csv.gz.
• Diagnostic and Procedural Codes: ICD-9 codes derived from DIAGNOSES_ICD.csv.gz and PROCEDURES_ICD.csv.gz.
• Pharmacological Data: Medication records from PRESCRIPTIONS.csv.gz, including drug types, dosages, and administration routes.
• Microbiology: Infectious disease data from MICROBIOLOGYEVENTS.csv.gz, including organism identifiers and antibiotic susceptibility.

Following text normalization, a SpaceTokenizer is utilized, and word embeddings are generated using the Word2Vec algorithm. The embedding layer is configured with a vector size of 100 and a minimum word count frequency of three, and is trained with five epochs to capture semantic relationships within the corpus.

5.2. Hyperparameter Tuning

The architecture employed is tuned with several hyperparameters to ensure the best possible results. The specific architectural hyperparameters are detailed in

Table 4. Hyperparameters for MMACNet.

Parameter	Value
Embedding dimension	100
Sequence length (L)	1500 tokens
Kernel size	100
Number of filter maps	128
Convolutional block depth	6
Objective function	BCE
Regularization coefficient $\lambda$	0.25
Embedding dropout	0.6
Fully connected dropout	0.3
Batch size	512
Activation function	ReLU
Batch normalization	True
Target classes—rare subset ($N_{rare}$)	568
Target classes—full dataset ($N_{full}$)	8930
Tabular Fusion
Fusion strategy	Late Fusion
Tabular hidden dimension	50

.

Unlike standard shallow CNNs, this experiment utilizes a deep convolutional stack with a depth of 6 and batch normalization enabled. This allows the model to learn more complex hierarchical patterns in the clinical text. The model is configured to classify 568 distinct labels for the rare disease subset, representing only the rare specific subset of conditions as per ICD-9 definitions. Furthermore, for the sake of comparability, a separate experimentation has also been designed to automatically detect all unique ICD (i.e., 8930 after data preprocessing) codes represented in the MIMIC-III dataset.

To mitigate overfitting, substantial regularization is applied. A dropout rate of $0.6$ is applied to the embedding layer, while the fully connected layers utilize a dropout rate of $0.3$. The fully connected layers are structured with dimensions $[1024,568]$, ensuring a gradual projection of features to the output space.

The configuration values were selected by validation Precision@8 over the exploratory runs we conducted, in which convolutional depth was varied over 2–12 and embedding dropout over 0.2–0.8; the depth of 6 was retained as the deepest stack that did not degrade validation Precision@8. No test-split data were used at any point in model selection.

We state plainly that this is not an exhaustive search and that the configuration is therefore not claimed to be optimal, but a single full-label training run over 8930 classes costs approximately 27 GPU-hours on the hardware described in Section 5.5, so a cross-validated search over even a modest grid would require several hundred GPU-hours and would additionally break comparability with the fixed benchmark split. We therefore treat a full sensitivity study as a separate study in the future work presented in Section 8.3.

5.3. Training Protocol

The training pipeline is optimized for the high-dimensional, multi-label classification of International Classification of Diseases (ICD) codes from multimodal electronic health records. The network’s parameters are iteratively updated by minimizing a composite objective function. This loss unifies a primary multi-label classification criterion, namely the binary cross-entropy of Equation (4), with a description-based regularization term $\lambda {\mathcal{L}}_{reg}$ that enforces semantic alignment between the learned label weights and their textual definitions. To ensure stable convergence and mitigate overfitting on the highly imbalanced MIMIC-III dataset, the optimization routine is heavily supported by residual skip connections, layer-wise batch normalization, and strategic dropout across the feature refinement layers.

The model parameters are optimized using the Adam optimizer with a learning rate of $1\times {10}^{-3}$. To facilitate convergence and escape local minima which is a common issue in high-dimensional non-convex optimization, a Cosine Annealing learning rate scheduler is employed with $T_{max}$ set to 200 epochs.

The objective function is the binary cross-entropy of Equation (4), evaluated against the multi-hot label vector. The training is performed with a batch size of 512, utilizing 16 worker threads for efficient data loading. To ensure the stability of the training loop, the incomplete batches were discarded at the end of an epoch.

To prevent overfitting and minimize computational waste, an early stopping mechanism is implemented. The monitoring metric is prec_at_8 (precision at 8). Training is halted if there is no improvement in the validation Precision@8 for 15 consecutive epochs (patience). The best model is explicitly saved based on the maximization of the Precision@8 metric.

5.4. Evaluation Framework

The experimental results are evaluated using a suite of metrics standard in multi-label clinical classification. As specified in the evaluation configuration, the primary metrics include the following:

• Precision@k: Specifically $k=8$ and $k=15$. This metric is prioritized as the stopping criterion, reflecting the clinical need for accurate top-ranked suggestions.
• F1 Scores: Both Macro-F1 and Micro-F1 are calculated to assess performance across rare (macro) and frequent (micro) classes.
• AUC Scores: Macro-AUC and Micro-AUC provide an aggregate measure of classification thresholds.

All experiments are conducted with five random seeds and GPU acceleration enabled to ensure deterministic and efficient execution.

5.5. Computational Environment and Cost

Training and inference were carried out on one NVIDIA H100 80 GB GPU, with 2× Intel(R) Xeon(R) Gold 6442Y CPU clocked at 2.60 GHz and 128 GB of system memory, under CUDA version 12.6.2 and PyTorch 2.1.2.

Table 5. Computational cost of MMAC-Net. Baseline figures, where available, are those reported by the corresponding publications and are marked accordingly. Here, M = millions, GB = gigabytes, H = hours.

Model	Source	Parameter	Peak Memory	Peak GPU Memory	Train Time	Modality
MMAC-Net (full label space)	measured	15.5 M	102.91 GB	57 GB	27 H	Multi-modal
CAML [42]	reported	6.2 M	—	—	—	Text
MultiResCNN [82]	reported	11.9 M	—	—	—	Text

reports the quantities that substantiate the parameter-efficiency claim made in Section 3.

6. Experimental Results and Comparison

6.1. Ablation Study

To evaluate the contribution of distinct data sources to the automated ICD coding task, we conducted an extensive ablation study. The performance of the proposed architecture was assessed across different combinations of unstructured clinical text (Notes), structured numerical data (Tabular), and structured categorical data (Categorical). The results, summarized in

Table 6. Comparison among different combinations of data modalities in identifying rare subset (where at least one of the co-occuring codes is rare as defined by OrphaNet) of ICD codes. Values are rounded to three decimal places.

Data Modalities	Precision		F1		AUC
	p@8	p@15	Macro	Micro	Macro	Micro
Notes + Tabular	0.096 ± 0.001	0.073 ± 0.001	0.029 ± 0.001	0.381 ± 0.002	0.811 ± 0.001	0.970 ± 0.003
Notes + Categorical	0.095 ± 0.002	0.087 ± 0.003	0.020 ± 0.001	0.381 ± 0.004	0.808 ± 0.004	0.971 ± 0.002
Notes + Tabular + Categorical	0.159 ± 0.003	0.163 ± 0.002	0.084 ± 0.001	0.513 ± 0.003	0.878 ± 0.006	0.985 ± 0.009
Notes (Baseline)	0.092 ± 0.001	0.053 ± 0.001	0.011 ± 0.001	0.368 ± 0.001	0.767 ± 0.001	0.966 ± 0.001

Note: Values boldfaced indicate the highest while second highest in their categories are underlined.

, demonstrate the efficacy of the multi-modal fusion strategy implemented in the MMACNet architecture. The best values are depicted using bold font and second bests are represented by underlining.

6.2. Comparison with State-of-the-Art

Table 7. Model performance on MIMIC-III (full dataset). All baseline figures are those reported by the corresponding publications.

Model	# of Labels	Macro-AUC	Micro-AUC	Macro-F1	Micro-F1	P@8
CNN [42]	8922	0.835	0.974	0.034	0.420	0.619
CAML [42]	8922	0.893	0.985	0.056	0.506	0.704
MultiResCNN [82]	8930	0.910 ± 0.002	0.986 ± 0.001	0.085 ± 0.007	0.552 ± 0.005	0.734 ± 0.002
DCAN [27]	8930	0.848 ± 0.009	0.979 ± 0.001	0.066 ± 0.005	0.533 ± 0.006	0.721 ± 0.001
TransICD [27]	8930	0.886 ± 0.010	0.983 ± 0.002	0.058 ± 0.001	0.497 ± 0.001	0.666 ± 0.000
Fusion [27]	8930	0.910 ± 0.003	0.986 ± 0.000	0.081 ± 0.002	0.560 ± 0.003	0.744 ± 0.002
Ensemble (BiLSTM + FCN) [14]	8930	0.922	0.988	0.120	0.571	0.739
MMAC-Net (proposed)	8930	0.981 ± 0.002	0.997 ± 0.003	0.641 ± 0.003	0.724 ± 0.005	0.875 ± 0.003

presents a comparative analysis of the proposed model against prominent state-of-the-art architectures on the full MIMIC-III dataset comprising all of the 8930 ICD codes. Specifically, the model achieves a Micro-AUC of 0.997, which is higher than both the CAML baseline and advanced fusion models (0.986). The proposed model attains a Precision@8 of 0.875, against the foundational CAML architecture (0.704) and the sequence model DCAN (0.721). Furthermore, the Macro-F1 score of 0.641 is higher than the PLM-CA implementation (0.261).

6.3. Comparative Analysis Between Rare Subset vs. The Whole Dataset

The comparative analysis of the loss trajectories in Figure 4 elucidates a critical dichotomy in model performance between the full dataset and the rare disease subset. The full dataset curves in Figure 4a,b demonstrate a robust learning process, characterized by a smooth, monotonically decreasing validation loss that closely tracks the training loss, indicating effective generalization and stable convergence. In stark contrast, the rare subset evaluation in Figure 4c,d reveals severe instability and stochastic behavior; the validation loss in Figure 4c exhibits high-variance oscillations without a clear downward trend, fluctuating significantly between $3.5$ and $4.0$. Furthermore, Figure 4d highlights a substantial generalization gap, where the training loss (blue) descends rapidly to approximately $1.5$, while the validation loss (red) remains stagnantly high. This divergence is indicative of severe overfitting, a primary shortcoming attributable to the extreme sparsity of the rare disease samples.

7. Results Analysis

7.1. Impact of Multi-Modal Data Fusion

The experimental results indicate a clear hierarchy in performance, with multi-modal configurations consistently outperforming the unimodal text-only baseline.

• Baseline Performance (Notes Only): The model relying solely on unstructured discharge summaries yielded the lowest performance across all metrics, with a Macro-F1 of $0.0109$ and a Precision@8 of $0.0921$. This confirms that while clinical narratives contain rich information, they are often insufficient on their own to capture the full clinical picture required for accurate coding, particularly for rare conditions. This insufficiency occurs because physicians frequently omit objective, structured data, such as specific laboratory thresholds, vital signs, or demographic baselines from their narrative summaries, as these are already accessible elsewhere in the electronic health record (EHR). Furthermore, the high variance and noise inherent to free-text notes exacerbate the challenge of identifying rare diseases, which already suffer from a lack of training examples.
• Synergy of Structured Data: The integration of structured data significantly enhanced the model’s predictive capability. The combination of Notes and Tabular resulted in a substantial increase in Macro-AUC (0.8106 vs. 0.7670) and nearly tripled the Macro-F1 score (0.0295 vs. 0.0109). This suggests that numerical features, such as the normalized prescription dosage and microbiology dilution values processed by the tabular branch, provide critical signals that help disambiguate complex code assignments.
• Optimal Configuration: The fully integrated model (Notes, Tabular and Categorical) achieved the highest performance across the majority of key metrics. It reached a peak Micro-F1 of $0.5132$ and a Precision@8 of $0.163$. This configuration leverages the late fusion strategy, where categorical embeddings (e.g., medications, microbiology events) and numerical features are concatenated with the text representation before the final classification layer.

To bypass these limitations, the MMACNet architecture employs a multi-modal fusion strategy. By integrating categorical and tabular data alongside the text representations, the model effectively anchors the subjective clinical narrative with objective patient data. This holistic approach is supported by the improvement observed when all modalities are combined, improving the Macro-F1 score to 0.084 and the Micro-AUC to 0.985, which was the largest improvement among the configurations we evaluated on this dataset.

7.2. Effective Late Fusion Mechanism

The results validate the design choice of using a separate multi-layer perceptron (MLP) branch for structured data. By processing tabular and categorical features independently before fusing them with the attention-weighted text vectors, the model preserves the unique statistical properties of each modality. The Notes, Tabular, and Categorical modality achieved a Macro-F1 of $0.0845$, the highest among all experiments, indicating that this fusion strategy effectively addresses the class imbalance inherent in rare disease identification.

7.3. Behavior in High-Dimensional Classification

Despite the extreme sparsity of the label space (with rare diseases appearing infrequently), the inclusion of auxiliary data modalities was associated with more stable the model’s learning. The Macro-AUC remained high (≈0.88) for both tabular-enhanced models, above the text-only baseline of $0.77$. This is consistent with the interpretation that structured features provide a grounding effect, helping the model maintain discriminative power even when textual cues are ambiguous or sparse.

7.4. Explainability of Predictions

For each code ℓ, the attention distribution ${\alpha }_{\ell }$ produced by equation-level per-label attention is already a distribution over document positions. We extract these weights from the trained checkpoint, and present them in two forms. Figure 5 shows token-level attention heat maps over de-identified excerpts for representative rare codes, and the most highly weighted n-grams per code aggregated over the test split is available as a supplementary material titled rare_code_top_ngrams.csv.

7.5. Case Studies

To illustrate the inference, we present eight contrasting admissions in Figure 6 selected from the stored test-set predictions. The test cases are presented with their labels and top-3 predictions, attention heatmap, and type of note annotation.

8. Conclusions and Future Work

This research addressed the critical challenge of automated International Classification of Diseases (ICD) coding, with a specific focus on the identification of rare pathologies within EHRs. By leveraging the heterogeneous data modalities available in the MIMIC-III critical care database, we developed a multi-modal multi-attention based deep learning network (MMAC-Net) for multi-label classification. The study systematically investigated the hypothesis that integrating structured clinical variables, specifically medications and microbiology events with unstructured discharge summaries serves to ground the model’s predictions and mitigate the sparsity issues inherent in rare disease code assignment.

The primary contributions and achievements of this work are summarized as follows:

• Multi-Modal Fusion Architecture: We successfully engineered a late fusion neural architecture that processes unstructured clinical text via a deep convolutional attention mechanism while simultaneously encoding structured categorical and numerical features through parallel perceptron branches.
• Validation of Structured Features: Through a rigorous ablation study, we demonstrated that the inclusion of structured data significantly enhances predictive performance. The fully integrated model (Notes, Tabular and Categorical) achieved a Macro-F1 of $0.0845$, nearly tripling the performance of the text-only baseline ($0.0109$), thereby validating the importance of multi-modal signals in disambiguating complex diagnoses.
• Competitive Performance: The proposed framework achieved a Micro-AUC of $0.997$ and a Precision@8 of $0.875$ on the full MIMIC-III dataset. These results are higher than the figures reported for baselines such as CAML and DCAN, though as explained in Section 3 and in the caption of Table 7, those figures are transcribed from their source publications rather than produced under our protocol, so the comparison is indicative and does not establish a ranking.

8.1. Limitations

We state the principal limitation first, because it bounds everything that follows. On the subset of admissions carrying rare codes the model overfits: the training loss falls while the validation loss does not, as Figure 4 shows. The model is therefore prone to memorizing sparse rare examples rather than learning generalisable discriminative features, and the results below should be read as evidence about multi-modal fusion on one retrospective intensive-care corpus rather than as a demonstration of deployable performance.

Secondly, The evaluation uses a single corpus, so no claim of cross-institutional generalization is made; hospitals differ in documentation and coding practice, and validation on external data is required before transferability can be assumed. The corpus is intensive-care only, which over-represents acute presentations and distorts the apparent prevalence of rare conditions relative to general hospital or primary care populations. The recording period of 2001 to 2012 and the use of ICD-9 rather than ICD-10 or ICD-11 limit transferability to current coding practice, and documentation conventions and billing incentives are specific to the United States.

8.2. Ethical Considerations and Human-in-the-Loop Use

Automated coding of rare conditions carries an asymmetric cost of error. A missed rare code delays recognition of a condition that may already have taken years to identify, while a false positive can trigger unnecessary investigation, patient anxiety and cost. Neither error is acceptable without review, and this asymmetry is the reason we frame MMAC-Net as a decision-support component that produces a ranked shortlist of candidate codes with visible supporting evidence, rather than as a system that assigns codes autonomously. Presenting a shortlist together with the attention-based evidence of Section 7.4 is also a partial safeguard against automation bias, since it allows a coder to check the basis of a suggestion instead of accepting it.

A second consideration concerns the inheritance of bias. The model is trained on codes assigned by human coders for administrative purposes, and it therefore reproduces the documented tendency of such codes to under-record rare conditions, particularly in populations with poorer access to specialist care. A system trained on these labels will be least reliable precisely where recognition is already weakest, and deployment would require monitoring stratified by patient demographics rather than aggregate performance alone. Accountability for the final code assignment remains with the human coder or clinician; nothing in this article supports removing that oversight.

8.3. Future Work

The following work process addresses the limitations above as each item requires training experiments beyond the scope of the present study.

• Hyperparameter search with cross-validation. A grid or random search over depth, embedding dimension, kernel size and dropout under k-fold cross-validation, with sensitivity curves, to place the configuration of Table 4 on a systematic footing.
• Imbalance-aware objectives. Weighted binary cross-entropy, focal, asymmetric and distribution-balanced losses, compared on the rare subset.
• External validation. Evaluation on MIMIC-IV in zero-shot and fine-tuned settings, noting that MIMIC-IV originates from the same institution and therefore provides temporal rather than cross-site validation; genuinely multi-site data would be required for the latter.
• Patient-disjoint split. Retraining under a SUBJECT_ID-disjoint partition to quantify memorization across repeat admissions.
• Temporal ablation. Censoring prescriptions and microbiology events at successive cut-offs to quantify how much of the multi-modal gain derives from end-of-stay evidence.
• Prospective formulation. Defining a prediction time before diagnosis and excluding evidence recorded after it, including the discharge summary itself, to move from retrospective coding towards decision support during the admission.

8.4. Constraints Observed

Despite the overall efficacy of the multi-modal architecture, the model remains fundamentally constrained by the inherent data scarcity of low-prevalence diseases. This severe lack of training samples results in an insufficient signal-to-noise ratio for extremely rare codes, rendering standard supervised learning paradigms highly susceptible to severe overfitting, even with multi-modal augmentation. Qualitative analysis of the loss landscapes confirms this generalization gap; while the model converges smoothly on the full dataset, the validation loss specifically for the rare disease subset exhibits high-variance stochastic oscillations and fails to decrease in tandem with the training loss. Consequently, this indicates that the model is prone to memorizing sparse rare examples rather than learning robust, generalizable discriminative features.

To address these current limitations and advance the field of automated rare disease coding, future research must tackle data scarcity and model generalization through advanced architectural and dataset enhancements. Specifically, integrating graph neural networks (GNNs) with hierarchical medical ontologies (such as ICD-9 or Orphanet) could significantly improve generalization by transferring learned features from common to rare conditions. Furthermore, the persistent challenge of data scarcity can be directly mitigated by employing few-shot learning techniques or utilizing large language models (LLMs) to generate high-fidelity synthetic clinical records. Finally, transitioning the evaluation framework to the contemporary MIMIC-IV dataset would yield higher temporal resolution and a more modern case mix, providing the richer feature sets necessary for precise rare phenotype extraction.