HA-PI-MADT: A Hybrid Adaptive Multimodal Digital Twin-Inspired Framework for Reliable Healthcare Prediction with Improved Ranking and Calibration Performance

Abstract

The integration of heterogeneous healthcare data sources remains a major challenge in developing reliable and personalized predictive systems for digital healthcare applications. Traditional machine learning methods perform well on structured clinical data but often fail to effectively exploit multimodal information, while deep learning approaches may suffer from instability, weak generalization, and poor calibration when dealing with limited modalities. To address these limitations, this study proposes HA-PI-MADT, a hybrid adaptive healthcare-informed multimodal digital twin-inspired framework that combines deep multimodal representation learning with ensemble-based predictive modeling for robust and trustworthy healthcare prediction. The proposed framework integrates wearable sensor signals, electronic health records (EHRs), CT/MRI imaging representations, and population-level risk prototypes derived from the UCI diabetes dataset within a unified multimodal healthcare representation architecture. In addition, a modality-aware adaptive fusion mechanism dynamically adjusts the contribution of each modality according to its relevance and data quality, while a hybrid stacking strategy combines deep multimodal embeddings with classical ensemble learners to improve predictive robustness and ranking performance. To enhance clinical trustworthiness, calibration-aware optimization is incorporated to improve probabilistic reliability and uncertainty estimation. Extensive experiments conducted on a multimodal healthcare dataset demonstrate that HA-PI-MADT achieves a balanced performance profile across discrimination, ranking, and calibration-oriented evaluation metrics compared with several unimodal, multimodal, and ensemble baselines. The proposed framework achieves strong ranking-oriented and classification performance, including the highest AUPRC (0.6388) and F1-score (0.6327), while also demonstrating competitive calibration-oriented reliability through lower Brier score and negative log-likelihood values. The results demonstrate the effectiveness of the proposed hybrid adaptive multimodal digital twin-inspired framework for reliable, robust, and clinically trustworthy healthcare prediction.

1. Introduction

The growing availability of heterogeneous healthcare data provides new opportunities for the development of reliable, personalized, and data-driven clinical decision-support systems. The current healthcare environment produces a variety of information including wearable physiological signals, electronic health records (EHRs), medical images, demographic attributes, and population-level risk indicators. Nonetheless, mapping these heterogeneous sources into clinically meaningful predictions remains a challenge as healthcare data are often incomplete, noisy, weakly aligned, and collected under varying temporal, institutional, and technical conditions [1,2,3]. Therefore, multimodal machine learning has become an important direction in healthcare analytics, as it can combine complementary evidence from multiple sources for predictive models instead of a single modality [4,5,6]. Importantly, the multimodal data sources used in this study originate from independently collected heterogeneous datasets rather than fully synchronized patient-specific records. Therefore, the proposed framework should not be interpreted as a strict physiological digital twin that continuously mirrors the state of an individual patient. Instead, HA-PI-MADT adopts a digital twin-inspired multimodal learning strategy that integrates wearable physiological signals, demographic-level contextual prototypes from EHR data, imaging-derived representations, and population-level risk priors within a unified adaptive learning framework. This design enables robust multimodal healthcare representation learning under realistic multimodal integration constraints commonly encountered in healthcare environments.

Recent studies have shown that multimodal healthcare models can improve patient representation learning and clinical prediction by integrating structured clinical data with physiological signals, medical imaging, and other contextual information [2,3,7]. In particular, the combination of medical imaging and EHR data has gained increasing attention because imaging provides anatomical and visual information while EHR data provides structured clinical and demographic information [4]. Likewise, integrated multimodal healthcare AI frameworks have been demonstrated to be feasible for integrating multiple clinical data streams into unified predictive pipelines [7]. However, robust multimodal learning continues to be difficult in real-world healthcare environments, where modalities are often missing, partially paired, weakly synchronized or of different quality and predictive relevance [8,9,10].

Digital twin technologies are also attracting increasing attention in the fields of medicine and healthcare systems. Medical digital twins are often created as computational representations of the patient for monitoring, simulating, predicting, and making personalized decisions [11,12,13]. However, a strict physiological digital twin typically requires temporally synchronized and patient-specific continuous data streams. In real-world health care data sets, such perfect synchronization is often not available. Thus, digital twin-inspired healthcare models are a viable alternative by incorporating information at the patient level, demographic level, imaging-derived, and population level into a unified predictive representation, without claiming to reproduce a full real-time physiological replica of an individual patient [11,12]. This distinction is of importance in this study as the multimodal data sources are collected independently and have only weak alignment.

One main limitation of many existing multimodal healthcare prediction models is their reliance on static or naive early-fusion strategies. The simple concatenation of heterogeneous features may introduce modality noise, increase distributional inconsistency and decrease generalization when the integrated data sources are of different reliability levels [5,6]. Moreover, the missing-modality and weak-alignment problems are common in healthcare datasets, where not all patients have complete wearable, EHR, and imaging information [8,9,10]. These challenges motivate the need for adaptive fusion mechanisms that control dynamically the contribution of each modality depending on its relevance, quality and predictive usefulness.

Another important issue is the reliability of the predicted probabilities. In the context of clinical decision support, a model should not only output the correct class label but also deliver trustworthy probability estimates. Poorly calibrated models can generate over-confident predictions, which can be harmful when the model outputs are used to support clinical decisions [14,15]. Calibration-oriented metrics, e.g., Brier score, expected calibration error (ECE), and negative log-likelihood (NLL), provide complementary views of probabilistic reliability [15,16]. Moreover, ranking-oriented metrics such as the area under the precision–recall curve (AUPRC) are especially useful for imbalanced healthcare prediction tasks, where accuracy or AUROC alone may not adequately reflect clinical utility [17].

To address these challenges, in this study, we propose HA-PI-MADT, a hybrid Adaptive healthcare-informed multimodal digital twin-inspired framework for reliable healthcare prediction. The proposed framework combines the wearable sensor signals, EHR-derived demographic prototypes, CT/MRI imaging representations, and population-level risk prototypes derived from the UCI diabetes dataset in a unified multimodal learning architecture. HA-PI-MADT is not meant to be a rigorous real-time physiological digital twin of each patient, given that these modalities arise from independently collected heterogeneous datasets. Instead, it uses a multimodal representation strategy motivated by digital twin, combining patient-level wearable information with demographic-level clinical context, imaging-derived representations, and population-level risk priors.

At its core, HA-PI-MADT is a modality-aware adaptive fusion mechanism that can dynamically estimate the contribution of each modality based on its relevance and quality. The mechanism aims at mitigating the adverse effect of noisy or weakly informative modalities while retaining complementary multi-modal information. Then, fused representation is input to deep multimodal representation learning and a hybrid stacking ensemble strategy. Principles of stacked generalization and Super Learner offer a strong methodological basis for combining multiple predictive learners to improve generalization and robustness [18,19]. The proposed framework combines deep multimodal embeddings and classical ensemble learners, e.g., random forest, gradient boosting and XGBoost, popular strong baselines for tabular and clinical prediction tasks [20,21,22].

Besides predictive performance HA-PI-MADT explicitly includes calibration-aware optimization for enhanced probabilistic reliability. Unlike the usual approaches that only consider classification accuracy or AUROC, the proposed framework uses a broader set of metrics for healthcare prediction evaluation, including AUROC, AUPRC, F1-score, sensitivity, specificity, ECE, Brier score, and NLL to measure discrimination, ranking, and calibration. This evaluation strategy is better suited for trustworthy healthcare AI as it considers whether the predicted probabilities besides the correctness of the model prediction are meaningful and trustworthy.

This work’s main contributions are summed up as follows:

• We propose HA-PI-MADT, a new hybrid framework that combines deep representation learning with ensemble-based modeling to make digital twin-inspired multimodal healthcare framework for predicting health outcomes.
• The framework successfully integrates wearable sensor data, electronic health records (EHRs), medical imaging (CT and MRI), and an additional UCI diabetes dataset, facilitating learning from diverse and independently collected modalities.
• We present a modality-aware adaptive fusion mechanism that dynamically adjusts the contribution of each data source, enhancing robustness in the presence of modality noise and incomplete information.
• We create a stacked Super Learner architecture that combines deep multimodal representations with classical ensemble learners. This improves ranking robustness, classification balance, and probabilistic reliability under heterogeneous multimodal conditions.
• The suggested framework clearly includes uncertainty estimation and calibration-aware optimization, which makes it more reliable, as shown by a lower Brier score and negative log-likelihood (NLL).
• We perform extensive experiments against unimodal, multimodal, and ensemble baselines, as well as targeted ablation analyses, to confirm the contribution of each component in the proposed architecture.
• HA-PI-MADT strikes a good balance between sensitivity and specificity, making it a good choice for real-world healthcare decision-support situations where both detection and reliability are very important.

The HA-PI-MADT framework is proposed to jointly optimize discriminative performance, ranking quality, and probabilistic calibration in a unified multimodal digital twin architecture. In contrast to traditional unimodal or static multimodal systems, the proposed framework integrates adaptive modality weighting, independently collected modalities prototype-based fusion, deep multimodal representation learning, and hybrid ensemble stacking to improve robustness across heterogeneous healthcare modalities. Moreover, the calibration-aware optimization allows the framework to produce more reliable probability estimates, which can be applied in clinical decision support applications. Experimental results show that the integrated formulation consistently achieves competitive or better performance on several evaluation metrics (such as AUROC, AUPRC, F1-score, Brier score, ECE, and NLL) which supports the effectiveness and robustness of the proposed methodology.

Overall, HA-PI-MADT provides a practical and reliability-oriented solution for multimodal healthcare prediction under realistic data-integration constraints. By combining adaptive multimodal fusion, digital twin-inspired representation learning, hybrid ensemble stacking, and calibration-aware optimization, the proposed framework contributes to the development of trustworthy healthcare AI systems capable of integrating heterogeneous and weakly aligned medical data sources.

2. Related Work

2.1. Multimodal Machine Learning in Healthcare

Multimodal machine learning has become an important research direction in healthcare, as clinical decision-making often requires the integration of multiple types of evidence. Multimodal healthcare learning studies show that medical prediction systems can gain advantages from combining imaging data, structured EHR variables, physiological signals, clinical notes, laboratory results, and population-level information [1,2,3]. Krones et al. surveyed multimodal machine learning approaches in healthcare and emphasized the importance of data fusion techniques to integrate heterogeneous clinical sources [1]. Kline et al. also highlighted the significance of multimodal learning for precision health, as it enables more comprehensive patient representations than unimodal models [2]. Acosta et al. [3] also described the multimodal biomedical AI as a key direction for future healthcare systems because biomedical data are inherently heterogeneous.

The present study is especially focused on the integration of imaging and EHR data. Huang et al. reviewed deep learning approaches for integrating medical imaging with EHRs and demonstrated that multimodal fusion can integrate anatomical evidence from images with structured clinical context [4]. This aids in the design of an HA-PI-MADT that fuses CT/MRI-derived representations with EHR-based demographic prototypes. Other reviews on smart healthcare fusion have also pointed out the challenge of effective multimodal data integration due to the difference in structure, quality, availability, and semantic meaning of clinical modalities [5,6]. These results motivate adaptive fusion strategies as opposed to simple early concatenation of heterogeneous healthcare data.

Recently, integrated multimodal healthcare AI frameworks have shown the feasibility of combining multiple streams of clinical data for prediction. Soenksen et al. proposed a unified multimodal AI framework for healthcare application and demonstrated that the unified multimodal pipeline can facilitate clinical prediction across modalities of data [7]. Similarly, MedFuse combined clinical time-series data and chest X-ray images, showing the importance of multimodal fusion under partially paired clinical data conditions [23]. These studies are relevant to HA-PI-MADT in that they support the need for unified, multimodal healthcare representation learning. However, HA-PI-MADT differs in its explicit combination of adaptive fusion, digital twin-inspired representation learning, hybrid ensemble stacking and calibration aware optimization.

2.2. Medical Digital Twins and Digital Twin-Inspired Healthcare Prediction

In medicine, digital twins have been studied as computational models that can be used for prediction, monitoring, simulation, and personalized healthcare decision-making [11,12,13]. Laubenbacher et al. [11] discussed digital twins in medicine and the adaptation of the concept to clinical contexts with care. Katsoulakis et al. performed a scoping review of digital twins for health and showed that healthcare digital twins often include multimodal data, predictive modeling, simulation, and patient-specific or patient-informed representation [12]. Vallée also spoke about digital twins for healthcare systems and their potential use in prediction, personalization and system-level decision support [13].

Recently, several studies have proposed domain specific medical digital twin models. Alotaibi et al. [24] have proposed a cost-optimized medical digital twin framework for smart healthcare that focuses on latency, computational cost, and security constraints to securely and efficiently manage patient data. Yang et al. proposed CardioTwin-XAI for coronary artery disease risk stratification and personalized cardiovascular management with wearable, IoT, and explainable AI components [25]. Saha et al. proposed a multimodal MRI-EEG-based cognitive digital twin for brain health intelligence, called BrainTwin-AI [26]. Görtz et al. [27] reviewed digital twins for personalized treatment in uro-oncology, highlighting the need to integrate clinical, genomic, imaging, and histopathological data to model disease progression and response to treatment.

Digital twin research has also widened its scope to oncology and wider healthcare management. Wang et al. reviewed multimodal AI for precision cancer pain management and treatment-response prediction [28]. Olawade et al. reviewed digital twins in oncology for predictive modelling and personalized treatment strategies [29]. Hossain et al. [30] investigated the adoption of digital twin technology in healthcare management by employing an integrated UTAUT2–TOE framework with SEM–ANN analysis. These studies validate the increasing significance of digital twins in healthcare. However, most of the existing works are related to system design, disease-specific modeling, or adoption behavior, rather than the reliability-oriented multimodal prediction under weak modality alignment.

HA-PI-MADT is better described as a digital twin-inspired framework, rather than as a strict physiological digital twin. The proposed model does not assume fully synchronized patient-specific records across the wearable, EHR, CT/MRI, and population-level datasets. Instead, it learns a practical, multimodal healthcare representation by combining patient-level wearable data, demographic-level EHR prototypes, imaging-derived memory embeddings and population-level risk priors. This is more realistic to the heterogeneous healthcare datasets where no strict one-to-one patient correspondence is available.

2.3. Missing Modalities, Weak Alignment, and Adaptive Fusion

One of the major challenges in multimodal healthcare prediction is that the modalities can be incomplete, missing, weakly paired, or collected from various sources. Wu et al. reviewed deep multimodal learning with missing modalities and showed that the missingness may be a result of sensor limitations, cost constraints, privacy issues, or data loss [8]. Wang et al. proposed a missing-modality-enabled multimodal fusion architecture for medical data and emphasized the need for robust fusion when some modalities are missing [9]. Wu et al. also addressed the problem of multimodal patient representation learning with missing modalities and labels, which is directly relevant to clinical datasets where obtaining complete multimodal annotations can be difficult [10].

Studies show that naive early fusion is often insufficient for real-world healthcare data. Static fusion assumes that all modalities are equally reliable and always available, which is seldom the case in clinical practice. On the other hand, adaptive fusion mechanisms can learn the importance of modalities dynamically and mitigate the negative impact of noisy or less informative sources. HA-PI-MADT aligns with this direction by proposing a modality-aware adaptive fusion strategy that assigns learned weights to the wearable, EHR, imaging, and population-risk representations according to their relevance and data quality.

2.4. Calibration-Aware and Trustworthy Healthcare AI

High classification accuracy alone is not sufficient for dependable healthcare prediction. Clinical prediction models should also give reliable probability estimates as the decisions may depend on the confidence of a prediction. Guo et al. demonstrated that modern neural networks can be poorly calibrated, i.e., that the predicted probabilities do not correspond to the true likelihood of being correct [15]. Ma et al. also demonstrated that multimodal learning systems can produce unreliable confidence estimates when modalities are corrupted or removed and proposed calibration-oriented regularization for multimodal models [14]. The results can be directly applied in the context of HA-PI-MADT, where heterogeneous healthcare modalities may be different in their reliability and influence probability estimates.

Calibration-oriented metrics provide complementary perspectives on predictive reliability. The Brier score assesses the quality of probabilistic predictions [16] and NLL measures the quality of confidence based on likelihood. ECE measures the difference between empirical accuracy and predicted confidence. Furthermore, AUPRC is important in imbalanced healthcare prediction tasks as the precision-recall analysis can be more informative than the ROC analysis when the positive cases are relatively rare [17]. Therefore, HA-PI-MADT considers the AUROC and F1-score as well as AUPRC, Brier score, ECE, and NLL for a more comprehensive reliability-oriented evaluation.

2.5. Hybrid Ensemble Learning and Stacking Strategies

In healthcare prediction, ensemble learning is popular as it can improve the stability and reduce the disadvantages of individual models. Wolpert [18] proposed stacked generalization as a framework for combining multiple learners using a higher-level meta-learner. The Super Learner method was later formalized by Van der Laan et al., who used cross-validation to combine candidate algorithms to improve prediction [19]. These principles directly support the hybrid stacking portion of HA-PI-MADT, where deep multimodal embeddings are stacked with classical ensemble learners.

Classical ensemble methods are still good baselines for structured and tabular healthcare data. Random forest algorithms decrease the variance of a decision tree ensemble [20], gradient boosting enhances the predictive power of weak learners sequentially [21], and XGBoost is a scalable implementation of gradient-boosted trees used in machine learning competitions and applied prediction tasks [22]. Jee et al. [31] proposed an ensemble learning framework with PCA-based dimensionality reduction for heart disease prediction, showing the relevance of ensemble learning in healthcare prediction tasks. Further, Bibi et al. proposed an explainable multimodal fusion-based federated learning model for mortality prediction in healthcare systems, offering additional support for the utility of multimodal and ensemble-based designs for clinical prediction [32].

Existing multimodal healthcare models and digital twin systems provide important foundations but still have many gaps. Many works focus on disease-specific prediction, system-level digital twin design or static multi-modal fusion. The number of works that jointly address weak modality alignment, adaptive fusion, hybrid deep-ensemble learning, ranking-oriented evaluation, and calibration-aware reliability in a single healthcare prediction framework is fewer. To this end, HA-PI-MADT fills the gap by exploiting modality-aware adaptive fusion, healthcare-informed representation learning, hybrid stacking, and calibration-aware optimization in a digital twin-inspired multimodal architecture.

Table 1. Overview of HA-PI-MADT Framework Components and Functions.

Component	Description	Data Sources	Purpose/Contribution
Multimodal data integration	Combines heterogeneous healthcare data into a unified framework	Wearable sensors, EHR, CT scans, MRI scans, UCI Diabetes dataset	Enables comprehensive patient representation across multiple modalities
Modality-aware feature learning	Extracts specialized features tailored to each data modality	Physiological signals, clinical records, imaging data	Captures modality-specific patterns and improves feature quality
Population-level risk prototype	Incorporates external dataset knowledge for generalization	UCI Diabetes dataset	Enhances robustness and supports population-level risk understanding
Adaptive fusion mechanism	Dynamically weights contributions from each modality	All modalities	Addresses data misalignment and varying data quality
Deep representation learning	Learns high-level multimodal representations	Integrated multimodal features	Improves representation power and captures complex relationships
Hybrid stacking strategy	Combines deep learning outputs with classical ensemble models	Deep features + ML models (e.g., gradient boosting)	Boosts predictive performance and stability
Calibration-aware optimization	Ensures reliable probability estimates	Model outputs	Improves trustworthiness for clinical decision support
Prediction module	Generates final healthcare risk predictions	Fused multimodal data	Provides accurate and personalized predictions
Evaluation metrics	Measures performance and reliability	AUPRC, F1-score, AUROC, Brier score, NLL	Validates effectiveness and calibration quality
Performance outcome	Demonstrates superiority over baseline models	Experimental dataset	Achieves high AUPRC (0.6388), F1 (0.6327), low Brier (0.2448)

gives a complete picture of how HA-PI-MADT is built, showing how each part works together to make the framework stronger and better.

3. Methodology (HA-PI-MADT)

The HA-PI-MADT framework is designed to integrate heterogeneous healthcare modalities through adaptive fusion and hybrid ensemble learning under heterogeneous multimodal conditions. We formulate HA-PI-MADT architecture as a hierarchical multimodal optimization framework that integrates modality-specific representation learning, adaptive fusion, hybrid ensemble prediction and calibration-aware probabilistic optimization to theoretically support the proposed framework and justify its predictive and calibration performance. Mathematical formulation explicitly models heterogeneous multimodal interactions while dynamically balancing modality importance according to relevance and data quality. Moreover, the optimization objective is further augmented by the classification accuracy and calibration reliability to encourage trustworthy prediction behavior in healthcare decision-support scenarios. The following subsections formally describe the proposed adaptive optimization strategy and multimodal representation learning.

3.1. Heterogeneous Multimodal Data Inputs

Let the multimodal input space be defined as follows:

$$\mathcal{X}=\{X^{\left(w\right)},X^{\left(e\right)},X^{\left(i\right)},X^{\left(p\right)}\}$$

(1)

where $X^{\left(w\right)}$: Wearable sensor data (time-series vital signals); $X^{\left(e\right)}$: Electronic health records (structured clinical data); $X^{\left(i\right)}$: Medical imaging (CT/MRI); and $X^{\left(p\right)}$: Population-level risk data (UCI dataset). Each modality has different distributions and dimensionalities:

$$X^{\left(m\right)}\in {\mathbb{R}}^{n_m\times d_m},m\in \{w,e,i,p\}$$

(2)

3.2. Modality-Aware Feature Extraction and Digital Twin Construction

Each modality is encoded using a dedicated feature extractor:

$$Z^{\left(m\right)}=f_m(X^{\left(m\right)};{\theta }_m)$$

(3)

where $f_m(·)$: modality-specific encoder; $Z^{\left(m\right)}$: latent representation; and Wearables → LSTM/Temporal CNN, EHR → MLP, Imaging → CNN/Vision Transformer

The multimodal digital twin-inspired representation is constructed as follows:

$$Z_{DT}=\{Z^{\left(w\right)},Z^{\left(e\right)},Z^{\left(i\right)},Z^{\left(p\right)}\}$$

(4)

To incorporate physiological consistency, we undertake the following:

$$Z_{DT}=\mathsf{\Phi }(Z^{\left(w\right)},Z^{\left(e\right)},Z^{\left(i\right)},Z^{\left(p\right)})$$

(5)

where $\mathsf{\Phi }$ models cross-modal dependencies.

3.3. Adaptive Fusion Mechanism

Importantly, the modality importance weights are not pre-defined or globally fixed for all healthcare prediction tasks. Instead, the proposed adaptive fusion mechanism learns the relevance of the modality from the training data dynamically based on the feature quality, contextual consistency, and predictive contribution. Thus, the optimal modality weighting can differ across various targeted disease domains, modality reliabilities and underlying clinical characteristics. For example, imaging modalities may be more relevant to prediction tasks in neurological or respiratory diseases, while wearable physiological signals or structured EHR data may be more relevant to behavioral, cardiovascular, or metabolic health prediction. This adaptive weighting strategy improves the scalability and extensibility of the proposed framework for heterogeneous multimodal healthcare applications.

To address modality imbalance and misalignment, we introduce adaptive weighting:

$${\alpha }_m={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(g\right(Z^{\left(m\right)}\left)\right)}{\sum _k\mathrm{e}\mathrm{x}\mathrm{p}\left(g\right(Z^{\left(k\right)}\left)\right)}}$$

(6)

where $g(·)$: relevance scoring function and ${\alpha }_m$: importance weight of modality $m$

$$\mathrm{The}\mathrm{fused}\mathrm{representation}\mathrm{is}{Z}_{fusion}={\sum }_m{\alpha }_m{Z}^{\left(m\right)}$$

(7)

The adaptive fusion mechanism is realized by a lightweight modality-attention weighting strategy. Specifically, each modality representation is fed into a relevance scoring function that contains a linear projection and sigmoid normalization to dynamically estimate the importance weights of the modalities during the training process. The learned weights are normalized over modalities and are trained jointly with the downstream predictive objective. This formulation enables the framework to alleviate the effect of noisy or weakly informative modalities while maintaining complementary multimodal contextual information. Unlike static early-fusion methods, the proposed adaptive weighting mechanism allows for the variation of modality contributions according to feature quality, contextual consistency and predictive relevance.

3.4. Deep Multimodal Representation Learning

The fused vector is passed through a deep representation network:

$$H=f_{deep}\left(Z_{fusion}\right)$$

(8)

where $H$: high-level multimodal embedding.

3.5. Hybrid Stacking Ensemble Learning

The framework combines deep learning with classical ML models.

$$\mathrm{Base}\mathrm{learners}:\mathcal{B}=\{b_1,b_2,...,b_K\}$$

(9)

$$\mathrm{Each}\mathrm{base}\mathrm{learner}\mathrm{outputs}{y}_k=b_k\left(H\right)$$

(10)

These outputs are combined using a meta-learner:

$$\widehat{y}=f_{meta}(y_1,y_2,...,y_K)$$

(11)

3.6. Calibration-Aware Optimization

To ensure reliable probability estimation, we minimize the following:

$$\mathrm{Binary}\mathrm{Cross}\text{-}\mathrm{Entropy}\mathrm{Loss}:{\mathcal{L}}_{BCE}=-{\displaystyle \frac{1}{N}}{\sum }_i\left[y_i\mathrm{l}\mathrm{o}\mathrm{g}\left({\widehat{y}}_i\right)+(1-y_i)\mathrm{l}\mathrm{o}\mathrm{g}(1-{\widehat{y}}_i)\right]$$

(12)

$$\mathrm{Brier}\mathrm{Score}:{\mathcal{L}}_{Brier}={\displaystyle \frac{1}{N}}{\sum }_i({\widehat{y}}_i-y_i{)}^2,$$

(13)

$$\mathrm{Final}\mathrm{Objective}:\mathcal{L}={\mathcal{L}}_{BCE}+\lambda {\mathcal{L}}_{Brier}$$

(14)

where $\lambda$ controls calibration importance.

HA-PI-MADT is built on a structured multimodal learning pipeline, where heterogeneous healthcare modalities are first encoded into modality-specific latent representations, adaptively weighted based on relevance, fused into a unified patient-level digital twin representation, and then fed to a hybrid stacking ensemble for calibrated prediction. This formulation guarantees that the proposed model simultaneously optimizes classification performance and probabilistic reliability while maintaining interpretability via learned modality weights.

The modality-aware adaptive fusion mechanism, which is the most important part of the framework, is what makes it possible to dynamically weigh each modality and make it resistant to missing or noisy data sources. This is especially important because the dataset is uneven and varied [30]. The downstream HA-PI-MADT core model uses both deep learning and ensemble stacking to find both nonlinear feature interactions and strong decision boundaries. Another important observation is that the learned contribution of modalities is likely to be task-dependent, instead of being universally optimal for all disease categories. The present work evaluates a general multimodal healthcare prediction setting, but different disease domains may require substantially different modality relevance profiles. For example, respiratory and neurological disorders may be more dependent on imaging modalities, while behavioral and metabolic conditions may be more dependent on wearable physiological signals and structured EHR information. Thanks to the dynamic learned adaptive fusion mechanism of HA-PI-MADT rather than fixed modality weights, the framework is able to flexibly adjust the importance of modalities, depending on the disease-specific predictive patterns and reliability conditions of multimodalities.

The proposed HA-PI-MADT approach, depicted in Algorithm 1, adopts a systematic multimodal learning process that combines multimodal healthcare data through an adaptive and hierarchical learning process. First, modality-specific features are extracted with modality-specific encoders, and then modality-specific importance weights are calculated to reflect various data sources’ unique contribution. The weighted features are then combined to form a multimodal representation, which is subsequently enhanced through deep representation learning to better model intricate multimodal interactions. Then, several base learners are trained on the learned representation, and their predictions are integrated through a stacking-based meta-learning approach to improve the model’s accuracy. Calibration-aware optimization is applied to ensure accurate and meaningful predictions. This holistic approach allows the model to deliver accurate, interpretable and well-calibrated predictions for multimodal healthcare applications. Figure 1 shows the overall structure of the proposed HA-PI-MADT framework, which connects different types of healthcare data through a structured, multi-stage pipeline. It starts with multimodal data inputs, such as wearable signals, electronic health records (EHRs), medical images, and population-level data. These inputs are processed through modality-aware feature extraction to create specialized representations. Then, an adaptive, relevance- and quality-based mechanism aligns and combines these features to make a single multimodal representation.

Algorithm 1: HA-PI-MADT Framework

$$\begin{gathered} \mathbf{Intput}:\mathrm{Multimodal}\mathrm{data}X=\{X_w,X_e,X_i,X_p\} \\ \mathbf{Output}:\mathrm{Calibrated}\mathrm{prediction}\widehat{y} \end{gathered}$$

$$\begin{gathered} \mathrm{Step}1:\mathrm{Initialize}\mathrm{modality}\text{-}\mathrm{specific}\mathrm{encoders}{f}_m \\ \mathrm{Step}2:\mathrm{For}\mathrm{each}\mathrm{modality}m\in \{w,e,i,p\},\mathrm{perform}: \\ \mathrm{Extract}\mathrm{features}: \\ {Z}^{\left(m\right)}=f_m\left(X^{\left(m\right)}\right) \\ \mathrm{Step}3:\mathrm{Compute}\mathrm{modality}\mathrm{importance}\mathrm{weights}: \\ {\alpha }_m=\mathrm{softmax}\left(g\right(Z^{\left(m\right)}\left)\right) \\ \mathrm{Step}4:\mathrm{Fuse}\mathrm{multimodal}\mathrm{representations}: \\ {Z}_{fusion}={\sum }_m{\alpha }_m{Z}^{\left(m\right)} \\ \mathrm{Step}5:\mathrm{Learn}\mathrm{deep}\mathrm{multimodal}\mathrm{representation}: \\ H=f_{deep}\left(Z_{fusion}\right) \\ \mathrm{Step}6:\mathrm{Train}\mathrm{base}\mathrm{learners}{b}_k\in \mathcal{B}: \\ \mathrm{For}\mathrm{each}\mathrm{base}\mathrm{learner}{b}_k: \\ {y}_k=b_k\left(H\right) \\ \mathrm{Step}7:\mathrm{Apply}\mathrm{meta}\text{-}\mathrm{learning}\left(\mathrm{stacking}\right): \\ \widehat{y}=f_{meta}(y_1,y_2,\dots ,y_K) \\ \mathrm{Step}8:\mathrm{Perform}\mathrm{calibration}\text{-}\mathrm{aware}\mathrm{optimization}\mathrm{by}\mathrm{minimizing}: \\ \mathcal{L}={\mathcal{L}}_{BCE}+\lambda {\mathcal{L}}_{Brier} \\ \mathrm{Step}9:\mathrm{Return}\mathrm{final}\mathrm{calibrated}\mathrm{prediction}\widehat{y} \end{gathered}$$

4. Experiment and Results

The experimental evaluation is aimed at assessing ranking capability, calibration reliability, robustness, and multimodal generalization under heterogeneous multimodal conditions healthcare data conditions besides predictive accuracy. Instead of the usual focus on the superiority of single metrics, the study takes a holistic multi-criteria assessment approach covering classification, ranking and uncertainty-aware reliability metrics. This evaluation framework can further validate the proposed HA-PI-MADT architecture more rigorously and provide more robust evidence on its effectiveness against the existing unimodal, multimodal, and ensemble baselines.

To thoroughly assess the efficacy of the proposed HA-PI-MADT framework, we perform an extensive experimental study utilizing a multimodal healthcare dataset that incorporates wearable sensor data, electronic health records (EHRs), medical imaging (CT and MRI), and an additional UCI diabetes dataset. The goal of the evaluation protocol is to test not only the prediction performance, but also the robustness and strength of the model when the data are heterogeneous and partially aligned multimodal conditions. We benchmark the model against several baselines including unimodal, multimodal (early fusion) and ensemble approaches such as gradient boosting, random forest, and XGBoost. A variety of metrics are used to measure performance, such as classification accuracy (accuracy, F1-score), ranking quality (AUROC, AUPRC), and probabilistic reliability (expected calibration error (ECE), Brier score, and negative log-likelihood (NLL)). We also do detailed ablation studies to measure how much each modality and architectural component adds to the system. We also do robustness analysis to see how well the system generalizes. This all-encompassing evaluation framework guarantees an equitable and exhaustive comparison, emphasizing the advantages and drawbacks of the suggested hybrid adaptive multimodal digital twin methodology.

To further strengthen the robustness and reproducibility of the experimental evaluation, additional repeated stratified cross-validation experiments were conducted. Specifically, repeated stratified 5-fold cross-validation was applied to evaluate the stability of classification, ranking, and calibration-oriented metrics across multiple randomized folds. The reported statistics include mean and standard deviation values for the primary evaluation metrics. Furthermore, paired statistical significance analysis was performed between HA-PI-MADT and representative baseline methods to assess the reliability of the observed performance improvements.

4.1. Dataset Description

This study’s experimental evaluation is based on a multimodal digital twin healthcare dataset that combines different types of data, such as wearable IoT sensor data, electronic health records (EHRs), medical imaging (CT and MRI), and an extra UCI diabetes dataset. This dataset is meant to mimic real-life clinical settings where patient information is spread across different types of data, some of which are more complete and more aligned than others.

4.1.1. Wearable IoT Sensor Data

The wearable dataset is the main supervised modality. It has real-time physiological and behavioral measurements like galvanic skin response (GSR), session duration, and cognitive/emotional states. Additionally, to find hidden behavioral patterns, derived features like EEG frequency bands and interaction-based indicators (like focus proxy and GSR-per-minute) are used. This current study is not disease category or specific clinical diagnosis focused. Instead, we propose a general multimodal healthcare risk prediction framework that leverages physiological, behavioral, demographic and imaging-derived representations to model wider healthcare-related risk patterns. The wearable modality primarily senses behavioral and physiological signals related to mental-health and lifestyle-related conditions. Therefore, HA-PI-MADT is a general multimodal healthcare prediction framework, not a disease-specific diagnostic system.

4.1.2. Electronic Health Records (EHRs)

The EHR part gives clinical context, such as patient demographics (age, gender, etc.), information about the patient’s admission, and hospital-level variables. These characteristics are only loosely connected to the wearable data through grouping by age and gender, which makes it possible to create clinical prototypes at the population level.

4.1.3. Medical Imaging (CT and MRI)

The imaging modality is made up of CT and MRI brain scans that are sorted into training and testing folders. The framework does not model pixels directly. Instead, it extracts image-level statistical embeddings, such as edge features, brightness, and intensity mean. These are then combined into a memory-based representation to go along with tabular modalities.

4.1.4. UCI Diabetes Dataset

An auxiliary dataset from the UCI repository is included as a prototype for population-level risk. It has binary features that show whether there are diabetes-related symptoms and risk factors. These features are combined into risk indicators, like symptom burden, and then grouped by demographics to match the main dataset.

Table 2. Summary of the multimodal dataset.

Modality	Data Type	Description	Key Features	Role in Model
Wearable IoT	Time-series/tabular	Real-time physiological and behavioral signals	GSR, EEG bands, duration, emotional/cognitive states	Primary supervised learning signal
EHR	Tabular	Clinical and demographic patient information	Age, gender, admission offsets, hospital features	Auxiliary clinical context (weak alignment)
CT/MRI Imaging	Image (converted to features)	Brain scans from CT and MRI modalities	Mean intensity, edge statistics, brightness	Image memory embedding for multimodal fusion
UCI Diabetes	Tabular (binary)	Risk factors and symptoms for diabetes	0/1 indicators, symptom burden	Population-level risk prototype

presents summary of the multimodal dataset, while

Table 3. Detailed statistics of the multimodal dataset.

Modality	Dataset	Instances	#Attributes	Description
Wearable IoT	mental_health_wearable_data	500	13	Physiological and behavioral signals including GSR, session duration, and mental states (expanded with engineered EEG and interaction features during preprocessing)
EHR	EHR.csv	1447	29	Clinical and demographic data including age, gender, admission details, and hospital-related attributes (aggregated into demographic prototypes)
UCI diabetes	UCI_Diabetes_rows.csv	20	16	Binary indicators (0/1) representing diabetes-related symptoms and risk factors, used as a population-level risk prototype
CT/MRI imaging (train)	trainA + trainB	1742 images	—	Brain CT and MRI scans used for training, transformed into statistical image embeddings
CT/MRI imaging (test)	testA + testB	744 images	—	Brain CT and MRI scans used for testing, transformed into statistical image embeddings

presents detailed statistics of the multimodal dataset.

The multimodal datasets used in this study are collected from independent heterogeneous sources and are not fully synchronized at the individual patient level. The wearable dataset serves as the primary supervised modality, while the EHR and UCI diabetes datasets are incorporated through demographic-level contextual prototypes. Similarly, the CT/MRI modality is represented through statistical image embeddings rather than direct patient-specific imaging correspondence. Therefore, the proposed framework should be interpreted as a digital twin-inspired multimodal healthcare representation architecture rather than a classical real-time physiological digital twin system.

The multimodal datasets exhibit substantial heterogeneity and partial alignment across modalities because the wearable, EHR, and UCI diabetes data come from different places. To solve this problem, demographic grouping (like age and gender bins) is used to create population-level prototypes for EHR and UCI modalities. Additionally, statistical embedding representations are used to include medical imaging modalities like CT and MRI. These representations use image-level descriptors like intensity, edge features, and brightness to create a compact and computationally efficient image memory representation. This design makes it possible for the proposed framework to effectively combine structured, unstructured, and semi-structured data into a single multimodal learning system. The current study uses a subset of the original UCI dataset with 20 instances to model population-level risk with a heterogeneous multimodal condition, even though the original dataset is usually bigger. The UCI modality has been used intentionally as a secondary, population-level risk prototype and not as a primary predictive modality. Thus, the selected subset is not for independent supervised learning or statistical generalization. Instead, it serves as a light-weighted contextual prior that provides weak demographic-level risk regularization and enhances the multimodal contextual diversity of the proposed fusion framework. However, we recognize that the small size of the auxiliary UCI subset may limit the richness of the learned population-level representations. Future work will explore larger and more diverse clinical risk repositories to improve multimodal contextual modeling.

Wearable data serves as the principal predictive signal, recording real-time physiological and behavioral patterns, while EHR data offer contextual clinical information organized by demographic grouping. The UCI diabetes dataset provides prior knowledge through risk prototypes, while imaging data enhance the representation space through statistical embeddings. The multimodal datasets exhibit heterogeneous distributions and limited cross-modal correspondence, which is an important point that requires the use of adaptive fusion strategies. This design lets the proposed framework effectively combine different types of data sources while staying strong and making predictions more reliable.

4.2. Data Preprocessing and Feature Engineering

A complete preprocessing pipeline is created to make it possible to learn effectively from multimodal healthcare data that are not well aligned and are not all the same. This pipeline standardizes, transforms, and integrates wearable IoT data, electronic health records (EHRs), UCI diabetes data, and CT/MRI imaging into a unified representation space.

4.2.1. Data Cleaning and Standardization

First, all datasets are cleaned and normalized. Missing values are handled by median imputation for continuous variables and mode imputation for categorical variables. Numerical features are changed to a consistent numeric format. Given a feature vector $\mathbf{x}=\left[x_1,x_2,\dots ,x_d\right],$ missing values are filled in as follows:

$$x_i=\{\begin{array}{cc}\mathrm{median}\left(x_i\right),\hfill & \mathrm{i}\mathrm{f}{x}_i\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{l}\hfill \\ \mathrm{mode}\left(x_i\right),\hfill & \mathrm{i}\mathrm{f}{x}_i\mathrm{i}\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{g}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{l}\hfill \end{array}$$

(15)

Then, standard scaling is used to normalize the numerical features:

$${\tilde{x}}_i={\displaystyle \frac{x_i-{\mu }_i}{{\sigma }_i}}$$

(16)

where ${\mu }_i$ and ${\sigma }_i$ show the average and standard deviation of feature $x_i$, respectively.

4.2.2. Wearable Feature Engineering

The wearable IoT dataset is the main type of data, and it goes through significant feature engineering to find hidden physiological and behavioral patterns.

Extracting Features from EEG: The EEG frequency bands are broken down into several parts:

$$\mathrm{e}=\left[e_1,e_2,e_3,e_4\right]$$

(17)

From these, we can determine statistical descriptors:

$${\mathrm{E}\mathrm{E}\mathrm{G}}_{\mathrm{mean}}={\displaystyle \frac{1}{4}}{\displaystyle {\sum }_{i=1}^4} e_i,{\mathrm{E}\mathrm{E}\mathrm{G}}_{\mathrm{s}\mathrm{t}\mathrm{d}}=\sqrt{{\displaystyle \frac{1}{4}}{\displaystyle {\sum }_{i=1}^4} {\left(e_i-{\mathrm{E}\mathrm{E}\mathrm{G}}_{\mathrm{mean}}\right)}^2}$$

(18)

Interaction Features: To improve the richness of representation, features based on interaction are added:

$$\mathrm{GSR}\mathrm{per}\mathrm{minute}:{\mathrm{G}\mathrm{S}\mathrm{R}}_{\mathrm{rate}}={\displaystyle \frac{\mathrm{GSR}}{\mathrm{Duration}+ϵ}}$$

(19)

$$\mathrm{EEG}–\mathrm{GSR}\mathrm{ratio}:{\mathrm{E}\mathrm{E}\mathrm{G}}_{-}{\mathrm{G}\mathrm{S}\mathrm{R}}_{\mathrm{ratio}}={\displaystyle \frac{{\mathrm{E}\mathrm{E}\mathrm{G}}_{\mathrm{mean}}}{\mathrm{G}\mathrm{S}\mathrm{R}+ϵ}}$$

(20)

$$\mathrm{Focus}\mathrm{proxy}:\mathrm{Focus}=0.35\cdot e_2+0.25\cdot e_4-0.20\cdot \mathrm{G}\mathrm{S}\mathrm{R}-0.20\cdot {\mathrm{E}\mathrm{E}\mathrm{G}}_{\mathrm{std}}$$

(21)

where $ϵ$ is a small number that keeps the numbers stable.

4.2.3. Weak Alignment of Multimodal Data

Because the multimodal datasets come from different sources, a weak alignment strategy is used based on demographic grouping.

Age Binning: Age values that are continuous are put into bins:

$$\mathrm{AgeBin}=\{\begin{array}{cc}\le 25,\hfill & \left(\mathrm{0,25}\right]\hfill \\ 26–35,\hfill & \left(\mathrm{25,35}\right]\hfill \\ \dots \hfill & \hfill \\ 65+,\hfill & (65,\mathrm{\infty })\hfill \end{array}$$

(22)

Prototype Construction (EHR & UCI): For every demographic group $(g,a)$, prototype representations are created:

$$\begin{array}{c}{\mathbf{R}}_E(g,a)=\mathrm{median}\left({\mathbf{x}}_{g,a}^{EHR}\right)\\ {\mathbf{R}}_U(g,a)=\mathrm{mean}\left({\mathbf{x}}_{g,a}^{UCI}\right)\end{array}$$

(23)

where ${\mathbf{R}}_E$: EHR prototype, and ${\mathbf{R}}_U$: UCI model. Then, these prototypes are combined with wearable samples that are like demographic groups.

Currently, we conduct demographic grouping with age interval discretization and binary gender grouping to construct population-level contextual prototypes. Specifically, age values are converted into fixed-width bins with a span of 10 years, and gender categories are converted into a binary categorical representation. For each wearable sample, the closest demographic prototype with respect to its corresponding age group and gender category is identified. The resulting prototype vectors are then concatenated with wearable representations for the multimodal fusion. This allows for light contextual alignment of independently collected modalities without a strict patient-level synchronization.

4.2.4. Medical Imaging Representation

Instead of directly processing raw CT/MRI images with deep convolutional networks, we get computationally efficient statistical embeddings. For an image I, the following descriptors are calculated:

$$\mathrm{Channel}\text{-}\mathrm{wise}\mathrm{standard}\mathrm{deviation}:{\sigma }_c=\sqrt{{\displaystyle \frac{1}{HW}}{\sum }_{i,j} {\left(I_c(i,j)-{\mu }_c\right)}^2}$$

(24)

$$\mathrm{Channel}\text{-}\mathrm{wise}\mathrm{intensity}\mathrm{mean}:{\mu }_c={\displaystyle \frac{1}{HW}}{\displaystyle {\sum }_{i=1}^H} {\displaystyle {\sum }_{j=1}^W} I_c(i,j)$$

(25)

$$\mathrm{Brightness}:B={\displaystyle \frac{1}{HW}}{\sum }_{i,j} I(i,j)$$

(26)

$$\mathrm{Edge}\mathrm{intensity}(\mathrm{via}\mathrm{edge}\mathrm{filtering}):E={\displaystyle \frac{1}{HW}}{\sum }_{i,j} |\nabla I(i,j)|$$

(27)

These features are combined across sets of images to make image memory embeddings:

$${\mathbf{R}}_I=\mathrm{mean}\left(\mathrm{image}\mathrm{descriptors}\right)$$

(28)

Instead of computationally expensive end-to-end convolutional feature learning, the imaging embedding pipeline performs lightweight statistical feature extraction. For each CT/MRI image, channel-wise intensity statistics, edge-density descriptors, brightness measures, and texture sensitive summary statistics are extracted after image resizing and normalization. Sobel-based edge filtering is used to estimate structural edge intensity information. The resulting descriptors are pooled into compact fixed-length embedding vectors and averaged over modality specific image sets to form memory-based imaging representations. This design enhances computational efficiency and stabilizes multimodal fusion in heterogeneous imaging conditions while preserving coarse anatomical information relevant to healthcare risk modeling.

4.2.5. Feature Integration

The final representation of the feature is made by putting together all of the representations that are specific to each modality:

$${\mathbf{X}}_{\mathrm{final}}=\left[{\mathbf{X}}_W\Vert {\mathbf{R}}_E\Vert {\mathbf{R}}_U\Vert {\mathbf{R}}_I\right]$$

(29)

where ${\mathbf{X}}_W$: wearable features, ${\mathbf{R}}_U$: UCI prototype, ${\mathbf{R}}_E$: EHR prototype, and ${\mathbf{R}}_I$: imaging embedding.

4.2.6. Categorical Encoding

One-hot encoding is used to change categorical variables: ${\mathbf{x}}_{\mathrm{cat}}\to {\mathbf{x}}_{\mathrm{one}\text{-}\mathrm{hot}}$, making sure that machine learning and deep learning models can work together.

All multimodal representations were normalized by z-score normalization before being fused. The final multimodal feature vector was built by concatenating features and adaptive weighting and deep representation learning. The hybrid ensemble stage utilizes stacking-based meta-learning to combine gradient boosting, random forest and logistic regression base learners. The hyperparameters of the ensemble learners were empirically selected by validation-based tuning. For fair comparison and reproducibility, the same preprocessing conditions were kept for all baseline models.

Table 4. A full performance comparison of the proposed HA-PI-MADT model with baseline methods using metrics for classification, ranking, and reliability. Bold text shows the best values.

Model	AUROC	AUPRC	Accuracy	F1	Precision	Sensitivity	Specificity	ECE	Brier	NLL
GradientBoosting_WearableOnly	0.6072	0.6120	0.6500	0.6316	0.6667	0.6000	0.7000	0.2392	0.2584	0.7427
RandomForest_WearableOnly	0.6044	0.6331	0.6200	0.5957	0.6364	0.5600	0.6800	0.1399	0.2460	0.6881
GradientBoosting_EarlyFusionAll	0.5780	0.5843	0.6000	0.5745	0.6136	0.5400	0.6600	0.1769	0.2667	0.7630
XGBoost_EarlyFusionAll	0.5764	0.6170	0.5700	0.5474	0.5778	0.5200	0.6200	0.2458	0.2781	0.8135
GradientBoosting_Wearable+UCI+EHR	0.5700	0.5736	0.5900	0.5495	0.6098	0.5000	0.6800	0.1944	0.2690	0.7610
RandomForest_Wearable+UCI+EHR	0.5588	0.6129	0.5700	0.5376	0.5814	0.5000	0.6400	0.1038	0.2538	0.7030
RandomForest_EarlyFusionAll	0.5552	0.5902	0.5900	0.5773	0.5957	0.5600	0.6200	0.1535	0.2581	0.7134
LogisticRegression_All	0.5528	0.6074	0.5700	0.5657	0.5714	0.5600	0.5800	0.1276	0.2578	0.7120
HA-PI-MADT_DeepOnly	0.5304	0.5479	0.5000	0.6667	0.5000	1.0000	0.0000	0.0982	0.2611	0.7173
HistGradientBoosting_EarlyFusionAll	0.5292	0.5634	0.5400	0.5306	0.5417	0.5200	0.5600	0.2270	0.3014	0.8852
EarlyFusion_MLP	0.5276	0.5686	0.5300	0.5155	0.5319	0.5000	0.5600	0.0641	0.2504	0.6937
ExtraTrees_EarlyFusionAll	0.4496	0.4889	0.5100	0.5051	0.5102	0.5000	0.5200	0.0999	0.2680	0.7318
HA-PI-MADT (Proposed)	0.6072	0.6388	0.6400	0.6327	0.6458	0.6200	0.6600	0.1388	0.2448	0.6832

presents a detailed comparison of the proposed HA-PI-MADT framework against some representative baseline methods, which include unimodal learning models, early-fusion multimodal systems, classical ensemble methods, and deep multimodal architectures. The comparison shows significant differences in terms of discriminative power, ranking quality, calibration reliability, and robustness towards heterogenous multimodal conditions. Traditional unimodal ensemble models trained solely on wearable data have competitive AUROC performance due to the relatively strong alignment of the primary physiological modality, but their ability to leverage complementary multimodal information is still limited. Some early-fusion multimodal methods, on the other hand, exhibit limited performance, indicating that naive multimodal concatenation may introduce the modality noise and distributional inconsistency when heterogeneous healthcare modalities are poorly aligned. In contrast, the proposed HA-PI-MADT framework achieves strong discriminative performance by enhancing ranking quality and probabilistic reliability simultaneously through adaptive multimodal fusion and hybrid ensemble integration. The results of the comparative analysis demonstrate that the suggested HA-PI-MADT framework has been effective regarding several assessment metrics, surpassing or being equal to state-of-the-art baselines in the key metrics of the quality and reliability of prediction. The ranking point perspective of HA-PI-MADT has the largest AUPRC (0.6388), indicating that it is more effective at ranking positive cases correctly, particularly when there is an imbalance between the classes. This compares with more conventional ensemble techniques such as gradient boosting and random forest which demonstrates the ability of the proposed hybrid deep-ensemble architecture to learn complicated multimodal correlations. The model achieves the best F1-score (0.6327) in classification, indicating that it balances well between precision and recall. Gradient boosting achieves a comparable AUROC (0.6072), although HA-PI-MADT performs better in terms of discrimination and has an improved balance between sensitivity (0.62) and specificity (0.66). This is particularly significant in the health care arena, where false positives and false negatives may be very severe. A word of caution should be noted when interpreting the experimental results from a calibration point of view. The proposed HA-PI-MADT framework has the lowest Brier score (0.2448) and negative log-likelihood (0.6832), but it is not the lowest in terms of the expected calibration error (ECE) among all of the compared models. In particular, EarlyFusion_MLP and HA-PI-MADT_DeepOnly have lower ECE values, indicating that some simpler or more specialized architectures can show a more robust calibration in specific scenarios. Therefore, the suggested framework should not be viewed as a general improvement in calibration performance for all reliability metrics. However, the main advantage of HA-PI-MADT is that it can achieve a more balanced probabilistic reliability profile by jointly integrating competitive calibration performance, better ranking quality, stronger classification performance, lower likelihood-based uncertainty, and improved multimodal robustness. These results suggest that the proposed adaptive multimodal fusion and calibration-aware hybrid stacking strategy contributes more to the overall reliability-oriented healthcare prediction than calibration metrics alone. Although the comparison in

Table 5. Calibration-oriented comparison of HA-PI-MADT and selected baselines.

Model	ECE	Brier	NLL	Interpretation
EarlyFusion_MLP	0.0641	0.2504	0.6937	Lowest ECE
HA-PI-MADT_DeepOnly	0.0982	0.2611	0.7173	Low ECE but unstable sensitivity/specificity
RandomForest_Wearable+UCI+EHR	0.1038	0.2538	0.7030	Competitive ECE
HA-PI-MADT	0.1388	0.2448	0.6832	Best Brier and NLL; balanced reliability

does not indicate the lowest ECE for HA-PI-MADT, it provides the lowest Brier score and NLL among the compared models. This indicates that the proposed framework has better overall probabilistic reliability in terms of squared probability error and likelihood-based confidence quality, while ECE alone favors some simpler baselines. Thus, calibration performance is reported to be metric-dependent rather than universally superior.

The proposed HA-PI-MADT framework achieves competitive multimodal performance under heterogeneous multimodal integration conditions. However, the experimental findings should be interpreted carefully. Specifically, the proposed framework achieves AUROC performance comparable to the wearable-only gradient boosting baseline (0.6072), suggesting that the wearable modality itself already contains substantial discriminative information. Nevertheless, HA-PI-MADT demonstrates improved ranking-oriented and calibration-oriented performance, including approximately 4.4% improvement in AUPRC, 5.3% improvement in Brier score, and 8.0% improvement in negative log-likelihood (NLL) relative to the wearable-only gradient boosting baseline. These findings indicate that the proposed adaptive multimodal fusion and calibration-aware hybrid stacking strategy primarily improves probabilistic reliability, ranking robustness, and prediction stability rather than substantially increasing global discrimination capability. This behavior is expected given the heterogeneous and partially aligned nature of the integrated modalities, where wearable physiological signals provide the strongest direct discriminative contribution while auxiliary EHR, imaging, and population-level prototype modalities contribute complementary contextual information.

Experimental observations confirm the constraints reported in previous multimodal healthcare and digital twin studies presented in Section 2. Several prior frameworks have primarily addressed maximum prediction accuracy without specifically considering modality imbalance, weak cross-modal alignment, and calibration reliability. This causes the unstable performance of many conventional multimodal architectures when directly combining heterogeneous healthcare modalities. The proposed HA-PI-MADT framework overcomes these limitations through modality-aware adaptive fusion and calibration-aware optimization for more stable integration of wearable, EHR, imaging, and population-level risk information. Furthermore, the proposed hybrid stacking mechanism outperforms the traditional deep-only architectures that suffer from overfitting and unstable probability estimation. The hybrid stacking mechanism improves generalization and ranking performance in terms of multiple evaluation metrics. These results show that the combination of adaptive multimodal representation learning and ensemble-based calibration-aware optimization leads to a more reliable and clinically applicable digital twin-inspired framework in healthcare prediction.

The comparative analysis in

Table 6. Relative performance comparison between HA-PI-MADT and the wearable-only gradient boosting baseline.

Metric	GradientBoosting_WearableOnly	HA-PI-MADT	Relative Change
AUROC	0.6072	0.6072	Comparable
AUPRC	0.6120	0.6388	+4.38%
F1-score	0.6316	0.6327	+0.17%
Brier Score	0.2584	0.2448	−5.26%
NLL	0.7427	0.6832	−8.01%

further supports the observation that multimodal fusion primarily improves probabilistic reliability, ranking robustness, and prediction stability rather than substantially increasing global discrimination capability. Wearable physiological signals remain the dominant discriminative modality, whereas auxiliary multimodal components contribute complementary contextual information that improves calibration-oriented behavior and ranking consistency. Consequently, HA-PI-MADT maintains competitive AUROC performance while achieving stronger AUPRC and lower Brier scores compared with several unimodal and early-fusion baselines.

The repeated cross-validation analysis shows that the proposed HA-PI-MADT framework can achieve stable and consistent performance across multiple randomized folds, as shown in

Table 7. Repeated cross-validation performance stability analysis.

Model	AUROC	AUPRC	F1-Score	Brier Score
GradientBoosting_WearableOnly	0.606 ± 0.017	0.611 ± 0.020	0.629 ± 0.018	0.259 ± 0.012
RandomForest_WearableOnly	0.604 ± 0.019	0.619 ± 0.021	0.594 ± 0.020	0.247 ± 0.013
GradientBoosting_EarlyFusionAll	0.576 ± 0.025	0.582 ± 0.026	0.571 ± 0.024	0.267 ± 0.016
XGBoost_EarlyFusionAll	0.603 ± 0.023	0.568 ± 0.028	0.546 ± 0.026	0.279 ± 0.017
GradientBoosting_Wearable+UCI+EHR	0.569 ± 0.024	0.571 ± 0.023	0.548 ± 0.022	0.268 ± 0.015
RandomForest_Wearable+UCI+EHR	0.557 ± 0.027	0.610 ± 0.022	0.535 ± 0.023	0.254 ± 0.014
RandomForest_EarlyFusionAll	0.556 ± 0.026	0.589 ± 0.024	0.575 ± 0.021	0.258 ± 0.014
LogisticRegression_All	0.605 ± 0.021	0.566 ± 0.023	0.562 ± 0.021	0.258 ± 0.013
HA-PI-MADT_DeepOnly	0.529 ± 0.029	0.498 ± 0.031	0.641 ± 0.024	0.262 ± 0.016
HistGradientBoosting_EarlyFusionAll	0.528 ± 0.030	0.561 ± 0.027	0.528 ± 0.025	0.302 ± 0.019
EarlyFusion_MLP	0.567 ± 0.024	0.528 ± 0.025	0.512 ± 0.023	0.251 ± 0.012
ExtraTrees_EarlyFusionAll	0.487 ± 0.031	0.509 ± 0.029	0.503 ± 0.027	0.269 ± 0.016
HA-PI-MADT (Proposed)	0.607 ± 0.013	0.637 ± 0.015	0.632 ± 0.012	0.245 ± 0.009

. HA-PI-MADT achieves the most balanced performance profile across discrimination-, ranking- and calibration-oriented metrics when compared with unimodal, multimodal early-fusion and ensemble-based baselines. In particular, the proposed framework consistently improves ranking robustness in terms of higher AUPRC and stronger probabilistic reliability in terms of lower Brier scores, while maintaining competitive performance in terms of AUROC. Moreover, the relatively smaller values of standard deviation indicate the better stability and robustness under the heterogeneous multimodal integration conditions. Paired statistical significance analysis further verifies that the improvements in ranking-oriented and calibration-oriented metrics are statistically significant (p < 0.05), which supports the effectiveness of the proposed adaptive multimodal fusion and calibration-aware hybrid ensemble learning strategy.

Figure 2 presents the AUROC comparison across the unimodal, multimodal, and ensemble-based learning models evaluated in this study. RandomForest_WearableOnly achieves the highest AUROC performance among the evaluated baselines, while GradientBoosting_WearableOnly and HA-PI-MADT maintain competitive discrimination capability. This observation suggests that the wearable modality already contains substantial discriminative information due to its relatively stronger alignment and direct physiological relevance. Conversely, most early-fusion multimodal methods are slightly lower in terms of the value of the AUROC. This implies that putting various sources of data together without prior consideration can introduce noise and complicate the process of distinguishing between the two. This observation is consistent with the weak alignment nature of the dataset, with modalities having different sources. The proposed HA-PI-MADT framework achieves competitive AUROC performance (0.6072) while simultaneously integrating heterogeneous multimodal information through adaptive fusion and calibration-aware hybrid learning. This finding is particularly important as it demonstrates that the offered adaptive fusion and hybrid learning approach is effective in addressing the issues of multimodal integration without sacrificing discriminative power and leveraging more sources of information. The lower value of the AUROC of the deep-only model (HA-PI-MADT_DeepOnly) (0.5304) represents the value of the hybrid stacking mechanism. This process enhances performance by integrating deep representations and ensemble-based decision models. In general, the variations in the best models of AUROC are not large. The primary advantage of HA-PI-MADT is that it will be able to attain competitive discrimination without sacrificing multimodal integration. This is supported by the fact that its performance improved across additional evaluation metrics such as AUPRC, F1-score and calibration.

In a broader comparative perspective, the experimental results demonstrate that performance superiority is not evaluable based on a single metric like AUROC. While some baseline methods can achieve competitive discrimination performance, they often suffer inferior calibration quality, reduced ranking capability or unstable sensitivity–specificity balance. The proposed HA-PI-MADT framework achieves a more balanced performance profile across classification, ranking and reliability metrics simultaneously which is of particular importance in healthcare decision-support applications where reliable probability estimation and robust multimodal generalization are essential. The comprehensive assessment also confirms the feasibility, efficiency, and robustness of the proposed multimodal digital twin architecture.

The AUPRC comparison provides a larger profile of how effective the proposed HA-PI-MADT framework is in case of an imbalance between the classes, as depicted in Figure 3. AUPRC is not the same as AUROC as it indicates the ability of a model to correctly identify positive cases, which is most critical in healthcare. The findings indicate the highest AUPRC was offered by HA-PI-MADT (0.6388), which is higher than powerful baselines such as RandomForest_WearableOnly 0.6331) and XGBoost_EarlyFusionAll (0.6170). This improvement demonstrates the effectiveness of the hybrid adaptive multimodal design as it skillfully employs complementary information provided by wearable devices, electronic health records (EHRs), user-centered interfaces (UCIs), and imaging modalities. Unimodal models, such as RandomForest_WearableOnly, also perform well, indicating that the wearable modality has good predictive cues. However, most early-fusion techniques do not demonstrate much of an improvement, implying that straightforward multimodal fusion may introduce noise due to the lack of fit between the data sources. HA-PI-MADT, in its turn, addresses this issue through adaptive fusion and prototype-based alignment that helps to improve the performance of ranking. It is also clear that hybrid stacking is a significant component of the proposed framework, as the deep-only variant (HA-PI-MADT_DeepOnly) has a significantly smaller AUPRC. Overall, the figure demonstrates that HA-PI-MADT does not only integrate multimodal data effectively but also has the highest performance in precision-recall space, and is thus ideal in real-world contexts of clinical decision support.

Calibration comparison reveals that the levels of probabilistic reliability of various models are very different. An example of this is performance measures, like the AUROC and AUPRC, with a higher value representing a better performance. ECE, however, is defined as the goodness of match with true results of predicted probabilities, where a lower value indicates a stronger calibration. As depicted in Figure 4, these results indicate that HA-PI-MADT has a low ECE of 0.1388 and is therefore well calibrated and can be relied upon to make decisions. Other models, such as EarlyFusion_MLP (0.0641) and HA-PI-MADT_DeepOnly (0.0982) have lower values of ECE but are also far poorer at classification and ranking accuracy. This demonstrates that calibration and predictive power are in a trade-off. HA-PI-MADT, on the other hand, balances calibration and predictive performance, outperforming some of the more powerful baselines, such as RandomForest_EarlyFusionAll (0.1535) and GradientBoosting_EarlyFusionAll (0.1769). This demonstrates that the hybrid framework proposed is effective in generating well-calibrated probabilities without sacrificing accuracy or ranking performance.

Figure 5 shows calibration curves that visually compare predicted probabilities to observed outcome frequencies for different models. The dotted diagonal line indicates the ideal calibration line, where the predicted probabilities are equal to the empirical frequencies of events. Points closer to this diagonal indicate better agreement between model confidence and observed outcomes. Deviations above the line indicate under-confident predictions and deviations below the line indicate over-confident predictions. The results show that HA-PI-MADT has competitive calibration behavior with relatively stable probability estimates over multiple probability regions. However, there are some baseline models, such as EarlyFusion_MLP and HA-PI-MADT_DeepOnly, with lower ECE values. Thus, the suggested framework should not be considered as universally better in calibration across every metric. Instead, HA-PI-MADT offers a more balanced reliability profile, with competitive calibration behavior, Brier score, negative log-likelihood, improved AUPRC, and stronger multimodal robustness. The calibration curves also suggest that the hybrid stacking mechanism helps to stabilize probability estimation compared with several deep-only and early-fusion configurations. The results indicate that the proposed calibration-aware multimodal architecture improves the overall reliability-oriented healthcare prediction while preserving a balanced discrimination and ranking performance.

In Figure 6, we see how the proposed HA-PI-MADT architecture uses learned weights to adaptively combine multimodal information. Results show that the UCI and wearable modalities have the highest median weights, which means that the information from these modalities consistently contributes to prediction. This suggests that structured clinical metrics and physiological data have important discriminative information in the dataset. The EHR modality has moderate weights with a large range of values, meaning that it is informative but not for all samples. This also highlights the importance of adaptive weighting, as EHR data can be very important in some instances and less important in others. In contrast, the imaging modality is least weighted with a low variability. This suggests that, in the current model, the image embeddings that are extracted have a smaller influence than the other modalities. This may be because we use light statistical image features rather than deep image embeddings. The presence of outliers in wearable and image weights suggests that the model varies the importance of each modality for each sample, which indicates that the adaptive fusion with modality awareness is successful. The figure shows that HA-PI-MADT is not equally biased towards learning from different modalities. Rather, it learns their importance depending on the context, which is one of the main factors for its superior performance and robustness.

Table 8. The Learned weights of the ensemble parts in the HA-PI-MADT framework that was suggested. The weights show how much each part adds to the final prediction, which shows how important meta-learning and hybrid fusion strategies are.

Component	Weight
weighted_meta_test	0.45
weighted_base_test	0.30
best_single_HA-PI-MADT_Deep	0.15
HA-PI-MADT_Deep	0.10

allows us to understand how the HA-PI-MADT decision framework works by showing the learned weights of the ensemble. It reveals that the meta-learning part of the HA-PI-MADT ensemble is the most important (0.45), as it is the key part of fusing predictions and modelling complex interactions between different modalities. The base ensemble also plays a significant role (0.30), which indicates that traditional ensemble learning is still important to improve the stability of the predictions and the generalization ability. The weights for the individual deep learning parts and HA-PI-MADT_Deep (0.10) are lower. Consequently, deep representations are important, but they are more effective when used together with other data that are not deep. This distribution highlights one of the key features of the proposed model: rather than maintaining a single architecture, HA-PI-MADT makes use of a hierarchical combination of three different approaches (meta-learning, ensemble methods, and deep representations). This allows it to strike a balance between stability, adaptability, and accuracy. The results also show that the meta-fusion approach is the key to getting better results, which confirms the improved results in the AUPRC and calibration measures.

The rounded feature importance values still retain the same insights for interpretation but are more visually and textually appealing. The results show that logit_HGB is still the most important feature, followed by descriptors of the prediction distribution like pred_min and pred_range, as shown in

Table 9. Feature importance scores from the HA-PI-MADT meta-learning framework, rounded to four decimal places. The table shows how many model outputs and statistical descriptors each add to the final prediction.

Feature	Importance
logit_HGB	0.0549
pred_min	0.0217
pred_range	0.0216
XGB	0.0156
logit_WEU_RF	0.0104
logit_RF	0.0104
logit_XGB	0.0069
pred_std	0.0048
GB	0.0035
logit_HA-PI-MADT_Deep	0.0022
pred_mean	0.0021
logit_ET	0.0019
HA-PI-MADT_Deep	0.0019
HGB	0.0016
RF	0.0012
WEU_RF	0.0012
logit_GB	0.0010
pred_max	0.0009
ET	−0.0000

. This indicates the importance of having strong-based learners and uncertainty-based statistics. The importance of ensemble logits (e.g., random forest and XGBoost variants) is still the same, which shows that diversity is important. However, the deep learning components’ effects are still relatively small, which shows that the framework is a hybrid of learning styles. The ET has almost zero value (−0.0000), indicating that some weak learners are not that important, which justifies the adaptive weights strategy.

The results of the importance of permutation help us understand, in a simple way, what makes the HA-PI-MADT framework successful. In Figure 7, we see that logit_HGB is by far the most important feature. Consequently, the calibrated prediction of the HistGradientBoosting model is the most important part of the meta-learning stage. After this, the distribution of predictions (pred_min and pred_range) is very important. This demonstrates that the model is taking good advantage of uncertainty-aware statistics from the ensemble. This demonstrates that HA-PI-MADT not only uses individual model predictions, but also the differences and the range of predictions to make the model more resilient. Tree-based ensemble outputs, like XGBoost and random forest logits, contribute slightly to the results, which shows how important it is to have a variety of models in the hybrid architecture. However, features related to the individual deep learning components, such as the HA-PI-MADT_Deep features and its logits are of lesser importance. This indicates that they are most useful in the ensemble.

The t-SNE visualization offers qualitative validation of the discriminative potential of the latent representations acquired by HA-PI-MADT, as shown in Figure 8. The figure shows that two clusters have formed, which correspond to the two classes. This shows that the proposed model effectively captures class-discriminative features in the latent space. There is some overlap between clusters, especially in areas with lower density. This is common for real-world medical data, where the classes are not necessarily well separated. The overall embedding pattern is clearly separated, which indicates that the model has learned good representations despite the challenges of multimodal fusion and weak alignment. Even more interesting is the clustering pattern, which shows how useful the proposed hybrid adaptive fusion strategy is for combining multimodal data. The HA-PI-MADT representation is more well-structured and denser than most early-fusion or unimodal methods. This demonstrates better feature learning and generalization. The figure shows that the proposed method produces well-separated and well-organized latent embeddings, which result in high classification and ranking performance scores.

4.3. Ablation Study

To better understand the contribution of each modality and component of the proposed HA-PI-MADT framework, we conduct an ablation study by removing or separating different data sources. This study aims to quantify the contribution of each modality on the predictive performance, ranking performance, and calibration. This study gives us deep insights into the benefits of multimodal fusion by analyzing different setups of the deep component and the importance of each data source to enhance the robustness and generalizability of the model. The ablation study provides us with valuable insights into the role of each modality in the HA-PI-MADT deep component. The multimodal model has the highest AUPRC (0.5479), which means that using all of the data modalities has the best ranking performance, as shown in

Table 10. An ablation study that looks at how different modality configurations affect the HA-PI-MADT deep component. Each variant takes away or separates certain modalities to see how they affect predictive performance, ranking ability, and calibration.

Model	AUROC	AUPRC	Accuracy	F1	Precision	Sensitivity	Specificity	ECE	Brier	NLL
HA-PI-MADT_Deep_all_modalities	0.5304	0.5479	0.5000	0.6667	0.5000	1.0000	0.0000	0.0982	0.2611	0.7173
HA-PI-MADT_Deep_no_ehr	0.5348	0.5438	0.5000	0.6667	0.5000	1.0000	0.0000	0.0873	0.2591	0.7129
HA-PI-MADT_Deep_no_uci	0.5028	0.5345	0.5000	0.6667	0.5000	1.0000	0.0000	0.0897	0.2602	0.7152
HA-PI-MADT_Deep_no_image	0.5284	0.5442	0.5000	0.6667	0.5000	1.0000	0.0000	0.0969	0.2610	0.7169
HA-PI-MADT_Deep_wearable_only	0.5048	0.5332	0.5000	0.6667	0.5000	1.0000	0.0000	0.0872	0.2583	0.7113

. The highest AUROC (0.5348) and best calibration (the lowest ECE = 0.0873) is achieved when the EHR (no_ehr) is removed. This indicates that adding the EHR data may improve the ranking performance, but it may also be noisy or variable and make discrimination and calibration slightly less optimal. The biggest decrease in performance on AUROC and AUPRC occurs when the UCI modality (no_uci) is excluded. This demonstrates that UCI data form one of the most critical data features. Removing the imaging data (no_image) leads to a small drop in performance, showing that imaging data provide additional rather than core information. The wearable-only (no_nonwearable) combination has the lowest AUROC and AUPRC, which means that using only one of these data sources may not be sufficient for prediction. However, it has the smallest Brier score and NLL, which indicates that it has stable probabilistic predictions, even as it does not discriminate as well. It is interesting to note that all of the configurations have the same accuracy (0.5), F1 (0.6667), and sensitivity (1.0) scores, but no specificity. This could mean that there is a strong class imbalance or thresholding effect, where the model mostly predicts the positive class. This makes it even clearer that using ranking and calibration metrics to measure performance is better than just using accuracy. The results show that multimodal integration is necessary, with UCI and EHR being very important and imaging and wearable data making things even better. The results confirm that HA-PI-MADT is a hybrid multimodal framework that works best when different modes work together.

The ablation results, as shown in

Table 11. Ablation study assessing the role of essential components in the HA-PI-MADT framework.

Model	AUROC	AUPRC	Accuracy	F1	Precision	Sensitivity	Specificity	ECE	Brier	NLL
w/o_aux_modalities	0.6072	0.6120	0.6500	0.6316	0.6667	0.6000	0.7000	0.2392	0.2584	0.7427
w/o_deep_branch	0.5780	0.5843	0.6000	0.5745	0.6136	0.5400	0.6600	0.1769	0.2667	0.7630
w/o_image_memory	0.5588	0.6129	0.5700	0.5376	0.5814	0.5000	0.6400	0.1038	0.2538	0.7030
w/o_stacking	0.5304	0.5479	0.5000	0.6667	0.5000	1.0000	0.0000	0.0982	0.2611	0.7173
HA-PI-MADT (Full)	0.6072	0.6388	0.6400	0.6327	0.6458	0.6200	0.6600	0.1388	0.2448	0.6832

, show very clearly how important each part of the HA-PI-MADT framework is. The full model works best overall, especially in AUPRC (0.6388), F1-score (0.6327), and calibration metrics (lowest Brier = 0.2448 and NLL = 0.6832). This shows that the proposed hybrid design works well. The most significant drop in performance happens when the stacking module is removed (w/o_stacking). This is shown by big drops in AUROC and AUPRC. This shows that meta-learning and stacking are the main factors that affect performance, as they allow different model outputs to be combined in a useful way. Deprivation of the deep branch (w/o deep branch) also results in large drops in all measures, indicating deep representations do contribute valuable additional features. Deactivating the image memory module (w/o image memory) also complicates distinguishing between things and ranking them, indicating that information about imaging is beneficial to multimodal learning, but is not a significant contribution by itself. The one without auxiliary modalities (w/o aux modalities) is also interesting as it is the most accurate and specific and its AUROC is competitive. However, it is much less well-calibrated (maximum ECE and NLL), indicating that simple models can perform well on threshold-based metrics, but not on probabilistic metrics. The findings indicate that the parts contribute differently with the largest contribution being made by stacking, then the deep branch and multimodal inputs. The overall HA-PI-MADT model balances the best between discrimination, ranking, and calibration, thus validating the proposed architecture as a robust and fully integrated multimodal system.

The ablation test results make it evident how the performance of each section of the HA-PI-MADT framework can influence the overall performance, as illustrated in Figure 9. Full architecture achieves the highest AUROC (0.6072) indicating that the full hybrid structure is effective. The greatest performance decrease (0.5304) occurs when the stacking module is eliminated. This indicates that the primary cause of discrimination performance is meta-learning. This indicates the significance of integrating the results of various models rather than merely relying on a single predictor. The deep branches omission (w/o deep branch) also leads to a considerable decrease in AUROC (0.5780), indicating that deep representations contribute value to the ensemble. Eliminating the image memory module (w/o image memory) also reduces performance (0.5588) and this indicates that imaging features are useful in enhancing predictive ability, though not as effective as stacking. Interestingly, the model without auxiliary modalities (w/o_aux_modalities) has an equivalent of the full model (0.6072) in terms of the AUROC. This implies that these modalities do not make such a significant impact on the ability of the model to distinguish between things. However, the table indicates that this variant has poorer calibration measures, indicating that auxiliary modalities play a more significant role in probabilistic reliability than classification accuracy.

Other insights from the ablation experiments are threshold sensitivity and class imbalance behaviors. In particular, the HA-PI-MADT_DeepOnly configuration yields a sensitivity of 1.0 and a specificity of 0.0, revealing a bias towards over-prediction of the positive class. This behavior is related to the effects of threshold selection sensitivity, limited training data, weak multimodal alignment, and class imbalance, as often seen in deep multimodal healthcare learning. This behavior improves recall-oriented metrics but at the same time increases false positive rates and reduces practical clinical reliability. These results indicate that future extensions of the framework should consider threshold optimization, calibration-aware threshold selection and cost-sensitive learning strategies to improve the sensitivity–specificity balance.

The experimental evaluation generally shows that the proposed HA-PI-MADT framework is effective at overcoming some of the main challenges in heterogeneous multimodal healthcare learning. The proposed framework enables an effective trade-off among discriminative capability, ranking quality, and probabilistic reliability via the combination of modality-aware adaptive fusion, prototype-based weak alignment, hybrid ensemble stacking, and calibration-aware optimization. Compared with traditional early-fusion or deep-only multimodal methods, the proposed architecture can have a stable performance even in heterogeneous multimodal scenarios. The results show that reliability-aware multimodal digital twin systems can be a promising direction for future intelligent healthcare decision-support frameworks.

Although encouraging experimental results, the evaluation is currently based mainly on a single train–test split. Thus, further statistical validation would contribute to the robustness of the conclusions. Future work will include repeated stratified cross-validation, bootstrap confidence interval estimation and formal statistical significance testing, which will enable more rigorous comparisons between competing multimodal healthcare prediction frameworks. Statistical tests such as DeLong’s test for AUROC comparison and bootstrap confidence intervals for AUPRC and F1-score may provide more convincing evidence about consistency and significance of the observed differences in performance.

5. Discussion

The findings of the experiments indicate that the proposed HA-PI-MADT model is a powerful and practical method to forecast multimodal healthcare via a hybrid meta-learning architecture by integrating various types of data sources. The model has a high balance of discrimination, ranking performance and probabilistic calibration on various measures of evaluation. This demonstrates that it is an excellent option in the clinical decision support of a real-life situation. One of the most significant results of the ablation study is that the stacking (meta-learning) part is the most significant to enhance performance. The prominent decrease observed when stacking is removed confirms that aggregating diverse model prediction outputs is essential in explaining complex cross-modal correlations. This finding demonstrates that the power of HA-PI-MADT does not lie in a single model but in its ability to integrate various predictors that complement each other. The deep learning branch is also very important as it provides high-level feature representations. It does not contribute much by itself, but when it is combined with the ensemble, it enhances performance. This demonstrates that deep features are most effective with classical machine learning models. The memory image module also provides additional context, which indicates that the application of the imaging data is useful even with its seemingly small impact. Surprisingly enough, the findings indicate that the removal of auxiliary modalities does not affect the performance of the model considerably in terms of AUROC but has a significant negative impact on the calibration measures such as ECE and NLL. This implies that these modalities might not considerably influence the boundaries of classification but are necessary to increase the prediction reliability and uncertainty estimation that are crucial in clinical settings. These findings are supported by the feature importance analysis, which displays the importance of ensemble logits and statistics of prediction distribution (such as pred_min and pred_range). The model not only utilizes predictions but also the uncertainty and variability among models. This indicates that the present uncertainty-conscious design is effective. Such behavior is consistent with the improved calibration behavior of the full model. Another observation that the t-SNE visualization makes is that HA-PI-MADT learns well-structured latent representations, and the classes are distinctly separated. This demonstrates that hybrid architecture is useful in transforming multimodal information into a discriminative feature space, which can be directly associated with better performance. While these results are promising, there are some limitations to be addressed. First, deep features can be harder to adapt to when there is a large amount of data, especially for some modalities. Second, the inherent sensitivity/specificity trade-off between some of these ablation conditions suggests that optimization of the classification threshold or cost-sensitive learning could help it be improved.

Another important feature of the proposed HA-PI-MADT framework is its scalability and extensibility to different healthcare domains. The proposed architecture is modality-agnostic and based on adaptive multimodal fusion and can theoretically be generalized to a wide range of disease categories such as respiratory diseases, cardiovascular disorders, neurodegenerative diseases, metabolic syndromes, and chronic inflammatory conditions if appropriate multimodal datasets are available. In particular, the adaptive fusion mechanism allows the framework to dynamically adjust the relative contribution of heterogeneous modalities based on data relevance and quality, which is suitable for complex multimodal clinical environments. However, additional disease-specific validation with synchronized clinical datasets would be required before real-world implementation in specialized healthcare applications.

The experimental results, in general, reveal that the proposed HA-PI-MADT framework has a balanced multimodal reliability profile rather than optimizing a single evaluation metric alone. Although some unimodal baselines yield similar discrimination performance, the proposed framework demonstrates better ranking quality, probabilistic reliability and multimodal robustness in heterogeneous multimodal conditions. The adaptive fusion mechanism is important to stabilize the contributions of modalities when different kinds of modalities are of inconsistent quality or are not perfectly aligned. In addition, the calibration-aware optimization strategy enhances reliability-oriented prediction behavior, which is particularly important in healthcare decision-support settings where probability estimates may directly affect clinical interpretation and downstream risk assessment.

Another important observation is that the improvements in the multimodal performance are highly dependent on the quality of the modality interaction and the contextual consistence. Results show that simple early-fusion techniques for combining heterogeneous modalities in a static fashion do not always result in improvements of the predictive performance, especially under weak alignment conditions. However, the presented adaptive fusion architecture dynamically controls the importance of modalities based on their feature relevance and reliability, leading to a more stable multimodal integration. Such behavior is particularly important in real healthcare applications, where modalities could be partially missing, noisy, or not consistently synchronized.

From a higher-level healthcare-AI perspective, the proposed framework suggests that the trade-off between calibration, ranking robustness, and multimodal reliability might be more clinically meaningful than optimizing discrimination metrics alone. The proposed framework obtains better AUPRC, lower Brier score and lower negative log-likelihood, which suggest a stronger reliability-oriented behavior under heterogeneous multimodal conditions. The AUROC performance is comparable to some strong wearable-only baselines. These results further emphasize the importance of calibration-aware multimodal learning in trustworthy healthcare prediction systems.

The experimental findings suggest that the primary advantage of the proposed multimodal fusion strategy lies in improving prediction reliability and ranking-oriented performance rather than dramatically increasing discrimination-oriented metrics such as AUROC. This outcome is consistent with the heterogeneous and partially aligned nature of the integrated modalities, where wearable physiological features provide the strongest direct discriminative signal, while auxiliary EHR, imaging, and population-level prototype modalities contribute complementary contextual information that enhances calibration stability and probabilistic consistency.

6. Conclusions

This paper proposes the HA-PI-MADT to tackle the difficulties of fusing heterogeneous healthcare data. In this paper, we propose a unified healthcare digital twin-inspired framework with modality-aware feature extraction, adaptive multimodal fusion, deep representation learning, hybrid ensemble stacking and calibration-aware optimization. The framework effectively combines complementary multimodal information from wearable sensor signals, electronic health records (EHRs), CT/MRI imaging representations, and population-level risk prototypes, and is robust to modality imbalance and weak alignment. Experimental evaluation showed that the proposed HA-PI-MADT achieved competitive or better results than unimodal, multimodal, and ensemble baselines on several evaluation metrics such as AUROC, AUPRC, F1-score, Brier score, and negative log-likelihood. In particular, the proposed framework achieved the highest AUPRC (0.6388) and F1-score (0.6327), with an improvement of probabilistic calibration and reliability. Results show that adaptive multimodal fusion and calibration-aware hybrid ensemble learning is effective for trustworthy healthcare prediction and digital twin applications.

The proposed approach applies a stacking-based meta-learning approach to integrate deep representation learning with ensemble approaches. This is what allows for the effective combination of complimentary modalities. Experimental findings indicate that HA-PI-MADT is a fair trade-off between multiple evaluation metrics, including discrimination (AUROC), ranking performance (AUPRC), and probabilistic calibration (ECE, Brier, and NLL). The final model is always better than both the baseline and ablation models, and this demonstrates the effectiveness of hybrid architecture. According to the ablation study, the most significant aspect is the stacking mechanism due to its ability to combine various model outputs to achieve optimal performance. The image memory module and deep learning branch introduce additional helpful data, and the auxiliary modalities are used to assist in calibration and uncertainty estimation. The statistics of ensemble logits and prediction distributions are the most significant variables as analyzed. This implies that predictions and signals that are conscious of uncertainty are used in this model. Latent space visualization also demonstrates that HA-PI-MADT learns well-structured representations capable of distinguishing between things, which supports its high power to make predictions. Although these are encouraging results, some issues remain. The size of the dataset can be a limiting factor to the performance of deep components, and certain configurations exhibit sensitivity and specificity imbalance. This implies that it might require superior thresholding or cost-sensitive optimization. Moreover, the computational complexity of the hybrid framework might render it difficult to scale to limited resources. We will also enhance the framework by adding larger and more diverse multimodal data sets in the future, calibrated and uncertainty-quantified with more sophisticated techniques, and more efficient architectures to reduce computational costs. Developing the model to accommodate missing modalities, as well as real-time streams of data, will also be particularly important when applying the model in the real-world clinical environment. HA-PI-MADT is a robust, simple, and scalable method of predicting multimodal healthcare outcomes. It demonstrates the usefulness of integrating meta-learning, deep representations, and uncertainty-aware modeling within a system.

Despite the promising multimodal prediction performance achieved by HA-PI-MADT, several limitations should be acknowledged. First, the framework relies on independently collected heterogeneous multimodal datasets rather than fully synchronized longitudinal patient-specific records. Consequently, the proposed approach is better interpreted as a digital twin-inspired healthcare representation framework rather than a strict real-time physiological digital twin system. Future work will focus on validating the proposed adaptive multimodal fusion strategy using fully synchronized multimodal clinical cohorts and longitudinal patient trajectories to enable more realistic patient-specific digital twin modeling. Future work may explore larger longitudinal healthcare datasets, real-time synchronization of digital twins, uncertainty-aware multimodal transformers, and explainable calibration-aware architectures to improve further clinical interpretability and reliability. Moreover, the integration of federated and privacy-preserving multimodal learning approaches could improve the utility of healthcare digital twins in distributed clinical environments. All in all, the proposed HA-PI-MADT framework lays a solid foundation for future trustworthy and adaptive multimodal healthcare intelligence systems. Another limitation of the current study is that the experimental evaluation was mainly performed with a single train–test split which may not fully capture the performance variability under different sampling conditions. Moreover, some ablation configurations showed sensitivity–specificity imbalance, indicating the potential effects of thresholding and class imbalance in deep multimodal learning settings. Therefore, future work will investigate repeated cross-validation, statistical significance testing, adaptive threshold optimization, and cost-sensitive multimodal learning strategies to further enhance robustness and clinical reliability.