A Multimodal Dual-Stream Cross-Attention Deep Learning Framework for Diabetic Foot Ulcer Classification

Abstract

Finding diabetic foot ulcers (DFUs) early and accurately is essential for improving patients’ quality of life and lowering the risk of amputation. RGB images, commonly used in automated DFU detection, have limitations such as lighting variations, color inconsistencies, and inability to directly reflect physiological information. Background/Objectives: Although thermal images can capture temperature anomalies associated with inflammation and circulatory disorders, they cannot provide consistent performance due to their low spatial resolution and limited availability in clinical datasets. Furthermore, the lack of paired RGB–thermal image pairs makes it difficult to develop effective multimodal deep learning models. Methods: This study proposes a two-stage multimodal deep learning approach to overcome these limitations. In the first stage, an RGB2T-cGAN (RGB to Thermal cGAN) model based on pix2pix was designed to generate synthetic thermal representations from RGB images that resemble clinical patterns, thereby addressing the missing modality problem. In the second stage, the Multimodal Dual-Stream Multi-Head Cross-Attention (MDS-MHCA) classifier model was developed, which processes DFU RGB and generated synthetic thermal images through separate streams, enabling the dynamic modeling of complementary information across modalities. Results: The proposed MDS-MHCA model achieved 99.06% accuracy, 99.09% recall, and 99.06% F1-score on the test set, demonstrating a clear advantage over models based solely on RGB (91.51% accuracy) or thermal (96.23% accuracy) modalities. Furthermore, patient-based 10-fold GroupKFold cross-validation results demonstrate that the model offers high generalization capability across different patient groups, with an average accuracy of 96.49 ± 1.04 and an AUC value of 0.9927 ± 0.0067. Conclusions: The findings reveal that the proposed approach, through the integration of synthetic thermal information and cross-attention-based multimodal fusion, overcomes the fundamental limitations of single-modality-based systems and offers a DFU detection system that is more robust and reliable and holds potential for integration into clinical decision support systems.

1. Introduction

Diabetes is a chronic metabolic disease with an increasing prevalence worldwide, leading to numerous health problems. The World Health Organization projects that the number of people with diabetes will double by 2030 [1]. Consequences of this disease include visual impairment; impaired blood flow; partial paralysis; stroke; sensory loss (neuropathy); and one of its most dangerous complications, diabetic foot ulcers (DFUs). DFUs occur when small wounds on the foot go unnoticed and grow over time, becoming infected and developing into ulcers [2]. In current clinical practice, DFU assessment is mostly based on manual methods. This manual assessment includes steps such as visual examination by the physician; measurement of ulcer size using planimetry; scoring of the area around the wound based on color, texture, and edema findings; and, if necessary, expert interpretation via photographs [3,4]. However, these methods are time-consuming and can lead to overlooking small lesions, especially in the early stages. As a result, small, undetectable wounds ulcerate and can lead to limb loss (amputation) in advanced stages. It is reported that a significant proportion of lower extremity amputations due to diabetes worldwide are caused by DFUs and that a diabetic patient loses a foot for this reason every 20 s [5]. This situation makes the rapid and reliable detection of foot ulcers critical both for patient quality of life and for reducing the financial burden on health systems [4]. Recently, Deep Learning (DL) based image processing approaches have achieved remarkable successes in the field of medical image analysis. Some of these include high performance in the detection and classification of DFUs [6,7], and skin lesions and cancer [8,9,10].

Specifically for DFUs, open competition datasets such as DFUC 2020 have accelerated model comparisons and popularized DL-based approaches in real-world problems [11]. However, most studies in the literature focus solely on RGB or thermal images [7,12]. Although RGB images are valuable, they are affected by external factors such as camera quality, color, lighting conditions, and shading, which can reduce classification performance [8]. On the other hand, while thermal images provide important clues by capturing heat anomalies associated with inflammation and circulatory disorders, in some thermal systems, surface tissue characteristics may not be represented as clearly as in RGB due to resolution and detail limitations; the potential and limitations of thermal imaging in detecting early risk in diabetic feet have been discussed in numerous studies [12,13]. Therefore, approaches based solely on RGB or thermal information may be limited in terms of generalization power in some cases [14]. Furthermore, the absence of matched RGB and thermal pairs in most DFU datasets is a factor that complicates the development of multimodal models [12,13].

This study proposes a multimodal DL-based classification architecture that utilizes and evaluates both RGB and thermal images. The proposed framework aims to achieve more reliable and accurate classification by combining the strengths and complementary features of the two distinct image sources, particularly in scenarios where RGB and thermal images exhibit limited performance [15,16]. In this context, by combining the color distribution and shape information provided by RGB with the heat anomalies provided by thermal images, pathological changes can be analyzed at both the macro and micro scales. To this end, this study proposes a pix2pix-based conditional GAN (cGAN) architecture to obtain thermal images from RGB images in the Kaggle DFU dataset, which is frequently used in the literature. The cGAN approaches are effectively used in the literature to generate pseudo maps from RGB images [16]. While the generator network was designed with an encoder–decoder structure, the discriminator network was trained using the PatchGAN approach, which controls local texture and edge continuity. The obtained pseudo-thermal images were fed into a multimodal classification network along with RGB images; the Multi-Head Cross-Attention (MHCA) mechanism was used in DL layers to model inter-modality relationships. Attention-based multimodal fusion has been shown in the literature to enhance discriminative features in medical imaging [17,18].

This study comprehensively addresses two fundamental limitations commonly encountered in the literature regarding DFU classification: (i) the absence of paired RGB and thermal images in clinical datasets and (ii) the limited generalization ability of models based on single modalities. In this context, the missing modality problem was addressed by developing a pix2pix-based conditional Generative Adversarial Network (GAN) model that generates pseudo-thermal representations from RGB images in the first stage. The generated synthetic thermal representations were evaluated as a complementary information source that enriches the multimodal learning process, in addition to aiming to replace actual thermal measurements. In the second stage, a novel Multimodal Dual-Stream Multi-Head Cross-Attention (MDS-MHCA) classification architecture was proposed, which processes RGB and synthetic thermal images through separate streams and dynamically models inter-modality interactions using the MHCA mechanism. Experimental results, supported by patient-based GroupKFold Cross-Validation (CV) and statistical significance tests, demonstrate that the proposed approach offers more stable and generalizable performance compared to models based on single modalities. In this regard, the study contributes at the application level to the DFU classification literature with its integrated structure that addresses attention-based multimodal fusion alongside synthetic modality generation. The main contributions of the study are as follows:

• The lack of paired RGB–thermal images, which is common in DFU datasets, was addressed using a cGAN-based approach that generates synthetic thermal maps from RGB images.
• Using a pix2pix-based conditional GAN, clinically interpretable pseudo-thermal maps were generated that highlight ulcer areas while preserving structural integrity.
• A novel dual-stream DL architecture has been proposed that processes RGB and synthetic thermal images in separate streams and models intermodal interactions using the MHCA mechanism.
• The proposed multimodal approach has provided a significant performance improvement in terms of accuracy, sensitivity, and F1-score compared to models trained with only RGB or only thermal inputs.
• The model’s generalization ability across different patient groups was validated using patient-based 10-fold GroupKFold CV; the architecture, which does not require actual thermal hardware, offers a feasible solution for clinical use.
• The performance improvement provided by the multimodal approach compared to single RGB-based models was validated with statistical significance tests.

Accordingly, the contribution of this work lies in its application-oriented framework design rather than in proposing a new learning paradigm. The Section 2 of this study provides a detailed review of existing literature on DFU detection; the Section 3 presents the DL architectures, multimodal fusion strategies, and GAN-based image-to-image transformation methods that form the theoretical foundation of this study. The Section 4 explains the proposed two-stage method (thermal image generation with GAN and Dual-Stream Cross-Attention-based multimodal classification) in detail; the Section 5 reports the experimental results, comparative analyses, and CV findings; and the Section 6 concludes this study by evaluating the results obtained.

2. Literature Review

Most studies on the automatic detection of DFU have been conducted using the widely adopted Kaggle DFU dataset in the literature [19,20,21]. Alzubaidi et al. [20] proposed a convolutional neural network (CNN) architecture specifically designed for DFU detection, named DFU-QUTNet, by combining the Kaggle DFU and QUT datasets. The study demonstrated that preferring a problem-specific architectural design over general-purpose transfer learning models enables the learning of more discriminative features. The achieved accuracy of 99.3% indicates that domain-specific architectures can deliver high performance in problems characterized by complex textural structures such as DFU. However, since the study relies on a single RGB modality, its generalization capability under different clinical conditions is addressed only to a limited extent. Nagaraju et al. [5], integrated an Inception-ResNet-v2–based DL architecture with the Sparrow Search Optimization (SSO) algorithm to optimize both the feature extraction and classification stages. The accuracy of 99.29% obtained through 10-fold CV demonstrates that heuristic optimization algorithms can contribute to more effective tuning of model parameters in DFU classification problems. Nevertheless, the computational cost of the optimization process and its applicability to real-time clinical scenarios are discussed in a limited manner.

Harahap et al. [22], conducted a comparative analysis of widely used CNN architectures, including ResNet152V2, ResNet101V2, MobileNetV2, and VGG19. Experimental results showed that ResNet152V2 offered the most successful performance, with 99% accuracy. This finding reveals that networks with deep and wide receptive areas can better represent complex textural distortions and ulcer boundaries in DFU images. However, all architectures used in the study were based on single RGB input, and the effect of complementary information that can be obtained from different modalities was not considered. Silva Dos Santos et al. [23] investigated the effect of model depth on DFU diagnosis by performing fine-tuning on multi-level CNN architectures. The study achieved 97.8% accuracy and emphasized that multi-layered representations play a significant role, especially in distinguishing different ulcer severity levels. While this approach reveals the hierarchical feature extraction power of deep networks, additional strategies to reduce the risk of dataset-dependent learning were addressed to a limited extent. Thotad et al. [24] focused on the class imbalance problem commonly encountered in DFU datasets and applied comprehensive data augmentation strategies. Using EfficientNet-B1 and EfficientNet-B3 architectures, an accuracy of 100% was achieved on certain sub-test sets, while the overall accuracy was reported as 98.97%. The study demonstrates that data augmentation can significantly improve performance in DFU classification. However, the extent to which these augmentation-based gains represent the variability present in real clinical data was not explicitly discussed. Verma [25] combined data augmentation and class balancing techniques with the EfficientNet-B0 architecture and achieved an accuracy of 96.4%. The study shows that imbalances in data distribution directly affect DFU classification performance and that appropriate balancing strategies can substantially enhance model effectiveness. Nevertheless, due to its reliance on a single modality, the study does not consider the potential advantages offered by multi-source data.

Özdaş [26] addressed the automatic detection of DFU by evaluating RegNet- and EfficientNet-based DL architectures on the Kaggle DFU dataset. In the study, model performance was improved through the use of transfer learning and data augmentation strategies. The findings indicate that deep CNN-based approaches can achieve high accuracy and strong generalization capability in DFU classification. However, the exclusive use of RGB images limits the exploration of potential gains from multimodal approaches, such as the integration of thermal images or clinical data.

Toofanee et al. [27] investigated DFU analysis from a longitudinal perspective using similarity learning. In this study, a Siamese neural network architecture was employed to model visual similarity between DFU images acquired at different time points, with the primary aim of assessing disease progression rather than performing direct classification. The study relies solely on RGB images and focuses on similarity-based evaluation and visualization. Using RGB images, the proposed approach demonstrated that similarity-based embeddings could capture meaningful temporal changes in DFU conditions.

Fadhel et al. [28] proposed a lightweight EfficientNet-B0-based model for real-time clinical applications and reported an accuracy of 97.8%. The study demonstrates the feasibility of low-complexity models in clinical settings. Nevertheless, the potential limitations of lightweight architectures in handling complex cases were not discussed in detail. Table 1 summarizes the classification performances and evaluation protocols of studies conducted on the Kaggle DFU dataset, which consists exclusively of RGB images. Studies employing real thermal imaging are based on different datasets and acquisition protocols and are therefore discussed separately in the literature review rather than included in this table. Overall, the reported results indicate that further improvements are still required for DFU classification.

Table 1. Summary of studies using the Kaggle DFU dataset.

Reference	Year	Train–Validation–Test Ratio	Classification Model	Best Acc (%)
[20]	2020	8:2 (train–test)	QUTNet + SVM	94.5%
[22]	2022	7:2:1	ResNet152V2	99%
[5]	2023	10-Fold CV	Inception-ResNet-v2 + SSAE	98.98%
[24]	2023	6:2:2	EfficientNet-B0 to B5	Avg ≈ 98.97%
[25]	2024	8:2 + external clinical test	EfficientNetB0 + CNN	96.4%
[23]	2025	7:1:2	Feature-level fusion of VGG-16 and VGG-19	95.91%
		5-Fold CV		95.93%

Studies on foot ulcer detection have been conducted using different datasets collected in clinical settings outside the DFU dataset. Yap et al. [7] developed a DL-based classification system using DFU images obtained from different clinical centers and drew attention to the problem of generalization between datasets. Cassidy et al. [11] presented a multi-center DFU dataset as part of the DFUC 2020 competition, evaluating model performance on data closer to real clinical application conditions. Wu et al. [12] and Panamonta et al. [13] focused on thermal imaging-based foot ulcer analysis, demonstrating that inflammation and circulatory disorders can be detected early through heat maps. Mrowiec et al. [14] reviewed the advantages and limitations of thermal imaging in diabetic feet, emphasizing that thermal information alone may not be sufficient but increases clinical value when used in conjunction with other modalities. These studies reveal that different data sources and imaging techniques provide important information in foot ulcer detection, but approaches based on single modalities may be limited in terms of generalization and reliability.

While these limitations persist in the DFU literature, interest in holistic approaches that combine different data sources has grown in recent years across medical image analysis and clinical decision support systems [15,29]. Particularly in studies that evaluate image-based and clinical information together, more reliable and stable decision mechanisms have been reported compared to methods based on single data sources. This trend has been supported in the literature by studies using RGB, thermal, MR, CT, and clinical data together; attention and cross-attention-based multimodal fusion methods have significantly improved performance by dynamically learning the relationships between directly accessible and real modalities belonging to the same sample [10,15,30]. While these studies assume multimodal data is available and combine it at the feature or attention level, this study generates GAN-based pseudo-thermal representations from RGB images in a scenario where thermal information is not directly available and incorporates this synthetic modality into the multimodal learning process alongside the original RGB data.

The vast majority of existing studies in the literature rely solely on RGB or thermal images. This study addresses the missing modality problem by combining the complementary properties of RGB and thermal streams in a single architecture and provides more stable results compared to single modalities thanks to multimodal fusion. In this respect, the study provides an innovative contribution to the DFU classification literature with its integrated structure that addresses both modality generation and modality fusion.

3. Background

3.1. CNN

CNN is deep neural network architectures consisting of convolution, pooling, and fully connected layers, widely used especially in image processing applications. CNNs are DL models designed to model spatial patterns and local dependencies [31]. The convolution operation, which is the basic building block of CNNs, is defined in Equation (1). The equation also denotes $X$ as the multi-channel input tensor representing the input image; $W$ as the learnable convolution filters; and $b_k$ as the bias term for the kth output channel. $i$ and $j$ denote spatial position indices, $c$ denotes the input channel, and $k$ denotes the output channel. The resulting $Z$ output from the equation constitutes the first stage of the model’s feature hierarchy. The sequential stacking of convolutional layers enables representation learning at different levels of abstraction [32].

$$Z_{i,j,k}=X_{i+m,j+n,c}\hspace{0.17em}{W}_{m,n,c,k}+b_k$$

(1)

Increasing the depth of neural networks on CNNs enables the learning of more complex representations, but it also leads to problems such as gradient vanishing or optimization difficulties [33]. Thanks to skip connections, direct information transfer between layers is achieved, gradient backpropagation is facilitated, and deeper networks can be trained more effectively. This feature contributes to learning more stable and discriminative features in medical images with low contrast, noise, and structural ambiguity [33]. Therefore, residual connections were used between CNN layers in the proposed model.

3.2. Multi-Head Cross-Attention and Dual-Stream Architecture

Attention mechanisms are structures that enable a model to dynamically learn the relationships between different feature representations and enhance the performance of modern DL architectures [34]. In the cross-attention mechanism, query vectors are obtained from one modality, while key and value vectors are derived from a different modality. This allows the model to learn which regions in one modality are related to information in another modality. The basic cross-attention operation is defined by measuring the similarity between the query, key, and value matrices, as shown in Equation (2).

$$\mathrm{Attention}\left(Q,K,V\right)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\mathrm{\top }}}{\sqrt{d_k}}}\right)V$$

(2)

In Equation (2), $Q$ represents query representations obtained from the first modality, while $K$ and $V$ represent key and value representations obtained from the second modality. This structure enables the model to highlight complementary information across modalities.

In the MHCA mechanism, this process is performed in parallel across different heads, and each head learns different inter-modality similarity relationships. The obtained head outputs are combined as in Equation (3) to form the final representation. This design increases the model’s expressive power and generalization capacity by enabling it to learn independent similarity relationships in different representation spaces [34].

$$\mathrm{MHA}\left(Q,K,V\right)=\mathrm{Concat}\left(hea{d}_1,\dots ,hea{d}_h\right)W_O$$

(3)

Dual-stream architectures are structures that aim to process representations obtained from different feature sources through independent streams and then combine them into a common representation [29]. This structure enables more discriminative representations to be obtained compared to models learned through a single stream. In general, a dual-stream architecture is expressed as in Equation (4).

$$F_1=f_1\left(X_1\right),F_2=f_2\left(X_2\right)$$

(4)

In Equation (4), $X_1$ and $X_2$ represent different feature sources; $f_1\left(\cdot \right)$ and $f_2\left(\cdot \right)$ represent the DL-based feature extraction functions for each flow. The obtained feature representations $F_1$ and $F_2$ are then combined into a common representation through a fusion process as described in Equation (5).

$$F_{\mathrm{fusion}}=\Phi \left(F_1,F_2\right)$$

(5)

This fusion process can be performed through concatenation, element-wise summation, or attention-based interaction mechanisms. In particular, attention-based approaches enable context-aware modeling of the relationships between representations extracted from the two streams, thereby facilitating more effective information integration [29].

3.3. Multimodal Fusion Strategies

Multimodal fusion techniques aim to combine information obtained from complementary data sources, such as audio, image, or different representations of images, into a unified representation. These approaches enable more accurate predictions, reduced uncertainty, and healthier modeling of complex clinical patterns by overcoming the limited representational power of individual modalities [35]. In the medical imaging, multimodal fusion enables more accurate detection of diseases by allowing the simultaneous evaluation of anatomical (RGB) and functional (thermal, MR, and PET) information. The literature commonly describes three basic fusion strategies: early fusion, late fusion, and attention-based fusion [36].

In the early fusion approach, data obtained from modalities are combined at the raw pixel level or immediately after low-level features are extracted to form a single common representation. In Equation (6), $X^{\left(x\right)}$ and $X^{\left(y\right)}$ represent the input data or low-level feature representations belonging to different modalities; the $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left(\cdot \right)$ operator denotes the concatenation of these representations in the channel dimension. The resulting $X_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}$ representation is processed through a single network and used for classification or prediction tasks. While this approach has low computational cost, it can compromise representational integrity in heterogeneous clinical data due to the direct transfer of statistical differences between modalities into the fusion [37].

$$X_{\mathrm{fusion}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left(X^{\left(x\right)},X^{\left(y\right)}\right).$$

(6)

In late fusion, each modality is processed independently through separate networks to extract high-level feature representations, which are then classified. The final decision is obtained by combining the individual model predictions through strategies such as voting, averaging, or weighted summation. In Equation (7), $F_x$ and $F_y$ denote the high-level features obtained from modality-specific networks, while $C$ represents the decision. Although late fusion preserves modality-specific information in separate streams, cross-modal interactions are modeled only at the fusion stage, which may limit the learned dependencies in the joint representation [37].

$$C=\mathrm{A}\mathrm{g}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(F_x,F_y\right).$$

(7)

Attention-based fusion approaches developed to overcome these limitations model the interaction between modalities using a learnable attention mechanism. In this method, each modality can dynamically update its own representation conditionally based on information from the other modality. A generalized attention-based fusion is defined in Equation (8), where the $\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(\cdot )$ operator adaptively integrates features from multiple modalities by capturing their interdependencies. This structure enables context-aware learning of cross-modal relationships, leading to more stable and discriminative representations, particularly in heterogeneous clinical data [29].

$$F_{\mathrm{fusion}}=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(F_x,F_y\right),$$

(8)

3.4. Generative Adversarial Networks and Image-to-Image Translation

GAN models are competitive artificial neural network architectures in which a generator ($G$) and a discriminator ($D$) network are trained together through an adversarial optimization process [38]. The generator network attempts to produce synthetic samples that mimic the real data distribution, while the discriminator network attempts to distinguish between real and generated samples. Equation (9) shows the loss of the GAN model.

$${min}_G{max}_D {\mathcal{L}}_{\mathrm{G}\mathrm{A}\mathrm{N}}\left(G,D\right)={\mathbb{E}}_{x\sim p_{\mathrm{data}}}\left[\mathrm{l}\mathrm{o}\mathrm{g}D\left(x\right)\right]+{\mathbb{E}}_{z\sim p_z}\left[\log \left(1-D\left(G\left(z\right)\right)\right)\right]$$

(9)

In Equation (9), $p_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ represents the actual data distribution, while $p_z$ represents the prior distribution in the noise space. The generator network $G$ aims to convert $z$ vectors sampled from the latent space into realistic samples, while the discriminator network $D$ attempts to distinguish whether the samples presented to it are from the true data distribution or generated by the generator. This min–max optimization problem aims for $D$ to produce high outputs for real samples and low outputs for generated samples, while $G$ strives to approximate the real data distribution as closely as possible with its generated samples in a way that deceives $D$. Thanks to these features, GAN-based approaches are widely used in many fields, such as medical image synthesis, modality conversion, super-resolution, noise reduction, and data augmentation, to produce synthetic images that are close to real data and clinically meaningful [39].

3.4.1. The Pix2Pix GAN Model

The pix2pix is an image-to-image translation model based on a conditional GAN (cGAN) [40]. The model aims to learn how to transform an image obtained from the source image domain into an image corresponding to the target image domain. This transformation process is modeled conditionally on the source image by the generator network $G$ [40]. Unlike traditional GANs, both the generator and discriminator networks are conditionally dependent on the source image. The conditional adversarial loss function used in pix2pix is defined as in Equation (10) [40].

$${\mathcal{L}}_{\mathrm{cGAN}}\left(G,D\right)={\mathbb{E}}_{x,y}\left[\mathrm{l}\mathrm{o}\mathrm{g}D\left(x,y\right)\right]+{\mathbb{E}}_{x,z}\left[\mathrm{l}\mathrm{o}\mathrm{g}\left(1-D\left(x,G\left(x,z\right)\right)\right)\right].$$

(10)

In Equation (10), $x$ represents the source image, while $y$ represents the corresponding actual target image. The $z$ represents the randomness. The discriminator network $D$ attempts to distinguish whether the image pairs (source–target) presented to it are from the real data distribution or generated by the generator network; the generator network $G$ aims to make its generated $G\left(x\right)$ outputs indistinguishable from real target images. This adversarial learning process ensures that the distributional alignment between source and target images is learned. However, using only adversarial loss can lead to the loss of fine structural details and local consistency, especially in medical images. To mitigate this problem, a pixel-based reconstruction loss, as defined in Equation (11), is additionally defined in the pix2pix model. The total objective function is expressed by the formula in Equation (12) as the weighted sum of the adversarial loss and the reconstruction loss.

$${\mathcal{L}}_{L1}\left(G\right)={\mathbb{E}}_{x,y}\left[\parallel y-G\left(x\right){\parallel }_1\right].$$

(11)

$$G^{*}=\arg \underset{G}{\min } \underset{D}{\max }({\mathcal{L}}_{\mathrm{c}\mathrm{G}\mathrm{A}\mathrm{N}}\left(G,D\right)+\lambda \hspace{0.17em}{\mathcal{L}}_{\mathcal{L}1}\left(G\right))\hspace{0.17em}$$

(12)

In Equation (12), $\lambda$ represents a weight coefficient that controls the balance between adversarial loss and reconstruction loss. Thanks to this combined loss function, the pix2pix model can learn the transformation while not only producing visually realistic outputs but also preserving the structural features of the target image. Due to these properties, pix2pix is widely used in many applications such as medical image segmentation, modality conversion, and image enhancement [41,42,43].

3.4.2. Generation of Synthetic Thermal Images

The high cost of thermal cameras, hardware limitations, and their lack of standard availability in every clinical setting lead to limited and incomplete thermal image-based datasets. This constitutes a significant obstacle in the development of multimodal analyses using both RGB and thermal images [44].

GAN-based image-to-image translation approaches allow for the synthetic generation of thermal-like maps from RGB images to overcome this limitation. Thanks to these methods, datasets containing only RGB images can be transformed into rich multimodal datasets consisting of conjugate RGB–thermal pairs. In conditional GAN-based models such as pix2pix, RGB images are considered as the source modality and thermal images as the target modality, and the model learns to estimate the corresponding thermal distribution for each RGB input. In the literature, it has been reported that such modality transformation approaches have been successfully used to synthetically complete missing modality, enhance data, and generate meaningful inputs for multimodal fusion systems [45,46]. However, to the best of our knowledge, no prior study has explored synthetic modality generation for DFU detection. Motivated by these observations, this study proposes a GAN-based framework to generate synthetic thermal representations from RGB images, enabling effective multimodal fusion for DFU detection in the absence of real thermal data.

3.5. Performance Metrics

The classification performance of the proposed framework was assessed using standard metrics, including accuracy, precision, recall, F1-score, and ROC-AUC. A high recall value is important for clinical safety in terms of reducing the number of patients who are incorrectly classified as healthy. Precision, on the other hand, contributes to reducing unnecessary clinical interventions by preventing false positive results [47]. In order to evaluate the discrimination power of the model independent of the decision threshold, the ROC–AUC (Receiver Operating Characteristic—Area Under Curve) metric was also used in this study. The ROC curve shows the relationship between the true positive rate and the false positive rate at different threshold values, and the area under the curve summarizes the overall discrimination capacity of the model [48]. Additionally, macro-F1 and weighted-F1 scores were reported to evaluate performance across classes in a balanced manner [49].

4. Materials and Methods

In this study, a two-stage DL approach was developed for the classification of DFUs. In the first stage, a conditional GAN-based pix2pix model was established to produce visually consistent pseudo-thermal maps from RGB images. This yielded a synthetic thermal modality that contained patterns similar to the temperature distribution around the ulcer, in addition to structural visual information. In this stage, Turbo Colormap and Wound Boosting techniques were integrated into the transformation process to enhance the visual discriminability and semantic consistency of the thermal modality.

In the second stage, a dual-stream CNN classifier architecture was proposed using both RGB and synthetic thermal images. Feature representations obtained from each modality were combined using a MHCA-based fusion mechanism. This enabled the model to dynamically learn inter-modality relationships in clinically critical regions such as ulcer margins, inflammation areas, and pathological tissue areas.

4.1. Dataset

In this study, the DFU dataset publicly shared on the Kaggle platform was used. The dataset is derived from previously published works by Alzubaidi et al., where two clinically relevant DFU datasets (DFU-1 and DFU-2) were introduced [20,21]. These datasets consist of expert-labeled foot images collected under controlled acquisition settings and have been widely used in multiple DFU classification studies. The dataset contains a total of 1055 RGB images, including ulcerated (512) and healthy (543) images. The dataset was randomly split into three subsets: 80% training, 10% validation, and 10% test (8:1:1). This division allowed the model performance to be monitored throughout the training process and provided an unbiased test evaluation. The ulcer class includes ulcer lesions that appear in different areas of the foot, disrupt tissue integrity, and vary in severity. Figure 1 shows sample images of healthy and ulcerated feet from the dataset. In addition to the random 8:1:1 protocol, a 10-fold GroupKFold CV strategy was employed to prevent data leakage—commonly observed in medical datasets and capable of artificially inflating model performance—by ensuring that images from the same patient were not distributed across training and test sets. Furthermore, to enhance model generalization-, Random Flip-, Random Rotation-, and Random Zoom-based data augmentation techniques were synchronously applied to both modalities.

Figure 1. Sample images from the DFU dataset: (a) ulcer, (b) normal.

4.2. The Proposed RGB2T-cGAN Model for Synthetic Thermal Image

Although the pix2pix model is used in the conversion of non-medical images, its application in medical images is limited [50]. However, GAN-based models are used for data augmentation in medical images for the production of spectral maps of skin lesions [51,52]. Since the RGB–Thermal conversion is also treated as a translation problem in this study, the structure of the cGAN architecture is compatible with the RGB–Thermal conversion. The RGB2T-cGAN (RGB to Thermal cGAN) model has been proposed to produce the second modality required for DFU multimodal classification synthetically (pseudo-thermal). In the RGB2T-cGAN model, the generator network is designed using a seven-level U-Net-based encoder–decoder architecture. The U-Net structure enables the lossless modeling of wound boundaries, tissue integrity disruptions, and color variations by transferring low-level anatomical features to high-level representations through skip connections [53]. On the discriminator, PatchGAN is used, enabling more accurate learning of high-frequency thermal textures by distinguishing between real and fake patches based on $70\times 70$ local patches. PatchGAN demonstrates superior performance in medical modality transformations where the preservation of structural details is critical [16]. A schematic view of the proposed RGB2T-cGAN model is presented in Figure 2. Two additional improvements were applied to enhance the visual distinctiveness and potential interpretability of the thermal maps generated by the proposed RGB2T-cGAN model. First, Turbo Colormap reduced perceptual distortions caused by non-linear color maps, enabling clearer representation of temperature gradients. Second, the developed Wound Boosting technique contributed to making the thermal profile clinically more meaningful by enhancing heat anomalies around the lesion. These improvements made thermal elevations, particularly in ulcer areas, visually more prominent.

Figure 2. An illustration of the proposed RGB2T-cGAN model.

The RGB2T-cGAN model consists of three main stages: (i) data preparation, (ii) loss-weighted cGAN training, and (iii) post-inference wound-focused reinforcement. In the first stage, RGB DFU images were converted to grayscale, and pseudo-thermal targets were obtained using OpenCV Colormap Turbo. The Turbo Colormap was preferred because it represents temperature gradients in a more perceptually linear manner. At the same time, a binary ulcer mask was created using the LAB color space and Otsu-based thresholding methods. These pseudo-thermal targets are not actual thermal camera measurements but synthetic representations that serve as references during GAN training. In the second stage, the U-Net-based generator network produces thermal-like maps from RGB images, while the discriminator network, with a PatchGAN architecture, performs adversarial learning by evaluating input–output pairs through $70\times 70$ local patches. This structure ensures that not only global image structures but also local thermal-like texture patterns are learned. During the training process, adversarial loss, Smooth L1, and VGG19-based perceptual loss are optimized together, ensuring both structural integrity and perceptual consistency. Furthermore, using a wound mask, the pixel-based loss function is weighted more heavily in lesion areas. In the final stage, the trained generator network produces raw thermal outputs from new RGB images; these outputs are then enhanced in terms of temperature through a post-processing step guided by the wound mask, resulting in clinically more interpretable final thermal maps. In this study, the discriminator evaluates the RGB input image and the thermal target image together to learn whether these two images are a true match. Thus, the model optimizes not only thermal similarity but also the structural consistency between the input image and the output.

The synthetic thermal images used in this study are not intended to serve as a data augmentation strategy in the conventional sense. Unlike standard augmentation techniques that increase data variability within the same modality, the proposed RGB-to-thermal transformation generates a complementary modality that encodes physiologically inspired temperature patterns related to inflammation and tissue abnormality. Each RGB image is deterministically mapped to a single synthetic thermal representation, which is used consistently during both training and inference. Furthermore, the synthetic thermal modality is processed through a dedicated network stream and integrated with RGB features via a cross-attention mechanism. This design enables the model to explicitly learn inter-modal relationships, supporting the interpretation of the synthetic thermal input as an auxiliary modality rather than as a form of stochastic data augmentation.

4.2.1. The Loss Function and Training of RGB2T-cGAN Model

While the standard pix2pix framework typically employs an L1 loss, this study designs a multi-component hybrid loss function to meet the high precision and noise robustness requirements of medical imaging. The total objective optimized during training consists of adversarial, pixel-wise, and perceptual loss components. To stabilize adversarial learning, a mean squared error-based Least Squares GAN loss was adopted, enabling the discriminator to learn a more stable separation between real and synthetic samples [39]. For pixel-level modeling, Smooth L1 loss—more robust to outliers than standard L1—was used to promote smoother and more balanced thermal transitions. Moreover, the Smooth L1 loss was weighted using wound masks to emphasize clinically critical lesion regions. To ensure semantic consistency beyond pixel accuracy, a perceptual loss based on the L1 distance between feature maps extracted from the first 16 layers of an ImageNet-pretrained VGG19 network was employed [54]. Accordingly, the overall objective function is defined in Equation (13).

$$L_G=L_{GAN}+{\lambda }_{pixel}{L}_{SmoothL1}+{\lambda }_{perc}{L}_{VGG}$$

(13)

In this study, $L_G$ denotes the total loss function optimized by the generator and is defined as a weighted combination of adversarial, pixel-based, and perceptual losses. $L_{GAN}$ represents the adversarial loss, encouraging the generator to produce thermal images that are classified as real by the discriminator. $L_{SmoothL1}$ measures the pixel-level $SmoothL1$ difference between the generated and target thermal images; this term is further weighted in wound mask regions to emphasize clinically critical areas and improve local accuracy. $L_{VGG}$ denotes the perceptual loss, which computes the difference between real and generated images using feature maps extracted from intermediate layers of a VGG-19 network, thereby preserving structural and high-level semantic consistency. The coefficients ${\lambda }_{pixel}$ and ${\lambda }_{perc}$ control the relative importance of the pixel-based and perceptual losses, respectively. In this study, ${\lambda }_{pixel}$ was set to 100 to prioritize pixel-level similarity, while ${\lambda }_{perc}$ was set to 10 to balance the contribution of the perceptual loss.

Training of the RGB2T-cGAN model was performed using Python (version 3.12) based DL libraries on an A100 GPU in the Google Colab. The hyperparameters used in the training process are summarized in Table 2.

Table 2. The hyperparameters of RGB2T-cGAN model for training.

Parameter	Value
Input size	256 × 256
Batch Size	4
Epoch	100
Learning Rate	2 × 10−4
Optimizer	Adam (beta_1 = 0.5, beta_2 = 0.999)

4.2.2. Wound Boosting Technique and Clinical Interpretability

The clinical interpretability of the generated thermal output is enhanced through the application of a novel Wound Boosting technique during the inference stage. This approach is a fully post-processing-based mechanism that is not involved in the training phase and aims to visually emphasize thermal intensity within wound regions. Specifically, using a previously obtained binary wound mask $M\left(x,y\right)\in \mathrm{0,1}$, the raw thermal output $T\left(x,y\right)$ produced by the generator is updated as defined in Equation (14).

$$T_{boost}\left(x,y\right)=T\left(x,y\right)\cdot \left(1+\alpha \cdot M\left(x,y\right)\right)$$

(14)

In this study, $\alpha =4$ was set as a fixed hyperparameter and remained unchanged throughout the entire inference process. Thermal values are preserved in regions outside the mask ($M\left(x,y\right)=0$), while thermal intensity is increased in a controlled manner only in pixels corresponding to ulcer tissue. This method enhances the thermal contrast between ulcer tissue and healthy surrounding tissue, thereby providing a more discriminative representation for both visual analysis and the attention mechanisms of the multimodal classification model.

4.2.3. Results of Synthetic Thermal Image Generation Using the RGB2T-cGAN

The RGB2T-cGAN model was applied to generate thermal images across the entire dataset. Figure 3 illustrates the evolution of the Generator Loss (A), Discriminator Loss (B), Smooth L1 Loss (C), and Perceptual (VGG19) Loss (D) over training iterations. The gradual decrease and convergence of these loss values indicate that the model is able to generate pseudo-thermal maps with improved structural and perceptual consistency.

Figure 3. Loss curves obtained during the training process of the RGB2T-cGAN.

Figure 4 presents representative examples of the generated pseudo thermal maps. Ulcer regions exhibit a pronounced temperature increase compared to surrounding tissues, and the combined effect of Wound Boosting, Turbo Colormap, and Smooth L1 mechanisms enhances the visual clarity of high-temperature areas. In contrast, healthy tissues are represented with lighter colors without artificial temperature amplification. The dataset used in this study was converted into thermal images using the RGB2T-cGAN model. Thus, since the representations obtained from thermal images by CNN-based classifiers will be different from those obtained from RGB images, the DFU diagnosis process from the images has been improved. In the next section, both modalities will be provided as input to the proposed multimodal CNN classifier.

Figure 4. Examples of pseudo thermal maps generated from RGB foot images in the DFU dataset.

The synthetic thermal images generated in this study are not intended to be pixel-level replicas of real thermal measurements. Since the Kaggle DFU dataset does not provide paired RGB–thermal images, a direct quantitative comparison using metrics such as SSIM or PSNR is not feasible. Therefore, the effectiveness of the synthetic modality is evaluated indirectly through its contribution to multimodal classification performance.

4.3. Proposed Multimodal Dual-Stream Multi-Head Cross-Attention for DFU

In the second phase of this study, to address the limitations of single modalities (RGB only or thermal only) in DFU diagnosis reported in the literature, a Multimodal Dual-Stream Multi-Head Cross-Attention (MDS-MHCA) DL model was proposed that combines the complementary information content of both modalities into a common representation. This model is structured with convolutional layers and residual blocks to extract high-level spatial and semantic representations. The synthetic thermal maps generated in Section 4.2 and the original RGB images were processed in separate streams using the dual-stream MHCA fusion approach, where the features processed in each stream can inform each other. This mechanism aims to improve classification performance by creating high-quality feature maps, enabling the learnable modeling of intermodal interactions by simultaneously modeling the structural texture information of RGB and the inflammation-induced temperature variations in the thermal modality. The structure of the proposed MDS-MHCA model is presented in Figure 5.

Figure 5. Schematic of the proposed MDS-MHCA classification model.

In the MDS-MHCA model, RGB and thermal images are processed independently through two separate streams at the model’s input stage. Each stream is designed using residual (skip-connected) blocks to prevent the gradient vanishing problem commonly encountered in deep networks and to enable more effective feature extraction. For both modalities, a convolution layer with a 7 × 7 kernel size and a stride of 2 is used in the input layer to preserve large-scale spatial structures in the images. Following this, each stream contains four residual blocks arranged sequentially, which add the input information to the processed output, enabling the network to learn in a more stable and deeper manner. As the network depth increases, the number of filters is increased from 64 to 128, enabling a gradual transition from low-level visual cues (edge, color, and intensity information) to high-level semantic features (ulcer tissue and temperature gradients). Feature maps obtained from CNN blocks are flattened in spatial dimensions and transferred to the Transformer-based Cross-Attention module. At this stage, CNN outputs are converted from $H\times W\times C$ dimensions to vector sequences representing spatial positions, with each position treated as a token. To enable the model to learn multimodal interactions more effectively, a four-headed (heads = 4) MHCA mechanism is used.

Feature vectors passed through the attention mechanism and normalized using layer normalization are summarized via the global average pooling 1D layer, reduced to one-dimensional representations, and then combined in the concatenate layer to form the final fusion representation. The deep supervision strategy was applied to increase the stability of the model during training and to encourage each modality to learn discriminative features independently. In this context, while the main output branch produces the final classification decision based on the combination of RGB and thermal features, the auxiliary outputs for RGB and thermal streams require each modality to learn discriminative representations independently. This multi-output structure enhances the generalization capability of multimodal fusion while effectively preventing the loss of modality-specific information. During model training, both the main fusion output and the auxiliary outputs for each modality were optimized using the deep supervision strategy. The total loss function used in this context is defined as in Equation (15).

$$L_{\mathrm{total}}=L_{\mathrm{main}}+{\lambda }_{\mathrm{aux}}\left(L_{\mathrm{rgb}}+L_{\mathrm{thermal}}\right)$$

(15)

All loss terms were computed using categorical cross-entropy, and the auxiliary loss weight was set to ${\lambda }_{\mathrm{aux}}=0.5$ to encourage each modality stream to learn discriminative features independently. To prevent the model from over-relying solely on the fused output and to ensure stable learning under missing, noisy, or low-quality modality conditions, the total loss $L_{\mathrm{total}}$ was used as the optimization objective. This loss design effectively enhances both the overall generalization capability and modality-specific discriminability of the multimodal classification network [55]. The proposed classifier was trained in Google Colab using an A100 GPU, and the main hyperparameters used during training are summarized in Table 3.

Table 3. The hyperparameters of MDS-MHCA model for training.

Parameter	Value
Input size	224 × 224
Batch Size	32
Epoch	60
Learning Rate	1 × 10−3
Optimizer	Adam
Multi-Head	4
Dropout	0.3
Early Stopping Patience	15

5. Classification Results and Analysis

In this section, the classification performances of the developed models are evaluated across RGB, synthetic thermal, and multimodal streams. First, to reveal the discriminative capability and limitations of each modality when used independently, the test results of the RGB-based and synthetic thermal-based models are reported using standard evaluation metrics, including accuracy, precision, recall, and F1-score. Second, to assess the robustness of the proposed MDS-MHCA model across different patient groups, both the patient-wise 10-fold GroupKFold CV results and the independent test set results are presented. Finally, the outcomes of the RGB, thermal, and multimodal approaches are statistically compared to demonstrate the superiority of the proposed model.

5.1. RGB Image Classification Results

The independent performance of the RGB modality was examined to assess the extent to which the model can discriminate surface morphology, color distribution, and texture-based structural features using RGB information alone. In this context, the model was evaluated on real RGB test images, and the corresponding classification metrics were computed. Figure 6 presents the performance metrics and the confusion matrix obtained on the test dataset. According to Figure 6, the model trained with the RGB modality achieved an accuracy of 94.34%, a precision of 92.50%, a recall of 91.82%, a Macro-F1 score of 91.49%, and an ROC–AUC score of 99.36%. When precision and recall are considered jointly, the results indicate that the model exhibits high sensitivity in identifying ulcerated tissues while producing a limited number of false positives in the healthy class.

Figure 6. Performance results and confusion matrix for the RGB test dataset. (a) Classification metric results on the RGB test dataset. (b) Confusion matrix of the RGB modality on the test dataset.

According to the confusion matrix shown in Figure 6, the RGB modality demonstrates strong performance in recognizing ulcerated tissues. However, a higher error rate is observed in the healthy class compared to the thermal modality. This can be attributed to the limited color and texture variations present in healthy tissues, which restrict the model’s ability to capture distinctive structural cues. An examination of the class-wise error distribution in the confusion matrix further reveals that the RGB modality remains limited in representing healthy tissues and is unable to fully capture the range of clinical variations when used alone. One representative example of a misclassified sample from Figure 6 is illustrated in Figure 7.

Figure 7. False positive example in the RGB modality.

As shown in Figure 7, while RGB images successfully represent the structural and visual characteristics of ulcers, their lack of direct functional information regarding inflammation and physiological changes can limit their discriminatory power in some clinical scenarios. This clearly supports the need to evaluate RGB information in conjunction with complementary modalities such as thermal imaging.

5.2. Synthetic Thermal Image Classification Results

The independent evaluation of the thermal modality was conducted to assess the extent to which the model can capture discriminative features based solely on temperature distribution. Figure 8 presents the performance metrics and the confusion matrix obtained on the thermal test dataset. The model trained with the thermal modality achieved an accuracy of 95.28%, a precision of 96.36%, a recall of 96.36%, and an ROC–AUC score of 99.79% on the held-out test set. The identical values of the Macro-F1 and Weighted-F1 scores indicate the absence of a notable performance imbalance between classes, demonstrating that the model can distinguish both classes with comparable effectiveness.

Figure 8. Performance results and confusion matrix for the synthetic thermal test set. (a) Classification metric results on the synthetic thermal test dataset. (b) Confusion matrix of the synthetic thermal modality on the test dataset.

To evaluate the discriminative capability of the model independent of the decision threshold, ROC and precision–recall analyses were performed. The high ROC–AUC value of 99.79% indicates that thermal information can effectively capture ulcer-induced temperature anomalies. Furthermore, the superior performance achieved using the thermal images generated by the proposed RGB2T-cGAN model, compared to the RGB images presented in Figure 6, further validates the effectiveness of the proposed thermal image generation approach.

In Figure 8, all ulcerated samples were correctly classified. However, four samples belonging to the healthy class were observed to be falsely classified. This type of error is related to the fact that patterns similar to local temperature increases in thermal maps can also occur in some healthy samples. In other words, while thermal modality produces a strong sensitivity in the ulcer class, ulcer-like thermal patterns in some healthy samples can lead to false alarms. Representative examples of the misclassified samples identified in the confusion matrix of Figure 8 are illustrated in Figure 9.

Figure 9. Examples of pseudo-thermal images labeled as healthy that were incorrectly classified as ulcer by the thermal stream.

An examination of the visual examples in Figure 9 indicates that, in some healthy cases, the thermal maps may produce ulcer-like heat concentrations due to the spatial distribution of texture or color gradients, which can trigger false positive predictions. Consequently, although the thermal modality provides a strong signal for ulcer detection, it may exhibit a tendency toward false positives when used in isolation under certain clinical scenarios.

5.3. Classification Results of the Proposed MDS-MHCA Model

The proposed MDS-MHCA classification model achieved more consistent and higher accuracy prediction performance compared to individual modalities by interactively combining the complementary information provided by both modalities. In this section, the model’s classification performance on real test images is comprehensively evaluated in terms of predictive accuracy, error analysis, discriminative capability, and training stability. Figure 10 presents the classification performance and confusion matrix of the proposed MDS-MHCA model. The MDS-MHCA model achieved 99.06% accuracy on the test set. The close correspondence between the macro-F1 (99.06%) and weighted-F1 (99.04%) scores of the model reveals that there is no significant performance imbalance between classes and that the model offers stable discrimination in both ulcerated and healthy samples. In addition, the high recall value (99.09%) indicates that nearly all clinically critical abnormal cases were correctly identified. According to the confusion matrix in Figure 10, no false negative errors occurred in the abnormal class, and only one healthy sample was classified as a false positive. The fact that there was only one misclassified sample reveals that the misclassifications in the healthy class observed in the single RGB model and the uncertainties due to the homogeneous temperature distribution in the thermal model were largely eliminated with the multimodal structure. This shows that the cross-attention mechanism in the MDS-MHCA model effectively aligns the structural cues observed in RGB with the physiological temperature changes in the thermal modality and that complementary information between modalities is successfully integrated. Furthermore, the ROC-AUC value of 99.79% of the model shows that the multimodal approach offers extremely high discrimination in both ulcer and healthy classes. Compared to the single-modality results, a clear improvement is observed over the RGB-based model.

Figure 10. Classification results and confusion matrix of the proposed MDS-MHCA model on the RGB + Thermal test dataset. (a) Classification metric results on the RGB + synthetic thermal test dataset. (b) Confusion matrix of the RGB + synthetic thermal modality on the test dataset.

The training behavior and stability of the MDS-MHCA model were assessed by analyzing the epoch-wise evolution of the loss, accuracy, and AUC metrics for the main output, as illustrated in Figure 11. When the training and validation curves in Figure 11 are examined together, it reveals that the model exhibits rapid convergence from the early epochs, follows a stable learning process without showing signs of overfitting, and has a high generalization capacity. In particular, the high and stable trend of the accuracy and AUC curves indicates that the proposed MDS-MHCA model is also effective in terms of training stability. In conclusion, the proposed MDS-MHCA model, by interactively integrating the structural and textural information provided by RGB with the physiological temperature cues provided by the thermal modality, offers a more reliable performance compared to individual modalities in DFU classification with 99.06% test accuracy, high recall, and stable F1 scores.

Figure 11. Loss, accuracy, and AUC curves of the MDS-MHCA model during the training and validation processes.

The high classification performance reported in this study should be interpreted in light of the dataset characteristics. To analyze potential dataset bias and image homogeneity, a quantitative assessment was performed using deep feature representations extracted with a pretrained ResNet50 network. The average intra-class variance was 25.99 for the normal class and 29.33 for the abnormal class, while the inter-class distance was measured as 28.82, resulting in a homogeneity ratio of 1.92. These values indicate a moderate level of image homogeneity, reflecting controlled acquisition conditions rather than a trivially separable dataset.

5.4. Patient-Wise Generalization Analysis of the MDS-MHCA Model

The generalization capability of the multimodal model was evaluated using a patient-wise 10-fold CV strategy based on the GroupKFold approach. This method prevents data leakage by preventing the simultaneous distribution of images from the same patient to the training and test layers. Figure 12 presents the 10-fold GroupKFold accuracy values obtained for the MDS-MHCA model.

Figure 12. Fold-wise accuracy distribution of the proposed MDS-MHCA model using 10-fold GroupKFold CV.

An examination of the 10-fold GroupKFold CV results presented in Figure 12 indicates that the proposed multimodal model exhibits high, consistent, and stable classification performance across different patient groups. The majority of fold-wise accuracy values remain above 95%, with the results concentrated within a narrow range. This distribution demonstrates that the model is capable of effective generalization without overfitting to specific patient groups.

On the other hand, the fact that the GroupKFold accuracy is lower than the 99.06% accuracy obtained on the held-out test dataset is an expected outcome. Nevertheless, the absence of a substantial performance degradation confirms the effectiveness of the multimodal approach. The achieved mean accuracy of 96.49%, together with a low standard deviation (±0.0104), indicates that the multimodal architecture delivers stable performance across different data splits. In addition, the model’s AUC score (0.9927 ± 0.0067) further confirms its strong patient-wise generalization capability. The variation in fold accuracies within the range of 95.24% to 98.10% highlights that the RGB and thermal modalities effectively complement each other through the cross-attention mechanism, enabling the model to maintain robust and consistent performance under patient-level partitioning.

5.5. Statistical Significance Analysis of Classification Performance

The statistical significance of the performance improvement achieved by the proposed MDS-MHCA model over the RGB-only model was evaluated using the McNemar test on the test dataset. The McNemar test examines whether the difference between the models is random by comparing the error distributions of two classifiers working on the same samples [56]. In this context, the MDS-MHCA model and the RGB modality model were compared on the test set. When the McNemar error table presented in Figure 13 is examined, it is seen that although there are eight samples that the MDS-MHCA model classified correctly while the RGB model classified incorrectly, there are no samples that the RGB model classified correctly and the MDS-MHCA model classified incorrectly.

Figure 13. McNemar comparison and error asymmetry between the MDS-MHCA and RGB models. (a) McNemar comparison between the multimodal and RGB models. (b) Error asymmetry.

The McNemar test result calculated based on the error distribution in Figure 13 was obtained as p = 0.0078. This value is below the significance level of p < 0.05 [56], indicating that the performance improvement provided by the multimodal model compared to the RGB model is statistically significant. To visually support the error asymmetry, the error asymmetry graph showing the direction of errors of the MDS-MHCA and RGB models on the paired samples on the test set is presented in Figure 13.

5.6. Ablation Analysis

Ablation analysis was performed to observe the effects of Wound Boosting and the number of attention heads used in the MDS-MHCA model. The number of headings used in the proposed MDS-MHCA model was determined to be four. To analyze the effect of the number of attention heads on classification performance, a sensitivity analysis was conducted by evaluating the proposed MDS-MHCA model with two, four, and eight attention heads. The performance metrics of the test dataset based on the number of titles are presented in Table 4. As the results indicate, increasing the number of heads from two to four significantly improves all evaluation metrics, suggesting that multiple attention heads enable more effective modeling of cross-modal relationships between RGB and synthetic thermal features. While configurations with four and eight heads yield comparable performance on the test set, increasing the number of heads beyond four does not provide additional benefits. This saturation effect implies that four attention heads are sufficient to capture the relevant inter-modal dependencies, whereas higher numbers primarily increase model complexity without improving generalization.

Table 4. Sensitivity analysis of the number of attention heads on the independent test set.

Number of Attention Heads	Accuracy	Precision	Recall	F1-Score
2 Heads	0.9528	0.9532	0.9524	0.9527
4 Heads (Proposed)	0.9906	0.9904	0.9909	0.9906
8 Heads	0.9906	0.9904	0.9909	0.9906

An ablation study was conducted to examine the effect of the Wound Boosting module and to assess whether it introduces label-dependent spatial priors. The classification results are presented in Table 5. As shown in Table 5, when Wound Boosting is disabled, the multimodal fusion model already outperforms the auxiliary RGB and thermal branches, achieving an accuracy of 0.9528 compared to 0.8679 and 0.8585, respectively. This indicates that the main discriminative capability arises from cross-attention-based multimodal fusion rather than lesion-focused pre-processing.

Table 5. Ablation study on the effect of the Wound Boosting module on classification performance.

Output Branch	Wound Boosting	Accuracy	Precision	Recall	F1-Score
MDS-MHCA	Without	0.9528	0.9554	0.9545	0.9528
MDS-MHCA	With	0.9906	0.9904	0.9909	0.9906
RGB (Aux)	Without	0.8679	0.8923	0.8727	0.8667
RGB (Aux)	With	0.9151	0.9250	0.9182	0.9149
Thermal (Aux)	Without	0.8585	0.8864	0.8636	0.8570
Thermal (Aux)	With	0.9623	0.9636	0.9636	0.9623

Enabling Wound Boosting improves performance across all branches; however, the largest gain is observed in the fusion model, where accuracy increases from 0.9528 to 0.9906. In contrast, the RGB and thermal branches exhibit more limited improvements (0.8679 to 0.9151 and 0.8585 to 0.9623, respectively). This asymmetric improvement pattern, highlighted in Table 5, suggests that Wound Boosting enhances multimodal feature interaction rather than acting as a label-dependent spatial shortcut.

5.7. Discussion

Classification results demonstrate that the proposed two-stage multi-modal framework significantly outperforms single-modal models in DFU classification. This performance improvement can be attributed to various interrelated factors embedded in the design of the RGB2T-cGAN and MDS-MHCA architectures.

Firstly, the error analysis of single-modal models reveals their inherent limitations. As shown in Figure 6, the model using only RGB tends to misclassify healthy skin with a reddish texture as an ulcer (false positive). This confirms that relying solely on visual surface features is insufficient to distinguish inflammation from pigmentation irregularities. In contrast, the synthetic thermal modality serves as a functional proxy by highlighting regions with density patterns specific to inflammation. The MDS-MHCA model effectively filters visual traps in RGB images by integrating these two streams, such that a reddish area on the skin is classified as an ulcer only if it is confirmed by a corresponding high-intensity signal in the thermal stream. The multimodal model effectively cross-validates hypotheses from both streams: an area flagged as suspicious by RGB but lacking a corresponding thermal anomaly is given less weight, and vice versa. This cross-modal validation is reflected in near-perfect sensitivity (99.09%) and a single false positive in the test set (Figure 10), indicating a robust reduction in both false negatives and false positives.

Secondly, the MHCA mechanism, unlike early or late fusion, allows the MHCA model to dynamically weight the importance of spatial regions. Thermal features act as a query function to focus the attention of the RGB stream on clinically lesion boundaries, thereby reducing the influence of irrelevant background structures. This interaction enables the model to learn to prioritize lesion-specific tissue information over general skin variations.

Thirdly, although thermal images are synthetically generated from RGB inputs, they serve as a feature transformation function that highlights critical regions and is highly non-linear. This process acts as a form of region-specific attention guidance. McNemar test results (p < 0.05) and ablation studies (Table 5) confirm that this structural support provided by the synthetic modality, when combined with the Wound Boosting technique, creates a more robust decision boundary compared to raw RGB data. The Wound Boosting process enhances the visibility of lesion areas in synthetic thermal maps, providing a stronger signal for the attention mechanism to focus on clinically relevant regions. The ablation study (Table 5) shows that Wound Boosting improves both single-modal streams, but the greatest gain is observed in the combined model, where accuracy raised from 95.28% to 99.06%.

Finally, the deep supervision strategy (Equation (15)) enables each modality stream to learn discriminative features independently before fusion. This prevents the model from collapsing into dominance by a single modality and encourages the extraction of complementary features.

5.8. Limitations and Future Works

This study has several limitations that must be acknowledged. First, the experimental evaluation is performed on a relatively small and carefully selected dataset acquired under controlled imaging conditions. Although patient-based CV and ablation analyses were used to mitigate overlearning, the lack of external validation limits the direct generalizability of the reported results to various clinical settings. Second, the synthetic thermal modality used in this study was obtained through a deterministic transformation from RGB images and is designed as an auxiliary input rather than a replacement for actual thermal imaging. While the proposed framework demonstrates that such synthetic representations can support multimodal learning, the synthetic modality may not fully capture all physiological variations in actual thermal measurements. Accordingly, its role should be interpreted as complementary rather than clinically equivalent to actual thermal data. Third, quantitative analysis of deep feature representations revealed a moderate level of image homogeneity in the dataset, suggesting controlled acquisition conditions rather than a trivially separable classification problem. Additionally, the lack of external or multicentric validation limits the ability to directly assess the generalizability of the proposed framework to various clinical settings. Moreover, the proposed framework has not been externally validated on independent or multi-center datasets, which limits the direct assessment of its generalizability to diverse clinical environments.

Future work will focus on validating the proposed framework using larger, multi-center datasets with heterogeneous acquisition conditions and, where possible, real thermal imaging. Additionally, incorporating clinical metadata and longitudinal information may further enhance robustness and clinical applicability.

6. Conclusions

DFUs are one of the most challenging clinical conditions in diabetes management, and late diagnosis and misdiagnosis can lead to irreversible consequences. Therefore, early determination of DFUs with high sensitivity and specificity has become a fundamental requirement of modern clinical practice. This study proposes a novel DL approach based on MDS-MHCA, supported by synthetic thermal image generation, aiming to overcome the limited representational power of individual modalities in the automated classification of DFUs. In the first stage of this study, the problem of missing thermal modality, frequently encountered in clinical settings, was addressed with a pix2pix-based conditional generative counter-network structure that generates clinically significant pseudo-thermal maps from RGB images. The resulting synthetic thermal representations provide a layer related to physiological patterns sensitive to inflammation and temperature anomalies, contributing to making the classification process sensitive not only to visual texture and morphology but also to the underlying biophysiological process. In the second stage, the proposed MDS-MHCA classifier models features extracted from RGB and synthetic thermal streams within two parallel pathways and integrates complementary cross-modal relationships through a MHCA mechanism. Thus, structural and physiological cues from both modalities are brought to a common representation at the level of complex clinical patterns that single modalities cannot capture. The classification results showed that the proposed model achieved over 99% accuracy and F1-score in the separated test set, offering a significant and consistent performance advantage compared to models trained with only RGB or only thermal images. In particular, the correct classification of almost all ulcer cases reveals that the model significantly reduces the clinically critical risk of false negatives. Patient-based 10-fold GroupKFold CV analyses showed that the model exhibited high generalization ability across different patient groups. A McNemar test was applied to examine whether the performance improvement of the model was statistically significant. The McNemar test results confirmed that the performance improvement provided by the multimodal approach compared to the RGB model was statistically significant.

The results obtained in this study clearly demonstrate that the synthetic generation of thermal information and its integration into the classification process using an attention-based multimodal architecture offers a powerful solution for the detection of DFUs. The synthetic thermal representations used in this study are not intended to replace real thermal images; rather, they are considered an auxiliary modality that enriches the multimodal learning process. It is assessed that the proposed approach, when integrated into clinical decision support systems, could contribute to earlier and more reliable detection of ulcers, thus reducing amputation risks and treatment costs. However, this study has some limitations, such as the size of the dataset used and the limited availability of real thermal images. Future studies that utilize multicenter and larger-scale datasets, combinations of real and synthetic thermal images, architectures with incomplete modality tolerance, and prospective validations integrated with clinical processes are planned.