Medical Image Segmentation Network Based on Dual-Encoder Interactive Fusion

Abstract

Hybrid CNN–Transformer networks seek to merge the local feature extraction capabilities of CNNs with the long-range dependency modeling abilities of Transformers, aiming to simultaneously address both local details and global contextual information. However, in many existing studies, CNNs and Transformers are often combined through the straightforward fusion of encoder features, which does not promote effective interaction between the two, thus limiting the potential benefits of each architecture. To overcome this shortfall, this study introduces a novel medical image segmentation (MIS) network, designated as DEFI-Net, which is based on dual-encoder interactive fusion. This network enhances segmentation performance by fostering interactive learning and feature fusion between the CNN and Transformer encoders. Specifically, during the encoding phase, DEFI-Net utilizes parallel encoding with both the CNN and Transformer to extract local and global features from the input images. The global–local interaction learning (GLIL) module then enables both the Transformer and CNN to assimilate global semantics and local details from each other, fully leveraging the strengths of the two encoders. In the feature fusion phase, the global–local feature fusion (GLFF) module integrates features from both encoders, using both global and local information to produce a more precise and comprehensive representation of features. Extensive experiments conducted on multiple public datasets, including multi-organ, cardiac, and colon polyp datasets, demonstrate that DEFI-Net surpasses several existing methods in terms of segmentation accuracy, thus highlighting its effectiveness and robustness in MIS tasks.

1. Introduction

Medical image segmentation (MIS) is crucial in aiding physicians to precisely ascertain the location, size, and shape of lesions, and it provides a robust foundation for accurate medical diagnostics and treatment planning. Medical images, unlike natural images, pose unique challenges such as intricate organ details, indistinct lesion boundaries, and low contrast. Therefore, achieving efficient and accurate segmentation is of critical importance.

With the rapid advancements in deep learning technology, convolutional neural networks (CNNs) have become the most prevalent approach for MIS tasks. Ronneberger et al. [1] introduced U-Net, a segmentation method based on a fully convolutional network that features a symmetric encoder–decoder structure with skip connections to preserve detailed information. The U-Net architecture has shown superior segmentation performance and its symmetric design has been widely adopted in various medical segmentation applications. Subsequent research has further refined U-Net to address diverse challenges in medical segmentation, yielding variants such as V-Net [2], attention U-Net [3], HADCNet [4], and I2U-Net [5]. However, the limited receptive field of convolutional kernels in CNNs allows for the excellent capture of local details but poses difficulties in modeling global contextual information, which limits the overall effectiveness of these segmentation methods [6,7,8,9,10].

Vision Transformers (ViTs) [11] have shown promise in enhancing the capability of networks to model long-range dependencies via self-attention mechanisms, effectively extracting global contextual information [12,13]. This has led to the development of improved methods based on ViTs, such as DeiT [14], PVT [15], and Swin-Transformer [16], which have been successfully applied in various vision-related tasks, demonstrating the efficacy of Transformers in visual applications. Inspired by these developments, several innovative approaches have integrated Transformers into networks specifically for MIS. Examples include Swin-UNet [17], TransDeepLab [18], and MISSFormer [19], which employ Transformers throughout both the encoding and decoding phases. While these methods excel at capturing global features, they often neglect local detail information. In response, some strategies have combined Transformers with CNN-based networks, such as TransUNet [20] and TransCeption [21], where Transformers are typically appended after the CNN encoding module. Although these hybrid approaches merge the strengths of CNNs and Transformers, they struggle to effectively integrate the features globally extracted by the Transformers with the local details captured by the CNNs, thereby compromising their segmentation capabilities. In contrast, other methods, like TransFuse [22], CTC-Net [23], and HiFormer [24], link CNNs and Transformers in parallel to concurrently extract global and local features. However, these methods frequently rely on simplistic feature fusion and lack interactive learning between the two encoder types, which prevents them from fully exploiting the complementary benefits of CNNs and Transformers.

Given the limitations inherent in existing hybrid architectures, which struggle to effectively combine the local detail extraction capabilities of CNNs with the global attention capabilities of Transformers, this study introduces a dual-encoder interactive learning network designed specifically for medical image segmentation. This network employs both Transformers and CNNs concurrently to extract image features, wherein the Transformers focus on global feature extraction and the CNNs concentrate on capturing local detail information. To better leverage the strengths of both architectures, we introduce the global–local interaction learning (GLIL) module, which facilitates interaction between the Transformer and CNN encoders. Additionally, a global–local feature fusion (GLFF) module is utilized to integrate features from both encoders, thereby enhancing the fusion process and achieving a more accurate feature representation. Extensive experiments were conducted on the Kvasir-SEG dataset [25], the ACDC cardiac dataset [26], and the Synapse multi-organ dataset [27] for validating the proposed method’s effectiveness.

The contributions of this study are presented below:

• The introduction of a novel network, DEFI-Net, notably enhancing the MIS performance through Transformer–CNN interactive learning. This interaction is facilitated by the GLIL module. The GLIL module consists of two parallel spatial attention modules: one module extracts global semantic information from the features of the Transformer encoder while the other captures local detail information from the features of the CNN encoder. Effective interaction between the two encoders is achieved by using the global semantic information to weight the features from the CNN encoder and refining the features from the Transformer encoder with local detail information.
• The development of a GLFF module that utilizes the global semantic information and local detail information produced by the GLIL module to integrate features from both the Transformer and CNN encoders. The GLFF module concatenates the features from the two encoders and employs a channel attention mechanism to enhance relevant channel features while suppressing less significant ones. The global semantic and local detail information from the GLIL module is then used to weight the concatenated features, resulting in a more precise feature representation.
• An evaluation of the robustness and generalizability of the proposed network through extensive experiments on three distinct medical image datasets: the Synapse multi-organ dataset, the ACDC cardiac dataset, and the Kvasir-SEG dataset. The experimental results demonstrate significant improvements over several existing MIS methods, thus further validating the effectiveness of the proposed approach.

2. Related Work

2.1. CNN for MIS

Convolution-based methods are widely used in medical image segmentation due to their outstanding performance. The U-Net architecture, introduced by Ronneberger et al. [1], features a symmetrical encoder–decoder structure. High-resolution features from the encoder are combined with low-resolution features from the decoder via skip connections, designed to retain detailed information. This has established U-Net as a leading framework for MIS due to its superior performance in this area. Building on this, Zhou et al. [28] developed UNet++, which enhances the U-Net architecture by adding additional skip connection pathways that link each decoder layer to multiple encoder layers. Additionally, a multilevel feature pyramid is created through the integration of cascading and skip connections. However, this architecture significantly increases the number of parameters. To overcome these drawbacks, Huang et al. [29] proposed Unet3+, which features full-scale skip connections and depth-supervised strategies to reduce the parameter count while effectively capturing both coarse and fine-grained features at multiple scales. With the growing use of attention mechanisms in various computer vision tasks, several researchers have explored their integration into medical segmentation networks. Oktay et al. [3] introduced the attention U-Net, which incorporates a gated attention mechanism within the skip connections to better segment small target areas. Similarly, Sinha et al. [30] developed the guided attention module, which includes spatial and channel self-attention submodules that help to adaptively integrate features across multiple scales. Drawing inspiration from ResNet [31], the Res-UNet [32] combines weighted attention mechanisms, skip connections, and residual connections within the U-Net architecture. This combination facilitates more effective feature modeling, enhancing the distinction between vascular and non-vascular structures and improving the accuracy of retinal vascular segmentation. Meanwhile, MultiResUNet [33] modifies the standard U-Net by replacing its convolutions with multi-scale convolution kernels and incorporating residual structures into the skip connections, achieving enhanced segmentation performance. Despite these advancements, convolutional neural networks, limited by the size of their convolutional kernels, still struggle to model long-range dependencies and effectively capture global contextual information.

2.2. Vision Transformer for MIS

In recent years, Transformer-based network architectures have proven to be highly effective in advancing the field of MIS. These architectures have attracted considerable attention for their capability to surmount the constraints traditionally associated with CNNs in capturing global contextual information. Among the purely Transformer-based approaches to MIS, TransDeepLab was introduced by Azad et al. [18]. This method, designed specifically for skin damage segmentation, incorporates various windowing strategies within the DeepLab framework. Similarly, Huang et al. [19] presented the MissFormer network, which employs a multi-scale global information fusion mechanism to substantially improve segmentation performance. Additionally, Swin-Unet, developed by Cao et al. [17], utilizes the Swin Transformer as a feature extractor. This model enhances its capability by incorporating a dynamic window mechanism, thus addressing the inherent limitations of the visual Transformer (ViT) in managing local inter-window interactions. Moreover, DS-TransUNet, proposed by Lin et al. [34], features a dual-branch Swin Transformer encoder to extract multi-scale features. It includes an innovative Transformer interactive fusion (TIF) module designed to effectively integrate multi-scale features from the encoder. This is achieved by fostering long-range dependencies between features of varying scales through a self-attention mechanism. In the realm of 3D medical image processing, the UNETR network introduced by Gao et al. [35] integrates a Transformer architecture with a U-shaped network design. This approach capitalizes on the Transformer encoder’s ability to serialize representations of 3D volumetric data, thereby significantly enhancing the capture of global multi-scale features. Despite these significant advancements, the application of Transformers in modeling long-range dependencies continues to face challenges, particularly in local feature extraction, where CNNs still hold a comparative advantage, especially in tasks involving MIS.

2.3. Combining CNN and Transformer for MIS

In recent years, efforts have been made to fully leverage the strengths of CNNs and Transformer networks by integrating these two architectures for MIS. One notable example is TransUnet [20], which innovatively combines Transformer networks and CNNs in a sequential manner. This model uses the Transformer network to process feature maps produced by the CNN, enhancing segmentation performance through capturing global information via a self-attention mechanism. Similarly, Gao et al. [36] introduced UTNet, which features an efficient self-attention module in its encoder to minimize the computational demands of capturing multi-scale long-range dependencies. Additionally, UTNet incorporates a unique self-attentive decoder designed to restore fine-grained details from skip connections. Contrasting with this serial integration, some recent studies have adopted a parallel approach to simultaneously extract both global and local features. For example, Zhang et al. [22] developed the TransFuse network, which uses separate encoders for CNNs and Transformers. These encoders independently extract image features, which are then fused and relayed to the decoder via a BiFusion module. In a similar vein, Yuan et al. [23] proposed a dual-encoder hybrid model combining CNN and Transformer technologies, where both encoders generate features that are subsequently fused and forwarded to the decoder for up-sampling through skip connections. Furthermore, Heidari et al. [24] introduced HiFormer, employing a combination of a Swin Transformer and a CNN-based encoder to create two distinct multi-scale feature representations. Additionally, Manzari et al. [37] developed BEFUnet, which enhances robustness and accuracy in edge segmentation through local cross-attention feature fusion and dual-encoder feature integration. Importantly, as depicted in Figure 1a, these parallel approaches typically fuse the features from both encoders in a straightforward manner and directly use them in the decoder, often lacking effective interaction between the different encoders.

2.4. Channels and Spatial Attention Mechanisms

The channel attention mechanism is predominantly utilized to process feature maps produced by CNNs. It concentrates on discerning the significance of individual channels within the input feature maps, thereby selectively focusing on the channel dimensions. The initial implementation of this mechanism is exemplified by the SENet model, introduced by Hu et al. [38]. This model incorporates the squeeze-and-excitation (SE) module, which fundamentally computes the weights for each channel using global average pooling and fully connected layers. These weights are then leveraged to modulate the network’s focus across various channels. To mitigate the computational burden imposed by the linear layers in the SE module, Wang et al. [39] developed the efficient channel attention (ECA) module. This module utilizes 1D convolution with a specific kernel size, K, to efficiently derive channel attention from the feature map. In comparison to the SE block, the ECA module requires fewer parameters and its 1D convolution dynamically adjusts the channel weights. However, the channel attention mechanism predominantly centers on the relevance of individual channels, often overlooking spatial information. This oversight can lead to the omission of crucial structural details. Such limitations are effectively countered by the spatial attention mechanism. The pioneering application of spatial attention was the spatial Transformer network (STN), proposed by Jaderberg et al. [40] in 2015. This network allows the model to dynamically concentrate on varying regions of the input by learning spatial transformation parameters. To amalgamate both spatial and channel attention mechanisms, Woo et al. [41] introduced the convolutional block attention module (CBAM). This module is a sophisticated attention mechanism that sequentially integrates channel and spatial attention, enhancing the model’s ability to comprehend features more holistically and capture complex interdependencies among them.

3. Methodology

Figure 2 presents an MIS network, designated as DEFI-Net, which is founded on a dual-encoder interactive fusion approach. Distinct from conventional methodologies that employ Transformers and CNNs in parallel and directly harness the fused features, DEFI-Net not only amalgamates features from disparate encoders but also promotes interactive learning between the Transformer and CNN. During the encoding phase, we construct a parallel encoder featuring a three-layer pyramid structure that incorporates both CNN and Transformer technologies for extracting multi-scale features from the input image. Subsequently, the GLIL module is utilized to facilitate interaction between the Transformer encoder and the CNN encoder at each pyramid layer, with the learned features being transmitted to subsequent, deeper layers of the network. Concurrently, the global–local feature fusion (GLGF) module is applied to efficiently amalgamate features from both encoders. Additionally, skip connections are implemented to aggregate features across various scales, culminating in the decoding process to generate the final segmentation output.

3.1. Transformer–CNN Dual-Encoder

Motivated by existing medical segmentation networks such as TransFuse [22] and CTC-Net [23], which utilize parallel encoders, we advance a dual-encoder architecture that capitalizes on interactive learning and feature fusion. This design is intended to enhance the network’s segmentation performance by exploiting the complementary strengths of different encoding systems. For the encoding phase, a dual-encoder network with a three-layer pyramid structure is established to effectively extract multi-scale features. In the Transformer encoder branch, for the input image $X\in {\mathbb{R}}^{H\times W\times C^{\prime }}$, patch embedding layers $P{E}_i$ with patch sizes of 4, 2, and 2 are sequentially applied between different Transformer blocks $T{F}_i$. This arrangement produces feature maps of sizes ${\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}\times C^{\prime }$, ${\displaystyle \frac{H}{8}}\times {\displaystyle \frac{W}{8}}\times 2C^{\prime }$, and ${\displaystyle \frac{H}{16}}\times {\displaystyle \frac{W}{16}}\times 4C^{\prime }$, while the Transformer layers focus on extracting global features $F_t^i$.

$$F_t^1=T{F}_1\left(P{E}_1\left(X\right)\right)\in {\mathbb{R}}^{\frac{H}{4}\times \frac{W}{4}\times C^{\prime }}$$

(1)

$$F_t^2=T{F}_2\left(P{E}_2\left(F_t^1\right)\right)\in {\mathbb{R}}^{\frac{H}{8}\times \frac{W}{8}\times 2C^{\prime }}$$

(2)

$$F_t^3=T{F}_3\left(P{E}_3\left(F_t^2\right)\right)\in {\mathbb{R}}^{\frac{H}{16}\times \frac{W}{16}\times 4C^{\prime }}$$

(3)

In the CNN encoder branch, ResNet-34 is directly implemented to extract multi-scale local features from images. The 2nd, 3rd, and 4th stages of ResNet align with the three blocks of the Transformer, maintaining consistent feature scales across both encoder branches.

3.2. Global–Local Interaction Learning Module

The exclusive use of either CNNs or Transformers as the encoding mechanism often fails to strike an optimal balance between extracting global features and capturing local detail features. Furthermore, existing medical segmentation networks that utilize hybrid CNN–Transformer architectures tend to merely fuse features from different encoders and directly employ them in the decoding phase, thereby overlooking the potential synergistic interactions among the encoders. To address this limitation and effectively harness the strengths of both CNNs and Transformers, the GLIL module has been developed. This module facilitates the exchange of information between the CNN and Transformer, promoting interactive learning. The GLIL module comprises two parallel spatial attention modules [41]: one extracts global information from the Transformer’s output features and the other captures detailed local information from the CNN’s output features. For an input feature F within the dimensions ${\mathbb{R}}^{H\times W\times C^{\prime \prime }}$, the spatial attention module conducts average pooling and max pooling along the channel dimension to generate compressed feature maps $F_{avg}$ and $F_{max}$, respectively, both in ${\mathbb{R}}^{H\times W\times 1}$.

$$F_{avg}=AvgPool\left(F\right)$$

(4)

$$F_{max}=MaxPool\left(F\right)$$

(5)

These compressed maps, $F_{avg}$ and $F_{max}$, are then concatenated along the channel dimension to form $F_{concat}\in {\mathbb{R}}^{H\times W\times 2}$. Next, a convolutional operation followed by a sigmoid activation function $\sigma$ is used for deriving the attention weights $\omega \in {\mathbb{R}}^{H\times W\times 1}$ for the input feature map F.

$$F_{concat}=Concat[F_{avg},F_{max}]$$

(6)

$$\omega =\sigma \left(Conv\left(F_{concat}\right)\right)$$

(7)

This architecture allows for the parallel extraction of global information ${\omega }_t$ from the Transformer and local detail knowledge ${\omega }_c$ from the CNN. To facilitate effective interaction between the Transformer and CNN, the global information is utilized for modulating the features $F_c$ extracted by the CNN while the local detail knowledge is used for modulating the features $F_t$ extracted by the Transformer. Additionally, residual connections are employed to enhance the stability of the interactive learning process.

$$F_c=F_c+\alpha \times (F_c\otimes {\omega }_t)$$

(8)

$$F_t=F_t+\beta \times (F_t\otimes {\omega }_c)$$

(9)

Through the implementation of the GLIL module, the output features of the Transformer and CNN are enabled to mutually enhance their learning and propagate effectively to the subsequent layer of the encoder. This method significantly enriches the interaction between the Transformer and CNN encoders, facilitating a more integrated and effective use of their respective advantages within the network.

3.3. Global–Local Feature Fusion Module

To enhance the integration of features from disparate encoders, we have developed a GLFF module. This module capitalizes on both global and local information to facilitate a more effective amalgamation of features. Specifically, the feature outputs from the Transformer, denoted as $F_t$, and the CNN, denoted as $F_c$, are combined through concatenation to yield the fused feature $F_M$:

$$F_M=Concat[F_t,F_c]$$

(10)

The GLFF module incorporates a channel attention mechanism [38] that prioritizes salient channel features while attenuating the less relevant ones. Initially, average pooling is applied along the spatial dimension to derive the channel-wise representation of the fused features, denoted as $F_{M_{a}vg}\in {\mathbb{R}}^{C\times 1\times 1}$. Following this, a nonlinear transformation is applied to $F_{M_{a}vg}$ using the sigmoid function $\sigma$ and a multi-layer perceptron (MLP), resulting in the generation of channel attention weights $\phi \in {\mathbb{R}}^{C\times 1\times 1}$. These weights are then applied element-wise to the concatenated features $F_M$ to highlight significant channel characteristics:

$$F_{M_{a}vg}=AvgPool\left(F_M\right)$$

(11)

$$\phi =\sigma \left(W_2\delta \left(W_1{\mathrm{F}}_{{\mathrm{M}}_{\mathrm{a}}\mathrm{vg}}\right)\right)$$

(12)

$$F_{M_{C}A}=F_{M_{a}vg}\otimes \phi$$

(13)

Here, $W_1$ and $W_2$ are the weight matrices corresponding to the fully connected layers, with $\delta$ denoting the ReLU activation function.

Subsequently, the integrated features are modulated using the global semantic information and local detail knowledge derived from the GLFF module, leading to a richer feature representation:

$$F_{Fusion}=F_{M_{C}A}+\gamma (({\omega }_t+{\omega }_c)\otimes F_{M_{C}A})$$

(14)

3.4. Loss Function

The model employs a composite loss function, which is a weighted sum of the CE loss and Dice loss, designated as the total loss:

$${\mathrm{L}}_{\mathrm{total}}=\lambda {\mathrm{L}}_{\mathrm{CE}}+(1-\lambda ){\mathrm{L}}_{\mathrm{Dice}}$$

(15)

Here, $L_{total}$ represents the total loss, $L_{CE}$ denotes the cross-entropy loss, and $L_{Dice}$ refers to the Dice loss. For the purposes of our experiment, we set $\lambda$ to 0.5.

4. Experiments and Results

4.1. Implementation Details

The experiments conducted utilized the DEIF-Net on a single NVIDIA RTX 3090 GPU. Optimization of the network parameters was achieved using the SGD optimizer. The experiments were run with a batch size of four across 150 epochs. The initial learning rate was set at 0.01, which was reduced by multiplying it by 0.9 at regular intervals during the training process. All inputs were uniformly resized to 224 × 224 pixels. We applied simple data augmentations such as random rotation and flip. The hyperparameters within the network were set as $\alpha =\beta =\gamma =0.5$. In the Transformer block, the channel dimension $C^{\prime }$ was set to 320.

4.2. Datasets

4.2.1. Synapse Multi-Organ Segmentation

The Synapse multi-organ dataset [27], originating from the MICCAI 2015 Multi-Atlas Abdomen Labeling Challenge, consists of 30 abdominal CT scans. Each scan is annotated to identify eight distinct organs: the stomach, spleen, pancreas, liver, right kidney, left kidney, gallbladder, and aorta. Importantly, each CT scan comprises between 85 and 198 axial slices, totaling 3779 slices across the dataset. In alignment with the data configuration used by TransUnet [20], the Synapse dataset was divided into 18 training cases (encompassing 2211 axial slices) and 12 test cases.

4.2.2. Automated Cardiac Diagnosis Challenge (ACDC) Segmentation

The ACDC [26], initiated by MICCAI in 2017, focuses on the segmentation of the myocardium (Myo), right ventricle (RV), and left ventricle (LV) in cardiac MRI data. The dataset features short-axis slices, extending from the base to the apex of the heart, with slice thicknesses ranging from 5 to 8 mm and spatial resolutions between 0.83 and 1.75 mm² per pixel. For experimental purposes, the dataset was segmented as 20 test cases, 10 validation cases, and 70 training cases (comprising 1930 axial slices), following the data setup of TranUnet [20].

4.2.3. Kvasir-SEG Dataset

The Kvasir-SEG dataset [25] consists of 1000 annotated images of colon polyps, featuring a range of resolutions from 332 × 487 to 1920 × 1072 pixels. The dataset was partitioned such that eighty percent was randomly allocated as the training set while the remaining twenty percent was evenly divided between the validation and test sets. These subsets facilitated the evaluation of the model’s performance.

4.3. Evaluation Metrics

In this research, several evaluation metrics were utilized to ascertain the effectiveness of the model. These metrics included the Dice similarity coefficient (DSC), the 95th percentile Hausdorff distance (HD95), and the intersection over union (IoU) coefficient. Each metric is elaborated upon in the subsequent sections.

4.3.1. Dice Similarity Coefficient (DSC) and Intersection over Union (IoU)

The DSC and the IoU are pivotal metrics for assessing the performance of segmentation models. These metrics measure the extent of overlap between the predicted outcomes and the ground truth labels, with values ranging from 0 to 1. A value of 0 indicates no overlap, whereas a value of 1 denotes perfect alignment, with higher values indicative of superior model performance.

The DSC is computed according to the following formula:

$$\mathrm{DSC}(T,P)={\displaystyle \frac{2|P\cap T|}{\left|P\right|+\left|T\right|}}$$

Here, P and T represent the sets of predicted and ground truth pixels, respectively, $|P\cap T|$ is the intersection of these sets, and $\left|P\right|+\left|T\right|$ is the sum of pixels in both sets.

The IoU is defined by the subsequent equation:

$$\mathrm{IoU}(T,P)={\displaystyle \frac{|P\cap T|}{|P\cup T|}}$$

In this formula, $|P\cup T|$ represents the union of the predicted and ground truth pixel sets.

4.3.2. Hausdorff Distance at the 95th Percentile (HD95)

The Hausdorff distance (HD) serves as a metric for evaluating the congruence of boundaries between segmentation results and their corresponding ground truths. It measures the distance between the peripheries of these entities, thereby focusing on the precision of boundary delineation. In contrast to the DSC, the HD places a greater emphasis on the accuracy of boundary localization.

For two non-empty sets, A and B, the HD is defined by the following equation:

$$HD(A,B)=max\left\{\underset{a\in A}{sup}\underset{b\in B}{inf}d(a,b),\underset{b\in B}{sup}\underset{a\in A}{inf}d(a,b)\right\}$$

Here, $d(a,b)$ represents the Euclidean distance between points a and b. Essentially, HD computes the greatest of all the shortest distances from points in one set to the nearest points in the other set. In the context of segmentation tasks, it is common to use HD95, which quantifies the maximum distance for 95% of the points in set A to their nearest points in set B. A superior alignment of the segmentation result with the ground truth can be indicated by a lower HD95 value.

4.4. Results on Synapse Multi-Organ Segmentation

For evaluating the segmentation efficacy of DEFI-Net on multi-organ medical images, we carried out a series of experiments using the Synapse dataset. The performance of DEFI-Net was benchmarked against numerous leading-edge segmentation methods. The comparative analysis included various architectures, such as CNN-based networks (e.g., U-Net [1], Att-UNet [3], Res-UNet [32]) and purely Transformer-based networks (e.g., Swin U-Net [17], TransDeepLab [18], MISSFormer [19])), along with hybrid models employing serial (e.g., TransUNet [20], TransCeption [21]) and parallel CNN–Transformer configurations (e.g., CTC-Net [23], TransFuse [22], HiFormer [24]). As depicted in

Table 1. Comparison results of the proposed method on the Synapse dataset, with the best results indicated in red and the second best in blue.

Methods	DSC (%) ↑	HD (mm) ↓	Aorta	Gallbladder	Kidney (L)	Kidney (R)	Liver	Pancreas	Spleen	Stomach
V-Net [2]	68.81	-	75.34	51.87	77.10	80.75	87.84	40.05	80.56	56.98
U-net [1]	76.85	39.70	89.07	69.72	77.77	68.60	93.43	53.98	86.67	75.58
Att-Unet [3]	77.77	36.02	89.55	68.88	77.98	71.11	93.57	58.04	87.30	75.75
Res-Unet [32]	78.56	22.51	87.95	53.44	84.89	87.38	93.74	52.56	90.65	78.21
TransUnet [20]	77.48	31.69	87.23	63.13	81.87	77.02	94.08	55.86	85.08	75.62
TransNorm [42]	78.40	30.25	86.23	65.10	82.18	78.63	94.22	55.34	89.50	76.01
CTC–Net [23]	78.41	-	86.46	63.53	83.71	80.79	93.78	59.73	86.87	72.39
Swin U-Net [17]	79.13	21.55	85.47	66.53	83.28	79.61	94.29	56.58	90.66	76.60
TransDeepLab [18]	80.16	21.25	86.04	69.16	84.08	79.88	93.53	61.19	89.00	78.40
TransFuse [22]	80.31	30.76	87.99	69.12	83.19	80.06	93.80	62.83	88.69	76.82
HiFormer [24]	80.39	14.70	86.21	65.69	85.23	79.77	94.61	59.52	90.99	81.08
MISSFormer [19]	81.96	18.20	86.99	68.65	85.21	82.00	94.41	65.67	91.92	80.81
TransCeption [21]	82.24	20.89	87.60	71.82	86.23	80.29	95.01	65.27	91.68	80.02
DEFI-Net (ours)	83.29	18.15	88.38	71.36	86.05	81.51	95.05	68.69	92.45	82.81

, methods such as U-Net and Att-U-Net, primarily based on CNNs, effectively segment the aorta but struggle with organs that exhibit large variations in shape and size. DEFI-Net demonstrated superior segmentation accuracy across multiple organs, achieving the highest average Dice similarity coefficient (DSC) of 83.29% and a competitive 95% Hausdorff distance (HD) of 18.15 mm. This performance surpasses that of both purely CNN-based and Transformer-based approaches, underscoring the synergistic benefits derived from integrating CNN and Transformer encoders, which significantly enhance segmentation accuracy across most organs. Moreover, DEFI-Net displayed superior segmentation outcomes when compared to hybrid models using both serial and parallel combinations. This highlights the critical role of effective encoder interactions in achieving enhanced model performance. Notably, DEFI-Net exhibited remarkable improvements in the segmentation of the pancreas, spleen, and stomach. The most significant enhancement was observed in the pancreas segmentation, where the DSC increased by 3.02% relative to other evaluated methods. In contrast to models like Swin-UNet [17] and HiFormer [24], DEFI-Net does not rely on pre-trained weights for initializing the Transformer module, further emphasizing its robust feature extraction capabilities and the efficacy of its feature interaction and fusion strategies at each stage.

Figure 3 presents various methods’ segmentation results. It is apparent that conventional convolution-based networks like U-Net [1] and Att-UNet [3] are less effective in accurately identifying and segmenting organs of varying sizes, such as the pancreas and stomach, often leading to under-segmentation or erroneous delineations. Similarly, Swin-UNet [17], TransUNet [20], and HiFormer [24] falter in stomach and pancreatic segmentation, failing to completely delineate the organs, showing issues such as under-segmentation and irregular boundaries. In stark contrast, DEFI-Net exhibits better performance in multi-organ segmentation tasks, accurately identifying and segmenting the pancreas and stomach with more precise, smooth-edged, and complete segmentation areas. These visualization results are consistent with the empirical data, further validating DEFI-Net’s substantial improvements in segmenting organs of varying sizes, particularly the pancreas and stomach.

4.5. Results on ACDC Segmentation

The DEIF-Net method was assessed using the ACDC dataset and compared with various other segmentation approaches. The results of these comparisons are detailed in

Table 2. Comparative results of the developed method on the ACDC dataset, with the second highest performance indicated by blue and the highest performance indicated by red.

Methods	DSC (%) ↑	HD (mm) ↓	RV	Myo	LV
U-Net [1]	89.36	1.81	90.25	83.28	94.57
Att-UNet [3]	90.28	1.56	90.38	84.89	95.58
TransUNet [20]	89.71	1.16	88.86	84.53	95.73
CTC–Net [23]	90.77	-	90.09	85.52	96.72
Swin U-Net [17]	90.00	1.32	88.55	85.62	95.83
HiFormer [24]	90.43	1.14	88.75	87.35	95.20
MISSFormer [19]	87.90	2.25	86.34	82.75	94.62
DAE-Former [43]	90.24	1.68	89.32	86.38	95.04
TransCeption [21]	88.73	1.72	87.68	83.87	94.66
CS-Unet [44]	91.37	-	89.20	89.47	95.42
DEIF-Net (ours)	92.02	1.08	90.75	89.27	96.04

. DEIF-Net outperformed others in DSC and HD95, achieving values of 92.02% and 1.08 mm, respectively. It also recorded the highest DSC accuracy for ventricular segmentation. In particular, DEIF-Net demonstrated improvements of 2.66% and 1.74% in DSC accuracy over the Unet [1] and Att-Unet [3] models, both of which employ convolutional network architectures. Furthermore, in contrast with Transformer-based models such as Swin-Unet [17] and MISSFormer [19], DEIF-Net showed enhancements of 2.02% and 4.12%, respectively. Additionally, it outperformed the hybrid Transformer and CNN-based models TransUnet [20] and HiFormer [24] by 2.31% and 1.59%, respectively. It is important to note that several methods utilizing hybrid Transformer and CNN architectures did not demonstrate significant advancements over traditional convolutional networks like Unet and Att-Unet. By contrast, DEIF-Net capitalizes on the synergistic interaction and feature fusion between different encoder types, effectively harnessing the strengths of both, which culminates in marked improvements in segmentation efficacy.

Further evaluations of DEIF-Net’s performance in cardiac organ segmentation were conducted through visual assessments, as depicted in Figure 4. DEIF-Net and other methods were generally successful in accurately segmenting the left ventricle and myocardium. However, the segmentation of the right ventricle posed challenges for several models, which exhibited issues such as incomplete segmentation and non-smooth segmentation boundaries. For example, Att-Unet [3] consistently failed to fully segment the right ventricle in various cases, while HiFormer [24], Swin-Unet [17], and TransUnet [20] struggled with effectively delineating the segmentation edges in multiple instances. In contrast, DEIF-Net not only accomplished complete segmentation of the right ventricle but also excelled at delineating the edges of various tissues, demonstrating reduced misdetection rates compared to the models mentioned.

4.6. Results on Kvasir-SEG Segmentation

For the evaluation of the efficacy of DEFI-Net, we carried out a series of experiments using the Kvasir-SEG dataset, with performance benchmarks compared against multiple existing approaches. DEFI-Net achieved the highest scores in both DSC and IoU, with results of 87.45% and 80.51%, respectively (

Table 3. Comparison results of the proposed method on the Kvasir-SEG dataset, with the best outcomes highlighted in red and the second best in blue.

Methods	DSC (%) ↑	IoU (%) ↑
V-Net [2]	80.79	68.07
U-Net [1]	82.99	71.01
Att-UNet [3]	82.85	70.45
MultiResUnet [33]	81.34	68.66
TransUnet [20]	85.61	78.15
SwinU-Net [17]	85.44	78.21
MISSFormer [19]	75.35	62.64
TransCeption [21]	77.32	65.21
HiFormer [24]	85.96	78.26
DEIF-Net	87.45	80.51

). In comparison to convolutional-network-based methods, including Att-UNet [3] and U-Net [1], DEFI-Net exhibited significant enhancements, improving DSC by 4.46% over U-Net and 4.60% over Att-UNet. Additionally, DEFI-Net surpassed hybrid CNN–Transformer methods, including TransUNet [20] and HiFormer [24]. Specifically, it improved DSC and IoU by 1.84% and 2.36% over TransUNet and by 1.49% and 2.25% over HiFormer, respectively. These findings underscore DEFI-Net’s superior segmentation capabilities on the Kvasir-SEG dataset.

Furthermore, the segmentation results of DEFI-Net and other comparative methods on the Kvasir-SEG dataset were visualized. As depicted in Figure 5, DEFI-Net demonstrated better performance in segmenting colon polyp images, outperforming other methods. Notably, while most competing methods encountered issues with incomplete segmentation, DEFI-Net effectively mitigated this challenge, thereby reinforcing its effectiveness in colon polyp image segmentation.

4.7. Ablation Experiments

For ascertaining the effectiveness of specific components within the DEFI-Net architecture, a series of ablation experiments were performed across various datasets. These experiments focused on evaluating the impact of disabling the GLIL and GLFF modules. A baseline network was established by directly concatenating the outputs of the encoder to assess the contributions of these modules. As demonstrated in

Table 4. Ablation experiments on three datasets.

Model	Synapse		ACDC		Kvasir SEG
	DSC ↑	HD95 ↓	DSC ↑	HD95 ↓	DSC ↑	IoU ↑
Concat	80.98	24.17	88.72	1.44	85.54	78.23
+GLIL	82.64	18.45	91.83	1.11	86.50	79.16
+GLFF	81.65	19.21	91.62	1.18	86.14	78.64
DEIF-Net	83.29	18.15	92.02	1.08	87.45	80.51

, integrating the GLIL module significantly enhanced segmentation performance, as evidenced by notable improvements in DSC and HD on the Synapse and ACDC datasets, and in DSC and IoU on the Kvasir-SEG dataset. These results highlight the critical role of encoder interactions and confirm that the GLIL module effectively integrates complementary features from both CNN and Transformer architectures. Conversely, when the GLIL module was disabled and replaced with the GLFF module, there was an improvement in segmentation accuracy, indicating that the GLFF module effectively amalgamates multi-scale encoder features to augment model performance. Ultimately, the highest segmentation performance was realized when all proposed modules were integrated, further validating the synergistic effect of combining the GLIL and GLFF components within the DEFI-Net framework.

4.8. Conclusions

A hybrid CNN–Transformer architecture named DEFI-Net that exhibits superior segmentation performance in MIS tasks is introduced. This advantage arises from effective interaction and feature fusion between the CNN and Transformer encoders. The interaction within DEFI-Net is facilitated by the GLIL, which allows the CNN to absorb global information from the Transformer and vice versa, enabling the Transformer to acquire local detail knowledge from the CNN. This promotes a robust interaction between the two encoders. Feature fusion is accomplished through the GLFF, which harnesses both global and local information to enhance the integration of features. Evaluations conducted on three distinct datasets underscore the superior performance of DEFI-Net compared to other methods. Additionally, ablation experiments validate the critical role of encoder interactions in achieving high segmentation accuracy. However, the dual-encoder structure of DEFI-Net introduces significant computational overhead. Future work will therefore focus on developing a more efficient network structure inspired by DEFI-Net, aiming to maintain high accuracy while reducing computational costs.