3D Multi-Organ and Tumor Segmentation Based on Re-Parameterize Diverse Experts

Abstract

Automated segmentation of abdominal organs and tumors in medical images is a challenging yet essential task in medical image analysis. Deep learning has shown excellent performance in many medical image segmentation tasks, but most prior efforts were fragmented, addressing individual organ and tumor segmentation tasks with specialized networks. To tackle the challenges of abdominal organ and tumor segmentation using partially labeled datasets, we introduce Re-parameterizing Mixture-of-Diverse-Experts (RepMode) to abdominal organ and tumor segmentation. Within the RepMode framework, the Mixture-of-Diverse-Experts (MoDE) block forms the foundation, learning generalized parameters applicable across all tasks. We seamlessly integrate the MoDE block into a U-shaped network with dynamic heads, addressing multi-scale challenges by dynamically combining experts with varying receptive fields for each organ and tumor. Our framework incorporates task encoding in both the encoder–decoder section and the segmentation head, enabling the network to adapt throughout the entire system based on task-related information. We evaluate our approach on the multi-organ and tumor segmentation (MOTS) dataset. Experiments show that DoDRepNet outperforms previous methods, including multi-head networks and single-network approaches, giving a highly competitive performance compared with the original single network with dynamic heads. DoDRepNet offers a promising approach to address the complexities of abdominal organ and tumor segmentation using partially labeled datasets, enhancing segmentation accuracy and robustness.

1. Introduction

Automated segmentation of abdominal organs and tumors in medical images is a key yet intricate task within medical image analysis [1,2,3]. This task holds significant importance in various computer-aided diagnosis applications, encompassing tasks such as lesion delineation, 3D reconstruction, and surgical planning. For abdominal multi-organ images, creating extensive fully annotated datasets for abdominal multi-organ images is a formidable undertaking, demanding both significant resources and time, especially in the case of 3D segmentation tasks. Currently, most benchmark datasets are typically limited in sample size, and most of them only annotate one or a few organs, rather than all abdominal organs, designating all task-irrelevant structures as background. For example, the pancreas-CT dataset exclusively provides labels for the pancreas and pancreas tumors, while the hepatic vessel dataset offers labels solely for hepatic vessels and tumors [4].

Recently, deep learning has made remarkable strides in medical image segmentation [5,6,7,8,9,10]. Previous abdominal organ and tumor segmentation tasks were typically addressed individually using specialized networks [11,12,13,14], leading to a dispersion of research efforts. Effectively acquiring representations of multiple organs and tumors by taking advantage of partially labeled datasets to enhance segmentation accuracy and robustness has attracted great interest. Broadly, prior approaches can be categorized into two main groups. One focuses on devising effective training strategies, which involve techniques such as knowledge distillation [15], cross-task consistency learning [16], and the design of jointly optimized loss functions [17]. The other strives to enhance a network’s structure [18,19]. They can be further grouped into multi-head networks [20] and a single network with dynamic heads [18,19]. A brief summary is provided in Table 1.

Multi-head networks consist of a partially shared architecture, incorporating a shared feature extractor and several task-dedicated decoders. Thanks to the common encoder, they can leverage the rich information from various partially annotated datasets. Nonetheless, multi-head networks face challenges in effectively co-training for multiple tasks, mainly due to the structural redundancy resulting from utilising individual decoders for each task. A single network with dynamic heads presents a versatile architecture designed to address various segmentation tasks by utilizing task encoding and a controller for generating segmentation heads equipped with dynamic convolutions. However, these models have a few limitations. Task-specific information, namely the dynamic convolution parameters, is set at the decoder’s end. This timing may pose challenges in making the model fully aware of the current task, especially when decoding sophisticated objectives. In addition, it is worth noting that abdominal organs and tumors naturally exhibit significant variations in size, giving rise to the multi-scale challenge. Currently, segmentation accuracy for small and highly deformable organs such as the pancreas, tubular structures, and tumors, is generally below 90%.

To address the above challenges inherent in abdominal organ and tumor segmentation using partially labeled datasets, we have adopted a unique approach known as Re-parameterizing Mixture-of-Diverse-Experts (RepMode) [21]. Its key feature is the dynamic organization of its parameters through the use of task-aware priors. Within the RepMode framework, the Mixture-of-Diverse-Experts (MoDE) block serves as a foundational component, designed to learn generalized parameters that can be applied across all the tasks. We have integrated RepMode into DoDNet, obtaining DoDNetRep, to suit the demands of abdominal organ and tumor segmentation, where each partially labeled dataset may encompass multiple labels, as illustrated in Figure 1.

In our framework, we make use of task encoding in both the encoder–decoder section and the segmentation head. This strategy enables the network to adapt its behavior throughout the entire system, including the multi-stage encoder–decoder section and the segmentation head, based on task-related information. Notably, we have seamlessly integrated the MoDE block at each stage of both the encoder and decoder within the DoDNet. This adaptation effectively addresses the multi-scale challenges inherent in the segmentation of abdominal organs and tumors. The integration empowers our network to combine experts with varying receptive fields dynamically for each specific organ and tumor, facilitating the learning of multi-scale features in a task-oriented way. The dynamic kernels within the segmentation head are generated adaptively by a controller, with this process being conditioned on the assigned segmentation task. Task-specific priors guide the controller in generating dynamic head kernels for each segmentation task, ensuring that our network can effectively handle the intricacies of abdominal organ and tumor segmentation from a set of partially annotated datasets. We performed experiments on the multi-organ and tumor segmentation (MOTS) dataset [18]. Our model outperformed both multi-head networks and previous present single-network methods, and obtained highly competitive performance compared with the original single network with dynamic heads [18]. DoDRepNet offers a promising approach to address the complexities of abdominal organ and tumor segmentation using partially labeled datasets, enhancing segmentation accuracy and robustness. Our contributions can be summarized as follows:

(1) We introduced RepMode into DoDNet, obtaining DoDNetRep, for abdominal organ and tumor segmentation using partially labeled datasets. DoDNetRep takes advantage of task-related information in both the encoder–decoder section and the dynamic segmentation head, enabling our network to combine experts with varying receptive fields dynamically for each specific organ and tumor, thus facilitating the learning of multi-scale features in a task-oriented way.

(2) We demonstrated that our model outperformed both multi-head networks and previous present single-network methods, and obtained a highly competitive performance compared with the original single network with dynamic heads on MOTS.

Table 1. Related work for medical image segmentation from partially labeled datasets.

Authors	Methods	Datasets
Zhang, L. et al. [15]	A multi-teacher knowledge distillation framework leveraging the soft labels	KiTS [22], MSD Spleen and Pancreas [23], TCIA [24], BTCV [25]
Shi, G. et al. [17]	Design of jointly optimized marginal loss and exclusion loss	BTCV [25], MSD Liver, MSD Spleen, MSD Pancreas [23], KiTS [22]
Chen., S. et al. [20]	Multi-head: transfer learning	LIDC [26], LiTS [27]
Fang, X. et al. [28]	Multi-head: pyramid input pyramid output feature abstraction network and a target adaptive loss	BTCV [25], LiTS [27], KiTS [22] and MSD Spleen [23]
Zhang, G. et al. [29]	Single network: conditional nnU-Net with a conditioning strategy for the decoder	LiTS [27], MSD Pancreas, MSD Spleen [23], KiTS [22], SLIVER07 [30], NIH pancreas [31], BTCV [25]
Zhang, J. et al. [18]	Single network: with dynamic heads leveraging one-hot task embedding	MOTS including LiTS [27], KiTS [22], and MSD Hepatic vessel and tumor, MSD Pancreas and tumor, MSD Colon tumor, MSD Lung tumor and MSD Spleen [23]
Liu, J. et al. [19]	Single network: with dynamic heads leveraging task embedding from Clip	MSD [23] and BTCV [25]

Table 1. Related work for medical image segmentation from partially labeled datasets.

Authors	Methods	Datasets
Zhang, L. et al. [15]	A multi-teacher knowledge distillation framework leveraging the soft labels	KiTS [22], MSD Spleen and Pancreas [23], TCIA [24], BTCV [25]
Shi, G. et al. [17]	Design of jointly optimized marginal loss and exclusion loss	BTCV [25], MSD Liver, MSD Spleen, MSD Pancreas [23], KiTS [22]
Chen., S. et al. [20]	Multi-head: transfer learning	LIDC [26], LiTS [27]
Fang, X. et al. [28]	Multi-head: pyramid input pyramid output feature abstraction network and a target adaptive loss	BTCV [25], LiTS [27], KiTS [22] and MSD Spleen [23]
Zhang, G. et al. [29]	Single network: conditional nnU-Net with a conditioning strategy for the decoder	LiTS [27], MSD Pancreas, MSD Spleen [23], KiTS [22], SLIVER07 [30], NIH pancreas [31], BTCV [25]
Zhang, J. et al. [18]	Single network: with dynamic heads leveraging one-hot task embedding	MOTS including LiTS [27], KiTS [22], and MSD Hepatic vessel and tumor, MSD Pancreas and tumor, MSD Colon tumor, MSD Lung tumor and MSD Spleen [23]
Liu, J. et al. [19]	Single network: with dynamic heads leveraging task embedding from Clip	MSD [23] and BTCV [25]

2. Materials and Methods

2.1. Problem Definition

We consider each partially labeled dataset as a task (organ or tumor segmentation or organ and tumor segmentation) and assume there are P partially labeled dataset in total. Each dataset can have several labels. In this context, ${\mathfrak{D}}_i={\{{\mathbf{V}}_{ij},{\mathbf{L}}_{ij}\}}_{j=1}^{n_i}$ denotes the i-th partially labeled dataset, comprising $n_i$ labeled volumes. Each volume in $\mathfrak{D}i$ is represented as $\mathbf{V}ij\in {\mathbb{R}}^{D\times W\times H}$, where $W\times H$ indicates the dimensions of each slice, while D signifies the slice number. The ground truth segmentation corresponding to this is represented as ${\mathbf{L}}_{ij}$, where the voxel labels belong to the set $0:\mathrm{background};1:\mathrm{organ};2:\mathrm{tumor}$.

To address the challenges in abdominal organ and tumor segmentation using partially labeled datasets, one straightforward approach involves training P segmentation networks, individually on each of the P datasets. The same as in [18], we endeavor to tackle this issue by employing a single network denoted as f. This can be written as follows

$$\underset{\mathit{\theta }}{min}\sum _{i=1}^m\sum _{j=1}^{n_i}\mathcal{L}(f({\mathbf{V}}_{ij};\mathit{\theta }),{\mathbf{L}}_{ij})$$

(1)

Different from Multi-Net, maintaining an individual $\mathit{\theta }$ for each task, we adopt a single network, as in DoDNet [18], generating a dynamic head with parameters ${\mathit{\theta }}_h$ conditioned on the assigned task. What is more, we aim to utilize a shared ${\mathit{\theta }}_{\mathit{f}}$ across all tasks and dynamically structure ${\mathit{\theta }}_{\mathit{f}}$ to accommodate specific task requirements by using re-parameterize diverse experts in the encoder and decoder.

2.2. Network Architecture

Our network, DoDRepNet, which leverages the re-parameterization of diverse experts [21], is composed of several essential components similar to DoDNet [18]: a shared encoder–decoder featuring the MoDE block, a task-encoding module, a dynamic filter generation module, and a dynamic segmentation head (as depicted in Figure 1). In the following sections, we will briefly describe the shared encoder–decoder network. Other modules are the same as those in DoDNet; please reference [18] for details.

The input to DoDRepNet consists of randomly selected samples from the preprocessed dataset, including preprocessed images and annotated images, along with the task identifier corresponding to the subset dataset of that sample. The task identifier serves as input to the task-encoding module, generating a task code represented as a one-hot vector. The preprocessed images and task code are input to a shared encoder–decoder architecture. The multi-stage encoder produces feature maps at each stage, which serve as inputs to the respective stages of the decoder. The final stage of the encoder outputs feature maps that undergo global average pooling (GAP) to obtain high-level image features. These features are concatenated with the task code and serve as input to the dynamic filter generation module, producing a task-specific controller used to generate parameters ${\mathit{\theta }}_h$ for the dynamic segmentation head. ${\mathit{\theta }}_h$, along with the output feature maps from the final stage of the decoder, are passed through the dynamic segmentation head to obtain the output of DoDRepNet, representing the segmented result of the preprocessed image predicted by the network. The loss function is employed to calculate the loss between the predicted result and the annotated image, and the network is trained using optimization algorithms. After training, given any sample (preprocessed image and annotated image) and the corresponding task identifier for that sample’s subset dataset, DoDRepNet can produce the segmentation result for the respective task.

The shared encoder–decoder relies on a 3D U-shaped design, primarily composed of downsampling and upsampling components, as seen in Figure 2. In detail, the downsampling path comprises two consecutive MoDE blocks, which are responsible for capturing task-specific feature maps and increasing the channel number. Following these blocks, there is a downsampling layer that uses a convolution with a $2\times 2\times 2$ kernel size and a stride of 2 to decrease the feature maps’ size by half. It is worth noting that we used group normalization and ReLU activation along with each MoDE block.

Within each upsampling stage, feature maps are enlarged and their channel number is halved using an upsampling layer with a $2\times 2\times 2$ kernel and a stride of 2. Following that, the upsampled feature maps are combined with the corresponding feature maps transmitted from the encoder. These combined feature maps then go through additional enhancement via two successive MoDE blocks. Ultimately, a MoDE block, excluding batch normalization and ReLU, is used to decrease the number of channels to match the number of output classes, with the goal of producing the final predictions for each target class.

It is worth noting that MoDE blocks are used both in the encoder and decoder, effectively promoting the acquisition of task-specific features and thus contributing to superior performance. The detailed information such as the channel sizes of each convolution and output size of feature maps are presented in

Table 2. The detailed information such as the channel sizes of each convolution and output size of feature maps in key layers of DoDRepNet. Upsampling in each stage of the decoder is not listed in this table for simplicity.

Stage	Layer Name	In Channel Size	Out Channel Size	Stride	Output Size
Encoder	Input	1	-	-	$1\times 64\times 192\times 192$
	Conv1	1	32	$1\times 1\times 1$	$32\times 32\times 192\times 192$
	Layer0	32	32	$1\times 1\times 1$	$32\times 64\times 192\times 192$
	Layer1	32	64	$2\times 2\times 2$	$64\times 64\times 192\times 192$
	Layer2	64	128	$2\times 2\times 2$	$128\times 64\times 192\times 192$
	Layer3	128	256	$2\times 2\times 2$	$256\times 64\times 192\times 192$
	Layer4	256	256	$2\times 2\times 2$	$256\times 64\times 192\times 192$
	fusionConv	256	256	$1\times 1\times 1$	$256\times 64\times 192\times 192$
Decoder	GAP	256	-	-	$256\times 1$
	Controller	256 + 7	162	$1\times 1\times 1$	$162\times 1\times 1\times 1$
	8resb	256	128	$1\times 1\times 1$	$128\times 64\times 192\times 192$
	4resb	128	64	$1\times 1\times 1$	$64\times 64\times 192\times 192$
	2resb	64	32	$1\times 1\times 1$	$32\times 64\times 192\times 192$
	1resb	32	32	$1\times 1\times 1$	$32\times 64\times 192\times 192$
	preclsConv	32	8	$1\times 1\times 1$	$8\times 64\times 192\times 192$
	SegHead	32	8	$1\times 1\times 1$	$2\times 64\times 192\times 192$

.

2.3. Mixture-of-Diverse-Experts Block

In order to address the diverse prediction subtasks effectively and enhance the network’s representational capacity for robust generalization, RepMode with MoDE [21] for dynamic parameter ${\mathit{\theta }}_{\mathit{f}}$ organizing consists of diverse expert design, gating module design, and gating re-parameterization.

We use the MoDE block as a fundamental building block of the U-shaped encoder–decoder network as a potent alternative to the conventional convolutional layer. Within the MoDE block, an array of diverse experts is meticulously crafted, each responsible for exploring a distinctive convolutional configuration. Furthermore, a gating module is intricately devised to harness task-specific information, enabling the generation of gating weights for dynamic parameter organization.

For the diverse expert design, there are two distinct types of expert diversity to enhance its capabilities: shape diversity, to address the multi-scale challenges by using different experts to possess a variety of receptive fields, and kernel diversity, using combinations of different kernels. Following these principles, the concept of “expert pairs” is used to form the multiple branch structure. Each expert pair consists of two key components: 3D convolutions (Conv) and 3D average poolings (Avgp). In essence, the MoDE block comprises expert pairs with three distinct receptive fields, promoting shape diversity. For more details, please refer to [21].

As for the gating module design, we encode the task-aware prior associated with each volume, ${\mathbf{x}}_n$, by transforming its subtask indicator, $l_n$, into a P-dimensional one-hot vector, ${\mathbf{k}}_n$, as in [18]. This encoding is expressed as follows:

$$k_{np}=\left\{\begin{array}{cc}\hfill 1,& \mathrm{if}p=l_n,\hfill \\ \hfill 0,& \mathrm{otherwise},\hfill \end{array}\right.p=1,2,\dots ,P,$$

(2)

where $k_{np}$ signifies the p-th element of ${\mathbf{k}}_n$. Next, the one-hot vector, ${\mathbf{k}}_n$, is introduced to the gating module, from which the gating weights, $\mathbf{M}$, are produced using a single-layer fully connected network, denoted as $\varphi (·)$, as depicted below: $\mathbf{M}=\varphi \left({\mathbf{k}}_n\right)={\left\{{\mathbf{m}}_t\right\}}_{t=1}^T$, where $T=5$. The subscript n in $\mathbf{M}$ is omitted for brevity. Each ${\mathbf{m}}_{\mathbf{t}}\in {\mathbb{R}}^{N_{\mathrm{O}}}$ represents the gating weights for the t-th expert, and these values are derived from $\mathbf{M}$, with $N_O$ representing the number of channels in the output feature maps. Lastly, $\mathbf{M}$ undergoes further processing to yield $\widehat{\mathbf{M}}$, which is expressed as a set $\widehat{\mathbf{M}}={\left\{{\widehat{\mathbf{m}}}_t\right\}}_{t=1}^T$ by applying the Softmax function. This operation ensures a balanced intensity among the various experts and can be defined as follows:

$$\begin{array}{c}\hfill {\widehat{m}}_{ti}=\frac{exp\left(m_{ti}\right)}{{\sum }_{j=1}^{T}exp\left(m_{ji}\right)},i=1,2,\dots ,N_{\mathrm{O}},\end{array}$$

(3)

where ${\mathbf{m}}_{\mathbf{ti}}$ (resp. ${\widehat{\mathbf{m}}}_{\mathbf{ti}}$) denotes the i-th element of ${\mathbf{m}}_t$ (resp. ${\widehat{\mathbf{m}}}_t$). Utilizing the resulting gating weights, $\widehat{\mathbf{M}}$, RepMode can undertake the dynamic organization of parameters for these experts, which are task-agnostic and adapt their behavior based on the task-aware prior.

Then, for gating re-parameterization, GatRep is designed according to principles of homogeneity and additivity that are established in [32,33]. The initial phase of GatRep involves the amalgamation of Avgp and Conv operations into a unified kernel. To streamline the explanation, let us consider the example of an Avgp–Conv expert. Here, we use ${\mathbf{F}}^{\mathrm{I}}$ to represent the input feature maps. Then, the output feature maps, denoted as ${\mathbf{F}}^{\mathrm{O}}$, can be articulated as follows:

$${\mathbf{F}}^{\mathrm{O}}=\mathbf{W}⊛({\mathbf{W}}^{\mathrm{a}}⊛{\mathbf{F}}^{\mathrm{I}}),$$

(4)

where ⊛ means the convolution operation. By applying the associative property, we can achieve an equivalent transformation for the equation in Equation (4) by initially consolidating ${\mathbf{W}}^{\mathrm{a}}$ and $\mathbf{W}$. This process can be denoted as follows:

$${\mathbf{F}}^{\mathrm{O}}=\underset{{\mathbf{W}}^{\mathrm{e}}}{\underbrace{(\mathbf{W}⊛{\mathbf{W}}^{\mathrm{a}})}}⊛{\mathbf{F}}^{\mathrm{I}},$$

(5)

With this process, the kernels of Avgp and Conv can be combined into a unified kernel for use in the subsequent stage.

The second phase of GatRep involves the consolidation of all experts in a manner specific to the subtask at hand. To facilitate this, we introduce a mapping function denoted as $\mathrm{Pad}(·,K^{\prime })$. This function effectively transforms a kernel to the kernel space $\mathcal{Z}\left(K^{\prime }\right)$ by employing zero-padding. Here, we set $K^{\prime }=5$, which represents the largest receptive field size among these experts. We use ${\widehat{\mathbf{F}}}^{\mathrm{O}}$ to represent the ultimate task-specific feature maps. The transformation can be expressed as follows:

$$\begin{array}{c}\hfill {\widehat{\mathbf{F}}}^{\mathrm{O}}=\underset{{\widehat{\mathbf{W}}}^{\mathrm{e}}}{\underbrace{\left(\sum _{t=1}^T{\widehat{\mathbf{g}}}_t\odot \mathrm{Pad}({\mathbf{W}}_t^{\mathrm{e}},K^{\prime })\right)}}⊛{\mathbf{F}}^{\mathrm{I}},\end{array}$$

(6)

where ⊙ represents channel-wise multiplication, and ${\mathbf{W}}_t^{\mathrm{e}}$ represents the kernel of the t-th expert. It is important to note that ${\mathbf{W}}_t^{\mathrm{e}}$ is an integrated kernel for an Avgp–Conv expert, or simply a Conv kernel for a Conv expert. Ultimately, ${\widehat{\mathbf{W}}}^{\mathrm{e}}$ is the dynamically generated resulting task-specific kernel by GatRep.

3. Results

3.1. Datasets

We used the multi-organ and tumor segmentation (MOTS) dataset, as in [18]. MOTS was presented for abdominal organ and tumor segmentation using partially labeled datasets. It consists of datasets for Liver and tumor [27], Kidney and tumor [22], and five datasets of Hepatic vessel and tumor, Pancreas and tumor, Colon tumor, Lung tumor, and Spleen from the Medical Segmentation Decathlon [23]. The dataset comprises 1155 3D abdominal CT volumes, gathered from diverse clinical sites worldwide. Of these, 920 volumes are allocated for training purposes, while the remaining 235 are reserved for testing. Notably, all volumes have been uniformly re-sampled to a consistent voxel size of 1.5 × 0.8 × 0.8 mm³, following the data preprocessing in [18]. In our experiments, we found out there are two bad cases in the Liver and tumor training dataset and two bad cases in the Liver and tumor testing dataset after preprocessing. The resolution of the volumes and the corresponding labels are different in these four cases, so we removed them, resulting 918 training cases and 233 testing cases.

3.2. Implementation Details

We performed experiments on a workstation equipped with two NVIDIA A100 GPUs. We adopted DoDNet from [18] as our baseline, then integrated the MoDE block with the GatRep module into DoDNet, obtaining our DoDRepNet. We maintained the remaining implementation settings consistent with those in [18] except with a batch size of 16. We reran the original DoDNet using their provided source code, and executed our DoDRepNet based on Re-parameterize Diverse Experts. To ensure fair comparisons, all models underwent the same training configurations, which included weight standardization, learning rate, optimizer, and other settings, as well as the inference strategy, except adding a few morphological operations for two cases for the spleen segmentation task in postprocessing.

3.3. Performance Metrics

This study employed the commonly used Dice similarity coefficient (Dice) and Hausdorff distance (HD) as performance metrics for evaluating the performance of segmentation methods. Dice is a measure of the spatial overlap between the predicted segmentation (output of an algorithm) and the ground truth segmentation (manual or reference segmentation). The Dice coefficient is calculated using the formula as follows:

$$\mathbf{Dice}=\frac{2\times (\mathbf{Pred}\cap \mathbf{GT})}{\mathbf{Pred}\cap \mathbf{GT}+\mathbf{Pred}\cup \mathbf{GT}},$$

(7)

where $\mathbf{Pred}$ denotes the predicted segmentation and $\mathbf{GT}$ is the ground truth segmentation.

HD measures the maximum distance between any point in the predicted segmentation and its closest point in the ground truth, as well as the maximum distance between any point in the ground truth and its closest point in the predicted segmentation. The HD is defined as follows:

$$\mathbf{HD}=\mathbf{max}({\mathbf{max}}_{\mathbf{i}}{\mathbf{min}}_{\mathbf{j}}\mathbf{d}({\mathbf{p}}_{\mathbf{i}},{\mathbf{q}}_{\mathbf{j}}),{\mathbf{max}}_{\mathbf{j}}{\mathbf{min}}_{\mathbf{i}}\mathbf{d}({\mathbf{q}}_{\mathbf{j}},{\mathbf{p}}_{\mathbf{i}})),$$

(8)

where ${\mathbf{p}}_{\mathbf{i}}$ and ${\mathbf{q}}_{\mathbf{j}}$ are points in $\mathbf{Pred}$ and $\mathbf{GT}$, and $\mathbf{d}$ is the distance metric (e.g., Euclidean distance).

A lower HD indicates better agreement between the predicted and ground truth segmentations, as it represents the maximum distance between corresponding points. In the context of medical image segmentation, these metrics help quantify how well an algorithm delineates structures of interest (such as organs or tumors) compared with the manually annotated ground truth.

3.4. Comparisons with State-of-the-Art Approaches

We conduct a comparative analysis of our DoDRepNet against state-of-the-art methods designed for addressing the challenge of partially labeled data. This evaluation was carried out on seven partially labeled tasks using the MOTS test set, as in [18]. The competing methods include: two multi-head networks, namely Multi-Head [20] and TAL [28], a single-network method that operates without task conditioning, known as Cond-NO, and three single-network methods incorporating task conditioning, specifically Cond-Input [34], Cond-Dec [35], and DoDNet [18]. The results of the above methods are copied from [18]. We added our results of reran DoDNet and our DoDRepNet, as listed in

Table 3. Performance (Dice, %, and Hausdorff distance (HD) ) of different approaches on MOTS test set, as in [18]. Please note that the term ’Average score’ serves as a composite metric, averaging the Dice or HD values across 11 organs and tumors.

Methods	Task 1: Liver				Task 2: Kidney				Task 3: Hepatic Vessel
	Dice		HD		Dice		HD		Dice		HD
	Organ	Tumor	Organ	Tumor	Organ	Tumor	Organ	Tumor	Organ	Tumor	Organ	Tumor
Multi-Nets	96.61	61.65	4.25	41.16	96.52	74.89	1.79	11.19	63.04	72.19	13.73	50.70
TAL [28]	96.18	60.82	5.99	38.87	95.95	75.87	1.98	15.36	61.90	72.68	13.86	43.57
Multi-Head [20]	96.75	64.08	3.67	45.68	96.60	79.16	4.69	13.28	59.49	69.64	19.28	79.66
Cond-NO	69.38	47.38	37.79	109.65	93.32	70.40	8.68	24.37	42.27	69.86	93.35	70.34
Cond-Input [34]	96.68	65.26	6.21	47.61	96.82	78.41	1.32	10.10	62.17	73.17	13.61	43.32
Cond-Dec [35]	95.27	63.86	5.49	36.04	95.07	79.27	7.21	8.02	61.29	72.46	14.05	65.57
DoDNet [18]	96.87	65.47	3.35	36.75	96.52	77.59	2.11	8.91	62.42	73.39	13.49	53.56
DoDNet 1	96.78	63.56	4.52	32.97	96.26	80.06	3.87	11.99	62.55	74.87	13.76	40.9
DoDRepNet [21,32,33]	96.99	66.69	3.29	25.31	96.89	82.68	1.97	14.61	63.6	76.65	13.45	29.06
Methods	Task 4: Pancreas				Task 5: Colon		Task 6: Lung		Task 7: Spleen		Average score
	Dice		HD		Dice	HD	Dice	HD	Dice	HD	Dice↑	HD↓
	Organ	Tumor	Organ	Tumor	Tumor	Tumor	Tumor	Tumor	Organ	Organ
Multi-Nets	82.53	58.36	9.23	26.13	34.33	103.91	54.51	53.68	93.76	2.65	71.67	28.95
TAL [28]	81.35	59.15	9.02	21.07	48.08	66.42	61.85	39.92	93.01	3.10	73.35	23.56
Multi-Head [20]	83.49	61.22	6.40	18.66	50.89	59.00	64.75	34.22	94.01	3.86	74.55	26.22
Cond-NO	65.31	46.24	36.06	76.26	42.55	76.14	57.67	102.92	59.68	38.11	60.37	61.24
Cond-Input [34]	82.53	61.20	8.09	31.53	51.43	44.18	60.29	58.02	93.51	4.32	74.68	24.39
Cond-Dec [35]	77.24	55.69	17.60	48.47	51.80	63.67	57.68	53.27	90.14	6.52	72.71	29.63
DoDNet [18]	82.64	60.45	7.88	15.51	51.55	58.89	71.25	10.37	93.91	3.67	75.64	19.50
DoDNet 1	82.54	59.82	8.61	28.56	48.86	58.88	61.5	18.5	94.74	2.13	74.54	20.66
DoDRepNet [21,32,33]	83.67	61.22	7.48	34.07	45.17	70.94	65.82	47.61	94.18	2.68	75.78	22.77

¹ Results of DoDNet rerun on our workstation.

. The best score of each metric in each task is denoted in red.

Our DoDRepNet, designed to master the task-specific amalgamation of various task-agnostic experts, surpasses the performance of current methods on the liver (both on Dice and HD), kidney (on Dice), hepatic vessel (both on Dice and HD), and pancreas (on Dice), and achieves better overall performance than DoDNet. Figure 3 displays a sample of each task and the segmentations of Ground Truth (GT), DoDNet, DoDRepNet, and the 3D rendering of the three segmentations. From top to bottom, the tasks are Liver and Liver tumor, Kidney and Kidney tumor, Hepatic vessel and tumor, Pancreas and tumor, Colon tumor, Lung tumor, and Spleen. Specifically, (a) is a axial slice of the original resampled volume, (b,c) are its segmentations of Ground Truth (GT), DoDNet, and DoDRepNet, respectively, and (e,f) are the 3D rendering of the segmentations of Ground Truth (GT), DoDNet, and DoDRepNet, respectively. From Figure 3, we can see that the segmentations of DoDRepNet in most tasks are more similar to GT than those of DoDNet and both segmentations of DoDRepNet and DoDNet are more smooth than GT thanks to the dynamic parameter ${\theta }_f$ organizing based on Re-parameterize Diverse Experts.

In Figure 4, we compared the speed–accuracy trade-off of DoDRepNet with previous methods, as listed in [18]. From Figure 4, we can see that DoDRepNet has a few more parameters than DoDNet due to the MoDE blocks. It also has a little more inference time than DoDNet. However, DoDRepNet achieves the best accuracy.

4. Discussion

A single network with dynamic heads [18,19] offers a good way for leveraging partially annotated data for multi-organ abdominal and tumor segmentation tasks. However, substantial differences in the relative sizes of distinct target organs lead to imbalances in segmentation objectives. In the training process, the size differences between organs create substantial competition, which can be detrimental to smaller organs. In order to address the diverse prediction subtasks effectively and enhance the network’s representational capacity for robust generalization, we introduce the MoDE block as a fundamental building block of the U-shaped encoder–decoder network with dynamic segmentation heads. The MoDE block is put forth as a potent alternative to the conventional convolutional layer. Within the MoDE block, an array of diverse experts is meticulously crafted, each responsible for exploring a distinctive convolutional configuration. Furthermore, a gating module is intricately devised to harness task-specific information, enabling the generation of gating weights for dynamic parameter organization.

This design enables DoDRepNet to focus on learning dynamic compositions of experts with varying receptive fields specific to each organ and tumor. This capability make it able to acquire multi-scale features adaptively in a task-specific way, addressing the multi-scale challenges of abdominal organ and tumor segmentation. Compared with DoDNet, which only uses task-related information at the decoder end for the segmentation head, we utilize it in both the encoder–decoder section and the segmentation head. By incorporating task-dependent gating and Mixture-of-Diverse-Experts, DoDRepNet can learn the generalized parameters for all tasks by combine experts with diverse configurations, and generate the specialized parameters for each task with gating re-parameterization (GatRep) in the encoder–decoder section. This approach offers greater flexibility in modeling various scenarios, thus improving segmentation performance.

While DoDRepNet showed promising segmentation performance, there were still some instances of suboptimal predictions. Additionally, our best model did not surpass the performance of other methods on all the organs and tumors. One limitation of our model is that the task-related information is a simple one-hot vector, the same as in DoDNet, which cannot offer task-specific semantic information. In addition, the current method does not consider anatomical prior among organs or of organs, which makes it fail when the boundaries between organs are quite unclear, resulting in necessary postprocessing, as in DoDNet [18]. Future work includes task-specific semantic information and anatomical priors, which may improve MOTS results. In addition, ensemble deep learning [36], combining several individual deep neural network models to obtain better generalization performance, has shown potential in tumor detection [37] and classification [38], and its utilization in MOTS is also a direction worth exploring.

5. Conclusions

We present DoDRepNet by integrating RepMode into DoDNet to suit the demands of abdominal organ and tumor segmentation. By dynamically organizing its parameters based on task-aware priors, we overcame the multi-scale challenge by combining experts with varying receptive fields, resulting in dynamic multi-scale feature learning. Task-dependent gating in both the encoder–decoder and segmentation head provides flexibility in modeling diverse scenarios. Our experiments on the MOTS dataset demonstrated that DoDRepNet outperformed other methods in several organ and tumor segmentation tasks, showcasing the effectiveness of DoDRepNet. However, some challenges remain, such as handling unclear organ boundaries and leveraging anatomical priors effectively.