W-Shaped Net: An Inter-Slice Super-Resolution Segmentation Deep Network for CT Scans of Hepatic Ducts

Abstract

The three-dimensional (3D) reconstruction of the hepatic duct tree is significant for the minimally invasive surgery of hepatobiliary stone disease, which can be influenced by the quantity of the CT scans of hepatic ducts. If insufficient CT scans with low inter-slice resolution are directly utilized for 3D reconstruction, discontinuities and gaps will emerge in the reconstructed hepatic duct tree. In this paper, a novel end-to-end deep learning framework is designed for the inter-slice super-resolution segmentation of the CT slices of hepatic ducts, which can improve the 3D reconstruction performance in the inter-slice dimension. Specifically, the framework cascades into an inter-slice super-resolution subnetwork and a segmentation subnetwork. A deep learning network is introduced as the inter-slice super-resolution subnetwork to generate an intermediate slice between two adjacent CT slices in the simulated CT scans with low inter-slice resolution. To capture the spatiotemporal correlation existing in the CT scans of hepatic ducts, the ConvLSTM is introduced into the U-Net-like segmentation subnetwork in the high-dimensional feature space. To further suppress the problems of discontinuities and gaps, a structure-aware loss function is proposed by incorporating the structural similarity index measure (SSIM) as a regulator to dynamically assign the contribution of the generated CT slice to the total loss of the designed framework. Experimental results demonstrate that our proposed framework performs better segmentations for hepatic ducts than several existing deep learning networks with the performance of 0.7690 DICE and a 0.7712 F1-score, which is beneficial for the 3D reconstruction of the hepatic duct tree.

1. Introduction

In recent years, the incidence of hepatobiliary stone disease has been rising annually in countries and regions worldwide, which is becoming one of the common liver diseases. This disease can cause recurrent infections that may trigger localized hepatic atrophy, thereby elevating the risk of intrahepatic cholangiocarcinoma [1,2]. A mainstream therapeutic approach [3,4,5], namely Percutaneous Transhepatic Cholangioscopy (PTCS) [5], has been clinically employed to investigate and extract stones from the hepatic ducts within the liver, which is a minimally invasive surgery. Compared with conventional hepatectomy, this type of surgery has the advantages of fewer sessions of anesthesia, less trauma to the patient, and a shorter postoperative hospital stay [5].

Before the surgery of PTCS, the surgeons can utilize the medical imaging techniques to determine the spatial structure of the hepatic duct tree and further localize the hepatobiliary stone in the hepatic duct tree, involving computed tomography (CT) techniques [6,7,8,9] and three-dimensional (3D) reconstruction techniques. The 3D reconstruction techniques can assist the surgeons in obtaining comprehensive preoperative diagnostic information and efficiently refining the pathway for cholangioscopic stone extraction during the surgery [10]. The CT techniques are utilized to acquire image sequences covering the patient’s liver, surrounding blood vessels, and other bodily tissues by using X-ray, which can be subsequently employed to reconstruct the 3D model of the affected tissues [11,12]. Generally, as the dose of X-ray increases, the quality of the images acquired by CT improves, which is beneficial for the fidelity of 3D reconstruction [13]. However, prolonged exposure to such high doses of X-ray can cause irreversible harm to the human body. One direct way to mitigate this issue is to enhance the performance of the equipment to accelerate the CT scanning speed. Another way is to reduce the frequency (or time) of CT scanning, which can easily lead to low inter-slice resolution for the acquired CT scans [14]. That is, the CT scans with low inter-slice resolution involve less slices than those with high inter-slice resolution. Thanks to the development of CT technology, modern CT devices are equipped with various inter-slice interpolation methods to increase the quantity of CT slices for the improvement of 3D reconstruction quality. Inter-slice interpolation can be considered a form of specialized super-resolution reconstruction, performed in the inter-slice dimension (z-axis) rather than in the intra-slice dimension (xy-axis). For clear explanations, Figure 1 illustrates the 3D reconstructed examples for the CT devices with high inter-slice resolution and with low inter-slice resolution. As indicated in Figure 1, after the CT scan acquired by the CT device with low inter-slice resolution is inter-slice interpolated, its corresponding reconstructed hepatic duct tree has a similar high quality to that reconstructed by the CT device with high inter-slice resolution. This illustration demonstrates that inter-slice interpolation allows even normal CT devices to compensate to some extent for the low inter-slice resolution in the generated CT scans, which is also an economical solution for many regional hospitals. Commonly, most normal CT devices integrate traditional inter-slice interpolation methods, such as linear interpolation [15] and spline interpolation [16], which can be further improved for high-resolution 3D reconstruction.

Due to its strong feature extraction and generalization capability, deep learning has been successfully employed in the field of medical image segmentation, which is beneficial for high-quality 3D construction. However, most deep learning models for segmentation still face the problem of discontinuities and gaps [17,18] when they are directly utilized to subsequently reconstruct 3D medical images by insufficient CT scans with low inter-slice resolution. To deal with this problem, some studies attempt to employ inter-slice interpolation to increase the number of CT scans, which is followed by segmentation [19]. This type of two-stage methods will bring two issues. One is that it does not fully consider the inter-slice relationship between the synthesized slices and the adjacent slices during segmentation, which may result in the undesirable discontinuities and gaps in the reconstructed 3D images [20]. Second, simply overlaying these two methods without a mechanism for interaction between them can result in an inherent problem, which is that accumulative errors from the two stages will notably degrade the segmentation of the tissues and further influence the 3D reconstruction, especially of the hepatic ducts with complex and tenuous shapes.

To address these limitations of the aforementioned two-stage methods, we designed a one-stage inter-slice super-resolution segmentation deep learning framework to segment the CT scans of hepatic ducts, cascading an inter-slice super-resolution subnetwork for inter-slice interpolation and a segmentation subnetwork for CT scan segmentation. It is noted that the two subnetworks are jointly trained in an end-to-end mode. Due to it being composed of two U-Net-like networks, it is termed a W-shaped Net for simplicity. To capture the correlations of adjacent slices, the ConvLSTM [21] is introduced into the decoder of the U-Net-like segmentation subnetwork. To build the connection between two subnetworks and suppress the potential accumulative errors, a novel structure-aware loss function is proposed to jointly optimize the inter-slice super-resolution subnetwork and the segmentation subnetwork.

In summary, our study has several main contributions, described as follows.

(1)

To deal with the problems of discontinuities and gaps that arise from the segmentation of insufficient CT scans acquired by CT equipment with low inter-slice resolution, a one-stage deep learning framework is designed to jointly integrate an inter-slice super-resolution subnetwork for inter-slice interpolation and a segmentation subnetwork for CT scan segmentation.

(2)

Since the spatiotemporal correlation exists in the CT scans of hepatic ducts, a U-Net-like segmentation subnetwork is designed by introducing a ConvLSTM to mine this spatiotemporal correlation in the high-dimensional feature space, which can alleviate the problems of discontinuities and even the wrong assignments of two segments of the same hepatic duct in two adjacent CT slices.

(3)

Subtle differences between the generated CT slices and the real ones will result in the discontinuities emerging between the segmented hepatic ducts in the generated CT slice and those in the two adjacent ones. To this end, a novel structure-aware loss function is proposed to jointly constrain the two subnetworks, in which the structural similarity index measure (SSIM) is incorporated as a regulator to dynamically assign the contribution of the generated CT slice to the total loss of the proposed framework.

2. Related Work

2.1. Deep Learning-Based Inter-Slice Interpolation for Medical Images

Recently, deep learning has been employed for inter-slice interpolation to generate some slices in the medical image sequences for subsequent processing, which mainly rely on optical flow techniques.

Wu et al. [19] designed an intermediate slice synthesis deep model for CT inter-slice augmentation, consisting of an optical flow estimation module based on the U-Net architecture and a global–local dual discriminator based on attention mechanisms. Guo et al. [22] designed a spatiotemporal volumetric interpolation network (SVIN) for 4D dynamic medical images (including 4D CT and 4D flow MRI), comprising a motion network for the construction of a spatiotemporal motion field and an interpolation network with a regression-based module for sequence volumetric interpolation.

Although these optical-flow-based deep learning methods can augment the medical slices by interpolating some slices between the adjacent slices and achieve good interpolation results, they heavily rely on optical flow estimation and pixel-level warping processing, resulting in substantially computational costs and memory usage. In this study, we introduce the Speconv [23], a flow-free deep convolutional network, as our inter-slice super-resolution subnetwork.

2.2. Deep Learning-Based Image Segmentation for Tubular Tissues in the Livers

Due to its superior capability of feature extraction and flexible network structure, U-Net and its variants have been widely employed for medical image segmentation [24]. Also, some studies have employed U-Net-like networks to segment the tubular tissues in the liver, such as hepatic vessels and hepatic ducts, from the medical images.

Huang et al. [25] employed a 3D U-Net for hepatic vessel segmentation in which a loss function with a variant of the DICE was proposed to address the pixel imbalance between the hepatic vessels and the background. Yan et al. [26] designed a novel 3D U-Net for hepatic vessel segmentation, which incorporated several attention-guided concatenation modules between the encoder and decoder to extract contextual information from lower-dimensional features guided by higher-dimensional features. Li et al. [27] integrated a graph connectivity constraint module into a lightweight 3D U-Net for hepatic vessel segmentation. Zhou et al. [28] proposed a sequence-based context-aware association network for hepatic vessel segmentation, integrating a slice-level attention module and a graph association module into the U-Net architecture.

Compared with the segmentation of hepatic vessels, the segmentation of hepatic ducts has several challenging issues, such as lower contrasts and sparser distributions. Thus, only a few studies focus on the segmentation of hepatic ducts. Fu et al. [29] proposed an end-to-end fully convolutional network for the segmentation of hepatic ducts, which integrated multi-scale dilated convolutions to extract abundant semantic features and various scale contextual information. To well constrain the designed network, they proposed a novel loss function by combining the online bootstrapped method and cross entropy.

Although these deep learning methods can segment the tubular tissues from the liver images, the discontinuities and gaps will emerge in the constructed 3D tissue image if they are directly utilized to segment insufficient CT scans with low inter-slice resolution.

3. Materials

The abdominal CT slices in this study were provided by the Department of Hepatobiliary Surgery at the First Affiliated Hospital of Guangzhou Medical University, which were acquired by a 256-row spiral CT equipment with the spatial resolution of 512 × 512 and the pixel spacing ranging between 0.56 mm and 0.74 mm. The thicknesses of the acquired CT slices ranged between 0.625 mm and 2.500 mm, which differ from those of ordinary spiral CT slices ranging between 2.5 mm and 5 mm. These acquired CT slices are considered as the ground truths with high inter-slice resolution. In this study, due to the limitations such as CT equipment and scanning time, we are unable to acquire CT slices with lower inter-slice resolution. Even though we have the slices with low inter-slice resolution, we could not evaluate whether the inter-slice interpolated slices have high qualities for 3D reconstruction, due to no real ones with high inter-slice resolution. Hence, we extracted the CT slices at intervals from the ground truths to simulate the CT scans with low inter-slice resolution. For example, assume that the CT scans have a total of 100 CT slices {1, 2, 3, 4, 5, …, 100}, then slice sampling with intervals can simulate the generation of the CT scans with low inter-slice resolution, i.e., the CT slices {1, 3, 5, …, 99}. Then, our motivation is to restore the filtered-out 50 CT slices {2, 4, 6, …, 100}, along with the CT slices {1, 3, 5, …, 99}, for the subsequent 3D reconstruction of the hepatic duct tree.

In this study, a total of 24 patients were recruited, and each patient’s CT data comprised approximately 100 to 200 high inter-slice resolution CT slices available for the inter-slice super-resolution segmentation of hepatic ducts. A total of 2522 CT slices with high inter-slice resolution were acquired for these cases, which were randomly divided into a training set with 1913 CT slices and a test set with 609 CT slices. All patient data collected for research purposes were obtained with informed consent, but due to confidentiality agreements, this dataset is currently not available for public access. Experienced hepatobiliary surgeons utilized the ITKSNAP [30] to perform the ground truth segmentation of hepatic ducts.

All the experiments in this paper were conducted on a desktop computer equipped with an NVIDIA GTX 3090 24G GPU and Ubuntu 20.04 Linux operating system. The code was implemented by using Pytorch v1.6. The proposed W-shaped Net was trained by the Adam optimizer with a batch size of 4, a learning rate of 0.001, and exponential decay rates (beta 1 and beta 2) with 0.9 and 0.999, respectively. The training process was conducted over 200 epochs. Here, several commonly used metrics were utilized to assess different networks for the segmentation of hepatic ducts, involving the dice similarity coefficient (DICE), sensitivity, specificity, and F1-score.

4. Methods

4.1. The Architecture of the W-Shaped Net

As illustrated in Figure 2, the designed W-shaped Net is composed of two cascading U-Net-like subnetworks, which are an inter-slice super-resolution subnetwork and a segmentation subnetwork, respectively. The former subnetwork consists of an encoder–decoder structure and a separable kernel estimation (SKE) module. Specifically, the encoder/decoder involves five downsampling blocks with a 2 × 2 AveragePooling layer and a 3 × 3 convolutional block (ConvB), and four upsampling blocks with a bilinear upsampling block (BUB) and a 3 × 3 ConvBs, respectively. The SKE involves four parallel blocks successively with a 3 × 3 ConvB, a bilinear upsampling layer (BUL) and a 3 × 3 convolutional layer (ConvL), which is utilized to produce the generated slices (or intermediate slices) between the adjacent abdominal CT slices. It is noted that the 3 × 3 ConvBs utilized in the encoder, decoder, and the SKE have the same network structure with three 3 × 3 ConvL-ReLU layers. The BUB involves BUL-3 × 3 ConvL-ReLU blocks. The segmentation subnetwork includes a CNN backbone with four residual blocks, a transformer block with 12 transformer layers, two high-dimensional ConvLSTM-3 × 3 ConvL-ReLU-batch normalization (BN) blocks, and three BUL-3 × 3 ConvL-ReLU-BN blocks. Two adjacent CT slices are input into the W-shaped Net to finally achieve three segmented CT slices with hepatic ducts.

4.2. Inter-Slice Super-Resolution Subnetwork

Our network configuration of an inter-slice super-resolution subnetwork is shown in

Table 1. The network configuration of our inter-slice super-resolution subnetwork.

Layer	Input Features Size	Main Operation	Output Channel
Input image × 2	(1,3,512,512) × 2	Concatenation	6
First convolution	(1,6,512,512)	Conv-3	32
Encoding layer 1	(1,32,512,512)	APL-2 × 2, ConvB-3	64
Encoding layer 2	(1,64,256,256)	APL-2 × 2, ConvB-3	128
Encoding layer 3	(1,128,128,128)	APL-2 × 2, ConvB-3	256
Encoding layer 4	(1,256,64,64)	APL-2 × 2, ConvB-3	512
Encoding layer 5	(1,512,32,32)	APL-2 × 2, ConvB-3	1024
Decoding layer 1	(1,1024,16,16)	BUB-2, ConvB-3	512
Decoding layer 2	(1,512,32,32)	BUB-2, ConvB-3	256
Decoding layer 3	(1,256,64,64)	BUB-2, ConvB-3	128
Decoding layer 4	(1,128,128,128)	BUB-2, ConvB-3	64
SKE	(1,64,256,256)	BUB-2, ConvB-3, 1D-Kernel, Relu	3
Output image	(1,3,512,512) × 3		3

. Since the flow-free network in [23] has the advantages of relatively small memory demand, high computational efficiency, and good interpolation performance, it is introduced into our proposed W-shaped Net as the inter-slice super-resolution subnetwork.

As illustrated in Figure 2, assume the two adjacent slices as $I_1,I_2\in {ℝ}^{3\times \mathrm{H}\times \mathrm{W}}$; they are initially concatenated along the channel dimension and subsequently pass through the encoder for downsampling to the high-dimensional feature maps with the size of ${\displaystyle \frac{H}{2^5}}\times {\displaystyle \frac{W}{2^5}}$. Thus, the low-dimensional features can primarily capture the contours, grayscale, and shapes of hepatic ducts in the slices, while the high-dimensional features can primarily represent the global semantic information of various tissues. This encoding can be formulated as

$$x_0=Conv{L}_3\left(concat\left(I_1,I_2\right)\right)$$

(1)

$$x_i=ConvB\left(AP\left(x_{i-1}\right)\right),i=1~5$$

(2)

where $ConvB\left(\cdot \right)$ and $AP\left(\cdot \right)$ represent the operations of 3 × 3 ConvB and AveragePooling (AP), respectively. $Conv{L}_3\left(\cdot \right)$ represents the 3 × 3 convolution operation. $x_0$ and $x_i$ denote the input features of the encoder and the output feature maps of the i-th layer in the encoder.

During the decoding process, the feature maps from each lower layer undergo the upsampling operation, followed by a residual addition with the corresponding outputs of the encoding layer. Next, they pass through the 3 × 3 convolution operation. The decoding can be formulated as

$$x_1^{\prime }=ConvB\left(add\left(BUB\left(x_5\right),x_4\right)\right)$$

(3)

$$x_j^{\prime }=ConvB\left(add\left(BUB\left(x_{j-1}^{\prime }\right),x_{5-j}\right)\right),j=2~4$$

(4)

where $add\left(\cdot \right)$ and $BUB\left(\cdot \right)$ represent the operations of an element-wise addition and a bilinear upsampling block, respectively. $x_j^{\prime }$ denotes the output feature maps of the j-th layer in the decoder. Subsequently, the extract feature maps from the decoder are inputted into the SKE module to generate an intermediate slice. Specifically, the SKE module is designed as a fully convolutional module for convolution-based interpolation. It has four 1D kernel operations, each pair of which is used to extract vertical and horizontal information from the feature maps from the decoder. Then, two types of paired information are correspondingly convolved with $I_1$ and $I_2$ to compute the colors of the pixels of the paired feature maps. The mechanism of the SKE, which approximates as a 2D kernel by estimating a pair of 1D kernels, can reduce the number of kernel parameters and accelerate the training speed, ultimately improving the visual quality of the generated slice. Finally, the paired feature maps are element-wise added to obtain the generated slice. The process of the SKE module can be formulated as

$$I_k^{\prime }=Conv{L}_{*}\left(I_k,Conv{L}_v^k\left(f_{\uparrow }\left(ConvB\left(x_4^{\prime }\right)\right)\right),Conv{L}_h^k\left(f_{\uparrow }\left(ConvB\left(x_4^{\prime }\right)\right)\right)\right),k=1,2$$

(5)

$$I_{is}=add\left(I_1^{\prime },I_2^{\prime }\right)$$

(6)

where $Conv{L}_{*}\left(\cdot \right)$ and $f_{\uparrow }\left(\cdot \right)$ represent the local convolution operation and bilinear upsampling operation. $Conv{L}_v^k\left(\cdot \right)$ and $Conv{L}_h^k\left(\cdot \right)$ denote the convolution operation for vertical feature information and for horizontal feature information, respectively. $I_{is}$ denotes the generated slice between the adjacent abdominal CT slices generated by the inter-slice super-resolution subnetwork.

4.3. Segmentation Subnetwork

Our network configuration of the segmentation subnetwork is shown in

Table 2. The network configuration of our segmentation subnetwork.

Layer	Input Features Size	Main Operation	Output Channel
Input image × 3	(1,3,512,512) × 3		3
ResBlock1	(1,3,512,512) × 3	APL-2 × 2, ConvL-3, GroupNorm, ReLu	64
ResBlock2	(1,64,256,256) × 3	APL-2 × 2, ConvL-3, GroupNorm, ReLu	256
ResBlock3	(1,256,128,128) × 3	APL-2 × 2, ConvL-3, GroupNorm, ReLu	512
ResBlock4	(1,512,64,64) × 3	APL-2 × 2, ConvL-3, GroupNorm, ReLu	1024
Transformer layer × 12	(1,1024,32,32) × 3	Flatten, ConvL-3, Self-attention	1024
Feature reshaper	(1,1024,768) × 3	ConvL-3, Reshape	1024
Decoding layer 1	(1,1024,32,32) × 3	ConvLSTM, ConvL-3, Relu, BN	512
Decoding layer 2	(1,512,32,32) × 3	ConvLSTM, BUB-2, ConvL-3, Relu, BN	256
Decoding layer 3	(1,256,64,64) × 3	BUB-2, ConvL-3, Relu, BN	128
Decoding layer 4	(1,128,128,128) × 3	BUB-2, ConvL-3, Relu, BN	64
Decoding layer 5	(1,64,256,256) × 3	BUB-2, ConvL-3, Relu, BN	64
Last conv	(1,64,512,512) × 3	ConvL-3	1

. We designed a segmentation subnetwork with the similar network architecture in [31]. Different from the network in [31] for single-image segmentation, our subnetwork is to deal with the CT scans, which can focus on the correlation among the two adjacent CT slices as well as the generated slice. Thus, to fully reveal this potential correlation, the ConvLSTM module is introduced into the high-dimensional feature space to mine the spatiotemporal correlation of the three CT slices, compared to the network in [31].

The three slices, $I_1, I_{is}, I_2\in {ℝ}^{3\times \mathrm{H}\times \mathrm{W}}$, are simultaneously input into the segmentation subnetwork, and they individually pass through four residual blocks and a transformer block [32,33] to generate their corresponding high-dimensional semantic features $T{B}_j$, formulated as

$$T{B}_j=TB\left(R{B}^4\left(\cdots R{B}^1\left(I_j\right)\right)\right),j=1,2,is$$

(7)

where $TB\left(\cdot \right)$ represents the operation of the transformer block with 12 transformer layers. $R{B}^1\left(\cdot \right)~R{B}^4\left(\cdot \right)$ denote the four residual operations, whose structure is illustrated in [34].

During the decoding process, high-dimensional semantic features $T{B}_j$ successively pass through five decoding blocks followed by a 3 × 3 convolutional layer to achieve the three segmented CT slices with hepatic ducts. It is noted that the ConvLSTM module is introduced into the first two decoding blocks to mine the spatiotemporal correlation of the three CT slices in the high-dimensional feature space. Moreover, the ConvLSTM module is only performed on the adjacent slices rather than the generated slice, since the generated slice possibly introduces a correlation bias if it is performed by this module. These operations can be formulated as

$$Se{g}_{is}=Conv{L}_3\left(Decoder\left(Reshape\left(T{B}_{is}\right)\right)\right)$$

(8)

$$Se{g}_i=Conv{L}_3\left(Decoder\prime \left(Reshape\left(T{B}_1,T{B}_2\right)\right)\right),i=1,2$$

(9)

where $Se{g}_{is}$ and $Se{g}_i$ denote the segmented intermediate slice and the two segmented adjacent slices, respectively, in which hepatic ducts are extracted from the slices. $Decoder$ and $Decode{r}^{\prime }$ represent five decoding blocks without and with the ConvLSTM module, respectively. Each decoding block with the ConvLSTM module involves a ConvLSTM module, a 3 × 3 convolutional layer, a ReLU layer, and a BN layer. It is noted that a bilinear upsampling layer (BUL) is added to the second decoder block and a skip connection is performed ahead of the BN layer from it, as illustrated in Figure 1.

4.4. Loss Function

The generated CT slice from the inter-slice super-resolution subnetwork is subtly different in appearance from the real slice, which influences the subsequent segmentation. To well constrain the inter-slice super-resolution subnetwork and the segmentation subnetwork, the quality evaluation of the generated slice should be considered in the total loss of the W-shaped Net. Three metrics are commonly used for image quality evaluation, involving Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR), and SSIM. Specifically, MSE calculates the difference between two images at each pixel, solely based on the pixel values at the current two pixels and neglecting the relationships with other pixels. Similarly, PSNR is calculated based on MSE. So, MSE and PSNR both disregard the local structural information of the image. SSIM evaluates the difference between the paired images at each pixel by considering not only individual pixels but also the local structural information, which is computed from three perspectives: luminance, contrast, and structure. Thus, we integrate the SSIM metric into the total loss of the W-shaped Net, which is utilized to regulate the influence of the generated CT slice. That is, a novel structure-aware loss function is defined as the total loss of the W-shaped Net, formulated as

$$L_{Total}={\displaystyle \frac{L_C+\frac{1}{3}\left(\frac{L_{Region}\left(Se{g}_1\right)}{SSI{M}_{is}}+SSI{M}_{is}\times L_{Region}\left(Se{g}_{is}\right)+\frac{L_{Region}\left(Se{g}_2\right)}{SSI{M}_{is}}\right)}{2}}$$

(10)

where $L_C$ represents the Charbonnier loss function [35]. $L_{Region}$ is the combined loss function commonly used in medical image segmentation, comprising cross-entropy loss and DICE loss, defined as

$$L_{Region}=L_{CE}+L_{Dice}$$

(11)

$$L_{CE}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum }_{c=1}^C{\displaystyle \sum }_{i=1}^N{g}_i^{c}log{s}_i^c$$

(12)

$$L_{Dice}=1-{\displaystyle \frac{2{{\displaystyle \sum }}_{c=1}^C{{\displaystyle \sum }}_{i=1}^N{g}_i^c{s}_i^c}{{{\displaystyle \sum }}_{c=1}^C{{\displaystyle \sum }}_{i=1}^N{g}_i^c+{{\displaystyle \sum }}_{c=1}^C{{\displaystyle \sum }}_{i=1}^N{s}_i^c}}$$

(13)

where $g_i^c$ and $s_i^c$ denote the ground truth and the predicted classes for the i-th pixel in the CT slice. In this work, we use the SSIM [36] as a regulator to dynamically assign the contribution of the generated CT slice to the total loss of the proposed framework, defined as

$$SSI{M}_{is}=SSIM\left(Se{g}_{is},G{T}_{is}\right)$$

(14)

where $SSI{M}_{is}$ is the SSIM value computed by the generated slice and its ground truth. The higher the quality of the generated slice is, the more significantly it contributes to the total loss, and vice versa.

5. Results

5.1. Comparisons of the Existing Deep Learning-Based Methods

As described in Section 3, our proposed W-shaped Net cascades the two subnetworks, i.e., the inter-slice super-resolution subnetwork and the segmentation subnetwork. The SpeConv [23] is introduced for the former subnetwork. The segmentation subnetwork is elaboratively designed by the inspiration of the Chen et al. method [31]. Thus, in this subsection, we conducted comparison experiments to validate the proposed W-shape Net in which several cascaded networks combining the SpeConv and different segmentation networks were employed for comparisons.

Comparison experiments are illustrated in

Table 3. Comparative experiments with different two-stage deep learning-based methods.

Methods	DICE	Sensitivity	Specificity	F1-Score
SpeConv [23] + Walsh et al. [37] (2024)	0.7027	0.6924	0.9990	0.7089
SpeConv + Imran et al. [40] (2023)	0.7210	0.7829	0.9988	0.7215
SpeConv + Wang et al. [38] (2023)	0.7198	0.7070	0.9991	0.7198
SpeConv + AlSalmi et al. [41] (2023)	0.7284	0.7959	0.9984	0.7337
SpeConv + Cao et al. [39] (2022)	0.6589	0.6912	0.9986	0.6653
SpeConv + Chen et al. [31] (2021)	0.7410	0.7649	0.9990	0.7432
Ours	0.7690	0.7792	0.9990	0.7712

. Walsh et al. [37] and Wang et al. [38] cannot well grasp the spatial relationships of different pixels since it performs the segmentation by means of the operations of convolutions and upsampling. Thus, they cannot precisely localize the tenuous hepatic ducts with complex traveling structures, which results in the two models, the SpeConv + Walsh et al. [37] method and the SpeConv + Wang et al. [38] method, not being able to achieve fair segmentation performance. Although the Cao et al. [39] method employs the window-based self-attention mechanism to capture global and local spatial information, it is a pure transformer-based network requiring a larger volume of data for training. This fact influences the segmentation of the SpeConv + Cao et al. [39] method with the worst performance of a 0.6589 DICE and a 0.6653 F1-score. Imran et al. [40] adopts an appropriate dense connection design and uses a CNN-based feature-preserving block to mitigate the loss of spatial information, which results in it achieving better segmentation results when combined with SpeConv+, compared to the two CNN-based methods, i.e., Walsh et al. [37] and Wang et al. [38]. However, its pure CNN architecture cannot capture global long-term dependencies among CT slices well, which degrades its segmentation, with the DICE of 0.7210 and the F1-score of 0.7215.

AlSalmi et al. [41] introduces an attention gate (AG) mechanism in the fusion of multi-scale information, leading to the network overly focusing on boundary areas with gray values similar to the hepatic ducts. Although this improves the probability of correctly predicting the target areas with the best performance of 0.7959 sensitivity, it also leads to an increase in false predictions. Both the Chen et al. [31] and our network introduce the self-attention blocks after the CNN-based feature extraction to allow for a more effective focus on relevant high-level features and thus to avoid the prediction of more erroneous areas. So, the SpeConv + Chen et al. [31] method and ours can achieve the two best segmentation performances for hepatic ducts. Especially, our model achieves the best performance of a 0.7690 DICE and a 0.7712 F1-score. It can be attributed to two reasons. One is that the ConvLSTM can well reveal the spatiotemporal correlation existing in the CT scans of hepatic ducts in the high-dimensional feature space. The other is that structure-aware loss function involving the SSIM regulator can assign the contribution of the generated CT slice to the total loss of the proposed framework.

Figure 3 and Figure 4, respectively, illustrate the segmentation results for hepatic ducts and corresponding 3D reconstruction visualizations achieved by different models. In Figure 3, it is evident that when the Speconv network is separately paired by Walsh et al. [41] and Cao et al. [39], the resulting visual segmentations exhibit many over-segmentations and mis-segmentations. When the Speconv network is combined with Chen et al. [31], some missed segmentations emerge. Our designed W-shaped Net demonstrates superior accuracy in hepatic duct segmentation, closely resembling real hepatic duct segmentation results. Moreover, as shown in Figure 4, the 3D reconstruction obtained by our model exhibits fewer artifacts and discontinuities compared to other models, which is highly similar to the ground truth.

5.2. Ablation Experiments

5.2.1. Influence of ConvLSTM

In this subsection, we conducted experiments to discuss the influence of the ConvLSTM module on the proposed W-shaped Net in which the networks without/with several ConvLSTM modules were established for the segmentation of hepatic ducts.

As indicated in

Table 4. Segmentation results achieved by the W-shaped Net with different ConvLSTM modules.

Models	DICE	Sensitivity	Specificity	F1-Score
w/ConvLSTM (added in L1&L2) (Ours)	0.7690	0.7792	0.9990	0.7712
w/o ConvLSTM	0.7559	0.7807	0.9991	0.7611
w/ConvLSTM (added in L1)	0.7608	0.8137	0.9988	0.7608
w/ConvLSTM (added in L1&L2&L3)	0.7032	0.8346	0.9985	0.7128
w/ConvLSTM (added in L1&L2&L3&L4)	0.7300	0.7728	0.9989	0.7322

Note: As indicated in Table 2, “L1” and “L2” refer to decoding layers 1 and 2 in the segmentation subnetwork, respectively, and so forth.

, the W-shaped Net without ConvLSTM performs worse than those with the ConvLSTM modules in the first decoder block and in the first two decoder blocks. This is because the ConvLSTM effectively learns the high-dimensional correlations of the adjacent CT slices involving hepatic ducts. Especially, the model with two ConvLSTM modules achieves the best performance of 0.7690 DICE and a 0.7712 F1-score. However, it is interesting that the segmentation performance decreases significantly when the ConvLSTM modules are integrated into the decoders at the high-dimensional and low-dimensional spaces. This is possibly because the feature information of other tissues is wrongly integrated at the low-dimensional feature spaces due to their subtle grayscale differences from the hepatic ducts. Moreover, when the ConvLSTM modules are simultaneously incorporated into the first three decoders, the corresponding model achieves the best sensitivity of 0.8346 but the worst performance for three metrics. The facts indicate that many pixels of other tissues are wrongly considered as those of hepatic ducts, which will puzzle the pre-operation and even influence the precise minimally invasive surgery for the extraction of hepatobiliary stones.

5.2.2. Influence of the Regulator in the Loss Function

As indicated in Equation (10), the SSIM is introduced as a regulator to dynamically assign the contribution of the generated CT slice to the total loss of the designed W-shaped Net. In this subsection, we conducted several experiments to demonstrate its influence on the designed network.

As indicated in (10), the smaller the SSIM regulator is, the larger the difference between the generated CT slice and the ground truth is, the lesser influence of the generated CT slice and the stronger influence of the two adjacent CT slices on the W-shaped Net are, and vice versa. Thus, as shown in

Table 5. Segmentation results achieved by the W-shaped Net with different loss regulators.

Models	DICE	Sensitivity	Specificity	F1-Score
w/SSIM (Ours)	0.7690	0.7792	0.9990	0.7712
w/o SSIM	0.7626	0.7515	0.9992	0.7626
SSIM→MSE	0.6387	0.6379	0.9995	0.6439
SSIM→PSNR	0.7541	0.7869	0.9990	0.7541

, the total loss with the SSIM regulator results in higher segmentation performance in model predictions compared to that without SSIM, which demonstrates its ability of dynamically allocating the contribution of generated inter-slices to the segmentation subnetwork. Moreover, the models with the MSE regulator and with the PSNR regulator perform worse than those without any loss regulator, which further demonstrates that the SSIM can better characterize the structural correlations of hepatic ducts in different CT slices.

6. Discussions

6.1. Interpretability of the W-Shaped Net

In this section, we visually evaluate the qualities of the generated CT slices and the segmented results achieved by our designed W-shaped Net. To do so, one duct tree is utilized to visually illustrate its reconstructed 3D image and some slices extracted from the 3D image, as illustrated in Figure 5 and Figure 6.

For the inter-slice super-resolution subnetwork, the SKE module is employed to estimate the two input slices using 1D kernels, and ultimately to obtain the generated slice. This module can compute the weights of the two input slices based on internal kernel parameters, which are visually displayed in Figure 5 and Figure 6. As indicated in Figure 5 and Figure 6, one of two weight maps has a higher brightness than the other one, which indicates that the information (pixel values) for the generated slice mainly comes from the weight map with high brightness. Moreover, the generated CT slices are visually similar to the ground truth slices, which demonstrates that the inter-slice super-resolution subnetwork works.

As indicated in Figure 5, the well-reconstructed region in the 3D image demonstrates a dispersed distribution within a certain area in the ground truth. The segmentation of three successive CT slices achieved by the W-shaped Net is highly prone to the ground truth, which demonstrates the effectiveness of the fusion mechanism between the CNN for local feature extraction and the transformer for global feature extraction. As indicated in Figure 6, the poorly reconstructed region in the 3D image demonstrates a wide continuous distribution within a certain area in the ground truth. Except for the generated slice, many hepatic duct regions are not well segmented from the two adjacent slices. This is possibly attributed to the poor handling of correlations among successive slices, since the ConvLSTM module may pay more attention to local correlations and overlook the global correlations.

6.2. Comparisons of Traditional Inter-Slice Interpolation

Normal CT devices mostly involve traditional inter-slice interpolation methods to improve the inter-slice resolution. In our study, we utilize deep learning (DL) for inter-slice interpolation. Thus, in this section, we conducted a comparison experiment to demonstrate the superiority of our inter-slice interpolation subnetwork, in which two widely used traditional inter-slice interpolation methods (linear and spline) are separately substituted for this subnetwork.

As indicated in

Table 6. Generated results via the W-shaped Net with different inter-slice interpolation methods.

Methods	SSIM	PSNR
DL-based interpolation (Ours)	0.9855	36.90
Linear interpolation	0.9704	35.18
Spline interpolation	0.9792	36.23

and

Table 7. Segmentation results via the W-shaped Net with different inter-slice interpolation methods.

Methods	DICE	Sensitivity	Specificity	F1-Score
DL-based interpolation (Ours)	0.7690	0.7792	0.9990	0.7712
Linear interpolation	0.7254	0.6993	0.9996	0.7254
Spline interpolation	0.7365	0.7422	0.9994	0.7370

, compared to the two traditional methods, our inter-slice interpolation subnetwork can generate better inter-slices, with an SSIM of 0.9855 and a PSNR of 36.90, which can further improve the subsequent segmentation performance with a DICE of 0.7690 and an F1-score of 0.7254.

6.3. Influence of the Number of Interpolated Inter-Slices

In this section, we conducted an experiment to discuss the segmentation performance of our method with different numbers of interpolated inter-slices between the two adjacent slices in the simulated CT scans with low inter-slice resolution. It is noted that we filtered out three slices from every four slices in the ground truth CT scans to simulate the generation of other CT slices with low inter-slice resolution. That is, assume that the CT scans have a total of 100 CT slices {1, 2, 3, 4, 5, …, 100}, then slice sampling with intervals can simulate the generation of the CT scans with low inter-slice resolution, i.e., the CT slices {1, 5, 9, …, 97}. Then, our method is separately performed on the two types of CT scans with low inter-slice resolution, i.e., for example, with twenty-five slices and with fifty slices, which indicates that three slices and one slice should, respectively, be generated between the two adjacent CT slices in the simulated CT scans with low inter-slice resolution.

As indicated in

Table 8. Segmentation results for different numbers of interpolated inter-slices.

Methods	DICE	Sensitivity	Specificity	F1-Score
Generate 1 Slice	0.7690	0.7792	0.9990	0.7712
Generate 3 Slices	0.7226	0.7341	0.9994	0.7226

, our method can also generate three inter-slices to achieve a fairly good segmentation performance. The decrease in the segmentation performance is attributed to the generation of more inter-slices.

6.4. Discussion on Practical Implications

As indicated in Figure 7, after scanning the patient with the CT equipment, some CT scans are obtained with possible low inter-slice resolution. Since our method can simultaneously perform both inter-slice interpolation and segmentation tasks, once these slices are input into our W-shaped Net model, it can generate segmentation results with high inter-slice resolution. Experienced surgeons then use 3D reconstruction software (like ITKSNAP 3.8) to convert these segmentation results into a 3D hepatic tree, which is subsequently analyzed and used for preoperative planning.

6.5. Limitations and Future Work

Although our designed W-shaped Net demonstrates superior segmentation performance and 3D reconstruction results for CT scans with improving inter-slice resolution, several issues will still need to be addressed in the future.

First, in this study, there are only 24 cases used for training, with only 1913 slices in total. Training with such a small dataset possibly leads to a potential overfitting issue for further poor generalization. As indicated in Figure 5 and Figure 6, when the hepatic ducts have a dispersed distribution within a certain area, our method performs well. However, when the hepatic ducts have a wide continuous distribution within a certain area, our method is not sensitive for good segmentation. Except for the mechanism of ConvLSTM, another possible reason is the insufficient training data where the hepatic ducts exhibit a continuous and large-range distribution. In the future, we will collect a large number of cases from hospitals through research cooperation, which can to some extent address this generalization issue.

Second, since the non-local block [42] has the ability of adaptively modeling along the temporal dimension to capture temporal cues well [43], this can be adapted to different scenarios by switching kernel functions. Thus, in the future, we will incorporate the non-local block into our proposed model to address the issue of ConvLSTM’s limited ability to capture global correlations among CT slices.

Third, for convenience, we adopted an existing inter-slice interpolation network (SpeConv [23]) as the inter-slice super-resolution subnetwork to generate the inter-slice between two adjacent CT slices. In the future, we will design a new inter-slice super-resolution subnetwork that dynamically adjusts the number of generated inter-slices according to the acquired CT scans with low inter-slice resolution to ensure that the 3D reconstruction of hepatic ducts meets the medical requirements well.

Fourth, the CT scans with low inter-slice resolution are simulated by slice sampling with intervals of CT scans with high inter-slice resolution, which inevitably have some discrepancies compared to the real-world data. In the future, we will validate our designed model in real-world slices with low inter-slice resolution for real clinical applications.

Also, since our W-shaped Net performs well for inter-slice interpolation and segmentation on complex and elongated hepatic duct structures, it can theoretically be extended to related fields involving similar structures. In the future, we will adapt our W-shaped Net to other medical CT data with similar anatomic structures.

7. Conclusions

Due to the limitations of low-performance CT equipment at some local hospitals, the acquired CT scans with a low inter-slice resolution cannot reconstruct high-resolution 3D hepatic duct trees. Even with the usage of some interpolation methods, the 2D segmentation results and 3D reconstruction results may not be ideal, which will influence the precise minimally invasive surgery for hepatobiliary stone disease.

In this paper, we design an end-to-end inter-slice super-resolution segmentation network for hepatic ducts, which cascades two U-Net-like subnetworks for two different tasks. The SpeConv is introduced as the inter-slice super-resolution subnetwork to obtain a generated slice between two adjacent abdominal CT slices involving hepatic ducts. The segmentation subnetwork is also an encoder–decoder structure, with several residual blocks and 12 transformer blocks for the encoder, and several upsampling blocks for the decoder. In the high-dimensional feature space, the ConvLSTM is introduced to deeply reveal spatiotemporal information among the CT scans of hepatic ducts. Our joint loss function with the SSIM regulator is designed to strengthen the interaction between two subnetworks.

Ablation experiments validate the significances of the introduction of the ConvLSTM and the SSIM regulator for the total loss on the designed W-shaped Net for hepatic duct segmentation. Comparative experiments indicate that our model outperforms the existing deep learning frameworks in terms of a 0.7690 DICE and a 0.7712 F1-score, which results in commendable 3D reconstruction for the hepatic duct tree.