A Symmetric Multiscale Detail-Guided Attention Network for Cardiac MR Image Semantic Segmentation

Abstract

Cardiac medical image segmentation can advance healthcare and embedded vision systems. In this paper, a symmetric semantic segmentation architecture for cardiac magnetic resonance (MR) images based on a symmetric multiscale detail-guided attention network is presented. Detailed information and multiscale attention maps can be exploited more efficiently in this model. A symmetric encoder and decoder are used to generate high-dimensional semantic feature maps and segmentation masks, respectively. First, a series of densely connected residual blocks is introduced for extracting high-dimensional semantic features. Second, an asymmetric detail-guided module is proposed. In this module, a feature pyramid is used to extract detailed information and generate detailed feature maps as part of the detail guidance of the model during the training phase, which are used to extract deep features of multiscale information and calculate a detail loss with specific encoder semantic features. Third, a series of multiscale upsampling attention blocks symmetrical to the encoder is introduced in the decoder of the model. For each upsampling attention block, feature fusion is first performed on the previous-level low-resolution features and the symmetric skip connections of the same layer, and then spatial and channel attention are used to enhance the features. Image gradients of the input images are also introduced at the end of the decoder. Finally, the predicted segmentation masks are obtained by calculating a detail loss and a segmentation loss. Our method demonstrates outstanding performance on the public cardiac MR image dataset, which can achieve significant results for endocardial and epicardial segmentation of the left ventricle (LV).

1. Introduction

In the process of the development of healthcare industries and embedded vision systems, cardiac image segmentation can promote and benefit them effectively. Image semantic segmentation aims at pixel-level semantic classification in a given image. It distinguishes meaningful objects (such as individual objects and regions of interest) by determining ground-truth boundary information. Traditionally, methods based on filtering [1], clustering [2], region [3,4], histograms [5], graphs [6,7], thresholds [8], the energy function [9,10,11], edges [12], and decomposition [13,14] are convenient for inputting various types of knowledge, but these types of methods have drawbacks, such as the simplicity of the image data to be segmented, insufficient use of public datasets, a low degree of automation, and limitations in terms of the effectiveness and segmentation of objects. Convolutional neural networks have overcome these problems and have been successful in many fields, such as image quality assessment [15,16,17,18,19,20,21,22,23,24,25], fault diagnosis [26,27], 3D point cloud analysis [28,29,30], video coding [31,32,33,34,35] and transmission [36,37], and image retrieval [38,39] The field of cardiac image segmentation is no exception. Cardiac magnetic resonance (MR) images have become the standard for noninvasive assessment of various cardiovascular functions. Therefore, cardiac MR image segmentation is crucial in the diagnosis and treatment process. In the image segmentation model, the context information and spatial resolution are particularly important and can correlate between pixels and retain the detailed information of the object. In cardiac MR image segmentation tasks, previous methods based on convolutional neural networks have attempted to preserve context information or spatial resolution to obtain higher accuracy [40,41,42,43,44,45,46,47,48].

However, detailed information, such as boundary and corner information, has not been exploited effectively, as the distinction between the boundary of cardiac MR images and surrounding pixels is not high, the shape changes greatly and is relatively fuzzy, and the grayscale and texture are relatively similar. Furthermore, the segmentation model needs to be more effective in understanding features of detailed information and contextual information. Moreover, contextual and detailed information, such as boundary and corner information, need to be fused further. In this work, a symmetric semantic segmentation architecture for cardiac magnetic resonance (MR) images based on a symmetric multiscale detail-guided attention network, which is an end-to-end trainable architecture, is presented. Detailed information and multiscale attention maps can be exploited more efficiently in this model. First, a series of densely connected residual blocks is introduced for extracting high-dimensional semantic features. Second, a detail-guided module is proposed. In this module, a feature pyramid is used to extract detailed information and generate detailed feature maps as part of the detail guidance of the model during the model training process, which are used to extract deep features of multiscale information and calculate a detail loss with semantic features. Third, multiscale upsampling blocks symmetrical to the encoder are employed in the decoder of the model. For each upsampling block, feature fusion is first performed on the previous-level low-resolution features and the symmetrical skip connections of the same layer, and then spatial and channel attention are used to enhance the features. A feature aggregation module and image gradients of the input images are also introduced at the top of the model and the end of the upsampling attention block, respectively. Finally, the predicted segmentation masks are obtained by calculating a detail loss and a segmentation loss. Our main contributions are as follows:

• We propose a semantic segmentation architecture that effectively extracts multiscale context and detailed features.
• We introduce a detail-guided module to extract detailed information and generate detailed feature maps in the encoder as part of the detail guidance of the model.
• We introduce a series of multiscale upsampling blocks that utilize spatial and channel attention, a feature aggregation module, and image gradients to extract multiscale and multilevel context.

In Section 2, we present related work on our method. Our method is derived in Section 3. Section 4 presents the experimental results. Finally, Section 5 draws conclusions.

2. Related Work

2.1. Resolution Preservation

The spatial resolution is preserved by an encoder–decoder architecture in these methods. Long et al. [49] proposed fully convolutional networks (FCNs) [49] that combine coarse and fine features. In DeconvNet [50] and SegNet [51], the resolution is preserved by a symmetrical network, and the fully connected layers are removed in SegNet to reduce the model parameters. LRR, proposed by Ghiasi et al. [52], performs upsampling operations on feature maps instead of score maps. In U-Net [53] and SPGNet [54], skip connections between the same level were added. Cheng et al. [55] proposed HigherHRNet, in which convolutional flows of different resolutions are connected with multiscale fusion across resolutions. Yu et al. [56] proposed BiSeNet, which uses convolution with more channels and reduces downsampling times to preserve resolution. Yan et al. [57,58] used Multi-Branch-CNN to maintain feature resolution. Zhang et al. [59] proposed a spatiotemporal video super-resolution network to reconstruct high-resolution images.

2.2. Context Embedding

Context methods include multilayer and multiscale context embedding methods. The multilayer context was extracted from the FCN, U-Net, and SPGNet. For the multiscale context, several segmentation models [60,61,62] have been introduced to extract the multiscale context via multiscale analysis. In the works of [63,64], dilated convolution was integrated to obtain representations of multiscale context. Both the DeepLab series proposed by Chen et al. [65,66,67] and the Denseaspp proposed by Yang et al. [68] applied the atrous spatial pyramid pooling module (ASPP) to increase receptive fields while maintaining high resolution. In DDRNets [69], proposed by Pan et al., deep aggregation pyramid pooling was used to enhance contextual information.

2.3. Attention Aggregation

Attention aggregation methods calculate the similarities of images, which can dynamically adjust the weights of different positions according to the content in the image so that the model can better automatically capture the information of interest. Zhao et al. [70] proposed PSANet, aggregating contextual information from all locations through similarity to generate dense pixel-level information, which introduced a spatial attention network to obtain attention context. Hu et al. [71] presented SENet, in which a squeeze-and-excitation unit was introduced to obtain the attention context. Jetley et al. [72] used spatial attention to enhance attention maps. Wang et al. [73] presented a nonlocal operation for capturing long-range dependencies. In addition, OCNet [74], DANet [75], CFNet [76], and CCNet [77] are also representative attention aggregation methods. Similarly, attention aggregations are also applied in image enhancement [78,79] and object detection tasks [80,81,82,83,84,85,86].

3. Proposed Method

A symmetric multiscale detail-guided attention network is presented for cardiac MR image semantic segmentation. An overview of our symmetric multiscale detail-guided attention network is shown in Figure 1. The input images are first sent to densely connected residual blocks after preprocessing. A detail-guided module is then employed on the corresponding labels to extract deep features of multiscale information, in which a feature pyramid is used to generate detailed feature maps, and a detail loss is calculated with semantic features. Next, a series of multiscale upsampling blocks symmetrical to the encoder is introduced to fuse the previous-level low-resolution features and the corresponding symmetrical skip connections and to enhance features via spatial and channel attention. Finally, the predicted segmentation masks are obtained via a multitask loss function, which calculates a detail loss and a segmentation loss.

3.1. CNN Architecture

To extract context information efficiently, a CNN architecture with multiscale context extraction is designed. We use densely connected residual blocks [87] to obtain semantic features. We introduce an ASPP module to extract multiscale context at the top of the encoder and decoder. For the decoder, multiscale upsampling attention blocks are integrated. To aggregate the multilevel features of the encoder, the outputs of ASPP and each upsampling attention block are concatenated with the cropped skip connection feature in the encoder. At the end of the upsampling attention block, image gradients of the input images are also introduced, followed by a normalized 3 × 3 convolution with batch normalization and ReLU activation $Con{v}_{3\times 3}$. In this hierarchical structure, the features of each level of the encoder have different scale perception capabilities, so effectively aggregating multilayer and multiscale information between levels helps to accurately segment objects of different sizes.

3.2. Detail-Guided Module

In our densely connected blocks, spatial detail information needs to be further extracted, so we use an asymmetric detail guidance module to extract spatial detail information in the encoder. The Laplacian image pyramid is used to process the ground-truth mask and utilize multiple levels of detail at different scales to generate detail feature maps. The segmentation head and densely connected block are then introduced to calculate a detail loss to guide the learning of spatial detail features. We use the detail guidance module to extract deep features of multiscale spatial detail information to fuse with the semantic features extracted in the encoder and pass them to the decoder through the downsampled densely connected blocks and corresponding skip connections. The detail feature maps $M_d$ on the ground truth G are computed as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill M_d=Con{v}_{1\times 1}{(L\left(G\right)}_{s=1}\left|\right|U\left(L{\left(G\right)}_{s=2}\right)\left|\right|U\left(L{\left(G\right)}_{s=4}\right))\end{array}\end{array}$$

(1)

where $L\left(G\right)$ denotes the Laplacian convolution on G and s denotes the stride. $U($·) is a bilinear upsampling function. $\left|\right|$ denotes the channelwise concatenation. $Con{v}_{1\times 1}$ is a normalized 1 × 1 convolution layer performed after the concatenation.

3.3. Multiscale Upsampling Attention Block

Multiscale upsampling attention blocks are introduced in the decoder of the model, which are symmetrical to the encoder. A diagram of the multiscale upsampling attention block is shown in Figure 2. For each upsampling block, the previous lower-resolution features are first upsampled via transposed convolution. Then, feature fusion is performed on the previous-level low-resolution features and the symmetrical skip connections of the same level from the densely connected residual blocks. After $Con{v}_{3\times 3}$, spatial and channel attention are used to enhance the ability to extract contextual and spatial information.

In the spatial attention path, two consecutive normalized 1 × 1 convolution layers $Con{v}_{1\times 1}$ are used to reduce the number of channels to 1, and a sigmoid function $\sigma$ is applied to rescale the feature values of each channel attention map to the interval [0, 1]. The spatial attention mechanism can be formulated as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill s\left(x\right)=x_c·\sigma (Con{v}_{1\times 1}\left(Con{v}_{1\times 1}\left(x\right)\right)\end{array}\end{array}$$

(2)

where $x_c$ denotes all the channels in the combined feature maps, which means stacking channelwise for matching output dimensions.

In the channel attention path, the squeeze-and-excitation module [71] is employed for channel weight adaptation. The module generates a scaling factor in [0, 1] for each channel of the input features and scales the channel according to this factor to obtain the output $c\left(x\right)$.

The output of the upsampling attention block is as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill UA\left(x\right)=(1+s(x\left)\right)\odot c\left(x\right)\end{array}\end{array}$$

(3)

where ⊙ denotes an elementwise product.

3.4. Loss Function

We apply a multitask loss for our multiscale detail-guided attention network, which combines detail loss and segmentation loss organically for these two tasks and performs well in our experiments. The total loss is given by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{total}=L_{detail}+L_{seg}\end{array}\end{array}$$

(4)

For detail loss, the detail maps obtained by the detail-guided module are binary feature maps, and the proportion of details in all the pixels is small. Therefore, we use binary cross-entropy and Dice loss as detail loss to jointly optimize learning to align the spatial detail features extracted by the encoder with the detail ground truth. The detail loss $L_{detail}$ is as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{detail}=L_{bce}(d,\widehat{d})+L_{dice}(d,\widehat{d})\end{array}\end{array}$$

(5)

where $d,\widehat{d}\in {\mathbb{R}}^{h\times w}$ are the ground-truth detail label and predicted detail, respectively. All the pixels are in the domain with height h and width w. $L_{detail}$ is composed of a binary cross-entropy loss $L_{\mathrm{bce}}(d,\widehat{d})$ and a Dice loss $L_{dice}\left(d,\widehat{d}\right)$, which are as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L_{\mathrm{bce}}(d,\widehat{d})=-{\displaystyle \frac{1}{N}}\sum _i^{\mathrm{N}}& \left[d_{i}log\left({\widehat{d}}_i\right)+\left(1-d_i\right)log\left(1-{\widehat{d}}_i\right)\right]\hfill \end{array}\end{array}$$

(6)

where N represents the domain with height h and width w. i is the i-th pixel of N, and $d_i$ and ${\widehat{d}}_i$ are the ground-truth detail label and predicted detail of the i-th pixel, respectively.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{dice}\left(d,\widehat{d}\right)=1-{\displaystyle \frac{2{\sum }_i^N{d}_i{\widehat{d}}_i+\epsilon }{{\sum }_i^N{\left(d_i\right)}^2+{\sum }_i^N{\left({\widehat{d}}_i\right)}^2+\epsilon }}\end{array}\end{array}$$

(7)

where $\epsilon$ is a smoothing item. Here, we set $\epsilon$ = 1.

The segmentation loss $L_{seg}$ is composed of $L_{ce}(y,\widehat{y})$ and $L_{dice}(y,\widehat{y})$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{seg}=L_{ce}+L_{dice}\end{array}\end{array}$$

(8)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{ce}(y,\widehat{y})=-{\displaystyle \frac{1}{N}}\sum _i^{\mathrm{N}}{y}_{i}log\left({\widehat{y}}_i\right)\end{array}\end{array}$$

(9)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L_{dice}(y,\widehat{y})=K-\sum _k{\displaystyle \frac{2{\sum }_i^N{y}_{ik}{\widehat{y}}_{ik}}{{\sum }_i^N{y}_{ik}+{\sum }_i^N{\widehat{y}}_{ik}+eps}}\end{array}\end{array}$$

(10)

where y and $\widehat{y}$ are the ground-truth coding and the predicted matrix, respectively. $y\in {\{0,1\}}^{h\times w\times K}$ and $\widehat{y}\in {[0,1]}^{h\times w\times K}$. $y_{ik}$ and ${\widehat{y}}_{ik}$ are the i-th pixel ground truth and prediction of category k. $eps$ is set to 1 × 10 ⁻⁷ for the numerical stability of the denominator.

4. Experiments

4.1. Implementation Details

Our method is tested on a public magnetic resonance image segmentation dataset—the Sunnybrook Left Ventricle Segmentation Challenge dataset [88]. The Sunnybrook dataset contains short-axis cine MR images of 45 subjects: hypertrophic (HYP), heart failure without/with infarction (HF-NI/HF-I), and healthy (N). It contains 15 cases each in the training set, validation set, and online set. The resolution is 256 × 256 for each slice, and all the training images are augmented through centre cropping, random rotation, and random horizontal/vertical flipping. We present the experimental results of the model implemented via PyTorch1.9 and trained on an NVIDIA 4090 GPU and an Intel^® Core^TM i9–13900K CPU, which was trained independently for 100 epochs with a batch size of 4 and the RAdam optimizer. The initial learning rate of the model was 0.001, which was exponentially decayed with a coefficient of 0.9. In the final stage, the loss curve gradually stabilizes.

4.2. Quantitative Results

The Jaccard coefficient, Dice similarity coefficient (DSC), and pixel accuracy are the evaluation metrics for our segmentation:

$$\begin{array}{c}\hfill Jaccard(P,G)={\displaystyle \frac{|P\cap G|}{|P\cup G|}}\end{array}$$

(11)

$$\begin{array}{c}\hfill Dice(P,G)={\displaystyle \frac{2|P\cap G|}{\left|P\right|+\left|G\right|}}\end{array}$$

(12)

The DSCs for the endocardium and epicardium of other works and our method are given in

Table 1. Dice scores of different methods.

Method	Endocardium	Epicardium
Romaguera et al. [40]	0.9200	0.9300
Trinh et al. [41]	0.9196	0.9505
SAUNet [42]	0.9520	0.9620
DR-Unet++ [43]	0.9131	0.9315
FCN-ACM [44]	0.9400	0.9600
nnU-net [45]	0.9527	0.9648
nnformer [46]	0.9572	0.9695
Swin-unet [47]	0.9548	0.9639
SwinUMamba [48]	0.9531	0.9636
Ours	0.9586	0.9718

. The best metric for each category is marked in bold. The top eight reported accuracies of previous works are listed. These previously proposed methods are fully automatic segmentation methods. Table 1 demonstrates the superiority of our multiscale detail-guided attention network, which can achieve the best DSC in the endocardium and epicardium compared with other methods. Our model can achieve DSC improvements of 0.69% and 1.02% in Dice scores of the endocardium and epicardium, respectively, over the second-ranked method.

For different cardiac phases (end-diastolic (ED) and end-systolic (ES)), the pixel accuracy and Jaccard and Dice scores for the endocardium and epicardium are presented in

Table 2. Pixel accuracy and Jaccard and Dice scores in the ED and ES cardiac phases.

	Cardiac Phase
Geometric Metric	End-Diastole		End-Systole
	Endocardium	Epicardium	Endocardium	Epicardium
Pixel Accuracy	0.9606	0.9781	0.9612	-
Jaccard	0.9198	0.9451	0.9212	-
DSC	0.9582	0.9718	0.9590	-

. Notably, only the contour of the endocardium is labeled in the ES phase in the dataset. As shown in Table 2, the endocardium segmentation in the ES phase is better than that in the ED phase. Some visualizations of segmentations in the ED and ES phases are given in Figure 3, which demonstrates the performance of our method in every subject group. HYP-06, HF-NI-11, HF-I-05, and SC-N-05 represent cavity hypertrophy subject No. 6, heart failure without infarction subject No. 11, heart failure with infarction subject No. 5, and normal subject No. 5, respectively.

4.3. Ablation Experiments

The baseline of our experiments is a basic model whose encoder consists of densely connected residual blocks and whose decoder consists mainly of transposed convolutions. The Jaccard and DSC for the endocardium and epicardium with different settings of ablation studies are presented in

Table 3. Ablation studies on the detail-guided module.

Method	Endocardium		Epicardium
	Jaccard	DSC	Jaccard	DSC
Baseline	0.8768	0.9344	0.8981	0.9463
Baseline + DG	0.8883	0.9408	0.9132	0.9546

,

Table 4. Ablation studies on the pooling operation.

Method	Endocardium		Epicardium
	Jaccard	DSC	Jaccard	DSC
Baseline	0.8768	0.9344	0.8981	0.9463
Baseline + DG + PPM	0.8898	0.9417	0.9161	0.9562
Baseline + DG + ASPP	0.8922	0.9430	0.9190	0.9578

,

Table 5. Ablation studies on the multiscale upsampling attention block.

Method	Endocardium		Epicardium
	Jaccard	DSC	Jaccard	DSC
Baseline	0.8768	0.9344	0.8981	0.9463
Baseline + DG + ASPP + UA	0.9113	0.9536	0.9367	0.9673

and

Table 6. Ablation studies on image gradients.

Method	Endocardium		Epicardium
	Jaccard	DSC	Jaccard	DSC
Baseline	0.8768	0.9344	0.8981	0.9463
Baseline + DG + ASPP + UA + Sobel	0.9155	0.9559	0.9406	0.9694
Baseline + DG + ASPP + UA + Canny	0.9205	0.9586	0.9451	0.9718

.

For the detail-guided module (DG) component, Table 3 shows that, compared with the method without detail guidance, the method with detail guidance achieves average performance improvements of 1.33% and 0.73%, respectively, in terms of the Jaccard and DSC.

The pooling operation component is applied at the top of the model to extract the multiscale context. We apply PPM and ASPP modules. Table 4 demonstrates that the method with the ASPP module attained average performance improvements of 1.55% and 0.86% in terms of the Jaccard and DSC, respectively, compared with the baseline. Compared with the method using the PPM module, the method using the ASPP module attained average performance improvements of 0.26% and 0.14% in terms of the Jaccard and DSC, respectively.

For the multiscale upsampling attention block (UA) component, it is applied as a part of the decoder of the model for further extracting context. Table 5 shows that the method with the UA module attained average performance improvements of 3.65% and 2.01% in terms of the Jaccard and DSC, respectively, compared with the baseline.

For the image gradient component, we perform methods without/with image gradients generated by a Sobel/Canny edge detector at the end of the upsampling attention block separately. Table 6 demonstrates that Canny performs better, and the method with the Canny image gradients attained average performance improvements of 4.54% and 2.48% in terms of the Jaccard and DSC, respectively, compared with the baseline.

5. Conclusions

A symmetric multiscale detail-guided attention network for cardiac MR image segmentation, which can extract context and detailed features effectively, is presented in this paper. In the encoder, a detail-guided module was introduced to extract detailed information and generate detailed feature maps, which were combined with semantic features from the densely connected residual blocks as part of the detail guidance of the model. An ASPP module was also applied to extract multiscale context. To aggregate multilayer and multiscale information between levels effectively, a series of upsampling blocks was also introduced in the decoder of the model. Moreover, a feature aggregation module and image gradients of the input images are also integrated to extract multiscale and multilevel context. Our method was tested on the public cardiac MR image dataset. Our method achieved favourable performance for endocardial and epicardial segmentation of the left ventricle. In future work, if context from different directions can be better integrated and the adaptive weight selection is performed based on the high-, medium-, and low frequency regions of cardiac medical images, then valuable pixel regions can be integrated, and the accuracy will be further improved.