Segmentation of Echocardiography Based on Deep Learning Model

Abstract

In order to achieve the classification of mitral regurgitation, a deep learning network VDS-UNET was designed to automatically segment the critical regions of echocardiography with three sections of apical two-chamber, apical three-chamber, and apical four-chamber. First, an expert-labeled dataset of 153 echocardiographic videos and 2183 images from 49 subjects was constructed. Then, the convolution layer in the VGG16 network was used to replace the contraction path in the original UNet network to extract image features, and depth supervision was added to the expansion path to achieve the segmentation of LA, LV, and MV. The results showed that the Dice coefficients of LA, LV, and MV were 0.935, 0.915, and 0.757, respectively. The proposed deep learning network can achieve simultaneous and accurate segmentation of LA, LV, and MV in multi-section echocardiography, laying a foundation for quantitative measurement of clinical parameters related to mitral regurgitation.

1. Introduction

Valvular heart disease is one of the causes of heart failure and has a high mortality rate. Mitral regurgitation (MR) is the most common type of heart valve disease [1]. Severe MR can cause various of complications, including heart failure, atrial fibrillation, infective endocarditis, pulmonary hypertension, and even sudden death [2]. The mechanism of MR can be broadly divided into degenerative MR and functional MR, which is mostly characterized by mitral valve (MV) prolapse/flail, leaflet redundancy/thickening, and chordal dysfunction [3]. Ventricular functional mitral regurgitation (vFMR) is commonly attributed to local or global left ventricular (LV) dysfunction and remodeling without structural MV abnormalities [4]. Recently, atrial functional mitral regurgitation (aFMR) has been increasingly recognized and is characterized by severe mitral annular dilatation, insufficient leaflet remodeling and decreased annular contractility in patients with long-standing atrial fibrillation (AF) [5]. Clinically, different treatment methods are adopted for different types of MR [6,7].

Echocardiography is currently the most important imaging method for the diagnosis and assessment of MR and has been widely used to determine etiology, qualitative, and quantitative MR severity, and cardiac hemodynamic changes [3,8,9]. According to the expert consensus of the American College of Cardiology on mitral regurgitation management, which was updated in 2020 [10], and the collection and analysis process of MR echocardiography listed in the American Echocardiography Guidelines [3], echocardiography analysis has high professional requirements for cardiologists and the quality of the analysis strongly depends on professional experience. Inexperience in the diagnosis and quantitative analysis of valvular disease may lead to inaccurate measurements, overestimation, or underestimation of valvular disease, which may delay the diagnosis and treatment of MR patients. Therefore, a unified standard, intelligent, and efficient auxiliary decision-making system is required in clinical practice to improve the accuracy and efficiency of MR diagnosis. The first step to achieve this goal is to segment the MV, atrium, and ventricle so that the required clinical parameters can be automatically and quantitatively measured.

Several studies have segmented the LV and left atrium (LA) using echocardiography. For example, Refs. [11,12,13,14,15] separated the LV at the end-diastolic and end-systolic stages to calculate the LV volume and ejection fraction. Currently, research on MV segmentation based on echocardiography is still in development. Sultanl et al. [16] proposed a method of locating the anterior MV region in m-mode ultrasound and transferring it back to conventional b-mode ultrasound images using a local active contour model to segment the anterior MV, with a Dice coefficient of 0.63. Costa et al. [17] used the UNet structure to segment the anterior and posterior MVs of the parasternal long-axis and apical four-chamber sections in patients with rheumatic heart disease, and the Dice coefficients were 0.742 and 0.795, respectively. Corinzia et al. [18] proposed a new automatic unsupervised segmentation method to achieve MV segmentation in the four-chamber section of the apex of the heart, with a Dice coefficient of 0.495.

Most existing work only achieves segmentation of one of the MVs, atria, or ventricles, which cannot meet the needs of MR typing. For example, the diagnosis of MV prolapse and non-ischemic cardiomyopathy requires both atrial and ventricular information. This study proposes a deep learning segmentation method combining UNet and the VGG16 network to achieve simultaneous segmentation of the LA, LV, and MV in echocardiography based on three different sections (apical two-chamber, apical three-chamber, and apical four-chamber).

2. Methods

2.1. Dataset

The dataset used in this study is from the Department of Cardiology, Zhongshan Hospital affiliated with Fudan University, and included 49 subjects. There were 20 cases of MV with normal function, 19 of MV prolapse, 8 of non-ischemic cardiomyopathy, and 2 of atrial dysfunction. A total of 153 video clips, each lasting for 2 s, containing one or more complete cardiac cycles, were used in this study. In this study, 10–20 frames of images were randomly selected from each video, and a total of 2183 images were selected, as shown in

Table 1. Dataset.

	NM	DMR	FMR	aFMR	Total
2CH	110	265	129	46	550
3CH	206	225	141	39	611
4CH	542	284	164	32	1022
Total	858	774	434	117	2183

. The quality score of the images met the quality level of 3 or above, as described in [19], that is, the boundaries of three or four chambers are visible or clear. LabelMe software was used by professionals to label the atrial, ventricular, and MV regions. Different sections and their labelling examples are presented in Figure 1.

The study was approved by the Regional Committee for Medical and Health Research Ethics and was conducted in compliance with the ethical principles of the Declaration of Helsinki.

2.2. Model Construction

2.2.1. Overall Architecture of the Model

In this study, an end-to-end multi-category semantic segmentation network for echocardiography was constructed to achieve accurate segmentation of the LA, LV, and MV based on different heart sections. The UNet structure [20] was selected as the backbone of the network model, and the model was modified accordingly.

The network model constructed in this manuscript can simultaneously achieve segmentation of three parts (LA, LV, and MV) from the input of three different perspectives (apical two-chamber, three-chamber, and four-chamber), and the specific locations of the segmentation targets on the perspectives were different from each other. For the segmentation targets, the size of the MV was significantly smaller than those of the LA and LV, and the range of activity of the MV was larger. To locate the specific positions of the LA, LV, and MV, the model needs high semantic information to provide the basis for category recognition. Accordingly, the contraction path of the UNet structure, namely the encoder part, was replaced by the convolution part of VGG16 so that the network could extract semantic information of lower resolution. In addition, because of the continuous movement of the targets in the process of cardiac contraction and diastolic recording during echocardiography, their edges in the input image frame were not clear, and the detailed information was worse after network down-sampling, which required feature fusion of different scales to supplement high-resolution information. To solve this problem, the hop layer connection in the UNet structure was used to supplement the information, and a deep supervision mechanism was designed. Labels were used to constrain the middle layer of the network to help better learn relevant information, and the multi-scale feature fusion of the network side was taken as the final output.

The network architecture of the manuscript, shown in Figure 2, consists of three parts:

• The contraction path used the VGG16 network for image feature extraction;
• The expanded path restored the image by layer hopping and transpose convolution;
• The deep supervision mechanism supplemented multi-scale information.

2.2.2. Contraction Path

A contraction path was used to extract the image features. The convolutional part of VGG16 was used as the contraction path of the model. A convolution module was added compared with the original UNet network. Each down-sampling process doubles the number of channels and reduces the size of the corresponding feature image by half, enabling the network to extract features with a lower resolution. The preliminary features extracted from the contraction path were used as input for subsequent segmentation tasks.

2.2.3. Expansion Path

The expansion path was used for image restoration, and the up-sampling process was performed by transposition convolution. The image features of the corresponding height contraction path were fused through the skip connection, and then the 3 × 3 fillless convolution and ReLU activation function were used twice to provide finer features for segmentation, such as gradients. Each up-sampling process magnified the image twice and halved the number of channels.

2.2.4. Deep Supervision Structure

The deep supervision architecture was used to constrain the middle layer of the network and make full use of all the resolution information in the network to obtain multiscale feature fusion. For the output of each height of the extended path, a 1 × 1 convolution was added once to map each feature vector to the desired number of categories. After the results at each height were spliced in the image channel, a 1 × 1 convolution was carried out for feature fusion so that all the resolution information in the network can be trained fully and convergence can be accelerated. Finally, the segmentation results of the network prediction were obtained.

2.3. Loss Function

To achieve segmentation of the MV, LA, and LV simultaneously, it is necessary to predict not only whether the pixel belongs to the segmentation target but also the classification of pixels (LA, LV, or MV).

Therefore, the network outputs three channel images simultaneously, and each channel image represents the prediction result that the input image belongs to a specific category. ${\mathcal{X}}_i$ is defined as the predictive image output by the network; $y_i$ is the real label of the segmentation target; and i = 1, 2, and 3 correspond to the LA, LV, and MV, respectively. A weighted multi-classification cross-entropy function was designed as a loss function in this manuscript. First, the softmax function was used to calculate the network output image ${\mathcal{X}}_i$ and obtain the probability distribution of ${\mathcal{X}}_i$ for category prediction ${\widehat{y}}_i$, as shown in Equation (1). The cross-entropy H of the target distribution and prediction distribution is shown in Equation (2). Because the classification of the true label $y_i$ of the segmentation target is known, ${\mathcal{X}}_i$ must only belong to $y_i$, and the probability of belonging to $y_j$ (j ≠ i) is 0. The real label $y_i$ is re-coded into one-hot form, and the cross-entropy H only needs to calculate the value corresponding to the prediction image and the real label category. Then, the category loss $\mathcal{l}\left({\mathcal{X}}_i\right)$ is obtained as illustrated in Equation (3):

$${\widehat{y}}_i=\frac{e^{{\mathcal{X}}_i}}{{\sum }_{j=1}^n{e}^{{\mathcal{X}}_j}},$$

(1)

$$H=-{\sum }_{i=1}^n\left(y_i\times \ln {\widehat{y}}_i\right),$$

(2)

$$\begin{array}{ll}\mathcal{l}\left({\mathcal{X}}_i\right)=& -{\displaystyle {\displaystyle \sum }_{i=1}^n}\left(y_i\times \ln {\widehat{y}}_i\right)\\ & =-{\displaystyle {\displaystyle \sum }_{i=1}^n}\left(y_i\times \ln \frac{e^{{\mathcal{X}}_i}}{{{\displaystyle \sum }}_{j=1}^n{e}^{{\mathcal{X}}_j}}\right)\\ & =-\ln \frac{e^{{\mathcal{X}}_i}}{{{\displaystyle \sum }}_{j=1}^n{e}^{{\mathcal{X}}_j}}\\ & =-{\mathcal{X}}_i+\ln {{\displaystyle \sum }}_{j=1}^n{e}^{{\mathcal{X}}_j}\end{array}$$

(3)

Considering that the MV region is far smaller than the LA and LV, and the position changes are greater, a larger weight value was taken for the MV than the LA and LV. Thus, the network model pays more attention to the MV position, as shown in Equation (4):

$$\mathrm{weight}\left[{\omega }_{\mathrm{A}},{\omega }_V,{\omega }_M\right]=\left[1,1,4\right],$$

(4)

where ${\omega }_{\mathrm{A}},{\omega }_V,{\omega }_M$ are the weight values of the LA, LV, and MV loss, respectively.

The final loss was set as the weighted sum of each loss category, as shown in Equation (5):

$$\begin{array}{ll}loss& =\mathrm{weight}\left[{\omega }_{\mathrm{A}}\right]\times \mathcal{l}\left({\mathcal{X}}_1\right)+\mathrm{weight}\left[{\omega }_V\right]\times \mathcal{l}\left({\mathcal{X}}_2\right)+weight\left[{\omega }_M\right]\times \mathcal{l}\left({\mathcal{X}}_3\right)\\ & =\mathcal{l}\left({\mathcal{X}}_1\right)+\mathcal{l}\left({\mathcal{X}}_2\right)+4\mathcal{l}\left({\mathcal{X}}_3\right)\end{array}$$

(5)

3. Results

3.1. Details of Train

The training process lasted for 200 epochs. The learning rate adjustment utilized the ReduceLROnPlateau strategy. It was initiated with value of 10⁻³, and decreased when the loss did not fall within 10 epochs. In Figure 3, the training loss drops very quickly and steadily, and eventually settles around 10⁻³. Furthermore, the RMSProp optimizer was integrated to update the network weights.

3.2. Segmentation Results

In Figure 4, the red contour is marked by professionals, and the green contour is the result of network segmentation. It can be observed that the LA can be segmented well in all three sections of echocardiography. However, the segmentations of the LV and MV are not as good as LA because the presence of the valve cord and papillary muscle causes the differences of the pixel gray values of the LV and MV and the aorta interferes with segmentation of the LV in the apical three-chamber view.

3.3. Result Evaluation

In this study, pixel accuracy (PA), category pixel accuracy (CPA), intersection over union (IoU), and Dice coefficients, as shown in Equations (6)–(9), are adopted to evaluate the segmentation results:

$$PA=\frac{TP+TN}{TP+TN+FP+FN},$$

(6)

$$CPA=\frac{TP}{TP+FP},$$

(7)

$$IoU=\frac{TP}{TP+FP+FN},$$

(8)

$$Dice=\frac{2TP}{2TP+FP+FN}$$

(9)

where TP and TN are true and true negative pixels, respectively, and FP and FN are the false and false negative pixels, respectively. The segmentation results are shown in

Table 2. Results.

	LA	LV	MV
PA	0.991 ± 0.001
CPA	0.934 ± 0.002	0.925 ± 0.003	0.794 ± 0.007
IoU	0.880 ± 0.002	0.847 ± 0.001	0.615 ± 0.003
Dice	0.935 ± 0.002	0.915 ± 0.002	0.757 ± 0.003

.

4. Discussion

4.1. Comparison and Analysis

Table 3. Comparison of the segmentation results of this study and other studies.

Author	Method	Dataset	Dice (LA)	Dice (LV)	Dice (MV)
Sultanl, M.S., et al. [16]	M-mode, LAC	62 videos	/	/	0.63
Costa, E., et al. [17]	U-Net	101 videos	/	/	0.742 (PLAX) 0.795 (A4C)
Corinzia, L., et al. [18]	NN-MitralSeg	39 videos	/	/	0.495
Pedrosa, J., et al. [11]	SSM, lAAOF	CETUS	/	0.909 (ED), 0.875 (ES)	/
Ali, Y., et al. [12]	Res-U	CAMUS	/	0.975 (ED), 0.972 (ES)	/
Liu, F., et al. [13]	PLANet	CAMUS & sub-EchoNet-Dynamic	/	0.942 (ED), 0.918 (ES)	/
Alexander Haak, et al. [21]	ASM	63D TEE volumes	0.92	/	/
Zyuzin, V., et al. [14]	Res-U	CAMUS	0.904	/	/
Zhao, C., et al. [15]	MS-Net	CAMUS & 127 videos	0.98	/	/
Proposed	VDS-UNet	153 videos	0.935	0.915	0.757

compares the segmentation results of the proposed method with those of other studies.

The segmentation in previous research was only conducted for a single object such as LA [14,15,21], LV [11,12,13], or MV [16,17,18]. In the cases of LV [11,12,13], it was segmented at a specific moment of cardiac cycle (ED/ES). In [17], UNET was used to segment a single object (MV) in two different views (PLAX, and A4C). The network proposed in [17] can provide more low-resolution information and high-resolution information of different scales to improve the MV segmentation results. In [12,14], ResNet50 or ResNet34 was used as the encoder of UNet to segment LA or LV without considering the size difference or position difference between MV, LA, and LV. Dice loss was used in the loss function.

In this proposed manuscript, a multi-category cross entropy loss function with weighted value was designed to give consideration to the segmentation effect of LA and LV as much as possible, and pay more attention to the MV position with a small target and wide variation range. The network model constructed can simultaneously segment three objects (LA, LV, and MV) of the echocardiographic three-section view (apical two-chamber, three-chamber, and four-chamber), and the input included images taken at any time in the cardiac cycle through feature fusion of information with different resolution scales. The segmentation Dice coefficients of the LV and LA were at the mean values, and the segmentation effect of the MV was better than that of other reported results. In addition, the network is valid in MV and MR.

4.2. Effectiveness of Model Construction

The network model proposed in this manuscript consists of three parts: the convolutional layer of VGG16 was used as the network contraction path, transposed convolution and skip connection were used as the network expansion path, and deep supervision architecture was used for multi-scale feature fusion. To prove the effectiveness of the proposed network structure, ablation experiments on all parts were performed. The hyperparameters in the training process of the network are consistent, and the results are presented in

Table 4. Effectiveness analysis.

	Dice (LA)	Dice (LV)	Dice (MV)
UNet	0.917	0.913	0.736
UNet (VGG16)	0.917	0.909	0.726
UNet (ds)	0.93	0.909	0.72
Propoesd	0.935	0.915	0.757

.

After the contraction path of the original UNet model was replaced by VGG16(UNet(VGG16)), although the network extracted more low-resolution features, but the segmentation results were not improved, the Dice coefficients of the LV and MV decreased by 0.004 and 0.01, respectively. After the original UNet model was enhanced with the supervision framework(UNet(ds)), the low-resolution and high-resolution features were better integrated; however, the semantic information extracted from the contraction path was reduced, and the segmentation effect for the MV with a smaller target and larger dynamic range was reduced. After applying both modifications to UNet (VDS-UNet) above, the Dice coefficients of the LA, LV, and MV were improved by 1.96%, 0.22%, and 2.72%, respectively, compared with the original UNet network.

5. Conclusions

In this manuscript, a deep learning segmentation method based on the combination of UNet and the VGG16 network is proposed to achieve simultaneous segmentation of the targets as LA, LV, and MV for apical two-chamber, three-chamber, and four-chamber echocardiography. The method was applied to segment targets from 2183 images extracted from 153 echocardiographic videos of 49 patients, with Dice coefficients of 0.935, 0.915, and 0.757, respectively. The method proposed in this manuscript can be used to quantitatively measure the relevant parameters of the LA, LV, and MV and lay a foundation for MR classification and grading.