Automated Clinical Decision Support for Coronary Plaques Characterization from Optical Coherence Tomography Imaging with Fused Neural Networks

Abstract

Deep Neural Networks (DNNs) are nurturing clinical decision support systems for the detection and accurate modeling of coronary arterial plaques. However, efficient plaque characterization in time-constrained settings is still an open problem. The purpose of this study is to develop a novel automated classification architecture viable for the real-time clinical detection and classification of coronary artery plaques, and secondly, to use the novel dataset of OCT images for data augmentation. Further, the purpose is to validate the efficacy of transfer learning for arterial plaques classification. In this perspective, a novel time-efficient classification architecture based on DNNs is proposed. A new data set consisting of in-vivo patient Optical Coherence Tomography (OCT) images labeled by three trained experts was created and dynamically programmed. Generative Adversarial Networks (GANs) were used for populating the coronary aerial plaques dataset. We removed the fully connected layers, including softmax and the cross-entropy in the GoogleNet framework, and replaced them with the Support Vector Machines (SVMs). Our proposed architecture limits weight up-gradation cycles to only modified layers and computes the global hyper-plane in a timely, competitive fashion. Transfer learning was used for high-level discriminative feature learning. Cross-entropy loss was minimized by using the Adam optimizer for model training. A train validation scheme was used to determine the classification accuracy. Automated plaques differentiation in addition to their detection was found to agree with the clinical findings. Our customized fused classification scheme outperforms the other leading reported works with an overall accuracy of 96.84%, and multiple folds reduced elapsed time demonstrating it as a viable choice for real-time clinical settings.

1. Introduction

Deep Neural Networks (DNNs) are fueling medical imaging technology, medical diagnostics, and healthcare in general [1]. DNNs also continue to be of significance in the field of cardiovascular imaging for the detection of coronary arterial plaques [2,3]. Coronary plaques are cholesterol deposits in the wall of the heart arteries and are the leading cause of death globally (projected one in four deaths). According to the World Health Organization (WHO), 85% of these deaths were due to plaque buildup that resulted in the narrowing of the coronary arteries through a process termed atherosclerosis.

Each layer of DNN optimized its weights based on the Boltzmann machine [4] to avoid overfitting and vanishing gradients when used in cardiac imaging problems. DNNs with sufficient depth can foster compact representation that requires fewer training examples to tune the parameters and produces better classification results [5,6]. Due to their ability to correlate deep features with each layer, they are preferred over conventional architectures for improved patient outcomes [7,8,9,10,11,12]. However, the bulk of parameters sometimes leads to overfitting and poor generalization [13], and this paved way for the evolution of other leading architectures including AlexNet, ResNet, and GoogleNet [14,15,16,17,18]. These deep learning models embed Cartesian and polar image representations for multi-path classification architectures [19,20,21] but at the cost of computational effort.

The resolve of this paper is to develop a time-efficient classification architecture for real-time clinical support systems and to validate Generative Adversarial Networks (GANs) for data augmentation. We propose a time-efficient hybrid fused Convolutional Neural Networks (CNNs) classifier. Our proposed architecture with Support Vector Machines (SVMs) embedded in high-density layers reduced the computational burden. Data augmentation was done to remove the class imbalance [22,23]. The objective functions of the generator and discriminator were optimized via the gradient descent method to make perfect aliases. Transfer learning was used to train our OCT dataset by freezing early layers of the architecture and fine-tuning them later on. For ground-truth annotation, three trained experts with daily experience in OCT-assisted interventions determined the plaque type in an A-Scan. All experts were partially provided the same and different images for labeling. For the same images, the final label was determined based on the consensus between the experts. The proposed architecture recorded the best classification performance in minimal time compared to other leading architectures, with potential viability for enhancing decision-making in clinical settings.

2. Methods

A novel dataset of 53 coronary stenosis in 39 patients was assessed using labels by an OCT available system (C7-XR, St. Jude Medical, St. Paul, MA, USA) using the C7 Dragonfly intravascular OCT catheter (St. Jude). This system provides spatial resolution up to 10 µm and tissue penetration up to 3 mm. Target vessels were those with stenosis (>30% diameter stenosis visual estimation). Serial stenosis, left main stenosis, by-pass graft stenosis, and anatomical characteristics were excluded from the study, as this would distort results. The study was approved by Galway Clinical Research Ethics Committee (GCREC) and informed consent was obtained from the patients. The dataset was built from acquired OCT images having A-scans. All experts were partially provided the same images and different images for labeling. For the same images, the final label was determined based on consensus between the experts. We classified based on the particular label versus the rest. Preprocessing steps were applied to raw OCT acquired images, and the A-line values within the guidewire were set to zero. Vulnerable plaques were determined and excluded using the flow diagram as illustrated in Figure 1.

Finally, we had 20% images labeled as “calcified,” 20% as “lipid plaques,” 15% were labeled as “fibro-lipidic plaque,” 15% “fibro-calcified,” 5% were labeled as “mixed plaque,” and 25% labeled as “no plaque”.

2.1. Data Augmentation Using GANs

For data amplification, we used GANs, as these networks facilitate hybridization and ease merging pre-existing models [24,25,26,27]. GAN generator consisting of a fully connected layer was used. It helps in projecting input to the next layers using stride convolutions and is presented in Figure 2. We approximated unknown distribution through a generator that maps samples from a fixed prior distribution. The Generator (G) was trained in parallel with a Discriminator (D) by searching a saddle point. Batch normalization was performed in each layer.

The generator creates realistic samples, and a discriminator was then used to distinguish between real and real-like copies as illustrated in Figure 3.

2.2. Proposed Fused Deep Learning Classifier

Figure 3a represents the leading CNN architecture (GoogleNet), where each block represents the input and output features maps. Figure 3b is our proposed fused classification architecture, where the densely connected layers of the CNNs were replaced by SVMs as indicated using dotted blocks. Radial Basis Function (RBF) was used to ensure faster convergence of the global hyper-plane. As reported elsewhere [28,29], we implemented transfer learning to ensure improved learning in the target domain. The labeled data was fed to the classifier and, via transfer learning, different features were extracted at different levels in the network. For transfer learning, we removed the final layer and used GoogleNet as our pre-trained model. Then, we unfroze convolution layers 4 and 5 while keeping the first three blocks frozen for the second pass of training. Finally, the replaced end layers were trained by freezing all convolutional layers of the module (GoogleNet). Fine-tuning was done by removing the fully connected nodes and embedding new layers as illustrated in Figure 3b. We normalized our network predictions based on the cross-entropy between the true label distribution and the predicted label. Hinge Loss was set up for the SVM classifier based on maximum margin classification. For model training, we minimized the cross-entropy loss by using the Adam optimizer with a learning rate of Lr = 10−4. To find the optimal schedule, we reduced the learning rate by a factor of two when the validation error saturated. In total, we trained each model for 100 epochs.

In our model, the intermediate output value Z was obtained as a result of convolution input data A from the previous layer with weight tensor W, as indicated in Equation (1). The model was trained without partitioning the replicas for memory optimization.

$$Z^l=W^l·A^{l-1}+b^l$$

(1)

where l is the number of layers and b is the bias term.

We minimized the cross-entropy loss L in our classification tasks using Equation (2).

$$L=-{\displaystyle \sum }_{i=1}^m{y}_{i}log{Y}_i$$

(2)

where m represents the number of classes, y is the ground truth label, and Y represents the softmax normalized model prediction.

Our model utilizes specific learning rates and suitable hyper-parameters including epoch count, batch size, and filter counts for each layer. Langrangian function (L_F) loss for back-propagation was computed using the expression in Equation (3).

$$L_F={\displaystyle \sum }_{i=1}^m{a}_i{y}_i{R}_b+b$$

(3)

where a_i represents the Langrangian multiplier and R_b is the Radial Basis Kernel function.

3. Results and Discussion

The gain of our proposed solution is that it not only detects but also characterizes coronary arterial plaques in real-time clinical settings. It provides better classification accuracy with much less computational effort as compared to the other leading classification architectures. Data Augmentation through GANs has proven handy for our limited arterial plaques dataset. However, data variability in other clinical datasets is yet to be validated in terms of generalization. Similarly, the effect of distributional shift caused by externalities is another key issue.

For all the experiments presented in this section, a train validation-test scheme was utilized. The presented results were calculated on the test set where the selection of hyper-parameters was made over the validation set. In the rest of this section, we compare the performance of different leading CNN-based classification architectures with our proposed fused CNN architecture for this dataset. Calculated results were cross verified with the preliminary expert’s marking and their automated characterization results as illustrated in Figure 4.

The confusion matrix was computed by calculating True Positive Rate (TPR), True Negative Rate (TNR), False Positive Rate (FPR), and False Negative Rate (FNR) along with Positive Predicted Value (PPV) and Negative Predicted Value (NPV) [30]. This is presented in

Table 1. Confusion Matrix for AlexNet Architecture.

1	11 8.8	0 0	1 0.8	0 0	0 0.0	1 0.8	84.6 15.4
2	0 0	23 18.4	2 1.6	2 1.6	1 0.8	1 0.8	79.3 20.7
3	0 0	3 2.4	33 26.4	1 0.8	1 0.8	0 0	86.8 13.2
4	1 0.8	3 2.4	2 1.6	21 16.8	0 0	0 0	77.8 22.2
5	0 0	0 0	2 1.6	0 0	7 5.6	0 0	77.8 22.2
6	2 1.6	0 0	0 0	0 0	0 0	7 5.6	77.8 22.2
	78.6 21.4	79.3 20.7	82.5 17.5	87.5 12.5	77.8 22.2	77.8 22.2	81.6 18.4
	1	2	3	4	5	6

.

The numbering on the left side and at the bottom of Table 1 represents different classes. Diagonal elements in the confusion matrix represent the number of correct predicted frames against each class label during the testing process. The cells in the last column indicate the percentage of correctly identified positive predictions (specificity) against each class. It was confirmed that the precision value for AlexNet remained above 77% for each of the six classes. The cells in the last row indicate the sensitivity of the model. Finally, the last diagonal cell is indicative of overall accuracy that remained 81.6%. After the training phase, the model was cross-validated by picking up a few random frames to mark its accuracy against real-time frames. The same procedure was repeated for the DenseNet model and the results are illustrated in

Table 2. Confusion Matrix for DenseNet architecture.

1	7 5.6	3 2.5	0 0	2 1.7	0 0	0 0	58.3 41.7
2	0 0	9 7.4	0 0	0 0	2 1.7	0 0	81.8 18.2
3	0 0	0 0	24 19.8	1 0.8	3 2.5	1 0.8	82.8 17.2
4	0 0	2 1.7	4 3.3	33 27.3	1 0.8	1 0.8	80.5 19.5
5	1 0.8	0 0	0 0	0 0	17 14.0	0 0	89.5 10.5
6	1 0.8	0 0	0 0	1 0.8	1 0.8	7 5.8	77.8 22.2
	77.8 22.2	64.3 35.7	85.7 14.3	89.2 10.8	70.8 29.2	77.8 22.2	80.8 19.9
	1	2	3	4	5	6

. GoogleNet resulted in calcified plaque detection with PPV of 58.3% and an overall accuracy of 80.2%, whereas

Table 3. Confusion Matrix for the proposed CNN architecture.

1	11 88	0 0	0 0	2 1.7	0 0	0 0	91.7% 8.3%
2	0 0	23 18.5	2 1.6	2 1.6	1 0.8	1 0.8	99.3% 20.7%
3	0 0	4 3.2	35 28	2 1.6	1 0.8	0 0	93.3% 16.7%
4	0 0	2 1.6	1 0.8	20 16	0 0	0 0	99.3% 16.7%
5	1 0.8	0 0	2 1.6	0 0	7 5.6	0 0	97.8% 22.2%
6	2 1	0 0	0 0	0 0.8	1 0.8	7 5	95.8% 22.2%
	78.6% 21.4%	79.3% 20.7%	87.5% 12.5%	83.3% 16.7%	77.8% 2.2%	77.8% 22.2%	96.84% 17.6%
	1	2	3	4	5	6

highlights that our proposed architecture has achieved the highest classification accuracy of 96.84%.

Figure 5a demonstrates the training process of AlexNet with a layer size of 25. AlexNet was fed with an input tensor having a dimension of 227 × 227 × 3 and the learning rate was kept initially at 0.5 for low-level feature learning. It achieved 83.20% validation accuracy for 40 epochs after 29 iterations for the pre-defined six classes. The training process took 2900 iterations and the elapsed time remained 380 min and 22 s. Figure 5b revealed that the training DenseNet with the same layer size and an input tensor having a dimension of 227 × 227 × 3 achieved 79.37% validation accuracy. However, the training process took 2900 iterations and the measured elapsed time was 3215 min and 31 s. Though both these architectures provide reasonable accuracies, they suffer from intense time computations. Hence, they are not ideal for any real-time clinical decision support.

As illustrated in Figure 6, the validation accuracy for the GoogleNet was found to be 83.4% with an elapsed time of 92 min and 37 s, but our modified architecture with SVM embedded in the output layers of GoogleNet resulted in an accuracy of 96.84%. This is 16% more than the state-of-the-art GoogleNet with an elapsed time of 20 s that is ideal for real-time implementations.

In this paper, cutting-edge models [31,32,33,34,35,36] were compared to our proposed architecture. DenseNet, AlexNet, and GoogleNet architectures were implemented and tested with our newly created dataset. AlexNet includes repeating convolutional layers, followed by max-pooling and then a few dense layers operations, whereas DenseNet relied heavily on the extensive computations. GoogleNet, with a quite different architecture from both, uses combinations of inception modules and 1 × 1 feature convolutions for feature selection. Each inception module captured salient features at different levels and concatenated them before feeding them to the next layer. Our proposed architecture is unique as it takes advantage of multi-level feature extraction, both general (5 × 5) and local (1 × 1), concurrently. The output format of our fully connected layer is 1 × 10 which significantly reduces the number of training parameters as compared to a conventional CNN model. By exploiting batch normalization, the sample distribution characteristics within the same layer were preserved while the distribution gap between layers was eliminated.

Table 4. Inter-comparison of the proposed architecture with the leading classification schemes in terms of accuracy and time.

Sr. No.	Pre-Trained Model	Input Dimensions	Elapsed Time	No of Epochs	Accuracy
1	AlexNet	227 × 227 × 3	380 min and 22 s	100	81.6%
2	GoogleNet	224 × 224 × 3	92 min and 37 s	100	82.64%
3	ResNet 50	224 × 224 × 3	1509 min and 21 s	100	79.34%
4	ResNet 101	224 × 224 × 3	1711 min and 8 s	100	73.73%
5	Densenet	224 × 224 × 3	3215 min and 31 s	100	80.17%
6	Proposed architecture	224 × 224 × 3	20 s	70	96.84 %

indicates the accuracy of leading architectures against our proposed architecture in terms of elapsed time, number of iterations, and number of layers involved. Elapsed time was found to be a function of several parameters. Proposed fused CNN took a minimum of 17 iterations per epoch to reach the highest classification accuracy of 96.84%. This inter-comparison substantiates the motivation behind the development of the proposed architecture.

4. Conclusions

We presented an in-depth exploration of plaque detection in OCT pullbacks using hybrid CNNs. We validated our model by creating a new dataset of OCT acquired images labeled by three trained experts. GANs were implemented for the synthetic creation of OCT images. Insertion of fully connected SVMs in the GoogleNet at the Softmax layer leads to better extraction of cross-entropy features and multi-class label prediction. Real-time processing in a few seconds is indicative of the potential of our model to become integrated into catheter laboratories for real-time assessments.