Reduction of Video Capsule Endoscopy Reading Times Using Deep Learning with Small Data

Abstract

Video capsule endoscopy (VCE) is an innovation that has revolutionized care within the field of gastroenterology, but the time needed to read the studies generated has often been cited as an area for improvement. With the aid of artificial intelligence, various fields have been able to improve the efficiency of their core processes by reducing the burden of irrelevant stimuli on their human elements. In this study, we have created and trained a convolutional neural network (CNN) capable of significantly reducing capsule endoscopy reading times by eliminating normal parts of the video while retaining abnormal ones. Our model, a variation of ResNet50, was able to reduce VCE video length by 47% on average and capture abnormal segments on VCE with 100% accuracy on three VCE videos as confirmed by the reading physician. The ability to successfully pre-process VCE footage as we have demonstrated will greatly increase the practicality of VCE technology without the expense of hundreds of hours of physician annotated videos.

1. Introduction

Upon its implementation in 2001, video capsule endoscopy (VCE) revolutionized the field of gastroenterology (GI) by allowing physicians to easily visualize areas of the bowel that were previously impossible to examine [1,2]. Despite being a powerful solution for examining 600 cm of small bowel, the field of capsule endoscopy has largely stood still since then [3]. Additionally, although VCE has a wide range of clinical applications, interpretation of VCE studies can be a daunting task, as it requires an individual with expertise to review more than 50,000–70,000 image frames, with 8 h of recording, looking for abnormal pathology that is often only present in one or two frames [4,5]. As a result of the limitations of human concentration [6], VCE readings have a significant miss rate of 5.9% for vascular lesions, 0.5% for ulcers, and 18.9% for neoplasms [7]. Over the years, VCE manufacturers have continuously refined their software to include advances in interpretation aids such as blood indicators, Quickview modes, and adaptive framerates, yet capsule reading remains a tedious and time-consuming process [8,9,10]. With the evolution of artificial intelligence (AI), computer-aided diagnosis (CAD) has shown promise in many areas of medicine, including pathology, dermatology, radiology and gastroenterology, where it has been used to decrease observational oversights (i.e., human error). With the above-mentioned limitations of VCE, CAD seems to be well poised to resolve several gaps in maximizing the function of VCE [11]. Though previous studies have been published regarding the use of CAD with VCE [12,13,14,15,16,17,18,19], these studies are either conducted on tens to hundreds of non-publicly available videos, or only use publicly available frame data. The aim of this study is to use CAD to identify abnormal vascular and inflammatory capsule endoscopy frames from full video footage, using a limited number of publicly available videos.

2. Materials and Methods

In this work, we used publicly available data from the $\kappa \acute{\alpha }\psi$o$\upsilon$$\lambda$$\alpha$ interactive database (KID) [20,21]. The database consists of two types of data: still frame images and videos. All data in the database was collected with a MiroCam device developed by IntroMedic Co in Seoul, Korea. The MiroCam has a recording frame rate of 3 frames per second [22]. However, the videos provided in the database are all MP4 files with a frame rate of five frames per second. Previous work [23,24,25,26,27,28,29,30] using this database focused on classification on the frame only data known as KID Datasets 1 and 2. These datasets were annotated by expert clinicians and come with both classification and segmentation labels. For the videos, only the KID video 1 contains classification labels and it is the only one to have been used in previous work [31] referencing the KID dataset. The other two videos are without expert annotation; with the help of our local Veteran’s Affairs (VA) Hospital, we are the first to use these previously unannotated KID videos. Our senior author, a VA physician with 20 years of GI experience, labeled videos 2 and 3 and adjusted the labels on the first video according to their findings. This resulted in three full-length capsule endoscopy videos with annotations that we used for cross-validation of our deep learning approach.

The goal of this work is to reduce the physician time needed to read the VCE videos; thus, our framework focused on removing confidently predicted normal frames which are not needed for diagnoses. Before training a model, we had to pre-process the videos by separating each into frames using the five frames per second speed of the MP4 files. All frames contain a time code in the top corner as seen in Figure 1, which are what the physician uses to denote the location of any abnormalities. The ground truth for this data is a list of segments with a start and ending time of when the abnormality is visible in the video. To provide the label for each image, we used optical character recognition to read the time code and determine if the frame was indicated as abnormal by the physician. The final step before training the deep learner was to remove the time code and IntroMedic text from the images as seen in Figure 2, to prevent the model from seeing this information and using it to make predictions. We decided to use a well-known and robust convolutional neural network architecture, ResNet50 [32], which has been trained to achieve high performance on various classification benchmark datasets. ResNet50, with initial weights from training on ImageNet [33], was fine-tuned with the KID videos dataset. The classification layer of ResNet was replaced by a single fully connected layer with two neurons as seen in Figure 3. This output layer used a sigmoid activation function to classify the frames as either normal or abnormal. All models were trained using stochastic gradient descent as the optimizer using a learning rate of 0.001 with class weights to account for the abnormal frames being a minority.

Typically, a network would have all its starting weights generated randomly prior to training, and learn optimal weights based on the training data. However, we had a very small dataset in this study, so we decided to use the final weights learned on the ImageNet dataset as the starting point [33]. As we only had three annotated videos to work with, it was decided that experiments would be conducted using a three-fold cross validation scheme. Thus, we would train a model with two videos, testing on the third, and repeat for all combinations. With our architecture and pre-processing pipeline chosen, our first experiment was to train a model using the videos and make predictions with that model. We found that because our dataset was small, this single model approach did not perform well. With small datasets, it is useful to use multiple networks in an ensemble approach to increase performance. Traditional ensembles require training multiple models that are used for voting on final predictions, which can be very time consuming. However, more recently, it has been shown that saving intermediate model’s weights while training can save time and still produce the desired effect of an ensemble [34].

While training the single model, we found that the training accuracy began to plateau around 99% after ten epochs. If training continues past this point, the model begins to over-fit to the training data and not generalize well on the test data. Thus, we decided to train for only ten epochs, saving the model weights after each epoch to use for voting. We then conducted a second experiment using the weights of nine of these intermediate models to vote on the class of each frame in the video. We found an increase in performance over the single model; however, overall, there were still many missing abnormal segments. KID videos 1 and 3 had significantly less abnormal frames than KID video 2, an imbalance which we identified as a potential problem.

Our solution was to augment the abnormal frames in these two videos using rotation. KID video 2 has 5050 abnormal frames; therefore, for KID video 1, we rotated each of the abnormal frames by 30 degrees resulting in 12 rotations, and in KID video 3, we rotated each of the abnormal frames by 72 degrees resulting in 5 rotations. These rotations increased the number of abnormal frames from 429 and 988 to 5148 and 4940 for videos 1 and 3 respectively. In our final experiment, the rotated abnormal images were only used in training the model, not in testing. An example of this rotation can be seen in Figure 4. This augmentation, combined with the use of intermediate model weights to vote on predictions, provided us with our best results.

3. Results

The ground truth for the videos is a list of time segments where an abnormality was observed and annotated by an expert. An abnormality may be visible for as short as a few seconds to as long as several minutes. We considered an abnormal segment detected if at least one frame from the segment is predicted as abnormal. In our first experiment, we trained the deep learning model detailed in Figure 3, for ten epochs, using the final epoch to make predictions on the test videos. The results for the three videos are shown in

Table 1. Single Network Trained on Non-Augmented Videos.

Video Name	Total Frames	Number of Abnormal Segments	Reduced Number of Frames	Abnormal Segments Detected
KID Video 1	28,480 (95 min)	22	616 (2 min)	3
KID Video 2	117,565 (391 min)	86	956 (3 min)	10
KID Video 3	74,762 (249 min)	11	4522 (15 min)	8

. This single network was able to identify 21 of the 119 abnormalities, while reducing the number of frames on average by 97%.

Though this method produces a sizeable reduction in video length, the number of abnormalities detected is less than 18%. In real world applications, there will always be a physician to look over the predictions; thus, normal frames being incorrectly classified as abnormal is acceptable. However, incorrectly classifying abnormal frames as normal could have dire consequences for patient outcomes. Therefore, our goal is to limit the number of missed abnormal segments. Since the single model did not perform well, we decided to use an ensemble approach to increase performance. As discussed above, we decided to save intermediate models while training to have them vote on predictions. In our second experiment, we again trained the model described in Figure 3 for ten epochs; however, we saved each of the ten intermediate models. We then had the first nine intermediate models make a prediction for each frame in the videos. Since the most important aspect is not to miss an abnormality, we decided on a voting scheme where all of the nine networks had to agree on a normal frame for it to be given that label. A paired t-test with a confidence interval of 95% showed a statistically significant improvement in the detection of abnormal segments with this ensemble in the three-fold cross validation over a single model as seen in

Table 2. Paired t-test Results Comparing Accuracy of Single Network vs. Ensemble with No Augmentation.

Model	n	Mean	Variance	t-calc	t-crit	df	p
Single Network	119	0.176	0.147	11.72	1.98	118	$1.66\times {10}^{-21}$
Ensemble No Aug	119	0.714	0.206

below.

As shown in

Table 3. Ensemble of Networks Trained on Non-Augmented Videos.

Video Name	Total Frames	Number of Abnormal Segments	Reduced Number of Frames	Abnormal Segments Detected
KID Video 1	28,480 (95 min)	22	10,747 (36 min)	21
KID Video 2	117,565 (391 min)	86	10,960 (37 min)	53
KID Video 3	74,762 (249 min)	11	33,963 (113 min)	11

, the ensemble method was able to identify 85 of the 119 abnormal segments, while reducing the number of frames by 75%. The voting was repeated using the last nine models, and it was found to perform marginally worse with only 80 of the 119 abnormal segments detected, and a reduction in number of frames by 76%.

This method performed very well on videos 1 and 3 but missed a large number of abnormal segments on video 2. We hypothesized this was due to the number of abnormal examples in the training set. Video 1 has 22 abnormal segments with a total of 429 abnormal frames, video 2 has 86 abnormal segments with a total of 5050 abnormal frames, and video 3 has 11 abnormal segments with a total of 988 abnormal frames. This means when testing on video 2 and training with videos 1 and 3, we have a significantly lower number of abnormal frames for training than the other two folds. Thus, to compensate for the large variance in abnormal frames per video, we decided to use augmentation to increase the number of frames in videos 1 and 3 as described previously. We then re-trained the networks using the videos with the augmented abnormal frames, again saving the ten intermediate models. Using the first nine models, the same voting procedure was used such that all networks needed to agree a frame was normal for it to be given that label. The results of this experiment with augmentation on the abnormal frames can be seen in

Table 4. Ensemble of Networks Trained on Augmented Videos.

Video Name	Total Frames	Number of Abnormal Segments	Reduced Number of Frames	Abnormal Segments Detected
KID Video 1	28,480 (95 min)	22	14,828 (49 min)	22
KID Video 2	117,565 (391 min)	86	84,672 (282 min)	86
KID Video 3	74,762 (249 min)	11	27,099 (90 min)	11

below.

The average reduction in number of frames is 47%, while the number of abnormal segments detected is 119 of 119. A paired t-test with a confidence interval of 95% showed that this ensemble method with augmented data had a statistically significant improvement in the detection of abnormal segments in the three-fold cross validation over the ensemble without augmentation as shown in

Table 5. Paired t-test Results Comparing Accuracy of Ensemble without Augmentation vs. Ensemble with Augmentation.

Model	n	Mean	Variance	t-calc	t-crit	df	p
Ensemble No Aug	119	0.714	0.206	6.87	1.98	118	$3.21\times {10}^{-10}$
Ensemble With Aug	119	1	0

. The same voting procedure was repeated using the last nine models. It was slightly worse in detecting 118 of 119 abnormalities while reducing the videos on average by the same 47% as voting the first nine models. Despite working with a small dataset of only two videos for training in each fold, we were able to develop an algorithm that successfully detected 100% of abnormal segments while reducing the reading time for a physician by 47% on average.

4. Discussion

The goal of our study was to reduce the physician time needed to read VCE studies by excluding frames classified as normal without sacrificing diagnostic accuracy. Our study used a robust convolutional neural network (CNN) that was trained and tested to classify a frame as normal or abnormal, with the goal of minimizing the number of false negatives. As such, using our CNN model, the physician can confidently exclude normal frames and carefully analyze abnormal frames to ensure accurate diagnosis. Using the method proposed in our study, we were able to reduce the video length on average by 47%. By doing this, our model was able to remove frames confidently classified as normal while accurately classifying frames within abnormal segments. Using this model, we captured frames from all 119 of the abnormal segments, for an accuracy of 100%. We would note that KID video 2 is of lesser quality than the two other videos in the database. Our approach has the promise of significantly reducing the reading time needed by a physician to review VCE images. This study contributes to the growing field of CNN in the detection of subtle lesions in the small bowel by demonstrating the ability to perform well with small training data. This is important due to the cost associated with collecting and annotating large amounts of medical data.

Similar studies have efficiently detected pathology in VCE in shorter time periods. As described in [35], the authors attempted to use AI to improve the performance of VCE using machine learning called Computer-Assisted Diagnosis for CAPsule Endoscopy (CAD-CAP). CAD-CAP had a sensitivity of 100% in the detection of angiectasias; however, this was done with clean and still images. Our study had the benefit of having the physician review all the abnormal frames, thus ensuring an accurate diagnosis. As described in [36], a custom CNN was trained and evaluated using a total of 5825 WCE videos and was able to detect 95.9% of abnormal findings. In [37], capsule endoscopy data was used to train a CNN to find protruding lesions in the small bowel. Using 30,000 images from 292 subjects, they obtained 0.91 AUC with a sensitivity of 90.7%. In [38], the authors obtained a sensitivity of 88.2% using a CNN trained on 5360 images when looking for erosions and ulcerations in the small bowel. In [39], the authors were able to train a CNN to classify abnormal pathology from normal pathology with sensitivities and specificity as high as 99.88% and 99.09% compared with physician reads that had sensitivities and specificity of 74.57% and 76.89%. Their method reduces reading times on average from 96.6 min to 5.9 min. However, their study involved the use of around 5000 videos collected and annotated by 21 physicians in China with unknown specialties, experience, and skill.

Our study has some limitations which offer opportunity for improvement. Our CNN model was trained only on two videos and tested on one, which limits the scale of our study and the results achieved. We hope to improve this by increasing the data available in training our CNN model, with the goal of establishing an algorithm that can delineate between normal and abnormal frames instantly. Our goal is to expand our data and to continue to train our CNN model to detect abnormal frameworks, ultimately reducing physician reading time.

Artificial intelligence (AI) is an emerging field in gastroenterology, particularly in VCE. Our study focused on training a CNN-based model to identify subtle inflammatory and vascular lesions, with the goal of developing an algorithm that can quickly differentiate between normal and abnormal frames. Our model demonstrated the ability to find all abnormal regions with significant reduction in physician reading time. Our study results demonstrate the benefit of using a CNN in processing VCE images. We believe our study lays an excellent foundation for further validation in large multi-center trials.