Fully Automated Electronic Cleansing Using CycleGAN in Computed Tomography Colonography

Abstract

In computed tomography colonography (CTC), an electric cleansing technique is used to mix barium with residual fluid, and colon residue is removed by image processing. However, a nonhomogenous mixture of barium and residue may not be properly removed. We developed an electronic cleansing method using CycleGAN, a deep learning technique, to assist diagnosis in CTC. In this method, an original computed tomography (CT) image taken during a CTC examination and a manually cleansed image in which the barium area was manually removed from the original CT image were prepared and converted to an image in which the barium was removed from the original CT image using CycleGAN. In the experiment, the electric cleansing images obtained using the conventional method were compared with those obtained using the proposed method. The average barium cleansing rates obtained by the conventional and proposed methods were 72.3% and 96.3%, respectively. A visual evaluation of the images showed that it was possible to remove only barium without removing the intestinal tract. Furthermore, the extraction of colorectal polyps and early stage cancerous lesions in the colon was performed as in the conventional method. These results indicate that the proposed method using CycleGAN may be useful for accurately visualizing the colon without barium.

1. Introduction

Colorectal cancer was the third most commonly diagnosed cancer worldwide in 2020, with an estimated 20 million new cases. Approximately 1 million people die annually from colorectal cancer [1,2]. As colorectal cancer has a good prognosis if detected and treated at an early stage, early detection and treatment are important to reduce the mortality rate. However, screening and biopsy checkup rates for colorectal cancer are low, and improving the screening checkup rate is a major challenge [3]. The main reason for the avoidance of colonoscopy is the taking of a large amount of laxatives as pretreatment before the examination, which are intestinal cleansing agents and are painful [4,5].

Computed tomography colonography (CTC), with its simple bowel pretreatment, is attracting attention as a new diagnostic method for colorectal cancer [6]. This examination uses a computed tomography (CT) scan to observe the interior colon, and fecal tagging has been devised as a method of labeling of the residue with an oral contrast agent to distinguish the intestinal tract from the intestinal residue [7,8,9]. Barium sulfate (barium) is the most commonly used oral contrast agent for fecal tagging [10]. Fecal tagging has facilitated discrimination between intestinal residues and lesions by labeling highly absorbable intestinal residues. Therefore, CTC with fecal tagging does not require the complete cleaning of the intestinal residues. Moreover, the accuracy of the detection of colorectal neoplastic lesions is ensured even if the amount of intestinal cleansing agents used in a conventional colonoscopy is reduced [5]. In addition, an electronic cleansing (cleansing) process that removes the residue labeled with barium is now available [8]. This technique removes the fecal tagging from the original CT images obtained via CTC using an algorithm that tracks the interior colon and removes regions of high CT values. This is now available on three-dimensional (3D) imaging workstations. In addition, a material discrimination method [11] and methods using deep learning [12] have been proposed for the cleansing process, and evaluations using phantoms and other methods have been conducted.

One problem with the cleansing process is that in cases of bad fecal tagging, the barium area is not correctly removed [13,14]. Bad fecal tagging signifies that the residue, residual liquid, and barium are nonuniformly mixed. In addition, a double layer phenomenon often occurs when only barium sinks, and unmixed intestinal fluid layer is formed on the surface. If the residue is badly tagged, the cleansing process becomes difficult and barium may remain in the same form as the lesion. In such cases, it may be difficult to distinguish barium from the lesion, and analysis and reading of the extracted lesion may take a long time. If the cleansing process is not available, it is necessary to disable this function for diagnosis, which complicates the reading of the lesion. To solve this problem, we focused on a domain transformation technique called a generative adversarial network (GAN) [15]. GANs were proposed by Goodfellow et al. in 2014 and are models that deepen learning by having two neural networks: one that generates images and another that determines whether the images are close to the real ones or compete with each other. By training features from the data, nonexistent data can be generated and transformed according to the features of the existing data. Various applications of GAN have been reported in the medical field [16,17,18]. Onishi et al. generated a large number of CT images of pulmonary nodules using GANs and used them as training data to discriminate between benign and malignant nodules [16]. Teramoto et al. generated high-resolution lung cell images using the progressive growth of GANs and improved the accuracy of differentiation between benign and malignant lung cancers [17]. Toda et al. generated CT images of lung nodules using Info GAN, which allows the control of image characteristics, and applied them to the histological classification of lung cancer [18]. In this study, we focused on CycleGAN [19], which is a derivative of GAN. CycleGAN is a method for realizing image transformation by learning the relationship between the fields or regions of two image data. CycleGAN can learn the relationship between two different images and obtain an image transformation algorithm without preparing a large number of paired images. Several applications of CycleGAN have been reported in the medical field. Hiasa et al. proposed a method to convert magnetic resonance head images into CT images and obtained good conversion results [20]. Zhou et al. proposed a method to remove noise from fluorodeoxyglucose positron emission tomography images and obtained good results in improving the positron emission tomography image quality at low doses [21]. However, no studies have been conducted to improve the above-mentioned problems of poor cleansing in clinical CTC examinations using CycleGAN [22]. If CycleGAN resolves the problems of double layer and uneven barium in CTC cleansing, the accuracy of CTC can be improved. Therefore, this study aimed to develop a cleansing process for CTC original images using CycleGAN, and its effectiveness was confirmed using actual clinical images.

2. Methods

2.1. Target Case

We collected images of 125 cases of CTC taken at the Yokkaichi Hazu Medical Center of the Japan Community Health Care Organization. Figure 1 shows the original CTC images collected in this study. One hundred cases were used for CycleGAN training, and the remaining 25 cases were used for evaluation. The data acquisition period for the 100 cases was from 20 April 2021 to 13 September 2021, and the data acquisition period for the 25 cases for validation was from 20 November 2021 to 15 January 2022.

As a pretreatment for CTC examination, the patient drank a test meal, barium, and 400 mL of water after each meal in the morning before the examination and drank a laxative and glass of water before bedtime. Subsequently, eating and drinking were prohibited until the day of the examination. All examinations were performed using Aquilion ONE (Canon Medical Systems) with FC05 as the reconstruction function and AIDR 3D as the reconstruction algorithm. The original CTC images were taken in the transverse section, covering the area from the subdiaphragm to the rectum. The number of slices varied with each case but was approximately 150. The slice interval was 3 mm, and the slice thickness was interpolated to 3 mm. Consequently, 11,161 images for training and 3802 images for evaluation were obtained.

2.2. Pretreatment Conditions

As the colon is a very long organ, the water content of residual fecal fluid differs between the rectum and cecum. Therefore, the condition of the pretreatment residuals also varies. The state of pretreatment barium greatly varies with each person depending on the state of the bowel of the patient [23]. The cleansing process depends on the barium status of the CT, and it is necessary to understand its current status. The condition of barium in the collected original CTC images can be classified into the following categories: homogeneous, uneven and heterogeneous, heterogeneous with a sticking to the intestinal wall, barium and residue separated into double layers, no barium in the residue, and no residue. The barium status of the original CTC images of the 25 cases was defined and evaluated, as shown in Figure 2. The assessment was conducted by a radiological technologist with 20 years of clinical experience and certified by X-ray CT technologists or radiological technologists. Figure 3 shows the distribution of barium in the rectum, sigmoid, descending, transverse and ascending colons, and cecum. The overall number of good cases was calculated by subtracting the number of bad cases from the number of good cases. The total number of good and bad cases was 14 and 11, respectively.

2.3. Manual Cleansing (MC) Image Creation

An overview of the creation of a manually cleaned image is shown in Figure 4. The first step was to manually extract only the barium in the collected original CTC images in 3D while displaying the 3D image of the barium area and confirming that calcification of blood vessels and stones were not selected. The 3D image obtained by subtracting the manually extracted barium area image from the original CTC image was converted to a two-dimensional axial image, which was used as an MC image for CycleGAN training. The VR function of Ziostation2 (Ziosoft, Tokyo, Japan) was used for manual selection of barium regions. MC images were created as training data for this method by performing the above process on the 100 cases collected. In this process, the MC images for training were not created for images with serious artifacts. Consequently, 11,161 original CTC images and 10,944 MC images were used in the training data.

2.4. Network Structures

In this study, CycleGAN was employed to generate cleansed CTC images. An outline of the CycleGAN architecture used in this study is shown in Figure 5. The datasets fed to it are the original CTC and MC images that were manually cleaned with barium. Generator G is trained to generate images that fool the discriminator from the original input CTC images, and discriminator Dy is trained to discriminate between real virtual MC and nonreal MC images. By alternately training the generator and discriminator, the generator can generate images that resemble real original CTC images. Simultaneously, generator F transforms the MC image into a CTC original image with a barium residue, and discriminator Dx identifies it from the real CTC original image. In addition, CycleGAN transforms one set of images to another set of images, restores them to the original set of images, compares the differences, and learns. Figure 6 shows the network structure of CycleGAN used in this study. ResNet [20] with nine residual blocks was used as the generator of CycleGAN, and the mechanism of PatchGAN [24] was employed as the discriminator.

To train CycleGAN, we set the number of training cycles to 100 and the batch size to 1. Adam was used as the optimization algorithm. The learning rate was set to 0.0002, and the loss function was set to the mean absolute error. Regarding the batch size, the evaluation was conducted while changing the parameters to 1, 4, and 16 during the preliminary experiments. Consequently, a batch size of 1 was adopted, which produced the best output images.

This study was conducted using an in-house Python program with the Keras and TensorFlow software libraries. Three types of images were used for evaluation: original CTC images, conventional cleansing images (hereafter referred to as “conventional method”), and images created using the proposed cleansing method (hereafter referred to as “proposed method”). The conventional cleansing method has no parameters for the cleansing process, and all processing was automatically conducted.

2.5. Evaluation Metrics

To verify the effectiveness of the proposed method, we evaluated the barium volume and cleansing rate (CR). For verification, only the colon region was extracted from each image, and the volume was calculated from the barium pixel values. The volume of barium after the cleansing process was defined as V_DC, the volume of barium in the original CTC image was defined as V_CTC, and the barium removal rate (CR) was defined as per the following Equation (1). The CR was calculated using the conventional method and the proposed method and compared.

CR = (1 − V_DC/V_CTC) × 100 [%]

(1)

It is necessary to extract a barium-only image to obtain the barium volume of the V_CTC. Extraction of barium from the original CTC image was performed using the barium-only image created, as described in Section 2.3. For the extraction of barium only from V_DC, subtraction of MC images, in which only the colon was removed from the cleansing images of the conventional method, was performed, and the barium area of the colon was extracted using the conventional and the proposed methods. To calculate the volume, the barium-eliminated image obtained via the conventional method was binarized, and the number of pixels in the barium region obtained was counted. The volume was calculated similarly to that for the cleansing image of the proposed method.

3. Results

3.1. Cleansing Images Generated Using CycleGAN

The cleansed images generated using the proposed method are shown in Figure 7. Although barium was removed up to the intestinal wall using the proposed method, no cases of excessive removal from the intestinal tract were observed.

Figure 8 shows the results of bilayer separation caused by the poor pretreatment of CTC. Barium was removed even when the barium and intestinal fluid were in double layers, and the concentration of barium was not uniform. In the conventional method, a large amount of barium remained, whereas, in the proposed method, barium was removed, even at the edge of the intestinal wall.

Figure 9 shows the results of applying the proposed method to images of the lesions. Both the conventional and proposed methods extracted rectal polyps of 13 mm and 9 mm in diameter, early stage cancer of the rectum, and perirectal lymph nodes without any loss of morphology due to processing. As shown in Figure 9, aortic calcification and renal calculus, which have high CT values similar to barium, were not removed from the images of the colorectal lesions. In addition, mass lesions of the liver and fatty liver lesions with low CT values were also depicted.

3.2. Results of Quantitative Evaluation

The total volume of barium in the 25 cases is shown in Figure 10a. The average volumes of the barium in the original, conventional, and proposed methods were 36,428.08 cm³, 9649.76 cm³, and 1359.88 cm³, respectively. Figure 10b shows the CRs. The average CRs of the conventional and proposed methods were 72.3% and 96.3%, respectively. Figure 11a,b show the barium CRs calculated for 14 and 11 cases of good and bad pretreatments, respectively. In the group of cases with good pretreatment, the barium CRs of the conventional and proposed methods were 79.0% and 96.4%, respectively. In the group of patients with poor pretreatment, the average barium CRs using the conventional and proposed methods were 63.8% and 96.1%, respectively.

4. Discussion

4.1. Images Generated by CycleGAN

The CycleGAN-generated image (Figure 7) of the proposed method shows that the cleansing process did not damage the intestinal tract, and only barium was removed. In the conventional method, there were cases in which barium remained in the image, such as false–positive lesions, or barium remained in the image without being removed at all. If barium is retained, it is necessary to confirm whether it is barium or a lesion from the original image to avoid pointing out a false–positive lesion, which adds time and burden to the preparation of the CTC imaging reports. The proposed method can remove more barium than the conventional method, and lesions remain unremoved, making it easier to detect.

To confirm that an excess area was not removed when the barium was removed, an outline of the intestinal tract wall surface was drawn and overlapped with the original CTC image and the CycleGAN-generated image, as shown in Figure 12. In cases with good barium pretreatment (Figure 12a), poor pretreatment (Figure 12b), and lesions (Figure 12c), the intestinal wall, and lesions were not removed, and only barium was selectively removed, indicating that this technique correctly recognized and removed barium.

However, even with the images generated using the proposed method, barium was not removed in some cases where there was no air space in the intestinal tract and the colon was completely filled with barium. This may be due to the fact that few of the original CTC images used in the study were filled with barium. CTC is also used to identify extraintestinal lesions as they are included in the range of imaging from the upper to lower abdomen. The Extracolonic Findings Reporting and Data System was used to determine extraintestinal organs. It has been reported that tumor lesions in abdominal organs other than the colon and lesions, such as aortic aneurysms, can be detected [25,26]. Therefore, lesions and organs outside the intestinal tract must be depicted in CycleGAN-generated images without defects. As shown in Figure 9, the proposed method showed no degradation of the lesions outside of the intestinal tract, indicating that it can be used to diagnose these lesions.

The CycleGAN introduced in this study consists of a network that performs two types of transformations and converts MC images into pseudo-CTC images. The observation of the results obtained from converting MC images to CTC images using CycleGAN showed that many good pretreatment patterns with uniform barium were generated, and barium images with nonuniform CT values that were poorly pretreated were not reproduced. In this situation, CycleGAN training showed a greater loss in the images of patients with poor barium generation. However, the results of the image quality evaluation and barium removal rate were acceptable, and we believe that this is not a problem for the purposes of this study.

4.2. Quantitative Evaluation Considerations

A comparison of the CRs of good and bad pretreatment with the conventional method showed a 15.2% difference, indicating that the conventional method is affected by the pretreatment conditions. An evaluation of the CR of barium in cases of poor pretreatment between the conventional and proposed methods showed that the proposed method was 32.3% higher than the conventional method. The CR of barium using the proposed method was not affected by the state of pretreatment, such as the separation of the double layers. The CR of unnecessary barium was higher than that of the conventional method, and the detection of colorectal lesions was expected to be enhanced using CycleGAN-generated images for image extraction.

4.3. Possible Interpretation

Previous studies [11,12] were performed on simulated stools and did not use barium, which is used in clinical practice; therefore, they were not compared with realistic assessments of intestinal conditions. Various intestinal conditions are expected to affect clinical CTCs. The proposed method is effective in clinical practice because it can remove barium under any pretreatment condition and extract the lesions.

It was also shown that the proposed method can accurately perform the cleansing process, even in cases with poor pretreatment. In clinical practice, patients are photographed in two positions, supine and prone, to consider the possibility of poor pretreatment.

We believe that the proposed method can achieve a good cleansing process. Therefore, it is possible to complete the examination using only one-way imaging. This reduces the radiation dose from the CT examination. In addition, the body movements of the patient, such as lying on a narrow CT table, will be reduced, and the burden of the patient will also be lessened.

In this study, CycleGAN was used for barium cleansing. This method has the advantage that an unpaired image can be used for training. In some cases, MC could not be performed because of artifacts in the image data. Therefore, in these cases, MC images were not created, and training was performed under unpaired conditions, where the numbers of original CTC images and MC images were different. The ability to train using unpaired images is useful for data collection.

Only a limited number of radiologists can read CTCs, and specific training is required. Furthermore, experienced physicians need to reconfirm the original CTC images by observing areas that are poorly cleansed from multiple directions. However, these are complicated tasks. To reduce the burden on physicians and improve accuracy, we believe that it is important to reduce the number of cleansing defect areas and facilitate the confirmation process [27].

This study had some limitations; we evaluated only 25 cases in the pretreatment state. To study images of various variations in normal and abnormal cases of the colon, it is necessary to increase the number of cases for further validation in the future. In clinical practice, diagnosis is performed by displaying virtual endoscopic images (virtual endoscopy) and virtual colon development images (virtual gross pathology) using images after imaging and cleansing. In the future, it is necessary to generate and evaluate these images.

5. Conclusions

In this study, we developed a cleansing process for the original CTC images using CycleGAN. The results of evaluating the cleansing images with the proposed and conventional methods show that the proposed method using CycleGAN can eliminate barium more accurately than the conventional method. These results indicate that the proposed method is effective for cleansing CTC.