Using the STEGO Neural Network for Scintigraphic Image Analysis ^†

Abstract

Currently, neural networks are being widely implemented for the diagnosis of various diseases, including cancer of various localizations and stages. The vast majority of such solutions use supervised or unsupervised convolutional neural networks, which require a great deal of training data. Using unsupervised image segmentation algorithms can be considered the preferred trend since their use significantly reduces the complexity of neural network training. So, developing unsupervised image segmentation algorithms is one of the topical tasks of machine learning. This year, a team of developers from Google, MIT, and Cornell University developed the STEGO algorithm, which is an unsupervised and non-convolutional neural network. As its author stated, the STEGO algorithm performs well at image segmentation problems compared with other machine learning models. And this algorithm does not need a large amount of training data, unlike convolutional neural networks, which are widely used for medical image analysis. So, the aim of this work is to check the possibility of using this neural network for scintigraphy image segmentation by testing whether the STEGO algorithm is relevant when applied to a scintigraphy dataset. To achieve this goal, the intersection over union metric (IoU) was chosen for evaluating the correctness of the detection of the location of metastases. The training dataset consists of scintigraphic images of patients with various types of cancer and various metastasis appearances. Another version of this metric (mIoU, mean intersection over union) was also used by the creators of STEGO to assess the quality of the model to segment images with different kinds of content. Since the calculated metrics are not good enough, the use of this algorithm for scintigraphic image analysis is not possible or requires the development of a special methodology for this.

1. Introduction

1.1. Artificial Intelligence and It’s Applications

Artificial intelligence (AI) in the world is one of the most promising information technology industries, and there are many examples of improving the living conditions of people through the implementation of machine learning algorithms into their everyday activities [1]. However, despite the already quite widespread use of AI in the economy and other human routines, there are areas of human life where the introduction of neural networks is quite specific, and medicine is among them. Using methods for processing data about a person by implementing a neural network to make a diagnosis, recommend medicines, and carry out a course of treatment entails a high level of responsibility for the lives of people to whom these methods will be applied. However, artificial intelligence in medicine is the use of machine learning models to search medical data and uncover insights to help improve health outcomes and patient experiences. Thanks to recent advances in computer science and informatics, artificial intelligence (AI) is quickly becoming an integral part of modern healthcare. AI algorithms and other applications powered by AI are being used to support medical professionals in clinical settings and in ongoing research.

1.2. Scintigraphy

Scintigraphy is a modern method of radionuclide visualization based on the registration of irradiation generated by the radioactive substance inside a patient. Scintigraphy is used for the evaluation of the functioning of various organs and tissues. Diagnostic methods such as X-ray, ultrasound, CT, or MRI are focused on detecting structural changes in body tissues, and are not always able to distinguish the disease in its early stages, when deviations manifest themselves at the level of biochemical changes in tissues. Radionuclide research methods are indicated for diseases of the heart, the brain, the kidneys, and the liver and look indispensable in identifying and assessing the prevalence of oncological processes. Currently, scintigraphy is widely used in the USA, Europe, and a number of other countries. So more than 17 million radionuclide studies on more than 15 million people were conducted in 2007 in the USA. In Europe, in the same year, over 12 million studies were conducted [2].

1.3. Machine Learning Usage in Scintigraphy

It was worth expecting the appearance of attempts to apply deep learning methods to radionuclide diagnostics. Recent research has been completed on these areas that has resulted in an AI model that demonstrates high-quality diagnostics of cancer bone metastasis [3,4]. The area under the curve (AUC) of receiver operating characteristic (ROC) was 0.988 for breast cancer, 0.955 for prostate cancer, 0.957 for lung cancer, and 0.971 for other cancers [3]. This model, after retraining, demonstrated performance comparable to that of physicians individually classifying bone metastasis. Further AI-consulted interpretation also improved diagnostic sensitivity and accuracy. In total, this AI model performed a valuable benefit for nuclear medicine physicians in the timely and accurate evaluation of cancer bone metastasis [3]. In the work of Papandrianos A. et al. [5], several convolutional neural network algorithms, such as esNet50, VGG16, MobileNet, and DenseNet, were compared with each other in terms of efficiency for bone metastasis diagnosis by means of the analysis of scintigraphy images in RGB and gray scales. As has been shown, some of these neural networks can achieve high accuracy, up to 92.50%, with the specific values of the models. However, convolutional neural networks have great disadvantages. For example, convolutional neural networks need a lot of training data and can’t recognize the orientation and position of objects [6]. Therefore, the authors [5] could not create any classification model suitable for practical use since they had only 408 images. A brief overview of some image processing algorithms regarding the task of scintigraphic image analysis were discussed in ref. [7]. In this work [7], the authors proposed a new machine learning approach in which the problem is rephrased to the multi-label learning problem and each bone will be labeled with a non-empty subset of all possible labels. The articles above do not exhaust the methods and algorithms of deep learning for scintigraphic images [8,9]. Thus, the problem of developing deep learning algorithms for the analysis of scintigraphic images is a task of current interest. Of special interest is the use of completely new architectures of neural networks and deep learning algorithms not originally intended for the analysis of medical images such as scintigraphy. One of such neural network architectures is the self-supervised transformer with energy-based graph optimization (STEGO) framework, which was published recently in [10]. This framework distils unsupervised features into high-quality, discrete semantic labels.

The purpose of this work is to evaluate the applicability of STEGO to the scintigraphic image segmentation problem and to compare it to other machine and deep learning methods that can be used for scintigraphic image analysis.

2. Dataset

Scintigraphy images were obtained during clinical research that was carried out at Saratov State Medical University named after V. I. Razumovsky. The inclusion criteria were the presence of kidney or prostate cancer with metabone metastases. The data for this research was prepared by the scientific department of Saratov State Medical University and contains information on 57 patients. The dataset includes data of 54 men and 3 women, patients ages are normally distributed (Figure 1). Patient ages normally distributed except two outliers (Figure 1). 17 patients have prostate cancer (about 30%), and patients with another type or combination of types of cancer account for 40 (70%).

Patient data was in DICOM format. It is a special medical data format that may contain a set of images of human bones and metastases on them for each patient on both the front and dorsal sides. The images can be represented by different color palettes without decreasing in quality due to DICOM format specifics (Figure 2). The dataset contains three different types of images (like in Figure 2) for the front and dorsal views of the body for each person in scope. This is raw scintigraphy images.

Since patient data contains images of the front and dorsal sides of the body, we can analyze the spread of metastases more precisely because metastasis can look varied on different points of view.

Metastasis means that cancer spreads to a different body part from where it started. Metastases most commonly develop when cancer cells break away from the main tumor and enter the bloodstream or lymphatic system. These systems carry fluids around the body. This means that the cancer cells can travel far from the original tumor and form new tumors when they settle and grow in a different part of the body. The main task of scintigraphy is to show these affected body parts by inserting the radioactive substance inside a patient.

The principle of color indication on scintigraphic images is as follows (Figure 2). For example, the color gamma of the left image in Figure 2 is blue/yellow, where the intensity of the yellow color indicates the size and quantity of bone metastases. That is, an area containing many metastases will be more intensely yellow than an area containing fewer metastases. The less metastasis an area contains, the bluer it looks. For the other two images in Figure 2, the principle of colour indication is the same, and yellow colour also shows the quantity of metastases.

3. Implementation of STEGO for Scintigraphic Image Analysis

The STEGO algorithm (hereinafter-STEGO) first time described in [10] by scientists from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), Microsoft, and Cornell University. As it was stated, STEGO can jointly discover and segment objects without any human labels at all, down to the pixel. The input to the STEGO is an image. The STEGO retrieves global image information by pooling spatial data. The neural network is made up of a frozen backbone that serves as a source of learning feedback and as an input to the segmentation head for predicting distilled features. This segmentation head is a simple feed forward network with ReLU activations. Thus, STEGO looks quite promising for medical image analysis.

For example, when it is used in an ensemble as an image preprocessing neural network, the output of the work can later be presented as an input to a classifier neural network that will predict the level and area of the lesion, diagnosis based on this area, and so on. The creation of ensembles for solving the problem of image segmentation of medical data has previously been discussed in various papers, such as ref. [11]. In this work, Robert Gordon University’s members considered the approach of creating a two-layer ensemble, which was validated using the Hausdorff and Dice metric. The algorithm proposed in ref. [11] showed the best results and has a 0.892 value of Dice Measure and a 48.831 value of Hausdorff measure out of a few other two-layer ensembles that contain quite popular neural networks such as VGG-16, ResNet-34, UNet, and others. Training of this network was carried out on Kvasir-SEG dataset (which consists of 1000 gastrointestinal polyp images) and the CAMUS dataset (the Cardiac Acquisitions for Multi-structure Ultrasound Segmentation).

In this work, raw scintigraphic images were taken as input data for STEGO. As a result, good low-resolution images 246 × 246 px were obtained, which almost ideally divide the input pre-scaled scintigraphic image into segments.

An example of STEGO output can be seen in Figure 3 and Figure 4. Both Figure 3 and Figure 4 show three images. The first image is the input image of STEGO, the second one is processed image by cluster prediction, and the last one is the image after linear probe prediction. As can be seen in Figure 3 and Figure 4, the resulting images of cluster predictions look almost uniformly colored, so the cluster prediction algorithm is not good enough for the scintigraphic images’ analysis. The resulting images of linear probe predictions look more promising for further analysis since areas with and without metastases are well separated there.

The patient in Figure 3a has metastases on the ribs since we see yellow colored areas on the initial image (the left image), and the body areas without metastases are colored in various shades of blue on the initial image. In the output image of the STEGO algorithm (linear probe predictions), the yellow blurry areas are replaced by red areas with clear boundaries. The same situation exists for the image of the patient with metastasis on the spine (Figure 3b). Thus, the areas of metastasis are clearly identified, and the resulting images of the linear probe prediction algorithm can be used further for building high-level models, for example, classification models.

It should be noted that the result of the STEGO of processing scintigraphic images is sensitive to the color scheme of these images. So the wrong segments can be highlighted (Figure 3 and Figure 4).

In Figure 4, the input image shows metastases located on the spine, but linear probe predictions of the STEGO algorithm split the image into two segments, green and red, and we can clearly see a large area with metastasis (left red area in the right image) between the arm and thigh of the patient, which is an obvious mistake.

If the color gamma changes from blue/yellow to rainbow colors, the performance of dividing into segments by STEGO can get worse (Figure 5). Thus, not all possible color palettes allow for high-quality of image segmentations.

4. Quality Metrics

To check the quality of STEGO performance on a scintigraphic image dataset, an IoU (intersection over union) was chosen as a standard metric for the problem of image segmentation. The IoU is the area of overlap between the predicted segmentation and the ground truth divided by the area of union between the predicted segmentation and the ground truth. To apply this metric, it is necessary to create a mask for each patient, which will determine the true position of metastases on the body. Since the size of the output image from STEGO is 246 × 246 px, after applying the algorithm, it is needed to increase the image size to the mask size (690 × 940 px) for the correct calculation of the metric.

Due to the fact that linear probe prediction in STEGO makes segments of images by coloring them into different color palettes, the metric is calculated as follows. STEGO output setting to grayscale, then metric applied over the mask and over every gray shade, thereby giving the list of results (intersection over union for the mask and each gray shade). Among the obtained results, the one that has the highest value determines the value of this metric relative to this particular image. The metric is applied like this to all images, and thus forming a list of metric values for each of the images in the dataset. The mIoU can be calculated by finding the average value of the elements of the resulting list.

According to the steps and actions described above, and also taking into account the above dataset and its characteristics, the value of the metric is about 0.05.

5. Conclusions

In the work, the possibility and efficiency of using the unsupervised algorithm STEGO for scintigraphic image analysis were studied.

It was shown that cluster predictions of the STEGO algorithm look almost uniformly colored, so the cluster prediction algorithm is not good enough for the scintigraphic image analysis. Whereas, the linear probe predictions of the STEGO algorithm look more promising for further analysis, but the quality of image segmentation highly depends on the original color scheme of the initial images.

Two metrics were used for quality checking. The intersection over union metric (IoU) was chosen for evaluating the correctness of detection of the location of metastases. The training dataset consists of scintigraphic images of patients with various types of cancer and various metastasis appearances. Another version of this metric (mIoU, mean intersection over union) was also used by the creators of STEGO to assess the quality of the model to segment images with different kinds of content.

Since the calculated metrics are not good enough, the use of the STEGO algorithm for scintigraphic image analysis is not possible, or requires the development of a special methodology for this.