Automated Detection of Liver Histopathological Findings Based on Biopsy Image Processing

Abstract

Hepatic steatosis is the accumulation of fat in the hepatic cells and the liver. Triglycerides and other kinds of molecules are included in the lipids. When there is some defect in the process, hepatic steatosis arise, during which the free fatty acids are taken by the liver and exuded as lipoproteins. Alcohol is the main cause of steatosis when excessive amounts are consumed for a long period of time. In many cases, steatosis can lead to inflammation that is mentioned as steatohepatitis or non-alcoholic steatohepatitis (NASH), which can later lead to fibrosis and finally cirrhosis. For automated detection and quantification of hepatic steatosis, a novel two-stage methodology is developed in this study. Initially, the image is processed in order to become more suitable for the detection of fat regions and steatosis quantification. In the second stage, initial candidate image regions are detected, and then they are either validated or discarded based on a series of criteria. The methodology is based on liver biopsy image analysis, and has been tested using 40 liver biopsy images obtained from patients who suffer from hepatitis C. The obtained results indicate that the proposed methodology can accurately assess liver steatosis.

1. Introduction

Steatosis or fatty liver is the term that describes the abnormal buildup fat in the liver. While the existence of fat in the liver is normal, if it is more than 10% of the liver weight then it constitutes a hazardous condition. Steatosis may be the result of a short-term or long-term condition, which by itself is not harmful, but may lead to different kinds of problems such as liver cirrhosis [1]. Fatty liver is a reversible condition that can be resolved with changed behaviors. Triglycerides are mostly included in “droplets” of fat in fatty liver [2]. Steatosis often has no symptoms and it does not hurt and may not be noticed for a long period of time, however, in severe cases, the liver size grows up to three times, thus leading to a very harmful condition.

1.1. Types of Fatty Liver

Fatty liver is categorized into four types: Nonalcoholic Fatty Liver (NAFL), Alcoholic Fatty Liver (AFL), Nonalcoholic Steatohepatitis (NASH), and Acute Fatty Liver of Pregnancy. NAFL is developed when fat accumulates in the liver and this is not related to alcohol. NAFL causes a buildup in the liver tissue and it is strongly associated with metabolic syndromes. When more than 10 percent of the liver is fat, NAFL is diagnosed [3]. AFL is the earliest stage of alcohol-related liver disease. Because of the alcohol related damage in the liver, it cannot break down fat. Although AFL may disappear if the patient abstains from alcohol, cirrhosis may also develop if alcohol use continues [3]. In NASH, swelling of the liver is induced when enough fat is accumulated. If left untreated, NASH can impair liver function and can cause liver failure and permanent scaring of the liver, while appetite loss, nausea, vomiting, abdominal pain, and yellowing of the skin (jaundice) are some of its symptoms [3]. Acute Fatty Liver of Pregnancy is a rare complication of pregnancy and can be life-threatening. Usually appearing in the third trimester, the symptoms may include pain in the upper-right abdomen, persistent nausea and vomiting, jaundice, and general malaise. Most women improve after delivery and have no lasting effects [3].

1.2. Causes of Liver Steatosis

Since regular alcohol consumption causes fat deposition in the liver, alcoholism is the most common reason for steatosis. Hepatic steatosis is developed by people who consume large amount of alcohol for a long period of time [4].

Some other reasons that can result in liver steatosis are:

(1)

Drugs. Cortisone, synthetic estrogens, contraceptives, amiodarone (Angoron), tamoxifen, and tetracyclines when consumed for a long time may cause hepatic steatosis [5].

(2)

Diabetes. Chances for fat deposition in the liver increases in cases where diabetes remains unregulated [5].

(3)

Obesity. Liver steatosis is caused by central obesity characterized by increased fat deposition in the abdomen [5].

(4)

Sudden weight loss. Crash diets leading to rapid weight loss can also cause fat deposition in the liver.

(5)

Rare causes. A series of diseases such as hepatitis C, Crohn’s disease, ulcerative colitis, Wilson’s disease and avitalipoproteinaimia are also considered rare causes for hepatic steatosis.

1.3. Diagnosis and Identification of Steatosis

Ultrasound (US), Computed Tomography (CT), Magnetic Resonance Imaging (MRI) and biopsy are imaging techniques which are used for the diagnosis of hepatic steatosis [5]. However, a liver biopsy examination is required for the determination of the severity of steatosis in the liver tissue [6]. Liver biopsies have some very important advantages over imaging methods since they provide high sensitivity in steatosis diagnosis and severity assessment. The disadvantage of a biopsy is that it is an invasive method; obtaining biopsy material for examination maximizes the performance of diagnosis [7] since non-invasive methods cannot estimate the degree of liver disease with certainty [8].

The steatosis on biopsy images can be identified by the shape and size of fat accumulation points, since those areas are expanded in circular shapes. In Figure 1, fat droplets are illustrated in the small enlarged area (blue circles).

1.4. Related Work

According to the literature, recent studies for the assessment of steatosis have been presented regarding quantitative staging, diagnosis, and treatment monitoring. Since Fatty Liver Diseases are currently the focal point of liver diseases, such studies have significantly increased. According to the employed technology, the image analysis methods for steatosis assessment can be categorized as (i) Histological/Biopsy; (ii) CT [9,10]; (iii) US [11,12]; and (iv) MRI analysis [2,13]. Reasonably, advantages and disadvantages of these methods can be also listed according to the technology used. For example, although a CT-based method is non-invasive, ionizing radiation is required. An extensive review has been recently presented by Goceri et al. [14] comparing studies from all categories.

It is obvious from the above comparative study that biopsy image analysis has been used less than other approaches. Early methods aim to correlate pathologist assessment for the steatosis staging with quantitative results of digital image analysis. Such an approach [15,16,17] either uses morphometric analysis, without providing imaging details, or employs software packages such as Image Pro Plus [18] for simple thresholding application. Classical image processing techniques, such as morphology operations and algorithms, have been also used. Liquori et al. [19] presented an automated method based on morphology, where fat regions are recognized according to color uniformity and circular shape. The procedure was tested in liver biopsy images obtained from rats, and the results are not numerical. Kong et al. [2] presented a watershed-based image analysis method in order to correlate the results with MRI lipid analysis. More sophisticated methods have been recently presented based on machine learning. The earliest one is presented by Roullier et al. [20], where modifications of the Fuzzy C-Means Algorithm was used to cluster the pixels of HSV (Hue, Saturation, Value) saturation images. Unsupervised clustering was also used by Nativ et al. [21], where features of the detected regions were extracted from the rules of Decision Trees. Sciarabba et al. [22] performed a multi-step procedure by using a clustering technique and a two-levels thresholding for shape parameters to correctly distinguish fat droplets from other not stained objects. Furthermore, a classification approach [23] which directly classifies all the white regions in liver tissue, has been proposed. The methodology is based on feature extraction from image regions and several supervised machine learning classifiers. Finally, a recently presented work [24] explores the spatial characteristics of fat globules using morphological filtering and sparse linear models.

2. Description of Methodology

The proposed methodology is developed in two stages. Initially, in the first stage, several image preprocessing steps are used to generate the image in which the fat detection and evaluation is made. In the second stage, some candidate image regions are tracked and then an initial check is carried out, based on the eccentricity of the region, to decide if a candidate region will be discarded (not fat) or will proceed to the next step. The candidate fat regions which are not rejected through the first stage, are then assessed further based on their shape (roundness). This feature is calculated based on the area and the perimeter of the region and expresses the region’s circularity (if the value is 1, then the region is a circle). Regions meeting the roundness criterion are categorized as fat. The second stage of the methodology is threshold-based and lies inside an iterative procedure; in each iteration, the threshold becomes stricter, thus, recognizing less fat regions, while the recursive procedure goes on until all candidate regions in the image are checked and classified as fat or not-fat (rejected). In Figure 2, a flowchart of the proposed methodology is presented.

2.1. Image Preprocessing

The first stage of the methodology is an image preprocessing stage, in which the initial liver biopsy image is processed using a number of image processing steps:

i

Image magnification. The methodology is designed to process low-resolution images. Thus, in the first step of the preprocessing stage the image is enlarged by 2×, to make more visible the joined regions. Bicubic interpolation is employed to calculate the additional pixels. A weighted average of pixels in the nearest 4-by-4 neighborhood is the new value of each pixel.

ii

Convert to grayscale. The image is converted from red, green, blue (RGB) to grayscale, using a weighted sum of R, G and Β:

$$I\_grayscale=0.2989R+0.5870G+0.1140B$$

(1)

where I_grayscale is the grayscale image and R, G, and B are the intensity values of each RGB channel, respectively [25].

iii

Histogram equalization. Histogram normalization is used to adjust the brightness of the image.

iv

Edge sharpening. This step is done by using the unsharp masking method which returns an upgraded version of the grayscale image, where the edges and features have been sharpened.

v

Convert to binary. Finally, the image is converted to binary, using histogram thresholding. The threshold was defined based on a trial-and-error approach, and it was set to 200.

The image processing steps of the first stage of the methodology are illustrated in Figure 3.

The steps of the preprocessing stage are illustrated in Figure 3. In the binary image, all white regions are recognized following a three steps algorithm: (i) search for the next unlabeled pixel; (ii) use a flood-fill algorithm to label all the pixels in the connected component; and (iii) repeat steps 1 and 2 until all the pixels are labelled. Based of the above, each connected region is labeled with a different label. Then, regions are filtered according to their size and all very small and extremely large regions are rejected to avoid false positive points. This is because very small regions (<10 pixels) may appear because of poor resolution of the image, while extremely large regions (>1000 pixels) are related to other types of areas such as central veins, portal veins and arteries, sinusoids, and bile ducts.

2.2. Second Stage

The second stage of the methodology consists of an iterative procedure. Initially, a morphological process is applied to all regions. The morphological opening performs an erosion operation followed by a dilation operation using a disk structuring element with a starting size of 2, and it increases in each iteration to separate the fat accumulation regions that are joined.

The eccentricity of each detected region in the image is calculated as:

$$\mathrm{eccentricity}=\sqrt{1-{\left(\frac{b}{c}\right)}^2},$$

(2)

where c is the half length of the major axis and b is the half length of the secondary axis. Eccentricity of a region shows where the center of the region is, and its values range from 0 to 1, indicating whether the region’s shape is circular or linear; if this value is 1, then the shape of the region is a straight line, while if it is 0 then the shape of the region is a circle. The first test was chosen to be more relaxed so we set the value of the threshold to 0.8. If the eccentricity of the point is greater than 0.8 then it is discarded, otherwise it goes to the next stricter control.

The roundness of each detected region in the image is calculated as:

$$\mathrm{roundness}=4\pi \frac{a}{p}$$

(3)

where, a and p are the area and perimeter of the region, respectively. The roundness determines if the shape of the region resembles a circle; if the value is 1, then the shape is a circle. The initial threshold for roundness is set to 0.6 and increases with each iteration until it reaches 0.9.

At the end of each iteration, an individual result image is generated, which includes all regions identified as fat in this iteration. To be sure that the regions are identified only once, all regions evaluated in each iteration are subtracted from the image (either classified as fat or rejected). If there are no other regions to evaluate, then the iterative procedure terminates. The final result image with all fat regions is generated after the end of the iterative procedure by merging all individual result images generated at each iterative procedure. An example of the iterative procedure is illustrated in Figure 4. In the first figure of each iterative procedure are the regions that meet the eccentricity criterion, in the second figure are the regions meeting the roundness criterion and in the third figure are the regions that are characterized as fat accumulation points.

3. Results

Data Set

Forty biopsy images where used for the validation of the proposed methodology. All samples were from patients with hepatitis C, and twenty images were selected from patients who also presented with steatosis while the remaining twenty were from patients without steatosis. The images were obtained in the University Hospital of Patras (Greece). Biopsies were digitized using a Nikon Eclipse 50i microscope with a DS-Fi2 high-definition color camera, at 2× magnification. All images have been characterized due to Ishak HAI (Histology Activity Index) while the fat regions in each image have been annotated by expert pathanatomists.

Using the experts’ annotations and the outcome of the methodology, all regions in the image were divided into the following categories: (1) True Positive (TP), being the regions categorized as fat and detected as fat from the proposed methodology; (2) False Positive (FP), being non-fat areas classified as fat; and (3) False Negative (FN), being fat annotated regions that were not recognized as fat from the methodology. Based on these categories, Classification Accuracy, Sensitivity, and Positive Predictive Value (PPV) for the detected fat regions were calculated:

$$\mathrm{Classification}\mathrm{Accuracy}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}},$$

(4)

$$\mathrm{Sensitivity}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}},$$

(5)

$$\mathrm{PPV}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}.$$

(6)

All Classification Accuracy, Sensitivity, and PPV results for the region detection are presented in

Table 1. Accuracy, Sensitivity, and Positive Predictive Value (PPV) results for the fat region detection.

#	Accuracy (%)	Sensitivity (%)	PPV (%)	#	Accuracy (%)	Sensitivity (%)	PPV (%)
1	93.05	96.56	96.23	11	88.11	88.11	100
2	92.97	94.44	98.35	12	94	94	100
3	93.18	93.18	100	13	91.89	91.89	100
4	86.22	89.05	96.44	14	92.97	92.97	100
5	97.78	97.78	100	15	97.11	97.11	100
6	93.44	100	93.44	16	94.70	98.42	96.15
7	90.41	90.41	100	17	97.62	97.61	100
8	79.35	86.90	90.12	18	98.75	98.75	100
9	93.18	95.35	97.62	19	86.57	100	86.56
10	91.84	91.84	100	20	95.45	100	95.45

. Results are only for the images from patients with steatosis.

Beside the region-based results, pixel-based results were also calculated. Annotated Steatosis is the real percentage of fat that exists in liver:

$$\mathrm{Annotated}\mathrm{Steatosis}=\frac{\mathrm{total}\mathrm{pixels}\mathrm{in}\mathrm{fat}\mathrm{annotated}\mathrm{regions}}{N},$$

(7)

with N being the total number of pixels in the liver tissue area, while Calculated Steatosis is the estimated fat percentage:

$$\mathrm{Calculated}\mathrm{Steatosis}=\frac{\mathrm{total}\mathrm{pixels}\mathrm{in}\mathrm{fat}\mathrm{detected}\mathrm{regions}}{N}.$$

(8)

Also, the absolute error, which is the difference between the two percentages, is calculated as:

$$\mathrm{Absolute}\mathrm{Error}=\left|\mathrm{Annotated}\mathrm{Steatosis}-\mathrm{Calculated}\mathrm{Steatosis}\right|.$$

(9)

The obtained results for Annotated Steatosis, Calculated Steatosis, and Absolute Error are presented in

Table 2. Annotated and Calculated Steatosis (%), and Absolute Error (%).

#	Annotated Steatosis (%)	Calculated Steatosis (%)	Absolute Error %	#	Annotated Steatosis (%)	Calculated Steatosis (%)	Absolute Error %
1	4.99	4.79	0.20	21	0	1.23	1.23
2	7.84	6.68	1.16	22	0	0.67	0.67
3	9.48	8.65	0.83	23	0	0.21	0.21
4	10.20	6.91	3.29	24	0	1.51	1.51
5	5.03	4.25	0.78	25	0	0.11	0.11
6	1.25	1.36	0.11	26	0	1.90	1.90
7	2.34	1.85	0.49	27	0	0.22	0.22
8	15.56	13.14	2.42	28	0	0.13	0.13
9	4.41	4.48	0.07	29	0	5.75	5.75
10	4.89	3.19	1.70	30	0	1.43	1.43
11	6.11	5.38	0.73	31	0	3.76	3.76
12	4.12	3.85	0.27	32	0	2.37	2.37
13	4.90	4.63	0.27	33	0	3.18	3.18
14	5.59	5.25	0.34	34	0	3.69	3.69
15	6.59	6.28	0.31	35	0	0.02	0.02
16	8.07	8.27	0.20	36	0	1.20	1.20
17	4.92	4.75	0.17	37	0	0.89	0.89
18	6.12	6	0.12	38	0	0	0
19	2.46	2.83	0.37	39	0	0.28	0.28
20	2.16	2.28	0.12	40	0	0.17	0.17

(for all images in the dataset). For the Absolute Error, the mean value (for all samples) is μ = 1.07 and the standard deviation is σ = 1.29. Furthermore, the Concordance Correlation Coefficient (CCC) was used to check the agreement between the steatosis percentage provided by the experts and the calculated value resulting from the proposed methodology [26]. Values for CCC range in [−1, 1], with 1 corresponding to perfect agreement, −1 to perfect disagreement, and 0 to no agreement. The CCC calculated for the proposed methodology results is 0.87.

4. Discussion and Conclusions

In Figure 5, the actual (annotated) and calculated percentage of steatosis for each image is presented. The first 20 samples include steatosis and the rest are normal (not presenting steatosis). The outcome shows that the proposed methodology can precisely detect fat and calculate liver steatosis. However, some normal samples presented a large false positive detection.

The proposed methodology is optimized as far as it concerns processing time, since it is designed to process low resolution images. Images were obtained in a low resolution (2× magnification) using plain pathology laboratory equipment (microscope). This is an important advantage of the proposed methodology in comparison to other methods presented in the literature which use high resolution images, since acquisition of high resolution images takes a lot of time (to manually obtain several figures and merge them into a single one) or sophisticated equipment (microscope with scanning and image merging ability). Therefore, the time to process a high-resolution image is significantly larger and computationally demanding. Another advantage of the proposed methodology is that it is fully automated, without any manual interference required in any step.

In order to assess the robustness of the methodology, a threshold analysis has been conducted for the intensity used in the histogram thresholding step to convert the image to binary (the v step in the image preprocessing stage) and on the value used for the eccentricity threshold (second stage). The average accuracy for the detection fat regions in the steatosis images (i.e., images 1–20) was calculated using grid search for values ranging from 180 to 220 for the gray level in the histogram thresholding and from 0.7 to 0.9 for the eccentricity threshold. The threshold analysis results are presented in Figure 6.

From the threshold analysis, it is clear that the gray level threshold (intensity) can range from 190 to 210 without greatly affecting the accuracy results (being > 80%), while the eccentricity threshold has a small impact on the accuracy. This is due to the fact that the eccentricity control is followed by the roundness control which becomes stricter in each iteration, and, thus, it is mainly to reduce the number of detected fat regions by rejecting those with shapes resembling straight lines. However, very small values result in large numbers of false negatives, since a lot of fat annotated regions are falsely discarded from the analysis.

Furthermore, the methodology at the moment is designed to process tissue images, which are obtained by cropping initial images (with tissue and background) until only tissue remains. Nevertheless, although this is a disadvantage of the proposed methodology, it is not a significant aspect in the clinical practice, since cutting a liver tissue image can be done very easily.

Table 3. Comparison with other methods from the literature.

Author/Year	Sample	Method	Results
Marsman et al., 2004 [16]	46 High-definition biopsy images	No details for image analysis. Correlation between the measurement of fat using automated software and the assessment of Pathologists.	High correlation value (r = 0.97)
Roullier et al., 2007 [20]	37 Images	Modification of Fuzzy C-Means Algorithm for pixel clustering.	High correlation with pathologist assessment (r2 > 0.85)
El Badry 2009 [17]	46 Images	Thresholding for white area detection and roundness criteria for lipid droplets.	Poor correlation with four pathologists (Spearman rank correlation coefficient: 0.82, 0.22, 0.28, 0.38)
Liquori et al., 2009 [19]	Biopsy images from rats	Morphology image preprocessing. Detect fat regions based on color and circular shape.	No method evaluation. Follow-up results for fat development during diet in rats
Turlin et al., 2009 [18]	97 Biopsy images	Image analysis using Image Pro Plus. Filtering and thresholding.	Strong correlation with pathologist’s grading (r = 0.89)
Kong et al., 2011 [2]	21,900 Steatosis regions	Image preprocessing. Separation of bonded areas, image rotation and deletion of small points.	High Pearson Correlation value with MRI (ρ = 0.92)
Nativ et al., 2014 [21]	54 Histological Images	K-means clustering and feature extraction using Decision trees.	Sensitivity 97% Specificity 60%
Vanderbeck et al., 2014 [23]	59 Biopsy images	Image preprocessing. Clustering pixel using the k-means algorithm. Supervised machine learning classifiers.	The overall accuracy of the classification algorithm is greater than 89%
Sciarabba et al., 2015 [22]	15 Images	Clustering using K-means and thresholding in shape features.	Detected steatosis 91% False positive ratio 5%
Nazre, 2016 [24]	38 High resolution images	Morphological filtering and sparse linear models.	Pearson’s correlation with pathologists (ρ = 0.90)
Proposed methodology	40 Low-resolution biopsy images	Image preprocessing. Examination of regions according to their eccentricity and roundness.	Region detection (accuracy > 90%) Steatosis assessment (Abs. Error: 1.07% ± 1.29%) Concordance Correlation Coefficient (CCC = 0.87)

summarizes studies presented in the literature and the proposed methodology. Although a direct comparison is not feasible, since each method is tested using a different dataset, the proposed methodology compares well in terms of accuracy and sensitivity.

The main advantage of the proposed methodology is that it can process images of low resolution, unlike most of the methods introduced in the literature, which can be obtained with standard laboratory equipment. Moreover, the processing time is reduced. Both of these characteristics (easily obtained images and small processing time) are important features that can allow the proposed methodology to be easily adopted in common, everyday clinical practice.