# Deep Learning Classification of Colorectal Lesions Based on Whole Slide Images

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. WSIs Database

- structures of normal colon glands (NG);
- structures of serrated lesions (SDL);
- structures of serrated lesions with dysplasia (SDH);
- structures of hyperplastic polyp, microvesicular type (HPM);
- structures of hyperplastic polyp, goblet-cell type (HPG);
- structures of adenomatous polyp, low-grade dysplasia (APL);
- structures of adenomatous polyp, high-grade dysplasia (APH);
- structures of tubular adenoma (TA);
- structures of villous adenoma (VA);
- structures of glandular intraepithelial neoplasia, low-grade (INL);
- structures of glandular intraepithelial neoplasia, high-grade (INH);
- structures of well differentiated adenocarcinoma (AKG1);
- structures of moderate differentiated adenocarcinoma (AKG2);
- structures of poorly differentiated adenocarcinoma (AKG3);
- structures of mucinous adenocarcinoma (MAK);
- structures of signet-ring cell carcinoma (SRC);
- structures of medullary adenocarcinoma (MC);
- structures of undifferentiated carcinoma (AKG4);
- Granulation tissue (GT).

#### 2.2. WSIs Preprocessing

#### 2.3. Approaches to the Problem Statement: Multi-Class and Multi-Label

#### 2.4. The Structure and Types of Neural Networks

#### 2.5. Training the Neural Network

- Training a single neural network to classify fragments into six target classes.
- Training six independent neural networks to solve the one-vs-rest binary classification problems.

#### 2.6. Converting Neural Network Outputs to Class Probabilities

#### 2.7. Training Neural Networks

#### 2.8. Evaluation. Train and Test Splitting

#### 2.9. PR Curves and Their Normalization

## 3. Results

## 4. Discussion

## 5. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Howlader, N.; Noone, A.M.; Krapcho, M.; Miller, D.; Brest, A.; Yu, M.; Ruhl, J.; Tatalovich, Z.; Mariotto, A.; Lewis, D.R.; et al. SEER Cancer Statistics Review, 1975–2016. Available online: https://seer.cancer.gov/csr/1975_2016/ (accessed on 31 August 2022).
- World Cancer Research Fund International. Colorectal Cancer Statistics. Available online: https://www.wcrf.org/cancer-trends/colorectal-cancer-statistics/ (accessed on 31 August 2022).
- Rawla, P.; Sunkara, T.; Barsouk, A. Epidemiology of Colorectal Cancer: Incidence, Mortality, Survival, and Risk Factors. Gastroenterol. Rev. Przegląd Gastroenterol.
**2019**, 14, 89–103. [Google Scholar] [CrossRef] - Thakur, N.; Yoon, H.; Chong, Y. Current Trends of Artificial Intelligence for Colorectal Cancer Pathology Image Analysis: A Systematic Review. Cancers
**2020**, 12, 1884. [Google Scholar] [CrossRef] - Goyal, H.; Mann, R.; Gandhi, Z.; Perisetti, A.; Ali, A.; Aman Ali, K.; Sharma, N.; Saligram, S.; Tharian, B.; Inamdar, S. Scope of Artificial Intelligence in Screening and Diagnosis of Colorectal Cancer. J. Clin. Med.
**2020**, 9, 3313. [Google Scholar] [CrossRef] - Xing, F.; Xie, Y.; Su, H.; Liu, F.; Yang, L. Deep Learning in Microscopy Image Analysis: A Survey. IEEE Trans. Neural Netw. Learn. Syst.
**2018**, 29, 4550–4568. [Google Scholar] [CrossRef] - Litjens, G.; Kooi, T.; Bejnordi, B.E.; Setio, A.A.A.; Ciompi, F.; Ghafoorian, M.; van der Laak, J.A.W.M.; van Ginneken, B.; Sánchez, C.I. A Survey on Deep Learning in Medical Image Analysis. Med. Image Anal.
**2017**, 42, 60–88. [Google Scholar] [CrossRef][Green Version] - LeCun, Y.; Bengio, Y.; Hinton, G. Deep Learning. Nature
**2015**, 521, 436–444. [Google Scholar] [CrossRef] - Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet Classification with Deep Convolutional Neural Networks. Commun. ACM
**2017**, 60, 84–90. [Google Scholar] [CrossRef][Green Version] - Heaton, J. Ian Goodfellow, Yoshua Bengio, and Aaron Courville: Deep Learning. Genet. Program. Evolvable Mach.
**2018**, 19, 305–307. [Google Scholar] [CrossRef][Green Version] - Khan, I.U.; Aslam, N. A Deep-Learning-Based Framework for Automated Diagnosis of COVID-19 Using X-Ray Images. Information
**2020**, 11, 419. [Google Scholar] [CrossRef] - Uysal, F.; Hardalaç, F.; Peker, O.; Tolunay, T.; Tokgöz, N. Classification of Shoulder X-Ray Images with Deep Learning Ensemble Models. Appl. Sci.
**2021**, 11, 2723. [Google Scholar] [CrossRef] - Masood, M.; Nazir, T.; Nawaz, M.; Mehmood, A.; Rashid, J.; Kwon, H.-Y.; Mahmood, T.; Hussain, A. A Novel Deep Learning Method for Recognition and Classification of Brain Tumors from MRI Images. Diagnostics
**2021**, 11, 744. [Google Scholar] [CrossRef] [PubMed] - Taheri Gorji, H.; Kaabouch, N. A Deep Learning Approach for Diagnosis of Mild Cognitive Impairment Based on MRI Images. Brain Sci.
**2019**, 9, 217. [Google Scholar] [CrossRef] [PubMed][Green Version] - Jang, B.-S.; Jeon, S.H.; Kim, I.H.; Kim, I.A. Prediction of Pseudoprogression versus Progression Using Machine Learning Algorithm in Glioblastoma. Sci. Rep.
**2018**, 8, 12516. [Google Scholar] [CrossRef][Green Version] - Jang, B.-S.; Park, A.J.; Jeon, S.H.; Kim, I.H.; Lim, D.H.; Park, S.-H.; Lee, J.H.; Chang, J.H.; Cho, K.H.; Kim, J.H.; et al. Machine Learning Model to Predict Pseudoprogression Versus Progression in Glioblastoma Using MRI: A Multi-Institutional Study (KROG 18-07). Cancers
**2020**, 12, 2706. [Google Scholar] [CrossRef] - Madabhushi, A.; Lee, G. Image Analysis and Machine Learning in Digital Pathology: Challenges and Opportunities. Med. Image Anal.
**2016**, 33, 170–175. [Google Scholar] [CrossRef][Green Version] - Wang, S.; Yang, D.M.; Rong, R.; Zhan, X.; Fujimoto, J.; Liu, H.; Minna, J.; Wistuba, I.I.; Xie, Y.; Xiao, G. Artificial Intelligence in Lung Cancer Pathology Image Analysis. Cancers
**2019**, 11, 1673. [Google Scholar] [CrossRef][Green Version] - Qaiser, T.; Tsang, Y.-W.; Taniyama, D.; Sakamoto, N.; Nakane, K.; Epstein, D.; Rajpoot, N. Fast and Accurate Tumor Segmentation of Histology Images Using Persistent Homology and Deep Convolutional Features. Med. Image Anal.
**2019**, 55, 1–14. [Google Scholar] [CrossRef][Green Version] - Khened, M.; Kori, A.; Rajkumar, H.; Krishnamurthi, G.; Srinivasan, B. A Generalized Deep Learning Framework for Whole-Slide Image Segmentation and Analysis. Sci. Rep.
**2021**, 11, 11579. [Google Scholar] [CrossRef] - Song, Z.; Zou, S.; Zhou, W.; Huang, Y.; Shao, L.; Yuan, J.; Gou, X.; Jin, W.; Wang, Z.; Chen, X.; et al. Clinically Applicable Histopathological Diagnosis System for Gastric Cancer Detection Using Deep Learning. Nat. Commun.
**2020**, 11, 4294. [Google Scholar] [CrossRef] - Noorbakhsh, J.; Farahmand, S.; Foroughi pour, A.; Namburi, S.; Caruana, D.; Rimm, D.; Soltanieh-ha, M.; Zarringhalam, K.; Chuang, J.H. Deep Learning-Based Cross-Classifications Reveal Conserved Spatial Behaviors within Tumor Histological Images. Nat. Commun.
**2020**, 11, 6367. [Google Scholar] [CrossRef] - Syrykh, C.; Abreu, A.; Amara, N.; Siegfried, A.; Maisongrosse, V.; Frenois, F.X.; Martin, L.; Rossi, C.; Laurent, C.; Brousset, P. Accurate Diagnosis of Lymphoma on Whole-Slide Histopathology Images Using Deep Learning. NPJ Digit. Med.
**2020**, 3, 63. [Google Scholar] [CrossRef] [PubMed] - Jones, A.D.; Graff, J.P.; Darrow, M.; Borowsky, A.; Olson, K.A.; Gandour-Edwards, R.; Datta Mitra, A.; Wei, D.; Gao, G.; Durbin-Johnson, B.; et al. Impact of Pre-analytical Variables on Deep Learning Accuracy in Histopathology. Histopathology
**2019**, 75, 39–53. [Google Scholar] [CrossRef] [PubMed] - Iizuka, O.; Kanavati, F.; Kato, K.; Rambeau, M.; Arihiro, K.; Tsuneki, M. Deep Learning Models for Histopathological Classification of Gastric and Colonic Epithelial Tumours. Sci. Rep.
**2020**, 10, 1504. [Google Scholar] [CrossRef] [PubMed][Green Version] - Yang, H.; Chen, L.; Cheng, Z.; Yang, M.; Wang, J.; Lin, C.; Wang, Y.; Huang, L.; Chen, Y.; Peng, S.; et al. Deep Learning-Based Six-Type Classifier for Lung Cancer and Mimics from Histopathological Whole Slide Images: A Retrospective Study. BMC Med.
**2021**, 19, 80. [Google Scholar] [CrossRef] - Ding, H.; Pan, Z.; Cen, Q.; Li, Y.; Chen, S. Multi-Scale Fully Convolutional Network for Gland Segmentation Using Three-Class Classification. Neurocomputing
**2020**, 380, 150–161. [Google Scholar] [CrossRef] - Shelhamer, E.; Long, J.; Darrell, T. Fully Convolutional Networks for Semantic Segmentation. IEEE Trans. Pattern Anal. Mach. Intell.
**2017**, 39, 640–651. [Google Scholar] [CrossRef] - Zhang, Y.; Chen, H.; Wei, Y.; Zhao, P.; Cao, J.; Fan, X.; Lou, X.; Liu, H.; Hou, J.; Han, X.; et al. From Whole Slide Imaging to Microscopy: Deep Microscopy Adaptation Network for Histopathology Cancer Image Classification. In Medical Image Computing and Computer Assisted Intervention—MICCAI 2019; Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.-T., Khan, A., Eds.; Springer International Publishing: Cham, Switzerland, 2019; Volume 11764, pp. 360–368. ISBN 978-3-030-32238-0. [Google Scholar]
- Chen, H.; Han, X.; Fan, X.; Lou, X.; Liu, H.; Huang, J.; Yao, J. Rectified Cross-Entropy and Upper Transition Loss for Weakly Supervised Whole Slide Image Classifier. In Medical Image Computing and Computer Assisted Intervention—MICCAI 2019; Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.-T., Khan, A., Eds.; Springer International Publishing: Cham, Switzerland, 2019; Volume 11764, pp. 351–359. ISBN 978-3-030-32238-0. [Google Scholar]
- Smith, L.N. A Disciplined Approach to Neural Network Hyper-Parameters: Part 1—Learning Rate, Batch Size, Momentum, and Weight Decay. arXiv
**2018**, arXiv:1803.09820. [Google Scholar] - WHO. Digestive System Tumours. In World Health Organization Classification of Tumours, 5th ed.; International Agency for Research on Cancer: Lyon, France, 2019; ISBN 978-92-832-4499-8. [Google Scholar]
- Buslaev, A.; Iglovikov, V.I.; Khvedchenya, E.; Parinov, A.; Druzhinin, M.; Kalinin, A.A. Albumentations: Fast and Flexible Image Augmentations. Information
**2020**, 11, 125. [Google Scholar] [CrossRef][Green Version] - Pan, S.J.; Yang, Q. A Survey on Transfer Learning. IEEE Trans. Knowl. Data Eng.
**2010**, 22, 1345–1359. [Google Scholar] [CrossRef] - He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; IEEE: Las Vegas, NV, USA, 2016; pp. 770–778. [Google Scholar]
- Tan, M.; Le, Q.V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv
**2020**, arXiv:1905.11946. [Google Scholar] - Tsuneki, M.; Kanavati, F. Deep Learning Models for Poorly Differentiated Colorectal Adenocarcinoma Classification in Whole Slide Images Using Transfer Learning. Diagnostics
**2021**, 11, 2074. [Google Scholar] [CrossRef] [PubMed]

**Figure 1.**Examples of labelled hematoxylin- and eosin (H and E)-stained colorectal tissue slides stained with the labelled hematoxylin and eosin (H and E) of different classes: (

**a**) tubular adenoma (TA); (

**b**) villous adenoma (VA); (

**c**) well-differentiated adenocarcinoma (AKG1); (

**d**) poorly differentiated adenocarcinoma (AKG3).

**Figure 4.**PR curves during neural network training: (

**a**) 750 iterations; (

**b**) 3000 iterations; (

**c**) 5250 iterations; and (

**d**) 6750 iterations.

**Figure 5.**Changes in metrics during the training process: (

**a**) the ROC-AUC metric for the adenocarcinoma G1 class; (

**b**) the PR-AUC metric for the adenocarcinoma G1 class; (

**c**) the ROC-AUC metric for the adenocarcinoma G2 class; and (

**d**) the PR-AUC metric for the adenocarcinoma G2 class.

**Figure 6.**The quality of EfficientNet-B4 prediction for all classes: (

**a**) the ROC-AUC metric and (

**b**) the PR-AUC metric.

**Figure 7.**The quality of ResNet-34 prediction for all classes: (

**a**) ROC-AUC metric and (

**b**) PR-AUC metric.

**Figure 8.**Examples of CNN predictions. The patches were painted when the predicted class probability was greater than 0.5.

Original patch | |||||

Processed patch |

CNN Architecture | Number of Parameters, Millions |
---|---|

ResNet-34 | 21.8 |

ResNet-50 | 25.6 |

ResNet-101 | 44.5 |

ResNet-152 | 60.2 |

EfficientNet-B0 | 5.3 |

EfficientNet-B1 | 7.8 |

EfficientNet-B2 | 9.2 |

EfficientNet-B3 | 12 |

EfficientNet-B4 | 19 |

Data Sets | Number of WSIs in Set | Patch Size | Class Names and Number of Patches in Each | |||||
---|---|---|---|---|---|---|---|---|

AKG1 | AKG2 | AKG3 | NG | TA | VA | |||

Train | 1071 | 224 × 224 | 39104 | 39573 | 1885 | 102101 | 288570 | 245649 |

500 × 500 | 7909 | 7831 | 356 | 20311 | 58977 | 50726 | ||

Validation | 357 | 224 × 224 | 7543 | 6543 | 502 | 45447 | 103798 | 46193 |

500 × 500 | 1486 | 1236 | 94 | 9053 | 21260 | 9664 | ||

Test | 357 | 224 × 224 | 9233 | 15640 | 601 | 38665 | 114408 | 48830 |

500 × 500 | 1857 | 3105 | 110 | 7646 | 23516 | 10242 |

Metrics | Classes | ||||||
---|---|---|---|---|---|---|---|

NG | AKG1 | AKG2 | AKG3 | TA | VA | ||

ROC-AUC | Metrics value | 0.96 | 0.85 | 0.94 | 0.91 | 0.80 | 0.84 |

CNN | EfficientNet-b4 | EfficientNet-b4 | EfficientNet-b4 | EfficientNet-b4 | EfficientNet-b4 | EfficientNet-b4/ResNet-34 | |

PR-AUC | Metrics value | 0.86 | 0.53 | 0.77 | 0.70 | 0.51 | 0.53 |

CNN | EfficientNet-b4 | EfficientNet-b4 | EfficientNet-b4 | ResNet-34 | EfficientNet-b4 | EfficientNet-b4 |

**Table 5.**Metric values for EfficientNet-b4 predictions. Patch-level evaluation corresponds to the WSI level in a way similar to the micro-averaging corresponding to the macro-averaging.

Metrics | Level | Classes | |||||
---|---|---|---|---|---|---|---|

NG | AKG1 | AKG2 | AKG3 | TA | VA | ||

Accuracy | Patch | 0.905 | 0.828 | 0.730 | 0.833 | 0.793 | 0.855 |

WSI | 0.838 | 0.871 | 0.876 | 0.974 | 0.664 | 0.886 | |

Precision | Patch | 0.944 | 0.200 | 0.339 | 1.000 | 0.308 | 0.612 |

WSI | 0.939 | 1.000 | 0.625 | 1.000 | 0.368 | 0.500 | |

Sensitivity | Patch | 0.463 | 0.009 | 0.669 | 0.000 | 0.190 | 0.372 |

WSI | 0.553 | 0.000 | 0.577 | 0.000 | 0.318 | 0.277 | |

Specificity | Patch | 0.994 | 0.992 | 0.741 | 1.000 | 0.914 | 0.952 |

WSI | 0.981 | 1.000 | 0.933 | 1.000 | 0.794 | 0.964 | |

NPV | Patch | 0.902 | 0.833 | 0.918 | 0.833 | 0.849 | 0.883 |

WSI | 0.813 | 0.871 | 0.920 | 0.974 | 0.756 | 0.912 | |

F1-score | Patch | 0.622 | 0.017 | 0.450 | 0.000 | 0.236 | 0.463 |

WSI | 0.696 | 0.000 | 0.600 | 0.000 | 0.341 | 0.357 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Soldatov, S.A.; Pashkov, D.M.; Guda, S.A.; Karnaukhov, N.S.; Guda, A.A.; Soldatov, A.V. Deep Learning Classification of Colorectal Lesions Based on Whole Slide Images. *Algorithms* **2022**, *15*, 398.
https://doi.org/10.3390/a15110398

**AMA Style**

Soldatov SA, Pashkov DM, Guda SA, Karnaukhov NS, Guda AA, Soldatov AV. Deep Learning Classification of Colorectal Lesions Based on Whole Slide Images. *Algorithms*. 2022; 15(11):398.
https://doi.org/10.3390/a15110398

**Chicago/Turabian Style**

Soldatov, Sergey A., Danil M. Pashkov, Sergey A. Guda, Nikolay S. Karnaukhov, Alexander A. Guda, and Alexander V. Soldatov. 2022. "Deep Learning Classification of Colorectal Lesions Based on Whole Slide Images" *Algorithms* 15, no. 11: 398.
https://doi.org/10.3390/a15110398