Development of a Deep-Learning Pipeline to Recognize and Characterize Macrophages in Colo-Rectal Liver Metastasis
Abstract
:Simple Summary
Abstract
1. Introduction
2. Materials and Methods
2.1. Dataset
2.2. Data Annotation
2.3. Data Augmentation
2.4. Deep Learning Models
2.5. Deep Learning Strategies
2.5.1. Semantic Segmentation: Per-Pixel Weights
2.5.2. Instance Segmentation
2.6. Metrics
3. Results
3.1. Dataset
3.2. Deep Learning Models
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
TME | Tumor Microenvironment |
TAM | Tumor Associated Macrophage |
CLM | Colo-Rectal Liver Metastases |
CNN | Convolutional Neural Network |
IoU | Intersection over Union |
SBD | Symmetric Best Dice |
CAD | Computer-Aided Diagnosis |
ROI | Region of Interest |
References
- Griffin, J.; Treanor, D. Digital pathology in clinical use: Where are we now and what is holding us back? Histopathology 2017, 70, 134–145. [Google Scholar] [CrossRef]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef]
- Haque, I.R.I.; Neubert, J. Deep learning approaches to biomedical image segmentation. Inform. Med. Unlocked 2020, 18, 100297. [Google Scholar] [CrossRef]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 2012, 25, 1097–1105. [Google Scholar] [CrossRef]
- Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.E.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going Deeper with Convolutions. arXiv 2014, arXiv:1409.4842. [Google Scholar]
- Wang, H.; Roa, A.C.; Basavanhally, A.N.; Gilmore, H.L.; Shih, N.; Feldman, M.; Tomaszewski, J.; Gonzalez, F.; Madabhushi, A. Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. J. Med. Imaging 2014, 1, 034003. [Google Scholar] [CrossRef]
- Bulten, W.; Pinckaers, H.; van Boven, H.; Vink, R.; de Bel, T.; van Ginneken, B.; van der Laak, J.; Hulsbergen-van de Kaa, C.; Litjens, G. Automated deep-learning system for Gleason grading of prostate cancer using biopsies: A diagnostic study. Lancet Oncol. 2020, 21, 233–241. [Google Scholar] [CrossRef] [Green Version]
- Yu, K.H.; Zhang, C.; Berry, G.J.; Altman, R.B.; Ré, C.; Rubin, D.L.; Snyder, M. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat. Commun. 2016, 7, 1–10. [Google Scholar] [CrossRef] [Green Version]
- Coudray, N.; Ocampo, P.S.; Sakellaropoulos, T.; Narula, N.; Snuderl, M.; Fenyö, D.; Moreira, A.L.; Razavian, N.; Tsirigos, A. Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning. Nat. Med. 2018, 24, 1559–1567. [Google Scholar] [CrossRef]
- Schmauch, B.; Romagnoni, A.; Pronier, E.; Saillard, C.; Maillé, P.; Calderaro, J.; Kamoun, A.; Sefta, M.; Toldo, S.; Zaslavskiy, M.; et al. A deep learning model to predict RNA-Seq expression of tumours from whole slide images. Nat. Commun. 2020, 11, 1–15. [Google Scholar]
- Angelova, M.; Mlecnik, B.; Vasaturo, A.; Bindea, G.; Fredriksen, T.; Lafontaine, L.; Buttard, B.; Morgand, E.; Bruni, D.; Jouret-Mourin, A.; et al. Evolution of metastases in space and time under immune selection. Cell 2018, 175, 751–765. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Galon, J.; Costes, A.; Sanchez-Cabo, F.; Kirilovsky, A.; Mlecnik, B.; Lagorce-Pagès, C.; Tosolini, M.; Camus, M.; Berger, A.; Wind, P.; et al. Type, density, and location of immune cells within human colorectal tumors predict clinical outcome. Science 2006, 313, 1960–1964. [Google Scholar] [CrossRef] [Green Version]
- Laghi, L.; Bianchi, P.; Miranda, E.; Balladore, E.; Pacetti, V.; Grizzi, F.; Allavena, P.; Torri, V.; Repici, A.; Santoro, A.; et al. CD3+ cells at the invasive margin of deeply invading (pT3–T4) colorectal cancer and risk of post-surgical metastasis: A longitudinal study. Lancet Oncol. 2009, 10, 877–884. [Google Scholar] [CrossRef]
- Fridman, W.H.; Zitvogel, L.; Sautès-Fridman, C.; Kroemer, G. The immune contexture in cancer prognosis and treatment. Nat. Rev. Clin. Oncol. 2017, 14, 717. [Google Scholar] [CrossRef] [PubMed]
- Pagès, F.; Mlecnik, B.; Marliot, F.; Bindea, G.; Ou, F.S.; Bifulco, C.; Lugli, A.; Zlobec, I.; Rau, T.T.; Berger, M.D.; et al. International validation of the consensus Immunoscore for the classification of colon cancer: A prognostic and accuracy study. Lancet 2018, 391, 2128–2139. [Google Scholar] [CrossRef]
- Marliot, F.; Chen, X.; Kirilovsky, A.; Sbarrato, T.; El Sissy, C.; Batista, L.; Van den Eynde, M.; Haicheur-Adjouri, N.; Anitei, M.G.; Musina, A.M.; et al. Analytical validation of the Immunoscore and its associated prognostic value in patients with colon cancer. J. Immunother. Cancer 2020, 8, e000272. [Google Scholar] [CrossRef] [PubMed]
- Mantovani, A.; Allavena, P.; Sica, A.; Balkwill, F. Cancer-related inflammation. Nature 2008, 454, 436–444. [Google Scholar] [CrossRef]
- Ruffell, B.; Affara, N.I.; Coussens, L.M. Differential macrophage programming in the tumor microenvironment. Trends Immunol. 2012, 33, 119–126. [Google Scholar] [CrossRef] [Green Version]
- Murray, P.J.; Allen, J.E.; Biswas, S.K.; Fisher, E.A.; Gilroy, D.W.; Goerdt, S.; Gordon, S.; Hamilton, J.A.; Ivashkiv, L.B.; Lawrence, T.; et al. Macrophage activation and polarization: Nomenclature and experimental guidelines. Immunity 2014, 41, 14–20. [Google Scholar] [CrossRef] [Green Version]
- Mantovani, A.; Marchesi, F.; Malesci, A.; Laghi, L.; Allavena, P. Tumour-associated macrophages as treatment targets in oncology. Nat. Rev. Clin. Oncol. 2017, 14, 399. [Google Scholar] [CrossRef]
- DeNardo, D.G.; Ruffell, B. Macrophages as regulators of tumour immunity and immunotherapy. Nat. Rev. Immunol. 2019, 19, 369–382. [Google Scholar] [CrossRef]
- Donadon, M.; Torzilli, G.; Cortese, N.; Soldani, C.; Di Tommaso, L.; Franceschini, B.; Carriero, R.; Barbagallo, M.; Rigamonti, A.; Anselmo, A.; et al. Macrophage morphology correlates with single-cell diversity and prognosis in colorectal liver metastasis. J. Exp. Med. 2020, 217, e20191847. [Google Scholar] [CrossRef]
- Cortese, N.; Carriero, R.; Laghi, L.; Mantovani, A.; Marchesi, F. Prognostic significance of tumor-associated macrophages: Past, present and future. In Seminars in Immunology; Elsevier: Amsterdam, The Netherlands, 2020; p. 101408. [Google Scholar]
- Brabandere, B.D.; Neven, D.; Gool, L.V. Semantic Instance Segmentation with a Discriminative Loss Function. arXiv 2017, arXiv:1708.02551. [Google Scholar]
- Shorten, C.; Khoshgoftaar, T.M. A survey on image data augmentation for deep learning. J. Big Data 2019, 6, 1–48. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention; Springer: Berlin/Heidelberg, Germany, 2015; pp. 234–241. [Google Scholar]
- Badrinarayanan, V.; Kendall, A.; Cipolla, R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 2481–2495. [Google Scholar] [CrossRef]
- Chen, L.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 40, 834–848. [Google Scholar] [CrossRef] [PubMed]
- Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Falk, T.; Mai, D.; Bensch, R.; Çiçek, Ö.; Abdulkadir, A.; Marrakchi, Y.; Böhm, A.; Deubner, J.; Jäckel, Z.; Seiwald, K.; et al. U-Net: Deep learning for cell counting, detection, and morphometry. Nat. Methods 2019, 16, 67–70. [Google Scholar] [CrossRef]
- Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; Chen, L.C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4510–4520. [Google Scholar]
- Hegde, P.S.; Chen, D.S. Top 10 challenges in cancer immunotherapy. Immunity 2020, 52, 17–35. [Google Scholar] [CrossRef]
- Bruni, D.; Angell, H.K.; Galon, J. The immune contexture and Immunoscore in cancer prognosis and therapeutic efficacy. Nat. Rev. Cancer 2020, 20, 662–680. [Google Scholar] [CrossRef]
- Quezada-Marín, J.I.; Lam, A.K.; Ochiai, A.; Odze, R.D.; Washington, K.M.; Fukayama, M.; Rugge, M.; Klimstra, D.S.; Nagtegaal, I.D.; Tan, P.H.; et al. Gastrointestinal tissue-based molecular biomarkers: A practical categorisation based on the 2019 World Health Organization classification of epithelial digestive tumours. Histopathology 2020, 77, 340–350. [Google Scholar] [CrossRef] [PubMed]
- Bera, K.; Schalper, K.A.; Rimm, D.L.; Velcheti, V.; Madabhushi, A. Artificial intelligence in digital pathology—New tools for diagnosis and precision oncology. Nat. Rev. Clin. Oncol. 2019, 16, 703–715. [Google Scholar] [CrossRef]
Model | IoU Mean | IoU StDev | SBD Mean | SBD StDev |
---|---|---|---|---|
Segnet Vanilla | 82.1318 | 1.59 | 40.5940 | 4.80 |
Unet Vanilla | 82.3440 | 2.59 | 39.8712 | 2.90 |
Segnet Weights | 54.4297 | 3.87 | 53.9024 | 3.47 |
Unet Weights | 59.6215 | 3.15 | 61.3407 | 2.21 |
Deeplab Instance | 89.1332 | 3.85 | 79.0028 | 3.72 |
Unet Instance | 75.4104 | 5.30 | 37.1577 | 5.30 |
Segnet Instance | 79.9746 | 2.15 | 60.7870 | 5.75 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Cancian, P.; Cortese, N.; Donadon, M.; Di Maio, M.; Soldani, C.; Marchesi, F.; Savevski, V.; Santambrogio, M.D.; Cerina, L.; Laino, M.E.; et al. Development of a Deep-Learning Pipeline to Recognize and Characterize Macrophages in Colo-Rectal Liver Metastasis. Cancers 2021, 13, 3313. https://doi.org/10.3390/cancers13133313
Cancian P, Cortese N, Donadon M, Di Maio M, Soldani C, Marchesi F, Savevski V, Santambrogio MD, Cerina L, Laino ME, et al. Development of a Deep-Learning Pipeline to Recognize and Characterize Macrophages in Colo-Rectal Liver Metastasis. Cancers. 2021; 13(13):3313. https://doi.org/10.3390/cancers13133313
Chicago/Turabian StyleCancian, Pierandrea, Nina Cortese, Matteo Donadon, Marco Di Maio, Cristiana Soldani, Federica Marchesi, Victor Savevski, Marco Domenico Santambrogio, Luca Cerina, Maria Elena Laino, and et al. 2021. "Development of a Deep-Learning Pipeline to Recognize and Characterize Macrophages in Colo-Rectal Liver Metastasis" Cancers 13, no. 13: 3313. https://doi.org/10.3390/cancers13133313