Image analysis is a driver for the development of artificial intelligence (AI) applications and great progress has been made in recent years following the introduction of convolutional neural networks (CNNs) [1
]. Morphological disciplines such as pathology are likely to benefit from the development of specific AI, and publications on the use of CNNs in pathology as well as neuropathology have begun to appear [3
]. An AI-supported workflow in pathology can already be envisioned.
Immune checkpoint markers are of special interest in cancer research because they may represent powerful new therapeutic targets, as suggested by the significant progress made, especially in the field of melanoma [5
]. Microvascular proliferation is one of the essential hallmarks of glioblastoma [10
]. It is a more specific and reliable sign of malignancy than necrosis, another key morphological feature that distinguishes glioblastoma from WHO grade III glioma. While it is not only endothelial cells that constitute the cellular substrate of microvascular proliferation [11
], recent reports have shown that CD276 (B7-H3), an immune checkpoint marker of prognostic significance [14
], is strongly expressed by abnormal endothelial cells in various cancers including glioblastoma (Figure S1
]. The inclusion of immunohistochemical (IHC) data has become a standard for cancer diagnostics and research.
Given the complex morphological characteristics of human tissue biopsies, their analysis requires an ever-increasing amount of computing resources and advanced learning algorithms. The arrival of more advanced deep CNNs in recent years has made the analysis of whole-slide tissue sections possible. For example, deep residual networks (ResNet) [20
] introduced skip connection to eliminate singularities and alleviate the vanishing gradient problem, which enables training of CNNs that are hundreds and even thousands of layers deep. One of the widely used ResNet models is ResNet-50, which is 50 layers deep and has been used as the reference when comparing network performances. Even more complex architecture such as Xception [21
] integrated skip connection with depth-wise separable convolution and achieved superior performance in comparison to its predecessors.
On the basis of advanced deep learning models, we have developed an integrated system, termed PathoFusion, which is an AI-based platform for marking, training, and recognition of pathological features in whole-slide images (WSIs). We have used tissue sections obtained from glioblastoma cases to evaluate the system. Adjacent tissue sections were processed for hematoxylin and eosin (H&E) staining and IHC (CD276), respectively, and scanned. PathoFusion is designed to meet three goals: (i) efficient training of convolutional neural networks to recognize key pathomorphological features in routine H&E-stained, scanned histological slides; (ii) improved model generation and increased effectiveness of feature recognition, thus requiring fewer physical cases than conventionally needed for neural network training; and (iii) establishing a method that allows the inclusion of immunohistochemical (IHC) data in the automated analysis given the great importance of immunohistochemistry in contemporary slide-based analyses.
Morphological diagnostic work in pathology, and especially in neuropathology—given the great structural complexity of the nervous system—has elements of an art. In fact, “painting with words” is a skill taught for the writing of useful microscopic reports that convey a synthesis of complex visual information, requiring both creativity and imagination for their generation. It may seem counterintuitive, therefore, that an AI should be capable of acquiring the equivalent of typical human skills, but similar developments are occurring in other fields, in- and outside of medicine—in software programming and even music, to name a few.
The PathoFusion framework introduced here provides new tools that may facilitate the transfer of pathomorphological feature detection knowledge to machines. The framework is suited for the analysis of pathomorphological hallmarks in whole histological slides, as also demonstrated in Video S1, allowing the development of a specific (neuro)pathological AI. Both marking and training times can be reduced when using the integrated system, accommodating the time constraints relevant to busy clinical consultants. Expert input is the key limiting factor, since the marking of pathological features (“ground truth”) cannot be delegated to less well-trained observers. On the other hand, a comparatively small cohort of human biopsies was required to carry out the necessary training, which resulted in high testing performance. Being able to train neural networks effectively on the basis of a relatively small number of cases should be useful in many scenarios.
The histological slides used in pathology and neuropathology can be scanned and converted into digital images. Those images contain an exceptionally rich variety of structural detail and staining patterns. The file size of these images is extremely large (gigabytes), which makes them very challenging to process using conventional methods. However, in all other respects, the images used here are comparable to the digital photographs that are widely employed and have become a predominant focus of AI research. CNNs have been found to be particularly suited for the analysis of images.
A number of earlier studies have demonstrated the utility of repurposing CNN models pretrained on natural image collections such as ImageNet for medical image analysis [27
]. This approach is referred to as transfer learning and has worked well in some cases where a close relationship exists between the source and target domains, but failed in other instances where knowledge learned from the source domain was less general across sub-domains [29
]. However, our study obviates the need for transfer learning through the use of a dual-path CNN model utilizing bifocal image tiles as input. Our model achieved better recognition performance than popular deep learning models, including ResNet-50 and Xception, pretrained on ImageNet, thus eliminating the need for resource-intensive pretraining.
The prognostic marker CD276 was chosen as a proof-of-concept example, allowing our AI system to validate its recognition ability autonomously by producing a fusion heatmap that demonstrates the overlap between a morphological feature, microvascular proliferation, and CD276 immunoreactivity. The specific task of mapping CD276 to a subset of blood vessels, and endothelial cells of abnormal tumor blood vessels in particular, can also be performed by a human observer, which is why this marker was chosen for machine recognition because a widely accepted feature is more convincing. A human observer may even be able to perform systematic recognition of a morphological feature in an entire histological section, but it would be a very tedious and time-consuming exercise. Interestingly, CD276 labeling in the present study was strong in tumor vasculature, as expected, but not limited to it when using the Ventana automated tissue staining system. We believe that the CD276 labeling noted when using this staining system deserves further analysis, which is beyond the scope of this manuscript. Extravascular CD276 labeling, which appeared to be partly cellular, is very interesting with regard to the function of CD276 as an immune checkpoint molecule.
Following training, our BCNNs reliably identified the key morphological features that they had been trained to recognize with AUC performances of 0.985 and 0.988 on H&E and immunohistochemical images, respectively. This formed the basis for the ability to correlate the occurrence of individual pathomorphological features with the tissue expression of CD276. Notably, the close association between CD276 and microvascular proliferation in GBM was faithfully reproduced and visualized by the heatmaps, permitting high-resolution (40× primary magnification) qualitative as well as quantitative morphological analyses of complete histological sections. The method presented here allows quantification of the occurrence of key morphological features and simultaneous matching of immunoreactivities (or other molecular histological data such as in situ hybridization results) to those features. This has not been possible before and may harbor significant potential for brain mapping projects (e.g., Allen Atlas). Another important quality of the PathoFusion framework consists of the independence of its prediction and fusion processes, which do not require human intervention once the BCNN model has been properly trained. The framework is expected to provide a comparable performance on other tumor types and also on non-neoplastic pathological lesions, provided that a qualified observer (experienced consultant) performs the marking, thus establishing the relevant ground truth which is necessary for CNN training.
There are several methods available to facilitate slide diagnostics—for example, Digital Slide Archive is a web-based interactive system for annotating whole-slide histopathology images which shows a similarity in function to the labeling module of PathoFusion; Cytomine is a machine learning-based open-source platform for collaborative analysis of multi-gigapixel images, but it requires users to provide customized scripts for image analysis. Distinct from Digital Slide Archive and Cytomine, Digipath is the product of a company that provides hardware (e.g., a scanner) as well as online training courses. Compared to all of these, PathoFusion is a light-weight automated framework for the analysis of multi-modal histopathology images, which provides functions ranging from annotation, training and whole-slide image detection to cross-modality quantitative analysis. PathoFusion is also flexible and can be integrated in existing annotation systems, such as Digital Slide Archive, or enhance existing hardware solutions, e.g., a scanner that is capable of detecting tissue structures as well as producing heatmaps for abnormalities.
The present study has the following limitations. Firstly, we acknowledge that training for the recognition of some pathomorphological features could be improved further by creating more subcategories as well as carefully defining any similar-appearing morphologies (mimics that can cause diagnostic pitfalls), e.g., geographic necrosis with and without bleeding, and palisading necrosis vs. tumor cell palisades without necrosis. Secondly, many additional pathomorphological features could be added to the training in order to be fully prepared for possible confounding pathological signs of unrelated but co-occurring diseases. Thirdly, the quantifications performed in this study were carried out for formal reasons and are not necessarily biologically understood. For instance, some of the novel CD276 immunohistochemical results such as lower-level CD276 expression outside of the tumor vasculature when using the Ventana system deserve additional study.