Towards Explainable Machine Learning from Remote Sensing to Medical Images—Merging Medical and Environmental Data into Public Health Knowledge Maps

Abstract

Both remote sensing and medical fields benefited a lot from the machine learning methods, originally developed for computer vision and multimedia. We investigate the applicability of the same data mining-based machine learning (ML) techniques for exploring the structure of both Earth observation (EO) and medical image data. Support Vector Machine (SVM) is an explainable active learning tool to discover the semantic relations between the EO image content classes, extending this technique further to medical images of various types. The EO image dataset was acquired by multispectral and radar sensors (WorldView-2, Sentinel-2, TerraSAR-X, Sentinel-1, RADARSAT-2, and Gaofen-3) from four different urban areas. In addition, medical images were acquired by camera, microscope, and computed tomography (CT). The methodology has been tested by several experts, and the semantic classification results were checked by either comparing them with reference data or through the feedback given by these experts in the field. The accuracy of the results amounts to 95% for the satellite images and 85% for the medical images. This study opens the pathway to correlate the information extracted from the EO images (e.g., quality-of-life-related environmental data) with that extracted from medical images (e.g., medical imaging disease phenotypes) to obtain geographically refined results in epidemiology.

1. Introduction

In recent years, there has been a rapid increase in data available, where many data sources have now reached the status of Big Data. For several years, we have been talking about Big Data. Big Data can be described in five different ways (called the 5 V’s): “by an enormous data volume, exhibiting a wide variety of types, requiring a high velocity of data processing, a need to deal with the veracity of data uncertainty, and to transform the data into value” [1].

In Earth observation (EO), the Copernicus Open Access Hub provides complete, free, and open access to Sentinel-1, Sentinel-2, Sentinel-3, and Sentinel-5P data products. With the launch of the Copernicus Sentinels and several new national missions [2], the volume of EO data has dramatically increased, reaching up to hundreds of petabytes of data [3]: the extraction of the valuable information from these data has become a significant challenge. In December 2021, 45 million Sentinel products were acquired since the start of operations: 410 PB volume of downloads has been generated by more than 500,000 users. Recent reports [4] show that the daily average of data generated per day by the Copernicus programme is 20.2 TB divided as follows: Sentinel-1 data 31%, Sentinel-2 data 47%, Sentinel-3 data 20%, and Sentinel-5P data 2%. The open and free access to such data opens new opportunities and poses challenges to the storage, processing, and analysis of such massive data. A state-of-the-art survey of several land cover datasets for multispectral and synthetic aperture radar (SAR) sensors is provided in [5].

In the medical field, there is a large number of medical datasets available [6] linked to different diseases [7]. The only restriction is that sometimes the medical data is governed by the protection of patient data. For example, regarding the neoplastic disease that is our case of investigation in this paper, the Cancer Imaging Archive [8] is one of the most extensive and publicly available medical archives funded by the Cancer Imaging Programme, which is part of the United States National Cancer Institute and managed by the Frederick National Laboratory for Cancer Research. The dataset contains images acquired from patients with cancer of different organs (e.g., lung, prostate, liver, breast, and colon); these images were acquired by sensors/devices (e.g., magnetic resonance imaging, computed tomography, and digital histopathology). Another freely available dataset is the one provided by WebPathology [9], which contains high-quality pathology images of benign and malignant neoplasms and related entities.

In this paper, we explain how information extracted from images (whether Earth observation or medical) using a data mining system can help users better understand the content after semantic labelling. Therefore, the first investigation is made with EO images, for which there is already a validation, and then continues with the investigation on the medical images.

Here, we present a machine learning system developed for EO images (radar and then extended for multispectral). For demonstration, several cities were selected, and depending on the availability of various sensors, we acquired various images (either radar or multispectral). Finally, after the system has been operated and validated by an expert in EO, we extend it to medical images.

This article touches on many Big Data notions, such as the large volume of data in both fields. Furthermore, these data need to be interpreted and understood using machine learning techniques (e.g., data mining) and semantically labelled. These notions are further used to show a connection between these semantic classes extracted by different sensors and the two domains; we use knowledge graphs. Finally, we involve expert users in both fields to understand and explain these results. In the future, we would like to develop methods for the explainability and trustworthiness of developed machine learning techniques.

Previous studies linking remote sensing and medicine have primarily focused on correlating environmental factors with disease incidence or using remote sensing-derived variables as external inputs to epidemiological models. In contrast, our work directly transfers semantic labelling and active learning workflows originally developed for Earth observation image analysis to medical imaging. This enables a unified framework, where environmental and clinical data can be analysed within the same methodological pipeline, facilitating integrative studies at the intersection of public health and precision medicine.

In the Big Data era, we face enormous data delivered daily by current instruments and stored in archives. As a result, we need processing systems to extract knowledge from such extensive archives. This paper is organised as follows: In Section 2, we present the implementation details of the proposed workflow method, and we describe the characteristics of each dataset compiled for each set of images. In Section 3, we show the classification results obtained by applying the proposed method and discusses specific details for each individual set of images acquired by the different sensors. Further, we detail the information that can be extracted using such a method from the EO and medical images. In Section 4, we analyse the findings of the current study starting in light of existing related studies on EO and medical images. This section ends with perspectives and future work, followed by the concluding remarks in Section 5.

2. Materials and Methods

2.1. Image Processing Workflow

The system’s workflow for generating semantic classification results is based on previous work described in [10], in which a satellite image time series was analysed in terms of temporal changes. However, the methodology now includes efficient exploitation of EO multi-sensor data and concentrates mainly on urban areas [5,11,12]. The novelty of the present work is the analysis of medical images using the same methodology.

The overall processing workflow is depicted in Figure 1. The methodology comprises six steps and is used for handling and analysing EO images and medical images to create semantic maps, domain ontologies, and knowledge-graph representations. In addition, this workflow can be used to create the city models based on selected EO images or patient models under medical follow-up based on collected medical images.

The key elements in image processing are, hence, the application of Gabor filters with 5 scales and 6 orientations to EO image patches, following the configuration established by [13,14,15]. For medical images, Weber Local Descriptors with 8 orientations and 18 excitation levels were used [15,16]. These parameters were selected to balance texture sensitivity and computational feasibility, enabling direct methodological transfer across domains. Classification was performed using a Support Vector Machine (SVM) with a chi-squared kernel, one-against-all strategy, and 5-fold cross-validation. Active learning cycles involved iterative expert feedback, with domain experts labelling positive and negative examples until semantic convergence was achieved. The model training using SVM with active learning and relevance feedback, following the workflow described in our previous EO publications, is summarised in

Table 1. Parameters used for model training on EO and medical images with SVM.

Parameter	EO Images	Medical Images
Kernel type	Chi-squared kernel	Chi-squared kernel
Multi-class strategy	One-against-all	One-against-all
Regularisation parameter (C)	1.0 (default)	1.0 (default)
Gamma (kernel coefficient)	1/(number of features)	1/(number of features)
Cross-validation strategy	5-fold cross-validation	5-fold cross-validation
Active learning cycles	3–5 iterations depending on dataset size	3–5 iterations depending on dataset size

.

These steps are as follows:

• Step 1: During a preparatory phase, the experts select the dates and target areas acquired by different satellites and download them to typical archives such as the one from the ESA Copernicus hub [17] or the DLR archives [18]. Each EO acquisition from the archive has two parts: the image and the metadata. Later, these two parts are used during operations, as shown in Figure 1. Like EO images, the medical images are acquired by various devices/sensors operated by experts and stored on servers as images and metadata associated with each medical image.
• Step 2: Tile each EO image into patches (with no overlapping) and with a pre-selected size, depending on the actual pixel ground sampling distance to cover the objects on the ground (see

  Table 2. List of selected EO images with their parameters.

  Location	Instrument	Type	Mode	No. of Sensor Bands/Selected Bands	Resolution	Polarisation	Patch Size (Pixels)	No. of Patches
  Berlin, Germany	Gaofen-3	C-band SAR	SpotLight (SL)	1/1	1 m	HH	256 × 256	2080
  	TerraSAR-X	X-band SAR	Multi-look Ground range Detected (MGD)	1/1	2.9 m	HH	160 × 160	1025
  Bucharest, Romania	TerraSAR-X	X-band SAR	Multi-look Ground range Detected (MGD)	1/1	2.9 m	HH	160 × 160	4455
  	WorldView-2	Multi-spectral	-	8/3 (RGB)	1.87 m	-	100 × 100	33,930
  Vancouver, Canada	RADARSAT-2	C-band SAR	Extended high (EH)	1/1	13.5 m	HH	160 × 160	660
  	TerraSAR-X	X-band SAR	Multi-look Ground range Detected (MGD)	1/1	2.9 m	HH	160 × 160	825
  Albania and Greece	TerraSAR-X	X-band SAR	Multi-look Ground range Detected (MGD)	1/1	2.9 m	HH	160 × 160	1872
  	Sentinel-1	C-band SAR	Interferometric Wide swath (IW)	1/1	20 m	VV/VH	128 × 128	26,260
  	Sentinel-2	Multi-spectral	-	13/3 (RGB)	10/20/60 m	-	120 × 120	8281

  ). The tiling is applied in the same way for the medical images, with no overlapping between the patches and with a patch size adapted to the content of the medical image (see

  Table 3. List of selected medical images with their parameters.

  Data	Instrument	No. of Bands/Selected Bands	Image Dimension (Pixels)	Sub-Images (Pixels)	Patch Size (Pixels)	No. of Patches
  Colorectal adenocarcinoma	Optical microscopy	3/3	1024 × 768	-	4 × 4	49,152
  Lung adenocarcinoma	Optical microscopy	3/3	avg. 9688 × 9832	890 × 801	4 × 4	44,400
  Non-small cell lung cancer	Computer tomography (CT) scan	1/1	avg. 1802 × 884	1372 × 672	4 × 4	57,624

  ). The size of the patch should be adapted to the image resolution and its content, so that the patch includes (as much as possible) a single object [11,12].
• Step 3: Extract the primitive features that describe the content of each original patch. For the EO images, one applies Gabor filters with 5 scales and 6 orientations [13,14,15], while for the medical images, one applies Weber local descriptors [15] with 8 orientations and 18 excitation levels or multispectral histograms 64 bins. In the case of Gabor filtering, we extract the Gabor coefficients from each image patch and compute the means and standard deviations of each set of coefficients (in total, 5 × 6 × 2 = 60 coefficients). In the case of Weber local descriptors, the features are extracted from each patch in a set of 144 (i.e., 8 × 18) coefficients. A detailed study in the use of various primitive feature extraction methods and different values of their parameters can be found in [5,19].
• Step 4: The classification of the primitive features of each original patch is made automatically, and the patch features are grouped into clusters (i.e., “mathematical groupings”) using a Support Vector Machine (SVM) [16] with relevant feedback. The aim is to obtain a feature-based image patch classification by assigning a single semantic label to each patch using a user-oriented terminology of real-world classes. For the SVM, a chi-squared kernel is selected, and a one-against-all approach is used. The activities of the expert users are called “active learning”, referring to the interactive selection of randomly selected positive and negative examples of target classes based on a proper visualisation of the individual patches, a visual comparison of the selected patches (using for comparison the Google Earth maps in the case of EO images and reference health datasets in the case of medical case), and human expert judgements about the actual patch content.
• Step 5: Generate a set of patches that are semantically correctly labelled. This step is finished once all the given patches have been labelled. However, some patches may remain unlabelled. If the missing labelling represents problem cases, an expert user must identify the most probable class, or the data can be assigned to an unclassified class. There is already a hierarchical semantic labelling scheme in the case of EO images (see [16]). However, regarding the medical images, the experience of expert physicians is significant for defining the semantic labels allocated to the classes.
• Step 6: Interpretation of the produced results. The first data product is the semantic classification results/maps of each image—or, in the case of image time series—the corresponding change maps. The second product is domain ontology representations [5], which help users extract the information and knowledge from the images. Finally, knowledge graphs can be created to explain the entire chain of relations (starting from the data, information extraction up to the semantic classes and their relations).

To summarise the process from Figure 1, we can say the following: the proposed methodology has two branches: one for Earth observation images and one for non-EO images. In detail, our approach consists of the following steps: acquire new data from one of the available EO satellites and store the EO image products in an archive, tile each image into patches with different sizes and integrate the appropriate metadata parameters, extract the primitive features from each patch (in this case, we used Gabor filters with 5 scales and 6 orientations), group them into classes using an active learning approach by applying a Support Vector Machine with relevance feedback, and provide a semantic label for each patch. Here, a list of already defined semantic labels exists, and the user can select an existing one or define a new one. Once EO experts have completed this process, users can create semantic classification maps of the given images. Like EO imagery, the procedure for medical images starts by acquiring images from various instruments such as optical microscopy images in classical haematoxylin-eosin (HE) and computed tomography (CT) images and storing them on a server. The following steps are similar; however, the size of the patches depends on the medical image parameters. Finally, the Weber local descriptors generate the extracted primitive features. Here, compared to the EO domain, the knowledge of field experts is very important in defining the semantic classes because there are no already predefined labels. After the images have been classified, the next step is to create semantic maps of the medical images.

2.2. Dataset Description

This subsection is dedicated to selecting and describing the images used in this paper. The images are divided into two main categories, namely EO images and medical images. In the case of EO images, we considered the availability of the selected images, the diversity of locations, and the available sensors (radar or multispectral instruments). In addition, we considered the following types of images collected from haematoxylin-eosin (HE) pathological slides and computed tomography (CT) scans in terms of medical images.

The data provided by each EO sensor are the images (in GeoTIFF or JPEG2000 format) and their metadata (in XML format). These metadata contain information about the acquisition parameters of the respective image (e.g., time of acquisition, incidence angle, resolution).

Similar to EO images, medical images are also accompanied by additional information such as clinical and demographic data (e.g., sex, weight, height, body mass index, area of residence) and relevant information from medical history, including current chronic diseases, treatments, and risk factors (usually in TXT or CSV format, separate from the map data).

2.2.1. Earth Observation Datasets

From the large amount of EO data provided by different instruments [2,20], we selected six instruments that flew over the following cities: Berlin, Bucharest, Vancouver, and some cities areas between Albania and Greece, such as Corfu. We grouped the instruments into multispectral and SAR imagers. The data were received/provided via proposals, project agreements, or downloaded from the sensor imagery samples, and are sometimes subject to copyright rules, which we adhered to. From the wide variety of available and existing instruments, we selected the following ones (in alphabetical order): Gaofen-3 [21], RADARSAT-2 [22], Sentinel-1 [23], Sentinel-2 [24], TerraSAR-X [25], and WorldView-2 [26]. Table 2 shows the most important parameters of each selected instrument. More examples can be found in [27,28]. The dimensions of the EO images range from 5000 × 5000 pixels for TerraSAR-X, to 10,000 × 10,000 pixels for Sentinel-2, and even up to 25,000 × 16,000 pixels for Sentinel-1.

2.2.2. Medical Datasets

Medical images from various clinical and biomedical research fields are widely available online. For instance, an extensive collection of radiological medical images created for the validation of AI methods can be found in [29]. Unfortunately, these images are subjected to sensitive personal data protection issues, so many clinical and demographic data and clinical history were unavailable [29]. The first two sets of images were acquired by optical microscopy, while the third set of images was acquired by computer tomography (CT) scan. The number of selected images for demonstration depends on the diversity of medical cases identified in each set of images. Table 3 shows the essential parameters of each selected dataset. All medical images used in this study were fully anonymized prior to analysis. Public datasets (WebPathology and TCIA) were used under their respective open-access policies. A small number of additional de-identified pathology images were collected retrospectively under institutional IRB approval no. 1575/02.02.2024. No patient identifiers were accessible to the research team, and no additional informed consent was required.

(1)

The first dataset of images is from 8 patients with colorectal adenocarcinoma; the total collected images at different magnifications (5×, 10×, 20×) are 180. They were collected using an optical microscopic device equipped with an RGB camera, being selected by a medical expert based on their expertise to include typical normal structures of the colon wall and pathological structures commonly found in colon adenocarcinoma. Usually, the prototypic colorectal cancer is a well-to-moderately differentiated adenocarcinoma consisting of tubular, anastomosing, and branching glands in a desmoplastic stroma. The surface component may be ulcerated or show papillary or villous architecture. In addition, residual adenoma is often present at the edge of the tumour [30,31].

(2)

The second dataset of images is from the patients with lung tumours; there are 11,210 CT images and 25 pathology slices collected from 6 patients. From these, we selected 10 images from 2 patients with lung adenocarcinoma. Usually, lung adenocarcinomas show an admixture of many architectural patterns such as acinar, papillary, micropapillary, lepidic, and solid growth patterns [32,33].

(3)

The third dataset of images is extracted from a collection of 52,072 images from 422 patients with non-small cell lung cancer (NSCLC) [34]. For these patients, pre-treatment CT scans lung tumours; manual delineation by a radiation oncologist of the 3D volume of the gross tumour volume and clinical outcome data are available in [31] for the Lung1 dataset. Typically, lung cancer pathology can identify two groups of cancer cells: small cell lung cancer (SCLC) and non-small cell lung cancer (NSCLC). Then, the last ones, the NSCLC, are divided again into squamous cell cancer (SCC), large cell cancer, and lung adenocarcinoma. Finally, in situ (ISA) and invasive are the two types of lung adenocarcinoma.

3. Results

This paper is based on an “intelligent system”, which starts from the data and reaches its knowledge [35]. The system has two stages: (1) the first stage is to transform the data (in our case, the images) into actual information. The data is in the form of images, where after several processing steps (see Figure 1), each patch being tiled from the image comprises an associated feature vector and its corresponding metadata (e.g., geographical position, acquisition data, and instrument type). The image is now converted into information. Then, in the second stage, the contoured image is transformed into knowledge about that image. Such transformation is triggered by the user who, with the help of his knowledge, classifies and groups the features of each patch into classes (categories) and then gives them meaning by defining semantics. One obtains, thus, a semantic catalogue for each domain [36]. This type of system is mono-directional, in which the information flows from the machine to the human. When all the results are stored in a database, each patch is associated with metadata, features, semantic labels, and user ID. All this information/knowledge can be used to make various queries in the database (in the opposite direction), or different knowledge representation models can be defined.

3.1. Semantic Classification Based on the Extracted Information

3.1.1. Earth Observation Images

This sub-section explains the classification results obtained for each target area, considering that various instruments have acquired the data. The results show that the number of obtained classes depends on the instrument type, resolution, and patch size. For EO data, we used for comparison the same type of features, namely, the results of Gabor filtering, where we extracted 60 coefficients (see Step 3 in Section 2). A detailed analysis of the impact of a selected patch and its resolution on the number of attainable classes has already been published in [34,35].

For a set of characteristic EO images, we selected from the available data four cities/areas, for which we could obtain simultaneous acquisitions of several instruments (SAR and multispectral). The investigation could be grouped into three categories: (1) instruments flew over the same city or a subarea of the city (e.g., Bucharest, Romania); (2) the acquired images mainly cover the same area of the city (e.g., Berlin, Germany) or different subareas of the city (e.g., Vancouver, Canada); (3) the acquired images cover the city together with its surrounding areas, up to the coverage of a neighbouring island (e.g., Corfu, Greece).

Bucharest [37] is analysed through the acquisitions made by TerraSAR-X and WorldView-2. After applying the method described in the Materials and Methods section, the semantic classification results show that the number of identified classes in the two images is the same (see Figure 2). However, in the case of multispectral data, the classification accuracy is higher [38]. The higher accuracy is due to (a) the better resolution of the WorldView-2 instrument (1.87 m instead of 2.9 m) and (b) the smaller patch size (100 × 100 pixels instead of 160 × 160 pixels). The smaller patch size shrinks the size of the classified object, thus avoiding a mix-up with other objects.

For Berlin (see Figure 3), the overlapping of the two images is partial, but parts of the city centre (e.g., Tiergarten Park and the Brandenburg Gate) are common. For comparison, we analysed two images acquired by similar types of SAR instruments. When we compare the classification results obtained by GF-3 and TerraSAR-X, we notice that even if GF-3 has a better resolution, the number of attainable surface cover classes is higher for TerraSAR-X due to the actual content of the image. The TerraSAR-X image also covers the northern part of the city, richer in content, and new categories appear, such as bridges, channels, etc.

For Vancouver (see Figure 4), we used images of two SAR instruments (TerraSAR-X and RADARSAT-2) covering separate parts of the city with different resolutions. The images show that there are situations in which the instruments cannot cover the same area of a city but only parts of it. However, this approach is helpful for repeated coverage of a city and its classification.

The images in Figure 5 were acquired by three different instruments (two SAR instruments and one multispectral instrument). These images were selected to obtain a reduced time interval between their acquisitions because a simultaneous acquisition was not possible due to the satellite revisiting times and—in the case of the multispectral instrument—due to cloud coverage of the investigated area.

When we analyse the results of the semantic classifications, we can observe the following: (a) comparing Sentinel-2 with Sentinel-1 (with resolutions of 10 m versus 20 m) reveals that the vegetation classes can be distinguished much better due to the high resolution of the images and the additional multispectral bands (Sentinel-2); (b) comparing TerraSAR-X with Sentinel-1 (resolutions of 3 m versus 20 m) shows a more refined set of semantic classes found by the former instrument. Here, the patch contains more than 80% of the most prominent semantic class, compared to the Sentinel-1 case, where we have a more mixed content in the patch.

Selecting a specific class of interest, such as aquaculture (i.e., fish cages in coastal seas), shows that identifying this category depends on the instrument’s resolution. This aquaculture class can be seen from three semantically classified images acquired with a resolution of up to 10 m (e.g., TerraSAR-X, Sentinel-2).

3.1.2. Medical Images

After the method has been successfully validated for EO images, now in this sub-section, we validate the method/system for several medical images acquired from humans with various sensors (as single-band or multi-band). Each image is explained based on the semantic classification results. We choose the most representative images per case for a demonstration from each set of medical images. Similar results are obtained for the other images from the datasets.

For medical data, we used for the comparison of each dataset the same type of features (see Step 3 in Materials and Methods). Based on the knowledge gained in the analysis of EO images, in the case of medical images, we tried different patch size (see Step 2 in Materials and Methods) for each set of medical images in order to better capture their contents and to describe the semantic classes as precisely as possible. Comparative results obtained using different patch sizes are presented in the following figures for each set of medical images. However, in the future, a more detailed study in this direction is necessary to identify the optimal patch size for each individual case.

To compare the results obtained with the method presented in Materials and Methods, we use an unsupervised method based on Latent Dirichlet Allocation [42], developed for topic representation of EO images during the H2020 ExtremeEarth project. Here, the method is applied to the medical images (with a patch size of 4 × 4 pixels) selected from the three datasets in order to compare the results with the one from Materials and Methods.

The first medical dataset analysed is that of the colorectal adenocarcinoma, acquired using optical microscopy. From the dataset, we chose the most representative images. For this set of images, the patch size is reduced by adapting the patch size to the resolution of the image and the content of the image. We compared the classification results using four different patch sizes, 48 × 48 pixels (with a total number of patches equal to 336), 24 × 24 pixels (with a total number of patches equal to 1344), and 16 × 16 pixels (with a total number of patches equal to 3072). The results of these different patch sizes (for the image with the ID KS-172) are presented in Figure 6 and Figure 7. For comparison, the previous results are compared with the LDA method with the patch size of 4 × 4 pixels (with a total number of patches equal to 49,152).

With a larger size of the patch, a larger area of the image is incorporated; therefore, more objects can be included in it (see Figure 6 and Figure 7), which is not desired for a better classification. By reducing the size of the patch (see Figure 8), we can obtain a better classification; therefore, the patch includes only one class.

In Figure 6 and Figure 7 (top part), class 5 around class 1 is the adventitia tunic around a blood vessel.

Regarding Figure 6, Figure 7 and Figure 8 (bottom part), when class 1 is inside the blood vessel, it is the lumen (an empty space); when it is in the submucosa, it represents the loose connective tissue. For class 2, when it is around the blood vessels, it represents their wall (tunica adventitia, muscular, intima), and when it is outside the crypto areas, it is the connective tissue.

Based on the results obtained in Figure 6, Figure 7 and Figure 8 and the expert observations, it can be seen that for a more detailed/finer classification, a smaller patch is required in order to retrieve the small areas/cells.

Further, the other selected images for evaluation (KS-031, KS-032, KS-040, KS-042, KS-145, KS-165, and KS-168) were classified. The results were similar with the one from Figure 7 (bottom part) and are not presented here due to space limitations.

Analysing the results from the above figures (Figure 6, Figure 7 and Figure 8), we can see that the proposed method works very well when the size of the patch is smaller. In the unsupervised case, the results are not as good (see Figure 8). What is very important in the case of medical images, due to the complexity of the image content, is that an expert (e.g., a doctor) and his knowledge in the field are needed in order to classify the image and group the patches into the classes, and further to provide the semantic meaning to the retrieved classes.

In the histopathology image of Figure 6 (top left), there are three structurally distinct areas: the structurally normal glandular zone, adipose tissue with blood vessels surrounded by connective tissue, where disturbances in organisation are identified (Figure 6, top right). In the case of the blood vessels, a distinction was made between the vascular walls and the interior of the vessel, with a clear differentiation within the latter between the area occupied by blood cells (predominantly erythrocytes) and the empty intraluminal space. There is an error zone, where classes related to a quasi-homogeneous pinkish area representing a fibrinogen accumulation zone are mixed. Small-sized areas are incorrectly identified as glandular tissue in perivascular regions and as vascular wall areas in the dominant glandular region. Due to their small size, it is difficult to achieve continuity of the vascular wall. Dividing the same image into 24 × 24 patches led to some errors (Figure 6, bottom right). The vascular wall was incorrectly identified as connective tissue, and the vascular lumen as adipose tissue. Additionally, there were a few patches where connective tissue was misidentified as glandular tissue. Reducing the patch size to 16 × 16 (Figure 7) allowed for the identification of vascular structures but still did not distinguish between the vascular wall and the blood-filled lumen. Moreover, the periglandular areas identified as vascular structures expanded. A positive aspect is the ability to differentiate between loose and dense connective tissue. However, distinguishing between adipose tissues and empty intraluminal spaces remains impossible. Unsupervised learning identifies a larger number of classes (Figure 8), but there are significant limitations regarding the distinction of areas with different histological characteristics. Confusions arise between areas of connective tissue and vascular wall areas, even inside the intraluminal blood clot, between the glandular lumen and areas of connective tissue, and so on.

Figure 9 displays images selected from the entire dataset from which we are taking the sub-images for our investigation. These areas are marked with a green rectangle and annotated from 1 to 19. The index of each sub-image is marked in the lower left corner of the green rectangle.

We compared the classification results using four different patch sizes, 48 × 48 pixels (with a total number of patches equal to 288), 32 × 32 pixels (with a total number of patches equal to 675), 16 × 16 pixels (with a total number of patches equal to 2750), and 8 × 8 pixels (with a total number of patches equal to 11,100). The results of this comparison is performed for a sub-image with the id 15 (see Figure 10 and Figure 11), and we can say that the best results, in terms of the number of retrieved classes and visual comparison (given by an specialist doctor) between the original image and classified one, are those obtained for the case when the patch was 8 × 8 pixels.

For the same sub-image id 15, the method in [42] is applied with different number of classes/topics (e.g., 6, 10, and 12) with a patch size of 4 × 4 pixels (with a total number of patches equal to 44,400). The results are presented in Figure 12.

Similar results are obtained for the other representative sub-images selected from Figure 9 and marked from 1 to 19. Figure 10 (top left) shows the image of a lung parenchyma with inflammatory infiltrate, in which a blood vessel and a few airways are being destroyed. The analysis of the image, starting from patches of 48 × 48 pixels, relatively correctly distinguishes these areas, with a few errors related to some areas of the lung parenchyma richer in connective tissue, which are mistakenly identified as vascular walls. It is noteworthy that a distinction can be made between the empty intraparenchymal areas that may correspond to airways (bronchi of various orders). Figure 10 (bottom, left) represents the result of the analysis of images with smaller dimensions (32 × 32 pixels).

It is notable that there is a clear distinction mentioned above between the empty intraparenchymal space and the empty space at the edge of the area occupied by the tissue sample on the slide. The only notable error remains the identification of some areas of connective parenchymal tissue as being vascular walls. Figure 11 (above, left) represents the result of the analysis using patches of size 16 × 16 pixels. The error related to the incorrect identification of areas of connective parenchymal tissue as vascular wall persists even in the wrongly identified areas, as can be seen in Figure 11 (top, left).

Figure 11 (bottom, left) represents the processing of the image using patches of 8 × 8 pixels. The details are more numerous, and the errors are smaller. Various types of classes could be identified, including areas of connective intraparenchymal tissue. The areas where there are errors related to the identification of intraparenchymal connective tissue as the vascular wall are small in size. The results of unsupervised learning become increasingly inefficient as the size of the patch used is smaller and the number of classes is larger, as shown in the images from Figure 12 (right).

The third medical dataset analysed is that of lung adenocarcinoma, which was acquired using CT. The patients, whose CT images are available in the third dataset and analysed here, are the same as the patients in the second dataset. From this dataset, we chose the most representative images, which contain different anomalies (suspicious or cancerous areas) and need to be analysed.

The following figures show images selected from the entire dataset and used in our investigation. We compared the classification results using four different patch sizes, 48 × 48 pixels (with a total number of patches equal to 140), 32 × 32 pixels (with a total number of patches equal to 315), 24 × 24 pixels (with a total number of patches equal to 609), and 16 × 16 pixels (with a total number of patches equal to 1333). The results of this comparison are given in Figure 13 and Figure 14, and we can say that, in the case of CT images, the best results are for the classification with a patch of 16 × 16 pixels, which not only has the highest number of classes retrieved by the method but also benefits from the expertise of the user (in this case, the specialist doctor).

Based on this result, Figure 12 shows the analysis of two other CT images: one collected from a patient with no diseases and one from a patient with a cancerous tumour. For the same set of CT images selected from the dataset, the unsupervised method [42] is applied (number of classes/topics is equal to 12) with a patch size of 4 × 4 pixels (with a total number of patches equal to 30,800). The results are presented in Figure 13. The training of the algorithm on computed tomography images allows the introduction of multiple classes representing anatomical structures and regions of the thorax with relatively few errors (Figure 13, top left). Figure 13 (top right) shows a CT image of the transverse section of the thorax at the level of T5, where the right lung is identified and a large-sized tumour is in the middle. For proper preparation, these tumours can be delimited without errors of misidentifying other structures or tumour tissue. Keeping the same number of classes (of anatomical structures) but reducing the size of the patches results in improved delineation of these areas (Figure 13 and Figure 14, bottom left). Reducing the patch size to 24 × 24 and 16 × 16 allows for refinement in the definitions of selecting anatomical regions in the thorax (Figure 14). Patches with a size of 16 pixels are optimal for delineating structures and anatomical regions in sections where the tumour is absent (Figure 15, top) or present (Figure 15, bottom). Also, it is noted that errors in identifying structures are greatly reduced, regardless of the sizes of the patches used in the analysis. Unsupervised learning using 4-pixel patches is not effective, neither in cases where the tumour is absent nor in those where it is present (Figure 16), even though the number of classes is the same as in supervised learning.

3.2. Knowledge Representation

Daily, through the Copernicus Programme, a huge amount of data (e.g., 12 TB of satellite images/day) is acquired, much of which remain unexplored in archives. This is a typical case where methods/algorithms are needed to explore these archives and extract useful information. One of the methods that can be applied for this purpose is the one based on active learning, with human interaction from a user who can guide the search and the semantic annotation in the desired direction by extracting information necessary for a given application [5].

In this section, we use the output of the proposed method (the semantic classes) in order to efficiently exploit the information/knowledge from the semantic labels and to analyse the relation between the different labels in a model of each observed target area. Based on the collected information, Figure 17, Figure 18 and Figure 19 show built models that include the semantic classes, which can lead to the creation of the model of the city that considers two major aspects: How green is the city? and How industrialised/populated is the city? Figure 17, Figure 18 and Figure 19 show three examples for the following cities: Bucharest (Figure 17), Berlin (Figure 18), and Vancouver (Figure 19).

Similar to the creation of the city model, a model of the patient or patients from the same city/region can be created. The next figure (Figure 20) shows such a model of two patients with cancer from the same city.

4. Discussion

4.1. Machine Learning Systems and Semantic Labelling

Annotation systems have been proposed for multimedia data for the first time. An example is the online LabelMe tool [43] for annotating pictures. Several research groups created systems for image understanding and annotation [44], including web engines in remote sensing. In the EO domain, most of the well-known systems are based on machine learning techniques (e.g., content-based information retrieval [45], data mining [46], deep neural networks [47], and a hybrid semi-supervised technique [48]. We would like to mention here: LandEX [49], GeoIRIS [45], EOLib [50], and CANDELA [27].

In recent years, with the advancement of artificial intelligence, especially machine learning techniques [51], these techniques have become more visible in the medical field [52], with the main direct goal of better diagnosis and diagnostic error elimination [53]. Based on our knowledge, in the medical field, most of the existing systems use segmentation techniques to separate different objects/classes in an image or use a content-based information retrieval approach to identify similar images. However, a system like the one we proposed for medicine based on an active learning approach (using a data mining technique) is not yet available.

The present study was designed as a proof of concept to explore the transferability of semantic labelling and active learning techniques from the field of Earth observation to medical imaging. Unlike deep learning-based approaches that dominate current medical image analysis, our system emphasises interpretability and integration with environmental datasets via knowledge graphs. While our results demonstrate feasibility, we do not claim superiority over mainstream methods. A comprehensive benchmarking study comparing our approach with state-of-the-art deep learning algorithms will be an important next step once larger and clinically curated datasets become available. Our approaches are systematic in terms of medical cases but are exhaustive in terms of testing the capabilities of the methodology. We found that this method is quite efficient at the histological level and remarkably efficient for delineating organs, structures, and anatomical regions based on CT images. This CT image analysis capability allows the methodology to be combined with radiomics, as we will show a bit further on.

There are already predefined semantic annotation schemes for labelling the EO data in remote sensing (including radar and multispectral images). One of them is presented in [5] and is further applied in this paper. From about 300 radar images, 354,000 patches were generated, to which 53 semantic labels have been assigned, using a data mining system and the knowledge of an expert user in the field. To this number of labels can be added the cases where two or more semantic classes are assigned to a patch in a multi-labelling approach (e.g., channels and high-density residential areas), plus the class Unclassified, which is assigned to the patches for which no suitable label was found.

The number of semantic classes identified in an EO image varies between 10 and 20 classes; this depends on the image’s location and content. From our experience [1,28,54], a more significant number of semantic classes are being identified in metropolitan areas.

In contrast, the number of semantic classes in the medical field is smaller but with much more complex image content. In this case, in the definition of semantic classes, the knowledge provided by an expert in oncological diagnostics is critical.

Graph-based representations of data, their interrelationships, and their linking with higher-level knowledge have already been proposed by several authors [55,56,57]. Typical applications are graph data for graph visualisation, graph matching, graph patterns and substructures, pattern and grammar learning, decision trees, and graph mining.

An essential step in using graphs for image content interpretation was the introduction of knowledge graphs by Google [58]. We can assume that the big commercial search engines (such as Bing and Wikipedia) rely on knowledge graphs in their search routines when they combine higher-level knowledge expressed as ontologies with digital information such as images.

4.2. Knowledge Graphs and Interpretability

The scientific goal of knowledge graphs [59] is to select image data combined with additional information and generate higher-level interpretation results from them. There are already several available knowledge graphs, among which we would like to mention Google Knowledge Graph, Wikidata, DBpedia, and YAGO [60]. A survey on knowledge graphs has been published by [61], while a detailed scientific paper dealing with knowledge graphs for remote sensing images has been published by [62]. A study paper for health knowledge graphs built from medical records has been presented in [63].

Although machine learning methods have reached a high level of maturity in many fields (e.g., computer vision, remote sensing), there is still a need for the explainability and trustworthiness of developed methods. Recently, in this domain, there have been some efforts to create explainable machine learning (XML) models, which are part of the larger domain of explainable artificial intelligence [64,65,66]. The three concepts that govern XML are interpretability, transparency, and explainability.

A first step in expanding the knowledge from EO to medicine was taken within a previous research TELEIOS project [67], where a data mining “laboratory” system was tested successfully with other types of images rather than the EO images for which it was developed. The image was a microscopic tissue of the stomach collected from a herbivorous animal. The RGB image had the size of 3136 × 2352 pixels that was tiled into 1131 patches with the size of 80 × 80 pixels. Based on this system, six semantic classes were extracted.

4.3. From AI for EO to AI for Health: The Case of Lung and Colorectal Cancers

Such an approach that is proposed in this paper for the medical field is an unexplored opportunity as it emerged when we compared the number of publications in remote sensing with those in medicine. This study was conducted on the IEEE Xplore website, an extensive database of articles with a remarkable impact, by querying based on keywords related to the two fields. The search was performed for keywords such as “machine learning” and “remote sensing” or “medical images” (see Figure 21a). Another comparison was made between “machine learning” and “Earth observation” or “Computed tomography” (see Figure 21b). Finally, from the available machine learning techniques, our proposed and selected method is the “active learning” method that was compared in the number of existing publications on “Earth observation” and “Computed tomography” (see Figure 21c). The main observation of this study is that the research in the field of Earth observation is much more developed than in the medical field (in several available publications). This conclusion is an additional reason that motivates us to find solutions applicable to the medical field.

A state-of-the art in the field of AI4EO (artificial intelligence for Earth observation) is already mentioned in many papers published in recent years in the field of remote sensing; for this reason, in this paper, we will not insist in this topic, but we will mention a series of articles on this topic [68,69,70].

AI and deep learning have been applied in computer-aided diagnosis and research, allowing for advanced analysis and learning through simulations of the human brain [71]. The applications of AI in lung cancer are diverse and include tasks such as segmentation, detection, cell counting, and gene mutation prediction, demonstrating their potential in improving various aspects of lung cancer diagnosis and treatment [72].

In the context of lung diseases, there is a high global burden that encompasses a spectrum of diseases, including cancer, tuberculosis, idiopathic pulmonary fibrosis, and COVID-19 [73]. Among these, lung cancer stands out as a major cause of cancer-related death [74]. Furthermore, AI has shown promise in the quantification of imaging biomarkers, which are essential for the diagnosis, risk stratification, and assessment of treatment responses in lung cancer patients [73], while other emerging applications include areas like multimodal data analysis, 3D pathology, and transplant rejection prediction [73]. In thoracic imaging, AI has played a pivotal role, with deep learning techniques driving significant progress [75]. Deep learning (DL) algorithms have been developed for various lung cancer-related tasks, underlining their importance in the domain [76]. These algorithms have had a profound impact on medical image analysis, especially in the context of lung imaging [77]. However, it is important to be aware of possible bias in supervised deep learning algorithms used for CT lung nodule detection and classification, highlighting the need for ongoing research and refinement in this field [78]. The potential of DL in lung cancer extends from screening to prognostication, showcasing its versatility and significance [76].

On the other hand, colorectal diseases, with a particular focus on colorectal cancer (CRC), are a significant health concern with high incidence, morbidity, and mortality rates, as highlighted in several studies [74,79,80,81]. AI plays an expansive role in the diagnosis, treatment, and prognosis of CRC being applied in various fields, including imaging, endoscopy, and pathology [82]. CRC is the second most common cancer in women and the third most common in men [83]. This prevalence is accompanied by increasing incidence rates, which present growing diagnostic challenges in the field of colorectal cancer [83]. Various diagnostic methods—including excreta and blood tests, colonoscopy biopsy samples [84], computer-aided endoscopy, and medical imaging—are employed in the diagnosis of colorectal cancer [74], since early and accurate diagnosis is of paramount importance in improving survival rates, increasing cure rates, reducing mortality, and minimising medical costs associated with the disease. Thus, in the context of treatment decisions for colorectal cancer, the preoperative assessments of a large variety of biological variables [81] hold significant importance as they guide personalised treatment strategies for individuals with colorectal cancer [79]. Neoadjuvant chemoradiotherapy is the standard treatment for Locally Advanced Rectal Cancer (LARC) [85], while early evaluation of colorectal cancer liver metastasis (CRCLM) is crucial for determining treatment strategies and improving survival outcomes in patients who are beyond the locally advanced disease [86]. While CT is also utilised in the detection of distant metastases [81], MRI serves as a crucial diagnostic tool, offering accurate evaluation of tumour location, local staging, restaging, invasion depth assessment, localization of radiotherapy, prediction of chemotherapy response, detection of high-risk factors, and prognosis assessment [81,87]. To evaluate the diagnostic accuracy of AI in detecting lymph node metastasis in CRC, systematic reviews have been conducted with a focus on radiomics and deep learning in CT/MRI studies [88]. The results indicate that while radiomics exhibits high heterogeneity, deep learning, although less prevalent, proves to be effective. In particular, AI models, with a special emphasis on deep learning, demonstrate the potential for accurate prediction of lymph node metastasis in CRC [88]. Additionally, DL techniques have emerged as valuable tools in histopathology, offering diagnosis assistance in CRC diagnosis and the ability to predict molecular phenotypes, prognostic features, and even assess the tumour microenvironment, all of which contribute to a deeper understanding and improved management of colorectal diseases [83].

4.4. Limitations and Perspectives of AI Applications

AI tools for lung cancer pathology have evolved, incorporating hand-crafted and deep learning-based unsupervised features [73]. This evolution aligns with the emergence of digital pathology (DP), driven by advancements in computational power and whole-slide imaging technology [72]. DP, coupled with AI tools, is aiding pathologists and pulmonologists in various aspects of their work, from remote support to routine diagnosis [73]. The synergy between AI and pathology continues to advance the field, with promising implications for the accurate diagnosis and treatment of lung cancer.

Recent developments in the field show promise for CRC pathological analysis, emphasising the importance of accurate diagnosis and treatment planning [84]. Goals in this context include the removal of pre-cancerous polyps and reducing risks associated with unnecessary polypectomies, highlighting the need for precision in medical interventions [80]. Artificial intelligence, in particular, is emerging as a significant advancement in the field, with the potential to revolutionise clinical decision-making and improve outcomes for patients with colorectal diseases, especially CRC [89]. There is a growing emphasis on AI leveraging in colonoscopy, specifically in the areas of polyp detection and characterisation [80] or image mining and analysis for new insights into tumour biology, which is directly impacting clinical practice and decision-making [89].

Studies in this domain have explored various model features, including gland segmentation, tumour classification, microenvironment characterisation, and prognosis prediction [84]. While other models show promise, many are still in their early stages of development. New developments in CRC research are focusing on morphological biomarkers, dynamic evaluation of metastases, genetics, and the role of the liver–tumour interface, all of which contribute to a deeper understanding of tumour biology [89].

4.5. Limitations of This Study

The reason why, within our study, there are identification errors when using pathology slides with conventional haematoxylin–eosin staining is the presence on the slide of areas where the pink–violet hues are very close in tone. These areas are physically smaller in size compared to the others (usually with less clear contours). Usually, errors occur interspersed in extensive areas that actually correspond to other structures. The level of detail is dependent on the size of the patches used in supervised learning.

Moreover, errors in delimiting small vessels and airways can occur following tissue processing for histopathological analysis. In pathology, the processing of tissues for optical microscopy observations involves several steps, each of which can induce changes (alterations) in blood vessels, airways, and other tissue structures. The main processing steps and the associated changes are as follows:

• Fixation using chemicals such as formalin to stabilise proteins and cellular structures to prevent autolysis and degradation. This process can induce artefacts by coagulating proteins and altering the appearance of blood vessels, causing contractions or stiffening.
• Dehydration of tissue samples with increasing concentrations of alcohol, which can lead to a reduction in the volume of blood vessels and small airways and their collapse, thus altering their microscopic appearance.
• Clearing, in which tissues are passed through clearing solutions (usually xylene or toluene), making them transparent and ready for paraffin infiltration. This can lead to additional alterations, such as vessel collapse and changes in the spatial relationships between structures, including lung airways.
• Paraffin infiltration leading to mechanical artefacts, such as distortion or displacement of tissue structures, including blood vessels. This process can make vessels and airways appear more rigid and collapsed than in their natural state.
• Microtomy or fine cutting of thin sections for microscopy, which can induce mechanical artefacts, such as cracking or distorting blood vessels. Sections may show compressed or deformed vessels and airways due to the pressure of the microtome blade.
• Staining involving various chemicals (e.g., haematoxylin and eosin), which can accentuate or blur certain structural details of blood vessels. Sometimes, dyes can cause precipitates or other colouring artefacts, which can mask or alter the natural appearance of the dishes.
• Fitting the tissue ultrathin sections onto the slides and covering them with another slide can induce mechanical pressure, which could compress or deform the blood vessels and airways.

The smaller the patch size, the more faithful the reproduction. It should be noted that areas, admittedly small in size, are incorrectly identified in the case of pathology slides, regardless of the size of the analysis patch. As a general rule, however, it can be deduced that the choice of analysis patch size depends on the objective of the analysis, i.e., the structures of interest for diagnosis and the precision required in their identification and delineation. A study in which the influence of the patch size vs. the EO image (resolution, size of objects in the EO image) was investigated in [11].

This process is used for the conjugate analysis of CT and/or (magnetic resonance imaging) MRI in conjunction with the analysis of pathological slides to identify biomarkers that can be extracted from texture analysis. Texture analysis in pathological images, at different organ levels (CT/MRI) and in tissue (pathological slide), provides a series of biomarkers that can be independent or correlated. If histopathological biomarkers are linked to histological type, degree of differentiation, and degree of invasion (perineural, intravascular), they can be correlated with imaging biomarkers (radiomics). Then, there is a shift in a diagnostic paradigm, whose integration into clinical practice is difficult to formulate, specify, or predict. However, our analysis can be complementary and even integrated into radiomics, with quite broad perspectives, especially if supervised learning can be defined in which to build relationships between patches beyond semantic classification. We think, first of all, of the topological relationships that allow the reduction in errors by relating a patch to neighbouring patches, when the analysis of the content would lead to a completely different classification than the patches in the analysed area.

Medical images also remain to be explored in various capture conditions. For example, CT images can be examined in different windows corresponding to optimal visualisation, depending on the anatomical region where the object is located, for instance, the chest window, brain window, etc., with or without contrast. Images from pathological slides can also be analysed at different sizes and in different colorations than the usual haematoxylin–eosin staining, such as immunohistochemical stains. In the case of both types of medical images mentioned, a rigorous definition of the classes belonging to healthy tissues and those corresponding to lesions is necessary. Last but not least, it is necessary to define criteria not only technical but also related to obtaining relevant information for diagnosis, so that it is possible to compare and evaluate in the form of benchmarking the various technical solutions of digital pathology.

The small sample size of the medical datasets used in this study represents an inherent limitation. As this work was designed as a proof-of-concept demonstration, the reported accuracy values are intended only to show the feasibility of applying EO-inspired semantic labelling to medical images, and not to establish clinical performance benchmarks. Future studies will involve significantly larger, multicentre datasets, enabling statistical hypothesis testing and more robust validation of the methodology.

However, the results in this study could help us to identify possible causes why one city has a higher number of patients with a particular disease (e.g., patients with cancer or schizophrenia) compared to another city where the number is smaller, considering the structure of the city (e.g., the surface of the green areas, number of hospitals, the surface of the industrial area). Such a perspective could be ground-breaking in medical epidemiology. These are the premises for a correlation between the information extracted from the Earth observation images related to environmental conditions and the information in medical images (such as disease radiological or histopathological phenotypes), providing in-depth knowledge about environment-related disease aetiologies. Furthermore, this analysis includes the temporal component of all types of images to study the evolution of the cities/areas or the patients in time. We would also like to mention that such a vision idea/conception has not been approached so far in the literature.

The results of the Earth observation image analysis should answer questions such as: How green is the city? How many parks are in the city? Is there a forest or a lake in the city or nearby? How many hospitals are there in the area? Is it an area with a high density of houses? Is it an industrialised city? This information should be grouped into a semantic model of the city/region, obtained based on these images and further validated using existing reference datasets (e.g., in situ measurements, CORINE Land Cover, Urban Atlas, OpenStreetMap), as well as the experience of remote sensing experts in the field.

Similarly, a semantic medical model can be obtained for a particular disease or several diseases by analysing the medical images. Comparing these results to previous hospital results and the physicians’ expertise validates them.

In creating a global health model of a city/area (see Figure 22), other information can be added to the previous two models. For instance, the Copernicus Sentinel-5P mission performs atmospheric measurements with a high spatio-temporal resolution. Thus, one can easily add the pollution level of the area, ozone levels, UV radiation, and climate monitoring and forecasting. Also, additional data layers can be added from the Copernicus Sentinel-3 mission dedicated to water vapour absorptions and atmospheric and aerosol monitoring. Figure 23 and Figure 24 show the domain ontology and the knowledge graph [5] created for the global health model of a city. In addition, the temporal component of images can be added to the model helping to observe the changes in a city. In addition, these urbanism or pollution features can be added to the medical information about monitored patients who, for example, are recovering from an illness.

Although this study does not include a full epidemiological case study, the unified methodology we propose has significant potential for future public health applications. By using identical semantic labelling workflows for both environmental and medical datasets, it becomes possible to generate geographically resolved maps that link environmental risk factors (e.g., pollution levels, urbanisation patterns) with disease phenotypes observed in medical images. Future work will focus on pilot studies, in which lung cancer imaging datasets will be correlated with EO-derived air quality data, as well as exploring links between colorectal cancer patterns and environmental indicators such as water quality and agricultural practices. These applications will require large, geographically indexed datasets and collaboration with public health institutions.

5. Conclusions

The described system comprises novel concepts to help Earth observation, medical, and other data users access and discover general information from large archives of images, and quickly collect the desired detailed information to act accordingly. This involves dealing with complicated spatial, structural, and temporal relationships between land surface categories and objects appearing in the image.

In this paper, we have shown the usefulness and the adaptation of the proposed method for the semantic labelling of various types of images acquired by different instruments.

In the first case, we considered EO images acquired by five different instruments covering four cities from different parts of the world. The method described in Figure 1 (top part) was successfully applied, and the remote sensing experts were able to semantically label these images with predefined labels [54], with an accuracy of 95% [11]. An important observation is that the better the sensor resolution, the more semantic classes and more accurate classification results. Also, an appropriate patch size depends on the sensor resolution; an optimal patch size vs. resolution is presented in [11,12]. A detailed comparison of different methods (e.g., CNN, auto-encoder, hybrid methods, etc.) applied to the EO images was performed in [12].

We considered medical images acquired with four different instruments in the second case. The method described in Figure 1 (bottom part) was successfully applied to medical images without major software modifications. Here, the expected image content is already considered when choosing an appropriate patch size and an efficient feature extraction technique. The grouping of the patches into semantic classes was made interactively by several experts in the field (physicians), who gave meaning to the extracted classes by assigning a best-fitting semantic label to each class (in this case, we used no predefined semantic labels like for the EO images). The visual accuracy obtained for these semantic classes lay between 80% and 85% (based on the feedback collected from the doctors). We point out that for medical images, a more detailed study must be performed about the optimal patch size and the features to be extracted, which depends on the type of the instrument used to acquire the medical image and on the content of the image.

The reason behind the lower (visual) results of the unsupervised learning may be the fact that the features are not captured by the 4 × 4 pixels. The size of this patch needs to be adjusted according to the resolution and size of the objects (class content too small) in the case of the medial images. Regarding the EO images, a patch of 4 × 4 pixels gives good results using this method [42].