Polarization-Based Digital Histology of Human Skin Biopsies Assisted by Deep Learning

Abstract

Mueller polarimetry has proven to be a powerful optical technique to complement medical doctors in their conventional histology analysis. In this work, various degenerative and malignant human skin lesions were evaluated ex vivo using imaging Mueller polarimetry. The Mueller matrix images of thin sections of biopsies were recorded and the differential decomposition of Mueller matrices was applied pixel-wise to extract the polarization fingerprint of the specimens under study. To improve the classification accuracy, a deep learning model was created. The results indicate the sensitivity of polarimetry to different skin lesions and healthy skin zones and their differentiation, while using standard histological analysis as a ground truth. In particular, the deep learning model was found sufficiently accurate to detect and differentiate between all eight classes in the data set. Special attention was paid to the overfitting problem and the reduction of the loss function of the model. Our approach is an effort in establishing digital histology for clinical applications by complementing medical doctors in their diagnostic decisions.

1. Introduction

Polarization-based optical modalities are sensitive to the micro-structure of biological tissue and have found its place in pre-clinical and clinical applications [1,2,3,4,5]. It was shown that polarized light can be used for non-invasive diagnosis of benign and malignant lesions [6,7,8,9,10,11,12]. Along with diffuse reflectance, fluorescence, Raman spectroscopy and other optical techniques, tissue polarimetry can be used as label-free modality providing supplementary information for skin diagnosis [10,13]. Specifically, polarization of light is sensitive to tissue morphology and can reveal relevant information about the tissue internal structure and reorganization. This would allow medical doctors to be assisted in their diagnosis by performing polarization-sensitive digital histology. In such cases, the availability of a physical model for the corresponding turbid medium [14] is highly desirable. Unfortunately, this is not always the case, while it remains of huge importance to interpret physically an experimentally measured Mueller matrix and consequently, to retrieve all polarization and depolarization properties contained in it. If one knows the physically realizable form of M for the medium of interest, all polarization and depolarization properties can be recovered via appropriately selected decomposition algorithms [15]. Regardless of the experimental geometry (either transmission or reflection), precise instrument calibration is needed (i.e., eigenvalue calibration method [16]), as well as another decomposition algorithm to filter data noise and/or experimental errors (a.k.a physical realizability filtering [17]). If one wants to interpret phenomenologically a given experimental Mueller matrix, conventionally one can use differential [18,19], Lu-Chipman [20] or symmetric decompositions [21], in order to relate the decomposition results to the polarization fingerprint of the samples under study. In this way, the need for an explicit physical model can be bypassed. Whenever tissue damage is caused by malignant or benign formations the above-mentioned optical properties are modified. As a result, the light scattering also changes and affects the (de)polarization properties of the samples under study.

In clinical practice, pathologists are in charge of conducting macroscopic and microscopic examinations of tissue biopsies and surgical specimens. These types of examinations aim at making accurate diagnosis and at localizing any zones of the tissues with abnormal structure [22]. However, this is far from a trivial task and requires extensive knowledge and experience. It is time consuming, involving lengthy sample preparation and examination of many histological slides [23]. To aid and supplement medical doctors, polarization-sensitive digital histology holds great promise as a fast, non-invasive and relatively cheap optical method [24].

With the large amount of experimental data, suitable data post-processing algorithms are required. These can potentially include classical statistical methods and/or even extend further to the rapidly advancing domain of artificial intelligence (AI) [25]. The latter could be efficiently utilized to mimic human-like intellect when making decisions by using artificial neural networks [26,27]. Especially, when necessary to work in the image processing domain, object detection or image recognition are pivotal tasks and find many applications as reported in various sources [28,29,30,31,32,33,34,35]. Usually, convolutional neural networks (CNN) are used for such tasks, and demonstrate very promising results. It becomes useful to compensate the absence of general physical model describing the light tissue interactions with polarized light by acquiring sufficient amount of large data sets with ground truth polarization-sensitive measurements and images. This approach could be used to construct a deep learning model for image classification. Thus, the work of the medical staff may be eased by means of classification of newly obtained data, while the classification time may be significantly reduced.

In this work we focus on several types of skin pathologies namely, the basic types of skin cancer, such as: basal cell carcinoma, squamous cell carcinoma and malignant melanoma. Secondly, we addresses different types of degenerative diseases affecting the collagen quantities and structure, namely: psoriasis, lupus erythomatosis, scleroderma and Syndrome of Reynaud. Indeed, to the best of our knowledge, there is not much work done on the polarimetric response of tissue samples with degenerative diseases all of which affect the collagen structure of the skin. The workflow included measuring the Mueller matrices of samples and classes with subsequent filtering and decomposition. Then different data post-processing algorithms were used to exclude the substrate contribution, which is often overlooked, from the polarimetric response. Then the workflow proceeded with obtaining thickness invariants for all polarimetric quantities. Finally, the images were segmented into patches, in order to form a polarimetric model with CNN for image classification in order to increase the diagnostic accuracy.

2. Materials and Methods

2.1. Theory

Following Azzam [36], the Mueller matrix M of an anisotropic continuous medium, measured with light propagating along the z direction, can be reduced to the so-called differential matrix m in transmission measurement configuration. Indeed, M can be shown to obey the differential equation [18]:

$$\frac{d{\mathbf{M}}_{\mathbf{z}}}{dz}=\mathbf{m}{\mathbf{M}}_{\mathbf{z}}.$$

(1)

Here, m is another 4 × 4 matrix, containing the means and the variances (or fluctuations) of polarization properties [18,19]:

$$\mathbf{m}={\mathbf{m}}_{\mathbf{m}}+{\mathbf{m}}_{\mathbf{u}}=\langle \mathbf{m}\rangle +\langle \Delta {\mathbf{m}}^{\mathbf{2}}\rangle \mathit{z},$$

(2)

where the spatial averaging $\langle \rangle$ is in the transverse plane. The random fluctuations in Equation (2) result from different reasons, and depend on the sample inner structure and reorganization. Fluctuations may originate, for instance, from heterogeneous structures, different scatterer sizes and non-uniform distribution of the scattering particles.

After rearranging, substituting m and integrating with respect to the boundary conditions, Equation (1) is found to admit the exponential solution [19]:

$$\ln \mathbf{M}=\langle \mathbf{m}\rangle z+\frac{1}{2}\langle \Delta {\mathbf{m}}^2\rangle z^2$$

(3)

The matrix logarithm can be obtained by solving the eigenvalue-eigenvector problem for M and evaluating the diagonal matrix D = diag[ln($d_i$)] whose elements $d_i$ are the eigenvalues of M. Finally, by using D and the matrix V whose columns are the eigenvectors of M, the matrix logarithm and the two, polarizing and depolarizing, parts of m can be calculated from [18]:

$$\mathbf{L}\equiv \ln \mathbf{M}={\mathbf{VDV}}^{-1}$$

(4a)

$${\mathbf{m}}_{\mathbf{m}}=\frac{1}{2}\left(\mathbf{L}-{\mathbf{GL}}^{\mathbf{T}}\mathbf{G}\right),$$

(4b)

$${\mathbf{m}}_{\mathbf{u}}=\frac{1}{2}\left(\mathbf{L}+{\mathbf{GL}}^{\mathbf{T}}\mathbf{G}\right),$$

(4c)

where G = diag(1,−1,−1,−1) is the Minkowski metric tensor. More precisely, the polarizing and the depolarizing parts of m can respectively be represented as [18]:

$${\mathbf{m}}_{\mathbf{m}}=\left[\begin{array}{cccc}0& p_1& p_2& p_3\\ p_1& 0& p_6& p_5\\ p_2& -p_6& 0& p_4\\ p_3& -p_5& -p_4& 0\end{array}\right],$$

(5a)

$${\mathbf{m}}_{\mathbf{u}}=\left[\begin{array}{cccc}\alpha & d_1& d_2& d_3\\ -d_1& {\alpha }_1& d_6& d_5\\ -d_2& d_6& {\alpha }_2& d_4\\ -d_3& d_5& d_4& {\alpha }_3\end{array}\right].$$

(5b)

The notations used denote the elementary polarization properties as follows: $p_1$—linear dichroism along x-y axis (LD), $p_2$—linear dichroism along $\pm {45}^{\circ }$ axis (LD’), $p_3$—circular dichroism (CD), $p_4$—linear birefringence along x-y axis (LB), $p_5$—linear birefringence along $\pm {45}^{\circ }$ axis (LB’) and $p_6$—circular birefringence (CB). For a depolarizing medium, the off-diagonal elements of the polarizing part m_m represent the mean values of the six elementary properties. In the depolarizing part m_u, the off-diagonal elements $d_i$ are the uncertainties of the six elementary properties, while the main diagonal contains the depolarizing factors ${\alpha }_i$ (provided that $\alpha$ = 0). Again, ${\alpha }_1$ is along the x-y axis, ${\alpha }_2$ is along the $\pm {45}^{\circ }$ axis and ${\alpha }_3$ is the circular component. Often, ${\alpha }_1$, ${\alpha }_2$ and ${\alpha }_3$ are referred to as anisotropic absorptions and are consequently denoted LA, LA’ and CA, while ${\alpha }_0$ is the isotropic component [18]. One can obtain the rotationally invariant net scalar retardance $\varphi$, the total depolarization ${\alpha }_t$ (provided that $\alpha$ = 0) and the orientation of the optical axis $\theta$ [15] as:

$$\varphi =\sqrt{p_6^2+p_5^2+p_4^2},$$

(6a)

$${\alpha }_t=\frac{1}{3}|{\alpha }_1+{\alpha }_2+{\alpha }_3|,$$

(6b)

$$\theta =\frac{1}{2}ta{n}^{-1}\left[\frac{p_5}{p_4}\right].$$

(6c)

In the above equations, the depolarization factors are bound within the interval (−∞, 0] for physically realizable differential Mueller matrices, while the factor of 1/2 in the azimuth definition is relating to a physical azimuth in the absence of a circular component. If one wants to relate it to an orientation angle for the Poincaré sphere representation, then the factor of 1/2 should be omitted.

The proposed decomposition algorithm can be adopted for use when studying experimentally the propagation of polarized light through an anisotropic continuous medium in transmission measurement configuration. Nevertheless, it is very important to compare results for samples with similar thicknesses, since the polarization properties scale linearly with sample thickness, while the depolarization ones evolve quadratically [18]. In reality, especially when dealing with biological samples, the thickness is always varying from one sample to another and the aforementioned considerations are not met. In order to avoid this obstacle, thickness-invariant parameters must be implemented. Analogously to Equation (1), the Beer-Lambert law has also an exponential solution [37]:

$$dI=-I{\mu }_{t}dz,$$

(7a)

$$I\left(z\right)=I_0{e}^{-{\mu }_{t}z},$$

(7b)

where I denotes the light intensity and ${\mu }_t$ is the total attenuation coefficient which includes both scattering and absorption. In practice, the latter is unknown in most cases but can be evaluated approximately from:

$$ln\left[\frac{I\left(z\right)}{I_0}\right]=-{\mu }_{t}z\approx ln\left(m_{11}\right).$$

(8)

The matrix element m₁₁ can be obtained experimentally from the non-normalized M, and each polarimetric quantity from Equation (6a) through Equation (6c) can be normalized, thus working with and considering only thickness-invariant parameters. Specifically, all polarization quantities must be normalized to ln(m₁₁), while all depolarizing ones, to ln²(m₁₁) [24].

2.2. Substrate Contribution Removal

Most often, all biomedical samples are placed on different types of substrates (glass, quartz, etc.). Ideally, the substrate would have no polarization response, but in practice there might be some residual one. When thin sections of tissue on top of the substrate are analyzed, their polarization response can be biased by the contribution of the polarimetric parameter values of bare substrate in the zones of tissue holes and cracks, which is undesired for accurate diagnosis and image classification. To overcome this obstacle, the following algorithm was proposed. After the calibration, the bare substrate was measured. Then any measurements of biological sample placed on a substrate would yield a cumulative polarization response. To filter the substrate contribution, the algorithm operates prior to normalizing the experimental 4 × 4 matrix and proceeding to the physical realizability test. The filtering of bare substrate zones is performed by using the following metric:

$$RMSE(x,y)=\sqrt{{\displaystyle \sum _{i=1}^{n=16}}\frac{{[{\widehat{I}}_i(x,y)-I_i(x,y)]}^2}{n}},$$

(9)

where RMSE stands for root-mean-squared error, the index i runs over the experimental matrix elements, $\widehat{I_i}$ represents the sample-with-substrate measurement, while $I_i$ is the bare substrate measurement. As each matrix element is an image with size 600 × 800 pixels, the above-mentioned metric is calculated for each pixel of the imaging sensor. An a priori threshold value t is assigned, depending on how soft or hard the filtering is desired to be. Its value is entirely empirical since depending on both sample and substrate types. If RMSE(x,y) > t, the pixel value is preserved since the difference in the intensity images between the sample and the substrate is significant. Conversely, if RMSE(x,y) ≤ t, then the pixel value is replaced by not-a-number value (NaN) because the difference in the intensity images between the sample and the substrate is not large enough, hence assuming a dominant substrate contribution to the detected signal. In this way, although at the price of increased computational time, the polarization response of the bare substrate zones can be filtered out using a relatively simple metric. From here on, the intensity matrix can be normalized and fed towards the physical realizability test, before decomposing the resulting Mueller matrix.

2.3. Ex Vivo Skin Samples

Thin histological sections of human skin biopsies were used in this study, complying with the local ethical committee. The results of the standard histological analysis provided by a pathologist were used as a ground truth for polarimetry measurements and analysis. The samples originate from 28 patients, 15 with tumor lesions and 13 with collagenosis. The patients are between 44 and 84 years old, in generally good health and no other abnormal conditions affecting the skin. In total, there are eight different tissue types (or classes) labelled as: H—healthy, SC—scleroderma, LU—lupus erythematosus, PS—psoriasis, RE—syndrome of Raynaud, SCC—squamous-cell carcinoma, BCC—basal-cell carcinoma, and MM—malignant melanoma.

The standard procedure for preparation of histological tissue slides in the clinical practice includes various processing steps: tissue fixation; fitting in the cassettes; dehydration; clearing; paraffin embedding; sectioning; mounting on microscopy glass slides; deparaffinization; protecting the tissue slide by mounting means and coverslip. Tissue fixation is performed through submerging of a tissue sample in 10% neutrally buffered formalin. The tissue is then trimmed to fit in a standard histology cassette, dehydrated through immersion in five baths with increasing concentrations of alcohol, cleaned with organic solvents and embedded in paraffin wax. The paraffin wax provides support to the tissue that allows thin sectioning. The thickness of the slides is usually around 4–5 μm to ensure single cell layer, since the average diameter of the tissue cells is around 10 μm. Thereupon the slides are deparaffinized. After the aforementioned processing the tissue section is transparent, hence there is a requirement for a staining procedure. The most common histology dyes are heamotoxylin and eosin. Heamotoxylin binds with the cell nuclei and stains them in purple-violet. Eosin binds with the structural proteins—collagen and elastin, thus, highlighting the tissue structural matrix in different shades of pink. To adapt the stained tissue slide to white light microscopy imaging and pathological examination, a mounting means and a coverslip are applied.

The skin cancer tissue slides used in this study are consecutive sections from a single tissue sample. One of the slides is processed through the standard procedure described above, the processing of the other slides follows the same clinical protocol up to the cutting stage. The samples prepared for polarimetric measurements were not stained, since we rely on the polarimetric Muller microscopy to provide the contrast for the different types of structures in the tissue slides. Initially, the paraffin wax was kept on the slide to protect the tissue from mechanical damage. However, our preliminary results proved that the presence of paraffin disrupts the specific optical properties of the normal and pathological tissues. Hence the paraffin was removed, following the standard protocol of three consecutive xylene baths for 5 min each and three consecutive 98% ethylene baths for 3 min each to remove xylene. Our previous experience demonstrates that samples thicker than the standard ones (4 μm) are better suited for polarimetric measurements, resulting in more informative experimental data [24]. Also, Dong et al. [38] have used tissue sections from breast tissue with a thickness of 12 μm and reported sufficient polarimetric contrast for the recognition of characteristic structures.

However, all of the investigated histological sections have nominal thicknesses varying from 4 μm to 12 μm and were placed on glass microscopy slides. Since the microscope data processing includes normalization to the thickness of the sample [24], this difference in thicknesses was eliminated and did not require additional corrections before including the data to the final dataset for classification.

Measurements for healthy skin were performed on the adjacent healthy tissue around the tumor for the samples of cancerous lesions. Collagen degenerative diseases affect the whole skin, hence no measurements of “healthy” originate from these samples. Depending on the size of the tumor and the adjacent healthy part, up to ten measurements for both areas were performed of one sample, without overlapping.

2.4. Imaging Mueller Polarimeter

The Mueller polarimeter used for this study is described in details in [39]. For the sake of clarity we provide a brief overview of the instrument. The light source is white-light LED and the wavelength of the incident light is selected through color filter with a central wavelength of 700 nm and full width at half maximum of 15 nm. Identical optical elements for the polarization state generator (PSG) and the polarization state analyzer (PSA) were used, with the only difference that these were placed in reverse order for the PSG and the PSA, respectively. The latter two include a horizontal linear polarizer, two ferroelectric liquid crystal (FLC) retarders (Meadowlark FPR-200-1550) and a quarter-wave retarder inserted between the two FLC retarders. For detection the light is focused onto a telephoto lens, coupled and set to infinity to a matrix photodetector (AV Stingray F-080B). The field of view was selected to be 400 μm, achieving a spatial resolution of 0.5 μm/pixel.

For a complete Mueller matrix measurement, 4 input and 4 output polarization states are required. All of these states can be reached with the voltage modulation of the optical elements in the PSG and the PSA, as discussed above. The resulting 16 polarization states generate 16 intensities to be measured, and the same number of intensity images. Then, 16 linear equations are solved, in order to reconstruct all elements of the Mueller matrix. The same approach was used during the calibration based on the eigenvalue calibration method, described in details in [16].

3. Results

3.1. Polarimetry

As can be seen from the polarimetric images in Figure 1, there are holes (shown by pixels rendered in black) in the skin tissue sections, and the contribution of glass substrate was filtered in these zones. This observation holds for all skin lesion images, hence the algorithm for the glass filtering clears very efficiently the unwanted contribution of the substrate. Second, one notices that the values of the polarimetric parameters for the degenerative and tumorous skin tissue images are generally lower than the corresponding values for healthy skin tissue images, used as a reference. It is to be noted that considerable reduction in the values of the depolarization properties of skin cancer tissue samples has already been reported elsewhere [40]. This could be attributed to the structural alterations of skin by the pathology, causing morphological changes in the tissue, thus, changing its optical properties and affecting its polarimetric response. In the healthy skin, the extracellular matrix is comprised of structural proteins and fibers, thus, forming a structure to which the polarized light is highly sensitive. All degenerative and malignant formations in the skin are characterized by the changes in the extracellular matrix and/or an infiltration of abnormal cells that have usually atypical size and shape. This leads to changes in morphology and loss of tissue optical anisotropy, resulting in different polarization response and increase in polarimetric contrast between the healthy skin and the pathological skin samples. For skin cancer samples it was initially expected to observe a decrease of the tissue anisotropy for all malignant sample types, since the process of cancerogenesis disturbs the tissues structure by disruption of the extracellular matrix [41,42,43] as well as by a loss of differentiation of the cells [44].

At this point, a parallel should be drawn between changes in the polarization and the depolarization properties of the tissues specimens. Whereas the former result from changes in the amplitudes and/or the phases of the electric field components, the latter are a consequence of coherence loss—both temporal and spatial. As a sensitive parameter for structural changes the scalar retardance $\varphi$ can be used. Also, the orientation angle of the optical axis $\theta$ can serve as indicator for randomization or loss of a preferential orientation in the image regions of interests (ROIs). On the other hand, light depolarization is quantified by the parameter ${\alpha }_t$.

Continuing the analysis, the orientation angle of the optical axis shows several predominant orientations, in different ROIs for all studied classes of skin tissue samples, whereas the orientation angle $\theta$ has been completely randomized at some zones. This could be related to the destruction of extracellular collagen matrix of healthy skin tissue upon development of a particular skin lesion.

To better evidence and compare the differences between all samples, one must use the measurements from all classes. As can be seen from the histograms in Figure 2 that include all measurements of all samples, the healthy samples demonstrate increased positive value of skewness for the distributions of both depolarization and retardance, while the distribution of the orientation angle appears to be almost uniform for the malignant class. There are also predominant orientation angles (around 0°) for both healthy and degenerative classes. The orientation and the alignment of collagen fibrils in the proximity of cancer are found to be the meaningful characteristics of collagen that may serve as the biomarkers of cancer [45,46]. The current understanding of the role of collagen polarimetric characteristics as diagnostic and prognostic parameters for cancer is still incomplete and is a subject of intensive research [47]. Le et al. [48] report lower values of the retardance for SCC, BCC and MM samples (by 75%, 58% and 70%, respectively), in comparison with the retardance of healthy skin.

The superposition of the distributions in Figure 2 could be attributed to the presence of the structures not affected by pathological alterations. In the best case scenario, there should not be any overlap between the distributions in order to discriminate the classes with different pathological status. To be able to distinguish different skin tissue classes in the presence of distribution overlap, the data were further post-processed with the artificial neural network model.

3.2. Deep Learning

For the purpose of the current study, 15 images per class for all skin lesions were selected, based on the medical doctors’ expertise and diagnosis. To increase the number of images in our data set, each image was segmented into 12 patches of 200 pixels by 200 pixels. Additionally, once filtered for the glass substrate contribution, the images of the Mueller matrix elements (except for $m_{11}$, because of its unit value after the normalization) were used as predictors, instead of using polarimetric parameters provided by the differential decomposition. This approach is not only time- and computationally-efficient, but is also information-preserving, since all (de)polarization information is entirely encoded in all Mueller matrix elements of the samples.

In total, there are eight classes of skin tissue at different pathological conditions and the final data set consists of 21,600 images, obtained from: 8 (pathological classes + healthy skin tissue) × 15 (images measured per class) × 15 (Mueller matrix elements) × 12 (patches per image). For each image, a categorical mask/variable was created to indicate its class: 0—H, 1—RE, 2—SC, 3—PS, 4—LU, 5—BCC, 6—SCC and 7—MM. This amount of data was found sufficient to be further split into data sets for training, validation and testing in the proportion 74:13:13 or 16,000:2800:2800 images. Generally, the neural network is to be used to extract complex features from the images that are unnoticeable by the medical doctors on the pixel level (we recall that the resolution per pixel is 0.5 μm). In such way, the diagnostic analysis of the medical doctors may be substantially complemented by the results obtained using neural network approach. For this task, a CNN was chosen, which architecture is provided in Figure 3.

The model was trained by selecting as optimiser “Adaptive Movement Estimation Algorithm” and adjusting learning rate, respectively, which is an extension to gradient descent. Initially, different model architectures were tried, but with poorer results, hence the chosen one was found to be the most optimal one for increasing the classification accuracy. In total, our model was trained on in total of 3,666,179 parameters. When there is a change in tissue inner structure and its extracellular collagen matrix reorganization due to the development of degenerative skin disease or tumor formation, the CNN model should be able to recognise the kind of alteration at the pixel level. Then, not only the medical doctors can be supported with another tool for diagnosis, but also this approach may have the potential to diagnose a particular kind of skin disease at earlier stage of development.

Initially, a 5 × 5 sliding window (a.k.a convolution kernel) was chosen with ReLU (rectifier linear unit) as activation function, respectively. To prevent overfitting, a pooling layer was added after each convolutional layer [26,27]. This downsizing technique examines the pixel values within another kernel and preserves only the maximum value in it. In this study, the kernel size for this step was chosen with size 3 × 3. This technique would help to make the model more general.

All the layers before the flatten one will learn the features of tissue structure of the skin samples. Taking all these features and passing them to two dense layers with 256 neurons (this value was found to be optimal for the current model) would result in learning the relationship between these features.

Finally, in the dense layer producing the classifications 8 neurons are required (since we have 8 classes), and the “softmax” activation function was used to convert the classification probabilities [26,27]. To further prevent the model from overfitting, the technique of dropout was used with a fraction of 0.2 as a regularization method [26,27], with additional batch normalization.

Randomly dropping out a proportion of the neurons during training has the effect of preventing the network from relying on any one particular set of neurons for making predictions [26,27]. By forcing the model to use different subsets of neurons for each training sample, one compels the model to learn more robust features that are useful across a variety of inputs.

First, the model learns from the training data and then it tries to minimize the loss function before making predictions on the labeled training data. The loss function used in the neural network was chosen to be the categorical cross-entropy, which is a subject to minimization from the optimizer [26,27]. The loss function is then minimised by the technique of back propagation and updating the weight and bias values at each step. Then, the model also tries to minimize the loss function of the validation data and to additionally perform classifications on it. In total, 300 epochs (i.e., 300 complete passes of the training dataset through the learning algorithm) with a batch size of 16 were found sufficient during the training phase to minimise the loss function and to reach satisfactory results for classification.

In Figure 4 and Figure 5 the results for the training and validation processes are presented. The curves of the loss function and the model accuracy are almost overlapping, indicating as minimum as possible or none at all overfitting, as was initially targeted. Hence, a good balance was reached between the model complexity with the available data and the desired purpose of classification of the skin images at different histological conditions.

Next, in order to evaluate the real model performance with the unseen data, the predictions and classifications were made with the testing data set. From 2800 labelled testing images only 213 were wrongly classified, which again confirms the excellent performance of the built Skin-HDT700 model (this model name was symbolically given to indicate the skin lesions of healthy (H), degenerative (D) and tumorous (T) tissue classes and polarimetric measurements performed at 700 nm). Eventually, the final classification accuracy with the testing data set was found to be ≈92%.

Next, for better clarity, the confusion matrix is shown in Figure 6. It has a practically almost diagonal form, with large values on the main diagonal (as intended and expected for a good classifier) and very small values for all off-diagonal elements (these values should be close to zero for a good classifier). Also, the classification report is included in

Table 1. Classification report including the values for precision, recall and F1-score, evaluating the classification model performance for the eight investigated classes.

Class	Precision	Recall	F1-Score
0—Healthy	0.98	0.93	0.95
1—Syndrome of Raynaud	0.86	0.92	0.89
2—Scleroderma	0.93	0.92	0.92
3—Psoriasis	0.90	0.91	0.91
4—Lupus erythematosus	0.96	0.92	0.94
5—Basal cell carcinoma	0.93	0.95	0.94
6—Squamous cell carcinoma	0.92	0.89	0.90
7—Malignant melanoma	0.92	0.96	0.94

, thus evaluating the precision, recall and the F1-score as classification metrics for each class, respectively. All of these metrics remain with similar and large values between all classes, thus indicating for a good and balanced classification performance of the Skin-HDT700 model.

Finally, the receiver operating characteristic (ROC) curves for all skin tissue classes are shown in Figure 7, again confirming the good classification capabilities of the model for all classes.

4. Discussion and Conclusions

In this study multiple histological tissue sections of skin tumors with the adjacent healthy tissue zones and different skin degenerative diseases were measured with the custom-built transmission Mueller microscope. A novel approach for bare substrate contribution removal based on relatively simple, but effective metric and algorithm, was developed and implemented. Further, data noise and/or experimental errors were filtered by using the Cloude’s physical realizability algorithm. The differential decomposition was utilized pixel-wise for the calculations of the polarization and depolarization parameters of the skin tissue samples. The resulting images were normalized in order to get thickness invariant images of polarization and depolarization parameters. Due to the inter-patient variability, an overlap of the distributions of total depolarization, scalar retardance and azimuth of the optical axis was observed for all classes.

To simplify and reduce the feature selection process, the images of all normalized Mueller matrix elements were used as predictors in order to construct a deep learning model for the multi-class image classification problem. The data were augmented by dividing each image into the patches, thus, feeding the model with more data for training. Special attention was paid to the prevention of overfitting, reduction of the loss function and increase of classification accuracy. The combination of imaging polarimetry and deep learning approach appears to provide an adequate solution for the identification of skin abnormalities as well as for partially overcoming the inter-patient variability problem.

Although the scope of the current study is entirely pilot-based, it holds great potential for the extension to digital histology analysis for pre-clinical and clinical applications. In order to achieve this goal, it is necessary to have a significantly larger number of samples available together with a larger data set. Nowadays a rapidly expanding field of AI has an exponentially growing number of applications and capabilities The current AI model Skin-HDT700 welds classification precision of around 92%. In comparison with previous work of Luu et al. [49] on diagnosis of skin cancer tissue sections through Mueller matrix elements and classification algorithm they also obtain overall classification precision of 93%.

The application of Skin-HDT700 classification model can be useful for the researchers and clinicians in classifying and distinguishing between healthy and abnormal skin tissues. Although computational time is usually not a crucial factor for research purposes, it is essential for clinical applications. For our model the whole training process was completed on the CPU for around 12 h. As for the classification time for a single image, assuming an already preprocessed image to classify is at avail, as well as the saved model has been loaded, then the classification of that image would be sufficient to complete within a few seconds. By improving the diagnosis, the survival rate and the quality of life for all affected patients can be also significantly improved. Additional unsupervised and supervised machine learning algorithms can be applied to enrich the supplementary information to provide to the medical doctors, as each Mueller matrix element may be more or less susceptible as a diagnostic marker to a given health condition. Nevertheless, these studies are beyond the scope of the current manuscript and will be assessed in future work.