Liptinite Segmentation in Microscopic Images via Deep Networks

Abstract

Maceral identification in images obtained with an immersive microscopy is one of the most important techniques for coal quality characterization. The objective of this paper is to explore the potential of semantic segmentation for the classification of liptinite macerals within microscope images. The following U-Net-based architectures were proposed for the task: a U-Net with a varying depth and feature map numbers, a U-Net extended with a proposed feature map attention mechanism, and a U-Net architecture with an encoder part replaced with a ResNet backbone. Two resolutions of input images were examined: 256 × 256 and 512 × 512 pixels. The training was conducted using constant and scheduled learning rate values. The results show a superior performance of the networks using a ResNet-based encoder, with the best IoU measure, equal 0.91, obtained with ResNet50. The other networks achieved worse results, but attention-supported U-Nets were considerably better than the basic versions. Both training approaches (constant and scheduled learning rates) yielded comparable results. The best results were better than those reported in the literature for other architectures of deep neural networks. It was also observed that the images presenting the greatest challenges to the networks were highly imbalanced, with the liptinite present only in a small area of the image. The architectures employing ResNet-based encoders were the only ones capable of surmounting these challenges.

1. Introduction

The organic matter of hard coal includes macerals of the vitrinite, liptinite, and inertinite groups. Individual macerals are formed from plant remains subjected to various coalification and transformation processes. Their physical and chemical properties change during the coalification process. Macerals are the basic, microscopically distinguishable components of hard coal. They are equivalents of minerals occurring in mineral rocks, with the difference being that they do not have a crystalline form and have a very diverse chemical composition [1,2,3,4,5,6].

Hard coals are heterogeneous in terms of chemical composition. This results from the diversity of plant material from which individual macerals were created and the variability of conditions in the biochemical and geochemical phases to which the plant material was subjected [2,6,7].

Petrographic studies of hard coal are important. They allow for determining the physicochemical and technological properties of hard coals [2,3,4,5,6,7,8,9]. Petrographic components are responsible for the properties of hard coals and their behavior in various coal processing technologies [2,6,9,10,11,12,13,14,15,16,17,18,19].

Knowledge of the content of macerals in hard coals is important when designing coal enrichment processes [12,13,20], and selecting coal for the gasification process [16,21,22,23] and combustion [6,13,15,24]. The petrographic composition of coal is also an essential parameter for composing coking mixtures and for forecasting the quality of coke [6,14]. The individual features that distinguish individual types of macerals include, among others: specific density, content of volatile parts, content of carbon, hydrogen, nitrogen, and oxygen elements [25,26].

Macerals are distinguished under a microscope, based on various physical features, namely color, reflectivity, and morphology [3,4,5,6]. Analyses of the petrographic composition and degree of coal organic matter carbonization, based on the vitrinite reflectivity index, have been used in Poland since the 1950s. Methods of petrographic analysis are based on microscopic identification of petrographic components by humans, which is laborious and requires extensive knowledge.

In the petrographic structure of hard coals, macerals of the vitrinite and inertinite groups predominate. Their total share in the structure of the organic matter of coal can reach up to 90% by volume. The content of liptinite group macerals in hard coals is small; it can reach up to 15% by volume, rarely up to 30% by volume [6,7,8,9]. Despite this, liptinite macerals can significantly affect the physicochemical and technological properties of coal. They show a high content of the element carbon. In a given coal, liptinite shows the highest content of volatile parts and the highest combustion heat of all macerals.

In hard coals, there are three types of liptinite group macerals: sporinite, cutinite, and resinite. In the microscopic image, the color of liptinite macerals varies from orange-brown or dark brown in low-coal coals to gray and light gray in coking coals. Their texture is also varied. It can be vermicular, ribbon-like, thread-like, oval, or irregular [6].

Sporinite represents spore skins. It consists of megaspores (as presented in Figure 1, Figure 2 and Figure 3) and smaller microspores (see Figure 3 and Figure 4). They are made of an organic substance with a complex chemical composition, resistant to the decomposition processes. In the microscopic image, they have oval, elliptical, and vermicular forms, strongly flattened. In reflected light, sporinite in weakly coals shows a brown color, often with orange internal reflections. In higher coals, its color changes to gray.

Cutinite, on the other hand, represents leaf epithelium (cuticle) or needle epithelium as presented in Figure 3 and Figure 5. The upper leaf cuticle is thicker and serrated on one side, while the lower leaf cuticle is thinner and has slits. In reflected light, the color of cutinite is dark gray to light gray.

Resinite is formed from plant resins and essential oils transformed into resins. It occurs very rarely in hard coals. Most often in the form of fillings of telinite cells, streaks, or lenses. In a microscopic image, in reflected light, it is dark gray and darker than sporanite and cutinite. It occurs very rarely.

The image processing and analyzing approach has been used with success in many applications, in fields such as biology, medicine, industry, and many others. There are also applications in discovering information on microscopic petrographic photographs [27]. The coal petrography and supporting discovery of solid fuels properties. The approach taken covers a wide area of ideas: from an investigation into a reflectance histogram, through the thresholding and segmentation based on color space [28], to the extensive use of color and texture features along with machine learning (ML) methods [29].

The Generative Adversarial networks (GAN) were used for resolution enhancement of coal microphotographs [25,30]. Usage of wide residual blocks along with multiscale attention allowed for obtaining more than satisfactory results. A very interesting attempt at using generative AI for enhancing and augmentation of training image sets was presented by Ma et al. [31]. The diffusion model was applied to generate artificial training images. The results were more than promising. The research of collotelinite (vitrinite) segmentation on microphotographs was made by Santos et al. [32]. The authors used Mask R-CNN for segmentation and then the reflectance of segmented maceral was determined by analyzing the gray level. There were reported applications of U-Net-based [33] architecture deep networks for macerals and intrusion detection on microphotographs [34,35,36]. Iwaszenko and Róg used a simplified U-Net architecture for maceral segmentation with good results for inertinite and vitrinite, and satisfactory (but not overwhelming) results for liptinite [34]. Fan et al. investigated the application of different modifications of U-Net for vitrinite, inertinite, and liptinite segmentation. The best results were obtained for U-Net with a ResNet [37] background and TransUNet [38,39]. However, the authors report liptinite as the most difficult maceral for segmentation. Wang et al. compared three different network architectures for semantic segmentation: U-Net, SegNet [40], and DeepLabV3+ [41]. The best results were achieved for the last architecture, and can be recognized as a reference for the automatic maceral groups segmentation.

Our work concentrates on liptinite segmentation on the microscopic images. The choice of liptinite as the only maceral group considered in the study was dictated by the observed difficulty in achieving an effective segmentation of it compared to other maceral groups (vitrinite, inertinite). Such a conclusion appears in previous works, although it is not always formulated directly [34,36,42]. In this paper, the focus is on U-type architecture. While networks of this type have been considered in previous works, the topic of studying their effectiveness in the context of liptinite appears to be open and promising for future research. The efficacy of networks based on this architecture in other applications suggests the potential for achieving high-fidelity segmentation of liptinite through the selection of network components and an appropriate training procedure. The research work carried out and presented in this article has confirmed this conjecture.

The selected deep network architectures based on U-Net were considered in the research, varying in depth and number of filters. The networks were trained “from scratch”, but commonly used backbone features extractors like ResNet were also examined (in different versions). An adjusted training procedure was proposed. The idea behind it was to pre-train the network from scratch using macerals easier to segment and then use them as initialization for liptinite segmentation.

2. Materials and Methods

2.1. Image Acquisition

For the purposes of this study, medium-sized samples for testing were taken in accordance with the ISO 18283:2006 standard Hard coal and coke—Manual sampling.

Samples of coal were obtained from the Polish Upper Silesian Coal Basin. The degree of carbonization of the organic substance of the collected coals, determined on the basis of the vitrinite reflectance index, did not exceed 0.75%.

Then, analytical samples with a grain size of less than 1 mm were prepared from the collected samples, from which microscopic briquettes were later prepared according to the PN-ISO standard ISO 7404-2:2009 Methods for the petrographic analysis of bituminous coal and anthracite—Part 2: Method of preparing coal samples.

Data on the tested coal samples (rank, the origin of the samples, and macerals compositions) are presented in

Table 1. Data on the tested coal samples.

Samples Origin	Number of Samples	Maceral Groups [% vol.]			Vitrinite Reflectance [%]
		Vitrinite	Liptinite	Inertinite
Poland, Upper Silesia	26	48–75	4–14	21–32	0.52–0.75

. The microscopic specimens were prepared via immersion of coal dust in a mixture of epoxy resin and hardener, obtained by mixing the components in a ratio of 8:1. The immersed microscopic specimens were left for at least 24 h until solidification. The solidified specimens were polished using a Struers LaboForce-3 grinding/polishing machine.

A Zeiss Axio Imager Z 2 m microscope (Figure 6) was used for the study. A magnification of 500 times and white light reflected in oil immersion (refractive index n = 1.5180) were used. Surfaces, on which the photographs were taken using an Axiocam 506 color camera, were selected. The set of microscope photographs obtained shows different macerals from the liptinite group, for which a mask set was developed. In the petrographic analysis, the participation of macerals groups is most important.

2.2. Images Preprocessing

Gathered visual information originated from two different microscopes (see section above). The sizes of acquired images varied from 1600 × 1200 pixels to 2752 × 2208 pixels. All images represented were taken with the same magnification—500×. To unify the input image dataset, all images were rescaled so that the width remained at 1600 pixels. Such resizing ensured the same size of the sample area represented by one pixel, with a size of 0.15652 × 0.15652 μm. Rescaling was performed using a Lanczos resampler [43]. No exposure corrections were made. Then, the image set was split into square tiles 512 × 512 pixels each. The resulting input set consisted of 590 images. It was divided into training and test sets in the proportion of 4:1. The images were normalized to fit the 0–1 interval. The normalization was performed per channel.

2.3. Training Procedure

The training procedure used images from the training set. To mitigate the overfitting and increase the trained networks’ generalization capabilities, an augmentation technique was used. However, the augmentation operations were limited to the operations that did not require image re-rasterization. Moreover, emphasis was placed on the use of transformations that could potentially generate images similar to those obtained during the image acquisition process. Say the transformations were chosen from the following set:

• Horizontal flip;
• Vertical flip;
• Rotation by angle 90°, 180°, 270°;

During the training phase, each image was subjected to one of the aforementioned transformations. The specific transformation was selected at random. Subsequently, with a probability of 0.5, the image underwent the transformation or remained unchanged. Depending on the experiment, the images were rescaled to 256 × 256 pixels or left in their original size (512 × 512). For the augmentation workflow, the Albumentations open-source library was used [44]. The testing set was not augmented in any way.

The training procedure assumed training the model for an arbitrarily chosen number of epochs. A series of preliminary experiments were conducted to determine the range of epochs that would yield the anticipated outcomes. The underlying assumption was that a point would be reached at which the network would exhibit indications of overfitting, as evidenced by the discrepancy in loss, IoU, and accuracy between training and test image sets. The findings indicated that, in our particular case, 800 epochs constituted the optimal training range. There were no automatic early stopping conditions applied. During the training, the binary cross entropy loss function was used:

$${\mathcal{L}}_{BCE}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left(y_i^{gt}\log y_i^{pred}+\left({1-y}_i^{gt}\right)\log \left(1-y_i^{pred}\right)\right)$$

(1)

where ${\mathcal{L}}_{BCE}$ stands for the loss function, $y_i^{gt}$, $y_i^{pred}$ are true and predicted class ($y_i^{gt}\in \left\{0,1\right\}$, $y_i^{pred}\in \left(0,1\right))$, and N is the batch size. As the input dataset was unbalanced (there were much more pixels belonging to the background than to the liptinite), the weighted form of the binary cross entropy function was used. The model quality was assessed using two measures: IoU (Intersection over Union—Jaccard index) and accuracy:

$$IoU={\displaystyle \frac{\left|Y^{gt}{\cap Y}^{pred}\right|}{Y^{gt}{\cup Y}^{pred}}}$$

(2)

$$accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(3)

The symbol $Y^{gt}$ denote a set of pixels representing ground truth on an image, while the symbol $Y^{pred}$ represents the set of image pixels classified as belonging to the sought object by the segmentation model. Similarly, the symbols TP, TN, FP, FN are true positive (foreground pixel classified as foreground), true negative (background pixel classified as background), false positive (background pixel classified as foreground), and false negative (foreground pixel classified as background). The value used for estimating the effectiveness of the model is the average computed on all images from the test set.

The ADAM [45] algorithm was used for loss function minimization. During training, two strategies for the learning rate value estimation were considered. The first assumed keeping the learning rate value constant during the training process, while the second lowered the learning rate by half every 100 epochs. Loss function, accuracy, and IoU were monitored for both training and test sets during training procedure.

2.4. Deep Network Architectures

The research was focused on a deep convolutional networks, belonging to the U-Net-originating networks family. The base U-Net network, previously designed for medical images semantic segmentation, consists of two branches: an encoder and a decoder. The important architecture detail, which differs U-Net from other encoder–decoder architectures is the existence of so-called skip connections: the links between corresponding layers in the encoder and decoder parts. They are responsible for transferring, in the form of feature maps, the spatial information from the encoder to the decoder, which would be lost otherwise. The conducted research considered the following architectures:

• U-Net (three variants with varying depths were examined);
• U-Net with channel attention (three variants with varying depths were examined);
• U-Net with ResNet backbone (five variants of ResNet backbones were examined).

A detailed description of the network architectures is presented in the subsections below. Each deep network variant was trained for squared tiles of sizes 512 × 512 pixels and 256 × 256 pixels. In the case of larger resolutions, no resizing was applied to the input data, while the smaller ones were resized versions of the bigger ones. Differentiations of the tile sizes were aimed at investigating potential resizing and resolution influence on segmentation results.

2.4.1. U-Net Architecture

The architecture of the used U-Net deep network is depicted in Figure 7. Both encoder and decoder parts extensively use convolutional neural networks. For the experiments, the depth of the network was changed from 4 to 6 sections composing the encoder and decoder blocks. The number of filters started at 16 on the shallowest layer, going up to 128, 256, and 512 in the deepest one (depending on the tested network depth). ReLU activation was used everywhere except the network output, where the sigmoid function was used instead. The overall depth of the network was lower than that originally proposed by Ronneberger [33]. The limitation was applied because of the size of the input image—at each level, the size of feature maps was halved in each dimension, so the deepest feature maps’ size determined the reasonable depth of the network.

2.4.2. U-Net with Channel Attention Architecture

For checking if slight attention can have an observable effect on liptinite segmentation effectiveness, a simple mechanism for channel-wise attention was proposed. The architecture of the network was modified on the concatenation paths by adding a vector of trainable weights. The weights were supposed to favor some of the concatenated feature maps while inhibiting the others. The structure of the proposed mechanism is depicted in Figure 8. Similarly to the case described in the previous section, three depths of the network were considered (from 128 to 512 filters in the deepest layer). The proposed attention mechanism is simple. At each level of the U-Net structure, the features maps, before concatenating with the corresponding layer in the decoder branch, were multiplied by a vector of attention coefficients α:

$${F\prime }_i^{en}={\mathsf{\alpha }}_i\otimes F_i^{en}$$

(4)

where $F_i^{en}$ is a tensor of features at level i, ${F\prime }_i^{en}$ is the resulting tensor of features, and ${\mathsf{\alpha }}_i$ is attention coefficients vector for layer i. The operator $\otimes$ is a multiplication—each features map from the features tensor $F_i^{en}$ is multiplied by the corresponding element of the attention coefficients’ vector ${\mathsf{\alpha }}_i$ (the operator resembles the well-known Hadamard product). The attention coefficients vector is subject to training.

2.4.3. U-Net with ResNet Backbone Architecture

The third of the investigated architectures used the encoder build with the usage of ResNet blocks. The encoder branch in each architecture consisted of four ResNet blocks, while the decoder had the same construction as the first and second architectures (up-sampling and convolutional layers). There were five different U-Net—ResNet configurations, which differed with the type of ResNet blocks used: ResNet18, ResNet34, ResNet50, ResNet101, and ResNet152. In each case, pre-trained ResNet parts were used. However, they were not frozen, and their weights were modified during training.

The complete list of deep network architectures is presented in

Table 2. Networks architectures used for liptinite segmentation experiments.

Network ID	Network Type	Architecture Details
UN1	U-Net	16, 32, 64, 128
UN2	U-Net	16, 32, 64, 128, 256
UN3	U-Net	16, 32, 64, 128, 256, 512
SU1	U-Net + Simple Attention	16, 32, 64, 128
SU2	U-Net + Simple Attention	16, 32, 64, 128, 256
SU3	U-Net + Simple Attention	16, 32, 64, 128, 256, 512
UR1	U-Net with ResNet encoder	ResNet18
UR2	U-Net with ResNet encoder	ResNet34
UR3	U-Net with ResNet encoder	ResNet50
UR4	U-Net with ResNet encoder	ResNet101
UR5	U-Net with ResNet encoder	ResNet152

Each of the networks was subject to training with the 256 × 256 resolution, as well as the 512 × 512 resolution, with the same procedure. All networks were implemented using pytorch. The computations were conducted on PC running a Microsoft Windows operating system. The computer was equipped with an Intel Core i7-12700H 2.70 GHz processor, 32 GB of RAM, and NVIDIA RTX3080Ti, 16 GB VRAM graphics card.

3. Results

The results present the best values obtained during the training procedure. First, the most effective approach to the training was established by the comparison of the results obtained with training using the constant learning rate coefficient to the scheduled learning rate training. The constant rate was chosen from the set of trials. The chosen value was different for the networks used, but fell between 10⁻³ and 10⁻⁴. For the scheduled learning rate corrections, the best results were achieved when the initial learning rate started with 10⁻³ and was halved every 100 epochs. In each case, the assumed number of epochs was equal to 800. However, when the signs of the lack of convergence were identified, the training was manually halted to conserve resources (only a few cases exhibited these characteristics). The results obtained for each of the networks and learning rate schedules are presented in

Table 3. The results of liptinite segmentations for constant and scheduled learning rates.

Network ID	Image Size	Constant Learning Rate			Scheduled Learning Rate
	[Pixel × Pixel]	lr	IoU	Accuracy	Starting lr	IoU	Accuracy
UN1	256 × 256	10−3	0.671076	0.984067	10−3	0.667536	0.983995
UN1	512 × 512	10−3	0.715285	0.988587	10−3	0.654678	0.984221
UN2	256 × 256	10−3	0.701296	0.985928	10−3	0.589239	0.977995
UN2	512 × 512	10−3	0.725503	0.987457	10−3	0.622029	0.981534
UN3	256 × 256	10−3	0.570244	0.975736	10−3	0.602232	0.978368
UN3	512 × 512	10−3	0.673112	0.997436	10−3	0.644637	0.980819
SU1	256 × 256	10−3	0.731232	0.987534	10−3	0.603762	0.980358
SU1	512 × 512	10−3	0.729422	0.988659	10−3	0.649881	0.983203
SU2	256 × 256	10−3	0.669764	0.984286	10−3	0.606607	0.979867
SU2	512 × 512	10−3	0.788368	0.991566	10−3	0.685189	0.985391
SU3	256 × 256	10−3	0.627331	0.981021	10−3	0.553564	0.973038
SU3	512 × 512	10−3	0.755102	0.989774	10−3	0.582619	0.979038
UR1	256 × 256	5 × 10−4	0.835293	0.993426	10−3	0.787711	0.990501
UR1	512 × 512	5 × 10−4	0.846189	0.994433	10−3	0.695672	0.987332
UR2	256 × 256	5 × 10−4	0.851052	0.994207	10−3	0.802847	0.992266
UR2	512 × 512	5 × 10−4	0.856250	0.994330	10−3	0.816170	0.992466
UR3	256 × 256	5 × 10−4	0.878621	0.995852	10−3	0.811808	0.992449
UR3	512 × 512	5 × 10−4	0.915585	0.997436	10−3	0.800134	0.993146
UR4	256 × 256	10−4	0.884272	0.996129	10−3	0.857781	0.994800
UR4	512 × 512	10−4	0.910468	0.997433	10−3	0.810178	0.993604
UR5	256 × 256	10−4	0.883951	0.996846	10−3	0.844847	0.993461
UR5	512 × 512	10−4	0.894659	0.996769	10−3	0.870872	0.995519

. The typical charts presenting the loss function values were presented in Figure 9, and the values for IoU for such cases were presented in Figure 10. It can be observed that symptoms of overfitting are easier to spot on the IoU chart.

The trained models were used for the computation of a set of segmentation quality measures. The IoU, acc (accuracy), rcl (recall), prc (precision), and F₁ measures were used. The recall, precision, and F₁ were calculated using the following formulas:

$$recall={\displaystyle \frac{TP}{TP+FN}}$$

(5)

$$precission={\displaystyle \frac{TP}{TP+FP}}$$

(6)

$$F_1={\displaystyle \frac{2TP}{2TP+FP+FN}}$$

(7)

The obtained results are presented in

Table 4. Network architectures and their best results. The best results in each column were marked in bold.

Network ID	Image Size [Pixel × Pixel]	acc (Accuracy)	rcl (Recall)	prc (Precision)	F1	IoU
UN1	256 × 256	0.679087	0.979058	0.791190	0.671076	0.679087
UN1	512 × 512	0.988587	0.722411	0.974642	0.819893	0.715286
UN2	256 × 256	0.985928	0.709571	0.980746	0.815265	0.701296
UN2	512 × 512	0.987457	0.732874	0.985958	0.833731	0.725503
UN3	256 × 256	0.978368	0.618260	0.953940	0.73140	0.602232
UN3	512 × 512	0.984650	0.681151	0.977873	0.791776	0.673112
SU1	256 × 256	0.987534	0.739890	0.98042	0.834904	0.731232
SU1	512 × 512	0.988659	0.73934	0.974415	0.830737	0.729422
SU2	256 × 256	0.984286	0.678011	0.972411	0.788763	0.669764
SU2	512 × 512	0.991566	0.801630	0.978020	0.876640	0.788368
SU3	256 × 256	0.981021	0.645645	0.925420	0.749697	0.627331
SU3	512 × 512	0.989774	0.764320	0.976705	0.850746	0.755102
UR1	256 × 256	0.993426	0.848134	0.976801	0.903709	0.835293
UR1	512 × 512	0.994433	0.853365	0.986271	0.908020	0.846189
UR2	256 × 256	0.994207	0.858564	0.988354	0.917532	0.851052
UR2	512 × 512	0.994330	0.864658	0.983132	0.916175	0.856250
UR3	256 × 256	0.995853	0.887876	0.982891	0.929751	0.878621
UR3	512 × 512	0.997436	0.926238	0.982401	0.952210	0.915586
UR4	256 × 256	0.996129	0.899227	0.976273	0.931553	0.884272
UR4	512 × 512	0.997433	0.920265	0.977116	0.944863	0.910468
UR5	256 × 256	0.996846	0.896490	0.983463	0.928265	0.883951
UR5	512 × 512	0.996769	0.902379	0.988267	0.937194	0.894659

. All measures were computed for each architecture as well as the image sizes. The table presents the best results achieved. For each model, the visualization of image segmentation was made for comparison with the ground truth. The typical segmentation results are depicted in Figure 11.

4. Discussion

In recent years, semantic segmentation has been a subject of research in the field of petrography [27,31,32,46]. Our research addresses the niche of petrography, important in the determination of the properties of coal, a sedimentary rock with a wide usage in the industry [47]. Our previous research [34] and other studies [29,36,42] show that vitrinite, liptinite, and inertinite expose various challenges for automatic segmentation. Among the three, liptinite was the most demanding and caused the most difficulties [34,36]. Therefore, the presented work focuses particularly on the liptinite, investigating the possibility of increasing the effectiveness of segmentation with the use of various deep network architectures (but still following the general U-Net architectural paradigm). In the following part of the discussion, the potential reasons for liptinite segmentation difficulties are considered, the influence of the network training protocol on the results is analyzed, and finally, the comparison of the achieved results with the results reported in the literature is presented.

4.1. Liptinite Segmentation Analysis

The liptinite was recognized as the most difficult one for segmentation out of all three maceral groups. Some of the researchers mention it directly [34,36], while in other research it can be observed in the measures used for segmentation description [42]. The hypothesis of the cause of the phenomenon was formulated by Fan et al. [36]. The authors postulate that the problem with segmentation is caused by its usual occurrence as worm-like structures on the vitrinite [5]. The other reason suggested is its lower presence compared to the other maceral groups, leading to a higher imbalance within the data, and therefore to difficulties in training. Following the presented hypothesis, the imbalance ratio was determined for the training set, yielding 11.218, which shows that the positive class (liptinite) is indeed in the vast minority. The imbalance problem was addressed by the application of weighted binary cross-entropy as a loss function:

$${\mathcal{L}}_{BCE}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left({w_{imb}y}_i^{gt}\log y_i^{pred}+\left({1-y}_i^{gt}\right)\log \left(1-y_i^{pred}\right)\right)$$

(8)

where $w_{imb}$ is the weight used for correcting the classes imbalance. The training procedure, with the loss function described by Equation (5), showed a much better convergence than the training with ordinary cross-entropy function [34]. The above is a point towards the confirmation of the hypothesis.

To further investigate the source of the possible poor performance of the proposed networks, the trained models were used to segment images from the test set, while simultaneous computations of the segmentation quality measures (IoU, acc, prc, rcl, F₁) for each of the images. For each model, the worst cases were identified and analyzed. There were two crucial questions, for which answers were sought:

• Does the effectiveness of segmentation depend on network architecture?
• Are there any easily visible structure/texture properties that can explain the problems with segmentation?

If the answer to the first question is negative, worse performance should be observed for the same image no matter which network architecture is used, and the order of the images sorted according to the selected effectiveness measure should be almost the same for each model. Otherwise, if the network architecture is crucial for performance (its influence on performance is stronger than the image properties), it should be observed that both the worst-performing image, and the images’ order are strongly connected with the network architecture.

The investigations on the obtained data showed that, to a certain level, both effects can be observed. However, the dominant is the influence of the image itself. There were images causing problems for every network architecture tested. This is especially true for the images that resulted in the poorest segmentation. It seemed that these two images caused problems for every network architecture tested. The results for those images are presented in Figure 12. While more differences were observed for the subsequent problematic images, the group of the ‘difficult’ ones can be identified.

Analyzing the results for the worst cases, it may be observed that most difficulties were caused by the images with only a tiny area occupied by segmented liptinite. The problems arose when the image structure was complicated—where there are many areas similar in shape and size, among which only one or two are actually liptinite. Though most of such images turned out to be difficult for segmentation for each architecture, it has to be noted that ResNet-based architectures performed better, even segmenting the image correctly, as presented in Figure 13. In general, the networks trained for higher-resolution images performed better.

4.2. Training Procedure Analysis

During the experiments, two training procedures were investigated and their influence on the results was analyzed. The first approach focused on networks trained using a constant learning rate, while during the second approach, the training rate was halved every 100 epochs. The value of the learning rate in the first approach was determined using a grid search. The training was limited to 200 epochs, and the used values formed a series of 10⁻², 5 × 10⁻³, 10⁻³, 5 × 10⁻⁴, 10⁻⁴, 10⁻⁵. The selection criterium for the value used during further training was to achieve the highest IoU measure after 200 epochs. Once the starting value was established, it was used as the constant learning rate for the training for 800 epochs, and also as an initial value for the training using a scheduled learning rate. Taking into consideration that the ADAM optimizer performs the lr adaptation itself, the used learning rate established the range within which the adaptation was made [45]. The results show that generally comparable results can be achieved for both methods. However, elaborating on the correct value of the learning rate for the URx architectures turned out to be tricky. Either the low convergence or halting of learning was observed, depending on the value used. In the case of UNx and SUx, the effect was less prominent. It has to be pointed out that, during the training of RUx networks, the pre-trained ResNet backbones were used. However, the backbone was not frozen during the training, allowing for adjustment of its weights as necessary. In conclusion, it is more profitable to use learning rate scheduling than a constant value. It allows comparable results to training with constant lr. While it is still a question of how to choose the initial value and scheduling algorithm, it turned out to be less rigorous than in the case of constant lr.

4.3. Results Analysis

The results of the experiments show that the best results, in the IoU measure, were obtained for the networks with ResNet encoders. The results for the best architecture, UR3, trained on 512 × 512 images, were as high as 0.9156 in terms of IoU. The same network was also superior in terms of accuracy (0.9974) and F₁ (0.9522). The obtained result was better than that achieved in earlier experiments. The results reported in [42], which can be used as a reference for the base values for the maceral identification, reveal the best IoU for liptinite at a level of 0.83. This value was obtained for DeepLabV3+ network. It is worth noting that none of “classic” U-Net architectures (UNx, SUx) were able to achieve better results. On the other hand, the networks with ResNet-based encoders were close to or better than this result. It is known from the literature that liptinite is more difficult to segment due to its form and visual properties in microscopic images [34,36,42]. Therefore, only more sophisticated architectures, with a greater ability for visual feature extraction proved to be successful in its segmentation. On the other hand, even the U-Net architectures can be expected to deliver reasonably good and practically usable results.

The application of a simple, map-oriented attention mechanism implemented in SUx networks turned out to be moderately successful. On the one hand, it allowed for achieving better results for almost all tested cases. On the other, the improvement was not good enough to overcome the results for the URx architectures or the best values reported for DeepLabV3+. Nevertheless, these networks are reasonably good at liptinite segmentation and efficient in terms of computation and memory requirements, so in cases where only limited hardware resources are available, they are still a good choice.

5. Conclusions

Identification of macerals in microscopic images is one of the methods of coal quality assessment. The process is usually difficult, and the involvement of experienced domain experts is crucial to achieve a good quality of the results. An attempt of using deep learning networks for the semantic segmentation of one of the maceral groups, the liptinite, was made. The liptinite was chosen as the most difficult for semantic segmentation. The U-Net-based architectures were used for the segmentation. Three groups of networks were examined: a classic U-Net architecture, but with a varying depth, a U-Net architecture with feature map simple attention mechanism implemented, and a U-Net architecture using a ResNet backbone in the encoder branch. The results show that the best results were achieved using the last approach, when the encoder was based on the ResNet50 network. The results were superior, and good enough to consider the architecture as a tool for potential automatic segmentation of liptinite. The other network architectures achieved worse results but were also much more efficient in terms of memory consumption and computation power requirements. The achieved results showed that even difficult macerals, such as liptinite can be effectively segmented using U-Net-based architectures. It was also shown that the most difficult images for the segmentation are those which contain a tiny amount of liptinite in them. Only the most complex architectures (ResNet-based) were capable of successfully segmentingsuch cases. In the case of this particular architecture, the best value of the IoU measure observed was equal to 0.91. The presented results can justify stating that the domain experts can be effectively supported by computer vision tools in the identification and segmentation of maceral groups in microscopic images.