Deep Learning Method for Wetland Segmentation in Unmanned Aerial Vehicle Multispectral Imagery

Abstract

This study highlights the importance of unmanned aerial vehicle (UAV) multispectral (MS) imagery for the accurate delineation and analysis of wetland ecosystems, which is crucial for their conservation and management. We present an enhanced semantic segmentation algorithm designed for UAV MS imagery, which incorporates thermal infrared (TIR) data to improve segmentation outcomes. Our approach, involving meticulous image preprocessing, customized network architecture, and iterative training procedures, aims to refine wetland boundary delineation. The algorithm demonstrates strong segmentation results, including a mean pixel accuracy (MPA) of 90.35% and a mean intersection over union (MIOU) of 73.87% across different classes, with a pixel accuracy (PA) of 95.42% and an intersection over union (IOU) of 90.46% for the wetland class. The integration of TIR data with MS imagery not only enriches the feature set for segmentation but also, to some extent, helps address data imbalance issues, contributing to a more refined ecological analysis. This approach, along with the development of a comprehensive dataset that reflects the diversity of wetland environments and advances the utility of remote sensing technologies in ecological monitoring. This research lays the groundwork for more detailed and informative UAV-based evaluations of wetland health and integrity.

1. Introduction

Wetlands [1] are not only breathtaking natural landscapes but also essential ecosystems that play a vital role in maintaining the ecological balance of our planet. These distinctive areas, marked by the presence of water, serve as a sanctuary for a diverse array of plant and animal species [2,3]. Among their most significant functions is their capacity to support biodiversity, providing a home for countless species including birds, fish, mammals, reptiles, and amphibians. The varied vegetation within wetlands offers a source of food and shelter for many animals, underpinning the survival of numerous species. Despite the extensive benefits they offer, wetlands face ongoing threats from human activities and natural disasters. Urbanization, agriculture, and pollution often lead to the degradation of wetland ecosystems, while the drainage of wetlands for development or the redirection of water for irrigation can have severe impacts on these delicate habitats. Natural disasters such as hurricanes, floods, and droughts also present significant challenges to wetland conservation and management efforts [4].

Effective conservation and management of wetlands require precise delineation and continuous monitoring of these areas [5,6]. Mapping wetlands and understanding their ecological attributes are crucial components of this process. Unmanned aerial vehicle (UAV) remote sensing technology has emerged as a powerful tool in this context, offering numerous advantages over satellite-based remote sensing. By tracking changes in wetlands over time, stakeholders such as conservationists and policymakers can make informed decisions and implement appropriate strategies for the protection and rehabilitation of wetland ecosystems [5,7]. As illustrated in Figure 1, UAVs provide high-resolution imagery, enabling the capture of detailed information about wetland surfaces. This high resolution is particularly important for accurately delineating wetland boundaries and identifying specific features such as vegetation types and water bodies. In addition to high resolution, UAVs offer flexibility in terms of deployment [8,9,10,11,12,13,14]. They can be flown at specific locations and times, providing targeted data collection for different wetland areas. This flexibility allows for customized monitoring approaches that can adapt to the unique characteristics of each wetland ecosystem. Cost-effectiveness is another advantage of UAVs. Compared to traditional methods such as ground surveys or manned aircraft, UAVs are more affordable to operate. They require lower operating costs and can be easily controlled by trained operators, reducing the need for extensive resources and personnel. Furthermore, UAVs can capture rich and detailed wetland surface information by carrying different sensors including RGB (red, green, and blue), multispectral (MS), and thermal infrared (TIR) sensors [15,16,17,18,19]. RGB sensors function similarly to the human eye. They capture visual information in these three primary colors, creating a comprehensive and vibrant image of the landscape. For example, they can reveal the rich diversity of a wetland’s flora and fauna or provide a clear view of the water’s surface. MS sensors are designed to capture data in specific spectral bands, which can be especially useful in analyzing vegetation health and stress. For instance, they can detect subtle changes in the color or reflectance of plant leaves, which may indicate disease or nutrient deficiency. In a wetland context, they might be used to monitor the health of aquatic plants or to identify areas of invasive species. TIR sensors measure thermal radiation, which can be used to assess water temperature and energy exchange processes in wetlands. These measurements can provide crucial insights into the wetland’s health and function. For example, changes in water temperature can affect the life cycles of many aquatic species, while understanding energy exchange processes can help scientists predict how the wetland might respond to environmental changes such as global warming.

Figure 1. Wetland segmentation of UAV MS images based on deep learning (DL).

UAV-based MS imagery has emerged as a valuable tool across various domains, including agriculture, environmental monitoring, urban planning, and disaster management. This technology provides detailed spectral information that supports analysis and decision-making processes. However, extracting and interpreting this information from the complexity and variability of captured scenes remains a significant challenge. Ongoing research and advancements in image processing algorithms and techniques are essential to develop advanced methods capable of extracting meaningful data from UAV MS imagery, thereby maximizing the potential of UAV remote sensing for wetland monitoring. As illustrated in Figure 1, deep learning [20,21,22,23], image semantic segmentation methods offer great promise in this regard. Notably, deep learning models, such as fully convolutional networks (FCNs) [24], DeepLab [25,26,27,28], SegNet [29], Unet [30], MobilenetV3 [31], and those based on transformer methods [32,33,34,35,36], have demonstrated significant progress in image segmentation tasks. Among these, the Segformer [37] model stands out for its ability to achieve high accuracy efficiently. This makes it particularly suitable for processing the large-scale UAV datasets typically used in wetland studies. By training the model on extensive and diverse datasets, it can learn to recognize and classify different wetland classes, such as water bodies, vegetation types, and landforms. The application of the Segformer model to wetland monitoring can provide valuable insights into the dynamics and health of wetland ecosystems. Recent studies have demonstrated the capability of deep learning models to process complex datasets and provide detailed insights into wetland ecosystems, which is crucial for understanding their health and dynamics [38,39,40,41,42,43,44,45].

Despite the widespread use of deep learning methods for RGB three-channel visible images, these approaches are often constrained by weather variability and shadows, which obscure critical features and reduce segmentation accuracy. To address these limitations, MS imaging provides richer information compared to traditional RGB imaging. Additionally, TIR imaging captures thermal radiation emitted by objects, offering complementary data. The integration of TIR features with MS data can significantly enhance the robustness and accuracy of wetland segmentation.

This study introduces a novel approach to improve the semantic segmentation of MS UAV imagery for wetland monitoring by integrating TIR data with deep learning techniques. The key contributions of this paper are:

(i)

The study presents a novel deep learning approach for MS UAV imagery segmentation that integrates MS data and TIR features, significantly improving wetland segmentation accuracy and robustness.

(ii)

The validation, through an experimental comparison of loss functions, that the incorporation of TIR image features effectively addresses the issue of data imbalance, a common challenge that leads to reduced accuracy in segmentation tasks.

(iii)

The construction of a comprehensive dataset specifically designed for semantic segmentation of UAV MS remote sensing images. This dataset includes a diverse range of land cover types, providing a valuable resource for training and testing deep learning models.

2. Materials

The study area is situated in the northern suburb of Zhang Ye Wetland Park, Gansu Province, Northwest China, as depicted in Figure 2. This region, which is part of the Zhang Ye Wetland Park, covers a total area of approximately 1347.65 hectares. The park is known for its rich biodiversity and diverse ecosystems, including wetlands, grasslands, and various water bodies. The research focus is on the area’s ecological characteristics and land use patterns. As depicted in Figure 3, the experiment employed a UAV, specifically the DJI M600 Pro (SZ DJI Technology Co., Ltd., Shenzhen, China), equipped with both an MS camera (Red Edge-M) (MicaSense Inc., Seattle, WA, USA) and a TIR camera (WIRIS Pro Sc) (Workswell Corp., Prague, Czech Republic). The Red Edge-M camera features five bands—red, green, blue, NIR, and REG—with a spectral response range of 0.4–0.9 µm. The WIRIS Pro Sc camera operates within the spectral range of 7.5–13.5 µm and has a single band. During the flights, the UAV maintained an altitude of approximately 300 m above ground level, resulting in a ground resolution of 0.4 m. The dimensions of the MS and TIR images for the entire area are 5947 × 12,192 and 3390 × 6203, respectively [46,47].

Figure 2. Location and the color position of the UAV image of the study area.

Figure 3. (a) UAV M600 Pro; (b) MS camera (Red Edge-M); (c) TIR camera WIRIS.

3. Methods

3.1. Technical Route

Our proposed method is designed to incorporate TIR information and deep learning technology into the UAV MS remote sensing image segmentation. The framework of this method consists of several key components, including data preprocessing, segmentation model, and post-processing.

As illustrated in Figure 4, each component plays a pivotal role in the overall process. The input model primarily focuses on data preprocessing, ensuring that the input data are suitable for further analysis before passing them to the segmentation model. The segmentation model employs an advanced algorithm to analyze the input data and identify meaningful patterns or segments within them, grouping similar data points based on specific criteria or characteristics. The output model then takes the segmented data and performs additional analysis or generates the desired outputs. Detailed explanations follow.

Figure 4. Technical route of the whole strategy Segformer* + Multispectral + TIR + Cross-Entropy.

3.1.1. Data Preprocessing

The input module of the Segmentation model is designed to accommodate both single-channel and multichannel inputs. This module primarily focuses on preprocessing the input data. Initially, the MS and TIR images are resized to a uniform dimension. Subsequently, a six-band MS image is generated through the concatenation of the TIR band with the other spectral bands. For model training, a UAV dataset was constructed through data annotation and slicing. As illustrated in Figure 5, five distinct objects within the study area were annotated. The UAV images utilized in this research are registered images sourced from a previous study [46,47]. The following is the flow chart of data processing:

Figure 5. (a) TIR image; (b) MS image; (c) label image.

(i)

Image Acquisition: DJ600 Pro is equipped with a TIR and MS camera to capture TIR and MS images.

(ii)

Image Stitching: Stitch together all the smaller images to obtain a larger image of the entire study area.

(iii)

Image Resampling: The MS and thermal infrared images are resized to a consistent dimension using bilinear interpolation. This method calculates new pixel values by averaging the four nearest pixel values, ensuring a smooth transition and balancing performance with image quality.

(iv)

Band Concatenation: The different spectral bands are combined to create a six-channel MS image.

(v)

Data Annotation and Slicing: We annotated large images from the entire study area and composed a semantic segmentation dataset comprising 1000 cropped images. The dataset was partitioned into 60% for training, 30% for validation, and 10% for testing.

(vi)

Training: We employ stand-alone synchronized training methods with the Adam optimizer in PyTorch, leveraging the combined strengths of AdaGrad and RMSProp. The initial learning rate is set to 1 × 10−⁴, with a batch size of 2000 (comprising 8 images per chip) and a learning rate decay of 0.7 every 30 epochs. Dropout is applied at a rate of 0.5, and L2 weight decay is set to 1 × 10−⁵. Image preprocessing follows the same protocol as Inception. Additionally, we utilize an exponential moving average with a decay rate of 0.9999. All convolutional layers incorporate batch normalization with an average decay of 0.9.

It is worth mentioning that the wetland category also includes vegetation, which refers to plant species specially adapted to living in a humid environment, characterized by the presence of water on or near the soil surface at least part of the year, and we uniformly label it as a wetland. In contrast, the general category of vegetation excludes wetland areas and includes plant life that typically grows on well-drained soil with no permanent water table at or near the surface of the soil.

3.1.2. Segmentation Model

As shown in Figure 6, our segmentation model Segformer * is a fine-tuned version of the original Segformer model. It consists of two main components: an encoder and a decoder. The encoder plays a crucial role in extracting feature information from the input image. Based on the original Segformer encoder structure, we have made enhancements to better handle multisensor data in our Segformer * model. Specifically, we have added a convolutional layer to the input stage of the encoder, allowing us to fuse the multisensor data information through convolution. This approach enhances the locality of the extracted features and effectively preserves the spatial structure information. Our experiments have led us to choose the MixB2 encoder structure in the original Segformer. On the other hand, we have maintained the original lightweight All-MLP decoder in our model. The decoder is responsible for fusing the multi-level features extracted by the encoder with the predictive semantic segmentation mask. This fusion process enables the model to generate accurate and detailed segmentation results.

Figure 6. Segformer* segmentation model.

It is important to note that the specific modifications and choices made in the encoder and decoder components were based on our experiments and the specific requirements of our segmentation task. These modifications enhance the capabilities of the SegFormer model in handling our specific use case. Let us elaborate on these aforementioned structures.

The encoder component of the Segformer incorporates a multilayer transformer block. A convolutional layer is integrated into the input stage of the encoder to facilitate the fusion of multisensor data. The encoder employs a transformer model known as MiT (multilayer in transformer), which utilizes the overlap patch embeddings (OPEs) structure for feature extraction and downsampling of the input image. The OPE structure involves dividing the input image into patches and applying linear projections to these patches. The resulting features are subsequently processed through the multi-headed self-attention layer and the Mix-FFN (feed-forward network) layer. In summary, the encoder can be described as follows:

$$Encode{r}_{out}=MLP(GELU(Con{v}_{3\times 3}(MLP(x_{in})\pm x_{in})))$$

(1)

$$x_{in}=SE(OPM(Con{v}_{3\times 3}(im{g}_{input})))$$

(2)

where SE is the efficient self-attention mechanism; $x_{in}$ is the output features from SE; $im{g}_{in}$ is the input image; OPM is the overlapped patch merging technique, utilized to merge overlapping patches into a complete image; Mix-FFN mixes a 3 × 3 convolution and a (multilayer perceptron) MLP into each FFN.

The lightweight All-MLP decoder is a compact decoder that processes the feature maps generated by the encoder output to obtain the final semantic segmentation outcome.

$$Decode{r}_{All-MLP}=ML{P}_3(ML{P}_2(upsampling(ML{P}_1(F_i)))$$

(3)

where $F_i\left[i=1,2,3,4\right]$ are the multi-level features from the Mix-FFN encoder; MLP1 layer unified channel dimension for Fi; up-sampling represents features that are up-sampled to a 1/4th and concatenated together; the MLP2 layer is adopted to fuse the concatenated features; the MLP3 layer takes the fused feature to predict the segmentation mask M with a ${\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}\times N_{cls}$ resolution; here, Ncls is the number of categories.

3.1.3. Data Post-Processing

The output module is responsible for image post-processing, as the original image is divided into smaller chunks during the training process to alleviate the computational burden on the model. Initially, the model predicts the category labels for each sub-chunk, and subsequently, these predictions are stitched together to form the final prediction for the entire image. The primary objective of the data post-processing phase is to aggregate the segmentation results of all small image segments to obtain the comprehensive segmentation of the entire area. This is achieved by directly stitching the small patches based on their pixel indices. The details of this process are illustrated in Figure 7.

Figure 7. Data post-processing.

In Figure 7, the input image has dimensions H × W. To ensure that the sliced images maintain a consistent size, the original image is padded to dimensions H_pad × W_pad, which are divisible by the slice patch size N × N. The coordinates (r, c) denote the position of each sliced image in the original image, where r and c represent the row and column indices. The efficacy of our method is rooted in the pixel indexing technique, which leverages the original image’s pixel indices to segment and reassemble the small patches. This approach ensures that each patch is accurately placed in its original position, thereby eliminating the risk of misalignment or artifacts at the patch boundaries.

3.2. Evaluation Indexes

To assess the performance of the models, five evaluation indexes are employed here. These indexes serve as a means to attain more accurate and efficient semantic segmentation models. Their main objectives include identifying the best-performing models, comprehending performance levels in various application scenarios, recognizing model limitations and methods for enhancing their performance, as well as replicating the results obtained in prior experiments. The main metrics include intersection over union (IoU) and its mean (MIOU), as well as pixel accuracy (PA) and its mean (MPA), recall, F1 score, and precision. Detailed formulas and definitions can be found in the referenced literature [48].

3.3. Loss Function

In the pursuit of accurate wetland segmentation from multi-source remote sensing images captured by UAVs, the selection of an appropriate loss function is critical. This section delves into the rationale behind comparing two loss functions: Softmax Cross-Entropy Loss and Sigmoid Focal Loss. By conducting a thorough comparison, we aim to contribute to the existing body of knowledge by providing empirical evidence on the effectiveness of Softmax Cross-Entropy Loss and Focal Loss in the context of UAV-based multi-source remote sensing image segmentation. This will not only guide future research in similar ecological segmentation tasks but also offer practical insights for researchers and practitioners working in the field of remote sensing and environmental monitoring.

3.3.1. Cross-Entropy Loss

Softmax Cross-Entropy Loss [49] is a cornerstone in the field of semantic segmentation due to its simplicity and widespread adoption. It accepts two parameters: input data and labels. The input data are the data predicted by the model and the labels are the actual categories. The function first converts the input data to a probability distribution using the Softmax function and then calculates the Cross-Entropy loss. The Cross-Entropy loss is used to assess the difference between the probability distribution of the model output and the actual category distribution and can be used as an objective function for model optimization. The function returns a tensor representing the value of the Cross-Entropy loss, and its algorithmic process consists of two steps: Softmax and Cross-Entropy.

$$P_c=soft\max (Z_{i,c})={\displaystyle \frac{e^{Z_{i,c}}}{{\displaystyle \sum _{j=1}^c{e}^{Z_{i,j}}}}}$$

(4)

where P is the probability distribution generated by the Softmax function with x_i denoting the output value associated with class i and n representing the total number of output categories. The Softmax function transforms the outputs into a probability distribution, assigning probabilities between 0 and 1 to each class and providing accuracy assessments for each respective category.

$$CrossEntropy=-{\displaystyle \frac{1}{\mathrm{M}}}{\displaystyle \sum _{M=1}^M}{\displaystyle \sum _{c=1}^c{y}_c\log (P_c)}$$

(5)

where c represents the label assigned to the sample; y_c represents the one-hot encoding of the label. y_c equals 1 if the sample falls into category c, and 0 otherwise. Pc represents the probability distribution obtained through the utilization of the softmax function.

3.3.2. Sigmoid Focal Loss

Sigmoid Focal Loss [50] is a loss function designed to address the data imbalance issue by reducing the weight of easily classified samples, allowing the model to focus more on hard-to-classify samples during training. In multi-label segmentation tasks, the Sigmoid Focal Loss is computed by calculating the loss for each class separately and then summing the weighted losses across all classes.

$$FL={\displaystyle \frac{1}{N}}{\displaystyle \sum _{\mathrm{i}=1}^N{\displaystyle \sum _{t=1}^c\left(-{\alpha }_t{\left(1-P_{it}\right)}^{\gamma }\log P_{it}\right)}}$$

(6)

N is the total pixel number. C is the class number. $P_{it}$ is the model’s predicted probability for each class. α (Alpha) and γ (gamma) are the key parameters, which are used to adjust the model’s focus on different classes of samples, especially to improve the recognition of difficult samples to optimize the performance of the multi-class segmentation task. α controls the balance between the easy and hard examples, and γ focuses the model attention on the hard, misclassified examples.

4. Results

4.1. Comparative Analysis of Single-Channel Segmentation Performance

To analyze the role of thermal infrared image features in image semantic segmentation, comparison experiments were conducted on different single-band images.

Based on the experimental results presented in Figure 8 and Table 1 and Table 2, thermal infrared image features demonstrate superior performance compared to other single-channel segmentation methods. The incorporation of thermal data significantly enhances adaptability, accuracy, and robustness across various applications. To further assess the efficacy of our approach, we conducted a comparative analysis with other methods. Table 3 provides a comprehensive comparison of the performance of different techniques in thermal infrared (TIR) single-channel segmentation, evaluating their effectiveness in segmenting TIR images.

Figure 8. Visualization of single-channel image segmentation. (Note: Segformer* represents the finetuned Segformer architecture in our research.).

Table 1. Pixel accuracy of different single-channel image segmentations.

Method	Data	PA(%)						MPA
		Background	Wetland	Water	Road	Building	Vegetation
Segformer [37]	R	73.22	82.12	42.63	53.98	74.07	37.20	72.68
	G	73.34	86.33	68.48	58.74	60.43	56.14	79.30
	B	72.50	90.44	64.66	39.09	25.91	10.35	79.00
	REG	47.49	02.98	73.28	01.85	04.00	00.00	69.03
	NIR	25.65	01.96	89.04	01.13	04.00	21.31	74.84
	TIR	67.50	93.94	75.00	76.89	70.54	84.38	82.14
Segformer*	R	66.78	83.13	40.24	46.13	32.70	58.98	71.52
	G	74.21	88.99	80.17	63.79	75.93	46.32	82.74
	B	71.70	90.60	79.88	61.56	77.07	65.05	83.51
	REG	43.14	92.74	85.25	04.22	03.86	00.37	75.63
	NIR	71.76	95.21	92.93	37.69	40.69	40.73	80.60
	TIR	74.78	92.44	77.45	80.45	61.82	77.42	83.28

Table 2. Intersection over union of different single-channel image segmentations.

Method	Data	IOU(%)						MIOU
		Background	Wetland	Water	Road	Building	Vegetation
Segformer	R	48.97	68.66	32.73	48.35	52.43	32.62	47.29
	G	52.40	77.26	54.70	46.89	50.86	40.97	53.83
	B	48.61	80.40	54.75	34.89	23.14	09.40	41.87
	REG	04.56	07.05	04.96	00.01	00.00	00.00	20.80
	NIR	21.52	01.79	58.58	01.13	04.00	14.50	28.96
	TIR	55.03	80.17	68.86	45.88	61.54	43.48	59.16
Segformer*	R	51.55	68.68	28.17	40.64	25.07	25.90	40.00
	G	58.73	80.28	62.50	55.70	48.96	40.05	57.70
	B	57.03	81.96	65.62	54.42	47.52	48.51	59.18
	REG	32.40	80.56	54.74	03.53	03.76	00.36	29.23
	NIR	58.14	87.48	80.14	32.68	35.88	38.45	55.46
	TIR	58.63	81.43	70.27	53.56	55.08	67.20	64.36

Table 3. Pixel accuracy of different methods in TIR single-channel segmentation.

Method	PA(%)						MPA
	Background	Wetland	Water	Road	Building	Vegetation
FCN [24]	38.89	53.87	77.76	02.77	19.40	01.15	52.05
MobilenetV3 [31]	20.73	71.10	18.55	07.46	19.23	04.89	44.49
DeepLabv3 [29]	80.07	72.82	65.17	49.20	44.85	57.32	71.49
Segformer [37]	69.31	94.58	89.96	81.61	81.36	89.37	86.41
Segformer*	75.55	92.44	91.15	90.88	77.90	85.27	87.06

As illustrated in Table 3 and Table 4, along with Figure 9, the Segformer* method demonstrates superior performance relative to other methods. Table 2 underscores the superior performance of TIR image features compared to other single-channel segmentation techniques. The performance metrics presented in the table provide valuable insights into the strengths and weaknesses of each method, thereby facilitating the selection of the most appropriate approach for TIR single-channel segmentation based on specific requirements.

Table 4. Intersection over union of different methods in TIR single-channel segmentation.

Method	IOU(%)						MIOU
	Background	Wetland	Water	Road	Building	Vegetation
FCN [24]	29.70	41.62	33.45	02.62	18.34	01.14	21.15
MobilenetV3 [32]	15.19	46.36	15.01	01.84	05.53	02.42	14.39
DeepLabv3 [29]	44.94	67.17	56.10	26.96	41.56	43.38	46.68
Segformer [33]	59.44	85.98	77.30	74.06	65.83	83.00	74.27
Segformer*	63.51	85.88	78.22	76.20	64.68	83.08	75.26

Figure 9. Different method comparison results of single-channel image segmentation.

4.2. Comparative Analysis of Multichannel Segmentation

In this experiment, we conducted a comparative analysis of multichannel images to elucidate the role of thermal infrared (TIR) image features in multichannel segmentation tasks. The channels examined included RGB (three channels), MS (five channels), and MS+TIR (six channels). Specifically, we compared the performance of our strategy with that of the original Segformer in multichannel image segmentation. Our primary focus was to analyze the impact of TIR image features on segmentation outcomes and to compare the performance differences between models trained on RGB images and those trained on multichannel images.

From the results presented in Table 5, Table 6, Table 7 and Table 8, and Figure 10 and Figure 11, it is evident that the Segformer* model excels in multichannel image segmentation. This superior performance can be attributed to its unique encoder architecture and sophisticated training strategy. The model effectively leverages multichannel image information, achieving more accurate segmentation results through advanced feature extraction and segmentation techniques. By incorporating additional channels, the model can capture more comprehensive information, thereby enhancing the accuracy and robustness of the segmentation outcomes. The inclusion of thermal infrared channels notably boosts the model’s performance, as thermal images provide valuable insights into temperature variations, facilitating the identification of defects, precise segmentation of targets, and detection of material changes.

Table 5. Pixel accuracy of multichannel image segmentation.

Method	Data	PA(%)						MPA(%)
		Background	Wetland	Water	Road	Building	Vegetation
Segformer	RGB	42.01	89.63	88.57	82.57	32.47	91.31	78.65
	MS	76.40	86.85	81.95	57.14	78.25	54.68	82.26
	MS + TIR	83.32	91.48	79.25	55.43	81.55	73.76	86.50
Segformer*	RGB	73.73	91.88	79.07	70.54	67.83	71.28	84.89
	MS	71.79	91.54	86.56	66.05	71.25	62.33	85.04
	MS+TIR	85.29	91.84	77.32	69.58	86.26	78.03	87.39

Table 6. Intersection over union of multichannel image segmentation.

Method	Data	IOU(%)						MIOU(%)
		Background	Wetland	Water	Road	Building	Vegetation
Segformer	RGB	37.03	83.82	66.76	49.13	26.34	21.89	47.50
	MS	55.48	80.06	68.41	45.04	36.92	46.45	55.39
	MS + TIR	60.06	87.24	73.07	53.21	52.97	59.33	64.31
Segformer*	RGB	57.40	83.55	73.20	57.64	45.77	50.17	61.29
	MS	55.19	86.73	67.76	57.80	54.17	48.53	61.70
	MS + TIR	63.46	86.76	71.84	64.98	67.29	68.81	70.52

Table 7. Pixel accuracy of different methods in multichannel image segmentation.

Method	PA(%)						MPA
	Background	Wetland	Water	Road	Building	Vegetation
FCN [24]	49.95	81.31	54.81	37.48	18.04	05.35	66.11
MobilenetV3 [32]	46.82	86.43	27.60	20.00	05.72	00.00	62.23
DeepLabv3 [24]	61.21	88.92	63.48	75.79	46.92	77.41	76.41
Segformer [33]	81.00	94.58	87.69	77.75	91.27	79.57	89.86
Segformer*	85.25	95.42	90.51	82.68	90.94	61.92	90.35

Table 8. Intersection over union of different methods in multichannel image segmentation.

Method	IOU(%)						MIOU
	Background	Wetland	Water	Road	Building	Vegetation
FCN [24]	30.79	64.88	41.47	22.24	14.77	05.25	29.90
MoilenetV3 [32]	26.44	63.93	24.93	10.99	05.01	00.00	21.88
DeepLabv3 [29]	47.60	72.89	52.92	42.90	40.06	41.51	49.65
Segformer [33]	70.57	89.36	84.46	72.78	75.76	46.61	73.26
Segformer*	66.46	90.46	77.71	70.27	70.15	67.75	73.87

Figure 10. Comparison results of multichannel image segmentation.

Figure 11. Different method comparison results of multichannel image segmentation.

4.3. Limitations of the Proposed Method

Our strategy has effectively demonstrated the utility of thermal infrared imagery in enhancing segmentation accuracy. However, the approach is not without limitations, which we discuss in this section. Data imbalance is a significant challenge in the dataset we constructed, as illustrated in Table 9. This imbalance can lead to biased model performance, where the model may overfit the majority classes and underperform on the minority classes, potentially skewing the model’s predictive capabilities.

Table 9. Class pixel number in our dataset.

Class	Background	Wetland	Water	Road	Building	Vegetation
Pixel number	13,619,456	39,929,764	13,833,990	992,358	281,814	3,848,442

Our strategy has effectively demonstrated the utility of thermal infrared imagery in enhancing segmentation accuracy. However, the approach is not without limitations, which we discuss in this section. Data imbalance is a significant challenge in the dataset we constructed, as illustrated in Table 9. This imbalance can lead to biased model performance, where the model may overfit the majority classes and underperform on the minority classes, potentially skewing the model’s predictive capabilities. The minor classes include roads, buildings, and vegetation, while the major classes are wetlands and water. To investigate the impact of the Segformer model, the loss function, and thermal infrared image features on addressing the class imbalance, we designed a series of comparative experiments. The following sections detail the experimental setup and findings. Initially, we identified the optimal values for α and γ in the Sigmoid Focal Loss function by conducting comparative experiments, testing various parameter combinations to determine the best hyperparameter settings for addressing class imbalance. Subsequently, we compared the Sigmoid Focal Loss function (with optimal values α = 0.90 and γ = 4) and the Cross-Entropy loss function within the Segformer* framework, focusing on their impact on channel segmentation results. This analysis aimed to determine whether the Focal Loss function could effectively mitigate data imbalance by prioritizing hard-to-classify examples.

As illustrated in Figure 12, we can elaborate on the impact of different loss functions on the segmentation outcomes. Specifically, the Sigmoid Focal Loss function was applied to address the data imbalance issue, particularly for the minority classes such as roads, buildings, and vegetation. The adoption of the Focal Loss function proved to be particularly effective in boosting the segmentation precision for the minority classes. This loss function is designed to focus on hard-to-classify examples and reduce the relative loss for well-classified examples, thereby providing a greater emphasis on the minority classes during the training process. As a result, the PA and IOU scores for roads, buildings, and vegetation showed a noticeable improvement when compared to the segmentation performance achieved with the standard Cross-Entropy loss. However, despite the Focal Loss’s efficacy in enhancing the segmentation accuracy for the minority classes, the overall segmentation performance, when evaluated across all categories, was found to be slightly inferior to the strategy that employed the Cross-Entropy loss in conjunction with thermal infrared data. This strategy demonstrated a more consistent improvement in segmentation metrics across the board, indicating that while the Focal Loss does address the issue of class imbalance, the complementary information provided by thermal infrared data may have a broader and more uniform positive impact on the segmentation results.

Figure 12. Comparison results of different channel image segmentation algorithms with different loss functions.

In addition, combining the experimental results of Figure 12 and Figure 13, it is evident that the inclusion of thermal infrared image features significantly enhanced the segmentation performance across all classes. For the minority classes such as roads, buildings, and vegetation, the accuracy and IoU scores showed an increase when thermal infrared data were incorporated. This indicates that thermal information helped to distinguish between classes that may appear similar in visible light, thus aiding in the identification of minority classes, and which have a relatively stable trend in accuracy across all categories. In the case of wetland and water classes, which are the majority classes in our dataset, the addition of thermal infrared features also led to a slight improvement in segmentation performance. This indicates that thermal data can complement visible imagery even for well-represented classes.

Figure 13. Examples of comparations of different single-channel segmentation for minor class.

In summary, the integration of thermal infrared features into the segmentation process yielded a relatively stable trend of increased accuracy for all classes, with particularly significant gains for the minority classes. Although the Focal Loss function successfully elevated the segmentation precision for the less represented classes, the comprehensive segmentation strategy that combined visible imagery with thermal infrared data and used the Cross-Entropy loss yielded superior overall segmentation performance, suggesting that a multifaceted approach to data and loss function selection is key to achieving optimal segmentation results.

As can be seen in Figure 14, there are some differences in the recall, precision, and F1 scores of the categories, but overall, the trend of precision changes in different categories is relatively balanced. Moreover, even on a few categories with small sample sizes, the model maintains relatively high-performance metrics, showing good generalization ability. Thus, our strategy performs satisfactorily on such unbalanced datasets, proving its effectiveness and robustness in dealing with the problem of data imbalance.

Figure 14. Evaluation of Segformer* + MS +TIR+ Cross-Entropy strategy in the presence of data imbalance.

This performance is largely attributed to the Segformer* model’s ability to effectively integrate multi-scale features and the robustness of its transformer architecture, which allows it to capture long-range dependencies and contextual information more effectively. Additionally, the thermal infrared image features play a crucial role in addressing the issue of data imbalance by enriching the feature set with unique spectral signatures that are not available in the visible spectrum, thereby providing a more discriminative representation for both minority and majority classes.

This adaptability to varied data distributions and the utilization of rich spectral data underscore the model’s potential for advancing the state-of-the-art in remote sensing imagery analysis. The Segformer*’s innovative design is particularly well-suited for remote sensing applications where the distribution of classes in the dataset may be skewed, and additional spectral information from thermal infrared images can provide valuable insights for enhanced feature discrimination and more reliable land cover classification.

5. Conclusions

This study introduces a novel wetland semantic segmentation method for multispectral remote sensing images acquired from UAVs, augmented by thermal infrared (TIR) image features. Employing the Enhanced Segformer* model within a deep learning framework, our approach achieves exceptional segmentation accuracy, with a mean pixel accuracy (MPA) of 90.35% and a mean intersection over union (MIOU) of 73.87%. For wetland classification, we attained a pixel accuracy (PA) of 95.42% and an intersection over union (IOU) of 90.46%. The incorporation of TIR imagery has been pivotal in enhancing segmentation performance. However, the reliance on TIR data introduces additional costs and complexities, necessitating future research to explore alternatives such as synthetic thermal imagery generation and the use of thermal imagery as a supplementary tool. Additionally, while Focal Loss has enhanced segmentation precision for minority classes, its efficacy without TIR data remains uncertain. Thus, optimizing loss functions and addressing data imbalance are critical areas for future investigation. Overall, this study not only advances remote sensing analysis for wetland mapping but also lays the groundwork for integrating diverse data sources into deep learning models, thereby paving the way for more robust and versatile environmental monitoring solutions.