Algorithm for Detecting Trees Affected by Pine Wilt Disease in Complex Scenes Based on CNN-Transformer

Abstract

Pine wilt disease, a highly destructive forest disease with rapid spread, currently has no effective treatments. Infected pine trees usually die within a few months, causing severe damage to forest ecosystems. A rapid and accurate detection algorithm for diseased trees is crucial for curbing the spread of this disease. In recent years, the combination of drone remote sensing and deep learning has become the main methods of detecting and locating diseased trees. Previous studies have shown that increasing network depth cannot improve accuracy in this task. Therefore, a lightweight semantic segmentation model based on a CNN-Transformer hybrid architecture was designed in this study, named EVitNet. This segmentation model reduces network parameters while improving recognition accuracy, outperforming mainstream models. The segmentation IoU for discolored trees reached 0.713, with only 1.195 M parameters. Furthermore, considering the diverse and complex terrain where diseased trees are distributed, a fine-tuning model approach was adopted. After a small amount of training, the IoU on new samples increased from 0.321 to 0.735, greatly enhancing the practicality of the algorithm. The model’s segmentation speed in the task of discolored trees identification meets the requirements of real-time performance, and its accuracy exceeds that of mainstream semantic segmentation models. In the future, it is expected to be deployed on drones for real-time recognition, accelerating the entire process of discovering and locating infected trees.

1. Introduction

Currently, nearly 10 pine tree diseases are prevalent, among which pine wilt disease (PWD), caused by the pine wood nematode (PWN), is the major one and is highly contagious and extremely destructive [1]. It was first discovered in North America [2] and is now prevalent in countries such as China, Japan, South Korea, the United States, Canada, Portugal, and Mexico. This disease has caused widespread death of pine trees in multiple countries, resulting in significant ecological damage [3]. In Japan, the first case of PWD was discovered in the early twentieth century, and by 2019, the disease had spread in all prefectures except Hokkaido. In China, the first infected pine tree was found in Nanjing in 1982, with a spread rate of 7000~10,000 hectares per year [4], and by 2023, the disease had spread to 19 provinces (encompassing 701 counties and cities) [5]. When the PWN invades pine trees, it first destroys the xylem tissue cells, hindering water transport and causing the needles to gradually turn reddish-brown [6]. Once infected, pine trees usually die within a few months [7]. Since there are no effective treatment measures for this disease, the most effective control measure is to locate and fell the infected trees for clearance [8]. A more pressing issue is that the PWN is now spreading to areas with previously unsuitable climates, moving northward and to higher altitudes [9]. To effectively prevent the further spread of pine wilt disease and avoid greater ecological damage, there is an urgent need to accelerate the process of locating infected trees for felling and crushing, with the goal of harmless treatment [10].

Artificial intelligence (AI) has shown transformative potential across multiple fields. For instance, Koy et al. [11] shows AI optimizing high-frequency trading strategies; Purohit et al. [12] highlights deep learning’s significance in medical image segmentation; and Chiphiko et al. [13] underscores AI’s role in enhancing IoT security protocols. These examples show AI’s broad applicability. Similarly, in pest and disease identification, AI significantly boosts efficiency [14,15]. Traditional monitoring of pine wilt disease mainly relies on manual surveys, which are time-consuming and labor-intensive. Moreover, most pine trees are located in areas with steep mountains and dense forests, making it difficult for survey personnel to reach [16]. Monitoring using drone and satellite remote sensing technologies can greatly improve efficiency. However, satellite remote sensing is still limited by its low spatial resolution and long revisit cycle, resulting in low accuracy and long monitoring intervals [17]. Drone remote sensing has the advantages of high flexibility, high spatial resolution, and low cost. Using drone remote sensing technology to monitor PWD-affected discolored trees (hereinafter referred to as “discolored trees”) has become the main method for detecting and locating discolored trees [18]. There have been many studies using object detection methods to identify discolored pine trees [19], but object detection methods often struggle with detection in scenes with overlap [20] and small targets [21]. In comparison, semantic segmentation is more suitable for remote sensing classification and recognition tasks. Remote sensing images are usually much larger than natural images, and semantic segmentation has a natural advantage in processing large-sized images. Additionally, semantic segmentation networks can extract delicate texture features of discolored trees at the pixel level, obtaining precise segmentation boundaries, which facilitates rapid estimation of the affected area.

Semantic segmentation was initially transformed to classification tasks through a sliding window approach, but this method was computationally redundant and inefficient. The advent of Fully Convolutional Networks (FCNs) [22] established the indelible status of encoder–decoder architecture-based networks in the domain of semantic segmentation. In the subsequent years, a plethora of segmentation networks grounded in the encoder–decoder structure emerged. For instance, UNet [23], which achieved state-of-the-art (SOTA) performance in the medical field when it was proposed, is a classic U-shaped symmetric network of the encoder–decoder architecture. It is typically suitable for datasets with low contrast, imbalance, and requirements for preserving fine details, such as medical datasets. In the general domain, Google introduced the prominent Deeplab series, which have now evolved to Deeplabv3+ [24]. Its excellent performance stems from the larger receptive field obtained through atrous convolution and the feature maps of different atrous rates fused via ASPP (Atrous Spatial Pyramid Pooling). In recent years, with the research on Transformers [25], technologies such as large models and multimodality have been better implemented. In the field of semantic segmentation, Meta introduced SAM [26]. This algorithm performs interactive segmentation and can “segment anything” without the need for retraining. As deep learning continues to evolve, current research trends are increasingly focused on combining a Transformer with a CNN to fully leverage the unique advantages of both.

Current research on semantic segmentation for discolored tree recognition mainly falls into two categories. One applies a single segmentation algorithm for identification, while the other evaluates the performance of multiple algorithms. There is a lack of specialized segmentation algorithms designed specifically for discolored tree recognition. As a result, mainstream segmentation algorithms often fall short in terms of accuracy and speed, and their robustness in complex environments leaves much to be desired. There is a notable absence of segmentation models specifically tailored for this task. In this study, we employed a lightweight Vision Transformer (ViT) as the encoder to capture the global features of discolored trees, thereby enhancing the network’s feature extraction capabilities. A Convolutional Neural Network (CNN) was utilized as the decoder, leveraging its inherent spatial inductive bias to ensure the precision of the decoding process. On the other hand, previous studies have overlooked the significant challenges to model generalizability posed by varying terrains, diverse tree species, and different sampling equipment. This task typically demands a high level of model generalizability. The optimal fine-tuning strategy will enable the model to maintain high accuracy with minimal new sample learning, thus enhancing its practicality. The main aims of this study are as follows:

(1)

To construct two high-quality semantic segmentation datasets from distinct terrain areas using different sampling equipment.

(2)

To develop an efficient and accurate discolored tree segmentation model (EVitNet) through the innovative integration of a lightweight ViT feature extraction network and a CNN upsampling method.

(3)

To identify the optimal fine-tuning strategy that improves the model’s practicality, allowing it to achieve high accuracy with only a small amount of new sample learning.

2. Materials and Methods

2.1. Data Collection

Two datasets of PWD-affected discolored trees were utilized in this study, with different sampling equipment being used in the two sampling areas (Figure 1). Dataset A was collected in Qingdao, Shandong (36°04′0.98″ N, 120°22′57.59″ E), using a DB-II fixed-wing drone equipped with a Sony Alpha 7R II camera, while Dataset B was collected in Yantai, Shandong (37°28′25.71″ N, 121°11′51.14″ E), using a DJI Phantom 4 Pro. The sampling area in Qingdao, Shandong, is mountainous terrain with an average elevation of 360 m and a maximum elevation of 1132.7 m. The main pine species are Pinus densiflora [27], and the photographed area covers 200 km². The sampling area in Yantai, Shandong, is flat coastal terrain with an average elevation of 9 m. The area of this sampling site is 0.7 km², and the main pine species are Pinus densiflora and Pinus thunbergii [28]. During sampling in both areas, images were captured at equal distances, with a flight direction overlap of no less than 75% and a lateral overlap of no less than 75%. Ultimately, more than 17,000 raw images with a resolution of 7952 × 5304 were obtained in Qingdao, Shandong, and a total of 1089 raw images with a resolution of 5472 × 3648 were obtained in Yantai, Shandong.

During sampling, equipment was selected based on the scene, with detailed dataset information (

Table 1. Details of Dataset A and Dataset B.

	Dataset A	Dataset B
Collection system	DB-II + Snoy alpha 7 R II	DJI Phantom 4 Pro
Resolution of raw data	7952 × 5304	5472 × 3648
Amount of raw data	17,000	1096
Area	200 km2	0.7 km2
Location	Qingdao, Shandong	Yantai, Shandong
Principal species	Pinus densiflora	Pinus densiflora and Pinus thunbergii

). In Qingdao, Shandong, given the large sampling area and high average elevation of discolored trees, a DB-II fixed-wing drone (Dabai Technology Co. Ltd.; China) equipped with a full-frame camera was used. Fixed-wing drones, with fast cruising speeds and long endurance, are suitable for large-scale data collection. They also have a high cruising altitude, fitting mountainous terrain. A high-pixel full-frame camera (Sony Alpha 7 R II, Sony Group Corporation, Japan) was used due to the high flight altitude. However, fixed-wing drones require a suitable runway for takeoff and landing. In contrast, quadcopters, though having lower sampling resolution, feature lower usage costs and fewer terrain restrictions, making them more appropriate for smaller scenes, such as those in Yantai, Shandong, so we chose the DJI Phantom 4 Pro (Da-Jiang Innovations Science and Technology Co., Ltd.; China) for sampling in Yantai.

2.2. Data Process

Raw aerial images contain a substantial amount of repetitive content. By utilizing Pix4DMapper software (version 4.5.6), the original aerial photographs were stitched together to create orthophotos (Figure 1), thereby achieving the goal of eliminating redundancy. Under the guidance of forest disease control experts, annotations were carried out based on the visual characteristics of discolored pine trees in remote sensing images. After pine trees are infected with PWN, their needles change from green to yellow or red, and resin secretion decreases [29]. These changes can be distinguished through spectral and texture feature analysis, which differentiates discolored trees from healthy pines and other similar-colored objects, enabling precise labeling. Subsequently, a script was employed to generate corresponding binary images based on the annotation information. Both the orthophotos and the binary images with annotation information were cropped to a size of 256 × 256. A total of 31,500 samples were obtained from the data collected in Qingdao, Shandong, with 2981 of them containing at least one discolored tree lesion. From the data gathered in Yantai, Shandong, 3753 samples were acquired, among which 524 had discolored tree lesions. Both datasets are characterized by extremely imbalanced sample distribution.

To facilitate the selection of samples containing discolored tree lesions, this study proposes an entirely new sample processing workflow, with the pseudocode presented in pseudocode 1. After processing with this code, samples containing lesions and their annotations can be stored in directory 1, while all samples can be stored in directory 2. This approach eliminates the need for manual screening of the dataset and also makes it easier to generate samples of different sizes in the future.

Pseudocode 1: Generate samples from xml annotation information

Open orthophoto to read image size and geographical transformation parameters Create the whole annotated image For annotations in xml file:   Convert the UTM coordinate list to the pixel coordinate list   Draw polygons according to pixel coordinates on the annotated image and fill them   For blocks (256 × 256) in annotated image:     If the block has annotation:       Save the label and corresponding orthophoto block to Directory 1     Save the label and corresponding orthophoto block to Directory 2

To facilitate subsequent experiments, samples containing lesions from the two sampling sites were used as two separate datasets, with each divided into a training set and a testing set at a 9:1 ratio. Some samples from the two datasets contain both typical discolored tree samples and various samples that may be difficult to identify (Figure 2). Dataset A includes situations such as similar color and texture of interfering objects and discolored trees, large-scale differences in discolored trees, various outlines, and different lighting conditions. In Dataset B, the sandy background poses strong interference to identification. Due to differences in terrain, tree species, and sampling equipment between the two sites, the two datasets differ significantly. The dataset from Qingdao, Shandong, is named Dataset A, and the dataset from Yantai, Shandong, is named Dataset B. Dataset A was used for model design and validation, while Dataset B was used to determine the fine-tuning learning strategy.

In the field of deep learning, the quality of data samples determines the final model performance and generalization ability. To ensure the model’s generalizability, data augmentation is commonly used to enhance the quality of data samples, generally by adding noise to the original images and performing random transformations [30]. Considering that the sample size is sufficient in this study, the efficient online augmentation method was adopted. During training, color space transformation and adjustment, random flipping, random blurring, random adding gray strips, random resizing, and random rotation were performed on the samples and labels to enhance the data, with the effects of augmentation (Figure 3).

2.3. Methods

2.3.1. The Semantic Segmentation Model for Discolored Trees, EVitNet

To enhance the accuracy and efficiency of discolored trees segmentation in complex scenes, this study constructed a lightweight discolored trees segmentation model, EVitNet (Easy Vision Transformer Net), based on Transformer and CNN as the basic network structures, featuring an encoder–decoder architecture. The algorithm is robust and capable of meeting the needs for precise and rapid segmentation of discolored trees in diverse scenarios. EVitNet combines the advantages of Transformer and CNN. The encoder section alternately employs CNN and Vision Transformer, enabling global feature encoding through Transformer while preserving spatial inductive bias via CNN. This effectively avoids the drawback of Transformer requiring massive data for training and reduces training costs [31]. The encoder with added Transformer blocks can better extract global features from the input, expanding the receptive field to the size of the entire image and enhancing the effectiveness of feature extraction. The decoder section draws on the upsampling structure of U-Net, imitating its skip connections to preserve shallow detail feature information, ensuring the precision of subsequent upsampling.

In the network structure diagram of EVitNet (Figure 4), for the input image, lightweight convolution based on depthwise separable convolution is employed. After the last three convolutional blocks, MobileVit Blocks are incorporated, enabling the network to perform global reorganization among sufficiently rich feature information. Before each downsampling, skip connections are used to retain all detail features. Each decoder in the network consists of a standard convolution and an expanded convolution. The expanded convolution enhances the upsampling accuracy without increasing the network parameters. After a series of upsampling operations, the final feature map is obtained, which is then mapped to two channels through a fully connected layer to complete the classification for each pixel.

2.3.2. EasyVit Backbone Network

In recent years, the Transformer architecture has revolutionized the fields of natural language processing and computer vision. In the realm of computer vision, ViT has achieved SOTA results in some tasks. However, ViT has a large number of parameters. Experiments have shown that using ViT with a large number of parameters does not lead to higher segmentation accuracy but significantly increases the computational burden. In this study, a lightweight ViT structure was adopted based on the design of MobileViT [32]. EasyViT was constructed using MobileViT Block and MobileNetv2 Block [33], and EasyViT was used as the feature extraction backbone network.

The MobileVit block commences with a standard convolutional layer, which is tasked with capturing the local feature information of the input tensor X. Subsequently, the feature information is projected into a higher-dimensional space through a convolutional operation, thereby enhancing the expressive power of the features and facilitating their subsequent flattening and input into the Transformer for global feature encoding. After obtaining the encoded results, they are restored to the original shape, and then the feature information is projected back to the input size through a convolutional operation and concatenated with the skip connection of the input X. Through a standard convolutional layer, the feature map with 2C channels is converted back to C channels, thus completing the global feature encoding in the spatial domain.

The MobileNetv2 block, a core component of MobileNetv2, shares a similar concept with the MobileViT block. Initially, project the feature information into a higher-dimensional space before further refining the features. For an input X, the MobileNetv2 block first expands the number of channels in the feature map through convolution. It then applies depthwise separable convolution and pointwise convolution separately. Replacing standard convolution with two separate convolutions reduces the number of parameters and computations. Finally, when the input and output feature maps have the same size, a residual connection is used to concatenate the two feature maps.

2.3.3. Decoder with Expanded Convolution

Controlling the number of network parameters is crucial for segmentation speed. Therefore, this study does not increase the parameters. Instead, it follows the design of MMDM [34] and replaces the traditional 3 × 3 standard convolutional blocks with three convolutions of 3 × 3, 1 × 3, and 3 × 1, effectively improving the accuracy of network upsampling. During training, the 9 parameters of the standard convolutional kernel expand to 15 parameters and then revert to 9 parameters during subsequent deployment, without increasing the number of parameters. This process is referred to as expanded convolution.

The expanded convolution block is illustrated (Figure 5). Since convolution has additivity, the outputs of several expanded convolutions are summed, which is equivalent to the outputs obtained by summing these convolution kernels. Therefore, expanded convolutions can be used to replace traditional convolutions. During training, the original convolution is decomposed into three convolutions, allowing for more parameters to refine the learning of feature processing methods. When deploying the model, the expanded convolution is merged into a standard convolution without adding extra parameters.

2.3.4. Model Fine-Tuning Method

After years of clearance and control, PWD is currently sporadic in multiple regions. The tree species affected by PWD are numerous and span many countries, making it difficult to achieve complete sampling coverage for the large number of tree species. Moreover, the economic viability of large-scale sampling and annotation is low. In response to this challenge, this study proposes fine-tuning the weights obtained through training, enabling the model to learn from a small amount of new data and maintain a high accuracy on the new dataset.

Fine-tuning the model is implemented through transfer learning, the main idea of which is to transfer labeled data or knowledge structures from related domains to enhance learning efficiency in the target domain. The premise of using transfer learning is that the source and target domains share some degree of similarity. The prior knowledge learned in the source domain is helpful for learning in the target domain. If the two domains are too different, using transfer learning may lead to negative transfer. In this study, both datasets are about PWD-affected discolored trees, which are highly similar and thus suitable for transfer learning. Dataset A, introduced in Section 2, is used as the base dataset because of its large scale, which allows for sufficient model training. Dataset B is used as the transfer learning validation dataset. The process of the two datasets and model fine-tuning is shown in Figure 6. The model is first trained conventionally on Dataset A. After training, based on the weights obtained from the previous training, a portion of samples from Dataset B is selected as incremental data for learning. A series of comparative experiments are designed to determine the optimal transfer learning strategy, focusing on whether to freeze the backbone network weights and what proportion of data to use for incremental learning.

2.4. Evaluation Metrics

In this study, four accuracy evaluation metrics were primarily used for the single category of discolored trees: IoU (Intersection over Union), F1 score, precision, and recall. Due to the presence of a large number of easily recognizable background negative samples in the samples, the mIoU (mean Intersection over Union) or mp (mean precision) would be artificially high, which is not conducive to comparing model performance. The identification accuracy of PWD-affected discolored trees by the model is the main focus, hence the primary use of single-category evaluation metrics for discolored trees. Initially used in object detection, IoU is employed to determine the difference between the predicted bounding box area and the true area, calculated by dividing the intersection area of the two regions by their union area. The value of IoU ranges from 0 to 1, with a value closer to 1 indicating a more accurate predicted box location. In semantic segmentation, the situation is similar, except that the shape of the detection and annotation areas changes from rectangles to irregular shapes. In a task with k categories, the calculation method for the IoU of the ith category is shown in Formula (1):

$$IoU={\displaystyle \frac{Area of Overlap}{Area of Union}}={\displaystyle \frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}+{\sum }_{j=0}^k{p}_{ji}-p_{ii}}}$$

(1)

In this context, ${\mathrm{p}}_{ij}$ represents the number of pixels where the true label is class $i$ and the predicted result is class $j$. When $i=j$, ${\mathrm{p}}_{ij}$ indicates the number of correctly predicted pixels, and when $i\ne j$, it indicates the number of incorrectly predicted pixels. Similarly, the calculation methods for precision and recall of class $i$ are shown in Equation (2) and Equation (3), respectively.

$$Precision={\displaystyle \frac{TP}{TP+FP}}={\displaystyle \frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ji}}}$$

(2)

$$Recall={\displaystyle \frac{TP}{TP+FN}}={\displaystyle \frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}}}$$

(3)

In this context, TP represents the number of correctly detected positive samples, while FP and FN denote the number of incorrectly detected positive samples and the number of incorrectly detected negative samples, respectively.

The F1 score is the harmonic mean of precision and recall, providing a more comprehensive representation of the model’s segmentation accuracy. The calculation method is presented in Formula (4):

$$F1-score={\displaystyle \frac{2TP}{2TP+FP+FN}}$$

(4)

The model complexity is primarily determined by the number of parameters. The evaluation indicators for model complexity are the number of model parameters and the amount of inference computation. The unit for the number of parameters is M (minions), which represents the size of the total number of learnable parameters in the entire model. The unit for the amount of inference computation is GFLOPs (Giga floating point operations), which indicates the number of floating point operations required for model inference. The model segmentation speed is measured by FPS (frames per second), which represents the number of images the model can segment per second.

2.5. Experimental Setting

The experiments involved in this study were conducted on a Windows-based computing platform. The deep learning framework employed was PyTorch 1.13, with CUDA version 11.7, and the Python version was 3.8. The platform was equipped with a GeForce RTX 3090 GPU (NVIDIA Corporation, USA) and an Intel(R) Core (TM) i9–10940X CPU (Intel Corporation, USA) running at 3.30 GHz.

3. Results

3.1. The Problem of Unbalanced Positive and Negative Samples

The task of identifying PWD-affected discolored trees is quite unique. After epidemic prevention and clearance, the distribution of discolored trees is currently sparse. Most of the samples obtained through sampling contain only background and no target of discolored trees, which is a case of extreme sample imbalance. We tried using the focal loss function, which can greatly improve network performance. Considering that most of the information in the background has no positive impact on model training, we conducted experiments on the training set of Dataset A. Following the data processing method mentioned before, we selected samples containing discolored tree lesions for training and then validated them on the same validation set. The model accuracy obtained was higher than that of training on all original training sets, and it greatly saved training time.

Training with the filtered training set not only resulted in higher accuracy but also reduced the training time by 76.19% (

Table 2. Comparison of different training sets and loss functions.

Training Scheme	IoUdiscoloredtree	IoUbackground	F1 Score	Precision	Recall	Time
original training set	0.510	0.997	0.676	0.846	0.563	8.4 h
original training set + focal loss	0.617	0.997	0.763	0.808	0.722	7.4 h
filtered training set	0.647	0.996	0.804	0.715	0.871	2.1 h
filtered training set + focal loss	0.668	0.997	0.802	0.797	0.807	2.0 h

The bold data in the table represents the optimal value in its respective column.

). When there is a severe imbalance between positive and negative samples, focal loss can bring about substantial performance improvements. After data filtering, the sample imbalance issue was largely resolved, the precision of the background did not reduce, and the precision of the discolored tree rose. Based on these experimental results, the new sample processing workflow mentioned before was proposed.

3.2. Comparison of Mainstream Models

A comparison was made between mainstream semantic segmentation models and the proposed EVitNet on the PWD-affected discolored tree Dataset A. The models included U-Net, DeepLabv3+, PSPNet [35], Segformer [36], HrNet [37], and Swin-UNet [38]. It is worth noting that SAM is an interactive model that requires inputting each target’s position for segmentation and is unable to perform automatic segmentation tasks. Therefore, it was not included in our comparison. The input size for all networks was standardized to 256 × 256. Additionally, all hyperparameters were kept consistent: batch size = 16, epochs = 100, and the Adam optimizer was used with a cosine annealing learning rate adjustment strategy. After training, the performance metrics mentioned before were calculated.

Based on the performance of each model on the discolored tree dataset, a scatter bubble chart was created (Figure 7). From the figure, it is evident that EVitNet exhibits significant advantages in terms of segmentation accuracy, parameter count, and speed.

The training loss and validation loss curves for each model are presented in Figure 8. The curves for the HrNet and DeepLabv3+ models exhibit a distinctive pattern, showing a noticeable secondary decline. This indicates that the complexity of the HrNet and DeepLabv3+ models is relatively high, which may not align well with the complexity of the discolored tree dataset, potentially leading them to become trapped in local optima during training. It illustrates that the implemented models have completed extensive training on this dataset, with all models reaching convergence criteria (Figure 8).

In the identification task of PWD-affected discolored trees, EVitNet achieved the highest IOU, F1 score, and pixel recall. Compared to the second-highest performing model, Unet, they were 4.4%, 3.3%, and 0.6% higher, respectively (

Table 3. Performance comparison of mainstream models.

Model	Backbone	IoU	F1 Score	Precision	Recall	Params (M)	GFLOPs	FPS
UNet (Baseline)	vgg	0.669	0.800	0.793	0.808	24.981	112.918	39.81
DeepLabv3+	mobilenet	0.472	0.645	0.588	0.715	5.818	53.026	100.45
HrNet	hrnetv2_w18	0.561	0.711	0.783	0.651	9.642	32.948	211.19
PSPNet	mobilenet	0.617	0.760	0.819	0.708	2.377	6.034	886.50
Segformer	Segformer_b0	0.655	0.786	0.812	0.768	3.720	13.696	485.17
Swin-Unet	Swin-Unet	0.665	0.799	0.817	0.781	98.30	26.33	162.43
EVitNet (Ours)	EasyVit	0.713	0.833	0.853	0.814	1.195	1.636	850.43

In the table, the bold data in the table represents the optimal value in its respective column.

). This may be because discolored tree lesions have many similarities with targets in the medical field. Therefore, Unet, which performs well in the medical field, also has good performance in this task. In this study, Unet was used as the baseline model. The worst-performing model was DeepLabv3+. Judging from the train loss and val loss curves, it may have fallen into a local optimum and ultimately performed worse than Unet. Among mainstream segmentation networks, PSPNet has the fastest segmentation speed and the highest pixel accuracy. This is because the input feature map of PSPNet is divided into branches of different sizes through the pyramid pooling layer. Compared with obtaining branches of different sizes through cascaded convolution, this simplifies the computation and greatly reduces the number of parameters. Unet uses a relatively complex VGG as the backbone network, resulting in a larger number of parameters and computational load. Through comparison, it can be found that the segmentation accuracy of the proposed EVitNet in this study outperforms the other networks and has achieved SOTA on the PWD-affected discolored tree dataset.

3.3. Model Fine-Tuning Study

In previous experiments, EVitNet was sufficiently trained on Dataset A, but its direct application for prediction on Dataset B yielded poor accuracy. On Dataset A, the single-category IoU for discolored trees segmentation was as high as 0.7134, while on Dataset B, the IoU dropped to 0.321. All models used for comparison in this study exhibited this phenomenon. On the one hand, this is due to the insufficient generalization ability of the models. On the other hand, it is because there are significant differences in sampling sensors, height, and terrain between the two datasets, leading to substantial differences in the distribution of feature information. In this study, an experiment for model fine-tuning was designed, with a series of training sample sizes set according to gradients and three different model fine-tuning strategies, yielding the corresponding experimental results (Figure 9).

The experimental results indicated that, due to the significant differences between the two datasets, the final model achieved the highest accuracy when fine-tuning without freezing the backbone network. For global fine-tuning, when the number of new samples used for training was more than 30%, the accuracy generally reached a relatively high level, which is higher than the IoU of the baseline model UNet (0.669).

3.4. Ablation Study

To assess the contribution of each component to the network accurately, ablation experiments were conducted. The MobileVit Block, MobileNetv2 block, ExpConv decoder, and fine-tuning learning were taken as the four basic factors. All training was carried out with the hyperparameter settings consistent with those in Section 3.3. IoU, F1 score, precision, and recall were used as the performance evaluation metrics for the models, where IoU_B refers to the IoU obtained when testing on Dataset B. The following ablation experiments were performed, and the experimental results are shown in

Table 4. The result of ablation study.

Ablation	MobileVit	Mv2	ExpCov	Fine-Turning	IoU	IoU_B	F1 Score	Precision	Recall	Params (M)	GFLOPs	FPS
Unet(baseline)					0.669	0.109	0.800	0.793	0.808	24.981	112.918	39.81
Unet+MobileVit	✓				0.687	0.211	0.816	0.819	0.812	1.331	1.937	771.76
Unet+MobileVit+Mv2	✓	✓			0.709	0.297	0.830	0.835	0.825	1.195	1.636	839.28
Unet+MobileVit+Mv2+ExpConv(EVitNet)	✓	✓	✓		0.713	0.321	0.833	0.853	0.814	1.195	1.636	850.43
Unet+MobileVit+Mv2+ExpConv+fine-turning	✓	✓	✓	✓	0.295	0.735	0. 470	0.314	0.831	1.195	1.636	845.69

In the table, the checks from the 2nd to the 5th columns show whether a module is used. In the following columns, bold numbers indicate the optimal value in that column.

.

Ablation of MobileViT Block: To lighten the model, the bottleneck part of the network was removed to reduce redundant channels in feature maps. This prevented the number of parameters from increasing exponentially with network depth, significantly reducing the network’s parameter count and computational load. To enhance the network’s ability to fuse global features, a lightweight ViT structure was introduced starting from the fourth encoder in the backbone network. This effectively improved segmentation accuracy while keeping the parameter size small. As shown in Table 3, compared to Unet, Unet + MobileViT has 23.65 M fewer parameters and a 1.8% increase in segmentation IoU. This indicates that redesigning the number of channels and incorporating a lightweight ViT can not only reduce the model’s parameter count but also improve the accuracy of identifying PWD-affected discolored trees.

Ablation of MobileNetv2 Block: To further improve the efficiency of feature extraction in the backbone network, the more efficient convolution module, MobileNetv2 Block, was used to replace standard convolutions. This slightly reduced the number of parameters and further enhanced the network’s segmentation accuracy. As shown in Table 3, the segmentation IoU increased by 2.2%, and the number of parameters decreased by 0.126 M. This suggests that, compared to standard convolutions, the MobileNetv2 Block is more suitable for the task of identifying PWD-affected discolored trees.

Ablation of Expanded Convolution in Decoder: Expanded convolutions were used to improve the accuracy of the model during the upsampling phase without increasing the number of network parameters. The standard convolution blocks after each decoder were replaced with expanded convolution blocks. As shown in Table 3, compared to Unet + MobileViT + Mv2, EVitNet achieved a slight improvement in segmentation accuracy. The idea behind expanded convolutions is similar to that of the MobileViT Block: Both expand the features, perform computations, and then compress them back to their original dimensions. This approach is very effective for the task of discolored trees identification.

Ablation of Model Fine-tuning: Among the various ablation experiments, the IoU_B on Dataset B was relatively low, indicating that the model trained on Dataset A could not be directly used to predict Dataset B. As shown in Table 3, after fine-tuning with a small portion of samples, EVitNet was able to achieve high accuracy on Dataset B.

4. Discussions

4.1. Network Performance Analysis

Currently, research on semantic segmentation methods for the identification of discolored trees is gradually increasing. Zhang et al. [39] were the first to attempt using segmentation methods for the recognition of discolored trees, comparing the Random Forest and U-Net algorithms. The latter, based on a CNN, can extract features more effectively and showed an absolute advantage in recognition accuracy. Li et al. [40] proposed an improved Mask R-CNN instance segmentation method for identifying discolored trees at various infection stages, achieving an overall accuracy of 71%. Zhi et al. [41] compared several algorithms, including U-Net, DeepLabv3+, FPN, and SAM, and found that UNet achieved the highest accuracy on discolored tree datasets in RGB, HSV, and mixed modes. Their research mainly compared and evaluated mainstream networks on their self-built discolored tree dataset. In contrast, our study not only assessed mainstream network performance on our self-built dataset but also proposed a new lightweight algorithm and used fine-tuning to address generalization issues. Due to dataset differences, direct comparison is not feasible. So, we reproduced their methods on our dataset for comparison, with the results presented in the earlier Results Section. These results echoed their findings, with UNet showing the highest accuracy among mainstream networks. In the task of identifying PWD-affected discolored trees, EVitNet significantly outperforms mainstream segmentation networks. The challenge of this task lies in distinguishing discolored tree lesions from similar ground objects. The uniqueness of EVitNet lies in its lightweight EasyVit backbone network, which combines a CNN and a Transformer. A CNN is responsible for extracting and processing local features to ensure precise segmentation boundaries, while a Transformer processes features from a global perspective to ensure accurate classification of pixel categories. Additionally, the idea of flattening the feature map, performing operations, and then contracting it is adopted in multiple parts of the network, saving parameter overhead and improving accuracy. In EVitNet, in addition to the residual connections between the encoder and decoder, residual connections are also used in the MV2 Block and MobileVit Block, preserving rich local detail features and ultimately achieving accurate semantic segmentation of discolored tree lesions.

Using the same hyperparameter settings as in Section 3.3 and the same training and testing samples, each model was trained for prediction, yielding the segmentation results for PWD-affected discolored trees. Figure 10 shows the segmentation results of representative samples under different lighting conditions and complexities. Five semantic segmentation models, PSPNet, Segformer, DeepLabv3+, HrNet, and Swin-Unet, were used for comparison.

In Figure 10, for sample (a), the contrast is clear, and there is little environmental interference, so all models are able to identify the discolored trees. However, DeepLabv3+ missed some pixels of the discolored trees. In samples (b), (c), and (d), due to strong environmental interference, poor lighting, and other reasons, many models had a large number of missed and false detections. The EVitNet proposed in this study has the best robustness, with only a small number of pixels missed or falsely detected. It can be seen that there are still some discrepancies between the EVitNet results and the label. These discrepancies correspond to the misclassified pixels. In addition to the pixels near the discolored tree lesions, there are also a few misclassified pixels in the background. For example, in sample (b), EVitNet misclassified some pixels on the roof as discolored trees. In the sample in sample (c), there is also a small portion of missed detections. However, these types of errors occur in all models, and there is currently no way to optimize them. Further research in the future is needed.

In this study, fine-tuning the model had a significant impact on the network’s segmentation performance. The comparison of the segmentation results before and after fine-tuning can be seen in Figure 11. The non-fine-tuned model refers to using EVitNet trained on Dataset A directly to identify samples in Dataset B, while the fine-tuned model refers to using EVitNet after fine-tuning with 40% of the training set to identify samples in Dataset B.

It can be seen from Figure 11 that the non-fine-tuned model has a serious problem of missed detection, but there are almost no false positives. This is because in the task of discolored trees recognition, most areas of the samples are negative samples. When the model encounters pixels it cannot recognize during the identification process, it is more likely to classify them into the background category. Additionally, some knowledge learned by the model on Dataset A is also applicable to Dataset B, so there are almost no false positives. After fine-tuning, the model’s missed detection rate dropped significantly, and the recognition accuracy reached a high level. Fine-tuning the model has greatly enhanced the practicality of EVitNet in different scenarios.

4.2. Advantages and Disadvantages

The quality of training samples has a significant impact on the model’s accuracy. The task of this study is to segment the discolored tree lesions. Training excessively on negative samples can adversely affect the model’s precision. Experiments have shown that training solely with samples containing discolored tree lesions yields a higher accuracy than using all samples. The datasets used in this study were annotated under the guidance of forestry disease experts, which, to some extent, excluded interference from discolored pine trees caused by non-PWD and other broad-leaved discolored trees. This allowed the model to learn from the experts’ judgment and experience. Currently, research on remote sensing monitoring of PWD-affected discolored trees mainly focuses on object detection algorithms. However, object detection algorithms represented by YOLO often miss small targets [42] and cannot precisely identify partially diseased branches and leaves of a single pine tree. Semantic segmentation models have lower requirements for image spatial resolution and are more suitable for large-scale monitoring of discolored trees. They can detect discolored trees earlier and more comprehensively. Moreover, the EVitNet proposed in this study has fewer parameters than the YOLO series of object detection algorithms. The research results of Xia et al. [43] indicate that an excessive number of network layers does not enhance the accuracy of segmentation but merely increases the computational load, so this study used a lightweight ViT to enhance the network’s ability to process global features, and the segmentation network was lightened, making the network complexity match the sample complexity. The images of PWD infected trees have low contrast and limited local spatial information. Traditional CNNs can only capture local features and ignore long-range pixel relationships, lacking global perception. This makes it hard to classify pixels based on the entire image context, especially when lesions are scattered. Global feature fusion can highlight key areas, and combining with local features improves segmentation accuracy. Using a Transformer for feature extraction solves the global perception issue but increases parameters, hindering widespread deployment. EVitNet combines the strengths of CNNs and Transformers, uses fewer channels, introduces a Transformer in deeper layers, shortens the self-attention sequence length, and significantly reduces parameters and computational complexity, achieving a balance between speed and accuracy. Additionally, we deployed EVitNet on the NVIDIA Jetson XAVIER NX developer kit. On this platform, the unquantized algorithm achieved a segmentation speed of 74.88 fps, with a peak instantaneous power consumption of 14.3 W and an average power consumption of 10.2 W. Using real-time semantic segmentation algorithms can improve the workflow of discovering and locating discolored trees and can timely curb the spread of PWD.

Although EVitNet performed well on the PWD-affected discolored trees dataset, the lightweight design reduced some parameters and lowered the model complexity, which may result in a gap between this model and SOTA algorithms in other fields. The EVitNet model was designed for discolored tree datasets and excels at segmenting similar lesion-type images. However, its performance in other real-world scenarios may be suboptimal. This study focused on identifying only one type of target with relatively simple and low-resolution image content. In contrast, real-world scenes often have more target varieties, complexity, and higher resolutions. The inconsistent feature distribution means the lightweight backbone network EasyVit used in this study may fail to extract sufficient features, leading to potential poor segmentation accuracy in other scenarios, which might restrict the algorithm’s applicability. It may only be suitable for tasks with similar complexity to PWD-affected discolored trees identification, such as cotton root rot, bacterial blight of rice, and powdery mildew in wheat [44,45]. In addition, the PWN can infect 57 species of pine plants and 13 non-pine species [46]. Annotating data for training semantic segmentation models has always been difficult. The fine-tuning learning method used in this study greatly improved the model’s segmentation performance on new datasets, but the new data still needs to be annotated, which is very time-consuming in the workflow of identifying discolored trees and is not conducive to accelerating the process. Further research is needed to find new methods to improve the model’s performance on new datasets.

5. Conclusions

The aims of this study were to overcome the limitations of current semantic segmentation technologies in recognizing discolored trees, such as the lack of specialized algorithms and sub-par performance of mainstream methods in accuracy, speed, and robustness in complex environments. We believe that precise annotation and more efficient segmentation networks can boost recognition accuracy. Thus, we propose creating a high-quality dataset, developing the EVitNet model with a lightweight ViT encoder and CNN decoder, and determining the optimal fine-tuning strategy. The EasyVit backbone has low parameter overhead and improves feature extraction in complex environments. The decoder, using expended convolutions, slightly improves segmentation accuracy without adding parameters. The fine-tuning learning method effectively tackles generalization challenges from different environments and sensors. The model can achieve high accuracy on new datasets with only a few new samples. It has an IoU of 0.713, an F1 score of 0.833, a segmentation speed of 850.43 fps, and 1.195 M parameters. EVitNet excels in recognizing discolored trees and can semantically segment PWD-affected trees accurately and efficiently. As PWD is sporadic in many regions, monitoring is crucial for control. This study offers technical support for faster and more accurate remote sensing monitoring. However, as mentioned in the limitations, the EvitNet may only achieve high accuracy on feature-similar datasets. Future work could explore its application to other data types, such as multispectral and hyperspectral samples. Additionally, semi-supervised or unsupervised learning could be considered to address the difficulty of dataset annotation.