A Multimodal Visual–Textual Framework for Detection and Counting of Diseased Trees Caused by Invasive Species in Complex Forest Scenes

Highlights

What is the main finding?

• We have developed an end-to-end multimodal disease detection and counting model that accurately detects and counts trees affected by various types of diseases in forest imagery by integrating rotational position encoding with a gated bidirectional cross-attention mechanism.

What is the implication of the main finding?

• This technological approach provides a scientific basis for forest ecological environment monitoring and forest disease prevention and control, facilitating the timely identification of disease risks and promoting the healthy development and sustainable management of ecosystems.

Abstract

With the large-scale invasion of alien species, forest ecosystems are facing severe challenges, and the health of trees is increasingly threatened. Accurately detecting and counting trees affected by such invasive species has become a critical issue in forest conservation and resource management. Traditional detection methods usually rely only on the information of a single modality of an image, lack linguistic or semantic guidance, and often can only model a specific diseased tree situation during training, making it difficult to achieve effective differentiation and generalization of multiple diseased tree types, which limits their practicality. To address the above challenges, we propose an end-to-end multimodal diseased tree detection model. In the visual encoder of the model, we introduce rotational positional encoding to enhance the model’s ability to perceive detailed structures of trees in images. This design enables more accurate extraction of features related to diseased trees, especially when processing images with complex environments. At the same time, we further introduce a cross-attention mechanism between image and text modalities, so that the model can realize the deep fusion of visual and verbal information, thus improving the detection accuracy based on understanding and recognizing the semantics of the disease. Additionally, this method possesses strong generalization capabilities, enabling effective recognition based on textual descriptions even when samples are not available. Our model achieves optimal results on the Larch Casebearer dataset and the Pests and Diseases Tree dataset, verifying the effectiveness and generalizability of the method.

1. Introduction

Trees are an indispensable part of the earth’s ecosystem; they not only provide us with valuable resources such as oxygen, timber, and medicinal herbs, but also play a vital role in regulating the climate, purifying the air, and preventing soil erosion. By absorbing carbon dioxide, forest ecosystems reduce greenhouse gas and mitigate global warming, and are a key component in the global carbon cycle. However, with the continuous invasion of invasive alien species (IAS), the growth of trees faces unprecedented threats. These IAS not only compete with native tree species for living space, nutrients, and water, but also seriously disturb the ecological balance of forests by spreading pests and diseases. Their invasion has already caused a great disaster to forestry ecology, affecting the health and sustainable development of forests. With the growing seriousness of this problem, it has become imperative to protect and restore forest ecosystems, resist invasive alien species, and detect and count the diseases of invasive alien species.

With the advancement of remote sensing technology, image recognition, computer vision, and artificial intelligence, more and more studies are exploring automated tree disease detection methods [1] based on image analysis. These efforts aim to improve the efficiency, accuracy, and intelligence of forest pest and disease monitoring. By collecting tree images and processing them with deep learning models, automatic identification and localization of diseases can be achieved, greatly reducing the workload of manually detecting diseased trees.

Following these advancements, numerous detection and recognition models have been developed. For instance, the You Only Look Once (YOLO) series [2] enables the identification of diseased trees by extracting visual features and performing multi-scale predictions on input images, while the Segment Anything Model (SAM) [3] provides powerful segmentation capabilities that precisely delineate tree regions for accurate disease localization and analysis. In 2020, Alaa et al. [4] proposed a palm tree disease detection method combining a convolutional neural network (CNN) [5] and a support vector machine (SVM) [6], effectively integrating the deep learning model’s feature extraction ability with the robust classification strength of traditional machine learning to distinguish visually similar disease symptoms. More recently, Lin et al. [7] introduced a Transformer-based model with graph-structured modulation, in which attention and feature centrality graphs jointly refine self-attention and feature representations, mitigating attention homogenization and improving detection accuracy. Despite these advances, current models still face significant challenges that limit their performance in complex real-world scenarios.

• In many complex and dense scenes, a large number of trees are clustered together with different sizes of individuals, and this high density and scale difference significantly increase the detection difficulty, which seriously affects the recognition and localization ability of the model.
• The generalization ability of many models is weak, as they usually only perform well on trained data and are less effective in recognizing unseen samples, making it difficult to adapt to complex and changing practical application scenarios.
• Traditional models rely only on single-modal information, which leads to their lack of sufficient semantic expressiveness to effectively capture richer and deeper semantic features, and the recognition accuracy of the models is greatly reduced when facing similar categories.

To cope with these challenges, this paper proposes a multimodal tree disease detection model, as shown in Figure 1, which is able to distinguish different classes of trees in an image compared to the traditional model that only realizes one modality as input. Thanks to the introduction of a textual modality [8], our model shows unprecedented multi-scene adaptability, and is able to efficiently train and differentiate statistics on tree images taken by unmanned aerial vehicles (UAVs) with multiple complex conditions at the same time. This breakthrough design covers different physiological states, including healthy, diseased, and insect-infested trees. We introduce rotational attention encoding [9] in the visual coder to enable the model to capture local and global features in the image more efficiently. In order to better fuse the image information and text information, we introduce gated bidirectional cross-attention [10] and a contrastive loss function, which deeply interact with the image information and text information to enhance the alignment and fusion of multimodal features. Finally, the fused multimodal features are inputted into the downstream task module of diseased tree detection and statistics to obtain accurate diseased tree conditions, which provides an efficient and accurate technical solution for forestry ecological protection.

Figure 1. Traditional methods vs. text-guided statistical methods: traditional methods are trained separately for each scenario and accept only one input modality, while our model is trained jointly across multiple scenarios and leverages text guidance together with image input to distinguish different classes.

In summary, our contributions are as follows:

• We propose an end-to-end deep learning model for disease detection that incorporates rotational position encoding in the image encoder, which allows the model to better understand the positional information of the trees in the image, and improves the model’s ability to capture the trees in complex scenes, thus increasing the accuracy.
• Our model demonstrates exceptional generalization capabilities, enabling it to extract universal features based on existing knowledge and adapt to diverse datasets, even when distributions shift. When confronted with unseen data scenarios, it maintains robust recognition performance, overcoming the limitations of traditional models that require extensive re-training in new environments.
• Our model fuses visual and textual information in a gated bidirectional cross-attention module, and the ability to incorporate textual information enables the model to be dynamically adapted to a wider range of real-world scenarios based on user-input descriptions or specific needs.

The rest of the paper is organized as follows: In Section 2, we describe the development of traditional disease detection with the emerging large model. In Section 3, we present the general architecture of our model, with each subsection describing each part of the model. In Section 4, we describe the experimental setup and experimental results, and show visualization images. In the Section 5 and Section 6, we present our discussion and conclusions.

2. Related Work

2.1. Breakthrough in Disease Detection

Since 2016, plant disease detection has advanced rapidly with the development of increasingly efficient and intelligent models. Anand et al. [11] first employed a K-means clustering algorithm to segment leaf images and extract texture features of diseased areas, followed by classification using neural networks, achieving accurate identification of diseased eggplant leaves and laying the groundwork for subsequent research. By 2019, Pantazi et al. [12] enhanced recognition accuracy by adopting the local binary pattern (LBP) for image feature extraction and integrating it with support vector machines (SVMs), enabling robust classification of multiple diseased leaf types without reliance on large-scale labeled datasets—an important step toward small-sample disease detection. In 2022, Abu-Zanona et al. [13] proposed a lightweight CNN architecture with only four convolutional and one fully connected layer, significantly reducing computational costs and training time while maintaining high classification performance, thereby providing an efficient solution for palm tree disease identification under resource-constrained environments. With the emergence of more powerful detection frameworks, Lyu et al. [14], in 2024, successfully applied YOLOv8 for efficient plant disease detection, leveraging its end-to-end capability, real-time inference speed, and superior localization precision to transform disease detection from simple classification into precise, target-level identification. By 2025, Amin et al. [15] introduced a Transformer-based model incorporating a multi-head external attention mechanism and synthetic data generation for Pine Wilt Disease (PWD) detection, effectively addressing data scarcity and computational complexity while maintaining high accuracy. In the same year, Li et al. [16] developed a large-scale real-world dataset for apricot tree disease and pest detection and proposed a lightweight detection framework, APNet, which integrates an Adaptive Thresholding Algorithm and multi-scale feature fusion to enhance small-target recognition under complex backgrounds. Around the same time, another lightweight single-stage model, PWDViTNet [17], was designed for early PWD detection in pine forests, combining the local feature extraction of CNNs with the global modeling capability of Vision Transformers, further enhanced by multi-scale fusion and attention mechanisms. Collectively, these advancements have enabled more accurate, automated, and efficient plant disease monitoring, laying a solid foundation for the continued evolution of intelligent agricultural technologies.

Although there are many models that can realize the classification and detection of diseased trees, their performance tends to degrade in complex scenes with dense targets and overlap between trees. To cope with such challenges, we introduce a rotational attention mechanism in the image encoder, which enhances the model’s ability to perceive details and thus localize each tree crown more accurately. In addition, we incorporate the density map point location information to effectively improve the model’s detection accuracy in complex environments.

2.2. Pre-Trained Large Models

Large models are pre-trained on large-scale datasets and mainly include large language models [18] and multimodal large models [19]. With powerful capabilities, such models have demonstrated excellent performance in many downstream tasks, which not only enhance the task-solving efficiency, but also promote the development and innovation of related fields.

Since 2017, large models have undergone rapid development, evolving from unimodal to multimodal domains. In 2017, Vaswani et al. [20] proposed the Transformer architecture in Attention Is All You Need, a brand-new model based entirely on the attention mechanism, which abandons the traditional convolutional neural network architecture and revolutionized the field of natural language processing. In 2018, the introduction of the Bidirectional Encoder Representations from Transformers (BERT) model [21] marked the successful application of the bidirectional Transformer architecture. By combining pre-training and fine-tuning strategies, BERT significantly improved the performance of the model and ushered in the era of pre-trained models. In the same year, ChatGPT series [22] appeared, and GPT-1 proposed the concept of generative pre-training for the first time, providing a new way of thinking about natural language generation tasks. Subsequently, GPT-2 and GPT-3 further expanded the scale and capability of the model, laying a solid foundation for large-scale language modeling. Progress in cross-modal learning has been equally impressive. For example, the Contrastive Language–Image Pre-training (CLIP) model [23] achieves efficient alignment of text and image information by pre-training on a large-scale dataset containing 400 million image–text pairs, which significantly improves performance on cross-modal tasks. In 2024, Zang et al. [24] proposed the ContextDET framework, pioneering the application of multimodal large language models to ‘contextual object detection’. Through an integrated generation–detection design, the model simultaneously localizes and recognizes visual objects within language completion, description, and question-answering tasks, transcending the limitations of traditional closed-class detection. In 2024, Zhu et al. [25] proposed the VA-Count framework, employing visual association and contrastive learning to address zero-shot object counting. Its sample augmentation and noise suppression modules significantly enhanced counting accuracy and generalization capabilities under unlabeled conditions. In 2025, Binyamin et al. [26] introduced the CountGen method, enabling diffusion models to accurately generate specified quantities of objects. By identifying object-specific features within self-attention layers and incorporating the layout correction network ReLayout, it achieves text-to-image generation with controllable quantity. These innovations further advance the multimodal field and greatly enhance the application potential of the models, opening up new possibilities for research and practice in computer vision [27], natural language processing [28], and cross-modal tasks [29].

Our model makes full use of the multimodal alignment capability of the large model to accurately detect and count the corresponding diseased trees through the text-guided model, providing an efficient and intelligent solution for forestry disease monitoring.

2.3. Zero-Shot Learning

In target detection and recognition tasks, traditional deep learning methods usually rely on sufficient pre-training for all target categories. Zero-shot learning is an innovative approach that can achieve accurate recognition and classification of unseen categories by exploiting semantic information between categories or other a priori knowledge without pre-training for a specific target category, thus demonstrating stronger generalization ability and flexibility to effectively deal with scenarios with new categories and sparse data.

The concept of zero-shot learning [30] was first introduced by Lampert et al. to address the problem of classification without training samples. They proposed two methods—direct attribute prediction and indirect attribute prediction—to classify unseen categories by attribute information. This research direction quickly attracted a lot of attention. In 2013, Norouzi et al. [30] proposed a method to establish the linkage between old and new categories using word vectors by generating an embedded representation of an image by using the probabilities of the old categories as the weighted weights of the word vectors, and then calculating the probabilities of the new categories by utilizing the cosine similarity between the word vectors of the new categories and the embedded representation of the image. Subsequently, Zhang et al. [31] found that projecting image features from visual space to semantic space leads to dimensionality catastrophe. For this reason, they proposed, in 2017, an inverse projection method to map semantic information to visual space, thus optimizing the distance between the image embedding representation and the label vector. In the same year, Kodirov et al. [32] projected visual features into semantic space based on encoder and decoder structures, and effectively mitigated the projection bias problem in zero-shot learning by reconstructing the original visual features. In 2019, Hou et al. [33] proposed the cross-domain adversarial generative network, which combines the traditional generative adversarial network with the variational autocoder, and is capable of generating samples of unseen categories to improve the zero-shot classification performance. Subsequently, with the advent of pre-trained models such as CLIP in 2021, image information was highly aligned with textual information, which significantly improved the performance of zero-shot classification tasks. After this, more and more large-scale pre-trained models continued to appear, such as Distillation with No Labels (DINO) [34], BLIP, etc., and the research focus shifted to how to efficiently utilize large-scale pre-trained models, cross-modal alignment techniques, and generative models to further improve the performance and generalization ability of zero-shot learning.

Building on the strengths of large-scale pre-trained models, our approach emphasizes robust generalization capabilities. This powerful generalization capability enables the model to effectively handle new scenarios.

3. Methods

As shown in Figure 2, the image and the prompt indicating the condition type of the trees are first used as inputs. A trainable image encoder and text encoder are then employed to generate corresponding image and text embeddings. The image embeddings are enhanced through a CNN and a self-attention mechanism, and then dynamically fused with the text embeddings via a bidirectional cross-attention gating mechanism. The decoder subsequently outputs the density features of the images, which are used to identify and count the number of diseased trees.

Figure 2. General architecture diagram, showing the complete processing pipeline from feature extraction and multimodal fusion to density map generation and tree counting.

3.1. CLIP Base

CLIP [23] is a cross-modal learning model proposed by OpenAI, which is able to process images and text simultaneously and learn the association between them. The core of CLIP is to reinforce the similarity between the corresponding images and texts by mapping the images and texts to the shared embedding space by means of comparative learning. Specifically, by inputting an image $\mathbf{I}$ and a text description $\mathbf{T}$, CLIP maps to the shared embedding space using an image encoder and a text encoder, respectively.

$${\mathbf{E}}_I=G\left(\mathbf{I}\right),$$

(1)

$${\mathbf{E}}_T=G\left(\mathbf{T}\right),$$

(2)

${\mathbf{E}}_I$ is image embedding and ${\mathbf{E}}_T$ is text embedding. Then, CLIP computes the similarity between the text embedding and the image embedding by calculating the cosine similarity between the two.

$$CLIP(I,T)={\displaystyle \frac{{\mathbf{E}}_I·{\mathbf{E}}_T}{∥{\mathbf{E}}_I∥∥{\mathbf{E}}_T∥}},$$

(3)

Here, · is the dot product of the image embedding and the text embedding, and $\parallel ·\parallel$ is the L2 paradigm (the modulus of the vectors).

3.2. Rotational Positional Encoding

In the image encoder, we incorporate rotational positional encoding [9], which utilizes an approach that deals with vector embedding in the two-dimensional case. It differs from traditional positional embedding in that it transforms the feature vectors by rotating them through a rotation matrix, encoding positional coding information directly into the feature space. This approach not only avoids the introduction of additional learnable parameters, but also efficiently captures the sequence or the spatial distribution in positional relationships in both visual and verbal tasks. After that, a scale factor is further introduced to enable the position coding to perceive information at different scales, thus enhancing the robustness and expressiveness of the model in dealing with multi-scale targets, such as the scale changes caused by differences in height and tree species in UAV images.

The core idea behind rotational positional encoding is to leverage frequency and angular information to construct rotation matrices using cosine and sine functions, which are then applied to feature vectors as rotation transformations. This rotation operation inherently preserves the vector magnitude while integrating positional cues into the feature representation. In this work, we become the first to apply rotational positional encoding to tree disease detection and counting. In the context of aerial images of dense tree canopies, we introduce a scale factor based on the actual scene, enabling the model to better capture the spatial distribution and relative arrangement of overlapping tree crowns, and thus improving its ability to accurately distinguish and count trees under complex canopy densities and varying spatial orientations.

$${\mathbf{S}}_{\mathrm{i}}=T\left(patch\right),$$

(4)

$$\begin{array}{cc}\hfill {\mathrm{f}}_{\{q,k\}}\left({\mathrm{x}}_m,m\right)& =\left(\begin{array}{cc}cos\left({\mathbf{S}}_{\mathrm{i}}·m·\theta \right)& -sin\left({\mathbf{S}}_{\mathrm{i}}·m·\theta \right)\\ sin\left({\mathbf{S}}_{\mathrm{i}}·m·\theta \right)& cos\left({\mathbf{S}}_{\mathrm{i}}·m·\theta \right)\end{array}\right)\hfill \\ \hfill & ·\left(\begin{array}{cc}{\mathbf{W}}_{\{q,k\}}^{\left(11\right)}& {\mathbf{W}}_{\{q,k\}}^{\left(12\right)}\\ {\mathbf{W}}_{\{q,k\}}^{\left(21\right)}& {\mathbf{W}}_{\{q,k\}}^{\left(22\right)}\end{array}\right)\left(\begin{array}{c}{\mathrm{x}}_m^{\left(1\right)}\\ {\mathrm{x}}_m^{\left(2\right)}\end{array}\right),\hfill \end{array}$$

(5)

where $\theta$ is a constant to control the rotation angle, ${\mathbf{W}}_{\{q,k\}}^{\left(ij\right)}$ is the element of row i and column j of the projection matrix for Query/Key, $\mathrm{x}$ is a given vector, and m denotes the position. In the image processing task of diseased trees, rotational attention coding significantly improves the model’s ability to recognize complex disease tree features through its unique orientation adaptation and dynamic weight assignment mechanism, and ${\mathbf{S}}_i$ is the scale factor. $T(·)$ is the scale function and $patch$ is the position index in the image block.

3.3. Contrastive Loss Function

We invoke a text-patch contrast loss function [35] designed to maximize textual cues and cue-related image-patch mutual information. The method maps the feature map ${\mathbf{z}}_x$ to the same channel dimension as the text embedding ${\mathbf{E}}_T$ through a linear projection ${\ell }_{\mathrm{p}}:R_{\mathrm{d}}\to R_n$ to ensure that both can be learned in the same feature space for comparison:

$$\left\{{\mathbf{E}}_p^i={\ell }_{\mathrm{p}}\left({\mathbf{z}}_x^i\right)\mid i=1,2,\dots ,{\mathrm{p}}^2\right\},$$

(6)

In Equation (6), we use ${\mathbf{E}}_p\in R_{\mathrm{p}\times \mathrm{p}\times n}$ to refer to the resulting patch embedding of the input image. In order to determine the location of the object, the ground truth density map y is maximally pooled, and thus the patch-level mask $\mathrm{M}\in R_{\mathrm{p}\times \mathrm{p}}$ is obtained. For more accurate statistics, we introduce a contrastive loss function $L_{\mathrm{con}}$:

$$L_{\mathrm{con}}=-log{\displaystyle \frac{{\sum }_{i\in P}exp\left(\frac{s\left({\mathbf{E}}_p^i,{\mathbf{E}}_T\right)}{\tau }\right)}{{\sum }_{i\in N}exp\left(\frac{s\left({\mathbf{E}}_p^i,{\mathbf{E}}_T\right)}{\tau }\right)+{\sum }_{k\in N}exp\left(\frac{s\left({\mathbf{E}}_p^k,{\mathbf{E}}_T\right)}{\tau }\right)}},$$

(7)

In Equation (7), P and N represent the positive and negative sample sets of patch embeddings, respectively, according to the corresponding masks, $\mathrm{s}(·)$ represents the cosine similarity, and $\tau$ is a temperature parameter. The core objective of this contrast loss is to bring the positive sample embeddings closer to the corresponding text embeddings in the feature space, while reducing the similarity between the negative samples and the text embeddings. This strategy is consistent with the optimization goal of CLIP pre-training, and experimental validation shows that jointly optimizing patch embeddings and text embeddings can significantly improve the overall performance of the model.

3.4. Gated Bidirectional Cross-Attention Module

This section proposes a structural module for deep fusion of image and text modalities, centered on a bidirectional cross-attention mechanism and a gated dynamic fusion mechanism. The designed bidirectional cross-attention simultaneously models semantic dependencies between modalities in both visual-to-text and text-to-visual directions. This overcomes the limitations of traditional unidirectional attention, which inadequately models modal information, enabling fine-grained, bidirectional semantic interaction. Building upon this foundation, the gated dynamic fusion mechanism is introduced. This mechanism employs a lightweight gated network to adaptively adjust the fusion weights of cross-attention information from both directions, enabling dynamic routing and weighted integration of intermodal information. The module first extracts the local structural features of the image by convolutional operations and up-samples or down-samples them according to a set target resolution to unify the feature scale. Subsequently, the module applies a self-attention mechanism [36] on the image features to enhance their contextual representation. Next, the text-to-image cross-attention mechanism and image-to-text cross-attention mechanism are executed sequentially to obtain corresponding multimodal information [37]. Subsequently, a weighted fusion is achieved through a gated dynamic fusion mechanism.

In UAV-acquired dense forest canopy imagery, the gated bidirectional cross-attention modules are designed to leverage textual cues, enabling more precise guidance of visual feature extraction and improved alignment between tree attributes and visual regions. The gating component selectively controls the flow of information, allowing the most relevant textual prompts describing tree attributes to guide visual feature extraction, while the bidirectional design reinforces this guidance by letting visual features and textual cues interact continuously. This dual mechanism helps the model focus on target areas, reduces misidentification, and ultimately improves overall prediction accuracy in dense and diverse forest environments.

$${\mathbf{E}}_{conv}={\mathbf{E}}_p+\text{ReLU}\left(\text{Conv}\right({\mathbf{E}}_p\left)\right),$$

(8)

$${\mathbf{E}}_{interp}=Interp\left({\mathbf{E}}_{conv},\mathrm{scale}=2\right),$$

(9)

$${\mathbf{E}}^{\prime }={\mathbf{E}}_{interp}+\text{Dropth}\left(\text{SelfAttn}\left(\text{LN}\left({\mathbf{E}}_{interp}\right)\right)\right),$$

(10)

$$\begin{array}{cc}\hfill \mathbf{F}=gfuse& (\text{Dropth}\left(\text{CrossAttn}(\text{LN}\left({\mathbf{E}}^{\prime }\right),{\mathbf{E}}_T)\right),\hfill \\ \hfill & \text{Dropth}\left(\text{CrossAttn}(\text{LN}\left({\mathbf{E}}_T\right),{\mathbf{E}}^{\prime })\right)),\hfill \end{array}$$

(11)

$${\mathbf{E}}^{″}={\mathbf{E}}^{\prime }+\mathbf{F},$$

(12)

$${\mathbf{E}}_{out}={\mathbf{E}}^{″}+Dropth\left(MLP\left(LN\left({\mathbf{E}}^{″}\right)\right)\right),$$

(13)

where $ReLU(·)$ is the activation function, $Interp(·)$ is the sampling function, and Dropth is a random discard function that we customized. $SelfAttn(·)$ is the self-attention layer, $CrossAttn(·)$ is the cross-attention layer, LN is the normalization layer, $Conv(·)$ is the convolutional layers, scale is the scaling factor, $gfuse(·)$ is a function that performs gated dynamic fusion on the integration of two types of information, and $MLP(·)$ is the multilayer perceptron.

3.5. Counting Decoder Module

The decoder consists of multiple convolutional layers, each of which progressively reduces the number of channels and improves the spatial resolution through convolutional, normalization, and activation operations, and gradually restores the size to the target size by bilinearly interpolating and up-sampling the feature maps at each layer.

$${\mathbf{E}}_{\mathrm{out}}^i=Conv\left(Interp\left({\mathbf{E}}_{\mathrm{out}}^{i-1},\mathrm{scale}=2\right)\right)\mathrm{for}i=1,2,\dots \mathrm{n},$$

(14)

$${\mathbf{E}}_{\mathrm{target}}=Interp\left({\mathbf{E}}_{\mathrm{out}}^i,\mathrm{scale}=\mathrm{target}\right),$$

(15)

$${\mathbf{E}}_{final}=\sigma \left({\mathbf{E}}_{target}\right),$$

(16)

$${\mathrm{N}}_{pred}=SUM\left({\mathbf{E}}_{final}\right),$$

(17)

$$L_{MSE}={\displaystyle \frac{1}{n}}\sum _{\mathrm{i}=1}^n{\left({\mathrm{N}}_{pred}^{\mathrm{i}}-{\mathrm{N}}_{gt}^{\mathrm{i}}\right)}^2,$$

(18)

where $\mathrm{Conv}(·)$ is the convolutional layers, $\mathrm{scale}$ is the scaling factor, $\mathrm{Interp}(·)$ is the sampling function, $\mathrm{target}$ is the size of the target image, $\sigma (·)$ is the Sigmoid function, $L_{MSE}$ is the mean square error loss function, $N_{out}^{\mathrm{i}}$ is the predicted result of the ith image, $N_{gt}^{\mathrm{i}}$ is the true result of the ith image, SUM is the sum of all elements in the image, and n is the number of quantities.

4. Experimentation and Analysis

4.1. Experimental Setup

Datasets: (1) We utilize the open-source dataset Larch Casebearer (Swedish Forest Agency (2021): Forest Damages – Larch Casebearer 1.0. National Forest Data Lab, Dataset) [38], which is a project initiated by the Swedish Forest Agency with support from Microsoft that monitors forest areas affected by the white moth. The project annotates the locations of healthy, mildly diseased, and highly diseased trees across five distinct regions. Healthy trees exhibit green foliage with no visible discoloration. Mildly diseased trees show slight yellowing of leaves or minor thinning. Severely diseased trees display severe discoloration or even dieback. In total, the dataset contains more than 1500 images of over 40,000 trees. Several sample images from dataset 1 are displayed within the black dashed box in Figure 3.

Figure 3. Sample diagram of the dataset, showing several randomly selected UAV-captured images.

(2) The open-source dataset PDT dataset (PDT: Uav Target Detection Dataset for Pests and Diseases Tree; Mingle Zhou, Rui Xing, Delong Han, Zhiyong Qi, Gang Li*; ECCV 2024) [39], which is from the Shandong Computer Science Center, is also utilized to collect aerial images of the pine forest area. It contains more than 4000 images with two types of trees in each image, healthy trees and trees damaged by the red fat-sized borer. Several sample images from dataset 2 are displayed within the blue dashed box in Figure 3.

Both datasets consist of aerial forest images captured vertically by unmanned aerial vehicles (UAVs). For both datasets, images with incomplete or missing annotations were removed to ensure data validity. In the Larch Casebearer dataset, this resulted in approximately 800 images with roughly 44,000 annotated instances used for experiments. Similarly, in the PDT dataset, around 3900 images with approximately 110,000 annotations were retained after filtering. To ensure consistency, all images were resized to a uniform resolution of 384 × 384 pixels. The datasets were divided into training, validation, and test sets with a ratio of approximately 5:1:1. All textual inputs in our framework are expressed as natural language prompts.

4.2. Comparison Methods

In order to validate the effectiveness of our model, we chose to focus on deep learning methods to compare with our method, which are as follows:

ALEXTNET [40]: The first model to apply deep convolutional neural networks to large-scale image classification and achieve a breakthrough.

RESNET50 [5]: A 50-layer convolutional neural network that introduces residual connectivity to solve the difficulty of training deep networks.

UNET [41]: Supports density map generation through fine segmentation capability, suitable for small-target or sparse counting tasks, with jump–junction fusion of multilayer features.

EFFICIENTNET [42]: An image classification model that combines high accuracy and efficiency through a composite scaling strategy.

ICC [43]: Rich features are extracted by multi-scale convolution, and accuracy and computational efficiency are balanced in counting.

MCNN [44]: A crowd counting model based on multi-column convolutional branching, adapting to different density and scale variations.

COUNTX [45]: A single-level model that does not distinguish between categories and can count the number of instances of any category based on image and text descriptions.

CLIP-COUNT [35]: A text-guided zero-shot object counting model capable of automatically completing object counting based on natural language prompts, demonstrating considerable application potential.

MSTNET [46]: A Transformer-based network for remote sensing segmentation that captures multi-scale features to improve accuracy.

GRAMFORMER [7]: A graph-modulated Transformer for crowd counting that diversifies attention maps via an anti-similarity graph and modulates node features using centrality encoding to address homogenization in dense scenes.

VL-COUNTER [47]: A text-guided zero-shot object counting model that leverages CLIP’s visual–linguistic alignment capability to generate high-quality density maps for unseen open-vocabulary objects.

DAVE [48]: A zero-shot object counting model that identifies and leverages high-quality exemplars to accurately count objects.

VA-COUNT [25]: An object counting model that combines detection and verification to reliably estimate counts.

4.3. Evaluation Metrics

In this paper, we use the mean absolute error (MAE) and the root mean square error (RMSE) to evaluate the performance of the model.

$$\mathrm{MAE}={\displaystyle \frac{1}{n}}\sum _{\mathrm{i}=1}^n\left|N_{pred}^{\mathrm{i}}-N_{gt}^{\mathrm{i}}\right|,$$

(19)

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}\sum _{\mathrm{i}=1}^n{\left(N_{pred}^{\mathrm{i}}-N_{gt}^{\mathrm{i}}\right)}^2},$$

(20)

where $N_{pred}^{\mathrm{i}}$ is the predicted number of the ith sample, $N_{gt}^{\mathrm{i}}$ is the true number of the ith sample, and n is the total number of samples.

We also use the structural similarity index measure (SSIM) and the peak signal-to-noise ratio (PSNR) to evaluate the similarity between the predicted and ground truth density maps. Equation (21) presents the formula for SSIM, and Equation (22) presents the formula for PSNR.

$$\mathrm{SSIM}(x,y)={\displaystyle \frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)({\sigma }_x^2+{\sigma }_y^2+C_2)}}$$

(21)

$$\mathrm{PSNR}=10·{log}_{10}\left({\displaystyle \frac{MA{X}_I^2}{\mathrm{MSE}}}\right)$$

(22)

where ${\mu }_x$, ${\mu }_y$, ${\sigma }_x^2$, ${\sigma }_y^2$, and ${\sigma }_{xy}$ represent the mean, variance, and covariance of x and y, $C_1$ and $C_2$ are constant values to prevent the division of zeros, and $MA{X}_I$ signifies the density map’s maximum value.

4.4. Measurement Setup

During the experiments, we use the ViT-B/16 encoder from the CLIP model as the image encoder, decoder-depth is set to 4, and decoder-head is set to 8. For MSE loss training, we perform 200 rounds, with 28 rounds of contrastive loss training before that, using a batch size of 32, with a learning rate set to 0.0001, and the adamw optimizer is used; the training is performed on an nvdia rtx- 4090 GPU. The image size of our dataset is 384 × 384.

4.5. Comparison Experiment

To demonstrate the reliability of our model, we introduce multiple comparative experiments across two datasets. In the Larch Casebearer dataset, we test three types: healthy, mildly diseased, and severely diseased. In the PDT dataset, we conduct tests on the severely diseased condition.

In the Larch Casebearer dataset, single-mode models struggle to effectively train across multiple tree typess simultaneously. To address this limitation, we adopt a strategy of independently training and testing each tree type. In contrast, our approach enables unified training of all types within a single framework by leveraging distinct prompts. For fair comparison with other models, we also employ type-wise testing during the evaluation phase.

As shown in Table 1, our model consistently achieves lower MAE and RMSE, and higher SSIM and PSNR across all test scenarios, outperforming the other comparative models. In addition, Table 2 shows that while the MAE of the other models generally exceeds 3, our model achieves a much lower MAE of 2.85, which is about 0.4 lower than the lowest MAE among the other models. Moreover, our model attains an SSIM of 92.1, whereas none of the other models exceed 90, and achieves a PSNR of 28.4, which is 0.7 higher than the best among the others. It can be observed that our model achieves the lowest MAE and RMSE, as well as the highest SSIM and PSNR across both datasets, suggesting that it not only delivers more accurate count estimations but also generates density maps with improved structural consistency. These comparative results clearly demonstrate the superior performance of our model in category-specific scenarios, thereby further validating its effectiveness.

Table 1. Comparison experiments using Larch Casebearer dataset. We tested our model against other models on the Larch Casebearer dataset, with the optimal results highlighted in bold.

Model	Healthy		Mild Disease		Severe Disease		SSIM	PSNR
	MAE	RMSE	MAE	RMSE	MAE	RMSE
ALEXNET	5.11	8.23	5.57	7.90	9.52	14.80	72.2	22.5
RESNET50	3.81	6.24	4.49	6.18	3.93	6.30	75.6	23.2
UNET	9.38	11.02	6.36	8.06	8.40	10.80	74.5	22.7
EFFICIENTNET	4.22	6.85	5.22	7.13	7.51	12.00	76.2	23.4
ICC	10.15	14.68	4.63	6.08	7.94	11.42	77.1	23.0
MCNN	4.98	7.82	7.41	9.81	4.91	7.93	70.2	22.0
COUNTX	4.97	6.21	8.28	11.09	7.86	8.98	82.6	24.6
CLIP-COUNT	3.95	5.96	4.98	6.75	5.05	6.82	85.6	25.9
MSTNET	5.02	8.10	6.28	7.85	6.23	8.50	84.1	24.9
GRAMFORMER	5.28	8.46	6.30	8.02	6.55	8.73	85.0	25.1
VL-COUNTER	3.93	6.36	4.63	6.43	4.56	5.98	86.3	26.5
DAVE	4.06	5.45	4.59	6.17	4.01	5.10	88.2	27.2
VA-COUNT	4.13	5.99	4.50	6.12	4.52	6.01	87.9	27.4
Ours	3.61	4.33	4.21	5.77	3.06	4.03	91.2	28.2

Table 2. Comparison experiments using the PDT dataset. We tested our model against other models on the PDT dataset, with the optimal results highlighted in bold.

Model	MAE	RMSE	SSIM	PSNR
ALEXNET	3.78	5.41	75.4	23.1
RESNET50	3.25	4.49	76.4	23.5
UNET	6.68	7.75	70.2	21.9
EFFICIENTNET	3.88	5.60	74.4	23.0
ICC	3.28	4.51	75.8	23.1
MCNN	4.32	6.01	73.6	22.7
COUNTX	6.56	9.02	82.6	25.1
CLIP-COUNT	4.20	6.13	85.3	26.0
MSTNER	5.28	6.86	84.7	25.4
GRAMFORMER	5.02	7.12	83.6	25.1
VL-COUNTER	4.05	5.83	86.8	27.0
DAVE	3.53	5.32	89.5	27.7
VA-COUNT	3.95	5.81	88.6	27.5
Ours	2.85	4.17	92.1	28.4

By accurately counting diseased trees within the target areas, our model not only proves its high precision and robustness in practical applications, but also provides a scientific foundation for disease monitoring and forest management. Specifically, the model enables efficient identification and counting of diseased trees, substantially reducing the cost and time of manual surveys while improving data accuracy and reliability. This achievement offers strong technical support for forest protection and ecological restoration, highlighting the great potential of deep learning technologies in natural resource management.

4.6. Ablation Experiment

To better validate each module in the model, we perform ablation experiments on it.

Our design idea is to use the image encoder and text encoder to output the corresponding information. We design the visual cue optimization in the image encoder, and we set the contrast loss function between the two corresponding outputs of the two modalities of information; after that, we fuse the two kinds of information, then generate the corresponding density maps by the decoder, and then compare them with the real density maps.

Firstly, we verify the reliability of our multimodal model by incorporating and removing textual information. As shown in Table 3 and Table 4, experiments are conducted on two datasets. The results clearly demonstrate that the MAE and RMSE values are consistently lower, while the SSIM and PSNR values are higher when textual input is included. These findings fully validate the effectiveness of our proposed multimodal model.

Table 3. Ablation experiment comparing model performance with and without textual input on the PDT dataset. The best results are highlighted in bold.

Setting	Healthy		Mild Disease		Severe Disease		SSIM	PSNR
	MAE	RMSE	MAE	RMSE	MAE	RMSE
With Text Input	3.61	4.33	4.21	5.77	3.06	4.03	91.2	28.2
Without Text Input	5.25	8.06	6.01	7.56	6.10	7.75	80.4	23.6

Table 4. Ablation experiment comparing model performance with and without textual input on the PDT dataset.

Setting	MAE	RMSE	SSIM	PSNR
With Text Input	2.85	4.17	92.1	28.4
Without Text Input	4.66	6.56	82.2	23.8

We define three variables: rotational position encoding (Rotate), contrast loss (Contrast), and fusion module (Fusion). ✓ indicates that the variable is enabled, while ✗ indicates that it is disabled. In the fusion module, Cross represents the fusion of image information and text information through a gated bidirectional cross-attention mechanism, and Add represents the operation of adding image information and text information. By designing these three variables, the effects of different variables on the model can be effectively observed. By choosing to enable and disable each module, the effectiveness of each module can be demonstrated. Val MAE and Val RMSE represent the mean absolute error and root mean square error of the validation set, and Test MAE and Val RMSE represent the mean absolute error and root mean square error of the test set, respectively. SSIM and PSNR denote the structural similarity index measure and peak signal-to-noise ratio, which are reported as the average values across the validation and test sets. In Table 5, it is easy to find that the model achieves the best results when using the rotational position encoding, contrast loss, and gated bidirectional cross-attention module, with a mean absolute error of 3.50 and a root mean square error of 4.75 for the validation set, and a mean absolute error of 3.62 and a root mean square error of 5.02 for the test set. In addition, the model attains an SSIM of 91.2 and a PSNR of 28.2, further demonstrating its superior reconstruction quality.

Table 5. Ablation experiments on the Larch Casebearer dataset, presenting the impact of different model components on the performance of the Larch Casebearer dataset. The best results are highlighted in bold.

Method	Rotate	Contrast	Fusion	Val MAE	Val RMSE	Test MAE	Test RMSE	SSIM	PSNR
1	✓	✗	Add	5.10	8.31	5.26	8.18	84.0	24.1
2	✗	✗	Add	5.32	8.30	5.70	8.86	83.7	24.0
3	✗	✓	Add	5.03	7.84	5.31	8.31	84.3	24.5
4	✓	✓	Add	4.90	7.53	5.21	8.40	85.0	25.3
5	✗	✗	Cross	4.63	5.87	4.62	6.25	86.7	26.3
6	✗	✓	Cross	4.09	5.30	3.90	5.10	87.4	27.1
7	✓	✗	Cross	4.20	5.13	4.47	6.02	87.6	27.4
8	✓	✓	Cross	3.50	4.75	3.62	5.02	91.2	28.2

In a continuation of the above design ideas, in Table 6, we find that when using the rotational position encoding, contrast loss, and gated bidirectional cross-attention module, the model achieves the best results, with a mean absolute error of 2.85 and a root mean square error of 4.17 in the validation set, and a mean absolute error of 2.91 and a root mean square error of 4.18 in the test set. Moreover, the model attains an SSIM of 92.1 and a PSNR of 28.4, further confirming the superior reconstruction quality and the reasonableness of introducing each module.

Table 6. Ablation experiments on the PDT dataset, presenting the impact of different model components on the performance of the PDT dataset. The best results are highlighted in bold.

Method	Rotate	Contrast	Fusion	Val MAE	Val RMSE	Test MAE	Test RMSE	SSIM	PSNR
1	✓	✗	Add	3.43	5.56	3.82	5.61	84.0	24.0
2	✗	✗	Add	3.95	6.43	4.02	6.26	83.5	23.7
3	✗	✓	Add	3.22	5.22	3.31	4.96	84.4	24.9
4	✓	✓	Add	3.30	5.36	3.58	5.26	84.6	24.7
5	✗	✗	Cross	3.68	5.17	3.82	5.25	86.3	26.4
6	✗	✓	Cross	3.43	4.90	3.42	4.68	88.0	26.9
7	✓	✗	Cross	3.39	4.58	3.55	4.79	88.7	27.2
8	✓	✓	Cross	2.85	4.17	2.91	4.18	92.1	28.4

4.7. Case Study

In order to better demonstrate the test results of our model on the two datasets, we have designed the “Case Study” Section. In this section, we will show some specific visualization results to visualize the prediction performance of our model in complex scenarios. We hope to convey the reliability of the model’s predictions more clearly through these specific cases.

Figure 4 shows the effect of the model’s visual recognition of diseased trees. The first row is the original image, the second row is the real point map, and the third row is the model-predicted point map. The colors of the points indicate different tree health states: orange for healthy trees, cyan for mildly diseased trees, and purple for severely diseased trees. The number following the color above each image is the number of trees of that type represented by that color. All images are taken from the Larch Casebearer dataset.

Figure 4. Visualization of test results: The first row is the original image, the second row is the real point map, and the third row is the model-predicted point map. The colors of the points indicate different tree health states: orange for healthy trees, cyan for mildly diseased trees, and purple for severely diseased trees. The number following the color above each image is the number of trees of that type represented by that color.

From the four examples in Figure 4, we can observe that even in complex scenarios such as dense trees and overlapping tree canopies, our model is still able to accurately localize the trees from the images according to the cue words. This is in comparison to traditional tree detection models, where usually, each model can only be trained for a single scene or a specific class of trees, and cannot handle tree detection tasks under different conditions. Our approach undoubtedly improves the practicality and simplicity by allowing the model to handle multiple tree types simultaneously during a single training session through a multi-task learning strategy. This not only improves the adaptability and robustness of the model, but also significantly enhances its practicality and simplicity.

4.8. Generalization Ability Test

To rigorously assess the generalization capability of our model, we conducted experiments on previously unseen test datasets from different geographic regions. This evaluation relies on quantitative performance metrics, allowing us to systematically examine the model’s robustness and adaptability across diverse spatial contexts. The results provide a comprehensive validation of the model’s effectiveness and reliability in scenarios not encountered during training.

The Kampe area is one of the four regions included in the Larch Casebearer dataset. It contains the largest number of annotations, and also has the highest average number of annotations per image. There are approximately 170 images from the Kampe area, with around 15,000 annotations. As shown in Table 7, although our model exhibits some decline in MAE, RMSE, SSIM, and PSNR, these metrics still remain within a reasonable range. This shows that the model not only is able to learn the features of a specific dataset, but also has some cross-domain adaptability, which validates the effectiveness and robustness of our approach when dealing with unknown data.

Table 7. Generalization experiment in Kampe area. We use images from the four locations in the Larch Casebearer dataset, excluding the Kampe area, as the training set, and evaluate the model on the Kampe area data to assess its generalization ability to unseen locations.

Area	MAE	RMSE	SSIM	PSNR
Kampe	8.90	13.24	85.1	26.2

The reason for our model’s strong generalization capability in classification, detection, and counting, which can effectively detect and roughly count trees in different scenarios even if it is not specifically trained on a specific dataset, is the model’s strong generalization performance and the migration learning mechanism, which extracts generalized features from existing knowledge and applies them to unknown data distributions.

5. Discussion

This study proposes an efficient approach to distinguish trees with varying health conditions for forestry statistical analysis, providing robust technical support for ecological monitoring and tree disease prevention. The proposed method demonstrates significant potential in assessing tree health status based on remote sensing data, offering strong support for the efficient management of forest resources. However, certain uncertainties remain, as the model’s performance may be affected by factors such as illumination conditions, shooting angles, atmospheric variations, and background interference.

In the experimental stage, two large-scale datasets were employed to provide a solid and representative foundation for model training. In the future, with the incorporation of more diverse and heterogeneous datasets, the diversity and coverage of training data are expected to be further enhanced. We plan to continuously collect and integrate additional relevant data resources to improve the model’s robustness and generalization ability under varying data distributions. The support of larger-scale data will not only optimize the model training process but also enhance its stability and accuracy in complex and diverse scenarios, thereby improving its adaptability and practical value. Furthermore, we intend to extend this method to broader geographic regions and integrate it with UAV-based canopy feature analysis to enhance its spatial scalability in diverse forest environments.

Our model is about 1.1 GB in size and processes each image in approximately 46 ms. It performs efficiently, but the speed is still not sufficient for real-time UAV aerial analysis. In future work, we will develop lighter and more optimized structures to improve real-time performance and deployment feasibility.

6. Conclusions

In this paper, we further improved the CLIP model by introducing rotational position encoding in the image encoder to improve the model’s ability to perceive diseased trees at different locations in the image, realized multimodal feature alignment and fusion through the gated bidirectional cross-attention module and the contrast loss function, utilized the decoder to generate the corresponding density map information for quantity counting, and ultimately constructed a deep learning model for diseased tree counting.

This deep learning model is applied to the task of recognizing and counting diseased trees in forests. It effectively extracts key regions from disease-affected images and integrates textual descriptions—such as disease types and characteristic terms—for accurate identification and classification. By enhancing the automation and precision of disease tree detection and tree counting, the proposed model holds promise for supporting forest resource protection and pest management. It can assist relevant authorities in early warning and informed decision-making, thereby providing technological support for ecological security and sustainable development.