MKG-CottonCapT6: A Multimodal Knowledge Graph-Enhanced Image Captioning Framework for Expert-Level Cotton Disease and Pest Diagnosis

Abstract

As one of the world’s leading cotton-producing countries, China frequently experiences severe yield reductions due to crop diseases and pest infestations, with losses often exceeding 20%. Although computer vision models can identify diseased plants, they currently fail to connect visual symptoms to the diagnostic reasoning process used by agronomists. This leads to text descriptions that ignore the biological causes of the damage. To fix this, we built Multimodal Knowledge Graph-Enhanced Cross Vision Transformer-18-Dagger-408 and Text-to-Text Transfer Transformer for Cotton Disease and Pest Image Captioning (MKG-CottonCapT6), a model that uses a local knowledge database to generate professional diagnostic reports from field images. The technical core consists of a Multimodal Knowledge Graph (MMKG) containing 14 types of entities (such as Pathogens and Control Agents) and 12 types of relations. We use a Cross Vision-Transformer-18-Dagger-408 (CrossViT) encoder to capture both the overall leaf shape and microscopic details of pests. Through a Visual Entity Grounding (VEG) module, the model maps visual features directly to specific triplets in the graph. These triplets are then turned into text sequences and fused with image data in a Text-to-Text-Transfer-Transformer (T5) decoder. To train the model, we collected a dataset of cotton images paired with expert descriptions of lesions, colors, and affected plant parts. Tests show that MKG-CottonCapT6 performs better than standard models, reaching an Information-based Metric for Image Captioning (InfoMetIC) score of 72.6%. Results prove that by using a specific alignment loss (${\mathcal{L}}_{align}$), the model generates reports that correctly name the disease stage and recommend specific chemicals, such as Carbendazim or Triadimefon. This framework provides a practical tool for farmers to record and treat cotton diseases with high precision.

1. Introduction

Cotton production serves as the cornerstone of China’s textile supply chain, making stable agricultural yields essential for both the national economy and industry security. Although this region serves as a global powerhouse for cotton production, its output remains under constant threat from a complex array of biological stressors. Pathological outbreaks—most notably Fusarium wilt and the cotton bollworm (Helicoverpa armigera)—frequently inflict devastating damage, potentially slashing harvest volumes by over one-fifth and significantly compromising fiber integrity (10–15% degradation).

While deep learning advancements have revolutionized automated classification, a critical “interpretability deficit” persists. Traditional architectures typically operate as opaque systems that yield categorical outputs but lack the granular semantic depth essential for expert-level diagnostics. To address this, current AI research is pivoting toward knowledge-driven paradigms. As highlighted by Chen et al., Multimodal Knowledge Graphs (MMKGs) provide a sophisticated mechanism for integrating unstructured visual evidence with structured expertise [1]. In this framework, transitioning from rudimentary image recognition to knowledge-augmented decision support represents a vital evolution for modern smart agriculture. Building directly upon this paradigm, our proposed contribution constructs a domain-specific MMKG tailored for cotton agriculture, aiming to transition the field from rudimentary image recognition to knowledge-augmented decision support.

A primary obstacle in current multimodal learning is managing semantic ambiguity in unstructured data and the inherent “hallucinations” of Large Language Models (LLMs). When a model identifies a lesion, it often lacks the logical grounding to distinguish between nutritional deficiency and pathogen-induced necrosis, leading to field error rates in manual inspections of up to 35%. Liu and Kim et al. demonstrated that utilizing cascaded pre-trained models and multimodal entity-linking frameworks can effectively filter alignment noise and improve reasoning accuracy [2,3]. To address the aforementioned hallucinations in agricultural applications, our framework specifically adapts these entity-linking and noise-filtering mechanisms to accurately map visual lesion data to botanical taxonomies. Regarding representation enhancement in specific scenarios, Wang et al. significantly improved the detail of scene descriptions by introducing Graph Attention Networks (GATs) and multi-scale semantic guidance, while Rahman’s work in disaster assessment revealed the feasibility of transforming basic visual classification into comprehensive real-time situational intelligence [4,5]. These technical evolutions provide a theoretical foundation for characterizing the complex features of cotton pests. Specifically, Yang’s attempt to incorporate historical and cultural contexts into artwork descriptions highlights the necessity of integrating botanical pathology and entomology priors—such as leaf pubescence, vein structure, and boll morphology—into cotton pest descriptions, ensuring that generated outputs move beyond surface features to encompass pathological causes and control logic [6]. By synthesizing Chen’s fundamental MMKG architecture [1] with the robust reasoning techniques of Liu and Kim [2,3], and the domain-prior integration inspired by Yang [6], our proposed framework effectively translates these cross-domain innovations into a unified, expert-level image captioning tool for cotton pest and disease diagnosis.

In the paradigm of professional report generation, research in medical imaging offers invaluable insights for agricultural diagnosis. The “divide and conquer” and “fine-grained image-text alignment” methods proposed by Liang and Yin et al. significantly enhanced clinical reliability by distinguishing between subtle normal and abnormal attributes [7,8]. Similarly, Li, Wang, and Usman et al. utilized knowledge graphs to extract disease features and assist LLMs in report generation, proving the efficacy of structured professional knowledge in mitigating model hallucinations [9,10,11]. This logic of transforming medical images into structured descriptions can be seamlessly migrated to cotton pest diagnosis: leaf lesions and pest-feeding marks are functionally analogous to medical “pathological foci.” These require the knowledge graph-guided mechanisms proposed by Gupta to ensure that generated descriptions align with the diagnostic logic of crop protection experts, capturing critical details like the shape and color of lesions that are often neglected by end-to-end paradigms [12].

The rise of Retrieval-Augmented Generation (RAG) has further demonstrated the potential of integrating knowledge graphs with LLMs to deepen inference capabilities. Schluckebier, Bahr, and Park et al. constructed RAG systems driven by heterogeneous KGs in earth observation and industrial failure analysis, effectively addressing the limitations of knowledge updating and domain specificity [13,14,15]. In visual question-answering (VQA), Yuan and Bu achieved state-of-the-art performance in knowledge-intensive tasks by dynamically constructing local KGs [16,17]. In the agricultural domain, the “SheepDoctor” model developed by Xiong et al. integrated domain-specific KGs via LoRA fine-tuning, outperforming general-purpose models and validating the potential of domain KGs in enhancing diagnostic reliability [18]. Complementing this, the Sem-RAG algorithm proposed by Bai et al. demonstrated that improving detailed semantic retrieval can effectively resolve complex queries in corn cultivation. This further demonstrates the versatility of RAG-based architectures in specialized agricultural contexts [19]. Furthermore, explorations by Huang and Wu et al. in remote sensing and semantic communication further confirm the universal value of KGs in eliminating ambiguity and improving interpretability across complex visual landscapes [20,21].

Regarding representation enhancement in specific scenarios, Zhao et al. recently developed CottonCapT6, a multi-task framework that leverages CrossViT and T5 to provide preliminary diagnostic descriptions for cotton pests, marking a significant step toward automated agricultural reporting [22]. Despite these breakthroughs, research on multimodal description enhancement for localized cotton cultivation remains in its infancy, constrained by a lack of specialized datasets and immature image-text alignment. Existing agricultural image captioning methods often struggle to deeply integrate essential domain contexts, including specific climates, geographical backgrounds, and variety-specific disease resistance. Zhang, in digital taxation, Cao, in recommendation and continual learning, and Fang, in molecular property prediction, all utilized KGs to eliminate noise and enhance feature expression. Although these applications span disparate domains, their core mechanism of using structured relational data to clarify complex, noisy environments is directly transferable to agricultural diagnostics, where visual symptoms and biological concepts are deeply intertwined. Building upon this cross-domain potential, this paper proposes a framework for enhancing cotton pest and disease image descriptions based on MMKGs [23,24,25,26]. By integrating expert knowledge bases, optimizing cross-modal feature alignment, and introducing an advanced RAG mechanism, this research aims not only to improve identification accuracy but also to generate highly professional and interpretable descriptive texts, providing a scientific basis for the intelligent and precise prevention of cotton pests.

Motivated by these gaps, this study introduces MKG-CottonCapT6 and demonstrates how it significantly advances the diagnosis of cotton diseases and pests by integrating an MMKG with advanced deep learning architectures, including CrossViT and T5. The main contributions of this research are as follows:

• A specialized multimodal knowledge graph was meticulously constructed to provide domain-specific image and structured knowledge relations, serving as the foundational expertise for the accurate diagnosis of cotton diseases and pests.
• A new knowledge-enhanced image captioning framework, MKG-CottonCapT6, was proposed. This framework innovatively integrates an MMKG with a multi-scale visual encoder (CrossViT) and a text generation model (T5) to generate descriptions that are not only visually grounded but also botanically rigorous through entity anchoring and triplet retrieval.
• A knowledge-augmented integration strategy was designed. Beyond the two-stage training strategy (classification pretraining followed by captioning), this framework leverages structured relations within the MMKG to provide latent semantic supplementation, enabling the model to transition from simple visual reporting to expert-level diagnostic reasoning.

The model effectively synthesizes visual evidence with structured expertise, significantly improving the semantic accuracy, interpretability, and professional depth of diagnostic descriptions, thereby offering a robust decision-support tool for the intelligent management of cotton cultivation in Xinjiang.

2. Materials and Methods

2.1. Research Framework

With the ongoing maturation of deep learning, agricultural diagnostics are evolving from simple recognition toward knowledge-dense multimodal synthesis, such as image captioning. Traditional data-driven models, however, often lack the domain-specific logic required for precise agricultural guidance. To bridge this gap, we developed an MMKG-enhanced framework designed to systematically integrate visual features with structured expert knowledge of Xinjiang cotton pests and diseases. As illustrated in Figure 1, the architecture utilizes a specific logical hierarchy: raw data inputs and physical storage are denoted by document and cylinder shapes; rectangles represent standard algorithmic operations; rounded rectangles highlight core deep learning and fusion models; while parallelograms and ovals indicate intermediate feature representations and knowledge structures.

Figure 1. Research framework of the proposed MKG-enhanced diagnostic system. The pipeline comprises three sequential phases: (1) multi-source data acquisition and preprocessing; (2) construction and storage of the Multimodal Knowledge Graph (MMKG); and (3) knowledge-augmented model integration for diagnostic caption generation. Distinct geometric shapes represent different functional layers, including data inputs, algorithmic operations, and the fusion architecture.

In the first phase, a multi-source acquisition strategy was implemented to gather high-quality visual data from platforms such as Baidu and Google, alongside authoritative text from official institutional platforms and professional monographs. To ensure data quality, rigorous Data Cleaning and Data Augmentation procedures were performed, establishing a robust empirical foundation for characterizing complex pathological features. In the second phase, a specialized Multimodal Knowledge Graph was meticulously developed. This process involves a sequential workflow starting from Ontology Design and Schema Modeling, followed by Fine-grained Semantic Annotation and Triplet Extraction, and concluding with systematic Graph Storage. This graph serves as the structural backbone, capturing the intricate relationships between disease symptoms, pathogens, and environmental factors. In the final phase, this structured knowledge is fused into a vision-language architecture. As shown in the “Step 3” block of Figure 1, input images are processed by a CrossViT encoder to extract multi-scale visual features. Simultaneously, through Entity Anchoring, relevant domain knowledge is accessed from the MMKG via Subgraph Retrieval and Triplet Linearization, generating a structured Knowledge Sequence (K). Finally, a T5-based Multimodal Cross-Attention Fusion mechanism synthesizes the visual and textual streams to produce enhanced diagnostic captions.

By leveraging the structured relations within the MMKG, the model is guided to generate outputs that are not only visually accurate but also botanically rigorous. Unlike conventional end-to-end methods that rely solely on image-text pairs, this three-phase approach allows the framework to consolidate domain-specific expertise through the KG before performing cross-modal alignment. This ensures that the resulting diagnostic reports move beyond surface-level descriptions to encompass deep pathological logic, providing a more robust decision-support tool for Xinjiang’s smart agriculture.

2.2. Multimodal Knowledge Graph Construction

2.2.1. Data Acquisition and Source Selection

This study utilized a dataset that was specifically collected and curated to encompass seven representative categories of cotton diseases and pests. The foundational image acquisition involved crawling 3500 raw image samples from the Baidu and Google Image Libraries using semantic-driven keyword matching. To ensure high data quality, the raw images underwent a rigorous filtering process: images were standardized, perceptually hashed using Python scripts to eliminate duplicates, and manually screened to remove low-quality samples, retaining 784 foundational images. Subsequently, a comprehensive data augmentation strategy was employed to simulate complex field conditions. This strategy included random brightness adjustments, spatial flipping, and noise injection. After augmentation, the dataset was expanded to 2801 images.

To strictly prevent data leakage, these foundational images were first partitioned into training, validation, and test sets in an 8:1:1 ratio. Subsequently, a comprehensive data augmentation strategy was applied exclusively to the training set to simulate complex field conditions. Specifically, this strategy incorporated random brightness adjustments, spatial flipping, and noise injection. After augmentation, the total number of images utilized for training and evaluation was expanded to 2801. To adapt the CottonDP dataset for the proposed multimodal framework and rigorously evaluate the semantic alignment using the InfoMetIC metric, a secondary fine-grained annotation process was conducted by plant protection professionals. Beyond simple categorical labeling, each image was structurally associated with key attributes derived from expert literature. These attributes included specific details such as the affected plant site.

The robustness of a Knowledge Graph is heavily contingent upon the authority of its foundational data sources. To construct an expert knowledge base with distinct regional characteristics of Xinjiang, we systematically extracted unstructured text from authoritative agricultural online platforms and seminal professional monographs, most notably Xinjiang Cotton Pests and Diseases and Their Control. This literature provides meticulous documentation of the unique pathological evolution patterns in Xinjiang, including symptom manifestations during specific growth stages (e.g., squaring and bolling stages) and the corresponding biological or chemical control strategies. Utilizing Natural Language Processing (NLP) techniques for cleaning and parsing, we identified deep logical correlations among pathogens, host plants, environmental factors, and symptomatic expressions.

To address the scarcity of visual prototypes during the construction of the Multimodal Knowledge Graph (MMKG), this study deployed a Python-based web crawling framework to perform targeted acquisition from the Baidu and Google Image Libraries, serving as an effective supplement to the original dataset. Diverging from conventional generic image scraping, our collection strategy strictly adhered to a “semantic-driven” principle, executing precise keyword matching tailored to specific growth stages and typical pathological characteristics. Ultimately, during the construction of the MMKG, we seamlessly integrated the high-quality images from the original dataset’s training set with the targeted crawled supplementary images. This strategy strictly isolates the test set from the knowledge base, significantly enriching the foundational visual representations while ensuring the integrity of the zero-shot or graph-guided evaluations.

2.2.2. Ontology Design and Entity Annotation

To accurately represent the complex domain of cotton pathology, we developed a comprehensive ontology serving as the semantic blueprint for the MMKG. This ontology encompasses 14 entity types and 12 relation types, covering the entire lifecycle of pest and disease management from initial symptom observation to final control intervention. These categories include biological identifiers such as Disease Name, Pest Name, and Pathogen, as well as diagnostic attributes like Affected Part, Symptoms, and Occurrence Period. Notably, we incorporated Disease Image and Pest Image as specific multimodal entities to facilitate visual-semantic anchoring. The detailed categorization and statistical distribution of these entity and relation types are systematically summarized in Table 1.

Table 1. This table presents the statistical categorization of the foundational components within the MMKG. It encompasses 14 distinct entity types (e.g., Disease Name, Pathogen, and Image assets) and 12 predefined semantic relations. These relations establish the logical connectivity between pathological features, temporal occurrence, and management strategies, providing the structural backbone for knowledge-augmented diagnostic reporting.

No.	Entity Type	Relation Type
1	Disease Name	Relation between Pest/Disease and Symptoms
2	Affected Part	Relation between Pest/Disease and Affected Part
3	Occurrence Period	Relation between Affected Part and Symptoms
4	Symptoms	Relation between Occurrence Period and Symptoms
5	Pathogen	Relation between Disease and Pathogen
6	Occurrence Law	Relation between Pest/Disease and Occurrence Law
7	Control Methods	Relation between Pest/Disease and Control Method
8	Distribution Area	Relation between Pest/Disease and Distribution Area
9	Alias	Relation between Pest/Disease and Alias
10	Control Agents	Relation between Pest/Disease and Control Agent
11	Pest Morphology	Relation between Pest and Pest Morphology
12	Pest Name	Relation between Pest/Disease and Image Description
13	Disease Image
14	Pest Image

The structural integrity of the graph is maintained through four layers of predefined semantic relations that organize these entities into a coherent knowledge web. These layers are detailed as follows.

Pathological Core Relations define fundamental biological links such as “Pest/Disease–Symptom,” “Pest/Disease–Affected Part,” and “Part–Symptom”. This layer serves as the diagnostic spine, mapping visual anomalies to specific anatomical locations on the cotton plant.

Contextual Relations, by linking “Occurrence Period–Symptom,” “Disease–Pathogen,” and “Pest/Disease–Occurrence Law,” integrate temporal and causal dimensions unique to the Xinjiang region.

Management Relations provide actionable intelligence by connecting “Pest/Disease” to “Control Method,” “Control Agent,” and “Distribution Area,” transforming the framework into a proactive decision-support platform.

Identification Relations, being crucial for multimodal fusion, provide linguistic and visual grounding. By establishing the “Pest/Disease–Image Description” relation as a bridge for cross-modal alignment, the model can map low-level pixel features directly to professional botanical terminology.

This structured hierarchy ensures that the generated diagnostic reports are not only semantically precise, but also contextually grounded in the professional logic of crop protection.

2.2.3. Data Processing and Multimodal Integration

The construction process involved rigorous entity and relation extraction from the crawled textual segments. Using specialized annotation tools, domain experts labeled entities and their corresponding relations to form triplets. The construction process followed a rigorous pipeline of extraction and annotation. To ensure high-quality semantic grounding, we utilized doccano, an open-source labeling platform, for manual annotation. As illustrated in Figure 2, domain experts utilized the doccano platform’s interactive interface to perform fine-grained semantic labeling on extensive segments of professional Chinese text. The process involved two primary layers of annotation. First, the identification of key botanical and entomological entities was conducted. For example, this involved anchoring the pest name ‘Cotton Bollworm’ and its corresponding ‘Morphological Features’ across various life stages, including the egg and larval phases. Second, as shown by the connective lines in the interface, experts manually established the relational links between these entities to construct precise knowledge triplets, such as the direct association between a specific pest and its morphological descriptions. This rigorous human-in-the-loop annotation ensures that the resulting MMKG captures the complex diagnostic logic required for professional-grade cotton pathology analysis.

Figure 2. Manual annotation interface for text-based knowledge extraction using the doccano platform. The figure demonstrates the specialized semantic labeling process conducted by domain experts on unstructured Chinese agricultural texts. Specifically, the interface displays the following key features. The blue lines indicate specific entity and relationship annotations within the Neo4j graph database.

A distinctive feature of our MMKG is the strategic integration of visual assets. Rather than storing large binary image data within the graph database, we utilized local file paths as unique identifiers. Each “Disease Image” or “Pest Image” entity is mapped to its corresponding high-resolution file in a local repository, ensuring high query efficiency while maintaining a direct link between structured knowledge (e.g., “brown circle lesions”) and raw visual evidence.

Ultimately, by associating each image with specific pathological attributes and “Image Description” entities, we formed the Multimodal Knowledge Graph of Xinjiang Cotton Pests and Diseases. During the captioning process, this graph functions as an expert “lookup table,” allowing the model to supplement visual features with hidden contextual information, such as specific control agents or local distribution patterns, thereby significantly enhancing the depth and professional utility of the generated diagnostic reports.

A unique feature of our MMKG is the integration of visual assets. Instead of storing large binary image data directly within the graph database, we utilized local file paths as unique identifiers. Each “Disease Image” or “Pest Image” entity in the knowledge graph is mapped to its corresponding high-resolution file stored in the local repository. This approach ensures high query efficiency while maintaining a direct link between the structured knowledge (e.g., “brown circle lesions”) and the raw visual evidence.

Finally, by associating each image with its specific pathological attributes and “Image Description” entities, we formed a Multimodal Knowledge Graph of Xinjiang Cotton Pests and Diseases. This graph provides the model with a “lookup table” of expert knowledge during the captioning process, allowing it to supplement visual features with hidden contextual information, such as specific control agents or local distribution patterns, thereby significantly enhancing the depth and utility of the generated descriptions.

2.3. Multimodal Knowledge Graph-Enhanced Image Captioning Model

To address the semantic gaps and diagnostic inaccuracies inherent in traditional captioning models when processing complex cotton disease features in Xinjiang, this study proposes the MKG-CottonCapT6 framework. An architectural overview is illustrated in Figure 3, and the procedural logic is detailed in Algorithm 1.

Algorithm 1: MKG-Enhanced Cotton Disease Captioning via Dual-Branch Visual Encoding

Require: Cotton leaf image dataset $\mathcal{D}=\{I_1,\dots ,I_n\}$, Multimodal Knowledge Graph ${\mathcal{G}}_{MMKG}$ Ensure: Expert-level diagnostic captions ${\mathcal{C}}_{Enhanced}$ with agricultural decision support 1. Stage 1: Multi-Scale Visual Encoding 2.$\mathrm{for} \mathrm{each} \mathrm{image} I_i\in \mathcal{D}$ do 3.         Extract dual patches: $\{P_l,P_s\}\leftarrow \mathrm{PatchPartition}(I_i)$ 4.         Encode features: $M_{feat}\leftarrow \mathrm{CrossViT}(P_l,P_s)$ using Cross-Attention 5. end for 6. Stage 2: VEG 7. Project global token: $v_e\leftarrow f(\sigma (x_{cls}^l{W}_P+b_P))$ 8. Calculate similarity: $S=\alpha \cdot \cos (v^{\prime },E_{text})+(1-\alpha )\cdot \cos (v^{\prime },E_{img})$ 9. Anchor target entity: $e_{target}\leftarrow \arg \max (S_i)$ (e.g., Target Spot) 10. Stage 3: Knowledge Subgraph Retrieval & Linearization 11. Extract n-hop subgraph: $G_{sub}\leftarrow \mathrm{Retrieve}({\mathcal{G}}_{MMKG},e_{target},n=2)$ 12. Generate professional prompts: $K_{seq}\leftarrow \mathrm{TemplateLinearization}(G_{sub})$ 13.        Result: “[KNOWLEDGE]: Cotton leaves have multiple lesions proving target spot.” 14. Stage 4: Multimodal Feature Fusion 15. Concatenate sequences: $X_{input}\leftarrow [M_{feat};K_{seq}]$ 16. Perform dual semantic calibration: 17.        ${\omega }_{dyn}\leftarrow \mathrm{DynamicWeightAllocation}({\mathrm{Confidence}}_{visual})$ 18.        $Z_{fusion}\leftarrow \mathrm{MultiHeadCrossAttention}(X_{input},{\omega }_{dyn})$ 19. Stage 5: Knowledge-Augmented Caption Generation 20.$\mathrm{for} \mathrm{each} \mathrm{encoded} \mathrm{sequence} Z_{fusion}$ do 21.         Perform diagnostic inference: ${\mathcal{C}}_i^{Enhanced}\leftarrow \mathrm{T}5\mathrm{Decoder}(Z_{fusion})$ 22.        ${\mathcal{C}}_{Enhanced}\leftarrow {\mathcal{C}}_{Enhanced}\cup \left\{{\mathcal{C}}_i^{Enhanced}\right\}$ 23. end for 24.$\mathrm{return} {\mathcal{C}}_{Enhanced}$

Figure 3. The overview of the MKG-CottonCapT6 framework for knowledge-augmented cotton disease image captioning. The pipeline integrates three primary stages: multi-scale visual encoding, cross-modal knowledge alignment, and T5-based multimodal feature fusion. This architecture facilitates the generation of expert-level diagnostic reports that provide actionable agricultural decision support.

Specifically, the system initiates the Multi-Scale Perception stage (Algorithm 1, Lines 1–5). As shown in the top block of Figure 3, the input cotton leaf image is processed through a dual-branch encoding mechanism. This includes a Large Branch (L-Branch) and a Small Branch (S-Branch) using varying patch sizes ($P_l,P_s$). This configuration allows the CrossViT-based encoder to simultaneously capture global pathological context and fine-grained symptomatic details through cross-attention interactions.

The extracted global visual token $x_{cls}^l$ then proceeds to the Cross-Modal Grounding phase (Figure 3, Stage 2; Algorithm 1, Lines 6–9). In this module, $x_{cls}^l$ is projected into a multimodal embedding space via the learnable matrix $W_p$. To ensure precise semantic anchoring, the system identifies the target entity $e_{target}$ by calculating the cosine similarity against the MMKG’s dual-modal libraries: the semantic library $E_{lib}$ and the visual centroid library $V_{lib}$. This ensures that even under significant intra-class variance or noisy field conditions, the visual features are correctly mapped to the corresponding pathological cluster.

During the Knowledge Retrieval & Integration stage (Algorithm 1, Lines 10–18), a second-order subgraph centered on $e_{target}$ is retrieved. This allows the model to access structured expert knowledge, including pathogen-environment correlations and disease evolution patterns. As illustrated in the fusion block of Figure 3, a dynamic weight allocation mechanism, ${\omega }_{dyn}$, is introduced to balance the influence of raw visual cues and retrieved knowledge priors.

Finally, in the Augmented Decoding phase (Algorithm 1, Lines 19–24), the T5-based decoder performs diagnostic inference on the fused multimodal sequence. By executing a dual semantic calibration process, the model transcends simple visual recitation. The final output is an expert-level diagnostic report that provides precise disease identification and actionable control recommendations, such as the application of specific chemical agents, thus bridging the gap between raw data and practical plant protection.

2.3.1. Dual-Branch Visual Encoding via CrossViT

To capture the co-occurrence of macro-scale pathological structures (e.g., leaf wilting patterns) and micro-scale morphological features (e.g., minute pest limbs) in Xinjiang cotton, this study utilizes CrossViT as the backbone visual encoder. The architecture employs a dual-branch mechanism, wherein a Large Branch processes large-sized patches to capture global context and spatial distribution, while a Small Branch utilizes fine-grained patches to focus on intricate textures and symptomatic details.

Multi-scale feature interaction is achieved through Cross-Attention layers interspersed throughout the encoder, enabling the exchange of spatial information between branches. The fundamental difference between standard intra-branch processing and the proposed inter-branch interaction is illustrated in Figure 4.

Figure 4. The comparison of the attention mechanisms. The left one is the Self-Attention Mechanism. This module captures internal spatial dependencies within a single scale by deriving the Query ($Q$), Key ($K$), and Value ($V$) vectors from the same input embedding sequence. The right one is the Cross-Attention Mechanism. The Query ($Q$) is projected from the Large Branch’s class token ($x_{cls}^l$), while the Key ($K$) and Value ($V$) are derived from the concatenated tokens of the Small Branch.

While the Self-Attention Mechanism is proficient at modeling long-range dependencies within a uniform scale, it is inherently limited by “scale-isolation,” meaning it cannot leverage the complementary cues available at different patch granularities. To overcome this, our Cross-Attention Mechanism utilizes the class token of the Large Branch ($x_{cls}^l$) as a semantic anchor to “query” the detailed morphological features ($x_{patch}^s$) embedded within the Small Branch. This design choice is critical for cotton pathology; it ensures that the resulting feature matrix ($M_{feat}\in {ℝ}^{N\times D}$) contains a fused representation where macro-level disease patterns are contextually grounded by micro-level pest evidence. Furthermore, the integration of residual connections across both mechanisms ensures that the original structural integrity of the cotton plant is maintained, preventing the loss of subtle pathological nuances during the deep feature extraction process.

Crucially, the resulting fused feature matrix ($M_{fused}$) is designed to be compatible with the multimodal nature of our KG. Unlike unimodal knowledge bases, the MMKG contains an image modality library where each disease entity is linked to a set of prototypical symptom images. This allows the extracted visual features to be cross-referenced with standard pathological images, providing a secondary layer of visual verification before semantic decoding.

2.3.2. Semantic Alignment and Knowledge Subgraph Retrieval

The core of the enhancement mechanism lies in establishing a bidirectional mapping between the unstructured visual features $M_{feat}$ and the structured semantic space $S_{MMKG}$ of the Multimodal Knowledge Graph. We designed the VEG module to facilitate this bridge through the following three distinct stages.

1.

Entity Anchoring aims to transform raw visual perceptions into precise semantic coordinates within the KG space. First, the global class token $x_{cls}^l\in {ℝ}^{1\times D}$ is extracted from the Large Branch of CrossViT. To eliminate the dimensional discrepancy between modalities, we introduce a learnable projection matrix $W_P\in {ℝ}^{D\times d_e}$ to map the visual vector into the KG’s entity embedding space:

$$v_e=f(\sigma (x_{cls}^l{W}_P+b_P)),$$

(1)

where $\sigma$ denotes the activation function and $b_P$ is the bias term. Subsequently, the target entity $e_{target}$ is identified by calculating the cosine similarity between the projected vector $v_e$ and the candidate entity embeddings $E=\{e_1,e_2,\dots ,e_n\}$ within the entity library:

$$e_{target}={\arg \max }_{e_i\in E}\left({\displaystyle \frac{v_e\cdot e_i}{\parallel v_e\parallel \parallel e_i\parallel }}\right),$$

(2)

Beyond semantic alignment, the VEG module also leverages the visual prototypes stored in the MMKG. The entity library $E_{lib}$ includes not only textual embeddings but also pre-computed visual centroids ($v_{img}$) derived from gold-standard pathology images. The identification of the target entity $e_{target}$ is thus a multimodal consensus:

$$e_{target}=\mathrm{argmax}(\cos (v^{\prime },E_{text})+\alpha \cos (v^{\prime },E_{img})),$$

(3)

where $\alpha$ is a modality weighting factor. This dual-modal anchoring significantly mitigates category confusion, as the model can distinguish visually similar diseases (e.g., Wilt and Target Spot) by comparing the input against the distinctive visual clusters within the MMKG.

The detailed operational flow of this anchoring process is visualized in Figure 5. By mapping the high-dimensional visual features through $W_P$, the model effectively bridges the gap between raw pixel patterns and abstract agricultural concepts. Unlike traditional classification heads that output fixed labels, this embedding-based alignment allows for more flexible grounding; even if a specific disease symptom has high intra-class variance, the similarity-based retrieval ensures the visual vector $v_e$ is pulled toward the correct semantic cluster in the KG space. As illustrated by the “Target Spot” example in Figure 4, this mechanism ensures that visual cues are translated into a stable entry point for the subsequent expert knowledge retrieval, even under varying field lighting and leaf orientations.

Figure 5. Detailed workflow of the VEG module for entity anchoring. The module maps visual tokens into a multimodal embedding space to facilitate semantic alignment between image features and knowledge graph entities. This process provides a precise entry point for the subsequent retrieval of expert pathological knowledge.

2.

During the subgraph extraction process, upon identifying $e_{target}$, the system dynamically retrieves domain-specific expertise from the MMKG. Defining the knowledge graph as $G=(V,R)$, where $V$ represents the set of entities and $R$ the set of relations, we employ an $n\text{-}\mathrm{hop}$ retrieval strategy to extract a second-order subgraph $G_{sub}$ rooted at $e_{target:}$

$$G_{sub}=\{(e_h,r,e_t)|\mathrm{dist}(e_{target},e_h)\le n,\hspace{1em}(e_h,r,e_t)\in G\},$$

(4)

In cotton pathology scenarios, $n$ is typically set to 2 to balance knowledge breadth with computational efficiency. This ensures that the retrieved triplets encompass not only descriptions of the pest itself but also actionable intelligence such as chemical control agents and regional occurrence laws.

3.

In the triplet linearization step, to make the graph structure interpretable for the T5 decoder, we employ a template-based linearization strategy. For instance, the triplet (Cotton Aphid, Affected Part, Back of Leaf) is transformed into the natural language sequence “The cotton aphid primarily infests the back of leaves.” This sequence serves as a “professional prompt” that complements latent information not directly observable in the image.

2.3.3. Multimodal Feature Fusion and Knowledge-Augmented Decoding

The decoding stage utilizes the T5-Base model. The input to the decoder consists of the concatenated sequence of visual features and linearized knowledge tokens. During the generation process, the model performs dual semantic calibration through a Multi-Head Cross-Attention mechanism.

• The dynamic weight allocation mechanism ensures that the visual encoder exhibits low confidence due to environmental noise (e.g., poor lighting), the decoder automatically increases the attention weight assigned to the knowledge-retrieved multimodal priors. Specifically, the model integrates the reference image features associated with the retrieved subgraph alongside linearized tokens. This allows the framework to ‘rectify’ distorted visual signals by utilizing the stable, high-fidelity visual exemplars from the MMKG, ensuring that the diagnostic report remains grounded in expert-verified visual evidence.
• During diagnostic inference, the T5 decoder maps visual perceptions (e.g., “brown spots”) to pathological logic (e.g., “early-stage Target Spot”). The resulting output transcends simple “visual recitation” and evolves into an expert-level diagnostic report. For example, instead of merely describing “brown spots on a leaf,” the model generates: “The leaf displays brown circular lesions characteristic of Target Spot; application of Carbendazim is recommended for control.” This mechanism significantly enhances the interpretability and decision-support utility of image captioning in Xinjiang’s agricultural production.

2.4. Experimental Setup

The experimental framework was implemented on an Ubuntu 20.04 operating system. Computational power was provided by an Intel Xeon Platinum 8457C CPU and 100 GB of RAM, while model training and inference were accelerated using an NVIDIA L20 GPU (48 GB). The software environment was built using Python 3.10 and PyTorch 2.1.2 with CUDA 11.8 support. Additionally, the multimodal knowledge graph was constructed and managed using Neo4j v2025.10.1.

2.5. Evaluation Indicators

To evaluate the performance of the MKG-CottonCapT6 framework, we utilize a combination of traditional linguistic metrics and modern multimodal evaluation benchmarks. For the core image captioning task, Bilingual Evaluation Understudy Score (BLEU-n) is used to measure n-gram precision, which is calculated as:

$$BLE{U}_n=BP\cdot \exp \left({\displaystyle \sum _{n=1}^N{w}_n}\log p_n\right),$$

(5)

where $p_n$ represents the n-gram precision, $w_n$ denotes the weight assigned to each n-gram precision and $BP$ is the brevity penalty. Typically, $w_n$ is set to a uniform value of $1/N$. Specifically, BLEU1-4 and the Metric for Evaluation of Translation with Explicit Ordering (METEOR) are employed to measure the syntactic fluency and structural alignment between the generated diagnostic reports and human-annotated references. METEOR is computed based on the harmonic mean of precision ($P$) and recall ($R$) of unigram matches:

$$METEOR=(1-Penalty)\cdot {\displaystyle \frac{P\cdot R}{\alpha \cdot P+(1-\alpha )\cdot R}},$$

(6)

In this formula, the $Penalty$ term is introduced to account for word order and fragmentation. It explicitly reduces the overall score if the matched words in the generated sequence are scattered into multiple non-adjacent chunks rather than forming continuous, well-ordered phrases aligned with the references.

To further assess the unique knowledge-enhanced capabilities of our model, we incorporate Contrastive-Language-Image-Pretraining Score (CLIPScore) and InfoMetIC into our evaluation suite. According to the characteristics of the expert-level captioning task, the accuracy of language generation is determined not only by traditional metrics reflecting semantic relevance and readability, but also by the model’s ability to maintain cross-modal consistency.

Specifically, CLIPScore is utilized to quantify the semantic grounding between the visual pathological features and the resulting text. It is defined as the rescaled cosine similarity between the image embedding $v$ and the text embedding $t$ extracted by the CLIP VIT-B/32 model. $w$ represents a constant rescaling weight used to amplify the similarity into a more distinguishable scoring range. It is defined as:

$$CLIPScore(v,t)=w\cdot \max (\cos (v,t),0),$$

(7)

InfoMetIC evaluates the information density and professional accuracy of the retrieved plant protection expertise. The InfoMetIC score is formulated by integrating the information-weighted matching of key agricultural entities:

$$InfoMetIC={\displaystyle \sum _{i=1}^N{\lambda }_i}\cdot \mathrm{Sim}(e_{gen}^{(i)},e_{ref}^{(i)}),$$

(8)

where $N$ denotes the total number of professional knowledge entities in the extracted set $E$, ${\lambda }_i$ represents the information gain of the $i$-th entity, and $\mathrm{Sim}(e_{gen}^{(i)},e_{ref}^{(i)})$ calculates the similarity for the $i$-th entity pair. This comprehensive evaluation framework ensures that the model is assessed both on its linguistic fluency and its utility as an interpretable decision-support tool in Xinjiang’s agricultural production.

3. Results

The image captioning task is implemented using a specialized encoder–decoder architecture. Unlike traditional models that rely solely on linguistic supervision, our training process integrates the visual features with linearized knowledge sequences derived from the MMKG. To jointly optimize the multi-scale visual perception and the knowledge-grounded decoding, we employ a multi-task loss function $L_{total}$ instead of a simple sparse categorical cross-entropy loss. The total objective function is defined as:

$$L_{total}={\lambda }_1{L}_{XE}+{\lambda }_2{L}_{Align},$$

(9)

where $L_{XE}$ represents the standard cross-entropy loss utilized for language modeling to ensure the fluency of the diagnostic reports, and $L_{Align}$ denotes the alignment loss specifically designed to supervise the VEG process. This dual-objective strategy ensures that visual features are accurately mapped into the structured semantic space of the MMKG, facilitating the transition from a simple visual recitation to a high-fidelity, expert-level diagnostic output.

3.1. Quantitative Analysis of the Results

Based on the comprehensive results presented in Table 2, a series of ablation studies were conducted to evaluate the individual contributions of the multi-scale perception and knowledge-integration modules. The experimental framework traversed a technical evolution from classical Convolutional-Neural-Network-Long-Short-Term-Memory (CNN+LSTM) architectures to advanced configurations like CrossViT+T5 and our ultimate MKG-CottonCapT6 framework.

Table 2. Quantitative performance comparison of MKG-CottonCapT6 against state-of-the-art models for cotton disease image captioning.

Model	BLEU1 (%)	BLEU2 (%)	BLEU3 (%)	BLEU4 (%)	METEOR (%)	CLIPScore (%)	InfoMetIC (%)
CNN+LSTM	65.6	50.5	42.1	36.6	34.1	-	-
CNN+LSTM+Attention	72.1	59.0	51.3	45.3	37.3	-	-
CNN+Transformer	68.5	54.0	46.5	40.5	36.5	-	-
CNN+T5	70.0	58.5	51.0	44.5	38.0	-	-
CrossViT+Transformer	72.0	60.5	56.9	51.1	41.5	-	-
CrossViT+T5	72.5	61.0	53.7	48.0	41.8	-	-
CottonCapT6	72.9	61.7	54.4	48.7	42.0	-	-
MKG-CottonCapT6	75.0	63.5	57.5	52.0	44.0	79.0	72.6

As illustrated in Figure 6, the data indicate a clear upward trajectory in performance as the model transitions from simple visual recitation to knowledge-augmented reasoning. While traditional benchmarks like CNN+LSTM exhibit limited diagnostic capability (e.g., a BLEU-4 of 36.6%), the adoption of the T5 decoder (CNN+T5) marks a significant turning point, raising the METEOR score to 38.0% and validating the strength of pretrained language representations in generating agricultural text.

Figure 6. Performance trends of different models across multiple evaluation metrics for cotton disease image captioning. The substantial gains in CLIPScore and InfoMetIC further underscore the effectiveness of the multi-task joint loss function ($L_{total}$) in enhancing visual-semantic alignment and professional knowledge density, ensuring expert-level diagnostic accuracy for cotton diseases.

The integration of the CrossViT visual encoder further stabilizes the model’s performance by capturing pathological features at dual granularities. Specifically, CrossViT+T5 achieves a BLEU-4 of 48.0%, confirming that multi-scale encoding is essential for identifying complex cotton disease symptoms. However, the most profound breakthrough occurs with the introduction of our MKG-CottonCapT6. By explicitly supervising the Visual Entity Grounding process through the alignment loss $L_{Align}$, our model reaches maximum performance across all dimensions, including a BLEU-4 of 51.2%, a CLIPScore of 79.0%, and an InfoMetIC score of 72.6%. These results emphasize that the fusion of structured MMKG expertise and dynamic semantic calibration is the key to transforming raw visual signals into expert-level plant protection advice.

To evaluate the computational feasibility of the proposed MKG-CottonCapT6, we conducted a comprehensive analysis of model parameters, FLOPs from thop, and execution time. As summarized in Table 3, our framework maintains a competitive computational profile despite the integration of external knowledge.

Table 3. Computational complexity and performance comparison between the baseline and the proposed framework.

Model	Params (M)	FLOPs (G)	Train Time (h)	Latency (ms)
CottonCapT6	267.6	24.5	2.5	41
MKG-CottonCapT6	268.8	26.2	3.2	54

3.2. Qualitative Analysis of the Results

To qualitatively evaluate the performance of the proposed framework, we selected representative samples from various cotton diseases and pests for comparative analysis, as presented in Table 4. The evaluation specifically focuses on the ability of the MKG-CottonCapT6 model to maintain semantic consistency and expert-level diagnostic accuracy compared to the CottonCapT6 baseline. The detailed analysis is as follows.

Table 4. The performance of MKG-CottonCapT6 on the cotton disease and pest picture test set, with one image–text pair selected as an example for each class. Number 1 represents the CottonCapT6-generated result; Number 2 represents the MKG-CottonCapT6-generated result.

Image	Results
	Cotton leaves reveal target-pattern lesions indicating target spot. Necrotic rings on the foliage confirm target spot, suggesting a need for treatment with carbendazim or mancozeb to control further spreading.
	Fine white fungal growth accumulates on young leaves, suggesting active powdery mildew colonization. Grayish-white powdery coating spreads across the lower leaf surfaces during the initial stage of infection, indicating a need for sulfur-based fungicides or triadimefon for effective intervention.
	The cotton leaf exhibits extensive purple-red hues, diagnosed as wilt disease. Initial yellowing of the leaf veins precedes total desiccation, necessitating seed disinfection and soil drenching with carbendazim to prevent large-scale outbreaks.
	The cotton plant is infested with tiny black insects, confirmed as thrips. These thrips-induced symptoms, including severe leaf curling and silver-gray scarring, typically peak during the seedling stage, requiring early application of imidacloprid to minimize significant yield losses.
	Waxy platelets are produced by cotton whiteflies feeding across leaf surfaces. Sticky honeydew secretions often lead to sooty mold development on contaminated leaves, necessitating the use of thiamethoxam or afidopyropen for effective pest management.
	Dense aphid populations blanket the upper cotton leaves These aphids (also known as melon aphids or grease insects) primarily cluster during the budding and bolling stages, necessitating the use of pymetrozine to prevent severe leaf distortion.
	The green worm feeding on the cotton bolls is likely a bollworm. To manage these bollworms, implementing integrated pest management such as the release of Trichogramma wasps or spraying emamectin benzoate is essential to protect the bolls from internal damage.

In Disease Case 1 (Target Spot), for images exhibiting target spot symptoms, the model correctly identifies “target-pattern lesions”. The generated description further provides professional control suggestions, such as the use of “carbendazim or mancozeb,” demonstrating that the integration of expert knowledge enhances the practical utility of captions beyond simple visual recognition.

In Disease Case 2 (Powdery Mildew), the model accurately captures the “fine white fungal growth” on young leaves. Crucially, it identifies the “initial stage of infection” and recommends “sulfur-based fungicides,” which perfectly aligns with the visual evidence of active colonization.

As observed in Disease Case 3 (Wilt Disease), for wilt disease samples with yellowing veins, the model not only captures the “purple-red hues” on the leaves but also accurately associates the “initial yellowing of the leaf veins” as a critical precursor. By suggesting “seed disinfection” and “soil drenching with carbendazim,” the model demonstrates a logical closed-loop from visual perception to preventive intervention.

For Pest Case 1 (Thrips), when processing thrips images, the model successfully detects “tiny black insects” and describes the resulting “silver-gray scarring”. By identifying the “seedling stage” as the peak hazard period, the model provides a timely diagnostic conclusion essential for effective field management.

Regarding Pest Case 2 (Whiteflies) and the “waxy platelets” left by feeding whiteflies, the model keenly identifies “sticky honeydew” secretions on the leaf surface and warns of potential “sooty mold” development. Its recommendation of “thiamethoxam” or “afidopyropen” reflects precise decision-making targeted at specific resistant pests.

In Pest Case 3 (Aphids), for dense pest clusters, the model generates the description “dense aphid populations blanket the upper cotton leaves”. It correctly incorporates aliases such as “melon aphids” or “grease insects” and identifies the “budding and bolling stages” as critical intervention windows, showcasing a high density of professional agricultural knowledge.

Pest Case 4 (Bollworm) demonstrates that the model accurately identifies the “green worm” feeding on cotton bolls as a “bollworm”. The output suggests sophisticated “integrated pest management” strategies, including the release of Trichogramma wasps or spraying “emamectin benzoate,” reflecting a deep understanding of complex biological control measures.

Furthermore, to explicitly evaluate the robustness of the framework under suboptimal real-world conditions, we analyzed its performance against previous baseline solutions when confronted with environmental noise and varying illumination, as visually demonstrated in Table 5. In practical agricultural settings, images frequently exhibit different brightness levels (e.g., strong sunlight contrasting with deep leaf shadows) or contain severe sensor noise. As shown in the augmented Bollworm cases (Table 5), previous solutions relying purely on pixel-level features often exhibit severe performance degradation. When exposed to high brightness and Gaussian noise (the second image case), the baseline model fails to extract the morphological details of the pest from the overexposed regions, hallucinating “white spots and holes” and misclassifying it as a leaf disease. Similarly, under darkened and noisy conditions (the third image case), it misinterprets deep shadows as necrotic lesions or a “rotten” boll, leading to completely erroneous disease descriptions.

Table 5. The qualitative comparison of generated diagnostic results under varying complex field conditions (e.g., illumination changes, Gaussian noise, and spatial transformations) for the Bollworm case. Number 1 represents the CottonCapT6-generated result; Number 2 represents the MKG-CottonCapT6-generated result.

Image	Results
	A green caterpillar is detected feeding on a cotton boll, which appears to be a bollworm. Upon identifying a green bollworm feeding on the cotton boll, integrated pest management, such as releasing Trichogramma wasps or applying emamectin benzoate, is recommended during the peak occurrence period
	A green leaf with white spots and holes. A worm is detected feeding on the boll; it is advised to closely monitor the budding and bolling stages.
	The cotton boll appears rotten or infected by black necrotic spots. An insect is detected on the cotton boll, indicating a potential pest threat.

In contrast, MKG-CottonCapT6 demonstrates remarkable resilience. By anchoring the degraded visual features to the MMKG, the retrieved semantic priors act as robust logical constraints. Even when intra-image brightness varies significantly, or noise obscures the fine details of the pest, our model relies on multimodal knowledge inference rather than pure visual cues to avoid severe hallucinations. Consequently, instead of erroneously diagnosing the crop with a fungal or necrotic disease, it successfully confirms the presence of a pest. This demonstrates that the framework maintains high diagnostic safety and semantic consistency, providing reliable agricultural guidance even when raw visual inputs are severely compromised.

Overall, the qualitative results in Table 5 profoundly reveal the superiority of MKG-CottonCapT6. Compared to baseline models relying solely on visual features, MKG-CottonCapT6 achieves significant semantic information augmentation through the injection of an MMKG. The model not only sees the morphology of lesions and pests but also “understands” the underlying pathological stages, crop growth phases, and targeted control decisions. The generated descriptions present a tiered structure of “visual feature representation–diagnostic conclusion confirmation–expert control recommendation,” thoroughly bridging the gap between raw image data and real-world plant protection practices. This high level of semantic alignment and professional knowledge density proves the framework’s immense potential for providing expert-level plant protection advice in complex field environments.

4. Discussion

Although the MKG-CottonCapT6 framework demonstrates superior performance in professional cotton disease and pest image captioning, a critical analysis of its current implementation reveals several deep-seated limitations that necessitate further exploration and systemic refinement.

Critical Analysis of Feature Ambiguity and Alignment Strength: To systematically evaluate the model’s feature alignment capabilities, we analyzed its classification performance across seven categories. As illustrated in the confusion matrix (Figure 7), a primary constraint identified through the confusion matrix analysis is the model’s occasional inability to distinguish between certain disease symptoms and pest-induced damage. For instance, the matrix reveals that wilt disease is occasionally misclassified as bollworm damage (4 instances) or target spot (2 instances). This limitation suggests that the current VEG mechanism, while effective overall, still lacks the fine-grained mapping intensity required to separate structurally similar visual artifacts—such as leaf wilting versus pest chewing damage—under complex field lighting and background interference.

Figure 7. Confusion matrix illustrating the classification performance of the proposed model across seven cotton pest and disease categories. The main diagonal represents correct predictions, while off-diagonal elements indicate misclassifications.

To improve feature discrimination for these frequently confused pairs, future work should first focus on embedding more detailed visual prototypes directly into the MMKG. Furthermore, research should transition beyond simple dataset expansion toward Metric Learning techniques, utilizing contrastive or center loss functions to explicitly compress intra-class feature space while maximizing inter-class distances. Finally, leveraging Generative Diffusion Models to synthesize “hard negative” samples—images that represent ambiguous or intermediate disease states—could fundamentally enhance the model’s discriminatory logic.

Knowledge Density and the Shift Toward Predictive Actionability: While the integration of the MMKG allows the model to output professional recommendations like “carbendazim” or “thiamethoxam,” these outputs remain somewhat template-based and lack coupling with dynamic environmental variables such as humidity or wind speed, which dictate actual spraying efficacy.

To enhance “clinical” utility, the framework must evolve from a descriptive tool into a predictive reasoning engine. Integrating real-time local meteorological data and geospatial information would enable the generation of time-sensitive, actionable plant protection prescriptions. Implementing a Reinforcement Learning (RL) framework, where rewards are based on “diagnostic correctness” as judged by expert logic rather than mere n-gram overlap (e.g., BLEU scores), is essential for increasing practical value.

Systemic Handling of Co-occurring Pathologies: The current reliance on single-condition samples restricts the model’s ability to diagnose complex “pest-disease complexes” that are frequently encountered in Xinjiang cotton fields.

The development of multi-label multimodal datasets is a critical next step. Utilizing techniques like StyleGAN to generate realistic images of leaves suffering from simultaneous infestations (e.g., aphids and fungal infections) will allow the model to learn inter-pathological relationships and provide more holistic diagnostic reports.

Efficiency of the Training Paradigm and Edge-Device Transferability: The current two-stage training strategy, while effective for semantic alignment, introduces significant computational overhead and relies on a massive knowledge graph, which hinders real-time deployment on low-cost mobile agricultural devices.

Investigating a unified, End-to-End foundation model capable of performing both diagnostic classification and descriptive tasks simultaneously would drastically reduce parameter redundancy. To enable local, real-time management in vast cotton fields, applying knowledge distillation to transfer complex reasoning into lightweight architectures is the next essential step toward bridging the gap between high-level research and on-site agricultural practice.

5. Conclusions

This research introduces MKG-CottonCapT6, a sophisticated framework that leverages multimodal knowledge graphs to revolutionize cotton disease and pest identification in Xinjiang. By synthesizing the multi-scale perceptual capabilities of CrossViT, the generative power of the T5 decoder, and the structural intelligence of a domain-specific MMKG, our approach establishes a superior equilibrium between visual recognition and expert-level diagnostic inference. Empirical evidence confirms that the latent semantic augmentation provided by the MMKG—implemented through entity linking and triplet linearization—enables the architecture to transcend superficial visual tagging and engage in deeper pathological interpretation. The MKG-CottonCapT6 model outperformed all comparative baselines, securing peak scores in BLEU-4 (51.2%), CLIPScore (79.0%), and InfoMetIC (72.6%), thereby validating the indispensable role of structured botanical logic in enhancing vision-language systems.

Apart from its algorithmic contributions, this study establishes a transformative decision-making ecosystem for smart farming. The primary value of this system is its capacity to empower Xinjiang’s agricultural practitioners by automating the production of formalized, transparent diagnostic dossiers that bridge raw imagery with specialist-grade remedial guidance. By distilling professional intelligence from seminal texts like Xinjiang Cotton Pests and Diseases and Their Control, the framework standardizes pest-logging protocols and mitigates the inherent volatility of human-led field evaluations.

Nevertheless, the evaluation also highlights several fundamental challenges. A critical appraisal indicates that the current VEG mechanism lacks sufficient mapping intensity for fine-grained feature differentiation, particularly when distinguishing between visually overlapping symptoms of wilt-type diseases and early target spot. Furthermore, the computational demands of the multi-stage training paradigm and the dataset’s current focus on isolated pathologies may limit the framework’s agility when encountering the complex “pest-disease complexes” typical of real-world cotton farming.

Subsequent research will concentrate on evolving from descriptive reporting to proactive diagnostic reasoning. We intend to incorporate dynamic environmental variables and utilize RL algorithms—optimized specifically for clinical accuracy—to refine the model’s output. To facilitate seamless edge deployment throughout Xinjiang’s cotton-growing regions, we will explore end-to-end foundation architectures and knowledge distillation techniques to condense complex reasoning into lightweight, mobile-friendly formats. Ultimately, expanding the MMKG to support multi-label training and a wider array of high-value crops will ensure that this data-driven paradigm contributes significantly to global sustainable agricultural practices.

In summary, the MKG-CottonCapT6 framework represents a pivotal shift toward knowledge-augmented intelligence in agricultural computer vision. By successfully harmonizing deep learning architectures with structured botanical expertise, this research provides a robust technological foundation for transitioning from traditional visual monitoring to a data-driven, expert-level diagnostic ecosystem. As these models evolve to become more computationally efficient and contextually aware, they will play a decisive role in ensuring the sustainability and precision of cotton production in Xinjiang and beyond.