Skip to Content
ElectronicsElectronics
  • Article
  • Open Access

2 February 2026

19 Pages

Multimodal Sample Correction Method Based on Large-Model Instruction Enhancement and Knowledge Guidance

,
,
,
and
1
Information and Telecommunication Center, State Grid Corporation of China, Beijing 100031, China
2
China Electric Power Research Institute, Beijing 100192, China
3
State Grid Information and Telecommunication Group Co., Ltd., Beijing 102211, China
4
Information and Communication Branch, State Grid Zhejiang Electric Power Company, Hangzhou 310007, China

Abstract

With the continuous improvement of power system intelligence, multimodal data generated during distribution network maintenance have grown exponentially. However, existing power multimodal datasets commonly suffer from issues such as low sample quality, frequent factual errors, and inconsistent instruction expressions caused by regional differences.Traditional sample correction methods mainly rely on manual screening or single-feature matching, which suffer from low efficiency and limited adaptability. This paper proposes a multimodal sample correction framework based on large-model instruction enhancement and knowledge guidance, focusing on two critical modalities: temporal data and text documentation. Multimodal sample correction refers to the task of identifying and rectifying errors, inconsistencies, or quality issues in datasets containing multiple data types (temporal sequences and text), with the objective of producing corrected samples that maintain factual accuracy, temporal consistency, and domain-specific compliance. Our proposed framework employs a three-stage processing approach: first, temporal Bidirectional Encoder Representations from Transformers (BERT) models and text BERT models are used to extract and fuse device temporal features and text features, respectively; second, a knowledge-injected assessment mechanism integrated with power knowledge graphs and DeepSeek’s long-chain-of-thought (CoT) capabilities is designed to achieve precise assessment of sample credibility; third, beam search algorithms are employed to generate high-quality corrected text, significantly improving the quality and reliability of multimodal samples in power professional scenarios. Experimental results demonstrate that our method significantly outperforms baseline models across all evaluation metrics (BLEU: 0.361, ROUGE: 0.521, METEOR: 0.443, F1-Score: 0.796), achieving improvements ranging from 21.1% to 73.0% over state-of-the-art methods: specifically, a 21.1% improvement over GECToR in BLEU, 26.5% over GECToR in ROUGE, 30.3% over Deep Edit in METEOR, and 11.8% over Deep Edit in F1-Score, with a reduction of approximately 35% in hallucination rates compared to existing approaches. These improvements provide important technical support for intelligent operation and maintenance of power systems, with implications for improving data quality management, enhancing model reliability in safety-critical applications, and enabling scalable knowledge-guided correction frameworks transferable to other industrial domains requiring high data integrity.

1. Introduction

With the deepening advancement of new power system construction, distribution network maintenance work is undergoing a fundamental transformation from traditional manual inspection to intelligent and automated maintenance [1,2,3]. Modern power systems continuously generate massive volumes of multimodal data through high-resolution sensors, unmanned aerial vehicle (UAV) inspections, mobile maintenance robots, and other equipment, including text documentation, time series signals, and other forms. The advancement of deep learning techniques has significantly enhanced data processing capabilities across various domains. In vision and language applications, recent developments include multimodal fusion networks [4] for cross-modal understanding, dual-stream architectures [5] for visual–linguistic alignment, scene understanding frameworks [6], and text-guided person re-identification methods [7]. Additionally, learning-based approaches have shown promising results in detection tasks [8], knowledge-driven reasoning [9], topological structure learning [10], and intelligent system optimization [11]. However, these general-purpose methods often lack domain-specific adaptations necessary for power system applications. These data provide critical information support for equipment condition analysis, fault detection, and defect identification. However, the explosive growth in data volume also brings unprecedented challenges: data quality issues.
In this context, we formally define the multimodal sample correction task as follows: Given a multimodal input sample S = { S t , S t e x t } consisting of temporal sequences S t and textual documentation S t e x t , the objective is to produce a corrected output S ′ = { S t ′ , S t e x t ′ } that (1) maintains factual accuracy with respect to domain knowledge constraints K , (2) ensures temporal consistency between time-series patterns and textual descriptions, (3) adheres to professional standards and regulatory requirements specific to power systems, and (4) eliminates hallucinations, semantic inconsistencies, and quality defects present in the original sample. The expected output includes corrected text with minimal edit distance while preserving semantic integrity, accompanied by credibility scores indicating the confidence level of corrections.
Distribution network maintenance data exhibit obvious regional characteristics with varying management requirements and instruction language expressions across different cities and regions. Traditional large models [12,13] cannot adequately meet regional maintenance needs. During data processing procedures such as OCR recognition and sample augmentation, substantial inaccurate or incomplete data are generated, leading to decreased model generalization ability. Real-time fluctuations in power system operating states further cause static data to become disconnected from actual states, exacerbating data timeliness issues.
Existing multimodal sample correction methods mainly fall into three categories: traditional rule-based correction methods, statistical learning-based correction methods, and deep learning-based correction methods [14,15]. Traditional methods offer fast computation but perform poorly when handling complex backgrounds, lighting variations, and geometric structures typical of power equipment. Statistical learning methods improve efficiency and robustness to some extent but often fail to distinguish deeper semantic differences. Deep learning models provide stronger representational capacity but require substantial computational resources and labeled datasets and still lack adequate semantic understanding of power-specific content. More critically, existing methods often fail to effectively distinguish between actual equipment state changes and superficial imaging differences, limiting their deployment in real-world power scenarios.
To address these limitations, this paper proposes a multimodal sample correction framework based on large-model instruction enhancement and knowledge guidance. The framework deeply integrates perceptual and semantic features, adopting a three-stage processing pipeline: “multimodal feature extraction and fusion → knowledge-injected assessment → sample correction,” as illustrated in Figure 1. First, temporal BERT models and text BERT models are used to extract device temporal features and text features, respectively, followed by cross-modal attention mechanisms to achieve effective fusion of features from different modalities. Second, a knowledge graph-enhanced reasoning mechanism integrated with DeepSeek’s long-chain-of-thought capabilities is designed to achieve precise assessment of sample credibility. Third, beam search algorithms are employed to generate high-quality corrected text, ensuring accuracy and consistency of correction results.
Figure 1. Overview of the three-stage multimodal sample correction methodology. The framework consists of multimodal feature extraction and fusion (temporal and text feature extraction with cross-modal attention), knowledge-injected assessment (knowledge graph construction, chain-of-thought reasoning, and credibility assessment), and sample correction (strategy selection and beam search repair).
The main contributions of this paper include the following: (1) designing a comprehensive multimodal feature extraction framework that integrates temporal attention mechanisms and multi-head self-attention for text processing, enabling effective handling of complex power system data, achieving F1-Score improvements of 5.1 percentage points over single-modality approaches (from 0.745 to 0.796); (2) proposing a knowledge graph-enhanced reasoning system that combines entity and relation embeddings with attention-weighted knowledge retrieval, achieving precise sample credibility assessment through weighted aggregation of multiple scoring components, with knowledge graph integration contributing 3.5 percentage points improvement in F1-Score and reducing factual errors by approximately 35%; (3) developing an adaptive correction strategy selection mechanism with gating networks and beam search algorithms that incorporate knowledge graph constraints, providing targeted solutions for different types of data quality issues in power professional scenarios, demonstrating 21.1–73.0% improvements across different evaluation metrics (BLEU, ROUGE, METEOR, F1-Score) compared to state-of-the-art baseline methods.

3. Methodology

3.1. Overall Framework

The proposed multimodal sample correction framework based on large-model instruction enhancement and knowledge guidance is illustrated in Figure 2. The framework adopts a three-stage processing pipeline: multimodal feature extraction and fusion, knowledge-injected assessment, and sample correction. This architectural design addresses the key challenges identified in existing methods: inadequate cross-modal fusion, lack of domain-specific knowledge integration, and insufficient reasoning verification mechanisms. The framework processes power system maintenance data containing both temporal sensor readings and textual documentation, producing corrected outputs that maintain factual accuracy, temporal consistency, and regulatory compliance.
Figure 2. Detailed framework architecture of the multimodal sample correction system. The system integrates temporal BERT and text BERT models for feature extraction, employs knowledge graphs and chain-of-thought reasoning for assessment, and utilizes beam search algorithms for sample correction, demonstrating the complete workflow from input processing to corrected output generation.
As shown in Figure 2, the framework operates through a systematic workflow that processes multimodal inputs and generates quality-assured corrected outputs. We now elaborate on each major component:
Stage 1: Multimodal Feature Extraction and Fusion. The input layer receives two complementary data streams: (i) temporal sequences X t = { x 1 , x 2 , … , x T } from power equipment sensors capturing operational parameters such as voltage, current, temperature, and load patterns over time; (ii) textual documentation X t e x t = { w 1 , w 2 , … , w N } comprising maintenance reports, operation logs, and equipment specifications. These heterogeneous modalities are processed through specialized encoder architectures. The temporal BERT encoder applies multi-head self-attention mechanisms to capture temporal dependencies and equipment-specific operational patterns, incorporating positional encodings that preserve chronological information. Simultaneously, the text BERT encoder processes linguistic content through bidirectional attention, extracting semantic features from technical terminology, operational procedures, and maintenance instructions. The extracted representations H t ∈ R T × d and H t e x t ∈ R N × d are then integrated through a cross-modal attention module, which computes attention weights measuring the relevance between temporal patterns and textual descriptions. This fusion mechanism generates a unified multimodal representation F f u s e d that encapsulates both temporal dynamics and semantic content, enabling downstream modules to reason about temporal–text consistency.
Stage 2: Knowledge-Injected Assessment. The fused multimodal features enter the assessment module, which evaluates sample credibility through knowledge-guided reasoning. This stage leverages a power system knowledge graph KG containing 15,432 equipment entities, 87 relation types, and 38,765 validated triplets encoding domain expertise including equipment specifications, operational constraints, regulatory requirements, and historical maintenance patterns. The knowledge graph provides structured domain knowledge through entity embeddings e i and relation embeddings r j learned via TransE, augmented with temporal decay factors to prioritize recently validated information. The DeepSeek-based long-chain-of-thought reasoning module generates intermediate reasoning steps, with each step constrained by knowledge graph triplets to prevent hallucinations and ensure domain compliance. Unlike standard CoT approaches using fixed-length reasoning chains, our method dynamically adjusts reasoning depth based on sample complexity C ( S ) , allocating more reasoning steps to samples with detected inconsistencies. A consistency verification mechanism validates each reasoning step against prior steps and knowledge constraints through scoring function Consistency ( r t | r < t , KG ) , triggering self-correction via beam search when consistency falls below threshold τ = 0.7 . The reasoning outputs feed into a hierarchical credibility assessment module that computes four complementary scores: (1) temporal consistency score s 1 measuring alignment between time series patterns and textual descriptions, (2) knowledge conformity score s 2 evaluating adherence to domain constraints, (3) semantic coherence score s 3 assessing logical consistency, and (4) regional compliance score s 4 verifying conformity to local regulations. These scores are aggregated through learned weights w i optimized during training, producing a final credibility score Credibility ∈ [ 0 , 1 ] indicating sample quality.
Stage 3: Sample Correction. Based on the credibility assessment and identified error patterns, an adaptive strategy selection mechanism determines the appropriate correction approach. A gating network processes the enhanced features F e n h a n c e d , credibility score, and context features C i (including error type distribution, sample source, and regional requirements) to select among three strategies: (i) minor editing for samples with localized errors, (ii) major restructuring for samples with structural inconsistencies, (iii) complete regeneration for samples with pervasive quality issues. The selected strategy guides a knowledge-constrained beam search decoder that generates corrected text. At each decoding step t, the decoder maintains B = 5 candidate sequences, computing generation probability P ( y t | y < t , x ) combined with knowledge graph conformity score KG _ Score ( y 1 : t ) weighted by λ = 0.3 . A knowledge-constrained pruning mechanism eliminates candidates violating hard constraints (e.g., incompatible equipment-operation pairs) at each step, reducing search space by 40% while preserving quality. The beam search incorporates length penalty α = 0.6 to balance output length and quality. The final output consists of corrected text y ∗ = arg max y Score ( y 1 : | y | ) accompanied by credibility scores and identified error types, providing transparency for downstream quality control.
This three-stage architecture enables end-to-end processing of multimodal power system data, addressing the limitations of existing methods through explicit knowledge integration, dynamic reasoning depth control, and adaptive correction strategies. The modular design facilitates component-wise optimization and allows flexible adaptation to different power grid operational scenarios and regional regulatory requirements.

3.2. Multimodal Feature Extraction and Fusion

3.2.1. Temporal Feature Extraction

For temporal data of power equipment, we employ temporal BERT models for feature extraction. Consider a temporal data sequence X t = { x 1 , x 2 , … , x T } , where x i ∈ R d represents the feature vector at the ith time step, T denotes the sequence length, and d is the feature dimension (set to 768 in our implementation).
Following the standard Transformer attention mechanism [23], the temporal attention mechanism computes attention weights for each time step:
A t , i , j = exp ( Q i K j T / d k ) ∑ k = 1 T exp ( Q i K k T / d k )
where Q i and K j are query and key vectors at positions i and j, respectively, and d k is the key dimension (set to 64 in our experiments). The temporal feature representation is computed as:
H t ( i ) = ∑ j = 1 T A t , i , j V j + P i
where V j is the value vector at position j, and P i is the positional encoding for the ith time step, computed using sinusoidal functions as in [23]. Novel contribution: Unlike standard temporal transformers, we introduce device-specific positional encodings that incorporate equipment-type information through learnable embeddings E d e v i c e ∈ R d , modifying Equation (2) to H t ( i ) = ∑ j = 1 T A t , i , j V j + P i + E d e v i c e , enabling better capture of device-specific temporal patterns as shown in Figure 2 (Temporal Feature Extraction module).

3.2.2. Text Feature Extraction

For power maintenance-related text data, we employ BERT models specifically pre-trained for the power domain. Consider an input text sequence X t e x t = { w 1 , w 2 , … , w N } , where w i represents the ith token, and N is the text sequence length (maximum 512 tokens in our implementation).
Following the BERT architecture [13], the multi-head self-attention mechanism computes attention scores as:
Attention ( Q , K , V ) = Softmax Q K T d k V
where Q , K , V ∈ R N × d k are query, key, and value matrices, respectively. The multi-head attention combines multiple attention heads:
MultiHead ( Q , K , V ) = Concat ( head 1 , … , head h ) W O
where head i = Attention ( Q W i Q , K W i K , V W i V ) , h = 12 is the number of attention heads, W i Q , W i K , W i V ∈ R d × d k are projection matrices, and W O ∈ R d × d is the output projection matrix, and the final text representation incorporates layer normalization and residual connections:
H t e x t ( l + 1 ) = LayerNorm ( H t e x t ( l ) + MultiHead ( H t e x t ( l ) ) )
where l indexes the transformer layer (total 12 layers), and LayerNorm denotes layer normalization [23]. Novel contribution: We extend the text BERT with domain-specific token type embeddings that distinguish between equipment names, operation commands, and descriptive text, enhancing the model’s ability to capture power system documentation structure as illustrated in Figure 2 (Text Feature Extraction module).

3.2.3. Multimodal Feature Fusion

To effectively fuse features from different modalities, we designed a cross-modal attention-based fusion mechanism. This mechanism can adaptively learn correlation relationships between different modalities, generating unified multimodal representations.

3.2.4. Cross-Modal Attention Mechanism

Given feature representations from two modalities H t and H t e x t , where H t ∈ R T × d and H t e x t ∈ R N × d , we first map them to a unified feature space through linear projections:
H ˜ t = H t W t + b t
H ˜ t e x t = H t e x t W t e x t + b t e x t
where W t , W t e x t ∈ R d × d are learnable projection weight matrices, and b t , b t e x t ∈ R d are bias vectors. Then, following standard practice [23], cross-modal attention weights are computed:
A t , t e x t = Softmax H ˜ t H ˜ t e x t T d k
where A t , t e x t ∈ R T × N represents the attention weights between temporal and text features. This component is illustrated in the Cross-Modal Attention module of Figure 2.

3.2.5. Feature Fusion

Based on the computed attention weights, feature fusion is performed:
F f u s e d = α 1 H ˜ t + α 2 A t , t e x t H ˜ t e x t
where α 1 and α 2 are learnable fusion weight parameters, initialized to 0.5 and optimized during training. Novel contribution: We introduce adaptive fusion weights computed via a gating network G ( H ˜ t , H ˜ t e x t ) = sigmoid ( W g [ H ˜ t ; H ˜ t e x t ] + b g ) , where W g ∈ R 2 × 2 d and b g ∈ R 2 , allowing the model to dynamically balance modality contributions based on input quality, as depicted in the Feature Fusion module of Figure 2.

3.3. Knowledge-Injected Assessment

3.3.1. Knowledge Graph Construction

Based on professional knowledge in the power maintenance domain, we construct a knowledge graph KG = { ( h , r , t ) } containing equipment entities, attribute relationships, and operational rules, where h , t ∈ E are head and tail entities from the entity set E (15,432 entities in total), and r ∈ R is a relation from the relation set R (87 relation types). Following TransE [32], entity embeddings are learned through:
e i = tanh ( W e · onehot ( i ) + b e )
where e i ∈ R d k g is the embedding for entity i, W e ∈ R d k g × | E | is the entity embedding matrix, b e ∈ R d k g is the bias vector, and d k g = 256 is the knowledge graph embedding dimension. Relation embeddings capture semantic relationships:
r j = tanh ( W r · onehot ( j ) + b r )
where r j ∈ R d k g is the embedding for relation j, and W r ∈ R d k g × | R | is the relation embedding matrix. The knowledge graph scoring function measures triplet validity:
Score ( h , r , t ) = ∥ e h + r j − e t ∥ 2 2
where lower scores indicate higher validity. Novel contribution: We extend the standard TransE model with temporal decay factors λ t = exp ( − γ · Δ t ) , where Δ t is the time elapsed since the triplet was last verified, and γ = 0.01 is a decay rate, allowing the knowledge graph to prioritize recently validated information. This enhancement addresses the dynamic nature of power system operations as shown in the Knowledge Graph module of Figure 2.

3.3.2. Long-Chain-of-Thought Reasoning Based on DeepSeek

Unlike standard chain-of-thought (CoT) reasoning that relies on fixed-length reasoning chains and static prompt templates, our DeepSeek-based long-CoT mechanism introduces three key innovations: (1) Dynamic reasoning length control: the reasoning chain length L is adaptively determined based on sample complexity C ( S ) through a learned gating function L = min ( L m a x , ⌈ β · C ( S ) ⌉ ) , where β is a scaling parameter, and C ( S ) quantifies the number of conflicting knowledge triplets, temporal anomalies, and semantic inconsistencies in the input sample; (2) Logical consistency maintenance: at each reasoning step t, a consistency checker validates the generated reasoning token r t against previously generated tokens r < t and knowledge graph constraints through a scoring function Consistency ( r t | r < t , KG ) = sigmoid ( W c [ h t ; h < t ; k r e l ] + b c ) , where k r e l represents relevant knowledge embeddings, and h t is the hidden state at step t; (3) Error propagation mitigation: a self-correction mechanism is triggered when consistency scores fall below threshold τ = 0.7 , allowing the model to backtrack and regenerate alternative reasoning paths using beam search with width B = 5 .
The reasoning process incorporates attention-weighted knowledge retrieval. Given query features q, relevant knowledge is retrieved through:
α i = exp ( q T k i ) ∑ j = 1 | KG | exp ( q T k j )
To ensure knowledge nodes are effectively constrained within the reasoning chain, we implement a structured integration mechanism. Each knowledge triplet ( h , r , t ) ∈ KG is represented as k i = [ e h ; r j ; e t ] , where embeddings are learned through Equations (10) and (11). During reasoning step t, knowledge constraints are enforced through a gating mechanism:
g k ( t ) = sigmoid ( W g k [ F f u s e d ; r t ; k i ] + b g k )
which determines the influence of each knowledge node on the current reasoning step. The knowledge-constrained reasoning state is then computed as:
h t k g = h t ⊙ ( 1 − g k ( t ) ) + ∑ i = 1 | KG | α i v i ⊙ g k ( t )
where ⊙ denotes element-wise multiplication. This mechanism ensures that knowledge graph constraints are directly integrated into each reasoning step, preventing the generation of reasoning chains that violate domain knowledge.
The knowledge-enhanced representation is computed as:
F e n h a n c e d = F f u s e d + ∑ i = 1 | KG | α i v i
where k i and v i are key and value representations of knowledge triplets, and the summation aggregates multiple relevant knowledge nodes weighted by their attention scores α i , enabling soft constraint enforcement that balances data-driven features with structured knowledge.

3.3.3. Sample Credibility Assessment

Credibility assessment combines multiple scoring components through weighted aggregation:
Credibility = σ ∑ i = 1 M w i · s i ( F e n h a n c e d )
where s i ( · ) are different scoring functions, including (1) temporal consistency score s 1 measuring alignment between temporal patterns and text descriptions, (2) knowledge conformity score s 2 evaluating adherence to knowledge graph constraints, (3) semantic coherence score s 3 assessing internal logical consistency, and (4) regional compliance score s 4 checking conformity to local regulations ( M = 4 total scoring functions), where w i are learned weights, optimized via gradient descent with constraints ∑ i = 1 M w i = 1 and w i ≥ 0 , and σ is the sigmoid function. Novel contribution: We introduce a hierarchical credibility assessment that first evaluates individual modalities separately then combines them with cross-modal consistency checks, as illustrated in the Credibility Assessment module of Figure 2, enabling fine-grained identification of quality issues across different data aspects.

3.4. Sample Correction

3.4.1. Correction Strategy Selection

The correction strategy is determined by a gating mechanism that considers multiple factors:
g i = Softmax ( W g [ F e n h a n c e d ; Credibility ; C i ] + b g )
where C i represents context features for strategy i, including error type distribution, sample source, and regional requirements, W g ∈ R K × ( d + 1 + d c ) , where K is the number of correction strategies (we use K = 3 : minor editing, major restructuring, and complete regeneration), d c is the context feature dimension (set to 128), and the concatenation [ F e n h a n c e d ; Credibility ; C i ] produces a combined feature vector. The final strategy selection is:
Strategy = arg max i g i
Novel contribution: Unlike fixed correction strategies, our adaptive selection mechanism dynamically chooses between different correction granularities based on error severity and type, as shown in the Strategy Selection module of Figure 2.

3.4.2. Beam Search Repair

The beam search process maintains top-B candidate sequences, where B = 5 in our implementation. At each step t, the probability of generating token y t is:
P ( y t | y < t , x ) = Softmax ( W o h t + b o )
where W o ∈ R | V | × d is the output projection matrix, | V | is the vocabulary size (50,000 tokens), h t ∈ R d is the hidden state at step t, and b o ∈ R | V | is the output bias. Following standard sequence-to-sequence models [23], the beam search score combines generation probability with knowledge constraints:
Score ( y 1 : t ) = 1 t ∑ i = 1 t log P ( y i | y < i , x ) + λ · KG _ Score ( y 1 : t )
where KG _ Score ( · ) measures consistency with knowledge graph constraints, and λ balances the two components, set to 0.3 based on validation set performance. Novel contribution: We enhance standard beam search with knowledge-constrained pruning that eliminates candidate sequences violating hard constraints from the knowledge graph (e.g., incompatible equipment–operation pairs) at each decoding step, reducing the effective search space by 40% while maintaining output quality. This is implemented in the Beam Search Repair module shown in Figure 2.

4. Experiments

4.1. Dataset Construction

To comprehensively evaluate the effectiveness of the proposed method, we constructed a specialized multimodal sample correction dataset for the power domain. The dataset primarily focused on core business scenarios such as distribution network dispatching maintenance applications and dispatching operation tickets, covering the most critical document types in power system operation and maintenance. The construction of the dataset fully considered the professional characteristics and regional differences of the power industry, ensuring the practicality and reliability of experimental results.
The dataset consisted of two main components: First, the safety monitoring internal Q&A data, constructed based on the power safety examination question bank, containing 7946 professional question–answer pairs covering core knowledge areas such as work ticket management, safety operation procedures, and equipment maintenance standards. These data originated from actual safety training and assessment scenarios, demonstrating strong practicality. Second, the distribution network dispatching Q&A data, constructed based on actual dispatching business in Xuancheng and Ma’anshan regions of Anhui Province, including distribution network dispatching maintenance application Q&A pairs and 1498 annotated samples for generating dispatching operation tickets from maintenance applications.
To enhance the model’s professional knowledge understanding capability, we also constructed an external knowledge base using 500 complete-version-annotated samples as knowledge support. These samples were carefully annotated by professional personnel, covering various typical scenarios and operational procedures of distribution network dispatching, providing rich professional knowledge background for the Q&A system and ensuring the accuracy and professionalism of answers.

4.2. Experimental Setup and Evaluation Metrics

This study adopted a multi-dimensional evaluation system to comprehensively measure model performance. Considering the characteristics of text correction tasks, we selected widely recognized evaluation metrics in the industry, including BLEU, ROUGE, METEOR, and F1-Score, which can evaluate the quality and authenticity of text correction from different perspectives. The BLEU score mainly evaluates the n-gram overlap between generated text and reference text, the ROUGE score focuses on evaluating text summary quality and content coverage, the METEOR score considers text similarity of synonyms and stems, while the F1-Score comprehensively considers the harmonic mean of precision and recall.
Implementation Details and Reproducibility: All experiments were conducted on a computing cluster equipped with four NVIDIA A100 GPUs (80 GB memory each), Intel Xeon Platinum 8358 CPUs (2.60 GHz, 64 cores), and 512 GB of RAM. We used PyTorch 2.0.1 with CUDA 11.8 for the implementation. To ensure reproducibility, we set random seeds across all libraries: Python random seed = 42, NumPy random seed = 42, PyTorch manual seed = 42, and CUDA manual seed = 42. The deterministic mode was enabled in PyTorch (torch.backends.cudnn.deterministic = True, torch.backends.cudnn.benchmark = False). Our temporal BERT model was initialized with 12 transformer layers, a hidden size of 768, and 12 attention heads. The text BERT model employed the bert-base-chinese pre-trained weights. Training used AdamW optimizer with a learning rate of 2 × 10−5, batch size of 16, and gradient accumulation steps of 4. The knowledge graph contained 15,432 entities and 38,765 relation triplets. All models were trained for 50 epochs with early stopping (patience = 5). The beam search width was set to B = 5 with length penalty α = 0.6 .
The experiment adopted a rigorous controlled design, selecting multiple representative baseline models for comparison, including T5 [39], BART [40], GPT-3 [12], Deep Edit [15], and GECToR [14] public models. These models represent the mainstream technical routes in the current text correction field, providing comprehensive performance benchmarks for this study. For fair comparison, all baseline models were fine-tuned on our dataset using their recommended hyperparameters with consistent early stopping criteria. The experiment conducted 10 independent validation experiments on datasets such as dispatching maintenance applications and dispatching operation tickets, ensuring statistical significance and reproducibility of results. Statistical significance was assessed using paired t-tests with a p < 0.05 threshold, and we report mean ± standard deviation across all runs with 95% confidence intervals computed using bootstrap resampling (10,000 iterations).

4.3. Experimental Results and Analysis

Through large-scale comparative experiments conducted on dispatching maintenance application and dispatching operation ticket datasets, our method achieved significant performance improvements on all evaluation metrics. As shown in Table 1, our method significantly outperformed other comparison models on the four key metrics of BLEU, ROUGE, METEOR, and F1-Score.
Table 1. Comparison of different methods on a multimodal sample correction task. Results shown as mean ± std across 10 runs. * indicates statistical significance (p < 0.05) compared to the best baseline (GECToR). The 95% confidence intervals for our method were as follows: BLEU [0.352, 0.369], ROUGE [0.509, 0.532], METEOR [0.436, 0.451], F1 [0.788, 0.804].
From the experimental results, we can see that compared to the best baseline model GECToR, our method improved by 6.3 percentage points on the BLEU score, 10.9 percentage points on the ROUGE score, 2.8 percentage points on the METEOR score, and 1.2 percentage points on the F1-Score. These improvements fully demonstrate the significant advantages of our method in text correction quality and authenticity.
Particularly noteworthy is that our method achieves the most significant improvement on the ROUGE score, indicating that our method has obvious advantages in maintaining text content integrity and coverage. This is mainly attributed to the multimodal feature fusion mechanism and knowledge graph-guided chain-of-thought reasoning, enabling the model to better understand the semantic structure and professional content of text, thereby generating more accurate and complete correction results.

4.4. Ablation Studies and Component Analysis

To deeply understand the contribution of each component to overall performance, we conducted comprehensive ablation experiments by systematically removing or modifying individual components of our framework. These experiments provide crucial insights into the relative importance of different architectural choices and help validate our design decisions.
The ablation study results are presented in Table 2, which shows the performance impact of removing various components from our complete framework. The experimental results demonstrate that multimodal feature fusion is a key factor for performance improvement, with each modality contributing significantly to the overall system performance.
Table 2. Ablation study results showing the contribution of different components. Results shown as mean ± std across 10 runs. Δ F1 indicates absolute percentage point difference from complete framework. All differences are statistically significant ( p < 0.001 ).
When using only text features, the model’s F1-Score was 0.745, demonstrating the baseline performance of single-modal approaches. Using only temporal features yielded an F1-Score of 0.728, showing that temporal information alone provides valuable but limited insights. The combination of text and temporal features achieved an F1-Score of 0.782, with an increase of 3.7 percentage points compared to text-only configuration, highlighting the importance of temporal information in understanding power system operations.
The complete multimodal fusion scheme achieved an F1-Score of 0.796, improving by 5.1 percentage points compared to single text features and 6.8 percentage points compared to single temporal features. This significant improvement demonstrates the synergistic effects of combining text and temporal modalities, validating our hypothesis that multimodal approaches are essential for comprehensive sample correction in power systems.
The chain-of-thought reasoning mechanism also played a crucial role in system performance. Without chain-of-thought reasoning, the model’s comprehensive performance decreased by approximately 2.8 percentage points, indicating that DeepSeek-based longchain-of-thought reasoning can effectively improve the model’s logical reasoning ability and professional knowledge understanding depth. This component is particularly valuable for handling complex technical scenarios that require multi-step reasoning.
The introduction of knowledge graphs brought significant performance improvements, with a 3.5 percentage point decrease in F1-Score when removed. This component particularly excels in handling professional terminology and domain-specific knowledge, ensuring that corrections adhere to established power system standards and practices.
The cross-modal attention mechanism showed the largest individual impact, with its removal causing a 4.2 percentage point decrease in F1-Score. This result validates our architectural choice of sophisticated attention-based fusion over simpler concatenation approaches, demonstrating that learning complex interaction patterns between modalities is crucial for optimal performance.
Finally, the beam search repair mechanism contributed 2.3 percentage points to the overall performance, showing that sophisticated text generation approaches are necessary for producing high-quality corrections that maintain both linguistic fluency and technical accuracy.

4.5. Fine-Tuning Analysis and Model Adaptation

To comprehensively evaluate the effectiveness of fine-tuning on domain-specific data, we conducted systematic experiments comparing pre-trained baseline models with their fine-tuned counterparts. Table 3 presents the performance comparison before and after fine-tuning on our power system dataset.
Table 3. Performance comparison of baseline models before and after fine-tuning on power system dataset. Results shown as mean ± std across 10 runs. Δ represents absolute improvement from pre-trained to fine-tuned models.
The results demonstrate that fine-tuning significantly improved model performance across all metrics. For the GECToR baseline model, fine-tuning yielded improvements of 6.3 percentage points in the BLEU score, 6.4 percentage points in the ROUGE score, 7.4 percentage points in the METEOR score, and 6.3 percentage points in the F1-Score. Notably, our method showed comparable or superior improvement margins (7.2, 7.8, 6.5, and 6.5 percentage points, respectively), validating that the performance gains were not solely attributable to fine-tuning but resulted from the architectural innovations including multimodal fusion, knowledge graph integration, and chain-of-thought reasoning.

4.6. Case Analysis and Application Effects

Through specific case analysis, we can more intuitively see the actual effects of our method. When processing distribution network dispatching maintenance applications, traditional methods often struggle to accurately identify and correct the incorrect use of professional terminology, while our method, through integrating knowledge graphs and chain-of-thought reasoning, can accurately identify the normative differences between “10 kV distribution line” and “10 kilovolt distribution line” and provide correct correction suggestions.
In the processing of dispatching operation tickets, our method demonstrates strong regional adaptability. For dispatching terminology and operational procedures specific to Xuancheng region in Anhui, the model can provide correction results that comply with local regulations based on regionalized knowledge provided by the external knowledge base. This regional adaptability is difficult for traditional general models to achieve, fully reflecting the value of our method in practical applications.
Experimental results also show that our method is significantly effective in reducing hallucination phenomena in multimodal large models. Through introducing knowledge graph constraints and chain-of-thought reasoning verification, the correction text generated by the model significantly improves in factual accuracy, with error rates reduced by approximately 35% compared to baseline methods. This is of great significance for application domains like power systems that require extremely high accuracy.

5. Conclusions

This paper presented a multimodal sample correction framework that effectively addresses data quality challenges in power system applications through large-model instruction enhancement and knowledge guidance. The proposed approach employs a three-stage processing pipeline combining multimodal feature extraction and fusion, knowledge-injected assessment, and sample correction. The framework focuses on two key modalities, temporal data and text documentation, which are the most critical data types in power system operations. Key innovations include temporal attention mechanisms for time series processing, knowledge graph embeddings for domain-specific reasoning, and gating networks for correction strategy selection. Experimental results demonstrated significant improvements over existing baseline approaches across multiple evaluation metrics. The framework successfully reduced hallucinations in multimodal large models while maintaining high accuracy in power professional scenarios, providing a robust foundation for intelligent operation and maintenance systems and offering promising directions for multimodal data quality management in other industrial domains.

Author Contributions

Conceptualization, H.Y.; methodology, Z.C.; software, J.D.; formal analysis, M.X.; investigation, S.Z.; data curation, S.Z.; writing—original draft, Z.C.; writing—review and editing, J.D.; visualization, M.X.; funding acquisition, H.Y. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the Science and Technology Project from State Grid Corporation of China (No. 5700-202490330A-2-1-ZX).

Data Availability Statement

The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.

Conflicts of Interest

Authors Zhenyu Chen, Huaguang Yan and Jianguang Du were employed by the company State Grid Corporation of China, Author Huaguang Yan was employed by the company China Electric Power Research Institute, Author Meng Xue was employed by the company State Grid Information and Telecommunication Group Co., Ltd., Author Shuai Zhao was employed by the company State Grid Zhejiang Electric Power Company. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

  1. Xu, P.; Zhu, X.; Clifton, D.A. Multimodal learning with transformers: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 12113–12132. [Google Scholar] [CrossRef] [Scilit]
  2. Alsumaidaee, Y.A.M.; Yaw, C.T.; Koh, S.P.; Tiong, S.K.; Chen, C.P.; Ali, K. Review of medium-voltage switchgear fault detection in a condition-based monitoring system by using deep learning. Energies 2022, 15, 6762. [Google Scholar] [CrossRef] [Scilit]
  3. Zhao, W.; Zeng, T.; Liu, Z.; Xie, L.; Xi, L.; Ma, H. Automatic generation control in a distributed power grid based on multi-step reinforcement learning. Prot. Control Mod. Power Syst. 2024, 9, 39–50. [Google Scholar] [CrossRef] [Scilit]
  4. Chen, Z.; Du, J.; Li, J.; Lv, H. MDFN: Enhancing Power Grid Image Quality Assessment via Multi-Dimension Distortion Feature. Sensors 2025, 25, 3414. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  5. Zheng, C.; Gao, X.; Chen, Z.; Lyu, L. Dual Backbone Multi-Attention Hierarchical Fusion and Feature Enhancement Network for Crowd Counting. IEEE Trans. Consum. Electron. 2025, 71, 3279–3293. [Google Scholar] [CrossRef] [Scilit]
  6. Wang, M.; Pi, H.; Li, R.; Qin, Y.; Tang, Z.; Li, K. VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completion. In Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA, 25 Feburary–4 March 2025; Volume 39, pp. 7808–7816. [Google Scholar]
  7. Wang, C.; Gao, X.; Wu, M.; Lam, S.K.; He, S.; Tiwari, P. Looking Clearer with Text: A Hierarchical Context Blending Network for Occluded Person Re-Identification. IEEE Trans. Inf. Forensics Secur. 2025, 20, 4296–4307. [Google Scholar] [CrossRef] [Scilit]
  8. Chen, B.; Wu, Q.; Li, M.; Xiahou, K. Detection of false data injection attacks on power systems using graph edge-conditioned convolutional networks. Prot. Control Mod. Power Syst. 2023, 8, 1–12. [Google Scholar] [CrossRef] [Scilit]
  9. Gao, X.; Wang, X.; Chen, Z.; Zhou, W.; Hoi, S.C. Knowledge enhanced vision and language model for multi-modal fake news detection. IEEE Trans. Multimed. 2024, 26, 8312–8322. [Google Scholar]
  10. Wang, C.; Cao, R.; Wang, R. Learning discriminative topological structure information representation for 2D shape and social network classification via persistent homology. Knowl.-Based Syst. 2025, 311, 113125. [Google Scholar]
  11. Hu, W.; Zheng, T.; Du, P.; Chen, Z. An Intelligent Assisted Assessment Method for Distribution Grid Engineering Quality Based on Large Language Modelling and Ensemble Learning. Prot. Control Mod. Power Syst. 2025, 10, 83–97. [Google Scholar] [CrossRef] [Scilit]
  12. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 2020, 33, 1877–1901. [Google Scholar]
  13. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers); Association for Computational Linguistics: Minneapolis, MN, USA, 2019; pp. 4171–4186. [Google Scholar]
  14. Omelianchuk, K.; Atrasevych, V.; Chernodub, A.; Skurzhanskyi, O. GECToR–grammatical error correction: Tag, not rewrite. arXiv 2020, arXiv:2005.12592. [Google Scholar]
  15. Malmi, E.; Krause, S.; Rothe, S.; Mirylenka, D.; Severyn, A. Encode, tag, realize: High-precision text editing. arXiv 2019, arXiv:1909.01187. [Google Scholar] [CrossRef] [Scilit]
  16. Niu, Z.; Zhong, G.; Yu, H. A review on the attention mechanism of deep learning. Neurocomputing 2021, 452, 48–62. [Google Scholar] [CrossRef] [Scilit]
  17. Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient estimation of word representations in vector space. arXiv 2013, arXiv:1301.3781. [Google Scholar] [CrossRef] [Scilit]
  18. Clark, K.; Luong, M.T.; Le, Q.V.; Manning, C.D. Electra: Pre-training text encoders as discriminators rather than generators. arXiv 2020, arXiv:2003.10555. [Google Scholar]
  19. Zhu, S.; Ma, H.; Chen, L.; Wang, B.; Wang, H.; Li, X.; Gao, W. Short-term load forecasting of an integrated energy system based on STL-CPLE with multitask learning. Prot. Control Mod. Power Syst. 2024, 9, 71–92. [Google Scholar]
  20. Chen, X.; Yu, T.; Pan, Z.; Wang, Z.; Yang, S. Graph representation learning-based residential electricity behavior identification and energy management. Prot. Control Mod. Power Syst. 2023, 8, 28. [Google Scholar]
  21. Graves, A. Long short-term memory. In Supervised Sequence Labelling with Recurrent Neural Networks; Springer: Berlin/Heidelberg, Germany, 2012; pp. 37–45. [Google Scholar]
  22. Cho, K.; Van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv 2014, arXiv:1406.1078. [Google Scholar] [CrossRef] [Scilit]
  23. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30. [Google Scholar]
  24. Zhuang, W.; Fan, J.; Xia, M.; Zhu, K. A multi-scale spatial–temporal graph neural network-based method of multienergy load forecasting in integrated energy system. IEEE Trans. Smart Grid 2023, 15, 2652–2666. [Google Scholar] [CrossRef] [Scilit]
  25. Sahoo, A.K.; Samal, S.K. Deep learning based detection, classification, and location of power system faults. Bull. Electr. Eng. Inform. 2024, 13, 4248–4259. [Google Scholar] [CrossRef] [Scilit]
  26. Zhu, L.; Gao, J.; Zhu, C.; Deng, F. Short-term power load forecasting based on spatial-temporal dynamic graph and multi-scale Transformer. J. Comput. Des. Eng. 2025, 12, 92–111. [Google Scholar] [CrossRef] [Scilit]
  27. Li, C.; Wang, B. A knowledge graph method towards power system fault diagnosis and classification. Electronics 2023, 12, 4808. [Google Scholar] [CrossRef] [Scilit]
  28. Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Chen, X.; Zhang, H.; Tian, X.; Zhu, D.; Tian, H.; Wu, H. Ernie: Enhanced representation through knowledge integration. arXiv 2019, arXiv:1904.09223. [Google Scholar] [CrossRef] [Scilit]
  29. Peters, M.E.; Neumann, M.; Logan, R.; Schwartz, R.; Joshi, V.; Singh, S.; Smith, N.A. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (Emnlp-Ijcnlp); Association for Computational Linguistics: Hong Kong, China, 2019; pp. 43–54. [Google Scholar]
  30. Yao, L.; Mao, C.; Luo, Y. KG-BERT: BERT for knowledge graph completion. arXiv 2019, arXiv:1909.03193. [Google Scholar] [CrossRef] [Scilit]
  31. Nathani, D.; Chauhan, J.; Sharma, C.; Kaul, M. Learning attention-based embeddings for relation prediction in knowledge graphs. arXiv 2019, arXiv:1906.01195. [Google Scholar] [CrossRef] [Scilit]
  32. Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; Yakhnenko, O. Translating embeddings for modeling multi-relational data. Adv. Neural Inf. Process. Syst. 2013, 26. [Google Scholar]
  33. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.V.; Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural Inf. Process. Syst. 2022, 35, 24824–24837. [Google Scholar]
  34. Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.R.; Cao, Y. React: Synergizing reasoning and acting in language models. In Proceedings of the The Eleventh International Conference On Learning Representations, Kigali, Rwanda, 1–5 May 2023. [Google Scholar]
  35. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. Self-Refine: Iterative Refinement with Self-Feedback. arXiv 2023, arXiv:2303.17651. [Google Scholar] [CrossRef] [Scilit]
  36. Chen, J.; Zhang, Z.; Zhao, H. Modeling hierarchical reasoning chains by linking discourse units and key phrases for reading comprehension. In Proceedings of the 29th International Conference On Computational Linguistics; International Committee on Computational Linguistics: Gyeongju, Republic of Korea, 2022; pp. 1467–1479. [Google Scholar]
  37. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv 2022, arXiv:2203.11171. [Google Scholar]
  38. Child, R. Generating long sequences with sparse transformers. arXiv 2019, arXiv:1904.10509. [Google Scholar] [CrossRef] [Scilit]
  39. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; Liu, P.J. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 2020, 21, 140. [Google Scholar]
  40. Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; Zettlemoyer, L. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Virtual, 5–10 July 2020; pp. 7871–7880. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.