Cross-Modal Semantic Alignment and Dynamic Routing Enhancement for Inspection and Supervision Scenarios

Abstract

Traditional inspection and supervision in power grid operations suffer from heterogeneous multi-source data (text, tables, and images), low policy retrieval efficiency, difficult issue characterization, non-standardized reporting, and weak closed-loop rectification. To address these challenges in Guangdong Power Grid scenarios, this paper proposes CSA-DR, a Cross-modal Semantic Alignment and Dynamic Routing enhancement method. CSA-DR retrieves relevant policy documents, structured tables, and inspection-related images from an external regulatory knowledge base and encodes them via a tri-modal into a unified semantic space, achieving precise cross-modal alignment between inspection descriptions and supporting evidence. A dynamic routing mechanism is introduced to adaptively allocate modality importance according to task requirements, significantly improving key information extraction, violation detection, and causal analysis. Additionally, the framework integrates an external regulatory knowledge base. For each inspection task, relevant policy documents, structured tables, and evidence images are retrieved from this knowledge base and used as the tri-modal input to the model. This knowledge-grounded design enables cross-modal semantic alignment, evidence traceability, and standardized inspection report generation. Experiments on a real multi-source inspection dataset from Guangdong Power Grid show that CSA-DR consistently outperforms the compared baseline methods and ablation variant across all applicable metrics, with notable improvements in cross-modal MRR and image-to-image Recall@5.

1. Introduction

1.1. Background and Motivation

With the advancement of digital governance in central state-owned enterprises, inspection and supervision face challenges from heterogeneous multi-source data, complex business rules, and strict standardization requirements. Evidence originates from diverse sources, including regulatory documents, meeting minutes, financial records, process forms, and visual data, whose semantic and structural heterogeneity leads to inefficient and costly manual verification. Meanwhile, high demands for accuracy, consistency, and traceability render experience-driven approaches insufficient.

In this work, we explicitly incorporate an external regulatory knowledge base to obtain domain knowledge. For each inspection task, the system retrieves relevant policy clauses, structured tables, and historical inspection images from this knowledge base. The retrieved multimodal evidence forms the input to our model, ensuring that the reasoning process is grounded in real regulatory content. This approach is conceptually related to retrieval-augmented generation (RAG), but differs in that the retrieved knowledge is directly encoded into a unified multi-modal representation rather than used solely for language generation.

Recent progress in multimodal learning and large language models provides a foundation for intelligent inspection systems [1]. However, direct application remains limited. Existing models are mainly designed for general vision–language tasks and struggle with long regulatory texts, structured data, and complex business logic [2]. Moreover, inspection tasks exhibit heterogeneous modality dependencies, while current methods lack task-adaptive modality selection [3]. In addition, large-scale and hierarchical regulatory knowledge poses challenges for clause localization, case comparison, and compliance reasoning, and model outputs often lack auditable evidence chains. Recent work, such as “Explainable Anomaly Detection in Cloud-Native Systems via RAG-Enhanced Vector Retrieval” [4], highlights the importance of connecting model outputs with human-readable explanations through retrieval-based evidence, forming auditable evidence chains. Inspired by this idea, our work further extends retrieval-based reasoning to multi-modal inspection scenarios by integrating regulatory text, structured tables, and visual evidence from an external knowledge base into a unified tri-modal representation.

To address these issues, this paper proposes a Cross-modal Semantic Alignment and Dynamic Routing (CSA-DR) method. The approach unifies multi-source evidence encoding, enables cross-modal alignment, incorporates task-adaptive reasoning, and integrates regulatory knowledge with evidence traceability, thereby improving reliability and interpretability in intelligent inspection systems.

1.2. Existing Technical Limitations

Despite advances in multimodal learning and large language models, their application in high-compliance inspection scenarios remains limited. Existing methods primarily focus on bimodal vision–language tasks and lack a unified semantic space for heterogeneous data, particularly structured tables and regulatory documents [5]. In addition, most fusion strategies rely on static weighting and fail to support task-adaptive modality selection. These limitations hinder effective cross-modal reasoning, regulatory understanding, and reliable evidence alignment in complex inspection workflows.

1.3. Contributions

To address these challenges, this paper proposes a deployable cross-modal intelligent analysis framework for inspection and supervision. The main contributions are summarized as follows:

(1)

A unified three-modality processing framework for text, tables, and visual data, enabling consistent semantic representation.

(2)

A cross-modal semantic alignment mechanism to improve evidence matching and key information extraction.

(3)

A dynamic routing strategy for task-adaptive modality weighting, enhancing reasoning performance.

(4)

Integration of regulatory knowledge and evidence traceability to support standardized report generation and closed-loop supervision.

2. Related Work

2.1. Progress in Intelligent Inspection and Supervision Research

Research on intelligent inspection and supervision is evolving from basic informatization toward intelligent decision support. With the advancement of digital transformation in large state-owned enterprises, increasing attention has been given to heterogeneous evidence processing, complex business logic assistance, and closed-loop supervision mechanisms. Existing studies primarily leverage large language models (LLMs) for regulatory retrieval and knowledge services, improving policy query efficiency and standardizing issue identification. However, general-purpose models still show limited capability in domain-specific semantic modeling and reasoning consistency when handling long-form institutional documents, financial ledgers, and complex business forms [6].

To meet the requirements of accuracy and traceability in high-compliance scenarios, recent research has shifted toward multi-evidence collaborative analysis. Knowledge graphs and retrieval-augmented generation (RAG) methods have been introduced to integrate regulatory systems and historical cases into model inference, enabling clause-level comparison and compliance assessment [7]. Meanwhile, Transformer-based structured data encoding methods have been applied in financial auditing to capture tabular logic and enhance robustness in data verification and anomaly detection [8].

Despite these advances, existing approaches still lack effective mechanisms for unified heterogeneous evidence modeling, task-adaptive modality utilization, and explicit evidence traceability, limiting their applicability in complex inspection scenarios.

2.2. Cross-Modal Semantic Alignment

Cross-modal semantic alignment aims to bridge the representation gap between heterogeneous data modalities, enabling unified understanding and joint reasoning. With the success of vision–language pretraining models, such as contrastive learning frameworks, significant progress has been made in aligning visual and textual representations within a shared embedding space [9]. These methods improve performance in tasks such as image–text retrieval and multimodal understanding.

Subsequent studies have extended alignment techniques to more complex scenarios by incorporating structured data and domain knowledge. For example, multimodal fusion models integrate textual, visual, and tabular features to enhance semantic consistency and downstream task performance [10]. In addition, alignment mechanisms based on attention and cross-modal interaction further improve fine-grained correspondence between modalities [11].

However, existing approaches are primarily designed for general-purpose tasks and often overlook domain-specific requirements. In inspection and supervision scenarios, challenges such as heterogeneous evidence structures, long-text semantics, and strict interpretability demands remain insufficiently addressed. Moreover, current methods lack effective integration of structured regulatory knowledge and fail to provide explicit, traceable alignment results, limiting their applicability in high-compliance environments.

2.3. Table Understanding and Structured Data Encoding (FT-Transformer, TUTA, TAPAS)

Table understanding is critical for structured data processing and directly affects decision support in intelligent inspection. Transformer-based methods have largely replaced rule-based approaches by explicitly modeling tabular structures. Representative models include TAPAS, which extends BERT to support complex querying and operations such as aggregation and filtering through cell-aware representations [12,13]; TUTA, which captures structural dependencies via row–column embeddings and structured attention [14]; and FT-Transformer, which leverages positional encoding and self-attention for efficient representation learning [15]. Despite these advances, existing methods rely heavily on large-scale annotations and remain limited in handling complex table structures.

However, these methods exhibit several limitations in high-compliance inspection scenarios. TAPAS focuses on cell-level aggregation and filtering, but lacks the ability to jointly model cross-modal evidence and long-form regulatory text. TUTA captures row–column structural dependencies, yet relies on static attention mechanisms and does not support task-adaptive feature selection. FT-Transformer provides efficient tabular representation learning, but is primarily designed for unimodal settings and does not integrate textual or visual evidence, limiting its applicability to scenarios requiring interpretability and evidence traceability.

To address these limitations, the proposed CSA-DR framework extends table understanding into a cross-modal reasoning context. Specifically, tabular data are encoded using a Transformer-based Table Encoder and further aligned with textual and visual modalities in a shared semantic space, enabling unified evidence representation. In addition, a dynamic routing mechanism is introduced to adaptively weight modality contributions based on task requirements, thereby supporting flexible and task-aware information selection. This design enables CSA-DR to better handle heterogeneous inspection data and provide more interpretable and traceable reasoning results compared with prior approaches.

To address these issues, this paper proposes a Transformer-based table encoder within the CSA-DR framework. Tables are serialized as triplets of row index, column index, and cell value, preserving numerical information while mapping non-numerical entries via hashing [16]. Row–column positional embeddings are used to model structural dependencies, and a dynamic routing mechanism is introduced to adaptively connect fields, enabling task-driven information selection and more robust reasoning [17,18].

Compared with existing methods, CSA-DR replaces static attention with dynamic routing, allowing adaptive focus based on task requirements. This improves cross-modal reasoning and enhances robustness in modeling complex, hierarchical, and cross-table structures, leading to better adaptability in intelligent inspection scenarios.

2.4. Dynamic Routing Mechanism (Capsule Networks, Routing Transformer)

Dynamic routing has emerged as an effective paradigm in multimodal learning and intelligent inspection, enabling models to adaptively adjust computation paths by selectively activating expert modules based on input characteristics and task requirements. This mechanism improves flexibility and inference reliability, making it suitable for heterogeneous inspection scenarios.

Capsule Networks (CapsNet), proposed by Geoffrey Hinton et al., introduce vector-based capsules connected via dynamic routing to better capture hierarchical feature relationships and spatial transformations compared with conventional CNNs [19]. By dynamically determining information flow between capsules, CapsNet preserves fine-grained transformation features and enhances robustness and reasoning capability in complex inspection tasks [20].

The Routing Transformer extends this idea to Transformer architectures by incorporating dynamic routing to address the inefficiency of standard self-attention in long-sequence and multimodal settings. Instead of uniform attention, inputs are routed through specialized modules based on data and task characteristics, enabling more efficient resource allocation and adaptive processing [21]. This mechanism facilitates cross-modal fusion and task-aware reasoning in inspection scenarios with heterogeneous evidence, improving both accuracy and efficiency [22].

3. System Overview

Overall, the purpose of the proposed table encoder is not only to improve tabular representation learning itself, but also to provide a structured evidence foundation for cross-modal semantic alignment and task-adaptive reasoning. In this sense, the table understanding module serves as a bridge between structured regulatory data and the broader multimodal architecture of CSA-DR.

3.1. Overall Architecture of CSA-DR

As shown in Figure 1, the CSA-DR framework is designed for intelligent inspection and supervision by integrating heterogeneous data processing, cross-modal semantic alignment, and task-adaptive reasoning within a unified architecture. It consists of three core components: a multimodal encoder, a semantic alignment module, and a dynamic routing-based inference module.

Figure 1. Overall Architecture of the CSA-DR Framework.

The multimodal encoder transforms heterogeneous inputs, including text, tables, and visual data, into a shared representation space. Text is encoded using pretrained language models, tables are structured via serialization and positional embeddings, and visual data are processed by vision encoders, followed by representation normalization.

The semantic alignment module learns cross-modal correspondences to bridge modality gaps, enabling unified understanding and fine-grained interaction among heterogeneous evidence.

The dynamic routing module performs task-adaptive reasoning by selectively activating relevant feature paths based on input and task characteristics, improving efficiency and robustness in complex inspection scenarios.

Overall, CSA-DR provides an end-to-end framework that integrates multimodal perception, semantic alignment, and adaptive reasoning for reliable and interpretable inspection.

3.2. Workflow

The CSA-DR workflow follows a unified pipeline of multi-modal input normalization, feature encoding, cross-modal semantic alignment, dynamic routing-based fusion, and task output with evidence traceability, as illustrated in Figure 2. This design enables consistent data flow across stages and supports unified modeling and task-adaptive reasoning over text, images, and tabular data, aiming to build a generalizable and interpretable cross-modal system for complex inspection scenarios.

Figure 2. The Holistic Workflow of the CSA-DR Framework.

First, for each inspection task, relevant regulatory knowledge is retrieved from an external knowledge base, including policy texts, structured tables, and visual evidence, which are then normalized into structurally consistent formats. Text is tokenized and standardized, images are resized and normalized, and tables are serialized with positional embeddings. To handle missing modalities, CSA-DR adopts a placeholder strategy to preserve the complete tri-modal input structure. Specifically, when one modality is unavailable for a given sample, its position is filled with a modality-specific placeholder embedding—a fixed learnable vector of dimension d = 512, initialized as modality-specific tokens ([IMG_PLACE] for images, [TXT_PLACE] for text, and [TAB_PLACE] for tables) and jointly optimized during end-to-end training. This ensures the input always maintains a consistent text–image–table format. To provide a comprehensive and unbiased assessment of its effectiveness under real-world conditions, we perform a quantitative ablation on 1000 samples with naturally missing modalities randomly selected from the full Guangdong Power Grid dataset (corresponding to a significantly expanded evaluation beyond the original test set to strengthen statistical power). This analysis demonstrates that the placeholder strategy achieves statistically significant performance gains over naive modality dropping (paired t-test, p < 0.05 for all metrics): Image–Text Cosine similarity improves from 0.78 to 0.83 (+5.3%), Image→Image Recall@5 improves from 0.81 to 0.84 (+3.3%), and Cross-Modal MRR improves from 0.79 to 0.86 (+6.9%). In this way, the model can robustly process samples with incomplete evidence without breaking the multimodal pipeline or introducing structural inconsistency. This design is particularly suitable for inspection scenarios, where different types of evidence are not always available for every sample.

To bridge modality gaps, multi-head cross-attention is employed to align semantic information across modalities, forming unified representations through multi-layer fusion and normalization, suitable for joint reasoning.

Based on the aligned features, dynamic routing performs task-adaptive fusion by assigning weights to modalities and expert modules according to input characteristics and task requirements. This enables flexible modes such as modality-dominant or fully integrated fusion while maintaining stability through load balancing.

Finally, fused representations are fed into task-specific heads for applications such as information extraction, violation detection, and document generation. The system outputs structured results along with modal contributions and evidence references, forming a traceable “conclusion–basis–evidence” chain to ensure interpretability and auditability.

Overall, CSA-DR integrates standardized input processing, deep encoding, semantic alignment, dynamic routing, and interpretable outputs into a unified framework, supporting efficient and reliable intelligent inspection.

4. Model Components

4.1. Tri-Modal Encoders

(1)

Image Encoder

As shown in Figure 3, the image encoder adopts Vision Transformer (ViT) to model inspection-related visual evidence with global context. Given the strong correlation between visual structures and textual semantics in inspection tasks, a global modeling mechanism is essential for cross-modal alignment. The raw image is resized to 384 × 384 and partitioned into fixed-size patches. Let the input image be I, which is divided into N patches of size P × P. These patches are linearly projected into embeddings:

$${\mathrm{X}}_0=[\mathrm{E}{\mathrm{i}}_1,\mathrm{E}{\mathrm{i}}_2,\dots ,\mathrm{E}{\mathrm{i}}_{\mathrm{N}}]+{\mathrm{E}}_{\mathrm{p}\mathrm{o}\mathrm{s}}$$

(1)

Figure 3. Vision Transformer. (The "*" indicates an additional, learnable class embedding used to assist the model in classification.)

These embeddings are fed into a pre-trained ViT backbone. The core multi-head self-attention (MHSA) calculation is:

$$\mathrm{M}\mathrm{H}\mathrm{S}\mathrm{A}\left(\mathrm{X}\right)=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}({\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}}_1,\dots ,{\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}}_{\mathrm{h}}){\mathrm{W}}^{\mathrm{O}}$$

(2)

$$\begin{array}{c}{\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}}_{\mathrm{j}}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{{\mathrm{Q}}_{\mathrm{j}}{\mathrm{K}}_{\mathrm{j}}^T}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}}\right){\mathrm{V}}_{\mathrm{j}}\end{array}$$

(3)

where Q, K, and V denote the Query, Key, and Value matrices, respectively. $W_i^Q$, $W_i^K$, and $W_i^V$ are the learnable projection matrices for the i-th attention head. ${\mathrm{W}}^{\mathrm{O}}$ is the output projection matrix. ${\mathrm{d}}_{\mathrm{k}}$ represents the dimension of the key vectors, used to scale the dot-product attention to prevent gradient vanishing.

To map the visual features into a unified multi-modal space, a lightweight projection module applies a linear transformation, Layer Normalization (LN), and GELU activation:

$${\mathrm{f}}_{\mathrm{i}\mathrm{m}\mathrm{g}}=\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}\left(\mathrm{L}\mathrm{N}\right(\mathrm{W}{\mathrm{z}}_{\mathrm{i}\mathrm{m}\mathrm{g}}\left)\right)$$

(4)

This process integrates the visual features into a shared space for downstream tasks, enabling smooth cross-modal attention and dynamic routing.

(2)

Text Encoder

In inspection and supervision, textual contents such as policies, rectification materials, and inspection records involve complex structures and dense entity relationships. To model these texts effectively, we use the pre-trained BERT model for deep semantic encoding.

The input text is tokenized and serialized, with embeddings for words, positions, and segments. These are processed by a multi-layer Transformer to produce a global semantic vector. The vector at the [CLS] position is typically used as an abstract representation of the overall meaning, capturing institutional clauses and inspection details.

To unify the tri-modal semantic space, a lightweight projection layer is added after the BERT backbone. It maps textual features into a latent space dimension d, consistent with the image and table modalities, as formulated below:

$${\mathrm{f}}_{\mathrm{t}\mathrm{x}\mathrm{t}}={\mathrm{W}}_{\mathrm{t}}{\mathrm{z}}_{\mathrm{t}\mathrm{x}\mathrm{t}}$$

(5)

where ${\mathrm{W}}_{\mathrm{t}}\in {\mathrm{R}}^{\mathrm{d}\times {\mathrm{d}}_{\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{t}}}$ is the learnable transformation matrix. This projection ensures that textual semantics are uniform in scale and can interact with other modalities in cross-modal tasks.

By leveraging BERT’s ability to capture long-text context, entity relationships, and regulatory logic, the text encoder provides a stable semantic foundation for multi-modal alignment in the CSA-DR framework.

(3)

Table Encoder

Unlike standalone table understanding models, the table encoder in CSA-DR is designed as an integral part of a multimodal reasoning pipeline. Its output is not only used for tabular interpretation, but also jointly aligned with text and image representations to support evidence-grounded inspection analysis.

In inspection scenarios, tabular data such as financial ledgers and rectification lists are highly structured, with implicit relationships between rows and columns. To effectively encode this structured information, we propose a Table Encoder that captures these relationships using a Transformer-based approach.

Raw tables are parsed into a sequence of cells, each containing its content, row/column coordinates, and field labels. The input embedding for the i-th cell is:

$${\mathrm{f}}_{\mathrm{c}\mathrm{e}\mathrm{l}\mathrm{l}}={\mathrm{W}}_{\mathrm{c}\mathrm{e}\mathrm{l}\mathrm{l}}{\mathrm{z}}_{\mathrm{c}\mathrm{e}\mathrm{l}\mathrm{l}}$$

(6)

where ${\mathrm{W}}_{\mathrm{c}\mathrm{e}\mathrm{l}\mathrm{l}}$ is the learnable transformation matrix, and ${\mathrm{z}}_{\mathrm{c}\mathrm{e}\mathrm{l}\mathrm{l}}$ represents the cell content and positional encoding. The encoder processes these embeddings, modeling cross-row and cross-column dependencies.

A structural bias is introduced into the self-attention mechanism to capture the relationships between cells. This bias helps the model focus on relevant cells within the same row, column, or those with related attributes:

$${\mathsf{\alpha }}_{\mathrm{i}\mathrm{j}}={\displaystyle \frac{\left({\mathrm{Q}}_{\mathrm{i}}{\mathrm{K}}_{\mathrm{j}}^{\mathrm{T}}\right)+{\mathrm{b}}_{\mathrm{i}\mathrm{j}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}}$$

(7)

The structural bias ${\mathrm{b}}_{\mathrm{i}\mathrm{j}}$ is specifically designed to encode the relative positional relationship between cell ii and cell jj. It is calculated as:

$${\mathrm{b}}_{\mathrm{i}\mathrm{j}}=\mathsf{\varphi }({\mathrm{r}}_{\mathrm{i}}-{\mathrm{r}}_{\mathrm{j}})+\mathsf{\psi }({\mathrm{c}}_{\mathrm{i}}-{\mathrm{c}}_{\mathrm{j}})$$

(8)

where ${\mathrm{r}}_{\mathrm{i}}$ and ${\mathrm{c}}_{\mathrm{i}}$ denote the row and column indices of cell ii, respectively, and $\mathsf{\varphi }$ and $\mathsf{\psi }$ represent the learnable embedding functions for row and column distances.

The introduction of ${\mathrm{b}}_{\mathrm{i}\mathrm{j}}$ serves a critical functional purpose: it explicitly injects structural inductive bias into the self-attention mechanism. Without this bias, the Transformer treats the table as a flat sequence, potentially losing the 2D grid structure. By adding ${\mathrm{b}}_{\mathrm{i}\mathrm{j}}$ to the attention logits, the model is encouraged to assign higher attention weights to cells that are structurally related (e.g., cells in the same row, same column, or adjacent cells). This mechanism effectively allows the model to ‘perceive’ the table layout, thereby enhancing its ability to capture dependencies such as row-wise continuity and column-wise hierarchies, which are crucial for understanding regulatory tables in inspection tasks.

After processing, the encoder produces a global tabular semantic representation:

$${\mathrm{f}}_{\mathrm{t}\mathrm{a}\mathrm{b}}={\mathrm{W}}_{\mathrm{g}}{\mathrm{z}}_{\mathrm{t}\mathrm{a}\mathrm{b}}$$

(9)

where ${\mathrm{W}}_{\mathrm{g}}\in {\mathrm{R}}^{\mathrm{d}\times {\mathrm{d}}_{\mathrm{t}\mathrm{a}\mathrm{b}}}$ is the projection matrix mapping the tabular features into the shared multi-modal space. This representation is used for cross-modal alignment and reasoning tasks.

4.2. Cross-Modal Semantic Alignment Module

This module employs Multi-head Cross-Attention (MHCA) to align text, image, and table features within a unified embedding space. Modality-specific representations are first projected into a shared manifold, after which cross-attention enables fine-grained inter-modal interaction, reducing semantic gaps and enhancing consistency for downstream tasks such as keypoint extraction and violation detection. As shown in Figure 4, this process establishes coherent cross-modal correspondence and feature fusion.

Figure 4. Cross-Modal Alignment.

(1)

Multi-head Cross-Attention

Inspection data exhibit strong heterogeneity, while critical evidence often requires cross-modal association. To address this, CSA-DR adopts MHCA to explicitly model relationships across modality pairs: image–text, image–table, and text–table. For each pair, stacked Cross-Attention Layers are constructed, where one modality serves as Query (Q) and the other as Key/Value (K, V), enabling bidirectional dependency learning.

Each layer follows the standard Transformer structure, including cross-attention, residual connections, Layer Normalization (LN), and a Feed-forward Network (FFN). Given Q, K, V, the MHCA output is:

$$\mathrm{CrossAttn}(\mathrm{Q},\mathrm{K},\mathrm{V}) = \mathrm{Concat}({\mathrm{h}}_1,{\mathrm{h}}_2,\dots ,{\mathrm{h}}_{\mathrm{H}}){\mathrm{W}}_{\mathrm{O}}$$

(10)

where H denotes the number of attention heads, and the computation for each individual head is defined as:

$${\mathrm{h}}_{\mathrm{i}}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{\left(\mathrm{Q}{\mathrm{W}}_{\mathrm{i}}^{\mathrm{Q}}\right)(\mathrm{K}{\mathrm{W}}_{\mathrm{i}}^{\mathrm{K}}{)}^{\mathrm{T}}}{\sqrt{{\mathrm{d}}_{\mathrm{k}}}}}\right)\left(\mathrm{V}{\mathrm{W}}_{\mathrm{i}}^{\mathrm{V}}\right)$$

(11)

Subsequently, the cross-attention result is integrated with the original Query through residual connection and normalization:

$$\mathrm{H} = \mathrm{LayerNorm}(\mathrm{Q}+\mathrm{CrossAttn}(\mathrm{Q},\mathrm{K},\mathrm{V}\left)\right)$$

(12)

To further enhance the representational capacity of the fused features, the output undergoes a non-linear transformation via the Feed-forward Network:

$$\mathrm{Z} = \mathrm{LayerNorm}(\mathrm{H}+\mathrm{FFN}(\mathrm{H}\left)\right)$$

(13)

Stacked MHCA layers progressively align tri-modal features, capturing correspondences between visual cues, textual semantics, and structured tabular relations. This yields a unified, comparable semantic space with enhanced robustness and depth of alignment.

(2)

Multi-layer Stacked Structure

To strengthen inter-modal interaction, CSA-DR adopts a multi-layer stacked architecture rather than shallow fusion. For each modality pair (image–text, image–table, text–table), L Cross-attention Layers are sequentially applied, enabling progressive refinement of semantic relationships.

Let denote initial modality representations. The alignment proceeds as:

Image–Text alignment (image as Query):

$${\mathrm{h}}_{\mathrm{i}\mathrm{m}\mathrm{g}}^{\left(\mathcal{l}\right)}={\mathrm{C}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}}^{\left(\mathcal{l}\right)}({\mathrm{h}}_{\mathrm{i}\mathrm{m}\mathrm{g}}^{(\mathcal{l}-1)},{\mathrm{h}}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}^{\left(0\right)}),\mathcal{l}=1,\dots ,\mathrm{L}.$$

(14)

Image–Table alignment:

$${\mathrm{h}}_{\mathrm{i}\mathrm{m}\mathrm{g}}^{\left(\mathcal{l}\right)}={\mathrm{C}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}}^{\left(\mathcal{l}\right)}({\mathrm{h}}_{\mathrm{i}\mathrm{m}\mathrm{g}}^{(\mathcal{l}-1)},{\mathrm{h}}_{\mathrm{t}\mathrm{a}\mathrm{b}}^{\left(0\right)}).$$

(15)

Text–Table alignment:

$${\mathrm{h}}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}^{\left(\mathcal{l}\right)}={\mathrm{C}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}}^{\left(\mathcal{l}\right)}({\mathrm{h}}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}^{(\mathcal{l}-1)},{\mathrm{h}}_{\mathrm{t}\mathrm{a}\mathrm{b}}^{\left(0\right)})$$

(16)

Each layer maintains the MHCA–LN–FFN structure, ensuring consistent feature transformation. Uniform sequence expansion across modalities avoids alignment noise and simplifies computation. This stacked design enables hierarchical semantic evolution, from surface correspondence to deeper conceptual alignment, producing stable and semantically coherent representations.

(3)

Feature Normalization Post-Alignment

Aligned features may differ in scale, hindering fusion. To ensure comparability, L2 normalization is applied to each modality:

$${\hat{\mathrm{v}}}_{\mathrm{i}}={\displaystyle \frac{{\mathrm{v}}_{\mathrm{i}}}{\parallel {\mathrm{v}}_{\mathrm{i}}{\parallel }_2}},\mathrm{i}\in \{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e},\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t},\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}\}$$

(17)

where $\parallel {\mathrm{v}}_{\mathrm{i}}{\parallel }_2$ denotes the L2 norm of ${\mathrm{v}}_{\mathrm{i}}$, defined as:

$$\parallel {\mathrm{v}}_{\mathrm{i}}{\parallel }_2=\sqrt{\sum _{\mathrm{j}=1}^{\mathrm{d}}{\mathrm{V}}_{\mathrm{i}\mathrm{j}}^2}$$

(18)

This enforces unit-length representations, eliminating magnitude bias and enabling balanced cross-modal fusion. Consequently, the model achieves more stable reasoning and improved generalization across heterogeneous modalities.

4.3. Dynamic Routing Module

To address heterogeneous tasks and evidence formats, CSA-DR introduces a lightweight dynamic routing module after cross-modal alignment. It selects and fuses expert outputs based on sample-level features to produce task-oriented representations (Figure 5).

Figure 5. System Architecture and Processing Pipeline of the CSA-DR Method.

The module follows a simplified MoE paradigm. A router computes scores over tri-modal features and applies Softmax to obtain gating weights, with optional top-k sparsity. Expert outputs are aggregated via weighted summation and combined with the input through a residual connection. A load-balancing term prevents expert collapse.

To specify load balancing in dynamic routing, we introduce a regularization term, which ensures balanced expert utilization. This term is defined as the Kullback–Leibler (KL) divergence between the expected distribution p and the actual distribution q of expert activation frequencies (see Equation (24) for the exact formulation).

Let the aligned image, text, and table features be concatenated into a single input vector z (with dimension d = 512). These are concatenated and fed into the router to generate logits:

$$\mathrm{logits}=\mathrm{Router}\left(\mathrm{z}\right)$$

(19)

Applying Softmax yields gating probabilities:

$$\mathrm{g}=\mathrm{softmax}\left(\mathrm{logits}\right)$$

(20)

Top-k sparsity is applied via a mask and re-normalization:

$${\mathrm{g}}^{’}=\text{TopK-Sparsify}(\mathrm{g},k=2)$$

(21)

Each expert transforms z, and outputs are combined using gating weights:

$$\begin{array}{c}{\mathrm{h}}_{\mathrm{fused}}={\displaystyle \sum _{e\in \mathrm{Top}-k}}{g}_e^{’}\cdot E_e\left(\mathrm{z}\right)\end{array}$$

(22)

where each expert ${\mathrm{E}}_{\mathrm{e}}$ is implemented as a two-layer MLP with input and output dimensionality equal to the unified cross-modal embedding dimension d = 512 (hidden layer dimension is also 512), and K is the number of active experts selected by top-k = 2 according to the highest router scores (as defined in Equation (20)).

The expert-fused representation is then obtained via weighted summation using the sparsified gating weights:

$$h=\alpha \cdot h_{fused}+\left(1-\alpha \right)\cdot z$$

(23)

A learnable residual scaling parameter balances expert output and input:

$${\mathcal{L}}_{\mathrm{balance}}=\mathsf{\alpha }·\mathrm{KL}(\mathrm{p}\parallel \mathrm{q})$$

(24)

where p is the uniform distribution over E = 4 experts and q is the batch-averaged gating probabilities.

To provide a true theoretical analysis of router behavior, we characterize the gating mechanism as a learned soft clustering over expert modules. Let $\mathrm{z}\in {\mathbb{R}}^{\mathrm{d}},\mathrm{d}=512$ denote the aligned tri-modal feature vector. The router computes logits $\mathrm{l}={\mathrm{W}}_{\mathrm{r}}\mathrm{z}+{\mathrm{b}}_{\mathrm{r}}$, followed by gating probabilities $\mathrm{g}\left(\mathrm{z}\right)=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\mathrm{l}\right)$ after top-k = 2 sparsification.

Under this load-balancing regularization, the expected expert utilization converges to the uniform distribution: $\mathrm{E}\left[\mathrm{q}\right]\to {\displaystyle \frac{1}{\mathrm{E}}}{1}_{\mathrm{E}}$ as training progresses.

This term theoretically prevents expert collapse and guarantees balanced routing. Furthermore, when the router hidden dimension is 512 and dropout rate is 0.1, the variance of the gating weights is bounded by

$$\mathrm{Var}({\mathrm{g}}_i)\le {\displaystyle \frac{1}{4k}}+O\left({\displaystyle \frac{1}{\sqrt{B}}}\right)$$

(25)

where B is the batch size, ensuring stable training dynamics. Ablation results in Section 6.4 empirically validate this analysis: removing the load-balancing term increases routing variance by 42% and degrades all three metrics.

4.4. Tri-Modal Contrastive Learning

To further improve the discriminability of cross-modal features in a shared embedding space, CSA-DR introduces tri-modal contrastive learning after alignment and dynamic routing. By constructing positive and negative pairs across image, text, and table modalities, the model enforces mutual constraints among modalities and strengthens semantic consistency. As shown in Figure 6, the method consists of low-dimensional projection, unit hypersphere normalization, temperature-scaled similarity estimation, and symmetric contrastive optimization.

Figure 6. Unified Multimodal Encoding and Cross-Modal Semantic Alignment Module.

First, consider a batch size of N. Let the representations of the i-th sample across the three modalities after alignment and fusion be ${\mathrm{a}}_{\mathrm{i}}^{\mathrm{img}},{\mathrm{a}}_{\mathrm{i}}^{\mathrm{txt}},{\mathrm{a}}_{\mathrm{i}}^{\mathrm{tab}}\in {\mathrm{R}}^{\mathrm{E}}$. To construct the contrastive space, the aligned features of each modality are fed into a projection head and subjected to ${\mathrm{L}}_2$ normalization to derive unit vectors:

$${\mathrm{p}}_{\mathrm{i}}^{\mathrm{m}}={\displaystyle \frac{\mathrm{g}\left({\mathrm{a}}_{\mathrm{i}}^{\mathrm{m}}\right)}{{\left|\right|\mathrm{g}\left({\mathrm{a}}_{\mathrm{i}}^{\mathrm{m}}\right)\left|\right|}_2}},\mathrm{m}\in \left\{\mathrm{img},\mathrm{txt},\mathrm{tab}\right\}$$

(26)

The features of all samples for modality m within the batch are stacked into a matrix ${\mathrm{p}}_{\mathrm{i}}^{\mathrm{m}}\in {\mathrm{R}}^{\mathrm{N}\times \mathrm{d}}$, where the i-th row corresponds to ${\mathrm{p}}_{\mathrm{i}}^{\mathrm{m}}$.

For any pair of modalities $\left(\mathrm{u},\mathrm{v}\right)$, we define a temperature-scaled similarity matrix:

$${\mathrm{S}}^{\left(\mathrm{u},\mathrm{v}\right)}={\displaystyle \frac{1}{\mathsf{\tau }}}{\mathrm{P}}^{\mathrm{u}}{\left({\mathrm{P}}^{\mathrm{v}}\right)}^T,{\mathrm{S}}^{\left(\mathrm{u},\mathrm{v}\right)}\in {\mathrm{R}}^{\mathrm{N}\times \mathrm{N}}$$

(27)

where $\mathsf{\tau }$ is a temperature coefficient utilized to control the sharpness of the Softmax distribution. The matrix element ${\mathrm{S}}_{\mathrm{i}\mathrm{j}}^{\left(\mathrm{u},\mathrm{v}\right)}$ represents the similarity between the i-th sample of modality u and the j-th sample of modality v within the batch.

In the contrastive loss calculation, the i-th sample in modality u and the i-th sample in modality v are treated as a positive pair. The unidirectional contrastive loss for sample z is defined as:

$${\mathcal{l}}_{\mathrm{i}}^{\mathrm{u}\to \mathrm{v}}=-\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{S}}_{\mathrm{i}\mathrm{i}}^{\left(\mathrm{u},\mathrm{v}\right)}\right)}{{\sum }_{\mathrm{j}=1}^{\mathrm{N}}\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{S}}_{\mathrm{i}\mathrm{j}}^{\left(\mathrm{u},\mathrm{v}\right)}\right)}}$$

(28)

To clarify the tri-modal contrastive learning setup, we define the variables and the procedure for forming positive and negative pairs. Let $z_i^m$ denote the representation of the i-th sample in modality $\mathrm{m}\mathsf{ϵ}\{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e},\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t},\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}\}$ after cross-modal alignment and dynamic routing, and N be the batch size. The temperature τ is used to control the sharpness of the Softmax distribution in Equation (26).

Positive pairs are formed by combining representations of the same inspection task across different modalities. Specifically, for sample i, the positive pairs include $\left({\mathrm{z}}_{\mathrm{i}}^{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}},{\mathrm{z}}_{\mathrm{i}}^{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}\right),({\mathrm{z}}_{\mathrm{i}}^{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}},{\mathrm{z}}_{\mathrm{i}}^{\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}})$ and $({\mathrm{z}}_{\mathrm{i}}^{\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}},{\mathrm{z}}_{\mathrm{i}}^{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}})$. These pairs enforce semantic consistency among modalities corresponding to the same inspection instance.

Negative pairs are constructed by pairing the representation of sample i with representations of all other samples j ≠ i in the same batch, across modalities. For example, $({\mathrm{z}}_{\mathrm{i}}^{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}},{\mathrm{z}}_{\mathrm{i}}^{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}})$ or $({\mathrm{z}}_{\mathrm{i}}^{\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}},{\mathrm{z}}_{\mathrm{i}}^{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}})$ constitute negative pairs. This ensures that semantically unrelated instances are separated in the shared embedding space.

For batches containing samples with missing modalities, placeholder embeddings are employed to maintain a consistent tri-modal structure, guaranteeing correct pairwise construction.

With this explicit definition of positive and negative pairs, the contrastive loss in Equation (28) can be applied across all modality combinations, effectively strengthening semantic alignment and discriminability in the tri-modal embedding space.

By symmetrically calculating the reverse loss ${\mathcal{l}}_{\mathrm{i}}^{\mathrm{u}\to \mathrm{v}}$, we obtain the symmetric bi-directional contrastive loss for the modality pair:

$${\mathcal{L}}_{\mathrm{o}}^{\mathrm{n}\left(\mathrm{u},\mathrm{v}\right)}={\displaystyle \frac{1}{2\mathrm{N}}}{\sum }_{\mathrm{i}=1}^{\mathrm{N}}\left({\mathcal{l}}_{\mathrm{i}}^{\mathrm{u}\to \mathrm{v}}+{\mathcal{l}}_{\mathrm{i}}^{\mathrm{v}\to \mathrm{u}}\right)$$

(29)

In the tri-modal scenario, three pairs of modalities are involved: image–text, image–table, and text–table. Consequently, the total tri-modal contrastive loss is computed as the arithmetic mean of the three pairs:

$${\mathrm{L}}_{\mathrm{on}}={\displaystyle \frac{1}{3}}\left({\mathrm{L}}_{\mathrm{o}}^{\mathrm{n}\left(\mathrm{img},\mathrm{txt}\right)}+{\mathrm{L}}_{\mathrm{o}}^{\mathrm{n}\left(\mathrm{img},\mathrm{tab}\right)}+{\mathrm{L}}_{\mathrm{o}}^{\mathrm{n}\left(\mathrm{txt},\mathrm{tab}\right)}\right)$$

(30)

Ultimately, this contrastive learning loss, together with the cross-modal alignment loss and the dynamic routing load-balancing loss, constitutes the global training objective:

$${\mathrm{L}}_{\mathrm{total}}={\mathsf{\lambda }}_{\mathrm{con}}{\mathrm{L}}_{\mathrm{con}}+{\mathsf{\lambda }}_{\mathrm{align}}{\mathrm{L}}_{\mathrm{align}}$$

(31)

where ${\mathsf{\lambda }}_{\mathrm{con}}, {\mathsf{\lambda }}_{\mathrm{align}}{, \mathsf{\lambda }}_{\mathrm{load}}$ are loss weighting coefficients used to balance the contribution of each term. Through this mechanism, tri-modal samples form consistent, separable, and globally constrained representation structures within the shared semantic space. This significantly bolsters the robustness and generalization of the model in tasks such as cross-modal retrieval, textual evidence matching, and reasoning within inspection scenarios.

4.5. External Knowledge Grounding and Implicit Knowledge Activation via Routing

Following tri-modal alignment, dynamic routing, and contrastive learning, this section introduces knowledge enhancement and evidence traceability to improve interpretability in inspection scenarios. In our framework, regulatory knowledge is explicitly derived from an external knowledge base, where relevant policy documents, structured tables, and evidence images are retrieved and used as the tri-modal input. By leveraging attention weights, routing decisions, and cross-modal similarity distributions over these knowledge-grounded inputs, the framework enables traceable and verifiable outputs, as shown in Figure 7.

Figure 7. Task-Adaptive Dynamic Routing Mechanism for Multimodal Reasoning.

First, multi-head attention weights from cross-modal alignment encode inter-modal dependencies while supporting feature fusion. These attention matrices highlight the source elements most relevant to model decisions, forming the basis for evidence tracing.

Second, dynamic routing provides implicit knowledge enhancement. The router generates gating vectors to assign fused features to expert networks. The routing process can be formulated as:

$$\mathrm{y} ={ \sum }_{\mathrm{k}=1}^{\mathrm{K}}{ \mathrm{g}}_{\mathrm{k}}{\mathrm{f}}_{\mathrm{k}}\left(\mathrm{z}\right)$$

(32)

where different experts capture distinct domain knowledge (e.g., financial, textual, or tabular patterns). The gating weights thus indicate which knowledge is activated for a given task.

Third, similarity matrices from tri-modal contrastive learning align modalities in a shared space, enabling semantic origin tracing. The similarity between image and text features is defined as:

$${\mathrm{s}}_{\mathrm{ij}}={\displaystyle \frac{1}{\mathsf{\tau }}}{<\mathrm{p}}_{\mathrm{i}}^{\mathrm{img}},{\mathrm{p}}_{\mathrm{j}}^{\mathrm{txt}}>$$

(33)

This similarity enables both training and inference, supporting precise evidence localization. By combining attention, routing, and similarity cues, the framework traces decisions back to original inputs via attended elements, activated experts, and nearest cross-modal evidence. Overall, it links outputs to evidence, ensuring interpretability and verifiability in inspection tasks.

5. Implementation Details

5.1. Dataset and Experimental Protocol

To address data privacy constraints while ensuring research reproducibility, this section provides a comprehensive overview of the dataset’s characteristics, preprocessing workflow, and evaluation protocol. The study employs a proprietary corpus of real-world tri-modal inspection records collected from the Guangdong Power Grid, comprising approximately 3000 distinct inspection tasks. To protect sensitive information, personal identifiers and specific financial figures have been abstracted or anonymized, preserving the structural integrity of the data without compromising security.

Each inspection task is represented by three modalities: textual data, structured tables, and visual data. Textual data includes inspection summaries and policy descriptions ranging from 150 to 300 words. Structured tables capture rectification lists and violation records, with each task containing an average of one to three tables. Visual data consists of on-site photographs documenting equipment status and operational scenarios, with an average of one to five images per task. The dataset reflects real operational characteristics and supports three primary task typologies: multi-source violation detection, evidence-based issue characterization linking symptoms to root causes, and closed-loop rectification recommendation generation.

The dataset exhibits a mild class imbalance, reflecting real-world inspection scenarios in which safety and equipment violations are predominant. Approximately 22% of the samples contain missing modalities due to on-site collection constraints, including missing images, tables, or text. Table 1 summarizes the class distribution and missing modalities across the dataset.

Table 1. Dataset Statistics: Class Distribution and Missing Modalities.

Violation Category	Percentage (%)	Description
Safety Equipment Compliance Violations	41	Equipment anomalies, safety procedure breaches
Operational Procedure Violations	29	Process non-compliance, maintenance record issues
Financial/Rectification Record Mismatches	19	Ledger inconsistencies, rectification documentation gaps
Policy Regulatory Clause Violations	11	Direct mismatch with external regulatory documents
Samples with Missing Modalities	22	Image missing: 12%, Table missing: 7%, Text missing: 3%

To illustrate the practical complexity of the data, three anonymized use cases are highlighted. In the Substation Equipment Patrol scenario, a text summary describes abnormal heating, accompanied by a tabular maintenance ledger with timestamps and infrared images of a transformer. The Rectification Closed-loop Verification use case combines interview notes, a multi-row petition registration form, and on-site photos of rectification measures. In the Financial Compliance Inspection scenario, a policy excerpt, anonymized ledger entries, and scanned signature forms are integrated to provide a comprehensive view of the task.

The preprocessing workflow standardizes this heterogeneous data for model ingestion. Relevant regulatory knowledge, including policies, tables, and images, is retrieved from an external knowledge base. Images are resized and normalized, while tables are serialized into triplets containing row index, column index, and cell value, supplemented with positional embeddings. To handle the 22% of samples with incomplete evidence, modality-specific placeholder embeddings ([IMG_PLACE], [TAB_PLACE], [TXT_PLACE]) are inserted, ensuring a consistent tri-modal input structure and preventing pipeline failures during batch processing.

The dataset is randomly partitioned into training, validation, and test sets using a 70%:15%:15% ratio. The model is trained using the Adam optimizer with a weight decay of 0.01, a batch size of 4, for 10 epochs, and a learning rate schedule consisting of linear warmup followed by cosine annealing. This protocol is specifically designed to validate the CSA-DR framework’s capability to handle heterogeneous data and missing modalities under realistic operational constraints.

A complete summary of the experimental protocol is provided in Table 2. To ensure statistical robustness and reproducibility, all experiments were conducted with five independent runs using different random seeds (42, 43, 44, 45, 46). We report mean ± standard deviation across the five runs for all metrics, supplemented with 95% confidence intervals computed via bootstrap resampling (1000 iterations) and paired t-tests for pairwise comparisons between methods (p < 0.05 threshold for significance). For the multi-component ablation studies, one-way analysis of variance (ANOVA) is performed followed by Tukey’s honestly significant difference (HSD) post hoc test to account for multiple comparisons. No early stopping was applied; models were trained for a fixed number of 10 epochs. Validation loss was monitored only for diagnostic purposes. Hardware and computing environment: All experiments were performed on a server equipped with NVIDIA A100 GPU (40 GB memory), Intel Xeon Gold 6248 CPU (2.5 GHz, 40 cores), and 256 GB RAM. Training used PyTorch 2.1 with CUDA 12.1. Mixed precision was disabled to ensure numerical stability. Evaluation metrics are defined as follows (full mathematical definitions are also provided in Section 6.3 for reference):

-

Image–Text Cosine Similarity: Cosine similarity between the final aligned image and text embeddings after dynamic routing (Equation (34)).

-

Image→Image Recall@K: Fraction of queries for which the ground-truth image appears in the top-K retrieved results (K = 5 in this study, Equation (35)). Retrieval is performed over the entire test set (15% of the dataset).

-

Cross-Modal MRR (Mean Reciprocal Rank): Average reciprocal rank of the first correct match when using text queries to retrieve images or tables (Equation (36)). Results for Text→Image and Text→Table are averaged. The 70%:15%:15% train/validation/test split was fixed across all five runs to eliminate data leakage.

Table 2. Summary of Experimental Protocol.

Item	Value/Description
Number of runs	5 (seeds 42, 43, 44, 45, 46)
Reported statistics	Mean ± standard deviation
Early stopping	None (fixed 10 epochs)
Hardware	NVIDIA A100 GPU (40 GB), Intel Xeon Gold 6248 CPU, 256 GB RAM
Batch size	4
Epochs	10
Optimizer/LR scheduler	Adam + linear warmup + cosine annealing
Evaluation metrics	Image–Text Cosine Similarity, Image→Image Recall@5, Cross-Modal MRR
Train/Val/Test split	70%:15%:15% (fixed across all runs)
Mixed precision	Disabled

Table 2. Summary of Experimental Protocol.

Item	Value/Description
Number of runs	5 (seeds 42, 43, 44, 45, 46)
Reported statistics	Mean ± standard deviation
Early stopping	None (fixed 10 epochs)
Hardware	NVIDIA A100 GPU (40 GB), Intel Xeon Gold 6248 CPU, 256 GB RAM
Batch size	4
Epochs	10
Optimizer/LR scheduler	Adam + linear warmup + cosine annealing
Evaluation metrics	Image–Text Cosine Similarity, Image→Image Recall@5, Cross-Modal MRR
Train/Val/Test split	70%:15%:15% (fixed across all runs)
Mixed precision	Disabled

5.2. Configuration and Hyperparameters

To ensure the reproducibility of the training process and the consistency of experimental results, a unified training configuration was adopted across all experiments, which is automatically loaded by the training script at execution. These configurations encompass core elements such as optimizer selection, learning rate scheduling strategies, gradient clipping, mixed-precision control, batch size, and epoch counts. The specific training hyperparameters employed in this study are detailed in Table 3. During training, critical operational data—including loss components for each training/validation epoch, learning rate curves, and router activation statistics—are recorded in real-time and saved as structured logs and checkpoints. This setup ensures that the experimental pipeline and primary results can be replicated given the same code and configuration.

Table 3. Lists the training hyperparameters practically employed in this study for the reference of reviewers and researchers (values are derived directly from the engineering configuration).

Parameter Name	Experimental Value	Description
Batch Size	4	Number of samples per iteration (mini-batch training)
Epochs	10	Total number of training cycles
Initial Learning Rate	1.0 × 10−4	Starting step size for the optimizer
Weight Decay	0.01	L2 regularization coefficient
LR Scheduler	Warmup + Cosine	Cosine annealing schedule with linear warmup
Minimum LR	1.0 × 10−6	Lower bound for the scheduler
Gradient Clipping	1.0	Threshold for gradient norm clipping
Mixed Precision	False	Whether to enable Automatic Mixed Precision (AMP)
Warmup Settings	2/0.1	Number of warmup epochs and initial ratio

Following the specification of training hyperparameters, it is essential to define the specific configurations of the model architecture and routing strategy, as these settings determine the model capacity, fusion mechanism, and inference behavior. The model implemented in this study adopts a unified semantic embedding space and incorporates an MoE-style dynamic routing module following cross-modal alignment. To regulate routing sparsity and expert utilization distribution, relevant routing hyperparameters are specified via configuration files prior to training. The detailed configurations for the model architecture and dynamic routing strategy are summarized in Table 4. During the training process, a load-balancing loss is applied to constrain expert invocation frequencies, thereby preventing expert collapse and enhancing overall robustness.

Table 4. Provides the key parameters for the model architecture and dynamic routing sections as utilized in the experiments.

Parameter Name	Experimental Value	Description
Embedding Dim	512	Unified cross-modal feature dimension (post-projection)
Image Encoder	ViT-Base/16	Pre-trained Vision Transformer (ViT-Base/16) backbone for visual feature extraction
Text Encoder	BERT-Tiny	Pre-trained language model for text encoding
Alignment Heads	8	Number of attention heads for cross-modal alignment
Alignment Layers	4	Number of stacked cross-modal alignment layers
Total Experts	4	Number of expert networks in the routing module
Active Experts	2	Number of experts activated per sample (Top-k)
Router Hidden Dim	512	Intermediate layer dimension of the Router MLP
Dropout Rate	0.1	Unified dropout ratio throughout the network
Temperature	0.07	Temperature parameter in the contrastive loss
Load Balance Weight	0.01	Weight for the routing load-balancing regularization

5.3. Hardware and Engineering Configuration

Hardware Configuration: All experiments were executed on a single NVIDIA A100 GPU (40 GB memory) server with Intel Xeon Gold 6248 CPU (2.5 GHz, 40 cores) and 256 GB RAM. The device parameter in the configuration file automatically selects CUDA if available; otherwise, it falls back to CPU. All training scripts were launched via start_all.sh with the configuration file config.yaml.

6. Applications and System Examples

6.1. Experimental Tasks

This section presents experimental tasks to evaluate the proposed CSA-DR method in inspection scenarios, focusing on multimodal alignment, fusion, and task-adaptive inference. The tasks are as follows:

(1)

Image–Text Similarity Computation

This task measures cross-modal alignment by computing similarity between image and text representations (e.g., cosine similarity), assessing the model’s ability to capture semantic correspondence across modalities.

(2)

Image-to-Image Retrieval

Given a query image, the model retrieves semantically similar images from a database. This evaluates its capability in visual representation, similarity computation, and ranking.

(3)

Cross-Modal Retrieval

This task retrieves images or tables based on text queries, testing the model’s ability to align and transfer semantics across modalities for effective retrieval.

Overall, these tasks evaluate similarity, matching, and retrieval performance, providing a comprehensive validation of CSA-DR’s multimodal fusion and inference capabilities.

6.2. System Example

This section presents deployment examples of an intelligent multimodal retrieval system built on CSA-DR. As shown in Figure 8, the interface supports text-to-image/table and image-to-image/table retrieval for textual, visual, and tabular evidence. In the text-query case, the user input “Substation Operation and Maintenance Center Li,” and the system returned the most relevant images and tables with similarity scores. By retrieving the table titled “Petition and Complaint Registration Form,” it further identified the corresponding interview records associated with Li, demonstrating fine-grained cross-modal evidence linkage.

Figure 8. Regulatory Knowledge Integration and Evidentiary Chain Construction.

Figure 9 shows the image-query interface. After a user uploads a query image, the system extracts visual features and aligns them with the database to retrieve semantically similar images and related tabular records. Compared with the baseline, the optimized CSA-DR retrieval yields higher similarity scores, with image matching accuracy reaching 99.1%, indicating improved cross-modal retrieval precision and stronger associations between visual content and structured evidence.

Figure 9. Illustrates the interface for the image-to-image/table retrieval task.

Overall, the system demonstrates robust cross-modal alignment across text, image, and table modalities, enabling accurate and efficient evidence retrieval in practical scenarios. These results validate the effectiveness of CSA-DR in enhancing multimodal consistency and supporting real-world deployment.

6.3. Evaluation Metrics

To evaluate CSA-DR in inspection and patrol scenarios, we adopt three metrics covering semantic alignment, retrieval accuracy, and ranking quality, as summarized in Figure 10.

Figure 10. Unified evaluation of CSA-DR: (a) image–text similarity distribution, (b) image-to-image retrieval Recall@K, and (c) cross-modal retrieval MRR.

For image–text similarity, cosine similarity is computed between image and text features:

$$\mathrm{C}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}(\mathrm{t},\mathrm{i})={\displaystyle \frac{\mathrm{t}\cdot \mathrm{i}}{\parallel \mathrm{t}{\parallel }_2\parallel \mathrm{i}{\parallel }_2}}$$

(34)

Positive pairs concentrate in high-similarity regions, while negative pairs remain low, indicating effective cross-modal alignment.

For image-to-image retrieval, Recall@K is defined as:

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}@\mathrm{K}={\displaystyle \frac{1}{\mathrm{N}}}\sum _{\mathrm{i}=1}^{\mathrm{N}}\mathrm{I}({\mathrm{g}}_{\mathrm{i}}\in {\mathrm{R}}_{\mathrm{i}}^{\left(5\right)})$$

(35)

Recall@5 evaluates whether the ground-truth image appears in the top-ranked results, reflecting efficient visual retrieval performance.

For cross-modal retrieval, MRR is defined as:

$$\mathrm{M}\mathrm{R}\mathrm{R}={\displaystyle \frac{1}{\mathrm{N}}}\sum _{\mathrm{i}=1}^{\mathrm{N}}{\displaystyle \frac{1}{{\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}}_{\mathrm{i}}}}$$

(36)

Higher MRR indicates better ranking quality. In this study, Text→Image and Text→Table results are averaged for overall evaluation.

Overall, these metrics jointly evaluate CSA-DR in terms of semantic alignment, retrieval accuracy, and ranking quality, forming a compact and comprehensive framework for intelligent inspection and supervision.

In inspection and patrol scenarios, where evidence exists in heterogeneous forms such as text, images, and tables, higher requirements are imposed on accuracy, interpretability, and traceability. To address the limitations of general multimodal models in handling long texts, structured tables, and task-driven modality selection, CSA-DR integrates cross-modal semantic alignment with dynamic routing for unified encoding and adaptive fusion. The proposed three-dimensional evaluation framework—comprising image–text similarity distribution, image-to-image Recall@5, and cross-modal MRR—systematically validates the model’s capability in multimodal understanding and evidence-oriented retrieval, providing a reliable basis for practical deployment.

6.4. Quantitative Comparison, Variability Analysis and Systematic Ablation of Dynamic Routing

To rigorously evaluate the impact of the dynamic routing module, we conducted five independent runs (seeds = 42–46) following the experimental protocol detailed in Section 5.1. Results are reported as mean ± standard deviation across the five runs. Furthermore, we performed a systematic ablation study on the key components of the dynamic routing module, including:

(1)

W/o DR (remove entire routing module);

(2)

W/o Load Balancing (remove KL divergence term);

(3)

W/o Top-k Sparsity (use all 4 experts instead of top-2);

(4)

W/o Residual Scaling (set residual scaling parameter α = 0);

(5)

Different expert numbers (K = 2 vs. K = 4);

(6)

Full CSA-DR (proposed).

The full CSA-DR achieves statistically significant improvements over the w/o DR baseline (paired t-test, p < 0.01 for all three metrics; 95% bootstrap CI: Image–Text Cosine [0.82, 0.84], Image→Image Recall@5 [0.87, 0.89], Cross-Modal MRR [0.85, 0.87]). One-way ANOVA across all ablation variants confirmed overall significant differences (F = 18.7, p < 0.001), with post hoc Tukey tests verifying pairwise superiority of the full model. The standard deviations are consistently small (≤0.02), indicating high reproducibility. As illustrated in Figure 11, the inclusion of the load-balancing term leads to a more uniform expert activation distribution, whereas its removal results in noticeable imbalance across experts. In addition to the dynamic routing ablation, we further evaluate the effectiveness of the placeholder strategy on samples with naturally missing modalities in the dataset. This evaluation encompasses a large-scale ablation on 1000 samples with naturally missing modalities randomly selected from the full Guangdong Power Grid dataset, thereby ensuring high statistical robustness, eliminating selection bias, and fully reflecting realistic inspection scenarios where certain types of evidence (e.g., images or tables) may be unavailable. These results, when viewed together with the main experimental outcomes on the full test set reported in Table 5 (which inherently benefit from the placeholder strategy across all samples, including those with missing modalities), confirm the robustness of CSA-DR’s handling of incomplete multimodal inputs and support reliable inferences for practical deployment (paired t-test p < 0.05; 95% bootstrap CI for CSA-DR (Placeholder): Image–Text Cosine [0.82, 0.84], Image→Image Recall@5 [0.83, 0.85], Cross-Modal MRR [0.85, 0.87]).

Figure 11. Expert activation frequency with and without the load-balancing term (averaged over the test set).

Table 5. Systematic Ablation Study of Dynamic Routing Module (Mean ± Std over 5 runs).

Method	Image–Text Cosine	Image→Image Recall@5	Cross-Modal MRR
Text–Image Bimodal Baseline	0.75 ± 0.02	N/A	0.78 ± 0.01
Text–Table Bimodal Baseline	N/A	N/A	0.76 ± 0.02
CSA-DR w/o DR	0.80 ± 0.01	0.85 ± 0.01	0.82 ± 0.01
CSA-DR w/o Load Balancing	0.81 ± 0.02	0.86 ± 0.02	0.83 ± 0.02
CSA-DR w/o Top-k	0.82 ± 0.01	0.86 ± 0.01	0.84 ± 0.01
CSA-DR w/o Residual Scaling	0.81 ± 0.02	0.85 ± 0.02	0.83 ± 0.02
CSA-DR (K = 4 experts)	0.82 ± 0.01	0.87 ± 0.01	0.85 ± 0.01
CSA-DR (full, proposed)	0.83 ± 0.01	0.88 ± 0.01	0.86 ± 0.01

All improvements over the w/o DR baseline are statistically significant (paired t-test p < 0.01; 95% bootstrap CI reported above for the full model). One-way ANOVA across ablation variants: F = 18.7, p < 0.001 (Tukey’s HSD post hoc confirms pairwise differences).

In addition to the dynamic routing ablation, we further evaluate the effectiveness of the placeholder strategy on samples with naturally missing modalities in the dataset. This evaluation encompasses a large-scale ablation on 1000 samples with naturally missing modalities randomly selected from the full Guangdong Power Grid dataset, thereby ensuring high statistical robustness, eliminating selection bias, and fully reflecting realistic inspection scenarios where certain types of evidence (e.g., images or tables) may be unavailable. Table 6. Comparison of missing-modality handling strategies on the large-scale missing-modality subset (N = 1000). These results, when viewed together with the main experimental outcomes on the full test set reported in Table 5 (which inherently benefit from the placeholder strategy across all samples, including those with missing modalities), confirm the robustness of CSA-DR’s handling of incomplete multimodal inputs and support reliable inferences for practical deployment.

Table 6. Comparison of missing-modality handling strategies on the large-scale missing-modality subset (N = 1000).

Method	Image–Text Cosine	Image→Image Recall@5	Cross-Modal MRR
Modality Dropping	0.78	0.81	0.79
CSA-DR (Placeholder)	0.83	0.84	0.86

Improvements are statistically significant (paired t-test, p < 0.05; 95% bootstrap CI: Image–Text Cosine [0.82, 0.84], Image→Image Recall@5 [0.83, 0.85], Cross-Modal MRR [0.85, 0.87]).

As shown in Table 6, the proposed placeholder strategy consistently outperforms modality dropping across all three metrics, delivering relative improvements of +5.3% in Image–Text Cosine similarity, +3.3% in Image→Image Recall@5, and +6.9% in Cross-Modal MRR. More importantly, by using the modality-specific learnable placeholder vectors described in Section 3.2, it preserves a consistent tri-modal input structure, which simplifies the downstream processing pipeline and ensures stable behavior under incomplete inputs. This indicates that, even when performance gains are moderate, maintaining structural consistency through placeholder embeddings brings measurable benefits for practical inspection scenarios.

6.5. Traceable Path Display Accuracy Evaluation

To evaluate the accuracy of the traceable path display, we conducted a human-in-the-loop experiment on 100 randomly selected inspection samples. For each sample, the attended evidence highlighted by CSA-DR was compared with human-verified ground truth violations.

The evaluation metric is the matching rate, defined as the proportion of attended evidence items correctly aligned with the human-verified violations. Table 7 summarizes the results:

Table 7. Human-in-the-Loop Evaluation of CSA-DR Traceable Path Display.

Sample Type	Number of Samples	Correctly Matched	Matching Rate (%)
All Samples	100	87	87%

The results indicate that 87% of the attended evidence items correctly correspond to the human-verified violations, demonstrating that the traceable path display is largely reliable and interpretable. This evaluation provides quantitative support for the interpretability and auditability of CSA-DR outputs in inspection scenarios.

6.6. Computational Analysis

The computational overhead of CSA-DR mainly comes from three components: tri-modal encoding, cross-modal semantic alignment, and dynamic routing. First, the model processes text, tables, and images with modality-specific encoders, which increases inference cost compared with single-modality systems. Second, the cross-modal alignment module introduces additional attention computation across modality pairs, which is necessary to establish a unified semantic space and improve evidence consistency. Third, the routing module adds expert selection and weighted fusion operations, which further increase model complexity but also improve task adaptivity and interpretability.

Although this version does not provide exact FLOPs or latency measurements, the architecture is designed to remain deployment-friendly. The routing module is lightweight and can be configured with a small number of active experts, so the computational burden can be controlled through top-k selection, expert width, and alignment depth. In practical deployment, the retrieval stage can also be precomputed or cached offline, which reduces online latency. Therefore, CSA-DR is suitable for intelligent inspection scenarios where accuracy, traceability, and moderate inference efficiency are simultaneously required.

For L1/L2-level intelligent assistants, real-time responsiveness is an important deployment consideration. Future work will include systematic benchmarking of FLOPs, inference latency, memory consumption, and throughput under different hardware settings to further quantify the deployment efficiency of the proposed framework.

6.7. Failure Case Analysis

To provide a systematic study of failure cases as requested by the reviewer, we analyze 200 misclassified samples randomly selected from the test set, averaged over five independent runs. The overall failure rate of CSA-DR (full) is 12.4%. We further categorize the failures by violation type, following Table 1, and inspect the corresponding routing behavior together with the associated cross-modal limitations. This analysis helps characterize when the router assigns suboptimal expert weights and how such routing decisions relate to downstream errors. The quantitative breakdown of failure rates and their primary causes across different violation categories is summarized in Table 8.

Table 8. Failure Rate and Primary Failure Cause by Violation Category (Mean ± Std over 5 Runs).

Violation Category	Failure Rate (%)	Primary Failure Cause
Safety Equipment Compliance Violations	9.2 ± 1.1	Low-quality visual evidence weakens image–table alignment, reducing the router’s ability to exploit complementary signals
Operational Procedure Violations	11.8 ± 1.4	Text-dominant inputs over-activate the textual expert, diminishing the contribution of structured evidence
Financial/Rectification Record Mismatches	18.7 ± 2.3	Sparse tabular entries are under-emphasized, and placeholder embeddings cannot fully recover missing numerical context
Policy Regulatory Clause Violations	14.6 ± 1.7	Long regulatory text introduces noise, biasing routing toward visually similar evidence instead of precise clause matching

(1)

Financial Mismatch (Table-dominant):

The input contains a rectification ledger with multiple missing values and a low-resolution infrared image. The router assigns higher weights to the textual and visual branches, but the sparse numerical cells in the table remain under-represented. As a result, the model fails to detect the mismatch. This suggests that the placeholder embedding can preserve the input structure, but it only partially compensates for missing numerical context when the table evidence is heavily incomplete.

(2)

Policy Clause Violation (Text-heavy):

A long regulatory passage is paired with visually similar images. In this case, the routing module over-emphasizes the visual branch, which leads to over-reliance on superficial visual similarity rather than precise textual clause matching. This indicates that long-text noise can distort the routing decision and weaken clause-level reasoning.

(3)

Safety Violation (Multi-modal conflict):

The image and table provide conflicting evidence, while the text offers only partial support. Although the dynamic router attempts to combine these modalities, the model still fails to resolve the conflict effectively. This suggests that the current cross-modal alignment module is not always sufficient when modality signals are inconsistent or of low quality.

Overall, these failure cases show that although dynamic routing significantly improves average performance, rare or noisy modality combinations, especially sparse tables, long texts, and conflicting evidence, remain challenging. The interpretability analysis of routing weights further indicates that failures are often associated with over-confident activation of a single expert and insufficient balance across modalities. These findings suggest that future improvements may come from adaptive expert capacity, stronger conflict-aware fusion, and memory-augmented routing.

7. Discussion

7.1. Data Privacy and Security Considerations

Although CSA-DR improves evidence grounding and traceability by retrieving policy documents, structured tables, and visual evidence from an external knowledge base, real-world inspection and supervision scenarios often involve sensitive organizational information. Accordingly, data privacy should be carefully considered throughout data collection, knowledge retrieval, model training, and inference. At present, CSA-DR does not explicitly incorporate privacy-preserving mechanisms in training or deployment. As discussed in recent studies on privacy-preserving data sharing technologies [23], techniques such as differential privacy, federated learning, homomorphic encryption, and secure multi-party computation provide effective solutions to mitigate data leakage risks while maintaining analytical utility. These methods are particularly relevant for collaborative data sharing and secure computation in privacy-sensitive environments, and they also provide a natural foundation for addressing the remaining practical challenges discussed below.

7.2. Limitations and Challenges

CSA-DR targets inspection and supervision scenarios by addressing multimodal understanding and reasoning across images, text, and tables. Through cross-modal alignment and dynamic routing, it improves data fusion and task-adaptive inference, achieving strong performance in tasks such as similarity evaluation, issue detection, and evidence traceability. In addition, the model incorporates an external regulatory knowledge base, where relevant policy documents, structured tables, and evidence images are retrieved and used as the tri-modal inputs.

The model aligns heterogeneous data into a unified semantic space via multi-head cross-attention, while dynamic routing adaptively weights modalities to focus on critical evidence, enhancing accuracy and interpretability. This supports the requirement of “traceable and verifiable” conclusions in inspection tasks.

However, limitations remain, including weaker performance on complex long texts and hierarchical tables, reliance on large annotated datasets, and increased computational cost from dynamic routing.

Overall, CSA-DR provides an effective approach for intelligent inspection and supervision, though further optimization in efficiency, complex data handling, and deployment is needed.