Defect Identification and Diagnosis for Distribution Network Electrical Equipment Based on Fused Image and Voiceprint Joint Perception

Abstract

As the scale of distribution networks expand, existing defect identification methods face numerous challenges, including limitations in single-modal feature identification, insufficient cross-modal information fusion, and the lack of a multi-stage feedback mechanism. To address these issues, we first propose a joint perception of image and voiceprint features based on bidirectional coupled attention, which enhances deep interaction across modalities and overcomes the shortcomings of traditional methods in cross-modal fusion. Secondly, a defect identification and diagnosis method of distribution network electrical equipment based on two-stage convolutional neural networks (CNN) is introduced, which makes the network pay more attention to typical and frequent defects, and enhances defect diagnosis accuracy and robustness. The proposed algorithm is compared with two baseline algorithms. Baseline 1 is a long short term memory (LSTM)-based algorithm that performs separate feature extraction and processing for image and voiceprint signals without coupling the features of the two modalities, and Baseline 2 is a traditional CNN algorithm that uses classical convolutional layers for feature learning and classification through pooling and fully connected layers. Compared with two baselines, simulation results demonstrate that the proposed method improves accuracy by 12.1% and 33.7%, recall by 12.5% and 33.1%, and diagnosis efficiency by 22.92% and 60.42%.

1. Introduction

Accurate and efficient defect identification and diagnosis of electrical equipment in distribution networks is the foundation for handling distribution network accidents, rapidly restoring power, and maintaining reliable operation [1]. With the advancement of modern power systems, the increasing deployment of power electronic devices, along with the growing proportion of high-voltage direct current (DC) systems and renewable energy, the scale and structure of distribution networks becomes increasingly complex. Moreover, factors such as adverse weather and pollution have increased the risk of faults, leading to frequent power outages and equipment damage, which severely impacts the stability of the power system [2,3,4]. Certain critical faults, such as high impedance faults (HIFs), are difficult to detect because they generate low fault currents and their arc characteristics resemble those of normal operations (such as load switch operations). These factors demand higher standards for defect identification and diagnosis of electrical equipment in distribution networks. By monitoring equipment images (such as ultraviolet, infrared, visible light, etc., which reflect external conditions such as cracks, corrosion, etc., [5]) or voiceprints (which reflect abnormal noise during operation [6]), the working state of equipment can be identified, allowing for the timely detection and resolution of potential defects, thereby effectively avoiding faults. However, due to the complex and dynamic environment of distribution networks, including strong electromagnetic interference, equipment vibration, environmental noise, and other factors, single-modal defect identification technologies face many challenges. For example, image feature identification has a high misjudgment rate under low light or strong light reflection, while voiceprint identification suffers from decreased accuracy under complex noise interference. Additionally, different equipment states and fault modes depend on single-modal features to different extents. For instance, in a humid environment, partial discharge activity increases, and voiceprint analysis of partial discharge signals is more effective. In contrast, defects such as surface cracks or corrosion are more reliant on image feature identification. Therefore, how to integrate image and voiceprint features under complex conditions, fully utilizing their complementarity, and establish a full-stage identification and diagnosis mechanism for comprehensive and precise monitoring of distribution network equipment defects, remains a key technical challenge in the field of electrical equipment defect identification in distribution networks.

By extracting core information from electrical equipment images and voiceprint features, the accuracy and robustness of defect identification can be improved. Researchers have conducted extensive research. A new method for power equipment fault diagnosis based on infrared thermal imaging was proposed [7]. The you only look once version 5 (YOLOv5) convolutional neural network is adopted and enhanced to improve the efficiency and accuracy of defect diagnosis. In [8], Kong et al. developed a logistic membership function-based algorithm for infrared image enhancement. This method reduces thermal image noise and improves contrast through blur processing, enabling efficient batch processing and power equipment fault diagnosis. A data augmentation method based on a cycle consistent generative adversarial network (CycleGAN) for visible light image features was proposed [9], which converts easily obtainable visible light images into difficult-to-obtain infrared images, achieving the expansion of a power equipment infrared image pool. A transformer fault voiceprint identification model based on Mel frequency domain coefficients (MFCC) and deep learning was proposed [10]. This model greatly enhanced the accuracy of fault identification for transformer acoustic signals in a low signal-to-noise ratio environment. In [11], Satea et al. introduced a method for detecting high impedance faults, which combines third harmonic current/voltage amplitude variations with wavelet analysis. This approach enhances detection reliability and significantly reduces the risk of mistaking normal events for faults. Although the above studies have made some progress, they mostly focus on single-modal data and fail to fully utilize the fusion of image and voiceprint features. This limitation impedes accurate physical state capture in electrical equipment, degrading fault identification performance. Meanwhile, feature fusion based on attention mechanisms has shown excellent potential in fault diagnostics and analysis of multi-source heterogeneous data. It has become an important development direction for defect identification and diagnosis. A fault identification and diagnosis method for unmanned aerial vehicles (UAVs) was proposed in [12]. Through intelligent analysis and real-time monitoring of flight images and voiceprint data, efficient fault diagnosis of UAV flight status is achieved. However, these methods overlook horizontal–vertical attention coupling. This limitation prevents effective capture of image–voiceprint interactive features, leading to slow convergence and low accuracy. Consequently, distribution network equipment defects lack rapid, precise detection and diagnosis.

Artificial intelligence (AI) is increasingly being applied across various fields, providing creative solutions to tackle complex problems. For example, in the healthcare industry [13], Kh-Madhloom et al. introduced a multi-layer encryption system combining deoxyribonucleic acid (DNA) computing and advanced encryption standard (AES) algorithms to improve the security of patient healthcare images, showcasing the potential of AI in safeguarding sensitive medical data. In video compression [14], Hassen et al. developed an optimal video frame compression model based on a quantum genetic algorithm, utilizing quantum machine learning techniques to optimize video multicast efficiency over the internet.

Similarly, in the field of defect identification and diagnosis for electrical equipment in distribution networks, the application of AI technology, particularly convolutional neural networks (CNN), is crucial. CNN-based techniques, such as the fusion of image and voiceprint features, enable automatic defect identification and diagnosis in electrical equipment. This approach significantly enhances the efficiency and safety of equipment maintenance, reduces operational risks, and helps lower maintenance costs by leveraging the powerful capabilities of deep learning in image and audio data analysis. In [15], Qiu et al. fused CNN-extracted image features via adaptive weights for equipment state assessment, effectively reducing maintenance costs. In [16], Xing et al. combined 1D-CNN-processed gas data and residual-compressed infrared imaging with cross-modal attention, achieving precise transformer health evaluation. MFCC combined with CNN were used to effectively diagnose abnormal states in transformers, achieving an accuracy of 98% under low noise conditions [17]. A multi-source data fusion algorithm, integrating multi-source feature extraction and a multi-scale CNN fusion network, was introduced to simultaneously process image and voiceprint data, ensuring precise equipment failure monitoring [18]. However, existing methods typically employ single-stage CNN networks, lacking multi-stage adaptive adjustment mechanisms, and cannot dynamically update attention weights based on temporal features, which limits the model’s focus on typical and frequent defects. Additionally, existing methods do not incorporate closed-loop feedback mechanisms, making it difficult to achieve precise diagnosis in complex environments, leading to insufficient diagnostic accuracy and robustness.

Despite progress in existing research, defect identification and diagnosis of electrical equipment in distribution networks still face the following challenges. Firstly, traditional defect identification and diagnosis methods rely only on single-modal data, failing to fully extract the rich information contained in different modalities and neglecting cross-modal interactions and correlations. This leads to low identification and diagnosis accuracy and increased risks of misdetection or missed detection. Secondly, existing defect identification and diagnosis methods based on image and voiceprint fusion do not consider the bidirectional coupling of horizontal and vertical attention. This results in insufficient fine control of modality interactions, limiting the effective transmission of cross-modal information and further affecting identification and diagnosis efficiency. Lastly, current defect identification and diagnosis methods often use single CNN networks, lacking adaptive adjustments and closed-loop feedback mechanisms in multi-stage networks, and neglecting temporal features. This leads to low identification accuracy, poor stability, and insufficient intelligent early warning capabilities, making it unsuitable for electrical equipment defect identification scenarios involving image and voiceprint data fusion.

In response to these challenges, we propose a joint perception method for image and voiceprint features based on bidirectional coupled attention to improve the efficiency of defect identification and diagnosis. Secondly, a two-stage CNN based method for identifying and diagnosing defects in distribution network electrical equipment is proposed to improve the accuracy and robustness of diagnosis.

The main contributions of this paper are as follows:

• Joint Perception of Image and Voiceprint Features based on Bidirectional Coupled Attention: Firstly, an initial feature extraction of image and voiceprint data is performed using an LSTM network. Then, the features from the two modalities are concatenated and coupled through affine transformation. Subsequently, bidirectional coupled attention is constructed for both image and voiceprint modalities, including vertical self-attention and horizontal coupled attention. Vertical self-attention uses a multi-head attention mechanism to adapt to more complex mapping relationships. Horizontal coupled attention integrates information from other modalities when computing attention scores, enabling deep interaction across modalities. This prevents the inefficiency of relying solely on single-modal information and bridges the semantic gap between modalities, improving the efficiency of defect identification and diagnosis.
• Defect Identification and Diagnosis of Distribution Network Electrical Equipment based on Two-stage CNN: Firstly, a preliminary defect identification method based on a temporal-adaptive CNN is proposed, which analyzes electrical equipment feature data and adapts the attention weights based on temporal features, such as the time intervals and correlations of the data. This improves the network’s focus on typical and frequently occurring defects, enhancing the adaptability of the preliminary identification stage. Secondly, a precise diagnostic method based on closed-loop CNN is introduced. This method adjusts the diagnostic network using closed-loop feedback based on actual defect conditions and preliminary identification results, achieving refined diagnosis in complex environments, thus significantly improving diagnostic accuracy and robustness.

2. Joint Perception of Image and Voiceprint Features Based on Bidirectional Coupled Attention

A joint perception of image and voiceprint features based on bidirectional coupled attention is proposed, as shown in Figure 1. On the basis of traditional feature extraction research, waveform and temporal features of image and voiceprint information are obtained. Affine transformation is used to combine the two types of features of image and voiceprint to obtain feature vectors containing waveform and temporal information. To ensure there is no conflict and to guarantee convergence when two sets of inputs feed into the interaction information, we have implemented a comprehensive approach that includes affine transformation, concatenation, and multi-head attention mechanisms. In the subsequent defect identification and diagnosis of electrical equipment, temporal information is used to improve model performance.

2.1. Extraction and Construction of Image and Voiceprint Features

Firstly, based on LSTM network, initial features of defects in distribution network electrical equipment contained in images and voiceprint signals are extracted. Assuming the time slot set is $\mathcal{T}=\{1,\dots ,t,\dots T\}$, where $t$ is the slot index and $T$ is the total number of slots. The process of extracting image signal features can be expressed as

$$\left\{\begin{array}{l}{V}_{\mathrm{I}}\left(t\right)=sigmoid\left({\eta }_V^{\mathrm{I}}\left[S_{\mathrm{I}}\left(t-1\right),I{N}_{\mathrm{I}}\left(t\right)\right]+{\lambda }_V^{\mathrm{I}}\right)\\ J_{\mathrm{I}}\left(t\right)=sigmoid\left({\eta }_J^{\mathrm{I}}\left[S_{\mathrm{I}}\left(t-1\right),I{N}_{\mathrm{I}}\left(t\right)\right]+{\lambda }_J^{\mathrm{I}}\right)\\ O_{\mathrm{I}}\left(t\right)=sigmoid\left({\eta }_O^{\mathrm{I}}\left[S_{\mathrm{I}}\left(t-1\right),I{N}_{\mathrm{I}}\left(t\right)\right]+{\lambda }_O^{\mathrm{I}}\right)\\ Q_{\mathrm{I}}\left(t\right)=V_{\mathrm{I}}\left(t\right)Q_{\mathrm{I}}\left(t-1\right)+Q_{\mathrm{I}}\left(t\right)\tanh \left({\eta }_Q^{\mathrm{I}}\left[S_{\mathrm{I}}\left(t-1\right),I{N}_{\mathrm{I}}\left(t\right)\right]+{\lambda }_Q^{\mathrm{I}}\right)\\ S_{\mathrm{I}}\left(t\right)=O_{\mathrm{I}}\left(t\right)\tanh \left(Q_{\mathrm{I}}\left(t\right)\right)\end{array}\right.$$

(1)

where: $V_{\mathrm{I}}\left(t\right)$, $J_{\mathrm{I}}\left(t\right)$, and $O_{\mathrm{I}}\left(t\right)$ represent the state information of the forget gate, input gate, and output gate of the image signal feature extraction LSTM network, respectively. ${\eta }_V^{\mathrm{I}}$, ${\eta }_J^{\mathrm{I}}$, ${\eta }_O^{\mathrm{I}}$, and ${\eta }_Q^{\mathrm{I}}$ represent the weight parameters of the forget gate, input gate, output gate, and candidate cell of the image signal feature extraction LSTM network, respectively. ${\lambda }_V^{\mathrm{I}}$, ${\lambda }_J^{\mathrm{I}}$, ${\lambda }_O^{\mathrm{I}}$, and ${\lambda }_Q^{\mathrm{I}}$ are the biases, respectively. $I{N}_{\mathrm{I}}\left(t\right)$ denotes the input to the image signal feature extraction LSTM network, i.e., the image signal input. $S_{\mathrm{I}}\left(t-1\right)$ is the hidden layer output of the image signal feature extraction LSTM network in slot $t-1$. $Q_{\mathrm{I}}\left(t\right)$ is the candidate cell state of the image signal feature extraction LSTM network in time slot $t$. $sigmoid$ and $\tanh$ are the activation functions used to update the state information of the forget gate, input gate, and output gate, as well as to create the new state information for the candidate cell.

The process of extracting voiceprint signal features can be expressed as

$$\left\{\begin{array}{l}{V}_{\mathrm{A}}\left(t\right)=sigmoid\left({\eta }_V^{\mathrm{A}}\left[S_{\mathrm{A}}\left(t-1\right),I{N}_{\mathrm{A}}\left(t\right)\right]+{\lambda }_V^{\mathrm{A}}\right)\\ J_{\mathrm{A}}\left(t\right)=sigmoid\left({\eta }_J^{\mathrm{A}}\left[S_{\mathrm{A}}\left(t-1\right),I{N}_{\mathrm{A}}\left(t\right)\right]+{\lambda }_J^{\mathrm{A}}\right)\\ O_{\mathrm{A}}\left(t\right)=sigmoid\left({\eta }_O^{\mathrm{A}}\left[S_{\mathrm{A}}\left(t-1\right),I{N}_{\mathrm{A}}\left(t\right)\right]+{\lambda }_O^{\mathrm{A}}\right)\\ Q_{\mathrm{A}}\left(t\right)=V_{\mathrm{A}}\left(t\right)Q_{\mathrm{A}}\left(t-1\right)+Q_{\mathrm{A}}\left(t\right)\tanh \left({\eta }_Q^{\mathrm{A}}\left[S_{\mathrm{A}}\left(t-1\right),I{N}_{\mathrm{A}}\left(t\right)\right]+{\lambda }_Q^{\mathrm{A}}\right)\\ S_{\mathrm{A}}\left(t\right)=O_{\mathrm{A}}\left(t\right)\tanh \left(Q_{\mathrm{A}}\left(t\right)\right)\end{array}\right.$$

(2)

where: $V_{\mathrm{A}}\left(t\right)$, $J_{\mathrm{A}}\left(t\right)$, and $O_{\mathrm{A}}\left(t\right)$ represent the state information of the forget gate, input gate, and output gate of the voiceprint signal feature extraction LSTM network, respectively. ${\eta }_V^{\mathrm{A}}$, ${\eta }_J^{\mathrm{A}}$, ${\eta }_O^{\mathrm{A}}$, and ${\eta }_Q^{\mathrm{A}}$ represent the weight parameters of the forget gate, input gate, output gate, and candidate cell of the voiceprint signal feature extraction LSTM network, respectively. ${\lambda }_V^{\mathrm{A}}$, ${\lambda }_J^{\mathrm{A}}$, ${\lambda }_O^{\mathrm{A}}$, and ${\lambda }_Q^{\mathrm{A}}$ are the biases of the forget gate, input gate, output gate, and candidate cell of the voiceprint signal feature extraction LSTM network. $I{N}_{\mathrm{A}}\left(t\right)$ is the input to the voiceprint signal feature extraction LSTM network. $S_{\mathrm{A}}\left(t-1\right)$ denotes the output of the hidden layer of the voiceprint signal feature extraction LSTM network in slot $t-1$. $Q_{\mathrm{A}}\left(t-1\right)$ is the candidate cell state of the voiceprint signal feature extraction LSTM network in time slot $t-1$.

2.2. Construction of Bidirectional Coupled Attention

Consider that the initial defect features of the distribution network electrical equipment extracted through the image signal feature extraction LSTM network include waveform feature ${\mathit{N}}_{\mathrm{I}}\left(t\right)$ and temporal feature ${\mathit{\tau }}_{\mathrm{I}}\left(t\right)$. The initial defect features extracted through the voiceprint signal feature extraction LSTM network include waveform feature ${\mathit{N}}_{\mathrm{A}}\left(t\right)$ and temporal feature ${\mathit{\tau }}_{\mathrm{I}}\left(t\right)$. These waveform and temporal features from both the image and voiceprint modalities are then concatenated and coupled through affine transformation.

First, the temporal feature vectors of the two modalities are concatenated into a matrix, which is then multiplied by a weight vector $\mathit{\theta }$ and fused into a multimodal offset $\mathit{b}\left(t\right)$, expressed as

$$\mathit{b}\left(t\right)=\left[{\mathit{\tau }}_{\mathrm{I}}\left(t\right),{\mathit{\tau }}_{\mathrm{A}}\left(t\right)\right] \mathit{\theta }$$

(3)

Then, $\mathit{b}\left(t\right)$ is used to provide a unified offset for each modality to perform affine transformation. Then, each of the modal representations containing temporal information are obtained, expressed as

$$\left\{\begin{array}{l}{\mathit{Q}}_{\mathrm{I}}\left(t\right)={\mathit{W}}_{\mathrm{I}}^{\mathrm{Q}}{\mathit{N}}_{\mathrm{I}}\left(t\right)+\mathit{b}\left(t\right)\\ {\mathit{K}}_{\mathrm{I}}\left(t\right)={\mathit{W}}_{\mathrm{I}}^{\mathrm{K}}{\mathit{N}}_{\mathrm{I}}\left(t\right)+\mathit{b}\left(t\right)\\ {\mathit{V}}_{\mathrm{I}}\left(t\right)={\mathit{W}}_{\mathrm{I}}^{\mathrm{V}}{\mathit{N}}_{\mathrm{I}}\left(t\right)+\mathit{b}\left(t\right)\end{array}\right.$$

(4)

$$\left\{\begin{array}{l}{\mathit{Q}}_{\mathrm{A}}\left(t\right)={\mathit{W}}_{\mathrm{A}}^{\mathrm{Q}}{\mathit{N}}_{\mathrm{A}}\left(t\right)+\mathit{b}\left(t\right)\\ {\mathit{K}}_{\mathrm{A}}\left(t\right)={\mathit{W}}_{\mathrm{A}}^{\mathrm{K}}{\mathit{N}}_{\mathrm{A}}\left(t\right)+\mathit{b}\left(t\right)\\ {\mathit{V}}_{\mathrm{A}}\left(t\right)={\mathit{W}}_{\mathrm{A}}^{\mathrm{V}}{\mathit{N}}_{\mathrm{A}}\left(t\right)+\mathit{b}\left(t\right)\end{array}\right.$$

(5)

where: ${\mathit{Q}}_{\mathrm{I}}\left(t\right)$, ${\mathit{K}}_{\mathrm{I}}\left(t\right)$, and ${\mathit{V}}_{\mathrm{I}}\left(t\right)$ denote the query vector, key vector, and value vector of image modality in time slot $t$, respectively. ${\mathit{Q}}_{\mathrm{A}}\left(t\right)$, ${\mathit{K}}_{\mathrm{A}}\left(t\right)$, and ${\mathit{V}}_{\mathrm{A}}\left(t\right)$ represent the query vector, key vector, and value vector of voiceprint modality in time slot $t$, respectively. ${\mathit{W}}_{\mathrm{I}}^{\mathrm{Q}}$, ${\mathit{W}}_{\mathrm{I}}^{\mathrm{K}}$, and ${\mathit{W}}_{\mathrm{I}}^{\mathrm{V}}$ are the three weight matrices of the image modality, respectively. ${\mathit{W}}_{\mathrm{A}}^{\mathrm{Q}}$, ${\mathit{W}}_{\mathrm{A}}^{\mathrm{K}}$, and ${\mathit{W}}_{\mathrm{A}}^{\mathrm{V}}$ are the three weight matrices of the voiceprint modality, respectively.

Next, bidirectional coupled attention is constructed separately for the image and voiceprint modalities, including vertical self-attention and horizontal coupled attention.

Vertical self-attention adopts a multi-head attention mechanism to accommodate more complex mapping relationships. The attention scores between the query vector and the key vector are computed using the dot product function and the softmax function. These scores are used to obtain the weighted sum of the value vectors, resulting in the modality representation after the self-attention calculation, expressed as

$$Attention\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)=\mathrm{softmax}\left(\left({\mathit{Q}}_{\mathrm{I}}\left(t\right){\mathit{K}}_{\mathrm{I}}^{\mathrm{T}}\left(t\right)\right)/\sqrt{d_{\mathrm{I}}}\right){\mathit{W}}^{\mathrm{o}}$$

(6)

$$Attention\left({\mathit{Q}}_{\mathrm{A}}\left(t\right),{\mathit{K}}_{\mathrm{A}}\left(t\right),{\mathit{V}}_{\mathrm{A}}\left(t\right)\right)=\mathrm{softmax}\left(\left({\mathit{Q}}_{\mathrm{A}}\left(t\right){\mathit{K}}_{\mathrm{A}}^{\mathrm{T}}\left(t\right)\right)/\sqrt{d_{\mathrm{A}}}\right){\mathit{W}}^{\mathrm{o}}$$

(7)

where: $Attention\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)$ is the result of the self-attention operation of the image modality in time slot $t$. $d_{\mathrm{I}}$ is the output dimension of the vertical self-attention in the image modality. $d_{\mathrm{A}}$ is the output dimension of the vertical self-attention in the voiceprint modality. ${\mathit{W}}^{\mathrm{o}}$ is the linear transformation of the output.

The above operations are repeated $h$ times, with each iteration considered a head. The results of these $h$ times are concatenated and linearly transformed to obtain the result of multi-head attention. To prevent the inner product from becoming too large, the computed similarity weight is typically divided by the key vector’s dimension, with each head having distinct linear transformation parameters. The specific process is expressed as

$$H_{hea{d}_j}^{\mathrm{I}}\left(t\right)=Attention\left({\mathit{Q}}_{\mathrm{I}}\left(t\right){\mathit{W}}_j^{\mathrm{Q}},{\mathit{K}}_{\mathrm{I}}\left(t\right){\mathit{W}}_j^{\mathrm{K}},{\mathit{V}}_{\mathrm{I}}\left(t\right){\mathit{W}}_j^{\mathrm{V}}\right)$$

(8)

$$H_{hea{d}_j}^{\mathrm{A}}\left(t\right)=Attention\left({\mathit{Q}}_{\mathrm{A}}\left(t\right){\mathit{W}}_j^{\mathrm{Q}},{\mathit{K}}_{\mathrm{A}}\left(t\right){\mathit{W}}_j^{\mathrm{K}},{\mathit{V}}_{\mathrm{A}}\left(t\right){\mathit{W}}_j^{\mathrm{V}}\right)$$

(9)

$$MHA\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)=Concat\left(H_{hea{d}_1}^{\mathrm{I}},\cdots ,H_{hea{d}_j}^{\mathrm{I}},\cdots ,H_{hea{d}_h}^{\mathrm{I}}\right){\mathit{W}}^{\mathrm{o}}$$

(10)

$$MHA\left({\mathit{Q}}_{\mathrm{A}}\left(t\right),{\mathit{K}}_{\mathrm{A}}\left(t\right),{\mathit{V}}_{\mathrm{A}}\left(t\right)\right)=Concat\left(H_{hea{d}_j}^{\mathrm{A}},\cdots ,H_{hea{d}_j}^{\mathrm{A}},\cdots ,H_{hea{d}_j}^{\mathrm{A}}\right){\mathit{W}}^{\mathrm{o}}$$

(11)

where: $H_{hea{d}_j}^{\mathrm{I}}\left(t\right)$ is the head obtained from the image modality’s vertical self-attention operation at the $j$-th step in time slot $t$. $H_{hea{d}_j}^{\mathrm{A}}\left(t\right)$ is the head obtained from the voiceprint modality’s vertical self-attention operation at the $j$-th step in time slot $t$. ${\mathit{W}}_j^{\mathrm{Q}}$, ${\mathit{W}}_j^{\mathrm{K}}$, and ${\mathit{W}}_j^{\mathrm{V}}$ represent the weight matrices for the query vector, key vector, and value vector at the $j$-th operation. $MHA\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)$ represents the vertical self-attention of the image modality in time slot $t$. $MHA\left({\mathit{Q}}_{\mathrm{A}}\left(t\right),{\mathit{K}}_{\mathrm{A}}\left(t\right),{\mathit{V}}_{\mathrm{A}}\left(t\right)\right)$ represents the vertical self-attention of the voiceprint modality in time slot $t$. $Concat\left(\cdot \right)$ indicates the concatenation of features.

2.3. Cross-Modal Deep Interaction

Horizontal coupled attention introduces information from other modalities when calculating the attention score. Through cross-modal information interaction, bridging the semantic gap between modalities, the efficiency of defect identification and diagnosis is improved. The horizontal coupled attention of the image modality calculates the interactive attention score of the image modality and the voiceprint modality, which is similar to the vertical self-attention calculation process of the image modality, but the query vector comes from the voiceprint modality. The query vector of the voiceprint modality is used to guide the multi-head attention of the image modality, so that the final obtained modal representation can contain the interaction information between the modalities, which is specifically expressed as

$$CrossAttention\left({\mathit{Q}}_{\mathrm{A}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)=\mathrm{softmax}\left(\left({\mathit{Q}}_{\mathrm{A}}\left(t\right){\mathit{K}}_{\mathrm{I}}^{\mathrm{T}}\left(t\right)\right)/\sqrt{d_{\mathrm{I}}}\right){\mathit{W}}^{\mathrm{o}}$$

(12)

$$H_{hea{d}_j}^{\mathrm{I},\mathrm{Cross}}\left(t\right)=CrossAttention\left({\mathit{Q}}_{\mathrm{A}}\left(t\right){\mathit{W}}_j^{\mathrm{Q}},{\mathit{K}}_{\mathrm{I}}\left(t\right){\mathit{W}}_j^{\mathrm{K}},{\mathit{V}}_{\mathrm{I}}\left(t\right){\mathit{W}}_j^{\mathrm{V}}\right)$$

(13)

$$CrossMHA\left({\mathit{Q}}_{\mathrm{A}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)=Concat\left(H_{hea{d}_1}^{\mathrm{I},\mathrm{Cross}},\cdots ,H_{hea{d}_j}^{\mathrm{I},\mathrm{Cross}},\cdots ,H_{hea{d}_h}^{\mathrm{I},\mathrm{Cross}}\right){\mathit{W}}^{\mathrm{o}}$$

(14)

where: $CrossAttention\left({\mathit{Q}}_{\mathrm{A}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)$ denotes the result of the interaction attention operation of the image modality. $H_{hea{d}_j}^{\mathrm{I},\mathrm{Cross}}\left(t\right)$ represents the head obtained from the image modality’s horizontal coupled attention operation at the $j$-th step in time slot $t$. $CrossMHA\left({\mathit{Q}}_{\mathrm{A}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)$ represents the horizontal coupled self-attention for the image modality in time slot $t$.

Similarly, the horizontal coupled attention computation process for the voiceprint modality is expressed as

$$CrossAttention\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{A}}\left(t\right),{\mathit{V}}_{\mathrm{A}}\left(t\right)\right)=\mathrm{softmax}\left(\left({\mathit{Q}}_{\mathrm{I}}\left(t\right){\mathit{K}}_{\mathrm{A}}^{\mathrm{T}}\left(t\right)\right)/\sqrt{d_{\mathrm{A}}}\right){\mathit{W}}^{\mathrm{o}}$$

(15)

$$H_{hea{d}_j}^{\mathrm{A},\mathrm{Cross}}\left(t\right)=CrossAttention\left({\mathit{Q}}_{\mathrm{I}}\left(t\right){\mathit{W}}_j^{\mathrm{Q}},{\mathit{K}}_{\mathrm{A}}\left(t\right){\mathit{W}}_j^{\mathrm{K}},{\mathit{V}}_{\mathrm{A}}\left(t\right){\mathit{W}}_j^{\mathrm{V}}\right)$$

(16)

$$CrossMHA\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{A}}\left(t\right),{\mathit{V}}_{\mathrm{A}}\left(t\right)\right)=Concat\left(H_{hea{d}_1}^{\mathrm{A},\mathrm{Cross}},\cdots ,H_{hea{d}_j}^{\mathrm{A},\mathrm{Cross}},\cdots ,H_{hea{d}_h}^{\mathrm{A},\mathrm{Cross}}\right){\mathit{W}}^{\mathrm{o}}$$

(17)

where: $CrossAttention\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{A}}\left(t\right),{\mathit{V}}_{\mathrm{A}}\left(t\right)\right)$ represents the result of the interaction attention operation for the image modality in time slot $t$. $H_{hea{d}_j}^{\mathrm{A},\mathrm{Cross}}\left(t\right)$ represents the head obtained from the image modality’s horizontal coupled attention operation at the $j$-th step in time slot $t$. $CrossMHA\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{A}}\left(t\right),{\mathit{V}}_{\mathrm{A}}\left(t\right)\right)$ represents the horizontal coupled self-attention for the image modality in time slot $t$.

3. Defect Identification and Diagnosis of Distribution Network Electrical Equipment Based on Two-Stage CNN

To enable defect identification and diagnosis in distribution network electrical equipment through the joint perception of image and voiceprint data, a feature identification method for image and voiceprint data fusion based on a feature temporal attention mechanism must be explored. The schematic diagram is shown in Figure 2. Firstly, the bidirectional coupling attention obtained from the feature joint perception method in the previous section is input into the CNN network for preliminary identification of image and voiceprint data features. Secondly, the feature identification and fusion of joint perception data are carried out based on the closed-loop CNN network, enabling the closed-loop update of fine diagnostic parameters. This two-stage approach ensures effective backpropagation through the entire network, allowing the model to iteratively refine its parameters based on feedback from both stages. Compared to single CNNs with multiple layers, our two-stage CNN architecture offers higher accuracy and better robustness in defect identification, especially in complex environments.

3.1. Preliminary Identification of Image and Voiceprint Data Based on Attention Mechanism with Time Sequence Characteristics

In this section, the CNN network is introduced to perform preliminary identification of data features for image and voiceprint joint perception. CNN, a deep learning model, has achieved significant success in computer vision. Its end-to-end structure enables effective feature extraction.

The raw electrical equipment defect data are extracted by a convolutional layer with relevant features. These features are further compressed through the pooling layer, reducing the size of the feature image and voiceprint while enhancing learning robustness. Global features are extended into matrices and fed into fully connected layers for classification or regression. Therefore, CNN significantly reduces the complexity of the network structure through weight self-adjustment. Figure 3 shows the flow chart of the defect identification and diagnosis for distribution network electrical equipment based on two-stage CNN.

The image and voiceprint joint perception data are input into the convolutional layer. The convolutional layer extracts features from the input data, generating a feature map using a filter. The convolution operation is performed using the Swish activation function, which is expressed as

$$Swish\left(x\right)={\displaystyle \frac{x}{1+{\mathrm{e}}^{-x}}}$$

(18)

A new feature map is generated from the features extracted by the convolutional layer. The pooling layer removes redundant features while preserving prominent ones, helping to improve CNN accuracy. The fully connected layer consists of densely interconnected neurons in a feedforward network. Thus, the fully connected layer ultimately receives the features extracted by the convolutional and pooling layers as inputs.

Next, we perform preliminary identification of image and voiceprint data features based on CNN. First, we acquire the image data ${\mathit{X}}_1\left(t\right)=\{x_{I_1}(t),x_{I_2}(t),\dots ,x_{I_{n_1}}(t)\}$ and voiceprint data ${\mathit{X}}_2\left(t\right)=\{x_{A_1}(t),x_{A_2}(t),\dots ,x_{A_{n_2}}(t)\}$ based on joint perception. Here, $n_1$ represents the number of image data streams, and $n_2$ represents the number of voiceprint data streams. Using CNN, we perform preliminary identification of the image and voiceprint fusion data features, and the resulting preliminary identification data features are expressed as

$${\mathit{Y}}_i^f\left(t\right)={\Gamma }_{\mathrm{CNN}}\left({\mathit{W}}_0\cdot A\left(t\right)\cdot Swish\left({\mathit{X}}_i\left(t\right)\right)+b\right)=\{y_{i,1}(t),y_{i,2}(t),\dots ,y_{i,n_i}(t)\}.$$

(19)

where: $i\in \{I,A\}$ represents the data modality label. $\{y_{i,1}(t),y_{i,2}(t),\dots ,y_{i,n_i}(t)\}$ is the set of data features identified in time slot $t$. $n_i(t)$ is the data traffic in time slot $t$, i.e., the number of preliminary identified features. $y_{i,q}(t)$ is the feature identified for the $q$-th data stream in time slot $t$. ${\Gamma }_{\mathrm{CNN}}\left(\cdot \right)$ denotes the output function of CNN. $Swish(\cdot )$ denotes the activation function. $b$ is the bias term of the convolutional neural network. ${\mathit{W}}_0$ is the initial attention coefficient, and $\mathit{A}(t)$ is the multi-channel coupling temporal interaction attention matrix in time slot $t$, generated by the bidirectional coupling attention group layer discussed in Section 2, which can be expressed as

$$\mathit{A}(t)=\left[\begin{array}{ll}MHA\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)& CrossMHA\left({\mathit{Q}}_{\mathrm{I}}\left(t\right),{\mathit{K}}_{\mathrm{A}}\left(t\right),{\mathit{V}}_{\mathrm{A}}\left(t\right)\right)\\ CrossMHA\left({\mathit{Q}}_{\mathrm{A}}\left(t\right),{\mathit{K}}_{\mathrm{I}}\left(t\right),{\mathit{V}}_{\mathrm{I}}\left(t\right)\right)& MHA\left({\mathit{Q}}_{\mathrm{A}}\left(t\right),{\mathit{K}}_{\mathrm{A}}\left(t\right),{\mathit{V}}_{\mathrm{A}}\left(t\right)\right)\end{array}\right]$$

(20)

For convenience in representation, the head index in Section 2 is replaced by the data stream index in this section. ${\mathit{A}}_k(t)$ is divided into the self-attention weight set ${\mathit{A}}_i^o(t)$ and the interaction attention set $A_i^c\left(t\right)$, where ${\mathit{A}}_i^o\left(t\right)=\{A_{i,1}^o(t),A_{i,2}^o(t),\dots ,A_{i,q}^o(t),\dots ,A_{i,n_i}^o(t)\}$ and ${\mathit{A}}_i^c\left(t\right)=\{A_{i,1}^c(t),A_{i,2}^c(t),\dots ,A_{i,q}^c(t),\dots ,A_{i,n_i}^c(t)\}$. $A_{i,q}^o(t)$ and $A_{i,q}^c(t)$ are the self-attention weights and interaction attention weights for the $q$-th data stream of data modality $i$ in time slot $t$. The subsequent discussion will adopt this form for definition.

Finally, the feature set of the whole data of image and voiceprint fusion can be obtained, which can be expressed as

$${\mathit{Y}}_{all}\left(t\right)=\left\{{\mathit{Y}}_i^f\left(t\right)\left|i=1,2\right.\right\}$$

(21)

To further improve the attention of the identification network to typical and frequent defects and enhance the adaptability of preliminary identification, we introduce the attention weight of temporal interaction. First, we calculate the similarity of the data features from the fusion of image and voiceprint for the $n_i(t)$-th data stream in time slot $t$, which can be expressed as

$$p_{j,k}^i\left(t\right)={\sigma }_1\left(y_{i,j}(t)\odot y_{i,k}(t)\right)+{\sigma }_2\left(y_{i,j}(t)\otimes y_{i,k}(t)\right)$$

(22)

where: $p_{j,k}^i\left(t\right)$ represents the similarity between the pre-identified features of the $j$-th and $k$-th $(j\ne k)$ channels in time slot $t$. $y_{n,g}(t)$ and $y_{n,j}(t)$ represent the pre-identified features of the $g$-th and $j$-th channels in the identification zone n in time slot $t$, respectively. ${\sigma }_1$ and ${\sigma }_2$ are the weight coefficients. $\odot$ is the operator for calculating the waveform similarity between features, and $\otimes$ is the operator for calculating the temporal similarity between features. Specifically, the smaller the time interval between the earlier start time and the later end time of two features, the greater their temporal similarity is.

If the similarity between two features is greater than the threshold $p_0$, i.e., $p_{j,k}^i\left(t\right)>p_0$, then the two features are considered similar. For a series of features, if all features are pairwise similar, they form a feature group. For example, if $R\{p_{1,2}^1\left(t\right),p_{1,3}^1\left(t\right),p_{2,3}^1\left(t\right)\}>p_0$, then the 1st, 2nd, and 3rd features of the image data are considered to belong to the same feature group, where $R\{p_{1,2}^1\left(t\right),p_{1,3}^1\left(t\right),p_{2,3}^1\left(t\right)\}$ represents the average value of $p_{1,2}^1\left(t\right),p_{1,3}^1\left(t\right),p_{2,3}^1\left(t\right)$.

Similarly, for several sets of feature groups obtained from image data and voiceprint data, the interaction similarity is solved across modalities to obtain the set of interaction feature groups. Since the features in the interaction feature group are similar, they can be treated as a single feature group. Therefore, for the features corresponding to each interaction feature group, the temporal series interaction attention weight is computed based on the data count in the interaction feature group set, which is expressed as

$$A_{i,h}^c\left(t\right)={\mathit{P}}_h{\lambda }_{i,h}\left(t\right)=\left\{A_{i,1}^c\left(t\right),\cdots ,A_{i,h}^c\left(t\right),\cdots ,A_{i,m_i}^c\left(t\right)\right\}$$

(23)

where: $A_{i,h}^c\left(t\right)$ represents the temporal interaction attention weight for feature $y_h(t)$. ${\lambda }_{i,h}\left(t\right)$ represents the temporal interaction attention weight coefficient for feature $y_h(t)$. ${\mathit{P}}_h$ denote the number of elements in the interaction feature group set ${\mathit{P}}_h\left(t\right)$, and ${\mathit{P}}_h\left(t\right)$ represents the $h$-th interaction feature group. $y_h(t)$ corresponds to the feature in the interaction feature group ${\mathit{P}}_h\left(t\right)$. The higher the value of ${\mathit{P}}_h$, the more data within a specific time range have been identified as that feature, meaning the feature is representative and typical, and thus the feature attention is higher. The minimum value of ${\mathit{P}}_h$ is 1, meaning that in time slot $t$, only one channel identifies this unique feature, forming a feature group that only contains that feature. Since the feature is rare, the feature attention is lower.

Furthermore, the temporal interaction attention in time slot $t$ is expressed as

$${\mathit{A}}_i\left(t\right)={\mathit{A}}_i^o\left(t\right)+{\mathit{A}}_i^c\left(t\right)$$

(24)

where: ${\mathit{A}}_i\left(t\right)=\{A_{i,1}(t),A_{i,2}(t),\dots ,A_{i,k}(t),\dots A_{i,n_i}(t)\}$. $A_{i,k}(t)$ represents the temporal interaction attention weights for the $k$-th data stream in time slot $t$, where $k$ is the data stream index.

3.2. Precise Diagnosis Method of Image and Voiceprint Defects Based on Closed-Loop CNN

Based on the acquisition of temporal interactive attention, the attention weights in Formula (19) can be modified, and the temporal features and cross-module data information of the image and voiceprint fusion data can be fully mined. This enables the precise diagnosis of image and voiceprint data features, which can be expressed as

$$\mathit{Y}\left(t\right)={\Gamma }_{\mathrm{CNN}}^{\prime }\left(\left({\mathit{W}}_1+{\mathit{W}}_2\right)\cdot Swish\left({\mathit{Y}}_{\mathrm{all}}\left(t\right)\right)+b\right)$$

(25)

where: ${\mathit{W}}_i=\{W_{i,1},W_{i,2},\dots ,W_{i,n_i}\},i=1,2$, $W_{i,k}=W_{i,k,0}{A}_{i,k}(t)$. $W_{i,k}$ represents the corrected attention coefficients for the $k$-th data, and $W_{i,k,0}$ represents the initial attention coefficient for the $k$-th data. $\mathit{Y}\left(t\right)=\{y_1(t),\dots ,y_q(t),\dots y_n(t)\}$ represents the fine identification result of the image and voiceprint fused data features in time slot $t$. $y_q(t)$ represents the $q$-th fine diagnostic result of the image and voiceprint features in time slot $t$, and $q$ represents the fine diagnostic feature index. $n$ represents the number of fine-diagnosed features.

In addition, we propose a method in this section for updating the feature diagnosis model of image and voiceprint data based on closed-loop feedback, which adaptively updates the preliminary identification of image and voiceprint features, the temporal interactive attention of image and voiceprint fusion data, and the fine diagnosis of defect features of image and voiceprint fusion data according to the feature identification results and defect conditions. This can realize the dynamic optimization of the diagnostic model and enhance the comprehensiveness, robustness, and reliability of defect diagnosis.

Specifically, the full data feature set ${\mathit{Y}}_{\mathrm{all}}\left(t\right)$ and the fine diagnostic results from the backend $\mathit{Y}\left(t\right)$ are used. If the features diagnosed by the backend are not identified by the frontend, these features are added to the original full data feature set ${\mathit{Y}}_{\mathrm{all}}^{\prime }\left(t\right)=\left\{{\mathit{Y}}_{\mathrm{all}}\left(t\right),y_k\left(t\right)\right\}$, resulting in a new full data feature set. This new set is then input into the temporal interaction attention for attention identification, where $y_k\left(t\right)$ represents the features diagnosed by the backend that are not identified by the frontend.

If the self-attention weight of $y_k\left(t\right)$ is greater than the average self-attention weight of the full data feature set, the attention of feature $y_k\left(t\right)$ takes precedence, which is expressed as

$$A_{i,k}^o\left(t\right)>\mathrm{E}\left[{\mathit{A}}_i^o\left(t\right)\right]$$

(26)

where: $A_{i,k}^o\left(t\right)$ represents the self-attention weight of feature $y_k\left(t\right)$, and $\mathrm{E}\left[{\mathit{A}}_i^o\left(t\right)\right]$ represents the average self-attention weight of the full data feature set.

Based on this, the preliminary identification of image and voiceprint fused data features is updated. Specifically, the self-attention weight of the feature $y_k\left(t\right)$ in $\mathit{A}(t)$ is updated to $A_{i,k}^o\left(t\right)$.

If the attention of feature $y_k\left(t\right)$ does not take precedence, but based on historical defect feature diagnosis results, the appearance of E is indeed followed by defect E (which has not been detected by the data features). In that case, feature $y_k\left(t\right)$ can be considered as a potential feature for defect E. In this case, the temporal interaction attention parameters for the image and voiceprint fused data are adjusted to increase the model’s attention to feature $y_k\left(t\right)$, thereby improving the accuracy of early warning for the potential defect E. Specifically, the correlation $z_k^{\mathrm{E}}\left(t\right)$ between feature $y_k\left(t\right)$ and the occurrence of defect E is calculated, expressed as

$$z_k^{\mathrm{E}}\left(t\right)=\mathrm{num}\left({\displaystyle \frac{y_k\left(t\right)}{{\mathit{D}}_{\mathrm{E}}\left(t\right)}}\right)$$

(27)

where: $\mathrm{num}\left(·\right)$ is a counting function and ${\mathit{D}}_{\mathrm{E}}\left(t\right)$ represents the historical feature identification results of defect E. This means that the higher the frequency of feature $y_k\left(t\right)$ appearing in the historical feature identification results of defect E, the greater the correlation between feature $y_k\left(t\right)$ and the occurrence of defect E. Further, the temporal interaction attention parameters for the image and voiceprint fused data are adjusted, expressed as

$${\lambda }_k\left(t+1\right)=\left\{\begin{array}{l}{\lambda }_k\left(t\right)+\beta \exp \left(z_k\left(t\right)-z_0\right),z_k\left(t\right)>z_0\\ {\lambda }_k\left(t\right),z_k\left(t\right)\le z_0\end{array}\right.$$

(28)

where: ${\lambda }_k\left(t\right)$ represents the temporal interaction attention weight coefficient for feature $y_k\left(t\right)$. ${\lambda }_k\left(t+1\right)$ represents the temporal interaction attention weight coefficient for feature $y_k\left(t+1\right)$. $z_0$ is the correlation threshold and $\beta$ is the update step size. This equation means that when the correlation $z_k\left(t\right)$ between feature $y_k\left(t\right)$ and defect E exceeds the threshold $z_0$, the temporal interaction attention weight coefficient for feature $y_k\left(t\right)$ is increased in the time slot $t+1$, thereby increasing the attention to feature $y_k\left(t\right)$.

If a feature with a dominant attention identified by the front-end is repeatedly screened out by the back-end image and voiceprint fusion data feature diagnosis, the CNN network in the image and voiceprint fusion data feature diagnosis area is updated in a closed loop, i.e.,

$${\mathbf{\pi }}_{\mathrm{CNN}}\left(k+1\right)={\mathbf{\pi }}_{\mathrm{CNN}}\left(k\right)-{\log }_{{\mathbf{\pi }}_{\mathrm{CNN}}}\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{t=1}^N{\left[{\mathit{Y}}_{\mathrm{all}}\left(t\right)-\mathit{Y}\left(t\right)\right]}^2}}$$

(29)

where: ${\mathbf{\pi }}_{\mathrm{CNN}}$ represents the CNN network for the fine diagnosis of the image and voiceprint data features. $\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{t=1}^N{\left[{\mathit{Y}}_{\mathrm{all}}\left(t\right)-\mathit{Y}\left(t\right)\right]}^2}}$ represents the identification error caused by discarding feature $y_k\left(t\right)$ during the fine diagnosis of the image and voiceprint fused data features.

4. Simulation Verification

In order to verify the accuracy of the proposed method, a dataset with 5 years of historical data and 10,000 samples is constructed, including the image and the voiceprint signal of the equipment at the fault location in the distribution network. From the fault dataset, seven typical fault samples of electrical equipment in the distribution network are selected: partial discharge of medium voltage switchgear, mechanical fault, loose transformer core, DC bias, heavy overload, transmission line damage, and bird’s nest. The initial 7500 samples were used for training, while the remaining 2500 were designated for testing. During training, the maximum number of iterations is set to 200, with a learning rate of 0.1. The training set data and results are used to train the image and voiceprint feature joint perception network based on bidirectional coupled attention, while the test set fault data are fed into the trained network for detection.

In addition, we select two algorithms for comparative analysis to verify the effectiveness of the proposed algorithm. Baseline 1 adopts the LSTM algorithm [19], which performs separate feature extraction and processing for the image and voiceprint signals, without splicing and coupling the features of the two modalities. LSTM is mainly used to capture time sequence features and identify potential defects by analyzing the signal at each time step. However, when the environmental background noise is large or the image features are blurred, LSTM is less robust in defect identification. Baseline 2 uses the traditional CNN algorithm [20], which uses the classical convolutional layer for feature learning and classifies through pooling and fully connected layers. However, it does not perform closed-loop feedback adjustment based on the actual defect situation, which limits the accuracy and robustness of defect identification. Traditional CNN also has certain shortcomings in the fusion of multimodal information and it is difficult to fully combine the complementarity between image and voiceprint features, which limits its identification effect under complex working conditions. The baseline algorithm uses the same training set and test set as the proposed algorithm for simulation. We verify the performance of the proposed algorithm by comparing the accuracy of defect identification and the convergence performance with two baseline algorithms.

In order to preliminarily test the effectiveness of the electrical equipment defect identification model in this paper, after model training on the dataset, the confusion matrix is used to measure the identification accuracy of the model for each state sample, as shown in Figure 4. The identification accuracy of defects is higher than 97%. This indicates that the image and voiceprint feature joint perception model based on bidirectional coupled attention proposed in this paper has good sensitivity and identification ability for electrical equipment defects.

To evaluate the accuracy of this algorithm, five metrics are selected: accuracy, precision, recall, average precision ($mAP$), and F1 score.

$$F1={\displaystyle \frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(30)

$$\mathrm{Precision}={\displaystyle \frac{N_{\mathrm{TP}}}{N_{\mathrm{TP}}+N_{\mathrm{FP}}}}$$

(31)

$$\mathrm{Recall}={\displaystyle \frac{N_{\mathrm{TP}}}{N_{\mathrm{TP}}+N_{\mathrm{FN}}}}$$

(32)

$$mAP={\displaystyle \sum _{m=1}^{M}A}{P}_m/M$$

(33)

where: $N_{\mathrm{TP}}$, $N_{\mathrm{FP}}$, and $N_{\mathrm{FN}}$ represent the number of true detections, false detections, and missed detections, respectively. $AP$ refers to the area under the Precision–Recall (P-R) curve, and $m$ represents the number of detection categories.

Figure 5 shows the comparison of the defect identification results. Compared with the two baseline algorithms, the accuracy of the proposed algorithm is improved by 12.1% and 33.7%, the precision is increased by 11.3% and 34.3%, the recall rate is increased by 12.5% and 33.1%, the F1 value is increased by 10.4% and 34.1%, and the average accuracy is increased by 16.2 and 34.3%, respectively. The proposed algorithm analyzes the temporal characteristics of image and voiceprint data, and adaptively adjusts the attention weight of the network according to the time interval and correlation. This can strengthen the attention to typical and frequent defects, improving the accuracy of preliminary identification. Additionally, the closed-loop CNN realizes multi-stage iterative optimization by feeding back the results of preliminary identification to the fine diagnosis network. This allows the diagnosis network to carry out detailed and accurate diagnosis for the actual defects in complex environments, improving robustness and accuracy.

Figure 6 demonstrates the change of mAP versus slots. The average accuracy value of the proposed algorithm rises to about 0.85 in the first 30 slots, and finally stabilizes at 0.931. Baseline 2 shows a slow increase in mAP to 0.6 around 50 training slots, eventually stabilizing at 0.611. Compared with the two baseline algorithms, the convergence speed of the proposed algorithm is increased by 22.92% and 60.42%, respectively. The proposed algorithm uses the vertical self-attention mechanism to explore the intricate relationships within the modality through the multi-head attention mechanism, so that the network can adaptively pay attention to more critical features and improve the model’s perception of complex defects. Horizontal coupling attention introduces cross-modal information interaction, so that the semantic gap between the image and the voiceprint can be effectively bridged. This significantly enhances collaboration between different modalities while improving the efficiency and accuracy of defect identification.

Figure 7 shows the spatial distribution of features after image and voiceprint joint perception and fault feature extraction. The three coordinate axes represent the principal component of the image features, the principal component of the voiceprint feature, and the multi-modal fusion feature, respectively. Each point corresponds to a fault sample, and the fault sample is mapped into the three-dimensional space through principal component analysis to form the feature point. The close aggregation of the feature points means that the discriminative nature of the fault type in the high-dimensional feature space is enhanced, which indicates that the method successfully separates the features of different fault types and achieves high effectiveness and accuracy in the process of feature extraction and fault identification. It is worth noting that defect 4 and defect 7 are not close to the center, which further highlights their distinct characteristics and effective separation from other defects. This is a desirable outcome for accurate fault identification and diagnosis. The vertical self-attention mechanism can mine more complex feature associations within image and voiceprint modalities. The horizontal coupling attention mechanism eliminates the semantic gap between different modalities through the exchange of cross-modal information. This allows the features of image and voiceprint to complement each other, thereby improving the accuracy of defect identification. Through this bidirectional coupled attention mechanism, the model can not only capture the unique characteristics of each modality, but also enhance the synergistic effect between them, so that fault samples of the same type can be tightly clustered together in the feature space. Through the effective fusion of image and voiceprint features, the proposed method successfully extracts the core features of the equipment fault and can effectively map these features to the low-dimensional space. In the samples of the same fault type, the aggregation of feature points indicates that the features of the fault type are strengthened and clearly resolved under the multi-modal information fusion, so as to achieve more efficient fault identification and diagnosis. Compared with the traditional single-modal feature extraction method, the accuracy and robustness of identification are significantly improved. Especially in complex backgrounds and with multi-source interference, the model can maintain strong identification ability through the complementarity of multi-modal features.

Considering the potential abnormal states in the network, two types of distortion are added to the collected images and voiceprint signals of the equipment defect to verify the robustness of the identification and diagnosis model. The first one is Gaussian white noise and the second one is a decrease in the quality of the image. As a rule of thumb, we randomly put 90 percent of the distortion samples into the training dataset, and the rest of the experimental images and voiceprint signals are put into the test dataset at the beginning of each experiment.

Table 1. Comparison of detection results of each algorithm after adding distortion.

	Accuracy	Precision	Recall	Average Precision	F1 Score
Proposed Algorithm	0.871	0.833	0.901	0.916	0.897
Baseline 1	0.733	0.781	0.693	0.763	0.699
Baseline 2	0.493	0.477	0.510	0.442	0.491

shows that the proposed algorithm maintains accurate fault target detection even with added noise. Although its detection accuracy slightly declines, the average accuracy and recall rate remain high, demonstrating strong anti-noise capability. However, as signal distortion increases, the model’s identification accuracy decreases. This is because the proposed algorithm integrates image and voiceprint features using a bidirectional coupled attention mechanism, along with multi-modal information fusion, to realize multi-modal joint perception. Integrating and complementing information from different feature sources allows key features to be preserved in distorted signals while reducing noise interference. By incorporating multi-dimensional voiceprint information, the model’s noise resistance and robustness are enhanced, which is why the decline in performance is not significant. In addition, the adaptive weight adjustment of the temporal adaptive CNN and the feedback mechanism of the closed-loop CNN further improve the model’s adaptability to abnormal inputs, so that the model can identify the effective features of interest through optimization during the training process. This ensures that detection accuracy is maintained even under distorted signals.

5. Conclusions

This paper addressed the issues of efficiency and accuracy in defect identification and diagnosis of electrical equipment in distribution networks by proposing a defect identification and diagnosis method for distribution network electrical equipment based on fused image and voiceprint joint perception. Through initial feature extraction and concatenation coupling, a bidirectional attention mechanism was constructed to enhance the efficiency of defect identification and diagnosis. Meanwhile, by combining temporal adaptability and a closed-loop feedback mechanism, precise diagnosis of equipment defects is achieved, significantly improving diagnostic accuracy and robustness. Baseline 1, an LSTM-based algorithm that performs separate feature extraction and processing for image and voiceprint signals, and Baseline 2, a traditional CNN algorithm that uses classical convolutional layers for feature learning and classification, are chosen as comparison benchmarks. Compared to the baselines, the proposed method significantly enhances defect identification accuracy by 12.1% and 33.7%, increases recall by 12.5% and 33.1%, and boosts diagnostic efficiency by 22.92% and 60.42%. Future research will focus on the integration of more modal signals, particularly combining voltage, current, and electromagnetic signals with image and voiceprint features, to provide multi-dimensional data support for fault identification and diagnosis. Additionally, further exploration will be conducted on how to leverage deep learning and other advanced machine learning methods to achieve deeper multimodal signal fusion, supporting more accurate fault identification and diagnosis.