Global and Local Knowledge Distillation Method for Few-Shot Classification of Electrical Equipment

Abstract

With the increasing utilization of intelligent mobile devices for online inspection of electrical equipment in smart grids, the limited computing power and storage capacity of these devices pose challenges for deploying large algorithm models, and it is also difficult to obtain a substantial number of images of electrical equipment in public. In this paper, we propose a novel distillation method that compresses the knowledge of teacher networks into a compact few-shot classification network, employing a global and local knowledge distillation strategy. Central to our method is exploiting the global and local relationships between the features exacted by the backbone of the teacher network and the student network. We compared our method with recent state-of-the-art (SOTA) methods on three public datasets, and we achieved superior performance. Additionally, we contribute a new dataset, namely, EEI-100, which is specifically designed for electrical equipment image classification. We validated our method on this dataset and demonstrated its exceptional prediction accuracy of 94.12% when utilizing only 5-shot images.

1. Introduction

As an essential component of the power system, daily inspection of electrical equipment is imperative to ensure the secure and stable operation of the power system [1]. Image classification is a crucial prerequisite for monitoring the condition of electrical equipment based on image information. In recent years, machine learning has made significant progress in the field of image classification for electrical equipment. Bogdann presented a machine learning method for determining the state of each switch by analyzing images of the switches in power distribution substations [2]. Zhang implemented FINet based on improved YOLOv5 to inspect the insulators and their defects in order to ensure the safety and stability of the power system [3]. To address the few fault cases and deficient monitoring information in transformer diagnostic tasks, Xu provides an improved few-shot learning method based on approximation space and belief functions [4]. Yi proposed a label distribution CNNs classifier to estimate the aging time of the conductor morphology of the high-voltage transmission line [5]. It is noteworthy that the majority of these investigations have concentrated on a restricted range of electrical apparatus. These models necessitate a substantial quantity of training data to guarantee optimal performance. Nevertheless, acquiring adequate electrical equipment images in a practical setting may prove challenging, and the proportion of labeled samples is minimal. To a certain extent, the classification of electrical equipment images is not really a big data problem; rather, it belongs to the few-shot learning (FSL) domains.

Few-shot classification (FSC) is the task of classifying samples with a limited number of instances while showcasing rapid generalization capabilities. Recently, a large number of few-shot image classification algorithms have been proposed. Depending on the learning paradigm used, these methods can be broadly divided into two categories: (1) meta-learning-based methods and (2) transfer-based learning methods. Meta-learning is a promising approach that leverages episodic training to simulate the real test environment by randomly selecting several subtasks. Optimization-based meta-learning methods employ a two-tier optimization process to learn the optimizer for quickly processing new tasks. Model-Agnostic Meta-Learning (MAML) [6] obtains the optimal initialization parameters of the model through meta-training, enabling the model to adapt to new tasks after a few gradient updates. In addition, the learning rate and gradient direction are also important factors for the optimizer [7,8]. However, these methods require the storage and computation of higher-order derivatives, resulting in high memory and computational costs. On the other hand, metric-based methods use nonparametric classifiers as the basic learners, avoiding the aforementioned issues. The key factors of these methods are feature extraction and similarity measurement, which offer ample room for improvement. PARN [9] proposed a feature extractor that learned an offset for each cell in the convolution kernel in order to extract more efficient features, building deformable convolutional layers. CC+rot [10] improved the transfer ability of feature extractors by adopting auxiliary self-supervised tasks. Zhang et al. [11] used the pre-trained visual saliency detection model to segment the foreground and background of the image and then extract the foreground and background features, respectively. With the proven effectiveness of attention mechanisms in extracting discriminating features, several few-shot classification (FSC) tasks have adopted this method, including CAN [12], AWGIM [13], and CTM [14]. Additionally, in metric-based meta-learning methods, the measurement of similarity is also crucial. SEN [15] combines Euclidean distance and norm distance to improve the effectiveness of Euclidean distance measurement in high-dimensional spaces. FPN [16] calculated the reconstruction error between the support sample and the query sample as the similarity score. DN4 [17] and Deep EMD [18] obtain rich similarity measures directly from local features.

Recent studies have indicated that the FSC of the transfer-learning method can attain comparable performance to that of the meta-learning method with complex episodic training. Such methods typically combine pre-trained feature extractors on all base-class datasets with arbitrary traditional classifiers to make classification decisions for query samples of unknown classes. It has been shown that pre-training the entire base class dataset using the cross-entropy loss function followed by fine-tuning the pre-trained model using support samples of the visible class can provide a powerful baseline for FSC tasks [19]. Since then, several efforts have been proposed to improve the representation performance of feature extractors. For example, Neg-Cosin [20] proposed the use of the non-negative interval Cosine loss function to optimize the model, thereby increasing the distance between the training sample and its corresponding parametric prototype, which can effectively improve the generalization performance of the model. S2M2 [21] used manifold mixing as an effective regularization method to improve the generalization performance of the model. Refs. [22,23] used rotation prediction and mirror prediction as self-supervised tasks to add to the pre-training process, and experimental results showed that self-supervised tasks are effective methods to improve feature representation performance.

In the pursuit of high-accuracy classification performance, many studies tend to employ highly complex networks to enhance the representation of features. As a result, deploying these methods in real-world applications often requires significant computing resources (such as storage space and computing power) and leads to significant time delays. However, power inspection heavily relies on intelligent mobile devices, including inspection robots and UAVs. Considering the limited storage capacity of these devices, it is essential to ensure that the classification model’s size is not excessively large. Therefore, in this study, we utilize a knowledge distillation-based model compression algorithm to achieve few-shot image classification, aiming to reduce the number of model parameters.

Knowledge distillation [24,25] is one of the most effective model compression methods and has garnered significant research interest in both industry and academia due to its simple training strategy and effective performance. It leverages the knowledge acquired by a teacher network to guide the training of a small-scale student network, enabling the latter to achieve comparable performance despite having fewer parameters. In order to make better use of the knowledge information contained in the teacher network, AT [26] proposed to take the spatial attention of the hidden layer features of the teacher network as knowledge and instruct the student network to imitate its attention feature map. RKD [27] proposed a relational knowledge distillation method, which used distance and angle to measure the relationship between sample features, as a valid type of knowledge during distillation. Peng et al. [28] used the kernel function to obtain higher-order relationships between the features of samples as effective distillation knowledge. Although the model compression methods based on feature relation knowledge mentioned above can effectively improve the performance of small-capacity student networks, the current work only focuses on the local relationship between individual sample features, ignoring the global relationship between the features of samples. Therefore, this paper proposes a method based on global and local knowledge distillation and applies it to the task of FSC for electrical equipment images.

In this paper, we propose three main contributions:

• We present a novel distillation approach that compresses the knowledge of teacher networks into a compact student network, enabling efficient few-shot classification. The incorporation of global and local relationship strategies during the distillation process effectively directs the student network towards achieving performance levels akin to those of the teacher network.
• We contribute a new dataset that contains 100 classes of electrical equipment with 4000 images. The dataset contains a wide range of various pieces of electrical equipment, including power generation equipment, distribution equipment, industrial electrical equipment, and household electrical equipment.
• We demonstrate the effectiveness of our proposed method by validating it on three public datasets and comparing it with the SOTA methods on the electrical image dataset we introduced. Our proposed method outperforms all other methods and achieves the best performance.

2. Methodology

2.1. Problem Definition

In few-shot image classification tasks, given a certain size of image dataset I, it is randomly divided into three subsets: I_train, I_val, and I_test. I_train is used as the base dataset for pre-training the classification model. Assuming that the pre-training set has C_b categories, the m-th image sample is denoted as x_m, and its corresponding label is represented as y_m. I_val is used for validation, while I_test is used as the new class dataset for testing the trained model. For I_val and I_test, multiple N-way-K-shot subtasks are randomly sampled, with each task consisting of a support sample set (I_S) and a query sample set (I_Q). I_S is constructed by randomly selecting N categories from I_val or I_test, and then randomly selecting K samples from each category. The set of the n-th category is denoted as ${{\displaystyle I}}_n={{\displaystyle \{({{\displaystyle I}}_k,{{\displaystyle y}}_k)\}}}_{k=1}^K$, and the k-th image in the n-th category is denoted as I_k. I_Q is composed of Q samples randomly selected from each residual sample category, denoted as ${{\displaystyle I}}_Q={{\displaystyle \left\{{{\displaystyle I}}_q\right\}}}_{q=1}^Q$, where I_q represents the q-th query sample. Therefore, the problem of few-shot image classification can be described as using the model trained on the base class dataset and the support sample set to make classification decisions for query samples.

2.2. FSC Network Based on Global and Local Knowledge Distillation

We propose a novel few-shot electrical image classification algorithm based on knowledge distillation. Figure 1 shows the overall architecture of our network. We first trained a high-performance teacher network through self-supervised learning, and we then guided the training of the student network by the teacher network. To fully utilize the prior knowledge of the teacher network, we designed a knowledge distillation method based on global and local relationships. This method can transfer the global and local features of the images extracted by the teacher network to the student network, enabling the compact student network to learn more effective features about the images and achieve better image classification.

2.2.1. Pre-Train of Teacher Network

The teacher model consists of a backbone convolutional neural network and two linear classifiers. The backbone network f_θ(•) is used for feature extraction of images, one classifier L_w(•) is used for predicting the base class of image samples, and the other classifier L_r(•) is used for predicting the rotation category in self-supervised tasks. Additionally, each classifier is followed by a Softmax layer. M image samples are randomly selected from the base class dataset, and each image is rotated at 0°, 90°, 180°, and 270°, with its corresponding rotation label as ${{\displaystyle \widehat{y}}}_m$ = [0, 1, 2, 3].

When image x_m is fed into the teacher network, the d-dimensional feature representation f_θ(x_m) is extracted by the backbone network. The classification scores of the base class prediction classifier and the rotation prediction classifier for the features are expressed as S_b and S_r, as shown in Equation (1):

$$\{\begin{array}{c}{{\displaystyle {{\displaystyle S}}_b=L}}_w({{\displaystyle f}}_{\theta }({{\displaystyle x}}_m))\\ {{\displaystyle {{\displaystyle S}}_r=L}}_r({{\displaystyle f}}_{\theta }({{\displaystyle x}}_m)).\end{array}$$

(1)

Furthermore, the aforementioned classification scores are transformed into base class and rotation class prediction probabilities through a Softmax layer, as shown in Equation (2):

$$\{\begin{array}{c}{{\displaystyle p}}_{}({{\displaystyle y}}_m=c|{{\displaystyle x}}_m)=\frac{{{\displaystyle e}}^{{{\displaystyle S}}_{bc}}}{{\displaystyle \sum _{c=1}^{{{\displaystyle C}}_b}{{\displaystyle e}}^{{{\displaystyle S}}_{bc}}}}\\ {{\displaystyle p}}_{}({{\displaystyle \widehat{y}}}_m=r|{{\displaystyle x}}_m)=\frac{{{\displaystyle e}}^{{{\displaystyle S}}_{rr}}}{{\displaystyle \sum _{r=1}^4{{\displaystyle e}}^{{{\displaystyle S}}_{rr}}}},\end{array}$$

(2)

where S_bc denotes the c-th element of the score vector S_b, S_rr denotes the r-th element of the score vector S_r, C_b denotes the number of base class labels, and $p({{\displaystyle y}}_m=c|{{\displaystyle x}}_m)$ and $p({{\displaystyle \widehat{y}}}_m=r|{{\displaystyle x}}_m)$ are the probability output values of the base classifier and the rotation classifier, respectively. The cross-entropy loss function and the self-supervised loss function are calculated to obtain the training loss function, as shown in Equation (3):

$$L(\theta ,w,r)=-{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{c=1}^{{{\displaystyle C}}_b}{{\displaystyle y}}_{mc}}}\log p({{\displaystyle y}}_m=c|{{\displaystyle x}}_m)-{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{r=1}^4{{\displaystyle \widehat{y}}}_{mr}}}\log p({{\displaystyle \widehat{y}}}_m=r|{{\displaystyle x}}_m),$$

(3)

where y_mc denotes the c-th element of the one-hot encoded vector of y_m, and ${{\displaystyle \widehat{y}}}_{mr}$ denotes the c-th element of the one-hot encoded vector of ${{\displaystyle \widehat{y}}}_m$. Based on the loss function in Equation (3), the parameters of the teacher network are optimized to complete the pre-training process.

2.2.2. Global and Local Knowledge Distillation

Firstly, a student network is constructed, which consists of a backbone neural network B_Φ(•) composed of a small number of convolutional layers and a linear classifier C_H(•). Next, a batch of M images randomly selected from the base dataset I_train is inputted into both the teacher network and the student network. The m-th image is represented by feature maps ${{\displaystyle z}}_m^t{{\displaystyle =f}}_{\theta }({{\displaystyle I}}_m)$ and ${{\displaystyle {{\displaystyle z}}_m^s=B}}_{\varphi }({{\displaystyle I}}_m)$ obtained from the backbone of the teacher network and the student network, respectively. Finally, the features are fed into the linear classifier to obtain the output value of the student network, as shown in Equation (4):

$${{\displaystyle {{\displaystyle S}}_m=C}}_H({{\displaystyle z}}_m^s).$$

(4)

Furthermore, the above output classification scores are transformed into classification prediction probabilities through the Softmax layer, as shown in Equation (5):

$${{\displaystyle p}}_s({{\displaystyle y}}_m=c|{{\displaystyle x}}_m)=\frac{{{\displaystyle e}}^{{{\displaystyle S}}_{mc}}}{{\displaystyle \sum _{c=1}^{{{\displaystyle C}}_b}{{\displaystyle e}}^{{{\displaystyle S}}_{mc}}}},$$

(5)

where S_mc denotes the c-th element of the score vector S_m.

The equation for calculating the cross-entropy loss function between the output values of a student network and the true labels is shown in Equation (6):

$${{\displaystyle l}}_1(\varphi ,H)=-{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{c=1}^{{{\displaystyle C}}_b}{{\displaystyle y}}_{mc}}}\log {{\displaystyle p}}_s({{\displaystyle y}}_m=c|{{\displaystyle x}}_m).$$

(6)

In order to enable students to learn the representation of global features of images by the teacher network through online learning, we adopt the maximum mean discrepancy between the feature spaces of the two networks as the global loss function, which is calculated by Equation (7):

$${{\displaystyle l}}_2(\varphi ,H)=\frac{1}{{{\displaystyle M}}^2}{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{m^{\prime }=1}^M{z}_m^t{z}_{m^{\prime }}^t{}^T}}+\frac{1}{{{\displaystyle \mathrm{M}}}^2}{\displaystyle \sum _{m=1}^{\mathrm{M}}{\displaystyle \sum _{m^{\prime }=1}^{\mathrm{M}}{{\displaystyle z}}_m^s{z}_{m^{\prime }}^s{}^T}}-\frac{1}{{{\displaystyle \mathrm{M}}}^2}{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{m^{\prime }=1}^M{{\displaystyle z}}_m^t}}{z}_{m^{\prime }}^s{}^T.$$

(7)

In addition, we calculate the Euclidean distance between each sample feature in the two networks as the local loss function, and its calculation formula is shown in Equation (8):

$${{\displaystyle l}}_3(\varphi ,H)=\frac{1}{M}{\displaystyle \sum _{m=1}^M{{\displaystyle \Vert {{\displaystyle z}}_m^t-{{\displaystyle z}}_m^s\Vert }}^2}.$$

(8)

In summary, the total loss function for the student network is shown in Equation (9). Based on Equation (9), the student network is trained, and the parameters in the network are updated until optimal, thereby completing the knowledge distillation process from the teacher network to the student network.

$$L(\varphi ,H)={{\displaystyle l}}_1(\varphi ,H)+{{\displaystyle \alpha }}_1{{\displaystyle l}}_2(\varphi ,H)+{{\displaystyle \alpha }}_2{{\displaystyle l}}_3(\varphi ,H).$$

(9)

2.2.3. Few-Shot Evaluation

After completing the knowledge distillation task in Section 2.2.2, the base classifiers in the student network are first removed. Furthermore, the parameters of the backbone neural network B_Φ(•) are fixed, and features are extracted from both the support and query samples. Finally, based on the N-way-K-shot method, the query samples are classified using Equation (10), where the features of the k-th support sample and the q-th query sample are denoted as B_Φ(I_k) and B_Φ(I_q), respectively, and g_φ{•} is a classifier with parameters φ. Any traditional classifier can be used to complete the classification prediction task.

$${{\displaystyle \widehat{y}}}_q={{\displaystyle g}}_{\phi }\{{{\displaystyle B}}_{\varphi }({{\displaystyle I}}_q)|{\displaystyle \sum _{k=1}^{NK}{{\displaystyle B}}_{\varphi }({{\displaystyle I}}_k)}\}.$$

(10)

3. Experiments

3.1. EEI-100 Dataset

We invested a substantial amount of time in constructing a dataset for electrical equipment image classification, EEI-100. It contains 100 classes of electrical equipment with 4000 images. The majority of the images were obtained through on-site collection, with a small number of images sourced from online platforms. To the best of our knowledge, this is one of the first datasets specifically designed for the classification of electrical equipment. This dataset is an extension of our previous EEI-40 [29]. It includes substation equipment, distribution station equipment, and common electrical equipment, ranging from large-scale equipment such as heavy-duty transformers to small-scale equipment such as circuit breakers. A few images from the proposed dataset are illustrated in Figure 2 (more images are illustrated in Appendix A).

3.2. Experiments on Public Datasets

3.2.1. Experiment Setup

We evaluate our knowledge distillation method on three widely used public datasets, namely MiniImageNet, CIFAR-FS, and CUB. The experiments were conducted on a workstation equipped with an NVIDIA 3090Ti GPU and implemented using Pytorch software. To ensure a fair comparison with current few-shot image classification methods, a commonly used 4-layer convolutional neural network (CNN) and ResNet12 were adopted as the student network and teacher network, respectively. During the training phase, we used the SGD optimizer to optimize our models in all experiments, where momentum was set to 0.9 and weight decay was set to 5 × 10⁻⁴. We trained for 100 epochs with an initial learning rate of 0.025, which was reduced by half after 60 epochs. In the testing phase, we conducted 5-way-1-shot and 5-way-5-shot tests. Specifically, we randomly performed 2000 classification subtasks on the testing dataset. In each subtask, 15 images were randomly selected from each class as query images for testing. The evaluation criterion for the algorithm’s classification performance was the average accuracy of all subtasks, and the standard deviation of the accuracy under a 95% confidence interval should also be provided.

Please note that we also conducted a similar experiment on the 1080Ti GPU and achieved comparable performance. This significantly alleviates the economic burden associated with model training. Leveraging a 4-layer CNN architecture, the student model occupies a mere 2 MB in size, which is approximately 50 times smaller than the teacher model. This lightweight model can be deployed on diverse edge processors, substantially lowering the hardware requirements for its implementation.

3.2.2. Parametric Analysis Experiment

It can be seen from Equation (9) that α₁ and α₂ are important hyperparameters in the process of distilling the student network. Initially, we conducted experiments in which we temporarily ignored the influence of global knowledge by setting α₁ to 0. Through this analysis, we observed that α₂ near 1 yielded the best model’s performance. Building upon this observation, we proceeded to fix α₂ at 1, and the value of parameter α₁ was varied with a step size of 0.1 within the range of [0, 1]. The test accuracy of the student network under different values of α₁ is shown in Figure 3a,b. The results indicate that the model performance is optimal when the value of α₁ is 0.5. Therefore, the value of α₁ was set to 0.5, and then α₂ was varied with a step size of 0.01 within the range of [0, 0.1]. The test accuracy of the student network under different values of α₂ is shown in Figure 3c,d. The results reveal that the optimal value for α₂ is 0.1.

Additionally, after completing the search for α₁ and α₂ within their respective ranges, we extended our exploration beyond the boundaries of [0, 1] and [0, 0.1]. However, we found that no values outside of these ranges yielded superior results.

3.2.3. Ablation Studies

The innovation of this work lies in proposing a knowledge distillation algorithm for global and local relationships. In order to verify the effectiveness of the proposed method, detailed ablation experiments were conducted on three public datasets. The knowledge distillation algorithms using only global and local relationships are denoted as Global and Local, respectively, and their fusion is denoted as Global-Local. The classification accuracies of these methods on 5-way-1-shot and 5-way-5-shot tasks are shown in

Table 1. Results (%) of ablation experiment.

Method	Backbone	MiniImageNet		CIFAR-FS		CUB
		1-Shot	5-Shot	1-Shot	5-Shot	1-Shot	5-Shot
Global	Conv4	57.32 ± 0.84	72.90 ± 0.64	66.40 ± 0.93	80.44 ± 0.67	70.20 ± 0.93	83.88 ± 0.57
Local	Conv4	57.65 ± 0.83	73.06 ± 0.64	66.63 ± 0.93	80.64 ± 0.67	70.12 ± 0.93	83.66 ± 0.57
Global-Local	Conv4	57.86 ± 0.83	73.38 ± 0.62	67.04 ± 0.91	80.84 ± 0.68	70.44 ± 0.92	84.19 ± 0.56

.

The results in Table 1 indicate that for both 5-way 1-shot and 5-way 5-shot tasks on all datasets, the classification accuracy of Global-Local is consistently higher than that of Global and Local. The experiments demonstrate that global and local relationships are complementary, and their fusion can extract richer image features. Therefore, the knowledge distillation algorithm based on global and local relationships can further improve the performance of knowledge distillation.

3.2.4. Comparison Experiment Compared with Existing Methods

This paper compares our method with the SOTA methods in recent years, which are mainly divided into two categories: meta-learning-based methods and transfer learning-based methods. The comparison results with these methods are shown in

Table 2. Comparison results (%) of the experiment on three public datasets.

Method	Backbone	MiniImageNet		CIFAR-FS		CUB
		1-Shot	5-Shot	1-Shot	5-Shot	1-Shot	5-Shot
Meta-learning
Relational	Conv4	50.44 ± 0.82	65.32 ± 0.70	55.00 ± 1.00	69.30 ± 0.80	62.45± 0.98	76.11± 0.69
MetaOpt SVM	Conv4	52.87 ± 0.57	68.76 ± 0.48	-	-	-	-
PN+rot	Conv4	53.63 ± 0.43	71.70 ± 0.36	-	-	-	-
CovaMNet	Conv4	51.19 ± 0.76	67.65± 0.63	-	-	52.42 ± 0.76	63.76 ± 0.64
DN4	Conv4	51.24 ± 0.74	71.02 ± 0.64	-	-	46.84 ± 0.81	74.92 ± 0.64
MeTAL	Conv4	52.63 ± 0.37	70.52 ± 0.29	-	-
HGNN	Conv4	55.63 ± 0.20	72.48 ± 0.16	-	-	69.02 ± 0.22	83.20 ± 0.15
DSFN	Conv4	50.21 ± 0.64	72.20 ± 0.51	-	-	-	-
PSST	Conv4	-	-	64.37 ± 0.33	80.42± 0.32	-	-
Transfer-learning
Baseline++	Conv4	48.24 ± 0.75	66.43 ± 0.63	-	-	60.53 ± 0.83	79.34 ± 0.61
Neg-Cosine	Conv4	52.84 ± 0.76	70.41 ± 0.66	-	-	-	-
SKD	Conv4	48.14	66.36	-	-	-	-
CGCS	Conv4	55.53 ± 0.20	72.12 ± 0.16	-	-	-	-
Our method	Conv4	57.86 ± 0.83	73.38 ± 0.62	67.04 ± 0.91	80.84 ± 0.68	70.44 ± 0.92	84.19 ± 0.56

- indicates that the method described in the literature was not evaluated on certain datasets.

.

According to the results in Table 2, the following observations can be made:

• On the MiniImageNet dataset, our proposed method achieves the best classification performance. Compared with the best-performing method in the meta-learning-based category, HGNN, our method outperforms it by 2.23% and 0.9% on 1-shot and 5-shot classification tasks, respectively. In the transfer learning-based category, compared with the best performing method, CGCS, our method outperforms it by 2.33% and 1.26% on 1-shot and 5-shot classification tasks, respectively.
• On the CIFAR-FS dataset, our proposed method also achieves top performance. Our method outperforms the best-performing method, PSST, by 2.67% and 0.42% on 1-shot and 5-shot classification tasks, respectively.
• On the CUB-200-2011 dataset, our proposed method achieves the highest classification performance. Our method outperforms the best-performing method, HGNN, by 1.42% and 0.99% on 1-shot and 5-shot classification tasks, respectively.

3.3. Experiments on EEI-100 Dataset

Furthermore, we compare the performance of our proposed method with the SOTA methods on the EEI-100 dataset. The experimental process employs the same parameter selection strategy as before.

3.3.1. Parametric Analysis Experiment

By following the approach outlined in Section 3.2.2, the values of parameters α₁ and α₂ are determined to optimize the performance of the model on EEI-100. Experimental results show that α₁ has the optimal value of 0.6 within the range of [0.1, 1], as illustrated in Figure 4a. Similarly, α₂ has the optimal value of 0.1 within the range of [0.01, 0.1], as illustrated in Figure 4b.

3.3.2. Comparison Experiment with Existing Methods

To demonstrate the superiority of our proposed method in the classification of electrical equipment images, this section presents a comparative experiment with three existing methods, namely, CGCS, Neg-Cosine, and HGNN, on the EEI-100 dataset. These three methods have recently achieved good performance on public datasets. The classification accuracy of the test set is presented in

Table 3. Comparison result (%) of the experiment on EEI-100 dataset.

Method	1-Shot	5-Shot
CGCS	72.85 ± 0.68	89.68 ± 0.27
Neg-Cosine	74.57 ± 0.63	90.54 ± 0.25
HGNN	75.61 ± 0.62	93.54 ± 0.24
Our method	75.80 ± 0.67	94.12 ± 0.20

. Specifically, our method achieves the highest classification accuracy (up to 94.12%) compared with the other methods.

4. Conclusions and Future Work

In conclusion, this paper proposes a novel few-shot electrical image classification algorithm based on knowledge distillation. By leveraging the few-shot learning method and employing global and local knowledge distillation, our algorithm achieves high classification accuracy with only a limited number of image samples. The results obtained on the newly introduced EEI-100 dataset demonstrate that our method achieves a remarkable prediction accuracy of 94.12% using just 5-shot images. The lightweight and high-performance nature of our model enables its practical application in the online inspection of electrical equipment in smart grids, effectively enhancing the efficiency of detection and maintenance in the power system. Furthermore, the training and deployment of our model do not impose significant hardware requirements, making it accessible to a wide range of researchers.

Regarding future work, we plan to explore a pre-training method to separate the foreground and background, as different backgrounds may negatively affect distillation. Additionally, we plan to use a multi-stage fusion of global and local features during the distillation process. This can provide a better understanding of the underlying structure of the complex model and the relationship between different stages of the model.