Training Small Networks for Scene Classification of Remote Sensing Images via Knowledge Distillation

Abstract

Scene classification, aiming to identify the land-cover categories of remotely sensed image patches, is now a fundamental task in the remote sensing image analysis field. Deep-learning-model-based algorithms are widely applied in scene classification and achieve remarkable performance, but these high-level methods are computationally expensive and time-consuming. Consequently in this paper, we introduce a knowledge distillation framework, currently a mainstream model compression method, into remote sensing scene classification to improve the performance of smaller and shallower network models. Our knowledge distillation training method makes the high-temperature softmax output of a small and shallow student model match the large and deep teacher model. In our experiments, we evaluate knowledge distillation training method for remote sensing scene classification on four public datasets: AID dataset, UCMerced dataset, NWPU-RESISC dataset, and EuroSAT dataset. Results show that our proposed training method was effective and increased overall accuracy (3% in AID experiments, 5% in UCMerced experiments, 1% in NWPU-RESISC and EuroSAT experiments) for small and shallow models. We further explored the performance of the student model on small and unbalanced datasets. Our findings indicate that knowledge distillation can improve the performance of small network models on datasets with lower spatial resolution images, numerous categories, as well as fewer training samples.

1. Introduction

With the rapid development of remote sensing (RS) techniques, a large number of algorithms have been proposed to automatically process massive earth observation data. Scene classification is the one of fundamental procedures of RS images analysis and of much importance in many RS applications, such as land-use/land-cover (LULC) [1,2,3,4], agriculture [5,6,7,8], forestry [9,10], and hydrology [11].

The core task of scene classification is to identify the land-cover type of remotely sensed image patches automatically. Numerous supervised machine learning algorithms have been used in scene classification. These algorithms can be categorized into the following three types [12]: low-level, mid-level and high-level methods. Low-level methods first extract low-level hand-crafted features, including SIFT (scale invariant feature transform) [13,14], HOG (histogram of oriented gradients) [15], structural texture similarity [16], LBP (local binary patterns) [17], Gabor descriptors [18], etc. Features extracted by methods are utilized to train shallow classifiers such as Support Vector Machines (SVMs) [19] or K-Nearest Neighbor algorithms (KNNs) [20] to identify the categories of scene images. These scene classification methods based on low-level features are efficient on some structures and arrangements, but cannot easily depict the highly diverse and non-homogeneous spatial distributions in images [21].Mid-level methods build scene representations by coding low-level local attributes. The bag-of-visual-words (BoVW) model is one of the most frequently used approaches [22]. To improve the classification accuracy, various low-level local descriptors are combined as complemented features in the standard BoVW model [23], Gaussian mixture model (GMM) [24], and other pyramid-based models [25,26,27] for scene classification. In addition, topic models are introduced to combine visual semantic information to encode higher order spatial information between low-level local visual words [28,29,30,31,32]. High-level methods are based on deep learning (DL) models. DL models achieve the state-of-the-art in image recognition, speech recognition, semantic segmentation, object detection, natural language processing [33,34,35,36], and RS image scene classification. Many classic DL models in the field of computer vision (CV) have been shown to be effective in RS scene classification [12,37,38,39,40,41,42,43,44]. Most are based on deep convolutional neural network (CNN) models, such as AlexNet [45], CaffeNet [46], VGGNet [47], deep residual networks (ResNet) [48] and DenseNets [49]. Among these approaches, the CNN-based high-level models outperform the state-of-the-art for scene classification tasks in remote sensing [50]. They can deal with scenes that are more complex and achieve higher overall accuracy by learning deep visual features from large training datasets, in contrast to shallow models and low-level methods that rely on manual feature extraction [51].

However, deep CNNs contain more parameters to train, thus they cost more computational resources and time on training and predicting. For example, a 102-convolutional-layer CNN model, which contains 42.4 M parameters, costs 14 ms to classify a $224\times 224\times 3$ scene image while a simple 4-convolutional-layer CNN model costs 8.77 ms and only contains 1 M parameters, as detailed in Section 3.1 of this paper. This is an unacceptable cost of time and storage space in special situations, such as embedded devices [52,53,54] or during on-orbit processing [55]. In contrast, a small and shallow model is fast and uses little space, but will not yield accurate and precise results when trained directly on ground truth data [33].

Under these circumstances, model compression techniques become imperative. Generalized model compression improves the performance of a shallow and fast model by learning a cumbersome, but better performing model, or by simplifying the structure of the cumbersome network. There are three mainstream types of model compression algorithms: network pruning, network quantization and Teacher-Student Training (TST). Network pruning is a technique that reduces the size of networks by removing neurons or weights that are less important based on certain standard [52,56,57,58], while network quantization attempts to reduce the precision of weights or features [59,60,61]. In contrast, TST methods impart knowledge from a teacher model into a student model by learning distributions or outputs of some specific layers [62,63,64,65,66,67,68,69,70].

TST is easily confused with transfer learning. In transfer learning, we first train a base model on a certain dataset and task, and then transfer the learned features to another target network to be trained on different target dataset and task [71,72]. A common use of transfer learning in the field of remote sensing is to fine-tune an ImageNet-pretrained model on a remote sensing dataset [2,40,73]. In a TST process, however, the teacher and student models are trained on the same dataset.

Knowledge distillation (KD) is one kind of TST method, first defined in [63]. In that paper, authors distill knowledge from an ensemble of models into a single smaller model via high-temperature softmax training. In this paper, we introduce the KD into remote sensing scene classification for the first time to improve the performance of small and shallow network model. We then conducted experiments on several public datasets to verify the effectiveness of KD and make a quantitative analysis to explore the optimum parameter settings. We will also discuss performance of KD on different types of datasets. In addition, we tested whether knowledge can still be distilled in the absence of a certain type of training samples or in the absence of sufficient training data sets. For convenience in our work, we simplified the KD training process by only learning from one cumbersome model. The rest of this paper is organized as follows. Section 2 will describe the teacher-student training method and our knowledge distillation framework. In Section 3, results and analysis of experiments on several datasets are detailed. Our conclusion and future work are discussed in Section 4.

2. Method

As a special case of teacher-student training (TST), knowledge distillation (KD) imitates high-temperature softmax output from the cumbersome teacher model, serving as our training framework. In this section, we first describe the different TST methods, and introduce the KD training methods we adopted in our research.

2.1. Teacher-Student Training

TST is one of the mainstream model compression methods. In TST processing, a cumbersome pre-trained model is regarded as a teacher, the untrained small and shallow model is a student. The student model not only learns hard target from the ground truth data, but also matches its output to the output of the teacher model. This is because the output of a softmax layer (soft target) contains more information than one-hot labeled dataset (hard target).

A general TST process first trains the cumbersome model directly on dataset and then train the student model by minimizing the following loss function using mini-batch stochastic gradient descent (MSGD) [74] method:

$$L_{TST}\left(X\right)=\lambda ·H_1(S\left(X\right),Y_{GT})+H_2(S^{*}\left(X\right),T^{*}\left(X\right))$$

(1)

where X denotes a batch of input data, $S\left(X\right)$ is the softmax output of student model, $Y_{GT}$ is the ground truth label corresponding to the input X, and $S^{*}\left(X\right)$ and $T^{*}\left(X\right)$ are the output of a certain layer in student model and teacher model, $\lambda$ is a non-negative constant. The first term in this loss function is a ground truth constraint. If $\lambda =0$, the supervised information is only provided by the teacher model, instead of the ground truth data. If $\lambda$ goes higher, the output probability distribution of the trained student model is more like the teacher model. $H_1$(X,Y) and $H_2$(X,Y) in Equation (1) can be any common loss functions, such as mean squared error (MSE) or categorical crossentropy (CE):

$$MSE(x,y)=\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^d{\left({\mathbf{x}}_{ik}-{\mathbf{y}}_{ik}\right)}^2$$

(2)

$$CE(x,y)=-\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^d\left[{\mathbf{x}}_{ik}\log \left({\mathbf{y}}_{ik}\right)+\left(1-{\mathbf{x}}_{ik}\right)\log \left(1-{\mathbf{y}}_{ik}\right)\right]$$

(3)

where m denotes the batch size, d denotes the size of input vector $\mathbf{x}$, and ${\mathbf{x}}_{ik}$ indicates the kth element of the ith input samples.

The simplest and the naive method in TST matches the output probability distribution of the last softmax layer (MS) in the student model to the teacher model. In this case, $S^{*}\left(X\right)=S\left(X\right)$ and $T^{*}\left(X\right)$ is the softmax output of the teacher model. Thus, the loss function of MS mode can be defined as:

$$L_{MS}\left(X\right)=\lambda ·CE(S\left(X\right),Y_{GT})+CE(S\left(X\right),T\left(X\right))$$

(4)

To better impart knowledge from teacher to student, Bucilua, C., et al. [62] proposed the matching logits (ML) method. Logits are the inputs to the final softmax layer of a network. Here, the discrepancy among categories in the logits form is more significant than the probabilities form. As a result, the CE of the second term in the loss function are replaced by MSE because the value of a logit can be any real number. The loss function in ML mode now becomes:

$$L_{ML}\left(X\right)=\lambda ·CE(S\left(X\right),Y_{GT})+MSE(S_{logits}\left(X\right),T_{logits}\left(X\right))$$

(5)

where $S_{logits}\left(X\right)$ and $T_{logits}\left(X\right)$ are the logits output of their model. In that work, the authors conducted several experiments and verified that the ML mode could improve small models by learning from model ensembles in simple machine learning tasks.

In addition to these methods as mentioned, various researchers in the field have proposed other means for TST training. Huang, Z., et al. [69] proposed an idea of matching the distributions of neuron selectivity patterns (NST) between two networks, where a new loss function through minimizing the maximum mean discrepancy (MMD) is designed to match these distributions. In [70], the authors compressed wide and shallow networks into thin but deeper networks, the FitNet, by learning intermediate-level hints from teacher’s hidden layers. Yim [68] transfers the knowledge distilled from the flow between layers, computed by the inner product between features, and generated into the FSP matrix. Different from the previous methods, Chen, T., et al. [64] accelerates the training of a larger network by transferring knowledge from a network to the new larger network. Lopez-Paz, D., et al. [66] combines knowledge distillation with privileged information [75], deriving into a generalized distillation.

2.2. Knowledge Distillation Framework

2.2.1. Knowledge from Probability Distribution

The last output layer of neural networks is the softmax layer that transforms the logits $z_i$ into a probability $p_i$ via:

$$p_i=softmax\left(z_i\right)=\frac{\exp \left(z_i\right)}{{\sum }_j\exp \left(z_j\right)}$$

(6)

However, the normal softmax output always leads to an approximate one-hot vector. An example is shown in

Table 1. An example of high-temperature softmax output with different temperature.

Temperature	C1	C2	C3	C4	C5	C6	Entropy 1
1	0.0000	1.0000	0.0000	0.0000	0.0000	0.0000	0.0000
100	0.0523	0.8566	0.0022	0.0320	0.0160	0.0409	0.8764
200	0.1338	0.5416	0.0275	0.1046	0.0741	0.1183	1.9934
500	0.1687	0.2951	0.0896	0.1529	0.1331	0.1606	2.4898
logits	$-304.63$	$-25.01$	$-620.91$	$-353.81$	$-422.91$	$-329.22$

¹ Log2-based entropy.

and Figure 1, which indicates that normal softmax (temperature = 1) made the probability of C2 class tends to one and others tend to zero. The entropy of the output also tends to zero. In practice, a remote sensing scene image consists of several categories of pixels. Thus, information for non-maximum probability categories provides additional supervision for training.

To improve the discrimination ability and generalizability of a model, Hinton, G., et al. [63] introduced high temperature softmax function instead of normal softmax or logits. High temperature softmax function was first used in the field of reinforcement learning [76], denoted as:

$$p_i=\frac{\exp (z_i/T)}{{\sum }_j\exp (z_j/T)}$$

(7)

where T is the temperature. The normal softmax is a special case if $T=1$.

Softmax with high temperature could increase the entropy of the categorical vector which helps to learn more knowledge from the probability distribution of a complex scene. An example of categorical probability distributions of high-temperature softmax output with different temperature is shown in Table 1 and Figure 1. C1∼C6 denote six categories, and the example input logit data are listed in the last row of the table. When $T=1$ (normal softmax), the probability of C2 tends to 1 and others tend to 0. If T goes higher, the entropy of the categorical distribution becomes higher. It can be inferred from this example that if the student model learns high-temperature softmax output from the teacher model, it will distill more knowledge of categorical probability distribution. The experiments in Section 3 will verify this inference.

2.2.2. KD Training Process

By introducing the high-temperature softmax into our framework in former subsection, we divide the whole training process of knowledge distillation into two procedures. First, train the teacher model directly on dataset, which is shown in Figure 2a. The target is to let $T\left(X\right)$, the softmax output of the teacher model, fit the ground truth $Y_{GT}$. Then, distill the knowledge via high-temperature softmax, as shown in Figure 2b. The student model outputs two branches: high-temperature softmax outputs distill knowledge from the teacher model and the normal softmax outputs learn to match the ground truth label. Thus, the total loss of KD process $L_{KD}\left(X\right)$ is:

$$L_{KD}\left(X\right)=\lambda ·CE(S\left(X\right),Y_{GT})+CE(S_T\left(X\right),T_T\left(X\right))$$

(8)

where $S_T$ and $T_T\left(X\right)$ denote the T-temperature softmax output of the student model and the teacher model respectively. In extreme conditions, such as lacking of training samples, the teacher’s high-temperature output can even provide supervision to the student model without any ground truth data (set $\lambda =0$). In prediction or production environment, the trained student model only outputs normal softmax result, as shown in Figure 2c.

As the higher-temperature softmax output from the teacher model contains different information than the ground truth dataset, our KD framework provides the student model with more categorical information in scene classification tasks.

3. Experimental Results and Analysis

In this section, to test the performance of our distillation framework, we conducted several experiments on four remote sensing scene classification datasets: AID dataset [51] (Figure 3a), UCMerced dataset [22] (Figure 3b), NWPU-RESISC dataset [77] (Figure 3c) and EuroSAT dataset [2] (Figure 3d). The general information of each dataset is listed in

Table 2. General information of datasets.

Dataset	Resolution	Size	Category Count	Training Samples	Validation Samples	Test Samples
AID	0.5–8 m	600 × 600	30	5000	2507	2493
UCMerced	0.3 m	256 × 256	21	1050	525	525
NWPU-RESISC	0.2–3 m	256 × 256	45	15,750	7875	7875
EuroSAT	10–60 m	64 × 64	10	16,200	5400	5400

.

For each dataset, we first train a large deep network model and a small shallow one by direct training methods. Then we train the small model by our proposed KD methods. For comparison, other model compression methods including ML were also processed, and we analyzed in detail the experimental results (The implementation of the framework was based on Keras 2.1.1 [78] and TensorFlow 1.4.0 [79]).

3.1. Experiments on AID Dataset

To evaluate the performance and robustness of our proposed KD framework for remote sensing image scene classification, we first designed and conducted several experiments on the AID dataset.

3.1.1. Dataset Description

The AID dataset is a large-scale public data set for aerial scene classification, provided by [51]. It contains 10,000 manually labeled remote sensing scene images from around the world. All images in the AID dataset were collected from Google Earth (https://www.google.com/earth/). Each is 600 × 600 pixels with RGB three spectral bands. The task in our experiments was to classify all scene images into thirty categories. The specific categories are: airport, bare land, baseball field, beach, bridge, center, church, commercial, dense residential, desert, farmland, forest, industrial, meadow, medium residential, mountain, park, parking, playground, pond, port, railway station, resort, river, school, sparse residential, square, stadium, storage tanks, and viaduct.

The dataset was divided into three parts for each category: training data (around 50%), validation data (around 25%) and test data (around 25%). Considered from computational resources, all images were re-sampled from $600\times 600\times 3$ to $224\times 224\times 3$ pixels. In total, there were 5000 images in the training dataset, 2507 as validation data and 2493 in the test dataset.

In the experiments, overall accuracy ($OA$), kappa coefficient (K) [80], precision (P), recall (R) and F1-score ($F1$) of the test dataset were adopted as the accuracy assessment metrics. By introducing the confusion matrix, a table with two rows and two columns that reports the number of true positives ($tp$), false positives ($fp$), false negatives ($fn$), and true negatives ($tn$), we can define these assessment metrics as follows. Overall accuracy ($OA$) is defined as the number of correctly predicted images divided by the total number of predicted images, denoted as:

$$OA=\frac{N_{true}}{N}$$

(9)

while $N_{true}$ denotes the number of correctly predicted images, and N stands for the total number of predicted images. Precision (P) and recall (R) for one-class classification are then defined as:

$$P=\frac{tp}{tp+fp},R=\frac{tp}{tp+fn}$$

(10)

and F1-score is the harmonic mean of the precision and recall, which can be calculated by:

$$F{1}_{oneclass}=\frac{2}{1/R+1/P}=2·\frac{P·R}{P+R}$$

(11)

when it turns to multi-class situation, we use weighted F1-score as our metric:

$$F1=\sum _{k=1}^{K}w\left(k\right)·F{1}_{oneclass}^k,w\left(k\right)=\frac{n_k}{N}$$

(12)

while weight $w\left(k\right)$ is determined by the number of samples of each category, $F{1}_{oneclass}^k$ is correspondingly calculated in each category. Kappa coefficient (K) is another metric that measures inter-rater agreement for categorical items, in multi-class situation, K is defined as:

$$K=\frac{OA-P_e}{1-P_e}$$

(13)

where $P_e$ is the hypothetical probability of chance agreement, calculated as:

$$P_e=\frac{1}{N^2}·\sum _k{n}_{k1}{n}_{k2}$$

(14)

where k denotes the number of categories, and $n_{ki}$ is the number of times rater i predicted category k.

3.1.2. Structure of Networks and Direct Training

We choose the classic 101-layer deep residual network (ResNet-101) [48] as the teacher model (The structure of ResNet-101 can be found at https://github.com/KaimingHe/deep-residual-networks and http://ethereon.github.io/netscope/#/gist/b21e2aae116dc1ac7b50). For comparison, we designed a shallow and simple CNN with four convolutional layers and only one fully connected layer. To prevent over-fitting, we add a Dropout layer [81] between the convolutional layers and fully connected layers. The specific structure of the student model is detailed in Appendix A.1.

Both two models were trained by the common back-propagation (BP) algorithm [82] with a batch size of 24. We adopted Adadelta [83] as the weights updating optimization method. Each model was trained for 100 epochs. All the experimental results on AID dataset were processed on a desktop PC with Intel Core i7 6700K (4C8T), 32GB RAM and Nvidia GeForce GTX1080 Ti (11264MB memory). After training one epoch, the validation OA (VOA) will be recorded and the model achieved the highest VOA was used to make a final accuracy assessment on the test dataset. In our experiments, we used the following data augmentation policies for generalization purpose:

• random scaling in the range [0.8, 1.2];
• random rotation by [−30, 30] degrees;
• random vertically and horizontally flipping.

3.1.3. KD Training and Results

To analyze the knowledge distillation (KD) methods discussed in Section 2, we conducted a series of experiments by training the student model via KD with different T (1, 5, 10, 20, 50 and 100) and $\lambda$ (1 and 0.1) parameters. For comparative purposes, we also did experiments by directly training and matching logits (ML) training the student model on the training dataset. The complete results are listed in

Table 3. Results of Knowledge Distillation training experiments on AID dataset.

				VOA	OA	K	F1	FPS	Parameters
student model	direct training			0.8466	0.8295	0.8234	0.8299	113.96	1,012,830
	ML training	$\lambda$
		1		0.7140	0.7084	0.6980	0.7098
		100		0.8161	0.8087	0.8018	0.8070
	KD training	T	$\lambda$
		1	1	0.8373	0.8448	0.8392	0.8438
		1	0.1	0.8289	0.8287	0.8226	0.8282
		5	1	0.8516	0.8480	0.8426	0.8484
		5	0.1	0.8524	0.8363	0.8305	0.8348
		10	1	0.8500	0.8367	0.8309	0.8352
		10	0.1	0.8560	0.8580	0.8530	0.8593
		20	1	0.8416	0.8355	0.8297	0.8347
		20	0.1	0.8484	0.8383	0.8326	0.8353
		50	1	0.8404	0.8335	0.8276	0.8336
		50	0.1	0.8400	0.8307	0.8247	0.8308
		100	1	0.8536	0.8484	0.8430	0.8478
		100	0.1	0.8444	0.8311	0.8251	0.8304
teacher model (ResNet-101)				0.8888	0.8524	0.8471	0.8542	71.31	42,437,278

(The items in bold in each table mean the optimum results of all.). In addition to accuracy assessment metrics, we also recorded the FPS (frames per second) values in Table 3.

There were three key findings as shown in Table 3:

• KD training is effective. It could increase OA by approximately 3%, compared to the direct training way. However, ML training did not seem to work or even reduce the OA. For further analysis, we draw the validation loss curves and VOA curves of these three types of training methods, which are shown in Figure 4a,b. From training curves, we could find that direct training leads to faster convergence (50 epochs) but falls into local optima while KD training could always reduce the loss.
• The student model learned more knowledge from the teacher model via higher temperature softmax output. Different from T, the effect of $\lambda$ parameter is not clear. When T is 1, 5, 50, or 100, the bigger $\lambda$ the better OA achieved. However, if T is set to 10 or 20, the smaller $\lambda$ value performed better. To further analyze $\lambda$, we drew four subfigures in Figure 5 to demonstrate the relationship between our four metrics and the temperature T. The curves shows that the trend of VOA are similar to other metrics whether $\lambda =0.1$ or $\lambda =1$ on AID dataset.
• From a macro point of view, KD training methods could improve the performance of a network model, in terms of OA, K or F1. In test data evaluation, it even surpassed the deep teacher model by 60% higher speed and using only 2.4% model parameters.

3.1.4. KD Training on Small Dataset

If a student model could learn knowledge from a teacher model via KD training on the complete dataset, it should also work on a small part of the dataset. To verify this idea, we implemented extra experiments of KD training on a small part of AID dataset. We took 20% training samples (1000 images) and 20% validation data (507 images) of the original AID dataset and evaluated accuracy on the complete test dataset.

The results are shown in

Table 4. Results of training the student model on part of AID dataset

Dataset	Training Method	VOA	OA	K	F1	Training Time per Epoch (s)
100%	direct training	0.8466	0.8295	0.8234	0.8299	128.1
20%	direct training	0.7219	0.5872	0.5728	0.5929	32.0
	KD training (T = 10, $\lambda$ = 0.1)	0.7298	0.6554	0.6432	0.6567
	KD training (T = 10, $\lambda$ = 1.0)	0.7416	0.6221	0.6089	0.6234

. As shown, small dataset lead to a shorter training time (It includes validation time.) but poorer training results. However, the KD training was still better than direct training under such conditions. In addition, if we decreased $\lambda$, the weight of the first term in the loss function (Equation (8)), the generalizability of the student model would be greatly enhanced.

3.1.5. Remove One Category

Instead of training on a small dataset, we then removed all image samples of the “airport” category in training data and validation data in this experiment. The test results after KD training ($T=10$ and $\lambda =0.1$) 100 epochs, are shown in

Table 5. Results of Knowledge Distillation training experiments on AID dataset without Airports.

Category Name	Direct Training			KD Training without Airport			Fine-Tuned on Unlabeled Data
	P	R	F1	P	R	F1	P	R	F1
airport	0.7359	0.8667	0.7959	0.9130	0.2333	0.3717	0.7234	0.7556	0.7391
average	0.8421	0.8295	0.8299	0.8461	0.8363	0.8306	0.8588	0.8420	0.8438

.

From the results, in the absence of “airport” samples, the KD training process still achieved better F1-score than direct training on complete dataset. From the point of view of the distilled model, it has never seen “airport” before. However, it got high precision (0.913), although the recall is very low (0.2333). If we continued to fine-tune the student model only 10 epochs on the unlabeled data (2493 images with 90 airports, all labels were removed when fine-tuning) via KD, the F1-score of Airport class increased from 0.3717 to 0.7391 and the average F1-score increased by approximately 1.7%.

3.1.6. The Relationship between the Optimal Temperature and the Number of Categories

In KD training process, the temperature (T) is a significant factor. When $T=1$, the output probability vector tends to an one-hot vector. However, if T approaches infinity, the output probability of all categories tends to the same, which made the student model hard to learn from the teacher model. Intuitively, the more categories, the more information the probability vector contained in the outputs of a model. Therefore, it is easy to speculate that the optimal temperature for KD training is negatively related to the number of categories.

To evaluate the relationship between the optimal temperature and the number of categories, we conducted extra KD training experiments on three subsets of the AID dataset. These three AID subsets contain 25, 20, and 15 categories separately, constructed by removing 5, 10, and 15 categories in the original AID dataset. In experiments, the $\lambda$ is a constant with a value of 0.1 and the range of temperature T is [1, 100]. The structure model and the teacher model are the same as before, except their output softmax layers (The output of the last softmax layer has the same size as the number of categories).

Results of these experiments are shown in

Table 6. Results of Knowledge Distillation training experiments with different numbers of category and temperatures on AID dataset.

Student Model	T	30 Categories			25 Categories			20 Categories			15 Categories
		OA	K	F1	OA	K	F1	OA	K	F1	OA	K	F1
direct training		0.830	0.823	0.830	0.840	0.833	0.840	0.870	0.863	0.870	0.878	0.869	0.876
KD training	1	0.829	0.823	0.828	0.837	0.830	0.837	0.863	0.856	0.863	0.884	0.875	0.883
	5	0.836	0.831	0.835	0.837	0.830	0.833	0.872	0.865	0.872	0.869	0.860	0.869
	10	0.858	0.853	0.859	0.846	0.839	0.846	0.868	0.861	0.867	0.879	0.870	0.879
	20	0.838	0.833	0.835	0.841	0.835	0.841	0.871	0.864	0.870	0.880	0.871	0.880
	50	0.831	0.825	0.831	0.851	0.845	0.850	0.870	0.863	0.871	0.871	0.861	0.870
	100	0.831	0.825	0.830	0.839	0.832	0.839	0.873	0.866	0.873	0.890	0.882	0.890
teacher model		0.852	0.847	0.854	0.870	0.865	0.871	0.880	0.874	0.880	0.894	0.886	0.893

. According to the metrics (OA, K, and F1-score), the optimal temperature for KD training is 10 for the complete AID dataset, 50 for the 25-category subset, 100 for the 20-category and 15-category subset. It is obvious that the optimal T increased as number of categories decreased on AID dataset, which verified our speculation.

3.1.7. Evaluating Our Proposed KD Method on AID Dataset

We detailed a series of experiments to evaluate KD training framework on AID dataset. As shown by the results, we found that KD could increase OA than direct training and ML training, and achieved the optimum performance when $T=10$ and $\lambda =0.1$. We then did same experiments on only 20% of training data in the AID dataset. It also showed that KD training was still better than direct training. From these results, we infer that our framework could still distill knowledge of a certain category even there is no sample from that category. In addition, the student model got better results by continuous fine-tuning on unlabeled samples via unsupervised KD training. This further verifies that the knowledge of the probability distribution within classes can be effectively distilled. Moreover, we found the optimal temperature for KD training is negatively related to the number of categories on AID dataset.

3.2. Additional Experiments

To evaluate the effectiveness and applicability of KD framework, we conducted additional experiments on three different datasets of remote sensing scenes: UCMerced dataset, NWPU-RESISC dataset and EuroSAT dataset.

3.2.1. Experiments on UCMerced Dataset

The UCMerced dataset is a widely-used remotely sensed image scene dataset, which consists of a total of only 2100 image patches each of size $256\times 256$ with a ground sample distance (GSD) of 0.3 m and covering 21 land-cover classes [22]. Images in UCMerced dataset were extracted from the USGS National Map Urban Area Imagery collection for various urban areas around the country. We split UCMerced dataset into three parts: 1050 images for training, 525 images for validating and 525 images for testing, respectively. Due to a relatively small number of training samples, we adopt two four-convolutional-layer CNN models as the teacher model and the student model to avoid overfitting. The specific structures of two models are shown in Appendix A.2. The other training settings are the same as the AID experiments discussed in Section 3.1.

As we can see from

Table 7. Results of Knowledge Distillation training experiments on UCMerced dataset.

				OA	K	F1	FPS	Parameters
student model	direct training			0.6838	0.6680	0.6710	334.40	66,933
	KD training	T	$\lambda$
		1	1	0.6933	0.6780	0.6918
		1	0.1	0.7105	0.6960	0.7079
		5	1	0.7200	0.7060	0.7201
		5	0.1	0.6933	0.6780	0.6820
		10	1	0.7124	0.6980	0.7092
		10	0.1	0.6990	0.6840	0.6871
		20	1	0.6838	0.6680	0.6787
		20	0.1	0.7067	0.6920	0.7063
		50	1	0.7181	0.7040	0.7098
		50	0.1	0.7067	0.6920	0.6993
		100	1	0.7352	0.7220	0.7338
		100	0.1	0.6724	0.6560	0.6691
teacher model				0.8438	0.8360	0.8402	137.12	3,658,389

, KD training seems to be effective on UCMerced dataset as almost all KD experiment results point to be better than the direct training result of the student model. In accordance with all the metrics, KD training achieves its optimal performance when T = 100 and $\lambda$ = 1. Similar to our experiment results on small AID dataset, KD method is still practicable to small datasets.

3.2.2. Experiments on NWPU-RESISC Dataset

NWPU-RESISC is a large-scale benchmark dataset for remote sensing scene classification, covering 45 scene classes with 700 images in each class. Images within each class are with a size of $256\times 256$ pixels in the red-green-blue(RGB) color space, while the spatial resolution of those images vary from about 30 to 0.2 m per pixel. The 31,500 images, extracted by experts from Google Earth, cover more than 100 countries and regions all around the world. 15,750 of them are training samples, 7875 of them are validation samples, and the rest 7875 images are test samples. The teacher model and the student model are the same as the experiments on AID dataset, except the output layers.

Results of KD training experiments on NWPU-RESISC dataset are recorded in

Table 8. Results of Knowledge Distillation training experiments on NWPU-RESISC dataset.

				OA	K	F1	FPS	Parameters
student_model	direct training			0.7896	0.7848	0.7894	303.67	1,389,165
	KD training	T	$\lambda$
		1	1	0.7911	0.7864	0.7904
		1	0.1	0.7705	0.7653	0.7705
		5	1	0.7945	0.7899	0.7923
		5	0.1	0.8000	0.7955	0.7983
		10	1	0.7945	0.7899	0.7940
		10	0.1	0.7926	0.7879	0.7912
		20	1	0.7939	0.7892	0.7933
		20	0.1	0.7873	0.7825	0.7862
		50	1	0.7907	0.7860	0.7891
		50	0.1	0.7893	0.7845	0.7880
		100	1	0.7915	0.7868	0.7907
		100	0.1	0.7912	0.7865	0.7892
teacher model (ResNet-101)				0.8703	0.8674	0.8692	88.90	42,468,013

. Performances of KD training methods with different settings of T and $\lambda$ seem to be close on NWPU-RESISC dataset, it performs better when T = 5 and $\lambda$ = 0.1. Different from those datasets with fewer categories, NWPU-RESISC contains 45 classes, including land-use and land-cover classes, which is challenging with high within-class diversity and between-class similarity. KD method proves to be effective on NWPU-RESISC dataset with 45 categories.

3.2.3. Experiments on EuroSAT Dataset

EuroSAT dataset is another widely-used scene classification dataset based on medium-resolution satellite images covering 13 different spectral bands, and consisting of 10 different land-use and land-cover classes. It contains 27,000 images, each image patch measures $64\times 64$ pixels, with a ground sample distance (GSD) varying from 10 to 60 m. Data in EuroSAT dataset is gathered from satellite images of cities in over 30 European countries. Like previous experiments, EuroSAT dataset was split into three parts: 16,200 images for training, 5400 images for validating and 5400 images for testing, respectively. In experiments, we only exploited red, green, blue three spectral bands as the input of networks. The teacher model and the student model (Appendix A.1) are the same as the experiments on AID dataset, except the output layers.

Table 9. Results of Knowledge Distillation training experiments on EuroSAT dataset.

				OA	K	F1	FPS	Parameters
student model	direct training			0.9333	0.9258	0.9336	440.00	511,050
	KD training	T	$\lambda$
		1	1	0.9363	0.9291	0.9360
		1	0.1	0.9363	0.9291	0.9362
		5	1	0.9376	0.9305	0.9376
		5	0.1	0.9300	0.9221	0.9299
		10	1	0.9352	0.9279	0.9352
		10	0.1	0.9335	0.9260	0.9334
		20	1	0.9389	0.9320	0.9388
		20	0.1	0.9283	0.9203	0.9286
		50	1	0.9320	0.9243	0.9319
		50	0.1	0.9391	0.9322	0.9389
		100	1	0.9430	0.9365	0.9429
		100	0.1	0.9398	0.9330	0.9397
teacher model (ResNet-101)				0.9474	0.9415	0.9471	98.93	42,396,298

shows the results of KD training experiments on EuroSAT dataset. All of these models easily achieve outstanding OA over 90%, and KD training model reaches optimum when T = 100 and $\lambda$ = 1. Compared with datasets such as AID and NWPU-RESISC, KD methods still seem to work well on EuroSAT dataset that contains smaller-patch-size and lower-spatial-resolution images.

3.2.4. Discussions

Through conducting additional experiments on three other different datasets of remote sensing images with different settings of T and $\lambda$, we evaluate the effectiveness of KD method and find that it ends up with different results on different datasets. In general, KD method is practicable to datasets with lower spatial resolution images, numerous categories, as well as fewer training samples.

4. Conclusions

In this work, we introduced knowledge distillation framework to improve the performance of small neural networks in the field of remote sensing scene image classification. The core concept behind this framework is to let the small model learn the categorical probability distribution from the pre-trained and well-performed cumbersome model via matching high-temperature softmax output. To evaluate this framework, we conducted several experiments on the AID datasets. The experimental results showed that the KD framework was effective and increased overall accuracy (3% in AID experiments, 5% in UCMerced experiments, 1% in NWPU-RESISC and EuroSAT experiments) for small models and knowledge could be well distilled via high-temperature softmax. In experiments on AID dataset, we also found that the KD training framework helped to train networks on small or unbalanced datasets. In addition, based on the experimental results on AID dataset, we initially concluded that if the dataset contains fewer categories, the KD framework needs a larger temperature-value T to achieve better results. Moreover, to test the effectiveness and applicability of our framework, we conducted experiments on three different datasets of remote sensing scenes: UCMerced dataset, NWPU-RESISC dataset and EuroSAT dataset. The results of these additional experiments show that KD method can improve the performance of small network models on datasets with lower spatial resolution images, numerous categories, as well as fewer training samples.

In the future, we plan to investigate how to integrate KD framework with other model compression methods. Another interesting opportunity for future work is to apply KD framework to other fields of remote sensing, such as semantic segmentation and object detection.