Rotation Invariance Regularization for Remote Sensing Image Scene Classification with Convolutional Neural Networks

Abstract

Deep convolutional neural networks (DCNNs) have shown significant improvements in remote sensing image scene classification for powerful feature representations. However, because of the high variance and volume limitations of the available remote sensing datasets, DCNNs are prone to overfit the data used for their training. To address this problem, this paper proposes a novel scene classification framework based on a deep Siamese convolutional network with rotation invariance regularization. Specifically, we design a data augmentation strategy for the Siamese model to learn a rotation invariance DCNN model that is achieved by directly enforcing the labels of the training samples before and after rotating to be mapped close to each other. In addition to the cross-entropy cost function for the traditional CNN models, we impose a rotation invariance regularization constraint on the objective function of our proposed model. The experimental results obtained using three publicly-available scene classification datasets show that the proposed method can generally improve the classification performance by 2~3% and achieves satisfactory classification performance compared with some state-of-the-art methods.

1. Introduction

Remote sensing image scene classification [1] has been widely used in many practical applications such as urban planning, environment monitoring, and natural hazard detection [2,3,4,5,6]. The goal of scene classification is to assign a unique label (e.g., airport or playground) to a query remote sensing image. However, scene classification is a challenging problem due to the complex spatial structure and land cover categories diversity in remote sensing scene images. Thus, many scene classification methods have been proposed over the past years [7,8,9,10,11,12].

Recently, many deep convolutional neural network (DCNN)-based models have been proposed [13,14,15,16], replacing scene classification based on handcrafted-features such as color histograms [17], scale-invariant feature transform (SIFT) [18], histogram of oriented gradients (HOG) [19], and global image descriptor (GIST) [20]. These models can achieve better classification performance due to the powerful feature representation and generalization ability of pre-trained DCNN models (e.g., AlexNet [21], VGG-VD16 [22], GoogLeNet [23], and ResNet [24]) on the ImageNet [25]. However, although these methods have significantly improved the classification performance, two challenging problems still remain in remote sensing scene classification: the small scale of labeled training data has limited the potential of DCNN-based methods for scene classification due to the overfitting problem; the orientations of geospatial objects are diverse because remote sensing images are taken from the upper airspace, but objects in the nature scene images generally show small orientation variations due to the gravity of Earth [26].

To address this problem, we propose an effective and robust approach to learn a rotation-invariant DCNN model for remote sensing scene classification that is achieved by adopting a Siamese DCNN [27,28] architecture and introducing a new rotation invariance regularization (RIR) on the basis of the existing successful DCNN architectures (e.g., VGG-VD16 [22]). Our architecture has two identical DCNN layers that combine the identification and contrastive modules. The identification module is the traditional DCNN model that accepts an input image and predicts its label by optimizing the cross-entropy objective. The contrastive module compares the outputs of the final softmax layer of original and rotated images and minimizes the Jensen–Shannon divergence [29] in the probability distribution space. As shown in Figure 1, the scene images with different rotation angles contain the same information, and thus it should produce the same outputs including the similarity of the other categories. However, because the probability distributions has the correct class at a very high probability, with all other class probabilities very close to 0, the Kullback–Leibler (KL) divergence of images with the same category will be very small. Therefore, we take the soft labels obtained by the “softmax temperature” [30] to provide more information as to which classes of the original image are more similar to the rotated image. Comprehensive evaluations on three scene datasets and comparisons with state-of-the-art methods, including traditional DCNN architectures without RIR, demonstrate the effectiveness of the proposed method.

The remainder of this paper is organized as follows. We briefly review related works on DCNN-based scene classification and rotation invariance features in Section 2. In Section 3, we describe the proposed method in detail. The experimental datasets and setup are shown in Section 4. Section 5 reports comprehensive results with exhaustive comparison and discussions. Finally, conclusions are drawn in Section 6.

2. Related Work

Remote sensing scene classification has been investigated by a wide variety of methods in recent years. In this section, we briefly review the existing DCNN-based scene classification algorithms and rotation invariance in DCNN.

2.1. DCNN-Based Scene Classification

The goal of remote sensing classification is to categorize scene images into a discrete set of meaningful land use classes according to the image contents. In recent years, various deep learning methods, particularly DCNNs [21,22,31,32,33,34], have shown powerful feature representation and attracted increasing research attention. However, the small scale of remote sensing scene datasets makes training a DCNN model from scratch unrealistic. Fortunately, DCNNs pre-trained on large natural image datasets (e.g., ImageNet [25]) show a powerful generalization ability in different domains [6,7,8,9,11,12].

Despite their effectiveness, it is problematic to directly use DCNN features for remote sensing scene classification because the method is sensitive to the object orientation. This can can lead to misclassification of scene images that are of the same category with different orientations and therefore influences the classification performance.

2.2. Rotation Invariance Features

During recent decades, most of the works for scene classification were based on handcrafted features such as color histograms [17,35], gray level co-occurrence matrix (GLCM) [36], Gabor feature [37], local binary patterns (LBPs) [38], SIFT [18], GIST [20], and HOG [19]. The global features (e.g., color histogram, GLCM, Gabor feature, LBPs, and GIST) that describe the overall image as a whole to generalize the entire object or scene are generally rotation-invariant. Thus, they can be directly used for scene classification. By contrast, the local features (e.g., SIFT and HOG) that describe the image patches (key points in the image) are usually applied to construct global representation such as the bag-of-visual-words (BoVW) models [2,39,40,41]. These middle-level BoVW methods represent each image as a frequency histogram features, so that these features are rotation-invariant for scene classification.

These human-engineered features that require domain expertise greatly limit the representation capability for the complex scene images. Inspired by the recent success achieved by DCNNs in the computer vision community, many CNN models have been extended for various applications including remote sensing scene classification and have shown impressive feature representation capability [11,15,42,43,44]. However, these DCNNs are not actually invariant to large rotations of the input data [45] and this inevitably influences the performance of DCNN features for scene classification.

To overcome this limitation, many investigations of invariant features have been carried out based on DCNNs [14,26,46,47,48,49,50,51]. For examples, TI-Pooling [47] utilizes the tiny rotation invariance of max-pooling that considers a set of rotation angles to find the optimal canonical instance. The concentric circle pooling method [14] introduces the concentric circle-based partition strategy [50,52] in the context of DCNN-based models for scene classification. The spatial transformer network (STN) [53] includes an additional network module that can rotate or scale an input image or feature map to remove rotation variance, but the problem of the complex transformation of parameters using a CNN has not been solved well to date [54]. In addition to these architecture design methods, data augmentation-based methods can more directly learn rotation invariance information by rotating the image at random orientation [49,51,55,56]. Furthermore, this approach can alleviate the overfitting problem that arises from the inadequate available samples. Zhou et al. [51] propose a novel data augmentation strategy based on siamese DCNN to learn rotation-invariant features. The rotation-invariant convolutional neural networks (RICNN) [49] present a rotation-invariant layer on the basis of the existing CNN architecture to address this problem. These two methods introduce additional middle layers and impose regularization constraints to enforce the input and its rotated images with similar features.

Different from the previous methods, we propose a novel rotation invariance regularization network that we call RIR-net by using Siamese DCNNs [27] and soft label [30] for DCNN-based remote sensing scene classification methods. Specifically, we utilize Siamese DCNNs to combine identification and constrastive models that can learn in parallel to recognize the same images with different rotation orientations. The soft label is proposed in knowledge distillation to decrease the gap between the probabilities of different classes that can measure the similarity among the classes for the input and its rotated images. Soft labels contain more information than binary hard labels. We can further reduce the amount of overfitting by forcing the model to contain co-label effects during training. Unlike the previous DCNN-based scene classification methods, our proposed methods can be directly applied to existing DCNNs and effectively handle the problems of rotation variations in remote sensing scene classification.

3. Proposed Method

We utilize Siamese CNNs to learn rotation invariance for remote sensing scene classification. Figure 2 presents the overall architecture. The network consists of two identical pre-trained CNN models that share weights and predict the categories of the input image pair simultaneously. The CNN models can either be the popular CNN architectures such as AlexNet [21], VGG-VD16 [22], GoogLeNet [23], and ResNet [24], or other network architectures. The loss of the whole network includes two parts: identification and regularization losses. For the identification loss, we use the cross-entropy loss to measure the distance between the probability outputs and labels for the input and its rotated images. For the regularization loss, we propose the RIR method to alleviate the rotation variation problem on the basis of data augmentation. The overall framework of our proposed method consists of two modules: data augmentation and rotation invariance regularization. We next describe these two modules in detail.

3.1. Data Augmentation

For the limited size of the available samples for remote sensing scene classification, data augmentation is an effective approach to increase the number of training sets and assist DCNNs to avoid overfitting. Rather than using a range of transform operations such as shifts, flips, and zooms, we only adapt rotation and color jitter augmentations to verify the effectiveness of our proposed methods. For simplicity and consistency, we randomly rotate the input images to generate input image pairs. The color channels of the input images are randomly jittered to avoid shortcuts due to chromatic aberration [57]. The horizontal and vertical flipping augmentations certainly can be used with rotation and color jitter augmentations.

Specifically, we denote a rotation transformed image $x^r=g\left(x\right|{\theta }_r)$, where $g\left(x\right|{\theta }_r)$ is the operator that applies the rotation transformation with angle ${\theta }_r$ to a scene image x. The images at the two rotated angles of each image pair should be different. The CNN models are pre-trained on large natural image datasets, ImageNet [25], to avoid overfitting given the small scale of the remote sensing data sets.

3.2. Rotation Invariance Regularization

Based on the pre-trained DCNN models, we apply Siamese CNNs to realize our RIR method. For these DCNN models, we replace the final 1000-way softmax classification layer with an C-way softmax classification layer, where C denotes the number of categories. The whole architecture takes a pair of images as the input that are the same image with different rotation angles. Different from these baseline DCNNs, the whole network is trained by optimizing a new objective function that consists of two cross-entropy losses and one RIR term. The proposed regularization constraint term is imposed to enforce the training samples rotated at different angles to produce identical probability distribution over classes.

However, in these baseline DCNNs, the correct answers with very high confidence are always generated during the training phase. While we seek to directly minimize the distance of the probability distribution of the image pair, it will be very small since the probabilities are so close to one or zero. In addition, much of the information that resides in the ratios of very small probabilities fails to be used for the learned function. For example, a category of forest may be given a probability of ${10}^{-3}$ of being an agricultural category and of ${10}^{-4}$ being a golf course category. This indicates that the characteristic of the forest category, at least for some images in this category, are similar to those of the agricultural and golf course categories. This is valuable information but it does not influence the cross-entropy loss function because the target labels are hardly 0 or 1. Inspired by the “distillation” in [30], we produce soft labels by raising the temperature of the final softmax in the RIR term.

In the proposed RIR methods, the parameters of all of the layers except for the final softmax layer are transferred from the baseline DCNNs to avoid overfitting in scene classification. Then, we fine-tune the pre-trained layers with a smaller learning rate and a higher learning rate for the last randomly initialized softmax layer. For a pair of input images ($x_i,x_i^r$), let $z_i$ and $z_i^r$ denote the outputs of the final layer for each class. Thus, the probabilities for each class, $p_i\left[c\right]$ and $p_i^r\left[c\right]$, can be computed by

$$\begin{array}{c}\hfill p_i\left[c\right]=\phi (z_i\left[c\right];T)=\frac{e^{(z_i\left[c\right]/T)}}{{\sum }_{j=1}^C{e}^{(z_i\left[j\right]/T)}}\\ \hfill p_i^r\left[c\right]=\phi (z_i^r\left[c\right];T)=\frac{e^{(z_i^r\left[c\right]/T)}}{{\sum }_{j=1}^C{e}^{(z_i^r\left[j\right]/T)}},\end{array}$$

(1)

where T is the temperature, and C is the number of categories. $\phi \left(x\right)$ will be the normal “softmax” with the temperature T of 1 and will produce a softer probability distribution over classes with a high value for T. This is often used in knowledge distillation to transfer knowledge to the distilled model. We introduce it to mine the rotation invariance of DCNNs for scene classification.

We define the training samples $X={\left\{x_i\right\}}_{i=1}^N$, their corresponding rotation transformed samples $X_r={\left\{x_i^r\right\}}_{i=1}^N$, and labels $Y={\left\{y_i\right\}}_{i=1}^N$, where $y_i$ denotes the target possibilities of sample $x_i$ with one-hot encoding and N denotes the number of training samples. To achieve rotation invariance for any set of the training sample pairs ($x_i,x_i^r$), we propose a new objective function with RIR by the following formula:

$$L(X,Y)=\lambda L_{id}(X,Y)+(1-\lambda )R(X,X_r),$$

(2)

where $L_{id}$ and R are the losses of the identification and contrastive terms, and $\lambda$ denotes the tradeoff parameter that tunes the weighted average between the two components of the loss. The first component $L_{id}$ forces the optimization toward approximating the ground truth labels for the training samples, whereas the second component $R(X,X_r)$ forces the optimization toward a similarly softened softmax distribution between the pair of the training samples before and after the rotation. This model will degenerate to the traditional DCNN with rotation augmentation operation while $\lambda =1$.

We use the cross-entropy loss for the identification term and RIR for the contrastive term. The identification term $L_{id}$ adopts softmax as the predicted classification probabilities. For C categories, we can compute $L_{id}$ by

$$\begin{array}{cc}\hfill L_{id}(X,Y)=& -\frac{1}{N}\sum _{i=1}^N(〈y_i,log\left(\phi (z_i,1)\right)〉+〈y_i,log\left(\phi (z_i^r,1)\right)〉)\hfill \\ \hfill =& -\frac{1}{N}\sum _{i=1}^N\sum _{c=1}^C(y_i\left[c\right]·log\left(\phi (z_i,1)\right)\left[c\right]+y_i\left[c\right]·log\left(\phi (z_i^r,1)\right)\left[c\right]),\hfill \end{array}$$

(3)

where $〈y,log\left(\phi \right(f\left(x\right),1\left)\right)〉$ is the cross-entropy loss for the sample $(x,y)$, and $\phi \left(f\right(x),1)$ is the normal softmax score that can be viewed as a special case of the temperature $T=1$.

The contrastive term $R(X,X_r)$ is a RIR constraint. We enforce the output classification probabilities of the original and rotated training samples (X and $X_r$) to be as identical as possible. We use the Jensen–Shannon divergence [29] to measure the similarity between the predicted probability distribution of the input image pairs. The regularization constraint term is defined as

$$\begin{array}{cc}\hfill R(X,X_r)=& \frac{1}{N}\sum _{i=1}^{N}JS(p_i\left|\right|p_i^r)\hfill \\ \hfill =& \frac{1}{2N}\sum _{i=1}^N\left(KL\right(p_i\left|\right|m_i)+KL(p_i^r\left|\right|m_i\left)\right),\hfill \end{array}$$

(4)

where $KL(·)$ denotes the Kullback–Leibler divergence [58] that is a measure the differences between the probability distributions, $JS(·)$ denotes the Jensen–Shannon divergence, $m_i$ are the average probabilities of $p_i$ and $p_i^r$, and are computed by

$$m_i=\frac{p_i+p_i^r}{2}=\frac{\phi (z_i,T)+\phi (z_i^r,T)}{2}.$$

(5)

Due to the cross-entropy loss in identification term, the regularization constraint term requires a large T to become effective. However, noise may be introduced by a large T and this will enlarge the probabilities of too many categories in the output classification vector. Thus, we multiple $R(X,X_r)$ by $2·T^2$, to obtain

$$\begin{array}{cc}\hfill R(X,X_r)=& \frac{1}{N}\sum _{i=1}^{N}JS(p_i\left|\right|p_i^r)·(2·T^2)\hfill \\ \hfill =& \frac{T^2}{N}\sum _{i=1}^N\left(KL\right(p_i\left|\right|m_i)+KL(p_i^r\left|\right|m_i\left)\right)\hfill \\ \hfill =& \frac{T^2}{N}\sum _{i=1}^N\sum _{c=1}^C(p_i\left[c\right]·log\frac{p_i\left[c\right]}{m_i\left[c\right]}+p_i^r\left[c\right]·log\frac{p_i^r\left[c\right]}{m_i\left[c\right]}).\hfill \end{array}$$

(6)

To prevent division by 0 in $KL(·)$, we add a small constant $ϵ$, e.g., $1^{-10}$, to the probabilities $p_i$ and $p_i^r$. By incorporating Equations (3) and (6), the loss function is given by

$$\begin{array}{cc}\hfill L(X,Y;\lambda ,T)=& \frac{-\lambda }{N}\sum _{i=1}^N\sum _{c=1}^C(y_i\left[c\right]·log\left(\phi (z_i,1)\right)\left[c\right]+y_i\left[c\right]·log\left(\phi (z_i^r,1)\right)\left[c\right])+\hfill \\ \hfill & \frac{(1-\lambda )T^2}{N}\sum _{i=1}^N\sum _{c=1}^C((p_i\left[c\right]+ϵ)·log\frac{p_i\left[c\right]+ϵ}{m_i\left[c\right]+ϵ}+\hfill \\ \hfill & (p_i^r\left[c\right]+ϵ)·log\frac{p_i^r\left[c\right]+ϵ}{m_i\left[c\right]+ϵ}).\hfill \end{array}$$

(7)

The loss function $L(X,Y;\lambda ,T)$ defined in Equation (7) imposes a regularization constraint to achieve rotation invariance on basis of the existing DCNN architectures. The gradient for each output unit of the input image $x_i$, $\partial L/\partial z_i\left[c\right]$, can be computed by

$$\begin{array}{cc}\hfill \frac{\partial L(X,Y;\lambda ,T)}{\partial z_i\left[c\right]}& =\frac{\lambda }{N}\sum _{i=1}^N(\phi (z_i,1)\left[c\right]-y_i\left[c\right])+\hfill \\ \hfill & \frac{(1-\lambda )T^2}{N}\sum _{i=1}^N\sum _{j=1}^C(\frac{1}{T^2}{p}_i\left[j\right]·m_i\left[c\right]-\frac{1}{T^2}{p}_i\left[c\right])\hfill \\ \hfill & =\frac{\lambda }{N}\sum _{i=1}^N(\phi (z_i,1)\left[c\right]-y_i\left[c\right])+\hfill \\ \hfill & \frac{1-\lambda }{N}\sum _{i=1}^N\sum _{j=1}^C(p_i\left[j\right]·m_i\left[c\right]-p_i\left[c\right]).\hfill \end{array}$$

(8)

We can minimize this loss function by using the stochastic gradient descent (SGD) [59] or adaptive moment estimation (Adam) [60] methods.

The RIR model can be expanded from two input images to multiple input images with different rotation transformations. Then, the regularization constraint term becomes

$$R(X,X_r)=\frac{1}{KN}\sum _{i=1}^N\sum _{k=1}^K\left(KL\right(p_i^{r_k}\left|\right|m_i\left)\right),$$

(9)

where K is the total number of rotation transformations for each $x_i$, and $p_i^{r_k}$ is the soft probability of the image $x_i^{r_k}$, which is the version of the input image $x_i$ rotated by angle ${\theta }_{r_k}$. The average probability $m_i$ can be computed by

$$m_i=\frac{1}{K}\sum _{k=1}^K{p}_i^{r_k}=\frac{1}{K}\sum _{k=1}^K\phi (z_i^{r_k},T).$$

(10)

Then, we can compute the loss function $L(X,Y;\lambda ,T)$ by

$$\begin{array}{cc}\hfill L(X,Y;& \lambda ,T)=\frac{-\lambda }{KN}\sum _{i=1}^N\sum _{c=1}^C\sum _{k=1}^K(y_i\left[c\right]·log\left(\phi (z_i^{r_k},1)\right)\left[c\right])+\hfill \\ \hfill & \frac{(1-\lambda )T^2}{KN}\sum _{i=1}^N\sum _{c=1}^C\sum _{k=1}^K((p_i^{r_k}\left[c\right]+ϵ)·log\frac{p_i^{r_k}\left[c\right]+ϵ}{m_i\left[c\right]+ϵ}).\hfill \end{array}$$

(11)

For the input image $x_i^{r_k}$, the gradient $\partial L/\partial z_i^{r_k}\left[c\right]$ of each output unit is same as Equation (8) besides the value of average probability $m_i$. Due to the stability of mean value, a larger value of K gives better performance, but requires greater computational capability and iterations.

The proper value of T is important for network performance. If T is too high, it disturbs the identification term to affect classification accuracy. However, if T is too small, we cannot benefit from the regularization term. Inspired by the deterministic annealing process [61], we can slowly increase the value of $T\left(s\right)$ by the step s to benefit from the RIR without impacting the classification performance.

$$T\left(s\right)=\left\{\begin{array}{cccc}\hfill & 1\hfill & \hfill & s\le S_1\hfill \\ \hfill & \frac{s-S_1}{S_2-S_1}(T_f-1)+1\hfill & \hfill & S_1\le s\le S_2\hfill \\ \hfill & T_f\hfill & \hfill & S_2\le s,\hfill \end{array}\right.$$

(12)

where s is the current step, $S_1$ and $S_2$ are the steps at which the increase in the temperature is started and stopped, respectively, for the optimizer in this paper. We evaluate $T_f$ together with the tuning parameter $\lambda$.

4. Experiments

4.1. Data Sets

We evaluate the performance of our proposed method on three challenging remote sensing scene image datasets. These are the UC Merced land use (UC) dataset [39], the Aerial Image dataset (AID) [62], and the NWPU-RESISC45 (NWPU) dataset [1].

The UC (http://vision.ucmerced.edu/datasets/landuse.html) [39] dataset contains 21 typical remote sensing scene categories. There are 100 images with a size of $256\times 256$ pixels for each class in the red–green–blue color space. The pixel resolution of images is approximately one foot. Two examples of ground truth images from each class are shown in Figure 3. The high interclass similarity among the categories (e.g., medium and dense residential areas) results in some challenges for the classification of the UC dataset. The AID (http://www.lmars.whu.edu.cn/xia/AID-project.html) [62] dataset contains 10,000 images with a size of $600\times 600$ pixels within 30 scene classes. The number of images for different categories varies from 220 to 420. The spatial resolution changes from approximately 8 m to 0.5 m per pixel. Figure 4 shows examples of the AID dataset. The NWPU (http://www.escience.cn/people/JunweiHan/NWPU-RESISC45.html) [1] dataset contains 31,500 images with a size of $256\times 256$ pixels. There are 45 scene classes, and each class consists of 700 images. The spatial resolution changes from approximately 30 m to 0.2 m per pixel for most of the scene categories. The classification of the NWPU dataset is more challenging because of the rich image variations, large within-class diversity, and high between-class similarity.

4.2. Experimental Setup

We consider three kinds of widely used DCNNs, including AlexNet [21], VGG-VD16 [22], and ResNet50 [24] as baselines to evaluate the performance improvement for the RIR. We utilize these models pre-trained on ImageNet dataset and then fine-tune them to adopt them to the remote sensing scene images. We replace the flatten layer in the AlexNet [21] and VGG-VD16 [22] with an adaptive average pooling layer for arbitrary input image size. Thus, the original image sizes are used for these data sets.

We randomly select proportional samples of each class for training, and the remaining samples are used for testing 10 times. These 10-time random splits are prepared prior to the experiments. For the UC dataset [39], we randomly select 10%, 20%, 50%, 80% of samples for training, and the remaining samples are used for testing. For the AID dataset [62], the training ratios are set to 20% and 50%, and the remaining 80% and 50%, respectively, are used for testing. For the NWPU dataset [1], we set the training ratios to 10% and 20%, and the remaining 90% and 80%, respectively, are used for testing.

To quantitatively evaluate the performance of the proposed method, we adopt the average accuracy (AA), overall accuracy (OA), and confusion matrix as the evaluation metrics in this paper. The OA is recorded as the number of correctly classified samples divided by the total number of samples. The AA is defined as the average classification accuracy of each class. The confusion matrix is a summary of prediction results that analyze the errors and confusions between different scene classes. It can be computed by the number of correct and incorrect predictions for each class and organized into a table.

We implement these networks by means of an open-source neural network library named Pytorch [63]. Experiments in this work are implemented using VSCode 1.31.1/Ubuntu 18.04 and conducted on a workstation equipped with a single NVIDIA GeForce RTX 2080 8 GB GPU.

4.3. Implementation

To transfer features from the aforementioned DCNNs to the scene classification, we fine-tune these DCNNs on each scene dataset individually. The final classification layer trained on the ImageNet dataset is substituted by a randomly initialized C-way softmax layer, where C is the number of classes in the scene dataset. We use the original input image dimension for all data sets.

We set the rotation angles ${\theta }_r$ as 0, 90, 180, or 270 degrees to decrease the impact of discrepancy from random rotation. For the proposed network, we set a higher learning rate for the final layer (randomly initialized) to achieve fast convergence of the model and a lower learning rate for the other fine-tuning layer in order to avoid too rapid and too high distortion of the pre-trained weights. Based on the pre-trained DCNN architectures, we train the models via Adam with a momentum of (0.9, 0.99).

5. Results and Discussion

5.1. Tradeoff of Temperature and Tuning Parameters

In the proposed method, there are two main parameters, the temperature parameter $T_f$ and tuning parameter $\lambda$, that can affect the performance of scene classification. The temperature affects the quality of the RIR and the setting of $\lambda$ also affects the overall classification accuracy of the proposed method. These parameters are determined in a heuristic way for the scene classification. We set temperature $T_f$ from $\{1.5,2.0,3.0,4.5,6.0,8.0,10.0,20.0\}$ and the tuning parameter $\lambda$ in $\{0.05,0.1,0.5,0.95,0.99\}$, respectively. Using a higher value for $T_f$ or a lower value for $\lambda$ strengthens the regularization effect on the scene classification task.

Table 1. Parameters utilized for models with different baseline deep convolutional neural networks (DCNNs) on the UC data set.

Baseline DCNN	#Iterations	Batch Size	L.R. ${}^1$	L.R. (Final Layer)	#Iterations (L.R. Decay)	Decay Factor	Training Times (h)
AlexNet [21]	2000	64	${10}^{-4}$	${10}^{-3}$	1000	0.1	0.200
VGG-VD16 [22]	6000	16	${10}^{-4}$	${10}^{-3}$	3000	0.1	0.836
ResNet50 [24]	4000	24	${10}^{-4}$	${10}^{-3}$	2000	0.1	0.471

${}^1$ L.R. denotes the learning rate.

summarizes the detailed parameters used for the models with different baseline DCNNs on the UC dataset. The learning rate is set to ${10}^{-3}$ for the last fully connected layer and ${10}^{-4}$ for the other layers. These models are trained for 5000 iterations and the learning rate is dynamically divided by a factor of 0.1 every 4000 iterations. To facilitate evaluation, the step parameters $S_1$ and $S_2$ are set to 0. We also list the training times of the no-RIR and RIR DCNNs by hours. Because the gradient of the regularization term has a low computational complexity, the training times of these two DCNNs are almost equal.

Figure 5 and Figure 6 summarize the training and testing results with the AlexNet-based RIR-net on the UC dataset. We evaluate this method with different parameter settings. The classification is measuring in terms of OA for all categories. It is found that the RIR with a softer label does indeed help the baseline DCNNs to generalize better, but requires more iterations for convergence. As shown in Figure 5a, these methods show fast convergence except for the RIR-nets with excessively high parameter $\lambda$ of 0.99. This is because underfitting occurs when the model is regularized too much for high $\lambda$. In addition, the models with high $\lambda$ (e.g., 0.95) show divergence for low temperature T (e.g., 2.0), but convergence for high T (e.g., 10.0). The curves in Figure 6 indicate that the losses of the RIR-nets are higher for the regularization term. As shown in Figure 6b, larger T and $\lambda$ can lead to a more stable convergence of the loss.

We evaluate the hyper-parameters of temperature T and tuning parameter $\lambda$ for the AlexNet-based RIR-net and baseline in Figure 7. As observed from Figure 7, the optimal value of the tuning parameter $\lambda$ is generally 0.5 under most temperatures T. For too large $\lambda$ or too small T, the RIR-nets obtain poor results that are even worse than the model that only uses cross-entropy criteria (84.01% and 89.81%, respectively). The performance of the RIR-nets with other settings are clearly better than the baseline model. Thus, higher temperatures T are better for scene classification accuracies.

Table 2. Classification accuracies (%) on the UC data set.

	T.R. ${}^1$ = 10%	T.R. = 20%	T.R. = 50%	T.R. = 80%
no RIR + AlexNet	$83.{89}_{\left(1.62\right)}$	$89.{81}_{\left(0.80\right)}$	$95.{20}_{\left(0.70\right)}$	$96.{54}_{\left(0.66\right)}$
no RIR + VGG-VD16	$88.{08}_{\left(1.19\right)}$	$92.{46}_{\left(1.14\right)}$	$95.{96}_{\left(0.86\right)}$	$97.{38}_{\left(0.45\right)}$
no RIR + ResNet50	$91.{97}_{\left(0.77\right)}$	$95.{23}_{\left(0.47\right)}$	$97.{79}_{\left(0.34\right)}$	$98.{75}_{\left(0.35\right)}$
RIR + AlexNet	$85.{76}_{\left(0.92\right)}$	$91.{03}_{\left(0.62\right)}$	$95.{91}_{\left(0.42\right)}$	$96.{96}_{\left(0.69\right)}$
RIR + VGG-VD16	$90.{09}_{\left(0.77\right)}$	$93.{96}_{\left(0.86\right)}$	$96.{96}_{\left(0.49\right)}$	$97.{92}_{\left(0.81\right)}$
RIR + ResNet50	$93.{03}_{\left(0.84\right)}$	$95.{98}_{\left(0.46\right)}$	$98.{28}_{\left(0.34\right)}$	$99.{15}_{\left(0.40\right)}$

${}^1$ T.R. denotes the training ratio.

summarizes the results for different baseline network architectures under different training ratios. Our results show considerable improvement over the no-RIR baseline DCNNs, which have same architecture with RIR-net except for the regularization term. The larger performance gain of RIR-net over no-RIR are due to the smaller training samples. For each baseline DCNN, the RIR method can improve the classification accuracies by almost 2% on the UC dataset with the training ratio of 10%. With respect to the variance, these baseline DCNNs with RIR are mostly lower than the corresponding baseline DCNNs. This result indicates that the RIR method can not only reduce the prediction error, but also boosts the generalization of DCNNs.

For the VGG-VD16-based RIR-net and baseline DCNN, the confusion matrices on the UCM dataset with a training ratio of 50% are provided in Figure 8. It is observed that the performances for each class are improved except classes (forest) and (tennis court) categories that decrease to lower than 1%. The classification accuracies of most scene categories achieve a satisfactory level, and some categories are close to 100%. The category (building) that is easily confused with category (dense residential), is the most improved category. In addition, the performances of classes (dense residential) and (sparse residential) are boosted, but that of their commonly confused category (medium residential) is unchanged, possibly due to the limitations of the representation capability of the VGG-VD16 architectures for the remote sensing scene classification.

In

Table 3. Comparison of overall accuracies (OA) (%) on the UC dataset (training ratio = 80%).

Method	Training Ratio
	50%	80%
BoVW (SIFT) [64]	$73.{48}_{\left(1.39\right)}$	$75.{52}_{\left(1.77\right)}$
AlexNet [62]	$93.{98}_{\left(0.67\right)}$	$95.{02}_{\left(0.81\right)}$
VGG-VD16 [62]	$94.{14}_{\left(0.69\right)}$	$95.{21}_{\left(1.20\right)}$
Scenario (II) [9]	–	$96.{90}_{\left(0.77\right)}$
AlexNet + SPP [65]	$94.{77}_{\left(0.46\right)}$	$96.{67}_{\left(0.94\right)}$
CCP-net [14]	–	$97.{52}_{\left(0.97\right)}$
AlexNet + MSCP [43]	–	$97.{29}_{\left(0.63\right)}$
VGG-VD16 + MSCP [43]	–	$98.{36}_{\left(0.58\right)}$
TEX-Net + VGG-VD16 [66]	$94.{22}_{\left(0.50\right)}$	$95.{31}_{\left(0.69\right)}$
VGG-VD16 + Siamese [28]	$85.{14}_{\left(0.53\right)}$	$92.{38}_{\left(0.16\right)}$
ResNet50 + Siamese [28]	$90.{95}_{\left(0.41\right)}$	$94.{29}_{\left(0.39\right)}$
RADC-Net [67]	$94.{79}_{\left(0.42\right)}$	$97.{05}_{\left(0.48\right)}$
$\mathrm{AlexNet}+{\mathrm{RIR}}_{\left(\mathrm{ours}\right)}$	$95.{91}_{\left(0.42\right)}$	$96.{96}_{\left(0.69\right)}$
$\mathrm{VGG}-\mathrm{VD}16+{\mathrm{RIR}}_{\left(\mathrm{ours}\right)}$	$96.{96}_{\left(0.49\right)}$	$97.{92}_{\left(0.81\right)}$
$\mathrm{ResNet}50+{\mathrm{RIR}}_{\left(\mathrm{ours}\right)}$	$98.{28}_{\left(0.34\right)}$	$99.{15}_{\left(0.40\right)}$

, we compare our method with the previously reported state-of-the-art methods. Deep-learning-based methods exhibit superior performance compared to the handcrafted-feature method. The RIR-net obtains higher classification accuracies than the other pre-trained DCNN-based methods with the AlexNet or VGG-VD16. AN examination of the data presented in Table 3 shows that RIR-net achieves more desirable classification accuracies for a smaller number of training samples. Our best network with pre-trained ResNet50 achieves 98.28% and 99.15% on the testing set with the training ratios of 50% and 80%, respectively. Compare with the Siamese CNN in [28], the RIR-net achieves significantly better performance, especially for the training ratio of 50%. In addition, the RIR-net is generally independent of the architecture, so that we expect that it will further improve the deeper and larger DCNN architectures, e.g., VGG-VD19 [22] and ResNet101 [24].

5.2. Effect of Increasing Temperature Strategy

In this section, we compare the results of the proposed RIR-net with fixed and increasing temperature T on the NWPU dataset. The detailed parameters on the NWPU dataset are shown in

Table 4. Parameters utilized for models with different baseline DCNNs on the NWPU data set.

Baseline DCNN	#Iterations	Batch Size	L.R.	L.R. (Final Layer)	#Iterations (L.R. Decay)	Decay Factor	Training Times (h)
AlexNet	5000	64	${10}^{-4}$	${10}^{-3}$	3000	0.1	0.499
VGG-VD16	15,000	16	${10}^{-4}$	${10}^{-3}$	7000	0.1	1.948
ResNet50	12,000	24	${10}^{-4}$	${10}^{-3}$	6000	0.1	1.388

for different baseline DCNNs. The hyper-parameters of temperature T and tuning parameter $\lambda$ for the AlexNet-based RIR-net with the training ratio = 10% are evaluated in Figure 9. For each tuning parameter $\lambda$, the classification accuracies gradually increase with increasing temperature T. For the NWPU dataset, the gap between RIR-net with $T=0.5$ and $T=0.95$ is smaller than that of the UCM dataset. This may be caused by the use of more training samples and steps. Except for the $T=0.99$, all of the RIR-nets are superior to the baseline DCNN models. Because more training steps are required for higher temperature, we select the optimal temperature $T=10.0$ and tuning parameter $\lambda =0.5$ for the NWPU data set.

In addition, we evaluate the strategy of slowly increasing T as described in Equation (12). To demonstrate the effectiveness of this strategy, we take larger $T=20$ and $\lambda =0.5$ on the NWPU dataset with a training ratio of 10%. The AlexNet architecture is employed with the parameters $S_1=1000$ and $S_2=2000$. As shown in Figure 10, the RIR-nets with increasing or fixed T finally achieve the same testing accuracies and losses. With increasing T, the accuracy increases sharply from #iteration 1000 at which the temperature starts to rise. It is interesting that the no-RIR baselines have a growth point at #iteration 2000, that the learning rate starts to decay. However, the performance of RIR-nets was not improved further after the learning rate decay. It can be concluded that the temperature increasing strategy resembles the learning rate decay method. For fair comparison, we still adopt the learning decay method in the rest of the experiments.

The results in the

Table 5. Classification accuracies (%) on the NWPU data set.

	T.R. = 10%	T.R. = 20%
no RIR + AlexNet	$82.{37}_{\left(0.19\right)}$	$86.{67}_{\left(0.25\right)}$
no RIR + VGG-VD16	$86.{07}_{\left(0.27\right)}$	$90.{10}_{\left(0.17\right)}$
no RIR + ResNet50	$90.{93}_{\left(0.23\right)}$	$93.{27}_{\left(0.19\right)}$
RIR + AlexNet	$84.{46}_{\left(0.14\right)}$	$87.{68}_{\left(0.19\right)}$
RIR + VGG-VD16	$88.{42}_{\left(0.21\right)}$	$91.{34}_{\left(0.21\right)}$
RIR + ResNet50	$92.{05}_{\left(0.23\right)}$	$94.{06}_{\left(0.15\right)}$

show considerable improvement over the no-RIR baselines on the NWPU dataset. An examination of the data presented in Table 5 shows that the performance of AlexNet and VGG-VD16 based architectures is boosted more significantly than that of the ResNet50 based architecture.

As described in Equation (9), we evaluate our RIR-net for $K=4$ where all rotated samples are used in the regularization term. With same hyper-parameters $T=10.0$ and $\lambda =0.5$, the AlexNet-based RIR-net with $K=4$ obtain classification accuracy of $84.82\pm 0.17\%$ when using the training ratio of 10%. The larger K is conducive to the performance improvement. However, due to the greater GPU memory and iteration requirements, we use $K=2$ in the experiments reported in this paper.

Figure 11 shows the confusion matrices conducted with the VGG-VD16-based RIR-net and baseline DCNN on the NWPU dataset with training ratio of 20%. It is observed that our proposed RIR-net can improve the classification accuracies on nearly all classes, particularly for categories (bridge), (circular farmland), (intersection), (medium residential), (palace), and (wetland) categories. As shown in Figure 11, RIR-net reduces the confusion between the classes (circular farmland) and (dense residential), (medium residential) and (mobile home park), and (wet land) and (lake). These results indicate that our method extracts a more discriminative description for the remote sensing scene images.

We compare our proposed RIR-net with some other pre-trained CNN-based classification methods. As shown in

Table 6. Comparison of the classification results (%) on the NWPU dataset (training ratio = 10% and 20%).

Method	Training Ratio
	10%	20%
AlexNet + BoW [42]	$55.{22}_{\left(0.39\right)}$	$59.{22}_{\left(0.18\right)}$
VGG-VD16 + BoW [42]	$82.{65}_{\left(0.31\right)}$	$84.{32}_{\left(0.17\right)}$
Fine-tuned AlexNet [1]	$81.{22}_{\left(0.19\right)}$	$85.{16}_{\left(0.18\right)}$
Fine-tuned VGG-VD16 [1]	$87.{15}_{\left(0.45\right)}$	$90.{36}_{\left(0.18\right)}$
AlexNet + MSCP [43]	$81.{70}_{\left(0.23\right)}$	$85.{58}_{\left(0.16\right)}$
VGG-VD16 + MSCP [43]	$85.{33}_{\left(0.17\right)}$	$88.{93}_{\left(0.14\right)}$
AlexNet + SPP [65]	$82.{13}_{\left(0.30\right)}$	$84.{64}_{\left(0.23\right)}$
AlexNet + D-CNN [11]	$85.{56}_{\left(0.20\right)}$	$87.{24}_{\left(0.12\right)}$
VGG-VD16 + D-CNN [11]	$89.{22}_{\left(0.50\right)}$	$91.{89}_{\left(0.22\right)}$
VGG-VD16 + Siamese [28]	–	$90.{06}_{\left(3.23\right)}$
ResNet50 + Siamese [28]	–	$92.{28}_{\left(3.27\right)}$
RADC-Net [67]	$85.{72}_{\left(0.25\right)}$	$87.{63}_{\left(0.28\right)}$
AlexNet + SAFF [44]	$80.{05}_{\left(0.29\right)}$	$84.{00}_{\left(0.17\right)}$
VGG-VD16 + SAFF [44]	$84.{38}_{\left(0.19\right)}$	$87.{86}_{\left(0.14\right)}$
$\mathrm{AlexNet}+{\mathrm{RIR}}_{\left(\mathrm{ours}\right)}$	$84.{46}_{\left(0.14\right)}$	$87.{68}_{\left(0.19\right)}$
$\mathrm{VGG}-\mathrm{VD}16+{\mathrm{RIR}}_{\left(\mathrm{ours}\right)}$	$88.{42}_{\left(0.21\right)}$	$91.{34}_{\left(0.21\right)}$
$\mathrm{ResNet}50+{\mathrm{RIR}}_{\left(\mathrm{ours}\right)}$	$92.{05}_{\left(0.23\right)}$	$94.{06}_{\left(0.15\right)}$

, the RIR-nets achieve higher accuracy than the baseline DCNNs. It is important to note that the gain in the performance is not only due to the data augmentation for the comparison in Table 5. The RIR-net performs better than all of the other methods based on the same DCNN architectures, but shows slightly worse performance than the latest D-CNN model [11] under the training ratio of 10%. Compared to the MSCP [43] and SAFF [44] methods, the AlexNet-based RIR-net obtains comparable performance to that of other VGG-VD16-based architectures. The highest classification accuracies are obtained by the ResNet50-based RIR-net that achieves 92.05% and 94.06% for the training ratios of 10% and 20%, respectively.

5.3. Validation on the AID Data Set

With optimal temperature and tuning parameters based on the UC and NWPU datasets, we evaluate the proposed RIR-net on the AID dataset. As shown in

Table 7. Parameters utilized for the models with different baseline DCNNs on the AID data set.

Baseline DCNN	#Iterations	Batch Size	Input Size	L.R.	L.R. (Final Layer)	#Iterations (L.R. Decay)	Decay Factor	Training Times (h)
AlexNet	6000	32	$512\times 512$	${10}^{-4}$	${10}^{-3}$	3000	0.1	1.232
VGG-VD16	12,000	16	$256\times 256$	${10}^{-4}$	${10}^{-3}$	6000	0.1	1.494
ResNet50	10,000	24	$256\times 256$	${10}^{-4}$	${10}^{-3}$	5000	0.1	1.130

, we set the parameters with different input sizes due to the greater GPU memory required by the larger image size in the AID dataset. We transform the size of input images into $512\times 512$ and $256\times 256$ for AlexNet and the other two baseline DCNNs, respectively.

Table 8. Classification accuracies (%) on the AID data set.

	T.R. = 20%	T.R. = 50%
no RIR + AlexNet	$90.{54}_{\left(0.30\right)}$	$93.{61}_{\left(0.14\right)}$
no RIR + VGG-VD16	$91.{98}_{\left(0.30\right)}$	$94.{86}_{\left(0.27\right)}$
no RIR + ResNet50	$94.{01}_{\left(0.25\right)}$	$95.{96}_{\left(0.26\right)}$
RIR + AlexNet	$91.{95}_{\left(0.20\right)}$	$94.{56}_{\left(0.11\right)}$
RIR + VGG-VD16	$93.{34}_{\left(0.18\right)}$	$95.{57}_{\left(0.23\right)}$
RIR + ResNet50	$94.{95}_{\left(0.17\right)}$	$96.{48}_{\left(0.21\right)}$

shows the results for different baseline DCNNs with temperature $T=10$ and $\lambda =0.5$.

As shown in Figure 8, our RIR-net can also improve the classification accuracy for the large size of input images. It is observed that the RIR-nets are better than the no-RIR DCNNs for the results on the AID datasets with optimal temperature and tuning parameters based on the UC and NWPU datasets. For the VGG-VD16 and ResNet50 architecture, the performance boost on the input image of large size and is not much reduced by shrinking the image size.

Figure 12 shows the confusion matrix of the VGG-VD16-based RIR-net under the training ratio of 20%. The classification accuracies of most of the categories are increased. For the classes (center) and (stadium), the performances are decreased by less than 1%. The classes (school) and (resort) achieve a greater than 5% enhancement in performance due to the discrimination improvement between the classes (school) and (church), (resort) and (church), and (resort) and (sparse residential). These classes usually share building objects, thereby making these categories similar and their classification challenging.

Table 9. Comparison of the OAs (%) on the AID dataset (training ratio = 20% and 50%).

Method	Training Ratio
	20%	50%
VGG-VD16 [62]	$86.{59}_{\left(0.29\right)}$	$89.{64}_{\left(0.36\right)}$
GoogLeNet [62]	$83.{44}_{\left(0.40\right)}$	$86.{39}_{\left(0.55\right)}$
AlexNet + MSCP [43]	$88.{99}_{\left(0.38\right)}$	$92.{36}_{\left(0.21\right)}$
VGG-VD16 + MSCP [43]	$91.{52}_{\left(0.21\right)}$	$94.{42}_{\left(0.17\right)}$
AlexNet + SPP [65]	$87.{44}_{\left(0.45\right)}$	$91.{45}_{\left(0.38\right)}$
RADC-Net [67]	$88.{12}_{\left(0.43\right)}$	$92.{53}_{\left(0.19\right)}$
AlexNet + SAFF [44]	$87.{51}_{\left(0.36\right)}$	$91.{83}_{\left(0.27\right)}$
VGG-VD16 + SAFF [44]	$90.{25}_{\left(0.29\right)}$	$93.{83}_{\left(0.28\right)}$
$\mathrm{AlexNet}+{\mathrm{RIR}}_{\left(\mathrm{ours}\right)}$	$91.{95}_{\left(0.20\right)}$	$94.{56}_{\left(0.11\right)}$
$\mathrm{VGG}-\mathrm{VD}16+{\mathrm{RIR}}_{\left(\mathrm{ours}\right)}$	$93.{34}_{\left(0.18\right)}$	$95.{57}_{\left(0.23\right)}$
$\mathrm{ResNet}50+{\mathrm{RIR}}_{\left(\mathrm{ours}\right)}$	$94.{95}_{\left(0.17\right)}$	$96.{48}_{\left(0.21\right)}$

summarizes the comparison of our results with the state-of-the-art methods on the AID dataset. Our results with the AlexNet are superior to the results obtained using the previous AlexNet-based methods, and even surpass the previous VGG-VD16-based methods. The best performances (94.45% and 96.48%) are achieved by the ResNet50-based RIR-net with the training ratios of 20% and 50%.

5.4. Visualization of the Softmax Outputs

In this section, we visualize the outputs of the last layer of baseline CNN and RIR-net. Figure 13 shows two test examples of different scene classes (building and golf course) and their corresponding probabilities of categories for the baseline DCNN and RIR-net. These test examples are rotated by 0, 90, 180, 270 degrees, respectively. We use softmax with temperature $T=5.0$ to facilitate visualization. The softmax outputs of AlexNet-based RIR-net and baseline are shown in Figure 13. As shown in Figure 13, the output of CNN of different test examples of the same category are not rotation-invariant. For the scene image of #4 (building category), the most similar categories are #6 (dense residential), #12 (medium residential), and #13 (mobile home park). For the CNN model, the major differences between the outputs of the images with different orientations are found for the categories #6 (dense residential), #13, and #19 (storage tank). For the scene image of #9 (golf course), it appears similar to the categories of #3 (beach), #16 (river), and #18 (sparse residential). Particularly for the #16, some of this scene image is misclassified because the baseline CNN cannot capture the invariance information between the images with different orientations. In contrast, the output of RIR-net of these examples of the same object class are clearly more similar. These results demonstrate the effectiveness of the proposed RIR method.

6. Conclusions

In this study, we propose a novel and effective method based on the Siamese convolutional neural networks that aim to increase the robustness and reduce the overfitting for the remote sensing scene classification. We enforce the output classification probabilities of the training samples before and after the rotation to be similar by optimizing a new objective function with the RIR. To compare the probability distributions, we introduce the soft labels proposed in the “softmax temperature”. By raising the temperature of the final softmax, we enlarge the diversity of the output distribution to be characteristic of the similarities between different categories. The quantitative comparison results on the publicly available remote sensing scene classification datasets demonstrate the performance gain of the proposed method compares with state-of-the-art approaches. In the future, we plan to investigate the effectiveness of this regularization for the semi-supervised methods and the metric learning-based DCNN architectures.