A Novel Two-Stream Network for Few-Shot Remote Sensing Image Scene Classification

Abstract

Recently, remote sensing image scene classification (RSISC) has gained considerable interest from the research community. Numerous approaches have been developed to tackling this issue, with deep learning techniques standing out due to their great performance in RSISC. Nevertheless, there is a general consensus that deep learning techniques usually need a lot of labeled data to work best. Collecting sufficient labeled data usually necessitates substantial human labor and resource allocation. Hence, the significance of few-shot learning to RSISC has greatly increased. Thankfully, the recently proposed discriminative enhanced attention-based deep nearest neighbor neural network (DEADN4) method has introduced episodic training- and attention-based strategies to reduce the effect of background noise on the classification accuracy. Furthermore, DEADN4 uses deep global–local descriptors that extract both the overall features and detailed features, adjusts the loss function to distinguish between different classes better, and adds a term to make features within the same class closer together. This helps solve the problem of features within the same class being spread out and features between classes being too similar in remote sensing images. However, the DEADN4 method does not address the impact of large-scale variations in objects on RSISC. Therefore, we propose a two-stream deep nearest neighbor neural network (TSDN4) to resolve the aforementioned problem. Our framework consists of two streams: a global stream that assesses the likelihood of the whole image being associated with a particular class and a local stream that evaluates the probability of the most significant area corresponding to a particular class. The ultimate classification outcome is determined by putting together the results from both streams. Our method was evaluated across three distinct remote sensing image datasets to assess its effectiveness. To assess its performance, we compare our method with a range of advanced techniques, such as MatchingNet, RelationNet, MAML, Meta-SGD, DLA-MatchNet, DN4, DN4AM, and DEADN4, showcasing its encouraging results in addressing the challenges of few-shot RSISC.

1. Introduction

Remote sensing images (RSIs) offer a wealth of data related to land cover targets [1,2,3,4,5,6], finding extensive applications in diverse contexts, including road identification [7], ecological surveillance [8], disaster forecasting, and numerous other domains [9,10]. Classifying RSIs into different scenes based on the collected data is a key focus of the current research [11,12,13,14,15,16,17,18], aiming to classify new images into relevant classes [19,20,21,22,23]. The objective of few-shot scene classification is to accurately categorize images into their respective classes despite limited examples [24], a capability that is vital given the challenge of obtaining sufficient labeled data in numerous practical applications. The potential for few-shot classification of RSIs is significant because of its usefulness in many areas, thereby reducing the need for field investigation and manual labeling.

In the domain of conventional image classification, deep learning has shown great effectiveness [25,26] but usually demands extensive labeled datasets to achieve the optimal training results [27,28,29,30,31]. A neural network utilizing a well-designed architecture will achieve a good performance when equipped with comprehensive foundational knowledge [32,33]. Nevertheless, not having enough labeled data can cause deep models to overfit and incomplete feature representations. In diverse remote sensing applications, acquiring tagged examples is difficult and made harder by the time-consuming task of creating labeled datasets [34]. Moreover, training deep neural networks necessitates substantial computational power, and if the settings of the hyperparameters are not suitable, the process may need to be repeated [35,36,37,38].

Hence, as for remote sensing image scene classification (RSISC), the importance of few-shot learning cannot be overstated. The basic idea of few-shot learning is to create methods that rapidly adapt to a limited set of tagged examples, mainly to give models the ability to learn fast [39,40,41]. As for few-shot learning, it is crucial that the model learns deep features that are both well separated and easily distinguishable so that it can recognize novel classes despite having only a few tagged samples. Snell et al. [42] proposed Prototypical Networks for few-shot classification, learning a metric space for classification through computation of the distance to class prototypes, demonstrating that simple designs outperform complex architectures and meta-learning. Yan et al. [43] presented a straightforward and effective approach seamlessly integrable into both meta-learning and transfer-learning frameworks, isolating confusing negatives while preserving the impact of hard negatives, adjusting the gradients by their category affinity, and reducing the effects of false negatives without extra parameters. Zeng et al. [44] proposed a novel framework for few-shot RSISC which introduced a feature-generating model and prototype calibration, employing a self-attention encoder to filter irrelevant data, expanding the support set via generated features, and refining the prototypes to improve the category feature representation. Vinyals et al. [45] presented an episodic training method that dealt with the difficulties inherent in few-shot learning. Throughout the training phase, a support set is formed by randomly picking K images from each of the C classes within the dataset, yielding a collection of $C\times K$ images. After that, N images are taken from the rest of the dataset for each of the C chosen classes to make up a query set. Each episode combines one support set with one query set, and this process is repeated until the model converges. Many few-shot models use a method that focuses on learning distances, where they calculate the similarity measure between support and query instances directly to train the classifier based on this distance. However, these methods may not fully use the network’s ability to extract good features, which will significantly affect these methods’ discriminative ability.

Large variations in the object sizes are also a difficult problem in few-shot RSISC. In RSIs, the same target may look very different in different images because of factors like the camera’s angle and the height of the sensor. As depicted in Figure 1, scenes like airplanes, oil tanks, and power plants photographed at different heights show large size differences, even within the same type of object. Furthermore, because of a scene’s characteristics, the sizes of the objects in it can differ, as shown by the river scene in the figure, which has several different parts, like small rivers and streams. From Figure 1, it can be observed that the sizes of key objects differ, occupying just a small part of the whole picture and often being around many unrelated features and objects. Obviously, it does not make sense to use one big object to match pictures that contain many small ones.

To solve this problem, Chen et al. [46] developed the discriminative enhanced attention-based deep nearest neighbor neural network (DEADN4). Initially, the DEADN4 model employs a deep local–global descriptor (DLGD) to perform classification by combining localized and global characteristics, thereby facilitating the distinction between various classes. After this, to strengthen the similarity within the same class, DEADN4 uses a special loss to improve the overall details, bringing features in the same class closer together and reducing the differences within the class. Finally, DEADN4 enhances the accuracy of classification by incorporating a cosine margin into the Softmax loss, making the differences between features from different classes clearer. However, the DEADN4 method does not solve the problem of the large size differences among the objects in the RSI.

To address the abovementioned problem, this paper proposes a two-stream deep nearest neighbor neural network (TSDN4), accomplished by constructing a global–local two-stream network. This network has two parts: the global stream, which looks at the whole image to find out whether it belongs to a certain class, and the local stream, which focuses on the most important part of the image to see whether it falls into a particular class. The final classification result is obtained by combining the results from both parts. To find the most important areas in the image, the key area localization (KAL) method is used to link the two parts mentioned above. Section 2 discusses earlier work within this domain, while Section 3 introduces our proposed method. Next, Section 4 discusses the results of our experiments and analysis. Lastly, Section 6 highlights the main conclusions from our research.

2. Related Work

The method proposed in this paper builds upon DEADN4 [46]; therefore, this section mainly focuses on introducing DEADN4. The primary objective of DEADN4 is to tackle the challenge posed by few-shot RSISC, which comprises two key components: a deep embedding module that utilizes attention mechanisms, denoted as $f_{\psi }(·)$, and a metric module, represented by $f_{\phi }(·)$. The core function of the $f_{\psi }(·)$ module is to derive representative details from the images and generate attention maps highly pertinent to the scene class. As for every input image X, the function $f_{\psi }\left(X\right)$ produces a feature map of the dimensions $H\times W\times d$, comprising $H\times W$ DLGD instances, each with d dimensions. A DLGD consists of a deep local descriptor (DLD) and a deep global descriptor (DGD). DLDs extract features from various regions of the image, while the DGD captures the global details, serving as a useful guide to enhance the distinctiveness of each local feature.

Moreover, the $f_{\psi }(·)$ module includes a component for acquiring class-distinctive attention capabilities. This component classifies the local features into pertinent and non-pertinent segments according to the scene classification. The main goal here is to minimize the impact of background noise and highlight features that are important to the scene class. Lastly, within the $f_{\phi }(·)$ module, the classification of a query image is acquired by contrasting its features with those of the support images.

To summarize, DEADN4 employs the $f_{\psi }(·)$ module to derive features pertinent to scene classification from the images, and then the $f_{\phi }(·)$ module assesses the feature resemblance between the query and support images to enable RSISC.

The DEADN4 loss function is defined as follows:

$$L_{DEADN4}=L_s+\lambda L_c$$

(1)

where the overall loss $L_{DEADN4}$ includes contributions from the central loss $L_c$ and the class loss $L_s$, with $\lambda$ as a scalar that adjusts the influence of $L_c$.

The central loss term $L_c$ in Equation (1) can be described as follows:

$$L_c={\displaystyle \frac{1}{2}}\sum _{i=1}^C\sum _{j=1}^K{∥f_{ij}-f_{ci}∥}_2^2$$

(2)

where C denotes the overall count of classes, K indicates the quantity of instances per class in the support set, $f_{ij}$ is the global feature of the j-th instance within the support set’s i-th class, and $f_{ci}$ denotes the mean global feature across all instances associated with the i-th class in the support set.

The class loss $L_s$, as mentioned in Equation (1), is defined as follows:

$$L_s={\displaystyle \frac{1}{C\times N}}\sum _{i=1}^C\sum _{j=1}^N-log{p}_{ij}={\displaystyle \frac{1}{C\times N}}\sum _{i=1}^C\sum _{j=1}^N-log{\displaystyle \frac{e^{f_{\phi }\left(f_{\psi }\left(q_j\right),c_i\right)-mk\times M}}{e^{f_{\phi }\left(f_{\psi }\left(q_j\right),c_i\right)-mk\times M}+{\sum }_{t\ne i}{e}^{f_{\phi }\left(f_{\psi }\left(q_j\right),c_t\right)}}}$$

(3)

where $p_{ij}$ stands for the likelihood of accurately categorizing the query image, k represents the number of nearest neighbors, m denotes the DLGD’s count, and M serves as an additional margin enhancing the resemblance $cos\left(x_i,{\widehat{x}}_i\right)$ between descriptors.

3. Methods

3.1. Architecture

To tackle the challenge of significant variations in the objects within images, this paper proposes a method for scene image classification that combines global and local metrics. As shown in Figure 2, a two-stream network that combines both global and local measurements is designed. Within this framework, the network’s blue pathway represents the global stream for measuring the overall image, calculating the resemblance between images and classes. Meanwhile, the network’s green pathway represents the local stream, which focuses on measuring a particular area within the image. It emphasizes the most significant object by focusing on key local areas when calculating the similarities between areas. The two-stream network proposed in this paper allows for separate measurements of the entire image and specific areas from inputs of different sizes, with their classification results eventually being combined.

The global stream uses the DEADN4 model, while the local stream employs a prototype network. To identify the most important local areas within the whole image, this paper further introduces a weakly supervised key area detection method, called KAL, as shown in the path in Figure 2. KAL, using scene-related attention maps, is a clear method for finding key local areas, connecting the two streams. It aims to precisely identify the most significant area and find the coordinates of its bounding box $B=[x_1,x_2,y_1,y_2]$, where $x_1$ and $x_2$ are the left and right edges of the area’s width, and $y_1$ and $y_2$ represent the upper and lower edges of the area’s height.

3.2. Key Area Localization

In an image, finding key areas is the most important step, connecting the global and local streams. The flowchart for the KAL strategy is displayed in Figure 3. Specifically, using the attention maps related to the scene classes from DEADN4, a bounding box is created to help classify the local stream. KAL consists of two sub-steps: vector construction and a greedy boundary search.

3.2.1. Vector Construction

The attention maps related to the scene classes can measure how important each part of an image is. So, using this attention map, we can find areas with high energy, which are the key areas. Because searching in two-dimensional space is complex, in this paper, the attention map is simplified into two one-dimensional energy vectors, one along the height and one along the width of the space, which is formulated as follows:

$$\left\{\begin{array}{cc}\hfill V_w& =\sum _{i=0}^{H}MM(i,W)\hfill \\ \hfill V_h& =\sum _{j=0}^{W}MM(H,j)\hfill \end{array}\right.$$

(4)

where $MM$ is the attention map we obtain, $MM(i,j)$ is the value located at position $(i,j)$ in $MM$, the total height is H, and the total width is W. By summing $MM$ along its height and width, we obtain $V_w\in {\mathbb{R}}^W$ and $V_h\in {\mathbb{R}}^H$, respectively.

To quickly find the most important one-dimensional area in an energy vector, this paper proposes a energy-based greedy boundary search method, which is formulated as follows:

$$\left\{\begin{array}{c}\hfill E_{[0:W]}=\sum _{i=0}^W{V}_w\left(i\right)\\ \hfill E_{\left[x_1:x_2\right]}=\sum _{i=x_1}^{x_2}{V}_w\left(i\right)\end{array}\right.$$

(5)

where $E_{[0:W]}$ represents the total energy of all of the components in the width vector $V_w$, and $E_{[x_1:x_2]}$ is the sum of the energies from spatial widths $x_1$ to $x_2$.

3.2.2. Greedy Boundary Search

This paper defines the key areas in an image as the areas that occupy the minimum space while containing no less than a certain proportion of the overall energy, specifically where $E_{\left[x_1:x_2\right]}/E_{[0:W]}>E_{Tr}$, with $E_{Tr}$ representing a hyperparameter for the energy ratio. Based on this criterion, a greedy-like algorithm can be employed to search for the most critical areas, which is outlined in Algorithm 1. The greedy boundary search algorithm is described as follows:

Algorithm 1 Greedy Boundary Search

Require: Width vector: $V_w\in {\mathbb{R}}^W$

Ensure: Width boundaries of the most critical area: $[x_1,x_2]$

$x_1\leftarrow 0$

$x_2\leftarrow W/2$

for $i=0\to W/2$ do

if $E_{\left[x_1:x_2\right]}<E_{[i:i+W/2]}$ then

$x_1\leftarrow i$

$x_2\leftarrow i+W/2$

end if

end for

if $E_{\left[x_1:x_2\right]}/E_{[0:W]}>E_{Tr}$ then

while $E_{\left[x_1:x_2\right]}/E_{[0:W]}>E_{Tr}$ do

if $V_w(x_1+1)<V_w(x_2-1)$ then

$x_1\leftarrow x_1+1$

else

$x_2\leftarrow x_2-1$

end if

end while

else

while $E_{\left[x_1:x_2\right]}/E_{[0:W]}<E_{Tr}$ do

if $V_w(x_1-1)>V_w(x_2+1)$ then

$x_1\leftarrow x_1-1$

else

$x_2\leftarrow x_2+1$

end if

end while

end if

$x_1\leftarrow I_w{x}_1/W$

$x_2\leftarrow I_w{x}_2/W$

Step 1: Initialize $x_1$ and $x_2$.

Step 2: This involves repeatedly adjusting the limits of $\left[x_1,x_2\right]$ to make the energy $E_{\left[x_1:x_2\right]}$ closer to $E_{tr}$. After Step 1, two states are possible: the ratio of energy in the $\left[x_1,x_2\right]$ area to the total energy in the width direction, $E_{\left[x_1:x_2\right]}/E_{[0:W]}$, is either higher or lower than $E_{tr}$. When the ratio $E_{\left[x_1:x_2\right]}/E_{[0:W]}$ is higher than $E_{tr}$, the $\left[x_1,x_2\right]$ area needs to contract in the direction of the slowest energy decrease until the ratio is no longer higher than $E_{tr}$. On the other hand, when the ratio $E_{\left[x_1:x_2\right]}/E_{[0:W]}$ is lower than $E_{tr}$, the area must expand in the direction of the fastest energy increase until it is no longer below $E_{tr}$. This paper’s greedy boundary search can identify the most compact yet informative area.

Step 3: The boundary coordinates are mapped from the feature map to the input image. Since the aforementioned boundary coordinates are located on the feature map, it is necessary to acquire the corresponding coordinates on the image, represented as $x_i=I_w{x}_f/W$, where $x_i$ is the coordinate in the width direction on the image, $x_f$ is the coordinate on the feature map, $I_w$ represents the width size of the image, and W indicates the width size of the feature map. The width boundaries of the most critical area in the image, denoted as $\left[x_1,x_2\right]$, can be obtained using the three steps described above. By employing the same algorithm, the height boundaries of the most crucial area, represented as $\left[y_1,y_2\right]$, can similarly be obtained.

3.2.3. The Two-Stream Architecture

The key areas’ positions can be obtained using the KAL module, represented by $B=[x_1,x_2,y_1,y_2]$. The classification results for the global stream are obtained from query and support images inputted into the DEADN4 model, whereas the classification scores of the local stream are generated by utilizing bilinear interpolation to feed the key areas of the query and support images into the prototype network and obtain classification scores. The prototype network is designed to utilize a mapping function that maps input images into a lower-dimensional embedding space, and a prototype representation is learned for each class. Lastly, we compute the distances between the query image’s features and the prototype representation of each class using a standard distance measure for classification purposes. In the prototype network, convolutional neural networks are typically used as the mapping function, with the average of the embedding features for all images within a class being taken as the prototype representation for that class.

Let the mapping function $f_{\varphi }$: ${\mathbb{R}}^D\to {\mathbb{R}}^F$ be defined to transform sample data of the dimension D into an embedding space of the dimension F. The prototype representation for each class is computed as follows:

$$c{c}_k={\displaystyle \frac{1}{\left|S_k\right|}}\sum _{\left(x{x}_i,y{y}_i\right)\in S_k}{f}_{\varphi }\left(x{x}_i\right)$$

(6)

where $S_k$ represents the dataset for class k, $xx$ denotes a sample in $S_k$, and $yy$ is its corresponding class. The prototype representation $c{c}_k$ corresponds to the mean of all of the embedding features within the support set for the respective class.

For a given query image $x{x}_q$, the likelihood that the query image falls into class k can be determined as follows:

$$p_{\varphi }(y{y}_q=k\mid x{x}_q)={\displaystyle \frac{exp\left(-d\left(f_{\varphi }\left(x{x}_q\right),{cc}_k\right)\right)}{{\sum }_{k^{\prime }}exp\left(-d\left(f_{\varphi }\left(x{x}_q\right),{cc}_{k^{\prime }}\right)\right)}}$$

(7)

where $f_{\varphi }\left(x_q\right)$ denotes the mapping features obtained from the query image $x{x}_q$ through the mapping function, $d(·)$ represents the distance function, and the cosine distance is used in this paper for computation. It should be noted that since the cosine distance offers advantages such as directional consistency, scale invariance, suitability for high-dimensional data, computational efficiency, and strong robustness and the global stream also employs the cosine distance for measurement, the cosine distance is adopted here as well for consistency.

Based on the actual labels of the images and the forecasted likelihood distribution, the local stream loss function $L_l$ is formulated as the negative log-probability, expressed using the formula below:

$$L_l={\displaystyle \frac{1}{C\times N}}\sum _{c=1}^C\sum _{q=1}^N-log{p}_{\varphi }\left(y{y}_q=c\mid x{x}_q\right)$$

(8)

where N signifies the overall count of samples in the query set, and C denotes the overall count of classes in the query set.

3.2.4. The Loss Function

Here is the overall loss function for the network:

$$L_{Total}=L_{DEADN4}+L_l$$

(9)

where $L_{Total}$ represents the total loss, which is the combination of $L_{DEADN4}$ from Equation (1) and $L_l$ from Equation (8).

4. Results

4.1. Dataset Description

In the experiments of this paper, the datasets utilized are the NWPU-RESISC45 [47] dataset, the UC Merced [48] dataset, and the WHU-RS19 [49] dataset. To facilitate comparison, the partitioning of the three datasets is the same as that for DLA-MatchNet [27].

4.1.1. The NWPU-RESISC45 Dataset

The NWPU-RESISC45 dataset, originating from Northwestern Polytechnical University in China, comprises a collection of 31,500 images sourced from Google Earth. The images encompass a diverse variety of over 100 countries and territories, created through a combination of satellite imagery, aerial photography, and geographical information. The dataset includes a variety of weather conditions, scene variations, sizes, seasonal settings, and lighting conditions. The images in this dataset typically have resolutions ranging from 30 m to $0.2$ m and are presented in a full color palette, including red, green, and blue. Figure 4 shows 45 distinct scene classes, with each containing a set of 700 images with dimensions of $256\times 256$. In the experimental setup, the dataset is partitioned into three parts: a training subset with 25 classes, a validation subset with 10 classes, and a testing subset with another 10 classes.

4.1.2. The UC Merced Dataset

The UC Merced dataset, introduced in 2010, is based on detailed maps provided by the United States Geological Survey National Map, showcasing a variety of areas in the USA. The RGB dataset maintains a high image resolution of $0.3$ m, spanning 21 varied scene categories. Within this dataset, each of the 21 categories comprises 100 detailed land use images, with each measuring $256\times 256$ dimensions, as illustrated in Figure 5. To conduct the experiments, the dataset is partitioned into three distinct sets: a training subset with 10 classes, a validation subset with 6 classes, and a testing subset with another 5 classes.

4.1.3. The WHU-RS19 Dataset

Released by Wuhan University in China, the WHU-RS19 dataset is a collection of images sourced directly from the famous Google Earth platform. The RGB dataset employs a fine resolution of $0.5$ m. Displayed in Figure 6, the dataset contains a diverse collection of 19 distinct scene classes. Each of these carefully selected classes contains a minimum of 50 representative samples, with each having a resolution of $600\times 600$ pixels. The dataset includes a total of 1005 scene images. In the experimental setup, the dataset is partitioned into three distinct parts: a training subset for 9 specific classes, a validation subset for 5 selected classes, and a testing subset comprising another 5 classes.

4.2. The Experimental Setting

4.2.1. The Experimental Software and Hardware Environment

The detailed specifications of the software configurations and hardware setups used in this paper are thoroughly presented in

Table 1. The experimental setup included both software and hardware components.

Hardware Environment	CPU	Intel(R) Core(TM) i7-7800X CPU @ 3.50 GHz 32 GB
	GPU	NVIDIA Geforce RTX 2080Ti 11 GB
Software Environment	OS	Linux Ubuntu 18.04 LTS
	Programming Language	Python 3.6
	Deep Learning Framework	PyTorch 1.4.0
	CUDA	Cuda 10.0

.

4.2.2. The Experimental Design

We performed experimental evaluations of solving both 5-way 1-shot and 5-way 5-shot problems across the three aforementioned datasets. In order to assess the effectiveness of our method, we benchmarked it against eight well-known few-shot learning methods: MatchingNet [45], RelationNet [50], MAML [51], Meta-SGD [52], DLA-MatchNet [27], DN4 [53], DN4AM [54], and DEADN4 [46]. Given that our method builds upon the DEADN4 framework, we benchmarked it against DEADN4, utilizing an identical embedding network for a fair evaluation. We evaluated the classification results utilizing the mean top-1 accuracy, along with the $95\%$ confidence intervals (CIs), as described in [54].

The global stream of our method adopts the network from DEADN4, while the local stream uses a prototype network, and both streams utilize ResNet18 models for their embedding modules. Random cropping is applied to input images, resizing them to $224\times 224$ pixels, and these resized images are then subjected to enhancements such as random horizontal flipping, brightness variations, color augmentation, and contrast enhancement. In the classification module, we specify the count of nearest neighbors as $k=3$, configure the loss function hyperparameter M to $0.01$, and set the parameter $E_{Tr}$ to $70\%$. The training set comprises 300,000 scenario sets in total, where in the 5-way 1-shot experiment, each scenario set includes 5 support images and 75 query images, while in the 5-way 5-shot experiment, each scenario set includes 25 support images and 50 query images. The Adam [55] algorithm is employed during network training, and the initial learning rate is set to 0.0001. The learning rate undergoes a decay every 100,000 scenarios, and for validation, 600 randomly generated scenario sets are utilized on the validation dataset. The top-performing model’s mean top-1 accuracy is used as the current network’s training outcome, and the model achieving the highest scores is saved as the final version. During the testing phase, we randomly sample 600 scenario sets from the test set five times, calculate the average top-1 accuracy for each iteration, and determine the final testing outcome as the mean of the results obtained from these iterations. Additionally, the $95\%$ CI is provided.

4.3. The Experimental Results

In this paper, corresponding experiments are performed on the three aforementioned datasets, and comparisons are made with other few-shot learning methods.

Table 2. Comparison of results of various methods tested on the NWPU-RESISC45 dataset.

Method	5-Way 1-Shot	5-Way 5-Shot
MatchingNet	54.46% ± 0.77%	67.87% ± 0.59%
RelationNet	58.61% ± 0.83%	78.63% ± 0.52%
MAML	37.36% ± 0.69%	45.94% ± 0.68%
Meta-SGD	60.63% ± 0.90%	75.75% ± 0.65%
DLA-MatchNet	68.80% ± 0.70%	81.63% ± 0.46%
DN4	66.39% ± 0.86%	83.24% ± 0.87%
DN4AM	70.75% ± 0.81%	86.79% ± 0.51%
DEADN4	73.56% ± 0.83%	87.28% ± 0.50%
Our Method	73.84% ± 0.80%	87.86% ± 0.51%

,

Table 3. Comparison of results of various methods tested on the UC Merced dataset.

Method	5-Way 1-Shot	5-Way 5-Shot
MatchingNet	46.16% ± 0.71%	66.73% ± 0.56%
RelationNet	48.89% ± 0.73%	64.10% ± 0.54%
MAML	43.65% ± 0.68%	58.43% ± 0.64%
Meta-SGD	50.52% ± 2.61%	60.82% ± 2.00%
DLA-MatchNet	53.76% ± 0.62%	63.01% ± 0.51%
DN4	57.25% ± 1.01	79.74% ± 0.78%
DN4AM	65.49% ± 0.72%	85.73% ± 0.47%
DEADN4	67.27% ± 0.74%	87.69% ± 0.44%
Our Method	68.12% ± 0.81%	88.57% ± 0.52%

and

Table 4. Comparison of results of various methods tested on the WHU-RS19 dataset.

Method	5-Way 1-Shot	5-Way 5-Shot
MatchingNet	60.60% ± 0.68%	82.99% ± 0.40%
RelationNet	60.54% ± 0.71%	76.24% ± 0.34%
MAML	46.72% ± 0.55%	79.88% ± 0.41%
Meta-SGD	51.54% ± 2.31%	61.74% ± 2.02%
DLA-MatchNet	68.27% ± 1.83%	79.89% ± 0.33%
DN4	82.14% ± 0.80%	96.02% ± 0.33%
DN4AM	85.05% ± 0.52%	96.94% ± 0.21%
DEADN4	86.89% ± 0.57%	97.63% ± 0.19%
Our Method	87.34% ± 0.62%	98.25% ± 0.15%

show the experimental results of various few-shot classification methods on the three datasets.

5. Discussion

From the results in Table 2, Table 3 and Table 4, it is evident that our method outperforms the others across all three datasets, achieving the highest average accuracy. Additionally, when compared to DEADN4, our method demonstrates improvements of varying degrees across all three datasets, particularly a significant enhancement on the UC Merced dataset, with an average increase in accuracy of $0.85\%$. The UC Merced test set comprises scene classes with multi-scale characteristics, such as rivers, sparse residential areas, and tennis courts, indicating that our method has effectively addressed the multi-scale challenges in RSISC.

To visually illustrate the efficacy of the KAL method, the localization results on the NWPU-RESISC45 dataset have been visualized, as depicted in Figure 7. Based on the visual outcomes, it is clear that our method accurately locate key areas within the scenes. In the airplane scene, targets of varying sizes can be accurately localized, while objects with slight scale variations in other scenes, such as rivers, storage tanks, thermal power stations, and parking lots, are also successfully located. This, in turn, benefits the classification processes of the local stream.

6. Conclusions

This paper introduces a few-shot scene classification model utilizing a dual-stream network. Our method constructs a global–local two-stream framework, integrating global and local streams, to calculate the likelihood of a sample belonging to a class by analyzing both the entire image and its most crucial regions. The KAL method is also designed to identify the most crucial areas within the entire image. Our method utilizes a greedy strategy to quickly identify the most critical areas within the overall image while linking the two aforementioned streams. To assess our method’s performance, tests are carried out on three public RSI datasets, and the visual outcomes of the KAL method on the NWPU-RESISC45 dataset are presented. Through a qualitative analysis and ablation experiments, our method can successfully tackle the multi-scale challenge in scene classification for few-shot RSIs. In the context of few-shot RSISC, mislabeled samples significantly impact the classification accuracy, and exploring strategies to mitigate their effects presents a promising direction for future research.