Few-Shot Object Detection in Remote Sensing Imagery via Fuse Context Dependencies and Global Features

Abstract

The rapid development of Earth observation technology has promoted the continuous accumulation of images in the field of remote sensing. However, a large number of remote sensing images still lack manual annotations of objects, which makes the strongly supervised deep learning object detection method not widely used, as it lacks generalization ability for unseen object categories. Considering the above problems, this study proposes a few-shot remote sensing image object detection method that integrates context dependencies and global features. The method can be used to fine-tune the model with a small number of sample annotations based on the model trained in the base class, as a way to enhance the detection capability of new object classes. The method proposed in this study consists of three main modules, namely, the meta-feature extractor (ME), reweighting module (RM), and feature fusion module (FFM). These three modules are respectively used to enhance the context dependencies of the query set features, improve the global features of the support set that contains annotations, and finally fuse the query set features and support set features. The baseline of the meta-feature extractor of the entire framework is based on the optimized YOLOv5 framework. The reweighting module of the support set feature extraction is based on a simple convolutional neural network (CNN) framework, and the foreground feature enhancement of the support sets was made in the preprocessing stage. This study achieved beneficial results in the two benchmark datasets NWPU VHR-10 and DIOR. Compared with the comparison methods, the proposed method achieved the best performance in the object detection of the base class and the novel class.

1. Introduction

Object detection can help further decision-making analysis by obtaining the object location and category information of the region of interest (ROI) in an image [1,2,3,4]. Object detection is one of the important tasks in the understanding of satellite images, and it plays an indispensable role in military decision making, urban management, environmental monitoring, disaster detection, public safety, etc. [5,6,7,8,9]. Moreover, the result obtained by object detection techniques can also be used to track moving objects, which is of great significance for analyzing the state and movement pattern of the objects [10,11,12]. Currently, studies are relatively mature in strong supervised object detection technology using remote sensing images and have achieved a large number of excellent research results. However, for some specific research fields, it is extremely difficult to obtain a large amount of data to build a sample library to acquire excellent results, such as disaster data and military damage data [13,14,15,16]. Therefore, how to use limited remote sensing images to study the object detection problem under a small number of manually annotated samples is a topic of great research significance.

Object detection technology in remote sensing images has been relatively mature after decades of development, from the early methods based on geometric principles, methods based on low-level image feature extraction (color, texture, etc.), and traditional machine learning methods to current deep learning methods [17,18,19]. These gradually got rid of the complicated steps of manually designing features and developed toward a higher level of less artificial assistance and intelligent direction. The current mainstream deep learning methods include (1) two-stage detection models, such as R-CNN [20], Fast R-CNN [21], and faster R-CNN [22], which usually have high detection accuracy but slow speed, and (2) single-stage detection models, such as YOLO series [23] and SSD [24], which are usually faster but the detection accuracy is relatively low.

Additionally, some graph-based methods [25,26] are used to enhance the context information of remote sensing images, thereby enhancing the performance of object detection. Examples include the research on building an embedded graph attention network using latent spaces and semantic relationships among objects [27], the research on constructing semantic graphs by combining semantic graphs with word embeddings of object category labels on graph nodes [28], and the research on constructing semantic graphs by combined with the graph aggregation network to update the weight of the object node to achieve enhanced attention to the rotation angle and scale of the object [29].

These relatively mature models are deployed in application projects to solve some practical problems and improve production efficiency. However, remote sensing images are different from general natural images in that they have more complex spectral features, scene features, multiscale features, etc. Therefore, the application of these strongly supervised network models in object detection tasks in remote sensing images still faces many challenges [30,31]. The major challenges currently facing remote sensing object detection tasks are how to detect when there are very few samples, and how to use the trained model to adapt to infer new categories of objects.

To solve these practical problems, a few-shot remote sensing image object detection method was proposed. Few-shot object detection aims to expand the task of new category recognition and detection by training known category objects. It learns from the perspective of the model and learns the characteristics of the annotated base class. Its essence is to realize the transferability of knowledge through fine-tuning, and finally realize the detection of new categories. At present, few-shot object detection methods mainly include methods based on metric learning, methods based on meta-learning, methods based on data enhancement, and methods based on multimodal data [32,33,34,35]. However, the methods based on data enhancement and multimodal data are directly driven by data, but the generalization performance of these models is weak; thus, it is difficult to detect objects in other scenarios. The methods based on metric learning and meta-learning start from the perspective of model learning, learn and absorb the characteristics of data to form a model, and then identify the objects. This kind of method based on the model often faces problems such as incomplete features, as well as insufficient feature fusion ability and model learning ability, making the detection and recognition of the base class and new class inaccurate.

Addressing the above problems, this study proposes a few-shot remote sensing image object detection approach that integrates the global features of the support set (S) and the enhanced context dependency of the query set (Q). This method includes two parts: the base class training stage and the new class fine-tuning training stage. Moreover, our proposed network contains two important feature extraction modules and one feature fusion module. They are the meta-feature extractor (ME) for query set feature extraction, the reweighting module (RM) for support set feature extraction, and the feature fusion module (FFM) for fusing the two features. Among them, ME uses the optimized YOLOv5 [36,37] framework as the backbone, and focuses on the contextual dependencies of the query set objects through graph convolution. The RM is applied to obtain the global features of the support set, and then use the FFM to fuse it and the query set enhancement features obtained from ME; the output results are sent to the prediction layer of the model for prediction.

In summary, the main contributions of this study are as follows:

• This study innovatively proposes a few-shot remote sensing image object detection method based on meta-learning, which integrates the global features of the S and the enhanced contextual dependencies of the Q, improving the final feature representation ability and object detection performance.
• In the ME structure, a feature representation structure that takes into account the contextual long-distance dependencies of the Q was constructed, focusing on the regional similarity of the query features, and optimizing the encoding performance of the query features.
• In the RM, the global feature pyramid extraction (GFPE) module is constructed to enhance the global feature representation of the S. Simultaneously, a new fusion module of query meta-features and support features is designed, which enhances the salient representation ability after feature fusion.

2. Related Work

As a key research topic in the computer vision domain, object detection technology is of great significance for understanding the high-level semantic information of images. In recent years, with the continuous development of artificial intelligence (AI) technology, a large number of object detection methods based on deep learning have been proposed and applied in pedestrian detection, automatic driving, aerospace object detection, etc. [38,39,40].

2.1. General Object Detection

In the general field, the method of object detection is divided into two categories: two-stage and one-stage. Among them, the earliest two-stage method originated from R-CNN in 2014. This model appeared in the public eye with the combined mode of region proposal + CNN + support vector machine (SVM). It first generates a candidate area, and then judges whether there is an object to be detected in the candidate area, as well as the specific category of the object. Subsequently, inspired by this idea, representative two-stage methods such as SPP-Net [41], fast R-CNN, and faster R-CNN appeared one after the other. To meet specific application requirements, on the basis of these two-stage approaches, predecessors proposed a series of improved methods. For example, in the field of face detection, Sun et al. [42] perfected the shortcomings of faster R-CNN by combining improvement strategies such as feature splicing, multiscale features training, and joint pretraining, and achieved better face detection performance. Similarly, to solve the problem of poor vehicle detection results caused by large changes in vehicle size, severe occlusion, or truncated vehicles in the image, Hoanh et al. [43] replaced the backbone of faster R-CNN with MobileNet, and replaced the non-maximum suppression (NMS) algorithm with the soft-NMS algorithm, thus realizing fast and accurate vehicle detection. In addition, the impact of the loss function on the model is important; hence, some researchers proposed to use IoU-balanced loss functions to improve the positioning accuracy without sacrificing efficiency. Experiments proved that the proposed loss function is indeed helpful to improve the performance of faster R-CNN [44]. The two-stage object detection method directly solves the object or background candidates as a classification problem, which has the advantage of high detection accuracy. However, the YOLO series, as a representative network of one-stage, has the characteristics of fast speed. It redefines object detection as a regression problem, directly predicting image pixels as objects and their bounding box attributes [45].

In intelligent driving or assisted driving, to reduce the interference of irrelevant scene information, a real-time detection framework called increase–decrease YOLO (ID-YOLO) was used to simulate the driver’s selective attention mechanism, paying more attention to all objects in the driving scene, while suppressing other irrelevant objects [46]. To solve the problem that small objects are easily missed, Luo et al. [47] proposed to build a context matrix instead of the original classification matrix based on YOLOv3, and replaced the NMS module with the context filtering algorithm, finally obtaining a better detection effect, with an average accuracy of over 90%. In addition, in the actual application deployment, Wang et al. [48] took advantage of the fast speed of the YOLOv4 network and proposed a Trident-YOLO framework, which has played a huge role in mobile devices with limited computing power, whereby the parameter amount of the original modules was reduced, significantly improving the performance of object detection.

The above object detection algorithms have played an important role in different general fields and have achieved good results in practical applications. However, there are still a lot of challenges in object detection based on remote sensing images, including the problems of multiscale objects [49], dense objects [50], and arbitrary direction [51]. Scale, as the biggest feature of object detection in remote sensing, is the primary concern, which directly affects the performance of object detection. Aiming at the limited receptive field of the feature pyramid network (FPN) during feature extraction, Dong et al. [52] proposed to use the gated context-aware module (G-CAM) that adaptively perceives multiscale contextual information to enhance object detection performance. Similarly, since faster R-CNN weakly matches the scale variability of different objects in remote sensing images and the poor positioning performance of small objects, an enhanced feature extractor was proposed for faster R-CNN, such that small and dense objects have a stronger response [13]. Moreover, Zhang et al. [53] proposed a scale-adaptive proposal network (SAPNet), which consists of multilayer RPN and detection subnetworks that generate multiscale object proposals. The results showed that SAPNet significantly improves the accuracy of multi-object detection in remote sensing images. The arbitrary direction of the object in the remote sensing image often easily leads to the missed detection of the objects marked in the form of a horizontal bounding box (HBB). Therefore, a form of the oriented bounding box (OBB) with angle information annotation was applied. Combining high-level semantic information and low-level features can improve multiscale representation capabilities for remote sensing images, and an effective OBB object detector can be used to define objects in any direction, making object positioning more accurate, and ultimately yielding excellent object detection capabilities [14,54]. In addition to focusing on the objects in any direction, Li et al. [55] focused on solving the problem of the blurred appearance of similar objects (for example, roads and bridges). They achieved a high average accuracy (87%) with a dual-channel feature fusion network to learn local and contextual information about objects. In addition, Wang et al. [56] refined the object features by simulating the rotation angle and used the Bhattacharyya distance of the Gaussian distribution as a loss function to strengthen the learning ability and convergence ability, the experimental results on multiple datasets showed that the proposed network has excellent performance. Additionally, in terms of specific applications, more detailed object detection studies can be reviewed. Sun et al. [57] applied the object detection technology to the detection of composite objects (sewage treatment plants, golf courses, airports, etc.) and proposed a unified part-based convolutional neural network (PBNet) specifically for detecting composite objects. The network treats composite objects as a set of parts and incorporates individual part features into contextual information to improve overall composite object detection performance. Li et al. [58] proposed a CNN-based ship fine-grained detection network, which improved the efficiency and accuracy of fine-grained ship detection in large-format remote sensing images. Furthermore, to get rid of the existing detection methods relying too much on instance-level labels, Chen et al. [59] proposed a weakly supervised object detection network based on collaborative learning. The method introduces the joint pooling module and optimizes the candidate proposals and the object distribution characteristics; good performance was achieved.

2.2. Few-Shot Object Detection

The current deep learning object detection technology relies too much on annotated samples, which hinders the development of AI industrialization. In recent years, researchers have paid more and more attention to learning a model with excellent effect with a small amount of data annotation. Few-shot learning is a solution proposed to solve the problem of fewer annotated samples. At present, there is paradigm research based on meta-learning and paradigm research based on transfer learning [35,60,61,62,63].

The object detection based on meta-learning is to use the meta-learner to learn meta-knowledge from the query set and support set, and then complete the detection of the new class by adjusting the meta-knowledge. Generally, the few-shot object detection in remote sensing images focuses on multiscale feature enhancement, the relationship between support features and query features, etc. Specifically, a feature attention highlight module was proposed, which greatly promotes the detection performance of few-shot objects in remote sensing images [64]. Similarly, Zhang et al. [65] proposed a few-shot enhancing class features approach, which was applied to solve the large gap within the class and cause confusion in the recognition results. Furthermore, some researchers focused on the connection between query features and support features. To make full use of the information provided by the support set, Wang et al. [66] proposed a diversity measurement module, which was used to measure diversity information to obtain more meta-feature knowledge and strengthen the connection between support features and query features. Zhang et al. [67] proposed a few-shot remote sensing image detection method of self-adaptive global similarity and two-way foreground stimulator, which improved the spatial similarity and asymmetry problems between support features and query features. Zhu et al. [68] used the meta-learning method to integrate faster R-CNN into the reweighting module, and fine-tuned the reweighting module and the last layer of faster R-CNN in the meta-test stage, achieving good few-shot object detection performance. The huge scale and direction changes of objects in remote sensing images are major difficulties in few-shot object detection. Integrating FPNs and using prototype features to enhance query features are new ideas to solve this problem. The TINet proposed by Liu et al. explicitly aligns query features and support features while ensuring geometric invariance, which can obtain performance gains while maintaining the same inference speed [69]. In addition, some researchers explored lightweight models, which have a direct connection to the actual application deployment. Li et al. [70] proposed a meta-learning-based few-shot object detection method (FSODM). The model used YOLOv3-tiny as the main architecture, and a multiscale lightweight object detection framework was designed, achieving excellent results in both benchmark datasets. On the basis of the FSODM infrastructure, Zhou et al. [71] made a lightweight improvement for SAR image object detection, changed the meta-feature extractor to a lighter DarknetS model, and considered the correlation and saliency between different classes of the support set. The new model has achieved excellent performance in SAR images. In addition, for the detection of important time-sensitive objects such as ships and airplanes, more refined type identification is required. Therefore, Zhang et al. [72] proposed a few-shot multiclass ship detection algorithm with the attention mechanism and multi-relation detectors. Moreover, because the remote sensing images own complex background features, and as it is difficult to extract fine-grained features, Liu et al. [73] proposed an improved task sampling strategy for optimizing the object distribution, achieving excellent performance in fine-grained few-shot airplane object detection.

The two-stage fine-tuning approach (TFA) is a common method for few-shot object detection in transfer learning, and most examples are improved compared to faster R-CNN. In few-shot detection in the general field, the problem of sample scarcity often makes it difficult for models such as faster R-CNN to extract effective features, resulting in low object detection accuracy. Therefore, a large number of researchers have proposed combining transfer learning methods to solve the problem of few-shot object detection. Combined with the transfer learning, by adding an attention mechanism, and improving the loss function based on the faster R-CNN network, this move can significantly improve the generalization ability of the few-shot object detection [74]. Similarly, Zhou et al. [75] proposed a few-shot airplane detection method with a feature scale selection pyramid for the problem of small foreground size and low contrast in satellite video data. This method takes full advantage of the two-stage fine-tuning strategy and the characteristics of satellite video data. In addition, the low-shot mechanism has the problem of lack of internal class variation, while the high-shot mechanism has the problem of misalignment between the learning distribution and the real distribution, which will lead to poor detection of new categories. Therefore, the learning of model knowledge can be enhanced and transferred to guide new category detection, thereby improving the competitiveness of the model [76]. In addition, in the research on the efficiency of the few-shot object detection, some researchers proposed an efficient pre-training transmission framework without calculation increments, mainly by designing a knowledge-inherited initializer to reweight the frame classifier, which effectively facilitated the knowledge transfer process and increased the speed of adaptation [77]. Recent studies have shown that text data are beneficial for few-shot object detection. Lu et al. [73] proposed an FSOD method guided by text-modal knowledge, which uses text-modal knowledge to highly summarize the universality and uniqueness of the category, greatly reducing the confusing classification problem of new categories.

3. Methods

3.1. Overall Framework

Inspired by [70], a similar structure of the few-shot remote sensing image object detection framework based on meta-learning is proposed in this study. It proposes a few-shot remote sensing image object detection method that fuses the global features of S and the enhanced context dependencies of Q. The whole framework includes three important modules (ME, RM, and FFM) and the final bounding box prediction layer, as shown in Figure 1. The proposed framework consists of two stages: base training and fine-tuning training. The base training is used to train the dataset of visible categories and learn meta-knowledge from visible categories; the fine-tuning training stage transfers the meta-knowledge learned in the base training stage to train unseen category datasets.

Figure 1. The framework of the proposed network. This network contains three important modules: meta-feature extractor, reweighting module, and feature fusion module. The meta-feature extractor uses optimized YOLOv5 as the backbone to extract the meta-features of the query set. The reweighting module uses a global feature pyramid extractor to extract multiscale global features of the support set, and the fusion features through the feature fusion module operation are used as the input (F₀, F₁, and F₂) of the prediction layer, which is eventually used to detect the object.

The input of the proposed framework includes Q and S, where the Q can be regarded as test samples for evaluating the performance of the task, which does not have annotations; the S is used as training samples, including the images and annotations for individual instances. The few-shot object detection is also called the N-ways K-shot detection task, where N and K represent N categories in the support set S, and each category has K instance annotations. In addition, the Q generally contains Nq images, and these images come from the same set of the same category C in the S. In this study, we use the annotations in the support set S as an auxiliary for frequency-sensitive foreground enhancement (FE) processing to incorporate frequency knowledge into the feature learning of the support set S.

Specifically, as shown in Figure 2, we first use the annotation information of the support set S to generate the mask (Figure 2b), and then use the mask to intercept the annotation part of the supporting image (Figure 2a), forming a mask with only the foreground object category; secondly, this study constructs a frequency extractor as shown in Figure 2 to reduce the noise of the input image. In this study, the frequency extractor uses discrete Fourier transform [78] to realize the frequency representation of the two-dimensional (2D) image data f(w, h) of size W × H, and the formula is shown below.

$$F\left(k,l\right)=\sum _{w=0}^{W-1} \sum _{h=0}^{H-1} f\left(w,h\right)e^{-2\pi i\frac{kw}{W}}{e}^{-2\pi i\frac{lh}{H}},$$

(1)

where k ∈ [0, W − 1] and l ∈ [0, H − 1] represent the coordinates of the two-dimensional image, respectively. Meanwhile, i in the 2D Fourier transform is the imaginary unit.

Figure 2. Schematic diagram of foreground enhancement (FE) processing. (a) represents the original image from the support set S, (b) is the generated mask, (c) is the image obtained by frequency extractor, and (d) is the final output result obtained by fuse the (b,c). Where, the mask of the support set S was used to crop the foreground (i.e., the ground track field), and the frequency extractor was applied to perform frequency transformation to remove some noise, enhancing the object feature expression.

To better represent the frequencies of different bands, the representation of Cartesian coordinates is first converted to polar coordinates (r and θ), as shown in Equation (2), where AI(r) represents the average intensity of the signal of the two-dimensional image in the radial distance. Meanwhile, to enhance the difference between the high-frequency bands of the image, we take the average intensity AI(r) of the signal as input, and this vector is input into a high-pass filter (F_hp), finally obtaining the one-dimensional spectral vector V_I of input image I. The formula is shown in Equation (3).

$$\left(r,\theta \right)=F\left(k,l\right):r=\sqrt{k^2+l^2},\theta =\arctan \frac{l}{k},AI\left(r\right)=\frac{1}{2\pi }{\int }_0^{2\pi } \left|F\left(r,\theta \right)\right|d\theta .$$

(2)

$$v_I=F_{hp}\left(AI\left(r\right)\right),F_{hp}\left(x\right)=\left\{\begin{array}{c}x, r>r_{\tau }\hfill \\ 0, \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}\right.,$$

(3)

where $r_{\tau }$ is the threshold radius of high-pass filtering. V_I’s dimension representation is ${\mathbb{R}}^{Class\times 1}$, which is reshaped as ${\mathbb{R}}^{Class\times 1\times 1\times 1}$, followed by matrix multiplication with the original support set feature S_i $\in {\mathbb{R}}^{Class\times 3\times W\times H}$. Its output feature map is shown in Figure 2c. Here, Class represents the number of visible categories contained in the support set S. V_I can easily distinguish whether there is noise on the image in the high-frequency band; thus, it can be used for noise reduction. Lastly, the object-only image obtained from the first step in Figure 2 is merged with the image shown in Figure 2c to obtain the final output image in Figure 2d. Figure 2d reduces the noise of the background except for the object part, which is beneficial to the enhancement of object features. Figure 2d is the final input of the RM.

3.2. Meta-Feature Extractor

ME is used for the multiscale representation of the features of Q. Usually, the same object in a remote sensing image has feature representations of different scales. Therefore, adding a multiscale feature representation structure is beneficial to the detection of objects of different sizes. The ME module designed in this study is optimized with YOLOv5 as the backbone. The overall structure of the module is shown in Figure 3, including Backbone, PANet, and Output. Among them, the original backbone, CSPDarkNet, of the YOLOv5 model was replaced with a more lightweight G-GhostNet [79] module. This module can minimize network parameters and improve detection accuracy. Simultaneously, G-GhostNet can achieve a balance between accuracy and GPU delay, by deleting some activations layers based on C-GhostNet [80]. In addition, to obtain the context dependence of each object scale feature, inspired by [81], a graph convolutional unit (GCU) is introduced. The design of this unit is to enhance the close connection between similar pixels of each scale feature in the Output layer and deepen the difference between dissimilar pixels.

Figure 3. Meta-feature extractor module. The overall structure is an optimization of YOLOv5. G-GhostNet is used as the backbone structure, and PANet is used as the neck layer, with the multiscale GCUs structure as the output layer.

In this study, the specific settings of G-GhostNet are as shown in Table 1. The G-Ghost Bottleneck structure in the Backbone layer in Figure 3 corresponds to the specific operations shown in Stage L1, Stage L2, and Stage L3 in Table 1, respectively. The output feature Yn is the fusion of each stage with the complex feature $Y_n^c\in {\mathbb{R}}^{(1-\mu )c\times W\times H}$ obtained by the Block operation and the cheap feature $Y_n^g\in {\mathbb{R}}^{\mu \mathrm{c}\times W\times H}$ obtained by the Cheap operation. The value range of μ is 0 ≤ h ≤ 1; here, the value of μ is taken as 0.5. The specific expression is shown in Equation (4), where $Y_n^c$ is generally the feature obtained after the residual operation, and $Y_n^g$ is generally the feature obtained after 1 × 1 or 3 × 3 convolution.

$$Y_n=\left[Y_n^c,Y_n^g\right].$$

(4)

Table 1. The overall framework of G-GhostNet set in this study. Block represents a residual bottleneck, the Cheap operation is generally a 1 × 1 or 3 × 3 convolution to obtain cheap features, and Output represents the size of the output feature map and the number of channels at this stage.

Stage	Output	Operator	Stride
Stem	256 × 256 × 32	Conv 3 × 3	2
L0	128 × 128 × 128	Block	2
	128 × 128 × 128	Block × 1 Cheap	1
	128 × 128 × 128	Concat	1
L1	64 × 64 × 256	Block	2
	64 × 64 × 256	Block × 1 Cheap	1
	64 × 64 × 256	Concat	1
L2	32 × 32 × 512	Block	2
	32 × 32 × 512	Block × 5 Cheap	1
	32 × 32 × 512	Concat	1
L3	16 × 16 × 1024	Block	2
	16 × 16 × 1024	Block × 5 Cheap	1
	16 × 16 × 1024	Concat	1

In addition, in Figure 3, the multiscale features obtained by the 1 × 1 convolution of the output layer lack contextual long-range dependence. Therefore, the GCU is established by using the graph structure to strengthen the dependency between adjacent pixels. The structure of GCU is shown in Figure 4, and the specific calculation method is shown in Algorithm 1.

Figure 4. Schematic diagram of graph convolutional unit.

First, the feature X of the output layer after 1 × 1 convolution is reprojected spatially, and the K-means [82] algorithm is used to initialize the W ${\in \mathbb{R}}^{c1\times V}$ and Variance${\in \mathbb{R}}^{c1\times V}$. Here, C1 represents the channel size of the input feature X, and V represents the number of regions where the feature map X is divided, which is also the number of nodes in the graph structure. In Step 1 of Algorithm 1, the calculation formulas of any element of the probability matrix Q ${\in \mathbb{R}}^{H\times W\times c1}$ and the coding matrix Z ${\in \mathbb{R}}^{c1\times V}$ are shown in Equations (5) and (6). In order to achieve multiscale context long-range dependence, this study sets the value of V to 4 and 8, respectively. Therefore, the output features of GCU_i (i = 1, 2, 3) in Figure 3 are the feature fusion of the three features, which includes its own feature X, the feature $\tilde{X}$_V4 when V = 4, and the feature $\tilde{X}$_V8 when V = 8.

$$q_{ij}^k=\frac{\exp \left(-{∥\left(x_{ij}-w_k\right)/{\sigma }_k∥}_2^2/2\right)}{\sum _k \exp \left(-{∥\left(x_{ij}-w_k\right)/{\sigma }_k∥}_2^2/2\right)},$$

(5)

$$z_k=\frac{z_k^{\prime }}{{∥z_k^{\prime }∥}_2},z_k^{\prime }=\frac{\frac{1}{\sum _{ij} q_{ij}^k}\sum _{ij} q_{ij}^k\left(x_{ij}-w_k\right)}{{\sigma }_k},$$

(6)

where $q_{ij}^k$ is the probability of each pixel belonging to the region (node) k, k ∈ (1,V), $x_{ij}$ represents the feature of the pixel in the i-th row and j-th column of the two-dimensional feature map, and w_k represents the feature vector of the k-th node. In addition, ${\sigma }_k$ is the variance of all dimensions of node k, and it is normalized to (0,1) by the Sigmoid function.

Algorithm 1 Graph convolutional unit (GCU) algorithm

1: Input: A feature map X 2: Output: The enhanced context-dependent features representation $\tilde{\mathit{X}}$ 3: while in training(test) stage: do 4: // Step 1: The feature map X is projected into the graph structure to obtain the probability matrix Q, encoding feature Z and adjacency matrix A 5:   Init W, Variance ← KMeans(n_clusters = V) (X) 6:   probability matrix Q ← f (xij, ${\omega }_k$) encoding feature Z ← f ($q_{ij}^k\in Q, {\omega }_k\in W,{\sigma }_k\in \mathit{V}\mathit{a}\mathit{r}\mathit{i}\mathit{a}\mathit{n}\mathit{c}\mathit{e}$) adjacency matrix A ← $Z^{T}Z$ 7: // Step 2: The graph convolution operation, $W_g{\in \mathbb{R}}^{c1\times c2}$ is a random weight matrix 8: $\tilde{\mathit{Z}}$ ← f (A$Z^T{W}_g$), $W_g{\in \mathbb{R}}^{c1\times c2}$ 9: // Step 3: Reverse reprojection 10:  $\tilde{\mathit{X}}=Q{\tilde{Z}}^T$ 11: end while

3.3. Reweighting Module

The RM is used to extract meta-feature knowledge from the S, as well as to guide the positioning and object recognition of each image in the Q. The input of the RM is the changed form of V_I mentioned in Equation (3) (Section 3.1). The specific structure of the RM is shown in Figure 5. Figure 5a shows the overall structure of the RM. After multiple convolution and pooling operations, it is input into the GFPE shown in Figure 1, inspired by [83], and finally three multiscale feature weights (R₀${\in \mathbb{R}}^{\mathrm{W}/16\times \mathrm{H}/16\times 512}$, R₁${\in \mathbb{R}}^{\mathrm{W}/32\times \mathrm{H}/32\times 1024}$, and R₂${\in \mathbb{R}}^{\mathrm{W}/8\times \mathrm{H}/8\times 256}$) of S are obtained. Figure 5b shows the global feature extraction (GFE) block in GFPE. The module obtains the attention weight of each feature map (such as F${\in \mathbb{R}}^{W1\times \mathrm{H}1\times \mathrm{C}1}$ in Figure 5b), and then passes weighted averaging combines all features to generate global contextual features. Its calculation formula is shown below.

$${GF}_{ctx}=\sum _{i=1}^{N_p}\left(e^{W_g{F}_i}/\sum _{m=1}^{N_p}{e}^{W_g{F}_m}\right)F_i,$$

(7)

where F${\in \mathbb{R}}^{W1\times \mathrm{H}1\times \mathrm{C}1}$ = ${\left\{F_i\right\}}_{i=1}^{N=3}$ is the input feature map of the GFE block, F_i is any feature in F, N_p = W₁ × H₁ is the feature dimension in the feature map, and Z is the output feature map of the GFE block. For global feature extraction, W_g (1 × 1 convolution kernel) is first used to compress the feature map channel to 1, and then the Softmax function is used to activate W_gF_i. A 1 × 1 × H₁W₁ convolution is used to reshape, and then the attention weights are obtained. Finally, the weight $e^{W_g{F}_i}/\sum _{m=1}^{N_p}{e}^{W_g{F}_m}$ is applied to aggregate any feature F_i to obtain the global context feature $G{F}_{ctx}$.

Figure 5. Reweighting module: (a) the entire module structure of the reweighting module; (b) the GE block in the global feature pyramid extractor (shown by GFPE in Figure 1). In addition, the feature map output of each layer is shown in the feature dimension, e.g., W₁ × H₁ × C₁ denotes a feature map with width W₁, height H₁, and channel number C₁; ⊗ denotes matrix multiplication.

To establish the interdependence between channels, the global context features $G{F}_{ctx}$ are weighted and reassigned. The redistribution calculation formula is as follows:

$$Z=W_{t2}\left(\mathrm{ReLU}\left(LN\left(W_{t1}\left(G{F}_{ctx}\right)\right)\right)\right),$$

(8)

where $W_{t1}$ (1 × 1 convolution kernel) is used for channel dimensionality reduction of the feature map, followed by LayerNorm for feature normalization and ReLU for nonlinear activation; finally, $W_{t2}$ (1 × 1 convolution kernel) is used to increase the dimension of the feature map channel, restoring the size of the feature channel C₁, and the feature map Z is output.

3.4. Feature Fusion Module

To transfer the meta-knowledge of the RM to the ME, inspired by the omni-dimensional (OD) convolution [84], we propose a dynamic FFM, as shown in Figure 6, which implements the fusion of the output feature maps of the two-part structure. FFM introduces a multidimensional attention mechanism through a parallel strategy to learn more flexible attention to the four dimensions of the convolution kernel space. FFM can be expressed by Equations (9) and (10).

$${y=(\alpha w_1\odot \alpha f_1\odot \alpha c_1\odot \alpha s_1\odot W_1+\cdots +\alpha w_n\odot \alpha f_n\odot \alpha c_n\odot \alpha s_n\odot W_n)\times F}_r,$$

(9)

$$F_{f=}F.conv2d\left(F_{m,}y\right),$$

(10)

where F_r represents the feature map output (R₀, R₁, or R₂) by the RM, αw_i represents the attention scalar of the convolution kernel W_i, and αs_i ${\in \mathbb{R}}^{k\times k }$, αc_i ${\in \mathbb{R}}^{c_{in} }$, and αfi ${\in \mathbb{R}}^{c_{out}}$ represent the three newly introduced attentions, which work along the spatial dimension, input channel dimension, and output channel dimension, respectively. These four attentions are calculated using the multi-head attention module πi(x) [85,86]. By gradually multiplying the convolution W_i along the position, channel, filter, and kernel dimensions with different attention, the convolution operation can strengthen the difference of the input feature map in each dimension, as well as provide better performance to capture rich meta-knowledge contextual information. F.conv2d represents a two-dimensional convolution operation with weights, and F_m represents the feature map generated by the ME.

Figure 6. Schematic diagram of feature fusion module (FFM).

3.5. Loss Function

In order to obtain the excellent performance of few-shot remote sensing image object detection, a good loss function design is essential. In this study, we construct the loss function between the predicted bounding box and the ground-truth bounding box from the localization and classification parts of the object detection, so that the training phase gradually reaches convergence. First, for the localization loss, we adopt the mean square error loss to penalize the deviation between the predicted localization and the true localization. Its calculation formula is shown below.

$${\mathcal{L}}_{\mathrm{l}\mathrm{o}\mathrm{c}}=\frac{1}{N_{\mathrm{p}\mathrm{o}\mathrm{s}}}\sum _{\mathrm{p}\mathrm{o}\mathrm{s}} \sum _c {\left({\mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r}\mathrm{d}}_t^c-{\mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r}\mathrm{d}}_p^c\right)}^2.$$

(11)

Similar to [70], pos means that this study only considers the loss of positive anchor boxes, while ignoring the loss of negative anchor boxes. In practical situations, we choose the localization threshold empirically; for example, if the IOU between the predicted anchor box and the ground-truth anchor box is greater than 0.7, it is considered a positive anchor box; if the IOU between the predicted anchor box and the ground-truth anchor box is less than 0.3, it is considered a negative anchor box. In addition, c represents a coordinate selector to select from four coordinate representations {x, y, w, h} of a specific bounding box, respectively.

Since the current IOU loss does not take into account the mismatched direction between the ground-truth anchor box and the predicted anchor box, which leads to slow convergence and low efficiency, it is proposed to use the SIOU loss function [87] as the constraint between the predicted anchor box and the ground-truth anchor box.

$$L_{box}=1-IoU+\frac{\Delta +\Omega }{2},$$

(12)

where $\Delta$ represents distance cost, $\Omega$ represents shape cost, and $\Delta$ is a cost function redefined based on angle cost.

In addition, the confidence loss ${\mathcal{L}}_{\mathrm{obj} }$ needs to be paid attention to in object detection, and confidence loss focuses on the possibility of an object in a certain area. In this study, the confidence loss is a binary cross-entropy loss, which comprehensively considers objectness loss (${\mathcal{L}}_o$) and non-objectness loss (${\mathcal{L}}_{\mathrm{noobj} }$). Its calculation formula is shown below.

$$\begin{gathered} {\mathcal{L}}_{\mathrm{obj} }=w_{\mathrm{obj} }\cdot {\mathcal{L}}_o+w_{\mathrm{noobj} }\cdot {\mathcal{L}}_{\mathrm{noobj}} \\ =w_{\mathrm{obj} }\cdot \left(\frac{1}{N_{\mathrm{pos} }}\sum _{\mathrm{pos} } -\log {\mathcal{P}}_o\right)+w_{\mathrm{noobj} }\cdot \left(\frac{1}{N_{\mathrm{neg} }}\sum _{\mathrm{neg} } -\log \left(1-{\mathcal{P}}_o\right)\right), \end{gathered}$$

(13)

where ${\mathcal{P}}_o$ indicates represents the possibility of containing the object. To balance ${\mathcal{L}}_o$ and ${\mathcal{L}}_{\mathrm{noobj} }$, $w_{\mathrm{obj} }$ and $w_{\mathrm{noobj} }$ represent the weight, considering that, in actual situations, there are generally more negative boxes than positive boxes.

The cross-entropy loss function is used as the object classification loss, and the calculation method is shown in Equations (14) and (15).

$${\mathcal{R}}_c=\frac{e^{r_p^c}}{\sum _{c=1}^N e^{r_p^c}},$$

(14)

$${\mathcal{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}=\frac{1}{N_{\mathrm{pos} }}\sum _{\mathrm{pos} } \sum _c -y_c\log {\mathcal{R}}_c,$$

(15)

where $y_c$ represents the true category annotation of category c, and ${\mathcal{R}}_c$ represents the classification probability belonging to category c. ${\mathcal{R}}_c$ is normalized by the softmax function. $r_p^c$ (c = 1, 2, 3,…, N) represents one of the N predicted anchor boxes.

Ultimately, the loss function constraints in the entire few-shot object detection are shown in Equation (16).

$$\mathcal{L}={\mathcal{L}}_{\mathrm{loc} }+{\mathcal{L}}_{\mathrm{obj} }+{\mathcal{L}}_{\mathrm{cls}}.$$

(16)

4. Experiment and Results

4.1. Datasets and Evaluation Metrics

To verify the effectiveness of the method proposed in this study, this study conducts experimental verification using the two datasets of NWPU VHR-10 [88] and DIOR [2]. The specific dataset introduction is shown in Table 2.

Table 2. Details of the experimental dataset.

Datasets	Data Collection	Image Numbers	Types	Resolution	Image Size	Base Classes: Novel Classes
NWPU VHR-10	Google Earth/ISPRS Vaihingen dataset	800	10	0.5–2.0 m	Long side: 500–1200 pixels	7:3
DIOR	Google Earth	23,463	20	0.5–30 m	800 × 800 pixels	15:5

NWPU VHR-10: This dataset contains a total of 800 remote sensing images collected from the Google Earth and ISPRS Vaihingen datasets. The NWPU VHR-10 dataset is divided into two categories: one where each image has at least one manually labeled object instance, with a total of 650 images; the other where each image does not contain object instances, with a total of 150 images. The dataset contains 10 types of objects. According to the classification standards of most studies, the base classes include the ship, the storage tank, the basketball court, the ground track field, the harbor, the bridge, and the vehicle. The novel classes include the airplane, the baseball diamond, and the tennis court. Furthermore, in this study, few-shot experiments were used to evaluate the performance of the proposed method in 3/5/10-shot cases, respectively.

DIOR: This dataset contains a total of 23,463 images and is a large-scale benchmark dataset. The source of the dataset is mainly Google Earth, and a total of 20 categories are marked, including 192,472 instances. The size of each image is 800 × 800 pixels, and the image resolution range is 0.5–30 m. According to the general classification standard, the base classes include airport, basketball court, bridge, chimney, dam, expressway service area, expressway toll station, golf course, ground track field, harbor, overpass, ship, stadium, storage tank, and vehicle. There are five types of novel classes, including airplanes, baseball fields, tennis courts, train stations, and windmills. Furthermore, in this study, few-shot experiments were used to evaluate the performance of the proposed method in 3/5/10-shot cases, respectively.

To verify the performance of the proposed method and the advanced comparison methods, we use mean average precision (mAP) as the evaluation index, and its calculation process is shown in Equations (17)–(20).

$$\mathrm{P} =\frac{TP}{TP+FP},$$

(17)

$$\mathrm{R} =\frac{TP}{TP+FN},$$

(18)

$$A{P}_c={\int }_0^1 P_c\left(R\right)dR,$$

(19)

$$mA{P}_k=\frac{1}{k}\sum _{c=1}^k A{P}_c,$$

(20)

where TP denotes the number of true positives, FP represents the number of false positives, FN is the number of false negatives, and FP denotes the number of false positives. The AP calculation formula of a certain category c is shown in Equation (19). mAP_k is the average accuracy of the k-shot model in the case of k-shot.

4.2. Experiment Settings

This study uses an RTX 3090 graphics processing unit (24 G) for the training and prediction of the proposed network model, as well as the training and prediction process of the comparative experiment. In the experiment, due to the nonuniform size of the image, we resized the image; the image was reshaped to 512 × 512 pixels and sent to the network for experimentation. Moreover, we set up a mini-batch with a size of eight. The initial learning rate of network training was set to 1 × 10⁻⁴, and the weight decay was 5 × 10⁻⁵. For the base training of the NWPU VHR 10 dataset, the proposed model was trained for 12k iterations. In addition, for 3-shot, 5-shot, and 10-shot experiments, the models experienced 2k, 2.5k, and 4k iterations respectively. At the same time, for the base training of the DIOR dataset, the proposed model was trained for 8k iterations. In addition, for the 3-shot, 5-shot, and 10-shot experiments, the model experienced 2k, 3k, and 4k iterations, respectively.

4.3. Results on the NWPU VHR 10 Dataset

Table 3 presents the quantitative evaluation results of the experiments using the NWPU VHR 10 dataset in different state-of-the-art FSOD networks, including YOLOv5 [89], faster R-CNN (ResNet101) [22], FSRW [90], FSODM [70], TFA [91], PAMS-Det [92], G-FSDet [93], CIR-FSD [94], SAGS-TFS [67], TINet [69], and the proposed network.

Table 3. Comparisons among different FSOD networks on the NWPU VHR 10 dataset.

Class	Shot	YOLOv5	Faster R-CNN (ResNet101)	FSRW	FSODM	TFA	PAMS-Det	G-FS Det	CIR-FSD	SAGS-TFS	TINet	Ours
	Base Class Results
Ship		0.80	0.88	0.77	0.72	0.86	0.88	-	0.91	-	-	0.91
Storage tank		0.52	0.49	0.80	0.71	0.89	0.89	-	0.88	-	-	0.90
Basketball court		0.58	0.56	0.51	0.72	0.89	0.90	-	0.91	-	-	0.91
Ground track field		0.99	1.00	0.94	0.91	0.99	0.99	-	0.99	-	-	1.00
Harbor		0.67	0.66	0.86	0.87	0.84	0.84	-	0.80	-	-	0.90
Bridge		0.56	0.57	0.77	0.76	0.78	0.80	-	0.87	-	-	0.90
Vehicle		0.70	0.74	0.38	0.76	0.87	0.89	-	0.89	-	-	0.90
Mean		0.69	0.70	0.76	0.78	0.87	0.88	0.89	0.89	-	-	0.92
	Novel Class Results
Airplane	3	0.06	0.09	0.13	0.15	0.12	0.21	-	0.52	0.35	-	0.54
	5	0.10	0.19	0.24	0.58	0.51	0.55	-	0.67	0.64	-	0.69
	10	0.18	0.20	0.20	0.60	0.60	0.61	-	0.71	0.66	-	0.68
Baseball diamond	3	0.14	0.19	0.12	0.57	0.61	0.76	-	0.79	0.76	-	0.80
	5	0.20	0.23	0.39	0.84	0.78	0.88	-	0.88	0.82	-	0.87
	10	0.28	0.35	0.74	0.88	0.85	0.88	-	0.88	0.87	-	0.89
Tennis court	3	0.12	0.12	0.11	0.25	0.13	0.16	-	0.31	0.43	-	0.35
	5	0.15	0.17	0.11	0.16	0.19	0.20	-	0.37	0.52	-	0.52
	10	0.15	0.17	0.26	0.48	0.49	0.50	-	0.53	0.64	-	0.62
Mean	3	0.11	0.13	0.12	0.32	0.29	0.37	0.49	0.54	0.51	0.56	0.56
	5	0.15	0.20	0.24	0.53	0.49	0.55	0.56	0.64	0.66	0.64	0.69
	10	0.20	0.24	0.40	0.65	0.65	0.66	0.72	0.70	0.72	0.72	0.72

As shown in Table 3, the model proposed in this study achieved excellent mAP scores in the prediction results of both the base class and the novel class. In the base class detection results, the mAP score reached 92%, indicating that the network proposed in this study can achieve satisfactory results under a large number of samples. For the novel class detection, we performed experiments under 3-shot, 5-shot, and 10-shot settings, and the mAP reached 0.56, 0.69, and 0.72, respectively. Compared with several state-of-the-art few-shot object detectors, our model also had advantages. For example, compared with SAGS-TFS, our proposed model achieved a 5% improvement in mAP value in the 3-shot experimental setting and a 3% improvement in the 5-shot setting. As the number of object instance annotations increased, the mAP of the model also tended to be stable. Under the 10-shot setting, our model and the SAGS-TFS model achieved the same results, both 0.72. Meanwhile, compared with TINet, our model achieved the same performance in the 3-shot and 10-shot settings, and had a slight advantage in the 5-shot setting. However, for the prediction results of fully supervised YOLOv5 and faster R-CNN (ResNet101), these models achieved poor results when the training samples were gradually reduced. Simultaneously, when comparing the prediction results of the same type of meta-learning-based models such as FSRW and FSODM, our proposed method also achieved outstanding results. Especially in the experimental results under the few-shot setting, according to the numerical comparison of mAP, the model proposed in this study was 19%, 16%, and 7% better than FSODM. TFA, PAMS-Det, G-FSDet, and CIR-FSD are all two-stage fine-tuning methods. This type of model can generally achieve better results, especially if the prediction results in the base class are better, while the prediction results in the novel class are slightly lower. This is because of the introduction of novel class samples, whereby the model is still not perfect in adjusting the balance of foreground and foreground proportions.

Figure 7 shows the qualitative visualization results of the method proposed in this study for the 10-shot setting of the base class (results shown in light-blue background) and novel classes (results shown in light-yellow background) in the NWPU VHR 10 dataset. Combined with the quantitative analysis of Table 3, our model had excellent performance in the detection results of the base class and achieved a score of mAP above 0.90. Therefore, Figure 7 shows a good visualization effect of the base class. Especially for the ground track field, full-coverage detection was realized, and the mAP reached 100%, which shows that our model had the best detection effect for large-scale objects. Furthermore, for the detection of small and densely distributed objects such as ships and vehicles, the model proposed in this study showed a good detection performance. At the same time, the detection performance of novel classes also achieved the best visualization when compared with some state-of-the-art models. For example, it can be seen from Table 3 that FSODM had a large number of missed objects in novel class detection, while two-stage fine-tuning networks such as CIR-FSD had a higher probability of missed detection in tennis courts. The meta-learning-based model we proposed considers the enhancement of context information and global information, which can strengthen the distinction between the foreground and background of the tennis court, thus improving the accuracy of tennis court detection.

Figure 7. The visualization detection results of the NWPU VHR 10 dataset under the base class setting and the 20-shot setting. Results with light-blue background include the visual displays of the base class, and results with light-yellow background include the visual displays of the novel class.

4.4. Results on the DIOR Dataset

Table 4 shows the quantitative evaluation results of DIOR dataset in different state-of-the-art FSOD networks. Table 4 and Figure 8 jointly give some of the most advanced few-shot object detection models, and the quantitative results and qualitative visualization experimental results of the network proposed in this study in the DIOR dataset are shown.

Table 4. Comparisons among different FSOD networks on the DIOR dataset.

Class	Shot	YOLOv5	Faster R-CNN (ResNet101)	FSRW	FSODM	TFA	PAMS-Det	G-FS Det	CIR-FSD	SAGS-TFS	TINet	Ours
	Base Class Results
Airport		0.59	0.73	0.59	0.63	0.76	0.78	-	0.87	-	-	0.85
Basketball court		0.71	0.69	0.74	0.80	0.78	0.79	-	0.88	-	-	0.88
Bridge		0.26	0.26	0.29	0.32	0.52	0.52	-	0.55	-	-	0.56
Chimney		0.68	0.72	0.70	0.72	0.66	0.69	-	0.79	-	-	0.80
Dam		0.40	0.57	0.52	0.45	0.54	0.55	-	0.72	-	-	0.75
Expressway service area		0.55	0.59	0.63	0.63	0.66	0.67	-	0.86	-	-	0.87
Expressway toll station		0.45	0.45	0.48	0.60	0.60	0.62	-	0.78	-	-	0.69
Golf course		0.60	0.68	0.61	0.61	0.79	0.81	-	0.84	-	-	0.87
Ground track field		0.65	0.65	0.54	0.61	0.77	0.78	-	0.83	-	-	0.85
Harbor		0.31	0.31	0.52	0.43	0.50	0.50	-	0.57	-	-	0.56
Overpass		0.46	0.45	0.49	0.46	0.50	0.51	-	0.64	-	-	0.65
Ship		0.10	0.10	0.33	0.50	0.66	0.67	-	0.72	-	-	0.75
Stadium		0.65	0.67	0.52	0.45	0.75	0.76	-	0.77	-	-	0.81
Storage tank		0.21	0.21	0.26	0.43	0.55	0.57	-	0.70	-	-	0.72
Vehicle		0.17	0.19	0.29	0.39	0.52	0.54	-	0.56	-	-	0.54
Mean		0.45	0.48	0.50	0.54	0.63	0.65	0.71	0.74	-	-	0.74
	Novel Class Results
Airplane	5	0.02	0.03	0.09	0.09	0.13	0.14	-	0.20	0.17	-	0.21
	10	0.08	0.09	0.15	0.16	0.17	0.17	-	0.20	0.24	-	0.23
	20	0.09	0.09	0.19	0.22	0.24	0.25	-	0.27	0.33	-	0.35
Baseball field	5	0.09	0.09	0.33	0.27	0.51	0.54	-	0.50	0.50	-	0.49
	10	0.27	0.31	0.45	0.46	0.53	0.55	-	0.55	0.51	-	0.51
	20	0.30	0.35	0.52	0.50	0.56	0.58	-	0.62	0.53	-	0.64
Tennis court	5	0.10	0.12	0.47	0.57	0.24	0.24	-	0.50	0.62	-	0.63
	10	0.12	0.13	0.54	0.60	0.41	0.41	-	0.50	0.63	-	0.66
	20	0.20	0.21	0.55	0.66	0.50	0.50	-	0.55	0.63	-	0.68
Train station	5	0.00	0.00	0.09	0.11	0.13	0.17	-	0.24	0.17	-	0.18
	10	0.00	0.02	0.07	0.14	0.15	0.17	-	0.23	0.22	-	0.25
	20	0.02	0.04	0.18	0.16	0.21	0.23	-	0.28	0.28	-	0.30
Windmill	5	0.01	0.01	0.13	0.19	0.25	0.31	-	0.20	0.23	-	0.31
	10	0.10	0.12	0.18	0.24	0.30	0.34	-	0.36	0.24	-	0.40
	20	0.12	0.21	0.26	0.29	0.33	0.36	-	0.37	0.35	-	0.39
Mean	5	0.04	0.05	0.22	0.25	0.25	0.28	0.31	0.33	0.34	0.29	0.36
	10	0.11	0.13	0.28	0.32	0.31	0.33	0.37	0.38	0.37	0.38	0.41
	20	0.15	0.18	0.34	0.36	0.37	0.38	0.40	0.43	0.42	0.43	0.47

Figure 8. The visualization detection results of the DIOR dataset under the base class setting and the 20-shot setting. Results with light-blue background include the visual displays of the base class, and results with light-yellow background include the visual displays of the novel class.

In Table 4 and Figure 8, compared with the current state-of-the-art models, our proposed model achieved the highest mAP value and the most accurate visual detection results in the base class and the novel class; the highest base class detection mAP value reached 0.74, reaching the same level as the CIR-FSD model in base class detection. In the novel class, we performed experiments under 5-shot, 10-shot, and 20-shot settings, and the mAP values reached 0.36, 0.41, and 0.47 respectively. It can be found that, as the number of annotations increased, the effect improved, which is also in line with the principle of supervised learning. In the similar few-shot learning comparison, we found that the method proposed in this study improved the mAP value by 24% in the base class compared with the results of the FSRW network. Corresponding to the settings under 5-shot, 10-shot, and 20-shot settings, the detection results of the novel class were improved by 14%, 13%, and 13%, respectively, in terms of mAP values. In addition, for the G-FSDet network, since the prediction category of the network in the few-shot experiment differed from the experimental settings in this study and other network models, we took the maximum mAP of the network in different prediction categories as the comparison accuracy. In general, the detection accuracies of G-FSDet in the base class and novel class were better, but there were still objects with missed detection under very few samples.

Figure 8 shows the qualitative visualization results of the proposed model in the base class (results shown in light-blue background) and for the 20-shot setting (results shown in light-yellow background), respectively. According to the visualization results, our model paid more attention to reducing the missed detection rate of each dense object in the few-shot detection of DIOR dataset and had better detection performance than models such as FSODM, TFA, PAMS-Det, G-FSDet, CIR-FSD, SAGS-TFS and TINet. For example, in the dense object detection of airplanes, baseball fields, and tennis courts (Figure 8), there were only a small number of missed objects in the entire prediction set.

5. Discussion

5.1. Ablation Experiment

In order to further discuss the specific role and rationality of the proposed network in few-shot remote sensing image object detection, we discuss and analyze the effectiveness of important modules in the proposed model. Our ablation experiments are based on the NWPU VHR-10 dataset. Our baseline setting is a combined framework of ME based on the YOLOv5 framework and RM based on simple CNN network feature extraction. Then, the FE, GFPE, GCU, and FFM modules were added in turn for ablation experiments. The specific quantitative evaluation analysis is shown in Table 5.

Table 5. Ablation experiment on the NWPU VHR-10 Dataset.

Baseline	FE	GFPE	GCU	FFM	mAP
					3-Shot	5-Shot	10-Shot
√					0.32	0.51	0.62
√	√				0.36	0.54	0.64
√	√	√			0.48	0.60	0.68
√	√	√	√		0.52	0.64	0.69
√	√	√	√	√	0.56	0.69	0.72

Effectiveness of FE: The proposed FE module aims to enhance the ability to express foreground features in visible category images. After the FE module is operated, its output is used as the input of the RM to guide the object feature enhancement representation. As shown in Figure 9a, we visualized the feature maps of the middle layer before and after adding the FE module under the 10-shot setting (all feature maps in Figure 9 are based on the 10-shot setting). It can be observed from Figure 9a, when the FE module was not added, the shallow shape features of the foreground object and the background were more obvious, but the foreground and background did not have differential saliency expressions. When the FE module was added, the feature expression of the background was weakened, while the feature expression of the foreground was enhanced. Table 5 shows that the mAP values after adding the FE module were increased by 4%, 3%, and 2% respectively compared with the mAP values before adding the 3-shot, 5-shot, and 10-shot settings, which also directly shows that the introduced FE module is effective for few-shot object detection.

Figure 9. Effectiveness analysis of FE/GFPE/GCU/FFM. (a–d) represent the middle layer heat maps before and after adding different modules. The red rectangles indicate the salience representation of adding different modules.

Effectiveness of GFPE: The role of GFPE is to extract the multiscale features of the S, and its output feature map is used as the weight feature of the FFM module to fuse with the output feature of the ME. As shown in Figure 9b, after adding the GFPE module, the deep semantic information of the object features was more prominently expressed. As shown in Figure 9b, it can be seen from the details displayed in the red rectangle that the feature map without the GFPE module reflected the phenomenon of multi-object adhesion; for example, the boundaries between the harbors in Figure 9b were not completely distinguished. On the contrary, after adding the GFPE module, the harbor feature was more specific, and the difference between the harbor and the background was also more prominent. At the same time, it can also be seen from Table 5 that, after adding the GFPE module, the mAP under each experimental setting increased by 12%, 6%, and 4%, which reflects the importance of the GFPE module.

Effectiveness of GCU: As an important structure in the ME, the GCU module is used to guide the enhancement of the context-dependent information of the meta-features of the Q. By setting the multiple graph convolution nodes such as V = 4 and V = 8, and fusing the original features, the meta-features fusion of multiscale context information of the Q is realized. It can be observed from Figure 9c that the meta-features guided by the GCU module were more focused on the expression of object semantic information. The first line shows the expression of ship features before and after the introduction of the GCU module. It can be found that, after the introduction of GCU, the semantic information of the ship was more obvious, as shown in the details of the red rectangle. In addition, regarding the display of the second row of airplane features, before the introduction of the GCU module, the features of the airplane and the background were difficult to distinguish. As shown in the details of the red rectangle, after the GCU module was introduced, the semantic information of the aircraft was more obvious, and objects could be well distinguished from other objects and the background. At the same time, combined with the specific quantitative indicators in Table 5, the accuracy was 12%, 6%, and 4% higher than the 3-shot, 5-shot, and 10-shot settings without adding the GCU module. This also fully demonstrates the role played by the GCU module, which is consistent with the information reflected in the heat map after adding the GCU module.

Effectiveness of FFM: The function of FFM is to aggregate the multiscale features generated by ME and the multiscale weight features generated by RM to generate fusion features for final object detection. Through the middle layer representation of the Q features in Figure 9d, it can be found that the most important difference before and after adding the FFM module was that FFM could focus more on characterizing the object features, strengthening the difference with the background information, and enhancing the feature representation of the object. As shown in the red rectangle details in Figure 9d, before adding the FFM module, the object feature representation was weaker. Combined with the quantitative representation results of Table 5, FFM played a role in enhancing the characteristics of the Q. With small sample annotations, the accuracy increased by 4%, 5%, and 3% under 3-shot, 5-shot, and 10-shot settings, respectively.

5.2. Comprehensive Analysis

The detection ability of the base class and novel class in special cases: In order to show that the network proposed in this study has the capability of comprehensive detection of the base class and novel class, we selected some samples with multiple categories in the NWPU VHR-10 dataset for display. Figure 10 shows the combinations of ground track field + basketball court + baseball diamond, bridge + baseball diamond, basketball court + tennis court, and harbor + tennis court, which are all base classes + novel classes. The realization of these combinations is extremely challenging; the first three images show successful cases, while the fourth is a typical failure case. The failure of the fourth image was mainly due to the missed detection and false detection of the harbor class. The main reasons for imperfect detection of the fourth image were as follows: (1) most of the training sets of the harbor class were full of ships, while the test set had harbors that were not full of ships or had no ships; (2) the scale gap between harbor classes was large, resulting in a particularly large-scale harbor being detected as multiple harbors.

Figure 10. Comprehensive visual display of base + novel classes object detection results. The ground track field + basketball court + baseball diamond, bridge + baseball diamond, basketball court + tennis court, and harbor + tennis court combinations are shown respectively.

In addition, there were some cases of detection failure in the NWPU VHR-10 dataset, as shown in Figure 11: (1) failures caused by inter-class similarity, e.g., the nonobvious characteristic differences between a few vehicles and airplanes, and between airplanes and ships; (2) failures caused by objects being too densely distributed, e.g., an increase in the probability of repeated detection or missed detection due to the close-to-close distribution of various courts; (3) a small number of missed detections caused by noise, e.g., vehicle missed detection caused by tree occlusion or shadow. Overall, our proposed method has certain scene adaptability and generalization.

Figure 11. Failed detection cases are marked with red ovals. The cases correspond to failed detection of interclass similarity, false detection due to dense distribution, and missed detection caused by tree occlusion, respectively.

Model complexity and inference time: The complexity and inference time of the model are important factors for evaluating performance. Therefore, this subsection makes a statistical comparison of the complexity and inference time of the model proposed in this study and other representative state-of-the-art few-shot object detection models, as shown in Table 6. Table 6 shows that the complexity of the FSODM network was the highest, and the number of parameters reached 81.25 M, followed by the model proposed in this study, which reached 65.01 M. Simultaneously, compared with these superior models, we calculated the computational complexity of the model in the test phase with a batch size of 4 under the same standard as other papers, and the computational complexity of our model was at a medium level, reaching 197.89 GFLOPs.

Table 6. Model complexity comparison and inference time between our network and other FSOD approaches.

Model	FSODM	TFA	G-FSDet	CIR-FSD	SAGS-TFS	TINet	Ours
Params (M)	81.25	58.21	74.08/60.19	63.35	49.58	-	65.01
FLOPs (G)	216.38	154.70	-	158.98	327.17	-	197.89
Time (per image)	0.15	0.28	-	0.35	0.13	0.25	0.11

Although the model parameters proposed in this study were relatively high, the inference time per image was the lowest at 0.11 s. This shows that the proposed network could achieve optimal performance benefits at the cost of increasing a small number of parameters. In addition, TFA, G-FSDet, CIR-FSD, SAGS-TFS are two-stage fine-tuning approaches; Although the other models had slightly fewer parameters compared to our proposed model in this study, the single image inference time was higher for them. This can be attributed to the fact that our proposed model has less architectural complexity during the inference stage.In particular, G-FSDet model had a model complexity of 60.19 M after the reparameterization technique, and a complexity of 74.08 M before the reparameterization technique. In summary, the proposed model achieved a superior few-shot object detection performance.

6. Conclusions

This study proposed a few-shot remote sensing image object detection method that fuses the context dependencies of the query set and the global features of the support set. It mainly includes three important structures: ME, RM, and FFM. Firstly, the RM passes the multiscale support set global features as meta-knowledge to the query set; secondly, the optimized YOLOv5 model is used as the baseline of the ME, combined with the multiscale GCUs module to enhance context dependence relationship, and the FFM structure is combined with the meta-knowledge of the RM to make the query set features more prominently expressed and achieve the optimal detection performance. Experiments on NWPU VHR-10 and DIOR datasets showed that the proposed model outperformed some current advanced FSODM detection models and had good detection generalization performance for different remote sensing images.

However, combined with the failure cases in the comparative experiments and comprehensive experiments in Section 5, the proposed model still has room for improvement. For example, we will pay more attention to the detection of objects with large-scale differences and focus on the detection of small objects with stronger density. These problems will be important topics for our future work.