CRTED: Few-Shot Object Detection via Correlation-RPN and Transformer Encoder–Decoder

Abstract

Few-shot object detection (FSOD) aims to address the challenge of requiring a substantial number of annotations for training in conventional object detection, which is very labor-intensive. However, the existing few-shot methods achieve high precision with the sacrifice of time for exhaustive fine-tuning or have poor performance in novel-class adaptation. We presume the major reason is that the valuable correlation feature among different categories is insufficiently exploited, hindering the generalization of knowledge from base to novel categories for object detection. In this paper, we propose few-shot object detection via Correlation-RPN and transformer encoder–decoder (CRTED), a novel training network to learn object-relevant features of inter-class correlation and intra-class compactness while suppressing object-agnostic features in the background with limited annotated samples. And we also introduce a four-way tuple-contrast training strategy to positively activate the training progress of our object detector. Experiments over two few-shot benchmarks (Pascal VOC, MS-COCO) demonstrate that our proposed CRTED without further fine-tuning can achieve comparable performance with current state-of-the-art fine-tuned works. The codes and pre-trained models will be released.

1. Introduction

In recent years, object detection [1,2,3,4,5,6] has seen remarkable advancements through the application of deep neural models and large-scale training. Nevertheless, conventional object detection techniques typically depend extensively on vast amounts of quantity and quality annotated data and necessitate extended training duration, which has sparked the recent pursuit of few-shot object detection. The challenge in few-shot learning lies in the significant diversity of real-world objects, and despite noteworthy advancements, existing methods [7,8,9,10] primarily focus on image classification, seldom delving into the complexities of few-shot object detection. This might be due to the non-trivial nature of transferring knowledge from few-shot classification to few-shot object detection.

The core difficulty in object detection with limited examples lies in pinpointing unseen objects against a cluttered background. This is essentially a general problem of locating objects from a few annotated examples in new categories. Potential bounding boxes often overlook unseen objects or generate numerous false detections in the background. We argue that this is due to the sub-optimal scoring of promising bounding boxes by a region proposal network (RPN), making it challenging to detect novel objects. This distinction underscores the inherent difference between few-shot classification and object detection. Additionally, recent efforts in few-shot object detection [11,12,13] necessitate fine-tuning, preventing their direct application to novel categories.

In this paper, we attempt to address the problem mentioned above in few-shot object detection. First, we propose a novel network structure named Correlation-RPN based on a general RPN to activate the model to pay more attention to object-relevant regions and help learn the matching correlation between a query and support image feature, for generalizing the knowledge learned from base classes to novel classes. Secondly, we integrally migrate the transformer encoder–decoder into our framework. With the new feature coding mechanism, we utilize the decoder to obtain the correlational metric of feature representation after feature extraction in the backbone of our network. Thus, we introduce a four-way tuple-contrast training strategy to positively activate the training progress of our object detector.

The main contributions of this work include the following:

●

We propose a novel correlation-aware region proposal network structure called Correlation-RPN and migrate it to object detectors, improving detectors’ capacity of object localization and generalization;

●

We redesign a new feature coding mechanism and integrally migrate the encoder–decoder of the transformer into our model to effectively learn support–query feature similarity representation;

●

With our presented four-way tuple-contrast training strategy, CRTED without further fine-tuning can achieve comparable performance with most of the representative methods for few-shot object detection.

2. Related Work

2.1. General Object Detection

Object detection remains a key topic in computer vision, particularly with the rise of deep learning. CNN-based methods, always pre-trained on vast datasets, have gained popularity. These methods split into two categories: proposal-based and proposal-free detectors. The RCNN series [14,15,16] falls into the former, relying on pre-trained CNNs to classify region proposals from a selective search. SPP-Net [17] and Fast-RCNN [15] evolved from RCNN, extracting regional features via an RoI pooling layer from convolutional maps. Faster-RCNN [16] introduced a region proposal network (RPN) to enhance proposal quality. In contrast, YOLO [3,18,19,20] pioneered the proposal-free approach, using a single CNN for classification and bounding box prediction. Later works refined YOLO with default anchors for shape adjustment or multi-scale training. Proposal-free methods are simpler and faster but still rely heavily on annotated samples, limiting their performance in few-shot scenarios.

2.2. Few-Shot Object Detection

The challenging few-shot object detection (FSOD) problem aims to detect objects or novel classes at the instance level with limited annotations. Prior works on few-shot object detection can be mainly categorized into three paradigms: meta-learning, transfer learning and metric learning approaches. Meta-learning methods aim at devising a periodic and stage-wise meta-training paradigm to train a class-agnostic meta-model to help knowledge transfer from base classes to novel classes with few annotated labels, known as “Learning to learn”. Meta-RPN [21] and Meta faster-cnn [13] are proposed to generate class-relevant proposals while improving the instance alignment. Transfer learning-based methods, also known as fine-tuning-based methods, emphasize performing fine-tuning for few-shot object detection, which trained base classes and novel classes together, and fine-tuning the whole model, where the model is only trained on the base classes and then fine-tuned on a balanced set including both base classes and novel classes. MPSR [22] adopts a manually defined positive sample refinement branch to mitigate the object scale scarcity issue for few-shot object detection. Specially, TFA [23] pre-trains a base detector from abundant samples on a base set and fine-tunes it for novel classes. Metric learning approaches focus on learning good embedding spaces or appropriate metrics that facilitate downstream tasks, including cosine similarity [24], Euclidean distance to the class center, graph distances and so on. RepMet [25] achieves current SOTA results for few-shot object detection by simultaneously learning the parameters of the backbone network, the embedding space and the multimodal distribution of each training category within it in an end-to-end training manner. In [21], the authors exploit the similarity metric between the support set and query set in a few-shot setup to detect novel objects and suppress false detection in the background.

2.3. Transformer Encoder–Decoder

The transformer architecture, initially designed for machine translation, has been widely applied to various computer vision tasks. One notable example is DETR, a representative class of object detectors that leverage the strengths of transformers. These models employ a transformer encoder–decoder structure to understand and learn the relationships between the global image context and objects, using CNN features as input and producing final predictions. As a variant of DETR, ViDT [26] introduced a pre-trained transformer to replace the CNN backbone while maintaining a randomly initialized transformer neck. More recently, ViTDet [27] and MIMDet [28] have attempted to capitalize on the powerful network architecture pre-trained by MAE for object detection tasks. However, ViTDet only utilizes the pre-trained MAE encoder and discards the pre-trained decoder. In contrast, MIMDet retains the entire encoder–decoder for feature extraction, focusing on leveraging the reconstruction capabilities of the MAE decoder to mask input image patches, reducing additional inference costs. Unlike these approaches, imTED [29] employs a fully pre-trained transformer encoder–decoder that not only extracts features but also performs representation transformation, which offers a comprehensive utilization of the transformer’s capabilities, enhancing the performance of object detection tasks.

3. Approach

In this section, we will walk through the whole architecture designs in our proposed CRTED step by step. The structure of CRTED is exhibited in Figure 1. First, before introducing CRTED, we consider the few-shot object detection task it aims to achieve. We start with the preliminaries on the few-shot object detection setup that motivate our method. Then, we present our network architecture in detail for few-shot object detection. Finally, we describe the learning procedure of CRTED.

3.1. Preliminaries

• Problem Definition In this work, we focus on processing the task of few-shot object detection. Given two sets of classes, we define a base set C_base and a novel set C_novel, where C_base ∩ C_novel = ∅. We define two datasets, a base dataset D_base with sufficient annotated objects of C_base and a novel dataset D_novel with few annotated objects of C_novel. A few-shot object detector aims at classifying and localizing objects of C_base ∪ C_novel by learning from D_base ∪ D_novel. In a task of N_n-way K-shot object detection with N_n = |C_novel|, there are exactly K annotated instances for each novel class in D_novel. The goal of this work is to train a model that can detect novel classes in C_novel by only providing K-shot labeled samples for C_novel and abundant images from C_base. Basically, images from C_base are split into a support image set S_b containing support images $s_c$ with a close-up of the target object and a query image set Q_b containing query images $q_c$ which potentially contains objects belonging to the support class. Given all support images S_b, our model learns to detect objects in Q_b. For convenience, we denote C_base, C_novel, D_base and D_novel as C_b, C_n, D_b and D_n in the following sections.
• Rethink Region-Based Object Detectors The majority of current few-shot object detection methods rely heavily on the Faster R-CNN framework [16], which leverages a region proposal network to generate potentially relevant bounding boxes to facilitate subsequent detection tasks. The RPN plays a pivotal role, as it must not only distinguish between objects and the background but also filter out negative objects belonging to non-matching support categories. However, under the few-shot detection setting, where support image information is extremely limited, the RPN often struggles. It tends to indiscriminately focus on every potential object with a high objectness score, regardless of whether they belong to the support category or not. This behavior can hamper the generalization of knowledge from base classes to novel classes, and it also places a significant burden on the subsequent classification task of the detector, as it has to deal with a large number of irrelevant objects. Previous studies [1,3,4,16,21,30] have attempted to address this challenge by generating more accurate region proposals. Nevertheless, the issue persists, stemming from the inherent limitations of region-based object detection frameworks within the few-shot learning context. To truly address this challenge, it is essential to develop novel strategies that can effectively leverage the limited support image information, enhancing the discriminative capabilities of the RPN and ensuring that it focuses only on relevant objects, thus improving the overall performance of few-shot object detection systems.
• Rethink Transformer-Based Detection Frameworks The transformer [31] emerged as a revolutionary self-attention-based building block specifically tailored for machine translation tasks. This architecture revolutionizes the way sequences are processed, updating each element by scanning through the entire sequence and subsequently aggregating information from it. Seeking to harness the immense potential of the transformer, DETRs [1] introduced an innovative approach by integrating a transformer encoder–decoder architecture into an object detector. This integration enabled the system to tackle the intricate challenge of attending to multiple support classes within a single forward pass. Nevertheless, a notable issue persists: the vision transformers employed in DETRs were randomly initialized, limiting their capabilities to solely processing feature representations extracted by the backbone network. This constraint underscores the need for further advancements to fully unlock the potential of vision transformers in object detection tasks.

3.2. Architecture

• Correlation-Aware Region Proposal Network In generic object detection, an RPN is useful to provide region proposals and generate object-relevant anchors while suffering from performance drop when used in few-shot object detection, due to the low-quality region proposals for novel classes and the fatigue to capture inter-class correlation among different classes. Taking inspiration from the success of the RPN-based FSOD framework [21], we propose a novel network structure based on a general RPN which learns the matching correlation between the support set S_b and queries Q_b. Figure 2 shows the overall architecture of our proposed Correlation-RPN. Correlation-RPN can make use of the support information to be sensitively aware of the similarities and differences between S_b and Q_b, and is also able to provide high-quality region proposals with objects of target or novel classes, while relatively depressing proposals in the background or in other categories.

Figure 2. The overall view of the Correlation-RPN design. S_b: support set; Q_b: queries; X: support feature map; Y: query feature map. F_s and F_q denote the support–query feature extractor. F denotes the similarity of the support–query feature map, which includes a 1 × 1 × C and 3 × 3 × C vector. Then, the correlation map computed is to be fed into the RPN for generating proposals.

Figure 2. The overall view of the Correlation-RPN design. S_b: support set; Q_b: queries; X: support feature map; Y: query feature map. F_s and F_q denote the support–query feature extractor. F denotes the similarity of the support–query feature map, which includes a 1 × 1 × C and 3 × 3 × C vector. Then, the correlation map computed is to be fed into the RPN for generating proposals.

Specifically, we compute the correlational metric between the feature map of S_b and Q_b in a depth-wise manner. The similarity map is then utilized to build the region proposal generation. In particular, we denote the support features of S_b as $X\in R^{H\times W\times C}$ and the feature map of the query $q_c$ as $Y\in R^{H\times W\times C}$; the similarity is defined as follows:

$${\mathit{F}}_{h,w,c}=\sum _{i,j}{\alpha X}_{i,j,c}·{\beta Y}_{h+i-1,w+j-1,c},\hspace{1em}\hspace{1em}i,j\in \left\{1,\dots ,K\right\}$$

(1)

where F is the resultant correlation feature map, and α, β are control coefficients to prevent overly favoring features of either side. Here, X is used as the kernel to slide on the query feature map [32,33] in a depth-wise cross-correlation way [34]. Our work adopts the top architecture of the attention RPN [21]. We empirically find that a kernel size of K = 1 performs well in our case, since we argue that a global feature can provide a great object prior for objectness classification, consistent with [16]. In our case, the kernel is calculated by averaging on the support feature map X. The correlation map is processed simultaneously by a 1 × 1 convolution and a 3 × 3 convolution followed by the objectiveness branch and regression branch. Correlation-RPN is trained jointly with the network and elaborated as in Section 4.3.

• Feature Metric Matching Based on the idea of [1,29], we integrally migrate the transformer encoder–decoder as the pillar of the correlational metric aggregation module into our object detector. The feature metric matching is accomplished in the transformer encoder by the multi-head attention mechanism. Specifically, given the support features of S_b, denoted as $F_s\in R^{H\times d}$, and the query image $q_c$, of which the feature map is denoted as $F_q\in R^{H\times W\times d}$, the matching coefficients M can be obtained by the following:

  $$\begin{array}{c}\mathit{M}\left(HW,d,c\right)=Match\left(F_s,F_q\right)\\ =Softmax\left(\frac{\left(F_s\left(d,c\right)·S\right){\left(F_q\left(HW,d\right)·S\right)}^T}{\sqrt{d}}\right),\hspace{1em}\hspace{1em}c\in \left\{1,\dots ,C\right\}\end{array}$$

  (2)

  where HW is the feature spatial size, d is the feature dimensionality, C is the number of support categories and S is a cosine similarity shared by $F_s$ and $F_q$, to ensure they are embedded into the same linear feature projection space. The cosine similarity as the correlational metric of each pair of the feature representation of $F_s$ and $F_q$ is calculated via the following:

  $$cos\left(f_s,f_q\right)=\frac{1}{C}\sum _i^C\left(\frac{f_s·f_q}{|f_s|·|f_q|}\right),\hspace{1em}\hspace{1em}i\in \left\{1,\dots ,C\right\}$$

  (3)

  where $f_s$ and $f_q$ denote the single feature representation of $F_s$ and $F_q$. Finally, for each $q_c$ that may contain multiple complex instances, we ensure a fact that the choice of the m potential support cases for each of these instances is the same. Therefore, the average correlation score of the m same potential support objects can be considered as the similarity or shared feature representation between $q_c$ and the m potential S_b, with which we prefer the $s_c$ containing the most similar support instance as the powerful support patch of $s_c$ in training. And this process has experimentally demonstrated to be helpful for our training. The effectiveness of support–query feature similarity metric mining, i.e., distinguishing support objects similar to the query, is discussed in Section 4.3.
• Encoding Matching In order to achieve class-agnostic object prediction, we propose the utilization of a carefully crafted set of predefined task encodings, which serve as a bridge between the given support classes and the abstract task encodings space. By mapping the support classes to these encodings, we ensure that the final object predictions are constrained within the task encodings space rather than being limited to predicting specific classes on the surface level. Drawing inspiration from the positional encodings employed in the transformer architecture, we implement task encodings $\mathit{T}\in R^{H\times d}$ utilizing sinusoidal functions. This allows us to capture both local and global patterns within the task encodings space, enhancing the representational power of our approach.

Furthermore, encoding matching and feature metric matching share the same matching coefficients. This ensures consistency across different matching processes and simplifies the overall pipeline. The matched encodings Q_E are simply obtained through a straightforward process, further streamlining the prediction framework:

$${\mathit{Q}}_E=\mathit{M}⨂\mathit{T},$$

(4)

where $⨂$ denotes sinusoidal functional multiplication. In essence, our approach offers a more flexible and generalizable framework for object prediction, enabling us to transcend the limitations of traditional class-specific prediction methods and move towards a more abstract and powerful representation of objects.

• Modeling Background for Object Prediction Generally, under a few-shot object detection setup, the background does not belong to any target classes and usually takes up a lot of space in a support or query image. Images where objects only account for a small proportion and most of the area is complex background, which we also called hard samples, are as shown in Figure 3. Taking the consideration of this reason, we propose a learnable prototype BG-P and a corresponding task encoding BG-E (fixed to zeros), for explicitly modeling the background class. This can significantly eliminate the matching ambiguity when a query is very hard to match to any of the given support classes. And we additionally introduce background suppression (BS) regularization as an auxiliary branch to help in addressing this problem, which will be described in detail in the next section. The final output of the feature metric matching module can be obtained via the following equation:

  $${\mathit{Q}}_F=\mathsf{\tau }(\mathit{M}·\sigma \left(F_s\right),{BS(F}_q)),$$

  (5)

  where $\mathsf{\tau }(·)$ denotes the Hadamard product, $BS(·)$ denotes the background suppression operation and $\sigma (·)$ denotes the sigmoid function. By applying the matching coefficients M, we filter out features not matched to S_b, producing a feature map Q_F, denoted as Equation (5), that inhibits the negative impact of hard samples and highlights class-related objects from query set Q_b for each individual support class.

3.3. Training Procedure

• Two-Stage Training strategy Our training procedure consists of two stages: the base-class training stage on samples from C_b (C_train = C_b), followed by K-shot few-shot fine-tuning stage on a balanced set of samples from both C_b and C_n (C_train = C_b ∪ C_n). More precisely, in the second stage, where only K labeled samples are individually available for each class in C_n, K samples are randomly selected for each class in C_b to balance the training iterations between C_b and C_n.

Generally, a naive training strategy is matching objects of the same class by constructing a training pair $\widetilde{p_c}(q_c,s_c)$ where $q_c$ and $s_c$ are both in the same c-th class. However, a powerful model should not only be able to perform query–support feature similarity mining but also allow for capturing the inter-class correlation among different categories. For this reason, according to the different matching results in Figure 4, we present a novel four-way tuple-contrast training strategy to match the same category while distinguishing different categories. We randomly choose a query image $q_c$, a support image $s_c$ and a hard sample $s_h$ containing the same c-th category object and one other support image $s_n$ containing a different n-th category object, to construct the training pair $\widetilde{p_t}(q_c,s_c,s_h,s_n)$, where c ≠ n. In the training pair $\widetilde{p_t}(q_c,s_c,s_h,s_n)$, only the objects of the c-th category in $q_c$ are needed and annotated as foreground, while all other objects are neglected and treated as background.

During training, our model learns to match every proposal generated by Correlation-RPN in $q_c$ with the object of $s_c$. Thus, the model needs to not only match the same category objects from $\widetilde{p_c}(q_c,s_c)$ and $\widetilde{p_h}(s_c,s_h)$ but also distinguish objects in different categories from $\widetilde{p_n}(q_c,s_n)$. Nevertheless, there are a massive amount of background proposals, especially with $s_h$, which usually dominate the training. Taking the consideration of this reason, we adjust these training pairs $\tilde{p}$ to balance the ratio of proposals between queries and supports. The ratio of $\tilde{p}$ is kept as 2:1:1 for $\widetilde{p_c}(q_c,s_c)$, $\widetilde{p_h}(s_c,s_h)$ and $\widetilde{p_n}(q_c,s_n)$. According to their matching scores, we pick all N $\widetilde{p_c}(q_c,s_c)$ and select the top 2N $\widetilde{p_h}(s_c,s_h)$ and top N $\widetilde{p_n}(q_c,s_n)$, respectively, and calculate the matching loss on the selected training pairs.

• Detection Loss Function During the training process, we use the multi-task loss on each sampled proposal. It is worth mentioning that we choose the different loss function while optimizing the network in two stages. In the first stage, for each bounding box $B$, we predict a 2D classification vector $p\in [0,1]$ to represent the probability for the target object and background, respectively. Inspired by [35,36,37], concretely, for a mini-batch of N RoI box features ${\left\{r_{x,y}^i,u_{x,y}^i,t_{x,y}^i\right\}}_i^N$, our first stage loss ${\mathit{L}}_{first}$ is defined as follows with considerations tailored for detection:

  $${\mathit{L}}_{first}=\frac{1}{N}\sum _i^{N}f\left(u_{x,y}^i,p_i\right)·{\mathit{L}}_{r_{x,y}^i}$$

  (6)

  $${\mathit{L}}_{r_{x,y}^i}=\frac{1}{N_{t_i}-1} \sum _{j=1,j\ne i}^N\overrightarrow{\mathrm{S}} \left\{t_i=t_j\right\}·\log \frac{\mathrm{e}\mathrm{x}\mathrm{p}(\widetilde{r_i}·\widetilde{r_j}/\vartheta )}{{\sum }_{k=1}^N\overrightarrow{\mathrm{S}} \left\{k\ne i\right\}·\mathrm{e}\mathrm{x}\mathrm{p}(\widetilde{r_i}·\widetilde{r_k}/\vartheta )},$$

  (7)

  where x, y denotes locations, and $\vartheta$ is the hyper-parameter temperature as in [36]. $r_{x,y}^i$ refers to the encoded RoI feature of the detector head for the i-th region proposal generated by Correlation-RPN, $u_{x,y}^i$ denotes the IOU score of $r_{x,y}^i$ with a matched ground truth bounding box $B^{\ast }$ and $t_{x,y}^i$ denotes the truth annotation.

The second stage output also includes the vector $p\in [0,1]$ for distinguishing between background and target object classes. Different from the first stage, following the parameterization in [14], we present a new regression vector $t=\left(t_x,t_y,t_w,t_h\right)$, to specify a scale-invariant translation and height/width shift of log-space relative to a region proposal. In the second stage, we adopt binary cross entropy (BCE) loss for classification and smooth L1 loss for regression. In combination,

$${\mathit{L}}_{second}=\frac{{\lambda }_{cls}}{N} \sum _i^N\mathit{B}\mathit{C}\mathit{E}\left(c_i^{\ast },p_i\right)+\frac{1}{N}\sum _i^N\overrightarrow{\mathrm{S}} \left\{c_i^{\ast }==1\right\}· {\mathit{L}}_{smooth}\left(t_i-u_{x,y}^i\right),$$

(8)

where $c^{\ast }$ refers to the class label for the target object, and ${\lambda }_{cls}$ denotes a balancing factor, which we empirically set to 2.

Our total loss function is the combination of the first (Equations (6) and (7)) and second stage loss (Equation (8)):

$${\mathit{L}}_{total}={\mathit{L}}_{first}+\rho ·{\mathit{L}}_{second},$$

(9)

where $\rho$ = 3 is a balancing factor for second stage loss.

• Background Suppression (BS) Regularization In our proposed structure of CRTED, feature metric matching is developed with the encoder architecture design in the transformer by the multi-head attention mechanism. It is sure that this design can moderate the training stress for objects with various sizes, but it may still disturb the detector performance for localization in the scenario of hard samples, especially when in the few-shot condition. For this reason, we propose novel background suppression (BS) regularization, by utilizing object knowledge in the domain of ground truth bounding boxes for each training pair $\widetilde{p_c}(q_c,s_c)$. Specifically, for $q_c$ in $\widetilde{p_c}(q_c,s_c)$, we first obtain the middle-level $F_q$ of the target domain generated from Correlation-RPN. Then, we adopt a masking method that enables the ground truth labels of target objects in the image $s_c$ to be mapped to the convolutional cube. Consequently, we can identify the feature regions corresponding to the background, namely ${\mathit{R}}_{BS}$. To minimize the adverse effects of background disturbances, we choose L2 regularization to penalize the activation of ${\mathit{R}}_{BS}$:

  $${\mathit{L}}_{BS}=BS\left(F_q\right)={\left|\right|{\mathit{R}}_{BS}\left|\right|}_2$$

  (10)

With this ${\mathit{L}}_{BS}$, CRTED can depress regions of indifference while paying more attention to where we are interested, which is especially important for training in few-shot learning. More details and visualization results of the experiment are shown in Section 4.3.

• Proposal Consistency Control One of the differences between image classification and object detection is that the former extracts semantics from the entire image, while the classification signals for the latter come from region proposals. We adopt a settled IoU threshold ${\mathit{T}}_{iou}$ to assure the consistency of proposals, with the consideration that low-IoU proposals may result in excessive deviation to the center of regressed objects and therefore might include irrelevant semantic information. In the following formula, $f\left(·\right)$ is responsible for controlling the consistency of proposals, defined with the proposal consistency threshold φ:

  $${\mathit{T}}_{iou}=f\left(u_{x,y}^i\right)=\frac{1}{N}\sum _i^N\overrightarrow{\mathrm{S}} \left\{u_i\ge \phi \right\}· r\left(u_{x,y}^i\right),$$

  (11)

  where $r\left(·\right)$ can be re-weighted for object proposals with different levels of IoU scores. We experimentally find that φ = 0.75 is a good cutoff point on which the detector head can be trained according to most centered object recommendations.

4. Experiments

We mainly perform extensive experiments in both few-shot object detection benchmark datasets PASCAL VOC and MS COCO to assess the effectiveness of our proposed CRTED.

4.1. Few-Shot Object Detection Benchmarks

• Pascal VOC In [38], the study consists of images with object annotations of 20 categories where the categories split for C_b and C_n are 15 and 5 separately. We use training set D_b ∪ D_n from Pascal VOC 07+12 trainval sets for training, where D_n is randomly sampled from previously unseen novel classes with K-shot in {1, 2, 3, 5, 10}. Following existing studies [12,39,40], we consider the same three random partitions of base/novel classes and samplings introduced. Each split is referred to as follows: Novel Split set 1: {“bottle”, “aeroplane”, “sofa”, “cow”, “horse”/others}; Novel Split set 2: {“bus”, “horse”, “motorbike”, “cow”, “sofa”/others} and Novel Split set 3: {“boat”, “cat”, “aeroplane”, “sheep”, “sofa”/others}. For fair comparison, in each partition, we use the same sampled novel instances and report AP50 for the detection precision for C_b (bAP50) and C_n (nAP50) on the Pascal VOC 07 test set. The results are averaged over 10 randomly sampled support datasets.
• MS COCO In [41], a large-scale and more challenging object detection dataset is presented, which consists of 80 categories where |C_b| = 60, |C_n| = 20 and C_n are common to Pascal VOC. The training set D_b ∪ D_n is from the MS COCO 2017 training set, and we perform evaluations on five K images from the COCO 2017 val dataset, for which the number of shots is set to 1, 2, 3, 5, 10 and 30. The COCO-style detection precision of C_b ∪ C_n (AP), C_b (bAP) and C_n (nAP) is reported. The results are averaged over five randomly sampled support datasets.

4.2. Implementation Details

We follow the training pipeline of [21], and the basic deep architecture of our CRTED is trained end-to-end on four V100 GPUs parallel with optimizer Adam and a batch size of eight for each $\widetilde{p_t}(q_c,s_c,s_h,s_n)$. During training, we find that more training iterations may result in the model over-fitting to the training set D_b ∪ D_n and damage performance. The learning rate is thus experimentally set to 0.01 during the first training stage and gradually decayed to 0.0002 for 500 later iterations, which can lead to a better convergence point. We first perform the evaluation on the MS COCO 2017 training dataset and ensure only simple class objects appear in images for each $\widetilde{p_t}(q_c,s_c,s_h,s_n)$. During few-shot training, for one $q_c$ belonging to C_b ∪ C_n in $\widetilde{p_t}(q_c,s_c,s_h,s_n)$, we provide 40 support images, containing 30 belonging to C_b and 10 belonging to C_n, termed as four-way 10-shot contrastive training.

4.3. Ablation Studies

• Evaluation of Correlation-RPN We conduct sufficient experiments to assess our Correlation-RPN on different training strategies. To evaluate the proposal quality, we first compare the precision and recall on the top 50 proposals of the regular RPN, attention RPN and our proposed Correlation-RPN at a 0.5 IoU threshold. In addition, we also add the average best overlap ratio (ABO) across the ground truth bounding box $B^{\ast }$ as one of our evaluation metrics. As demonstrated in Table 1, our model with Correlation-RPN reveals better performance than the other two counterparts under the same training pairs and K-shot, producing performance improvement in all the evaluations, which indicates that our proposed RPN architecture can generate more object-relevant proposals to benefit the total detection prediction.

Table 1. Ablation studies on proposed Correlation-RPN and other counterparts.

Method			Precision	Recall	AP	ABO
Regular RPN	Attention RPN	Correlation-RPN
√			0.7923	0.8804	54.5	0.7127
	√		0.8345	0.9130	56.9	0.7282
		√	0.8509	0.9214	57.1	0.7335

Table 1. Ablation studies on proposed Correlation-RPN and other counterparts.

Method			Precision	Recall	AP	ABO
Regular RPN	Attention RPN	Correlation-RPN
√			0.7923	0.8804	54.5	0.7127
	√		0.8345	0.9130	56.9	0.7282
		√	0.8509	0.9214	57.1	0.7335

Specially, the visualization comparison of the attention from superficial layers, between our Correlation-RPN and other two counterparts, is also clearly offered in Figure 5. The results confirm that our Correlation-RPN has a better ability to pay attention to the target domain and provide more high-quality proposals. Especially when dealing with challenging samples, particularly objects that are significantly occluded, our method demonstrates an impressive ability to achieve a high level of confidence in accurate recognition, which ensures reliable and robust performance even in situations where traditional methods might struggle.

• Analysis of Matching Procedure for CRTED Figure 6 clearly shows the feature visualization of different object classes learned with and without the matching procedure under the same constraint. As demonstrated, with the matching procedure introduced to learn inter-class correlation and capture intra-class compactness, different classes are better separated from each other, which helps to reduce model misclassification and boost generalization ability among similar categories. Specially, Table 2 clearly verifies the effectiveness of our proposed matching procedure. When we adopt our method into the model, regardless of the number of support classes, CRTED can still boost the detection performance of novel classes under the one-shot setup, which indicates the capacity of our designed matching procedure in support–query feature similarity metric mining and the strong power of the transformer encoder–decoder. It is worth nothing that when there are multiple support classes, MP is able to exploit the inter-class similarity of the support–query image to obtain improvement in few-shot detection performance, especially for about 2.8% mAP under the five-shot setting.

  Figure 6. The t-SNE visualization of object classification in the feature space with and without our proposed matching procedure MP on the novel split set 2 of Pascal VOC.

  Figure 6. The t-SNE visualization of object classification in the feature space with and without our proposed matching procedure MP on the novel split set 2 of Pascal VOC.

  Table 2. Ablation studies to validate the effectiveness of our presented MP. C means the number of support classes.

  Method	MP	C	Novel mAP (IoU = 0.5)
  			1	2	3	5	10
  CRTED		1	28.2	43.3	51.6	54.0	60.3
  	√	1	31.3	45.2	53.1	56.8	63.0
  		5	33.7	46.5	52.4	57.1	61.8
  	√	5	37.3	51.1	54.5	58.2	63.3

  Table 2. Ablation studies to validate the effectiveness of our presented MP. C means the number of support classes.

  Method	MP	C	Novel mAP (IoU = 0.5)
  			1	2	3	5	10
  CRTED		1	28.2	43.3	51.6	54.0	60.3
  	√	1	31.3	45.2	53.1	56.8	63.0
  		5	33.7	46.5	52.4	57.1	61.8
  	√	5	37.3	51.1	54.5	58.2	63.3

• Impact of Background Suppression (BS) Regularization For the purpose of enhancing few-shot detection, we mainly evaluate whether our proposed BS regularization method can boost transfer learning for CRTED. As shown in Figure 7, our background suppression (BS) regularization can effectively help CRTED to reduce the background disturbances. And it is clearly shown in Table 3 that our proposed regularization method can significantly improve the performance, when the support images $s_c$ in the training set $\widetilde{p_t}(q_c,s_c,s_h,s_n)$ are scarce in the few-shot domain, especially in one-shot. Additionally, it is worth mentioning that we try to pick objects from as many categories as possible to verify the effect, which shows that BS is generally robust for different categories.

  Figure 7. Feature heatmap of background suppression (BS) regularization. Samples of different categories are selected from COCO val set. BS can effectively weaken negative influence from background disturbances and then activate CRTED to pay attention to object-relevant regions.

  Figure 7. Feature heatmap of background suppression (BS) regularization. Samples of different categories are selected from COCO val set. BS can effectively weaken negative influence from background disturbances and then activate CRTED to pay attention to object-relevant regions.

  Table 3. The regularized transfer learning for CRTED of AP50 on the Pascal VOC dataset on C_b ∪ C_n. BL: baseline. BS: background suppression (BS) regularization. The mAP results show that our proposed BS regularization method can significantly boost the baseline CRTED, when in few-shot object detection.

  Shots for Split 1	1	2	3	5	10
  CRTED BL	68.7	69.4	70.8	73.6	75.5
  CRTED BL+BS	69.8	70.2	72.0	75.4	76.5
  Shots for Split 2	1	2	3	5	10
  CRTED BL	65.5	66.8	69.9	71.3	73.3
  CRTED BL+BS	67.7	68.0	71.3	71.8	73.7
  Shots for Split 3	1	2	3	5	10
  CRTED BL	67.7	68.7	71.4	72.7	74.8
  CRTED BL+BS	68.6	70.4	72.7	73.9	75.1

  Table 3. The regularized transfer learning for CRTED of AP50 on the Pascal VOC dataset on C_b ∪ C_n. BL: baseline. BS: background suppression (BS) regularization. The mAP results show that our proposed BS regularization method can significantly boost the baseline CRTED, when in few-shot object detection.

  Shots for Split 1	1	2	3	5	10
  CRTED BL	68.7	69.4	70.8	73.6	75.5
  CRTED BL+BS	69.8	70.2	72.0	75.4	76.5
  Shots for Split 2	1	2	3	5	10
  CRTED BL	65.5	66.8	69.9	71.3	73.3
  CRTED BL+BS	67.7	68.0	71.3	71.8	73.7
  Shots for Split 3	1	2	3	5	10
  CRTED BL	67.7	68.7	71.4	72.7	74.8
  CRTED BL+BS	68.6	70.4	72.7	73.9	75.1

• Ablation of Training CRTED Refer to

  Table 4. Experimental results for different training strategies.

  Training Strategy	AP	AP50	AP75
  $\widetilde{p_c}$ 1-shot	48.5	55.9	41.1
  $\widetilde{p_t}$ 1-shot	48.7	56.2	41.2
  $\widetilde{p_c}$ 5-shot	57.7	68.1	47.3
  $\widetilde{p_t}$ 5-shot	58.1	68.4	47.7
  $\widetilde{p_c}$ 10-shot	58.8	69.2	48.4
  $\widetilde{p_t}$ 10-shot	59.7	70.5	48.8

  . We train our network with different training strategies and obtain 1.3% AP₅₀ improvement in the $\widetilde{p_t}$ 10-shot training strategy, compared with the $\widetilde{p_c}$ 10-shot training strategy. It is straightforward that the model performs better at the training strategy with $\widetilde{p_t}$ than with $\widetilde{p_c}$, which shows the importance of training pairs $\widetilde{p_n}$ and $\widetilde{p_h}$, included in $\widetilde{p_t}$. With adding $\widetilde{p_n}(q_c,s_n)$, even the object in $q_c$ belonging to an unseen class, the model can learn inter-class correlation and intra-class compactness from novel classes to enhance generalization ability. And the model can improve robustness with $\widetilde{p_h}(s_c,s_h)$, especially $q_c$ which is a hard sample. It is clear that with larger K-shot training, we achieve better performance, which also indicates a certain number of support images is beneficial to few-shot learning. We think that controlling the number of $s_h$ and $s_n$ to 1 can be sufficient in training the model for distinguishing different classes. And from Figure 8, our proposed training strategy can positively activate the training progress of our detector. Our full model thus adopts the $\widetilde{p_t}$ 10-shot training strategy.

4.4. Comparison with State-of-the-Art Object Detectors

Comparisons between our approaches and state-of-the-art few-shot object detectors on Pascal VOC and COCO are shown in

Table 5. A performance comparison of nAP₅₀ on Pascal VOC. Red and green fonts denote the best and second-best performance, respectively. With fine-tuning, CRTED achieves a new SOTA performance on the 1-shot, 2-shot and 3-shot setting and obtains comparable performance on 5-shot and 10-shot, demonstrating its strong generalization capability.

Method	Fine-Tune	nAP50 (Avg. on Splits for Each Shot)
		1	2	3	5	10
FSRW [12]	√	16.6	17.5	25.0	34.9	42.6
Meta R-CNN [13]	√	11.2	15.3	20.5	29.8	37.0
TFAfc [23]	√	27.6	30.6	39.8	46.6	48.7
TFAcos [23]	√	31.4	32.6	40.5	46.8	48.3
FSDetView [39]	√	26.9	20.4	29.9	31.6	37.7
A-RPN [21]	×	18.1	22.6	24.0	25.0	-
AirDet [42]	×	21.3	26.8	28.6	29.8	-
DiGeo [43]	√	31.6	36.1	45.8	51.2	55.1
CRTED (Ours)	×	21.6	27.0	29.2	30.2	35.4
	√	32.2	36.6	46.4	51.4	55.5

and

Table 6. Performance comparison of AP with k-shot on COCO validation dataset. Red and green fonts denote best and second-best performance, respectively. CRTED achieves comparable performance on baseline without fine-tuning and outperforms most of representative methods with fine-tuning, which indicates its strong power, especially in few-shot settings.

Method	Venue	Fine-Tune	Shots
			1	2	3	5	10	30
FSRW [12]	ICCV 2019	√	-	-	-	-	5.6	9.2
Meta R-CNN [13]	ICCV 2019	√	-	-	-	-	8.7	12.4
TFAfc [23]	ICML 2020	√	2.8	4.1	6.3	7.9	9.1	-
TFAcos [23]	ICML 2020	√	3.1	4.2	6.1	7.6	9.1	12.1
FSDetView [39]	ECCV 2020	√	2.2	3.4	5.2	8.2	12.5	-
MPSR [22]	ECCV 2020	√	3.3	5.4	5.7	7.2	9.8	-
A-RPN [21]	CVPR 2020	×	4.3	4.7	5.3	6.1	7.4	-
W. Zhang et al. [44]	CVPR 2021	√	4.4	5.6	7.2	-	-	-
FSCE [45]	CVPR 2021	√	-	-	-	-	11.1	15.3
FADI [46]	NIPS 2021	√	5.7	7.0	8.6	10.1	12.2	16.1
AirDet [42]	ECCV 2022	×	6.0	6.6	7.0	7.8	8.7	12.1
VFA [47]	AAAI 2023	√	-	-	-	-	16.2	18.9
CRTED	Ours	×	5.9	6.6	7.2	8.0	8.9	12.5
		√	6.3	7.3	8.7	9.6	12.8	19.3

. Following the default novel split setting of data with different K-shots in previous researches, our proposed CRTED without further fine-tuning can show comparable performance with fine-tuned methods or even new SOTA results for few-shot object detection. Our CRTED outperforms A-RPN without fine-tuning by 2.6% on AP metrics, which demonstrates the strong generalization ability of our detector, especially in the few-shot scenario. Specially, in terms of AP over different classes, our CRTED obtains comparable performance for several cases on Pascal VOC and performs better than most of the representative fine-tuned models with one-shot on the COCO val dataset, boosting from about ~0.3% to ~3.8% in improvement.

5. Conclusions

This paper presents a novel few-shot object detection model, CRTED, which introduces a newly proposed object-relevant region-based module Correlation-RPN and the powerful structure of the transformer encoder–decoder. Our model without fine-tuning has been trained and validated on Pascal VOC and COCO datasets, and extensive qualitative experimental results have been given. Specifically, with the proposed matching procedure, BS regularization and the novel four-way tuple-contrast training strategy, CRTED can perform comparably or even better than those detectors with exhaustive fine-tuning in the same evaluation. We hope that this work can provide good inspiration for further works in few-shot object detection.