Object Detection Using Improved Bi-Directional Feature Pyramid Network

Abstract

Conventional single-stage object detectors have been able to efficiently detect objects of various sizes using a feature pyramid network. However, because they adopt a too simple manner of aggregating feature maps, they cannot avoid performance degradation due to information loss. To solve this problem, this paper proposes a new framework for single-stage object detection. The proposed aggregation scheme introduces two independent modules to extract global and local information. First, the global information extractor is designed so that each feature vector can reflect the information of the entire image through a non-local neural network (NLNN). Next, the local information extractor aggregates each feature map more effectively through the improved bi-directional network. The proposed method can achieve better performance than the existing single-stage object detection methods by providing improved feature maps to the detection heads. For example, the proposed method shows 1.6% higher average precision (AP) than the efficient featurized image pyramid network (EFIPNet) for the MicroSoft Common Objects in COntext (MS COCO) dataset.

1. Introduction

For a long time, object detection has been one of the major topics in the computer vision field, because it is highly utilized in various applications. Recent deep learning-based object detection methods have two mainstreams, i.e., two-stage [1,2] and single-stage approaches [3,4]. The two-stage model is an early methodology that combines a region proposal network (RPN) that estimates the location of an object and a network that classifies the estimated object. It has received attention because it provides much higher performance than existing detection techniques. However, the two-stage model inherently requires considerable computational cost, so a single-stage model was born. The single-stage model extracts feature maps at several stages of convolutional neural networks (CNNs) and concurrently estimates location and class through detection heads. Thus, it provides a more efficient structure than the two-stage model. Because the single-stage model is suitable for real edge devices, it has recently attracted great attention.

The latest single-stage object detection techniques are mainly based on the feature pyramid network (FPN) [4,5,6,7,8,9,10,11]. FPN contributed to improving the performance of existing single-stage methods by aggregating information of feature maps of different sizes in a top-down or bottom-up manner. However, previous FPN-based techniques still have a problem with information loss occurring during the aggregation process, because they adopt a somewhat naive architecture [6,9,11]. In detail, aggregated feature maps do not have enough multi-scale feature information, which degrades performance. Meanwhile, a few approaches to improving performance by delivering global feature information to FPN have been reported [7,8,10]. Since they used a shallow CNN to extract global information, they have a problem with the extracted information still having local characteristics. Therefore, we claim that a more sophisticated network architecture is required to meet the purpose of each part of the proposed method in order to further improve the performance of FPN.

To solve the above-mentioned problems, this paper proposes advanced modules to improve detection performance. The proposed algorithm is composed of aggregation architecture and a global information extraction model (GIEM). First, the proposed aggregation architecture is designed to minimize information loss by properly fusing the aggregation modules of existing FPN-based techniques. Second, the proposed global information extraction model is constructed so that each point of the feature maps has global information through a non-local neural network (NLNN). Since the proposed method is more advanced than the conventional schemes, it can effectively provide information necessary for detection. As a result, the proposed detector can achieve high performance at a reasonable cost. The experimental results proved that the proposed method is superior to the existing techniques for PASCAL Visual Object Classes (VOC) and MS COCO datasets. For example, in the MS COCO dataset, the proposed method provides 1.6% higher AP than EFIPNet [10], a state-of-the-art (SOTA) technique.

The contribution points of the proposed method are as follows.

• This paper proposes an improved Bi-FPN that minimizes information loss by combining aggregation models of existing FPN-based single object detectors.
• GIEM introduces NLNN to understand the relation between the feature vector and the entire feature map, i.e., obtain global information.

2. Related Work

2.1. Multi-Scale Feature Representation

As for the existing single-stage object detection techniques [4,5,6,9,11,12,13,14,15], it is most common to aggregate feature maps of various sizes. The lower the level in a feature pyramid, the larger the feature map, so semantic information tends to be weak, but geometric information tends to be strong. On the other hand, as the level increases, the opposite tendency exists. Thus, if feature maps of multiple levels are properly aggregated, detection performance can be improved, because diverse information can be utilized together. Note that iterative connections (IC) are mainly adopted as a way to aggregate feature maps [6,9,12]. However, IC aggregates feature maps in a simple weighted sum way. Thus, the geometric and semantic information of each feature map is not properly aggregated, resulting in information loss. In particular, the disadvantage is that the detection performance of small objects is not satisfactory.

2.2. Global Information Extraction Network

Recently, many methods of utilizing additional information to improve the performance of single-stage object detectors have been proposed. Among them, a few methods that employ the entire image information, that is, global information, are attracting attention because of their relatively high performance [7,8,10]. They first resize an input image to a small resolution version and then extract features through CNNs. Next, local and global information are fused by injecting the extracted global information into each level of the feature pyramid. The authors of [7,10] have a disadvantage in that they cannot extract feature maps of sufficiently various sizes inherently because global information is obtained only by shallow CNNs. To mitigate this problem, the authors of [8] adopted convolutional layers with three different dilations. However, since the above-mentioned methods simply extract feature maps through convolutional layers, the extracted feature maps are still quite dependent on local characteristics.

3. Methods

In order to improve the existing naïve aggregation framework that causes information loss, this paper proposes a new single-stage object detector. Figure 1 shows the overall structure. VGG-16 [16] was used as the backbone network. The local information extraction model (LIEM) extracted multi-scale object context from feature maps (conv4_3, fc7, conv8_2, conv9_2) of the backbone network. At the same time, the global information extraction model (GIEM) extracted global information from down-sampled images. Finally, the aggregation network fused the feature maps extracted by LIEM and GIEM. As a result, the proposed method can improve classification and regression performance for objects of various sizes. From the following sections, each part of the proposed method is described in detail.

3.1. Local Information Extraction Model (LIEM)

This section describes the LIEM part of the proposed method. A total of four levels of feature maps are generated by the backbone network VGG-16 and extra layers as in Figure 1. Then, the improved bi-directional FPN (IBi-FPN) extracts local information from these feature maps (see Figure 2). Next, feature maps are iteratively fused in top-down and bottom-up manners. When fusing feature maps of different levels, the following weighted sum is used.

$$WS\left(f_1,f_2\right)=Conv\left(\frac{w_1\ast f_1+w_2\ast Resize\left(f_2\right)}{w_1+w_2+\epsilon }\right),$$

(1)

where $f_1$ and $f_2$ are two input feature maps, $w_1$ and $w_2$ are trained weights, and $\epsilon$ is set to 10⁻⁴. Conv is composed of a convolutional layer with 3 × 3 kernel and stride of 1, batch normalization, and ReLU. Resize is a function to match the size of feature maps of two levels and usually uses bilinear interpolation or a max pooling layer. The fused feature maps have richer context and location information than the original feature maps. However, due to such a fusion, features necessary for each level are smoothened, so objects of a specific size may not be detected. To prevent this problem, the Maxout function (MO) extracts important information from the original feature map and the fused feature map through the following equation.

$$L{c}_{}^{mx}=MO\left(f_1,f_2\right),$$

(2)

Here, the smallest feature map is used as it is because it has little effect on fusion. Finally, feature maps extracted from LIEM are expressed as follows.

$$Lc=\left\{L{c}_1^{mx},L{c}_2^{mx},L{c}_3^{mx},L{c}_4^{bt}\right\},$$

(3)

where $mx$ and $bt$ indicate Maxout and bottom-up manner fusion, respectively. Therefore, the proposed LIEM not only allows feature maps to have richer context and location information but also allows each level to retain important information originally possessed by the Maxout function.

3.2. Global Information Extraction Model (GIEM)

This section describes GIEM to obtain global information. First, in order to lower computational complexity, an input image is down-sampled. In order to reduce the computational cost in down-sampling, we employ three max pooling layers rather than a single large max pooling layer. Next, the down-sampled feature map is transformed into an enriched feature map (EFM) by three convolutional layers and one non-local block. This process is expressed by the following equation.

$$Gl{b}_1=\phi \left(I\right),$$

(4)

where $\phi$ indicates a combination of three convolutional layers, batch normalization, ReLU, and one non-local block. I denotes a down-sampled image. Figure 3 shows the non-local block used here. A non-local block (NLB) contributes to extracting global information of a given feature map by analyzing the relation between information in the feature map. Unlike a conventional NLB, the proposed NLB separates g and $\left(\Phi ,\theta \right)$. This separation contributes to more focus on finding relations and improving features. The g function generates richer feature maps through a squeeze-and-excitation block. In addition, in order to obtain feature maps of the same size as the feature map with local information obtained beforehand, the size of the feature map is repeatedly reduced through a non-local block. This is expressed as the following equation.

$$Gl{b}_i=\partial \left(Gl{b}_k\right),$$

(5)

where k is an index with k = 1, …, n−1th. Additionally, ∂ denotes one non-local block. The output of GIEM is defined by

$$Glb=\left\{Gl{b}_1,Gl{b}_2,Gl{b}_3,Gl{b}_4\right\},$$

(6)

The proposed GIEM allows pixels of each feature map to improve information through global relation through a non-local block. Therefore, the proposed GIEM provides richer information than the previous techniques.

3.3. Aggregation Network

This section describes how to aggregate feature maps obtained from the proposed LIEM and GIEM. The proposed aggregation network is constructed based on [8] (see Figure 1c). In this paper, the function for aggregating local and global features was determined heuristically, which is represented by

$$f_i=Con{v}_i^1\left(L{c}_i\otimes Gl{b}_i\right)\oplus Con{v}_i^2\left(f_{i-1}\right),$$

(7)

where i is an index with i = 1, …, n and $Con{v}_{}^s$ operation is composed of a convolution (kernel 3 × 3, stride s), BatchNorm, and ReLU layers. Further, $\otimes$ and $\oplus$ indicate element-wise multiplication and addition, respectively. Therefore, the information obtained from the proposed LIEM and GIEM is aggregated in this way, which minimizes information loss and improves context and location information.

4. Experiments

4.1. Datasets

PASCAL VOC [17] is a dataset with 20 classes. We constructed a training set composed of 16,000 images from the trainval of VOC2007 and VOC2012 and trained the network with the dataset. Additionally, it was evaluated with the VOC2007 test set consisting of 5000 images. The threshold of intersection over union (IOU) for calculating mean average precision (mAP) was set to 0.5.

MS COCO [18] is a dataset with 80 classes. A network was trained with a trainval dataset composed of 120,000 images and was evaluated with a test set composed of 40,000 images. For the evaluation with the MS COCO dataset, well-known pycocotools were used, and the average precision (AP) was calculated by averaging the results for the IOU thresholds in the range of 0.5 to 0.95.

4.2. Implementation Details

The backbone network VGG-16 was pre-trained on ImageNet [19]. For the MS COCO dataset, the initial learning rate was set to 2 $\times$ 10⁻³ and decreased by 10% at 90 and 120 epochs. For the PASCAL VOC dataset, the learning rate was set to 2 $\times$ 10⁻³ and gradually decreased to 10⁻⁴ and 2 $\times$ 10⁻⁵ at 150 and 200 epochs, respectively. We deployed the same settings as the standard SSD [3], where we used the same loss function, scales, and aspect ratios of the default’s boxes and the data augmentation method. Further, the weight decay was set to 0.0005, the momentum was set to 0.9, and the batch size was set to 32. We considered two input sizes, i.e., 300 $\times$ 300 and 320 $\times$ 320.

4.3. Evaluation Results

PASCAL VOC 2007: We compared the proposed method with various single- and two-stage methods.

Table 1. Comparison of the proposed method and the conventional models on the PASCAL VOC 2007 dataset.

Model	Backbone	Input Size	mAP	FPS
Two-Stage Detectors:
Faster RCNN [2]	ResNet101	1000 × 600	76.4	-
R-FCN [22]	ResNet101	1000 × 600	80.5	-
Single-Stage Detectors:
CHFANet [23]	VGG16	300 × 300	79.9	-
FERNet [24]	VGG16	300 × 300	80.2	-
EFIPNet [10]	VGG16	300 × 300	80.4	111
SFDet [25]	VGG16	300 × 300	78.8	-
RFBNet [26]	VGG16	300 × 300	78.6	-
DES [27]	VGG16	300 × 300	79.7	76.8
DFPR [20]	VGG16	300 × 300	80	-
GFR-DSOD [9]	DSOD	300 × 300	78.9	-
SSD [3]	VGG16	300 × 300	77.93	64.23
IBi-FPN (Ours)	VGG16	300 × 300	79.92	45.68
EFGRNet [7]	VGG16	320 × 320	81.37	44.4
DAFS [21]	VGG16	320 × 320	80.6	-
RefineDet [28]	VGG16	320 × 320	80	40.3
DSSD [29]	ResNet101	321 × 321	78.6	9.5
GFR-DSOD [9]	DSOD	320 × 320	79.2	-
RON [30]	VGG16	320 × 320	76.6	-
SSD [3]	VGG16	320 × 320	78.35	-
IBi-FPN (Ours)	VGG16	320 × 320	80.37	44

shows the comparison result on the PASCAL VOC 2007 test set. Input image size and performance figures of other techniques are cited as they are in the paper. In the case of the 300 $\times$ 300 input size and VGG-16 backbone network, the proposed method provides a significant performance gain of about 2% over SSD. However, the proposed method provides a little bit lower performance than SOTA models such as EFIPNet [10] and deep feature pyramid reconfiguration (DFPR) [20]. On the other hand, compared with different backbone networks, the proposed method provides a gain of 1.02% in terms of mAP over gated feature reuse-deeply supervised object detectors (GFR-DSOD) [9]. Although the proposed method effectively utilizes the information of feature maps to improve the performance of SSD, its performance is slightly less than the SOTA technique’s. Despite the slightly insufficient performance, we have obtained insight that if the components of the proposed method are fused with other algorithms, they can contribute to improving the performance of the algorithms. Detailed analysis is covered in the ablation study. Further, in the case of the 320 $\times$ 320 input size, the proposed object detector achieved mAP of 80.37, which is a significant gain of 2.02% over SSD. However, the proposed scheme shows 0.23% and 1.00% lower performance than dynamic anchor feature selection (DAFS) [21] and enriched feature guided refinement network (EFGRNet), respectively. In general, the performance of object detection techniques is highly dependent on datasets. In Table 1, the performance of the proposed method is inferior to that of EFIPNet, but in

Table 2. Comparison of the proposed method and the conventional models on the MS COCO dataset.

Methods	Input Size	Backbone	FPS	AP (0.5:0.95)	AP (0.5)	AP (0.75)	APs (0.5:0.95)	APm (0.5:0.95)	APl (0.5:0.95)
FAENet [31]	300 × 300	VGG-16	-	28.3	47.9	29.7	10.5	30.9	41.9
SSD [3]	300 × 300	VGG-16	50	25.1	43.1	25.8	6.6	25.9	47.6
LSNet [8]	300 × 300	VGG-16	76.9	32	51.5	33.8	12.6	34.9	47
EFIPNet [10]	300 × 300	VGG-16	71.4	30	48.8	31.7	10.9	32.8	46.3
IBi-FPN (Ours)	300 × 300	VGG-16	45.43	31.6	50.3	33	10.7	36.9	47.7
EFGRNet [7]	320 × 320	VGG-16	47.62	33.2	53.4	35.4	13.4	37.1	47.9
DAFS [21]	320 × 320	VGG-16	46	31.2	50.8	33.4	10.8	34	47.1
PASSD [13]	320 × 320	VGG-16	40	31.4	51.6	33.6	12.0	35.1	45.8
M2Det [5]	320 × 320	VGG-16	33.4	33.5	52.4	35.6	14.4	37.6	47.6
IBi-FPN (Ours)	320 × 320	VGG-16	42.01	32	50.9	33.7	12	37	47.1

regarding MS COCO, the proposed method is superior. The processing speed of the algorithm normally relies on the computer hardware. In fact, the computing power used by EFIPNet is stronger than that of ours. This leads to a difference in speed. The authors urge the reviewer to consider this tendency.

Figure 4 shows an example of detecting an object in an image of PASCAL VOC 2007 using the proposed method and SSD, respectively. We find that the proposed method shows better detection performance than the SSD by making good use of local and global information extracted from the SSD. In particular, the proposed method has qualitatively excellent performance by detecting objects that even the SSD cannot detect.

MS COCO: Table 2 compares the proposed method with the latest single-stage models on MS COCO dataset. In the case of the 300 $\times$ 300 input size, the proposed object detector provides a gain of 6.5% over SSD. In the case of the 320 × 320 input size, the proposed object provides a gain of 0.8% over DAFS [29]. The proposed scheme shows 1.2% and 1.3% lower performance than EFGRNet [7] and M2Det, respectively. However, note that the proposed method has a fast speed of 42.01 FPS for 320 × 320.

4.4. Ablation Study and Discussion

Through the following ablation study, each part of the proposed method is described in detail. To verify the performance of each module of the proposed method,

Table 3. The effect of each model of improved bi-directional FPN (IBi-FPN) on performance for the PASCAL VOC 2007 dataset.

Methods					mAP
Standard SSD	BiFPN	LIEM	GIEM
			NLNN	Split Branch
√	-	-	-	-	78.35
-	√	-	-	-	79.53
-	-	√	-	-	79.84
-	-	√	√	-	80.16
-	-	√	√	√	80.37

shows the results of the ablation study on the PASCAL VOC2007 dataset. Firstly, we compared the proposed LIEM with standard SSD. The proposed LIEM provided a gain of 1.49% in terms of mAP. Thus, we can find that the Maxout for fusion in the proposed method effectively fuses more features than the weighted sum of the existing BiFPN. In the case of GIEM, conventional NLNN and the proposed non-local block were compared. Both GIEMs effectively extracted global information and improved detection performance. In particular, the proposed non-local block shows 0.21% higher performance than NLNN, which proves that the proposed technique is effective. Therefore, we can say that every module of the proposed framework has a meaningful effect on the performance of object detection.

It is true that the performance of the proposed method is somewhat inferior to that of the SOTA technique. In other words, since the proposed method requires a certain amount of additional computation, it shows a slightly lower FPS than the SOTA scheme with similar performance. Nevertheless, it is worth noting that the proposed method gives significant intuition to the object detection field. That is to say, a meaningful way to improve the existing naïve approach was suggested. Therefore, if a light-weighting technique is applied to the proposed method, a computationally efficient object detector will be developed.

5. Conclusions

We propose a new single-stage object detection framework which contains LIEM, GIEM, and an aggregation network. LIEM based on a bi-directional model extracts high context and location information, and GIEM captures rich global features using non-local blocks. Additionally, the aggregation network generates informative feature maps by aggregating local and global features. Experimental results show that the proposed object detector achieves the desirable AP performance of 31.6% on MS COCO dataset, which is better than the existing single- and two-stage approaches. For example, the proposed method shows 1.6% higher AP than EFIPNet, which is a significant step forward in the object detection task. If further performance improvement is made in the future, it will be possible to apply the proposed method to the industry. Moreover, the processing speed of our object detector is 45.43 FPS, which is suitable for the real-time application of object detection. However, it is true that the processing speed of the proposed method still does not reach SOTA. Therefore, further improving the computational efficiency is our future research task.