Semi-Autonomous Learning Algorithm for Remote Image Object Detection Based on Aggregation Area Instance Refinement

Abstract

Semi-autonomous learning for object detection has attracted more and more attention in recent years, which usually tends to find only one object instance with the highest score in each image. However, this strategy usually highlights the most representative part of the object instead of the whole object, which may lead to the loss of a lot of important information. To solve this problem, a novel end-to-end aggregate-guided semi-autonomous learning residual network is proposed to perform object detection. Firstly, a progressive modified residual network (MRN) is applied to the backbone network to make the detector more sensitive to the boundary features of the object. Then, an aggregate-based region-merging strategy (ARMS) is designed to select high-quality instances by selecting aggregation areas and merging these regions. The ARMS selects the aggregation areas that are highly related to the object through association coefficient, and then evaluates the aggregation areas through a similarity coefficient and fuses them to obtain high-quality object instance areas. Finally, a regression-locating branch is further developed to refine the location of the object, which can be optimized jointly with regional classification. Extensive experiments demonstrate that the proposed method is superior to state-of-the-art methods.

1. Introduction

With the rapid development of modern remote sensing technology, remote sensing image processing has been widely used in various fields, such as object detection [1,2,3], road mapping [4,5,6], agricultural planning [7,8,9], semantic analysis [10,11,12] and urban planning [13,14,15]. Recently, thanks to the stronger feature representation ability of Convolutional Neural Networks (CNNs) [16,17] and the availability of rich datasets with case-level annotations, the object detection of remote sensing images has achieved breakthrough performance. However, collecting such accurate annotations is labor-intensive and time-consuming. In contrast, scene-level tags can be easily obtained. This paper mainly studies the application of weakly supervised scene-level tags to realize semi-autonomous learning of object detections in remote sensing images.

In the existing remote sensing image object detection methods, the data used for the training network usually have object-level markers, which include the specific number, location, size or direction of all objects in the scene. However, the label at the scene level only records the category type of the main objects in each scene, and does not contain the specific object information in the scene. In addition, scenes with the same label usually contain different numbers of objects with different positions, sizes and orientations. The lack of this object information brings great challenges to the learning process.

With the aid of multiple instance learning, researchers [16,18,19,20,21,22,23,24,25,26,27] have shown that deep networks trained with only image-level/scene-level tags can generate pseudo ground truth, thus effectively predicting object location information. Most semi-autonomous learning object detection methods are divided into two stages [22,28,29,30]. In the first stage, a series of candidate boxes are generated by the object proposal method [31,32], and in the second stage, the features obtained from each proposal are regarded as instance-level classification problems. The core of these methods is to treat each image as a bag of potential object instance, and then train the instance classifier under the constraint of multi-instance learning. Promising results have been reported in the aforementioned methods. In [23], Ren et al. proposed an unpacking algorithm which iteratively removes negative instances from positive bags; this algorithm can effectively reduce the ambiguity of positive bags. However, the method ignores the accuracy of positive bags and may result in less accurate positive bags. Bilen et al. [22] proposed an effective multi-instance learning method, namely, the weakly supervised deep detection network (WSDDN), which can operate at the image region level and perform region selection and classification at the same time. Nevertheless, this method lacks displayed labels to classify instances. Later, Tang et al. [28] proposed a new online instance classifier refinement (OICR) algorithm, which integrates the multiple instance learning and instance classifier optimization process into a single deep network, and at the same time, it proposes to optimize instance classifier online by using a spatial relationship. This method can learn more distinguished instance classifiers by assigning binary labels on display. However, this method makes the network pay more attention to part of objects and ignores the complete objects. Thereafter, a proposal cluster learning (PCL) is presented to [29], to realize a refined instance classifier by learning the generated proposal clusters. As the proposals in the same cluster are associated with the same object, this can make the network focus more on the whole object instead of the parts of objects. In [30], Feng et al. designed a dual contextual refinement strategy to shift the focus of the detection network from locally different parts to the whole object by using local and global context information.

Although these studies have achieved good results, the development of semi-autonomous learning in remote sensing images still faces two major challenges. The first challenge is that deep residual networks, such as ResNet [33] and DenseNet [34], have become the standard backbones of many computer vision tasks, while the advanced semi-autonomous learning methods for object detection still rely on common networks, such as VGG [35]. The fundamental problem is unstable for the header of semi-autonomous learning, and is particularly sensitive to model initialization. This may propagate uncertain and wrong gradients back to the backbone, thus deteriorating visual representation learning. Especially in remote sensing image, the background is complex, the features are chaotic and the object size is small. The large kernel convolution kernel and non-maximum down-sampling in traditional ResNet will lead to the loss of highly informative features in the original image [36].

The second challenge is that most methods of semi-autonomous learning tend to select only the candidate frames with the highest scores to train the corresponding detectors. However, the suggestions with the highest scores and their associated suggestions usually cover only a part of the object instead of the whole object, especially in the large-scale and chaotic remote sensing image. Therefore, the above methods may not be enough to extract the complete object features. In addition, remote sensing images usually contain multiple instances of the same kind. If the highest score and its associated suggestions are simply selected as pseudo GT, other instances of the same kind may be missed, resulting in suboptimal object detectors [29].

For the first challenge, we designed a modified residual network (MRN) to solve the problem of losing highly informative features in the original image. Specifically, this paper replaces large convolution kernel and non-maximum down-sampling in a traditional residual network with a small kernel convolution and maximum down-sampling. It could enhance the robustness of network information flow by extracting finer object boundary features from the original image and keeping the information of instances. In the process of inputting remote sensing images into the deep neural network, continuous convolution and pooling will reduce the image size and increase the receptive field, which will lose some resolution and make some image details unable to be reconstructed. To solve this problem, the hole convolution is used in ResNet to increase the receptive field without the loss of feature information by the pooling operation. Each convolution output contains a wider range of feature information than an ordinary convolution, which is beneficial in obtaining the global information of the object features in remote sensing images.

In order to address the second challenge, this paper proposes an end-to-end aggregation-based region-merging strategy (ARMS), which combines similar regions in the cluster and deletes redundant regions in the cluster to select high-quality regions. In the process of network refinement, this strategy shifts the focus of network detection from part of the object to the whole object, and reduces the influence of background area on the network. More specifically, this paper does not directly select the suggestion with the highest score as pseudo-supervision, but selects other areas related to the suggestion with the highest score, and merges these related areas to form a complete object area, generating a new confidence level according to the merged form.

In this paper, MRN and ARMS are combined to form a progressive aggregation area instance refinement network. The MRN solve the problem of losing highly informative features in the original image. It extracts finer object boundary features in image processing, and preserves the information of instances. The ARMS is designed to select high-quality instances by selecting aggregation areas and merging these regions. ARMS selects the aggregation areas that are highly related to the object information through the association coefficient, and then evaluates the aggregation areas through the similarity coefficient and fuses them to obtain high-quality object instance areas. Moreover, a regression-locating branch is further developed to refine the location of the object, which can be optimized jointly with regional classification. Experiments on augmented NWPU VHR-10 and LEVIR datasets clearly demonstrate that the proposed method significantly outperforms previous state-of-the-art semi-autonomous learning object detection approaches on LEVIR and augmented NWPU datasets. In summary, the main contributions of this paper can be summarized as follows: (1) An end-to-end framework is proposed to realize semi-autonomous learning object detection based on image level annotations, which can jointly optimize the region classification and regression to improve the accuracy of object location. (2) The MRN is applied as the backbone to implement a lightweight network structure, which provides finer object boundaries and preserves the information of small instances, enhancing the robustness of information flow in the network. (3) A novel ARMS algorithm is proposed to utilize the cluster information of the region to obtain high-quality object features, and further suppress the interference of messy background information.

2. Materials and Methods

The overall architecture of the proposed framework is illustrated in Figure 1. The core goal of this method is to mine more accurate positive instances by making full use of aggregation areas under a weak supervision setting. For this reason, an instance refinement framework of the aggregation area based on a modified residual network is designed. Specifically, for each input image, the selective search method [32] is applied to generate about 2000 suggestions, MRN is used to extract the features of the image, and then the data enter the multiple instance learning (MIL) branch to generate instance-level supervision. Then, the AMSR further refines the instances, and finally uses the classification and regression branch to process the features at the same time to get the final result.

2.1. Modified Residual Network for Semi-Autonomous Learning

The residual network is a variant network proposed by Mike et al. in 2015, which has achieved excellent results in the field of image recognition and object detection [37,38]. The residual network has less computation and can solve the problems of network degradation and gradient dispersion. Deep residual networks, such as ResNet and DenseNet, have become the standard backbone of many computer vision tasks [39,40,41], but the advanced semi-autonomous learning methods still rely on common networks, such as VGG. The fundamental problem is that the header of the semi-autonomous learning network is particularly sensitive to model initialization. This may propagate uncertain and wrong gradients back to the backbone and deteriorate visual representation learning. Training the semi-autonomous learning network with ResNet backbone will reduce the identification of the proposed features, and will be weak in localizing object instances. Discovered by [36], the proposed semi-autonomous learning algorithm in this paper takes MRN as the backbone network.

In the traditional plain network, the data source of the network can only come from the previous layer; as shown in Figure 2a, the data flow down layer by layer. In the convolutional network, the data will flow into the next layer after down sampling, and data will be compressed in this process. In the residual structure, a design similar to the “short circuit” is introduced, in which the data output from the previous layer will directly skip over several layers and be introduced into the input part of the following data layer, as shown in the Figure 2b. Simply put, the uncompressed vector data of the previous layer and the further compressed data of the later layer will be used as the later input data together. More abundant dimension characteristic values are introduced into the network, so that the network can learn more abundant features. The residual convolution block is composed of several residual convolution submodules, as shown in Figure 2b. Assuming that the input of layer $l$ is $x_l$, then the input of layer $l+1$ is

$$x_{l+1}=H\left(x_l,w_f\right)=x_l+F\left(x_l,w_l\right)$$

(1)

where $w_l=\left\{w_{l,k}|1\le k\le K\right\}$ is the parameter of layer $l$, and $K$ represents the number of residual unit layers. Sequential cycle analogy is denoted as

$$x_{l+2}=x_{l+1}+F\left(x_{l+1},w_{l+1}\right)=x_l+F\left(x_l,w_l\right)+F\left(x_{l+1},w_{l+1}\right)$$

(2)

$$x_L=x_l+{\displaystyle \sum _{i=1}^{L-1}H\left(x_i,w_i\right)}$$

(3)

In Formula (3), it can be seen that the feature $x_L$ in layer $L$ can be divided into two parts—the first part is shallow network representation $x_l$, and the second part is the mapping of residual function ${\displaystyle {\sum }_{i=1}^{L-1}H\left(x_i,w_i\right)L}$, which indicates that the model is a residual form in any unit. Different from the plain network, the feature in layer $L$ is the product of a series of vectors, that is, ${\displaystyle \prod {}_{i=0}^{L-1}}{W}_i{x}_i$. In addition, a network often has the best network level; therefore, many deep networks have many redundant network layers. At this time, it is hoped that these redundant layers can complete identity mapping to ensure that the input and output through this identity layer are identical. As shown in Formula (3), in forward propagation, the input can be propagated directly from any lower level to higher level, which is equivalent to containing a natural identity mapping. This characteristic can solve the problem of network degradation to a certain extent. This also explains why the output in Figure 2b is $H\left(x\right)=F\left(x\right)+x$. In the residual network, for features with any depth, the result is the sum of all the previous residual modules, which increases the diversity of features. Similarly, the residual network also has good back propagation characteristics. Assuming that the loss is $E$, according to the chain derivation rule, we can get

$$\frac{\partial E}{{\partial x}_l}=\frac{\partial E}{{\partial x}_L}\cdot \frac{{\partial x}_L}{{\partial x}_l}=\frac{\partial E}{{\partial x}_L}\left(1+\frac{\partial }{{\partial x}_l}{\displaystyle \sum _{i=1}^{L-1}H\left(x_i,w_i\right)}\right)$$

(4)

It can be seen from the Formula (4) that the gradient consists of two parts—one is the information flow without any weight, another one is the weighted information flow. The linear characteristics of the connection between the two parts ensure that the information can be transmitted back to the shallow layer. At the same time, the formula also shows that for a small batch-size, the possibility of gradient disappearance becomes smaller.

Although the residual network has powerful advantages for visual tasks, it cannot be directly applied to semi-autonomous learning. This is because semi-autonomous learning is particularly sensitive to model initialization. Firstly, the large kernel convolution weakens the information of the object boundary, which leads to the uncertainty around the object boundary. Secondly, non-maximum down-sampling may also damage the flow of information, which makes small instances difficult to be perceived. This is because non-maximum down-sampling may not retain the activation and gradient of the information flowing through the network under weak supervision. Inspired by [36], a MRN is proposed in this paper, and the small kernel convolution and max-pooling are used to improve the robustness of information flow, which makes the object boundary more detailed. Specifically, the original stem block is replaced by three conservative 3 × 3 convolutions, and the first and third convolutions are followed by the max-pooling layer. For down-sampling, max-pooling is applied to change the average-pooling operation. The general depth residual network usually uses five stages to extract the whole image feature map. In the process of inputting remote sensing images into the deep neural network, continuous convolution and pooling operations will be carried out on the images, and this operate will lose some resolution and make some image details unable to be reconstructed. To solve this problem, this paper uses hole convolution to increase the receptive field when extracting the high-resolution weakly supervised feature so that each output of convolution contains a wider range of feature information than ordinary convolution, and it is beneficial to obtain the global information of object features. Specifically, the size of the space is fixed after the third stage, and the expansion convolution with a rate of 2 is added in the subsequent stage.

Table 1. The design details of MRN.

Layer Name	Convolution Kernels	Rate of Dilated Convolutions
Conv1	$3\times 3, \mathrm{conv}$ $2\times 2, \mathrm{maxpool}$ $3\times 3, \mathrm{conv}$ $3\times 3, \mathrm{conv}$ $2\times 2, \mathrm{maxpool}$	$-$
Conv2	$\left[\begin{array}{c}3\times 3, 64\\ 3\times 3, 64\end{array}\right]\times 2$	$-$
Conv3	$\left[\begin{array}{c}3\times 3, 128\\ 3\times 3, 128\end{array}\right]\times 2$	$-$
Conv4	$\left[\begin{array}{c}3\times 3, 256\\ 3\times 3, 256\end{array}\right]\times 2$	2
Conv5	$\left[\begin{array}{c}3\times 3, 512\\ 3\times 3, 512\end{array}\right]\times 2$	2
	$2\times 2,\mathrm{maxpool},\mathrm{stride}2$

displays some parameters of MRN. Figure 3 visualizes the feature map of the VGG backbone network and MRN. The first column is the original image, the second column is the feature map extracted by VGG-16 backbone network and the third column is the feature map extracted by MRN; all these feature maps are come from one specific CNN layer. It can be seen that the MRN can obtain clearer boundary features, especially for the small object in the complex scene. For the scene with multiple objects, the MRN can obtain more complete global information.

2.2. Multiple Instance Learning Branch (MIL Branch)

The image only contains the image-level label indicating whether the object category appears or not. In order to train a standard object detector, it is necessary to mine instance-level supervision, such as the bounding-box annotation. Therefore, this paper needs to introduce a MIL branch to initialize the pseudo ground truth (GT) annotations, such as [22,28]. The OICR network based on WSDDN is chosen in this paper because it is effective and can realize end-to-end training. WSDNN adopts a dual-stream network: classification stream $x_{cls}$ and detecting data stream $x_{det}$. Instance-level prediction can be achieved by aggregating these two streams. The scores of region proposal can be defined as

$$x_{cr}=x_{cls}\odot x_{det}$$

(5)

$$x_{cls}=\frac{e^{x_{ij}^c}}{{\displaystyle \sum {}_{k=1}^C{e}^{x_{ij}^c}}}, x_{det}=\frac{e^{x_{ij}^d}}{{\displaystyle \sum {}_{r=1}^{\left|R\right|}{e}^{x_{ir}^d}}}$$

(6)

where $x_{ij}^c$ comes from a matrix of data $x^c\in {ℝ}^{C\times \left|R\right|}$ and $x_{ij}^d$ comes from a matrix of scores $x^d\in {ℝ}^{C\times \left|R\right|}$; $\odot$ means element-wise operation, $C$ represents the number of image categories and $R$ denotes the number of object proposals. Suppose the image label is denoted as $Y={\left[y_1,y_2,\dots ,y_C\right]}^T\in {ℝ}^{C\times 1}$, $y_C=0$ indicates that there is no object in the image and $y_C=1$ indicates that one or more objects exist in the image. Then, the image-level class prediction score can be generated by summing up the scores over all proposals.

$$S_c={\displaystyle \sum _{r=1}^{\left|R\right|}{x}_{cr}}$$

(7)

The two-stream network of multiple instance can be trained as:

$$los{s}_{wsddn}=-{\displaystyle \sum _{c=1}^C\left\{y_C\log S_c+\left(1-y_C\right)\log \left(1-S_c\right)\right\}}$$

(8)

A preliminary classification result about the object instance can be obtained by instantiating the branch, but the result at this time is a very rough initial result. In order to get a finer classification result, similar to [29], this paper designs a refinement branch to be added after the instantiation branch. Different from OICR, our refinement branch contains two sub-branches, namely, the classification branch and location branch. In each refinement branch, the proposed ARMS strategy is introduced to obtain the most representative object area by using the information of the aggregation area; further, to discover instances with more complete characteristics by merging aggregation areas.

2.3. Aggregate-Based Region Merging Strategy (ARMS)

This section will explain how to mine trusted instances to achieve semi-autonomous object detection when only image-level tags are available. Firstly, the aggregation area is generated to shift the focus of the detection network from local different areas to the whole object area. Then, the aggregated local information is merged and the tag information is propagated to other possible instances.

2.3.1. Identification and Selection of Aggregation Regions

It is needed to select an aggregation center before selecting an aggregation area. As the top region can always detect a part of the object, the proposal with the higher score is selected as the clustering center, and then the aggregation region is obtained according to the degree of spatial overlap with the aggregation center after obtaining the proposal score. Although the proposal with the high score may not be the most complete, it can at least cover a part of the objects. The gathering area formed by taking it as the center will contain some objects or even the whole object, on the assumption that the $R$ object proposals have boxes $B=\left(b_1,b_2,\dots ,b_R\right)$ and a score $S=\left(s_1,s_2,\dots ,s_R\right)$. Suppose the object category $c$ exists in the image—that is $y_c=1$. Firstly, a series of the highest-scores are selected by

$$r_0^c=\arg \max (s_r^c)$$

(9)

Then, this proposal is chosen as the $n\mathrm{th}$ center of the aggregation area, i.e., $R_0^c=(b_n,y_n,s_n)=(b_{r_0^c},c,s_{r_0^c})$, where $b_{r_0^c}$ denotes the box of $r_0^c\mathrm{th}$ proposal region. To take advantage of aggregated information in the aggregation area, an association coefficient $\mathit{\sigma }\left(R_0^c,R_n^c\right)$ is introduced to describe the closeness between the center of the aggregation area and other proposals. The composition of the $n\mathrm{th}$ aggregation region corresponding to category $c$ can be expressed as

$$R_n^c=\left\{R_0^c|\mathit{\sigma }\left(R_0^c,R_i{}^c\right)>\mathit{\lambda }\right\}$$

(10)

$$\mathit{\sigma }\left(R_0^c,R_i^c\right)=\frac{area(b_0^c\cap b_i^c)}{area\left(b_0^c\right)}$$

(11)

$$area=\left(x_{\max }^i-x_{\min }^0+1\right)\ast \left(y_{\max }^i-y_{\max }^0+1\right)$$

(12)

where $b=\left(x_{\min },y_{\min },x_{\max },y_{\max }\right)$ denotes the coordinates of proposals. $x_{\min }$ and $y_{\min }$ represent the coordinates of the upper left corner, $x_{\max }$ and $y_{\max }$ represent the coordinates in the lower right corner and $\mathit{\lambda }$ is a threshold that is applied to select areas close to the central area to form the aggregation area.

2.3.2. Aggregation Region Merging

The selected aggregation area set $R_n^c$ can provide supplementary information to the object, which is helpful to obtain complete object features. Although the area with the highest score is not the most complete object region, it can cover at least a part of the object. Then, taking this region as the clustering center and gradually merging the adjacent areas, a nearly complete region can be obtained. However, one of the main problems is that the number of these regions is not fixed and may contain useless background information. Merging arbitrary regions may lead to negative effects. In order to solve this problem, a graph is designed to evaluate the similarity of these aggregation areas, and then the aggregation areas with high similarities are merged by the greedy strategy, so as to obtain the object areas with more complete features. The concept of the similarity coefficient $S{M}_{mn}\left(f_{mn},g_{mn}\right)$ is introduced to evaluate the similarity between areas. The similarity coefficient is divided into two parts—feature similarity and geometric similarity. It is well known that regions with similar features will have close detection scores. At the same time, the regions with large overlapping areas are more closely related. Therefore, the label information of local instances through a similarity coefficient measurement can be propagated to the global instance. The similarity measure can be expressed as:

$$S{M}_{mn}\left(f_{mn},g_{mn}\right)=\{\begin{array}{c}{f}_{mn}=\left|s_m/s_n-1\right|\\ g_{mn}=\mathit{\sigma }\left(R_m,R_n\right)\end{array}$$

(13)

where $R_m,R_n\in R_n^c$. $f_{mn}$ and $g_{mn}$ represent the feature similarity and geometric similarity between the region $m$ and region $n$, respectively. The regions in the aggregation area $R_n^c$ are sorted according to the score. A graph $G=\left(V,E\right)$ is established based on the similarity measure. The node $V$ in the graph denotes the aggregation area. Edge $E=e\left(V_m,V_n\right)$ represents the connection between nodes and it is determined by the similarity between two nodes.

$$e\left(V_m,V_n\right)=\{\begin{array}{l}1 \mathrm{if} f_{mn}<{\mathit{\tau }}_s \mathrm{and} g_{mn}>{\mathit{\tau }}_R\\ 0 \mathrm{otherwise}\end{array}$$

(14)

where ${\mathit{\tau }}_s$ denotes the feature metric threshold and ${\mathit{\tau }}_R$ is the geometric metric threshold. Therefore, when the similarity between two nodes is high, they will be connected together. Then, the greedy merging strategy will be adopted to merge these two regions with high enough similarity to obtain a new region. Since regions with similar features have similar detection scores, redefining the merged regions can shift the focus of the detection network from partial object information to complete object information. The score and coordinates of the merged area are defined as:

$$s_A=s_m+s_n$$

(15)

$$A\left(x_{\min }, y_{\min }, y_{\min },y_{\max }\right)=\left\{{}_{x_{\max }=\min \left(x_{\max }^m,x_{\max }^n\right), y_{\max }=\min \left(y_{\max }^m,y_{\max }^n\right)}^{x_{\min }=\max \left(x_{\min }^m,x_{\min }^n\right),y_{\min }=\max \left(y_{\min }^m,y_{\min }^n\right)}\right\}$$

(16)

As the aggregation regions are all closely related to the regions with the highest score, these regions contain at least a part of the features of the object, and further merging these parts of the features can obtain nearly complete features of the object. The ARMS mines more accurate instance information to a certain extent and obtains higher region scores, and provides more accurate monitoring information for subsequent object feature extraction. Figure 4 shows the process of aggregation region merging in ARMS. The Algorithm 1 summarizes the whole process of ARMS. In the first row, the relationship between region A and region C satisfies the description in Formula (14) according to the $S{M}_{AC}$, then region A and region C are merged. The same is done between region A and region B. In the last row, all regions do not meet the consolidation conditions in Formula (14), so there are no merge operations between regions.

Algorithm 1 Aggregate-Based Region Merging Strategy

Input: Image-level label $\mathbf{Y}={\left[y_1,y_2,\dots ,y_C\right]}^T$; proposal boxes $B=\left(b_1,b_2,\dots ,b_R\right)$; proposal score matrix $S=\left(s_1,s_2,\dots ,s_R\right)$.

Output: merge region score $s_A$; bounding box $A\left(x_{\min }, y_{\min }, y_{\min },y_{\max }\right)$

1: for c = 1 to $C$ do;

2:    if $y_c$ = 1 then;

3:    Choose the highest-score proposal $R_0^c$ by Equation (9);

4:    Find aggregation region $R_i^c$ by Equations (10)–(12);

5:    Build a graph between aggregation areas;

6:    Compute similarity between $R_m$ and $R_n$ by Equation (13);

7:    Merge aggregation region by Equation (14);

8:    Generates the instance-level score and coordinate by Equations (15) and (16);

9:    End if;

10:     End for.

2.4. Multi-Task Classification and Regression Branch

After executing the above process, the pseudo GT can be obtained, and the subsequent classification task can be carried out in a fully supervised way. Between WSDDN and OICR, the subsequent classification only includes one branch of classification. Inspired by [42], this paper uses a multi-task classification and regression branch to perform detection tasks at the same time, similarly to Faster-RCNN [43], in which the classification branch outputs the discrete probability distribution, which is calculated by a softmax on the $C+1$ outputs of FC layer. The regression branch predicts a bounding-box regression of each $C$ object classes. Suppose the output of the classification branch and regression branch are $p^c\in {ℝ}^{(C+1)\times 1}$ and $b^c=\left(b_x^c,b_y^c,b_w^c,b_h^c\right)$, respectively. Suppose $los{s}_{det}$ is used for the classification and bounding-box regression. The loss function of the network is defined as

$$loss=los{s}_{wsddn}+los{s}_{det}$$

(17)

$$los{s}_{det}=los{s}_{cls}\left(y^c,p^c\right)+\mathit{\lambda }los{s}_{reg}\left(t^c,b^c\right)$$

(18)

where $los{s}_{cls}$ and $los{s}_{reg}$ represent classification loss and regression loss, respectively. $\mathit{\lambda }$ is the balance parameter that controls the balance between these two losses. $los{s}_{cls}$ and $los{s}_{reg}$ can be calculated as:

$$los{s}_{cls}\left(y^c,p^c\right)=-\frac{1}{\left|R\right|}{\displaystyle \sum _{r=1}^{\left|R\right|}{\displaystyle \sum _c^{C+1}{\omega }_r{y}_r^c\log p_r^c}}$$

(19)

$$los{s}_{reg}\left(t^c,b^c\right)={\displaystyle \sum _{i\in \left(x,y,w,h\right)}smoot{h}_{L_1}\left(t_i^c-b_i^c\right)}$$

(20)

$$smoot{h}_{L_1}\left(x\right)=\{\begin{array}{c}0.5x^2 \hspace{0.17em}\mathrm{if}\left|x\right|<1\\ \left|x\right|-0.5 \mathrm{otherwise}\end{array}$$

(21)

where $t^c=\left(t_x^c,t_y^c,t_w^c,t_h^c\right)$ denote the pseudo GT bounding box and $y^c$ represents the real label. ${\omega }_r$ denotes the loss weight and can be calculated following the weights calculation method in [28]. $R$ is the number of region proposals.

3. Results

3.1. Dataset and Evaluation Metrics

The proposed method is evaluated on the augmented NWPU VHR-10 [44] and LEVIR [45] datasets. The NWPU VHR consists of 650 images (400 × 400 pixels) from ten object categories. This dataset is composed of three groups: the train, test and valuation set. In this experiment, 650 images were expanded to 1950 images by rotating the images 90 degrees, 180 degrees and 270 degrees, respectively. Half of the images are used for training data and half for testing. All of the 1950 images were scrambled, and the training data and test data were randomly selected from these images in proportion to ensure the balance and randomness of the training and test data. In the test, two standard metrics are used to evaluate the performance of the proposed method. Average precision (AP) is measured to evaluate the proposed model on the test set, and the correct position (CorLoc) is applied to evaluate the positioning accuracy of the proposed model.

3.2. Comparisons with State-of-the-Art Methods

In order to evaluate the robustness and effectiveness of the proposed method, several widely used semi-autonomous learning methods are taken for performance comparison. These semi-autonomous learning methods have made breakthrough achievements in the field of object detection, including CSC [46], WSDDN [22], OICR [28] and PCL [29]. CSC is a weak supervision method that designs a new change segmentation and classification module based on image-level semantic tags. WSDDN initiated the transformation of the weak supervision problem into an architecture of the implicitly learning object detector. OICR and PCL add new online instance classifier optimization branches on the basis of WSDDN. In addition, the results of the proposed method are compared with fully supervised object detection methods, such as Faster-RCNN [43], YOLOv4 [47], DCIFF-CNN [1], YOLOv4-CSP [48] and YOLOv5 (https://github.com/ultralytics/yolov5.git, accessed on 1 December 2021). Among them, Faster-RCNN and YOLOv4 are representative of CNN object detection methods, while DCIFF-CNN is a deep learning method for remote sensing images.

3.2.1. Detection Results on Augmented NWPU VHR-10 Dataset

Table 2. AP detection results of different methods for augmented NWPU-VHR 10 dataset (Bold indicates the best result).

			Fully Supervised Methods				Semi-Autonomous Learning Methods
	Methods	Faster-RCNN	YOLOv4	DCIFF-CNN	YOLOv4-CSP	YOLOv5	WSDDN	OICR	PCL	Ours
Objects
Airplane		89.41	94.60	89.65	97.61	96.81	12.90	22.54	30.28	38.03
Ship		76.73	67.58	74.82	67.01	79.87	50.62	70.00	70.00	71.25
Storage tank		57.20	83.78	77.62	94.15	92.31	31.43	58.00	56.00	82.00
Baseball diamond		88.74	94.30	89.84	92.92	95.76	90.91	94.07	94.92	96.19
Tennis court		64.39	74.09	79.66	86.23	92.43	5.96	1.23	2.45	22.09
Basketball court		69.37	57.93	89.87	90.09	89.50	39.20	29.49	16.10	33.05
Ground-track-field		89.67	91.42	90.41	92.63	95.00	98.05	99.57	99.11	99.13
Harbor		78.97	74.13	88.76	91.68	95.62	5.32	9.09	20.45	18.18
Bridge		72.75	56.42	78.99	67.50	80.67	1.21	3.61	3.35	8.64
Vehicle		57.80	69.06	76.14	82.42	85.01	9.30	19.23	20.41	61.54
mAP		74.50	76.33	83.58	86.22	90.30	34.49	40.68	41.31	53.01

shows the AP detection results of the different methods for the augmented NWPU-VHR 10 dataset. It can be observed that the proposed method achieves the highest mean average precision (mAP) value, and the mAP value obtained by WSDDN is slightly lower. Compared with other methods, WSDDN lacks a multi-instance refinement branch, while OICR and PCL combine the multi-instance learning and multi-stage instance classifier. The refinement branch makes the ability of the instance classifier iteratively enhanced and improves the detection performance. The difference is that in the process of refinement, ARMS combines the regions with the highest scores and the aggregation regions closely related to it, so as to obtain more complete object features, while OICR simply selects a series of regions with the highest scores as corresponding categories. For objects of storage tanks and vehicles that are small in size and crowded, the proposed method obtains outstanding performance. This is because ARMS can distinguish multiple instances in dense areas more effectively while extracting the complete features of the object. At the same time, the added positioning branches can locate the object more accurately and eliminate the redundant bounding box. For objects of the ground-track-field and bridge, WSDDN gets better AP values. However, this method is unstable in detecting small objects. In addition, the method in this paper further narrows the gap between the semi-autonomous learning method and the fully supervised learning method. The dataset in the semi-autonomous learning method is based on the image level label, while the data in the full supervision method are based on the object level label, and the results of the two methods are quite different. The particularity of the dataset brings many challenges and difficulties to the detection of semi-autonomous learning methods. We strive to make the detection accuracy under the semi-autonomous learning condition close to that under a fully supervised condition. In particular, for the object categories such as the ship, storage tank, baseball diamond, ground-track-field, vehicles, etc., the proposed method has achieved equivalent or even better results than the fully supervised method. Therefore, the results in Table 2 show that the proposed method is robust and effective in multi-task remote sensing image object detection.

Table 3. CorLoc detection results of different methods for augmented NWPU-VHR 10 dataset (Bold indicates the best result).

	Methods	OICR	WSDDN	PCL	Ours
Objects
Airplane		22.54	12.9	30.28	38.03
Ship		70.00	50.62	70	71.25
Storage tank		58.00	31.43	56	82
Baseball diamond		94.07	90.91	94.92	96.19
Tennis court		1.23	5.96	2.45	22.09
Basketball court		29.49	39.2	16.1	33.05
Ground-track-field		99.57	98.05	99.11	99.13
Harbor		9.09	5.32	20.45	18.18
Bridge		3.61	1.21	3.35	8.64
Vehicle		19.23	9.3	20.41	61.54
mCorLoc		40.68	34.49	41.31	53.01

shows the performance of the augmented NWPU VHR-10 dataset measured by CorLoc. For the fully supervised method, the training data all have specific positioning labels. Since CorLoc is the evaluation criterion for positioning accuracy, it is only applicable to the semi-autonomous learning method with image-level labels. Compared with WSDDN, OICR and PCL, the proposed method gets 19.12%, 12.33% and 11.70% improvement, respectively. For the object of harbor, PCL gets a better CorLoc value. However, for the object of tennis court and basketball court, which are similar in shape and size, PCL gets a poor result. This is because the background is complex, and for objects with high similarity, choosing only the proposal with the highest score as a positive example will lead to the loss of a large amount of object information. Because ARMS combines the area with the highest score and the gathering area closely related to it, it can mine even more complete positive instance information, which will be of great improvement to the positioning results.

The precision-recall curves (PRCs) of several semi-autonomous learning methods are displayed in Figure 5. In general, the precision will decrease with the increase of recall. Therefore, the performance of a model cannot be comprehensively measured by only the precision and recall corresponding to a certain point, and it should be measured by the overall performance of the PRCs. It can be observed that the proposed method can achieve better performance than other methods in most object categories. For the object of ground-track-field, all methods can achieve stable PRCs. This is because this object is large in size and has special features compared with other objects, which is of great help to object detection. For the object of tennis courts and basketball courts, which have similar characteristics and similar environments, the proposed method obviously has more outstanding precision and recall. It can not only accurately identify the object, but also obtain a high-precision detection rate. It is indicated that the proposed aggregation region instance refinement framework can effectively mine multi-task positive instances to accurately identify different objects. For the object of storage tank and vehicle, they are both object categories that are small in size. Moreover, there are a lot of instances in the image they appear, and the instances are relatively crowded. The proposed method can obtain more prominent PRCs for this kind of object. This benefits from the proposed ARMS, which utilizes the aggregation region to form a more complete object feature to solve the problem of missing object information in multiple instances. For the object of bridge, the proposed method obtains the suboptimal PRCs. This is due to the irregular shape of the bridge leading to some difficulties in object detection. However, the proposed method is still superior to most methods. Generally speaking, these results show that the proposed framework, based on aggregation region instance refinement, can mine richer positive instance features and generate more accurate object information by merging aggregation regions.

From the above results, it can be observed that the semi-autonomous learning method proposed in this paper can achieve better performance than other semi-autonomous learning methods in most object categories. WSDDN proposes an implicit learning object detector based on weak supervision, which can simultaneously select and classify regions at the level of image regions. However, compared with other methods, it is difficult and challenging to train classifiers because of the lack of multi-instance refinement branches. OICR proposes a novel online case classifier refinement algorithm, which integrates MIL and the instance classifier refinement procedure into a single depth network. However, OICR tends to select only the most confident candidate box to train the corresponding detector. However, the proposal with the highest score often covers only a part of the object rather than the whole object, which will affect the performance of the algorithm. PCL generates a proposed cluster, and learns the refined instance classifier through an iterative process. It spatially associates the proposals in the same cluster. This can prevent the network from paying too much attention to part of the object instead of the whole object. However, there are usually many similar examples in remote sensing images. Simply selecting the highest score and its related suggestions as pseudo GT may miss other similar examples, resulting in suboptimal object detectors. This paper proposes an end-to-end aggregation-based region merging strategy. By selecting other regions related to the proposal with the highest score and merging these related regions, a complete object region is formed, and a new confidence level is generated according to the merging form. In the iterative process, similar areas in the cluster are merged and redundant areas in the cluster are deleted to select high-quality areas. In the process of network refinement, this strategy shifts the focus of the detection network from part of the object to the whole object; at the same time, it reduces the influence of the background area on the network, and can generate its own cluster areas for many similar instances.

3.2.2. Detection Results on LEVIR Dataset

The LEVIR dataset contains three types of objects, namely, airplane, ship (including offshore and open sea) and oil port. Although it contains few kinds of objects, the data structure of this dataset is allocated reasonably. What is more, the image environment of the dataset is diverse and contains most types of ground features, such as a city, village, mountain and ocean.

Table 4. AP detection results of different methods for LEVIR dataset (Bold indicates the best result).

		Objects	Airplane	Ship	Oil Port	mAP
	Methods
Fully supervised methods	Faster-RCNN		77.60	70.21	65.34	71.10
	YOLOv4		79.81	73.34	70.40	74.52
	DCIFF-CNN		80.62	77.34	70.73	76.23
	YOLOv4-CSP		93.68	86.79	87.79	89.42
	YOLOv5		95.63	88.62	88.90	91.05
Semi-autonomous learning methods	CSC		0.81	0.13	10.76	3.90
	WSDDN		16.02	0.54	16.10	10.88
	OICR		22.06	0.41	33.93	18.80
	PCL		39.92	13.65	39.92	29.93
	Ours		52.41	14.57	40.06	35.68

shows the AP detection results of different methods for the LEVIR dataset. It can be observed that the proposed method is far superior to other semi-autonomous learning methods. Compared with other methods, the results obtained by CSC are slightly insufficient. This is because CSC does not add instance refinement branch and lacks effective supervision information in the subsequent classification. As CSC is also one of the typical semi-autonomous learning methods, it is compared with the semi-autonomous learning method proposed in this paper despite its poor results. As shown in the table, the proposed method outperforms other methods by about 6% in the mAP metric. Especially for the airplane class, it is about 12% higher than that of PCL. In addition, the proposed method further narrows the gap between the semi-autonomous learning method and the full supervision method. The above results prove that the proposed method is robust to various background images.

The comparisons of the CorLoc metric on the LEVIR dataset are reported in

Table 5. CorLoc detection results of different methods for LEVIR dataset (Bold indicates the best result).

	Objects	Airplane	Ship	Oil Port	mCorLoc
Methods
CSC		0.00	0.57	13.64	4.74
WSDDN		37.55	0.00	36.36	24.64
OICR		36.80	1.14	65.91	34.62
PCL		49.44	0.00	54.55	34.66
Ours		72.12	24.00	75.00	57.04

. It is clear that, on average, the proposed method outperforms the previous state-of-the-art methods by a large margin. The proposed method outperforms other methods by about 12% in the mCorLoc metric, especially for the airplane class. Because the dataset contains not only ships far from the coast but also ships close to the coast, with the coast being a complex existence for ships, the environment information of ships is extremely complex and changeable. As shown in Table 5, there is a big deviation between WSDDN and PCL in positioning ship objects. However, the proposed method can overcome these difficulties. This is because the proposed framework can obtain more complete object features in the process of mining more positive examples, which provides more accurate supervision information for the subsequent refinement process. This shows that the proposed method is robust in complex and changeable environments.

The PRCs of several semi-autonomous learning methods for the LEVIR dataset are displayed in Figure 6. It can be observed that the proposed method is superior to other methods. Especially for the objects that have complex background environments, such as the airplane, the proposed method obtains prominent and stable PRC. For the object of the ship, the PRC obtained by OICR is obviously a little worse; this is because the background of the ship is changeable, which include both the offshore scene and open sea scene. Even so, the proposed method still performs well and gets stable PRC. This is a benefit from the modified residual network proposed in this paper, which can extract highly informative features from the original image. Generally speaking, the proposed environment is robust in complex and changeable scenes.

3.3. Qualitative Results

Some detection results on the augmented NWPU VHR-10 are displayed in Figure 7. It can be observed that for most objects, the proposed method can obtain accurate and tight bounding boxes. Figure 7(b(1),b(5)) show the detection result of the harbor. For this crowded object with large-scale change, the proposed method can still obtain a more accurate positioning frame. However, it can be observed in Figure 7(b(1)) that there are still some shortcomings. The detection ability in the irregular scene of the crowded harbor still needs to be improved. The detection of the storage tank is displayed in Figure 7(a(5),b(3)); the object is small in size and crowded in image. It can be observed that for this small object, the proposed method has a certain rate of missing inspection. This is because the proportion of the storage tank is especially small, which causes some difficulties to the detection. Figure 7(c(1),c(2)) show the detection results of vehicle, whose environment is complex. It can be observed from the figures that there is a certain error detection rate. However, for the ship, ground-track-field and other objects with obvious features or simple scenes, the proposed method can obtain excellent detection performance.

Figure 8 shows the detection results of some methods for LEVIR. The first row denotes the PCL method, and the second row is the method proposed in this paper. It can be observed from the figure that the proposed method is superior to PCL. For the airplane, PCL has some error detection results. For the ship, especially those near the coast, the results obtained by the two methods are not good enough, but in comparison, the error detection rate of the proposed method is less. This is because the coastal scene has certain influence on object detection. For the oil port object, it is obvious that the proposed method has a less missed detection rate. Generally speaking, these show that the proposed method is robust to complex and changeable environments.

The runtime comparisons between the proposed method and other semi-autonomous learning methods are shown in

Table 6. Computation times for different methods in terms of LEVIR dataset.

Methods	CSC	WSDDN	OICR	PCL	Ours
Testing time (second/image)	0.55	1.64	1.69	2.12	2.14

, where the runtime of the proposal generation is not considered. It can be observed that with the improvement of the method, the algorithm takes more and more time. The method of CSC is efficient due to the lack of MIL. Although the proposed method is more competitive than other methods, the proposed method takes almost the same testing time as PCL. This is because the proposed approach is an end-to-end network. Although a small number of additional test computations are required compared to WSDDN and OICR, the proposed method obtains much better detection results than other methods.

4. Discussion

To evaluate the effectiveness of the proposed approach, ablation experiments are constructed to analyze the effects of the key components of ARMS, such as the modified residual network, ARMS and regression branches.

4.1. MRN

In order to verify the effectiveness of the improved residual network, experiments were carried out on different backbone networks, including VGG-16, RESNET-18 and the MRN, as shown in Figure 9. In this experiment, except for the different backbone networks, the other settings are the same. It can be observed that the performance of mAP decreased from 34.78% to 33.90% by directly replacing VGG-16 with resnet-18, especially for the objects with high similarity and indistinguishability, such as the tennis court and basketball court. For a vehicle with a small size and complex scene, the AP value drops from 23.23% to 7.02%. Therefore, it may be counterproductive to simply and rudely replace VGG-16 with resnet-18. On the contrary, it can be observed that the mAP value steadily increased from 34.78% to 38.72% after replacing VGG-16 with the modified residual network. For the small-sized object such as the harbor and vehicle, the AP value can be kept stable or even increased, which further confirms that the proposed method is effective and stable in the multi-task object detection under semi-autonomous learning.

4.2. ARMS

In this section, the proposed ARMS is compared to a training strategy similar to OICR, which only selected the candidates with the highest credibility to refine the next classifier. In this experiment, the way of instance refinement is different and other settings are the same. As shown in Figure 10, it can be observed that ARMS greatly improves the detection performance of the network, and the value of mAP increases from 31.9% to 38.72%. Interestingly, the proposed ARMS is more effective for small objects with particularly complex backgrounds and object classes with multiple instances in the image, such as the vehicle and airplane. Moreover, it can clearly distinguish the similar objects, such as tennis courts and basketball courts. Therefore, the proposed ARMS can mine more positive examples and classify more accurately. The main reason is that in the refinement process, OICR only chooses the suggestions with the highest scores as the supervision information in the next stage, which makes it easy to confuse the background with the positive instance, as well as with the different types of positive instances, while ignoring other similar goals in the scene. However, ARMS makes use of the suggestions with the highest scores and the aggregation areas closely related to them, which can fully mine the correct examples and classify them accurately.

4.3. Regression Branches

Figure 11 shows the contribution of the regression branch to the whole framework. In this experiment, the way of regression is different and other settings are the same. It can be observed from the table that the performance is obviously improved after the regression branch is added, and the mAP is improved from 34.7% to 38.72%. Especially for small objects with multiple instances and close distance, such as oil tanks and airplanes, the positioning branch can accurately find its exact position. This is due to the combination of the regression branch and classification branch, which can effectively solve the over-fitting problem caused by the weak detector when distinguishing parts, and at the same time, it can locate the object more accurately.

5. Conclusions

A novel end-to-end aggregate-guided semi-autonomous learning residual network is presented in this paper to handle the object detection problem in remote sensing images. Specially, ARMS is designed to select high-quality regions by merging similar regions in the cluster and deleting redundant regions in the cluster. Moreover, a progressive residual network is applied to the backbone network to make the detector more sensitive to small objects. Meanwhile, a regression locating branch is further developed to refine the location of the object, which can be optimized jointly with regional classification. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art object detection of the remote sensing image. For the NWPU dataset with a large number of object categories, the semi-autonomous learning methods WSDDN, OICR and PCL achieved 34.49%, 40.68% and 41.31% AP values, respectively, while the semi-autonomous learning method proposed in this paper achieved a 53.01% AP value. This indicates that the proposed method is obviously superior to other methods. For the LEVIR dataset with a relatively balanced number of objects, the semi-autonomous learning methods WSDDN, OICR and PCL have achieved 10.88%, 18.80% and 29.93% AP values, respectively, while the semi-autonomous learning method proposed in this paper has achieved a 35.68% AP value. It can be proved that the proposed method outperforms the state-of-the-art methods.