Accurate Instance Segmentation for Remote Sensing Images via Adaptive and Dynamic Feature Learning

Abstract

Instance segmentation for high-resolution remote sensing images (HRSIs) is a fundamental yet challenging task in earth observation, which aims at achieving instance-level location and pixel-level classification for instances of interest on the earth’s surface. The main difficulties come from the huge scale variation, arbitrary instance shapes, and numerous densely packed small objects in HRSIs. In this paper, we design an end-to-end multi-category instance segmentation network for HRSIs, where three new modules based on adaptive and dynamic feature learning are proposed to address the above issues. The cross-scale adaptive fusion (CSAF) module introduces a novel multi-scale feature fusion mechanism to enhance the capability of the model to detect and segment objects with noticeable size variation. To predict precise masks for the complex boundaries of remote sensing instances, we embed a context attention upsampling (CAU) kernel instead of deconvolution in the segmentation branch to aggregate contextual information for refined upsampling. Furthermore, we extend the general fixed positive and negative sample judgment threshold strategy into a dynamic sample selection (DSS) module to select more suitable positive and negative samples flexibly for densely packed instances. These three modules enable a better feature learning of the instance segmentation network. Extensive experiments are conducted on the iSAID and NWU VHR-10 instance segmentation datasets to validate the proposed method. Attributing to the three proposed modules, we have achieved 1.9% and 2.9% segmentation performance improvements on these two datasets compared with the baseline method and achieved the state-of-the-art performance.

1. Introduction

With the development of earth observation technologies, a large number of high-resolution remote sensing images (HRSIs) are available, raising the high demands of automatic image analysis by intelligent means [1,2,3,4,5,6,7,8,9,10]. The ways to interpret remote sensing images, e.g., object detection, instance segmentation, etc., have recently attracted much attention. Instance segmentation for HRSIs aims at accurately detecting the location of the objects of interest (e.g., vehicles, ships, etc.) in the image, predicting their categories and classifying each instance at the pixel level. It is conducive to the automatic analysis and utilization of the spatial and spectral information of HRSIs, which has a wide range of applications in earth observation, e.g., disaster estimation, urban planning, traffic management, etc.

Recently instance segmentation has been extensively studied based on deep learning both for natural images and aerial images [1,3,4,5,9,11,12,13,14,15,16,17,18,19,20,21,22,23,24]. The generic instance segmentation methods for natural images can be roughly divided into detection-based and segmentation-based approaches from the point of view of top-down and bottom-up schemes, respectively. The detection-based methods first find out the bounding boxes of the instances of interest through an advanced object detector such as Fast-RCNN [25], Faster-RCNN [26], and R-FCN [27]. Then they perform semantic segmentation in the detection boxes to segment the mask for each instance. Early work [28,29,30,31,32] adopted mask proposals to perform category-agnostic object segments. FCIS [33] uses the position-sensitive inside/outside score maps for instance-aware semantic segmentation with the help of fully convolutional network (FCN) [34]. Mask-RCNN [11] inherits the classification and regression structure from Faster-RCNN [26] and adds a fully convolutional mask branch to predict the category of each pixel. Inspired by Mask-RCNN, [12,13,14,15] introduce different designs and structures to refine mask prediction for two-stage instance segmentation. HTC [16] integrates the cascade method into joint multi-stage processing and uses spatial context to further improve the accuracy of segmented masks. YOLACT [35] adds the mask branch to an existing one-stage target detection model, which realizes real-time instance segmentation with competitive results reported by two-stage methods on the MS COCO dataset [36] for the first time. BlendMask [17] combines the instance-level information (such as bounding boxes) and the per-pixel semantic information for mask prediction.

The segmentation-based methods first perform the pixel-level prediction on the image and then use a clustering algorithm to group the pixels into different instances. Bai et al. [37] and Hsu et al. [38] applied the traditional watershed transform and the pairwise relationship between pixels to perform pixel-wise clustering. Liu et al. [39] employed a sequence of neural networks to handle the problems of occlusion and large differences in the number of objects in instance segmentation. Neven et al. [40] proposed a new loss function to achieve real-time instance segmentation with high accuracy. PolarMask [41] models the instance contours in a polar coordinate and transforms the instance segmentation into center point classification and dense distance regression tasks. While [42,43] adopt the location and size information to assign category labels for each pixel within an object, which converts the instance segmentation into a pixel-wise classification problem.

Benefiting from the vigorous success in natural images, the research of instance segmentation for HRSIs has made great progress in recent years. However, Limited by the lack of aerial instance segmentation datasets, previous instance segmentation methods [1,3,4,5,9] mainly focused on some or other specific categories, such as ship, vehicle, and building. Since Su et al. [19,20] extended the NWPU VHR-10 object detection dataset [44,45,46] for instance segmentation and the release of the first large-scale instance segmentation dataset named iSAID [47] in the field of aerial imagery, researchers began to turn their attention to multi-category instance segmentation for HRSIs. Su et al. [19] designed the precise RoI pooling incorporated into Mask-RCNN to improve the performance of object detection and instance segmentation in aerial images. HQ-ISNet [20] introduces the HRFPN to reserve high-resolution features of HRSIs and utilizes the ISNetV2 to predict more accurate masks. Ran et al. [21] proposed an adaptive fusion and mask refinement (AFMR) network to learn complementary spatial features and conduct the content-aware mask refinement. CPISNet [23] presented a novel adaptive feature extraction network and a proposal consistent cascaded architecture to boost the integral network performance. Zhang et al. [22] proposed a Semantic Attention (SEA) module to alleviate the background interference, and they modified the original mask branch to ameliorate the under-segmentation phenomena.

Remote sensing images are usually taken from a high altitude with the bird’s-eye view depending on the perspective of the earth observation platforms. The particular imaging platforms and various kinds of objects lead to some distinctive characteristics of instances presented in HRSIs, such as large scale variation, complex contours, and dense block distribution, as shown in Figure 1. These three issues are the main difficulties complicating the instance segmentation task of HRSIs. This paper specifically focuses on the problems of instance segmentation in HRSIs, which is challenging mainly due to the following three aspects:

Huge scale variation. The intra-class and inter-class scale variation among objects in remote sensing images are much more significant than those in natural images. It will increase the difficulty of feature learning in deep neural networks to a great extent. A general solution to this problem is adopting multi-scale feature extraction and fusion, which has been used in [11,14,15,48,49,50,51,52]. However, these multi-scale feature pyramid based methods such as feature pyramid network (FPN) usually assume that large objects should be represented by high-level features, while small objects should be associated with shallow features. The inconsistency across multi-scale feature maps limits their capability both in object detection and instance segmentation when the scale of objects changes too sharply. Liu et al. [13] proposed a bottom-up path aggregation strategy by using accurate low-level locating information to enhance the feature representation entirely. However, this fixed fusion mechanism does not fully explore the self-learning merit of neural networks. Therefore, it is important to design adaptive feature fusion methods for the combination of useful information at different levels.

Objects with arbitrary shapes. There are numerous objects with arbitrary shapes in HRSIs, such as ships, harbors, helicopters. Since instance segmentation requires to achieve pixel-level accurate classification, these irregular shapes will bring significant challenges to the segmentation task. Zhao et al. [4] attempted to use building boundary regularization to deal with the complex boundary segmentation of buildings. Cheng et al. [5] constructed an energy map in polar coordinates, and then minimized the energy function to evolve the polygon outlines, thereby achieving accurate automatic building segmentation with arbitrary shapes. Limited by the dataset, [4,5] are only for single category segmentation. As for multi-class segmentation, Huang et al. [14] improved their results by using the mask-IoU score to correct the misalignment between the mask quality and mask score. Kirillov et al. [15] converted the instance segmentation into a rendering problem and obtained more nuanced mask predictions on the object contours through an iterative subdivision algorithm. However, current methods ignore the impact of feature upsampling operators when predicting the precise masks. The most widely used bilinear and nearest neighbor interpolation only take the sub-pixel neighborhood into consideration, and the deconvolution operation utilizes the same upsampling kernel across the entire image. These immutable upsampling operators fail to take full advantage of the contextual semantic information required by complex boundary segmentation.

Densely packed small objects. Cities and ports commonly contain considerable densely distributed small objects in aerial images. Since the size of these small objects is only a few to a dozen pixels and they always distribute in blocks, the close arrangements will lead to mutual interference among themselves whether in detection or segmentation. Mou et al. [1] proposed a semantic boundary-aware unified multitask learning ResFCN for vehicle instance segmentation. Feng et al. [3] improved the detection and segmentation performance of small dense objects via implementing multi-level information fusion on the mask prediction branch. Another solution is to improve the quality of the candidate proposals which are important for the subsequent detection and segmentation. Most detection-based instance segmentation methods adopt a hard threshold to divide positive and negative samples. However, simply using a constant IoU threshold as the criterion to separate training samples is not sufficient. As illustrated in [53,54,55], the dynamic sample selection strategy can solve the incompatibility between the hard threshold settings and the dynamic training process. For that reason, the network can generate candidate boxes of higher quality to facilitate subsequent localization and pixel-wise classification.

In this paper, we design an end-to-end multi-category instance segmentation network for HRSIs, where three key modules based on adaptive and dynamic feature learning are proposed to address the above three problems. Firstly, a cross-scale adaptive fusion (CSAF) module is proposed to integrate the multi-scale information, which can enhance the capability of the model to detect and segment objects with obvious size variation. Secondly, to deal with the complex boundaries of remote sensing instances, we embed a context attention upsampling (CAU) module in the segmentation branch to get more delicate masks. Thirdly, we devise a Dynamic Sample Selection (DSS) module to select more appropriate candidate boxes, especially for densely packed small objects. We evaluate the proposed method on two public remote sensing datasets.

The main contributions of this paper can be summarized as:

• We propose the CSAF module with a novel multi-scale information fusion mechanism that can learn the fusion weights adaptively according to different input feature maps from the FPN.
• We extend the original deconvolution layer into the CAU module in the segmentation branch which can obtain more refined predicted masks by generating different upsampling kernels with contextual information.
• Instead of the traditional fixed threshold for determining positive and negative samples, the DSS module employs a dynamic threshold calculation algorithm to select more representative positive/negative samples.
• The proposed method has achieved state-of-the-art performance on two challenging public datasets for instance segmentation task in HRSIs.

The rest of this paper is organized as follows. Section 2 describes the proposed method in detail. The experimental results and analysis are reported in Section 3. Section 4 gives some discussions about the proposed method. Finally, we draw a conclusion in Section 5.

2. Methodology

For instance, segmentation, we take PANet [13] as a strong baseline for the outstanding performance in aerial images by following the benchmark paper [47]. The overall framework of the proposed method is depicted in Figure 2. The Resnet-FPN backbone is used to extract multi-scale feature maps from an input image. Then we adopt the cross-scale adaptive fusion (CSAF) module to automatically fuse semantic information across multi-level feature maps. The RPN-based dynamic sample selection (DSS) module is employed for candidate proposals generation. We extend the general fixed positive and negative sample judgment threshold in the region proposal network (RPN) into a dynamic threshold selection strategy to choose high-quality training samples. Finally, the proposals are sent to the bounding box branch for object detection and to the mask branch for instance segmentation where a context attention upsampling (CAU) module is embedded for upsampling. In this section, we describe the three proposed modules in detail.

2.1. Cross-Scale Adaptive Fusion

Most state-of-the-art methods attempt to cope with objects across a wide range of scales via constructing a multi-level feature pyramid [11,13,15,49,56]. The commonly used fusion method is manually setting the fixed fusion weights through a sum or concatenation operator. Different from this way, we design a cross-scale adaptive fusion (CSAF) module to enable the network to learn the fusion weights across feature layers of different levels autonomously. Inspired by [57], we propose a fully connected fusion mechanism between the top-down and bottom-up paths in the PANet. It can adaptively search the optimal fusion operation by adjusting the spatial fusion weights according to the specific pyramidal feature maps. The structure of the cross-scale adaptive fusion module is shown in Figure 3.

Given an input image, the ResNet-FPN backbone extracts the multi-level features and that are sent to the cross-scale adaptive fusion module. We denote the feature map at level $l\left(l\in \{2,3,4,5\}\mathrm{in}\mathrm{FPN}\right)$ as $P^l$. Since the size of these feature maps from FPN is inconsistent, for each level l, we need firstly resize the features maps $P^n$ at all other levels n to the same size as $P^l$. During the process of resizing, we use the nearest neighbor interpolation to enlarge the feature maps. However, the downsampling method can be a bit more complicated. When the resizing scale is 1/2, we employ a 2-stride max pooling operation directly; for 1/4, we adopt the combination of a max pooling layer and a convolution layer whose kernel size is $3\times 3$ with the stride of 2; and when the scale is 1/8, two consecutive max pooling layers and a two-stride $3\times 3$ convolution are used. After the rescaling step, we further depict the feature fusion mechanism in Figure 4.

To formulate our method, we denote $P^l$ as the target feature for rescaling and fusion and the other features $P^n$ at level n as the source feature maps. For the feature maps rescaling from source level n to target level l, the feature vector at location $(i,j)$ is $P_{ij}^{n\to l}$. The core of this mechanism is that the network can learn the appropriate fusion weights according to the input multi-scale feature maps adaptively. We firstly use an $1\times 1$ convolution layer to compute the weight scalar maps ${\lambda }^{n,l}$ from $P^{2\to l}, P^{3\to l}, P^{4\to l}$ and $P^{5\to l}$, which indicates that $\lambda$ can be learned through standard back-propagation. Then a softmax function is resorted to normalize these weight maps to obtain the fusion scores. We define the final fusion scores $\alpha$ as follows:

$${\alpha }_{ij}^{n\to l}=\frac{e^{{\lambda }_{ij}^{n\to l}}}{{\sum }_{k=2}^5{e}^{{\lambda }_{ij}^{k\to l}}},$$

(1)

where ${\alpha }_{ij}^{n\to l}$ are the outputs of the softmax function with ${\lambda }_{ij}^{n\to l}$ as control parameters, respectively. Notice that ${\alpha }_{ij}^{n\to l}$ are shared by all channels at location $(i,j)$ and they are simple scalar variables and ${\sum }_{n=2}^5{\alpha }_{ij}^{n\to l}=1$. Finally, the fused feature map at target level l can be computed as:

$${\widehat{P}}_{ij}^l=\sum _{n=2}^5{\alpha }_{ij}^{n,l}·P_{ij}^{n\to l},$$

(2)

where ${\widehat{P}}_{ij}^l$ implies the $(i,j)$-th vector in the output feature maps ${\widehat{P}}^l$ across channels. ${\alpha }_{ij}^{n,l}$ refers to the fusion weights for the feature maps between the source level n and the target level l.

Through the proposed learning mechanism, we can get the output fused feature maps $\left\{{\widehat{P}}_2,{\widehat{P}}_3,{\widehat{P}}_4,{\widehat{P}}_5\right\}$. We denote the final output feature maps as $\left\{O_2,O_3,O_4,O_5\right\}$, which will be sent to the following RPN for proposal generation. Each feature map $O_i(i\in \{3,4,5\})$ is obtained from the fused feature ${\widehat{P}}_i$ and a higher resolution feature map $O_{i-1}$ through lateral connection and a $0.5\times$ downsampling. The fusion of these two feature maps is a simple addition operation, which is consistent with the bottom-up path in the original PANet. Additionally, $O_2$ is simply ${\widehat{P}}_2$ without any operation. Through the bottom-up pathway, we integrate the cross-scale fused features into the information flow of PANet. Thus, the semantic information from high-level feature maps is cross linked to the small objects, while the location information from shallow feature pyramids can also be used for the detection and segmentation of large instances.

2.2. Context Attention Upsampling

HRSIs contain many instances with complex boundaries which makes it difficult to achieve pixel-level classification and limits the final segmentation performance. It is mainly because that the network has lost the detailed information of the object boundaries after multiple downsampling operations. However, existing conventional upsampling methods cannot completely recover these lost details. In PANet, the output of mask predictions is obtained from $14\times 14$ RoI features and a deconvolution layer is employed to upsample them by $2\times$. Yet this deconvolution layer uses the fixed upsampling kernel across the whole image, paying no attention to the semantic characteristic of specific proposals. In this paper, we replace the deconvolution operation with a context attention upsampling (CAU), where the upsampling kernel can be adaptively generated according to different contextual information.

The pipeline of proposed CAU module is dipicted in Figure 5. We denote the RoI feature map $\Psi$ from RoIAlign operation as the source feature and the upsampled feature map ${\Psi }^{\prime }$ as its corresponding target feature in CAU. The width and height of $\Psi$ are W and H, and we denote the number of its channel as C. With an upsampling scale $\eta$, the size of the RoI feature map is upsampled from $W\times H\times C$ to $\eta W\times \eta H\times C$. Firstly, we use an $1\times 1$ convolution layer to compress the channel of $\Psi$, whose output feature shape is referred to $W\times H\times C_f$. The fewer number of channels implies fewer parameters and less computational cost during upsampling. Compressing the channel from C to $C_f$ will not lead to performance decline while being more efficient. It can be inferred that each position $p=(i,j)$ on $\Psi$ corresponds to a size of $\eta \times \eta$ target region $p^{\prime }=(\eta \times i,\eta \times j)$ on ${\Psi }^{\prime }$. Instead of using the fixed sampling kernel, a context aware kernel is designed for feature reassembly, whose size is denoted as $k_{up}\times k_{up}$. Since the upsampling kernel is dynamically generated by the contextual semantic feature map of each location, the shape of the final predicted kernel map for the whole feature map can be written as $\eta W\times \eta H\times {k_{up}}^2$. We adopt a $3\times 3\times C_{up}(C_{up}=\eta \times \eta \times {k_{up}}^2)$ convolution layer to encode the compressed feature map and a shuffle operation to generate such upsampling kernel on-the-fly. Afterward, we utilize a softmax function for spatial normalization on the rearranged kernel feature map. The reason for normalization is to ensure the sum of kernel values equal to 1, which will not change the mean values of the feature map. Finally, we integrate the input RoI feature and the upsampling kernel, thus the target feature ${\Psi }_{p^{\prime }}^{\prime }$ on position $p^{\prime }$ can be denoted as

$${\Psi }_{p^{\prime }}^{\prime }=\sum _{n=-r}^r\sum _{m=-r}^r{\omega }_{p^{\prime }(n,m)}·{\Psi }_{(i+n,j+m)},$$

(3)

where $r=k_{up}/2$ and ${\omega }_{p^{\prime }(n,m)}$ is the upsampling kernel generated by the above steps. $(i,j)$ denote the original position on RoI feature $\Psi$.

With the adaptive sampling kernel, the RoI feature map is upsampled in a context attention manner and retains much more instance-specific semantic information [58]. As shown in Section 3.5, our mask prediction is much more accurate, especially for instances with complex boundaries.

2.3. Dynamic Sample Selection

How to separate positive and negative samples accurately is a noteworthy problem in both of object detection and instance segmentation tasks. The most widely used strategy is to set a hard threshold for the IoU between the anchor bounding box and that of the corresponding one of ground truth. When the IoU is higher than the threshold, the anchor box will be assigned a positive label; otherwise, it will be assigned a negative label. Taking a binary classification loss for simplicity, the paradigm can be formulated as follows:

$$label=\left\{\begin{array}{cc}1,\hfill & ifmaxIoU(a,G)\ge t_{+},\hfill \\ 0,\hfill & ifmaxIoU(a,G)<t_{-},\hfill \\ -1,\hfill & otherwise.\hfill \end{array}\right.$$

(4)

Here a stands for a anchor bounding box, G represents the set of ground-truths, $t_{+}$ and $t_{-}$ are the positive and negative threshold for IoU. Labels of $1,0,-1$ stand for positives, negatives and ignored samples, respectively. As for RPN in Faster R-CNN, $t_{+}$ and $t_{-}$ are set to 0.5 by default. So the definition of positives and negatives is essentially pre-set.

In HRSIs, many instances like vehicles and ships present a densely packed distribution. As shown in Figure 6, a region proposal may contain several instances with similar appearances. These adjacent instances can be background interfering noises when extracting accurate features for a certain instance, which increases the difficulty of selecting positive and negative samples. Using manually set and fixed sample judging thresholds adopted by most classic methods is not suitable for remote sensing images because there are a large number of densely packed instances with complex and irregular boundaries. Thereby, we propose a dynamic sample selection (DSS) module with a novel positive and negative samples selection strategy and a constrained $IoU$ calculation method. It works to assign positive and negative samples dynamically, improve the quality of positive samples, reduce the interference between adjacent instances, and alleviate the pressure of coordinate regression. The DSS finally enhances the performance of both object detection and instance segmentation.

2.3.1. The DSS Strategy

The proposed dynamic scheme is applied in the process of generating positive candidate samples, replacing the original simple IoU threshold filter strategy. Specifically, given an input image, we denote the set of ground-truth boxes as G and the number of feature maps as l. The sets of all anchor boxes and anchor boxes on the $i_{th}$$\left(i\in \{1,2,\cdots ,l\}\right)$ feature level are written as A and $A_i$, respectively. For each ground-truth box $g\in G$, we first compute the distances between the center of each anchor box in $A_i$ and the center of g from feature pyramid level 1 to level l. The top-k closest anchor boxes are labeled as the set of candidate positive samples $C_i$ belonging to g on level i. Then each ground-truth box will have $k\times l$ candidate positive samples in total. Secondly, we compute the IoU between the candidates set and their corresponding ground-truth box g. The IoU set is denoted as $D_g$ whose mean and standard deviation are $m_g$ and $v_g$. After that, a dynamic IoU threshold for g is computed as $t_g=m_g+v_g$. Finally, according to the label assignment rule of object detection in Equation (4), the proposed adaptive training sample selection can be formulated as follows:

$$label=\left\{\begin{array}{cc}1,\hfill & ifIoU(c,g)\ge t_g,\hfill \\ 0,\hfill & ifIoU(c,g)<t_g.\hfill \end{array}\right.$$

(5)

where g represents the ground-truth box, c stands for a candidate positive sample of $C_i$ whose center is in g, and $t_g$ is the dynamic threshold for IoU. 1 and 0 stand for the positive and negative samples. Additionally, if an anchor box is assigned to multiple ground-truth boxes, the one with the greatest IoU will be selected.

2.3.2. Constrained IoU Calculation

The common IoU for comparing similarity between two boxes is computed by:

$$IoU(c,g)=\frac{\left|c\cap g\right|}{\left|c\cup g\right|}.$$

(6)

In order to obtain high-quality positive samples, we add a penalty item on the basis of IoU. It is defined as:

$$Io{U}_p(c,g)=IoU(c,g)-\Delta ,$$

(7)

$$\Delta =\frac{\left|R\backslash (c\cup g)\right|}{\left|R\right|}.$$

(8)

Here, c denotes a candidate positive sample, g is the corresponding ground-truth box, R represents the smallest rectangle enclosing them, and $\Delta$ stands for the proposed penalty item.

The calculation process of $\Delta$ is illustrated in Figure 7. We first find the minimum enclosing rectangle R of the candidate positive sample c and its ground-truth box g. Then we calculate the area occupied by R excluding c and g, which next divide by the total area of R. This represents a normalized measure that focuses on the empty area in the minimum enclosing rectangle of c and g. Finally, the constrained IoU is attained by subtracting this ratio from the IoU value. The value range of $Io{U}_p$ is within $(-1, 1]$. When the candidate sample coincides with its ground-truth box, $Io{U}_p$ equals to $IoU$. When the two are far apart, $Io{U}_p$ approaches $-1$. $Io{U}_p$ holding the property of $IoU$ such as insensitivity for scale variety, and can reflect the relative position between the candidate positive sample and ground-truth box. As shown in Figure 8, the relative positions of the candidate positive sample and the ground-truth box in Figure 8a,b are different while their $IoU$ are the same. We are not able to judge which candidate sample is better according to $IoU$. However, if we use the $Io{U}_p$ as the criterion, the candidate box in (a) is superior than that of (b). The coordinate regression difficulty of the candidate positive sample in Figure 8a is lower than that in Figure 8b, and we only need to regress the horizontal axis in (a). From this perspective, the proposed constrained $Io{U}_p$ is able to distinguish the two candidates and more suitable than the original $IoU$.

3. Experiments

3.1. Datasets Description

We evaluate the proposed method on two public remote sensing datasets: the iSAID [47] and the NWPU VHR-10 instance segmentation dataset [19,20].

iSAID: The iSAID [47] dataset released in 2019 is the first benchmark dataset for the multi-category instance segmentation in aerial imagery that contains object-level and pixel-level annotations. It consists of 2806 remote sensing images with size ranging from 800 to 13,000 in width. There are totally 655,451 instances annotated in 15 categories including plane (PL), ship, storage tank (ST), baseball diamond (BD), tennis court (TC), basketball court (BC), ground track field (GTF), harbor (HB), bridge (BR), large vehicle (LV), small vehicle (SV), helicopter (HC), roundabout (RA), swimming pool (SP) and soccer ball field (SBF). For dataset splitting, half of the images are used for training, $1/6$ images for validating, and $1/3$ for testing. When the input image size is too large, the computational complexity of the network will rise sharply, which will bring a great burden to the computing device. Therefore, following the iSAID benchmark paper [47], we crop the original images into $800\times 800$ patches with the stride of 200 pixels. After the data preprocessing, the number of images for the training set, validation set and test set are 28,029, 9512, and 19,377, respectively. We evaluate our model both on the iSAID validation set and its test set. Since the ground-truth annotations of the test set are not available, the results are tested on the official evaluation server [59]

NWPU VHR-10 Dataset: The NWPU VHR-10 dataset [44,45,46] is a geospatial object detection dataset and was extended by Su et al. [19,20] for instance segmentation. It contains 650 positive images (the image contains at least one object) and 150 negative images (the image does not contain any object) with the sizes ranging from $533\times 597$ to $1728\times 1028$ pixels. In the positive image set, 757 airplanes (PL), 302 ships (SH), 655 storage tanks (ST), 390 baseball diamonds (BD), 524 tennis courts (TC), 159 basketball courts (BC), 163 ground track fields (GTF), 224 harbors (HB), 124 bridges (BR), and 477 vehicles (VC) were manually annotated with bounding boxes and instance masks for ground truths. In consistence with previous studies [19,20,22,23], we randomly divide the original positive images into two parts: $70\%$ for the training set and $30\%$ for the test set which include 454 images and 196 images, respectively.

3.2. Evaluation Metrics

We use the standard COCO metrics [36] for evaluation: $AP$ (averaged over IoU threshold from 0.5 to 0.95 with stride of 0.05), $A{P}_{50}$ (IoU threshold is 0.5), $A{P}_{75}$ (IoU threshold is 0.75), $A{P}_S$, $A{P}_M$ and $A{P}_L$, where S, M and L represent small $(0<\mathrm{area}<{32}^2\mathrm{pixels})$, medium $({32}^2<\mathrm{area}<{96}^2\mathrm{pixels})$ and large objects $(\mathrm{area}>{96}^2\mathrm{pixels})$, respectively. Considering the significant scale difference between HRSIs and natural images [22,23,47], we set the area range of different instances for the iSAID dataset by following its benchmark paper [47]. Specifically, the area of small instances ranges from ${10}^2$ to ${144}^2$, medium instances range from ${144}^2$ to ${512}^2$ pixels and large instances range from ${512}^2$ and over. In addition, we use $A{P}^{seg}$ and $A{P}^{bb}$ to represent the experimental results of mean average precision of instance segmentation and object detection in Section 3.4.

3.3. Implementation Details

For fair comparisons, all experiments are implemented on PyTorch framework and the hyper-parameters are basically the same as the settings in PANet [13,47]. We use ResNet-FPN as the backbone network and RPN to generate candidate object proposals. Our models are trained on 2 NVIDIA Tesla V100 GPUs with the batch size of 4 for 180 k iterations. The initial learning rate is set as 0.025 and decreased by a factor of 10 at 120 k and 160 k iteration. Besides, we adopt the SGD for training optimization using the weight decay of 0.0001 and the momentum of 0.9. As large scale variation exists among aerial images, we use scale augmentation for the training set at five scales (1200, 1000, 800, 600, 400). For the testing phase, the test images are resized to 1000 pixels on the long side. Considering the number of objects in HRSIs is much larger than in natural images, during evaluation, we set the number of detection boxes as 1000 instead of 100 by default. The source code is available on the website [60].

3.4. Quantitative Results

3.4.1. Results on iSAID

As shown in

Table 1. Overall results of instance segmentation and object detection on iSAID validation set.

Method	${\mathbf{AP}}^{\mathbf{seg}}$	${\mathbf{AP}}_{50}^{\mathbf{seg}}$	${\mathbf{AP}}_{75}^{\mathbf{seg}}$	${\mathbf{AP}}_{\mathbf{S}}^{\mathbf{seg}}$	${\mathbf{AP}}_{\mathbf{M}}^{\mathbf{seg}}$	${\mathbf{AP}}_{\mathbf{L}}^{\mathbf{seg}}$	${\mathbf{AP}}^{\mathbf{bb}}$	${\mathbf{AP}}_{50}^{\mathbf{bb}}$	${\mathbf{AP}}_{75}^{\mathbf{bb}}$	${\mathbf{AP}}_{\mathbf{S}}^{\mathbf{bb}}$	${\mathbf{AP}}_{\mathbf{M}}^{\mathbf{bb}}$	${\mathbf{AP}}_{\mathbf{L}}^{\mathbf{bb}}$
Mask R-CNN [11]	35.4	57.7	37.8	37.4	50.0	53.0	40.4	62.3	44.3	42.6	48.4	62.4
PANet [13]	38.2	62.4	41.0	40.4	51.9	55.5	43.2	66.0	47.6	45.5	49.4	58.4
BlendMask [17]	36.8	60.5	38.2	20.8	45.5	53.1	43.6	64.1	47.6	28.4	50.1	55.2
PointRend [15]	38.5	62.2	40.8	39.9	56.0	53.5	43.6	65.2	47.9	44.9	58.0	48.8
ours	40.1	64.6	43.0	42.0	56.6	77.3	46.2	68.8	51.3	48.3	58.3	80.1

,

Table 2. Class-wise results of instance segmentation on iSAID validation set.

Method	$\mathbf{AP}$	SV	LV	PL	ST	Ship	SP	HB	TC	GTF	SBF	BD	BR	BC	RA	HC
Mask R-CNN	35.4	15.5	37.3	51.5	38.5	44.3	33.5	28.3	76.8	26.0	38.7	49.8	20.3	34.2	31.7	4.8
PANet	38.2	18.0	41.0	54.0	39.6	54.4	35.4	30.3	78.1	29.9	39.3	51.7	20.7	38.5	35.5	7.0
BlendMask	36.8	14.7	37.8	54.7	40.3	41.0	34.2	30.4	78.4	23.6	41.4	51.9	20.9	38.5	35.6	8.1
PointRend	38.5	16.4	40.7	54.4	37.4	49.2	32.6	31.3	79.2	35.3	43.0	52.7	22.1	39.4	35.6	8.3
ours	40.1	20.0	41.9	54.1	41.7	56.6	36.0	31.4	78.0	31.5	45.4	55.2	22.4	43.9	35.2	8.1

,

Table 3. Class-wise results of object detection on iSAID validation set.

Method	$\mathbf{AP}$	SV	LV	PL	ST	Ship	SP	HB	TC	GTF	SBF	BD	BR	BC	RA	HC
Mask R-CNN	40.4	19.7	42.8	69.3	38.7	46.3	37.7	47.3	76.0	40.4	37.4	49.1	24.5	32.6	30.5	14.3
PANet	43.2	22.4	45.8	70.6	39.5	57.0	40.2	50.1	77.6	41.8	37.1	51.4	24.6	36.6	35.3	18.5
BlendMask	43.6	19.9	44.7	74.3	42.5	47.7	39.5	49.4	80.0	34.9	43.6	52.6	26.0	39.1	37.6	21.3
PointRend	43.6	20.3	45.0	70.7	37.0	52.8	36.1	50.9	79.9	45.8	42.7	53.3	25.9	39.1	36.3	17.8
ours	46.2	25.8	47.5	71.0	42.3	60.6	41.6	53.6	78.3	44.2	45.0	55.8	27.8	42.9	34.8	21.5

,

Table 4. Overall results of instance segmentation and object detection on iSAID test set.

Method	${\mathbf{AP}}^{\mathbf{seg}}$	${\mathbf{AP}}_{50}^{\mathbf{seg}}$	${\mathbf{AP}}_{75}^{\mathbf{seg}}$	${\mathbf{AP}}_{\mathbf{S}}^{\mathbf{seg}}$	${\mathbf{AP}}_{\mathbf{M}}^{\mathbf{seg}}$	${\mathbf{AP}}_{\mathbf{L}}^{\mathbf{seg}}$	${\mathbf{AP}}^{\mathbf{bb}}$	${\mathbf{AP}}_{50}^{\mathbf{bb}}$	${\mathbf{AP}}_{75}^{\mathbf{bb}}$	${\mathbf{AP}}_{\mathbf{S}}^{\mathbf{bb}}$	${\mathbf{AP}}_{\mathbf{M}}^{\mathbf{bb}}$	${\mathbf{AP}}_{\mathbf{L}}^{\mathbf{bb}}$
Mask R-CNN	33.4	56.8	34.7	35.8	46.5	23.9	37.2	60.8	40.7	39.8	43.7	16.0
PANet	38.4	61.8	41.4	41.2	47.2	11.8	43.8	65.3	49.3	46.2	52.6	17.5
blendmask	36.7	59.5	39.0	39.5	44.7	10.6	43.1	62.3	48.4	45.8	49.0	19.1
PointRend	38.1	61.3	41.0	40.7	48.6	16.9	43.5	64.6	48.5	45.8	53.0	22.8
D2det	37.5	61.0	39.8	-	-	-	-	-	-	-	-	-
ours	39.7	62.7	42.8	41.9	54.6	25.5	46.1	66.4	52.2	48.5	53.8	26.2

and

Table 5. Class-wise results of instance segmentation on iSAID test set.

Method	$\mathbf{AP}$	SV	LV	PL	ST	Ship	SP	HB	TC	GTF	SBF	BD	BR	BC	RA	HC
Mask R-CNN	33.4	16.9	30.4	41.7	32.0	48.8	36.7	29.6	72.9	25.9	26.7	39.6	15.2	43.1	36.0	5.6
PANet	38.4	17.6	32.1	45.0	37.3	50.7	38.3	33.3	76.2	28.8	38.9	53.4	18.9	48.5	41.3	15.2
BlendMask	36.7	15.8	31.1	45.4	35.8	47.0	38.5	34.5	74.8	22.7	33.9	52.9	16.4	49.1	40.4	11.8
PointRend	38.1	16.3	32.3	46.2	36.3	47.7	37.8	36.5	75.0	28.6	36.9	51.8	17.5	48.2	41.5	13.2
D2det	37.5	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
ours	39.7	18.9	34.4	47.2	37.5	54.7	40.8	35.7	77.6	31.2	40.0	54.5	19.3	49.4	41.9	15.5

, we report the quantitative results achieved by PANet baseline and the proposed method for instance segmentation and object detection tasks on iSAID validation and test sets. We also compared our results with that of those state-of-the-art methods including BlendMask [17], PointRend [15], D2Det [51], etc., for validation.

The overall comparison results on the iSAID validation set are presented in Table 1. The bold numbers represent the highest indicators. It can be seen that our method achieves the highest $40.1\%$ $A{P}^{seg}$ and $46.2\%$ $A{P}^{bb}$ which indicates our superiority both in instance segmentation and object detection tasks. Compared with the baseline method PANet, we yield better performance in all six metrics. The $A{P}^{seg}$ and $A{P}^{bb}$ have improved by $1.9\%$ and $3.0\%$, respectively, which shows the proposed modules effectively improve the overall performance of the model. Furthermore, our method achieves 1.6%, 4.7%, and 11.8% higher values in segmenting small, medium, and large instances in HRSIs while the results of object detection have been also improved to some extents, which indicates that the proposed method has advantages in handling significant scale variation in HRSIs. Compared with other state-of-the-art methods in recent two years, we outperform BlendMask and PointRend by margins of 3.3% and 1.6% respectively in overall $A{P}^{seg}$ and other indices have been also improved correspondingly.

To further verify the capability of our method, we report the class-wise results in Table 2 and Table 3. From Table 2, our method has higher $AP$ than PANet in most categories except tennis court (TC) and roundabout (RA). The performance of our model and PANet in these two categories is almost the same, with the difference of only 0.1% and 0.3% respectively. For categories with irregular boundaries and rich detailed information, such as harbor (HB), we have achieved significant performance improvements by 1.1% and 2.5% in $A{P}^{seg}$ and $A{P}^{bb}$. It confirms that our method can handle the instances with arbitrary shapes in aerial images. Furthermore, for the categories containing a great amount of small and densely distributed objects such as small vehicle (SV) and ship, segmentation accuracy has improved by about 2% while 4% for object detection. These quantitative results show that our model can effectively address the problems of insufficient and inaccurate segmentation for hard samples in multi-class instance segmentation of HRSIs. Considering the results of comparison with other advanced methods, we achieve better performance in many categories while lower $AP$ within 1% decrease for plane (PL), tennis court (TC), and roundabout (RA). It is mainly because the number of objects for these categories is too small (300 vs. 232,457). Therefore, $AP$ can be easily affected by a tiny deviation in prediction.

Table 4 and Table 5 show the comparison results on iSAID test set. The models we use here are trained with the ResNet-101-FPN and all results are tested on the official iSAID evaluation server. Although using deeper networks such as ResNet-152 can improve the performance for remote sensing images to some extent, which is mentioned in [47], it also indicates greatly increasing parameter quantities and heavier computation, and the training and inference time also increase multiply. Considering the computing resources and the model size, we take PANet as our baseline instead of PANet++ in [47] for comparison under the backbone of ResNet-101-FPN. As reported in Table 4, with the designed three modules, we achieve the best performance in every evaluation metric. The average AP^seg and AP^bb rises by 1.3% and 2.3% compared with PANet. Besides, our method is superior to the other multi-category instance segmentation methods. From the class-wise results in Table 5, we also display higher $AP$ in all categories of iSAID than the original PANet. Notice that the $AP$ increment of large vehicle (LV), plane (PL), harbor (HB) is 2.3%, 2.2%, 2.4% and that of ship is up to 4%. Instances in these categories have obvious characteristics like scale variation, complex contours, and dense block distribution, which are the problems we focus on most and want to solve. The improvement shows that our model can effectively address these issues.

Considering the comparison results with other methods in these two tables, similar to the class-wise performance on the validation set, our method has better capability in the main categories such as vehicle (SV and LV) and ship which make up the vast majority of the dataset. The instances in these categories are mostly small-scale and densely distributed. The improvement of the results indicates that our method is superior in detecting and segmenting densely packed small objects. As for the subtle decrease in the result of harbor (HB) compared with PointRend [15], we attribute this to the iterative subdivision algorithm adopted by PointRend, which helps to better segment the edges of these instances with extremely complex contours. However, PointRend is much more complicated than our method for the mask branch of the upsampling convolution, which achieves the comparable segmentation performance but the implementation approach is much simpler.

3.4.2. Results on NWPU VHR-10 Instance Segmentation Dataset

We report the overall performance of the proposed method on the NWPU VHR-10 test set in

Table 6. Overall results of instance segmentation and object detection on NWPU VHR-10 test set.

Method	${\mathbf{AP}}^{\mathbf{seg}}$	${\mathbf{AP}}_{50}^{\mathbf{seg}}$	${\mathbf{AP}}_{75}^{\mathbf{seg}}$	${\mathbf{AP}}_{\mathbf{S}}^{\mathbf{seg}}$	${\mathbf{AP}}_{\mathbf{M}}^{\mathbf{seg}}$	${\mathbf{AP}}_{\mathbf{L}}^{\mathbf{seg}}$	${\mathbf{AP}}^{\mathbf{bb}}$	${\mathbf{AP}}_{50}^{\mathbf{bb}}$	${\mathbf{AP}}_{75}^{\mathbf{bb}}$	${\mathbf{AP}}_{\mathbf{S}}^{\mathbf{bb}}$	${\mathbf{AP}}_{\mathbf{M}}^{\mathbf{bb}}$	${\mathbf{AP}}_{\mathbf{L}}^{\mathbf{bb}}$
Mask R-CNN	62.8	90.2	69.0	42.3	61.3	69.5	60.6	90.8	69.2	52.2	60.9	54.8
PANet	64.8	92.3	72.7	45.1	63.7	72.3	66.3	91.3	77.3	57.9	57.5	60.2
PointRend	65.4	88.1	73.0	42.8	63.4	77.5	64.9	88.3	76.3	54.0	65.1	60.7
BlendMask	65.7	91.3	73.7	41.2	64.5	69.8	68.0	91.1	79.0	56.7	68.1	57.7
ours	67.7	93.3	76.7	48.2	65.0	78.3	69.4	93.1	81.5	58.4	69.5	65.6

. It can be seen that our method presents the best capability in both instance segmentation and object detection among the state-of-the-art with the highest $A{P}^{seg}$ and $A{P}^{bb}$ of 67.7% and 69.4%. Comparing with the results of PANet, we achieve higher performance on NWPU VHR-10 dataset where the improvements are up to 2.9% in $A{P}^{seg}$ and 3.1% in $A{P}^{bb}$. Our improvements in $A{P}_S$ and $A{P}_L$ indicate that our method can detect and segment objects of different sizes better. Notice that the results of instance segmentation are better than that of object detection for Mask R-CNN [11] and PointRend [15], which can be attributed to the characteristics of the dataset. Most instances in the NWPU VHR-10 dataset are with regular boundaries and easy to segment. However, similar object appearances can lead to mutual interference that means they are difficult to detect accurately.

Table 7. Class-wise results of instance segmentation on NWPU VHR-10 test set.

Method	$\mathbf{AP}$	PL	SH	ST	BD	TC	BC	GTF	HB	BR	VC
Mask R-CNN	62.8	43.6	50.4	79.0	82.9	73.0	76.4	83.1	53.9	35.1	50.8
PANet	64.8	50.6	53.5	78.4	83.5	73.0	78.1	87.2	58.6	33.8	51.6
PointRend	65.4	54.5	53.2	75.7	84.3	72.4	74.4	90.1	58.8	35.9	54.7
BlendMask	65.7	48.1	51.1	79.8	84.0	72.4	76.7	91.5	58.9	39.6	54.6
ours	67.7	51.7	55.6	79.8	84.4	73.9	83.0	93.3	60.7	42.3	57.5

and

Table 8. Class-wise results of object detection on NWPU VHR-10 test set.

Method	$\mathbf{AP}$	PL	SH	ST	BD	TC	BC	GTF	HB	BR	VC
Mask R-CNN	60.6	64.8	49.4	76.6	82.8	66.3	69.1	67.9	43.4	29.3	55.9
PANet	66.2	79.0	61.5	75.4	80.7	76.9	74.9	74.1	51.5	32.3	56.3
PointRend	63.8	75.6	60.1	76.3	81.5	71.4	68.4	72.8	43.4	31.9	56.2
BlendMask	68.0	78.5	60.3	76.6	83.7	75.3	79.1	77.2	54.3	35.4	59.7
ours	69.4	82.9	63.2	77.0	83.9	76.2	81.4	77.4	54.8	36.2	60.6

depict the class-wise segmentation and detection results. Compared with the baseline method, our model achieves higher performance in 9 categories except a comparable $A{P}^{bb}$ for tennis court (TC). The segmentation $AP$ of vehicle (VC), basketball court (BC) and ground track field (GTF) have gained more than 5% improvement, which proves the effectiveness of the cross-scale adaptively fusion in dealing with the scale variation problem. It can make full use of multi-scale information and is conducive to better detection and segmentation of objects with different sizes appearing in HRSIs. Besides, with the help of the proposed context attention upsamping, the ability of our model to cope with complex contours has been greatly enhanced, which can be varified from the improvement of our method in the two categories of harbor (HB) and airplane (PL). Furthermore, for the vehicle (VC) class, we obtain 5.9% and 4.3% performance gains in segmentation and detection results. That is because the proposed dynamic sample selection can greatly improve the quality of positive samples, reduce the interference between adjacent targets and enhance the segmentation ability of densely distributed objects.

Compared with other state-of-the-art methods, the proposed method shows better performance in both instance segmentation and object detection tasks, which achieves the highest accuracy in most categories, except airplane (PL). Although the boundaries of many instances in the NWPU VHR-10 dataset are relatively regular, accurate segmentation of airplanes is still challenging due to the existence of wings and propellers with overly complex boundaries. The proposed method alleviates this problem to a certain extent through a simple upsampling operation, but there is still a slight gap compared with PointRend [15]. We attribute it to the iterative subdivision algorithm proposed by PointRend, which is more conducive to segmenting the elaborate part of instances such as the wings of an airplane. However, the implementation of PointRend is much more complicated than ours because it requires pointwise optimization for the hard-to-segment points in object boundaries. Moreover, our method has more advantages in handling the huge scale variations among different instances and dense packed small objects, which PointRend cannot realize.

3.5. Qualitative Results

The class-wise visualization instance segmentation results are shown in Figure 9 and Figure 10. It can be seen that the proposed method has achieved great visual performance in each category and shows satisfactory capabilities in detecting and segmenting instances with different sizes (e.g., ship and ground track field). For densely packed small instances (e.g., small vehicle and large vehicle), there are few cases of false and missing segmentation. In addition, the outlines of the predicted masks are clear and accurate, which is shown in the visualization results on categories such as helicopter, airplane, and harbor.

To further validate the instance segmentation performance, the qualitative results of the proposed approach in comparison with the baseline method PANet on iSAID and the NWPU VHR-10 dataset are depicted in Figure 11 and Figure 12. The first column (a) are the original images in the dataset and the second column (b) are their corresponding ground-truth masks. Columns (c) and (d) denote predicted instance results of PANet and the proposed method. These three images correspond to three types of situations that are difficult to deal with in HRSIs, respectively. The first row is images with huge scale variation among different objects. There are small vehicles in the right part of the image whose sizes are extremely small while the size of the ships and harbor is hundreds of times their size. As we can see, our model eliminates the missing detection and under-segmentation of PANet. The reason is that the cross-scale adaptive fusion strategy can take full advantage of cross-scale feature information and optimize the feature extraction procedure through the network. Therefore, our method can effectively deal with the huge scale differences among objects and remove the false and missing detection phenomenon. The second problem is the existence of a large number of objects with irregular contours and complex boundaries in remote sensing images. As shown in the second row of Figure 11, the shape of objects such as harbors is too complex to segment. Our method can segment the harbors more accurately than the baseline, which suggests that the proposed strategy can indeed mitigate the under-segmentation problem of objects with arbitrary shape. The third row of images shows that the remote sensing images contain large quantity dense and small objects which are formidable to detect and segment. The visualization results demonstrate that our method addresses this issue as mentioned above.

The visualization results of the proposed method on NWPU VHR-10 instance segmentation dataset are shown in Figure 12. Our overall performance is better than that of the baseline method PANet. It can be observed from the first and second rows that PANet has false and missing detection of objects (e.g., the bridge and the vehicle) with different sizes, while our method can handle this situation. It is because adaptive multi-scale feature fusion can make fuller use of high-level and low-level object-wise semantic information and enhance the ability to detect and segment both of large and small objects. As shown in the third row, our model compensates for under segmentation problem of PANet, which comes from interference caused by similar backgrounds. We attribute this refinement to high-quality proposals selected by DSS. It strengthens the ability of feature representation and improves the quality of the candidate boxes sent into the regression and mask branches. Similarly, when objects distribute in dense blocks, our segmentation masks of objects are much smoother than the baseline. It is mainly because the proposed context attention upsampling has advantages over other general strategies when segmenting elaborate boundaries. The module generates convolution kernels associated with the semantic content of the feature map, which can guide the subsequent segmentation operation.

3.6. Ablation Study

To verify the effectiveness of each proposed strategy, this section tests their performance on iSAID and NWPU VHR-10 datasets. The results are shown in

Table 9. Validations of different strategies on iSAID validation set.

AUG	CSAF	CAU	DSS	Instance Segmentation						Object Detection
				$\mathbf{AP}$	${\mathbf{AP}}_{\mathbf{50}}$	${\mathbf{AP}}_{\mathbf{75}}$	${\mathbf{AP}}_{\mathbf{S}}$	${\mathbf{AP}}_{\mathbf{M}}$	${\mathbf{AP}}_{\mathbf{L}}$	$\mathbf{AP}$	${\mathbf{AP}}_{\mathbf{50}}$	${\mathbf{AP}}_{\mathbf{75}}$	${\mathbf{AP}}_{\mathbf{S}}$	${\mathbf{AP}}_{\mathbf{M}}$	${\mathbf{AP}}_{\mathbf{L}}$
				34.2	56.6	35.8	19.6	42.3	46.6	41.7	60.9	46.6	26.9	47.8	51.0
✓				38.2	62.4	41.0	40.4	51.9	55.5	43.2	66.0	47.6	45.5	49.4	58.4
✓	✓			39.4	63.1	42.6	41.2	56.2	71.5	45.7	67.0	50.8	47.5	57.2	77.4
✓	✓	✓		39.7	63.4	42.9	41.4	56.4	71.9	45.8	67.2	51.1	47.5	57.3	77.4
✓	✓	✓	✓	40.1	64.6	43.0	42.0	56.6	77.3	46.2	68.8	51.3	48.3	58.3	80.1

and

Table 10. Validations of different strategies on NWPU VHR-10 instance segmentation test set.

AUG	CSAF	CAU	DSS	Instance Segmentation						Object Detection
				$\mathbf{AP}$	${\mathbf{AP}}_{\mathbf{50}}$	${\mathbf{AP}}_{\mathbf{75}}$	${\mathbf{AP}}_{\mathbf{S}}$	${\mathbf{AP}}_{\mathbf{M}}$	${\mathbf{AP}}_{\mathbf{L}}$	$\mathbf{AP}$	${\mathbf{AP}}_{\mathbf{50}}$	${\mathbf{AP}}_{\mathbf{75}}$	${\mathbf{AP}}_{\mathbf{S}}$	${\mathbf{AP}}_{\mathbf{M}}$	${\mathbf{AP}}_{\mathbf{L}}$
				63.1	90.5	68.6	42.0	61.9	69.6	62.3	90.3	73.6	54.2	63.5	52.2
✓				64.8	91.3	72.7	45.1	63.7	72.3	66.3	91.3	77.3	57.9	57.5	60.2
✓	✓			65.4	92.4	72.6	45.8	64.1	73.4	68.0	92.5	79.0	58.3	68.6	61.8
✓	✓	✓		66.6	91.3	75.7	46.9	64.8	74.8	68.2	92.8	79.6	58.4	68.6	62.2
✓	✓	✓	✓	67.7	93.3	76.7	48.2	65.0	78.3	69.4	93.1	81.5	58.5	69.5	65.6

. AUG denotes the random multi-scale scaling strategy used in the training phase; CSAF is the proposed cross-scale adaptive fusion module; CAU represents the Context Attention Upsampling module; and DSS denotes the dynamic sample selection module. Our method is to deploy the above three modules based on PANet and the results without any proposed strategy in Table 9 and Table 10 are tested with the baseline PANet.

It can be observed from Table 9 that after adding multi-scale training to the benchmark method PANet, the performance of instance segmentation task and the object detection task has been improved by 4% and 1.5%, respectively. Since the random multi-scale scaling strategy introduces the prior knowledge of artificial visual invariance into the network, the generalization performance of the model is greatly enhanced. Besides, with the CASF module, our results have been further improved by 1.2% for segmentation and 2.5% for detection. Thanks to the efficient use of information brought by the adaptive fusion strategy, the detection and segmentation accuracy of both small and large objects have been promoted in varying degrees. Furthermore, with the help of CAU, our $A{P}^{seg}$ result has improved by 0.3% while 0.1% for $A{P}^{bb}$. Because this strategy is only embedded in the branch of instance segmentation, and it is relatively independent of the object detection branch, so that the improvement in the target detection task is not as obvious as the former. As shown in the last line, through the DSS module, we achieved the best results with the highest segmentation $AP$ 40.1% and detection $AP$ 46.2%. These improvements can be attributed to the fact that our sample selection strategy can dynamically adjust the threshold for selecting positive and negative samples changing with the training process. DSS makes full use of the statistical characteristics of different input images and generates samples of higher quality. We also add a penalty item when calculating IoU in DSS, which reflects the relative position of the candidate bounding boxes and the ground truth to some extent compared with the original IoU. Therefore, our model is more discernible to the quality of candidate boxes.

Table 10 displays the results of our ablation experiments on the NWPU VHR-10 instance segmentation dataset. Similar to the results on the iSAID validation set, each of the proposed strategies has achieved further performance improvement. By using scale augmentation (AUG) during the training procedure, the overall performance of PANet improves a lot. The cross-scale adaptive fusion (CSAF) makes the network more invariant to the size change of various objects. After using CSAF, the segmentation results of the model for large and small instances has been obviously improved where $A{P}_S$ and $A{P}_L$ has been boost by 0.7% and 1.1%. As for the performance of CAU, similar to the results on iSAID, our segmentation result has dramatically improved by 1.2%, while only 0.2% for object detection, which is because the proposed module mainly works on the mask branch and has little influence on the detection branch. Finally, based on the above strategies, our model utilizes the DSS module and gets the highest $A{P}^{seg}$ and $A{P}^{bb}$ of 67.7% and 69.4%, which achieves the state-of-the-art performance on the dataset.

4. Discussion

In this paper, we study the instance segmentation task for remote sensing images, and the proposed instance segmentation method has achieved impressive results. However, there are some limitations in this work, which need to be further improved and optimized. The first limitation is that it is difficult to well segment instances with large aspect ratios such as bridges. Our detection and segmentation $AP$ results for bridges on both iSAID and NWPU VHR-10 instance segmentation datasets are not very satisfactory. The default anchor ratio settings commonly adopted by most state-of-the-art methods (including our baseline PANet [13]) are 1:1, 1:2, and 2:1. These default anchors can indeed well represent most of the instances in the two datasets whose aspect ratios do not vary significantly. However, they are not appropriate for the instances with large aspect ratios in remote sensing images, which brings difficulties for subsequent location regression and instance-wise segmentation. Secondly, since the HRSIs are usually taken from the bird’s-eye view, instances in HRSIs always present arbitrary orientations [47,61]. However, current detection-based instance segmentation methods still use horizontal candidate boxes to represent the rotated instances, paying no attention to their orientation information. The horizontal candidate boxes typically lead to misalignment between the bounding boxes and the oriented instances, which brings challenges for extracting object features and identifying the instance’s accurate localization [7,8], thus also affects the segmentation results. Therefore, introducing the orientation information into the instance segmentation network for aerial images is conducive to predicting accurate oriented candidate boxes with rotation angles, which can further improve the performance of the detect-then-segment models.

5. Conclusions

In this paper, we design an end-to-end multi-category instance segmentation network for high-resolution remote sensing images aiming at address the problems of the huge scale variation, arbitrary instance shapes, and numerous densely packed small objects in HRSIs. Firstly, we propose a cross-scale adaptive fusion module with a novel multi-scale information fusion mechanism to enhance the capability of the model to detect and segment objects with obvious size differences. Then, we use a context attention upsampling module to substitute the original deconvolution layer in the segmentation branch which can obtain more refined predicted masks. Finally, We extend the traditional fixed positive and negative sample judgment threshold into a dynamic sample selection module to select more suitable positive and negative samples flexibly. It has obvious advantages in the segmentation of densely packed objects. The proposed method has achieved the state-of-the-art performance on two challenging public datasets for instance segmentation task in HRSIs.