Integrating Prior Knowledge into Attention for Ship Detection in SAR Images

Abstract

Although they have achieved great success in optical images, deep convolutional neural networks underperform for ship detection in SAR images because of the lack of color and textual features. In this paper, we propose our framework which integrates prior knowledge into neural networks by means of the attention mechanism. Because the background of ships is mostly water surface or coast, we use clustering algorithms to generate the prior knowledge map from brightness and density features. The prior knowledge map is later resized and fused with convolutional feature maps by the attention mechanism. Our experiments demonstrate that our framework is able to improve various one-stage and two-stage object detection algorithms (Faster R-CNN, RetinaNet, SSD, and YOLOv4) on two benchmark datasets (SSDD, LS-SSDD, and HRSID).

1. Introduction

Ship detection has received increasing attention in the field of synthetic aperture radar (SAR) for its broad application for military [1,2,3], marine traffic monitoring [4,5], harbor surveillance [6,7], etc. Following their success in optical images, deep learning-based methods are replacing traditional methods, which rely on manually designed feature extraction, as the most popular models for SAR images [8,9,10,11,12,13]. Among the deep learning methods, deep convolutional neural networks (CNNs) overwhelm other branches in the computer vision.

Despite the popularity of CNNs for various tasks of SAR images, ship detection remains a challenging task for three reasons. Firstly, SAR images do not have important features, such as color and textual features, which are vital for object detection. Secondly, ships are normally quite small in SAR images. Therefore, resolutions are too small to easily locate ships [14,15,16,17]. Thirdly, for complex backgrounds (e.g., coast), ship detection is heavily affected by the scattering points from the background [6,18,19,20,21].

Although the imaging mechanism of SAR images poses great obstacles, it provides supplementary information for ship detection. In most cases of SAR images, the backgrounds of ships are water surface (e.g., ocean, river) or coast. On the one hand, due to complex scatterings from the ships, such as volume scattering and double-bounce, the intensity of echoes generated by ships is significantly stronger than that of the water surface [22,23]. On the other hand, due to the structure and material of the ship, the ship has many strong scattering objects [24], i.e., strong scattering points. Studies [25,26] have shown that the strong scattering points of the ship are more dense than the surrounding areas.

Over the recent years, a series of explorations have been proposed to combine prior knowledge with deep learning methods. Gao et al. [27] extracted the SAR polarization features [28] using the power–entropy (PE) decomposition theory. After that, the polarization features were fed into RetinaNet [29] as the input. Zhang et al. [30] defined the saliency value of a pixel as the color contrast of all other pixels in the image. They generated high-quality slices using a saliency detection method to suppress background clutter, before using CNNs to further detect ship targets. Their method substantially improved the detection capability of offshore ships while ensuring the inshore ship detection performance. However, due to the over-suppression of saliency detection, their method did not perform well when facing small ships with very few pixels. Sun et al. [31] pointed out that the general deep learning-based detectors perform poorly because they ignore the characteristics of SAR images. They grouped all the extracted strong scattering points, and generated the initial ROI areas based on the grouping result. Compared to those extracting only a portion of strong scattering points as key points, Sun et al.’s method performed better for small targets. However, the method of getting the initial ROI at the beginning of training was trained insufficiently, which makes the initial ROIs inaccurate and causes the model to fail to learn valid knowledge. Although previous works recognize the importance of prior knowledge for the CNNS, they combine the prior knowledge by feeding it into the networks. Therefore, the networks could not access the raw information which may be missing from the prior knowledge.

In this paper, we propose a framework which integrates prior knowledge into CNNs by fusing it with an attention mechanism. In our model, the prior knowledge is used to guide the attention of deep networks. Our framework can be categorized into two stages: prior knowledge representation and prior knowledge integration. The aim of the first stage is to generate the probability score for each pixel of which how likely it is on the ship area or background. Based on the findings from other works [28,31,32], brightness and density are two promising features for prior knowledge. To this end, we employ k-means clustering as brightness filtering because there are ships and backgrounds in a bounding box. Similarly, DBSCAN clustering is utilized as the density filtering. For the second stage, we down-pool the original prior knowledge map into the same size as the feature map. After that, position-wise multiplication integrates the resized prior knowledge map into the CNNs.

Our contributions can be summarized into three items.

• To our best knowledge, our framework is the first that can integrate prior knowledge into arbitrary CNN-based detectors using attention mechanism for SAR images.
• By the attention mechanism, the deep learning models can learn from both prior knowledge and vanilla images.
• Our experiments exhibit the superiority of our method using object detection algorithms (Faster R-CNN, RetinaNet, YOLOv4, and SSD) on three SAR image datasets (SSDD, LS-SSDD, and HRSID).

The remainder of this paper is organized as follows. Section 2 presents the related works about traditional methods, CNN-based object detection algorithms, and attention mechanisms. Section 3 details our intuition and approaches to represent prior knowledge. Section 4 demonstrates the combination of prior knowledge and deep CNNs. Section 5 displays the settings, descriptions, and results of our experiments.

2. Related Work

2.1. General CNN-Based Object Detection

Neural networks have a family of variants, including convolutional neural networks (CNNs) [33], recurrent neural networks (RNNs) [34], auto-encoders [35,36], transformers [37,38], etc. CNNs are one of the most vital branches that are widely applied in computer vision tasks, including object recognition [39], object detection [29,40,41,42], segmentation [43], and image generation [44]. CNN-based object detectors can be grouped into single-stage and two-stage algorithms. Faster R-CNN [45] is one of the most widely used two-stage detectors, which is an extension of Fast R-CNN [42]. The Faster R-CNN replaces the selective search of Fast R-CNN with region proposal networks (RPN). The RPN enables the end-to-end training and further reduces the computational complexity. The single-stage algorithms do not rely on proposed regions, such as typical single-stage algorithms YOLO [41,46,47], SSD [48], and RetinaNet [29], which perform classification and regression directly from feature maps. The YOLO [46] models the detection task as a pure regression problem. It divides the image into grids of patches, and performs the prediction and classification of the bounding boxes for the center of each grid. The core of RetinaNet [29] is focal loss, a loss function derived from a modification of binary cross-entropy. Focal loss makes the model more focused on hard-to-classify samples during training by reducing the weights of easy-to-classify samples. YOLOv4 [41] uses PANet [49] instead of the commonly used FPN as the neck of the network and assembles various optimization strategies for CNNs, including Mosaic data augmentation method, Mish activation function [50], and CIoU loss function [51], to improve the detection accuracy of the algorithm.

In the deep learning era, the CNNs have dominated the object detection in SAR images. Lv et al. [52] combined a sliding window and Faster R-CNN approach to detect pylons in SAR images and used a data enhancement strategy to train it. Their detector has good immediacy and detection rate in large-scene SAR images. Ge et al. [53] improved the feature extraction capability of the algorithm for azimuthally inscribed objects by adding SENet and inverted residuals to the backbone of YOLOv5. Their proposed detector was suitable for detecting azimuthally sensitive targets, such as aircraft. Zhang et al. [54] cross-coupled the edge-aware network by a residual space pyramid set and attention mechanism to help object recognition. In addition, they improved the model’s multi-scale feature extraction capability by semi-dense connectivity based on residual convolution blocks. Their proposed SAR image detector performed well on SAR oil tank and residential area images. Sun et al. [55] used transformer to build the neck of YOLOv5. They found out that adding the structure of transformer in the high level of YOLOv5 can help the model to obtain global background information, which improved the performance of YOLOv5. This method achieved good results in detecting military vehicles in large scenarios.

Deep CNNs have made a lot of breakthroughs for ship detection in SAR images as well. Tang et al. [8] proposed a new ship detection model FLNet based on YOLOv5. This model combined traditional image processing and deep learning-based methods, with improved accuracy and recall. Zhu et al. [9] used FCOS as the baseline, redesigned the feature extraction method, and redefined the sample according to the statistical features of SAR ships to reduce the missed detection rate of small ships. Li et al. [10] replaced the original RPNs with a fixed number of prior anchor boxes with K-means, applied cascade amplification and feature fusion to design the feature extraction network, and improved the speed and accuracy of the Faster R-CNN. To effectively aggregate the sparse and meaningful clues of small ships, Shi et al. [11] introduced a deformable attention mechanism based on the Swin Transformer to change the original self-attention mechanism. Yu et al. [12] constructed a highly accurate and highly generalizable network, FIERNet, by combining feature extraction and fusion modules. The differentiable neural structure search model proposed by Li et al. [13] employs a new channel cropping scheme and loss function that can generate significantly lighter neural networks with guaranteed accuracy.

2.2. Prior Knowledge for Detection SAR Images

Before the popularity of deep learning, constant false alarm rate (CFAR) was one of the most seminal feature extraction methods [56] for object detection in SAR images. CFAR is a statistical detection method based on hypothesis testing theory, which ensures a constant false alarm probability by selecting a suitable threshold, and then determines whether the target signal exists under this condition. The MS-CFAR [57], which takes into account the statistical characteristics of the tested cell, has a more stable performance compared to other algorithms based on CFAR. The statistical clutter edge selector designed in [58] can obtain a uniform clutter field before CFAR, improving the detection performance of the classical CFAR detector. Moreover, the idea of improving the algorithm by using clutter boundary statistical features is also reflected in [59]. The bi-parametric clutter map CFAR detection method proposed by Wang et al. [60] estimates the local threshold by following the standard deviation and mean values of the new clutter map, which has better applicability and target detection capability compared to the mono-parametric method. Zhou et al. [61] chose to use the half-sided Gaussian distribution for modeling and proposed a new method called HG-CFAR. This method has higher efficiency relative to the CFAR method based on a Gaussian distribution.

Specifically, there is a group of works which are for ship detection only. Li et al. [62] found that the grayscale values on SAR images fluctuate due to the influence of speckle noise. Such fluctuations take a regular form in uniform regions and an irregular form in non-homogeneous regions. The ship region is a homogeneous region while the ship boundary region and the artificial target are non-homogeneous regions. Based on this phenomenon, they designed a new texture feature IUDF for helping ship detection in SAR images. The proposed algorithm using IUDF has high accuracy and processing speed. The ship, as a scatterer, has a strong coherent backscattering signal, and this feature can be formed by evaluating the sub-resolution electromagnetic field, in which Gambardella et al. [63] considered the Rice factor as a parameter that is sensitive to a dominant scatterer such as a ship. Therefore, they designed a filtering technique based on the Rice factor, by which the accuracy and efficiency of the ship detection algorithm can be effectively improved. Zhang et al. [64] also designed a filtering technique for SAR ships, but their technique was based on the strong double-bounced scattering phenomenon occurring from ships. In [65], the authors found that the azimuth ambiguities caused by the undersampling of the echo signal during SAR imaging can lead to false alarms of ships. As a result, they located and eliminated the ambiguous points in SAR images in advance according to the imaging coefficients of SAR, as a way to reduce the interference of orientation-ambiguous points to SAR ship detection.

2.3. Attention Mechanisms

Attention mechanisms make the model focus on more critical features. Squeeze-and-Excitation Networks (SENet) [66] and Convolutional Block Attention Module (CBAM) [67] are two of the most popular attention mechanisms. SENet computes channel weights by compressing the information of all points in the space and uses the fully connected layer to exploit the correlation between channels, effectively improving the accuracy of the network. In [68,69], the researchers added SENet to the feature extraction network to enhance the representation of vehicle features. Compared to SENet, which simply uses the channel attention mechanism, CBAM additionally uses the spatial attention mechanism. One of the advantages of CBAM is that it is plug-and-play and can be easily incorporated into various algorithms. Zhu et al. [70] integrated CBAM into YOLO to help YOLO better find regions of interest. Wang et al. [71] made their VGG-style network more focused on effective features by embedding CBAM. Feng et al. [72] combined CBAM with SSD to effectively suppress useless information and improved the pedestrian detection accuracy of the algorithm.

Attention mechanism has been extensively used for object detection in SAR images. Zhang et al. [73] designed a lightweight attention mechanism which has significant performance and efficiency for the detection task of multi-class SAR targets in both moving and stationary situations. Combined with feature pyramids, SENet achieved competitive performance in the SAR bridge detection [74], because the SENet enables detectors to suppress irrelevant information while obtaining additional valid information. Guo et al. [75] used a mixture of feature pyramid networks and CBAM to further process the features enhanced by scattering information, effectively adapting to the discrete and variable nature of aircraft in SAR images.

Attention mechanism is useful for ship detection as well. The ship attention module proposed by Sun et al. [21] distinguishes the strong scattering points of artificial facilities on land and ships in water by generating a feature map containing rich texture features, which effectively reduces false alarms in the land area and improves the accuracy of the algorithm. Wang et al. [7] designed a Spatial Group-wise Enhance (SGE) attention module to reduce the problem of adjacent ships being ignored. SGE reduces the computational effort in the form of channel grouping, while enhancing the spatial features of each group of channels, providing more semantic features. Their experimental results showed that their method has better detection capability for densely docked ships and makes a contribution to ship detection in complex scenarios. Shao et al. [76] designed a dynamic shrinkage attention mechanism to address the impact of speckle noise on SAR images, which can automatically learn the soft threshold required by the denoising algorithm to achieve feature-level denoising.

3. Representing Prior Knowledge

3.1. Analysis

SAR images are different from optical images due to different imaging mechanisms. Optical images record multi-band data, while SAR images record single-band data, such as the amplitude and phase of one band of echoes. Therefore, SAR images are usually single-channel pictures, which have a pixel value range of 0 to 255, inclusively. The digital image element, or pixel value, reflects the intensity of the scattered echoes before the imaging. The stronger the echo signal, the larger the pixel value is. Figure 1 visualizes ships in optical images (first row) and SAR images (second row), respectively. In the optical images, we can easily locate ships by the contour, color differences, and texture. However, there is a large amount of speckle noise in SAR images, and the ship targets are usually blurred. Ship targets in SAR images not only lack rich color information, but also possess unclear texture features. This delivers difficulties for the detection of ships in SAR images.

The intensity of scattered echoes roughly depends on two types of factors [77]. The first factor is the operating parameters of the radar system, including the operating wavelength of the radar sensor, the angle of incidence, the polarization mode, etc. The second factor is the characteristics of the ground target, including the roughness of the ground surface, the radar scattering cross section (RCS), the complex permittivity of the target, etc. They mainly affect the type of scattering occurring from the target and thus the intensity of the scattered echoes [77,78,79,80]. Ships generally have simultaneous volume scattering, surface scattering, and double bounce [81]. The echoes generated by double-bounce are very strong. Even if only surface scattering is considered, the echoes generated by the ships will be stronger than those from the sea surface. We refer to this phenomenon as the brightness difference. Figure 2 illustrates this phenomenon. The ships look significantly brighter than the sea surface.

From Figure 3, we can conclude the strong scattering points on land also produce high-intensity echoes, which leads to the pixels on the SAR image having similar brightness as ships. Therefore, brightness alone is not enough to distinguish ships from the background. SAR imaging has the inherent defect of coherent speckle noise [82]. In SAR images, the grayscale values of neighboring pixel points can vary randomly due to coherence, which is around a certain mean value. As a result, a uniform scatterer can also appear to vary unevenly between bright and dark on a SAR image. Targets with different structures are affected differently by speckle noise due to the different distributions of the basic scatterers in their bodies. We describe this phenomenon in terms of the density of bright spots in the region. From Figure 3, we can discover that the density of the bright spots of the ships in the red boxes is significantly higher than that of the background in the green boxes.

3.2. Generating by Brightness

As aforementioned in Section 3.1, brightness is important to distinguish ships from the water surface. Therefore, we can use an interval of pixel values to separate ships from the water surface. Only pixels within are considered as ships. Since the images from training set and test set are i.i.d., the interval which fits for the training set suits the test set as well.

We perform clustering on pixels of each bounding box (annotations of training set) individually. we use K-means [83] clustering with two classes (one for the foreground and the other for the background) to cluster the pixel values. The cluster which has a higher value as the center than that of the other cluster represents the foreground. From the foreground pixels, we find a pair of median value $me{d}_i$ and max value $ma{x}_i$, where i indicates the global index of bounding boxes. The global median value $med$ and max value $max$ could be calculated as:

$$med=arg\underset{x}{min}\frac{1}{n}\sum _i{(me{d}_i-x)}^2$$

(1)

$$max=arg\underset{x}{min}\frac{1}{n}\sum _i{(ma{x}_i-x)}^2$$

(2)

Here, n is the total number of bounding boxes. The final interval is $[2·median-max,max]$.

As can be seen from Figure 4, most of the points in the clustering results are distributed at higher pixel value levels. It can be illustrated that most pixels in the bounding boxes represent ships while others are noises. The intervals from Equations (1) and (2) visualized on the second row suggest the effectiveness of our approach.

3.3. Generating by Density

As discussed in Section 3.1, brightness is insufficient to separate ships from land, but density is a promising feature, because the pixels on the ships are more dense than that on land. We use DBSCAN [84] for the filtering in this stage. The input of the DBSCAN is the foreground pixels of the brightness filtering. DBSCAN owns two hyper-parameters to tune: $Eps$ and $MinPts$. Two points are considered neighbors only if the distance between them is no bigger than $Eps$. A core point has at least $MinPts$ points in its neighbors.

Instead of adopting the strategy introduced in the original paper [84], we use the metric IoU to tune the hyper-parameters. Experiments are conducted using different combinations of $Eps$ and $Minpts$, and the final set of parameters with the highest IoU is selected. The IoU of a single image and the IoU of the whole dataset are calculated as shown in Equations (3) and (4):

$$IoU=\frac{U}{\widehat{X}+X-U}$$

(3)

$$AverageIoU=\frac{{\sum }_{i=1}^n{I}_i\times B_i}{{\sum }_{i=1}^n{B}_i}$$

(4)

In Equation (3), U represents the number of points where the clustering result overlaps with the real ship. X represents the number of points in the clustering result. $\widehat{X}$ represents the number of real ship points. In Equation (4), n is the total number of images. $I_i$ represents the IoU of the ith image. $B_i$ represents represents the number of targets on the ith image.

After sequentially applying K-Means clustering and DBSCAN clustering, for each pixel on the SAR images, we obtain a binary value for it. A pixel with value one is considered a part of ship, while a pixel with value zero is taken as background. Such matrix is denoted as prior knowledge map because the values are calculated based on prior knowledge of SAR images. The prior knowledge map has the same size as the SAR image but with a single channel.

4. Combining Prior Knowledge Map

Figure 5 depicts the comprehensive architecture of our framework. The framework has three important parts: the backbone, our attention component (BDAM), and the prior knowledge map. The backbone can be ResNet [85], VGG [86], GoogleNet [87], etc. Our attention component (BDAM) does not change the sizes of the input and output of the backbone. We insert it between stages of the backbone. The shape of the input and output of BDAM is equally the same.

4.1. Resizing Prior Knowledge Map

The diagram of our attention component is depicted in Figure 6. The original prior knowledge map, on which each point indicates if it is foreground or background, has the size of $(H,W,1)$. In order to be attended with the feature map, we need to convert the prior knowledge map to the same size as the feature map. Let us assume that a feature map has the size $(h,w)$. We divide the original prior knowledge map into $m\times n$ windows, where $m=\lfloor \frac{H}{h}\rfloor$ and $n=\lfloor \frac{W}{w}\rfloor$.

The value of each window is the number of foreground points within. Intuitively, the window with more foreground points requires more attention. We normalize the counting of foreground points into $[0,1]$ as the confidence score. The process of converting the original prior knowledge map into the resized prior knowledge map is depicted in Figure 7.

4.2. Integrating Prior Knowledge Map

Traditional attention mechanisms (such as SENet) serve as self-attention, which have one input tensor. In our work, there are two input tensors: one is for the resized prior knowledge map; and the other is the feature map of CNNs. The output tensor is considered as a better representation of the feature map of CNNs. The output is fed into latter stages of the CNNs.

Our BDAM extends the structure of CBAM. The detailed structure of BDAM is shown in Figure 6. Given an intermediate feature map, CBAM (depicted in Figure 6a) successively infers attention maps along two separate dimensions, channel and spatial. After that, the attention maps are multiplied to the input feature map for adaptive feature refinement. Our module (BDAM) uses the CNN feature map for channel attention and the prior knowledge map for spatial attention (Brightness and Density Spatial Attention Mechanism, BDSM), which is shown in Figure 6b.

The intermediate CNN feature map $\mathbf{F}\in {\mathbb{R}}^{C\times H\times W}$ and the prior knowledge map $\mathbf{P}\in {\mathbb{R}}^{C^{\prime }\times H^{\prime }\times W^{\prime }}$ are used as the input of BDAM. $\mathbf{F}$ is passed through the channel attention mechanism to obtain the channel attention weight ${\mathbf{M}}_{\mathbf{c}}\in {\mathbb{R}}^{C\times 1\times 1}$, while $\mathbf{P}$ is passed through the spatial attention mechanism to obtain the spatial attention weight ${\mathbf{M}}_{\mathbf{s}}\in {\mathbb{R}}^{1\times H\times W}$. The channel attention weight and spatial attention weight are multiplied to get the final attention weight $\mathbf{M}\in {\mathbb{R}}^{C\times H\times W}$. This process can be described as:

$$\begin{array}{cc}\hfill \mathbf{M}& ={\mathbf{M}}_{\mathbf{c}}\left(\mathbf{F}\right)\otimes {\mathbf{M}}_{\mathbf{s}}\left(\mathbf{P}\right)\hfill \\ \hfill {\mathbf{F}}^{\prime }& =\mathbf{M}\otimes \mathbf{F}\hfill \end{array}$$

(5)

where ⊗ represents the element-wise multiplication, and ${\mathbf{F}}^{\prime }$ is the feature map after adjusting the weights.

The channel attention module can be divided into two branches: max-pooling and average-pooling. Max-pooling and average-pooling operations are performed on the CNN feature maps to obtain ${\mathbf{F}}_{\mathbf{max}}$ and ${\mathbf{F}}_{\mathbf{avg}}$. ${\mathbf{F}}_{\mathbf{max}}$ and ${\mathbf{F}}_{\mathbf{avg}}$ are fed into the multilayer perceptron (MLP), and the two results output by the MLP are summed element-wise. The merged tensor is finally passed through the Sigmoid function to obtain the channel weights. This process can be described as:

$${\mathbf{M}}_{\mathbf{c}}\left(\mathbf{F}\right)=Sigmoid({\mathbf{W}}_{\mathbf{0}}\left({\mathbf{W}}_{\mathbf{1}}\left({\mathbf{F}}_{\mathbf{max}}\right)\right)+{\mathbf{W}}_{\mathbf{0}}\left({\mathbf{W}}_{\mathbf{1}}\left({\mathbf{F}}_{\mathbf{avg}}\right)\right))$$

(6)

where ${\mathbf{W}}_{\mathbf{0}}\in {\mathbb{R}}^{C\times C/r}$,${\mathbf{W}}_{\mathbf{1}}\in {\mathbb{R}}^{C/r\times C}$ represent the weights of the MLP, and r is the reduction ratio.

In the BDSM, the prior knowledge map is first resized by a multilayer convolutional neural network. The resized prior knowledge map is subjected to maxi-pooling and average-pooling operations in the channel direction. Then, the outputs ${\mathbf{P}}_{\mathbf{max}}$ and ${\mathbf{P}}_{\mathbf{avg}}$ of the two branches are concatenated and fed into a convolutional layer with a convolutional kernel of size $7\times 7$. The spatial attention weights are calculated as:

$$\begin{array}{cc}\hfill {\mathbf{M}}_{\mathbf{s}}\left(\mathbf{P}\right)& =Sigmoid\left(f\right(\left[MaxPool\right(\mathbf{W}\left(\mathbf{P}\right));AvgPool(\mathbf{W}\left(\mathbf{P}\right)\left]\right)\hfill \\ \hfill & =Sigmoid\left(f\left([{\mathbf{P}}_{\mathbf{max}};{\mathbf{P}}_{\mathbf{avg}}]\right)\right)\hfill \end{array}$$

(7)

where $\mathbf{W}$ represents the weights of the initial layers of convolution and f represents the last layer of convolution.

5. Experiments and Results

5.1. Environment

Our experiments are conducted with one Tesla V100 DGXS 32GB GPU. The operating system was Ubuntu 18.04. We used PyTorch 1.8.0, cuda 11.6, and cudnn 7.6.5 as our GPU platform.

5.2. Datasets

The datasets used in the experiment are SSDD [88], LS-SSDD [89], and HRSID [90]. These datasets contain SAR ships only. SSDD has 1160 images with 2456 ships. SSDD derives from three satellites: RadarSat-2, TerraSAR-X, and Sentinel-1, which display ships at multiple scales and in multiple scenarios. The HRSID comes from three radar satellites: TerraSAR-X, TanDEMX, and Sentinel-1B, with a total of 5604 images containing 16,951 ships. The data of LS-SSDD are from one satellite, Sentinel-1, with a total of 9000 images. The three datasets are annotated as the format of PASCAL VOC. The size of ships within these three datasets varies widely. According to the MS COCO [91] definition of target size, LS-SSDD consists of no big ships and almost small ships, while SSDD and HRSID contain various sizes of ships.

Table 1. Statistics of datasets.

Datasets	The Largest Ship (pixel ${}^2$)	The Smallest Ship (pixel${}^2$)	Average (pixel${}^2$)	Number of Pictures
HRSID	522,400	3	1808	5604
SSDD	62,878	28	1882	1160
LS-SSDD	5822	6	381	9000

demonstrates the details of the three datasets. Because of the scarcity, SAR datasets are smaller than other detection datasets, such as COCO which contains 328,000 images.

For the SSDD, we follow the data split instructed by its owner [92]. Those images with a tail number between 2 and 7 or 0 are divided into the training set, while those with a tail number of 8 are used for validation during the training process and, finally, the remaining images are used for testing. The division of validation, test, and training sets in the HRSID dataset is randomly divided according to 1:1:8. The authors of LS-SSDD gave a fixed division of the dataset into a training set of 6000 and a test set of 3000. In the experiments of this paper, we randomly delineated 1000 from the 6000 training sets as the validation set.

Table 2. Distribution of datasets.

Datasets	Training	Validation	Testing	Total
SSDD	812	116	232	1160
HRSID	4538	505	561	5604
LS-SSDD	5000	1000	3000	9000

shows the number of images for different purposes after the division of the three datasets.

5.3. Implementation Details

In this experiment, the backbone networks of all algorithms are loaded with parameters pretrained on the ImageNet dataset. Among them, the backbone networks of the algorithms are ResNet50 except for YOLOv4 whose backbone network is CSPDarkNet53 [41]. Faster R-CNN and RetinaNet additionally add FPN as the neck of the network.

The optimizer used in the training process is SGD, whose parameters $momentum$ and $weight$$decay$ are set to 0.937 and 0.0001. The training epochs of the other three algorithms are 25, except for YOLOv4 where the training epochs are 200. The data augmentation strategies used in training the four algorithms are RandomFlip and Resize. MixUp and Mosaic data augmentation strategies are additionally used in the YOLOv4 training, where Mosaic is only used in the first 70% of epochs (140 epochs).

5.4. Evaluation Criteria

Following the convention of MS COCO [91], we adopt the following six metrics for evaluation: $AP$ (Average Precision), $A{P}_{50}$, $A{P}_{75}$, $A{P}_S$, $A{P}_M$, and $A{P}_L$. $A{P}_{50}$, $A{P}_{75}$ are the AP values obtained with IoU fixed at 0.5 and 0.75. $A{P}_S$, $A{P}_M$, and $A{P}_L$ calculate the average AP values for all small, medium, and large targets, respectively.

5.5. Hyper-Parameter Analysis

As mentioned in Section 3.3, clustering parameters are the key point of the attention mechanism.

Table 3. The brightness intervals of SSDD, LS-SSDD, and HRSID.

Datasets	Median of the Interval	Interval Width
SSDD	0.9531	0.0928
HRSID	0.8245	0.3443
LS-SSDD	0.8507	0.2346

displays the brightness intervals of the ships obtained with brightness clustering in the datasets.

From the table, we observe that the same class of targets may have different brightness intervals in different datasets. The cause is the different system parameters used in the acquisition of different datasets. Therefore, it is necessary to use brightness clustering to learn the brightness intervals of the targets in the dataset in advance.

Figure 8 shows the performance of different $Eps$ and $MinPts$ on three datasets. The more leftward the horizontal coordinate in this graph, the smaller the ratio of $Minpts$ to $Ep{s}^2$. It can be found that as the ratio rises, the curves of SSDD and HRSID will first rise to a certain value and then start to fluctuate in a small range, while the curve of LS-SSDD starts to fall after fluctuating for a period of time. According to the trend of the curve changes in Figure 8, the combination used for the experiments on SSDD is $Eps=5$ and $MinPts=5$, while the combination of $Eps=7$ and $MinPts=7$ is used for the experiments on HRSID. And for the LS-SSDD, we choose the combination of $Eps=6$ and $MinPts=5$. Using these parameters, the average IoUs obtained on SSDD, LS-SSDD, and HRSID are 0.754, 0.463, and 0.632.

Figure 9 is the visualization of the clustering results on some images in the training datasets. For the image of ships on the sea, the filtered points are all on the ships. Even for a complex background as in the image of ships near the shore, our approach filtered out most noisy points.

5.6. Time Analysis

We test the time required to represent and combine the prior knowledge individually.

Table 4. Time consumption for representing the prior knowledge.

Operation	Total Time Spent	Average Time Spent (Per Image)	FPS
brightness	71.3828 s	0.0879 s	11.375
density	217.8586 s	0.2683 s	3.727
the complete process	289.2406 s	0.1781 s	5.615

and

Table 5. Time consumption of combining the prior knowledge with SSD.

Operation	Total Time Spent	Average Time Spent (Per Image)	FPS
brightness	7.2804 s	0.00897 s	111.532
density	7.0043 s	0.0086 s	115.929
the prior knowledge map	2.1932 s	0.0027 s	370.233
the complete process	18.0784 s	0.0223 s	44.916

detail the total time spent, the average time spent per image, and the FPS for the different programs in the two phases. The images used to test the time performance are from the training set of SSDD, with a total of 812 images.

Representing the prior knowledge includes calculating metrics (e.g., IoU metrics of DBSCAN) for better hyper-parameter selection, which is approximately eight times slower than combining the prior knowledge. However, because the hyper-parameters are preserved during training and inference of models, the heavy cost is one-off and is the disposal in time complexity. Table 5 demonstrates the complete elapsed time for generating and representing prior knowledge as well as the elapsed time for each component during training and inference. The complete program takes an average of 0.0223 seconds to process an image on the GPU and achieves an FPS of over 44.

5.7. Qualitative Results

To prove the validity of BDAM, we add it to four popular detectors (Faster R-CNN, YOLOv4, RetinaNet, and SSD) for experiments and compare it with the original algorithms. In addition, we compare the effects with attention mechanisms EAM [17], BCA [93], and CBAM [67]. The first two are specially designed for SAR images while the last one is one of the most popular for general purpose.

Table 6. Experimental results of different algorithms on dataset SSDD. Bolded data represent the best and underlined data represent the second best.

Algorithm	$\mathit{AP}$	${\mathit{AP}}_{50}$	${\mathit{AP}}_{75}$	${\mathit{AP}}_{\mathit{S}}$	${\mathit{AP}}_{\mathit{M}}$	${\mathit{AP}}_{\mathit{L}}$
Faster R-CNN	0.659	0.959	0.776	0.638	0.700	0.678
+ BDAM	0.664	0.961	0.798	0.643	0.702	0.720
+ CBAM	0.655	0.959	0.781	0.634	0.693	0.681
+ EAM	0.657	0.961	0.776	0.635	0.692	0.678
+ BCA	0.660	0.961	0.762	0.639	0.701	0.663
RetinaNet	0.639	0.921	0.750	0.616	0.687	0.601
+ BDAM	0.641	0.942	0.762	0.621	0.681	0.678
+ CBAM	0.637	0.925	0.764	0.612	0.689	0.620
+ EAM	0.634	0.927	0.750	0.605	0.696	0.635
+ BCA	0.641	0.931	0.758	0.612	0.702	0.656
SSD	0.548	0.885	0.643	0.515	0.610	0.596
+ BDAM	0.555	0.891	0.651	0.520	0.626	0.598
+ CBAM	0.544	0.885	0.645	0.511	0.623	0.558
+ EAM	0.551	0.897	0.631	0.520	0.615	0.545
+ BCA	0.499	0.834	0.558	0.450	0.585	0.548
YOLOv4	0.499	0.937	0.472	0.453	0.633	0.637
+ BDAM	0.508	0.942	0.464	0.452	0.647	0.729
+ CBAM	0.507	0.958	0.464	0.457	0.639	0.657
+ EAM	0.504	0.940	0.485	0.452	0.644	0.675
+ BCA	0.505	0.952	0.452	0.454	0.632	0.713

,

Table 7. Experimental results of different algorithms on dataset HRSID. Bolded data represent the best and underlined data represent the second best.

Algorithm	$\mathit{AP}$	${\mathit{AP}}_{50}$	${\mathit{AP}}_{75}$	${\mathit{AP}}_{\mathit{S}}$	${\mathit{AP}}_{\mathit{M}}$	${\mathit{AP}}_{\mathit{L}}$
Faster R-CNN	0.626	0.866	0.716	0.502	0.768	0.599
+ BDAM	0.625	0.876	0.719	0.505	0.764	0.628
+ CBAM	0.625	0.875	0.712	0.505	0.761	0.612
+ EAM	0.621	0.866	0.710	0.500	0.763	0.615
+ BCA	0.613	0.855	0.707	0.493	0.749	0.618
RetinaNet	0.561	0.787	0.627	0.380	0.765	0.507
+ BDAM	0.567	0.796	0.633	0.387	0.764	0.566
+ CBAM	0.561	0.785	0.627	0.380	0.765	0.494
+ EAM	0.558	0.784	0.625	0.377	0.763	0.546
+ BCA	0.554	0.779	0.615	0.382	0.753	0.475
SSD	0.449	0.686	0.510	0.255	0.674	0.537
+ BDAM	0.457	0.694	0.518	0.257	0.684	0.539
+ CBAM	0.454	0.690	0.513	0.259	0.683	0.528
+ EAM	0.453	0.691	0.506	0.250	0.683	0.538
+ BCA	0.437	0.666	0.498	0.246	0.662	0.421
YOLOv4	0.550	0.914	0.603	0.395	0.686	0.607
+ BDAM	0.552	0.920	0.618	0.389	0.702	0.650
+ CBAM	0.550	0.912	0.610	0.377	0.690	0.621
+ EAM	0.540	0.920	0.578	0.376	0.680	0.619
+ BCA	0.545	0.912	0.600	0.382	0.690	0.616

and

Table 8. Experimental results of different algorithms on dataset LS-SSDD. LS-SSDD does not contain large ships, so the metrics of $A{P}_L$ do not apply to it. Bolded data represent the best and underlined data represent the second best.

Algorithm	$\mathit{A}\mathit{P}$	$\mathit{A}{\mathit{P}}_{50}$	$\mathit{A}{\mathit{P}}_{75}$	$\mathit{A}{\mathit{P}}_{\mathit{S}}$	$\mathit{A}{\mathit{P}}_{\mathit{M}}$
Faster R-CNN	0.258	0.722	0.073	0.255	0.339
+ BDAM	0.262	0.741	0.093	0.258	0.352
+ CBAM	0.258	0.722	0.087	0.255	0.342
+ EAM	0.257	0.727	0.093	0.250	0.346
+ BCA	0.235	0.675	0.070	0.230	0.352
RetinaNet	0.213	0.607	0.072	0.206	0.363
+ BDAM	0.223	0.638	0.075	0.219	0.352
+ CBAM	0.211	0.599	0.069	0.204	0.385
+ EAM	0.218	0.632	0.063	0.214	0.333
+ BCA	0.207	0.598	0.058	0.203	0.324
SSD	0.158	0.488	0.045	0.148	0.325
+ BDAM	0.169	0.517	0.045	0.160	0.343
+ CBAM	0.162	0.517	0.042	0.157	0.298
+ EAM	0.139	0.476	0.030	0.133	0.301
+ BCA	0.052	0.210	0.006	0.052	0.122
YOLOv4	0.284	0.819	0.092	0.280	0.402
+ BDAM	0.287	0.830	0.087	0.284	0.408
+ CBAM	0.284	0.826	0.090	0.280	0.405
+ EAM	0.273	0.809	0.088	0.270	0.408
+ BCA	0.256	0.789	0.074	0.252	0.400

show the performance of these four algorithms on the three datasets, where the bold data represent the one with the best results for a certain metric of the same algorithm, and the underlined data represent the one with the second best results. Figure 10 and Figure 11 show the recognition results of some images.

As illustrated in Table 6, Table 7 and Table 8, BDAM performs well with all detectors on all three datasets. Namely, the BDAM achieves 10 winners and 2 runners-up with Faster R-CNN; 9 winners and 2 runners-up with RetinaNet; 10 winners and 2 runners-up with SSD; and 7 winners and 2 runners-up with RetinaNet. We can notice that all baselines have a stable improvement on the detection ability of large targets after adding BDAM. For example, on the SSDD data, RetinaNet’s $A{P}_L$ improves by 7.7% after adding BDAM while it improves by 9.2% on the HRSID dataset, which is a significant improvement.

Table 6 and Table 7 reveal that performance of each method on $A{P}_M$ is better than that of $A{P}_S$ and $A{P}_L$. Although there is no large ship in Table 8, the performance of $A{P}_M$ is better than $A{P}_S$. In addition, from the three tables, we observe that the improvement of our model is insignificant on $A{P}_M$. It indicates that the detectors we used learned features well on medium size targets. BDAM could not improve detectors further if sufficient information has been utilized.

On SSDD (Table 6), BDAM combined with YOLOv4 underperforms other combinations. Unlike other detectors, YOLOv4 uses Mosaic as one of its data augmentation strategies. Mosaic extends CutMix by mixing four images at a time as one training image, which aims to enrich the background. However, the reality images are likely to contain no objects, which is equivalent to adding a lot of noise to the training process of the model. Adding appropriate noise to the training process can be beneficial to prevent the model from overfitting. However, it hinders BDAM. If an image does not contain objects, then the prior knowledge map obtained by clustering is likely to be an all-0 image. It makes BDAM not work well but increases the training burden of the model. Therefore, we follow the YOLOX [47] approach. Instead of using Mosaic in all epochs, we only use Mosaic in the first 70% epochs. However, limited by the amount of data, BDAM still does not perform as well on SSDD as it does on HRSID.

5.8. Ablation Study

5.8.1. Insertion Strategy

The insertion of BDAM can be divided into two ways, one is inserted between the backbone’s stages, and the other is inserted into the backbone’s blocks. BDAM does not fit into the second scheme. There is a big difference between BDAM and CBAM: the input of spatial attention in BDAM is the prior knowledge map while the input of CBAM is the output of the upper convolutional layer. If we want to insert BDAM into the block, it is not as easy as CBAM, and we have to solve the input problem for each block. For one input image, only one prior knowledge map can be generated. It means that we need to generate multiple weight maps of different sizes from a single prior knowledge map, which sounds like feeding a single image into a convolutional neural network to obtain a multi-scale feature map. However, the prior knowledge map is different from images, which are obtained by clustering based on our prior knowledge, and it contains certain interpretability itself. However, if it is fed into a very deep network to extract weighted graphs at multiple scales, we cannot guarantee whether the knowledge recorded on the prior knowledge map can be retained. Moreover, the insertion of BDAM into the block causes the effect of pre-trained parameters to be affected. According to He et al.’s study [94] on pre-trained parameters, the amount of data in HRSID and SSDD does not support us to train from scratch. The data in

Table 9. Performance of insertion in block and insertion in stage on Faster R-CNN. Bolded data represent the best and underlined data represent the second best.

Algorithm	$\mathit{AP}$	${\mathit{AP}}_{50}$	${\mathit{AP}}_{75}$	${\mathit{AP}}_{\mathit{S}}$	${\mathit{AP}}_{\mathit{M}}$	${\mathit{AP}}_{\mathit{L}}$
+ BDAM	0.664	0.961	0.798	0.643	0.702	0.720
+ BDAM *	0.427	0.727	0.453	0.516	0.280	0.166

confirm our concerns, where the BDAM with * represents the insertion into the block.

If we take the first way, since the size of the feature map output from each stage is different, we can change the problem to which size of the feature map we should adjust with BDAM. We conduct experiments on SSDD using Faster R-CNN. The backbone used in the experiments is ResNet50, and the results of the experiments are shown in

Table 10. Experimental results of different size weighting maps. Bolded data represent the best and underlined data represent the second best.

Algorithm	$\mathit{AP}$	${\mathit{AP}}_{50}$	${\mathit{AP}}_{75}$	${\mathit{AP}}_{\mathit{S}}$	${\mathit{AP}}_{\mathit{M}}$	${\mathit{AP}}_{\mathit{L}}$
Faster R-CNN	0.659	0.959	0.776	0.638	0.700	0.678
Input Size/4	0.659	0.959	0.779	0.639	0.696	0.663
Input Size/16	0.661	0.969	0.790	0.639	0.709	0.676
Input Size/64	0.664	0.961	0.798	0.643	0.702	0.720
Input Size/256	0.661	0.959	0.779	0.634	0.713	0.681
Input Size/1024	0.659	0.960	0.774	0.638	0.705	0.664

.

According to the experimental results, we find that using BDAM to adjust the feature maps that are reduced by a factor of 8, each relative to the initial input length and width, works best. That is, it is better to insert BDAM between the second stage and the third stage.

5.8.2. Attention Components

BDAM is a hybrid attention mechanism, in which the channel attention mechanism follows the implementation of CBAM. Our innovation and work focus on the spatial attention mechanism of BDAM. We call our spatial attention mechanism Brightness and Density Spatial Attention Mechanism (BDSM). Previous comparative experiments can only prove the effectiveness of BDAM, but are not enough to prove the effectiveness of BDSM.

To demonstrate the effectiveness of BDSM, we design the ablation experiments. The accuracy of the four algorithms with only the channel attention mechanism, only BDAM, and only BDSM is compared. In addition, spatial attention in CBAM uses the feature map output at the upper layer of the feature extraction network as input, which is very different from BDSM. Therefore, we also compared it with CBAM in the ablation experiment to demonstrate the effect of the prior knowledge map.

Table 11. Comparison of BDSM with different attention mechanisms. Bolded data represent the best and underlined data represent the second best.

Algorithm	$\mathit{AP}$	${\mathit{AP}}_{50}$	${\mathit{AP}}_{75}$	${\mathit{AP}}_{\mathit{S}}$	${\mathit{AP}}_{\mathit{M}}$	${\mathit{AP}}_{\mathit{L}}$
Faster R-CNN	0.626	0.866	0.716	0.502	0.768	0.599
+ BDAM	0.625	0.876	0.719	0.505	0.764	0.628
+ BDSM	0.626	0.868	0.716	0.506	0.762	0.678
+ CAM	0.622	0.856	0.713	0.500	0.763	0.622
+ CBAM	0.625	0.875	0.712	0.505	0.761	0.612
RetinaNet	0.561	0.787	0.627	0.380	0.765	0.507
+ BDAM	0.565	0.795	0.630	0.387	0.766	0.563
+ BDSM	0.567	0.796	0.633	0.385	0.767	0.558
+ CAM	0.551	0.777	0.624	0.370	0.752	0.525
+ CBAM	0.561	0.785	0.627	0.380	0.765	0.494
SSD	0.449	0.686	0.510	0.255	0.674	0.537
+ BDAM	0.457	0.694	0.518	0.257	0.684	0.539
+ BDSM	0.457	0.693	0.518	0.261	0.682	0.536
+ CAM	0.454	0.692	0.509	0.255	0.683	0.526
+ CBAM	0.454	0.690	0.513	0.259	0.683	0.528
YOLOv4	0.550	0.914	0.603	0.395	0.686	0.607
+ BDAM	0.552	0.920	0.618	0.389	0.702	0.650
+ BDSM	0.556	0.920	0.622	0.399	0.695	0.641
+ CAM	0.549	0.908	0.599	0.389	0.687	0.565
+ CBAM	0.550	0.912	0.610	0.377	0.690	0.621

presents the results of the ablation experiments. The only dataset we used in the ablation experiment is HRSID.

Comparing BDSM with BDAM, under four baselines and six evaluation criteria, BDAM wins a total of 12 winners and 9 runners-up, while BDSM wins a total of 11 winners and 8 runners-up. BDSM performs similarly to BDAM, and even slightly better than BDAM occasionally. For example, BDSM achieves 4 winners and 2 runner-ups in RetinaNet, and 3 winners and 1 runner-up in $AP$ metrics. BDAM adds CAM to BDSM, but BDAM has a humble improvement over BDSM. It indicates that BDSM plays a much larger role than CAM in BDAM.

Comparing CAM with BDAM, the effect of adding CAM is worse than adding BDAM for all metrics of different baselines. BDAM adds BDSM to CAM, and its effect has been greatly improved. This suggests the effectiveness of BDSM. The difference between BDAM and CBAM is the different spatial attention mechanisms, and BDAM beats CBAM in all metrics and baselines. This also shows that BDAM’s spatial attention (BDSM) is superior to CBAM’s spatial attention.

From the above comparative experiments, we believe that BDSM is effective and plays a major role in BDAM.

6. Discussion

Overall, our paper establishes a simple way of integrating prior knowledge into the CNN-based detectors by attention mechanisms. Through analyzing the ships in SAR images, we discovered that clustering extracts the prior knowledge easily. From the extensive experiments, we demonstrate that BDAM could be successfully applied to various CNN-based detectors easily. For example, on the SSDD data, RetinaNet’s $A{P}_L$ improves by 7.7% after adding BDAM while it improves by 9.2% on the HRSID dataset, which is a significant improvement. Furthermore, the ablation study implies that the most important component in the BDAM is BDSM.

However, the limitation of the BDAM is obvious. As a proof-of-concept, our prior knowledge is constructed by the fact that backgrounds in most images are shores or water surfaces for ship detection. The task is hard to escalate into general object detection in SAR images.