Inshore Ship Detection Based on Multi-Modality Saliency for Synthetic Aperture Radar Images

Abstract

Synthetic aperture radar (SAR) ship detection is of significant importance in military and commercial applications. However, a high similarity in intensity and spatial distribution of scattering characteristics between the ship target and harbor facilities, along with a fuzzy sea-land boundary due to the strong speckle noise, result in a low detection accuracy and high false alarm rate for SAR ship detection with complex inshore scenes. In this paper, a new inshore ship detection method based on multi-modality saliency is proposed to overcome these challenges. Four saliency maps are established from different perspectives: an ocean-buffer saliency map (OBSM) outlining more accurate coastline under speckle noises; a local stability saliency map (LSSM) addressing pixel spatial distribution; a super-pixel saliency map (SPSM) extracting critical region-based features for inshore ship detection; and an intensity saliency map (ISM) to highlight target pixels with intensity distribution. By combining these saliency maps, ship targets in complex inshore scenes can be successfully detected. The method provides a novel interdisciplinary perspective (surface metrology) for SAR image segmentation, discovers the difference in spatial characteristics of SAR image elements, and proposes a novel robust CFAR procedure for background clutter fitting. Experiments on a public SAR ship detection dataset (SSDD) shows that our method achieves excellent detection performance, with a low false alarm rate, in offshore scenes, inshore scenes, inshore scenes with confusing metallic port facilities, and large-scale scenes. The results outperform several widely used methods, such as CFAR-based methods and super-pixel methods.

1. Introduction

Ship detection in synthetic aperture radar (SAR) images has significant implications for various military and commercial domains, such as maritime surveillance and military reconnaissance [1]. Despite the considerable advances in ship detection techniques over the years, most existing methods are tailored to offshore ship detection, i.e., ships in a marine area, and have limited applicability to inshore ship detection [2]. Compared to offshore ship detection, inshore ship detection poses several challenges. First, the background consists of both ocean and harbor/land region, which complicates the estimation of background clutter. Second, ships and harbor facilities have similar scattering characteristics in terms of intensity and spatial distribution, resulting in high false alarms. Third, land/dock area and ships may be connected, leading to low detection accuracy. Last, the contour and edge features are difficult to extract accurately due to the intrinsic multiplicative speckle noise [3]. Therefore, SAR ship detection for inshore scenes still represents the most challenging scenario in SAR target detection.

The most fundamental feature in SAR target detection is the amplitude; as a result, the constant false alarm rate (CFAR) detection method, which lays a particular emphasis on the amplitude feature, has become the most widely used algorithm for SAR ship detection [4]. The CFAR detection methods model the statistical distribution of the background clutters to get an adaptive threshold, and then search for pixels that are unusually bright compared to the surrounding background clutters [5]. It typically achieves good performance in offshore scenes since ships tend to appear as small bright targets with discernible shape information [6]. However, for inshore scenes, the conditions are totally different, and several problems appear for the traditional CFAR detection. The CFAR methods focus on pixel level and cannot identify whether adjacent pixels belong to a target or clutter regions. Therefore, in most cases, it requires further morphological procedures to cluster pixels [7]. For inshore ship detection, this morphological clustering will connect ship pixels with strong land clutters, resulting in false negative detections. Another problem mainly arises in scenes with large targets or large parts of high intensity land clutters. The distribution weight of high intensities is so large that the statistical fitting for background clutters tends to exhibit excess kurtosis and a long tail. This inaccurate fitting usually gives a higher intensity threshold, resulting in the loss of detected ship structures such as holes and fractures [2,8,9,10].

Motivated by the target pixel clustering requirements, some researchers employ the super-pixel (SP) method to overcome the inshore ship detection challenge. The SP methods typically returns a set of locally coherent pixels, which benefits the extraction of region-based features [11,12]. The simple linear iterative clustering (SLIC) segmentation method [13] and the maximally stable extremal regions (MSER) method [14] are the two most widely used SP methods in SAR target detection. SLIC adopts k-means clustering to generate SPs, which cover the whole image region, whether it is the ship target or background clutter. Therefore, the extracted SP features have to be labeled by a CFAR detector or some prior knowledge. The main problem of SLIC is the inherent over-segmentation, leading to excessive computational resources allocated for quantities of background clutter SPs (BCSPs), while the ship target SPs (STSPs) are much fewer [8]. In addition, the number of total SPs is normally predetermined in SLIC, so when there exists strong noise, the target region cannot be outlined accurately, and the achieved SPs will include both true target region and the surrounding speckle noise [2,8].

The MSER method is another category of SP methods, which defines the feature blobs by the extremal property of the intensity function in the blob region and on its outer boundary. The MSER method can return a more accurate contour of the target region, even with strong speckle noise, and typically finds only the bright regions, which avoids the over-segmentation. However, a batch of nested MSER regions may be generated in a specific area since there are noise and numbers of local height extrema in real SAR images [15]. In addition, the extracted regions of harbor facilities and ship targets are quite similar, and the absence of background clutter patches makes it impossible to exclude false alarms by comparison. Therefore, such approaches often also require accurate and robust sea–land segmentation. However, current sea–land segmentation approaches are generally designed based on an amplitude segmentation method, such as the OTSU method, and are quite sensitive to dense speckle noise and strong sea clutter [2,16,17], resulting in inaccurate sea–land boundaries.

Motivated by the successful application of the human attention theory in optical remote sensing images [18], saliency analysis has been introduced to SAR target detection [19,20]. The commonly used saliency in SAR detection is considered as a perceptual feature that describes the prominence of an object or location compared to its contextual surroundings and easily attracts visual attention [21] for small and weak target detection. For inshore ship detection, the key challenge is the false alarm in harbor region and inaccurate sea–land segmentation instead of small and weak targets. As a result, the common saliency is not suitable for our problem. In this paper, a broader saliency is defined to alleviate the complexity caused by dense speckle noises, harbor facilities, and strong sea–land clutters for more accurate sea–land segmentation and a lower false alarm rate.

With the rapid development of deep learning, the convolution neural network (CNN) based methods are gradually used in SAR ship detection [22,23,24,25,26,27]. The data-driven deep learning approaches can build an end-to-end system to learn the hierarchical features automatically and detect ship targets simultaneously without human intervention and have realized state-of-the-art detection performance on widely used benchmark datasets. However, such performance usually requires a large amount of data, and the fundamental assumption that training and test data are identically and independently distributed (a.k.a. i.i.d. assumption) [28]. In most datasets, the inshore scenario is relatively scarce compared to the offshore scenario, making it harder to satisfy the i.i.d. assumption. Some studies found that the CNN based detectors tend to be weaker on inshore ship detection because the features extracted by convolution are easily affected by noise. In some extreme cases, changing one pixel can lead to totally different detection results [29]. Additional constraints are usually required to achieve reliable predictions under non i.i.d. assumption, but the prediction of most current deep models is short of a physical explanation. In consideration of the special physical characteristics underlying SAR images, it is promising to combine the data-driven CNN methods with traditional model-driven methods to enhance the interpretability and generalization in the future [30]. Therefore, a reliable model-driven method can still be a boost and supplement for the future development of CNN based models.

In this paper, we propose a multi-modality saliency (MMS) method for ship detection in SAR images of complex inshore scenes with similar port facilities. The method extracts four types of modality information from the images: the MSER feature, the scale feature, the spatial distribution feature, and the intensity distribution feature. Each type of feature is converted into a saliency map: the super-pixel saliency map (SPSM), the ocean-buffer saliency map (OBSM), the local stability saliency map (LSSM), and the intensity saliency map (ISM). The method combines these saliency maps to detect ships with accurate contours and exclude false alarms with low stability, weak intensity, or located in land regions. The main contributions of this paper are as follows:

• This paper provides a novel interdisciplinary perspective of surface metrology for SAR image segmentation by addressing the scale difference of the main elements of SAR images with the suitable usage of a second-order Gaussian regression filter.
• This paper defines the spatial characteristics of SAR image elements as the local stability calculated by the moving standard deviation. It discovers distinct differences in the local stability between ships, sea regions, land regions, and connections between the sea and other elements, which demonstrates the detection effectiveness of the proposed local stability.
• This paper proposes a novel robust constant false alarm rate (CFAR) procedure that can eliminate noise and large ships’ effects on background clutters’ statistical fitting. This procedure can also be widely used for fitting other distributions.

2. Related Works

In the following, we review several bodies of literature that are relevant to the objective of our paper.

2.1. Constant False Alarm Rate (CFAR) Based Method

Some studies take CFAR detection as a preprocessing procedure, and adopt further refinement procedures to exclude false alarms, e.g., Ao and Xu used a Gamma distribution based CFAR detector to refine the land–sea segmentation from the global 250 m water mask database, then used the Eigen–ellipse to exclude non-ship objects [31]. Wang et al. applied a diagonal ship–island ratio enhance operator to the CFAR detected image to weaken the non-ship pixel clusters, and a morphological erosion can be further applied to eliminate the island elements [32]. Other studies focused on improving the fitting ability of CFAR, e.g., Leng et al. proposed a bilateral CFAR method to reduce the influence of SAR ambiguities and sea clutter, by means of a combination of the intensity distribution and the spatial distribution of SAR images [4]. Tao et al. employed a mixture of Gamma distribution with truncated statistics to fit inhomogeneous background clutters [9]. In our work, we propose a robust CFAR approach to lessen the influence of high intensity land clutters and large targets.

2.2. Super-Pixel (SP) Method

One crucial point for the SP method is how to locate the fewest ship target SPs (STSPs) among the numerous over-segmented results. Wang et al. proposed a density feature to locate STSPs [7]. Liu et al. proposed a weighted information entropy feature to discriminate STSPs [8]. Wang et al. proposed a fluctuation feature to discriminate STSPs [2]. Lin et al. designed a Fisher feature vector at the super-pixel level, which can better describe the feature differences between the STSPs and BCSPs [33]. However, these features address poor discriminative ability when the intensity and spatial distributions of ship and harbor facility pixels are similar. In addition, their results show that the SLIC method cannot accurately outline the target region when strong noise exists. In our work, we employ the MSER method to avoid over-segmentation and propose a nested feature removal method to reduce the consumption of computational resources.

2.3. Saliency

The method based on visual saliency simulates the visual information processing mechanism in nature and designs multiple filters to decompose SAR images to obtain features such as target grayscale, direction, and texture. This method first extracts the salient regions of the whole scene, then studies the spatial relationship of local features, selects some relatively optimized features to retain local salient pixels, and finally constructs the visual saliency map of the scene. Cui et al. designed a new saliency detector for polarized SAR ship detection through similarity testing [34], while Yang et al. designed a curvature-based saliency detector based on the microstructural characteristics of statistical manifolds [35]. There are also studies using saliency on inshore ship detection [3,36], but it seemed that methods based on singular saliency result in port facilities that are visually such as ships being difficult to distinguish, resulting in limited detection accuracy of complex scenes. The proposed multi-modality saliency method in this article is mainly inspired by the concept of saliency.

2.4. Convolutional Neural Network (CNN)

Although the learned features are more abstract and difficult for humans to understand, the CNN based methods have attracted more and more attention due to their significant advantages such as higher accuracy, faster speed, and simpler design processes. Qian et al. proposed a novel object detection method based on improved bounding box regression and multi-level features fusion to improve the precision of object localization [25]. An et al. proposed a DrBox-v2 detector, with a multi-layer prior box generation strategy for small scale targets, a modified encoding scheme for more precisely estimating the position and orientation of targets, and a focal loss (FL) combined with hard negative mining (HNM) technique to mitigate the issue of the imbalance between positive and negative samples [37]. Chen et al. designed a deep neural network based on an attention mechanism, using GloU loss to detect ships in multi-scale and complex scenes [38,39]. As we mentioned before, the CNN based detection methods are outside of the scope of this article, but the proposed MMS method can be a boost and supplement for the future CNN based methods.

3. Materials and Methods

Figure 1 shows the workflow of our method. We use four modalities to detect ships in SAR images. First, we extract non-overlapping regions with a non-nested MSER procedure and form a super-pixel saliency map (SPSM), where the region features are not labeled and usually contain quantities of false alarms. Second, we apply an improved robust K-CFAR detector to obtain an intensity saliency map (ISM) and use it to filter out false alarms from SPSM. This gives us the stage I results. Third, we calculate the local stability of each region with a moving standard deviation (σ) and create a local stability saliency map (LSSM). We use LSSM to refine the stage I results to obtain the stage II results. Fourth, we filter out low frequency components with a second order Gaussian regression filter and segment the image with a four-level OTSU method. We then dilate the lowest level to get an ocean-buffer saliency map (OBSM), which excludes targets near the coast from the stage II results. This gives us the final ship detection results. In real time application, these four saliency processes can be implemented in parallel. We arranged the workflow in Figure 1 for a better illustration.

The proposed SPSM, OBSM, LSSM, and ISM are introduced in detail below.

3.1. Super-Pixel Saliency Map (SPSM)

To reduce the computational cost of over-segmentation by the SLIC method, we employ the MSER method for extracting super-pixel region features [14]. The MSER method scans the intensity range [0, 255] of the input SAR image incrementally and detects region features that have relatively stable areas across different intensity levels. The intensity level increases by a constant step size. The MSER method constructs a series of binary cross section images based on the increasing intensity level. For each intensity level, $h$, the image pixels are assigned ‘1’ if their intensity is lower than or equal to $h$, and ‘0’ otherwise. Each connected region of ‘1’ pixels is called an extremal region. As $h$ increases, new extremal regions appear, and the existing ones expand and merge with each other. When $h$ reaches the maximum intensity of 255, there is only one extremal region that covers the whole image. The MSER method tracks both the area and the area growth rate of the nested extremal regions. When two regions merge, the tracking of the smaller one terminates. For a given region, the MSER method measures the variation of its area size between adjacent intensity thresholds. If this variation is below a threshold value, then this region area is regarded as a maximally stable extremal region (MSER). In our application, the intensity step is set as 2, and the growth rate threshold is set as 0.2.

Real SAR images contain many local intensity peaks and valleys. Therefore, for a given region, the MSER detector will produce a set of nested MSER region features due to its intrinsic extremal property. Figure 2 illustrates the nested feature removal process. Figure 2a displays an original SAR image, and Figure 2b shows the MSER extraction result of Figure 2a. Figure 2c presents the non-nested MSER regions of Figure 2a, which are collectively regarded as our SPSM. The nested feature removal method is proposed as follows:

(a)

Get the gravity center, $G{C}_i$, of each MSER region, $M{R}_i$, where $i\in \left[1,2,\dots ,N\right]$, and $N$ is the total number of the extracted MSER features.

(b)

For any two gravity centers, $G{C}_i$ and $G{C}_j$, if $\left|G{C}_i-G{C}_j\right|<20$ pixels, the smaller one of $M{R}_i$ and $M{R}_j$ is eliminated.

(c)

Repeat the above procedures until all the remaining gravity centers are more than 20 pixels apart from each other.

3.2. Intensity Saliency Map (ISM)

Pixel intensity is the fundamental characteristics of SAR images, and CFAR detection methods model the statistical distribution of the background clutters to obtain an adaptive intensity threshold. However, the threshold relies heavily on the statistical fitting accuracy for the background clutters. In inhomogeneous scenes, the intensity histogram tends to exhibit excess kurtosis and a long tail [2], which require an asymmetric (or non-gaussian) distribution for optimal fitting. In the existing literature, some parametric models have been proposed to fit the long heavy tail feature of a heterogeneous clutter, such as the K distribution [40]. However, the modeling ability of a singular parametric model is doubtable for inshore scene probability density function (PDF), and the computational cost is too high for mixture parametric models [9]. As a result, here, we chose the nonparametric kernel density estimator (KDE) to model the PDF of inshore scene clutters. The KDE is given by [41]:

$$f_h\left(x\right)=\frac{1}{n}{\displaystyle \sum }_{j=1}^n\frac{1}{h}K(\frac{x-x_j}{h})$$

(1)

where $x_1,x_2,\dots ,x_n$ are $n$ identically and independently distributed samples forming an unknown PDF of inshore scene clutters. $h$ is the bandwidth that indicates the width of kernel function. $K$ is a kernel function satisfying ${{\displaystyle \int }}_{-\infty }^{+\infty }K\left(x\right)dx=1$. In our work, a Gaussian kernel function is employed:

$$f_h\left(x\right)=\frac{1}{nh\sqrt{2\pi }}\sum \exp (-\frac{{\left(x-x_j\right)}^2}{2h^2})$$

(2)

where in our case, Gaussian basis functions are used to approximate univariate data, the optimal choice for bandwidth $h$, which is also the bandwidth that minimizes the mean integrated squared error [42]:

$$h={(\frac{4{\widehat{\sigma }}^5}{3n})}^{\frac{1}{5}}\approx 1.06\widehat{\sigma }{n}^{-\frac{1}{5}}$$

(3)

where $\widehat{\sigma }$ is the estimated standard deviation of $x_1,x_2,\dots ,x_n$. Then, the corresponding CDF is given by:

$$F\left(x\right)=\frac{1}{n}{\displaystyle \sum }_{j=1}^n\frac{1}{h}\mathsf{\Phi }(\frac{x-x_j}{h})$$

(4)

where $\mathsf{\Phi }$ is the CDF of standard normal distribution. Given the value of the global false alarm rate, Pfa, the threshold, $T$, can be calculated from:

$$Pfa=1-\frac{1}{n}{\displaystyle \sum }_{j=1}^n\frac{1}{h}\mathsf{\Phi }(\frac{T-x_j}{h})$$

(5)

However, when applying the KDE to SAR images with big targets and large area land clutters, as shown in Figure 3a, the PDF generated by KDE (green line in Figure 3b) shows overfitting to outliers in the high intensity edge, resulting in a higher threshold $T$ and increasing the risk of missed detections (as shown in Figure 4a) under the same global false alarm rate Pfa (in this case, we set Pfa at 0.005). We found in Figure 3b that, although KDE overfits the high intensity edge, the curve does not follow the histogram well; there are outliers in the residual between the fitting curve and the histogram (yellow line in Figure 3b). Inspired by this phenomenon, we added a weighting function, $w$, to the samples, $x_1,x_2,\dots ,x_n$, to reduce the effect of large ships and land clutters. Considering both the convergence property and calculation speed, the Turkey estimator is introduced as the weighting function:

$$w\left(r\right)=\left\{\begin{array}{cc}\frac{c^2}{6}\left(1-{\left[1-{\left(r/c\right)}^2\right]}^2\right)& if \left|r\right|\le c\\ \frac{c^2}{6}& if \left|r\right|>c\end{array}\right.$$

(6)

where $r=h\left(x\right)-f\left(x\right)$, $h\left(x\right)$ is the histogram of $x_1,x_2,\dots ,x_n$, and $f\left(x\right)$ is the PDF of KDE. $c^{\left(i\right)}=4.4478\times meidan\left(\left|r^{\left(i\right)}\right|\right)$, where $i$ is the current number of iterations. If $\left|c^{\left(i\right)}-c^{\left(i-1\right)}\right|<\left|c^{\left(i-1\right)}\right|\epsilon$, and $\epsilon ={10}^{-3}$, the iteration ends, and $f\left(x\right)$ is the robust PDF of KDE fitting. The red line in Figure 3b shows the robust KDE PDF; it is clear that there is no overfitting to ship targets and strong land clutters. Figure 4b shows the detection result by a robust KDE CFAR detector; as can be seen, more details of ship targets are retained in Figure 4b, and the result is termed as our ISM. No matter how well the model fits the background clutters, there are still false alarms detected in land regions, so simple intensity-based saliency may be not enough for inshore scene target detection.

3.3. Local Stability Saliency Map (LSSM)

Ships are distinct from the ocean or land not only in the intensity domain, but also in the spatial domain. Bright pixels of ships are often contiguous and concentrated in a small area, while those of the background are relatively discrete and unstable. Elements of a SAR image typically exhibit the following difference in spatial characteristics: (a) ships appear as bright, concentrated pixels, with the highest local stability; (b) bright pixels are distributed uniformly in the marine region, whose local stability is slightly lower than ships; (c) due to the large-scale ridges and landscapes, the local stability of the land region is lower than the marine region; and (d) the connections between the sea and other elements have the lowest local stability. Local stability can be measured by the standard deviation of the pixel intensities in a local region. For a vector $x$ with $N$ dimensions, its standard deviation, $\sigma$, is defined as follows:

$$\sigma =\sqrt{\frac{{{\displaystyle \sum }}_{i=1}^N{x}_i^2}{N}-{(\frac{{{\displaystyle \sum }}_{i=1}^N{x}_i}{N})}^2}$$

(7)

The calculation of the local standard deviation can be accelerated in the frequency domain: we construct a template image, $S_T$, with the same size as the SAR image, S, and set all its values to ‘1’. We also create a convolution kernel, $K$. Then, we compute the local mean matrix of pixel intensity, $\overline{S_K}$, by $\overline{S_K}=\left(K\ast S\right)/\left(K\ast S_T\right)$, and the local mean matrix of pixel intensity squared, $\overline{S_K^2}$, by $\overline{S_K^2}=\left(K\ast S^2\right)/\left(K\ast S_T\right)$, where $\ast$ denotes convolution. The local standard deviation matrix is then obtained by:

$$S_{\sigma }=\sqrt{\overline{S_K^2}-{(\overline{S_K})}^2}$$

(8)

The process of local stability is illustrated in Figure 5. Figure 5a displays an original SAR image, and Figure 5b shows its corresponding local standard deviation matrix. Figure 5c presents the histogram of Figure 5b, where the $\sigma$ value of the red rectangle indicates the central area of ships, the $\sigma$ value of the green rectangle represents the sea region, the $\sigma$ value of the blue rectangle signifies the land region, and the $\sigma$ value of the yellow rectangle marks the boundaries between sea and other elements. We define an adaptive threshold by detecting the breaking point between the red and green rectangles, which is usually less than 10. It can be seen in Figure 5d that, not only ship targes, but metallic harbor facilities are identified by the LSSM as well, so simple LSSM is not enough for inshore scene detection.

3.4. Ocean-Buffer Saliency Map (OBSM)

Ships can only appear in marine areas for ship detection in harbor regions. Therefore, we need to first segment the sea from the land. Previous studies often used intensity-based segmentation methods, such as Otsu’s method, with morphology operations to connect isolated land regions and eliminate small holes. However, these methods are sensitive to strong noise and sea clutter. We consider three main elements in SAR images: noise, ship targets, and landforms. These elements have different scales. We address this scale difference by using surface metrology methods [43]. We extract features with interested scales for further analysis by filtering them. We use the second order Gaussian regression filter [44] to avoid boundary effects and reduce noise, which can be defined by the following minimization problem:

$$\begin{array}{c}{{\displaystyle \int }}_0^{ly}{{\displaystyle \int }}_0^{lx}\rho \left(\begin{array}{c}{z}_0\left(\xi ,\eta \right)-z_f\left(x,y\right)\\ -{\beta }_{10}\left(x,y\right)\left(\xi -x\right)-{\beta }_{01}\left(x,y\right)\left(\eta -y\right)\\ -{\beta }_{20}\left(x,y\right){\left(\xi -x\right)}^2-{\beta }_{02}\left(x,y\right){\left(\eta -y\right)}^2\\ -{\beta }_{11}\left(x,y\right)\left(\xi -x\right)\left(\eta -y\right)\end{array}\right)s\left(\xi -x,\eta -y\right)d\xi d\eta \\ \Rightarrow mi{n}_{z_f\left(x,y\right),{\beta }_{10}\left(x,y\right),{\beta }_{01}\left(x,y\right),{\beta }_{20}\left(x,y\right),{\beta }_{02}\left(x,y\right),{\beta }_{11}\left(x,y\right)}\end{array}$$

(9)

We obtain the resulting filtration surface, $z_f$, by zeroing the partial derivatives in the directions of $z_f$, ${\beta }_{10}\left(x,y\right)$, ${\beta }_{01}\left(x,y\right)$, ${\beta }_{20}\left(x,y\right)$, ${\beta }_{11}\left(x,y\right)$, and ${\beta }_{02}\left(x,y\right)$. We define these terms in Equation (9) as follows:

$z_f$: the filtration result of the input SAR image $z_0$;

$x$ and $y$: spatial coordinates in two orthogonal directions.

$s\left(x,y\right)=\frac{1}{{\alpha }^2{\lambda }_{cx}{\lambda }_{cy}}exp\lceil -\pi {\left(\frac{x}{\alpha {\lambda }_{cx}}\right)}^2-\pi {\left(\frac{y}{\alpha {\lambda }_{cy}}\right)}^2\rceil$: Gaussian weighting function, where $\alpha =\sqrt{log\left(2\right)/\pi }$.

${\lambda }_{cx}$ and ${\lambda }_{cy}$: the cutoff wavelengths in $x$ and $y$ directions;

${\beta }_{10}\left(x,y\right)$ and ${\beta }_{01}\left(x,y\right)$: first-order coefficients;

${\beta }_{20}\left(x,y\right)$, ${\beta }_{11}\left(x,y\right)$, and ${\beta }_{02}\left(x,y\right)$: second-order coefficients;

$\rho \left(r\right)=r^2/2$: error metric function of the estimated residual.

Figure 6 and Figure 7 show the filtration results of SAR images in an inshore scene with different levels of speckle noise. We use the structural similarity index [45] to assess their difference. The structural similarity index between Figure 6a and Figure 7a is 56.7%. The structural similarity index between their filtration results is 76.3%. The increase indicates that using a second order Gaussian regression filter can reduce noise sensitivity.

To further identify the marine area, the four-level Otsu’s method is employed for adaptive thresholding based on the maximization of inter-class variance [46]. The filtered image, $z_f$, is first rescaled to the gray level range of 0~255 using the following formula:

$$\frac{z_f-\min \left(z_f\right)}{\max (z_f)-\min \left(z_f\right)}\times 255$$

(10)

The intensity level of the filtered image is now [0, 255], and $n_i$ denotes the number of pixels with intensity, $i$, where $i\in \left\{0,1,2,\dots , 255\right\}$. The probability of occurrence of a gray level $i$ is $p_i=n_i/{{\displaystyle \sum }}_{i=0}^{255}{n}_i$. For a four-level segmentation with three thresholds $\left[t_1,t_2,t_3\right]$, the total inter-class variance, $f$, can be expressed as:

$$f=\frac{{\left(\mu w_{t_1}-{\mu }_{t_1}\right)}^2}{w_{t_1}}+\dots +\frac{{\left({\mu }_{t_j}-\mu w_{t_j}+\mu w_{t_{j-1}}-{\mu }_{t_{j-1}}\right)}^2}{w_{t_j}-w_{t_{j-1}}}+\dots +\frac{{\left(\mu w_{t_3}-{\mu }_{t_3}\right)}^2}{1-w_{t_3}}$$

(11)

In this equation, $w_{t_j}$ represents the cumulative probability of the occurrence of the gray level $\left[0, 1,\dots ,t_j\right]$, and $w_{t_j}={{\displaystyle \sum }}_{i=0}^{t_j}{p}_i$. ${\mu }_{t_j}$ denotes the mean value at the gray level $\left[0,t_j\right]$, and ${\mu }_{t_j}={{\displaystyle \sum }}_{i=0}^{t_j}i·p_i$. The optimal thresholds $\left[t_1,t_2,t_3\right]$ that maximize Equation (11) can be obtained using the Nelder–Mead simplex method [47]. The lowest level segment with intensity values lower than $t_1$ corresponds to the marine area.

The process of sea–land segmentation is illustrated in Figure 8. Figure 8a displays the original SAR image, Figure 8b shows the segmentation result obtained by using the second order Gaussian regression filter and four-level Otsu’s method, and Figure 8c depicts the extracted marine area. Since ships only appear in or near the ocean, a buffer area is created by performing a morphological dilation operation with a disk-shaped structuring element whose radius is approximately equal to the width of ships. Figure 8d presents the resulting OBSM in blue where ships are located, while the red area indicates land.

4. Results

4.1. Implementation Details

Experiments were performed using SAR images with typical near-shore, inshore scenes from the SSDD dataset [48]. The hardware environment used was the Intel Core i7-6700HQ CPU at 2.60-GHZ and 16-GB RAM. The images were arranged into four groups: offshore heterogeneous scenes, inshore scenes without port facilities, inshore scenes with confusing port facilities, and large-scale scenes. The details of images in each group are listed in tables along with the detection result. The proposed MMS method is compared with the classic K-CFAR based method, Gamma-CFAR based method, and a SLIC-CFAR method. For K-CFAR and Gamma-CFAR based methods, the CFAR detectors are first applied to the input SAR image, then some prior knowledge, such as size and aspect ratio, are used to further eliminate non-ship objects. A morphological closing operation is also required since there are often holes remaining in the CFAR detection results. For the SLIC-CFAR method, SPs are first generated with the SLIC method, then a global detection is performed using weighted information entropy to extract interested SPs. These SPs are further selected by a local CFAR detection with neighboring SPs. A clustering operation is applied to the remaining foreground SPs, and finally, prior knowledge is used to further remove non-ship SPs.

The detection results are quantitatively evaluated by the F1 score. The F1 score is calculated by using three well established metrics in information retrieval, namely true positive (TP), false positive (FP), and false negative (FN). For inshore ship detection, shape distortion caused by adhesion between ships and land may affect the further identification process. Therefore, in this paper, a pixel-level F1 score such as the overlap rate between the detection result and ground truth is used, where TP denotes the number of pixels that belong to correctly detected ships, FP indicates the number of pixels that correspond to false alarms, and FN represents the number of pixels that are missed by the detection algorithm.

The recall measures the proportion of a correctly detected ship area in all ground truths, and is defined as:

$$recall=\frac{TP}{TP+FN}$$

(12)

In addition, the precision evaluates the proportion of the correctly detected ship area in all detection results:

$$precision=\frac{TP}{TP+FP}$$

(13)

The F1 score is used to assess the overall detection performance:

$$F1=\frac{2\times precision\times recall}{precision+recall}$$

(14)

4.2. Comparison with Other Methods

Figure 9 presents the detection results of four methods on offshore scenes with an inhomogeneous background, in which Figure 9a,b show the SAR image 1 and 2 that were tested for detection with the ground truth marked by a red polygon. The size and resolution of the two images were listed in

Table 1. Information of Image 1 and Image 2.

Image	Resolution (m)	Size (Pixels)
1	2	495 $\times$ 284
2	5	505 $\times$ 305

. Figure 9c,d show the detection results of the K-CFAR method; there are false alarms representing bright land ridges that remained because they resemble ships in shape and intensity. Figure 9e,f show a similar problem with the Gamma-CFAR method. Figure 9g shows less false alarms with the SLIC-CFAR method because super-pixels representing bright ridges relate to each other and can be excluded by size. However, in Figure 9h, a ship target is miss detected with the SLIC-CFAR method. Figure 9i,j show the detection result of the proposed MMS method, which could suppress the false alarm land clutters and retain good, detailed target characteristics, but some side slopes are also detected. The pixel-level F1 score describing the degree of the pixel level coincidence between the detection results and ground truth, along with the computing time, are listed in

Table 2. Pixel-level F1 score and computing time of the four methods on Image 1 and Image 2.

Methods	K-CFAR		Gamma-CFAR		SLIC-CFAR		MMS
	F1 (%)	Time (s)	F1 (%)	Time (s)	F1 (%)	Time (s)	F1 (%)	Time (s)
Image 1	24.28	0.06	19.07	0.08	34.11	0.61	78.02	0.40
Image 2	42.91	0.05	42.73	0.11	44.56	0.74	72.64	0.37

.

Figure 10 presents the detection results of the four methods on inshore scenes without port facilities, in which Figure 10a,b show the SAR image 3 and 4 that were tested for detection with the ground truth marked by a red polygon. The size and resolution of the two images were listed in

Table 3. Information for Image 3 and Image 4.

Image	Resolution (m)	Size (Pixels)
3	1	549 $\times$ 494
4	1	502 $\times$ 374

. Figure 10c shows false alarms and miss detected parts of the ship with the K-CFAR method because the long heavy tail feature of a heterogeneous clutter affects fitting, resulting in a high threshold. The same miss detections are found in Figure 10d. The detection results in Figure 10e,f with the Gamma-CFAR method are quite similar with those using the K-CFAR method. Figure 10g shows heavy false alarms and miss detections with the SLIC-CFAR method, while Figure 10h shows pretty good detection results because there are less confusing concentrated bright clutters in Image 4. Figure 10i,j show the detection result of the proposed MMS method, the ship contours are well outlined, and no false alarms are observed. Numerical results and computing time are listed in

Table 4. Pixel-level F1 score and computing time of the four methods on Image 3 and Image 4.

Methods	K-CFAR		Gamma-CFAR		SLIC-CFAR		MMS
	F1 (%)	Time (s)	F1 (%)	Time (s)	F1 (%)	Time (s)	F1 (%)	Time (s)
Image 3	37.95	0.09	37.95	0.07	35.90	1.27	79.56	0.16
Image 4	54.90	0.08	54.90	0.10	67.92	0.94	68.80	0.12

.

Figure 11 presents the detection results of the four methods on inshore scenes with confusing port facilities, in which Figure 11a–c show the SAR Images 5, 6, and 7 that were tested for detection with the ground truth marked by a red polygon. The size and resolution of the three images were listed in

Table 5. Information on Image 5, Image 6, and Image 7.

Image	Resolution (m)	Size (Pixels)
5	1	389 $\times$ 318
6	1	500 $\times$ 373
7	1	521 $\times$ 391

. Figure 11d–f show the detection results with the K-CFAR method. Both ships and confusing metallic port facilities are detected. The detection results in Figure 11g–i with the Gamma-CFAR method are quite similar to those using the K-CFAR method; the difference lies in the threshold value affected by different model fitting. Figure 11j–l show the detection results with the SLIC-CFAR method, in addition to false alarms, some parts of the metallic port facilities are missing in Figure 11j, and a whole target is miss detected in Figure 11l. The reason for the miss detection is that a local CFAR detector is applied to the detected SPs by comparing them with their neighboring SPs. When there are large bright structures in the image, the intensity of the neighboring SPs will be too high, resulting in miss detection of the central SPs. Figure 11m–o show the detection results of the proposed MMS method; all ship targets are detected with no false alarms (confusing port facilities), but there are some side lopes and land structures connected with the ship targets. Numerical results and computing time are listed in

Table 6. Pixel-level F1 score and computing time of the four methods on Image 5, Image 6, and Image 7.

Methods	K-CFAR		Gamma-CFAR		SLIC-CFAR		MMS
	F1 (%)	Time (s)	F1 (%)	Time (s)	F1 (%)	Time (s)	F1 (%)	Time (s)
Image 5	49.65	0.03	52.96	0.04	54.22	0.52	67.70	0.12
Image 6	37.88	0.07	35.98	0.03	46.92	0.82	75.73	0.53
Image 7	53.50	0.05	53.50	0.04	25.98	1.00	73.42	0.28

.

Figure 12 shows the detection results of the four methods on large scenes in which Figure 12a,b show the SAR Image 8 and 9 that were tested for detection with ground truth marked by a red polygon. For CFAR-based methods, the detection results show missing parts of the detected target ships with few false alarms, as shown in Figure 12c–f. The reason for the missing parts is due to the high threshold caused by the overfitting to land clutters, while few false alarms is because no confusing large port facilities exist in Image 8 and 9. For the SLIC-CFAR method, there are still missing targets, as shown in Figure 12g,h, due to the local CFAR procedure based on SPs. All ship targets are detected with no false alarms using the proposed MMS method, as shown in Figure 12i,j, while some land structures are connected with the detected targets in Figure 12i. The size and resolution of the three images were listed in

Table 7. Information for Image 8 and Image 9.

Image	Resolution (m)	Size (Pixels)
8	1	1074 $\times$ 1506
9	1	1023 $\times$ 1524

. Numerical results and computing time are listed in

Table 8. Pixel-level F1 score and computing time of the four methods on Image 8 and Image 9.

Methods	K-CFAR		Gamma-CFAR		SLIC-CFAR		MMS
	F1 (%)	Time (s)	F1 (%)	Time (s)	F1 (%)	Time (s)	F1 (%)	Time (s)
Image 8	43.82	0.15	46.18	0.09	48.83	17.32	71.39	0.66
Image 9	47.29	0.15	42.68	0.15	35.13	15.19	79.81	0.54

.

4.3. Ablation Tests

To further evaluate the concrete contributions of the saliency maps in our MMS method, we removed the OBSM, SPSM, and LSSM once each time and checked the performances of the degraded MMS methods. We thought it better to conduct the ablation tests on images with different scenes that other methods performed not well, so we chose Image 1 (offshore scene), Image 3 (inshore scene without port facilities), and Image 7 (inshore scene with confusing port facilities). The ISM is used as the backbone since it addresses the fundamental characteristics of pixel intensity, which is key to SAR ship detection; therefore, ISM is not included in the ablation test. Figure 13 shows the ablation tests, in which Figure 13d–f show the detection results without SPSM. The SPSM addresses the clustering characteristics of detected pixels, ships are often connected with other facilities, as shown in Figure 13f, and may be wrongly excluded, as shown in Figure 13e. Figure 13g–i show the detection results without LSSM, in which asymmetric land clutter clusters are detected along coast lines. Figure 13j–l show the detection results without OBSM; some concentrated land elements and facilities are falsely detected among the land regions.

5. Discussion and Conclusions

This paper proposed a novel multi-modality saliency (MMS) method for ship detection in SAR images with complex inshore scenes. The MMS method was validated on images with offshore scenes, inshore scenes without port facilities, inshore scenes with confusing port facilities, and large-scale scenes. The performance was compared with those of several existing methods to demonstrate the effectiveness and superiority. For all three scene types, the MMS method can successfully detect the ship targets without false alarms, while either miss detection or false alarms exist in the detection results of other methods. However, side lopes and some small land structures are found connected with the detected ships using MMS method, since a limitation of the MMS method is that it relies heavily on the performance of the MSER method for ship contour detection. A future research direction is to develop a clustering and pruning process for the MMS method for a more accurate ship contour, avoiding excluding ships connected with port facilities. Ablation tests are also conducted on three scene types to evaluate the concrete contributions of the saliency maps in our MMS method. Ships are falsely connected with land clutters without SPSM. Asymmetric land clutter clusters are detected along coast lines without LSSM. Concentrated land elements and facilities are falsely detected among the land regions without OBSM.

We can further draw the following conclusions from this paper: (1) It shows that the main elements of SAR images in complex scenes (noise, ship targets, and landforms) can be well separated by their scale difference from the perspective of surface metrology. This segmentation method is hardly affected by speckle noise, which surpasses most existing methods, such as Otsu’s method. This provides a novel interdisciplinary perspective for SAR image segmentation. (2) It utilizes the spatial characteristics of SAR image elements to mark the potential ships. For now, we only employ pixels with the most local stability, but further research can employ pixels with other extents of local stability for better ship locating. (3) It proposes a novel robust CFAR procedure that can eliminate the effects of noise and large ships on the background clutters’ statistical fitting. The robust KDE CFAR detection is just one application, and this procedure is also available for other statistical models. In summary, the proposed MMS method exploits multi-modality information of SAR images and transforms them into saliency maps. The fusion of these saliency maps enables effective and accurate ship detection in various scenes of SAR images. The experimental results demonstrate that the MMS method can achieve high detection performance in both inshore and offshore scenes, regardless of confusing port facilities, and outperforms the widely used methods, such as CFAR-based methods and super-pixel methods.