1. Introduction
Infrared ship target detection is an important technology for maritime search and track applications [
1,
2], where both accuracy and robustness are indispensable. However, because of the long imaging distance, for thermal infrared (TIR) images, the signal intensity of a small ship target is usually very weak without sufficient texture and structure information. More importantly, the complicated sea clutter such as sun glint, tail wave, island, sea fog and sea-sky line are usually capricious without predictable shape, which reduces the accuracy of TIR ship target detection. Moreover, the variable size and irregular shape of a ship target also further restrict the robustness of target detection. For above-mentioned reasons, infrared ship detection has attracted many researchers and a number of ship target detection algorithms are designed [
3,
4,
5].
The representative TIR ship detection algorithms can be roughly divided into two categories, namely, detect-before-track (DBT) strategy and track-before-detect (TBD) strategy. By taking full advantage of the continuity of moving target and the stationarity of background, the TBD-based detection methods, such as frame difference method [
6], three-plot correlation filter [
7] and multi-stage hypothesis test filter [
8], have achieved outstanding performance for the ships with continuous trajectory. However, in many maritime scenes, the background is changeable, and the target trajectory might be discontinuous, so the performance of TBD methods would degrade sharply. Compared with TBD strategy-based methods, the DBT-based methods have many advantages, such as less prior knowledge and faster computational speed and can work stably for the target without continuous trajectory under variable background. Therefore, the DBT-based ship detection methods are of great significance and have been drawing much attention from researchers recently. The existing DBT-based ship detection methods can be approximately classified into four categories: The target/background modeling-based methods, the image segmentation-based methods, the human visual system (HVS)-based methods, and the machine learning-based methods.
In the target/background modeling-based methods, the ship target and background are separated by modeling the TIR imaging properties of ship target or background. Lagaras et al. [
9] introduced an end-to-end temperature difference model for the detection and classification of TIR ships at the environmental conditions with an analysis-based scanning detector. Wang et al. [
10] proposed a TIR ship detection method by modeling ship radiation anomalies with a nonlinear statistical Gaussian mixture model. These methods try to extract the ship target from background via modeling on the premise of acquiring abundant knowledge about the infrared radiation characteristics of ship target and background, but it is difficult to meet these requirements. To solve this problem, Kim et al. [
11,
12] estimated the background and distinguished the target from sea clutter by using the heterogeneous background removal filter. Furthermore, the statistical histogram curve transforms were also developed for the infrared maritime target detection based on the model assumption that the ship target region is much brighter than the background, such as the methods mentioned in [
13,
14]. These background removal filters and statistical histogram curve transforms are excellent for the infrared point ship target with relatively high positive contrast. However, for low contrast or negative contrast, the ship targets are dim, and the background clutter is intricate, so these methods may rapidly reduce the detection performance.
The image segmentation-based methods have the advantages of simplicity and efficiency, which are widely used for ship detection in TIR images. The classical threshold segmentation methods such as 2D Otsu [
15], minimum error [
16], and 2D maximum entropy [
17] are well known in infrared ship target segmentation for their simplicity and easy-implementation. Nevertheless, these classical threshold segmentation methods are sensitive to noise and cannot detect small or low contrast ship targets due to the fact that their performance is easily affected by the clutter intensity information. To overcome the disturbance of noise and background clutter, the mean shift-based ship segmentation algorithms developed in [
18,
19] have achieved considerable detection performance for infrared ship in sea clutter. Whereas, because the mean shift methods are based on region clustering and merging, they may obtain a wrong detection when the ship target has low contrast or point size. In addition, the active contour-based Chan–Vese models [
20,
21] are also commonly used in the field of ship target detection because they can effectively segment the targets in homogenous background by extracting topology structure. Unfortunately, for complex background with heterogeneous sea clutter, the low-contrast ship target and sea clutter might be similar in topology structure, so serious false detection might happen.
The human visual system-based methods are based on the local contrast measure and selective attention mechanism of the ship target region, and therefore, the feature saliency map calculation of infrared ship target is the foremost topic for HVS-based methods. Mumtaz et al. [
22] adopted graph-based visual saliency (GBVS) method to compute a saliency map, and then extracted the ship target by using multilevel thresholding of the saliency map. The GBVS-based method may extract strongly salient clutter regions and fail to detect real ship target when the sea clutter is heavy. To conquer this problem, Liu et al. [
23,
24] proposed an effective infrared ship target detection method based on saliency map fusion by exploiting multi-features of ship target, including local contrast, edge information, and brighter intensity. Following, Bai et al. [
25] presented a new detection method for low-contrast infrared ship targets by analyzing the fuzzy inference system that integrates both local saliency information and global spatial feature. The two methods can acquire excellent performance for the detection of infrared ships in complex background clutter. However, these methods are based on the assumption that target regions are comparatively brighter than the dark sea surface, so they cannot detect the negative-source dark ship targets submerged in relatively bright backgrounds.
In the machine learning-based methods, the infrared ship detection problem is considered as a two-class (ship target and background) recognition problem. In these methods, the infrared images are depicted by multiple feature vectors, and then the ship target class and background class can be distinguished by classifiers, such as feedforward neural network [
26], extreme learning machine [
27], artificial neural network [
28], and convolutional neural network [
29,
30]. These machine learning-based methods can easily detect diverse ship targets under intricate background clutter in some cases. However, regrettably, these methods must spend plenty of time training samples and selecting features. Moreover, in practical TIR maritime applications, these machine learning-based methods fail to generate enough training samples due to the complexity and variability of the sea clutter, leading to the deterioration of the ship target detection capability.
Comparing the advantages and disadvantages of above-mentioned methods, although many studies have been focused on the detection of TIR ship targets against complex backgrounds in the past decades, it is still an open issue. Actually, in TIR remote sensing images, the natural scene background has strong local self-similarity, but the small ship target as a manmade object will destroy these characteristics of background, so compared with the background clutter even for heavy sea clutter, the ship target has solid intensity, contrast, contour, and shape features. To further overcome the disturbance of heavy sea clutter on the detection of ship targets with low-contrast, multiple targets, unknown brightness, and different sizes, we propose an effective and robust small ship detection scheme, based on the morphological reconstruction and multi-feature analysis of TIR imaging characteristics between ship targets and background clutter. Firstly, a pre-processing procedure based on closing (opening)-based gray-level morphological reconstruction (GMR) is introduced to remove noise and smooth intricate background clutter while preserving the ship target signals including intensity, shape, and contour information. Secondly, according to the intensity and local region contrast features of TIR ship targets, the intensity foreground saliency map (IFSM) and brightness contrast saliency map (BCSM) are computed and fused to well enhance potential ship targets, and an adaptive threshold is applied to segment candidate ship targets. Then, motivated by the contour characteristic of target region in GMR pre-processed image, a novel contour descriptor of TIR ship target named as average eigenvalue measure of structure tensor (STAEM) is proposed to characterize candidate ship targets and eliminate residual clutter simultaneously. After that, based on the statistics and observation of shape parameters of TIR ship targets selected from a comprehensive ship database namely visible and infrared ships (VAIS) [
31], shape knowledge is obtained and utilized to further distinguish true ship targets from non-ship targets. Finally, the dual approach is adopted to detect both bright and dark ship targets in TIR image simultaneously. Extensive experiments show that the proposed ship target detection scheme outperforms the compared state-of-the-art algorithms under diverse backgrounds, and is suitable for ship targets with unknown brightness, variable sizes, and quantities.
Figure 1 gives the flow chart of the proposed TIR small ship target detection method. The red boxes have enlarged true ship targets, and the purple boxes have marked several highest STAEM values of false targets. The STAEM value of ship target is 0.3281, and the four largest STAEM values of false targets are 0.1431, 0.1149, 0.0960, and 0.0792, respectively.
denotes pixel-wise multiplication manner, and
denotes pixel-wise addition manner.
Figure 1a is the input TIR ship image, and the final automatically detected ship target image is shown in
Figure 1g. During this process, a dual approach for both bright and dark ship target detection is introduced.
Figure 1b1–b4 shows the GMR-based pre-processed images, and
Figure 1c1–c4 gives the saliency detection results of IFSM and BCSM.
Figure 1(d1,d2) shows the final fused saliency maps of bright and dark ship target, respectively.
Figure 1(e1,e2) is the step of extracting candidate ship targets and eliminating residual clutter by STAEM.
Figure 1(f1,f2) shows the detected bright and dark ship target maps after two-step ship verification method, respectively.
There are four contributions in this paper: (1) Traditional infrared ship target detection methods suffer the disturbance of heavy sea clutter. In this paper, the GMR-based pre-processing procedure is introduced to efficiently remove noise and smooth intricate background clutter. Moreover, to deeply explore the intensity and local region contrast features of TIR ship targets after GMR-based pretreatment, the IFSM and BCSM are computed and fused to highlight potential ship targets and suppress sea clutter. As far as we know, it is the first time that gray-level morphological reconstruction is used for suppressing heavy sea clutter and perceiving the saliency map detection for potential ship targets. (2) The STAEM is presented as a valid contour descriptor to further depict the candidate ship targets and eliminate residual clutter simultaneously. The proposed STAEM is a novel measure to describe the contour information of a connected region. (3) Based on the statistics and observation of the shape parameters of TIR ship targets selected from VAIS database, a statistical shape knowledge is generated and utilized to further extract true ship targets from candidate targets. Because the VAIS database contains 1242 TIR ship images composed of 264 different types of ships captured during the daytime and nighttime with variable view-angles and diverse distances, the shape knowledge obtained by statistics and observation on this database could be more widely used in the field of TIR small ship detection. (4) Combining the above methods and their advantages, an efficient and robust infrared small ship detection scheme is developed and is superior to the state-of-the-art ship target detection methods.
The structure of this paper is organized as follows: The morphological reconstruction and multi-feature analysis based on the intrinsic TIR imaging characteristics between ship targets and sea clutter are discussed in
Section 2. The TIR ship detection algorithm based on morphological reconstruction and multi-feature analysis is proposed, and the whole details of the novel and robust scheme are shown in
Section 3. Extensive experiments are included in
Section 4 to evaluate the performance of the proposed algorithm, and the results show that the ship detection performance by the proposed method is significantly enhanced. Finally,
Section 5 gives the conclusions of this paper.
5. Conclusions and Future Work
A new method based on gray-level morphological reconstruction and multi-feature analysis is proposed in this paper to detect small ship targets under heavy maritime background clutter. The proposed TIR ship target detection method can automatically segment the small ship target from the sea background clutter which has intricate texture and strong contrast. By using the opening or closing based GMR, the intricate sea clutter is removed while the intensity, shape, and contour information of ship target are retained, so the proposed ship target detection method is robust to heavy sea clutter. Considering the intensity and contrast features of TIR ship targets after GMR-based pretreatment, the IFSM and BCSM are computed and fused, so the potential ship targets can be well highlighted and the complex clutter can be suppressed simultaneously. Furthermore, considering the contour and shape features of TIR ship targets, a two-step ship verification strategy including STAEM-based contour descriptor and the statistical shape knowledge constraint is constructed, so the true ship targets are efficiently extracted from residual non-ship clutter. Moreover, the dual approach is applied by directly adding bright and dark ship target maps, so both bright and dark ship targets in TIR image can be simultaneously detected. Extensive experiments verify that the proposed small ship detection algorithm has a better detection performance than compared state-of-the-art methods, including THF/BHF, 2DME, 2DO, MSS, IMFS, IFCM, CVM, and HCST. The experimental results also demonstrate that the proposed method can work stably for ship target with unknown brightness, variable quantities, sizes, and shapes.
However, although the proposed method has achieved considerable detection results, the method is based on the assumption that ship targets are viewed as uniform regions under the sea background in thermal infrared images due to long imaging distance, so it cannot work well for segmenting the whole ship target with uneven intensities in near-distance or near-infrared imaging. Therefore, combining the proposed method with region growing method [
47] to develop a high-quality ship segmentation algorithm for ship targets with uneven intensities is one important direction in our future studies. Besides, the proposed method can accurately detect both bright and dark ship targets by directly adding bright and dark ship target maps in most cases but may cause some false alarms in the ship target detection submerged in dense sun-glint clutter, as
Figure 14j6 illustrates. Therefore, combining multi-frame verification [
48] or deep multi-feature fusion [
27,
49] strategy to recognize ship target submerged in heavy sun-glint clutter is our another important research direction.