A Review of Methods for Ship Detection with Electro-Optical Images in Marine Environments

Abstract

The ocean connects all continents and is an important space for human activities. Ship detection with electro-optical images has shown great potential due to the abundant imaging spectrum and, hence, strongly supports human activities in the ocean. A suitable imaging spectrum can obtain effective images in complex marine environments, which is the premise of ship detection. This paper provides an overview of ship detection methods with electro-optical images in marine environments. Ship detection methods with sea–sky backgrounds include traditional and deep learning methods. Traditional ship detection methods comprise the following steps: preprocessing, sea–sky line (SSL) detection, region of interest (ROI) extraction, and identification. The use of deep learning is promising in ship detection; however, it requires a large amount of labeled data to build a robust model, and its targeted optimization for ship detection in marine environments is not sufficient.

1. Introduction

Human activities in the ocean are becoming more frequent and include maritime transportation, maritime trade, marine fishing, and military activities [1]. Ships are important carriers for human activities in the ocean, so the monitoring and management of maritime ships is critical.

With the rise of autonomous vehicles, autonomous ships have attracted more attention, especially for long-distance and short-distance cargo ships and special ships participating in high-risk military operations [2]. Some ships are equipped with automatic identification systems (AISs) and broadcast their own position and other information through radio signals, which can be received by surrounding ships to avoid collisions. However, little marine crafts (canoes, kayaks, sailboats, small fishing boats, etc.) are often not equipped with AIS [3], and the signals emitted by AIS are vulnerable to interference. Therefore, for ship management in the marine environment and the unmanned and autonomous navigation of ships, the ability to accurately detect ships is needed.

Radar is widely used in ship detection, but it has some limitations. Radar is vulnerable to the interference of objects around a port such as metal facilities. In addition, it cannot detect small nonmetallic (wood or glass fiber) marine crafts [4]. For synthetic aperture radar (SAR), speckle noise and motion blurring in SAR images make feature extraction difficult [5,6,7,8]. Due to the imaging mechanism of SAR, images are difficult to comprehend and interpret [5].

Ship detection with electro-optical images has attracted the interest of researchers. The images are intuitive and easy to understand for the human eye, and the rich imaging spectrum also meets the imaging needs in various scenarios. Visible spectrum imaging has high resolution and can obtain color and texture information. Thermal infrared can obtain images of a scene under night vision and low-light conditions. Short-wave infrared is less affected by atmospheric scattering and has a strong penetrating ability for fog and haze. The abundant imaging spectrum can enable ship detection in marine environments to obtain clear and effective images under various light and weather conditions. This is beneficial to the accuracy and robustness of ship detection.

There have been some reviews on ship detection. For example, Kanjir [3] reviewed articles on ship detection using optical satellite images. Prasad [9] paid more attention to video-based object detection and tracking in the marine environment. Spraul [10] compared the performance of several deep learning methods in maritime ship detection. Fingas [11] discussed ship detection methods based on airborne platforms. These authors all mentioned the subject of ship detection. However, ship detection with electro-optical images in marine environments still lacks sufficient discussion. For example, discussions on applicable scenarios of different imaging spectra are lacking. The characteristics of a marine environment and a ship’s appearance make ship detection different from generic object detection. Therefore, ship detection workflows in marine environments are special. In addition to traditional ship detection methods, deep learning has been introduced into ship detection by researchers in recent years. In the open sea, there is often insufficient texture information, and it is difficult to select appropriate hand-crafted features. Deep learning can automatically learn robust features from data. However, current deep learning methods do not fully consider the characteristics of ship detection in marine environments, which limits their performance. To our knowledge, there is no article that fully discusses these issues.

This paper focuses on ship detection with electro-optical images in marine environments to meet the needs of ship management and flow monitoring in ports and coasts, as well as the unmanned and autonomous navigation of ships. In this paper, Section 2 introduces the applicable scenarios of different imaging spectra and the image characteristics of ship detection in marine environments. Section 3 presents the traditional ship detection methods and deep learning methods and discusses the advantages and limitations of different methods. Section 4 presents concluding remarks.

2. Electro-Optical Images in Ship Detection: Spectral Bands and Characteristics

Acquiring clear and effective images is a prerequisite for ship detection. Appropriate spectral bands can obtain better images in harsh and changeable marine environments. Although objects can be enhanced in an original image by using image processing methods, it increases the burden of computing devices, especially for some embedded devices. The characteristics of electro-optical images, including the appearance characteristics of ships and the components of maritime scenarios, determine the ship detection workflow.

2.1. Image Spectral Bands in Ship Detection

To fulfil accurate ship detection in various maritime scenarios (night vision, fog, or haze), it is worth considering selecting a suitable imaging spectrum. The changeable weather conditions and light in marine environments make it difficult for imagers to obtain clear and effective images. This paper counts the imaging spectrum commonly used in ship detection.

Table 1. Types of electro-optical images commonly used in ship detection.

Types of Electro-Optical Images		Articles
Visible		Chen [12], Cane [13], Marie [14], Chen [15], Chen [16], Liu [17], Shan [18], Gal [19], Lin [20], Chen [21], Lee [22], Feng [23], Shan [24], Fefilatyev [25]
IR	N/A	Tang [26], Liu [27], Hu [28], Lin [29]
	MWIR	Özertem [30], Wang [31]
	LWIR	Sun [32], Lu [33], Bai [34], Leira [35], Bai [36], Mumtaz [37], Singh [38], Zhang [39], Xu [40], Zhou [41], Schöller [42], Li [43], Westlake [44]
Visible + IR	N/A	Islam [45], Wei [46]
	Visible + MWIR	Nita [47]
	Visible + LWIR	Zhang [48], Ribeiro [49], Farahnakian [50]
	Visible + SWIR + LWIR	Stets [51]
	Visible + SWIR + MWIR + LWIR	Bouma [52]

Visible: refers to visible-band images; IR: infrared images, including short-wave infrared (SWIR), medium-wave infrared (MWIR), and long-wave infrared (LWIR); N/A, means that the type of image was not specified in the paper.

shows the types of electro-optical images commonly used in ship detection. Figure 1 shows the percentage of various optical images. It can be seen from the statistical data that infrared images are used the most, followed by visible images. Among the infrared image types, the most commonly used is long-wave infrared (LWIR).

Different imaging spectral bands have unique advantages. LWIR and medium-wave infrared (MWIR) can be imaged without any natural (sun or moon) or artificial light. For night vision, there is a temperature difference between the ship’s power cabin and the sea–sky background, which is beneficial for thermal infrared imaging. However, for some small marine crafts (canoes, etc.), due to the fact of long-term immersion in the sea, they have reached a thermal balance with the seawater. In these cases, LWIR is usually unable to distinguish the target. In daytime ship detection, LWIR imaging has no obvious advantages [51]. Visible-band camera technology is well established and widely used. For daylight imaging, or in clear weather, visible-band cameras can provide more information than LWIR and MWIR imaging. Visible-band cameras are sensitive to sudden changes in light. Severe weather conditions reduce the effectiveness of visible-band cameras. Short-wave infrared (SWIR) is currently rarely used but has great potential. Both SWIR and visible-band cameras rely on reflected light for imaging, and their imaging mechanisms are similar. Therefore, SWIR can obtain more information than thermal infrared such as texture information. SWIR penetrates fog and haze much better than sensitive sensors in the visible spectral range, especially for long-range distances [53].

Limited by the spectral response or imaging mechanisms, it is difficult for a single-band sensor to simultaneously take into account spatial resolution, penetration (such as smoke, fog, and rain), and sensitivity. The combination of different band sensors can meet the needs of all-day and all-weather ship detection in a marine environment and better cope with the complex environment. For example, the combination of thermal infrared and visible-band cameras can provide daytime and night detection. A visible-band camera has a high resolution and can also compensate for the texture and color information missing from thermal infrared. SWIR has a strong permeability for fog and haze and can effectively be used to supply LWIR and visible-band imaging. Currently, most researchers use images of different spectral bands separately. Multi-sensor data fusion can provide comprehensive information, give play to the advantages of each sensor, and obtain more accurate detection results. However, data fusion will also increase the computational load. Existing fusion methods mainly include pixel-level fusion, feature-level fusion, and decision-level fusion [54]. Farahnakian [50] uses three fusion methods to detect ships. Their research shows that feature-level fusion performs better than other fusion architectures.

2.2. Image Characteristics with a Sea–Sky Background

Ship detection in marine environments is usually performed with a sea–sky background, and the components of typical scenarios include sea, sky, and sea–sky regions. In sea–sky regions, the sea–sky line (SSL) is the boundary between the sea and sky regions. Ships usually appear near the SSL, so the SSL can be used as a pivotal clue for ship detection. In the following discussion on the workflow of ship detection, it can be seen that the SSL is the main reason for the difference between ship detection and generic object detection.

Figure 2 shows typical scenes captured by ship detection in a marine environment.

Unlike observing ships by satellite or airborne platforms, the complete contour of a ship cannot be obtained in a marine environment. A ship’s appearance will change due to the detection distance and view angle. The appearance of a ship is easily affected by lighting, occlusion, and the observation angle. In addition, wide intraclass variations in some types of ships also make it difficult to classify ships [56]. These bring challenges to the classification task of ships (such as passenger ships, and cargo ships). Some research in the field of computer vision may provide some inspiration to solve this problem such as person re-identification [57] and vehicle re-identification [58,59,60]. They face similar problems such as the effect different viewing angles have on the appearance of a target. Currently, most researchers only distinguish between ships and non-ships in ship detection in marine environments.

The marine environment is complex and changeable. The motion of floating offshore platforms, buoys, and ships is affected by waves, currents, wind, and variations in water depth owing to the tide [61,62,63,64]. For example, in certain seas, such as Nigeria’s Bonga Field in offshore West Africa, near-surface currents are usually powerful and vary with time and location [63]. But for ship detection in marine environments, most researchers pay more attention to the impact of ship appearance changes on identification. Especially for some types of ships, their wide intraclass variations bring challenges to ship detection [56].

3. Ship Detection Methods from Electro-Optical Images

Ship detection can be regarded as a subcategory of generic object detection. Due to the particularity of a marine environment and the characteristics of ships, ship detection cannot fully follow the paradigm of generic object detection. Currently, deep learning has achieved impressive performance in object detection, and more researchers have introduced deep learning into ship detection. Detectors based on deep learning can be directly used for ship detection, but it is worth considering combining the characteristics of marine environments. For ease of understanding, this paper uses illustrations to show the scheme of ship detection workflow including traditional methods and deep learning methods.

3.1. Traditional Ship Detection Methods

From the current literature, this paper summarizes the workflow of ship detection in marine environments. Traditional object detection uses sliding windows of different scales to search for candidate regions. In ship detection, detecting the location of SSL can effectively reduce the search range of candidate areas and reduce interference. The SSL is the main reason for the difference between ship detection and generic object detection. In addition, in the stage of region of interest (ROI) extraction and ship identification, many researchers use SSL as an important clue.

The workflow of ship detection is shown in Figure 3. Traditional ship detection consists of four steps: preprocessing, SSL detection, ROI extraction, and recognition.

3.1.1. Preprocessing

The captured images usually contain noise and interfering objects such as sea clutter and clouds. The first ship detection step for some researchers is to remove or mitigate the environmental effects.

Tang [26] used Gaussian smoothing to filter noise. Li [43] used gray morphological reconstruction based on opening-and-closing operations to address background clutter. Lu [33] used a median filter to eliminate noise. Sun [32] used a wavelet transform to suppress the background. Bouma [52] estimated the background intensity and subtracted the background intensity from the image to eliminate the change in the background.

In preprocessing, it is difficult to completely eliminate the environmental effects. At present, most of the preprocessing methods used are only for significant noise interference and have trouble addressing strong sea clutter or some extreme situations.

3.1.2. SSL Detection

In maritime scenarios, the SSL is the boundary between the sky and the ocean, and it is conducive to locating the ocean area and reducing the search range. Figure 4 shows the process of reducing the candidate search regions after the SSL is detected. In long-range ship detection, ships usually appear near the SSL. There are some SSL-based methods that apply subsequent steps such as ROI extraction and identification. Many researchers, such as Prasad [9] and Lipschutz [65], have summarized related methods for SSL detection. This paper summarizes the common methods of SSL extraction in ship detection.

Transformation

Methods based on transformation first extract all edges in an image to generate an edge map. Then, the Hough transform (Wei [46], Özertem [30], and Chen [12]) or Radon transform (Tang [26]) is used to detect all lines in the image. Finally, the correct SSL result will be selected according to some criteria. For example, Tang [26] comprehensively judged the candidate SSL locations through a gradient transform, the gradient direction, and the average gray difference between the sea and sky.

The SSL detection method based on transformation is mature and easy to use, but it requires image edge extraction. Therefore, when the contrast of an image is low or the sea–sky boundary is blurred, it is difficult to obtain a clear edge image. An edge detector only responds to local changes and is easily disturbed by noise. In this case, the detection of the SSL is difficult or results in misidentification. In addition, the brightness of the sea surface is nonuniform, and sea clutter can produce many edge features, resulting in misjudgments.

Region Features

Generally, the various features of the sea or sky region change slowly, but there will be drastic changes at the sea–sky boundary. Therefore, the location of the SSL can be determined according to the change in features in an image. This article summarizes the features that are used to describe regions as shown in

Table 2. The features used to describe regions in sea–sky line detection.

Articles	Description Features of Regions
Bouma [52]	Local intensity differences
Wang [31]	Threshold, standard deviation map (SDM) features, width, height, centroid, and area
Fefilatyev [66]	Intensity, mean, standard deviation, smoothness, third moment, uniformity, entropy
Liang [67]	Texture, threshold
Sun [68]	Gradient features
Liang [69]	Probability distribution, physical characteristics: texture, linear features, color, and context information
Lin [20]	Gradient integral curve of an image

. It is difficult to classify various regional description features and describe their advantages and disadvantages. In particular, many authors use combined features. For details on the proposed algorithms, please refer to the original works.

This method uses some features to represent an image and distinguishes different regions through the differences in features. It considers the global image features and prevents the interference of local noise. Nonetheless, it needs to determine the distribution of features in an image, which requires a large amount of computation.

Semantic Segmentation

Semantic segmentation can mark the pixels and make a more detailed division of regions (ocean, sky, and mixed region) to obtain a more accurate location of the SSL. For example, Yang [70] used the Gaussian mixture model (GMM) and Jeong [71] used a neural network to semantically segment images. A segmented image can be used to determine the sky and ocean area and locate the SSL.

This method is usually complex or needs labeled training samples. Due to the large amount of computation, semantic segmentation may not meet the real-time requirement on embedded devices with limited computing resources.

Summary of SSL Detection Methods

Table 3. Summary of the sea–sky line detection methods.

Methods of SSL Detection	Articles	Advantages	Disadvantages
Transformation	Wei [46], Tang [26], Özertem [30], Chen [12]	The algorithms are mature and easy to use.	(1) The effect is poor when the image is blurry and has low contrast; (2) Sea clutter can cause misjudgments.
Region attributes	Bouma [52], Wang [31], Fefilatyev [66], Liang [67], Sun [68], Liang [69], Lin [20]	It prevents the influence of local noise and is more robust.	(1) Needs to choose the appropriate hand-crafted features; (2) Large amount of computation.
Semantic segmentation	Yang [70], Jeong [71]	It can obtain a more accurate location of the SSL.	It may not meet the real-time requirements on embedded computing devices.

shows the advantages and limitations of each SSL detection method.

3.1.3. ROI Extraction

After extracting the location of SSL, the next step is to find the area containing candidate ships, that is, the ROI. Some researchers use SSL information, while others use common threshold methods. In the face of low-contrast images, such as thermal infrared images, some researchers use saliency detection to extract ROIs. Frequency domain methods are also commonly used.

SSL-Based

The SSL is an effective clue for ROI extraction. Reasonable use of the SSL can effectively reduce the scope of a search. Especially for long-range detection, ships usually appear near SSL.

Researchers search for candidate ROIs near the SSL and usually combine other information to comprehensively judge the correct ROI. For example, Tang [26] used gradient and shape information, Lu [33] used the distance rule, and Chen [12] used the mutation of gray values. Some researchers also use a fixed-height search range near the SSL (Fefilatyev [25]) or cut the image directly near the SSL to extract ROIs (Shan [18]).

However, the location error of the SSL will reduce the accuracy of ROI extraction. Therefore, the premise of using this method is to obtain the accurate position of the SSL.

Threshold-Based

There is usually an obvious intensity difference between ships and the background. Ships and the background can be distinguished by a threshold. Extracting ROIs based on thresholds is a traditional method.

Researchers use different methods to determine the appropriate threshold. Wang [72] analyzed the complexity of an image. Singh [38] used linguistic quantifiers to determine the appropriate threshold. Bouma [52] estimated the intensity of the background and detected the target using hysteresis thresholding.

When the contrast between a ship and the background is low (especially for thermal infrared images), it is difficult to determine an appropriate threshold. In addition, sea clutter can also affect threshold-based methods.

Saliency Detection

Sometimes the contrast of an image is low, and the intensity distribution is uneven. To effectively detect ships, visual saliency detection can be used to find saliency regions according to global information instead of relying on local information. To prevent interference caused by local noise to a certain extent, the bottom-up visual saliency detection method based on data is commonly used. It judges the difference between the target region and its surrounding pixels by the color, the brightness, the edge, and other features.

Researchers have used different strategies to detect salient regions. Li [43] used intensity and contrast features. Liu [27] proposed an improved light nonlocal depth feature (NLDF) for low-contrast infrared images. Mumtaz [37] used graph-based visual saliency (GBVS). Lin [20] fused a variety of visual features, such as gradient texture, brightness, and color features, to obtain a salient image.

When there is a large area with strong sea clutter, it will cause misidentification of the saliency regions. Therefore, the suppression of sea clutter is a key issue.

Frequency Domain

There are usually obvious differences between the sea surface and a ship in the frequency domain. The background and target can be separated by setting an appropriate threshold.

The commonly used frequency domain methods are the Fourier transform and wavelet transform methods. Zhou [41] combined the fractional Fourier transform (FRFT) and high-order statistical filtering. The target and sea clutter were separated by a high-order statistical curve (HOSC). Sun [32] used a wavelet transform.

When the sea clutter is strong, it is difficult to separate the background and a ship simply by the features of the frequency spectrum; however, when these methods are combined with morphological analysis, they can usually achieve better results. For example, Zhou [41] combined shape features to obtain more accurate results.

Summary of the ROI Extraction Methods

Table 4. Summary of the ROI extraction methods.

Methods of ROI Extraction	Articles	Advantages	Disadvantages
SSL based	Lu [33], Chen [12], Shan [18], Fefilatyev [25]	It is simple and fast.	The location error of the SSL will reduce the accuracy of ROI extraction.
Threshold based	Wang [72], Singh [38], Bouma [52]	It is simple and easy to use.	(1) When the contrast between a ship and the background is low, it is difficult to determine an appropriate threshold; (2) Sea clutter will affect the selection of the threshold.
Saliency detection	Li [43], Liu [27], Mumtaz [37], Lin [20]	(1) Suitable for thermal infrared images; (2) It works well for low-contrast images.	A large area of strong sea clutter can cause misidentification of the saliency regions.
Frequency domain	Zhou [41], Sun [32]	This method can suppress the background noise well.	When the sea clutter is strong, it is difficult to separate the background and target only by frequency spectrum features.

summarizes the advantages and disadvantages of various ROI extraction methods.

3.1.4. Identification

The extracted ROIs may contain false alarms such as sea clutter and islands. This step removes false alarms and identifies the correct candidate ships. There are two kinds of recognition results: ship and non-ship. Few authors mention the classification of ship types. This article is more concerned with identifying the correct ships and non-ships.

Prior Knowledge

Ships have fixed sizes, shapes, lengths, and widths, so some researchers have used prior knowledge to identify ships.

Tang [26] used the aspect ratio, the contrast ratio, the duty ratio, and other features for ship recognition. Özertem [30] removed false alarms according to prior knowledge of a ship’s size and the size of the target’s edge on the SSL. Lu [33] found that a ship is usually near the SSL and has the brightest grayscale in an infrared image, so the distance rule and gray rule were combined to determine whether a target was a ship.

However, the robustness of ship identification using prior knowledge is weak, and it is easily disturbed by waves, clouds, and islands in the ocean, resulting in false alarms.

Classifier

Some researchers have used feature vectors to describe ships and used samples to train classifiers to obtain more robust results.

Xu [40] proposed a rotation-invariant descriptor, a circle histogram of oriented gradient (C-HOG), to describe ships. This descriptor can identify infrared ships with different rotation angles. Finally, they used support vector machine (SVM) as a classifier. Lin [29] used three types of features: size, shape, and texture; constructed a 10-dimensional feature vector to describe a ship; used SVM for offline training to effectively remove false alarms.

This method needs to select appropriate hand-crafted features, and the training of the classifier depends on the training samples.

Summary of Ship Identification

Table 5. Summary of the various ship identification methods.

Methods of Ship Identification	Articles	Features	Advantages	Disadvantages
Prior knowledge	Tang [26], Özertem [30], Lu [33], Li [43]	Aspect ratio, contrast ratio, duty ratio, ship size, grayscale, distance rule, gray rule, and shape	Simple and easy to use.	The robustness is weak, for example, a change in the detection angle, the detection distance, and light conditions make the ship show different characteristics.
Classifier	Xu [40], Lin [29]	C-HOG, shape, size, and texture	The method is robust.	(1) This method needs to extract appropriate hand-crafted features; (2) It relies on training samples.

summarizes the various ship identification methods.

3.2. Ship Detection Based on Deep Learning

Deep learning has great potential in various fields, such as biology and physics, and it is not limited to computer vision. It breaks through some of the limitations of traditional methods and has achieved remarkable results.

Currently, in object detection, most of the state-of-the-art methods use deep learning networks as the backbone and detection networks to extract features from images and perform classification and location tasks [73]. At present, ship detection based on deep learning mostly borrows methods from object detection.

There are many open-source deep learning frameworks that provide basic deep learning components and rich application programming interfaces (APIs), which provide convenience for researchers to use deep learning methods to deal with ship detection. Popular open-source deep learning frameworks are shown in

Table 6. Summary of deep learning frameworks.

Deep Learning Frameworks	Organization	Interface
TensorFlow [74]	Google	Python, C++, Java, and Go
Keras [75]	Google	Python, R
PyTorch [76]	Facebook	Python, C++
MXNet [77]	Amazon	Python, C++, Clojure, Julia, Perl, R, Scala, and Java
CNTK [78]	Microsoft	Python, C++, C#/.NET, and Java
Theano [79]	Université de Montréal	Python

.

3.2.1. One-Stage and Two-Stage Detectors

Traditional object detection methods using hand-crafted features have encountered a development bottleneck. Convolutional neural networks (CNNs) can extract high-level semantic information from images and obtain more robust features. In 2012, AlexNet [80] achieved impressive results in the ImageNet competition. Since then, some researchers have tried to introduce CNNs into object detection. In 2014, Girshick et al. proposed R-CNN [81] and used CNN for object detection for the first time. At present, object detection based on deep learning has become mainstream.

Object detection based on deep learning can be divided into two categories: (a) two-stage detectors, which mainly include R-CNN [81], Fast R-CNN [82], and Faster R-CNN [83]; (b) one-stage detectors, which mainly include YOLO (YOLO v2 [84] and YOLO v3 [85]) and SSD (SSD [86] and DSSD [87]).

The two-stage detectors have a proposal generation stage, and the generated proposals are sent to the next stage for object classification and location regression. A one-stage detector directly predicts the category of objects on the extracted feature map and regresses the positions of the objects. Figure 5 shows two-stage and one-stage detectors. Two-stage detectors usually have better detection performance, while one-stage detectors have faster inference speeds.

3.2.2. Deep Learning in Ship Detection

In view of the excellent performance of CNNs in the field of object detection, more researchers have used deep learning for ship detection in recent years. Figure 6 shows the development trend of traditional methods and deep learning methods in ship detection.

One-Stage and Two-Stage Detectors in Ship Detection

This paper sorted the CNN models commonly used in ship detection as shown in

Table 7. Application of deep learning models in ship detection.

Methods of Deep Learning	Articles	Model	Advantages	Disadvantages
One stage	Chen [16], Westlake [44], Chen [21]	YOLO	One-stage detectors are time efficient, have fast detection speeds, and are suitable for deployment to embedded platforms. They are suitable for shipborne platforms.	The detection accuracy is usually less than that of a two-stage detector.
	Stets [51]	Retina-Net
	Lee [22], Hu [28]	Other
Two stage	Marie [14]	Fast R-CNN	Two-stage detectors commonly obtain more accurate detection results. They are suitable for shore-static platforms.	The detection speed is slow, the calculation amount is large, and they are not suitable to deploy on embedded platforms.
	Chen [15], Shan [24]	Other
Others	Gal [88], Farahnakian [50], Liu [27], Cane [13], Nita [47]

.

Many researchers have used one-stage detectors. In ship detection, the detection speed needs to be considered. For example, for the anticollision warning of ships, there are high requirements for the real-time performance of ship detection. The one-stage models usually have faster inference speeds and can meet real-time tasks. In addition, many shipborne platforms need to consider the volume and power consumption of computing equipment. Shipborne platforms usually use embedded devices for ship detection, where one-stage models have less computation and lower power consumption and are suitable for deployment on embedded platforms.

Two-stage detectors usually have better detection accuracy but also require a large number of calculations and high consumption of power. Shore-static platforms usually monitor ships in a port, where the moving speed of the ships is slow, and there is no need to strictly limit the volume and power consumption of the computing device. In these cases, the two-stage model can be deployed on a desktop computer.

As the observation distance changes, ships have different scales in an image. The sizes of ships are also diverse. The scale change in ships brings challenges to the current detection networks based on predesigned anchors [24]. In addition, in the long-distance detection of small marine crafts, the ship usually occupies a few pixels in the image. This is unfavorable for the feature extraction of a CNN. Especially for a one-stage network, small marine crafts in an image may be ignored.

Currently, researchers mainly start with two factors to improve the performance of detection networks in ship detection: multiscale features and anchor settings.

Researchers use multiscale features or high-resolution feature maps to improve the detection ability of ships with different sizes. Generally, in a neural network, shallow features contain more structural and geometric information of an object [89], which is conducive to the regression of the target. High-level features represent more semantic information, which is conducive to the classification of objects [89]. Therefore, combining the features of different layers can obtain complementary information. To enhance the detection capability of small marine crafts, Chen [15] extracted proposals of different scales in a multilayer convolution feature map. In the detection network, they used high-resolution convolution feature maps to extract targets. Hu [28] used a scale transform module to establish a feature pyramid network (FPN) to cope with the changes in ship appearance caused by different imaging distances. The module used context information to combine high-level features and low-level features. Shan [24] used ResNet-50 as a backbone network and added an FPN structure to solve the problem of variation scales, especially for some small marine crafts. They used the feature maps output by the three convolutional layers as the input of three region proposal networks (RPNs), which improve the discriminative ability at different scales.

Anchors are a set of initial, fixed-size candidate boxes [28]. The shape and size of an anchor will affect the accuracy and speed of object recognition and regression. The diversity of a ship’s shape and size as well as changes in the observation distance, bring challenges to the selection of anchors. Currently, the commonly used method is to analyze the ground truth bounding boxes in the training set to select suitable shapes and sizes for the anchor. For example, Hu [28] and Shan [24] used k-means clustering to cluster all ground truth bounding boxes in the training data set, and the cluster centers were used as the initial anchors. However, the effect of this method depends on whether the training data set contains enough ship sizes and shapes.

Discussion about the SSL

The current deep learning-based ship detection methods have not considered the use of the SSL. In Section 3.1.2., it was mentioned that the SSL is important in ship detection, can effectively reduce the search area, and can remove some interference areas. To the best of our knowledge, there is no application of the SSL in ship detection based on deep learning.

Some researchers mentioned that before sending an image to the detection network, they will first screen the potential candidate regions. For example, Marie [14] analyzed video streams to determine potential key points of objects or backgrounds. Using the SSL may be a more effective way to reduce the candidate region. For some lightweight SSL extraction methods, their calculations are acceptable. In two-stage detectors, proposal generation is time consuming. The detection of the SSL can improve the efficiency of two-stage detectors and give better play to their advantages of detection accuracy. Figure 7 shows this process.

3.3. Comparison of Traditional Methods and Deep Learning

Traditional ship detection methods do not need a large number of training samples and consume relatively few computing resources, and they have strong interpretability. However, traditional methods have poor adaptability to complex scenes and need to select appropriate hand-crafted features. Especially for thermal infrared images, because they lack detailed object information and retain only the contour and spatial information of an object, it is difficult to select appropriate hand-crafted features.

Deep learning models avoid selecting hand-crafted features, automatically learn more robust image features through backbone networks, and then locate and recognize objects. Deep learning methods have strong adaptability to complex scenes, especially in marine environments, which usually encounter drastic environmental changes. However, to obtain better prediction results, a deep learning model requires a large number of training samples; otherwise, it will experience overfitting. At present, there are few public data sets of ships. There is still a big gap between some existing ships public data sets and generic object data sets in terms of data scale.

Table 8. Advantages and disadvantages of traditional methods and deep learning methods in ship detection.

Methods	Advantages	Disadvantages
Traditional methods	(1) Easy to implement; (2) They do not require many training data.	(1) It is difficult to adapt to complex scenes; (2) Need to select appropriate hand-crafted features.
Deep learning	(1) It can adapt to complex scenes; (2) No need to select hand-crafted features; (3) It provides a more promising solution for ship detection based on thermal infrared images.	Need many hand-labeled training data in real scenarios.

summarizes the advantages and disadvantages of traditional and deep learning methods.

3.4. Validation

Compared with generic object detection, ship detection lacks public data sets, especially for multispectral data. Many researchers use private data sets for research on ship detection, which is not conducive to comparison among different methods. Therefore, this paper only provides methods for evaluating performance of ship detection.

The intersection over union (IoU) is used to measure the localization accuracy in ship detection. It can check whether the IoU between the predicated bounding box and the ground truth is greater than a predefined threshold. IoU is defined as in Equation (1):

$$IoU=\frac{Area(bbox)\cap Area(gt)}{Area(bbox)\cup Area(gt)}$$

(1)

where bbox is the predicated bounding box, gt is the ground truth, and Area denotes the number of pixels. If IoU is greater than the predefined threshold, the predicated box is considered to correctly detect ships; that is a true positive (TP). For false detection, it is called a false positive (FP). For missed detection, it is called a false negative (FN). After obtaining the total number of TPs, FPs, and FNs, the precision and recall are computed as:

$$\mathrm{Precision}=\frac{Sum(TP)}{Sum(TP)+Sum(FP)}$$

(2)

$$\mathrm{Recall}=\frac{Sum(TP)}{Sum(TP)+Sum(FN)}$$

(3)

4. Conclusions

This paper reviewed ship detection methods with electro-optical images in marine environments and mainly included two parts: electro-optical images and ship detection methods. Suitable imaging spectral bands can suppress interference from the environment to capture clear and effective images. At present, the most commonly used images by researchers are visible and LWIR images. Although SWIR images are rarely used, they are promising for ship detection in marine environments. The combination of multiple spectral bands can better cope with complex marine environments. Among the traditional ship detection methods, the SSL makes ship detection different from generic object detection. Using the location of the SSL can effectively reduce the search for candidate regions. Deep learning avoids the selection of hand-crafted features, uses artificial neural networks to classify and regress objects, and achieves impressive results in ship detection. However, deep learning does not use SSL information, which is worth considering in future research. In addition, deep learning relies on labeled training samples, which also affects its application in ship detection because there are few public data sets of ships at present.