Maritime Infrared Target Detection Using a Dual-Mode Background Model

Abstract

With the rapid development of marine business, the intelligent detection of ship targets has become the key to marine safety. However, it is difficult to accurately detect maritime infrared targets due to severe sea clutter interference in strong wind waves or dim sea scenes. To adapt to diverse marine environments, a dual-mode sea background model is proposed for target detection. According to the global contrast of the image, the scene is divided into the sea surface with violent changes and the sea surface with stable changes. In the first stage, the preliminary background model suitable for steadily changing scenes is proposed. The pixel-level foreground mask is generated through the background block filter and the posterior probability criterion. Moreover, the learning rate parameter is adjusted using the detection results of two adjacent frames. In the second stage, the background model suitable for highly fluctuating scenes is proposed. Moreover, the local correlation feature is used to enhance the local contrast of the frame. The experimental results for the different scenes show that the proposed method has a better detection performance than the other comparison algorithms.

1. Introduction

Infrared maritime target detection is the basis for maritime target search and tracking systems. It is used for emergency rescue, marine environmental protection, territorial sea protection and other fields [1,2,3,4,5,6]. However, the infrared imaging quality is affected by the resolution of the infrared sensor, environment temperature, imaging distance, etc. In many cases, the gray distribution of the target is uneven and the outline is not clear [7,8,9]. Moreover, due to the low ambient temperature difference, the contrast between the target and the wave clutter is low. In addition, interference caused by illumination, large and random sea waves, weather and imaging noise results in a complex and diverse sea surface with low signal-to-clutter ratios. All these factors lead to difficulties for infrared maritime target detection. Consequently, for various complex maritime sea backgrounds, robust and accurate infrared target detection is a challenging task.

Up to now, a variety of methods have been proposed, including single-frame methods and background modeling methods for infrared maritime target detection. Compared with the background modeling methods, single-frame methods often require less computation and run faster. The key to solving this problem is to separate the targets from interference by background filtering and using feature differences. The global background subtraction filter and the adaptive row mean subtraction filter [10] are iteratively applied for background suppression and target enhancement, respectively. In addition to using background filtering, Yang et al. [11] proposed an integrated target saliency measure method using the local heterogeneity property of the targets and the non-local self-correlation property of the background to segment the targets. Zhang et al. [12] formulated the foreground segmentation problem with the feature difference in the similarity of directional edge components between the targets and interference. However, because this kind of method uses less information, its detection effect in a challenging sea scene still requires improvement. Zhang et al. [13] characterized the local intensity and gradient properties to detect small infrared targets. However, the intensity value of the target is not always greater than that of its locally neighboring pixels.

1.1. Related Work

Background modeling is a typical sequential detection method, in which the moving boats are considered as the foreground and the dynamic wave clutter is considered as the background. At present, background modeling methods are broadly categorized into statistical models, non-parametric models, neural network models, etc. Due to significant illumination changes and dynamic movement of the seawater over time, it is challenging to model variations in the sea background using these methods.

The statistical modeling approaches represent distributions of background changes using parametric models. The most popular statistical modeling approaches are the single Gaussian background model [14], Gaussian mixture model (GMM) [15] and improved GMM-based approaches [16,17,18]. The single Gaussian background models every pixel as a Gaussian probability density function, and the Gaussian mixture model uses the mixture of several weighed Gaussians to model every pixel. However, these approaches have a poor performance when the foreground objects move slowly, as exemplified by the shareable Gaussian mixture models in [19]. In [20], T. S. F. Haines et al. modeled every pixel with the mixture of several weighed Dirichlet processes for background subtraction. In [21], Kratika Garg et al. employed Bayesian probabilistic modeling for the block-based feature and proposed a rapid and robust background modeling technique for foreground detection (RRBM). The background modeling approach, by eliminating gray-level co-occurrence matrix features with an improved local Zernike moment and color components of intensity, effectively categorized the foreground and background pixels [22].

In our previous work, two background statistical modeling methods for maritime infrared target detection in the Fourier domain were proposed, including the adaptive single Gaussian background modeling algorithm (FASGM) [23] and the background modeling algorithm of multi-feature fusion (BMMFF) [24]. The FASGM method designs a comprehensive discrimination flag based on the variance in the test frames’ amplitude spectrum within a short temporal window and the initial Gaussian decision. The adaptive Gaussian discrimination coefficients are set based on each category, and an entropy filter in the Fourier domain is designed to enhance the contrast. The FASGM method has an excellent detection performance for dim targets; however, a small amount of seawater is still mistakenly detected for the highly fluctuating sea surface. The BMMFF method uses multiple features that can effectively distinguish the target and seawater, i.e., the background subtraction feature, global correlation feature and oscillation speed feature of the test frames’ amplitude spectrum. The background subtraction feature extracts the foreground by comparing the local statistical characteristics of the test frame and the background. Two strategies are used to update the background in cases with or without sudden waves in two adjacent test frames. The BMMFF method has a good performance under the influence of interference from sudden and continuous bright waves, but for a small number of dim scenes, its performance needs to be improved.

The non-parametric method establishes a non-parametric background model for each pixel through density estimation or historical information. The advantage is that it can adapt to any unknown data distribution without specifying the potential model and explicit estimation parameters. The coarse-to-fine detection algorithm extracts foreground objects on the basis of non-parametric background and foreground models represented by binary descriptors [25]. The visual background extraction (VIBE) method is a universal and non-parametric background subtraction method for video sequences, which categorizes the pixel by comparing its intensity with a set of historical background values [26]. The background subtraction algorithm [27] based on a modified ViBe model uses an improved local binary similarity pattern descriptor instead of simply relying on pixel intensities as its core component (LOBSTER). The algorithm described in [28] is also a variant of ViBe, which uses the dynamic adaptive learning rate and pixel-based adaptive threshold (PBAS). The universal pixel-level segmentation method, which uses spatiotemporal binary features and color information (SuBSENSE), is characterized as a non-parametric method [29]. The locally statistical dual-mode background subtraction approach (LSDS) [30] adopts a local intensity pattern comparison for foreground detection based on the assumption that the background pattern is generally more homogeneous than the foreground pattern. The dual-target non-parametric background model (DTNBM) [31] fuses the gray value and gradient as the feature to signify each pixel.

Moreover, in recent years, the background subtraction algorithm, based on spatial features learned by convolutional neural networks, was proposed. In [32], the algorithm uses a background model reduced to a single background image and a scene-specific training dataset to feed the convolutional neural network, which proves itself able to learn how to subtract the background from an input image patch. L. Yang et al. [33] presented a deep background modeling approach utilizing a fully convolutional network. In the network block constructing the deep background model, the coarse information of the high-layer information and the fine information of the low layer are combined using skip architecture, which can improve feature representation. In [34], a multi-view receptive field encoder–decoder convolutional neural network was introduced for video foreground extraction. Moreover, in [35], the multiscale 3D fully convolutional network was proposed for the mapping of image sequences to pixel-wise classification results and to learn deep and hierarchical multiscale spatial-temporal features of the input image sequence.

1.2. Motivation and Highlights of This Paper

Many methods employ different background modeling strategies for infrared maritime target detection and achieve relatively good detection results. However, there still remain some difficulties in infrared target detection in ocean background conditions. Due to strong, dynamic wave clutter, it is difficult for the background model to adapt to changes in the actual sea scene. Moreover, the low contrast between the target and background makes it difficult for each pixel to be correctly classified by the background model. In addition, factors such as illumination changes, weather changes and image noise may increase the complexity of the model. Due to the need to suppress sea clutter with serious interference and to obtain accurate detection results, a method that can detect targets in various sea scenes will have wide a scope for application.

In this paper, we present a dual-mode background modeling technique that can be used to effectively detect foreground masks in various sea scenes. The proposed method divides the seawater into sea surfaces with stable changes or violent fluctuations using global contrast. For the steadily changing sea surface, a relatively simple background-updating method is adopted. For the violently fluctuating sea surface, a background-updating method with a variable learning rate is adopted. The preliminary background model used in the first stage is suitable for the steadily changing scene, which generates accurate pixel-level foreground masks through the background sub-block filter and the posterior probability criterion. The precise background model used in the second stage is suitable for highly fluctuating scenes, and the model is maintained using the modified learning rate. In addition, the background subtraction result is filtered using the local linear correlation feature, which is helpful for detecting the foreground in various scenes.

The model has better applicability to different sea surfaces based on the concept of sea surface classification. Moreover, based on the concept of combining background subtraction with posterior probability judgment, the background model can effectively suppress various types of sea clutter without the need for excessive subsequent morphological processing. The work conducted in this study has certain reference value for the theoretical development of maritime infrared sequence image target detection. The main highlights of this paper are three-fold:

• A dual-mode background modeling approach which can accurately detect foregrounds in various sea scenes is proposed. For the steadily changing scene, background sub-block filtering based on the pixel property combined with a posterior criterion is adopted. For the violently fluctuating scene, background sub-block filtering based on the regional property combined with a posterior criterion is adopted.
• The global contrast of the image is adopted to judge whether it belongs to a steadily changing scene or a highly fluctuating scene.
• The local correlation feature between the test frame and the updated background is proposed to suppress the local sea clutter.

The rest of the paper is organized as follows. Section 2 explains the proposed approach in detail. In Section 3, the proposed approach is evaluated and compared with other state-of-the-art methods. Finally, in Section 4, we draw our conclusions.

2. Method

The method proposed basically includes three major steps: the modeling of the background, filtering in the frequency domain and the detection of the foreground. A flowchart of the proposed method is shown in Figure 1. Details are provided in the following sections.

2.1. Initialization of the Sea Background Model

Considering that the amplitude spectrum sequence of a maritime infrared image sequence is more stable than the gray value sequence, background construction is carried out in the Fourier domain [23]. First, one transforms each frame of the infrared maritime sequence image to the Fourier domain in the row direction. Then, the amplitude spectrum of the kth frame is obtained. Let $I_k(x,y)$ be the kth frame with a size of $M\times N$ and $A_k(x,y)$ be the amplitude spectrum of the kth frame, where $x=0,1,\dots ,M$ and $y=0,1,\dots ,N$.

Suppose that there are n frames of continuous pure seawater images for background model training and the value of n is related to the length of the captured video. To initialize the background model, the first frame of the sequence image is used. The mean parameter is initialized as the amplitude spectrum of the first frame, and the variance parameter is initialized as a zero matrix. That is, $B_1=A_1$ and ${\sigma }_1={\left[\mathbf{0}\right]}_{M\times N}$.

When the kth pure seawater image is read in, the model is updated as follows:

$$B_k=[(k-1)·B_{k-1}+A_k]/k$$

(1)

$${\sigma }_k=[{(A_k-B_k)}^2+(k-2)·{\sigma }_{k-1}]/(k-1)+{(B_k-B_{k-1})}^2$$

(2)

Here, $B_k$ and ${\sigma }_k$ are the mean and variance parameter of the kth updated background model, respectively.

2.2. The Preliminary Background Model

The learning rate parameter is important for the background model, which determines the accuracy of the updated background. Since the sea surface changes randomly, the learning rate parameter of the background model should not be constant. Assuming that the change in the sea surface is relatively stable, the learning rate value is first set as a constant. The learning rate value is then adjusted according to the area difference in the detected target between two adjacent test frames.

For the steadily changing sea surface, when the target appears, there will be a clear difference in the amplitude spectrum between the target area and the sea background area. Thus, we use the change in the amplitude spectrum of each frequency point relative to the background, i.e., the amplitude spectrum difference, as the key feature for the classification process.

For each frequency point $(i,j)$ in a local sub-block with a size of $r\times r$, the amplitude spectrum difference between the test frame and the background model is calculated as follows:

$$▵A(i,j)=\mid B(i,j)-A_k(i,j)\mid$$

(3)

Each sub-block consisting of $▵A(i,j)$ is marked as a region of interest (RoI). The template value of the background block filter is set as $\lambda$ times the standard deviation of the sequence, namely,

$$BF(i,j)=\lambda ·\sqrt{{\sigma }_k(i,j)}$$

(4)

A subtraction operation is performed on each RoI and the background filter value $BF$, and the response value of the filter is 1 if the difference is positive, namely

$$R(i,j)=1,if▵A(i,j)-BF(i,j)>0$$

(5)

For the steadily changing sea surface, the amplitude spectrum at location $(i,j)$ deviates greatly from that of the sea background if $R(i,j)$ is equal to 1. Otherwise, it is considered that the amplitude spectrum at location $(i,j)$ is close to that of the sea background. However, for scenes with large waves, some points meeting Equation (5) belong to the sea background.

To eliminate the interference of these sea clutter points, a posterior probability criterion is used. For each frequency point $(i,j)$ with an amplitude spectrum difference $▵A(i,j)$ in the local sub-block, the posterior probability of it being the foreground can be described as follows:

$$P\left(f\right|▵A(i,j))=\left\{\begin{array}{cc}1,\hfill & \hfill \mathrm{if}m·\sum _{(i,j)\in RoI}R(i,j)>r\times r,\\ 0,\hfill & \hfill otherwise.\end{array}\right.$$

(6)

Here, m is an adjustable parameter, and its value is generally set as $[2,4]$ for different scenes. The posterior probability $P\left(f\right|▵A(i,j))$ constitutes the preliminary binary detection image in the Fourier domain.

For the background update procedure, we follow a selective update strategy, as follows:

$$B_k=(1-a)·B_{k-1}+a·(1-P\left(f\right|▵A))·A_k$$

(7)

$$\begin{array}{c}\hfill {\sigma }_k=(1-a)·{\sigma }_{k-1}+a·(1-P\left(f\right|▵A))·{(A_k-{\mu }_k)}^2\end{array}$$

(8)

Here, the learning rate a is assumed to be an adjustable constant.

2.3. Adjustment of the Learning Rate of the Background Model

Due to the continuity of the sea scene over a short time, if the detection results of two adjacent frames differ greatly, this indicates that there are clear errors in the detection results of the two frames, and the learning rate value of the background model needs to be modified. That is, the learning rate is set to a larger value to incorporate the latest information on the sea scene. Otherwise, if the detection results of two adjacent frames are close, this indicates that the sea scene changes stably, and the learning rate value need not be changed.

Let $S_k$ be the connected area of each frequency point in the binary image $P\left(f\right|▵A)$ of the k-th test frame. Thus, the modification strategy for the learning rate value is as follows:

$${\alpha }_k(i,j)=\left\{\begin{array}{cc}1,\hfill & \hfill \mathrm{if}|S_k(i,j)-S_{k-1}(i,j)|>T,\\ a,\hfill & \hfill otherwise.\end{array}\right.$$

(9)

Here, ${\alpha }_k(i,j)$ denotes the learning rate value of the background model at location $(i,j)$ for the k-th test frame, and T is an adjustable threshold.

2.4. The Precise Background Model

The preliminary background model is useful for classifying sea waves and targets for the steadily changing sea surface. However, for the violently fluctuating sea surface, the preliminary background model needs to be modified to obtain better detection effects. Thus, global contrast is used to determine whether the sea surface belongs to a steadily changing scene or a highly fluctuating scene. The global contrast is defined as follows:

$$C=\mu ·\sqrt{\sum _{(x,y)}{(I_k(x,y)-\mu )}^2}$$

(10)

Here, $\mu$ denotes the average gray value of the k-th test frame. If C is greater than the threshold $C_h$ ($C_h$ is a user-settable threshold for global contrast), it is considered that the seawater fluctuates violently, and the preliminary background model is modified.

Within the local sub-block centered at point $(i,j)$, the feature difference between the test frame and the background model is calculated as follows:

$$▵A(i,j)=\mid \overline{B}(i,j)-{\overline{A}}_k(i,j)\mid$$

(11)

Here, ${\overline{A}}_k$ is the mean of the k-th test frame’s amplitude spectrum in the sub-block, and $\overline{B}$ is the mean of the updated background’s mean parameter in the sub-block.

Each sub-block consisting of $▵A(i,j)$ is marked as a region of interest (RoI). The template value of the background block filter is set as ${\lambda }_1$ times the standard deviation of the updated background’s mean parameter in the sub-block, namely,

$$BF(i,j)={\lambda }_1·v_k(i,j)$$

(12)

A subtraction operation is performed on each RoI and the background filter value $BF$, and the response value of the filter is 1 if the difference is positive in Equation (5). Similarly, for each frequency point $(i,j)$ with an amplitude spectrum difference $▵A(i,j)$ in the local sub-block, the posterior probability of each point being foreground is described in Equation (6).

Background maintenance must meet the requirement that the model can absorb the random variations in the sea scene. To accurately update the background model, the modified learning rate is used.

If $|A_k(i,j)-A_{k-1}(i,j)|>BF(i,j)$, the sea scenes at location $(i,j)$ of the two consecutive frames are clearly different, and the model is updated as follows:

$$B_k(i,j)=A_k(i,j)$$

(13)

$$\begin{array}{c}\hfill {\sigma }_k(i,j)=(1-{\alpha }_k(i,j))·{\sigma }_{k-1}(i,j)+{\alpha }_k(i,j)·(1-P\left(f\right|▵A(i,j)))·{(B_k(i,j)-B_{k-1}(i,j))}^2\end{array}$$

(14)

If $|A_k(i,j)-A_{k-1}(i,j)|\le BF(i,j)$, the sea scenes at location $(i,j)$ of the two consecutive frames are relatively close, and the model is updated as follows:

$$B_k(i,j)=(1-{\alpha }_k(i,j))·B_{k-1}(i,j)+{\alpha }_k(i,j)·(1-P\left(f\right|▵A(i,j)))·A_k(i,j)$$

(15)

$$\begin{array}{c}\hfill {\sigma }_k(i,j)=(1-{\alpha }_k(i,j))·{\sigma }_{k-1}(i,j)+{\alpha }_k(i,j)·(1-P\left(f\right|▵A(i,j)))·{(A_k(i,j)-B_k(i,j))}^2\end{array}$$

(16)

Here, ${\alpha }_k(i,j)$ denotes the learning rate of the background model at location $(i,j)$ for the k-th test frame, $P\left(f\right|▵A)$ is the binary detection result, and $B_k(i,j)$ and ${\sigma }_k(i,j)$ are the mean and variance at location $(i,j)$ of the k-th updated background, respectively.

2.5. Local Correlation Feature between the Test Frame and the Background

Due to the local similarity between seawater regions, the similarity between the local seawater area of the test frame and the corresponding updated background area is strong, while the similarity between the local target area of the test frame and the corresponding updated background area is weak. Therefore, the local correlation feature between the test frame and the updated background can enhance the local target area and suppress the local sea clutter.

$A_k$ and $B_k$ are divided into small sub-blocks with sizes of $p\times s$ each, and each sub-block is transformed into the 2D Fourier domain, namely,

$$\begin{array}{c}\hfill {\mathbb{B}}_i(u,v)=FF{T}_2\left(B_k^i(x,y)\right),(u=0,1,\cdots ,p-1;v=0,1,\cdots ,s-1)\end{array}$$

(17)

$$\begin{array}{c}\hfill O_i(u,v)=FF{T}_2\left(A_k^i(x,y)\right),(u=0,1,\cdots ,p-1;v=0,1,\cdots ,s-1)\end{array}$$

(18)

Here, $FF{T}_2(·)$ denotes the 2D Fourier transformation, and $B_k^i$ and $A_k^i$ denote the ith sub-block of $B_k$ and $A_k$, respectively. ${\mathbb{B}}_i(u,v)$ and $O_i(u,v)$ denote the 2D Fourier frequency spectra of $B_k^i(x,y)$ and $A_k^i(x,y)$, respectively.

The local linear correlation of $B_k^i$ and $A_k^i$ is calculated as follows:

$$L_i(x,y)=IF{T}_2(\frac{{\mathbb{B}}_i(u,v)·O_i^{*}(u,v)}{|{\mathbb{B}}_i(u,v)·O_i^{*}(u,v)|})$$

(19)

Here, $IF{T}_2(·)$ denotes the 2D inverse Fourier transformation, and $O_i^{*}(u,v)$ denotes the conjugate of $O_i(u,v)$.

Then, the local correlation feature between $B_k^i$ and $A_k^i$ is obtained by a reverse transformation:

$$L{C}_i(x,y)=max\left\{L_i(x,y)\right\}-L_i(x,y)$$

(20)

Here, $L_N$ is the number of sub-blocks of the test frame. Clearly, the local correlation feature between the seawater sub-block of the test frame and the sub-block of the updated background is small. Similarly, the local correlation feature between the target sub-block of the test frame and the sub-block of the updated background is large. Therefore, feature $LC$ is regarded as a filter in the Fourier domain.

$$LC(x,y)=\left\{L{C}_i(x,y)\right\},i=1,2,\cdots ,L_N$$

(21)

where $L_N$ is the number of sub-blocks of the test frame. Obviously, the local correlation feature between the seawater sub-block of the test frame and the sub-block of the updated background is small. Similarly, the local correlation feature between the target sub-block of the test frame and the sub-block of the updated background is large. Therefore, feature $LC$ is regarded as a filter in the Fourier domain.

The filtering process of the background subtraction result $P\left(f\right|▵A)·A_k$ by filter $LC$ is shown in Figure 2. Figure 2a shows the original test frames. Figure 2b shows the amplitude spectrum of the original test frames. Feature $LC$ between the test frame and the updated background is shown in Figure 2c. Figure 2d shows the filtering results of the background subtraction result $P\left(f\right|▵A)·A_k$ in the Fourier domain. The filtered images in the spatial domain are shown in Figure 2e. It can be seen from Figure 2e that the hull in the filtered images using filter $LC$ is relatively complete, and the image contrast is much higher than that of the original test frames. Thus, feature $LC$ is conducive to improving the local contrast of the test frame.

2.6. Foreground Detection

The local correlation feature $LC$ can enhance the local contrast of the test frame; thus, it is used to filter the background subtraction result $P\left(f\right|▵A)·A_k$ in the Fourier domain. Then, one applies inverse Fourier transformation to the filtering result to obtain the detection result in the spatial domain, namely,

$$fr=IFT(P\left(f\right|▵A)·A_k·LC·exp\left({\mathsf{\Phi }}_k\right))$$

(22)

Here, $IFT(·)$ denotes the inverse Fourier transformation in the row direction, and $A_k$ and ${\mathsf{\Phi }}_k$ denote the amplitude spectrum and phase spectrum of the kth test frame, respectively.

However, a small amount of sea clutter still remains in the preliminary detection result $fr$. Therefore, the salient target area is extracted, and the significance map of the preliminary detection result $fr$ is obtained [23]. One builds a new image using the significance maps of both the preliminary detection result $fr$ and the original test frame, as follows:

$$img1=p_1·imgs+p_2·img$$

(23)

Here, $imgs$ and $img$ are the significance maps of the preliminary detection result $fr$ and the original test frame, respectively. Additionally, $imgs$ and $img$ have the same significant target area, and $p_1$ and $p_2$ are two adjustable parameters.

Then, threshold segmentation is carried out for $img1$ to obtain the final detection result. The threshold is set as follows:

$$th=\tau ·meanb$$

(24)

Here, $meanb$ is the mean value of $img1$. $\tau$ is a control parameter, and its range is $(0,1)$. The foreground region is obtained as follows:

$$fo=\left\{\begin{array}{cc}1,\hfill & \hfill \mathrm{if}img1>th,\\ 0,\hfill & \hfill otherwise.\end{array}\right.$$

(25)

3. Experimental Results

3.1. Results and Analysis

Figure 3 and Figure 4 show the detection results of maritime infrared sequence images obtained using the proposed method. Figure 3a and Figure 4a show the original test frames from different maritime infrared scenes. It can be seen that the sea scenes have different contrasts and various types of sea clutter; hence, it is challenging to accurately detect the targets. Figure 3b and Figure 4b show the preliminary detection results before segmentation. As seen from Figure 3b and Figure 4b, most of the complex and diverse backgrounds have been effectively suppressed, and the targets have been accurately located in the scene. Figure 3c and Figure 4c show the fusion results of the significance maps of both the preliminary results and the original images. It can be seen that the target area is enhanced and that the sea clutter is further suppressed using the fusion strategy. The detection results in Figure 3d and Figure 4d demonstrate that most of the sea clutter is removed and that the ship targets are accurately detected in various scenes.

3.2. Comparison of the Results

We compared the proposed approach with the state-of-the-art techniques using maritime videos that were captured by our research group using an infrared thermographer. The visual comparison results are shown in Figure 5 and Figure 6, demonstrating that the proposed method outperforms the comparative methods, including SuBSense [29], LSDS [30], RRBM [21], LOBSTER [27], PBAS [28], FASGM [23] and BMMFF [24] in terms of visual effects. Figure 5a and Figure 6a show different maritime infrared scenes with different targets and seawater fluctuations. The results of SuBSense, LSDS, RRBM, LOBSTER, PBAS, FASGM, BMMFF and the proposed method are shown in Figure 5b and Figure 6b, Figure 5c and Figure 6c, Figure 5d and Figure 6d, Figure 5e and Figure 6e, Figure 5f and Figure 6f, Figure 5g and Figure 6g, Figure 5h and Figure 6h, and Figure 5i and Figure 6i, respectively.

4. Experimental Analysis

4.1. Qualitative Comparison and Discussion

The SuBSense method performs well for images with smaller waves and higher contrast, such as those in the fifth and sixth rows in Figure 6b. However, partial sea clutter is incorrectly classified as the target in the high sea clutter scene, as in the third, seventh and eleventh rows in Figure 6b. In addition, this method fails if the target is dark and weak, as in the third, fifth, seventh and ninth rows in Figure 5b.

For dark scenes with positive local contrast, LSDS can basically extract the weak target, as in the first, third and fifth rows in Figure 5c. However, for the dark scenes with weaker local contrast, some of the targets are misclassified as the background components, as shown in the sixth and seventh rows in Figure 5c. In addition, the representative image is defined as the median of the images in the buffer, which is not accurate for highly dynamic scenes. Thus, the detection effect is poor for the scene with brighter and larger sea waves, as shown in the third, seventh and eleventh rows in Figure 6c.

The RRBM method utilizes the robust block-level feature for modeling the background. However, the block intensity variance feature still cannot represent an accurate sea background for varying sea clutter. For example, in the fourth and seventh rows of Figure 6d, much sea clutter is incorrectly classified as the target. The detected targets are incomplete for the dim scenes with uneven targets, as in the second, ninth and twelfth rows in Figure 5d and the ninth row in Figure 6d. In addition, the partial dim hull is missed in the scenes with weak targets, such as the sixth row in Figure 5d.

The LOBSTER method is based on a modified ViBe model and improved local binary similarity feature, which is sometimes unsuitable for noisy or blurred regions. Such a case is illustrated in the third, seventh and eleventh rows of Figure 6e. It can be noted that many pixels are falsely labeled as foreground, since the improved local binary similarity vectors of the background model and the high dynamic sea clutter are still quite different. Moreover, the detected hull of dim targets in low-contrast scenes is incomplete compared with the real ship, as in the third, fifth and seventh rows in Figure 5e. This is due to the non-representative feature used for the local comparisons.

The PBAS method is effective for scenes with high contrast, such as those in the first and eighth rows in Figure 6f. However, the ever-changing sea waves seriously affect the detection performance. While the targets are detected, some waves are not removed, as can be seen from the third and eleventh rows in Figure 6f. Moreover, the targets in the dark and weak sea scenes are not completely detected, as in the seventh, ninth and eleventh rows in Figure 5f and the ninth row in Figure 6f. The targets are missed in the second, third and sixth rows in Figure 5f.

The FASGM method is effective for most different sea clutter scenes, as shown in Figure 5g and Figure 6g. However, its detection performance is still slightly inferior to that of the proposed method. The FASGM method has lower detection integrity for targets than the proposed method, as shown in the first to fourth, seventh and eleventh rows in Figure 5g. In addition, a small portion of the sea clutter is enhanced along with the target, as in the ninth and twelfth rows in Figure 5g and the third and fourth rows in Figure 6g.

The BMMFF method can successfully segment targets in most sea scenes and filter out most of the sea clutter. However, a significant number of detected targets are missing, and some sea clutter remains in the results, as shown in the seventh and eighth rows of Figure 5h. In some scenes with large sea waves, a small amount of sea clutter still remains in the detected results, as in the fourth and tenth rows in Figure 5h and the fourth, eighth and twelfth rows in Figure 6h.

The proposed method can accurately detect different targets in different sea scenes, as shown in Figure 5i and Figure 6i. As can be seen from the third, seventh and eleventh rows in Figure 6i, the method can detect targets in sea scenes with violent fluctuations better than the other comparative methods. From the second, third, seventh and eleventh rows in Figure 5i and the ninth row in Figure 6i, it can be seen that the detection results of the uneven dim targets show more details and are much closer to the actual targets. The sixth row in Figure 5i shows that the method can accurately detect the weak targets in the challenging sea scene. The reason for the good performance is that the proposed method uses different and effective modeling strategies for seawaters with different fluctuations, rendering it applicable to various sea scenes. By using the background sub-block filter and the posterior probability criterion, the sea clutter is effectively suppressed. In addition, the adjustment of the learning rate is introduced, and the local linear correlation feature is used to enhance the local contrast of the test frame. Thus, the proposed method achieves a good detection result.

4.2. Quantitative Comparison

For the quantitative evaluation of test results, three measures are used: the misclassification error ($ME$) [36], relative foreground area error ($RAE$) [36] and f-measure values ($FM$) [20]. $ME$ denotes the percentage of background pixels incorrectly classified as the foreground and foreground pixels incorrectly classified as the background. $RAE$ denotes the area accuracy of the detected result compared with the true image. The $FM$ is defined as the harmonic mean of the precision ($Pr$) and recall ($Re$). The three measures are defined as follows:

$$ME=1-\frac{|B_o\bigcap B_T|+|F_o\bigcap F_T|}{|B_o|+|F_o|}$$

(26)

$$RAE=\left\{\begin{array}{cc}\frac{A_o-A_T}{A_o},\hfill & \hfill \mathrm{if}{A}_o>A_T;\\ \frac{A_T-A_o}{A_T},\hfill & \hfill \mathrm{otherwise}.\end{array}\right.$$

(27)

$$FM=2·\frac{Pr·Re}{Pr+Re}$$

(28)

$$Pr=\frac{TruePositive}{TruePositive+FalsePositive}$$

(29)

$$Re=\frac{TruePositive}{TruePositive+FalseNegative}$$

(30)

Here, $B_o$ and $F_o$ denote the background and foreground pixels of the original image, respectively. $B_T$ and $F_T$ denote the background and foreground pixels of the detected result, respectively. $|·|$ denotes the cardinality of a set. $A_o$ and $A_T$ denote the areas of the true target and detected target, respectively. $Pr$ is the proportion of the correct classification of ship pixels to all actual detected pixels. The $Re$ is the proportion of correct classifications of ship pixels relative to the total true ship pixels.

Figure 7 shows detailed comparisons of twelve representative scenes in $ME$, $RAE$ and $FM$. We find that our background model approach has a superior performance compared with the other classical background modeling approaches for most of complex scenes. The $ME$, $RAE$ and $FM$ performances of the other comparison algorithms are poor for some scenes with serious interference. On the whole, the proposed method offers the most stable performance for different scenes.

Table 1. The average values of three metrics of comparison methods.

Methods	AME	ARAE	AFM
SuBSENSE	0.0186	0.4310	0.5121
LSDS	0.0121	0.4837	0.5422
RRBM	0.0203	0.4739	0.4579
LOBSTER	0.0378	0.4700	0.4073
PBAS	0.0125	0.5781	0.4407
FASGM	0.0059	0.1067	0.7544
BMMFF	0.0065	0.1293	0.7216
Proposed	0.0049	0.0979	0.8281

lists the average values of three metrics for twelve scenes, and $AME$, $ARAE$ and $AFM$ represent the average values of $ME$, $RAE$ and $FM$, respectively. It can be seen that the proposed method has the lowest average $ME$ and $RAE$, and highest average $FM$ value compared with the other methods. The average value of $RAE$ for the proposed method is reduced by approximately $33.31\%$ compared with SuBSENSE. The average value of $ME$ for the proposed method is reduced by approximately $0.72\%$ compared with LSDS, while the average $FM$ value of the proposed method is increased by approximately $28.59\%$ compared with LSDS. The average values of $ME$ and $RAE$ for the proposed method are reduced by approximately $0.1\%$ and $0.88\%$ compared with FASGM, respectively, while the average value of $FM$ for the proposed method is increased by approximately $7.37\%$ compared with FASGM, which demonstrates that the proposed method performs better overall.

The variance values of three metrics for twelve scenes are reported in

Table 2. The variance values of three metrics of comparison methods.

Methods	VME	VRAE	VFM
SuBSENSE	4.6642 × 10−4	0.0964	0.0686
LSDS	7.8767 × 10−5	0.0589	0.0721
RRBM	4.3311 × 10−4	0.0670	0.0492
LOBSTER	0.0027	0.1162	0.1043
PBAS	9.0512 × 10−5	0.1145	0.0856
FASGM	1.7060 × 10−5	0.0106	0.0145
BMMFF	2.3645 × 10−5	0.0144	0.0158
Proposed	1.6279 × 10−5	0.0101	0.0135

, and $VME$, $VRAE$ and $VFM$ represent the variance values of $ME$, $RAE$ and $FM$, respectively. It can be seen that the proposed method has the lowest variance values of $ME$, $RAE$ and $FM$ compared with the other methods. The variance values of $ME$ and $RAE$ for the proposed method are reduced by approximately $0.0062\%$ and $4.88\%$ compared with LSDS, respectively, while the variance value of $FM$ for the proposed method is reduced by approximately $3.57\%$ compared with RRBM. The variance values of $RAE$ and $FM$ for the proposed method are reduced by approximately $0.05\%$ and $0.1\%$ compared with FASGM, respectively, which demonstrates that the proposed method is the most stable and robust for various scenes in terms of $RAE$ and $FM$. The variance value of $ME$ for the proposed method is reduced by approximately $0.00008\%$ compared with FASGM, which demonstrates that the two methods perform similarly on stability in term of $ME$.

5. Conclusions and Future Work

In this paper, we introduced a novel dual-mode background modeling technique for the foreground detection of maritime infrared scenes. First, a rule based on global contrast is adopted to judge whether the scene is a steadily changing scene or a highly fluctuating scene. For the steadily changing scene, background sub-block filtering, based on the pixel property combined with a posterior probability criterion, is adopted for foreground detection. For the violently fluctuating scene, background sub-block filtering, based on the regional property combined with a posterior probability criterion, is adopted for foreground detection. Moreover, the learning rate of the background model is modified using the area difference of two adjacent frames’ connected regions. Moreover, the local correlation feature between the test frame and the updated background is proposed to suppress the local sea clutter. It achieves robustness for scenes with large waves and dark seawater. This can be attributed to our robust background model and our classification framework that effectively detects the foreground in different scenes. The experimental results of different sea scenes confirm that the proposed method achieves a higher accuracy than the state-of-the-art techniques. Thus, the proposed algorithm has good application value in the field of maritime intelligent object detection and is convenient for coastal defense. Moreover, this research has important reference value for the development of and improvement in target detection theory using maritime infrared images.

In the future, we will aim to explore the deep background subtraction approach with a deep neural network so as to improve the detection performance and applicability of the approach for different types of scenes, such as sea scenes with complex cloud interference and sea surfaces with extremely uneven brightness.