Infrared Small-Faint Target Detection Using Non-i.i.d. Mixture of Gaussians and Flux Density

Abstract

The robustness of infrared small-faint target detection methods to noisy situations has been a challenging and meaningful research spot. The targets are usually spatially small due to the far observation distance. Considering the underlying assumption of noise distribution in the existing methods is impractical; a state-of-the-art method has been developed to dig out valuable information in the temporal domain and separate small-faint targets from background noise. However, there are still two drawbacks: (1) The mixture of Gaussians (MoG) model assumes that noise of different frames satisfies independent and identical distribution (i.i.d.); (2) the assumption of Markov random field (MRF) would fail in more complex noise scenarios. In real scenarios, the noise is actually more complicated than the MoG model. To address this problem, a method using the non-i.i.d. mixture of Gaussians (NMoG) with modified flux density (MFD) is proposed in this paper. We firstly construct a novel data structure containing spatial and temporal information with an infrared image sequence. Then, we use an NMoG model to describe the noise, which can be separated with the background via the variational Bayes algorithm. Finally, we can select the component containing true targets through the obvious difference of target and noise in an MFD maple. Extensive experiments demonstrate that the proposed method performs better in complicated noisy scenarios than the competitive approaches.

1. Introduction

Distant and faint target detection is of great importance to infrared systems, as anti-missile techniques and early-warning systems. Due to the unique characteristic of these military tasks, the targets need to be detected accurately as early as possible in the infrared search and track systems to provide ample time for deployment and striking back. However, the target usually occupies only a few pixels and lacks texture information due to the very far observation distance. The backgrounds are very complex, including sky background and sea–sky background, which means the acquired infrared images are usually contaminated by a clutter background and a varying noise. The contrast between targets, background and the varying noise might be very poor. The low signal-to-clutter ratio (SCR) and signal-to-noise ratio (SNR) make the infrared targets very faint. Therefore, robust infrared small and faint target detection technique remains a valuable research hotspot [1,2,3].

To achieve a satisfying target detection performance, many approaches have been proposed for different scenarios, including two types: Track-before-detection (TBD) approaches [4,5] and detection-before-track (DBT) approaches [6,7,8]. TBD approaches have good detection performance for targets with continuous track motion, such as 3D matched filters [9] and its improved versions [10,11]. DBT approaches focus on suppressing the clutter background while enhancing the target in single frame, and are more efficient than TBD approaches. TBD approaches are widely used in practical engineering. At present, the common types of DBT methods are filtering, human visual system (HVS) and multi-feature based approaches. Filtering methods analyze spatial continuity of an input image, and the target is modeled as a break point, such as max-median filter [12], top-hat filter [13] and 2D least mean square (TDLMS) filter [14]. HVS based methods [15,16,17] assume that there is a significant contrast between background and target regions. Multi-feature based methods [18,19,20] represent the target characteristics and background region with features used to train the classifiers.

Moreover, the low-rank and sparse component recovery based approach, as a subdiscipline of the low-rank representation (LRR) [21], has become very popular in recent years. In this approach, the background regions are assumed to change gradually, and a special low-rank data structure can be constructed with the original images, such as a 2D matrix and a 3D tensor. With the recovery of the low-rank background, the dim target can be separated from the original image. Gao proposed an infrared image-patch (IPI) model [22], which constructs a low-rank matrix by sliding window. The IPI model uses vanilla nuclear norm minimization (NNM) [23] and $l_1$ [24] to regularize the background and the target, respectively. The performance of NNM in a low-rank component estimation problem would degrade in a noisy scenario. The solution for this problem is to replace NNM with a more suitable regularizer. Thus, Dai proposed a weighted IPI approach [25] and a non-negative IPI approach [26], and Guo proposed a reweighted WIPI model (ReWIPI) based on weighted nuclear norm minimization (WNNM) [27]. In the view of the dimension of data, Dai proposed a reweighted infrared patch-tensor (RIPT) method to generalize the low-rank matrix to low-rank tensor for mining more spatial information [28]. However, the RIPT method unfolds the background patch tensor as three matrices and regularizes it via the sum of nuclear norms (SNN) [29], which is suboptimal and inefficient. To remedy this issue, Sun proposed a weighted tensor nuclear norm with IPT (WNRIPT) method [30].

However, most of the existing low-rank component recovery based approaches [22,25,26,27,28,30] only use the Frobenius loss term [31] to constrain the noise, which models the noise as an independent and identically distributed (i.i.d.) Gaussian distribution. In practical applications, the infrared images usually include complex instrumental noise that degrades the performance for target detection. The complex noise degrades the performance of the target detection severely. A robust method, capable of distinguishing different kinds of noise, is needed.

To this end, a state-of-the-art method [32] digs out valuable information in time domain and uses a mixture of Gaussian (MoG) noise models [33] to model the target component and noise component together. The MoG model characterizes each pixel in the image and updates the mixed Gaussian model after the new image is acquired. It matches each pixel in the current image with the MoG model, and the matched pixels are classified into background regions [34,35]. Finally, the Markov random field (MRF) method [34] is used to detect the target. However, the noise distribution in different frames is modeled as i.i.d. MoG distributions substantially in [32], which is not suitable for complex noisy scenarios. In addition, the MRF model does not provide a robust noise estimate in complex scenarios, since its performance is based on the assumption that the noise component does not arise in the neighborhood region of the targets. However, the noise permeates through the whole image, including the target.

We propose a small and faint target detection approach based on a non-i.i.d. MoG (NMoG) model [36] and modified flux density (MFD) maple [37]. The noise distributions in different frames (sequences of images) is assumed to follow non-i.i.d. for improving the robustness in real scenarios. The target is considered as a kind of noise extracted from the background via an NMoG low-rank matrix factorization (NMoG-LRMF) model, solved by a variational Bayes (VB) algorithm. In a second step, the MFD maple [37] method is used to distinguish the true target from the noise, accounting for the fact that target flux density differs from the noise in infrared gradient vector field.

This paper is organized as follows. The proposed method is described in Section 2. Section 3 provides the experimental results to validate the effectiveness of the proposed method. Finally, we conclude our work in Section 4.

2. The Proposed Model

2.1. Spatio-Temporal Patch Model

Given an infrared image sequence, we can get a 3D cube patch tensor by storing each frame into its slice. We vectorize each slice and get a 2D matrix. The procedure is given in Figure 1. Note that it is possible to reconstruct the image sequence from the processed 2D matrix via inverse operation. Assume an infrared image sequence $f_1,f_2,\cdots ,f_P\in {\mathbb{R}}^{m\times n}$ transformed into a matrix $\mathit{F}$ with size of $N\times P$, where $N=m\times n$ and P denote the spatial and temporal dimensions. We divide $\mathit{F}$ into background component $\mathit{B}$ and noise $\mathit{E}$, described as:

$$\mathit{F}=\mathit{B}+\mathit{E},$$

(1)

and the small-faint target component $\mathit{T}$ is considered as a sparse noise component in $\mathit{E}$ [32].

2.2. Background Component

In low-rank recovery based methods, background regions are assumed to vary slowly, and there are a lot of repeated regions. The low-rank matrix $\mathit{B}$ [32] is modeled as follows:

$$\mathit{B}=\mathit{U}{\mathit{V}}^T=\sum _{l=1}^R{\mathit{u}}_{·l}{\mathit{v}}_{·l}^T,$$

(2)

where $\mathit{U}\in {\mathbb{R}}^{N\times R}$ and $\mathit{V}\in {\mathbb{R}}^{P\times R}$, and their l-$\mathrm{th}$ columns are represented as ${\mathit{u}}_{·l}$ and ${\mathit{v}}_{·l}$. R is the initial rank of $\mathit{B}$. The intrinsic low-rank nature of $\mathit{B}$ is guaranteed by assuming ${\mathit{u}}_{·l}$ and ${\mathit{v}}_{·l}$ generated according to a Gaussian distribution:

$${\mathbf{u}}_{·l}\sim N\left({\mathbf{u}}_{·l}\left|0,{\gamma }_l^{-1}{I}_N\right.\right),{\mathbf{v}}_{·l}\sim N\left({\mathbf{v}}_{·l}\left|0,{\gamma }_l^{-1}{I}_P\right.\right),$$

(3)

where ${\mathit{I}}_N\left({\mathit{I}}_P\right)$ is the $N\times N\left(P\times P\right)$ identity matrix. ${\gamma }_l$ denotes the precision of ${\mathit{u}}_{·l}$ and ${\mathit{v}}_{·l}$ that satisfies:

$${\gamma }_l\sim \mathrm{Gam}\left({\gamma }_l\left|{\xi }_0,{\delta }_0\right.\right),$$

(4)

where $\mathrm{Gam}\left({\gamma }_l\left|{\xi }_0,{\delta }_0\right.\right)$ represents a gamma distribution, and ${\xi }_0,{\delta }_0$ are scales. The low-rank component can be estimated accurately by this model [38].

2.3. Noise Component

In [32], the noise of different frames are assumed to be i.i.d., which is not practical in real and complex scenarios. Thus, we use the NMoG model [36] to model the noise distributions in different frames, namely noise distribution of images in different frames are nonidentical. The $ij$-$\mathrm{th}$ element of the noise $\mathbf{E}$ can be divided into K components as below:

$$e_{ij}\sim \sum _{k=1}^K{\pi }_{jk}N\left(e_{ij}\left|{\mu }_{jk},{\tau }_{jk}^{-1}\right.\right),$$

(5)

where ${\pi }_{jk}$ denotes the mixing proportion that is non-negative, and ${\sum }_{k=1}^K{\pi }_{jk}=1$. ${\mu }_{jk}$ and ${\tau }_{jk}$ denote mean and precision, respectively. Instead of setting the MoG parameters, i.e., ${\pi }_{jk}$, ${\mu }_{jk}$ and ${\tau }_{jk}$, as unchanging value for k-$\mathrm{th}$ Gaussian component, we vary them in different frames. Equation (5) can be equivalently expressed as a two-level generative model by introducing the indicator variables ${\mathbf{z}}_{ij}$. ${\mathbf{z}}_{ij}$ is the hidden variable generated from Multinomial distribution with parameter ${\mathit{\pi }}_j$:

$$\begin{array}{cc}\hfill e_{ij}& \sim \prod _{k=1}^K{N\left(e_{ij}\left|{\mu }_{jk},{\tau }_{jk}^{-1}\right.\right)}^{z_{ijk}}\hfill \\ \hfill {\mathit{z}}_{ij}& \sim \mathrm{Multinominal}\left({\mathit{z}}_{ij}\left|{\mathit{\pi }}_j\right.\right)\hfill \end{array}$$

(6)

where ${\mathit{z}}_{ij}=\left(z_{ij1},\dots ,z_{ijK}\right)\in {\left\{0,1\right\}}^K$, ${\sum }_{k=1}^K{z}_{ijk}=1$. Multinomial( ) represents the multinomial Dirichlet distribution. The conjugate priors of ${\mu }_{jk}$, ${\tau }_{jk}$ and the mixing proportions ${\mathit{\pi }}_j=[{\pi }_{j1},\dots ,{\pi }_{jK}]$ are also defined for completing the Bayesian model:

$$\begin{array}{cc}\hfill {\mu }_{jk},{\tau }_{jk}& \sim N\left({\mu }_{jk}\left|m_0,{\left({\beta }_0{\tau }_{jk}\right)}^{-1}\right.\right)\mathrm{Gam}\left({\tau }_{jk}\left|c_0,d\right.\right)\hfill \\ \hfill d& \sim \mathrm{Gam}\left(d\left|{\eta }_0,{\lambda }_0\right.\right)\hfill \\ \hfill {\mathit{\pi }}_j& \sim \mathrm{Dir}\left({\mathit{\pi }}_j\left|{\mathit{\alpha }}_0\right.\right)\hfill \end{array}$$

(7)

where ${\beta }_0,m_0,c_0,d$ are the hyper-parameters, and d satisfies Gam distribution with hyper-parameters ${\eta }_0,{\lambda }_0$. Dir(.) is a Dirichlet distribution parameterized by ${\mathbf{\alpha }}_0=\left({\alpha }_{01},\dots ,{\alpha }_{0K}\right)$. Then, the noise component can be modeled by Equations (6) and (7).

Combining Equations (2)–(7) together, Bayes’ theorem is used to estimate from $\mathit{F}$ the values of all parameters:

$$p\left(\mathit{U},\mathit{V},\mathcal{Z},\mathit{\mu },\mathit{\tau },\mathit{\pi },\mathit{\gamma },d\left|\mathit{F}\right.\right)$$

(8)

where $\mathcal{Z}={\left\{{\mathit{z}}_{ij}\right\}}_{N\times P},\mathit{\mu }={\left\{{\mu }_{jk}\right\}}_{B\times K},\mathit{\tau }={\left\{{\tau }_{jk}\right\}}_{B\times K},\mathit{\pi }=\left({\mathit{\pi }}_1,\dots ,{\mathit{\pi }}_P\right)$ and $\mathit{\gamma }=\left({\gamma }_1,\dots ,{\gamma }_R\right)$.

2.4. Variational Inference

In this section, the posterior of parametric model Equation (8) is inferred by the VB approach [39]. VB obtains the objective parameters $\mathit{x}$ finding the minimum Kullback–Leibler (KL) divergence between the approximated distribution $q\left(\mathit{x}\right)$ and the actual distribution $p\left(\mathit{x}\left|\mathit{D}\right.\right)$ with the known observation $\mathit{D}$, which can be formulated as below:

$$\begin{array}{cc}\hfill q^{*}\left(\mathit{x}\right)& =\underset{q\in \mathsf{\Omega }}{\min }KL\left(q\left(\mathit{x}\right)∥p\left(\mathit{x}\left|\mathit{D}\right.\right)\right),\hfill \end{array}$$

(9)

where $\mathsf{\Omega }$ is the constrained probability densities for obtaining the feasible solution. We can factorize $q\left(\mathit{\theta }\right)$ as $q\left(\mathit{\theta }\right)={\prod }_i{q}_i\left({\mathit{\theta }}_i\right)$ by mean field theory, and the posterior distribution Equation (8) can be approximated with the following form:

$$\begin{array}{cc}\hfill & p\left(\mathit{U},\mathit{V},\mathcal{Z},\mathit{\mu },\mathit{\pi },\mathit{\tau },\mathit{\gamma },\mathit{d}\right)=\prod _{i}q\left({\mathit{u}}_{i·}\right)\prod _{j}q\left({\mathit{u}}_{j·}\right)\hfill \\ & \prod _{ij}q\left({\mathit{z}}_{ij}\right)\times \prod _{j}q\left({\mathit{\mu }}_j,{\mathit{\tau }}_j\right)q\left({\mathit{\pi }}_j\right)\prod _{l}q\left({\gamma }_l\right)q\left(d\right).\hfill \end{array}$$

(10)

2.4.1. Estimation of Noise Component

For the noise component in the j-$\mathrm{th}$ frame, we need to estimate four parameters, ${\mathit{\mu }}_j$, ${\mathit{\tau }}_j$, $\mathcal{Z}$ and ${\mathit{\pi }}_j$. Firstly we update ${\mathit{\mu }}_j$ and ${\mathit{\tau }}_j$ in the following way:

$$\begin{array}{c}\hfill q^{*}\left({\mathit{\mu }}_j,{\mathit{\tau }}_j\right)=\prod _{k}N\left({\mu }_{jk}\left|m_{jk},\frac{1}{{\beta }_{jk}{\tau }_{jk}}\right.\right)\mathrm{Gam}\left({\tau }_{jk}\left|c_{jk},d_{jk}\right.\right),\end{array}$$

(11)

where

$$\begin{array}{cc}\hfill & m_{jk}=\frac{1}{{\beta }_{jk}}\left\{m_0{\beta }_0+\sum _i〈z_{ijk}〉\left(f_{ij}-〈{\mathit{u}}_{i·}〉{〈{\mathit{v}}_{j·}〉}^T\right)\right\},\hfill \\ \hfill & {\beta }_{jk}={\beta }_0+\sum _i〈z_{ijk}〉,c_{jk}=c_0+\frac{1}{2}\sum _i〈z_{ijk}〉,\hfill \\ \hfill & d_{jk}=〈d〉+\frac{1}{2}\left\{\sum _i〈z_{ijk}〉〈{\left(f_{ij}-{\mathit{u}}_{i·}{{\mathit{v}}_{j·}}^T\right)}^2〉+{\beta }_0{m_0}^2\right.\hfill \\ \hfill & \left.-\frac{1}{{\beta }_{jk}}{\left(\sum _i〈z_{ijk}〉\left(f_{ij}-〈{\mathit{u}}_{i·}〉{〈{\mathit{v}}_{j·}〉}^T\right)+{\beta }_0{m}_0\right)}^2\right\},\hfill \end{array}$$

(12)

where $f_{ij}$ denotes the $ij$-$\mathrm{th}$ element of the $\mathit{F}$. The variables ${\mathit{z}}_{ij}$ can be derived in closed form as below:

$$\begin{array}{c}\hfill q\left({\mathit{z}}_{ij}\right)={\prod }_k{r}_{ijk}^{z_{ijk}}\end{array}$$

(13)

where

$$r_{ijk}=\frac{{\rho }_{ijk}}{{\sum }_k{\rho }_{ijk}},$$

(14)

$$\begin{array}{cc}\hfill \ln {\rho }_{ijk}=& 〈\ln {\pi }_{jk}〉-\frac{1}{2}\ln 2\pi +\frac{1}{2}〈\ln {\tau }_{jk}〉\hfill \\ \hfill & -\frac{1}{2}〈{\tau }_{jk}{\left(f_{ij}-{\mu }_{jk}-{\mathbf{\mu }}_{i·}{\mathit{v}}_{j·}^T\right)}^2〉.\hfill \end{array}$$

(15)

Finally, we update ${\mathit{\pi }}_j$ by:

$$\begin{array}{c}\hfill q\left({\mathit{\pi }}_j\right)={\prod }_k{\pi }_{jk}^{{\alpha }_{jk}-1},\end{array}$$

(16)

where ${\alpha }_{jk}={\alpha }_0+{\sum }_i〈z_{ijk}〉$, and the hyper-parameter d is updated by the following equation:

$$\begin{array}{c}\hfill q\left(d\right)=\mathrm{Gam}\left(d\left|\eta ,\lambda \right.\right),\end{array}$$

(17)

where $\eta ={\eta }_0+c_{0}KP$ and $\lambda ={\lambda }_0+{\sum }_{j,k}〈{\tau }_{jk}〉$.

2.4.2. Estimation of Background Component

For the background component, we need to estimate three parameters, including $\mathit{U},\mathit{V}$ and $\mathbf{\gamma }$. ${\mathit{u}}_{i·}\left(i=1,\dots ,N\right)$ can be estimated as follows:

$$\begin{array}{c}\hfill q\left({\mathit{u}}_{i·}\right)=N\left({\mathit{u}}_{i·}\left|{\mathit{\mu }}_{{\mathit{u}}_{i·}},{\Sigma }_{{\mathit{u}}_{i·}}\right.\right),\end{array}$$

(18)

where

$$\begin{array}{c}{\mathit{\mu }}_{{\mathit{u}}_{i·}}=\left\{\sum _{j,k}〈z_{ijk}〉〈{\tau }_{jk}〉\left(f_{ij}-〈{\mu }_{jk}〉\right)〈{\mathit{v}}_{j·}〉\right\}{\Sigma }_{{\mathit{u}}_{i·}},\hfill \\ {\Sigma }_{{\mathit{u}}_{i·}}={\left\{\sum _{j,k}〈z_{ijk}〉〈{\tau }_{jk}〉〈{\mathit{v}}_{j·}^T{\mathit{v}}_{j·}〉+〈\mathsf{\Gamma }〉\right\}}^{-1}.\hfill \end{array}$$

Similarly, ${\mathit{v}}_{j·}\left(j=1,\dots ,P\right)$ is estimated by:

$$\begin{array}{c}\hfill q\left({\mathit{v}}_{j·}\right)=N\left({\mathit{v}}_{j·}\left|{\mathit{\mu }}_{{\mathit{v}}_{j·}},{\Sigma }_{{\mathit{v}}_{j·}}\right.\right),\end{array}$$

(19)

where

$$\begin{array}{c}{\mathit{\mu }}_{{\mathit{v}}_{j·}}=\left\{\sum _{j,k}〈z_{ijk}〉〈{\tau }_{jk}〉\left(f_{ij}-〈{\mu }_{jk}〉\right)〈{\mathbf{u}}_{j·}〉\right\}{\Sigma }_{{\mathit{v}}_{j·}},\hfill \\ {\Sigma }_{{\mathit{v}}_{j·}}={\left\{\sum _{j,k}〈z_{ijk}〉〈{\tau }_{jk}〉〈{\mathit{u}}_{j·}^T{\mathit{u}}_{j·}〉+〈\mathsf{\Gamma }〉\right\}}^{-1}.\hfill \end{array}$$

$\mathsf{\Gamma }=\mathrm{diag}\left(〈\mathit{\gamma }〉\right)$, ${\gamma }_l$ is a decisive factor for guaranteeing low-rank property of $\mathit{B}$ by removing the corresponding rows when its value is very large [38], which can be estimated by:

$$\begin{array}{c}\hfill q\left({\gamma }_l\right)=\mathrm{Gam}\left({\gamma }_r\left|{\xi }_l,{\delta }_l\right.\right),\end{array}$$

(20)

where

$$\begin{array}{c}{\xi }_l={\xi }_0+\frac{1}{2}\left(m+n\right),\hfill \\ {\delta }_l={\delta }_0+\frac{1}{2}\sum _i〈u_{il}^2〉+\frac{1}{2}\sum _j〈v_{jl}^2〉.\hfill \end{array}$$

In the following experiments, we set $m_0=0$, and ${\alpha }_0,{\beta }_0,c_0,d_0,{\eta }_0,{\lambda }_0,{\xi }_0,{\delta }_0$ are initialized with ${10}^{-6}$ [36].

2.5. Target Extraction

In this section, we firstly select the noise component containing small-faint target. Then, the MFD method [37] is used to extract the target from the noise.

2.5.1. Selecting Noise Component Containing Target

We obtain the noise component $\mathit{E}$ separating it from the background component, and we can divide it into K components ${\mathit{E}}^1,\dots ,{\mathit{E}}^K$ according to the maximum probability criteria [32]:

$$\begin{array}{c}\hfill e_{ij}^m=\left\{\begin{array}{c}{e}_{ij,}\mathrm{if}m=\arg \underset{k=1,\dots ,K}{\max }\left(r_{ijk}\right)\hfill \\ 0,\mathrm{else}\hfill \end{array}.\right.\end{array}$$

(21)

The K components are restored to sequences ${\overline{\mathit{E}}}^1,\dots ,{\overline{\mathit{E}}}^K$ by the aforementioned method in Section 2.1. Note that the intensity of the true target is quite different from the noise. Instead of using variance guided method in [32], we calculate the difference between the minimum and maximum of intensity and select the largest one ${\overline{\mathit{E}}}^i$ as the component containing target, which can be described as follows:

$$\begin{array}{c}\hfill i=\arg \underset{k=1,\dots ,K}{\max }\left(\max \left({\overline{\mathit{E}}}^k\right)-\min \left({\overline{\mathit{E}}}^k\right)\right)\end{array}$$

(22)

The following experimental results demonstrate that this method is effective.

2.5.2. Extracting Target by MFD

Figure 2 gives the results of a representative infrared noisy image using NMoG method with $K=3$, and subfigure (c) is the slice containing the true target. It is observed from Figure 2c that the restored slice containing true target is still contaminated by pixel noise. Thus, we use the MFD method [37] to wipe out the noise and enhance the target.

The noise component $\mathit{E}$ containing the target is firstly transformed into a gradient vector field by:

$$\begin{array}{cc}\hfill I\left(x,y\right)& =\left[\begin{array}{c}{e^{\prime }}_x\left(x,y\right)\hfill \\ {e^{\prime }}_y\left(x,y\right)\hfill \end{array}\right]\hfill \\ \hfill {e^{\prime }}_x\left(x,y\right)& =\frac{e\left(x+1,y\right)-e\left(x-1,y\right)}{2}\hfill \\ \hfill {e^{\prime }}_y\left(x,y\right)& =\frac{e\left(x,y+1\right)-e\left(x,y-1\right)}{2}\hfill \end{array}$$

(23)

where $e\left(x,y\right)$ denotes the value of $\mathit{E}$ at location $(x,y)$, ${e^{\prime }}_x\left(x,y\right)$ and ${e^{\prime }}_y\left(x,y\right)$ are the gradient value in the x-direction and y-direction.

From Figure 3b,d, it can be observed that both the true target and bright noise residuals are a sink in gradient vector field. But the gradient vectors of noise pixel focus on 4 directions, and MFD method can compute the flux density of each pixel after removing its four largest gradient vectors, which is defined as follows [37]:

$$\begin{array}{c}\hfill \mathrm{M}{\mathrm{FD}}_s\left(x,y\right)=\sum _{\left(x^{\prime },y^{\prime }\right)\in {\mathbf{O}}^{\prime }\left(x,y,s\right)}\frac{\mathbf{I}\left(x^{\prime },y^{\prime }\right)·{\overrightarrow{n}}_o\left(x^{\prime }-x,y^{\prime }-y\right)}{8s-4}\end{array}$$

(24)

where $\mathrm{M}{\mathrm{FD}}_s$ is s-scale MFD, s denotes the scale variable, ${\mathbf{O}}^{\prime }$ denotes the subset of $\mathbf{O}$, which removes four pixels containing the four largest gradient vectors. Note that the number of pixels on the curve is $8s-4$. $\mathit{O}$ represents the neighborhood region as:

$$\begin{array}{c}\hfill \mathit{O}\left(x,y,s\right)=\left\{\left(x^{\prime },y^{\prime }\right)\left|max\left(\left|x^{\prime }-x\right|,\left|y^{\prime }-y\right|\right)=s\right.\right\}\end{array}$$

(25)

and the norm vector on the boundary point ${\overrightarrow{n}}_o\left(x,y\right)$ is defined as follows:

$$\begin{array}{cc}\hfill {\overrightarrow{n}}_o\left(x,y\right)& =\left[\begin{array}{c}{n}_{ox}\left(x,y\right)\hfill \\ n_{oy}\left(x,y\right)\hfill \end{array}\right]\hfill \\ \hfill n_{ox}\left(x,y\right)& =\left\{\begin{array}{c}-1,x=k\hfill \\ 1,x=-k\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.\hfill \\ \hfill n_{oy}\left(x,y\right)& =\left\{\begin{array}{c}-1,y=k\hfill \\ 1,y=-k\hfill \\ 0,\mathrm{else}\hfill \end{array}\right.\hfill \end{array}$$

(26)

where $n_{ox}\left(x,y\right)$ and $n_{oy}\left(x,y\right)$ are the value in the x-direction and y-direction.

Figure 3 shows that noisy pixels are wiped out according to their MFD value. This is because the MFD value of the noisy pixels is much smaller than that of the real target, which is usually a negative element. Following this property, the corresponding noise pixels are wiped out in the target image. Thus, we obtain an initial result by the following equation:

$$\mathit{T}\left(x,y\right)={\overline{\mathit{E}}}^i\left(x,y\right)*\mathrm{M}{\mathrm{FD}}_s{\left(x,y\right)}_{+}$$

(27)

where $\mathit{T}\left(x,y\right)$ denotes the initial target image, $\mathrm{M}{\mathrm{FD}}_s{\left(x,y\right)}_{+}$ is the result by setting the positive elements and negative elements in the original MFD maple to 1 and 0, respectively. Finally, we use an adaptive threshold to further separate the target [22], which is described as below:

$$T=\mu +k\sigma$$

(28)

where $\mu$ and $\sigma$ are the mean value and standard deviation of the small target image. k is a empirical value, and we set it as 0.05 in our experiment. The framework of our method is shown in Figure 1, and the detection procedure is given in Algorithm 1.

Algorithm 1

Input: Infrared image sequence $f_1,f_2,\cdots ,f_P\in {\mathbb{R}}^{m\times n}$.

Initialize: Set parameters $\left(m_0,{\beta }_0,c_0,d_0,{\eta }_0,{\lambda }_0\right)={10}^{-6}$ in noise prior. Low-rank background component ${\mathbf{U}}_0,{\mathbf{V}}_0$ and ${\alpha }_0$ parameters in the model prior $\left({\xi }_0,{\delta }_0\right)$, scale parameter $s=1$ in MFD method, iteration number $t=1$.

Step 1: Construct the spatio-temporal patch image $\mathbf{F}$ with the input infrared image sequence using the method in Section 2.1.

Step 2: Build NMoG noise model under the Bayesian framework by Equations (2) and (5).

Step 3: While not converged do:

1. Update approximate posterior of noise component ${\mathcal{Z}}^t,{\mathbf{\pi }}^t$ by Equations (13)–(16), ${\mathit{\mu }}^t,{\mathit{\tau }}^t$ by Equations (11) and (12) and $d^t$ by Equation (17), respectively.

2. Update approximate posterior of background component $\mathit{U},\mathbf{V}$ by Equations (18) and (19).

3. Update approximate posterior of parameters in noise component ${\mathit{\gamma }}^t$ by Equation (20).

4. Update $t=t+1$.

end While

Step 4: Noise component $\mathit{E}$ by $\mathit{E}=\mathit{F}-\mathit{U}{\mathit{V}}^t$. Decompose $\mathit{E}$ into K components by Equation (21), and reconstruct noise components into the corresponding image sequences by method in Section 2.1.

Step 5: Select the true target images by Equation (22).

Step 6: Calculate the original MFD map of the target images by Equations (23) and (24).

Step 7: Obtain the separated target images by using both MFD maple and adaptive threshold, which can be computed by Equation (27).

Output: Separated target image sequence.

3. Experiments

To validate the effectiveness of the proposed approach, extensive experiments are performed on simulated and real infrared image sequences in this section.

3.1. Metrics and Comparative Methods

In this paper, we use the receiver operating characteristic (ROC) to show the relationship between the detection probability $P_d$ and false alarm rate $F_a$, and they are described as below [22,25,26,27,28,32]:

$$P_d=\frac{\mathrm{number}\mathrm{of}\mathrm{true}\mathrm{detections}}{\mathrm{number}\mathrm{of}\mathrm{actual}\mathrm{targets}}$$

(29)

$$F_a=\frac{\mathrm{number}\mathrm{of}\mathrm{false}\mathrm{detections}}{\mathrm{number}\mathrm{of}\mathrm{images}}$$

(30)

In addition, the local signal-to-noise ratio gain (LSNRG), background suppression factor (BSF), signal to clutter ratio gain (SCRG) and contrast gain (CG) metrics are also used in our work, and the detailed definitions can be found in [28,32]. We also introduce a local background region for computing the LSNRG and SCRG [22], which is displayed in Figure 4. The width of neighboring region d is set as 20 here.

Nonetheless, the accuracy of the low-rank background estimation is also an important metric, since less estimation error means better preservation of strong edges in the background component. Thus, we use another metric, namely accuracy of background recovery (ABR), which is defined as:

$$\mathrm{ABR}=\frac{{∥{\mathit{B}}_{\mathrm{out}}∥}_F}{{∥{\mathit{B}}_{\mathrm{in}}∥}_F}$$

(31)

where ${\mathit{B}}_{\mathrm{in}}$ and ${\mathit{B}}_{\mathrm{out}}$ are the background before and after processing.

The five baseline methods for comparison including two classical filtering methods, i.e., top-hat [13] and max-median filtering [12], and three low-rank matrix analysis methods IPI [22] and RIPT [28] (using spatial information) and the MRF-MoG [32] (using spatio-temporal information and assuming i.i.d. MoG noise) method.

Table 1. Parameter setting of methods.

Methods	Acronyms	Parameter Settings
max-median filter	max-median	Support size: $5\times 5$
top-hat method	top-hat	Structure shape: Square, structure size: $3\times 3$
Infrared Patch-Image Mode	IPI	Patch size: $50\times 50$, sliding step: 10, $\lambda =\frac{L}{\sqrt{min\left(n_1,n_2,n_3\right)}}$, $L=1$, $\epsilon =1e-7$
Reweighted Infrared Patch-Tensor Model	RIPT	Patch size: $50\times 50$, sliding step: 10, $\lambda =\frac{L}{\sqrt{min\left(n_1,n_2,n_3\right)}}$, $L=1$, $h=10$, $\epsilon =1e-7$
Mixture of Gaussians with Markov random field	MoG with MRF	Noise component number: $K=3$
Mixture of Non-i.i.d. Gaussians with Modified Flux Density	NMoG with MFD	Noise component number: $K=3$

gives the detailed parameter settings, where $n_1,n_2,n_3$ denotes the dimensions of the infrared patch tensor [28].

3.2. Simulated and Real Datasets

The noise of real infrared images usually includes five typical types: Gaussian noise, Poisson noise, impulse noise, dead pixels or lines, and salt and pepper noise. To validate the effectiveness of the proposed approach in complex noisy situations, five consecutive real infrared image sequences are used as original images to add the mixture of the above five types of noises, and these original images are approximately noise-free. Additive white Gaussian noise with two SNR value are added to each frame of five sequences, and the SNR are in the range of [10, 15] dB and [15, 20] dB, respectively. The location of pixels corrupted by different noises are chosen randomly. We choose forty frames of Sequences 1–4 to add with various types of noise and different intensity. Finally, we add the mixture of noise to each frame in sequence 5. The details are described in

Table 2. Characteristics of noisy infrared sequences.

Sequence	Number of Frames	Image Resolution (pixels)	Noise Characteristics	Background Characteristics	SCR	$\stackrel{\_\_\_\_\_\_}{\mathbf{SCR}}$
1	135	$220\times 140$	Gaussian + Deadline Noise	Sea-Sky Clutters	0.25∼10.11	3.49
2	108	$280\times 228$	Gaussian + Salt and Pepper Noise	Heavy cloud-sky clutters	0.13∼8.24	2.71
3	114	$250\times 200$	Gaussian + Poisson Noise	Heavy cloud-sky clutters	0.11∼3.30	1.33
4	123	$281\times 240$	Gaussian + Impulse Noise	Heavy cloud-sky clutters	0.02∼4.32	1.90
5	102	$200\times 150$	Mixture Noise	Heavy cloud-sky clutters	0.05∼10.24	3.09

, and their representative frames are displayed in the first column of Figure 9.

SCR is defined as follows [40]:

$$\mathrm{SCR}=\frac{\left|{\mu }_t-{\mu }_b\right|}{{\sigma }_b}$$

(32)

where ${\mu }_t$ is the average pixel value of the target region, ${\mu }_b$ and ${\sigma }_b$ denote the average pixel value and the standard deviation of the neighborhood region. Based on definition of SCR, the average SCR value of targets is used to characterize the noisy sequence, which is defined as follows [22]:

$$\stackrel{\_\_\_\_\_}{\mathrm{SCR}}=\frac{1}{N}\sum _{i=1}^N\mathrm{SC}{\mathrm{R}}_i$$

(33)

where N denotes the number of targets and $\mathrm{SC}{\mathrm{R}}_i$ denotes $i\mathrm{th}$ target.

Then we also carry out comparison experiments with three real infrared image sequences contaminated by heavy noise.

3.3. Effect of Component Number

Here, we vary K from 2 to 7 for analyzing the influence of noise component parameter K on the performance of the proposed model. For quantitative analysis, the experiments have fixed false-alarm rates ($F_a$) by changing the segmentation thresholds on Sequences 1–5, which are given in

Table 3. The detection performance of the proposed method with different K values.

Metric	K	Sequence 1	Sequence 2	Sequence 3	Sequence 4	Sequence 5
$F_a$		$F_a$ = 0.01/image	$F_a$ = 0.1/image	$F_a$ = 0.5/image	$F_a$ = 2/image	$F_a$ = 0.25/image
$P_d$	2	0.98	0.90	0.90	0.90	0.85
	3	1.00	0.90	0.94	0.93	0.87
	4	0.96	0.90	0.87	0.85	0.86
	5	0.96	0.90	0.86	0.84	0.80
	6	0.96	0.89	0.84	0.84	0.81
	7	0.96	0.89	0.85	0.82	0.80

. The bold format number indicates the highest score. Besides, we also display the ROC curves in Figure 5. We can observe from the result that there is no significant difference in performance when K is larger than 2. From Figure 5a,d, it can be seen that $F_a$ of $K=2$ are higher than that of other K values, this is because the target component might contain the sparse noise, which could not be wiped out by the threshold. However, it is also improper to set K too large. From Figure 5a,c–e, the probability of detection is decreasing as K becoming larger when $K\ge 4$ due to the true targets might lose in the separated target component. In addition, considering the computation complexity is increasing with larger K, K is set as 3 in experiments.

3.4. Effect of MFD

To demonstrate the superiority of the MFD method over other methods, we perform comparative experiments on a representative image of simulated Sequence 5, including the MRF [32] and the ablated version (NMoG without MFD). From Figure 6, we can observe that the MFD method can effectively wipe out the bright noise, while the other two methods lose the true target and have many residual noise pixels, and these residuals could cause a high false alarm ratio.

3.5. Performance of Multiple Targets Scene

Considering the number of targets may change in different scenes, such as antimissile systems, we test the effectiveness of the proposed method in multi-target scenarios (the number of the targets is 3). The method of embedding a synthetic target into the images can be found in [22]. The representative images and the corresponding results are displayed in the first row and second row of Figure 7. All the targets are detected successfully by the proposed method.

3.6. Comparisons to Baseline Methods

3.6.1. Experiments on Simulated Data

In this experiment, we focus on analyzing and comparing the performance of different approaches on real infrared images with synthetic noise. To illustrate the difference between the original images and noisy images, we display the gray histograms of five representative frames in Figure 8. The representative images are chosen from one image of the corresponding 40 noisy images of Sequences 1–4 and from one image of Sequence 5 randomly. It can be observed from Figure 8 that the distributions of original and noisy images are quite different. Figure 9 shows the corresponding target images of different approaches. We can observe that both max-median filter and top-hat filter can not suppress the noise pixels clearly, and these residuals would increase $F_a$. Besides, top-hat filter loses the target in Sequences 2 and 5. The performances of both max-median filter and top-hat filter are limited by the filtering size required to be fixed as an input parameter without any knowledge of the target size. Their performances degrade heavily when the filter size deviates from the target size.

From the comparison between the results of filtering based approaches and low-rank based approaches, we conclude that the latter can achieve better performance than the former ones. All the targets can be detected by IPI method, but many noise pixels are also retained due to the deficiency effects [28], especially for Sequences 2, 4 and 5. This phenomenon demonstrate that the IPI method is quite sensitive to salt and pepper noise and impulse noise. The RIPT approach has better background suppression ability than IPI approach, but we can find that it is also sensitive to salt and pepper noise from the corresponding results of Sequences 2 and 5. Moreover, the RIPT method fails in Sequence 3. MoG-MRF only detects the true targets of Sequence 1 and 4, the unsatisfying performance of MoG-MRF is because the i.i.d. MoG assumption is not suitable to the case when the noise distribution between different frames is nonidentical. Besides, the segmentation performance of MRF would degrade when the noise pixel is adjacent to true targets in complex noisy cases. From the last column of Figure 9, it can be observed that all targets are detected correctly by the proposed model while noise pixels and clutters being suppressed completely.

In addition, we also use five metrics to analyze the performance of different approaches quantitatively. The LSNRG, BSF and SCRG values of different approaches for the representative images are given in

Table 4. Quantitative evaluation of different methods for the representative images of Sequences 1–3.

	31st Frame of Sequence 1			3rd Frame of Sequence 2			41st Frame of Sequence 3
Method	LSNRG	BSF	SCRG	LSNRG	BSF	SCRG	LSNRG	BSF	SCRG
Max-Meidan	1.6209	14.8329	2.7474	0.4829	7.0193	3.6076	0.4640	14.4141	0.0792
top-hat	Inf	Inf	Inf	Miss	Miss	Miss	1.0311	2.3991	0.2823
IPI	4.1182	1.5731	6.207	0.976	1.5658	0.6981	0.9828	1.3869	0.2679
RIPT	Inf	Inf	Inf	0.6207	6.4585	13.3034	0	6.0941	32.9413
MoG-MRF	Inf	Inf	Inf	Miss	Miss	Miss	0	6.8651	53.2848
NMoG-MFD	Inf	Inf	Inf	Inf	Inf	Inf	Inf	Inf	Inf

and

Table 5. Quantitative evaluation of different methods for the representative images of Sequences 4 and 5.

	4th Frame of Sequence 4			22nd Frame of Sequence 5
Method	LSNRG	BSF	SCRG	LSNRG	BSF	SCRG
Max-Meidan	1.2486	9.5636	3.2359	0.2618	2.4965	0.366
top-hat	Inf	Inf	Inf	Miss	Miss	Miss
IPI	2.1088	3.8868	4.8508	0.9329	1.3979	0.4839
RIPT	Inf	Inf	Inf	0.6674	2.901	3.6575
MoG-MRF	Inf	Inf	Inf	Miss	Miss	Miss
NMoG-MFD	Inf	Inf	Inf	Inf	Inf	Inf

. The Inf means that the standard deviation of neighboring background is zero, and the high scores in the above three metrics only reflect the good suppression performance in a local region. Note that the values of low-rank based methods in the above three metrics are usually Inf, as the results of RIPT method, MoG-MRF method and the proposed method on Sequences 1 and 4. Considering the above phenomenon, the average CG and ABR values of all images are also computed for further comparison [32], as listed in

Table 6. The evaluation results of average contrast gain (CG) and accuracy of background recovery (ABR) values of different methods for all image sequences.

	Sequence 1		Sequence 2		Sequence 3		Sequence 4		Sequence 5
Method	CG	ABR	CG	ABR	CG	ABR	CG	ABR	CG	ABR
Max-Meidan	2.3312	0.9221	1.2661	0.9457	2.4625	0.9519	1.7005	0.868	1.4464	0.8981
top-hat	3.8321	0.9066	5.0661	0.9286	5.6079	0.9398	4.1474	0.9289	3.1628	0.9192
IPI	2.6185	0.8601	1.5188	0.8374	1.7097	0.9067	2.5819	0.9447	1.8794	0.8861
RIPT	3.1073	0.9179	3.7042	0.9303	6.1015	0.9423	3.0008	0.9321	2.1123	0.8993
MoG-MRF	4.7533	0.9327	5.2441	0.9700	6.3158	0.9508	4.0663	0.9584	3.8108	0.9435
NMoG-MFD	4.7798	0.9801	5.6895	0.9841	8.1627	0.9837	5.1757	0.9849	3.8432	0.9825

. For quantitative analysis, the experiments have fixed false-alarm rates ($F_a$) by changing the segmentation thresholds on Sequences 1–5, which are given in

Table 7. The detection performance of different methods on Sequences 1–5.

Metric	Methods	Sequence 1	Sequence 2	Sequence 3	Sequence 4	Sequence 5
$F_a$		$F_a$ = 1/image	$F_a$ = 15/image	$F_a$ = 2.5/image	$F_a$ = 2/image	$F_a$ = 11/image
	max-median	0.84	0	0.49	0.30	0
	Top-hat	0.46	0.28	0.41	0.22	0.26
$P_d$	IPI	0.91	0.05	0.93	0.90	0.27
	RIPT	0.91	0.17	0.93	0.91	0.54
	MoG-MRF	0.90	0.52	0.88	0.74	0.75
	NMoG-MFD	1.00	0.90	0.94	0.93	0.87

. In conclusion, the proposed approach achieves the best performance. In conclusion, the proposed approach achieves the best performance.

Moreover, we show the ROC curves of different approaches in Figure 10. From the result, we can see that the $F_a$ of max-median on Sequences 2 and 5 are very high. The performance of the proposed approach is superior to other approaches on Sequences 1–3 and 5, which achieves the highest $P_d$ with very low $F_a$, this is because the noise pixels and background residuals are suppressed thoroughly by the proposed method. As for Sequence 4, IPI achieves higher $P_d$ than that of the proposed method when $F_a\le 1.1$. However, the proposed method can achieve higher probability of detection when $F_a>1.1$. The ROC curves of IPI and RIPT on Sequences 2 and 5 demonstrate that they are sensitive to salt and pepper noise, and the performance of MoG with MRF method is not satisfying due to the identical noise distribution assumption fails in complex noise case.

3.6.2. Experiments on Real Data

We also carry out additional experiments on three real and noisy infrared image sequences, namely, Sequences 6–8. Briefly, we use the most important metric, i.e., the ROC curves of 6 tested method on real image sequences, to compare their performance, which are shown in Figure 11. In addition,

Table 8. The detection performance of different methods on Sequences 6–8.

Metric	Methods	Sequence 6	Sequence 7	Sequence 8
$F_a$		$F_a$ = 2/image	$F_a$ = 2/image	$F_a$ = 2/image
	max-median	0	0	0
	Top-hat	0.11	0.14	0.51
$P_d$	IPI	0.86	0.33	0.04
	RIPT	0.34	0.28	0.32
	MoG-MRF	0.63	0.14	0.85
	NMoG-MFD	0.89	0.90	1.00

shows the quantitative analysis, and the proposed approach achieves the highest $P_d$ with the same $F_a$. The results demonstrate the superiority of the proposed approach on target detection, background clutter and noise suppression ability over other competitive methods, because the NMoG model and MFD maple improve the robustness of the proposed approach to different kinds of noise.

3.7. Complexity Analysis

Here, we analyze and compare the complexity of different approaches, which are listed in

Table 9. Algorithm complexity and computational time comparisons of different methods.

Method	Complexity	Times(s)
max-median	$\mathcal{O}\left(mn{L}^2\right)$	392.997661
top-hat	$\mathcal{O}\left(mn{L}^{2}logL\right)$	2.639046
IPI	$\mathcal{O}\left(m{n}^2\right)$	682.764355
RIPT	$\mathcal{O}\left(t{n}_1{n}_2{n}_3\left(n_1{n}_2+n_2{n}_3+n_1{n}_3\right)\right)$	224.866089
MoG-MRF	$\mathcal{O}\left(t\left(\left(N+P\right)R^3+kNPR\right)+mn(w+1)\right)$	3002.7214
NMoG-MFD	$\mathcal{O}\left(t\left(\left(N+P\right)R^3+kNPR\right)+mn{\left(2s+1\right)}^2+mn\right)$	482.9220

. $(m,n)$ and L denote the image size and the structure element, respectively. $(n_1,n_2,n_3)$ represent the dimensions of the tensor in RIPT model, and the details can be found in [30]. As for the proposed method, let $\mathit{F}\in {\mathbb{R}}^{N\times P}$, we firstly need to infer the parameters in NMoG model, and its complexity is $\mathcal{O}\left(\left(N+P\right)R^3+KNPR\right)$ in each iteration. For computing MFD maple of an image with size of $m\times n$, the whole computational cost is $\mathcal{O}\left(mn{\left(2s+1\right)}^2\right)$. For target segmentation, the cost of this step is $\mathcal{O}\left(mn\right)$. Thus, the entire computation cost of the proposed method is $\mathcal{O}\left(t\left(\left(N+P\right)R^3+kNPR\right)+mn{\left(2s+1\right)}^2+mn\right)$, where t is the iteration number. The MoG with MRF method uses median operation to reconstruct image sequences, and its complexity is $\mathcal{O}\left(mnw\right)$, where w denotes the number of overlapped pixels during the transformation from the spatio-temporal patch image to the reconstruction image [32]. In addition, we compare the computational time of different approaches on whole Sequence 6. It can be observed from the result that MOG with MRF method is the slowest while the top-hat filter is the fastest. The processing time of the RIPT approach is shorter than the IPI approach and max-median filter. The proposed approach is slower than RIPT method and two filtering methods, but the superiority of its performance over other baseline methods can make up for this deficiency.

4. Conclusions

In this paper, we propose a novel infrared small and faint target detection approach based on NMoG and MFD models for complex and noisy scenarios. The proposed model can finely accord with the noise characteristics embeded in real infrared image sequences by using the NMoG model. We model the recovery of a low-rank background component and noise component as an LRMF model, which can be solved by the VB algorithm. Finally, the target can be extracted correctly from the noise by using MFD maple. Experimental results show that the proposed approach performs better than other competitive approaches, since it is more robust to complex noisy scenarios in real application.