Distributed Target Detection in Unknown Interference

Abstract

Interference can degrade the detection performance of a radar system. To overcome the difficulty of target detection in unknown interference, in this paper we model the interference belonging to a subspace orthogonal to the signal subspace. We design three effective detectors for distributed target detection in unknown interference by adopting the criteria of the generalized likelihood ratio test (GLRT), the Rao test, and the Wald test. At the stage of performance evaluation, we illustrate the detection performance of the proposed detectors in the presence of completely unknown interference (not constrained to lie in the above subspace). Numerical examples indicate that the proposed GLRT and Wald test can provide better detection performance than the existing detectors.

1. Introduction

Multichannel signal detection is a fundamental problem in the signal processing community [1,2,3,4,5,6,7,8]. Kelly first derived the generalized likelihood ratio test (GLRT) for multichannel radar systems in unknown noise in 1986 [9]. Then, many other detectors were proposed, such as the adaptive matched filter (AMF) [10], the adaptive coherence estimator (ACE) [11], and so on. Besides the possible target, there is often interference [12]. In [13], the problem of distributed target detection in the presence of known interference was considered, and several detectors were proposed according to the GLRT criterion. The interference in [13] is known in the sense that the interference lies in a known subspace but its coordinate is unknown. The related Rao tests were derived in [14], while the point-target case of the Wald test was given in [15]. Recently, it was pointed out in [16] that the detectors proposed in [14,15] are essentially derived according the variations of the Rao test and Wald test, respectively. According to the criteria of the GLRT, the Rao test, and the Wald test, three detectors were proposed in [17] for the problem of point target detection in a kind of partially known interference, called orthogonal interference therein. It was shown in [18] that the GLRT in [17] can also be derived according to the criterion of gradient test.

In fact, the operating environment of the radar system usually changes rapidly. Hence, there may be no reliable prior information about the interference. In [19], two detectors were proposed according to the GLRT and the Wald test for the problem of point target detection in the presence of completely unknown interference, which can be caused by static objects or a hostile jammer. The statistical performance of the GLRT in [19] was investigated in [20]. It was shown in [16] that both the Rao and Wald tests coincided with the GLRT. Moreover, It was pointed out in [16] that the Wald test proposed in [19] was essentially a variation of the Wald test.

Note that with the improvement of radar resolution, a target usually occupies successive multiple range bins. In other words, a target is usually a distributed target [21,22]. However, to the best of our knowledge, no effective detector has been proposed for the problem of detecting a distributed target in completely unknown interference. To bridge this gap, in this study, we consider the problem of distributed target detection in the presence of completely unknown interference. We adopt the criteria of the GLRT, the Rao test, and the Wald test to design detectors, because there is no optimum detector. This is usually called the uniformly most powerful (UMP) test (There is no UMP test, due to the fact that the interference and noise covariance matrix are both unknown.). We evaluate the detection performance of both of the proposed detectors through simulation data. The results show that the proposed detectors according to the GLRT and the Wald test can provide higher probabilities of detection (PDs) than the existing detectors.

2. Problem Formulation

Assume that a radar system has N antennas. A distributed target, if present, occupies K successive range bins. Then, the test data reflected by the distributed target can be denoted by an $N\times K$ matrix $\mathbf{X}$. Under the null hypothesis ${\mathrm{H}}_0$, $\mathbf{X}$ contains noise $\mathbf{W}$ and completely unknown interference $\mathbf{Q}$. In contrast, under the alternative hypothesis ${\mathrm{H}}_1$, $\mathbf{X}$ contains noise $\mathbf{W}$, completely unknown interference $\mathbf{Q}$, and signal $\mathbf{H}$, which is assumed to lie in a known subspace spanned by an $N\times p$ full-column-rank matrix $\mathbf{A}$. Hence, we have $\mathbf{H}=\mathbf{A}\mathbf{B}$, where the $p\times K$ matrix denotes the coordinate of the signal. We adopt the method in [19] to model the completely unknown interference at the stage of detector design. Precisely, it is temporally assumed that $\mathbf{Q}$ lies in a subspace orthogonal to the signal subspace. Hence, we have $\mathbf{Q}=\mathbf{J}\mathbf{D}$, where $\mathbf{J}$ is an $N\times (N-p)$ full-column-rank matrix satisfying (We use (1) to constrain the unknown interference mainly for mathematical tractability. The constraint in (1) is abandoned at the stage of detection performance evaluation.)

$${\mathbf{A}}^H\mathbf{J}={\mathbf{0}}_{p\times (N-p)},$$

(1)

with ${(·)}^H$ denoting the conjugate transpose; $\mathbf{D}$ is an unknown $(N-p)\times K$ matrix. In practice, the noise covariance matrix of $\mathbf{W}$, denoted as $\mathbf{R}$, is unknown. A common method to overcome this problem is using training data, usually collected in the vicinity of the test data. Assume that there are M training data, ${\mathbf{x}}_m$, $m=1,2,\dots ,M$, sharing the same noise covariance matrix $\mathbf{R}$. To summarize, we have the following binary hypothesis test:

$$\left\{\begin{array}{c}{\mathrm{H}}_0:\mathbf{X}=\mathbf{J}\mathbf{D}+\mathbf{W},{\mathbf{X}}_M={\mathbf{W}}_M,\hfill \\ {\mathrm{H}}_1:\mathbf{X}=\mathbf{A}\mathbf{B}+\mathbf{J}\mathbf{D}+\mathbf{W},{\mathbf{X}}_M={\mathbf{W}}_M,\hfill \end{array}\right.$$

(2)

where ${\mathbf{X}}_M=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_M]$, and ${\mathbf{W}}_M$ is the noise matrix in ${\mathbf{X}}_M$ (It seems that the detection problem in (2) is similar to that in [13,14]. However, there is an essential difference. The interference matrix $\mathbf{J}$ in (2) can be obtained by the singular valued decomposition of $\mathbf{A}$. In other words, no prior information about the interference is needed for (2). In contrast, the interference subspace needs to be set in advance in [13,14]).

3. Detector Derivations

Since there is no optimum detector for the detection problem in (2), we adopt the criteria of the GLRT, the Rao test, and the Wald test to design detectors and then compare their detection performance in the next section.

3.1. GLRT

The GLRT can be expressed as

$$t_{\mathrm{GLRT}}=\frac{\underset{\mathbf{B},\mathbf{D},\mathbf{R}}{max}{f}_1(\mathbf{X},{\mathbf{X}}_M)}{\underset{\mathbf{D},\mathbf{R}}{max}{f}_0(\mathbf{X},{\mathbf{X}}_M)},$$

(3)

where $f_1(\mathbf{X},{\mathbf{X}}_M)$ and $f_0(\mathbf{X},{\mathbf{X}}_M)$ are the joint probabilty density functions (PDFs) of $\mathbf{X}$ and ${\mathbf{X}}_M$ under Hypotheses ${\mathrm{H}}_1$ and ${\mathrm{H}}_0$, respectively. To obtain the GLRT for (2), we can adopt the mathematical derivations similar to that in [13]. For simplicity, we omit the derivation procedure and just list the resulting GLRT

$$t_{\mathrm{GLRT}}=\det ({\mathbf{I}}_K+{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{X}}),$$

(4)

where $\tilde{\mathbf{X}}={\mathbf{S}}^{-\frac{1}{2}}\mathbf{X}$, $\tilde{\mathbf{J}}={\mathbf{S}}^{-\frac{1}{2}}\mathbf{J}$, $\mathbf{S}={\mathbf{X}}_M{\mathbf{X}}_M^H$, ${\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }={\mathbf{I}}_K-{\mathbf{P}}_{\tilde{\mathbf{J}}}$, ${\mathbf{P}}_{\tilde{\mathbf{J}}}=\tilde{\mathbf{J}}{\left({\tilde{\mathbf{J}}}^H\tilde{\mathbf{J}}\right)}^{-1}{\tilde{\mathbf{J}}}^H$, $\det (·)$ denotes the determinant of a square matrix.

3.2. Rao Test

To give the Rao test, we first need the concept of Fisher information matrix (FIM), defined as [23]

$$\mathbf{I}(\Theta )=\mathrm{E}\left[\frac{\partial lnf(\mathbf{X};\Theta )}{\partial {\Theta }^{\ast }}\frac{\partial lnf(\mathbf{X};\Theta )}{\partial {\Theta }^T}\right],$$

(5)

where $\Theta$ stands for the unknown parameter set, $\mathrm{E}(·)$, $\partial (·)$, ${(·)}^{\ast }$, and ${(·)}^T$ denote statistical expectation, partial derivative, conjugate, and transpose, respectively. The unknown parameter set is usually partitioned as $\Theta ={\left[{\Theta }_{\mathrm{r}}^T,{\Theta }_{\mathrm{s}}^T\right]}^T$, where ${\Theta }_{\mathrm{r}}$ and ${\Theta }_{\mathrm{s}}$ are the related parameters and the nuisance parameters, respectively. Then, the FIM is partitioned as

$$\mathbf{I}(\Theta )=\left[\begin{array}{cc}{\mathbf{I}}_{\mathrm{r},\mathrm{r}}(\Theta )\hfill & {\mathbf{I}}_{\mathrm{r},\mathrm{s}}(\Theta )\hfill \\ {\mathbf{I}}_{\mathrm{s},\mathrm{r}}(\Theta )\hfill & {\mathbf{I}}_{\mathrm{s},\mathrm{s}}(\Theta )\hfill \end{array}\right].$$

(6)

The Rao test for complex-valued parameters is [23]

$$t_{\mathrm{Rao}}={\left.{\displaystyle \frac{\partial ln{f}_1(\mathbf{X};\Theta )}{\partial {\Theta }_{\mathrm{r}}}}\right|}_{\Theta ={\hat{\Theta }}_0}^T{\left[{\mathbf{I}}^{-1}\left({\hat{\Theta }}_0\right)\right]}_{\mathrm{r},\mathrm{r}}{\left.{\displaystyle \frac{\partial ln{f}_1(\mathbf{X};\Theta )}{\partial {\Theta }_{\mathrm{r}}^{\ast }}}\right|}_{\Theta ={\hat{\Theta }}_0},$$

(7)

where ${\Theta }_{\mathrm{r}}={\mathrm{vec}}^T\left(\mathbf{B}\right)$, ${\Theta }_{\mathrm{s}}=[{\Theta }_{{\mathrm{s}}_1}^T,{\Theta }_{{\mathrm{s}}_2}^T]={[{\mathrm{vec}}^T\left(\mathbf{D}\right),{\mathrm{vec}}^T\left(\mathbf{R}\right)]}^T$, the notation $\mathrm{vec}(·)$ denotes the vectorization, ${\hat{\Theta }}_0$ is the maximum likelihood estimate (MLE) of $\Theta$ under hypothesis ${\mathrm{H}}_0$, and

$${\left[{\mathbf{I}}^{-1}(\Theta )\right]}_{\mathrm{r},\mathrm{r}}={\left[{\mathbf{I}}_{\mathrm{r},\mathrm{r}}(\Theta )-{\mathbf{I}}_{\mathrm{r},\mathrm{s}}(\Theta ){\mathbf{I}}_{\mathrm{s},\mathrm{s}}^{-1}(\Theta ){\mathbf{I}}_{\mathrm{s},\mathrm{r}}(\Theta )\right]}^{-1}.$$

(8)

In the following we successively derive the sub-matrices for the FIMs in (8). The joint probability density function (PDF) of $\mathbf{X}$ and ${\mathbf{X}}_M$ under hypothesis ${\mathrm{H}}_1$ is

$$f_1(\mathbf{X},{\mathbf{X}}_M)=\frac{\exp \left[-\mathrm{tr}\left({\mathbf{R}}^{-1}\mathbf{S}\right)-\mathrm{tr}\left({\mathbf{R}}^{-1}{\mathbf{X}}_1{\mathbf{X}}_1^H\right)\right]}{{\pi }^{N(K+M)}\det {\left(\mathbf{R}\right)}^{K+M}},$$

(9)

where ${\mathbf{X}}_1=\mathbf{X}-\mathbf{A}\mathbf{B}-\mathbf{J}\mathbf{D}$. Taking the partial derivative of the logarithm of (9) with respect to $\mathrm{vec}\left(\mathbf{B}\right)$ and $\mathrm{vec}\left({\mathbf{B}}^{\ast }\right)$, we have

$$\frac{\partial \ln f_1(\mathbf{X},{\mathbf{X}}_M)}{\partial \mathrm{vec}\left(\mathbf{B}\right)}=\mathrm{vec}\left({\mathbf{A}}^T{\mathbf{R}}^{-T}{\mathbf{X}}_1^{\ast }\right)$$

(10)

$$\frac{\partial \ln f_1(\mathbf{X},{\mathbf{X}}_M)}{\partial \mathrm{vec}\left({\mathbf{B}}^{\ast }\right)}=\mathrm{vec}\left({\mathbf{A}}^H{\mathbf{R}}^{-1}{\mathbf{X}}_1\right).$$

(11)

Substituting (10) and (11) into (5), we have

$$\begin{array}{cc}\hfill {\mathbf{I}}_{\mathrm{r},\mathrm{r}}(\Theta )& =\mathrm{E}\left\{\mathrm{vec}\left({\mathbf{A}}^H{\mathbf{R}}^{-1}{\mathbf{X}}_1\right){\left[\mathrm{vec}\left({\mathbf{A}}^T{\mathbf{R}}^{-T}{\mathbf{X}}_1^{\ast }\right)\right]}^T\right\}\hfill \\ \hfill & =\mathrm{E}\left\{\mathrm{vec}\left({\mathbf{A}}^H{\mathbf{R}}^{-1}{\mathbf{X}}_1\right){[{\mathbf{I}}_K\otimes \left({\mathbf{A}}^T{\mathbf{R}}^{-T}\right)\mathrm{vec}\left({\mathbf{X}}_1^{\ast }\right)]}^T\right\}\hfill \\ \hfill & =\left({\mathbf{I}}_K\otimes {\mathbf{A}}^H{\mathbf{R}}^{-1}\right)\mathrm{E}\left[\mathrm{vec}\left({\mathbf{X}}_1\right)\mathrm{vec}{}^H\left({\mathbf{X}}_1\right)\right]\left({\mathbf{I}}_K\otimes {\mathbf{R}}^{-1}\mathbf{A}\right)\hfill \\ \hfill & ={\mathbf{I}}_K\otimes \left({\mathbf{A}}^H{\mathbf{R}}^{-1}\mathbf{A}\right),\hfill \end{array}$$

(12)

where in the second equality we have used $\mathrm{vec}\left({\mathbf{F}}_1{\mathbf{F}}_2{\mathbf{F}}_3\right)=({\mathbf{F}}_3^T\otimes {\mathbf{F}}_1)\mathrm{vec}\left({\mathbf{F}}_2\right)$ for comparable matrices ${\mathbf{F}}_1$, ${\mathbf{F}}_2$, and ${\mathbf{F}}_3$.

Taking the partial derivative of the logarithm of (9) with respect to $\mathrm{vec}\left(\mathbf{D}\right)$ and $\mathrm{vec}\left({\mathbf{D}}^{\ast }\right)$, we have the following two equalities

$$\frac{\partial \ln f_1(\mathbf{X},{\mathbf{X}}_M)}{\partial \mathrm{vec}\left(\mathbf{D}\right)}=\mathrm{vec}\left({\mathbf{J}}^T{\mathbf{R}}^{-T}{\mathbf{X}}_1^{\ast }\right)$$

(13)

$$\frac{\partial \ln f_1(\mathbf{X},{\mathbf{X}}_M)}{\partial \mathrm{vec}\left({\mathbf{D}}^{\ast }\right)}=\mathrm{vec}\left({\mathbf{J}}^H{\mathbf{R}}^{-1}{\mathbf{X}}_1\right).$$

(14)

Using (11) and (13), we have

$$\begin{array}{cc}\hfill {\mathbf{I}}_{\mathrm{r},{\mathrm{s}}_1}(\Theta )& =\left({\mathbf{I}}_K\otimes {\mathbf{A}}^H{\mathbf{R}}^{-1}\right)\mathrm{E}\left[\mathrm{vec}\left({\mathbf{X}}_1\right)\mathrm{vec}{}^H\left({\mathbf{X}}_1\right)\right]\left({\mathbf{I}}_K\otimes {\mathbf{R}}^{-1}\mathbf{J}\right)\hfill \\ \hfill & =\left(\mathbf{I}\otimes {\mathbf{A}}^H{\mathbf{R}}^{-1}\right)\left({\mathbf{I}}_K\otimes \mathbf{R}\right)\left({\mathbf{I}}_K\otimes {\mathbf{R}}^{-1}\mathbf{A}\right)\hfill \\ \hfill & ={\mathbf{I}}_K\otimes \left({\mathbf{A}}^H{\mathbf{R}}^{-1}\mathbf{J}\right).\hfill \end{array}$$

(15)

We can similarly obtain ${\mathbf{I}}_{{\mathrm{r}}_1,{\mathrm{s}}_2}(\Theta )=\mathbf{0}$ and ${\mathbf{I}}_{{\mathrm{s}}_1,{\mathrm{s}}_1}(\Theta )={\mathbf{I}}_K\otimes \left({\mathbf{J}}^H{\mathbf{R}}^{-1}\mathbf{J}\right)$. According to [16], we have

$$\begin{array}{c}\hfill {\mathbf{I}}_{{\mathrm{s}}_2,{\mathrm{s}}_2}(\Theta )=(M+K){\mathbf{R}}^{-T}\otimes {\mathbf{R}}^{-1}.\end{array}$$

(16)

It follows

$$\begin{array}{c}\hfill {\mathbf{I}}_{\mathrm{r},\mathrm{s}}(\Theta )=[{\mathbf{I}}_K\otimes \left({\mathbf{A}}^H{\mathbf{R}}^{-1}\mathbf{J}\right)\mathbf{0}]\end{array}$$

(17)

$$\begin{array}{c}\hfill {\mathbf{I}}_{\mathrm{s},\mathrm{s}}(\Theta )=\left[\begin{array}{cc}{\mathbf{I}}_K\otimes \left({\mathbf{J}}^H{\mathbf{R}}^{-1}\mathbf{J}\right)& \mathbf{0}\\ \mathbf{0}& (K+M){\mathbf{R}}^{-T}\otimes {\mathbf{R}}^{-1}\end{array}\right].\end{array}$$

(18)

Substituting (12), (17), and (18) into (8) results in

$${\left[{\mathbf{I}}^{-1}(\Theta )\right]}_{\mathrm{r},\mathrm{r}}={\mathbf{I}}_K\otimes {\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1},$$

(19)

where $\overline{\mathbf{A}}={\mathbf{R}}^{-\frac{1}{2}}\mathbf{A}$, $\overline{\mathbf{J}}={\mathbf{R}}^{-\frac{1}{2}}\mathbf{J}$, ${\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }={\mathbf{I}}_N-{\mathbf{P}}_{\overline{\mathbf{J}}}$, and ${\mathbf{P}}_{\overline{\mathbf{J}}}=\overline{\mathbf{J}}{\left({\overline{\mathbf{J}}}^H\overline{\mathbf{J}}\right)}^{-1}{\overline{\mathbf{J}}}^H$.

Substituting (10) and (19) into (7) results in the Rao test for given $\mathbf{R}$ and $\mathbf{D}$

$$\begin{array}{cc}\hfill t_{\mathrm{Rao}}& ={\mathrm{vec}}^T\left({\mathbf{A}}^T{\mathbf{R}}^{-T}{\mathbf{X}}_0^{\ast }\right)[{\mathbf{I}}_K\otimes {\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1}]\mathrm{vec}\left({\mathbf{A}}^H{\mathbf{R}}^{-1}{\mathbf{X}}_0\right)\hfill \\ \hfill & ={\mathrm{vec}}^T\left({\mathbf{A}}^T{\mathbf{R}}^{-T}{\mathbf{X}}_0^{\ast }\right)\mathrm{vec}\left[{\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1}{\overline{\mathbf{A}}}^H{\overline{\mathbf{X}}}_0\right]\hfill \\ \hfill & =\mathrm{tr}\left[{(\overline{\mathbf{X}}-\overline{\mathbf{J}}\mathbf{D})}^H\overline{\mathbf{A}}{\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1}{\overline{\mathbf{A}}}^H(\overline{\mathbf{X}}-\overline{\mathbf{J}}\mathbf{D})\right],\hfill \end{array}$$

(20)

where $\mathrm{tr}(·)$ denotes the trace of a square matrix. Setting (13) to be zero result the MLE of $\mathbf{D}$ under hypothesis ${\mathrm{H}}_0$

$$\hat{\mathbf{D}}={\left({\mathbf{J}}^T{\mathbf{R}}^{-T}\mathbf{J}\right)}^{-1}{\mathbf{J}}^T{\mathbf{R}}^{-T}\mathbf{X}.$$

(21)

Substituting (21) into (20) results in the Rao test for given $\mathbf{R}$

$$\begin{array}{c}\hfill t_{\mathrm{Rao}}=\mathrm{tr}\left[{\overline{\mathbf{X}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}{\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1}{\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{X}}\right]=\mathrm{tr}\left({\overline{\mathbf{X}}}^H{\mathbf{P}}_{{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}}\overline{\mathbf{X}}\right),\end{array}$$

(22)

According to (1), we have ${\mathbf{P}}_{{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}}={\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }$. Hence, we can re-express (22) as

$$t_{\mathrm{Rao}}=\mathrm{tr}\left({\overline{\mathbf{X}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{X}}\right),$$

(23)

which can be extended as

$$t_{\mathrm{Rao}}=\mathrm{tr}[{\mathbf{X}}^H{\mathbf{R}}^{-1}\mathbf{X}-{\mathbf{X}}^H{\mathbf{R}}^{-1}\mathbf{J}{\left({\mathbf{J}}^H{\mathbf{R}}^{-1}\mathbf{J}\right)}^{-1}{\mathbf{J}}^H{\mathbf{R}}^{-1}\mathbf{X}].$$

(24)

It is straightforward to show that

$${\hat{\mathbf{R}}}_0=\frac{1}{M+K}{\mathbf{S}}^{\frac{1}{2}}({\mathbf{I}}_N+{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{X}}{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }){\mathbf{S}}^{\frac{1}{2}}.$$

(25)

Performing matrix inversion to (25) and dropping the constant results in

$${\hat{\mathbf{R}}}_0^{-1}={\mathbf{S}}^{-\frac{1}{2}}[{\mathbf{I}}_N-{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{X}}{({\mathbf{I}}_K+{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{X}})}^{-1}{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }]{\mathbf{S}}^{-\frac{1}{2}}.$$

(26)

Substituting (26) into (24), we have the final Rao test

$$t_{\mathrm{Rao}}=\mathrm{tr}\left[{\tilde{\mathbf{X}}}^H\tilde{\mathbf{X}}-{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{X}}{({\mathbf{I}}_K+{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }{\tilde{\mathbf{X}}}^H)}^{-1}{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{X}}\right].$$

(27)

3.3. Wald Test

The Wald test for complex-valued parameters is [23]

$$t_{\mathrm{Wald}}={({\hat{\Theta }}_{{\mathrm{r}}_1}-{\Theta }_{{\mathrm{r}}_0})}^H{\left\{{\left[{\mathbf{I}}^{-1}\left({\hat{\Theta }}_1\right)\right]}_{\mathrm{r},\mathrm{r}}\right\}}^{-1}({\hat{\Theta }}_{{\mathrm{r}}_1}-{\Theta }_{{\mathrm{r}}_0})$$

(28)

where ${\hat{\Theta }}_{{\mathrm{r}}_1}$, ${\Theta }_{{\mathrm{r}}_0}$, and ${\hat{\Theta }}_1$ are the MLE of ${\Theta }_{\mathrm{r}}$ under hypothesis ${\mathrm{H}}_1$, the value of ${\hat{\Theta }}_{\mathrm{r}}$ under hypothesis ${\mathrm{H}}_0$, and the MLE of $\Theta$ under hypothesis ${\mathrm{H}}_1$, respectively. In a manner similar to (8), we have

$${\left\{{\left[{\mathbf{I}}^{-1}(\Theta )\right]}_{\mathrm{r},\mathrm{r}}\right\}}^{-1}={\mathbf{I}}_K\otimes {\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}.$$

(29)

The quantity ${\mathbf{X}}_1$ in (9) can be written as ${\mathbf{X}}_1=\mathbf{X}-\mathbf{C}\mathbf{F}$, where $\mathbf{C}=[\mathbf{A},\mathbf{J}]$ and $\mathbf{F}=[\mathbf{B};\mathbf{D}]$. Then nulling the derivative of (9) with respect to $\mathbf{F}$ results in the MLE of $\mathbf{F}$ as

$$\hat{\mathbf{F}}={\left({\mathbf{F}}^T{\mathbf{R}}^{-T}\mathbf{F}\right)}^{-1}{\mathbf{F}}^T{\mathbf{R}}^{-T}\mathbf{X}.$$

(30)

Hence, the MLE of $\mathbf{B}$ is the first p column of (30). In a manner similar to [15], we can derive the MLE of $\mathbf{B}$ as

$$\hat{\mathbf{B}}={\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1}{\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{X}}.$$

(31)

Substituting (29) and (31) into (28) results in the Wald test for given $\mathbf{R}$

$$\begin{array}{cc}\hfill t_{\mathrm{Wald}}=& {\mathrm{vec}}^H\left[{\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1}{\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{X}}\right][{\mathbf{I}}_K\otimes {\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}]\hfill \\ \hfill & ·\mathrm{vec}\left[{\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1}{\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{X}}\right]\hfill \\ \hfill =& {\mathrm{vec}}^H\left[{\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1}{\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{X}}\right]\mathrm{vec}\left[{\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{X}}\right]\hfill \\ \hfill =& \mathrm{tr}\left[{\overline{\mathbf{X}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}{\left({\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{A}}\right)}^{-1}{\overline{\mathbf{A}}}^H{\mathbf{P}}_{\overline{\mathbf{J}}}^{\perp }\overline{\mathbf{X}}\right],\hfill \end{array}$$

(32)

which can be extended as

$$\begin{array}{cc}\hfill t_{\mathrm{Wald}}=& \mathrm{tr}\left\{{\mathbf{X}}^H[{\mathbf{R}}^{-1}-{\mathbf{R}}^{-1}\mathbf{J}{\left({\mathbf{J}}^H{\mathbf{R}}^{-1}\mathbf{J}\right)}^{-1}{\mathbf{J}}^H{\mathbf{R}}^{-1}]\right.\hfill \\ \hfill & ·\mathbf{A}{[{\mathbf{A}}^H{\mathbf{R}}^{-1}\mathbf{A}-{\mathbf{A}}^H{\mathbf{R}}^{-1}\mathbf{J}{\left({\mathbf{J}}^H{\mathbf{R}}^{-1}\mathbf{J}\right)}^{-1}{\mathbf{J}}^H{\mathbf{R}}^{-1}\mathbf{A}]}^{-1}\hfill \\ \hfill & \left.·{\mathbf{A}}^H[{\mathbf{R}}^{-1}-{\mathbf{R}}^{-1}\mathbf{J}{\left({\mathbf{J}}^H{\mathbf{R}}^{-1}\mathbf{J}\right)}^{-1}{\mathbf{J}}^H{\mathbf{R}}^{-1}]\mathbf{X}\right\}.\hfill \end{array}$$

(33)

To obtain the final Wald test, we need the MLE of $\mathbf{R}$ under hypothesis ${\mathrm{H}}_1$. Similar to (25), we have

$${\hat{\mathbf{R}}}_1=\frac{1}{M+K}{\mathbf{S}}^{\frac{1}{2}}({\mathbf{I}}_N+{\mathbf{P}}_{\tilde{\mathbf{C}}}^{\perp }\tilde{\mathbf{X}}{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{C}}}^{\perp }){\mathbf{S}}^{\frac{1}{2}}.$$

(34)

Performing matrix inversion to (34) and dropping the constant results in

$${\hat{\mathbf{R}}}_1^{-1}={\mathbf{S}}^{-\frac{1}{2}}[{\mathbf{I}}_N-{\mathbf{P}}_{\tilde{\mathbf{C}}}^{\perp }\tilde{\mathbf{X}}{({\mathbf{I}}_K+{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{C}}}^{\perp }\tilde{\mathbf{X}})}^{-1}{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{C}}}^{\perp }]{\mathbf{S}}^{-\frac{1}{2}}.$$

(35)

Post-multiplying (35) by $\mathbf{A}$ and $\mathbf{J}$ results in

$${\hat{\mathbf{R}}}_1^{-1}\mathbf{A}={\mathbf{S}}^{-1}\mathbf{A}$$

(36)

and

$${\hat{\mathbf{R}}}_1^{-1}\mathbf{J}={\mathbf{S}}^{-1}\mathbf{J},$$

(37)

respectively. Substituting these two equalities into (33) results in

$$\begin{array}{c}\hfill t_{\mathrm{Wald}}=\mathrm{tr}\left[{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{A}}{\left({\tilde{\mathbf{A}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{A}}\right)}^{-1}{\tilde{\mathbf{A}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{X}}\right]=\mathrm{tr}\left({\tilde{\mathbf{X}}}^H{\mathbf{P}}_{{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{A}}}\tilde{\mathbf{X}}\right).\end{array}$$

(38)

It can be shown that ${\mathbf{P}}_{{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{A}}}={\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }$. Hence, we can rewrite (38) as

$$\begin{array}{c}\hfill t_{\mathrm{Wald}}=\mathrm{tr}\left({\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{J}}}^{\perp }\tilde{\mathbf{X}}\right).\end{array}$$

(39)

4. Performance Comparison

In this section, we investigate the detection performance of the proposed detectors with the existing ones. We compare the detection performance of the proposed detectors with the GLRT (We choose the GLRT0 for comparison, because the GLRT is the most common criterion for detector design.) for the detection problem in (2) when the interference is not taken into consideration, which is given by [24,25]

$$t_{\mathrm{GLRT}0}=\frac{\det ({\mathbf{I}}_K+{\tilde{\mathbf{X}}}^H\tilde{\mathbf{X}})}{\det ({\mathbf{I}}_K+{\tilde{\mathbf{X}}}^H{\mathbf{P}}_{\tilde{\mathbf{A}}}^{\perp }\tilde{\mathbf{X}})}.$$

(40)

For convenience, we denote the detector in (40) as GLRT0.

In Figure 1 and Figure 2, the noise covariance matrix is assumed to have the form $\mathbf{R}(i_1,i_2)={\sigma }^2{\rho }^{|i_1-i_2|}$, $i_1,i_2=1,2,\dots ,N$, and we choose ${\sigma }^2=1$ and $\rho =0.95$. The signal-to-noise ratio (SNR) is defined as

$$\mathrm{SNR}=\frac{1}{{\sigma }^2}\mathrm{tr}\left({\mathbf{B}}^H{\mathbf{A}}^H\mathbf{A}\mathbf{B}\right).$$

(41)

Similarly, the interference-to-noise ratio (INR), when assuming the actual unknown interference, has the form ${\mathbf{Q}}_r={\mathbf{J}}_r{\mathbf{D}}_r$, which is defined as

$$\mathrm{INR}=\frac{1}{{\sigma }^2}\mathrm{tr}\left({\mathbf{D}}_r^H{\mathbf{J}}_r^H{\mathbf{J}}_r{\mathbf{D}}_r\right).$$

(42)

Figure 1 displays the PDs of the detectors under different SNRs when the target signal has the form $\mathbf{H}=\mathbf{a}{\mathbf{b}}^H$,

$$\mathbf{a}={[1,{\mathrm{e}}^{-j2\pi {\theta }_t},\dots ,{\mathrm{e}}^{-j2\pi (N-1){\theta }_t}]}^T,$$

(43)

${\theta }_t$ is the normalized spatial frequency or normalized Doppler frequency, set to be ${\theta }_t=0.2$, and $\mathbf{b}$ is chosen to satisfy a certain SNR. Moreover, the unknown interference has the form ${\mathbf{Q}}_r={\mathbf{J}}_r{\mathbf{D}}_r$, where ${\mathbf{J}}_r=[{\mathbf{j}}_1,{\mathbf{j}}_2]$,

$${\mathbf{j}}_i={[1,{\mathrm{e}}^{-j2\pi {\theta }_i},\dots ,{\mathrm{e}}^{-j2\pi (N-1){\theta }_i}]}^T,$$

(44)

${\theta }_1=0.25$, ${\theta }_2=0.4$, and ${\mathbf{D}}_r$ is chosen to satisfy a given INR. The results show that the proposed GLRT and Wald test provide higher PDs than the GLRT0, which in turn has the higher PD than the Rao test. Moreover, the proposed GLRT has the same PD as the proposed Wald test for the chosen parameters. In fact, the GLRT is statistically equivalent to the Wald test when $p=1$. This is shown in Appendix A.

Figure 2 shows the PDs of the detectors under different SNRs when the signal has the form $\mathbf{H}=\mathbf{A}\mathbf{B}$, $\mathbf{A}=[{\mathbf{a}}_1,{\mathbf{a}}_2]$,

$${\mathbf{a}}_i={[1,{\mathrm{e}}^{-j2\pi {\theta }_{t_i}},\dots ,{\mathrm{e}}^{-j2\pi (N-1){\theta }_{t_i}}]}^T,$$

(45)

${\theta }_{t_1}=-0.2$, ${\theta }_{t_2}=0.3$, and the normalized interference angles are the same as those in Figure 1. The results indicate that the proposed Wald test has the highest PD, and the performance improvement in terms of SNR when PD = 0.8 is more than 2.5 dB, compared with the GLRT0. Moreover, comparing the results in Figure 1 and Figure 2 highlights that the PDs of the detectors in Figure 1 are lower than those in Figure 2. This is because the angles of the target and interference are closer in Figure 1 than in Figure 2.

In Figure 3, the covariance matrix has the form $\mathbf{R}(i_1,i_2)=0.{95}^{|i_1-i_2|}{\mathrm{e}}^{-(i_1-i_2)f_{d_c}}$, with ${\sigma }^2=0.95$ and $f_{d_c}=0.05$. This model of the covariance matrix can be taken as a generalization of that in Figure 1 and Figure 2, since the case of $f_{d_c}=0$ degenerates into the one adopted in Figure 1 and Figure 2. The results show that the proposed GLRT and the Wald test have much higher PDs than the GLRT0. Moreover, all the detectors suffer from performance loss for the chosen parameter setting, compared with the results in Figure 1 and Figure 2. Gathering the results in Figure 1, Figure 2, Figure 3 and Figure 4 indicates that the construction of the covariance matrix can affect the detection performance.

Figure 4 and Figure 5 display the PDs of the detectors under different INRs. In Figure 4, the target normalized angle is ${\theta }_t=0.2$, and the interference normalized angles are ${\theta }_1=0.25$, ${\theta }_2=-0.2$, and ${\theta }_3=0.4$. In Figure 5, the target normalized angles are the same as those in Figure 4, while the interference normalized angles are ${\theta }_1=0.25$, ${\theta }_2=0.3$, and ${\theta }_3=0.4$. The results show that the PDs of the detectors decrease with the increase of the INR. Comparing the results in Figure 4 and Figure 5 indicates that when the interference is close to the target, the PD of a detector will decrease.

5. Conclusions

In this paper, we considered the problem of detecting a distributed target in unknown interference. To devise effective detectors, we temporally assumed that the interference is orthogonal to the signal subspace. Then, we proposed three detectors according to the GLRT, the Rao test, and the Wald test. An interesting finding is that the detection statistics of the three proposed detectors do not directly depend on the signal matrix. Instead, they depend on the matrix which is orthogonal to the signal matrix. Moreover, it is found that the GLRT coincides with the Wald test when the dimension of the signal subspace is equal to unity. Numerical examples show that all the three proposed detectors can effectively detect the target in the presence of unknown interference. The GLRT and the Wald test can achieve better detection performance than the existing detectors.