The Geometry of Signal Detection with Applications to Radar Signal Processing

Abstract

The problem of hypothesis testing in the Neyman–Pearson formulation is considered from a geometric viewpoint. In particular, a concise geometric interpretation of deterministic and random signal detection in the philosophy of information geometry is presented. In such a framework, both hypotheses and detectors can be treated as geometrical objects on the statistical manifold of a parameterized family of probability distributions. Both the detector and detection performance are geometrically elucidated in terms of the Kullback–Leibler divergence. Compared to the likelihood ratio test, the geometric interpretation provides a consistent but more comprehensive means to understand and deal with signal detection problems in a rather convenient manner. Example of the geometry based detector in radar constant false alarm rate (CFAR) detection is presented, which shows its advantage over the classical processing method.

1. Introduction

Signal detection in the presence of noise is a basic issue in statistical signal processing and is fundamental to target detection applications in modern radar signal processing systems. Classical approaches to binary hypothesis testing are usually in the form of a likelihood ratio test. It has been proved that the optimal detector for a known deterministic signal in Gaussian noise is termed the matched filter, while for a random signal with known statistics, it is named the estimator–correlator in which the signal estimate is correlated with the data [1]. The essence of a basic detection problem can be regarded as decision or discrimination between two probability distributions with different models or parameters. While the likelihood ratio test provides an optimal solution to hypothesis testing problems, its meaning is not obvious in terms of the discrimination nature of a detection problem. The term “discrimination” is usually related to a decision or comparison in terms of distance between two items. There is no doubt that a better understanding of the detection problem will benefit from interpretations of its discrimination nature. Furthermore, it is promising to develop new approaches revealed by such interpretations to deal with the detection problem itself.

In statistics and information theory, a parameterized family of identically distributed probability distributions is usually regarded as a statistical manifold [2]. The statistical manifold of probability distributions has a natural geometrical structure that characterizes the intrinsic statistical properties of the family of distributions. Information geometry is the study of intrinsic properties of manifolds of probability distributions [3], where the ability of the data to discriminate those distributions is translated into a Riemannian metric [4]. The main tenet of information geometry is that many important notions in probability theory, information theory and statistics can be treated as structures in differential geometry by regarding a space of probabilities as a differentiable manifold endowed with a Riemannian metric and a family of affine connections [5]. By providing a means to analyze the Riemannian geometric properties of various families of probability density functions, information geometry offers comprehensive results about statistical models simply by considering them as geometrical objects.

The geometric theory of statistics, pioneered in the 1940s by Rao [4] and further developed by Chentsov [6], Efron [7,8] and Amari [5,9], has found a wide range of applications in the theory of statistical inference [10,11], the study of Boltzmann machine [12], the learning of neural networks [13], the Expectation–Maximization (EM) algorithm [14], all with certain degrees of success. In the last two decades, its application has spanned several discipline areas such as information theory [15,16], systems theory [17,18], mathematical programming [19], and statistical physics [20]. As this new general theory reveals the capability of defining a new perspective on existing questions, many researchers are extending their work on this geometric theory of information to new areas of application and interpretation. For example, in the area of communication coding/decoding and channel analysis, the most important milestone is the work of Richardson where the geometric perspective clearly indicates the relationship between turbo decoding and maximum-likelihood decoding [21], while a geometric characterization of multiuser detection for synchronous direct-sequence code division multiple access (DS/CDMA) channels is presented in [22]; In the area of estimation and filtering, Smith [23] studied the intrinsic Cramér–Rao bounds on estimation accuracy for the estimation problems on arbitrary manifolds, where the set of intrinsic coordinates are not apparent. Srivastava et al. [24,25] addressed the geometric subspace estimation and target tracking problems under a Bayesian framework. In particular, in the area of hypothesis testing and signal detection, the geometrical interpretation of multiple hypothesis testing problems in the asymptotic limit was developed by Westover [26], while the interpretation of Bayes risk error in Bayesian hypothesis testing was demonstrated by Varshney [27] and it has been proved that the Bayes risk error is a member of the class of Bregman divergences. Furthermore, Barbaresco et al. [28,29,30] developed a new Matrix constant false alarm rate (CFAR) detection method based on Bruhat–Tits complete metric space and upper-half Siegel space in symplectic geometry, which improves the detection performance with respect to the classical CFAR detection approach based on Doppler Filter Bank or fast Fourier transform (FFT).

Information geometry opens a new prospective to study the intrinsic geometrical nature of information theory and provides a new way to deal with existing statistical problems. For signal detection in statistical signal processing, information geometry can also provide new viewpoints and better understandings in the analysis of the detection process where both hypotheses and detector can be regarded as geometrical objects on the statistical manifold. In this paper, geometrical interpretations of deterministic and random signal detection are presented from the viewpoint of information geometry. Under such interpretations, the optimal detector is a “generalized minimum distance detector” where the decision is made by comparing distances from the signal distribution estimates to two hypotheses in the sense of the Kullback–Leibler divergence (KLD). Results, presented in this paper, illustrate that the generalized minimum distance detector provides a consistent but more comprehensive means to understand signal detection problems compared to the likelihood ratio test. An example of the geometry based detector in radar CFAR detection is presented, which shows its advantage over the classical processing method.

In the next section, the equivalence between likelihood ratio test and the Kullback–Leibler divergence is discussed, followed by the geometric interpretation of classical likelihood ratio test from the viewpoint of information geometry. In Section 3 and Section 4, the geometry of deterministic and random signal detection as well as the detection performance is elucidated geometrically, respectively. An example of the geometry based detector in radar CFAR detection is presented in Section 5. Finally, conclusions are made in Section 6.

2. Geometric Interpretation of the Classical Likelihood Ratio Test

2.1. Equivalence between Likelihood Ratio Test and Kullback–Leibler Divergence

In statistics, the likelihood ratio test is a basic form for hypothesis testing. Suppose $x_1,x_2,\dots ,x_N$ are N independent and identically distributed (i.i.d.) observations from a statistical model whose probability density function is $q\left(x\right)$, i.e., $x_1,x_2,\dots ,x_N\stackrel{i.i.d.}{\sim }q\left(x\right)$. Consider two hypotheses for $q\left(x\right)$, denoted by $p_0\left(x\right)$ and $p_1\left(x\right)$, which are referred to as the null hypothesis and alternative hypothesis, respectively. Then, the binary hypothesis testing problem is to decide between two possible hypotheses based on the observed data $x_1,x_2,\dots ,x_N$ [1]. The likelihood ratio for the underlying hypothesis test problem is

$$L=\prod _{i=1}^N\frac{p_1\left(x_i\right)}{p_0\left(x_i\right)}.$$

(1)

The normalized log likelihood ratio can be represented by

$$\overline{l}=\frac{1}{N}lnL=\frac{1}{N}\sum _{i=1}^{N}ln\frac{p_1\left(x_i\right)}{p_0\left(x_i\right)}.$$

(2)

The normalized log likelihood ratio $\overline{l}$ is itself a random variable, and is an arithmetic average of N i.i.d. random variables $l_i=ln\frac{p_1\left(x_i\right)}{p_0\left(x_i\right)}$. According to the strong law of large numbers [31] that for large N, as

$$\overline{l}\stackrel{P}{⟶}E\left[l_i\right],$$

(3)

where P denotes the convergence in probability, E is the expectation of random variables.

As $l_i,i=1,\dots ,N$ are i.i.d. random variables and asymptotically

$$\begin{array}{cc}\hfill \overline{l}=E\left[l_i\right]& =\int q\left(x\right)ln\frac{p_1\left(x\right)}{p_0\left(x\right)}dx=\int q\left(x\right)ln\left[\frac{p_1\left(x\right)}{p_0\left(x\right)}\frac{q\left(x\right)}{q\left(x\right)}\right]dx\hfill \\ & =\int q\left(x\right)\left[ln\frac{q\left(x\right)}{p_0\left(x\right)}-ln\frac{q\left(x\right)}{p_1\left(x\right)}\right]dx\hfill \\ & =\int q\left(x\right)ln\frac{q\left(x\right)}{p_0\left(x\right)}dx-\int q\left(x\right)ln\frac{q\left(x\right)}{p_1\left(x\right)}dx.\hfill \end{array}$$

(4)

The quantity $\int q\left(x\right)ln\frac{q\left(x\right)}{p\left(x\right)}dx$ is known as the Kullback–Leibler divergence (KLD) [32] from q to p, which can be denoted by

$$D(q\parallel p)=\int q\left(x\right)ln\frac{q\left(x\right)}{p\left(x\right)}dx.$$

(5)

The Kullback–Leibler divergence is widely used to measure the similarity (distance) between two distributions and is also termed the relative entropy [33] in information theory. It should be noted that the Kullback–Leibler divergence is not a genuine metric of distance as it fails to be symmetric or satisfy the triangle inequality, which is justified by the Pinsker–Csiszar Inequality [34]. However, it has been proved that the KLD plays a central role in the theory of statistical inference [8,9].

The Kullback–Leibler divergence can be regarded as the expected log-likelihood ratio. The equivalence between the likelihood ratio and Kullback–Leibler divergence is indicated by Equation (4) as

$$\overline{l}=D(q\parallel p_0)-D(q\parallel p_1).$$

(6)

Therefore, for large N, the classical log likelihood ratio test $lnL\underset{{\mathcal{H}}_0}{\stackrel{{\mathcal{H}}_1}{\gtrless }}\gamma$ can be performed by

$$D(q\parallel p_0)-D(q\parallel p_1)\underset{{\mathcal{H}}_0}{\stackrel{{\mathcal{H}}_1}{\gtrless }}\frac{1}{N}\gamma \triangleq {\gamma }^{\prime }.$$

(7)

In particular, for $\gamma =0$ where the prior probabilities of two hypotheses are equal, the decision becomes

$$D(q\parallel p_0)\underset{{\mathcal{H}}_0}{\stackrel{{\mathcal{H}}_1}{\gtrless }}D(q\parallel p_1).$$

(8)

In such a case, the likelihood ratio test is equivalent to choosing the model that is “closer” to q in the sense of the Kullback–Leibler divergence. This is referred to as a minimum distance detector. For the general case given by Equation (7), the test can be regarded as a generalized minimum distance detector with a threshold ${\gamma }^{\prime }$. Similar results can also be found in [26,33], where the analysis is performed on a probability simplex. In practical applications, typically in radar and sonar systems, the Neyman–Pearson criterion [35] is employed so that the detector is designed to maximize the probability of detection $P_D$ for a given probability of false alarm $P_F$. In such a case, the probability of detection, i.e., the miss probability $P_M$ of the detector, is especially concerned as a performance index of the detector. Generally speaking, the miss probability of the Neyman–Pearson detector decays exponentially with a fixed level as the sample size increases, and the error exponent K is a widely used measure as the rate of exponential decay [36], i.e.,

$$K\triangleq \underset{n\to \infty }{lim}-\frac{1}{N}log{P}_M$$

(9)

under the given false-alarm constraint.

Stein’s lemma [33,37] shows that the best error exponent under a fixed false-alarm constraint is given by the Kullback–Leibler divergence $D(p_0\parallel p_1)$ from the distribution of hypothesis $p_0$ to $p_1$, i.e.,

$$K=D(p_0\parallel p_1)$$

(10)

and

$$P_M\doteq 2^{-NK},given{P}_F=\alpha ,$$

(11)

where ≐ denotes equality to first order in the exponent, i.e., $a_n\doteq b_n$ means $\underset{n\to \infty }{\lim }\frac{1}{n}log\frac{a_n}{b_n}=0$.

The error exponent provides a good performance index for Neyman–Pearson detectors in the large sample regime. Furthermore, discussions of the error exponent and its properties are addressed in [36]. In the following sections of this paper, the above notions will be demonstrated through discussions on the geometry of deterministic and random signal detection.

2.2. Geometric Interpretation of the Classical Likelihood Ratio Test

The previous analysis of the geometry of hypothesis testing is performed on a probability simplex [38], which is a generalization of the notion of a triangle or tetrahedron to arbitrary dimension. The vertex labels of a probability simplex correspond to a single probability set to 1, while edge labels correspond to a sum of two probabilities set to 1 and triangular faces correspond to a sum of three probabilities set to 1. The probability simplex provides a simple structure to analyze testing problems of probability distributions by mapping the type (empirical histogram) of data to a point on the simplex. However, it is not accomplished in dealing with continuous probability distributions as well as a complete structure to perform higher level of analysis. In a sense, information geometry provides a more powerful mathematical tool to deal with statistical problems in a geometrical manner, in which a parameterized family of probability distributions can be conveniently referred to as a statistical manifold with precise geometrical structure.

As mentioned earlier, the essence of a basic hypothesis testing problem can be regarded as discrimination between two identically distributed distributions with different parameters. From the viewpoint of information geometry, probability distributions of hypotheses can be conveniently regarded as two elements of a statistical manifold. Consider the parameterized family of probability distributions $S=\left\{p\right(\mathit{x}\left|\mathbf{\theta }\right)\}$, where $\mathit{x}$ is a random variable and $\mathbf{\theta }=({\theta }_1,\dots ,{\theta }_n)$ is a parameter vector specifying the distribution. The family S is regarded as a statistical manifold with $\mathbf{\theta }$ as its (possibly local) coordinate system [2].

Figure 1 illustrates the definition of a statistical manifold. For a given state of interest $\mathbf{\theta }$ in the parameter space $\mathsf{\Theta }\in {\mathbb{R}}^n$, the measurement $\mathit{x}$ in the sample space $\mathit{X}\in {\mathbb{R}}^m$ is a realisation of to a probability distribution $p\left(\mathit{x}\right|\mathbf{\theta })$. Each probability distribution $p\left(\mathit{x}\right|\mathbf{\theta })$ is labelled by a point $s\left(\mathbf{\theta }\right)$ on the manifold S. The parameterized family of probability distributions $S=\left\{p\right(\mathit{x}\left|\mathbf{\theta }\right)\}$ forms an n-dimensional statistical manifold, where $\mathbf{\theta }$ plays the role of a coordinate system of $\mathit{S}$.

For a sequence of observations $\mathit{x}={(x_1,x_2,\dots ,x_n)}^T$, the binary hypothesis testing problem is that of deciding whether this sequence is originated from a source (hypothesis ${\mathcal{H}}_1$) with a probability distribution $p\left(\mathit{x}\right|{\mathbf{\theta }}_1)$ or from a source (hypothesis ${\mathcal{H}}_0$) with a probability distribution $p\left(\mathit{x}\right|{\mathbf{\theta }}_0)$, based on observations $\mathit{x}$. The probability distribution $p\left(\mathit{x}\right|{\mathbf{\theta }}_i),i=1,2$, associated with the hypothesis ${\mathcal{H}}_i$, is a element of a parametric family of probability density functions $S=\left\{p\right(\mathit{x}\left|\mathbf{\theta }\right),\mathbf{\theta }\in \mathsf{\Theta }\}$, where $\mathsf{\Theta }\in {\mathbb{R}}^n$ is the parameter set. This family of distributions $S=\left\{p\right(\mathit{x}\left|\mathbf{\theta }\right)\}$ makes up of a statistical manifold, where the parameter $\mathbf{\theta }$ is the coordinate of the measurement distribution on the statistical manifold. Distributions of the two model (hypotheses) $p\left(\mathit{x}\right|{\mathbf{\theta }}_0)$ and $p\left(\mathit{x}\right|{\mathbf{\theta }}_1)$ can be regarded as two “datum marks” on the manifold, while the distribution $p\left(\mathit{x}\right|\mathbf{\theta })$ summarized from observations $\mathit{x}$ is an estimated distribution on the manifold. Then, the above hypothesis testing problem becomes a discrimination problem where the decision is made by comparing distances from the signal distribution estimates to two hypotheses in the sense of the Kullback–Leibler divergence, i.e., selecting the model that is “closer” to the signal distribution estimates. The geometric interpretation of classical likelihood ratio test from the viewpoint of information geometry can be illustrated by Figure 2. The above analysis can be extended to the complex parameter $\mathbf{\theta }$ and/or complex measurement $\mathit{x}$.

3. Geometry of Deterministic Signal Detection

3.1. Signal Model and Likelihood Ratio Test

The detection of a known deterministic signal in Gaussian noise is a basic detection problem encountered in statistical signal processing. Meanwhile, in radar applications, it usually involves the complex data. Consider the detection of a known deterministic signal in complex Gaussian noise. The hypothesis testing problem is [1]

$$\left\{\begin{array}{cc}{\mathcal{H}}_0:x\left(n\right)=w\left(n\right),\hfill & \hfill n=0,1,\dots ,N-1,\\ {\mathcal{H}}_1:x\left(n\right)=s\left(n\right)+w\left(n\right),\hfill & \hfill n=0,1,\dots ,N-1,\end{array}\right.$$

(12)

where $s\left(n\right)$ is a known complex signal and $w\left(n\right)$ is a complex correlated Gaussian noise with zero mean. The above discrete quantities can be written as the vector forms,

$$\mathit{x}={\left[x\left(0\right),x\left(1\right),\dots ,x(N-1)\right]}^T,\mathit{s}={\left[s\left(0\right),s\left(1\right),\dots ,s(N-1)\right]}^T,\mathit{w}={\left[w\left(0\right),w\left(1\right),\dots ,w(N-1)\right]}^T.$$

Then, under the two hypotheses, the distributions of measurement error can be represented by

$$\left\{\begin{array}{c}{\mathcal{H}}_0:\mathit{x}\sim \mathcal{CN}\left(\mathbf{0},\mathsf{\Sigma }\right),\hfill \\ {\mathcal{H}}_1:\mathit{x}\sim \mathcal{CN}\left(\mathit{s},\mathsf{\Sigma }\right),\hfill \end{array}\right.$$

(13)

where Σ is the covariance of noise vector $\mathit{w}$ and $\mathcal{CN}$ denotes the complex Gaussian distribution.

Consequently, the underlying detection problem for a known deterministic signal can be regarded as discrimination between two distributions with the same covariance but different means. According to the Neyman–Pearson criterion, the likelihood ratio test reduces to deciding ${\mathcal{H}}_1$ if

$$T\left(\mathit{x}\right)=\mathrm{Re}\left({\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{x}\right)>\gamma ,$$

(14)

where $T\left(\mathit{x}\right)$ is the test statistic and H denotes the complex conjugate transpose. The detector described by Equation (14) is referred to as a generalized replica–correlator or generalized matched filter [1].

3.2. Geometry of Deterministic Signal Detection

As the vectorial measurement $\mathit{x}$ is from either of two hypotheses described in Equation (13), its covariance is Σ for either hypothesis. However, the mean of the measurements $\mathit{x}$ is unknown and should be estimated in order to decide which hypothesis is true. The measurement distribution can be summarized (estimated) by

$$p\left(\mathit{x}\right|\mathbf{\theta })\sim \mathcal{CN}\left(\mathit{x},\mathsf{\Sigma }\right),$$

(15)

where the mean is estimated by the mean of measurements, e.g., $\mathit{x}$.

The two datum marks on the manifold can be respectively denoted by

$$p\left(\mathit{x}\right|{\mathbf{\theta }}_1)\sim \mathcal{CN}\left(\mathit{s},\mathsf{\Sigma }\right),p\left(\mathit{x}\right|{\mathbf{\theta }}_0)\sim \mathcal{CN}\left(0,\mathsf{\Sigma }\right).$$

(16)

For two multivariate Gaussian distributions $p_0={\mathcal{CN}}_0\left({\mathbf{\mu }}_0,{\mathsf{\Sigma }}_0\right)$ and $p_1={\mathcal{CN}}_1\left({\mathbf{\mu }}_1,{\mathsf{\Sigma }}_1\right)$, the Kullback–Leibler divergence from $p_0$ to $p_1$ can be calculated via the closed form as [32]

$$D(p_0\parallel p_1)=\frac{1}{2}\left[ln\left(\frac{det{\mathsf{\Sigma }}_1}{det{\mathsf{\Sigma }}_0}\right)+\mathrm{tr}\left({\mathsf{\Sigma }}_1^{-1}{\mathsf{\Sigma }}_0\right)+{\left({\mathbf{\mu }}_1-{\mathbf{\mu }}_0\right)}^H{\mathsf{\Sigma }}_1^{-1}\left({\mathbf{\mu }}_1-{\mathbf{\mu }}_0\right)-N\right],$$

(17)

where “tr” and “det” denote the trace and determinant of a matrix, respectively.

According to Equation (17), the KLDs from the measurement distribution $p\left(\mathit{x}\right|\mathbf{\theta })$ to hypotheses $p\left(\mathit{x}\right|{\mathbf{\theta }}_1)$ and $p\left(\mathit{x}\right|{\mathbf{\theta }}_0)$ are given by

$$\begin{array}{cc}\hfill D(p\parallel p_1)& =\frac{1}{2}{(\mathit{x}-\mathit{s})}^H{\mathsf{\Sigma }}^{-1}(\mathit{x}-\mathit{s}),\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill D(p\parallel p_0)& =\frac{1}{2}{\mathit{x}}^H{\mathsf{\Sigma }}^{-1}\mathit{x}.\hfill \end{array}$$

(19)

Thus, the distance difference is

$$D(p\parallel p_0)-D(p\parallel p_1)=\frac{1}{2}\left[{\mathit{x}}^H{\mathsf{\Sigma }}^{-1}\mathit{x}-{(\mathit{x}-\mathit{s})}^H{\mathsf{\Sigma }}^{-1}(\mathit{x}-\mathit{s})\right]=\mathrm{Re}\left({\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{x}\right)-\frac{1}{2}{\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{s}.$$

(20)

By incorporating the non-data-dependent term into the threshold we decide ${\mathcal{H}}_1$ if

$$\mathcal{D}\left(\mathit{x}\right)=\mathrm{Re}\left({\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{x}\right)>\gamma ,$$

(21)

where $\mathcal{D}\left(\mathit{x}\right)$ is the test statistic of the generalized minimum distance detector.

By comparing the results given by Equation (14) and Equation (21), the detector defined by the generalized minimum distance detector coincides with the result given by the likelihood ratio test. However, the former illustrates a clear geometric interpretation of the detection problem in terms of its discrimination nature.

For above detectors, the test statistic is a complex Gaussian random variable under either hypothesis, being a linear transformation of $\mathit{x}$. Its statistics can be represented by

$$T\left(\mathit{x}\right)\sim \left\{\begin{array}{cc}\mathcal{N}\left(\mathbf{0},{\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{s}/2\right)\hfill & \hfill \mathrm{under}{\mathcal{H}}_0,\\ \mathcal{N}\left({\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{s},{\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{s}/2\right)\hfill & \hfill \mathrm{under}{\mathcal{H}}_1,\end{array}\right.$$

(22)

where we have noted that ${\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{s}$ is real.

For a given probability of false alarm $P_F$, the threshold is given by

$$\gamma =\sqrt{{\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{s}/2}{Q}^{-1}\left(P_F\right),$$

(23)

where $Q(·)$ is the right tail cumulative distribution function of the standard Gaussian distribution.

The corresponding probability of detection is given by [1]

$$P_D=Q\left[Q^{-1}\left(P_F\right)-\sqrt{d^2}\right],$$

(24)

where the deflection coefficient is defined by $d^2=2{\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{s}$, which can be regarded as the signal-to-noise ratio (SNR) of the underlying detection problem. Equation (24) indicates that the probability of detection increases monotonically with ${\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{s}$.

From the geometrical viewpoint, as shown in Equations (10) and (11), the performance of the Neyman–Pearson detector under a given false-alarm constraint can be measured by the error exponent, which is associated with the Kullback–Leibler divergence between two hypothesis distributions. In fact, in the deterministic signal detection example, the deflection coefficient $d^2$ is equivalent to the KLD $D(p_0\parallel p_1)$ from $p\left(\mathit{x}\right|{\mathbf{\theta }}_0)$ to $p\left(\mathit{x}\right|{\mathbf{\theta }}_1)$, which denotes distributions of the two hypotheses, as

$$D(p_0\parallel p_1)=\frac{1}{2}{\mathit{s}}^H{\mathsf{\Sigma }}^{-1}\mathit{s}.$$

(25)

The result in Equation (25) indicates that the detection performance depends on the distance between two hypothesis distributions, which gives a clear geometric meaning of the detection performance by viewing the detection process as a discrimination between two hypotheses.

4. Geometry of Random Signal Detection

In the last section, we discussed the deterministic signal detection problem as well as its geometry. It is able to detect signals in the presence of noise by detecting the change in the mean of a test statistic. This is because the presence of a deterministic signal alters the mean of the received data. However, in some cases, the signal is more appropriately to be modeled as a random process with a known covariance structure. In this section, we consider the detection from the random signal model as well as the geometry of random signal detection.

4.1. Signal Model and Likelihood Ratio Test

The signal model of a random signal with known statistics is similar to the deterministic case given by Equation (12). The only difference is that the signal $s\left(n\right)$ is assumed as a zero-mean complex Gaussian random process, and under the two hypotheses, the distributions of measurement error can be represented by

$$\left\{\begin{array}{c}{\mathcal{H}}_0:\mathit{x}\sim \mathcal{CN}\left(\mathbf{0},{\mathsf{\Sigma }}_w\right),\hfill \\ {\mathcal{H}}_1:\mathit{x}\sim \mathcal{CN}\left(\mathbf{0},{\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w\right),\hfill \end{array}\right.$$

(26)

where ${\mathsf{\Sigma }}_s$ and ${\mathsf{\Sigma }}_w$ denote the covariance matrices of the random signal and noise, respectively.

Consequently, the underlying detection problem for a random signal with known statistics in the presence of noise can be regarded as discrimination between two distributions with the same mean but different covariance matrices. The Neyman–Pearson detector decides ${\mathcal{H}}_1$ if the likelihood ratio exceeds a threshold or if

$$T\left(\mathit{x}\right)={\mathit{x}}^H{\mathsf{\Sigma }}_w^{-1}{\mathsf{\Sigma }}_s{\left({\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w\right)}^{-1}\mathit{x}>\gamma ,$$

(27)

where $T\left(\mathit{x}\right)$ is the test statistic.

4.2. Geometry of Random Signal Detection

As measurements $\mathit{x}$ are from either of the two hypotheses described in Equation (26), its mean is zero for either hypothesis. However, the covariance of $\mathit{x}$ is uncertain and should be estimated in order to decide which hypothesis is true. Assume that both the signal and noise are zero mean wide sense stationary (WSS) random process; then, the covariance of data $\mathit{x}$ is defined by

$$\mathit{R}\triangleq E\left[(\mathit{x}-{\mathbf{\mu }}_x){(\mathit{x}-{\mathbf{\mu }}_x)}^H\right]=\left[\begin{array}{cccc}{r}_0& {\overline{r}}_1& \cdots & {\overline{r}}_{N-1}\\ r_1& r_0& \ddots & ⋮\\ ⋮& \ddots & \ddots & {\overline{r}}_1\\ r_{N-1}& \cdots & r_1& r_0\end{array}\right],$$

(28)

where $r_k=E\left[x_n{\overline{x}}_{n+k}\right]$ is called the correlation coefficient and $\overline{x}$ denotes the complex conjugate of x. $\mathit{R}$ is a Toeplitz Hermitian Positive Definite matrix with ${\mathit{R}}^H=\mathit{R}$. According to the ergodicity of a WSS random process, the correlation coefficients of the signal can be calculated by averaging over time instead of its statistical expectation, as

$${\widehat{r}}_k=\frac{1}{N}\sum _{n=0}^{N-1-\left|k\right|}x\left(n\right)\overline{x}(n+k),\left|k\right|⩽N-1.$$

(29)

Thus, the measurement error distribution can be summarized (estimated) by

$$p\left(\mathit{x}\right|\mathbf{\theta })\sim \mathcal{CN}\left(\mathbf{0},\mathit{R}\right),$$

(30)

where the covariance $\mathit{R}$ is estimated from measurements $\mathit{x}$.

Then, the two datum marks on the manifold can be, respectively, denoted by

$$p\left(\mathit{x}\right|{\mathbf{\theta }}_1)\sim \mathcal{CN}\left(\mathbf{0},{\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w\right),p\left(\mathit{x}\right|{\mathbf{\theta }}_0)\sim \mathcal{CN}\left(\mathbf{0},{\mathsf{\Sigma }}_w\right).$$

(31)

Using Equation (17), the KLDs from the measurement distribution $p\left(\mathit{x}\right|\mathbf{\theta })$ to hypotheses $p\left(\mathit{x}\right|{\mathbf{\theta }}_1)$ and $p\left(\mathit{x}\right|{\mathbf{\theta }}_0)$ are given by

$$\begin{array}{cc}\hfill D(p\parallel p_1)& =\frac{1}{2}\left[\mathrm{tr}\left({({\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w)}^{-1}\mathit{R}-\mathbf{I}\right)-lndet\left({({\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w)}^{-1}\mathit{R}\right)\right],\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill D(p\parallel p_0)& =\frac{1}{2}\left[\mathrm{tr}\left({\mathsf{\Sigma }}_w^{-1}\mathit{R}-\mathbf{I}\right)-lndet\left({\mathsf{\Sigma }}_w^{-1}\mathit{R}\right)\right].\hfill \end{array}$$

(33)

Consequently, the distance difference is

$$\begin{array}{cc}\hfill D(p\parallel p_0)-D(p\parallel p_1)& =\frac{1}{2}\left[\mathrm{tr}\left(\left({\mathsf{\Sigma }}_w^{-1}-{({\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w)}^{-1}\right)\mathit{R}\right)-lndet\left(({\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w){\mathsf{\Sigma }}_w^{-1}\right)\right]\hfill \\ & =\frac{1}{2}\mathrm{tr}\left[{\mathsf{\Sigma }}_w^{-1}{\mathsf{\Sigma }}_s{({\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w)}^{-1}\mathit{R}\right]-\frac{1}{2}lndet\left(\mathbf{I}+{\mathsf{\Sigma }}_s{\mathsf{\Sigma }}_w^{-1}\right).\hfill \end{array}$$

(34)

By incorporating the non-data-dependent term into the threshold, we decide ${\mathcal{H}}_1$ if

$$\mathcal{D}\left(\mathit{x}\right)=\mathrm{tr}\left[{\mathsf{\Sigma }}_w^{-1}{\mathsf{\Sigma }}_s{({\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w)}^{-1}\mathit{R}\right]>\gamma ,$$

(35)

where $\mathcal{D}\left(\mathit{x}\right)$ is the test statistic of the generalized minimum distance detector.

As the test statistic in Equation (27) is a quadratic form in the data and thus will not be a Gaussian random variable, the detection performance of the detector is quite often difficult to determine analytically [1], especially when the signal is a correlated random process with arbitrary covariance matrix. While the detector given in Equation (35) is associated with an estimated covariance $\mathit{R}$ based on measurements, analysis of its detection performance is also non-trivial. Thus, here, the performance comparison of two detectors is performed via Monte Carlo simulations. In addition, 1,000,000 Monte Carlo runs are performed and the performances of two detectors are illustrated in Figure 3. In the simulation, the number of measurement samples is $N=100$ and the probabilities of false alarm $P_F$ are assumed $P_F={10}^{-4},{10}^{-5}$, respectively. Figure 3 illustrates that the detection performance of two detectors coincide with each other, i.e., the likelihood ratio-based detector is equivalent to the geometry-based detector.

The error exponent of the above Neyman–Pearson detector under the fixed false-alarm constraint is given by the Kullback–Leibler divergence from $p_0$ to $p_1$, as

$$D(p_0\parallel p_1)=\frac{1}{2}\left[\mathrm{tr}\left({({\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w)}^{-1}{\mathsf{\Sigma }}_w\right)-lndet\left({({\mathsf{\Sigma }}_s+{\mathsf{\Sigma }}_w)}^{-1}{\mathsf{\Sigma }}_w\right)-N\right].$$

(36)

As a special case, when the signal and noise are both white random processes with variances ${\sigma }_s^2$ and ${\sigma }_w^2$, the error exponent is simplified by

$$\begin{array}{cc}\hfill K& =\frac{N}{2}\left(\frac{{\sigma }_w^2}{{\sigma }_s^2+{\sigma }_w^2}+ln\frac{{\sigma }_s^2+{\sigma }_w^2}{{\sigma }_w^2}-1\right)\hfill \\ & =\frac{N}{2}\left[\frac{1}{\mathrm{SNR}+1}+ln(\mathrm{SNR}+1)-1\right],\hfill \end{array}$$

(37)

where $\mathrm{SNR}=\frac{{\sigma }_s^2}{{\sigma }_w^2}$ denotes the signal-to-noise ratio.

As

$$\frac{dK}{d\mathrm{SNR}}=\frac{N·\mathrm{SNR}}{2{(\mathrm{SNR}+1)}^2}>0,$$

(38)

the error exponent is monotonically increasing as SNR increases. Moreover, as shown in Figure 4, the error exponent K increases linearly with respect to SNR at high SNR. The result indicates that the detection performance improves as the divergence between two hypotheses increases, which gives a clear geometric interpretation of the detection performance of the Neyman–Pearson detector.

5. Application to Radar Target Detection

In this section, the application of geometry based detector to radar constant false alarm rate (CFAR) detection with pulse Doppler radar is presented. The performance between the geometry based CFAR detector and the classical cell-averaging CFAR (CA-CFAR) detector is compared. As illustrated in Figure 5, in the classical CA-CFAR detector for pulse Doppler radar, the data $z_i$ in each range resolution cell is obtained from a square-law detector of Doppler filter banks, where the Doppler power spectral density of the sample data $\mathit{x}$ is estimated by fast Fourier transform (FFT). The CA-CFAR detector performs by comparing the data $z_D$ in the test cell to an adaptive detection threshold γ such that a CFAR is maintained. The detection threshold γ is determined by estimating the background clutter power of the test cell and multiplying a scaling factor T based on the desired probability of false alarm $P_F$. The power of the background clutter is estimated by the arithmetical mean of the data in the background range cells, which is given by

$$\overline{z}=\frac{1}{M}\sum _{i=1}^M{z}_i.$$

(39)

Due to the poor Doppler resolution as well as the energy spread of the Doppler filter banks, the classical CA-CFAR detector based on Doppler spectral estimation suffers from a severe performance degradation. Barbaresco et al. [28] developed a matrix CFAR detector based on Cartan’s geometry of the symmetric positive-definite (SPD) matrices space, which improves the detection performance of the classical CA-CFAR detector based on Doppler filter banks. As illustrated in Figure 6, the data ${\mathit{R}}_i$ in each resolution cell is a covariance matrix estimated by the sample data $\mathit{x}$ according to Equation (28). Then, calculate the distance between the covariance matrix ${\mathit{R}}_D$ of the cell under test and the mean matrix $\overline{\mathit{R}}$ of the reference cells around the cell under test. Finally, the detection is made by comparing the distance between ${\mathit{R}}_D$ and $\overline{\mathit{R}}$ with an adaptive detection threshold γ, which is related to the clutter power level as well as a multiplier corresponding to the desired probability of false alarm $P_F$. In [28], the Riemannian distance measure is used when calculating the distance as well as the mean matrix, as the geometry of the manifold of SPD matrices is considered.

The Riemannian distance between the covariance matrix of the cell under test and the mean matrix of reference cells is calculated by

$$d^2\left({\mathit{R}}_D,\overline{\mathit{R}}\right)={\parallel ln\left({\mathit{R}}_D^{-1/2}\overline{\mathit{R}}{\mathit{R}}_D^{-1/2}\right)\parallel }^2=\sum _{i=1}^N{ln}^2\left({\lambda }_k\right),$$

(40)

where ${\lambda }_k$ is the $k^{th}$ eigenvalue of ${\mathit{R}}_D^{-1/2}\overline{\mathit{R}}{\mathit{R}}_D^{-1/2}$.

The Riemannian mean of a set of SPD matrices $\{{\mathit{R}}_1,{\mathit{R}}_2,\dots ,{\mathit{R}}_M\}$ is calculated via the minimum sum of distance between $\mathit{R}$ and ${\mathit{R}}_i$ as [39]

$$\overline{\mathit{R}}=arg\underset{\mathit{R}}{min}\sum _{i=1}^M{d}^2\left(\mathit{R},{\mathit{R}}_i\right),$$

(41)

where $d(·,·)$ represents the Riemannian distance between two matrices.

One approach to solve Equation (41) is to use an iterative gradient algorithm that is based on the Jacobi field and the exponential map. In summary, the Karcher barycenter based Riemannian mean is given by the following iteration [40]

$${\overline{\mathit{R}}}_{k+1}={\overline{\mathit{R}}}_k^{1/2}exp\left({\eta }_k\sum _{i=1}^{M}ln\left({\overline{\mathit{R}}}_k^{-1/2}{\mathit{R}}_i{\overline{\mathit{R}}}_k^{-1/2}\right)\right){\overline{\mathit{R}}}_k^{1/2},$$

(42)

where ${\eta }_k$ denotes the step-length of iterations.

Then, the matrix CFAR detector based on Riemannian distance can be performed as

$$d\left({\mathit{R}}_D,\overline{\mathit{R}}\right)\underset{{\mathcal{H}}_0}{\stackrel{{\mathcal{H}}_1}{\gtrless }}\gamma .$$

(43)

The matrix CFAR detector proposed in [28] improves the performance of the classical CA-CFAR detector. However, as discussed in Section 2, the likelihood ratio test is equivalent to the generalized minimum distance detector in terms of the Kullback–Leibler divergence, which indicates that the Kullback–Leibler divergence is suitable for detection applications. Thus, it is necessary to compare the performance of matrix CFAR detector in terms of the Riemannian distance measure and Kullback–Leibler divergence. The Kullback–Leibler divergence between two matrix is given by Equation (17), where the mean of a set of SPD matrices $\{{\mathit{R}}_1,{\mathit{R}}_2,\dots ,{\mathit{R}}_M\}$ is calculated by

$${\overline{\mathit{R}}}_{KL}=arg\underset{\mathit{R}}{min}\sum _{i=1}^M{D}^2(\mathit{R}\parallel {\mathit{R}}_i)={\left(\frac{1}{M}\sum _{i=1}^M{\mathit{R}}_i^{-1}\right)}^{-1}.$$

(44)

The performance of: (i) the classical CA-CFAR detector based on Doppler spectral estimation, (ii) the Riemannian distance-based matrix CFAR detector, and (iii) the Kullback–Leibler divergence-based matrix CFAR detector are compared via Monte Carlo simulations. The data samples are from $N=7$ received pulses. A total of 50 range cells are considered, while $M=14$ range cells are used for averaging. One moving target with Doppler frequency $f_d=2.65\mathrm{Hz}$ is located at the 19th range cell. As the Gaussian assumption is no longer met in many situations of practical interest, such as the high-resolution radar and sea clutter, a compound-Gaussian model is more appropriate to describe the non-Gaussian radar clutter. The compound-Gaussian clutter can be written as the product of two mutually independent random processes, where the fast fluctuating ‘speckle’ component is a zero-mean complex Gaussian process and the comparatively slow varying ‘texture’ component is a nonnegative real random process that describes the underlying mean power level of the resultant clutter process [41,42]. The most common clutter models, such as the Weibull and K-distribution, are compatible with the compound-Gaussian model. In the simulation, the clutter is assumed as a Weibull distribution with scale parameter $\alpha =1$ and shape parameter $\beta =3$.

The performance of the three detectors are presented in Figure 7. In addition, 100,000 Monte Carlo runs are performed in the simulation. The probabilities of false alarm $P_F$ are assumed to be ${10}^{-3},{10}^{-4}$, respectively. For the underlying detectors, the detection thresholds are not only related to the background clutter power but are also associated with the threshold multiplier, and a closed form expression of the threshold multiplier does not usually exist. Moreover, for Weibull distributed clutter, the threshold multiplier of the underlying detector depends on the desired probability of false alarm $P_F$, the number of bins averaged M, as well as the shape parameter β [43]. Consequently, it is difficult to determine the detection threshold analytically. In the simulation, the detection threshold is obtained via Monte Carlo runs. The test statistic in the absence of target is calculated via 100,000 Monte Carlo runs. As the effects of the clutter power, the number of bins averaged M and the shape parameter β are included in the test statistic; then, the detection threshold is determined according to the given probability of false alarm $P_F$. The result illustrates that both the two geometry based detectors outperform the classical CA-CFAR detector. Moreover, the performance of the Kullback–Leibler divergence-based matrix CFAR detector is better than the Riemannian distance-based matrix CFAR detector with 4–6 dB signal-to-clutter ratio (SCR) improvement.

Figure 8 illustrates a geometrical interpretation of the classical and geometry-based detection procedures. The classical CA-CFAR detector performs in the Euclidean space, where the Euclidean distance measure is used to define the divergence between two elements. However, the geometry based detector performs in the statistical manifold, in which the intrinsic divergence of the covariance matrices can be properly exploited and utilized, thus improving the detection performance.

Different distance measures adopted in detectors may result in different performance of detection. The Riemannian distance and Kullback–Leibler divergence are two measures used to evaluate the divergence between the data under test and clutter. The quality of the estimated Riemannian mean and Kullback–Leibler mean of SPD matrices are analyzed as follows. As the accuracy of the estimated mean matrix is a function of the number of samples (SPD matrices) used for averaging [40], when the number of samples M tends to infinity, the true mean matrix is obtained. In the simulation, 1000 samples are used to approximately estimate the true mean matrix. Then, for the case of a few number of samples available in detection, the distances between the estimated mean of few samples and the true mean matrix in terms of the Riemannian distance as well as Kullback–Leibler divergence are calculated via 300 Monte Carlo runs. The average of the 300 distances is presented as a function of the number of samples in Figure 9. It represents the distance (divergence) between the estimated mean matrix and the true mean matrix.

It can be observed that both of the Riemannian mean and Kullback–Leibler mean asymptotically approach the true mean matrix as the number of samples for averaging increases. It seems that the Kullback–Leibler mean is of smaller divergence to the true mean matrix than Riemannian mean in quantity. However, as Riemannian distance and Kullback–Leibler divergence are two different distance measures, the values of the two measures are incomparable. As discussed in the geometric interpretation of classical likelihood ratio test presented in Section 2, the likelihood ratio test is equivalent to the generalized minimum distance detector in the sense of Kullback–Leibler divergence, rather than Riemannian distance, which provides evidence that the KLD plays a central role in the theory of statistical inference. Therefore, Kullback–Leibler divergence is probably more appropriate to the case of hypothesis testing than Riemannian distance measure, which brings a performance improvement in detection.

6. Conclusions

In this paper, the problem of hypothesis testing in the Neyman–Pearson formulation is considered from a geometric viewpoint. In particular, we have described a concise geometric interpretation of deterministic and random signal detection in the philosophy of information geometry. In such a framework, both hypotheses and detectors can be regarded as geometrical objects on the statistical manifold of a parameterized family of probability distributions. Both the detector and detection performance are elucidated geometrically in terms of the Kullback–Leibler divergence. Compared to the likelihood ratio test, the geometric interpretation provides a consistent but more comprehensive means to understanding signal detection problems. Such methodology may play an important role in the development and understanding of the theoretical aspects of algorithms in radar signal processing.