The Geometry of Generalized Likelihood Ratio Test

Abstract

The generalized likelihood ratio test (GLRT) for composite hypothesis testing problems is studied from a geometric perspective. An information-geometrical interpretation of the GLRT is proposed based on the geometry of curved exponential families. Two geometric pictures of the GLRT are presented for the cases where unknown parameters are and are not the same under the null and alternative hypotheses, respectively. A demonstration of one-dimensional curved Gaussian distribution is introduced to elucidate the geometric realization of the GLRT. The asymptotic performance of the GLRT is discussed based on the proposed geometric representation of the GLRT. The study provides an alternative perspective for understanding the problems of statistical inference in the theoretical sense.

1. Introduction

The problem of hypothesis testing under statistical uncertainty arises naturally in many practical contexts. In these cases, the probability density functions (PDFs) under either or both hypotheses need not be completely specified, resulting in the inclusion of unknown parameters in the PDFs to express the statistical uncertainty in the model. The class of hypothesis testing problems with unknown parameters in the PDFs is commonly referred to as composite hypothesis testing [1]. The generalized likelihood ratio test (GLRT) is one of the most widely used approaches in composite hypothesis testing [2]. It involves estimating the unknown parameters via the maximum likelihood estimation (MLE) to implement a likelihood ratio test. In practice, the GLRT appears to be asymptotically optimal in the sense of the Neyman–Pearson criterion and usually gives satisfactory results [3]. As the GLRT combines both estimation and detection to deal with the composite hypothesis testing problem, its performance, in general, will depend on the statistical inference performance of these two aspects. However, in the literature, there is no general analytical result associated with the performance of the GLRT [1].

In recent years, the development of new theories in statistical inference has been characterized by the emerging trend of geometric approaches and their powerful capabilities, which allows one to analyze statistical problems in a unified perspective. It is important to link the GLRT to the geometrical nature of estimation and detection, which provides a new viewpoint on the GLRT. The general problem of composite hypothesis testing involves a decision between two hypotheses where the PDFs are themselves functions of unknown parameters. One approach to the understanding of performance limitations of statistical inference is via the theory of information geometry. In this context, the family of probability distributions with a natural geometrical structure is defined as a statistical manifold [4]. Information geometry studies the intrinsic properties of statistical manifolds which are endowed with a Riemannian metric and a family of affine connections derived from the log-likelihood functions of probability distributions [5]. It provides a way of analyzing the geometrical properties of statistical models by regarding them as geometric objects.

The geometric theory of statistics was firstly introduced in the 1940s by Rao [6], where the Fisher information matrix was regarded as a Riemannian metric on the manifold of probability distributions. Then, in 1972, a one-parameter family of affine connections was introduced by Chentsov in [7]. Meanwhile, Efron [8] defined the concept of statistical curvature and discussed its basic role in the high-order asymptotic theory of statistical inference. In 1982, Amari [5,9] developed a duality structure theory that unified all of these theories in a differential-geometric framework, leading to a large number of applications.

In the area of hypothesis testing, the geometric perspectives have acquired relevance in the analysis and development of new approaches to various testing and detection contexts. For example, Kass and Vos [10] provided a detailed introduction to the geometrical foundations of asymptotic inference of curved exponential families. Garderen [11] presented a global analysis of the effects of curvature on hypothesis testing. Dabak [12] induced a geometric structure on the manifold of probability distributions and enforced a detection theoretic specific geometry on it, while Westover [13] discussed the asymptotic limit in the problems of multiple hypothesis testing from the geometrical perspective. For the development of new approaches to hypothesis testing, Hoeffding [14] proposed an asymptotically optimal test for multinomial distributions in which the testing can be denoted in terms of the Kullback–Leibler divergence (KLD) between the empirical distribution of the measurements and the null hypothesis, where the alternate distribution is unrestricted. In the aspect of signal detection, Barbaresco et al. [15,16,17] studied the geometry of Bruhat–Tits complete metric space and upper-half Siegel space and introduced a matrix constant false alarm rate (CFAR) detector which improves the detection performance of the classical CFAR detection.

As more and more new analyses and new approaches have benefited from the geometric and information-theoretic perspectives of statistics, it appears to be important to clarify the geometry of existing problems that is promising to gain new ways to deal with the statistical problems. In this paper, a geometric interpretation of the GLRT is sought from the perspective of information geometry. Two pictures of the GLRT are presented for the cases where unknown parameters are and are not the same under each hypothesis, respectively. Under such an interpretation, both detection and estimation associated with the GLRT are regarded as geometric operations on the statistical manifold. As a general consideration, curved exponential families [9], which include a large number of the most common used distributions, are taken into account as the statistical model of hypothesis testing problems. A demonstration of one-dimensional curved Gaussian distribution is introduced to elucidate the geometric realization of the GLRT. The geometric structure of the curved exponential families developed by Efron [8] in 1975 and Amari [9] in 1982 provides a theoretical foundation for the analysis. The geometric formulation of the GLRT presented in this paper makes it possible for several advanced notions and conclusions in the information geometry theory to be transferred and applied to the performance analysis of the GLRT.

The main contributions of this paper are summarized as follows:

• A geometric interpretation of the GLRT is proposed based on the differential geometry of curved exponential families and duality structure theory developed by Amari [9]. Two geometric pictures of the GLRT are presented in the theoretical sense, which provides an alternative perspective for understanding the problems of statistical inference.
• The asymptotic performance of the GLRT is discussed based on the proposed geometric representation of the GLRT. The information loss when performingthe MLE using a finite number of samples is related to the flatness of the submanifolds determined by the GLRT model.

In the next section, alternative viewpoints on the likelihood ratio test and the maximum likelihood estimation are introduced from the perspective of information theory. The equivalences between the Kullback–Leibler divergence, likelihood ratio test, and the MLE are highlighted. The principles of information geometry are briefly introduced in Section 3. In Section 4, the geometric interpretation of the GLRT is presented in consideration of the geometry of curved exponential families. We present an example of the GLRT where a curved Gaussian distribution with one unknown parameter is involved, and a further discussion on the geometry of the GLRT. Finally, conclusions are obtained in Section 5.

2. Information-Theoretic Viewpoints on Likelihood Ratio Test and Maximum Likelihood Estimation

In statistics, the likelihood ratio test and maximum likelihood estimation are two fundamental concepts related to the GLRT. The likelihood ratio test is a very general form of testing model assumptions, while the maximum likelihood estimation is one of the most common approaches to parameter estimation. Both of them are associated with the Kullback–Leibler divergence [18], which is equivalent to the relative entropy [19] in information theory.

For a sequence of observations $\mathit{x}={(x_1,x_2,\dots ,x_N)}^T\in {\mathbb{R}}^N$ which is independently and identically distributed (i.i.d.), the binary hypothesis testing problem is used to decide whether this sequence $\mathit{x}$ originates from the null hypothesis ${\mathcal{H}}_0$ or the alternative hypothesis ${\mathcal{H}}_1$ with probability distributions $p_0\left(\mathit{x}\right)$ and $p_1\left(\mathit{x}\right)$, respectively. The likelihood ratio is given by

$$L=\frac{p_1\left(\mathit{x}\right)}{p_0\left(\mathit{x}\right)}=\prod _{i=1}^N\frac{p_1\left(x_i\right)}{p_0\left(x_i\right)}$$

(1)

Assume $q\left(x\right)$ is the empirical distribution (frequency histogram acquired via Monte Carlo tests) of observed data. For large N, in accordance with the strong law of large numbers [20], the log likelihood ratio test in the Neyman–Pearson formulation $lnL\underset{{\mathcal{H}}_0}{\stackrel{{\mathcal{H}}_1}{\gtrless }}\gamma$ is equivalent to

$$D(q\parallel p_0)-D(q\parallel p_1)\underset{{\mathcal{H}}_0}{\stackrel{{\mathcal{H}}_1}{\gtrless }}\frac{1}{N}\gamma \triangleq {\gamma }^{\prime }$$

(2)

where $\underset{{\mathcal{H}}_0}{\stackrel{{\mathcal{H}}_1}{\gtrless }}$ denotes that the test is to decide ${\mathcal{H}}_1$ if “>” is satisfied, or to decide ${\mathcal{H}}_0$, and vice versa. The quantity

$$D(q\parallel p)={\int }_{{\mathbb{R}}^N}q\left(x\right)ln\frac{q\left(x\right)}{p\left(x\right)}dx$$

(3)

is the KLD from $q\left(x\right)$ to $p\left(x\right)$. Note that x is dropped from the notion D for simplifying the KLD expression without confusion.

Equation (2) indicates that the likelihood ratio test is equivalent to choosing the hypothesis that is “closer” to the empirical distribution in the sense of the KLD. The test can be referred to as a generalized minimum dissimilarity detector in a geometric viewpoint.

Now, consider another, slightly different problem where the observations $\mathit{x}$ are from a statistical model represented by $p\left(\mathit{x}\right|\mathbf{\theta })$ with unknown parameters $\mathbf{\theta }$. The problem is to estimate the unknown parameters $\mathbf{\theta }$ based on observations $\mathit{x}$. The likelihood function for the underlying estimation problem is

$$p\left(\mathit{x}\right|\mathbf{\theta })=\prod _{i=1}^{N}p\left(x_i\right|\mathbf{\theta })$$

(4)

In a similar way, for large N, maximizing the likelihood (4) to find the maximum likelihood estimate of $\mathbf{\theta }$ is equivalent to finding $\mathbf{\theta }$, which minimizes the KLD $D(q\parallel p_{\mathbf{\theta }})$, i.e.,

$$\widehat{\mathbf{\theta }}=arg\underset{\mathbf{\theta }}{min}D(q\parallel p_{\mathbf{\theta }})$$

(5)

where $p_{\mathbf{\theta }}$ is used as a surrogate for $p\left(\mathit{x}\right|\mathbf{\theta })$.

The above results provide an information-theoretic view to the problem of hypothesis testing and maximum likelihood estimation in statistics. From the perspective of information difference, these results have profound geometric meanings and can be geometrically analyzed and viewed in the framework of information geometry theory, from which additional insights into the analysis of these statistical problems, as well as their geometric interpretations, are obtained.

3. Principles of Information Geometry

3.1. Statistical Manifold

Information geometry studies the natural geometric structure of the parameterized family of probability distributions $S=\left\{p\right(\mathit{x}\left|\mathbf{\theta }\right)\}$ specifying by a parameter vector $\mathbf{\theta }:=[{\theta }_1,\dots ,{\theta }_n]$, in which $\mathit{x}$ is the samples of a random variable $\mathit{X}$. When the probability measure on the sample space is continuous and differentiable and the mapping $\mathbf{\theta }\mapsto p\left(\mathit{x}\right|\mathbf{\theta })$ is injective [5], the family S is considered as a statistical manifold with $\mathbf{\theta }$ as its coordinate system [4].

Figure 1 demonstrates the diagram of a statistical manifold. For a given parameter vector $\mathbf{\theta }\in \mathsf{\Theta }\subset {\mathbb{R}}^n$, the measurement $\mathit{x}$ in the sample space $\mathit{X}$ is an instantiation of a probability distribution $p\left(\mathit{x}\right|\mathbf{\theta })$. Each $p\left(\mathit{x}\right|\mathbf{\theta })$ in the family of distributions is specified by a point $s\left(\mathbf{\theta }\right)$ on the manifold S. The n-dimensional statistical manifold is composed of the parameterized family of probability distributions $S=\left\{p\right(\mathit{x}\left|\mathbf{\theta }\right)\}$ with $\mathbf{\theta }$ as a coordinate system of S.

Various families of probability distributions correspond to specific structures of the statistical manifold. Information geometry takes the statistical properties of samples as the geometric structure of a statistical manifold, and utilizes differential geometry methods to measure the variation of information contained in the samples.

3.2. Fisher Information Metric and Affine Connections

The metric and connections associated with a manifold are two important concepts in information geometry. For a statistical manifold consisting of a parameterized family of probability distributions, the Fisher information matrix (FIM) is usually adopted as a Riemannian metric tensor of the manifold [6], which is defined by the inner product between tangent vectors at a point on the manifold. It is denoted by $\mathit{G}\left(\mathbf{\theta }\right)=\left[g_{ij}\left(\mathbf{\theta }\right)\right]$, where

$$g_{ij}\left(\mathbf{\theta }\right)=E\left\{\frac{\partial logp\left(\mathit{x}\right|\mathbf{\theta })}{\partial {\theta }_i}·\frac{\partial logp\left(\mathit{x}\right|\mathbf{\theta })}{\partial {\theta }_j}\right\}$$

(6)

where $\{\partial log(·)/\partial {\theta }_i\}$ is considered as a basis for the vector space of random variable $\mathit{X}$. The tangent space of S at $\mathbf{\theta }$, denoted as ${\mathcal{T}}_{\mathbf{\theta }}\left(S\right)$, is identified as the vector space. Based on the above definition, the FIM metric determines how the information distance is measured on the statistical manifold.

When considering the relationships between two tangent spaces ${\mathcal{T}}_{\mathbf{\theta }}$ and ${\mathcal{T}}_{\mathbf{\theta }+d\mathbf{\theta }}$ at two neighboring points $\mathbf{\theta }$ and $\mathbf{\theta }+d\mathbf{\theta }$ (d is the differential operator), an affine connection is defined by which the two tangent spaces become feasible for comparison. When the connection coefficients are all identically 0, then S is flat manifold that “locally looks like” a Euclidean space with zero curvatures everywhere. The most commonly used connection is called $\alpha$-connections [9],

$${\stackrel{\alpha }{\mathsf{\Gamma }}}_{jim}\left(\mathbf{\theta }\right)=E_{\mathbf{\theta }}\left\{{\partial }_j{\partial }_{i}l(\mathit{x},\mathbf{\theta }){\partial }_{m}l(\mathit{x},\mathbf{\theta })\right\}+\frac{1-\alpha }{2}{E}_{\mathbf{\theta }}\left\{{\partial }_{j}l(\mathit{x},\mathbf{\theta }){\partial }_{i}l(\mathit{x},\mathbf{\theta }){\partial }_{m}l(\mathit{x},\mathbf{\theta })\right\}$$

(7)

where ${\stackrel{\alpha }{\mathsf{\Gamma }}}_{jim}$ denotes the connection coefficients with $i,j,m=1,\dots ,n$, and

$${\partial }_i=\frac{\partial }{\partial {\theta }_i}\mathrm{and}l(\mathbf{\theta },\mathit{x})=logp\left(\mathit{x}\right|\mathbf{\theta }).$$

(8)

In (7), $\alpha =0$ corresponds to the Levi–Civita connection, while $\alpha =1$ defines the e-connection and $\alpha =-1$ defines the m-connection. Under the e-connection and m-connection, an exponential family with natural parameter $\mathbf{\theta }$ coordinate and a mixture family with expectation parameter $\mathbf{\eta }$ coordinate are both flat manifolds [9]. Statistical inference with respect to the exponential family greatly benefits from the geometric properties of the flat manifold. By using the methods of differential geometry, many additional insights into the intrinsic structure of probability distributions can be obtained, which opens a new perspective on the analysis of statistical problems. In the next section, a geometric interpretation of the GLRT and further discussions are sought based on the principles of information geometry.

4. Geometry of the Generalized Likelihood Ratio Test

As a general treatment, the curved exponential families, which encapsulate many important distributions for real-world problems, are considered as the statistical model for the hypothesis testing problems discussed in this paper. In this section, the MLE solution to parameter estimation for curved exponential families is derived. We then present two pictures of the GLRT, which are sketched based on the geometric structure of the curved exponential families developed by Efron [8] in 1975 and Amari [9] in 1982, to illustrate the information geometry of the GLRT. An example of the GLRT for a curved Gaussian distribution with a single unknown parameter is given, which is followed by a further discussion on the geometric formulation of the GLRT.

4.1. The MLE Solution to Statistical Estimation for Curved Exponential Families

Exponential families contain lots of the most commonly used distributions, including the normal, exponential, Gamma, Beta, Poisson, Bernoulli, and so on [21]. The curved exponential families are the distributions whose natural parameters are nonlinear functions of “local” parameters. The canonical form of a curved exponential family [9] is expressed as

$$p\left(\mathit{x}\right|\mathit{u})=exp\left\{C\left(\mathit{x}\right)+{\mathbf{\theta }}^T\left(\mathit{u}\right)\mathit{F}\left(\mathit{x}\right)-\phi \left(\mathbf{\theta }\left(\mathit{u}\right)\right)\right\}=p\left(\mathit{x}\left|\mathbf{\theta }\right(\mathit{u})\right)$$

(9)

where $\mathit{x}\in X$ is a vector of samples, $\mathbf{\theta }:=[{\theta }_1,\dots ,{\theta }_n]$ are the natural parameters, $\mathit{u}\in {\mathbb{R}}^m(m<n)$ are local parameters standing for the parameters of interest to be estimated, which is specified by (9), while $\mathit{F}\left(\mathit{x}\right):={[F_1\left(\mathit{x}\right),\cdots ,F_n\left(\mathit{x}\right)]}^T$ denote sufficient statistics with respect to $\mathbf{\theta }=({\theta }_1,\cdots ,{\theta }_n)$, which take values from the sample space X. $\phi$ is the potential function of the curved exponential family and it is found from the normalization condition ${\int }_{X}p\left(\mathit{x}\right|\mathbf{\theta })d\mathit{x}=1$, i.e.,

$$\phi \left(\mathbf{\theta }\right)=log{\int }_{X}exp\left\{C\left(\mathit{x}\right)+\sum _i^n\left({\theta }_i{F}_i\left(\mathit{x}\right)\right)\right\}d\mathit{x}$$

(10)

The term “curved” is due to the fact that the curved exponential family in (9) is a submanifold of the canonical exponential family $p\left(\mathit{x}\right|\mathbf{\theta })$ by the embedding $\mathit{u}⟶\mathbf{\theta }\left(\mathit{u}\right)$.

Let $l(\mathbf{\theta },\mathit{x})=logp\left(\mathit{x}\right|\mathbf{\theta })$ be the log-likelihood and ${\nabla }_{\mathit{u}}\mathbf{\theta }$ be the Jacobian matrix of the natural parameter $\mathbf{\theta }$. According to (9),

$$\begin{array}{cc}\hfill \nabla l(\mathbf{\theta },\mathit{x})& =\nabla \left\{{\mathbf{\theta }}^T\left(\mathit{u}\right)\mathit{F}\left(\mathit{x}\right)-\phi \left(\mathbf{\theta }\left(\mathit{u}\right)\right)\right\}\hfill \\ \hfill & =\nabla {\mathbf{\theta }}^T\left(\mathit{u}\right)\left[\mathit{F}\left(\mathit{x}\right)-\mathbf{\eta }\left(\mathit{u}\right)\right]\hfill \end{array}$$

(11)

where $\mathbf{\eta }\left(\mathit{u}\right)$ is the expectation of the sufficient statistics $\mathit{F}\left(\mathit{x}\right)$, i.e.,

$$\mathbf{\eta }\left(\mathit{u}\right):=E_{p(·,\mathit{u})}\left\{\mathit{F}\left(\mathit{x}\right)\right\}$$

(12)

and is called the expectation parameter, which defines a distribution of mixture family [4]. The natural parameter $\mathbf{\theta }\left(\mathit{u}\right)$ and expectation parameter $\mathbf{\eta }\left(\mathit{u}\right)$ are connected by the Legendre transformation [9], as

$$\mathbf{\eta }={\nabla }_{\mathbf{\theta }}\phi \left(\mathbf{\theta }\right),\mathbf{\theta }={\nabla }_{\mathbf{\eta }}\varphi \left(\mathbf{\eta }\right)$$

(13)

where $\varphi \left(\mathbf{\eta }\right)$ is defined by

$$\varphi \left(\mathbf{\eta }\right)=\underset{\mathbf{\theta }}{max}\{{\mathbf{\eta }}^T\mathbf{\theta }-\phi \left(\mathbf{\theta }\right)\}$$

(14)

Therefore,

$${\nabla }_{\mathit{u}}\phi \left(\mathit{u}\right)={\nabla }_{\mathit{u}}{\mathbf{\theta }}^T\left(\mathit{u}\right){\nabla }_{\mathbf{\theta }}\phi \left(\mathbf{\theta }\right)={\nabla }_{\mathit{u}}{\mathbf{\theta }}^T\left(\mathit{u}\right)\mathbf{\eta }\left(\mathit{u}\right)$$

(15)

Thus, the maximum likelihood estimator $\widehat{\mathit{u}}$ of the local parameter in (9) can be obtained by the following likelihood equation:

$${\nabla }_{\mathit{u}}l\left(\widehat{\mathit{u}}\right)={\nabla }_{\mathit{u}}logp\left(\mathit{x}\right|\mathit{u})={\nabla }_{\mathit{u}}{\mathbf{\theta }}^T\left(\widehat{\mathit{u}}\right)\left[\mathit{F}\left(\mathit{x}\right)-\mathbf{\eta }\left(\widehat{\mathit{u}}\right)\right]=0$$

(16)

Equation (16) indicates that the solution to the MLE can be found by mapping the data $\mathit{F}\left(\mathit{x}\right)$ onto ${\mathcal{F}}_{\mathcal{B}}:=\{\mathbf{\eta }\left(\mathit{u}\right):\mathit{u}\in {\mathbb{R}}^m\}$ orthogonally to the tangent of ${\mathcal{F}}_{\mathcal{A}}:=\{\mathbf{\theta }\left(\mathit{u}\right):\mathit{u}\in {\mathbb{R}}^m\}$. As $\mathbf{\theta }\left(\mathit{u}\right)$ and $\mathbf{\eta }\left(\mathit{u}\right)$ live in two different spaces ${\mathcal{F}}_{\mathcal{A}}$ and ${\mathcal{F}}_{\mathcal{B}}$, the inner product between dual spaces is defined as ${\langle \mathbf{\theta }\left(\mathit{u}\right),\mathbf{\eta }\left(\mathit{u}\right)\rangle }_{\mathsf{\Gamma }}:=\mathbf{\theta }{\left(\mathit{u}\right)}^T·\mathsf{\Gamma }·\mathbf{\eta }\left(\mathit{u}\right)$ with a metric $\mathsf{\Gamma }$. For the flat manifold, the identity matrix serves as the metric $\mathsf{\Gamma }$. By analogy with the MLE for the universal distribution given by (5), Hoeffding [14] presented another interpretation for the MLE of the curved exponential family. In the interpretation, $\mathbf{\eta }\left(\widehat{\mathit{u}}\right)$ represents a point in ${\mathcal{F}}_{\mathcal{B}}$ which is located closest to the data point in the sense of the Kullback–Leibler divergence, i.e.,

$$\widehat{\mathit{u}}=arg\underset{\mathbf{\eta }\left(\mathit{u}\right)\in {\mathcal{F}}_{\mathcal{B}}}{min}D\left[\mathit{F}\left(\mathit{x}\right)\parallel {\mathbf{\eta }}_{\mathit{u}}\right]$$

(17)

where $D\left[\mathit{F}\left(\mathit{x}\right)\parallel {\mathbf{\eta }}_{\mathit{u}}\right]$ denotes the Kullback–Leibler divergence from the multivariate joint distributions of $\mathit{F}\left(\mathit{x}\right)$ to ${\mathbf{\eta }}_{\mathit{u}}$.

Based on the above analysis, there are two important spaces related to a curved exponential family. One is called the natural parameter space, denoted by $\left\{\mathbf{\theta }\right\}\subset {\mathcal{A}}^n$, which denotes the enveloping space including all the distributions of exponential families, and the other is called the expectation parameter space, denoted by $\left\{\mathbf{\eta }\right\}\subset {\mathcal{B}}^n$, denoting the dual space of ${\mathcal{A}}^n$. The two spaces are “dual” with each other and flat under the e-connection and m-connection, respectively. The curved exponential family (9) is regarded as submanifolds embedded in the two spaces, and the data can also be immersed in these spaces in the form of sufficient statistics $\mathit{F}\left(\mathit{x}\right)$. Consequently, the estimators, such as the MLE given by (16), associated with the curved exponential families can be geometrically performed in the two spaces.

4.2. Geometric Demonstration of the Generalized Likelihood Ratio Test

As mentioned earlier, the GLRT is one of the most widely used approaches in composite hypothesis testing problems with unknown parameters in the PDFs. The data $\mathit{x}$ have the PDF $p\left(\mathit{x}\right|{\mathit{u}}_0;{\mathcal{H}}_0)$ under hypothesis ${\mathcal{H}}_0$ and $p\left(\mathit{x}\right|{\mathit{u}}_1;{\mathcal{H}}_1)$ under hypothesis ${\mathcal{H}}_1$, where ${\mathit{u}}_0$ and ${\mathit{u}}_1$ are unknown parameters under each hypothesis. The GLRT enables a decision by means of replacement of the unknown parameters by their maximum likelihood estimates (MLEs) to implement a likelihood ratio test. The GLRT decides ${\mathcal{H}}_1$ if

$$L_G\left(\mathit{x}\right)=\frac{p\left(\mathit{x}\right|{\widehat{\mathit{u}}}_1;{\mathcal{H}}_1)}{p\left(\mathit{x}\right|{\widehat{\mathit{u}}}_0;{\mathcal{H}}_0)}>\gamma$$

(18)

where ${\widehat{\mathit{u}}}_i$ is the MLE of ${\mathit{u}}_i$ (by maximizing $p\left(\mathit{x}\right|{\mathit{u}}_i;{\mathcal{H}}_i)$).

From the perspective of information geometry, the probability distribution $p\left(\mathit{x}\right|{\mathit{u}}_i;{\mathcal{H}}_i)$ is an element of the parameterized family of PDFs $S=\left\{p\right(\mathit{x}\left|\mathit{u}\right),\mathit{u}\in \mathsf{\Omega }\}$, where $\mathsf{\Omega }\subset {\mathbb{R}}^m$ is the parameter set. For the curved exponential family $S=\left\{p\right(\mathit{x}\left|\mathit{u}\right)\}$, it can be regarded as a submanifold embedding in the natural parameter space $\left\{\mathbf{\theta }\right\}\subset {\mathcal{A}}^n$, which includes all the distributions of exponential families. The curved exponential family S can be represented by a curve (or surface) $\{\mathbf{\theta }=\mathbf{\theta }(\mathit{u}\left)\right\}$ embedded in the enveloping space ${\mathcal{A}}^n$ by the nonlinear mapping $\mathit{u}⟶\mathbf{\theta }\left(\mathit{u}\right)$. The expectation parameter space $\left\{\mathbf{\eta }\right\}\subset {\mathcal{B}}^n$ of S is a dual flat space to the natural parameter space $\left\{\mathbf{\theta }\right\}$, while the “realizations” of sufficient statistics $\mathit{F}\left(\mathit{x}\right)$ of the distribution $p\left(\mathit{x}\right|\mathbf{\theta })$ can be immersed in this space. Consequently, the MLE is performed in the space ${\mathcal{B}}^n$ by mapping the samples $\mathit{F}\left(\mathit{x}\right)$ onto the submanifold specified by $\{\mathbf{\eta }=\mathbf{\eta }(\mathit{u}\left)\right\}$ under the m-projection.

As the parameters ${\mathit{u}}_0$ and ${\mathit{u}}_1$, as well as their dimensionalities, may or may not be the same under the null and alternative hypotheses, two pictures of the GLRT are presented for the two cases: one is with the same unknown parameters under each hypotheses and the other is with different parameters or different dimensionalities. The picture for the first case is illustrated in Figure 2a. In this case, distributions under two hypotheses share the same form and the same unknown parameter $\mathit{u}$. However, the parameter takes different value sets under different hypotheses. The family of $S=\left\{p\right(\mathit{x}\left|\mathit{u}\right)\}$ can be smoothly embedded as a surface ${\mathcal{F}}_{\mathcal{B}}$ specified by $\{\mathbf{\eta }\left(\mathit{u}\right):\mathit{u}\in {\mathbb{R}}^m\}$ in the space ${\mathcal{B}}^n$. The hypotheses $p\left(\mathit{x}\right|{\mathit{u}}_i;{\mathcal{H}}_i)$ with unknown ${\mathit{u}}_i$ define two “uncertainty volumes” ${\mathsf{\Omega }}_0$ and ${\mathsf{\Omega }}_1$ on the submanifold ${\mathcal{F}}_{\mathcal{B}}$. These volumes are the collections of probability distributions specified by the value sets of the unknown parameter ${\mathit{u}}_i$. The measurements $\mathit{x}$ are immersed in ${\mathcal{B}}^n$ in the form of sufficient statistics $\mathit{F}\left(\mathit{x}\right)$. Consequently, the MLE can be found by “mapping” the samples $\mathit{F}\left(\mathit{x}\right)$ onto the uncertainty volumes ${\mathsf{\Omega }}_0$ and ${\mathsf{\Omega }}_1$ on ${\mathcal{F}}_{\mathcal{B}}$. The points $p_0$ and $p_1$ in Figure 2 are the corresponding projections, i.e., the MLEs of the unknown parameter under two hypotheses. As indicated in (17), the MLEs can also be obtained by finding the points on ${\mathsf{\Omega }}_0$ and ${\mathsf{\Omega }}_1$ which are located closest to the data point in the sense of KLD, i.e.,

$${\widehat{\mathit{u}}}_0=arg\underset{\mathbf{\eta }\left(\mathit{u}\right)\in {\mathsf{\Omega }}_0}{inf}D\left[\mathit{F}\left(\mathit{x}\right)\parallel {\mathbf{\eta }}_{\mathit{u}}\right],{\widehat{\mathit{u}}}_1=arg\underset{\mathbf{\eta }\left(\mathit{u}\right)\in {\mathsf{\Omega }}_1}{inf}D\left[\mathit{F}\left(\mathit{x}\right)\parallel {\mathbf{\eta }}_{\mathit{u}}\right]$$

(19)

and the corresponding minimum KLDs can be represented by

$$D_0=D\left[\mathit{F}\left(\mathit{x}\right)\parallel p_0\right],D_1=D\left[\mathit{F}\left(\mathit{x}\right)\parallel p_1\right],$$

(20)

respectively.

It should be emphasized that the above “mapping” is a general concept. When the parameters to be estimated are not restricted by a given “value set”, the MLE is simply obtained by maximizing the likelihood and the projections will fall onto the submanifold ${\mathcal{F}}_{\mathcal{B}}$. However, if the parameters to be estimated are restricted in a given “value set”, the MLE should be operated by maximizing the likelihood with respect to the given parameter space. In the case where the projections fall outside the “uncertainty volumes”, the MLE solutions are given by those points which are closest to the data point described by (19).

Let $\mathcal{R}({\mathbf{\eta }}_0,\rho )$ be a divergence sphere centered at ${\mathbf{\eta }}_0$ with radius $\rho$; that is, the submanifold of the enveloping space ${\mathcal{B}}^n$ consisting of points $\mathbf{\eta }$ for which the KLD $D({\mathbf{\eta }}_0\parallel \mathbf{\eta })$ is equal to $\rho$. Denote this divergence sphere by

$$\mathcal{R}({\mathbf{\eta }}_0,\rho )=\{\mathbf{\eta }\in {\mathcal{B}}^n|D({\mathbf{\eta }}_0\parallel \mathbf{\eta })=\rho \}$$

(21)

Then, the closest point in (19) may be more easily found via the divergence sphere with center $\mathit{F}\left(\mathit{x}\right)$ and radius $D_i$ tangent to ${\mathsf{\Omega }}_i$ at $p_i$, as illustrated in Figure 3. Consequently, according to (2), the GLRT can be geometrically performed by comparing the difference between the minimum KLDs $D_0$ and $D_1$ with a threshold ${\gamma }^{\prime }$, i.e.,

$$D\left[\mathit{F}\left(\mathit{x}\right)\parallel p_0\right]-D\left[\mathit{F}\left(\mathit{x}\right)\parallel p_1\right]\underset{{\mathcal{H}}_0}{\stackrel{{\mathcal{H}}_1}{\gtrless }}{\gamma }^{\prime }$$

(22)

In practice, the Neyman–Pearson criterion is commonly employed to determine the threshold ${\gamma }^{\prime }$ in (22) and the detector is of maximum probability of detection $P_D$ under a given probability of false alarm $P_F$. As a commonly used performance index, the missingprobability $P_M$ usually decays exponentially as the sample size increases. The rate of exponential decay can be represented by [22], as

$$K\triangleq \underset{n\to \infty }{lim}-\frac{1}{N}log{P}_M$$

(23)

Based on Stein’s lemma, for a constant false-alarm constraint, the best error exponent is related to the Kullback–Leibler divergence $D(p_0\parallel p_1)$ from $p_0$ to $p_1$ [23], i.e.,

$$K=D(p_0\parallel p_1)$$

(24)

and

$$P_M\doteq 2^{-NK}$$

(25)

where ≐ denotes the first-order equivalence in the exponent. For example,

$$a_n\doteq b_n\mathrm{means}\underset{n\to \infty }{lim}\frac{1}{n}log\frac{a_n}{b_n}=0.$$

(26)

In the above sense, the KLD from $p_0$ to $p_1$ is equivalent to the signal-to-noise ratio (SNR) of the underlying detection problem. Therefore, information geometry offers an insightful geometrical explanation for the detection performance of a Neyman–Pearson detector.

In the second case, the dimensionality of the unknown parameters ${\mathit{u}}_0$ and ${\mathit{u}}_1$ is different, while the dimensionality of the enveloping spaces is common for both hypotheses due to the same measurements $\mathit{x}$. However, the hypotheses may correspond to two separated submanifolds, ${\mathsf{\Omega }}_0$ and ${\mathsf{\Omega }}_1$, embedded in ${\mathcal{B}}^n$ caused by the different dimensionality between the unknown parameters. As illustrated in Figure 2b, a surface and a curve are used to denote the submanifolds ${\mathsf{\Omega }}_0$ and ${\mathsf{\Omega }}_1$, corresponding to the two hypotheses, respectively. Similar to the first case, the GLRT with different unknown parameters may also be geometrically interpreted.

4.3. A Demonstration of One-Dimensional Curved Gaussian Distribution

Consider the following detection problem:

$$\left\{\begin{array}{c}{\mathcal{H}}_0:u<0\hfill \\ {\mathcal{H}}_1:u>0\hfill \end{array}\right.$$

(27)

The measurement originates from a curved Gaussian distribution

$$x\sim \mathcal{N}(u,u^2{a}^2)$$

(28)

where a is a positive constant and u is an unknown parameter.

The probability density function of the measurement is

$$p\left(x\right|u)=\frac{1}{\sqrt{2\pi }a\left|u\right|}exp\left[-\frac{{(x-u)}^2}{2a^2{u}^2}\right]$$

(29)

By reparameterization, the probability density function can be represented in the general form of a curved exponential family as

$$\begin{array}{cc}\hfill p\left(x\right|u)& =exp\left\{\frac{1}{a^{2}u}x-\frac{1}{2a^2{u}^2}{x}^2-\frac{1}{2a^2}-ln\left(a\left|u\right|\right)-\frac{1}{2}ln\left(2\pi \right)\right\}\hfill \\ \hfill & =exp\left\{lna+{\theta }_{1}x+{\theta }_2{x}^2-\left[-\frac{{\theta }_1^2}{4{\theta }_2}-\frac{1}{2}ln(-{\theta }_2)+\frac{1}{2}ln\pi \right]\right\}\hfill \\ \hfill & =exp\left\{C\left(x\right)+F_1\left(x\right){\theta }_1+F_2\left(x\right){\theta }_2-\phi ({\theta }_1,{\theta }_2)\right\}\hfill \end{array}$$

(30)

where $C\left(x\right)=lna$ and the potential function $\phi$ is

$$\phi ({\theta }_1,{\theta }_2)=-\frac{{\theta }_1^2}{4{\theta }_2}-\frac{1}{2}ln(-{\theta }_2)+\frac{1}{2}ln\pi$$

(31)

The above distributions with local parameter u correspond to a one-dimensional curved exponential family embedded in the natural parameter space $\mathcal{A}$. The natural coordinates are

$${\theta }_1=\frac{1}{a^{2}u},{\theta }_2=-\frac{1}{2a^2{u}^2}$$

(32)

which defines a parabola (denoted by ${\mathcal{F}}_{\mathcal{A}}$)

$${\theta }_2=-\frac{a^2}{2}{\theta }_1^2$$

(33)

in $\mathcal{A}$. The underlying distribution (28) can also be represented in the expectation parameter space $\mathcal{B}$ with expectation coordinates

$${\eta }_1=\frac{\partial \phi }{\partial {\theta }_1}=u,{\eta }_2=\frac{\partial \phi }{\partial {\theta }_2}=(a^2+1)u^2$$

(34)

which also defines a parabola (denoted by ${\mathcal{F}}_{\mathcal{B}}$)

$${\eta }_2=(a^2+1){\eta }_1^2$$

(35)

in $\mathcal{B}$.

The sufficient statistics $\mathit{F}\left(x\right)$ obtained from samples x can be represented by

$$\mathit{F}\left(x\right)={[x,x^2]}^T$$

(36)

Figure 4 shows the expectation parameter space and illustrates geometric interpretation of the underlying GLRT, where the blue parabola in the figure denotes embedding of the curved Gaussian distribution with parameter u. The submanifolds associated with two hypotheses can be geometrically represented by the blue parabolas (specified by ${\eta }_1<0$ and ${\eta }_1>0$, respectively). Without loss of generality, assume that $a=1,u=2$. The blue dots signify $N=100$ observations (measurements) in the expectation parameter space with the coordinates $(x,x^2)$. The statistical mean of the measurements are used to calculate the sufficient statistics $\mathit{F}\left(x\right)$ which are denoted by a red asterisk. The MLEs of parameter u under two hypotheses are obtained by finding the points on the two submanifolds which are closest to the data point in the sense of KLD. According to (22), the GLRT can be geometrically performed by comparing the difference between the minimum KLDs $D_0$ and $D_1$ with a threshold ${\gamma }^{\prime }$.

4.4. Discussions

The geometric formulation of the GLRT presented above provides additional insights into the GLRT. To the best of our knowledge, there is no general analytical result associated with the performance of the GLRT in the literature [1]. The asymptotic analysis is only valid under the conditions that (1) the data sample size is large; and (2) the MLE asymptotically attains the Cramér-Rao lower bound (CRLB) of the underlying estimation problems.

It is known that the MLE with natural parameters is a sufficient statistic for an exponential family, and achieves the CRLB if a suitable measurement function is chosen for the estimation [8]. For the curved exponential families the MLE is not, in general, an efficient estimator, which means that the variance of MLE may not achieve CRLB with a finite number of samples. This indicates that when using a finite number of samples there will be a deterioration in performance for both MLE and GLRT when the underlying statistical model is a curved exponential family. There will be an inherent information loss (compared with the Fisher information) when implementingan estimation process if the statistical model is of nonlinearity. Roughly speaking, if the embedded submanifold ${\mathcal{F}}_{\mathcal{B}}$ in Figure 2a and ${\mathsf{\Omega }}_0$, ${\mathsf{\Omega }}_1$ in Figure 2b are curved, the MLEs will not achieve the CRLB due to the inherent information loss caused by the non-flatness of the statistical model. The information loss may be quantitatively calculated using the e-curvature of the statistical model [9].

Consequently, if the statistical model associated with a GLRT is not flat, i.e., the submanifolds shown in Figure 2 are curved, there will be a deterioration in performance for the GLRT using a finite number of samples. As sample size N increases, the sufficient statistics $\mathit{F}\left(\mathit{x}\right)$ will be better matched to the statistical model and thus closer to the submanifolds (see Figure 2), and the divergence from data to the submanifold associated with the true hypothesis ${\mathcal{H}}_i$ will be shorter. Asymptotically, as $N\to \infty$, the sufficient statistics will fall onto the submanifold associated with the true hypothesis ${\mathcal{H}}_i$, so that the corresponding divergence $D_i$ reduces to zero. By then, the GLRT achieves a perfect performance.

5. Conclusions

In this paper, the generalized likelihood ratio test is addressed from a geometric viewpoint. Two pictures of the GLRT are presented in the philosophy of the information geometry theory. Both the detection and estimation associated with a GLRT are regarded as geometric operations on the manifolds of a parameterized family of probability distributions. As demonstrated in this work, the geometric interpretation of GLRT provides additional insights in the analysis of GLRT.

Potentially, more constructive analysis can be generalized based on the information geometry of GLRT. For example, the error exponent defined by (24) and (25) provides a useful performance index for the detection process associated with GLRT. When $p_0$ and $p_1$ in (24) are the estimates of an MLE (rather than the true values) of unknown parameters under each hypothesis, there may be a deterioration in performance in the estimation process. Determining how to incorporate such an “estimation loss” into the error exponent is an issue. Another open issue is the GLRT with PDFs of different forms for each hypothesis, which leads to a different distribution embedding associated with each hypothesis.