Discriminant Analysis under f-Divergence Measures

Abstract

In statistical inference, the information-theoretic performance limits can often be expressed in terms of a statistical divergence between the underlying statistical models (e.g., in binary hypothesis testing, the error probability is related to the total variation distance between the statistical models). As the data dimension grows, computing the statistics involved in decision-making and the attendant performance limits (divergence measures) face complexity and stability challenges. Dimensionality reduction addresses these challenges at the expense of compromising the performance (the divergence reduces by the data-processing inequality). This paper considers linear dimensionality reduction such that the divergence between the models is maximally preserved. Specifically, this paper focuses on Gaussian models where we investigate discriminant analysis under five f-divergence measures (Kullback–Leibler, symmetrized Kullback–Leibler, Hellinger, total variation, and ${\chi }^2$). We characterize the optimal design of the linear transformation of the data onto a lower-dimensional subspace for zero-mean Gaussian models and employ numerical algorithms to find the design for general Gaussian models with non-zero means. There are two key observations for zero-mean Gaussian models. First, projections are not necessarily along the largest modes of the covariance matrix of the data, and, in some situations, they can even be along the smallest modes. Secondly, under specific regimes, the optimal design of subspace projection is identical under all the f-divergence measures considered, rendering a degree of universality to the design, independent of the inference problem of interest.

1. Introduction

1.1. Motivation

Consider a simple binary hypothesis testing problem in which we observe an n-dimensional sample X and aim to discern the underlying model according to:

$$\begin{array}{c}\hfill {\mathsf{H}}_0: X\sim \mathbb{P}\hspace{1em}\mathrm{vs}.\hspace{1em}{\mathsf{H}}_1: X\sim \mathbb{Q}.\end{array}$$

(1)

The optimal decision rule (in the Neyman-Pearson sense) involves computing the likelihood ratio $\frac{\mathrm{d}\mathbb{P}}{\mathrm{d}\mathbb{Q}}\left(X\right)$ and the performance limit (sum of type I and type II errors) is related to the total variation distance between $\mathbb{P}$ and $\mathbb{Q}$. We emphasize that our focus is on the settings in which the n elements of X are not statistically independent, in which case the likelihood ratio $\frac{\mathrm{d}\mathbb{P}}{\mathrm{d}\mathbb{Q}}\left(X\right)$ cannot be decomposed into the product of the coordinate-level likelihood ratios. One of the key practical obstacles to solve such problems pertains to the computational cost of finding and performing the statistical tests. This renders a gap between the performance that is information-theoretically viable (unbounded complexity) versus a performance possible under bounded computational complexity [1,2].

Dimensionality reduction techniques have become an integral part of statistical analysis in high dimensions [3,4,5,6]. In particular, linear dimensionality reduction methods have been developed and used for over a century for various reasons, such as their low computational complexity and simple geometric interpretation, as well as for a multitude of applications, such as data compression, storage, and visualization, to name only a few. These methods linearly map the high-dimensional data to lower dimensions while ensuring that the desired features of the data are preserved. There exist two broad sets of approaches to linear dimensionality reduction in one dataset X, which we review next.

1.2. Related Literature

(1)Feature extraction: In one set of approaches, the objective is to select and extract informative and non-redundant features in the dataset X. These approaches are generally unsupervised. These widely-used approaches are principal component analysis (PCA), and its variations [7,8,9], multidimensional scaling (MDS) [10,11,12,13], and sufficient dimensionality reduction (SDR) [14]. The objective of PCA is to retain as much variation in the data in a lower dimension by minimizing the reconstruction error. In contrast, MDS aims to maximize the scatter of the projection and maximizes an aggregate scatter metric. Finally, the objective of SDR is to design an orthogonal mapping of the data that makes the data X and the responses conditionally independent (given the projected data). There exist extensive variations to the three approaches, and we refer the reader to Reference [6] for more discussions.

(2)Class separation: In another set of approaches, the objective is to perform classification in the lower dimensional space. These approaches are supervised. Depending on the problem formulation and the underlying assumptions, the resulting decision boundaries between the models can be linear or non-linear. One approach pertinent to this paper’s scope is discriminant analysis (DA), that leverages the distinction between given models and designs a mapping such that its lower-dimensional output exhibits maximum separation across different models [15,16,17,18,19,20]. In general, this approach generates two matrices: within-class and between-class scatter matrices. The within-class scatter matrix shows the scatter of the samples around their respective class means, whereas, in contrast, the between-class scatter matrix captures the scatter of the samples around the mixture mean of all the models. Subsequently, a univariate function of these matrices is formed such that it increases when the between-class scatter becomes larger, or when the within-class scatter becomes smaller. Examples of such a function of between-class and within-class matrices is a classification index that includes the ratio of their determinants, difference of their determinants, and ratio of their traces [17]. These approaches focus on reducing the dimension to one and maximize separability between the two classes. There exist, however, studies that consider reducing to dimensions higher than one and separation across more than two classes. Finally, depending on the structure of the class-conditional densities, the resulting shape of the decision boundaries give rise to linear and quadratic DA.

The f-divergences between a pair of probability measures quantifies the similarity between them. Shannon [21] introduced the mutual information as a divergence measure, which was later studied comprehensively by Kullback and Leibler [22] and Kolmogorov [23], establishing the importance of such measures in information theory, probability theory, and related disciplines. The family of f-divergences, independently introduced by Csiszár [24], Ali and Silvey [25], and Morimoto [26], generalize the Kullback–Leibler divergence which enable characterizing the information-theoretic performance limits of a wide range of inference, learning, source coding, and channel coding problems. For instance, References [27,28,29,30] consider their application to various statistical decision-making problems [31,32,33,34]. More recent developments on the properties of f-divergence measures can be found in References [31,35,36,37].

1.3. Contributions

The contribution of this paper has two main distinctions from the existing literature on DA. First, DA generally focuses on the classification problem for determining the underlying model of the data. Secondly, motivated by the complexities of finding the optimal decision rules for classification (e.g., density estimation), the existing criteria used for separation are selected heuristically. In this paper, we study this problem by referring to the family of f-divergences as measures of the distinction between a pair of probability distributions. Such a choice has three main features: (i) it enables designing linear mappings for a wider range of inference problems (beyond classification); (ii) it provides the designs that are optimal for the inference problem at hand; and (iii) it enables characterizing the information-theoretic performance limits after linear mapping. Our analyses are focused on Gaussian models. Even though we observe that the design of the linear mapping has differences under different f-divergence measures, we have two main observations in the case of zero-mean Gaussian models: (i) the optimal design of the linear mapping is not necessarily along the most dominant components of the data matrix; and (ii) in certain regimes, irrespective of the choice of the f-divergence measure, the design of the linear map that retains the maximal divergence between the two models is robust. In such cases, this makes the optimal design of the linear map independent of the inference problem at hand rendering a degree of universality (in the considered space of the Gaussian probability measures).

The remainder of the paper is organized as follows. Section 2 provides the linear dimensionality reduction model, and it provides an overview of the f-divergence measures considered in this paper. Section 3 formulates the problem, and it helps to facilitate the mathematical analysis in subsequent sections. In Section 4, we provide a motivating operational interpretation for each f-divergence measure and then characterize an optimal design of the linear mapping for zero-mean Gaussian models. Section 5 considers numerical simulations for inference problems associated with the f-divergence measure of interest for zero-mean Gaussian models. Section 6 generalizes the theory to non-zero mean Gaussian models and discusses numerical algorithms that help characterize the design of the linear map, and Section 7 concludes the paper. A list of abbreviations used in this paper is provided on page 22.

2. Preliminaries

Consider a pair of n-dimensional Gaussian models:

$$\begin{array}{c}\hfill \mathbb{P}: \mathcal{N}({\mathit{\mu }}_{\mathbb{P}},{\mathbf{\Sigma }}_{\mathbb{P}}), \mathrm{and} \mathbb{Q}: \mathcal{N}({\mathit{\mu }}_{\mathbb{Q}},{\mathbf{\Sigma }}_{\mathbb{Q}}),\end{array}$$

(2)

where ${\mathit{\mu }}_{\mathbb{P}},{\mathit{\mu }}_{\mathbb{Q}}$ and ${\mathbf{\Sigma }}_{\mathbb{P}},{\mathbf{\Sigma }}_{\mathbb{Q}}$ are two distinct mean vectors and covariance matrices, respectively, and $\mathbb{P}$ and $\mathbb{Q}$ denote their associated probability measures. The nature selects one model and generates a random variable $X\in {\mathbb{R}}^n$. We perform linear dimensionality reduction on X via matrix $\mathbf{A}\in {\mathbb{R}}^{r\times n}$, where $r<n$, rendering

$$\begin{array}{c}\hfill Y \stackrel{▵}{=}\mathbf{A}·X.\end{array}$$

(3)

After linear mapping, the two possible distributions of Y induced by matrix $\mathbf{A}$ are denoted by ${\mathbb{P}}_{\mathbf{A}}$ and ${\mathbb{Q}}_{\mathbf{A}}$, where

$$\begin{array}{c}\hfill \begin{array}{cc}{\mathbb{P}}_{\mathbf{A}} :\hfill & \mathcal{N}(\mathbf{A}·{\mathit{\mu }}_{\mathbb{P}},\mathbf{A}·{\mathbf{\Sigma }}_{\mathbb{P}}·{\mathbf{A}}^{\top })\hfill \\ {\mathbb{Q}}_{\mathbf{A}}:\hfill & \mathcal{N}(\mathbf{A}·{\mathit{\mu }}_{\mathbb{Q}},\mathbf{A}·{\mathbf{\Sigma }}_{\mathbb{Q}}·{\mathbf{A}}^{\top })\hfill \end{array}.\end{array}$$

(4)

Motivated by inference problems that we discuss in Section 3, our objective is to design the linear mapping parameterized by matrix $\mathbf{A}$ that ensures that the two possible distributions of Y, i.e., ${\mathbb{P}}_{\mathbf{A}}$ and ${\mathbb{Q}}_{\mathbf{A}}$, are maximally distinguishable. That is, to design $\mathbf{A}$ as a function of the statistical models (i.e., ${\mathit{\mu }}_{\mathbb{P}}$, ${\mathit{\mu }}_{\mathbb{Q}}$, ${\mathbf{\Sigma }}_{\mathbb{P}}$ and ${\mathbf{\Sigma }}_{\mathbb{Q}}$) such that relevant notions of f-divergences between ${\mathbb{P}}_{\mathbf{A}}$ and ${\mathbb{Q}}_{\mathbf{A}}$ are maximized. We use a number of f-divergence measures for capturing the distinction between ${\mathbb{P}}_{\mathbf{A}}$ and ${\mathbb{Q}}_{\mathbf{A}}$, each with a distinct operational meaning under specific inference problems. For this purpose, we denote the f-divergence of ${\mathbb{Q}}_{\mathbf{A}}$ from ${\mathbb{P}}_{\mathbf{A}}$ by $D_f\left(\mathbf{A}\right)$, where

$$\begin{array}{c}\hfill D_f\left(\mathbf{A}\right) \stackrel{▵}{=}{\mathbb{E}}_{{\mathbb{P}}_{\mathbf{A}}}\left[f\left(\frac{\mathrm{d}{\mathbb{Q}}_{\mathbf{A}}}{\mathrm{d}{\mathbb{P}}_{\mathbf{A}}}\right)\right].\end{array}$$

(5)

We use the shorthand $D_f\left(\mathbf{A}\right)$ for the canonical notation $D_f({\mathbb{Q}}_{\mathbf{A}} \parallel {\mathbb{P}}_{\mathbf{A}})$ for emphasizing the dependence on $\mathbf{A}$ and for the simplicity in notations. ${\mathbb{E}}_{{\mathbb{P}}_{\mathbf{A}}}$ denotes the expectation with respect to ${\mathbb{P}}_{\mathbf{A}}$, and $f:(0,+\infty )\to \mathbb{R}$ is a convex function that is strictly convex at 1 and $f\left(1\right)=0$. Strict convexity at 1 ensures that the f-divergence between a pair of probability measures is zero if and only if the probability measures are identical. Given the linear dimensionality reduction model in (3), the objective is to solve

$$\begin{array}{cc}\hfill \mathcal{P}: \underset{\mathbf{A}\in {\mathbb{R}}^{r\times n}}{max}& D_f\left(\mathbf{A}\right),\hfill \end{array}$$

(6)

for the following choices of the f-divergence measures.

• Kullback–Leibler (KL) divergence for $f\left(t\right)=tlogt$:

  $$\begin{array}{c}\hfill D_{\mathsf{KL}}\left(\mathbf{A}\right) \stackrel{▵}{=}{\mathbb{E}}_{{\mathbb{Q}}_{\mathbf{A}}}\left[log\frac{\mathrm{d}{\mathbb{Q}}_{\mathbf{A}}}{\mathrm{d}{\mathbb{P}}_{\mathbf{A}}}\right].\end{array}$$

  (7)
  We also denote the KL divergence from ${\mathbb{P}}_{\mathbf{A}}$ to ${\mathbb{Q}}_{\mathbf{A}}$ by $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$.
• Symmetric KL divergence for $f\left(t\right)=(t-1)logt$:

  $$\begin{array}{c}\hfill D_{\mathsf{SKL}}\left(\mathbf{A}\right) \stackrel{▵}{=}{D}_{\mathsf{KL}}({\mathbb{Q}}_{\mathbf{A}} \parallel {\mathbb{P}}_{\mathbf{A}})+D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}}).\end{array}$$

  (8)
• Squared Hellinger distance for $f\left(t\right)={(1-\sqrt{t})}^2$:

  $$\begin{array}{c}\hfill {\mathsf{H}}^2\left(\mathbf{A}\right) \stackrel{▵}{=}{\int }_{{\mathbb{R}}^r}{\left(\sqrt{\mathrm{d}{\mathbb{Q}}_{\mathbf{A}}}-\sqrt{\mathrm{d}{\mathbb{P}}_{\mathbf{A}}}\right)}^2.\end{array}$$

  (9)
• Total variation distance for $f\left(t\right)=\frac{1}{2}·|t-1|$:

  $$\begin{array}{c}\hfill d_{\mathsf{TV}}\left(\mathbf{A}\right) \stackrel{▵}{=}\frac{1}{2}{\int }_{{\mathbb{R}}^r}\left|\mathrm{d}{\mathbb{Q}}_{\mathbf{A}}-\mathrm{d}{\mathbb{P}}_{\mathbf{A}}\right|.\end{array}$$

  (10)
• ${\chi }^2$-divergence for $f\left(t\right)={(t-1)}^2$:

  $$\begin{array}{c}\hfill {\chi }^2\left(\mathbf{A}\right) \stackrel{▵}{=}{\int }_{{\mathbb{R}}^r}\frac{{(\mathrm{d}{\mathbb{Q}}_{\mathbf{A}}-\mathrm{d}{\mathbb{P}}_{\mathbf{A}})}^2}{\mathrm{d}{\mathbb{P}}_{\mathbf{A}}}.\end{array}$$

  (11)
  We also denote the ${\chi }^2$-divergence from ${\mathbb{P}}_{\mathbf{A}}$ to ${\mathbb{Q}}_{\mathbf{A}}$ by ${\chi }^2({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$.

3. Problem Formulation

In this section, without loss of generality, we focus on the setting where one of the covariance matrices is the identity matrix, and the other one has a covariance matrix $\mathbf{\Sigma }$ in order to avoid complex representations. One key observation is that the design of $\mathbf{A}$ under different measures has strong similarities. We first note that, by defining $\overline{\mathbf{A}} \stackrel{▵}{=}\mathbf{A}·{\mathbf{\Sigma }}_{\mathbb{P}}^{1/2}$, $\mathit{\mu } \stackrel{▵}{=}{\mathbf{\Sigma }}_{\mathbb{P}}^{-1/2}·({\mathit{\mu }}_{\mathbb{Q}}-{\mathit{\mu }}_{\mathbb{P}})$, and $\mathbf{\Sigma } \stackrel{▵}{=}{\mathbf{\Sigma }}_{\mathbb{P}}^{-1/2}·{\mathbf{\Sigma }}_{\mathbb{Q}}·{\mathbf{\Sigma }}_{\mathbb{P}}^{-1/2}$, designing $\mathbf{A}$ for maximally distinguishing

$$\begin{array}{c}\hfill \mathcal{N}(\mathbf{A}·{\mathit{\mu }}_{\mathbb{P}},\mathbf{A}·{\mathbf{\Sigma }}_{\mathbb{P}}·{\mathbf{A}}^{\top }) \mathrm{and} \mathcal{N}(\mathbf{A}·{\mathit{\mu }}_{\mathbb{Q}},\mathbf{A}·{\mathbf{\Sigma }}_{\mathbb{Q}}·{\mathbf{A}}^{\top })\end{array}$$

(12)

is equivalent to designing $\overline{\mathbf{A}}$ for maximally distinguishing

$$\begin{array}{c}\hfill \mathcal{N}(\mathbf{0},\overline{\mathbf{A}}·{\overline{\mathbf{A}}}^{\top }) \mathrm{and} \mathcal{N}(\overline{\mathbf{A}}·\mathit{\mu },\overline{\mathbf{A}}·\mathbf{\Sigma }·{\overline{\mathbf{A}}}^{\top }).\end{array}$$

(13)

Hence, without loss of generality, we focus on the setting where ${\mathit{\mu }}_{\mathbb{P}}=\mathbf{0}$, ${\mathbf{\Sigma }}_{\mathbb{P}}={\mathbf{I}}_n$, and ${\mathbf{\Sigma }}_{\mathbb{Q}}=\mathbf{\Sigma }$. Next, we show that determining an optimal design for $\mathbf{A}$ can be confined to the class of semi-orthogonal matrices.

Theorem 1.

For every $\mathbf{A}$, there exists a semi-orthogonal matrix $\overline{\mathbf{A}}$ such that $D_f\left(\overline{\mathbf{A}}\right)=D_f\left(\mathbf{A}\right)$.

Proof. 

See Appendix A.    □

This observation indicates that we can reduce the unconstrained problem in (6) to the following constrained problem:

$$\begin{array}{cc}\hfill \mathcal{Q}: & \underset{\mathbf{A}\in {\mathbb{R}}^{r\times n}}{max} D_f\left(\mathbf{A}\right) \mathrm{s}.\mathrm{t}. \mathbf{A}·{\mathbf{A}}^{\top }={\mathbf{I}}_r.\hfill \end{array}$$

(14)

We show that the design of $\mathbf{A}$ in the case of $\mathit{\mu }=\mathbf{0}$, under the considered f-divergence measures, directly relates to analyzing the eigenspace of matrix $\mathbf{\Sigma }$. For this purpose, we denote the non-negative eigenvalues of $\mathbf{\Sigma }$ ordered in the descending order by $\{{\lambda }_i:i\in \left[n\right]\}$, where for an integer m we have defined $\left[m\right]=\{1,\dots ,m\}$. For an arbitrary permutation function $\pi :\left[n\right]\to \left[n\right]$, we denote the permutation of $\{{\lambda }_i:i\in \left[n\right]\}$ with respect to $\pi$ by $\{{\lambda }_{\pi \left(i\right)}:i\in \left[n\right]\}$. We also denote the eigenvalues of $\mathbf{A}·\mathbf{\Sigma }·{\mathbf{A}}^{\top }$ ordered in the descending order by $\{{\gamma }_i:i\in \left[r\right]\}$. Throughout the analysis, we frequently use Poincaré separation theorem [38] for finding the row space of matrix $\mathbf{A}$ with respect to the eigenvalues of $\mathbf{\Sigma }$.

Theorem 2

(Poincaré Separation Theorem). Let Σ be a real symmetric $n\times n$ matrix and $\mathbf{A}$ be a semi-orthogonal $r\times n$ matrix. The eigenvalues of Σ denoted by $\{{\lambda }_i:i\in \left[n\right]\}$ (sorted in the descending order) and the eigenvalues of $\mathbf{A}·\mathbf{\Sigma }·{\mathbf{A}}^{\top }$ denoted by $\{{\gamma }_i:i\in \left[r\right]\}$ (sorted in the descending order) satisfy

$$\begin{array}{c}\hfill {\lambda }_{n-(r-i)}\le {\gamma }_i\le {\lambda }_i, \forall i\in \left[r\right].\end{array}$$

(15)

Finally, we define the following functions, which we will refer to frequently throughout the paper:

$$\begin{array}{cc}\hfill h_1\left(\mathbf{A}\right)& \stackrel{▵}{=}\mathbf{A}·\mathbf{\Sigma }·{\mathbf{A}}^{\top },\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill h_2\left(\mathbf{A}\right)& \stackrel{▵}{=}{\mathit{\mu }}^{\top }·{\mathbf{A}}^{\top }·\mathbf{A}·\mathit{\mu },\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill h_3\left(\mathbf{A}\right)& \stackrel{▵}{=}{\mathit{\mu }}^{\top }·{\mathbf{A}}^{\top }·{\left[h_1\left(\mathbf{A}\right)\right]}^{-1}·\mathbf{A}·\mathit{\mu }.\hfill \end{array}$$

(18)

In the next sections, we analyze the design of $\mathbf{A}$ under different f-divergence measures. In particular, in Section 4 and Section 5, we focus on zero-mean Gaussian models for $\mathbb{P}$ and $\mathbb{Q}$ where we provide an operational interpretation of the measure in the dichotomous mode in (4). Subsequently, we will discuss the generalization to non-zero mean Gaussian models in Section 6.

4. Main Results for Zero-Mean Gaussian Models

In this section, we analyze problem $\mathcal{Q}$ defined in (14) for each of the f-divergence measures separately. Specifically, for each case, we briefly provide an inference problem as a motivating example, in the context of which we relate the optimal performance limit of that inference problem to the f-divergence of interest. These analyses are provided in Section 4.1, Section 4.2, Section 4.3, Section 4.4 and Section 4.5. Subsequently, we provide the main results on the optimal design of the linear mapping matrix $\mathbf{A}$ in Section 4.6.

4.1. Kullback Leibler Divergence

4.1.1. Motivation

The KL divergence, being the expected value of the log-likelihood ratio, captures, at least partially, the performance of a wide range of inference problems. One specific problem whose performance is completely captured by $D_{\mathsf{KL}}\left(\mathbf{A}\right)$ is the quickest change-point detection. Consider an observation process (time-series) $\{X_t:t\in \mathbb{N}\}$ in which the observations $X_t\in {\mathbb{R}}^n$ are generated by a distribution with probability measure $\mathbb{P}$ specified in (2). This distribution changes to $\mathbb{Q}$ at an unknown (random or deterministic) time $\kappa$, i.e.,

$$\begin{array}{c}\hfill \begin{array}{c}{X}_t\sim \mathbb{P}\hspace{1em}t<\kappa , \mathrm{and} X_t\sim \mathbb{Q}\hspace{1em}t\ge \kappa \hfill \end{array}.\end{array}$$

(19)

Change-point detection algorithms sample the observation process sequentially and aim to detect the change point with the minimal delay after it occurs subject to a false alarm constraint. Hence, the two key figures of merit capturing the performance of a sequential change-point detection algorithm are the average detection delay ($\mathsf{ADD}$) and the rate of false alarms. Whether the change-point $\kappa$ is random or deterministic gives rise to two broad classes of quickest change-point detection problems, namely the Bayesian setting ($\kappa$ is random) and minimax setting ($\kappa$ is deterministic). Irrespective of their discrepancies in settings and the nature of performance guarantees, the $\mathsf{ADD}$ for the (asymptotically) optimal algorithms are in the form [39]:

$$\begin{array}{c}\hfill \mathsf{ADD}\sim \frac{c_1}{D_{\mathsf{KL}}(\mathbb{Q} \parallel \mathbb{P})}.\end{array}$$

(20)

Hence, after the linear mapping induced by matrix $\mathbf{A}$, for the $\mathsf{ADD}$, we have

$$\begin{array}{c}\hfill \mathsf{ADD}\sim \frac{c_2}{D_{\mathsf{KL}}({\mathbb{Q}}_{\mathbf{A}} \parallel {\mathbb{P}}_{\mathbf{A}})},\end{array}$$

(21)

where $c_1$ and $c_2$ are constants specified by the false alarm constraints. Clearly, the design of $\mathbf{A}$ that minimizes the $\mathsf{ADD}$ will be maximizing the disparity between the pre- and post-change distributions ${\mathbb{P}}_{\mathbf{A}}$ and ${\mathbb{Q}}_{\mathbf{A}}$, respectively.

4.1.2. Connection between $D_{\mathsf{KL}}$ and $\mathbf{A}$

By noting that $\mathbf{A}$ is a semi-orthogonal matrix and recalling that the eigenvalues of $h_1\left(\mathbf{A}\right)$ are denoted by $\{{\gamma }_i:i\in \left[r\right]\}$, simple algebraic manipulations simplify $D_{\mathsf{KL}}({\mathbb{Q}}_{\mathbf{A}} \parallel {\mathbb{P}}_{\mathbf{A}})$ to:

$$\begin{array}{cc}\hfill D_{\mathsf{KL}}({\mathbb{Q}}_{\mathbf{A}} \parallel {\mathbb{P}}_{\mathbf{A}})& =\frac{1}{2}\left[log\frac{1}{|h_1\left(\mathbf{A}\right)|}-r+Tr\left[h_1\left(\mathbf{A}\right)\right]+h_2\left(\mathbf{A}\right)\right].\hfill \end{array}$$

(22)

By setting, and leveraging, Theorem 2, the problem of finding an optimal design for $\mathbf{A}$ that solves (14) can be found as the solution to:

$$\begin{array}{c}\hfill \underset{\{{\gamma }_i : i\in \left[r\right]\}}{max}\sum _{i=1}^r{g}_{\mathsf{KL}}\left({\gamma }_i\right) \mathrm{s}.\mathrm{t}. {\lambda }_{n-(r-i)}\le {\gamma }_i\le {\lambda }_i \forall i\in \left[r\right],\end{array}$$

(23)

where we have defined

$$\begin{array}{c}\hfill g_{\mathsf{KL}}\left(x\right) \stackrel{▵}{=}\frac{1}{2}(x-logx-1).\end{array}$$

(24)

Likewise, finding the optimal design for $\mathbf{A}$ that optimizes $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ when $\mathit{\mu }=0$ can be found by replacing $g_{\mathsf{KL}}\left({\gamma }_i\right)$ by $g_{\mathsf{KL}}\left(\frac{1}{{\gamma }_i}\right)$ in (23). In either case, the optimal design of $\mathbf{A}$ is constructed by choosing r eigenvectors of $\mathbf{\Sigma }$ as the rows of $\mathbf{A}$. The results and observations are formalized in Section 4.6.

4.2. Symmetric KL Divergence

4.2.1. Motivation

The KL divergence discussed in Section 4.1 is an asymmetric measure of separation between two probability measures. It is symmetrized by adding two directed divergence measures in reverse directions. The symmetric KL divergence has applications in model selection problems in which the model selection criteria is based on a measure of disparity between the true model and the approximating models. As shown in Reference [40], using the symmetric KL divergence outperforms the individual directed KL divergences since it better reflects the risks associated with underfitting and overfitting of the models, respectively.

4.2.2. Connection between $D_{\mathsf{SKL}}$ and $\mathbf{A}$

For a given $\mathbf{A}$, the symmetric KL divergence of interest specified in (8) is given by

$$\begin{array}{c}\hfill D_{\mathsf{SKL}}\left(\mathbf{A}\right)=\frac{1}{2}·\left[Tr\left({\left[h_1\left(\mathbf{A}\right)\right]}^{-1}+h_1\left(\mathbf{A}\right)\right)+h_2\left(\mathbf{A}\right)+h_3\left(\mathbf{A}\right)\right]-r.\end{array}$$

(25)

By setting $\mathit{\mu }=0$, and leveraging Theorem 2, the problem of finding an optimal design for $\mathbf{A}$ that solves (14) can be found as the solution to:

$$\begin{array}{c}\hfill \underset{\{{\gamma }_i : i\in \left[r\right]\}}{max}\sum _{i=1}^r{g}_{\mathsf{SKL}}\left({\gamma }_i\right) \mathrm{s}.\mathrm{t}. {\lambda }_{n-(r-i)}\le {\gamma }_i\le {\lambda }_i \forall i\in \left[r\right],\end{array}$$

(26)

where we have defined

$$\begin{array}{c}\hfill g_{\mathsf{SKL}}\left(x\right) \stackrel{▵}{=}\frac{1}{2}\left(x+\frac{1}{x}-2\right).\end{array}$$

(27)

4.3. Squared Hellinger Distance

4.3.1. Motivation

Squared Hellinger distance facilitates analysis in high dimensions, especially when other measures fail to take closed-form expressions. We will discuss an important instance of this in the next subsection in the analysis of $d_{\mathsf{TV}}$. Squared Hellinger distance is symmetric, and it is confined in the range $[0,2]$.

4.3.2. Connection between ${\mathsf{H}}^2$ and $\mathbf{A}$

For a given matrix $\mathbf{A}$, we have the following closed-form expression:

$$\begin{array}{c}\hfill {\mathsf{H}}^2\left(\mathbf{A}\right)= \begin{array}{cc}\hfill & 2-2\frac{|4·h_1{\left(\mathbf{A}\right)|}^{\frac{1}{4}}}{|h_1\left(\mathbf{A}\right)+{\mathbf{I}}_r{|}^{\frac{1}{2}}}·exp\left(-\frac{{\mathit{\mu }}^{\top }·{\mathbf{A}}^{\top }·{\left[h_1\left(\mathbf{A}\right)+{\mathbf{I}}_r\right]}^{-1}·\mathbf{A}·\mathit{\mu }}{4}\right).\hfill \end{array}\end{array}$$

(28)

By setting $\mathit{\mu }=0$, and leveraging Theorem 2, the problem of finding an optimal design for $\mathbf{A}$ that solves (14) can be found as the solution to:

$$\begin{array}{c}\hfill \underset{\{{\gamma }_i : i\in \left[r\right]\}}{max}\prod _{i=1}^r{g}_{\mathsf{H}}\left({\gamma }_i\right) \mathrm{s}.\mathrm{t}. {\lambda }_{n-(r-i)}\le {\gamma }_i\le {\lambda }_i \forall i\in \left[r\right],\end{array}$$

(29)

where we have defined

$$\begin{array}{c}\hfill g_{\mathsf{H}}\left(x\right) \stackrel{▵}{=}\frac{{(x+1)}^2}{x}.\end{array}$$

(30)

4.4. Total Variation Distance

4.4.1. Motivation

The total variation distance appears as the key performance metric in binary hypothesis testing and in high-dimensional inference, e.g., Le Cam’s method for the binary quantization and testing of the individual dimensions (which is in essence binary hypothesis testing). In particular, for the simple binary hypothesis testing model in (65), the minimum total probability of error (sum of type-I and type-II error probabilities) is related to the total variation $d_{\mathsf{TV}}\left(\mathbf{A}\right)$. Specifically, for a decision rule $d:X\to \{{\mathsf{H}}_0,{\mathsf{H}}_1\}$, the following holds:

$$\begin{array}{c}\hfill \underset{d}{inf} [{\mathbb{P}}_{\mathbf{A}}(d={\mathsf{H}}_1)+{\mathbb{Q}}_{\mathbf{A}}(d={\mathsf{H}}_0)]=1-d_{\mathsf{TV}}\left(\mathbf{A}\right).\end{array}$$

(31)

The total variation between two Gaussian distributions does not have a closed-form expression. Hence, unlike the other settings, an optimal solution to (6) in this context cannot be obtained analytically. Alternatively, in order to gain intuition into the structure of a near optimal matrix $\mathbf{A}$, we design $\mathbf{A}$ such that it optimizes known bounds on $d_{\mathsf{TV}}\left(\mathbf{A}\right)$. In particular, we use two sets of bounds on $d_{\mathsf{TV}}\left(\mathbf{A}\right)$. One set is due to bounding it via the Hellinger distance, and another set is due to a recent study that established upper and lower bounds that are identical up to a constant factor [41].

4.4.2. Connection between $d_{\mathsf{TV}}$ and $\mathbf{A}$

(1) Bounding by Hellinger Distance: The total variation distance can be bounded by the Hellinger distance according to

$$\begin{array}{c}\hfill \frac{1}{2}{\mathsf{H}}^2\left(\mathbf{A}\right)\le d_{\mathsf{TV}}\left(\mathbf{A}\right)\le \mathsf{H}\left(\mathbf{A}\right)\sqrt{1-\frac{{\mathsf{H}}^2\left(\mathbf{A}\right)}{4}}.\end{array}$$

(32)

It can be readily verified that these bounds are monotonically increasing with ${\mathsf{H}}^2\left(\mathbf{A}\right)$ in the interval $[0,2]$. Hence, they are maximized simultaneously by maximizing the squared Hellinger distance as discussed in Section 4.3. We refer to this bound as the Hellinger bound.

(2) Matching Bounds up to a Constant: The second set of bounds that we used are provided in Reference [41]. These bounds relate the total variation between two Gaussian models to the Frobenius norm (FB) of a matrix related to their covariance matrices. Specifically, these FB-based bounds on the total variation $d_{\mathsf{TV}}\left(\mathbf{A}\right)$ are given by

$$\begin{array}{c}\hfill \frac{1}{100}\le \frac{d_{\mathsf{TV}}\left(\mathbf{A}\right)}{min\{1,\sqrt{{\sum }_{i=1}^r{g}_{\mathsf{TV}}\left({\gamma }_i\right)}\}}\le \frac{3}{2},\end{array}$$

(33)

where we have defined

$$\begin{array}{c}\hfill g_{\mathsf{TV}}\left(x\right) \stackrel{▵}{=}{\left(\frac{1}{x}-1\right)}^2.\end{array}$$

(34)

Since the lower and upper bounds on $d_{\mathsf{TV}}\left(\mathbf{A}\right)$ are identical up to a constant, they will be maximized by the same design of $\mathbf{A}$.

4.5. ${\chi }^2$-Divergence

4.5.1. Motivation

${\chi }^2$-divergence appears in a wide range of statistical estimation problems for the purpose of finding a lower bound on the estimation noise variance. For instance, consider the canonical problem of estimating a latent variable $\theta$ from the observed data X, and denote two candidate estimates by $p\left(X\right)$ and $q\left(X\right)$. Define $\mathbb{P}$ and $\mathbb{Q}$ as the probability measures of $p\left(X\right)$ and $q\left(X\right)$, respectively. According to the Hammersly-Chapman-Robbins (HCR) bound on the quadratic loss function, for any estimator $\widehat{\theta }$, we have

$$\begin{array}{c}\hfill {\mathsf{var}}_{\theta }\left(\widehat{\theta }\right)\ge \underset{p\ne q}{sup}\frac{{\left[{\mathbb{E}}_{\mathbb{Q}}\left[q\left(X\right)\right]-{\mathbb{E}}_{\mathbb{P}}\left[p\left(X\right)\right]\right]}^2}{{\chi }^2(\mathbb{Q} \parallel \mathbb{P})},\end{array}$$

(35)

which, for unbiased estimators p and q, simplifies to the Cramér-Rao lower bound

$$\begin{array}{c}\hfill {\mathsf{var}}_{\theta }\left(\widehat{\theta }\right)\ge \underset{p\ne q}{sup}\frac{{(q-p)}^2}{{\chi }^2(\mathbb{Q} \parallel \mathbb{P})},\end{array}$$

(36)

depending on $\mathbb{P}$ and $\mathbb{Q}$ through their ${\chi }^2$-divergence. Besides the applications to estimation problems, ${\chi }^2$ is easier to compute compared to some of other f-divergence measures (e.g., total variation). Specifically, for product distributions ${\chi }^2$ tensorizes to be expressed in terms of the one-dimensional components that are easier to compute than the KL divergence and TV variation distance. Hence, a combination of bounding other measures with ${\chi }^2$ and then analyzing ${\chi }^2$ appears in a wide range of inference problems.

4.5.2. Connection between ${\chi }^2$ and $\mathbf{A}$

By setting $\mathit{\mu }=0$, for a given matrix $\mathbf{A}$, from (11), we have the following closed-form expression:

$$\begin{array}{cc}\hfill {\chi }^2\left(\mathbf{A}\right)& =\frac{1}{|h_1\left(\mathbf{A}\right)|\sqrt{|2{\left(h_1\left(\mathbf{A}\right)\right)}^{-1}-{\mathbf{I}}_r|}}-1\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & =\prod _{i=1}^r{g}_{{\chi }_1}\left({\gamma }_i\right)-1,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hfill \end{array}$$

(38)

where we have defined

$$\begin{array}{c}\hfill g_{{\chi }_1}\left(x\right) \stackrel{▵}{=}\frac{1}{\sqrt{x(2-x)}}.\end{array}$$

(39)

As we show in Appendix C, for ${\chi }^2\left(\mathbf{A}\right)$ to exist (i.e., be finite), all the eigenvalues $\{{\lambda }_i:i\in \left[r\right]\}$ should fall in the interval $(0,2)$. Subsequently, finding the optimal design for $\mathbf{A}$ that optimizes ${\chi }^2({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ when $\mathit{\mu }=0$ can be done by replacing $g_{{\chi }_1}$ in (38) by $g_{{\chi }_2}$, which is given by

$$\begin{array}{c}\hfill g_{{\chi }_2}\left(x\right) \stackrel{▵}{=}\sqrt{\frac{x^2}{2x-1}}.\end{array}$$

(40)

Based on this, and by following a similar line of argument as in the case of the KL divergence, designing an optimal $\mathbf{A}$ reduces to identifying a subset of the eigenvalues of $\mathbf{\Sigma }$ and assigning their associated eigenvectors as the rows of matrix $\mathbf{A}$. These observations are formalized in Section 4.6.

4.6. Main Results

In this section, we provide analytical closed-form solutions to design optimal matrices $\mathbf{A}$ for the following f-divergence measures: $D_{\mathsf{KL}}$, $D_{\mathsf{SKL}}$, ${\mathsf{H}}^2$, and ${\chi }^2$. The total variation measure $d_{\mathsf{TV}}$ does not admit a closed-form for Gaussian models. In this case, we provide a design for $\mathbf{A}$ that optimizes the bound we have provided for $d_{\mathsf{TV}}$ in Section 4.4. Due to their structural similarities of the results, we group and treat $D_{\mathsf{KL}}$, $D_{\mathsf{SKL}}$, and $d_{\mathsf{TV}}$ in Theorem 3. Similarly, we group and treat ${\mathsf{H}}^2$ and ${\chi }^2$ in Theorem 4.

Theorem 3

($D_{\mathsf{KL}}$, $D_{\mathsf{SKL}}$, $d_{\mathsf{TV}}$). For a given function $g:\mathbb{R}\to \mathbb{R}$, define the permutations:

$$\begin{array}{c}\hfill {\pi }^{\ast } \stackrel{▵}{=}arg\underset{\pi }{max}\sum _{i=1}^{r}g\left({\lambda }_{\pi \left(i\right)}\right).\end{array}$$

(41)

Then, for $D_f\left(\mathbf{A}\right)\in \{D_{\mathsf{KL}}\left(\mathbf{A}\right),D_{\mathsf{SKL}}\left(\mathbf{A}\right),d_{\mathsf{TV}}\left(\mathbf{A}\right)\}$ and functions $g_f$$\in \{g_{\mathsf{KL}},g_{\mathsf{SKL}},g_{\mathsf{TV}}\}$:

• For maximizing $D_f$, set $g=g_f$ and select the eigenvalues of $\mathbf{A}\mathbf{\Sigma }{\mathbf{A}}^{\top }$ as

  $$\begin{array}{c}\hfill {\gamma }_i={\lambda }_{{\pi }^{\ast }\left(i\right)}, for i\in \left[r\right].\end{array}$$

  (42)
• Row $i\in \left[r\right]$ of matrix $\mathbf{A}$ is the eigenvector ofΣassociated with the eigenvalue ${\gamma }_i$.

Proof. 

See Appendix B.    □

By further leveraging the structures of functions $g_{\mathsf{KL}},g_{\mathsf{SKL}}$, and $g_{\mathsf{TV}}$, we can simplify approaches for designing the matrix $\mathbf{A}$. Specifically, note that the functions $g_{\mathsf{KL}},g_{\mathsf{SKL}},and g_{\mathsf{TV}}$ are all strictly convex functions taking their global minima at $x=1$. Based on this, we have the following observations.

Corollary 1

($D_{\mathsf{KL}}$, $D_{\mathsf{SKL}}$, $d_{\mathsf{TV}}$). For maximizing $D_f\left(\mathbf{A}\right)\in \{D_{\mathsf{KL}}\left(\mathbf{A}\right),D_{\mathsf{SKL}}\left(\mathbf{A}\right),d_{\mathsf{TV}}\left(\mathbf{A}\right)\}$, when ${\lambda }_n\ge 1$, we have ${\gamma }_i={\lambda }_i$ for all $i\in \left[r\right]$, and the rows of $\mathbf{A}$ are eigenvectors of Σ associated with its r largest eigenvalues, i.e., $\{{\lambda }_i:i\in \left[r\right]\}$.

Corollary 2

($D_{\mathsf{KL}}$, $D_{\mathsf{SKL}}$, $d_{\mathsf{TV}}$). For maximizing $D_f\left(\mathbf{A}\right)\in \{D_{\mathsf{KL}}\left(\mathbf{A}\right),D_{\mathsf{SKL}}\left(\mathbf{A}\right),d_{\mathsf{TV}}\left(\mathbf{A}\right)\}$, when ${\lambda }_1\le 1$, we have ${\gamma }_i={\lambda }_{n-r+i}$ for all $i\in \left[r\right]$, and the rows of $\mathbf{A}$ are eigenvectors of Σ associated with its r smallest eigenvalues, i.e., $\{{\lambda }_i:i\in \{n-r+1,\cdots ,n\}\}$.

Remark 1.

In order to maximize $D_f\left(\mathbf{A}\right)\in \{D_{\mathsf{KL}}\left(\mathbf{A}\right),D_{\mathsf{SKL}}\left(\mathbf{A}\right),d_{\mathsf{TV}}\left(\mathbf{A}\right)\}$ when ${\lambda }_n\le 1\le {\lambda }_1$, finding the best permutation of eigenvalues involves sorting all the n eigenvalues ${\lambda }_i$’s and subsequently performing r comparisons as illustrated in Algorithm 1. This amounts to $\mathcal{O}(n·log(n\left)\right)$ time complexity instead of $\mathcal{O}(n·log(r\left)\right)$ time complexity involved in determining the design for $\mathbf{A}$ in the case of Corollaries 1 and 2, which require finding the r extreme eigenvalues in determining the design for ${\pi }^{\ast }$.

Remark 2.

The optimal design of $\mathbf{A}$ often does not involve being aligned with the largest eigenvalues of the covariance matrixΣ, which is in contrast to some of the key approaches to linear dimensionality reduction that generally perform linear mapping along the eigenvectors associated with the largest eigenvalues of the covariance matrix. When the eigenvalues ofΣare all smaller than 1, in particular, $\mathbf{A}$ will be designed by choosing eigenvectors associated with the smallest eigenvalues ofΣin order to preserve largest separability.

Next, we provide the counterpart results for the ${\mathsf{H}}^2$ and ${\chi }^2$-divergence measures. Their major distinction from the previous three measures is that, for these two, $D_f\left(\mathbf{A}\right)$ can be decomposed into a product of individual functions of the eigenvalues $\{{\gamma }_i:i\in \left[r\right]\}$. Next, we provide the counterparts of Theorem 3 and Corollaries 1 and 2 for ${\mathsf{H}}^2$ and ${\chi }^2$.

Theorem 4

(${\mathsf{H}}^2$, ${\chi }^2$). For a given function $g:\mathbb{R}\to \mathbb{R}$, define the permutations:

$$\begin{array}{c}\hfill {\pi }^{\ast } \stackrel{▵}{=}arg\underset{\pi }{max}\prod _{i=1}^{r}g\left({\lambda }_{\pi \left(i\right)}\right).\end{array}$$

(43)

Then, for $D_f\left(\mathbf{A}\right)\in \{{\mathsf{H}}^2\left(\mathbf{A}\right),{\chi }^2\left(\mathbf{A}\right),{\chi }^2({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})\}$ and functions $g_f\in \{g_{\mathsf{H}},g_{{\chi }_1},g_{{\chi }_2}\}$:

• For maximizing $D_f$, set $g=g_f$ and select the eigenvalues of $\mathbf{A}\mathbf{\Sigma }{\mathbf{A}}^{\top }$ as

  $$\begin{array}{c}\hfill {\gamma }_i={\lambda }_{{\pi }^{\ast }\left(i\right)}, for i\in \left[r\right].\end{array}$$

  (44)
• Row $i\in \left[r\right]$ of matrix $\mathbf{A}$ is the eigenvector ofΣassociated with the eigenvalue ${\gamma }_i$.

Proof. 

See Appendix C.    □

Next, note that $g_{\mathsf{H}}$ is a strictly convex function taking its global minimum at $x=1$. Furthermore, $g_{{\chi }_i}$ for $i\in \left[2\right]$ are strictly convex over $(0,2)$ and take their global minimum at $x=1$.

Corollary 3

(${\mathsf{H}}^2$, ${\chi }^2$). For maximizing $D_f\left(\mathbf{A}\right)\in \{{\mathsf{H}}^2\left(\mathbf{A}\right),{\chi }^2\left(\mathbf{A}\right),{\chi }^2({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})\}$, when ${\lambda }_n\ge 1$, we have ${\gamma }_i={\lambda }_i$ for all $i\in \left[r\right]$, and the rows of $\mathbf{A}$ are eigenvectors of Σ associated with its r largest eigenvalues, i.e., $\{{\lambda }_i:i\in \left[r\right]\}$.

Corollary 4

(${\mathsf{H}}^2$, ${\chi }^2$). For maximizing $D_f\left(\mathbf{A}\right)\in \{{\mathsf{H}}^2\left(\mathbf{A}\right),{\chi }^2\left(\mathbf{A}\right),{\chi }^2({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})\}$, when ${\lambda }_1\le 1$, we have ${\gamma }_i={\lambda }_{n-r+i}$ for all $i\in \left[r\right]$, and the rows of $\mathbf{A}$ are eigenvectors of Σ associated with its r smallest eigenvalues, i.e., $\{{\lambda }_i:i\in \{n-r+1,\cdots ,n\}\}$.

Algorithm 1: Optimal Permutation ${\pi }^{\ast }$ When ${\lambda }_n\le 1\le {\lambda }_1$

1: Initialize $i\leftarrow n$, $j\leftarrow 1$, $p_k\leftarrow {\lambda }_k \forall k\in \{i,j\}$, ${\pi }^{\ast }\leftarrow \varnothing$ 2: Sort the eigenvalues of $\mathbf{\Sigma }$ in descending order $\{{\lambda }_k:k\in \left[n\right]\}$ 3: while$|{\pi }^{\ast }|\ne r$do 4:   if $g_f\left(p_i\right)>g_f\left(p_j\right)$ then 5:     ${\pi }^{\ast }\leftarrow {\pi }^{\ast }\cup \left\{p_i\right\}$ 6:     $i\leftarrow i-1$ 7:   else 8:     ${\pi }^{\ast }\leftarrow {\pi }^{\ast }\cup \left\{p_j\right\}$ 9:     $j\leftarrow j+1$ 10:   end if 11: end while 12: return${\pi }^{\ast }$

Finally, we remark that, unlike the other measures, total variation does not admit a closed-form, and we used two sets of tractable bounds to analyze this case of total variations. By comparing the design of $\mathbf{A}$ based on different bounds, we have the following observation.

Remark 3.

We note that both sets of bounds lead to the same design of $\mathbf{A}$ when either ${\lambda }_1\le 1$ or ${\lambda }_n\ge 1$. Otherwise, each will be selecting a different set of the eigenvectors ofΣto construct $\mathbf{A}$ according to the functions

$$\begin{array}{c}\hfill g_{\mathsf{H}}\left(x\right)=\frac{{(x+1)}^2}{x} \mathrm{versus} g_{\mathsf{TV}}\left(x\right)={\left(\frac{1}{x}-1\right)}^2.\end{array}$$

(45)

5. Zero-Mean Gaussian Models–Simulations

5.1. KL Divergence

In this section, we show gains of the above analysis for the KL divergence measure $D_{\mathsf{KL}}\left(\mathbf{A}\right)$ through simulations on a change-point detection problem. We focus on the minimax setting in which the change-point $\kappa$ is deterministic. The objective is to detect a change in the stochastic process $X_t$ with minimal delay after the change in the probability measure occurs at $\kappa$ and define $\tau \in \mathbb{N}$ as the time that we can form a confident decision. A canonical model to quantify the decision delay is the conditional average detection delay ($\mathsf{CADD}$) due to Pollak [42]

$$\begin{array}{cc}\hfill \mathsf{CADD}\left(\tau \right)& \stackrel{▵}{=}\underset{\kappa \ge 1}{sup} {\mathbb{E}}_{\kappa } [\tau -\kappa | \tau \ge \kappa ],\hfill \end{array}$$

(46)

where ${\mathbb{E}}_{\kappa }$ is the expectation with respect to the probability distribution when the change happens at time $\kappa$. The objective of this formulation is to optimize the decision delay for the worst-case realization of the random change-point $\kappa$ (that is, the change-point realization that leads to the maximum decision delay), while the constraints on the false alarm rate are satisfied. In this formulation, this worst-case realization is $\kappa =1$, in which case all the data points are generated from the post-change distribution. In the minimax setting, a reasonable measure of false alarms is the mean-time to false alarm, or its reciprocal, which is the false alarm rate ($\mathsf{FAR}$) defined as

$$\begin{array}{cc}\hfill \mathsf{FAR}\left(\tau \right)& \stackrel{▵}{=}\frac{1}{{\mathbb{E}}_{\infty }\left[\tau \right]} ,\hfill \end{array}$$

(47)

where ${\mathbb{E}}_{\infty }$ is the expectation with respect to the distribution when a change never occurs, i.e., $\kappa \stackrel{▵}{=}\infty$. A standard approach to balance the trade-off between decision delay and false alarm rates involves solving [42]

$$\begin{array}{c}\hfill \underset{\tau }{min}\mathsf{CADD}\left(\tau \right) \mathrm{s}.\mathrm{t}. \mathsf{FAR}\left(\tau \right)\le \alpha ,\end{array}$$

(48)

where $\alpha \in {\mathbb{R}}_{+}$ controls the rate of false alarms. For the quickest change-point detection formulation in (48), the popular cumulative sum (CuSum) test generates the optimal solutions, involving computing the following test statistic:

$$\begin{array}{c}\hfill W\left[t\right] \stackrel{▵}{=}{\displaystyle \underset{1\le k\le t+1}{max}}\sum _{i=k}^t\log \left(\frac{d{\mathbb{Q}}_{\mathbf{A}}\left(X_i\right)}{d{\mathbb{P}}_{\mathbf{A}}\left(X_i\right)}\right).\end{array}$$

(49)

Computing $W\left[t\right]$ follows a convenient recursion given by

$$\begin{array}{c}\hfill W\left[t\right] \stackrel{▵}{=}{\left(W[t-1]+\log \left(\frac{d{\mathbb{Q}}_{\mathbf{A}}\left(X_t\right)}{d{\mathbb{P}}_{\mathbf{A}}\left(X_t\right)}\right)\right)}^{+},\end{array}$$

(50)

where $W\left[0\right]=0$. The CuSum statistic declares a change at a stopping time $\tau$ given by

$$\begin{array}{c}\hfill \tau \stackrel{▵}{=}\inf \{t\ge 1:W\left[t\right]>C\},\end{array}$$

(51)

where C is chosen such that the constraint on $\mathsf{FAR}\left(\tau \right)$ in (48) is satisfied.

In this setting, we consider two zero-mean Gaussian models with the following pre- and post-linear dimensionality reduction structures:

$$\begin{array}{c}\hfill \begin{array}{cc}\mathbb{P}: \hfill & \mathcal{N}(\mathbf{0},{\mathbf{I}}_n) \mathrm{and} \mathbb{Q}: \mathcal{N}(\mathbf{0},\mathbf{\Sigma })\hfill \\ {\mathbb{P}}_{\mathbf{A}}: \hfill & \mathcal{N}(\mathbf{0},{\mathbf{I}}_r) \mathrm{and} {\mathbb{Q}}_{\mathbf{A}}: \mathcal{N}(\mathbf{0},h_1\left(\mathbf{A}\right))\hfill \end{array},\end{array}$$

(52)

where the covariance matrix $\mathbf{\Sigma }$ is generated randomly, and its eigenvalues are sampled from a uniform distribution. In particular, for the original data dimension n, $\ast \lceil 0.9n\rceil$ eigenvalues are sampled such that $\{{\lambda }_i\sim \mathcal{U}(0.064,1)\}$, and the remaining eigenvalues are sampled such that $\{{\lambda }_i\sim \mathcal{U}(1,4.24)\}$. We note that this is done since the objective function lies in the same range for the eigenvalues within the range $[0.0649,1]$ and $[1,4.24]$. In order to consider the worst case detection delay, we set $\kappa =1$ and generate stochastic observations according to the model described in (52) that follows the change-point detection model in (19). For every random realization of covariance matrix $\mathbf{\Sigma }$, we run the CuSum statistic (50), where we generate $\mathbf{A}$ according to the following two schemes:

(1) Largest eigen modes: In this scheme, the linear map $\mathbf{A}$ is designed such that its rows are eigenvectors associated with the r largest eigenvalues of $\mathbf{\Sigma }$.

(2) Optimal design: In this scheme, the linear map $\mathbf{A}$ is designed such that its rows are eigenvectors associated with r eigenvalues of $\mathbf{\Sigma }$ that maximize $D_{\mathsf{KL}}\left(\mathbf{A}\right)$ according to Theorem 3.

In order to evaluate and compare the performance of the two schemes, we compute the $\mathsf{ADD}$ obtained by running a Monte-Carlo simulation over 5000 random realizations of the stochastic process $X_t$ following the change-point detection model in (19) for every random realization of $\mathbf{\Sigma }$ and for each reduced dimension $1\le r\le 9$. The detection delays obtained are then averaged again over 100 random realizations of covariance matrices $\mathbf{\Sigma }$ for each reduced dimension r. Figure 1 shows the plot for $\mathsf{ADD}$ versus r for multiple initial data dimension n and for a fixed $\mathsf{FAR}=\frac{1}{5000}$. Owing to the dependence on $D_{\mathsf{KL}}\left(\mathbf{A}\right)$ given in (21), the delay associated with the optimal linear mapping in Theorem 3 achieves better performance.

Figure 1. Comparison of the average detection delay ($\mathsf{ADD}$) under the optimal design and largest eigen modes schemes for multiple reduced data dimensions r as a function of original data dimension n for a fixed false alarm rate ($\mathsf{FAR}$) which is equal to $1/5000$.

5.2. Symmetric KL Divergence

In this section, we show the gains of the analysis by numerically computing $D_{\mathsf{SKL}}\left(\mathbf{A}\right)$. We follow the pre- and post-linear dimensionality reduction structures given in (52), where the covariance matrix $\mathbf{\Sigma }$ is randomly generated following the setup used in Section 5.1. As plotted in Figure 2, by choosing the design scheme for $D_{\mathsf{SKL}}\left(\mathbf{A}\right)$ according to Theorem 3, the optimal design outperforms other schemes.

Figure 2. Comparison of the empirical average computed for the optimal design and largest eigen modes schemes for multiple reduced data dimensions r as a function of original data dimension n.

5.3. Squared Hellinger Distance

We consider a Bayesian hypothesis testing problem given class a priori parameters $p_{{\mathbb{P}}_{\mathbf{A}}}$, $p_{{\mathbb{Q}}_{\mathbf{A}}}$ and Gaussian class conditional densities for the linear dimensionality reduction model in (52). Without loss of generality, we assume a 0–1 loss function associated with misclassification for the hypothesis test. In order to quantify the performance of the Bayes decision rule, it is imperative to compute the associated probability of error, also known as the Bayes error, which we denote by $P_e$. Since, in general, computing $P_e$ for the optimal decision rule for multivariate Gaussian conditional densities is intractable, numerous techniques have been devised to bound $P_e$. Owing to its simplicity, one of the most commonly employed metric is the Bhattacharyya coefficient given by

$$\begin{array}{c}\hfill \mathsf{BC}\left(\mathbf{A}\right) \stackrel{▵}{=}{\int }_{{\mathbb{R}}^{\mathsf{r}}}\sqrt{\mathrm{d}{\mathbb{P}}_{\mathbf{A}}·\mathrm{d}{\mathbb{Q}}_{\mathbf{A}}}.\end{array}$$

(53)

The metric in (53) facilitates upper bounding the error probability as

$$\begin{array}{c}\hfill P_e\le \sqrt{p_{{\mathbb{P}}_{\mathbf{A}}} p_{{\mathbb{Q}}_{\mathbf{A}}}}·\mathsf{BC}\left(\mathbf{A}\right),\end{array}$$

(54)

which is widely referred to as the Bhattacharrya bound. Relevant to this study is that the squared Hellinger distance is related to the Bhattacharyya coefficient in (53) through

$$\begin{array}{c}\hfill {\mathsf{H}}^2\left(\mathbf{A}\right)=2-\mathsf{BC}\left(\mathbf{A}\right).\end{array}$$

(55)

Hence, maximizing the Hellinger distance ${\mathsf{H}}^2\left(\mathbf{A}\right)$ results in a tighter bound on $P_e$ from (54). To show the performance numerically, we compute the $\mathsf{BC}\left(\mathbf{A}\right)$ via (55). For the pre- and post-linear dimensionality reduction structures as given in (52), the covariance matrix $\mathbf{\Sigma }$ is randomly generated following the setup used in Section 5.1. As plotted in Figure 3, by employing the design scheme according to Theorem 4, the optimal design results in a smaller $\mathsf{BC}\left(\mathbf{A}\right)$ and, hence, a tighter upper bound on $P_e$ in comparison to other schemes.

Figure 3. Comparison of the empirical average of the Bhattacharyya coefficient $\mathsf{BC}\left(\mathbf{A}\right)$ under optimal design and largest eigen modes schemes for multiple reduced data dimensions r as a function of original data dimension n.

5.4. Total Variation Distance

Consider a binary hypothesis test with Gaussian class conditional densities following the model in (52) and equal class a priori probabilities, i.e., $p_{{\mathbb{P}}_{\mathbf{A}}}=p_{{\mathbb{Q}}_{\mathbf{A}}}$. We define $c_{ij}$ as the cost associated with deciding in favor of ${\mathsf{H}}_i$ when the true hypothesis is ${\mathsf{H}}_j$ such that $0\le i,j\le 1$, and denote the densities associated with measures ${\mathbb{P}}_{\mathbf{A}}$, ${\mathbb{Q}}_{\mathbf{A}}$ by $f_{{\mathbb{P}}_{\mathbf{A}}}$ and $f_{{\mathbb{Q}}_{\mathbf{A}}}$, respectively. Without loss of generality, we assume a 0–1 loss function such that $c_{ij}=1 \forall i\ne j$ and $c_{ii}=0 \forall i$. The optimal Bayes decision rule that minimizes the error probability is given by

$$\begin{array}{c}\hfill \frac{f_{{\mathbb{P}}_{\mathbf{A}}}\left(x\right)}{f_{{\mathbb{Q}}_{\mathbf{A}}}\left(x\right)} \stackrel{d={\mathsf{H}}_1}{\underset{d={\mathsf{H}}_0}{\lessgtr }} 1.\end{array}$$

(56)

Since the total variation distance cannot be computed in closed-form, we numerically compute the error probability $P_e$ under the two bounds (Hellinger-based and FB-based) introduced in Section 4.4.2 to quantify the performance of the design of matrix $\mathbf{A}$ for the underlying inference problem. The covariance matrix $\mathbf{\Sigma }$ is randomly generated following the setup used in Section 5.1. As plotted in Figure 4, by optimizing the Hellinger-based bound according to Theorem 4 and optimizing the FB-based bound according to Theorem 3, the two design schemes achieve a smaller $P_e$. We further observe that the bounds due to FB-based are loose in comparison to Hellinger-based bounds. Therefore, we choose not to plot the lower bound on $P_e$ for the FB-based bounds in Figure 4.

Figure 4. Comparing the logarithm of the empirical average value for $P_e$ under the two bounds on $d_{\mathsf{TV}}\left(\mathbf{A}\right)$ (Hellinger-based and Frobenius norm (FB)-based) with the largest eigen modes scheme for multiple projected data dimensions r as a function of initial data dimension n.

5.5. ${\chi }^2$-Divergence

In this section, we show the gains of the proposed analysis through numerical evaluations by numerically computing ${\chi }^2\left(\mathbf{A}\right)$, to find a lower bound on the noise variance ${\mathsf{var}}_{\theta }\left(\widehat{\theta }\right)$ up to a constant. Following the pre- and post-linear dimensionality reduction structures given in (52), the covariance matrix $\mathbf{\Sigma }$ is randomly generated following the setup used in Section 5.1. As shown in Figure 5, constructing the optimal design according to Theorem 4 achieves a tighter lower bound in comparison to the other scheme.

Figure 5. Comparison of the lower bound on noise variance given by $\frac{1}{{\chi }^2\left(\mathbf{A}\right)}$ under the optimal and largest eigen modes schemes for multiple reduced data dimensions r as a function of original data dimension n.

6. General Gaussian Models

In the previous section, we focused on $\mathit{\mu }=0$. When $\mathit{\mu }\ne 0$, optimizing each f-divergence measure under the semi-orthogonality constraint does not render closed-form expressions. Nevertheless, to provide some intuitions, we provide a numerical approach to the optimal design of $\mathbf{A}$, which might also enjoy some local optimality guarantees. To start, note that the feasible set of solutions given by ${\mathcal{M}}_n^r \stackrel{▵}{=}\{\mathbf{A}\in {\mathbb{R}}^{r\times n}:\mathbf{A}·{\mathbf{A}}^{\top }={\mathbf{I}}_r\}$ owing to the orthogonality constraints in $\mathcal{Q}$ is often referred to as the Stiefel manifold. Therefore, solving $\mathcal{Q}$ requires designing algorithms that optimize the objective while preserving manifold constraints during iterations.

We employ the method of Lagrange multipliers to formulate the Lagrangian function. By denoting the matrix of Lagrangian multipliers by $\mathbf{L}\in {\mathbb{R}}^{r\times r}$, the Lagrangian function of problem (14) is given by

$$\begin{array}{c}\hfill \mathcal{L}(\mathbf{A},\mathbf{L})=D_f\left(\mathbf{A}\right)+⟨\mathbf{L},\mathbf{A}·{\mathbf{A}}^{\top }-{\mathbf{I}}_r⟩.\end{array}$$

(57)

From the first order optimality condition, for any local maximizer ${\mathbf{A}}^{\ast }$ of (14), there exists a Lagrange multiplier ${\mathbf{L}}^{\ast }$ such that

$$\begin{array}{c}\hfill {\nabla }_{\mathbf{A}}\mathcal{L}(\mathbf{A},\mathbf{L}){|}_{{\mathbf{A}}^{\ast },{\mathbf{L}}^{\ast }}=0,\end{array}$$

(58)

where we denote the partial derivative with respect to $\mathbf{A}$ by ${\nabla }_{\mathbf{A}}$. In what follows, we iterate the design mapping $\mathbf{A}$ using the gradient ascent algorithm in order to find a solution for $\mathbf{A}$. As discussed in the next subsection, this solution is guaranteed to be at least locally optimal.

6.1. Optimizing via Gradient Ascent

We use an iterative gradient ascent-based algorithm to find the local maximizer of $D_f\left(\mathbf{A}\right)$ such that $\mathbf{A}\in {\mathcal{M}}_n^r$. The gradient ascent update at any given iteration $k\in \mathbb{N}$ is given by

$$\begin{array}{c}\hfill {\mathbf{A}}^{k+1}={\mathbf{A}}^k+\alpha ·{\nabla }_{\mathbf{A}}\mathcal{L}(\mathbf{A},\mathbf{L}){|}_{{\mathbf{A}}^k}.\end{array}$$

(59)

Note that, following this update, since the new point ${\mathbf{A}}^{k+1}$ in (59) may not satisfy the semi-orthogonality, i.e., ${\mathbf{A}}^{k+1}\notin {\mathcal{M}}_n^r$, it is imperative to establish a relation between the multipliers $\mathbf{L}$ and ${\mathbf{A}}^k$ in every iteration k to ensure a constraint-preserving update scheme. In particular, to enforce the semi-orthogonality constraint on ${\mathbf{A}}^{k+1}$, a relationship between the multipliers and the gradients in every iteration k is derived. Following a similar line of analysis for gradient descent in Reference [43], the relationship between multipliers and the gradients is provided in Appendix E. More details on the analysis of the update scheme can be found in Reference [43], and a detailed discussion on the convergence guarantees of classical steepest descent update schemes adapted to semi-orthogonality constraints can be found in Reference [44].

In order to simplify ${\nabla }_{\mathbf{A}}\mathcal{L}(\mathbf{A},\mathbf{L})$ and state the relationships, we define $\Lambda \stackrel{▵}{=}\mathbf{L}+{\mathbf{L}}^{\top }$ and subsequently find a relationship between $\Lambda$ and ${\mathbf{A}}^k$ in every iteration k. This is obtained by right-multiplying (59) by ${\mathbf{A}}^{k+1}$ and solving for $\Lambda$ that enforces the semi-orthogonality constraint on ${\mathbf{A}}^{k+1}$. To simplify the analysis, we take a finite Taylor series expansion of $\Lambda$ around $\alpha =0$ and choose $\alpha$ such that the error in forcing the constraint is a good approximation of the gradient of the objective subjected to $\mathbf{A}·{\mathbf{A}}^{\top }={\mathbf{I}}_r$. As derived in the Appendix E, by simple algebraic manipulations, it can be shown that the matrices ${\Lambda }_0,{\Lambda }_1$, and ${\Lambda }_2$, for which the finite Taylor series expansion of $\Lambda \approx {\Lambda }_0+\alpha ·{\Lambda }_1+{\alpha }^2·{\Lambda }_2$ is a good approximation of the constraint, are given by

$$\begin{array}{cc}\hfill {\Lambda }_0& \stackrel{▵}{=}-\frac{1}{2}\left[{\nabla }_{\mathbf{A}}{D}_f\left(\mathbf{A}\right)·{\left(\mathbf{A}\right)}^{\top }+\mathbf{A}·{{\nabla }_{\mathbf{A}}{D}_f\left(\mathbf{A}\right)}^{\top }\right],\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill {\Lambda }_1& \stackrel{▵}{=}-\frac{1}{2}\left[\left({\nabla }_{\mathbf{A}}{D}_f\left(\mathbf{A}\right)+{\Lambda }_0\mathbf{A}\right)·{\left({\nabla }_{\mathbf{A}}{D}_f\left(\mathbf{A}\right)+{\Lambda }_0\mathbf{A}\right)}^{\top }\right],\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill {\Lambda }_2& \stackrel{▵}{=}-\frac{1}{2}\left[{\Lambda }_1·\mathbf{A}·{{\nabla }_{\mathbf{A}}{D}_f\left(\mathbf{A}\right)}^{\top }+{\nabla }_{\mathbf{A}}{D}_f\left(\mathbf{A}\right)·{\left(\mathbf{A}\right)}^{\top }·{\Lambda }_1+{\Lambda }_0·{\Lambda }_1+{\Lambda }_1·{\Lambda }_0\right].\hfill \end{array}$$

(62)

Additionally, we note that, since finding the global maximum is not guaranteed, it is imperative to initialize ${\mathbf{A}}^0$ close to the estimated maximum. In this regard, we leverage the structure of the objective function for each f-divergence measure as given in Appendix D. In particular, we observe that the objective of each f-divergence measure can be decomposed into two objectives: the first not involving $\mathit{\mu }$ (making this objective a convex problem as shown in Section 4), and the second objective a function of $\mathit{\mu }$. Hence, leveraging the structure of the solution from Section 4, we initialize ${\mathbf{A}}^0$ such that it maximizes the objective in the case of zero-mean Gaussian models. We further note that, while there are more sophisticated orthogonality constraint-preserving algorithms [45], we find that our method adopted from Reference [43] is sufficient for our purpose, as we show next through numerical simulations.

6.2. Results and Discussion

The design of $\mathbf{A}$ when $\mathit{\mu }\ne 0$ is not characterized analytically. Therefore, we resort to numerical simulations to show the gains of optimizing f-divergence measures when $\mathit{\mu }\ne 0$. In particular, we consider the linear discriminant analysis (LDA) problem where the goal is to design a mapping $\mathbf{A}$ and perform classification in the lower dimensional space (of dimension r). Without loss of generality, we assume $n=10$ and consider Gaussian densities with the following pre- and post-linear dimensionality reduction structures:

$$\begin{array}{c}\hfill \begin{array}{cc}\mathbb{P}: \hfill & \mathcal{N}(\mathbf{0},{\mathbf{I}}_n) \mathrm{and} \mathbb{Q}: \mathcal{N}(\mathit{\mu },\mathbf{\Sigma })\hfill \\ {\mathbb{P}}_{\mathbf{A}}: \hfill & \mathcal{N}(\mathbf{0},{\mathbf{I}}_r) \mathrm{and} {\mathbb{Q}}_{\mathbf{A}}: \mathcal{N}(\mathbf{A}·\mathit{\mu },h_1\left(\mathbf{A}\right))\hfill \end{array},\end{array}$$

(63)

where the covariance matrix $\mathbf{\Sigma }$ is generated randomly the eigenvalues of which are sampled from a uniform distribution ${\{{\lambda }_i\sim \mathcal{U}(0,1)\}}_{i=1}^{10}$. For the model in (63), we consider two kinds of performance metrics that have information-theoretic performance interpretations: (i) the total probability of error related to the $d_{\mathsf{TV}}\left(\mathbf{A}\right)$, and (ii) the exponential decay of error probability related to $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$. In what follows, we demonstrate that optimizing appropriate f-divergence measures between ${\mathbb{P}}_{\mathbf{A}}$ and ${\mathbb{Q}}_{\mathbf{A}}$ lead to better performance when compared to the performance of the popular Fisher’s quadratic discriminant analysis (QDA) classifier [20]. In particular, the Fisher’s approach sets $r=1$ and designs $\mathbf{A}$ by solving

$$\begin{array}{c}\hfill \underset{\mathbf{A}\in {\mathbb{R}}^{1\times n}}{arg\; max} \frac{{(\mathit{\mu }·{\mathbf{A}}^{\top })}^2}{\mathbf{A}·({\mathbf{I}}_n+\mathbf{\Sigma })·{\mathbf{A}}^{\top }}.\end{array}$$

(64)

In contrast, we design $\mathbf{A}$ such that the information-theoretic objective functions associated with the total probability of error (captured by $d_{\mathsf{TV}}\left(\mathbf{A}\right)$) and the exponential decay of error probability (captured by $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$) are minimized. The structure of the objective functions is discussed in Total probability of error and Type-II error subjected to type-I error constraints. Both methods and Fisher’s method, after projecting the data into a lower dimension, deploy optimal detectors to discern the true model. It is noteworthy that, in both methods the data in the lower dimensions has a Gaussian model, and the conventional QDA [20] classifier is the optimal detector. Hence, we emphasize that our approach aims to have a design for $\mathbf{A}$ that maximizes the distance between the probability measures after reducing the dimensions, i.e., the distance between ${\mathbb{P}}_{\mathbf{A}}$ and ${\mathbb{Q}}_{\mathbf{A}}$. Since this distance captures the quality of the decisions, our design of $\mathbf{A}$ outperforms that of Fisher’s. For each comparison, we consider various values for $\mathit{\mu }$ and compare the appropriate performance metrics with that of Fisher’s QDA for each. In all cases, the data is synthetically generated, i.e., sampled from a Gaussian distribution where we consider 2000 data points associated with each measure $\mathbb{P}$ and $\mathbb{Q}$.

6.2.1. Schemes for Linear Map

(1) Total Probability of Error: In this scheme, the linear map $\mathbf{A}$ is designed such that $d_{\mathsf{TV}}\left(\mathbf{A}\right)$ is optimized via gradient ascent iterations until convergence. As discussed in Section 4.4.1, since the total probability of error is the key performance metric that arises while optimizing $d_{\mathsf{TV}}\left(\mathbf{A}\right)$, it is expected that optimizing $d_{\mathsf{TV}}\left(\mathbf{A}\right)$ will result in a smaller total error in comparison to other schemes that optimize other objective functions (e.g., Fisher’s QDA). We note that, since there do not exist closed-form expressions for the total variation distance, we maximize bounds on $d_{\mathsf{TV}}\left(\mathbf{A}\right)$ instead via the Hellinger bound in (33) as a proxy to minimize the total probability of error. The corresponding gradient expression to optimize ${\mathsf{H}}^2\left(\mathbf{A}\right)$ (to perform iterative updates as in (59)) is derived in closed-form and is given in Appendix D.

(2) Type-II Error Subjected to Type-I Error Constraints: In this scheme, the linear map $\mathbf{A}$ is designed such that $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ is optimized via gradient ascent iterations until convergence. In order to establish a relation, consider the following binary hypothesis test:

$$\begin{array}{c}\hfill {\mathsf{H}}_0 :X\sim {\mathbb{P}}_{\mathbf{A}} \mathrm{versus} {\mathsf{H}}_1 :X\sim {\mathbb{Q}}_{\mathbf{A}}.\end{array}$$

(65)

When minimizing the probability of type-II error subjected to type-I error constraints, the optimal test guarantees that the probability of type-II error decays exponentially as

$$\begin{array}{c}\hfill \underset{s\to \infty }{lim}\frac{-log\left({\mathbb{Q}}_{\mathbf{A}}(d={\mathsf{H}}_0)\right)}{s}=D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}}),\end{array}$$

(66)

where we have define $d:X\to \{{\mathsf{H}}_0,{\mathsf{H}}_1\}$ as the decision rule for the hypothesis test, and s denotes the sample size. As a result, $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ appears as the error exponent for hypothesis test in (65). Hence, it is expected that optimizing $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ will result in a smaller type-II error for the same type-I error when comparing with a method that optimizes other objectives (e.g., Fisher’s QDA). The corresponding gradient expression to optimize the $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ is derived in closed-form and is given in Appendix D.

For the sake of comparison and reference, we also consider schemes in which $\mathbf{A}$ is designed to optimize the objectives $D_{\mathsf{KL}}\left(\mathbf{A}\right)$, the largest eigen modes (LEM), and the smallest eigen modes (SEM), which carry no specific operational significance in the context of the binary classification problem. In the case of LEM and SEM schemes, the linear map $\mathbf{A}$ is designed such that the rows of $\mathbf{A}$ are the eigenvector associated with the largest and smallest modes of the matrix $\mathbf{\Sigma }$, respectively. Furthermore, we define $𝟙$ as the vector of all those of appropriate dimension.

6.2.2. Performance Comparison

After learning the linear map $\mathbf{A}$ for each scheme described in Section 6.2.1, we perform classification in the lower dimensional space of dimension r to find the type-I, type-II, and total probability of error for each scheme. Table 1, Table 2, Table 3 and Table 4 tabulate the results for various choices of the mean parameter $\mathit{\mu }$. We have the following important observations: (i) we observe that optimizing ${\mathsf{H}}^2\left(\mathbf{A}\right)$ results in a smaller total probability of error in comparison to the total error obtained by optimizing the Fisher’s objective; it is important to note that the superior performance is observed despite maximizing bounds on $d_{\mathsf{TV}}\left(\mathbf{A}\right)$ (that is sub-optimal) and not the distance itself; and (ii) we observe that except for the case of $\mathit{\mu }=0.8·𝟙$, optimizing $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ results in a smaller type-II error in comparison to that obtained by optimizing the Fisher’s objective indicating a gain in optimizing $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ in comparison to the Fisher’s objective in (64).

Table 1. $\mathit{\mu }=0.2·𝟙,r=1$.

	Fisher’s QDA	${\mathit{D}}_{\mathsf{KL}}({\mathbb{P}}_{\mathit{A}} \parallel {\mathbb{Q}}_{\mathit{A}})$	${\mathsf{H}}^2\left(\mathit{A}\right)$	${\mathit{D}}_{\mathsf{KL}}\left(\mathit{A}\right)$	SEM	LEM
${\mathbb{P}}_{\mathbf{A}}(d={\mathsf{H}}_1)$	331/2000	331/2000	331/2000	331/2000	337/2000	915/2000
${\mathbb{Q}}_{\mathbf{A}}(d={\mathsf{H}}_0)$	1226/2000	63/2000	63/2000	63/2000	64/2000	811/2000
Total Error	1557/4000	394/4000	394/4000	394/4000	401/4000	1726/4000

Table 2. $\mathit{\mu }=0.4·𝟙,r=1$.

	Fisher’s QDA	${\mathit{D}}_{\mathsf{KL}}({\mathbb{P}}_{\mathit{A}} \parallel {\mathbb{Q}}_{\mathit{A}})$	${\mathsf{H}}^2\left(\mathit{A}\right)$	${\mathit{D}}_{\mathsf{KL}}\left(\mathit{A}\right)$	SEM	LEM
${\mathbb{P}}_{\mathbf{A}}(d={\mathsf{H}}_1)$	344/2000	344/2000	344/2000	345/2000	347/2000	782/2000
${\mathbb{Q}}_{\mathbf{A}}(d={\mathsf{H}}_0)$	594/2000	63/2000	63/2000	63/2000	64/2000	739/2000
Total Error	938/4000	407/4000	407/4000	408/4000	411/4000	1521/4000

Table 3. $\mathit{\mu }=0.6·𝟙,r=1$.

	Fisher’s QDA	${\mathit{D}}_{\mathsf{KL}}({\mathbb{P}}_{\mathit{A}} \parallel {\mathbb{Q}}_{\mathit{A}})$	${\mathsf{H}}^2\left(\mathit{A}\right)$	${\mathit{D}}_{\mathsf{KL}}\left(\mathit{A}\right)$	SEM	LEM
${\mathbb{P}}_{\mathbf{A}}(d={\mathsf{H}}_1)$	326/2000	326/2000	335/2000	318/2000	335/2000	638/2000
${\mathbb{Q}}_{\mathbf{A}}(d={\mathsf{H}}_0)$	137/2000	55/2000	108/2000	57/2000	61/2000	669/2000
Total Error	463/4000	381/4000	443/4000	375/4000	396/4000	1307/4000

Table 4. $\mathit{\mu }=0.8·𝟙,r=1$.

	Fisher’s QDA	${\mathit{D}}_{\mathsf{KL}}({\mathbb{P}}_{\mathit{A}} \parallel {\mathbb{Q}}_{\mathit{A}})$	${\mathsf{H}}^2\left(\mathit{A}\right)$	${\mathit{D}}_{\mathsf{KL}}\left(\mathit{A}\right)$	SEM	LEM
${\mathbb{P}}_{\mathbf{A}}(d={\mathsf{H}}_1)$	264/2000	264/2000	159/2000	255/2000	307/2000	561/2000
${\mathbb{Q}}_{\mathbf{A}}(d={\mathsf{H}}_0)$	25/2000	53/2000	64/2000	55/2000	60/2000	580/2000
Total Error	289/4000	317/4000	214/4000	310/4000	367/4000	1141/4000

It is important to note that the convergence of the gradient ascent algorithm only guarantees a locally optimal solution. While we have restricted the results that consider a maximum separation of $\mathit{\mu }=0.8·𝟙$, we have performed additional simulations for larger separation between models (greater $\mathit{\mu }>0.8$). We have the following observations: (i) solution for the linear map $\mathbf{A}$ obtained through gradient ascent becomes highly sensitive to the initialization ${\mathbf{A}}^0$; specifically, it was observed that optimizing the Fisher’s objective outperforms optimizing ${\mathsf{H}}^2\left(\mathbf{A}\right)$ for various initializations ${\mathbf{A}}^0$, and vice versa, for other random initializations; and (ii) the gradient ascent solver becomes more prone to getting stuck at the local maxima for larger separations between the models. We conjecture that the odd observation in the case of $\mathit{\mu }=0.8·𝟙$ when optimizing $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ (where optimizing the Fisher’s objective outperforms optimizing $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$) supports this observation. Furthermore, we note that, since the problem is convex for $\mathit{\mu }=\mathbf{0}$, a deviation from this assumption moves the problem further from being convex, making the solver prone to getting stuck at the locally optimal solutions for larger separation between the Gaussian models.

6.2.3. Subspace Representation

In order to gain more intuition towards the learned representations, we illustrate the 2-dimensional projections of the original 10-dimensional data obtained after optimizing the corresponding f-divergence measures. For brevity, we only show the plots for $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ and ${\mathsf{H}}^2\left(\mathbf{A}\right)$. Figure 6 and Figure 7 plot the two-dimensional projections of the synthetic dataset that optimize $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$ and ${\mathsf{H}}^2\left(\mathbf{A}\right)$, respectively. As expected, it is observed that the total probability of error is smaller when optimizing ${\mathsf{H}}^2\left(\mathbf{A}\right)$. Figure 8 shows the variation in the objective function as a function of gradient ascent iterations. As the iterations grow, the objective functions eventually converges to a locally optimal solution.

Figure 6. Two-dimensional projected data obtained by optimizing $D_{\mathsf{KL}}({\mathbb{P}}_{\mathbf{A}} \parallel {\mathbb{Q}}_{\mathbf{A}})$.

Figure 7. Two-dimensional projected data obtained by optimizing ${\mathsf{H}}^2\left(\mathbf{A}\right)$.

Figure 8. Convergence of the gradient ascent algorithm as a result of optimizing ${\mathsf{H}}^2\left(\mathbf{A}\right)$.

7. Conclusions

In this paper, we have considered the problem of discriminant analysis such that separation between the classes is maximized under f-divergence measures. This approach is motivated by dimensionality reduction for inference problems, where we have investigated discriminant analysis under Kullback–Leibler, symmetrized Kullback–Leibler, Hellinger, ${\chi }^2$, and total variation measures. We have characterized the optimal design for the linear transformation of the data onto a lower-dimensional subspace for each in the case of zero-mean Gaussian models and adopted numerical algorithms to find the design of the linear transformation in the case of general Gaussian models with non-zero means. We have shown that, in the case of zero-mean Gaussian models, the row space of the mapping matrix lies in the eigenspace of a matrix associated with the covariance matrix of the Gaussian models involved. While each f-divergence measure favors specific eigenvector components, we have shown that all the designs become identical in certain regimes, making the design of the linear mapping independent of the inference problem of interest.