Extensions of Some Statistical Concepts to the Complex Domain

Abstract

This paper deals with the extension of principal component analysis, canonical correlation analysis, the Cramer–Rao inequality, and a few other statistical concepts in the real domain to the corresponding complex domain. Optimizations of Hermitian forms under a linear constraint, a bilinear form under Hermitian-form constraints, and similar maxima/minima problems in the complex domain are discussed. Some vector/matrix differential operators are developed to handle the above types of problems. These operators in the complex domain and the optimization problems in the complex domain are believed to be new and novel. These operators will also be useful in maximum likelihood estimation problems, which will be illustrated in the concluding remarks. Detailed steps are given in the derivations so that the methods are easily accessible to everyone.

1. Introduction

In most textbooks on statistics, scalar/vector/matrix-variate random variables in the complex domain and the corresponding statistical analysis in the complex domain, are not discussed. But, in a large number of physical situations, it is natural or more convenient to represent variables in the complex domain. Hence, statistical techniques in the complex domain are required for data analysis in these situations. In the physical science and engineering literature, there are a number of papers dealing with random variables in the complex domain. Data reduction techniques such as principal component analysis in the complex domain seem to be the main topic in these areas. Most of the papers in these applied areas, concentrate on developing algorithms for computing eigenvalues and eigenvectors, which are useful and relevant in principal component analysis, independent component analysis, factor analysis, and so on. Statistical analysis in the complex domain is widely used in the analysis of multi-look return signals in radar [1], in multi-task learning in artificial intelligence and machine learning [2], in problems such as signal processing [3], in principal component analysis and independent component analysis in analyzing meteorological data in the complex domain [4], in optimal allocation of resources, especially energy resources [5], in holography, microscopy and optical metrology [6], in delayed mixing in speech processing, in biomedical signal analysis, and in financial data modeling, etc. [7].

In the present paper, vector/matrix differential operators in the complex domain are defined. Then, these are applied in optimizing a Hermitian form under a linear constraint, a bilinear form under Hermitian-form constraints, etc. Then, as applications of these optimization problems, the real-domain techniques of principal component analysis and canonical correlation analysis are extended to the complex domain. Other statistical concepts such as the Cramer–Rao inequality, least square procedure, and related aspects are extended to the complex domain. Detailed derivations are given so that the methods will be accessible even to beginners.

The following notation is used in this paper: Scalar variables, whether mathematical or random, are denoted by lower-case letters such as $x,y$. Vector/matrix variables are denoted by capital letters such as $X,Y$. Scalar constants are denoted by $a,b$, etc., and vector, matrix constants by $A,B$, etc. The wedge product of the differentials $\mathrm{d}x$ and $\mathrm{d}y$ is defined as $\mathrm{d}x\wedge \mathrm{d}y=-\mathrm{d}y\wedge \mathrm{d}x$, where x and y are two real scalar variables, so that $\mathrm{d}x\wedge \mathrm{d}x=0,\mathrm{d}y\wedge \mathrm{d}y=0$. Let $X=\left(x_{jk}\right)$ be an $m\times n$ matrix with distinct real scalar variables $x_{jk}$ as elements, then $\mathrm{d}X={\wedge }_{j=1}^m{\wedge }_{k=1}^n\mathrm{d}{x}_{jk}$. The transpose of a matrix X is denoted by a prime as $X^{\prime }$. For a $p\times p$ matrix X, if $X=X^{\prime }$ (symmetric), then $\mathrm{d}X={\wedge }_{j\ge k}\mathrm{d}{x}_{jk}={\wedge }_{j\le k}\mathrm{d}{x}_{jk}$. Variables in the complex domain are written with a tilde, such as $\tilde{x},\tilde{y},\tilde{X},\tilde{Y}$. If $\tilde{X}$ is a $p\times q$ matrix in the complex domain, then $\tilde{X}=X_1+i{X}_2,i=\sqrt{(-1)},X_1,X_2$ are real, then $\mathrm{d}\tilde{X}$ is defined as $\mathrm{d}\tilde{X}=\mathrm{d}{X}_1\wedge \mathrm{d}{X}_2$. The determinant of a real $p\times p$ matrix Y is written as $\left|Y\right|$ or as $\det \left(Y\right)$ and if $\tilde{Y}$ is in the complex domain, then the absolute value of the determinant of $\tilde{Y}$ is written as $\left|\det \right(Y\left)\right|$. Also, $\mathrm{tr}\left(Y\right)$ means the trace of the square matrix Y. A $p\times p$ real matrix A being positive definite is written as $A>O$, and $\tilde{X}={\tilde{X}}^{*}>O$ indicates that the matrix $\tilde{X}$, in the complex domain, is Hermitian positive definite. Other notation is explained when it occurs for the first time.

This paper is organized as follows: Section 2 starts with defining a vector differential operator in the complex domain. Then, with the help of this operator, some optimization problems such as optimizing a linear form subject to Hermitian-form constraint, a Hermitian form under Hermitian-form constraint, and a bilinear form with Hermitian-form constraints are discussed. Then, a matrix differential operator in the complex domain is defined. Differentiations of the trace of a product of matrices and determinant of a matrix, by using the matrix differential operator in the complex domain, are dealt with. Section 3 deals with the extension of principal component analysis to the complex domain. Section 4 delves into the extension of canonical correlation analysis to the complex domain. Section 5 examines the extension of the Cramer–Rao inequality, Cauchy–Schwarz inequality, and the least square estimation procedure to the complex domain. Detailed steps are given in each situation.

2. Optimization Involving Linear Forms, Traces, and Determinants

Here, we will consider linear forms and their differentiations first, and then, we will consider some situations of optimizing a Hermitian form with linear constraint and optimization of a linear form with Hermitian-form constraint. We consider some basic results to start with. Let

$$A=\left[\begin{array}{c}{a}_1\\ ⋮\\ a_p\end{array}\right],\tilde{X}=\left[\begin{array}{c}{\tilde{x}}_1\\ ⋮\\ {\tilde{x}}_p\end{array}\right],A=A_1+i{A}_2,i=\sqrt{(-1)},\tilde{X}=X_1+i{X}_2$$

where $A_1,A_2,X_1,X_2$ are real vectors. Assume that A is a non-null arbitrary coefficient vector and $\tilde{X}$ is a known vector random variable in the complex domain.

Theorem 1.

For A and $\tilde{X}$ as defined above,

$$\tilde{u}=A^{*}\tilde{X}\Rightarrow \frac{\partial \tilde{u}}{\partial A}=2\tilde{X}.$$

(1)

Proof. 

This can be easily seen from the following: $\tilde{u}=(A_1^{\prime }-i{A}_2^{\prime })(X_1+i{X}_2)=A_1^{\prime }{X}_1+A_2^{\prime }{X}_2+i(A_1^{\prime }{X}_2-A_2^{\prime }{X}_1)$ Then,

$$\frac{\partial \tilde{u}}{\partial A_1}=X_1+i{X}_2 \mathrm{and} \frac{\partial \tilde{u}}{\partial A_2}=X_2-i{X}_1$$

(2)

from the corresponding real variable case. That is,

$$(\frac{\partial }{\partial A_1}+i\frac{\partial }{\partial A_2})\tilde{u}=\frac{\partial \tilde{u}}{\partial A}=2(X_1+i{X}_2)=2\tilde{X}.$$

□

Now, we consider a slightly more general result. Consider $\tilde{u}=A^{*}{\mathrm{\Sigma }}_{11}\tilde{X},{\mathrm{\Sigma }}_{11}={\mathrm{\Sigma }}_{11}^{*}>O$. Let $\frac{\partial }{\partial A^c}=(\frac{\partial }{\partial A_1}-i\frac{\partial }{\partial A_2})$ and $\frac{\partial }{\partial A^{*}}=(\frac{\partial }{\partial A_1^{\prime }}-i\frac{\partial }{\partial A_2^{\prime }})$. Then, from $\left(2\right)$ note that $\frac{\partial \tilde{u}}{\partial A^c}=O$ and $\frac{\partial \tilde{u}}{\partial A^{*}}=O;$$\frac{\partial \tilde{u}}{\partial \tilde{X}}=O$; $\frac{\partial \tilde{u}}{\partial {\tilde{X}}^c}=2A$ and $\frac{\partial \tilde{u}}{\partial {\tilde{X}}^{*}}=2A^{\prime }$, where $A^c$ means the complex conjugate of A.

Theorem 2.

For $\tilde{X}$ and A as defined above, let $\tilde{u}=A^{*}{\mathrm{\Sigma }}_{11}\tilde{X}$. Let ${\mathrm{\Sigma }}_{11}$ be a $p\times p$ matrix, free of the elements in A and $\tilde{X}$. Then,

$$\frac{\partial \tilde{u}}{\partial A}=2{\mathrm{\Sigma }}_{11}\tilde{X}.$$

(3)

Proof. 

Since ${\mathrm{\Sigma }}_{11}$ and $\tilde{X}$ are free of the elements in A, we may take ${\mathrm{\Sigma }}_{11}\tilde{X}=B=B_1+i{B}_2$, $i=\sqrt{(-1)},B_1,B_2$ real. Then, from Theorem 1 the result follows. □

Now, consider a $p\times 1$ vector A and a $p\times p$ matrix $\mathrm{\Sigma }$ and the Hermitian form $\tilde{u}=A^{*}\mathrm{\Sigma }A$ in the complex domain where $\mathrm{\Sigma }={\mathrm{\Sigma }}^{*}>O$ is free of the elements in A. Then, we have the following result:

Theorem 3.

For the Hermitian form as defined above, where $A=A_1+i{A}_2,\mathrm{\Sigma }={\mathrm{\Sigma }}_1+i{\mathrm{\Sigma }}_2,$$\mathrm{\Sigma }={\mathrm{\Sigma }}^{*}>O,A_1,A_2,{\mathrm{\Sigma }}_1,{\mathrm{\Sigma }}_2$ are real,

$$\frac{\partial }{\partial A}\left[\tilde{u}\right]=\frac{\partial }{\partial A}\left[A^{*}\mathrm{\Sigma }A\right]=2\mathrm{\Sigma }A.$$

(4)

Proof. 

Opening up $\tilde{u}$, we have the following:

$$\tilde{u}=A^{*}\mathrm{\Sigma }A=(A_1^{\prime }-i{A}_2^{\prime })\mathrm{\Sigma }(A_1+i{A}_2)=A_1^{\prime }\mathrm{\Sigma }{A}_1+A_2^{\prime }\mathrm{\Sigma }{A}_2+i(A_1^{\prime }\mathrm{\Sigma }{A}_2-A_2^{\prime }\mathrm{\Sigma }{A}_1).$$

(5)

Since $\mathrm{\Sigma }={\mathrm{\Sigma }}^{*}$, we have ${\mathrm{\Sigma }}_1={\mathrm{\Sigma }}_1^{\prime }$ (real symmetric), ${\mathrm{\Sigma }}_2=-{\mathrm{\Sigma }}_2^{\prime }$ (real skew symmetric). $A_1^{\prime }\mathrm{\Sigma }{A}_1=A_1^{\prime }{\mathrm{\Sigma }}_1{A}_1+i{A}_1^{\prime }{\mathrm{\Sigma }}_2{A}_1=A_1^{\prime }{\mathrm{\Sigma }}_1{A}_1$ because $A_1^{\prime }{\mathrm{\Sigma }}_2{A}_1=0$ due to ${\mathrm{\Sigma }}_2$ being real skew symmetric. Similarly, $A_2^{\prime }\mathrm{\Sigma }{A}_2=A_2^{\prime }{\mathrm{\Sigma }}_1{A}_2$. Then, from the results in the real case, we have the following:

$$\frac{\partial }{\partial A_1}\left[A_1^{\prime }{\mathrm{\Sigma }}_1{A}_1\right]=2{\mathrm{\Sigma }}_1{A}_1,\frac{\partial }{\partial A_2}\left[A_2^{\prime }{\mathrm{\Sigma }}_1{A}_2\right]=2{\mathrm{\Sigma }}_1{A}_2.$$

(6)

Consider

$$i(A_1^{\prime }\mathrm{\Sigma }{A}_2-A_2^{\prime }\mathrm{\Sigma }{A}_1)=i[(A_1^{\prime }{\mathrm{\Sigma }}_1{A}_2-A_2^{\prime }{\mathrm{\Sigma }}_1{A}_1)+i(A_1^{\prime }{\mathrm{\Sigma }}_2{A}_2-A_2^{\prime }{\mathrm{\Sigma }}_2{A}_1)].$$

But, ${\mathrm{\Sigma }}_1={\mathrm{\Sigma }}_1^{\prime }$ and ${\left(A_2^{\prime }{\mathrm{\Sigma }}_1{A}_1\right)}^{\prime }=A_1^{\prime }{\mathrm{\Sigma }}_1{A}_2$ and real $1\times 1$, and hence, $A_1^{\prime }{\mathrm{\Sigma }}_1{A}_2-A_2^{\prime }{\mathrm{\Sigma }}_1{A}_1=0$. Also, $A_2^{\prime }{\mathrm{\Sigma }}_2{A}_1=-A_1^{\prime }{\mathrm{\Sigma }}_2{A}_2$ because ${\mathrm{\Sigma }}_2=-{\mathrm{\Sigma }}_2^{\prime }$, and hence, $A_1^{\prime }{\mathrm{\Sigma }}_2{A}_2-A_2^{\prime }{\mathrm{\Sigma }}_2{A}_1=2A_1^{\prime }{\mathrm{\Sigma }}_2{A}_2=-2A_2^{\prime }{\mathrm{\Sigma }}_2{A}_1$. Therefore,

$$i(A_1^{\prime }\mathrm{\Sigma }{A}_2-A_2^{\prime }\mathrm{\Sigma }{A}_1)=i^{2}2A_1^{\prime }{\mathrm{\Sigma }}_2{A}_2=-i^{2}2A_2^{\prime }{\mathrm{\Sigma }}_2{A}_1.$$

Then, from the real case,

$$\begin{array}{cc}\hfill \frac{\partial }{\partial A_1}\left[i^{2}2A_1^{\prime }{\mathrm{\Sigma }}_2{A}_2\right]& =2i^2{\mathrm{\Sigma }}_2{A}_2=2\left(i{\mathrm{\Sigma }}_2\right)\left(i{A}_2\right)\Rightarrow \hfill \\ \hfill \frac{\partial }{\partial A_1}\left(A^{*}\mathrm{\Sigma }A\right)& =2{\mathrm{\Sigma }}_1{A}_1+2\left(i{\mathrm{\Sigma }}_2\right)\left(i{A}_2\right)\hfill \\ \hfill \frac{\partial }{\partial A_2}\left(-2i^2{A}_2^{\prime }{\mathrm{\Sigma }}_2{A}_1\right)& =-2i^2{\mathrm{\Sigma }}_2{A}_1=2{\mathrm{\Sigma }}_2{A}_1\Rightarrow \hfill \\ \hfill \frac{\partial }{\partial A_2}\left(A^{*}\mathrm{\Sigma }A\right)& =2{\mathrm{\Sigma }}_1{A}_2+2{\mathrm{\Sigma }}_2{A}_1\hfill \end{array}$$

(7)

from the corresponding real case. Then, from (2)–(5),

$$\begin{array}{cc}\hfill \frac{\partial }{\partial A}\left[A^{*}\mathrm{\Sigma }A\right]& =(\frac{\partial }{\partial A_1}+i\frac{\partial }{\partial A_2})\left[A^{*}\mathrm{\Sigma }A\right]\hfill \\ \hfill & =[2{\mathrm{\Sigma }}_1{A}_1+2\left(i{\mathrm{\Sigma }}_2\right)\left(i{A}_2\right)]+i[2{\mathrm{\Sigma }}_1{A}_2+2{\mathrm{\Sigma }}_2{A}_1]\hfill \\ \hfill & =2{\mathrm{\Sigma }}_1{A}_1+i2{\mathrm{\Sigma }}_1{A}_2+2i{\mathrm{\Sigma }}_2(A_1+i{A}_2)=2{\mathrm{\Sigma }}_{1}A+2i{\mathrm{\Sigma }}_{2}A=2\mathrm{\Sigma }A.\hfill \end{array}$$

This establishes the result. □

2.1. Optimization of a Linear Form Subject to Hermitian-Form Constraint

Consider a linear form $\tilde{u}=A^{*}\tilde{X}$, where A and $\tilde{X}$ are $p\times 1$ non-null vectors, with A being arbitrary, and $\tilde{X}$ being a known vector variable with variance $\mathrm{Var}\left(\tilde{u}\right)=A^{*}{\mathrm{\Sigma }}_{11}A$, where ${\mathrm{\Sigma }}_{11}={\mathrm{\Sigma }}_{11}^{*}>O$ is the covariance matrix in $\tilde{X}$. Consider the optimization of $\tilde{u}$, subject to the constraint that the variance of $\tilde{u}$ is fixed, say unity, that is, $A^{*}{\mathrm{\Sigma }}_{11}A=1$. Then, we have the following result:

Theorem 4.

For the linear form $\tilde{u}=A^{*}\tilde{X}$ and the constraint $A^{*}{\mathrm{\Sigma }}_{11}A=1$, as defined above,

$$\underset{A^{*}{\mathrm{\Sigma }}_{11}A=1}{max}\left[A^{*}\tilde{X}\right]=\sqrt{{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}}.$$

(8)

Proof. 

Let $\tilde{w}=A^{*}\tilde{X}-\lambda (A^{*}{\mathrm{\Sigma }}_{11}A-1)$, where $\lambda$ is a Lagrangian multiplier. Observe that ${\mathrm{\Sigma }}_{11}={\mathrm{\Sigma }}_{11}^{*}$ and we assume that it is Hermitian positive definite. From previous results (1) and (2),

$$\frac{\partial \tilde{w}}{\partial A}=2\tilde{X}-\lambda 2{\mathrm{\Sigma }}_{11}A=O\Rightarrow \tilde{X}=\lambda {\mathrm{\Sigma }}_{11}A\Rightarrow \lambda =A^{*}\tilde{X}.$$

(9)

That is,

$$A=\frac{1}{\lambda }{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}\Rightarrow A^{*}=\frac{1}{{\lambda }^c}{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\Rightarrow \lambda =A^{*}\tilde{X}=\frac{1}{{\lambda }^c}{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}.$$

(10)

But from $\left(9\right)$, $\lambda =A^{*}\tilde{X}$ which means that the maximum of our linear form is the largest $\lambda$ and the minimum of our linear form is the smallest $\lambda$. But, from $\left(9\right)$ and $\left(10\right)$, $\lambda {\lambda }^c={\left|\lambda \right|}^2={\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}$, where $\left|\lambda \right|$ means the absolute value of $\lambda$ and ${\lambda }^c$ is the complex conjugate of $\lambda$.

$$\underset{A^{*}{\mathrm{\Sigma }}_{11}A=1}{max}\left[\tilde{u}\right]=\sqrt{{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}},{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}>0$$

being positive definite Hermitian form. This completes the proof. □

2.2. Optimization of a Hermitian Form Subject to Linear Constraint

Now, consider a problem of optimizing $A^{*}{\mathrm{\Sigma }}_{11}A$ subject to $A^{*}\tilde{X}=\alpha$ fixed, where A and $\tilde{X}$ are $p\times 1$ vectors and ${\mathrm{\Sigma }}_{11}={\mathrm{\Sigma }}_{11}^{*}>O$ is a $p\times p$ Hermitian positive definite matrix, free of the elements of A. Then, we have the following result:

Theorem 5.

For the Hermitian form $A^{*}{\mathrm{\Sigma }}_{11}A$ and linear form constraint $A^{*}\tilde{X}=\alpha$, we have

$$\underset{A^{*}\tilde{X}=\alpha }{max}\left[A^{*}{\mathrm{\Sigma }}_{11}A\right]=+\infty ,\underset{A^{*}\tilde{X}=\alpha }{min}\left[A^{*}{\mathrm{\Sigma }}_{11}A\right]=\frac{{\left|\alpha \right|}^2}{{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}}.$$

(11)

Proof. 

Let $\lambda$ be a Lagrangian multiplier and consider $\tilde{w}=A^{*}{\mathrm{\Sigma }}_{11}A-\lambda (A^{*}\tilde{X}-\alpha )$. From (1) and (2), we have

$$\frac{\partial \tilde{w}}{\partial A}=2{\mathrm{\Sigma }}_{11}A-2\lambda \tilde{X}\Rightarrow {\mathrm{\Sigma }}_{11}A=\lambda \tilde{X}\Rightarrow A^{*}{\mathrm{\Sigma }}_{11}A=\lambda A^{*}\tilde{X}=\lambda \alpha .$$

(12)

From $\left(12\right)$,

$$A=\lambda {\mathrm{\Sigma }}_{11}^{-1}\tilde{X}\Rightarrow \alpha =A^{*}\tilde{X}={\lambda }^c{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}\Rightarrow \left|\lambda \right|=\frac{\left|\alpha \right|}{{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}}$$

(13)

because ${\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}$ is Hermitian positive definite, where $\left|\lambda \right|$ and $\left|\alpha \right|$ denote the absolute values of $\lambda$ and $\alpha$, respectively. Then, $max\left[{\tilde{X}}^{*}A\tilde{X}\right]=+\infty$ since A is arbitrary and since the linear restriction cannot eliminate the effect of A fully. Hence, we look for the minimum. From $\left(13\right)$,

$$\left|\lambda \right|=\frac{\left|\alpha \right|}{{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}}\Rightarrow \left|\lambda \alpha \right|=\frac{{\left|\alpha \right|}^2}{{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}}.$$

Hence,

$$\underset{A^{*}\tilde{X}=1}{min}\left[A^{*}{\mathrm{\Sigma }}_{11}A\right]=\frac{{\left|\alpha \right|}^2}{{\tilde{X}}^{*}{\mathrm{\Sigma }}_{11}^{-1}\tilde{X}}.$$

This completes the proof. □

2.3. Differentiation of Traces of Matrix Products and Matrix Differential Operator

Now, we consider a few matrix-variate cases, where also the problem is essentially optimization involving vector variable situations. Let $A=\left(a_{jk}\right)$ be a $p\times q$ matrix of arbitrary elements $a_{jk}$. Let $\tilde{X}=\left({\tilde{x}}_{jk}\right)$ be a $p\times q$ matrix in the complex domain with distinct scalar complex variables ${\tilde{x}}_{jk}$ as elements. Consider $\tilde{u}=\mathrm{tr}\left(A^{*}\tilde{X}\right)$. Let $A_j$ and ${\tilde{X}}_j$ be the j-th columns of A and $\tilde{X}$, respectively. Then, $\tilde{u}={\sum }_{j=1}^q{A}_j^{*}{\tilde{X}}_j$. Let $A_j=A_{j1}+i{A}_{j2},$$i=\sqrt{(-1)},A_{j1},A_{j2}$ are real $p\times 1$ vectors. Let ${\tilde{u}}_j=A_j^{*}{\tilde{X}}_j$. Consider the operator

$$\frac{\partial }{\partial A_j}=(\frac{\partial }{\partial A_{j1}}+i\frac{\partial }{\partial A_{j2}}),\frac{\partial }{\partial A_j^{*}}=(\frac{\partial }{\partial A_{j1}^{\prime }}-i\frac{\partial }{\partial A_{j2}^{\prime }}),\frac{\partial }{\partial A_j^c}=(\frac{\partial }{\partial A_{j1}}-i\frac{\partial }{\partial A_{j2}})$$

and then, $\frac{\partial {\tilde{u}}_j}{\partial A_j}=\frac{\partial \tilde{u}}{\partial A_j}$ since we are taking partial derivatives. Then, we have the following result:

Theorem 6.

Let $\tilde{u},A,A_j,{\tilde{X}}_j$ be as defined above. Then,

$$\frac{\partial \tilde{u}}{\partial A}=\frac{\partial }{\partial A}\left[\mathrm{tr}\left(A^{*}\tilde{X}\right)\right]=2\tilde{X}.$$

(14)

Proof. 

From the previous result,

$$\begin{array}{cc}\hfill \frac{\partial {\tilde{u}}_j}{\partial A_j}& =(\frac{\partial }{\partial A_{j1}}+i\frac{\partial }{\partial A_{j2}}){\tilde{u}}_j\hfill \\ \hfill & =(\frac{\partial }{\partial A_{j1}}+i\frac{\partial }{\partial A_{j2}})\left(A_j^{*}{\tilde{X}}_j\right)=2{\tilde{X}}_j,j=1,...,q,\hfill \end{array}$$

from Theorem 1. But, $\frac{\partial {\tilde{u}}_j}{\partial A_j}=\frac{\partial \tilde{u}}{\partial A_j}$. Now, stack up these side by side as q columns for $j=1,...,q$. Note that $A=[A_1,...,A_q],\frac{\partial }{\partial A}=[\frac{\partial }{\partial A_1},...,\frac{\partial }{\partial A_q}]$, and $\frac{\partial \tilde{u}}{\partial A_j}=2{\tilde{X}}_j$, and hence, $\frac{\partial \tilde{u}}{\partial A}=2\tilde{X}$. This completes the proof. □

For the next result to be established, we need a result on matrices, which will be stated here as a lemma.

Lemma 1.

For two $p\times p$ real matrices A and B, where $A=A^{\prime }$ (symmetric) and $B=-B^{\prime }$ (skew symmetric), $\mathrm{tr}\left(A^{\prime }B\right)=0$.

Proof. 

For two $p\times p$ matrices $A_1$ and $B_1$, it is well known that $\mathrm{tr}\left(A_1\right)=\mathrm{tr}\left(A_1^{\prime }\right)$ and $\mathrm{tr}\left(A_1{B}_1\right)=\mathrm{tr}\left(B_1{A}_1\right)$. Then, for our matrices A and B, $A=A^{\prime },B=-B^{\prime }$, we have $\mathrm{tr}\left(A^{\prime }B\right)=\mathrm{tr}{\left(A^{\prime }B\right)}^{\prime }=\mathrm{tr}\left(B^{\prime }A\right)=-\mathrm{tr}\left(BA\right)=-\mathrm{tr}\left(AB\right)=-\mathrm{tr}\left(A^{\prime }B\right)$. That is, $\mathrm{tr}\left(A^{\prime }B\right)=-\mathrm{tr}\left(A^{\prime }B\right)\Rightarrow \mathrm{tr}\left(A^{\prime }B\right)=0\Rightarrow \mathrm{tr}\left(AB\right)=0,\mathrm{tr}\left(A{B}^{\prime }\right)=0$. □

For the next problem, let $A=A^{*}$ be a $p\times p$ Hermitian matrix of arbitrary elements $a_{jk}$. Let $\tilde{X}={\tilde{X}}^{*}$ be a Hermitian matrix in the complex domain with distinct complex scalar variables as elements except for the Hermitian property. Let $A=A_1+i{A}_2,\tilde{X}=X_1+i{X}_2$, where $A_1,A_2,X_1,X_2$ are real. Then, $A_1=A_1^{\prime },X_1=X_1^{\prime }$ (symmetric), $A_2=-A_2^{\prime },X_2=-X_2^{\prime }$ (skew symmetric). Let $\tilde{u}=\mathrm{tr}\left(A^{*}\tilde{X}\right)$. Then, we have the following result:

Theorem 7.

Let $A,\tilde{X},A,A_1,A_2,X_1,X_2$, and $\tilde{u}=\mathrm{tr}\left(A^{*}\tilde{X}\right)$ be as defined above. Then,

$$\frac{\partial \tilde{u}}{\partial A}=2\tilde{X}-\mathrm{diag}\left(\tilde{X}\right)$$

where $\mathrm{diag}\left(\tilde{X}\right)$ means a diagonal matrix consisting of only the diagonal elements from $\tilde{X}$.

Proof. 

Opening up $\tilde{u}$, we have the following:

$$\begin{array}{cc}\hfill \tilde{u}& =\mathrm{tr}\left(A^{*}\tilde{X}\right)=\mathrm{tr}\left\{(A_1^{\prime }-i{A}_2^{\prime })(X_1+i{X}_2)\right\}\hfill \\ \hfill & =\mathrm{tr}\{A_1^{\prime }{X}_1+A_2^{\prime }{X}_2+i(A_1^{\prime }{X}_2-A_2^{\prime }{X}_1)\}\hfill \\ \hfill & =\mathrm{tr}(A_1^{\prime }{X}_1+A_2^{\prime }{X}_2)\hfill \end{array}$$

since $\mathrm{tr}\left(A_1^{\prime }{X}_2\right)=0,\mathrm{tr}\left(A_2^{\prime }{X}_1\right)=0$ by Lemma 1. Now, from the known result for symmetric matrices in the real case, we have

$$\frac{\partial }{\partial A_1}\left[\mathrm{tr}\left(A_1{X}_1\right)\right]=2X_1-\mathrm{diag}\left(X_1\right).$$

(15)

If the $(j,k)$-th element, for $j\ne k$, in $A_2$ is $+a_{jk}$, then the $(j,k)$-th element in $A_2^{\prime }$ is $-a_{jk}$ and the $(k,j)$-th element in $A_2^{\prime }$ is $a_{jk}$. Hence, $\mathrm{tr}\left(A_2^{\prime }{X}_2\right)=2{\sum }_{j<k}{a}_{jk}{x}_{jk}={\sum }_{j>k}{a}_{jk}{x}_{jk}$. Hence,

$$\frac{\partial }{\partial A_2}\left[\mathrm{tr}\left(A_2^{\prime }{X}_2\right)\right]=2X_2$$

(16)

since the diagonal elements in $A_2$ and $X_2$ are already zeros. Combining $\left(15\right)$ and $\left(16\right)$, we have

$$\frac{\partial }{\partial A}\left[\mathrm{tr}\left(A^{*}\tilde{X}\right)\right]=2\tilde{X}-\mathrm{diag}\left(\tilde{X}\right)$$

which corresponds to the result in the real case. □

2.4. Differentiation of a Determinant in the Complex Domain

Here, we will start with defining the derivative of a scalar quantity with respect to a matrix or we will define a matrix differential operator first. Let $X=\left(x_{jk}\right)$ be an $m\times n$ real matrix and let $\tilde{X}=\left({\tilde{x}}_{jk}\right)$ be an $m\times n$ matrix in the complex domain. Then, we can always write $\tilde{X}=X_1+i{X}_2$, where $i=\sqrt{(-1)},X_1,X_2$ are real $m\times n$ matrices. Then, matrix differential operators $\frac{\partial }{\partial X}$ and $\frac{\partial }{\partial \tilde{X}}$ will be defined as the following:

$$\frac{\partial }{\partial X}=\left[\begin{array}{ccc}\frac{\partial }{\partial x_{11}}& ...& \frac{\partial }{\partial x_{1n}}\\ ⋮& ...& ⋮\\ \frac{\partial }{\partial x_{m1}}& ...& \frac{\partial }{\partial x_{mn}}\end{array}\right] \mathrm{and} \frac{\partial }{\partial \tilde{X}}=(\frac{\partial }{\partial X_1}+i\frac{\partial }{\partial X_2}).$$

Consider a $p\times p$ nonsingular matrix $\tilde{X}$ in the complex domain and let $|\tilde{X}|$ be its determinant. Then, we have the following result:

Theorem 8.

For the $p\times p$ nonsingular matrix $\tilde{X}$ in the complex domain,

$$\frac{\partial }{\partial \tilde{X}}\left[\right|\tilde{X}\left|\right]=2|{\tilde{X}}^c|{\left({\tilde{X}}^{*}\right)}^{-1}for a general \tilde{X}$$

and

$$\frac{\partial }{\partial \tilde{X}}\left[\right|\tilde{X}\left|\right]=|\tilde{X}|[2{\tilde{X}}^{-1}-\mathrm{diag}\left({\tilde{X}}^{-1}\right)] for \tilde{X}={\tilde{X}}^{*}.$$

Proof. 

The cofactor expansion of a determinant holds in the real and complex domains and it is the following:

$$\begin{array}{cc}\hfill |\tilde{X}|& ={\tilde{x}}_{11}{\tilde{C}}_{11}+...+{\tilde{x}}_{1p}{\tilde{C}}_{1p}\hfill \\ \hfill & ={\tilde{x}}_{21}{\tilde{C}}_{21}+...+{\tilde{x}}_{2p}{\tilde{C}}_{2p}\hfill \\ \hfill & ⋮\hfill \\ \hfill & ={\tilde{x}}_{p1}{\tilde{C}}_{p1}+...+{\tilde{x}}_{pp}{\tilde{C}}_{pp}\hfill \end{array}$$

(17)

where ${\tilde{C}}_{jk}$ is the cofactor of ${\tilde{x}}_{jk}$ for all j and k. Let $\tilde{C}=\left({\tilde{C}}_{jk}\right)$ be the matrix of cofactors. For a general matrix, all ${\tilde{x}}_{jk}$’s are distinct and the corresponding ${\tilde{C}}_{jk}$’s are also distinct. Hence, in the real case, taking the partial derivative of the j-th line in $\left(17\right)$ we have $\frac{\partial }{\partial x_{jk}}\left[\right|X\left|\right]=C_{jk}$ for all j and k. Hence, in the real case

$$\frac{\partial }{\partial X}\left[\right|X\left|\right]=C=|X^{\prime }|{\left(X^{\prime }\right)}^{-1}=\left|X\right|{\left(X^{\prime }\right)}^{-1}.$$

This is a known result. But, in the complex case, the situation is different. Before we tackle the complex-domain situation, we will develop some necessary tools. It is a known result that ${\tilde{X}}^{-1}=\frac{1}{|\tilde{X}|}{\tilde{C}}^{\prime }$. But when $\tilde{X}={\tilde{X}}^{*}$ (Hermitian), the eigenvalues are real, and hence, the determinant is real. Then, ${\tilde{X}}^{-1}$ is Hermitian, thereby $\tilde{C}$ is also Hermitian. The following results will be helpful when we apply our matrix differential operators to scalar functions of matrices in the complex domain. For two scalar complex variables $\tilde{x}$ and $\tilde{y}$ the following can be easily verified.

$$\begin{array}{cc}\hfill \frac{\partial }{\partial \tilde{x}}\left[\tilde{x}\tilde{y}\right]=& 0,\frac{\partial }{\partial {\tilde{x}}^c}\left[{\tilde{x}}^c{\tilde{y}}^c\right]=0\hfill \\ \hfill \frac{\partial }{\partial \tilde{x}}\left[{\tilde{x}}^c\tilde{y}\right]& =2\tilde{y},\frac{\partial }{\partial \tilde{x}}\left[{\tilde{x}}^c{\tilde{y}}^c\right]=2{\tilde{y}}^c,\frac{\partial }{\partial {\tilde{x}}^c}\left[\tilde{x}\tilde{y}\right]=2\tilde{y}.\hfill \end{array}$$

(18)

For convenience, let us consider the term

$$\begin{array}{cc}\hfill {\tilde{u}}_{jk}& ={\tilde{x}}_{jk}^c{\tilde{C}}_{jk}^c=(x_{1jk}-i{x}_{2jk})(C_{1jk}-i{C}_{2jk})\hfill \\ \hfill & =x_{1jk}{C}_{1jk}-x_{2jk}{C}_{2jk}-i(x_{1jk}{C}_{2jk}+x_{2jk}{C}_{1jk})\hfill \end{array}$$

(19)

where ${\tilde{x}}_{jk}=x_{1jk}+i{x}_{2jk},{\tilde{C}}_{jk}=C_{1jk}+i{C}_{2jk}$, with $x_{1jk},x_{2jk},C_{1jk},C_{2jk}$ being real scalar quantities. Note that ${\tilde{x}}_{jk}={\tilde{x}}_{kj}^{*}$ when $\tilde{X}$ is Hermitian. Hence, for example, when we differentiate with respect to ${\tilde{x}}_{21}$ it is equivalent to differentiating with respect to ${\tilde{x}}_{12}^{*}$ and vice versa. When ${\tilde{x}}_{12}^{*}{\tilde{C}}_{12}^{*}$ is differentiated with respect to ${\tilde{x}}_{12}$ it is equivalent to differentiating ${\tilde{x}}_{21}{\tilde{C}}_{21}$ with respect to ${\tilde{x}}_{12}$, and so on. Then,

$$\frac{\partial }{\partial x_{1jk}}\left[{\tilde{u}}_{jk}\right]=C_{1jk}-i{C}_{2jk},\frac{\partial }{\partial x_{2jk}}\left[{\tilde{u}}_{jk}\right]=-C_{2jk}-i{C}_{1jk},\Rightarrow \frac{\partial }{\partial {\tilde{x}}_{jk}}\left[{\tilde{u}}_{jk}\right]=2{\tilde{C}}_{jk}^c.$$

(20)

This is the derivative of the complex conjugate of the $(j,k)$-th element in $\left(18\right)$ for all j and k when $\tilde{X}$ is a general matrix with distinct scalar complex variables as elements. Then,

$$\begin{array}{cc}\hfill \frac{\partial }{\partial \tilde{X}}\left[\right|\tilde{X}\left|\right]& =0,\frac{\partial }{\partial \tilde{X}}\left[\right|{\tilde{X}}^c\left|\right]=2{\tilde{C}}^c=2\left|{\tilde{X}}^c\right|{\left[{\tilde{X}}^{*}\right]}^{-1}\hfill \\ \hfill \frac{\partial }{\partial {\tilde{X}}^c}\left[\right|\tilde{X}\left|\right]& =2\tilde{C}=2\left|\tilde{X}\right|{\left({\tilde{X}}^{\prime }\right)}^{-1}.\hfill \end{array}$$

(21)

When $\tilde{X}={\tilde{X}}^{*}$ (Hermitian) we have $|\tilde{X}|=|{\tilde{X}}^{*}|=|{\tilde{X}}^c|$. Then, the result in $\left(20\right)$ holds for the k-th term in the j-th line in $\left(18\right)$. But, when $\tilde{X}$ is Hermitian, there is one more element contributing to $x_{1jk}$ and $x_{2jk}$. This is the j-th term in the k-th line of $\left(18\right)$. Thus, the sum of the contributions coming from these two terms is the derivative of $|{\tilde{X}}^c|$ with respect to ${\tilde{x}}_{jk}$. The sum of the contributions is the following, observing that $x_{1jk}=x_{1kj},C_{1jk}=C_{1kj},$$x_{2jk}=-x_{2kj},C_{2jk}=-C_{2kj}$:

$$\begin{array}{cc}\hfill & x_{1jk}{C}_{1jk}-x_{2jk}{C}_{2jk}-i(x_{1jk}{C}_{2jk}+x_{2jk}{C}_{1jk})\hfill \\ \hfill & +x_{1kj}{C}_{1kj}-x_{2kj}{C}_{2kj}-i(x_{1kj}{C}_{2kj}+x_{2kj}{C}_{1kj})\hfill \\ \hfill & =2x_{1jk}{C}_{1jk}-2x_{2jk}{C}_{2jk}-i[x_{1jk}(C_{2jk}+C_{2kj})+C_{1jk}(x_{2jk}+x_{2kj})]\hfill \\ \hfill & =2x_{1jk}{C}_{1jk}-2x_{2jk}{C}_{2jk} \mathrm{since} C_{2jk}+C_{2kj}=0,x_{2jk}+x_{2kj}=0.\hfill \end{array}$$

Now,

$$\frac{\partial }{\partial {\tilde{x}}_{jk}}[{\tilde{x}}_{jk}^c{\tilde{C}}_{jk}^c+{\tilde{x}}_{kj}^c{\tilde{C}}_{kj}^c]=2{\tilde{C}}_{jk}^c=2{\tilde{C}}_{kj}$$

for all $j\ne k$. When $j=k$, the diagonal elements in $\tilde{X}$ and $\tilde{C}$ are real, and hence, the term occurs only once, and therefore,

$$\frac{\partial }{\partial {\tilde{x}}_{jj}}\left[{\tilde{x}}_{jj}^c{\tilde{C}}_{jj}^c\right]={\tilde{C}}_{jj}^c=C_{jj}.$$

Since we are taking the partial derivative, the derivatives are the same as differentiation of $|{\tilde{X}}^c|$. Hence,

$$\begin{array}{cc}\hfill \frac{\partial }{\partial \tilde{X}}\left[\right|{\tilde{X}}^c\left|\right]& =2{\tilde{C}}^c-\mathrm{diag}\left({\tilde{C}}^c\right)=\left|{\tilde{X}}^{*}\right|[2{\left({\tilde{X}}^{*}\right)}^{-1}-\mathrm{diag}\left({\left({\tilde{X}}^{*}\right)}^{-1}\right)]\hfill \\ \hfill & =|\tilde{X}|[2{\tilde{X}}^{-1}-\mathrm{diag}\left({\tilde{X}}^{-1}\right)]\hfill \end{array}$$

and then,

$$\frac{\partial }{\partial \tilde{X}}\left[\right|\tilde{X}\left|\right]=|\tilde{X}|[2{\tilde{X}}^{-1}-\mathrm{diag}\left({\tilde{X}}^{-1}\right)].$$

Here, we have used two properties that for a $p\times p$ matrix B, $\left|B\right|=|B^{\prime }|$ and ${\left(B^c\right)}^{\prime }=B^{*}=B$ if B is Hermitian. Then, we have the following theorem: □

Theorem 9.

When the $p\times p$ nonsingular matrix $\tilde{X}$ in the complex domain is Hermitian, that is, $\tilde{X}={\tilde{X}}^{*}$, then,

$$\frac{\partial }{\partial \tilde{X}}\left[\right|\tilde{X}\left|\right]=|\tilde{X}|[2{\tilde{X}}^{-1}-\mathrm{diag}\left({\tilde{X}}^{-1}\right)].$$

In the following sections, we will consider some applications of the results obtained in Section 2.

3. Principal Component Analysis in the Complex Domain

This is an application of the mathematical problem of optimization of a Hermitian form under a Hermitian-form constraint.

In many physical situations, variables occur in pairs, such as time and phase, and hence, the most appropriate representation of such variables is through complex variables because a scalar complex variable can be taken as a pair of real variables. Let $\tilde{x}=x_1+i{x}_2,$$i=\sqrt{(-1)},x_1,x_2$ are real scalar variables, be a scalar complex variable. Then, a statistical density associated with $\tilde{x}$ is a real-valued scalar function $f\left(\tilde{x}\right)$ of $\tilde{x}$ such that $f\left(\tilde{x}\right)\ge 0$ over the entire complex plane, something like a hill on the complex plane, so that the total volume under the surface is unity, that is, ${\int }_{\tilde{x}}f\left(\tilde{x}\right)\mathrm{d}\tilde{x}=1$, where $\mathrm{d}\tilde{x}=\mathrm{d}{x}_1\wedge \mathrm{d}{x}_2$, the wedge product of the differentials $\mathrm{d}{x}_1$ and $\mathrm{d}{x}_2$, respectively. Then, the center of gravity of the hill $f\left(\tilde{x}\right)$ is at $E\left[\tilde{x}\right]={\int }_{\tilde{x}}\tilde{x}f\left(\tilde{x}\right)\mathrm{d}\tilde{x}$ and the square of the measure of scatter in $\tilde{x}$ is ${\sigma }^2=E\left[(\tilde{x}-E\left(\tilde{x}\right)){(\tilde{x}-E\left(\tilde{x}\right))}^{*}\right]$, where $E\left[\right(·\left)\right]$ means the expected value when $\tilde{x}$ is a scalar complex random variable and $f\left(\tilde{x}\right)$ is its density.

If the scatter is small, then the variable $\tilde{x}$ is concentrated near the center of gravity $E\left[\tilde{x}\right]$. If ${\sigma }^2$ is large, then $\tilde{x}$ is spread thin far and wide, and hence, it is more or less unrecognizable. If a large number of scalar variables are being considered as possible variables to be included into a model, then the ones with the larger scatter are the important variables to be included into the model. For convenience, one can consider linear functions of such variables because linear functions also contain individual variables. A linear function is of the form $\tilde{u}=a_1^{*}{\tilde{x}}_1+...+a_p^{*}{\tilde{x}}_k$, where $a_1^{*},...,a_p^{*}$ are constants with ${\tilde{x}}_1,...,{\tilde{x}}_p$ being scalar complex variables. For example, $a_2^{*}=0=...=a_p^{*}=0$ gives ${\tilde{x}}_1$, where a * indicates the conjugate transpose and for a scalar quantity y, ${\tilde{y}}^{*}={\tilde{y}}^c$ only, where c in the exponent indicates the complex conjugate. We may write the linear function as $\tilde{u}=A^{*}\tilde{X}$, where

$$A=\left[\begin{array}{c}{a}_1\\ ⋮\\ a_p\end{array}\right],A^{\prime }=[a_1,...,a_p] \mathrm{and} \tilde{X}=\left[\begin{array}{c}{\tilde{x}}_1\\ ⋮\\ {\tilde{x}}_p\end{array}\right],{\tilde{X}}^{\prime }=[{\tilde{x}}_1....,{\tilde{x}}_p]$$

where a prime indicates the transpose. The expected value of $A^{*}\tilde{X}$ is $E\left[\tilde{u}\right]=A^{*}E\left[\tilde{X}\right]$ and the variance–covariance matrix or the covariance matrix in $\tilde{X}$ is denoted by $\mathrm{\Sigma }>O$ (Hermitian positive definite). Then, the variance of $\tilde{u}$, denoted by $\mathrm{Var}\left(\tilde{u}\right)$, is given by $\mathrm{Var}\left(\tilde{u}\right)=E[\left(A^{*}(\tilde{X}-E\left(\tilde{X}\right)){(\tilde{X}-E\left(\tilde{X}\right))}^{*}A\right]=A^{*}\mathrm{\Sigma }A$. The most important linear function is that linear function having the maximum variance. Hence, we may compute ${max}_A\left[A^{*}\mathrm{\Sigma }A\right]$. But, since $\mathrm{\Sigma }>O$ we have a positive definite Hermitian form in $A^{*}\mathrm{\Sigma }A$ and its maximum over A is at $+\infty$. Thus, unrestricted maximization does not make sense. Without loss of generality we may take $A^{*}A=1$ because this can always be achieved for any non-null A. Then, the maximization amounts to maximizing $A^{*}\mathrm{\Sigma }A$ within the unit sphere $A^{*}A=1$. There will be a maximum and a minimum in this case. We may incorporate the restriction by using a Lagrangian multiplier. Consider

$$\tilde{w}=A^{*}\mathrm{\Sigma }A-\lambda (A^{*}A-1)$$

(22)

where $\lambda$ is a Lagrangian multiplier. Maximization will be achieved by using the following result, which will be stated as a lemma.

Lemma 2.

Let $\tilde{Y}=Y_1+i{Y}_2,i=\sqrt{(-1)},Y_1,Y_2$ are $p\times 1$ vectors with distinct real scalar variables as elements. Let $B=B^{*}$ be a $p\times p$ constant Hermitian matrix. Let the partial differential operator $\frac{\partial }{\partial Y}$ be defined as

$$\frac{\partial }{\partial \tilde{Y}}=\frac{\partial }{\partial Y_1}+i\frac{\partial }{\partial Y_2}.$$

Then,

$$\frac{\partial }{\partial \tilde{Y}}\left[{\tilde{Y}}^{*}B\tilde{Y}\right]=2B\tilde{Y},B=B^{*}>O,\frac{\partial }{\partial \tilde{Y}}\left[{\tilde{Y}}^{*}\tilde{Y}\right]=2\tilde{Y}.$$

Proof. 

Let $B=B^{*}=B_1+i{B}_2$. When B is Hermitian, $B_1=B_1^{\prime },B_2=-B_2^{\prime }$, that is, $B_1$ is real symmetric and $B_2$ is real skew symmetric. Let $\tilde{Y}=Y_1+i{Y}_2,i=\sqrt{(-1)},Y_1,Y_2$ are real. Then, ${\tilde{Y}}^{*}B\tilde{Y}=(Y_1^{\prime }-i{Y}_2^{\prime })B(Y_1+i{Y}_2)=Y_1^{\prime }B{Y}_1+Y_2^{\prime }B{Y}_2+i(Y_1^{\prime }B{Y}_2-Y_2^{\prime }B{Y}_1)$, where $Y_1^{\prime }B{Y}_1=Y_1^{\prime }{B}_1{Y}_1+i{Y}_1^{\prime }{B}_2{Y}_1=Y_1^{\prime }{B}_1{Y}_1$ because $Y_1^{\prime }{B}_2{Y}_1=0$ due to $B_2=-B_2^{\prime }$. From the real case it is well known that $\frac{\partial }{\partial Y_1}\left[Y_1^{\prime }{B}_1{Y}_1\right]=2B_1{Y}_1$. Similarly, $\frac{\partial }{\partial Y_2}\left[Y_2^{\prime }B{Y}_2\right]=2B_1{Y}_2$. Now,

$$\begin{array}{cc}\hfill i\frac{\partial }{\partial Y_1}[Y_1^{\prime }B{Y}_2-Y_2^{\prime }B{Y}_1]& =i\frac{\partial }{\partial Y_1}[Y_1^{\prime }{B}_1{Y}_2-Y_2^{\prime }{B}_1{Y}_1+i(Y_1^{\prime }{B}_2{Y}_2-Y_2^{\prime }{B}_2{Y}_1)]\hfill \\ \hfill & =0+i^2\frac{\partial }{\partial Y_1}[Y_1^{\prime }{B}_2{Y}_2+Y_1^{\prime }{B}_2{Y}_2]=i^{2}2B_2{Y}_2=-2B_2{Y}_2.\hfill \end{array}$$

Similarly,

$$\frac{\partial }{\partial Y_2}[-Y_1^{\prime }B{Y}_2-Y_2^{\prime }B{Y}_1]=-2i^2{B}_2{Y}_1=2B_2{Y}_1$$

where we have used two properties. When $B_1=B_1^{\prime }$, that is, it is real symmetric, we have $Y_1^{\prime }{B}_1{Y}_2=Y_2^{\prime }{B}_1{Y}_1$ because both are $1\times 1$ real, and hence, each is equal to its transpose, and therefore, the difference is zero. When $B_2=-B_2^{\prime }$ we have $Y_2^{\prime }{B}_2{Y}_1=-Y_1^{\prime }{B}_2{Y}_2$, because both are $1\times 1$ real, and hence, each is equal to its transpose. Then, from the real operator operating on a real linear form the result follows. Similar results hold when differentiating with respect to the operator $\frac{\partial }{\partial Y_2}$. Thus, we have the following:

$$\begin{array}{cc}\hfill \frac{\partial }{\partial Y_1}\left[{\tilde{Y}}^{*}B\tilde{Y}\right]& =2B_1{Y}_1-2B_2{Y}_2=2B_1{Y}_1+i^{2}2B_2{Y}_2 \mathrm{and}\hfill \\ \hfill \frac{\partial }{\partial Y_2}\left[{\tilde{Y}}^{*}B\tilde{Y}\right]& =2B_1{Y}_2+2B_2{Y}_1.\hfill \end{array}$$

Therefore,

$$\begin{array}{cc}\hfill (\frac{\partial }{\partial Y_1}+i\frac{\partial }{\partial Y_2})\left[{\tilde{Y}}^{*}B\tilde{Y}\right]& =2B_1(Y_1+i{Y}_2)+2i{B}_2(Y_1+i{Y}_2)=2B_1\tilde{Y}+2i{B}_2\tilde{Y}\hfill \\ \hfill & =2B\tilde{Y}.\hfill \end{array}$$

Hence, the result. For $B=I$, the identity matrix, the result on ${\tilde{Y}}^{*}\tilde{Y}$ follows, that is, $\frac{\partial }{\partial \tilde{Y}}\left({\tilde{Y}}^{*}\tilde{Y}\right)=2\tilde{Y}$. Now, by using Lemma 1 we can differentiate $\tilde{w}$ in $\left(22\right)$. That is,

$$\begin{array}{cc}\hfill \frac{\partial }{\partial A}\tilde{w}& =O,\frac{\partial }{\partial \lambda }=0\Rightarrow \mathrm{\Sigma }A-\lambda A=O\hfill \\ \hfill & \Rightarrow (\mathrm{\Sigma }-\lambda I)A=O\Rightarrow |\mathrm{\Sigma }-\lambda I|=0\hfill \end{array}$$

(23)

where $\left|\right(·\left)\right|$ means the determinant of the square matrix $(·)$. From $\left(23\right)$, $\mathrm{\Sigma }A=\lambda A$ and pre-multiplying by $A^{*}$ and using the fact that $A^{*}A=1$, we have $A^{*}\mathrm{\Sigma }A=\lambda$. Hence,

$$\underset{A^{*}A=1}{max}\left[A^{*}\mathrm{\Sigma }A\right]={\lambda }_1 \mathrm{and} \underset{A^{*}A=1}{min}\left[A^{*}\mathrm{\Sigma }A\right]={\lambda }_p$$

(24)

where ${\lambda }_1$ is the largest eigenvalue of $\mathrm{\Sigma }={\mathrm{\Sigma }}^{*}>O$ and ${\lambda }_p$ is the smallest eigenvalue of $\mathrm{\Sigma }$. When $\mathrm{\Sigma }$ is Hermitian, all its eigenvalues are real and when it is Hermitian positive definite, all its eigenvalues are real positive also. Hence, the procedure is the following: Take the largest eigenvalue of $\mathrm{\Sigma }$, say ${\lambda }_1$. Then, through (23), compute an eigenvector corresponding to ${\lambda }_1$, that is, solve $\mathrm{\Sigma }A={\lambda }_{1}A$ for an A. Then, normalize this eigenvector through $A_1^{*}{A}_1=1$. That is, if an eigenvector corresponding to ${\lambda }_1$ is ${\alpha }_1$, then compute $A_1=\frac{1}{\sqrt{{\alpha }_1^{*}{\alpha }_1}}{\alpha }_1$. This $A_1$ is the normalized eigenvector corresponding to ${\lambda }_1$. Now, consider ${\tilde{u}}_1=A_1^{*}\tilde{X}$. This ${\tilde{u}}_1$ is the first principal component in the sense of the linear function having the maximum variance. Now, take the second largest eigenvalue ${\lambda }_2$. Go through the same procedure and construct the normalized eigenvector $A_2$ corresponding to ${\lambda }_2$. Then, ${\tilde{u}}_2=A_2^{*}\tilde{X}$ is the second principal component. Continue the process and stop the process when the variance of ${\tilde{u}}_j$, namely, ${\lambda }_j$, falls below a preassigned number. If there is no preassigned number, then ${\tilde{u}}_1,...,{\tilde{u}}_p$ will be the p principal components. Here, we have assumed that the eigenvalues of $\mathrm{\Sigma }$ are distinct. When the eigenvalues are distinct, we can show that the eigenvectors corresponding to the distinct eigenvalues of a symmetric or Hermitian matrix are orthogonal to each other. Hence, our principal components will be orthogonal to each other in the sense that the joint dispersion in the pair $({\tilde{u}}_i,{\tilde{u}}_j)$ is 0 for $i\ne j$ or the covariance between ${\tilde{u}}_i$ and ${\tilde{u}}_j$ is zero when $i\ne j$. The covariance is defined as the following: Let $\tilde{U}$ be a $p\times 1$ vector in the complex domain and let $\tilde{V}$ be a $q\times 1$ vector in the complex domain. Then, the covariance of $\tilde{U}$ on $\tilde{V}$ is defined and denoted as $\mathrm{Cov}(\tilde{U},\tilde{V})=E\left[(\tilde{U}-E\left(\tilde{U}\right)){(\tilde{V}-E\left(\tilde{V}\right))}^{*}\right]$, whenever this expected value exists, so that when $\tilde{V}=\tilde{U}$, then $\mathrm{Cov}(\tilde{U},\tilde{V})=\mathrm{Cov}\left(\tilde{U}\right)$, the covariance matrix in $\tilde{U}$, and when $p=1$, it is the variance of the scalar complex variable $\tilde{u}$. □

When the covariance matrix $\mathrm{\Sigma }$ is unknown, then we may construct sample principal components. Let our population be the $p\times 1$ vector $\tilde{X},{\tilde{X}}^{\prime }=[{\tilde{x}}_1,...,{\tilde{x}}_p]$, where ${\tilde{x}}_j,j=1,...,p$ are distinct scalar complex variables. Consider n independently and identically distributed (iid) such p-vectors. Then, we have a simple random sample of size n from $\tilde{X}$. Then, the sample matrix is the $p\times n,n>p$ matrix, denoted as the following:

$$\tilde{\mathbf{X}}=[{\tilde{X}}_1,...,{\tilde{X}}_n]=\left[\begin{array}{cccc}{\tilde{x}}_{11}& {\tilde{x}}_{12}& ...& {\tilde{x}}_{1n}\\ {\tilde{x}}_{21}& {\tilde{x}}_{22}& ...& {\tilde{x}}_{2n}\\ ⋮& ⋮& ...& ⋮\\ {\tilde{x}}_{p1}& {\tilde{x}}_{p2}& ...& {\tilde{x}}_{pn}\end{array}\right].$$

Let the sample average be denoted by $\overline{\tilde{X}}=\frac{1}{n}[{\tilde{X}}_1+...+{\tilde{X}}_n]$ and the matrix of sample averages be denoted by the bold letter $\overline{\tilde{\mathbf{X}}}=[\overline{\tilde{X}},...,\overline{\tilde{X}}]$. Then, the sample sum of products matrix $\tilde{S}$ is given by

$$\tilde{S}=[\tilde{\mathbf{X}}-\overline{\tilde{\mathbf{X}}}]{[\tilde{\mathbf{X}}-\overline{\tilde{\mathbf{X}}}]}^{*}=\left({\tilde{s}}_{jk}\right)$$

where ${\tilde{s}}_{jk}={\sum }_{r=1}^n({\tilde{x}}_{jr}-{\overline{\tilde{x}}}_j){({\tilde{x}}_{rk}-{\overline{\tilde{x}}}_k)}^{*}$. The motivation in using the sample sum of products matrix $\tilde{S}$ is that $\frac{1}{n-1}\tilde{S}$ is an unbiased estimator of $\mathrm{\Sigma }$. Since we will be normalizing the eigenvectors, operate with $\tilde{S}$ itself. Compute the eigenvalues of $\tilde{S}$. Take the largest eigenvalue of $\tilde{S}$. Call it $m_1$. Construct an eigenvector corresponding to $m_1$ and normalize it through $M_1^{*}{M}_1=1$, where $M_1$ is the normalized eigenvector corresponding to $m_1$. Then, ${\tilde{v}}_1=M_1^{*}\tilde{X}$ is the first sample principal component. When the columns of the sample matrix are not linearly related then we have $\tilde{S}={\tilde{S}}^{*}>O$ (Hermitian positive definite) and all eigenvalues $m_1,...,m_p$ will be positive. We assume that the eigenvalues are distinct $m_1>m_2>...>m_p$. This will be true almost surely. Now, take $m_2$ and construct $M_2$ and the second principal component ${\tilde{v}}_2=M_2^{*}\tilde{X}$ and continue the process. We can show that the covariances between ${\tilde{v}}_j$ and ${\tilde{v}}_k$ will be zeros for all $j\ne k$. This property follows from the fact that when the matrix is symmetric or Hermitian, the eigenvectors corresponding to distinct eigenvalues are orthogonal. When the population $\tilde{X}$ is p-variate complex Gaussian, then we can show that $\tilde{S}$ will be a complex Wishart distributed with degrees of freedom $n-1$ and parameter matrix $\mathrm{\Sigma }$. The distributions of the largest, smallest, and j-th largest eigenvalues and the corresponding eigenvectors of $\tilde{S}$ in the complex domain are given in [8].

4. Canonical Correlation Analysis in the Complex Domain

This is an application of the mathematical problem of optimization of a bilinear form in the complex domain, under two Hermitian-form constraints. The following application is regarding the prediction of one set of variables by using another set of variables.

Consider two sets of scalar complex variables

$$S_1=\{{\tilde{x}}_1,...,{\tilde{x}}_p\} \mathrm{and} S_2=\{{\tilde{y}}_1,...,{\tilde{y}}_q\}$$

where p need not be equal to q. Consider the appended vector

$$\begin{array}{cc}\hfill \tilde{Z}& =\left[\begin{array}{c}\tilde{X}\\ \tilde{Y}\end{array}\right],\tilde{X}=\left[\begin{array}{c}{\tilde{x}}_1\\ ⋮\\ {\tilde{x}}_p\end{array}\right],\tilde{Y}=\left[\begin{array}{c}{\tilde{y}}_1\\ ⋮\\ {\tilde{y}}_q\end{array}\right],E\left[\tilde{X}\right]={\mu }_x,E\left[\tilde{Y}\right]={\mu }_y,\hfill \\ \hfill \mathrm{\Sigma }& =E\left[\left[\begin{array}{c}\tilde{X}-{\mu }_x\\ \tilde{Y}-{\mu }_y\end{array}\right][{\tilde{X}}^{*}-{\mu }_x^{*},{\tilde{Y}}^{*}-{\mu }_y^{*}]\right]=\mathrm{Cov}\left(\tilde{Z}\right)=\left[\begin{array}{cc}{\mathrm{\Sigma }}_{11}& {\mathrm{\Sigma }}_{12}\\ {\mathrm{\Sigma }}_{21}& {\mathrm{\Sigma }}_{22}\end{array}\right]\hfill \end{array}$$

where ${\mathrm{\Sigma }}_{11}=\mathrm{Cov}\left(\tilde{X}\right),{\mathrm{\Sigma }}_{22}=\mathrm{Cov}\left(\tilde{Y}\right),{\mathrm{\Sigma }}_{12}=\mathrm{Cov}(\tilde{X},\tilde{Y})$, and ${\mathrm{\Sigma }}_{12}^{*}={\mathrm{\Sigma }}_{21}$, and vice-versa. That is, $\mathrm{\Sigma }$ is the covariance matrix in $\tilde{Z}$, ${\mathrm{\Sigma }}_{11}$ is the covariance matrix in $\tilde{X}$, ${\mathrm{\Sigma }}_{22}$ is the covariance matrix in $\tilde{Y}$, and so on. Also, $\mathrm{\Sigma }={\mathrm{\Sigma }}^{*}>O,{\mathrm{\Sigma }}_{11}={\mathrm{\Sigma }}_{11}^{*}>O,{\mathrm{\Sigma }}_{22}={\mathrm{\Sigma }}_{22}^{*}>O$. Our aim is to predict the variables in the set $S_1$ by using the variables in the set $S_2$ and vice-versa, and obtain the “best” predictors; “best” in the sense of having the maximum joint dispersion. In order to represent each set $S_1$ and $S_2$, we will take arbitrary linear functions of the variables in each set. Consider linear functions $\tilde{u}=A^{*}\tilde{X}$ and $\tilde{v}=B^{*}\tilde{Y}$, where

$$A=\left[\begin{array}{c}{a}_1\\ ⋮\\ a_p\end{array}\right],B=\left[\begin{array}{c}{b}_1\\ ⋮\\ b_q\end{array}\right],\begin{array}{cc}\tilde{u}& =A^{*}\tilde{X}=a_1^{*}{\tilde{x}}_1+...+a_p^{*}{\tilde{x}}_p\\ \tilde{v}& =B^{*}\tilde{Y}=b_1^{*}{\tilde{y}}_1+...+b_q^{*}{\tilde{y}}_q\end{array},$$

where $\tilde{X}$ and $\tilde{Y}$ are listed above already. Since $a_j$ and $b_j$ are scalar constant quantities, $a_j^{*}=a_j^c,j=1,...,p$ and $b_j^{*}=b_j^c,j=1,...,q$. Variances for linear functions are already seen in Section 2. Therefore, $\mathrm{Var}\left(\tilde{u}\right)=A^{*}{\mathrm{\Sigma }}_{11}A,\mathrm{Var}\left(\tilde{v}\right)=B^{*}{\mathrm{\Sigma }}_{22}B$ and $\mathrm{Cov}(\tilde{u},\tilde{v})=A^{*}{\mathrm{\Sigma }}_{12}B,$${\mathrm{\Sigma }}_{21}={\mathrm{\Sigma }}_{12}^{*},B^{*}{\mathrm{\Sigma }}_{21}A=\mathrm{Cov}(\tilde{v},\tilde{u})$. Here, ${\mathrm{\Sigma }}_{12}$ and ${\mathrm{\Sigma }}_{21}$ can be taken as measures of joint dispersion or joint variation between $\tilde{X}$ and $\tilde{Y}$ and $A^{*}{\mathrm{\Sigma }}_{12}B=\mathrm{Cov}(\tilde{u},\tilde{v})$ as the joint dispersion between $\tilde{u}$ and $\tilde{v}$. As a criterion for the “best” predictor of $\tilde{u}$ by using $\tilde{v}$ and vice versa, we may take that the pair $\tilde{u}$ and $\tilde{v}$ have the maximum joint variation. The best predictor of $\tilde{u}$ by using $\tilde{v}$ as that pair having the maximum $A^{*}{\mathrm{\Sigma }}_{12}B$ and the best predictor of $\tilde{v}$ by using $\tilde{u}$ as that pair having the maximum value for $B^{*}{\mathrm{\Sigma }}_{21}A$. Since covariances depend upon the units of measurements of the variables involved, we may take a scale-free covariance by taking

$$\rho =\frac{\mathrm{Cov}(\tilde{u},\tilde{v})}{\sqrt{\mathrm{Var}\left(\tilde{u}\right)\mathrm{Var}\left(\tilde{v}\right)}}=\frac{A^{*}{\mathrm{\Sigma }}_{12}B}{\sqrt{\left[A^{*}{\mathrm{\Sigma }}_{11}A\right]\left[B^{*}{\mathrm{\Sigma }}_{22}B\right]}}.$$

Further, as explained in Section 2, without loss of generality we may take $A^{*}{\mathrm{\Sigma }}_{11}A=1$ and $B^{*}{\mathrm{\Sigma }}_{22}B=1$ or confine the bilinear form (hyperboloid) within unit positive definite Hermitian forms (ellipsoids) in order to prevent them from going to $+\infty$. Hence, our procedure simplifies to optimizing $A^{*}{\mathrm{\Sigma }}_{12}B$ subject to the conditions $A^{*}{\mathrm{\Sigma }}_{11}A=1,$$B^{*}{\mathrm{\Sigma }}_{22}B=1$ and computing that pair of A and B which will maximize $A^{*}{\mathrm{\Sigma }}_{12}B$. As before, we may use the Lagrangian multipliers ${\lambda }_1$ and ${\lambda }_2$ and consider the function

$$\tilde{w}=A^{*}{\mathrm{\Sigma }}_{12}B-{\lambda }_1(A^{*}{\mathrm{\Sigma }}_{11}A-1)-{\lambda }_2(B^{*}{\mathrm{\Sigma }}_{22}B-1).$$

(25)

In order to optimize this $\tilde{w}$ we need one result on differentiation of a bilinear form, which will be stated as a lemma.

Lemma 3.

Let $\tilde{X}=X_1+i{X}_2,i=\sqrt{(-1)},X_1,X_2$ are real, $p\times 1$ vectors of distinct real scalar variables $x_{1j}$ and $x_{2j}$, respectively, where ${\tilde{x}}_j=x_{j1}+i{x}_{j2},j=1,...,p$. Let $\tilde{Y}=Y_1+i{Y}_2,$$i=\sqrt{(-1)},Y_1,Y_2$ are real, $q\times 1$ vectors of distinct real scalar variables $y_{j1}$ and $y_{j2}$, respectively, where ${\tilde{y}}_j=y_{j1}+i{y}_{j2},j=1,...,q$, where $x_{j1},x_{j2},y_{j1},y_{j2}$ are real. Let the partial differential operators be as defined in Section 2, namely,

$$\frac{\partial }{\partial X_1}=\left[\begin{array}{c}\frac{\partial }{\partial x_{11}}\\ ⋮\\ \frac{\partial }{\partial x_{p1}}\end{array}\right],\frac{\partial }{\partial X_2}=\left[\begin{array}{c}\frac{\partial }{\partial x_{12}}\\ ⋮\\ \frac{\partial }{\partial x_{p2}}\end{array}\right],\frac{\partial }{\partial \tilde{X}}=(\frac{\partial }{\partial X_1}+i\frac{\partial }{\partial X_2})$$

and similar operators involving $\tilde{Y}=Y_1+i{Y}_2$. Then,

$$\frac{\partial }{\partial \tilde{X}}\left[{\tilde{X}}^{*}A\tilde{Y}\right]=2A\tilde{Y} and \frac{\partial }{\partial \tilde{Y}}\left[{\tilde{Y}}^{*}{A}^{*}\tilde{X}\right]=2A^{*}\tilde{X}.$$

Proof. 

Opening up ${\tilde{X}}^{*}A\tilde{Y}$ we have the following:

$${\tilde{X}}^{*}A\tilde{Y}=(X_1^{\prime }-i{X}_2^{\prime })A(Y_1+i{Y}_2)=X_1^{\prime }A{Y}_1+X_2^{\prime }A{Y}_2+i(X_1^{\prime }A{Y}_2-X_2^{\prime }A{Y}_1).$$

Then, from the known results in the real case, we have the following:

$$\frac{\partial }{\partial X_1}\left[{\tilde{X}}^{*}A\tilde{Y}\right]=A{Y}_1+iA{Y}_2 \mathrm{and} \frac{\partial }{\partial X_2}\left[{\tilde{X}}^{*}A\tilde{Y}\right]=A{Y}_2-iA{Y}_1$$

irrespective of whether A is real or in the complex domain. Then,

$$\frac{\partial }{\partial \tilde{X}}\left[{\tilde{X}}^{*}A\tilde{Y}\right]=(\frac{\partial }{\partial X_1}+i\frac{\partial }{\partial X_2})\left[{\tilde{X}}^{*}A\tilde{Y}\right]=2A\tilde{Y}.$$

Similarly, $\frac{\partial }{\partial \tilde{Y}}\left[{\tilde{Y}}^{*}{A}^{*}\tilde{X}\right]=2A^{*}\tilde{X}.$ This completes the proof. □

Now, differentiating $\tilde{w}$ in $\left(25\right)$, we have the following:

$$\begin{array}{cc}\hfill \frac{\partial \tilde{w}}{\partial A}=O& \Rightarrow {\mathrm{\Sigma }}_{12}B-{\lambda }_1{\mathrm{\Sigma }}_{11}A=O\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill \frac{\partial {\tilde{w}}^{*}}{\partial B}=O& \Rightarrow {\mathrm{\Sigma }}_{21}A-{\lambda }_2{\mathrm{\Sigma }}_{22}B=O.\hfill \end{array}$$

(27)

Now, premultiply $\left(26\right)$ by $A^{*}$ and $\left(27\right)$ by $B^{*}$ to obtain the following, observing that ${\mathrm{\Sigma }}_{21}={\mathrm{\Sigma }}_{12}^{*}$:

$$A^{*}{\mathrm{\Sigma }}_{12}B={\lambda }_1,B^{*}{\mathrm{\Sigma }}_{21}A={\lambda }_2,\Rightarrow {\lambda }_2={\lambda }_1^c$$

or ${\lambda }_1=\lambda ,{\lambda }_2={\lambda }^c$. Take B from $\left(27\right)$ and substitute in $\left(26\right)$ to obtain the following:

$$({\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}-\lambda {\lambda }^{c}I)A=O,\lambda {\lambda }^c={\left|\lambda \right|}^2.$$

(28)

This shows that $\lambda {\lambda }^c={\left|\lambda \right|}^2=\mu$ is an eigenvalue of ${\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}$, where the matrix is $p\times p$. From symmetry, it follows that $\mu$ is also an eigenvalue of ${\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}$, where the matrix is $q\times q$. Hence, all the nonzero $\mu$’s are common to both of these matrices. Hence, the procedure is the following: If $p\le q$, then compute the eigenvalues of ${\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}$, otherwise, compute the eigenvalues of the other matrix ${\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}$, both will give the same nonzero eigenvalues. Let ${\mu }_1$ be the largest and ${\mu }_r$ be the smallest nonzero eigenvalues. Then, we have the results

$$\underset{A^{*}{\mathrm{\Sigma }}_{11}A=1,B^{*}{\mathrm{\Sigma }}_{22}B=1}{max}\left[A^{*}{\mathrm{\Sigma }}_{12}B\right]={\mu }_1$$

and

$$\underset{A^{*}{\mathrm{\Sigma }}_{11}A=1,B^{*}{\mathrm{\Sigma }}_{22}B=1}{min}\left[A^{*}{\mathrm{\Sigma }}_{12}B\right]={\mu }_r.$$

Then, the procedure is the following: If $p\le q$, then compute all the eigenvalues of ${\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}$. Let the largest eigenvalue be ${\mu }_1$. Then, compute one eigenvector corresponding to ${\mu }_1$. Use the equation $({\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}-{\mu }_1{\mathrm{\Sigma }}_{11})A=O$. Let it be $A_{11}$. Then, normalize it through $A_{11}^{*}{\mathrm{\Sigma }}_{11}{A}_{11}=1$. That is, compute $A_1=\frac{1}{\sqrt{A_{11}^{*}{\mathrm{\Sigma }}_{11}{A}_{11}}}{A}_{11}$. Then, compute ${\tilde{u}}_1=A_1^{*}\tilde{X}$. Then, use the same eigenvalue ${\mu }_1$ and compute one eigenvector from the equation $({\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}-{\mu }_1{\mathrm{\Sigma }}_{22})B=O$. Let it be $B_{11}$. Then, normalize it through $B_{11}^{*}{\mathrm{\Sigma }}_{22}{B}_{11}=1$, that is, compute $B_1=\frac{1}{\sqrt{B_{11}^{*}{\mathrm{\Sigma }}_{22}{B}_{11}}}{B}_{11}$. Now, compute ${\tilde{v}}_1=B_1^{*}\tilde{Y}$. Then, $({\tilde{u}}_1,{\tilde{v}}_1)$ is the first pair of canonical variables in the sense ${\tilde{u}}_1$ is the best predictor of ${\tilde{v}}_1$ and vice-versa. Now, take the second largest eigenvalue ${\mu }_2$ of ${\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}$. Then, compute one eigenvector corresponding to ${\mu }_2$, that is, solve the equation $({\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}-{\mu }_2{\mathrm{\Sigma }}_{11})A=O.$ Let $A_{21}$ be that eigenvector. Normalize it, that is, compute $A_2=\frac{1}{\sqrt{A_{21}^{*}{\mathrm{\Sigma }}_{11}{A}_{21}}}{A}_{21}$. Now, compute ${\tilde{u}}_2=A_2^{*}\tilde{X}$. Use the same ${\mu }_2$ and solve for B from the equation $({\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}-{\mu }_2{\mathrm{\Sigma }}_{22})B=O$. Let $B_{21}$ be one solution. Then, normalize it, that is, compute $B_2=\frac{1}{\sqrt{B_{21}^{*}{\mathrm{\Sigma }}_{22}{B}_{21}}}{B}_{21}$. Now, compute ${\tilde{v}}_2=B_2^{*}\tilde{Y}$. Then, $({\tilde{u}}_2,{\tilde{v}}_2)$ is the second pair of canonical variables. Continue the process until ${\mu }_j$ falls below a preassigned limit. If there is no such preassigned limit, then compute all the pairs, that is, p if $p\le q$; otherwise q. If $q<p$, then start with the computation of the eigenvalues of ${\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}$ and proceed parallel to the steps used in the case $p\le q$. Observe that the symmetric format of ${\mathrm{\Sigma }}_{11}^{-1}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}$ is ${\mathrm{\Sigma }}_{11}^{-\frac{1}{2}}{\mathrm{\Sigma }}_{12}{\mathrm{\Sigma }}_{22}^{-1}{\mathrm{\Sigma }}_{21}{\mathrm{\Sigma }}_{11}^{-\frac{1}{2}}$. This form is also available from the same starting Equation $\left(28\right)$. The symmetric format can always be written in the form $C^{*}C$ for some matrix C, and hence, the symmetric form is either Hermitian positive definite or Hermitian positive semi-definite, and therefore, all the nonzero eigenvalues are positive. Let us assume that the eigenvalues ${\mu }_1,{\mu }_2,...$ are distinct, ${\mu }_1>{\mu }_2>...>{\mu }_p$ if $p\le q$. It is a known result that eigenvectors corresponding to distinct eigenvalues of Hermitian or symmetric matrices are orthogonal to each other. Hence, ${\tilde{u}}_1,{\tilde{u}}_2,...$ are non-correlated. Similarly, ${\tilde{v}}_1,{\tilde{v}}_2,...$ are non-correlated.

If ${\mathrm{\Sigma }}_{11},{\mathrm{\Sigma }}_{12},{\mathrm{\Sigma }}_{22}$ are not available, then take a simple random sample of size n, $n>p+q$, from $\tilde{Z}=\left[\begin{array}{c}\tilde{X}\\ \tilde{Y}\end{array}\right]$. Then, compute the sample sum of products matrix:

$$\tilde{S}=\left[\begin{array}{cc}{\tilde{S}}_{11}& {\tilde{S}}_{12}\\ {\tilde{S}}_{21}& {\tilde{S}}_{22}\end{array}\right],$$

as for the case of principal component analysis. Now, continue with ${\tilde{S}}_{jk}$’s as with ${\mathrm{\Sigma }}_{jk}$’s. Then, we will obtain the pairs of sample canonical variables. Some distributional aspects of sample canonical variables and the sample canonical correlation matrix $\tilde{U}={\tilde{S}}_{11}^{-\frac{1}{2}}{\tilde{S}}_{12}{\tilde{S}}_{22}^{-1}{\tilde{S}}_{21}{\tilde{S}}_{11}^{-\frac{1}{2}}$ are discussed in [8]. Note that when $S_{11}$ is $1\times 1$, then we have the square of the multiple correlation coefficient in $\tilde{U}=\tilde{u}$, which is scalar. In this case, our starting set $S_1$ will have only one complex scalar variable and the set $S_2$ will have q variables. Here, the problem is to predict one variable in $S_1$ by using the variables in $S_2$. The exact distributions of the canonical correlation matrix $\tilde{U}$ and the square of the absolute value of the multiple correlation coefficient $\tilde{u}$ are available in explicit forms in [8].

5. Covariance and Correlation in the Complex Domain

Consider scalar complex variables first. Let ${\tilde{x}}_1$ and ${\tilde{y}}_1$ be two scalar complex variables or scalar variables in the complex domain. Then, the mean values or expected values of ${\tilde{x}}_1$ and ${\tilde{y}}_1$ are, respectively, $E\left[{\tilde{x}}_1\right]={\int }_{{\tilde{x}}_1}{\tilde{x}}_{1}f\left({\tilde{x}}_1\right)\mathrm{d}{\tilde{x}}_1,E\left[{\tilde{y}}_1\right]={\int }_{{\tilde{y}}_1}{\tilde{y}}_{1}g\left({\tilde{y}}_1\right)\mathrm{d}{\tilde{y}}_1$, where $E[·]$ means the expected value of $[·]$, and f and g are the densities of ${\tilde{x}}_1$ and ${\tilde{y}}_1$, respectively. Let $\tilde{x}={\tilde{x}}_1-E\left[{\tilde{x}}_1\right],\tilde{y}={\tilde{y}}_1-E\left[{\tilde{y}}_1\right]$. Let $\tilde{x}=x_{11}+i{x}_{12},i=\sqrt{(-1)},x_{11},x_{12}$ are real scalar variables, and let $\tilde{y}=y_{11}+i{y}_{12},i=\sqrt{(-1)},y_{11},y_{12}$ are real scalar variables. Then, $E\left[\tilde{x}\right]=0,$$E\left[\tilde{y}\right]=0$. Then, the variance, denoted by $\mathrm{Var}(·)$, and covariance, denoted by $\mathrm{Cov}(·,·)$, are the following:

$$\begin{array}{cc}\hfill \mathrm{Var}\left({\tilde{x}}_1\right)& =\mathrm{Var}\left(\tilde{x}\right)=E\left[\tilde{x}{\tilde{x}}^{*}\right]={\sigma }_{11}\hfill \\ \hfill \mathrm{Var}\left({\tilde{y}}_1\right)& =\mathrm{Var}\left(\tilde{y}\right)=E\left[\tilde{y}{\tilde{y}}^{*}\right]={\sigma }_{22}\hfill \\ \hfill \mathrm{Cov}({\tilde{x}}_1,{\tilde{y}}_1)& =\mathrm{Cov}(\tilde{x},\tilde{y})=E\left[\tilde{x}{\tilde{y}}^{*}\right]={\sigma }_{12}\hfill \end{array}$$

where, for example, ${\tilde{x}}^{*}={\tilde{x}}^c$, with * indicating the conjugate transpose and c indicating the conjugate only. Here, we have only scalar variables, and hence, the complex conjugate transpose is only complex conjugate. For convenience, we have used the notation ${\sigma }_{11},{\sigma }_{22},{\sigma }_{12}$. Then, ${\sigma }_{21}=E\left[\tilde{y}{\tilde{x}}^{*}\right]=E\left[\tilde{y}{\tilde{x}}^c\right]=\mathrm{Cov}(\tilde{y},\tilde{x})$. For a scalar complex variable $\tilde{x}=x_{11}+i{x}_{12},$${\tilde{x}}^{*}=x_{11}^{\prime }-i{x}_{12}^{\prime }=x_{11}-i{x}_{12}={\tilde{x}}^c$. Let us examine the variances of sum and difference. Let

$$\tilde{u}=\frac{{\tilde{x}}_1-E\left[{\tilde{x}}_1\right]}{\sqrt{Var\left(\widetilde{x_1}\right)}}=\frac{\tilde{x}}{\sqrt{{\sigma }_{11}}},\tilde{v}=\frac{{\tilde{y}}_1-E\left[{\tilde{y}}_1\right]}{\sqrt{\mathrm{Var}\left({\tilde{y}}_1\right)}}=\frac{\tilde{y}}{\sqrt{{\sigma }_{22}}}.$$

Then,

$$\begin{array}{cc}\hfill \mathrm{Var}(\tilde{u}+\tilde{v})& =E\left[(\tilde{u}+\tilde{v}-E(\tilde{u}+\tilde{v})){(\tilde{u}+\tilde{v}-E(\tilde{u}+\tilde{v}))}^{*}\right]\hfill \\ \hfill & =E[\frac{\tilde{x}{\tilde{x}}^{*}}{{\sigma }_{11}}+\frac{\tilde{y}{\tilde{y}}^{*}}{{\sigma }_{22}}+\frac{\tilde{x}{\tilde{y}}^{*}+\tilde{y}{\tilde{x}}^{*}}{\sqrt{{\sigma }_{11}{\sigma }_{22}}}]\hfill \end{array}$$

This can be simplified as the following:

$$\begin{array}{cc}\hfill \mathrm{Var}(\tilde{u}+\tilde{v})& =2+2\frac{[\mathrm{Cov}(\tilde{x},\tilde{y})+\mathrm{Cov}(\tilde{y},\tilde{x})]}{2\sqrt{{\sigma }_{11}{\sigma }_{22}}}\ge 0\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill \mathrm{Var}(\tilde{u}-\tilde{v})& =\frac{E\left[\tilde{x}{\tilde{x}}^{*}\right]}{{\sigma }_{11}}+\frac{E\left[\tilde{y}{\tilde{y}}^{*}\right]}{{\sigma }_{22}}-\frac{E[\tilde{x}{\tilde{y}}^{*}+\tilde{y}{\tilde{x}}^{*}]}{\sqrt{{\sigma }_{11}{\sigma }_{22}}}\hfill \\ \hfill & =2-2\frac{[\mathrm{Cov}(\tilde{x},\tilde{y})+\mathrm{Cov}(\tilde{y},\tilde{x})]}{2\sqrt{{\sigma }_{11}{\sigma }_{22}}}\ge 0\hfill \end{array}$$

(30)

From $\left(29\right)$ and $\left(30\right)$ we have

$$-1\le \frac{\mathrm{Cov}(\tilde{x},\tilde{y})+\mathrm{Cov}(\tilde{y},\tilde{x})}{2\sqrt{{\sigma }_{11}{\sigma }_{22}}}\le 1.$$

(31)

Let us examine the quantity, $\mathrm{Cov}(\tilde{x},\tilde{y})+\mathrm{Cov}(\tilde{y},\tilde{x})=E\left[\tilde{x}{\tilde{y}}^{*}\right]+E\left[\tilde{y}{\tilde{x}}^{*}\right].$ Note that

$$\begin{array}{cc}\hfill E\left[\tilde{x}{\tilde{y}}^{*}\right]& =E\left[(x_{11}+i{x}_{12})(y_{11}-i{y}_{12})\right]=E[x_{11}{y}_{11}+x_{12}{y}_{12}+i(x_{12}{y}_{11}-x_{11}{y}_{12})]\hfill \\ \hfill E\left[\tilde{y}{\tilde{x}}^{*}\right]& =E[x_{11}{y}_{11}+x_{12}{y}_{12}+i(y_{12}{x}_{11}-y_{11}{x}_{12})]\Rightarrow \hfill \\ \hfill E\left[\tilde{x}{\tilde{y}}^{*}\right]& +E\left[\tilde{y}{\tilde{x}}^{*}\right]=\mathrm{Cov}(\tilde{x},\tilde{y})+\mathrm{Cov}(\tilde{y},\tilde{x})=2E[x_{11}{y}_{11}+x_{12}{y}_{12}]=2E[\Re \left(\tilde{x}{\tilde{y}}^{*}\right)]\hfill \end{array}$$

Hence,

$$\begin{array}{cc}\hfill \frac{1}{2}\left[\frac{\mathrm{Cov}(\tilde{x},\tilde{y})+\mathrm{Cov}(\tilde{y},\tilde{x})}{\sqrt{{\sigma }_{11}{\sigma }_{22}}}\right]& =E\left[\frac{x_{11}{y}_{11}+x_{12}{y}_{12}}{\sqrt{{\sigma }_{11}{\sigma }_{22}}}\right]\hfill \\ \hfill & =E\left[\frac{\Re \left(\tilde{x}{\tilde{y}}^{*}\right)}{\sqrt{{\sigma }_{11}{\sigma }_{22}}}\right].\hfill \end{array}$$

Therefore, we may define the correlation coefficient in the complex domain, denoted by $\tilde{r}$, as the following:

$$\tilde{r}=\frac{\Re \left(\mathrm{Cov}(\tilde{x},\tilde{y})\right)}{\sqrt{{\sigma }_{11}{\sigma }_{22}}}\Rightarrow -1\le \tilde{r}\le 1$$

(32)

where $\Re (·)$ denotes the real part in $(·)$. Also, note that $|\tilde{x}{\tilde{y}}^{*}{|}^2=|\tilde{x}{|}^2{|\tilde{y}|}^2$. This will motivate us to examine the dot product of two vectors in the complex domain and the Cauchy–Schwarz inequality in the complex domain.

5.1. The Cauchy–Schwarz Inequality in the Complex Domain

Let $\tilde{X},{\tilde{X}}^{\prime }=[{\tilde{x}}_1,...,{\tilde{x}}_p]$, and $\tilde{Y},{\tilde{Y}}^{\prime }=[{\tilde{y}}_1,...,{\tilde{y}}_p]$ be two $p\times 1$ vectors in the complex domain. We will define the dot product between $\tilde{X}$ and $\tilde{Y}$, and it will be denoted and defined as the following: $\tilde{X}\odot \tilde{Y}={\tilde{X}}^{*}\tilde{Y}$ and

$$\begin{array}{cc}\hfill |\tilde{X}\odot \tilde{Y}{|}^2& =|{\tilde{X}}^{*}\tilde{Y}{|}^2=|{\tilde{x}}_1^c{\tilde{y}}_1{|}^2+...+{\left|{\tilde{x}}_p^c{\tilde{y}}_p\right|}^2\hfill \\ \hfill & =|{\tilde{x}}_1{|}^2|{\tilde{y}}_1{|}^2+...+|{\tilde{x}}_p{|}^2{\left|{\tilde{y}}_p\right|}^2\hfill \\ \hfill & =(x_{11}^2+x_{12}^2)(y_{11}^2+y_{12}^2)+...+(x_{p1}^2+x_{p2}^2)(y_{p1}^2+y_{p2}^2)\hfill \\ \hfill & \le (x_{11}^2+x_{12}^2+...+x_{p1}^2+x_{p2}^2)(y_{11}^2+y_{12}^2+...+y_{p1}^2+y_{p2}^2)\hfill \\ \hfill & \Rightarrow |\tilde{X}\odot \tilde{Y}|\le |\tilde{X}\left|\right|\tilde{Y}|.\hfill \end{array}$$

(33)

Thus, the Cauchy–Schwarz inequality holds for the complex domain also.

5.2. Minimum-Variance Unbiased Estimators in the Complex Domain

In the class of linear estimators $A^{*}\tilde{X}$ for the parametric function $g\left(\theta \right)$, which linear function is the minimum-variance unbiased estimator for $g\left(\theta \right)$? Let

$$A=\left[\begin{array}{c}{a}_1\\ ⋮\\ a_p\end{array}\right],\tilde{X}=\left[\begin{array}{c}{\tilde{x}}_1\\ ⋮\\ {\tilde{x}}_p\end{array}\right]\Rightarrow E\left[A^{*}\tilde{X}\right]=A^{*}E\left[\tilde{X}\right]=A^{*}\tilde{\mu }=g\left(\theta \right),$$

and $\mathrm{Var}\left(A^{*}\tilde{X}\right)=A^{*}\mathrm{\Sigma }A,E\left[\tilde{X}\right]=\tilde{\mu }$ where $\mathrm{\Sigma }$ is the covariance matrix in $\tilde{X}$. Our aim here is to minimize $A^{*}\mathrm{\Sigma }A$ subject to the constraint $A^{*}\tilde{\mu }=g\left(\theta \right)$ (given). Let $\lambda$ be a Lagrangian multiplier and let $\tilde{w}=A^{*}\mathrm{\Sigma }A-\lambda (A^{*}\tilde{\mu }-g\left(\theta \right))$. Then,

$$\frac{\partial \tilde{w}}{\partial A}=O\Rightarrow 2\mathrm{\Sigma }A-2\lambda \tilde{\mu }=O\Rightarrow \mathrm{\Sigma }A=\lambda \tilde{\mu }.$$

(34)

That is, $A^{*}\mathrm{\Sigma }A=\lambda A^{*}\tilde{\mu }=\lambda g\left(\theta \right)$. Hence, $g\left(\theta \right)$ multiplied by the minimum value of $\lambda$ gives the minimum of $A^{*}\mathrm{\Sigma }A$. From $\left(34\right)$,

$$A=\lambda {\mathrm{\Sigma }}^{-1}\tilde{\mu }\Rightarrow g\left(\theta \right)=A^{*}\tilde{\mu }={\lambda }^c{\tilde{\mu }}^{*}{\mathrm{\Sigma }}^{-1}\tilde{\mu }\Rightarrow {\lambda }^c=\frac{g\left(\theta \right)}{{\tilde{\mu }}^{*}{\mathrm{\Sigma }}^{-1}\tilde{\mu }}.$$

Then,

$$\lambda g\left(\theta \right)=\frac{g{g}^{*}}{{\tilde{\mu }}^{*}{\mathrm{\Sigma }}^{-1}\tilde{\mu }}=\frac{{\left|g\right|}^2}{{\tilde{\mu }}^{*}{\mathrm{\Sigma }}^{-1}\tilde{\mu }}$$

is the minimum value of the variance of our linear estimator. Hence, the minimum-variance unbiased estimator is

$$T\left(\tilde{X}\right)=\frac{g}{{\tilde{\mu }}^{*}{\mathrm{\Sigma }}^{-1}\tilde{\mu }}\left[{\tilde{\mu }}^{*}{\mathrm{\Sigma }}^{-1}\tilde{X}\right].$$

Note also that

$$g=A^{*}\tilde{\mu }=\left(A^{*}{\mathrm{\Sigma }}^{\frac{1}{2}}\right)\left({\mathrm{\Sigma }}^{-\frac{1}{2}}\tilde{\mu }\right)\le \sqrt{A^{*}\mathrm{\Sigma }A}\sqrt{\tilde{\mu }{\mathrm{\Sigma }}^{-1}\tilde{\mu }}\Rightarrow \sqrt{A^{*}\mathrm{\Sigma }A}\ge \frac{g}{{\tilde{\mu }}^{*}{\mathrm{\Sigma }}^{-1}\tilde{\mu }}.$$

The first inequality follows from the inequality established in Section 5.1.

5.3. Cramer–Rao-Type Inequality in the Complex Domain

Let ${\tilde{x}}_1,...,{\tilde{x}}_n$ be a simple random sample from the population designated by the density $f\left({\tilde{x}}_j\right)$, where ${\tilde{x}}_j,j=1,...,n$ are independently and identically distributed (iid). Then, the joint density $L={\prod }_{j=1}^{n}f\left({\tilde{x}}_j\right)$. Let $T({\tilde{x}}_1,...,{\tilde{x}}_n)$, denoted as $T\left(\tilde{X}\right)$, be a statistic with the expected value $g\left(\theta \right)$. That is, $E\left[T\right]={\int }_{\tilde{X}}TL\mathrm{d}\tilde{X}=g\left(\theta \right)$. Differentiating with respect to $\theta$, we have

$$\frac{\partial g\left(\theta \right)}{\partial \theta }={\int }_{\tilde{X}}T\left(\frac{\partial L}{\partial \theta }\right)\mathrm{d}\tilde{X}={\int }_{\tilde{X}}T\left(\frac{\partial lnL}{\partial \theta }\right)L\mathrm{d}\tilde{X}=E\left[T\left(\frac{\partial lnL}{\partial \theta }\right)\right]$$

(35)

where we have assumed that the support of $\tilde{X}$ is free of $\theta$ and differentiation inside the integral is valid. But, from the total integral being one, we have

$${\int }_{\tilde{X}}L\mathrm{d}\tilde{X}=1\Rightarrow {\int }_{\tilde{X}}\left(\frac{\partial lnL}{\partial \theta }\right)L\mathrm{d}\tilde{X}=0\Rightarrow E\left[\frac{\partial lnL}{\partial \theta }\right]=0.$$

(36)

From $\left(35\right)$ and $\left(36\right)$, we have $E\left[T\left(\frac{\partial lnL}{\partial \theta }\right)\right]=\mathrm{Cov}(T,\frac{\partial lnL}{\partial \theta })$ because for any two scalar random variables u and v real, or $\tilde{u}$ and $\tilde{v}$ in the complex domain, $\mathrm{Cov}(u,v)=E\left[\right(u-E\left(u\right)\left)\right(v-E\left(v\right)\left)\right]=E\left[u\right(v-E\left(v\right)\left)\right]=E\left[\right(u-E\left(u\right)\left)v\right]$ and the corresponding results in the complex domain also hold. Therefore, since $E\left[\frac{\partial lnL}{\partial \theta }\right]=0,E\left[T\left(\frac{\partial lnL}{\partial \theta }\right)\right]=\mathrm{Cov}(T,\frac{\partial lnL}{\partial \theta })$. Then,

$$|\mathrm{Cov}(T,\frac{\partial lnL}{\partial \theta })|\le \sqrt{\mathrm{Var}\left(T\right)}\sqrt{\mathrm{Var}\left(\frac{\partial lnL}{\partial \theta }\right)}\Rightarrow \mathrm{Var}\left(\tilde{T}\right)\ge \frac{|\frac{\partial g}{\partial \theta }{|}^2}{\mathrm{Var}\left(\frac{\partial lnL}{\partial \theta }\right)}.$$

(37)

Note that

$$\mathrm{Var}\left(\frac{\partial lnL}{\partial \theta }\right)=E\left[{\left(\frac{\partial lnL}{\partial \theta }\right)}^2\right]=n\mathrm{Var}\left(\frac{\partial lnf}{\partial \theta }\right)=nE\left[{\left(\frac{\partial lnL}{\partial \theta }\right)}^2\right].$$

(38)

This shows that the Cramer–Rao-type inequality holds in the complex domain also.

5.4. Least Square Estimation in Linear Models in the Complex Domain

Let us examine whether the least square procedure, in the class of linear models, holds in the complex domain also. Let ${\tilde{x}}_1,...,{\tilde{x}}_k$ be preassigned complex numbers or observations on k random variables in the complex domain. Let $\tilde{y}$ be a scalar complex variable. Then, a linear model of ${\tilde{x}}_1,...,{\tilde{x}}_k$ for predicting $\tilde{y}$, can be of the following form:

$$\tilde{y}=a_0^c+{\beta }^{*}\tilde{X},\beta =\left[\begin{array}{c}{a}_1\\ ⋮\\ a_k\end{array}\right],\tilde{X}=\left[\begin{array}{c}{\tilde{x}}_1\\ ⋮\\ {\tilde{x}}_k\end{array}\right].$$

But, in order to predict $\tilde{y}$ by using linear predictors we must know the conditional distribution of $\tilde{y}$, given ${\tilde{x}}_1,...,{\tilde{x}}_k$, and also the conditional expectation must be linear in ${\tilde{x}}_1,...,{\tilde{x}}_k$. If the conditional distribution is not known, then we may use a distribution-free procedure. One such procedure is the estimation procedure by using the least square method. In this method, we set up a corresponding model of the following form for the j-th observation on $\tilde{y}$, namely, ${\tilde{y}}_j$, for $j=1,...,n$, where $n>k+1$ is the sample size.

$$\tilde{y}=a_0+a_1{\tilde{x}}_{1j}+...+a_k{\tilde{x}}_{kj}+{ϵ}_j,j=1,...,n$$

(39)

corresponding to the linear model in the real case, where ${ϵ}_j$ is the random part or the sum total contributions coming from unknown factors, corresponding to ${\tilde{y}}_j$. Then, if we sum up the observations and divide by n we obtain the sample averages $\overline{\tilde{y}},{\overline{\tilde{x}}}_r,r=1,...,k$, where, for example, ${\overline{\tilde{x}}}_r=\frac{1}{n}{\sum }_{j=1}^n{\tilde{x}}_{rj}$. Then, from $\left(39\right)$, we have

$$\begin{array}{cc}\hfill \overline{\tilde{y}}=a_0+a_1{\overline{\tilde{x}}}_1+...+a_k{\overline{\tilde{x}}}_k+0& \Rightarrow a_0=\overline{\tilde{y}}-a_1{\overline{\tilde{x}}}_1-...-a_k{\overline{\tilde{x}}}_k\hfill \\ \hfill & \Rightarrow c_0^c=\overline{\tilde{y}}-a_1^c{\overline{\tilde{x}}}_1-...-a_k^c{\overline{\tilde{x}}}_k.\hfill \end{array}$$

(40)

We have taken the error sum as zero without much loss of generality. Since $a_0$ is available from $\left(40\right)$, we may rewrite $\left(39\right)$ as follows:

$${\tilde{y}}_j-\overline{\tilde{y}}=a_1({\tilde{x}}_{1j}-{\overline{\tilde{x}}}_1)+...+a_k({\tilde{x}}_{kj}-{\overline{\tilde{x}}}_k)+{ϵ}_j-\overline{ϵ},j=1,...,n.$$

(41)

We may write all the equations in $\left(41\right)$ together as $\tilde{U}=\tilde{Z}\beta +e$, where

$$\tilde{U}=\left[\begin{array}{c}{\tilde{y}}_1-\overline{\tilde{y}}\\ ⋮\\ {\tilde{y}}_n-\overline{\tilde{y}}\end{array}\right],\tilde{Z}=\left[\begin{array}{ccc}{\tilde{x}}_{11}-{\overline{\tilde{x}}}_1& ...& {\tilde{x}}_{k1}-{\overline{\tilde{x}}}_k\\ ⋮& ...& ⋮\\ {\tilde{x}}_{1n}-{\overline{\tilde{x}}}_1& ...& {\tilde{x}}_{kn}-{\overline{\tilde{x}}}_k\end{array}\right],e=\left[\begin{array}{c}{\tilde{e}}_1\\ ⋮\\ {\tilde{e}}_n\end{array}\right],\beta =\left[\begin{array}{c}{a}_1\\ ⋮\\ a_k\end{array}\right].$$

Note that $\tilde{U}$ is an $n\times 1$ matrix, $\tilde{Z}$ is an $n\times k$ matrix, $\beta$ is a $k\times 1$ matrix, and $\tilde{e}$ is an $n\times 1$ matrix. Then, the sum of squares of the absolute values of the errors is the following:

$${\tilde{e}}^{*}\tilde{e}={(\tilde{U}-\tilde{Z}\beta )}^{*}(\tilde{U}-\tilde{Z}\beta ).$$

(42)

In the least square procedure, we minimize this error sum of squares of the absolute values of the errors, and then, estimate the parameter vector $\beta$. Note that ${(\tilde{U}-\tilde{Z}\beta )}^{*}={\tilde{U}}^{*}-{\beta }^{*}{\tilde{Z}}^{*}$ and

$$\begin{array}{cc}\hfill \frac{\partial {\tilde{e}}^{*}\tilde{e}}{\partial \beta }& =O\Rightarrow O-O-2{\tilde{Z}}^{*}\tilde{U}+2{\tilde{Z}}^{*}\tilde{Z}\beta =O\hfill \\ \hfill \beta & ={\left({\tilde{Z}}^{*}\tilde{Z}\right)}^{-1}{\tilde{Z}}^{*}\tilde{U}\hfill \end{array}$$

(43)

where we have assumed that ${\tilde{Z}}^{*}\tilde{Z}$ is a nonsingular matrix because ${\tilde{x}}_{rj}$’s are preassigned numbers, and hence, $\tilde{Z}$ can be taken as a full-rank matrix with rank $k<n$. Then, the estimated $\beta$, as per the least square estimate, again denoted by $\beta$, is $\beta ={\left({\tilde{Z}}^{*}\tilde{Z}\right)}^{-1}{\tilde{Z}}^{*}\tilde{U}$ and the estimated model for $\tilde{y}$ is $a_0^c+{\beta }^{*}\tilde{X},{\tilde{X}}^{\prime }=[{\tilde{x}}_1,...,{\tilde{x}}_k],{\beta }^{*}={\tilde{U}}^{*}\tilde{Z}{\left({\tilde{Z}}^{*}\tilde{Z}\right)}^{-1}$ or the estimated $\tilde{y}$ is

$$\tilde{y}=a_0^c+{\beta }^{*}\tilde{X}=a_0^c+{\tilde{U}}^{*}\tilde{Z}{\left({\tilde{Z}}^{*}\tilde{Z}\right)}^{-1}\tilde{X}$$

(44)

where $\beta$ is available from $\left(43\right)$ and $a_0^c=\overline{\tilde{y}}-{\beta }^{*}\overline{\tilde{X}}$, $\overline{\tilde{X}}={[{\overline{\tilde{x}}}_1,...,{\overline{\tilde{x}}}_k]}^{\prime }$. This shows that the least square procedure in the complex domain also runs parallel to that in the real domain. If $\tilde{e}$ is assumed to have an n-variate complex Gaussian distribution, then the inference problems also runs parallel to those in the real domain.

6. Concluding Remarks

In this paper, we have introduced vector/matrix differential operators in the complex domain. These differential operators in the complex domain are believed to be new. With the help of these operators, we have examined the optimization of a linear form with Hermitian-form constraint, optimization of a Hermitian form with linear form as well as Hermitian-form constraint, and optimization of a bilinear form with Hermitian-form constraints, where the linear forms and bilinear forms involve vectors and matrices in the complex domain. As applications of these optimization problems, we have extended principal component analysis and canonical correlation analysis to the complex domain. Also extended to the complex domain are the Cramer–Rao inequality, the Cauchy–Schwarz inequality, minimum-variance unbiased estimation, and least square analysis. If we use the general definition of a density $f\left(X\right)$ as a real-valued scalar function such that $f\left(X\right)\ge 0$ in the domain of X and ${\int }_{X}f\left(X\right)\mathrm{d}X=1$, where the argument X may be scalar or vector or matrix or a sequence of matrices in the real or complex domains [8], then the structures of the joint density, marginal density, conditional density, etc., will be parallel to those in the real domain. Then, we will be able to extend Bayesian analysis to the complex domain. One can also explore extending other multivariate statistical techniques such as factor analysis, classification problems, cluster analysis, analysis of variance, analysis of covariance, etc., to the complex domain. These are some of the open problems. Since the likelihood function L is a product of densities at the observed sample point, in the simple random sample case, this L will be a real-valued scalar function. Then, one can extend the maximum likelihood method of estimation to the complex domain. For example, in the p-variate complex Gaussian case, in the case of a simple random sample of size n, we have

$$lnL=c-npln|\tilde{\mathrm{\Sigma }}|-\mathrm{tr}\left({\tilde{\mathrm{\Sigma }}}^{-1}\tilde{S}\right)-\sum _{j=1}^n{({\tilde{X}}_j-\tilde{\mu })}^{*}{\tilde{\mathrm{\Sigma }}}^{-1}({\tilde{X}}_j-\tilde{\mu })$$

where c is a constant, $\tilde{S}$ is the sample sum of squares and cross-products matrix, $\tilde{\mu }$ is the population mean value, and $\tilde{\mathrm{\Sigma }}>O$ is the Hermitian positive definite covariance matrix. We have already established results on vector and matrix differential operators in the complex domain, operating on the trace and determinant involving a Hermitian positive definite matrix. Hence, all the terms in $\frac{\partial }{\partial \tilde{\mu }}[lnL]=O$ and $\frac{\partial }{\partial \tilde{\mathrm{\Sigma }}}[lnL]=O$ are defined and such equations are already solved in our discussion of our operators operating on traces and determinants. Thus, we will see that $\frac{\partial }{\partial \tilde{\mu }}[lnL]=O$ yields the sample average as the estimate/estimator of $\tilde{\mu }$, and $\frac{\partial }{\partial \tilde{\mathrm{\Sigma }}}[lnL]=O$, with the estimate on $\tilde{\mu }$, yields the estimate/estimator of $\tilde{\mathrm{\Sigma }}$ as $\frac{1}{n}\tilde{S}$. One can also examine the maximum likelihood estimation involving other scalar/vector/matrix-variate densities in the complex domain. The above are some of the open problems.