A Mathematical Investigation of a Continuous Covariance Function Fitting with Discrete Covariances of an AR Process ^†

Abstract

In this paper, we want to find a continuous function fitting through the discrete covariance sequence generated by a stationary AR process. This function can be determined as soon as the Yule–Walker equations are found. The procedure consists of two steps. At first the inverse zeros of the characteristic polynomial of the AR process must be fixed. The second step is based on the fact that an AR process can also be seen as a difference equation. By solving this difference equation, it is possible to determine a class of functions from which a candidate for a continuous covariance function can be determined. To analyze if this function is applicable as a positive definite covariance function, it is analyzed mathematically in view of the power spectral density compared to the characteristics of the power spectral density for the discrete covariances. Then it is shown that this function is positive semi-definite. At the end, a simulation of a stationary AR(3) process is elaborated to illustrate the derived properties.

1. Introduction

In geodesy the observations or analyzed signals are often discrete measurements which repeat at regular distances. For example, deformation observations (repeating in time) or data from satellite missions such as GOCE (repeating in time and space). It is a common way to describe regular and equidistant signals by auto regressive moving average (ARMA) processes [1,2,3,4,5].

Within this contribution we focus on the analysis of the AR part. This AR part defines the causal link between an observation and its predecessors. Additionally, Least Squares Collocation (LSC) (see e.g., [6,7,8]) and kriging [9,10] are benefiting from the use of AR processes. For example, the inverse of a covariance matrix, based on AR process, is a band matrix witch bandwidth equals the order of the AR process (see e.g., [11]).

However, to activate the full potential of LSC a continuous covariance function is indispensable. With this function it will be possible to predict a pseudo signal between the observations. Furthermore, it is possible to predict the signal outside the observation field and not only for a multiple of the sampling rate. Moreover, refs. [12,13] used a continuous covariance function to switch from one functional to another (Like using sea level heights to calculate sea level changes which are proportional to the stream velocity).

In this paper, we want to find an analytical description of a continuous function fitting through the discrete sequence of covariances generated by any stationary AR process. This function is derived from the coefficients of the AR process and the discrete covariances using a system of equations with a unique solution. The resulting function should be positive definite, and its spectrum is expected to correspond to the spectrum of the discrete AR process. It will turn out that this function is the continuous solution of the difference equation and correctly interpolates the discrete covariance sequence with appropriate basis functions, indeed following sampling theory/convolution theorem. In the end an example is given by a simulated AR process and the accompanying continuous covariance function as well as the two spectra are estimated.

2. Continuous Covariance Function

To find a suitable covariance function for any stationary AR process the definition of AR processes is a good starting point. Especially the transfer from the AR process to the difference equation approach will lead us to the continuous representation we looked for (The following definitions could be found in [14,15,16,17,18]. Here the notation of [17] is used).

2.1. Construction of a Continuous Covariance Function

The process ${\mathcal{S}}_t$ is called one-dimensional AR process of order p (AR(p) process) if it is described by the recursive equation

$$\begin{array}{c}\hfill {\mathcal{S}}_t={\alpha }_1{\mathcal{S}}_{t-1}+{\alpha }_2{\mathcal{S}}_{t-2}+\dots +{\alpha }_p{\mathcal{S}}_{t-p}+{\mathcal{E}}_t\end{array}$$

(1)

where ${\alpha }_1$, ${\alpha }_2$, ..., ${\alpha }_p$ are the coefficients of the AR process and ${\mathcal{E}}_t$ is a i.i.d. sequence with variance ${\sigma }_{\mathcal{E}}^2$ [17] (p. 58, Equation (3.4.31)). We assume that ${\alpha }_p\ne 0$, as otherwise the AR(p) process is also an AR$(p-1)$ process so that AR(p) is not well-defined (In addition, if ${\alpha }_p=0$ some formulas in this paper cannot be used).

An important quantity is the zeros (${\zeta }_l$) of an AR(p) process defined by the zeros of the characteristic polynomial

$$\begin{array}{cc}\hfill \chi \left(x\right)& =1-{\alpha }_1{x}^1-{\alpha }_2{x}^2-\dots -{\alpha }_p{x}^p\hfill \\ \hfill & =(x-{\zeta }_1)(x-{\zeta }_2)\dots (x-{\zeta }_p),\hfill \end{array}$$

(2)

see e.g., [17] (p. 58, Equation (3.4.32)).

An alternative definition is given by the auxiliary equation if we interpret the AR process as a difference equation which has the general solution (see [19], p. 134, Equation (3.33))

$$\begin{array}{cc}\hfill b\left(x\right)& =x^p-{\alpha }_1{x}^{p-1}-{\alpha }_2{x}^{p-2}-\dots -{\alpha }_p\hfill \\ \hfill & =(x-p_1)(x-p_2)\dots (x-p_p).\hfill \end{array}$$

(3)

These zeros $p_l$ only occur as real values or in pairs of complex conjugated zeros. Bear in mind that the zeros of the characteristic polynomial from Equation (2) are linked to the zeros of the auxiliary equation (cf. Equation (3)) by ${\zeta }_l=1/p_l$. AR processes are stationary if and only if the ${\zeta }_l$ are outside the unit circle, such that $|p_l|<1$. In the following we will restrict to $p_l$ for simplicity.

With this definitions in mind, the discrete covariances ${\mathsf{\Sigma }}_j$ of an AR(p) process, are linked with each other by the Yule–Walker (YW) equations (see e.g., [17], p. 59, Equation (3.4.36)), i.e.,

$$\begin{array}{ccc}\hfill {\mathsf{\Sigma }}_0& ={\alpha }_1{\mathsf{\Sigma }}_1+{\alpha }_2{\mathsf{\Sigma }}_2+\dots +{\alpha }_p{\mathsf{\Sigma }}_p+{\sigma }_{\mathcal{E}}^2\hfill & \end{array}$$

(4)

$$\begin{array}{ccc}\hfill {\mathsf{\Sigma }}_j& ={\alpha }_1{\mathsf{\Sigma }}_{|j-1|}+{\alpha }_2{\mathsf{\Sigma }}_{|j-2|}+\dots +{\alpha }_p{\mathsf{\Sigma }}_{|j-p|}\hfill & \hfill \mathrm{if}j\ne 0.\end{array}$$

(5)

The YW equation of higher order than 0 (Equation (5)) are basically homogeneous difference equations of order p,

$$\begin{array}{c}\hfill {\mathsf{\Sigma }}_j-{\alpha }_1{\mathsf{\Sigma }}_{|j-1|}-{\alpha }_2{\mathsf{\Sigma }}_{|j-2|}-\dots -{\alpha }_p{\mathsf{\Sigma }}_{|j-p|}=0.\end{array}$$

(6)

The general solution to the difference equation can be expressed by the powers of the zeros $p_l$ of the auxiliary Equation (3). The particular solution is fixed by the boundary conditions using the discrete covariances determined from the YW equations (cf. Equation (5)),

$$\begin{array}{c}\hfill {\mathsf{\Sigma }}_j=\sum _{l=1}^p{A}_l{p}_l^{\left|j\right|}.\end{array}$$

(7)

Here $A_l$ are coefficients which are complex if and only if the corresponding $p_l$ is complex. Furthermore, if there is a pair of complex conjugated $p_l$ then $A_l$ occur also as complex conjugated pairs (see e.g., [18], p. 134, Equation (3.5.44) or [19], p.163, f.).

At this point a new but now continuous function is defined, which can be seen as the continuous covariance function $\gamma \left(h\right):\mathbb{R}\to \mathbb{R}$ for any AR(p) process,

$$\begin{array}{c}\hfill \gamma \left(h\right):=\sum _{l=1}^p{A}_l{p}_l^{\left|h\right|}.\end{array}$$

(8)

Here $A_l$ and $p_l$ are the same as in Equation (7), but the domain changed. $j\in {\mathbb{N}}_0$ is replaced by $h\in \mathbb{R}$.

Attentive readers will have noticed that $\gamma \left(h\right)$ is complex if any $p_l\in {\mathbb{R}}^{-}$. Then $\gamma \left(h\right)\in \mathbb{C}$ for all $h\notin {\mathbb{N}}_0$. One important convention that will help with this inconsistency is the use of the real part $\mathrm{Re}\left(\gamma \left(h\right)\right)$ of the complex function (see Figure 1). This condition will not have any impact if $\gamma \left(h\right)$ is real (what is mostly the case), and furthermore a covariance function for real valued signals is defined to be a function in $\mathbb{R}$ not in $\mathbb{C}$.

2.2. Properties of the Continuous Covariance Function

Since with $\gamma \left(h\right)$ from Equation (8) a suitable function is found to fit through the discrete covariances from Equation (7), we want to analyze the power spectral densities of the continuous and the discrete functions. On this basis we can demonstrate that the Fourier transform of the continuous covariance function is positive semi-definite.

Initially the problem restricted to AR processes of order 1 and order 2 with two complex conjugated zeros. On the one hand any AR(p) process can be dissected into a product of AR(1) and AR(2) processes. This is a linear function, so the power spectral function is the product of the corresponding AR(1) processes and AR(2) processes. On the other hand, Equation (8) shows that the covariance function is a weighted sum of the real valued zeros, or pairs of complex conjugated zeros. So, the zeros are also in a linear relation, and so is the Fourier transform. So, it is only necessary to examine the spectrum and the Fourier transform for the first order AR process and second order AR process with two complex conjugated zeros.

For these specific types of AR processes there is an analytical solution to switch from AR coefficients ${\alpha }_l$ to the zeros $p_l$ (see [20]),

$$\begin{array}{cc}\hfill \mathrm{for}\mathrm{order}p=1& {\alpha }_1=p_1\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill \mathrm{and}\mathrm{for}\mathrm{order}p=2& \left\{\begin{array}{c}{\alpha }_1=p_1+p_2\hfill \\ {\alpha }_2=-p_1{p}_2\hfill \end{array}\right.\mathrm{with}{p}_1=p_2^{\ast }.\hfill \end{array}$$

(10)

2.2.1. Power Spectral Density

The power spectral density for an AR(p) process is well known (see e.g., [16], p. 244, Equation (11.20)) and is described by the transfer function

$$\begin{array}{c}\hfill {\mathcal{H}}^2\left(\nu \right)=\frac{{\sigma }_{\mathcal{E}}^2}{|1-{\sum }_{l=1}^p{\alpha }_l{e}^{-i2\pi \nu l}{|}^2}.\end{array}$$

(11)

In consideration of Equations (9) and (10) the power spectral density for any AR(1) process and AR(2) process with complex conjugated zero can be calculated explicitly. So, the power spectral density AR(1) process is generated via

$$\begin{array}{cc}\hfill {\mathcal{H}}^2\left(\nu \right)& =\frac{{\sigma }_{\mathcal{E}}^2}{1-2p_{1}cos\left(2\pi \nu \right)+p_1^2}.\hfill \end{array}$$

(12)

For the AR(2) process with zeros $p_1=p_2^{\ast }$ the power spectral density is a little more complicated and turns out to be

$$\begin{array}{c}\hfill {\mathcal{H}}^2\left(\nu \right)=\\ \hfill \frac{{\sigma }_{\mathcal{E}}^2}{1-2(p_1+p_2)cos\left(2\pi \nu \right)+p_1{p}_2\left(2+2cos\left(4\pi \nu \right)-(p_1+p_2)cos\left(2\pi \nu \right)+p_1{p}_2\right)+p_1^2+p_2^2}.\end{array}$$

(13)

Using the Fourier transform of the covariance function $\gamma \left(h\right)$

$$\begin{array}{c}\hfill \mathsf{\Gamma }\left(\nu \right):=\mathcal{F}\left\{\gamma \left(h\right)\right\}\left(\nu \right)=\sum _{l=1}^p{A}_l\frac{-2ln\left(p_l\right)}{{\left(ln\left(p_l\right)\right)}^2+{\left(2\pi \nu \right)}^2}\end{array}$$

(14)

is an alternative way to derive the power spectral density (For further derivations of the Fourier transform see Appendix A). However, ${\mathcal{H}}^2\left(\nu \right)\ne \mathsf{\Gamma }\left(\nu \right)$. Especially Equation (11) shows ${\mathcal{H}}^2(\nu$) as a function whose only parameter $\nu$ arise as power of the complex function $e^{-i2\pi l}$. Therefore ${\mathcal{H}}^2\left(\nu \right)$ is a repetitive function with period 1. In contrast, Equation (14) shows $\mathsf{\Gamma }\left(\nu \right)$ is aperiodic function with $\underset{\nu \to \infty }{lim}\mathsf{\Gamma }\left(\nu \right)=0$. To understand this circumstance two theorems are of importance:

• The discrete covariances of an AR(p) process (${\mathsf{\Sigma }}_j$) are equivalent to the product of the Dirac comb with the continuous covariance function $\gamma \left(h\right)$.
• The convolution theorem shows that multiplication in time domain results in convolution in frequency domain.

Combining these two pieces of information shows indeed that ${\mathcal{H}}^2\left(\nu \right)\ne \mathsf{\Gamma }\left(\nu \right)$ but

$$\begin{array}{c}\hfill {\mathcal{H}}^2\left(\nu \right)=\mathsf{\Gamma }\left(\nu \right)\ast \left(\sum _{k=-\infty }^{\infty }\delta (x-k)\right)\end{array}$$

(15)

where ${\sum }_{k=-\infty }^{\infty }\delta (x-k)$ is the Dirac comb of distance 1 and $\mathsf{\Gamma }\left(\nu \right)\ast \left({\sum }_{k=-\infty }^{\infty }\delta (x-k)\right)$ is the convolution of $\mathsf{\Gamma }\left(\nu \right)$ and the Dirac comb. The transitions from continuous functions to discrete sequences as well as the resulting Fourier transforms are shown in Figure 2 (For a more detailed method of calculation for AR(1) and AR(2) processes see Appendix C).

2.2.2. Positive Semi-Definite Function

Equation (15) shows that knowing if ${\mathcal{H}}^2\left(\nu \right)$ is positive semi-definite is not sufficient to guarantee that the Fourier transform of the continuous function $\gamma \left(h\right)$ is positive semi-definite too. Therefore, the explicit Fourier transforms of the AR(1) and AR(2) process are derived here. For the case of the AR(1) process it is easy to see that for the Fourier transform of the covariance function $\gamma \left(h\right)$

$$\begin{array}{c}\hfill \mathsf{\Gamma }\left(\nu \right)=\frac{{\sigma }_{\mathcal{E}}^2}{1-p^2}\frac{-2ln\left(p\right)}{{\left(ln\left(p\right)\right)}^2+{\left(2\pi \nu \right)}^2}>0\forall \nu \in \mathbb{R}.\end{array}$$

(16)

holds (For the derivation of this formula see Appendix B.1). Neither the squared terms could be less than 0 nor $1-p^2$ or $-ln\left(p\right)$ due to the fact that p lies within the unit circle.

For the AR(2) process things are not that obvious. In Appendix B.2 it is demonstrated that the Fourier transform of $\gamma \left(h\right)$ is

$$\begin{array}{cc}\hfill \mathsf{\Gamma }\left(\nu \right)& =2\mathrm{Re}\left(\frac{-{\sigma }_{\mathcal{E}}^2{p}_1}{(p_2-p_1)(1-p_1^2)(1-p_1{p}_2)}\frac{-2ln\left(p_1\right)}{{(ln\left(p_1\right))}^2+{\left(2\pi \nu \right)}^2}\right).\hfill \end{array}$$

(17)

To work with complex valued fractions, it is necessary to eliminate the imaginary part in the denominator. This is done by multiplying each term of the sum with the complex conjugated denominator divided by itself. Afterwards it is simple to pick the real part. To simplify the formula, we use the polar coordinates $p_1=r{e}^{i\varphi }$, and $p_2=r{e}^{-i\varphi }$ with $0<r<1$ and $0\le \varphi \le \pi$. So, it can be shown that the numerator is positive ($\mathsf{\Gamma }\left(\nu \right)\ge 0$) if and only if

$$\begin{array}{cc}\hfill \underset{:=f\left(r\right)}{\underbrace{-ln\left(r\right)coth(-ln(r\left)\right)}}& \ge \underset{:=g\left(\varphi \right)}{\underbrace{\varphi cot\left(\varphi \right)}}\hfill \end{array}$$

(18)

The function $f\left(r\right)$ and $g\left(\varphi \right)$ are displayed in Figure 3 and Figure 4 (Since $g\left(\varphi \right)=g(-\varphi )$, the negative values of $\varphi$ are not needed). On the one hand $f\left(r\right)$ is a declining function with infimum 1. On the other hand, the function $g\left(\varphi \right)$ is also declining, with a maximum of 1. Thus, infimum $f\left(r\right)\ge maxg\left(\varphi \right)$, which evaluate that Equation (18) is always satisfied.

3. Simulation

To visualize the results from Section 2 of an AR(3) process with two complex conjugated zeros was generated as an example. First to guarantee stationarity the roots are chosen as

$$\begin{array}{c}\hfill \left[\begin{array}{c}{p}_1\\ p_2\\ p_3\end{array}\right]=\left[\begin{array}{c}-0.252-0.126i\\ -0.252+0.126i\\ 0.306\end{array}\right].\end{array}$$

Let the variance of the white noise be ${\sigma }_{ϵ}^2$ = 1. After deriving the coefficients ${\alpha }_l$, $l\in \{1,2,3\}$, using Equation (3), we estimate the discrete covariances (${\mathsf{\Sigma }}_j$) of the AR(3) process by the reorganized YW equations (see [21], p. 32, Equation (183)). With Equation (8) a continuous function $\gamma \left(h\right)$ is fitted through the discrete covariances (see the left upper corner of Figure 5). Subsequently the power spectral density is set first by Equation (11) using the coefficients ${\alpha }_l$ and secondly by Equation (14). For the second step the coefficients $A_l$ are estimated by solving the system of equations

$$\begin{array}{c}\hfill \left[\begin{array}{c}{\mathsf{\Sigma }}_0\\ {\mathsf{\Sigma }}_1\\ {\mathsf{\Sigma }}_2\end{array}\right]=\left[\begin{array}{ccc}1& 1& 1\\ p_1& p_2& p_3\\ p_1^2& p_2^2& p_3^2\end{array}\right]\left[\begin{array}{c}{A}_1\\ A_2\\ A_3\end{array}\right].\end{array}$$

(19)

Here the zeros $p_l$ and covariances ${\mathsf{\Sigma }}_l$ are known. Each row represents Equation (7) for $l\in \{1,2,3\}$. The coefficients $A_l$ are used in Equation (8) to estimate the power spectral density of the continuous covariance function. It must be mentioned that this example is an extreme one where the Fourier transform of the continuous covariance function has high values for frequencies higher the Nyquist frequency ${\nu }_n=0.5$. The Fourier transform of the continuous covariance function is not periodic at all (compare right upper corner of Figure 5). However, periodicity is the characteristic of the spectral density of a discrete AR process. Therefore, the periodicity is a result of the convolution of the Dirac comb and $\mathsf{\Gamma }\left(\nu \right)$.

4. Conclusions and Outlook

In this paper, it was shown that the choice of a valid continuous covariance function for AR(p) processes is given by the function

$$\begin{array}{c}\hfill \gamma \left(h\right)=\sum _{l=1}^p{A}_l{p}_l^{\left|h\right|}.\end{array}$$

Here $h\in \mathbb{R}$ is the lag, $p_l$ are the roots of the characteristic polynomial, and $A_l$ follows from the unique solution of Equation (14) for p arbitrarily chosen discrete covariances ${\mathsf{\Sigma }}_{j_1}$, ${\mathsf{\Sigma }}_{j_2}$, ..., ${\mathsf{\Sigma }}_{j_p}$ (with $j_i\in {\mathbb{N}}_0$, and $j_i\ne j_k\iff i\ne k$):

$$\begin{array}{c}\hfill \left[\begin{array}{c}{A}_1\\ A_2\\ \dots \\ A_p\end{array}\right]={\left[\begin{array}{cccc}{p}_1^{j1}& p_2^{j1}& \dots & p_p^{j1}\\ p_1^{j2}& p_2^{j2}& \dots & p_p^{j2}\\ \dots & \dots & \dots & \dots \\ p_1^{jp}& p_2^{jp}& \dots & p_p^{jp}\end{array}\right]}^{-1}\left[\begin{array}{c}{\mathsf{\Sigma }}_{j_1}\\ {\mathsf{\Sigma }}_{j_2}\\ \dots \\ {\mathsf{\Sigma }}_{j_p}\end{array}\right].\end{array}$$

Due to the convolution theorem, the power spectral density of $\gamma \left(h\right)$ might be different to the power spectral density of the discrete AR(p) process. Nevertheless, the proof was given that $\gamma \left(h\right)$ is still positive semi-definite (cf. Section 2.2.2), and consequently meets all conditions for a suitable covariance function. The Fourier transforms $\mathsf{\Gamma }\left(\nu \right)$ and ${\mathcal{H}}^2\left(\nu \right)$ may not vary much for $\nu \in [-1,1]$ and the simulation (see Section 3) is an extreme example. Anyway $\gamma \left(h\right)$ is an exponential function, so it is easy to use it as functional for covariance function propagation or LSC.

In further works the continuous covariance function $\gamma \left(h\right)$ could be extended to a function for an autoregressive moving average (ARMA) process to examine its properties. It is not yet demonstrated that the oscillation of $\mathrm{Re}\left(\gamma \right(h\left)\right)$ leads towards the minimal frequency if there is a real negative zero ($p_l=0$ for any l).