Using Subspace Algorithms for the Estimation of Linear State Space Models for Over-Differenced Processes

Abstract

Subspace algorithms like canonical variate analysis (CVA) are regression-based methods for the estimation of linear dynamic state space models. They have been shown to deliver accurate (consistent and asymptotically equivalent to quasi-maximum likelihood estimation using the Gaussian likelihood) estimators for stably invertible stationary autoregressive moving average (ARMA) processes. These results use the assumption that there are no zeros of the spectral density on the unit circle corresponding to the state space system. In this technical study, we consider vector processes made stationary by applying differencing to all variables, ignoring potential co-integrating relations. This leads to spectral zeros violating the above mentioned assumptions. We show consistency for the CVA estimators, closing a gap in the literature. However, a simulation exercise shows that over-differencing (while leading to consistent estimation of the transfer function) also complicates inference for CVA estimators, not just maximum likelihood-based estimators. This is also demonstrated in a real-world data example. The result also applies to seasonal differencing. The present paper hence suggests working with original data, not working in differences.

1. Introduction

Subspace algorithms such as the canonical variate analysis (CVA) (Larimore, 1983) are used for the estimation of linear dynamical state space systems for (multivariate) time series. CVA is popular since it is numerically cheap, consisting of a series of regressions, asymptotically equivalent to quasi-maximum likelihood estimation (using the Gaussian likelihood; qMLE) for stationary processes, and it is robust to the existence of simple unit roots (see Bauer, 2005 for a survey). It has also been shown to provide consistent estimates for GARCH effects being present in the innovation process (Bauer, 2008) and even fractionally integrated processes when the system order tends to infinity with appropriate sample sizes (Bauer, 2018). In addition to asymptotic properties, recently, finite sample properties have also been investigated for subspace algorithms, which also apply to CVA (He et al., 2025; see also Tsiamis et al., 2023).

The algorithm fits a state space system in innovation form

$$y_t=C{x}_t+{\epsilon }_t,x_{t+1}=A{x}_t+B{\epsilon }_t,t\in \mathbb{Z},$$

(1)

to an observed time series $y_t\in {\mathbb{R}}^s,t=1,\dots ,T$. Here $A\in {\mathbb{R}}^{n\times n},B\in {\mathbb{R}}^{n\times s}, C\in {\mathbb{R}}^{s\times n}$ define the state space system with system order n. In this paper, we will assume that the system is minimal (the state dimension cannot be reduced; see (Hannan & Deistler, 1988), Chapter 1, for details) and stable (such that all eigenvalues of A are smaller than one in modulus).

The innovation representation corresponds to the Wold representation of the process ${\left(y_t\right)}_{t\in \mathbb{Z}}$, if and only if the eigenvalues of the matrix $\underline{A}=A-BC$ are inside or on the unit circle. In this case, $|{\lambda }_{max}\left(\underline{A}\right)|\le 1$, where ${\lambda }_{max}\left(M\right)$ denotes a maximum modulus eigenvalue of the matrix M.

The asymptotic properties for CVA when the data are generated from a state space system in innovation representation documented in the literature are restricted to the case of stably invertible processes where the strict inequality $|{\lambda }_{max}\left(\underline{A}\right)|<1$ holds. However, this restriction may be violated in particular for economic data if the data are transformed to stationarity by temporal differencing without taking possible co-integrating relations into account. If co-integrating relations between the component variables exist and the whole time series is differenced, this leads to over-differencing in some directions, introducing spectral zeros at frequency $\omega =0$. A similar situation occurs if HP filtering is used to extract a trend: Sakarya and de Jong (2022) show that the spectrum of the detrended cyclical process then has a zero at frequency zero. Similar effects occur for seasonal differencing.

Processes that have spectral zeros are often called non-invertible. Funovits (2024), dealing with structural modelling involving non-invertible autoregressive moving average (ARMA) processes, excludes spectral zeros from his model class and argues that such processes in fact are invertible in the sense that long autoregressive approximations exist. These approximations, however, have non-standard properties; see below. Hence, we use the term ‘stably invertible’ for $|{\lambda }_{max}\left(\underline{A}\right)|<1$.1

In such a situation, the properties of the subspace estimators are currently unknown. This study closes this gap using results from Poskitt (2006) related to the autoregressive approximation of processes with spectral zeros. We show that CVA provides consistent estimators for the impulse response sequence also in the case of some spectral zeros at $\omega =0$. From the proof, it is clear that analogous results also hold for zeros at different frequencies. Consistency is obtained for the integer parameter p of CVA (corresponding to the lag length of an autoregressive approximation) tending to infinity at a certain rate. We investigate the asymptotic bias arising for finite lag lengths and show that for typical choices, it is not negligible, as it tends to zero slower than $1/\sqrt{T}$, the typical convergence rate involved in asymptotic normality.

The significance of this result is demonstrated in a simulation exercise whose main implications are that inference in the over-differenced situation is unreliable, be it using CVA or qMLE or prediction error method (PEM) estimators (each suffering from different problems). The simulations illustrate the advantages of working with the original data rather than preprocessed data involving seasonal adjustment or differencing to transform the data to stationarity.

This paper is organized as follows: in the next section, the main idea of CVA is presented. Afterwards, the results of this study are described in Section 3. Demonstrations are provided in Section 4. Section 5 concludes the paper. Proofs are relegated to the appendices.

2. Canonical Variate Analysis

The CVA method of estimation is performed in three steps and uses two integers, $f,p$ (‘future’ and ‘past’), and information of the system order n (compare Bauer, 2005):

• Obtain an estimate ${\widehat{x}}_t$ of the state $x_t$ for $t=p+1,\dots ,T+1$.
• Estimate C by regressing $y_t$ onto ${\widehat{x}}_t$. This step provides residuals ${\widehat{\epsilon }}_t=y_t-\widehat{C}{\widehat{x}}_t,$ $t=p+1,\dots ,T$.
• Estimate A and B by regressing ${\widehat{x}}_{t+1}$ onto ${\widehat{x}}_t$ and ${\widehat{\epsilon }}_t,t=p+1,\dots ,T$.

The essential idea of CVA lies in the estimation of $x_t$, which uses the representation of the joint vector $Y_t^{+}={(y_t^{\prime },y_{t+1}^{\prime },\dots ,y_{t+f-1}^{\prime })}^{\prime }$ for some integer $f\ge n$ as the state space system implies:

$$Y_t^{+}={\mathcal{O}}_f{x}_t+{\mathcal{E}}_f{E}_t^{+},x_t={\mathcal{K}}_p{Y}_t^{-}+\delta x_t\left(p\right),$$

(2)

where ${\mathcal{E}}_f\in {\mathbb{R}}^{fs\times fs}$ contains the impulse response coefficients and

$${\mathcal{O}}_f={(C^{\prime },A^{\prime }{C}^{\prime },\dots ,{\left(A^{f-1}\right)}^{\prime }{C}^{\prime })}^{\prime }\in {\mathbb{R}}^{fs\times n}$$

denotes the observability matrix, which has full column rank due to minimality. Furthermore

$${\mathcal{K}}_p=\mathbb{E}{x}_t{\left(Y_t^{-}\right)}^{\prime }{\left(\mathbb{E}{Y}_t^{-}{\left(Y_t^{-}\right)}^{\prime }\right)}^{-1}\in {\mathbb{R}}^{n\times ps}$$

denotes the regression coefficient for explaining $x_t$ by $Y_t^{-}={(y_{t-1}^{\prime },\dots ,y_{t-p}^{\prime })}^{\prime }\in {\mathbb{R}}^{ps}$ for integer $p\ge n$, leading to the approximation $x_t\left(p\right)={\mathcal{K}}_p{Y}_t^{-}$. Then $\delta x_t\left(p\right)=x_t-x_t\left(p\right)$ denotes the approximation error.

Under the strict minimum-phase assumption implying ${\underline{A}}^p\to 0$ for $p\to \infty$, we may use2 ${\mathcal{K}}_p=[B,\underline{A}B,{\underline{A}}^{2}B,\dots ,{\underline{A}}^{p-1}B]$, leading to

$$x_t={\mathcal{K}}_p{Y}_t^{-}+{\underline{A}}^p{x}_{t-p}$$

to infer that the variance of the approximation error $\delta x_t\left(p\right)$ can be bounded by ${\underline{A}}^p\left(\mathbb{E}{x}_{t-p}{x}_{t-p}^{\prime }\right){\left({\underline{A}}^p\right)}^{\prime }$ such that it is of order $O\left({\rho }_0^{2p}\right)$, where $1>{\rho }_0>\left|{\lambda }_{max}\left(\underline{A}\right)\right|$. If in that case, $p=p\left(T\right)=-c(logT)/(2log{\rho }_0)$ is used, we obtain

$${\rho }_0^p=exp(-c(logT)/(2log{\rho }_0)log{\rho }_0)=exp(-c(logT)/2)=T^{-c/2}$$

such that the variance of the approximation error is of order $T^{-c}$. If $c>1$, this implies that the approximation error is negligible in the usual $\sqrt{T}$ asymptotics. In the stably invertible case, it is known that the lag length of a vector autoregressive approximation of the process ${\left(y_t\right)}_{t\in \mathbb{Z}}$ selected using information criteria like BIC tends to infinity roughly as $-logT/(2log{\rho }_0)$ (Hannan & Deistler, 1988, see Theorems 6.6.3 and 6.6.4). Thus, selecting $p\left(T\right)$ by the AIC optimal autoregressive lag length has been suggested (Bauer, 2005). For integrated processes, twice this number has been suggested such that the approximation error decays as $1/T$.

For ${\rho }_0=1$, this argument does not work any more. Poskitt (2006) shows that also in this case, the approximation error decreases to zero, albeit not at the same speed, as shown in the following example:

Example 1.

Consider $y_t={\epsilon }_t-{\epsilon }_{t-1}\in {\mathbb{R}}^s$ for independent identically distributed white noise ${\left({\epsilon }_t\right)}_{t\in \mathbb{Z}}$ with expectation zero and variance $\mathsf{\Omega }>0$. This can be represented in state space form as

$$y_t=I_s{x}_t+{\epsilon }_t,x_{t+1}=0_{s\times s}{x}_t-I_s{\epsilon }_t$$

and hence $x_t=-{\epsilon }_{t-1}$ and $(A,B,C)=(0_{s\times s},-I_s,I_s)$. Following Poskitt (2006), we see that ${\mathcal{K}}_p=-[{\displaystyle \frac{p}{p+1}}{I}_s,{\displaystyle \frac{p-1}{p+1}}{I}_s,\dots ,{\displaystyle \frac{1}{p+1}}{I}_s]$, implying that

$$\begin{array}{ccc}\hfill x_t\left(p\right)& =& {\mathcal{K}}_p{Y}_t^{-}={\displaystyle \frac{-1}{p+1}}\sum _{j=1}^p(p+1-j)({\epsilon }_{t-j}-{\epsilon }_{t-j-1})\hfill \\ & =& {\displaystyle \frac{-1}{p+1}}{\displaystyle \sum _{j=1}^p}(p+1-j){\epsilon }_{t-j}-{\displaystyle \frac{-1}{p+1}}{\displaystyle \sum _{j=2}^{p+1}}(p+2-j){\epsilon }_{t-j}\hfill \\ & =& {\displaystyle \frac{-1}{p+1}}\left(p{\epsilon }_{t-1}-\sum _{j=2}^p{\epsilon }_{t-j}-{\epsilon }_{t-p-1}\right)=-{\epsilon }_{t-1}+{\displaystyle \frac{1}{p+1}}{\displaystyle \sum _{j=1}^{p+1}}{\epsilon }_{t-j}.\hfill \end{array}$$

Denoting ${\overline{\epsilon }}_{t-1}\left(p\right)={\sum }_{j=1}^{p+1}{\epsilon }_{t-j}/(p+1)$, we obtain $x_t=x_t\left(p\right)-{\overline{\epsilon }}_{t-1}\left(p\right),{\epsilon }_t\left(p\right)=y_t-x_t\left(p\right)={\epsilon }_t-{\overline{\epsilon }}_{t-1}\left(p\right)$ such that the approximation error $\delta x_t\left(p\right)=-{\overline{\epsilon }}_{t-1}\left(p\right)$. It follows that $\mathbb{E}\delta x_t\left(p\right)\delta x_t{\left(p\right)}^{\prime }={\displaystyle \frac{1}{p+1}}\mathsf{\Omega }$. Thus the approximation error tends to zero in mean square, but the variance is of order $1/p$ and does not decrease exponentially.

This example is typical. The same arguments show that the variance of the approximation error for $y_t=\Delta u_t=u_t-u_{t-1}$ for stationary process ${\left(u_t\right)}_{t\in \mathbb{Z}}$ with non-singular spectral density at $\omega =0$ (not necessarily white noise) is at most of order $p^{-1}$. In order for this error to be neglectable in typical asymptotic analyses, we require $p^{-1}\sqrt{T}\to 0$ (see Example 2 below). Thus, in this setting, p must increase much faster than the usual rate proportional to $logT$ for stably invertible processes to produce a neglectable bias term.

Note that in the over-differenced situation with ${\rho }_0=1$, the properties of BIC optimal lag length selection do not follow from Theorem 6.6.3 (Hannan & Deistler, 1988). The arguments in the corresponding proofs, however, suggest that the corresponding estimate $\widehat{p}$ is close to the minimizer of $logdet{\mathsf{\Omega }}_p+2p{s}^2/T$, where ${\mathsf{\Omega }}_p$ denotes the one-step prediction error variance for an autoregression of order p. In the example above, one obtains ${\mathsf{\Omega }}_p=\mathsf{\Omega }{\displaystyle \frac{p+2}{p+1}}$. The minimizer of $logdet{\mathsf{\Omega }}_p+2p{s}^2/T$ equals the minimizer of $logdet\mathsf{\Omega }+slog\left((p+2)/(p+1)\right)+2(p+1)s^2/T$. Approximating $log(1+{\displaystyle \frac{1}{p+1}})$ as ${\displaystyle \frac{1}{p+1}}$, one obtains $\sqrt{{\displaystyle \frac{T}{2s}}}-1$ as the minimizer. This illustrates that in the over-differenced situation, one might expect a relatively large choice of the lag length roughly proportional to $\sqrt{T}$ in the autoregressive approximation chosen by BIC. The simulations in Section 4 demonstrate this behaviour.

3. Results

Poskitt (2006) derives results for the estimation accuracy of the autoregressive approximation coefficients. In his Theorem 5, he states that uniformly in $0<p\le H_T$ for some upper bound $H_T=O\left(\sqrt{T/logT}\right)$ and using $Q_T^2=(\log T)/T$, we have

$${\displaystyle \sum _{j=1}^p}{|{\widehat{\alpha }}_p\left(j\right)-{\alpha }_p\left(j\right)|}^2=O\left({\displaystyle \frac{p}{{\lambda }_{min}{\left({\mathsf{\Gamma }}_p\right)}^2}}{Q}_T^2\right)$$

where $O(.)$ denotes an almost certain convergence at the given rate. Here ${\alpha }_p\left(j\right)$ denotes the autoregressive coefficients in a lag p approximation for ${\left(y_t\right)}_{t\in \mathbb{Z}}$ obtained from

$$[{\alpha }_p\left(1\right),\dots ,{\alpha }_p\left(p\right)]=\left(\mathbb{E}{y}_t{\left(Y_t^{-}\right)}^{\prime }\right){\mathsf{\Gamma }}_p^{-1},{\mathsf{\Gamma }}_p={\left(\mathbb{E}{Y}_t^{-}{\left(Y_t^{-}\right)}^{\prime }\right)}^{-1}$$

and ${\widehat{\alpha }}_p\left(j\right)$, which are the corresponding least squares estimates. Poskitt (2006) uses a univariate setting; however, the extension to multivariate time series in our framework is obvious.3

In this study, we do not investigate autoregressive processes but state space processes with spectral zeros. We focus on the case of simple spectral zeros obtained by one-time over-differencing:

Assumption 1.

The stationary process ${\left(y_t\right)}_{t\in \mathbb{Z}},y_t\in {\mathbb{R}}^s$ is generated using a rational, stable, and stably invertible transfer function $k\left(z\right)=I_s+{\sum }_{j=1}^{\infty }{K}_j{z}^j, K_j=C_{\circ }{A}_{\circ }^{j-1}{B}_{\circ }$ (which has all its zeros and poles outside the unit circle) and an orthonormal matrix $M=[M_c,M_{s-c}]\in {\mathbb{R}}^{s\times s},M^{\prime }M=I_s$, $M_c\in {\mathbb{R}}^{s\times c}$, where $0<c\le s$ is an integer, as (L denoting the backward-shift operator and $\Delta =(1-L)$)

$${\left(y_t\right)}_{t\in \mathbb{Z}}=M\left[\begin{array}{cc}\Delta I_c& 0\\ 0& I_{s-c}\end{array}\right]M^{\prime }k\left(L\right){\left({\epsilon }_t\right)}_{t\in \mathbb{Z}}.$$

Here ${\left({\epsilon }_t\right)}_{t\in \mathbb{Z}}$ denotes a zero mean ergodic, stationary, martingale difference sequence with respect to the sequence ${\mathcal{F}}_t$ of sigma fields spanned by the past of ${\epsilon }_t$, fulfilling

$$\begin{array}{ccc}\hfill \mathbb{E}\left({\epsilon }_t\right|{\mathcal{F}}_{t-1})=0& ,& \mathbb{E}\left({\epsilon }_t{\epsilon }_t^{\prime }\right|{\mathcal{F}}_{t-1})=\mathbb{E}\left({\epsilon }_t{\epsilon }_t^{\prime }\right)=\mathsf{\Omega }.\hfill \end{array}$$

Furthermore, $\mathbb{E}{\epsilon }_{t,j}^4<\infty ,j=1,\dots ,s$.

We use the same noise assumptions as Poskitt (2006) and Hannan and Deistler (1988). The conditional homoskedasticity assumption $\mathbb{E}\left({\epsilon }_t{\epsilon }_t^{\prime }\right|{\mathcal{F}}_{t-1})=\mathbb{E}\left({\epsilon }_t{\epsilon }_t^{\prime }\right)$ can be replaced by the weaker assumption ${lim}_{k\to \infty }\mathbb{E}\left({\epsilon }_t{\epsilon }_t^{\prime }\right|{\mathcal{F}}_{t-k})=\mathsf{\Omega }$, allowing for GARCH-type processes (Bauer, 2008). We do not attempt to weaken the assumption of finite fourth moments. Clearly such processes have a spectral density of rank $s-c$ (which is hence singular for $c>0$) for $\omega =0$ due to the differencing). At all other frequencies, the rank equals s, since $k\left(L\right)$ is assumed to be stably invertible.

These assumptions are fulfilled when examining first differences of a co-integrated I(1) process. Examples of such processes include multivariate time series, wherein some components are stationary, while others show the random walk-like behaviour typical for I(1) processes. In this situation, differencing the whole multivariate process will over-difference the stationary components. This can be avoided by only differencing the integrated components, if such a partitioning of the multivariate time series exists, which of course requires knowledge on which components are stationary and which are integrated.

Often, however, all components of a multivariate time series will be integrated, while linear combinations are stationary. This corresponds to a process where the number of common trends driving the time series is smaller than the dimension of the time series, the difference being equal to the co-integrating rank. This is reasonable in systems with many variables where inference on the co-integrating rank is difficult. But even in smaller systems, the decision on the number of co-integrating relations is sometimes not simple, as documented in examples in (Johansen, 1995). If in such situations, the co-integrating rank is specified as too small or if, for simplicity, the full process is differenced, spectral zeros result.

To apply this result in our setting, note that the straightforward multivariate extension4 to Theorem 2 of Palma and Bondon (2003)’s work implies that ${\lambda }_{min}\left(\mathbb{E}{Y}_t^{-}{\left(Y_t^{-}\right)}^{\prime }\right)$ is bounded from below by $\underline{c}{p}^{-2}$ for ${\left(y_t\right)}_{t\in \mathbb{Z}}$ according to Assumption 1.

This implies that the bound above amounts to $p^{5}logT/T$, which tends to zero, if $p=c\lfloor T^{\delta }\rfloor$ for $0<\delta <0.2$. Note, however, that for this rate of increase, the approximation error (with variance of order $p^{-1}$) is larger than $O(1/\sqrt{T})$ and hence dominates the asymptotic distribution of terms like $\sqrt{T}(\widehat{A}-A_{\circ })$; see Theorem 2 below.

The results from the autoregressive setting can be used here almost immediately if $f\ge n$ is fixed and $p=f\tilde{p}$, where $\tilde{p}=\tilde{p}\left(T\right)=o\left(T^{\delta }\right)$ depends on the sample size. This implies that for the approximation of $x_t$, the unrestricted estimate ${\widehat{\beta }}_{f,p}=\langle Y_t^{+},Y_t^{-}\rangle {\langle Y_t^{-},Y_t^{-}\rangle }^{-1}$ equals an autoregressive model for $Y_t^{+}$. Here and below, we use the notation $\langle a_t,b_t\rangle =T^{-1}{\sum }_{t=p+1}^{T-f}{a}_t{b}_t^{\prime }$ for two processes ${\left(a_t\right)}_{t\in \mathbb{Z}}$ and ${\left(b_t\right)}_{t\in \mathbb{Z}}$. This matrix ${\widehat{\beta }}_{f,p}$—which in the limit has rank n but in finite samples is of full rank—is then low-rank-approximated, leading to the estimate ${\widehat{\mathcal{O}}}_f{\widehat{\mathcal{K}}}_p$ of ${\mathcal{O}}_f{\mathcal{K}}_p$.

In order to identify the factors ${\widehat{\mathcal{O}}}_f$ and ${\widehat{\mathcal{K}}}_p$ from the product, we use a selector matrix $S_f\in {\mathbb{R}}^{n\times fs}$ such that $S_f{\mathcal{O}}_f=I_n$. Such a matrix always exists (cf. for example the overlapping echelon forms; Section 2.6 of Hannan and Deistler (1988)). Since the results below correspond to estimates of the impulse response coefficients (which are invariant in this respect), this choice of the state basis can be assumed without restriction of generality.

The second and third step of CVA then amount to least squares using the estimate ${\widehat{x}}_t={\widehat{\mathcal{K}}}_p{Y}_t^{-}$ of the state. If instead we had access to the state approximation $x_t\left(p\right)={\mathcal{K}}_p{Y}_t^{-}$ as well as population instead of sample moments, we would obtain the following matrices:

$$\begin{array}{ccc}\hfill A_p& =& \mathbb{E}{x}_{t+1}\left(p\right)x_t{\left(p\right)}^{\prime }{\left(\mathbb{E}{x}_t\left(p\right)x_t{\left(p\right)}^{\prime }\right)}^{-1}\hfill \\ \hfill B_p& =& \mathbb{E}{x}_{t+1}\left(p\right){\epsilon }_t{\left(p\right)}^{\prime }{\left(\mathbb{E}{\epsilon }_t\left(p\right){\epsilon }_t{\left(p\right)}^{\prime }\right)}^{-1}\hfill \\ \hfill C_p& =& \mathbb{E}{y}_t{x}_t{\left(p\right)}^{\prime }{\left(\mathbb{E}{x}_t\left(p\right)x_t{\left(p\right)}^{\prime }\right)}^{-1}.\hfill \end{array}$$

If the approximation errors tend to zero and the convergence of sample covariances to population quantities is uniform in p, then consistency for $p\to \infty$ follows (for the proof, see Appendix A):

Theorem 1.

Let the process ${\left(y_t\right)}_{t\in \mathbb{Z}}$ be generated according to Assumption 1. Let the CVA procedure be applied with $f\ge n$ not depending on T and $p=p\left(T\right)\to \infty$ for $T\to \infty$ such that $p\left(T\right)=o\left(T^{\delta }\right),0<\delta <0.2$.

Then

$$\begin{array}{ccc}\hfill max\{\parallel \widehat{A}-A_p\parallel ,\parallel \widehat{B}-B_p\parallel ,\parallel \widehat{C}-C_p\parallel \}& =& O\left(\sqrt{p{\left(T\right)}^{5}log\left(T\right)/T}\right),\hfill \\ \hfill max\{\parallel A_{\circ }-A_p\parallel ,\parallel B_{\circ }-B_p\parallel ,\parallel C_{\circ }-C_p\parallel \}& \to & 0\hfill \end{array}$$

for $p=p\left(T\right)\to \infty$ as $T\to \infty$. Consequently, $\widehat{C}{\widehat{A}}^j\widehat{B}\to C_{\circ }{A}_{\circ }^j{B}_{\circ },j=0,1,2,\dots$ almost certainly in that case.

Note that these two error bounds are differently influenced by the integer p: a large p value reduces the approximation errors such as $A_p-A_{\circ }$ but increases the sampling error $\widehat{A}-A_p$. It is the first one that tends to zero slower for spectral zeros than in the stably invertible case.

Example 2.

Consider, as in Example 1, $y_t=\Delta {\epsilon }_t$ for white noise ${\left({\epsilon }_t\right)}_{t\in \mathbb{Z}}$. Then $x_t\left(p\right)=-{\epsilon }_{t-1}+{\overline{\epsilon }}_{t-1}\left(p\right)$ and ${\epsilon }_t\left(p\right)={\epsilon }_t-{\overline{\epsilon }}_{t-1}\left(p\right)$. It follows that $\mathbb{E}{x}_t\left(p\right){\epsilon }_t{\left(p\right)}^{\prime }=0,$

$$\begin{array}{ccc}\hfill \mathbb{E}{\epsilon }_t\left(p\right){\epsilon }_t{\left(p\right)}^{\prime }={\displaystyle \frac{p+2}{p+1}}\mathsf{\Omega }& ,& \mathbb{E}{x}_t\left(p\right)x_t{\left(p\right)}^{\prime }={\displaystyle \frac{p}{p+1}}\mathsf{\Omega },\hfill \\ \hfill \mathbb{E}{x}_{t+1}\left(p\right){\epsilon }_t{\left(p\right)}^{\prime }=-(1-{\displaystyle \frac{1}{{(p+1)}^2}})\mathsf{\Omega }& ,& \mathbb{E}{x}_{t+1}\left(p\right)x_t{\left(p\right)}^{\prime }=-{\displaystyle \frac{1}{{(p+1)}^2}}\mathsf{\Omega }.\hfill \end{array}$$

Thus $A_p=A_{\circ }-I_s{\displaystyle \frac{1}{p(p+1)}},B_p=B_{\circ }+{\displaystyle \frac{1}{p+1}}{I}_s,C_p=C_{\circ }$.

This shows for the special case that the system $(A_p,B_p,C_p)$ for fixed p is a biased estimate of the true system $(0,-I_s,I_s)$. The bias is of order $p^{-1}$. This is typical, as is demonstrated by the next result.

Theorem 2.

Let the process ${\left(y_t\right)}_{t\in \mathbb{Z}}$ be generated as in Theorem 1.

(I) 

Assume that $\mathbb{E}\delta x_t\left(p\right)\delta x_t{\left(p\right)}^{\prime }=O\left(g_p\right)$. Then we obtain

$$max\{\parallel A_{\circ }-A_p\parallel ,\parallel B_{\circ }-B_p\parallel ,\parallel C_{\circ }-C_p\parallel \}=O\left(g_p^{1/2}\right).$$

(II) 

$g_p=1/p$ provides an upper bound on the achievable error in the sense that $\mathbb{E}\delta x_t\left(p\right)\delta x_t{\left(p\right)}^{\prime }=O(1/p)$.

In order for this bound of the bias to be asymptotically negligible, $\sqrt{p}$ has to grow faster than $T^{1/2}$. The bound may be conservative; however, Example 2 shows that, in general, $g_p$ cannot be smaller than $1/p^2$ such that the bias is of order $O(1/p)$. Even in this case, the required increase is faster than the upper bound $H_T=\sqrt{T/logT}$ used above, such that with our methods, we cannot derive results for the asymptotic distribution of the system estimates.

Additionally note that typically the upper bound for selecting the lag length is $H_T=c\lfloor T^{1/4}\rfloor$ such that the bias derived above will dominate the asymptotics.

4. Illustration

In order to demonstrate the theoretical results, in this section a simulation exercise is discussed and the approaches are applied to a Danish dataset which (Johansen, 1995) used as a demonstration example for co-integration analysis in the vector error correction framework.

4.1. Simulations

We simulate a test example to indicate the relative performance of three different estimators for the original as well as the once-differenced time series:

• CVA: The subspace procedure described above, where p is chosen according to twice the AIC optimal lag length in an autoregressive approximation.
• qMLE: Maximum likelihood estimation based on the Gaussian likelihood. Here both stability and strict minimum-phase assumptions are imposed using a barrier function approach. The data is assumed to be stationary and the corresponding likelihood is optimized.
• PEM: Prediction error methods use the assumption of $x_0=0$. With this initialization, the Kalman filter collapses to the inverse system. Again, stability and strict minimum-phase assumptions are imposed using a barrier function approach.

qMLE and PEM are initialized using the CVA estimate but also employ randomization to reduce the probability of being trapped in local minima. For the data-generating process, we use a bivariate system:

$$\begin{array}{ccc}\hfill y_t& =& x_t+e_t,\hfill \\ \hfill x_{t+1}& =& \left(\begin{array}{cc}0.7& 0\\ 0& 1\end{array}\right)x_t+\left(\begin{array}{cc}1& 0\\ 0& 0.5\end{array}\right)e_t.\hfill \end{array}$$

Here ${\left(e_t\right)}_{t\in \mathbb{Z}}$ denotes a bivariate standard normal error process. Consequently, the process is a bivariate I(1) process with independent innovations. The first component is stationary, while the second is integrated. We apply the estimation procedures to ${\left(y_t\right)}_{t\in \mathbb{Z}}$ and to $\Delta {\left(y_t\right)}_{t\in \mathbb{Z}}$, which has a state space representation of order three:

$$\begin{array}{ccc}\hfill \Delta y_t& =& \left(\begin{array}{ccc}0& -0.3& 0\\ 0& 0& -0.5\end{array}\right)x_t+e_t,\hfill \\ \hfill x_{t+1}& =& \left(\begin{array}{ccc}0.7& 0& 0\\ 1& 0& 0\\ 0& 0& 0\end{array}\right)x_t+\left(\begin{array}{cc}1& 0\\ 0& 0\\ 0& 1\end{array}\right)e_t.\hfill \end{array}$$

We generate $M=1000$ datasets of sample size $T\in \{100,200,400,800,1600\}$ and estimate a state space system with $n=2$ for ${\left(y_t\right)}_{t\in \mathbb{Z}}$ and $n=3$ for $\Delta {\left(y_t\right)}_{t\in \mathbb{Z}}$. Hereby $\widehat{f}=\widehat{p}=2{\widehat{k}}_{AIC}$ is chosen, where ${\widehat{k}}_{AIC}$ denotes the lag length of an autoregressive approximation chosen using AIC with $p=10$ used as the upper bound for the original data and $p=20$ for the differenced data.

The results can be seen in Figure 1. Plot (a) of that figure provides a plot of the mean squared error of the impulse response estimates times the sample size T. For ${\left(y_t\right)}_{t\in \mathbb{Z}}$, we calculate the impulse response corresponding to the differenced series. A convergence rate of order $O_P\left(T^{-1/2}\right)$ would imply that the curves level off for large sample sizes. While this seems to be the case for CVA and PEM applied to the original data, all curves for the differenced series show an increase for the larger sample size. The superiority of the estimates for the original series is visible for all three estimators. CVA applied to the differenced series performs worst, particularly for large sample sizes. Additionally, it is visible that qMLE performs worse than PEM for the integrated process. This happens as the qMLE assumes stationary initialization of the state process in the Kalman filter. In the integrated case, this leads to very large initial uncertainty of the state, reflected in the poorer estimation. This is also visible in the estimates of the $(2,2)$ element of $K_1=CK$, the first element of the impulse response, as shown in plots (c) for $T=200$ and (d) for $T=1600$. All but the qMLE estimates appear to be Gaussian-distributed around the correct mean −0.5. In plot (c), the superiority of the estimates for the original series is visible in a higher concentration around $-0.5$. Plot (b) provides mean selected lag lengths, documenting that for the original series the lag length selection roughly behaves as $-logT/(2log{\rho }_0)$ (printed as red stars), while for the differenced series, the increase is much faster.

While the impulse response estimates for the large sample size appear to be Gaussian-distributed, this is not the case for all system-dependent quantities. Figure 2a,b provide density estimates for the estimate of the sum of all entries of $\underline{A}=A-BC$. In (a), for the original data, one notices a bias for the qMLE estimates (compensating for the large initial state uncertainty), while in (b), for the differenced data, the CVA estimates show a downward bias. Plot (c) provides the deviation of the mean of the largest modulus of the eigenvalues of $\underline{A}$ for the original process (with true value $0.5$), as well as for the differenced process (with true value 1).5 The larger downward bias for the qMLE estimates for the original data and the CVA estimates for the difference data is clearly visible but diminishes with sample size. Note that for $T=400$ onward for the differenced data for PEM and qMLE, the mean maximum modulus almost reaches the maximum allowed value, implying that almost all values are equal to the maximum. Plot (d) provides the ecdf for $T=400$ of the maximum modulus of the eigenvalues of $\underline{A}$. While for CVA, the maximum modulus is almost always estimated below 0.98, for PEM and qMLE, the opposite is true, with a large point mass located at 0.99. Only in 6% of the cases is the maximum modulus smaller than $0.989$. Therefore the maximum modulus for the qMLE and PEM estimates for the differenced data does not follow a normal distribution, not even for $T=1600$.

Consequently, for the differenced series, inference provides obstacles for qMLE and PEM in the form of non-Gaussian distributions with discrete components and a dominating bias for CVA.

4.2. The Danish Dataset

The simulation evidence from the last subsection is briefly complemented with a real-world data set. The Danish dataset was used by Johansen (1995) for the illustration of co-integration modeling using vector error correction representation. The dataset comprises four time series observed quarterly over the span of 55 quarters from 1974:1 to 1987:3. The variables are log real money (m2), log real income (y), the bank deposit rate for interest-bearing deposits (ide), and the bond rate (ibo). From the plots on p. 26 of Johansen (1995)’s work and the accompanying univariate tests, one may conclude that all four series appear to be integrated with stationary differences. Johansen (1995) uses seasonal dummies as deterministics.

Johansen (1995) tests for the number of co-integrating relations and states on p. 112 ‘From Table 7.1 we find that there is no clear evidence for co-integration. […] We choose to maintain the hypothesis that r = 1, that is, that there is one co-integrating relation’. This is a typical situation where the number of co-integrating relations (if any) is not certain. Economic theory would imply its existence, but the data provides borderline tests and no clear-cut results.

In addition to the estimation of the model, Johansen (1995) also tests—under the assumption of one co-integrating relation—for restrictions on the co-integrating relation.

We used the dataset to estimate models for the original data as well as for the differenced process. Seasonal dummies are included. We use CVA and PEM as in the simulation section. The results are provided in

Table 1. Mean square error in sample for the three estimators, for the original data, and for the differenced data.

Dataset	Estimator	m2	y	ibo	ide
original data	VECM(1,1)	0.0195	0.0215	0.0077	0.0054
	PEM	0.0163	0.0207	0.0075	0.0048
	CVA	0.0219	0.0197	0.0085	0.0049
differenced data	VAR(2)	0.0202	0.0201	0.0073	0.0053
	PEM	0.0168	0.0202	0.0070	0.0048
	CVA	0.0200	0.0217	0.0088	0.0061

.

For the original data, $\widehat{p}=2$ was used as in Johansen (1995). As deterministic terms, quarterly dummies were added. For PEM and CVA, $\widehat{n}=7$ resulted in the best fit (according to AIC). For PEM, three common trends corresponding to one co-integrating relation minimize AIC.

For the differenced data, $\widehat{p}=2$ was chosen. Lag length selection using AIC is very unstable, though. The results depend heavily on the upper limit if a uniform evaluation sample is used. For PEM and CVA, $\widehat{n}=8$ was selected for the differenced data, which corresponds to the first difference of the model for the original data.

We can see from Table 1 that PEM for the original data results in the best one-step-ahead prediction error except for y, where CVA performs better, and ide, where PEM on the differenced series provides better forecasts.

In terms of the zeros of the transfer function for the original and for the differenced data, the PEM estimates provide zeros on the boundary to being unstable. For the differenced data, six out of the eight zeros have a modulus larger than $0.9$. Also for the original data, the PEM model has three zeros with a modulus larger than $0.9$. Thus, these estimates are also not well behaved.

Computing the spectral density at zero frequency according to the PEM model for the differenced data, we obtain for the square roots of the eigenvalues $(0.0476,0.0175,0.0065,0.0023)$. This indicates that the rank is either two or three. For the first difference of the PEM model for the original data, we obtain $(0.0416,0.0168,0.0006,0.000)$, since there are only three common trends contained in the model. The strength of the third is, thus, also very weak.

With the PEM model for the original data, we can use the trace statistic in the state space error correction (SPECM) formulation including a restricted constant (see Matuschek, 2020), providing the eigenvalues $(0.9030,0.2732,0.1202,0.0246)$. The corresponding trace statistics equal $(195.52,25.02,8.11,1.32)$, resulting in rank $r=1$ with the usual sequential testing technique (using critical values $(53.92,35.08,20.27,9.12)$).

Within the SPECM, one can also test restrictions on the co-integrating relations, which is estimated as $\widehat{\beta }={(-5.83,1.00,-1.08,4.52,-2.37)}^{\prime }$, normalized such that the coefficients for m2 equals 1. The first entry corresponds to the restricted constant. These values are close to the ones by Johansen (1995), shown in Section 7.3.1. One notices that the coefficient for y is close to $-1$. Economic theory further conjectures that the coefficients for ibo and ide should sum to zero. Testing this restriction on $\widehat{\beta }$ within the SPECM, we get a test statistic of $3.92$, which corresponds to a p-value of 0.14 for the limiting ${\chi }_2^2$ distribution. The restricted estimate $\tilde{\beta }={(-6.11,1.00,-1.00,4.73,-4.73)}^{\prime }$. This is to be compared to $(-6.21,1.00,-1.00,5.88,-5.88)$, as provided by Johansen (1995) with a different model and different specification of the deterministics (i.e., leaving out the seasonal dummies).

Finally, the spectral density of the differenced process in the direction of the co-integrating relation $\tilde{\beta }$ (omitting the coefficient corresponding to the constant and scaling the vector to have a norm equal to one) is corresponding to the model for the first differences equal to $1.5148\times {10}^{-5}$. The smallest eigenvalue of the spectrum equals $5.18\times {10}^{-6}$ and the second to last equals $4.26\times {10}^{-5}$, while the largest eigenvalue equals $0.0023$. This hints at the singularity of the spectrum at zero for the differenced process as well as to the fact that the co-integrating relation found points in the direction of the singularity.

Thus, the analysis with the state space systems adds evidence to Johansen’s choice of $r=1$ as well as no evidence against the equalities in the co-integrating vectors suggested by economic theory.

5. Conclusions

In this technical paper, we show that working with first differences does not invalidate consistency for CVA. This is a relief in situations where one is not sure about the existence of co-integrating relations.

Inference, on the other hand, gets more complicated as in a situation where some of the variables are over-differenced, the asymptotic distribution is typically dominated by the bias term for CVA estimates. Note that for partially over-differenced series, the quasi-maximum likelihood estimators are also known to not be asymptotically normal, as the true system lies at the boundary of the parameter space. This implies that inference is close to impossible for over-differenced processes.

The results also imply that a higher order of differencing and spectral zeros at other frequencies introduced, for example, from yearly differencing, can be dealt with using exactly the same methods, as the results from Poskitt (2006) and Palma and Bondon (2003) also hold for these cases. In such situations, the consistency of CVA estimators of the impulse response sequence again follows for $p=p\left(T\right)$ increasing at a sufficiently slow rate.