Value of Information in the Binary Case and Confusion Matrix ^†

Abstract

The simplest Bayesian system used to illustrate ideas of probability theory is a coin and a boolean utility function. To illustrate ideas of hypothesis testing, estimation or optimal control, one needs to use at least two coins and a confusion matrix accounting for the utilities of four possible outcomes. Here we use such a system to illustrate the main ideas of Stratonovich’s value of information (VoI) theory in the context of a financial time-series forecast. We demonstrate how VoI can provide a theoretical upper bound on the accuracy of the forecasts facilitating the analysis and optimization of models.

1. Introduction

The concept of value of information has different definitions in the literature [1,2]. Here we follow the works of Ruslan Stratonovich and his colleagues, who were inspired by Shannon’s work on rate distortion [3] and made a number of important developments in the 1960s [2]. These mainly theoretical results are gaining new interest thanks to the advancements in data science and machine learning and the need for a deeper understanding of the role of information in learning. We shall review the value of information theory in the context of optimal estimation and hypothesis testing, although the context of optimal control is also relevant.

Consider a probability space $(\mathsf{\Omega },P,\mathcal{A})$ and a random variable $x:\mathsf{\Omega }\to X$ (a measurable function). The optimal estimation of $x\in X$ is the problem of finding an element $y\in Y$ maximizing the expected value of some utility function $u:X\times Y\to \mathbb{R}$ (or minimizing for cost $-u$). The optimal value is

$$U\left(0\right):=\underset{y\in Y}{sup}{\mathbb{E}}_{P\left(x\right)}\left\{u(x,y)\right\},$$

where zero designates the fact that no information about the specific value of $x\in X$ is given, only the prior distribution $P\left(x\right)$. At the other extreme, let $z\in Z$ be another random variable that communicates full information about each realization of x. This entails that there is an invertible function $z=f\left(x\right)$ such that $x=f^{-1}\left(z\right)$ is determined uniquely by the ‘message’ $z\in Z$. The corresponding optimal value is

$$U(\infty ):={\mathbb{E}}_{P\left(x\right)}\left\{\underset{y\left(z\right)}{sup}u(x,y\left(z\right))\right\},$$

where an optimal y is found for each z (i.e., optimization over all mappings $y:Z\to Y$). In the context of estimation, variable x is the response (i.e., the variable of interest) and z is the predictor. The mapping $y\left(z\right)$ represents a model with output $y\in Y$.

Let $I\in [0,\infty ]$ be the intermediate amounts of information, and let $U\left(I\right)\in \left[U\right(0),U(\infty \left)\right]$ be the corresponding optimal values. The value of information is the difference [4]:

$$V\left(I\right):=U\left(I\right)-U\left(0\right).$$

There are, however, different ways in which the information amount I and the quantity $U\left(I\right)$ can be defined, leading to different types of the value function $V\left(I\right)$. For example, consider a mapping $f:X\to Z$ with a constraint $\left|Z\right|\le e^I<\left|X\right|$ on the cardinality of its image. The mapping f partitions its domain into a finite number of subsets $f^{-1}\left(z\right)=\{x\in X:f\left(x\right)=z\}$. Then, given a specific partition $z\left(x\right)$, one can find optimal $y\left(z\right)$ maximizing the conditional expected utility ${\mathbb{E}}_{P(x\mid z)}\{u(x,y)\mid z\}$ for each subset $f^{-1}\left(z\right)\ni x$. This optimization should be repeated for different partitions $z\left(x\right)$, and the optimal value $U\left(I\right)$ is defined over all partitions $z\left(x\right)$, satisfying the cardinality constraint $ln\left|Z\right|\le I$:

$$U\left(I\right):=\underset{z\left(x\right)}{sup}\left[{\mathbb{E}}_{P\left(z\right)}\left\{\underset{y\left(z\right)}{sup}{\mathbb{E}}_{P(x\mid z)}\{u(x,y)\mid z\}\right\}:ln\left|Z\right|\le I\right]$$

(1)

Here, $P\left(z\right)=P\{x\in f^{-1}\left(z\right)\}$. The quantity $I=ln\left|Z\right|$ is called Hartley’s information, and the difference $V\left(I\right)=U\left(I\right)-U\left(0\right)$ in this case is the value of Hartley’s information. One can relax the cardinality constraint and replace it with the constraint on entropy $H\left(Z\right)\le I$, where $H\left(Z\right)=-{\mathbb{E}}_{P\left(z\right)}\{lnP\left(z\right)\}\le ln\left|Z\right|$. In this case, $V\left(I\right)$ is called the value of Boltzmann’s information [4].

One can see from Equation (1) that the computation of the value of Hartley’s or Boltzmann’s information is quite demanding and may involve a procedure such as the k-means clustering algorithm or training a multilayer neural network. Thus, using these values of information is not practical due to high computational costs. The main result of Stratonovich’s theory [4] is that the upper bound on Hartley’s or Boltzmann’s values of information is given by the value of Shannon’s information, and that asymptotically all these values are equivalent (Theorems 11.1 and 11.2 in [4]). The value of Shannon’s information is much easier to compute.

Recall the definition of Shannon’s mutual information [3]:

$$I(X,Y):={\mathbb{E}}_{W(x,y)}\left\{ln\frac{P(x\mid y)}{P\left(x\right)}\right\}=H\left(X\right)-H(X\mid Y)=H\left(Y\right)-H(Y\mid X),$$

where $W(x,y)=P(x\mid y)Q\left(y\right)$ is the joint probability distribution on $X\times Y$, and $H(X\mid Y)$ is the conditional entropy. Under broad assumptions on the reference measures (see Theorem 1.16 in [4]), the following inequalities are valid:

$$0\le I(X,Y)\le min\left\{H\right(X),H(Y\left)\right\}\le min\{ln|X|,ln|Y\left|\right\}.$$

The value of Shannon’s information is defined using the quantity:

$$U\left(I\right):=\underset{P(y\mid x)}{sup}[{\mathbb{E}}_W\left\{u(x,y)\right\}:I(X,Y)\le I]$$

(2)

The optimization above is over all conditional probabilities $P(y\mid x)$ (or joint measures $W(x,y)=P(y\mid x)P\left(x\right)$) satisfying the information constraint $I(X,Y)\le I$. Contrast this with $U\left(I\right)$ for Hartley’s or Boltzmann’s information (1), where optimization is over the mappings $y\left(x\right)=y\circ z\left(x\right)$. As was pointed out in [5], the relation between functions (1) and (2) is similar to that between optimal transport problems in the Monge and Kantorovich formulations. Joint distributions optimal in the sense of (2) are found using the standard method of Lagrange multipliers (e.g., see [4,6]):

$$W(x,y;\beta )=P\left(x\right)Q\left(y\right)e^{\beta u(x,y)-\gamma (\beta ,x)},$$

(3)

where parameter ${\beta }^{-1}$, called temperature, is the Lagrange multiplier associated with the constraint $I(X,Y)\le I$. Distributions P and Q are the marginals of W, and function $\gamma (\beta ,x)$ is defined by normalization ${\sum }_{x,y}W(x,y;\beta )=1$. In fact, taking partial traces of solution (3) gives two equations:

$$\begin{array}{cc}\hfill \sum _{x}W(x,y)=Q\left(y\right)& \Rightarrow \sum _x{e}^{\beta u(x,y)-\gamma (\beta ,x)}P\left(x\right)=1\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill \sum _{y}W(x,y)=P\left(x\right)& \Rightarrow \sum _y{e}^{\beta u(x,y)}Q\left(y\right)=e^{\gamma (\beta ,x)}\hfill \end{array}$$

(5)

Equation (5) defines function $\gamma (\beta ,x)=ln{\sum }_y{e}^{\beta u(x,y)}Q\left(y\right)$. If the linear transformation $T(·)={\sum }_x{e}^{\beta u(x,y)}(·)$ has an inverse, then from Equation (4) one obtains $e^{-\gamma (\beta ,x)}P\left(x\right)=T^{-1}\left(1\right)$ or

$$\gamma (\beta ,x)=-ln\sum _{y}b(x,y)+lnP\left(x\right)={\gamma }_0(\beta ,x)-h\left(x\right),$$

where ${\gamma }_0(\beta ,x):=-ln{\sum }_{y}b(x,y)$, $b(x,y)$ is the kernel of the inverse transformation $T^{-1}$, and $h\left(x\right)=-lnP\left(x\right)$ is random entropy or surprise. Integrating the above with respect to measure $P\left(x\right)$ we obtain

$$\mathsf{\Gamma }\left(\beta \right):=\sum _x\gamma (\beta ,x)P\left(x\right)={\mathsf{\Gamma }}_0\left(\beta \right)-H\left(X\right),$$

where ${\mathsf{\Gamma }}_0\left(\beta \right):={\sum }_x{\gamma }_0(\beta ,x)P\left(x\right)$. Function $\mathsf{\Gamma }\left(\beta \right)$ is the cumulant generating function of optimal distribution (3). Indeed, the expected utility and Shannon’s information for this distribution are

$$U\left(\beta \right)={\mathsf{\Gamma }}^{\prime }\left(\beta \right)={\mathsf{\Gamma }}_0^{\prime }\left(\beta \right),I\left(\beta \right)=\beta {\mathsf{\Gamma }}^{\prime }\left(\beta \right)-\mathsf{\Gamma }\left(\beta \right)=H\left(X\right)-[{\mathsf{\Gamma }}_0\left(\beta \right)-\beta {\mathsf{\Gamma }}_0^{\prime }\left(\beta \right)].$$

The first formula can be obtained directly by differentiating $\mathsf{\Gamma }\left(\beta \right)$, and the second by substitution of (3) into the formula for Shannon’s mutual information. Function ${\mathsf{\Gamma }}_0\left(\beta \right)-\beta {\mathsf{\Gamma }}_0^{\prime }\left(\beta \right)$ is clearly the conditional entropy $H(X\mid Y)$ because $I(X,Y)=H\left(X\right)-H(X\mid Y)$.

Note that information is the Legendre–Fenchel transform $I\left(U\right)=sup\{\beta U-\mathsf{\Gamma }(\beta \left)\right\}$ of convex function $\mathsf{\Gamma }\left(\beta \right)$ (indeed, $U={\mathsf{\Gamma }}^{\prime }\left(\beta \right)$). The inverse of $I\left(U\right)$ is the optimal value $U\left(I\right)$ from Equation (2) defining the value of Shannon’s information, and it is the Legendre–Fenchel transform $U\left(I\right)=inf\{{\beta }^{-1}I-F\left({\beta }^{-1}\right)\}$ of concave function $F\left({\beta }^{-1}\right)=-{\beta }^{-1}\mathsf{\Gamma }\left(\beta \right)$, which is called free energy.

The general strategy for computing the value of Shannon’s information is to derive the expressions for $U\left(\beta \right)$ and $I\left(\beta \right)$ from function ${\mathsf{\Gamma }}_0\left(\beta \right)$ (alternatively, one can obtain $U\left({\beta }^{-1}\right)$ and $I\left({\beta }^{-1}\right)$ from free energy $F_0\left({\beta }^{-1}\right)=-{\beta }^{-1}{\mathsf{\Gamma }}_0\left(\beta \right)$). Then the dependency $U\left(I\right)$ is obtained either parametrically or by excluding $\beta$. Let us now apply this to the simplest $2\times 2$ case.

2. Value of Shannon’s Information for the $2\times 2$ System

Let $X\times Y=\{x_1,x_2\}\times \{y_1,y_2\}$, and let $u:X\times Y\to \mathbb{R}$ be the utility function, which we can represent by a $2\times 2$ matrix:

$$\parallel u(x,y)\parallel =\left[\begin{array}{cc}u(x_1,y_1)& u(x_1,y_2)\\ u(x_2,y_1)& u(x_2,y_2)\end{array}\right]=\left[\begin{array}{cc}{u}_{11}& u_{12}\\ u_{21}& u_{22}\end{array}\right]=\left[\begin{array}{cc}{c}_1+d_1& c_1-d_1\\ c_2-d_2& c_2+d_2\end{array}\right]$$

It is called the confusion matrix in the context of hypothesis testing, where rows correspond to the true states $\{x_1,x_2\}$, and columns correspond to accepting or rejecting the hypothesis $\{y_1,y_2\}$. The set of all joint distributions $W(x,y)$ is a 3-simplex (tetrahedron), shown in Figure 1. The 2D surface in the middle is the set of all product distributions $W(x,y;0)=P\left(x\right)Q\left(y\right)$, which correspond to the minimum $I(X,Y)=0$ of mutual information (independent $x,y$). With no additional information about x, the decision $y_1$ to accept or $y_2$ to reject the hypothesis is completely determined by the utilities and prior probabilities $P\left(x_1\right)=p$ and $P\left(x_2\right)=1-p$. Thus, one has to compare expected utilities ${\mathbb{E}}_P\{u\mid y_1\}=p{u}_{11}+(1-p)u_{21}$ and ${\mathbb{E}}_P\{u\mid y_2\}=p{u}_{12}+(1-p)u_{22}$. The output distribution $Q\left(y\right)$ is an elementary $\delta$-distribution:

$$Q\left(y_1\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\frac{p}{1-p}\ge \frac{u_{22}-u_{21}}{u_{11}-u_{12}}=\frac{d_2}{d_1}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

The optimal value corresponding to $I=0$ information is $U\left(0\right)=p{c}_1+(1-p)c_2+|p{d}_1-(1-p)d_2|$. In the case when $c_1=c_2=c$ and $d_1=d_2=d$, the condition for $y_1$ is $d(2p-1)\ge 0$ and $U\left(0\right)=c+d|2p-1|$. With $c=1/2$ and $d=1/2$, the value $U\left(0\right)=\frac{1}{2}+\frac{1}{2}|2p-1|$ represents the best possible accuracy for prior probabilities $P\left(x\right)\in \{p,1-p\}$. If additional information about x is communicated, say by some random variable $z\in Z$, then the maximum possible improvement $V\left(I\right)=U\left(I\right)-U\left(0\right)$ is the value of this information. The first step in deriving function $U\left(I\right)$ for the value of Shannon’s information (2) is to obtain the expression for function $\mathsf{\Gamma }\left(\beta \right)={\mathsf{\Gamma }}_0\left(\beta \right)-H\left(X\right)$.

Writing Equation (4) in the matrix form $\parallel e^{\beta u(x,y)}{\parallel }^{T}P\left(x\right)e^{-\gamma (\beta ,x)}=1$ and using the inverse matrix $(\parallel e^{\beta u(x,y)}{{\parallel }^T)}^{-1}$ gives the solution for function $e^{-{\gamma }_0(\beta ,x)}=P\left(x\right)e^{-\gamma (\beta ,x)}$:

$$\left[\begin{array}{c}p{e}^{-\gamma (\beta ,x_1)}\\ (1-p)e^{-\gamma (\beta ,x_2)}\end{array}\right]={\left[\begin{array}{cc}{e}^{\beta u_{11}}& e^{\beta u_{21}}\\ e^{\beta u_{12}}& e^{\beta u_{22}}\end{array}\right]}^{-1}\left[\begin{array}{c}1\\ 1\end{array}\right]=\frac{1}{det\parallel e^{\beta u}{\parallel }^T}\left[\begin{array}{cc}{e}^{\beta u_{22}}& -e^{\beta u_{21}}\\ -e^{\beta u_{12}}& e^{\beta u_{11}}\end{array}\right]\left[\begin{array}{c}1\\ 1\end{array}\right],$$

where $det\parallel e^{\beta u}{\parallel }^T=e^{\beta (u_{11}+u_{22})}-e^{\beta (u_{12}+u_{21})}=2e^{\beta (c_1+c_2)}sinh\left[\beta (d_1+d_2)\right]$. This gives two equations:

$$\begin{array}{cc}\hfill p{e}^{-\gamma (\beta ,x_1)}& =\frac{e^{\beta u_{22}}-e^{\beta u_{21}}}{e^{\beta (u_{11}+u_{22})}-e^{\beta (u_{12}+u_{21})}}=e^{-\beta c_1}\frac{sinh\left(\beta d_2\right)}{sinh\left[\beta (d_1+d_2)\right]}=:e^{-{\gamma }_0(\beta ,x_1)}\hfill \\ \hfill (1-p)e^{-\gamma (\beta ,x_2)}& =\frac{e^{\beta u_{11}}-e^{\beta u_{12}}}{e^{\beta (u_{11}+u_{22})}-e^{\beta (u_{12}+u_{21})}}=e^{-\beta c_2}\frac{sinh\left(\beta d_1\right)}{sinh\left[\beta (d_1+d_2)\right]}=:e^{-{\gamma }_0(\beta ,x_2)}\hfill \end{array}$$

Therefore, the expression for function ${\mathsf{\Gamma }}_0\left(\beta \right):=p{\gamma }_0(\beta ,x_1)+(1-p){\gamma }_0(\beta ,x_2)$ is

$${\mathsf{\Gamma }}_0\left(\beta \right)=\beta [p{c}_1+(1-p)c_2]+ln|sinh\left[\beta (d_1+d_2)\right]|-pln|sinh\left(\beta d_2\right)|-(1-p)ln|sinh\left(\beta d_1\right)|.$$

Its first derivative ${\mathsf{\Gamma }}_0^{\prime }\left(\beta \right)$ gives the expression for $U\left(\beta \right)$:

$$U\left(\beta \right)=p{c}_1+(1-p)c_2+(d_1+d_2)coth\left[\beta (d_1+d_2)\right]-p{d}_{2}coth\left(\beta d_2\right)-(1-p)d_{1}coth\left(\beta d_1\right).$$

The expression for information is obtained from $I\left(\beta \right)=H\left(X\right)-[{\mathsf{\Gamma }}_0\left(\beta \right)-\beta {\mathsf{\Gamma }}_0^{\prime }\left(\beta \right)]$, where $H\left(X\right)=-plnp-(1-p)ln(1-p)$. Two functions $U\left(\beta \right)$ and $I\left(\beta \right)$ define parametric dependency $U\left(I\right)$ for the value of Shannon’s information (2).

Notice that function ${\mathsf{\Gamma }}_0\left(\beta \right)$ (and hence $U\left(\beta \right)$ and $I\left(\beta \right)$) depends in general on $P\left(x\right)\in \{p,1-p\}$. If, however, $c_1=c_2=c$ and $d_1=d_2=d$, then, using the formula $\frac{sinh\left(2x\right)}{sinh\left(x\right)}=2cosh\left(x\right)$, we obtain simplified expressions: ${\mathsf{\Gamma }}_0\left(\beta \right)=\beta c+ln\left[2cosh\left(\beta d\right)\right]$ and

$$\begin{array}{c}U\left(\beta \right)=c+dtanh\left(\beta d\right),I\left(\beta \right)=H\left(X\right)-[ln[2cosh\left(\beta d\right)]-\beta dtanh(\beta d\left)\right].\end{array}$$

Let us denote $\theta :=\frac{U-c}{d}=tanh\left(\beta d\right)\in [0,1]$. Then the expression for information is

$$\begin{array}{cc}\hfill I\left(\theta \right)& =H\left(X\right)-ln[2cosh\left({tanh}^{-1}\theta \right)]+\theta {tanh}^{-1}\theta \hfill \\ \hfill & =H\left(X\right)+ln\frac{1}{2}+\frac{1}{2}ln(1-{\theta }^2)+\frac{1}{2}\theta ln\frac{1+\theta }{1-\theta }\hfill \\ \hfill & =H_2\left[p\right]-H_2\left[\frac{1+\theta }{2}\right].\hfill \end{array}$$

In the first step we used the formulae $cosh\left({tanh}^{-1}\theta \right)=\frac{1}{\sqrt{1-{\theta }^2}}$ and ${tanh}^{-1}\theta =\frac{1}{2}ln\frac{1+\theta }{1-\theta }$. The last equation is written using binary entropies $H_2\left[p\right]=-plnp-(1-p)ln(1-p)$, which shows that an increase of information in a binary system is directly related to an increase of the probability $(1+\theta )/2\ge max\{p,1-p\}$ due to conditioning on the ‘message’ $z\in Z$ about the realization of $x\in X$. Additionally, substituting $\theta =(U-c)/d$ we obtain the closed-form expression:

$$I\left(U\right)=H_2\left[p\right]-H_2\left[\frac{1}{2}+\frac{1}{2}\frac{U-c}{d}\right]$$

(6)

Let us derive the equations for the output probabilities $Q\left(y\right)={\sum }_{x}P(y\mid x)P\left(x\right)$. This can be done using Equation (5), which in the matrix form is $\parallel e^{\beta u(x,y)}\parallel Q\left(y\right)=e^{\gamma (\beta ,x)}$. Thus, we obtain

$$\left[\begin{array}{c}q\\ 1-q\end{array}\right]={\left[\begin{array}{cc}{e}^{\beta u_{11}}& e^{\beta u_{12}}\\ e^{\beta u_{21}}& e^{\beta u_{22}}\end{array}\right]}^{-1}\left[\begin{array}{c}{e}^{\gamma (\beta ,x_1)}\\ e^{\gamma (\beta ,x_2)}\end{array}\right]=\frac{1}{det\parallel e^{\beta u}\parallel }\left[\begin{array}{cc}{e}^{\beta u_{22}}& -e^{\beta u_{12}}\\ -e^{\beta u_{21}}& e^{\beta u_{11}}\end{array}\right]\left[\begin{array}{c}{e}^{\gamma (\beta ,x_1)}\\ e^{\gamma (\beta ,x_2)}\end{array}\right],$$

where $det\parallel e^{\beta u}\parallel =e^{\beta (u_{11}+u_{22})}-e^{\beta (u_{12}+u_{21})}=2e^{\beta (c_1+c_2)}sinh\left[\beta (d_1+d_2)\right]$. This gives two equations:

$$Q\left(y_1\right)=\frac{p}{1-e^{-2\beta d_2}}+\frac{1-p}{1-e^{2\beta d_1}},Q\left(y_2\right)=\frac{1-p}{1-e^{-2\beta d_1}}+\frac{p}{1-e^{2\beta d_2}}.$$

It is easy to check that $Q\left(y_1\right)+Q\left(y_2\right)=1$. Additionally, if $p=1-p$, then $Q\left(y_1\right)\ge 0$ and $Q\left(y_2\right)\ge 0$ for all $\beta \ge 0$. However, when $p\ne 1-p$, there exists ${\beta }_0>0$ such that either $Q\left(y_1\right)<0$ or $Q\left(y_2\right)<0$ for $\beta \in [0,{\beta }_0)$. The value ${\beta }_0$ can be found from $Q\left(y_1\right)=0$ or $Q\left(y_2\right)=0$. For $d_1=d_2=d$ this value is

$${\beta }_0=\frac{1}{2d}\left|ln\left(\frac{p}{1-p}\right)\right|.$$

One can show that $I\left({\beta }_0\right)=0$ and $U\left({\beta }_0\right)=c+d|2p-1|$. Thus, the output probabilities are non-negative for all $\beta \ge {\beta }_0$, which corresponds to positive information $I\ge 0$ and $U\left(I\right)\ge U\left(0\right)$.

It is important to note that in the limit $\beta \to \infty$, corresponding to an increase of information to its maximum, the output probabilities $Q\left(y\right)\in \{q,1-q\}$ converge to $P\left(x\right)\in \{p,1-p\}$.

3. Application: Accuracy of Time-Series Forecasts

In this section, we illustrate how the value of information can facilitate the analysis of the performance of data-driven models. Here we use financial time-series data and predict the signs of future log returns. Thus, if $s\left(t\right)$ and $s(t-1)$ are prices of an asset at two time moments, then $r\left(t\right)=ln\left[s\right(t)/s(t-1\left)\right]$ is the log-return at t. The models will try to predict whether the future log return $r(t+1)$ is positive or negative. Thus, we have a $2\times 2$ system, where $x\in \{x_1,x_2\}$ is the true sign, and $y\in \{y_1,y_2\}$ is the prediction. The accuracy of different models will be evaluated against the theoretical upper bound, defined by the value of information.

The data used here are from the set of close-day prices $s\left(t\right)$ of several cryptocurrency pairs between 1 January 2019 and 11 January 2021. Figure 2 shows the price of Bitcoin against USD (left) and the corresponding log returns (right). Predicting price changes is very challenging. In fact, in economics, log returns are often assumed to be independent (and hence prices $s\left(t\right)$ are assumed to be Markov). Indeed, one can see no obvious relation on the left chart on Figure 3, which plots logreturns $r\left(t\right)$ (abscissa) and $r(t+1)$ (ordinates). In reality, however, some amounts of information and correlations exist, which can be seen from the plot of the autocorrelation function for BTC/USD shown on the right chart of Figure 3.

The idea of autoregressive models is to use the small amounts of information between the past and future values for forecasts. In addition to autocorrelations (correlations between the values of $\left\{r\right(t\left)\right\}$ at different times), information can be increased by using cross-correlations (correlations between log-returns of different symbols in the dataset). Thus, the vector of predictors used here is an $m\times n$-tuple, where m is the number of symbols used, and n is the number of time lags. In this paper, we report the results of models using the range $m\in \{1,2,\dots ,5\}$ of symbols (BTC/USD, ETH/USD, DAI/BTC, XRP/BTC, IOT/BTC) and $n\in \{2,3,\dots ,20\}$ of lags. This means that the models used predictors $(z_1,\dots ,z_{m\times n})$, where $m\times n$ ranged from 2 to 100. The model output $y\left(z\right)$ is the forecast of the sign $x\in \{-1,1\}$ (the response) of future log return $r(t+1)$ of BTC/USD. Here we report results from the following models:

• Logistic regression (LM). This model has no hyperparameters.
• Partial least squares discrimination (PLSD). We used the SIMPLS algorithm [7] with three components.
• Feed-forward neural network (NN). Here we used one hidden layer with three logistic units.

In order to analyse the performance of models using the value of information, one has to estimate the amount of information between the predictors $z_1,\dots ,z_{m\times n}$ and the response variable x. Here we employ two methods. The first uses the following Gaussian formula [4]:

$$I(X,Z)\approx \frac{1}{2}\left[lndet{K}_z+lndet{K}_x-lndet{K}_{z\oplus x}\right],$$

where $K_i$ are the covariance matrices. Because the distributions of log returns are generally not Gaussian, this formula is an approximation (in fact, it gives a lower bound). The second method is based on the discretization of continuous variables. Because models were used to predict signs of log returns, here we used discretization into two subsets. Figure 4 shows the average amounts of information $I(X,Z)$ in the training sets, computed using the Gaussian formula (left) and using binary discretization (right). Information (ordinates) is plotted against the number n of lags (abscissa) and for $m\in \{1,2,\dots ,5\}$ symbols (different curves). One can see that the amounts of information using Gaussian approximation (left) are generally lower than those using discretization (right). We note, however, that linear models can only use linear dependencies (correlations), which means that Gaussian approximation is sufficient for assessing the performance of linear models, such as LM and PLSD. Non-linear models, on the other hand, can potentially use all information present in the data. Therefore, we used information estimated with the second method to assess the performance of NN.

For each collection of predictors $(z_1,\dots ,z_{m\times n})$ and response x, the data were split into multiple training and testing subsets using the following rolling window procedure: we used 200- and 50-day data windows for training and testing, respectively; after training and testing the models, the windows were moved forward by 50 days and the process repeated. Thus, the data of approximately 700 days (January 2019 to January 2021) were split into $(700-200)/50=10$ pairs of training and testing sets. The results reported here are the average of results from these 10 subsets.

Figure 5 shows the accuracies of models plotted against information amounts I in the training data. The top row shows results on the training sets (i.e. fitted values) and the bottom row for new data (i.e., predicted values). Different curves are plotted for different numbers of symbols $m\in \{1,\dots ,5\}$. The theoretical upper bounds are shown by the $\mathrm{Accuracy}\left(I\right)$ curves computed using the inverse of function (6) with $c=d=1/2$ and $p=1/2$. Here we note the following observations:

• The accuracy of fitting the training data closely follows theoretical curve $\mathrm{Accuracy}\left(I\right)$. The accuracy of predicting new data (testing sets) is significantly lower.
• Increasing information increases the accuracy on training data, but not necessarily on new data.
• Models using $m>1$ symbols appear to achieve better accuracy than models using $m=1$ symbol with the same amounts of information. Thus, surprisingly, cross-correlations potentially provide more valuable information for forecasts than autocorrelations.

4. Discussion

We have reviewed the main ideas of Stratonovich’s value of information theory [2,4] and applied it to the simplest $2\times 2$ Bayesian system. We explicitly performed the main computations for the cumulant generating function $\mathsf{\Gamma }\left(\beta \right)={\mathsf{\Gamma }}_0\left(\beta \right)-H\left(X\right)$ and derived functions $U\left(\beta \right)$ and $I\left(\beta \right)$ defining the dependency $U\left(I\right)$ and the value of Shannon’s information $V\left(I\right)=U\left(I\right)-U\left(0\right)$. The main application of the considered binary example the is evaluation of the accuracy of model predictions or hypothesis testing. The analysis the of performance of data-driven models can be enriched by the use of the value of information. However, one needs to be careful about the estimation of the amount of information in the data. Gaussian approximation of mutual information can be used for linear models. However, other techniques should be used for the analysis of non-linear models, such as neural networks. Here we applied the value of information to the analysis of financial time-series forecasts. These methods can be generalized to many other machine learning and data science problems.