InfoMat: Leveraging Information Theory to Visualize and Understand Sequential Data ^†

Abstract

Despite the widespread use of information measures in analyzing probabilistic systems, effective visualization tools for understanding complex dependencies in sequential data are scarce. In this work, we introduce the information matrix (InfoMat), a novel and intuitive matrix representation of information transfer in sequential systems. InfoMat provides a structured visual perspective on mutual information decompositions, enabling the discovery of new relationships between sequential information measures and enhancing interpretability in time series data analytics. We demonstrate how InfoMat captures key sequential information measures, such as directed information and transfer entropy. To facilitate its application in real-world datasets, we propose both an efficient Gaussian mutual information estimator and a neural InfoMat estimator based on masked autoregressive flows to model more complex dependencies. These estimators make InfoMat a valuable tool for uncovering hidden patterns in data analytics applications, encompassing neuroscience, finance, communication systems, and machine learning. We further illustrate the utility of InfoMat in visualizing information flow in real-world sequential physiological data analysis and in visualizing information flow in communication channels under various coding schemes. By mapping visual patterns in InfoMat to various modes of dependence structures, we provide a data-driven framework for analyzing causal relationships and temporal interactions. InfoMat thus serves as both a theoretical and empirical tool for data-driven decision making, bridging the gap between information theory and applied data analytics.

1. Introduction

Information theory plays a key role in the analysis of dynamics in stochastic systems [1,2,3,4]. Through the lens of information theory, one can study the temporal evolution of dependence in a sequential system, which is often interpreted as the exchange of information between its interacting components. For example, in communication channels, directed information quantifies the information flow from the encoder to the channel [5,6]. Consequently, the feedback capacity of communication channels, which is characterized by the optimization of average directed information, can be interpreted as the maximum amount of information flow from the transmitter to the channel [7]. Another pertinent example is neuroscience [8], where information theory is widely used for the analysis and processing of collected data. For example, transfer entropy [9] is used to infer functional connectivity and to analyze information flow from recorded data, such as EEG and fMRI [10]. Beyond communications and neuroscience, information theory was shown useful for various fields of sequential analysis, encompassing control [11,12,13], reinforcement learning [14,15], causal inference [16,17,18], and various machine learning tasks [19,20,21].

In contrast to its wide use to infer and quantify relations in time series data, information theory fails to offer simple visualization tools to demonstrate its merits, and existing visualization techniques are not applicable to such settings. A common example of dependence visualization is the cross-correlation matrix [22]. For a given random vector pair $(X,Y)\in {\mathbb{R}}^{d_x+d_y}$, the correlation matrix is an ${\mathbb{R}}_{d_x\times d_y}$-valued matrix, whose $(i,j)$th entry is the correlation between $(X_i,Y_j)$. While being a powerful tool for simple exploratory analysis, the performance of the correlation matrix heavily relies on the data domain and structure of the joint distribution. Specifically, it provides a complete characterization of dependence only when the vectors’ entries are univariate Gaussian sequences with linear relations. Furthermore, the correlation matrix does not generalize to the conditional setting, which is crucial for visualization in time series analysis.

A common visualization of information theoretic relationships is the Venn diagram [4], which serves as a visualization of mutual information, through decomposition into joint and conditional entropies. Finally, the information diagram [23], which is based on the Taylor diagram [24], explores the interplay between entropies and mutual information in a geometric fashion by mapping dependence into an angle between the marginal entropy vectors. However, this method is limited to a random pair. Beyond the visualization of information measures, information theory is widely used to evaluate other visualization and rendering techniques [25]. In this work, we attempt to close this gap, and propose a new visualization method.

Contributions

In this work, we propose the information matrix (InfoMat), a novel visualization tool of the information transfer in dynamic stochastic systems. Given two stochastic sequences of length m, the InfoMat arranges the conditional mutual information terms that describe the evolution of dependence in an $m\times m$ matrix. By arranging the $m^2$ conditional information terms that describe the temporal processes’ evolution in an $m\times m$ matrix, various dependence patterns emerge in the InfoMat though visual patterns. We show how the InfoMat captures popular sequential information measures such as the directed information and transfer entropy through linear operations. We then demonstrate the InfoMat utilities for both theoretical and practical methods.

For theoretical purposes, the InfoMat provides a visual representation of existing information theoretic conservation laws and decompositions, while revealing new relationships. The relationships are proven by characterizing different subsets of the matrix with corresponding information measures. Additionally, the InfoMat serves as a practical visualization tool for arbitrary sequential data. Using a heatmap representation, we can link various dependence structures in the data with visual patterns. We propose a Gaussian mutual information estimator of the InfoMat that relies on the calculation of sample covariance matrices and analyze its theoretical guarantees, while empirically demonstrating the power of the resulting visualization tool. When a Gaussian estimator is insufficient, we develop a neural InfoMat estimator, which is based on masked normalizing flows (MAFs), which expand the class of distributions captured by the InfoMat. We demonstrate the estimated InfoMat to visualize the power of optimal coding schemes in communication channels with memory and for the visualization of information flows in real-world datasets.

The rest of this paper is organized as follows. Section 2 presents required background and preliminaries, followed by Section 3, which presents the InfoMat and demonstrates its capabilities for the visualization of theoretical information-theoretic conservation laws. Then, Section 4 discusses the Gaussian estimator of the InfoMat, while Section 5 explores neural estimation. Finally, Section 6 demonstrates the utility of the estimated InfoMat for visualizing dependence structures in sequential data, and Section 7 provides concluding remarks and discusses future work.

2. Background and Preliminaries

2.1. Notation

Sets are denoted by calligraphic letters, e.g., $\mathcal{X}$. When $\mathcal{X}$ is finite, we use $\left|\mathcal{X}\right|$ for its cardinality. For any $n\in \mathbb{N}$, ${\mathcal{X}}^n$ is the n-fold Cartesian product of $\mathcal{X}$, while $x^n=(x_1,\dots ,x_n)$ denotes an element of ${\mathcal{X}}^n$. For $i,j\in \mathbb{Z}$ with $i\le j$, we use the shorthand $x_i^j\triangleq (x_i,\dots ,x_j)$; the subscript is omitted when $i=1$. Expectations are denoted by $\mathbb{E}$. When $\mathcal{X}$ is countable, we use p for the PMF associated with the probability measure P. Random variables are denoted by upper-case letters, e.g., X. The Kullback–Leibler (KL) divergence between P and Q, with $P\ll Q$, is ${\mathsf{D}}_{\mathsf{KL}}(P\parallel Q)\triangleq {\mathbb{E}}_P\left[log{\displaystyle \frac{\mathrm{d}P}{\mathrm{d}Q}}\right]$. The mutual information between $(X,Y)\sim P_{XY}$ is $I(X;Y)\triangleq {\mathsf{D}}_{\mathsf{KL}}(P_{XY}\parallel P_X\otimes P_Y)$, where $P_X$ and $P_Y$ are the marginals of $P_{XY}$. The entropy of a discrete random variable $X\sim P$ is $H\left(X\right)\triangleq -\mathbb{E}\left[logp\left(X\right)\right]$.

2.2. Sequential Information Measures

Consider a pair of jointly distributed sequences of length m, $(X^m,Y^m)\sim P_{X^m,Y^m}$. The causal nature of sequential communication systems impedes mutual information from properly describing sequential information flows, as it decomposes into non-causal conditional mutual information terms, i.e., [26]

$$I(X^m;Y^m)=\sum _{i=1}^{m}I(X_i;Y^m|X^{i-1}),$$

To this end, several causal adaptations of mutual information to time series data were developed in the literature. The first is directed information [5], which was originally developed to characterize information rates in communication channels with feedback. The directed information from $X^m$ to $Y^m$ is given by

$$I(X^m\to Y^m)\triangleq \sum _{i=1}^{m}I(X^i;Y_i|Y^{i-1}),$$

(1)

where the directed information in the opposite direction $I(Y^m\to X^m)$ is defined symmetrically. The second information measure, which gained popularity in neuroscience and physics is transfer entropy [9,27,28]. For parameters $(m,k,l)$, the transfer entropy is given by

$$T_m^{X\to Y}(k,l)\triangleq I(X_{m-k}^{m-1};Y_m|Y_{m-l}^{m-1}),$$

(2)

Transfer entropy and directed information follow various decompositions and information conservation laws. We will further discuss them through the lenses of the proposed InfoMat in the upcoming section.

In physics, time series measures—such as approximate entropy [29], sample entropy [30], permutation entropy [31], and dispersion entropy [32]—are widely used to quantify the local complexity and irregularity of sequential data. While these scalar metrics provide valuable insights into local dynamical properties, they are conceptually distinct from directed information and transfer entropy, as they are not based on quantification through conditional mutual information.

2.3. Information Decomposition and Conservation

In stochastic systems with memory, the temporal evolution of the dependence between the system’s elements can be viewed through the lenses of information exchange. This exchange of information can be captured within conversation laws, which quantify the total amount of information flow in a given system. Specifically, for a system with m time steps and two sources $(X^m,Y^m)\sim P_{X^m,Y^m}$, Massey [33] proved the following law of information conservation

$$I(X^m;Y^m)=I(X^m\to Y^m)+I(\mathsf{D}\circ Y^m\to X^m)$$

(3)

where $({\mathsf{D}}^k\circ X^m)$ is a left concatenation of k ‘dummy’ deterministic symbols with $X_{k+1}^m$ and $(\mathsf{D}\circ X^m)=({\mathsf{D}}^1\circ X^m)$. The authors of [34] propose a modification of (3) that distinguishes between past and present effects, given by

$$\begin{array}{c}\hfill I(X^m;Y^m)=I(\mathsf{D}\circ X^m\to Y^m)+I(\mathsf{D}\circ Y^m\to X^m)+I_{\mathsf{inst}}(X^m,Y^m),\end{array}$$

(4)

where $I_{\mathsf{inst}}(X^m,Y^m)\triangleq {\sum }_{i=1}^{n}I(X_i;Y_i|X^{i-1},Y^{i-1})$ is the instantaneous information, which measures the symmetric dependence between $X^m$ and $Y^m$, given a shared history. Information conservation has been shown to be useful for the analysis of causal information flows in neural spike trains and financial data [35,36].

3. Information Matrix

Fix $m\in \mathbb{N}$. We are interested in characterizing the interaction between two stochastic sequences (the sequences can be thought of as an m-step sample from some underlying joint stochastic process) of length m in a visually meaningful manner. Denote the sequences with $(X^m,Y^m)\sim P_{X^m,Y^m}$. Our emphasis is on the information-theoretic description of this interaction, which can be seen as the evolution of dependence between the interacting components over time. This characterization can alternatively be viewed as a transfer of information. The entire dependence structure between $X^m$ and $Y^m$ is captured by the m-fold mutual information, given by the following chain rule [4]

$$I(X^m;Y^m)=\sum _{i=1}^m\sum _{j=1}^{m}I(X_i;Y_j|X^{i-1},Y^{j-1}).$$

(5)

The above decomposition implies that, along m steps, the interaction is characterized with $m^2$ conditional mutual information terms. This acts as our motivation for the following definition and this work.

We define the InfoMat as the following $m\times m$ matrix:

$${\mathrm{I}}^{X,Y}\in {\mathbb{R}}_{\ge 0}^{m\times m},{\mathrm{I}}_{i,j}^{X,Y}\triangleq I(X_i;Y_j|X^{i-1},Y^{j-1}).$$

(6)

The InfoMat captures all the information transfer within the two-user system $(X^m,Y^m)$ as (5) implies that

$$I(X^m;Y^m)=\sum _{i,j}{\mathrm{I}}_{i,j}^{X,Y}={\mathbf{1}}_m{\mathrm{I}}^{X,Y}{\mathbf{1}}_m^{⊺},$$

where ${\mathbf{1}}_m\in {\mathbb{R}}^m$ is an m-length vector of ones and the inequality follows from the chain rule for mutual information [4].

As we further show, the InfoMat allows for the interpretations of the chain rule (5), and other information decomposition laws of $I(X^m;Y^m)$ as grouping the entries of ${\mathrm{I}}^{X,Y}$ into meaningful subsets that sum up to $I(X^m;Y^m)$. These decompositions are often termed information conservation laws [33]. We will show that such laws, whose proofs are usually technical and algebraic, can be easily visualized by coloring subsets of the entries of ${\mathrm{I}}^{X,Y}$.

3.1. Visualizing Sequential Information Measures

We begin by demonstrating that the aforementioned sequential information measures can be recovered through the summation of elements of the InfoMat. Directed information (1) is given by the sum of a triangular sub-matrix, i.e.,

$$I(X^m\to Y^m)={\mathbf{1}}^{\mathsf{T}}\left(\begin{array}{cccc}{\mathrm{I}}_{1,1}^{X,Y}& {\mathrm{I}}_{1,2}^{X,Y}& \dots & {\mathrm{I}}_{1,m}^{X,Y}\\ 0& {\mathrm{I}}_{2,2}^{X,Y}& \ddots & ⋮\\ ⋮& \ddots & \ddots & {\mathrm{I}}_{m-1,m}^{X,Y}\\ 0& \dots & 0& {\mathrm{I}}_{m,m}^{X,Y}\end{array}\right)\mathbf{1}.$$

(7)

Due to its direction sensitivity, directed information in each direction is associated with a certain triangular sub-matrix. Specifically, the direction $X^m\to Y^m$ corresponds to the upper triangular part of ${\mathrm{I}}^{X,Y}$, while the lower triangular part represents the direction $Y^m\to X^m$.

It is also useful to define the time-delayed directed information, which describes the causal flow of information under a time delay at the transmitting node. For a delay of $k<m$, the k-time-delayed directed information is given by $I({\mathsf{D}}^k\circ X^m\to Y^m)\triangleq {\sum }_{i=1}^{m}I(X^{i-k};Y_i|Y^{i-1})$. In ${\mathrm{I}}^{X,Y}$, a k-delay of directed information corresponds to a right shift of k indices in the $X^m\to Y^m$ direction, and a down shift in the $Y^m\to X^m$ direction.

Next, the transfer entropy term $T_{i+1}^{X\to Y}(i,i)=I(X^i;Y_{i+1}|Y^i)$, which quantifies the causal effect of $X^i$ on $Y_{i+1}$ given $Y^i$ is given by the sum over a column of length i in row $i+1$. For example, we have

$$T_3^{X\to Y}(2,2)={\mathbf{1}}^{\mathsf{T}}\left(\begin{array}{ccccc}0& \dots & {\mathrm{I}}_{1,3}^{X,Y}& \dots & 0\\ ⋮& 0& {\mathrm{I}}_{2,3}^{X,Y}& \ddots & ⋮\\ ⋮& \ddots & 0& \ddots & ⋮\\ ⋮& \ddots & \ddots & 0& ⋮\\ 0& \dots & \dots & \dots & 0\end{array}\right)\mathbf{1}.$$

(8)

Note that a transfer entropy does not include terms on the InfoMat diagonal. This stresses that transfer entropy focuses on strict past influence on present interaction, which is a key distinction from directed information [37]. The relation between mentioned information measures and the patterns in ${\mathrm{I}}^{X,Y}$ are summarized in

Table 1. Visual shapes of dependence patterns in ${\mathrm{I}}^{X,Y}$.

Information measure	Visual pattern in ${\mathrm{I}}^{X,Y}$
$I(X^m\to Y^m)$	Upper triangular with diagonal
$I({\mathsf{D}}^k\circ X^m\to Y^m)$	Upper triangular, side $(m-k)$
$T_{i+1}^{X\to Y}(i,i)$	Col. in row $i+1$ with length i

.

3.2. Capturing Information Conservation Laws

Having identified the relation between information measures and their patterns in ${\mathrm{I}}^{X,Y}$, we can visualize the aforementioned laws of information conservation [33,34]. Such rules are often derived via algebraic manipulation of information measures and therefore may lack intuition and may fail to provide a deeper understanding of the underlying interaction. Recall that Massey’s information conservation law is given by (3)

$$I(X^m;Y^m)=I(X^m\to Y^m)+I(\mathsf{D}\circ Y^m\to X^m)$$

(9)

Following the identification of DI with triangular submatrices of ${\mathrm{I}}^{X,Y}$, (3) follows by coloring index subsets and summing over each color group.

$$I(X^m;Y^m)={\mathbf{1}}^{\mathsf{T}}\left(\begin{array}{cccc}{\mathrm{I}}_{1,1}^{X,Y}& {\mathrm{I}}_{1,2}^{X,Y}& \dots & {\mathrm{I}}_{1,m}^{X,Y}\\ {\mathrm{I}}_{2,1}^{X,Y}& {\mathrm{I}}_{2,2}^{X,Y}& \ddots & ⋮\\ ⋮& \ddots & \ddots & {\mathrm{I}}_{m-1,m}^{X,Y}\\ {\mathrm{I}}_{m,1}^{X,Y}& \dots & {\mathrm{I}}_{m,m-1}^{X,Y}& {\mathrm{I}}_{m,m}^{X,Y}\end{array}\right)\mathbf{1}.$$

(10)

Considering the finer decomposition using instantaneous information (4), we note that $I_{\mathsf{inst}}(X^n,Y^n)=\mathsf{Trace}\left({\mathrm{I}}^{X,Y}\right)$. We can therefore identify this decomposition by excluding the diagonal from the upper sub-triangle.

$$I(X^m;Y^m)={\mathbf{1}}^{\mathsf{T}}\left(\begin{array}{cccc}{\mathrm{I}}_{1,1}^{X,Y}& {\mathrm{I}}_{1,2}^{X,Y}& \dots & {\mathrm{I}}_{1,m}^{X,Y}\\ {\mathrm{I}}_{2,1}^{X,Y}& {\mathrm{I}}_{2,2}^{X,Y}& \ddots & ⋮\\ ⋮& \ddots & \ddots & {\mathrm{I}}_{m-1,m}^{X,Y}\\ {\mathrm{I}}_{m,1}^{X,Y}& \dots & {\mathrm{I}}_{m,m-1}^{X,Y}& {\mathrm{I}}_{m,m}^{X,Y}\end{array}\right)\mathbf{1}.$$

(11)

Next, we will leverage the constructed relations between InfoMat patterns and sequential information measures to derive new information-theoretic decompositions and formulas.

3.3. Developing New Information-Theoretic Relations

Beyond the visualization of existing relationships, the simplicity of the InfoMat visualization allows us to develop new meaningful information-theoretic equivalences. We begin with the following proposition that relates the two information measures of interest.

Proposition 1

(Transfer entropic decomposition of directed information). For $(X^m,Y^m)\sim P_{X^m,Y^m}$ and $1\le k\le m$, we have

$$I({\mathsf{D}}^k\circ X^m\to Y^m)=\sum _{i=1}^{m-k}{T}_{i+1}^{X\to Y}(i,i).$$

Proof. 

We provide the proof for $k=1$. Recall that directed information corresponds to the upper triangular part of ${\mathrm{I}}^{X,Y}$. We note that $T_{i+1}^{X\to Y}(i,i)$ corresponds to a column in ${\mathrm{I}}^{X,Y}$ that begins in the $(i+1)$th column and has length i. Thus, we color the triangular part (directed information) as follows

$$\left(\begin{array}{ccccc}& {\mathrm{I}}_{1,2}^{X,Y}& {\mathrm{I}}_{1,3}^{X,Y}& \dots & {\mathrm{I}}_{1,m}^{X,Y}\\ & & {\mathrm{I}}_{2,3}^{X,Y}& \ddots & ⋮\\ & & & \ddots & ⋮\\ & & & & {\mathrm{I}}_{m-1,m}^{X,Y}\\ & & & & \end{array}\right)$$

The relation then follows by summing over the upper triangular part and dividing the sum into the corresponding rows, i.e.,

$$\begin{array}{cc}\hfill & I(\mathsf{D}\circ X^n\to Y^n)=T_2^{X\to Y}(1,1)+T_3^{X\to Y}(2,2)+\cdots +T_m^{X\to Y}(m-1,m-1)\hfill \end{array}$$

The proof similarly extends to $k>1$ with similar steps.    □

The proof demonstrates the simplicity of InfoMat, as it boils down to identifying the aforementioned information measures as entry subsets within ${\mathrm{I}}^{X,Y}$. Equipped with Proposition 1, we can provide an information conservation rule in terms of transfer entropy terms.

Proposition 2

(Conservation of transfer entropy). Let $(X^m,Y^m)\sim P_{X^m,Y^m}$. Then

$$I(X^m;Y^m)=\sum _{i=1}^{m-1}{T}_{i+1}^{X\to Y}(i,i)+T_{i+1}^{Y\to X}(i,i)+I_{\mathsf{inst}}(X^m,Y^m).$$

Finally, we propose a recursive decomposition of directed information via transfer entropy.

Proposition 3

(Directed information chain rule). Let $(X^m,Y^m)\sim P_{X^m,Y^m}$. Then

$$\begin{array}{c}\hfill I({\mathsf{D}}^k\circ X^m\to Y^m)=I({\mathsf{D}}^{k+1}\circ X^m\to Y^m)+\sum _{i=1}^{m-k}I(X_{i-k};Y_i|Y^{i-1}).\end{array}$$

The proofs of Propositions 2 and 3 follow by arguments and tools similar to Proposition 1 and are given in Appendix A.1 for completeness. The new derived information-theoretic conservation laws and decompositions elucidate the relationships between mutual information, directed information, and transfer entropy. Strengthening these connections is crucial, as each measure has distinct tools and applications, thereby enhancing their cross-utilization potential and potentially bridging various applications. While these proposed relations can indeed be derived through existing algebraic manipulations of mutual information, the technical complexity may make these deductions less apparent.

3.4. Relating Dependency Structures and Visual Patterns in the InfoMat

We now show that, beyond the derivation of various information decomposition laws, the InfoMat can be used to identify temporal patterns in the data. To this end, we relate various dependence structures to corresponding visual patterns in the InfoMat. We later demonstrate those relations on data via various estimates of the InfoMat (Section 6). To this end, we assume in this section that the sequence $(X^m,Y^m)$ is an m-fold projection of some jointly stationary stochastic process defined over $\mathcal{X}\times \mathcal{Y}$. As a first example, we note that when the joint process is independent and identically distributed (i.i.d.), every conditional mutual information with $i\ne j$ vanishes. The corresponding InfoMat takes the form ${\mathrm{I}}^{X,Y}=I(X;Y){\mathrm{I}}_m$ with ${\mathrm{I}}_m$ being the m-dimensional identity matrix.

Next, we focus on a specific case of interest, when the joint process is jointly Markov with some order k. In this case, we know that, for ${\mathrm{I}}_{i,j}^{X,Y}$, when both i and j are larger than k, we obtain a similar value of conditional mutual information due to joint stationarity. This implies that within the square block within ${\mathrm{I}}^{X,Y}$ that is determined by the indices $\left\{\right(i,j\left)\right|i>k,j>k\}$, we have a Toeplitz structure (which is a matrix whose constant along its diagonals.). Furthermore, when $|i-j|>k$, due to the joint Markov nature of the process, we have ${\mathrm{I}}_{i,j}^{X,Y}=0$. This implies a banded structure of the InfoMat, which is determined by a k-width ‘Markov band’ outside the main diagonal. This is useful for InfoMat estimation. As we further elaborate in the next sections, estimating the InfoMat may be generally computationally expensive, as we are required to estimate the $m^2$ conditional mutual information terms. However, when the joint process is a stationary Markov of order k, the number of distinct conditional mutual information terms reduces to $O\left(km\right)$. We demonstrate this relation between dependence structures and visual patterns on a simple example in Figure 1. The discussion readily extends to asymmetric Markov orders for the X and Y processes.

Finally, we note that, as initially observed in Section 3.1, various areas in the InfoMat correspond to different directions of information. Specifically, values in the upper triangular correspond to information flow in the direction $X^m\to Y^m$, while information flow in the opposite direction is represented by the lower triangular. Therefore, the trace of ${\mathrm{I}}^{X,Y}$ represented the instantaneous exchange of information, also quantifying its operator norm. We believe that further relations and linear algebraic can be unveiled, with task and setting specific structures. In Section 6, we demonstrate these relations on InfoMat estimates from data.

4. InfoMat Estimation via Gaussian Mutual Information

Beyond its theoretical merit as a proof visualization tool, we argue that the InfoMat is also effective for the visualization and analysis of dependence structures in time series data. However, the underlying data distribution is often unknown. Even if it is known, the corresponding conditional mutual information terms may not be given in closed form. To this end, to utilize the InfoMat as a visualization tool in real data setting, an estimator is required. In this section, we propose an approximation of ${\mathrm{I}}^{X,Y}$ from samples of the joint distribution $P_{X^m,Y^m}$.

Estimating ${\mathrm{I}}^{X,Y}$ boils down to the estimation of $m^2$ conditional mutual information terms. Without assumptions on the data distribution, the complexity and performance of mutual information estimators tend to deteriorate with the length of the conditioned joint history. That is, the bigger $(i,j)$ are, the more samples are required and the worse the performance of the single ${\mathrm{I}}^{X,Y}$ entry is expected to be. To that end, we begin by proposing a data-efficient estimation of mutual information, focusing on an approximation that follows from the Gaussian mutual information term, which is given in closed form. In this case, estimating the entries of ${\mathrm{I}}^{X,Y}$ boils down to the estimation of the corresponding covariance matrices, whose guarantees are well studied, and for which we can use estimators with parametric error rates. We begin by constructing and analyzing the proposed estimator. Then, we analyze its theoretical performance.

4.1. Proposed Estimator

Let $(X^n,Y^n)\sim P_{X^n,Y^n}$ be a given dataset from which we want to estimate ${\mathrm{I}}^{X,Y}$. For simplicity, we assume that all variables have zero mean. We begin by using the following representation of conditional mutual information.

Lemma 1.

Let $(X^m,Y^m)\sim P_{X^m,Y^m}$, and $1\le i,j\le m$. Then,

$$\begin{array}{cc}\hfill I(X_i;Y_j|X^{i-1},Y^{j-1})& =H(X^i,Y^{j-1})+H(X^{i-1},Y^j)-H(X^{i-1},Y^{j-1})-H(X^i,Y^j).\hfill \end{array}$$

(12)

If $(X^m,Y^m)$ are jointly Gaussian, then

$$I(X_i;Y_j|X^{i-1},Y^{j-1})={\mathrm{I}}_{G,i,j}^{X,Y}=\triangleq {\displaystyle \frac{1}{2}}log{\displaystyle \frac{\left|{\mathrm{K}}_{X^i,Y^{j-1}}\right|\left|{\mathrm{K}}_{X^{i-1},Y^j}\right|}{\left|{\mathrm{K}}_{X^{i-1},Y^{j-1}}\right|\left|{\mathrm{K}}_{X^i,Y^j}\right|}},$$

(13)

where ${\mathrm{K}}_Z$ is the covariance matrix of $Z\sim P_Z$, and $|{\mathrm{K}}_Z|$ representing its determinant.

To estimate ${\mathrm{I}}_{i,j}^{X,Y}$ from a dataset $(x^n,y^n)$ using the Gaussian estimator (13), we estimate the corresponding sample covariance matrices ${\widehat{\mathrm{K}}}_{i,j}\triangleq {\widehat{\mathrm{K}}}_{X^i,Y^j}$, and plug those into (13). We denote the Gaussian estimator of ${\mathrm{I}}_{i,j}^{X,Y}$ with ${\widehat{\mathrm{I}}}_{G,i,j}^{X,Y}(x^n,y^n)$. Finally, a Gaussian estimator of ${\mathrm{I}}^{X,Y}$ is an $m\times m$ matrix whose $(i,j)$ entry is given by ${\widehat{\mathrm{I}}}_{G,i,j}^{X,Y}$. The Gaussian mutual information estimation procedure is summarized in Algorithm 1.

Algorithm 1 Gaussian InfoMat Estimation

input: Data $(x^n,y^n)$, matrix length m output: Gaussian estimate of ${\mathrm{I}}^{X,Y}$

Initialize ${\widehat{\mathrm{I}}}_{G,i,j}^{X,Y}=0$ for $(i,j)\in (1,\dots ,m)\times (1,\dots ,m)$ for $(i,j)$ in $(1,\dots ,m)\times (1,\dots ,m)$ do     Divide $(x^n,y^n$) into datasets $${({(x^{i-1},y^{j-1})}_l,{(x^i,y^{j-1})}_l,{(x^{i-1},y^j)}_l,{(x^i,y^j)}_l)}_{l=1}^N$$     Calculate sample covariance matrices     Calculate ${\widehat{\mathrm{I}}}_{G,i,j}^{X,Y}$ via (13). return Estimated InfoMat.

The proposed Gaussian estimator for ${\mathrm{I}}^{X,Y}$ relies on the estimation of covariance matrices. Therefore, it inherits its guarantees from those of log determinants of sample covariance matrices. We assume that the underlying data-generating process is stationary, and obtain the dataset for the estimation of ${\mathrm{I}}_{i,j}^{X,Y}$ by dividing $(x^n,y^n)$ into the corresponding $i\times j$ sequences. We thus have the following

Proposition 4

(Gaussian estimator performance guarantees). Let $(X^n,Y^n)$ be a sequence of jointly Gaussian random vectors over ${\mathbb{R}}^{d_x+d_y}$ and let $d=max(d_x,d_y)$. Then

1. 

Bias: ${lim}_{n\to \infty }\mathbb{E}\left[{\widehat{\mathrm{I}}}_{G,i,j}^{X,Y}\right]={\mathrm{I}}_{G,i,j}.$

2. 

Variance: ${lim}_{n\to \infty }\mathsf{Var}\left({\widehat{\mathrm{I}}}_{G,i,j}^{X,Y}\right)=O\left({\displaystyle \frac{d{m}^2}{n}}\right)=O\left({\displaystyle \frac{1}{n}}\right).$

The proof follows from the analysis in [38,39], and the dependence on $m^2$ follows from the division of the dataset $(x^n,y^n)$ into the corresponding subsequences.

We note that, as m grows, the performance of the Gaussian estimator deteriorates, as the corresponding conditional mutual information term considers higher dimensional variables. To this end, we propose an alternative dataset acquisition approach—divide $(x^n,y^n)$ into $n-m$ samples such that the lth subsequence for $infoma{t}_{i,j}$ is given by $(x_l^{l+i},y_l^{l+j})$. This provides us with a significantly larger effective dataset when n is not bigger than m by orders of magnitude. However, the resulting sampled sequences are no longer i.i.d., and therefore, the estimator’s guarantees no longer hold. We refer to such a dataset as a “correlated dataset”, and use it for the visualization of real-world data when data availability is low.

While estimating ${\mathrm{I}}_{G,i,j}^{X,Y}$ is a simpler task, a Gaussian approximation can capture only partial information when the data distribution is far from a joint Gaussian. We propose an upper bound on the error of using the Gaussian approximation.

Proposition 5.

Let $(X,Y,Z)\sim P_{X,Y,Z}$ and let $\left(P_{X_G,Y_G,Z_G}\right)$ be the corresponding Gaussian joint distribution with the same moments as $\left(P_{Z,Y,Z}\right)$. We have the following bound:

$$\begin{array}{cc}\hfill \left|I(X;Y|Z)-I(X_G;Y_G|Z_G)\right|& \le \hfill \\ & {\mathsf{D}}_{\mathsf{KL}}(P_{X,Y|Z}\parallel P_{X_G,Y_G|Z}|P_Z)-{\mathsf{D}}_{\mathsf{KL}}(P_{X|Z}\otimes P_{Y|Z}\parallel P_{X_G|Z}\otimes P_{Y_G|Z}\left|P_Z\right)\hfill \\ & +\underset{z\in \mathcal{Z}}{max}{\mathsf{D}}_{\mathsf{KL}}(P_{X_G,Y_G|Z_G=z}\parallel P_{X_G|Z_G=z}\otimes P_{Y_G|Z_g=z})d_{\mathsf{TV}}(p_Z,p_{z_G})\hfill \end{array}$$

(14)

The proof of Proposition 5 is given in Appendix A.2.

4.2. InfoMat Estimation for Discrete Datasets

When the data domain is discrete, i.e., $X^m$ and $Y^m$ are drawn from some finite sets $\mathcal{X}$ and $\mathcal{Y}$, respectively, the Gaussian mutual information estimator is no longer valid. In such a case, we propose a plug-in estimator of mutual information. The estimator relies on the entropic factorization in Lemma 1, followed by a standard plug-in estimator for each entropy term. For completeness, we provide more details on the plug-in estimation methodology and demonstrate its application to the InfoMat in Appendix A.3. In the proposed applications, the plug-in estimator had demonstrated satisfactory results. However, its complexity grows exponentially with the size of conditioned history. In these situations, context tree weighting methodologies [36] can be utilized, to which our approach seamlessly extends.

5. Beyond Gaussian—Neural Estimation

Despite its simplicity and data efficiency, the Gaussian mutual information approximation can only fully capture the dependence structure under strong assumptions, which are often violated [40]. To this end, we propose a conditional mutual information estimator that does not require joint Gaussianity. The algorithm relies on the concept of neural estimation [41,42,43,44], which utilizes a variational formula of the measure of interest and optimizes it with neural networks. With the purpose of maintaining the simplicity of the Gaussian method, we turn to a recent scheme that leverages normalizing flows. We utilize the recently proposed diffeomorphic conditional mutual information estimator [39] that leverages a type of network optimization scheme termed MAFs [45]. MAF-based estimators of mutual information map the jointly distributed pair into a corresponding Gaussian pair, such that the learned mapping is a diffeomorphism (which is a differentiable invertible map with a differentiable inverse). In what follows, we introduce the proposed estimator and discuss its performance.

5.1. Masked Autoregressive Flows

This section provides a high-level description of MAFs. For an in-depth discussion, we refer the reader to [39,45]. Consider a pair $(X,Y)$ that has a conditional distribution $P_{XY|Z}$ for some random variables Z, such that $P_{XY|Z=z}$ is absolutely continuous for any $z\in \mathcal{Z}$. The employed estimator consists of two stages and relies on obtaining a diffeomorphism that maps $(X,Y)$ into a Gaussian pair $(X^{\prime },Y^{\prime })$. The second stage consists of calculation of the $I(X^{\prime };Y^{\prime }|Z)$ which has a simple form and will be a proxy for $I(X;Y|Z)$.

To learn a parametric diffeomorphism, we optimize an MAF. MAFs map samples from a (usually simple) base distribution $p\left(U\right)$ to samples from an arbitrary target distribution $p\left(X\right)$, assuming both are absolutely continuous and defined on $\mathcal{U}\subseteq {\mathbb{R}}^d$ and $\mathcal{X}\subseteq {\mathbb{R}}^d$ for some finite $d\in \mathbb{N}$. We denote the MAF with $T_{\theta }$. MAFs are designed such that their Jacobian is a triangular matrix. This implies that its determinant is simply given by the product of its diagonal. This property is crucial for the design as we are interested in representing the likelihood of the target distribution as a function of the likelihood of the base distribution and the partial derivatives along the parametric map $p_{\theta }$. For example, when $p\left(U\right)=\mathsf{Unif}{[0,1]}^d$, the parametric likelihood of X under $T_{\theta }$ is given by

$$log{p}_{\theta }\left(x\right)=log\left(d\right)+\sum _{i=1}^{d}log\left|{\displaystyle \frac{\partial u_i}{\partial x_i}}\right|,$$

(15)

where $\left|{\displaystyle \frac{\partial u_i}{\partial x_i}}\right|$ is the Jacobian of $T_{\theta }$. The map $T_{\theta }$ can be realized by neural networks with masked weight matrices and monotone activations [45]. Training an MAF consists of maximum likelihood optimization. Therefore, it is trained using minibatch gradient methods with (15) serving as the loss for $\theta$. This scheme readily adapts to conditional distributions $p_{\theta }\left(x\right|z)$. The generalization considers a conditioner model $g_{{\theta }^{\prime }}\left(z\right)$ with parameters ${\theta }^{\prime }$, whose purpose is to transfer the relevant information in Z about X into $T_{\theta }$. The conditional MAF is trained similarly to the unconditional MAF, with the slight change that now $x_i=f_{\theta }\left(u\right)+g_{{\theta }^{\prime }}\left(z\right)$. Note that the conditioner model need not to be a diffeomorphism.

5.2. Proposed Estimator

Equipped with an MAF model, we can discuss the proposed diffeomorphic conditional mutual information estimator from [39]. We demonstrate the method on an unconditional mutual information term, and then discuss the required modification to introduce conditioning. Recall that our goal is to map the pair $(X,Y)$ into a Gaussian pair $(X^{\prime },Y^{\prime })$. We break this task into a concatenation of two stages. First, we will map $(X,Y)$ into a uniform distribution over ${[0,1]}^{d_x+d_y}$ by learning a MAF. Then, the uniform distribution will be mapped into a Gaussian distribution using the inverse of the Gaussian cumulative distribution function (CDF) This method is often referred to as generalized inverse transform sampling [46].

To map $(X,Y)$ into a pair of uniform random variables, the learned MAF maps each variable into a mixture of Gaussian CDFs. Specifically, for both X and Y, we learn a mapping $\tau :$ as follows

$$\tau (x_i,h_i)=\sum _{j=1}^k{w}_{i,j}\left(h_i\right)\Phi (x_i;{\mu }_{i,j}\left(h_i\right),{\sigma }_{i,j}^2\left(h_i\right)),$$

where $\Phi$ is the Gaussian CDF with mean $\mu$ and variance ${\sigma }^2$, ${\left(w_{i,j}\right)}_{i,j}$ are the model parameters, and $h_i$ is some parametric autoregressive summary map, i.e., $h_i=h_i\left(x_{<i}\right)$. As explained in [39], the reasoning behind this design choice is that $\partial \tau /\partial x_i$ is a Gaussian mixture model, which is known to be a universal approximator of probability distributions. Finally, the decomposition of $\tau$ and the inverse Gaussian CDF yields a parametric model that maps an arbitrary random variable into a Gaussian variable. Thus, when applied to X and Y, we result with a pair of Gaussian variables $X^{\prime }$ and $Y^{\prime }$.

Having mapped $(X,Y)$, which are distributed according to the conditional distribution $P_{XY|Z}$ into the Gaussian pair, we can use the following result on conditional mutual information invariance.

Proposition 6

(Conditional mutual information invariance). Let $(X,Y,Z)\sim P_{X,Y,Z}$ and denote by $(X^{\prime },Y^{\prime })=(f_{\theta }(X,Z),g_{\varphi }(Y,Z))$, where $f_{\theta }$ and $g_{\varphi }$ are conditional diffeomorphisms. Then,

$$I(X;Y|Z)=I(X^{\prime };Y^{\prime }|Z)$$

(16)

Proposition 6 is a slight modification of ([39] (Lemma 2)). The existence of optimal MAFs is guaranteed by the universal approximation properties of normalizing flows [47]. Finally, the Gaussian mutual information term is calculated from sample covariance matrices, as elaborated in Section 4. The MAF-based scheme is depicted in Figure 2, and the algorithm steps are summarized in Algorithm 2.

To estimate ${\mathrm{I}}^{X,Y}$ from a given dataset, we apply Algorithm 2 to each coordinate pair $(i,j)$. The data are split in a similar fashion to Algorithm 1, but due to the parametric nature of the estimator, we optimize using iterative minibatch-gradient descent. In the training phase, we optimize all DMI models in parallel through the optimization of the corresponding maximum-likelihood loss (15) for a fixed number of epochs. When the training concludes, for each entry, ${\mathrm{I}}_{i,j}^{X,Y}$, we feed the corresponding dataset through the optimized MAFs and estimate the sample covariance matrices, from which we calculate the mutual information term. Neural network optimization is considerably expensive. However, ref. [39] shows that the DMI outperforms existing conditional mutual information estimators in terms of sample requirements. The proposed method boils down to the optimization of $m^2$ MAF models, which may be computationally expensive. We believe that this complexity can be alleviated by incorporating recurrent architectures or attention models. However, this investigation is out of the scope of this work.

Algorithm 2 Neural InfoMat Estimation

input: Data $(x^n,y^n)$, matrix length m, number of epochs N. output: Neural estimate of ${\mathrm{I}}^{X,Y}$

Initialize MAF parameters ${\theta }_{i,j}$ for $(i,j)\in (1,\dots ,m)\times (1,\dots ,m)$ for $(i,j)$ in $(1,\dots ,m)\times (1,\dots ,m)$ do     Divide $(x^n,y^n$) into datasets $${({(x^{i-1},y^{j-1})}_l,{(x^i,y^{j-1})}_l,{(x^{i-1},y^j)}_l,{(x^i,y^j)}_l)}_{l=1}^N$$     Optimize $T_{{\theta }_{i,j}}$ for N epoch via maximum likelihood (15)     Calculate ${\widehat{\mathrm{I}}}_{G,i,j}^{X,Y}$ via (13) on $T_{{\theta }_{i,j}}$ to the sample set. return Estimated InfoMat.

Remark 1

(Performance of normalizing flows). MAFs are a recent promising method for the estimation of (conditional) mutual information using the expressive power of neural network, and were shown to outperform previous methodologies in an array of experiments [39]. However, using the MAF method separately on X and Y yields a pair of Gaussian variables, which are not guaranteed to be jointly Gaussian [48]. Consequently, the estimated mutual information (through the Gaussian formula) is, in general, a lower bound of $I(X;Y)$. Nonetheless, the proposed method showed good empirical performance in considered tasks, as we show in Section 6.

Remark 2

(Computational Complexity and Scalability). Estimating the $m\times m$ InfoMat, where each entry represents a conditional mutual information term, poses significant computational challenges. In the neural estimation approach, each of the $m^2$ entries is estimated via a separate neural network, leading to an overall computational complexity of $O\left(T{m}^2\right)$, where T is the complexity of estimating a single conditional mutual information term. This quickly becomes prohibitive as m increases, both in terms of training time and memory usage. This complexity can potentially be reduced by incorporating weight or state sharing through recurrent architectures, which we aim to explore in future work. The current implementation offers two estimation options: the first is the neural estimator, which provides a more accurate estimation when ample data are available, albeit with increased computational overhead. The second is the Gaussian mutual information Estimator, which relies on covariance matrix estimation and offers a more computationally efficient alternative that performs well in low-data regimes or for moderate values of m.

6. Visualization of Information Transfer

In this section, we demonstrate the utility of the InfoMat as a visualization tool for sequential data. We demonstrate how one can use the relations between information measures and InfoMat entry subsets (Section 3) to deduce temporal interactions in sequential data. We show that the InfoMat provides a more informative mode of information compared to existing measures, and propose measurements that can be coupled with the InfoMat estimate to deduce relationships in the data. Throughout, we adopt the interpretation that higher directed information in a certain direction implies a higher causal effect [36,43]. We analyze the InfoMat through its heatmap representation. Specifically, the heatmap X-axis corresponds to the X-process time index and Y-axis corresponds to the Y-process time index, as per the InfoMat definition (6). An implementation of considered experiments can be found at https://github.com/DorTsur/infomat (accessed on 22 March 2025).

6.1. Synthetic Data—Gaussian Processes

We begin with a sequential Gaussian process, which allows us to clearly present the relations between various dependence structures and the resulting InfoMat structure. Specifically, we consider the following joint Gaussian autoregressive (AR) process

$$\begin{array}{cc}\hfill X_t& =\sum _{k=0}^{{\overline{k}}_x}{\alpha }_k^X{X}_{t-k}+{\alpha }_k^Y{Y}_{i-k}+N_t^X,t\in \mathbb{N}\hfill \\ \hfill Y_t& =\sum _{k=0}^{{\overline{k}}_y}{\beta }_k^X{X}_{t-k}+{\beta }_k^Y{Y}_{t-k}+N_t^Y,\hfill \end{array}$$

where $N_t^Y$ and $N_t^Y$ are samples of a centered i.i.d. Gaussian processes with covariance matrices ${\mathrm{K}}_{N_x}$ and ${\mathrm{K}}_{N_y}$, respectively. By controlling the values for the AR model parameters ${({\overline{k}}_x,{\overline{k}}_y,{\alpha }_k^X,{\alpha }_k^Y,{\beta }_k^X,{\beta }_k^Y)}_{k=1}^m$, we induce various dependence structures on the sequence $(X^m,Y^m)$, which we then visualize via ${\mathrm{I}}^{X,Y}$. All visualizations in this subsection are obtained via Algorithm 1, with $n\approx {10}^5$ samples, which as. We assume that the samples are given from the stationary distribution by omitting the first $max({\overline{k}}_x,{\overline{k}}_y)$ samples.).

We begin with a simple i.i.d. setting by taking ${\beta }_k^X=\gamma$ for $\gamma \in (0,1)$ and nullifying the rest of the parameters. In this case, $X_i\perp \perp Y_j$ for $i\ne j$. Thus, all shared information is instantaneous, implying that we should expect a diagonal InfoMat. As seen in Figure 3a, this is indeed the case. The corresponding InfoMat captures the dependence structure, resulting in a diagonal matrix.

Next, denote ${\overline{x}}_x={\overline{k}}_y=\overline{k}$. We increase the values of AR weights for $\overline{k}>0$, inducing dependence in the history of the joint process. We consider a symmetric dependence structure and set ${\alpha }_k^X={\alpha }_k^Y={\beta }_k^X={\beta }_k^Y=0.3$ for $j\in \{0,\dots ,\overline{k}\}$ considering several values of $\overline{k}$. The value of $\overline{k}$ can be interpreted as how far into the past the dependence of the present terms extends. As shown in Figure 3b,c, the larger $\overline{k}$ is, the bigger the ‘information band’ around the diagonal, whose width depends on the value of $\overline{k}$. Consequently, we may deduce that the farther away we reside on the off-diagonal, the farther in history we observe dependence.

Finally, we demonstrate how more complex temporal structures can be captured via the InfoMat. Specifically, we take two cases in which the ARMA parameters are time-varying, leading to nontrivial temporal relations between the processes. The first considers that ARMA parameters vary over time, i.e., the amount of parameters which are not nullified depends on the $(i,j)$ value. As seen in Figure 4a, this results in a decay in the aforementioned ‘dependence band’. Second, we demonstrate a case of pure unidirectional information transfer in a single direction by introducing a time delay in the parameters and a significant difference between their values. As shown in Figure 4b, the information transfer in dense in the lower triangular, which correspond to the DI term $I(Y^m\to X^m)$ (see Section 3). We conclude that the InfoMat successfully captures nontrivial temporal dependence structures.

6.2. Expressiveness of Neural Estimation

As previously discussed, the efficient InfoMat estimation through Gaussian conditional mutual information estimation comes at the cost of a mismatch when the relations are, e.g., nonlinear. Herein, we demonstrate the utility of neural estimation (Section 5) to the InfoMat capabilities by comparing its performance with the Gaussian mutual information estimator. We take an i.i.d. jointly Gaussian dataset with correlation coefficient $\rho =0.9$. We then apply a cyclic shift of $T<m$ to the samples $Y^m$ within each m-length sequence, effectively resulting in a time-shift dependence structure. We introduce nonlinearity by considering the mapping $X_i\mapsto log\left(X_i\right)$ and $Y_i\mapsto Y_i^3$. Such mappings are invertible and therefore the overall mutual information should be preserved.

As visualized in Figure 5, the neural estimator successfully recovers the correct structure in the data, while the Gaussian mutual information estimator fails to provide a meaningful visualization in the given setting. As the mappings are invertible, the resulting mutual information is $0.83$ [nats] on nonzero entries, which are approximately the corresponding values in the neural estimator of ${\mathrm{I}}^{X,Y}$. However, this accuracy comes at the cost of training $m^2$ neural nets, which is significantly slower than calculating sample covariance matrices. Consequently, we argue that one should consider neural estimation when the functional nature of the data are complex, and consider the Gaussian estimator when m is large, or when simple, initial results may be of need.

6.3. Visualizing Information Flow in Physiological Data

Directed information and transfer entropy have been previously addressed as measures of causal effect between two interacting processes [36,49]. To that end, they have been used to quantify and compare flows of information in stochastic systems. We demonstrate this paradigm with the InfoMat, while showing its applicability to visualize dynamics in real-world datasets. We consider the Apnea dataset from Santa Fe Time Series Competition (https://physionet.org/content/santa-fe/1.0.0/ (accessed on 22 March 2025)) [50,51]. The Apnea dataset is common benchmark for the evaluation of transfer entropy estimation. It is a sequential dataset which consists of measures of heart rate and chest volume (representing respiration force). The Apnea dataset was previously addressed in the literature as a case study to understand the relation between sequential information measures and causal effect. We estimate the InfoMat on the Apnea dataset, denoting the interacting processes at hand being ${\left(X_t\right)}_{t\in \mathbb{N}}=\mathsf{Heart}$ and ${\left(Y\right)}_{t\in \mathbb{N}}=\mathsf{Breath}$. The visualization of the estimated InfoMat is given in Figure 6.

It was shown in [37,49] that the transfer entropy in the direction $\mathsf{Breath}\to \mathsf{Heart}$ is higher than the transfer entropy measures in the other direction. By calculating the directed information in each direction, we recover the same conclusion on the relationship in the data. Specifically, we have

$$\widehat{I}(\mathrm{D}\circ X^n\to Y^n)=0.14<0.41=\widehat{I}(\mathrm{D}\circ Y^n\to X^n),{\widehat{I}}_{inst}(X^n,Y^n)=0.034.$$

This calculation implies that the causal effect in the $\mathsf{Breath}\to \mathsf{Heart}$ direction is the prominent one, which is in agreement with the previous literature. This conclusion provides another validation of the InfoMat method consistency.

Beyond its agreement with known previous results, the InfoMat provides a more informative observation of the exchange of control between the two measures. For example, we observe that most of the information transfer occurs in the first upper subdiagonal, which corresponds to $I(X_{i-1};Y_i|X^{i-2},Y^{i-1})$, i.e., most information is transferred from the previous time step to the next one. In the opposite direction, we have a significantly smaller information transfer. Surprisingly, information is transferred from the steps further in the process past, i.e., the effect is from $Y_{t-4}^{t-2}$ to $X_t$. These results further motivate the use of the InfoMat as a visual tool for the task of exploratory data analysis, we apply it to real-world data.

6.4. Visualizing Coding Schemes Effect

As a final application, we demonstrate an application of the InfoMat to analyze information flow in digital communication schemes. In this case, $X^m$ and $Y^m$ represent the input and output of some communication channel with memory. The joint distribution of $(X^m,Y^m)$ is determined by the channel transition kernel input distribution. The purpose of the encoder, which generates the sequence $X^m$ according to some causally conditional law $P(X^m\parallel Y^{m-1})\triangleq {\prod }_{i=1}^{m}P\left(X_i\right|Y^{i-1}{X}^{i-1})$ [26], is to maximize the downstream directed information. Formally, under mild assumptions, the feedback capacity is characterized by the following optimization problem [7]

$$\mathsf{C}=\underset{n\to \infty }{lim}\underset{P(X^n\parallel Y^{n-1})}{sup}{\displaystyle \frac{1}{n}}I(X^n\to Y^n).$$

(17)

where $\mathsf{C}$ is termed the ‘feedback capacity’ of the communication channel, which is determined by the causal conditional law $P(Y^m\parallel X^m)\triangleq {\prod }_{i=1}^{m}P\left(Y_i\right|Y^{i-1}{X}^i)$. Solving the channel capacity optimization (17) provides the user with the value of maxima achievable rate of reliable communications, coupled with a coding scheme that achieves this rate under block length asymptotics. In what follows, we visualize the InfoMat for several channels, under both a channel oblivious coding scheme and the capacity-achieving coding scheme. As the data for this application are discrete, we utilize a simple plug-in estimator for the considered conditional mutual information terms. For completeness, we provide a characterization and analysis of the plug-in entropy estimator in Appendix A.3.

6.4.1. Ising Channel

As a first example, we consider the Ising channel [52], which is a popular example of a communication channel with memory, which adheres to the famous Ising model. In this case, the channel law is defined according to the relation

$$Y_t=\left\{\begin{array}{c}{X}_t,\mathsf{w}.\mathsf{p}.0.5\hfill \\ X_{t-1},\mathsf{w}.\mathsf{p}.0.5\hfill \end{array}\right.$$

The feedback capacity-achieving coding scheme was obtained in [53] by representing the Ising channel is a finite state channel [7], which allowed for a dynamic programming approximation of the corresponding optimization.

We estimate the InfoMat in the Ising channel under two coding schemes. The first, which we refer to as the oblivious scheme, sends $X^m\stackrel{i.i.d.}{\sim }\mathsf{Ber}(1/2)$ independently of the channel outputs. (Figure 7a). The second generates $X^m$ according to [53] (Figure 7b). We note that the oblivious scheme generates an InfoMat with most of its information in the main diagonal and a small residue in the off-diagonal. The diagonal information follows from the i.i.d. scheme, and it is constant along all time steps. The nonzero off-diagonal entries are due to event $Y_t=X_{t-1}$ which injects memory through the channel transition kernel. When we consider the feedback capacity-achieving scheme from [53], we result with a non-constant pattern of information flow. Specifically, most of the information is sent along the off diagonal, i.e., most of the information is sent through the effect of past channel inputs. Additionally, we note that the amount of conveyed information is non-constant. This is a result of the underlying finite-state machine that defines the evolution of $X^m$ according to past inputs, outputs and states ([53] Figure 5).

Finally, we calculate the normalized directed information for each scheme, as it serves as the proxy for the information rate in the channel. We have

$${\displaystyle \frac{1}{m}}{\widehat{I}}_{\mathsf{i}.\mathsf{i}.\mathsf{d}.}(X^m\to Y^m)\approx 0.45,{\displaystyle \frac{1}{m}}{\widehat{I}}_{\mathsf{opt}}(X^m\to Y^m)\approx 0.56$$

We note that, as expected, the capacity-achieving scheme yields a significantly greater normalized directed information. The resulting quantity is very close to the theoretical capacity values $\left(0.575\right)$. We conjecture that the mismatch results from the plug-in estimation error. Finally, we note that, in contrast to the i.i.d. scheme, the optimal scheme generates information in the direction $Y\to X$ as well, which quantifies the usage of feedback in the scheme.

6.4.2. Trapdoor Channel

Next, we visualize the effect of the coding strategy on the transfer of information in the Trapdoor channel. The trapdoor channel is an example of a binary finite-state channel whose state and output evolve according to the relation

$$Y_t=\left\{\begin{array}{cc}{\mathsf{Z}}_{1/2}\left(X_t\right),\hfill & \mathrm{if}{S}_{t-1}=0\hfill \\ {\mathsf{S}}_{1/2}\left(X_t\right),\hfill & \mathrm{otherwise}\hfill \end{array}\right.,S_t=S_{t-1}\oplus X_t\oplus Y_t,$$

where ${\mathsf{Z}}_{1/2}$ and ${\mathsf{S}}_{1/2}$ denote the Z- and S-channels with a probability of $1/2$. The output of a ${\mathsf{Z}}_p$$\left({\mathsf{S}}_p\right)$ channel equals its input when the latter is 0 (1) and distributes according to $\mathsf{Ber}\left(p\right)$ otherwise. These channels are fundamental and have been extensively investigated in the literature [4,54]. Again, we consider the channel oblivious coding scheme and the optimal coding scheme from [7], as can be seen in Figure 8. We note that the highest amount of conveyed information is in the beginning of transmission. Notably, the information transfer under the optimal Trapdoor scheme is more uniform than the optimal Ising coding scheme. In this case, the estimated normalized directed information is

$${\displaystyle \frac{1}{m}}{\widehat{I}}_{\mathsf{i}.\mathsf{i}.\mathsf{d}.}(X^m\to Y^m)\approx 0.441,{\displaystyle \frac{1}{m}}{\widehat{I}}_{\mathsf{opt}}(X^m\to Y^m)\approx 0.663$$

Again, we note that the optimal coding strategy induces information transfer in the backward direction $Y\to X$ due to the incorporation of feedback into the input distribution.

7. Conclusions

In this work, we developed the InfoMat, a matrix representation of information exchange. We showed the utility of the InfoMat for the visual proofs of information conservation laws via matrix coloring arguments, which allowed us to expand the existing set of decompositions for information measures. Then, we proposed several estimators of the InfoMat, which were studied both theoretically and empirically. Equipped with the InfoMat estimators, we presented several applications of the InfoMat as a visualization tool to analyze the dependence structures and information transfer in sequential datasets in various settings. For future work, we aimed to develop a computationally efficient neural estimator of the InfoMat using weight sharing, sequential architectures [55], and slicing techniques [56]. Given this work’s simplicity and the popularity of information measures, the InfoMat can serve as an effective tool for data exploration in sequential data analysis pipelines. Furthermore, we believe that the InfoMat can be highly useful for a myriad of contemporary research fields that involve time series. Such fields encompass empowerment [14], which characterizes robust sequential decision-making via information theory, and causal inference [16], in which information theory has been shown to be beneficial. Finally, we aim to investigate the multivariate extensions of the InfoMat, as it is central to contemporary setting. This extension can be obtained by constructing higher-dimensional information tensors that capture conditional mutual information among multiple signals. while this extension can uncover rich patterns of inter-dependencies in complex datasets, it also introduces new challenges in computational scalability and visualization clarity.