A Measure of Information Available for Inference

Abstract

The mutual information between the state of a neural network and the state of the external world represents the amount of information stored in the neural network that is associated with the external world. In contrast, the surprise of the sensory input indicates the unpredictability of the current input. In other words, this is a measure of inference ability, and an upper bound of the surprise is known as the variational free energy. According to the free-energy principle (FEP), a neural network continuously minimizes the free energy to perceive the external world. For the survival of animals, inference ability is considered to be more important than simply memorized information. In this study, the free energy is shown to represent the gap between the amount of information stored in the neural network and that available for inference. This concept involves both the FEP and the infomax principle, and will be a useful measure for quantifying the amount of information available for inference.

1. Introduction

Sensory perception comprises complex responses of the brain to sensory inputs. For example, the visual cortex can distinguish objects from their background [1], while the auditory cortex can recognize a certain sound in a noisy place with high sensitivity, a phenomenon known as the cocktail party effect [2,3,4,5,6,7]. The brain (i.e., a neural network) has acquired these perceptual abilities without supervision, which is referred to as unsupervised learning [8,9,10]. Unsupervised learning, or implicit learning, is defined as the learning that happens in the absence of a teacher or supervisor; it is achieved through adaptation to past environments, which is necessary for higher brain functions. An understanding of the physiological mechanisms that mediate unsupervised learning is fundamental to augmenting our knowledge of information processing in the brain.

One of the consequent benefits of unsupervised learning is inference, which is the action of guessing unknown matters based on known facts or certain observations, i.e., the process of drawing conclusions through reasoning and estimation. While inference is thought to be an act of the conscious mind in the ordinary sense of the word, it can occur even in the unconscious mind. Hermann von Helmholtz, a 19th-century physicist/physiologist, realized that perception often requires inference by the unconscious mind and coined the word unconscious inference [11]. According to Helmholtz, conscious inference and unconscious inference can be distinguished based on whether conscious knowledge is involved in the process. For example, when an astronomer computes the positions or distances of stars in space based on images taken at various times from different parts of the orbit of the Earth, he or she performs conscious inference because the process is “based on a conscious knowledge of the laws of optics”; by contrast, “in the ordinary acts of vision, this knowledge of optics is lacking” [11]. Thus, the latter process is performed by the unconscious mind. Unconscious inference is crucial for estimating the overall picture from partial observations.

In the field of theoretical and computational neuroscience, unconscious inference has been translated as the successive inference of the generative process of the external world (in terms of Bayesian inference) that animals perform in order to achieve perception. One hypothesis, the so-called internal model hypothesis [12,13,14,15,16,17,18,19], states that animals reconstruct a model of the external world in their brain through past experiences. This internal model helps animals infer hidden causes and predict future inputs automatically; in other words, this inference process happens unconsciously. This is also known as the predictive coding hypothesis [20,21]. In the past decade, a mathematical foundation for unconscious inference, called the free-energy principle (FEP), has been proposed [13,14,15,16,17], and is a candidate unified theory of higher brain functions. Briefly, this principle hypothesizes that parameters of the generative model are learned through unsupervised learning, while hidden variables are inferred in the subsequent inference step. The FEP provides a unified framework for higher brain functions including perceptual learning [14], reinforcement learning [22], motor learning [23,24], communication [25,26], emotion, mental disorders [27,28], and evolution. However, the difference between the FEP and a related theory, namely the information maximization (infomax) principle, which states that a neural network maximizes the amount of sensory information preserved in the network [29,30,31,32], is still not fully understood.

In this study, the relationship between the FEP and the infomax principle is investigated. As one of the most simple and important examples, the study focuses on blind source separation (BSS), which is the task of separating sensory inputs into hidden sources (or causes) [33,34,35,36]. BSS is shown to be a subset of the inference problem considered in the FEP, and variational free energy is demonstrated to represent the difference between the information stored in the neural network (which is the measure of the infomax principle [29]) and the information available for inferring current sensory inputs.

2. Methods

2.1. Definition of a System

Let us suppose $s\equiv {(s_1,\cdots ,s_N)}^T$ as hidden sources that follow $p(s|\lambda )\equiv {\prod }_{i}p(s_i|\lambda )$ parameterized by a hyper-parameter set $\lambda$; $x\equiv {(x_1,\cdots ,x_M)}^T$ as sensory inputs; $u\equiv {(u_1,\cdots ,u_N)}^T$ as neural outputs; $z\equiv {(z_1,\cdots ,z_M)}^T$ as background noises that follow $p(z|\lambda )$ parameterized by $\lambda$; $ϵ\equiv {({ϵ}_1,\cdots ,{ϵ}_M)}^T$ as reconstruction errors; and $f\in {\mathbb{R}}^M$, $g\in {\mathbb{R}}^N$, and $h\in {\mathbb{R}}^M$ as nonlinear functions (see also

Table 1. Glossary of expressions.

Expression	Description
Generative process	A set of stochastic equations that generate the external world dynamics
Recognition model	A model in the neural network that imitates the inverse of the generative process
Generative model	A model in the neural network that imitates the generative process
$s\in {\mathbb{R}}^N$	Hidden sources
$x\in {\mathbb{R}}^M$	Sensory inputs
$\theta$	A set of parameters
$\lambda$	A set of hyper-parameters
$\vartheta \equiv \{s,\theta ,\lambda \}$	A set of hidden states of the external world
$u\in {\mathbb{R}}^N$	Neural outputs
$W\in {\mathbb{R}}^{N\times M},V\in {\mathbb{R}}^{M\times N}$	Synaptic strength matrices
$\gamma$	State of neuromodulators
$\phi \equiv \{u,W,V,\gamma \}$	A set of the internal states of the neural network
$z\in {\mathbb{R}}^M$	Background noises
$ϵ\in {\mathbb{R}}^M$	Reconstruction errors
$p(x)$	The actual probability density of x
$p(\phi |x),p(x,\phi ),p(\phi )$	Actual probability densities (posterior densities)
$p^{\ast }(u|\gamma ),p^{\ast }(\phi )\equiv p^{\ast }(u|\gamma )p^{\ast }(W,V,\gamma )$	Prior densities
$p^{\ast }(x|\phi )\equiv p^{\ast }(x|u,V,\gamma )$	Likelihood function
$p^{\ast }(x),p^{\ast }(\phi |x),p^{\ast }(x,\phi )$	Statistical models
${\Delta }_x\equiv {\prod }_i{\Delta }_{x_i}$	Finite spatial resolution of x, ${\Delta }_{x_i}>0$
${\langle ·\rangle }_{p(x)}\equiv \int ·p(x)dx$	Expectation of · over $p(x)$
$H\left[x\right]\equiv {\langle -log(p(x){\Delta }_x)\rangle }_{p(x)}$	Shannon entropy of $p(x){\Delta }_x$
${\langle -log(p^{\ast }(x){\Delta }_x)\rangle }_{p(x)}$	Cross entropy of $p^{\ast }(x){\Delta }_x$ over $p(x)$
${\mathcal{D}}_{KL}[p(·)||p^{\ast }(·)]\equiv {〈log\frac{p(·)}{p^{\ast }(·)}〉}_{p(·)}$	KLD between $p(·)$ and $p^{\ast }(·)$
$I[x;\phi ]\equiv {\mathcal{D}}_{KL}[p(x,\phi )||p(x)p(\phi )]$	Mutual information between x and $\phi$
$S(x)\equiv log\frac{p(x)}{p^{\ast }(x)}$	Surprise
$\overline{S}\equiv {\langle S(x)\rangle }_{p(x)}$	Surprise expectation
$F(x)\equiv S(x)+{\mathcal{D}}_{KL}[p(\phi |x)||p^{\ast }(\phi |x)]$	Free energy
$\overline{F}\equiv {\langle F(x)\rangle }_{p(x)}$	Free energy expectation
$X[x;\phi ]\equiv {〈log\frac{p^{\ast }(x,\phi )}{p(x)p(\phi )}〉}_{p(x,\phi )}$	Utilizable information between x and $\phi$

). The generative process of the external world (or the environment) is described by a stochastic equation as:

$$\mathrm{Generative}\mathrm{process}:x=f(s)+z.$$

(1)

Recognition and generative models of the neural network are defined as follows:

$$\mathrm{Recognition}\mathrm{model}:u=g(x),$$

(2)

$$\mathrm{Generative}\mathrm{model}:x=h(u)+ϵ.$$

(3)

Figure 1 illustrates the structure of the system under consideration. For the generative model, the prior distribution of u is defined as $p^{\ast }(u|\gamma )={\prod }_i{p}^{\ast }(u_i|\gamma )$ with a hyper-parameter set $\gamma$ and the likelihood function as $p^{\ast }(x|h(u),\gamma )=\mathcal{N}[x;h(u),{\Sigma }_{ϵ}(\gamma )]$, where $p^{\ast }$ indicates a statistical model and $\mathcal{N}$ is a Gaussian distribution characterized by the mean $h(u)$ and covariance ${\Sigma }_{ϵ}(\gamma )$. Moreover, suppose $\theta$, $W\in {\mathbb{R}}^{N\times M}$, and $V\in {\mathbb{R}}^{M\times N}$ as parameter sets for f, g, and h, respectively, $\lambda$ as a hyper-parameter set for $p(s|\lambda )$ and $p(z|\lambda )$, and $\gamma$ as a hyper-parameter set for $p^{\ast }(u|\gamma )$ and $p^{\ast }(x|h(u),\gamma )$. Here, hyper-parameters are defined as parameters that determine the shape of distributions (e.g., the covariance matrix). Note that W and V are assumed as synaptic strength matrices for feedforward and backward paths, respectively, while $\gamma$ is assumed as a state of neuromodulators similarly to [13,14,15]. In this study, unless specifically mentioned, parameters and hyper-parameters refer to slowly changing variables, so that W, V, and $\gamma$ can change their values. Equations (1)–(3) are transformed into probabilistic representations.

$$\begin{array}{cc}\hfill \mathrm{Generative}\mathrm{process}:p(s,x|\theta ,\lambda )& =p(x|s,\theta ,\lambda )p(s|\lambda )\hfill \\ \hfill & =\int \delta (x-f(s;\theta )-z)p(z|\lambda )p(s|\lambda )dz\hfill \\ \hfill & =p(z=x-f|s,\theta ,\lambda )p(s|\lambda ),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill \mathrm{Recognition}\mathrm{model}:p(x,u|W)& =p(x|u,W)p(u|W)\hfill \\ \hfill & =p(u|x,W)p(x)\hfill \\ \hfill & =p(u-g(x;W)|x,W)p(x),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill \mathrm{Generative}\mathrm{model}:p^{\ast }(x,u|V,\gamma )& =p^{\ast }(x|u,V,\gamma )p^{\ast }(u|\gamma )\hfill \\ \hfill & =\int \delta (x-h(u;V)-ϵ)p^{\ast }(ϵ|V,\gamma )p^{\ast }(u|\gamma )dϵ\hfill \\ \hfill & =p^{\ast }(ϵ=x-h|u,V,\gamma )p^{\ast }(u|\gamma ).\hfill \end{array}$$

(6)

Note that $\delta (·)$ is Dirac’s delta function and $p^{\ast }(x|u,V,\gamma )\equiv p(x|u,V,\gamma ,m)$ is a statistical model given a model structure m. For simplification, let $\vartheta \equiv \{s,\theta ,\lambda \}$ be a set of hidden states of the external world and $\phi \equiv \{u,W,V,\gamma \}$ be a set of internal states of the neural network. By multiplying $p(\theta ,\lambda )$ by Equation (4), $p(W,V,\gamma )$ by Equation (5), and $p^{\ast }(W,V,\gamma )=p^{\ast }(W)p^{\ast }(V)p^{\ast }(\gamma )$ by Equation (6), Equations (4)–(6) become

$$\mathrm{Generative}\mathrm{process}:p(x,\vartheta )=p(x|\vartheta )p(\vartheta )=p(z=x-f|\vartheta )p(\vartheta ),$$

(7)

$$\mathrm{Recognition}\mathrm{model}:p(x,\phi )=p(x|\phi )p(\phi )=p(ϵ=x-h|\phi )p(\phi ),$$

(8)

$$\mathrm{Generative}\mathrm{model}:p^{\ast }(x,\phi )=p^{\ast }(x|\phi )p^{\ast }(\phi )=p^{\ast }(ϵ=x-h|\phi )p^{\ast }(\phi ),$$

(9)

where $p^{\ast }(\phi )=p^{\ast }(u|\gamma )p^{\ast }(W,V,\gamma )$ is the prior distribution for $\phi$ and $p^{\ast }(x,\phi )\equiv p(x,\phi |m)$ is a statistical model given a model structure m, which is determined by the shapes of $p^{\ast }(\phi )$ and $p^{\ast }(x|\phi )\equiv p^{\ast }(x|u,V,\gamma )$. The expression of $p^{\ast }(x,\phi )$ is used instead of $p(x,\phi |m)$ to emphasize the difference between $p(x,\phi )$ and $p^{\ast }(x,\phi )$. While $p(x,\phi )\equiv p(u|x,W)p(W,V,\gamma |x)p(x)$ is the actual joint probability of $(x,\phi )$ (which corresponds to the posterior distribution), $p^{\ast }(x,\phi )$, i.e., the product of the likelihood function and the prior distribution, represents the generative model that the neural network expects $(x,\phi )$ to follow. Typically, elements of $p^{\ast }(W,V,\gamma )$ are supposed to be independent of each other, $p^{\ast }(W,V,\gamma )={\prod }_{i{i}^{\prime }}{p}^{\ast }(W_{i{i}^{\prime }}){\prod }_{j{j}^{\prime }}{p}^{\ast }(V_{j{j}^{\prime }}){\prod }_k{p}^{\ast }({\gamma }_k)$. For example, sparse priors about parameters are sometimes used to prevent the over-learning [37], while a generative model with sparse priors for outputs is known as a sparse coding model [38,39]. As shown later, the inference and learning are achieved by minimizing the difference between $p(x,\phi )$ and $p^{\ast }(x,\phi )$. At that time, minimizing the difference between $p(V,W,\gamma )$ and $p^{\ast }(V,W,\gamma )$ acts as a constraint or a regularizer that prevents over-learning (see Section 2.3 for details).

2.2. Information Stored in the Neural Network

Information is defined as the negative log of probability [40]. When $\mathrm{Prob}(x)$ is the probability of given sensory inputs x, its information is given by $-log\mathrm{Prob}(x)$ [nat], where 1 nat = 1.4427 bits. When x takes continuous values, by coarse graining, $-log\mathrm{Prob}(x)$ is replaced with $-log(p(x){\Delta }_x)$, where $p(x)$ is the probability density of x and ${\Delta }_x\equiv {\prod }_i{\Delta }_{x_i}$ is the product of the finite spatial resolutions of x’s elements (${\Delta }_{x_i}>0$). The expectation of $-log(p(x){\Delta }_x)$ over $p(x)$ gives the Shannon entropy (or average information), which is defined by

$$H\left[x\right]\equiv {〈-log(p(x){\Delta }_x)〉}_{p(x)}\left[\mathrm{nat}\right],$$

(10)

where ${\langle ·\rangle }_{p(x)}\equiv \int ·p(x)dx$ represents the expectation of · over $p(x)$. Note that the use of $-log(p(x){\Delta }_x)$ instead of $-logp(x)$ is useful because this $H\left[x\right]$ is non-negative ($d\mathrm{Prob}(x)=p(x){\Delta }_x$ takes a value between 0 and 1). This is a coarse binning of x and the spatial resolution ${\Delta }_x$ takes a small but nonzero value so that the addition of constant $-log{\Delta }_x$ has no effect except for sliding the offset value. If and only if $p(x)$ is Dirac’s delta function (strictly, $p(x)=1/{\Delta }_x$ at one bin and 0 otherwise), $H\left[x\right]=0$ is realized. For the system under consideration (Equations (7)–(9)), the information shared between the external world states $(x,\vartheta )$ and the internal states of the neural network $\phi$ is defined by mutual information [41]

$$I[(x,\vartheta );\phi ]\equiv {〈log\frac{p(x,\vartheta ,\phi )}{p(x,\vartheta )p(\phi )}〉}_{p(x,\vartheta ,\phi )}\left[\mathrm{nat}\right].$$

(11)

Note that $p(x,\vartheta ,\phi )$ is the joint probability of $(x,\vartheta )$ and $\phi$. Moreover $p(x,\vartheta )$ and $p(\phi )$ are their marginal distributions, respectively. This mutual information takes a non-negative value and quantifies how much $(x,\vartheta )$ and $\phi$ are related with each other. High mutual information indicates the internal states are informative for explaining the external world states, while zero mutual information means they are independent of each other.

However, the only information that the neural network can directly access is the sensory input. This is the case because the system under consideration can be described as a Bayesian network (see [42,43] for details on the Markov blanket). Hence, the entropy of the external world states under a fixed sensory input gives information that the neural network cannot infer. Moreover, there is no feedback control from the neural network to the external world in this setup. Thus, under a fixed x, $\vartheta$ and $\phi$ are conditionally independent of each other. From $p(\vartheta ,\phi |x)=p(\vartheta |x)p(\phi |x)$, we can obtain

$$\begin{array}{c}\hfill I[(x,\vartheta );\phi ]={〈log\frac{p(\vartheta |x)p(\phi |x)p(x)}{p(\vartheta |x)p(x)p(\phi )}〉}_{p(\vartheta |x)p(\phi |x)p(x)}={〈log\frac{p(\phi |x)}{p(\phi )}〉}_{p(\phi ,x)}=I[x;\phi ].\end{array}$$

(12)

Using Shannon entropy, $I[x;\phi ]$ becomes

$$I[x;\phi ]=H\left[x\right]-H[x|\phi \left]\right[\mathrm{nat}],$$

(13)

where

$$H[x|\phi ]\equiv {〈-log\left(p(x|\phi ){\Delta }_x\right)〉}_{p(x,\phi )}$$

(14)

is the conditional entropy of x given $\phi$. Thus, maximization of $I[(x,\vartheta );\phi ]$ is the same as maximization of $I[x;\phi ]$ for this system. As $I[x;\phi ]$, $H\left[x\right]$, and $H[x|\phi ]$ are non-negative, $I[x;\phi ]$ has the range $0\le I[x;\phi ]\le H\left[x\right]$. Zero mutual information occurs if and only if x and $\phi$ are independent, while $I[x;\phi ]=H\left[x\right]$ occurs if and only if x is fully explained by $\phi$. In this manner, $I[x;\phi ]$ describes the information about the external world stored in the neural network. Note that this $I[x;\phi ]$ can be expressed using the Kullback–Leibler divergence (KLD) [44] as $I[x;\phi ]\equiv {\mathcal{D}}_{KL}\left[p(x,\phi )||p(x)p(\phi )\right]$. The KLD takes a non-negative value and indicates the divergence between two distributions.

The infomax principle states that “the network connections develop in such a way as to maximize the amount of information that is preserved when signals are transformed at each processing stage, subject to certain constraints” [29], see also [30,31,32]. According to the infomax principle, the neural network is hypothesized to maximize $I[x;\phi ]$ to perceive the external world. However, $I[x;\phi ]$ does not fully explain the inference capability of a neural network. For example, if neural outputs just express the sensory input itself ($u=x$), $I[x;\phi ]=H\left[x\right]$ is easily achieved, but this does not mean that the neural network can predict or reconstruct input statistics. This is considered in the next section.

2.3. Free-Energy Principle

If one has a statistical model determined by model structure m, the information calculated based on m is given by the negative log likelihood $-logp(x|m)$, which is termed as the surprise (or the marginal likelihood) of the sensory input and expresses the unpredictability of the sensory input for the individual. The neural network is considered to minimize the surprise in the sensory input using the knowledge about the external world, to perceive the external world [13]. To infer if an event is likely to happen based on the past observation, a statistical (i.e., generative) model is necessary; otherwise it is difficult to generalize sensory inputs [45]. Note that the surprise is the marginal over the generative model; hence, the neural network can reduce the surprise by optimizing its internal states, while Shannon entropy of the input is determined by the environment. When the actual probability density and a generative model are given by $p(x)$ and $p^{\ast }(x)\equiv p(x|m)$, respectively, the cross entropy ${\langle -log(p^{\ast }(x){\Delta }_x)\rangle }_{p(x)}$ is always larger than or equal to Shannon entropy $H\left[x\right]$ because of the non-negativity of KLD. Hence, in this study, the input surprise is defined by

$$S(x)\equiv -log{p}^{\ast }(x)+logp(x)\left[\mathrm{nat}\right]$$

(15)

and its expectation over $p(x)$ by

$$\overline{S}\equiv {\langle S(x)\rangle }_{p(x)}={\mathcal{D}}_{KL}[p(x)||p^{\ast }(x)]={〈-log(p^{\ast }(x){\Delta }_x)〉}_{p(x)}-H\left[x\right]\left[\mathrm{nat}\right].$$

(16)

This definition of $S(x)$ is to ensure $\overline{S}$ is non-negative and $\overline{S}=0$ if and only if $p^{\ast }(x)=p(x)$. Since $H\left[x\right]$ is determined by the environment and constant for the neural network, minimization of this $\overline{S}$ is the same meaning as minimization of ${〈-log(p^{\ast }(x){\Delta }_x)〉}_{p(x)}$.

As the sensory input is generated by the external world generative process, consideration of the structure and dynamics placed in the background of the sensory input can provide accurate inference. According to the internal model hypothesis, animals develop the internal model in their brain to increase the accuracy and efficiency of inference [12,13,14,15,17,18,19]; thus, internal states of the neural network $\phi$ are hypothesized to imitate the hidden states of the external world $\vartheta$. A problem is that $-log{p}^{\ast }(x)=-log(\int p^{\ast }(x,\phi )d\phi )$ is intractable for the neural network, because the integral of $p^{\ast }(x,\phi )$ placed in the logarithm function. The FEP hypothesizes that the neural network calculates an upper bound of $-log{p}^{\ast }(x)$ instead of the exact value as a proxy, which is more tractable [13] (because $-logp(x)$ is fixed, the free energy is sometimes defined including or excluding this term). This upper bound is termed as variational free energy:

$$F(x)\equiv S(x)+{\mathcal{D}}_{KL}[p(\phi |x)||p^{\ast }(\phi |x)]={〈-log{p}^{\ast }(x,\phi )+logp(x,\phi )〉}_{p(\phi |x)}\left[\mathrm{nat}\right].$$

(17)

Note that $p(\phi |x)\equiv p(u|x,W)p(W,V,\gamma |x)$ expresses the belief about hidden states of the external world encoded by internal states of the neural network, termed as the recognition density. Due to the non-negativity of KLD, $F(x)$ is guaranteed to be an upper bound of $S(x)$ and $F(x)=S(x)$ holds if and only if $p^{\ast }(\phi |x)=p(\phi |x)$. Furthermore, the expectation of $F(x)$ over $p(x)$ is defined by

$$\begin{array}{cc}\hfill \overline{F}& \equiv {\langle F(x)\rangle }_{p(x)}={\mathcal{D}}_{KL}[p(x,\phi )||p^{\ast }(x,\phi )]\hfill \\ \hfill & ={〈-log(p^{\ast }(x|\phi ){\Delta }_x)〉}_{p(x,\phi )}+{〈-log(p^{\ast }(\phi ){\Delta }_{\phi })〉}_{p(\phi )}-H[\phi |x]-H\left[x\right]\left[\mathrm{nat}\right],\hfill \end{array}$$

(18)

where ${〈-log(p^{\ast }(x|\phi ){\Delta }_x)〉}_{p(x,\phi )}$ is the negative log likelihood and called the accuracy [15]. The second and third terms are the cross entropy of $\phi$ and the conditional entropy of $\phi$ given x, $H[\phi |x]\equiv {\langle -log(p(\phi |x){\Delta }_{\phi })\rangle }_{p(x,\phi )}$, where the difference between them is called the complexity [15]. The last term $H\left[x\right]$ is a constant. $\overline{F}$ indicates the difference between the actual probability $p(x,\phi )$ and the generative model $p^{\ast }(x,\phi )$. Given the non-negativity of KLD, $\overline{F}$ is always larger than or equal to non-negative value $\overline{S}$, and $\overline{F}=\overline{S}=0$ holds if and only if $p^{\ast }(x,\phi )=p(x,\phi )$. The FEP hypothesized that $\overline{F}$ is minimized by optimizing neural activities (u), synaptic strengths (W and V; i.e., synaptic plasticity), and activities of neuromodulators ($\gamma$).

The accuracy ${〈-log(p^{\ast }(x|\phi ){\Delta }_x)〉}_{p(x,\phi )}$ quantifies the amplitude of the reconstruction error. Minimization of the accuracy is the maximum likelihood estimation [10] and provides a solution that (at least locally) minimizes the reconstruction error. Whereas, minimization of the complexity ${〈-log(p^{\ast }(\phi ){\Delta }_{\phi })〉}_{p(\phi )}-H[\phi |x]$ makes $p(\phi )$ closer to $p^{\ast }(\phi )$. As $p^{\ast }(\phi )=p^{\ast }(u|\gamma )p^{\ast }(W,V,\gamma )$ usually supposes the elements of $\phi$ are mutually independent, this acts as the maximization of the entropy under a constraint. Hence, this leads to the increase of the independence between internal states, which helps neurons to establish an efficient representation, as pointed out by Jaynes’ max entropy principle [46,47]. This is essential for BSS [33,34,35,36] because the optimal parameters that minimize the accuracy are not always uniquely determined. Due to this, the maximum likelihood estimation alone does not always identify the generative process behind the sensory inputs. As $\overline{F}$ is the sum of costs for the maximum likelihood estimation and BSS, free-energy minimization is the rule to simultaneously minimize the reconstruction error and maximize the independence of the internal states. It is recognized that animals perform BSS [2,3,4,5,6,7]. Interestingly, even in vitro neural networks perform BSS, which is accompanied by significant reduction of free energy in accordance with the FEP and Jaynes’ max entropy principle [48].

2.4. Information Available for Inference

We now consider how free energy expectation $\overline{F}$ relates to mutual information $I[x;\phi ]$. According to unconscious inference and the internal model hypothesis, the aim of a neural network is to predict x, and for this purpose, it infers hidden states of the external world. While the neural network is conventionally hypothesized to express sufficient statistics of the hidden states of the external world [14], here it is hypothesized that internal states of the neural network are random variables and the probability distribution of them imitates the probability distribution of the hidden states of the external world. The neural network hence attempts to match the joint probability of the sensory inputs and the internal states with that of the sensory inputs and the hidden states of the external world. To do so, the neural network shifts the actual probability of internal states $p(x,\phi )=p(x|\phi )p(\phi )$ closer to those of the generative model $p^{\ast }(x,\phi )=p^{\ast }(x|\phi )p^{\ast }(\phi )$ that the neural network expects $(x,\phi )$ to follow (note that here, $p(x|\phi )=p(x|u,W)$ and $p^{\ast }(x|\phi )=p^{\ast }(x|u,V,\gamma )$). This means that the shape or structure of $p^{\ast }(x,\phi )$ is pre-defined, but the argument $(x,\phi )$ can still change. From this viewpoint, the difference between these two distributions is associated with the loss of information.

The amount of information available for inference can be calculated using the following three values related to information loss: (i) because $H\left[x\right]$ is information of the sensory input and $I[x;\phi ]$ is information stored in the neural network, $H\left[x\right]-I[x;\phi ]=H[x|\phi ]$ indicates the information loss in the recognition model (Figure 2); (ii) the difference between actual and desired (prior) distributions of internal states ${\mathcal{D}}_{KL}[p(\phi )||p^{\ast }(\phi )]$ quantifies the information loss for inferring internal states using the prior (i.e., blind state separation). This is a common approach used in BSS methods [33,34,35,36]; and (iii) the difference between distributions of the actual reconstruction error and the reconstruction error under the given model $\langle {\mathcal{D}}_{KL}[p(x|\phi )||p^{\ast }(x|\phi ){]\rangle }_{p(\phi )}$ quantifies the information loss for representing inputs using internal states. Therefore, by subtracting these three values from $H\left[x\right]$, a mutual-information-like measure representing the inference capability is obtained:

$$\begin{array}{cc}\hfill X[x;\phi ]& \equiv H\left[x\right]-H[x|\phi ]-{\mathcal{D}}_{KL}[p(\phi )||p^{\ast }(\phi )]-\langle {\mathcal{D}}_{KL}[p(x|\phi )||p^{\ast }(x|\phi ){]\rangle }_{p(\phi )}\hfill \\ \hfill & ={〈log\frac{p^{\ast }(x,\phi )}{p(x)p(\phi )}〉}_{p(x,\phi )}\left[\mathrm{nat}\right],\hfill \end{array}$$

(19)

which is called utilizable information in this study. This utilizable information $X[x;\phi ]$ is defined by replacing $p(x,\phi )$ in $I[x;\phi ]$ with $p^{\ast }(x,\phi )$, immediately yielding

$$\overline{F}=I[x;\phi ]-X[x;\phi ]\left[\mathrm{nat}\right].$$

(20)

Hence, $\overline{F}$ represents the gap between the amount of information stored in the neural network and the amount that is available for inference, which is equivalent to the information loss in the generative model. Note that the sum of losses in the recognition and generative models $H\left[x\right]-X[x;\phi ]=\overline{F}+H[x|\phi ]$ is an upper bound of $\overline{F}$ because of the non-negativity of $H[x|\phi ]$ (Figure 2). As $H[x|\phi ]$ is generally nonzero, $F(x)+H[x|\phi ]$ does not usually reach zero, even when $p(x,\phi )=p^{\ast }(x,\phi )$.

Furthermore, $X[x;\phi ]$ is transformed into

$$X[x;\phi ]=H\left[x\right]-L_X-L_A,$$

(21)

where

$$L_X\equiv {〈-log(p^{\ast }(x|\phi ){\Delta }_x)〉}_{p(x,\phi )}$$

(22)

is the so-called reconstruction error, which is similar to the reconstruction error for principal component analysis (PCA) [49], while

$$L_A\equiv {\mathcal{D}}_{KL}[p(\phi )||p^{\ast }(\phi )]$$

(23)

is a generalization of Amari’s cost function for independent component analysis (ICA) [50].

PCA is one of the most popular dimensionality reduction methods. It is used to remove background noise and extract important features from sensory inputs [49,51]. In contrast, ICA is a BSS method used to decompose a mixture set of sensory inputs into independent hidden sources [34,36,50,52,53]. Theoreticians hypothesize that the PCA- and ICA-like learning underlies BSS in the brain [3]. This kind of extraction of the hidden representation is also an important problem in machine learning [54,55]. Equation (21) indicates that $X[x;\phi ]$ consists of PCA- and ICA-like parts, i.e., maximization of $X[x;\phi ]$ can perform both dimensionality reduction and BSS (Figure 2). Their relationship is discussed in the next section.

3. Comparison between the Free-Energy Principle and Related Theories

In this section, the FEP is compared with other theories. As described in the Methods, the aim of the infomax principle is to maximize mutual information $I[x;\phi ]$ (Equation (13)), while the aim of the FEP is to minimize free energy expectation $\overline{F}$ (Equation (18)), while maximization of utilizable information $X[x;\phi ]$ (Equation (19)) means to do both of them simultaneously.

3.1. Infomax Principle

The generative process and the recognition and generative models defined in Equations (1)–(3) are assumed. For the sake of simplicity, let us suppose $W,V$, and $\gamma$ follow Dirac’s delta functions; then, the goal of the infomax principle is simplified to maximization of the mutual information between the sensory inputs x and the neural outputs u:

$$I[x;u|W]={〈log\frac{p(x,u|W)}{p(x)p(u|W)}〉}_{p(x,u,W)}=H\left[x\right]-H[x|u,W]=H[u|W]-H[u|x,W].$$

(24)

Here W, V, and $\gamma$ are still variables, and W is optimized according to the learning while V and $\gamma$ do not directly contribute to minimization of $I[x;u|W]$. For the sake of simplicity, let us suppose $dim(x)\ge dim(u)$ and a linear recognition model $u=g(x)=Wx$, with full-rank matrix W. As $H[u|x,W]=\mathrm{const}.$ is usually assumed and u has an infinite range, $I[x;u|W]=H[u|W]+\mathrm{const}.$ monotonically increases as the variance of u increases. Thus, $I[x;u|W]$ without any constraint is insufficient for deriving learning algorithms for PCA or ICA. To perform PCA and ICA based on the infomax principle, one may consider mutual information between the sensory inputs and the nonlinearly transformed neural outputs $\psi (u)={(\psi (u_1),\cdots ,\psi (u_N))}^T$ with an injective nonlinear function $\psi (·)$. This mutual information is given by:

$$I[x;\psi (u)|W]={〈log\frac{p(x,\psi (u)|W)}{p(x)p(\psi (u)|W)}〉}_{p(x,\psi (u),W)}=H[\psi (u)|W]-H[\psi (u)|x,W].$$

(25)

When nonlinear neural outputs have a finite range (e.g., between 0 and 1), the variance of u should be maintained in the appropriate range. The infomax-based ICA [52,53] is formulated based on this constraint. From $p(\psi (u)|W)=|\partial u/\partial \psi (u)|p(u|W)={({\prod }_i{\psi }^{\prime }(u_i))}^{-1}p(u|W)$, $H[\psi (u)|W]$ becomes $H[\psi (u)|W]={\langle -log\left\{{({\prod }_i{\psi }^{\prime }(u_i))}^{-1}p(u|W){\Delta }_u\right\}\rangle }_{p(u,W)}=H[u|W]+{\langle {\sum }_{i}log{\psi }^{\prime }(u_i)\rangle }_{p(u,W)}$. Since $H[\psi (u)|x,W]=\mathrm{const}.$ holds, Equation (25) becomes:

$$I[x;\psi (u)|W]=H[u|W]+{〈\sum _{i}log{\psi }^{\prime }(u_i)〉}_{p(u,W)}+\mathrm{const}.$$

(26)

This is the cost function that is usually considered in the studies on the infomax-based ICA [52,53]. The following section shows that PCA and ICA are performed by the maximization of Equation (26) as well as the FEP.

3.2. Principal Component Analysis

Both the infomax principle and FEP yield a cost function of PCA. One of the most popular data compression methods, PCA is defined by minimization of the error when the inputs are reconstructed from the compressed representation (i.e., u in this study) [49]. It is known that PCA is derived from the infomax principle under a constraint on the internal states. Although maximization of the mutual information between x and u under the orthonormal constraint on W is usually considered [29], here let us consider another solution. Suppose $dim(x)>dim(u)$, $V=W^T$, and $log{\psi }^{\prime }(u_i)=u_i^2/2+\mathrm{const}$. From Equation (24), $H[u|W]=H[x]-H[x|u,W]+\mathrm{const}.$ holds. Since the reconstruction error is given by $ϵ=x-W^{T}u=(I-W^{T}W)x$ for the linear system under consideration, we obtain $H[x|u,W]=\langle -log\{p(x)|\partial x/\partial ϵ|{\Delta }_x{\}\rangle }_{p(x,\phi )}=H\left[x\right]+\langle log|I-W^{T}W{|\rangle }_{p(\phi )}$. Thus, Equation (26) becomes:

$$I[x;\psi (u)|W]={〈-log|I-W^{T}W|+\frac{1}{2}{|u|}^2〉}_{p(x,\phi )}+\mathrm{const}.$$

(27)

The first term of Equation (27) is maximized if $W{W}^T=I$ holds (i.e., if W is an orthogonal matrix; here, a coarse graining with a finite resolution of W is supposed). To maximize the second term, outputs u need to be involved in a subspace spanned by the first to the N-th major principal components of x. Therefore, maximization of Equation (27) performs PCA.

Further, PCA is also derived by minimization of $L_X$ (Equation (22)), under the assumption that the reconstruction error follows a Gaussian distribution $p^{\ast }(x|\phi )=p^{\ast }(x|u,W,V,\gamma )=\mathcal{N}[x;W^{T}u,{\gamma }^{-1}I]$. Here, $\gamma >0$ is a scalar hyper-parameter that scales the precision of the reconstruction error. Hence, the cost function is given by:

$$L_X={〈\frac{\gamma }{2}{ϵ}^Tϵ-\frac{1}{2}log|\gamma |〉}_{p(\phi )}+\mathrm{const}.$$

(28)

When $\gamma$ is fixed, the derivative of Equation (28) with respect to W gives the update rule for the least mean square error PCA [49]. As this cost function quantifies the magnitude of the reconstruction error, the algorithm that minimizes Equation (28) yields the low-dimensional compressed representation that minimizes the loss incurred in reconstructing the sensory inputs. This algorithm is the same as Oja’s subspace rule [51], up to an additional term that does not essentially change its behavior (see, e.g., [56] for a comparison between them). The $L_X$ here is also in the same form as the cost function for an auto-encoder [54].

Moreover, when the priors of $u,W,V$, and $\gamma$ are flat, ${〈-log{p}^{\ast }(u|W)〉}_{p(u,W)}$ and ${\mathcal{D}}_{KL}[p(W,V,\gamma )||p^{\ast }(W,V,\gamma )]$ are constants with respect to u, W, V, and $\gamma$, because $p(W,V,\gamma )$ is supposed to be a delta function. Hence, the free energy expectation (Equation (18)) becomes $\overline{F}=L_X-H[x|\phi ]-H[u|W]=L_X+\mathrm{const}.$, where $\mathrm{const}.$ is a constant with respect to u, W, and V. In this case, the optimization of W gives the minimum of $\overline{F}$ because u and V are determined by W while $\gamma$ is fixed. Thus, under this condition, $\overline{F}$ is equivalent to the cost function of the least mean square error PCA.

3.3. Independent Component Analysis

It is known that ICA yields independent representation of input data by maximizing the independence between the outputs [52,53]. Thus, ICA reduces the redundancy and yields an efficient representation. When sensory inputs are generated from hidden sources, representing the hidden sources is usually the most efficient representation. Both the infomax principle and FEP yield a cost function of ICA. Let us suppose that sources $s_1,\cdots ,s_N$ independently follow an identical distribution $p_0(s_i|\lambda )$. The infomax-based ICA is derived from Equation (26) [52,53]. If $\psi (u_i)$ is defined to satisfy ${\psi }^{\prime }(u_i)=p_0(u_i|\gamma )$, negative mutual information $-I[x;\psi (u)|W]$ becomes the KLD between the actual and prior distributions up to a constant term,

$$-I[x;\psi (u)|W]+\mathrm{const}.={〈logp(u|W)-log{p}_0(u|\gamma )〉}_{p(\phi )}={〈{\mathcal{D}}_{KL}[p(u|W)||p_0(u|\gamma )]〉}_{p(W,V,\gamma )}=L_A.$$

(29)

The $L_A$ here is known as Amari’s ICA cost function [50], which is a reduction of (23). While both $-I[x;\psi (u)|W]$ and $L_A$ provide the same gradient descent rule, formulating $I[x;\psi (u)|W]$ requires nonlinearly transformed neural outputs $\psi (u)$. By contrast, $L_A$ straightforwardly represents that ICA is performed by minimization of the KLD between $p(u|W)$ and $p^{\ast }(u|\gamma )=p_0(u|\gamma )$. Indeed, if $dim(u)=dim(x)=N$, the background noise is small, and the priors of $W,V$, and $\gamma$ are flat, we obtain $\overline{F}={〈{\mathcal{D}}_{KL}[p(u|W)||p^{\ast }(u|\gamma )]〉}_{p(W,V,\gamma )}=L_A$. Therefore, ICA is a subset of the inference problem considered in the FEP, and the derivation from the FEP is simpler, although both the infomax principle and FEP yield the same ICA algorithm.

Furthermore, when $dim(x)>dim(u)$, minimization of $\overline{F}$ can perform both dimensionality reduction and BSS. When the priors of $W,V$, and $\gamma$ are flat, free energy expectation (Equation (18)) approximately becomes $\overline{F}\approx L_X+L_A+\mathrm{const}.=-X[x;u|W,V,\gamma ]+\mathrm{const}$. Here, $\gamma$ is fixed so that $\mathrm{const}.$ is a constant with respect to $x,u,W$ and V. Conditional entropy $H[x|u,W]$ is ignored in the calculation because it is typically of a smaller order than $L_X$ when $\Sigma (\gamma )$ is not fine-tuned. As $\gamma$ parameterizes the precision of the reconstruction error, it controls the ratio of PCA to ICA. Hence, as $\gamma$ decreases to zero, the solution shifts from a PCA-like to an ICA-like solution.

Unlike the case with the scalar $\gamma$ described above, if ${\Sigma }_{ϵ}(\gamma )$ is fine-tuned by high-dimensional $\gamma$ to minimize $\overline{F}$, ${\Sigma }_{ϵ}={\langle ϵ{ϵ}^T\rangle }_{p(x,\phi )}$ is obtained. Under this condition, $L_X$ is equal to $H[x|u,W]$ up to a constant term, and thereby, $\overline{F}=L_A+\mathrm{const}.$ is obtained. This indicates that $\overline{F}$ consists only of the ICA part. These comparisons suggest that low-dimensional $\gamma$ is better for performing noise reduction than high-dimensional $\gamma$.

4. Simulation and Results

The difference between the infomax principle and the FEP is illustrated by a simple simulation using a linear generative process and a linear neural network (Figure 3). For simplification, it is assumed that u quickly converge to $u=Wx$ compared to the change of s (adiabatic approximation).

For the results shown in Figure 3, s denotes two-dimensional hidden sources following an identical Laplace distribution with zero mean and unit variance; x denotes four-dimensional sensory inputs; u denotes two-dimensional neural outputs; z denotes four-dimensional background Gaussian noises following $\mathcal{N}[z;0,{\Sigma }_z]$; $\theta$ denotes a $4\times 2$-dimensional mixing matrix; W is a $2\times 4$-dimensional synaptic strength matrix for the bottom-up path; and V is a $4\times 2$-dimensional synaptic strength matrix for the top-down path. The priors of $W,V$, and $\gamma$ are supposed to be flat as in Section 3. Sensory inputs are determined by $x=\theta s+z$, while neural outputs are determined by $u=Wx$. The reconstruction error is given by $ϵ=x-Vu$ and used to calculate $H[x|\phi ]$ and $L_A$. Horizontal and vertical axes in the figure are conditional entropy $H[x|\phi ]$ (Equation (14)) and free energy expectation $\overline{F}$ (Equation (18)), respectively. Simulations were conducted 100 times with randomly selected $\theta$ and ${\Sigma }_z$ for each condition. For each simulation, ${10}^8$ random sample points were generated and probability distributions were calculated using the histogram method.

First, when W is randomly chosen and V is defined by $V=W^T$, both $H[x|\phi ]$ and $\overline{F}$ are scattered (black circles in Figure 3) because neural outputs represent random mixtures of sources and noises. Next, when W is optimized according to either Equation (27) or (28) under the constraint of $V=W^T$, the neural outputs express the major principal components of the inputs, i.e., the network performs PCA (blue circles in Figure 3). This is the case when $H[x|\phi ]$ is minimized. In contrast, when $W,V$, and ${\Sigma }_{ϵ}(\gamma )$ are optimized according to the FEP (see Equation (18)), the neural outputs represent the independent components that match the prior source distribution; i.e., the network performs BSS or ICA while reducing the reconstruction error (red circles in Figure 3). For linear generative processes, the minimization of $\overline{F}$ can reliably and accurately perform both dimensionality reduction and BSS because the outputs become independent of each other and match the prior belief if and only if the outputs represent true sources up to permutation and sign-flip. As the utilizable information consists of PCA and ICA cost functions (see Equation (21)), the maximization of $X[x;\phi ]$ leads to a solution that is a compromise between the solutions for the infomax principle and the FEP. Interestingly, the infomax optimization (i.e., PCA) provides a W that makes $\overline{F}$ closer to zero than random states, which indicates that the infomax optimization contributes to the free energy minimization. Note that, for nonlinear systems, there are many different transformations that make the outputs independent of each other [57]. Hence, there is no guarantee that minimization of $\overline{F}$ can identify the true sources of nonlinear generative models.

In summary, the aims of the FEP and infomax principle are similar to each other. In particular, when both the sources and noises follow Gaussian distributions, their aims become the same. Conversely, the optimal synaptic weights under the FEP are different from those under the infomax principle when sources follow non-Gaussian distributions. Under this condition, the maximization of the utilizable information leads to a compromise solution between those for the FEP and the infomax principle.

5. Discussion

In this study, the FEP is linked with the infomax principle, PCA, and ICA. It is more likely that the purpose of a neural network in a biological system is to minimize the surprise of sensory inputs to realize better inference rather than maximize the amount of stored information. For example, the visual input captured by a video camera contributes to the stored information, but this amount of information is not equal to the amount of information available for inference. The surprise expectation represents the difference between actual and inferred observations; the free energy expectation provides the difference between recognition and generative models. Utilizable information is introduced to quantify the inference and generalization capability of sensory inputs. Using this approach, the free energy expectation can be explained as the gap between the information stored in the neural network and that available for inference.

To perform ICA based on the infomax principle, one needs to tune the nonlinearity of the neural outputs to ensure the derivative of the nonlinear I/O function matches the prior distribution. Conversely, under the FEP, ICA is straightforwardly derived from the KLD between the actual probability distribution and the prior distribution of u. Especially, in the absence of background noise and prior knowledge of the parameters and hyper-parameters, the free energy expectation is equivalent to the surprise expectation as well as Amari’s ICA cost function, which indicates that ICA is a subproblem of the FEP.

The variational free energy quantifies the gap between the actual probability and the generative model and is a straightforward extension of the cost functions for BSS in the sense that it comprises the cost function for PCA [49] and ICA [50] in some special cases. Apart from that, there are studies that use the gap between the actual probability and the product of the marginal distributions to perform BSS [58] or to evaluate the information loss [59,60]. While the relationship between the product of the marginal distributions and the generative model is non-trivial, the comparison would lead to a deeper understanding about how the information of the external world is encoded by the neural network. In the subsequent work, we would like to see how the FEP and the infomax principle are related to those approaches.

The FEP is a rigorous and promising theory from theoretical and engineering viewpoints because various learning rules are derived from the FEP [14,15]. However, to be a physiologically plausible theory of the brain, the FEP needs to satisfy certain physiological requirements. There are two major requirements: first, physiological evidence that shows the existence of learning or self-organizing processes under the FEP is required. The model structure under the FEP is consistent with the structure of cortical microcircuits [19]. Moreover, in vitro neural networks performing BSS reduce free energy [48]. It is known that the spontaneous prior activity of a visual area enables it to learn the properties of natural pictures [61]. These results suggest the physiological plausibility of the FEP. Nevertheless, further experiments and consideration of information-theoretical optimization under physiological constraints [62] are required to prove the existence of the FEP in the biological brain. Second, the update rule must be a biologically plausible local learning rule, i.e., synaptic strengths must be changed by signals from connected cells or widespread liquid factors. While the synaptic update rule for a discrete system is local [17], the current rule for a continuous system [14] is a non-local rule. Recently developed biologically-plausible three-factor learning models in which Hebbian learning is mediated by a third modulatory factor [56,63,64,65] may help reveal the neuronal mechanism underlying unconscious inference. Therefore, it is necessary to investigate how actual neural networks infer the dynamics placed in the background of the sensory input and whether this is consistent with the FEP (see also [66] for the relationship between the FEP and spike-timing dependent plasticity [67,68]). This may help develop a biologically plausible learning algorithm through which an actual neural network might develop its internal model. Characterization of information from a physical viewpoint may also help understand how the brain physically embodies the information [69,70]. In the subsequent work, we would like to investigate this relationship.

In summary, this study investigated the differences between two types of information: information stored in the neural network and information available for inference. It was demonstrated that free energy represents the gap between these two types of information. This result clarifies the difference between the FEP and related theories and can be utilized for understanding unconscious inference from a theoretical viewpoint.