Information Is (Only) Probability ^†

Abstract

Is information a real entity, or is it only a useful mathematical notion? Shannon’s equation and a consolidated tradition have led many scientists and laymen to consider information as some kind of physical, additive quantity. The very well-known argument in favour of information as a physical quantity has not reached any definitive conclusion. Much has been derived from Shannon’s formula, which was introduced to provide a convenient logarithmic measure that might be practically useful and close to most intuitive feelings. Shannon never insisted on the ontological status of information, which might be revealed to be a lot less committing than is often believed. The bottom line is that information might be a convenient mathematical notation to express the probability of independent configurations in a system.

1. Introduction

Here, I will show that there is a mathematical derivation that reduces Shannon’s formula to standard probability theory between independent events. I present an alternative formulation of entropy based on remapping events with different probabilities over multiple equiprobable events. This suggests that, as happens with entropy, there are not more probable configurations, but rather there are many indistinguishable configurations that are mapped onto convenient “more probable” states. Since an alternative formula that does not require any ontological commitment is available, the contention is that information does not exist (its existence does not make any difference). If H can be expressed as the product between the probabilities of independent events, there is no need to add any new entity—information is simply a convenient way to express the probability of different configurations of a physical system.

2. Shannon’s Equation Revisited

Shannon’s equation and a consolidated tradition have led many scientists and laymen to consider information as some kind of physical, additive quantity. The standard formula that all schoolboys learn is the familiar sum of products between probabilities and their logarithms [1]:

$$H\left(X\right)={\displaystyle \sum }_i^N{p}_{i}Lo{g}_2{p}_i$$

(1)

The aim of this paper is to show that the above function (1) can be rewritten in terms of standard probability theory, and that it thus does not entail any ontological commitment. Namely, that it is possible to show that:

$$H\left(X\right)=-\frac{1}{M}Lo{g}_2{\displaystyle \prod }_j^M{\tilde{p}}_j$$

(2)

where ${{\tilde{p}}_j}^{\prime }s$ ($M\ge N)$ maps the probabilities, $p_i$, of the random variable, X, using a method that will be described below. It can be shown that Equations (1) and (2) are mathematically equivalent (in the sense of providing the same set of solutions). Since Equation (2) is a function of a product of probabilities, it follows that H expresses, logarithmically, the probability of independent events, i.e., P(AB) = P(A)P(B). Thus, by Ockham’s principle, if Equation (2) were valid it would drain all ontological commitments of Equation (1). Information as a real quantity could be dismissed.

A few steps show how Equation (2) can be derived from Equation (1). First, Equation (1) can be rewritten as an exponent, and thus the sum can be reformulated as a product with exponents:

$$2^{-H}=2^{{\displaystyle \sum }_i^N{p}_{i}Lo{g}_2{p}_i}={\displaystyle \prod }_i^N{\left(2^{Lo{g}_2{p}_i}\right)}^{p_i}={\displaystyle \prod }_i^N{p}_i{}^{p_i}$$

(3)

Equation (3) begins to take some of the spell away. The next step consists of removing exponents and suggesting suitable independent events. Let d be the greatest common factor among all $p_i$ (assuming $p_i\in \mathbb{Q}, \forall i$, $d\in \mathbb{Q}$, d is the greatest rational, such that $p_i/d=q\in \mathbb{N}, \forall i$). By means of d, Equation (3) can then be rewritten as (Figure 1):

$${\displaystyle \prod }_i^N{p}_i{}^{p_i}=p_1{}^{p_1}\cdot p_2{}^{p_2}\cdot \dots \cdot p_N{}^{p_N}=p_1{}^d\cdot p_1{}^d\cdot \dots \cdot p_N{}^d={\tilde{p}}_1{}^d\cdot {\tilde{p}}_2{}^d\cdot \dots \cdot {\tilde{p}}_M{}^d$$

(4)

whereas each factor, $p_i{}^{p_i}$, is rewritten as a product, ${\displaystyle \prod }_k^{K_i}{\left(p_i\right)}^d$, where $K_i=p_i/d, K_i\ge 1, M=\sum _i^N\frac{p_i}{d}$. Of course, if $d=p_i$, $K_i=1$. Thus, each $p_i$ can be revisited by—and thus substituted with—a series of independent events, ${\tilde{x}}_i, 1\le i\le M$, each having a probability, ${\tilde{p}}_i$. The set of probabilities, ${\tilde{p}}_i$, is built by multiplying each $p_i$ $K_i$ times. Equation (3) can be rewritten as:

$$2^{-H}={\displaystyle \prod }_i^N{p}_i{}^{p_i}={\displaystyle \prod }_j^M{\tilde{p}}_i{}^d={\left[{\displaystyle \prod }_j^M{\tilde{p}}_i\right]}^d$$

(5)

Additionally, then, since $M=\sum _{}^N\left(p_i/d\right)=\sum _{}^N{p}_i/d=1/M$, it is trivial to obtain Equation (2). The derivation from Equation (1) to Equation (2) might appear complicated, but it is not. For instance, if the elements are equiprobable symbols ($p_i=\frac{1}{N}),$ the familiar Hartley’s expression, $H=Lo{g}_{2}N$, will follow [2]. An important caveat: $\prod _{}^N{p}_i$ does not correspond to a probability ($p\left(i\cap j\right)=0, \forall i\ne j$), $\prod _{}^M{\tilde{p}}_i$ does. In fact, $\prod _{}^M{\tilde{p}}_i$ is the probability of any sequence of M events whose frequency matches exactly that of the outcomes of X (Figure 2). In the future, it would be interesting to check whether this proposal might be extended to the continuous case, thereby providing an even more compelling case.

Consider a handful of numerical examples.

First, suppose having two outcomes with probabilities $\left\{\frac{1}{3}, \frac{1}{2}\right\}$, then $d=\frac{1}{6}$, since $\frac{1}{6}$ is their greatest common factor, i.e., d is a rational number by which $\frac{1}{3}/\frac{1}{6}=2$, $\frac{1}{2}/\frac{1}{6}=3$. In fact, ${\left(\frac{1}{3}\right)}^{\frac{1}{3}}\cdot {\left(\frac{1}{3}\right)}^{\frac{1}{3}}$=${\left(\frac{1}{3}\right)}^{\frac{1}{6}}{\left(\frac{1}{3}\right)}^{\frac{1}{6}}\cdot {\left(\frac{1}{2}\right)}^{\frac{1}{6}}{\left(\frac{1}{2}\right)}^{\frac{1}{6}}{\left(\frac{1}{2}\right)}^{\frac{1}{6}}$.

Second, suppose having three outcomes, $x_1, x_2, x_3$, with probabilities $\left\{\frac{1}{6},\frac{1}{3},\frac{1}{2}\right\}$, N = 3. Their greatest common factor common is d = ⅙. By multiplying all elements according to the ratio $K_i=p_i/d$, a set, $\left\{{\tilde{x}}_j\right\}, M=6$, where each element has a probability of $d=\frac{1}{6}=\frac{1}{M}$, we obtain $\left\{p_1, p_2,p_3\right\}\Rightarrow \left\{{\tilde{p}}_1, {\tilde{p}}_2, {\tilde{p}}_3, {\tilde{p}}_4, {\tilde{p}}_5,{\tilde{p}}_6\right\}$. Equation (1) obtains $H=-\left(\frac{1}{6}Lo{g}_2\frac{1}{6}+\frac{1}{3}Lo{g}_2\frac{1}{3}+\frac{1}{2}Lo{g}_2\frac{1}{2}\right)\cong 1.459147917\dots$. Equation (2) obtains $H=-\frac{1}{6}Lo{g}_2\left(\frac{1}{6}\cdot \frac{1}{3}\cdot \frac{1}{3}\cdot \frac{1}{2}\cdot \frac{1}{2}\cdot \frac{1}{2}\right)\cong 1.459147917\dots$

Finally, suppose having two outcomes with probabilities $\left\{\frac{3}{4},\frac{1}{4}\right\}$, N = 2. Then, $d=\frac{1}{4}$, M = 4. $\left\{p_1, p_2\right\}\Rightarrow \left\{{\tilde{p}}_1, {\tilde{p}}_2, {\tilde{p}}_3, {\tilde{p}}_4\right\}$. The probability $\prod _{}^M{\tilde{p}}_i$ represents the probability of any sequence of M outcomes of the kind 0001, 0100, and 1000 (or whatever matches X’s profile).

3. Discussion

Is information something real or not? In many fields, the notion that information is a physical quantity has led scientists to debatable conclusions about both its nature and role [3,4]. Many arguments have been advanced both in favour of and against the reality of information. It is fair to maintain that information cannot be directly measured, as can be done with other, less problematic physical quantities (say, mass and charge). It is impossible to tell how much information is contained in a physical system unless more knowledge about the relation between that physical system and another system is available. It is also disputable whether information has any real causal efficacy, or if it is an epiphenomenal notion, as is centres of mass. The very well-known argument in favour of information as a physical quantity has not reached any definitive conclusion [5]. In this context, much have been derived from Equation (1), which, as Shannon himself stated, was introduced mostly because “the logarithmic measure is more convenient […] it is practically more useful […] it is mathematically more suitable […] it is nearer to our intuitive feelings” [1]. Shannon hardly provides final arguments for the physical existence of information.

An interesting outcome of Equation (2) is that it is achieved by remapping events with different probabilities over multiple equiprobable events. This suggests that, as happens with entropy, there are not more probable configurations, but rather there are many indistinguishable configurations that are mapped onto convenient “more probable” states. Therefore, Equation (2) (and the steps from three to six) is a way to unpack one’s limited knowledge or perspective about physical states and represent them in a neutral way where each state is just as probable.

This is not to say that the notion of information is not a very successful and useful one. Yet, the existence of an alternative formula that does not require any ontological commitment to information makes a strong cause against its existence—the existence of information does not seem to make any difference. Equations (1) and (2) causally overdetermine what a system does, and, since Equation (1) is ontologically less parsimonious, for the Ockham’s razor Equation (2) is to be preferred.

In this contribution, Equation (2) expresses H in terms of the product between the probabilities of independent events. Therefore, if Equation (2) was used in place of Equation (1) (despite being more cumbersome), there would be no need to add any new entity—information is simply a convenient way to express the probability of different configurations of a physical system. Since Equation (2) is mathematically equivalent to Equation (1), and it does not require anything but probabilities, it follows that Equation (2) provides a mathematical proof that information does not exist apart from as a useful conceptual tool.