On a 2-Relative Entropy

Abstract

We construct a 2-categorical extension of the relative entropy functor of Baez and Fritz, and show that our construction is functorial with respect to vertical morphisms. Moreover, we show such a ‘2-relative entropy’ satisfies natural 2-categorial analogues of convex linearity, vanishing under optimal hypotheses, and lower semicontinuity. While relative entropy is a relative measure of information between probability distributions, we view our construction as a relative measure of information between channels.

1. Introduction

Let X and Y be finite sets which are the input and output alphabets of a discrete memoryless channel $X\stackrel{f}{⇝}Y$ with probability transition matrix $f_{yx}$, representing the probability of the output y given the input x. Every input x then determines a probability distribution on Y which we denote by $f^x$, so that $f^x\left(y\right)=f_{yx}$ for all $x\in X$ and $y\in Y$. The channel $X\stackrel{f}{⇝}Y$ together with the choice of a prior distribution p on X will be denoted $\left(f\right|p)$, and such data then determine a distribution $\vartheta \left(f\right|p)$ on $X\times Y$ given by $\vartheta {\left(f\right|p)}_{(x,y)}=p_x{f}_{yx}$. Given a second channel $X\stackrel{g}{⇝}Y$ with prior distribution q on X, the chain rule for relative entropy says that the relative entropy $D\left(\vartheta \left(f\right|p),\vartheta (g\left|q\right)\right)$ is given by

$$D\left(\vartheta \left(f\right|p),\vartheta (g\left|q\right)\right)=D(p,q)+\sum _{x\in X}{p}_{x}D(f^x,g^x).$$

(1)

As the RHS of (1) involves precisely the datum of the channels f and g together with the prior distributions p and q, we view the quantity $D\left(\vartheta \left(f\right|p),\vartheta (g\left|q\right)\right)$ as a relative measure of information between the channels$\left(f\right|p)$ and $\left(g\right|q)$. In particular, since from a Bayesian perspective $D(p,q)$ may be thought of as the amount of information gained upon discovering that the assumed prior distribution p is actually q, it seems only natural to think of $D\left(\vartheta \left(f\right|p),\vartheta (g\left|q\right)\right)$ as the amount of information gained upon learning that the assumed channel $\left(f\right|p)$ is actually the channel $\left(g\right|q)$.

To make such a Bayesian interpretation more precise, we build upon the work of Baez and Fritz [1], who formulated a type of Bayesian inference as a process $X\to Y$ (including a set of conditional hypotheses on the outcome of the process), which given a prior distribution p on X yields distributions r on Y and q on X in such a way that the relative entropy $D(p,q)$ has an operational meaning as a quantity associated with a Bayesian updating with respect to the process $X\to Y$. (Here, X may be thought of more generally as the set of possible states of some system to be measured, while Y may be thought of as the possible outcomes of the measurement.) Baez and Fritz then proved that up to a constant multiple, the map on such processes given by

$$(X\to Y)\mapsto \mathrm{RE}(X\to Y):=D(p,q)$$

(2)

is the unique map satisfying the following axioms.

• Functoriality: Given a composition of processes $X\to Y\to Z$,

  $$\mathrm{RE}(X\to Y\to Z)=\mathrm{RE}(X\to Y)+\mathrm{RE}(Y\to Z).$$
• Convex Linearity: Given a collection of processes $U^x\to V^x$ indexed by the elements $x\in X$ of a finite probability space $(X,p)$,

  $$\mathrm{RE}\left(\sum _{x\in X}{p}_x(U^x\to V^x)\right)=\sum _{x\in X}{p}_x\mathrm{RE}\left(U^x\to V^x\right).$$
• Vanishing Under Optimal Hypotheses: If the conditional hypotheses associated with a process $X\to Y$ are optimal, then

  $$\mathrm{RE}(X\to Y)=0.$$
• Continuity: The map $(X\to Y)\mapsto \mathrm{RE}(X\to Y)$ is lower semi-continuous.

While Baez and Fritz facilitated their exposition using the language of category theory [2], knowing that a category consists of a class of objects together with a class of composable arrows (i.e., morphisms) between objects is all that is needed for an appreciation of their construction. From such a perspective, the aforementioned processes $X\to Y$ are morphisms in a category $\mathbf{FinStat}$, and the relative entropy assignment given by (2) is then a map from morphisms in $\mathbf{FinStat}$ to $[0,\infty ]$.

In what follows, we elevate the construction of Baez and Fritz to the level of 2-categories (or more precisely, double categories), whose 2-morphisms may be viewed as certain processes between processes, or rather, processes which connect one channel to another. In particular, in Section 2, we formally introduce the category $\mathbf{FinStat}$ introduced by Baez and Fritz, and then, in Section 3, we review their functorial characterization of relative entropy using $\mathbf{FinStat}$. In Section 4, we construct a category ${\mathbf{FinStat}}_{\mathbf{2}}$ which is a 2-level extension of $\mathbf{FinStat}$, and in Section 5, we define a convex structure on ${\mathbf{FinStat}}_{\mathbf{2}}$. In Section 6, we define a relative measure of information between channels which we refer to as conditional relative entropy, and show that it is convex linear and functorial with respect to vertical morphisms in ${\mathbf{FinStat}}_{\mathbf{2}}$. The conditional relative entropy is then used in Section 7 to define a relative entropy assignment ${\mathrm{RE}}_2$ on 2-morphisms via the chain rule as given by (1) (for more on the chain rule for relative entropy one may consult Chapter 2 of [3]). Moreover, we show that such a ‘2-relative entropy’ satisfies the natural 2-level analogues of axioms 1–4 as satisfied by the relative entropy map RE.

As abstract as a relative entropy of processes between processes may seem, Shannon’s Noisy Channel Coding Theorem—which is a cornerstone of information theory—is essentially a statement about transforming a noisy channel into a noiseless one via a sequence of codings and encodings. From such a viewpoint, information theory is fundamentally about processes (i.e., a sequence of codings and encodings), between processes (i.e., channels), and it is precisely this viewpoint with which we will proceed. Furthermore, there is a growing recent interest in axiomatic and categorical approaches to information theory [4,5,6,7,8,9,10,11,12,13], and the present work is a direct outgrowth of such activity.

2. The Category $\mathbf{FinStat}$

In this section, we introduce the first-level structure of interest, which is the category $\mathbf{FinStat}$ introduced by Baez and Fritz [1]. Though we use the language of categories, knowing that a category consists of a class of composable arrows between a class of objects is sufficient for the comprehension of all categorical notions in this work.

Definition 1.

Let X and Y be finite sets. Adiscrete memoryless channel(or simplychannelfor short) $X\stackrel{f}{⇝}Y$ associates every $x\in X$ with a probability distribution $f^x$ on Y. In such a case, the sets X and Y are referred to as theset of inputsandset of outputsof the channel f, respectively, and $f^x\left(y\right)$ is the probability of receiving the output y given the input x, which will be denoted by $f_{yx}$.

Definition 2.

If $X\stackrel{f}{⇝}Y$ and $Y\stackrel{g}{⇝}Z$ are channels, then the composition $X\stackrel{g\circ f}{⇝}Z$ is given by

$$g_{zx}=\sum _{x\in X}{g}_{zy}{f}_{yx}$$

for all $x\in X$ and $y\in Y$.

Remark 1.

If $X\stackrel{f}{⇝}Y$ is a channel such that for every $x\in X$ there exists a $y\in Y$ with $f_{yx}=1$, then such a y is necessarily unique given x, and as such, f may be identified with a function $f:X\to Y$. In such a case, we say that the channel f ispure(ordeterministic), and from here on, we will not distinguish the difference between a pure channel and the associated function from its set of inputs to its set of outputs.

Definition 3.

If ★ denotes a set with a single element, then a channel $★\stackrel{p}{⇝}X$ is simply a probability distribution on X, and, in such a case, we will use $p_x$ to denote the probability of x as given by p for all $x\in X$. The pair $(X,p)$ is then referred to as a finite probability space.

Notation 1.

The datum of a channel $X\stackrel{f}{⇝}Y$ together with a prior distribution $★\stackrel{p}{⇝}X$ on its set of inputs will be denoted $\left(f\right|p)$.

Definition 4.

Let $\mathbf{FinStat}$ denote the category whose objects are finite probability spaces, and whose morphisms $(X,p)⟶(Y,q)$ consist of the following data:

• A function $f:X\to Y$ such that $f\circ p=q$;
• A channel $Y\stackrel{s}{⇝}X$ such that $f\circ s={\mathrm{id}}_Y$. In other words, $Y\stackrel{s}{⇝}X$ is astochastic sectionof $f:X\to Y$.

A morphism in $\mathbf{FinStat}$ is then summarized by a diagram of the form

(3)

and a composition of morphisms in $\mathbf{FinStat}$ is obtained via function composition and composition of stochastic sections. In such a case, it is straightforward to show that a composition of stochastic sections is a stochastic section, etc. The morphism corresponding to diagram (3) will often be denoted $(f,p,s)$.

Remark 2.

Note that in diagram (3), a straight arrow is used for $X\stackrel{f}{⟶}Y$ as f is a function, as opposed to a noisy channel.

Remark 3.

The operational interpretation of diagram (3) is as follows. The set X is thought of as the set of possible states of the system, and $f:X\to Y$ is then thought of as a measurement process, so that Y is then thought of as the set of possible states of some measuring apparatus. The stochastic section $Y\stackrel{s}{⇝}X$ is then thought of as a set of hypotheses about the state of the system given a state of the measuring apparatus. In particular, $s_{xy}$ is thought of as the probability the system was in state x given the state y of the measuring apparatus.

Definition 5.

If the stochastic section $Y\stackrel{s}{⇝}X$ in diagram (3) is such that $s\circ q=p$, then s will be referred to as anoptimal hypothesisfor $\left(f\right|p)$.

Definition 6.

Let $(X,p)$ be a finite probability space, and let $U^x\stackrel{{\mu }^x}{⇝}{V}^x$ be a collection of channels with prior distributions $★\stackrel{q^x}{⇝}{U}^x$ indexed by X. Theconvex combinationof $\left({\mu }^x\right|q^x)$ with respect to $(X,p)$ is the channel

$$\left.\left(\coprod _{x\in X}{U}^x\stackrel{{⨁}_{x\in X}{\mu }^x}{⇝}\coprod _{x\in X}{V}^x\right|\underset{x\in X}{⨁}{p}_x{q}^x\right),$$

where ${⨁}_{x\in X}{\mu }^x$ is the channel given by

$${\left(\underset{x\in X}{⨁}{\mu }^x\right)}_{vu}=\left\{\begin{array}{c}{\mu }_{vu}^x\mathrm{if}(v,u)\in V^x\times U^x\mathrm{for}\mathrm{some}x\in X\hfill \\ 0\mathrm{otherwise}\hfill \end{array}\right.,$$

with prior distribution $★\stackrel{{⨁}_{x\in X}{p}_x{q}^x}{⇝}\coprod U^x$ given by

$${\left(\underset{x\in X}{⨁}{p}_x{q}^x\right)}_u=p_{x_u}{q}_u^{x_u},$$

where $x_u$ is such that $u\in U^{x_u}$. Such a convex combination will be denoted ${⨁}_{x\in X}{p}_x\left({\mu }^x\right|q^x)$.

3. The Baez and Fritz Characterization of Relative Entropy

We now recall the Baez and Fritz characterization of relative entropy in $\mathbf{FinStat}$.

Definition 7.

Let $(X,p)$ be a finite probability space, and let

$$\left(U^x\stackrel{{\mu }^x}{⟶}{V}^x,★\stackrel{q^x}{⇝}{U}^x,V^x\stackrel{s^x}{⇝}{U}^x\right)$$

be a collection of morphisms in $\mathbf{FinStat}$ indexed by X. Theconvex combinationof $({\mu }^x,q^x,s^x)$ with respect to $(X,p)$ is the morphism ${⨁}_{x\in X}{p}_x({\mu }^x,q^x,s^x)$ in $\mathbf{FinStat}$ corresponding to the diagram

(4)

where $r^x={\mu }^x\circ q^x$ for all $x\in X$.

Definition 8.

Let $(f,p,s)$ be a morphism in $\mathbf{FinStat}$, and let $r=s\circ f\circ p$. Therelative entropyof $(f,p,s)$ is the non-negative extended real number $RE(f,p,s)\in [0,\infty ]$ given by

$$RE(f,p,s)=D(p,r),$$

where $D(p,r)={\sum }_x{p}_{x}log(p_x/r_x)$ is the relative entropy between the distributions p and r on X.

Definition 9.

Let $F:\mathbf{FinStat}\to [0,\infty ]$ be a map from the morphisms in $\mathbf{FinStat}$ to the extended non-negative reals $[0,\infty ]$.

• F is said to befunctorialif and only if for every composition $(g\circ f,p,s\circ t)$ of morphisms in $\mathbf{FinStat}$ we have

  $$F(g\circ f,p,s\circ t)=F(f,p,s)+F(g,f\circ p,t).$$
• F is said to beconvex linearif and only if for every convex combination ${⨁}_{x\in X}$$p_x({\mu }^x,q^x,s^x)$ of morphisms in $\mathbf{FinStat}$ we have

  $$F\left(\underset{x\in X}{⨁}{p}_x({\mu }^x,q^x,s^x)\right)=\sum _{x\in X}{p}_{x}F({\mu }^x,q^x,s^x).$$
• F is said to bevanishing under optimal hypothesesif and only if for every morphism $(f,p,s)$ in $\mathbf{FinStat}$ with s an optimal hypothesis we have

  $$F(f,p,s)=0.$$
• F is said to belower semicontinuousif and only if for every sequence of morphisms $(f,p_n,s_n)$ in $\mathbf{FinStat}$ converging to a morphism $(f,p,s)$ we have

  $$F(f,p,s)\le \underset{n\to \infty }{lim\; inf}F(f,p_n,s_n).$$

Theorem 1

(The Baez and Fritz Characterization of Relative Entropy). Let $\mathbb{S}$ be the collection of maps from the morphisms in $\mathbf{FinStat}$ to $[0,\infty ]$ which are functorial, convex linear, vanishing under optimal hypotheses and lower semicontinuous. Then, the following statements hold.

1.

The relative entropy $\mathrm{RE}$ is an element of $\mathbb{S}$;

2.

If $F\in \mathbb{S}$, then $F=c\mathrm{RE}$ for some non-negative constant $c\in \mathbb{R}$.

4. The Category ${\mathbf{FinStat}}_2$

In this section, we introduce the second-level structure of interest, namely, the double category ${\mathbf{FinStat}}_2$, which is a 2-level extension of $\mathbf{FinStat}$.

Definition 10.

Let ${\mathbf{FinStat}}_2$ denote the 2-category whose objects and 1-morphisms coincide with those of $FinStat$, and whose 2-morphisms are constructed as follows. Given 1-morphisms

a 2-morphism $♠:(\mu ,p,s)\Rightarrow (\nu ,q,t)$ consists of channels $X\stackrel{f}{⇝}Y$ and $X^{\prime }\stackrel{f^{\prime }}{⇝}{Y}^{\prime }$ such that

• $f\circ p=q$
• $f^{\prime }\circ p^{\prime }=q^{\prime }$
• $\nu \circ f=f^{\prime }\circ \mu$

The 2-morphism $♠:(\mu ,p,s)\Rightarrow (\nu ,q,t)$ may then be summarized by the following diagram.

(5)

Remark 4.

A crucial point is that in the above diagram, all arrows necessarily commute except for any compositions involving the the outer ‘wings’, s and t. For example, the compositions $s\circ \mu \circ p$ and $t\circ f^{\prime }\circ \mu$ need not be equal to p and f, respectively.

Remark 5.

Diagram (5) should be thought of as a flattened out pyramid, whose base is the inner square and whose vertex is obtained by the identification of the upper and lower stars in the diagram.

Remark 6.

For an operational interpretation of a 2-morphism in ${\mathbf{FinStat}}_2$ as given by diagram (5), one may consider X and Y as sample spaces associated with all possible outcomes of experiments ${\mathcal{E}}_X$ and ${\mathcal{E}}_Y$. As the sets X and Y are endowed with prior distributions p and q, the maps $\mu :X\to X^{\prime }$ and $\nu :Y\to Y^{\prime }$ are then random variables with values in $X^{\prime }$ and $Y^{\prime }$, and the stochastic sections s and t then represent conditional hypotheses about the outcomes of the measurements corresponding to μ and ν. The channels $f:X⇝Y$ and $f:X^{\prime }⇝Y^{\prime }$ then represent stochastic processes such that taking the measurement μ followed by $f^{\prime }$ results in the same process as first letting X evolve according to f and then taking the measurement ν.

Example 1.

For a real-life scenario which realizes a 2-morphism in $\mathbf{FinStat}$, suppose two experimenters Alice and Bob are collaborating on a project to verify predictions of a theory. As such, Alice and her data analyst partner Alicia travel to a mountain in Brazil during a solar eclipse to perform experiments while Bob and and his data-analyst partner Bernie travel to a mountain in Montenegro at the same time for the same purpose. Alice and Bob will then perform experiments in their separate locations and hand their results over to Alicia and Bernie, who will then analyze the data to produce numerical results. At the end of each day, Alice will report her results to Bob over a noisy channel while Alicia will report her results to Bernie over a noisy channel, so that Bob and Bernie may compare their results with Alice and Alicia’s. We then summarize such a scenario with the following 2-morphism in ${\mathbf{FinStat}}_2$:

(6)

In diagram (6), p and q are assumed prior distributions on Alice and Bob’s measurements, while s and t are empirical conditional distributions on Alice and Bob’s measurements given the data outcomes of Alicia and Bernie’s analysis. Moreover, if the communication channel f is less reliable than $f^{\prime }$, then the composition $t\circ f^{\prime }\circ \mu$ provides a Bayesian updating for the channel f.

For vertical composition of 2-morphisms, suppose $♣:({\mu }^{\prime },p^{\prime },s^{\prime })\Rightarrow ({\nu }^{\prime },q^{\prime },t^{\prime })$ is the 2-morphism summarized by the following diagram.

The vertical composition $♣\circ ♠$ is then summarized by the following diagram.

For horizontal composition, let $♡:(\nu ,q,t)\Rightarrow (\xi ,r,u)$ be a 2-morphism summarized by the following diagram.

The horizontal composition $♡\circ ♠$ is then summarized by the following diagram.

5. Convexity in ${\mathbf{FinStat}}_2$

We now generalize the convex structure on morphisms in $\mathbf{FinStat}$ to 2-morphisms in ${\mathbf{FinStat}}_2$. For this, let $(X,p)$ be a finite probability space, and let ${♠}^x$ be a collection of 2-morphisms in ${\mathbf{FinStat}}_2$ indexed by X, where ${♠}^x$ is summarized by the following diagram.

Definition 11.

Theconvex sum${⨁}_{x\in X}{p}_x{♠}^x$ is the 2-morphism in ${\mathbf{FinStat}}_2$ summarized by the following diagram.

6. Conditional Relative Entropy in ${\mathbf{FinStat}}_2$

We now introduce a measure of information associated with 2-morphisms in ${\mathbf{FinStat}}_2$ which we refer to as ‘conditional relative entropy’. The results proved in this section are essentially all lemmas for the results proved in the next section, where we introduce a 2-level extension of the relative entropy map RE, and show that it satisfies the 2-level analogues of the characterizing axioms of relative entropy.

Definition 12.

With every 2-morphism

we associate the non-negative extended real number $\mathrm{CE}(♠)\in [0,\infty ]$ given by

$$\mathrm{CE}(♠)=\sum _{x\in X}{p}_{x}D(f^x,{(t\circ f^{\prime }\circ \mu )}^x),$$

(7)

where $D(-,-)$ is the standard relative entropy. We refer to $\mathrm{CE}(♠)$ as the conditional relative entropy of ♠.

Remark 7.

We refer to $\mathrm{CE}(♠)$ as conditional relative entropy as its defining formula (7) is structurally similar to the defining formula for conditional entropy. In particular, if $X\stackrel{f}{⇝}Y$ is a channel with prior distribution $★\stackrel{p}{⇝}X$, then the conditional entropy $H\left(f\right|p)$ is given by

$$H\left(f\right|p)=\sum _{x\in X}{p}_{x}H\left(f^x\right),$$

where $H\left(f^x\right)$ is the Shannon entropy of the distribution $f^x$ on Y.

Proposition 1.

Conditional relative entropy in ${\mathbf{FinStat}}_2$ is convex linear, i.e., if $(X,p)$ is a finite probability space and ${♠}^x$ is a collection of 2-morphisms in ${\mathbf{FinStat}}_2$ indexed by X, then

$$CE\left(\underset{x\in X}{⨁}{p}_x{♠}^x\right)=\sum _{x\in X}{p}_{x}CE\left({♠}^x\right).$$

Proof. 

Suppose ${⨁}_{x\in X}{p}_x{♠}^x$ is summarized by the following diagram

and let $U={\coprod }_{x\in X}{U}_x$, $\mu ={\oplus }_{x\in X}{\mu }^x$, $f={\oplus }_{x\in X}{f}^x$, $f^{\prime }={\oplus }_{x\in X}{f}^{\prime x}$ and $t={\oplus }_{x\in X}{t}^x$. We then have

$$\begin{array}{ccc}\hfill \mathrm{CE}\left(\underset{x\in X}{⨁}{p}_x{♠}^x\right)& =& \sum _{u\in U}{q}_{u}D(f^u,{(t\circ f^{\prime }\circ \mu )}^u)\hfill \\ & =& \sum _{x\in X}\sum _{u_x\in U_x}{p}_x{q}_{u_x}^{x}D(f^{u_x},{(t\circ f^{\prime }\circ \mu )}^{u_x})\hfill \\ & =& \sum _{x\in X}{p}_x\sum _{u_x\in U_x}{q}_{u_x}^{x}D({\left(f_x\right)}^{u_x},{(t^x\circ f_x^{\prime }\circ {\mu }^x)}^{u_x})\hfill \\ & =& \sum _{x\in X}{p}_x\mathrm{CE}\left({♠}^x\right),\hfill \end{array}$$

as desired. □

Theorem 2.

Conditional relative entropy in ${\mathbf{FinStat}}_2$ is functorial with respect to vertical composition, i.e., if $♣\circ ♠$ is a vertical composition in ${\mathbf{FinStat}}_2$, then $CE(♣\circ ♠)=CE(♣)+CE(♠)$.

Lemma 1.

Let $X\stackrel{f}{⇝}Y\stackrel{g}{⇝}Z$ be a composition of channels.

1.

If f is a pure channel, then ${(g\circ f)}_{zx}=g_{zf\left(x\right)}$;

2.

If g is a stochastic section of a pure channel $Z\stackrel{h}{\to }Y$, then ${(g\circ f)}_{zx}=g_{zh\left(z\right)}{f}_{h\left(z\right)x}$.

Proof. 

The statements 1 and 2 follow immediately from the definitions of pure channel and stochastic section. □

Lemma 2.

Let ♠ be a 2-morphism in ${\mathbf{FinStat}}_2$ as summarized by the diagram

Then,

1.

$CE(♠)={\sum }_{x\in X}{\sum }_{y\in Y}{p}_x{f}_{yx}log\left(\frac{f_{yx}}{t_{y\nu \left(y\right)}{f}_{\nu \left(y\right)\mu \left(x\right)}^{\prime }}\right)$ for all $x\in X$;

2.

$p_{x^{\prime }}^{\prime }{f}_{y^{\prime }{x}^{\prime }}^{\prime }={\sum }_{x\in {\mu }^{-1}\left(x^{\prime }\right)}{\sum }_{y\in {\nu }^{-1}\left(y^{\prime }\right)}{p}_x{f}_{yx}$ for all $(x^{\prime },y^{\prime })\in X^{\prime }\times Y^{\prime }$.

Proof. 

To prove item (1), let $x\in X$ and $y\in Y$. Then,

$$\begin{array}{ccc}\hfill {(t\circ f^{\prime }\circ \mu )}_{yx}& =& {(t\circ (f^{\prime }\circ \mu ))}_{yx}\hfill \\ & =& \sum _{y^{\prime }\in Y^{\prime }}{t}_{y{y}^{\prime }}{(f^{\prime }\circ \mu )}_{y^{\prime }x}\hfill \\ & =& \sum _{y^{\prime }\in Y}{t}_{y{y}^{\prime }}{f}_{y^{\prime }\mu \left(x\right)}^{\prime }\hfill \\ & =& t_{y\nu \left(y\right)}{f}_{\nu \left(y\right)\mu \left(x\right)}^{\prime },\hfill \end{array}$$

where the third equality follows from Lemma 1 since $\mu$ is a pure channel, and the fourth equality follows also from Lemma 1 since t is a stochastic section of a pure channel. We then have

$$\begin{array}{ccc}\hfill \mathrm{CE}(♠)& =& \sum _{x\in X}{p}_{x}D(f^x,{(t\circ f^{\prime }\circ \mu )}^x)\hfill \\ & =& \sum _{x\in X}\sum _{y\in Y}{p}_x{f}_{yx}log\left(\frac{f_{yx}}{{(t\circ f^{\prime }\circ \mu )}_{yx}}\right)\hfill \\ & =& \sum _{x\in X}\sum _{y\in Y}{p}_x{f}_{yx}log\left(\frac{f_{yx}}{t_{y\nu \left(y\right)}{f}_{\nu \left(y\right)\mu \left(x\right)}^{\prime }}\right),\hfill \end{array}$$

as desired.

To prove item (2), the condition $\nu \circ f=f^{\prime }\circ \mu$ is equivalent to the equation ${(\nu \circ f)}_{y^{\prime }x}={(f^{\prime }\circ \mu )}_{y^{\prime }x}$ for all $y^{\prime }\in Y^{\prime }$ and $x\in X$. In addition, since

$${(f^{\prime }\circ \mu )}_{y^{\prime }x}=f_{y^{\prime }\mu \left(x\right)}^{\prime },$$

and

$${(\nu \circ f)}_{y^{\prime }x}=\sum _{y\in Y}{\nu }_{y^{\prime }y}{f}_{yx}=\sum _{y\in {\nu }^{-1}\left(y^{\prime }\right)}{\nu }_{y^{\prime }y}{f}_{yx}=\sum _{y\in {\nu }^{-1}\left(y^{\prime }\right)}{f}_{yx},$$

it follows that $f_{y^{\prime }\mu \left(x\right)}^{\prime }={\sum }_{y\in {\nu }^{-1}\left(y^{\prime }\right)}{f}_{yx}$, thus, for all $x^{\prime }\in X^{\prime }$ and $y^{\prime }\in Y$, it follows that $f_{y^{\prime }{x}^{\prime }}^{\prime }={\sum }_{y\in {\nu }^{-1}\left(y^{\prime }\right)}{f}_{yx}$ for all $x\in {\mu }^{-1}\left(x^{\prime }\right)$. As such, we have

$$p_{x^{\prime }}^{\prime }{f}_{y^{\prime }{x}^{\prime }}^{\prime }=\left(\sum _{x\in {\mu }^{-1}\left(x^{\prime }\right)}{p}_x\right)\left(\sum _{y\in {\nu }^{-1}\left(y^{\prime }\right)}{f}_{yx}\right)=\sum _{x\in {\mu }^{-1}\left(x^{\prime }\right)}\sum _{y\in {\nu }^{-1}\left(y^{\prime }\right)}{p}_x{f}_{yx},$$

as desired. □

Proof of Theorem 2.

Suppose ♠ and ♣ are such that the vertical composition $♣\circ ♠$ is summarized by the following diagram.

By item (1) in Lemma 2, we then have

$$\mathrm{CE}(♣\circ ♠)=\sum _{x\in X}\sum _{y\in Y}{p}_x{f}_{yx}log\left(\frac{f_{yx}}{{(t\circ t^{\prime })}_{y({\nu }^{\prime }\circ \nu )\left(y\right)}{f}_{({\nu }^{\prime }\circ \nu )\left(y\right)({\mu }^{\prime }\circ \mu )\left(x\right)}^{\prime \prime }}\right),$$

and since t is a section of the pure channel $\nu$, by item (2) of Lemma 1, it follows that for every $y\in Y$ we have

$${(t\circ t^{\prime })}_{y({\nu }^{\prime }\circ \nu )\left(y\right)}=t_{y\nu \left(y\right)}{t}_{\nu \left(y\right)({\nu }^{\prime }\circ \nu )\left(y\right)}^{\prime }.$$

As such, we have

$$\begin{array}{ccc}\hfill \mathrm{CE}(♣\circ ♠)& =& \sum _{x\in X}\sum _{y\in Y}{p}_x{f}_{yx}log\left(\frac{f_{yx}}{t_{y\nu \left(y\right)}{t}_{\nu \left(y\right)({\nu }^{\prime }\circ \nu )\left(y\right)}^{\prime }{f}_{({\nu }^{\prime }\circ \nu )\left(y\right)}^{″}}\right)\hfill \\ & =& \sum _{x\in X}\sum _{y\in Y}{p}_x{f}_{yx}log\left(\frac{f_{yx}{f}_{\nu \left(y\right)\mu \left(x\right)}^{\prime }}{t_{y\nu \left(y\right)}{f}_{\nu \left(y\right)\mu \left(x\right)}^{\prime }{t}_{\nu \left(y\right)({\nu }^{\prime }\circ \nu )\left(y\right)}^{\prime }{f}_{({\nu }^{\prime }\circ \nu )\left(y\right)({\mu }^{\prime }\circ \mu )\left(x\right)}^{″}}\right)\hfill \\ & =& \sum _{x\in X}\sum _{y\in Y}{p}_x{f}_{yx}log\left(\frac{f_{yx}}{t_{y\nu \left(y\right)}{f}_{\nu \left(y\right)\mu \left(x\right)}^{\prime }}\right)\hfill \\ & +& \sum _{x\in X}\sum _{y\in Y}{p}_x{f}_{yx}log\left(\frac{f_{\nu \left(y\right)\mu \left(x\right)}^{\prime }}{t_{\nu \left(y\right)({\nu }^{\prime }\circ \nu )\left(y\right)}^{\prime }{f}_{({\nu }^{\prime }\circ \nu )\left(y\right)({\mu }^{\prime }\circ \mu )\left(x\right)}^{″}}\right)\hfill \\ & =& \sum _{x\in X}\sum _{y\in Y}{p}_x{f}_{yx}log\left(\frac{f_{yx}}{t_{y\nu \left(y\right)}{f}_{\nu \left(y\right)\mu \left(x\right)}^{\prime }}\right)\hfill \\ & +& \sum _{x^{\prime }\in X^{\prime }}\sum _{y^{\prime }\in Y^{\prime }}{p}_{x^{\prime }}^{\prime }{f}_{y^{\prime }{x}^{\prime }}^{\prime }log\left(\frac{f_{y^{\prime }{x}^{\prime }}^{\prime }}{t_{y^{\prime }{\nu }^{\prime }\left(y^{\prime }\right)}^{\prime }{f}_{{\nu }^{\prime }\left(y^{\prime }\right){\mu }^{\prime }\left(x^{\prime }\right)}^{″}}\right)(\mathrm{by}\mathrm{item}\left(2\right)\mathrm{of}\mathrm{Lemma} 2)\hfill \\ & =& \sum _{x\in X}{p}_{x}D(f^x,{(t\circ f^{\prime }\circ \mu )}^x)+\sum _{x^{\prime }\in X^{\prime }}{p}_{x^{\prime }}^{\prime }D(f^{\prime x^{\prime }},{(t^{\prime }\circ f^{″}\circ {\mu }^{\prime })}^{x^{\prime }})\hfill \\ & =& \mathrm{CE}(♣)+\mathrm{CE}(♠),\hfill \end{array}$$

as desired (the second-to-last equality follows by item (1) of Lemma 2). □

7. Relative Entropy in ${\mathbf{FinStat}}_2$

In this section, we introduce a 2-level extension of the relative entropy map RE introduced by Baez and Fritz, and show that it satisfies the natural 2-level analogues of functoriality, convex linearity, vanishing under optimal hypotheses, and lower semicontinuity.

Definition 13.

With every 2-morphism

we associate the non-negative extended real number ${\mathrm{RE}}_2(♠)\in [0,\infty ]$ given by

$${\mathrm{RE}}_2(♠)=\mathrm{RE}(\mu ,p,s)+\mathrm{CE}(♠),$$

(8)

which we refer to as the 2-relative entropy of ♠. We note that the quantity $\mathrm{RE}(\mu ,p,s)$ appearing on the RHS of (8) is the relative entropy associated with the morphism $(\mu ,p,s)$ in $\mathbf{FinStat}$, so that

$$\mathrm{RE}(\mu ,p,s)=D(p,s\circ \mu \circ p),$$

where $D(-,-)$ is the standard relative entropy.

Proposition 2.

2-Relative entropy is convex linear, i.e., if $(X,p)$ is a finite probability space and ${♠}^x$ is a collection of 2-morphisms in ${\mathbf{FinStat}}_2$ indexed by X, then

$$R{E}_2\left(\underset{x\in X}{⨁}{p}_x{♠}^x\right)=\sum _{x\in X}{p}_{x}R{E}_2\left({♠}^x\right).$$

Proof. 

Suppose ${⨁}_{x\in X}{p}_x{♠}^x$ is summarized by the following diagram

By Theorem 1, we know that the relative entropy RE is convex linear over 1-morphisms in ${\mathbf{FinStat}}_2$, and by Proposition 1, we know conditional relative entropy is convex linear over 2-morphisms in ${\mathbf{FinStat}}_2$, thus

$$\begin{array}{c}\hfill \mathrm{RE}\left(\underset{x\in X}{⨁}{p}_x({\mu }^x,q^x,s^x)\right)=\sum _{x\in X}{p}_x\mathrm{RE}({\mu }^x,q^x,s^x)\&\mathrm{CE}\left(\underset{x\in X}{⨁}{p}_x{♠}^x\right)=\sum _{x\in X}{p}_x\mathrm{CE}\left({♠}^x\right).\end{array}$$

(9)

We then have

$$\begin{array}{ccc}\hfill {\mathrm{RE}}_2\left(\underset{x\in X}{⨁}{p}_x{♠}^x\right)& =& \mathrm{RE}\left(\underset{x\in X}{⨁}{p}_x({\mu }^x,q^x,s^x)\right)+\mathrm{CE}\left(\underset{x\in X}{⨁}{p}_x{♠}^x\right)\hfill \\ & \stackrel{\left(9\right)}{=}& \sum _{x\in X}{p}_x\mathrm{RE}({\mu }^x,q^x,s^x)+\sum _{x\in X}{p}_x\mathrm{CE}\left({♠}^x\right)\hfill \\ & =& \sum _{x\in X}{p}_x\left(\mathrm{RE}({\mu }^x,q^x,s^x)+\mathrm{CE}\left({♠}^x\right)\right)\hfill \\ & =& \sum _{x\in X}{p}_x{\mathrm{RE}}_2\left({♠}^x\right),\hfill \end{array}$$

as desired. □

Theorem 3.

Relative entropy is functorial with respect to vertical composition, i.e., if $♣\circ ♠$ is a vertical composition in ${\mathbf{FinStat}}_2$, then ${\mathrm{RE}}_2(♣\circ ♠)={\mathrm{RE}}_2(♣)+{\mathrm{RE}}_2(♠)$.

Proof. 

Suppose ♠ and ♣ are such that the vertical composition $♣\circ ♠$ is summarized by the following diagram.

Then,

$$\begin{array}{ccc}\hfill {\mathrm{RE}}_2(♣\circ ♠)& =& \mathrm{RE}({\mu }^{\prime }\circ \mu ,p,s\circ s^{\prime })+\mathrm{CE}(♣\circ ♠)\hfill \\ & =& \mathrm{RE}(\mu ,p,s)+\mathrm{RE}({\mu }^{\prime },p^{\prime },s^{\prime })+\mathrm{CE}(♣)+\mathrm{CE}(♠)\hfill \\ & =& {\mathrm{RE}}_2(♣)+{\mathrm{RE}}_2(♠),\hfill \end{array}$$

where the second equality follows from Theorems 1 and 2. □

Proposition 3.

Let $♠:(\mu ,p,s)\Rightarrow (\nu ,q,t)$ be a 2-morphism in $\mathbf{FinStat}$, and suppose s and t are optimal hypotheses for $\left(\mu \right|p)$ and $\left(\nu \right|q)$ as (defined in Definition 5). Then, ${\mathrm{RE}}_2(♠)=0$.

Proof. 

Since s and t are optimal hypotheses, it follows that $\mathrm{RE}(\mu ,p,s)=\mathrm{CE}(♠)=0$, from which the proposition follows. □

Proposition 4.

The 2-relative entropy ${\mathrm{RE}}_2$ is lower semicontinuous.

Proof. 

Since the 2-relative entropy ${\mathrm{RE}}_2$ is a linear combination of 1-level relative entropies, and 1-level relative entropies are lower semicontinuous by Theorem 1, it follows that ${\mathrm{RE}}_2$ is lower semicontinuous. □

8. Conclusions, Limitations and Future Research

In this work, we have constructed a 2-categorical extension ${\mathrm{RE}}_2$ of the relative entropy functor RE of Baez and Fritz [1], yielding a new measure of information which we view as a relative measure of information between noisy channels. Moreover, we show that our construction satisfies natural 2-level analogues of functoriality, convex linearity, vanishing under optimal hypotheses and lower semicontinuity. As the relative entropy functor of Baez and Fritz is uniquely characterized by such properties, it is only natural to question if our 2-level extension ${\mathrm{RE}}_2$ of RE is also uniquely characterized by the 2-level analogues of such properties. It would also be interesting to investigate alternative versions of 2-morphisms in $\mathbf{FinStat}$ where the 2-morphisms are less restrictive, such as where the base pyramid of a 2-morphism is not assumed to be commutative. While taking the 2-relative entropy associated with such morphisms would not be functorial, such less restrictive morphisms would provide more flexibility for potential applications. Finally, as there are many other categories of interest with respect to information theory [8,14,15], it would be interesting to investigate 2-level extensions of such categories as well.