Rényi Cross-Entropy Measures for Common Distributions and Processes with Memory

Abstract

Two Rényi-type generalizations of the Shannon cross-entropy, the Rényi cross-entropy and the Natural Rényi cross-entropy, were recently used as loss functions for the improved design of deep learning generative adversarial networks. In this work, we derive the Rényi and Natural Rényi differential cross-entropy measures in closed form for a wide class of common continuous distributions belonging to the exponential family, and we tabulate the results for ease of reference. We also summarise the Rényi-type cross-entropy rates between stationary Gaussian processes and between finite-alphabet time-invariant Markov sources.

1. Introduction

The Rényi entropy [1] of order $\alpha$ of a probability mass function p with finite support $\mathbb{S}$ is defined as

$$H_{\alpha }\left(p\right)=\frac{1}{1-\alpha }ln\sum _{x\in \mathbb{S}}p{\left(x\right)}^{\alpha }$$

(1)

for $\alpha >0,\alpha \ne 1$. The Rényi entropy generalizes the Shannon entropy,

$$H\left(p\right)=-\sum _{x\in \mathbb{S}}p\left(x\right)lnp\left(x\right),$$

(2)

in the sense that as $\alpha \to 1$, $H_{\alpha }\left(p\right)\to H\left(p\right)$. Several other Rényi-type information measures have been put forward, each obeying the condition that their limit as $\alpha$ goes to one reduces to a Shannon-type information measure. This includes the Rényi divergence (of order $\alpha$) between two discrete distributions p and q with common finite support $\mathbb{S}$, given by

$$D_{\alpha }\left(p\right|\left|q\right)=\frac{1}{\alpha -1}ln\sum _{x\in \mathbb{S}}p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }$$

(3)

which reduces to the familiar Kullback–Leibler divergence,

$$D_{\mathrm{KL}}\left(p\right|\left|q\right)=\sum _{x\in \mathbb{S}}p\left(x\right)ln\frac{p\left(x\right)}{q\left(x\right)},$$

(4)

as $\alpha \to 1$. Note that in some cases [2], there may exist multiple Rényi-type generalisations for the same information measure (particularly for mutual information).

Many of these definitions admit natural counterparts in the case when the involved distributions have a probability density function (pdf). This gives rise to information measures such as the Rényi differential entropy for pdf p with support $\mathbb{S}$,

$$h_{\alpha }\left(p\right)=\frac{1}{1-\alpha }ln{\int }_{\mathbb{S}}p{\left(x\right)}^{\alpha }dx,$$

(5)

and the Rényi differential divergence between pdfs p and q with common support $\mathbb{S}$,

$$D_{\alpha }(p\parallel q)=\frac{1}{\alpha -1}ln{\int }_{\mathbb{S}}p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }dx.$$

(6)

The Rényi cross-entropy between distributions p and q is an analogous generalization of the Shannon cross-entropy

$$H(p;q)=-\sum _{x\in \mathbb{S}}p\left(x\right)lnq\left(x\right).$$

(7)

Two definitions for this measure have been recently suggested. In light of the fact that Shannon’s cross-entropy satisfies $H(p;q)=D(p\parallel q)+H\left(p\right)$, a natural definition of the Rényi cross-entropy is:

$${\tilde{H}}_{\alpha }(p;q)D_{\alpha }\left(p\right|\left|q\right)+H_{\alpha }\left(p\right).$$

(8)

This definition was indeed proposed in [3] in the continuous case, with the differential cross-entropy measure given by

$${\tilde{h}}_{\alpha }(p;q)D_{\alpha }\left(p\right|\left|q\right)+h_{\alpha }\left(p\right).$$

(9)

In contrast, prior to [3], the authors of [4] introduced the Rényi cross-entropy in their study of shifted Rényi measures expressed as the logarithm of weighted generalized power means. Specifically, upon simplifying Definition 6 in [4], their expression for the Rényi cross-entropy between distributions p and q is given by

$$H_{\alpha }\left(p;q\right)\frac{1}{1-\alpha }ln\sum _{x\in \mathbb{S}}p\left(x\right)q{\left(x\right)}^{\alpha -1}.$$

(10)

For the continuous case, (10) can be readily converted to yield the Rényi differential cross-entropy between pdfs p and q:

$$h_{\alpha }(p;q)\frac{1}{1-\alpha }ln{\int }_{\mathbb{S}}p\left(x\right)q{\left(x\right)}^{\alpha -1}dx.$$

(11)

Note that both (8) and (10) reduce to the Shannon cross-entropy $H(p;q)$ as $\alpha \to 1$ [5]. A similar result holds for (9) and (11), where the Shannon differential cross-entropy,

$$h(p;q)=-{\int }_{\mathbb{S}}p\left(x\right)lnq\left(x\right)dx,$$

(12)

is obtained. Further, the Rényi (differential) entropy is recovered in all equations when $p=q$ (almost everywhere). These properties alone make these definitions viable extensions of the Shannon (differential) cross-entropy.

Finding closed-form expressions for the cross-entropy measure in (9) for continuous distributions is direct, since the Rényi divergence and the Rényi differential entropy were already calculated for numerous distributions in [6,7], respectively. However, deriving the measure in (11) is more involved. We hereafter refer to the measures ${\tilde{H}}_{\alpha }(p;q)$ in (8) and ${\tilde{h}}_{\alpha }(p;q)$ in (9) as the Natural Rényi cross-entropy and the Natural Rényi differential cross-entropy, respectively; while we plainly call the measures $H_{\alpha }(p;q)$ in (10) and $h_{\alpha }(p;q)$ (11) as the Rényi cross-entropy and the Rényi differential cross-entropy, respectively.

In a recent conference paper [8], we showed how to calculate the Rényi differential cross-entropy $h_{\alpha }(p;q)$ between distributions of the same type from the exponential family. Building upon the results shown there, the purpose of this paper is to derive in closed form the expression of $h_{\alpha }(p;q)$ for thirteen commonly used univariate distributions from the exponential family, as well as for multivariate Gaussians, and tabulate the results for ease of reference. We also analytically derive the Natural Rényi differential cross-entropy ${\tilde{h}}_{\alpha }(p;q)$ for the same set of distributions. Finally, we present tables summarising the Rényi and Natural Rényi (differential) cross-entropy rate measures, along with their Shannon counterparts, for two important classes of sources with memory, namely stationary Gaussian sources and finite-state time-invariant Markov sources.

Motivation for determining formulae for the Rényi cross-entropy originates from the use of the Shannon differential cross-entropy as a loss function for the design of deep learning generative adversarial networks (GANs) in [9]. The parameter $\alpha$, ubiquitous to all Rényi information measures, allows one to fine-tune the loss function to improve the quality of the GAN-generated output. This can be seen in [3,5,10], which used the Rényi differential cross-entropy, and the Natural Rényi differential cross-entropy measures, respectively, to generalize the original GAN loss function (which is recovered as $\alpha \to 1$), resulting in both improved GAN system stability and performance for multiple image datasets. It is also shown in [5,10] that the introduced Rényi-centric generalized loss function preserves the equilibrium point satisfied by the original GAN via the so-called Jensen-Rényi divergence [11], a natural extension of the Jensen–Shannon divergence [12] upon which the equilibrium result of [9] is established. Other GAN systems that utilize different generalized loss functions were recently developed and analysed in [13,14,15] (see also the references therein for prior work).

The rest of this paper is organised as follows. In Section 2, the formulae for the Rényi differential cross-entropy and Natural Rényi differential cross-entropy for distributions from the exponential family are given. In Section 3, these calculations are systematically carried out for fourteen pairs of distributions of the same type within the exponential family, and the results are presented in two tables. The Rényi and Natural Rényi differential cross-entropy rates are presented in Section 4 for stationary Gaussian sources; furthermore, the Rényi and Natural Rényi cross-entropy rates are provided in Section 5 for finite-state time-invariant Markov sources. Finally, the paper is concluded in Section 6.

2. Rényi and Natural Rényi Differential Cross-Entropies for Distributions from the Exponential Family

An exponential family is a class of probability distributions over a support $\mathbb{S}\subseteq {\mathbb{R}}^n$ defined by a parameter space $\Theta \subseteq {\mathbb{R}}^m$ and functions $b:\mathbb{S}\mapsto \mathbb{R}$, $c:\Theta \mapsto \mathbb{R}$, $T:\mathbb{S}\mapsto {\mathbb{R}}^m$, and $\eta :\Theta \mapsto {\mathbb{R}}^m$ such that the pdf in this family has the form

$$f\left(x\right)=c\left(\theta \right)b\left(x\right)exp\left(\langle \eta \left(\theta \right),T\left(x\right)\rangle \right),x\in \mathbb{S}$$

(13)

where $\langle ·,·\rangle$ denotes the standard inner product in ${\mathbb{R}}^m$. Alternatively, by using the (natural) parameter $\eta =\eta \left(\theta \right)$, the pdf can also be written

$${\displaystyle f\left(x\right)=b\left(x\right)exp\left(\langle \eta ,T\left(x\right)\rangle +A\left(\eta \right)\right),}$$

(14)

where $A\left(\eta \right):\eta (\Theta )\mapsto \mathbb{R}$ with $A\left(\eta \right)=lnc\left(\theta \right)$. Examples of important pdfs we consider from the exponential family are included in Appendix A.

In [8], the cross-entropy between pdfs $f_1$ and $f_2$ of the same type from the exponential family was proven to be

$$h_{\alpha }\left(f_1;f_2\right)=\frac{A\left({\eta }_1\right)-A\left({\eta }_h\right)+ln{E}_h}{1-\alpha }-A\left({\eta }_2\right),$$

(15)

where

$$E_h={\mathbb{E}}_{f_h}\left[b{\left(X\right)}^{\alpha -1}\right]=\int b{\left(x\right)}^{\alpha -1}{f}_h\left(x\right)dx.$$

(16)

Here, $f_h$ refers to a distribution of the same type as $f_1$ and $f_2$ within the exponential family with natural parameter

$${\eta }_h{\eta }_1+(\alpha -1){\eta }_2.$$

(17)

It can also be shown that the Natural Rényi differential cross-entropy between $f_1$ and $f_2$ is given by

$${\tilde{h}}_{\alpha }\left(f_1;f_2\right)=\frac{A\left({\eta }_{\alpha }\right)-A\left(\alpha {\eta }_1\right)+ln{E}_{\alpha }}{1-\alpha }-A\left({\eta }_2\right),$$

(18)

where

$${\eta }_{\alpha }=\alpha {\eta }_1+(1-\alpha ){\eta }_2,$$

(19)

and

$$E_{\alpha }={\mathbb{E}}_{f_{\alpha 1}}\left[b{\left(X\right)}^{\alpha -1}\right]=\int b{\left(x\right)}^{\alpha -1}{f}_{\alpha 1}\left(x\right)dx$$

(20)

where $f_{\alpha 1}$ refers to a distribution of the same type as $f_1$ and $f_2$ within the exponential family with natural parameter $\alpha {\eta }_1$.

3. Tables of Rényi and Natural Rényi Differential Cross-Entropies

Table 1. Rényi differential cross-entropies.

Name	${\mathit{h}}_{\mathit{\alpha }}({\mathit{f}}_1;{\mathit{f}}_2)$
Beta	${\displaystyle lnB(a_2,b_2)+\frac{1}{\alpha -1}ln\frac{B(a_h,b_h)}{B(a_1,b_1)}}$
	$a_h:=a_1+(\alpha -1)(a_2-1)$,$a_h>0$
	$b_h:=b_1+(\alpha -1)(b_2-1)$,$b_h>0$
$\begin{array}{c}\chi \\ \left(\mathbf{scaled}\right)\end{array}$	${\displaystyle \frac{1}{2}\left(k_{2}ln{\sigma }_2^2{\sigma }_h^2-ln2{\sigma }_h^2\right)+ln\Gamma \left(\frac{k_2}{2}\right)}$
	${\displaystyle +\frac{1}{\alpha -1}\left(ln\Gamma \left(\frac{k_h}{2}\right)-ln\Gamma \left(\frac{k_1}{2}\right)-\frac{k_1}{2}ln{\sigma }_1^2{\sigma }_h^2\right)}$
	${\sigma }_h^2:=\frac{1}{{\sigma }_1^2}+\frac{\alpha -1}{{\sigma }_2^2}$,${\sigma }_h^2>0$
	$k_h:=k_1+(\alpha -1)(k_2-1)$,$k_h>0$
$\begin{array}{c}\chi \\ \left(\mathbf{non-scaled}\right)\end{array}$	${\displaystyle \frac{1}{2}\left(k_{2}ln\alpha -ln2\alpha \right)+ln\Gamma \left(\frac{k_2}{2}\right)}$
	${\displaystyle +\frac{1}{\alpha -1}\left(ln\Gamma \left(\frac{k_h}{2}\right)-ln\Gamma \left(\frac{k_1}{2}\right)-\frac{k_1}{2}ln\alpha \right)}$
	$k_h:=k_1+(\alpha -1)(k_2-1)$,$k_h>0$
${\chi }^{\mathbf{2}}$	${\displaystyle \frac{1}{1-\alpha }\left(\frac{{\nu }_1}{2}ln\left(\alpha \right)-ln\Gamma \left(\frac{{\nu }_1}{2}\right)+ln\Gamma \left(\frac{{\nu }_h}{2}\right)\right)}$
	${\displaystyle +\frac{2-{\nu }_2}{2}ln\left(\alpha \right)+ln2\Gamma \left(\frac{{\nu }_2}{2}\right)}$
	${\nu }_h:={\nu }_1+(\alpha -1)({\nu }_2-2)$,${\nu }_h>0$
Exponential	${\displaystyle \frac{1}{1-\alpha }ln\frac{{\lambda }_1}{{\lambda }_h}-ln{\lambda }_2}$
	${\lambda }_h:={\lambda }_1+(\alpha -1){\lambda }_2$,${\lambda }_h>0$
Gamma	${\displaystyle ln\Gamma \left(k_2\right)+k_{2}ln{\theta }_2}$
	${\displaystyle +\frac{1}{1-\alpha }\left(ln\frac{\Gamma \left(k_h\right)}{\Gamma \left(k_1\right)}-k_{h}ln{\theta }_h-k_{1}ln{\theta }_1\right)}$
	${\theta }_h:=\frac{{\theta }_1+(a-1){\theta }_2}{(\alpha -1){\theta }_1{\theta }_1}$,${\theta }_h>0$
	$k_h:=k_1+(\alpha -1)k_2$,$k_h>0$
$\begin{array}{c}\mathbf{Gaussian}\\ \left(\mathbf{univariate}\right)\end{array}$	${\displaystyle \frac{1}{2}\left(ln\left(2\pi {\sigma }_2^2\right)+\frac{1}{1-\alpha }ln\left(\frac{{\sigma }_2^2}{{\left({\sigma }^2\right)}_h}\right)+\frac{{({\mu }_1-{\mu }_2)}^2}{{\left({\sigma }^2\right)}_h}\right)}$
	${\left({\sigma }^2\right)}_h:={\sigma }_2^2+(\alpha -1){\sigma }_1^2$,${\left({\sigma }^2\right)}_h>0$
$\begin{array}{c}\mathbf{Gaussian}\\ \left(\mathbf{multivariate}\right)\end{array}$	${\displaystyle \frac{1}{2-2\alpha }\left(-ln\left|A\right||{\Sigma }_1|+\left(1-\alpha \right)ln{\left(2\pi \right)}^n\left|{\Sigma }_2\right|-d\right)}$
	$A:={\Sigma }_1^{-1}+(\alpha -1){\Sigma }_2^{-1}$, $A\succ 0$
	${\displaystyle d:={\mu }_{\mathbf{1}}^{\mathbf{T}}{\Sigma }_1^{-1}{\mu }_{\mathbf{1}}+\left(\alpha -1\right){\mu }_{\mathbf{2}}^{\mathbf{T}}{\Sigma }_2^{-1}{\mu }_{\mathbf{2}}}$
	${\displaystyle -({\mu }_{\mathbf{1}}^{\mathbf{T}}{\Sigma }_1^{-1}+\left(\alpha -1\right){\mu }_{\mathbf{2}}^{\mathbf{T}}{\Sigma }_2^{-1})A^{-1}({\Sigma }_1^{-1}{\mu }_{\mathbf{1}}+\left(\alpha -1\right){\Sigma }_2^{-1}{\mu }_{\mathbf{2}})}$
$\begin{array}{c}\mathbf{Gumbel}\\ ({\beta }_{\mathbf{1}}={\beta }_{\mathbf{2}}=\beta )\end{array}$	${\displaystyle \frac{1}{1-\alpha }\left(ln\frac{\Gamma (2-\alpha )}{\beta }-\frac{{\mu }_1}{\beta }-\alpha ln{\eta }_h\right)+\frac{{\mu }_2}{\beta }}$
	${\displaystyle {\eta }_h:=e^{-{\mu }_1/\beta }+(\alpha -1)e^{-{\mu }_2/\beta }}$,${\eta }_h>0$
Half-Normal	${\displaystyle \frac{1}{2}\left(ln\left(\frac{\pi {\sigma }_2^2}{2}\right)+\frac{1}{1-\alpha }ln\left(\frac{{\sigma }_2^2}{{\left({\sigma }^2\right)}_h}\right)\right)}$
	${\left({\sigma }^2\right)}_h:={\sigma }_2^2+(\alpha -1){\sigma }_1^2$,${\left({\sigma }^2\right)}_h>0$
$\begin{array}{c}\mathbf{Laplace}\\ ({\mathbf{\mu }}_{\mathbf{1}}={\mathbf{\mu }}_{\mathbf{2}}=\mathbf{0})\end{array}$	${\displaystyle ln\left(2b_2\right)+\frac{1}{1-\alpha }ln\left(\frac{b_2}{2b_h}\right)}$
	${\displaystyle b_h:=b_2+(1-\alpha )b_1}$,$b_h>0$
$\begin{array}{c}\mathbf{Maxwell}\\ \mathbf{Boltzmann}\end{array}$	${\displaystyle \frac{1}{2}\left(ln2\pi +3ln{\sigma }_2^2\right)+ln{\sigma }_h^2}$
	${\displaystyle +\frac{1}{1-\alpha }\left(ln\frac{\Gamma \left(2\alpha \right)}{\Gamma \left(\alpha \right)}-\frac{3}{2}ln{\sigma }_1^2{\sigma }_h^2\right)}$
	${\sigma }_h^2:={\sigma }_1^{-2}+(\alpha -1){\sigma }_2^{-2}$,${\sigma }_h^2>0$
$\begin{array}{c}\mathbf{Pareto}\\ ({\mathit{m}}_{\mathbf{1}}={\mathit{m}}_{\mathbf{2}}=\mathit{m})\end{array}$	${\displaystyle -lnm-ln{\lambda }_2+\frac{1}{1-\alpha }ln\frac{{\lambda }_1}{{\lambda }_h}}$
	${\lambda }_h:={\lambda }_1+\left(\alpha -1\right)\left({\lambda }_2+1\right)$,${\lambda }_h>0$
Rayleigh	${\displaystyle \frac{ln{\sigma }_1^2-\alpha ln{\sigma }_h^2+ln\Gamma \left(\frac{1-\alpha }{2}\right)}{1-\alpha }+ln2{\sigma }_2^2}$
	${\sigma }_h^2:={\sigma }_1^{-2}+(\alpha -1){\sigma }_2^{-2}$,${\sigma }_h^2>0$

and

Table 2. Natural Rényi differential cross-entropies.

Name	${\tilde{\mathit{h}}}_{\mathit{\alpha }}({\mathit{f}}_1;{\mathit{f}}_2)$
Beta	${\displaystyle lnB(a_2,b_2)+\frac{1}{\alpha -1}ln\frac{B(a_{\alpha },b_{\alpha })}{B\left(\alpha \left(a_1-1\right)+1,\alpha \left(b_1-1\right)+1\right)}}$
	$a_{\alpha }:=\alpha a_1+(1-\alpha )a_2$, $a_{\alpha }>0$
	$b_{\alpha }:=\alpha b_1+(1-\alpha )b_2$, $b_{\alpha }>0$
$\begin{array}{c}\chi \\ \left(\mathbf{scaled}\right)\end{array}$	${\displaystyle \frac{1}{2}\left(-ln\frac{2{\sigma }_1^2}{\alpha }+k_{2}ln{\sigma }_2^2{\sigma }_{\alpha }^2\right)+ln\Gamma \left(\frac{k_2}{2}\right)}$
	${\displaystyle +\frac{1}{1-\alpha }\left(\frac{\alpha k_{1}ln\frac{{\sigma }_{\alpha }^2{\sigma }_1^2}{\alpha }}{2}-ln\Gamma \left(\frac{k_{\alpha }}{2}\right)+ln\Gamma \left(\frac{\alpha (k_1-1)+1}{2}\right)\right)}$
	${\sigma }_{\alpha }^2:=\frac{\alpha }{{\sigma }_1^2}+\frac{1-\alpha }{{\sigma }_2^2}$, ${\sigma }_{\alpha }^2>0$
	$k_{\alpha }:=\alpha k_1+(1-\alpha )k_2$, $k_{\alpha }>0$
$\begin{array}{c}\chi \\ \left(\mathbf{non-scaled}\right)\end{array}$	$\frac{-ln2\alpha }{2}+ln\Gamma \left(\frac{k_2}{2}\right)$
	${\displaystyle +\frac{1}{1-\alpha }\left(-ln\Gamma \left(\frac{k_{\alpha }}{2}\right)-\frac{\alpha k_{1}ln\alpha }{2}+ln\Gamma \left(\frac{\alpha (k_1-1)+1}{2}\right)\right)}$
	$k_{\alpha }:=\alpha k_1+(1-\alpha )k_2$, $k_{\alpha }>0$
${\chi }^{\mathbf{2}}$	${\displaystyle \frac{1}{1-\alpha }\left(-ln\Gamma \left(\frac{{\nu }_{\alpha }}{2}\right)+\alpha ln\Gamma \left(\frac{{\nu }_1}{2}\right)\right)+ln\Gamma \left(\frac{{\nu }_2}{2}\right)}$
	${\nu }_{\alpha }:=\alpha {\nu }_1+(1-\alpha )k$, ${\nu }_{\alpha }>0$
Exponential	${\displaystyle \frac{1}{1-\alpha }ln\frac{{\lambda }_1}{\alpha {\lambda }_{\alpha }}-ln{\lambda }_2}$
	${\lambda }_{\alpha }:=\alpha {\lambda }_1+(1-\alpha ){\lambda }_2$, ${\lambda }_{\alpha }>0$
Gamma	${\displaystyle ln\Gamma \left(k_2\right)+k_{2}ln{\theta }_2}$
	${\displaystyle +\frac{1}{1-\alpha }\left(ln\frac{\Gamma \left(k_1\right)}{\Gamma \left(k_{\alpha }\right)}-k_{\alpha }ln{\theta }_{\alpha }-{\alpha }^2{k}_{1}ln{\theta }_1\right)}$
	${\theta }_{\alpha }:=\alpha {\theta }_1^{-1}+(1-\alpha ){\theta }_2^{-1}$, $k_{\alpha }:=\alpha k_1+(1-\alpha )k_2$, ${\theta }_{\alpha }>0$
$\begin{array}{c}\mathbf{Gaussian}\\ \left(\mathbf{univariate}\right)\end{array}$	${\displaystyle \frac{1}{2}\left(ln\left(2\pi {\sigma }_2^2\right)+\frac{{({\mu }_1-{\mu }_2)}^2}{{\left({\sigma }^2\right)}_{\alpha }}+\frac{1}{1-\alpha }ln\left(\frac{\alpha {\sigma }_2^2}{{\left({\sigma }^2\right)}_{\alpha }}\right)\right)}$
	${\left({\sigma }^2\right)}_{\alpha }:=\alpha {\sigma }_2^2+(1-\alpha ){\sigma }_1^2$, ${\left({\sigma }^2\right)}_{\alpha }>0$
$\begin{array}{c}\mathbf{Gaussian}\\ \left(\mathbf{multivariate}\right)\end{array}$	${\displaystyle \frac{1}{2-2\alpha }\left(-ln\left|\alpha \right|+ln\left|A\right||{\Sigma }_1|+d\right)+\frac{1}{2}ln\frac{{\left(2\pi \right)}^n{\left|{\Sigma }_1\right|}^2}{|{\Sigma }_2|}}$
	$A:=\alpha {\Sigma }_1^{-1}+(1-\alpha ){\Sigma }_2^{-1}$, $A\succ 0$
	${\displaystyle d:={({\mu }_{\mathbf{1}}-{\mu }_{\mathbf{2}})}^T{\Sigma }_{1}A{\Sigma }_2({\mu }_{\mathbf{1}}-{\mu }_{\mathbf{2}})}$
$\begin{array}{c}\mathbf{Gumbel}\\ ({\beta }_{\mathbf{1}}={\beta }_{\mathbf{2}}=\beta )\end{array}$	${\displaystyle \frac{{\mu }_2+\alpha {\mu }_1}{\beta }+\frac{1}{1-\alpha }\left(ln\frac{\Gamma (2-\alpha ){\eta }_{\alpha }}{\alpha \beta }+\frac{{\mu }_1}{\beta }\right)}$
	${\displaystyle {\eta }_{\alpha }:=\alpha e^{-{\mu }_1/\beta }+(1-\alpha )e^{-{\mu }_2/\beta }}$, ${\eta }_{\alpha }>0$
Half-Normal	${\displaystyle \frac{1}{2}\left(ln\left(\frac{\pi {\sigma }_2^2}{2}\right)+\frac{1}{1-\alpha }ln\left(\frac{\alpha {\sigma }_2^2}{{\left({\sigma }^2\right)}_{\alpha }}\right)\right)}$
	${\left({\sigma }^2\right)}_{\alpha }:=\alpha {\sigma }_2^2+(1-\alpha ){\sigma }_1^2$, ${\left({\sigma }^2\right)}_{\alpha }>0$
$\begin{array}{c}\mathbf{Laplace}\\ ({\mathbf{\mu }}_{\mathbf{1}}={\mathbf{\mu }}_{\mathbf{2}}=\mathbf{0})\end{array}$	${\displaystyle \frac{ln{b}_{\alpha }+ln\alpha b_1}{1-\alpha }+ln2b_2}$
	${\displaystyle b_{\alpha }:=\frac{\alpha }{b_1}+\frac{1-\alpha }{b_2}}$, $b_{\alpha }>0$
Maxwell Boltzmann	${\displaystyle \frac{-ln2+3ln{\sigma }_2^2}{2}+ln\frac{\alpha }{{\sigma }_1^2}}$
	${\displaystyle +\frac{1}{1-\alpha }\left(\frac{3}{2}ln\frac{{\sigma }_{\alpha }{\sigma }_1^2}{\alpha }-\alpha ln\frac{\sqrt{\pi }}{2}+ln\Gamma \left(\alpha +\frac{1}{2}\right)\right)}$
	${\sigma }_{\alpha }^2:=\frac{\alpha }{{\sigma }_1^2}+\frac{1-\alpha }{{\sigma }_2^2}$, ${\sigma }_{\alpha }^2>0$
$\begin{array}{c}\mathbf{Pareto}\\ ({\mathit{m}}_{\mathbf{1}}={\mathit{m}}_{\mathbf{2}}=\mathit{m})\end{array}$	${\displaystyle \frac{1}{1-\alpha }\left(ln{\lambda }_{\alpha }-ln\left(1-\alpha ({\lambda }_1-1)\right)\right)-ln{\lambda }_{2}m}$
	${\lambda }_{\alpha }:=\alpha {\lambda }_1+\left(1-\alpha \right){\lambda }_2$, ${\lambda }_{\alpha }>0$
Rayleigh	${\displaystyle \frac{ln{\sigma }_1^2{\left({\sigma }^2\right)}_{\alpha }+ln\alpha +ln\Gamma \left(\frac{1-\alpha }{2}\right)}{1-\alpha }+ln2{\sigma }_1^2}$
	${\left({\sigma }^2\right)}_{\alpha }:=\alpha {\sigma }_1^{-2}+\frac{1}{2}ln\frac{2{\sigma }_1^4{\sigma }_2^4}{\alpha }$, ${\left({\sigma }^2\right)}_{\alpha }>0$

list Rényi and Natural differential cross-entropy expressions, respectively, between common distributions of the same type from the exponential family (which we describe in Appendix A for convenience). The closed-form expressions were derived using (15) and (18), respectively. In the tables, the subscript i is used to denote that a parameter belongs to pdf $f_i$, $i=1,2$.

4. Rényi and Natural Rényi Differential Cross-Entropy Rates for Stationary Gaussian Processes

In [8], the Rényi differential cross-entropy rate between two stationary zero-mean Gaussian processes is derived. We present its formula alongside the expressions of the Shannon and Natural Rényi differential cross-entropy rates in

Table 3. Differential cross-entropy rates for stationary zero-mean Gaussian sources.

Information Measure	Rate	Constraint
$\begin{array}{c}\mathrm{Shannon}\hfill \\ \mathrm{Differential}\hfill \\ \mathrm{Cross-Entropy}\hfill \end{array}$	${\displaystyle \frac{1}{2}ln2\pi +\frac{1}{4\pi }{\int }_0^{2\pi }\left[lng\left(\lambda \right)+\frac{f\left(\lambda \right)}{g\left(\lambda \right)}\right]d\lambda }$	$g\left(\lambda \right)>0$
$\begin{array}{c}\mathrm{Natural}\mathrm{R}\text{é}\mathrm{nyi}\hfill \\ \mathrm{Differential}\hfill \\ \mathrm{Cross-Entropy}\hfill \end{array}$	${\displaystyle \frac{1}{2}ln4{\pi }^2{\alpha }^{\frac{1}{\alpha -1}}+\frac{1}{4\pi (1-\alpha )}{\int }_0^{2\pi }ln\frac{j\left(\lambda \right)}{{g\left(\lambda \right)}^{\alpha }}d\lambda }$	${\displaystyle \frac{j\left(\lambda \right)}{g\left(\lambda \right)}>0}$
$\begin{array}{c}\mathrm{R}\text{é}\mathrm{nyi}\hfill \\ \mathrm{Differential}\hfill \\ \mathrm{Cross-Entropy}\hfill \end{array}$	${\displaystyle \frac{ln2\pi }{2}+\frac{1}{4\pi (1-\alpha )}{\int }_0^{2\pi }\left[(2-\alpha )lng\left(\lambda \right)-lnh\left(\lambda \right)\right]d\lambda }$	${\displaystyle \frac{g\left(\lambda \right)}{h\left(\lambda \right)}>0}$

. Here, $f\left(\lambda \right)$ is the spectral density of the first zero-mean Gaussian process, $g\left(\lambda \right)$ is the spectral density of the second Gaussian process,

$$h\left(\lambda \right)=f\left(\lambda \right)+(\alpha -1)g\left(\lambda \right),$$

(21)

and

$$j\left(\lambda \right)=\alpha f\left(\lambda \right)+(1-\alpha )g\left(\lambda \right).$$

(22)

5. Rényi and Natural Rényi Cross-Entropy Rates for Markov Sources

In [8], the Rényi cross-entropy rate between finite-state time-invariant Markov sources was established, using, as in [16], tools from the theory of non-negative matrices and Perron–Frobenius theory (e.g., cf. [17,18]). This measure, as well as the Shannon and Natural Rényi differential cross-entropy rates, are derived and summarised in

Table 4. Cross-entropy rates for time-invariant Markov sources.

Information Measure	Rate
$\begin{array}{c}\mathrm{Shannon}\hfill \\ \mathrm{Cross-Entropy}\hfill \end{array}$	${\displaystyle -{\pi }_p^T\left(P\odot \dot{ln}Q\right)\mathbf{1}}$
$\begin{array}{c}\mathrm{Natural}\mathrm{R}\text{é}\mathrm{nyi}\hfill \\ \mathrm{Cross-Entropy}\hfill \end{array}$	${\displaystyle \frac{1}{\alpha -1}ln\frac{\lambda (P^{\alpha }\odot Q^{1-\alpha })}{\lambda \left(P^{\alpha }\right)}}$
$\begin{array}{c}\mathrm{R}\text{é}\mathrm{nyi}\hfill \\ \mathrm{Cross-Entropy}\hfill \end{array}$	${\displaystyle \frac{1}{1-\alpha }ln\lambda (P\odot Q^{\alpha -1})}$

. Here, P and Q are the $m\times m$ (stochastic) transition matrices associated with the first and second Markov sources, respectively, where both sources have a common alphabet of size m. To allow any value of the Rényi parameter $\alpha$ in $(0,1)\cup (1,\infty )$, we assume that the transition matrix Q of the second Markov chain has positive entries ($Q>0$); however, the transition matrix P of the first Markov chain is taken to be an arbitrary stochastic matrix. For simplicity, we assume that the initial distribution vectors, p and q, of both Markov chains also have positive entries ($p>0$ and $q>0$). This condition can be relaxed via the approach used to prove Theorem 1 in [16]. Moreover, ${\pi }_p^T$ denotes the stationary probability row vector associated with the first Markov chain, and $\mathbf{1}$ is an m-dimensional column vector in which each element equals one. Furthermore, ⊙ denotes element-wise multiplication (i.e., the Hadamard product operation), and $\dot{ln}$ is the element-wise natural logarithm.

Finally, the definition of $\lambda \left(R\right):{\mathbb{R}}^{m\times m}\mapsto \mathbb{R}$ for a matrix R is more involved. If R is irreducible, $\lambda \left(R\right)$ is its largest positive eigenvalue. Otherwise, rewriting R in its canonical form as detailed in Proposition 1 in [16], we have that $\lambda \left(R\right)=max({\lambda }^{*},{\lambda }_{*})$, where ${\lambda }^{*}$ is the maximum of all largest positive eigenvalues of (irreducible) sub-matrices of R corresponding to self-communicating classes, and ${\lambda }_{*}$ is the maximum of all largest positive eigenvalues of sub-matrices of R corresponding to classes reachable from an inessential class.

6. Conclusions

We have derived closed-form formulae for the Rényi and Natural Rényi differential cross-entropies of commonly used distributions from the exponential family. This is of potential use to further studies in information theory and machine learning, particularly problems where deep neural networks, trained according to a Shannon cross-entropy loss function, can be improved via generalized Rényi-type loss functions in virtue of the extra degree of freedom provided by the Rényi ($\alpha$) parameter. In addition, we have provided formulae for the Rényi and Natural Rényi differential cross-entropy rates for stationary zero-mean Gaussian processes and expressions for the cross-entropy rates for Markov sources. Further work includes expanding the present collection by considering distributions such as Levy or Weibull and investigating cross-entropy measures based on the f-divergence [19,20,21], starting with Arimoto’s divergence [22].