Comparing the Zeta Distributions with the Pareto Distributions from the Viewpoint of Information Theory and Information Geometry: Discrete versus Continuous Exponential Families of Power Laws ^†

Abstract

We consider the zeta distributions, which are discrete power law distributions that can be interpreted as the counterparts of the continuous Pareto distributions with a unit scale. The family of zeta distributions forms a discrete exponential family with normalizing constants expressed using the Riemann zeta function. We present several information-theoretic measures between zeta distributions, study their underlying information geometry, and compare the results with their continuous counterparts, the Pareto distributions.

1. Introduction

Zeta distributions [1,2] are parametric discrete distributions with probability mass functions indexed by a scalar parameter $s\in (1,\infty )$ whose support is the set of positive integers $\mathbb{N}$:

$$p_s\left(x\right)=Pr[X=x]\propto \frac{1}{x^s},x\in \mathcal{X}=\mathbb{N}=\{1,2,\dots \}.$$

(1)

The normalization function $\zeta \left(s\right)$ of the zeta distributions $p_s\left(x\right)=\frac{1}{\zeta \left(s\right)}\frac{1}{x^s}$ such that ${\sum }_{x\in \mathbb{N}}{p}_s\left(x\right)=1$ is the real Riemann zeta function [3,4,5]:

$$\zeta \left(s\right)=\sum _{i=1}^{\infty }\frac{1}{i^s}=1+\frac{1}{2^s}+\frac{1}{3^s}+\dots ,s>1.$$

(2)

The set of zeta distributions $\mathcal{Z}=\{p_s\left(x\right):s\in (1,\infty )\}$ forms a discrete exponential family [6,7] with natural parameter $\theta \left(s\right)=s$ lying in the natural parameter space $\mathsf{\Theta }=(1,\infty )$, the sufficient statistic $t\left(x\right)=-logx$, and the cumulant function or log-normalizer $F\left(\theta \right)=log\zeta \left(\theta \right)$. Therefore, it follows from the theory of exponential families [7] that $log\zeta \left(\theta \right)$ is a strictly convex and real analytic function (see Figure 1). Thus, the pmf of zeta distributions can be rewritten in the canonical form of exponential families as:

$$p_s\left(x\right)=exp\left(\theta \left(s\right)t\left(x\right)-F\left(\theta \right(s\left)\right)\right).$$

(3)

The characteristic function is thus ${\varphi }_s\left(t\right)=\frac{\zeta (s+it)}{\zeta \left(s\right)}$.

Thus, a zeta distribution $p_s\left(x\right)$ can be interpreted as the discrete equivalent of a Pareto distribution $q_s\left(x\right)$ of scale 1 and shape $s-1$ with probability density function $q_s\left(x\right)=\frac{s-1}{x^s}$ for $x>1$ (see

Table 1. Comparisons between the Zeta family and the Pareto subfamily. The function $\zeta \left(s\right)$ is the real zeta function.

	Zeta Distribution	Pareto Distribution
Univariate Uni-Order Exponential Family $exp\left(\mathit{\theta }\mathit{t}\right(\mathit{x})-\mathit{F}(\mathit{\theta }\left)\right)$
	Discrete EF	Continuous EF
PMF/PDF	$p_s\left(x\right)=\frac{1}{x^s\zeta \left(s\right)}$	$q_s\left(x\right)=\frac{s-1}{x^s}$
Support $\mathcal{X}$	$\mathbb{N}=\{1,2,\dots \}$	$(1,\infty )$
Natural parameter $\theta$	s	s
Cumulant $F\left(\theta \right)$	$log\zeta \left(\theta \right)$	$-log(\theta -1)$
Sufficient statistic $t\left(x\right)$	$-logx$	$-logx$
Moment parameter $\eta$	$\frac{{\zeta }^{\prime }\left(\theta \right)}{\zeta \left(\theta \right)}$	$-\frac{1}{s-1}$
Conjugate $F^{*}\left(\eta \right)$	$-H\left[p_s\right]=-{\sum }_{i=1}^{\infty }\frac{1}{i^s\zeta \left(s\right)}log\left(i^s\zeta \left(s\right)\right)$	$\eta -1-log(-\eta )$
Maximum likelihood estimator	$\widehat{\eta }=\frac{{\zeta }^{\prime }\left(\widehat{\theta }\right)}{\zeta \left(\widehat{\theta }\right)}=-\frac{1}{n}{\sum }_{i=1}^{n}log{x}_i$	$\widehat{s}=\frac{n}{{\sum }_{i=1}^{n}log{x}_i}$
Fisher information	${\sum }_{i=0}^{\infty }\mathsf{\Lambda }\left(i\right)log\left(i\right)i^{-s}$	$\frac{1}{{(s-1)}^2}$
Entropy $-F^{*}\left(\eta \left(s\right)\right)$	${\sum }_{i=1}^{\infty }\frac{1}{i^s\zeta \left(s\right)}log\left(i^s\zeta \left(s\right)\right)$	$1+\frac{1}{s-1}-log(s-1)$
Bhattacharyya coefficient $I_{\alpha }$	$\frac{\zeta (\alpha s_1+(1-\alpha )s_2)}{\zeta {\left(s_1\right)}^{\alpha }\zeta {\left(s_2\right)}^{1-\alpha }}$	$\frac{\alpha s_1+(1-\alpha )s_2}{s_1^{\alpha }{s}_2^{1-\alpha }}$
Kullback-Leibler divergence	$log\left(\zeta \left(s_2\right)\right)-{\sum }_{i=1}^{\infty }\frac{1}{i^s\zeta \left(s\right)}log\left(i^s\zeta \left(s\right)\right)-s_2\frac{{\zeta }^{\prime }\left(s_1\right)}{\zeta \left(s_1\right)}$	$log\left(\frac{s_1-1}{s_2-1}\right)+\frac{s_2-s_1}{s_1-1}$

).

The zeta function is known to be irrational at many positive odd integers [8,9,10] and can be calculated using Bernoulli numbers [11] for positive even integers: $\zeta \left(2n\right)=\frac{{(-1)}^{n+1}{B}_{2n}{\left(2\pi \right)}^{2n}}{2\left(2n\right)!},n\in \mathbb{N}$. The zeta function can be calculated fast [12] and precisely [13]. The derivatives of the zeta function have also been studied [12,14].

The zeta distributions are related to the Zipf distributions [15] $p_{s,N}\left(x\right)\propto \frac{1}{x^s}$ for $x\in \{1,\dots ,N\}$ and the Zipf–Mandelbrot distributions [16,17] $p_{s,q,N}\left(x\right)\propto \frac{1}{{(x+q)}^s}$ for $x\in \{1,\dots ,N\}$, which play an important role in quantitative linguistics. See [6] for more details. The Zipf distributions and the Zipf–Mandelbrot distributions both have finite support and can be interpreted as truncated zeta distributions (right truncation for Zipf distributions and both left & right truncations for the Zipf–Mandelbrot distributions) with normalizing constants, which can be calculated approximately using properties of the zeta function [18]. Left-only truncations of the Zeta distributions are called Hurwitz zeta distributions [19]. Similarly, truncated Pareto distributions are used in applications [20]. Notice that truncated distributions of an exponential family with fixed truncation support form another exponential family [21]. The zeta distributions are infinite divisible [19,22]: A random variable following a zeta distribution can be expressed as the probability distribution of the sum of an arbitrary number of independent and identically distributed random variables. In applications, it is important to quantitatively discriminate between zeta distributions (see, for example [23,24] or [25]). Mixtures of zeta distributions have also been used to model social networks [26]. In general, products of exponential families yield other exponential families. The products of d zeta distributions form an exponential family called the zeta-star distributions [22].

In this paper, we first study various information-theoretic measures between zeta distributions by considering them as a discrete exponential family [7]: That is, we consider the $\alpha$-divergences [27] between zeta distributions in Section 2, and study their limit Kullback–Leibler-oriented divergences when $\alpha \to 1$ and $\alpha \to 0$ in Section 3. We then compare these results with the counterpart results obtained for the continuous exponential family of Pareto distributions in Section 4. Finally, we conclude this work in Section 5.

2. Amari’s $\mathbf{\alpha }$-Divergences and Sharma–Mittal Divergences

To measure the dissimilarity between two zeta distributions $p_{s_1}$ and $p_{s_2}$, one can use the α-divergences [27] defined for a real $\alpha \in (0,1)$ as follows:

$$D_{\alpha }[p_{s_1}:p_{s_2}]:=\frac{1}{\alpha (1-\alpha )}\left(1-I_{\alpha }[p_{s_1}:p_{s_2}]\right)=D_{1-\alpha }[p_{s_2}:p_{s_1}],$$

(4)

where

$$I_{\alpha }[p_1,p_2]:=\sum _{i=1}^{\infty }{p}_1{\left(x\right)}^{\alpha }{p}_2{\left(x\right)}^{1-\alpha }=I_{1-\alpha }[p_2:p_1],$$

(5)

is the $\alpha$-Bhattacharyya coefficient (a similarity measure also called an affinity coefficient).

It follows from [28] that the skewed Bhattacharyya coefficient amounts to a skewed Jensen divergence between the natural parameters of the exponential family $\mathcal{E}$:

$$I_{\alpha }[p_{s_1}:p_{s_2}]=exp\left(-J_{F,\alpha }(s_1:s_2)\right),$$

(6)

where $J_{F,\alpha }$ is the skewed Jensen divergence induced by a strictly convex and smooth convex function $F\left(\theta \right)$:

$$\begin{array}{ccc}\hfill J_{F,\alpha }(s_1:s_2)& :=& \alpha F\left(s_1\right)+(1-\alpha )F\left(s_2\right)-F(\alpha s_1+(1-\alpha )s_2)\ge 0,\hfill \end{array}$$

(7)

$$\begin{array}{ccc}& =& log\left(\frac{\zeta {\left(s_1\right)}^{\alpha }\zeta {\left(s_2\right)}^{1-\alpha }}{\zeta (\alpha s_1+(1-\alpha )s_2)}\right).\hfill \end{array}$$

(8)

Thus, we have the $\alpha$-divergences between two zeta distributions $p_{s_1}$ and $p_{s_2}$ available in closed form.

Theorem 1

($\alpha$-divergences between two zeta distributions). The α-divergence for $\alpha \in (0,1)$ between two zeta distributions $p_{s_1}$ and $p_{s_2}$ is:

$$D_{\alpha }[p_{s_1}:p_{s_2}]=\frac{1}{\alpha (1-\alpha )}\left(1-\frac{\zeta (\alpha s_1+(1-\alpha )s_2)}{\zeta {\left(s_1\right)}^{\alpha }\zeta {\left(s_2\right)}^{1-\alpha }}\right).$$

It follows that when $s_1$, $s_2$, and $\alpha s_1+(1-\alpha )s_2$ are all positive even integers, we can evaluate exactly the $\alpha$-divergences between $p_{s_1}$ and $p_{s_2}$.

Example 1.

Consider $s_1=4$ and $s_2=12$ with $\alpha =\frac{1}{2}$ so that $\alpha s_1+(1-\alpha )s_2=8$. Using the formula [11] $\zeta \left(2n\right)=\frac{{(-1)}^{n+1}{B}_{2n}{\left(2\pi \right)}^{2n}}{2\left(2n\right)!},n\in \mathbb{N}$ where $B_{2n}$ denotes the Bernoulli numbers, the zeta functions can be calculated exactly at 4, 8 and 12: $\zeta \left(4\right)=\frac{{\pi }^4}{90}$, $\zeta \left(8\right)=\frac{{\pi }^8}{9450}$, and $\zeta \left(12\right)=\frac{691{\pi }^{12}}{638512875}$. The α-divergence for $\alpha =\frac{1}{2}$ is the squared Hellinger divergence $D_{\frac{1}{2}}[p_{s_1},p_{s_2}]={\sum }_{i=1}^{\infty }{\left(\sqrt{p_{s_1}\left(i\right)}-\sqrt{p_{s_2}\left(i\right)}\right)}^2$. Thus, we find the exact squared Hellinger divergence: $D_{\frac{1}{2}}[p_4,p_{12}]=4\left(1-3\sqrt{\frac{715}{6910}}\right)\simeq 0.139929\dots$.

Let us report another example where the squared Hellinger divergence is expressed using the zeta function:

Example 2.

We consider $s_1=3$, $s_2=7$ and $\alpha =\frac{1}{2}$ so that $\alpha s_1+(1-\alpha )s_2=5$. Then, we have $D_{\frac{1}{2}}[p_3,p_7]=4\left(1-\frac{\zeta \left(5\right)}{\sqrt{\zeta \left(3\right)\zeta \left(7\right)}}\right)\simeq 0.23261\dots$

Since ${lim}_{\alpha \to 1}{D}_{\alpha }[p_{s_1}:p_{s_2}]=D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]$ is the Kullback–Leibler divergence [27] (KLD)

$$D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]:=\sum _{i=1}^{\infty }{p}_{s_1}\left(i\right)log\frac{p_{s_1}\left(i\right)}{p_{s_2}\left(i\right)},$$

(9)

we can approximate the KLD by $D_{1-ϵ}[s_1:s_2]$ for a small value of $ϵ$ (say, $ϵ={10}^{-3}$) using fast methods to compute the zeta function [12].

Corollary 1

(Approximation of the Kullback–Leibler divergence). The Kullback–Leibler divergence between two zeta distributions $p_{s_1}$ and $p_{s_2}$ can be approximated for small values $ϵ>0$ by

$$D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]\simeq D_{1-ϵ}[p_{s_1}:p_{s_2}]=\frac{1}{ϵ(1-ϵ)}\left(1-\frac{\zeta ((1-ϵ)s_1+ϵs_2)}{\zeta {\left(s_1\right)}^{1-ϵ}\zeta {\left(s_2\right)}^{ϵ}}\right).$$

Example 3.

We let $1-ϵ=0.99$, $1-ϵ=0.999$, $1-ϵ=0.9999$, $1-ϵ=0.99999$, and find the following numerical approximations:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]& {\simeq }_{1-ϵ=0.99}& 0.473\dots \hfill \\ \hfill D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]& {\simeq }_{1-ϵ=0.999}& 0.482\dots \hfill \\ \hfill D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]& {\simeq }_{1-ϵ=0.9999}& 0.483\dots \hfill \\ \hfill D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]& {\simeq }_{1-ϵ=0.99999}& 0.483\dots \hfill \end{array}$$

We can also calculate the KLD $D_{\mathrm{KL}}[p_{s_1}^{{\mathcal{X}}_1}:p_{s_2}^{{\mathcal{X}}_2}]$ between two truncated zeta distributions with nested supports ${\mathcal{X}}_1\subseteq {\mathcal{X}}_2$. See [21]. A truncated zeta distribution on the support $\{a,a+1,\dots ,b\}\subset \mathbb{N}$ (with $b>a$) has pmf $p_s^{a,b}\left(x\right)=\frac{p_s\left(x\right)}{{\mathsf{\Phi }}_s\left(b\right)-{\mathsf{\Phi }}_s\left(a\right)}$ where ${\mathsf{\Phi }}_s\left(u\right)$ is the cumulative distribution function ${\mathsf{\Phi }}_s\left(u\right)={\sum }_{x\in \{1,\dots ,u\}}{p}_s\left(x\right)=\frac{1}{\zeta \left(s\right)}{\sum }_{x\in \{1,\dots ,u\}}\frac{1}{x^s}$.

The Chernoff information [29] is defined by $C[p_1,p_2]=-log{min}_{\alpha \in (0,1)}{I}_{\alpha }[p_1,p_2]$. The unique optimal value ${\alpha }^{*}$ maximizing the Chernoff $\alpha$-divergences $C_{\alpha }[p_1,p_2]=-log{I}_{\alpha }[p_1,p_2]$ is called the Chernoff exponent [29] due to its role in bounding the probability of error in Bayesian hypothesis testing. When both pdfs or pmfs belong to the same exponential family, we have [29]

$$C[p_{{\theta }_1},p_{{\theta }_2}]=J_F({\theta }_1:{\left({\theta }_1{\theta }_2\right)}_{{\alpha }^{*}})=B_F({\theta }_1:{\left({\theta }_1{\theta }_2\right)}_{{\alpha }^{*}})=B_F({\theta }_2:{\left({\theta }_1{\theta }_2\right)}_{{\alpha }^{*}}),$$

(10)

where $B_F$ denotes the Bregman divergence (corresponding to the KLD) and ${\left({\theta }_1{\theta }_2\right)}_{{\alpha }^{*}}={\alpha }^{*}{\theta }_1+(1-{\alpha }^{*}){\theta }_2$. For a uniorder exponential family such as the zeta distributions, a closed-form formula for the optimal Chernoff exponent ${\alpha }^{*}$ is reported in [29]: ${\alpha }^{*}=\frac{{F^{\prime }}^{-1}\left(\frac{F\left({\theta }_2\right)-F\left({\theta }_1\right)}{{\theta }_2-{\theta }_1}\right)-{\theta }_1}{{\theta }_2-{\theta }_1}$.

The Sharma–Mittal divergences [30] between two densities p and q is a biparametric family of relative entropies defined by

$$D_{\alpha ,\beta }[p:q]=\frac{1}{\beta -1}\left({\left(\int p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\mathrm{d}x\right)}^{\frac{1-\beta }{1-\alpha }}-1\right),\forall \alpha >0,\alpha \ne 1,\beta \ne 1.$$

(11)

The Sharma–Mittal divergence is induced from the Sharma–Mittal entropies, which unify the extensive Rényi entropies with the non-extensive Tsallis entropies [30]. The Sharma–Mittal divergences include the Rényi divergences ($\beta \to 1$) and the Tsallis divergences ($\beta \to \alpha$), and in the limit case of $\alpha ,\beta \to 1$, the Kullback–Leibler divergence [31]. When both densities $p=p_{{\theta }_1}$ and $q=p_{{\theta }_2}$ belong to the same exponential family, we have the following closed-form formula [31]:

$$D_{\alpha ,\beta }[p_{{\theta }_1}:p_{{\theta }_2}]=\frac{1}{\beta -1}\left(e^{-\frac{1-\beta }{1-\alpha }{J}_{F,\alpha }\left({\theta }_1:{\theta }_2\right)}-1\right).$$

(12)

Thus, we get the following theorem:

Theorem 2.

For $\alpha >0$, $\alpha \ne 1$, $\beta \ne 1$, the Sharma–Mittal divergence between two zeta distributions $p_{s_1}$ and $p_{s_2}$ is

$$D_{\alpha ,\beta }[p_{s_1}:p_{s_2}]=\frac{1}{\beta -1}\left({\left(\frac{\zeta (\alpha s_1+(1-\alpha )s_2)}{\zeta {\left(s_1\right)}^{\alpha }\zeta {\left(s_2\right)}^{1-\alpha }}\right)}^{\frac{1-\beta }{1-\alpha }}-1\right).$$

3. The Kullback–Leibler Divergence between Two Zeta Distributions

It is well-known that the KLD between two probability mass functions of an exponential family amounts to a reverse Bregman divergence induced by the cumulant function [32]: $D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]=B_F^{*}({\theta }_1:{\theta }_2):=B_F({\theta }_2:{\theta }_1)$ (with ${\theta }_1=s_1$ and ${\theta }_2=s_2$). Furthermore, this Bregman divergence amounts to a Fenchel–Young divergence [33] so that we have

$$D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]=B_F({\theta }_2:{\theta }_1)=F\left(\theta \left(s_2\right)\right)+F^{*}\left(\eta \left(s_1\right)\right)-\theta \left(s_2\right)\eta \left(s_1\right),$$

(13)

where $F^{*}\left(\eta \right)$ denotes the Legendre convex conjugate of F, $\theta \left(s\right)=s$ and $\eta \left(s\right)=F^{\prime }\left(\theta \left(s\right)\right)=E_{p_s}\left[t\left(x\right)\right]=-E_{p_s}[logx]$, see [7]. Moreover, the convex conjugate $F^{*}\left(\eta \left(s\right)\right)$ corresponds to the negentropy [34]: $F^{*}\left(\eta \left(s\right)\right)=-H\left[p_s\right]$, where the entropy of a zeta distribution $p_s$ is defined by:

$$H\left[p_s\right]:=\sum _{i=1}^{\infty }{p}_s\left(i\right)log\frac{1}{p_s\left(i\right)}.$$

(14)

Using the fact that ${\sum }_{i=1}^{\infty }{p}_s\left(i\right)=1={\sum }_{i=1}^{\infty }\frac{1}{i^s\zeta \left(s\right)}$, we can express the entropy as follows:

$$\begin{array}{ccc}\hfill H\left[p_s\right]& =& \sum _{i=1}^{\infty }\frac{1}{i^s\zeta \left(s\right)}log{i}^s+log\left(\zeta \left(s\right)\right)\sum _{i=1}^{\infty }\frac{1}{i^s\zeta \left(s\right)},\hfill \\ & =& \sum _{i=1}^{\infty }\frac{1}{i^s\zeta \left(s\right)}log\left(i^s\zeta \left(s\right)\right).\hfill \end{array}$$

Since $F\left(\theta \right)=log\zeta \left(\theta \right)$, we have $\eta \left(\theta \right)=F^{\prime }\left(\theta \right)=\frac{{\zeta }^{\prime }\left(\theta \right)}{\zeta \left(\theta \right)}$. The function $\frac{{\zeta }^{\prime }\left(\theta \right)}{\zeta \left(\theta \right)}$ has been tabulated in [35] (page 400). Notice that the maximum likelihood estimator [7] of n independently and identically distributed observations $x_1,\dots ,x_n$ is $\widehat{\eta }=\frac{1}{n}{\sum }_{i=1}^{n}t\left(x_i\right)$. Thus we have:

$$\widehat{\eta }=\frac{{\zeta }^{\prime }\left(\widehat{\theta }\right)}{\zeta \left(\widehat{\theta }\right)}=-\frac{1}{n}\sum _{i=1}^{n}log{x}_i.$$

(15)

The inverse of the zeta function ${\zeta }^{-1}(·)$ has been studied in [36].

Proposition 1

(KLD between zeta distributions). The Kullback–Leibler divergence between two zeta distributions can be written as:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]& =& log\left(\zeta \left(s_2\right)\right)-H\left[p_{s_1}\right]+s_2{E}_{p_{s_1}}[logx],\hfill \\ & =& log\left(\zeta \left(s_2\right)\right)-\sum _{i=1}^{\infty }\frac{1}{i^{s_1}\zeta \left(s_1\right)}log\left(i^{s_1}\zeta \left(s_1\right)\right)-s_2\frac{{\zeta }^{\prime }\left(s_1\right)}{\zeta \left(s_1\right)}.\hfill \end{array}$$

Moreover, the logarithmic derivative of the zeta function can be expressed using the von Mangoldt function [37] (page 1850) for $\theta >1$:

$$\eta \left(\theta \right)=\frac{{\zeta }^{\prime }\left(\theta \right)}{\zeta \left(\theta \right)}=-\sum _{i=1}^{\infty }\frac{\mathsf{\Lambda }\left(i\right)}{i^{\theta }},$$

(16)

where $\mathsf{\Lambda }\left(i\right)=logp$ is $i=p^k$ for some prime p and integer $k\ge 1$ and 0 otherwise. Notice that the zeta function can be calculated using Euler product formula: $\zeta \left(\theta \right)={\prod }_{p:\mathrm{prime}}\frac{1}{1-p^{-\theta }}$.

Theorem 3.

The Kullback–Leibler divergence between two zeta distributions can be expressed using the real zeta function ζ and the von Mangoldt function Λ as:

$$D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]=log\left(\zeta \left(s_2\right)\right)-\sum _{i=1}^{\infty }\frac{1}{i^s\zeta \left(s\right)}log\left(i^s\zeta \left(s\right)\right)+s_2\sum _{i=1}^{\infty }\frac{\mathsf{\Lambda }\left(i\right)}{i^{s_1}}.$$

Example 4.

Consider $s_1=4$ and $s_2=12$. Letting $1-ϵ=0.9999$ and using Corollary 1, we obtain

$$D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]\simeq D_{1-ϵ}[p_{s_1}:p_{s_2}]=0.430479743738878\dots$$

Let us now calculate the KLD using Theorem 3; we obtain $log\left(\zeta \left(s_2\right)\right)=log\frac{691{\pi }^{1}2}{638512875}$, $H\left[p_{s_1}\right]\simeq 0.3337829096182664\dots$ (using 100 terms), and $\eta \left(s_1\right)=-0.06366938697034288\dots$ (using 100 terms) so that we have

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{s_1}:p_{s_2}]& =& log\left(\zeta \left(s_2\right)\right)-\sum _{i=1}^{\infty }\frac{1}{i^s\zeta \left(s\right)}log\left(i^s\zeta \left(s\right)\right)+s_2\sum _{i=1}^{\infty }\frac{\mathsf{\Lambda }\left(i\right)}{i^{s_1}},\hfill \end{array}$$

(17)

$$\begin{array}{ccc}& \simeq & 0.430495790304827\dots \hfill \end{array}$$

(18)

It is well-known that the KLD between two arbitrarily close zeta distributions $p_s$ and $p_{s+\mathrm{d}s}$ amounts to half of the quadratic distance induced by the Fisher information:

$$D_{\mathrm{KL}}[p_s:p_{s+\mathrm{d}s}]\approx \frac{1}{2}I\left(s\right)\mathrm{d}{s}^2,$$

(19)

where

$$I\left(s\right)=E_{p_s}\left[{{(log{p}_s\left(x\right))}^{\prime }}^2\right]=-E_{p_s}\left[{(log{p}_s\left(x\right))}^{\prime \prime }\right],$$

(20)

where the first-order and second-order derivatives are taken with respect to the parameter s. Thus, for uniorder exponential families, the Fisher information matrix is

$$I\left(s\right)=-E_{p_s}\left[{(log{p}_s\left(x\right))}^{\prime \prime }\right]={(log\zeta \left(s\right))}^{\prime \prime }=\frac{\zeta \left(s\right){\zeta }^{\prime \prime }\left(s\right)-{\zeta }^{\prime }{\left(s\right)}^2}{{\zeta }^2\left(s\right)}.$$

(21)

This second-order derivative ${(log\zeta \left(s\right))}^{\prime \prime }$ has been studied in [38]. We have

$$I\left(s\right)=\sum _{n=1}^{\infty }\mathsf{\Lambda }\left(n\right)log\left(n\right)n^{-s}$$

(22)

where $\mathsf{\Lambda }\left(n\right)$ is the Von Mangoldt function.

4. Comparison of the Zeta Family with a Pareto Subfamily

The zeta distribution is also called the “pure power-law distribution” in the literature [2].

We can compute the $\alpha$-divergences between two Pareto distributions $q_{s_1}$ and $q_{s_2}$ with fixed scale 1 and respective shapes $s_1-1$ and $s_2-1$. The Pareto density writes $q_s\left(x\right)=\frac{s-1}{x^s}$ for $x\in \mathcal{X}=(1,\infty )$. The family of such Pareto distributions forms a continuous exponential family with natural parameter $\theta =s$, sufficient statistic $t\left(x\right)=-log\left(x\right)$, and convex cumulant function $F\left(\theta \right)=-log(\theta -1)$ for $\theta \in \mathsf{\Theta }=(1,\infty )$. Thus we have [28]:

$$\begin{array}{ccc}\hfill I_{\alpha }[q_1:q_2]=\int q_{s_1}{\left(x\right)}^{\alpha }{q}_{s_2}{\left(x\right)}^{1-\alpha }\mathrm{d}x& =& exp(-J_{F,\alpha }({\theta }_1:{\theta }_2)),\hfill \end{array}$$

(23)

$$\begin{array}{ccc}& =& \frac{\alpha s_1+(1-\alpha )s_2}{s_1^{\alpha }{s}_2^{1-\alpha }},\hfill \end{array}$$

(24)

and we obtain the following closed form for the $\alpha$-divergences between two Pareto distributions $q_{s_1}$ and $q_{s_2}$:

$$D_{\alpha }[q_{s_1}:q_{s_2}]=\frac{1}{\alpha (1-\alpha )}\left(1-\frac{\alpha s_1+(1-\alpha )s_2}{s_1^{\alpha }{s}_2^{1-\alpha }}\right).$$

(25)

The moment parameter is $\eta \left(\theta \right)=F^{\prime }\left(\theta \right)=-\frac{1}{\theta -1}$ so that $\theta \left(\eta \right)=1-\frac{1}{\eta }$ and $F^{*}\left(\eta \right)=\theta \left(\eta \right)\eta -F\left(\theta \left(\eta \right)\right)=\eta -1-log(-\eta )$. It follows that the KLD is

$$D_{\mathrm{KL}}[q_{s_1}:q_{s_2}]=B_F({\theta }_2:{\theta }_1)=log\left(\frac{s_1-1}{s_2-1}\right)+\frac{s_2-s_1}{s_1-1}.$$

(26)

The differential entropy of the Pareto distribution $q_s$ is

$$h\left[q_s\right]=-{\int }_1^{\infty }{q}_s\left(x\right)log{q}_s\left(x\right)\mathrm{d}x=-F^{*}\left(\eta \left(s\right)\right)$$

(27)

with $\eta \left(s\right)=-\frac{1}{s-1}$. We find that

$$h\left[q_s\right]=1+\frac{1}{s-1}-log(s-1).$$

(28)

Example 5.

For comparison, we calculate the KLD between two Pareto distributions with parameters $s_1=4$ and $s_2=12$. We find

$$D_{\mathrm{KL}}[q_{s_1}:q_{s_2}]=log\frac{3}{11}+\frac{8}{3}\simeq 1.367383682536406\dots$$

5. Conclusions

Table 1 compares the discrete exponential family of zeta distributions with the continuous exponential family of Pareto distributions with fixed scale 1.

In general, it is interesting to consider discrete counterparts of continuous exponential families. For example, the discrete Gaussian distributions or discrete normal distributions defined as maximum entropy distributions have been studied in [39,40]. The log-normalizer or cumulant function of the discrete Gaussian distributions are related to the Riemann theta function [41]. Given a prescribed sufficient statistics $t\left(x\right)$, we may define the continuous exponential family with respect to the Lebesgue measure $\mu$ as the probability density functions $p\left(x\right)$ maximizing the differential entropy under the moment constraint $E_p\left[t\left(x\right)\right]=\eta$. The corresponding discrete exponential family is obtained by the distributions with probability mass functions maximizing Shannon entropy under the moment constraint $E_p\left[t\left(x\right)\right]=\eta$.

Additional material is available online at https://franknielsen.github.io/ZetaParetoExpFam/index.html (accessed on 18 October 2022).