Representational Rényi Heterogeneity

Abstract

A discrete system’s heterogeneity is measured by the Rényi heterogeneity family of indices (also known as Hill numbers or Hannah–Kay indices), whose units are the numbers equivalent. Unfortunately, numbers equivalent heterogeneity measures for non-categorical data require a priori (A) categorical partitioning and (B) pairwise distance measurement on the observable data space, thereby precluding application to problems with ill-defined categories or where semantically relevant features must be learned as abstractions from some data. We thus introduce representational Rényi heterogeneity (RRH), which transforms an observable domain onto a latent space upon which the Rényi heterogeneity is both tractable and semantically relevant. This method requires neither a priori binning nor definition of a distance function on the observable space. We show that RRH can generalize existing biodiversity and economic equality indices. Compared with existing indices on a beta-mixture distribution, we show that RRH responds more appropriately to changes in mixture component separation and weighting. Finally, we demonstrate the measurement of RRH in a set of natural images, with respect to abstract representations learned by a deep neural network. The RRH approach will further enable heterogeneity measurement in disciplines whose data do not easily conform to the assumptions of existing indices.

1. Introduction

Measuring heterogeneity is of broad scientific importance, such as in studies of biodiversity (ecology and microbiology) [1,2], resource concentration (economics) [3], and consistency of clinical trial results (biostatistics) [4], to name a few. In most of these cases, one measures the heterogeneity of a discrete system equipped with a probability mass function.

Discrete systems assume that all observations of a given state are identical (zero distance), and that all pairwise distances between states are permutation invariant. This assumption is violated when relative distances between states are important. For example, an ecosystem is not biodiverse if all species serve the same functional role [5]. Although species are categorical labels, their pairwise differences in terms of ecological functions differ and thus violate the discrete space assumptions. Mathematical ecologists have thus developed heterogeneity measures for non-categorical systems, which they generally call “functional diversity indices” [6,7,8,9,10,11]. These indices typically require a priori discretization and specification of a distance function on the observable space.

The requirement for defining the state space a priori is problematic when the states are incompletely observable: that is, when they may be noisy, unreliable, or invalid. For example, consider sampling a patient from a population of individuals with psychiatric disorders and assigning a categorical state label corresponding to his or her diagnosis according to standard definitions [12]. Given that psychiatric conditions are not defined by objective biomarkers, the individual’s diagnostic state will be uncertain. Indeed, many of these conditions are inconsistently diagnosed across raters [13], and there is no guarantee that they correspond to valid biological processes. Alternatively, it is possible that variation within some categorical diagnostic groups is simply related to diagnostic “noise,” or nuisance variation, but that variation within other diagnostic groups constitutes the presence of sub-strata. Appropriate measurement of heterogeneity in such disciplines requires freedom from the discretization requirement of existing non-categorical heterogeneity indices.

Pre-specified distance functions may fail to capture semantically relevant geometry in the raw feature space. For example, the Euclidean distance between Edmonton and Johannesburg is relatively useless since the straight-line path cannot be traversed. Rather, the appropriate distances between points must account for the data’s underlying manifold of support. Representation learning addresses this problem by learning a latent embedding upon which distances are of greater semantic relevance [14]. Indeed, we have observed superior clustering of natural images embedded on Riemannian manifolds [15] (but also see Shao et al. [16]), and preservation of semantic hierarchies when linguistic data are embedded on a hyperbolic space [17].

Therefore, we seek non-categorical heterogeneity indices without requisite a priori definition of categorical state labels or a distance function. The present study proposes a solution to these problems based on the measurement of heterogeneity on learned latent representations, rather than on raw observable data. Our method, representational Rényi heterogeneity (RRH), involves learning a mapping from the space of observable data to a latent space upon which an existing measure (the Rényi heterogeneity [18], also known as the Hill numbers [19] or Hannah–Kay indices [20]) is meaningful and tractable.

The paper is structured as follows. Section 2 introduces the original categorical formulation of Rényi heterogeneity and various approaches by which it has been generalized for application on non-categorical spaces [8,10,21]. Limitations of these indices are highlighted, thereby motivating Section 3, which introduces the theory of Representational Rényi Heterogeneity (RRH), which generalizes the process for computing many indices of biodiversity and economic equality. Section 4 provides an illustration of how RRH may be measured in various analytical contexts. We provide an exact comparison of RRH to existing non-categorical heterogeneity indices under a tractable mixture of beta distributions. To highlight the generalizability of our approach to complex latent variable models, we also provide an evaluation of RRH applied to the latent representations of a handwritten image dataset [22] learned by a variational autoencoder [23,24]. Finally, in Section 5 we provide a summary of our findings and discuss avenues for future work.

2. Existing Heterogeneity Indices

2.1. Rényi Heterogeneity in Categorical Systems

There are many approaches to derive Rényi heterogeneity [18,19,20]. Here, we loosely follow the presentation of Eliazar and Sokolov [25] by using the metaphor of repeated sampling from a discrete system X with event space $\mathcal{X}=\left\{1,2,\dots ,n\right\}$ and probability distribution $\mathbf{p}={\left(p_i\right)}_{i=1,2,\dots ,n}$. The probability that $q\in {\mathbb{N}}_{>1}$ independent and identically distributed (i.i.d.) realizations of X, sampled with replacement, will be identical is

$${\mathbb{P}}_X\left[x_1=x_2=\cdots =x_q\right]=\sum _{i=1}^n{p}_i^q.$$

(1)

Now let $X_{\ast }$ be an idealized reference system with a uniform probability distribution over $n_{\ast }$ categorical states, ${\mathbf{p}}_{\ast }={\left(n_{\ast }^{-1}\right)}_{i=1,2,\dots ,n_{\ast }}$, and let $\left(x_{\ast 1},x_{\ast 2},\dots ,x_{\ast q}\right)$ be a sample of q i.i.d. realizations of $X_{\ast }$ such that

$${\mathbb{P}}_X\left[x_1=x_2=\cdots =x_q\right]={\mathbb{P}}_{X_{\ast }}\left[x_{\ast 1}=x_{\ast 2}=\cdots =x_{\ast q}\right]=\sum _{i=1}^{n_{\ast }}{n}_{\ast }^{-q}.$$

(2)

We call $X_{\ast }$ an “idealized” categorical system because its probability distribution is uniform, and it is a “reference” system for X in that the probability of drawing homogeneous samples of q observations from both systems is identical. Substituting Equation (2) into Equation (1) and solving for $n_{\ast }$ yields the Rényi heterogeneity of order q,

$${\mathsf{\Pi }}_q\left(\mathbf{p}\right)={\left(\sum _{i=1}^n{p}_i^q\right)}^{\frac{1}{1-q}}=n_{\ast },$$

(3)

whose units are the numbers equivalent of system X [1,26,27,28], insofar as $n_{\ast }$ is the number of states in an “equivalent” (idealized reference) system $X_{\ast }$. Thus far, we have restricted the parameter q to take integer values greater than 1 solely to facilitate this intuitive derivation in a concise fashion. However, the elasticity parameter q in Equation (3) can be any real number (but $q\ne 1$), although in the context of heterogeneity measurement only $q\ge 0$ are used [1,25]. Despite Equation (3) being udefined at $q=1$ directly, L’Hôpital’s rule can be used to show that the limit $q\to 1$ exists, wherein it corresponds to the exponential of Shannon’s entropy [28,29], known as perplexity [30].

Equation (3) is the exponential of Rényi’s entropy [18], and is alternatively known as the Hill numbers in ecology [1,19], Hannah–Kay indices in economics [20], and generalized inverse participation ratio in physics [25]. Interestingly, it generalizes or can be transformed into several heterogeneity indices that are commonly employed across scientific disciplines (Table 1).

Table 1. Relationships between Rényi heterogeneity and various diversity or inequality indices for a system X with event space $\mathcal{X}=\{1,2,\dots ,n\}$ and probability distribution $\mathbf{p}={\left(p_i\right)}_{i=1,2,\dots ,n}$. The function $𝟙[·]$ is an indicator function that evaluates to 1 if its argument is true or to 0 otherwise.

Index	Expression
Observed richness [31]	${\mathsf{\Pi }}_0\left(\mathbf{p}\right)={\sum }_{i=1}^n𝟙[p_i>0]$
Perplexity [30]	${\mathsf{\Pi }}_1\left(\mathbf{p}\right)=exp\left\{-{\sum }_{i=1}^n{p}_{i}log{p}_i\right\}$
Inverse Simpson concentration [1]	${\mathsf{\Pi }}_2\left(\mathbf{p}\right)={\left({\sum }_{i=1}^n{p}_i^2\right)}^{-1}$
Berger-Parker Diversity Index [32,33]	${\mathsf{\Pi }}_{\infty }\left(\mathbf{p}\right)={\left({max}_i{p}_i\right)}^{-1}$
Rényi entropy [18]	$R_q\left(\mathbf{p}\right)=log{\mathsf{\Pi }}_q\left(\mathbf{p}\right)$
Shannon entropy [29]	$H\left(\mathbf{p}\right)=log{\mathsf{\Pi }}_1\left(\mathbf{p}\right)$
Tsallis entropy [34]	$T_q\left(\mathbf{p}\right)=\frac{1}{q-1}\left(1-{\mathsf{\Pi }}_q{\left(\mathbf{p}\right)}^{1-q}\right)$
Simpson concentration [35]	$\mathrm{Simpson}\left(\mathbf{p}\right)={\left({\mathsf{\Pi }}_2\left(\mathbf{p}\right)\right)}^{-1}$
Gini-Simpson index [36]	$\mathrm{GSI}\left(\mathbf{p}\right)=1-\mathrm{Simpson}\left(\mathbf{p}\right)$
Generalized entropy index [3,37]	$\mathrm{GEI}\left(\mathbf{p}\right)=\frac{1}{q(q-1)}\left[{\left(\frac{1}{n}{\mathsf{\Pi }}_q\left(\mathbf{p}\right)\right)}^{1-q}-1\right]$

2.1.1. Properties of the Rényi Heterogeneity

Equation (3) satisfies several properties that render it a preferable measure of heterogeneity. These have been detailed elsewhere [1,20,25,28,33,38], but we focus on three properties that are of particular relevance for the remainder of this paper.

First, ${\mathsf{\Pi }}_q$ satisfies the principle of transfers [39,40] which states that any equality-increasing transfer of probability between states must increase the heterogeneity. The maximal value of ${\mathsf{\Pi }}_q$ is attained if and only if $p_i=p_j$ for all $(i,j)\in \{1,2,\dots ,n\}$. This property follows from Schur-concavity of Equation (3) [20].

Second, ${\mathsf{\Pi }}_q$ satisfies the replication principle [1,38,41], which is equivalent to stating that Equation (3) scales linearly with the number of equally probable states in an idealized categorical system [25]. More formally, consider a set of systems $X_1,X_2,\dots ,X_N$ with probability distributions ${\mathbf{p}}_1,{\mathbf{p}}_2,\dots ,{\mathbf{p}}_N$ over respective discrete event spaces ${\mathcal{X}}_1,{\mathcal{X}}_2,\dots ,{\mathcal{X}}_N$. These systems are also assumed to satisfy the following properties:

• Event spaces are disjoint: ${\mathcal{X}}_i\cap {\mathcal{X}}_j=\varnothing$ for all $(i,j)\in \{1,2,\dots ,N\}$ where $i\ne j$
• All systems have equal heterogeneity: ${\mathsf{\Pi }}_q\left({\mathbf{p}}_1\right)={\mathsf{\Pi }}_q\left({\mathbf{p}}_2\right)=\cdots ={\mathsf{\Pi }}_q\left({\mathbf{p}}_i\right)=\cdots ={\mathsf{\Pi }}_q\left({\mathbf{p}}_N\right)$

The replication principle states that if we combine $X_1,X_2,\dots ,X_N$ into a pooled system X with probability distribution $\overline{\mathbf{p}}$, then

$${\mathsf{\Pi }}_q\left(\overline{\mathbf{p}}\right)=N{\mathsf{\Pi }}_q\left({\mathbf{p}}_i\right)$$

(4)

must hold (see Appendix A for proof that Rényi heterogeneity satisfies the replication principle).

The replication principle suggests that Equation (3) satisfies a property known as decomposability, in that the heterogeneity of a pooled system can be decomposed into that arising from variation within and between component subsystems. However, we require that this property be satisfied when either (A) subsystems’ event spaces are overlapping, or (B) subsystems do not have equal heterogeneity. The decomposability property will be particularly important for Section 3, and so we detail it further in Section 2.1.2.

2.1.2. Decomposition of Categorical Rényi Heterogeneity

Consider a system X defined by pooling subsystems $X_1,X_2,\dots ,X_N$ with potentially overlapping event spaces ${\mathcal{X}}_1,{\mathcal{X}}_2,\dots ,{\mathcal{X}}_N$, respectively. The event space of the pooled system is defined as

$$\mathcal{X}={\cup }_{i=1}^N{\mathcal{X}}_i=\left\{1,2,\dots ,n\right\}.$$

(5)

Furthermore, we define the matrix $\mathbf{P}={\left(p_{ij}\right)}_{i=1,2,\dots ,N}^{j=1,2,\dots ,n}$ whose i^th row is the probability of system $X_i$ being observed in each state $j\in \{1,2,\dots ,n\}$.

It may be the case that some subsystems comprise a larger proportion of X than others. For instance, if the probability distribution for subsystem $X_i$ was estimated based on a larger sample size than that of $X_j$, one may want to weight the contribution of $X_i$ higher. Thus, we define a column vector of weights $\mathbf{w}={\left(w_i\right)}_{i=1,2,\dots ,N}$ over the N subsystems such that ${\sum }_{i=1}^N{w}_i=1$ and $w_i\ge 0$ for all i. The probability distribution over states in the pooled system X may thus be computed as $\overline{\mathbf{p}}={\sum }_{i=1}^N{w}_i{\mathbf{p}}_i$, from which the definition of pooled heterogeneity follows:

$${\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{P},\mathbf{w}\right)={\left[\sum _{j=1}^n{\left(\sum _{i=1}^N{w}_i{p}_{ij}\right)}^q\right]}^{\frac{1}{1-q}}.$$

(6)

One can interpret ${\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{P},\mathbf{w}\right)$ as the effective number of states in the pooled categorical system X.

Jost [28] showed that the within-group heterogeneity, which is the effective number of unique states arising from individual component systems, can be defined as

$${\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathbf{P},\mathbf{w}\right)={\left[\frac{{\sum }_{i=1}^N{w}_i^q\left({\sum }_{j=1}^n{p}_{ij}^q\right)}{{\sum }_{k=1}^N{w}_k^q}\right]}^{\frac{1}{1-q}},$$

(7)

For example, in the case where all subsystems have disjoint event spaces with heterogeneity equal to constant $\nu$, then they each contribute $\nu$ unique states to the pooled system X.

Deriving the between-group heterogeneity ${\mathsf{\Pi }}_q^{\mathrm{B}}\left(\mathbf{P},\mathbf{w}\right)$, is thus straightforward. If the effective total number of states in the pooled system is ${\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{P},\mathbf{w}\right)$, and the effective number of unique states contributed by distinct subsystems is ${\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathbf{P},\mathbf{w}\right)$, then

$${\mathsf{\Pi }}_q^{\mathrm{B}}\left(\mathbf{P},\mathbf{w}\right)=\frac{{\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{P},\mathbf{w}\right)}{{\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathbf{P},\mathbf{w}\right)}$$

(8)

is the effective number of completely distinct subsystems in the pooled system X. A word of caution is warranted. If we require that within-group heterogeneity is a lower bound on pooled heterogeneity [42], then (Jost [28], see Proofs 2 and 3) showed that Equation (8) will hold (A) at any value of q when weights are equal (i.e., $w_i=1/N$ for all $i\in \{1,2,\dots ,N\}$), or (B) only at $q=0$ and $q=1$ if weights are unequal.

2.1.3. Limitations of Categorical Rényi Heterogeneity

The chief limitation of Rényi heterogeneity (Equation (3)) is its assumption that all states in a system X (with event space $\mathcal{X}=\{1,2,\dots ,n\}$ and probability distribution $\mathbf{p}={\left(p_i\right)}_{i=1,2,\dots ,n}$) are categorical. More formally, the dissimilarity between a pair of observations $(x,y)\in \mathcal{X}$ from this system is defined by the discrete metric

$$d^{\ast }(x,y)=1-{\delta }_{xy},$$

(9)

where ${\delta }_{xy}$ is Kronecker’s delta, which takes a value of 1 if $x=y$ and 0 otherwise. Since the discrete metric assumption is an idealization, we have continued to use the asterisk to qualify an arbitrary distance function $d(·,·)$ as categorical in nature. The resulting expected pairwise distance matrix between states in X is

$${\mathbf{D}}^{\ast }={\left[d^{\ast }(i,j)\right]}_{i=1,2,\dots ,n}^{j=1,2,\dots ,n}=\mathbf{1}{\mathbf{1}}^{\top }-\mathbf{I},$$

(10)

where $\mathbf{1}={\left(1\right)}_{i=1,2,\dots ,n}$ is a column vector of ones, and $\mathbf{I}={\left({\delta }_{ij}\right)}_{i=1,2,\dots ,n}^{j=1,2,\dots ,n}$ is the $n\times n$ identity matrix.

Clearly, many systems of interest in the real world are not categorical. For example, although we may label a sample of organisms according to their respective species, there may be differences between these taxonomic classes that are relevant to the functioning of the ecosystem as a whole [5]. It is also possible that no valid and reliable set of categorical labels is known a priori for a system whose event space is naturally non-categorical.

2.2. Non-Categorical Heterogeneity Indices

Consider a system X with probability distribution $\mathbf{p}={\left(p_i\right)}_{i=1,2,\dots ,n}$ defined over event space $\mathcal{X}=\{1,2,\dots ,n\}$ and equipped with dissimilarity function $d_X(·,·)$. We assume that $d_X$ is more general than the discrete metric (Equation (9)), and further still need not be a true (metric) distance. For such systems, there are three heterogeneity indices whose units are numbers equivalent, and respect the replication principle [6,8,10,11,21]. Much like our derivation of the Rényi heterogeneity in Section 2.1, these indices quantify the heterogeneity of a non-categorical system as the number of states in an idealized reference system, but differ primarily in how the idealized reference is defined. We begin with a discussion of the Numbers-Equivalent Quadratic Entropy (Section 2.2.1), followed by the Functional Hill Numbers (Section 2.2.2) and the Leinster–Cobbold index [10] (Section 2.2.3).

2.2.1. Numbers Equivalent Quadratic Entropy

Rao [43] introduced the diversity index commonly known as Rao’s quadratic entropy (RQE),

$$Q_1\left(\mathbf{D},\mathbf{p}\right)=\sum _{i=1}^n\sum _{j=1}^n{D}_{ij}{p}_i{p}_j$$

(11)

where $\mathbf{D}$ is an $n\times n$ matrix where $D_{ij}=d_X(i,j)$ for states $(i,j)\in \mathcal{X}$.

Ricotta and Szeidl [21] assume that $D_{ij}=1$ means that states i and j are maximally dissimilar (i.e., categorically different), and that $D_{ij}=0$ means $i=j$, which occurs when $\mathcal{X}$ is a categorical system. An arbitrary dissimilarity matrix $\mathbf{D}$ can be rescaled to respect this assumption by applying the following transformation:

$$\tilde{\mathbf{D}}=\frac{\mathbf{D}-{min}_{ij}{D}_{ij}}{{max}_{ij}{D}_{ij}-{min}_{ij}{D}_{ij}}.$$

(12)

Under this transformation, Ricotta and Szeidl [21] search for an idealized categorical reference system $X_{\ast }$ with event space ${\mathcal{X}}_{\ast }=\{1,2,\dots ,n_{\ast }\}$, probability distribution ${\mathbf{p}}_{\ast }={\left(n_{\ast }^{-1}\right)}_{i=1,2,\dots ,n_{\ast }}$, and RQE equal to that of X. For a column vector of ones, $\mathbf{1}={\left(1\right)}_{i=1,2,\dots ,n_{\ast }}$, and the identity matrix $\mathbf{I}={\left({\delta }_{ij}\right)}_{i=1,2,\dots ,n_{\ast }}^{j=1,2,\dots ,n_{\ast }}$, this is

$$Q_1\left(\tilde{\mathbf{D}},\mathbf{p}\right)=Q_1\left(\mathbf{1}{\mathbf{1}}^{\top }-\mathbf{I},{\mathbf{p}}_{\ast }\right).$$

(13)

Expanding the right-hand side, we have

$$Q_1\left(\tilde{\mathbf{D}},\mathbf{p}\right)=\sum _{i=1}^{n_{\ast }}\sum _{j=1}^{n_{\ast }}{n}_{\ast }^{-2}\left(1-{\delta }_{ij}\right)=1-\frac{1}{n_{\ast }}.$$

(14)

Recalling that ${\mathsf{\Pi }}_q\left({\mathbf{p}}_{\ast }\right)=n_{\ast }$ and substituting into Equation (14) yields

$${\mathsf{\Pi }}_q\left({\mathbf{p}}_{\ast }\right)={\left[1-Q_1\left(\tilde{\mathbf{D}},\mathbf{p}\right)\right]}^{-1},$$

(15)

which establishes the units of ${\left[1-Q_1\left(\tilde{\mathbf{D}},\mathbf{p}\right)\right]}^{-1}$ as numbers equivalent.

For consistency, we require that ${\mathsf{\Pi }}_q\left({\mathbf{p}}_{\ast }\right)={\mathsf{\Pi }}_q\left(\mathbf{p}\right)$ if $\tilde{\mathbf{D}}$ were categorical. This only holds at $q=2$:

$${\left[1-Q_1\left(\tilde{\mathbf{D}},\mathbf{p}\right)\right]}^{-1}={\left[1-\sum _{i=1}^n\sum _{j=1}^n{p}_i{p}_j\left(1-{\delta }_{ij}\right)\right]}^{-1}={\left(\sum _{i=1}^n{p}_i^2\right)}^{-1}={\mathsf{\Pi }}_2\left({\mathbf{p}}_{\ast }\right).$$

(16)

Based on this result, Ricotta and Szeidl [21] define the numbers equivalent quadratic entropy ${\widehat{Q}}_e$ as

$${\widehat{Q}}_e\left(\tilde{\mathbf{D}},\mathbf{p}\right)={\left(1-Q_1\left(\tilde{\mathbf{D}},\mathbf{p}\right)\right)}^{-1}.$$

(17)

This can be interpreted as the inverse Simpson concentration of an idealized categorical reference system whose average pairwise distance between states is equal to $Q_1\left(\tilde{\mathbf{D}},\mathbf{p}\right)$.

2.2.2. Functional Hill Numbers

Chiu and Chao [8] derived the Functional Hill Numbers, denoted $F_q$, based on a similar procedure to that of Ricotta and Szeidl [21]. However, whereas ${\widehat{Q}}_e$ uses a purely categorical system as the idealized reference, $F_q$ requires only that

$$Q_1\left(\mathbf{D},\mathbf{p}\right)=\sum _{i=1}^{n_{\ast }}\sum _{j=1}^{n_{\ast }}{Q}_1\left(\mathbf{D},\mathbf{p}\right)p_{\ast i}{p}_{\ast j}=\sum _{i=1}^{n_{\ast }}\sum _{j=1}^{n_{\ast }}{Q}_1\left(\mathbf{D},\mathbf{p}\right)n_{\ast }^{-2},$$

(18)

which means that the idealized reference system is one for which the between-state distance matrix is set to $Q_1\left(\mathbf{D},\mathbf{p}\right)$ everywhere (or to 0 along the leading diagonal and $Q_1\left(\mathbf{D},\mathbf{p}\right)n_{\ast }/(n_{\ast }-1)$ on the off diagonals).

Chiu and Chao [8] generalized Rao’s quadratic entropy to include the elasticity parameter $q\ge 0$

$$Q_q\left(\mathbf{D},\mathbf{p}\right)=\sum _{i=1}^n\sum _{j=1}^n{D}_{ij}{\left(p_i{p}_j\right)}^q,$$

(19)

and sought to find $n_{\ast }$ for the idealized reference system satisfying Equation (18) and the following:

$$Q_q\left(\mathbf{D},\mathbf{p}\right)=\sum _{i=1}^{n_{\ast }}\sum _{j=1}^{n_{\ast }}{Q}_1\left(\mathbf{D},\mathbf{p}\right){\left(\frac{1}{n_{\ast }}\frac{1}{n_{\ast }}\right)}^q.$$

(20)

Solving Equation (20) for $n_{\ast }$ yields the functional Hill numbers of order q:

$$F_q\left(\mathbf{D},\mathbf{p}\right)={\left(\frac{Q_q\left(\mathbf{D},\mathbf{p}\right)}{Q_1\left(\mathbf{D},\mathbf{p}\right)}\right)}^{\frac{1}{2(1-q)}}=n_{\ast },$$

(21)

which is the effective number of states in an idealized categorical reference system whose distance function is scaled by a factor of $Q_1\left(\mathbf{D},\mathbf{p}\right)n_{\ast }/(n_{\ast }-1)$.

2.2.3. Leinster–Cobbold Index

The index derived by Leinster and Cobbold [10], denoted $L_q$, is distinct from ${\widehat{Q}}_e$ and $F_q$ in two ways. First, for a given system X, the $L_q$ is not derived based on finding an idealized reference system $X_{\ast }$ whose average between-state dissimilarity is equal to that of X. Second, it does not use a dissimilarity matrix; rather, it uses a measure of similarity or affinity.

The Leinster–Cobbold index may be derived by simple extension of Equation (3). Assuming X has state space $\mathcal{X}=\{1,2,\dots ,n\}$ with probability distribution $\mathbf{p}={\left(p_i\right)}_{i=1,2,\dots ,n}$, we note that

$${\mathsf{\Pi }}_q\left(\mathbf{p}\right)={\left(\sum _{i=1}^n{p}_i^q\right)}^{\frac{1}{1-q}}={\left[\sum _{i=1}^n{p}_i{\left(\mathbf{I}\mathbf{p}\right)}_i^{q-1}\right]}^{\frac{1}{1-q}}.$$

(22)

Here, $\mathbf{I}$ is the $n\times n$ identity matrix representing the pairwise similarities between states in X. The Leinster–Cobbold index generalizes $\mathbf{I}$ to be any $n\times n$ similarity matrix $\mathbf{S}$, yielding the following formula:

$$L_q\left(\mathbf{S},\mathbf{p}\right)={\left[\sum _{i=1}^n{p}_i{\left(\sum _{j=1}^n{S}_{ij}{p}_j\right)}^{q-1}\right]}^{\frac{1}{1-q}}.$$

(23)

The similarity matrix can be obtained from a dissimilarity matrix by the transformation $S_{ij}=e^{-u{D}_{ij}}$, where $u\ge 0$ is a scaling factor. When $u=0$, then $\mathbf{S}$ is 1 everywhere. Conversely, when $u\to \infty$, then $\mathbf{S}$ approaches $\mathbf{I}$. The Leinster–Cobbold index can thus be interpreted as an effective number if the states are in an idealized reference system (i.e., one with uniform probabilities over states) whose topology is also governed by the similarity matrix $\mathbf{S}$.

2.2.4. Limitations of Existing Non-Categorical Heterogeneity Indices

We illustrate several limitations of the ${\widehat{Q}}_e$, $F_q$, and $L_q$ indices using a simple 3-state system X with event space $\mathcal{X}=\{1,2,3\}$ over which we specify a probability distribution

$$\mathbf{p}\left(\kappa \right)=\left\{\begin{array}{cc}{\left(1,0,0\right)}^{\top }\hfill & \kappa =0\hfill \\ {\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)}^{\top }\hfill & \kappa =1\hfill \\ {\left(0,0,1\right)}^{\top }\hfill & \kappa =\infty \hfill \\ {\left(\frac{1}{1+\sqrt{\kappa }+\kappa },\frac{\sqrt{\kappa }}{1+\sqrt{\kappa }+\kappa },\frac{\kappa }{1+\sqrt{\kappa }+\kappa }\right)}^{\top }\hfill & \mathrm{Otherwise}\hfill \end{array}\right.$$

(24)

where $0\le \kappa$ is a parameter that smoothly varies the level of inequality. When $\kappa =1$ the distribution is perfectly even (Figure 1A). Since an undirected graph of the system is arranged in a triangle with height h and base b, we also specify the following parametric distance matrix,

$$\mathbf{D}(h,b)=\left(\begin{array}{ccc}0& b& \sqrt{\frac{b^2}{4}+h^2}\\ b& 0& \sqrt{\frac{b^2}{4}+h^2}\\ \sqrt{\frac{b^2}{4}+h^2}& \sqrt{\frac{b^2}{4}+h^2}& 0\end{array}\right),$$

(25)

which allows us to smoothly vary the level of dissimilarity between states in X. Importantly, Equation (25) allows us to generate distance matrices that are either metric (when $h<b\sqrt{3}/2$; Definition 1) or ultrametric (when $h\ge b\sqrt{3}/2$; Definition 2). This is illustrated in Figure 1B.

Figure 1. Illustration of simple three-state system under which we compare existing non-categorical heterogeneity indices. Panel A depicts a three state system X as an undirected graph, with node sizes corresponding to state probabilities governed by Equation (24). As $0\le \kappa$ diverges further from $\kappa =1$, the probability distribution over states becomes more unequal. Panel B visually represents the parametric pairwise distance matrix $\mathbf{D}(h,b)$ of Equation (25) (h is height, b is base length, $D_{ij}$ is distance between states i and j). In the examples shown in Panels B and C, we set $b=1$. Specifically, we provide visual illustration of settings for which the distance function on X is a metric (Definition 1; when $h<b\sqrt{3}/2$) or ultrametric (Definition 2; when $h\ge b\sqrt{3}/2$). Panel C compares the numbers equivalent quadratic entropy (solid lines marked ${\widehat{Q}}_e$; Section 2.2.1), functional Hill numbers (at $q=1$, dashed lines marked $F_1$; Section 2.2.2), and the Leinster–Cobbold Index (at $q=1$, dotted lines marked $L_1$; Section 2.2.3) for reporting the heterogeneity of X. The y-axis reports the value of respective indices. The x-axis plots the height parameter for the distance matrix $\mathbf{D}(h,1)$ (Equation (25) and Panel B). The range of h at which $\mathbf{D}(h,1)$ is only a metric is depicted by the gray shaded background. The range of h shown with a white background is that for which $\mathbf{D}(h,1)$ is ultrametric. For each index, we plot values for a probability distribution over states that is perfectly even ($\kappa =1$; dotted markers) or skewed ($\kappa =10$; vertical line markers). Panel D shows the sensitivity of the Leinster–Cobbold index ($L_1$; y-axis) to the scaling parameter $0\le u$ (x-axis) used to transform a distance matrix into a similarity matrix ($S_{ij}=e^{-u{D}_{ij}}$). This is shown for three levels of skewness for the probability distribution over states (no skewness at $\kappa =1$, dotted markers; significant skewness at $\kappa =10$, vertical line markers; extreme skewness at $\kappa =100$, square markers).

Definition 1 (Metric distance).

A function $d:\mathcal{X}\times \mathcal{X}\to {\mathbb{R}}_{\ge 0}$ on a set $\mathcal{X}$ is a metric if and only if all of the following conditions are satisfied for all $(x,y,z)\in \mathcal{X}$:

1 

Non-negativity: $d(x,y)\ge 0$

2 

Identity of indiscernibles: $d(x,y)=0\iff x=y$

3 

Symmetry: $d(x,y)=d(y,x)$

4 

Triangle inequality: $d(x,z)\le d(x,y)+d(y,z)$

Definition 2 (Ultrametric distance).

A function $d:\mathcal{X}\times \mathcal{X}\to {\mathbb{R}}_{\ge 0}$ on a set $\mathcal{X}$ is ultrametric if and only if, for all $(x,y,z)\in \mathcal{X}$, criteria 1-3 for a metric are satisfied (Definition 1), in addition to the ultrametric triangle inequality:

$$d(x,z)\le max\left\{d(x,y),d(y,z)\right\}$$

(26)

Figure 1C compares the ${\widehat{Q}}_e,F_q$, and $L_q$ indices when applied to X across variation in between-state distances (via Equation (25)) and skewness in the probability distribution over states (Equation (24)). With respect to the numbers equivalent quadratic entropy (${\widehat{Q}}_e$; Section 2.2.1), we note that its behavior is categorically different with respect to whether the distance matrix is ultrametric. That is ${\widehat{Q}}_e$ increases with the triangle height parameter h (Equation (25)) until it passes the ultrametric threshold, after which it decreases monotonically with h. The behavior of ${\widehat{Q}}_e$ is sensible in the ultrametric range. When the distance matrix is scaled, as in Equation (12), pulling one of the three states in X further away from the remaining two should function similarly to progressively merging the latter states. Thus, the behavior of ${\widehat{Q}}_e$ is highly sensitive to whether a given distance matrix is ultrametric (which will often not be the case in real-world applications).

With respect to $F_q$, a notable benefit in comparison to ${\widehat{Q}}_e$ is that $F_q$ behaves consistently regardless of whether distance is ultrametric. However, Figure 1 shows other drawbacks. First, we can see that $F_q$ becomes insensitive to $\mathbf{D}(h,1)$ when $\mathbf{p}\left(\kappa \right)$ is perfectly even (shown analytically in Appendix A). Second, $F_q$ can paradoxically estimate a greater number of states than the theoretical maximum allows. That this occurs when the state probability distribution is more unequal violates the principle of transfers [20,33,39,40] (Section 2.1.1). This is made more problematic since Figure 1C shows it occurs when one state is being pushed closer to the others (i.e., with smaller values of h). To summarize, the functional Hill numbers are estimating more states than are really present despite the reduction in between-state distances and greater inequality in the probability mass function.

Figure 1C shows that the Leinster-Cobbold index compares favorably to $F_q$ because the former does not lose sensitivity to dissimilarity when $\mathbf{p}\left(\kappa \right)$ is perfectly even. However, Figure 1D shows that the Leinster-Cobbold index is particularly sensitive to the form of similarity transformation. In the present case, the maximal value of the $L_q$ gradually approaches 3 as u grows (and only when $u\to \infty$ does it reach 3), while progressively losing sensitivity to distance. As mentioned by Leinster and Cobbold [10], the choice of u or other similarity transformation is dependent on the importance assigned to functional differences between states. However, it is not clear how a given similarity transformation (e.g., u), and therefore the idealized reference system of $L_q$, should be validated.

Above all of the idiosyncratic limitations of existing numbers equivalent heterogeneity indices, we must highlight two basic assumptions they all share. First, they continue to assume that some valid and reliable categorical partitioning on X is known a priori. Second, they assume that a distance function specified a priori describes semantically relevant geometry of the system in question. These two limitations are not independent, since an unreliable categorical partitioning of the state space will lead to erroneous estimates of the pairwise distances between states. Thus, we seek an approach for measuring heterogeneity that has neither these limitations, nor those shown above to be specific to the other numbers equivalent heterogeneity indices for non-categorical systems.

3. Representational Rényi Heterogeneity

In this section, we propose an alternative approach to the indices of Section 2.2 that we call representational Rényi heterogeneity (RRH). It involves transforming X into a representation Z, defined on an unobservable or latent event space $\mathcal{Z}$, that satisfies two criteria:

• The representation Z captures the semantically relevant variation in X
• Rényi heterogeneity can be directly computed on Z

Satisfaction of the first criterion can only be ascertained in a domain-specific fashion. Since Z is essentially a model of X, investigators must justify that this model is appropriate for the scientific question at hand. For example, an investigator may evaluate the ability of X to be reconstructed from representation Z under cross-validation. The second criterion simply means that the transformation of $X\to Z$ must specify a probability distribution on $\mathcal{Z}$ upon which the Rényi heterogeneity can be directly computed.

Figure 2 illustrates the basic idea of RRH. However, the specifics of this framework differ based on the topology of the representation Z. Thus, the remainder of this section discusses the following approaches:

Figure 2. Graphical illustration of the two main approaches for computing representational Rényi heterogeneity. In both cases, we map sampled points on an observable space $\mathcal{X}$ onto a latent space $\mathcal{Z}$, upon which we apply the Rényi heterogeneity measure. The mapping is illustrated by the curved arrows, and should yield a posterior distribution over the latent space. Panel A shows the case in which the latent space is categorical (for example, discrete components of a mixture distribution on a continuous space). Panel B illustrates the case in which the latent space has non-categorical topology. A special case of the latter mapping may include probabilistic principal components analysis. When the latent space is continuous, we must derive a parametric form for the Rényi heterogeneity.

A.

Application of standard Rényi heterogeneity (Section 2.1) when Z is a categorical representation

B.

Deriving parametric forms for Rényi heterogeneity when Z is a non-categorical representation

3.1. Rényi Heterogeneity on Categorical Representations

Let X be a system defined on an observable space $\mathcal{X}$ that is non-categorical and $n_x$-dimensional. Consider the scenario in which the semantically relevant variation in X is categorical: for instance, images of different object categories stored in raw form as real-valued vectors. An investigator may be interested in measuring the effective number of states in X with respect to this categorical variation. This requires transforming X into a semantically relevant categorical representation Z upon which Equation (3) can be applied.

Assume we have a large random sample of N points $\mathbf{X}={\left({\mathbf{x}}_i\right)}_{i=1,2,\dots ,N}$ from system X. We can conceptualize each discrete observation ${\mathbf{x}}_i$ in this sample as the single point in the event space of a perfectly homogeneous subsystem $X_i$. When pooled, the subsystems ${\left\{X_i\right\}}_{i=1,2,\dots ,N}$ constitute X. The contribution weights of each subsystem to X as a whole are denoted $\mathbf{w}={\left(w_i\right)}_{i=1,2,\dots ,N}$, where ${\sum }_{i=1}^N{w}_i=1$ and $w_i\ge 0$.

We now specify a vector-valued function $\mathbf{f}:\mathcal{X}\to \mathcal{P}\left(\mathcal{Z}\right)$ such that $\mathbf{x}\mapsto \mathbf{f}\left(\mathbf{x}\right)={\left[f_j\left(\mathbf{x}\right)\right]}_{j=1,2,\dots ,n_z}$ is a mapping from $n_x$-dimensional coordinates on the observable space, $\mathbf{x}\in \mathcal{X}$, onto an $n_z$-dimensional discrete probability distribution over $\mathcal{Z}=\{1,2,\dots ,n_z\}$. Thus, $\mathbf{f}\left({\mathbf{x}}_i\right)$ can be conceptualized as mapping subsystem $X_i$ onto its categorical representation $Z_i$. After defining $\mathbf{f}$, the effective number of states in the latent representation of $X_i$ can be computed as

$${\mathsf{\Pi }}_q\left({\mathbf{x}}_i\right)={\left(\sum _{j=1}^{n_z}{f}_j^q\left({\mathbf{x}}_i\right)\right)}^{\frac{1}{1-q}}.$$

(27)

When ${\mathsf{\Pi }}_q\left({\mathbf{x}}_i\right)=1$, then $\mathbf{f}$ assigns $\mathbf{x}$ to a single category with perfect certainty. Conversely, when ${\mathsf{\Pi }}_q\left({\mathbf{x}}_i\right)=n_z$, then either ${\mathbf{x}}_i$ belongs to all categorical states with equal probability, or $\mathbf{f}$ is maximally uncertain about the mapping of point ${\mathbf{x}}_i$.

Mapping all points $\mathbf{X}$ onto the categorical latent space yields a collection of subsystems ${\left\{Z_i\right\}}_{i=1,2,\dots ,N}$, which generate Z when pooled. Using Equation (6), we can compute the effective number of total states in Z as the pooled heterogeneity:

$${\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)={\left[\sum _{j=1}^{n_z}{\left(\sum _{i=1}^N{w}_i{f}_j\left({\mathbf{x}}_i\right)\right)}^q\right]}^{\frac{1}{1-q}},$$

(28)

Unfortunately, ${\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)$ counts some heterogeneity that is due to uncertainty in the model (i.e., that quantified by Equation (27)). We, therefore, compute the effective number of states in Z per point $\mathbf{x}\in \mathcal{X}$ using the within-group heterogeneity formula (Equation (7)):

$${\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathbf{X},\mathbf{w}\right)={\left[\frac{{\sum }_{i=1}^N{w}_i^q\left({\sum }_{j=1}^{n_z}{f}_j^q\left({\mathbf{x}}_i\right)\right)}{{\sum }_{k=1}^N{w}_k^q}\right]}^{\frac{1}{1-q}}.$$

(29)

Finally, the effective number of states (points) in X—with respect to the categorical variation modeled by Z—can then be computed using the between-group heterogeneity formula (Equation (8)):

$${\mathsf{\Pi }}_q^{\mathrm{B}}\left(\mathbf{X},\mathbf{w}\right)=\frac{{\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)}{{\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathbf{X},\mathbf{w}\right)}.$$

(30)

Example 1 demonstrates that current methods of measuring biodiversity and wealth concentration can be viewed as special cases of categorical RRH.

Example 1 (Classical measurement of biodiversity and economic equality as categorical RRH).

Definitions necessary for this example are shown in Table 2. The traditional analysis of species diversity and economic equality can be recovered from an RRH-based formulation when $\mathbf{f}$ is assumed to be deterministic and $\mathbf{w}={\left(N^{-1}\right)}_{i=1,2,\dots ,N}$. In this case within-group heterogeneity can be shown to reduce to 1:

$$\begin{array}{cc}\hfill {\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathbf{X},\mathbf{w}\right)& ={\left[\sum _{i=1}^N\frac{N^{-q}}{{\sum }_{k=1}^N{N}^{-q}}\left(\sum _{j=1}^{n_z}{f}_j^q\left({\mathbf{x}}_i\right)\right)\right]}^{\frac{1}{1-q}}\hfill \\ \hfill & ={\left[\sum _{i=1}^N{N}^{-1}\left(1\right)\right]}^{\frac{1}{1-q}}\hfill \\ \hfill & =1.\hfill \end{array}$$

(31)

Table 2. Definitions in formulation of classical biodiversity and economic equality analysis as categorical representational Rényi heterogeneity. Superscripted indexing on $\mathbf{x}={\left(x_i\right)}^{i=1,\dots ,n_x}$ denotes that this is a row vector.

	Analytical Context
Symbol	Biodiversity	Economic Equality
X	Ecosystem, whose observation yields an organism denoted by vector $\mathbf{x}={\left(x_i\right)}^{i=1,\dots ,n_x}\in \mathcal{X}$	A system of resources, whose observation yields an asset denoted by vector $\mathbf{x}={\left(x_i\right)}^{i=1,\dots ,n_x}\in \mathcal{X}$
$\mathcal{X}\subseteq {\mathbb{R}}^{n_x}$	$n_x$-dimensional feature space of organisms in the ecosystem	$n_x$-dimensional feature space of assets in the economy, whose topology is such that the “economic” or monetary value is equal at each coordinate $\mathbf{x}\in \mathcal{X}$
$\mathcal{Z}=\left\{\mathbf{z}\in {\left\{0,1\right\}}^{n_z}:{\sum }_{i=1}^{n_z}{z}_i=1\right\}$	$n_z$-dimensional space of one-hot species labels	$n_z$-dimensional space of one-hot labels over wealth-owning agents
$\mathbf{f}:\mathcal{X}\to \mathcal{P}\left(\mathcal{Z}\right)$	A model that performs the mapping $\mathbf{x}\mapsto \mathbf{f}\left(\mathbf{x}\right)$ of organisms to discrete probability distributions over $\mathcal{Z}$	A model that performs the mapping $\mathbf{x}\mapsto \mathbf{f}\left(\mathbf{x}\right)$ of assets to discrete probability distributions over $\mathcal{Z}$
$N_i\in {\mathbb{N}}_{+}$	The number of organisms observed belonging to species $i\in \left\{1,\dots ,n_z\right\}$	The number of equal valued assets belonging to agent $i\in \left\{1,\dots ,n_z\right\}$
$N={\sum }_{i=1}^{n_z}{N}_i$	The total number of organisms observed	The total quantity of assets observed
$\mathbf{X}={\left(x_{ij}\right)}_{i=1,\dots ,N}^{j=1,\dots ,n_x}$	A sample of N organisms	A sample of N assets
$\mathbf{w}={\left(w_i\right)}_{i=1,\dots ,N}$	Sample weights, such that $w_i\ge 0$ and ${\sum }_{i=1}^N{w}_i=1$

Thus, we have

$$\begin{array}{cc}\hfill {\mathsf{\Pi }}_q^{\mathrm{B}}\left(\mathbf{X},\mathbf{w}\right)& ={\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)\hfill \\ \hfill & ={\left[\sum _{j=1}^{n_z}{\left(\sum _{i=1}^N{N}^{-1}{f}_j\left({\mathbf{x}}_i\right)\right)}^q\right]}^{\frac{1}{1-q}}\hfill \\ \hfill & ={\left[\sum _{j=1}^{n_z}{\left(\frac{N_j}{N}\right)}^q\right]}^{\frac{1}{1-q}},\hfill \end{array}$$

(32)

which yields the categorical Rényi heterogeneity (Hill numbers for biodiversity analysis and Hannah–Kay indices in the economic setting [19,20]), and by extension many diversity indices to which it is connected (Table 1). Thus, traditional analysis of species biodiversity and economic equality are special cases of representational Rényi heterogeneity where the representation is specified by a mapping onto degenerate distributions over categorical labels. The only differences lie in the definition of observable and latent spaces, and the representational models.

In the case of biodiversity analysis, the model $\mathbf{f}$ in real-world practice may simply be a human expert assigning species labels to a sample of organisms from a field study. In the economic setting, one may speculate that $\mathbf{f}$ would essentially reduce to contracts specifying ownership of assets, whose value is deemed by market forces.

3.2. Rényi Heterogeneity on Non-Categorical Representations

In Section 3.1, we dealt with instances in which semantically relevant variation in X is categorical, such as when object categories are embedded in images stored as real-valued vectors. Here, we consider scenarios in which the semantically relevant information in an observable system X is non-categorical: for instance, where a piece of text contains information about semantic concepts best represented as real-valued “word vectors” [44,45]. Measuring the effective number of distinct states in X with respect to this continuous variation requires transforming X into a semantically relevant continuous representation Z upon which procedures analogous to those of Section 3.1 may be undertaken.

Let Z be defined on an $n_z$-dimensional event space $\mathcal{Z}\subseteq {\mathbb{R}}^{n_z}$ over which there exists a family of parametric probability distributions $\mathcal{P}\left(\mathcal{Z}\right)$ of a form chosen by the experimenter. Let $f:\mathcal{X}\to \mathcal{P}\left(\mathcal{Z}\right)$ be a model that performs the mapping $\mathbf{x}\mapsto f(·|\mathbf{x})$ from a point $\mathbf{x}\in \mathcal{X}$ on the observable space to a probability density on $\mathcal{Z}$. For example, if $\mathcal{P}\left(\mathcal{Z}\right)$ is the family of multivariate Gaussians, then $f\left(\mathbf{z}\right|{\mathbf{x}}_i)=\mathcal{N}\left(\mathbf{z}\right|{\mu }_i,{\Sigma }_i)$, where ${\mu }_i$ and ${\Sigma }_i$ are the Gaussian mean and covariance functions at ${\mathbf{x}}_i$, respectively. Given a sample $\mathbf{X}={\left({\mathbf{x}}_i\right)}_{i=1,2,\dots ,N}$, as in Section 3.1, we compute the continuous analogue of Equation (27) as follows

$${\mathsf{\Pi }}_q\left({\mathbf{x}}_i\right)={\left({\int }_{\mathcal{Z}}{f}^q\left(\mathbf{z}\right|{\mathbf{x}}_i)\mathrm{d}\mathbf{z}\right)}^{\frac{1}{1-q}}.$$

(33)

This formula yields the effective size of the domain of a uniform distribution on ${\mathbb{R}}^{n_z}$ whose Rényi heterogeneity is equal to ${\mathsf{\Pi }}_q\left({\mathbf{x}}_i\right)$ (proof is given in Appendix A). Thus, it is possible for ${\mathsf{\Pi }}_q\left({\mathbf{x}}_i\right)$ to be less than 1, though it will remain non-negative.

Similar to the procedure in Section 3.1, we now define a continuous version of the within-observation heterogeneity

$${\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathbf{X},\mathbf{w}\right)={\left[\sum _{i=1}^N\frac{w_i^q}{{\sum }_{j=1}^N{w}_j^q}{\int }_{\mathcal{Z}}{f}^q\left(\mathbf{z}\right|{\mathbf{x}}_i)\mathrm{d}\mathbf{z}\right]}^{\frac{1}{1-q}},$$

(34)

which estimates the effective size of the latent space occupied per observable point $\mathbf{x}\in \mathcal{X}$.

In order to compute the pooled heterogeneity ${\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)$, the experimenter must specify the form of the pooled distribution, here denoted ${\overline{f}}_{\mathbf{w}}$. The conceptually most simple approach is non-parametric, using a model average,

$${\overline{f}}_{\mathbf{w}}\left(\mathbf{z}|\mathbf{X}\right)=\sum _{i=1}^N{w}_{i}f\left(\mathbf{z}\right|{\mathbf{x}}_i),$$

(35)

whereby the pooled heterogeneity would be

$${\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)={\left[{\int }_{\mathcal{Z}}{\left(\sum _{i=1}^N{w}_{i}f\left(\mathbf{z}\right|{\mathbf{x}}_i)\right)}^q\mathrm{d}\mathbf{z}\right]}^{\frac{1}{1-q}}.$$

(36)

The integral in Equation (36) may often be analytically intractable and potentially difficult to solve accurately in high dimensions with numerical methods. Furthermore, some areas of $\mathcal{Z}$ may be assigned low probability by $f\left(\mathbf{z}\right|{\mathbf{x}}_i)$ for all $i\in \{1,2,\dots ,N\}$. This is not a problem as the sample $\mathbf{X}$ becomes infinitely large. However, with finite samples, it may be the case that some representational states in $\mathcal{Z}$ are unlikely simply because we have not sampled from the corresponding regions of $\mathcal{X}$. An alternative to Equation (35) is therefore to specify a parametric pooled distribution

$${\overline{f}}_{\mathbf{w}}\left(·|\mathbf{X}\right)={\Xi }_f\left(\mathbf{X},\mathbf{w}\right),$$

(37)

where ${\Xi }_f$ is a deterministic function that combines $f(·|{\mathbf{x}}_i)$ for $i\in \{1,2,\dots ,N\}$ into a valid probability density on $\mathcal{Z}$. In this case, the pooled Rényi heterogeneity is simply

$${\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)={\left({\int }_{\mathcal{Z}}{\overline{f}}_{\mathbf{w}}^q\left(\mathbf{z}\right|\mathbf{X})\mathrm{d}\mathbf{z}\right)}^{\frac{1}{1-q}}.$$

(38)

Using either Equation (36) or (38) as the pooled heterogeneity and Equation (34) as the within-group heterogeneity, the effective number of distinct states in X—with respect to the non-categorical representation Z—can then be computed using Equation (30).

Figure 3 demonstrates the difference between the parametric and non-parametric approaches to pooling for non-categorical RRH, and Example 2 demonstrates one approach to parametric pooling for a mixture of multivariate Gaussians.

Figure 3. Illustration of approaches to computing the pooled distribution on a simple representational space $\mathcal{Z}=\mathbb{R}$. In this example, two points on the observable space, $({\mathbf{x}}_1,{\mathbf{x}}_2)\in \mathcal{X}$, are mapped onto the latent space via model $f(·|{\mathbf{x}}_i)$ for $i\in \{1,2\}$, which indexes univariate Gaussians over $\mathcal{Z}$ (depicted as hatched patterns for ${\mathbf{x}}_1$ and ${\mathbf{x}}_2$, respectively). A pooled distribution computed non-parametrically by model-averaging (Equation (35)) is depicted as the solid black line. The parametrically pooled distribution (see Example 2) is depicted as the dashed black line. The parametric approach implies the assumption that further samples from $\mathcal{X}$ would yield latent space projections in some regions assigned low probability by $f\left(z\right|{\mathbf{x}}_1)$ and $f\left(z\right|{\mathbf{x}}_2)$.

Example 2 (Parametric pooling of multivariate Gaussian distributions).

Let $\mathbf{X}={\left({\mathbf{x}}_i\right)}_{i=1,2,\dots ,N}$ be a sample of $n_x$-dimensional vectors from a system X with event space $\mathcal{X}\subseteq {\mathbb{R}}^{n_x}$. Let Z be a latent representation of X with $n_z$-dimensional event space $\mathcal{Z}={\mathbb{R}}^{n_z}$. Let

$$f\left(\mathbf{z}\right|{\mathbf{x}}_i)=\mathcal{N}\left(\mathbf{z}|{\mathit{\mu }}_i,{\Sigma }_i\right)$$

(39)

be a model that returns a multivariate Gaussian density with mean ${\mu }_i$ and covariance ${\Sigma }_i$ given point ${\mathbf{x}}_i\in \mathcal{X}$. Finally, let $\mathbf{w}={\left(w_i\right)}_{i=1,2,\dots ,N}$ be weights assigned to each sample in $\mathbf{X}$ such that $w_i\ge 0$ and ${\sum }_{i=1}^N{w}_i=1$.

If one assumes that the pooled distribution over $\mathcal{Z}$ given the set of components $f\left(\mathbf{z}\right|{\mathbf{x}}_1),f\left(\mathbf{z}\right|{\mathbf{x}}_2),\dots ,f\left(\mathbf{z}\right|{\mathbf{x}}_N)$ is itself a multivariate Gaussian,

$${\overline{f}}_{\mathbf{w}}\left(\mathbf{z}|\mathbf{X}\right)=\mathcal{N}\left(\mathbf{z}\right|{\mathit{\mu }}_{\ast },{\mathsf{\Sigma }}_{\ast })$$

(40)

with $n_z\times 1$ pooled mean,

$${\mathit{\mu }}_{\ast }=\sum _{i=1}^N{w}_i{\mathit{\mu }}_i$$

(41)

and $n_z\times n_z$ pooled covariance matrix

$${\Sigma }_{\ast }=-{\mathit{\mu }}_{\ast }{\mathit{\mu }}_{\ast }^{\top }+\sum _{i=1}^N{w}_i\left[{\Sigma }_i+{\mathit{\mu }}_i{\mathit{\mu }}_i^{\top }\right],$$

(42)

then the pooled heterogeneity ${\mathsf{\Pi }}_q^{\mathrm{P}}$ is therefore simply the Rényi heterogeneity of a multivariate Gaussian,

$${\mathsf{\Pi }}_q\left(\Sigma \right)=\left\{\begin{array}{cc}\mathrm{Undefined}\hfill & q=0\hfill \\ {\left(2\pi e\right)}^{\frac{n_z}{2}}\sqrt{\left|\Sigma \right|}\hfill & q=1\hfill \\ {\left(2\pi \right)}^{\frac{n_z}{2}}\sqrt{\left|\Sigma \right|}\hfill & q=\infty \hfill \\ {\left(2\pi \right)}^{\frac{n_z}{2}}{q}^{\frac{n_z}{2(q-1)}}\sqrt{\left|\Sigma \right|}\hfill & \mathrm{Otherwise}\hfill \end{array}\right.$$

(43)

evaluated at ${\Sigma }_{\ast }$. The derivation is provided in Appendix A [46]. Equation (43) at ${\Sigma }_{\ast }$ is interpreted as the effective size of space $\mathcal{Z}$ occupied by the complete latent representation of X under model f.

The within-group heterogeneity can be obtained for the set of components ${\left[f\left(\mathbf{z}\right|{\mathbf{x}}_i)\right]}_{i=1,2,\dots ,N}$ by solving Equation (34) for the Gaussian densities, yielding:

$${\mathsf{\Pi }}_q^{\mathrm{W}}\left({\Sigma }_{1:N},\mathbf{w}\right)=\left\{\begin{array}{cc}\mathrm{Undefined}\hfill & q=0\hfill \\ exp\left\{\frac{1}{2}\left(n_z+\sum _{i=1}^N{w}_{i}log\left|2\pi {\Sigma }_i\right|\right)\right\}\hfill & q=1\hfill \\ 0\hfill & q=\infty \hfill \\ {\left(2\pi \right)}^{\frac{n_z}{2}}{\left(\sum _{i=1}^N\frac{{\overline{w}}_i^q{\left|{\Sigma }_i\right|}^{\frac{1}{2}}}{q^{\frac{n_z}{2}}}\right)}^{\frac{1}{1-q}}\hfill & \mathrm{Otherwise}\hfill \end{array}\right.,$$

(44)

where we denote ${\Sigma }_{1:N}={\left\{{\Sigma }_i\right\}}_{i=1,2,\dots ,N}$ for parsimony, and ${\overline{w}}_i=w_i{\left({\sum }_{j=1}^N{w}_j^q\right)}^{-1/q}$. Equation (44) estimates the effective size of the $n_z$-dimensional representational space occupied per state $\mathbf{x}\in \mathcal{X}$.

The effective number of states in X with respect to the continuous representation Z is thus the between-group heterogeneity ${\mathsf{\Pi }}_q^{\mathrm{B}}$ which can be computed as the ratio ${\mathsf{\Pi }}_q\left({\Sigma }_{\ast }\right)/{\mathsf{\Pi }}_q^{\mathrm{W}}\left({\Sigma }_{1:N},\mathbf{w}\right)$. The properties of this decomposition—specifically the conditions under which ${\mathsf{\Pi }}_q^{\mathrm{B}}\ge 1$ (Lande’s requirement [28,42])—are discussed further elsewhere [46].

4. Empirical Applications of Representational Rényi Heterogeneity

In this section, we demonstrate two applications of RRH under assumptions of categorical (Section 4.1) and continuous (Section 4.2) latent spaces. First, Section 4.1, uses a simple closed-form system consisting of a mixture of two beta distributions on the (0,1) interval to give exact comparisons of the behavior of RRH against that of existing non-categorical heterogeneity indices (Section 2.2). This experiment provides evidence that existing non-categorical heterogeneity indices can demonstrate counterintuitive behavior under various circumstances. Second, Section 4.2 demonstrates that RRH can yield heterogeneity measurements that are sensible and tractably computed, even for highly complex mappings $f:\mathcal{X}\to \mathcal{P}\left(\mathcal{Z}\right)$. There, we use a deep neural network to compute the effective number of observations in a database of handwritten images with respect to compressed latent representations on a continuous space.

4.1. Comparison of Heterogeneity Indices Under a Mixture of Beta Distributions

Consider a system X with event space $\mathcal{X}$ on the open interval $(0,1)$, containing an embedded, unobservable, categorical structure represented by the latent system Z with event space $\mathcal{Z}=\left\{1,2\right\}$. The systems’ collective behavior is governed by the joint distribution of a beta mixture model (BMM),

$$p(x,z)=𝟙[z=1](1-{\theta }_1){\mathrm{Beta}}_{{\theta }_2,{\theta }_3}\left(x\right)+𝟙[z=2]{\theta }_1{\mathrm{Beta}}_{{\theta }_3,{\theta }_2}\left(x\right),$$

(45)

where ${\mathrm{Beta}}_{\alpha ,\beta }\left(x\right)$ is the probability density function for a beta distribution with shape parameters $\alpha ,\beta$, and $\mathit{\theta }=\left({\theta }_1,{\theta }_2,{\theta }_3\right)$ are parameters. The indicator function $\mathbb{1}[·]$ evaluates to 1 if its argument is true, and to 0 otherwise. The prior distribution is

$$p\left(z\right)=𝟙[z=1](1-{\theta }_1)+𝟙[z=2]{\theta }_1,$$

(46)

and marginal probability of observable data is as follows (see Figure 4 for illustrations):

$$p\left(x\right)=(1-{\theta }_1){\mathrm{Beta}}_{{\theta }_2,{\theta }_3}\left(x\right)+{\theta }_1{\mathrm{Beta}}_{{\theta }_3,{\theta }_2}\left(x\right).$$

(47)

Figure 4. Demonstration of data-generating distribution (top row; Equations (45)–(47)), and relationship between the representational model’s decision threshold (Equations (48) and (50)) and categorical representational Rényi heterogeneity (bottom row). The optimal decision boundary (Equation (50)) is shown as a gray vertical dashed line in all plots. Each column depicts a specific parameterization of the data-generating system (parameters are stated above the top row). Top Row: Probability density functions for data-generating distributions. Shaded regions correspond to the two mixture components. Solid black lines denote the marginal distribution (Equation (47)). The x-axis represents the observable domain, which is the (0,1) interval. Bottom Row: Effect of varying categorical representational Rényi heterogeneity (RRH) for $q\in \{1,2,\infty \}$ across different category assignment thresholds for the beta-mixture models shown in the top row. Varying levels of decision boundary are plotted on the x-axis. The y-axis shows the resulting between-observation RRH. Black dots highlight the RRH computed at the optimal decision boundary.

To facilitate exact comparisons between heterogeneity indices, below, let us assume we have a model $f:\mathcal{X}\to \mathcal{P}\left(\mathcal{Z}\right)$ that maps an observation $x\in \mathcal{X}$ onto a degenerate distribution over $\mathcal{Z}$:

$$f_{\theta }\left(z\right|x)=𝟙[z=1]𝟙[x\le \tau \left(\theta \right)]+𝟙[z=2]𝟙[x>\tau \left(\theta \right)].$$

(48)

The subscripting of $f_{\theta }$ denotes that the model is optimized such that the threshold $0\le \tau \left(\theta \right)\le 1$ is the solution to

$$p\left(z=1|x=\tau (\mathit{\theta })\right)=p\left(z=2|x=\tau (\mathit{\theta })\right),$$

(49)

which is

$$\tau \left(\mathit{\theta }\right)=\left\{\begin{array}{cc}{\left[{\left({\theta }_1^{-1}-1\right)}^{\frac{1}{2({\theta }_2-{\theta }_3)}}{\left(1-{\theta }_1\right)}^{\frac{1}{2({\theta }_2-{\theta }_3)}}{\theta }_1^{-\frac{1}{2({\theta }_2-{\theta }_3)}}+1\right]}^{-1}\hfill & {\theta }_2-{\theta }_3\ne 0\hfill \\ 0\hfill & \left(({\theta }_2={\theta }_3)\wedge ({\theta }_1>\frac{1}{2})\right)\hfill \\ 1\hfill & \mathrm{Otherwise}\hfill \end{array}\right.$$

(50)

Under this model, the categorical RRH at point $x\in \mathcal{X}$ is

$${\mathsf{\Pi }}_q\left(x\right)={\left(\sum _{i=1}^2{f}_{\theta }^q\left(z=i|x\right)\right)}^{\frac{1}{1-q}}={\left({𝟙}^q\left[x\le \tau \left(\mathit{\theta }\right)\right]+{𝟙}^q\left[x>\tau \left(\mathit{\theta }\right)\right]\right)}^{\frac{1}{1-q}}=1.$$

(51)

The expected value of $f_{\theta }(z=2|x)$ with respect to the data generating distribution (Equation (47)) is

$$\begin{array}{cc}\hfill {\overline{f}}_{\theta }(z=2)& ={\mathbb{E}}_{x\sim p\left(x\right)}\left[f_{\theta }(z=2|x)\right]\hfill \\ \hfill & ={\int }_0^{1}p\left(x\right)𝟙\left[x>\tau \left(\mathit{\theta }\right)\right]\mathrm{d}x\hfill \\ \hfill & ={\int }_{\tau \left(\theta \right)}^{1}p\left(x\right)\mathrm{d}x\hfill \\ \hfill & =(1-{\theta }_1)I_x^1\left({\theta }_2,{\theta }_3\right)+{\theta }_1{I}_x^1\left({\theta }_3,{\theta }_2\right),\hfill \end{array}$$

(52)

where $I_{x_0}^{x_1}(a,b)$ is the generalized regularized incomplete beta function (BetaRegularized[$x_0,x_1,a,b$] command in the Wolfram language and betainc($a,b,x_0,x_1$,regularized=True) in Python’s mpmath package). Equation (52) implies that ${\overline{f}}_{\theta }(z=1)=1-{\overline{f}}_{\theta }(z=2)$. The pooled heterogeneity is thus expressed as a function of $\theta$ as follows:

$${\mathsf{\Pi }}_q^{\mathrm{P}}\left(\theta \right)=\left\{\begin{array}{cc}\sum _{i=1}^2𝟙[{\overline{f}}_{\theta }(z=i)>0]\hfill & q=0\hfill \\ exp\left\{-\sum _{i=1}^2{\overline{f}}_{\theta }(z=i)log{\overline{f}}_{\theta }(z=i)\right\}\hfill & q=1\hfill \\ {\left(\underset{i}{max}{\overline{f}}_{\theta }(z=i)\right)}^{-1}\hfill & q=\infty \hfill \\ {\left(\sum _{i=1}^2{\overline{f}}_{\theta }^q(z=i)\right)}^{\frac{1}{1-q}}\hfill & \mathrm{Otherwise}\hfill \end{array}\right..$$

(53)

As a function of $\theta$, the within-group heterogeneity is

$$\begin{array}{cc}\hfill {\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathit{\theta }\right)& ={\left[{\int }_0^1\frac{p^q\left(x\right)}{{\int }_0^1{p}^q\left(u\right)\mathrm{d}u}{\left(\sum _{i=1}^2{f}_{\theta }(z=i|x)\right)}^q\mathrm{d}x\right]}^{\frac{1}{1-q}}\hfill \\ \hfill & ={\left[{\int }_0^1\frac{p^q\left(x\right)}{{\int }_0^1{p}^q\left(u\right)\mathrm{d}u}\left(1\right)\mathrm{d}x\right]}^{\frac{1}{1-q}}\hfill \\ \hfill & =1,\hfill \end{array}$$

(54)

and therefore the between-group heterogeneity is ${\mathsf{\Pi }}_q^{\mathrm{B}}\left(\mathit{\theta }\right)={\mathsf{\Pi }}_q^{\mathrm{P}}\left(\mathit{\theta }\right)$.

Analytic expressions for the existing non-categorical heterogeneity indices ${\widehat{Q}}_e$ (Equation (17)), $F_q$ (Equation (21)), and $L_q$ (Equation (23)) were computed as “best-case” scenarios, as follows. First, the probability distributions over states for all expressions was the true prior distribution (Equation (46)). Distance matrices—and by extension, the similarity matrix for $L_q$—were computed using the closed-form expectation of the absolute distance between two beta-distributed random variables (see Appendix B and the Supplementary Materials).

Figure 5 compares the categorical RRH against ${\widehat{Q}}_e$, $F_q$, and $L_q$ for BMM distributions of varying degrees of separation, and across different mixture component weights ($0.5\le {\theta }_1<1$). Without significant loss of generality, we show only those comparisons at $q=1$ (which excludes the numbers equivalent quadratic entropy), and $q=2$.

Figure 5. Comparison of categorical representational Rényi heterogeneity (${\mathsf{\Pi }}_q$), the functional Hill numbers ($F_q$), the numbers equivalent quadratic entropy (${\widehat{Q}}_e$), and the Leinster–Cobbold index ($L_q$) within the beta mixture model. Each row of plots corresponds to a given separation between the beta mixture components. Column 1 illustrates the beta mixture distributions upon which indices were compared. The x-axis plots the domain of the distribution (open interval between 0 and 1). The y-axis shows the corresponding probability density. Different line styles in Column 1 provides visual examples of the effect of changing the ${\theta }_1$ parameter over the range [0.5,1]. Column 2 compares ${\mathsf{\Pi }}_q$ (solid line), $F_q$ (dashed line), and $L_q$ (dotted line), each at elasticity $q=1$. The x-axis shows the value of the $0.5\le {\theta }_1<1$ parameter at which the indices were compared. Index values are plotted along the y-axis. Column 3 compares the indices shown in Column 2, as well as ${\widehat{Q}}_e$ (dot-dashed line).

The most salient differences between these indices occur when the BMM mixture components completely overlap (i.e., at ${\theta }_2={\theta }_3$). The RRH correctly identifies that there is effectively only one component, regardless of mixture weights. Only the Leinster–Cobbold index showed invariance to the mixture weights when ${\theta }_2={\theta }_3$, but it could not correctly identify that data were effectively unimodal.

The other stark difference arose when the mixture components were furthest apart (here when ${\theta }_2=5$ and ${\theta }_3=20$). At this setting, the functional Hill numbers showed a paradoxical increase in the heterogeneity estimate as the prior distribution on components was skewed. The Leinster–Cobbold index was appropriately concave throughout the range of prior weights, but it never reached a value of 2 at its peak (as expected based on the predictions outlined in Section 2.2.3). Conversely, the RRH was always concave and reached a peak of 2 when both mixture components were equally probable.

4.2. Representational Rényi Heterogeneity is Scalable to Deep Learning Models

In this example, the observable system X is that of images of handwritten digits defined on an event space $\mathcal{X}={[0,1]}^{784}$ of dimension $n_x=784$ (the black and white images are flattened from $28\times 28$ pixel matrices into 784-dimensional vectors). Our sample $\mathbf{X}={\left(x_{ij}\right)}_{i=1,2,\dots ,N}^{j=1,2,\dots ,784}$ from this space is the familiar MNIST training dataset [22] (Figure 6), which consists of $N=60,000$ images roughly evenly distributed across digits $\{0,1,\dots ,9\}$, and where approximately 10% of all images come from each class. We assume each image carries equal importance, given by a weight vector $\mathbf{w}={\left(N^{-1}\right)}_{i=1,2,\dots ,N}$. We are interested in measuring the heterogeneity of X with respect to a continuous latent representation Z defined on event space $\mathcal{Z}={\mathbb{R}}^2$. In the present example, this space is simply the continuous 2-dimensional compression of an image that best facilitates its reconstruction. We choose a dimensionality of $n_z=2$ for the latent space in order to facilitate a pedagogically useful visualization of the latent feature representation, below. Unlike Section 4.1, in the present case we have no explicit representation of the true marginal distribution over the data, $p\left(\mathbf{x}\right)$.

Figure 6. Sample images from the MNIST dataset [22].

Having defined the observable and latent spaces, measuring RRH now requires defining a model $f:\mathcal{X}\to \mathcal{P}\left(\mathcal{Z}\right)$ that maps a (flattened) image vector ${\mathbf{x}}_i\in \mathcal{X}$ onto a probability distribution over the latent space. Our chosen model is the encoder module of a pre-trained convolutional variational autoencoder (cVAE) provided by the (https://colab.research.google.com/github/smartgeometry-ucl/dl4g/blob/master/variational_autoencoder.ipynb, Smart Geometry Processing Group at University College London) (Figure 7) [23,24]:

$$f_{\varphi }\left(\mathbf{z}\right|{\mathbf{x}}_i)=\mathcal{N}\left(\mathbf{z}|\mathbf{m}\left({\mathbf{x}}_i\right),\mathbf{C}\left({\mathbf{x}}_i\right)\right)$$

(55)

where $\varphi$ are the encoder’s parameters, which specify a convolutional neural network (CNN) whose output layer returns a $2\times 1$ mean vector $\mathbf{m}\left({\mathbf{x}}_i\right)$ and a $2\times 1$ log-variance vector $\mathbf{s}\left({\mathbf{x}}_i\right)$ given ${\mathbf{x}}_i$. For simplicity, we denote the latter as the $2\times 2$ diagonal covariance matrix $\mathbf{C}\left({\mathbf{x}}_i\right)={\left(e^{s_j\left({\mathbf{x}}_i\right)}{\delta }_{jk}\right)}_{j=1,2}^{k=1,2}$. Further details of the cVAE and its training can be found in Kingma and Welling [23,24], although the specific implementation in this paper was a pre-trained implementation by the (https://colab.research.google.com/github/smartgeometry-ucl/dl4g/blob/master/variational_autoencoder.ipynb, Smart Geometry Processing Group at University College London). Briefly, the cVAE learns to generate a compressed latent representation (via encoder $f_{\varphi }$, which is an approximate posterior distribution) that contains enough information about the input ${\mathbf{x}}_i$ to facilitate its reconstruction by a “decoder” module. The objective function is a lower bound on the model evidence $p\left(\mathbf{x}\right)$, which if maximized is equivalent to minimizing the Kullback–Leibler divergence between the approximate and true (but unknown) posteriors $f_{\varphi }$ and $p\left(\mathbf{z}\right|\mathbf{x})$, respectively.

Figure 7. Panel A: Illustration of the convolutional variational autoencoder (cVAE) [23]. The computational graph is depicted from top to bottom. An n_x-dimensional input data X_i (white rectangle) is passed through an encoder (in our experiment this is a convolutional neural network, CNN) which parameterizes an n_z-dimensional multivariate Gaussian over the coordinates z_i for the image’s embedding on the latent space $\mathcal{Z}={\mathbb{R}}^2$. The latent embedding can then be passed through a decoder (blue rectangle) which is a neural network employing transposed convolutions (here denoted CNN^⊤) to yield a reconstruction ${\hat{X}}_i$ of the original input data. The objective function for this network is a variational lower bound on the model evidence of the input data (see Kingma and Welling [23] for details). Panel B: Depiction of the latent space learned by the cVAE. This model was a pre-trained model from the (https://colab.research.google.com/github/smartgeometry-ucl/dl4g/blob/master/variational_autoencoder.ipynb, Smart Geometry Processing Group at University College London).

The continuous RRH under the model in Equation (55) for a single example ${\mathbf{x}}_i\in \mathcal{X}$ can be computed by merely evaluating the Rényi heterogeneity of a multivariate Gaussian (Equation (43) in Example 2) for the covariance matrix given by $\mathbf{C}\left({\mathbf{x}}_i\right)$. This is interpreted as the effective area of the 2-dimensional latent space consumed by representation of ${\mathbf{x}}_i$.

Since the handwritten digit images belong to groups of “Zeros, Ones, Twos, …, Nines,” this section will call the quantity ${\mathsf{\Pi }}_q^{\mathrm{W}}$ the within-observation heterogeneity (rather than the “within-group” heterogeneity) in order to avoid its interpretation as measuring the heterogeneity of a group of digits. Rather, it is interpreted as the effective area of latent space consumed by representation of a single observation $\mathbf{x}\in \mathcal{X}$ on average. It is computed by evaluation of Equation (44) at $\mathbf{C}\left(\mathbf{X}\right)={\left\{\mathbf{C}\left({\mathbf{x}}_i\right)\right\}}_{i=1,2,\dots ,N}$, given uniform weights on samples.

Finally, to compute the pooled heterogeneity ${\mathsf{\Pi }}_q^{\mathrm{P}}$, we use the parametric pooling approach detailed in Example 2, wherein the pooled distribution is a multivariate Gaussian with mean and covariance given by Equations (41) and (42), respectively. The pooled heterogeneity is then merely Equation (43) evaluated at ${\mathbf{C}}_{\ast }\left(\mathbf{X}\right)$, and represents the total amount of area in the latent space consumed by the representation of X under $f_{\varphi }$. The effective number of observations in X with respect to the continuous latent representation Z is, therefore, given by the between-observation heterogeneity:

$${\mathsf{\Pi }}_q^{\mathrm{B}}\left(\mathbf{C}\left(\mathbf{X}\right),\mathbf{w}\right)=\frac{{\mathsf{\Pi }}_q^{\mathrm{P}}\left({\mathbf{C}}_{\ast }\left(\mathbf{X}\right)\right)}{{\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathbf{C}\left(\mathbf{X}\right),\mathbf{w}\right)}.$$

(56)

Equation (56) gives the effective number of observations in X because it uses the entire sample $\mathbf{X}$ (of course, assuming $\mathbf{X}$ provides adequate coverage of the observable event space). However, one could compute the effective number of observations in a subset of $\mathbf{X}$, if necessary. Let ${\mathbf{X}}^{\left(j\right)}={\left({\mathbf{x}}_k\right)}_{k=1,2,\dots ,N_j}$ be the subset of $N_j$ points in $\mathbf{X}$ found in the observable subspace ${\mathcal{X}}_j\subset \mathcal{X}$ (such as the subspace of MNIST digits corresponding to a given digit class). Given corresponding weights ${\mathbf{w}}^{\left(j\right)}={\left(N_j^{-1}\right)}_{k=1,2,\dots ,N_j}$, Equation (56) is then simply

$${\mathsf{\Pi }}_q^{\mathrm{B}}\left(\mathbf{C}\left({\mathbf{X}}^{\left(j\right)}\right),{\mathbf{w}}^{\left(j\right)}\right)=\frac{{\mathsf{\Pi }}_q^{\mathrm{P}}\left({\mathbf{C}}_{\ast }\left({\mathbf{X}}^{\left(j\right)}\right)\right)}{{\mathsf{\Pi }}_q^{\mathrm{W}}\left(\mathbf{C}\left(\mathbf{X}\right),{\mathbf{w}}^{\left(j\right)}\right)}.$$

(57)

Figure 8 shows the effective number of observations in the subsets of MNIST images belonging to each image class, under the continuous representation learned by the cVAE. One can appreciate that the MNIST class of “Ones” (in the training set) has the smallest effective number of observations. Subjective visual inspection of the MNIST samples in Figure 6 may suggest that the Ones are indeed relatively more homogeneous as a group than the other digits (this claim is given further objective support in Appendix C based on deep similarity metric learning [47,48]).

Figure 8. Heterogeneity for the subset of MNIST training data belonging to each digit class respectively projected onto the latent space of the convolutional variational autoencoder (cVAE). The leftmost plot shows the pooled heterogeneity for each digit class (the effective total area of latent space occupied by encoding each digit class). The middle plot shows the within-observation heterogeneity (the effective total area of latent space per encoded observation of each digit class, respectively). The rightmost plot shows the between-observation heterogeneity (the effective number of observations per digit class). Recall that Rényi heterogeneity on a continuous distribution gives the effective size of the domain of an equally heterogeneous uniform distribution on the same space, which explains why the within-observation heterogeneity values here are less than 1.

Figure 9 demonstrates the correspondence of between-observation heterogeneity (i.e., the effective number of observations) and the visual diversity of different samples from the latent space of our cVAE model. For each image in the MNIST training dataset, we computed the effective location of its latent representation: $\mathbf{m}\left({\mathbf{x}}_i\right)$ for $i\in \{1,2,\dots ,N\}$. For each of these image representations, we defined a “neighborhood” including the 49 other images whose latent coordinates were closest in Euclidean distance (which is sensible on the latent space given the Gaussian prior). For all such neighbourhoods defined, we then reconstructed the corresponding images on $\mathcal{X}$, whose between-observation heterogeneity was then computed using Equation (57). Figure 9b shows the estimated effective number of observations for the latent neighborhoods with the greatest and least heterogeneity. One can appreciate that neighborhoods with ${\mathsf{\Pi }}_q^{\mathrm{B}}$ close to 1 include images with considerably less diversity than neighborhoods with ${\mathsf{\Pi }}_q^{\mathrm{B}}$ closer to the upper limit of 49. These data suggest that the between-observation heterogeneity—which is the effective number of observations in X with respect to the latent features learned by a cVAE—can indeed correspond to visually appreciable sample diversity.

Figure 9. Visual illustration of MNIST image samples corresponding to different levels of representational Rényi heterogeneity under the convolutional variational autoencoder (cVAE). Panel (a) illustrates the approach to this analysis. Here, the surface $\mathcal{Z}$ shows hypothetical contours of a probability distribution over the 2-dimensional latent feature space. The surface $\mathcal{X}$ represents the observable space, upon which we have projected an “image” of the latent space $\mathcal{Z}$ for illustrative purposes. We first compute the expected latent locations m(x_i) for each image ${\mathrm{x}}_i\in \mathcal{X}$ (A₁) We then define the latent neighbourhood of image x_i as the 49 images whose latent locations are closest to m(x_i) in Euclidean distance. (A₂) Each coordinate in the neighbourhood of m(x_i) is then projected onto a corresponding patch on the observable space of images. (A₃) These images are then projected as a group back onto the latent space, where Equation (57) can be applied, given equal weights over images, to compute the effective number of observations in the neighbourhood of x_i. Panel (b) plots the most and least heterogeneous neighbourhoods so that we may compare the estimated effective number of observations with the visually appreciable sample diversity.

5. Discussion

This paper introduced representational Rényi heterogeneity, a measurement approach that satisfies the replication principle [1,38,41] and is decomposable [28] while requiring neither a priori (A) categorical partitioning nor (B) specification of a distance function on the input space. Rather, the experimenter is free to define a model that maps observable data onto a semantically relevant domain upon which Rényi heterogeneity may be tractably computed, and where a distance function need not be explicitly manipulated. These properties facilitate heterogeneity measurements for several new applications. Compared to state-of-the-art comparator indices under a beta mixture distribution, RRH more reliably quantified the number of unique mixture components (Section 4.1), and under a deep generative model of image data, RRH was able to measure the effective number of distinct images with respect to latent continuous representations (Section 4.2). In this section, we further synthesize our conclusions, discuss their implications, and highlight open questions for future research.

The main problem we set out to address was that all state of the art numbers equivalent heterogeneity measures (Section 2.2) require a priori specification of a distance function and categorical partitioning on the observable space. To this end, we showed that RRH does not require categorical partitioning of the input space (Section 3). Although our analysis under the two-component BMM assumed that the number of components was known, RRH was the only index able to accurately identify an effectively singular cluster (i.e., where mixture components overlapped; Figure 5). We also showed that the categorical RRH did not violate the principle of transfers [39,40] (i.e., it was strictly concave with respect to mixture component weights), unlike the functional Hill numbers (Figure 5). Future studies should extend this evaluation to mixtures of other distributional forms in order to better characterize the generalizability of our conclusions.

Section 3.1 and Section 3.2 both showed that RRH does not require specification of a distance function on the observable space. Instead, one must specify a model that maps the observable space onto a probability distribution over the latent representation. This is beneficial since input space distances are often irrelevant or misleading. For example, latent representations of image data learned by a convolutional neural network will be robust to translations of the inputs since convolution is translation invariant. However, pairwise distances on the observable space will be exquisitely sensitive to semantically irrelevant translations of input data. Furthermore, semantically relevant information must often be learned from raw data using hierarchical abstraction. Ultimately, when (A) pre-defined distance metrics are sensitive to noisy perturbations of the input space, or (B) the relevant semantic content of some input data is best captured by a latent abstraction, the RRH measure will be particularly useful.

The requirement of specifying a representational model $f:\mathcal{X}\to \mathcal{P}\left(\mathcal{Z}\right)$ implies the additional problem of model selection. In Section 3, we noted that the determination of whether a model is appropriate must be made in a domain-specific fashion. For instance, the method by which ecologists assign species labels prior to measurement of species diversity implies the use of a mapping from the observable space of organisms to a degenerate distribution over species labels (Example 1). In Section 4.2, we used the encoder module of a cVAE (a generative model based on a convolutional neural network architecture [23,24]) to represent images as 2-dimensional real-valued vectors in order to demonstrate our ability to capture variation in digits’ written forms (see Figure 7B and Figure 9). Someone concerned with measuring heterogeneity of image batches in terms of the digit-class distribution could choose a categorical latent representation corresponding to the digit classes (this would return the effective number of digit classes per sample). Regardless, the model used to map between observations and the latent space should be validated using either explanatory power (e.g., maximization of a lower bound on the model evidence), generalizability (e.g., out of sample predictive power), or another approach that is justifiable within the investigator’s scientific domain of interest.

In addition to the results of empirical applications of RRH in Section 4, we were also able to show that RRH generalizes the process by which species diversity and indices of economic equality are computed (Example 1). In doing so, we are able to clarify some of the assumptions inherent in those indices. Specifically, that assignment of species or ownership labels (in ecological and economic settings, respectively) corresponds to mapping from an observable space, such as the space of organisms’ identifiable features or the space of economic resources, onto a degenerate distribution over the categorical labels (Table 2). It is possible that altering the form of that mapping may yield new insights about ecological and economic diversity.

In conclusion, we have introduced an approach for measuring heterogeneity that requires neither (A) categorical partitioning nor (B) distance measure on the observable space. Our RRH method enables measurement of heterogeneity in disciplines where categorical entities are unreliably defined, or where relevant semantic content of some data is best captured by a hierarchical abstraction. Furthermore, our approach includes many existing heterogeneity indices as special cases, while facilitating clarification of many of their assumptions. Future work should evaluate the RRH in practice and under a broader array of models.