Maximizing diversity in biology and beyond

Entropy, under a variety of names, has long been used as a measure of diversity in ecology, as well as in genetics, economics and other fields. There is a spectrum of viewpoints on diversity, indexed by a real parameter q giving greater or lesser importance to rare species. Leinster and Cobbold proposed a one-parameter family of diversity measures taking into account both this variation and the varying similarities between species. Because of this latter feature, diversity is not maximized by the uniform distribution on species. So it is natural to ask: which distributions maximize diversity, and what is its maximum value? In principle, both answers depend on q, but our main theorem is that neither does. Thus, there is a single distribution that maximizes diversity from all viewpoints simultaneously, and any list of species has an unambiguous maximum diversity value. Furthermore, the maximizing distribution(s) can be computed in finite time, and any distribution maximizing diversity from some particular viewpoint q>0 actually maximizes diversity for all q. Although we phrase our results in ecological terms, they apply very widely, with applications in graph theory and metric geometry.


Introduction
For decades, ecologists have used entropy-like quantities as measures of biological diversity. The basic premise is that given a biological community or ecosystem containing n species in proportions p 1 , . . . , p n , the entropy of the probability distribution (p i ) indicates the extent to which the community is balanced or 'diverse'. Shannon entropy itself is often used; so too are many variants, as we shall see. But almost all of them share the property that for a fixed number n of species, the entropy is maximized by the uniform distribution p i = 1/n. However, there is a growing appreciation that this crude model of a biological community is too far from reality, in that it takes no notice of the varying similarities between species. For instance, we would intuitively judge a meadow to be more diverse if it consisted of ten dramatically different plant species than if it consisted of ten species of grass. This has led to the introduction of measures that do take into account inter-species similarities [34,26]. In mathematical terms, making this refinement essentially means extending the classical notion of entropy from probability distributions on a finite set to probability distributions on a finite metric space.
The maximum entropy problem now becomes more interesting. Consider, for instance, a pond community consisting of two very similar species of frog and one species of newt. We would not expect the maximum entropy (or diversity) to be achieved by the uniform distribution (1/3, 1/3, 1/3), since the community would then be 2/3 frog and only 1/3 newt. We might expect the maximizing distribution to be closer to (1/4, 1/4, 1/2); the exact answer should depend on the degrees of similarity of the species involved. We return to this scenario in Example 8.2.
For the sake of concreteness, this paper is written in terms of an ecological scenario: a community of organisms classified into species. However, nothing that we do is intrinsically ecological, or indeed connected to any specific branch of science. Our results apply equally to any collection of objects classified into types.
It is well understood that Shannon entropy is just one point (albeit a special one) on a continuous spectrum of entropies, indexed by a parameter q ∈ [0, ∞]. This spectrum has been presented in at least two ways: as the Rényi entropies H q [35] and as the so-called Tsallis entropies S q (actually introduced as biodiversity measures by Patil and Taillie prior to Tsallis's work in physics, and earlier still in information theory [37,32,13]): Both H q and S q converge to Shannon entropy as q → 1. Moreover, H q and S q can be obtained from one another by an increasing invertible transformation, and in this sense are interchangeable.
When H q or S q is used as a diversity measure, q controls the weight attached to rare species, with q = 0 giving as much importance to rare species as common ones, and the limiting case q = ∞ reflecting only the prevalence of the most common species. Different values of q produce genuinely different judgements on which of two distributions is the more diverse. For instance, if over time a community loses some species but becomes more balanced, then the Rényi and Tsallis entropies decrease for q = 0 but increase for q = ∞. Varying q therefore allows us to incorporate a spectrum of viewpoints on the meaning of the word 'diversity'.
Here we use the diversity measures introduced by Leinster and Cobbold [26], which both (i) reflect this spectrum of viewpoints by including the variable parameter q, and (ii) take into account the varying similarities between species. We review these measures in Sections 2-4. In the extreme case where different species are assumed to have nothing whatsoever in common, they reduce to the exponentials of the Rényi entropies, and in other special cases they reduce to other diversity measures used by ecologists. In practical terms, the measures of [26] have been used to assess a variety of ecological systems, from communities of microbes [38,1] and crustacean zooplankton [17] to alpine plants [5] and arctic predators [4], as well as being applied in non-biological contexts such as computer network security [39].
Mathematically, the set-up is as follows. A biological community is modelled as a probability distribution p = (p 1 , . . . , p n ) (representing the proportions of species) together with an n × n matrix Z (whose (i, j)-entry represents the similarity between species i and j). From this data, a formula gives a real number q D Z (p) for each q ∈ [0, ∞], called the 'diversity of order q' of the community. As for the Rényi entropies, different values of q make different judgements: for instance, it may be that for two distributions p and p , 1 D Z (p) < 1 D Z (p ) but 2 D Z (p) > 2 D Z (p ). Now consider the maximum diversity problem. Fix a list of species whose similarities to one another are known; that is, fix a matrix Z (subject to hypotheses to be discussed). The two basic questions are: • Which distribution(s) p maximize the diversity q D Z (p) of order q?
• What is the value of the maximum diversity sup p q D Z (p)?
This can be interpreted ecologically as follows: if we have a fixed list of species and complete control over their abundances within our community, how should we choose those abundances in order to maximize the diversity, and how large can we make that diversity?
In principle, both answers depend on q. After all, we have seen that if distributions are ranked by diversity then the ranking varies according to the value of q chosen. But our main theorem is that, in fact, both answers are independent of q: Theorem 6.1 (Main theorem) There exists a probability distribution on {1, . . . , n} that maximizes q D Z for all q ∈ [0, ∞]. Moreover, the maximum diversity sup p So, there is a 'best of all possible worlds': a distribution that maximizes diversity no matter what viewpoint one takes on the relative importance of rare and common species.
This theorem merely asserts the existence of a maximizing distribution. However, a second theorem describes how to compute all maximizing distributions, and the maximum diversity, in a finite number of steps (Theorem 7.1).
Better still, if by some means we have found a distribution p that maximizes the diversity of some order q > 0, then a further result asserts that p maximizes diversity of all orders (Corollary 6.4). For instance, it is often easiest to find a maximizing distribution for diversity of order ∞ (as in Example 8.1 and Proposition 9.1), and it is then automatic that this distribution maximizes diversity of all orders.
Let us put these results into context. First, they belong to the huge body of work on maximum entropy problems. For example, the normal distribution has the maximum entropy among all probability distributions on R with a given mean and variance, a property which is intimately connected with its appearance in the central limit theorem. This alone would be enough motivation to seek maximum entropy distributions in other settings (such as the one at hand), quite apart from the importance of maximum entropy in thermodynamics, machine learning, macroecology, and so on.
Second, we will see that maximum diversity is very closely related to the emerging invariant known as magnitude. This is defined in the extremely wide generality of enriched category theory [24,Section 1], and specializes in interesting ways in a variety of mathematical fields. For instance, it automatically produces a notion of the Euler characteristic of an (ordinary) category, closely related to topological Euler characteristic [22]; in the context of metric spaces, magnitude encodes geometric information such as volume and dimension [2,31,41]; in graph theory, magnitude is a new invariant that turns out to be related to a graded homology theory for graphs [25,14]; and in algebra, magnitude produces an invariant of associative algebras that can be understood as a homological Euler characteristic [7].
This work is self-contained. To that end, we begin by explaining and defining the diversity measures in [26] (Sections 2-4). Then come the results: preparatory lemmas in Section 5, and the main results in Sections 6 and 7. Examples are given in Sections 8-10, including results on special cases such as when the similarity matrix Z is either the adjacency matrix of a graph or positive definite. Perhaps counterintuitively, a distribution that maximizes diversity can eliminate some species entirely. This is addressed in Section 11, where we derive necessary and sufficient conditions on Z for maximization to preserve all species. Finally, we state some open questions (Section 12).
The main results of this paper previously appeared in a preprint of Leinster [23], but the proofs we present here are substantially simpler. Of the new results, Lemma 11.1 (the key to our results on preservation of species by maximizing distributions) borrows heavily from an argument of Fremlin and Talagrand [9]. In (a), there are four species, with the first dominant and the others relatively rare. In (b), the fourth species is absent but the community is otherwise evenly balanced.
and positive definite if this inequality is strict.

A spectrum of viewpoints on diversity
Ecologists began to propose quantitative definitions of biological diversity in the mid-twentieth century [36,40], setting in motion more than fifty years of heated debate, dozens of further proposed diversity measures, hundreds of scholarly papers, at least one book devoted to the subject [28], and consequently, for some, despair (already expressed by 1971 in a famously-titled paper of Hurlbert [16]). Meanwhile, parallel discussions were taking place in disciplines such as genetics [19], economists were using the same formulas to measure wealth inequality and industrial concentration [11], and information theorists were developing the mathematical theory of such quantities under the name of entropy rather than diversity.
Obtaining accurate data about an ecosystem is beset with practical and statistical problems, but that is not the reason for the prolonged debate. Even assuming that complete information is available, there are genuine differences of opinion about what the word 'diversity' should mean. We focus here on one particular axis of disagreement, illustrated by the examples in Figure 1.
One extreme viewpoint on diversity is that preservation of species is all that matters: 'biodiversity' simply means the number of species present (as is common in the ecological literature as well as the media). Since no attention is paid to the abundances of the species present, rare species count for exactly as much as common species. From this viewpoint, community (a) of Figure 1 is more diverse than community (b), simply because it contains more species.
The opposite extreme is to ignore rare species altogether and consider only those that are most common. (This might be motivated by a focus on overall ecosystem function.) From this viewpoint, community (b) is more diverse than community (a), because it is better-balanced: (a) is dominated by a single common species, whereas (b) has three common species in equal proportions.
Between these two extremes, there is a spectrum of intermediate viewpoints, attaching more or less weight to rare species. Different scientists have found it appropriate to adopt different positions on this spectrum for different purposes, as the literature amply attests.
Rather than attempting to impose one particular viewpoint, we will consider all equally. Thus, we use a one-parameter family of diversity measures, with the 'viewpoint parameter' q ∈ [0, ∞] controlling one's position on the spectrum. Taking q = 0 will give rare species as much importance as common species, while taking q = ∞ will give rare species no importance at all.
There is one important dimension missing from the discussion so far. We will consider not only the varying abundances of the species, but also the varying similarities between them. This is addressed in the next section.

Distributions on a set with similarities
In this section and the next, we give a brief introduction to the diversity measures of Leinster and Cobbold [26]. We have two tasks. We must build a mathematical model of the notion of 'biological community' (this section). Then, we must define and explain the diversity measures themselves (next section).
In brief, a biological community will be modelled as a finite set (whose elements are the species) equipped with both a probability distribution (indicating the relative abundances of the species) and, for each pair of elements of the set, a similarity coefficient (reflecting the similarities between species).
Let us now consider each of these aspects in turn. First, we assume a community or system of individuals, partitioned into n ≥ 1 species. The word 'species' need not have its standard meaning: it can denote any unit thought meaningful, such as genus, serotype (in the case of viruses), or the class of organisms having a particular type of diet. It need not even be a biological grouping; for instance, in [29] the units are soil types. For concreteness, however, we write in terms of an ecological community divided into species. The division of a system into species or types may be somewhat artificial, but this is mitigated by the introduction of the similarity coefficients (as shown in [26], p.482).
Second, each species has a relative abundance, the proportion of organisms in the community belonging to that species. Thus, listing the species in order as 1, . . . , n, the relative abundances determine a vector p = (p 1 , . . . , p n ). This is a probability distribution: p i ≥ 0 for each species i, and n i=1 p i = 1. Abundance can be measured in any way thought relevant, e.g. number of individuals, biomass, or (in the case of plants) ground coverage.
Critically, the word 'diversity' refers only to the relative, not absolute, abundances. If half of a forest burns down, or if a patient loses 90% of their gut bacteria, then it may be an ecological or medical disaster; but assuming that the system is well-mixed, the diversity does not change. In the language of physics, diversity is an intensive quantity (like density or temperature) rather than an extensive quantity (like mass or heat), meaning that it is independent of the system's size.
The third and final aspect of the model is inter-species similarity. For each pair (i, j) of species, we specify a real number Z ij representing the similarity between species i and j. This defines an n × n matrix Z = (Z ij ) 1≤i,j≤n . In [26], similarity is taken to be measured on a scale of 0 to 1, with 0 meaning total dissimilarity and 1 that the species are identical. Thus, it is assumed there that In fact, our maximization theorems will only require the weaker hypotheses together with the requirement that Z is a symmetric matrix. (In the appendix to [26], matrices satisfying (2) were called 'relatedness matrices'.) Just as the meanings of 'species' and 'abundance' are highly flexible, so too is the meaning of 'similarity': The simplest similarity matrix Z is the identity matrix I. This is called the naive model in [26], since it embodies the assumption that distinct species have nothing in common. Crude though this assumption is, it is implicit in the diversity measures most popular in the ecological literature [26, Table 1].
Example 3.2 With the rapid fall in the cost of DNA sequencing, it is increasingly common to measure similarity genetically (in any of several ways). Thus, the coefficients Z ij may be chosen to represent percentage genetic similarities between species. This is an effective strategy even when the taxonomic classification is unclear or incomplete [26], as is often the case for microbial communities [38]. Example 3.3 Given a suitable phylogenetic tree, we may define the similarity between two present-day species as the proportion of evolutionary time before the point at which the species diverged.
Example 3.4 In the absence of more refined data, we can measure species similarity according to a taxonomic tree. For instance, we might define

The diversity measures
Here we state the definition of the diversity measures of [26], which we will later seek to maximize. We then explain the reasons for this particular definition. As in Section 3, we take a biological community modelled as a finite probability distribution p = (p 1 , . . . , p n ) together with an n × n matrix Z satisfying (2). As explained in Section 2, we define not one diversity measure but a family of them, indexed by a parameter q ∈ [0, ∞] controlling the emphasis placed on rare species. The diversity of order q of the community is Here supp(p) is the support of p (Conventions, p.5), Zp is the column vector obtained by multiplying the matrix Z by the column vector p, and (Zp) i is its ith entry. The hypotheses (2) imply that (Zp) i > 0 whenever i ∈ supp(p), and so q D Z (p) is well-defined.
Although this formula is invalid for q = 1, it converges as q → 1, and 1 D Z (p) is defined to be the limit. The same is true for q = ∞. Explicitly, The applicability, context and meaning of definition (3) are discussed at length in [26]. Here we briefly review the principal points.
First, the definition includes as special cases many existing quantities going by the name of diversity or entropy. For instance, in the naive model Z = I, the diversity q D I (p) is the exponential of the Rényi entropy of order q, and is also known in ecology as the Hill number of order q. (References for this and the next two paragraphs are given in [26, Table 1].) Continuing in the naive model Z = I and specializing further to particular values of q, we obtain other known quantities: 0 D I (p) is species richness (the number of species present), 1 D I (p) is the exponential of Shannon entropy, 2 D I (p) is the Gini-Simpson index (the reciprocal of the probability that two randomlychosen individuals are of the same species), and ∞ D I (p) = 1/ max i p i is the Berger-Parker index (a measure of the dominance of the most abundant species). Now allowing a general Z, the diversity of order 2 is 1 i,j p i Z ij p j . Thus, diversity of order 2 is the reciprocal of the expected similarity between a random pair of individuals. (The meaning given to 'similarity' will determine the meaning of the diversity measure: taking the coefficients Z ij to be genetic similarities produces a genetic notion of diversity, and similarly phylogenetic, taxonomic, and so on.) Up to an increasing, invertible transformation, this is the wellstudied quantity known as Rao's quadratic entropy.
Given distributions p and p on the same list of species, different values of q may make different judgements on which of p and p is the more diverse. For instance, with Z = I and the two distributions shown in Figure 1, taking q = 0 makes community (a) more diverse and embodies the first 'extreme viewpoint' described in Section 2, whereas q = ∞ makes (b) more diverse and embodies the opposite extreme.
It is therefore most informative if we calculate the diversity of all orders q ∈ [0, ∞]. The graph of q D Z (p) against q is called the diversity profile of p. Two distributions p and p can be compared by plotting their diversity profiles on the same axes. If one curve is wholly above the other then the corresponding distribution is unambiguously more diverse. If they cross (as in Figure 2(b), p.14), then the judgement as to which is the more diverse depends on how much importance is attached to rare species.
The formula for q D Z (p) can be understood as follows.
First, for a given species i, the quantity (Zp) i = j Z ij p j is the expected similarity between species i and an individual chosen at random. Differently put, (Zp) i measures the ordinariness of the ith species witin the community; in [26], it is called the 'relative abundance of species similar to the ith'. Hence the mean ordinariness of an individual in the community is i p i (Zp) i . This measures the lack of diversity of the community, so its reciprocal is a measure of diversity. This is exactly 2 D Z (p).
To explain the diversity of orders q = 2, we recall the classical notion of power mean. Let p = (p 1 , . . . , p n ) be a finite probability distribution and let Chapter II]. This definition is extended to t = 0 and t = ±∞ by taking limits in t, which gives Now, when we take the 'mean ordinariness' in the previous paragraph, we can replace the ordinary arithmetic mean (the case t = 1) by the power mean of order t = q − 1. Again taking the reciprocal, we obtain exactly (3). That is, for all p, Z, and q ∈ [0, ∞]. So in all cases, diversity is the reciprocal mean ordinariness of an individual within the community, for varying interpretations of 'mean'. The diversity measures q D Z (p) have many good properties, discussed in [26]. Crucially, they are effective numbers: that is, for all q and n. This gives meaning to the quantities q D Z (p): if q D Z (p) = 32.8, say, then the community is nearly as diverse as a community of 33 completely dissimilar species in equal proportions. With the stronger assumptions (1) on Z, the value of q D Z (p) always lies between 1 and n.
Diversity profiles are decreasing: as less emphasis is given to rare species, perceived diversity drops. More precisely: Proposition 4.1 Let p be a probability distribution on {1, . . . , n} and let Z be an n×n matrix satisfying (2). If (Zp) i has the same value K for all i ∈ supp(p) Proof This is immediate from equation (4) and a classical result on power means [12,Theorem 16]: M t (p, x) is increasing in t, strictly so unless x i has the same value K for all i ∈ supp(p), in which case it has constant value K.
So, any diversity profile is either constant or strictly decreasing. The first part of the next lemma states that diversity profiles are also continuous: Lemma 4.2 Fix an n × n matrix Z satisfying (2). Then: Proof See Propositions A1 and A2 of the appendix of [26].
Finally, the measures have the sensible property that if some species have zero abundance, then the diversity is the same as if they were not mentioned at all. To express this, we introduce some notation: given a subset B ⊆ {1, . . . , n}, we denote by Z B the submatrix (Z ij ) i,j∈B of Z.
Proof This is trivial, and is also an instance of a more general naturality property (Lemma A13 in the appendix of [26]).

Preparatory lemmas
For the rest of this work, fix an integer n ≥ 1 and an n × n symmetric matrix Z of nonnegative reals whose diagonal entries are positive (that is, strictly greater than zero). Also write ∆ n = (p 1 , . . . , p n ) ∈ R n : p i ≥ 0, p 1 + · · · + p n = 1 for the set of probability distributions on {1, . . . , n}.
To prove the main theorem, we begin by making two apparent digressions. Let M be any matrix. A weighting on M is a column vector w such that Mw is the column vector whose every entry is 1. It is trivial to check that if both M and its transpose have at least one weighting, then the quantity i w i is independent of the choice of weighting w on M; this quantity is called the When M is symmetric (the case of interest here), |M| is defined just as long as M has at least one weighting. When M is invertible, M has exactly one weighting and |M| is the sum of all the entries of M −1 .
The second digression concerns the dichotomy expressed in Proposition 4.1: every diversity profile is either constant or strictly decreasing. We now ask: which distributions have constant diversity profile?
This question turns out to have a clean answer in terms of weightings and magnitude. To state it, we make some further definitions.
Let B ⊆ {1, . . . , n}, and let 0 = w ∈ [0, ∞) B be a nonnegative vector. Then there is a probability distribution p(w) on {1, . . . , n} defined by In particular, let B be a nonempty subset of {1, . . . , n} and w a nonnegative weighting on Z B = (Z ij ) i,j∈B . Then w = 0, so p(w) is defined, and p(w Lemma 5.2 The following are equivalent for p ∈ ∆ n : i. p is invariant; ii. (Zp) i = (Zp) j for all i, j ∈ supp(p); iii. p = p(w) for some nonnegative weighting w on Z B and some nonempty subset B ⊆ {1, . . . , n}.
Proof (i) ⇐⇒ (ii) is immediate from Proposition 4.1. For (ii) =⇒ (iii), assume (ii). Put B = supp(p) and write K = (Zp) i for any i ∈ B. Then K > 0, so we may define w ∈ R B by w i = p i /K (i ∈ B). Evidently p = p(w) and w is nonnegative. Furthermore, w is a weighting on Z B , since whenever i ∈ B, Finally, for (iii) =⇒ (ii) and 'moreover', take B and w as in (iii). Then supp(p(w)) ⊆ B, so for all i ∈ supp(p(w)), Hence q D Z (p(w)) = |Z B | for all q ∈ [0, ∞] by Proposition 4.1.
We now prove a result that is much weaker than the main theorem, but will act as a stepping stone in the proof. Proof Let q ∈ (0, 1). Then q D Z is continuous on the compact space ∆ n (Lemma 4.2(ii)), so attains a maximum at some point p. Take j, k ∈ supp(p) such that (Zp) j is least and (Zp) k is greatest. By Lemma 5.2, it is enough to prove that (Zp) j = (Zp) k .
Define δ j ∈ ∆ n by taking (δ j ) i to be the Kronecker delta δ ji , and δ k similarly. Then p + t(δ j − δ k ) ∈ ∆ n for all real t sufficiently close to 0, and where (5) holds because p is a supremum, (6) is a routine computation, (7) and (9) follow from the defining properties of j and k, and (8) uses the symmetry of Z. Equality therefore holds throughout, and in particular in (9). Hence (Zp) j = (Zp) k , as required.
An alternative proof uses Lagrange multipliers, but is complicated by the possibility that q D Z attains its maximum on the boundary of ∆ n .
The result we have just proved only concerns the maximization of q D Z for specific values of q, but the following lemma will allow us to deduce results about maximization for all q simultaneously.
Lemma 5.5 For 0 ≤ q ≤ q ≤ ∞, any invariant distribution that maximizes q D Z also maximizes q D Z . In particular, any invariant distribution that maximizes 0 D Z is maximizing.
Proof Let 0 ≤ q ≤ q ≤ ∞ and let p be an invariant distribution that maximizes q D Z . Then for all r ∈ ∆ n , since diversity profiles are decreasing (Proposition 4.1). Proof An equivalent statement is that there exists an invariant maximizing distribution. To prove this, choose a decreasing sequence (q λ ) ∞ λ=1 in (0, 1) converging to 0. By Lemma 5.3, we can choose for each λ ≥ 1 an invariant distribution p λ that maximizes q λ D Z . Since ∆ n is compact, we may assume (by passing to a subsequence if necessary) that the sequence (p λ ) converges to some point p ∈ ∆ n . We will show that p is invariant and maximizing.
To show that p is maximizing, first note that p λ maximizes q λ D Z whenever λ ≥ λ ≥ 1 (by Lemma 5.5). Fixing λ and letting λ → ∞, this implies that p maximizes q λ D Z , since q λ D Z is continuous (Lemma 4.2(ii)).
Thus, p maximizes q λ D Z for all λ. But q λ → 0 as λ → ∞, and diversity is continuous in its order (Lemma 4.2(i)), so p maximizes 0 D Z . Since p is invariant, Lemma 5.5 implies that p is maximizing.
The theorem can be understood as follows (Figure 2(a)). Each particular value of the viewpoint parameter q ranks the set of all distributions p in order of diversity, with p placed above p when q D Z (p) > q D Z (p ). Different values of q rank the set of distributions differently. Nevertheless, there is a distribution p max that is at the top of every ranking. This is the content of the first half of Theorem 6.1.
Alternatively, we can visualize the theorem in terms of diversity profiles (Figure 2(b)). Diversity profiles may cross, reflecting the different priorities embodied by different values of q. But there is at least one distribution p max whose profile is above every other profile; moreover, its profile is constant. Theorem 6.1 immediately implies:

Corollary 6.2 Every maximizing distribution is invariant.
This result can be partially understood as follows. For Shannon entropy, and more generally any of the Rényi entropies, the maximizing distribution is obtained by taking the relative abundance p i to be the same for all species i. This is no longer true when inter-species similarities are taken into account. However, what is approximately true is that diversity is maximized when (Zp) i , the relative abundance of species similar to the ith, is the same for all species i. This follows from Corollary 6.2 together with the characterization of invariant distributions in Lemma 5.2(ii); but it is only 'approximately true' because it is only guaranteed that (Zp) i = (Zp) j when i and j both belong to the support of p, not for all i and j. It may in fact be that some or all maximizing distributions do not have full support, a phenomenon we examine in Section 11.
The second half of Theorem 6.1 tells us that associated with the matrix Z is a numerical invariant, the constant value of a maximizing distribution: We show how to compute D max (Z) in the next section.
If a distribution p maximizes diversity of order 2, say, must it also maximize diversity of orders 1 and ∞? The answer turns out to be yes: Proof Let q ∈ (0, ∞] and let p be a distribution maximizing q D Z . Then where the first inequality holds because diversity profiles are decreasing. So equality holds throughout. Now q D Z (p) = 0 D Z (p) with q = 0, so Proposition 4.1 implies that p is invariant. But also 0 D Z (p) = D max (Z), so p maximizes 0 D Z . Hence by Lemma 5.5, p is maximizing.
The significance of this corollary is that if we wish to find a distribution that maximizes diversity of all orders q, it suffices to find a distribution that maximizes diversity of a single nonzero order.
The hypothesis that q > 0 in Corollary 6.4 cannot be dropped. Indeed, take Z = I. Then 0 D I (p) is species richness (the cardinality of supp(p)), which is maximized by any distribution p of full support, whereas 1 D I (p) is the exponential of Shannon entropy, which is maximized only when p is uniform.

The computation theorem
The main theorem guarantees the existence of a maximizing distribution p max , but does not tell us how to find it. It also states that q D Z (p max ) is independent of q, but does not tell us what its value is. The following result repairs both omissions. ii. The maximizing distributions are precisely those of the form p(w) where w is a nonnegative weighting on Z B for some B attaining the maximum in (10).
where (11) follows from the fact that there is an invariant maximizing distribution (Theorem 6.1), (12) follows from Lemma 5.2, and (13) follows from the trivial fact that |Z B | ≥ 0 = |Z ∅ | whenever Z B admits a nonnegative weighting. This proves (i). Every maximizing distribution is invariant (Corollary 6.2), so (ii) follows from Lemma 5.2. This algorithm takes exponentially many steps in n, and Remark 9.7 provides strong evidence that the time taken cannot be reduced to a polynomial in n. But the situation is not as hopeless as it might appear, for two reasons.
First, each step of the algorithm is fast, consisting as it does of solving a system of linear equations. For instance, in an implementation in Matlab on a standard laptop, with no attempt at optimization, the maximizing distributions of 25 × 25 matrices were computed in a few seconds. * Second, for certain classes of matrices Z, we can make substantial improvements in computing time, as observed in Section 10.

Simple examples
The next three sections give examples of the main results, beginning here with some simple, specific examples.

Example 8.1
First consider the naive model Z = I, in which different species are deemed to be entirely dissimilar. As noted in Section 4, q D I (p) is the exponential of the Rényi entropy of order q. It is well-known that Rényi entropy of any order q > 0 is maximized uniquely by the uniform distribution. This result also follows trivially from Corollary 6.4: for clearly ∞ D I (p) = 1/ max i p i is uniquely maximized by the uniform distribution, and the corollary implies that the same is true for all values of q > 0. Moreover, D max (I) = |I| = n. Example 8.2 For a general matrix Z satisfying (1), a two-species system is always maximized by the uniform distribution p 1 = p 2 = 1/2. When n = 3, however, nontrivial examples arise. For instance, take the system shown in Figure 3, consisting of one species of newt and two species of frog. Let us first consider intuitively what we expect the maximizing distribution to be, then compare this with the answer given by Theorem 7.1.
If we ignore the fact that the two frog species are more similar to each other than they are to the newt, then (as in Example 8.1) the maximizing distribution is (1/3, 1/3, 1/3). At the other extreme, if we regard the two frog species as essentially identical then effectively there are only two species, newts and frogs, so the maximizing distribution gives relative abundance 0.5 to the newt and 0.5 to the frogs. So with this assumption, we expect diversity to be maximized by the distribution (0.5, 0.25, 0.25). * We thank Christina Cobbold for carrying out this implementation. Intuitively, then, the maximizing distribution should lie between these two extremes. And indeed, it does: implementing the algorithm in Remark 7.2 (or using Proposition 10.3 below) reveals that the unique maximizing distribution is (0.478, 0.261, 0.261).
One of our standing hypotheses on Z is symmetry. The last of our simple examples shows that if Z is no longer assumed to be symmetric, then the main theorem fails in every respect.

Example 8.3
Let Z = 1 1/2 0 1 , which satisfies all of our standing hypotheses except symmetry. Consider a distribution p = (p 1 , p 2 ) ∈ ∆ 2 . If p is (1, 0) or (0, 1) then q D Z (p) = 1 for all q. Otherwise, From (14) it follows that sup p∈∆2 0 D Z (p) = 2. However, this supremum is not attained; 0 D Z (p) → 2 as p → (1, 0), but 0 D Z (1, 0) = 1. Equations (15) and (16) imply that sup Perhaps surprisingly, nonsymmetric similarity matrices Z do have practical uses. For example, it is shown in [26, Proposition A7] that the mean phylogenetic diversity measures of Chao, Chiu and Jost [6] are a special case of the measures q D Z (p), obtained by taking a particular Z depending on the phylogenetic tree concerned. This Z is usually nonsymmetric, reflecting the asymmetry of evolutionary time. More generally, the case for dropping the symmetry axiom for metric spaces was made in [21], and Gromov has argued that symmetry 'unpleasantly limits many applications' [10, p.xv]. So the fact that our maximization theorem fails for nonsymmetric Z is an important restriction.

Maximum diversity on graphs
Consider those matrices Z for which each similarity coefficient Z ij is either 0 or 1. A matrix Z of this form amounts to a (finite, undirected) reflexive graph with vertex-set {1, . . . , n}, with an edge between i and j if and only if Z ij = 1. (That is, Z is the adjacency matrix of the graph.) Our standing hypotheses on Z then imply that Z ii = 1 for all i, so every vertex has a loop on it; this is the meaning of reflexive.
What is the maximum diversity of the adjacency matrix of a graph?
To state the answer, we recall some terminology. Vertices x and y of a graph are adjacent, written x ∼ y, if there is an edge between them. (In particular, every vertex of a reflexive graph is adjacent to itself.) A set of vertices is independent if no two distinct vertices are adjacent. The independence number α(G) of a graph G is the maximal cardinality of an independent set of vertices of G.
Proposition 9.1 Let G be a reflexive graph with adjacency matrix Z. Then the maximum diversity D max (Z) is equal to the independence number α(G).
Proof We will maximize the diversity of order ∞ and apply Theorem 6.1. For any probability distribution p on the vertex-set {1, . . . , n}, we have First we show that D max (Z) ≥ α(G). Choose an independent set B of maximal cardinality, and define p ∈ ∆ n by For each i ∈ supp(p), the sum on the right-hand side of equation (17) is 1/α(G). Hence ∞ D Z (p) = α(G), and so α(G) ≤ D max (Z). Now we show that D max (Z) ≤ α(G). Let p ∈ ∆ n . Choose an independent set B ⊆ supp(p) with maximal cardinality among all independent subsets of supp(p). Then every vertex of supp(p) is adjacent to at least one vertex in B, otherwise we could adjoin it to B to make a larger independent subset. Hence i∈B j : i∼j So there exists i ∈ B such that j : i∼j p j ≥ 1/#B, where #B denotes the cardinality of B. But #B ≤ α(G), so max i∈supp(p) j : i∼j as required.
Remark 9.2 The first part of the proof (together with Corollary 6.4) shows that a maximizing distribution can be constructed by taking the uniform distribution on some independent set of largest cardinality, then extending by zero to the whole vertex-set. Except in the trivial case Z = I, this maximizing distribution never has full support. We return to this point in Section 11. . The independence number of G is 2; this, then, is the maximum diversity of Z. There is a unique independent set of cardinality 2, and a unique maximizing distribution, (1/2, 0, 1/2). where B(i, ε) = {j : d(i, j) ≤ ε}. The number log N (d, ε) is known as the ε-entropy of d [20]. Now define a matrix Z ε by Then Z ε is the adjacency matrix of the reflexive graph G with vertices {1, . . . , n} and i ∼ j if and only if d(i, j) ≤ ε. Thus, a subset of B ⊆ {1, . . . , n} is independent in G if and only if d(i, j) > ε for every i, j ∈ B. It is a consequence of the triangle inequality that and so by Proposition 9.1, Recalling that log q D Z extends the classical notion of Rényi entropy, this thoroughly justifies the name of ε-entropy (which was originally justified by vague analogy).
The moral of the proof of Proposition 9.1 is that by performing the simple task of maximizing diversity of order ∞, we automatically maximize diversity of all other orders. Here is an example of how this can be exploited.
Recall that every graph G has a complement G, with the same vertex-set as G; two vertices are adjacent in G if and only if they are not adjacent in G. Thus, the complement of a reflexive graph is irreflexive (has no loops), and vice versa. A set B of vertices in an irreflexive graph X is a clique if all pairs of distinct elements of B are adjacent in X. The clique number ω(X) of X is the maximal cardinality of a clique in X. Thus, ω(X) = α(X).
Corollary 9.6 Let X be an irreflexive graph. Then sup p (i,j) : i∼j where the supremum is over probability distributions p on the vertex-set of X, and the sum is over pairs of adjacent vertices of X.
Proof Write {1, . . . , n} for the vertex-set of X, and Z for the adjacency matrix of the reflexive graph X. Then for all p ∈ ∆ n , Hence by Theorem 6.1 and Proposition 9.1, It follows from this proof and Remark 9.2 that (i,j) : i∼j p i p j can be maximized as follows: take the uniform distribution on some clique in X of maximal cardinality, then extend by zero to the whole vertex-set.
Remark 9.7 Proposition 9.1 implies that computationally, finding the maximum diversity of an arbitrary Z is at least as hard as finding the independence number of a reflexive graph. This is a very well-studied problem, usually presented in its dual form (find the clique number of an irreflexive graph) and called the maximum clique problem [18]. It is NP-hard, so on the assumption that P = NP, there is no polynomial-time algorithm for computing maximum diversity, nor even for computing the support of a maximizing distribution.

Positive definite similarity matrices
The theory of magnitude of metric spaces runs most smoothly when the matrices Z concerned are positive definite [30,31]. We will see that positive (semi)definiteness is also an important condition when maximizing diversity.
Any positive definite matrix is invertible and therefore has a unique weighting.
If M is positive definite then the supremum is attained by exactly the nonzero scalar multiples x of the unique weighting on M.
Proof This is a small extension of [24,Proposition 2.4.3]. Choose a weighting w on M. By the Cauchy-Schwarz inequality, or equivalently for all x ∈ R n . Equality holds when x is a scalar multiple of w, and if M is positive definite, it holds only then. Finally, taking x = (1, 0, . . . , 0) T in (18) and using positive semidefiniteness gives |M| > 0.
From this, we deduce: be the unique weighting on Z B (which is positive definite), and write x ∈ R n for the extension of y by zero to {1, . . . , n}. Then y = 0, x = 0, and But x does not have full support, so by hypothesis, it is not a scalar multiple of the unique weighting on Z. Hence by Lemma 10.1, We now apply this result on magnitude to the maximization of diversity. In particular, if Z is positive semidefinite and has a nonnegative weighting, then its maximum diversity can be computed in polynomial time.
Corollary 10.4 If Z is positive definite with positive weighting, then its unique maximizing distribution has full support.
In other words, when Z has these properties, its maximizing distribution eliminates no species. Here are three classes of such matrices Z.
Example 10.5 Call Z ultrametric if Z ik ≥ min{Z ij , Z jk } for all i, j, k and Z ii > max j =k Z jk for all i. (Under the assumptions (1) on Z, the latter condition just states that distinct species are not completely similar.) If Z is ultrametric then Z is positive definite with positive weighting, by [24,Proposition 2.4.18].
Such matrices arise in practice: for instance, Z is ultrametric if it is defined from a phylogenetic or taxonomic tree as in Examples 3.3 and 3.4.
Example 10.6 Let r ∈ ∆ n be a probability distribution of full support, and write Z for the diagonal matrix with entries 1/r 1 , . . . , 1/r n . Then for 0 < q < ∞, The right-hand side is the Rényi relative entropy or Rényi divergence Proof Since Z is real symmetric, it is diagonalizable with real eigenvalues. By the hypotheses on Z and the Gershgorin disc theorem [15, Theorem 6.1.1], every eigenvalue of Z is in the interval (0, 2). It follows that Z is positive definite and that every eigenvalue of I − Z is in (−1, 1). Hence I − Z is similar to a diagonal matrix with entries in (−1, 1), and so ∞ k=0 (I − Z) k converges to (I − (I − Z)) −1 = Z −1 . Thus, Writing e = (1 · · · 1) T , the unique weighting on Z is w = Z −1 e. The hypotheses on Z imply that Z − I has nonnegative entries and (2I − Z)e has positive entries. Hence by (19), entrywise, and so w is positive.
Thus, a matrix Z that is ultrametric, or satisfies (1) and is strictly diagonally dominant, has many special properties: the maximum diversity is equal to the magnitude, there is a unique maximizing distribution, the maximizing distribution has full support, and both the maximizing distribution and the maximum diversity can be computed in polynomial time.

Preservation of species
We saw in Examples 9.3 and 9.4 that for certain similarity matrices Z, none of the maximizing distributions has full support. Mathematically, this simply means that maximizing distributions sometimes lie on the boundary of ∆ n . But ecologically, it may sound shocking: is it reasonable that diversity can be increased by eliminating some species?
We argue that it is. Consider, for instance, a forest consisting of one species of oak and ten species of pine, with each species equally abundant. Suppose that an eleventh species of pine is added, again with equal abundance (Figure 4). This makes the forest even more heavily dominated by pine, so it is intuitively reasonable that the diversity should decrease. But now running time backwards, the conclusion is that if we start with a forest containing the oak and all eleven pine species, eliminating the eleventh should increase diversity.
To clarify further, recall from Section 3 that diversity is defined in terms of the relative abundances only. Thus, eliminating species i causes not only a decrease in p i , but also an increase in the other relative abundances p j . If the ith species is particularly ordinary within the community (like the eleventh species oak pine Figure 4: Hypothetical community consisting of one species of oak ( ) and ten species of pine (•), to which one further species of pine is then added (•). Distances between species indicate degrees of dissimilarity (not to scale). of pine), then eliminating it makes way for less ordinary species, resulting in a more diverse community.
The instinct that maximizing diversity should not eliminate any species is based on the assumption that the distinction between species is of high value. (After all, if two species were very nearly identical-or in the extreme, actually identical-then losing one would be of little importance.) If one wishes to make that assumption, one must build it into the model. This is done by choosing a similarity matrix Z with a low similarity coefficient Z ij for each i = j. Thus, Z is close to the identity matrix I (assuming that similarity is measured on a scale of 0 to 1). Example 10.7 guarantees that in this case, there is a unique maximizing distribution and it does not, in fact, eliminate any species.
(The fact that maximizing distributions can eliminate some species has previously been discussed in the ecological literature in the case q = 2; see Pavoine and Bonsall [33] and references therein.) We now derive necessary and sufficient conditions for a similarity matrix Z to admit at least one maximizing distribution of full support, and also necessary and sufficient conditions for every maximizing distribution to have full support. The latter conditions are genuinely more restrictive; for instance, if Z = 1 1 1 1 then some but not all maximizing distributions have full support.
Lemma 11.1 If at least one maximizing distribution for Z has full support then Z is positive semidefinite and admits a positive weighting. Moreover, if every maximizing distribution for Z has full support then Z is positive definite.
Proof Fix a maximizing distribution p of full support. Maximizing distributions are invariant (Corollary 6.2), so by (i) =⇒ (iii) of Lemma 5.2, |Z| p is a weighting of Z and |Z| > 0. In particular, Z has a positive weighting. Now we imitate the proof of Proposition 3B of [9]. For each s ∈ R n such that n i=1 s i = 0, define a function f s : R → R by f s (t) = (p + ts) T Z(p + ts).
Using the symmetry of Z and the fact that |Z| p is a weighting, we obtain Now s i = 0 and p has full support, so p + ts ∈ ∆ n for all real t sufficiently close to zero. But f s (t) = 1/ 2 D Z (p + ts) for such t, so f s has a local minimum at 0. Hence s T Zs ≥ 0. It follows that f s is everywhere positive.
We have shown that s T Zs ≥ 0 whenever s ∈ R n with s i = 0. Now take x ∈ R n with x i = 0. Put s = x/ x i − p. Then s i = 0, and Hence Z is positive semidefinite. For 'moreover', assume that every maximizing distribution for Z has full support. By (21), we need only show that s T Zs > 0 whenever s = 0 with s i = 0. Given such an s, choose t ∈ R such that p + ts lies on the boundary of ∆ n . Then p + ts does not have full support, so is not maximizing, so does not maximize 2 D Z (by Corollary 6.4). Hence f s (t) > f s (0), which by (20) implies that s T Zs > 0.
We can now prove the two main results of this section.
Proposition 11.2 The following are equivalent: i. there exists a maximizing distribution for Z of full support; ii. Z is positive semidefinite and admits a positive weighting.
Proof (i) =⇒ (ii) is the first part of Lemma 11.1. For the converse, assume (ii) and choose a positive weighting w. Then |Z| > 0, so p = w/ |Z| is a probability distribution of full support. We have q D Z (p) = |Z| for all q, by Lemma 5.2. But the computation theorem implies that D max (Z) = |Z B | for some B ⊆ {1, . . . , n} such that Z B admits a weighting, so D max (Z) ≤ |Z| by Lemma 10.2. Hence p is maximizing. Proposition 11.3 The following are equivalent: i. every maximizing distribution for Z has full support; ii. Z has exactly one maximizing distribution, which has full support; iii. Z is positive definite with positive weighting; For (i) =⇒ (iv), assume (i). Let ∅ = B {1, . . . , n}. Choose a maximizing distribution p for Z B , and denote by p its extension by zero to {1, . . . , n}.
Then p does not have full support, so there is some q ∈ [0, ∞] such that p fails to maximize q D Z . Hence where the second equality is by the absent species lemma.
For (iv) =⇒ (i), assume (iv). Let p be a maximizing distribution for Z. Write B = supp(p), and denote by p the restriction of p to B. Then for any q, again by the absent species lemma. Hence by (iv), B = {1, . . . , n}.

Open questions
The main theorem, the computation theorem and Corollary 6.4 answer all the principal questions about maximizing the diversities q D Z . Nevertheless, certain questions remain.
First, there are computational questions. We have found two classes of matrix Z for which the maximum diversity and maximizing distributions can be computed in polynomial time: ultrametric matrices (Example 10.5) and those close to the identity matrix I (Example 10.7). Both are biologically significant. Are there other classes of similarity matrix for which the computation can be performed in less than exponential time?
Second, we may seek results on maximization of q D Z (p) under constraints on p. There are certainly some types of constraint under which both parts of Theorem 6.1 fail, for trivial reasons: if we choose two distributions p and p whose diversity profiles cross (Figure 2(b)) and constrain our distribution to lie in the set {p, p }, then there is no distribution that maximizes q D Z for all q simultaneously, and the maximum value of q D Z also depends on q. But are there other types of constraint under which the main theorem still holds?
In particular, the distribution might be constrained to lie close to a given distribution p. The question then becomes: if we start with a distribution p and have the resources to change it by only a given small amount, what should we do in order to maximize the diversity?
Third, we have seen that every symmetric matrix Z satisfying (2) (for instance, every symmetric matrix of positive reals) has attached to it a real number, the maximum diversity D max (Z). What is the significance of this invariant?
We know that it is closely related to the magnitude of matrices. This has been most intensively studied in the context of metric spaces. By definition, the magnitude of a finite metric space X is the magnitude of the matrix Z = (e −d(i,j) ) i,j∈X ; see [24,27,30], for instance. In the metric context, the meaning of magnitude becomes clearer after one extends the definition from finite to compact spaces (which is done by approximating them by finite subspaces). Magnitude for compact metric spaces has recognizable geometric content: for example, the magnitude of a 3-dimensional ball is a cubic polynomial in its radius [2,Theorem 2], and the magnitude of a homogeneous Riemannian manifold is closely related to its total scalar curvature [41,Theorem 11].
Thus, it is natural to ask: can one extend Theorem 6.1 to some class of 'infinite matrices' Z? (For instance, Z might be the form (x, y) → e −d(x,y) arising from a compact metric space. In this case, the maximum diversity of order 2 is a kind of capacity, analogous to classical definitions in potential theory; for a compact subset of R n , it coincides with the Bessel capacity of an appropriate order [31].) And if so, what is the geometric significance of maximum diversity in that context?
There is already evidence that this is a fruitful line of enquiry. In [31], Meckes gave a definition of the maximum diversity of order 2 of a compact metric space, and used it to prove a purely geometric theorem relating magnitude to fractional dimensions of subsets of R n . If this maximum diversity can be shown to be equal to the maximum diversity of all other orders then further geometric results may come within reach.
The fourth and final question concerns interpretation. Throughout, we have interpreted q D Z (p) in terms of ecological diversity. However, there is nothing intrinsically biological about any of our results. For example, in an informationtheoretic context, the 'species' might be the code symbols, with two symbols seen as similar if one is easily mistaken for the other; or if one wishes to transmit an image, the 'species' might be the colours, with two colours seen as similar if one is an acceptable substitute for the other (much as in rate distortion theory [8]). Under these or other interpretations, what is the significance of the theorem that the diversities of all orders can be maximized simultaneously?