Bounded Rational Decision-Making from Elementary Computations That Reduce Uncertainty

In its most basic form, decision-making can be viewed as a computational process that progressively eliminates alternatives, thereby reducing uncertainty. Such processes are generally costly, meaning that the amount of uncertainty that can be reduced is limited by the amount of available computational resources. Here, we introduce the notion of elementary computation based on a fundamental principle for probability transfers that reduce uncertainty. Elementary computations can be considered as the inverse of Pigou–Dalton transfers applied to probability distributions, closely related to the concepts of majorization, T-transforms, and generalized entropies that induce a preorder on the space of probability distributions. Consequently, we can define resource cost functions that are order-preserving and therefore monotonic with respect to the uncertainty reduction. This leads to a comprehensive notion of decision-making processes with limited resources. Along the way, we prove several new results on majorization theory, as well as on entropy and divergence measures.


Introduction
In rational decision theory, uncertainty may have multiple sources that ultimately share the commonality that they reflect a lack of knowledge on the part of the decision-maker about the environment. A paramount example is the perfectly rational decision-maker [99] that has a probabilistic model of the environment and chooses its actions so as to maximize the expected utility entailed by the different choices. When we consider bounded rational decision-makers [87], we may add another source of uncertainty arising from the decision-maker's limited processing capabilities, since the decision-maker will not only accept a single best choice, but will accept any satisficing option. Today, bounded rationality is an active research topic that crosses multiple scientific fields such as economics, political science, decision theory, game theory, computer science, and neuroscience [81,66,59,12,31,63,45,89,101,42,91,92,90,46,16,70,58,1,30], where uncertainty is one of the most important common denominators.
Uncertainty is often equated with Shannon entropy in information theory [85], measuring the average number of yes/no-questions that have to be answered to resolve the uncertainty. Even though Shannon entropy has many desirable properties, there are plenty of alternative suggestions for entropy measures in the literature, known as generalized entropies, such as Rényi entropy [77] or Tsallis entropy [95]. Closely related to entropies are divergence measures, which express how probability distributions differ from a given reference distribution. If the reference distribution is uniform then divergence measures can be expressed in terms of entropy measures, which is why divergences can be viewed as generalizations of entropy, for example the Kullback-Leibler divergence [52] generalizing Shannon entropy.
Here, we introduce the concept of elementary computation based on a slightly stronger notion of uncertainty than is expressed by Shannon entropy, or any other generalized entropy alone, but is equivalent to all of them combined. Equating decision-making with uncertainty reduction, this leads to a new comprehensive view of decision-making with limited resources. Our main contributions can be summarized as follows: (i) Based on a fundamental concept of probability transfers related to the Pigou-Dalton principle of welfare economics [23], we promote a generalized notion of uncertainty reduction of a probability distribution that we call elementary computation. This leads to a natural definition of cost functions that quantify the resource costs for uncertainty reduction necessary for decision-making. We generalize these concepts to arbitrary reference distributions. In particular, we define Pigou-Dalton-type transfers for probability distributions relative to a reference or prior distribution, which induce a preorder that is slightly stronger than Kullback-Leibler divergence, but is equivalent to the notion of divergence given by all f -divergences combined. We prove several new characterizations of the underlying concept, known as relative majorization. (ii) An interesting property of cost functions is their behavior under coarsegraining, which plays an important role in decision-making and formalizes the general notion of making abstractions. More precisely, if a decision in a set Ω is split up into two steps by partitioning Ω = i A i and first deciding in the set of (coarse-grained) partitions {A i } i and secondly choosing a finegrained option inside the selected partition A i , then it is an important question how the cost for the total decision-making process differs from the sum of the costs in each step. We show that f -divergences are superadditive with respect to coarse-graining, which means that decision-making costs can potentially be reduced by splitting up the decision into multiple steps. In this regard, we find evidence that the well-known property of Kullback-Leibler divergence of being additive under coarse-graining might be viewed as describing the minimal amount of processing cost that cannot be reduced by a more intelligent decision-making strategy. (iii) We define bounded rational decision-makers as decision-making processes that are optimizing a given utility function under a constraint on the cost function, or minimizing the cost function under a minimal requirement on expected utility. As a special case for Shannon-type information costs, we arrive at information-theoretic bounded rationality, which may form a normative baseline for bounded-optimal decision-making in the absence of process-dependent constraints. We show that bounded-optimal posteriors with informational costs trace a path through probability space that can itself be seen as an anytime decision-making process, where each step optimally trades off utility and processing costs. (iv) We show that Bayesian inference can be seen as a decision-making process with limited resources given by the number of available datapoints. Section 2 deals with (i) and (ii), aiming at a general characterization of decisionmaking in terms of uncertainty reduction. Item (iii) is covered in Section 3, deriving information-theoretic bounded rationality as a special case. Section 4 illustrates the concepts with an example including item (iv). Sections 5 and 6 contain a general discussion and concluding remarks.

Notation
Let R denote the real numbers, R + := [0, ∞) the set of non-negative real numbers, and Q the rational numbers. We write |A| for the number of elements contained in a countable set A, and B \ A for the set difference, that is the set of elements in B that are not in A. P Ω denotes the set of probability distributions on a set Ω, in particular, any p ∈ P Ω is normalized so that p(Ω) = E p [1] = 1. Random variables are denoted by capital letters X, Y, Z, while their explicit values are denoted by small letters x, y, z. For the probability distribution of a random variable X we write p(X), and p(x) for the values of p(X). Correspondingly, the expectation E[f (X)] is also written as We also write f p := 1 n n i=1 f (x n ), to denote the approximation of E p [f ] by an average over samples {x 1 , . . . , x n } from p ∈ P Ω .

Decision-making with limited resources
In this section, we develop the notion of a decision-making process with limited resources following the simple assumption that any decision-making process (i) reduces uncertainty (ii) by spending resources. Starting from an intuitive interpretation of uncertainty and resource costs, these concepts are refined incrementally until a precise definition of a decision-making process is given at the end of this section (Definition 2.11) in terms of elementary computations. Here, a decision-making process is a comprehensive term that describes all kinds of biological as well as artificial systems that are searching for solutions to given problems, for example a human decision-maker that burns calories while thinking, or a computer that uses electric energy to run an algorithm. However, resources do not necessarily refer to a real consumable quantity but can also represent more explicit resources (like time) as a proxy, for example the number of binary comparisons in a search algorithm, the number of forward simulations in a reinforcement learning algorithm, the number of samples in a Monte Carlo algorithm, or, even more abstractly, they can express the limited availability of some source of information, like for example the number of datapoints that are available to an inference algorithm (see Section 4).
2.1. Uncertainty reduction by eliminating options. In its most basic form, the concept of decision-making can be formalized as the process of looking for a decision x ∈ Ω in a discrete set of options Ω = {x 1 , . . . , x N }. We say that a decision x ∈ Ω is certain, if repeated queries of the decision-maker will result in the same decision, and it is uncertain, if repeated queries can result in different decisions. Uncertainty reduction then corresponds to reducing the amount of uncertain options. Hence, a decision-making process that transitions from a space Ω of options to a strictly smaller subset A Ω reduces the amount of uncertain options from N = |Ω| to N A := |A| < N , with the possible goal to eventually find a single certain decision x * . Such a process is generally costly, the more uncertainty is reduced the more resources it costs ( Figure 1). The explicit mapping between uncertainty reduction and resource cost depends on the details of the underlying process and on which explicit quantity is taken as the resource. For example, if the resource is given by time (or any monotone function of time), then a search algorithm that eliminates options sequentially until the target value is found (linear search) is less cost efficient than an algorithm that takes a sorted list and in each step removes half of the options by comparing the mid point to the target (logarithmic search). Abstractly, any real-valued function C on the power set of Ω that satisfies C(A ) < C(A) whenever A A might be used as a cost function in the sense that C(A) quantifies the expenses of reducing the uncertainty from Ω to A ⊂ Ω.
In utility theory decision-making is modelled as an optimization process that maximizes a so-called utility function U : Ω → R (which can itself be an expected utility with respect to a probabilistic model of the environment, in the sense of von Neumann and Morgenstern [99]). A decision-maker that is optimizing a given utility function U obtains a utility of 1 on average after reducing the amount of uncertain options from N to N A < N (see Figure  2). A decision-maker that completely reduces uncertainty by finding the optimum x * = argmax x∈Ω U (x) is called rational (w.l.o.g. we can assume that x * is unique, by redefining Ω in the case when it is not). Since uncertainty reduction generally comes with a cost, a utility optimizing decision-maker with limited resources, correspondingly called bounded rational (see Section 3), in contrast will obtain only uncertain decisions from a subset A ⊂ Ω. Such decision-makers seek satisfactory rather than optimal solutions, for example by taking the first option that satisfies a minimal utility requirement, which Herbert Simon calls a satisficing solution [87].
Summarizing, we conclude that a decision-making process with decision space Ω that successively eliminates options can be represented by a mapping φ between subsets of Ω, together with a cost function C that quantifies the total expenses of arriving at a given subset, For example, a rational decision-maker can afford C({x * }), whereas a decisionmaker with limited resources can typically only afford uncertainty reduction with cost C(A) < C({x * }). From a probabilistic perspective, a decision-making process as described above is a transition from a uniform probability distribution over N options to a uniform probability distribution over N < N options, that converges to the Dirac measure δ x * centered at x * in the fully rational limit. From this point of view, the restriction to uniform distributions is artificial. A decision-maker that is uncertain about the optimal decision x * might indeed have a bias towards a subset A without completely excluding other options (the ones in A c = Ω\A), so that the behavior must be properly described by a probability distribution p ∈ P Ω . Therefore, in the following section, we extend (2.1) and (2.2) to transitions between probability Figure 2. Decision-making as utility optimization process.
distributions. In particular, we must replace the power set of Ω by the space of probability distributions on Ω, denoted by P Ω .

Probabilistic decision-making.
Let Ω be a discrete decision space of N = |Ω|< ∞ options, so that P Ω consists of discrete distributions p, often represented by probability vectors p = (p 1 , . . . , p N ). However, many of the concepts presented in this and the following section can be generalized to the continuous case [62,44]. Intuitively, the uncertainty contained in a distribution p ∈ P Ω is related to the relative inequality of its entries, the more similar its entries are, the higher the uncertainty. This means that uncertainty is increased by moving some probability weight from a more likely option to a less likely option. It turns out that this simple idea leads to a concept widely known as majorization [36,10,72,13,62,11], which has roots in the economic literature of the early 19th century [60,74,23], where it was introduced to describe income inequality, later known as the Pigou-Dalton Principle of Transfers. Here, the operation of moving weight from a more likely to a less likely option corresponds to the transfer of income from one individual of a population to a relatively poorer individual (also known as a Robin Hood operation [10]). Since a decision-making process can be viewed as a sequence of uncertainty reducing computations, we call the inverse of such a Pigou-Dalton transfer an elementary computation. Definition 2.1 (Elementary computation). A transformation on P Ω of the form T ε : p → (p 1 , . . . , p m + ε, . . . , p n − ε, . . . , p N ) , where m, n are such that p m ≤ p n , and 0 < ε ≤ pn−pm 2 , is called a Pigou-Dalton transfer (see Figure 3). We call its inverse T −1 ε an elementary computation.
Since making two probability values more similar or more dissimilar are the only two possibilities to minimally transform a probability distribution, elementary computations are the most basic principle of how uncertainty is reduced. Hence, we conclude that a distribution p has more uncertainty than a distribution p if and only if p can be obtained from p by finitely many elementary computations (and permutations, which are not considered an elementary computation due to the choice of ε). Definition 2.2 (Uncertainty). We say that p ∈ P Ω contains more uncertainty than p ∈ P Ω , denoted by p ≺ p , (2.4) if and only if p can be obtained from p by a finite number of elementary computations and permutations. The transfer of probability from a more likely to a less likely option increases uncertainty.
Note that, mathematically this defines a preorder on P Ω , i.e. a reflexive (p ≺ p for all p ∈ P Ω ) and transitive (if p ≺ p , p ≺ p then p ≺ p for all p, p , p ∈ P Ω ) binary relation. In the literature, there are different names for the relation between p and p expressed by Definition 2.2, for example p is called more mixed than p [79], more disordered than p [78], more chaotic than p [13], or an average of p [36]. Most commonly however, p is said to majorize p , which started with the early influences of Muirhead [65], and Hardy, Littlewood, and Pólya [36] and was developed by many authors into the field of majorization theory (a standard reference is by Marshall, Olkin, and Arnold [62]), with far reaching applications until today, especially in nonequilibrium thermodynamics and quantum information theory [15,41,34].
There are plenty of equivalent (arguably less intuitive) characterizations of p ≺ p , some of which we are summarizing below. However, one characterization makes use of a concept very closely related to Pigou-Dalton transfers, known as T-transforms [13,62], which expresses the fact that moving some weight from a more likely option to a less likely option is equivalent to taking (weighted) averages of the two probability values. More precisely, a T-transform is a linear operator on P Ω with a matrix of the form T = (1 − λ)I + λΠ, where I denotes the identity matrix on R N , Π denotes a permutation matrix of two elements, and 0 ≤ λ ≤ 1. If Π permutes p m and p n , then (T p) k = p k for all k ∈ {m, n}, and (T p) m = (1 − λ)p m + λp n , (T p) n = λp m + (1 − λ)p n . (2.5) Hence, a T-transform considers any two probability values p m and p n of a given p ∈ P Ω , calculates their weighted averages with weights (1−λ, λ) and (λ, 1−λ), and replaces the original values with these averages. From (2.5) it follows immediately that a T-transform with parameter 0 < λ ≤ 1 2 and a permutation Π of p m , p n with p m ≤ p n is a Pigou-Dalton transfer with ε = (p n − p m )λ. Also allowing 1 2 ≤ λ ≤ 1 means that T-transfers include permutations, in particular, p ≺ p if and only if p can be derived from p by successive applications of finitely many T-transforms Due to a classic result by Hardy, Littlewood and Pólya [36, p.49], this characterization can be stated in an even simpler form by using doubly stochastic matrices, i.e. matrices A = (A ij ) i,j with A ij ≥ 0 and i A ij = 1 = j A ij for all i, j. By writing xA := A T x for all x ∈ R N , and e := (1, . . . , 1), these conditions are often stated as A ij ≥ 0 , Ae = e , eA = e .
(2.6) Note that doubly stochastic matrices can be viewed as generalizations of T-transforms in the sense that a T-transform takes an average of two entries, whereas if p = pA with a doubly stochastic matrix A, then p j = i A ij p i is a convex combination, or a weighted average, of p with coefficients (A ij ) i for each j. This is also, why p is then called more mixed than p [79]. Therefore, similar to T-transforms, we Figure 4. Comparability of probability distributions in N = 3. The region in the center consists of all p that are majorized by p, i.e. p p , whereas the outer region consists of all p that majorize p, p ≺ p . The bright regions are not comparable to p.
might expect that if p is the result of an application of a doubly stochastic matrix, p = pA, then p is an average of p and therefore contains more uncertainty than p. This is exactly what is expressed by characterization (iii) in the following theorem. A similar characterization of p ≺ p is that p must be given by a convex combination of permutations of the elements of p (see property (iv) below). Without having the concept of majorization, Schur proved that functions of the form p → i f (p i ) with a convex function f are monotone with respect to the application of a doubly stochastic matrix [83] (see property (v) below). Functions of this form are an important class of cost functions for probabilistic decision-makers, as we will discuss in Example 2.2. Theorem 2.3 (Characterizations of p ≺ p [62]). For p, p ∈ P Ω , the following are equivalent: (i) p ≺ p, i.e. p contains more uncertainty than p (Definition 2.2) (ii) p is the result of finitely many T-transforms applied to p (iii) p = pA for a doubly stochastic matrix A where p ↓ denotes the decreasing rearrangement of p.
As argued above, the equivalence between (i) and (ii) is straight-forward. The equivalences between (ii), (iii), and (vi) are due to Muirhead [65] and Hardy, Littlewood, and Pólya [36]. The implication (v) ⇒ (iii) is due to Karamata [47] and Hardy, Littlewood, and Pólya [37], whereas (iii) ⇒ (v) goes back to Schur [83]. Mathematically, (iv) means that p belongs to the convex hull of all permutations of the entries of p, and the equivalence (iii) ⇔ (iv) is known as the Birkhoffvon Neumann theorem. Here, we have stated all relations for probability vectors p ∈ P Ω , even though they are usually stated for all p, p ∈ R N with the additional requirement that  more uncertainty than uniform distributions over N < N options, since we have In particular, if A ⊂ A ⊂ Ω, then the uniform distribution over A contains less uncertainty than the uniform distribution over A , which shows that the notion of uncertainty introduced in Definition 2.2 is indeed a generalizatin of the notion of uncertainty given by the number of uncertain options introduced in the previous section.
Note that, ≺ only being a preorder on P Ω , in general, two distributions p , p ∈ P Ω are not necessarily comparable, i.e. we can have both p ≺ p and p ≺ p . In Figure 4, we visualize the regions of all comparable distributions for two exemplary distributions on a three-dimensional decision space (N = 3), represented on the two-dimensional simplex of probability vectors p = (p 1 , p 2 , p 3 ). For example, can not be compared under ≺, since 1 2 > 2 5 , but 3 4 < 4 5 . Cost functions can now be generalized to probabilistic decision-making by noting that the property C(A ) < C(A) whenever A A in (2.2) means that C is strictly monotonic with respect to the preorder given by set inclusion. Definition 2.4 (Cost functions on P Ω ). We say that a function C : P Ω → R + is a cost function, if it is strictly monotonically increasing with respect to the preorder ≺, i.e. if p ≺ p ⇒ C(p ) ≤ C(p) , (2.8) with equality only if p and p are equivalent, p ∼ p, which is defined as p ≺ p and p ≺ p . Moreover, for a parametrized family of posteriors (p r ) r∈I , we say that r is a resource parameter with respect to a cost function C, if the mapping I → R + , r → C(p r ) is strictly monotonically increasing.
Since monotonic functions with respect to majorization were first studied by Schur [83], functions with this property are usually called (strictly) Schur-convex [62,Ch. 3].
Example (Generalized entropies). From (v) in Theorem 2.3 it follows that functions of the form where f is strictly convex, are examples of cost functions. Since many entropy measures used in the literature can be seen to be special cases of (2.9) (with a concave f ), functions of this form are often called generalized entropies [18]. In particular, for the choice f (t) = t log t, we have C(p) = −H(p), where H(p) denotes the Shannon entropy of p. Thus, if p contains more uncertainty than p in the sense of Definition 2.2 (p ≺ p) then the Shannon entropy of p is larger than the Shannon entropy of p and therefore p contains also more uncertainty in the sense of classical information theory than p. Similarly, for f (t) = − log(t) we obtain the (negative) Burg entropy, and for functions of the form f (t) = ±t α for α ∈ R\{0, 1} we get the (negative) Tsallis entropy, where the sign is chosen depending on α such that f is convex (see e.g. [32] for more examples). Moreover, the composition of any (strictly) monotonically increasing function g with (2.9) generates another class of cost functions, which contains for example the (negative) Rényi entropy [77]. Note also that entropies of the form (2.9) are special cases of Csiszár's f-divergences [20] for uniform reference distributions (see Example 2.3 below). In Figure 5, several examples of cost functions are shown for N = 3. In this case, the 2-dimensional probability simplex P Ω is given by the triangle in R 3 with edges (1, 0, 0), (0, 1, 0), and (0, 0, 1). Cost functions are visualized in terms of their level sets. We prove in Proposition A.1 in the appendix that all cost functions of the form (2.9) are superadditive with respect to coarse-graining. This seems to be a new result and an improvement upon the fact that generalized entropies (and f -divergences) satisfy information monotonicity [5]. More precisely, if a decision in Ω, represented by a random variable Z, is split up into two steps by partitioning Ω = i∈I A i and first deciding about the partition i ∈ I, correspondingly described by a random variable X with values in I, and then choosing an option inside of the selected where C(X) := C(p(X)) and C(Y |X) := E p(X) [C(p(Y |X))]. For symmetric cost functions (such as (2.9)) this is equivalent to The case of equality in (2.10) and (2.11) (see Figure 6) is sometimes called separability [51], strong additivity [21], or recursivity [4], and it is often used to characterize Shannon entropy [26,96,48,55,77,2]. In fact, we also show in the appendix (Proposition A.2) that cost functions C that are additive under coarse-graining are proportional to the negative Shannon entropy −H. See also Example 2.3 in the next section, where we discuss the generalization to arbitrary reference distributions.
We can now refine the notion of a decision-making process introduced in the previous section as a mapping φ together with a cost function C satisfying (2.2). Instead of simply mapping from sets A to smaller subsets A A by successively eliminating options, we now allow φ to be a mapping between probability distributions such that φ(p) can be obtained from p by a finite number of elementary computations (without permutations), and we require C to be a cost function on P Ω , so that Here, C(p) quantifies the total costs of arriving at a distribution p, and p p means that p ≺ p and p ≺ p . In other words, a decision-making process can be viewed as traversing probability space by moving pieces of probability from one option to another option such that uncertainty is reduced.
Up to now we have ignored one important property of a decision-making process, the distribution q with minimal cost, i.e. satisfying C(q) ≤ C(p) for all p, which must be identified with the initial distribution of a decision-making process with Figure 6. Additivity under coarse-graining. If the cost for Z = (X, Y ) is the sum of the costs for X and the cost for Y given X, then the cost function is proportional to Shannon entropy.
cost function C. As one might expect (see Figure 5), it turns out that all cost functions according to Definition 2.4 have the same minimal element.
for all p, in particular the uniform distribution corresponds to the initial state of all decisionmaking processes with cost function C satisfying (2.12). In particular, it contains the maximum amount of uncertainty with respect to any entropy measure of the form (2.9), known as the second Khinchin axiom [51], e.g. for Shannon entropy 0 ≤ H(p) ≤ log N . Proposition 2.5 follows from characterization (iv) in Theorem 2.3 after noticing that every p ∈ P Ω can be transformed to a uniform distribution by permuting its elements cyclically (see Proposition A.3 in the appendix for a detailed proof).
Regarding the possibility that a decision-maker may have prior information, for example originating from the experience of previous comparable decision-making tasks, the assumption of a uniform initial distribution seems to be artificial. Therefore, in the following section we arrive at the final notion of a decision-making process by extending the results of this section to allow for arbitrary initial distributions.
2.3. Decision-making with prior knowledge. From the discussion at the end of the previous section we conclude that, in full generality, a decision-maker transitions from an initial probability distribution q ∈ P Ω , called prior, to a terminal distribution p ∈ P Ω , called posterior. Note that, since once eliminated options are excluded from the rest of the decision-making process, a posterior p must be absolutely continuous with respect to the prior q, denoted by p q, i.e. p(x) can be non-zero for a given x ∈ Ω only if q(x) is non-zero.
The notion of uncertainty (Definition 2.2) can be generalized with respect to a non-uniform prior q ∈ P Ω by viewing the probabilities q i as the probabilities Q(A i ) of partitions A i of an underlying elementary probability spaceΩ = i A i of equally likely elements under Q, in particular Q represents q as the uniform distribution oñ Ω (see Figure 7). The similarity of the entries of the corresponding representation P ∈ PΩ of any p ∈ P Ω (its uncertainty) then contains information about how close p is to q, which we call the relative uncertainty of p with respect to q (Definition 2.6 below).
The formal construction is as follows: Let p, q ∈ P Ω be such that p q and q i ∈ Q. The case when q i ∈ R then follows from a simple approximation of each Figure 7. Representation of q and p by Q and P onΩ (Example 2.3), such that the probabilities q i and p i are given by the probabilities of the partitions A i with respect to Q and P , respectively. entry by a rational number. Let α ∈ N be such that α q i ∈ N for all i ∈ {1, . . . , N }, for example α could be chosen as the least common multiple of the denominators of the q i . The underlying elementary probability spaceΩ then consists of α elements and there exists a partitioning {A i } i=1,...,N ofΩ such that where Q denotes the uniform distribution onΩ. In particular, it follows that i.e. Q represents q inΩ with respect to the partitioning {A i } i . Similarly, any p ∈ P Ω can be represented as a distribution onΩ by requiring that P (A i ) = p i for all i ∈ {1, . . . , N } and letting P to be constant inside of each partition, i.e. similar to (2.15) we have P (A i ) = |A i | P (ω) = p i for all ω ∈ A i and therefore by (2.14) Note that, if q i = 0 then p i = 0 by absolute continuity (p q) in which case we can either exclude option i from Ω or set P (ω) = 0.
Then q i corresponds to the probability of the i-th partition under the uniform distribution Q = 1 6 (1, . . . , 1), Figure 7). Importantly, if the components of the representation Λ q p := P in PΩ given by (2.16) are similar to each other, i.e. if P is close to uniform, then the components of p must be very similar to the components of q, which we express by the concept of relative uncertainty. Definition 2.6 (Uncertainty relative to q). We say that p ∈ P Ω contains more uncertainty with respect to a prior q ∈ P Ω than p ∈ P Ω , denoted by p ≺ q p, if and only if Λ q p contains more uncertainty than Λ q p, i.e.
As we will show in Theorem 2.7 below, it turns out that ≺ q coincides with a known concept called q-majorization [97], majorization relative to q [44,62], or mixing distance [80]. Due to the lack of a characterization by partial sums, it is usually defined as a generalization of characterization (iii) in Theorem 2.3, that is p is q-majorized by p iff p = pA, where A is a so-called q-stochastic matrix, which means that it is a stochastic matrix (Ae = e) with qA = q. In particular, ≺ q does not depend on the choice of α in the definition of Λ q . Here, we provide two new characterizations of q-majorization, the one given by Definition 2.6, and one using partial sums generalizing the original definition of majorization.
Theorem 2.7 (Characterizations of p ≺ q p). The following are equivalent (i) p ≺ q p, i.e. p contains more uncertainty relative to q than p (Def. 2.6) (ii) Λ q p can be obtained from Λ q p by a finite number of elementary computations and permutations on PΩ To prove that (i), (iii), and (v) are equivalent (see Proposition A.4 in the appendix), we use of the fact that Λ q : P Ω → PΩ has a left inverse Λ −1 q : Λ q (P Ω ) → P Ω . This can be verified by simply multiplying the corresponding matrices given in the proof of Proposition A.4. The equivalence betweeen (iii) and (iv) is shown in [44] (see also [80,62]). Characterization (ii) follows immediately from Definition 2.2 and Definition 2.6.
As required from the discussion at the end of the previous section, q is indeed minimal with respect to ≺ q , which means that it contains the most amount of uncertainty with respect to itself.
Proposition 2.8 (The prior is minimal). The prior q ∈ P Ω is the unique minimal element in P Ω with respect to ≺ q , that is This follows more or less directly from Proposition 2.5 and the equivalence of (i) and (iii) in Theorem 2.7 (see Proposition A.5 in the appendix for a detailed proof).
Order-preserving functions with respect to ≺ q generalize cost functions introduced in the previous section (Definition 2.4). According to Proposition 2.8, such functions have a unique minimum given by the prior q. Since cost functions are used in Definition 2.11 below to quantify the expenses of a decision-making process, we require their minimum to be zero, which can always be achieved by redefining a given cost function by an additive constant. Definition 2.9 (Cost functions relative to q). We say that a function C q : 19) with equality only if p ∼ q p, i.e. if p ≺ q p and p ≺ q p . Moreover, for a parametrized family of posteriors (p r ) r∈I , we say that r is a resource parameter with respect to Example (f -divergences). From (iv) in Theorem 2.7 it follows that functions of the form where f is continuous and strictly convex with f (1) = 0, are examples of cost functions relative to q. Many well-known divergence measures can be seen to belong to this class of relative cost functions, also known as Csiszár's f -divergences [20]: the Kullback-Leibler divergence (or relative entropy), the squared 2 distance, the Hartley entropy, the Burg entropy, the Tsallis entropy, and many more [32,21] (see Figure 8 for visualizations of some of them in N = 3 relative to a non-uniform prior). As a generalizition of Proposition A.1 (superadditivity of generalized entropies), we prove in Proposition A. 6 in the appendix that f -divergences are superadditive under coarse-graining, that is, for Z = (X, Y ) This generalizes (2.10) to the case of a non-uniform prior. Similar to entropies, the case of equality in (2.21) is sometimes called composition rule [40], chain rule [57], or recursivity [21], and is often used to characterize Kullback-Leibler divergence [40,63,21,57]. Indeed, we also show in the appendix (Proposition A.7) that all additive cost functions with respect to q are proportional to Kullback-Leibler divergence (relative entropy). This goes back to Hobson's modification [40] of Shannon's original proof [85], after establishing the following monotonicity property for uniform distributions: If f (M, N ) denotes the cost C u N (u M ) of a uniform distribution u M over M elements relative to a uniform distribution u N over N ≥ M elements, then (see Figure 9) Note that, even though our proof of Proposition A.7 uses additivity under coarse graining to show the monotonicity property (2.22), it is easy to see that any relative In terms of decision-making, superadditivity under coarse-graining means that decision-making costs can potentially be reduced by splitting up the decision into multiple steps, for example by a more intelligent search strategy. For example, if N = 2 k for some k ∈ N and C q is superadditive, then the cost for reducing uncertainty to a single option, i.e. p = (1, 0, . . . , 0), when starting from a uniform distribution q, satisfies where q n := ( 1 n , . . . , 1 n ), and we have set C q 2 (1, 0) = 1 as unit cost (corresponding to 1 bit in the case of Kullback-Leibler divergence). Thus, intuitively the property of the Kullback-Leibler divergence of being additive under coarse-graining might be viewed as describing the minimal amount of processing costs that must be contained in any cost function, because it cannot be reduced by changing the decision-making process. Therefore, in the following we call cost functions that are proportional to the Kullback-Leibler divergence simply informational costs.
In contrast to the previous section, in the definition of ≺ q and its characterizations we have never used elementary computations on P Ω directly. This is due to the fact that permutations do interact with the uncertainty relative to q, and therefore ≺ q cannot be characterized by a finite number of elementary computations and permutations on P Ω . However, we can still define elementary computations relative to q by the inverse of Pigou-Dalton transfers T ε of the form (2.3) such that T ε p q p for ε > 0, which is arguably the most basic form of how to generate uncertainty with respect to q.
Even for small ε, a regular Pigou-Dalton transfer does not necessarily increase uncertainty relative to q, because the similarity of the components now needs to be considered with respect to q. Instead, we compare the components of the representation P = Λ q p of p ∈ P Ω , and move some probability weight ε ≥ 0 from P (A n ) to P (A m ) whenever P (ω) ≤ P (ω ) for ω ∈ A m and ω ∈ A n , by distributing ε evenly among the elements in A m (see Figure 10), denoted by the transformationT ε . Here, ε must be small enough such that the inequality 1 (2.23) Figure 10. Pigou-Dalton transfer relative to q. A distribution p ∈ P Ω is transformed relative to q by first moving some amount of weight ε ≥ 0 from P (A n ) to P (A m ) where n, m are such that P (ω) ≤ P (ω ) whenever ω ∈ A m and ω ∈ A n , with ε small enough such that this relation remains true after the transformation, and then mapping the transformed distribution back to P Ω by Λ −1 q (see Definition 2.10).
By construction,T ε minimally increases uncertainty in PΩ while staying in the image of P Ω under Λ q , by keeping the values of P constant in each partition, and therefore T ε := Λ −1 qTε Λ q can be considered as the most basic way of how to increase uncertainty relative to q. Definition 2.10 (Elementary computation relative to q). We call a transformation on P Ω of the form T ε : p → (p 1 , . . . , p m + ε, . . . , p n − ε, . . . , p N ) , (2.24) with m, n such that pm qm ≤ pn qn , and ε satisfying (2.23), a Pigou-Dalton transfer relative to q, and its inverse an elementary computation relative to q.
We are now in the position to state our final definition of a decision-making process.
Definition 2.11 (Decision-making process). A decision-making process is a gradual transformation of a prior q ∈ P Ω to a posterior p ∈ P Ω , such that each step decreases uncertainty relative to q. This means that p is obtained from q by successive application of a mapping φ between probability distributions on Ω, such that φ(p ) can be obtained from p by finitely many elementary computations relative to q, in particular where C q (p ) quantifies the total costs of a distribution p , and p q p means that p ≺ q p and p ≺ q p .
In other words, a decision-making process can be viewed as traversing probability space from prior q to posterior p by moving pieces of probability from one option to another option such that uncertainty is reduced relative to q, while expending a certain amount of resources determined by the cost function C q .

Bounded rationality
3.1. Bounded rational decision-making. In this section, we consider decisionmaking processes that trade off utility against costs. Such decision-makers either maximize a utility function subject to a constraint on the cost function, for example an author of a scientific article that optimizes the article's quality until a deadline is reached, or minimizing the cost function subject to a utility constraint, for example a high-school student that minimizes effort such that the requirement to pass a certain class is achieved. In both cases, the decision-makers are called bounded rational, since in the limit of no resource constraints they coincide with rational decision-makers.
In general, depending on the underlying system, such an optimization process might have additional process dependent constraints that are not directly given by resource limitations, for example in cases when the optimization takes place in a parameter space that has less degrees of freedom than the full probability space P Ω . Abstractly, this is expressed by allowing the optimization process to search only in a subset Γ ⊂ P Ω . Definition 3.1 (Bounded rational decision-making process). Let U : Ω → R be a given utility function, and Γ ⊂ P Ω . A decision-making process with prior q, posterior p * ∈ Γ, and cost function C q is called bounded rational if its posterior satisfies for a given upper bound C 0 ≥ 0, or equivalently for a given lower bound U 0 ∈ R. In the case when the process constraints disappear, i.e. if Γ = P Ω , then a bounded rational decision-maker is called bounded-optimal.
The equivalence between (3.1) and (3.2) is easily seen from the equivalent optimization problem given by the formalism of Lagrange multipliers [25], where the cost or utility constraint is expressed by a trade-off between utility and cost, or cost and utility, with a trade-off parameter given by the Lagrange multiplier β, which is chosen such that the constraint given by C 0 or U 0 is satisfied. It is easily seen from the maximization problem on the right side of (3.3) that a larger value of β decreases the weight of the cost term and thus allows for higher values of the cost function. Hence, β parametrizes the amount of resources the decision-maker can afford with respect to the cost function C q , and, at least in non-trivial cases (non-constant utilities) it is therefore a resource parameter with respect to C q in the sense of Definition 2.9. In particular, for β → 0, the decision-maker minimizes its cost function irrespective of the expected utility, and therefore stays at the prior, p 0 = q, whereas β → ∞ makes the cost function disappear so that the decisionmaker becomes purely rational with a Dirac posterior centered on the optima x * of the utility function U . For example, in Figure 11 we can see how the posteriors (p β ) β≥0 of boundedoptimal decision-makers with different cost functions for N = 3 and with utility  This generalizes a result in [78] to the case of non-uniform priors, by making use of our new characterization (v) of ≺ q , by which it suffices to show that the function β → l−1 i=1 (p β,i ) ↓ + a q (k, l)(p β,l ) ↓ is monotonically increasing for all k, l specified in Theorem 2.7 (see Proposition A.8 in the appendix for details). For simplicity, we restrict ourselves to the case of the Kullback-Leibler divergence, however the proof is analogous for cost functions of the form (2.20) with f being differentiable and strictly convex on [0, 1] (so that f is strictly monotonically increasing and thus invertible on [0, 1], see [78] for the case of uniform priors).
Hence, for any β > 0, the posteriors (p β ) β <β of a bounded-optimal decisionmaking process with the Kullback-Leibler divergence as cost function can be regarded as the steps of a decision-making process (i.e. satisfying (2.25)) with posterior p β , where each step optimally trades off utility against informational cost. This means that with increasing β the posteriors p β do not only decrease entropy in the sense of the Kullback-Leibler divergence, but also in the sense of any other cost function.
The important case of bounded-optimal decision-makers with informational costs is termed information-theoretic bounded rationality [68,92,70] and is studied more closely in the following sections.
3.2. Information-theoretic bounded rationality. For bounded-optimal decisionmaking processes with informational costs the unconstrained optimization problem (3.3) takes the form max p∈PΩ F[p], where which has a unique maximum p β , the bounded-optimal posterior given by with normalization constant Z β . This form can easily be derived by finding the zeros of the functional derivative of the objective functional (3.5) with respect to p (with an additional normalization constraint), whereas the uniqueness follows from the convexity of the mapping p → D KL (p q). For the actual maximum of F we obtain Due to its analogy with physics, in particular thermodynamics (see e.g. [70]), the maximization of (3.5) is known as the Free Energy principle of informationtheoretic bounded rationality, pioneered in [68,92,70], further developed in [29,33], and applied in recent studies of artificial systems, such as generative neural networks trained by Markov chain Monte Carlo methods [38], or in reinforcement learning as an adaptive regularization strategy [56,35], as well as in recent experimental studies on human behavior [71,82]. Note that there is a formal connection of (3.5) and the Free Energy principle of active inference [28], however, as discussed in [33,Sec. 6

.3], both Free Energy principles have conceptually different interpretations.
Example (Bayes rule as a bounded-optimal posterior). In Bayesian inference, the parameter θ of the distribution p θ of a random variable Y is inferred from a given dataset d = {y 1 , . . . , y N } of observations of Y by treating the parameter itself as a random variable Θ with a prior distribution q(Θ). The parametrized distribution of Y evaluated at an observation y i ∈ d given a certain value of Θ, i.e. p(y i |Θ=θ), is then understood as a function of θ, known as the likelihood of the datapoint y i under the assumption of Θ = θ. After seeing the dataset d, the belief about Θ is updated by using Bayes rule This takes the form of a bounded-optimal posterior (3.6) with β = N and utility function given by the average log-likelihood per datapoint, since then Bayes rule reads The corresponding Free Energy (3.5), that is maximized by (3.7), coincides with the variational Free Energy F var from Bayesian statistics. Indeed, from (3.8) it is easy to see that F assumes its maximum when p(Θ) is proportional to q(Θ)p(d|Θ), that is when p(Θ) is given by (3.7). In the literature, F var is used in the variational characterization of Bayes rule, in cases when the form (3.7) cannot be achieved exactly but instead is approximated by optimizing (3.8) over the parameters ϑ of a parametrized distribution p ϑ (Θ) [39,61]. The decision about Z = (X, Y ) is equivalent to a two-step process consisting of the decision about X and the decision about Y given X. The objective function for the first step is the Free Energy of the second step.
In the following section, we will see that the Free Energy F of a bounded rational decision-making process satisfies a recursivity property, which allows the interpretation of F as a certainty-equivalent.
3.3. The recursivity of F and the value of a decision problem. Consider a bounded-optimal decision-maker with an informational cost function deciding about a random variable Z with values in Ω that is decomposed into the random variables X and Y , i.e. Z = (X, Y ). This decomposition can be understood as a two-step process, where first a decision about a partition A i of the full search space Ω = i∈I A i is made, represented by a random variable X with values in I, followed by a decision about Y inside the partition selected by X (see Figure 6).
Since p(Z) = p(X)p(Y |X), by the additivity of the Kullback-Leibler divergence (Proposition A.7), we have denotes the Free Energy of the second step, In particular, the Free Energy F β [p(Y |X)] of the second decision-step plays the role of the utility function of the first decision-step (see Figure 12). In Equation (3.9), the two decision-steps have the same resource parameter β, controlling the strength of the constraint on the total informational costs More generally, each step might have a separate information-processing constraint, which requires two resource parameters β 1 and β 2 , and results in the total Free Energy Example. Consider a bounded-rational decision-maker with informational cost function and a utility function U defined on a set Ω = {z 1 , . . . , z 4 } with values as given in Figure 13 and an information-processing bound of 0.2 bits (β ≈ 0.9). If we partition Ω into the disjoint subsets {z 1 , z 2 } and {z 3 , z 4 }, then the decision about Z can be decomposed into two steps, Z = (X, Y ), the decision about X corresponding to the choice of the partition and the decision about Y given X corresponding to the choice of z i inside the given partition determined by X. According to Equation (3.9), the choice of the partition X = x i is not in favor of the achieved expected utility inside each partition, but of the Free Energy (see Figure 13). Therefore, a bounded rational decision-maker that has the choice among decisionproblems ideally should base its decision not on the expected utility that might be achieved but on the Free Energy of the subordinate problems. In other words, the Free Energy quantifies the value of a decision-problem, that besides of the achieved average utility also takes the information-processing costs into account.

3.4.
Multi-task decision-making and the optimal prior. So far, we have considered decision-making problems with utility functions defined on Ω only, modelling a single decision-making task. This is extended to multi-task decision-making problems by utility functions of the form U : where the additional variable w ∈ W represents the current state of the world. Different world states w in general lead to different optimal decisions x * (w) := argmax x∈Ω U (w, x). For example, in a chess game the optimal moves depend on the current board configurations the players are faced with.
The prior q for a bounded-rational multi-task decision-making problem may either depend or not depend on the world state w ∈ W. In the first case, the multi-task decision-making problem is just given by multiple single-task problems, i.e. for each w ∈ W, q(X|W = w) and p(X|W = w) are the prior and posterior of a bounded rational decision-making process with utility function x → U (x, w), as described in the previous sections. In the case when there is a single prior for all world states w ∈ W, the Free Energy is where p(W ) is a given world state distribution. Note that, for simplicity we assume that β is independent of w ∈ W, which means that only the average informationprocessing is constrained, in contrast to the information-processing being constrained for each world state which in general would result in β being a function of w. Similarly to single-task decision-making (Equation (3.6)) the maximum of (3.10) is achieved by with normalization constant Z β (w). Interestingly, the deliberation cost in Similarly to (3.6), Equation (3.12) follows from finding the zeros of the functional derivative of the Free Energy with respect to q(X) (modified by an additional term for the normalization constraint).
Optimizing the Free Energy F[p(X|W ), q(X)] for both prior and posterior can be achieved by iterating Equations (3.11) and (3.12). This results in an alternating optimization algorithm, originally developed independently by Blahut and Arimoto to calculate the capacity of a memoryless channel [14,9] (see [22] for a convergence proof by Csiszár and Tusnády). Note that in particular, that the information-processing cost is now given by the mutual information I(W ; X) between the random variables W and X. In this form, we can see that the Free Energy optimization with respect to prior and posterior is equivalent to the optimization problem in classical rate distortion theory [86], where U is given by the negative of the distortion measure. Similarly as in rate-distortion theory, where compression algorithms are anaylized with respect to the rate-distortion function, any decision-making system can now be analyzed with respect to informational bounded-optimality. More precisely, when plotting the achieved expected utility against the information-processing resources of a bounded-rational decision-maker with optimal prior, we obtain a pareto-optimality curve that forms an efficiency-frontier that cannot be surpassed by any decision-making process (see Figure 14(c)).

Multi-task decision-making with unknown world state distribution.
A bounded rational decision-making process with informational cost and utility U : W × Ω → R that has an optimal prior q * (X) given by the marginal (3.12) must have perfect knowledge about the world state distribution p(W ). In contrast, here we consider the case when the exact shape of the world state distribution is unknown to the decision-maker and therefore has to be inferred from the already seen world states. More precisely, we assume that the world state distribution is parametrized by a parameter θ ∈ R, i.e. p(W ) = p θtrue (W ) for a given parametrized distribution p θ (W ). Since the true parameter θ true is unknown, θ is treated as a random variable by itself, so that p θ (W ) = p(W |Θ = θ). After a dataset d = {w 1 , . . . , w N } ∈ W N of samples from p(W |Θ = θ true ) has been observed the joint distribution of all involved random variables can be written as where p(Θ) denotes the decision-maker's prior belief about Θ, and p(D = d|Θ) = N i=1 p(w i |Θ) is the likelihood of the previously observed world states. Therefore, the resulting (multi-task) Free Energy (see Equation (3.10)) is given by It turns out that we obtain Bayesian inference as a byproduct of optimizing (3.13) with respect to the prior q(X|D). Indeed, by calculating the functional derivative with respect to q(X|D) of the Free Energy (3.13) plus an additional term for the normalization constraint of q(X|D) (with Lagrange multiplier λ), we can see that any distribution q * (X|D) that optimizes (3.13) must satisfy where λ ∈ R is chosen such that q * (X|D=d) ∈ P Ω for any d ∈ W N . This is equivalent to where Z D denotes the normalization constant of q * (X|D), given by with p * (Θ|D) as defined in Equation (3.14). Hence, we have shown (3.14) 4. Example: Absolute identification task with known and unknown stimulus distribution Consider a bounded rational decision-maker with a multi-task utility function U such that for each w ∈ W, U (w, x) is non-zero for only one choice x ∈ Ω as shown in Figure 14. Here, the decision and world spaces are both finite sets of N = 20 elements. The world state distribution p(W ) is given by a mixture of two Gaussian distributions as shown in Figure 14(b). Due to some world states w ∈ W being more likely than others, there are some choices x ∈ Ω that are less likely to be optimal. 4.1. Known stimulus distribution. As can be seen in Figure 14(c) (dashed line), here it is not ideal to have a uniform prior distribution, q(x) = 1 N for all x ∈ Ω. Instead, if the world state distribution is known perfectly and the prior has the form suggested by Equation (3.12), i.e. q(x) = w p(w)p β (x|w), then, as can be seen in Figure 14(c) (solid line), achieving the same expected utility as with a uniform prior requires less informational resources. In particular, the explicit form of q * depends on the resource parameter β, see Figure 14(d). For low resource availability (small β), only the choices that correspond to the most probable world states are considered. However, for β → ∞, we have  Figure 14. Absolute identification task with known world state distribution. (a) Utility function, (b) world states distribution (a mixture of two Gaussians), (c) expected utility as a function of information-processing resources for a bounded-optimal decisionmaker with a uniform and with an optimal prior (the shaded region cannot be reached by any decision-making process), and (d) exemplary optimal priors q * (X) for different information-processing bounds.
because here lim β→∞ p β (x|w) = δ w,x is the posterior of a rational decision-maker, where δ w,x denotes the Kronecker-δ (which is only non-zero if w=x). Hence, for decision-makers with abundant information-processing resources (large β) the optimal prior q * (X) approaches the form of the world state distribution p(W ) (since here W = Ω).

Unknown stimulus distribution.
In the case when the decision-maker has to infer its knowledge about p(W ) from a set of samples d = {w 1 , . . . , w N }, we know from Section 3.5 that this is optimally done via Bayesian inference. Here, we assume a mixture auf two Gaussians as a parametrization of p(W ), so that θ = (µ 1 , µ 2 , σ 1 , σ 2 ), where µ i and σ i denote mean and standard-deviation of the i-th component respectively (for simplicity, with fixed equal weights for the two mixture components).
In Figure 15(a), we can see how different values of N affect the belief about the world state distribution, p(W |D) = E p(Θ|D) [p(W |Θ)], when p(Θ|D) is given by the Bayes posterior (3.14) with a uniform prior belief p(Θ). The resulting expected utilities (averaged over samples from p(D|θ true )) as functions of available information-processing resources are displayed in Figure 15(b), which shows how the performance of a bounded-rational decision-maker with optimal prior and perfect knowledge about the true world state distribution is approached by bounded rational decision-makers with limited but increasing knowledge given by the sample size N . Average expected utility Expected utility as a function of information optimal prior, N = 30 optimal prior, N = 10 optimal prior, N = 4 optimal prior, N = 1 Figure 15. Absolute identification task with unknown world state distribution. (a) Average of inferred world state distributions for different sizes N of datasets (standard-deviations across datasets indicated by error bars). (b) Resulting utility-information curves of a bounded-rational decision-maker with optimal prior that has to infer the world state distribution from datasets with different sizes N .
Abstractly, we can view Equation (3.14) as the bounded optimal solution to the decision-making problem that starts with a prior p(Θ) and arrives at a posterior p(Θ|D = d) after processing the samples in d = {w 1 , . . . , w N } (see also Example 3.2). In fact, the posteriors shown in Figure 15(a) satisfy the requirements for a decision-making process with resource given by the number of datapoints N , when averaged over p(D). In particular, by increasing N the posteriors contain less and less uncertainty with respect to the preorder ≺ given by majorization. Accordingly, if we plot the achieved expected utility against the number of samples, we obtain an optimality curve similar to Figure 14(c) and Figure 15(b). In Figure 16 we can see how Bayesian Inference outperforms Maximum Likelihood when evaluated with respect to the average expected utility of a bounded-rational decision-maker with 2 bits of information-processing resources.

Discussion
In this work, we have developed a generalized notion of decision-making in terms of uncertainty reduction. Based on the simple idea of transferring pieces of probability between the elements of a probability distribution, which we call elementary computations, we have promoted a notion of uncertainty which is known in the literature as majorization, a preorder ≺ on P Ω . Taking non-uniform initial distributions into account, we extended the concept to the notion of relative uncertainty, which corresponds to relative majorization ≺ q . Despite of the large amount of research that has been done on majorization theory, from the early works [65,60,36] through further developments [79,10,8,72,19,13,62] to modern applications [15,41,34], there is a lack of results on the more general concept of relative majorization. This does not seem to be due to a lack of interest, as can be seen from  Figure 16. Optimality curve given by Bayesian inference. The average expected utility as a function of N achieved by a boundedrational decision-maker that infers the world state distribution with Bayes rule (3.14) forms an efficiency frontier that cannot be surpassed by any other inference scheme, like for example Maximum Likelihood, when starting from the same prior belief about the world.
the results [97,80,44,54], but mostly because relative majorization looses some of the appealing properties of majorization which makes it harder to deal with, for example that permutations no longer leave the ordering ≺ q invariant, in contrast to the case of a uniform prior. This restriction does, however, not affect our application of the concept to decision-making, as permutations are not considered as elementary computations, since they do not diminish uncertainty. By reducing the non-uniform to the uniform case, we managed to prove new results on relative majorization (Theorem 2.7), which then enabled new results in other parts of the paper (Example 2.3, Propositions A.6, A.8), and allowed an intuitive interpretation of our final definition of a decision-making process (Definition 2.11) in terms of elementary computations with respect to non-uniform priors (Definition 2.10).
More precisely, starting from stepwise elimination of uncertain options (Section 2.3), we have argued that decision-making can be formalized by transitions between probability distributions (Section 2.2), and arrived at the concept of decisionmaking processes traversing probability space from prior to posterior by successively moving pieces of probability between options such that uncertainty relative to the prior is reduced (Section 2.1). Such transformations can be quantified by cost functions, which we define as order-preserving functions with respect to ≺ q and capture the resource costs that must be expended by the process. We have shown (Propositions A.1, A.6) that many known generalized entropies and divergences, which are examples of such cost functions (Examples 2.2 and 2.3), satisfy superadditivity with respect to coarse-graining. This means that under such cost functions, decisionmaking costs can potentially be reduced by a more intelligent search strategy, in contrast to Kullback-Leibler divergence, which was characterized as the only additive cost function (Proposition A.7). There are plenty of open questions for further investigation in that regard. First, it is not clear under which assumptions on the cost functions C q superadditivity could be improved to C q (p) = αD KL (p q)+r(p, q) with α > 0 and r(p, q) ≥ 0. Additionally, it would be an interesting challenge to find sufficient conditions implying super-additivity that include more cost functions than f -divergences. The field of information geometry might give further insights on the topic, since there are studies in similar directions, in particular characterizations of divergence measures in terms of information monotonicity and the data-processing inequality [43,5,6,7]. One interesting result is the characterization of Kullback-Leibler divergence as the single divergence measure being both an f -divergence and a Bregman divergence.
In Section 3, bounded rational decision-makers were defined as decision-making processes that are maximizing utility under constraints on the cost function, or equivalently minimizing resource costs under a minimal utility requirement. In the important case of additive cost functions (i.e. proportional to Kullback-Leibler divergence), this leads to information-theoretic bounded rationality [68,92,70,29,33], which has precursors in the economic and game-theoretic literature [64,66,63,101,90,42,93,91,92,46,100,58]. We have shown that the posteriors of a bounded rational decision-maker with increasing informational constraints leave a path in probability space that can itself be considered an anytime decision-making process, in each step perfectly trading off utility against processing costs (Prop. 3.2). In particular, this means that the path of a bounded rational decision-maker with informational cost decreases uncertainty with respect to all cost functions, not just Kullback-Leibler divergence. We have also studied the role of the prior in bounded rational multi-task decision-making, where we have seen that imperfect knowledge about the world state distribution leads to Bayesian inference as a byproduct, which is in line with the characterization of Bayesian inference as minimizing prediction surprise [69], but also demonstrates the wide applicability of the developed theory of decision-making with limited resources.
Finally, in Section 4, we have presented the results of a simulated bounded rational decision-maker solving an absolute identification task with and without knowledge about the world state distribution. Additionally, we have seen that Bayesian inference can be considered a decision-making process with limited resources by itself, where the resource is given by the number of available datapoints.

Conclusion
To our knowledge, this is the first principled approach to decision-making based on the intuitive idea of Pigou-Dalton-type probability transfers (elementary computations). Information-theoretic bounded rationality has been introduced by other axiomatic approaches before [68,63]. For example, in [68] a precise relation between rewards and information value was derived by postulating that systems will choose those states with high probability that are desirable for them. This leads to a direct coupling of probabilities and utility, where utility and information inherit the same structure, and only differ with respect to normalization-see [17] for similar ideas. In contrast, we assume utility and probability to be independent objects a priori that only have a strict relationship in the case of bounded-optimal posteriors. The approach in [63] introduces Kullback-Leibler divergence as disutility for decision control. Based on Hobson's characterization [40], the authors argue that cost functions should be monotonic with respect to uniform distributions (property (2.22)) and invariant under decomposition, which coincides with additivity under coarsegraining (see Examples 2.2, 2.3). Both of these assumptions are special cases of our more general treatment, where cost functions must be monotonic with respect to elementary computations and are generally not restricted to being additive.
In the literature, there are many mechanistic models of decision-making that instantiate decision-making processes with limited resources. Examples include reinforcement learning algorithms with variable depth [49,50], Markov chain Monte Carlo (MCMC) models where only a certain number of samples can be evaluated [100,67,38], and evidence accumulation models that accumulate noisy evidence until either a fixed threshold is reached [53,75,94,76,84] or where thresholds are determined dynamically by explicit cost functions depending on the number of allowed evidence accumulation steps [27,24]. Many of these concrete models may be described abstractly by resource parameterizations (Def. 2.9). More precisely, in such cases the posteriors {p r } r∈I ⊂ Γ ⊂ P Ω are generated by an explicit process with process constraints Γ and resource parameter r. For example, in diffusion processes r may correspond to the amount of time allowed for evidence accumulation, in Monte Carlo algorithms r may reflect the number of MCMC steps, and in a reinforcement learning agent r may represent the number of forward-simulations. If the resource restriction is described by a monotonic cost function r → c r [27,24], then the process can be optimized by a maximization problem of the form max r∈I,p∈Γr where M, M are non-negative constants, Γ r ⊂ Γ denotes the subset of probability distributions with resource r, and C q denotes a cost function such that r → C q (p) for p ∈ Γ r is strictly monotonically increasing. In particular, such cases can also be regarded as bounded rational decision-making problems of the form (3.1).
Bounded rationality models in the literature come in a variety of flavors. In the heuristics and biases paradigm the notion of optimization is often dismissed in its entirety [31], even though decision-makers still have to have a notion of options being better or worse, for example in order to adapt their aspiration levels in a satisficing scheme [88]. We have argued that from an abstract normative perspective we can formulate satisficing in probabilistic terms, such that one could investigate the efficiency of heuristics within this framework. Another prominent approach to bounded rationality is given by systems capable of decision-making about decisionmaking, i.e. meta-decision-making. Explicit decision-making processes composed of two decision steps have been studied, for example, in the reinforcement learning literature [49,50,73,98], where the first step is represented by a meta decision about whether a cheap model-free or a more expensive model-based learning algorithm is used in the second step. The meta step consists of a trade-off between the estimated utility against the decision-making costs of the second decision step. In the information-theoretic framework of bounded rationality this could be seen as a natural property of multi-step decision-making and the recursivity property (3.9) from which follows that the value of a decision-making problem is given by its Free Energy, that besides of the achieved utility also takes the corresponding processing costs into account. Another prominent approach to bounded rationality is computational rationality [58] where the focus lies on finding bounded-optimal programs that solve constrained optimization problems presented by the decision-maker's architecture and the task environment. As described above, such architectural constraints could be represented by a process dependent subset Γ ⊂ P Ω , and in fact our resource costs could be included into such a subset Γ r as well. From this point of view, both frameworks would look for bounded-optimal solutions in that the search space is first restricted and then the best solution in the restricted search space is found. However, our search space would consist of distributions describing probabilistic input-output maps, whereas the search space of programs would be far more detailed.
The notion of decision-making presented in this work, intuitively developed from the basic concept of uncertainty reduction given by elementary computations and motivated by the simple idea of progressively eliminating options, on the one hand provides a promising theoretical playground that is open for further investigation (e.g. superadditivity of cost functions and minimality of relative entropy), potentially providing new insights into the connection between the fields of rationality theory and information theory, and on the other hand serves the purpose of a general framework to describe and analyze all kinds of decision-making processes (e.g. in terms of bounded-optimality).
Proof. As we will see in the following proof, strict convexity is not needed for superadditivity, but it is required for the definition of a cost function. First, since i p i = 1, notice that we can always redefine the convex function f in (A.1) by f c (t) := f (t) − c(t − 1) for an arbitrary constant c ∈ R without changing C(p) for all p ∈ P Ω . Hence, without loss of generality, we may assume f (1) = 0, so that t = 1 is a global minimum of f (since f (t) ≥ f (1) + (t − 1) f (1) = f (1) due to convexity). Since C is symmetric, superadditivity under coarse-graining is equivalent to By setting u = p 1 + p 2 and v = p1 p1+p2 , this is equivalent to By the symmetry of F v under the replacement of v by 1−v, without loss of generality, we may assume that v ≤ 1 2 , so that uv ≤ u(1−v) ≤ u. Since f is convex, f is monotonically increasing on [0, 1], and thus f (uv) ≤ f (u(1−v) ≤ f (u). In particular, and thus, since f (v) + f (1−v) ≥ 0, it follows that F v (u) ≤ 0, which completes the proof.
Proposition A.2 (Characterization of Shannon entropy, Example 2.2). If a cost function C is additive under coarse-graining, that is if C(Z) = C(X) + C(Y |X) with the notation from Proposition A.1, then C = −αH for some α ≥ 0, i.e. C is proportional to the negative Shannon entropy −H. Proof. Since uniform distributions over N options are majorized by uniform distributions over N < N options (see (2.7)), it follows for any cost function C that the function f defined by f (N ) := C 1 N , . . . , 1 N is monotonically increasing. Therefore the claim follows directly from Shannon's proof [85], where he shows that this monotonicity, additivity under coarse-graining, and continuity determine Shannon entropy up to a constant factor. It follows that N i=1 Π i (p) = e for all p ∈ P Ω , where e = (1, . . . , 1) as above, and therefore the uniform distribution 1 N e is given by a convex combination of permutations of p, so that (iv) in Theorem 2.3 implies 1 N e ≺ p. There is many different ways to prove uniqueness. A direct way is to assume there exists q ∈ P with q ≺ 1 N e for all p ∈ P Ω , so that by (iii) in Theorem 2.3 there exists a stochastic matrix A with 1 N eA = q. But since eA = e (A stochastic), it follows that q = 1 N e. An indirect way would be to use that if q ≺ p for all p ∈ P Ω , then from Example 2.2 we know that this implies H(q) ≥ H(p) for all p ∈ P Ω , where H denotes the Shannon entropy, simply because −H is a cost function. In particular, q maximizes H and therefore coincides with the uniform distribution 1 N e.
(i) ⇒ (iii): Assuming that p ≺ q p we have Λ q p ≺ Λ q p and therefore, by (iii) in Theorem 2.3, there exists a doubly stochastic matrix B such that Λ q p = B T Λ q p. With A T := Λ −1 q B T Λ q it follows that where we have used that (Λ −1 q ) T e = e and Λ T q e = e which is easy to check, and Be = e from B being a stochastic matrix. Note that, by slightly abusing notation, here e is always the constant vector (1, . . . , 1) irrespective of the number of its entries (N or α, depending on which space the operator is defined). Moreover, we have A T q = Λ −1 q B T Λ q q = 1 α Λ −1 q B T e = 1 α Λ −1 q e = q where we have used that Λ q q by definition is the uniform distribution on PΩ, i.e. Λ q q = 1 α e, and therefore also Λ −1 q e = α q. In particular, since also A ij ≥ 0 (B, Λ q , Λ −1 q have non-negative entries), it follows that A is a q-stochastic matrix such that p = A T p = pA.
(i) ⇐ (iii): Similarly, if A is a q-stochastic matrix with p = pA, then Λ q p = B T Λ q p, where B := (Λ −1 q ) T AΛ T q satisfies B T e = α Λ q A T q = α Λ q q = e and Be = (Λ −1 q ) T Ae = (Λ −1 q ) T e = e, where we have used that Λ −1 q e = αq, Λ q q = 1 α e, Λ T q e = e, and (Λ −1 q ) T e = e. In particular, since also B ij ≥ 0, B is doubly stochastic and therefore Λ q p ≺ Λ q p, i.e. p ≺ q p.
Proposition A.5 (Prop. 2.8). The prior q ∈ P Ω is the unique minimal element in P Ω with respect to ≺ q , that is q ≺ q p for all p ∈ P Ω .
Proof. Let p ∈ P Ω , and let P := Λ q p denote its representation in PΩ. Then Q ≺ P by Proposition A.3 (uniform distributions are minimal) and therefore q = Λ −1 q Q ≺ q p, in particular q is minimal with respect to ≺ q . For uniqueness, let p be possibly another minimal element. Then p ≺ q q and therefore by (iii) in Theorem 2.7 there exists a q-stochastic matrix A with p = qA. But since A is q-stochastic, qA = q, and thus p = q. Proposition A.6 (Example 2.3: Superadditivity of f -divergences under coarsegraining). All relative cost functions of the form with f (strictly) convex and differentiable on [0, 1], and f (1) = 0, are superadditive with respect to coarse-graining, that is, for Z = (X, Y ), whenever C q (X) := C q(X) (p(X)) and C q (Y |X) := E p(X) [C q(Y |X) (p(Y |X))], which for cost functions that are symmetric with respect to permutations of ((p i , q i )) i=1,...,N (such as (2.20)) is equivalent to C q (p) ≥ C (q1+q2,q3...,q N ) (p 1 +p 2 , p 3 , . . . , p N )+(p 1 +p 2 ) C ( q 1 Proof. This is a simple corollary to Proposition A.1, after establishing the following interesting property of cost functions of the form (A.5): where Λ q denotes the representation mapping defined in Section 2.3 that maps q to a uniform distribution on an elementary decision spaceΩ of |Ω| = α elements, given by (Λ q p)(ω) = 1 is a disjoint partition of Ω such that |A i | = α q i . Equation (A.7) then follows from where we have used that w∈Ai 1 = |A i | = αq i . Hence the case of a non-uniform prior reduces to the case of a uniform prior, which was shown in Proposition A.1.
If C q is a continuous cost function relative to q that is additive under coarsegraining, that is C q (Z) = C q (X) + C q (Y |X) in the notation of Proposition A.6, then C q (p) = α D KL (p q) (A.8) for some α ≥ 0, where D KL (p q) denotes the Kullback-Leibler divergence D KL (p q) = i p i log(p i /q i ).
Proof. First, we show that any relative cost function that is additive under coarsegraining satisfies (2. Once (A.9) has been established, then (A.8) goes back to a result by Hobson [40] (see also [63]), whose proof is a modification of Shannon's axiomatic characterization [85]. The first property in (A.9) actually is true for all relative cost functions: For q = u N with N = |Ω|, we have p ≺ q p iff p ≺ p and thus the first property follows from (2.7), and the same is true in the case when N < |Ω|, since we always assume that p , p are absolutely continuous with respect to q, which allows to redefine Ω to only contain the N options covered by q.
For the proof of the second property in (A.9), we let the random variable X indexing the partitions E 1 , E 2 of Ω, where E 1 denotes the support of u N and E 2 = Ω\E 1 its complement, and Y representing the choice inside of the selected partition E i given X = i. Letting q(Z) = u N , and p(Z) = u N , then it follows from addivity under coarse-graining that C q(X) (p(X)) = C u N (u N ) = f (N , N ), and letting p(Z) = u M , we obtain f (M, N ) = C q(X) (p(X)) + C(Y |X = 1) = f (N , N ) + f (M, N ) , since p(X=1) = 1, p(X=2) = 0, and C(Y |X = 1) = C u N (u M ). Therefore it follows that f (M, N ) ≥ f (M, N ).
Proposition A.8 (Prop. 3.2). If (p β ) β≥0 is a family of bounded-optimal posteriors given by (3.3) with cost function C q (p) = D KL (p q), then β is a resource parameter with respect to any cost function, in particular q = p 0 q p β q p β ∀β , β with β < β . (A.10) Proof. Part of the proof generalizes a result in [78] to the case of a non-uniform prior q, by making use of our new characterization (v) of ≺ q , by which it suffices to show that β → l−1 i=1 (p β,i ) ↓ + a q (k, l)(p β,l ) ↓ is an increasing function of β for all k, l specified in Theorem 2.7. By (3.6) below, we have where U i are the decreasingly arranged utility values U (x) for x ∈ Ω, so that also p β,i /q i is arranged decreasingly. From the ordering of the U i , it is easy to see that p j + p k+1   with the notation p k := p β,k , from which it follows that S k := k i=1p k U i withp k := p k / k j=1 p j , is monotonically decreasing in k (with S N = E p [U ]), and therefore x i ≥ 0, which completes the proof of p β ≺ q p β . It remains to see that p β ≺ q p β . This follows again from (v) in Theorem 2.7, more precisely from the requirement that p β,1 ≤ p β ,1 if p β ≺ q p β , after establishing that β → Z −1 β e βU1 is monotonically increasing. For the latter, note that since the U i are ordered decreasingly, q i e βUi from which follows that ∂ β (Z −1 β e βU1 ) ≥ 0, which completes the proof.