Minimum Description Length Codes Are Critical

Abstract

In the Minimum Description Length (MDL) principle, learning from the data is equivalent to an optimal coding problem. We show that the codes that achieve optimal compression in MDL are critical in a very precise sense. First, when they are taken as generative models of samples, they generate samples with broad empirical distributions and with a high value of the relevance, defined as the entropy of the empirical frequencies. These results are derived for different statistical models (Dirichlet model, independent and pairwise dependent spin models, and restricted Boltzmann machines). Second, MDL codes sit precisely at a second order phase transition point where the symmetry between the sampled outcomes is spontaneously broken. The order parameter controlling the phase transition is the coding cost of the samples. The phase transition is a manifestation of the optimality of MDL codes, and it arises because codes that achieve a higher compression do not exist. These results suggest a clear interpretation of the widespread occurrence of statistical criticality as a characterization of samples which are maximally informative on the underlying generative process.

1. Introduction

It is not infrequent to find empirical data which exhibits broad frequency distributions in the most disparate domains. Broad distributions manifest in the fact that if outcomes are ranked in order of decreasing frequency of their occurrence, then the rank frequency plot spans several orders of magnitude on both axes. Figure 1 reports few cases (see caption for details), but many more have been reported in the literature (see e.g., [1,2]). A straight line in the rank plot corresponds to a power law frequency distribution, where the number of outcomes that are observed k times behave as $m_k\sim k^{-\mu -1}$ (with $1/\mu$ being the slope of the rank plot). Yet, as Figure 1 shows, empirical distributions are not always power laws, even though they are broad nonetheless. Countless mechanisms have been advanced to explain this behaviour [1,2,3,4,5,6]. It has recently been suggested that broad distributions arise from efficient representations, i.e., when the data samples relevant variables, which are those carrying the maximal amount of information on the generative process [7,8,9]. Such Maximally Informative Samples (MIS) are those for which the entropy of the frequency with which outcomes occur—called relevance in [8,9]—is maximal at a given resolution, which is measured by the number of bits needed to encode the sample (see Section 1.1). MIS exhibit power law distributions with the exponent $\mu$ governing the tradeoff between resolution and relevance [9]. This argument for the emergence of broad distributions is independent of any mechanism or model. A direct way to confirm this claim is to check that samples generated from models that are known to encode efficient representations are actually maximally informative. In this line, [10] found strong evidence that MIS occur in the representations that deep learning extracts from data. This paper explores the same issue in efficient coding as defined in Minimum Description Length [11].

Figure 1. Rank plot of the frequencies across a broad range of datasets. Log-log plots of rank versus frequency from diverse datasets: survey of 4962 species of trees across 116 families sampled from the Amazonian lowlands [12], survey of 1053 species of trees across 376 genera and 89 families sampled across a 50 hectare plot in the Barro Colorado Island (BCI), Panama [13], counts indicating the inclusion of each 13,001 LEGO parts on 2613 distributed toy sets [14,15] and the number of genes that are regulated by each of the 203 transcription factors (TFs) in E. coli [16] and 188 TFs in S. cerevisiae (yeast) [17] through binding with transcription factor binding sites (TFBS).

Regarding empirical data as a message sent from nature, we expect it to be expressed in an efficient manner if relevant variables are chosen. This requirement can be made quantitative and precise, in information theoretic terms, following Minimum Description Length (MDL) theory [11]. MDL seeks the optimal encoding of data generated by a parametric model with unknown parameters (see Section 1.2). MDL derives a probability distribution over samples that embodies the requirement of optimal encoding. This distribution is the Normalized Maximum Likelihood (NML). This paper studies the NML as a generative process of samples and studies both its typical and atypical properties. In a series of cases, we find that samples generated by NMLs are typically close to being maximally informative, in the sense of [9], and that their frequency distribution is typically broad. In addition, we find that NMLs are critical in a very precise sense, because they sit at a second order phase transition that separates typical from atypical behavior. More precisely, we find that large deviations, for which the resolution attains atypically low values, exhibit a condensation phenomenon whereby all N points in the sample coincide. This is consistent with the fact that NML correspond to efficient coding of random samples generated from a model, so that codes achieving higher compression do not exist. Large deviations enforcing higher compression force parameters to corners of the allowed space where the model becomes deterministic.

The rest of the paper is organized as follows: the rest of the introduction lays the background of what follows by recalling the characterization of samples in terms of resolution and relevance, as in [9], and the derivation of NML in MDL, following [11]. Section 2.1 discusses typical properties of NML and Section 2.2 discusses large deviations of the coding cost. We conclude with a series of remarks on the significance of these results.

Setting the Scene

Let $\widehat{s}=(s^{(1)},\dots ,s^{(N)})$ be a sample of N observations, $s^{(i)}\in \chi$, of a system where $\chi$ is a countable finite state space. We define $k_s$ as the number of observations in $\widehat{s}$ for which $s^{(i)}=s$, i.e., the frequency of s. The number of states s that occur k times will be denoted as $m_k$. Both $k_s$ and $m_k$ depend on the sample $\widehat{s}$.

We assume that $\widehat{s}$ is generated in a series of independent experiments or observations, all in the same conditions. This is equivalent to taking $\widehat{s}$ as a sequence of N independent draws from an unknown distribution $p(s)$ (i.e., the generative process).

1.1. Resolution, Relevance and Maximally Informative Samples

The information content of the sample is measured by the number of bits needed to encode a single data point. This is given by Shannon entropy [18]. Taking the frequency $k_s/N$ as the probability of point s, this leads to:

$$\widehat{H}\left[s\right]=-\sum _{s\in \chi }\frac{k_s}{N}log\frac{k_s}{N}=-\sum _k\frac{k{m}_k}{N}log\frac{k}{N},$$

(1)

where the $\widehat{}$ indicates that the entropy is computed from the empirical frequency. This quantity specifies the level of detail of the description provided by the variable s. At one extreme, all the data points are equal, i.e., $s^{(i)}=s$, $\forall i=1,\dots ,N$ such that $m_k=0$ for $k=1,\dots ,N-1$ and $m_{k=N}=1$. With this, one finds that $\widehat{H}\left[s\right]=0$. On the other extreme, all the data points are different, i.e., $s^{(i)}\ne s^{(j)}$, $\forall i\ne j$, such that $m_{k=1}=N$ and $m_{k^{\prime }}=0$, $\forall k^{\prime }>1$. Hence, one finds that $\widehat{H}\left[s\right]=logN$. This is why we call $\widehat{H}\left[s\right]$ as the resolution, following [9]. The resolution clearly depends on the cardinality of $\chi$. Only a part of $\widehat{H}\left[s\right]$ provides information on the generative process $p(s)$ and this is given by the relevance

$$\widehat{H}\left[k\right]=-\sum _k\frac{k{m}_k}{N}log\frac{k{m}_k}{N}.$$

(2)

A simple argument, which is elaborated in detail in [9], is that the empirical frequency $k_s/N$ is the best estimate of $p(s)$, so conditional on $k_s$, the sample does not contain any further information on $p(s)$. Note that $k_s$ is a function of s, which implies $\widehat{H}[s,k]=\widehat{H}\left[s\right]\ge \widehat{H}\left[k\right]$. Therefore, the difference $\widehat{H}\left[s\right]-\widehat{H}\left[k\right]=\widehat{H}[s|k]$ quantifies the amount of noise the sample contains.

We call $\widehat{s}$ a Maximally Informative Sample (MIS) if $m_k$ is such that the relevance is maximal at a given resolution $\widehat{H}\left[s\right]=H_0$. This implies the maximization of the functional

$$\mathcal{F}=\widehat{H}\left[k\right]+\mu (\widehat{H}\left[s\right]-H_0)+\lambda \left(\sum _{k}k{m}_k-N\right)$$

(3)

over $m_k$, where the Lagrange multipliers $\mu$ and $\lambda$ are adjusted to enforce the conditions $\widehat{H}\left[s\right]=H_0$ and ${\sum }_{k}k{m}_k=N$, respectively. As shown in [7,8], MIS exhibit a power law frequency distribution

$$m_k\approx c{k}^{-1-\mu }$$

(4)

where c is a normalization constant such that ${\sum }_{k}k{m}_k=N$. As $H_0$ varies from 0 to $logN$, MISs trace a curve in the resolution-relevance plane (see solid lines in Figure 2 and Figure 3 (B, C)) with $\mu$ as the negative slope. As discussed in [9,10], $\mu$ quantifies the trade-off between resolution and relevance: a decrease in resolution of one bit leads to an increase of $\mu$ bits in relevance. The point $\mu =1$, which corresponds to Zipf’s law, sets the limit beyond which further reduction in $\widehat{H}\left[s\right]$ results in lossy compression, because, for $\mu <1$, the increase in $\widehat{H}\left[k\right]$ cannot compensate the loss in resolution.

Figure 2. Properties of the typical samples generated from the NML of the Dirichlet model. (A) A plot showing the frequency distribution of the typical samples of the Dirichlet NML code. Given S, the cardinality of the state space, $\chi$, with $S=1.0\times {10}^3$ (orange dots), $5.0\times {10}^3$ (green squares), and $1.0\times {10}^4$ (red triangles), we compute the average frequency distribution across 100 generated samples from the Dirichlet NML of size $N=10S$ such that the average frequency per state, $\rho$, is fixed. This is compared against the theoretical calculations (solid black line) for $q(k|z^{*})$ in Equation (19). (B) Plot showing the degeneracy, $m_k$, of the frequencies, k, in a representative typical sample of length $N={10}^3$ generated from the Dirichlet NML code with average frequencies per spike: $\rho =100$ (yellow triangle), $\rho =10$ (orange x-mark) and $\rho =2$ (red cross). The corresponding dashed lines depict the best-fit line. (C,D) Plots of $\widehat{H}\left[s\right]$ versus $\widehat{H}\left[k\right]$ for the typical samples of the Dirichlet NML code. For a fixed size of the data, N ($N={10}^3$ in C and $N={10}^4$ in D), we have drawn 100 samples from the Dirichlet NML code varying $\rho$, ranging from 2 to 100. The results are compared against the $\widehat{H}\left[k\right]$ and $\widehat{H}\left[s\right]$ for maximally informative samples (MIS, solid black line) and random samples (dashed black lines). For the MIS, the theoretical lower bound is reported [8]. For the random samples, we compute the averages of $\widehat{H}\left[s\right]$ and $\widehat{H}\left[k\right]$ over ${10}^7$ realizations of random distributions of N balls in L boxes, with L ranging from 2 to ${10}^7$. Here, each box corresponds to one state $s=1,\dots ,L$ and $k_s$ is the number of balls in box s. Note that all the calculated values for $\widehat{H}\left[k\right]$ and $\widehat{H}\left[s\right]$ are normalized by $logN$.

Figure 3. Properties of typical samples for the NML codes of the paramagnet. (A) Plots showing the degeneracy, $m_k$, of the frequencies, k, in a representative typical sample of length $N={10}^4$ generated from the NML of a paramagnet with different number of independent spins: $n=4$ (blue star), $n=12$ (red cross) and $n=20$ (yellow diamond). The corresponding dashed lines depict the best-fit line. (B,C) Plots of the $\widehat{H}\left[k\right]$ versus $\widehat{H}\left[s\right]$ of the typical samples generated from the paramagnet NML code for varying sizes of the data, $N={10}^4$ (B) and $N={10}^5$ (C), and for varying number of spins, n, ranging from 3 to 20. Given N and n, we compute the $\widehat{H}\left[k\right]$ and $\widehat{H}\left[s\right]$ over 100 realizations of the NML code of a paramagnet. The results are compared against the $\widehat{H}\left[k\right]$ and $\widehat{H}\left[s\right]$ for maximally informative samples (solid black line) and random samples (dashed black line) as described in Figure 2. Note that all the calculated $H\left[k\right]$ and $H\left[s\right]$ are normalized by $logN$.

1.2. Minimum Description Length and the Normalized Maximum Likelihood

The main insight of MDL is that learning from data is equivalent to data compression [11]. In turn, data compression is equivalent to assigning a probability distribution over the space of samples.This section provides a brief derivation of this distribution whereas the rest of the paper discuss its typical and atypical properties. We refer the interested reader to [11,19] for a more detailed discussion of MDL.

From an information theoretic perspective, one can think of the sample, $\widehat{s}$, as a message generated by some source (e.g., nature) that we wish to compress as much as possible. This entails translating $\widehat{s}$ in a sequence of bits. A code is a rule that achieves this for any $\widehat{s}\in {\chi }^N$ and its efficiency depends on whether frequent patterns are assigned short codewords or not. Conversely, any code implies a distribution $P(\widehat{s})$ over the space of samples and the cost of encoding the sample $\widehat{s}$ under the code P is given by [18]

$$E=-logP(\widehat{s})$$

(5)

bits (assuming logarithm base two). Optimal compression is achieved when the code P coincides with the data generating process [18].

Consider the situation where the data is generated as independent draws from a parametric model $f(s|\theta )$. If the value of $\theta$ were known, then the optimal code would be given by $P(\widehat{s})={\prod }_{i}f(s^{(i)}|\theta )\equiv f(\widehat{s}|\theta )$. MDL seeks to derive P in the case where $\theta$ is not known (Indeed, MDL aims at deriving efficient coding under f irrespective of whether $f(s|\theta )$ is the “true” generative model or not. This allows one to compare different models and choose the one providing the most concise description of the data). This applies, for example, to the situation where $\widehat{s}$ is a series of experiments or observation aimed at measuring the parameters $\theta$ of a theory.

In hind sight, i.e., upon seeing the sample, the best code is $f(\widehat{s}|\widehat{\theta })$, where $\widehat{\theta }(\widehat{s})$ is the maximum likelihood estimator for $\theta$, and it depends on the sample $\widehat{s}$. Therefore, one can define the regret $\mathcal{R}$, as the additional encoding cost that one needs to spend to encode the sample $\widehat{s}$, if one uses the code $P(\widehat{s})$ to compress $\widehat{s}$, i.e.,

$$\mathcal{R}=-logP(\widehat{s})-\underset{\theta }{min}\left[-logf(\widehat{s}|\theta )\right].$$

(6)

Notice that ${min}_{\theta }\left[-logf(\widehat{s}|\theta )\right]=-logf(\widehat{s}|\widehat{\theta }(\widehat{s}))$. $\mathcal{R}$ is called regret of P relative to f for sample $\widehat{s}$ because it depends both on P and on $\widehat{s}$.

MDL derives the optimal code, $\overline{P}(\widehat{s})$, that minimizes the regret, assuming that for any P the source produces the worst possible sample [11]. The solution [20]

$$\overline{P}(\widehat{s})=\frac{f(\widehat{s}|\widehat{\theta }(\widehat{s}))}{{\sum }_{\widehat{x}\in {\chi }^N}f(\widehat{x}|\widehat{\theta }(\widehat{x}))}.$$

(7)

is called the Normalized Maximum Likelihood (NML). The optimal regret is given by

$$\overline{\mathcal{R}}=log\sum _{\widehat{s}\in {\chi }^N}f(\widehat{s}|\widehat{\theta }(\widehat{s}))$$

(8)

which is known in MDL as the parametric complexity (Notice that $e^{\overline{\mathcal{R}}}$ can be seen as a partition sum. Hence, throughout the paper, we shall refer to the parametric complexity as the UC partition function.). For models in the exponential family, Rissanen showed that the parametric complexity is asymptotically given by [21]

$$\overline{\mathcal{R}}\simeq \frac{k}{2}log\frac{N}{2\pi }+log\int \sqrt{detI(\theta )}d\theta +\mathcal{O}(1)$$

(9)

where $I(\theta )$ is the Fisher information matrix with the matrix elements defined by an expectation $I_{ij}(\theta )=-{⟨\frac{{\partial }^{2}logf(s|\theta )}{\partial {\theta }_i\partial {\theta }_j}⟩}_{\theta }$ over the parametric model $f(s|\theta )$ (see Appendix A for a simple derivation). The NML code is a universal code because it achieves a compression per data point which is as good as the compression that would be achieved with the optimal choice of $\theta$ when one has large enough samples. This is easy to see, because the regret $\overline{\mathcal{R}}/N$ per data point vanishes in the limit $N\to \infty$, hence the NML code achieves the same compression as $f(\widehat{s}|\widehat{\theta })$.

Notice also that the optimal regret, $\overline{\mathcal{R}}$, in Equation (8) is independent of the sample $\widehat{s}$. It indeed provides a measure of complexity of the model f that can be used in model selection schemes. For exponential families, MDL procedure penalizes models with a cost which equals the one obtained in Bayesian model selection [22] under a Jeffreys prior. Indeed, considering $\overline{P}(\widehat{s})$ as a generative model for samples, one can show that the induced distribution on $\theta$ is given by Jeffreys prior (see Appendix A).

2. Results

2.1. NML Codes Provide Efficient Representations

In this section we consider $\overline{P}$ as a generative model for samples and we investigate its typical properties for some representative statistical models.

2.1.1. Dirichlet Model

Let us start by considering the Dirichlet model distribution $f(s|\theta )={\theta }_s$, $\forall s\in \chi$. The parameters ${\theta }_s\ge 0$ are constrained by the normalization condition ${\sum }_{s\in \chi }{\theta }_s=1$. Let $S=|\chi |$ denote the cardinality of $\chi$ and define, for convenience, $\rho =N/S$ as the average number of points per state. Because each observation is mutually independent, the likelihood of a sample $\widehat{s}$ given $\theta =({\theta }_1,\dots ,{\theta }_S)$ can be written as

$$f(\widehat{s}|\theta )=\prod _{s\in \chi }{\theta }_s^{k_s},$$

(10)

where $k_s$ is the number of times that the state s occurs in the sample $\widehat{s}$. From here, it can be seen that ${\widehat{\theta }}_s=k_s/N$ is the maximum likelihood estimator for ${\theta }_s$. Thus, the universal code for the Dirichlet model can now be constructed as

$$\overline{P}(\widehat{s})=e^{-\overline{\mathcal{R}}}\prod _{s\in \chi }{\left(\frac{k_s}{N}\right)}^{k_s}$$

(11)

which can be read as saying that for each s, the code needs $-k_{s}log(k_s/N)+\overline{\mathcal{R}}/N$ bits. In terms of the frequencies, $\{k_1,\dots ,k_S\}$, the universal codes can be written as

$$\overline{P}(k_1,\dots ,k_S)=e^{-\overline{\mathcal{R}}}\frac{N!}{{\prod }_{s\in \chi }{k}_s!}\prod _{s\in \chi }{\left(\frac{k_s}{N}\right)}^{k_s}\delta \left(\sum _{s\in \chi }{k}_s-N\right)$$

(12)

wherein the multinomial coefficient, $\frac{N!}{{\prod }_{s\in \chi }{k}_s!}$, counts the number of samples with a given frequency profile $k_1,\dots ,k_S$. In order to compute the optimal regret $\overline{\mathcal{R}}$, we have to evaluate the partition function

$$\begin{array}{cc}\hfill e^{\overline{\mathcal{R}}}& =\frac{N!}{N^N{e}^{-N}}{\int }_{-\pi }^{\pi }\frac{d\mu }{2\pi }{e}^{i\mu N}\left[\sum _{k_1=0}^{\infty }\frac{k_1^{k_1}{e}^{-k_1}{e}^{-i\mu k_1}}{k_1!}\right]\cdots \left[\sum _{k_S=0}^{\infty }\frac{k_S^{k_S}{e}^{-k_S}{e}^{-i\mu k_S}}{k_S!}\right]\hfill \end{array}$$

(13)

$$\begin{array}{c}=\frac{N!}{N^N{e}^{-N}}{\int }_{-\pi }^{\pi }\frac{d\mu }{2\pi }{e}^{i\mu N}{\left[\mathcal{N}(i\mu )\right]}^S\hfill \end{array}$$

(14)

$$\begin{array}{c}\simeq \sqrt{2\pi N}{\int }_{-\pi }^{\pi }\frac{d\mu }{2\pi }{e}^{S\mathsf{\Phi }(i\mu )}\hfill \end{array}$$

(15)

where

$$\mathsf{\Phi }(z)=\rho z+log\mathcal{N}(z)$$

(16)

and

$$\mathcal{N}(z)=\sum _{k=0}^{\infty }\frac{k^k{e}^{-(1+z)k}}{k!}.$$

(17)

The integral in Equation (15) is dominated by the value where the function $\varphi$ attains its saddle point value $z^{*}(\rho )$, which is given by the condition

$$\frac{d\mathsf{\Phi }}{dz}=\rho -{⟨k⟩}_z=0$$

(18)

where the average ${⟨\dots ⟩}_z$ is taken with respect to the distribution

$$q(k|z)=\frac{1}{\mathcal{N}(z)}\frac{k^k{e}^{-(1+z)k}}{k!}.$$

(19)

Gaussian integration around the saddle point leads then to

$$e^{\overline{\mathcal{R}}}\simeq \sqrt{\rho }\frac{e^{S\mathsf{\Phi }(z^{*}(\rho ))}}{\sqrt{{⟨k^2⟩}_{z^{*}}-{⟨k⟩}_{z^{*}}^2}}$$

(20)

where we used the identity ${\mathsf{\Phi }}^{″}(z)=-\left[{⟨k^2⟩}_z-{⟨k⟩}_z^2\right]$.

The distribution Equation (12) can also be written introducing the Fourier representation of the delta function

$$\overline{P}(k_1,\dots ,k_S)=\frac{N!e^{-\overline{\mathcal{R}}}}{N^N{e}^{-N}}{\int }_{-\pi }^{\pi }\frac{d\mu }{2\pi }{e}^{i\mu N}\prod _{s\in \chi }\frac{k_s^{k_s}{e}^{-(1+i\mu )k_s}}{k_s!}.$$

(21)

For typical sequences $k_1,\dots ,k_S$, the integral is also dominated by the value $\mu =-i{z}^{*}(\rho )$ that dominates Equation (15), which means that the distribution factorizes as

$$\overline{P}(k_1,\dots ,k_S)\simeq \prod _{s\in \chi }q(k_s|z^{*}).$$

(22)

This means that the NML is, to a good approximation, equivalent to S independent draws from the distribution $q(k|z^{*})$ or, equivalently, that the distribution $q(k|z^{*})$ is the one that characterizes typical samples. This is fully confirmed by Figure 2A, which compares $q(k|z^{*})$ with the empirical distribution of $k_s$ drawn from $\overline{P}$. For large k, we find $q(k|z^{*})\sim e^{-z^{*}k}/\sqrt{k}$, which shows that the distribution of frequencies is broad, with a cutoff at $1/z^{*}$. This underlying broad distribution is confirmed by Figure 2B which shows the dependence of the degeneracy $m_k$ with the frequency k.

In the regime where $\rho \gg 1$ and k is large, the cutoff extends to large values of k and we find $z^{*}(\rho )\simeq \frac{1}{2\rho }$ (see Appendix B.1). In addition, the parametric complexity can be computed explicitly via Equation (9) in this regime, with the result

$$\overline{\mathcal{R}}\simeq \frac{S}{2}(1+log\rho )-\frac{1}{2}log(2\rho ),\rho \gg 1.$$

(23)

The coding cost of a typical sample is given by

$$\begin{array}{cc}\hfill E& =-log\overline{P}(\widehat{s})\hfill \end{array}$$

(24)

$$\begin{array}{c}=-\sum _{s\in \chi }{k}_{s}log\frac{k_s}{N}+\overline{\mathcal{R}}\hfill \end{array}$$

(25)

$$\begin{array}{c}=N\widehat{H}\left[s\right]+\overline{\mathcal{R}}.\hfill \end{array}$$

(26)

The number of samples with encoding cost E can be computed in the following way. The number of samples that correspond to a given degeneracy $m_k$ of the states that occurs $k_s=k$ times in $\widehat{s}$, is given by

$$\frac{N!}{{\prod }_k(k{m}_k)!}.$$

(27)

Therefore, the number of samples with coding cost E is

$$\begin{array}{cc}\hfill W(E)& =\sum _{\{m_k\}\in \mathcal{M}(E)}\frac{N!}{{\prod }_k(k{m}_k)!}\hfill \end{array}$$

(28)

$$\begin{array}{c}=\sum _{\{m_k\}\in \mathcal{M}(E)}{e}^{logN!-{\sum }_{k}log(k{m}_k)!}\hfill \end{array}$$

(29)

$$\begin{array}{c}\sim \sum _{\{m_k\}\in \mathcal{M}(E)}{e}^{N\widehat{H}\left[k\right]},\rho \gg 1\hfill \end{array}$$

(30)

where $\mathcal{M}(E)$ is the set of all sequences $\{m_k\}$ that are consistent with samples in ${\chi }^N$ and satisfy Equation (26). The last expression assumes $logM!\simeq MlogM-M$, which is reasonable for $M=k{m}_k\gg 1$, i.e., when $\rho \gg 1$. In this regime we expect the sum over $\mathcal{M}(E)$ to be dominated by samples with maximal $\widehat{H}\left[k\right]$. Indeed, Figure 2C,D show that samples drawn from $\overline{P}$ achieve values of $\widehat{H}\left[k\right]$ close to the theoretical maximum, especially in the region $\rho \gg 1$.

2.1.2. A Model of Independent Spins

In order to corroborate our results for the Dirichlet model, we study the properties of the universal codes for a model of independent spins, i.e., a paramagnet. For a single spin, $s=\pm 1$, in a local field h, the probability distribution is given by

$$P(s|h)=\frac{e^{sh}}{2coshh}.$$

(31)

Thus for a sample $\widehat{s}$ of size N,

$$P(\widehat{s}|h)=e^{\left[Nmh-Nlog(2coshh)\right]}$$

(32)

where $m=\frac{1}{N}{\sum }_{i=1}^M{s}^{(i)}$ is the local magnetization. The maximum likelihood estimate for h is $\widehat{h}(m)={tanh}^{-1}m$, hence the universal code for a single spin can be written as

$$\overline{P}(\widehat{s})=e^{N\left[mh(m)-log\left(2coshh(m)\right)\right]-\overline{\mathcal{R}}}$$

(33)

where $\overline{\mathcal{R}}\simeq \frac{1}{2}log\frac{\pi N}{2}$ (see Appendix B.2). Note that a sample with a magnetization m can be realized by considering the permutation of the up-spins ($s=1$, where there are $\ell =\frac{N+Nm}{2}$ of such spins) and the permutation of the down-spins ($s=-1$, where there are $N-\ell$ of such spins). Consequently, the magnetization for samples drawn from $\overline{P}$ has a broad distribution given by the arcsin law (see Appendix B.2)

$$\begin{array}{cc}\hfill \overline{P}(m)& =\left(\genfrac{}{}{0pt}{}{N}{\frac{N-Nm}{2}}\right)e^{N\left[m{tanh}^{-1}m-log\left(2cosh\left({tanh}^{-1}m\right)\right)\right]-\overline{\mathcal{R}}}\hfill \end{array}$$

(34)

$$\begin{array}{c}\simeq \frac{1}{\pi \sqrt{1-m^2}},m\in [-1,1].\hfill \end{array}$$

(35)

It is straightforward to see that the model of a single spin is equivalent to a Dirichlet model with two states $\chi =\{-1,+1\}$. In terms of the number ℓ of up-spins, using $m=\frac{2\ell -N}{N}$, the NML for a single spin can be written as

$$\begin{array}{c}\hfill \overline{P}(\ell )=e^{-\overline{\mathcal{R}}}\left(\genfrac{}{}{0pt}{}{N}{\ell }\right){\left(\frac{\ell }{N}\right)}^{\ell }{\left(1-\frac{\ell }{N}\right)}^{N-\ell }.\end{array}$$

(36)

The NML for a paramagnet with n independent spins reads as

$$\overline{P}({\ell }_1,\dots ,{\ell }_n)=e^{-n\overline{\mathcal{R}}}\prod _{i=1}^n\left(\genfrac{}{}{0pt}{}{N}{{\ell }_i}\right){\left(\frac{{\ell }_i}{N}\right)}^{{\ell }_i}{\left(1-\frac{{\ell }_i}{N}\right)}^{N-{\ell }_i}.$$

(37)

Figure 3 reports the properties of the typical samples of the NML of a paramagnet. We observed that the frequency distribution of typical samples is broad (Figure 3A) and that typical samples attain values of $H\left[k\right]$ very close to the maximum for a given value of $\widehat{H}\left[s\right]$ (Figure 3B,C). As the size N of data increases, the NML enters the well-sampled regime where $\widehat{H}\left[k\right]\simeq \widehat{H}\left[s\right]$, indicating that the data processing inequality [18] is saturated. In this regime, typical samples are those which maximize the entropy $\widehat{H}\left[s\right]$.

2.1.3. Sherrington-Kirkpatrick Model

In the following sections, we extend our findings to systems of interacting variables (graphical models) and discuss the properties of typical samples drawn from the corresponding NML distribution. We shall first consider models in which the observed variables are interacting either directly (Sherrington-Kirkpatrick model) and then restricted Boltzmann machines, where the variables interact indirectly through hidden variables.

In this section, $s=(s_1,\dots ,s_n)$ is a configuration of n spins $s_i\in \{\pm 1\}$. In the Sherrington-Kirkpatrick (SK) model, the distribution of s, considers all interactions up to two-body

$$\begin{array}{c}\hfill P(s|\mathit{J},\mathit{h})=\frac{1}{Z(\mathit{J},\mathit{h})}exp\left[\sum _i{h}_i{s}_i+\sum _{i<j}{J}_{ij}{s}_i{s}_j\right],s=(s_1,\dots ,s_n)\end{array}$$

(38)

where the partition function

$$Z(\mathit{J},\mathit{h})=\sum _{s_1=\pm 1}\cdots \sum _{s_n=\pm 1}exp\left[\sum _i{h}_i{s}_i+\sum _{i<j}{J}_{ij}{s}_i{s}_j\right]$$

(39)

is a normalization constant which depends on the pairwise couplings, $\mathit{J}$ with $J_{ij}=J_{ji}$ being the coupling strength between $s_i$ and $s_j$, and external local fields, $\mathit{h}$. Thus, given a sample, $\widehat{s}=(s^{(1)},\dots ,s^{(N)})$ of N observations, the likelihood reads as

$$\begin{array}{c}\hfill P(\widehat{s}|\mathit{J},\mathit{h})=exp\left[N\sum _i{h}_i{m}_i+N\sum _{i<j}{J}_{ij}{c}_{ij}-NlogZ(\mathit{J},\mathit{h})\right].\end{array}$$

(40)

where $m_i=\frac{1}{N}{\sum }_{l=1}^N{s}_i^{(l)}$ and $c_{ij}=\frac{1}{N}{\sum }_{l=1}^N{s}_i^{(l)}{s}_j^{(l)}$ are the magnetization and pairwise correlation respectively. Note that all the needed information about the SK model is encapsulated in the free energy, $\varphi (\mathit{J},\mathit{h})=logZ(\mathit{J},\mathit{h})$. Indeed, the maximum likelihood estimators for the couplings, $\widehat{\mathit{J}}$, and local fields, $\widehat{\mathit{h}}$, are the solutions of the self-consistency equations

$$\frac{\partial \varphi (\mathit{J},\mathit{h})}{\partial h_i}=m_i,\frac{\partial \varphi (\mathit{J},\mathit{h})}{\partial J_{ij}}=c_{ij},i,j=1,\dots ,n.$$

(41)

The universal codes for the SK model then reads as

$$\overline{P}(\widehat{s})=exp\left[N\left(\sum _i{\widehat{h}}_i{m}_i+\sum _{i<j}{\widehat{J}}_{ij}{c}_{ij}-\varphi (\widehat{\mathit{J}},\widehat{\mathit{h}})\right)-\overline{\mathcal{R}}\right].$$

(42)

However, unlike for the Dirichlet model and the paramagnet model, the UC partition function, $e^{\overline{\mathcal{R}}}$, for the SK model is analytically intractable (For SK models which possess some particular structures, a calculation of the UC partition function has been done in [23]). To this, we resort to a Markov chain Monte Carlo (MCMC) approach to sample the universal codes (See Appendix C.1). Figure 4A,C shows the properties of the typical samples drawn from the universal codes of the SK model in Equation (42).

Figure 4. Properties of typical samples for the NML codes of two graphical models: the Sherrington-Kirkpatrick (SK) model and the restricted Boltzmann machine (RBM). Left panels (A,C) show plots of the degeneracy, $m_k$, of the frequency, k, for representative typical samples generated from the NML codes for the SK model (A) and the RBM given a number of hidden variables, $n_h=7$ (B) for different number of (visible) spins, n. The corresponding dashed lines show the best-fit lines. On the other hand, right panels (B,D) show plots of the $\widehat{H}\left[k\right]$ versus $\widehat{H}\left[s\right]$ of the typical samples drawn from the NML codes for the SK model (B) and the RBM with $n_h=7$ (D) for $N={10}^3$ and for varying number of spins, n ranging from 3 to 12. Given N and n of a graphical model, we compute the $\widehat{H}\left[k\right]$ and $\widehat{H}\left[s\right]$ for 100 samples drawn from the respective NML codes through a Markov chain Monte Carlo (MCMC) approach (see Appendix C.1). Note that for the RBM, varying $n_h$ do not qualitatively affect the observations made in this paper. As before, the $\widehat{H}\left[k\right]$ and $\widehat{H}\left[s\right]$ are normalized by $logN$ and the typical NML samples are compared against maximally informative samples (solid black line) and random samples (dashed black line) as described in Figure 2.

2.1.4. Restricted Boltzmann Machines

We consider a restricted Boltzmann machine (RBM) wherein one has a layer composed of $n_v$ independent visible boolean units, $\mathit{v}=(v_1,\dots ,v_{n_v})$, which are interacting with $n_h$ independent hidden boolean units, $\mathit{h}=(h_1,\dots ,h_{n_h})$, in another layer where $v_i,h_i=0,1$. The probability distribution can be written down as

$$P(\mathit{v},\mathit{h}|\mathbf{\theta }=(\mathit{a},\mathit{b},\mathit{w}))=\frac{1}{Z(\mathbf{\theta })}exp\left(\sum _{i=1}^{n_v}{a}_i{v}_i+\sum _{j=1}^{n_h}{b}_j{h}_j+\sum _{i=1}^{n_v}\sum _{j=1}^{n_h}{v}_i{w}_{ij}{h}_j\right)$$

(43)

where the partition function

$$Z(\mathbf{\theta })=\sum _{v_1=0,1}\cdots \sum _{v_{n_v}=0,1}\sum _{h_1=0,1}\cdots \sum _{h_{n_h}=0,1}exp\left(\sum _{i=1}^{n_v}{a}_i{v}_i+\sum _{j=1}^{n_h}\left(b_j+\sum _{i=1}^{n_v}{v}_i{w}_{ij}\right)h_j\right)$$

(44)

is a function of the parameters, $\mathbf{\theta }$, with $w_{ij}$ is the interaction strength between $v_i$ and $h_j$, $\mathit{a}$ and $\mathit{b}$ are the local fields acting on the visible $\mathit{v}$ and hidden $\mathit{h}$ units respectively. Because the hidden units, $\mathit{h}$, are mutually independent, we can factorize and then marginalize the sum over the hidden variables, $\mathit{h}$, to obtain the distribution of a single observation, $\mathit{v}$, as

$$P(\mathit{v}|\mathbf{\theta })=\frac{1}{Z(\mathbf{\theta })}exp\left[\sum _{i=1}^{n_v}{a}_i{v}_i+\sum _{j=1}^{n_h}log2cosh\left(\sum _{i=1}^{n_v}{v}_i{w}_{ij}+b_j\right)\right].$$

(45)

Then, the probability distribution for a sample, $\widehat{\mathit{v}}=({\mathit{v}}^{(1)},\dots ,{\mathit{v}}^{(N)})$, of N observations is simply

$$P(\widehat{\mathit{v}}|\mathbf{\theta })=\prod _{k=1}^{N}p({\mathit{v}}^{(k)}|\mathbf{\theta }).$$

(46)

The parameters, $\widehat{\mathbf{\theta }}$, can be estimated by maximizing the likelihood using the Contrastive Divergence (CD) algorithm [24,25] (see Appendix C.2). Once the maximum likelihood parameters, $\widehat{\mathbf{\theta }}$, have been inferred, then the universal codes for the RBM can be built as

$$\overline{P}(\widehat{\mathit{v}})=e^{-\overline{\mathcal{R}}}\prod _{k=1}^{N}p({\mathit{v}}^{(k)}|\widehat{\mathbf{\theta }}).$$

(47)

In addition, like in the SK model, the UC partition function, $e^{\overline{\mathcal{R}}}$, for the RBM cannot be solved analytically. To this, we also resort to a MCMC approach to sample the universal codes (See Appendix C.1). Figure 4B,D shows the properties of the typical samples drawn from the universal codes of the RBM in Equation (47).

Taken together, we see that even for models that incorporate interactions, the typical samples of the NML i) have broad frequency distributions and ii) they achieve values of $\widehat{H}\left[k\right]$ close to the maximum, given $\widehat{H}\left[s\right]$. Due to computational constraints, we only present the results for $N={10}^3$ however, we expect that increasing N will only shift the NML towards the well-sampled regime.

2.2. Large Deviations of the Universal Codes Exhibit Phase Transitions

In this section, we focus on the distribution of the resolution $\widehat{H}\left[s\right]$ for samples $\widehat{s}$ drawn from $\overline{P}$. We note that

$$\widehat{H}\left[s\right]=\frac{1}{N}\sum _{i=1}^{N}log\frac{k_{s^{(i)}}}{N}$$

has the form of an empirical average. Hence, we expect it to attain a given value for typical samples drawn from $\overline{P}$. This also suggests that the probability to draw samples with resolution $\widehat{H}\left[s\right]=E$ different from the typical value has the large deviation form $P\{\widehat{H}\left[s\right]=E\}\sim e^{-NI(E)}$, to leading order for $N\gg 1$. In order to establish this result and to compute the function $I(E)$, as in [26] and [27], we observe that

$$\begin{array}{cc}\hfill P\{\widehat{H}\left[s\right]=E\}& =\sum _{\widehat{s}}\overline{P}(\widehat{s})\delta \left(\widehat{H}\left[s\right]-E\right)\hfill \end{array}$$

(48)

$$\begin{array}{c}={\int }_{-\infty }^{\infty }\frac{Ndq}{2\pi }\sum _{\widehat{s}}\overline{P}(\widehat{s})e^{iqN(\widehat{H}\left[s\right]-E)},\hfill \end{array}$$

(49)

where we used the integral representation of the $\delta$ function and $\overline{P}(\widehat{s})$ is the NML distribution in Equation (7). Upon defining

$$\sum _{\widehat{s}}\overline{P}(\widehat{s})e^{iqN\widehat{H}\left[s\right]}=e^{N\varphi (iq)},$$

(50)

let us assume, as in the Gärtner–Ellis theorem [26], that $\varphi (iq)$ is finite for $N\gg 1$ for all q in the complex plane. Then Equation (49) can be evaluated by a saddle point integration

$$\begin{array}{cc}\hfill P\{\widehat{H}\left[s\right]=E\}& ={\int }_{-\infty }^{\infty }\frac{Nd\alpha }{2\pi }{e}^{-N[i\alpha E-\varphi (i\alpha )]}\hfill \end{array}$$

(51)

$$\begin{array}{c}\sim e^{-N[\beta E-\varphi (\beta )]},\hfill \end{array}$$

(52)

where we account only for the leading order. $\beta$ is related to the saddle point value $q^{*}=-i\beta$ that dominates the integral and it is given by the solution of the saddle point condition

$$E=\frac{d}{d\beta }\varphi (\beta ).$$

(53)

Equation (52) shows that the function $I(E)$ is the Legendre transform of $\varphi (\beta )$, i.e.,

$$I(E)=-\beta E+\varphi (\beta )$$

(54)

with $\beta (E)$ given by the condition (53), as in the Gärtner–Ellis theorem [26]. Further insight and a direct calculation from the definition in Equation (50) reveals that Equation (53) can also be written as

$$E=\sum _{\widehat{s}}{\overline{P}}_{\beta }(\widehat{s})\widehat{H}\left[s\right]$$

(55)

which is the average of $\widehat{H}\left[s\right]$ over a “tilted” probability distribution [26]

$${\overline{P}}_{\beta }(\widehat{s})=\overline{P}(\widehat{s})e^{N\left\{\beta \widehat{H}\left[s\right]-\varphi (\beta )\right\}},$$

(56)

hence $\beta$ arises as the Lagrange multiplier enforcing the condition $\widehat{H}\left[s\right]=E$. Conversely, when $\beta (E)$ is fixed by the condition Equation (e̊fapp3:saddle2), samples drawn from ${\overline{P}}_{\beta }$ have $\widehat{H}\left[s\right]\simeq E$. In other words, ${\overline{P}}_{\beta }$ describes how large deviations with $\widehat{H}\left[s\right]=E$ are realized. Therefore, typical samples that realize such large deviations can be obtained by sampling the distribution ${\overline{P}}_{\beta }(\widehat{s})$ in Equation (56). Figure 5 show that, for Dirichlet models, samples obtained from ${\overline{P}}_{\beta }$ exhibit a sharp transition at $\beta =0$. The resolution (see green lines in Figure 5) sharply vanishes for negative values of $\beta$ as a consequence of the fact that the distribution localizes to samples where almost all outcomes coincide, i.e., $s_i=\overline{s}$. This is evidenced by the fact that the maximal frequency $k_{\overline{s}}={max}_s{k}_s$ approaches N very fast (see purple lines in Figure 5). In other words, $\beta =0$ marks a localization transition where the symmetry between the states in $\chi$ is broken, because one state $\overline{s}$ is sampled an extensive number of times $k_{\overline{s}}\propto N$.

Figure 5. Typical realizations of large deviations from the NML code of the Dirichlet model. For a fixed parameter, $\beta$ ranging from $\beta =-1$ to $\beta =1$, samples are obtained from ${\overline{P}}_{\beta }$ in Equation (56) for varying length of the dataset, N ($N={10}^4$ in solid lines with circle markers and $N={10}^5$ in dashed lines with square markers). The resolution $\widehat{H}\left[s\right]$ normalized by $logN$ (in green lines) and the maximal frequency $k_{\overline{s}}$ normalized by N (in purple lines) are calculated as an average over 100 realizations of ${\overline{P}}_{\beta }$ given $\beta$. The point $\beta =0$ corresponds to the typical samples that are realized from the Dirichlet NML code in Equation (12).

One direct way to see this is to consider the Dirichlet model and use the “tilted” distribution in Equation (56) to compute the distribution

$$q_{\beta }(k|z)=\frac{1}{\mathcal{N}(z)}\frac{k^{(1-\beta )k}{e}^{-(1+z)k}}{k!}.$$

of $k_s$ following the same steps leading to Equation (19), where again z is fixed by the condition ${\sum }_k{q}_{\beta }(k|z)k=N/S$. For $\beta \ge 0$, we again find, as in Equation (22), that $k_s$ can be considered as independent draws from the same distribution $q_{\beta }(k|z)$. For $\beta <0$, we find that the distribution $q_{\beta }(k|z)$ develops a sharp maximum at $k=N$ indicating that, as mentioned above, the sample concentrates on one state $\overline{s}$.

This behavior is generic whenever the underlying model $f(s|\theta )$ itself localizes for certain values $\overline{\theta }$ of the parameters, i.e., when $f(s|\overline{\theta })={\delta }_{s,\overline{s}}$. In order to see this, notice that, in general, we can write

$$f(\widehat{s}|\widehat{\theta }(\widehat{s}))=\prod _{s}f{(s|\widehat{\theta }(\widehat{s}))}^{k_s}.$$

(57)

Thus, by inserting the identity $e^{-N\widehat{H}\left[s\right]+N\widehat{H}\left[s\right]}$, the NML distribution in Equation (7) can be re-cast as

$$\overline{P}(\widehat{s})=e^{-N\widehat{H}\left[s\right]-N{D}_{KL}(\widehat{p}||\widehat{\theta })-\overline{\mathcal{R}}}$$

(58)

where ${\widehat{p}}_s=k_s/N$ is the empirical distribution and

$$D_{KL}(\widehat{p}||\widehat{\theta })=\sum _s{\widehat{p}}_{s}log\frac{{\widehat{p}}_s}{f(s|\widehat{\theta }(\widehat{s}))}$$

(59)

is a Kullback-Leibler divergence.

Now, we observe that

$$\begin{array}{cc}\hfill e^{N\varphi (\beta )}& =e^{-\overline{\mathcal{R}}}\sum _{\widehat{s}}{e}^{-(1-\beta )N\widehat{H}\left[s\right]-N{D}_{KL}(\widehat{p}||\widehat{\theta })}\hfill \end{array}$$

(60)

$$\begin{array}{c}\ge e^{-\overline{\mathcal{R}}}\sum _{\widehat{s}}{e}^{-(1-\beta )N\widehat{H}\left[s\right]-N{D}_{KL}(\widehat{p}||{\theta }_0)}\hfill \end{array}$$

(61)

$$\begin{array}{c}=e^{-\overline{\mathcal{R}}}\hfill \end{array}$$

(62)

where the inequality in Equation (61) derives from the fact that $\widehat{\theta }(\widehat{s})$, the maximum likelihood estimator for sample $\widehat{s}$, is replaced by a generic value ${\theta }_0$ and consequently, $D_{KL}(\widehat{p}||\widehat{\theta })\le D_{KL}(\widehat{p}||{\theta }_0)$. The equality in Equation (62), instead, derives from the choice ${\theta }_0=\overline{\theta }$ such that $f(s|\overline{\theta })={\delta }_{s,\overline{s}}$. Under this choice, only the term corresponding to “localized” samples where $s^{(i)}=s_0$ for all points in the sample, survive in the sum on $\widehat{s}$. For such localized samples, $\widehat{H}\left[s\right]=D_{KL}(\widehat{p}||{\theta }_0)=0$, hence Equation (62) follows.

Because of the logarithmic dependence of the regret $\overline{\mathcal{R}}$ on N (see Equation (9)), Equation (62) implies that, for all $\beta$,

$$\varphi (\beta )\ge \overline{\mathcal{R}}/N\simeq 0$$

(63)

for $N\gg 1$. Given that $\widehat{H}\left[s\right]\ge 0$ in Equation (55), then $E\ge 0$ and therefore, Equation (53) implies that $\varphi (\beta )$ is a non-decreasing function of $\beta$. In addition, $\varphi (0)=0$ by Equation (50). Taken together, these facts require that $\varphi (\beta )=0$ for all values $\beta \le 0$. On the other hand, for $\beta >0$, the function $\varphi (\beta )$ is analytic with all finite derivatives, which corresponds to higher moments of $\widehat{H}\left[s\right]$ under ${\overline{P}}_{\beta }$. Therefore, $\beta =0$, which corresponds to the typical behavior of the NML, coincides with a second order phase transition point because the function $\varphi (\beta )$ exhibits a discontinuity in the second derivative. In terms of ${\overline{P}}_{\beta }(\widehat{s})$, the phase transition separates a region ($\beta \ge 0$) where all samples $\widehat{s}$ have a finite probability from a region ($\beta <0$) where only one sample, the one with $s^{(i)}=\overline{s},\forall i$, has non-zero probability and $\widehat{H}\left[s\right]=0$.

The phase transition is a natural consequence of the fact that NML provide efficient coding of samples generated from $f(s|\theta )$. It states that codes ${\overline{P}}_{\beta }$ that achieve a compression different from the one achieved by the NML only exist for higher coding costs. Codes with lower coding cost only describe non-random samples that correspond to deterministic models $f(s|\overline{\theta })={\delta }_{s,\overline{s}}$.

3. Discussion

The aim of this paper is to elucidate the properties of efficient representations of data corresponding to universal codes that arise in MDL. Taking NML as a generative model, we find that typical samples are characterized by broad frequency distributions and that they achieve values of the relevance which are close to the maximal possible $\widehat{H}\left[k\right]$.

In addition, we find that samples generated from NML are critical in a very precise sense. If we force NML to use less bits to encode samples, then the code localizes on deterministic samples. This is a consequence of the fact that if there were codes that required fewer bits, then NML would not be optimal.

This contributes to the discussion on the ubiquitous finding of statistical criticality [1,4] by providing a clear understanding of its origin. It suggests that statistical criticality can be related to a precise second order phase transition in terms of large deviations of the coding cost. This phase transition separates random samples that span a large range of possible outcomes (the set $\chi$ in the models discussed above) from deterministic ones, where one outcome occurs most of the time. The phase transition is accompanied by a spontaneous symmetry breaking in the permutation between samples. The frequencies of outcomes in the symmetric phase ($\beta \ge 0$) are generated as independent draws from the same distribution, that is sharply peaked for $\beta >0$ as can be checked in the case of the Dirichlet model. Instead, for $\beta <0$, only one state is sampled. In the typical case, $\beta =0$, the symmetry between outcomes is weakly broken, as there are outcomes that occur more frequently than others. At $\beta =0$, the samples maintain the maximal discriminative power over outcomes. This type of phase transitions in large deviations is very generic, and it occurs in large deviations whenever the underlying distribution develops fat tails (see e.g., [27]).

This leads to the conjecture that broad distributions arise as a consequence of efficient coding. More precisely, broad distributions arise when the variables sampled are relevant, i.e., when they provide an optimal representation. This is precisely the point which has been made in [7,8,9]. The results in the present paper add a new perspective whereby maximally informative samples can be seen as universal codes.

Appendix B.1. Dirichlet Model

In the regime where $\rho \gg 1$ and k large such that we can employ Stirling’s approximation, $k!=\sqrt{2\pi k}{k}^k{e}^{-k}$, the normalization can be calculated as

$$\begin{array}{cc}\hfill \sum _{k=0}^{\infty }\frac{k^k{e}^{-k}{e}^{-z^{*}(\rho )k}}{k!}& \approx \sum _{k=0}^{\infty }\frac{e^{-z^{*}(\rho )k}}{\sqrt{2\pi k}}\hfill \end{array}$$

(A14)

$$\begin{array}{c}={\int }_0^{\infty }\frac{e^{-z^{*}(\rho )k}dk}{\sqrt{2\pi k}}\hfill \end{array}$$

(A15)

$$\begin{array}{c}=\frac{1}{\sqrt{2\pi }}\sqrt{\frac{\pi }{z^{*}(\rho )}}\hfill \end{array}$$

(A16)

$$\begin{array}{c}=\frac{1}{\sqrt{2z^{*}(\rho )}}.\hfill \end{array}$$

(A17)

Similarly, we can also calculate

$$\begin{array}{cc}\hfill \sum _{k=0}^{\infty }\frac{k^{k+1}{e}^{-k}{e}^{-z^{*}(\rho )k}}{k!}& \approx \sum _{k=0}^{\infty }\frac{k{e}^{-z^{*}(\rho )k}}{\sqrt{2\pi k}}\hfill \end{array}$$

(A18)

$$\begin{array}{c}={\int }_0^{\infty }\sqrt{\frac{k}{2\pi }}{e}^{-z^{*}(\rho )k}dk\hfill \end{array}$$

(A19)

$$\begin{array}{c}=\frac{1}{\sqrt{2\pi }}\frac{\sqrt{\pi }}{2{(z^{*}(\rho ))}^{\frac{3}{2}}}\hfill \end{array}$$

(A20)

$$\begin{array}{c}=\frac{1}{{(2z^{*}(\rho ))}^{\frac{3}{2}}}\hfill \end{array}$$

(A21)

and thus, the saddle point value $z^{*}$ can now be evaluated as

$$z^{*}(\rho )\simeq \frac{1}{2\rho }.$$

(A22)

In the same regime, given the determinant $detI(\theta )$ of the Fisher information matrix for the Dirichlet model,

$$detI(\theta )=\prod _{s\in \chi }\frac{1}{{\theta }_s}$$

(A23)

the parametric complexity can be approximated as

$$\begin{array}{cc}\hfill e^{\overline{\mathcal{R}}}& \simeq {\left(\frac{N}{2\pi }\right)}^{(S-1)/2}\int d\theta \sqrt{detI(\theta )}\hfill \end{array}$$

(A24)

$$\begin{array}{c}={\left(\frac{N}{2\pi }\right)}^{(S-1)/2}\frac{\mathsf{\Gamma }{(\frac{1}{2})}^S}{\mathsf{\Gamma }(\frac{S}{2})}\hfill \end{array}$$

(A25)

$$\begin{array}{c}\simeq \frac{e^{\frac{S}{2}(1+log\rho )}}{\sqrt{2\rho }}\hfill \end{array}$$

(A26)

which, together with Equation (20) and the fact and the variance ${⟨k^2⟩}_{z^{*}}-{⟨k⟩}_{z^{*}}^2=2{\rho }^2$, implies that $\mathsf{\Phi }(z^{*}(\rho ))=\frac{1}{2}(1+log\rho )$.

Appendix B.2. Paramagnet Model

The parametric complexity for the paramagnetic Ising model, given $\overline{P}(m)$ in Equation (34), is given by

$$e^{\overline{\mathcal{R}}}=\sum _{M=-N}^N\left(\genfrac{}{}{0pt}{}{N}{\frac{N-M}{2}}\right)e^{\left[M{tanh}^{-1}(M/N)-Nlog\left(2cosh\left({tanh}^{-1}(M/N)\right)\right)\right]}.$$

(A27)

where $M=-N,-N+2,\dots ,N-2,N$ runs on $N+1$ values. When $N\gg 1$, the magnetization, $m=M/N$, can be treated as a continuous variable and consequently, the sum can be approximated as an integral: ${\sum }_M\dots \simeq \frac{N}{2}{\int }_{-1}^{1}dm\dots$. Hence, by using the identities ${tanh}^{-1}(m)=\frac{1}{2}log\frac{1+m}{1-m}$, $cosh\left({tanh}^{-1}(m)\right)=\frac{1}{\sqrt{1-m^2}}$ and $K!\simeq K^{K}e-K\sqrt{2\pi K}$, one finds that

$$\begin{array}{cc}\hfill e^{\overline{\mathcal{R}}}& \simeq \frac{N}{2}{\int }_{-1}^1\frac{1}{\sqrt{2\pi N(1-m^2)}}\hfill \end{array}$$

(A28)

$$\begin{array}{c}=\sqrt{\frac{\pi N}{2}}.\hfill \end{array}$$

(A29)

Appendix C.1. Sampling Universal Codes through Markov Chain Monte Carlo

Unlike the Dirichlet model and the independent spin model, analytic calculations for the Sherrington-Kirkpatrick (SK) model and the restricted Boltzmann machine (RBM) are generally not possible, because the partition function Z, and consequently, the UC partition function $e^{\overline{\mathcal{R}}}$, is computationally intractable. In order to sample the NML for these graphical models, we turn to a Markov chain Monte Carlo (MCMC) approach in which the transition probability, $P(\widehat{s}\to {\widehat{s}}^{\prime })$, can be built using the following heuristics:

• Starting from the sample, $\widehat{s}$, we calculate the maximum likelihood estimates, $\widehat{\mathbf{\theta }}(\widehat{s})$, of the parameters of the model, $p(\widehat{s}|\theta )$ by either solving Equation (41) for the SK model or by Contrastive Divergence (CD${}_{\kappa }$) [24,25] for the RBM (see Appendix C.2).
• We generate a new sample, ${\widehat{s}}^{\prime }$ from $\widehat{s}$ by flipping a spin in randomly selected r points $s^{(i)}$ of the sample. The number of selected spins, r, must be chosen carefully such that r must be large enough to ensure faster mixing but small enough so the new inferred model, $p({\widehat{s}}^{\prime }|\theta )$, is not too far from the starting model, $p(\widehat{s}|\theta )$.
• The maximum likelihood estimators, $\widehat{\mathbf{\theta }}({\widehat{s}}^{\prime })$ for the new sample are calculated as in Step 1.
• Compute

  $$\Delta E=logp({\widehat{s}}^{\prime }|\widehat{\mathbf{\theta }}({\widehat{s}}^{\prime }))-logp(\widehat{s}|\widehat{\mathbf{\theta }}(\widehat{s}))$$

  (A30)

  and accept the move $\widehat{s}\to {\widehat{s}}^{\prime }$ with probability $min\left(e^{N\Delta E},1\right)$.

Appendix C.2. Estimating RBM Parameters through Contrastive Divergence

Given a sample, $\widehat{\mathit{v}}=({\mathit{v}}^{(1)},\dots ,{\mathit{v}}^{(N)})$, of N observations, the log-likelihood for the restricted Boltzmann machine (RBM) is given by

$$log\mathcal{L}(\mathbf{\theta })=\sum _{k=1}^{N}log\sum _{\mathit{h}}P({\mathit{v}}^{(k)},\mathit{h}|\mathbf{\theta }).$$

(A31)

The inference of the parameters, $\mathbf{\theta }$, proceeds by updating $\mathbf{\theta }$ such that the log-likelihood, $log\mathcal{L}(\theta )$, is maximized. This updating formulation for the parameters is given by

$$\Delta \theta =\frac{ϵ}{N}\frac{\partial log\mathcal{L}(\mathbf{\theta })}{\partial \theta }$$

(A32)

where $ϵ$ is the learning rate parameter. The corresponding gradients for the parameters, $\mathit{w}$, $\mathit{a}$ and $\mathit{b}$ can then be written down respectively as

$$\begin{array}{cc}\hfill \frac{\partial log\mathcal{L}(\mathbf{\theta })}{\partial w_{ij}}& =\sum _{k=1}^N\left[\sum _{\mathit{h}}{v}_i^{(k)}{h}_{j}P({\mathit{v}}^{(k)},\mathit{h}|\mathbf{\theta })-\sum _{\mathit{v}}\sum _{\mathit{h}}{v}_i{h}_{j}P(\mathit{v},\mathit{h}|\mathbf{\theta })\right]\hfill \end{array}$$

(A33)

$$\begin{array}{cc}\hfill \frac{\partial log\mathcal{L}(\mathbf{\theta })}{\partial a_i}& =\sum _{k=1}^N\left[\sum _{\mathit{h}}{v}_i^{(k)}P({\mathit{v}}^{(k)},\mathit{h}|\mathbf{\theta })-\sum _{\mathit{v}}\sum _{\mathit{h}}{v}_{i}P(\mathit{v},\mathit{h}|\mathbf{\theta })\right]\hfill \end{array}$$

(A34)

$$\begin{array}{cc}\hfill \frac{\partial log\mathcal{L}(\mathbf{\theta })}{\partial b_j}& =\sum _{k=1}^N\left[\sum _{\mathit{h}}{h}_{j}P({\mathit{v}}^{(k)},\mathit{h}|\mathbf{\theta })-\sum _{\mathit{v}}\sum _{\mathit{h}}{h}_{j}P(\mathit{v},\mathit{h}|\mathbf{\theta })\right]\hfill \end{array}$$

(A35)

where the first terms denote averages over the data distribution while the second terms denote averages over the model distribution.

Here, we use the contrastive divergence (CD) approach which is a variation of the steepest gradient descent of $\mathcal{L}(\mathbf{\theta })$. Instead of performing the integration over the model distribution, CD approximates the partition function by averaging over distribution obtained after taking $\kappa$ Gibbs sampling steps away from the data distribution.

To do this, we exploit the factorizability of the conditional distributions of the RBM. In particular, the conditional probability for the forward propagation (i.e., sampling the hidden variables given the visible variables) from $\mathit{v}$ to $h_j$ reads as

$$P(h_j=1|\mathit{v},\mathbf{\theta })=\frac{1}{1+exp\left(-b_j-{\sum }_i{v}_i{w}_{ij}\right)}.$$

(A36)

Similarly, the conditional probability for the backward propagation (i.e., sampling the visible variables from the hidden variables) from $\mathit{h}$ to $v_i$ reads as

$$P(v_i=1|\mathit{h},\mathbf{\theta })=\frac{1}{1+exp\left(-a_i-{\sum }_j{h}_j{w}_{ij}\right)}.$$

(A37)

The Gibbs sampling is done by propagating a sample, ${\mathit{v}}^{(k)}={\mathit{v}}^{(k)}(0)$, forward and backward $\kappa$ times: ${\mathit{v}}^{(k)}(0)\to {\mathit{h}}^{(k)}(0)\to {\mathit{v}}^{(k)}(1)\to \dots \to {\mathit{h}}^{(k)}(\kappa -1)\to {\mathit{v}}^{(k)}(\kappa )\to {\mathit{h}}^{(k)}(\kappa )$. And thus, the Gibbs sampling approximates the gradient in Equation (A33) as

$$\frac{\partial log\mathcal{L}(\mathbf{\theta })}{\partial w_{ij}}=\sum _{k=1}^N\left[v_i^{(k)}(0)h_j^{(k)}(0)-v_i^{(k)}(\kappa )h_j^{(k)}(\kappa )\right].$$

(A38)

In the CD approach, each parameter update for a batch is called an epoch. While larger $\kappa$ approximates well the partition function, it also induces an additional computational cost. To find the global minimum more efficiently, we randomly divided the samples into groups of mini-batches. This approach introduces stochasticity and consequently reduces the likelihood of the learning algorithm to be confined in a local minima. However, a mini-batch approach can result in data-biased sampling. To circumvent this issue, we adopted the Persistent CD (PCD) algorithm where the Gibbs sampling extends to several epochs, each using different mini-batches. In the PCD approach, the initial visible variable configuration, ${\mathit{v}}^{(k)}(0)$, was set to random for the first mini-batch, but the final configurations, $({\mathit{v}}^{(k)}(\kappa ),{\mathit{h}}^{(k)}(\kappa ))$, of the current batches become the initial configuration for the next mini-batches. In this paper, we performed Gibbs sampling at $\kappa =10$ steps where we update the parameters, $\mathbf{\theta }$, are updated at 2500 epochs at a rate $ϵ=0.01$ with 200 mini-batches per epochs. For other details regarding inference of parameters of the RBM, we refer the reader to [24,25].

Appendix C.3. Source Codes

All the calculations in this manuscript were done using personalized scripts written in Python 3. The source codes are accessible online (https://github.com/rcubero/UniversalCodes (accessed on 8 May 2012)).