Abstract
We study the channel output distribution induced by a random code of rate R from the perspective of statistical physics. The central object is the partition function , where is the channel output vector, is the code, and plays the role of inverse temperature. More precisely, our focus is on the associated annealed free energy, , where the expectation is with respect to the randomness of . This quantity encodes the full Rényi spectrum of the output distribution. The single-letter formula derived for the annealed free energy decomposes into two branches, which reflect a “competition” between two populations of codewords. One is the bulk branch, , which is driven by typical codewords, and the other one is the sparse branch , which is driven by atypical codewords, where the qualifiers ‘typical’ and ‘atypical’ are in a sense that will become apparent later. We analyze the phase structure of each branch separately and characterize their competition. Both branches are derived for all . The phase boundary , where the two branches are equal, is analyzed for , where it has an explicit closed-form expression. The phase diagram in the first quadrant of the plane has four regions separated by three boundaries: (bulk branch transition), (bulk–sparse competition boundary), and (sparse branch transition), all meeting at the point , where is the mutual information induced by the input type and the channel. Applications to guesswork, channel resolvability, and hypothesis testing are discussed, and all the results are illustrated with a numerical example of a Z-channel.
1. Introduction
Given a codebook of size and a discrete memoryless channel (DMC), , the induced channel output distribution is given by the mixture
This object is central to channel resolvability [1] and the soft-covering problem (see, e.g., [2,3,4] and the references therein). Classical results, in this context, focus on a single threshold: if , is the mutual information induced by the input distribution and the channel W, then the distribution converges to the i.i.d. product law in total variation as well as in some other metrics between probability distributions.
This description, however, captures average behavior only. It says nothing about the internal geometry of : how the probability mass is distributed across output sequences, how many codewords support a typical and an atypical output, or how the self-information, , fluctuates. These questions require a deeper journey beyond the Shannon entropy and mutual information.
In this work, we study through the function
where the second representation is readily recognized as a partition function, with playing the role of inverse temperature and being the energy function (a.k.a. the Hamiltonian) associated with every micro-state . In other words, this is identified as the canonical partition function of a statistical-mechanics system whose micro-states are channel output sequences. Two special values bracket the range: (normalization, ) and (the mode, or the ground state in physics jargon). In general, is exactly the definition of , the Rényi entropy of order , pertaining to the output distribution , which is non-increasing in . The associated free energy encodes the exponential growth/decay rate of as a function of n. Since depends on the random codebook and, actually, should be denoted , we study the free energy behavior for the average code, which is identified with the annealed free energy,
where the existence of the limit will be evident from the exponentially tight analysis in the proof of Theorem 1 in the sequel. Here denotes the expectation operator with respect to (w.r.t.) the randomness of the code . Its phase structure—the rich dependence on and R—is the main subject of this work.
It should be emphasized that many earlier statistical-mechanics analyses of random coding were based on Derrida’s random energy model (REM) (see, e.g., [5,6] and the references therein). These are based on partition functions whose micro-states were the channel inputs (i.e., the codewords) for a fixed channel output sequence , that is,
for fixed , whereas, here, the micro-states are the channel outputs, as mentioned earlier. This difference is significant because, here, each energy term, , is itself a log-partition function over codewords. This two-level structure makes the problem considerably harder and richer than that of an ordinary REM.
The main contributions of this work are as follows. We derive the annealed free energy and analyze the phase structure of its two branches (Theorems 1 and 3). The free energy decomposes into a bulk branch and a sparse branch, reflecting a competition between two populations of codewords. The bulk branch is driven by typical codewords and the sparse branch is driven by atypical codewords, where the meanings of the qualifiers ‘typical’ and ‘atypical’ will become apparent in the sequel. Both branches are derived for all and analyzed separately. The bulk branch, (Section 3), has a single phase transition at a critical rate —the mutual information of the bulk optimizer (the unconstrained optimizer of the bulk branch)—where a rate constraint changes from inactive to active. A similar comment applies to the sparse branch, , whose phase boundary is . The total annealed free energy has a phase boundary given by an explicit closed-form formula (Theorem 3(b), Section 3). Together, the three boundaries divide the quadrant into four regions, denoted A, B, C, and D. By construction, always, with equality in regions A and B and strict inequality in regions C and D, where the sparse branch dominates. For the sparse branch, a fully explicit closed-form formula, in terms of channel transition probabilities and alone, holds whenever the rate is below (see Equation (29)). All the results are illustrated in a numerical example in Section 4.
A few words are in order with regard to earlier related work and the differences relative to the present work. The model (1) is the canonical object of channel resolvability and the soft-covering problem. Han and Verdú [1] established the resolvability threshold under total variation, for , and showed the threshold is tight. Hayashi [3] derived exponential convergence rates under total variation and normalized relative entropy, a.k.a. the Kullback–Leibler (KL) divergence. Hou and Kramer [4] extended the analysis to the Rényi divergence , showing the critical rate remains for all but the exponents differ. Yu and Tan [7] characterized the Rényi resolvability—the minimum rate R required for the Rényi divergence to vanish asymptotically. They showed that, for , the threshold remains , while, for , it is strictly larger than and depends on .
Our work is complementary to [7]: rather than characterizing the threshold rate at which a divergence to vanishes, we study the exact value of the annealed free energy at every rate R and inverse temperature , revealing a two-branch phase diagram with three distinct phase boundaries. We analyze the below-threshold regime with the same precision as the above-threshold regime and expose qualitative phenomena—condensation, sparse-branch dominance, and the annealed/quenched gap—invisible to divergence-based analyses. In particular, our bulk branch boundary, , for is precisely the Rényi resolvability threshold of [7], now seen as one of three phase boundaries in a richer thermodynamic landscape.
In general, all the earlier results, in this context, share a common feature: they measure how close is to a target distribution via some metric, and they all identify as the single relevant threshold. The present paper asks a fundamentally different and complementary question: not how close is to a target but what is the internal geometry of itself?
Our partition function encodes the Rényi structure of without reference to any external target distribution and reveals phenomena invisible to total variation or KL divergence. It turns out that the classical threshold, , is only the beginning of the story: the quadrant is divided into four distinct regions by three phase boundaries, all passing through the point . Concretely:
- Even for (where ), the annealed free energy reveals a condensed phase: when for some , the bulk branch of has an active rate constraint, indicating that probability mass concentrates on outputs supported by sub-exponentially few codewords—output condensation that is invisible to total variation distance.
- The annealed free energy has a phase boundary where the sparse branch takes over from the bulk branch, signaling that the atypical codewords dominate the ensemble average.
- Since exactly for every fixed codebook (by the definition of Rényi entropy), the bulk branch encodes the Rényi entropy rate of the output mixture for a typical codebook (for and ; this is proved in Theorem 2; see also Appendix A). The full annealed free energy thus encodes the complete Rényi-order profile of the output distribution, of which classical soft covering () is a single cross-section.
In summary, this work does not contradict soft-covering results but refines and extends them by revealing the fine structure of that lies beneath the single threshold . These structural findings have direct operational consequences, which we now describe.
- 1.
- Guesswork and Rényi entropy. Arikan [8] showed that guesswork moments grow as . The annealed free energy encodes the Rényi entropy rate of the output mixture, extending Arıkan’s analysis to the case where the source is itself a random-coding mixture. The phase structure of produces regime changes in guessing complexity that are absent in the i.i.d. case.
- 2.
- Hypothesis testing. The Chernoff exponent [9] for testing against equals , directly connecting our free energy to the optimal test. With being uniform, for , meaning the two distributions are not exponentially distinguishable at and above the soft-covering threshold. The phase structure of produces qualitative regime changes in as R varies, and it connects to the Rényi resolvability results of Yu and Tan [7]; see Section 5.
- 3.
- Statistical mechanics of codes. Sourlas [10] established the connection between linear codes and spin-glass models. Montanari [11] analyzed the phase transition in turbo codes. Mézard and Montanari [5] (Chapters 5 and 6) and Merhav [6] provided a comprehensive treatment of random coding via statistical physics, with the partition function summing over codewords (inputs) for a fixed received sequence. The present work complements [6] by placing the partition function over outputs: the resulting two-level hierarchical model has a distinct phase structure governed by the code rate R rather than by signal-to-noise ratio.
The outline of the remainder of this article is as follows. Section 2 defines the model and establishes the notation conventions. Section 3 derives the annealed free energy and analyzes the phase structure of its bulk and sparse branches; the phase boundary has an explicit closed-form formula for . Section 4 presents the phase diagram (four regions, ). Section 5 discusses applications and the implications of our results. Finally, Section 6 summarizes the paper and provides an outlook.
2. Model, Definitions and Notation
Throughout the paper, n denotes the block length. An n-vector over alphabet is displayed as , being the single-letter finite alphabet. Similarly, , where the single-letter alphabet is also finite. The empirical distribution (type) of is denoted for all . A similar definition applies to joint types of pairs of sequences, . A DMC W is a stochastic matrix , and, for sequences and , we write . Throughout the sequel, all logarithms are natural (base e). Accordingly, information measures are given in nats. We use the standard exponential-equivalence notation to mean ; i.e., f and g have the same exponential growth/decay rate. The notation of information measures is as follows. is the Kullback–Leibler (KL) divergence,
Throughout the paper, denotes a joint distribution on with X-marginal given by , being a fixed input distribution on the finite input alphabet . When needs to appear as a subscript in an information measure, we abbreviate by Q to avoid cumbersome notation. Thus, , , and stand for , , and , respectively, which are the entropy of Y, the conditional entropy of Y given X, and the mutual information between X and Y, respectively, all induced by . On the other hand, when appears as an argument of a certain functional, the full notation is used. The notation (without subscript) refers to the mutual information induced by the fixed single-letter channel input distribution and the channel transition probability matrix W, i.e., . The KL divergence between a conditional distribution and W, with weighting by , is defined by
A random codebook of rate and block length n is a collection and , where the codewords are drawn independently and uniformly at random from the type class,
the set of all n-sequences with empirical distribution exactly . Given codebook , the induced output distribution is
For , the partition function is defined as
The annealed free energy is
where the limit will be evident in the subsequent analysis. For a joint type on with marginal , define:
Note that, for of joint type , . We shall often use the identity
Fix an output sequence of type . The type-class enumerator counts how many codewords have a joint type (joint empirical distribution) with :
where is the joint type class associated with the joint distribution .
Using the method of types [12,13], it is readily observed that, given , each randomly selected codeword, , satisfies with probability
where is the conditional type class, and where we have used the fact that . Since codewords are drawn independently at random from , it is clear that is a binomial random variable with M trials and a probability of single success of the exponential order of .
3. Annealed Free Energy and Its Phase Structure
3.1. Statistical-Mechanics Perspective
Before presenting the formulas, we pause to explain the statistical-mechanics framework, why the qualifiers “annealed” and “quenched” are used, and why the analysis goes beyond a simple saddle-point calculation.
3.1.1. Annealed vs. Quenched: Origin of the Terminology
The terms come from metallurgy and were adopted by the statistical-mechanics community to describe two ways of averaging over quenched (i.e., frozen) disorder. In a disordered system—such as a spin glass—the disorder (e.g., the random coupling constants) is fixed for each physical realization but is drawn from some distribution. Annealing is the physical process of heating a material and cooling it slowly so that the disorder equilibrates with the thermal fluctuations; mathematically, this corresponds to averaging the partition function before taking its logarithm as if the disorder itself participates in the thermal equilibrium:
Quenching is the opposite process: the disorder is frozen rapidly and does not equilibrate. The free energy of a typical frozen realization is then
i.e., the logarithm is taken before averaging. In our setting, the “disorder” is the random codebook ; once drawn, it is fixed and does not change with . The annealed free energy defined in (10) is . By Jensen’s inequality (since log is concave), always. The main subject of this paper is (Theorems 1 and 3 below); whether and when is addressed in Theorem 2.
3.1.2. The Annealed Free Energy and Its Information-Theoretic Meaning
The partition function is defined in (9). Its exponential growth rate encodes how the output probability mass is distributed: large emphasizes the most probable outputs (low energy, high ), while weights all outputs equally. The special value gives (normalization), and, as shown below, is proportional to the Rényi entropy of the output mixture (defined in the sequel). The annealed free energy thus encodes the Rényi entropy profile of the output distribution averaged over codebooks. Its exact single-letter formula—derived in Theorem 1—reveals a competition between two populations of codewords, called the bulk branch and the sparse branch , to be defined precisely in the sequel.
3.1.3. Why This Is More than a Saddle-Point Calculation
A naive approach would replace by its expectation and compute deterministically. This is correct only if concentrates around its mean at the exponential scale, i.e., if with high probability. When this self-averaging property holds, the annealed and quenched free energies coincide and a saddle-point over the output type gives the answer. However, self-averaging can fail: the annealed average may be dominated by a sub-exponential fraction of codebooks that are far from typical, specifically those containing a codeword whose joint type with some output satisfies (a “sparse-type” codeword, in the terminology of the sequel). These atypical codebooks contribute negligibly in probability but dominate the expectation of , producing an annealed free energy strictly larger than the quenched one. Identifying when this happens—and characterizing both the typical behavior and the atypical behavior as a function of —requires tracking two competing saddle points simultaneously rather than a single one. This competition produces the two-branch structure of and the three phase boundaries (to be derived in Theorem 3).
3.1.4. Phase Transitions and the Two-Level Hierarchical Structure
The phase boundaries of —to be characterized explicitly in Theorem 3 and illustrated in Section 4—signal qualitative changes in which configurations dominate . This is qualitatively different from earlier statistical-mechanics analyses of random codes (see [5,6]), which placed the partition function over codewords (inputs) for a fixed output : , as in (4). In that case, the energies are approximately i.i.d. across codewords (analogous to Derrida’s random energy model), and the analysis reduces to a single saddle point. In the present work, the energy of each output micro-state is , which is itself a log-partition function over codewords—a two-level hierarchical structure. This makes the energies neither independent nor identically distributed across output sequences and is the source of the richer two-branch phase diagram with three distinct boundaries.
3.2. Main Result: Formula for the Annealed Free Energy
In this section, we derive a single-letter expression for the annealed free energy and investigate its phase structure in the plane. The first main result is the following.
Theorem 1
(Annealed Free Energy, ). For all :
with
For convenience, we define the functional
which is the common building block of both branches. Using the identity , we may present the two branches of the annealed free energy as
Proof of Theorem 1.
We begin with the trivial identity,
We next focus on for a fixed . Define so that . Since is deterministic,
The problem reduces to computing . For each ,
where the dotted equalities hold since the sum contains at most non-negative terms. Now, the evaluation of requires the following lemma, which is Theorem 4.2 of [14], and the proof is therein.
Lemma 1
(Moments of Binomial Enumerator). Let with and . Then,
The case corresponds to a non-exponential behavior of , as can be seen by observing the limiting behavior of both cases. (As a side remark, when N is for some constant , then, in the limit of , N becomes a Poissonian random variable with parameter ; that is, for every non-negative integer k, and, therefore, the asymptotic th moments are constants).
Lemma 1 is now used with the assignments and . Accordingly, one must distinguish between types for which as opposed to those with . Now,
Since the latter expression involves maximization over when is held fixed, it is clear that it depends on only via . The number of with is of the exponential order of . Multiplying the per- contributions by and yields the total contribution of type , and, finally, summing over all , which is exponentially equivalent to maximizing over all , yields the asserted expression of . The part pertaining to types with is identified with , and the one with is associated with . □
For future reference, we need the following definition.
Definition 1
(Sparse-feasible and bulk-feasible types). For a given rate , a joint distribution with is called:
- Sparse-feasible if ; i.e., it satisfies the constraint of the sparse branch optimization;
- Bulk-feasible if ; i.e., it satisfies the constraint of the bulk branch optimization. Every is either sparse-feasible or bulk-feasible (or both if ).
3.3. The Sparse Branch: Formula and Phase Structure
The sparse branch of the annealed free energy,
captures the contribution of sparse-type codewords: those whose joint type with the output satisfies . For fixed , such codewords are rarely encountered in a typical codebook of the ensemble.
Definition 2
(The annealed optimizer and its mutual information ). For each , define
It can be readily shown that maximizes over all with . Accordingly, define
The curve marks the point where crosses from sparse-feasible (, so optimizes the sparse branch) to bulk-feasible (, so optimizes the bulk branch).
- Closed-form formula. By Lemma 1, when , the th moment of the enumerator satisfies regardless of . The resulting annealed calculation giveswhereThe formula ceases to hold when as is then bulk-feasible () and hence does not comply with the rate constraint of the sparse branch.
- Phase structure. The sparse branch has its own phase transition at :
- −
- For : , the closed-form linear formula. The sparse branch is maximized by .
- −
- For : is not sparse-feasible; the closed form does not apply and the sparse branch must be evaluated directly from (26).
- Operational meaning. The sparse branch governs the behavior of the ensemble average when the latter is inflated by atypical codebooks. Specifically, when (which occurs for sufficiently small R, as characterized by the phase boundary derived in Theorem 3 below), the annealed free energy satisfies , meaning that seems to be dominated by a sub-exponential fraction of codebooks with atypical codewords (those containing a sparse-type codeword with ). A typical codebook has .
3.4. The Bulk Branch: Operational Meaning
We now explain why the bulk branch deserves a separate study within the analysis of . As said before, the bulk branch, , is the component of driven by typical codewords—those whose joint type with the output satisfies , i.e., codewords that are plausible given the code rate. A typical random codebook contains no sparse-type codewords (those with ) with high probability, so, for a typical fixed codebook, at the exponential scale. The annealed free energy can exceed (when the sparse branch dominates) because it is inflated by the rare codebooks that happen to contain a sparse-type codeword. In the language of statistical physics, is the quenched free energy, which is the free energy of a typical random code, while is the annealed free energy (the free energy of the disorder-averaged partition function). To support this observation, we now state the self-averaging property of for and . The proof appears in the Appendix A.
Theorem 2
(Self-averaging and quenched free energy). For and :
- (i)
- (ii)
The condition is precisely the regime (regions A and B in the phase diagram) where the bulk branch dominates the annealed free energy: (Theorem 3). This is needed for the proof of the a.s. upper bound: Markov’s inequality on gives eventually a.s., and this equals only when . The lower bound of the proof holds under the weaker condition , so region C (), where a typical codebook still lives in the bulk phase but the annealed average is inflated by rare sparse-type codebooks, remains open. Whether the result also extends to the condensed phase or to likewise remains open.
By the definition of Rényi entropy, , so exactly for every codebook. Since a.s. (Theorem 2), the bulk branch encodes the Rényi entropy rate of the output mixture at order :
for a typical fixed codebook.
3.5. The Bulk Branch: Formula and Phase Structure
Definition 3
(The bulk optimizer and its mutual information ). For each , define
the unconstrained maximizer of the bulk objective, and set
Obviously, for ,
The curve is therefore the bulk phase boundary: for the rate constraint is inactive (bulk phase), while for it is active (condensed phase).
The next lemma establishes an inequality relation between and .
Lemma 2
(Ordering of the two optimizers). For :
with strict inequality for . At , .
Proof.
By the optimality of in the bulk objective:
which, for , is equivalent to
where the right-hand side is clearly non-negative since the denominator is positive, and the numerator is non-negative as is the global maximum of , which cannot be smaller than . Therefore, is non-negative as well. □
3.6. Phase Structure of the Annealed Free Energy
The annealed free energy formula (Theorem 1) holds for all . However, the analysis of the phase boundary between the bulk and sparse branches is restricted here to the range for two reasons. The first reason is motivational: In the statistical-mechanics language, is the low-temperature regime where is closely related to Rényi entropies of order at least one. This range captures the operationally most relevant quantities: (Shannon entropy and resolvability threshold), (collision entropy and birthday attacks), (minimum entropy and ), and general (guessing moments [8]. The second reason is that the range does not appear to lend itself to closed-form analysis, the main issue being the lack of guarantee concerning the uniqueness of the solution R to the equation , which is the phase boundary of the total annealed free energy, as will be defined next. In particular, for , the branches and may or may not cross, and existence and uniqueness of a crossing are not established in general.
Definition 4
(Annealed Phase Boundary). For , the annealed phase boundary is the unique rate at which the bulk and sparse branches are equal:
Existence and uniqueness of are established in Theorem 3(b) below, together with an explicit formula.
In order to keep track of the various phase boundaries and the quantities associated with them so far, the following table summarizes those ingredients (Table 1).
Table 1.
Summary of the two optimizers.
The following theorem provides a characterization of the phase structure of the annealed free energy.
Theorem 3.
For any DMC W, any , and any , :
- (a)
- For , is sparse-feasible and
- (b)
- (Annealed phase boundary, .) LetFor all , , which is the unique rate at which , and
The curve is the annealed phase boundary.
Proof.
Part (a) was already shown earlier (but we include it here as part of the theorem for completeness since it is used in part (b)): Observe that, for , we have , so it is sparse-feasible. Since achieves the global maximum, , so .
As for part (b), observe that, for , both simplified formulas hold simultaneously: since ) and (by part (a) since ). Setting and solving gives immediately
Substituting , we end up with:
Since and , the second term is non-negative, giving , with strict inequality since strictly for (Lemma 2).
For the upper bound, recall from the proof of Lemma 2 that
which gives
Substituting into (43): , again with strict inequality for . Hence , confirming the formula is valid.
To establish the uniqueness of the solution to the equation , it remains to rule out any additional solutions outside the interval . For , is still sparse-feasible (since ), so part (a) gives . The bulk constraint is active, so . Hence,
The right side is positive at (as shown above) and increasing as R decreases (since ). Hence for all . For , consider the following. Since , the bulk constraint is inactive and . Since and (from the formula of ), we have:
where the strict inequality uses and , and the last inequality is because . Hence for all . □
4. A Numerical Example
In this section, we provide a numerical example and illustrate the behavior of the two branches of the annealed free energy as well as the phase boundary curves.
All phase diagrams are computed for an example of a Z-channel. The details are as follows. The input and output alphabets are , and the crossover probability is , i.e., , , , and . The input type is given by . Accordingly, the resulting channel mutual information is .
Figure 1 plots all the phase boundaries together in the relevant quadrant of the plane.
Figure 1.
Phase boundaries for the Z-channel, . Red: , the bulk branch boundary (bulk rate constraint activates). Blue: , the annealed boundary where bulk = sparse (closed-form formula, Theorem 3(b)). Green:, the sparse branch boundary. All three meet at (filled dot). Four regions (Remark 4): A (): unconstrained, constrained (sparse closed form does not apply); bulk dominates; B (): bulk branch dominates (); C (): sparse branch dominates (); D (): sparse branch dominates (), bulk condensed.
Three phase boundary curves are visible in Figure 1:
- (red): the bulk branch boundary, where the unconstrained bulk optimizer crosses the rate constraint. At : .
- (blue): the annealed boundary where the bulk and sparse branches are equal, given by the closed-form Formula (40). At : . Decreasing in , staying above .
- (green): the sparse branch boundary, where (which maximizes ) has . For the sparse closed form ceases to hold. At : .
- All three curves meet at the point , confirming that the soft-covering threshold is the unique point where all regions collapse to a single point.
For , the three phase boundaries divide the quadrant into four regions:
- A (, above all curves): is unconstrained (). is constrained: since , the sparse optimizer is bulk-feasible and the sparse closed-form expression does not apply. Bulk dominates: (since ).
- B (): Bulk branch dominates: . The bulk rate constraint is inactive (); no condensation.
- C (): Sparse branch dominates: (closed-form linear formula). Rare-event codewords inflate the ensemble average; a typical codebook is in the bulk phase.
- D (, below the red curve): Sparse branch dominates: . The bulk branch is condensed (rate constraint active; bulk optimizer constrained to ), but the annealed free energy is inflated by rare sparse-type codebooks.
- For the very same z-channel example, Figure 2, Figure 3 and Figure 4 display and as functions of for three representative values of R (one below , another one equal to , and yet another one above ) and contrasts them with the i.i.d. reference free energyFigure 2. . The sparse branch lies above the bulk branch for all , so the annealed free energy is dominated by the sparse branch; both lie well above the i.i.d. reference, reflecting ensemble inflation by rare codebooks.Figure 3. nats (the soft-covering threshold). Both branches and the i.i.d. reference all vanish at (since ) and diverge differently for .Figure 4. . The bulk branch dominates throughout and tracks the i.i.d. reference closely, confirming that, above the soft-covering threshold, the output mixture approximates in the Rényi sense. In all three figures, , with strict inequality below : the random-coding mixture has strictly higher Rényi entropy than the i.i.d. output distribution at every order .
A central message of this paper is that the classical soft-covering threshold is far from the end of the story, and, as this numerical example illustrates, we refine and deepen those results in this work. As mentioned in the Introduction, classical results, like those of [1,3,7], tell us that (and in other metrics) if and only if and provide exponential rates for this convergence. Our statistical-physics analysis reveals that, even well above this threshold, retains non-trivial internal structure that is invisible to total variation. Figure 2, Figure 3 and Figure 4 make this visible. The three plots of , , and the i.i.d. reference against for . At the classical threshold (second diagram), all three quantities vanish together at as required but diverge for : the free energy lies strictly above for all , meaning that has strictly higher Rényi entropy than at every order. Above the threshold at (third diagram), the bulk branch closely tracks , confirming the soft-covering intuition that the output mixture approaches the i.i.d. law, but the gap, while small, remains nonzero at finite , and quantifying it requires . Below the threshold at (first diagram), the gap is large and grows with : the sparse branch dominates the ensemble average, and the random-coding output is far from i.i.d. in the Rényi sense even though TV-distance arguments do not apply here at all.
Specifically, is the Rényi entropy rate of at order for a typical codebook, and its phase transition at implies a qualitative change in how probability mass is distributed over output sequences. In the bulk phase (), a typical output sequence is supported by exponentially many codewords. The output distribution is diffuse and “spread out” in a way that is consistent with the intuition behind soft covering. In the condensed phase (), a typical output appears to be supported by only sub-exponentially few codewords, meaning the mass of concentrates on a sparse set of sequences even though the TV distance to may already be negligible. This condensation is visible in Figure 2: the first diagram () shows for all , so the ensemble average is dominated by the rare sparse-branch codebooks, while the third diagram () shows , confirming typical-codebook dominance throughout.
The gap visible in all three diagrams has a direct operational meaning: it measures how much the random-coding output mixture exceeds the i.i.d. distribution in Rényi entropy at order . Classical soft-covering results establish that this gap vanishes in total variation for . The figures show that, in Rényi entropy, the gap is nonzero for every and every finite R, decaying to zero only as .
5. Applications and Implications
In this section, we summarize several applications and implications of the results in this work. Further investigation of these applications is deferred to future work.
- 1.
- Refined output statistics. Resolvability shows for . Our phase diagram reveals that, even in this regime, may be in the condensed phase (below for large ), where mass appears to concentrate on outputs supported by a sub-exponential number of codewords. This is invisible to the total variation distance and other distances.
- 2.
- Guesswork. [8]. Since and (the full annealed free energy, including both bulk and sparse branches), the guessing exponent at is . The phase structure of —both its bulk phase transition at and its bulk–sparse crossover at —therefore produces regime changes in guessing complexity absent in the i.i.d. case.
- 3.
- Security. In the condensed phase, even when in total variation, only a sub-exponential number of codewords may generate a typical output. A computationally powerful adversary can identify the message by examining which codewords are consistent with the observed output. Semantic security may therefore require operating strictly in the bulk phase for all relevant to the adversary’s test.
- 4.
- Hypothesis testing and the Chernoff exponent. A natural question is how well one can distinguish the random-code output mixture from the i.i.d. reference . With (achieved by choosing appropriately), this is a direct test of whether soft covering has succeeded. The annealed Chernoff exponent for this binary hypothesis test equalswhere is the annealed free energy of Theorem 1. Since both branches of are fully characterized by our results, (49) is completely determined. The optimizer satisfies , so the phase structure of directly governs the behavior of : the bulk/sparse phase boundaries produce regime changes in the Chernoff exponent as R varies.
With , two clean consequences follow. First, if and only if : the two distributions are not exponentially distinguishable for ; i.e., no test can achieve an exponentially small total error probability exactly at the soft-covering threshold. (Sub-exponential error rates, e.g., polynomial in n, are not excluded). Second, (49) is the Legendre–Fenchel transform of with respect to , evaluated at ; it therefore equals , which is precisely the Rényi divergence at order . This connects (49) directly to the Rényi resolvability results of Yu and Tan [7], who showed that if and only if for , while, for , the threshold is —the bulk condensation boundary of our phase diagram. Our analysis provides the exact rate of divergence at every R and and reveals how the two-branch phase structure of determines qualitative changes in this rate.
6. Conclusions
We developed a statistical-mechanics framework for the output distribution of random codes, centered on the annealed free energy of the partition function and its two-branch phase structure. This dichotomy between the two branches has concrete operational consequences. For guesswork, the phase structure of —both the bulk phase transition at and the bulk–sparse crossover at —translates directly (via Arıkan’s formula [8]) into regime changes in the exponential growth rate of guessing moments . Similar comments apply to hypothesis testing and perhaps other application areas. The annealed free energy adds yet another layer: its phase boundary separates the regime where the ensemble average is dominated by rare atypical codebooks (for example, the sparse branch dominates in Figure 2) from the regime where it reflects the behavior of a typical codebook (the bulk branch dominates in Figure 4). The gap in the sparse-dominant regime quantifies the extent to which ensemble averages can be misleading—a warning that is particularly relevant when analyzing the security of specific random codes rather than random coding in the mean.
Future research directions include: a rigorous derivation of exponentially tight error exponents for the hypothesis test of Application 4 via the type-class enumeration method of [14], together with a phase-diagram analysis of the Neyman–Pearson exponent tradeoff in the plane; a full treatment of the guesswork regime changes produced by the bulk/sparse phase boundaries; quantitative semantic security bounds in terms of and ; extension to Markov and mixed sources; connections to mismatched decoding exponents; and a complete treatment of the regime. Finally, we note that the self-averaging property of has been established in Theorem 2: for and , the bulk branch coincides with the quenched free energy . Whether the same holds in region C (), the condensed phase , or for remains an interesting open problem.
Funding
This research received no external funding.
Data Availability Statement
The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.
Conflicts of Interest
The author declares no conflicts of interest.
Appendix A. Proof of Theorem 2
Throughout this proof, we define
We begin with part (i). To establish the upper bound, observe that, by Jensen’s inequality, , where the last equality is by the postulate . It follows that . As for the lower bound, fix any joint type with and , and let be its Y-marginal. For every output sequence of type , each codeword of joint type with contributes exactly to , so
and, since ,
As observed before, is binomial with trials and a probability of a single success of the exponential order of , so its mean is exponentially . By the Chernoff bound, for any ,
which is doubly exponentially small (see also Theorem 4.1 of [14] and, in particular, eqs. (D.16) and (D.17) in its proof therein, which establish the doubly exponential decay). A union bound over all joint types and all sequences gives
where is the event that for all types and all simultaneously. On , the bound (A3) gives
Taking the supremum over all with and , we obtain
Taking expectations of both sides of (A7) and using deterministically (since by Jensen):
Since , we get , which, together with
implies
thus completing the proof of part (i).
Proceeding to part (ii), we begin by proving that is an almost sure upper bound. By Markov’s inequality, for any ,
where we have used the fact that , which holds because for . Since , the Borel–Cantelli lemma yields for every eventually a.s., or, equivalently, a.s. For the compatible almost sure lower bound, note that we have already proved in (A7) that, whenever occurs, , which means that , or, equivalently, . Thus,
where the summability of is due to the fact that each term is bounded by a doubly exponentially small quantity. By the Borel–Cantelli lemma, eventually a.s., or, equivalently, a.s., which, together with the matching upper bound, yields a.s., completing the proof.
References
- Han, T.S.; Verdú, S. Approximation theory of output statistics. IEEE Trans. Inf. Theory 1993, 39, 752–772. [Google Scholar] [CrossRef] [Scilit]
- Cuff, P. Soft covering with high probability. In Proceedings of the 2016 IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain, 10–15 July 2016; pp. 2963–2967. [Google Scholar]
- Hayashi, M. General nonasymptotic and asymptotic formulas in channel resolvability and identification capacity and their application to wire-tap channel. IEEE Trans. Inf. Theory 2006, 52, 1562–1575. [Google Scholar] [CrossRef] [Scilit]
- Hou, J.; Kramer, G. Effective secrecy: Reliability, confusion and stealth. In Proceedings of the 2014 IEEE International Symposium on Information Theory, Honolulu, HI, USA, 29 June–4 July 2014; pp. 601–605. [Google Scholar]
- Mézard, M.; Montanari, A. Information, Physics, and Computation; Oxford University Press: New York, NY, USA, 2009. [Google Scholar]
- Merhav, N. Statistical physics and information theory. Found. Trends Commun. Inf. Theory 2010, 6, 1–212. [Google Scholar] [CrossRef] [Scilit]
- Yu, L.; Tan, V.Y.-F. Rényi resolvability and its applications to the wiretap channel. IEEE Trans. Inf. Theory 2019, 65, 1862–1897. [Google Scholar] [CrossRef] [Scilit]
- Arıkan, E. An inequality on guessing and its application to sequential decoding. IEEE Trans. Inf. Theory 1996, 42, 99–105. [Google Scholar] [CrossRef] [Scilit]
- Blahut, R.E. Hypothesis testing and information theory. IEEE Trans. Inf. Theory 1974, 20, 405–417. [Google Scholar] [CrossRef] [Scilit]
- Sourlas, N. Spin-glass models as error-correcting codes. Nature 1989, 339, 693–695. [Google Scholar] [CrossRef] [Scilit]
- Montanari, A. Turbo codes: The phase transition. Eur. Phys. J. B 2001, 18, 321–333. [Google Scholar] [CrossRef] [Scilit]
- Csiszár, I.; Körner, J. Information Theory: Coding Theorems for Discrete Memoryless Systems, 2nd ed.; Cambridge University Press: Cambridge, UK, 2011. [Google Scholar]
- Csiszár, I. The method of types. IEEE Trans. Inf. Theory 1998, 44, 2505–2523. [Google Scholar] [CrossRef] [Scilit]
- Merhav, N.; Weinberger, N. A toolbox for refined information-theoretic analyses with applications. Found. Trends Commun. Inf. Theory 2025, 22, 1–184. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.



