Skip to Content
EntropyEntropy
  • Feature Paper
  • Article
  • Open Access

8 June 2026

21 Pages

A Statistical-Physics Refinement of Soft Covering

The Viterbi Faculty of Electrical and Computer Engineering, Technion—Israel Institute of Technology, Technion City, Haifa 3200003, Israel
This article belongs to the Special Issue Coding Theory and Its Applications

Abstract

We study the channel output distribution induced by a random code of rate R from the perspective of statistical physics. The central object is the partition function Z n ( β | C ) = ∑ y n [ P Y n | C ( y n ) ] β , where y n is the channel output vector, C is the code, and β > 0 plays the role of inverse temperature. More precisely, our focus is on the associated annealed free energy, ψ ( β , R ) = lim n → ∞ 1 n log E [ Z n ( β | C ) ] , where the expectation is with respect to the randomness of C . This quantity encodes the full Rényi spectrum of the output distribution. The single-letter formula derived for the annealed free energy decomposes into two branches, which reflect a “competition” between two populations of codewords. One is the bulk branch, ψ b ( β , R ) , which is driven by typical codewords, and the other one is the sparse branch ψ s ( β , R ) , which is driven by atypical codewords, where the qualifiers ‘typical’ and ‘atypical’ are in a sense that will become apparent later. We analyze the phase structure of each branch separately and characterize their competition. Both branches are derived for all β > 0 . The phase boundary R 🟉 ( β ) , where the two branches are equal, is analyzed for β ≥ 1 , where it has an explicit closed-form expression. The phase diagram in the first quadrant of the ( β , R ) plane has four regions separated by three boundaries: R = I b ( β ) (bulk branch transition), R = R 🟉 ( β ) (bulk–sparse competition boundary), and R = I s ( β ) (sparse branch transition), all meeting at the point ( β , R ) = ( 1 , I ( X ; Y ) ) , where I ( X ; Y ) is the mutual information induced by the input type and the channel. Applications to guesswork, channel resolvability, and hypothesis testing are discussed, and all the results are illustrated with a numerical example of a Z-channel.

1. Introduction

Given a codebook C = { x n ( m ) } m = 1 M of size M = e n R and a discrete memoryless channel (DMC), { W ( y | x ) } , the induced channel output distribution is given by the mixture
P Y n ( y n ) = 1 M ∑ m = 1 M W n ( y n | x n ( m ) ) .
This object is central to channel resolvability [1] and the soft-covering problem (see, e.g., [2,3,4] and the references therein). Classical results, in this context, focus on a single threshold: if R > I ( X ; Y ) , I ( X ; Y ) is the mutual information induced by the input distribution and the channel W, then the distribution P Y n converges to the i.i.d. product law P Y ⊗ n in total variation as well as in some other metrics between probability distributions.
This description, however, captures average behavior only. It says nothing about the internal geometry of P Y n : how the probability mass is distributed across output sequences, how many codewords support a typical and an atypical output, or how the self-information, − 1 n log P Y n ( y n ) , fluctuates. These questions require a deeper journey beyond the Shannon entropy and mutual information.
In this work, we study P Y n through the function
Z n ( β ) = ∑ y n [ P Y n ( y n ) ] β = ∑ y n exp { − β log [ 1 / P Y n ( y n ) ] } , β > 0 ,
where the second representation is readily recognized as a partition function, with β > 0 playing the role of inverse temperature and log [ 1 / P Y n ( y n ) ] = − log P Y n ( y n ) being the energy function (a.k.a. the Hamiltonian) associated with every micro-state  y n . In other words, this is identified as the canonical partition function of a statistical-mechanics system whose micro-states are channel output sequences. Two special values bracket the range: Z n ( 1 ) = 1 (normalization, β = 1 ) and lim β → ∞ [ Z n ( β ) ] 1 / β = max y n P Y n ( y n ) (the mode, or the ground state in physics jargon). In general, 1 1 − β log Z n ( β ) is exactly the definition of H β ( P Y n ) , the Rényi entropy of order β ≠ 1 , pertaining to the output distribution P Y n , which is non-increasing in β . The associated free energy  ψ ( β , R ) encodes the exponential growth/decay rate of Z n ( β ) as a function of n. Since Z n ( β ) depends on the random codebook and, actually, should be denoted Z n ( β | C ) , we study the free energy behavior for the average code, which is identified with the annealed free energy,
ψ ( β , R ) = lim n → ∞ 1 n log E { Z n ( β | C ) } ,
where the existence of the limit will be evident from the exponentially tight analysis in the proof of Theorem 1 in the sequel. Here E { · } denotes the expectation operator with respect to (w.r.t.) the randomness of the code C . Its phase structure—the rich dependence on β and R—is the main subject of this work.
It should be emphasized that many earlier statistical-mechanics analyses of random coding were based on Derrida’s random energy model (REM) (see, e.g., [5,6] and the references therein). These are based on partition functions whose micro-states were the channel inputs (i.e., the codewords) for a fixed channel output sequence y n , that is,
Z n input ( β ) = ∑ m = 1 M [ W n ( y n | x n ( m ) ) ] β ,
for fixed y n , whereas, here, the micro-states are the channel outputs, as mentioned earlier. This difference is significant because, here, each energy term, − log P Y n ( y n ) , is itself a log-partition function over codewords. This two-level structure makes the problem considerably harder and richer than that of an ordinary REM.
The main contributions of this work are as follows. We derive the annealed free energy and analyze the phase structure of its two branches (Theorems 1 and 3). The free energy decomposes into a bulk branch and a sparse branch, reflecting a competition between two populations of codewords. The bulk branch is driven by typical codewords and the sparse branch is driven by atypical codewords, where the meanings of the qualifiers ‘typical’ and ‘atypical’ will become apparent in the sequel. Both branches are derived for all β > 0 and analyzed separately. The bulk branch, ψ b ( β , R ) (Section 3), has a single phase transition at a critical rate R = I b ( β ) —the mutual information of the bulk optimizer (the unconstrained optimizer of the bulk branch)—where a rate constraint changes from inactive to active. A similar comment applies to the sparse branch, ψ s ( β , R ) , whose phase boundary is R = I s ( β ) . The total annealed free energy ψ ( β , R ) has a phase boundary R 🟉 ( β ) given by an explicit closed-form formula (Theorem 3(b), Section 3). Together, the three boundaries divide the ( β , R ) quadrant { R ≥ 0 , β ≥ 1 } into four regions, denoted A, B, C, and D. By construction, ψ ( β , R ) ≥ ψ b ( β , R ) always, with equality in regions A and B and strict inequality in regions C and D, where the sparse branch dominates. For the sparse branch, a fully explicit closed-form formula, in terms of channel transition probabilities and β alone, holds whenever the rate is below I s ( β ) (see Equation (29)). All the results are illustrated in a numerical example in Section 4.
A few words are in order with regard to earlier related work and the differences relative to the present work. The model (1) is the canonical object of channel resolvability and the soft-covering problem. Han and Verdú [1] established the resolvability threshold R = I ( X ; Y ) under total variation, ∥ P Y n − P Y ⊗ n   ∥ TV → 0 for R > I ( X ; Y ) , and showed the threshold is tight. Hayashi [3] derived exponential convergence rates under total variation and normalized relative entropy, a.k.a. the Kullback–Leibler (KL) divergence. Hou and Kramer [4] extended the analysis to the Rényi divergence D α ( P Y n ∥ P Y ⊗ n ) , showing the critical rate remains I ( X ; Y ) for all α ∈ ( 0 , ∞ ) but the exponents differ. Yu and Tan [7] characterized the Rényi resolvability—the minimum rate R required for the Rényi divergence D α ( P Y n | C ∥ P Y ⊗ n ) to vanish asymptotically. They showed that, for α ≤ 1 , the threshold remains I ( X ; Y ) , while, for α > 1 , it is strictly larger than I ( X ; Y ) and depends on α .
Our work is complementary to [7]: rather than characterizing the threshold rate at which a divergence to P Y ⊗ n vanishes, we study the exact value of the annealed free energy at every rate R and inverse temperature β , revealing a two-branch phase diagram with three distinct phase boundaries. We analyze the below-threshold regime with the same precision as the above-threshold regime and expose qualitative phenomena—condensation, sparse-branch dominance, and the annealed/quenched gap—invisible to divergence-based analyses. In particular, our bulk branch boundary, R = I b ( β ) , for β > 1 is precisely the Rényi resolvability threshold of [7], now seen as one of three phase boundaries in a richer thermodynamic landscape.
In general, all the earlier results, in this context, share a common feature: they measure how close P Y n is to a target distribution P Y ⊗ n via some metric, and they all identify R = I ( X ; Y ) as the single relevant threshold. The present paper asks a fundamentally different and complementary question: not how close P Y n is to a target but what is the internal geometry of P Y n itself?
Our partition function Z n ( β ) encodes the Rényi structure of P Y n without reference to any external target distribution and reveals phenomena invisible to total variation or KL divergence. It turns out that the classical threshold, R = I ( X ; Y ) , is only the beginning of the story: the ( β , R ) quadrant { β ≥ 1 , R ≥ 0 } is divided into four distinct regions by three phase boundaries, all passing through the point ( β , R ) = ( 1 , I ( X ; Y ) ) . Concretely:
  • Even for R > I ( X ; Y ) (where ∥ P Y n − P Y ⊗ n ∥ TV   → 0 ), the annealed free energy reveals a condensed phase: when R < I b ( β ) for some β > 1 , the bulk branch of ψ has an active rate constraint, indicating that probability mass concentrates on outputs supported by sub-exponentially few codewords—output condensation that is invisible to total variation distance.
  • The annealed free energy has a phase boundary R = R 🟉 ( β ) where the sparse branch takes over from the bulk branch, signaling that the atypical codewords dominate the ensemble average.
  • Since log Z n ( β | C ) = ( 1 − β ) H β ( P Y n | C ) exactly for every fixed codebook (by the definition of Rényi entropy), the bulk branch ψ b ( β , R ) encodes the Rényi entropy rate H β ( P Y n | C ) / n of the output mixture for a typical codebook (for β ≥ 1 and R > R 🟉 ( β ) ; this is proved in Theorem 2; see also Appendix A). The full annealed free energy thus encodes the complete Rényi-order profile of the output distribution, of which classical soft covering ( β → 1 ) is a single cross-section.
In summary, this work does not contradict soft-covering results but refines and extends them by revealing the fine structure of P Y n that lies beneath the single threshold R = I ( X ; Y ) . These structural findings have direct operational consequences, which we now describe.
1.
Guesswork and Rényi entropy. Arikan [8] showed that guesswork moments E [ G ( Y n ) s ] grow as e n s H 1 / ( 1 + s ) ( P Y n ) . The annealed free energy encodes the Rényi entropy rate of the output mixture, extending Arıkan’s analysis to the case where the source is itself a random-coding mixture. The phase structure of ψ ( β , R ) produces regime changes in guessing complexity that are absent in the i.i.d. case.
2.
Hypothesis testing. The Chernoff exponent [9] for testing P Y n | C against P Y ⊗ n equals ξ ( R ) = max 0 ≤ β ≤ 1 [ ( 1 − β ) log | Y | − ψ ( β , R ) ] , directly connecting our free energy to the optimal test. With P Y being uniform, ξ ( R ) = 0 for R ≥ I ( X ; Y ) , meaning the two distributions are not exponentially distinguishable at and above the soft-covering threshold. The phase structure of ψ ( β , R ) produces qualitative regime changes in ξ ( R ) as R varies, and it connects to the Rényi resolvability results of Yu and Tan [7]; see Section 5.
3.
Statistical mechanics of codes. Sourlas [10] established the connection between linear codes and spin-glass models. Montanari [11] analyzed the phase transition in turbo codes. Mézard and Montanari [5] (Chapters 5 and 6) and Merhav [6] provided a comprehensive treatment of random coding via statistical physics, with the partition function summing over codewords (inputs) for a fixed received sequence. The present work complements [6] by placing the partition function over outputs: the resulting two-level hierarchical model has a distinct phase structure governed by the code rate R rather than by signal-to-noise ratio.
The outline of the remainder of this article is as follows. Section 2 defines the model and establishes the notation conventions. Section 3 derives the annealed free energy and analyzes the phase structure of its bulk and sparse branches; the phase boundary R * ( β ) has an explicit closed-form formula for β ≥ 1 . Section 4 presents the phase diagram (four regions, β ≥ 1 ). Section 5 discusses applications and the implications of our results. Finally, Section 6 summarizes the paper and provides an outlook.

2. Model, Definitions and Notation

Throughout the paper, n denotes the block length. An n-vector over alphabet X is displayed as x n = ( x 1 , x 2 , … , x n ) ∈ X n , X being the single-letter finite alphabet. Similarly, y n ∈ Y n , where the single-letter alphabet Y is also finite. The empirical distribution (type) of x n is denoted P ^ x n ( a ) = 1 n # { i : x i = a } for all a ∈ X . A similar definition applies to joint types of pairs of sequences, ( x n , y n ) . A DMC W is a stochastic matrix W : X → Y , and, for sequences x n and y n , we write W n ( y n | x n ) = ∏ i = 1 n W ( y i | x i ) . Throughout the sequel, all logarithms are natural (base e). Accordingly, information measures are given in nats. We use the standard exponential-equivalence notation f ( n ) ≐ g ( n ) to mean lim n → ∞ 1 n log f ( n ) g ( n ) = 0 ; i.e., f and g have the same exponential growth/decay rate. The notation of information measures is as follows. D ( P ∥ Q ) is the Kullback–Leibler (KL) divergence,
D ( P ∥ Q ) = ∑ x ∈ X P ( x ) log P ( x ) Q ( x ) .
Throughout the paper, Q X Y denotes a joint distribution on X   ×   Y with X-marginal given by Q X = P X , P X being a fixed input distribution on the finite input alphabet X . When Q X Y needs to appear as a subscript in an information measure, we abbreviate Q X Y by Q to avoid cumbersome notation. Thus, H Q ( Y ) , H Q ( Y | X ) , and I Q ( X ; Y ) stand for H Q X Y ( Y ) , H Q X Y ( Y | X ) , and I Q X Y ( X ; Y ) , respectively, which are the entropy of Y, the conditional entropy of Y given X, and the mutual information between X and Y, respectively, all induced by Q X Y . On the other hand, when Q X Y appears as an argument of a certain functional, the full notation is used. The notation I ( X ; Y ) (without subscript) refers to the mutual information induced by the fixed single-letter channel input distribution P X and the channel transition probability matrix W, i.e., I ( X ; Y ) = I P X W ( X ; Y ) . The KL divergence between a conditional distribution Q Y | X and W, with weighting by P X , is defined by
D ( Q Y | X ∥ W | P X ) = ∑ x ∈ X P X ( x ) ∑ y ∈ Y Q Y | X ( y | x ) log Q Y | X ( y | x ) W ( y | x ) .
A random codebook of rate R > 0 and block length n is a collection C = { x n ( m ) } m = 1 M and M = ⌊ e n R ⌋ , where the codewords x n ( m ) are drawn independently and uniformly at random from the type class,
T ( P X ) = x n ∈ X n : P ^ x n ( a ) = P X ( a ) ∀ a ∈ X ,
the set of all n-sequences with empirical distribution exactly P X . Given codebook C , the induced output distribution is
P Y n | C ( y n ) = 1 M ∑ m = 1 M W n ( y n | x n ( m ) ) .
For β > 0 , the partition function is defined as
Z n ( β | C ) = ∑ y n [ P Y n | C ( y n ) ] β .
The annealed free energy is
ψ ( β , R ) = lim n → ∞ 1 n log E C Z n ( β | C ) ,
where the limit will be evident in the subsequent analysis. For a joint type Q X Y on X × Y with marginal Q X = P X , define:
ℓ ( Q X Y ) = H Q ( Y | X ) + D ( Q Y | X ∥ W | P X ) .
Note that, for ( x n , y n ) of joint type Q X Y , W n ( y n | x n ) = e − n ℓ ( Q X Y ) . We shall often use the identity
I Q ( X ; Y ) + ℓ ( Q X Y ) = H Q ( Y ) + D ( Q Y | X ∥ W | P X ) .
Fix an output sequence y n of type Q Y . The type-class enumerator N ( Q X Y | y n ) counts how many codewords have a joint type (joint empirical distribution) Q X Y with y n :
N ( Q X Y | y n ) = # m : ( x n ( m ) , y n ) ∈ T ( Q X Y ) ,
where T ( Q X Y ) is the joint type class associated with the joint distribution Q X Y .
Using the method of types [12,13], it is readily observed that, given y n , each randomly selected codeword, x n ( m ) , satisfies ( x n ( m ) , y n ) ∈ T ( Q X Y ) with probability
| T ( Q X | Y | y n ) | | T ( P X ) | ≐ e n H Q ( X | Y ) e n H Q ( X ) = e − n I Q ( X ; Y ) ,
where T ( Q X | Y | y n ) = { x n : ( x n , y n ) ∈ T ( Q X Y ) } is the conditional type class, and where we have used the fact that Q X = P X . Since codewords are drawn independently at random from T ( P X ) , it is clear that N ( Q X Y | y n ) is a binomial random variable with M trials and a probability of single success of the exponential order of e − n I Q ( X ; Y ) .

3. Annealed Free Energy and Its Phase Structure

3.1. Statistical-Mechanics Perspective

Before presenting the formulas, we pause to explain the statistical-mechanics framework, why the qualifiers “annealed” and “quenched” are used, and why the analysis goes beyond a simple saddle-point calculation.

3.1.1. Annealed vs. Quenched: Origin of the Terminology

The terms come from metallurgy and were adopted by the statistical-mechanics community to describe two ways of averaging over quenched (i.e., frozen) disorder. In a disordered system—such as a spin glass—the disorder (e.g., the random coupling constants) is fixed for each physical realization but is drawn from some distribution. Annealing is the physical process of heating a material and cooling it slowly so that the disorder equilibrates with the thermal fluctuations; mathematically, this corresponds to averaging the partition function before taking its logarithm as if the disorder itself participates in the thermal equilibrium:
ψ ann ( β , R ) = lim n → ∞ 1 n log E C Z n ( β | C ) .
Quenching is the opposite process: the disorder is frozen rapidly and does not equilibrate. The free energy of a typical frozen realization is then
ψ que ( β , R ) = lim n → ∞ 1 n E C log Z n ( β | C ) ,
i.e., the logarithm is taken before averaging. In our setting, the “disorder” is the random codebook C ; once drawn, it is fixed and does not change with β . The annealed free energy ψ ( β , R ) defined in (10) is ψ ann ( β , R ) . By Jensen’s inequality (since log is concave), ψ ann ( β , R ) ≥ ψ que ( β , R ) always. The main subject of this paper is ψ ann ( β , R ) (Theorems 1 and 3 below); whether and when ψ ann = ψ que is addressed in Theorem 2.

3.1.2. The Annealed Free Energy and Its Information-Theoretic Meaning

The partition function Z n ( β | C ) = ∑ y n [ P Y n | C ( y n ) ] β is defined in (9). Its exponential growth rate ( 1 / n ) log Z n encodes how the output probability mass is distributed: large β emphasizes the most probable outputs (low energy, high P Y n | C ), while β → 0 weights all outputs equally. The special value β = 1 gives Z n ( 1 | C ) = 1 (normalization), and, as shown below, ( 1 / n ) log Z n ( β | C ) is proportional to the Rényi entropy H β ( P Y n | C ) of the output mixture (defined in the sequel). The annealed free energy ψ ( β , R ) = lim n 1 n log E [ Z n ( β | C ) ] thus encodes the Rényi entropy profile of the output distribution averaged over codebooks. Its exact single-letter formula—derived in Theorem 1—reveals a competition between two populations of codewords, called the bulk branch  ψ b ( β , R ) and the sparse branch  ψ s ( β , R ) , to be defined precisely in the sequel.

3.1.3. Why This Is More than a Saddle-Point Calculation

A naive approach would replace P Y n | C ( y n ) by its expectation E [ P Y n | C ( y n ) ] and compute Z n deterministically. This is correct only if Z n concentrates around its mean at the exponential scale, i.e., if ( 1 / n ) log Z n ≈ ( 1 / n ) log E [ Z n ] with high probability. When this self-averaging property holds, the annealed and quenched free energies coincide and a saddle-point over the output type Q Y gives the answer. However, self-averaging can fail: the annealed average E [ Z n ( β | C ) ] may be dominated by a sub-exponential fraction of codebooks that are far from typical, specifically those containing a codeword whose joint type Q X Y with some output y n satisfies I Q ( X ; Y ) > R (a “sparse-type” codeword, in the terminology of the sequel). These atypical codebooks contribute negligibly in probability but dominate the expectation of Z n , producing an annealed free energy strictly larger than the quenched one. Identifying when this happens—and characterizing both the typical behavior ψ b and the atypical behavior ψ s as a function of ( β , R ) —requires tracking two competing saddle points simultaneously rather than a single one. This competition produces the two-branch structure of ψ ( β , R ) and the three phase boundaries (to be derived in Theorem 3).

3.1.4. Phase Transitions and the Two-Level Hierarchical Structure

The phase boundaries of ψ ( β , R ) —to be characterized explicitly in Theorem 3 and illustrated in Section 4—signal qualitative changes in which configurations dominate Z n . This is qualitatively different from earlier statistical-mechanics analyses of random codes (see [5,6]), which placed the partition function over codewords (inputs) for a fixed output y n : Z n input ( β ) = ∑ m = 1 M [ W n ( y n | x n ( m ) ) ] β , as in (4). In that case, the energies − log W n ( y n | x n ( m ) ) are approximately i.i.d. across codewords (analogous to Derrida’s random energy model), and the analysis reduces to a single saddle point. In the present work, the energy of each output micro-state y n is − log P Y n | C ( y n ) , which is itself a log-partition function over codewords—a two-level hierarchical structure. This makes the energies neither independent nor identically distributed across output sequences y n and is the source of the richer two-branch phase diagram with three distinct boundaries.

3.2. Main Result: Formula for the Annealed Free Energy

In this section, we derive a single-letter expression for the annealed free energy and investigate its phase structure in the ( β , R ) plane. The first main result is the following.
Theorem 1
(Annealed Free Energy, β > 0 ). For all β > 0 :
ψ ( β , R ) = max ψ b ( β , R ) , ψ s ( β , R ) ,
with
ψ b ( β , R ) = sup Q X Y : Q X = P X I Q ( X ; Y ) ≤ R H Q ( Y ) − β [ I Q ( X ; Y ) + ℓ ( Q X Y ) ] ,
ψ s ( β , R ) = R ( 1 − β ) + sup Q X Y : Q X = P X I Q ( X ; Y ) > R { H Q ( Y | X ) − β ℓ ( Q X Y ) } .
For convenience, we define the functional
F ( Q X Y ) : = H Q ( Y | X ) − β ℓ ( Q X Y ) ,
which is the common building block of both branches. Using the identity H Q ( Y ) = I Q ( X ; Y ) + H Q ( Y | X ) , we may present the two branches of the annealed free energy as
ψ b ( β , R ) = sup Q X Y : Q X = P X I Q ( X ; Y ) ≤ R ( 1 − β ) I Q ( X ; Y ) + F ( Q X Y )
ψ s ( β , R ) = R ( 1 − β ) + sup Q X Y : Q X = P X I Q ( X ; Y ) > R F ( Q X Y ) .
Proof of Theorem 1. 
We begin with the trivial identity,
E [ Z n ( β ) ] = E ∑ y n [ P Y n | C ( y n ) ] β = ∑ y n E P Y n | C ( y n ) β .
We next focus on E [ P Y n | C ( y n ) β ] for a fixed y n . Define S ( y n ) = ∑ m = 1 M W n ( y n | x n ( m ) ) so that P Y n | C ( y n ) = S ( y n ) / M . Since M = e n R is deterministic,
E P Y n | C ( y n ) β = M − β E [ S ( y n ) ] β = e − n β R E [ S ( y n ) ] β .
The problem reduces to computing E [ S ( y n ) β ] . For each y n ,
E { [ S ( y n ) ] β } = E ∑ Q X Y : Q X = P X Q Y = P ^ y n N ( Q X Y | y n ) · e − n ℓ ( Q X Y ) β ≐ E max Q X Y : Q X = P X Q Y = P ^ y n N ( Q X Y | y n ) · e − n ℓ ( Q X Y ) β = E max Q X Y : Q X = P X Q Y = P ^ y n [ N ( Q X Y | y n ) ] β · e − n β ℓ ( Q X Y ) ≐ E ∑ Q X Y : Q X = P X Q Y = P ^ y n [ N ( Q X Y | y n ) ] β · e − n β ℓ ( Q X Y ) = ∑ Q X Y : Q X = P X Q Y = P ^ y n E { [ N ( Q X Y | y n ) ] β } · e − n β ℓ ( Q X Y ) ≐ max Q X Y : Q X = P X Q Y = P ^ y n E { [ N ( Q X Y | y n ) ] β } · e − n β ℓ ( Q X Y ) ,
where the dotted equalities hold since the sum contains at most poly ( n ) non-negative terms. Now, the evaluation of E { N ( Q X Y | y n ) ] β } requires the following lemma, which is Theorem 4.2 of [14], and the proof is therein.
Lemma 1 
(Moments of Binomial Enumerator). Let N ∼ Binomial ( e n A , e − n B ) with A , B > 0 and β > 0 . Then,
E { N β } ≐ e n β ( A − B ) A > B e − n ( B − A ) A < B
The case A = B corresponds to a non-exponential behavior of E { N β } , as can be seen by observing the limiting behavior of both cases. (As a side remark, when N is Binomial ( e n A , λ e − n A ) for some constant λ > 0 , then, in the limit of n → ∞ , N becomes a Poissonian random variable with parameter λ ; that is, P r { N = k } → λ k e − λ k ! for every non-negative integer k, and, therefore, the asymptotic β th moments are constants).
Lemma 1 is now used with the assignments A = R and B = I Q ( X ; Y ) . Accordingly, one must distinguish between types { Q X Y } for which R > I Q ( X ; Y ) as opposed to those with R < I Q ( X ; Y ) . Now,
E { [ S ( y n ) ] β } ≐ max Q X Y : Q X = P X Q Y = P ^ y n E { [ N ( Q X Y | y n ) ] β } · e − n β ℓ ( Q X Y ) = max { max Q X Y : Q X = P X , Q Y = P ^ y n I Q ( X ; Y ) ≤ R E { [ N ( Q X Y | y n ) ] β } · e − n β ℓ ( Q X Y ) , max Q X Y : Q X = P X , Q Y = P ^ y n I Q ( X ; Y ) ≥ R E { [ N ( Q X Y | y n ) ] β } · e − n β ℓ ( Q X Y ) } ≐ max { max Q X Y : Q X = P X , Q Y = P ^ y n I Q ( X ; Y ) ≤ R e n β [ R − I Q ( X ; Y ) ] · e − n β ℓ ( Q X Y ) , max Q X Y : Q X = P X , Q Y = P ^ y n I Q ( X ; Y ) ≥ R e − n [ I Q ( X ; Y ) − R ] · e − n β ℓ ( Q X Y ) } .
Since the latter expression involves maximization over { Q X Y } when Q Y = P ^ y n is held fixed, it is clear that it depends on y n only via P ^ y n . The number of { y n } with P ^ y n = Q Y is of the exponential order of e n H Q ( Y ) . Multiplying the per- y n contributions by e n H Q ( Y ) and e − n β R yields the total contribution of type Q Y , and, finally, summing over all { Q Y } , which is exponentially equivalent to maximizing over all { Q Y } , yields the asserted expression of ψ ( β , R ) . The part pertaining to types with I Q ( X ; Y ) ≤ R is identified with ψ b ( β , R ) , and the one with I Q ( X ; Y ) > R is associated with ψ s ( β , R ) . □
For future reference, we need the following definition.
Definition 1
(Sparse-feasible and bulk-feasible types). For a given rate R > 0 , a joint distribution Q X Y with Q X = P X is called:
  • Sparse-feasible if I Q ( X ; Y ) > R ; i.e., it satisfies the constraint of the sparse branch optimization;
  • Bulk-feasible if I Q ( X ; Y ) ≤ R ; i.e., it satisfies the constraint of the bulk branch optimization. Every Q X Y is either sparse-feasible or bulk-feasible (or both if I Q ( X ; Y ) = R ).

3.3. The Sparse Branch: Formula and Phase Structure

The sparse branch of the annealed free energy,
ψ s ( β , R ) = R ( 1 − β ) + sup Q X Y : Q X = P X I Q ( X ; Y ) > R F ( Q X Y ) ,
captures the contribution of sparse-type codewords: those whose joint type Q X Y with the output y n satisfies I Q ( X ; Y ) > R . For fixed y n , such codewords are rarely encountered in a typical codebook of the ensemble.
Definition 2
(The annealed optimizer Q β s and its mutual information I s ( β ) ). For each β > 0 , define
Q β s ( y | x ) = [ W ( y | x ) ] β ∑ y ′ ∈ Y [ W ( y ′ | x ) ] β x ∈ X .
It can be readily shown that Q β s maximizes F ( Q X Y ) = H Q ( Y | X ) − β ℓ ( Q X Y ) over all Q X Y with Q X = P X . Accordingly, define
I s ( β ) : = I Q β s ( X ; Y ) .
The curve R = I s ( β ) marks the point where Q β s crosses from sparse-feasible ( I Q β s ( X ; Y ) > R , so Q β s optimizes the sparse branch) to bulk-feasible ( I Q β s ( X ; Y ) ≤ R , so Q β s optimizes the bulk branch).
  • Closed-form formula. By Lemma 1, when I Q ( X ; Y ) > R , the β th moment of the enumerator satisfies E { [ N ( Q X Y | y n ) ] β } ≐ e n [ R − I Q ( X ; Y ) ] regardless of β . The resulting annealed calculation gives
    ψ s ( β , R ) = R ( 1 − β ) + C ( β ) whenever R < I s ( β ) ,
    where
    C ( β ) = ∑ x ∈ X P X ( x ) log ∑ y ∈ Y [ W ( y | x ) ] β .
    The formula ceases to hold when R > I s ( β ) as Q β s is then bulk-feasible ( I Q β s ( X ; Y ) ≤ R ) and hence does not comply with the rate constraint of the sparse branch.
  • Phase structure. The sparse branch has its own phase transition at R = I s ( β ) :
    −
    For R < I s ( β ) : ψ s ( β , R ) = R ( 1 − β ) + C ( β ) , the closed-form linear formula. The sparse branch is maximized by Q β s .
    −
    For R > I s ( β ) : Q β s is not sparse-feasible; the closed form R ( 1 − β ) + C ( β ) does not apply and the sparse branch must be evaluated directly from (26).
  • Operational meaning. The sparse branch governs the behavior of the ensemble average E [ Z n ( β | C ) ] when the latter is inflated by atypical codebooks. Specifically, when ψ s ( β , R ) > ψ b ( β , R ) (which occurs for sufficiently small R, as characterized by the phase boundary R 🟉 ( β ) derived in Theorem 3 below), the annealed free energy satisfies ψ ( β , R ) = ψ s ( β , R ) , meaning that E [ Z n ( β | C ) ] seems to be dominated by a sub-exponential fraction of codebooks with atypical codewords (those containing a sparse-type codeword with I Q ( X ; Y ) > R ). A typical codebook has 1 n log Z n ( β | C ) ≈ ψ b ( β , R ) .

3.4. The Bulk Branch: Operational Meaning

We now explain why the bulk branch ψ b ( β , R ) deserves a separate study within the analysis of ψ ( β , R ) . As said before, the bulk branch, ψ b ( β , R ) , is the component of ψ ( β , R ) driven by typical codewords—those whose joint type with the output satisfies I Q ( X ; Y ) ≤ R , i.e., codewords that are plausible given the code rate. A typical random codebook contains no sparse-type codewords (those with I Q ( X ; Y ) > R ) with high probability, so, for a typical fixed codebook, Z n ( β | C ) ≐ e n ψ b ( β , R ) at the exponential scale. The annealed free energy ψ ( β , R ) = max { ψ b ( β , R ) , ψ s ( β , R ) } can exceed ψ b ( β , R ) (when the sparse branch dominates) because it is inflated by the rare codebooks that happen to contain a sparse-type codeword. In the language of statistical physics, ψ b ( β , R ) is the quenched free energy, which is the free energy of a typical random code, while ψ ( β , R ) is the annealed free energy (the free energy of the disorder-averaged partition function). To support this observation, we now state the self-averaging property of Z n ( β | C ) for β ≥ 1 and R ≥ R 🟉 ( β ) . The proof appears in the Appendix A.
Theorem 2
(Self-averaging and quenched free energy). For β ≥ 1 and R > R 🟉 ( β ) :
(i) 
lim n → ∞ 1 n E log Z n ( β | C ) = ψ b ( β , R ) .
(ii) 
1 n log Z n ( β | C ) → a . s . ψ b ( β , R ) .
The condition R > R 🟉 ( β ) is precisely the regime (regions A and B in the phase diagram) where the bulk branch dominates the annealed free energy: ψ ( β , R ) = ψ b ( β , R ) (Theorem 3). This is needed for the proof of the a.s. upper bound: Markov’s inequality on E [ Z n ] gives ( 1 / n ) log Z n ≤ ψ ( β , R ) eventually a.s., and this equals ψ b ( β , R ) only when R > R 🟉 ( β ) . The lower bound of the proof holds under the weaker condition R > I b ( β ) , so region C ( I b ( β ) < R < R 🟉 ( β ) ), where a typical codebook still lives in the bulk phase but the annealed average is inflated by rare sparse-type codebooks, remains open. Whether the result also extends to the condensed phase R ≤ I b ( β ) or to β < 1 likewise remains open.
By the definition of Rényi entropy, H β ( Y ) = 1 1 − β log ∑ y P ( y ) β , so log Z n ( β | C ) = log ∑ y n P Y n | C ( y n ) β = ( 1 − β ) H β ( P Y n | C ) exactly for every codebook. Since 1 n log Z n ( β | C ) → ψ b ( β , R ) a.s. (Theorem 2), the bulk branch encodes the Rényi entropy rate of the output mixture at order β :
ψ b ( β , R ) ≐ 1 − β n H β ( P Y n | C )
for a typical fixed codebook.

3.5. The Bulk Branch: Formula and Phase Structure

Definition 3
(The bulk optimizer Q β b and its mutual information I b ( β ) ). For each β > 0 , define
Q β b = arg max Q X Y : Q X = P X ( 1 − β ) I Q ( X ; Y ) + F ( Q X Y ) ,
the unconstrained maximizer of the bulk objective, and set
I b ( β ) : = I Q β b ( X ; Y ) .
Obviously, for R ≥ I b ( β ) ,
ψ b ( β , R ) = ψ b , u ( β ) : = sup Q X Y : Q X = P X ( 1 − β ) I Q ( X ; Y ) + F ( Q X Y ) .
The curve R = I b ( β ) is therefore the bulk phase boundary: for R ≥ I b ( β ) the rate constraint I Q ( X ; Y ) ≤ R is inactive (bulk phase), while for R < I b ( β ) it is active (condensed phase).
The next lemma establishes an inequality relation between I b ( β ) and I s ( β ) .
Lemma 2
(Ordering of the two optimizers). For β ≥ 1 :
I b ( β ) ≤ I s ( β ) ,
with strict inequality for β > 1 . At β = 1 , I b ( 1 ) = I s ( 1 ) = I ( X ; Y ) .
Proof. 
By the optimality of Q β b in the bulk objective:
F ( Q β b ) + ( 1 − β ) I b ( β ) ≥ F ( Q β s ) + ( 1 − β ) I s ( β ) = C ( β ) + ( 1 − β ) I s ( β ) ,
which, for β > 1 , is equivalent to
I s ( β ) − I b ( β ) ≥ C ( β ) − F ( Q β b ) β − 1
where the right-hand side is clearly non-negative since the denominator β − 1 is positive, and the numerator is non-negative as C ( β ) is the global maximum of F ( Q X Y ) , which cannot be smaller than F ( Q β b ) . Therefore, I s ( β ) − I b ( β ) is non-negative as well. □

3.6. Phase Structure of the Annealed Free Energy

The annealed free energy formula (Theorem 1) holds for all β > 0 . However, the analysis of the phase boundary between the bulk and sparse branches is restricted here to the range β ≥ 1 for two reasons. The first reason is motivational: In the statistical-mechanics language, β ≥ 1 is the low-temperature regime where Z n ( β ) is closely related to Rényi entropies of order at least one. This range captures the operationally most relevant quantities: β = 1 (Shannon entropy and resolvability threshold), β = 2 (collision entropy and birthday attacks), β → ∞ (minimum entropy and − log max y n P Y n ( y n ) ), and general β > 1 (guessing moments E [ G ( Y n ) s ] ≐ e n s H 1 / ( 1 + s ) [8]. The second reason is that the range 0 < β < 1 does not appear to lend itself to closed-form analysis, the main issue being the lack of guarantee concerning the uniqueness of the solution R to the equation ψ b ( β , R ) = ψ s ( β , R ) , which is the phase boundary of the total annealed free energy, as will be defined next. In particular, for 0 < β < 1 , the branches ψ b ( β , R ) and ψ s ( β , R ) may or may not cross, and existence and uniqueness of a crossing are not established in general.
Definition 4
(Annealed Phase Boundary). For β ≥ 1 , the annealed phase boundary R 🟉 ( β ) is the unique rate at which the bulk and sparse branches are equal:
ψ b β , R 🟉 ( β ) = ψ s β , R 🟉 ( β ) .
Existence and uniqueness of R 🟉 ( β ) ∈ ( I b ( β ) , I s ( β ) ) are established in Theorem 3(b) below, together with an explicit formula.
In order to keep track of the various phase boundaries and the quantities associated with them so far, the following table summarizes those ingredients (Table 1).
Table 1. Summary of the two optimizers.
The following theorem provides a characterization of the phase structure of the annealed free energy.
Theorem 3.
For any DMC W, any P X , and any β ≥ 1 , R > 0 :
(a) 
For R ≤ I s ( β ) , Q β s is sparse-feasible and
ψ s ( β , R ) = R ( 1 − β ) + C ( β ) .
(b) 
(Annealed phase boundary, β ≥ 1 .) Let
R 🟉 ( β ) : = C ( β ) − ψ b , u ( β ) β − 1 , β > 1 , R 🟉 ( 1 ) : = I ( X ; Y ) .
For all β ≥ 1 , R 🟉 ( β ) ∈ [ I b ( β ) , I s ( β ) ] , which is the unique rate at which ψ s ( β , R ) = ψ b ( β , R ) , and
ψ ( β , R ) = ψ s ( β , R ) R < R 🟉 ( β ) , ψ b ( β , R ) R ≥ R 🟉 ( β ) .
The curve R = R 🟉 ( β ) is the annealed phase boundary.
Proof. 
Part (a) was already shown earlier (but we include it here as part of the theorem for completeness since it is used in part (b)): Observe that, for R ≤ I s ( β ) , we have I Q β s ( X ; Y ) ≥ R , so it is sparse-feasible. Since Q β s achieves the global maximum, F ( Q β s ) = sup I Q ( X ; Y ) > R F ( Q X Y ) = C ( β ) , so ψ s ( β , R ) = R ( 1 − β ) + C ( β ) .
As for part (b), observe that, for R ∈ ( I b ( β ) , I s ( β ) ) , both simplified formulas hold simultaneously: ψ b ( β , R ) = ψ b , u ( β ) since R > I b ( β ) ) and ψ s ( β , R ) = R ( 1 − β ) + C ( β ) (by part (a) since R < I s ( β ) ). Setting ψ b ( β , R ) = ψ s ( β , R ) and solving gives immediately
R 🟉 ( β ) = C ( β ) − ψ b , u ( β ) β − 1 .
Substituting ψ b , u ( β ) = F ( Q β b ) + ( 1 − β ) I b ( β ) , we end up with:
R 🟉 ( β ) = I b ( β ) + C ( β ) − F ( Q β b ) β − 1 .
Since β − 1 > 0 and C ( β ) ≥ F ( Q β b ) , the second term is non-negative, giving R 🟉 ( β ) ≥ I b ( β ) , with strict inequality since F ( Q β b ) < C ( β ) strictly for β > 1 (Lemma 2).
For the upper bound, recall from the proof of Lemma 2 that
C ( β ) − F ( Q β b ) ≤ ( β − 1 ) I s ( β ) − I b ( β ) ,
which gives
C ( β ) − F ( Q β b ) β − 1 ≤ I s ( β ) − I b ( β ) .
Substituting into (43): R 🟉 ( β ) ≤ I s ( β ) , again with strict inequality for β > 1 . Hence R 🟉 ( β ) ∈ ( I b ( β ) , I s ( β ) ) , confirming the formula is valid.
To establish the uniqueness of the solution R 🟉 ( β ) to the equation ψ b ( β , R ) = ψ s ( β , R ) , it remains to rule out any additional solutions outside the interval [ I b ( β ) , I s ( β ) ] . For R ≤ I b ( β ) , Q β s is still sparse-feasible (since R ≤ I b ( β ) < I s ( β ) ), so part (a) gives ψ s ( β , R ) = R ( 1 − β ) + C ( β ) . The bulk constraint is active, so ψ b ( β , R ) ≤ ψ b , u ( β ) . Hence,
Δ ( β , R ) : = ψ s ( β , R ) − ψ b ( β , R ) = R ( 1 − β ) + C ( β ) − ψ b ( β , R ) ≥ R ( 1 − β ) + C ( β ) − ψ b , u ( β ) .
The right side is positive at R = I b ( β ) (as shown above) and increasing as R decreases (since 1 − β < 0 ). Hence Δ ( β , R ) > 0 for all R ≤ I b ( β ) . For R ≥ I s ( β ) , consider the following. Since I b ( β ) ≤ I s ( β ) ≤ R , the bulk constraint is inactive and ψ b ( β , R ) = ψ b , u ( β ) . Since R ≥ I s ( β ) > R 🟉 ( β ) and ψ b , u ( β ) = R 🟉 ( β ) ( 1 − β ) + C ( β ) (from the formula of R 🟉 ( β ) ), we have:
ψ b ( β , R ) = ψ b , u ( β ) = R 🟉 ( β ) ( 1 − β ) + C ( β ) > R ( 1 − β ) + C ( β ) ≥ ψ s ( β , R ) ,
where the strict inequality uses R > R 🟉 ( β ) and 1 − β < 0 , and the last inequality is because ψ s ( β , R ) = R ( 1 − β ) + sup { Q : Q X = P X , I Q ( X ; Y ) > R } F ( Q X Y ) ≤ R ( 1 − β ) + C ( β ) . Hence Δ ( β , R ) < 0 for all R ≥ I s ( β ) . □

4. A Numerical Example

In this section, we provide a numerical example and illustrate the behavior of the two branches of the annealed free energy as well as the phase boundary curves.
All phase diagrams are computed for an example of a Z-channel. The details are as follows. The input and output alphabets are X = Y = { 0 , 1 } , and the crossover probability is p = 0.45 , i.e., W ( 0 | 0 ) = 1 , W ( 1 | 0 ) = 0 , W ( 0 | 1 ) = 0.45 , and W ( 1 | 1 ) = 0.55 . The input type is given by P X = ( 0.5 , 0.5 ) . Accordingly, the resulting channel mutual information is I ( X ; Y ) = 0.2441 .
Figure 1 plots all the phase boundaries together in the relevant quadrant of the ( β , R ) plane.
Figure 1. Phase boundaries for the Z-channel, β ≥ 1 . Red: R = I b ( β ) , the bulk branch boundary (bulk rate constraint activates). Blue: R = R 🟉 ( β ) , the annealed boundary where bulk = sparse (closed-form formula, Theorem 3(b)). Green: R = I s ( β ) , the sparse branch boundary. All three meet at ( β , R ) = ( 1 , I ( X ; Y ) ) (filled dot). Four regions (Remark 4): A ( R > I s ( β ) ): ψ b unconstrained, ψ s constrained (sparse closed form does not apply); bulk dominates; B ( R 🟉 ( β ) < R < I s ( β ) ): bulk branch dominates ( ψ ( β , R ) = ψ b ( β , R ) ); C ( I b ( β ) < R < R 🟉 ( β ) ): sparse branch dominates ( ψ ( β , R ) = ψ s ( β , R ) ); D ( R < I b ( β ) ): sparse branch dominates ( ψ ( β , R ) = ψ s ( β , R ) ), bulk condensed.
Three phase boundary curves are visible in Figure 1:
  • R = I b ( β ) (red): the bulk branch boundary, where the unconstrained bulk optimizer crosses the rate constraint. At β = 1 : I b ( 1 ) = I ( X ; Y ) .
  • R = R 🟉 ( β ) (blue): the annealed boundary where the bulk and sparse branches are equal, given by the closed-form Formula (40). At β = 1 : R 🟉 ( 1 ) = I ( X ; Y ) . Decreasing in β , staying above I b ( β ) .
  • R = I s ( β ) (green): the sparse branch boundary, where Q β s (which maximizes F ( Q X Y ) = H Q ( Y | X ) − β ℓ ( Q X Y ) ) has I Q β s ( X ; Y ) = R . For R > I s ( β ) the sparse closed form ceases to hold. At β = 1 : I s ( 1 ) = I ( X ; Y ) .
  • All three curves meet at the point ( β , R ) = ( 1 , I ( X ; Y ) ) , confirming that the soft-covering threshold is the unique point where all regions collapse to a single point.
For β ≥ 1 , the three phase boundaries divide the quadrant { β ≥ 1 , R > 0 } into four regions:
  • A ( R > I s ( β ) , above all curves): ψ b ( β , R ) is unconstrained ( I b ( β ) < R ). ψ s ( β , R ) is constrained: since I s ( β ) < R , the sparse optimizer Q β s is bulk-feasible and the sparse closed-form expression R ( 1 − β ) + C ( β ) does not apply. Bulk dominates: ψ ( β , R ) = ψ b ( β , R ) (since R 🟉 ( β ) < I s ( β ) ).
  • B ( R 🟉 ( β ) < R < I s ( β ) ): Bulk branch dominates: ψ ( β , R ) = ψ b ( β , R ) . The bulk rate constraint is inactive ( R > I b ( β ) ); no condensation.
  • C ( I b ( β ) < R < R 🟉 ( β ) ): Sparse branch dominates: ψ ( β , R ) = ψ s ( β , R ) = R ( 1 − β ) + C ( β ) (closed-form linear formula). Rare-event codewords inflate the ensemble average; a typical codebook is in the bulk phase.
  • D ( R < I b ( β ) , below the red curve): Sparse branch dominates: ψ ( β , R ) = ψ s ( β , R ) = R ( 1 − β ) + C ( β ) . The bulk branch is condensed (rate constraint active; bulk optimizer constrained to I Q ( X ; Y ) = R ), but the annealed free energy is inflated by rare sparse-type codebooks.
  • For the very same z-channel example, Figure 2, Figure 3 and Figure 4 display ψ b ( β , R ) and ψ s ( β , R ) as functions of β ≥ 1 for three representative values of R (one below I ( X ; Y ) , another one equal to I ( X ; Y ) , and yet another one above I ( X ; Y ) ) and contrasts them with the i.i.d. reference free energy
    Figure 2. R 1 = 0.12 < I ( X ; Y ) . The sparse branch ψ s ( β , R 1 ) lies above the bulk branch for all β ≥ 1 , so the annealed free energy is dominated by the sparse branch; both lie well above the i.i.d. reference, reflecting ensemble inflation by rare codebooks.
    Figure 3. R 2 = I ( X ; Y ) = 0.2441 nats (the soft-covering threshold). Both branches and the i.i.d. reference all vanish at β = 1 (since Z n ( 1 ) = 1 ) and diverge differently for β > 1 .
    Figure 4. R 3 = 0.35 > I ( X ; Y ) . The bulk branch dominates throughout and tracks the i.i.d. reference closely, confirming that, above the soft-covering threshold, the output mixture approximates P Y ⊗ n in the Rényi sense. In all three figures, ψ iid ( β ) ≤ ψ ( β , R ) , with strict inequality below I ( X ; Y ) : the random-coding mixture has strictly higher Rényi entropy than the i.i.d. output distribution at every order β ≥ 1 .
ψ iid ( β ) : = 1 n log ∑ y n [ P Y ( y n ) ] β = log ∑ y ∈ Y [ P Y ( y ) ] β ,
where P Y is the output distribution induced by P X and W. This is the free energy of the product distribution P Y ⊗ n , i.e., the distribution to which P Y n converges for R > I ( X ; Y ) based on well-known soft-covering results. Clearly, ψ iid ( β ) is a purely smooth function with no phase transitions whatsoever.
A central message of this paper is that the classical soft-covering threshold R = I ( X ; Y ) is far from the end of the story, and, as this numerical example illustrates, we refine and deepen those results in this work. As mentioned in the Introduction, classical results, like those of [1,3,7], tell us that ∥ P Y n − P Y ⊗ n ∥ TV → 0 (and in other metrics) if and only if R > I ( X ; Y ) and provide exponential rates for this convergence. Our statistical-physics analysis reveals that, even well above this threshold, P Y n retains non-trivial internal structure that is invisible to total variation. Figure 2, Figure 3 and Figure 4 make this visible. The three plots of ψ b ( β , R ) , ψ s ( β , R ) , and the i.i.d. reference ψ iid ( β ) = log ∑ y [ P Y ( y ) ] β against β for R 1 < I ( X ; Y ) = R 2 < R 3 . At the classical threshold R 2 = I ( X ; Y ) (second diagram), all three quantities vanish together at β = 1 as required but diverge for β > 1 : the free energy ψ ( β , R 2 ) lies strictly above ψ iid ( β ) for all β > 1 , meaning that P Y n has strictly higher Rényi entropy than P Y ⊗ n at every order. Above the threshold at R 3 > I ( X ; Y ) (third diagram), the bulk branch closely tracks ψ iid ( β ) , confirming the soft-covering intuition that the output mixture approaches the i.i.d. law, but the gap, while small, remains nonzero at finite β , and quantifying it requires ψ b ( β , R 3 ) . Below the threshold at R 1 < I ( X ; Y ) (first diagram), the gap ψ ( β , R 1 ) − ψ iid ( β ) is large and grows with β : the sparse branch dominates the ensemble average, and the random-coding output is far from i.i.d. in the Rényi sense even though TV-distance arguments do not apply here at all.
Specifically, ψ b ( β , R ) is the Rényi entropy rate of P Y n at order β for a typical codebook, and its phase transition at R = I b ( β ) implies a qualitative change in how probability mass is distributed over output sequences. In the bulk phase ( R > I b ( β ) ), a typical output sequence is supported by exponentially many codewords. The output distribution is diffuse and “spread out” in a way that is consistent with the intuition behind soft covering. In the condensed phase ( R < I b ( β ) ), a typical output appears to be supported by only sub-exponentially few codewords, meaning the mass of P Y n concentrates on a sparse set of sequences even though the TV distance to P Y ⊗ n may already be negligible. This condensation is visible in Figure 2: the first diagram ( R 1 < I ( X ; Y ) ) shows ψ s ( β , R 1 ) > ψ b ( β , R 1 ) for all β ≥ 1 , so the ensemble average is dominated by the rare sparse-branch codebooks, while the third diagram ( R 3 > I ( X ; Y ) ) shows ψ b ( β , R 3 ) > ψ s ( β , R 3 ) , confirming typical-codebook dominance throughout.
The gap ψ ( β , R ) − ψ iid ( β ) ≥ 0 visible in all three diagrams has a direct operational meaning: it measures how much the random-coding output mixture exceeds the i.i.d. distribution in Rényi entropy at order β . Classical soft-covering results establish that this gap vanishes in total variation for R > I ( X ; Y ) . The figures show that, in Rényi entropy, the gap is nonzero for every β > 1 and every finite R, decaying to zero only as R → ∞ .

5. Applications and Implications

In this section, we summarize several applications and implications of the results in this work. Further investigation of these applications is deferred to future work.
1.
Refined output statistics. Resolvability shows ∥ P Y n − ( P Y ) n ∥ TV → 0 for R > I ( X ; Y ) . Our phase diagram reveals that, even in this regime, P Y n may be in the condensed phase (below I b ( β ) for large β ), where mass appears to concentrate on outputs supported by a sub-exponential number of codewords. This is invisible to the total variation distance and other distances.
2.
Guesswork. E [ G ( Y n ) s ] ≐ e n s H 1 / ( 1 + s ) ( P Y n ) [8]. Since H β ( P Y n | C ) = 1 ( 1 − β ) n log Z n ( β | C ) and 1 n log Z n ( β | C ) ≐ ψ ( β , R ) (the full annealed free energy, including both bulk and sparse branches), the guessing exponent at β = 1 / ( 1 + s ) is s n ( 1 − β ) log Z n ( β | C ) ≐ s 1 − β ψ ( β , R ) . The phase structure of ψ ( β , R ) —both its bulk phase transition at R = I b ( β ) and its bulk–sparse crossover at R = R 🟉 ( β ) —therefore produces regime changes in guessing complexity absent in the i.i.d. case.
3.
Security. In the condensed phase, even when P Y n ≈ ( P Y ) n in total variation, only a sub-exponential number of codewords may generate a typical output. A computationally powerful adversary can identify the message by examining which codewords are consistent with the observed output. Semantic security may therefore require operating strictly in the bulk phase for all β relevant to the adversary’s test.
4.
Hypothesis testing and the Chernoff exponent. A natural question is how well one can distinguish the random-code output mixture P Y n | C from the i.i.d. reference P Y ⊗ n . With P Y = Unif ( Y ) (achieved by choosing P X appropriately), this is a direct test of whether soft covering has succeeded. The annealed Chernoff exponent for this binary hypothesis test equals
ξ ( R ) = max 0 ≤ β ≤ 1 ( 1 − β ) log | Y | − ψ ( β , R ) ,
where ψ ( β , R ) is the annealed free energy of Theorem 1. Since both branches of ψ ( β , R ) are fully characterized by our results, (49) is completely determined. The optimizer β 🟉 ∈ [ 0 , 1 ] satisfies ψ ′ ( β 🟉 , R ) = − log | Y | , so the phase structure of ψ ( β , R ) directly governs the behavior of ξ ( R ) : the bulk/sparse phase boundaries produce regime changes in the Chernoff exponent as R varies.
With P Y = Unif , two clean consequences follow. First, ξ ( R ) = 0 if and only if R ≥ I ( X ; Y ) : the two distributions are not exponentially distinguishable for R ≥ I ( X ; Y ) ; i.e., no test can achieve an exponentially small total error probability exactly at the soft-covering threshold. (Sub-exponential error rates, e.g., polynomial in n, are not excluded). Second, (49) is the Legendre–Fenchel transform of ψ ( β , R ) with respect to β , evaluated at log | Y | ; it therefore equals 1 n log ∑ y n P Y n | C ( y n ) β 🟉 | Y | − n ( 1 − β 🟉 ) , which is precisely the Rényi divergence D α ( P Y n | C ∥ P Y ⊗ n ) at order α = β 🟉 ∈ [ 0 , 1 ] . This connects (49) directly to the Rényi resolvability results of Yu and Tan [7], who showed that D α ( P Y n | C ∥ P Y ⊗ n ) → 0 if and only if R > I ( X ; Y ) for α ≤ 1 , while, for α > 1 , the threshold is R = I b ( α ) —the bulk condensation boundary of our phase diagram. Our analysis provides the exact rate of divergence at every R and α and reveals how the two-branch phase structure of ψ ( β , R ) determines qualitative changes in this rate.

6. Conclusions

We developed a statistical-mechanics framework for the output distribution of random codes, centered on the annealed free energy ψ ( β , R ) of the partition function Z n ( β ) = ∑ y n [ P Y n ( y n ) ] β and its two-branch phase structure. This dichotomy between the two branches has concrete operational consequences. For guesswork, the phase structure of ψ ( β , R ) —both the bulk phase transition at R = I b ( β ) and the bulk–sparse crossover at R = R 🟉 ( β ) —translates directly (via Arıkan’s formula [8]) into regime changes in the exponential growth rate of guessing moments E [ G ( Y n ) s ] . Similar comments apply to hypothesis testing and perhaps other application areas. The annealed free energy adds yet another layer: its phase boundary R = R 🟉 ( β ) separates the regime where the ensemble average is dominated by rare atypical codebooks (for example, the sparse branch dominates in Figure 2) from the regime where it reflects the behavior of a typical codebook (the bulk branch dominates in Figure 4). The gap ψ ( β , R ) − ψ b ( β , R ) > 0 in the sparse-dominant regime quantifies the extent to which ensemble averages can be misleading—a warning that is particularly relevant when analyzing the security of specific random codes rather than random coding in the mean.
Future research directions include: a rigorous derivation of exponentially tight error exponents for the hypothesis test of Application 4 via the type-class enumeration method of [14], together with a phase-diagram analysis of the Neyman–Pearson exponent tradeoff in the ( R , τ ) plane; a full treatment of the guesswork regime changes produced by the bulk/sparse phase boundaries; quantitative semantic security bounds in terms of I b ( β ) and R 🟉 ( β ) ; extension to Markov and mixed sources; connections to mismatched decoding exponents; and a complete treatment of the β < 1 regime. Finally, we note that the self-averaging property of Z n ( β | C ) has been established in Theorem 2: for β ≥ 1 and R > R 🟉 ( β ) , the bulk branch ψ b ( β , R ) coincides with the quenched free energy lim n → ∞ 1 n E { log Z n ( β | C ) } . Whether the same holds in region C ( I b ( β ) < R < R 🟉 ( β ) ), the condensed phase R ≤ I b ( β ) , or for β < 1 remains an interesting open problem.

Funding

This research received no external funding.

Data Availability Statement

The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.

Conflicts of Interest

The author declares no conflicts of interest.

Appendix A. Proof of Theorem 2

Throughout this proof, we define
U n : = 1 n log Z n ( β | C ) .
We begin with part (i). To establish the upper bound, observe that, by Jensen’s inequality, E { U n } ≤ 1 n log E [ Z n ( β | C ) ] → ψ ( β , R ) = ψ b ( β , R ) , where the last equality is by the postulate R ≥ R 🟉 ( β ) . It follows that lim sup n → ∞ E { U n } ≤ ψ b ( β , R ) . As for the lower bound, fix any joint type Q X Y with Q X = P X and I Q ( X ; Y ) ≤ R , and let Q Y be its Y-marginal. For every output sequence y n of type Q Y , each codeword of joint type Q X Y with y n contributes exactly e − n ℓ ( Q X Y ) to ∑ m W n ( y n | x n ( m ) ) , so
P Y n | C ( y n ) ≥ N ( Q X Y | y n ) M · e − n ℓ ( Q X Y ) ,
and, since β ≥ 1 ,
Z n ( β | C ) ≥ ∑ y n : P ^ y n = Q Y N ( Q X Y | y n ) M β e − n β ℓ ( Q X Y ) .
As observed before, N ( Q X Y | y n ) is binomial with e n R trials and a probability of a single success of the exponential order of e − n I Q ( X ; Y ) , so its mean μ n is exponentially e n ( R − I Q ( X ; Y ) ) → ∞ . By the Chernoff bound, for any δ ∈ ( 0 , 1 ) ,
Pr N ( Q X Y | y n ) ≤ ( 1 − δ ) μ n ≤ exp − δ 2 μ n 2 = exp − δ 2 2 e n [ R − I Q ( X ; Y ) ] ,
which is doubly exponentially small (see also Theorem 4.1 of [14] and, in particular, eqs. (D.16) and (D.17) in its proof therein, which establish the doubly exponential decay). A union bound over all joint types { Q X Y } and all | T ( Q Y ) | ≤ e n H Q ( Y ) sequences y n gives
Pr { E n c } ≤ ( n + 1 ) | X | | Y | · e n H Q ( Y ) · exp − δ 2 2 e n [ R − I Q ( X ; Y ) ] ⟶ 0 ,
where E n is the event that N ( Q X Y | y n ) ≥ ( 1 − δ ) μ n for all types Q X Y and all y n ∈ T ( Q Y ) simultaneously. On E n , the bound (A3) gives
Z n ( β | C ) ≥ | T ( Q Y ) | · ( 1 − δ ) μ n M β e − n β ℓ ( Q X Y ) ≐ ( 1 − δ ) β · exp n H Q ( Y ) − β I Q ( X ; Y ) + ℓ ( Q X Y ) .
Taking the supremum over all Q X Y with Q X = P X and I Q ( X ; Y ) ≤ R , we obtain
U n ≥ ψ b ( β , R ) + log ( 1 − δ ) n on E n .
Taking expectations of both sides of (A7) and using U n ≥ ( 1 − β ) log | Y | deterministically (since Z n ( β ) ≥ | Y | n ( 1 − β ) by Jensen):
E { U n } ≥ ψ b ( β , R ) + log ( 1 − δ ) n Pr { E n } + ( 1 − β ) log | Y | · Pr { E n c } .
Since Pr [ E n ] → 1 , we get lim inf n → ∞ E { U n } ≥ ψ b ( β , R ) , which, together with
lim sup n → ∞ E { U n } ≤ ψ b ( β , R ) ,
implies
lim n → ∞ E { U n } = ψ b ( β , R ) ,
thus completing the proof of part (i).
Proceeding to part (ii), we begin by proving that ψ b ( β , R ) is an almost sure upper bound. By Markov’s inequality, for any ε > 0 ,
Pr U n ≥ ψ b ( β , R ) + ε ≤ E [ Z n ( β | C ) ] e n ( ψ b ( β , R ) + ε ) ≐ e − n ε ,
where we have used the fact that 1 n log E [ Z n ( β | C ) ] → ψ b ( β , R ) , which holds because ψ ( β , R ) = ψ b ( β , R ) for R > R 🟉 ( β ) . Since ∑ n e − n ε < ∞ , the Borel–Cantelli lemma yields U n ≤ ψ b ( β , R ) + ε for every ε > 0 eventually a.s., or, equivalently, lim sup n → ∞ U n ≤ ψ b ( β , R ) a.s. For the compatible almost sure lower bound, note that we have already proved in (A7) that, whenever E n occurs, U n ≥ ψ b ( β , R ) + log ( 1 − δ ) n , which means that E n ⊆ F n : = { U n ≥ ψ b ( β , R ) + log ( 1 − δ ) n } , or, equivalently, F n c ⊆ E n c . Thus,
∑ n = 1 ∞ P r U n < ψ b ( β , R ) + log ( 1 − δ ) n = ∑ n = 1 ∞ Pr { F n c } ≤ ∑ n = 1 ∞ Pr { E n c } < ∞
where the summability of Pr { E n c } is due to the fact that each term is bounded by a doubly exponentially small quantity. By the Borel–Cantelli lemma, U n − log ( 1 − δ ) n ≥ ψ b ( β , R ) eventually a.s., or, equivalently, lim inf n → ∞ U n ≥ ψ b ( β , R ) a.s., which, together with the matching upper bound, yields lim n → ∞ U n = ψ b ( β , R ) a.s., completing the proof.

References

  1. Han, T.S.; Verdú, S. Approximation theory of output statistics. IEEE Trans. Inf. Theory 1993, 39, 752–772. [Google Scholar] [CrossRef] [Scilit]
  2. Cuff, P. Soft covering with high probability. In Proceedings of the 2016 IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain, 10–15 July 2016; pp. 2963–2967. [Google Scholar]
  3. Hayashi, M. General nonasymptotic and asymptotic formulas in channel resolvability and identification capacity and their application to wire-tap channel. IEEE Trans. Inf. Theory 2006, 52, 1562–1575. [Google Scholar] [CrossRef] [Scilit]
  4. Hou, J.; Kramer, G. Effective secrecy: Reliability, confusion and stealth. In Proceedings of the 2014 IEEE International Symposium on Information Theory, Honolulu, HI, USA, 29 June–4 July 2014; pp. 601–605. [Google Scholar]
  5. Mézard, M.; Montanari, A. Information, Physics, and Computation; Oxford University Press: New York, NY, USA, 2009. [Google Scholar]
  6. Merhav, N. Statistical physics and information theory. Found. Trends Commun. Inf. Theory 2010, 6, 1–212. [Google Scholar] [CrossRef] [Scilit]
  7. Yu, L.; Tan, V.Y.-F. Rényi resolvability and its applications to the wiretap channel. IEEE Trans. Inf. Theory 2019, 65, 1862–1897. [Google Scholar] [CrossRef] [Scilit]
  8. Arıkan, E. An inequality on guessing and its application to sequential decoding. IEEE Trans. Inf. Theory 1996, 42, 99–105. [Google Scholar] [CrossRef] [Scilit]
  9. Blahut, R.E. Hypothesis testing and information theory. IEEE Trans. Inf. Theory 1974, 20, 405–417. [Google Scholar] [CrossRef] [Scilit]
  10. Sourlas, N. Spin-glass models as error-correcting codes. Nature 1989, 339, 693–695. [Google Scholar] [CrossRef] [Scilit]
  11. Montanari, A. Turbo codes: The phase transition. Eur. Phys. J. B 2001, 18, 321–333. [Google Scholar] [CrossRef] [Scilit]
  12. Csiszár, I.; Körner, J. Information Theory: Coding Theorems for Discrete Memoryless Systems, 2nd ed.; Cambridge University Press: Cambridge, UK, 2011. [Google Scholar]
  13. Csiszár, I. The method of types. IEEE Trans. Inf. Theory 1998, 44, 2505–2523. [Google Scholar] [CrossRef] [Scilit]
  14. Merhav, N.; Weinberger, N. A toolbox for refined information-theoretic analyses with applications. Found. Trends Commun. Inf. Theory 2025, 22, 1–184. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.