Entropy Regularization in the Mean-Field Free-Energy Machine for Combinatorial Optimization

Abstract

The free-energy machine (FEM) relaxes discrete combinatorial optimization by minimizing expected energy plus a temperature-dependent negative-entropy penalty over product distributions. This contribution investigates the computational and mathematical role of that entropy term. At high temperature, entropy drives the optimizer toward the uniform product law; explicit relative-entropy and total-variation bounds quantify this effect, while an influence condition and a restricted Hessian characterize uniqueness, contraction, and the first possible loss of stability. As temperature decreases, the entropy coefficient vanishes. Uniform and variational convergence then connect free energy to the original discrete objective, obtaining an inverse-temperature excess-energy bound, ground-state concentration, and entropy selection among degenerate optima. Extraction bounds, path tracking, perturbation stability, and factorwise complexity further distinguish the effects of entropy, annealing, and replica selection. A batched annealed FEM algorithm is evaluated on exactly enumerable Ising, MaxCut, and Max-3SAT instances; the results reveal the predicted entropy-to-energy transition and show problem-dependent benefits of annealing and replication. For empirical-loss energies, a PAC-Bayes inequality additionally calibrates inverse temperature to sample size. The analysis characterizes entropy-dominated, instability, and energy-dominated regimes without attributing global guarantees to the nonconvex optimizer.

1. Introduction

Combinatorial optimization is concerned with selecting a configuration from a finite but typically enormous set so as to minimize a prescribed cost. It underlies graph partitioning, scheduling, routing, satisfiability, coding, resource allocation, and various inference tasks. A convenient generic formulation can be stated by

$$E_{★}=\underset{\sigma \in \mathcal{A}}{\min }E\left(\sigma \right),\mathcal{A}={\displaystyle \prod _{i=1}^d}{\mathcal{A}}_i,\left|{\mathcal{A}}_i\right|=K_i,$$

(1)

where d is the number of variables, $K_i$ is the cardinality of ${\mathcal{A}}_i$, and E may contain pairwise, higher-order, or global interactions. Even when every variable is binary and E is quadratic, this framework includes broad classes of NP-hard problems [1,2,3,4,5]. Exact enumeration scales as $\left|\mathcal{A}\right|={\prod }_i{K}_i$, while worst-case polynomial-time exact algorithms are not expected for unrestricted instances. This gap between a compact algebraic description and an exponentially large configuration space has motivated approximation algorithms, continuous relaxations, local search, tabu and extremal dynamics, and problem-specific heuristics [6,7,8,9].

A specially prominent approach comes from statistical physics. The cost E is regarded as an energy and is embedded into the Gibbs distribution

$${\pi }_{\beta }\left(\sigma \right)={\displaystyle \frac{e^{-\beta E\left(\sigma \right)}}{{\sum }_{\tau \in \mathcal{A}}{e}^{-\beta E\left(\tau \right)}}}.$$

At high temperature, corresponding to small inverse temperature $\beta =1/T$, the law is diffuse and explores many configurations. At low temperature it concentrates on ground states. Simulated annealing implements this principle with a stochastic Markov chain and has a classical asymptotic convergence theory under logarithmically slow cooling, although the schedules required by the worst-case theory are often impractical [10,11]. The rationale of why low-temperature landscapes contain many competing pure states and metastable basins is delineated by spin-glass theory [12,13,14]. The foregoing ideas further inspired analog, quantum, optical, and digital Ising machines, as well as noisy mean-field solvers and simulated-bifurcation [15,16,17].

Mean-field annealing replaces sampling from the Gibbs law by deterministic optimization over independent marginal distributions. Its early versions inferred self-consistency equations for binary networks and updated their mean magnetizations along a temperature schedule [18,19]. This approach has two superiorities: (1) expectations of sparse local energies can often be evaluated exactly and differentiated without Monte Carlo noise; (2) all marginal probabilities can be updated in parallel. Its limitation is equally appreciable: a product distribution cannot reproduce correlated Gibbs mixtures, and the resulting free-energy landscape need not be convex. Classical variational treatments of graphical models describe this trade-off by means of tractable free-energy approximations [20,21,22]. TAP (Thouless-Anderson-Palmer), Bethe, belief-propagation, and structured variational corrections improve fidelity in selected frameworks but typically sacrifice the simplicity and universality of naive mean field [19,23,24].

The free-energy machine (FEM) of Shen et al. [25] transforms this classical principle into a general GPU-oriented solver. It allows multivalued variables and higher-order interactions, represents each marginal by softmax logits, calculates the expected energy with automatic differentiation or explicit formulas, and runs many independently initialized replicas in a batch. FEM has been demonstrated on MaxCut, balanced graph partitioning, and MaxSAT instances with up to millions of variables. Its central objective for a product law $P_p={⨂}_{i=1}^d{p}_i$ is denoted by

$$F_{\beta }\left(p\right)=U\left(p\right)-{\displaystyle \frac{1}{\beta }}H\left(P_p\right)=\sum _{\sigma }{P}_p\left(\sigma \right)E\left(\sigma \right)+{\displaystyle \frac{1}{\beta }}\sum _{\sigma }{P}_p\left(\sigma \right)\log P_p\left(\sigma \right),$$

(2)

where $\beta >0$ is inverse temperature, $U\left(p\right)={\mathbb{E}}_{P_p}E$, and $H\left(P_p\right)={\sum }_{i}H\left(p_i\right)$. In implementation, each marginal is parameterized by logits and a batch of independently initialized product laws is optimized when $\beta$ increases. The negative-entropy is pivotal to the purposed behavior: at the beginning it favors broad marginals and suppresses premature symmetry breaking; later the expected energy dominates as the temperature $1/\beta$ decreases.

Shen et al. [25], moreover, elucidated this annealing process as a qualitative three-stage evolution from an entropy-dominated nearly uniform regime, through a distributional transition, to an energy-dominated low-temperature regime. The aim of this contribution is not to introduce this phenomenological picture, but to determine which parts of it admit explicit finite-temperature bounds, stability criteria, and asymptotic guarantees.

Despite strong empirical results, several basic theoretical problems on (2) remain open at the level of a general finite-state energy. Is the uniform initialization an exact optimum or only an approximation at the starting temperature? When is the high-temperature solution unique, and at what point can distinct branches appear? Does global minimization at low temperature recover a ground state, at what rate, and how does the product constraint behave when the ground state is degenerate? What can be guaranteed after coordinatewise mode rounding or batched sampling? How sensitive is the solution to approximate energies and gradients? Finally, which statements concern the variational objective itself, and which require an optimizer to find its global minimum? Answering these questions is crucial to separate the role of entropy from the separate roles of annealing, numerical optimization, and replica parallelism.

The analysis belongs to a broader field of work on variational free energies. Mean-field accuracy has been studied in terms of nonlinear large deviations, high-temperature conditions, and log-concavity [26,27,28]. The geometry and convergence of coordinate-ascent and gradient-based variational algorithms are analyzed for Ising, community-detection, and Bayesian models [29,30,31,32]. Generalized Gibbs posteriors and their variational approximations connect free-energy minimization to statistical learning [33,34,35,36]. Entropic regularization is employed to control the geometry and expressiveness of variational approximations [37]. The analysis deliberately builds on several classical tools, including mean-field annealing arguments, Dobrushin-type influence bounds, variational convergence, reverse-KL variational representations, and PAC-Bayes change-of-measure inequalities. This work lies in specializing and combining these ingredients for the finite-state product free-energy objective in (2), and in deriving a set of quantitative conclusions that clarify the role of entropy, optimization error, annealing, and replica selection in FEM.

From an information-theoretic perspective, the entropy term in (2) is closely connected to the maximum-entropy principle, which selects a probability law by maximizing entropy subject to available constraints [38]. Such ideas are applied to incorporate prescribed ensemble-averaged observables into statistical-mechanical simulations while minimally perturbing a reference ensemble [39,40]. In the current setting, since

$$-H\left(P_p\right)=D_{\mathrm{KL}}(P_p\parallel u)-\log \left|\mathcal{A}\right|,$$

where u denotes the uniform law on $\mathcal{A}$, the FEM objective is equivalent, up to an additive constant, to   

$$U\left(p\right)+{\displaystyle \frac{1}{\beta }}{D}_{\mathrm{KL}}(P_p\parallel u).$$

This formulation suggests the more general regularization $U\left(p\right)+{\beta }^{-1}{D}_{\mathrm{KL}}(P_p\parallel Q)$ with a prescribed reference law Q, thereby associating the Shannon-entropy term to relative-entropy and cross-entropy representations. If the reference law is updated by virtue of information accumulated during optimization or sampling, the regularization becomes history dependent, conceptually related to adaptive-bias constructions such as metadynamics [41]. In this broader interpretation, $1/\beta$ manipulates the trade-off between energy minimization and information-theoretic proximity to the reference law. The analysis in this paper, however, remains restricted to the fixed uniform-reference case in (2).

With the above-mentioned distinction in mind, the FEM-specific contributions are summarized as follows.

• Starting from the classical reverse-KL representation of mean-field free energy, the FEM objective is elaborated quantitatively in the finite-state product setting. This gets explicit high-temperature bounds on proximity to the uniform law, an influence-based uniqueness and contraction criterion, and a restricted-Hessian condition for the first possible loss of local stability.
• A FEM-specific low-temperature analysis is obtained, deducing an explicit $\log \left|\mathcal{A}\right|/\beta$ excess-energy bound, ground-state concentration, entropy selection among degenerate optima, and guarantees for approximate minimizers, mode rounding, sampling, and replicas. Furthermore, robust one-flip optima are showcased to generate low-temperature stationary branches.
• Canonical annealing and stability ideas are specialized to the FEM dynamics to infer a quantitative path-tracking bound in the contractive regime, a perturbation-stability estimate for approximate energies, and exact factorwise per-step complexity. These results are then incorporated into a batched annealed FEM algorithm and evaluated on exactly enumerable Ising, MaxCut, and Max-3SAT instances.
• High-dimensional scaling regimes are derived and, for data-dependent energies, a PAC-Bayes oracle inequality is yielded with a statistically calibrated inverse temperature [42,43,44], where these results separate variational guarantees from assumptions about the nonconvex numerical optimizer.

The remainder of the paper is organized as follows. Section 2 develops the variational representation and the high- and low-temperature theory, including the emergence of low-temperature stationary points and all proofs adjacent to their results. Section 3 analyzes annealing, perturbation stability, and computational complexity. A concrete batched algorithm and controlled experiments with exact ground-state certificates are presented in Section 4. The high-dimensional phase diagram and the statistical-learning guarantees are manifested in Section 5. Section 6 specializes the general results to Ising/QUBO (quadratic unconstrained binary optimization), Potts MaxCut (maximum cut), and MaxSAT (maximum satisfiability), and summarizes practical implications. Finally, the limitations and research directions are concluded in Section 7.

2. Variational Foundations and Thermal Regimes

This section aims to develop the variational foundations of FEM and characterize its behavior across various temperature regimes. It first determines the Gibbs-projection interpretation and quantifies the mean-field approximation error. The entropy-dominated high-temperature regime is then analyzed in terms of convergence toward the uniform law, uniqueness and contraction under weak interactions, and the onset of local instability. Finally, the energy-dominated low-temperature regime is examined through zero-temperature convergence, solution extraction guarantees, and stationary branches associated with discrete local optima.

Set

$$M=\left|\mathcal{A}\right|={\displaystyle \prod _{i=1}^d}{K}_i,u\left(\sigma \right)=M^{-1},{\pi }_{\beta }\left(\sigma \right)=Z_{\beta }^{-1}{e}^{-\beta E\left(\sigma \right)},Z_{\beta }=\sum _{\sigma \in \mathcal{A}}{e}^{-\beta E\left(\sigma \right)}.$$

Let $\mathcal{M}=\{{⨂}_i{p}_i:p_i\in {\Delta }_{K_i}\}$ be the product family and

$$J_{\beta }\left(p\right)=\beta F_{\beta }\left(p\right)=\beta U\left(p\right)-H\left(P_p\right).$$

(3)

The rescaling removes the singularity of $F_{\beta }$ at $\beta =0$ without changing its minimizers for $\beta >0$.

Theorem 1

(Gibbs projection, existence, and interiority). For any $\beta >0$,

$$J_{\beta }\left(p\right)=D_{\mathrm{KL}}(P_p\parallel {\pi }_{\beta })-\log Z_{\beta }.$$

(4)

Consequently, FEM is the reverse-KL projection of ${\pi }_{\beta }$ onto $\mathcal{M}$. The minimum is attained. Every minimizer $p_{\beta }$ is interior: $p_{\beta ,i}\left(a\right)>0$ for each i and $a\in {\mathcal{A}}_i$.

Define the conditional mean energy

$$e_i(a;p_{-i})={\mathbb{E}}_{P_{p,-i}}\left[E(a,{\sigma }_{-i})\right].$$

(5)

Then for any global minimizer and any interior stationary point, one gets

$$p_i\left(a\right)={\displaystyle \frac{\exp \{-\beta e_i(a;p_{-i})\}}{{\sum }_{b\in {\mathcal{A}}_i}\exp \{-\beta e_i(b;p_{-i})\}}}.$$

(6)

Conversely, every solution of (6) is an interior stationary point of $F_{\beta }$.

Proof of Theorem 1.

Since $P_p$ is a product law,

$$D_{\mathrm{KL}}(P_p\parallel {\pi }_{\beta })=\sum _{\sigma }{P}_p\left(\sigma \right)\{\log P_p\left(\sigma \right)+\beta E\left(\sigma \right)+\log Z_{\beta }\}=J_{\beta }\left(p\right)+\log Z_{\beta },$$

which verifies (4). The product of finite-dimensional closed simplices is compact and $J_{\beta }$ is continuous under the convention $0\log 0=0$; thus, a minimizer exists.

Suppose a minimizer has $p_i\left(a\right)=0$. Move mass $\epsilon$ from a positive coordinate b to a, holding all other marginals fixed. The change in $\beta U$ is $O\left(\epsilon \right)$ since the energy is finite. The new entropy contribution to $-H$ contains $\epsilon \log \epsilon$, whose negative magnitude dominates every linear term as $\epsilon \downarrow 0$. The objective therefore decreases, which leads to a contradiction. Hence, each minimizer is interior.

Holding $p_{-i}$ fixed, the terms depending on $p_i$ are

$$\sum _a{p}_i\left(a\right)e_i(a;p_{-i})+{\beta }^{-1}\sum _a{p}_i\left(a\right)\log p_i\left(a\right).$$

The Lagrange multiplier for ${\sum }_a{p}_i\left(a\right)=1$ gives $e_i\left(a\right)+{\beta }^{-1}(\log p_i\left(a\right)+1)+{\lambda }_i=0$, which is equivalent to (6).

The same deduction also delineates all interior stationary points. Indeed, if p is an interior stationary point, then for each i, stationarity with respect to variations of $p_i$ within the simplex implies that there exists a Lagrange multiplier ${\lambda }_i$ satisfying

$$e_i(a;p_{-i})+{\beta }^{-1}\{\log p_i\left(a\right)+1\}+{\lambda }_i=0$$

for each $a\in A_i$. Solving these equations together with ${\sum }_a{p}_i\left(a\right)=1$ gives (6). Conversely, if p satisfies (6), then

$$e_i(a;p_{-i})+{\beta }^{-1}\{\log p_i\left(a\right)+1\}$$

is constant in a for each i. Hence the directional derivative of $F_{\beta }$ vanishes along every tangent direction of the product simplex, so p is an interior stationary point.    □

Theorem 1 clarifies exactly what role entropy plays. It is equivalent, up to the constant $-{\beta }^{-1}\log M$, to the penalty ${\beta }^{-1}{D}_{\mathrm{KL}}(P_p\parallel u)$. Hence, high temperature pulls the product law toward a uniform prior; it does not penalize statistical dependence, which has already been removed by the feasible set. This interpretation is expressed in terms of the standard information-geometric relation between entropy and relative entropy [45,46].

Corollary 1

(Logit gradient). Let

$$p_i\left(a\right)={\displaystyle \frac{e^{h_i\left(a\right)}}{{\sum }_b{e}^{h_i\left(b\right)}}}$$

and

$$g_i\left(a\right)=e_i(a;p_{-i})+{\beta }^{-1}\{\log p_i\left(a\right)+1\}.$$

Then

$${\displaystyle \frac{\partial F_{\beta }}{\partial h_i\left(a\right)}}=p_i\left(a\right)\left(g_i\left(a\right)-\sum _b{p}_i\left(b\right)g_i\left(b\right)\right),$$

(7)

which is the centered-gradient identity used in FEM. The logits have the gauge invariance $h_i\mapsto h_i+c_i\mathbf{1}$.

Proof of Corollary 1.

Holding $p_{-i}$ fixed, the terms of $F_{\beta }$ that depend on $p_i$ are

$$\sum _b{p}_i\left(b\right)e_i(b;p_{-i})+{\beta }^{-1}\sum _b{p}_i\left(b\right)\log p_i\left(b\right),$$

thus

$${\displaystyle \frac{\partial F_{\beta }}{\partial p_i\left(b\right)}}=e_i(b;p_{-i})+{\beta }^{-1}\{\log p_i\left(b\right)+1\}=g_i\left(b\right).$$

Furthermore, the softmax Jacobian satisfies

$${\displaystyle \frac{\partial p_i\left(b\right)}{\partial h_i\left(a\right)}}=p_i\left(b\right)\{\mathbf{1}(a=b)-p_i\left(a\right)\}.$$

Therefore, by means of the chain rule,

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial F_{\beta }}{\partial h_i\left(a\right)}}& =\sum _b{g}_i\left(b\right)p_i\left(b\right)\{\mathbf{1}(a=b)-p_i\left(a\right)\}\hfill \\ & =p_i\left(a\right)g_i\left(a\right)-p_i\left(a\right)\sum _b{p}_i\left(b\right)g_i\left(b\right),\hfill \end{array}$$

which is (7).    □

2.1. Mean-Field Approximation Error to the Gibbs Law

Define the reverse-KL approximation error by

$$A_{\beta }=\underset{P\in \mathcal{M}}{\min }{D}_{\mathrm{KL}}(P\parallel {\pi }_{\beta }).$$

(8)

The exact Gibbs free energy is $-{\beta }^{-1}\log Z_{\beta }$, whereas the optimal mean-field free energy is

$$F_{\beta }^{\mathrm{MF}}=-{\beta }^{-1}\log Z_{\beta }+{\beta }^{-1}{A}_{\beta }.$$

(9)

Proposition 1

(Two universal projection bounds). Let $R_E=osc\left(E\right)={\max }_{\sigma }E\left(\sigma \right)-{\min }_{\sigma }E\left(\sigma \right)$. Then

$$0\le A_{\beta }\le \min \left\{{\displaystyle \frac{{\beta }^2{R}_E^2}{8}},\log M\right\}.$$

(10)

Specifically, if $P_{p_{\beta }}$ is a global FEM minimizer (Throughout this paper, statements about a “global FEM minimizer” concern the mathematical minimum of (2)), then

$$d_{\mathrm{TV}}(P_{p_{\beta }},{\pi }_{\beta })\le \sqrt{A_{\beta }/2}\le {\displaystyle \frac{\beta R_E}{4}}$$

(11)

whenever the high-temperature bound is used.

Proof of Proposition 1.

Employing the uniform product law as a competitor,

$$A_{\beta }\le D_{\mathrm{KL}}(u\parallel {\pi }_{\beta })=\log {\mathbb{E}}_u\exp \{-\beta (E-{\mathbb{E}}_{u}E)\}\le {\beta }^2{R}_E^2/8,$$

where the last step follows from Hoeffding’s lemma. If ${\sigma }_{★}$ is a ground state, the point mass ${\delta }_{{\sigma }_{★}}$ is also a product law and

$$A_{\beta }\le D_{\mathrm{KL}}({\delta }_{{\sigma }_{★}}\parallel {\pi }_{\beta })=\log \sum _{\sigma }{e}^{-\beta (E\left(\sigma \right)-E_{★})}\le \log M.$$

Therefore, Equation (11) is obtained in light of Pinsker’s inequality and the minimizing identity $D_{\mathrm{KL}}(P_{p_{\beta }}\parallel {\pi }_{\beta })=A_{\beta }$.

If $p_{\beta }$ is a global FEM minimizer, then Theorem 1 and the definition of $A_{\beta }$ imply $D_{\mathrm{KL}}(P_{p_{\beta }}\parallel {\pi }_{\beta })=A_{\beta }$. Applying Pinsker’s inequality therefore obtains

$$d_{\mathrm{TV}}(P_{p_{\beta }},{\pi }_{\beta })\le \sqrt{{\displaystyle \frac{1}{2}}{D}_{\mathrm{KL}}(P_{p_{\beta }}\parallel {\pi }_{\beta })}=\sqrt{{\displaystyle \frac{A_{\beta }}{2}}}.$$

In light of the high-temperature bound $A_{\beta }\le {\beta }^2{R}_E^2/8$ from (10), one further gets

$$d_{\mathrm{TV}}(P_{p_{\beta }},{\pi }_{\beta })\le \sqrt{{\displaystyle \frac{A_{\beta }}{2}}}\le \sqrt{{\displaystyle \frac{{\beta }^2{R}_E^2}{16}}}={\displaystyle \frac{\beta R_E}{4}},$$

which confirms (11).    □

The first bound in (10) is useful only in a genuine high-temperature scaling. The second is temperature-uniform but does not say that the mean-field law reproduces correlations of ${\pi }_{\beta }$. Reverse KL is mode-seeking, especially when the Gibbs law is multimodal.

2.2. The Entropy-Dominated Regime

2.2.1. Uniform Limit and Its First Correction

Let $p_i^u\left(a\right)=K_i^{-1}$ and $P_u={⨂}_i{p}_i^u=u$.

Theorem 2

(Quantitative uniform regime). For any global minimizer $p_{\beta }$,

$$\begin{array}{cc}\hfill D_{\mathrm{KL}}(P_{p_{\beta }}\parallel u)& =\log M-H\left(P_{p_{\beta }}\right)\le \beta R_E,\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill d_{\mathrm{TV}}(P_{p_{\beta }},u)& \le \sqrt{\beta R_E/2},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{d}}{\displaystyle \sum _{i=1}^d}{d}_{\mathrm{TV}}(p_{\beta ,i},p_i^u)& \le \sqrt{{\displaystyle \frac{\beta R_E}{2d}}}.\hfill \end{array}$$

(14)

Hence, $p_{\beta }\to p^u$ as $\beta \downarrow 0$. Moreover, the locally unique solution emanating from $p^u$ has expansion

$$p_{\beta ,i}\left(a\right)={\displaystyle \frac{1}{K_i}}-{\displaystyle \frac{\beta }{K_i}}\left(e_i(a;p_{-i}^u)-U\left(p^u\right)\right)+O\left({\beta }^2\right).$$

(15)

Therefore, exact uniformity at positive β requires all conditional mean energies $e_i(a;p_{-i}^u)$ to be constant in a.

Proof of Theorem 2.

Since $p_{\beta }$ is a global minimizer of $J_{\beta }\left(p\right)=\beta U\left(p\right)-H\left(P_p\right)$, its objective value cannot exceed that of the feasible uniform product law $p^u$. Hence $J_{\beta }\left(p_{\beta }\right)\le J_{\beta }\left(p^u\right)$, or equivalently,   

$$\beta U\left(p_{\beta }\right)-H\left(P_{p_{\beta }}\right)\le \beta U\left(p^u\right)-H\left(P_{p^u}\right).$$

Since $P_{p^u}=u$ is uniform on the M-element configuration space $\mathcal{A}$, $H\left(P_{p^u}\right)=\log M$. Therefore,

$$\beta U\left(p_{\beta }\right)-H\left(P_{p_{\beta }}\right)\le \beta U\left(p^u\right)-\log M.$$

Thus,

$$D_{\mathrm{KL}}(P_{p_{\beta }}\parallel u)=\log M-H\left(P_{p_{\beta }}\right)\le \beta \{U\left(p^u\right)-U\left(p_{\beta }\right)\}\le \beta R_E,$$

and (13) is determined by Pinsker’s inequality (apply it to $P_{p_{\beta }}$ and u) and (12). Product additivity of KL and Cauchy–Schwarz infer

$${\displaystyle \frac{1}{d}}\sum _i{d}_{\mathrm{TV}}(p_{\beta ,i},u_i)\le {\displaystyle \frac{1}{d}}\sum _i\sqrt{D_{\mathrm{KL}}(p_{\beta ,i}\parallel u_i)/2}\le \sqrt{D_{\mathrm{KL}}(P_{p_{\beta }}\parallel u)/\left(2d\right)}.$$

The mean-field map is the constant map $p^u$ at $\beta =0$, so a unique smooth branch locally is obtained by the implicit function theorem (IFT). Differentiating the softmax equation at zero determines

$${\left.{\displaystyle \frac{d}{d\beta }}{p}_{\beta ,i}\left(a\right)\right|}_{\beta =0}=-K_i^{-1}\left(e_i(a;p_{-i}^u)-K_i^{-1}\sum _b{e}_i(b;p_{-i}^u)\right).$$

Since $p_i^u\left(b\right)=1/K_i$ and $P_{p^u}=p_i^u\otimes P_{p^u,-i}$, the average conditional energy satisfies

$${\displaystyle \frac{1}{K_i}}\sum _{b\in A_i}{e}_i(b;p_{-i}^u)=\sum _{b\in A_i}{p}_i^u\left(b\right){\mathbb{E}}_{P_{p^u,-i}}\left[E(b,{\sigma }_{-i})\right]={\mathbb{E}}_{P_{p^u}}\left[E\left(\sigma \right)\right]=U\left(p^u\right).$$

Hence,

$${\left.{\displaystyle \frac{d}{d\beta }}{p}_{\beta ,i}\left(a\right)\right|}_{\beta =0}=-{\displaystyle \frac{1}{K_i}}\left[e_i(a;p_{-i}^u)-U\left(p^u\right)\right].$$

Using $p_{0,i}\left(a\right)=1/K_i$, the first-order Taylor expansion of the smooth branch $p_{\beta }$ at $\beta =0$ yields (15).    □

Equation (15) formalizes the statement that uniform initialization maximizes entropy. The optimizer starts moving immediately unless symmetry cancels the first-order energy signal.

2.2.2. Weak Interactions: Uniqueness and Global Contraction

For $i\ne j$, denote the influence of coordinate j on the energy preference at coordinate i as

$$\begin{array}{cc}\hfill c_{ij}=\underset{a,b\in {\mathcal{A}}_i}{\max }\underset{x_{-i-j}}{\max }\underset{s,t\in {\mathcal{A}}_j}{\max }|& E(a,s,x_{-i-j})-E(b,s,x_{-i-j})\hfill \\ \hfill & -E(a,t,x_{-i-j})+E(b,t,x_{-i-j})|.\hfill \end{array}$$

(16)

Set $C={\max }_i{\sum }_{j\ne i}{c}_{ij}$. Let $S_{\beta }$ be the parallel softmax map on the right-hand side of (6), and equip the product of marginal simplices with

$$d_{\infty }(p,q)=\underset{i}{\max }{d}_{\mathrm{TV}}(p_i,q_i).$$

Theorem 3

(High-temperature uniqueness). For all $p,q$,

$$d_{\infty }(S_{\beta }p,S_{\beta }q)\le {\alpha }_{\beta }{d}_{\infty }(p,q),{\alpha }_{\beta }={\displaystyle \frac{\beta C}{4}}.$$

(17)

If $\beta C<4$, the FEM stationary point is unique and is the unique global minimizer. Parallel mean-field iteration converges geometrically:

$$d_{\infty }(p^{\left(t\right)},p_{\beta })\le {\alpha }_{\beta }^t{d}_{\infty }(p^{\left(0\right)},p_{\beta }).$$

(18)

The damped update $p^{(t+1)}=(1-\eta )p^{\left(t\right)}+\eta S_{\beta }{p}^{\left(t\right)}$, $0<\eta \le 1$, has contraction factor $1-\eta (1-{\alpha }_{\beta })$.

Lemma 1

(Softmax comparison). For finite vectors $f,g$ and $s_{\beta }\left(f\right)\left(a\right)\propto e^{-\beta f\left(a\right)}$,

$$d_{\mathrm{TV}}(s_{\beta }\left(f\right),s_{\beta }\left(g\right))\le \tanh \left({\displaystyle \frac{\beta osc(f-g)}{4}}\right)\le {\displaystyle \frac{\beta osc(f-g)}{4}}.$$

(19)

Proof. 

Let $P=s_{\beta }\left(f\right)$, $Q=s_{\beta }\left(g\right)$, and $R\left(a\right)=P\left(a\right)/Q\left(a\right)$. Then

$$\log R\left(a\right)=-\beta (f\left(a\right)-g\left(a\right))+\log (Z_g/Z_f),$$

thus

$$osc\left(\log R\right)=\beta osc(f-g)=:L.$$

Let $m={\min }_{a}R\left(a\right)$ and $M={\max }_{a}R\left(a\right)$. Then $M/m\le e^L$, while

$${\mathbb{E}}_{Q}R=\sum _{a}Q\left(a\right)R\left(a\right)=1.$$

Moreover,

$$d_{\mathrm{TV}}(P,Q)={\displaystyle \frac{1}{2}}{\mathbb{E}}_Q|R-1|.$$

Since $R\in [m,M]$ and ${\mathbb{E}}_{Q}R=1$, convexity of $r\mapsto |r-1|$ verifies

$$d_{\mathrm{TV}}(P,Q)\le {\displaystyle \frac{(M-1)(1-m)}{M-m}}.$$

Writing $M/m=t^2$, the bound above is maximized over $m\le 1\le M$ at $m=t^{-1}$ and $M=t$, yielding

$$d_{\mathrm{TV}}(P,Q)\le {\displaystyle \frac{t-1}{t+1}}.$$

Using $t\le e^{L/2}$, one gives

$$d_{\mathrm{TV}}(P,Q)\le {\displaystyle \frac{e^{L/2}-1}{e^{L/2}+1}}=\tanh (L/4).$$

Substituting $L=\beta osc(f-g)$ yields the first inequality in (19), and the second in terms of $\tanh x\le x$ for $x\ge 0$.    □

Proof of Theorem 3.

For given $a,b\in {\mathcal{A}}_i$, telescope from $P_{p,-i}$ to $P_{q,-i}$ one marginal at a time. The definition of $c_{ij}$ and the variational characterization of total variation infer

$$|[e_i(a;p_{-i})-e_i(b;p_{-i})]-[e_i(a;q_{-i})-e_i(b;q_{-i})]|\le \sum _{j\ne i}{c}_{ij}{d}_{\mathrm{TV}}(p_j,q_j).$$

Taking the maximum over $a,b$ bounds the oscillation of the two conditional energy vectors. In light of Lemma 1, one determines

$$d_{\mathrm{TV}}({\left(S_{\beta }p\right)}_i,{\left(S_{\beta }q\right)}_i)\le {\displaystyle \frac{\beta }{4}}\sum _{j\ne i}{c}_{ij}{d}_{\mathrm{TV}}(p_j,q_j),$$

which proves (17).

If $\beta C<4$, then ${\alpha }_{\beta }=\beta C/4<1$, so (17) reveals that $S_{\beta }$ is a contraction on the complete metric space $({\prod }_i{\Delta }_{K_i},d_{\infty })$. Banach’s fixed-point theorem hence guarantees a unique fixed point $p_{\beta }$. Moreover, for the parallel iteration $p^{(t+1)}=S_{\beta }{p}^{\left(t\right)}$,

$$d_{\infty }(p^{(t+1)},p_{\beta })=d_{\infty }(S_{\beta }{p}^{\left(t\right)},S_{\beta }{p}_{\beta })\le {\alpha }_{\beta }{d}_{\infty }(p^{\left(t\right)},p_{\beta }),$$

and iteration of this inequality determines

$$d_{\infty }(p^{\left(t\right)},p_{\beta })\le {\alpha }_{\beta }^t{d}_{\infty }(p^{\left(0\right)},p_{\beta }),$$

which proves (18).

By means of Theorem 1, a global minimizer $p^{★}$ of $F_{\beta }$ exists and is interior. Every such minimizer meets the softmax fixed-point Equation (6), and thus $S_{\beta }{p}^{★}=p^{★}$. Since $S_{\beta }$ has a unique fixed point, $p^{★}=p_{\beta }$; therefore $p_{\beta }$ is the unique global minimizer.

Finally, define the damped map

$$T_{\beta ,\eta }\left(p\right)=(1-\eta )p+\eta S_{\beta }p.$$

According to convexity of total variation under mixtures, for each i,

$$\begin{array}{cc}\hfill d_{\mathrm{TV}}\left({\left(T_{\beta ,\eta }p\right)}_i,{\left(T_{\beta ,\eta }q\right)}_i\right)& \le (1-\eta )d_{\mathrm{TV}}(p_i,q_i)+\eta d_{\mathrm{TV}}\left({\left(S_{\beta }p\right)}_i,{\left(S_{\beta }q\right)}_i\right)\hfill \\ & \le \left[(1-\eta )+\eta {\alpha }_{\beta }\right]d_{\infty }(p,q).\hfill \end{array}$$

Taking the maximum over i gets

$$d_{\infty }(T_{\beta ,\eta }p,T_{\beta ,\eta }q)\le \left[1-\eta (1-{\alpha }_{\beta })\right]d_{\infty }(p,q),$$

thus the damped update has contraction factor $1-\eta (1-{\alpha }_{\beta })$.    □

The condition is sufficient, not necessary. It is a general discrete-state analogue of the familiar high-temperature Dobrushin condition. Its failure marks the end of a theorem-certified single-basin regime, not necessarily the physical transition itself; see [47] for the corresponding Gibbs measure theory.

2.2.3. Hessian Criterion and the First Bifurcation

Let the tangent space of the product simplex be

$$\mathsf{T}=\left\{v=(v_1,\dots ,v_d):\sum _{a\in {\mathcal{A}}_i}{v}_i\left(a\right)=0\mathrm{for}\mathrm{every}i\right\}.$$

At an interior point p, the restricted Hessian of $J_{\beta }$ is formalized by

$${\mathcal{H}}_{\beta }\left(p\right)[v,v]=\sum _{i,a}{\displaystyle \frac{v_i{\left(a\right)}^2}{p_i\left(a\right)}}+\beta {\nabla }^{2}U\left(p\right)[v,v],v\in \mathsf{T}.$$

(20)

Proposition 2

(Local stability threshold). Let p be an interior stationary point of $J_{\beta }$, and let $\mathsf{T}$ be the tangent space defined above. Define

$$D\left(p\right):=diag\{1/p_i\left(a\right)\},{\mathcal{H}}_{\beta }\left(p\right):=D\left(p\right)+\beta {\nabla }^{2}U\left(p\right).$$

Then, for any $v\in \mathsf{T}$,

$$v^{\top }{\mathcal{H}}_{\beta }\left(p\right)v=\sum _{i,a}{\displaystyle \frac{v_i{\left(a\right)}^2}{p_i\left(a\right)}}+\beta {\nabla }^{2}U\left(p\right)[v,v].$$

If ${\mathcal{H}}_{\beta }\left(p\right)$ is positive definite on $\mathsf{T}$, i.e.,

$$v^{\top }{\mathcal{H}}_{\beta }\left(p\right)v>0\mathit{for}\mathit{every}v\in \mathsf{T}\setminus \left\{0\right\},$$

then p is a strict local minimum. If there exists $v\in \mathsf{T}\setminus \left\{0\right\}$ such that

$$v^{\top }{\mathcal{H}}_{\beta }\left(p\right)v<0,$$

then p has a feasible direction of negative curvature and is locally unstable.

Along a smooth stationary branch $p=p\left(\beta \right)$, a necessary condition for loss of local stability is that the smallest eigenvalue of the restricted Hessian reaches zero. Equivalently, after the change of variables $w=D{\left(p\right)}^{1/2}v$, the feasible tangent space becomes $D{\left(p\right)}^{1/2}\mathsf{T}$, and the normalized Hessian is $D{\left(p\right)}^{-1/2}{\mathcal{H}}_{\beta }\left(p\right)D{\left(p\right)}^{-1/2}$. Thus the condition can be written as

$${\lambda }_{\min }\left(D{\left(p\right)}^{-1/2}{\mathcal{H}}_{\beta }\left(p\right)D{\left(p\right)}^{-1/2}{|}_{D{\left(p\right)}^{1/2}\mathsf{T}}\right)=0.$$

(21)

This condition provides an instance-specific candidate critical inverse temperature ${\beta }_c$ for the first possible loss of local stability along the branch.

Proof of Proposition 2.

Since $J_{\beta }\left(p\right)=\beta U\left(p\right)+{\sum }_{i,a}{p}_i\left(a\right)\log p_i\left(a\right)$ and ${\displaystyle \frac{d^2}{d{x}^2}}\left(x\log x\right)={\displaystyle \frac{1}{x}}$, the Hessian of $J_{\beta }$ at an interior point p is

$${\mathcal{H}}_{\beta }\left(p\right)=D\left(p\right)+\beta {\nabla }^{2}U\left(p\right),D\left(p\right)=diag\{1/p_i\left(a\right)\}.$$

Feasible perturbations must preserve the normalization of every marginal, so they satisfy ${\sum }_{a\in {\mathcal{A}}_i}{v}_i\left(a\right)=0$ for each i and hence belong to $\mathsf{T}$. Therefore, for $v\in \mathsf{T}$,

$$v^{\top }{\mathcal{H}}_{\beta }\left(p\right)v=\sum _{i,a}{\displaystyle \frac{v_i{\left(a\right)}^2}{p_i\left(a\right)}}+\beta {\nabla }^{2}U\left(p\right)[v,v],$$

which derives (20).

Assume p is an interior stationary point. For each feasible direction $v\in \mathsf{T}$, the first directional derivative vanishes. Thus,

$$J_{\beta }(p+\epsilon v)-J_{\beta }\left(p\right)={\displaystyle \frac{{\epsilon }^2}{2}}{v}^{\top }{\mathcal{H}}_{\beta }\left(p\right)v+o\left({\epsilon }^2\right).$$

If ${\mathcal{H}}_{\beta }\left(p\right)$ is positive definite on $\mathsf{T}$, the right-hand side is positive for every sufficiently small nonzero feasible perturbation, hence p is a strict local minimum. Conversely, if there exists $v\in \mathsf{T}\setminus \left\{0\right\}$ with $v^{\top }{\mathcal{H}}_{\beta }\left(p\right)v<0$, then sufficiently small perturbations in the direction v decrease the objective, so the stationary point is locally unstable.

Finally, consider a smooth stationary branch $p\left(\beta \right)$. Since $p\left(\beta \right)$ is interior, $D\left(p\right(\beta \left)\right)$ is positive definite. Under the invertible change of variables

$$w=D{\left(p\right)}^{1/2}v,$$

a feasible direction $v\in \mathsf{T}$ corresponds exactly to $w\in D{\left(p\right)}^{1/2}\mathsf{T}$, and

$$v^{\top }{\mathcal{H}}_{\beta }\left(p\right)v=w^{\top }D{\left(p\right)}^{-1/2}{\mathcal{H}}_{\beta }\left(p\right)D{\left(p\right)}^{-1/2}w.$$

Hence the normalized Hessian restricted to $D{\left(p\right)}^{1/2}\mathsf{T}$ has the same inertia as the original Hessian restricted to $\mathsf{T}$. Along a smooth branch the restricted Hessian varies continuously with $\beta$, so its eigenvalues vary continuously as well. Therefore, a change from positive-definite to indefinite curvature can occur only when the smallest restricted eigenvalue first reaches zero. Consequently, a necessary condition for loss of local stability is

$${\lambda }_{\min }\left(D{\left(p\right)}^{-1/2}{\mathcal{H}}_{\beta }\left(p\right)D{\left(p\right)}^{-1/2}{|}_{D{\left(p\right)}^{1/2}\mathsf{T}}\right)=0,$$

which gives (21).    □

The uniform law itself need not remain a stationary point for $\beta >0$. Accordingly, one should monitor the Hessian along the annealed branch rather than assert a universal phase transition at a fixed temperature.

2.2.4. Monotonicity Along Global Optima

Proposition 3

(Energy-entropy monotonicity). Let $T_1>T_2>0$ and let $p^{\left(k\right)}$ globally minimize $F_{T_k}=U-T_{k}H$. Then

$$H\left(P_{p^{\left(1\right)}}\right)\ge H\left(P_{p^{\left(2\right)}}\right),U\left(p^{\left(1\right)}\right)\ge U\left(p^{\left(2\right)}\right).$$

(22)

Thus, along globally optimized annealing, decreasing temperature cannot increase the expected energy or increase entropy. This conclusion may fail for a numerical trajectory that switches between non-global basins.

Proof of Proposition 3.

Optimality at $T_1$ and $T_2$ gives

$$U_1-T_1{H}_1\le U_2-T_1{H}_2,U_2-T_2{H}_2\le U_1-T_2{H}_1.$$

Adding the inequalities yields $(T_1-T_2)(H_1-H_2)\ge 0$, hence $H_1\ge H_2$. Substitution into either inequality determines $U_1\ge U_2$.    □

2.3. The Energy-Dominated Regime

Let

$$\mathcal{G}=\{\sigma :E\left(\sigma \right)=E_{★}\},\Delta =\underset{\sigma \notin \mathcal{G}}{\min }\{E\left(\sigma \right)-E_{★}\},$$

with the concentration statement below interpreted as vacuous if $\mathcal{G}=\mathcal{A}$.

Theorem 4

(Zero-temperature limit, rate, and entropy selection). Let $p_T$ be any global minimizer of $F_T=U-TH$. Then

$$0\le U\left(p_T\right)-E_{★}\le T\log M={\displaystyle \frac{\log M}{\beta }}.$$

(23)

The functionals $F_T$ converge uniformly, and therefore Γ-converge, to U on the compact product simplex as $T\downarrow 0$. Every accumulation point $p_0$ meets

$$P_{p_0}\left(\mathcal{G}\right)=1,H\left(P_{p_0}\right)=\max \{H\left(P_p\right):P_p\left(\mathcal{G}\right)=1\}.$$

(24)

In addition,

$$P_{p_T}\left({\mathcal{G}}^c\right)\le {\displaystyle \frac{T\log M}{\Delta }}.$$

(25)

If the ground state is unique, $P_{p_T}$ converges in total variation to its point mass.

The application of $\Gamma$-convergence follows the standard variational framework in [48]; in the current finite-dimensional setting, the stronger uniform convergence makes the argument particularly direct.

Proof of Theorem 4.

Compare $p_T$ with a ground-state point mass. Since point masses have zero entropy, one gets

$$U\left(p_T\right)-TH\left(P_{p_T}\right)\le E_{★}.$$

The Equation (23) is obtained by virtue of $U\left(p_T\right)\ge E_{★}$ and $H\left(P_{p_T}\right)\le \log M$. It is noteworthy that ${\sup }_p|F_T\left(p\right)-U\left(p\right)|\le T\log M$, so convergence is uniform. Moreover, since $0\le H\left(P_p\right)\le \log M$,

$$\underset{p\in \mathcal{M}}{\sup }|F_T\left(p\right)-U\left(p\right)|\le T\log M\to 0,$$

thus $F_T$ converges uniformly to U on the compact product simplex $\mathcal{M}$. Specifically, for any $T_n\downarrow 0$ and $p_n\to p$, continuity of U determines

$$F_{T_n}\left(p_n\right)=U\left(p_n\right)+o\left(1\right)\to U\left(p\right),$$

which implies the $\Gamma$-liminf inequality; the constant recovery sequence $p_n=p$ gives the corresponding $\Gamma$-limsup inequality. Hence, $F_T$ $\Gamma$-converges to U. Furthermore, compactness implies that any sequence of minimizers $p_{T_n}$ admits a convergent subsequence, say $p_{T_n}\to p_0$. For any $q\in \mathcal{M}$, optimality obtains $F_{T_n}\left(p_{T_n}\right)\le F_{T_n}\left(q\right)$. Passing to the limit using uniform convergence and continuity of U yields $U\left(p_0\right)\le U\left(q\right)$. Since q is arbitrary, $p_0$ is a global minimizer of U.

For any product law $P_p$, $U\left(p\right)=E_{★}$ if and only if it assigns probability one to $\mathcal{G}$. To infer entropy selection, compare $p_T$ with any q satisfying $U\left(q\right)=E_{★}$:

$$U\left(p_T\right)-E_{★}\le T\{H\left(P_{p_T}\right)-H\left(P_q\right)\}.$$

Since $U\left(p_T\right)\ge E^{★}$ and $T>0$, the preceding inequality implies $H\left(P_{p_T}\right)\ge H\left(P_q\right)$ for each q meeting $U\left(q\right)=E^{★}$. Let $T_n\downarrow 0$ be a subsequence such that $p_{T_n}\to p_0$. As shown above, $p_0$ minimizes U, and hence $U\left(p_0\right)=E^{★}$, or equivalently $P_{p_0}\left(G\right)=1$. Since the state space is finite and $x\mapsto -x\log x$ is continuous on $[0,1]$ under the convention $0\log 0=0$, entropy is continuous. Therefore,

$$H\left(P_{p_{T_n}}\right)⟶H\left(P_{p_0}\right).$$

Passing to the limit in $H\left(P_{p_{T_n}}\right)\ge H\left(P_q\right)$ gets $H\left(P_{p_0}\right)\ge H\left(P_q\right)$ for each q with $U\left(q\right)=E^{★}$. Since $U\left(q\right)=E^{★}$ if and only if $P_q\left(G\right)=1$, and $p_0$ itself satisfies this condition,

$$H\left(P_{p_0}\right)=\max \{H\left(P_p\right):P_p\left(G\right)=1\},$$

which, together with $P_{p_0}\left(G\right)=1$, proves (24).

Finally, since $E\left(\sigma \right)-E^{★}=0$ for $\sigma \in \mathcal{G}$ and $E\left(\sigma \right)-E^{★}\ge \Delta$ for $\sigma \in {\mathcal{G}}^c$,

$$\begin{array}{cc}\hfill U\left(p_T\right)-E^{★}& =\sum _{\sigma \in {\mathcal{G}}^c}{P}_{p_T}\left(\sigma \right)\left(E\left(\sigma \right)-E^{★}\right)\hfill \\ & \ge \Delta \sum _{\sigma \in {\mathcal{G}}^c}{P}_{p_T}\left(\sigma \right)=\Delta P_{p_T}\left({\mathcal{G}}^c\right).\hfill \end{array}$$

Combining this lower bound with (23) gives

$$\Delta P_{p_T}\left({\mathcal{G}}^c\right)\le U\left(p_T\right)-E^{★}\le T\log M,$$

and hence yields (25).    □

Since product support of $P_p$ is a Cartesian set ${\prod }_{i}supp\left(p_i\right)$, Equation (24) has a concrete combinatorial interpretation.

Corollary 2

(Rectangular selection under degeneracy). Let

$$L_{★}=\max \left\{\prod _i\left|B_i\right|:⌀\ne B_i\subseteq {\mathcal{A}}_i,\prod _i{B}_i\subseteq \mathcal{G}\right\}.$$

Each zero-temperature accumulation point is uniform on a maximizing Cartesian subset ${\prod }_i{B}_i\subseteq \mathcal{G}$, up to nonuniqueness among subsets with size $L_{★}$, and its entropy is $\log L_{★}$.

Proof of Corollary 2.

Let $p_0$ be a zero-temperature accumulation point. In light of Theorem 4,

$$P_{p_0}\left(G\right)=1,H\left(P_{p_0}\right)=\max \{H\left(P_p\right):P_p\left(G\right)=1\}.$$

For any product law $P_p={⨂}_i{p}_i$, let $B_i=supp\left(p_i\right)$. Since $P_p\left(\sigma \right)={\prod }_i{p}_i\left({\sigma }_i\right)$, its support is $supp\left(P_p\right)={\prod }_i{B}_i$. Thus,

$$P_p\left(G\right)=1⟺\prod _i{B}_i\subseteq G.$$

Thus every feasible product law in the entropy maximization of (24) is supported on a Cartesian subset of the ground-state set.

For a given Cartesian support ${\prod }_i{B}_i\subseteq G$, entropy additivity determines

$$H\left(P_p\right)=\sum _{i}H\left(p_i\right)\le \sum _i\log |B_i|=\log \prod _i\left|B_i\right|,$$

where equality holds if and only if each $p_i$ is uniform on $B_i$. In that case $P_p$ is uniform on ${\prod }_i{B}_i$. Therefore, maximizing entropy over all product laws supported on G is equivalent to maximizing the cardinality of a Cartesian subset of G:

$$L^{★}=\max \left\{\prod _i\left|B_i\right|:⌀\ne B_i\subseteq A_i,pro{d}_i{B}_i\subseteq G\right\}.$$

Consequently, each zero-temperature accumulation point is uniform on a maximizing Cartesian subset of G, and $H\left(P_{p_0}\right)=\log L^{★}$. If several distinct Cartesian subsets attain cardinality $L^{★}$, the maximizing product law, and hence the accumulation point, need not be unique.    □

This differs from the exact Gibbs limit, which is uniform over all ground states. If two optimal configurations cannot be mixed coordinatewise without creating a suboptimal configuration, a product law cannot represent their mixture; reverse KL selects one compatible rectangle, often a single mode.

2.3.1. Approximate Optimization, Rounding, and Replicas

Global minimization is not what a generic gradient approach guarantees. The following theorem sheds light on exactly where numerical error enters.

Theorem 5

(Approximate solution and extraction guarantees). Suppose $\widehat{p}$ is ${\epsilon }_{\mathrm{opt}}$-optimal:

$$F_T\left(\widehat{p}\right)\le \underset{p}{\min }{F}_T\left(p\right)+{\epsilon }_{\mathrm{opt}},a_T=T\log M+{\epsilon }_{\mathrm{opt}}.$$

Then

$${\mathbb{E}}_{P_{\widehat{p}}}[E-E_{★}]\le a_T.$$

(26)

If $\widehat{\sigma }\sim P_{\widehat{p}}$, then for any $r>0$,

$$\mathbb{P}\{E\left(\widehat{\sigma }\right)-E_{★}>r\}\le \min \{1,a_T/r\}.$$

(27)

If ${\widehat{\sigma }}^{\left(1\right)},\dots ,{\widehat{\sigma }}^{\left(R\right)}$ are independently sampled from $P_{\widehat{p}}$, then

$$\mathbb{P}\left\{\underset{1\le s\le R}{\min }E\left({\widehat{\sigma }}^{\left(s\right)}\right)-E_{★}>r\right\}\le \min \{1,{(a_T/r)}^R\}.$$

(28)

Finally, deterministic coordinatewise mode rounding satisfies

$$E\left({\sigma }^{\mathrm{mode}}\right)-E_{★}\le {\displaystyle \frac{a_T}{{\rho }_{\mathrm{mode}}}}\le M{a}_T,$$

(29)

where ${\sigma }_i^{\mathrm{mode}}\in {arg\; max}_a{\widehat{p}}_i\left(a\right)$ and ${\rho }_{\mathrm{mode}}=P_{\widehat{p}}\left({\sigma }^{\mathrm{mode}}\right)$.

Proof of Theorem 5.

Since a ground-state point mass is feasible, ${\min }_p{F}_T\left(p\right)\le E_{★}$. Hence,

$$U\left(\widehat{p}\right)-E_{★}=F_T\left(\widehat{p}\right)+TH\left(P_{\widehat{p}}\right)-E_{★}\le {\epsilon }_{\mathrm{opt}}+T\log M=a_T.$$

Inequality (27) is obtained by applying Markov’s inequality to the nonnegative random variable $E\left(\widehat{\sigma }\right)-E^{★}$, whose expectation is bounded by $a_T$ in terms of (26). Since the R samples are independent, the probability that all of them have excess energy greater than r is at most ${(a_T/r)}^R$, which yields (28). For the mode $s={\sigma }^{\mathrm{mode}}$,

$$a_T\ge {\mathbb{E}}_{P_{\widehat{p}}}(E-E_{★})\ge P_{\widehat{p}}\left(s\right)\{E\left(s\right)-E_{★}\}.$$

Since $P_{\widehat{p}}$ is a product law and ${\sigma }_i^{\mathrm{mode}}\in \arg {\max }_a{\widehat{p}}_i\left(a\right)$, one gets

$${\rho }_{\mathrm{mode}}=P_{\widehat{p}}\left({\sigma }^{\mathrm{mode}}\right)=\prod _i\underset{a}{\max }{\widehat{p}}_i\left(a\right).$$

For each i, ${\max }_a{\widehat{p}}_i\left(a\right)\ge 1/K_i$, since the $K_i$ probabilities sum to one. Thus, ${\rho }_{\mathrm{mode}}\ge {\prod }_i{K}_i^{-1}=M^{-1}$. Combining this with $a_T\ge {\rho }_{\mathrm{mode}}[E\left({\sigma }^{\mathrm{mode}}\right)-E^{★}]$ yields (29).    □

The random-sampling bound can be far sharper than the worst-case mode bound. The latter necessarily contains the mode mass: a product expectation can be small even when a low-probability coordinatewise mode has poor energy. In actuality, evaluating both the mode and several samples from each final product law is therefore theoretically defensible.

The R batched FEM replicas in [25] are optimized from different initial conditions rather than sampled from a single globally optimized law. If a random initialization reaches a target basin with probability s, and replicas behave independently, the best-of-R failure probability is ${(1-s)}^R$. This is an optimization-search guarantee, not an increase in the expressive power of the product family; s is problem- and optimizer-dependent and has no positive universal lower bound.

2.3.2. Why Low Temperature Creates Many Traps

Call ${\sigma }^{\circ }$ a strict one-flip local optimum if

$${\gamma }_i=\underset{a\ne {\sigma }_i^{\circ }}{\min }\{E(a,{\sigma }_{-i}^{\circ })-E\left({\sigma }^{\circ }\right)\}>0\mathrm{for}\mathrm{every}i.$$

(30)

Let $C_i={\sum }_{j\ne i}{c}_{ij}$, with $c_{ij}$ from (16).

Proposition 4

(Stationary points near discrete local optima). Fix $r\in (0,1)$. If

$$C_{i}r\le {\gamma }_i/2,(K_i-1)e^{-\beta {\gamma }_i/2}\le r\mathit{for}\mathit{every}i,$$

(31)

then the mean-field map $S_{\beta }$ has a fixed point in

$${\mathcal{B}}_r\left({\sigma }^{\circ }\right)=\{p:p_i\left({\sigma }_i^{\circ }\right)\ge 1-r\mathit{for}\mathit{every}i\}.$$

At that fixed point,

$$1-p_i\left({\sigma }_i^{\circ }\right)\le (K_i-1)\exp \left[-\beta \left({\gamma }_i-\sum _{j\ne i}{c}_{ij}\{1-p_j\left({\sigma }_j^{\circ }\right)\}\right)\right].$$

(32)

If its restricted Hessian (20) is positive definite, this point is a strict local minimum.

Proof of Proposition 4.

For $a\ne {\sigma }_i^{\circ }$, define $D_i^a\left(x_{-i}\right)=E(a,x_{-i})-E({\sigma }_i^{\circ },x_{-i})$. For $X_{-i}\sim P_{p,-i}$, by changing the coordinates of $X_{-i}$ one at a time to those of ${\sigma }_{-i}^{\circ }$ and using the definition of $c_{ij}$ in (16), we obtain

$$D_i^a\left(X_{-i}\right)\ge D_i^a\left({\sigma }_{-i}^{\circ }\right)-\sum _{j\ne i}{c}_{ij}\mathbf{1}\{X_j\ne {\sigma }_j^{\circ }\}.$$

Taking expectations and using $D_i^a\left({\sigma }_{-i}^{\circ }\right)\ge {\gamma }_i$, one obtains

$${\mathbb{E}}_{P_{p,-i}}{D}_i^a\ge {\gamma }_i-\sum _{j\ne i}{c}_{ij}\{1-p_j\left({\sigma }_j^{\circ }\right)\}.$$

Since $p\in {\mathcal{B}}_r\left({\sigma }^{\circ }\right)$, $1-p_j\left({\sigma }_j^{\circ }\right)\le r$, and hence

$${\mathbb{E}}_{P_{p,-i}}{D}_i^a\ge {\gamma }_i-C_{i}r\ge {\gamma }_i/2.$$

For $a\ne {\sigma }_i^{\circ }$, the softmax equation obtains

$${\displaystyle \frac{{\left(S_{\beta }p\right)}_i\left(a\right)}{{\left(S_{\beta }p\right)}_i\left({\sigma }_i^{\circ }\right)}}=\exp \left\{-\beta \left[e_i(a;p_{-i})-e_i({\sigma }_i^{\circ };p_{-i})\right]\right\}=\exp \left\{-\beta {\mathbb{E}}_{P_{p,-i}}{D}_i^a\right\}.$$

Since ${\mathbb{E}}_{P_{p,-i}}{D}_i^a\ge {\gamma }_i/2$, it follows that

$${\left(S_{\beta }p\right)}_i\left(a\right)\le e^{-\beta {\gamma }_i/2}.$$

Thus,

$$1-{\left(S_{\beta }p\right)}_i\left({\sigma }_i^{\circ }\right)=\sum _{a\ne {\sigma }_i^{\circ }}{\left(S_{\beta }p\right)}_i\left(a\right)\le (K_i-1)e^{-\beta {\gamma }_i/2}\le r.$$

Hence, for each $p\in {\mathcal{B}}_r\left({\sigma }^{\circ }\right)$ and each i,

$${\left(S_{\beta }p\right)}_i\left({\sigma }_i^{\circ }\right)\ge 1-r,$$

and hence $S_{\beta }\left({\mathcal{B}}_r\left({\sigma }^{\circ }\right)\right)\subseteq {\mathcal{B}}_r\left({\sigma }^{\circ }\right)$. The set ${\mathcal{B}}_r\left({\sigma }^{\circ }\right)$ is nonempty, compact, and convex, and $S_{\beta }$ is continuous. Brouwer’s fixed-point theorem therefore yields a point $p^{★}\in {\mathcal{B}}_r\left({\sigma }^{\circ }\right)$ satisfying $S_{\beta }{p}^{★}=p^{★}$.

At this fixed point, retaining the individual error probabilities rather than replacing them by the uniform bound r gets

$${\mathbb{E}}_{P_{p^{★},-i}}{D}_i^a\ge {\gamma }_i-\sum _{j\ne i}{c}_{ij}\{1-p_j^{★}\left({\sigma }_j^{\circ }\right)\}.$$

Using the softmax fixed-point equation and summing over $a\ne {\sigma }_i^{\circ }$, we therefore deduce (32). Finally, since a fixed point of $S_{\beta }$ is an interior stationary point of $F_{\beta }$, positive definiteness of the restricted Hessian on the simplex tangent space implies, by the second-order sufficient condition, that $p^{★}$ is a strict local minimum.    □

Therefore, each sufficiently robust one-flip optimum can seed a low-temperature stationary branch. Entropy smooths the landscape at the start of annealing, but it cannot remove the combinatorial multiplicity after the weak-interaction regime ends.

3. Annealing, Stability, and Computation

This section investigates several computational implications of the preceding variational analysis. It first examines the tracking behavior of damped mean-field iterations along the temperature-dependent solution path within the contractive regime. The sensitivity of FEM solutions to approximate or noisy energy evaluations is then quantified. Finally, the per-step computational complexity for factorized energies is analyzed, clarifying when exact objective and gradient evaluations remain tractable despite the difficulty of global optimization.

3.1. Tracking the High-Temperature Solution Path

The next theorem unveils an idealized damped mean-field update, which is a baseline for annealing analysis.

Theorem 6

(Adiabatic tracking under contraction). Assume ${\beta }_t\in [0,\overline{\beta }]$ and $\overline{\alpha }=\overline{\beta }C/4<1$. Let $p_{\beta }$ be the unique fixed point and set

$$B=\underset{p,i}{\sup }{osc}_{a\in {\mathcal{A}}_i}{e}_i(a;p_{-i})\le R_E,K={\displaystyle \frac{B}{4(1-\overline{\alpha })}}.$$

Then,

$$d_{\infty }(p_{\beta },p_{{\beta }^{\prime }})\le K|\beta -{\beta }^{\prime }|.$$

(33)

For the update

$$p^{(t+1)}=(1-\eta )p^{\left(t\right)}+\eta S_{{\beta }_{t+1}}{p}^{\left(t\right)},0<\eta \le 1,$$

write $q=1-\eta (1-\overline{\alpha })$. If ${\delta }_{\beta }={\max }_t|{\beta }_{t+1}-{\beta }_t|$, then

$$d_{\infty }(p^{\left(t\right)},p_{{\beta }_t})\le q^t{d}_{\infty }(p^{\left(0\right)},p_{{\beta }_0})+{\displaystyle \frac{K{\delta }_{\beta }}{\eta (1-\overline{\alpha })}}.$$

(34)

Proof of Theorem 6.

Applying Lemma 1 to the same conditional energy vector at two inverse temperatures, one gives

$$d_{\infty }(S_{\beta }p,S_{{\beta }^{\prime }}p)\le B|\beta -{\beta }^{\prime }|/4.$$

Combining this with contraction and the fixed-point identities obtains

$$d_{\infty }(p_{\beta },p_{{\beta }^{\prime }})\le \overline{\alpha }{d}_{\infty }(p_{\beta },p_{{\beta }^{\prime }})+B|\beta -{\beta }^{\prime }|/4,$$

which determines (33).

Define the damped map

$$T_{\beta }\left(p\right)=(1-\eta )p+\eta S_{\beta }p.$$

By means of convexity of total variation under mixtures and the uniform contraction bound $d_{\infty }(S_{\beta }p,S_{\beta }q)\le \overline{\alpha }{d}_{\infty }(p,q)$,

$$d_{\infty }(T_{\beta }p,T_{\beta }q)\le \{(1-\eta )+\eta \overline{\alpha }\}{d}_{\infty }(p,q)=q{d}_{\infty }(p,q),$$

where $q=1-\eta (1-\overline{\alpha })<1$. Moreover, according to $S_{{\beta }_{t+1}}{p}_{{\beta }_{t+1}}=p_{{\beta }_{t+1}}$, the point $p_{{\beta }_{t+1}}$ is also a fixed point of $T_{{\beta }_{t+1}}$. Thus, writing $e_t=d_{\infty }(p^{\left(t\right)},p_{{\beta }_t})$, one gets

$$\begin{array}{cc}\hfill e_{t+1}& =d_{\infty }\left(T_{{\beta }_{t+1}}{p}^{\left(t\right)},T_{{\beta }_{t+1}}{p}_{{\beta }_{t+1}}\right)\le q{d}_{\infty }(p^{\left(t\right)},p_{{\beta }_{t+1}})\le q\left\{e_t+d_{\infty }(p_{{\beta }_t},p_{{\beta }_{t+1}})\right\}\hfill \\ & \le q\left\{e_t+K|{\beta }_{t+1}-{\beta }_t|\right\}\le q{e}_t+qK{\delta }_{\beta },\hfill \end{array}$$

where the penultimate inequality follows from (33). Iterating this recursion yields

$$e_t\le q^t{e}_0+K{\delta }_{\beta }{\displaystyle \sum _{j=1}^t}{q}^j=q^t{e}_0+{\displaystyle \frac{q(1-q^t)}{1-q}}K{\delta }_{\beta }.$$

Since $q\le 1$ and $1-q=\eta (1-\overline{\alpha })$, one has

$$e_t\le q^t{e}_0+{\displaystyle \frac{K{\delta }_{\beta }}{\eta (1-\overline{\alpha })}},$$

which is (34).    □

The bound determines a precise meaning to “slow enough” annealing before the first bifurcation. After contractivity is lost, a universal path-tracking guarantee requires stronger, branch-specific curvature assumptions.

3.2. Stability to an Approximate or Noisy Energy

Theorem 7

(Energy-perturbation stability). Let E and $\tilde{E}$ satisfy $\parallel E-\tilde{E}{\parallel }_{\infty }\le {\epsilon }_E$. Their optimal free-energy values differ by at most ${\epsilon }_E$. If both mean-field maps are contractive with a common factor $\alpha <1$, their unique solutions satisfy

$$d_{\infty }(p_{\beta },{\tilde{p}}_{\beta })\le {\displaystyle \frac{\beta {\epsilon }_E}{2(1-\alpha )}}.$$

(35)

Proof of Theorem 7.

For any p, the two free-energy objectives have the same entropy term, and therefore

$$F_{\beta ,E}\left(p\right)-F_{\beta ,\tilde{E}}\left(p\right)=U_E\left(p\right)-U_{\tilde{E}}\left(p\right)={\mathbb{E}}_{P_p}[E\left(\sigma \right)-\tilde{E}\left(\sigma \right)].$$

Thus, by virtue of $\parallel E-\tilde{E}{\parallel }_{\infty }\le {\epsilon }_E$,

$$|F_{\beta ,E}\left(p\right)-F_{\beta ,\tilde{E}}\left(p\right)|\le {\mathbb{E}}_{P_p}|E\left(\sigma \right)-\tilde{E}\left(\sigma \right)|\le \parallel E-\tilde{E}{\parallel }_{\infty }\le {\epsilon }_E.$$

Since the preceding bound holds uniformly for each p, one infers

$$F_{\beta ,E}\left(p\right)\le F_{\beta ,\tilde{E}}\left(p\right)+{\epsilon }_E\mathrm{for}\mathrm{all}p.$$

Taking the minimum over p on the left and estimating the right-hand side at a minimizer of

$$F_{\beta ,\tilde{E}}$$

gives

$$\underset{p}{\min }{F}_{\beta ,E}\left(p\right)\le \underset{p}{\min }{F}_{\beta ,\tilde{E}}\left(p\right)+{\epsilon }_E.$$

Interchanging E and $\tilde{E}$ similarly deduces

$$\underset{p}{\min }{F}_{\beta ,\tilde{E}}\left(p\right)\le \underset{p}{\min }{F}_{\beta ,E}\left(p\right)+{\epsilon }_E.$$

Combining the two inequalities yields

$$\left|\underset{p}{\min }{F}_{\beta ,E}\left(p\right)-\underset{p}{\min }{F}_{\beta ,\tilde{E}}\left(p\right)\right|\le {\epsilon }_E,$$

which infers the optimal-value bound.

For each i, a, and p, the corresponding conditional energies satisfy

$$|e_{i,E}(a;p_{-i})-e_{i,\tilde{E}}(a;p_{-i})|\le {\mathbb{E}}_{P_{p,-i}}|E(a,{\sigma }_{-i})-\tilde{E}(a,{\sigma }_{-i})|\le {\epsilon }_E.$$

Thus,

$${osc}_a\left(e_{i,E}(a;p_{-i})-e_{i,\tilde{E}}(a;p_{-i})\right)\le 2{\epsilon }_E.$$

In terms of Lemma 1, one confirms

$$d_{\mathrm{TV}}\left({\left(S_{\beta ,E}p\right)}_i,{\left(S_{\beta ,\tilde{E}}p\right)}_i\right)\le {\displaystyle \frac{\beta {\epsilon }_E}{2}},$$

and taking the maximum over i yields

$$d_{\infty }(S_{\beta ,E}p,S_{\beta ,\tilde{E}}p)\le {\displaystyle \frac{\beta {\epsilon }_E}{2}}.$$

Now use the fixed-point identities $p_{\beta }=S_{\beta ,E}{p}_{\beta }$, ${\tilde{p}}_{\beta }=S_{\beta ,\tilde{E}}{\tilde{p}}_{\beta }$. Adding and subtracting $S_{\beta ,E}{\tilde{p}}_{\beta }$ and applying the triangle inequality and the common contraction bound with factor $\alpha$, we obtain

$$\begin{array}{cc}\hfill d_{\infty }(p_{\beta },{\tilde{p}}_{\beta })& \le d_{\infty }(S_{\beta ,E}{p}_{\beta },S_{\beta ,E}{\tilde{p}}_{\beta })+d_{\infty }(S_{\beta ,E}{\tilde{p}}_{\beta },S_{\beta ,\tilde{E}}{\tilde{p}}_{\beta })\hfill \\ & \le \alpha d_{\infty }(p_{\beta },{\tilde{p}}_{\beta })+{\displaystyle \frac{\beta {\epsilon }_E}{2}}.\hfill \end{array}$$

Since $\alpha <1$, rearranging derives (35).    □

The factor $\beta$ is unavoidable: low temperature amplifies small energy errors. This matters when $U\left(p\right)$ or its gradient is estimated by stochastic sampling, low precision arithmetic, or a learned surrogate.

3.3. Per-Step Complexity for Factorized Energies

Suppose the energy admits a factor-graph representation

$$E\left(\sigma \right)=\sum _{a\in \mathcal{F}}{E}_a\left({\sigma }_{S_a}\right),\left|S_a\right|\le k,K_i\le K.$$

(36)

Then,

$$U\left(p\right)=\sum _{a\in \mathcal{F}}\sum _{{\sigma }_{S_a}}{E}_a\left({\sigma }_{S_a}\right)\prod _{i\in S_a}{p}_i\left({\sigma }_i\right).$$

(37)

Proposition 5

(Arithmetic cost). For (36), the exact objective and all factor-to-variable gradient messages can be evaluated in

$$O\left(\sum _{a\in \mathcal{F}}\left|S_a\right|K^{|S_a|}\right)=O\left(\right|\mathcal{F}|k{K}^k)$$

(38)

arithmetic operations, plus $O\left(dK\right)$ for entropy. A batch of R replicas multiplies the arithmetic work and parameter memory by R, but the replica dimension is embarrassingly parallel. Hence, bounded arity and sparse factors determine polynomial per-step cost even though global free-energy minimization can remain NP-hard.

Proof of Proposition 5.

For a factor $a\in \mathcal{F}$, set $s_a=\left|S_a\right|\le k$ and

$$U_a\left(p\right)=\sum _{{\sigma }_{S_a}}{E}_a\left({\sigma }_{S_a}\right)\prod _{i\in S_a}{p}_i\left({\sigma }_i\right),$$

so that $U\left(p\right)={\sum }_{a\in \mathcal{F}}{U}_a\left(p\right)$. Since each variable has at most K states, the sum defining $U_a$ contains at most $K^{s_a}$ local configurations. Evaluating the factor value and the corresponding product weight for every local configuration therefore costs $O\left(K^{s_a}\right)$ arithmetic operations up to a factor linear in $s_a$.

For each $i\in S_a$ and $x\in {\mathcal{A}}_i$, the factor-to-variable derivative is

$${\displaystyle \frac{\partial U_a}{\partial p_i\left(x\right)}}=\sum _{\begin{array}{c}{\sigma }_{S_a}:{\sigma }_i=x\end{array}}{E}_a\left({\sigma }_{S_a}\right)\prod _{j\in S_a\setminus \left\{i\right\}}{p}_j\left({\sigma }_j\right).$$

All such messages can be accumulated in a single enumeration of the $K^{s_a}$ local configurations. For each configuration, the products excluding one coordinate can be formed for all $i\in S_a$ in $O\left(s_a\right)$ time, for example by prefix–suffix products, and the resulting contribution is added to the message indexed by $(i,{\sigma }_i)$. Thus the objective contribution and all messages associated with factor a require $O\left(s_a{K}^{s_a}\right)$ operations. Summing over factors gives

$$O\left(\sum _{a\in \mathcal{F}}\left|S_a\right|K^{|S_a|}\right)\le O\left(\right|\mathcal{F}|k{K}^k),$$

which proves (38).

The entropy term decomposes as

$$-H\left(P_p\right)={\displaystyle \sum _{i=1}^d}\sum _{x\in {\mathcal{A}}_i}{p}_i\left(x\right)\log p_i\left(x\right),$$

so evaluating it and its coordinate derivatives entails $O\left({\sum }_i{K}_i\right)=O\left(dK\right)$ additional operations. Repeating the same calculations for R independent replicas multiplies both arithmetic work and parameter storage by R, and this dimension is embarrassingly parallel since no operation couples different replicas. It is noteworthy that Proposition 5 concerns exact per-step evaluation only and does not imply polynomial-time global minimization of the nonconvex free-energy objective.    □

Indeed, a polynomial-time algorithm that globally minimized (2) for arbitrary energies through sufficiently large $\beta$ would solve any instance of (1), contradicting standard complexity assumptions for NP-hard COPs [2]. The tractability guarantee is therefore for each iteration and, in the contractive regime, for convergence to the unique high-temperature solution. However, it is not a general low-temperature global guarantee.

4. Annealed FEM Algorithm and Reproducible Experiments

The foregoing conclusions distinguish three mechanisms that can otherwise be conflated in an implementation: entropy controls the high-temperature geometry, annealing moves the objective through a family of landscapes, and replicas enhance the chance of entering a favorable basin. This section turns that distinction into an explicit algorithm and tests its observable consequences on small instances for which the true optimum can be certified. The experiments are deliberately diagnostic rather than a large-scale solver benchmark.

4.1. A Batched Annealed Algorithm

Algorithm 1 implements the exact objective in (2), which applies the centered gradient (7) to R product laws in parallel and allows any first-order optimizer. Centering the logits after every update fixes the softmax gauge without changing the marginals. The schedule may be specified stepwise, as presented, or interpolated continuously. The output rule evaluates both marginal modes and optional samples since Theorem 5 yields different guarantees for these two candidate classes.

Algorithm 1 Annealed free-energy machine with batched replicas

Require:  Energy E or its factor graph; inverse temperatures $0<{\beta }_1\le \dots \le {\beta }_M$; R replicas; $S_m$ updates per stage; first-order optimizer $\mathsf{Opt}$; final samples $Q\ge 0$   1: Initialize $h_{i,a}^{\left(r\right)}$ near zero for all $r,i,a$, independently   2: for $m=1,\dots ,M$ do   3:     for $s=1,\dots ,S_m$ do   4:         $p_i^{\left(r\right)}\leftarrow softmax\left(h_i^{\left(r\right)}\right)$ for every $r,i$   5:         Evaluate $U\left(p^{\left(r\right)}\right)$ and $\partial U/\partial p_i^{\left(r\right)}\left(a\right)$ factorwise   6:         $g_i^{\left(r\right)}\left(a\right)\leftarrow \partial U/\partial p_i^{\left(r\right)}\left(a\right)+{\beta }_m^{-1}\{\log p_i^{\left(r\right)}\left(a\right)+1\}$   7:         $G_i^{\left(r\right)}\left(a\right)\leftarrow p_i^{\left(r\right)}\left(a\right)\left[g_i^{\left(r\right)}\left(a\right)-{\sum }_b{p}_i^{\left(r\right)}\left(b\right)g_i^{\left(r\right)}\left(b\right)\right]$   8:         $h\leftarrow \mathsf{Opt}(h,G)$   9:         $h_{i,a}^{\left(r\right)}\leftarrow h_{i,a}^{\left(r\right)}-K_i^{-1}{\sum }_b{h}_{i,b}^{\left(r\right)}$ for every $r,i,a$ 10:     end for 11: end for 12: Form $\mathcal{C}$ from every coordinatewise mode and Q independent samples from each final product law 13: return ${arg\; min}_{\sigma \in \mathcal{C}}E\left(\sigma \right)$

For a sparse factor graph, each update has the cost in Proposition 5; no enumeration of the full configuration space is required. Near-zero logits correspond to near-uniform marginals, which is justified by Theorem 2. The tracking theorem bears gradual changes of $\beta$ while the contraction condition is established. After that condition fails, Algorithm 1 remains a heuristic: neither the temperature schedule nor the use of an adaptive optimizer supplies a global low-temperature convergence theorem.

4.2. Controlled Experimental Design

Three binary problem families are studied with $d=16$ variables and 20 independent instances per family. Sparse Ising models use edge density 0.35, Gaussian couplings scaled by $d^{-1/2}$, and weak Gaussian fields of standard deviation 0.08. Weighted MaxCut instances use Erdős–Rényi graphs with edge probability 0.30 and independent edge weights uniform on [0.5, 1.5]. Random Max-3SAT instances include $4d$ three-literal clauses. Exhaustive enumeration of all $2^{16}$ assignments is used only to calculate $E_{★}$ and $E_{\max }$ for evaluation; FEM energies and gradients are evaluated exactly in terms of local factors.

The annealed approach uses $R=64$ replicas, logits initialized from $N(0,0.{02}^2)$, 500 adaptive-gradient updates with learning rate 0.08, and an exponential schedule from $\beta =0.05$ to 25. Comparisons are made with the same FEM implementation run for 500 updates at the fixed value $\beta =25$ and with 64 single-spin simulated-annealing chains following the same inverse-temperature schedule [10,11]. The latter makes 500 proposals for each chain and is included as a stochastic reference, not as a compute-matched ranking. FEM candidates are the coordinatewise modes of the final marginals. For each instance, exact-ground-state success and the normalized excess are recorded as

$${\mathcal{E}}_{\mathrm{norm}}={\displaystyle \frac{E_{\mathrm{best}}-E_{★}}{E_{\max }-E_{★}}}.$$

(39)

All hyperparameters were fixed across the three problem families.

4.3. Results and Interpretation

Table 1 uncovers means over the 20 instances in each family. Annealing improves on the fixed low-temperature start for Ising and Max-3SAT in both success and excess energy. MaxCut is different: the fixed run succeeds on all instances, whereas annealing succeeds on 75%. A plausible explanation is provided by the label symmetry discussed in Section 6.2. For Potts MaxCut, the uniform law is stationary, so the entropy-dominated early stages of annealing keep the iterates close to a symmetric state until a symmetry-breaking instability is crossed. With a finite update budget, part of the optimization effort may therefore be spent near the symmetric branch, leaving fewer effective iterations for selecting and refining a favorable low-temperature branch. By contrast, the fixed-$\beta$ run starts directly in the symmetry-broken low-temperature landscape and avoids this delayed transition. This interpretation suggests that the MaxCut anomaly is more naturally attributed to the interaction among symmetry, annealing schedule, optimizer dynamics, and finite budget than to entropy being intrinsically detrimental to MaxCut. The present experiments do not distinguish these mechanisms uniquely, so this explanation should be regarded as consistent with the theoretical analysis rather than as a causal identification.

Table 1. Controlled $d=16$ experiments. Each entry aggregates 20 instances and 64 replicas or chains. Excess is the mean of (39); uncertainty is one standard error.

Problem	Method	Exact Success	Normalized Excess
Ising	Annealed FEM	$0.80$	$0.00319\pm 0.00175$
	Fixed-$\beta$ FEM	$0.60$	$0.02289\pm 0.00986$
	Simulated annealing	$1.00$	$0.00000\pm 0.00000$
MaxCut	Annealed FEM	$0.75$	$0.00186\pm 0.00092$
	Fixed-$\beta$ FEM	$1.00$	$0.00000\pm 0.00000$
	Simulated annealing	$1.00$	$0.00000\pm 0.00000$
Max-3SAT	Annealed FEM	$0.80$	$0.01074\pm 0.00499$
	Fixed-$\beta$ FEM	$0.65$	$0.01784\pm 0.00566$
	Simulated annealing	$1.00$	$0.00000\pm 0.00000$

Figure 1a,b changes the number of annealed FEM replicas while keeping the first $R\in \{1,4,16,64\}$ runs from the same batch. Replica selection garners a striking improvement for MaxCut, a slight gain for Ising, and no measurable gain for Max-3SAT on these instances. This is consistent with the basin calculation following Theorem 5: replication can amplify a nonzero basin-entry probability but cannot improve the product family or create a favorable basin. Figure 1c presents the predicted thermal behavior on a representative Ising instance: entropy begins near its maximum and falls as the expected-energy gap approaches zero. Figure 1d summarizes the method comparison. Simulated annealing reaches each exact optimum here, but this observation should not be extrapolated to large-scale solver performance since the instances are intentionally small and the workloads are not matched.

Figure 1. Exactly verifiable synthetic experiments. (a) Mean normalized excess and (b) exact-ground-state frequency for annealed FEM as the replica count grows. (c) Expected-energy gap and entropy along one Ising trajectory. (d) Exact success for annealed FEM, fixed-temperature FEM, and simulated annealing with 64 parallel runs.

5. High-Dimensional and Statistical Guarantees

5.1. High-Dimensional Phase Diagram

Consider a sequence indexed by the number of variables d, with state size $K_i\le K$, energy oscillation $R_d$, ground-state gap ${\Delta }_d$, and inverse temperature ${\beta }_d$. The finite-dimensional results immediately infer the scaling laws as follows.

Corollary 3

(Three asymptotic regimes). Let $p_d$ be a global FEM minimizer.

1.

If ${\beta }_d{R}_d/d\to 0$, then the average marginal distance to uniform vanishes:

$${\displaystyle \frac{1}{d}}\sum _i{d}_{\mathrm{TV}}(p_{d,i},u_i)\to 0.$$

Joint convergence to uniform follows under the stronger condition ${\beta }_d{R}_d\to 0$.

2.

The expected energy-density error satisfies

$${\displaystyle \frac{U\left(p_d\right)-E_{★,d}}{d}}\le {\displaystyle \frac{\log K}{{\beta }_d}}.$$

Thus ${\beta }_d\to \infty$ is sufficient for asymptotically optimal energy for each variable.

3.

Exact ground-set concentration can be obtained if

$${\displaystyle \frac{{\beta }_d{\Delta }_d}{d\log K}}\to \infty .$$

For extensive energies $R_d=O\left(d\right)$, the universal bounds separate a high-temperature marginal-uniform regime ${\beta }_d\to 0$ from a low-temperature energy-density regime ${\beta }_d\to \infty$. The intermediate scale ${\beta }_d=\Theta \left(1\right)$ is necessarily model-specific and contains the bifurcations observed in practice. Concentrating on an exact ground state can require a much lower temperature than obtaining a good energy density, especially when ${\Delta }_d$ is small.

Consistent with the qualitative three-stage annealing picture described by Shen et al. [25], the above-mentioned bounds render a quantitative organization of the FEM path into three regimes:

Regime	Typical scale	Certified or expected behavior
Entropy dominated	${\beta }_d{R}_d/d\to 0$	average marginals approach uniform; weak interactions give a unique, contractive fixed point
Bifurcation	${\beta }_d=\Theta \left(1\right)$ for extensive energies	curvature and the interaction spectrum determine branch formation; behavior is instance dependent
Energy dominated	${\beta }_d\to \infty$	energy density approaches the optimum, while exact ground-set concentration requires the stronger gap-dependent scaling in Corollary 3

5.2. Statistical Learning When Energy Is Empirical Risk

The preceding contexts are deterministic and apply to a single COP instance. Random data are required for the genuine statistical-learning interpretation. Suppose $Z_1,\dots ,Z_n$ are i.i.d., $0\le \ell (\sigma ;Z)\le 1$, and

$$E_n\left(\sigma \right)={\displaystyle \sum _{t=1}^n}\ell (\sigma ;Z_t),{\widehat{R}}_n\left(\sigma \right)=n^{-1}{E}_n\left(\sigma \right),R\left(\sigma \right)=\mathbb{E}\ell (\sigma ;Z).$$

(40)

For a law Q, set ${\widehat{R}}_n\left(Q\right)={\mathbb{E}}_Q{\widehat{R}}_n\left(\sigma \right)$ and $R\left(Q\right)={\mathbb{E}}_{Q}R\left(\sigma \right)$. Adding the constant ${\beta }^{-1}\log M$ and dividing the FEM objective by n obtains

$${\widehat{R}}_n\left(P_p\right)+{\displaystyle \frac{1}{\eta }}{D}_{\mathrm{KL}}(P_p\parallel u),$$

(41)

where $\eta =\beta n$. Thus, FEM is a product-restricted generalized Gibbs posterior with uniform prior and learning rate $\eta$; see [33,34,44].

Theorem 8

(PAC-Bayes oracle inequality). Let $\delta \in (0,1)$ and $\eta >0$. With probability at least $1-\delta$ over the sample, the exact minimizer ${\widehat{P}}_{\eta }\in \mathcal{M}$ of (41) satisfies, simultaneously for every comparator $Q\in \mathcal{M}$,

$$R\left({\widehat{P}}_{\eta }\right)\le R\left(Q\right)+{\displaystyle \frac{2\{D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )\}}{\eta }}+{\displaystyle \frac{\eta }{4n}}.$$

(42)

If the variational objective in (41) is optimized only to accuracy ${\epsilon }_n$, add ${\epsilon }_n$ to the right-hand side. Equivalently, in the original FEM parameterization,

$$R\left({\widehat{P}}_{\beta }\right)\le R\left(Q\right)+{\displaystyle \frac{2\{D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )\}}{\beta n}}+{\displaystyle \frac{\beta }{4}}.$$

(43)

Proof of Theorem 8.

For each given $\sigma$, Hoeffding’s lemma implies

$$\mathbb{E}\exp \left\{\eta (R\left(\sigma \right)-{\widehat{R}}_n\left(\sigma \right))\right\}\le e^{{\eta }^2/\left(8n\right)}.$$

Integrating the preceding exponential-moment bound with respect to the uniform prior u and exchanging the order of expectation gives

$${\mathbb{E}}_{\mathcal{D}}\left[{\mathbb{E}}_{\sigma \sim u}{e}^{\eta (R\left(\sigma \right)-{\widehat{R}}_n\left(\sigma \right))}\right]\le e^{{\eta }^2/\left(8n\right)}.$$

Therefore, by virtue of Markov’s inequality, with probability at least $1-\delta /2$ over the sample,

$${\mathbb{E}}_{\sigma \sim u}{e}^{\eta (R\left(\sigma \right)-{\widehat{R}}_n\left(\sigma \right))}\le {\displaystyle \frac{2}{\delta }}{e}^{{\eta }^2/\left(8n\right)},$$

or equivalently,

$$\log {\mathbb{E}}_{\sigma \sim u}{e}^{\eta (R\left(\sigma \right)-{\widehat{R}}_n\left(\sigma \right))}\le \log (2/\delta )+{\displaystyle \frac{{\eta }^2}{8n}}.$$

On this event, the Donsker–Varadhan change-of-measure inequality [49], applied with

$$f\left(\sigma \right)=\eta (R\left(\sigma \right)-{\widehat{R}}_n\left(\sigma \right)),$$

implies simultaneously for each law Q that

$$\begin{array}{cc}\hfill \eta \{R\left(Q\right)-{\widehat{R}}_n\left(Q\right)\}& \le D_{\mathrm{KL}}(Q\parallel u)+\log {\mathbb{E}}_u{e}^{\eta (R-{\widehat{R}}_n)}\hfill \\ & \le D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )+{\displaystyle \frac{{\eta }^2}{8n}}.\hfill \end{array}$$

Dividing by $\eta$ yields

$$R\left(Q\right)\le {\widehat{R}}_n\left(Q\right)+{\displaystyle \frac{D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )}{\eta }}+{\displaystyle \frac{\eta }{8n}}.$$

Applying the same argument to the reverse deviation ${\widehat{R}}_n\left(\sigma \right)-R\left(\sigma \right)$, we obtain, by Hoeffding’s lemma, for each given $\sigma$,

$${\mathbb{E}}_{\mathcal{D}}\exp \left\{\eta ({\widehat{R}}_n\left(\sigma \right)-R\left(\sigma \right))\right\}\le e^{{\eta }^2/\left(8n\right)}.$$

Integrating with respect to u and applying Markov’s inequality shows that, with probability at least $1-\delta /2$,

$$\log {\mathbb{E}}_{\sigma \sim u}{e}^{\eta ({\widehat{R}}_n\left(\sigma \right)-R\left(\sigma \right))}\le \log (2/\delta )+{\displaystyle \frac{{\eta }^2}{8n}}.$$

On this event, the Donsker–Varadhan inequality implies simultaneously for every law Q that

$${\widehat{R}}_n\left(Q\right)\le R\left(Q\right)+{\displaystyle \frac{D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )}{\eta }}+{\displaystyle \frac{\eta }{8n}}.$$

Let ${\mathcal{E}}_{+}$ and ${\mathcal{E}}_{-}$ be the events on which the forward and reverse bounds hold, respectively. Since each has probability at least $1-\delta /2$, the union bound gives

$$Pr({\mathcal{E}}_{+}\cap {\mathcal{E}}_{-})\ge 1-\delta .$$

Hence, on a single event $\mathcal{E}={\mathcal{E}}_{+}\cap {\mathcal{E}}_{-}$ of probability at least $1-\delta$, both inequalities hold simultaneously for all laws Q.

For such an event $\mathcal{E}$, since the two bounds hold simultaneously for all laws, the forward bound may in particular be applied to the data-dependent minimizer ${\widehat{P}}_{\eta }$. Thus,

$$\begin{array}{cc}\hfill R\left({\widehat{P}}_{\eta }\right)& \le {\widehat{R}}_n\left({\widehat{P}}_{\eta }\right)+{\displaystyle \frac{D_{\mathrm{KL}}({\widehat{P}}_{\eta }\parallel u)+\log (2/\delta )}{\eta }}+{\displaystyle \frac{\eta }{8n}}\hfill \\ \hfill & \le {\widehat{R}}_n\left(Q\right)+{\displaystyle \frac{D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )}{\eta }}+{\displaystyle \frac{\eta }{8n}}\hfill \\ \hfill & \le R\left(Q\right)+{\displaystyle \frac{2\{D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )\}}{\eta }}+{\displaystyle \frac{\eta }{4n}}.\hfill \end{array}$$

If the normalized variational objective in (41) is optimized only to additive accuracy ${\epsilon }_n$, let ${\tilde{P}}_{\eta }$ be the resulting approximate minimizer. Then, for each $Q\in \mathcal{M}$,

$${\widehat{R}}_n\left({\tilde{P}}_{\eta }\right)+{\displaystyle \frac{1}{\eta }}{D}_{\mathrm{KL}}({\tilde{P}}_{\eta }\parallel u)\le {\widehat{R}}_n\left(Q\right)+{\displaystyle \frac{1}{\eta }}{D}_{\mathrm{KL}}(Q\parallel u)+{\epsilon }_n.$$

Thus, the variational-optimality step above acquires an additive ${\epsilon }_n$, and the same argument yields

$$R\left({\tilde{P}}_{\eta }\right)\le R\left(Q\right)+{\displaystyle \frac{2\{D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )\}}{\eta }}+{\displaystyle \frac{\eta }{4n}}+{\epsilon }_n.$$

Finally, in light of the relation $\eta =\beta n$ from (41), one deduces

$${\displaystyle \frac{2\{D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )\}}{\eta }}={\displaystyle \frac{2\{D_{\mathrm{KL}}(Q\parallel u)+\log (2/\delta )\}}{\beta n}},$$

where ${\displaystyle \frac{\eta }{4n}}={\displaystyle \frac{\beta }{4}}$. Hence, Equation (43) is obtained for exact variational minimization, and an additional $+{\epsilon }_n$ appears on the right-hand side for an ${\epsilon }_n$-approximate minimizer of (41). □

Taking $Q={\delta }_{\sigma }$ yields $D_{\mathrm{KL}}(Q\parallel u)=\log M$. Optimizing the bound at

$${\eta }_{★}=\sqrt{8n\{\log M+\log (2/\delta \left)\right\}},{\beta }_{★}={\displaystyle \frac{{\eta }_{★}}{n}}$$

(44)

yields

$$R\left({\widehat{P}}_{{\eta }_{★}}\right)\le \underset{\sigma \in \mathcal{A}}{\min }R\left(\sigma \right)+\sqrt{{\displaystyle \frac{2\{\log M+\log (2/\delta \left)\right\}}{n}}}.$$

(45)

The conclusion differs from deterministic annealing. Sending $\beta$ to infinity is appropriate for minimizing the observed instance energy, whereas the generalization bound favors a finite, sample-size-dependent temperature. If $\log M={\sum }_i\log K_i=O\left(d\right)$, nonvacuous uniform learning requires the data size to dominate the effective combinatorial dimension. This section does not apply when the COP energy has no data-generating interpretation.

6. Representative Models and Practical Guidance

This section specializes the general FEM theory to several celebrated combinatorial optimization frameworks and summarizes its practical implications. The framework is first instantiated for QUBO/Ising energies, Potts MaxCut, and MaxSAT, elucidating how the general fixed-point, stability, and complexity results take concrete forms in these problems. The resulting implications for initialization, annealing, replica use, and solution extraction are then discussed, together with the main limitations of the theoretical guarantees.

6.1. QUBO and Ising Energies

For spins ${\sigma }_i\in \{-1,+1\}$, consider

$$E\left(\sigma \right)=-{\displaystyle \frac{1}{2}}{\sigma }^{\top }W\sigma -b^{\top }\sigma ,W=W^{\top },W_{ii}=0.$$

(46)

By setting $m_i=\mathbb{E}{\sigma }_i$, the product free energy becomes

$$F_{\beta }\left(m\right)=-{\displaystyle \frac{1}{2}}{m}^{\top }Wm-b^{\top }m+{\displaystyle \frac{1}{\beta }}\sum _i\left[{\displaystyle \frac{1+m_i}{2}}\log {\displaystyle \frac{1+m_i}{2}}+{\displaystyle \frac{1-m_i}{2}}\log {\displaystyle \frac{1-m_i}{2}}\right],$$

(47)

and (6) reduces to

$$m_i=\tanh \left(\beta \left[b_i+\sum _j{W}_{ij}{m}_j\right]\right).$$

(48)

At $b=0$, $m=0$ is stationary, and its Hessian is stated by

$${\nabla }^2{F}_{\beta }\left(0\right)={\beta }^{-1}I-W.$$

(49)

If ${\lambda }_{\max }\left(W\right)>0$, the first local instability occurs at ${\beta }_c=1/{\lambda }_{\max }\left(W\right)$. The more conservative global contraction condition is $\beta {\max }_i{\sum }_j\left|W_{ij}\right|<1$. This example distinguishes the spectral local threshold from a row-sum condition certifying global uniqueness.

6.2. Potts MaxCut

For K labels and

$$E\left(\sigma \right)=-\sum _{(i,j)\in \mathcal{E}}{W}_{ij}\{1-\mathbf{1}({\sigma }_i={\sigma }_j)\},$$

the mean energy is formulated by

$$U\left(p\right)=-\sum _{(i,j)\in \mathcal{E}}{W}_{ij}\left(1-{\displaystyle \sum _{a=1}^K}{p}_i\left(a\right)p_j\left(a\right)\right).$$

(50)

Up to an action-independent constant, the conditional energy is presented as

$$e_i(a;p_{-i})=\sum _{j:(i,j)\in \mathcal{E}}{W}_{ij}{p}_j\left(a\right),$$

and hence

$$p_i\left(a\right)\propto \exp \left\{-\beta \sum _{j:(i,j)\in \mathcal{E}}{W}_{ij}{p}_j\left(a\right)\right\}.$$

(51)

The uniform law is stationary by label symmetry, and its loss of stability is determined jointly by the graph spectrum and the $(K-1)$-dimensional label tangent space. Balance penalties introduce dense interactions which enlarge the influence constant C, thus shrinking the certified high-temperature regime. This symmetry also has algorithmic consequences for annealing. If an annealing schedule allocates too many updates to the stable or weakly unstable symmetric regime, a finite-budget optimizer may have insufficient time after the bifurcation to amplify symmetry-breaking perturbations and settle into a favorable branch. This inspires one possible explanation for the lower annealed-FEM success observed for MaxCut in Table 1 compared with the fixed low-temperature initialization.

6.3. MaxSAT

For the unsatisfied-clause energy employed in [25],

$$E\left(\sigma \right)={\displaystyle \sum _{m=1}^L}\prod _{i\in \partial m}\{1-\mathbf{1}({\sigma }_i=w_{mi})\},$$

(52)

the exact product expectation is expressed by

$$U\left(p\right)={\displaystyle \sum _{m=1}^L}\prod _{i\in \partial m}\{1-p_i\left(w_{mi}\right)\}.$$

(53)

For clauses of width at most k, Proposition 5 offers $O\left(Lk{2}^k\right)$ exact arithmetic for each replica and step. The local-optimum result, Proposition 4, is especially relevant because changing one Boolean variable affects only its incident clauses, so the margins and influences can be calculated locally.

6.4. Practical Consequences and Limitations

The theories concluded in the above-mentioned sections suggest the following disciplined interpretation of FEM.

• Initialize uniformly because it is the exact $\beta =0$ optimizer. At positive ${\beta }_{\min }$, Theorem 2 can be employed to determine whether the initialization is genuinely close to the variational optimum.
• Before the estimated critical region, an influence bound or an empirical Hessian can diagnose whether a unique trackable branch is plausible. In this region, slow annealing has the quantitative meaning in Theorem 6.
• Beyond the first instability, replicas explore different basins but do not change the mean-field family. Report the best energy together with the number of replicas and initializations, but do not interpret the batch as a correlated variational distribution.
• Evaluate coordinatewise modes and random samples at termination, and the two extraction approaches have different guarantees in Theorem 5.
• If a certified lower bound $L\le E_{★}$ is available from a relaxation, any returned law has the computable certificate

  $$U\left(\widehat{p}\right)-E_{★}\le F_T\left(\widehat{p}\right)-L+T\log M,$$

  without requiring the global FEM optimum.

Several limitations are noteworthy, on the contrary. First, as Corollary 2 explicitly portrayed, reverse-KL mean field cannot represent correlated mixtures. Second, the global low-temperature rates do not mean that a specific optimizer admits the global variational solution. Third, the Dobrushin bound is conservative and may fail well before an empirical phase transition. Fourth, the PAC-Bayes analysis assumes bounded i.i.d. losses; dependent data, heavy-tailed losses, or a deterministic COP require different tools. Finally, the gradient manipulations used in implementations can depart from the exact gradient (7), and such modifications should be analyzed as a separate algorithm rather than attributed automatically to the free-energy objective.

7. Discussion

The central benefit of entropy in FEM is mathematically transparent. At high temperature it selects a unique, nearly uniform and often contractive solution, allowing broad exploration. Lowering the temperature decreases the energy of a globally optimized solution, but also weakens convexification, creates branches near discrete local optima, and magnifies numerical energy errors. At zero temperature the global variational relaxation is exact in objective value because the product family contains each point mass, yet the route to that endpoint remains computationally hard.

The three thermal regimes analyzed here should not be interpreted as a newly proposed phenomenological picture of FEM annealing. Shen et al. [25] already described, through the evolution of entropy, internal energy, and marginal probabilities, a qualitative three-stage process in which entropy favors nearly uniform marginals at high temperature, a distributional transition occurs near a critical inverse temperature, and energy dominates as the temperature approaches zero. The present contribution provides a quantitative characterization of this pipeline for the finite-state product free-energy objective. Specifically, Theorem 2 bounds the high-temperature deviation from uniformity in relative entropy and total variation and identifies the first-order departure from the uniform law; Theorem 3 and Proposition 2 unveil sufficient uniqueness/contraction conditions and an instance-dependent Hessian criterion for the first possible loss of stability; and Theorem 4 displays the explicit low-temperature excess-energy bound $U\left(p_{\beta }\right)-E_{★}\le \log \left|\mathcal{A}\right|/\beta$, together with ground-state concentration and entropy selection under degeneracy. Corollary 3 then organizes these finite-dimensional conclusions into asymptotic entropy-dominated, model-specific bifurcation, and energy-dominated regimes. Thus, three-regime analysis in this work is intended as a rigorous quantitative refinement of the qualitative annealing mechanism proposed in [25], rather than as the first observation of that mechanism.

Several research directions are listed from the analysis. The influence bound could be sharpened by weighted norms or nonbacktracking operators, producing less conservative uniqueness regions for sparse graphs. Below the first bifurcation, branch-specific curvature and barrier estimates may infer invaluable finite-rate annealing guarantees. Structured variational families, TAP or Bethe corrections, and learned autoregressive laws could relieve the rectangular-support limitation while preserving part of FEM’s parallelism [23,50,51]. Moreover, a theory that combines data-dependent generalization, instance perturbations, and nonconvex optimizer dynamics would more closely match large-scale applications. Accordingly, the main novelty is not the individual classical tools, but the unified FEM-specific analysis that separates entropy effects, annealing behavior, numerical optimization, replica selection, and statistical regularization within a single finite-state framework. Finally, a deep extension would replace the uniform reference law by a dynamically updated $Q_t$, yielding $U\left(p\right)+{\beta }^{-1}{D}_{\mathrm{KL}}(P_p\parallel Q_t)$, which encodes information accumulated during optimization and connects FEM more directly to adaptive biased ensembles, but its convergence and stability require an exclusive analysis because the objective itself becomes history dependent.