Skip to Content
You are currently on the new version of our website. Access the old version .
EntropyEntropy
  • Feature Paper
  • Article
  • Open Access

2 March 2026

14 Pages

On Maximum Entropy Density Estimation with Relaxed Moment Constraints

and
1
Informatics Department, Thuongmai University, 79 Ho Tung Mau Street, Cau Giay District, Hanoi 100000, Vietnam
2
Institut de Mathématiques de Toulouse, Université de Toulouse, 118 Route de Narbonne, 31062 Toulouse, Cedex 9, France
3
Équipe OPTIM-Optimisation, École Nationale de l’Aviation Civile, 7 Avenue Édouard Belin, 31055 Toulouse, Cedex 4, France
*
Author to whom correspondence should be addressed.

Abstract

We study Maximum Entropy density estimation on continuous domains under finitely many moment constraints, formulated as the minimization of the Kullback–Leibler divergence with respect to a reference measure. To model uncertainty in empirical moments, constraints are relaxed through convex penalty functions, leading to an infinite-dimensional convex optimization problem over probability densities. The main contribution of this work is a rigorous convex-analytic treatment of such relaxed Maximum Entropy problems in a functional setting, without discretization or smoothness assumptions on the density. Using convex integral functionals and an extension of Fenchel duality, we show that, under mild and explicit qualification conditions, the infinite-dimensional primal problem admits a dual formulation involving only finitely many variables. This reduction can be interpreted as a continuous-domain instance of partially finite convex programming. The resulting dual problem yields explicit primal–dual optimality conditions and characterizes Maximum Entropy solutions in exponential form. The proposed framework unifies exact and relaxed moment constraints, including box and quadratic relaxations, within a single variational formulation, and provides a mathematically sound foundation for relaxed Maximum Entropy methods previously studied mainly in finite or discrete settings. A brief numerical illustration demonstrates the practical tractability of the approach.

1. Introduction

The Maximum Entropy principle provides a classical framework for estimating probability distributions from partial information, typically in the form of moment constraints. In its standard formulation, the problem is posed on a finite or discrete sample space and reduces to a finite-dimensional convex optimization problem, which has been extensively studied in statistics and machine learning.
In many situations of interest, however, the underlying sample space is continuous and no natural discretization is available. Estimating a probability density on a continuous domain under moment constraints then leads to an infinite-dimensional optimization problem, for which the usual finite-dimensional duality arguments no longer apply directly.
Such continuous-domain formulations arise naturally in a variety of applications, including density estimation for physical or environmental variables (e.g., wind or ocean directions, spatial intensity fields), inverse problems and signal processing, and nonparametric statistical modeling where the unknown distribution is supported on a continuum. In directional and circular statistics, for instance, probability densities on the unit circle or sphere are routinely modeled without discretization [1,2]. In these settings, discretization may introduce bias or artificial constraints, whereas a genuine functional formulation is often preferable.
Continuous or nonparametric variants of the Maximum Entropy principle have also been studied in the literature, with foundational contributions by Csiszár on I-divergence geometry and entropy-based inference under exact linear constraints [3,4]. While these works exploit convexity and information-theoretic inequalities to enforce strict moment conditions in a rigorous manner, other continuous-domain treatments proceed in a more formal manner. In many such approaches, the entropy functional and the moment constraints are written in a continuous setting and optimal densities are derived by variational arguments or by analogy with the discrete case. Although these derivations often lead to plausible expressions, the precise functional-analytic framework and the underlying duality structure are frequently left implicit or treated only at a formal level.
Classical treatments of the Maximum Entropy principle in information theory and statistical physics rely on the formal Lagrange multiplier arguments and variational calculus [5,6,7]. In practice, continuous problems are often handled either through discretization or binning strategies that yield finite-dimensional approximations, or via parametric exponential-family models. While frequently effective, such approaches depend on approximation choices that may influence stability and accuracy.
From a convex-analytic perspective, entropy minimization can be formulated within the theory of convex integral functionals [8]. In the discrete setting, Dudík et al. [9] developed a systematic framework for Maximum Entropy estimation with relaxed moment constraints. The present work extends this line of research to genuinely continuous domains, where the primal variable is an infinite-dimensional density.
The purpose of this paper is to provide a fully rigorous formulation of the continuous-domain Maximum Entropy problem. We consider the minimization of the Kullback–Leibler divergence with respect to a reference measure under finitely many (possibly relaxed) moment constraints, where the optimization variable is a probability density defined on a continuous domain. The primary problem is genuinely infinite-dimensional and is formulated in a space of measurable functions defined by integrability with respect to a prescribed family of feature functions.
More precisely, the functional framework is built upon a dual pair of Köthe-type spaces associated with the chosen feature family, allowing moment constraints to be well defined. This setting accommodates a wide range of feature functions, including unbounded ones, and provides a natural domain for entropy optimization.
Our main contribution is to show that, under explicit and mild assumptions ensuring well-posedness of the moment operators and nonemptiness of the relevant effective domains, this infinite-dimensional convex problem admits a dual formulation involving only finitely many variables. The analysis relies on convex integral functionals and an extension of Fenchel duality, and yields precise primal–dual optimality conditions.
The proposed framework is closely related in spirit to the discrete relaxed Maximum Entropy approach of Dudík et al. [9]. It can be viewed as its continuous-domain counterpart, in which the finite-dimensional structure of the dual problem is preserved while the primal variable becomes an infinite-dimensional density.
Since empirical moments are computed from finite samples, they are subject to sampling variability and measurement noise. Enforcing them as exact constraints may therefore lead to overfitting or even infeasibility. This observation motivates the introduction of relaxed constraints through convex penalty functions, which provide a principled way to incorporate uncertainty while retaining the variational structure of the Maximum Entropy principle.
The paper is organized as follows. Section 2 formulates the continuous Maximum Entropy problem and introduces relaxed moment constraints. Section 3 derives the dual problem and establishes primal–dual optimality conditions. Section 4 discusses several choices of relaxation potentials and their consequences for the dual formulation. A brief numerical illustration is presented in Section 5, and concluding remarks are provided in Section 6.

2. Statement of the Problem

We consider the optimization of the so-called differential entropy or its relative counterpart, the Kullback–Leibler divergence between probability measures.
Let Ω ⊂ R d be a bounded domain. We assume throughout that Ω is endowed with the σ -algebra of Borel sets. The space of Borelian functions on Ω is denoted by Bor ( Ω ) .
Points x 1 , … , x n are observed in Ω . These points are assumed to be drawn according to some unknown probability P on Ω , and the fundamental problem is that of recovering P. Feature functions f 1 , … , f m :   Ω → R are chosen, from which the following statistics are derived:
f k = 1 n ∑ j = 1 n f k ( x j ) , k = 1 , … , m .
The functions f 1 , … , f m , also called moment functions, define the moment constraints of the problem.
Recall that the above equation expresses the expectation of f k under the empirical measure
P emp = 1 n ∑ j = 1 n δ x j ,
in which δ x denotes the Dirac measure at x . The Maximum Entropy principle states, in its native form, that among all probability measures P that are consistent with the moment constraints
f k = ∫ f k d P , k = 1 , … , m ,
one should select the probability P ¯ , which minimizes the Kullback–Leibler relative entropy K ( P ∥ P ∘ ) of P with respect to P ∘ . Here, P ∘ is a reference measure, supported in Ω , which encodes the prior knowledge one may have about P. If no prior knowledge is available, it is customary to choose P ∘ to be the uniform measure on Ω . Recall that the Kullback–Leibler relative entropy of P with respect to P ∘ is defined as
K ( P ∥ P ∘ ) : = ∫ ln d P d P ∘ d P if P ≺ ≺ P ∘ , ∞ otherwise .
Here, d P / d P ∘ denotes the Radon–Nikodym derivative of P with respect to P ∘ , and the notation P ≺ ≺ P ∘ means that P is absolutely continuous with respect to P ∘ . We are facing the following optimization problem:
( P ∘ ) Minimize K ( P ∥ P ∘ ) s . t . 1 = ∫ Ω d P , y = ∫ Ω f d P .
The shorthand s.t. stands for subject to, while y = ( y 1 , … , y m ) ∈ R m is the vector whose k-th component equals f k and f is the R m -valued function whose k-th component equals f k .
Problem ( P ∘ ) is an infinite-dimensional convex problem with finitely many linear constraints. Although the optimized variable P lies in the space of Radon measures on Ω , it is equivalent to searching for a density with respect to P ∘ . As a matter of fact, minimizing the Kullback–Leibler divergence discards measures that are not absolutely continuous with respect to P ∘ .
Since empirical moments are computed from finite samples, they are subject to sampling variability and measurement noise. Enforcing them as exact constraints may therefore lead to overfitting or infeasible formulations; this motivates the introduction of relaxed constraints. Following Ref. [9], we introduce, for the purpose of relaxation, a convex potential function  U : R m → ( − ∞ , ∞ ] .
We therefore obtain the relaxed, functional counterpart of Problem ( P ∘ ) , namely
( P ) Minimize H ( p ) + U ( F p ) s . t . 1 = I p ,
in which the variable p is the Radon–Nikodym derivative of the searched measure P with respect to P ∘ . The neg-entropy functional H is defined by
H ( p ) : = ∫ h ∘ p ( x ) d P ∘ ( x ) with h ∘ ( t ) : = t ln t if t > 0 , 0 if t = 0 , ∞ if t < 0
and the linear operators F and I are, respectively, defined by
F p = ∫ Ω f ( x ) p ( x ) d P ∘ ( x ) and I p = ∫ Ω p ( x ) d P ∘ ( x ) .
For convenience, we group these linear operators into a single moment operator
A : p ↦ ( I p , F p ) ∈ R × R m ,
which collects the normalization and moment constraints. This notation will be used throughout the sequel. The potential function U is generally a convex non-negative function, which attains its minimum value at y . The case of non-relaxed constraint [9] corresponds to the choice U y ( s ) = U y ( 0 ) ( s ) = δ s | { y } . Recall that the indicator function of a set S is the function
δ s |   S : = 0 if s ∈ S , ∞ otherwise .
Clearly, this choice enforces the equality A p = y . In [9], the case of box constraints was also considered. It corresponds to the choice U y ( s ) = U y ( 1 ) ( s ) = δ s | B y , in which B is box
B y = ∏ k = 1 m [ y k − β k , y k + β k ]
with β = ( β 1 , … , β m ) ∈ ( R + ∗ ) . In this finite-dimensional setting, the convex conjugate of δ · | B y coincides with the support function of the set B y , namely
δ · | B y ★ ( σ ) = sup s ∈ B y σ , s ,
where the pairing is the standard Euclidean inner product. Since the convex conjugate of δ · | B y fails to be differentiable, one may consider an approximation of the box constraint that yields a differentiable conjugate, as in Section 5.2 of [9]. This will also be considered in the present paper. The least square relaxation corresponds to the choice
U y ( s ) = U y ( 2 ) ( s ) = 1 2 α ∥ y − s ∥ 2 ,
in which ∥ · ∥ denotes the Euclidean norm and α is some positive constant. We may also consider, more generally, the choice
U ( s ) = 1 2 α y − s Q 2
in which ∥ z ∥ Q 2 = z , z Q = z , Q z where Q is a positive definite symmetric matrix. An interesting example of this is obtained if we let Q = Σ , with Σ an estimate of the covariance matrix of the vector y : the relaxation then accounts for the variance of each principal component of the data vector regarded as a random vector.
Implicitly, the above problem is posed in a space of measurable functions for which all moment constraints are well defined. More precisely, the natural domain of the optimization problem consists of functions that are integrable against the chosen family of feature (moment) functions.
This space is a subspace of L P ∘ 1 ( Ω ) , the space of measurable functions that are integrable with respect to the reference measure P ∘ , and coincides with L P ∘ 1 ( Ω ) when all feature functions are bounded. In general, however, it is a proper subspace, since the moment operator is only well defined for densities p such that the pointwise products p f k , defined by ( p f k ) ( x ) = p ( x ) f k ( x ) , belong to L P ∘ 1 ( Ω ) for every k ∈ { 1 , … , m } .
More generally, for a family of feature functions Φ , the natural workspace of the moment problem is the Köthe dual space
L P ∘ ( Ω ; Φ ) : = p ∈ Bor ( Ω ) | ∀ f ∈ Φ , p f ∈ L P ∘ 1 ( Ω ) .
In the next section, we shall explore the dualization of problem ( P ) and show how this dualization provides algorithms for effective computation of Maximum Entropy solutions in our infinite-dimensional setting, without having to discretize the statistical problem under consideration.

3. Maximum Entropy Solutions

This section derives the dual formulation of the relaxed Maximum Entropy problem. Although the primary variable is an infinite-dimensional density, convex duality allows the problem to be reduced to the optimization of finitely many dual variables (Lagrange multipliers). The analysis relies on standard tools from convex analysis—in particular, convex conjugates, subdifferentials, and Fenchel duality—which provide a natural framework for establishing duality and optimality conditions. Standard references include [10,11,12], while results on convex integral functionals can be found in [8,13,14]. A recent overview in the context of entropy optimization is provided in [15].
For convenience, we define on R × R m the concave function
G ( η ∘ , η ) : = − δ η ∘ | { 1 } − U ( η )
and rewrite ( P ) in the equivalent form
( P ) Minimize ( H − G ∘ A ) ( p ) s . t . p ∈ L P ∘ ( Ω ; Φ )
in which A p : = ( I p , F p ) . Here, G ∘ A denotes the composition of functions, that is, ( G ∘ A ) ( p ) = G ( A p ) . We shall also write U ( η ) = V ( y − η ) with V a proper convex lower semi-continuous (and most often even) function. We note that the dual problem associated with ( P ) reads
( D ) Maximize D ( λ ∘ , λ ) : = ( G ★ − H ★ ∘ A ★ ) ( λ ∘ , λ ) s . t . ( λ ∘ , λ ) ∈ R × R m .
The symbol ★ can denote different, though related, mathematical notions depending on context. When used as a superscript on a function, as in H ★ , it typically refers to the convex conjugate (or Legendre–Fenchel transform) of H. When used as a subscript, as in G ★ , it denotes the concave conjugate. In the context of linear operators, the same symbol used as a superscript, as in A ★ , denotes the adjoint of the operator A .
Let us clarify the notion of adjunction being referred to here: in our context, the operator A is well-defined from the space L P ∘ ( Ω ; Φ ) with Φ = 1 Ω , f 1 , … , f m to R 1 + m , and its adjoint is defined with respect to the pairing of L P ∘ ( Ω ; Φ ) with
L P ∘ ★ ( Ω ; Φ ) : = φ ∈ Bor ( Ω ) | ∀ p ∈ L P ∘ ( Ω ; Φ ) , p φ ∈ L P ∘ 1 ( Ω )
realized via the standard integral bilinear product
p , ϕ : = ∫ Ω p ( x ) ϕ ( x ) d P ∘ ( x ) .
An immediate computation shows that the adjoint of A is then given by
A ★ : R 1 + m ⟶ L P ∘ ★ ( Ω ; Φ ) ( λ ∘ , λ ) ⟼ λ ∘ + λ , f .
We use the notation ri ( C ) for the relative interior of a convex set C, that is, the interior taken with respect to the affine hull of C (see [10]).
Theorem 1.
Assume That The Following Conditions Hold:
(i)
for every k ∈ { 1 , … , m } , the function f k is integrable over every set of finite measure;
(ii)
ri ( A dom H ) ∩ ri ( dom G ) ≠ ∅ ;
(iii)
ri dom G ★ ∩ ri dom ( H ★ ∘ A ★ ) ≠ ∅ ;
(iv)
the function V underlying the definition of G is closed and proper convex.
  • If ( λ ¯ ∘ , λ ¯ ) maximizes the dual function D, then the density p ¯ defined on Ω by
p ¯ ( x ) : = ( h ∘ ★ ) ′ [ A ★ ( λ ¯ ∘ , λ ¯ ) ] ( x ) = exp ( λ ¯ ∘ − 1 ) · exp λ ¯ , f ( x )
is optimal in problem ( P ) .
Proof. 
Step 1. By Fenchel duality, thanks to Condition (ii), the optimal value v ( P ) of Problem ( P ) coincides with that of the dual problem ( D ) , and dual attainment is achieved:
v ( P ) = max D ( λ ∘ , λ ) | ( λ ∘ , λ ) ∈ R 1 + m .
See, e.g., Theorem 4 of [15].
Step 2. We now turn to the computation of the dual function. We start by computing the concave conjugate of G:
G ★ ( λ ∘ , λ ) : = inf ( λ ∘ , λ ) , ( η ∘ , η ) − G ( η ∘ , η ) | ( η ∘ , η ) ∈ R 1 + m = inf λ ∘ η ∘ + λ , η + δ η ∘ | { 1 } + U ( η ) | ( η ∘ , η ) ∈ R 1 + m = λ ∘ + inf λ , η + V ( y − η ) | η ∈ R m = λ ∘ − sup − λ , η − V ( y − η ) | η ∈ R m = λ ∘ + y , λ − V ★ ( λ ) ,
in which the last equality is obtained by the change of variable η ′ = y − η . Let us now compute the convex conjugate of H in the pairing (4). Condition (i) implies, via Lemma 1 and Proposition 1 [16], that both L P ∘ ( Ω ; Φ ) and L P ∘ ★ ( Ω ; Φ ) are decomposable. Moreover, the entropy kernel h ∘ ( t ) is a measurable integrand that is proper convex and lower semi-continuous. Its convex conjugate is provided by
h ∘ ★ ( τ ) = exp ( τ − 1 ) , τ ∈ R .
Since L P ∘ ( Ω ; Φ ) is decomposable and H has a nonempty effective domain, it follows that, for every φ ∈ L P ∘ ★ ( Ω ; Φ ) ,
H ★ ( φ ) = ∫ Ω exp φ ( x ) − 1 d P ∘ ( x ) .
Otherwise expressed, conjugacy through the integral sign holds. Now, h ∘ ★ ★ = h ∘ since h ∘ is proper convex and lower semi-continuous. Moreover, it is readily seen that
L P ∘ ★ ★ ( Ω ; Φ ) : = p ∈ Bor ( Ω ) | ∀ φ ∈ L P ∘ ★ ( Ω ; Φ ) , p φ ∈ L P ∘ 1 ( Ω ) = L P ∘ ( Ω ; Φ ) .
See Property (f) of [16] (p. 207). It follows that conjugacy through the integral sign applies one more time: for all p ∈ L P ∘ ( Ω ; Φ ) ,
H ★ ★ ( p ) = ∫ Ω h ∘ ★ ★ p ( x ) d P ∘ ( x ) = H ( p ) .
Step 3. We shall prove here that the dual solution satisfies
(a)
p ¯ ∈ ∂ H ★ ( A ★ ( λ ¯ ∘ , λ ¯ ) ) ;
(b)
H ★ ∘ A ★ has gradient A p ¯ at ( λ ¯ ∘ , λ ¯ ) .
The conclusion then follows from Theorem 6 of [15], since H ★ ★ = H (see Equation (6)) and G ★ ★ = G under Assumption (iv). Clearly, h ∘ is differentiable throughout R , and its derivative equals itself. By integrating the subdifferential inequality as in Proposition 6 of [15], for example, we see that the obvious inclusion ( h ∘ ★ ) ′ ( τ ) ∈ ∂ h ∘ ★ ( τ ) gives rise to the following inclusion:
( h ∘ ★ ) ′ ∘ φ ∈ ∂ H ★ ( φ )
for every φ ∈ L P ∘ ★ ( Ω ; Φ ) , thanks to Equation (5). Replacing φ by A ★ ( λ ¯ ∘ , λ ¯ ) yields (a). Finally, differentiating
( H ★ ∘ A ★ ) ( λ ∘ , λ ) = ∫ Ω exp λ ∘ + λ , f ( x ) d x
through the integral sign shows that
∇ ( H ★ ∘ A ★ ) ( λ ¯ ∘ , λ ¯ ) = A p ¯ .
Thus, (b) is also satisfied, and the proof is complete. □
As a byproduct of the above theorem, we see that the dual problem corresponding to Problem ( P ) consists of maximizing the function
D ( λ ∘ , λ ) : = λ ∘ + y , λ − V ★ ( λ ) − exp ( λ ∘ − 1 ) ∫ Ω exp λ , f ( x ) d P ∘ ( x ) .
By the first-order optimality condition for concave maximization (often referred to as Fermat’s principle; see [10]), we find that ( λ ¯ ∘ , λ ¯ ) is dual optimal if and only if
( 0 , 0 ) ∈ ∂ D ( λ ¯ ∘ , λ ¯ ) .
Here, the subdifferential is taken in the sense of concavity. Recall that the dual function is concave; thus, sufficiency is granted along with necessity. It is readily checked that the latter condition is equivalent to
exp ( λ ¯ ∘ − 1 ) = ∫ Ω exp λ ¯ , f ( x ) d x − 1 and 0 ∈ ∂ D ˜ ( λ ¯ ) ,
in which
D ˜ ( λ ) : = λ , y − V ★ ( λ ) − ln ∫ Ω exp λ , f ( x ) d x .
Therefore, maximizing the dual function in (7) boils down to maximizing the function D ˜ . We observe that D ˜ is concave and upper semi-continuous, as a consequence of the convexity and lower semi-continuity of the log-Laplace transform. We note that evaluating D ˜ ( λ ) relies on some numerical integration in dimension m. This is by no means a limitation, due to the existence of powerful numerical integration algorithms such as Monte Carlo integration, for example.
Let us summarize the above achievements. If the assumptions of Theorem 1 are satisfied, which clearly depend on the choice of the potential V and on that of the feature functions f 1 , … , f m , we compute the optimal vector λ ¯ by maximizing D ˜ , and we then obtain the Maximum Entropy solution p ¯ via the formula
p ¯ ( x ) = exp λ ¯ , f ( x ) ∫ Ω exp λ ¯ , f ( x ) d x , x ∈ Ω .
Some potentials V give rise to nice, differentiable, concave-unconstrained maximization problems. This happens whenever the conjugate V ★ has a full domain and is differentiable. A standard example is provided by the least square relaxation. However, in general, the function V ★ may suffer from drawbacks such as failing to have a full domain (so that the dual problem is constrained) or being non-differentiable. A standard example is provided by the case of box constraints.
In the smooth unconstrained case, quasi-Newtonian methods, such as the BFGS algorithm, are expected to provide a fast and accurate solution. If V ★ is not differentiable, the situation can still be addressed using the proximal gradient algorithm. As a matter of fact, in the latter case, the function to be maximized is then a composite concave function, that is to say, a sum of two concave functions, one being differentiable, the other being non-differentiable. The reader interested in the optimization of composite functions and the proximal gradient algorithm may refer to [17,18] and the references therein.

4. Relaxation

Following Ref. [9], we explore a number of potential functions to be used for the reconstruction of multivariate distributions. We assume throughout that the domain of H ★ ∘ A ★ is nonempty—a mild assumption on the family of feature functions.
No relaxation at all. In this case, V ( η ) = δ η | { 0 } . The convex conjugate V ★ ( λ ) is then the function identically equal to zero. Condition ( Q C ) reads y ∈ ri ( F dom H ) in this case, and Condition ( Q C ★ ) is automatically satisfied. The reason is that the effective domain of G ★ is then the whole of R m , and that ri dom ( H ★ ∘ A ★ ) is nonempty, since the relative interior of a nonempty convex set is nonempty.
Box constraints. The function V is defined in this case by
V ( η ) = δ η | ∏ k = 1 n [ − β k , β k ] = ∑ k = 1 n δ η k | [ − β k , β k ] .
Its convex conjugate is then given by
V ★ ( λ ) = ∑ k = 1 n β k | λ k | = β ⊙ λ | 1 ,
in which ⊙ denotes the pointwise product. In this case, Condition ( Q C ) reads
∏ k = 1 m ( y k − β k , y k + β k ) ⋂ ri ( F dom H ) ≠ ∅ .
Condition ( Q C ★ ) is, again, automatically satisfied. Since the corresponding function D ˜ is a composite function, the maximization of D ˜ will be efficiently performed by the proximal gradient algorithm.
Smoothed inner approximation of box constraints. Let ψ β : R → ( − ∞ , ∞ ] be defined by
ψ β ( η ) = η 2 ln β + η β − η + β ln 1 − η 2 β 2 if η ∈ ( − β , β ) , ∞ otherwise .
The smooth inner approximation of the box constraints with the same box as in the previous paragraph, together with its convex conjugate, are given, respectively, by
V ( η ) = ∑ k = 1 m ε k ψ β k ( η k ) and V ★ ( λ ) = ∑ k = 1 m β k ε k ln ch λ k ε k ,
in which ε k controls the proximity of V ( η ) to the target box constraint potential and, thereby, the proximity of the corresponding conjugate functions. Figure 1 illustrates the approximation of the box constraint potential and its conjugate. We note that the constraint qualification condition and its dual counterpart are exactly as in the box constraint case. The advantage here is that the dual function is differentiable, allowing the use of powerful quasi-Newtonian optimization methods.
Figure 1. Plots of ε ψ β ( η ) and its conjugate for β = 1 and various values of ε .
Least squares. Given a symmetric positive definite matrix Q, the weighted least squares relaxation corresponds to the case where
V ( η ) = 1 2 α η Q − 1 2 and V ★ ( λ ) = α 2 η Q 2 .
The standard least squares relaxation is obtained by letting Q = I , the identity matrix. If an empirical covariance matrix Σ of the moment vector is available, one may choose Q = Σ . The relaxation term then involves the quadratic form η ⊤ Σ − 1 η , i.e., a Mahalanobis-type distance. In the eigenbasis of Σ , this amounts to weighting each principal component inversely to its variance, so that directions with larger uncertainty are penalized less strongly.
Remarkably, things once again look nicer on the dual side: no matrix inversion is required to evaluate D ˜ ( λ ) . Finally, the constraint qualification and its dual counterpart are automatically fulfilled here.

5. Experimental Illustration

This section provides a brief numerical validation of the proposed continuous-domain Maximum Entropy formulation. The goal is not to perform an extensive empirical benchmark, but to illustrate that the dual approach yields stable and meaningful density reconstructions without discretizing the domain. We consider both synthetic data and a real-world meteorological dataset.

5.1. Synthetic Angular Densities

We first consider synthetic directional data on the unit circle, a classical setting in circular statistics. Three target densities are used: a unimodal von Mises distribution, a symmetric bimodal mixture, and a trimodal mixture. For each case, n = 2000 samples are generated.
Densities are reconstructed using trigonometric feature functions
φ k ( θ ) = ( cos ( k θ ) , sin ( k θ ) ) , k = 1 , … , K ,
with K ∈ { 1 , 2 , 4 , 8 } . A quadratic relaxation is employed, and the resulting finite-dimensional dual problem is solved using a quasi-Newton (BFGS) method. Normalization constants are evaluated by numerical quadrature.
Representative reconstructions are shown in Figure 2. In all cases, the Maximum Entropy solution correctly captures the qualitative structure of the target distribution, including multimodality, without any discretization of the domain. The regularization parameter α controls smoothness: small values preserve sharp features, whereas larger values produce smoother densities.
Figure 2. Reconstructed MaxEnt angular densities for unimodal, bimodal, and trimodal distributions under varying regularization strengths α ∈ { 0.01 , 0.1 , 1.0 } . Each subplot shows the true density (black solid), reconstructed density (red dashed), and sampled histogram (blue). Smaller α values preserve sharper peaks and fine directional structure, while larger α values yield smoother but less distinct reconstructions.

5.2. Wind Direction Reconstruction

We next evaluate the method on a real meteorological dataset consisting of aircraft-based wind measurements over Europe (January 2018) [19]. The main recorded variables relevant to our study are summarized in Table 1.
Table 1. Meteorological observation attributes used in the wind direction reconstruction.
The data include horizontal wind components ( w x , w y ) , from which wind directions are computed as
θ = mod ( atan 2 ( w y , w x ) , 2 π ) .
A random subset of N = 10 4 observations is used.
As a reference, we compare the proposed Maximum Entropy estimator with a circular kernel density estimate (KDE). Reconstruction quality is assessed using the L 2 distance, the Kullback–Leibler divergence, and held-out log-likelihood.
Figure 3, Figure 4 and Figure 5 show representative results for increasing harmonic orders K and different regularization strengths α . With few harmonics ( K = 2 ), the method already captures the dominant wind regimes. Increasing K improves fidelity, while moderate regularization prevents oscillations. A very small α may introduce mild overfitting. Quantitative results in Table 2 confirm that intermediate values (K = 4–6, α = 10 − 2 ) provide the best trade-off between accuracy and smoothness.
Figure 3. Reconstructed wind direction distribution using MaxEnt with K = 2 , α = 10 − 2 . The MaxEnt estimate (red dashed) accurately captures the two dominant wind regimes while maintaining smoothness and stability.
Figure 4. Reconstructed wind direction distribution using MaxEnt with K = 6 , α = 10 − 2 . The model achieves improved alignment with the KDE reference while preserving numerical stability and smooth transitions.
Figure 5. Reconstructed wind direction distribution using MaxEnt with K = 12 , α = 10 − 3 . Increasing model complexity allows for finer detail but may induce small oscillations near modal peaks.
Table 2. Quantitative comparison between MaxEnt and KDE on wind direction data.
Overall, these experiments illustrate that the proposed continuous-domain formulation produces stable and interpretable density estimates while avoiding discretization or kernel bandwidth tuning.

6. Conclusions

This paper developed a continuous-domain formulation of the Maximum Entropy principle for multivariate density estimation under moment constraints. By combining convex duality with the theory of convex integral functionals, we showed that an infinite-dimensional entropy minimization problem over probability densities admits a finite-dimensional dual representation. This result provides a rigorous and tractable foundation for Maximum Entropy estimation without discretizing the underlying domain.
The introduction of relaxed moment constraints through convex potential functions allows for a unified treatment of exact constraints, box constraints, and quadratic relaxations. On the dual side, this leads to concave optimization problems whose analytical and numerical properties depend explicitly on the chosen relaxation while remaining amenable to standard optimization algorithms.
From a conceptual standpoint, the proposed framework bridges the gap between classical discrete Maximum Entropy formulations and genuinely nonparametric continuous models. It preserves the interpretability and structural simplicity of the entropy principle while extending its applicability to settings where discretization may introduce bias or instability.
The selection of appropriate feature functions remains a central issue in continuous-domain Maximum Entropy modeling and deserves further investigation.
Future work will focus on principled choices of feature families and on extensions to structured domains, including spatial, physically informed, or space–time settings with partial observations.

Author Contributions

Conceptualization, P.M.; methodology, P.M. and T.L.N.; validation, T.L.N.; formal analysis, P.M.; investigation, P.M.; data curation, T.L.N.; writing—original draft preparation, P.M. and T.L.N.; writing—review and editing, P.M. and T.L.N.; visualization, T.L.N.; supervision, P.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

The data analyzed in this study are publicly available on Figshare at https://doi.org/10.6084/m9.figshare.6970403.v1 [19].

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Fisher, N.I. Statistical Analysis of Circular Data; Cambridge University Press: Cambridge, UK, 1995. [Google Scholar]
  2. Pewsey, A.; Neuhauser, M.; Ruxton, G. Circular Statistics in R; Oxford University Press: Oxford, UK, 2013. [Google Scholar]
  3. Csiszár, I. I-divergence geometry of probability distributions and minimization problems. Ann. Probab. 1975, 3, 146–158. [Google Scholar] [CrossRef] [Scilit]
  4. Csiszár, I. Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems. Ann. Stat. 1991, 19, 2032–2066. [Google Scholar] [CrossRef] [Scilit]
  5. Jaynes, E.T. Information Theory and Statistical Mechanics. Phys. Rev. 1957, 106, 620–630. [Google Scholar] [CrossRef] [Scilit]
  6. Jaynes, E.T. Probability Theory: The Logic of Science; Cambridge University Press: Cambridge, UK, 2003. [Google Scholar]
  7. Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; John Wiley & Sons: Hoboken, NJ, USA, 2006. [Google Scholar]
  8. Rockafellar, R.T. Convex integral functionals and duality. In Contributions to Nonlinear Functional Analysis; Academic Press: Cambridge, MA, USA, 1971; pp. 215–236. [Google Scholar] [CrossRef] [Scilit]
  9. Dudík, M.; Phillips, S.J.; Schapire, R.E. Maximum entropy density estimation with generalized regularization and an application to species distribution modeling. J. Mach. Learn. Res. 2007, 8, 1217–1260. [Google Scholar]
  10. Rockafellar, R.T. Convex Analysis; Princeton University Press: Princeton, NJ, USA, 1970; Volume 18. [Google Scholar]
  11. Hiriart-Urruty, J.B.; Lemaréchal, C. Fundamentals of Convex Analysis; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2004. [Google Scholar]
  12. Borwein, J.M.; Lewis, A.S. Convex Analysis and Nonlinear Optimization; Springer: Berlin/Heidelberg, Germany, 2006. [Google Scholar]
  13. Rockafellar, R.T. Integrals which are convex functionals. Pac. J. Math. 1968, 24, 525–539. [Google Scholar] [CrossRef] [Scilit]
  14. Rockafellar, R.T. Integral functionals, normal integrands and measurable selections. In Nonlinear Operators and the Calculus of Variations: Summer School Held in Bruxelles September 1975; Springer: Berlin/Heidelberg, Germany, 2006; pp. 157–207. [Google Scholar] [CrossRef] [Scilit]
  15. Maréchal, P. Entropy Optimization. In Convex Optimization. Theory, Algorithms and Applications; Springer: Berlin/Heidelberg, Germany, 2025; pp. 489–523. [Google Scholar] [CrossRef] [Scilit]
  16. Maréchal, P. A note on entropy optimization. In Approximation, Optimization and Mathematical Economics; Springer: Berlin/Heidelberg, Germany, 2001; pp. 205–211. [Google Scholar] [CrossRef] [Scilit]
  17. Beck, A. First-Order Methods in Optimization; SIAM-Society for Industrial and Applied Mathematics: Philadelphia, PA, USA, 2017. [Google Scholar]
  18. Teboulle, M. A simplified view of first order methods for optimization. Math. Program. 2018, 170, 67–96. [Google Scholar] [CrossRef] [Scilit]
  19. Sun, J.; Vû, H.; Ellerbroek, J.; Hoekstra, J.M. Supplemental dataset for “Weather field reconstruction using aircraft surveillance data and a novel meteo-particle model”. PLoS ONE 2018, 13, e0205029. [Google Scholar] [CrossRef] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.