Numerical Algorithms for Estimating Probability Density Function Based on the Maximum Entropy Principle and Fup Basis Functions

Abstract

Estimation of the probability density function from the statistical power moments presents a challenging nonlinear numerical problem posed by unbalanced nonlinearities, numerical instability and a lack of convergence, especially for larger numbers of moments. Despite many numerical improvements over the past two decades, the classical moment problem of maximum entropy (MaxEnt) is still a very demanding numerical and statistical task. Among others, it was presented how Fup basis functions with compact support can significantly improve the convergence properties of the mentioned nonlinear algorithm, but still, there is a lot of obstacles to an efficient pdf solution in different applied examples. Therefore, besides the mentioned classical nonlinear Algorithm 1, in this paper, we present a linear approximation of the MaxEnt moment problem as Algorithm 2 using exponential Fup basis functions. Algorithm 2 solves the linear problem, satisfying only the proposed moments, using an optimal exponential tension parameter that maximizes Shannon entropy. Algorithm 2 is very efficient for larger numbers of moments and especially for skewed pdfs. Since both Algorithms have pros and cons, a hybrid strategy is proposed to combine their best approximation properties.

1. Introduction

Many physical processes cannot be characterized deterministically owing to the presence of intrinsic or parametric uncertainty due to their physical nature, interpretation or measurements. Therefore, results are usually given in the form of a certain number of the first few statistical power moments or, rarely, as a probability density function (pdf) [1,2,3].

The maximum entropy principle, defined by Jaynes [4], is a versatile tool for statistical inference of the probability density function from its moments by maximizing Shannon entropy [5]. The principle states that among all possible pdfs that satisfy our incomplete information about the system, the one that maximizes entropy is the least biased estimate that can be made. It agrees with everything that is known but carefully avoids anything that is unknown [6,7].

In the past few decades, a great number of maximum entropy algorithms (MEAs) have been developed and applied in various fields of science, such as continuum mechanics [8], signal processing [9,10], chemical engineering [11,12,13] and heat transfer [14]. In coastal engineering, a MEAs are used as a powerful tool for the prediction of extreme significant wave heights [15,16,17] and for the distribution of streamwise velocity in open channels [18]. MEAs are especially popular in structural reliability analysis [19,20,21,22,23,24,25]. Their application goes much further, though; for example, MEAs used for the evaluation of the reliability of semiconductor devices [26]. MEAs have even been extended to multidimensional problems [27,28,29,30] and models in 2D domains and on surfaces [31].

MaxEnt algorithms for a higher number of moments are subjected to highly unbalanced nonlinearities, ill-conditioned Jacobian and Hessian matrices in Newton algorithms and many other numerical problems, such as insufficient arithmetic precision [3]. To overcome these difficulties, the MaxEnt algorithm uses orthogonal polynomials [2] and splines [32,33] instead of classic monomials.

Gotovac and Gotovac (2009) [34] employed a different and original idea using finite and localized basis functions with compact support closely related to algebraic polynomials, which made an efficient MaxEnt algorithm possible with more balanced nonlinearities and the ability to solve a higher number of moments. They used Fup basis functions of the second order with compact support, which are similar to wavelets and splines and belong to the class of atomic basis functions (ABF).

The history of the ABFs originates from V. L. and V. A. Rvachevs, [35]. The first monographs on the results of research were published in [36,37], while in [38,39] a detailed analysis of the current publications on ABF is provided, from the first publications until now. Furthermore, based on ABF theory, many different classes of weight functions have arisen, which are often used, especially in digital signal processing [40,41,42,43,44,45,46,47] and groundwater flow modeling [48,49,50,51]. A very interesting theory of atomic solitons is based on ABF theory that is used in new areas, like matter quantization, quantum gravity, Higgs fields, unified theory of nature, as well as in non-traditional areas, like medicine and life sciences, geology and financial markets [52,53,54,55].

In this paper, we will use two types of Fup basis functions that belong to the class of atomic basis functions (ABFs) [56]: the algebraic-type, ${\mathrm{Fup}}_{\mathrm{n}}\left(\mathrm{x}\right)$, and the exponential type, ${\mathrm{EFup}}_{\mathrm{n}}\left(\mathsf{\omega },\mathrm{x}\right)$, containing an additional tension parameter, ω. While atomic basis functions of the algebraic type have been used for many years, ABFs of the exponential type are still not widely used [57]. This work uses Fup and EFup basis functions of the fourth order by, which monomials up to the fourth degree can be represented accurately. This approximation property of the basis functions affects the numerical precision of the requested pdf since it is the first few moments that give the greatest impact on the solution function of the MaxEnt problem.

Based on the algebraic ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ and exponential ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$ basis functions, two Algorithms for solving the maximum entropy problem from known statistical moments are derived and presented. Algorithm 1 is a classical nonlinear algorithm [34], while Algorithm 2 is a fast linear algorithm. Exponential basis functions, ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$, use an unknown parameter, $\mathsf{\omega }$, that is selected considering the maximization of the Shannon entropy functional. Both Algorithms have their advantages and disadvantages, which will be demonstrated in various examples of the probability density functions.

2. Maximum Entropy Principle

Let $\mathrm{f}\left(\mathrm{x}\right)$ be an unknown probability density function (pdf) defined in a finite real interval that satisfies a basic normalized zero statistical moment, which says that all pdf outcomes present a certain event:

$${\displaystyle \underset{\mathrm{a}}{\overset{\mathrm{b}}{{\displaystyle \int }}}}\mathrm{f}\left(\mathrm{x}\right)\mathrm{dx}=1,\mathrm{f}\left(\mathrm{x}\right)\ge 0\forall \mathrm{x}\in \left[\mathrm{a},\mathrm{b}\right]$$

(1)

Suppose that the additional moment constraints on $\mathrm{f}$ are given in the form of classical statistical moments of the higher order:

$${\displaystyle \underset{\mathrm{a}}{\overset{\mathrm{b}}{{\displaystyle \int }}}}{\mathrm{x}}^{\mathrm{i}}\mathrm{f}\left(\mathrm{x}\right)\mathrm{dx}={\mathsf{\mu }}_{\mathrm{i}},\hspace{1em}\hspace{1em}\mathrm{i}=1,\dots ,\mathrm{m}$$

(2)

where ${\mathsf{\mu }}_{\mathrm{i}}$ represents given and known statistical moments. Without losing generality, the interval $\left[\mathrm{a},\mathrm{b}\right]$ can be transformed into the interval $\left[0,1\right],$ as we will write later in this article.

The estimation of the function, $\mathrm{f}\left(\mathrm{x}\right)$, by the maximum entropy principle is obtained by maximizing the Shannon entropy, $\mathrm{H}\left(\mathrm{f}\right)$, associated with the function, $\mathrm{f}$, with constraints given by Equation (2), where the Shannon entropy is given in the form:

$$\mathrm{H}\left(\mathrm{f}\right)=-{\displaystyle \underset{0}{\overset{1}{{\displaystyle \int }}}}\mathrm{f}\left(\mathrm{x}\right)\ln \left[\mathrm{f}\left(\mathrm{x}\right)\right]\mathrm{dx}$$

(3)

Jaynes [4] showed that maximizing $\mathrm{H}\left(\mathrm{f}\right)$ with respect to $\mathrm{f}$, under constraints (2), leads to the following analytical solution:

$$\mathrm{f}\left(\mathrm{x}\right)={\mathrm{e}}^{-{\mathsf{\lambda }}_0-{\mathsf{\lambda }}_1\mathrm{x}-\dots -{\mathsf{\lambda }}_{\mathrm{m}}{\mathrm{x}}^{\mathrm{m}}}$$

(4)

where Lagrange multipliers ${\mathsf{\lambda }}_0,\dots ,{\mathsf{\lambda }}_{\mathrm{m}}$ satisfy the following relations:

$${\displaystyle \underset{0}{\overset{1}{{\displaystyle \int }}}}\exp \left(-{\displaystyle {\displaystyle \sum }_{\mathrm{j}=1}^{\mathrm{m}}}{\mathsf{\lambda }}_{\mathrm{j}}{\mathrm{x}}^{\mathrm{j}}\right)\mathrm{dx}=\exp \left({\mathsf{\lambda }}_0\right)$$

(5)

$$\frac{{\displaystyle {\int }_0^1}{\mathrm{x}}^{\mathrm{i}}\exp \left(-{\displaystyle {\sum }_{\mathrm{j}=1}^{\mathrm{m}}}{\mathsf{\lambda }}_{\mathrm{j}}{\mathrm{x}}^{\mathrm{j}}\right)\mathrm{dx}}{{\displaystyle {\int }_0^1}\exp \left(-{\displaystyle {\sum }_{\mathrm{j}=1}^{\mathrm{m}}}{\mathsf{\lambda }}_{\mathrm{j}}{\mathrm{x}}^{\mathrm{j}}\right)\mathrm{dx}}={\mathsf{\mu }}_{\mathrm{i}},\hspace{1em}\hspace{1em}\mathrm{i}=1,\dots ,\mathrm{m}$$

(6)

From a practical point of view, determining the maximum entropy of a pdf from given moments comes down to solving a nonlinear system of $\mathrm{m}$ Equation (6), creating a challenging nonlinear numerical problem faced by unbalanced nonlinearities, numerical instability and lack of convergence, especially for larger numbers of moments.

3. Numerical Algorithms Using Fup Basis Functions

This chapter describes two basic numerical algorithms for estimating the probability density function from a finite number of known moments. Both algorithms for solving the classical moment problem were developed using atomic basis functions, one of the algebraic type, ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$, and one of the exponential type, ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$.

The function ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ is shown in Figure 1.

The function support contains six characteristic intervals. By a linear combination of these functions, shifted from each other by the length of the characteristic interval, algebraic polynomials up to the fourth degree can be accurately represented (see Appendix A, Appendix A.1). To approximate the pdf using the maximum entropy principle in the range $\left[0,1\right]$, a base of at least six basis functions, ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$, is required, as shown in Figure 2.

Therefore, the MaxEnt optimization procedure for estimating the probability density function starts from the sixth moment (with the zeroth moment engaged, i.e., $\mathrm{m}=5$). For numerical operations in the domain to be performed efficiently, it is necessary to modify basis functions that have non-zero values within the domain and the vertices of which are outside the domain by expressing them in the form of a linear combination of the original basis functions [50]. Figure 3 shows the basis functions in the observed domain for the estimation with six moments ($\mathrm{m}=5$).

Fup basis functions of the exponential type are much less well-investigated than those of the algebraic type [56]. The linear combination of the functions ${\mathrm{EFup}}_{\mathrm{n}}\left(\mathrm{x},\mathsf{\omega }\right)$ can accurately represent the exponential function $\mathrm{E}\left(\mathrm{x}\right)={\mathrm{e}}^{2^{\mathrm{n}}\mathsf{\omega }\mathrm{x}}$ (Appendix A, Appendix A.2). The ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$ function retains all the properties of the ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ function, such as smoothness, finiteness, polynomial development, etc. The special feature of these functions is that they contain the parameter $\mathsf{\omega }$, which allows them to change the slope depending on the parameter value, as shown in Figure 4.

Figure 4 shows function ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$ for different values of parameter $\mathsf{\omega }$. It can be seen that the function ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$ for $\mathsf{\omega }>0$ is tilted to the right, and if $\mathsf{\omega }<0$, the function is tilted to the left. In the boundary case when $\mathsf{\omega }=0,$ the function is symmetric and identical to the algebraic function ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$. Thus, a vector function space of the functions ${\mathrm{EFup}}_{\mathrm{n}}\left(\mathrm{x},\mathsf{\omega }\right)$ also contains functions ${\mathrm{Fup}}_{\mathrm{n}}\left(\mathrm{x}\right)$.

This property of EFup functions of the exponential type gives them an advantage over other finite functions (such as algebraic Fups, splines, wavelets), and it is their flexibility that allows them to “adapt” more to the solution. On a similar track, exponential splines that also contain a tension parameter are used in the literature; these are often applied to solve the singularly perturbed boundary problem [58,59]. When using the exponential type of atomic basis functions, the basic task is to determine the value of the parameter $\mathsf{\omega }$. Since the optimization problem of maximum entropy is the search for a pdf that maximizes the Shannon integral (3), the above will be taken as the criterion for choosing the parameter $\mathsf{\omega }$. Thus, of all possible solutions obtained for different values of the parameter $\mathsf{\omega }$, the one that gives the maximum value of the Shannon entropy functional (3) is adopted.

3.1. Classic MaxEnt Algorithm (Algorithm 1)

Here, we describe the procedure for solving the problem of pdf estimation from known moments using the principle of maximum entropy and ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ basis functions [34]. The procedure is simplified in relation to [34] and consists of the following steps:

• Let the set of $\mathrm{m}$ algebraic moments, ${\mathsf{\mu }}_{\mathrm{i}}$, be given on the general interval $\left[\mathrm{a},\mathrm{b}\right]$. Its values on the unit interval $\left[0,1\right]$ are calculated using a simple linear transformation.
• In expression (4), we express the monomials ${\mathrm{x}}^{\mathrm{i}},\mathrm{i}=0,\dots ,\mathrm{m}$ as a linear combination of basis functions ${\mathrm{Fup}}_4{}_{\mathrm{j}}\left(\mathrm{x}\right),\mathrm{j}=0,\dots ,\mathrm{m}$, as shown in Appendix A, Appendix A.1.
  Since the ${\mathrm{Fup}}_{\mathrm{n}}$ basis functions exactly describe algebraic monomials up to and including the $\mathrm{n}$th order in the algorithm from [34], the residual function ${\mathsf{\epsilon }}_{\mathrm{i}}\left(\mathrm{x}\right)$ is introduced, i.e.,:

  $${\mathrm{x}}^{\mathrm{i}}={\displaystyle {\displaystyle \sum }_{\mathrm{j}=0}^{\mathrm{m}}}{\mathrm{d}}_{\mathrm{ij}}{\mathrm{Fup}}_{\mathrm{n}}{}_{\mathrm{j}}\left(\mathrm{x}\right)+{\mathsf{\epsilon }}_{\mathrm{i}}\left(\mathrm{x}\right),\hspace{1em}\hspace{1em}\mathrm{i}=0,\dots ,\mathrm{m}$$

  (7)

  where

  $$\begin{array}{cc}{\mathsf{\epsilon }}_{\mathrm{i}}\left(\mathrm{x}\right)\equiv 0& \mathrm{for}\mathrm{i}\le \mathrm{n}\\ {\mathsf{\epsilon }}_{\mathrm{i}}\left(\mathrm{x}\right)\ne 0& \mathrm{for}\mathrm{i}>\mathrm{n}\end{array}$$

  (8)

  are residual functions that describe the difference between monomials of the mth order and their Fup_n approximation, and ${\mathrm{d}}_{\mathrm{ij}}$ presents the connection matrix, which depends on the Fup_n approximation and on the location and number of basis functions and, consequently, moments. For instance, the collocation procedure [48] calculates exact monomials in the collocation points, and residual functions ${\mathsf{\epsilon }}_{\mathrm{i}}\left(\mathrm{x}\right)$ fluctuate around zero, with significantly smaller values than Fup basis functions. For increasing numbers of moments and basis functions, the residual functions ${\mathsf{\epsilon }}_{\mathrm{i}}\left(\mathrm{x}\right)$ converge to zero, i.e., in the limit, for $\mathrm{m}\to \infty$, ${\mathsf{\epsilon }}_{\mathrm{i}}\left(\mathrm{x}\right)$ is zero. Given the fact that the influence of lower moments in the process of determining the pdf is the largest, in this paper, we choose the Fup basis function of the higher order, i.e., $\mathrm{n}=4$, which allows us to simplify the algorithm from [34] (leaving in mind that monomials above the fourth degree are now expressed approximately):

  $${\mathrm{x}}^{\mathrm{i}}\approx {\displaystyle {\displaystyle \sum }_{\mathrm{j}=0}^{\mathrm{m}}}{\mathrm{d}}_{\mathrm{ij}}{\mathrm{Fup}}_4{}_{\mathrm{j}}\left(\mathrm{x}\right),\hspace{1em}\hspace{1em}\mathrm{i}=0,\dots ,\mathrm{m}$$

  (9)

  Next, we calculate the connection matrix ${\mathrm{d}}_{\mathrm{ij}}$. Unlike [34], where the connection matrix ${\mathrm{d}}_{\mathrm{ij}}$ is obtained by the collocation method, here, it is obtained by the Galerkin method:

  $$\begin{array}{c}{\displaystyle {\displaystyle \sum }_{\mathrm{j}=0}^{\mathrm{m}}}{\mathrm{d}}_{\mathrm{ij}}{\displaystyle \underset{0}{\overset{1}{{\displaystyle \int }}}}{\mathrm{Fup}}_4{}_{\mathrm{j}}\left(\mathrm{x}\right){\mathrm{Fup}}_4{}_{\mathrm{k}}\left(\mathrm{x}\right)\mathrm{dx}={\displaystyle \underset{0}{\overset{1}{{\displaystyle \int }}}}{\mathrm{x}}^{\mathrm{i}}{\mathrm{Fup}}_4{}_{\mathrm{k}}\left(\mathrm{x}\right),\hspace{1em}\hspace{1em}\mathrm{i}=0,\dots ,\mathrm{m},\\ \mathrm{k}=0,\dots ,\mathrm{m}\end{array}$$

  (10)

  Multiplying expression (9) with the optimal pdf in the form

  $${\mathrm{f}}^{*}\left(\mathrm{x}\right)={\mathrm{e}}^{{\displaystyle {{\displaystyle \sum }}_{\mathrm{j}=0}^{\mathrm{m}}}{\mathsf{\gamma }}_{\mathrm{j}}{\mathrm{Fup}}_4{}_{\mathrm{j}}\left(\mathrm{x}\right)}$$

  (11)

  and integrating over the domain $\left[0,1\right]$, we can relate the classical algebraic and ${\mathrm{Fup}}_4$ moments

  $${\mathsf{\mu }}_{\mathrm{i}}={\displaystyle {\displaystyle \sum }_{\mathrm{j}=0}^{\mathrm{m}}}{\mathrm{d}}_{\mathrm{ij}}·{\mathsf{\mu }}_{\mathrm{j}}{}^{{\mathrm{Fup}}_4};\mathrm{i}=0,\dots ,\mathrm{m}$$

  (12)

  where ${\mathsf{\mu }}_{\mathrm{i}}$ represents the given moments of the requested pdf, while

  $${\mathsf{\mu }}_{\mathrm{i}}{}^{{\mathrm{Fup}}_4}={\displaystyle \underset{0}{\overset{1}{{\displaystyle \int }}}}{\mathrm{Fup}}_4{}_{\mathrm{i}}\left(\mathrm{x}\right)·{\mathrm{e}}^{{\displaystyle {{\displaystyle \sum }}_{\mathrm{j}=0}^{\mathrm{m}}}{\mathsf{\gamma }}_{\mathrm{j}}{\mathrm{Fup}}_4{}_{\mathrm{j}}\left(\mathrm{x}\right)}\mathrm{dx}$$

  (13)

  represents the moments of the ${\mathrm{Fup}}_4$ basis functions. Since the Lagrange multipliers, ${\mathsf{\gamma }}_{\mathrm{j}}$, are unknown, the system of Equation (13) must be solved iteratively.
• The algorithm starts with the initial pdf assumption, i.e., for the step $\mathrm{k}=0,$ the initial values of the Lagrange multipliers ${\mathsf{\gamma }}_{\mathrm{j}}{}^{\left(\mathrm{k}=0\right)},\mathrm{j}=0,\dots ,\mathrm{m}$ are selected, from which the initial pdf is calculated according to (11).
  For simplicity, for initial values, ${\mathsf{\gamma }}_{\mathrm{j}}{}^{\left(\mathrm{k}=0\right)},$ we take zero values, as the numerical procedure is not sensitive to the selection of the initial pdf. Then, the Fup moments ${\mathsf{\mu }}_{\mathrm{j}}{}^{{\mathrm{Fup}}_4};\mathrm{j}=0,\dots ,\mathrm{m}$ are determined in the $\mathrm{k}$th iterative step by solving system (12).
• From the known Fup moments, the problem of pdf estimation using the principle of maximum entropy is now solved:

  $${\displaystyle \underset{0}{\overset{1}{{\displaystyle \int }}}}{\mathrm{Fup}}_4{}_{\mathrm{i}}\left(\mathrm{x}\right)·{\mathrm{e}}^{{\displaystyle {{\displaystyle \sum }}_{\mathrm{j}=0}^{\mathrm{m}}}{\mathsf{\gamma }}_{\mathrm{j}}{\mathrm{Fup}}_4{}_{\mathrm{j}}\left(\mathrm{x}\right)}\mathrm{dx}={\mathsf{\mu }}_{\mathrm{i}}{}^{{\mathrm{Fup}}_4};\mathrm{i}=0,\dots ,\mathrm{m}$$

  (14)

  To determine the Lagrange multipliers, ${\mathsf{\gamma }}_{\mathrm{j}}$, the nonlinear system of Equation (14) is not solved at once but is solved iteratively by the classical Newton-Raphson method, equation by equation, using Romberg’s numerical integration.
• The index of the current equation, $\mathrm{i}$, is set. The correction of the ith Lagrange coefficient is performed using the classical Newton iterative procedure. If we demand that the residual function of the Lagrange coefficient, $\mathsf{\Psi }\left({\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}\right)$, weighs zero, then for the $\mathrm{k}+1$ iterative step, the relation holds:

  $${\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}+1}}={\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}-\frac{\mathsf{\Psi }\left({\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}\right)}{\mathsf{\Psi }\prime \left({\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}\right)}$$

  (15)

  In the current iterative step, $\mathrm{k}$, the correction of the value ${\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}$ is required in a way so that first the residual of the Lagrange coefficient belonging to a particular moment is determined:

  $$\mathsf{\Psi }\left({\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}\right)={\displaystyle \underset{0}{\overset{1}{{\displaystyle \int }}}}{\mathrm{Fup}}_4{}_{\mathrm{i}}\left(\mathrm{x}\right)·{\mathrm{e}}^{{\displaystyle {{\displaystyle \sum }}_{\mathrm{j}=0}^{\mathrm{m}}}{\mathsf{\gamma }}_{\mathrm{j}}{\mathrm{Fup}}_4{}_{\mathrm{j}}\left(\mathrm{x}\right)}\mathrm{dx}-{\mathsf{\mu }}_{\mathrm{i}}{}^{{\mathrm{Fup}}_4}$$

  (16)

  and then the first derivation of that residual:

  $$\mathsf{\Psi }\prime \left({\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}\right)={\displaystyle \underset{0}{\overset{1}{{\displaystyle \int }}}}{\mathrm{Fup}}_4{}_{\mathrm{i}}\left(\mathrm{x}\right)·{\mathrm{e}}^{{\displaystyle {{\displaystyle \sum }}_{\mathrm{j}=0}^{\mathrm{m}}}{\mathsf{\gamma }}_{\mathrm{j}}{\mathrm{Fup}}_4{}_{\mathrm{j}}\left(\mathrm{x}\right)}·{\mathrm{Fup}}_4{}_{\mathrm{i}}\left(\mathrm{x}\right)\mathrm{dx}$$

  (17)

  and the correction of the Lagrange multiplier in the $\mathrm{k}$th iterative step:

  $${\displaystyle {\mathsf{\Delta }\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}=-\frac{\mathsf{\Psi }\left({\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}\right)}{\mathsf{\Psi }\prime \left({\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}\right)},\hspace{1em}{\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}+1}}={\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}+\Delta {\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}}$$

  (18)

If all values of $\Delta {\displaystyle {\mathsf{\gamma }}_{\mathrm{i}}^{\mathrm{k}}},\mathrm{i}=0,\dots ,\mathrm{m}$ are less than the given tolerance, the iterative procedure is stopped; otherwise, step $\mathrm{k}+1$ is taken and point five is repeated until the convergence condition is satisfied or until the maximum number of given iterative steps is reached.

3.2. Fast Linear Algorithm (Algorithm 2)

Instead of solving a nonlinear system of Equation (13) in the classical MaxEnt problem, which, in some cases, can cause greater numerical instabilities and slow the solution convergence, the process of pdf estimation from known moments using the maximum entropy principle can be simplified by reducing the problem to solving the linear system of equations. The procedure is described using ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$ basis functions and consists of the following steps:

• The probability density function is assumed in the form of a linear combination of ${\mathrm{EFup}}_4$ basis functions:

  $$\mathrm{f}\left(\mathrm{x}\right)\approx {\displaystyle {\displaystyle \sum }_{\mathrm{j}=0}^{\mathrm{m}}}{\mathrm{c}}_{\mathrm{j}}·{\mathrm{EFup}}_4{}_{\mathrm{j}}\left(\mathrm{x},\mathsf{\omega }\right)$$

  (19)
  The procedure of pdf determination from the given moments, ${\mathsf{\mu }}_{\mathrm{i}}$, comes down to directly solving a linear system of equations:

  $${\mathrm{c}}_{\mathrm{j}}{\displaystyle \underset{0}{\overset{1}{{\displaystyle \int }}}}{\mathrm{x}}^{\mathrm{i}}·{\mathrm{EFup}}_4{}_{\mathrm{j}}\left(\mathrm{x},\mathsf{\omega }\right)\mathrm{dx}={\mathsf{\mu }}_{\mathrm{i}};\mathrm{i},\mathrm{j}=0,\dots ,\mathrm{m}$$

  (20)

  from which follow the coefficients of the linear combination, ${\mathrm{c}}_{\mathrm{j}}$.
• By including the calculated coefficients, ${\mathrm{c}}_{\mathrm{j}}$, in expression (19), a pdf estimation is obtained, which can also have negative values. Since the probability density function must be positive by definition, i.e., non-negative in the range $\left[0,1\right]$, the corresponding R-function [60] is used to remove the negative part of the function (or to control the lower limit of the pdf):

  $${\mathrm{f}}^{*}\left(\mathrm{x}\right)=\frac{1}{1+\mathsf{\alpha }}\left({\mathrm{f}}_1\left(\mathrm{x}\right)+{\mathrm{f}}_2\left(\mathrm{x}\right)+\sqrt{{\mathrm{f}}_1{\left(\mathrm{x}\right)}^2+{\mathrm{f}}_2{\left(\mathrm{x}\right)}^2-2\mathsf{\alpha }·{\mathrm{f}}_1\left(\mathrm{x}\right)·{\mathrm{f}}_2\left(\mathrm{x}\right)}\right)$$

  (21)

  where $\mathsf{\alpha }=\mathsf{\alpha }\left({\mathrm{f}}_1\left(\mathrm{x}\right),{\mathrm{f}}_2\left(\mathrm{x}\right)\right)$ is an arbitrary function that satisfies the condition:

  $$-1<\mathsf{\alpha }\left({\mathrm{f}}_1\left(\mathrm{x}\right),{\mathrm{f}}_2\left(\mathrm{x}\right)\right)\le 1$$

  (22)

  For $\mathsf{\alpha }=1$ follows:

  $${\mathrm{f}}^{*}\left(\mathrm{x}\right)=\frac{1}{2}\left\{{\mathrm{f}}_1\left(\mathrm{x}\right)+{\mathrm{f}}_2\left(\mathrm{x}\right)+\left|{\mathrm{f}}_1\left(\mathrm{x}\right)-{\mathrm{f}}_2\left(\mathrm{x}\right)\right|\right\}$$

  (23)

  In the case of maximum entropy, ${\mathrm{f}}_1\left(\mathrm{x}\right)$ represents the probability density function, and the function ${\mathrm{f}}_2\left(\mathrm{x}\right)$ is the lower limit that has a value of zero, so (21) becomes:

  $${\mathrm{f}}^{*}\left(\mathrm{x}\right)=\frac{1}{2}\left\{{\mathrm{f}}_1\left(\mathrm{x}\right)+\left|{\mathrm{f}}_1\left(\mathrm{x}\right)\right|\right\}$$

  (24)

  If $\mathrm{f}\left(\mathrm{x}\right)$ obtained from (19) i (20) is already a non-negative function, then (21) has no effect on the solution of the pdf.
• The function obtained according to expression (21) is normalized for the pdf to fulfill the basic probability condition (1). The moments of the pdf obtained in this way are calculated, compared with the given moments and the error of the calculated moments is estimated. The value of Shannon entropy is also calculated according to expression (3).
• The procedure is repeated for a number of real parameters, $\mathsf{\omega }$, within the selected interval. The interval can be arbitrary and contain positive and negative values of parameters. When $\mathsf{\omega }=0$, the ${\mathrm{EFup}}_4$ basis functions are identical to the ${\mathrm{Fup}}_4$ basis functions.
• A pdf with the parameter $\mathsf{\omega }$ that gives the highest value of Shannon entropy according to expression (3) is selected.

In the limit, when $\mathrm{m}\to \infty$, the obtained function $\mathrm{f}\left(\mathrm{x}\right)$ approaches the exact pdf function because it satisfies all moments. For a finite number of moments, the procedure is approximate, but the possibility of choosing the parameter, ω, which gives the maximum possible value of the functional (3) and also adapts to the shape of the pdf, often allows for a very good approximation of the pdf. On the other hand, the pdf vector space of the maximum entropy (4) shows that it belongs to a function of the exponential type; thus, the ${\mathrm{EFup}}_4$ basis functions make very good candidates for the linear approximation (19). The procedure is particularly suitable for greater numbers of moments when Algorithm 1 can fail.

4. Examples

In this section, the approximation possibilities of the proposed algorithms and their computational efficiency are explored through various examples of pdfs in order to show the advantages and disadvantages of nonlinear classical Algorithm 1 and linear Algorithm 2.

4.1. Example 1–Impulse Function

The efficiency of Algorithm 1 is demonstrated in the example of a numerically demanding probability density function, which, for the MaxEnt moment problem (1–2), represents a challenging numerical problem. It is an impulse function that is very similar to Dirac’s function in the form:

$$\mathsf{\delta }\left(\mathrm{x}\right)=\frac{256}{\sqrt{\mathsf{\pi }}·\mathrm{ERF}\left(128\right)}{\mathrm{e}}^{-4096{\left(4\mathrm{x}-2\right)}^2}$$

(25)

The peak of this function is in the coordinate $\mathrm{x}=0.5$ with a value of $144.4325$. Since this function is symmetric, the parameter $\mathsf{\omega }$ is zero, which means that Algorithm 1 and basis functions of the algebraic type ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ are used. It is interesting to show the convergence of the solution (Figure 5) that is obtained with only six basis functions, i.e., using the minimum number of moments for this algorithm, with an increase in the number of iterative nonlinear steps.

For $3·{10}^6$ iterations, the achieved pdf shape practically coincides with the shape of the analytical function.

Table 1. Absolute moment errors and pdf values in relation to the total number of iterations for $\mathrm{m}=5$.

Total Number of Iterations	5 × 103	5 × 104	105	106	3 × 106
Absolute moment error	0.26 × 10−2	0.68 × 10−3	0.42 × 10−3	0.12 × 10−4	0.29 × 10−7
Maximum pdf value	50.154	85.738	99.018	138.902	144.279

shows how the moment error calculated using Algorithm 1 decreases with the total number of iterations and how the value of the pdf increases at the point $\mathrm{x}=0.5$ of the observed area.

This example shows the accuracy and stability of numerical Algorithm 1 but also the deficiency of a nonlinear algorithm that requires a large number of iterations. With only six moments, an excellent solution is achieved. With a fast linear Algorithm 2 for this example of a symmetric impulse pdf function, it is not possible to achieve a satisfactory solution, nor can one be achieved with a larger number of moments due to, among other things, the oscillations that occur around the point $\mathrm{x}=0.5$. For example, the maximum pdf value obtained with 20 moments is 16.097, which is very far from the exact value.

4.2. Example 2—Bimodal Function

In the second example, we consider a bimodal function composed of two Gaussian distributions normalized to the domain $\left[0,1\right]$. Figure 6a–c show approximations of the bimodal pdf obtained by Algorithm 1, and Figure 6d–f show approximations obtained using Algorithm 2 for $\mathrm{m}=5,13$ and $20$. Since this function is approximately symmetric, a value close to zero is obtained for the parameter $\mathsf{\omega }$ so that the numerical calculation is done using ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ basis functions.

Improving the pdf estimation by increasing the number of moments taken into account, besides its shape, is monitored through the achieved accuracy of the moments and through the values of the Shannon entropy functional using expression (3). The exact value of the Shannon integral for a bimodal function is $\mathrm{H}\left(\mathrm{f}\right)=-0.42420$.

Table 2. Absolute moment error and values of functional (3) for Algorithm 1.

Number of Moments	6 (m = 5)	14 (m = 13)	21 (m = 20)
Absolute moment error	0.33 × 10−6	0.36 × 10−7	0.47 × 10−9
Value of the Shannon functional	−0.39841	−0.42283	−0.42419

and

Table 3. Absolute error of moments and values of functional (3) for Algorithm 2.

Number of Moments	6 (m = 5)	14 (m = 13)	21 (m = 20)
Absolute moment error	0.14 × 10−1	0.18 × 10−4	0.25 × 10−33
Value of Shannon functional	−0.24727	−0.42416	−0.42420

show that as the number of constraints of the required pdf function increases, the absolute moment error decreases, while the approximation of the Shannon entropy functional (3) improves for calculations using Algorithm 1 and Algorithm 2, respectively. Algorithm 1 approximately satisfies the moments due to solving a nonlinear system of equations and the selected accuracy criterion (“threshold”), as well as approximation of monomials with Fup₄ basis functions. On the other hand, functional (3) is also approximately calculated, and with an increasing number of moments, converges relatively slowly to the exact value of $\mathrm{H}\left(\mathrm{f}\right)=0.42420$. Algorithm 2 only solves a linear system that satisfies the given moments. With a small number of moments ($\mathrm{m}=5$) in the symmetric type of pdf, the solution is significantly worse than that of Algorithm 1, which is reflected not only in the visual effect but also in the accuracy of the moments and the value of the functional (3). Yet, when the number of moments reaches $\mathrm{m}=13$, very accurate results are obtained with a much simpler procedure.

The calculation time (“CPU time”) depending on the number of moments taken for both algorithms is graphically shown in Figure 7. The computer time is given in seconds and is obtained using the classic programming language Fortran on a computer with an Intel i7 2.80 GHz processor. As the number of moments increases, both algorithms require a nonlinear increase in computation time. When the number of moments exceeds the limit of $\mathrm{m}=10-12$ a significant saving of computation time is obtained with Algorithm 2. As seen in Figure 7, in the log-log scale, an approximately linear dependence of the number of moments and CPU time is obtained.

From the presented results, it can be concluded that the classical nonlinear MaxEnt Algorithm 1 gives an approximate pdf shape with only six moments. By increasing the number of moments or the number of basis functions, the calculated pdf shape approaches the exact form of the bimodal function, the accuracy of the moments increases and the value of the Shannon entropy functional also approaches the exact value. It can be said that a satisfactory solution is achieved with 21 moments.

However, due to the large number of iterative steps required to solve a nonlinear system of equations, this procedure is a computationally expensive and also very numerically sensitive. Algorithm 2 is a fast algorithm, is not numerically demanding and the procedure is stable. The system of linear equations is solved directly, and, as can be seen in Figure 7, the required computational time is significantly less than that of Algorithm 1. Note that the pdf estimation for a small number of moments is not sufficient; with nine moments, the obtained pdf shape has the characteristics of the required solution that Algorithm 1 provides with only six moments.

However, for a larger number of moments ($\mathrm{m}\ge 13$), the accuracy of the solution increases significantly (see Table 3) both in terms of pdf approximation approaching the exact function shape and in terms of the moment accuracy and the Shannon integral value (3). Using 21 moments, a pdf solution was achieved in which the absolute error of the moments was practically equal to zero and the value of $\mathrm{H}\left(\mathrm{f}\right)$ coincided with the exact value. The advantages of both numerical algorithms for estimating the pdf from known moments can be combined in a way by first obtaining the shape of the required function with six moments using the Fup MaxEnt Algorithm 1, from which one can conclude the basic characteristics of the function, and then with the fast Algorithm 2 to get a more accurate pdf estimation using a greater number of moments.

4.3. Example 3—Beta Function

The third example of the beta distribution is given in the form:

$$\mathrm{f}\left(\mathrm{x}\right)=3990{\mathrm{x}}^{18}{\left(1-\mathrm{x}\right)}^2$$

(26)

When applying atomic basis functions of exponential type ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$, the basic task is to determine the value of the parameter $\mathsf{\omega }$. Function (26) is asymmetric and inclined to the right, which means that the parameter $\mathsf{\omega }>0$ (see Figure 4). Since the principle of maximum entropy requires a pdf that maximizes the Shannon integral (3), this condition will be used as a criterion for selecting the parameter $\mathsf{\omega }$. The procedure is reduced to the simultaneous direct solution of the linear system (20) for different values of the parameter $\mathsf{\omega }$. Of all the numerical solutions thus obtained, the one that gives the maximum value of Shannon entropy (3) is adopted.

The procedure for determining the parameter $\mathsf{\omega }$ for pdf estimation by the principle of maximum entropy with 12 moments using basis functions ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$ is illustrated in Figure 8. The exact value of the integral (3) for the beta function of pdf (26) is $\mathrm{H}\left(\mathrm{f}\right)=1.29632$, and the closest value is obtained with $\mathsf{\omega }=22.5$.

Figure 9 compares the numerical solutions obtained by applying basis functions ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ and ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$ with Algorithms 1 and 2, respectively, for $\mathrm{m}=5$. The value of the parameter $\mathsf{\omega }$ for Algorithm 2, determined by the previously described procedure for pdf estimation with six moments, is $\mathsf{\omega }=23.0$.

From Figure 9, we can see the advantage that exponential finite basis functions have in relation to basis functions of the algebraic type when it comes to pdf approximation, which is quite an asymmetric function. The greater accuracy of numerical solutions obtained using basis functions ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$ can also be seen in

Table 4. Absolute moment errors and Shannon integral values obtained for $\mathrm{m}=5,11$ and $20$ for Algorithms 1 and 2 with ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ basis functions and ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$, respectively.

No. of Moments		6 (m = 5)	12 (m = 11)	21 (m = 20)
Abs. Moment Error	Algorithm 1	0.69 × 10−5	0.120 × 10−4	0.743 × 10−4
	Algorithm 2	0.337 × 10−2	0.144 × 10−4	0.194 × 10−6
Shannon Integral	Algorithm 1	−1.29590	−1.29571	−1.29686
	Algorithm 2	−1.29223	−1.29591	−1.29631
$\mathsf{\omega }$	Algorithm 2	23.0	22.5	17.2

, where the absolute errors are given of approximate moments for calculations with both algorithms using algebraic and exponential Fup basis functions.

Table 4 shows the convergence of the values of the Shannon entropy functionals (3) obtained by increasing the number of moments starting from $\mathrm{m}=5$ using Algorithms 1 and 2 and using the atomic basis functions ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ and ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$. In this example of the probability density function, the advantage that exponential finite functions have over basis functions of the algebraic type comes down to expression. It can be seen that the functions ${\mathrm{EFup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$ achieve a significantly better convergence of the numerical values of the Shannon integral to the exact value of $\mathrm{H}\left(\mathrm{f}\right)=-1.29632$. Since exponential functions adapt better to the shape of the MaxEnt problem solution thanks to the parameter $\mathsf{\omega }$, they ensure the stability of the numerical procedure, which is especially important for the classical MaxEnt algorithm. In this example, this indicated an extreme numerical sensitivity when using ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$ basis functions.

For practical application, i.e., for estimating the pdf from known moments when the form of the function is unknown, the advantages of both described numerical algorithms can be used. First, using the nonlinear MaxEnt Algorithm 1 with basis functions ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$, which are easier to apply than ${\mathrm{FEup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$, we get the appearance of the required function with six moments ($\mathrm{m}=5$), from which can be drawn certain conclusions about the basic characteristics of the pdf, such as its shape, position peaks, number of peaks and inclination, while the number of statistical moments required to accurately describe all the properties of the pdf can be assumed. If an approximately symmetric function is obtained, such a first approximation, the calculation can be continued with the basis functions ${\mathrm{Fup}}_4\left(\mathrm{x}\right)$.

If the first approximation gives an asymmetric function or a larger number of moments is required, then using the fast linear Algorithm 2, the parameter $\mathsf{\omega }$ is determined with the criterion for finding the maximum of the Shannon entropy functional (3), and the calculation continues with exponential functions ${\mathrm{FEup}}_4\left(\mathrm{x},\mathsf{\omega }\right)$. If it is a simple pdf for which less than six moments are enough, Algorithm 1 with ${\mathrm{Fup}}_2$ [34] or ${\mathrm{Fup}}_1$ basis functions can be used. In this way, using the accuracy and robustness of the classic nonlinear MaxEnt Algorithm 1 up to 10–12 moments, we can solve a large number of problems and diagnose the condition if more moments are needed, especially with pdf asymmetry, when linear Algorithm 2 becomes much simpler and more efficient. With this hybrid technique, a large number of challenging problems can be solved in a much easier way than could be solved individually with the two presented algorithms.

5. Conclusions

In this paper, the application has been shown of numerical algorithms based on the maximum entropy principle using Fup basis functions that belong to the class of atomic basis functions with a compact support. The pdf examples selected in this paper demonstrate the efficiency of two algorithms for probability-density-function estimation from a finite number of known moments. These algorithms use Fup basis functions of the algebraic and exponential types. The simulation results show the convergence properties of the proposed algorithms by increasing the number of moments taken into account. The example of a pdf in the form of a pulse function shows the accuracy and stability of the numerical procedure of Algorithm 1, where with only six moments, a satisfactory solution can be achieved by increasing the number of iterations in the calculation.

With such demanding pdfs, the number of iterative steps in the nonlinear procedure becomes very large. The characteristics of numerical solutions obtained with Algorithms 1 and 2 using algebraic finite basis functions are shown in the example of a demanding bimodal function. The classic nonlinear MaxEnt Algorithm 1 gives a good pdf shape approximation with only six moments. By increasing the number of moments or the number of basis functions, the approximate solution converges to the exact function. However, the numerical procedure is computationally expensive and sensitive to the computer’s precision, so the optimization problem becomes poorly conditioned. Fast linear Algorithm 2 ensures complete stability of the procedure, but a higher number of moments is required to satisfy the accuracy of the pdf estimation.

The third example of a pdf is a beta function that represents a polynomial of the 20th degree, which is why it is numerically demanding to approximate. Nonlinear Algorithm 1 showed special sensitivity in this pdf when applying algebraic finite basis functions. In the example of quite an asymmetric pdf function, the advantage of exponential finite basis functions over functions of the algebraic type can be noted both in the stability of the nonlinear classical MaxEnt algorithm and in the achieved accuracy of calculated pdf moments and calculated Shannon integral values as a basic criterion for pdf estimation by the principle of maximum entropy.

For practical applications, i.e., for pdf estimation from known moments when the shape of the function is unknown, we can use the advantages of both numerical algorithms described here, which we defined as a hybrid strategy in this article.