A Note on Burg ’ s Modified Entropy in Statistical Mechanics

Abstract: Burg’s entropy plays an important role in this age of information euphoria, particularly in understanding the emergent behavior of a complex system such as statistical mechanics. For discrete or continuous variable, maximization of Burg’s Entropy subject to its only natural and mean constraint always provide us a positive density function though the Entropy is always negative. On the other hand, Burg’s modified entropy is a better measure than the standard Burg’s entropy measure since this is always positive and there is no computational problem for small probabilistic values. Moreover, the maximum value of Burg’s modified entropy increases with the number of possible outcomes. In this paper, a premium has been put on the fact that if Burg’s modified entropy is used instead of conventional Burg’s entropy in a maximum entropy probability density (MEPD) function, the result yields a better approximation of the probability distribution. An important lemma in basic algebra and a suitable example with tables and graphs in statistical mechanics have been given to illustrate the whole idea appropriately.


Introduction
The concept of entropy [1] figured strongly in the physical sciences during the 19th century, especially in thermodynamics and statistical mechanics [2], as a measure of equilibrium and evolution of thermodynamic systems.Two main views were developed which were the macroscopic view formulated originally by Clausius and Carnot and the microscopic approach associated with Boltzmann and Maxwell.Since then, both the approaches have made introspection in natural thermodynamic and microscopically probabilistic systems possible.Entropy is defined as the measure of a system's thermal energy per unit temperature that is unavailable for doing useful work.Because work is obtained from ordered molecular motion, the amount of entropy is also a measure of molecular disorder or randomness of a system.The concept of entropy provides deep insight into the direction of spontaneous change for many day-to-day phenomena.Now, how entropy was developed by Rudolf Clausius [3] is discussed below.

Clausius's Entropy
To provide a quantitative measure for the direction of spontaneous change, Clausius introduced the concept of entropy as a precise way of expressing the second law of thermodynamics.The Clausius form of the second law states that spontaneous change for an irreversible process [4] in an isolated system (that is, one that does not exchange heat or work with its surroundings) always proceeds in the direction of increasing entropy.By the Clausius definition, if an amount of heat Q flows into a large heat reservoir at temperature T above absolute zero, then ∆s " Q T .This equation effectively gives an alternate definition of temperature that agrees with the usual definition.Assume that there are two heat reservoirs R 1 and R 2 at temperatures T 1 and T 2 .(such as the stove and the block of ice).
If an amount of heat Q flows from R 1 to R 2 , then the net entropy change for the two reservoirs is ∆s " Q ˆ1 T 2 ´1 T 1 ˙which is positive, provided that T 1 ą T 2 .Thus, the observation that heat never flows spontaneously from cold to hot is equivalent to requiring the net entropy change to be positive for a spontaneous flow of heat.When the system is in thermodynamic equilibrium, then dS " 0, i.e., if T 1 " T 2 , then the reservoirs are in equilibrium, no heat flows, and ∆s " 0. If the gas absorbs an incremental amount of heat dQ from a heat reservoir at temperature T and expands reversibly against the maximum possible restraining pressure P, then it does the maximum work and dW " PdV.The internal energy of the gas might also change by an amount dU as it expands.Then, by conservation of energy, dQ " dU `PdV.Because the net entropy change for the system plus reservoir is zero when maximum work is done and the entropy of the reservoir decreases by an amount dS reservoir " ´dQ T , this must be counterbalanced by an entropy increase of dS system " dU `PdV T " dQ T for the working gas so that dS system `dS reservoir " 0. For any real process, less than the maximum work would be done (because of friction, for example), and so the actual amount of heat dQ 1 absorbed from the heat reservoir would be less than the maximum amount dQ.For example, the gas could be allowed to expand freely into a vacuum and do no work at all.Therefore, it can be stated that dS system " dU `PdV T ě dQ 1 T with, dQ 1 " dQ in the case of maximum work corresponding to a reversible process.This equation defines S system as a thermodynamic state variable, meaning that its value is completely determined by the current state of the system and not by how the system reached that state.Entropy is a comprehensive property in that its magnitude depends on the amount of material in the system.In one statistical interpretation of entropy, it is found that for a very large system in thermodynamic equilibrium, entropy S is proportional to the natural logarithm of a quantity Ω corresponding to S and can be realized; that is, S " KlnΩ, in which K is related to molecular energy.On the other hand, entropy generation analysis [5][6][7][8][9][10][11] is used to optimize the thermal engineering devices for higher energy efficiency; it has attracted wide attention to its applications and rates in recent years.In order to access the best thermal design of systems, by minimizing the irreversibility, the second law of thermodynamics could be employed.Entropy generation is a criterion for the destruction of a systematized work.The development of the theory followed two conceptually different lines of thought.Nevertheless, they are symbiotically related, in particular through the work of Boltzmann.

Boltzmann's Entropy
In addition to thermodynamic (or heat-change) entropy, physicists also study entropy statistically [12,13].The statistical or probabilistic study of entropy is presented in Boltzmann's law.Boltzmann's equation is somewhat different from the original Clausius (thermodynamic) formulation of entropy.Firstly, the Boltzmann formulation is structured in terms of probabilities, while the thermodynamic formulation does not consist in the calculation of probabilities.The thermodynamic formulation can be characterized as a mathematical formulation, while the Boltzmann formulation is statistical.Secondly, the Boltzmann equation yields a value of entropy S while the thermodynamic formulation yields only a value for the change in entropy pdSq.Thirdly, there is a shift in content, as the Boltzmann equation was developed for research on gas molecules rather than thermodynamics.Fourthly, by incorporating probabilities, the Boltzmann equation focuses on microstates, and thus explicitly introduces the question of the relationship between macrostates and microstates.Boltzmann investigated such microstates and defined entropy in a new way such that the macroscopic maximum entropy state corresponded to a thermodynamic configuration which could be formulated by the maximum number of different microstates.He noticed that the entropy of a system can be considered as a measure of the disorder in the system and that in a system having many degrees of freedom, the number measuring the degree of disorder also measured the uncertainty in a probabilistic sense about the particular microstates.
The value W was originally intended to be proportional to the Wahrscheinlichkeit (means probability) of a macrostate for some probability distribution of a possible microstate, in which the thermodynamic state of a system can be realized by assigning different ξ and ρ of different molecules.The Boltzmann formula is the most general formula for thermodynamic entropy; however, his hypothesis was for an ideal gas of N identical particles, of which N i pi " 1, 2, . . . ., lq are the ith microscopic condition of position and momentum of a given distribution D i " pN 1 , N 2 , . . . ., N l q.Here, D 1 " pN, 0, . . . ., 0q, D 2 " pN ´1, 1, . . . ., 0q, . . .etc.For this state, the probability of each microstate system is equal, so it was equivalent to calculating the number of microstates associated with a macrostate.Then the statistical disorder is given by [14] Ns. Therefore, the entropy given by Boltzmann is: S B " KlnW Where lnW " ln Let us now take an approximate value of W for a large N!.Using Stirling's approximation lnN! -NlnN ´N, we have: p i lnp i where p i " N i N is the probability of the occurrence of i th microstates.Boltzmann was the first to emphasize the probabilistic meaning of entropy and the probabilistic nature of thermodynamics.

Information Theory andShannon's Entropy
Unlike the first two entropy approaches (thermodynamic entropy by Clausius and Boltzmann's entropy), the third major form of entropy did not fall within the field of physics, but was developed instead in a new field known as information theory [15][16][17] (also known as communication theory).A fundamental step in using entropy in new contexts unrelated to thermodynamics was provided by Shannon [18], who came to conclude that entropy could be used to measure types of disorder other than that of thermodynamic microstates.Shannon was interested in information theory [19,20], particularly in the ways in which information can be conveyed via a message.This led him to examine probability distributions in a very general sense and he worked to find a way of measuring the level of uncertainty in different distributions.
For example, suppose the probability distribution for the outcome of a coin toss experiment is P(H) = 0.999 and P(T) = 0.001.One is likely to notice that there is much more "certainty" than "uncertainty" about the outcome of this experiment and, consequently, the probability distribution.If, on the other hand, the probability distribution governing that same experiment were P(H) = 0.5 and P(T) = 0.5, then there is much less "certainty" and much more "uncertainty" when compared to the previous distribution.However, how can these uncertainties can be quantified?Is there some algebraic function which measures the amount of uncertainty in any probabilistic distribution in terms of the individual probabilities?From these types of simple examples and others, Shannon was able to devise a set of criteria which any measure of uncertainty may satisfy.He then tried to find an algebraic form which would satisfy his criteria and discovered that there was only one formula which fit.Let the probabilities of n possible outcomes E 1 , E 2 , . . . . . . ., E n of an experiment be p 1 , p 2 , .. . . ., p n , giving rise to the probability distribution P " pp 1 , p 2 , . . ., p n q; There is an uncertainty as to the outcome when the experiment is performed.Shannon suggested the measure ´n ř i"1 p i lnp i , which is identical to the previous entropy relation if the constant of probability is taken as the Boltzmann constant K. Thus, Shannon showed that entropy, which measures the amount of disorder in a thermodynamic system, also measures the amount of uncertainty in any probability distribution.Let us now give the formal definition of Shannon's entropy as follows: Consider a random experiment P " pp 1 , p 2 , . . ., p n q whose possible outcomes have probabilities p i , i " 1, 2, . . ., n that are known.Can we guess in advance which outcome we shall obtain?Can we measure the amount of uncertainty?We shall denote such an uncertainty measure by HpPq " H n pp 1 , p 2 , . . ., p n q.The most common as well as the most useful measure of uncertainty is Shannon's informational entropy (which should satisfy some basic requirements), which is defined as follows: Definition I: Let pp 1 , p 2 , . . ., p n q be the probability of the occurrence of the events E (iii) H n`1 pp 1 , p 2 , . . ., p n , 0q " H n pp 1 , p 2 , . . ., p n q, i.e., it should not change if there is an impossible outcome to the probability.(iv) Its minimum is 0 when there is no uncertainty about the outcome.Thus, it should vanish when one of the outcomes is certain to happen so that H n pp 1 , p 2 , . . ., p n q " 0; p i " 1, p j " 0, i ‰ j, i " 1, 2, . . ., n (v) It is the maximum when there is maximum uncertainty, which arises when the outcomes are equally likely so that H n pPq is the maximum when p 1 " p 2 " .. . . ." p n " 1 n .
(vi) The maximum value of H n pPq increases with n.
(vii) For two independent probability distributions Ppp 1 , p 2 , . . ., p n q and Qpq 1 , q 2 , . . ., q m q, the uncertainty of the joint scheme P Y Q should be the sum of their uncertainties: H n`m pP Y Qq " H n pPq `Hm pQq Shannon's entropy has various applications in the field of portfolio analysis, the measurement of economic analysis, transportation, and urban and regional planning as well as in the fields of statistics, thermodynamics, queuing theory, parametric estimation, etc.It has been used in the field non-commensurable and conflicting criteria [21] and in the nonlinear complexity of random sequences [22] as well.

Jaynes' Maximum Entropy (MaxEnt) Principle
Let the random variable of an experiment be X, and assume the probability mass associated with the value x i is p i , i.e., P X piq " PpXq " p i , i " 1, 2, . . . ., n.
Kapur [27,28] showed that there is always a concave function of a 1 , a 2 , . . . . . ., a m .We also note that all the probabilities given by p i are always positive.We naturally want to know whether there is another measure of entropy other than Shannon's entropy which, when maximized, subject to, p i g ri " a r , r " 1, 2, . . . . . . ., z, p i ě 0, gives positive probabilities and for which S max is possibly a concave function [29] of parameters.Kapur [30] studied that Burg's [31] measure of entropy, which has been very successfully used in spectral analysis, does always give positive probabilities.The maximum entropy principal of Jaynes has been used frequently to derive the distribution of statistical mechanics by maximizing the entropy of the system subject to some given constraints.The Maxwell-Boltzman distribution is obtained when there is only one constraint on a system which prescribes the expected energy per particle of the system by Bose-Einstein (B.E.) distribution, Fermi-Dirac (F.D.) distribution and intermediate statistics (I.S.) distributions; these are obtained by maximizing the entropy subject to two constraints by Kapur and Kesavan, and Kullback [32,33] and also by the present authors [34].

Formulation of MEPD in Statistical Mechanics Using Shannon's Measure of Entropy
Let p 1 , p 2 , .. . . ., p n be the probabilities of a particle having energy levels ε 1 ,ε 2 , . . ..,ε n , respectively, and let the expected value of energy be prescribed as ε; then, to get MEPD, we maximize the Shannon's measure of entropy: Let the Lagrangian be Differentiating with respect to p i 's, we get: lnp i `λ `µε i " 0 ñ p i " expp´λ ´µε i q where, λ, µ are to be determined by using Equation (2) so that Where Equation ( 4) is the well-known Maxwell-Boltzmann distribution from statistical mechanics which is used in many areas [35][36][37].

Burg's Entropy
we get where λ&µ are obtained by solving the equations Multiplying first and second Equation ( 8) by λ&µ respectively then adding We get so that from Equation (8), Then, µ " 0 is an obvious solution but that will give us λ " n and this will satisfy the second equation of (8) if m " n `1 2 .Now, Equation ( 10) is the nth degree polynomial in µ, and one of its roots is zero.
Its non-zero solutions will be obtained by solving an equation of pn ´1qth degree in µ.Lemma has been proved by Kapur as the following: Lemma: All the roots of n ř r"1 1 n ´µ pm ´iq " 1 are real; in other words, none of the roots can be complex.
Proof: Let µ " α `iβ, i " ?´1 be a pair of complex conjugate roots of Equation (10) ´1 , where m " k `l, k " rms and l is a "+" fraction.More precisely, f pµq Ñ `8 when µ Ñ points from one side and f pµq Ñ ´8 when µ Ñ points from other side.Again, lnp1 `ap i q ´lnp1 `aq, a ą 0 (11) ap i p1 ´pi q p1 `aq p1 `ap i q ą 0 (12) Therefore, B mod pPq is the monotonic increasing function of a.For the probability distribution pp, 1 ´pq it is showed (see: The measure of entropy Max P pB mod pPqq is the Burg's modified entropy.This is a better measure than the standard Burg's measure since it is always positive and there is no computational problem when p i is very small.In the above case, the maximum value increases with the number of possible outcomes n. lnp1 `ap i q ´lnp1 `aq " when a Ñ 8 , maximizing pB mod pPqq and pB pPqq will give the same result in both cases;again, if pB mod pPqq is maximized under the constraints we get 1 1 `ap i " rλ 0 `λ1 g 1 px i q `λ2 g 2 px i q `.. . . . . . .`λm g m px i qs, i " 1, 2, . . . . . . . . . . . ., n. Letting, we have The λ 1 i 's are determined by using Constraints ( 14) and ( 15) and this gives the MEPD when Burg's entropy is maximized as subject to Equation (14).

MBE and Its Concavity of S max under Prescribed Mean
Maximize B mod pPq " subjectto We obtain this using Lagranges multiplier mechanics: From the above equation: Therefore, and p i " a `n an `µ " pn `aq i ´am ´n pn `1q 2 where µ is determined as a function of m and that is S max will be a concave function of m if dµ dm ă 0, that is, if either µ ą 0, ă 0 when the denominator ă 0, ą 0, respectively.In the above case, when a " 1 we get S max and the derivative of S max as follows: " dµ dm Additionally, we have So that dµ dm Therefore, Therefore, from Equations ( 27) and (30), S max will be a concave function of m if either piq µ ą 0, Or Additionally, when µ " 0, d 2 S max dm 2 " 0, all the probabilities are equal, and from Equation ( 24) we have: We have, Now, if we proceed algebraically as done Section 2.3, we get: The obvious solution of the above problem is µ " 0 which will give λ " 1 p i " n, i.e., uniform distribution, and thus we get, m " n `1 2 .

Example
Let p 1 , p 2 , .. . . ., p 10 be the probabilities of a particle having energy levels ε 1 ,ε 2 , . . ..,ε 10 ; respectively, and let the expected value of energy be prescribed as m; then, we get the maximum entropy probability distribution (MEPD) with MBE as follows: There may be two cases, Case (i) when m ă n `1 2 " 5.5.
In this case, when m lies between 1 and 5.5, µ ą 0 implies dS max dm ą 0 and then S max is increasing.
In this case, when m lies between 5.5 and 10, µ ă 0 implies dS max dm ă 0 and then S max is decreasing.
When m " n `1 2 " 5.5, µ " 0, dS max dm " 0. it can be shown that S max will be concave if we prescribe E pg pxqq instead of E pxq, where g pxq is a monotonic increasing function of x; then we will apply the necessary changes.Again, since the concavity of S max has already been proven, this will enable us to handle the inequality constraint of the type m 1 ď

Simulated Results
Following Table (     If i p is plotted against i , we get rectangular hyperbolic types of curves (Table 3 and Figure 4).If i p is plotted against i , we get rectangular hyperbolic types of curves (Table 3 and Figure 4).If i p is plotted against i , we get rectangular hyperbolic types of curves (Table 3 and Figure 4).If p i is plotted against i, we get rectangular hyperbolic types of curves (Table 3 and Figure 4).

Conclusions
In the present paper we have presented different MEPDs and respective entropy measures with their properties.It has been found that MBE is a better measure than Burg's entropy when the maximized subject to the mean is prescribed, and it also has been shown that unlike Burg's entropy, the maximum value of MBE increases with n .The main problem here will consist of solving 1 z  simultaneous transcendental equations for the Lagranges multipliers.An application in statistical mechanics with simulated data has been studied with the help of Lingo11 software and corresponding graphs are provided.Now, one question arises: Will this result continue to hold for other moment constraints also?When we take generalized moment expectation of ) ( x g instead of expectation of x , then   x g must be a monotonic increasing function of x , and if i p becomes negative for some values of the moments, then we have to set those probabilities to zero and reformulate the problem for the remaining probabilities over the remaining range and solve it.

1 λ
m for m " 1 p0.25q 10 Solution: Maximizing the measure of entropy subject to the given constraints, we have p1 `pi q " " 55 `m, we get the probability distribution in the form of a table and also get the values of S max as described by Kapur and Kesavan.

Figure 4 .
Figure 4. Rectangular hyperbolic types of graphs.(a) when Measure and MEPD i lnp i was replaced by Burg's measure of entropy B pPq " ř i lnp i , it gave interesting results as shown by Kapur.Burg's measure of entropy is always negative, but this does not matter in entropy maximization, where it has been found that a probability distribution with maximum entropy satisfies the same constraint and it does not matter if all the entropies are negative.So, in Equation (1) when we use B pPq " ÿ i lnp i , with the constraints n

Table 1 .
The a values & maximum values of Burg's Modified Entropy.

Table 2 )
Found Using LINGO Software 2011 where Different Max-Entropy Values Are Given for Different m Values:
Graphs obtained from the above table are given on the next page (Figures1-3):

Table 3 .
p i and i graph.

Table 3 .
i p and i graph.