The Maximum Entropy Rate Description of a Thermodynamic System in a Stationary Non-Equilibrium State

In this paper we present a simple model to describe a rather general system in a stationary non-equilibrium state, which is an open system traversed by a stationary flux. The probabilistic description is provided by a non-homogeneous Markov chain, which is not assumed on the basis of a model of the microscopic interactions but rather derived from the knowledge of the macroscopic fluxes traversing the system through a maximum entropy rate principle.


Introduction
In a recent (2006, [1]) survey on the Maximum Entropy Production principle, reference is made to an early attempt (1967, [2,3]) to extend (a pioneering paper dealing with this extension problem was written by E.T. Jaynes himself, see [4]) the celebrated Jaynes' Maximum Entropy principle to thermodynamic systems in a stationary non-equilibrium.Informally stated, assumed that the dynamic is described by a Markov chain, the authors want to find a stochastic transition matrix such that: (i) it has a prescribed probability distribution as a stationary distribution, (ii) the associated chain evolution satisfies given constraints on the macroscopic scale (admissible microscopic evolution) and (iii) the selected transition matrix generates the maximum number of equally probable microscopic evolution paths.Motivated by their derivation, we proceed here to a self-contained independent approach to the same problem with, we think, more far-reaching consequences.
To state the main ideas, let us begin by recalling a restricted version of the ergodic theorem for Markov chains ( [5]).Let us suppose that the system has a finite state space χ = {1, . . ., n} and that its statistical description is given by a stationary, time-homogeneous Markov chain.
Theorem 1 Let P ∈ M at(n) be a stochastic matrix with positive entries and denote with P N the N th power of P .Then there exists a unique distribution π which is stationary for P , i.e., P * π = π where P * denotes the transposed matrix of P .Moreover, It is immediate to realize from the above Theorem that the stochastic matrix P determines its stationary distribution, while the contrary is false.In this paper we put forth a selection criterion to choose among all stochastic matrices admitting a fixed distribution π as stationary.In a sense, we want to select a preferred dynamics for the approach to equilibrium π.To this, recall that if we take the stationary distribution π as the initial distribution, then the Markov chain (P, π) defines a discrete-time, finite-states stationary stochastic process.Let X i , i ≥ 0 be the χ-valued random variable which describe the state of the system at time i.For a stationary process, we can define its entropy rate (see [6]) as where Ω = χ N and H(X 0 , . . ., X N −1 ) = − ω∈Ω p(ω) ln p(ω) In a sense, the entropy rate H is the thermodynamic N -limit of the entropy of the system formed by all realizations of length N of the process.Therefore, we will call it the system entropy rate in the sequel.Moreover, for the stationary process associated to (P, π) it holds that (see [5], [6]) Furthermore, in agreement with (ii) above, we assume that the knowledge of the macroscopic fluxes acting on the system in a stationary non-equilibrium state will force constraints on the two-dimensional probability distribution in the form of constraints P ∈ Λ on the associated stochastic matrix (see Section 2. below).We can now state our selection criterion, which we call the Maximum Entropy Rate principle and amounts to the following constrained extremum problem: M.E.R.P. Given a positive probability distribution π, find the stochastic matrix P ∈ Λ which admits π as a stationary distribution and that has maximum entropy rate H( P ).
This principle can be seen as an instance of a Maximum Entropy principle for the case of constraints on two-dimensional distributions.Its justification is provided in [7] (see also [8], [9]) using a large deviation theory type estimate on the empirical second order distributions P, in the same spirit of Sanov' Theorem for first order empirical distributions.
Remark.In the case that the only constraints on P are the stationarity and normalization ones, Pij = π j , i, j = 1, . . ., n.The answer to this problem can be easily found by using the elementary inequality, see [6] H(P ) = − i,j with equality if and only if Pij = π j , i, j = 1, . . ., n. (all rows are equal).This result is a simple consequence of the fact that the entropy H(π) can be seen as the entropy rate of a stationary process with i.i.d.random variables X i ∼ π and that conditioning reduces the entropy rate.It is immediate to verify that P admits π as a stationary distribution.By recalling the ergodic theorem, we see also that P N = P for every N .The starting point of the above M.E.R. principle is the probability distribution π, which is, apart from being positive, completely arbitrary.It affords the statistical description of our system at equilibrium.In case we have some macroscopic information on the system, for example its average energy, that we represent as where E i is the energy of the system in state i, then this information can be used to select a probability distribution π among all these satisfying the above constraint.By applying the Maximum Entropy principle one gets the Gibbs probability distribution.Here the inverse temperature β = 1 kT is uniquely determined by the value e.

Model of the System in a Stationary Non-Equilibrium State
We begin with the model of the energy exchange between our system χ and another system A. Let us suppose that the system is described by our stationary Markov chain (P, π).We suppose also that the whole system A ∪ χ is energetically insulated, that is if X t = i and X t+1 = j, then the microscopic energy conservation law holds Let us denote with the skew-symmetric matrix of the difference of energy between states and with the joint probability at one time step.The average energy transfer between A and χ is and it follows easily that if π is stationary for P , and if our microscopic energy conservation law holds, then hence the stationary distribution π describes the system χ at macroscopic equilibrium with A.

Coupling of the system with two environments
We now want to model the coupling of the system with two environments A and B. As before, the statistical description of the system is given by a Markov chain.As a simplifying assumption, we suppose that at every time step along the evolution of the chain, the system χ is in contact (i.e., the microscopic energy conservation holds) with only one of the environments, alternatively A and B. Therefore, now the system is described by the non-homogeneous Markov chain (m ∈ N) We will see that it is not restrictive to suppose that the stochastic matrices A, B have positive entries.Also, notice that the chain "observed" at even times t = 2m is a time-homogeneous chain described by the positive stochastic matrix AB, hence the ergodic theorem applies.Let us denote with π the unique stationary distribution for AB.If π is the distribution describing the system χ at t = 0, then we have Therefore, the probability distribution describing the chain switches between π (at even times) and A * π (at odd times).Let us compute as before the joint probability at one and two time-steps starting from an even time t = 2m .We have Reasoning as before, we can compute the averaged energy differences Modifications of the above formulae for the case that we start the observation of the chain at an odd time are straightforward.By a simple inspection of the above equalities we can draw the following conclusions: and also Moreover, if π is a stationary distribution for AB and A, then it is a stationary distribution for B also and Therefore, with a distribution π stationary for AB and A we model a system at equilibrium with the two environments, and with a probability distribution π stationary for AB but not for A, we model a system in a stationary non equilibrium state.Moreover, the condition of stationarity of π with respect to AB, which is sufficient (but not necessary in the general case) for the balance of energy to hold, is the one that will allow to compute the entropy rate.The value of the macroscopic energy flux can be specified by the macroscopic constraint introduced by the first equality in (2.2) Note that, as before, we made no assumption on the distribution π.We have to choose matrices A and B satisfying the following constraints (here The above constraints do not specify in the general case the matrices A and B. Therefore, we need to invoke a selection criterion, which will be our M.E.R. principle.We are led to investigate the existence of an entropy rate for the stochastic process described by the non-homogeneous Markov chain (2.1) with initial distribution π stationary for AB.Before turning to this, we look at the solution of our problem in case of zero flux, q = 0.It is immediate to see that if we limit ourselves to the case A = B, then the chain is ergodic and A = B = P , where Pij = π j is the maximum entropy rate solution.

Computation of the entropy rate
It is easy to see that the chain described by (2.1) with initial distribution π stationary for AB but not for A describes a non-stationary stochastic process.Moreover, the chain is not strongly ergodic, but, with the non restricting assumption that A and B have positive entries it is weakly ergodic (see e.g., [10] for the notion of strong and weak ergodicity and [11], [12] and the bibliography therein for the study of convergence of Markov processes using information theoretic tools.)We now show that the entropy rate is well defined and finite for the process at hand.This is not surprising since the inhomogeneity of the chain is very mild and the chain will become homogeneous with a suitable time-reparameterization, However, we will proceed with the chain as it is for simplicity' sake.Recall that the probability of a typical sequence ω ∈ Ω = χ N of length N of the chain with initial distribution π is To compute H(X 0 , . . ., X N −1 ) we use the well known chain rule (see [6]):

Therefore, we have
Hence, if the condition of stationary (AB) * π = π holds, the terms following H(X 0 ) alternate between the form of H(X Hence,

Application of the M.E.R. principle and solution
In this section we solve the constrained extremum problem for the objective function H(A, B) subject to the constraints (2.4) using the Lagrange multipliers method.The Lagrange function is and the necessary condition for the extremum are By simple computations we get the expression for the solution A and B in terms of the unknown multipliers as Âij = e λ j In the following we use x, y, z, Q as unknowns in place of the Lagrange multipliers.By using the normality constraint on A and B respectively we get By using the stationarity constraint of π with respect to AB we get the following equations for z k , k = 1, . . ., n, Since the above introduced matrix Q is a stochastic one with positive entries j Qij = 1, it is easy to see that the solution of the above equation for z k is z k = π k .Hence, x j = 1 and from (2.7) Before turning to the inflow constraint to determine the multiplier β, we note that the matrix Q admits a simpler form using the definition e −βE j j e −βE j =: πj Now the inflow constraint can be rewritten as Hence, for any given π and q, setting e = E π (E), the Lagrange multiplier β = β(e + q) is uniquely determined by the equation We conclude by noting that, if the relation e = E π (E) is seen as a constraint for the unknown probability distribution π, then the maximum entropy assignation for π is the Gibbs distribution where β = β(e). (2.9) We have found that the maximum entropy rate assignation for the non-stationary stochastic process described by the non-homogeneous Markov chain (2.1) defined by the stochastic matrices A and B and by the probability distribution π which is stationary for AB is where π is the Gibbs distribution for β = β(q + e) and π is the Gibbs distribution for β = β(e).As expected, the solution depends only on the macroscopic information supplied: the equilibrium energy e and the flow q.

Entropy rate and entropy production
By a direct computation from (1.1), (2.8), (2.10), the entropy rate of the process is the sum of two terms of the type Hence, from (2.5) Since the chain spends "half of its time" in a state with average energy e and the remaining half in a state with average energy e + q, the above formula proposes that the entropy rate be the time average of its "instantaneous" entropies.Which is the relation between the entropy rate of the stochastic process and the thermodynamic entropy of the system in a stationary non-equilibrium state?If q is small, we can consider the Taylor expansion of H(q, e) w.r.t.q and get H(q, e) = H(0, e) + ∂H ∂q (0, e)q + O(q 2 ) = S(e) + 1 2 Hence, by the well known identification β = 1/kT , H(q, e) = S(e) + 1 2 β(e)q + O(q 2 ) = S(e) + 1 2 q kT (e) + O(q 2 ) (2.12) In the above formula, the entropy rate is the sum of two terms, one of which S(e) is non negative while the other has the sign of q.It is appealing to interpret the first as the source term an the other as the flux term.Moreover, from the relation between the average energy e and the related multiplier β(e), we have that, if q is small e + q = e(β) Hence, up to O(2) order terms,

Entropy production of the A ∪ χ ∪ B system
Let us consider the insulated system A ∪ χ ∪ B and let us suppose that the two environments A and B are two thermostats respectively at temperatures T = 1/kβ(e + q) and T = 1/kβ(e).The net effect of putting the system χ alternatively in contact with A and with B is the flow in two time steps of the chain of an average energy amount q > 0 from a reservoir at higher temperature T to a reservoir a lower temperature T leaving the system χ unchanged, since the equilibrium distribution π is stationary for AB.By a standard non-equilibrium thermodynamics formula (see e.g.[13]) the entropy production in the π If we now compute the information divergence (also called relative entropy, see [6]) of the Gibbs distribution π with respect to Gibbs distribution π we find The information divergence it is not a symmetric function of the two probability distributions p, q while the symmetrized information divergence (see [14]) of p and q is a symmetric and non negative one The standard interpretation (see again [14]) of the symmetrized information divergence is a measure of the difficulty of assessing which is the statistical description (p or q) of the system on the basis of the observations of the system state.Since we have the following Proposition 2 The entropy production of the closed system A ∪ χ ∪ B is equal to the symmetrized information divergence between the probability distributions π, π Remark.In the literature (see e.g the books [15], [16] or the papers [17], [18]) there is a well established notion of entropy production rate for a stationary Markov chain with countable state space.Let (P, π) be the transition matrix and its unique stationary distribution respectively.Then the entropy production rate, also called the information gain of the stationary chain with respect to its time reversal, is defined as e P = i,j (π i P ij − π j P ji ) ln π i P ij π j P ji .
We immediately see that if the the chain satisfy the detailed balance condition π i P ij − π j P ji = 0, then the entropy production rate is zero.Let us show that this is the case for our χ system described by the maximum entropy rate matrices A and B defined in (2.10).Indeed, the chain observed at even times is described by the transition matrix AB with entries (AB) ij = π j , whose stationary distribution is the same π in (2.9), and this stationary Markov chain satisfy trivially the detailed balance condition.Therefore, the entropy production of the overall system A ∪ χ ∪ B is exclusively due to the energy exchange between the two reservoirs A and B as computed above.

Generalizations
In this section, we generalize the previous results to the case of a open system with k macroscopic observables E α with possibly different speed of relaxation to equilibrium and k stationary fluxes q α .

The case of k isochronous fluxes and Onsager' reciprocity relations
The considerations in Sections 2. and 2.1.developed for E apply without changes for every macroscopic observable E α .We consider therefore our M.E.R. problem with k constraints of the form (2.4).Since the constraints specifying the values q α of the k inflows are linear and independent, the last term in the r.h.s. of the Lagrange function The considerations and the results of Section 2.4.can be reformulated without changes for the vector of fluxes q = (q 1 , . . ., q k ) and equilibrium values e = (e 1 , . . ., e k ).In particular, the linear approximation of small fluxes formula (2.12) becomes H(q) = S(e) + 1 2 α β α (e)q α + O(q 2 ) (3.

The case of different speed of relaxation to equilibrium
In this section we sketch how to model a system where there are two observables X, Y with different speed of relaxation to equilibrium.As before, let the system χ be alternatively in contact with the two of the macroscopic fluxes traversing the system through a maximum entropy rate principle.With respect to existing applications of the M.E.R. principle, here we have introduced a Markov chain defined by two or more stochastic matrices.In this way we are able to describe a system in a stationary non-equilibrium state that may exhibit different speed of relaxation to equilibrium.We made reference to physical quantities such as the energy for the ease of interpretation of the results, but this is not necessary and the same model can be applied in other domains, such as economy or biology, as it is sufficiently general to take into account macroscopic constraints of different nature.