Open Markov processes: A compositional perspective on non-equilibrium steady states in biology

In recent work, Baez, Fong and the author introduced a framework for describing Markov processes equipped with a detailed balanced equilibrium as open systems of a certain type. These `open Markov processes' serve as the building blocks for more complicated processes. In this paper, we describe the potential application of this framework in the modeling of biological systems as open systems maintained away from equilibrium. We show that non-equilibrium steady states emerge in open systems of this type, even when the rates of the underlying process are such that a detailed balanced equilibrium is permitted. It is shown that these non-equilibrium steady states minimize a quadratic form which we call `dissipation.' In some circumstances, the dissipation is approximately equal to the rate of change of relative entropy plus a correction term. On the other hand, Prigogine's principle of minimum entropy production generally fails for non-equilibrium steady states. We use a simple model of membrane transport to illustrate these concepts.


ward thermody
amic equilibrium.Open systems can be maintained away from equilibrium via the exchange of energy and matter with the environment.In addition, biological systems typically consist of a large number of interacting parts.This paper presents a way of describing these 'parts' as morphisms in a category.A category consists of a collection of objects along with morphisms or arrows between objects, obeying certain conditions.We consider time-homogeneous Markov processes as a general framework for modeling various biological and biochemical systems whose dynamical equations are linear.Viewed as morphisms in a category, the 'open Markov processes' discussed in this paper provide a framework for describing open systems which can be combined to build larger systems.

Intuitively, one can think of a Markov process as specifying the dynamics of a probabilit or 'population' distribution that is spread across a finite set of states.A population distribution is a non-normalized probability distribution, see for example [15].The population of a particular state can be any non-negative real number.The total population in an open Markov process is not constant in time as population can flow in and out through certain boundary states.Part of the utility of Markov processes as models of physical or biological systems stems from the flexibility in choosing the correspondence between the states of the Markov process and the actual system it is to model.For instance, the states of a Markov process could correspond to different internal states of a particular molecule or chemical species.In this case, the transition rates describe the rates at which the molecule transitions among these states.Or, the states of a Markov process could correspond to a molecule's physical location.In this case, the transition rates encode the rates at which that molecule moves from place to place.

This paper is structured as follows.In Section 2 we give some preliminary definitions from the theory of Markov processes and explain the concept of an open Markov process.In Section 3 we introduce a model of membrane transport as a simple example of an open Markov process.In Section 4, we introduce the category DetBalMark.The objects in DetBalMark are finite sets of 'states' whose elements are labeled by non-negative real numbers which we call 'populations'.The morphisms in DetBalMark are Markov processes equipped with a detailed balanced equilibrium distribution as well as maps specifying input and output states.If the outputs of one process match the inputs of another process the two can be composed, yielding a new open Markov process.We refer to the union of the input and output states as the boundary of an open Markov process.

In Section 5, we show that if the populations at the boundary of an open detailed balanced Markov proce s are held fixed, then the non-equilibrium steady states which emerge minimize a quadratic form, which we call the 'dissipation,' subject to the constraint on the boundary populations.Depending on the values of the boundary populations these non-equilibrium steady states can exist arbitrarily far from the detailed balanced equilibrium of the underlying Markov process.In recent work [4], Baez, Fong and the author construct a functor : DetBalMark → LinRel from the category of open detailed balanced Markov process to the category of linear relations.

Applied to an open detailed balanced Markov process, this functor yields the subset of allowed steady sta e boundary population-flow pairs, providing an effective 'black-boxing' of open detailed balanced Markov processes.In Section 6 we show that, for fixed boundary populations, this principle of minimum dissipation approximates Prigogine's principle of minimum entropy production in the neighborhood of equilibrium plus a correction term involving only the flow of relative entropy through the boundary of the open Markov process.


Open Markov processes

In this section we define open Markov processes, describe the detailed balanced co

ition for equilibria a
d define non-equilibrium steady states for Markov processes.

An open Markov process, or open continuous time, discrete state Markov chain, is a triple (V, B, H) where is a finite set of states, B ⊆ V is the subset of boundary states and H : R V → R V is an infinitesimal stochastic Hamiltonian
H ij ≥ 0, i = j i H ij = 0.
For each i ∈ V the dynamica equation
dp i dt = j H ij p j , i ∈ V − B p i (t) = b i (t), i ∈ B.
The off-diagonal entries H ij , i = j are the rates at which population transitions from the j th to the i th state.A steady state di distribution which is constant in time: In a closed Markov process the total population is conserved:
dp i dt = 0 for all i ∈ V. A closedi dp i dt = i,j
H ij p j = 0, enabling one to talk about the relative probabilities of b ing in particular states.A steady-state distribution in a clos called an equilibrium.We say an equilibrium q ∈ [0, ∞) V of a Markov process is detailed balanced if
H ij q j = H ji q i for all i, j ∈ V.
An open detailed balanced Markov process is an open Markov at the populations of all states in a detailed balanced equilibrium are non-zero.For a pair of distinct states i, j ∈ V , the term H ij p j is the flow of population from j to i.The net flow of population from the j th state to the i th is
J ij (p) = H ij p j − H ji p i .
Summing the net flows into a particular state we can define the net inflow J i (p) ∈ R of a particular state to be
J i (p) p i .
Since j H ji p i = 0, the right side of this equation is the time derivative of the population at the i th st j (p) or J i (p) we have
dp i dt = j J ij (p) = J i (p).
The net flow between each pair of states vanishes identically in a detailed balanced equilibrium q:

J ij (q) = 0.

The exist nce of a detailed balanced equilibrium is equivalent to a condition on the rates of a Markov process due known as Kolmogorov's criterion [14], namely that
H i1i2 H i nite sequence of states i 1 , i 2 , . . ., i n of any length.This condition says that the product of the rates along any cycle is equal to the product of the rates along the same cycle in the reverse direction.

A non-equil brium steady state is a steady state in which the net flow between at least one pair of states is non-zero.Thus there could be population flowing between pairs of states, but in such a way that these flows still yield constant populations at all states.In a closed Markov process the existence of non-equilibrium steady states requires that the rates of the Markov process violate Kolmogorov's criterion.We show that open Markov processes with constant boundary populations admit non-equilibrium steady states even when the rates of the process satisfy Kolmogorov's criterion.Throughout this paper we use the term equilibrium to mean detailed balanced equilibrium.


Membrane diffu

on as an open Markov process

To illustrate t
ese ideas, we consider a simple model of the diffusion of neutral particles across a membrane as an open detailed balanced Markov process with three states V = {A, B, C}, input A and output C. The states A and C correspond to the each side of the membrane, while B corresponds within the membrane itself, see Figure 1.In this model, p A is the number of particles on one side of the membrane, p B the number of particles within the membrane and p C the number of particles on the other side of the membrane.The off-diagonal entires in the Hamiltonian H ij , i = j are the rates at which population hops from j to i.For example H AB is the rate at which population moves from B to A, or from inside the membrane to the top of the membrane.Let us assume that the membrane is symmetric in the sense that the rate at which particles hop from outside of the membrane to the interior is the same on either side, i.e.H BA = H BC = H in and H AB = H CB = H out .We can draw such an open Markov process as a labeled graph:
q A q B q C q A q C
Hi

e la
els on the

dges
are the corresponding transition rates.The states are labeled by their detailed balanced equilibrium populations, which, up to an overall scaling, are given by q A = q C = H in H out and q B = H 2 in .Suppose the populations p A and p C are externally maintained at constant values, i.e. whenever a particle diffuses from outside the cell into the membrane, the environment around the cell provides another particle and similarly when particles move from inside the membrane to the outside.We call (p A , p C ) the boundary populations.Given the values of p A and p C , the steady state population p B compatible with these values is
p B = H in p A + H in p C −H is steady state population minimizes the dissipation, subject to the constraints on p A and p C .We thus have a non-equilibrium steady state p = (p A , p B , p C ) with p B given in terms of the boundary populations above.From these values we can compute the boundary flows, J A , J C as
J A = j J Aj (p) = H out p B − ry populations this gives
J A = H in (p C − p A )2
and
J hat there is a constant net flow through the open Markov process.

As one would expect, if p A > C there is a positive flow from A to C and vice-versa.Of course, in actual membranes there exist much more complex transport mechanisms than the simple diffusion model presented here.A number of authors have modeled more complicated transport phenomena using the framework of networked master equation systems [20,30].

In our framework, we call the collection of all boundary population-flows pairs the steady state 'behavior' of the open Markov process.The main theorem of [4] constructs a functor from the category of open detailed balanced Markov process to the category of linear relations.Applied to an open detailed balanced Markov process, this functor yields the set of allowed steady state boundary population-flow pairs.One can imagine a situation in which only the populations and flows of boundary states are observable, thus characterizing a process in terms of its behavior.This provides an effective 'black-boxing' of open detailed balanced Markov processes.

As morphisms in a category, open de ailed balanced Markov processes can be composed, thereby building up more complex processes from these open building blocks.The fact that 'black-boxing' is accomplished via a functor means that the behavior of a composite Markov process can be built up from the composite behaviors of the open Markov processes from which it is built.In this paper we illustrate how this framework can be utilized to study linear master equation systems far from equilibrium with a particular emphasis on the modeling of biological phenomena.

Markovian or master equation systems ave a long history of being used to model and understand biological systems.We make no attempt to provide a complete review of this line of work.Schnakenberg, in his paper on networked master equation systems, defines the entropy production in a Markov process and shows that a quantity related to entropy serves as a Lyapunov function for master equation systems [29].His book [30] provides a number of biochemical applications of networked master equation systems.Oster, Perelson and Katchalsky developed a theory of 'networked thermodynamics' [19], which they went on to apply to the study of biological systems [20].Following the untimely passing of Katchalsky, Perelson and Oster went on to extend this work into the realm of chemical reactions [21].

Starting in the 1970's, T. L. Hill spearhe ded a line of research focused on what he called 'free energy transduction' in biology.A shortened and updated form of his 1977 text on the subject [10] was republished in 2005 [11].Hill applied various techniques, such as the use of the cycle basis, in the analysis of biological systems.His model of muscle contraction provides one example [12].

One quantity central to the study of non-equi ibrium systems is the rate of entropy production [9,24,18,8].Prigogine's principle of minimum entropy production [25] asserts that for nonequilibrium steady states that are near equilibrium, entropy production is minimized.This is an approximate principle that is obtained by linearizing the relevant equations about an equilibrium state.In fact, for open Markov processes, non-equilibrium steady states are governed by a different minimum principle that holds exactly, arbitrarily far from equilibrium.We show that for fixed boundary conditions, non-equilibrium steady states minimize a quantity we call 'dissipation'.If the populations of the non-equilibrium steady state are close to the population of the underlying detailed balanced equilibrium, one can show that dissipation is close to the rate of change of relative entropy plus a boundary term.Dissipation is in fact related to the Glansdorff-Prigogine criterion which states that a non-equilibrium steady state is stable if the second order variation of the entropy production is non-negative [8,29].

Starting in the 1990's, the Qians and their collabo ators developed a school studying nonequilibrium steady states, publishing a number of articles and books on the topic [13].More recently, results concerning fluctuations have been extended to master equation systems [1].In the past two decades, Hong Qian of the University of Washington and collaborators have published numerous results on non-equilibrium thermodynamics, biology and related topics [26,27,28].

This paper is part of a larger project which uses cat gory theory to unify a variety of diagrammatic approaches found across the sciences including, but not limited to, electrical circuits, control theory and bond graphs [3,2].We hope that the categorical approach will shed new light on each of these subjects as well as their interrelation, particularly as we generalize the results presented in this and recent papers to the more general, non-linear, setting of open chemical reaction networks.


The category of open detailed balanced Markov process

In this section we describe how open detailed balanced
Markov processes are the morphisms in a certain type of symmetric, monoidal, dagger-compact category.In previous work, Baez, Fong and the author [4] used the framework of decorated cospans [7] to construct the category DetBalMark.

Here we give an intuitive description of this category nd refer to those papers for the mathematical details.

An object in DetBalMark is a finite set with population , i.e. a finite set X together with a map p X : X → [0, ∞) assigning a population
p i ∈ [0, ∞) to each element i ∈ X. A morphism M : (X, p X ) → (Y, p Y ) consists
of an open th input and output maps i : (X, p X ) → (V, q) and o : (Y, p Y ) → (V, q) which preserve population so that p X = iq and p Y = oq.The boundary B ⊆ V of an open Markov process is the union of the images of the input and output maps
B = i(X) ∪ o(Y ).
One can draw an open detailed balanced Markov process as a labeled directed graph whose vertices are labeled by their equilibrium populations and with specified subsets of the ver n open detailed balanced Markov process, which we now think of as a morphism from the input X = {A} to the output Y = {C}:
q A q B q C X Y q A q C HBA HAB HCB HBC i o
This is a morphism in DetBalMark from X to Y where X simple example, X and Y both contain a single element, namely A and C respectively.Suppose we had another such membrane as depicted in Figure 2.This is a morphism in DetBalMark from with input Y = {C ′ } and output Z = {E}.Two open detailed balanced Markov processes can be composed if the detailed balanced equilibrium populations at the outputs of one match the detailed balanced equilibrium populations at the inputs of the other.This requirement guarantees that the composite of two open detailed balanced Markov process still admits a detailed balanced equilibrium.
q A q B q C X Y q A q C HBA HAB HCB HBC q C ′ q D q E Y Z q C ′ q E HDE HED H C ′ D H DC ′
If q C = q C ′ in our two membrane models we can compose them by identifying C with C ′ to yield an open detailed balanced Markov process

odeling the diffusion of neutral
articles across membranes arranged in series:
q A q B q C X q A HBA HAB HCB HBC q D q E Z q E HDC HCD HED HDE
Notice that the states corresponding to C and C ′ in

ach process ha
e been identified and become internal states in the composite which is a morphism from X = {A} to Z = {E}.This open Marko ffusion across two membranes in series, see Figure 3.One can 'black-box' an open detailed balanced Markov process by converting it into an electrical circuit, applying the already known black-boxing 3] and translating the result bac Markov processes [4].The key step in this process is the construction of a quadratic form which we call 'dissipation', analogous to power in electrical circuits, which is minimized when the kov process are in a steady state.


Principle of minimum dissipation

Here we show that by externally fixing the populations at boundary states, one induces st ady states which minimize a quadratic form which we call 'dissipation.'Definition 1.Given an open detailed balanced Markov process we define the dissipation functional of a population distribution p to be
D(p) = 1 2 i,j H ij q j p j q j − p i q i 2 .
Given boundary populations b ∈ R B , we can minimize this functional over all p which agree on the boundary.Differentiating the dissipation functional with respect to an internal population, we get
∂D(p) ∂p n = −2 j H nj p j q n .
Multiplying by

n 2 yields q n 2
∂D(p) ∂p n = − j H
nj p j ,
where we recognize the right-hand side from the open master equation for internal states.We see that for fixed boundary populations, the conditions for p to be a steady state, namely that Proof.This follows from Theorem 28 in [4].
dp i dt = 0 for all i ∈ V,
Given specified boundary populations, one can compute the steady state boundary flows by minimizing the dissipation subject to the boundary conditions.Indeed, there is a functor : DetBalMark → LinRel which maps open detailed balanced Markov processes to their steady state behaviors.This is the main result of our previous paper [4].The fact that this is a functor means that the behavior of a composit open detailed balanced Markov process can be computed as t .


Dissipation and Entropy Production

In the last section, we saw that non-equilibrium steady states with fixed boundary populations minimize the dissipation.In this section we relate the dissipation to a divergence between population distributions known in various circles as the relative entropy, relative information or the Kullback-Leibler divergence.The relative entropy is not symmetric and violates the triangle inequality, which is why it is called a 'divergence' rather than a metric, or distance function.We how that for population distributions near a detailed balanced equilibrium, the rate of change of the relative entropy is approximately equal to the dissipation plus a 'boundary term'.

The relative entropy of two distributions p, q is given by
I(p, q) = i p ing a detailed balanced equilibrium, the relative entropy with respect to this detailed balanced equilibrium distribution is monotonically decreasing with time, see for instance [14].There is an unfortunate sign convention in the definition of relative entropy: while entropy is typically increasing, relative entropy typicall in a closed Markov process.

In an open Markov process, the sign of the rate of change of relative entropy is indeterminate.Consider an open Markov process (V, B, H).For any two population distributions p(t) a the quantities
Dp i Dt = dp i dt − j∈V H ij p j and Dq i Dt = dq i dt − j∈V H ij q j ,
which measure how much the time derivatives of p(t) and q(t) fail to obey the master equation.Notice that Dpi Dt = 0 for i ∈ V − B, as the populations of internal states evolve according to the master equation.In terms o n be written as
d dt I(p(t), q(t)) = i,j∈V H ij p j ln p i q i − p ive entropy for a closed Markov process.Th [5,23].Thus, the rate of change of relative entropy ocess satisfies
d dt I(p(t), q(t)) ≤ i∈B Dp i Dt ∂I ∂p i + Dq i Dt ∂I ∂q i .
This inequality tells us that the rate of change of relative entropy in an open Markov processes is bounded by the rate at which relative entropy flows through its boundary.If q is an equilibrium solution of the master equation dq dt = Hq = 0, then the rate of change of relative entropy can be written as
d dt I(p(t), q) = i,j∈V (H ij p j − H ji p i ) ln p i q j q i p j + i∈B Dp i Dt ∂I ∂p i
Furthermore, if q satisfies detailed balance we can write this as
d dt I(p(t), q) = − 1 2 i,j∈V J ij A ij + i∈B Dp i Dt ∂I ∂p i .,
where
J ij (p) = H ij p j − H ji p i
is the thermodynamic flux from j to i and
A ij (p) = ln H ij p j H ji p i
is the conjugate thermodynamic force.This quan ity:
1 2 i,j∈V J ij A ij
is what Schnakenberg calls "the rate of entropy production" [29].This is always non-negative.Note that due to the sign convention in the definition of relative entropy, in the absence of the boundary term, a positive rate of entropy production corresponds to a decreasing relative entropy.

We shall shortly relate the rate of change of relative entropy to the dissipation for open detailed balanced Markov processes, but first let us consider the quantity A ij (p).It is the entropy production per unit flow from j to i.If J ij (p) > 0, i.e. if there is a positive net flow of population from j to i, then A ij (p) > 0. In addition, J ij (p) = 0 implies that A ij (p) = 0. T of entropy production is, by definition, non-negative.

In the realm of population dynamics, we can understand A ij (p) as the force resulting from a difference in chemical potential.Let us elaborate on this point to clarify the relation of our framework to the language of chemical potentials used in non-equi dealing with only a single type of molecule or chemical species.The states could correspond to different locations of the molecule, as in our example of membrane transport.Another possibility is that each state correspond to a different internal configuration of the molecule.In this setting the chemical potential µ chemical species in the following way:
µ i = µ o i + T ln(c i )
, where T is the te e and µ o i is the standard chemical potential.The difference in chemical potential between two states gives the force associated with the flow of population which seeks to reduce this difference in chemical potential
µ j − µ i = µ o j − µ o i + T ln c j c i .
In gen state is proportional to the population of t olume of the system c i = pi V .In this case, the volumes cancel out in the ratio of concentrat lation between chemical potential differences and population differences:
µ j − µ i = µ o j and we have 0 = µ o j − µ o i + T ln q j q i , or that
q j q i = e − µ o j −µ o i T .
If q sat lance, then this also gives an expression for the ratio of the transition rates Hji Hij in terms of the standard chemical potentials.Thus we can translate between difference as the force A ij (p) times the temperature of the system T :
µ j − µ i T = A ij (p).
Let us return to our expression for d p j − H ji p i ) ln q i p j q j p i + i∈B Dp i Dt ∂I ∂p i .
Consider the situatio the sense that
p i q i = 1 + ǫ i
where ǫ i ∈ R is the deviation in the ratio pi qi from unity.We collect these deviations in a vector denoted by ǫ.Expanding the logarithm to first order in ǫ we have that
d dt I(p(t), q) = − 1 2 i,j∈V (H ij p j − H ji p i ) (ǫ j − ǫ i ) + i∈B Dp es
d dt I(p(t), q) = − 1 2 i,j∈V (H ij p j − H ji p i ) p j q j − p i q s of order ǫ 2 i .When q is a detailed balanced equilibrium we can rewrite this quantity as
d dt I(p(t), q) = − 1 2 i,j H ij q j p j q j − p i q i 2 + i∈B Dp i Dt ∂I ∂p i + O(ǫ 2 ).
We re yields
d dt I(p(t), q) = −D(p) + i∈B Dp i Dt ∂I ∂p i + O(ǫ 2 ).
We se ation approximately minimizes the rate of decrease of relative entropy plus a term which depends on the boundary populations.In the case that boundary populations are held fixed so that dpi dt = 0, i ∈ B, we have that
Dp i Dt = − j∈V H ij p j , i ∈ B.
In this case, the rate of change of relative entropy can be written as
d dt I(p(t), q) = i∈V −B p i q i dp i dt + 2 i∈B Dp i Dt + O(ǫ 2 ).
Summarizing the results of this section, we have that for p arbitrarily far from the detailed b

anced equilibrium equilibrium q, the rate of relative
ntropy reduction can be written as
dI(p(t), q) dt = − 1 2 i,j J ij (p)A ij (p) + i∈B Dp i Dt ∂I ∂p i .
For p in the vicinity of a detailed balanced equilibrium we have that
dI(p(t), q) dt O(ǫ 2 )
where D(p) is the dissipation and ǫ i = pi qi − 1 measures the deviations of the populations p i from their equilibrium values.We have seen that in a non-equilibrium steady state with fixed boundary populations, dissipation is minimized.We showed that for steady states near equilibirum, the rate of change of relative ent y equal to minus the dissipation plus a boundary term.Minimum dissipation coincides with minimum entropy production onl etween populations which minimize dissipation and those which minimize entropy production:
q implicity, we have set all transition rates equal to one.In this case, the detailed balance equilibrium distribution is .If the populations p A and p C are externally fixed, then the population p B which minimizes the di the arithmetic mean of the boundary populations
p B = p A + p C 2 .
The rate of change of the relative entropy I(p(t), q) where q is the uniform detailed balanced equilibrium is given by
d dt I(p(t), q) = −(p A − p B ) ln( p A p B ) − (p B − p C ) ln( p B p C ) − 1 2 i,j∈V Jij Aij + (p A − p B )(ln(p A ) + 1) + (p C − p B )(ln(p C ) + 1) i∈B Dp i Dt ∂I ∂p i
.

Differentiating this quantity with respect to p B for fixed p A and p C yields the condition
p A + p C 2p B − ln(p B ) − 2 = 0.
The solution of this equation gives the population p B which extremizes the rate o

change of r
lative entropy, namely
p B = p A + p C 2W(pA+pC ) 2 e 2
, where W (x) is the Lambert W -function or the omega function which satisfies the following relation x = W (x)e W (x) .

The Lambert W -function is defined for x ≥ −1 e and double valued for x ∈ [ −1 e , This simple example illustrates the difference between distributions which minimize dissipation subject to boundary constraints and those which extremize the rate of change of relative entropy.For fixed boundary populations, dissipation is minimized in steady states arbitrarily far from equilibrium.For steady states in the neighborhood of the detailed balanced equilibrium, the rate of change of relative entropy is approximately equal to minus the dissipation plus a boundary term.


Discussion

Treating Markov processes as morphisms in a category leads naturally to open systems which admit non-equilibrium steady states, even when the transition rates of the underlying process satisfy Kolmogorov's criterion.Microscopically, all reactions should be reversible with perhaps a large disparity between the forward and reverse rates.Nonetheless, it is clear that biological organisms are capable, at least locally, of storing free energy.This is typically accomplished via the interaction with other systems or the environment.In this paper, the environment served as a reservoir maintaining boundary populations at constant values.Since open Markov processes are morphisms in the category DetBalMark, one can compose these open systems, thereby building up complicated systems in a systematic way.We saw that the non-equilibrium steady states which emerge minimize a quadratic form which depends on the deviation of the steady state populations from the populations of the underlying detailed balanced equilibrium.For steady states in the neighborhood of equilibrium, we saw that the dissipation is in fact the linear approximation of the rate of change of relativ