Nonequilibrium Thermodynamics in Biochemical Systems and Its Application

Abstract

Living systems are open systems, where the laws of nonequilibrium thermodynamics play the important role. Therefore, studying living systems from a nonequilibrium thermodynamic aspect is interesting and useful. In this review, we briefly introduce the history and current development of nonequilibrium thermodynamics, especially that in biochemical systems. We first introduce historically how people realized the importance to study biological systems in the thermodynamic point of view. We then introduce the development of stochastic thermodynamics, especially three landmarks: Jarzynski equality, Crooks’ fluctuation theorem and thermodynamic uncertainty relation. We also summarize the current theoretical framework for stochastic thermodynamics in biochemical reaction networks, especially the thermodynamic concepts and instruments at nonequilibrium steady state. Finally, we show two applications and research paradigms for thermodynamic study in biological systems.

1. Introduction

How should a living system, e.g., a cell, be described from a physical aspect? In 1943, Erwin Schrödinger provided a systematic point of view in his speech, which explained the basic physical logic of cellular activities. The content, which was quite novel and interesting to people especially those with physical background, was later summarized into a book named What is life [1] and became famous worldwide. In this book, he proposed that, the key difference between a living cell and a “dead” system is its ORDER. In a living cell, countless biochemical reactions occur regularly, orderly and endlessly, which would not happen in a non-living system. For a system without living activity, for example, a box of gas mixture, when put into a fixed environment its intrinsic reactions will soon enter the inactive, stable, orderless, dead thermodynamic equilibrium state where all the ordered intrinsic activities will stop and the entropy of the system reaches maximum, whereas living system maintains an ordered and relatively steady state and never reaches equilibrium. Why could this ordered steady state for living systems be maintained? According to the 2nd law of thermodynamics, the entropy of an isolated system would never decrease. Hence, as Schrödinger pointed out, the key reason is that living cells exchange energy and materials with external environment, absorb necessary energy and nutrients for biochemical reactions and expel waste and heat from the reactions. In terms of statistical physics, the system is maintained at a low entropy state to make a living, by absorbing “negative entropy” from environment to nullify the entropy production from its necessary biological activities.

Schrödinger’s point of view had great impact on biology and physics. It has been widely accepted now that metabolism, which describes how a living system continuously consumes necessary nutrients and energy taken from environment for life-sustaining, is the essence of living activities. On the other hand, this point of view first explicitly connected biological phenomenon with the theory of statistical physics, hence drew a large group of statistical physicists’ attention to systems far from equilibrium, i.e., nonequilibrium systems. Despite the success in explaining various non-living phenomena, the traditional equilibrium and near-equilibrium statistical physics had long been difficult to use in living system. After Schrödinger proposed his idea, physicists started to realize that open nonequilibrium systems (represented by living systems) are quite complicated and a new angle was needed.

Another succeeding revolutionary idea comes from the Brussels School. During 1950s, the founder of Brussels School Prigogine and some other researchers discovered the so-called “minimal entropy production principle” [2,3,4] for systems near equilibrium. This principle declares that for systems near equilibrium where the response to thermodynamic force is linear, the steady state (under a certain thermodynamic force) would be the state where entropy production rate takes minimum. However, this principle cannot be used to explain some complicated nonequilibrium phenomena, for example, Rayleigh-Bénard convection [5,6,7,8] and Belousov–Zhabotinsky reaction oscillation [9,10,11,12,13]. These systems are maintained at a highly ordered state, so the entropy production in these systems must be high and should not be at minimum. One common characteristic in these systems is that the response to thermodynamic force is highly nonlinear, and this nonlinearity is why minimal entropy production principle does not apply and ordered patterns arise.

Since Prigogine and colleges found that it’s impossible to apply the minimal entropy production principle to systems arbitrarily far from equilibrium, they proposed the famous dissipative structure theory [14,15]: for systems that are far from equilibrium, the response functions to various thermodynamic forces go beyond linear regime described by linear response theory and arrive at the nonlinear regime. In this regime, the system may be maintained at a relatively steady state (the nonequilibrium steady state) by dissipating free energy and producing entropy, which can exhibit complex and ordered dynamics.

Dissipative structure theory, combined with Schrödinger’s novel idea, pointed out several fundamental requirements for ordered systems including living systems in terms of statistical physics:

• The system must be open. By exchanging energy with external environment, a system’s ordered steady state is maintained with free energy dissipated and entropy (produced by nonequilibrium activities) expelled. In living systems and other chemical reaction systems, mass exchange is also needed in form of chemical reactants
• The system must be driven far enough from equilibrium. If the system is at or near equilibrium, it would be described by minimal entropy production principle and not be available for a steady ordered state.
• The nonlinearity in the system must be strong enough. Week nonlinearity would not lead to a complex and non-trivial dynamics.

When the above requirements are satisfied, self-organized steady ordered structure, i.e., dissipative structure, may arise.

Prigogine’s theory shed light on the research on thermodynamics for system far from equilibrium and its relation with complex nonlinear systems including living systems. With a series of proceeding work since 1980s on nonequilibrium thermodynamics, it’s now possible to explore the thermodynamic rules within living systems.

2. The Development of Nonequilibrium Statistical Physics and Stochastic Thermodynamics

Modern statistical mechanics is based on stochastic theory, where the dynamics of a system is described by stochastic process [16,17]. When the system is near equilibrium, its dynamics is simple fluctuation near the equilibrium state. As introduced above, it has been quite clear for long to physicsts how to describe such fluctuation. However, it was hard for physicists to discover the universal rules applicable to systems arbitrarily far from equilibrium, until recent several decades. The landmark results are the so-called Jarzynski equality [18,19], a series of fluctuation theorems [20,21,22,23] among which the most famous one is Crooks’ fluctuation theorem [21], and the so-called thermodynamic uncertainty relation (TUR) [24,25,26,27].

Jarzynski equality describes the relation between the work done and the free energy change during a general isothermal process. When the system is at a unique certain macroscopic state A, its Hamiltonian $H_A$ is also fixed. When put into a heat bath with temperature T, according to equilibrium statistical physics, it corresponds to one certain equilibrium state whose probability of each microscopic state follows Boltzmann distribution, $p_s^A=Z_A^{-1}exp[-\beta H_A\left(s\right)]$, with $H_A\left(s\right)$ the Hamiltonian for the microscopic state s, $Z_A={\sum }_{s}exp[-\beta H_A\left(s\right)]$ the partition function, and $\beta ={\left(k_{B}T\right)}^{-1}$. The free energy at equilibrium state $F_A=k_{B}Tln{Z}_A$ is also fixed. When the macroscopic state is changed to another macroscopic state B by doing work, e.g., pulling a spring, pushing a piston, or changing the conformation of a protein complex, the Hamiltonian would be changed to $H_B$. Same as above, macroscopic state B also corresponds to a unique certain equilibrium state with free energy $F_B=k_{B}Tln{Z}_B$. According to the 2nd law, in thermodynamic limit, the work done during this process should no smaller than their free energy difference $\mathsf{\Delta }F=F_B-F_A$. However, thermodynamic limit only applies in ensemble sense. For each single realization of this stochastic process, the initial state of the system follows the Boltzmann distribution, and is affected by noise from heat bath. Hence during this process, the system would not always follow the same trajectory in phase space, and the work done in each single realization W may also be different. Previous study including the 2nd law actually just stated that the average work done $\langle W\rangle \ge \mathsf{\Delta }F$. Jarzynski pointed out that, we should focus not only the average work done $\langle W\rangle$, but also the work distribution for each realization, and he found the following equality:

$$\langle e^{-\beta W}\rangle =e^{-\beta \mathsf{\Delta }F}.$$

(1)

It could also be easily checked with Jensen’s inequality that Equation (1) simply implies the 2nd law $\langle W\rangle \ge \mathsf{\Delta }F$.

Jarzynski equality is a great breakthrough to 2nd law, and was confirmed by experiments in 2000s [28,29,30]. For hundreds of years, Jarzynski equality for the first time shows how the 2nd law rules thermodynamic processes in the form of equality and beyond its previous inequality form, and is applicable to systems arbitrarily far from equilibrium.

Crooks’ fluctuation theorem describes the relation between the probability of a trajectory in phase space and the corresponding entropy production [21]. Suppose there is an amount of entropy production $\omega$ along one trajectory (forward trajectory), then the corresponding reverse process (reverse trajectory) would naturally produce $-\omega$ entropy (absorb $\omega$ entropy). Denote the probability of the forward trajectory as $P_F\left(\omega \right)$ and the probability of reverse trajectory as $P_R(-\omega )$. When $\omega >0$, according to the 2nd law, in thermodynamic limit $P_R(-\omega )/P_F\left(\omega \right)\to 0$, i.e., the process that decreases the overall entropy would never occur. However, in small systems where stochasticity is not negligible, it’s possible to occur with a small probability. This stochasticity is described by Crooks’ fluctuation theorem, which declares that in small systems the ratio $P_R(-\omega )/P_F\left(\omega \right)$ is exponentially suppressed by $\omega$, i.e.,

$$\frac{P_R(-\omega )}{P_F\left(\omega \right)}=e^{-\omega /k_B}.$$

(2)

According to this formula, the larger $\omega$ is, the more impossible would the reverse process be. This is compatible with the 2nd law, since $\omega$ is an extensive quantity and is proportional to the particle number. For a “macroscopic” trajectory, where the particle involved in is around ∼1 mol, the entropy production is in the order of ∼$1\mathrm{mol}\times N_A{k}_B$∼${10}^{23}{k}_B$, so $\omega /k_B$ is very large. In this sense, $e^{-\omega /k_B}\approx 0$, and the probability of the reverse process is almost impossible. For a small system, $\omega$ is finite, so the process that “seemingly violate” the 2nd law would occur more frequently. Such fluctuation resulted from stochasticity is why “fluctuation theorem” gets its name. This quantitative relation between the fluctuation probability and the corresponding entropy production has been confirmed by experiments [31,32,33].

Crooks’ fluctuation theorem, just like Jarzynski equality, is also a breakthrough to the 2nd law. Actually, by considering Jarzynski equality, Crooks’ fluctuation theorem can be expressed in a simpler form:

$$\langle e^{-\omega /k_B}\rangle =1.$$

(3)

These theories are the most breakthrough in recent 30 years. For the first time, physicists could be able to explore laws in statistical physics far away from equilibrium with these theoretical instruments. They also indicate that, only the statistical laws in mesoscopic systems, where system are small so fluctuation is not negligible but not too small for statistical physics to fail, are fundamental.

Seifert and some proceeding researchers extend these theories together with definition of some other thermodynamic variables to the motion of single particle. They calculated the entropy production rate in this case and proved the corresponding fluctuation theorem [34]:

$$\langle e^{-\mathsf{\Delta }{s}_{tot}/k_B}\rangle =1,$$

(4)

where $\mathsf{\Delta }{s}_{tot}$ is the entropy production for single particle along a trajectory. This is the same with Crooks’ fluctuation theorem. Meanwhile, there are some other forms of fluctuation theorems, which are catalogued in a review from Seifert [23].

After these work, statistical physics can be applied to a much wider range of studies. Researchers now are able to discuss the systems not only at or near equilibrium, but also the systems arbitrarily far from equilibrium and systems even small to single particle with external stochastic force, in terms of thermodynamics. Thus, a new branch named stochastic thermodynamics [23,34,35,36,37,38,39,40] is established and soon becomes a research hotspot in statistical physics and some related fields.

With the branch established, recently an inequality named “thermodynamic uncertainty relation” (TUR) was proposed [24,25,26,27] which shows the relation between the fluctuation and the entropy production at nonequlibrium steady state, and this relation is one of the most important and impacting theoretical results in the last years. Generally, the microstates of the system could form lots of cycles. At nonequilibrium steady state, there would be cyclic fluxes on these cycles (see Section 3.3 below for more details on cycle theory). Quantitatively, denoting the stochastic net cyclic flux on a cycle as j and the corresponding entropy production rate on this cycle as $e_p^s$, a relation will hold:

$$\frac{{\sigma }_j^2}{{\langle j\rangle }^2}·e_p^s\ge 2k_B,$$

(5)

where ${\sigma }_j^2$ is the variance of j and $\langle j\rangle$ is the average of j. It’s clear that if the entropy production rate $e_p^s$ is increased, the lower bound of the relative fluctuation ${\sigma }_j^2/{\langle j\rangle }^2$ of the cyclic flux could be suppressed. This relation can also be written in the integral form in a finite time interval $\tau$:

$$\frac{{\sigma }_{j_{\tau }}^2}{{\langle j_{\tau }\rangle }^2}·{\mathsf{\Sigma }}_{\tau }\ge 2k_B,$$

(6)

where $\langle j_{\tau }\rangle =\langle j\rangle \times \tau$ is on average how many times the cycle is completed in the time interval $\tau$ and ${\mathsf{\Sigma }}_{\tau }$ is the entropy produced in $\tau$. These relations are the called thermodynamic uncertainty relations (TURs).

TUR shows how the stochastic dynamic quantities (fluctuations) are related to thermodynamic quantities (entropy production rates). By increasing entropy production rate, the noise of the dynamics may be suppressed, and vice versa. It also reveals that, it’s impossible to reduce the fluctuation and entropy production rate simultaneously to infinitely small. This relation is attracting more and more attention, and is one of the most important theoretical results in this field in the past years.

3. Nonequilibrium Steady State in Chemical Reaction Networks and Its Thermodynamics

As introduced above, self-organized dissipative structure will maintain its ordered state for relatively long time, although far from equilibrium. Therefore, thermodynamic rules and properties in such steady states, i.e., nonequilibrium steady states (NESSs), are quite interesting to physicists especially biophysicists. The thermodynamic laws and methodologies in NESS, especially those in biochemical NESS, are of great importance to study the thermodynamic properties of living systems. Here we introduce the current thermodynamic theories in chemical reaction networks.

3.1. Description of Chemical Reactions with Stochastic Process

In the framework of stochastic process, chemical reactions are considered as a series of Markov processes described by chemical master equations (CMEs) [41,42]. Consider a general chemical reaction:

$$\sum _{i=1}^J{s}_i{X}_i\underset{k_{-}}{\overset{k_{+}}{\rightleftharpoons }}\sum _{i=1}^J{r}_i{X}_i,$$

(7)

where $X_i,i=1,2,\dots ,J$ is the i-th component and $s_i,r_i,i=1,2,\dots ,J$ the corresponding stoichiometric coefficient. Denoting $n_i$ as the molecule number of $X_i$, the microscopic state of the system can be represented by $\overrightarrow{n}=(n_1,n_2,\dots ,n_J)$. For the forward reaction, after the reaction happens, $n_i$ will be changed to $n_i+(r_i-s_i)$ with the reaction frequency proportional to probability that the reactant molecules collide, which is also proportional to

$$\prod _{i=1}^J\frac{n_i!}{(n_i-s_i)!}.$$

Similar for the reverse reaction, after the reverse reaction happens $n_i$ will be changed to $n_i-(r_i-s_i)$ the reaction frequency proportional to

$$\prod _{i=1}^J\frac{n_i!}{(n_i-r_i)!}.$$

Hence, considering other effects such as volume, the corresponding CME describing the evolution of the system’s probability density $P(\overrightarrow{n},t)$ can be written as [41]

$$\begin{array}{c}\frac{\mathrm{d}P(\overrightarrow{n},t)}{\mathrm{d}t}=\left\{k_{+}V\left(\prod _{i=1}^J{\mathbb{E}}^{s_i-r_i}-1\right)\prod _{j=1}^J\left[\frac{n_j!}{(n_j-s_j)!V^{s_j}}\right]\right.\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hfill \\ \hfill \left.+k_{-}V\left(\prod _{i=1}^J{\mathbb{E}}^{r_i-s_i}-1\right)\prod _{j=1}^J\left[\frac{n_j!}{(n_j-r_j)!V^{r_j}}\right]\right\}P,\end{array}$$

(8)

where V is the volume and $\mathbb{E}$ is the “step operator” defined by its effect on arbitrary function $f\left(n\right)$: $\mathbb{E}f\left(n\right)=f(n+1)$.

If the system consists of m reactions ($m>1$) forming a chemical reaction network (CRN),

$$\sum _{i=1}^J{s}_i^{\rho }{X}_i\underset{k_{-}^{\rho }}{\overset{k_{+}^{\rho }}{\rightleftharpoons }}\sum _{i=1}^J{r}_i^{\rho }{X}_i,\rho =1,2,\dots ,m,$$

(9)

the CME that describes this CRN should add up all the terms corresponding to each reaction, i.e.,

$$\begin{array}{c}\frac{\mathrm{d}P(\overrightarrow{n},t)}{\mathrm{d}t}=\sum _{\rho =1}^m\left\{k_{+}^{\rho }V\left(\prod _{i=1}^J{\mathbb{E}}^{s_i^{\rho }-r_i^{\rho }}-1\right)\prod _{j=1}^J\left[\frac{n_j!}{(n_j-s_j^{\rho })!V^{s_j^{\rho }}}\right]\right.\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hfill \\ \hfill \left.+k_{-}^{\rho }V\left(\prod _{i=1}^J{\mathbb{E}}^{r_i^{\rho }-s_i^{\rho }}-1\right)\prod _{j=1}^J\left[\frac{n_j!}{(n_j-r_j^{\rho })!V^{r_j^{\rho }}}\right]\right\}P.\end{array}$$

(10)

This is how to describe a general CRN. Sometimes this can be simplified. If the reaction system only consists of a series of unimolecular reactions (for example, a CRN that describes the decoration or allosteric transition of a single protein complex), a simplified framework can be applied [36]. As is shown in Figure 1, suppose there is a system consisting of three components $X_1,X_2,X_3$ which can be transferred to each other by unimolecular reactions, i.e.,

$$X_1\underset{k_{21}}{\overset{k_{12}}{\rightleftharpoons }}{X}_2\underset{k_{32}}{\overset{k_{23}}{\rightleftharpoons }}{X}_3\underset{k_{13}}{\overset{k_{31}}{\rightleftharpoons }}{X}_1.$$

The dynamics of this system can be regarded as a single molecule jumping between the three states $X_1,X_2,X_3$, with the probability stopping at each state $p_i$ equals to the concentration of each component at steady state. Thus the evolution of $p_i$ is governed by the following master equation:

$$\frac{\mathrm{d}{p}_i}{\mathrm{d}t}=\sum _{j\ne i}(k_{ji}{p}_j-k_{ij}{p}_i),$$

(11)

which has a much simpler form than the general Equation (10).

In summary, generally the dynamics of a CRN can be regarded the system jump between different microscopic states via Markov processes and can be described by a CME. Actually, such description in terms of stochastic process is proposed much earlier than the research on NESS or stochastic thermodynamics and had already brought in many important applications. For example, the precise simulation of chemical reactions—Gillespie algorithm [43].

3.2. Thermodynamic Quantities in Chemical Reaction Networks Out of Equilibrium

How to define the thermodynamic quantities in chemical reaction systems? Although they are well defined in equilibrium statistical mechanics, these definitions are traditionally only proved to hold at equilibrium state. Luckily, some of the definitions can be extend to nonequilibrium systems. The extension is strictly established by Ge and Qian [44], which is partly reported below. Their original work ignored the cases where different microstates can be connected by multiple reactions, so the content below is slightly modified. It should be noted that the extension naturally converges to standard equilibrium definitions in thermodynamics, similarly to what happens in other literature [45].

Consider a general CRN, where any two microscopic states i and j are connected by ${\sigma }_{ij}$ reactions. The dynamics of this CRN is governed by the following CME:

$$\frac{\mathrm{d}{p}_i}{\mathrm{d}t}=\sum _j\sum _{\alpha =1}^{{\sigma }_{ij}}(q_{ji}^{\alpha }{p}_j-q_{ij}^{\alpha }{p}_i),i=1,2,\dots$$

(12)

with $p_i$ the probability at state i and $q_{ji}^{\alpha }$ the reaction rate resulted from $\alpha$-th reaction that switch the system from state i to j. Mathematically, it can be proved that for a quite wide range of CRN, there exists a unique steady state distribution $p_i^s$. Hence the internal energy for microstates can be defined with self-consistency:

$$u_i\equiv -k_{B}Tln{p}_i^s,$$

(13)

and total internal energy is $U={\sum }_i{p}_i{u}_i$. The entropy can be defined by Gibbs entropy:

$$S\equiv -k_B\sum _i{p}_{i}ln{p}_i,$$

(14)

thus the free energy is

$$F\equiv U-TS=-k_{B}T\sum _i{p}_{i}ln\frac{p_i}{p_i^s}.$$

(15)

By taking time derivatives to F and S, it can be determined how free energy and entropy vary over time:

$$\frac{\mathrm{d}F}{\mathrm{d}t}=-\frac{1}{2}{k}_{B}T\sum _{i,j}\sum _{\alpha =1}^{{\sigma }_{ij}}(p_i{q}_{ij}^{\alpha }-p_j{q}_{ji}^{\alpha })ln\left[\frac{p_j\left(t\right)p_i^s}{p_i\left(t\right)p_j^s}\right],$$

(16)

and

$$\frac{\mathrm{d}S}{\mathrm{d}t}=e_p-\frac{h_d\left(t\right)}{T},$$

(17)

where

$$\begin{array}{c}{e}_p=\frac{1}{2}{k}_B\sum _{i,j}\sum _{\alpha =1}^{{\sigma }_{ij}}(p_i{q}_{ij}^{\alpha }-p_j{q}_{ji}^{\alpha })ln\frac{p_i{q}_{ij}^{\alpha }}{p_j{q}_{ji}^{\alpha }},\end{array}$$

(18)

$$\begin{array}{c}{h}_d=\frac{k_{B}T}{2}\sum _{i,j}\sum _{\alpha =1}^{{\sigma }_{ij}}(p_i{q}_{ij}^{\alpha }-p_j{q}_{ji}^{\alpha })ln\frac{q_{ij}^{\alpha }}{q_{ji}^{\alpha }}.\end{array}$$

(19)

$e_p$ is the entropy production rate of the chemical reactions, and is never negative. $h_d$ is the heat dissipation rate describing how much heat is dissipated and expelled to external environment.

At steady state $p_i=p_i^s$, it’s clear that $\mathrm{d}F/\mathrm{d}t=\mathrm{d}S/\mathrm{d}t=0,e_p^s=h_d^s/T$. This indicates that, when $e_p^s=0$, system is at equilibrium and there is no heat exchange with external environment. When $e_p^s>0$, system is driven to a NESS, continuously dissipating energy with a rate $e_p^s$ and expelling these entropy to external environment with a rate $h_d^s/T$ in the form of heat. Therefore, considering energy conservation law, the free energy dissipation rate of the system is

$$\dot{W}=T{e}_p^s=\frac{1}{2}{k}_{B}T\sum _{i,j}\sum _{\alpha =1}^{{\sigma }_{ij}}(p_i^s{q}_{ij}^{\alpha }-p_j^s{q}_{ji}^{\alpha })ln\frac{p_i^s{q}_{ij}^{\alpha }}{p_j^s{q}_{ji}^{\alpha }}.$$

(20)

In some literature, it’s also called “housekeeping heat” $Q_{hk}$ [44], with the name suggesting it’s the necessary energy input to maintain such a NESS. In addition, $Q_{hk}>0$ for NESS, which is in accordance with Schrödinger and Prigogine’s theory.

When the system’s microscopic states are so close that can be described by some continuous variables $\overrightarrow{x}=(x_1,x_2,\dots )$ (for example, in thermodynamic limit where the total molecular number goes to infinity, the system is described by its concentration), the CME can be approximated by a Fokker-Planck equation [41,46]:

$$\frac{\partial P(\overrightarrow{x},t)}{\partial t}=-\sum _{\alpha }\frac{\partial J_{\alpha }}{\partial x_{\alpha }},$$

(21)

with $\alpha$ the reaction index and the sum runs over all the possible chemical reactions. $J_{\alpha }=F_{\alpha }P-D_{\alpha }{\partial }_{x_{\alpha }}P$ is the chemical flux resulted from $\alpha$-th reaction, with $x_{\alpha }$ the generalized coordinate corresponding to the direction of this reaction, $F_{\alpha }$ and $D_{\alpha }$ the corresponding probability drift force and diffusion constant. Note that $x_{\alpha }$ may be different from $x_1,x_2,\dots$, because the direction of the reaction may not be parallel to any of $x_1,x_2,\dots$ For example, a system consisting of n components can be described by the concentration of each component $c_i,i=1,2,\dots ,n$, thus $\overrightarrow{x}=(c_1,c_2,\dots ,c_n)$. Suppose there is a reaction $X_1\rightleftharpoons X_2$, whose forward reaction would equally increase $c_2$ and decrease $c_1$. The generalized coordinate of this reaction is $c_2-c_1$, which is not parallel to any of $c_1,c_2,\dots ,c_n$.

Similar to Equation (20), it can be proved that the free energy dissipation rate at steady state $P_s\left(\overrightarrow{x}\right)$ is [47]

$$\dot{W}=k_{B}T\sum _{\alpha }\int \frac{J_{\alpha }^2}{D_{\alpha }{P}_s}\mathrm{d}\overrightarrow{x}.$$

(22)

Equation (22) can also be obtained by taking Equation (20) in the limit of state distance going to zero, indicating the two cases are essentially the same.

3.3. Cycle Theory and the Break of Detailed Balance

As stated above, generally there would be a steady state in a CRN. However, in different cases these steady states have different physical meanings. If the reaction rates in the CRN satisfies the so-called detailed balance condition, i.e., each forward probability flux from a reaction is nullified by the reverse probability flux from the corresponding reverse reaction, the steady state is the equilibrium state and there is no net probability flux in the system. If the detailed balance condition is not satisfied, net probability flux would arise and entropy would continuously be produced. For the system that doesn’t have a source or a sink, such net flux must be a cyclic flux. There are many work on the such cyclic flux since Hill [48], and it has been widely accepted that the existence of cyclic flux in a system is one of the most important signature for the system to be nonequilibrium.

The system shown in Figure 1 is a simple case to illustrate this concept. By definition, the probability flux of each reaction is $J_{ij}=k_{ij}{p}_i-k_{ji}{p}_j,$ and the thermodynamic force is [48]

$$X_{ij}=k_{B}Tln\frac{k_{ij}}{k_{ji}}+k_{B}Tln\frac{p_i}{p_j}=k_{B}Tln\frac{k_{ij}{p}_i}{k_{ji}{p}_j}.$$

According to Equation (18), the entropy production rate of each chemical reaction is $e_{p,ij}=T^{-1}{J}_{ij}\times X_{ij}\ge 0$. Denoting the steady state by $p_i^s,i=1,2,3,$ at steady state the fluxes should satisfy $J_{12}=J_{23}=J_{31}\equiv J^s$. Hence the total free energy dissipation rate at steady state is

$$\dot{W}=T{e}_p^s=k_{B}T{J}^{s}ln\frac{k_{12}{k}_{23}{k}_{31}}{k_{13}{k}_{32}{k}_{21}}=k_{B}T{J}^{s}ln\gamma \ge 0,$$

(23)

with $\gamma \equiv \left(k_{12}{k}_{23}{k}_{31}\right)$/$\left(k_{13}{k}_{32}{k}_{21}\right)$.

Each factor in Equation (23) has a quite clear physical meaning. $J^s$ is the net cyclic flux in the cycle. When $J^s>0$ there will be a clockwise net flux $X_1\to X_2\to X_3\to X_1$, and when $J^s<0$ there will be a counter-clockwise net flux $X_1\to X_3\to X_2\to X_1$. $\gamma$ is the ratio of the clockwise rate product and the counter-clockwise rate product, and $ln\gamma$ is the thermodynamic force of the cycle. When detailed balance is satisfied and $k_{ij}{p}_i^s=k_{ji}{p}_j^s$ holds for every reaction, $J^s=0,ln\gamma =0$, suggesting both the net cyclic flux and the thermodynamic force are 0, thus the entropy production rate is 0 and the system is at equilibrium state. When detailed balance is broken, both the net cyclic flux and the thermodynamic force will arise, $\gamma \ne 1$, with a positive entropy production rate indicating the system is at nonequilibrium state. Hence, $\gamma \ne 1$ is equivalent to the break of detailed balance.

How is the thermodynamic force introduced and detailed balance broken in a realistic systems? Qian illustrated the answer to this question with a simple example in a review [36]. In short, when detailed balance is broken, this network must be coupled with some reactions that involve molecules whose amount is under controlled by external constraints or regulations. For example, in the system shown in Figure 1, suppose the reaction $X_1\rightleftharpoons X_2$ is actually coupled with an ATP hydrolysis reaction and the real elementary reaction is

$$X_1+\mathrm{ATP}\underset{k_{21}^0}{\overset{k_{12}^0}{\rightleftharpoons }}{X}_2+\mathrm{ADP},$$

where inorganic phosphate Pi can be considered as a constant and is ignored. The reaction rates and the original rates are related by $k_{12}=k_{12}^0\left[ATP\right]$, $k_{21}^0=k_{21}\left[ADP\right]$. By definition, the reaction potential of each reaction is $\mathsf{\Delta }{\mu }_{ji}=-k_{B}Tln(k_{ij}/k_{ji}),$ and it’s easy to show that the thermodynamic force over a cycle $\mathsf{\Delta }{\mu }_{cycle}\equiv \mathsf{\Delta }{\mu }_{12}+\mathsf{\Delta }{\mu }_{23}+\mathsf{\Delta }{\mu }_{31}=k_{B}Tln\gamma$. In absence of external regulation, the system is closed and naturally should satisfy detailed balance, thus $\mathsf{\Delta }{\mu }_{cycle}=0$ and

$${\gamma }^{eq}=\frac{k_{12}^0{k}_{23}{k}_{31}{\left[ATP\right]}^{eq}}{k_{13}{k}_{32}{k}_{21}^0{\left[ADP\right]}^{eq}}=\frac{k_{12}^{eq}{k}_{23}{k}_{31}}{k_{13}{k}_{32}{k}_{21}^{eq}}=exp\left(\frac{\mathsf{\Delta }{\mu }_{cycle}}{k_{B}T}\right)=1.$$

(24)

when the external regulation takes place and fixes the concentration ratio $\left[ATP\right]/\left[ADP\right]$ to a constant value different from ${\left[ATP\right]}^{eq}/{\left[ADP\right]}^{eq}$, the rate ratio $k_{12}/k_{21}$ will be effectively tuned, thus

$$\gamma =\frac{k_{12}{k}_{23}{k}_{31}}{k_{13}{k}_{32}{k}_{21}}=\frac{k_{12}^0{k}_{23}{k}_{31}{\left[ATP\right]}^{eq}}{k_{13}{k}_{32}{k}_{21}^0{\left[ADP\right]}^{eq}}·\frac{\left[ATP\right]{\left[ADP\right]}^{eq}}{\left[ADP\right]{\left[ATP\right]}^{eq}}=\frac{\left[ATP\right]{\left[ADP\right]}^{eq}}{\left[ADP\right]{\left[ATP\right]}^{eq}}\ne 1,$$

(25)

and $\mathsf{\Delta }{\mu }_{cycle}\ne 0.$ Therefore, for a general reaction coupled with ATP hydrolysis, e.g., phosphorylation, by tuning the concentration ratio between ATP and ADP, the thermodynamic force on a cycle $\mathsf{\Delta }{\mu }_{cycle}$ could be tuned and detailed balance can be broken. In addition, it can be checked that

$$ln\gamma =ln\frac{\left[ATP\right]{\left[ADP\right]}^{eq}}{\left[ADP\right]{\left[ATP\right]}^{eq}}=\frac{\mathsf{\Delta }{\mu }_{ATP,0}}{k_{B}T}+ln\frac{\left[ATP\right]}{\left[ADP\right]}=\frac{\mathsf{\Delta }{\mu }_{ATP}}{k_{B}T},$$

(26)

where $\mathsf{\Delta }{\mu }_{ATP,0}$ is the ATP hydrolysis energy under standard condition. Therefore, $\mathsf{\Delta }{\mu }_{cycle}$ is exactly the free energy that one ATP molecule can produce by hydrolysis at the condition. This result explicitly shows how external energy source is pumped in to drive the system, and is interesting and quite important.

The above discussion can be extended to arbitrary CRNs without sink or source. If there is no cycle in the network, detailed balance will never be broken and the steady state is always equilibrium state. If there are more than one cycle, these cycles can be decomposed to a series of linear independent basis cycles $C_i$, whose ratio of the rate products in clockwise reactions and counter-clockwise reactions ${\gamma }_i$ determines whether detailed balance is broken on the basis cycle and the corresponding thermodynamic force. The net flux in the system at steady state can also be decomposed into a series of cyclic fluxes $J_{c_i}^s$, and the total free energy dissipation rate is adding up all the dissipation on each basis cycle, i.e., $k_{B}T{\sum }_i{J}_{c_i}^{s}ln{\gamma }_i$. More details about the cycle decomposition can be found in a review by Schnakenberg and some proceeding literature [49,50,51,52]. From their work, it turns out that such decomposition of thermodynamic observables into cycles might be even more fundamental than transition themselves.

At the end of this section, we note that it’s also possible to study the CRN by deterministic thermodynamics and don’t need to concern stochastic theory. For example, the above cycle theory can be obtained merely by deterministic methods, as is shown in the work done by Rao and Esposito [52]. In the framework of mean-field thermodynamics, by analyzing the stoichiometric matrix of the CRN with method from linear algebra, the conservation law and basis cycles can be calculated. And by calculating the chemical potential difference $\mathsf{\Delta }\mu =k_{B}Tln\gamma$ on each basis cycle, the thermodynamic force can be determined. The dynamics of the system is governed by law of mass action, so the steady state flux can also be determined. Hence, the total dissipation rate can be calculated just like above, which is in accordance with stochastic theory. In addition, such methodologies from mean-field thermodynamics are also capable to determine whether the detailed balance is broken in a CRN with irreversible reactions, for example the work done by Gorban and collaborators [53,54]. Their results pointed out that irreversible reactions cannot be involved in any cycles to prevent divergence of entropy production, and are also in accordance with stochastic theory. Some other notable results are obtained in the last decades following similar methodology [55,56,57,58,59,60,61,62]. This mean-field methodology to study thermodynamics is traditionally Brussels School would use.

The advantage to adopt the stochastic theory is that we can calculate not only the thermodynamic quantities themselves such as entropy production rate but also their uncertainties and errors. This is critical in living systems, since the system size is usually small and fluctuation or noise may play an important role in various biological mechanisms [63,64,65]. And such fluctuation is deeply related to entropy production rate with TUR Equations (5) and (6) mentioned above. It shows that, no matter how to design the parameters in a CRN, it’s impossible to suppress the relative fluctuation and entropy production rate to infinitely small at the same time.

4. Thermodynamics for Information Processing in Living Systems

In realistic systems, the form of maintaining the steady state by dissipation free energy is to fulfill various biological functions by burning molecular fuels. Some functions directly transfer the chemical energy in the fuel to mechanical energy, for example the famous molecular motor [66,67,68,69,70,71] which plays an important role in translocation in cells and motion of bacteria. Some functions involve the synthesis of complex molecules [72,73,74,75,76], e.g., DNA and protien complex.

Except these functions where the energy have clear physical or chemical purposes, there are another large group of functions that only involve in the signal transduction processes. These functions are called information processing functions. The signal is usually transduced by a series of pathways and networks, where the allosteric transition and modification (methylation, phosphorylation, ubiquitination, etc.) to the signal molecule play the important role. For example, the key step in the chemotaxis network in E.coli is the methylation and demethylation of the chemoreceptor dimer [77]. Such modification processes are usually accompanied by hydrolysis of the energy molecules (ATP, GTP, etc.), hence free energy will be dissipated when the information is processed.

A natural question is, from the physics point of view, why free energy dissipation is needed during information processing? The hint comes from the development of information theory in modern statistical physics, especially a branch called “information thermodynamics” [55,78,79,80,81] which studies the thermodynamic cost to manipulate information or vise versa. Hence, an intuitive understanding is that the free energy dissipation is the necessary cost to process the information, or could be used to improve the processing accuracy. Actually, there has been some pioneer works revealing the cost for different information processing functions, suggesting this understanding should generally be right. Here we show some typical results for examples.

4.1. The Accuracy of Specificity and Kinetic Proofreading

The study on the accuracy of specificity and kinetic proofreading is one of the earliest study that applies nonequilibrium thermodynamic concepts to information processing in living systems. This study could trace back to Hopfield’s work in 1974 [82], which has already become a classical research paradigm in this area.

This problem arises from the synthesis of complex molecules, such as DNA replication. As is widely known, successful DNA replication is achieved through base paring. For a certain site in the template DNA strand, the affinity of the “correct” base that should be paired to this site is much larger than the “wrong” base. This property that the specific ligand could bind to specific substrate or receptor is the so-called specificity. However, although small, the probability of a wrong base pair is not exactly zero. By calculating the free energy difference between correct and wrong base pairs, researchers found that, if the affinity difference is the only reason leading to specificity, the error rate during DNA replication is around ${10}^{-4}$∼${10}^{-5}$. This error rate is too large for gene to maintain its relative invariance. But actually, in realistic cases, for eukaryotes the error rate is only around ${10}^{-9}$ [82], much smaller than the estimation by affinity. Hence there must be some other mechanism to reduce the error, which is interesting to researchers for long.

Under this circumstance, Hopfield proposed a so-called kinetic proofreading model [82]. This model supposes that there are one receptor and two ligands in the model, one “correct” and one “wrong”. The affinity difference between the two ligands at equilibrium leads to a basic binding error rate $f_0$. On top of that, if an irreversible “proofreading reaction” is introduced, dissociating the ligand from the receptor and allowing the receptor to choose the ligand again, the error rate could be further reduced. In this case, the system is driven out of equilibrium because the “proofreading reaction” breaks detailed balance. The error rate could be even reduced to $f_0^2$ under some specific conditions, which well explains the small error rate during DNA replication.

In Hopfield’s work, due to the limitation of the development of thermodynamic theory at that time, he didn’t carefully discuss the relation between the error rate reduction and how far the system is driven out of equilibrium. In a more recent work in 2006, Qian discussed the same question with more details [83]. As is shown in Figure 2A, Hong presented Hopfield’s model in a more rigorous way with nonequilibrium thermodynamics and cycle theory. Ligand L can bind with receptor R forming the complex $RL$, and $RL$ could be activated to $R{L}^{\ast }$ by hydrolyzing ATP. The activated $R{L}^{\ast }$ could then dissociate to free ligand L and receptor R. Denoting the ligand concentration by $\left[L\right]$, the rate product ratio is

$$\gamma =\frac{k_1^0\left[L\right]k_2{k}_3}{k_{-1}{k}_{-2}{k}_{-3}^0\left[L\right]}=\frac{k_1{k}_2{k}_3}{k_{-1}{k}_{-2}{k}_{-3}},$$

(27)

with $k_1=k_1^0\left[L\right],k_{-3}=k_{-3}^0\left[L\right]$. As discussed in previous section, it can be proved that $k_{B}Tln\gamma$ is exactly the energy to hydrolyze one ATP molecule. When the concentration ration of ATP and ADP $\left[T\right]/\left[D\right]$ is large enough, $\gamma >1$, ATP can be effectively hydrolyzed, and the system is driven out of equilibrium.

On top of that, suppose there are two ligands $L^{\prime }$ (the correct ligand) and L (the wrong ligand) in the system with the same concentration. Their structures are similar (so have the same $k_1,k_2,k_{-2},k_{-3}$) but $L^{\prime }$ has a much larger affinity than L. This leads to a smaller dissociation rate than L:

$$\frac{k_{-1}^{\prime }}{k_{-1}}=\frac{k_3^{\prime }}{k_3}\equiv \theta <1.$$

The error rate can be defined by the affinity ratio for the two ligands at activated state,

$$f=\frac{\left[R{L}^{\ast }\right]/\left(\left[R\right]\left[L\right]\right)}{\left[R{L^{\prime }}^{\ast }\right]/\left(\left[R\right]\left[L^{\prime }\right]\right)},$$

(28)

It can be proved that, at equilibrium state $f=\theta$. When the system is driven out of equilibrium, $\gamma >1$, f can be smaller than $\theta$. It can be calculated that, for a fixed $\gamma$, the minimal error rate by tuning other parameters is

$$f_{min}\left(\gamma \right)=\theta {\left(\frac{1+\sqrt{\gamma \theta }}{\sqrt{\gamma }+\sqrt{\theta }}\right)}^2.$$

(29)

It can be checked that $f_{min}\left(\gamma \right)$ monotonically decreases with $\gamma$, and when $\gamma \to \infty$, $f_{min}\to {\theta }^2$, which is exactly Hopfield’s result.

In this paper, Hong also proposed that $\gamma$ gives a universal thermodynamic limitation to the error rate $f_{min}$, which would not be violated no matter what structure the reaction network has. The essence of the specificity is the competitive binding between the two ligands and one receptor, which can be represented by a single reaction:

$$L+R{L^{\prime }}^{\ast }\rightleftharpoons L^{\prime }+R{L}^{\ast },$$

(30)

with the equilibrium constant equals to $\theta$. The error rate can be defined as

$$f=\frac{\left[R{L}^{\ast }\right]}{\left[R{L^{\prime }}^{\ast }\right]}.$$

(31)

Given that the concentration of L and $L^{\prime }$ are equal, the free energy difference between $R{L}^{\ast }$ and $R{L^{\prime }}^{\ast }$ at steady state is

$$\mathsf{\Delta }\mu =-k_{B}Tln\theta +k_{B}Tln\frac{\left[R{L}^{\ast }\right]}{\left[R{L^{\prime }}^{\ast }\right]}=k_{B}Tln\frac{f}{\theta }.$$

(32)

when at equilibrium, $\mathsf{\Delta }\mu$ should be zero and $f=\theta$. If this reaction is coupled to a ATP hydrolysis reaction and driven out of equilibrium so that $\mathsf{\Delta }\mu =-\mathsf{\Delta }{\mu }_{ATP}=-k_{B}Tln\gamma ,$$\gamma \ne 1$, we have the following relation

$$f=\frac{\left[R{L}^{\ast }\right]}{\left[R{L^{\prime }}^{\ast }\right]}=\frac{\theta }{\gamma }.$$

(33)

This is the thermodynamic limit given by $\gamma$.

As is shown in Figure 2A, different solid lines illustrate the relation between the minimal error rate $f_{min}$ with different number of proofreading cycles and the energy parameter $\gamma$. The solid lines all lie above the dotted line representing the thermodynamic limit Equation (33), confirming that the accuracy of specificity is limited by the thermodynamic cost.

4.2. The Accuracy of Oscillators and the Energy Cost

Oscillatory behaviors exist in many biological systems and are crucial in controlling the timing of various living activities. However, due to noise, any oscillator cannot keep its high accuracy forever. The time that the oscillator takes to complete one cycle is never exactly equal to the mean period. Affected by noise, it would be slightly longer or shorter. With more cycle completed, the error will accumulate more and more, and finally lost the function of timing. In terms of physics, after a long time, the phase of the oscillator will lost its coherence to initial phase. How to overcome the noise and maintain coherence in a relatively long time, is an important question.

In Ref. [84], researchers thoroughly studied this problem. As is shown in Figure 3A, researchers enumerated all the possible motifs that could generate oscillation in three-node networks, including activator-inhibitor model, repressilator and substrate-depletion model. Researchers provided three realistic examples for each of the three motifs, along with a famous Brusselator model (which actually belongs to substrate-depletion model) and studied the oscillation accuracy and the energy cost for the respective cases.

As is shown in Figure 3B, researchers defined a phase diffusion constant D to describe the coherence of the oscillatory network. The results reveal that all the four models can oscillate only when far from equilibrium, and the dimensionless phase diffusion constant $D/T$ (T is the oscillation period) inversely depends on the energy dissipation in a period $\mathsf{\Delta }W$:

$$V\times \frac{D}{T}\approx C+\frac{W_0}{\mathsf{\Delta }W-W_c},$$

(34)

with $W_c$ a critical energy, V the system volume, C and $W_0$ two constants. This equation shows that the accuracy of the oscillators inversely depends on the additional energy to the minimal requirement for oscillation, and is used to fit the simulation of the 4 models, as is shown in Figure 3C–E. From the fitting, it turns out that Equation (34) is applicable for all the four models.

To understand the origin of this inverse dependence, researchers consider the noisy Stuart-Landau equation that describes a general system near Hopf bifurcation:

$$\frac{\mathrm{d}r}{\mathrm{d}t}=ar-c{r}^3+{\eta }_r,\frac{\mathrm{d}\theta }{\mathrm{d}t}=b+d{r}^2+{\eta }_{\theta },$$

(35)

where ${\eta }_r,{\eta }_{\theta }$ are noise terms satisfying

$$\langle {\eta }_r\left(t\right){\eta }_r\left(t^{\prime }\right)\rangle =2\mathsf{\Delta }\delta (t-t^{\prime }),\langle {\eta }_{\theta }\left(t\right){\eta }_{\theta }\left(t^{\prime }\right)\rangle =\frac{2\mathsf{\Delta }\delta }{r^2}(t-t^{\prime }).$$

In this system, by calculating the energy dissipation and phase diffusion constant with parameters $a,b,c,d,\mathsf{\Delta }$ in Equation (35), Equation (34) can be analytically obtained when $a\ll bc/d$, and $C,W_0,W_c$ can be expressed by:

$$C=0,W_c=\frac{8\pi d}{c}>0,W_0=\left(\frac{d^2}{2c}+2c\right)\left(\frac{2\pi b}{c\mathsf{\Delta }}+\frac{2\pi d^2}{b{c}^2}\right)\mathsf{\Delta },$$

hence the inverse relation Equation (34) holds in a quite large range and is relatively general.

In some biological systems, timing function is accomplished by the collective oscillation phase of a group of oscillating molecules. For such systems, synchronization among different molecular oscillators is of great importance to generate and maintain the correlation of the collective phase, which is critical for macroscopic oscillation accuracy. A recent work shows that the maximal achievable synchrony among a group of identical molecular oscillators is also inversely dependent on the additional energy cost [85].This result, along with Equation (34), shows that oscillation accuracy can be improved by additional energy both in single oscillator level and collective level.

5. Outlook

In this review, we summarize the history and current research status of biological thermodynamics. It’s still a new research branch and hotspot to study the thermodynamics in living systems, which could lead to great fundamental improvement in both physics and biology.

In terms of physics, living systems are realistic systems far from equilibrium, which is a good research object for nonequilibrium statistical physics. By studying the biological functions and their thermodynamic cost, as the examples we show above, the availability of the theories from nonequilibrium statistical physics can be examined by realistic cases. Furthermore, with such applications in realistic systems, physicists could accumulate intuitive feelings that which concepts are of more importance, so that they are able to develop more realistic and useful thermodynamic theories and instruments. It’s also possible to adopt experimental tools from biological systems to test physical theoretical results, just like the test for Jarzynski equality [28,29,30] and fluctuation theorems [31,32,33].

In terms of biology, theoretical tools from thermodynamics and statistical physics could provide another new angle to understand the general, fundamental laws in living systems. New paradigms can also be established with the new theoretical tools to study quantitative behavior in living systems, such as spatial positioning by self-organization [86], understanding fluctuation in cell growth [87] and characterizing enzymes as active matter [88]. In addition, laws from thermodynamics could also help answer the design principles of biological systems, which might be useful to synthetic biology [89,90,91,92] and other biological engineering with in silico design.