Fluctuation Theorem of Information Exchange between Subsystems that Co-Evolve in Time

Abstract

Sagawa and Ueda established a fluctuation theorem of information exchange by revealing the role of correlations in stochastic thermodynamics and unified the non-equilibrium thermodynamics of measurement and feedback control. They considered a process where a non-equilibrium system exchanges information with other degrees of freedom such as an observer or a feedback controller. They proved the fluctuation theorem of information exchange under the assumption that the state of the other degrees of freedom that exchange information with the system does not change over time while the states of the system evolve in time. Here we relax this constraint and prove that the same form of the fluctuation theorem holds even if both subsystems co-evolve during information exchange processes. This result may extend the applicability of the fluctuation theorem of information exchange to a broader class of non-equilibrium processes, such as a dynamic coupling in biological systems, where subsystems that exchange information interact with each other.

1. Introduction

Biological systems possess information processing mechanisms for their survival and heredity [1,2,3]. They, for example, sense external ligand concentrations [4,5], transmit information through signaling networks [6,7,8], and coordinate gene expressions [9] by secreting and sensing signaling molecules [10]. Cells even implement time integration by copying states of environment into molecular states inside the cells to reduce their sensing errors [11,12]. Therefore it is crucial to reveal the role of information in thermodynamics to properly understand complex biological information processes.

Historically, information has entered into the realm of thermodynamics by the name of Maxwell’s demon. The demon observes the speed of molecules in a box that is divided into two portions by a partition in which there is a small hole, and lets the fast particles pass from the lower-half of the box to the upper-half, and only the slow particles pass from the upper-half to the lower-half by opening/closing the hole without expenditure of work (see Figure 1a). This results in raising the temperature of the upper-half of the box and lower that of the lower-half, indicating that the second law of thermodynamics, which implies heat flows spontaneously from hotter to colder places, might hypothetically be violated [13]. This paradox shows that information can affect thermodynamics of a physical system, or information is a physical element [14].

Szilard has devised a much simpler model that carries the essential role of information in Maxwell’s thought experiment. The Szilard engine consists of a single particle in a box which is surrounded by a heat reservoir of constant temperature. A cycle of the engine begins with inserting a partition in the middle of the box. Depending on whether the particle is in the left-half or in the right-half of the box, one controls a lever such that a weight can be lifted during the wall moves quasi-statically in the direction that the particle pushes (see Figure 1b). If the partition reaches an end of the box, the partition is removed and a new cycle begins again with inserting a partition at the center. Since the energy required for lifting the weight comes from the heat reservoir, this engine corresponds to a perpetual-motion machine of the second kind, where the single heat reservoir is spontaneously cooled and the corresponding thermal energy is converted into mechanical work cyclically, which is prohibited by the second-law of thermodynamics [15].

Szilard interprets the coupling between the location of the particle and the direction of the lever as a sort of memory faculty and points out that the coupling is the main cause that enables an amount of work to be extracted from the heat reservoir. He infers, therefore, that establishing the coupling must be accompanied by a production of entropy (dissipation of heat into the environment) which compensates for the lost heat in the reservoir. In [16], Sagawa and Ueda have proved this idea in the form of a fluctuation theorem of information exchange, generalizing the second-law of thermodynamics by taking information into account:

$$〈e^{-\sigma +\Delta I}〉=1,$$

(1)

where $\sigma$ is the entropy production of a system X, and $\Delta I$ is the change of mutual information between the system X and another system Y, such as a demon, during a process ${\lambda }_t$ for $0\le t\le \tau$. Here the bracket indicates the ensemble average over all microscopic trajectories of X and over all states of Y. By Jensen’s inequality [17], Equation (1) implies

$$〈\sigma 〉\ge 〈\Delta I〉.$$

(2)

This tells indeed that establishing a correlation between the two subsystems, $〈\Delta I〉>0$, accompanies an entropy production, $〈\sigma 〉>0$, and expenditure of this correlation, $〈\Delta I〉<0$, serves as a source of entropy decrease, $〈\sigma 〉<0$. In proving this theorem, they have assumed that the state of system Y does not evolve in time. This assumption causes no problem for simple models of measurement and feedback control. However, in biological systems, it is not unusual that both subsystems that exchange information with each other co-evolve in time. For example, transmembrane receptor proteins transmit signals through thermodynamic coupling between extracellular ligands and conformation of intracellular parts of the receptors during a dynamic allosteric transition [18,19]. In this paper, we relax the constraint that Sagawa and Ueda have assumed, and generalize the fluctuation theorem of information exchange to be applicable to more involved situations, where the two subsystems can influence each other so that the states of both systems co-evolve in time.

2. Results

2.1. Theoretical Framework

We consider a finite classical stochastic system composed of subsystems X and Y that are in contact with a heat reservoir of inverse temperature $\beta \equiv 1/\left(k_{B}T\right)$ where $k_B$ is the Boltzmann constant and T is the temperature of the reservoir. We allow both systems X and Y to be driven far from equilibrium by changing external parameter ${\lambda }_t$ during time $0\le t\le \tau$ [20,21,22]. We assume that time evolutions of subsystems X and Y are described by a classical stochastic dynamics from $t=0$ to $t=\tau$ along trajectories $\left\{x_t\right\}$ and $\left\{y_t\right\}$, respectively, where $x_t$ ($y_t$) denotes a specific microstate of X (Y) at time t for $0\le t\le \tau$ on each trajectory. Since both trajectories fluctuate, we repeat the process ${\lambda }_t$ with appropriate initial joint probability distribution $p_0(x,y)$ over all microstates $(x,y)$ of systems X and Y. Then the joint probability distribution $p_t(x,y)$ would evolve for $0\le t\le \tau$. Let $p_t\left(x\right):=\int p_t(x,y)dy$ and $p_t\left(y\right):=\int p_t(x,y)dx$ be the corresponding marginal probability distributions. We assume

$$p_0(x,y)\ne 0forall(x,y)$$

(3)

so that we have $p_t(x,y)\ne 0$, $p_t\left(x\right)\ne 0$, and $p_t\left(y\right)\ne 0$ for all x and y during $0\le t\le \tau$.

Now, the entropy production $\sigma$ during process ${\lambda }_t$ for $0\le t\le \tau$ is given by

$$\sigma :=\Delta s+\beta Q_b,$$

(4)

where $\Delta s$ is the sum of changes in stochastic entropy along $\left\{x_t\right\}$ and $\left\{y_t\right\}$, and $Q_b$ is heat dissipated into the reservoir (entropy production in the reservoir) [23,24]. In detail, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \Delta s& :=\Delta s_x+\Delta s_y,\hfill \\ \hfill \Delta s_x& :=-ln{p}_{\tau }\left(x_{\tau }\right)+ln{p}_0\left(x_0\right),\hfill \\ \hfill \Delta s_y& :=-ln{p}_{\tau }\left(y_{\tau }\right)+ln{p}_0\left(y_0\right).\hfill \end{array}\end{array}$$

(5)

We note that the stochastic entropy $s\left[p_t(\circ )\right]:=-ln{p}_t(\circ )$ of microstate ∘ at time t can be interpreted as uncertainty of occurrence of ∘ at time t: The greater the probability that state ∘ occurs, the smaller the uncertainty of occurrence of state ∘.

Now we consider situations where system X exchanges information with system Y during process ${\lambda }_t$. By this, we mean that trajectory $\left\{x_t\right\}$ of system X evolves depending on the trajectory $\left\{y_t\right\}$ of system Y. Then, information $I_t$ at time t between $x_t$ and $y_t$ is characterized by the reduction of uncertainty of $x_t$ due to given $y_t$ [16]:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill I_t(x_t,y_t)& :=s\left[p_t\left(x_t\right)\right]-s\left[p_t\left(x_t\right|y_t)\right]\hfill \\ & =ln\frac{p_t(x_t,y_t)}{p_t\left(x_t\right)p_t\left(y_t\right)},\hfill \end{array}\end{array}$$

(6)

where $p_t\left(x_t\right|y_t)$ is the conditional probability distribution of $x_t$ given $y_t$. We note that this is called the (time-dependent form of) thermodynamic coupling function [19]. The larger the value of $I_t(x_t,y_t)$ is, the more information is being shared between $x_t$ and $y_t$ for their occurrence. We note that $I_t(x_t,y_t)$ vanishes if $x_t$ and $y_t$ are independent at time t, and the average of $I_t(x_t,y_t)$ with respect to $p_t(x_t,y_t)$ over all microstates is the mutual information between the two subsystems, which is greater than or equal to zero [17].

2.2. Proof of Fluctuation Theorem of Information Exchange

Now we are ready to prove the fluctuation theorem of information exchange in this general setup. We define reverse process ${\lambda }_t^{\prime }:={\lambda }_{\tau -t}$ for $0\le t\le \tau$, where the external parameter is time-reversed [25,26]. Here we set the initial probability distribution $p_0^{\prime }(x,y)$ for the reverse process as the final (time $t=\tau$) probability distribution for the forward process $p_{\tau }(x,y)$ so that we have

$$\begin{array}{c}\hfill \begin{array}{c}\hfill p_0^{\prime }\left(x\right)=\int p_0^{\prime }(x,y)dy=\int p_{\tau }(x,y)dy=p_{\tau }\left(x\right),\\ \hfill p_0^{\prime }\left(y\right)=\int p_0^{\prime }(x,y)dx=\int p_{\tau }(x,y)dx=p_{\tau }\left(y\right).\end{array}\end{array}$$

(7)

Then, by Equation (3), we have $p_t^{\prime }(x,y)\ne 0$, $p_t^{\prime }\left(x\right)\ne 0$, and $p_t^{\prime }\left(y\right)\ne 0$ for all x and y during $0\le t\le \tau$. We also consider the time-reversed conjugate for each $\left\{x_t\right\}$ and $\left\{y_t\right\}$ for $0\le t\le \tau$ as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{x_t^{\prime }\right\}:=\left\{x_{\tau -t}^{*}\right\},\\ \hfill \left\{y_t^{\prime }\right\}:=\left\{y_{\tau -t}^{*}\right\},\end{array}\end{array}$$

(8)

where * denotes momentum reversal. The microscopic reversibility condition connects the time-reversal symmetry of the microscopic dynamics to non-equilibrium thermodynamics, and reads in this framework as follows [23,27,28,29]:

$$\frac{p(\left\{x_t\right\},\left\{y_t\right\}|x_0,y_0)}{p^{\prime }(\left\{x_t^{\prime }\right\},\left\{y_t^{\prime }\right\}|x_0^{\prime },y_0^{\prime })}=e^{\beta Q_b},$$

(9)

where $p(\left\{x_t\right\},\left\{y_t\right\}|x_0,y_0)$ is the conditional joint probability distribution of paths $\left\{x_t\right\}$ and $\left\{y_t\right\}$ conditioned at initial microstates $x_0$ and $y_0$, and $p^{\prime }(\left\{x_t^{\prime }\right\},\left\{y_t^{\prime }\right\}|x_0^{\prime },y_0^{\prime })$ is that for the reverse process. Now we have the following:

$$\begin{array}{ccc}\hfill \frac{p^{\prime }(\left\{x_t^{\prime }\right\},\left\{y_t^{\prime }\right\})}{p(\left\{x_t\right\},\left\{y_t\right\})}& =& \frac{p^{\prime }(\left\{x_t^{\prime }\right\},\left\{y_t^{\prime }\right\}|x_0^{\prime },y_0^{\prime })}{p(\left\{x_t\right\},\left\{y_t\right\}|x_0,y_0)}·\frac{p_0^{\prime }(x_0^{\prime },y_0^{\prime })}{p_0(x_0,y_0)}\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill & =& \frac{p^{\prime }(\left\{x_t^{\prime }\right\},\left\{y_t^{\prime }\right\}|x_0^{\prime },y_0^{\prime })}{p(\left\{x_t\right\},\left\{y_t\right\}|x_0,y_0)}·\frac{p_0^{\prime }(x_0^{\prime },y_0^{\prime })}{p_0^{\prime }\left(x_0^{\prime }\right)p_0^{\prime }\left(y_0^{\prime }\right)}·\frac{p_0\left(x_0\right)p_0\left(y_0\right)}{p_0(x_0,y_0)}·\frac{p_0^{\prime }\left(x_0^{\prime }\right)}{p_0\left(x_0\right)}·\frac{p_0^{\prime }\left(y_0^{\prime }\right)}{p_0\left(y_0\right)}\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill & =& exp\{-\beta Q_b+I_{\tau }(x_{\tau },y_{\tau })-I_0(x_0,y_0)-\Delta s_x-\Delta s_y\}\hfill \end{array}$$

(12)

$$\begin{array}{ccc}\hfill & =& exp\{-\sigma +\Delta I\}.\hfill \end{array}$$

(13)

To obtain Equation (11) from Equation (10), we multiply Equation (10) by $\frac{p_0^{\prime }\left(x_0^{\prime }\right)p_0^{\prime }\left(y_0^{\prime }\right)}{p_0^{\prime }\left(x_0^{\prime }\right)p_0^{\prime }\left(y_0^{\prime }\right)}$ and $\frac{p_0\left(x_0\right)p_0\left(y_0\right)}{p_0\left(x_0\right)p_0\left(y_0\right)}$, which are 1. We obtain Equation (12) by applying Equations (5)–(7) and (9) consecutively to Equation (11). Finally, we set $\Delta I:=I_{\tau }(x_{\tau },y_{\tau })-I_0(x_0,y_0)$, and use Equation (4) to obtain Equation (13) from Equation (12).

We note that Equation (13) generalizes the detailed fluctuation theorem in the presence of information exchange that is proved in [16]. Now we obtain the generalized version of Equation (1) by using Equation (13) as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill 〈e^{-\sigma +\Delta I}〉& =\int e^{-\sigma +\Delta I}p(\left\{x_t\right\},\left\{y_t\right\})d\left\{x_t\right\}d\left\{y_t\right\}\hfill \\ & =\int p^{\prime }(\left\{x_t^{\prime }\right\},\left\{y_t^{\prime }\right\})d\left\{x_t^{\prime }\right\}d\left\{y_t^{\prime }\right\}=1.\hfill \end{array}\end{array}$$

(14)

Here we use the fact that there is a one-to-one correspondence between the forward and the reverse paths due to the time-reversal symmetry of the underlying microscopic dynamics such that $d\left\{x_t\right\}=d\left\{x_t^{\prime }\right\}$ and $d\left\{y_t\right\}=d\left\{y_t^{\prime }\right\}$ [30].

2.3. Corollary

Before discussing a corollary, we remark one thing: we have used similar notation to that used by Sagawa and Ueda in [16], but there is an important difference. Most importantly, their entropy production ${\sigma }_{\mathrm{su}}$ reads as follows:

$${\sigma }_{\mathrm{su}}:=\Delta s_{\mathrm{su}}+\beta Q_b,$$

where $\Delta s_{\mathrm{su}}:=\Delta s_x$. In [16], system X is in contact with the heat reservoir, but system Y is not. Nor does system Y evolve over time. Thus they have considered entropy production in system X and the bath. In this paper, both systems X and Y are in contact with the reservoir, and system Y also evolves in time. Thus both subsystems X and Y as well as the heat bath contribute to the entropy production as expressed in Equations (4) and (5). Keeping in mind this difference, we apply Jensen’s inequality to Equation (14) to obtain

$$〈\sigma 〉\ge 〈\Delta I〉.$$

(15)

It tells us that firstly, establishing correlation between X and Y accompanies entropy production, and secondly, established correlation serves as a source of entropy decrease.

Now as a corollary, we refine the generalized fluctuation theorem in Equation (14) by including energetic terms. To this end, we define local free energy ${\mathcal{F}}_x$ of system X at $x_t$ and ${\mathcal{F}}_y$ of system Y at $y_t$ as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathcal{F}}_x(x_t,t)& :=E_x(x_t,t)-Ts\left[p_t\left(x_t\right)\right]\hfill \\ \hfill {\mathcal{F}}_y(y_t,t)& :=E_y(y_t,t)-Ts\left[p_t\left(y_t\right)\right],\hfill \end{array}\end{array}$$

(16)

where $E_x$ and $E_y$ are internal energy of systems X and Y, respectively, and $s\left[p_t(\circ )\right]:=-ln{p}_t(\circ )$ is stochastic entropy [23,24]. Here T is the temperature of the heat bath and argument t indicates dependency of each terms on external parameter ${\lambda }_t$. During the process ${\lambda }_t$, work done on the systems is expressed by the first law of thermodynamics as follows:

$$W:=\Delta E+Q_b,$$

(17)

where $\Delta E$ is the change in internal energy of the systems. If we assume that systems X and Y are weakly coupled, in that interaction energy between X and Y is negligible compared to internal energy of X and Y, we may have

$$\Delta E:=\Delta E_x+\Delta E_y,$$

(18)

where $\Delta E_x:=E_x(x_{\tau },\tau )-E_x(x_0,0)$ and $\Delta E_y:=E_y(y_{\tau },\tau )-E_y(y_0,0)$ [31]. We rewrite Equation (12) by adding and subtracting the change of internal energy $\Delta E_x$ of X and $\Delta E_y$ of Y as follows:

$$\begin{array}{ccc}\hfill \frac{p^{\prime }(\left\{x_t^{\prime }\right\},\left\{y_t^{\prime }\right\})}{p(\left\{x_t\right\},\left\{y_t\right\})}& =& exp\{-\beta (Q_b+\Delta E_x+\Delta E_y)+\Delta I+\beta \Delta E_x-\Delta s_x+\beta \Delta E_y-\Delta s_y\}\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill & =& exp\{-\beta (W-\Delta {\mathcal{F}}_x-\Delta {\mathcal{F}}_y)+\Delta I\},\hfill \end{array}$$

(20)

where we have applied Equations (16)–(18) consecutively to Equation (19) to obtain Equation (20). Here $\Delta {\mathcal{F}}_x:={\mathcal{F}}_x(x_{\tau },\tau )-{\mathcal{F}}_x(x_0,0)$ and $\Delta {\mathcal{F}}_y:={\mathcal{F}}_y(y_{\tau },\tau )-{\mathcal{F}}_y(y_0,0)$. Now we obtain fluctuation theorem of information exchange with energetic terms as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill 〈e^{-\beta (W-\Delta {\mathcal{F}}_x-\Delta {\mathcal{F}}_y)+\Delta I}〉& =\int e^{-\beta (W-\Delta {\mathcal{F}}_x-\Delta {\mathcal{F}}_y)+\Delta I}p(\left\{x_t\right\},\left\{y_t\right\})d\left\{x_t\right\}d\left\{y_t\right\}\hfill \\ & =\int p^{\prime }(\left\{x_t^{\prime }\right\},\left\{y_t^{\prime }\right\})d\left\{x_t^{\prime }\right\}d\left\{y_t^{\prime }\right\}=1,\hfill \end{array}\end{array}$$

(21)

which generalizes known relations in the literature [31,32,33,34,35,36]. We note that Equation (21) holds under the weak-coupling assumption between systems X and Y during the process ${\lambda }_t$. By Jensen’s inequality, Equation (21) implies

$$〈W〉\ge 〈\Delta {\mathcal{F}}_x+\Delta {\mathcal{F}}_y+\frac{\Delta I}{\beta }〉.$$

(22)

We remark that $〈\Delta {\mathcal{F}}_x〉+〈\Delta {\mathcal{F}}_y〉$ in Equation (22) is the difference in non-equilibrium free energy, which is different from the change in equilibrium free energy that appears in similar relations in the literature [32,33,34,35,36].

3. Examples

3.1. Measurement

Let X be a device (or a demon) which measures the state of other system and Y be a measured system, both of which are in contact with a heat bath of inverse temperature $\beta$ (see Figure 2a). We consider a dynamic measurement process, which is described as follows: X and Y are prepared separately in equilibrium such that X and Y are not correlated initially, i.e., $I_0(x_0,y_0)=0$ for all $x_0$ and $y_0$. At time $t=0$, device X is put in contact with system Y so that the coupling of X and Y occurs due to their (weak) interactions until time $t=\tau$, at which a single measurement process finishes. We note that system Y is allowed to evolve in time during the process. Since each process fluctuates, we repeat the measurement many times to obtain probability distribution $p_t(x,y)$ for $0\le t\le \tau$.

A distinguished feature of the framework in this paper is that mutual information $I_t(x_t,y_t)$ in Equation (6) enables us to obtain the time-varying amount of established information during the dynamic coupling process, unlike other approaches where they either provide the amount of information at a fixed time [31,36,37] or one of the system is fixed during the coupling process [16]. For example, let us assume that the probability distribution $p_t(x_t,y_t)$ at an intermediate time t is as shown in

Table 1. The joint probability distribution of x and y at an intermediate time t: Here we assume for simplicity that both systems X and Y have two states, 0 (left) and 1 (right).

$\mathit{X}\setminus \mathit{Y}$	0 (Left)	1 (Right)
0 (Left)	1/3	1/6
1 (Right)	1/6	1/3

.

Then we have the following:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill I_t(x_t=0,y_t=0)& =ln\frac{1/3}{(1/2)·(1/2)}=ln(4/3),\hfill \\ \hfill I_t(x_t=0,y_t=1)& =ln\frac{1/6}{(1/2)·(1/2)}=ln(2/3),\hfill \\ \hfill I_t(x_t=1,y_t=0)& =ln\frac{1/6}{(1/2)·(1/2)}=ln(2/3),\hfill \\ \hfill I_t(x_t=1,y_t=1)& =ln\frac{1/3}{(1/2)·(1/2)}=ln(4/3),\hfill \end{array}\end{array}$$

(23)

so that $〈\Delta I〉=(1/3)ln(4/3)+(1/6)ln(2/3)+(1/6)ln(2/3)+(1/3)ln(4/3)\approx ln\left(1.06\right)$. Thus by Equation (15) we obtain the lower bound of the average entropy production for the coupling that has been established until time t from the uncorrelated initial state, as follows: $〈\sigma 〉\ge 〈\Delta I〉\approx ln1.06.$ If there is no measurement error at final time $\tau$ such that $p_{\tau }(x_{\tau }=0,y_{\tau }=1)=p_{\tau }(x_{\tau }=1,y_{\tau }=0)=0$ and $p_{\tau }(x_{\tau }=0,y_{\tau }=0)=p_{\tau }(x_{\tau }=1,y_{\tau }=1)=1/2$, then we may have $〈\sigma 〉\ge 〈\Delta I〉=ln2$, which is greater than $ln1.06$.

3.2. Feedback Control

Unlike the case in [16], we need not to exchange subsystems X and Y to consider feedback control after the measurement. Thus we proceed continuously to feedback control immediately after each measurement process at time $\tau$ (see Figure 2b). We assume that correlation $I_{\tau }(x_{\tau },y_{\tau })$ at time $\tau$ is given by the values in Equation (23) and final correlation at later time ${\tau }^{\prime }$ is zero, i.e., $I_{{\tau }^{\prime }}(x_{{\tau }^{\prime }},y_{{\tau }^{\prime }})=0$. By feedback control, we mean that external parameter ${\lambda }_t$ for $\tau \le t\le {\tau }^{\prime }$ is manipulated in a pre-determined manner [16], while systems X and Y co-evolve in time, such that the established correlation is used as a source of work while $I_t(x_t,y_t)$ for $\tau \le t\le {\tau }^{\prime }$ is decreased, not necessarily monotonically. Equation (21) provides an exact relation on the energetics of this process. We rewrite its corollary, Equation (22), with respect to extractable work $W_{\mathrm{ext}}:=-W$ as follows:

$$〈W_{\mathrm{ext}}〉\le -〈\Delta {\mathcal{F}}_x+\Delta {\mathcal{F}}_y+\frac{\Delta I}{\beta }〉.$$

(24)

Then the extractable work on top of the conventional bound, $-〈\Delta {\mathcal{F}}_x+\Delta {\mathcal{F}}_y〉$, is additionally given by $-\Delta I/\beta =ln\left(1.06\right)$, which comes from the consumption of the established correlation.

4. Conclusions

We have proved the fluctuation theorem of information exchange, Equation (14), which holds even during the co-evolution of two systems that exchange information with each other. Equation (14) tells us that establishing correlation between two systems necessarily accompanies entropy production which is contributed by both systems and the heat reservoir, as expressed in Equations (4) and (5). We have also proved, as a corollary of Equation (14), the fluctuation theorem of information exchange with energetic terms, Equation (21), under the assumption of weak coupling between the two subsystems. Equation (21) reveals the exact relationship between non-equilibrium free energy of both sub-systems and mutual information that is established/consumed through their interactions. This more generalized framework than that in [16], enables us to apply thermodynamics of information to biological systems, where molecules generate/consume correlations through their information processing mechanisms [4,5,6]. Since the new framework is applicable to fully non-equilibrium situations, thermodynamic coupling during a dynamic allosteric transition, for example, may be analyzed based on this theoretical framework beyond current equilibrium thermodynamic approach [18,19].