A Cost/Speed/Reliability Tradeoff to Erasing ^†

Abstract

We present a Kullback–Leibler (KL) control treatment of the fundamental problem of erasing a bit. We introduce notions of reliability of information storage via a reliability timescale ${\tau }_r$, and speed of erasing via an erasing timescale ${\tau }_e$. Our problem formulation captures the tradeoff between speed, reliability, and the KL cost required to erase a bit. We show that rapid erasing of a reliable bit costs at least $log2-log\left(1-e^{-\frac{{\tau }_e}{{\tau }_r}}\right)>log2$, which goes to $\frac{1}{2}log\frac{2{\tau }_r}{{\tau }_e}$ when ${\tau }_r>>{\tau }_e$.

1. Introduction

Biological systems are remarkably ordered at multiple scales and dimensions, from the spatial order witnessed in the packing of DNA inside the nucleus, the arrangement of cells to form tissues, and organs, and whole organisms, to the temporal order witnessed in the execution of various cellular processes. Superficially, such order might appear to violate the second law of thermodynamics which requires an increase in disorder with overwhelmingly high probability. In fact, there is no violation since biological systems expend energy to bring about and maintain this increase in order.

We would like to understand this “energy to order” conversion quantitatively. What are the fundamental limits to this conversion? Order can be measured in terms of information by counting the number of bits required to describe that order. From this point of view, understanding how much energy is required to create order becomes an instance of the investigation of the connection between information processing and thermodynamics. The basic information processing operation that increases order is the operation of “erasing” or resetting a bit to state 0. To fix ideas, imagine erasing random chalk marks from a blackboard, to leave it in a neat and ordered state.

Szilard [1] and later Landauer [2] have argued from the second law of thermodynamics that erasing at temperature T requires at least $k_{B}Tlog2$ units of energy, where $k_B$ is Boltzmann’s constant. The Szilard engine is a simple illustration of this result. Imagine a single molecule of ideal gas in a cylindrical vessel. If this molecule is in the left half of the vessel, think of that as encoding the bit “0”, and the bit “1” otherwise. Erasing this Brownian bit corresponds to ensuring that the molecule lies on the left half, for example by compressing the ideal gas to half its volume. For a heuristic analysis, we may use the ideal gas law $PV=k_{B}T$, integrating the expression $dW=-PdV$ for work from limits V to $V/2$ to obtain $W=k_{B}Tlog2$. More rigorous and general versions of this calculation are known, which also clarify why this is a lower bound [3,4,5].

In practice, one finds that both man-made and biological instrumentation often require energy substantially more than $k_{B}Tlog2$ to perform erasing [6,7]. John von Neumann remarked on this large gap in his 1949 lectures at the University of Illinois [8]. Bennett [9] has remarked that DNA polymerases come close to the $k_{B}Tlog2$ bound. To copy a single base, a DNA polymerase hydrolyzes a triphosphate molecule to a monophosphate, which provides close to $18k_{B}T$ at temperature $T=300$ K. Note that this is still almost two orders of magnitude away from $k_{B}Tlog2$. Furthermore, it is not clear whether the comparison is valid at all since copying and erasing are different operations.

How does one explain this large gap? Note that the result of $k_{B}Tlog2$ holds only in the isothermal limit, which takes infinite time. In practice, we want erasing to be performed rapidly, say in time ${\tau }_e$, which requires extra entropy production. For intuition, suppose one wants to compress a gas in finite time ${\tau }_e$. The gas heats up, and pushes back, increasing the work required.

Several groups [10,11,12] have recognized that rapid erasing requires entropy production which pushes up the cost of erasing beyond $k_{B}Tlog2$, and have obtained bounds for this problem. A grossly oversimplified, yet qualitatively accurate, sketch of these various results is obtained by considering the energy cost of compressing the Szilard engine rapidly. Specializing a result from finite-time thermodynamics [13] to the case of the Szilard engine, one obtains an energy cost $\left(1+\frac{k_{B}log2}{\sigma {\tau }_e-k_{B}log2}\right)k_{B}Tlog2$ where σ is the coefficient of heat conductivity of the vessel.

The bounds obtained by such considerations depend on technological parameters like the heat conductivity σ, and not just on fundamental constants of physics and the requirement specifications of the problem. If one varies over the technological parameters as well, e.g., allowing $\sigma \to \infty$, the energy cost tends to $k_{B}Tlog2$. Does there exist a more fundamental analysis for the cost of erasing that is independent of technological parameters, and improves on $k_{B}Tlog2$? This is the open question we address in this paper.

Our contribution: We follow up on von Neumann’s suggestion [8] that the gap was “due to something like a desire for reliability of operation”. Swanson [14] and Alicki [15] have also looked into issues of reliability. We introduce the notion of “reliability timescale” ${\tau }_r$, and explicitly consider the three-way trade-off between speed, reliability, and cost.

The other novelty of our approach is in bringing the tools of Kullback–Leibler (KL) control [16,17] to bear on the problem of erasing a bit. The intuitive idea is that the control can reshape the dynamics as it pleases but pays for the deviation from the uncontrolled dynamics. The cost of reshaping the dynamics is a relative entropy or KL divergence between the controlled and the uncontrolled dynamics, expressed as measures on path space.

We find the optimal control for rapid erasing of a reliable bit, and argue that it requires cost of at least $log2-log\left(1-e^{-\frac{{\tau }_e}{{\tau }_r}}\right)>log2$, which goes to $\frac{1}{2}log\frac{2{\tau }_r}{{\tau }_e}$ when ${\tau }_r>>{\tau }_e$. Importantly, our answer does not depend on any technological parameters, but only on the requirement specifications ${\tau }_r$ and ${\tau }_e$ of the problem.

2. The Erasing Problem

As a model of a bit, consider a two-state continuous-time Markov chain with states 0 and 1 and the passive or uncontrolled dynamics given by transition rates $k_{01}$ from state 0 to state 1 and $k_{10}$ from state 1 to state 0.

The transition rates $k_{01}$ and $k_{10}$ model spontaneous transitions between the states when no one is looking at the bit or trying to erase it. The time independence of these rates represents the physical fact that the system is not being driven.

Such finite Markov chain models often arise in physics by “coarse-graining”. For example, for the case of the Szilard engine, the transition rate $k_{10}$ models the rate at which the molecule enters the left side, conditioned on it currently being on the right side.

Apart from their importance in approximating the behavior of real physical systems, finite Markov chains are also important to thermodynamics from a logical point of view. They may be viewed as finite models of a mathematical theory of thermodynamics. The terms “theory” and “model” are to be understood in their technical sense as used in mathematical logic. We develop this remark no further here since doing so would take us far afield.

Suppose the distribution at time t is $(p_0\left(t\right),p_1\left(t\right))$ with $p_1\left(t\right)=1-p_0\left(t\right)$. Then, the time evolution of the bit is described by the ordinary differential equation (ODE):

$$\begin{array}{c}\hfill {\dot{p}}_0\left(t\right)=-k_{01}{p}_0\left(t\right)+k_{10}(1-p_0\left(t\right)).\end{array}$$

(1)

Setting ${\pi }_0=k_{10}/(k_{01}+k_{10})$ and the reliability timescale ${\tau }_r:=1/(k_{01}+k_{10})$, this admits the solution

$$\begin{array}{c}\hfill p_0\left(t\right)={\pi }_0+e^{-t/{\tau }_r}(p_0\left(0\right)-{\pi }_0).\end{array}$$

(2)

Here, ${\tau }_r$ represents the time scale on which memory is preserved. The smaller the rates $k_{01}$ and $k_{10}$, the larger is the value of ${\tau }_r$, and the slower the decay to equilibrium, so that the system remembers information for longer.

Fix a required erasing time ${\tau }_e$. Fix $p\left(0\right)={\pi }_0$. We want to control the dynamics with transition rates $u_{01}\left(t\right)$ and $u_{10}\left(t\right)$ to achieve $p\left({\tau }_e\right)=(1,0)$, where

$$\begin{array}{cc}\hfill {\dot{p}}_0\left(t\right)& =-u_{01}\left(t\right)p_0\left(t\right)+u_{10}\left(t\right)(1-p_0\left(t\right)).\hfill \end{array}$$

(3)

We want to find the cost of the optimal protocol $u_{01}^{*}\left(t\right)$ and $u_{10}^{*}\left(t\right)$ to achieve this objective, according to a cost function which we introduce next. In particular, when $k_{01}=k_{10}$, the equilibrium distribution $\pi =({\pi }_0,1-{\pi }_0)$ takes the value $(1/2,1/2)$, and we can interpret this task as erasing a bit of reliability ${\tau }_r=1/(k_{01}+k_{10})$ in time ${\tau }_e$.

Kullback–Leibler Cost

Define the path space $\mathcal{P}:={\{0,1\}}^{[0,{\tau }_e]}$ of the two-state Markov chain. This is the set of all paths in the time interval $[0,{\tau }_e]$ that jump between states 0 and 1 of the Markov chain. Each path can also be succinctly described by its initial state, and the times at which jumps occur. We can also effectively think of the path space as the limit as $h\to 0$ of the space ${\mathcal{P}}_h:={\{0,1\}}^{\{0,h,2h,\dots ,Nh={\tau }_e\}}$ corresponding to the discrete-time Markov chain that can only jump at clock ticks of h units (Figure 1).

Figure 1. The discrete-time path space ${\mathcal{P}}_h$. A specific path is labeled in red.

Once the rates $u_{01}\left(t\right),u_{10}\left(t\right)$ and the initial distribution $p\left(0\right)=p$ for the Markov chain are fixed, there is a unique probability measure ${\mu }_{u,p}$ on path space which intuitively assigns to every path the probability of occurrence of that path according to the Markov chain evolution (Equation (3)) with initial conditions p.

For pedagogic reasons, we first describe the discrete-time measure ${\mu }_{u,p}^h$ for a single path $i=(i_0,i_1,\dots ,i_N)\in {\mathcal{P}}_h$. First, we describe the transition probabilities of the discrete-time Markov chain. For $a,b\in \{0,1\}$ with $a\ne b$, for all times t, define $u_{aa}^h\left(t\right):=1-h{u}_{ab}\left(t\right)$ and $u_{ab}^h\left(t\right):=h{u}_{ab}\left(t\right)$ as the probability of jumping to a and to b, respectively, in the time step t, conditioned on being in state a. Then, the probability of the path i under control u is given by:

$${\mu }_{u,p}\left(i\right):=p_{i_0}\prod _{j=0}^{N-1}{u}_{i_j,i_{j+1}}^h\left(jh\right).$$

We describe the continous-time case now. We could obtain the measure ${\mu }_{u,p}$ from ${\mu }_{u,p}^h$ by sending $h\to 0$, but it can also be described more directly. Fix $i_0\in \{0,1\}$, and consider the set of paths $\mathcal{S}={\mathcal{S}}_{i_0,t_1,t_2,\dots ,t_n}$ starting at $i_0$ with jumps occurring around times $t_1<t_2<\cdots <t_n$ within infinitesimal intervals $d{t}_1,d{t}_2,\dots ,d{t}_n$ and leading to the trajectory $(i_0,i_1,\dots ,i_n)\in {\{0,1\}}^{n+1}$. Setting $t_0=0$:

$${\mu }_{u,p}\left(\mathcal{S}\right):=p_{i_0}\prod _{j=0}^{n-1}{e}^{-{\int }_{t_j}^{t_{j+1}}{u}_{i_j{i}_{j+1}}\left(s\right)ds}{u}_{i_j{i}_{j+1}}\left(t_{j+1}\right)d{t}_{j+1},$$

where $p_{i_0}$ is the probability of starting at $i_0$, $e^{-{\int }_0^{t_1}{u}_{i_0{i}_1}\left(s\right)ds}$ is the probability of not jumping in the time interval $(0,t_1)$, $u_{i_0{i}_1}\left(t_1\right)d{t}_1$ is the probability of jumping from $i_0$ to $i_1$ in the interval $(t_1,t_1+d{t}_1)$ and so on. This is the well-known Feynam–Kac formula for this Markov chain.

Specializing to $u_{01}\left(t\right)=k_{01}$ and $u_{10}\left(t\right)=k_{10}$, we obtain the probability measure ${\mu }_{k,p}$ induced on $\mathcal{P}$ by the passive dynamics (Equation (1)) with initial conditions p.

We declare the Kullback–Leibler (KL) cost $D({\mu }_{u,p}\parallel {\mu }_{k,p})$ as the cost for implementing the control u. More generally, for a physical system with path space $\mathcal{P}$, passive dynamics corresponding to a measure ν on $\mathcal{P}$, and a controlled dynamics with a control corresponding to a measure μ on $\mathcal{P}$, we declare $D(\mu \parallel \nu )$ as the cost for implementing the control. This cost function has been widely used in control theory [16,17,18,19,20,21,22,23,24,25,26,27]. In Section 4, we will explore some other interpretations of this cost function.

3. Solution to the Erasing Problem

Out of all controls $u\left(t\right)$, we want to find a control $u^{*}\left(t\right)$ that starts from

$$p\left(0\right)=\pi =\left(\frac{k_{10}}{(k_{01}+k_{10}},\frac{k_{01}}{k_{01}+k_{10}}\right)$$

and achieves $p\left({\tau }_e\right)=(1,0)$ while minimizing the relative entropy $D({\mu }_{u^{*},\pi }\parallel {\mu }_{k,\pi })$,

$$\begin{array}{cc}& \text{To}\text{find}:u^{*}=arg\underset{u}{inf}D({\mu }_{u,\pi }\parallel {\mu }_{k,\pi }),\hfill \\ & \text{Subject}\text{to}:{\mu }_{u,\pi }\left({\tau }_e\right)=(1,0).\hfill \end{array}$$

This question can be described within the framework of a well-studied problem in optimal control theory that has a closed-form solution [16,17,28]. Following Todorov [16], we introduce the optimal cost-to-go function $v\left(t\right)=(v_0\left(t\right),v_1\left(t\right))$. We intend $v_i\left(t\right)$ to denote the expected cumulative cost for starting at state i at time $t<{\tau }_e$, and reaching a distribution close to $(1,0)$ at time ${\tau }_e$.

To discourage the system from being in state 1 at time ${\tau }_e$, define $v_1\left({\tau }_e\right)=+\infty$ and $v_0\left({\tau }_e\right)=0$. Suppose the control performs actions $u_{01}\left(t\right)$ and $u_{10}\left(t\right)$ at time t. Fix a small time $h>0$. Define the transition probability $u_{ij}^h\left(t\right)$ as the probability that a trajectory starting in state i at time t will be found in state j at time $t+h$. When $i\ne j$, $u_{ij}^h\left(t\right)\approx h{u}_{ij}\left(t\right)$, whereas $u_{ii}^h\left(t\right)\approx 1-u_{ij}^h\left(t\right)$ ignoring terms of size $O\left(h^2\right)$. We define $k_{ij}^h$ similarly.

Let “log” denote the natural logarithm. To derive the law satisfied by the optimal cost-to-go $v\left(t\right)$, we approximate $v\left(t\right)$ by the backward recursion relations:

$$\begin{array}{cc}\hfill v_0\left(t\right)& =\underset{u_{01}\left(t\right)}{min}\mathbb{E}\left[v_i(t+h)+log\frac{u_{0i}^h\left(t\right)}{k_{0i}^h}\right],\hfill \\ \hfill v_1\left(t\right)& =\underset{u_{10}\left(t\right)}{min}\mathbb{E}\left[v_i(t+h)+log\frac{u_{1i}^h\left(t\right)}{k_{1i}^h}\right],\hfill \end{array}$$

(4)

where the first expectation is over $i{\sim }_{\text{law}}(u_{00}^h\left(t\right),u_{01}^h\left(t\right))$, and the second is over $i{\sim }_{\text{law}}(u_{10}^h\left(t\right),u_{11}^h\left(t\right))$, and the approximation ignores terms of size $O\left(h^2\right)$. As $h\to 0$, the second terms $\mathbb{E}log\frac{u_{ji}^h\left(t\right)}{k_{ji}^h}$ approach the relative entropy cost in path space over the time interval $(t,t+h)$.

Equation (4) says that the cost-to-go from state 0 at time t equals the cost of the control $u\left(t\right)$ plus the expected cost-to-go in the new state i reached at time $t+h$. The cost of the control is measured by relative entropy of the control dynamics relative to the passive dynamics, over the time interval $(t,t+h)$.

Define the desirability $z_0\left(t\right)=e^{-v_0\left(t\right)}$ and $z_1\left(t\right)=e^{-v_1\left(t\right)}$. Define

$$\begin{array}{cc}\hfill G_0\left[z\right]\left(t\right)& =k_{00}^h{z}_0\left(t\right)+k_{01}^h{z}_1\left(t\right),\hfill \\ \hfill G_1\left[z\right]\left(t\right)& =k_{10}^h{z}_0\left(t\right)+k_{11}^h{z}_1\left(t\right).\hfill \end{array}$$

We can rewrite Equation (4) as:

$$\begin{array}{cc}\hfill log{z}_0\left(t\right)& =log{G}_0\left[z\right](t+h)-\underset{u_{01}\left(t\right)}{min}\mathbb{E}\left[log\frac{u_{0i}^h\left(t\right)G_0\left[z\right](t+h)}{k_{0i}^h{z}_i(t+h)}\right],\hfill \\ \hfill log{z}_1\left(t\right)& =log{G}_1\left[z\right](t+h)-\underset{u_{10}\left(t\right)}{min}\mathbb{E}\left[log\frac{u_{1i}^h\left(t\right)G_1\left[z\right](t+h)}{k_{1i}^h{z}_i(t+h)}\right].\hfill \end{array}$$

(5)

Since the last term is the relative entropy of $(u_{j0}^h\left(t\right),u_{j1}^h\left(t\right))$ relative to the probability distribution $(k_{j0}^h{z}_0(t+h)/G_j\left[z\right](t+h),k_{j1}^h{z}_1(t+h)/G_j\left[z\right](t+h))$, its minimum value is 0, and is achieved by the protocol $u^{*}$ given by:

$$\begin{array}{c}\hfill \frac{u_{ji}^{*}\left(t\right)}{k_{ji}}=\underset{h\to 0}{lim}\frac{e^{-v_i(t+h)}}{G_j\left[z\right](t+h)}=\frac{e^{-v_i\left(t\right)}}{e^{-v_j\left(t\right)}},\end{array}$$

(6)

when $i\ne j$.

It remains to solve for $z\left(t\right)$ and the optimal cost. From Equation (5), at the optimal control $u^{*}$, the desirability $z\left(t\right)$ must satisfy the equation $-logz\left(t\right)=-logG\left[z\right](t+h)+0$, so that:

$$\begin{array}{c}\hfill \left(\begin{array}{c}{z}_0\left(t\right)\\ z_1\left(t\right)\end{array}\right)=\left(\begin{array}{cc}1-k_{01}h& k_{01}h\\ k_{10}h& 1-k_{10}h\end{array}\right)\left(\begin{array}{c}{z}_0(t+h)\\ z_1(t+h)\end{array}\right),\end{array}$$

which simplifies to $\frac{dz}{dt}=-Kz$ in the limit $h\to 0$, where $K=\left(\begin{array}{cc}-k_{01}& k_{10}\\ k_{10}& -k_{10}\end{array}\right)$ is the infinitesimal generator of the Markov chain. This equation has the formal solution $z({\tau }_e-t)=e^{Kt}z\left({\tau }_e\right)$ where $z\left({\tau }_e\right)=\left(\begin{array}{c}1\\ 0\end{array}\right)$. In the symmetric case, $k_{01}=k_{10}$,

$$z\left(t\right)=\left(\begin{array}{c}1/2\\ 1/2\end{array}\right)+e^{-\frac{{\tau }_e-t}{{\tau }_r}}\left(\begin{array}{c}1/2\\ -1/2\end{array}\right),$$

where ${\tau }_r=1/(k_{01}+k_{10})$. Substituting $t=0$ and taking logarithms, we find the cost-to-go function at time 0:

$$v\left(0\right)=\left(\begin{array}{c}log2\\ log2\end{array}\right)-\left(\begin{array}{c}log\left(1+e^{-{\tau }_e/{\tau }_r}\right)\\ log\left(1-e^{-{\tau }_e/{\tau }_r}\right)\end{array}\right).$$

When $\nu \left(0\right)=(1/2,1/2)$ with $k_{01}=k_{10}$, the cost $C_{\text{erase}}({\tau }_r,{\tau }_e,T)$ required for erasing a bit of reliability ${\tau }_r=1/(k_{01}+k_{10})$ in time ${\tau }_e$ is at least:

$$\begin{array}{c}\hfill log2-\frac{1}{2}log\left(1-e^{-2{\tau }_e/{\tau }_r}\right).\end{array}$$

(7)

Note that $C_{\text{erase}}\ge log2$ with equality when ${\tau }_e/{\tau }_r\to \infty$, since $1-e^{-2{\tau }_e/{\tau }_r}\le 1$. From Equation (7), $C_{\text{erase}}\ge \frac{1}{2}log\frac{2{\tau }_r}{{\tau }_e}$ when ${\tau }_r>>{\tau }_e$.

4. Interpreting the KL Cost

One motivation for our cost function comes from the field of KL control theory. We now compare other possible meanings to this cost function.

4.1. Path Space Szilard–Landauer Correspondence

The correspondence between information and thermodynamics was revealed in the work of Szilard, and clarified by Landauer. More rigorous and general treatments of this correspondence have been worked out recently [3,4,5]. We first recall this result, and then show how our cost function is a formal extension of this result.

Consider a physical system with finite state space S and energy $E:S\to \mathbb{R}$. (More general state spaces S can be handled by replacing the sum by an appropriate integral. For our present purposes, it suffices to assume S is finite.) Define the Gibbs distribution π at temperature T by

$$\pi \left(i\right)=\frac{e^{-E_i/k_{B}T}}{{\sum }_{j\in S}{e}^{-E_j/k_{B}T}},$$

for all $i\in S$. Define the free energy:

$$F\left(p\right):=\sum _{i\in S}{p}_i{E}_i-k_{B}T\sum _{i\in S}{p}_{i}log\frac{1}{p_i},$$

where p is a probability distribution.

Define the relative entropy $D(p\parallel q)={\sum }_{i\in S}{p}_{i}log\frac{p_i}{q_i}$ with Euler’s constant for the base of the logarithm. Following Jaynes [29], assume that equilibrium π corresponds to a maximally uninformative state of the system, so that we have zero information about the system when it is at equilibrium. Recall that a nat is the unit of information when logarithms are taken to the base of Euler’s constant. 1 bit $=log2$ nats. Then, the relative entropy $D(p\parallel \pi )$ has an axiomatic identification with the amount of information in nats that we know about the system when it is in a nonequilibrium state p [4].

The following identity is easily verified:

$$\begin{array}{c}\hfill F\left(p\right)-F\left(\pi \right)=k_{B}TD(p\parallel \pi ).\end{array}$$

(8)

The conceptual significance of this simple identity is that it supplies a dictionary between thermodynamics and information theory [4]. In particular, erasing a bit corresponds to increasing relative entropy, which, in turn, corresponds—via the identity—to increasing available free energy $F\left(p\right)-F\left(\pi \right)$ by $k_{B}Tlog2$, recovering the classical result of Szilard as an alternative statement of the second law of thermodynamics. In the other direction, charging a battery corresponds to increasing available free energy which in turn corresponds—via Identity Equation (8)—to erasing of information. This relates the energy efficiency of charging a battery to the energy required to erase a bit.

Now, consider our cost function $D(\mu \parallel \nu )$. The relative entropy $D(\mu \parallel \nu )$ counts the number of nats erased by the control in path space, relative to the passive dynamics. Since the Szilard–Landauer principle asserts that erasing one bit requires at least $k_{B}Tlog2$ units of energy, our cost function may be viewed as a Path Space Szilard–Landauer Principle, formally extending Identity Equation (8) to path space.

4.2. Thermodynamic Interpretation

We wish to compare the cost $D(\mu \parallel \nu )$ with the usual thermodynamic expected work $\Delta W$. We will quickly outline how thermodynamic quantities can be defined for a two-state Markov chain.

4.2.1. Thermodynamics on a Two-State Markov Chain

The ideas we present here are well-known in the nonequilibrium thermodynamics community, for example see Propp’s thesis [30]. The construction can be carried out more generally, but the generalization is not necessary for our present purposes.

• Consider again the two-state continuous-time Markov chain with passive dynamics given by transition rates $k_{01}$ and $k_{10}$.

  
  Let $E_0$ and $E_1$ denote the internal energy of states “0” and “1”, respectively. Then, the equilibrium distribution is given by ${\pi }_0\propto e^{-E_0/k_{B}T}$ and ${\pi }_1\propto e^{-E_1/k_{B}T}$. We also have $k_{01}{\pi }_0=k_{10}{\pi }_1$ from detailed balance. Together this yields

  $$\begin{array}{c}\hfill E_0-E_1=k_{B}Tlog\frac{k_{01}}{k_{10}}.\end{array}$$

  (9)
• Now consider the same two-state system with a control applied to it by means of a field of potential $\varphi \left(t\right)=({\varphi }_0\left(t\right),{\varphi }_1\left(t\right))$ so that the potential energy in state i becomes $E_i+{\varphi }_i\left(t\right)$. The transition rates due to the control become $u_{01}\left(t\right)$ and $u_{10}\left(t\right)$. By a reasoning similar to how we derived Equation (9), we get

  $$E_0+{\varphi }_0-E_1-{\varphi }_1=k_{B}Tlog\frac{u_{01}}{u_{10}}.$$
  Combining with Equation (9), this yields

  $$\begin{array}{c}\hfill {\varphi }_0-{\varphi }_1=k_{B}Tlog\frac{u_{01}{k}_{10}}{u_{10}{k}_{01}}.\end{array}$$

  (10)
• Given a distribution $p=(p_0,p_1)$ on the states, we can define the following thermodynamic quantities:

  • Expected internal energy $E\left(p\right)=p_0{E}_0+p_1{E}_1$.
  • Entropy $S\left(p\right)=-p_{0}log{p}_0-p_{1}log{p}_1$.
  • Nonequilibrium free energy $F\left(p\right)=E\left(p\right)-k_{B}TS\left(p\right)$.
• Given a transition from state i to state j in the presence of the control field, we can define the following thermodynamic quantities:

  • Heat dissipated $Q_{ij}\left(t\right)=E_i+{\varphi }_i\left(t\right)-E_j-{\varphi }_j\left(t\right)$.
  • Work done by the control $W_{ij}\left(t\right)={\varphi }_i\left(t\right)-{\varphi }_j\left(t\right)$. This expression for work can be traced back to Sekimoto [31], and is commonly employed in the field of Stochastic Thermodynamics to describe the work done by switching on a control field [32].
  • Entropy increase of the system $S_{ij}\left(t\right)=log\frac{p_i\left(t\right)}{p_j\left(t\right)}$.

  The first law of thermodynamics manifests as $W_{ij}=Q_{ij}+E_i-E_j$.
• Suppose the system is described at time t by a distribution $p\left(t\right)=(p_0\left(t\right),p_1\left(t\right))$. Define the Current $J_{ij}\left(t\right)=p_i\left(t\right)u_{ij}\left(t\right)-p_j\left(t\right)u_{ji}\left(t\right)$ so that ${\dot{p}}_0\left(t\right)=-J_{01}\left(t\right)$.
• We can further compute

  $$\begin{array}{cc}\hfill \frac{dE}{dt}& =J_{01}\left(t\right)(E_1\left(t\right)-E_0\left(t\right))=k_{B}T{J}_{01}\left(t\right)log\frac{k_{10}}{k_{01}},\hfill \\ \hfill \frac{dW}{dt}& =J_{01}\left(t\right)W_{01}\left(t\right)=J_{01}\left(t\right)({\varphi }_0\left(t\right)-{\varphi }_1\left(t\right))=k_{B}T{J}_{01}\left(t\right)log\frac{u_{01}\left(t\right)k_{10}}{u_{10}\left(t\right)k_{01}},\hfill \\ \hfill \frac{dQ}{dt}& =J_{01}\left(t\right)Q_{01}\left(t\right)=J_{01}\left(t\right)(E_0+{\varphi }_0\left(t\right)-E_1-{\varphi }_1\left(t\right))=k_{B}T{J}_{01}\left(t\right)log\frac{u_{01}\left(t\right)}{u_{10}\left(t\right)},\hfill \\ \hfill \frac{dS}{dt}& =J_{01}\left(t\right)S_{01}\left(t\right)=J_{01}\left(t\right)log\frac{p_0\left(t\right)}{p_1\left(t\right)},\hfill \\ \hfill \frac{dF}{dt}& =k_{B}T{J}_{01}\left(t\right)log\frac{k_{10}{p}_1\left(t\right)}{k_{01}{p}_0\left(t\right)}.\hfill \end{array}$$
• Define Total Entropy Production $S_{\text{tot}}\left(t\right)$ to be the total entropy produced from time 0 to time t. In other words, $S_{\text{tot}}\left(0\right)=0$ and

  $$\frac{d{S}_{\text{tot}}\left(t\right)}{dt}=\frac{1}{k_{B}T}\frac{dQ}{dt}+\frac{dS}{dt}.$$
  After simplification,

  $$\begin{array}{c}\hfill \frac{d{S}_{\text{tot}}\left(t\right)}{dt}=(p_0\left(t\right)u_{01}\left(t\right)-p_1\left(t\right)u_{10}\left(t\right))log\frac{p_0\left(t\right)u_{01}\left(t\right)}{p_1\left(t\right)u_{10}\left(t\right)}\ge 0,\end{array}$$

  (11)

  which is a statement of the second law of thermodynamics.
• The following identity is immediate

  $$\begin{array}{c}\hfill \frac{dW}{dt}=\frac{dF}{dt}+\frac{d{S}_{\text{tot}}}{dt}\end{array}$$

  and is another form of the first law.

4.2.2. Thermodynamic Cost for Rapid Erasing of a Reliable Bit

How much does it cost for rapid erasing of a reliable bit, with the cost function equal to $\Delta W$? We claim that it costs $k_{B}log2$. In particular, neither the reliability timescale ${\tau }_r$ nor the erasing timescale ${\tau }_e$ appear in this answer.

Suppose we can erase a $({\tau }_r,{\tau }_e)$ bit for work W. First, note that $\frac{dW}{dt}$ is a function of $k_{01}/k_{10}$, $u_{01}\left(t\right)/u_{10}\left(t\right)$ and $J_{01}\left(t\right)$ as in Equation (6). In particular, simultaneously sending the rates $k_{01}$ and $k_{10}$ as low as possible while keeping their ratio the same has no effect on the work. Thus, if we can erase a $({\tau }_r,{\tau }_e)$ bit for work W, then we can erase a $(A{\tau }_r,{\tau }_e)$ bit for work W, for an arbitrarily large constant A. In particular, it is enough for us to demonstrate a protocol when ${\tau }_r=1$.

Now, note that $\frac{dW}{dt}$ depends only on the ratio $u_{01}\left(t\right)/u_{10}\left(t\right)$ and not on the actual values of the rates. We can also erase a $({\tau }_r,{\tau }_e/2)$ bit for work W by taking the $({\tau }_r,{\tau }_e)$-protocol $\left(u_{ij}\left(t\right)\right)$ and defining new rates $v_{ij}\left(t\right)=2u_{ij}\left(2t\right)$. Since $v_{01}\left(t\right)/v_{10}\left(t\right)=u_{01}\left(2t\right)/u_{10}\left(2t\right)$, it follows from a simple calculation that the work required does not change.

By taking a limit of this time scaling argument, we only need to erase a $(1,+\infty )$ bit. Here, the infinite-time isothermal protocol, which proceeds by raising the `1’ well infinitesimally, waiting for the system to equilibrate, and repeating, erases for a total work of $k_{B}Tlog2$ since that is the free energy difference between the initial and final state, and there is no extra dissipation. This establishes our claim.

A more detailed version of this calculation can be found in [33]. This work assumes that there is a maximum energy limit $E_{\text{max}}$ to which a state can be raised, so that there will be some small error to erasing. It also makes another assumption about thermalization timescale which translates in our setting to assuming that there is a maximum value to the rates $u_{01}\left(t\right)$ and $u_{10}\left(t\right)$. With these assumptions, they show that the cost of rapid erasing is slightly more than $k_{B}log2$ and goes to $k_{B}Tlog2$ very quickly as the timescale of thermal relaxation becomes smaller and $E_{\text{max}}$ goes to infinity.

4.2.3. Link between KL-Cost and Thermodynamic Work

We will now characterize entropy production in terms of time reversal. This will allow us to make a link between KL-cost and the thermodynamic work W.

We first recall the notion of time reversal of a Markov chain. Usually time-reversal is defined for time-homogeneous Markov chains. However, for the purposes of characterizing entropy production in terms of time reversal, we will work with a definition that applies to time-inhomogeneous Markov chains also. Instead of giving this definition in full generality, we work with a Markov chain with a finite state space. This is sufficient for our purposes and allows us to avoid dealing with certain technical issues.

Note that given a matrix U with positive non-diagonal entries and $U·1=0$, there is a nonnegative vector v such that $v·U=0$. This can be shown by applying the Perron–Frobenius theorem to the exponential matrix $e^U$.

Definition 1 

(Time-reversal). Consider a continuous-time time-inhomogeneous Markov chain with state space $\left[n\right]=\{1,2,\dots ,n\}$ described by a time-dependent transition matrix $U\left(t\right)={\left(u_{ij}\left(t\right)\right)}_{i,j\in \left[n\right]}$ (so that at time t, $u_{ij}\left(t\right)$ denotes the rate of jumping to state j given that the system is in state i). Let $\pi \left(t\right)$ be a sequence of stationary probability distributions on $\left[n\right]$, i.e., $\pi \left(t\right)U\left(t\right)=0$ and ${\pi }_i\ge 0$ for all $i\in \left[n\right]$ and ${\sum }_i{\pi }_i=1$. Then, the time-reversal Markov chain is described by the time-dependent transition matrix $\widehat{U}\left(t\right)={\left({\widehat{u}}_{ij}\left(t\right)\right)}_{i,j\in \left[n\right]}$ where

$${\widehat{u}}_{ij}\left(t\right)=\frac{{\pi }_i\left(t\right)u_{ij}\left(t\right)}{{\pi }_i\left(t\right)}.$$

A Markov chain is reversible if $\widehat{U}=U$.

The justification for considering time-reversal comes from Bayes’ rule. Reversible Markov chains are well-known to be characterized by the conditions of existence of a detailed balanced equilibrium, as well as by the Kolmogorov chain conditions. In particular, two-state Markov chains are always reversible.

For the special case of Equation (3) in particular, given a distribution q at time ${\tau }_e$, the time reversal Markov chain evolves in time according to the ODE:

$$\begin{array}{c}\hfill {\dot{q}}_0\left(t\right)=-u_{01}\left(t\right)q_0\left(t\right)+u_{10}\left(t\right)(1-q_0\left(t\right)),q\left({\tau }_e\right)=q.\end{array}$$

(12)

We see that the difference is that, in Equation (3), the boundary condition was specified at time 0, whereas here the boundary condition is specified at time ${\tau }_e$.

We define the time-reversed measure ${\mu }_{u,q}^{\text{rev}}$ as the measure on path space corresponding to the process described by Equation (12). Strictly speaking, we should write ${\mu }_{u,q,{\tau }_e}^{\text{rev}}$ to denote the time at which the boundary condition is provided to the differential equation, but we will avoid this by using the convention that we are always going to set the boundary condition at time ${\tau }_e$ when considering the time-reversal.

The following result is key to our comparison.

Theorem 1. 

Run the control dynamics Equation (3) forward from initial condition $p\left(0\right)$ upto time ${\tau }_e$ to obtain the distribution $p\left({\tau }_e\right)$. Consider the measure ${\mu }_{u,p\left({\tau }_e\right)}^{\mathit{rev}}$. Then, the total entropy production $S_{\mathit{tot}}\left({\tau }_e\right)$ from time 0 to time ${\tau }_e$ equals

$$S_{\mathit{tot}}\left({\tau }_e\right)=D({\mu }_{u,p\left(0\right)}\parallel {\mu }_{u,p\left({\tau }_e\right)}^{\mathit{rev}}).$$

Proof. 

We will show that the time derivative of the right-hand side (RHS) equals the rate of entropy production. This will prove the theorem.

Fix a time $t\in [0,{\tau }_e]$. Let the probability distribution at time t be represented by $p\left(t\right)=(p_0\left(t\right),p_1\left(t\right))$. Let $f_{ij}:=p_i\left(t\right)u_{ij}\left(t\right)$ denote the flow rate from state i to state $j\ne i$ at time t. Then, $p_i(t+h)=p_i\left(t\right)+h(f_{ji}-f_{ij})+o\left(h\right)$ where $i\ne j$ and $o\left(h\right)$ denotes terms $g\left(h\right)$ such that ${lim}_{h\to 0}g\left(h\right)/h\to 0$.

We will consider the probabilities of the four Markov chain transitions $0\to 0$, $0\to 1$, $1\to 0$ and $1\to 1$ in the interval $(t,t+h)$ in the limit $h\to 0$ according to ${\mu }_{u,p\left(0\right)}$ and according to ${\mu }_{u,p\left({\tau }_e\right)}^{\text{rev}}$. Up to terms of size $o\left(h^2\right)$, we have for $i\ne j$:

$$\begin{array}{cc}\hfill {\mu }_{u,p\left(0\right)}(i\to j)& =h{p}_i\left(t\right)u_{ij}\left(t\right)=h{f}_{ij},\hfill \\ \hfill {\mu }_{u,p\left({\tau }_e\right)}^{\text{rev}}(i\to j)& =h{p}_j(t+h)u_{ji}\left(t\right)=h(f_{ji}+h(f_{ij}-f_{ji})).\hfill \end{array}$$

The increment in the relative entropy in the time interval $(t,t+h)$ equals, up to $o\left(h^2\right)$ terms:

$$\begin{array}{cc}& (1-h{f}_{01})log\frac{1-h{f}_{01}}{1-h{f}_{10}-h^2(f_{01}-f_{10})}+h{f}_{01}log\frac{h{f}_{01}}{h{f}_{10}+h^2(f_{01}-f_{10})}\hfill \\ & +h{f}_{10}log\frac{h{f}_{10}}{h{f}_{01}+h^2(f_{10}-f_{01})}+(1-h{f}_{10})log\frac{1-h{f}_{10}}{1-h{f}_{01}-h^2(f_{10}-f_{01})}.\hfill \end{array}$$

The off-diagonal terms contribute:

$$\begin{array}{c}(1-h{f}_{01})log\frac{1-h{f}_{01}}{1-h{f}_{10}-h^2(f_{01}-f_{10})}+(1-h{f}_{10})log\frac{1-h{f}_{10}}{1-h{f}_{01}-h^2(f_{10}-f_{01})}\hfill \\ \approx (1+o\left(h\right))(h{f}_{10}-h{f}_{01}+o\left(h^2\right))+(1+o\left(h\right))(h{f}_{01}-h{f}_{10}+o\left(h^2\right))\hfill \\ \approx o\left(h^2\right).\hfill \end{array}$$

Divide by h, and take the limit $h\to 0$. We can ignore the off-diagonal terms. The diagonal terms sum to the rate of entropy production $\frac{d{S}_{\text{tot}}}{dt}$ as in Equation (11), and we are done.  ☐

By the First Law of Thermodynamics and Theorem 1,

$$\begin{array}{c}\hfill \Delta W=\Delta F+k_{B}TD({\mu }_{u,p}\parallel {\mu }_{u,p\left({\tau }_e\right)}^{\text{rev}}),\end{array}$$

(13)

where the increase in free energy of the system

$$\Delta F=k_{B}T\left(D(p\left({\tau }_e\right)\parallel \pi )-D(p\left(0\right)\parallel \pi )\right)$$

by Equation (8). Now, to compare our cost function with $\Delta W$.

Theorem 2. 

The $KL$-cost equals change in free energy by $k_{B}T$ plus a path-space relative entropy term that resembles entropy production:

$$\begin{array}{c}\hfill k_{B}TD({\mu }_{u,p}\parallel {\mu }_{k,p})=\Delta F+k_{B}TD({\mu }_{u,p}\parallel {\mu }_{k,p\left({\tau }_e\right)}^{\mathit{rev}}),\end{array}$$

(14)

where $p\left({\tau }_e\right)$ is—as in Equation (13)—the solution to the control dynamics Equation (3) at time ${\tau }_e$.

Proof. 

Using Equation (8), we can rewrite the claim as

$$D({\mu }_{u,p}\parallel {\mu }_{k,p})+D(p\left(0\right)\parallel \pi )=D({\mu }_{u,p}\parallel {\mu }_{k,p\left({\tau }_e\right)}^{\text{rev}})+D(p\left({\tau }_e\right)\parallel \pi ).$$

Both left-hand side (LHS) and RHS equal $D({\mu }_{u,p}\parallel {\mu }_{k,\pi })$. The assertion for the LHS is straightforward. The assertion for the RHS is true because time-reversal dynamics was defined to keep the stationary distribution π remaining stationary under time reversal.  ☐

Comparing Equations (13) and (14), a KL control treatment replaces the total entropy production $D({\mu }_{u,p}\parallel {\mu }_{u,p\left({\tau }_e\right)}^{\text{rev}})$ in Equation (13) by the new term $D({\mu }_{u,p}\parallel {\mu }_{k,p\left({\tau }_e\right)}^{\text{rev}})$ which compares the control dynamics with the time reversal of the passive dynamics. This suggests an interpretation as follows. If we applied the control during some time interval $[0,{\tau }_e]$, and remembered what control we applied, then the entropy production is correctly given by $D({\mu }_{u,p}\parallel {\mu }_{u,p\left({\tau }_e\right)}^{\text{rev}})$. However, the information that a control was applied also needs to be stored somewhere. If we forget that a control was applied, and if application of the control is very rare, then our default model for the dynamics should be much closer to the passive dynamics. In this case, entropy production may be closer to the value $D({\mu }_{u,p}\parallel {\mu }_{k,p\left({\tau }_e\right)}^{\text{rev}})$.

4.3. Large Deviations Interpretation

Our cost function $D(\mu \parallel \nu )$ also admits a large deviation interpretation which was, remarkably, already noted by Schrödinger in 1931 [34,35,36,37]. Motivated by quantum mechanics, Schrödinger asked: conditioned on a more or less astonishing observation of a system at two extremes of a time interval, what is the least astonishing way in which the dynamics in the interval could have proceeded? Specializing to our problem of erasing, suppose an ensemble of two-state Markov chain with passive dynamics given by Equation (1) was observed at time 0 and at time ${\tau }_e$. Suppose the empirical state distribution over the ensemble was found to be the equilibrium distribution π at time 0, and (1, 0) at time ${\tau }_e$, respectively. This would be astonishing because no control has been applied, yet the ensemble has arrived at a state of higher free energy. Conditioned on this rare event having taken place, what is the least unlikely measure ${\mu }^{*}$ on path space via which the process took place?

By a statistical treatment of multiple single particle trajectories, Schrödinger found that the likelihood of an empirical measure μ on path space falls exponentially fast with the relative entropy $D(\mu \parallel \nu )$, where ν is the measure induced by the passive dynamics. In particular, the least unlikely measure ${\mu }^{*}$ is that measure which—among all μ whose marginals at time 0 and time ${\tau }_e$ respect the observations—minimizes $D(\mu \parallel \nu )$. Thus, for the problem of erasing, where $k_{01}=k_{10}$, the measure μ varies over all measures that have marginal $(1/2,1/2)$ at time 0 and marginal (1, 0) at time ${\tau }_e$, and ${\mu }^{*}$ is that measure among all such μ that minimizes $D(\mu \parallel {\mu }_{k,(1/2,1/2)})$. Thus, our optimal control produces in expectation the least surprising trajectory among all controls that perform rapid erasing.

4.4. Gibbs Measure

Equation (6) is not accidental for this example, but is in fact a general feature when the cost function is relative entropy [28]. More abstractly, the Radon–Nikodym derivative (i.e., “probability density”) $\frac{d{\mu }^{*}}{d\nu }$ of the measure ${\mu }^{*}$ induced on path space by the optimal control $u^{*}$ is a Gibbs measure with respect to the measure ν induced by the passive dynamics, with the cost-to-go function $v\left(t\right)$ playing the role of an energy function. In other words, mathematically our problem is precisely the free energy minimization problem so familiar from statistical mechanics. There is also a possible physical interpretation: we are choosing paths in $\mathcal{P}$ as microstates, instead of points in phase space. The idea of paths as microstates has occurred before [38].

5. Conclusions

Since charging a battery can also be thought of as erasing a bit [4], our result may also hold insights into the limits of efficiencies of rapidly charging batteries that must simultaneously hold their energy for a long time.

So long as the noise is Markovian, we conjecture that the KL cost for erasing the two-state Markov chain is a lower bound for more general cases—for example, for bits with Langevin dynamics [39]—which is a stochastic differential equation expressing Newton’s laws of motion with Brownian noise perturbations.