Random Walk Approximation for Stochastic Processes on Graphs

Abstract

We introduce the Random Walk Approximation (RWA), a new method to approximate the stationary solution of master equations describing stochastic processes taking place on graphs. Our approximation can be used for all processes governed by non-linear master equations without long-range interactions and with a conserved number of entities, which are typical in biological systems, such as gene regulatory or chemical reaction networks, where no exact solution exists. For linear systems, the RWA becomes the exact result obtained from the maximum entropy principle. The RWA allows having a simple analytical, even though approximated, form of the solution, which is global and easier to deal with than the standard System Size Expansion (SSE). Here, we give some theoretically sufficient conditions for the validity of the RWA and estimate the order of error calculated by the approximation with respect to the number of particles. We compare RWA with SSE for two examples, a toy model and the more realistic dual phosphorylation cycle, governed by the same underlying process. Both approximations are compared with the exact integration of the master equation, showing for the RWA good performances of the same order or better than the SSE, even in regions where sufficient conditions are not met.

1. Introduction

Stochastic processes are ubiquitous in nature and generate fluctuations that are not just noise, in particular in biochemical reactions occurring in single cells, where the copy number of the reactants can be relatively small, i.e., of the order of few hundreds [1,2]. At first glance, these molecules present in a small number were neglected and considered as unimportant, also because of the experimental difficulties to detect them. However, it soon became clear that in a biological system, all the reactions are coupled and even low copy numbers of, for instance, messenger RNA in bacteria [1] can determine the fate of the cell and result in peculiar properties due to large fluctuations, which can be very far from standard Poisson statistics [3]. Therefore, all stochastic properties became important to make predictions in that it was realized that noise can govern gene expression and in turn evolution [4,5], and it is necessary to adapt to new biological challenges [6]. Sometimes, fluctuations themselves can even generate phase transitions [7].

In these systems, it has thus been understood that considering only the macroscopic dynamics, neglecting then the stochasticity of the process, can be misleading and works only in very large systems. Therefore, a stochastic description, based on the master equation (ME), started being necessary [8]. Although in some simple cases, the ME can be solved analytically, mainly for linear processes, i.e., when the transition rates are constant, as for Brownian motion, most of the time, this is not possible. Those are, of course, cases of great interest, where self-organization phenomena occur, and the actual state of the system regulates its stability and evolution [9] and where the cell fate is determined by positive or negative feedback loops [10]. Moreover, nonlinearities are often at the origin of interesting phase transitions of biologic importance. Examples are gene expression networks [11], networks of chemical reactions showing non-equilibrium steady states and net fluxes, such as signal cellular pathways [12,13,14,15] and the phospho/dephosphorylation cycle of AMPA receptors in a single synaptic spine [16]. When the number of states of the system under study is finite, the process can be seen as a random walk on a graph. In this picture, a state of the system would be each possible observed form of it, quantified by some observables such as levels of proteins, messengers, organelles, or phenotypes. For instance, for gene expression networks, it would be the number of attractors (i.e., the number of cell types in the Waddington’s epigenetic landscape [11]) or the different levels of phosphorylation of a molecule in dual phosphorylation cycles [9] or again the number of chemicals involved in a signal transduction pathway. Each state of the system is then related to each other with some transition rates, generating a weighted graph, in which the weights associated to each edge are the transition rates and the nodes are the possible states of the system. This is a directed graph, since the pairs of nodes defining an edge are ordered by the direction of the transition; moreover, interesting behaviors, such as bifurcation phenomena, appear when the rates are not symmetric.

Recently, a lot of effort has been put into solving ME associated with non-linear systems on both developing accurate numerical techniques [17] and developing theoretical frameworks and approximation methods. Concerning the former, most of the numerical methods are derived from the progenitor of Monte Carlo sampling for the ME, the Gillespie algorithm [18], and the one exploiting the structure of the ME is remarkable [19]. On the other hand, the latter are mainly built on known results about the linear case, exploiting, for instance, the reaction velocities and then neglecting fast variables [20]. Theoretical methods to describe these systems at a mesoscopic scale can be divided into two main approaches: solving directly the ME by approximating the rates as constants or using the SSE, which transforms the ME in a diffusion equation by developing in terms of the system size [8,21], which is also known as van Kampen’s $\Omega$-expansion.

Here, we present a new approach for giving an approximation of the solution of the ME exploiting the graph structure, the Random Walk Approximation (RWA), and we compare it with commonly used methods. Our method leverages both the two approaches mentioned above: we use the solution of the linear ME (the multinomial distribution), but we do not impose the transition rates to be constant, therefore hardly modifying the multinomial shape. This, as we will show, has two main advantages. First, we obtain a global solution and not only a local one, as it is with the SSE or with the pure multinomial approximation obtained approximating the transition rates as constant. The global solution is therefore able to approximate also the tail of the distribution and can contain itself the phase transition; if one exists, it allows indeed to establish whether a system undergoes a critical bifurcation point without further calculations. Second, it is much simpler and easier to use than the SSE, since all is needed is to solve an eigenvalue problem. Indeed, it gives an analytical solution that can be easily used for further calculations, such as, for instance, the computation of the entropy production of the process. Our method is general and can be applied to all systems governed by a ME, no matter the number of states, as long as the transition rates are given.

The paper is organized as follows. In Section 2, we introduce the RWA from a mathematical point of view and give some theoretical results on its performance. In Section 3, we briefly explain the numerical and analytical methods used as comparison with our RWA. In Section 4, we describe the common backbone of the models to which the approximation is applied and then give the explicit theoretical results for, first, a simple toy model considered to check the validity of the RWA and its essential properties, and, second, a more relevant model of biological importance, the dual phospho/dephosphorylation cycle (PdPC). Those models are chosen because analytical calculations were possible and for the ease of numerical implementation, which is useful to clarify how and to what extent the RWA can be successfully applied. In Section 5, we show the results of the application of the RWA to the described models and finally give a global discussion of our work.

2. The Random Walk Approximation

We introduce here the main topic of the paper, namely the Random Walk Approximation. In particular, among some theoretical results about its expected behavior, we state the sufficient conditions for it to be a reliable approximation in the sense of the ${\ell }^1$-norm.

2.1. Linear Master Equation for One Step Processes

We consider a generic system that is described by an ensemble of $i=1,\cdots ,M$ states with transition rates ${\pi }_{ij}$ from the state $j\to i$, which defines the probability that a “particle” (individual of the system) changes its state on a time unit. Therefore, one obtains a weighted graph (network) with M nodes and link weights given by the rates ${\pi }_{ij}$. If one considers the dynamics of N identical independent particles, the statistical properties of the system are described by the particle distribution function $\rho (\overrightarrow{n},t)$, which gives the probability of the network state $\overrightarrow{n}=(n_1,\cdots ,n_M)$ where $n_i$ is the number of particles in the state i at time t. Since

$$|\overrightarrow{n}|=\sum _i{n}_i=N$$

is fixed, we are realizing a microcanonical ensemble (generalizations are possible by introducing an external node representing the reservoir coupled with the system). The evolution of the distribution function $\rho (\overrightarrow{n},t)$ is the master equation:

$$\frac{\partial \rho }{\partial t}(\overrightarrow{n},t)=\frac{1}{N}\sum _{i,j}\left[{\pi }_{ij}{E}_j^{+}{E}_i^{-}{n}_j-{\pi }_{ji}{n}_i\right]\rho (\overrightarrow{n},t),$$

(1)

assuming the one-step process approximation (i.e., the probability that two particles move simultaneously in a time interval $\Delta t$ is assumed to be $o(\Delta t$)). The one-step process is a continuous time Markov process, whose transition rates matrix allows only transitions between neighboring network states, which is not a strong experimental assumption, as long as you have a time resolution much higher than the typical transition time of a particle. The symbols $E_i^{\pm }$ denote the Van Kampen operators defined by:

$$E_i^{\pm }f\left(\overrightarrow{n}\right)=f(\overrightarrow{n}\pm {\widehat{e}}_i)$$

for any function $f\left(\overrightarrow{n}\right)$, where ${\widehat{e}}_i$ is the standard canonical base of ${\mathbb{Z}}^M$.

If the transition rates matrix ${\pi }_{ij}$ does not satisfy the detailed balance condition, the same is true for the ME (1); however, it is possible to prove that the stationary solution for the linear random walk is, regardless of whether detailed balance is verified or not, a maximum entropy distribution with the constraint that the average value of the particles in each node is finite (see Appendix A). In the following, detailed balance will not be assumed; therefore, we will deal, in general, with non-equilibrium processes. Moreover, in Equation (1), the distribution function $\rho \left(\overrightarrow{n}\right)$ is extended to all possible states $\overrightarrow{n}\in {\mathbb{Z}}^M$, defining $\rho (\overrightarrow{n},t)=0$ for all the non-physical states $|\overrightarrow{n}|\ne N$. Since the particles do not interact, it is also possible to analytically compute the solution $\rho (\overrightarrow{n},t)$ from the spectral properties of the Laplacian matrix [22] of the graph

$$L_{ki}\left(\overrightarrow{n}\right)={\delta }_{ki}{d}_k\left(\overrightarrow{n}\right)-{\pi }_{ki}\left(\overrightarrow{n}\right)d_k\left(\overrightarrow{n}\right)=\sum _i{\pi }_{ik}\left(\overrightarrow{n}\right),$$

(2)

where, for the linear case, the rates are independent of $\overrightarrow{n}$. In particular, the stationary solution is given by the multinomial distribution

$$\rho \left(\overrightarrow{n}\right)=N!\prod _{i=1}^M\frac{p_i^{n_i}}{n_i!},$$

(3)

where the vector $\overrightarrow{p}=(p_1,\dots ,p_M)$ is the eigenvector corresponding to the null eigenvalue of L and the covariance matrix of the solution reads

$$C_{ij}=N{p}_i{\delta }_{ij}-N{p}_i{p}_j.$$

(4)

We note that all the other eigenvalues of the Laplacian are positive and the second smallest eigenvalue is known as the Fiedler’s eigenvalue [22].

The multinomial solution is the same elegantly obtained by the application of the maximum entropy principle, as we show in Appendix A.

For $N\gg 1$, the multinomial converges to a symmetric distribution and the average values $\langle n_i\rangle =N{p}_i$ are also the mode of the multinomial distribution. The average dynamics can be directly computed from the ME

$$\begin{array}{ccc}\hfill \langle {\dot{n}}_k\rangle & =& \sum _{\left|n\right|=N}{n}_i\frac{\partial \rho }{\partial t}(\overrightarrow{n},t)=\frac{1}{N}\sum _{i,j}\sum _{\left|n\right|=N}{n}_k\left[{\pi }_{ij}{E}_j^{+}{E}_i^{-}{n}_j-{\pi }_{ji}{n}_i\right]\rho (\overrightarrow{n},t)\hfill \\ & =& \sum _{i,j}\sum _{\left|n\right|=N}\left[{\pi }_{ij}{E}_j^{+}{E}_i^{-}{\pi }_{ij}(n_k-{\delta }_{jk}+{\delta }_{ik})n_j-{\pi }_{ji}{n}_k{n}_i\right]\rho (\overrightarrow{n},t)\hfill \\ & =& \sum _j({\pi }_{kj}\langle n_j\rangle -{\pi }_{jk}\langle n_k\rangle ),\hfill \end{array}$$

(5)

whose critical points are the average values.

2.2. The Non-Linear Case

We consider now how the previous results generalize to non-linear random walks on graphs, when the transition probability rates depend on the network states ${\pi }_{ij}={\pi }_{ij}\left(\overrightarrow{n}\right)$. These models allow to simulate the effect of particle interactions at the node, but a physical interpretation is needed to justify the one-step process assumption in the formulation of the ME. Indeed, each time a particle moves, the transition probabilities are instantaneously updated before another particle moves. Therefore, a synchronous evolution of the network in which many particles move at the same time gives rise to a different dynamical system. In the case ${\pi }_{ij}\left(\overrightarrow{n}\right)={\pi }_{ij}(n_i,n_j)$, the interactions are local (Markov random field) and the one-step process assumption is physically justified. The corresponding ME is:

$$\frac{\partial \rho }{\partial t}(\overrightarrow{n},t)=\frac{1}{N}\sum _{i,j}\left[E_j^{+}{E}_i^{-}{\pi }_{ij}\left(\overrightarrow{n}\right)n_j-{\pi }_{ji}\left(\overrightarrow{n}\right)n_i\right]\rho (\overrightarrow{n},t).$$

(6)

We observe that the factor $1/N$ that defines the one-step process is a time scaling and scales the spectral properties of the Laplacian operator (defined hereafter), introducing a Fiedler’s eigenvalue of $O\left(N^{-1}\right)$. Then, it is convenient to scale the time by a factor $1/N$ and remove this factor from the equation. Moreover, we assume ${\pi }_{ij}\left(\overrightarrow{n}\right)={\pi }_{ij}\left(\overrightarrow{x}\right)$ where $x_i=n_i/N$, that is, the particle interactions depend on the density at each node i, in the limit $N\gg 1$. The stationary points of Equation (6) correspond to the eigenvectors $\overrightarrow{p}\left(\overrightarrow{x}\right)$ associated with the null eigenvalue of the Laplacian (2) (here in the density-dependent version) that satisfy the self-consistent condition:

$$p_i^{\ast }\left({\overrightarrow{x}}^{\ast }\right)=x_i^{\ast }\mathrm{with}\sum _i{x}_i^{\ast }=1,$$

(7)

where ${\overrightarrow{x}}^{\ast }$ is the stationary solution of the deterministic dynamics associated with ME (6), ${\sum }_i{L}_{ji}\left(x_i^{\ast }\right)x_i^{\ast }=0$.

An explicit solution for the stationary distribution of the ME (6) is difficult due to the non-linear nature of the problem, except in the cases where the detailed balance holds. Indeed, detailed balance introduces the constraint of zero current between each pair of states, i.e., the process is at equilibrium, leading to a Maxwell–Boltzmann distribution with a potential energy depending on the state of the system (differently from the standard linear case) [9,23]. The potential energy can be computed by recursion from a function of the rates of the process, starting from an arbitrary value which will be uniquely determined by the normalization of $\rho$.

In order to build an approximate stationary solution, we introduce the eigenvector of components $p_k\left(\overrightarrow{x}\right)$ and consider the multinomial-like solution, which we refer to as the Random Walk Approximation of the stationary solution:

$${\rho }_{\overrightarrow{p}}^{\ast }\left(\overrightarrow{n}\right)=C\left(N\right)N!\prod _{k=1}^M\frac{p_k^{n_k}\left(\overrightarrow{x}\right)}{n_k!}{x}_k=n_k/N,$$

(8)

where $C\left(N\right)$ is a normalizing factor. We are now interested in analyzing the behavior of the RWA with respect to the system size N and giving an estimate of the approximation error. First, in order to show that $C\left(N\right)$ does not have a strong dependence on N, we use a perturbative approach by considering small perturbations of $\overrightarrow{p}\left(\overrightarrow{x}\right)$ around the stationary points: $p_k\left(\overrightarrow{x}\right)=p_k^{\ast }+\Delta p_k\left(\overrightarrow{x}\right)$, where the perturbations $\Delta p_k$ satisfy the condition ${\sum }_k\Delta p_k=0$. Then, by injecting this into (8), we show that $C\left(N\right)=O\left(1\right)$ for $N\gg 1$, with a weak dependence on N (see detailed calculation in Appendix B).

We now give some conditions on the validity of Equation (7) for the critical points. For $N\gg 1$, we compute the modes of the distribution (8) from the condition

$$log\frac{n_i}{N{p}_i}-\sum _k\frac{\partial p_k}{\partial x_i}\frac{n_k}{N{p}_k}\simeq 0$$

If one introduces the $z_k=n_k/\left(N{p}_k\right)=x_k/p_k$, the relation can be written in the form

$$log{z}_i=\sum _k\frac{\partial p_k}{\partial x_i}{z}_k,$$

(9)

which is clearly satisfied for $z_k=1$, since:

$$\sum _k\frac{\partial p_k\left(\overrightarrow{x}\right)}{\partial x_i}=0.$$

Therefore, as long as the matrix $\partial p/\partial x$ (i.e., the transpose of the Jacobian matrix of the eigenvector $\overrightarrow{p}\left(\overrightarrow{x}\right)$) has all the eigenvalues $\lambda <1$, the self-consistent average solution $x_i^{\ast }=p_i^{\ast }\left({\overrightarrow{x}}^{\ast }\right)$ (the unperturbed solution for $\Delta p_k=0$) is the only critical point of the distribution (8) and the distribution is peaked at the critical point with a spread $O\left(\sqrt{N}\right)$. This is because one cannot have a tangency condition between $log{z}_i$ and the right-hand side of Equation (9) in such a way that self-consistency is verified. When we have an eigenvalue $\lambda \ge 1$, the perturbation $\Delta p_k$ may introduce other solutions to Equation (9) that are critical points for the RWA distribution but not for the stationary distribution of Equation (6). We remark that this condition is also the condition necessary for a bifurcation phenomenon, i.e., when the distribution (8) becomes bi-(or multi-)modal. Indeed, if one considers the self-consistent Equation (7) for the average dynamics, the existence of a bifurcation is equivalent to the existence of a null eigenvalue for the matrix

$${\delta }_{ik}-\frac{\partial p_i}{\partial x_k},$$

computed at the critical point. In other words, when a bifurcation phenomenon occurs for the self-consistent critical points of the average dynamics (7), the RWA (8) may have spurious stationary points that do not correspond to those of the average dynamics. Then, in this case, it is not guaranteed that the RWA is a good global approximation of the ME stationary solution.

We are now ready to compute the error of the approximation of the RWA (8) and its scaling with N. We will show that it depends on the derivative of the probabilities $p_i\left(\overrightarrow{x}\right)$. The intuition behind this is that fluctuations should be small with respect to the inhomogeneity of the system, which is represented by the derivatives of the $p_i\left(\overrightarrow{x}\right)$. An estimate of the error $|{\rho }_{\overrightarrow{p}}^{\ast }\left(\overrightarrow{n}\right)-{\rho }^s\left(\overrightarrow{n}\right)|,$ where ${\rho }^s$ is the exact stationary distribution, can be achieved by substituting the distribution (8) in the ME:

$${\ell }^1\text{-error}=\sum _{i,j}\left[E_j^{+}{E}_i^{-}{\pi }_{ij}\left(\overrightarrow{x}\right)n_j-{\pi }_{ji}\left(\overrightarrow{x}\right)n_i\right]{\rho }_{\overrightarrow{p}}^{\ast }\left(\overrightarrow{n}\right)$$

When $N\gg 1$, the main contribution to the error is due to the dependence of $p_k$ from the densities $\overrightarrow{x}$:

$${\ell }^1\text{-error}\simeq \frac{1}{N}\sum _{i,j}{\pi }_{ij}\left(\overrightarrow{x}\right)n_j\sum _k\left(\frac{\partial p_k}{\partial x_j}-\frac{\partial p_k}{\partial x_i}\right)\frac{n_k}{p_k}{\rho }_{\overrightarrow{p}}^{\ast }\left(\overrightarrow{n}\right).$$

(10)

Considering the fluctuations around the critical point of order $\Delta n_i=p_{i}O\left(\sqrt{N}\right)$ in the limit $N\gg 1$, the largest contribution to Equation (10) is proved to be of order $O\left(∥\frac{\partial p}{\partial x}∥\right)$ (details in Appendix C). Finally, we can write the estimate for the ${\ell }^1$-error:

$${∥{\rho }_{\overrightarrow{p}}^{\ast }\left(\overrightarrow{n}\right)-{\rho }^s\left(\overrightarrow{n}\right)∥}_1=O\left(∥\frac{\partial p}{\partial x}∥\right),$$

(11)

this means that the error does not depend on N, since the rates depend only on the densities $\overrightarrow{x}$ and not on N. The error is then independent of N if the distribution reaches its peak at the critical values with a spread $O\left(\sqrt{N}\right)$. At a bifurcation of the critical point, when a bimodal distribution is expected, the previous estimate (11) could no longer be valid for both the increased spread of the peak and for the existence of a very small Fiedler’s eigenvalue for the matrix $L_{ij}\left(\overrightarrow{x}\right)$.

3. Methods

As stated previously, except for systems in which the detailed balance condition holds, it is a challenging task to compute the stationary distribution because of the non-linearity of the problem. In this section, a standard method that will be applied in the following, the System Size Expansion (SSE) is presented and adapted to the situation of interest. The exact solution of the ME is given by the numerical integration of the ME, obtained with the Runge–Kutta (RK) algorithm [24,25] (RK5(4) or Runge–Kutta–Dormand–Prince, Python 3.10.4, mainly from module scipy v1.9.1), in an adapted way for one-step processes, which fastens the numerical convergence [26]. Details are given in Appendix D, and the code for all the results on the dual PdPC used in this paper is publicly available on GitHub [27].

3.1. System Size Expansion

The SSE considers the thermodynamic limit $N\to \infty$ of the master Equation (1), or (6) for non-linear processes, in the continuous variables $\overrightarrow{x}=\overrightarrow{n}/N$. Developing the ME up to terms of order $O\left(N^{-2}\right)$ one obtains the Fokker–Planck (FP) equation associated to the ME. If for linear cases, the SSE is straightforward [8], the calculations for the non-linear case are a little more cumbersome. The details of the calculations for both cases are given in Appendix E. In both cases, the SSE locally approximates the multinomial distribution (8) as $N\to \infty$ by a Gaussian distribution in the neighborhood $\delta \overrightarrow{x}=\overrightarrow{x}-{\overrightarrow{x}}^{\ast }$ of the critical point ${\overrightarrow{x}}^{\ast }$. The general solution for the non-linear case is the Gaussian:

$${\rho }_{SSE}^{\ast }\left(\delta \overrightarrow{x}\right)=\frac{1}{{\left(2\pi \right)}^{3/2}{\left|\mathsf{\Sigma }\right|}^{1/2}}exp\left[-\frac{1}{2}\delta {\overrightarrow{x}}^T{\mathsf{\Sigma }}^{-1}\delta \overrightarrow{x}\right],$$

(12)

where the covariance matrix $\mathsf{\Sigma }$ solves the continuous Lyapunov equation [8]:

$$-\overline{A}\mathsf{\Sigma }-\mathsf{\Sigma }{\overline{A}}^T+D^{\ast }=0$$

(13)

in which we denoted as $\overline{A}$ the matrix such that:

$${\overline{A}}_{ij}=\sum _k\left[L_{ik}^{\ast }{\delta }_{jk}+\frac{\partial L_{ik}}{\partial x_j^{\ast }}{x}_k^{\ast }\right],$$

(14)

where $L_{ik}^{\ast }$ is the Laplacian at the critical point, and with a slight abuse of notation, we denoted $\frac{\partial L_{ik}}{\partial x_j^{\ast }}$ as the derivative of the Laplacian calculated at the critical point. The positive symmetric matrix $D^{\ast }$ is the diffusion matrix:

$$D_{ij}\left(\overrightarrow{x}\right)=\frac{1}{N}\left[\sum _k{\delta }_{ij}\left({\pi }_{ik}\left(\overrightarrow{x}\right)x_k+{\pi }_{ki}\left(\overrightarrow{x}\right)x_i\right)-\left({\pi }_{ij}\left(\overrightarrow{x}\right)x_j+{\pi }_{ji}\left(\overrightarrow{x}\right)x_i\right)\right],$$

calculated as well at the critical point. In the non-linear case, these equations can only be solved numerically.

4. Model

The general setup we chose to test the RWA consists of a three-state (chemical species) process ruled by two underlying reaction cycles; a general scheme is reported in Figure 1. This is a typical setup for biochemical reactions in single cells, and it is a still reasonably simple setup to apply the RWA.

The state of the system is represented by a three-dimensional vector $\overrightarrow{n}=(n_A,n_B,n_C),$ and it can be reduced to two dimensions $\overrightarrow{n}=(n_A,N-n_A-n_C,n_C)$ by assuming the conservation on the total number of particles $|\overrightarrow{n}|=N$. The dependence of the dynamics with respect to $\overrightarrow{n}$ can be removed by defining the concentration vector $\overrightarrow{x}=\overrightarrow{n}/N$. The transition rates ${\pi }_{ij}$ from a species j to a species i may depend on this vector, and they allow to build a parameterized transition matrix $\Pi \left(\overrightarrow{x}\right)$ and Laplacian matrix $L\left(\overrightarrow{x}\right)$.

In order to analyze the time evolution of the system, we write down explicitly the deterministic equation of the dynamics for the density (or concentration) vector:

$$\left\{\begin{array}{c}\frac{d{x}_A}{dt}={\pi }_{AB}\left(\overrightarrow{x}\right)x_B-{\pi }_{BA}\left(\overrightarrow{x}\right)x_A\hfill \\ \frac{d{x}_C}{dt}={\pi }_{CB}\left(\overrightarrow{x}\right)x_B-{\pi }_{BC}\left(\overrightarrow{x}\right)x_C\hfill \end{array}\right.$$

(15)

Notice that this system of equations is equivalent to the system associated to the dynamics of the graph obtainable through the Laplacian matrix:

$$\frac{d\overrightarrow{x}}{dt}=-L\left(\overrightarrow{x}\right)\overrightarrow{x}$$

(16)

in which the equation associated to the time derivative of $x_B$ is redundant as a consequence of the constraint on the concentrations $|\overrightarrow{x}|=1$. The critical states of the system, which can be stable or unstable, correspond to the vectors ${\overrightarrow{x}}^{\ast }$ with a null time derivative in (15): this is equivalent to considering the macroscopic dynamics. At the base of the RWA, there is the eigenvector with null eigenvalue of the Laplacian $L\left(\overrightarrow{x}\right)$, that for our three-state model depends on the current $\overrightarrow{x}$ by:

$$\begin{array}{cc}\hfill p_A\left(\overrightarrow{x}\right)& =\frac{{\pi }_{AB}\left(\overrightarrow{x}\right){\pi }_{BC}\left(\overrightarrow{x}\right)}{{\pi }_{AB}\left(\overrightarrow{x}\right){\pi }_{BC}\left(\overrightarrow{x}\right)+{\pi }_{CB}\left(\overrightarrow{x}\right){\pi }_{BA}\left(\overrightarrow{x}\right)+{\pi }_{BA}\left(\overrightarrow{x}\right){\pi }_{BC}\left(\overrightarrow{x}\right)}\hfill \\ \hfill p_C\left(\overrightarrow{x}\right)& =\frac{{\pi }_{CB}\left(\overrightarrow{x}\right){\pi }_{BA}\left(\overrightarrow{x}\right)}{{\pi }_{AB}\left(\overrightarrow{x}\right){\pi }_{BC}\left(\overrightarrow{x}\right)+{\pi }_{CB}\left(\overrightarrow{x}\right){\pi }_{BA}\left(\overrightarrow{x}\right)+{\pi }_{BA}\left(\overrightarrow{x}\right){\pi }_{BC}\left(\overrightarrow{x}\right)}\hfill \end{array}$$

with the self-consistent condition (7).

If we want instead to consider the stochasticity of the process, the time evolution of the probability distribution $\rho (\overrightarrow{x},t)$ associated to the states of the system is constructed by taking into account all of the possible exchanges of the particles, as in (6). Explicitly, the ME reads:

$$\begin{array}{cc}\hfill \frac{\partial \rho (x_A,x_C,t)}{\partial t}=& {\pi }_{AB}\left(x_A-\frac{1}{N},x_C\right)\left(1-x_A+\frac{1}{N}-x_C\right)\rho \left(x_A-\frac{1}{N},x_C,t\right)+\hfill \\ \hfill & -{\pi }_{AB}(x_A,x_C)(1-x_A-x_C)\rho (x_A,x_C,t)\hfill \\ \hfill & +{\pi }_{BA}\left(x_A+\frac{1}{N},x_C\right)\left(x_A+\frac{1}{N}\right)\rho \left(x_A+\frac{1}{N},x_C,t\right)+\hfill \\ \hfill & -{\pi }_{BA}(x_A,x_C)x_A\rho (x_A,x_C,t)\hfill \\ \hfill & +{\pi }_{CB}\left(x_A,x_C-\frac{1}{N}\right)\left(1-x_A-x_C+\frac{1}{N}\right)\rho \left(x_A,x_C-\frac{1}{N},t\right)+\hfill \\ \hfill & -{\pi }_{CB}(x_A,x_C)(1-x_A-x_C)\rho (x_A,x_C,t)\hfill \\ \hfill & +{\pi }_{BC}\left(x_A,x_C+\frac{1}{N}\right)\left(x_C+\frac{1}{N}\right)\rho \left(x_A,x_C+\frac{1}{N},t\right)+\hfill \\ \hfill & -{\pi }_{BC}(x_A,x_C)x_C\rho (x_A,x_C,t),\hfill \end{array}$$

(17)

where we explicitly removed the dependence on $x_B$. Therefore, while the macroscopic approach provides an average dynamics of the system in the $N\to \infty$ limit, the ME describes statistically the time evolution of the probability distribution together with its stationary properties. In the thermodynamics limit $N\to \infty$, fluctuations are negligible, and the stationary state of the ME approach recovers the macroscopic kinetics, since the distribution converges toward a delta function peaked on the critical point of the average dynamics. If the system satisfies the detailed balance condition, the stationary solution coincides with the equilibrium solution and it can be computed in a closed form corresponding to a Maxwell–Boltzmann distribution [28]. However, in non-equilibrium steady states, an explicit solution of the stationary distribution cannot be obtained as a consequence of the effect of stationary currents.

According to the functional form of the transition matrix $\Pi \left(\overrightarrow{x}\right)$, we consider two different models that follow the scheme in Figure 1: a toy model and a biologically-inspired model, i.e., the dual PdPC.

4.1. Toy Model

As a consequence of their complex dynamics, biologically inspired models often depend on a large set of parameters, and the sensitivity on this high-dimensional parameters space can make it difficult to carry out a systematic study of the model itself. Therefore, before generalizing to a more reliable and known model such as the dual PdPC, we perform the analysis on a simple model depending on a one-dimensional parameter space. In particular, referring to Figure 1, the rates are:

$$\begin{array}{cc}\hfill {\pi }_{AB}& =1{\pi }_{BA}\left(\overrightarrow{x}\right)=\theta \left(x_A\right)\hfill \\ \hfill {\pi }_{CB}& =1{\pi }_{BC}\left(\overrightarrow{x}\right)=\theta \left(x_C\right)\hfill \end{array}$$

in which $\theta \left(z\right)$ is a threshold function depending on the unique control parameter $\alpha$ in the form:

$$\theta \left(x\right)=1-\alpha +\alpha {(1-x)}^2,\alpha \in [0,1].$$

(18)

When $\alpha =0$, the system is linear, and all the transitions are equally likely, whereas on the opposite, when $\alpha =1$, a large fraction of particles in states A and C drops the transition rates toward state B, leading to bistability. This model can be interpreted as a Markov process on a graph whose time evolution is governed by Equation (16). It is possible to compute the critical condition for the bifurcation of the symmetric equilibrium $x_1=x_3$ for the average self-consistent Equation (7):

$$f\left(x\right)=1+\frac{d\theta }{dx}\frac{1}{\theta \left(x\right)(2+\theta (x\left)\right)}=0,$$

(19)

of which a solution can be computed numerically and is for $0.8<\alpha <0.9$.

4.2. Dual Phospho/Dephosphorylation Cycles

We analyze here the dual phospho/dephosphorylation cycles. In order to analyze the time evolution of the system, we make use of the Michaelis–Menten (MM) approach, which provides an average description of the dynamics of the enzyme kinetics under the hypothesis of the quasi-steady-state approximation. In particular, the latter consists of assuming a constant concentration for the enzyme–substrate complex and results in rates being a non-linear function of the system state. The components of the concentration vector $x_A$, $x_B$ and $x_C$ represent, respectively, the unphosphorylated, phosphorylated and double-phosphorylated substrates. Each of the four reactions of the dual PdPC follows the same MM enzyme kinetic scheme, which consists of a reversible and an irreversible process. In particular, the former involves a substrate S which binds to an enzyme E, forming an enzyme–substrate complex $ES$. The latter uses this complex to produce a product P and regenerates the free enzyme E.

The whole system of reactions reads:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathrm{x}}_{\mathrm{A}}+{\mathrm{E}}_1\underset{{\mathrm{k}}_{b1}}{\overset{{\mathrm{k}}_{f1}}{\rightleftharpoons }}{\mathrm{E}}_1{\mathrm{x}}_{\mathrm{A}}\stackrel{{\mathrm{k}}_{c1}}{\to }{\mathrm{x}}_{\mathrm{B}}+{\mathrm{E}}_1\end{array}\\ \hfill \begin{array}{c}\hfill {\mathrm{x}}_{\mathrm{B}}+{\mathrm{E}}_1\underset{{\mathrm{k}}_{b2}}{\overset{{\mathrm{k}}_{f2}}{\rightleftharpoons }}{\mathrm{E}}_1{\mathrm{x}}_{\mathrm{B}}\stackrel{{\mathrm{k}}_{c2}}{\to }{\mathrm{x}}_{\mathrm{C}}+{\mathrm{E}}_1\end{array}\\ \hfill \begin{array}{c}\hfill {\mathrm{x}}_{\mathrm{B}}+{\mathrm{E}}_2\underset{{\mathrm{k}}_{b3}}{\overset{{\mathrm{k}}_{f3}}{\rightleftharpoons }}{\mathrm{E}}_2{\mathrm{x}}_{\mathrm{B}}\stackrel{{\mathrm{k}}_{c3}}{\to }{\mathrm{x}}_{\mathrm{A}}+{\mathrm{E}}_1\end{array}\\ \hfill \begin{array}{c}\hfill {\mathrm{x}}_{\mathrm{C}}+{\mathrm{E}}_2\underset{{\mathrm{k}}_{b4}}{\overset{{\mathrm{k}}_{f4}}{\rightleftharpoons }}{\mathrm{E}}_2{\mathrm{x}}_{\mathrm{C}}\stackrel{{\mathrm{k}}_{c4}}{\to }{\mathrm{x}}_{\mathrm{B}}+{\mathrm{E}}_2\end{array}\end{array}$$

in which for each reaction i, the kinetic constants $k_{fi}$, $k_{bi}$ and $k_{ci}$ represent the forward, backward and catalytic constants, respectively. Under the standard quasi-steady-state assumption (sQSSA) for the enzyme–substrate complex concentration, the transition rates assume the following form [29]:

$$\begin{array}{cc}\hfill {\pi }_{AB}\left(\overrightarrow{x}\right)& =\frac{k_4{v}_2}{k_4{k}_2+k_4{x}_B+k_2{x}_C}{\pi }_{BA}\left(\overrightarrow{x}\right)=\frac{k_3{v}_1}{k_1{k}_3+k_1{x}_B+k_3{x}_A}\hfill \\ \hfill {\pi }_{CB}\left(\overrightarrow{x}\right)& =\frac{k_1{v}_3}{k_1{k}_3+k_1{x}_B+k_3{x}_A}{\pi }_{BC}\left(\overrightarrow{x}\right)=\frac{k_2{v}_4}{k_2{k}_4+k_4{x}_B+k_2{x}_C}\hfill \end{array}$$

in which $k_i=(k_{bi}+k_{ci})/k_{fi}$ is the MM constant and $v_i$ represents the maximal rate velocity associated to reaction i.

We note that the literature proposes alternative methods to study the dynamics of the dual PdPC, different from the sQSSA [30], and our RWA can be applied to any kind of chosen reaction kinetics, since all of them depend on densities. However, for the purpose of this paper, which is to apply the RWA to a real case, we selected the sQSSA.

The complexity of the model is reduced by assuming $k_1=k_4=0.1$, $k_2=k_3=1$, $v_1=v_4=1$ and $v_2=v_3$. Therefore, $v_2$ is the control parameter which governs the bifurcation phenomena, which, with the given parameters occurs at $v_2=2.5$. By computing the critical states, one obtains a solution ${\overrightarrow{x}}_1^{\ast }$ of the form:

$${\overrightarrow{x}}_1^{\ast }=\left(\frac{k_1{v}_2}{k_2{v}_1+2k_1{v}_2},\frac{k_2{v}_1}{k_2{v}_1+2k_1{v}_2},\frac{k_1{v}_2}{k_2{v}_1+2k_1{v}_2}\right)$$

(20)

which exists independently of the value $v_2$. Notice that this solution automatically satisfies the constraints on the density $0\le x_{1,(A,B,C)}^{\ast }\le 1$, $|{\overrightarrow{x}}_1^{\ast }|=1$. Moreover, if the following conditions on $v_2$, which come from the existence of a stationary real solution of Equation (15) and the requirements for ${\overrightarrow{x}}^{\ast }$ to be a probability distribution, are satisfied simultaneously:

$$\begin{array}{cc}\hfill {[v_2-v_1(1+k_2)]}^2& \ge {\left[2k_1{v}_2\right]}^2\hfill \\ \hfill v_2& \ge v_1\hfill \\ \hfill v_2& \ge v_1(1+k_2),\hfill \end{array}$$

(21)

we have two additional and symmetric critical states in the form:

$$\begin{array}{c}\hfill {\overrightarrow{x}}_2^{\ast }=\left(x_{+}^s,\frac{k_2{v}_1}{v_2-v_1},x_{-}^s\right)\end{array}$$

(22)

$$\begin{array}{c}\hfill {\overrightarrow{x}}_3^{\ast }=\left(x_{-}^s,\frac{k_2{v}_1}{v_2-v_1},x_{+}^s\right)\end{array}$$

(23)

in which $x_{\pm }^s$ are the roots of the following second-order equation:

$${\left(x_{\pm }^s\right)}^2-\frac{x_{\pm }^s}{v_2-v_1}(v_2-v_1(1+k_2))+{\left(\frac{k_1{v}_2}{v_2-v_1}\right)}^2=0$$

Therefore, if the conditions in Equation (21) are met, then we have a bistable system, with two stable solutions ${\overrightarrow{x}}_2^{\ast }$, ${\overrightarrow{x}}_3^{\ast }$ and an unstable critical point ${\overrightarrow{x}}_1^{\ast }$. Otherwise, the latter is the unique stable point, and the system is monostable (with the chosen parameters, this occurs when $v_2<2.5$).

5. Results

We describe here the comparison between the RWA and the theoretical relations from Section 2 or the other approximation methods commonly used in the literature. In particular, we use the toy model to observe in action the discussion on RWA of Section 2 and the dual PdPC to prove its utility in a typical research situation compared to currently used methods.

5.1. Toy Model

A direct calculation gives $\partial p_i/\partial x_j\propto g\left(\alpha \right)$, where g is a function independent of N, monotonically increasing with respect to $\alpha$ (see Appendix F), which is the control parameter. We can check the validity of the estimate (11) by computing the RWA ${\ell }^1$-error with respect to the numerical solution of the ME associated to Equation (15). The results are plotted in Figure 2 (right) for different values of the parameter $\alpha$. We observe that for values $\alpha \le 0.6$, the ${\ell }^1$-error of the RWA is independent of the particle number, and it is proportional to the $\alpha$ value. When $\alpha =0.8$, we are near the bifurcation condition, and the error becomes sensitive to the particle number, increasing with the number of particles, since the perturbation approach starts being less accurate. However, at $\alpha =0.9$, when the distribution $\rho \left(\overrightarrow{n}\right)$ is bimodal (Figure 2), the existence of local peaks may reduce the ${\ell }^1$-error, but one has no warranty that spurious critical points exist (i.e., critical points for the distribution that are not solutions of (7)).

In Appendix F, we show that the error on the variance is also independent of N and increases quickly close to bifurcation, starting having the same dependence on N as the ${\ell }^1$-error.

Finally, at the bifurcation point, we also observe a change in the spectral properties of the Laplacian: this is shown in Figure A1c, where we plot the relaxation rate of the numerical solution of the ME associated to Equation (15) for the parameter values $\alpha =0.4$ (before the bifurcation) and $\alpha =0.9$ (after the bifurcation) at $N=200$ particles.

The change in the relaxation process depends on the Fiedler’s eigenvalue, as we discussed in Section 2: before the bifurcation, we have an explicit exponential relaxation dominated by the Fiedler’s eigenvalue that is far from zero; after the bifurcation, when the distribution is bimodal, the relaxation process shows different exponential slopes because the Fiedler’s eigenvalue (and possibly also other successive eigenvalues) is very close to zero, so the relaxation time is much longer.

5.2. Dual Phospho/Dephosphorylation Cycles

The dual PdPC show a more complex behavior that is more similar to what one can encounter in real case scenarios. This is the reason why we focus here on the comparison between the RWA and other common approximation methods usually applied in the literature. Those alternatives are: the SSE, probably the most used, and the standard multinomial solution of the ME, which is obtained by linearizing the ME and considering only the zero-order expansion of the rates around the critical point ${\overrightarrow{x}}^{\ast }$ and then using directly the linear solution (3). In Figure 3, we plot the error of each approximation method with respect to the direct numerical integration of the ME through the RK4(5) algorithm (Section 3). Here, we decided to use as error metric the Jensen–Shannon (JS) divergence [31,32]. It is a measure based on the entropy of the distribution, and it is symmetric and normalized to 1 (with ${log}_2$), giving then an absolute scale of comparison between the information content of the different approximations. The same plot, but with the ${\ell }^1$-norm, is given in Appendix F, showing the same scaling with respect to N, but with the drawback that is less accurate for the comparison, because it is more sensitive to the discretization of the space and it is not absolute. We note that our theoretical results of Section 2 are rigorous only with respect to the ${\ell }_1$-error. Nonetheless, since the JS divergence is based on the product of a probability distribution with the logarithm of a ratio between probabilities (which is of order 1 in N), and it does not involve exponentiation of $\rho$, it is expected to behave with N in the same way as the ${\ell }^1$-error.

Our results show two main things. First of all, as in Figure 2, the error of the RWA (both for the JS and for the ${\ell }^1$-norm, as shown in Appendix F) does not depend much on N, except at very low N, where stochastic fluctuations are important. This confirms the analytical result (10), because, by direct calculation, the derivatives $\frac{\partial p_i}{\partial x_j}$ are, in this model also independent of N, even if it does not have a simple proportionality with the control parameter $v_2$. Close to the bifurcation or after the bifurcation (when the system is bistable, plots (b) and (c) of Figure 3), some spurious dependence on N seems to appear, but especially for the JS measure, the error is still reasonably constant with N, suggesting that the RWA can be a good approximation even during and after the bifurcation. Second, the RWA, in addition to always performing significantly better than the standard multinomial approximation, is comparable to or better than the SSE, again even close to or after the bifurcation. Together, these numerical results suggest that the RWA can be reliable even when the mathematical conditions given in Section 2 are not met, and it is likely more general that what we proved. In this respect, we should note that after or close to the bifurcation, even the SSE has some issues of applicability. Indeed, the SSE (whose distribution is shown in Figure 4 for the monostable regime and Figure 5 for the bistable regime) in this case is constructed artificially, centering the two symmetric Gaussian distributions on the critical points ${\overrightarrow{x}}_2^{\ast }$ and ${\overrightarrow{x}}_3^{\ast }$. This works well for the parameters of Figure 5, but in general, when the two Gaussians are partially overlapped, this can lead to incorrect distributions. The issue would be even more problematic in systems with more than two stable states. This is a consequence of the fact that the SSE is a local approximation that only works in a neighborhood of ${\overrightarrow{x}}^{\ast }$, while the RWA is global.

In order to give some measures of the error on the whole approximated distribution, we can state that at the mesoscale for $N=200$ and $v_2=1.82$, the RWA makes a JS error of about 32%, while the SSE is 25%; at $v_2=2.47$, close to bifurcation, the RWA has an error of 29%, while the SSE is 50% and at $v_2=3.04$, in the bistable regime, the RWA has an error of 27%, while the SSE is 29%. These errors could seem high in general, but it has to be considered that these are errors on the information contained in the whole distribution, even considering the tails, which are usually neglected.

A visual picture of the results is given in Figure 4 and Figure 5, for the monostable and the bistable regime respectively, while the distributions close to criticality are shown in Figure A3. The numerical solution is shown in plot (a) of Figure 4 and Figure 5. We can observe that the SSE (plots (c) of Figure 4 and Figure 5) has good local performances around the critical point but fails in capturing the tails, which are better approximated by the global RWA (plot (b) of Figure 4 and Figure 5). In Figure 5, we notice that if the mean of the Gaussian is close to the boundary the SSE is poorer, in that the Gaussians are cut, resulting in a distortion of the final probability. This is likely why the SSE performs worse than the RWA close to the bifurcation and after, since the numerical distribution flattens and reaches the boundaries of the state space. This also explains why the SSE does not decrease with N in plots (b) and (c) of Figure 3 as expected, since it is a $N\to \infty$ approximation. On the other hand, the multinomial solution (plots (d) of Figure 4 and Figure 5) is always much worse than the other two, being able to capture only a very narrow neighborhood around the critical point. We remark that for all of them, the mode was the correct one, i.e., $x^{\ast }$, so the differences between the distributions were given by larger order moments.

The RWA has also the advantage that, being global, it allows an estimate of the transition rate between the two states, because it gives the energy barrier between the two stable states, which is in turn proportional to the Fiedler’s eigenvalue. Approximating the energy barrier is useful for the application of Kramers theory of transition rates.

Finally, we note that the numerical RK solution was compared with the solution obtained by means of the Gillespie algorithm [18] to ensure that the numerical integration was arrived at convergence to the stationary state. The results were exactly the same either considering the numerical integration or the Gillespie Monte Carlo simulation as benchmark, and therefore, the RK integrated solution can be considered as the “true” solution of the ME.

6. Discussion

We presented a new way to approximate the stationary distribution of stochastic processes governed by an ME, in general out of equilibrium (since we did not impose detailed balance), that can be mapped on a graph. We called this method the Random Walk Approximation, since it is based on the properties of the Laplacian matrix of random walk dynamics on graphs, which has been already successfully applied for instance for graph clustering [33,34]. Here, Laplacian theory on graphs is applied to justify the genesis of the RWA and in general to understand its range of applicability. The goal of the approximation is to give a stationary solution at the mesoscale, where stochasticity is important and the system is not at the thermodynamic limit ($N\to \infty$).

Summarizing, the essence of the RWA is to transform the complex problem of solving a non-linear ME (for the linear ME the analytical result and the RWA coincide) into an eigenvalue problem. Indeed, once the state-dependent rate matrix is given, all that is needed is to compute the null eigenvector of the Laplacian matrix associated to the process. We emphasize that this needs to be performed only once for each model, since the eigenvector will be parametrically dependent, and changing the parameters value can be easily achieved by replacement, while this is not the case for the numerical solutions of the ME. Therefore, the RWA can be preferable to the other techniques if one wants to make a systematic parametric study, exploring a large set of parameter values. Then, the eigenvector can easily be injected in formula (8) and computed for every point of the state space with any standard computational software, giving the stationary probability distribution.

Notably, the error of the RWA is related to the norm of the Jacobian matrix associated to the transition rates as a function of the local density. Therefore, the RWA is valid when we have an adiabatic variation of the transition rates in the state space. The RWA procedure can be applied in principle for any possible number of system states M, regardless of the number of particles N (even though for very large N, a deterministic approach may be often preferable), but both the complexity of the eigenvalue problem (which is of order $O\left(M^3\right)$ [35]) and the product in Equation (8) become numerically prohibitive for large M. However, it has to be said that in those cases, also the other methods, including the numerical ones, are computationally very expensive, and an ME approach is often impossible.

Although the RWA is computationally faster than both direct RK integration and the Gillespie algorithm, if one needs a very accurate prediction of the stationary distribution, those numerical methods are still in most cases preferred. The main added value of our approximation is to have a simple global analytical form that can be used for further computations on the stochastic system, such as entropy production, or to explore the importance of rare mesoscale states contained on the tail of the distribution. This is something that a numerical solution does not easily allow and something that, even in the simpler monostable regime, with the SSE is quite complicated to achieve, involving many steps, both numerical and analytical, as described in Section 3.1. Moreover, the RWA does not decrease its accuracy when the probability distribution is close to the boundaries, since all the constraints are automatically included in the procedure.

We also showed that the RWA has the advantage of including the bifurcation in itself, meaning that the multi-stability does not need to be added artificially as for the SSE, and therefore, an a priori stable solution of the deterministic dynamics is not needed. Nevertheless, as a drawback, the RWA is mathematically reliable only before the bifurcation, when one stable solution exists. Although this may be an issue in more complex models, at least for the systems that we studied in this paper, the RWA performs well even during and after the bifurcation, setting the foundation to further studies that may make our results more general, at least under some conditions.

Finally, with respect to the SSE, our approximation is simpler, global and has comparable, or better, accuracy, when the thermodynamic limit is not verified and the number of particles is not very large (of the order of Avogadro number).