Accelerated Diffusion-Based Sampling by the Non-Reversible Dynamics with Skew-Symmetric Matrices

Abstract

Langevin dynamics (LD) has been extensively studied theoretically and practically as a basic sampling technique. Recently, the incorporation of non-reversible dynamics into LD is attracting attention because it accelerates the mixing speed of LD. Popular choices for non-reversible dynamics include underdamped Langevin dynamics (ULD), which uses second-order dynamics and perturbations with skew-symmetric matrices. Although ULD has been widely used in practice, the application of skew acceleration is limited although it is expected to show superior performance theoretically. Current work lacks a theoretical understanding of issues that are important to practitioners, including the selection criteria for skew-symmetric matrices, quantitative evaluations of acceleration, and the large memory cost of storing skew matrices. In this study, we theoretically and numerically clarify these problems by analyzing acceleration focusing on how the skew-symmetric matrix perturbs the Hessian matrix of potential functions. We also present a practical algorithm that accelerates the standard LD and ULD, which uses novel memory-efficient skew-symmetric matrices under parallel-chain Monte Carlo settings.

1. Introduction

Sampling is one of the most widely used techniques for the approximation of posterior distribution in Bayesian inference [1]. Markov Chain Monte Carlo (MCMC) is widely used to obtain samples. In MCMC, Langevin dynamics (LD) is a popular choice for sampling from high-dimensional distributions. Each sample in LD moves toward a gradient direction with added Gaussian noise. LD efficiently explore around a mode of a target distribution using the gradient information without being trapped by local minima thanks to added Gaussian noise. Many previous studies theoretically and numerically proved LD’s superior performance [2,3,4,5]. Since non-reversible dynamics generally improves mixing performance [6,7], research on introducing non-reversible dynamics to LD for better sampling performance is attracting attention [8].

There are two widely known non-reversible dynamics for LD. One is underdamped Langevin dynamics (ULD) [9], which uses second-order dynamics. The other introduces perturbation, which consists of multiplying the skew-symmetric matrix by a gradient [8]. Here, we refer to the matrix as skew matrices for simplicity and this perturbation technique as skew acceleration. Much research has been done on ULD theoretically [9,10,11] and ULD is widely used in practice, which is also known as stochastic gradient Hamilton Monte Carlo [12]. In contrast, the application of the skew acceleration for standard Bayesian models is quite limited even though it is expected to show superior performance theoretically [8].

For example, skew acceleration has been analyzed focusing on sampling from Gaussian distributions [13,14,15,16,17], although assuming Gaussian distributions in Bayesian models is restrictive in practice. A recent study [8] theoretically showed that skew acceleration accelerates the dynamics around the local minima and saddle points for non-convex functions. Another work [18] clarified that the skew acceleration theoretically and numerically improves mixing speed when used as interactions between chains in parallel sampling schemes for non-convex Bayesian models.

Compared to ULD, what seems to be lacking for skew acceleration is a theoretical understanding of issues that are important to practitioners. The most significant problem is that no theory exists for selecting skew matrices. In existing studies, introducing a skew matrix into LD results in equal or faster convergence, denoting that a bad choice of skew matrix results in no acceleration. Thus, choosing appropriate skew matrices is critical. Furthermore, although ULD’s acceleration has been analyzed quantitatively, existing studies have only analyzed skew acceleration qualitatively. Thus, it is difficult to justify the usefulness of skew acceleration in practice compared to ULD. Another issue is that introducing skew matrices requires a vast memory cost in many practical Bayesian models.

The purpose of this study is to solve these problems from theoretical and numerical viewpoints and establish a practical algorithm for skew acceleration. The following are the two major contributions of this work.

Our contribution 1: We present a convergence analysis of skew acceleration for standard Bayesian model settings, including non-convex potential functions using Poincaré constants [19]. The major advantage of Poincaré constants is that we can analyze skew acceleration through a Hessian matrix and its eigenvalues and develop a practical theory about the selection of J and the quantitative assessment of skew acceleration.

Furthermore, we propose skew acceleration for ULD and present convergence analysis for the first time. Since ULD shows faster convergence than LD, combining skew acceleration with ULD is promising.

Our contribution 2: We develop a practical skew accelerated sampling algorithm for a parallel sampling setting with novel memory-efficient skew matrices. Since a naive implementation of skew acceleration requires a large memory cost to store skew matrices, memory-efficiency is critical in practice. We also present a non-asymptotic theoretical analysis for our algorithm in both LD and ULD settings under a stochastic gradient and Euler discretization. We clarify that introducing skew matrices accelerates the convergence of continuous dynamics, although it increases the discretization and stochastic gradient error. Then to the best of our knowledge, we propose the first algorithm that adaptively controls this trade-off using the empirical distribution of the parallel sampling scheme.

Finally, we verify our algorithm and theory in practical Bayesian problems and compare it with other sampling methods.

Notations: $I_d$ denotes a $d\times d$ identity matrix. Capital letters such as X represent random variables, and lowercase letters such as x represent non-random real values. ·, $\parallel ·\parallel$ and $|·|$ denote Euclidean inner products, distances and absolute values.

2. Preliminaries

In this section, we briefly introduce the basic settings of LD and non-reversible dynamics for the posterior distribution sampling in Bayesian inference.

2.1. LD and Stochastic Gradient LD

First, we introduce the notations and the basic settings of LD and stochastic gradient LD (SGLD), which is a practical extension of LD. Here $z_i$ denotes a data point in space $\mathbb{Z}$, $\left|\mathcal{Z}\right|$ denotes the total number of data points, and $x\in {\mathbb{R}}^d$ corresponds to the parameters of a given model, which we want to sample. Our goal is to sample from the target distribution with density $d\pi \left(x\right)\propto e^{-\beta U\left(x\right)}dx$, where potential function $U\left(x\right)$ is the summation of $u:{\mathbb{R}}^d\times \mathbb{Z}\to \mathbb{R}$, i.e., ${\displaystyle U\left(x\right)=\frac{1}{\left|\mathcal{Z}\right|}\sum _{i=1}^{\left|\mathcal{Z}\right|}u(x,z_i)}$. Function $u(·,·)$ is continuous and non-convex. The explicit assumptions made for it are discussed in Section 3.1. The SGLD algorithm [2,3] is given as a recursion:

$$\begin{array}{c}\hfill X_{k+1}=X_k-h\nabla \widehat{U}\left(X_k\right)+\sqrt{2h{\beta }^{-1}}{ϵ}_k,\end{array}$$

(1)

where $h\in {\mathbb{R}}^{+}$ is a step size, ${ϵ}_k\in {\mathbb{R}}^d$ is a standard Gaussian random vector, $\beta$ is a temperature parameter of $\pi$, and $\nabla \widehat{U}\left(X_k\right)$ is a conditionally unbiased estimator of true gradient $\nabla U\left(X_k\right)$. This unbiased estimate of the true gradient is suitable for large-scale data set since we can use not the full gradient, but a stochastic version obtained through a randomly chosen subset of data at each time step. This means that we can reduce the computational cost to calculate the gradient at each time step.

The discrete time Markov process in Equation (1) is the discretization of the continuous-time LD [2]:

$$\begin{array}{c}\hfill d{X}_t=-\nabla U\left(X_t\right)dt+\sqrt{2{\beta }^{-1}}d{w}_t,\end{array}$$

(2)

where $w_t$ denotes the standard Brownian motion in ${\mathbb{R}}^d$. The stationary measure of Equation (2) is $d\pi \left(x\right)\propto e^{-\beta U\left(x\right)}dx$.

2.2. Poincaré Inequality and Convergence Speed

In sampling, we are interested in the convergence speed to the stationary measure. The speed is often characterized by the the generator associated with Equation (2) and defined as:

$$\begin{array}{cc}\hfill \mathcal{L}f\left(X_t\right):& =\underset{s\to 0^{+}}{lim}\frac{\mathbb{E}\left(f\left(X_{t+s}\right)\right|X_t)-f\left(X_t\right)}{s}\hfill \\ \hfill & =\left(-\nabla U\left(X_t\right)·\nabla +{\beta }^{-1}\Delta \right)f\left(X_t\right),\hfill \end{array}$$

(3)

where $\Delta$ denotes a standard Laplacian on ${\mathbb{R}}^d$ and $f\in \mathcal{D}\left(\mathcal{L}\right)$ and $\mathcal{D}\left(\mathcal{L}\right)\subset {\mathrm{L}}^2\left(\pi \right)$ denote the $\mathcal{L}$ domain. This $-\mathcal{L}$ is a self-adjoint operator, which has only discrete spectrums (eigenvalues). $\pi$ with $\mathcal{L}$ has a spectral gap if the smallest eigenvalue of $-\mathcal{L}$ (other than 0) is positive. We refer to it as ${\rho }_0(>0)$. This spectral gap is closely related to Poincaré inequality. Internal energy is defined:

$$\begin{array}{c}\hfill \mathcal{E}\left(f\right):=-{\int }_{{\mathbb{R}}^d}f\mathcal{L}fd\pi .\end{array}$$

(4)

Please note that $\mathcal{E}\left(f\right)>0$ is satisfied. Then $\pi$ with $\mathcal{L}$ satisfies the Poincaré inequality with constant c, if for any $f\in \mathcal{D}\left(\mathcal{L}\right)$, $\pi$ with $\mathcal{L}$ satisfies:

$$\begin{array}{c}\hfill \int f^{2}d\pi -{\left(\int fd\pi \right)}^2\le c\mathcal{E}\left(f\right).\end{array}$$

(5)

The spectral gap characterizes this constant $c\le \frac{1}{{\rho }_0}$, which holds (see Appendix A.2 for details). We refer to best constant c as the Poincaré constant [19]. For notational simplicity, we define $m_0:=\frac{1}{c}$ and refer to this $m_0$ as the Poincaré constant.

In sampling, crucially, Poincaré inequality dominates the convergence speed in ${\chi }^2$ divergence:

$$\begin{array}{c}\hfill \int {\left(\frac{d{\mu }_t}{d\pi }-1\right)}^{2}d\pi :={\chi }^2({\mu }_t\parallel \pi )\le e^{-\frac{2m_0}{\beta }t}{\chi }^2({\mu }_0\parallel \pi ),\end{array}$$

(6)

where ${\mu }_t$ denotes the measure at time t induced by Equation (2) and ${\mu }_0$ is the initial measure (see Appendix A.3 for details). Thus, the larger Poincaré constant $m_0$ is, the faster convergence we have.

2.3. Non-Reversible Dynamics

In this section, we introduce the non-reversible dynamics. $\pi$ with $\mathcal{L}$ is reversible if for any test function $f,g\in \mathcal{D}\left(\mathcal{L}\right)$, $\pi$ with $\mathcal{L}$ satisfies

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^d}f\mathcal{L}gd\pi ={\int }_{{\mathbb{R}}^d}g\mathcal{L}fd\pi .\end{array}$$

(7)

If this is not satisfied, $\pi$ with $\mathcal{L}$ is non-reversible [19].

We introduce two non-reversible dynamics for LD. The first is ULD, which is given as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & d{X}_t={\Sigma }^{-1}{V}_{t}dt,\hfill \\ \hfill & d{V}_t=-\nabla U\left(X_t\right)dt-\gamma {\Sigma }^{-1}{V}_{t}dt+\sqrt{2\gamma {\beta }^{-1}}d{w}_t,\hfill \end{array}\end{array}$$

(8)

where $V\in {\mathbb{R}}^d$ is an auxiliary random variable, $\gamma \in \mathbb{R}$ is a positive constant, and $\Sigma$ is the variance of the stationary distribution of auxiliary random variable V. The stationary distribution is $\tilde{\pi }:=\pi \otimes \mathcal{N}(0,\Sigma )\propto e^{-\beta U\left(x\right)-\frac{1}{2}{\Sigma }^{-1}{\parallel v\parallel }^2}$, where $\mathcal{N}$ denotes a Gaussian distribution. The superior performance of ULD compared with LD has been studied rigorously [9,10,11]. ULD’s convergence speed is also characterized by the Poincaré constant [20]. In practice, we use discretization and the stochastic gradient for ULD, which is called the stochastic gradient Hamilton Monte Carlo (SGHMC) [10]. The second non-reversible dynamics is the skew acceleration given as

$$\begin{array}{c}\hfill d{X}_t=-(I+\alpha J)\nabla U\left(X_t\right)dt+\sqrt{2{\beta }^{-1}}d{w}_t,\end{array}$$

(9)

where J is a real value skew matrix and $\alpha \in {\mathbb{R}}^{+}$ is a positive constant. We call this dynamics S-LD. The stationary distribution of S-LD is still $\pi$, and S-LD shows faster convergence and smaller asymptotic variance [13,14,15,18].

3. Theoretical Analysis of Skew Acceleration

In this section, we present a theoretical analysis of skew acceleration in LD and ULD in standard Bayesian settings. We analyze acceleration through the Poincaré constant and connect it with the eigenvalues of the Hessian matrix, which allows us to obtain a practical criterion to choose skew matrices and quantitatively evaluate acceleration. We focus on a setting where a continuous SDE and a full gradient of the potential function is used in this section. The discretized SDE and stochastic gradient are discussed in Section 4.

3.1. Acceleration Characterization by the Poincaré Constant

First, we introduce the same four assumptions as a previous work [2], which showed the existence of the Poincaré constant about $m_0$ for LD (see Appendix C for details).

Assumption 1. 

(Upper bound of the potential function at the origin) Function u takes nonnegative real values and is twice continuously differentiable on ${\mathbb{R}}^d$, and constants A and B exist such that for all $z\in \mathbb{Z}$,

$$\begin{array}{c}\hfill \left|u\right(0,z\left)\right|\le A,\parallel \nabla u(0,z)\parallel \le B.\end{array}$$

(10)

Assumption 2. 

(Smoothness) Function u has Lipschitz continuous gradients; for all $z\in \mathbb{Z}$, positive constant M exists for all $x,y\in {\mathbb{R}}^d$,

$$\begin{array}{c}\hfill \parallel \nabla u(x,z)-\nabla u(y,z)\parallel \le M\parallel x-y\parallel .\end{array}$$

(11)

Assumption 3. 

(Dissipative condition) Function u satisfies the (m,b)-dissipative condition for all $z\in \mathbb{Z}$; for all $x\in {\mathbb{R}}^d$, $m>0$ and $b\ge 0$ exist such that

$$\begin{array}{c}\hfill -x·\nabla u(x,z)\le -{m\parallel x\parallel }^2+b.\end{array}$$

(12)

Assumption 4. 

(Initial condition) Initial probability distribution ${\mu }_0$ of $X_0$ has a bounded and strictly positive density $p_0$, and for all $x\in {\mathbb{R}}^d$,

$$\begin{array}{c}\hfill {\kappa }_0:=log{\int }_{{\mathbb{R}}^d}{e}^{{\parallel x\parallel }^2}{p}_0\left(x\right)dx<\infty .\end{array}$$

(13)

Please note that these assumptions allow us to consider the non-convex potential functions, which are common in practical Bayesian models. Furthermore, we make the following assumption about J.

Assumption 5. 

The operator norm of J is bounded:

$$\begin{array}{c}\hfill {\parallel J\parallel }_2\le 1.\end{array}$$

(14)

This means that the largest singular value of J is below 1.

Under these assumptions, we present the convergence behavior of skew acceleration using the Poincaré constant. First, we present the following S-LD result.

Theorem 1.

Under Assumptions 1–5, the S-LD of Equation (9) has exponential convergence,

$$\begin{array}{c}\hfill {\chi }^2({\mu }_t^{\alpha }\parallel \pi )\le e^{-\frac{2m\left(\alpha \right)}{\beta }t}{\chi }^2({\mu }_0\parallel \pi ),\end{array}$$

(15)

where ${\mu }_t^{\alpha }$ is the measure at time t induced by S-LD and $m\left(\alpha \right)$ is the Poincaré constant of S-LD defined by its generator

$$\begin{array}{c}\hfill {\mathcal{L}}_{\alpha }f\left(x\right):=\left(-(I+\alpha J)\nabla U\left(x\right)·\nabla +{\beta }^{-1}\Delta \right)f\left(x\right).\end{array}$$

(16)

Furthermore, $m\left(\alpha \right)$ satisfies $m\left(\alpha \right)\ge m_0$.

The proof is shown in Appendix C. This theorem states that introducing the skew matrices accelerates the convergence of LD by improving the convergence rate from $m_0$ to $m\left(\alpha \right)$. Although [18] obtained a similar result, we used the Poincaré constant and derived an explicit criterion when $m\left(\alpha \right)=m_0$ holds, as we discuss below.

Next, we also introduce skew acceleration in ULD. Since ULD shows faster convergence than LD in standard Bayesian settings [10,11], it is promising to combine skew acceleration with ULD to obtain a more efficient sampling algorithm. For that purpose, we propose the following SDE:

$$\begin{array}{cc}\hfill & d{X}_t={\Sigma }^{-1}{V}_{t}dt+{\alpha }_1{J}_1\nabla U\left(X_t\right)dt,\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & d{V}_t=-\nabla U\left(X_t\right)dt-\gamma ({\Sigma }^{-1}+{\alpha }_2{J}_2)V_{t}dt+\sqrt{2\gamma {\beta }^{-1}}d{w}_t,\hfill \end{array}$$

(18)

where $J_1$ and $J_2$ are real value skew matrices and ${\alpha }_1$ and ${\alpha }_2$ are positive constants. We assume that $J_1$ and $J_2$ satisfy Assumption 5. We refer to this method as skew underdamped Langevin dynamics (S-ULD) whose stationary distribution is $\tilde{\pi }=\pi \otimes \mathcal{N}(0,\Sigma )\propto e^{-\beta U\left(x\right)-\frac{1}{2}{\Sigma }^{-1}{\parallel v\parallel }^2}$. See Appendix B for details, which include discussions on other combinations of skew matrices. As for S-ULD, we need an additional assumption about the initial condition of $V_0$:

Assumption 6. 

(Initial condition) Initial probability distribution ${\mu }_0(x,v)$ of $(X_0,V_0)$ has a bounded and strictly positive density $p_0$ that satisfies,

$$\begin{array}{c}\hfill {\kappa }_0:=log{\int }_{{\mathbb{R}}^{2d}}{e}^{{\parallel x\parallel }^2+{\parallel v\parallel }^2}{p}_0\left((x,v)\right)dxdv<\infty .\end{array}$$

(19)

We then provide the following convergence theorem that resembles S-LD.

Theorem 2.

Under Assumptions 1–3, 5, 6, S-ULD has exponential convergence in ${\chi }^2$ divergence and its convergence rate is also characterized by $m\left(\alpha \right)$ as defined in Theorem 1. S-ULD’s convergence equals or exceeds ULD, of which convergence rate is characterized by $m_0$.

See Appendix C.2 for details. From these theorems, we confirmed that skew acceleration is effective in both S-LD and S-ULD, and the convergence speed is characterized by Poincaré constant $m\left(\alpha \right)$ defined by Equation (16).

3.2. Skew Acceleration from the Hessian Matrix

Our goal is to clarify what choices of J induce $m\left(\alpha \right)>m_0$, which leads to acceleration. Therefore, we discuss how Poincaré constant $m\left(\alpha \right)$ is connected to the eigenvalues and eigenvectors of the perturbed Hessian matrix $(I+\alpha J){\nabla }^{2}U\left(x\right)$. Next, we introduce the notations. We express the Hessian of $U\left(x\right)$ as $H\left(x\right)$ and the perturbed Hessian matrix as $H^{\prime }\left(x\right):=(I+\alpha J)H\left(x\right)$. Please note that H is a real symmetric matrix, which has real eigenvalues and diagonalizable. On the other hand, since $H^{\prime }$ is not symmetric, it has complex eigenvalues, although diagonalization is not assured (see Appendix E). We express pairs of eigenvectors and eigenvalues of $H^{\prime }\left(x\right)$ as ${\left\{(v_i^{\alpha }\left(x\right),{\lambda }_i^{\alpha }\left(x\right))\right\}}_{i=1}^d$, which are ordered as $\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right))\le \cdots \le \mathrm{Re}\left({\lambda }_d^{\alpha }\left(x\right)\right)$. Here, $\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ expresses the real part of complex value ${\lambda }_1^{\alpha }$ and $\mathrm{Im}$ denotes the imaginary part. We express those of $H\left(x\right)$ as ${\left\{(v_i^0\left(x\right),{\lambda }_i^0\left(x\right))\right\}}_{i=1}^d$ and order them as ${\lambda }_1^0\left(x\right)\le \cdots \le {\lambda }_d^0\left(x\right)$.

3.2.1. Strongly Convex Potential Function

Assume that U is an m-strongly convex function, where for all $x\in {\mathbb{R}}^d$, $m\le {\lambda }_1^0\left(x\right)$ holds. Poincaré constant $m_0$ of LD satisfies $m_0=m$ [19]. For the skew acceleration, since Poincaré constant satisfies $m\left(\alpha \right)=m^{\prime }\left(\alpha \right)$, where $m^{\prime }\left(\alpha \right)$ is the best constant that satisfies, for all x, $m^{\prime }\left(\alpha \right)\le \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ (see Appendix D.1). Therefore, studying the Poincaré constant is equivalent to studying the smallest (real part of the) eigenvalue of the Hessian matrix. Thus, the relation between ${\lambda }_1^0\left(x\right)$ and $\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ must be studied. The following theorem describes how the skew matrices change the smallest eigenvalue.

Theorem 3.

For all $x\in {\mathbb{R}}^d$, the real parts of the eigenvalues of $H^{\prime }$ satisfy

$$\begin{array}{c}\hfill m\le {\lambda }_1^0\left(x\right)\le \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)\le \cdots \le \mathrm{Re}\left({\lambda }_d^{\alpha }\left(x\right)\right)\le {\lambda }_d^0\left(x\right).\end{array}$$

(20)

The condition of ${\lambda }_1^0\left(x\right)=\mathrm{Re}\left({\lambda }_1^{\alpha }\right(x\left)\right)$ is shown in Remark 1.

Remark 1.

Denote the set of the eigenvectors of eigenvalue ${\lambda }_1^0\left(x\right)$ as $V_1^0$. If $V_1^0=\left\{v\right\}$ and $Jv=0$, then ${\lambda }_1^0\left(x\right)=\mathrm{Re}\left({\lambda }_1^{\alpha }\right(x\left)\right)$ holds. If the cardinality of set $V_1^0$ is larger than 1, and vectors $v,v^{\prime }\in V_1^0$ exist, such that ${\lambda }_1^0\alpha Jv=\left(\mathrm{Im}\left({\lambda }_1^{\alpha }\right)\right)v^{\prime }$ and ${\lambda }_1^0\alpha J{v}^{\prime }=-\left(\mathrm{Im}\left({\lambda }_1^{\alpha }\right)\right)v$, then ${\lambda }_1^0\left(x\right)=\mathrm{Re}\left({\lambda }_1^{\alpha }\right(x\left)\right)$ holds.

Refer to Appendix F for the proof. This is an extension of previous work [8,13]. If ${\lambda }_1^0\left(x\right)<\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ is satisfied for all x, we have $m_0<m\left(\alpha \right)$, i.e., acceleration occurs. We discuss how to construct J such that ${\lambda }_1^0\left(x\right)<\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ holds in Section 3.3.

3.2.2. Non-Convex Potential Function

The previous work [21] clarified that the Poincaré constant of the non-convex function is characterized by the negative eigenvalue of the saddle point. As shown in Figure 1, denote $x_1$ as the global minima, and $x_2$ is the local minima which has the second smallest value in $U\left(x\right)$. We express the saddle point with index one, i.e., there is only one negative eigenvalue at the point, between $x_1$ and $x_2$ as $x^{\ast }$. This means that the eigenvalues of $H\left(x^{\ast }\right)$ satisfies ${\lambda }_1^0\left(x^{\ast }\right)<0<{\lambda }_2^0\left(x^{\ast }\right)<\cdots <{\lambda }_d^0\left(x^{\ast }\right)$. Ref. [21] clarified that the saddle point $x^{\ast }$ characterizes the Poincaré constant as

$$\begin{array}{c}\hfill m_0^{-1}\propto \frac{1}{|{\lambda }_1\left(x^{\ast }\right)|}{e}^{\beta (U\left(x^{\ast }\right)-U\left(x_1\right)-U\left(x_2\right))}.\end{array}$$

(21)

When skew matrices are introduced, [8] clarified the following relation:

Theorem 4. 

([8]) ${\lambda }_1^{\alpha }\left(x^{\ast }\right)\le {\lambda }_1^0\left(x^{\ast }\right)<0$ and equality holds only if $J{v}_1^{\alpha }\left(x^{\ast }\right)=0$.

Figure 1. Double-potential example: Poincaré constant is related to the eigenvalue at $x^{\ast }$.

Note ${\lambda }_1^{\alpha }\left(x^{\ast }\right)$ is not a complex number. Thus, the skew acceleration reduces the negative eigenvalue and leads to a larger Poincaré constant (see Appendix D.2) and results in faster convergence.

In conclusion, introducing the skew matrix changes the Hessian’s eigenvalues and increase the Poincaré constant. If ${\lambda }_1^0\left(x\right)\ne \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ is satisfied, this leads to faster convergence for both convex and non-convex potential functions.

3.3. Choosing J

In this section, we present a method for choosing J that leads to ${\lambda }_1^0\left(x\right)\ne \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ to ensure the acceleration based on the equality conditions in Theorems 3 and 4. Combining these theorems, we obtain the following criterion:

Remark 2.

Given a point x, ${\lambda }_1^0\left(x\right)\ne \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ holds if either the following conditions are satisfied: $\left(i\right)$ when $V_1^0=\left\{v\right\}$, $Jv\ne 0$ is satisfied. $\left(ii\right)$ when $|V_1^0|>1$, $Jv\ne 0$ holds for any $v\in V_1^0$, and for any $v,v^{\prime }\in V_1^0$, ${\lambda }_1^0\alpha Jv=\left(\mathrm{Im}\left({\lambda }_1^{\alpha }\right)\right)v^{\prime }$ and ${\lambda }_1^0\alpha J{v}^{\prime }=-\left(\mathrm{Im}\left({\lambda }_1^{\alpha }\right)\right)v$ are not satisfied.

The first condition $\left(i\right)$ is easily satisfied if we choose J such that $\mathrm{Ker}J=\left\{0\right\}$. On the other hand, the second condition $\left(ii\right)$ is difficult to verify since H and its eigenvalues and eigenvectors generally depend on the current position of $X_t$. Instead of evaluating eigenvalues and eigenvectors of H and $H^{\prime }$ directly, we use the random matrix property shown in the next theorem.

Theorem 5.

Suppose the upper triangular entries of J follow a probability distribution that is absolutely continuous with respect to the Lebesgue measure. If $\mathrm{Ker}J=\left\{0\right\}$ is satisfied, then given a point $x\in {\mathbb{R}}^d$, ${\lambda }_1^0\left(x\right)\ne \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ holds with probability 1.

The proof is given in Appendix G.1. From this theorem, we simply generate J from some probability distribution, such as the Gaussian distribution. Then, we check whether $\mathrm{Ker}J=\left\{0\right\}$ holds. If $\mathrm{Ker}J=\left\{0\right\}$ does not hold, we generate a random matrix J again.

The above theorem is valid only at a given evaluation point x. We can extend the above theorem to all the points over the path of the discretized dynamics (see Appendix G.3). With this procedure, we can theoretically ensure that acceleration occurs with probability one for discretized dynamics.

3.4. Qualitative Evaluation of The Acceleration

So far, we have discussed skew acceleration qualitatively but not quantitatively. Although acceleration’s quantitative evaluation is critical for practical purposes, to the best of our knowledge, no existing work has addressed it. In this section, we present a formula that quantitatively assesses skew acceleration by analyzing the eigenvalues of the Hessian matrix.

Theorem 6.

With the identical notation as in Theorem 3, for all x, we have

$$\begin{array}{cc}\hfill & \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)={\lambda }_1^0\left(x\right)+{\alpha }^2\sum _{k=2}^d\frac{{\lambda }_1^0\left(x\right){\lambda }_k^0\left(x\right){\left|v_k^0\left(x\right)J{v}_1^0\left(x\right)\right|}^2}{{\lambda }_k^0\left(x\right)-{\lambda }_1^0\left(x\right)}+\mathcal{O}\left({\alpha }^3\right).\hfill \end{array}$$

(22)

In particular, at saddle point $x^{\ast }$, we have

$$\begin{array}{c}\hfill {\lambda }_1^{\alpha }\left(x^{\ast }\right)={\lambda }_1^0\left(x^{\ast }\right)+{\alpha }^2\sum _{k=2}^d\frac{{\lambda }_1^0\left(x^{\ast }\right){\lambda }_k^0\left(x^{\ast }\right){\left|v_k^0\left(x^{\ast }\right)J{v}_1^0\left(x^{\ast }\right)\right|}^2}{{\lambda }_k^0\left(x^{\ast }\right)-{\lambda }_1^0\left(x^{\ast }\right)}+\mathcal{O}\left({\alpha }^3\right).\end{array}$$

(23)

The proofs are shown in Appendix H. When focusing on Equation (22), if $U\left(x\right)$ is a strongly convex function, since for all $k>1$, ${\lambda }_k\left(x\right)>{\lambda }_1\left(x\right)>0$ holds and the second term in Equation (22) is positive. From this, $\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)>{\lambda }_1^0\left(x\right)$ holds. A similar relation holds for $\mathrm{Re}\left({\lambda }_d^{\alpha }\left(x\right)\right)$. In Equation (23), ${\lambda }_1^{\alpha }\left(x^{\ast }\right)<{\lambda }_1^0\left(x^{\ast }\right)<0$ holds. Thus, the changes of the Poincaré constants are proportional to ${\alpha }^2$. With these formulas, we can quantitatively evaluate the acceleration. We present numerical experiments to confirm our theoretical findings in Section 6.1.

4. Practical Algorithm for Skew Acceleration

In this section, we discuss skew acceleration in more practical settings compared to Section 3. First, we discuss the memory issue for storing J and the discretization of SDE and the stochastic gradient, which are widely used techniques in Bayesian inference. Finally, we present a practical algorithm for skew acceleration.

4.1. Memory Issue of Skew Acceleration and Ensemble Sampling

For d-dimensional Bayesian models, we need $\mathcal{O}\left(d^2\right)$ memory space to store skew matrices Js, and this is difficult for high-dimensional models. Instead of storing J, we can randomly generate Js at each time step following Theorem 5. However, we experimentally confirmed that using different Js at each step does not accelerate the convergence (see Section 6). Thus, we need to use a fixed J during the iterations.

As discussed below, we found that the previously proposed accelerated parallel sampling [18] can be a practical algorithm to resolve this memory issue. In that method, we simultaneously updated N samples of the model’s parameters with correlation. In such a parallel sampling scheme, a correlation exists among multiple Markov chains, it is more efficient than a naive parallel-chain MCMC, where the samples are independent. We express the n-th sample at time t as $X_t^{\left(n\right)}\in {\mathbb{R}}^d$ and the joint state of all samples at time t as $X_t^{\otimes N}:={(X_t^{\left(1\right)},\dots ,X_t^{\left(N\right)})}^{\top }\in {\mathbb{R}}^{dN}$. We express the joint stationary measure as ${\pi }^{\otimes N}:=\pi \otimes \cdots \otimes \pi \left(x^{\otimes N}\right)\propto e^{-\beta {\sum }_{i=1}^{N}U\left(x^{\left(i\right)}\right)}$. We express the sum of the potential function as $U^{\otimes N}:={\sum }_{i=1}^{N}U\left(x^{\left(i\right)}\right)$. We then consider the following dynamics:

$$\begin{array}{cc}\hfill & d{X}_t^{\otimes N}=-(I_{dN}+\alpha J)\nabla U^{\otimes N}\left(X_t^{\otimes N}\right)dt+\sqrt{2{\beta }^{-1}}d{w}_t,\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & \nabla U^{\otimes N}\left(X_t^{\otimes N}\right):={\left(\nabla U\left(X_t^{\left(1\right)}\right),\dots ,\nabla U\left(X_t^{\left(N\right)}\right)\right)}^{\top }.\hfill \end{array}$$

(25)

We call this dynamics skew parallel LD (S-PLD). N-independent parallel LD (PLD) is coupled with the skew matrix. Since each chain in PLD is independent of the other, the Poincaré constant of PLD is also $m_0$. Ref. [18] argued that the Poincaré constant of S-PLD, $m(\alpha ,N)$, satisfies $m(\alpha ,N)\ge m_0$. This means S-PLD shows faster convergence than PLD. As discussed in Section 3.2, these Poincaré constants are characterized by the smallest eigenvalue of the Hessian matrix ${\nabla }^2{U}^{\otimes N}\left(x^{\otimes N}\right)$ and $(I_{dN}+\alpha J){\nabla }^2{U}^{\otimes N}\left(x^{\otimes N}\right)$ where $x^{\otimes N}\in {\mathbb{R}}^{dN}$. We denote these smallest eigenvalues as ${\lambda }_1^0\left(x^{\otimes N}\right)$ and $\mathrm{Re}{\lambda }_1^{\alpha }\left(x^{\otimes N}\right)$. As discussed in Section 3.2, acceleration occurs if ${\lambda }_1^0\left(x^{\otimes N}\right)\ne \mathrm{Re}{\lambda }_1^{\alpha }\left(x^{\otimes N}\right)$ is satisfied.

In [18], they failed to specify the choice of J whose naive construction of J requires $\mathcal{O}\left(d^2{N}^2\right)$ memory cost. To reduce the memory cost, we propose the following skew matrix:

$$\begin{array}{c}\hfill J:=J_0\otimes I_d,\end{array}$$

(26)

where $J_0$ is a $N\times N$ skew matrix and ⊗ is a Kronecker product. We then have the following lemma:

Lemma 1.

If $J_0$ is generated based on Theorem 5 and $\mathrm{Ker}{J}_0=\left\{0\right\}$ is satisfied, then given a point $x^{\otimes N}$, J does not satisfy the equality condition in Theorems 3,4, which means ${\lambda }_1^0\left(x^{\otimes N}\right)\ne \mathrm{Re}{\lambda }_1^{\alpha }\left(x^{\otimes N}\right)$ with probability 1.

See Appendix G.2 for the proof. Thus, from this lemma, we only need to prepare and store $J_0$, which requires $\mathcal{O}\left(N^2\right)$ memory, which does not depend on d. In practical settings, this is a significant reduction of the memory size since the number of parallel chains is smaller than the dimension of models. Please note that we can ensure the acceleration with this J.

Lemma 2.

Under Assumptions 1–5, assume J satisfies the condition of Lemma 1. Then S-PLD shows

$$\begin{array}{c}\hfill {\chi }^2({\mu }_t^{\alpha ,\otimes N}\parallel {\pi }^{\otimes N})\le e^{-\frac{2m(\alpha ,N)}{\beta }t}{\chi }^2({\mu }_0^{\otimes N}\parallel {\pi }^{\otimes N}),\end{array}$$

(27)

where ${\mu }_t^{\alpha ,\otimes N}$ is the measure at time t induced by S-PLD, and ${\mu }_0^{\otimes N}$ is the initial measure defined as the product measure of ${\mu }_0$.

See Appendix I.1 for the proofs. Thus, combined with Lemma 2, S-PLD converges faster than PLD. We also considered the ensemble version of ULD (parallel ULD (PULD)) and its skew accelerated version:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & d{X}_t^{\otimes N}={\Sigma }^{-1}{V}_t^{\otimes N}dt+{\alpha }_1{J}_1\nabla U^{\otimes N}\left(X_t^{\otimes N}\right)dt,\hfill \\ \hfill & d{V}_t^{\otimes N}=-\nabla U^{\otimes N}\left(X_t^{\otimes N}\right)dt-\gamma ({\Sigma }^{-1}+{\alpha }_2{J}_2)V_t^{\otimes N}dt+\sqrt{2\gamma {\beta }^{-1}}d{w}_t,\hfill \end{array}\end{array}$$

(28)

where $J_1$ and $J_2\in {\mathbb{R}}^{dN\times dN}$ are real-valued skew-symmetric matrices, and ${\alpha }_1$ and ${\alpha }_2\in {\mathbb{R}}_{+}$ are positive constants and $V_t^{\otimes N}={\left(V_t^{\left(1\right)},\dots ,V_t^{\left(N\right)}\right)}^{\top }\in {\mathbb{R}}^{dN}$. We refer to this dynamics as skew PULD (S-PULD) whose faster convergence can be assured similar to Lemma 2 as shown in Appendix I.2.

4.2. Discussion of the Discretization of SDE and Stochastic Gradient and Practical Algorithm

In this section, we further consider practical settings for S-PLD and S-PULD. We discretize these continuous dynamics, e.g., by the Euler-Maruyama method, and approximate the gradient by the stochastic gradient. Although introducing skew matrices accelerates the convergence of continuous dynamics, it simultaneously increases the discretization and stochastic gradient error, resulting in a trade-off. We present a practical algorithm that controls this trade-off.

4.2.1. Trade-off Caused by Discretization and Stochastic Gradient

We consider the following discretization and stochastic gradient for S-PLD and S-PULD:

$$\begin{array}{c}\hfill X_{k+1}^{\otimes N}=X_k^{\otimes N}-h(I_{dN}+\alpha J)\nabla {\widehat{U}}^{\otimes N}\left(X_k^{\otimes N}\right)+\sqrt{2h{\beta }^{-1}}{ϵ}_k,\end{array}$$

(29)

and

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & X_{k+1}^{\otimes N}=X_k^{\otimes N}+{\Sigma }^{-1}{V}_k^{\otimes N}h+\alpha J\nabla {\widehat{U}}^{\otimes N}\left(X_k^{\otimes N}\right)h\hfill \\ \hfill & V_{k+1}^{\otimes N}=V_k^{\otimes N}-\nabla {\widehat{U}}^{\otimes N}\left(X_k^{\otimes N}\right)h-\gamma {\Sigma }^{-1}{V}_k^{\otimes N}h+\sqrt{2\gamma {\beta }^{-1}h}{ϵ}_k,\hfill \end{array}\end{array}$$

(30)

where ${ϵ}_k\in {\mathbb{R}}^{dN}$ is a standard Gaussian random vector. $\nabla {\widehat{U}}^{\otimes N}\left(X^{\otimes N}\right)$ is an unbiased estimator of the gradient $\nabla U^{\otimes N}\left(X^{\otimes N}\right)$. We refer to Equation (29) as skew-SGLD and Equation (30) as skew-SGHMC. For skew-SGHMC, we dropped $J_2$ of S-PULD to decrease the parameters, shown in Appendix B. Please note that skew-SGLD is the identical as the previous dynamics [18]. We introduce an assumption about the stochastic gradient:

Assumption 7. 

(Stochastic gradient) There exists a constant $\delta \in [0,1)$ such that

$$\begin{array}{c}\hfill \mathbb{E}[\parallel \nabla \widehat{U}\left(x\right)-\nabla U\left(x\right){\parallel }^2]\le 2\delta \left(M^2{\parallel x\parallel }^2+B^2\right).\end{array}$$

(31)

Given a test function f with $L_f$ lipschitzness, we approximate $\int fd\pi$ by skew-SGLD or skew-SGHMC, with estimator $\frac{1}{N}{\sum }_{n=1}^{N}f\left(X_k^{\left(n\right)}\right)$. The bias of skew-SGLD is upper-bounded as

Theorem 7.

Under Assumptions 1–7, for any $k\in \mathbb{N}$ and any $h\in (0,1\wedge \frac{m}{4M^2})$ obeying $kh\ge 1$ and $\beta m\ge 2$, we have

$$\begin{array}{cc}\hfill & \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|\le L_f(\underset{\left(i\right)}{\underbrace{C_1\left(\alpha \right)kh}}+\underset{\left(ii\right)}{\underbrace{C_2{e}^{-{\beta }^{-1}m(\alpha ,N)kh}}})\hfill \end{array}$$

(32)

and $C_1$ and $C_2$ depends on the constants of Assumptions 1–7, for the details see Appendix J.

We present a tighter bias bound in Section 4.3 under a stronger assumption. We can show a similar upper bound for the skew-SGHMC using the same proof strategy. This bound resembles of a previous one [18]; ours shows improved dependency on $kh$. The previous results of [18] are also limited to LD, not including skew-SGHMC.

Please note that $\left(i\right)$ corresponds to the discretization and stochastic gradient error and $\left(ii\right)$ corresponds to the convergence behavior of S-PLD, which is continuous dynamics. Since $C_1\left(\alpha \right)\ge C_1(\alpha =0)$, skew acceleration increases the discretization and stochastic gradient error. On the other hand, since $m(\alpha ,N)\ge m_0$, the convergence of the continuous dynamics is accelerated. Thus, skew acceleration causes a trade-off. When $\alpha$ is sufficiently small, we derive the explicit dependency of $\alpha$ for this trade-off from an asymptotic expansion. Using the quantitative evaluation of skew acceleration in Theorem 6, we obtain

$$\begin{array}{cc}\hfill & \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|\le \underset{\left(i\right)}{\underbrace{(d_1\alpha +d_2{\alpha }^2)kh}}-\underset{\left(ii\right)}{\underbrace{{\alpha }^2{d}_0{e}^{-{\beta }^{-1}{m}_{0}kh}}}+\mathcal{O}\left({\alpha }^3\right)+\mathrm{const},\hfill \end{array}$$

(33)

where $d_0$ to $d_2$ are positive constants obtained by the asymptotic expansion. See Appendix K for the details. In the above expression, $\left(i\right)$ and $\left(ii\right)$ correspond to $\left(i\right)$ and $\left(ii\right)$ of Equation (32). Thus, by choosing appropriate $\alpha$, we can control the trade-off.

4.2.2. Practical Algorithm Controlling the Trade-off

Since calculating the optimal $\alpha$ that minimizes Equation (33) at each step is computationally demanding, we adaptively tune the value of $\alpha$ by measuring the acceleration with kernelized Stein discrepancy (KSD) [22]. Our idea is to update samples under different $\alpha$ and $\alpha +\eta$, and compare KSD between the stationary and empirical distributions of these different interaction strengths. Here, $\eta \in {\mathbb{R}}^{+}$ is a small increment of $\alpha$. We denote the samples at the $(k+1)$th step, which is obtained by Equation (29) as $X_{k+1,\alpha }^{\otimes N}:=X_{k,\alpha }^{\otimes N}-h(I_{dN}+\alpha J)\nabla {\widehat{U}}^{\otimes N}\left(X_{k,\alpha }^{\otimes N}\right)+\sqrt{2h{\beta }^{-1}}{ϵ}_k$, (or (30) as ${\displaystyle X_{k+1,\alpha }^{\otimes N}:=X_k^{\otimes N}+{\Sigma }^{-1}{V}_k^{\otimes N}h+\alpha J\nabla {\widehat{U}}^{\otimes N}\left(X_k^{\otimes N}\right)h}$). We denote the samples, which are obtained by replacing the above $\alpha$ by $\alpha +\eta$, as $X_{k+1,\alpha +\eta }^{\otimes N}$. We denote the KSD between the measure of $X_{k+1,\alpha }^{\otimes N}$ and stationary measure $\pi$ as $KSD(k+1,\alpha )$ and estimate the differences of empirical KSD:

$$\begin{array}{cc}\hfill & \Delta :=\widehat{KSD}(k+1,\alpha )-\widehat{KSD}(k+1,\alpha +\eta ),\hfill \end{array}$$

(34)

where KSD is estimated by

$$\begin{array}{cc}\hfill & \widehat{KSD}(k,\alpha )=\frac{1}{N(N-1)}\sum _{i=1}^N{u}_q(X_{k,\alpha }^{\left(i\right)},X_{k,\alpha }^{\left(j\right)}),\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill & u_q(x,x^{\prime }):={\nabla }_{x}log\pi {\left(x\right)}^{\top }l(x,x^{\prime }){\nabla }_{x}log\pi \left(x^{\prime }\right)+{\nabla }_{x}log\pi {\left(x\right)}^{\top }{\nabla }_{x^{\prime }}l(x,x^{\prime })\hfill \\ \hfill & +{\nabla }_{x}l{(x,x^{\prime })}^{\top }{\nabla }_{x}log\pi +\mathrm{Tr}{\nabla }_{x,x^{\prime }}l(x,x^{\prime }),\hfill \end{array}$$

(36)

where l denotes a kernel and we use an RBF kernel. If $\Delta >0$, which indicates that the empirical distribution of $X_{k+1,\alpha +\eta }^{\otimes N}$ is closer to the stationary distribution than that of $X_{k+1,\alpha }^{\otimes N}$. Thus, we should increase the interaction strength from $\alpha$ to $\alpha +\eta$. If $\Delta <0$, we decrease it to $\alpha -\eta$. We also update $\eta$ to $c\eta$ where $c\in (0,1]$. The overall process is shown in Algorithm 1. Detailed discussions of the algorithm including how to select ${\alpha }_0,{\eta }_0$, and c are shown in Appendix L.

Algorithm 1 Tuning $\alpha$

Input: $X_k^{\otimes N},{\eta }_k,{\alpha }_k,c$ Output: ${\alpha }_{k+1},{\eta }_{k+1}$   1: Calculate $X_{k+1,{\alpha }_k}^{\otimes N}$ and $X_{k+1,{\alpha }_k+{\eta }_k}^{\otimes N}$.   2: Calculate $\Delta :=\widehat{KSD}(k+1,{\alpha }_k)-\widehat{KSD}(k+1,{\alpha }_k+{\eta }_k)$   3: if $\Delta >0$ then   4:  Update ${\alpha }_{k+1}={\alpha }_k+{\eta }_k$   5:  Update ${\eta }_{k+1}={\eta }_k$   6: else   7:  Update ${\alpha }_{k+1}=|{\alpha }_k-{\eta }_k|$   8:  Update ${\eta }_{k+1}=c{\eta }_k$   9: end if

Finally, we present Algorithm 2, which describes the whole process. We update the value of $\alpha$ once every $k^{\prime }$ step. Please note that its computational cost is not much larger than that of Equation (30). We only calculate the eigenvalues of J once, which requires $\mathcal{O}\left(N^3\right)$. The calculation of different KSDs is computationally inexpensive since we can re-use the gradient, which is the most computationally demanding part.

Algorithm 2 Proposed algorithm

Input: $X_0^{\otimes N},h,{\alpha }_0,\eta ,k^{\prime },K,c,(V_0^{\otimes N},\gamma ,{\Sigma }^{-1})$ Output: $X_K^{\otimes N}$   1: Make a $N\times N$ random matrix $J_0$ and check $\ker J_0=\left\{0\right\}$   2: Set $J=J_0\otimes I_d$   3: for $k=0$ to K do   4:  if $\lfloor \frac{k}{k^{\prime }}\rfloor =0$ then   5:   Update $\alpha$ by Algorithm 1   6:  end if   7:  Update $X_k^{\otimes N}$ by Equation (29) (for skew-SGLD)   8:  (Update $(X_k^{\otimes N},V_k^{\otimes N})$ by Equation (30) for skew-SGHMC)   9: end for

4.3. Refined Analysis for the Bias of Skew-SGLD

When using a constant step size for skew-SGLD, the bound in Theorem 7 is meaningless since the first term of Equation (32) will diverge. Here, following [23], we present a tighter bound for the bias of skew-SGLD under a stronger assumption.

Theorem 8.

Under Assumptions 1–7, for any $k\in \mathbb{N}$ and any $h\in (0,1\wedge \frac{\lambda (\alpha ,N)}{4\sqrt{2}{M}^2}\wedge \frac{m}{4M^2})$ obeying $kh\ge 1$ and $\beta m\ge 2$, we have

$$\begin{array}{cc}\hfill & \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|\le L_f\sqrt{\frac{2}{\lambda (\alpha ,N)}}\sqrt{e^{-\lambda (\alpha ,N)kh}\mathrm{KL}\left({\mu }_0\right|\pi )+\frac{C_3\left(\alpha \right)}{\lambda (\alpha ,N)}},\hfill \end{array}$$

(37)

where

$$\begin{array}{c}\hfill \lambda (\alpha ,N):={\left(\frac{1}{(1+m{(\alpha ,N)}^{-1}\beta C\left(m_0\right))2\pi e^2}+\frac{3}{2}m{(\alpha ,N)}^{-1}\right)}^{-1}\end{array}$$

(38)

and constants $C_3\left(\alpha \right)$ and $C\left(m_0\right)$ depend on the constants of Assumptions 1–7. Moreover, $\lambda (\alpha ,N)$ satisfies $\lambda (\alpha ,N)\ge \lambda (\alpha =0,N)$. For the details, see Appendix M.

Proof is shown in Appendix M. Please note that even if we use a constant step size for skew-SGLD, the bound in Theorem 8 will not diverge. Here we need the stronger assumption about a step size compared to Theorem 7. From Equation (37), the convergence behavior is characterized by $\lambda (\alpha ,N)$ and the bias bound become smaller when $\lambda (\alpha ,N)$ become larger. From the definition of $\lambda (\alpha ,N)$, the larger $m(\alpha ,N)$ is, the larger $\lambda (\alpha ,N)$ we obtain. Thus, as we had seen so far, introducing the skew matrices leads to the larger Poincaré constant, and thus, this leads to larger $\lambda (\alpha ,N)$.

Previous work [18] clarified that if $\alpha$ is sufficiently small, introducing skew matrices improves the Poincaré constant by a constant factor, which means that we have $m(\alpha ,N)-m_0\approx \mathcal{O}\left({\alpha }^2\right)$, where $\mathcal{O}\left({\alpha }^2\right)$ depends on the eigenvector and eigenvalues of the generator $\mathcal{L}$. On the other hand, from Theorem 8, for any $\xi >0$, to achieve the bias smaller than $\xi$, it suffice to run skew-SGLD at least for $k\ge \frac{2}{\lambda (\alpha ,N)h}ln\frac{L_f}{\xi }\sqrt{\frac{\mathrm{KL}\left({\mu }_0\right|\pi )}{2\lambda (\alpha ,N)}}$ iterations using the appropriate step size h and under the assumption that $\delta$ and $\alpha$ are small enough (see Appendix M.2 for details). Combined with these observations, introducing skew matrices into SGLD improves the computational complexity for a constant order. Our numerical experiments show that even constant improvement results in faster convergence in practical Bayesian models.

5. Related Work

In this section, we discuss the relationship between our method and other sampling methods.

5.1. Relation to Non-Reversible Methods

As we discussed in Section 1, our work extends the existing analysis of non-reversible dynamics [8,18] and presents a practical algorithm. Compared to those previous works, we focus on the practical setting of Bayesian sampling and derive the explicit condition about J for acceleration. We also derived a formula to quantitatively evaluate skew acceleration based on the asymptotic expansion of the eigenvalues of the perturbed Hessian matrix. A previous work [24], which derived the optimal skew matrices when the target distribution is Gaussian, requires $\mathcal{O}\left(d^3\right)$ computational cost to derive optimal skew matrices, and it is unclear whether it works for non-convex potential functions. On the other hand, our construction method for skew matrices is simple, computationally cheap, and can be applied to general Bayesian models.

Our work analyzes skew acceleration for ULD, which is more effective than LD in practical problems. Another work [8,18] only analyzed skew acceleration for LD. A previous work [17] combined a non-reversible drift term with ULD. Unlike our method, this work’s purpose was to reduce the asymptotic variance of the expectation of a test function and is mainly focusing on sampling from Gaussian distribution.

To the best of our knowledge, our work is the first to focus on the memory issue of skew acceleration and develop a memory-efficient skew matrix for ensemble sampling. Our work also presents an algorithm that controls the trade-off for the first time. Another work [18] identified the trade-off and handled it by cross-validation, which is computationally inefficient, unfortunately.

Finally, we point out an interesting connection between our skew-SGHMC and the magnetic HMC (M-HMC) [25]. M-HMC accelerates HMC’s mixing time by introducing a “magnetic” term into the Hamiltonian. That magnetic term is expressed by special skew matrices. Although a previous work [25] argued that M-HMC is numerically superior to a standard HMC, its theoretical property remains unclear. Thus, our work can analyze the theoretical behavior of magnetic HMC.

5.2. Relation to Ensemble Methods

Our proposed algorithm is based on ensemble sampling [26]. Ensemble sampling, in which multiple samples are simultaneously updated with interaction, has been attracting attention numerically and theoretically because of improvements in memory size, computational power, and parallel processing computation schemes [26]. There are successful, widely used ensemble methods, including SVGD [27] and SPOS [28], with which we compare our proposed method numerically in Section 6. Although both show numerically good performance, it is unclear how the interaction term theoretically accelerates the convergence since they are formulated as a McKean–Vlasov process, which is non-linear dynamics, complicating establishing a finite sample convergence rate. Our algorithm is an extension of another work [18], where the interaction was composed of a skew-acceleration term and can be rigorously analyzed. Compared to that previous work [18], we analyzed skew acceleration, focused on the Hessian matrix, and developed practical algorithms, as discussed in Section 4.2, and derived the explicit condition when acceleration occurs, which was unclear [18].

Another difference among SPOS, SVGD, and [18] is that they use first-order methods; our approach uses the second-order method. Little work has been done on ensemble sampling for second-order dynamics. Recently a second-order ensemble method was proposed [29], based on gradient flow analysis. Although its method showed good numerical performance, its theoretical property for finite samples remains unclear since it proposed a scheme as a finite sample approximation of the gradient flow. In contrast, our proposed method is a valid sampling scheme with a non-asymptotic guarantee.

6. Numerical Experiments

The purpose of our numerical experiments is to confirm the acceleration of our algorithm proposed in Section 4 in various commonly used Bayesian models including Gaussian distribution (toy data), latent Dirichlet allocation (LDA), and Bayesian neural net regression and classification (BNN). We compared our algorithm’s performance with other ensemble sampling methods: SVGD, SPOS, standard SGLD, and SGHMC. In all the experiments, the values and the error bars are the mean and the standard deviation of repeated trials. For all the experiments we set $\gamma =1$ and ${\Sigma }^{-1}=300$ for SGHMC and Skew-SGHMC. As for the hyperparameters of our proposed algorithm, the selection criterion is discussed in Appendix L.

6.1. Toy Data Experiment

The target distribution is the multivariate Gaussian distribution, $\pi =N(\mu ,\Omega )$ where we generated ${\Omega }^{-1}=A^{\top }A$ and each element of $A\in {\mathbb{R}}^{2d\times d}$ is drawn from the standard Gaussian distribution. The dimension of the target distribution is $d=50$, we approximate by 20 samples using the proposed ensemble methods. We tested these toy data because the LD for this target distribution is known as the Ornstein–Uhlenbeck process, and its theoretical properties have been studied extensively e.g., [30]. Thus, by studying the convergence behavior of these toy data, we can understand our proposed method more clearly.

First, we confirmed how the skew-symmetric matrix affects the eigenvalues of the Hessian matrix, as discussed in Section 3, where we only showed the asymptotic expansion for the smallest real part of the eigenvalues and saddle point. Here we can show a similar expansion for the largest real part:

$$\begin{array}{cc}\hfill & {\displaystyle \mathrm{Re}\left({\lambda }_{dN}^{\alpha }\right)={\lambda }_{dN}^0+{\alpha }^2\sum _{k=1}^{dN-1}\frac{{\lambda }_d{N}^0{\lambda }_k^0{\left|v_k^{0}J{v}_{dN}^0\right|}^2}{{\lambda }_k^0-{\lambda }_{dN}^0}+\mathcal{O}\left({\alpha }^3\right).}\hfill \end{array}$$

(39)

$\mathrm{Re}\left({\lambda }_{dN}^{\alpha }\right)\le {\lambda }_{dN}^{\alpha }$ holds.

Then we observed how the largest and smallest real parts of the eigenvalues of $(I+\alpha J){\Omega }^{-1}$ depend on $\alpha$. The results are shown in Figure 2, where we averaged 10 trials over a randomly made J with fixed A. The upper-left, upper-right, and lower figures show $\mathrm{Re}\left({\lambda }_1\left(\alpha \right)\right)$, $\mathrm{Re}\left({\lambda }_{dN}\left(\alpha \right)\right)$, and $\mathrm{Re}\left({\lambda }_1\left(\alpha \right)\right)/\mathrm{Re}\left({\lambda }_{dN}\left(\alpha \right)\right)$. These behaviors are consistent with Theorem 3. When $\alpha$ is small, its behavior is close to the quadratic function proved in Theorem 3.

Figure 2. Eigenvalue changes (averaged over ten trials).

Next, we observed the convergence behavior of skew-SGLD and skew-SGHMC. We measured the convergence by maximum mean discrepancy (MMD) [31] between the empirical and stationary distributions. For MMD, we used 2000 samples for the target distribution, and we used the Gaussian kernel whose bandwidth is set to the median distance of these 2000 samples. We used gradient descent (GD), with step size $h=1\times {10}^{-4}$. The results are shown in Figure 3. The proposed method shows faster convergence than naive parallel sampling, which is consistent with Table 2.

Figure 3. Convergence behavior of toy data in MMD (averaged over ten trials).

6.2. LDA Experiment

We tested with an LDA model using the ICML dataset [32] following the same setting as [33]. We used 20 samples for all the methods. Minibatch size is 100. We used step size $h=5\times {10}^{-4}$. First, we confirmed the effectiveness of our proposed Algorithm 1, which adaptively tunes $\alpha$ values. For that purpose, we compared the final performance obtained by our methods with a previous method [18], in which $\alpha$ is selected by cross-validation (CV). Here instead of CV, we just fixed $\alpha$ during the sampling and refer to it as fixed $\alpha$. We also tested the case when J is generated randomly at each step with fixed $\alpha$, as discussed in Section 4.1. We refer to it as random J. The results are shown in Figure 4 where skew-SGLD was used. We found that our method showed competitive performance with the best performance of fixed $\alpha$. For the computational cost, we used $k^{\prime }=2$ in Algorithm 2, and our method needed twice the wall clock time than each fixed $\alpha$. This means that our algorithm greatly reduces the total computational time since we tried more than two $\alpha$s in the fixed $\alpha$ for CV. We also found that since using different Js at each step did not accelerate the performance, we need to store and fix J during the sampling for acceleration. Next, we compared our method with other ensemble sampling schemes and observed the convergence speed. The result is shown in Figure 5. Skew-SGLD and skew-SGHMC outperformed SGLD and SGHMC, which is consistent with our theory.

Figure 4. Final performances of LDA under different values of $\alpha$ (averaged over ten trials).

Figure 5. LDA experiments (Averaged over 10 trials).

6.3. BNN Regression and Classification

We tested with the BNN regression task using the UCI dataset [34], following a previous setting Liu and Wang [27]. We used one hidden layer neural network model with ReLU activation and 100 hidden units. We used 10 samples for all the methods. We used the minibatch size 100. We used step size $h=5\times {10}^{-5}$. The results are shown in Table 1 and Table 2. We also tested on BNN classification task using the MNIST dataset. The result is shown in Figure 6. We used one hidden layer neural network model with ReLU activation and 100 hidden units. Batchsize is 500 and we set step size $h=5\times {10}^{-5}$. Our proposed methods outperformed other ensemble methods. Please note that skew-SGHMC and skew-SGLD consistently outperformed SGHMC and SGLD.

Table 1. Benchmark results on test RMSE for regression task.

Dataset	Avg. Test RMSE
	SVGD	SPOS	SGLD	Skew-SGLD	SGHMC	Skew-SGHMC
Concrete	5.709 ± 0.040	5.239 ± 0.199	5.009 ± 0.091	4.973 ± 0.057	4.949 ± 0.144	4.790 ± 0.081
Kin8nm	0.0731 ± 0.0006	0.0688 ± 0.0003	0.0693 ± 0.0006	0.0689 ± 0.0005	0.0687 ± 0.0001	0.0683 ± 0.0003
Energy	0.520 ± 0.060	0.456 ± 0.030	0.428 ± 0.045	0.412 ± 0.045	0.406 ± 0.019	0.403 ± 0.008
Bostonhousing	3.306 ± 0.005	3.107 ± 0.173	2.948 ± 0.084	2.930 ± 0.095	3.053 ± 0.093	2.986 ± 0.143
Winequality	0.619 ± 0.001	0.618 ± 0.007	0.641 ± 0.003	0.634 ± 0.004	0.614 ± 0.004	0.613 ± 0.004
PowerPlant	4.219 ± 0.012	4.160 ± 0.009	4.129 ± 0.002	4.118 ± 0.006	4.112 ± 0.009	4.105 ± 0.008
Yacht	0.475 ± 0.049	0.467 ± 0.110	0.464 ± 0.058	0.442 ± 0.046	0.464 ± 0.078	0.432 ± 0.051

Table 2. Benchmark results on test negative log likelihood for regression task.

Dataset	Avg. Test Negative Log Likelihood
	SVGD	SPOS	SGLD	Skew-SGLD	SGHMC	Skew-SGHMC
Concrete	−3.157 ± 0.008	−3.124 ± 0.025	−3.052 ± 0.009	−3.049 ± 0.012	−3.046 ± 0.025	−3.033 ± 0.021
Kin8nm	1.153 ± 0.0084	1.212 ± 0.008	1.223 ± 0.002	1.223 ± 0.005	1.230 ± 0.0015	1.235 ± 0.0025
Energy	−0.816 ± 0.102	−0.976 ± 0.079	−0.867 ± 0.056	−0.845 ± 0.021	−0.843 ± 0.045	−0.844 ± 0.041
Bostonhousing	−2.98 ± 0.000	−2.644 ± 0.027	−2.548 ± 0.016	−2.539 ± 0.002	−2.574 ± 0.019	−2.561 ± 0.017
Winequality	−1.012 ± 0.000	−0.959 ± 0.007	−0.976 ± 0.006	−0.968 ± 0.005	−0.941 ± 0.007	−0.938 ± 0.005
PowerPlant	−2.871 ± 0.004	−2.850 ± 0.004	−2.844 ± 0.002	−2.842 ± 0.001	−2.838 ± 0.004	−2.835 ± 0.003
Yacht	−1.184 ± 0.06	−1.372 ± 0.07	−1.077 ± 0.066	−1.078 ± 0.030	−1.083 ± 0.030	−1.079 ± 0.051

Figure 6. MNIST classification (Averaged over ten trials).

7. Conclusions

We studied skew acceleration for LD and ULD from practical viewpoints and concluded that the improved eigenvalues of the perturbed Hessian matrix caused acceleration and derived the explicit condition for acceleration. We described a novel ensemble sampling method, which couples multiple SGLD or SGHMC with memory-efficient skew matrices. We also proposed a practical algorithm that controls the trade-off of faster convergence and larger discretization and stochastic gradient error and numerically confirmed the effectiveness of our proposed algorithm.

Appendix A.1. Wasserstein Distance and Kullback–Leibler Divergence

In this paper, we use the Wasserstein distance. Let us define the Wasserstein distance. Let $(E,d)$ be a metric space (appropriate space such as Polish space) with $\sigma$ field $\mathcal{A}$, where $d(·,·)$ is $\mathcal{A}\times \mathcal{A}$-measurable. Let $\mu$, $\nu$ are probability measures on E, and $p\ge 1$. The Wasserstein distance of order p with cost function d between $\mu$ and $\nu$ is defined as

$$\begin{array}{c}\hfill W_p^d(\mu ,\nu )=\underset{\pi \in \Pi (\mu ,\nu )}{inf}{\left(\int \int d{(x,y)}^{p}d\pi (x,y)\right)}^{1/p},\end{array}$$

(A1)

where $\Pi (\mu ,\nu )$ is the set of all joint probability measures on $E\times E$ with marginals $\mu$ and $\nu$. In this paper, we work on the space ${\mathbb{R}}^d$. As for the distance, we use the Euclidean distance, $\parallel ·\parallel$. For simplicity, we express the p-Wasserstein distance with the Euclidean distance as $W_p$. The various properties of Wasserstein distance are summarized in [35]. We define the Kullback–Leibler (KL) divergence as

$$\begin{array}{c}\hfill \mathrm{KL}(\nu \parallel \mu )=\left\{\begin{array}{c}\int log\frac{d\nu }{d\mu }d\nu ,\nu \ll \mu ,\hfill \\ +\infty ,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(A2)

Appendix A.2. Markov Diffusion and Generator

Here we introduce the additional explanation about the generator of the Markov diffusion process. Given an SDE,

$$\begin{array}{c}\hfill d{X}_t=-\nabla U\left(X_t\right)dt+\sqrt{2{\beta }^{-1}}dw\left(t\right),\end{array}$$

(A3)

and we denote the corresponding Markov semigroup as $P={\left\{P_t\right\}}_{t>0}$ and define the Kolmogorov operator as $P_s$ which is defined as $P_{s}f\left(X_t\right)=\mathbb{E}\left[f\left(X_{t+s}\right)\right|X\left(t\right)]$, where $f:{\mathbb{R}}^d\to \mathbb{R}$ is some bounded test function in ${\mathrm{L}}^2\left(\mu \right)$. A property $P_{s+t}=P_s\circ P_t$ is called Markov property. A probability measure $\pi$ is the stationary distribution when it satisfies for all measurable bounded function f and t, ${\int }_{{\mathbb{R}}^d}{P}_{t}fd\pi ={\int }_{{\mathbb{R}}^d}fd\pi$.

We denote the infinitesimal generator of the associated Markov group as $\mathcal{L}$ and we call it a generator for simplicity. The linearity of the operators of $P_t$ with the semigroup property indicates that $\mathcal{L}$ is the derivative of $P_t$ as

$$\begin{array}{c}\hfill \frac{1}{h}(P_{t+h}-P_t)=P_t\frac{1}{h}(P_h-Id)=\frac{1}{h}(P_h-Id)P_t,\end{array}$$

(A4)

where $Id$ is the identity map. In addition, taking $h\to 0$, we have $\partial P_t=\mathcal{L}{P}_t=P_t\mathcal{L}$. From the Hille–Yoshida theory [19], there exists a dense linear subspace of ${\mathrm{L}}^2\left(\pi \right)$ on which $\mathcal{L}$ exists. We refer it as $\mathcal{D}\left(\mathcal{L}\right)$. If the Markov semigroup is associated with the SDE of Equation (A3), the generator can be written as

$$\begin{array}{cc}\hfill \mathcal{L}f\left(X_t\right):& =\underset{h\to 0^{+}}{lim}\frac{\mathbb{E}\left(f\left(X_{t+h}\right)\right|X_t)-f\left(X_t\right)}{h}=\left(-\nabla U\left(X_t\right)·\nabla +{\beta }^{-1}\Delta \right)f\left(X_t\right),\hfill \end{array}$$

(A5)

where $\Delta$ is the Laplacian in the standard Euclidean space. The generator satisfies $\mathcal{L}1=0,{\int }_{{\mathbb{R}}^d}\mathcal{L}fd\pi =0$.

Appendix A.3. Poincaré Inequality

We use the Poincaré inequality to measure the speed of convergence to the stationary distribution. In this section, we summarize definitions and useful properties of them and see [19] for more details. We define the Dirichlet form $\mathcal{E}\left(f\right)$ for all bounded functions $f\in \mathcal{D}\left(\mathcal{L}\right)$ where $\mathcal{D}\left(\mathcal{L}\right)$ denotes the domain of $\mathcal{L}$ as

$$\begin{array}{c}\hfill \mathcal{E}\left(f\right):=-{\int }_{{\mathbb{R}}^d}f\mathcal{L}fd\pi .\end{array}$$

(A6)

$\mathcal{E}\left(f\right)>0$ is satisfied. By the partial integration, we have $\mathcal{E}\left(f\right)=-{\int }_{{\mathbb{R}}^d}f\mathcal{L}fd\pi =\frac{1}{\beta }{\int }_{{\mathbb{R}}^d}{\parallel \nabla f\parallel }^{2}d\pi$. We define a Dirichlet domain, $\mathcal{D}\left(\mathcal{E}\right)$, which is the set of functions $f\in L^2\left(\pi \right)$ and satisfies $\mathcal{E}\left(f\right)<\infty$.

We say that $\pi$ with $\mathcal{L}$ satisfies a Poincaré inequality with a positive constant c if for any $f\in \mathcal{D}\left(\mathcal{E}\right)$, $\pi$ with $\mathcal{L}$ satisfies,

$$\begin{array}{c}\hfill \int f^{2}d\pi -{\left(\int fd\pi \right)}^2\le c\mathcal{E}\left(f\right).\end{array}$$

(A7)

This constant c is closely related to a spectral gap. If the smallest eigenvalue of $\mathcal{L}$, $\lambda$, is greater than 0, then it is called the spectral gap. If the spectral gap $\lambda >0$ exists, then it is written as

$$\begin{array}{c}\hfill \lambda :={\inf }_{f\in \mathcal{D}\left(\mathcal{E}\right)}\left\{\frac{\mathcal{E}\left(f\right)}{\int f^{2}d\pi }:f\ne 0,\int fd\pi =0\right\}.\end{array}$$

(A8)

From this, a constant c which satisfies $c\ge 1/\lambda$, can also satisfy the Poincaré inequality. To check the existence of the spectral gap, one approach is to use the Lyapunov function, which is developed by Bakry et al. [36].

We can also express the Poincaré inequality via chi divergence. Let us define the ${\chi }^2$ divergence for $\mu \ll \pi$ as

$$\begin{array}{c}\hfill {\chi }^2(\mu \parallel \pi ):={∥\frac{d\mu }{d\pi }-1∥}_{L_{\pi }^2}^2={\int }_{{\mathbb{R}}^d}{\left|\frac{d\mu }{d\pi }-1\right|}^{2}d\pi .\end{array}$$

(A9)

Then, we express the Poincaré inequality with a constant c for all $\mu \ll \pi$ as

$$\begin{array}{c}\hfill {\chi }^2(\mu \parallel \pi )\le c\mathcal{E}\left(\sqrt{\frac{d\mu }{d\pi }}\right).\end{array}$$

(A10)

We obtain the following exponential convergence results from the above functional inequalities for measures.

Theorem A1. 

(Exponential convergence in the variance, Theorem 4.2.5 in [19]) When π satisfies the Poincaré inequality with a constant c, it implies the exponential convergence in the variance with a rate $2/c$, i.e., for every bounded function $f:{\mathbb{R}}^d\to \mathbb{R}$,

$$\begin{array}{c}\hfill {\mathrm{Var}}_{\pi }\left(P_{t}f\right)\le e^{-2t/c}{\mathrm{Var}}_{\pi }\left(f\right),\end{array}$$

(A11)

where ${\mathrm{Var}}_{\pi }\left(f\right):={\int }_{{\mathbb{R}}^d}{f}^{2}d\pi -{\left({\int }_{{\mathbb{R}}^d}fd\pi \right)}^2$.

We also introduce the important property of Poincaré inequality as for the product measures. These relations play important roles in our analysis.

Theorem A2. 

(Stability under the product, Proposition 4.3.1 in [19]) If ${\mu }_1$ and ${\mu }_2$ on ${\mathbb{R}}^d$ satisfy the Poincaré inequalities with a constant $c_1$ and $c_2$, then the product ${\mu }_1\otimes {\mu }_2$ on ${\mathbb{R}}^d\otimes {\mathbb{R}}^d$ satisfies the Poincaré inequality with the constant $max(c_1,c_2)$.

Appendix C.1. Proof for S-LD

First, under Asuumptions 1–5, LD has a spectral gap, and its Poincaré constant is upper bounded as

$$\begin{array}{c}\hfill \frac{1}{m_0}\le \frac{2C(d+b\beta )}{m\beta }\exp \left(\frac{2}{m}(M+B)(b\beta +d)+\beta (A+B)\right)+\frac{1}{m\beta (d+b\beta )}.\end{array}$$

(A19)

and this is derived in [2].

Next, we introduce the generator of S-LD

$$\begin{array}{c}\hfill {\mathcal{L}}_{\alpha }f\left(x\right)=\left(-\nabla U_{\alpha }\left(x\right)·\nabla +{\beta }^{-1}\Delta \right)f\left(x\right),\end{array}$$

where $\nabla U_{\alpha }\left(x\right):=\nabla U\left(x\right)+\alpha J\nabla U\left(x\right)$.

The proof is almost similar to [18] of Theorem 12.

Proof of Theorem 1. 

Since the generator ${\mathcal{L}}_{\alpha =0}$ is self-adjoint, and the suitable growth condition, the spectral of ${\mathcal{L}}_{\alpha =0}$ is discrete [19]. We denote the spectrum of ${\mathcal{L}}_{\alpha =0}$ as ${\left\{{\lambda }_k\right\}}_{k=0}^{\infty }\in \mathbb{R}$ and corresponding normalized eigenvectors as ${\left\{e_k\right\}}_{k=0}^{\infty }$, which are the real functions. We order the spectrum as $0>{\lambda }_0>{\lambda }_1>\dots$. Thus, $m_0=-{\lambda }_0$.

As for ${\mathcal{L}}_{\alpha }$, although it is not a self-adjoint operator, from Proposition 1 in Franke et al. [39], it has discrete complex spectrums. We denote the spectrum of ${\mathcal{L}}_{\alpha }$ as $\lambda +i\mu \in \mathbb{C}$ where $\lambda ,\mu \in \mathbb{R}$ and corresponding normalized eigenvector as $u+iv$ where $u,v$ are the real functions and then we have

$$\begin{array}{c}\hfill {\mathcal{L}}_{\alpha }(u+iv)=(\lambda +i\mu )(u+iv).\end{array}$$

(A20)

From this definition, by checking the real parts and complex parts, following relations are derived

$$\begin{array}{c}\hfill {\mathcal{L}}_{\alpha }u=\lambda u-\mu v,\end{array}$$

(A21)

$$\begin{array}{c}\hfill {\mathcal{L}}_{\alpha }v=\lambda v+\mu u.\end{array}$$

(A22)

Due to the divergence-free drift property, for any bounded real value test function $g\left(x\right)$,

$$\begin{array}{cc}\hfill \int g({\mathcal{L}}_{\alpha =0}-{\mathcal{L}}_{\alpha })gd\pi & =\int \alpha g\gamma ·\nabla gd\pi =-\int \alpha g\gamma ·\nabla gd\pi ,\hfill \end{array}$$

(A23)

where we used the partial integral. This means that for any bounded real function $g\left(x\right)$,

$$\begin{array}{c}\hfill \int g{\mathcal{L}}_{\alpha =0}gd\pi =\int g{\mathcal{L}}_{\alpha }gd\pi .\end{array}$$

(A24)

(This only holds for real functions.) Then, we can evaluate the real part of the eigenvalue $\lambda$ as follows,

$$\begin{array}{cc}\hfill \int u{\mathcal{L}}_{\alpha =0}ud\pi +\int v{\mathcal{L}}_{\alpha =0}vd\pi & =\int u{\mathcal{L}}_{\alpha }ud\pi +\int v{\mathcal{L}}_{\alpha }vd\pi =\lambda \left(\int u^{2}d\pi +\int v^{2}d\pi \right)=\lambda .\hfill \end{array}$$

(A25)

Then, by expanding the eigenfunction $u,v$ by the eigenfunction $\left\{e_k\right\}$,

$$\begin{array}{cc}\hfill \lambda =\int u{\mathcal{L}}_{\alpha =0}ud\pi +\int v{\mathcal{L}}_{\alpha =0}vd\pi & =\sum _k{\lambda }_k\left({\left(\int u{e}_{k}d\pi \right)}^2+{\left(\int v{e}_{k}d\pi \right)}^2\right)\hfill \\ \hfill & \le {\lambda }_0\sum _k\left({\left(\int u{e}_{k}d\pi \right)}^2+{\left(\int v{e}_{k}d\pi \right)}^2\right)\le {\lambda }_0.\hfill \end{array}$$

(A26)

Thus, the real part of the eigenvalue of ${\mathcal{L}}_{\alpha }$ is smaller than the smallest eigenvalue of ${\mathcal{L}}_{\alpha }$. This means that the spectral gap of ${\mathcal{L}}_{\alpha }$ is larger than that of ${\mathcal{L}}_{\alpha =0}$, i.e., $m\left(\alpha \right)\ge m_0$ holds. □

Appendix C.2. Proof of Theorem 2 (S-ULD)

Proof of Theorem 2. 

To prove the S-ULD, we use the result of [20], which characterize the convergence of ULD via the Poincaré constant. Let us denote ${\tilde{\mu }}_t$ as the measure induced by ULD. Then from Theorem 1 of [20], if $\pi$ with $\mathcal{L}$ has the Poincaré constant $m_0$, we have

$$\begin{array}{c}\hfill {\chi }^2({\tilde{\mu }}_t\parallel \tilde{\pi })\le \frac{1+\overline{ϵ}}{1-\overline{ϵ}}{e}^{-{\lambda }_{\gamma }t}{\chi }^2({\tilde{\mu }}_t\parallel \tilde{\pi }).\end{array}$$

(A27)

where $\overline{ϵ}$ and ${\lambda }_{\gamma }$ is given as follows.

$$\begin{array}{c}\hfill {\lambda }_{\gamma }=\frac{\Lambda (\gamma ,\overline{ϵ}min(\gamma ,{\gamma }^{-1}))}{1+\overline{ϵ}min(\gamma ,{\gamma }^{-1})},\end{array}$$

(A28)

where

$$\begin{array}{cc}\hfill & \Lambda (\gamma ,ϵ)=\frac{\gamma {\Sigma }^{-1}-\frac{1}{1+\frac{m_0{\Sigma }^{-1}}{\beta }}}{2}-\frac{1}{2}\sqrt{{(S_{--}-S_{++})}^2+{\left(S_{-+}\right)}^2},\hfill \end{array}$$

(A29)

$$\begin{array}{cc}\hfill & S_{--}=ϵ{\lambda }_{ham},\hfill \end{array}$$

(A30)

$$\begin{array}{cc}\hfill & S_{-+}=-ϵ(R_{ham}+\gamma {\Sigma }^{-1}/2),\hfill \end{array}$$

(A31)

$$\begin{array}{cc}\hfill & S_{++}=\gamma {\Sigma }^{-1}-ϵ,\hfill \end{array}$$

(A32)

$$\begin{array}{cc}\hfill & {\lambda }_{ham}=1-{\left(1+\frac{m_0{\Sigma }^{-1}}{\beta }\right)}^{-1},\hfill \end{array}$$

(A33)

$$\begin{array}{cc}\hfill & ϵ=\overline{ϵ}min(\gamma ,{\gamma }^{-1}),\hfill \end{array}$$

(A34)

where $\overline{ϵ}$ is arbitrary sufficiently small positive value such that $\Lambda (\gamma ,\overline{ϵ}min(\gamma ,{\gamma }^{-1}))>0$ is satisfies. As for $R_{ham}$, if there exists a positive constant K, such that ${\nabla }^{2}U\ge -KI$, then $R_{ham}\le \sqrt{max\{K,2\}}$. In our assumption, this corresponds to $\beta M$, thus $R_{ham}\le \sqrt{max\{\beta M,2\}}$. From the above definitions, we can see that the larger $m_0$ is, i.e., the larger the Poincaré constant is the faster convergence ULD shows.

This can also be confirmed numerically, see Figure A1, which shows how the $\Lambda$ changes under different $m_0$. We set ${\Sigma }^{-1}=100$. From the figure, the larger the Poincaré constant is, the larger $\Lambda$ becomes.

Figure A1. The convergence rate of ULD under the different Poincaré constants.

So far, we confirmed that the convergence speed of S-ULD is characterized by the Poincaré constant of $\mathcal{L}$. When we consider S-ULD, we simply add the skew matrices term to the generator of the ULD in the proof of Proposition 1 in [20]. This means that we simply replace the Poincaré constant from $m_0$ to $m\left(\alpha \right)$ in the proof of Proposition 1 in [20]. Then, $m_0$ will be replaced with $m\left(\alpha \right)$ that indicates the faster convergence. □

Appendix D.1. Strongly Convex Potential Function

When we consider LD with m-strongly convex potential function, then the Poincaré constant is m, this means exponential convergence with rate m (See [19] for the detail).

We then consider the S-LD with m-strongly convex function. In this setting, by considering the synchronous coupling technique [11], we can show that the variance decays exponentially with the rate of the smallest real part of the eigenvalue. This is because that by preparing two S-LD $(X_t,Y_t)$ given as

$$\begin{array}{c}\hfill d{X}_t=-(I+\alpha J)\nabla U\left(X_t\right)dt+\sqrt{2{\beta }^{-1}}d{w}_t,d{Y}_t=-(I+\alpha J)\nabla U\left(Y_t\right)dt+\sqrt{2{\beta }^{-1}}d{w}_t^{\prime }.\end{array}$$

(A35)

Then we evaluate the behavior of $\parallel X_t-Y_t{\parallel }^2$. From Ito lemma and considering the synchronous coupling, we obtain

$$\begin{array}{c}\hfill \frac{d}{dt}\parallel X_t-Y_t{\parallel }^2=-(X_t-Y_t)·\frac{(I+\alpha J)}{\beta }(\nabla U\left(X_t\right)-\nabla U\left(Y_t\right))\le -\frac{2m\left(\alpha \right)}{\beta }{\parallel X_t-Y_t\parallel }^2,\end{array}$$

(A36)

where $m\left(\alpha \right)$ is the constant that satisfies $m\left(\alpha \right)\le \mathrm{Re}{\lambda }_1^{\alpha }\left(x\right)$ for all x, see Appendix E for details. This means that variance decays exponentially with the rate $\frac{2m\left(\alpha \right)}{\beta }$. From the fundamental property of the Poincaré constant (Theorem 4.2.5 in [19]), $m\left(\alpha \right)$ is the Poincaré constant. Thus the imaginary part has no effect on the continuous dynamics. Thus, the Poincaré inequality is the smallest real part of the perturbed Hessian matrix.

Appendix D.2. Non-Convex Potential Function

As we discussed in Section 3.1, [21] derived the sharper estimation for the Poincaré constant for the non-convex potential function. It is easy to verify that their assumptions are satisfied under our assumption 1–5. Following the main paper, we denote $x_1$ global minima, and $x_2$ is the local minima which have the second smallest value in $U\left(x\right)$. We express the saddle point between $x_1$ and $x_2$ as $x^{\ast }$. To be more precise, the saddle point that characterizes the Poincaré constant is known as the critical point with index one defined as

$$\begin{array}{c}\hfill U\left(x^{\ast }\right)=inf\left\{\underset{s\in [0,1]}{max}U\left(\gamma \left(s\right)\right):\gamma \in C([0,1],{\mathbb{R}}^d),\gamma \left(0\right)=x_1,\gamma \left(1\right)=x_2\right\},\end{array}$$

(A37)

and the eigenvalue of ${\nabla }^{2}U\left(x^{\ast }\right)$ has one negative eigenvalue and $d-1$ positive eigenvalues. We express them as ${\lambda }_1\left(x^{\ast }\right)<0<{\lambda }_2\left(x^{\ast }\right)<\dots ,{\lambda }_d\left(x^{\ast }\right)$.

Ref. [21] studied the Poincaré constant by decomposing the non-convex potential focusing on attractors. By focusing on attractors, they showed that the non-convex potential can be decomposed into the sum of approximately Gaussian distributions. They proved that the Poincaré constant is characterized by the local Poincaré constants, these are derived by the approximate Gaussian distribution on the attractors and their surrounding regions. In addition, they proved that the dominant term of the Poincaré constant is specified by the saddle points between the global minima and the point which takes the second smallest value for $U\left(x\right)$. From Theorem 2.12 and Corollary 2.15 in [21], the Poincaré constant is characterized by

$$\begin{array}{c}\hfill m_0^{-1}\approx \frac{\sqrt{\det H\left(x^{\ast }\right)}}{\sqrt{Z|{\lambda }_1\left(x^{\ast }\right)|\det H\left(x_1\right)}\sqrt{\det H\left(x_2\right)}}{e}^{\beta (U\left(x^{\ast }\right)-U\left(x_1\right)-U\left(x_2\right))}\propto \frac{1}{|{\lambda }_1\left(x^{\ast }\right)|}{e}^{\beta (U\left(x^{\ast }\right)-U\left(x_1\right)-U\left(x_2\right))},\end{array}$$

(A38)

where Z is the normalizing constant of $e^{-\beta U\left(x\right)}$.

Next, we discuss how this estimate changes when skew matrices are applied. When the skew matrices are introduced, from lemma A.1 in [40], at the saddle point, there exists a unique negative real eigenvalue ${\lambda }_1^{\alpha }\left(x^{\ast }\right)<0$ for the perturbed Hessian matrix even if $(I+\alpha J)H$ is not a symmetric matrix.

Then from Proposition 5 in [8], that negative eigenvalue of the perturbed Hessian is smaller than that of the un-perturbed Hessian matrix at the saddle point. This means that ${\lambda }_1^{\alpha }\left(x^{\ast }\right)\le {\lambda }_1\left(x^{\ast }\right)<0$ holds.

Finally, from Theorem 5.1 in [41] and Theorem 2.12 in [21], this improvement of the negative eigenvalue of the saddle point directly leads to the larger Poincaré constant.

Appendix G.1. Proof of Theorem 5

Proof. 

The proof is the straightforward consequence of lemma in [43], that is

Lemma in ([43]) If $f(x_1,\dots ,x_m)$ is a polynomial in real variables $x_1,\dots ,x_m$, which is not identically zero, then the subset $N_m=\left\{(x_1,\dots ,x_m)\right|f(x_1,\dots ,x_m)=0\}$ of the Euclidean m-space ${\mathbb{R}}^m$ has the Lebesgue measure zero.

We use this lemma to prove that the probability of ${\lambda }_1={\alpha }_1$ is 0 by showing that the probability mass of ${\lambda }_1={\alpha }_1$ has Lebesgue measure zero.

We use the same notation as in Appendix F. Recall Equation (A64), which is the condition of equality about ${\lambda }_1={\alpha }_1$. We express the elements of $a_1$ and $b_1$ as $a_1={(a_1^1,\dots ,a_1^d)}^{\top }$ and $b_1={(b_1^1,\dots ,b_1^d)}^{\top }$. Then the equality condition can be written as

$$\begin{array}{cc}\hfill & \sum _{i=1}^d(\sum _{j=1}^d{\lambda }_{1}c\alpha J_{i,j}{a}_1^j+{\beta }_1{c}^{\prime }{b}_1^i){)}^2+\sum _{i=1}^d(\sum _{j=1}^d{\lambda }_1{c}^{\prime }\alpha J_{i,j}{b}_1^j-{\beta }_{1}c{a}_1^i){)}^2=0.\hfill \end{array}$$

(A66)

Then we define the polynomial about $\left\{J_{i.j}\right\}$

$$\begin{array}{c}\hfill f(J_{1,2},\dots ,J_{d-1,d})=\sum _{i=1}^d(\sum _{j=1}^d{\lambda }_{1}c\alpha J_{ij}{a}_1^j+{\beta }_1{c}^{\prime }{b}_1^i){)}^2+\sum _{i=1}^d(\sum _{j=1}^d{\lambda }_1{c}^{\prime }\alpha J_{ij}{b}_1^j-{\beta }_{1}c{a}_1^i){)}^2.\end{array}$$

(A67)

To apply lemma of [43], we must confirm that $f(J_{1,2},\dots ,J_{d-1,d})$ is not always 0. This is clear from the definition of f since we generate $J_{1,2},\dots ,J_{d-1,d}$ randomly from the distribution that is absolutely continuous with respect to Lebesgue measure and ${\lambda }_1\ne 0$ and $c^2+c^{\prime 2}=1$ and either $a_1,b_1\ne 0$.

Then, given an evaluation point x, from lemma of [43], the subset of $\left\{J_{i,j}\right\}\in {\mathbb{R}}^{d(d-1)/2}$ that satisfies $f(J_{1,2},\dots ,J_{d-1,d})=0$ has Lebesque measure zero. Thus, if we generate $\left\{J_{i,j}\right\}$ from the probability measure which is absolutely continuous with respect to Lebesque measure, (such as Gaussian distribution), $f(J_{1,2},\dots ,J_{d-1,d})=0$ holds probability 0. This concludes the proof. □

Appendix G.2. Proof of Lemma 1

Proof. 

We first discuss the condition about $\mathrm{Ker}{J}_0=\left\{0\right\}$. Since $J=J_0\otimes I_d$, and we denote the set of eigenvalues of $J_0$ as $\left\{{\omega }_i\right\}$. In general, the eigenvalues of the matrix that is composed of the Kronecker product with two matrices, e.g., A and B, are given as the product of each eigenvalue of A and B [44]. Thus, since J is the Kronecker product of $J_0$ and $I_d$, if $J_0$ does not have 0 as an eigenvalue, J does not have 0 as an eigenvalue.

Next, we discuss another equality condition. We use the similar notation as in Appendix F, but now the dimension of the matrix J is $dN$. We express the eigenvalue which has the smallest real part as ${\lambda }_1^{\alpha }$ and its eigenvector as ${\omega }_1^{\alpha }=a_1+i{b}_1$. The elements of $a_1$ and $b_1$ as $a_1={(a_1^1,\dots ,a_1^d,a_1^{d+1},\dots ,a_1^{dN})}^{\top }\in {\mathbb{R}}^{dN}$ and $b_1={(b_1^1,\dots ,b_1^d,\dots ,b_1^{d+1},\dots ,b_1^{dN})}^{\top }$. We also express these as $a_1={(a_1^{\left(1\right)},\dots ,a_1^{\left(N\right)})}^{\top }\in {\mathbb{R}}^{dN}$ where $a_1^{\left(i\right)}={(a_1^{(i-1)d+1},\dots ,a_1^{id})}^{\top }\in {\mathbb{R}}^d$.

We use the Kronecker product property:

$$\begin{array}{c}\hfill J{a}_1=(J_0\otimes I_d)a_1={\left(\sum _{i=1}^N{J}_{0|i,1}{a}_1^{\left(i\right)},\dots ,\sum _{i=1}^N{J}_{0|i,N}{a}_1^{\left(i\right)}\right)}^{\top },\end{array}$$

(A68)

where $J_{0|i,j}$ indicates the element of i-th row and j-th column of $J_0$ where we use the property of the Kronecker product and the Vec operator in the second equality [44].

The proof is almost similar to Appendix G.1. Then the equality condition can be written as

$$\begin{array}{cc}\hfill & \sum _{n=1}^N{∥{\lambda }_{1}c\alpha \sum _i{J}_{0|i,n}{a}_1^{\left(i\right)}+{\beta }_1{c}^{\prime }{b}_1^{\left(n\right)}∥}^2+\sum _{n=1}^N{∥{\lambda }_1{c}^{\prime }\alpha \sum _i{J}_{0|i,n}{b}_1^{\left(i\right)}+{\beta }_{1}c{a}_1^{\left(n\right)}∥}^2=0,\hfill \end{array}$$

(A69)

where $\parallel ·\parallel$ is the d-dimensional Euclidean norm since $a_1^{\left(n\right)},b_1^{\left(n\right)}\in {\mathbb{R}}^d$. Then we define the polynomial about $\left\{J_{i.j}\right\}$

$$\begin{array}{c}\hfill f(J_{1,2},\dots ,J_{N-1,N})=\sum _{n=1}^N{∥{\lambda }_{1}c\alpha \sum _i{J}_{0|i,n}{a}_1^{\left(i\right)}+{\beta }_1{c}^{\prime }{b}_1^{\left(n\right)}∥}^2+\sum _{n=1}^N{∥{\lambda }_1{c}^{\prime }\alpha \sum _i{J}_{0|i,n}{b}_1^{\left(i\right)}+{\beta }_{1}c{a}_1^{\left(n\right)}∥}^2.\end{array}$$

(A70)

In a similar discussion with Appendix G.1, it is clear that f is not always 0. Thus, given an evaluation point x, from lemma of [43], the subset of $\left\{J_{i,j}\right\}\in {\mathbb{R}}^{N(N-1)/2}$ that satisfies $f(J_{1,2},\dots ,J_{N-1,d})=0$ has Lebesque measure zero. Thus, if we generate $\left\{J_{i,j}\right\}$ from the probability measure which is absolutely continuous with respect to Lebesque measure, (such as Gaussian distribution), $f(J_{1,2},\dots ,J_{N-1,N})=0$ holds probability 0. This concludes the proof. □

Appendix G.3. Extending the Theorem to the Path

About Theorem 5 and Lemma 1, the statement holds true when we fix an evaluation point x. To ensure the acceleration, we need to extend Theorem 5 and Lemma 1 from a single evaluation point to the path of the stochastic process for S-LD, S-PLD, S-ULD, and S-PULD.

First, the condition of $\mathrm{Ker}{J}_0=\left\{0\right\}$ is not related to the evaluation point. Thus, we need to consider the equality condition for $\mathrm{Re}{\lambda }_1^{\alpha }={\lambda }_1^0$. As for this condition, as we had seen in Theorem 5 and Lemma 1, if we generate the random matrix J which is absolutely continuous with respect to Lebesgue measure, then the equality condition is not satisfied with probability 1 at the given evaluation point. The important point in those proof is to prove that the event when the equality holds has Lebesgue measure 0 at the given evaluation point using the lemma of [43].

Let us consider when two evaluation points are given (e.g., $x_1$, $x_2$), and we check whether the random matrix J satisfies the above equality condition or not. We can easily prove that at each evaluation point, such an event (we express them as $S_1$ and $S_2$) has Lebesgue measure 0 using the lemma of [43] (We refer to this as $P\left(S_1\right)=0$ and $P\left(S_2\right)=0$ where P is the law induced by generating the random matrix that has independent $d(d-1)/2$ elements). So, the volume of the event of sum of $S_1$ and $S_2$ are also 0 ($P(S_1\bigcup S_2=0$). By repeating this procedure, when given a finite number of evaluation points, $(x_1,\dots ,x_k)$, the sum of such probability is 0 (this indicates $P(S_1\bigcup S_2,\dots ,\bigcup S_k)=0$).

When we consider the discretized dynamics of S-LD, S-PLD, and so on, and update samples up to k-iterations, then there exist k evaluation points. So, by applying the above discussion, we can ensure that along the path of the discretized dynamics, the equality condition does not hold with probability 1. On the other hand, as for the continuous dynamics, the evaluation point is infinite, thus when we cannot conclude that the probability that the equality does not hold is 1.

Appendix H.1. Asymptotic Expansion When the Smallest Eigenvalue of H(x) Is Positive

We work on the similar matrix of $H^{\prime }$, that is $H+\alpha V$ where $V:=H^{1/2}J{H}^{1/2}$. See Appendix G.1 for the detail. Please note that this similar matrix only exists when the smallest eigenvalue of $H\left(x\right)$ is positive. Thus, the following discussion cannot apply to the case at the saddle point, where negative eigenvalues appear. We discuss the saddle point expansion later.

From the definition, we have

$$\begin{array}{c}\hfill H^{\prime }{w}_j=H{w}_j+\alpha V{w}_j={\mu }_j{w}_j=({\lambda }_j+\delta {\lambda }_j)(v_j+\delta v_j),\end{array}$$

(A74)

We rearrange this equation as

$$\begin{array}{c}\hfill H{v}_j+H\delta v_j+\alpha V{v}_j+\alpha V\delta v_j={\lambda }_j{v}_j+\delta {\lambda }_j{v}_j+{\lambda }_j\delta v_j+\delta {\lambda }_j\delta v_j.\end{array}$$

(A75)

First, we focus on the first-order expansion. This means we neglect high-order terms. Then, we have

$$\begin{array}{c}\hfill H{v}_j+H\delta v_j+\alpha V{v}_j={\lambda }_j{v}_j+\delta {\lambda }_j{v}_j+{\lambda }_j\delta v_j.\end{array}$$

(A76)

By multiplying $v_j$ to Equation (A76) from the left-hand side, we have

$$\begin{array}{c}\hfill {\lambda }_j+{\lambda }_j{v}_j^{\top }\delta v_j+\alpha v_j^{\top }V{v}_j={\lambda }_j+\delta {\lambda }_j+{\lambda }_j{v}_j^{\top }\delta v_j,\end{array}$$

(A77)

Since $v_j^{\top }V{v}_j=0$ due to the skew-symmetric property of V. Thus, we have

$$\begin{array}{c}\hfill \delta {\lambda }_j=0,\end{array}$$

(A78)

up to the first-order expansion. Then we substitute this into Equation (A76) and multiplying $v_i$ where $i\ne j$, we have

$$\begin{array}{c}\hfill {\lambda }_i{c}_{ji}+\alpha v_i^{\top }V{v}_j={\lambda }_j{c}_{ji}.\end{array}$$

(A79)

Then we have

$$\begin{array}{c}\hfill c_{ji}=\frac{\alpha v_i^{\top }V{v}_j}{{\lambda }_j-{\lambda }_i}.\end{array}$$

(A80)

Then we obtain

$$\begin{array}{c}\hfill \delta v_j=\alpha \sum _{i\ne j}^d\frac{v_i^{\top }V{v}_j}{{\lambda }_j-{\lambda }_i}{v}_i.\end{array}$$

(A81)

We substitute this into Equation (A75), and multiplying $v_j^{\top }$, we have

$$\begin{array}{cc}\hfill & v_j^{\top }H\alpha \sum _{i\ne j}^d\frac{v_i^{\top }V{v}_j}{{\lambda }_j-{\lambda }_i}{v}_i+\alpha v_j^{\top }V{v}_j+\alpha v_j^{\top }V\alpha \sum _{i\ne j}^d\frac{v_i^{\top }V{v}_j}{{\lambda }_j-{\lambda }_i}{v}_i\hfill \\ \hfill & =\delta {\lambda }_j{v}_j^{\top }{v}_j+{\lambda }_j{v}_j^{\top }\alpha \sum _{i\ne j}^d\frac{v_i^{\top }V{v}_j}{{\lambda }_j-{\lambda }_i}{v}_i+\delta {\lambda }_j{v}_j^{\top }\alpha \sum _{i\ne j}^d\frac{v_i^{\top }V{v}_j}{{\lambda }_j-{\lambda }_i}{v}_i.\hfill \end{array}$$

(A82)

Since $v_j^{\top }V{v}_j=0$ and $v_j^{\top }{v}_i=0$ and $v_j^{\top }{v}_j=1$, we have

$$\begin{array}{c}\hfill {\alpha }^2\sum _{i\ne j}^d\frac{v_i^{\top }V{v}_j}{{\lambda }_j-{\lambda }_i}{v}_j^{\top }V{v}_i=\delta {\lambda }_j.\end{array}$$

(A83)

Thus, we have

$$\begin{array}{c}\hfill {\mu }_j-{\lambda }_j={\alpha }_j+i{\beta }_j-{\lambda }_j=-{\alpha }^2\sum _{i\ne j}^d\frac{{\left(v_i^{\top }V{v}_j\right)}^2}{{\lambda }_j-{\lambda }_i}.\end{array}$$

(A84)

Thus, by taking the real part, and note that $\mathrm{Re}{\lambda }_j\left(\alpha \right)={\alpha }_j$, we have

$$\begin{array}{cc}\hfill \mathrm{Re}{\lambda }_j\left(\alpha \right)-{\lambda }_j& ={\alpha }^2\mathrm{Re}\sum _{i\ne j}^d\frac{{\left(v_i^{\top }V{v}_j\right)}^2}{{\lambda }_i-{\lambda }_j}+\mathcal{O}\left({\alpha }^3\right)={\alpha }^2\sum _{i\ne j}^d\frac{{\lambda }_i{\lambda }_j{\left(v_i^{\top }J{v}_j\right)}^2}{{\lambda }_i-{\lambda }_j}+\mathcal{O}\left({\alpha }^3\right).\hfill \end{array}$$

(A85)

This concludes the proof.

Appendix H.2. Expansion of the Eigenvalue at the Saddle Point

Here we derive the formula of the expansion of the eigenvalue at the saddle point. Since the smallest eigenvalue is negative, we cannot use the similar matrix as shown above. Instead, we use the relation,

$$\begin{array}{c}\hfill {\mu }_{j}H{w}_j=H{\mu }_j{w}_j=H(I+\alpha J)H{w}_j\end{array}$$

(A86)

where we used the definition of the eigenvalues and eigenvectors. Here, we express $H^{\prime }:=(I+\alpha J)H$ and its pairs of eigenvalues and eigenvectors as ${\left\{({\mu }_i,w_i)\right\}}_{i=1}^d$. As introduced in the above, we substitute the expansion to Equation (A86), then we obtain

$$\begin{array}{c}\hfill ({\lambda }_j+\delta {\lambda }_j)H(v_j+\delta v_j)=H(I+\alpha J)H(v_j+\delta v_j)\end{array}$$

(A87)

Then, in the same way as above, since ${\left\{v_j\right\}}_{j=1}^d$ are the eigenvalues of H and they can be used as an orthogonal basis, we expand $\delta v$ by this basis. This means

$$\begin{array}{c}\hfill \delta v_j=\sum _{k=1}^d{c}_{jk}{v}_k,\end{array}$$

(A88)

where $c_{jk}=\delta v_j^{\top }{v}_k$. By multiplying $v_i$ to Equation (A87) where $i\ne j$ from left-hand side and neglecting high-order terms, we have

$$\begin{array}{c}\hfill c_{ji}=\frac{{\lambda }_j}{{\lambda }_j-{\lambda }_i}\left(v_i^{\top }\alpha J{v}_i\right).\end{array}$$

(A89)

Next, Then by multiplying $v_j$ to Equation (A87) from left-hand side, we have

$$\begin{array}{c}\hfill v_{j}H\left(\alpha J\right)H\delta v_j=\left(\delta {\lambda }_j\right)({\lambda }_j+{\lambda }_j{v}_j^{\top }\delta v_j)\end{array}$$

(A90)

Then by substituting $\delta v_j$ with coefficient Equation (A89), we have

$$\begin{array}{cc}\hfill \delta {\lambda }_j& ={\alpha }^2\sum _{i\ne j}^d\frac{{\lambda }_i{\lambda }_j{\left(v_i^{\top }J{v}_j\right)}^2}{{\lambda }_i-{\lambda }_j}+\mathcal{O}\left({\alpha }^3\right)\hfill \end{array}$$

(A91)

This concludes the proof.

Appendix I.1. Proof of Lemma 2

First, we introduce the notations. We express the random variables of S-PLD as $Y_t^{\otimes N}$. We express the measure induced by S-PLD as ${\mu }_t^{\otimes N}\left(\alpha \right)$, which uses the $\alpha J$ as an interaction term. Thus, we express the measure of PLD as ${\mu }_{kh}^{\otimes N}\left(0\right)$, we can decompose the measure as marginals. We also denote the marginal measure of S-PLD for $Y_t^{\left(n\right)}$${\nu }_t^{\left(n\right)}\left(\alpha \right)$. Please note that initial distribution is ${\mu }_0^{\otimes N}$ and its marginals are ${\mu }_0$ as defined in Assumption 4.

Please note that the marginal measure of PLD is the same as those of LD if the initial measures are all the same, thus each marginal satisfy the Poincaré constant $m_0$. This is also the result of the tensorization property of the spectral gap (Proposition 4.3.1 in Bakry et al. [19]).

As for the initial condition, from the fact that ${\chi }^2$ divergence is the special case of Renyi divergence ($\alpha =4$), and from the tensorization property of the Renyi divergence (see Theorem 28 in [45]), we have

$$\begin{array}{c}\hfill {\chi }^2({\mu }_t^{\otimes N}\left(0\right),{\pi }^{\otimes N})\le e^{-2{\beta }^{-1}{m}_{0}t}{\chi }^2({\mu }_0^{\otimes N},{\pi }^{\otimes N})=\sum _{n=1}^N{e}^{-2{\beta }^{-1}{m}_{0}t}{\chi }^2({\mu }_0,\pi ).\end{array}$$

(A92)

Then we have

$$\begin{array}{c}\hfill {\chi }^2({\mu }_t^{\otimes N}\left(0\right),{\pi }^{\otimes N})\le e^{-2{\beta }^{-1}{m}_{0}t}{\chi }^2({\mu }_0^{\otimes N},{\pi }^{\otimes N})=N{e}^{-2{\beta }^{-1}{m}_{0}t}{\chi }^2({\mu }_0,\pi ).\end{array}$$

(A93)

If the skew acceleration is applied, from the same discussion as S-LD (see Appendix C.1), S-PLD has the Poincaré constant which is larger than $m_0$. We express it as $m(\alpha ,N)(\ge m_0)$. Then we have

$$\begin{array}{c}\hfill {\chi }^2({\mu }_t^{\otimes N}\left(\alpha \right),{\pi }^{\otimes N})\le N{e}^{-2{\beta }^{-1}m(\alpha ,N)t}{\chi }^2({\mu }_0,\pi ).\end{array}$$

(A94)

At first, since there exists a constant N in the convergence bound, this bound seems not useful. However, as we discussed below, when we bound the bias or variance, these bound is meaningful. For example, let us consider approximating the true expectation $\int f\left(x\right)d\pi \left(x\right)$ by the ensemble samples $\frac{1}{N}{\sum }_{n=1}^{N}f\left(X_t^{\left(n\right)}\right)$. Then we are interested in bounding the error

$$\begin{array}{c}\hfill \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|.\end{array}$$

(A95)

For this purpose, we can bound this by 2-Wasserstein distance as

$$\begin{array}{c}\hfill \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|,\le \frac{L_f}{\sqrt{N}}{W}_2({\mu }_{kh}^{\otimes N}\left(\alpha \right),{\pi }^{\otimes N})\end{array}$$

(A96)

where we assumed that f shows $L_f$ lipschitzness and used the fact that $\frac{1}{N}{\sum }_{n=1}^{N}f\left(x^{\left(n\right)}\right)$ shows $L_f/\sqrt{N}$ lipschitzness.

To bound the distance, we use the basic relation

$$\begin{array}{c}\hfill W_2^2({\nu }_{kh}\left(\alpha \right),{\pi }^{\otimes N})\le 2\frac{1}{m(\alpha ,N)}{\chi }^2({\mu }_{kh}^{\otimes N}\left(\alpha \right),{\pi }^{\otimes N}),\end{array}$$

(A97)

where $m(\alpha ,N)$ is the Poincaré constant. This is established by the definition of Wasserstein distance and ${\chi }^2$-divergence, see [46] for the detail. Then combined with above relations, we obtain the bias bound of S-PLD as

$$\begin{array}{c}\hfill \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|\le L_f\sqrt{\frac{2}{m(\alpha ,N)}}{e}^{-{\beta }^{-1}m(\alpha ,N)kh}{\chi }^2{({\mu }_0,\pi )}^{1/2}.\end{array}$$

(A98)

In the same way, we obtain the bias bound of PLD as

$$\begin{array}{c}\hfill \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|\le L_f\sqrt{\frac{2}{m_0}}{e}^{-{\beta }^{-1}{m}_{0}kh}{\chi }^2{({\nu }_0,\pi )}^{1/2}.\end{array}$$

(A99)

Thus, while the explicit dependency on N disappeared, but S-PLD shows faster convergence through the relation of $m(\alpha ,N)\ge m_0$. Moreover, if we use the skew matrices, which does not satisfy the equality condition, we have $m(\alpha ,N)>m_0$.

Appendix I.2. Proof for S-ULD

We can characterize the convergence rate almost in the same way as Appendix C.2. The derivation is the same above, thus we only show the result

$$\begin{array}{c}\hfill \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|\le L_f\sqrt{\frac{2}{m(\alpha ,N)}}\sqrt{\frac{1+\overline{ϵ}}{1-\overline{ϵ}}}{e}^{-{\lambda }_{\gamma }/2kh}{\chi }^2{({\nu }_0^0,\pi )}^{1/2}.\end{array}$$

(A100)

where $\overline{ϵ}$ and ${\lambda }_{\gamma }$ is given as follows.

$$\begin{array}{c}\hfill {\lambda }_{\gamma }=\frac{\Lambda (\gamma ,\overline{ϵ}min(\gamma ,{\gamma }^{-1}))}{1+\overline{ϵ}min(\gamma ,{\gamma }^{-1})},\end{array}$$

(A101)

and

$$\begin{array}{cc}\hfill & \Lambda (\gamma ,ϵ)=\frac{\gamma {\Sigma }^{-1}-\frac{1}{1+\frac{m_0{\Sigma }^{-1}}{\beta }}}{2}-\frac{1}{2}\sqrt{{(S_{--}-S_{++})}^2+{\left(S_{-+}\right)}^2},\hfill \end{array}$$

(A102)

$$\begin{array}{cc}\hfill & S_{--}=ϵ{\lambda }_{ham},\hfill \end{array}$$

(A103)

$$\begin{array}{cc}\hfill & S_{-+}=-ϵ(R_{ham}+\gamma {\Sigma }^{-1}/2),\hfill \end{array}$$

(A104)

$$\begin{array}{cc}\hfill & S_{++}=\gamma {\Sigma }^{-1}-ϵ,\hfill \end{array}$$

(A105)

$$\begin{array}{cc}\hfill & {\lambda }_{ham}=1-{\left(1+\frac{m(\alpha ,N){\Sigma }^{-1}}{\beta }\right)}^{-1},\hfill \end{array}$$

(A106)

$$\begin{array}{cc}\hfill & ϵ=\overline{ϵ}min(\gamma ,{\gamma }^{-1}),\hfill \end{array}$$

(A107)

where $\overline{ϵ}$ is arbitrary sufficiently small positive value such that $\Lambda (\gamma ,\overline{ϵ}min(\gamma ,{\gamma }^{-1}))>0$ is satisfies. and

$$\begin{array}{c}\hfill R_{ham}\le \sqrt{max\{M,2\}}.\end{array}$$

(A108)

Appendix J.1. Proof of Lemma A1

Proof. 

We prove this lemma using the Donsker–Varadhan representation of the relative entropy [48]. The relative entropy admits the dual representation as:

$$\begin{array}{c}\hfill \mathrm{KL}(P\left(X\right)\parallel Q\left(Y\right))=\underset{T:{\Omega }^{\otimes N}\to \mathbb{R}}{sup}{\mathbb{E}}_{P\left(X\right)}\left[T\right]-log{\mathbb{E}}_{Q\left(Y\right)}\left[e^T\right],\end{array}$$

(A136)

where supremum is taken over all function T of which the expectation of $e^T$ and T are finite. We then restrict the function class into a class $\mathcal{F}\left(T\right)=\left\{T\left(X\right)\right|\exists T_i:{\Omega }_i\to \mathbb{R},s.t.T\left(X\right)={\sum }_{i=1}^N{T}_i\left(X_i\right)\}$ where each expectation of $e^{T_i}$ and $T_i$ are finite. Then by definition,

$$\begin{array}{c}\hfill \mathrm{KL}(P\left(X\right)\parallel Q\left(Y\right))=\underset{T:\Omega \to \mathbb{R}}{sup}{\mathbb{E}}_{P\left(X\right)}\left[T\right]-log{\mathbb{E}}_{Q\left(Y\right)}\left[e^T\right]\ge \underset{T\in \mathcal{F}}{sup}{\mathbb{E}}_{P\left(X\right)}\left[\sum _i{T}_i\right]-log{\mathbb{E}}_{Q\left(Y\right)}\left[e^{{\sum }_i{T}_i}\right].\end{array}$$

(A137)

Then we have

$$\begin{array}{cc}\hfill \mathrm{KL}\left(P\right(X)\parallel Q(Y\left)\right)& \ge \underset{T\in \mathcal{F}}{sup}\sum _i{\mathbb{E}}_{P_i\left(X_i\right)}\left[T_i\right]-log\prod _i{\mathbb{E}}_{Q_i\left(Y_i\right)}\left[e^{T_i}\right]\hfill \\ \hfill & =\underset{T\in \mathcal{F}}{sup}\sum _i\left({\mathbb{E}}_{P_i\left(X_i\right)}\left[T_i\right]-log{\mathbb{E}}_{Q_i\left(Y_i\right)}\left[e^{T_i}\right]\right)\hfill \\ \hfill & =\sum _i\underset{T_i:{\Omega }_i\to \mathbb{R}}{sup}{\mathbb{E}}_{P_i\left(X_i\right)}\left[T_i\right]-log{\mathbb{E}}_{Q_i\left(Y_i\right)}\left[e^{T_i}\right]\hfill \\ \hfill & =\sum _{i=1}^N\mathrm{KL}(P_i\left(X_i\right)\parallel Q_i\left(Y_i\right)).\hfill \end{array}$$

(A138)

□

Appendix M.1. Estimation of the Logarithmic Sobolev Constant

In this section, we estimate the logarithmic Sobolev constants using the technique of restricted logarithmic Sobolev inequality, which was introduced in [49].

The technique of [49] estimates the constant of the logarithmic Sobolev inequality as follows. Assume that $\pi$ on ${\mathbb{R}}^d$ with $\mathcal{L}$ satisfies the Poincaré inequality with constant m. Then, for any function u on ${\mathbb{R}}^d$ that satisfies

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^d}ud\pi =0\mathrm{and}{\int }_{{\mathbb{R}}^d}{u}^{2}d\pi =1,\end{array}$$

(A175)

we find a constant b that satisfies

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^d}{u}^{2}ln{u}^{2}d\pi \le b{\int }_{{\mathbb{R}}^d}-u\mathcal{L}ud\pi .\end{array}$$

(A176)

Then the logarithmic constant is larger than $2{(b+\frac{3}{m})}^{-1}$. Thus, we only need to focus on the restricted function class to estimate a constant b. We slightly change the Lemma 3.2 of [49] that estimate the constant b in Equation (A176) to apply it in our setting. In Lemma 3.2 of [49], it was proved that if u on ${\mathbb{R}}^d$ satisfies the conditions in Equation (A175), then for any $t\in (0,1)$, we have

$$\begin{array}{cc}\hfill & {\int }_{{\mathbb{R}}^d}-u\mathcal{L}ud\pi -t\pi e^2{\int }_{{\mathbb{R}}^d}{u}^{2}ln{u}^{2}d\pi \ge (1-t)m+t\beta {\int }_{{\mathbb{R}}^d}(-\frac{1}{2}\mathcal{L}U\left(x\right)-\pi e^{2}U\left(x\right))u^{2}d\pi ,\hfill \end{array}$$

(A177)

where we assume that $\pi \propto e^{-\beta U\left(x\right)}$ satisfies the Poincaré inequality with constant m. If there exists a constant C such that

$$\begin{array}{c}\hfill -C\ge \beta {\int }_{{\mathbb{R}}^d}(-\frac{1}{2}\mathcal{L}U\left(x\right)-\pi e^{2}U\left(x\right))u^{2}d\pi >-\infty ,\end{array}$$

(A178)

then by setting $t=m/(m+|C\left|\right)$, we can show that

$$\begin{array}{cc}\hfill & {\int }_{{\mathbb{R}}^d}-u\mathcal{L}ud\pi -m/(m+|C\left|\right)\pi e^2{\int }_{{\mathbb{R}}^d}{u}^{2}ln{u}^{2}d\pi >0.\hfill \end{array}$$

(A179)

Thus, the constant b in Equation (A176) is $b=t=m/(m+|C\left|\right)$ and the logarithmic constant is $2(m/(m+\left|C\right|)+\frac{3}{m}{)}^{-1}$.

Thus, We analyze the constant C. The first term of the integral in Equation (A178) is lower-bounded bounded by

$$\begin{array}{c}\hfill -{\mathbb{E}}_{\pi }\left[\mathcal{L}U\left(x\right)u^2\right]\ge -|{\mathbb{E}}_{\pi }\left[U\left(x\right)\mathcal{L}U\left(x\right)\right]{|}^{1/2}|{\mathbb{E}}_{\pi }\left[u^2\mathcal{L}{u}^2\right]{|}^{1/2}\ge -2{\mathbb{E}}_{\pi }[\parallel \nabla U\left(x\right){{\parallel }^2]}^{1/2},\end{array}$$

(A180)

where we used the property of $\mathcal{L}$, see [19] for details. As for the second term, it is lower-bounded by

$$\begin{array}{cc}\hfill -|{\mathbb{E}}_{\pi }\left[U\left(x\right)u^2\right]\ge -\sqrt{|{\mathbb{E}}_{\pi }\left[U^2\left(x\right)u^2\right]}& \ge -\sqrt{\frac{1}{m}|{\mathbb{E}}_{\pi }\left[\right(U\left(x\right)\left|u\right|\left)\mathcal{L}\right(U\left(x\right)\left|u\right|\left)\right]|}\hfill \\ \hfill & \ge -\sqrt{\frac{8}{m}{\mathbb{E}}_{\pi }{[\parallel \nabla U\left(x\right)\parallel }^2{]}^{1/2}}.\hfill \end{array}$$

(A181)

Thus, by setting

$$\begin{array}{c}\hfill -C:={\mathbb{E}}_{\pi }{[\parallel \nabla U\left(x\right)\parallel ]}^{1/2}+\sqrt{\frac{8}{m_0}{\mathbb{E}}_{\pi }{[\parallel \nabla U\left(x\right)\parallel }^2{]}^{1/2}},\end{array}$$

(A182)

we can estimate the logarithmic constant as $2(m/(m+\left|C\right|)+\frac{3}{m}{)}^{-1}$.

In our setting, this is modified to

$$\begin{array}{c}\hfill \lambda (\alpha ,N)={\left(\frac{1}{{(1+\beta m(\alpha ,N)}^{-1}|C\left(m_0\right)\left|\right)2\pi e^2}+\frac{3}{2m(\alpha ,N)}\right)}^{-1}.\end{array}$$

(A183)

where

$$\begin{array}{c}\hfill -C\left(m_0\right):={\mathbb{E}}_{{\pi }^{\otimes N}}[\parallel \nabla U^{\otimes N}\left(x\right){\parallel ]}^{1/2}+\sqrt{\frac{8}{m_0}{\mathbb{E}}_{{\pi }^{\otimes N}}[\parallel \nabla U^{\otimes N}\left(x\right){{\parallel }^2]}^{1/2}}.\end{array}$$

(A184)

Finally, if we increase $m(\alpha ,N)$, $\lambda (\alpha ,N)$ increases. Thus, since $m(\alpha ,N)\ge m(\alpha =0,N)$, we obtain $\lambda (\alpha ,N)\ge \lambda (\alpha =0,N)$.

Appendix M.2. Computational Complexity

To derive the computational complexity, for simplicity, we assume that $\delta \le h$ and We also set ${\alpha }^2\le h$ for simplicity. This means that the variance of the stochastic gradient is small enough and we use small $\alpha$. Then the bias is

$$\begin{array}{cc}\hfill \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|& \le L_f\sqrt{\frac{2}{\lambda (\alpha ,N)}}\sqrt{e^{-\lambda (\alpha ,N)kh}\mathrm{KL}\left({\mu }_0\right|\pi )+\frac{C_3\left(\alpha \right)}{\lambda (\alpha ,N)}}\hfill \\ \hfill & \le L_f\sqrt{\frac{2}{\lambda (\alpha ,N)}}\left(\sqrt{e^{-\lambda (\alpha ,N)kh}\mathrm{KL}\left({\mu }_0\right|\pi )}+\sqrt{\frac{C_3\left(\alpha \right)}{\lambda (\alpha ,N)}}\right),\hfill \end{array}$$

(A185)

where

$$\begin{array}{cc}\hfill & C_3\left(\alpha \right):=h\left(2\frac{dM}{\beta }(m+4M)+8C_0^{\prime }({(1+h^{1/2})}^2+1)\right),\hfill \end{array}$$

(A186)

$$\begin{array}{cc}\hfill & C_0^{\prime }:=2\left(M^2\left({\kappa }_0+2\left(1\vee \frac{1}{m}\right)\left(b+2{(1+h^{1/2})}^2{B}^2+\frac{d}{\beta }\right)\right)+B^2\right).\hfill \end{array}$$

(A187)

Then we define

$$\begin{array}{c}\hfill C_3^{\prime }:=2\frac{dM}{\beta }(m+4M)+8C_0^{\prime }({(1+h^{1/2})}^2+1),\end{array}$$

(A188)

and use the step size that satisfies $h=\frac{\lambda (\alpha ,N)\xi }{2\sqrt{2}{C}_3^{\prime }{L}_f}$. Then when we use

$$\begin{array}{c}\hfill k\ge \frac{2}{\lambda (\alpha ,N)h}ln\frac{L_f}{\xi }\sqrt{\frac{\mathrm{KL}\left({\mu }_0\right|\pi )}{2\lambda (\alpha ,N)}},\end{array}$$

(A189)

we have

$$\begin{array}{cc}\hfill \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|& \le \frac{\xi }{2}+\frac{\xi }{2}\le \xi .\hfill \end{array}$$

(A190)