Accelerated Diffusion-Based Sampling by the Non-Reversible Dynamics with Skew-Symmetric Matrices

Abstract

Langevin dynamics (LD) has been extensively studied theoretically and practically as a basic sampling technique. Recently, the incorporation of non-reversible dynamics into LD is attracting attention because it accelerates the mixing speed of LD. Popular choices for non-reversible dynamics include underdamped Langevin dynamics (ULD), which uses second-order dynamics and perturbations with skew-symmetric matrices. Although ULD has been widely used in practice, the application of skew acceleration is limited although it is expected to show superior performance theoretically. Current work lacks a theoretical understanding of issues that are important to practitioners, including the selection criteria for skew-symmetric matrices, quantitative evaluations of acceleration, and the large memory cost of storing skew matrices. In this study, we theoretically and numerically clarify these problems by analyzing acceleration focusing on how the skew-symmetric matrix perturbs the Hessian matrix of potential functions. We also present a practical algorithm that accelerates the standard LD and ULD, which uses novel memory-efficient skew-symmetric matrices under parallel-chain Monte Carlo settings.

1. Introduction

Sampling is one of the most widely used techniques for the approximation of posterior distribution in Bayesian inference [1]. Markov Chain Monte Carlo (MCMC) is widely used to obtain samples. In MCMC, Langevin dynamics (LD) is a popular choice for sampling from high-dimensional distributions. Each sample in LD moves toward a gradient direction with added Gaussian noise. LD efficiently explore around a mode of a target distribution using the gradient information without being trapped by local minima thanks to added Gaussian noise. Many previous studies theoretically and numerically proved LD’s superior performance [2,3,4,5]. Since non-reversible dynamics generally improves mixing performance [6,7], research on introducing non-reversible dynamics to LD for better sampling performance is attracting attention [8].

There are two widely known non-reversible dynamics for LD. One is underdamped Langevin dynamics (ULD) [9], which uses second-order dynamics. The other introduces perturbation, which consists of multiplying the skew-symmetric matrix by a gradient [8]. Here, we refer to the matrix as skew matrices for simplicity and this perturbation technique as skew acceleration. Much research has been done on ULD theoretically [9,10,11] and ULD is widely used in practice, which is also known as stochastic gradient Hamilton Monte Carlo [12]. In contrast, the application of the skew acceleration for standard Bayesian models is quite limited even though it is expected to show superior performance theoretically [8].

For example, skew acceleration has been analyzed focusing on sampling from Gaussian distributions [13,14,15,16,17], although assuming Gaussian distributions in Bayesian models is restrictive in practice. A recent study [8] theoretically showed that skew acceleration accelerates the dynamics around the local minima and saddle points for non-convex functions. Another work [18] clarified that the skew acceleration theoretically and numerically improves mixing speed when used as interactions between chains in parallel sampling schemes for non-convex Bayesian models.

Compared to ULD, what seems to be lacking for skew acceleration is a theoretical understanding of issues that are important to practitioners. The most significant problem is that no theory exists for selecting skew matrices. In existing studies, introducing a skew matrix into LD results in equal or faster convergence, denoting that a bad choice of skew matrix results in no acceleration. Thus, choosing appropriate skew matrices is critical. Furthermore, although ULD’s acceleration has been analyzed quantitatively, existing studies have only analyzed skew acceleration qualitatively. Thus, it is difficult to justify the usefulness of skew acceleration in practice compared to ULD. Another issue is that introducing skew matrices requires a vast memory cost in many practical Bayesian models.

The purpose of this study is to solve these problems from theoretical and numerical viewpoints and establish a practical algorithm for skew acceleration. The following are the two major contributions of this work.

Our contribution 1: We present a convergence analysis of skew acceleration for standard Bayesian model settings, including non-convex potential functions using Poincaré constants [19]. The major advantage of Poincaré constants is that we can analyze skew acceleration through a Hessian matrix and its eigenvalues and develop a practical theory about the selection of J and the quantitative assessment of skew acceleration.

Furthermore, we propose skew acceleration for ULD and present convergence analysis for the first time. Since ULD shows faster convergence than LD, combining skew acceleration with ULD is promising.

Our contribution 2: We develop a practical skew accelerated sampling algorithm for a parallel sampling setting with novel memory-efficient skew matrices. Since a naive implementation of skew acceleration requires a large memory cost to store skew matrices, memory-efficiency is critical in practice. We also present a non-asymptotic theoretical analysis for our algorithm in both LD and ULD settings under a stochastic gradient and Euler discretization. We clarify that introducing skew matrices accelerates the convergence of continuous dynamics, although it increases the discretization and stochastic gradient error. Then to the best of our knowledge, we propose the first algorithm that adaptively controls this trade-off using the empirical distribution of the parallel sampling scheme.

Finally, we verify our algorithm and theory in practical Bayesian problems and compare it with other sampling methods.

Notations: $I_d$ denotes a $d\times d$ identity matrix. Capital letters such as X represent random variables, and lowercase letters such as x represent non-random real values. ·, $\parallel ·\parallel$ and $|·|$ denote Euclidean inner products, distances and absolute values.

2. Preliminaries

In this section, we briefly introduce the basic settings of LD and non-reversible dynamics for the posterior distribution sampling in Bayesian inference.

2.1. LD and Stochastic Gradient LD

First, we introduce the notations and the basic settings of LD and stochastic gradient LD (SGLD), which is a practical extension of LD. Here $z_i$ denotes a data point in space $\mathbb{Z}$, $\left|\mathcal{Z}\right|$ denotes the total number of data points, and $x\in {\mathbb{R}}^d$ corresponds to the parameters of a given model, which we want to sample. Our goal is to sample from the target distribution with density $d\pi \left(x\right)\propto e^{-\beta U\left(x\right)}dx$, where potential function $U\left(x\right)$ is the summation of $u:{\mathbb{R}}^d\times \mathbb{Z}\to \mathbb{R}$, i.e., ${\displaystyle U\left(x\right)=\frac{1}{\left|\mathcal{Z}\right|}\sum _{i=1}^{\left|\mathcal{Z}\right|}u(x,z_i)}$. Function $u(·,·)$ is continuous and non-convex. The explicit assumptions made for it are discussed in Section 3.1. The SGLD algorithm [2,3] is given as a recursion:

$$\begin{array}{c}\hfill X_{k+1}=X_k-h\nabla \widehat{U}\left(X_k\right)+\sqrt{2h{\beta }^{-1}}{ϵ}_k,\end{array}$$

(1)

where $h\in {\mathbb{R}}^{+}$ is a step size, ${ϵ}_k\in {\mathbb{R}}^d$ is a standard Gaussian random vector, $\beta$ is a temperature parameter of $\pi$, and $\nabla \widehat{U}\left(X_k\right)$ is a conditionally unbiased estimator of true gradient $\nabla U\left(X_k\right)$. This unbiased estimate of the true gradient is suitable for large-scale data set since we can use not the full gradient, but a stochastic version obtained through a randomly chosen subset of data at each time step. This means that we can reduce the computational cost to calculate the gradient at each time step.

The discrete time Markov process in Equation (1) is the discretization of the continuous-time LD [2]:

$$\begin{array}{c}\hfill d{X}_t=-\nabla U\left(X_t\right)dt+\sqrt{2{\beta }^{-1}}d{w}_t,\end{array}$$

(2)

where $w_t$ denotes the standard Brownian motion in ${\mathbb{R}}^d$. The stationary measure of Equation (2) is $d\pi \left(x\right)\propto e^{-\beta U\left(x\right)}dx$.

2.2. Poincaré Inequality and Convergence Speed

In sampling, we are interested in the convergence speed to the stationary measure. The speed is often characterized by the the generator associated with Equation (2) and defined as:

$$\begin{array}{cc}\hfill \mathcal{L}f\left(X_t\right):& =\underset{s\to 0^{+}}{lim}\frac{\mathbb{E}\left(f\left(X_{t+s}\right)\right|X_t)-f\left(X_t\right)}{s}\hfill \\ \hfill & =\left(-\nabla U\left(X_t\right)·\nabla +{\beta }^{-1}\Delta \right)f\left(X_t\right),\hfill \end{array}$$

(3)

where $\Delta$ denotes a standard Laplacian on ${\mathbb{R}}^d$ and $f\in \mathcal{D}\left(\mathcal{L}\right)$ and $\mathcal{D}\left(\mathcal{L}\right)\subset {\mathrm{L}}^2\left(\pi \right)$ denote the $\mathcal{L}$ domain. This $-\mathcal{L}$ is a self-adjoint operator, which has only discrete spectrums (eigenvalues). $\pi$ with $\mathcal{L}$ has a spectral gap if the smallest eigenvalue of $-\mathcal{L}$ (other than 0) is positive. We refer to it as ${\rho }_0(>0)$. This spectral gap is closely related to Poincaré inequality. Internal energy is defined:

$$\begin{array}{c}\hfill \mathcal{E}\left(f\right):=-{\int }_{{\mathbb{R}}^d}f\mathcal{L}fd\pi .\end{array}$$

(4)

Please note that $\mathcal{E}\left(f\right)>0$ is satisfied. Then $\pi$ with $\mathcal{L}$ satisfies the Poincaré inequality with constant c, if for any $f\in \mathcal{D}\left(\mathcal{L}\right)$, $\pi$ with $\mathcal{L}$ satisfies:

$$\begin{array}{c}\hfill \int f^{2}d\pi -{\left(\int fd\pi \right)}^2\le c\mathcal{E}\left(f\right).\end{array}$$

(5)

The spectral gap characterizes this constant $c\le \frac{1}{{\rho }_0}$, which holds (see Appendix A.2 for details). We refer to best constant c as the Poincaré constant [19]. For notational simplicity, we define $m_0:=\frac{1}{c}$ and refer to this $m_0$ as the Poincaré constant.

In sampling, crucially, Poincaré inequality dominates the convergence speed in ${\chi }^2$ divergence:

$$\begin{array}{c}\hfill \int {\left(\frac{d{\mu }_t}{d\pi }-1\right)}^{2}d\pi :={\chi }^2({\mu }_t\parallel \pi )\le e^{-\frac{2m_0}{\beta }t}{\chi }^2({\mu }_0\parallel \pi ),\end{array}$$

(6)

where ${\mu }_t$ denotes the measure at time t induced by Equation (2) and ${\mu }_0$ is the initial measure (see Appendix A.3 for details). Thus, the larger Poincaré constant $m_0$ is, the faster convergence we have.

2.3. Non-Reversible Dynamics

In this section, we introduce the non-reversible dynamics. $\pi$ with $\mathcal{L}$ is reversible if for any test function $f,g\in \mathcal{D}\left(\mathcal{L}\right)$, $\pi$ with $\mathcal{L}$ satisfies

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^d}f\mathcal{L}gd\pi ={\int }_{{\mathbb{R}}^d}g\mathcal{L}fd\pi .\end{array}$$

(7)

If this is not satisfied, $\pi$ with $\mathcal{L}$ is non-reversible [19].

We introduce two non-reversible dynamics for LD. The first is ULD, which is given as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & d{X}_t={\Sigma }^{-1}{V}_{t}dt,\hfill \\ \hfill & d{V}_t=-\nabla U\left(X_t\right)dt-\gamma {\Sigma }^{-1}{V}_{t}dt+\sqrt{2\gamma {\beta }^{-1}}d{w}_t,\hfill \end{array}\end{array}$$

(8)

where $V\in {\mathbb{R}}^d$ is an auxiliary random variable, $\gamma \in \mathbb{R}$ is a positive constant, and $\Sigma$ is the variance of the stationary distribution of auxiliary random variable V. The stationary distribution is $\tilde{\pi }:=\pi \otimes \mathcal{N}(0,\Sigma )\propto e^{-\beta U\left(x\right)-\frac{1}{2}{\Sigma }^{-1}{\parallel v\parallel }^2}$, where $\mathcal{N}$ denotes a Gaussian distribution. The superior performance of ULD compared with LD has been studied rigorously [9,10,11]. ULD’s convergence speed is also characterized by the Poincaré constant [20]. In practice, we use discretization and the stochastic gradient for ULD, which is called the stochastic gradient Hamilton Monte Carlo (SGHMC) [10]. The second non-reversible dynamics is the skew acceleration given as

$$\begin{array}{c}\hfill d{X}_t=-(I+\alpha J)\nabla U\left(X_t\right)dt+\sqrt{2{\beta }^{-1}}d{w}_t,\end{array}$$

(9)

where J is a real value skew matrix and $\alpha \in {\mathbb{R}}^{+}$ is a positive constant. We call this dynamics S-LD. The stationary distribution of S-LD is still $\pi$, and S-LD shows faster convergence and smaller asymptotic variance [13,14,15,18].

3. Theoretical Analysis of Skew Acceleration

In this section, we present a theoretical analysis of skew acceleration in LD and ULD in standard Bayesian settings. We analyze acceleration through the Poincaré constant and connect it with the eigenvalues of the Hessian matrix, which allows us to obtain a practical criterion to choose skew matrices and quantitatively evaluate acceleration. We focus on a setting where a continuous SDE and a full gradient of the potential function is used in this section. The discretized SDE and stochastic gradient are discussed in Section 4.

3.1. Acceleration Characterization by the Poincaré Constant

First, we introduce the same four assumptions as a previous work [2], which showed the existence of the Poincaré constant about $m_0$ for LD (see Appendix C for details).

Assumption 1. 

(Upper bound of the potential function at the origin) Function u takes nonnegative real values and is twice continuously differentiable on ${\mathbb{R}}^d$, and constants A and B exist such that for all $z\in \mathbb{Z}$,

$$\begin{array}{c}\hfill \left|u\right(0,z\left)\right|\le A,\parallel \nabla u(0,z)\parallel \le B.\end{array}$$

(10)

Assumption 2. 

(Smoothness) Function u has Lipschitz continuous gradients; for all $z\in \mathbb{Z}$, positive constant M exists for all $x,y\in {\mathbb{R}}^d$,

$$\begin{array}{c}\hfill \parallel \nabla u(x,z)-\nabla u(y,z)\parallel \le M\parallel x-y\parallel .\end{array}$$

(11)

Assumption 3. 

(Dissipative condition) Function u satisfies the (m,b)-dissipative condition for all $z\in \mathbb{Z}$; for all $x\in {\mathbb{R}}^d$, $m>0$ and $b\ge 0$ exist such that

$$\begin{array}{c}\hfill -x·\nabla u(x,z)\le -{m\parallel x\parallel }^2+b.\end{array}$$

(12)

Assumption 4. 

(Initial condition) Initial probability distribution ${\mu }_0$ of $X_0$ has a bounded and strictly positive density $p_0$, and for all $x\in {\mathbb{R}}^d$,

$$\begin{array}{c}\hfill {\kappa }_0:=log{\int }_{{\mathbb{R}}^d}{e}^{{\parallel x\parallel }^2}{p}_0\left(x\right)dx<\infty .\end{array}$$

(13)

Please note that these assumptions allow us to consider the non-convex potential functions, which are common in practical Bayesian models. Furthermore, we make the following assumption about J.

Assumption 5. 

The operator norm of J is bounded:

$$\begin{array}{c}\hfill {\parallel J\parallel }_2\le 1.\end{array}$$

(14)

This means that the largest singular value of J is below 1.

Under these assumptions, we present the convergence behavior of skew acceleration using the Poincaré constant. First, we present the following S-LD result.

Theorem 1.

Under Assumptions 1–5, the S-LD of Equation (9) has exponential convergence,

$$\begin{array}{c}\hfill {\chi }^2({\mu }_t^{\alpha }\parallel \pi )\le e^{-\frac{2m\left(\alpha \right)}{\beta }t}{\chi }^2({\mu }_0\parallel \pi ),\end{array}$$

(15)

where ${\mu }_t^{\alpha }$ is the measure at time t induced by S-LD and $m\left(\alpha \right)$ is the Poincaré constant of S-LD defined by its generator

$$\begin{array}{c}\hfill {\mathcal{L}}_{\alpha }f\left(x\right):=\left(-(I+\alpha J)\nabla U\left(x\right)·\nabla +{\beta }^{-1}\Delta \right)f\left(x\right).\end{array}$$

(16)

Furthermore, $m\left(\alpha \right)$ satisfies $m\left(\alpha \right)\ge m_0$.

The proof is shown in Appendix C. This theorem states that introducing the skew matrices accelerates the convergence of LD by improving the convergence rate from $m_0$ to $m\left(\alpha \right)$. Although [18] obtained a similar result, we used the Poincaré constant and derived an explicit criterion when $m\left(\alpha \right)=m_0$ holds, as we discuss below.

Next, we also introduce skew acceleration in ULD. Since ULD shows faster convergence than LD in standard Bayesian settings [10,11], it is promising to combine skew acceleration with ULD to obtain a more efficient sampling algorithm. For that purpose, we propose the following SDE:

$$\begin{array}{cc}\hfill & d{X}_t={\Sigma }^{-1}{V}_{t}dt+{\alpha }_1{J}_1\nabla U\left(X_t\right)dt,\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & d{V}_t=-\nabla U\left(X_t\right)dt-\gamma ({\Sigma }^{-1}+{\alpha }_2{J}_2)V_{t}dt+\sqrt{2\gamma {\beta }^{-1}}d{w}_t,\hfill \end{array}$$

(18)

where $J_1$ and $J_2$ are real value skew matrices and ${\alpha }_1$ and ${\alpha }_2$ are positive constants. We assume that $J_1$ and $J_2$ satisfy Assumption 5. We refer to this method as skew underdamped Langevin dynamics (S-ULD) whose stationary distribution is $\tilde{\pi }=\pi \otimes \mathcal{N}(0,\Sigma )\propto e^{-\beta U\left(x\right)-\frac{1}{2}{\Sigma }^{-1}{\parallel v\parallel }^2}$. See Appendix B for details, which include discussions on other combinations of skew matrices. As for S-ULD, we need an additional assumption about the initial condition of $V_0$:

Assumption 6. 

(Initial condition) Initial probability distribution ${\mu }_0(x,v)$ of $(X_0,V_0)$ has a bounded and strictly positive density $p_0$ that satisfies,

$$\begin{array}{c}\hfill {\kappa }_0:=log{\int }_{{\mathbb{R}}^{2d}}{e}^{{\parallel x\parallel }^2+{\parallel v\parallel }^2}{p}_0\left((x,v)\right)dxdv<\infty .\end{array}$$

(19)

We then provide the following convergence theorem that resembles S-LD.

Theorem 2.

Under Assumptions 1–3, 5, 6, S-ULD has exponential convergence in ${\chi }^2$ divergence and its convergence rate is also characterized by $m\left(\alpha \right)$ as defined in Theorem 1. S-ULD’s convergence equals or exceeds ULD, of which convergence rate is characterized by $m_0$.

See Appendix C.2 for details. From these theorems, we confirmed that skew acceleration is effective in both S-LD and S-ULD, and the convergence speed is characterized by Poincaré constant $m\left(\alpha \right)$ defined by Equation (16).

3.2. Skew Acceleration from the Hessian Matrix

Our goal is to clarify what choices of J induce $m\left(\alpha \right)>m_0$, which leads to acceleration. Therefore, we discuss how Poincaré constant $m\left(\alpha \right)$ is connected to the eigenvalues and eigenvectors of the perturbed Hessian matrix $(I+\alpha J){\nabla }^{2}U\left(x\right)$. Next, we introduce the notations. We express the Hessian of $U\left(x\right)$ as $H\left(x\right)$ and the perturbed Hessian matrix as $H^{\prime }\left(x\right):=(I+\alpha J)H\left(x\right)$. Please note that H is a real symmetric matrix, which has real eigenvalues and diagonalizable. On the other hand, since $H^{\prime }$ is not symmetric, it has complex eigenvalues, although diagonalization is not assured (see Appendix E). We express pairs of eigenvectors and eigenvalues of $H^{\prime }\left(x\right)$ as ${\left\{(v_i^{\alpha }\left(x\right),{\lambda }_i^{\alpha }\left(x\right))\right\}}_{i=1}^d$, which are ordered as $\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right))\le \cdots \le \mathrm{Re}\left({\lambda }_d^{\alpha }\left(x\right)\right)$. Here, $\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ expresses the real part of complex value ${\lambda }_1^{\alpha }$ and $\mathrm{Im}$ denotes the imaginary part. We express those of $H\left(x\right)$ as ${\left\{(v_i^0\left(x\right),{\lambda }_i^0\left(x\right))\right\}}_{i=1}^d$ and order them as ${\lambda }_1^0\left(x\right)\le \cdots \le {\lambda }_d^0\left(x\right)$.

3.2.1. Strongly Convex Potential Function

Assume that U is an m-strongly convex function, where for all $x\in {\mathbb{R}}^d$, $m\le {\lambda }_1^0\left(x\right)$ holds. Poincaré constant $m_0$ of LD satisfies $m_0=m$ [19]. For the skew acceleration, since Poincaré constant satisfies $m\left(\alpha \right)=m^{\prime }\left(\alpha \right)$, where $m^{\prime }\left(\alpha \right)$ is the best constant that satisfies, for all x, $m^{\prime }\left(\alpha \right)\le \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ (see Appendix D.1). Therefore, studying the Poincaré constant is equivalent to studying the smallest (real part of the) eigenvalue of the Hessian matrix. Thus, the relation between ${\lambda }_1^0\left(x\right)$ and $\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ must be studied. The following theorem describes how the skew matrices change the smallest eigenvalue.

Theorem 3.

For all $x\in {\mathbb{R}}^d$, the real parts of the eigenvalues of $H^{\prime }$ satisfy

$$\begin{array}{c}\hfill m\le {\lambda }_1^0\left(x\right)\le \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)\le \cdots \le \mathrm{Re}\left({\lambda }_d^{\alpha }\left(x\right)\right)\le {\lambda }_d^0\left(x\right).\end{array}$$

(20)

The condition of ${\lambda }_1^0\left(x\right)=\mathrm{Re}\left({\lambda }_1^{\alpha }\right(x\left)\right)$ is shown in Remark 1.

Remark 1.

Denote the set of the eigenvectors of eigenvalue ${\lambda }_1^0\left(x\right)$ as $V_1^0$. If $V_1^0=\left\{v\right\}$ and $Jv=0$, then ${\lambda }_1^0\left(x\right)=\mathrm{Re}\left({\lambda }_1^{\alpha }\right(x\left)\right)$ holds. If the cardinality of set $V_1^0$ is larger than 1, and vectors $v,v^{\prime }\in V_1^0$ exist, such that ${\lambda }_1^0\alpha Jv=\left(\mathrm{Im}\left({\lambda }_1^{\alpha }\right)\right)v^{\prime }$ and ${\lambda }_1^0\alpha J{v}^{\prime }=-\left(\mathrm{Im}\left({\lambda }_1^{\alpha }\right)\right)v$, then ${\lambda }_1^0\left(x\right)=\mathrm{Re}\left({\lambda }_1^{\alpha }\right(x\left)\right)$ holds.

Refer to Appendix F for the proof. This is an extension of previous work [8,13]. If ${\lambda }_1^0\left(x\right)<\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ is satisfied for all x, we have $m_0<m\left(\alpha \right)$, i.e., acceleration occurs. We discuss how to construct J such that ${\lambda }_1^0\left(x\right)<\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ holds in Section 3.3.

3.2.2. Non-Convex Potential Function

The previous work [21] clarified that the Poincaré constant of the non-convex function is characterized by the negative eigenvalue of the saddle point. As shown in Figure 1, denote $x_1$ as the global minima, and $x_2$ is the local minima which has the second smallest value in $U\left(x\right)$. We express the saddle point with index one, i.e., there is only one negative eigenvalue at the point, between $x_1$ and $x_2$ as $x^{\ast }$. This means that the eigenvalues of $H\left(x^{\ast }\right)$ satisfies ${\lambda }_1^0\left(x^{\ast }\right)<0<{\lambda }_2^0\left(x^{\ast }\right)<\cdots <{\lambda }_d^0\left(x^{\ast }\right)$. Ref. [21] clarified that the saddle point $x^{\ast }$ characterizes the Poincaré constant as

$$\begin{array}{c}\hfill m_0^{-1}\propto \frac{1}{|{\lambda }_1\left(x^{\ast }\right)|}{e}^{\beta (U\left(x^{\ast }\right)-U\left(x_1\right)-U\left(x_2\right))}.\end{array}$$

(21)

When skew matrices are introduced, [8] clarified the following relation:

Theorem 4. 

([8]) ${\lambda }_1^{\alpha }\left(x^{\ast }\right)\le {\lambda }_1^0\left(x^{\ast }\right)<0$ and equality holds only if $J{v}_1^{\alpha }\left(x^{\ast }\right)=0$.

Note ${\lambda }_1^{\alpha }\left(x^{\ast }\right)$ is not a complex number. Thus, the skew acceleration reduces the negative eigenvalue and leads to a larger Poincaré constant (see Appendix D.2) and results in faster convergence.

In conclusion, introducing the skew matrix changes the Hessian’s eigenvalues and increase the Poincaré constant. If ${\lambda }_1^0\left(x\right)\ne \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ is satisfied, this leads to faster convergence for both convex and non-convex potential functions.

3.3. Choosing J

In this section, we present a method for choosing J that leads to ${\lambda }_1^0\left(x\right)\ne \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ to ensure the acceleration based on the equality conditions in Theorems 3 and 4. Combining these theorems, we obtain the following criterion:

Remark 2.

Given a point x, ${\lambda }_1^0\left(x\right)\ne \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ holds if either the following conditions are satisfied: $\left(i\right)$ when $V_1^0=\left\{v\right\}$, $Jv\ne 0$ is satisfied. $\left(ii\right)$ when $|V_1^0|>1$, $Jv\ne 0$ holds for any $v\in V_1^0$, and for any $v,v^{\prime }\in V_1^0$, ${\lambda }_1^0\alpha Jv=\left(\mathrm{Im}\left({\lambda }_1^{\alpha }\right)\right)v^{\prime }$ and ${\lambda }_1^0\alpha J{v}^{\prime }=-\left(\mathrm{Im}\left({\lambda }_1^{\alpha }\right)\right)v$ are not satisfied.

The first condition $\left(i\right)$ is easily satisfied if we choose J such that $\mathrm{Ker}J=\left\{0\right\}$. On the other hand, the second condition $\left(ii\right)$ is difficult to verify since H and its eigenvalues and eigenvectors generally depend on the current position of $X_t$. Instead of evaluating eigenvalues and eigenvectors of H and $H^{\prime }$ directly, we use the random matrix property shown in the next theorem.

Theorem 5.

Suppose the upper triangular entries of J follow a probability distribution that is absolutely continuous with respect to the Lebesgue measure. If $\mathrm{Ker}J=\left\{0\right\}$ is satisfied, then given a point $x\in {\mathbb{R}}^d$, ${\lambda }_1^0\left(x\right)\ne \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)$ holds with probability 1.

The proof is given in Appendix G.1. From this theorem, we simply generate J from some probability distribution, such as the Gaussian distribution. Then, we check whether $\mathrm{Ker}J=\left\{0\right\}$ holds. If $\mathrm{Ker}J=\left\{0\right\}$ does not hold, we generate a random matrix J again.

The above theorem is valid only at a given evaluation point x. We can extend the above theorem to all the points over the path of the discretized dynamics (see Appendix G.3). With this procedure, we can theoretically ensure that acceleration occurs with probability one for discretized dynamics.

3.4. Qualitative Evaluation of The Acceleration

So far, we have discussed skew acceleration qualitatively but not quantitatively. Although acceleration’s quantitative evaluation is critical for practical purposes, to the best of our knowledge, no existing work has addressed it. In this section, we present a formula that quantitatively assesses skew acceleration by analyzing the eigenvalues of the Hessian matrix.

Theorem 6.

With the identical notation as in Theorem 3, for all x, we have

$$\begin{array}{cc}\hfill & \mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)={\lambda }_1^0\left(x\right)+{\alpha }^2\sum _{k=2}^d\frac{{\lambda }_1^0\left(x\right){\lambda }_k^0\left(x\right){\left|v_k^0\left(x\right)J{v}_1^0\left(x\right)\right|}^2}{{\lambda }_k^0\left(x\right)-{\lambda }_1^0\left(x\right)}+\mathcal{O}\left({\alpha }^3\right).\hfill \end{array}$$

(22)

In particular, at saddle point $x^{\ast }$, we have

$$\begin{array}{c}\hfill {\lambda }_1^{\alpha }\left(x^{\ast }\right)={\lambda }_1^0\left(x^{\ast }\right)+{\alpha }^2\sum _{k=2}^d\frac{{\lambda }_1^0\left(x^{\ast }\right){\lambda }_k^0\left(x^{\ast }\right){\left|v_k^0\left(x^{\ast }\right)J{v}_1^0\left(x^{\ast }\right)\right|}^2}{{\lambda }_k^0\left(x^{\ast }\right)-{\lambda }_1^0\left(x^{\ast }\right)}+\mathcal{O}\left({\alpha }^3\right).\end{array}$$

(23)

The proofs are shown in Appendix H. When focusing on Equation (22), if $U\left(x\right)$ is a strongly convex function, since for all $k>1$, ${\lambda }_k\left(x\right)>{\lambda }_1\left(x\right)>0$ holds and the second term in Equation (22) is positive. From this, $\mathrm{Re}\left({\lambda }_1^{\alpha }\left(x\right)\right)>{\lambda }_1^0\left(x\right)$ holds. A similar relation holds for $\mathrm{Re}\left({\lambda }_d^{\alpha }\left(x\right)\right)$. In Equation (23), ${\lambda }_1^{\alpha }\left(x^{\ast }\right)<{\lambda }_1^0\left(x^{\ast }\right)<0$ holds. Thus, the changes of the Poincaré constants are proportional to ${\alpha }^2$. With these formulas, we can quantitatively evaluate the acceleration. We present numerical experiments to confirm our theoretical findings in Section 6.1.

4. Practical Algorithm for Skew Acceleration

In this section, we discuss skew acceleration in more practical settings compared to Section 3. First, we discuss the memory issue for storing J and the discretization of SDE and the stochastic gradient, which are widely used techniques in Bayesian inference. Finally, we present a practical algorithm for skew acceleration.

4.1. Memory Issue of Skew Acceleration and Ensemble Sampling

For d-dimensional Bayesian models, we need $\mathcal{O}\left(d^2\right)$ memory space to store skew matrices Js, and this is difficult for high-dimensional models. Instead of storing J, we can randomly generate Js at each time step following Theorem 5. However, we experimentally confirmed that using different Js at each step does not accelerate the convergence (see Section 6). Thus, we need to use a fixed J during the iterations.

As discussed below, we found that the previously proposed accelerated parallel sampling [18] can be a practical algorithm to resolve this memory issue. In that method, we simultaneously updated N samples of the model’s parameters with correlation. In such a parallel sampling scheme, a correlation exists among multiple Markov chains, it is more efficient than a naive parallel-chain MCMC, where the samples are independent. We express the n-th sample at time t as $X_t^{\left(n\right)}\in {\mathbb{R}}^d$ and the joint state of all samples at time t as $X_t^{\otimes N}:={(X_t^{\left(1\right)},\dots ,X_t^{\left(N\right)})}^{\top }\in {\mathbb{R}}^{dN}$. We express the joint stationary measure as ${\pi }^{\otimes N}:=\pi \otimes \cdots \otimes \pi \left(x^{\otimes N}\right)\propto e^{-\beta {\sum }_{i=1}^{N}U\left(x^{\left(i\right)}\right)}$. We express the sum of the potential function as $U^{\otimes N}:={\sum }_{i=1}^{N}U\left(x^{\left(i\right)}\right)$. We then consider the following dynamics:

$$\begin{array}{cc}\hfill & d{X}_t^{\otimes N}=-(I_{dN}+\alpha J)\nabla U^{\otimes N}\left(X_t^{\otimes N}\right)dt+\sqrt{2{\beta }^{-1}}d{w}_t,\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & \nabla U^{\otimes N}\left(X_t^{\otimes N}\right):={\left(\nabla U\left(X_t^{\left(1\right)}\right),\dots ,\nabla U\left(X_t^{\left(N\right)}\right)\right)}^{\top }.\hfill \end{array}$$

(25)

We call this dynamics skew parallel LD (S-PLD). N-independent parallel LD (PLD) is coupled with the skew matrix. Since each chain in PLD is independent of the other, the Poincaré constant of PLD is also $m_0$. Ref. [18] argued that the Poincaré constant of S-PLD, $m(\alpha ,N)$, satisfies $m(\alpha ,N)\ge m_0$. This means S-PLD shows faster convergence than PLD. As discussed in Section 3.2, these Poincaré constants are characterized by the smallest eigenvalue of the Hessian matrix ${\nabla }^2{U}^{\otimes N}\left(x^{\otimes N}\right)$ and $(I_{dN}+\alpha J){\nabla }^2{U}^{\otimes N}\left(x^{\otimes N}\right)$ where $x^{\otimes N}\in {\mathbb{R}}^{dN}$. We denote these smallest eigenvalues as ${\lambda }_1^0\left(x^{\otimes N}\right)$ and $\mathrm{Re}{\lambda }_1^{\alpha }\left(x^{\otimes N}\right)$. As discussed in Section 3.2, acceleration occurs if ${\lambda }_1^0\left(x^{\otimes N}\right)\ne \mathrm{Re}{\lambda }_1^{\alpha }\left(x^{\otimes N}\right)$ is satisfied.

In [18], they failed to specify the choice of J whose naive construction of J requires $\mathcal{O}\left(d^2{N}^2\right)$ memory cost. To reduce the memory cost, we propose the following skew matrix:

$$\begin{array}{c}\hfill J:=J_0\otimes I_d,\end{array}$$

(26)

where $J_0$ is a $N\times N$ skew matrix and ⊗ is a Kronecker product. We then have the following lemma:

Lemma 1.

If $J_0$ is generated based on Theorem 5 and $\mathrm{Ker}{J}_0=\left\{0\right\}$ is satisfied, then given a point $x^{\otimes N}$, J does not satisfy the equality condition in Theorems 3,4, which means ${\lambda }_1^0\left(x^{\otimes N}\right)\ne \mathrm{Re}{\lambda }_1^{\alpha }\left(x^{\otimes N}\right)$ with probability 1.

See Appendix G.2 for the proof. Thus, from this lemma, we only need to prepare and store $J_0$, which requires $\mathcal{O}\left(N^2\right)$ memory, which does not depend on d. In practical settings, this is a significant reduction of the memory size since the number of parallel chains is smaller than the dimension of models. Please note that we can ensure the acceleration with this J.

Lemma 2.

Under Assumptions 1–5, assume J satisfies the condition of Lemma 1. Then S-PLD shows

$$\begin{array}{c}\hfill {\chi }^2({\mu }_t^{\alpha ,\otimes N}\parallel {\pi }^{\otimes N})\le e^{-\frac{2m(\alpha ,N)}{\beta }t}{\chi }^2({\mu }_0^{\otimes N}\parallel {\pi }^{\otimes N}),\end{array}$$

(27)

where ${\mu }_t^{\alpha ,\otimes N}$ is the measure at time t induced by S-PLD, and ${\mu }_0^{\otimes N}$ is the initial measure defined as the product measure of ${\mu }_0$.

See Appendix I.1 for the proofs. Thus, combined with Lemma 2, S-PLD converges faster than PLD. We also considered the ensemble version of ULD (parallel ULD (PULD)) and its skew accelerated version:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & d{X}_t^{\otimes N}={\Sigma }^{-1}{V}_t^{\otimes N}dt+{\alpha }_1{J}_1\nabla U^{\otimes N}\left(X_t^{\otimes N}\right)dt,\hfill \\ \hfill & d{V}_t^{\otimes N}=-\nabla U^{\otimes N}\left(X_t^{\otimes N}\right)dt-\gamma ({\Sigma }^{-1}+{\alpha }_2{J}_2)V_t^{\otimes N}dt+\sqrt{2\gamma {\beta }^{-1}}d{w}_t,\hfill \end{array}\end{array}$$

(28)

where $J_1$ and $J_2\in {\mathbb{R}}^{dN\times dN}$ are real-valued skew-symmetric matrices, and ${\alpha }_1$ and ${\alpha }_2\in {\mathbb{R}}_{+}$ are positive constants and $V_t^{\otimes N}={\left(V_t^{\left(1\right)},\dots ,V_t^{\left(N\right)}\right)}^{\top }\in {\mathbb{R}}^{dN}$. We refer to this dynamics as skew PULD (S-PULD) whose faster convergence can be assured similar to Lemma 2 as shown in Appendix I.2.

4.2. Discussion of the Discretization of SDE and Stochastic Gradient and Practical Algorithm

In this section, we further consider practical settings for S-PLD and S-PULD. We discretize these continuous dynamics, e.g., by the Euler-Maruyama method, and approximate the gradient by the stochastic gradient. Although introducing skew matrices accelerates the convergence of continuous dynamics, it simultaneously increases the discretization and stochastic gradient error, resulting in a trade-off. We present a practical algorithm that controls this trade-off.

4.2.1. Trade-off Caused by Discretization and Stochastic Gradient

We consider the following discretization and stochastic gradient for S-PLD and S-PULD:

$$\begin{array}{c}\hfill X_{k+1}^{\otimes N}=X_k^{\otimes N}-h(I_{dN}+\alpha J)\nabla {\widehat{U}}^{\otimes N}\left(X_k^{\otimes N}\right)+\sqrt{2h{\beta }^{-1}}{ϵ}_k,\end{array}$$

(29)

and

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & X_{k+1}^{\otimes N}=X_k^{\otimes N}+{\Sigma }^{-1}{V}_k^{\otimes N}h+\alpha J\nabla {\widehat{U}}^{\otimes N}\left(X_k^{\otimes N}\right)h\hfill \\ \hfill & V_{k+1}^{\otimes N}=V_k^{\otimes N}-\nabla {\widehat{U}}^{\otimes N}\left(X_k^{\otimes N}\right)h-\gamma {\Sigma }^{-1}{V}_k^{\otimes N}h+\sqrt{2\gamma {\beta }^{-1}h}{ϵ}_k,\hfill \end{array}\end{array}$$

(30)

where ${ϵ}_k\in {\mathbb{R}}^{dN}$ is a standard Gaussian random vector. $\nabla {\widehat{U}}^{\otimes N}\left(X^{\otimes N}\right)$ is an unbiased estimator of the gradient $\nabla U^{\otimes N}\left(X^{\otimes N}\right)$. We refer to Equation (29) as skew-SGLD and Equation (30) as skew-SGHMC. For skew-SGHMC, we dropped $J_2$ of S-PULD to decrease the parameters, shown in Appendix B. Please note that skew-SGLD is the identical as the previous dynamics [18]. We introduce an assumption about the stochastic gradient:

Assumption 7. 

(Stochastic gradient) There exists a constant $\delta \in [0,1)$ such that

$$\begin{array}{c}\hfill \mathbb{E}[\parallel \nabla \widehat{U}\left(x\right)-\nabla U\left(x\right){\parallel }^2]\le 2\delta \left(M^2{\parallel x\parallel }^2+B^2\right).\end{array}$$

(31)

Given a test function f with $L_f$ lipschitzness, we approximate $\int fd\pi$ by skew-SGLD or skew-SGHMC, with estimator $\frac{1}{N}{\sum }_{n=1}^{N}f\left(X_k^{\left(n\right)}\right)$. The bias of skew-SGLD is upper-bounded as

Theorem 7.

Under Assumptions 1–7, for any $k\in \mathbb{N}$ and any $h\in (0,1\wedge \frac{m}{4M^2})$ obeying $kh\ge 1$ and $\beta m\ge 2$, we have

$$\begin{array}{cc}\hfill & \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|\le L_f(\underset{\left(i\right)}{\underbrace{C_1\left(\alpha \right)kh}}+\underset{\left(ii\right)}{\underbrace{C_2{e}^{-{\beta }^{-1}m(\alpha ,N)kh}}})\hfill \end{array}$$

(32)

and $C_1$ and $C_2$ depends on the constants of Assumptions 1–7, for the details see Appendix J.

We present a tighter bias bound in Section 4.3 under a stronger assumption. We can show a similar upper bound for the skew-SGHMC using the same proof strategy. This bound resembles of a previous one [18]; ours shows improved dependency on $kh$. The previous results of [18] are also limited to LD, not including skew-SGHMC.

Please note that $\left(i\right)$ corresponds to the discretization and stochastic gradient error and $\left(ii\right)$ corresponds to the convergence behavior of S-PLD, which is continuous dynamics. Since $C_1\left(\alpha \right)\ge C_1(\alpha =0)$, skew acceleration increases the discretization and stochastic gradient error. On the other hand, since $m(\alpha ,N)\ge m_0$, the convergence of the continuous dynamics is accelerated. Thus, skew acceleration causes a trade-off. When $\alpha$ is sufficiently small, we derive the explicit dependency of $\alpha$ for this trade-off from an asymptotic expansion. Using the quantitative evaluation of skew acceleration in Theorem 6, we obtain

$$\begin{array}{cc}\hfill & \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|\le \underset{\left(i\right)}{\underbrace{(d_1\alpha +d_2{\alpha }^2)kh}}-\underset{\left(ii\right)}{\underbrace{{\alpha }^2{d}_0{e}^{-{\beta }^{-1}{m}_{0}kh}}}+\mathcal{O}\left({\alpha }^3\right)+\mathrm{const},\hfill \end{array}$$

(33)

where $d_0$ to $d_2$ are positive constants obtained by the asymptotic expansion. See Appendix K for the details. In the above expression, $\left(i\right)$ and $\left(ii\right)$ correspond to $\left(i\right)$ and $\left(ii\right)$ of Equation (32). Thus, by choosing appropriate $\alpha$, we can control the trade-off.

4.2.2. Practical Algorithm Controlling the Trade-off

Since calculating the optimal $\alpha$ that minimizes Equation (33) at each step is computationally demanding, we adaptively tune the value of $\alpha$ by measuring the acceleration with kernelized Stein discrepancy (KSD) [22]. Our idea is to update samples under different $\alpha$ and $\alpha +\eta$, and compare KSD between the stationary and empirical distributions of these different interaction strengths. Here, $\eta \in {\mathbb{R}}^{+}$ is a small increment of $\alpha$. We denote the samples at the $(k+1)$th step, which is obtained by Equation (29) as $X_{k+1,\alpha }^{\otimes N}:=X_{k,\alpha }^{\otimes N}-h(I_{dN}+\alpha J)\nabla {\widehat{U}}^{\otimes N}\left(X_{k,\alpha }^{\otimes N}\right)+\sqrt{2h{\beta }^{-1}}{ϵ}_k$, (or (30) as ${\displaystyle X_{k+1,\alpha }^{\otimes N}:=X_k^{\otimes N}+{\Sigma }^{-1}{V}_k^{\otimes N}h+\alpha J\nabla {\widehat{U}}^{\otimes N}\left(X_k^{\otimes N}\right)h}$). We denote the samples, which are obtained by replacing the above $\alpha$ by $\alpha +\eta$, as $X_{k+1,\alpha +\eta }^{\otimes N}$. We denote the KSD between the measure of $X_{k+1,\alpha }^{\otimes N}$ and stationary measure $\pi$ as $KSD(k+1,\alpha )$ and estimate the differences of empirical KSD:

$$\begin{array}{cc}\hfill & \Delta :=\widehat{KSD}(k+1,\alpha )-\widehat{KSD}(k+1,\alpha +\eta ),\hfill \end{array}$$

(34)

where KSD is estimated by

$$\begin{array}{cc}\hfill & \widehat{KSD}(k,\alpha )=\frac{1}{N(N-1)}\sum _{i=1}^N{u}_q(X_{k,\alpha }^{\left(i\right)},X_{k,\alpha }^{\left(j\right)}),\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill & u_q(x,x^{\prime }):={\nabla }_{x}log\pi {\left(x\right)}^{\top }l(x,x^{\prime }){\nabla }_{x}log\pi \left(x^{\prime }\right)+{\nabla }_{x}log\pi {\left(x\right)}^{\top }{\nabla }_{x^{\prime }}l(x,x^{\prime })\hfill \\ \hfill & +{\nabla }_{x}l{(x,x^{\prime })}^{\top }{\nabla }_{x}log\pi +\mathrm{Tr}{\nabla }_{x,x^{\prime }}l(x,x^{\prime }),\hfill \end{array}$$

(36)

where l denotes a kernel and we use an RBF kernel. If $\Delta >0$, which indicates that the empirical distribution of $X_{k+1,\alpha +\eta }^{\otimes N}$ is closer to the stationary distribution than that of $X_{k+1,\alpha }^{\otimes N}$. Thus, we should increase the interaction strength from $\alpha$ to $\alpha +\eta$. If $\Delta <0$, we decrease it to $\alpha -\eta$. We also update $\eta$ to $c\eta$ where $c\in (0,1]$. The overall process is shown in Algorithm 1. Detailed discussions of the algorithm including how to select ${\alpha }_0,{\eta }_0$, and c are shown in Appendix L.

Algorithm 1 Tuning $\alpha$

Input: $X_k^{\otimes N},{\eta }_k,{\alpha }_k,c$ Output: ${\alpha }_{k+1},{\eta }_{k+1}$   1: Calculate $X_{k+1,{\alpha }_k}^{\otimes N}$ and $X_{k+1,{\alpha }_k+{\eta }_k}^{\otimes N}$.   2: Calculate $\Delta :=\widehat{KSD}(k+1,{\alpha }_k)-\widehat{KSD}(k+1,{\alpha }_k+{\eta }_k)$   3: if $\Delta >0$ then   4:  Update ${\alpha }_{k+1}={\alpha }_k+{\eta }_k$   5:  Update ${\eta }_{k+1}={\eta }_k$   6: else   7:  Update ${\alpha }_{k+1}=|{\alpha }_k-{\eta }_k|$   8:  Update ${\eta }_{k+1}=c{\eta }_k$   9: end if

Finally, we present Algorithm 2, which describes the whole process. We update the value of $\alpha$ once every $k^{\prime }$ step. Please note that its computational cost is not much larger than that of Equation (30). We only calculate the eigenvalues of J once, which requires $\mathcal{O}\left(N^3\right)$. The calculation of different KSDs is computationally inexpensive since we can re-use the gradient, which is the most computationally demanding part.

Algorithm 2 Proposed algorithm

Input: $X_0^{\otimes N},h,{\alpha }_0,\eta ,k^{\prime },K,c,(V_0^{\otimes N},\gamma ,{\Sigma }^{-1})$ Output: $X_K^{\otimes N}$   1: Make a $N\times N$ random matrix $J_0$ and check $\ker J_0=\left\{0\right\}$   2: Set $J=J_0\otimes I_d$   3: for $k=0$ to K do   4:  if $\lfloor \frac{k}{k^{\prime }}\rfloor =0$ then   5:   Update $\alpha$ by Algorithm 1   6:  end if   7:  Update $X_k^{\otimes N}$ by Equation (29) (for skew-SGLD)   8:  (Update $(X_k^{\otimes N},V_k^{\otimes N})$ by Equation (30) for skew-SGHMC)   9: end for

4.3. Refined Analysis for the Bias of Skew-SGLD

When using a constant step size for skew-SGLD, the bound in Theorem 7 is meaningless since the first term of Equation (32) will diverge. Here, following [23], we present a tighter bound for the bias of skew-SGLD under a stronger assumption.

Theorem 8.

Under Assumptions 1–7, for any $k\in \mathbb{N}$ and any $h\in (0,1\wedge \frac{\lambda (\alpha ,N)}{4\sqrt{2}{M}^2}\wedge \frac{m}{4M^2})$ obeying $kh\ge 1$ and $\beta m\ge 2$, we have

$$\begin{array}{cc}\hfill & \left|\mathbb{E}\frac{1}{N}\sum _{n=1}^{N}f\left(X_k^{\left(n\right)}\right)-{\int }_{{\mathbb{R}}^d}fd\pi \right|\le L_f\sqrt{\frac{2}{\lambda (\alpha ,N)}}\sqrt{e^{-\lambda (\alpha ,N)kh}\mathrm{KL}\left({\mu }_0\right|\pi )+\frac{C_3\left(\alpha \right)}{\lambda (\alpha ,N)}},\hfill \end{array}$$

(37)

where

$$\begin{array}{c}\hfill \lambda (\alpha ,N):={\left(\frac{1}{(1+m{(\alpha ,N)}^{-1}\beta C\left(m_0\right))2\pi e^2}+\frac{3}{2}m{(\alpha ,N)}^{-1}\right)}^{-1}\end{array}$$

(38)

and constants $C_3\left(\alpha \right)$ and $C\left(m_0\right)$ depend on the constants of Assumptions 1–7. Moreover, $\lambda (\alpha ,N)$ satisfies $\lambda (\alpha ,N)\ge \lambda (\alpha =0,N)$. For the details, see Appendix M.

Proof is shown in Appendix M. Please note that even if we use a constant step size for skew-SGLD, the bound in Theorem 8 will not diverge. Here we need the stronger assumption about a step size compared to Theorem 7. From Equation (37), the convergence behavior is characterized by $\lambda (\alpha ,N)$ and the bias bound become smaller when $\lambda (\alpha ,N)$ become larger. From the definition of $\lambda (\alpha ,N)$, the larger $m(\alpha ,N)$ is, the larger $\lambda (\alpha ,N)$ we obtain. Thus, as we had seen so far, introducing the skew matrices leads to the larger Poincaré constant, and thus, this leads to larger $\lambda (\alpha ,N)$.

Previous work [18] clarified that if $\alpha$ is sufficiently small, introducing skew matrices improves the Poincaré constant by a constant factor, which means that we have $m(\alpha ,N)-m_0\approx \mathcal{O}\left({\alpha }^2\right)$, where $\mathcal{O}\left({\alpha }^2\right)$ depends on the eigenvector and eigenvalues of the generator $\mathcal{L}$. On the other hand, from Theorem 8, for any $\xi >0$, to achieve the bias smaller than $\xi$, it suffice to run skew-SGLD at least for $k\ge \frac{2}{\lambda (\alpha ,N)h}ln\frac{L_f}{\xi }\sqrt{\frac{\mathrm{KL}\left({\mu }_0\right|\pi )}{2\lambda (\alpha ,N)}}$ iterations using the appropriate step size h and under the assumption that $\delta$ and $\alpha$ are small enough (see Appendix M.2 for details). Combined with these observations, introducing skew matrices into SGLD improves the computational complexity for a constant order. Our numerical experiments show that even constant improvement results in faster convergence in practical Bayesian models.

5. Related Work

In this section, we discuss the relationship between our method and other sampling methods.

5.1. Relation to Non-Reversible Methods

As we discussed in Section 1, our work extends the existing analysis of non-reversible dynamics [8,18] and presents a practical algorithm. Compared to those previous works, we focus on the practical setting of Bayesian sampling and derive the explicit condition about J for acceleration. We also derived a formula to quantitatively evaluate skew acceleration based on the asymptotic expansion of the eigenvalues of the perturbed Hessian matrix. A previous work [24], which derived the optimal skew matrices when the target distribution is Gaussian, requires $\mathcal{O}\left(d^3\right)$ computational cost to derive optimal skew matrices, and it is unclear whether it works for non-convex potential functions. On the other hand, our construction method for skew matrices is simple, computationally cheap, and can be applied to general Bayesian models.

Our work analyzes skew acceleration for ULD, which is more effective than LD in practical problems. Another work [8,18] only analyzed skew acceleration for LD. A previous work [17] combined a non-reversible drift term with ULD. Unlike our method, this work’s purpose was to reduce the asymptotic variance of the expectation of a test function and is mainly focusing on sampling from Gaussian distribution.

To the best of our knowledge, our work is the first to focus on the memory issue of skew acceleration and develop a memory-efficient skew matrix for ensemble sampling. Our work also presents an algorithm that controls the trade-off for the first time. Another work [18] identified the trade-off and handled it by cross-validation, which is computationally inefficient, unfortunately.

Finally, we point out an interesting connection between our skew-SGHMC and the magnetic HMC (M-HMC) [25]. M-HMC accelerates HMC’s mixing time by introducing a “magnetic” term into the Hamiltonian. That magnetic term is expressed by special skew matrices. Although a previous work [25] argued that M-HMC is numerically superior to a standard HMC, its theoretical property remains unclear. Thus, our work can analyze the theoretical behavior of magnetic HMC.

5.2. Relation to Ensemble Methods

Our proposed algorithm is based on ensemble sampling [26]. Ensemble sampling, in which multiple samples are simultaneously updated with interaction, has been attracting attention numerically and theoretically because of improvements in memory size, computational power, and parallel processing computation schemes [26]. There are successful, widely used ensemble methods, including SVGD [27] and SPOS [28], with which we compare our proposed method numerically in Section 6. Although both show numerically good performance, it is unclear how the interaction term theoretically accelerates the convergence since they are formulated as a McKean–Vlasov process, which is non-linear dynamics, complicating establishing a finite sample convergence rate. Our algorithm is an extension of another work [18], where the interaction was composed of a skew-acceleration term and can be rigorously analyzed. Compared to that previous work [18], we analyzed skew acceleration, focused on the Hessian matrix, and developed practical algorithms, as discussed in Section 4.2, and derived the explicit condition when acceleration occurs, which was unclear [18].

Another difference among SPOS, SVGD, and [18] is that they use first-order methods; our approach uses the second-order method. Little work has been done on ensemble sampling for second-order dynamics. Recently a second-order ensemble method was proposed [29], based on gradient flow analysis. Although its method showed good numerical performance, its theoretical property for finite samples remains unclear since it proposed a scheme as a finite sample approximation of the gradient flow. In contrast, our proposed method is a valid sampling scheme with a non-asymptotic guarantee.

6. Numerical Experiments

The purpose of our numerical experiments is to confirm the acceleration of our algorithm proposed in Section 4 in various commonly used Bayesian models including Gaussian distribution (toy data), latent Dirichlet allocation (LDA), and Bayesian neural net regression and classification (BNN). We compared our algorithm’s performance with other ensemble sampling methods: SVGD, SPOS, standard SGLD, and SGHMC. In all the experiments, the values and the error bars are the mean and the standard deviation of repeated trials. For all the experiments we set $\gamma =1$ and ${\Sigma }^{-1}=300$ for SGHMC and Skew-SGHMC. As for the hyperparameters of our proposed algorithm, the selection criterion is discussed in Appendix L.

6.1. Toy Data Experiment

The target distribution is the multivariate Gaussian distribution, $\pi =N(\mu ,\Omega )$ where we generated ${\Omega }^{-1}=A^{\top }A$ and each element of $A\in {\mathbb{R}}^{2d\times d}$ is drawn from the standard Gaussian distribution. The dimension of the target distribution is $d=50$, we approximate by 20 samples using the proposed ensemble methods. We tested these toy data because the LD for this target distribution is known as the Ornstein–Uhlenbeck process, and its theoretical properties have been studied extensively e.g., [30]. Thus, by studying the convergence behavior of these toy data, we can understand our proposed method more clearly.

First, we confirmed how the skew-symmetric matrix affects the eigenvalues of the Hessian matrix, as discussed in Section 3, where we only showed the asymptotic expansion for the smallest real part of the eigenvalues and saddle point. Here we can show a similar expansion for the largest real part:

$$\begin{array}{cc}\hfill & {\displaystyle \mathrm{Re}\left({\lambda }_{dN}^{\alpha }\right)={\lambda }_{dN}^0+{\alpha }^2\sum _{k=1}^{dN-1}\frac{{\lambda }_d{N}^0{\lambda }_k^0{\left|v_k^{0}J{v}_{dN}^0\right|}^2}{{\lambda }_k^0-{\lambda }_{dN}^0}+\mathcal{O}\left({\alpha }^3\right).}\hfill \end{array}$$

(39)

$\mathrm{Re}\left({\lambda }_{dN}^{\alpha }\right)\le {\lambda }_{dN}^{\alpha }$ holds.

Then we observed how the largest and smallest real parts of the eigenvalues of $(I+\alpha J){\Omega }^{-1}$ depend on $\alpha$. The results are shown in Figure 2, where we averaged 10 trials over a randomly made J with fixed A. The upper-left, upper-right, and lower figures show $\mathrm{Re}\left({\lambda }_1\left(\alpha \right)\right)$, $\mathrm{Re}\left({\lambda }_{dN}\left(\alpha \right)\right)$, and $\mathrm{Re}\left({\lambda }_1\left(\alpha \right)\right)/\mathrm{Re}\left({\lambda }_{dN}\left(\alpha \right)\right)$. These behaviors are consistent with Theorem 3. When $\alpha$ is small, its behavior is close to the quadratic function proved in Theorem 3.

Next, we observed the convergence behavior of skew-SGLD and skew-SGHMC. We measured the convergence by maximum mean discrepancy (MMD) [31] between the empirical and stationary distributions. For MMD, we used 2000 samples for the target distribution, and we used the Gaussian kernel whose bandwidth is set to the median distance of these 2000 samples. We used gradient descent (GD), with step size $h=1\times {10}^{-4}$. The results are shown in Figure 3. The proposed method shows faster convergence than naive parallel sampling, which is consistent with Table 2.

6.2. LDA Experiment

We tested with an LDA model using the ICML dataset [32] following the same setting as [33]. We used 20 samples for all the methods. Minibatch size is 100. We used step size $h=5\times {10}^{-4}$. First, we confirmed the effectiveness of our proposed Algorithm 1, which adaptively tunes $\alpha$ values. For that purpose, we compared the final performance obtained by our methods with a previous method [18], in which $\alpha$ is selected by cross-validation (CV). Here instead of CV, we just fixed $\alpha$ during the sampling and refer to it as fixed $\alpha$. We also tested the case when J is generated randomly at each step with fixed $\alpha$, as discussed in Section 4.1. We refer to it as random J. The results are shown in Figure 4 where skew-SGLD was used. We found that our method showed competitive performance with the best performance of fixed $\alpha$. For the computational cost, we used $k^{\prime }=2$ in Algorithm 2, and our method needed twice the wall clock time than each fixed $\alpha$. This means that our algorithm greatly reduces the total computational time since we tried more than two $\alpha$s in the fixed $\alpha$ for CV. We also found that since using different Js at each step did not accelerate the performance, we need to store and fix J during the sampling for acceleration. Next, we compared our method with other ensemble sampling schemes and observed the convergence speed. The result is shown in Figure 5. Skew-SGLD and skew-SGHMC outperformed SGLD and SGHMC, which is consistent with our theory.

6.3. BNN Regression and Classification

We tested with the BNN regression task using the UCI dataset [34], following a previous setting Liu and Wang [27]. We used one hidden layer neural network model with ReLU activation and 100 hidden units. We used 10 samples for all the methods. We used the minibatch size 100. We used step size $h=5\times {10}^{-5}$. The results are shown in

Table 1. Benchmark results on test RMSE for regression task.

Dataset	Avg. Test RMSE
	SVGD	SPOS	SGLD	Skew-SGLD	SGHMC	Skew-SGHMC
Concrete	5.709 ± 0.040	5.239 ± 0.199	5.009 ± 0.091	4.973 ± 0.057	4.949 ± 0.144	4.790 ± 0.081
Kin8nm	0.0731 ± 0.0006	0.0688 ± 0.0003	0.0693 ± 0.0006	0.0689 ± 0.0005	0.0687 ± 0.0001	0.0683 ± 0.0003
Energy	0.520 ± 0.060	0.456 ± 0.030	0.428 ± 0.045	0.412 ± 0.045	0.406 ± 0.019	0.403 ± 0.008
Bostonhousing	3.306 ± 0.005	3.107 ± 0.173	2.948 ± 0.084	2.930 ± 0.095	3.053 ± 0.093	2.986 ± 0.143
Winequality	0.619 ± 0.001	0.618 ± 0.007	0.641 ± 0.003	0.634 ± 0.004	0.614 ± 0.004	0.613 ± 0.004
PowerPlant	4.219 ± 0.012	4.160 ± 0.009	4.129 ± 0.002	4.118 ± 0.006	4.112 ± 0.009	4.105 ± 0.008
Yacht	0.475 ± 0.049	0.467 ± 0.110	0.464 ± 0.058	0.442 ± 0.046	0.464 ± 0.078	0.432 ± 0.051

and

Table 2. Benchmark results on test negative log likelihood for regression task.

Dataset	Avg. Test Negative Log Likelihood
	SVGD	SPOS	SGLD	Skew-SGLD	SGHMC	Skew-SGHMC
Concrete	−3.157 ± 0.008	−3.124 ± 0.025	−3.052 ± 0.009	−3.049 ± 0.012	−3.046 ± 0.025	−3.033 ± 0.021
Kin8nm	1.153 ± 0.0084	1.212 ± 0.008	1.223 ± 0.002	1.223 ± 0.005	1.230 ± 0.0015	1.235 ± 0.0025
Energy	−0.816 ± 0.102	−0.976 ± 0.079	−0.867 ± 0.056	−0.845 ± 0.021	−0.843 ± 0.045	−0.844 ± 0.041
Bostonhousing	−2.98 ± 0.000	−2.644 ± 0.027	−2.548 ± 0.016	−2.539 ± 0.002	−2.574 ± 0.019	−2.561 ± 0.017
Winequality	−1.012 ± 0.000	−0.959 ± 0.007	−0.976 ± 0.006	−0.968 ± 0.005	−0.941 ± 0.007	−0.938 ± 0.005
PowerPlant	−2.871 ± 0.004	−2.850 ± 0.004	−2.844 ± 0.002	−2.842 ± 0.001	−2.838 ± 0.004	−2.835 ± 0.003
Yacht	−1.184 ± 0.06	−1.372 ± 0.07	−1.077 ± 0.066	−1.078 ± 0.030	−1.083 ± 0.030	−1.079 ± 0.051

. We also tested on BNN classification task using the MNIST dataset. The result is shown in Figure 6. We used one hidden layer neural network model with ReLU activation and 100 hidden units. Batchsize is 500 and we set step size $h=5\times {10}^{-5}$. Our proposed methods outperformed other ensemble methods. Please note that skew-SGHMC and skew-SGLD consistently outperformed SGHMC and SGLD.

7. Conclusions

We studied skew acceleration for LD and ULD from practical viewpoints and concluded that the improved eigenvalues of the perturbed Hessian matrix caused acceleration and derived the explicit condition for acceleration. We described a novel ensemble sampling method, which couples multiple SGLD or SGHMC with memory-efficient skew matrices. We also proposed a practical algorithm that controls the trade-off of faster convergence and larger discretization and stochastic gradient error and numerically confirmed the effectiveness of our proposed algorithm.